2024-07-26T06:27:04.086 INFO:root:teuthology version: 0.0.1.dev379+g53ce146 2024-07-26T06:27:04.112 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-07-26T06:27:04.207 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818656 branch: wip-vshankar-testing-20240725.121331-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/fsx}} email: vshankar@redhat.com first_in_suite: false job_id: '7818656' kernel: client: sha1: distro flavor: default kdb: 1 sha1: 3e9255e907100f88049f320ca33d977e29155d12 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240725.121331-debug ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: global: ms die on skipped message: false mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.dir_split install: ceph: flavor: default sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240725.121331-debug sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5532 sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 sleep_before_teardown: 0 subset: 1001/1024 suite: fs suite_branch: wip-vshankar-testing-20240725.121331-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi042.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPGBs+kIALCUShVkk+houIPHtrMappOrOHV+aj8fEjifNVltDM5LaaLdPFItW7m0+LSaVqSWdCeouPi9pkwwULg= smithi163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH1Tcb8RYclFbH3VKTw3TKrx3GUZIAN1y46MBJfyH4dtXUgsz72yvHXmiMhToFKC6J465emhLf/umEzNmb1pcio= smithi164.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHvbdaPg6RXVWvJc94xi/xfqx6OZiNy0c+LGvuCEX3dFw8NxInrEJEym7blCMewWdvgCSBj3ggMOOdmRWZGzyxE= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 53ce1462e129f6eb4071986336534c740fdebd31 timestamp: 2024-07-26_04:18:59 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525 2024-07-26T06:27:04.207 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa; will attempt to use it 2024-07-26T06:27:04.208 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks 2024-07-26T06:27:04.208 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-07-26T06:27:04.209 INFO:teuthology.task.internal:Checking packages... 2024-07-26T06:27:04.248 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528' 2024-07-26T06:27:04.249 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-07-26T06:27:04.249 INFO:teuthology.packaging:ref: None 2024-07-26T06:27:04.249 INFO:teuthology.packaging:tag: None 2024-07-26T06:27:04.249 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240725.121331-debug 2024-07-26T06:27:04.249 INFO:teuthology.packaging:sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T06:27:04.249 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240725.121331-debug 2024-07-26T06:27:04.396 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5320.ge8f88bb1 2024-07-26T06:27:04.397 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-07-26T06:27:04.404 INFO:teuthology.task.internal:no buildpackages task found 2024-07-26T06:27:04.404 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-07-26T06:27:04.469 INFO:teuthology.task.internal:Saving configuration 2024-07-26T06:27:04.492 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-07-26T06:27:04.499 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-07-26T06:27:04.524 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi042.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818656', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 06:21:34.653665', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPGBs+kIALCUShVkk+houIPHtrMappOrOHV+aj8fEjifNVltDM5LaaLdPFItW7m0+LSaVqSWdCeouPi9pkwwULg='} 2024-07-26T06:27:04.549 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818656', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 06:21:34.652415', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH1Tcb8RYclFbH3VKTw3TKrx3GUZIAN1y46MBJfyH4dtXUgsz72yvHXmiMhToFKC6J465emhLf/umEzNmb1pcio='} 2024-07-26T06:27:04.571 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818656', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 06:21:34.650825', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHvbdaPg6RXVWvJc94xi/xfqx6OZiNy0c+LGvuCEX3dFw8NxInrEJEym7blCMewWdvgCSBj3ggMOOdmRWZGzyxE='} 2024-07-26T06:27:04.571 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-07-26T06:27:04.577 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-07-26T06:27:04.577 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-07-26T06:27:04.577 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-07-26T06:27:04.577 INFO:teuthology.run_tasks:Running task console_log... 2024-07-26T06:27:04.680 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7265a0bac0>, signals=[15]) 2024-07-26T06:27:04.680 INFO:teuthology.run_tasks:Running task internal.connect... 2024-07-26T06:27:04.687 INFO:teuthology.task.internal:Opening connections... 2024-07-26T06:27:04.687 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi042.front.sepia.ceph.com 2024-07-26T06:27:04.689 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T06:27:04.779 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2024-07-26T06:27:04.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T06:27:04.854 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2024-07-26T06:27:04.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T06:27:04.930 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-07-26T06:27:04.937 DEBUG:teuthology.orchestra.run.smithi042:> uname -m 2024-07-26T06:27:04.955 INFO:teuthology.orchestra.run.smithi042.stdout:x86_64 2024-07-26T06:27:04.956 DEBUG:teuthology.orchestra.run.smithi042:> cat /etc/os-release 2024-07-26T06:27:05.013 INFO:teuthology.orchestra.run.smithi042.stdout:NAME="CentOS Stream" 2024-07-26T06:27:05.014 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION="9" 2024-07-26T06:27:05.014 INFO:teuthology.orchestra.run.smithi042.stdout:ID="centos" 2024-07-26T06:27:05.014 INFO:teuthology.orchestra.run.smithi042.stdout:ID_LIKE="rhel fedora" 2024-07-26T06:27:05.014 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_ID="9" 2024-07-26T06:27:05.014 INFO:teuthology.orchestra.run.smithi042.stdout:PLATFORM_ID="platform:el9" 2024-07-26T06:27:05.014 INFO:teuthology.orchestra.run.smithi042.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T06:27:05.014 INFO:teuthology.orchestra.run.smithi042.stdout:ANSI_COLOR="0;31" 2024-07-26T06:27:05.014 INFO:teuthology.orchestra.run.smithi042.stdout:LOGO="fedora-logo-icon" 2024-07-26T06:27:05.014 INFO:teuthology.orchestra.run.smithi042.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T06:27:05.014 INFO:teuthology.orchestra.run.smithi042.stdout:HOME_URL="https://centos.org/" 2024-07-26T06:27:05.014 INFO:teuthology.orchestra.run.smithi042.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T06:27:05.015 INFO:teuthology.orchestra.run.smithi042.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T06:27:05.015 INFO:teuthology.orchestra.run.smithi042.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T06:27:05.015 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server 2024-07-26T06:27:05.041 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2024-07-26T06:27:05.060 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2024-07-26T06:27:05.060 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2024-07-26T06:27:05.117 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Stream" 2024-07-26T06:27:05.117 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="9" 2024-07-26T06:27:05.117 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2024-07-26T06:27:05.117 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2024-07-26T06:27:05.118 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="9" 2024-07-26T06:27:05.118 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el9" 2024-07-26T06:27:05.118 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T06:27:05.118 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2024-07-26T06:27:05.118 INFO:teuthology.orchestra.run.smithi163.stdout:LOGO="fedora-logo-icon" 2024-07-26T06:27:05.118 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T06:27:05.118 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/" 2024-07-26T06:27:05.118 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T06:27:05.118 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T06:27:05.118 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T06:27:05.119 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2024-07-26T06:27:05.150 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2024-07-26T06:27:05.167 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2024-07-26T06:27:05.168 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2024-07-26T06:27:05.225 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="CentOS Stream" 2024-07-26T06:27:05.226 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="9" 2024-07-26T06:27:05.226 INFO:teuthology.orchestra.run.smithi164.stdout:ID="centos" 2024-07-26T06:27:05.226 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="rhel fedora" 2024-07-26T06:27:05.226 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="9" 2024-07-26T06:27:05.226 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el9" 2024-07-26T06:27:05.226 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T06:27:05.226 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2024-07-26T06:27:05.226 INFO:teuthology.orchestra.run.smithi164.stdout:LOGO="fedora-logo-icon" 2024-07-26T06:27:05.226 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T06:27:05.226 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://centos.org/" 2024-07-26T06:27:05.227 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T06:27:05.227 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T06:27:05.227 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T06:27:05.227 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2024-07-26T06:27:05.249 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-07-26T06:27:05.260 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-07-26T06:27:05.283 INFO:teuthology.task.internal:Checking for old test directory... 2024-07-26T06:27:05.284 DEBUG:teuthology.orchestra.run.smithi042:> test '!' -e /home/ubuntu/cephtest 2024-07-26T06:27:05.287 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2024-07-26T06:27:05.289 DEBUG:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2024-07-26T06:27:05.305 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-07-26T06:27:05.311 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-07-26T06:27:05.311 DEBUG:teuthology.orchestra.run.smithi042:> test -z $(ls -A /var/lib/ceph) 2024-07-26T06:27:05.344 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2024-07-26T06:27:05.346 DEBUG:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2024-07-26T06:27:05.443 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-07-26T06:27:05.519 INFO:teuthology.run_tasks:Running task kernel... 2024-07-26T06:27:05.533 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': 1, 'sha1': '3e9255e907100f88049f320ca33d977e29155d12'} 2024-07-26T06:27:05.533 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-07-26T06:27:05.533 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-07-26T06:27:05.533 DEBUG:teuthology.orchestra.run.smithi042:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T06:27:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:27:05.551 DEBUG:teuthology.orchestra.run.smithi042:> uname -r 2024-07-26T06:27:05.609 INFO:teuthology.orchestra.run.smithi042.stdout:5.14.0-437.el9.x86_64 2024-07-26T06:27:05.609 INFO:teuthology.task.kernel:Running kernel on smithi042: 5.14.0-437.el9.x86_64 2024-07-26T06:27:05.610 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum install -y kernel 2024-07-26T06:27:08.286 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - BaseOS 7.8 MB/s | 8.2 MB 00:01 2024-07-26T06:27:11.240 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-07-26T06:27:16.538 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 6.5 MB 00:01 2024-07-26T06:27:18.397 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 18 kB 00:00 2024-07-26T06:27:19.609 INFO:teuthology.orchestra.run.smithi042.stdout:Extra Packages for Enterprise Linux 21 MB/s | 22 MB 00:01 2024-07-26T06:27:25.116 INFO:teuthology.orchestra.run.smithi042.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-07-26T06:27:27.180 INFO:teuthology.orchestra.run.smithi042.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-07-26T06:27:27.180 INFO:teuthology.orchestra.run.smithi042.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-07-26T06:27:27.271 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-07-26T06:27:27.288 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-07-26T06:27:27.288 INFO:teuthology.orchestra.run.smithi042.stdout: Package Architecture Version Repository Size 2024-07-26T06:27:27.288 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-07-26T06:27:27.288 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-07-26T06:27:27.288 INFO:teuthology.orchestra.run.smithi042.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-07-26T06:27:27.288 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-07-26T06:27:27.289 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-07-26T06:27:27.289 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-07-26T06:27:27.289 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-07-26T06:27:27.289 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:27:27.289 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-07-26T06:27:27.289 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-07-26T06:27:27.289 INFO:teuthology.orchestra.run.smithi042.stdout:Install 4 Packages 2024-07-26T06:27:27.289 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:27:27.290 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 84 M 2024-07-26T06:27:27.290 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 126 M 2024-07-26T06:27:27.290 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-07-26T06:27:28.334 INFO:teuthology.orchestra.run.smithi042.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 1.5 MB/s | 1.5 MB 00:00 2024-07-26T06:27:28.834 INFO:teuthology.orchestra.run.smithi042.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 12 MB/s | 17 MB 00:01 2024-07-26T06:27:29.601 INFO:teuthology.orchestra.run.smithi042.stdout:(3/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 16 MB/s | 36 MB 00:02 2024-07-26T06:27:30.659 INFO:teuthology.orchestra.run.smithi042.stdout:(4/4): kernel-modules-core-5.14.0-480.el9.x86_6 13 MB/s | 30 MB 00:02 2024-07-26T06:27:30.660 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-07-26T06:27:30.660 INFO:teuthology.orchestra.run.smithi042.stdout:Total 25 MB/s | 84 MB 00:03 2024-07-26T06:27:31.431 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-07-26T06:27:31.838 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-07-26T06:27:31.839 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-07-26T06:27:32.880 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-07-26T06:27:32.880 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-07-26T06:27:35.470 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-07-26T06:27:36.170 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-07-26T06:27:36.269 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-26T06:27:39.091 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-26T06:27:39.234 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-26T06:27:44.437 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-26T06:27:44.495 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-07-26T06:27:49.729 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-07-26T06:28:21.159 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-07-26T06:28:21.202 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-07-26T06:28:21.564 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-07-26T06:28:21.564 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-07-26T06:28:21.564 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-07-26T06:28:21.564 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-07-26T06:28:22.361 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-07-26T06:28:22.361 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:28:22.362 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-07-26T06:28:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-5.14.0-480.el9.x86_64 2024-07-26T06:28:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-07-26T06:28:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-07-26T06:28:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-07-26T06:28:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:28:22.362 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-07-26T06:28:22.760 DEBUG:teuthology.orchestra.run.smithi042:> rpm -q kernel | sort -rV | head -n 1 2024-07-26T06:28:22.841 INFO:teuthology.orchestra.run.smithi042.stdout:kernel-5.14.0-480.el9.x86_64 2024-07-26T06:28:22.842 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-07-26T06:28:22.842 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-07-26T06:28:22.842 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-07-26T06:28:22.842 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-07-26T06:28:22.842 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-07-26T06:28:22.842 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi042.front.sepia.ceph.com, path=None, version=distro) 2024-07-26T06:28:22.843 DEBUG:teuthology.orchestra.run.smithi042:> rpm -q kernel | sort -rV | head -n 1 2024-07-26T06:28:22.900 INFO:teuthology.orchestra.run.smithi042.stdout:kernel-5.14.0-480.el9.x86_64 2024-07-26T06:28:22.901 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-07-26T06:28:22.901 DEBUG:teuthology.orchestra.run.smithi042:> sudo rpm -qi grub2-tools 2024-07-26T06:28:22.945 INFO:teuthology.orchestra.run.smithi042.stdout:Name : grub2-tools 2024-07-26T06:28:22.945 INFO:teuthology.orchestra.run.smithi042.stdout:Epoch : 1 2024-07-26T06:28:22.945 INFO:teuthology.orchestra.run.smithi042.stdout:Version : 2.06 2024-07-26T06:28:22.945 INFO:teuthology.orchestra.run.smithi042.stdout:Release : 68.el9 2024-07-26T06:28:22.945 INFO:teuthology.orchestra.run.smithi042.stdout:Architecture: x86_64 2024-07-26T06:28:22.945 INFO:teuthology.orchestra.run.smithi042.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-07-26T06:28:22.945 INFO:teuthology.orchestra.run.smithi042.stdout:Group : Unspecified 2024-07-26T06:28:22.945 INFO:teuthology.orchestra.run.smithi042.stdout:Size : 8264254 2024-07-26T06:28:22.945 INFO:teuthology.orchestra.run.smithi042.stdout:License : GPLv3+ 2024-07-26T06:28:22.945 INFO:teuthology.orchestra.run.smithi042.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-07-26T06:28:22.946 INFO:teuthology.orchestra.run.smithi042.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-07-26T06:28:22.946 INFO:teuthology.orchestra.run.smithi042.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-07-26T06:28:22.946 INFO:teuthology.orchestra.run.smithi042.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-07-26T06:28:22.946 INFO:teuthology.orchestra.run.smithi042.stdout:Packager : builder@centos.org 2024-07-26T06:28:22.946 INFO:teuthology.orchestra.run.smithi042.stdout:Vendor : CentOS 2024-07-26T06:28:22.946 INFO:teuthology.orchestra.run.smithi042.stdout:URL : http://www.gnu.org/software/grub/ 2024-07-26T06:28:22.946 INFO:teuthology.orchestra.run.smithi042.stdout:Summary : Support tools for GRUB. 2024-07-26T06:28:22.946 INFO:teuthology.orchestra.run.smithi042.stdout:Description : 2024-07-26T06:28:22.946 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:28:22.946 INFO:teuthology.orchestra.run.smithi042.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-07-26T06:28:22.946 INFO:teuthology.orchestra.run.smithi042.stdout:customizable bootloader with modular architecture. It supports a rich 2024-07-26T06:28:22.946 INFO:teuthology.orchestra.run.smithi042.stdout:variety of kernel formats, file systems, computer architectures and 2024-07-26T06:28:22.946 INFO:teuthology.orchestra.run.smithi042.stdout:hardware devices. 2024-07-26T06:28:22.947 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:28:22.947 INFO:teuthology.orchestra.run.smithi042.stdout:This subpackage provides tools for support of all platforms. 2024-07-26T06:28:22.947 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-07-26T06:28:22.948 INFO:teuthology.task.kernel:Updating grub on smithi042 to boot 5.14.0-480.el9.x86_64 2024-07-26T06:28:22.948 DEBUG:teuthology.orchestra.run.smithi042:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-07-26T06:28:23.571 INFO:teuthology.orchestra.run.smithi042.stderr:Generating grub configuration file ... 2024-07-26T06:28:24.992 INFO:teuthology.orchestra.run.smithi042.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-07-26T06:28:25.030 INFO:teuthology.orchestra.run.smithi042.stderr:done 2024-07-26T06:28:25.032 DEBUG:teuthology.orchestra.run.smithi042:> mktemp 2024-07-26T06:28:25.048 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.IPz3ts7Sm7 2024-07-26T06:28:25.048 DEBUG:teuthology.orchestra.run.smithi042:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.IPz3ts7Sm7 2024-07-26T06:28:25.117 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.IPz3ts7Sm7 2024-07-26T06:28:25.265 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.IPz3ts7Sm7 is 6KB 2024-07-26T06:28:25.314 DEBUG:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.IPz3ts7Sm7 2024-07-26T06:28:25.329 DEBUG:teuthology.orchestra.run.smithi042:> sudo /bin/ls /boot/loader/entries || true 2024-07-26T06:28:25.391 INFO:teuthology.orchestra.run.smithi042.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-07-26T06:28:25.391 INFO:teuthology.orchestra.run.smithi042.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-07-26T06:28:25.392 INFO:teuthology.orchestra.run.smithi042.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-07-26T06:28:25.392 INFO:teuthology.orchestra.run.smithi042.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-07-26T06:28:25.393 DEBUG:teuthology.orchestra.run.smithi042:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-07-26T06:28:25.545 DEBUG:teuthology.orchestra.run.smithi042:> sudo shutdown -r now 2024-07-26T06:28:55.549 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-07-26T06:28:55.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-07-26T06:28:55.551 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T06:29:55.554 DEBUG:teuthology.orchestra.remote:timed out 2024-07-26T06:30:04.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-07-26T06:30:04.557 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T06:30:04.917 DEBUG:teuthology.orchestra.run.smithi042:> true 2024-07-26T06:30:05.272 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-07-26T06:30:05.272 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-480.el9.x86_64"... 2024-07-26T06:30:05.273 DEBUG:teuthology.orchestra.run.smithi042:> uname -r 2024-07-26T06:30:05.329 INFO:teuthology.orchestra.run.smithi042.stdout:5.14.0-480.el9.x86_64 2024-07-26T06:30:05.329 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-07-26T06:30:05.329 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-07-26T06:30:05.329 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-07-26T06:30:06.331 DEBUG:teuthology.parallel:result is None 2024-07-26T06:30:06.331 INFO:teuthology.run_tasks:Running task internal.base... 2024-07-26T06:30:06.340 INFO:teuthology.task.internal:Creating test directory... 2024-07-26T06:30:06.340 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T06:30:06.343 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T06:30:06.346 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T06:30:06.364 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-07-26T06:30:06.400 INFO:teuthology.run_tasks:Running task internal.archive... 2024-07-26T06:30:06.429 INFO:teuthology.task.internal:Creating archive directory... 2024-07-26T06:30:06.430 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T06:30:06.432 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T06:30:06.434 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T06:30:06.593 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-07-26T06:30:06.600 INFO:teuthology.task.internal:Enabling coredump saving... 2024-07-26T06:30:06.600 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T06:30:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:30:06.618 DEBUG:teuthology.orchestra.run.smithi164:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T06:30:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:30:06.635 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T06:30:06.637 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T06:30:06.660 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T06:30:06.689 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T06:30:06.704 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T06:30:06.717 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T06:30:06.717 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T06:30:06.733 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T06:30:06.755 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T06:30:06.756 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-07-26T06:30:06.763 INFO:teuthology.task.internal:Configuring sudo... 2024-07-26T06:30:06.764 DEBUG:teuthology.orchestra.run.smithi042:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T06:30:06.766 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T06:30:06.768 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T06:30:06.803 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-07-26T06:30:06.813 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-07-26T06:30:06.813 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T06:30:06.830 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T06:30:06.835 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T06:30:06.860 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T06:30:06.927 DEBUG:teuthology.orchestra.run.smithi042:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T06:30:06.972 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T06:30:07.054 DEBUG:teuthology.orchestra.run.smithi042:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T06:30:07.092 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-07-26T06:30:07.092 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T06:30:07.159 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T06:30:07.203 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T06:30:07.256 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T06:30:07.341 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T06:30:07.382 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:30:07.475 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T06:30:07.502 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T06:30:07.546 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T06:30:07.602 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T06:30:07.687 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T06:30:07.730 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:30:07.730 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T06:30:07.796 DEBUG:teuthology.orchestra.run.smithi042:> sudo service rsyslog restart 2024-07-26T06:30:07.799 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2024-07-26T06:30:07.801 DEBUG:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2024-07-26T06:30:07.835 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T06:30:07.845 INFO:teuthology.orchestra.run.smithi042.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T06:30:07.870 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T06:30:08.297 INFO:teuthology.run_tasks:Running task internal.timer... 2024-07-26T06:30:08.306 INFO:teuthology.task.internal:Starting timer... 2024-07-26T06:30:08.306 INFO:teuthology.run_tasks:Running task pcp... 2024-07-26T06:30:08.342 INFO:teuthology.run_tasks:Running task selinux... 2024-07-26T06:30:08.373 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-07-26T06:30:08.373 DEBUG:teuthology.orchestra.run.smithi042:> sudo service auditd rotate 2024-07-26T06:30:08.435 INFO:teuthology.orchestra.run.smithi042.stdout:Rotating logs: 2024-07-26T06:30:08.437 DEBUG:teuthology.orchestra.run.smithi163:> sudo service auditd rotate 2024-07-26T06:30:08.498 INFO:teuthology.orchestra.run.smithi163.stdout:Rotating logs: 2024-07-26T06:30:08.501 DEBUG:teuthology.orchestra.run.smithi164:> sudo service auditd rotate 2024-07-26T06:30:08.564 INFO:teuthology.orchestra.run.smithi164.stdout:Rotating logs: 2024-07-26T06:30:08.567 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-07-26T06:30:08.567 DEBUG:teuthology.orchestra.run.smithi042:> /usr/sbin/getenforce 2024-07-26T06:30:08.605 INFO:teuthology.orchestra.run.smithi042.stdout:Permissive 2024-07-26T06:30:08.606 DEBUG:teuthology.orchestra.run.smithi163:> /usr/sbin/getenforce 2024-07-26T06:30:08.636 INFO:teuthology.orchestra.run.smithi163.stdout:Permissive 2024-07-26T06:30:08.637 DEBUG:teuthology.orchestra.run.smithi164:> /usr/sbin/getenforce 2024-07-26T06:30:08.663 INFO:teuthology.orchestra.run.smithi164.stdout:Permissive 2024-07-26T06:30:08.663 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi042.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi163.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi164.front.sepia.ceph.com': 'permissive'} 2024-07-26T06:30:08.663 DEBUG:teuthology.orchestra.run.smithi042:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T06:30:08.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:30:08.690 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T06:30:08.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:30:08.720 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T06:30:08.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:30:08.749 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-07-26T06:30:08.750 DEBUG:teuthology.orchestra.run.smithi042:> sudo /usr/sbin/setenforce permissive 2024-07-26T06:30:08.833 DEBUG:teuthology.orchestra.run.smithi163:> sudo /usr/sbin/setenforce permissive 2024-07-26T06:30:08.864 DEBUG:teuthology.orchestra.run.smithi164:> sudo /usr/sbin/setenforce permissive 2024-07-26T06:30:08.890 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-07-26T06:30:08.900 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-07-26T06:30:08.916 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-07-26T06:30:08.917 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi042.front.sepia.ceph.com,smithi163.front.sepia.ceph.com,smithi164.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-07-26T06:35:39.498 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi042.front.sepia.ceph.com'), Remote(name='ubuntu@smithi163.front.sepia.ceph.com'), Remote(name='ubuntu@smithi164.front.sepia.ceph.com')] 2024-07-26T06:35:39.532 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-07-26T06:35:39.534 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T06:35:39.628 DEBUG:teuthology.orchestra.run.smithi042:> true 2024-07-26T06:35:39.712 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-07-26T06:35:39.713 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-07-26T06:35:39.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T06:35:39.803 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-07-26T06:35:39.891 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-07-26T06:35:39.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-07-26T06:35:39.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T06:35:39.968 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-07-26T06:35:40.052 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-07-26T06:35:40.052 INFO:teuthology.run_tasks:Running task clock... 2024-07-26T06:35:40.065 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-07-26T06:35:40.065 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T06:35:40.066 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T06:35:40.068 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T06:35:40.068 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T06:35:40.071 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T06:35:40.071 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T06:35:40.101 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T06:35:40.102 INFO:teuthology.orchestra.run.smithi042.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T06:35:40.116 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T06:35:40.118 INFO:teuthology.orchestra.run.smithi042.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T06:35:40.122 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T06:35:40.137 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T06:35:40.141 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: ntpd: command not found 2024-07-26T06:35:40.147 INFO:teuthology.orchestra.run.smithi042.stderr:sudo: ntpd: command not found 2024-07-26T06:35:40.152 INFO:teuthology.orchestra.run.smithi163.stdout:506 Cannot talk to daemon 2024-07-26T06:35:40.159 INFO:teuthology.orchestra.run.smithi042.stdout:506 Cannot talk to daemon 2024-07-26T06:35:40.162 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: ntpd: command not found 2024-07-26T06:35:40.166 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T06:35:40.173 INFO:teuthology.orchestra.run.smithi164.stdout:506 Cannot talk to daemon 2024-07-26T06:35:40.175 INFO:teuthology.orchestra.run.smithi042.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T06:35:40.180 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T06:35:40.269 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-07-26T06:35:40.270 INFO:teuthology.orchestra.run.smithi042.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T06:35:40.270 INFO:teuthology.orchestra.run.smithi042.stderr:bash: line 1: ntpq: command not found 2024-07-26T06:35:40.270 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T06:35:40.270 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T06:35:40.270 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-07-26T06:35:40.272 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T06:35:40.272 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-07-26T06:35:40.272 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:35:40.273 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:35:40.273 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:35:40.273 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:35:40.273 INFO:teuthology.orchestra.run.smithi042.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T06:35:40.273 INFO:teuthology.orchestra.run.smithi042.stdout:=============================================================================== 2024-07-26T06:35:40.273 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:35:40.273 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:35:40.273 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:35:40.273 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:35:40.274 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T06:35:40.274 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-07-26T06:35:40.275 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:35:40.275 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:35:40.275 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:35:40.275 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T06:35:40.275 INFO:teuthology.run_tasks:Running task install... 2024-07-26T06:35:40.284 DEBUG:teuthology.task.install:project ceph 2024-07-26T06:35:40.285 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'}} 2024-07-26T06:35:40.285 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'} 2024-07-26T06:35:40.285 INFO:teuthology.task.install:Using flavor: default 2024-07-26T06:35:40.291 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-07-26T06:35:40.291 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-07-26T06:35:40.292 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T06:35:40.292 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T06:35:40.294 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T06:35:40.295 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T06:35:40.296 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T06:35:40.296 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T06:35:40.437 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T06:35:40.437 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T06:35:40.451 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T06:35:40.451 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T06:35:40.470 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T06:35:40.470 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T06:35:40.548 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T06:35:40.548 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-07-26T06:35:40.548 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T06:35:40.562 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T06:35:40.562 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:35:40.562 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T06:35:40.577 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T06:35:40.577 DEBUG:teuthology.orchestra.run.smithi042:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T06:35:40.580 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T06:35:40.580 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:35:40.580 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T06:35:40.588 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T06:35:40.588 DEBUG:teuthology.orchestra.run.smithi164:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T06:35:40.604 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T06:35:40.605 DEBUG:teuthology.orchestra.run.smithi163:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T06:35:40.645 DEBUG:teuthology.orchestra.run.smithi042:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T06:35:40.653 DEBUG:teuthology.orchestra.run.smithi164:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T06:35:40.669 DEBUG:teuthology.orchestra.run.smithi163:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T06:35:40.723 DEBUG:teuthology.orchestra.run.smithi042:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T06:35:40.795 DEBUG:teuthology.orchestra.run.smithi163:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T06:35:40.798 DEBUG:teuthology.orchestra.run.smithi164:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T06:35:40.824 INFO:teuthology.orchestra.run.smithi042.stdout:check_obsoletes = 1 2024-07-26T06:35:40.825 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum clean all 2024-07-26T06:35:40.857 INFO:teuthology.orchestra.run.smithi163.stdout:check_obsoletes = 1 2024-07-26T06:35:40.857 INFO:teuthology.orchestra.run.smithi164.stdout:check_obsoletes = 1 2024-07-26T06:35:40.858 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum clean all 2024-07-26T06:35:40.859 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum clean all 2024-07-26T06:35:41.069 INFO:teuthology.orchestra.run.smithi042.stdout:45 files removed 2024-07-26T06:35:41.083 INFO:teuthology.orchestra.run.smithi164.stdout:45 files removed 2024-07-26T06:35:41.084 INFO:teuthology.orchestra.run.smithi163.stdout:45 files removed 2024-07-26T06:35:41.096 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T06:35:41.110 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T06:35:41.111 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T06:35:41.693 INFO:teuthology.orchestra.run.smithi163.stdout:ceph packages for x86_64 263 kB/s | 83 kB 00:00 2024-07-26T06:35:41.701 INFO:teuthology.orchestra.run.smithi164.stdout:ceph packages for x86_64 256 kB/s | 83 kB 00:00 2024-07-26T06:35:41.715 INFO:teuthology.orchestra.run.smithi042.stdout:ceph packages for x86_64 261 kB/s | 83 kB 00:00 2024-07-26T06:35:42.035 INFO:teuthology.orchestra.run.smithi163.stdout:ceph noarch packages 80 kB/s | 24 kB 00:00 2024-07-26T06:35:42.043 INFO:teuthology.orchestra.run.smithi164.stdout:ceph noarch packages 80 kB/s | 24 kB 00:00 2024-07-26T06:35:42.066 INFO:teuthology.orchestra.run.smithi042.stdout:ceph noarch packages 81 kB/s | 24 kB 00:00 2024-07-26T06:35:42.391 INFO:teuthology.orchestra.run.smithi042.stdout:ceph source packages 6.7 kB/s | 1.9 kB 00:00 2024-07-26T06:35:42.478 INFO:teuthology.orchestra.run.smithi164.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-07-26T06:35:42.485 INFO:teuthology.orchestra.run.smithi163.stdout:ceph source packages 4.5 kB/s | 1.9 kB 00:00 2024-07-26T06:35:44.094 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.2 MB 00:01 2024-07-26T06:35:44.183 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.2 MB 00:01 2024-07-26T06:35:47.228 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 9.8 MB/s | 20 MB 00:02 2024-07-26T06:35:47.709 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - AppStream 8.3 MB/s | 20 MB 00:02 2024-07-26T06:35:52.418 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - CRB 7.7 MB/s | 6.5 MB 00:00 2024-07-26T06:35:54.421 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 18 kB 00:00 2024-07-26T06:35:55.573 INFO:teuthology.orchestra.run.smithi042.stdout:Extra Packages for Enterprise Linux 22 MB/s | 22 MB 00:01 2024-07-26T06:35:56.021 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 622 kB/s | 8.2 MB 00:13 2024-07-26T06:35:57.545 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 1.0 MB/s | 6.5 MB 00:06 2024-07-26T06:35:59.063 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 18 kB 00:00 2024-07-26T06:35:59.463 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 8.5 MB/s | 20 MB 00:02 2024-07-26T06:35:59.992 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 28 MB/s | 22 MB 00:00 2024-07-26T06:36:01.179 INFO:teuthology.orchestra.run.smithi042.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-07-26T06:36:03.591 INFO:teuthology.orchestra.run.smithi042.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T06:36:03.592 INFO:teuthology.orchestra.run.smithi042.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T06:36:03.603 INFO:teuthology.orchestra.run.smithi042.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T06:36:03.604 INFO:teuthology.orchestra.run.smithi042.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T06:36:03.605 INFO:teuthology.orchestra.run.smithi042.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T06:36:03.606 INFO:teuthology.orchestra.run.smithi042.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T06:36:03.607 INFO:teuthology.orchestra.run.smithi042.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T06:36:03.610 INFO:teuthology.orchestra.run.smithi042.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T06:36:03.611 INFO:teuthology.orchestra.run.smithi042.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T06:36:03.678 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-07-26T06:36:03.680 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-07-26T06:36:03.710 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================= 2024-07-26T06:36:03.711 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-07-26T06:36:03.711 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================= 2024-07-26T06:36:03.711 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-07-26T06:36:03.711 INFO:teuthology.orchestra.run.smithi042.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T06:36:03.711 INFO:teuthology.orchestra.run.smithi042.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T06:36:03.711 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T06:36:03.711 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T06:36:03.711 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T06:36:03.712 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T06:36:03.712 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T06:36:03.712 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T06:36:03.712 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T06:36:03.712 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T06:36:03.712 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T06:36:03.712 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T06:36:03.712 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T06:36:03.712 INFO:teuthology.orchestra.run.smithi042.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T06:36:03.712 INFO:teuthology.orchestra.run.smithi042.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T06:36:03.712 INFO:teuthology.orchestra.run.smithi042.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T06:36:03.713 INFO:teuthology.orchestra.run.smithi042.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T06:36:03.713 INFO:teuthology.orchestra.run.smithi042.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T06:36:03.713 INFO:teuthology.orchestra.run.smithi042.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T06:36:03.713 INFO:teuthology.orchestra.run.smithi042.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T06:36:03.713 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T06:36:03.713 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T06:36:03.713 INFO:teuthology.orchestra.run.smithi042.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T06:36:03.713 INFO:teuthology.orchestra.run.smithi042.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T06:36:03.713 INFO:teuthology.orchestra.run.smithi042.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T06:36:03.713 INFO:teuthology.orchestra.run.smithi042.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T06:36:03.714 INFO:teuthology.orchestra.run.smithi042.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T06:36:03.714 INFO:teuthology.orchestra.run.smithi042.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T06:36:03.714 INFO:teuthology.orchestra.run.smithi042.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T06:36:03.714 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T06:36:03.714 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T06:36:03.714 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T06:36:03.714 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T06:36:03.714 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T06:36:03.715 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T06:36:03.715 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T06:36:03.715 INFO:teuthology.orchestra.run.smithi042.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T06:36:03.715 INFO:teuthology.orchestra.run.smithi042.stdout:Upgrading: 2024-07-26T06:36:03.715 INFO:teuthology.orchestra.run.smithi042.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T06:36:03.715 INFO:teuthology.orchestra.run.smithi042.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T06:36:03.715 INFO:teuthology.orchestra.run.smithi042.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T06:36:03.715 INFO:teuthology.orchestra.run.smithi042.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T06:36:03.715 INFO:teuthology.orchestra.run.smithi042.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T06:36:03.715 INFO:teuthology.orchestra.run.smithi042.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T06:36:03.715 INFO:teuthology.orchestra.run.smithi042.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T06:36:03.716 INFO:teuthology.orchestra.run.smithi042.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T06:36:03.716 INFO:teuthology.orchestra.run.smithi042.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T06:36:03.716 INFO:teuthology.orchestra.run.smithi042.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T06:36:03.716 INFO:teuthology.orchestra.run.smithi042.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T06:36:03.716 INFO:teuthology.orchestra.run.smithi042.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T06:36:03.716 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-07-26T06:36:03.716 INFO:teuthology.orchestra.run.smithi042.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T06:36:03.716 INFO:teuthology.orchestra.run.smithi042.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T06:36:03.716 INFO:teuthology.orchestra.run.smithi042.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T06:36:03.716 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T06:36:03.716 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T06:36:03.717 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T06:36:03.717 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T06:36:03.717 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T06:36:03.717 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T06:36:03.717 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T06:36:03.717 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T06:36:03.717 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T06:36:03.717 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T06:36:03.717 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T06:36:03.717 INFO:teuthology.orchestra.run.smithi042.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T06:36:03.717 INFO:teuthology.orchestra.run.smithi042.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T06:36:03.717 INFO:teuthology.orchestra.run.smithi042.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T06:36:03.718 INFO:teuthology.orchestra.run.smithi042.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T06:36:03.718 INFO:teuthology.orchestra.run.smithi042.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T06:36:03.718 INFO:teuthology.orchestra.run.smithi042.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T06:36:03.718 INFO:teuthology.orchestra.run.smithi042.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T06:36:03.718 INFO:teuthology.orchestra.run.smithi042.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T06:36:03.718 INFO:teuthology.orchestra.run.smithi042.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T06:36:03.718 INFO:teuthology.orchestra.run.smithi042.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T06:36:03.718 INFO:teuthology.orchestra.run.smithi042.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T06:36:03.718 INFO:teuthology.orchestra.run.smithi042.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T06:36:03.718 INFO:teuthology.orchestra.run.smithi042.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T06:36:03.719 INFO:teuthology.orchestra.run.smithi042.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T06:36:03.719 INFO:teuthology.orchestra.run.smithi042.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T06:36:03.719 INFO:teuthology.orchestra.run.smithi042.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T06:36:03.719 INFO:teuthology.orchestra.run.smithi042.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T06:36:03.719 INFO:teuthology.orchestra.run.smithi042.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T06:36:03.719 INFO:teuthology.orchestra.run.smithi042.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T06:36:03.719 INFO:teuthology.orchestra.run.smithi042.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T06:36:03.719 INFO:teuthology.orchestra.run.smithi042.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T06:36:03.719 INFO:teuthology.orchestra.run.smithi042.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T06:36:03.719 INFO:teuthology.orchestra.run.smithi042.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T06:36:03.719 INFO:teuthology.orchestra.run.smithi042.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T06:36:03.719 INFO:teuthology.orchestra.run.smithi042.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T06:36:03.720 INFO:teuthology.orchestra.run.smithi042.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T06:36:03.720 INFO:teuthology.orchestra.run.smithi042.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T06:36:03.720 INFO:teuthology.orchestra.run.smithi042.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T06:36:03.720 INFO:teuthology.orchestra.run.smithi042.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T06:36:03.720 INFO:teuthology.orchestra.run.smithi042.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T06:36:03.720 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T06:36:03.720 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T06:36:03.720 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T06:36:03.721 INFO:teuthology.orchestra.run.smithi042.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T06:36:03.721 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T06:36:03.721 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T06:36:03.721 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T06:36:03.721 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T06:36:03.721 INFO:teuthology.orchestra.run.smithi042.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T06:36:03.722 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T06:36:03.722 INFO:teuthology.orchestra.run.smithi042.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T06:36:03.722 INFO:teuthology.orchestra.run.smithi042.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T06:36:03.722 INFO:teuthology.orchestra.run.smithi042.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T06:36:03.722 INFO:teuthology.orchestra.run.smithi042.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T06:36:03.722 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T06:36:03.722 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T06:36:03.722 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T06:36:03.723 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T06:36:03.723 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T06:36:03.723 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T06:36:03.723 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T06:36:03.723 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T06:36:03.723 INFO:teuthology.orchestra.run.smithi042.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T06:36:03.723 INFO:teuthology.orchestra.run.smithi042.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T06:36:03.723 INFO:teuthology.orchestra.run.smithi042.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T06:36:03.723 INFO:teuthology.orchestra.run.smithi042.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T06:36:03.723 INFO:teuthology.orchestra.run.smithi042.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T06:36:03.724 INFO:teuthology.orchestra.run.smithi042.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T06:36:03.724 INFO:teuthology.orchestra.run.smithi042.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T06:36:03.724 INFO:teuthology.orchestra.run.smithi042.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T06:36:03.724 INFO:teuthology.orchestra.run.smithi042.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T06:36:03.724 INFO:teuthology.orchestra.run.smithi042.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T06:36:03.724 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T06:36:03.725 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T06:36:03.725 INFO:teuthology.orchestra.run.smithi042.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T06:36:03.725 INFO:teuthology.orchestra.run.smithi042.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T06:36:03.725 INFO:teuthology.orchestra.run.smithi042.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T06:36:03.725 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T06:36:03.725 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T06:36:03.725 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T06:36:03.725 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T06:36:03.725 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T06:36:03.725 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T06:36:03.725 INFO:teuthology.orchestra.run.smithi042.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T06:36:03.725 INFO:teuthology.orchestra.run.smithi042.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T06:36:03.726 INFO:teuthology.orchestra.run.smithi042.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T06:36:03.726 INFO:teuthology.orchestra.run.smithi042.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T06:36:03.726 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T06:36:03.726 INFO:teuthology.orchestra.run.smithi042.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T06:36:03.726 INFO:teuthology.orchestra.run.smithi042.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T06:36:03.726 INFO:teuthology.orchestra.run.smithi042.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T06:36:03.726 INFO:teuthology.orchestra.run.smithi042.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T06:36:03.726 INFO:teuthology.orchestra.run.smithi042.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T06:36:03.726 INFO:teuthology.orchestra.run.smithi042.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T06:36:03.726 INFO:teuthology.orchestra.run.smithi042.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T06:36:03.726 INFO:teuthology.orchestra.run.smithi042.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T06:36:03.727 INFO:teuthology.orchestra.run.smithi042.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T06:36:03.727 INFO:teuthology.orchestra.run.smithi042.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T06:36:03.727 INFO:teuthology.orchestra.run.smithi042.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T06:36:03.727 INFO:teuthology.orchestra.run.smithi042.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T06:36:03.727 INFO:teuthology.orchestra.run.smithi042.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T06:36:03.727 INFO:teuthology.orchestra.run.smithi042.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T06:36:03.727 INFO:teuthology.orchestra.run.smithi042.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T06:36:03.727 INFO:teuthology.orchestra.run.smithi042.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T06:36:03.727 INFO:teuthology.orchestra.run.smithi042.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T06:36:03.727 INFO:teuthology.orchestra.run.smithi042.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T06:36:03.727 INFO:teuthology.orchestra.run.smithi042.stdout:Installing weak dependencies: 2024-07-26T06:36:03.727 INFO:teuthology.orchestra.run.smithi042.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T06:36:03.728 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:36:03.728 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-07-26T06:36:03.728 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================= 2024-07-26T06:36:03.728 INFO:teuthology.orchestra.run.smithi042.stdout:Install 148 Packages 2024-07-26T06:36:03.728 INFO:teuthology.orchestra.run.smithi042.stdout:Upgrade 12 Packages 2024-07-26T06:36:03.728 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:36:03.729 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 233 M 2024-07-26T06:36:03.729 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-07-26T06:36:04.331 INFO:teuthology.orchestra.run.smithi042.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 34 kB/s | 6.6 kB 00:00 2024-07-26T06:36:04.573 INFO:teuthology.orchestra.run.smithi042.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 3.5 MB/s | 872 kB 00:00 2024-07-26T06:36:04.656 INFO:teuthology.orchestra.run.smithi042.stdout:(3/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 11 MB/s | 5.5 MB 00:00 2024-07-26T06:36:04.698 INFO:teuthology.orchestra.run.smithi042.stdout:(4/160): ceph-immutable-object-cache-19.0.0-532 1.2 MB/s | 148 kB 00:00 2024-07-26T06:36:04.765 INFO:teuthology.orchestra.run.smithi042.stdout:(5/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 21 MB/s | 2.3 MB 00:00 2024-07-26T06:36:04.987 INFO:teuthology.orchestra.run.smithi042.stdout:(6/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 5.3 MB/s | 1.5 MB 00:00 2024-07-26T06:36:05.066 INFO:teuthology.orchestra.run.smithi042.stdout:(7/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 16 MB/s | 4.9 MB 00:00 2024-07-26T06:36:05.334 INFO:teuthology.orchestra.run.smithi042.stdout:(8/160): ceph-common-19.0.0-5320.ge8f88bb1.el9. 17 MB/s | 20 MB 00:01 2024-07-26T06:36:05.375 INFO:teuthology.orchestra.run.smithi042.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 607 kB/s | 25 kB 00:00 2024-07-26T06:36:05.414 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-07-26T06:36:05.584 INFO:teuthology.orchestra.run.smithi042.stdout:(10/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el 20 MB/s | 10 MB 00:00 2024-07-26T06:36:05.634 INFO:teuthology.orchestra.run.smithi042.stdout:(11/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 4.4 MB/s | 225 kB 00:00 2024-07-26T06:36:05.701 INFO:teuthology.orchestra.run.smithi042.stdout:(12/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 407 kB/s | 27 kB 00:00 2024-07-26T06:36:05.768 INFO:teuthology.orchestra.run.smithi042.stdout:(13/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 476 kB/s | 32 kB 00:00 2024-07-26T06:36:05.876 INFO:teuthology.orchestra.run.smithi042.stdout:(14/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 6.9 MB/s | 765 kB 00:00 2024-07-26T06:36:05.968 INFO:teuthology.orchestra.run.smithi042.stdout:(15/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 1.7 MB/s | 158 kB 00:00 2024-07-26T06:36:06.052 INFO:teuthology.orchestra.run.smithi042.stdout:(16/160): librados-devel-19.0.0-5320.ge8f88bb1. 1.5 MB/s | 127 kB 00:00 2024-07-26T06:36:06.152 INFO:teuthology.orchestra.run.smithi042.stdout:(17/160): libradosstriper1-19.0.0-5320.ge8f88bb 4.9 MB/s | 503 kB 00:00 2024-07-26T06:36:06.214 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 18 kB 00:01 2024-07-26T06:36:06.344 INFO:teuthology.orchestra.run.smithi042.stdout:(18/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 13 MB/s | 18 MB 00:01 2024-07-26T06:36:06.386 INFO:teuthology.orchestra.run.smithi042.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 1.1 MB/s | 45 kB 00:00 2024-07-26T06:36:06.470 INFO:teuthology.orchestra.run.smithi042.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 1.6 MB/s | 138 kB 00:00 2024-07-26T06:36:06.537 INFO:teuthology.orchestra.run.smithi042.stdout:(21/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 14 MB/s | 5.3 MB 00:00 2024-07-26T06:36:06.554 INFO:teuthology.orchestra.run.smithi042.stdout:(22/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 2.2 MB/s | 188 kB 00:00 2024-07-26T06:36:06.588 INFO:teuthology.orchestra.run.smithi042.stdout:(23/160): python3-rados-19.0.0-5320.ge8f88bb1.e 7.2 MB/s | 372 kB 00:00 2024-07-26T06:36:06.613 INFO:teuthology.orchestra.run.smithi042.stdout:(24/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 5.9 MB/s | 355 kB 00:00 2024-07-26T06:36:06.638 INFO:teuthology.orchestra.run.smithi042.stdout:(25/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 2.3 MB/s | 116 kB 00:00 2024-07-26T06:36:06.663 INFO:teuthology.orchestra.run.smithi042.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.7 MB/s | 86 kB 00:00 2024-07-26T06:36:06.714 INFO:teuthology.orchestra.run.smithi042.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 3.3 MB/s | 170 kB 00:00 2024-07-26T06:36:06.781 INFO:teuthology.orchestra.run.smithi042.stdout:(28/160): ceph-grafana-dashboards-19.0.0-5320.g 514 kB/s | 34 kB 00:00 2024-07-26T06:36:07.215 INFO:teuthology.orchestra.run.smithi042.stdout:(29/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 24 MB/s | 44 MB 00:01 2024-07-26T06:36:07.240 INFO:teuthology.orchestra.run.smithi042.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 326 kB/s | 149 kB 00:00 2024-07-26T06:36:07.299 INFO:teuthology.orchestra.run.smithi042.stdout:(31/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 4.7 MB/s | 3.1 MB 00:00 2024-07-26T06:36:07.377 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 22 MB/s | 22 MB 00:01 2024-07-26T06:36:07.416 INFO:teuthology.orchestra.run.smithi042.stdout:(32/160): ceph-mgr-modules-core-19.0.0-5320.ge8 2.4 MB/s | 284 kB 00:00 2024-07-26T06:36:07.507 INFO:teuthology.orchestra.run.smithi042.stdout:(33/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 541 kB/s | 49 kB 00:00 2024-07-26T06:36:07.591 INFO:teuthology.orchestra.run.smithi042.stdout:(34/160): ceph-prometheus-alerts-19.0.0-5320.ge 198 kB/s | 16 kB 00:00 2024-07-26T06:36:07.664 INFO:teuthology.orchestra.run.smithi163.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T06:36:07.665 INFO:teuthology.orchestra.run.smithi163.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T06:36:07.676 INFO:teuthology.orchestra.run.smithi163.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T06:36:07.677 INFO:teuthology.orchestra.run.smithi163.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T06:36:07.678 INFO:teuthology.orchestra.run.smithi163.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T06:36:07.679 INFO:teuthology.orchestra.run.smithi163.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T06:36:07.680 INFO:teuthology.orchestra.run.smithi163.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T06:36:07.683 INFO:teuthology.orchestra.run.smithi163.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T06:36:07.684 INFO:teuthology.orchestra.run.smithi163.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T06:36:07.742 INFO:teuthology.orchestra.run.smithi042.stdout:(35/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 27 MB/s | 14 MB 00:00 2024-07-26T06:36:07.747 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-07-26T06:36:07.760 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-07-26T06:36:07.760 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-07-26T06:36:07.761 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-07-26T06:36:07.761 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-07-26T06:36:07.761 INFO:teuthology.orchestra.run.smithi163.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T06:36:07.761 INFO:teuthology.orchestra.run.smithi163.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T06:36:07.761 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T06:36:07.761 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T06:36:07.761 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T06:36:07.761 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T06:36:07.761 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T06:36:07.761 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T06:36:07.761 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T06:36:07.761 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T06:36:07.762 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T06:36:07.762 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T06:36:07.762 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T06:36:07.762 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T06:36:07.762 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T06:36:07.762 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T06:36:07.762 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T06:36:07.762 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T06:36:07.762 INFO:teuthology.orchestra.run.smithi163.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T06:36:07.762 INFO:teuthology.orchestra.run.smithi163.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T06:36:07.762 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T06:36:07.763 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T06:36:07.763 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T06:36:07.763 INFO:teuthology.orchestra.run.smithi163.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T06:36:07.763 INFO:teuthology.orchestra.run.smithi163.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T06:36:07.763 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T06:36:07.763 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T06:36:07.763 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T06:36:07.763 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T06:36:07.763 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T06:36:07.763 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T06:36:07.763 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T06:36:07.763 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T06:36:07.763 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T06:36:07.764 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T06:36:07.764 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T06:36:07.764 INFO:teuthology.orchestra.run.smithi163.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T06:36:07.764 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrading: 2024-07-26T06:36:07.764 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T06:36:07.764 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T06:36:07.764 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T06:36:07.764 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T06:36:07.764 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T06:36:07.764 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T06:36:07.764 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T06:36:07.764 INFO:teuthology.orchestra.run.smithi163.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T06:36:07.765 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T06:36:07.765 INFO:teuthology.orchestra.run.smithi163.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T06:36:07.765 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T06:36:07.765 INFO:teuthology.orchestra.run.smithi163.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T06:36:07.765 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-07-26T06:36:07.765 INFO:teuthology.orchestra.run.smithi163.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T06:36:07.765 INFO:teuthology.orchestra.run.smithi163.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T06:36:07.765 INFO:teuthology.orchestra.run.smithi163.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T06:36:07.765 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T06:36:07.765 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T06:36:07.765 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T06:36:07.765 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T06:36:07.765 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T06:36:07.766 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T06:36:07.766 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T06:36:07.766 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T06:36:07.766 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T06:36:07.766 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T06:36:07.766 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T06:36:07.766 INFO:teuthology.orchestra.run.smithi163.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T06:36:07.766 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T06:36:07.766 INFO:teuthology.orchestra.run.smithi163.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T06:36:07.766 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T06:36:07.766 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T06:36:07.766 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T06:36:07.767 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T06:36:07.767 INFO:teuthology.orchestra.run.smithi163.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T06:36:07.767 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T06:36:07.767 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T06:36:07.767 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T06:36:07.767 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T06:36:07.767 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T06:36:07.767 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T06:36:07.767 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T06:36:07.767 INFO:teuthology.orchestra.run.smithi163.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T06:36:07.767 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T06:36:07.767 INFO:teuthology.orchestra.run.smithi163.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T06:36:07.768 INFO:teuthology.orchestra.run.smithi163.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T06:36:07.768 INFO:teuthology.orchestra.run.smithi163.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T06:36:07.768 INFO:teuthology.orchestra.run.smithi163.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T06:36:07.768 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T06:36:07.768 INFO:teuthology.orchestra.run.smithi163.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T06:36:07.768 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T06:36:07.768 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T06:36:07.768 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T06:36:07.768 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T06:36:07.768 INFO:teuthology.orchestra.run.smithi163.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T06:36:07.768 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T06:36:07.768 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T06:36:07.769 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T06:36:07.769 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T06:36:07.769 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T06:36:07.769 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T06:36:07.769 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T06:36:07.769 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T06:36:07.769 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T06:36:07.769 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T06:36:07.769 INFO:teuthology.orchestra.run.smithi163.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T06:36:07.769 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T06:36:07.769 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T06:36:07.769 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T06:36:07.769 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T06:36:07.770 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T06:36:07.770 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T06:36:07.770 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T06:36:07.770 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T06:36:07.770 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T06:36:07.770 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T06:36:07.770 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T06:36:07.770 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T06:36:07.770 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T06:36:07.770 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T06:36:07.770 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T06:36:07.770 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T06:36:07.771 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T06:36:07.771 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T06:36:07.771 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T06:36:07.771 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T06:36:07.771 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T06:36:07.771 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T06:36:07.771 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T06:36:07.771 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T06:36:07.771 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T06:36:07.771 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T06:36:07.771 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T06:36:07.771 INFO:teuthology.orchestra.run.smithi163.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T06:36:07.771 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T06:36:07.772 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T06:36:07.772 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T06:36:07.772 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T06:36:07.772 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T06:36:07.772 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T06:36:07.772 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T06:36:07.772 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T06:36:07.772 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T06:36:07.772 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T06:36:07.772 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T06:36:07.772 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T06:36:07.772 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T06:36:07.773 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T06:36:07.773 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T06:36:07.773 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T06:36:07.773 INFO:teuthology.orchestra.run.smithi163.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T06:36:07.773 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T06:36:07.773 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T06:36:07.773 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T06:36:07.773 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T06:36:07.773 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T06:36:07.773 INFO:teuthology.orchestra.run.smithi163.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T06:36:07.773 INFO:teuthology.orchestra.run.smithi163.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T06:36:07.773 INFO:teuthology.orchestra.run.smithi163.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T06:36:07.773 INFO:teuthology.orchestra.run.smithi163.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T06:36:07.774 INFO:teuthology.orchestra.run.smithi163.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T06:36:07.774 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T06:36:07.774 INFO:teuthology.orchestra.run.smithi163.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T06:36:07.774 INFO:teuthology.orchestra.run.smithi163.stdout:Installing weak dependencies: 2024-07-26T06:36:07.774 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T06:36:07.774 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:36:07.774 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-07-26T06:36:07.774 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-07-26T06:36:07.774 INFO:teuthology.orchestra.run.smithi163.stdout:Install 148 Packages 2024-07-26T06:36:07.774 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrade 12 Packages 2024-07-26T06:36:07.774 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:36:07.775 INFO:teuthology.orchestra.run.smithi042.stdout:(36/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 1.5 MB/s | 274 kB 00:00 2024-07-26T06:36:07.775 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 233 M 2024-07-26T06:36:07.775 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-07-26T06:36:07.859 INFO:teuthology.orchestra.run.smithi042.stdout:(37/160): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-07-26T06:36:07.884 INFO:teuthology.orchestra.run.smithi042.stdout:(38/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 156 kB/s | 18 kB 00:00 2024-07-26T06:36:07.917 INFO:teuthology.orchestra.run.smithi042.stdout:(39/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 4.3 MB/s | 771 kB 00:00 2024-07-26T06:36:08.226 INFO:teuthology.orchestra.run.smithi042.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 217 kB/s | 80 kB 00:00 2024-07-26T06:36:08.426 INFO:teuthology.orchestra.run.smithi042.stdout:(41/160): libquadmath-11.4.1-3.el9.x86_64.rpm 384 kB/s | 195 kB 00:00 2024-07-26T06:36:08.577 INFO:teuthology.orchestra.run.smithi042.stdout:(42/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 723 kB/s | 253 kB 00:00 2024-07-26T06:36:08.635 INFO:teuthology.orchestra.run.smithi042.stdout:(43/160): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-07-26T06:36:08.686 INFO:teuthology.orchestra.run.smithi042.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 980 kB/s | 106 kB 00:00 2024-07-26T06:36:08.744 INFO:teuthology.orchestra.run.smithi042.stdout:(45/160): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-07-26T06:36:08.786 INFO:teuthology.orchestra.run.smithi042.stdout:(46/160): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-07-26T06:36:08.798 INFO:teuthology.orchestra.run.smithi163.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-07-26T06:36:08.853 INFO:teuthology.orchestra.run.smithi042.stdout:(47/160): python3-urllib3-1.26.5-6.el9.noarch.r 1.9 MB/s | 215 kB 00:00 2024-07-26T06:36:09.028 INFO:teuthology.orchestra.run.smithi042.stdout:(48/160): python3-cryptography-36.0.1-4.el9.x86 2.0 MB/s | 1.2 MB 00:00 2024-07-26T06:36:09.107 INFO:teuthology.orchestra.run.smithi163.stdout:(2/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 11 MB/s | 5.5 MB 00:00 2024-07-26T06:36:09.112 INFO:teuthology.orchestra.run.smithi042.stdout:(49/160): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2024-07-26T06:36:09.132 INFO:teuthology.orchestra.run.smithi163.stdout:(3/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 2.6 MB/s | 872 kB 00:00 2024-07-26T06:36:09.157 INFO:teuthology.orchestra.run.smithi163.stdout:(4/160): ceph-immutable-object-cache-19.0.0-532 2.9 MB/s | 148 kB 00:00 2024-07-26T06:36:09.163 INFO:teuthology.orchestra.run.smithi042.stdout:(50/160): elfutils-libelf-devel-0.191-4.el9.x86 432 kB/s | 22 kB 00:00 2024-07-26T06:36:09.196 INFO:teuthology.orchestra.run.smithi042.stdout:(51/160): bison-3.7.4-5.el9.x86_64.rpm 2.7 MB/s | 943 kB 00:00 2024-07-26T06:36:09.222 INFO:teuthology.orchestra.run.smithi042.stdout:(52/160): boost-program-options-1.75.0-8.el9.x8 554 kB/s | 107 kB 00:00 2024-07-26T06:36:09.247 INFO:teuthology.orchestra.run.smithi042.stdout:(53/160): flex-2.6.4-9.el9.x86_64.rpm 3.7 MB/s | 313 kB 00:00 2024-07-26T06:36:09.249 INFO:teuthology.orchestra.run.smithi163.stdout:(5/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 17 MB/s | 1.5 MB 00:00 2024-07-26T06:36:09.275 INFO:teuthology.orchestra.run.smithi042.stdout:(54/160): flexiblas-3.0.4-8.el9.x86_64.rpm 433 kB/s | 33 kB 00:00 2024-07-26T06:36:09.298 INFO:teuthology.orchestra.run.smithi042.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 362 kB/s | 18 kB 00:00 2024-07-26T06:36:09.323 INFO:teuthology.orchestra.run.smithi042.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-07-26T06:36:09.356 INFO:teuthology.orchestra.run.smithi042.stdout:(57/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-07-26T06:36:09.382 INFO:teuthology.orchestra.run.smithi042.stdout:(58/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 774 kB/s | 45 kB 00:00 2024-07-26T06:36:09.440 INFO:teuthology.orchestra.run.smithi042.stdout:(59/160): libtool-2.4.6-46.el9.x86_64.rpm 9.7 MB/s | 581 kB 00:00 2024-07-26T06:36:09.474 INFO:teuthology.orchestra.run.smithi042.stdout:(60/160): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-07-26T06:36:09.491 INFO:teuthology.orchestra.run.smithi163.stdout:(6/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 20 MB/s | 4.9 MB 00:00 2024-07-26T06:36:09.499 INFO:teuthology.orchestra.run.smithi042.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 434 kB/s | 25 kB 00:00 2024-07-26T06:36:09.525 INFO:teuthology.orchestra.run.smithi042.stdout:(62/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-07-26T06:36:09.541 INFO:teuthology.orchestra.run.smithi163.stdout:(7/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 5.5 MB/s | 2.3 MB 00:00 2024-07-26T06:36:09.633 INFO:teuthology.orchestra.run.smithi042.stdout:(63/160): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-07-26T06:36:09.667 INFO:teuthology.orchestra.run.smithi042.stdout:(64/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 292 kB 00:00 2024-07-26T06:36:09.692 INFO:teuthology.orchestra.run.smithi042.stdout:(65/160): openblas-0.3.21-2.el9.x86_64.rpm 601 kB/s | 35 kB 00:00 2024-07-26T06:36:09.942 INFO:teuthology.orchestra.run.smithi163.stdout:(8/160): ceph-common-19.0.0-5320.ge8f88bb1.el9. 15 MB/s | 20 MB 00:01 2024-07-26T06:36:09.992 INFO:teuthology.orchestra.run.smithi163.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 504 kB/s | 25 kB 00:00 2024-07-26T06:36:10.167 INFO:teuthology.orchestra.run.smithi042.stdout:(66/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.2 MB/s | 3.0 MB 00:00 2024-07-26T06:36:10.317 INFO:teuthology.orchestra.run.smithi163.stdout:(10/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 22 MB/s | 18 MB 00:00 2024-07-26T06:36:10.368 INFO:teuthology.orchestra.run.smithi163.stdout:(11/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 4.4 MB/s | 225 kB 00:00 2024-07-26T06:36:10.426 INFO:teuthology.orchestra.run.smithi163.stdout:(12/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 465 kB/s | 27 kB 00:00 2024-07-26T06:36:10.477 INFO:teuthology.orchestra.run.smithi163.stdout:(13/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 635 kB/s | 32 kB 00:00 2024-07-26T06:36:10.610 INFO:teuthology.orchestra.run.smithi163.stdout:(14/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el 9.7 MB/s | 10 MB 00:01 2024-07-26T06:36:10.652 INFO:teuthology.orchestra.run.smithi163.stdout:(15/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 4.3 MB/s | 765 kB 00:00 2024-07-26T06:36:10.677 INFO:teuthology.orchestra.run.smithi163.stdout:(16/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 2.3 MB/s | 158 kB 00:00 2024-07-26T06:36:10.706 INFO:teuthology.orchestra.run.smithi163.stdout:(17/160): librados-devel-19.0.0-5320.ge8f88bb1. 2.5 MB/s | 127 kB 00:00 2024-07-26T06:36:10.728 INFO:teuthology.orchestra.run.smithi163.stdout:(18/160): libradosstriper1-19.0.0-5320.ge8f88bb 9.8 MB/s | 503 kB 00:00 2024-07-26T06:36:10.778 INFO:teuthology.orchestra.run.smithi163.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 902 kB/s | 45 kB 00:00 2024-07-26T06:36:10.853 INFO:teuthology.orchestra.run.smithi163.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 1.8 MB/s | 138 kB 00:00 2024-07-26T06:36:10.912 INFO:teuthology.orchestra.run.smithi163.stdout:(21/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 3.2 MB/s | 188 kB 00:00 2024-07-26T06:36:10.959 INFO:teuthology.orchestra.run.smithi042.stdout:(67/160): postgresql-13.14-1.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-07-26T06:36:10.979 INFO:teuthology.orchestra.run.smithi163.stdout:(22/160): python3-rados-19.0.0-5320.ge8f88bb1.e 5.5 MB/s | 372 kB 00:00 2024-07-26T06:36:11.054 INFO:teuthology.orchestra.run.smithi163.stdout:(23/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 15 MB/s | 5.3 MB 00:00 2024-07-26T06:36:11.080 INFO:teuthology.orchestra.run.smithi163.stdout:(24/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 3.5 MB/s | 355 kB 00:00 2024-07-26T06:36:11.105 INFO:teuthology.orchestra.run.smithi163.stdout:(25/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 2.3 MB/s | 116 kB 00:00 2024-07-26T06:36:11.130 INFO:teuthology.orchestra.run.smithi163.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.7 MB/s | 86 kB 00:00 2024-07-26T06:36:11.181 INFO:teuthology.orchestra.run.smithi163.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 3.3 MB/s | 170 kB 00:00 2024-07-26T06:36:11.247 INFO:teuthology.orchestra.run.smithi163.stdout:(28/160): ceph-grafana-dashboards-19.0.0-5320.g 516 kB/s | 34 kB 00:00 2024-07-26T06:36:11.306 INFO:teuthology.orchestra.run.smithi163.stdout:(29/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 15 MB/s | 3.1 MB 00:00 2024-07-26T06:36:11.332 INFO:teuthology.orchestra.run.smithi163.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 1.7 MB/s | 149 kB 00:00 2024-07-26T06:36:11.434 INFO:teuthology.orchestra.run.smithi042.stdout:(68/160): postgresql-contrib-13.14-1.el9.x86_64 1.9 MB/s | 911 kB 00:00 2024-07-26T06:36:11.552 INFO:teuthology.orchestra.run.smithi042.stdout:(69/160): postgresql-private-libs-13.14-1.el9.x 1.1 MB/s | 133 kB 00:00 2024-07-26T06:36:11.785 INFO:teuthology.orchestra.run.smithi042.stdout:(70/160): openblas-openmp-0.3.21-2.el9.x86_64.r 2.2 MB/s | 4.7 MB 00:02 2024-07-26T06:36:11.869 INFO:teuthology.orchestra.run.smithi042.stdout:(71/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 2.0 MB/s | 4.4 MB 00:02 2024-07-26T06:36:11.927 INFO:teuthology.orchestra.run.smithi042.stdout:(72/160): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-07-26T06:36:11.961 INFO:teuthology.orchestra.run.smithi042.stdout:(73/160): protobuf-3.14.0-13.el9.x86_64.rpm 5.7 MB/s | 1.0 MB 00:00 2024-07-26T06:36:12.036 INFO:teuthology.orchestra.run.smithi042.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 816 kB/s | 61 kB 00:00 2024-07-26T06:36:12.040 INFO:teuthology.orchestra.run.smithi163.stdout:(31/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 22 MB/s | 44 MB 00:02 2024-07-26T06:36:12.140 INFO:teuthology.orchestra.run.smithi163.stdout:(32/160): ceph-mgr-diskprediction-local-19.0.0- 9.2 MB/s | 7.4 MB 00:00 2024-07-26T06:36:12.165 INFO:teuthology.orchestra.run.smithi163.stdout:(33/160): ceph-mgr-modules-core-19.0.0-5320.ge8 2.2 MB/s | 284 kB 00:00 2024-07-26T06:36:12.191 INFO:teuthology.orchestra.run.smithi163.stdout:(34/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 988 kB/s | 49 kB 00:00 2024-07-26T06:36:12.195 INFO:teuthology.orchestra.run.smithi042.stdout:(75/160): python3-devel-3.9.18-3.el9.x86_64.rpm 1.5 MB/s | 244 kB 00:00 2024-07-26T06:36:12.216 INFO:teuthology.orchestra.run.smithi163.stdout:(35/160): ceph-prometheus-alerts-19.0.0-5320.ge 327 kB/s | 16 kB 00:00 2024-07-26T06:36:12.241 INFO:teuthology.orchestra.run.smithi163.stdout:(36/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 5.3 MB/s | 274 kB 00:00 2024-07-26T06:36:12.283 INFO:teuthology.orchestra.run.smithi163.stdout:(37/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 438 kB/s | 18 kB 00:00 2024-07-26T06:36:12.319 INFO:teuthology.orchestra.run.smithi163.stdout:(38/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 7.3 MB/s | 771 kB 00:00 2024-07-26T06:36:12.337 INFO:teuthology.orchestra.run.smithi042.stdout:(76/160): python3-jinja2-2.11.3-6.el9.noarch.rp 1.7 MB/s | 248 kB 00:00 2024-07-26T06:36:12.388 INFO:teuthology.orchestra.run.smithi042.stdout:(77/160): python3-jsonpatch-1.21-16.el9.noarch. 510 kB/s | 26 kB 00:00 2024-07-26T06:36:12.438 INFO:teuthology.orchestra.run.smithi042.stdout:(78/160): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-07-26T06:36:12.478 INFO:teuthology.orchestra.run.smithi163.stdout:(39/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 12 MB/s | 14 MB 00:01 2024-07-26T06:36:12.563 INFO:teuthology.orchestra.run.smithi042.stdout:(79/160): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-07-26T06:36:12.613 INFO:teuthology.orchestra.run.smithi042.stdout:(80/160): python3-markupsafe-1.1.1-12.el9.x86_6 697 kB/s | 35 kB 00:00 2024-07-26T06:36:12.708 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-07-26T06:36:13.062 INFO:teuthology.orchestra.run.smithi163.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 103 kB/s | 80 kB 00:00 2024-07-26T06:36:13.228 INFO:teuthology.orchestra.run.smithi163.stdout:(41/160): libquadmath-11.4.1-3.el9.x86_64.rpm 260 kB/s | 195 kB 00:00 2024-07-26T06:36:13.379 INFO:teuthology.orchestra.run.smithi163.stdout:(42/160): libgfortran-11.4.1-3.el9.x86_64.rpm 758 kB/s | 803 kB 00:01 2024-07-26T06:36:13.879 INFO:teuthology.orchestra.run.smithi163.stdout:(43/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 310 kB/s | 253 kB 00:00 2024-07-26T06:36:14.080 INFO:teuthology.orchestra.run.smithi042.stdout:(81/160): postgresql-server-13.14-1.el9.x86_64. 2.3 MB/s | 5.8 MB 00:02 2024-07-26T06:36:14.112 INFO:teuthology.orchestra.run.smithi163.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 145 kB/s | 106 kB 00:00 2024-07-26T06:36:14.231 INFO:teuthology.orchestra.run.smithi042.stdout:(82/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 250 kB 00:00 2024-07-26T06:36:14.262 INFO:teuthology.orchestra.run.smithi163.stdout:(45/160): python3-cryptography-36.0.1-4.el9.x86 1.2 MB/s | 1.2 MB 00:01 2024-07-26T06:36:14.389 INFO:teuthology.orchestra.run.smithi042.stdout:(83/160): python3-babel-2.9.1-2.el9.noarch.rpm 2.4 MB/s | 6.0 MB 00:02 2024-07-26T06:36:14.414 INFO:teuthology.orchestra.run.smithi042.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 222 kB 00:00 2024-07-26T06:36:14.465 INFO:teuthology.orchestra.run.smithi042.stdout:(85/160): python3-prettytable-0.7.2-27.el9.noar 844 kB/s | 42 kB 00:00 2024-07-26T06:36:14.490 INFO:teuthology.orchestra.run.smithi042.stdout:(86/160): python3-packaging-20.9-5.el9.noarch.r 769 kB/s | 77 kB 00:00 2024-07-26T06:36:14.599 INFO:teuthology.orchestra.run.smithi042.stdout:(87/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-07-26T06:36:14.629 INFO:teuthology.orchestra.run.smithi163.stdout:(46/160): python3-pycparser-2.20-6.el9.noarch.r 180 kB/s | 135 kB 00:00 2024-07-26T06:36:14.649 INFO:teuthology.orchestra.run.smithi042.stdout:(88/160): python3-protobuf-3.14.0-13.el9.noarch 1.4 MB/s | 269 kB 00:00 2024-07-26T06:36:14.699 INFO:teuthology.orchestra.run.smithi042.stdout:(89/160): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-07-26T06:36:14.758 INFO:teuthology.orchestra.run.smithi042.stdout:(90/160): python3-requests-oauthlib-1.3.0-12.el 923 kB/s | 54 kB 00:00 2024-07-26T06:36:14.808 INFO:teuthology.orchestra.run.smithi042.stdout:(91/160): python3-pyasn1-modules-0.4.8-6.el9.no 1.3 MB/s | 279 kB 00:00 2024-07-26T06:36:14.846 INFO:teuthology.orchestra.run.smithi163.stdout:(47/160): automake-1.16.2-8.el9.noarch.rpm 3.1 MB/s | 689 kB 00:00 2024-07-26T06:36:14.867 INFO:teuthology.orchestra.run.smithi042.stdout:(92/160): python3-toml-0.10.2-6.el9.noarch.rpm 716 kB/s | 42 kB 00:00 2024-07-26T06:36:14.871 INFO:teuthology.orchestra.run.smithi163.stdout:(48/160): python3-requests-2.25.1-8.el9.noarch. 171 kB/s | 125 kB 00:00 2024-07-26T06:36:14.926 INFO:teuthology.orchestra.run.smithi042.stdout:(93/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 757 kB/s | 44 kB 00:00 2024-07-26T06:36:14.939 INFO:teuthology.orchestra.run.smithi164.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T06:36:14.940 INFO:teuthology.orchestra.run.smithi164.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T06:36:14.951 INFO:teuthology.orchestra.run.smithi164.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T06:36:14.951 INFO:teuthology.orchestra.run.smithi164.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T06:36:14.952 INFO:teuthology.orchestra.run.smithi164.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T06:36:14.953 INFO:teuthology.orchestra.run.smithi164.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T06:36:14.955 INFO:teuthology.orchestra.run.smithi164.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T06:36:14.957 INFO:teuthology.orchestra.run.smithi164.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T06:36:14.958 INFO:teuthology.orchestra.run.smithi164.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T06:36:15.014 INFO:teuthology.orchestra.run.smithi163.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 755 kB/s | 107 kB 00:00 2024-07-26T06:36:15.020 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-07-26T06:36:15.041 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================= 2024-07-26T06:36:15.041 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-07-26T06:36:15.041 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================= 2024-07-26T06:36:15.041 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-07-26T06:36:15.041 INFO:teuthology.orchestra.run.smithi164.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T06:36:15.042 INFO:teuthology.orchestra.run.smithi164.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T06:36:15.042 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T06:36:15.042 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T06:36:15.042 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T06:36:15.042 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T06:36:15.042 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T06:36:15.042 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T06:36:15.042 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T06:36:15.042 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T06:36:15.042 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T06:36:15.043 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T06:36:15.043 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T06:36:15.043 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T06:36:15.043 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T06:36:15.043 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T06:36:15.043 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T06:36:15.043 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T06:36:15.043 INFO:teuthology.orchestra.run.smithi164.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T06:36:15.043 INFO:teuthology.orchestra.run.smithi164.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T06:36:15.043 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T06:36:15.043 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T06:36:15.044 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T06:36:15.044 INFO:teuthology.orchestra.run.smithi164.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T06:36:15.044 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T06:36:15.044 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T06:36:15.044 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T06:36:15.044 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T06:36:15.044 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T06:36:15.044 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T06:36:15.044 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T06:36:15.044 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T06:36:15.044 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T06:36:15.045 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T06:36:15.045 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T06:36:15.045 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T06:36:15.045 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T06:36:15.045 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrading: 2024-07-26T06:36:15.045 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T06:36:15.045 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T06:36:15.045 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T06:36:15.045 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T06:36:15.045 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T06:36:15.045 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T06:36:15.046 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T06:36:15.046 INFO:teuthology.orchestra.run.smithi164.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T06:36:15.046 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T06:36:15.046 INFO:teuthology.orchestra.run.smithi164.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T06:36:15.046 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T06:36:15.046 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T06:36:15.046 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-07-26T06:36:15.046 INFO:teuthology.orchestra.run.smithi164.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T06:36:15.047 INFO:teuthology.orchestra.run.smithi164.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T06:36:15.047 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T06:36:15.047 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T06:36:15.047 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T06:36:15.047 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T06:36:15.047 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T06:36:15.047 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T06:36:15.047 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T06:36:15.047 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T06:36:15.047 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T06:36:15.047 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T06:36:15.048 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T06:36:15.048 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T06:36:15.048 INFO:teuthology.orchestra.run.smithi164.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T06:36:15.048 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T06:36:15.048 INFO:teuthology.orchestra.run.smithi164.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T06:36:15.048 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T06:36:15.048 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T06:36:15.048 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T06:36:15.048 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T06:36:15.048 INFO:teuthology.orchestra.run.smithi164.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T06:36:15.049 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T06:36:15.049 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T06:36:15.049 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T06:36:15.049 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T06:36:15.049 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T06:36:15.049 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T06:36:15.049 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T06:36:15.049 INFO:teuthology.orchestra.run.smithi164.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T06:36:15.049 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T06:36:15.049 INFO:teuthology.orchestra.run.smithi164.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T06:36:15.049 INFO:teuthology.orchestra.run.smithi164.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T06:36:15.049 INFO:teuthology.orchestra.run.smithi164.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T06:36:15.050 INFO:teuthology.orchestra.run.smithi164.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T06:36:15.050 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T06:36:15.050 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T06:36:15.050 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T06:36:15.050 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T06:36:15.050 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T06:36:15.050 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T06:36:15.050 INFO:teuthology.orchestra.run.smithi164.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T06:36:15.050 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T06:36:15.050 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T06:36:15.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T06:36:15.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T06:36:15.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T06:36:15.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T06:36:15.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T06:36:15.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T06:36:15.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T06:36:15.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T06:36:15.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T06:36:15.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T06:36:15.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T06:36:15.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T06:36:15.051 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T06:36:15.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T06:36:15.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T06:36:15.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T06:36:15.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T06:36:15.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T06:36:15.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T06:36:15.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T06:36:15.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T06:36:15.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T06:36:15.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T06:36:15.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T06:36:15.052 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T06:36:15.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T06:36:15.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T06:36:15.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T06:36:15.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T06:36:15.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T06:36:15.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T06:36:15.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T06:36:15.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T06:36:15.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T06:36:15.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T06:36:15.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T06:36:15.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T06:36:15.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T06:36:15.053 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T06:36:15.054 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T06:36:15.054 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T06:36:15.054 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T06:36:15.054 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T06:36:15.054 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T06:36:15.054 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T06:36:15.054 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T06:36:15.054 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T06:36:15.054 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T06:36:15.054 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T06:36:15.054 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T06:36:15.055 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T06:36:15.055 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T06:36:15.055 INFO:teuthology.orchestra.run.smithi164.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T06:36:15.055 INFO:teuthology.orchestra.run.smithi164.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T06:36:15.055 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T06:36:15.055 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T06:36:15.055 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T06:36:15.055 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T06:36:15.055 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T06:36:15.055 INFO:teuthology.orchestra.run.smithi164.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T06:36:15.055 INFO:teuthology.orchestra.run.smithi164.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T06:36:15.055 INFO:teuthology.orchestra.run.smithi164.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T06:36:15.055 INFO:teuthology.orchestra.run.smithi164.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T06:36:15.055 INFO:teuthology.orchestra.run.smithi164.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T06:36:15.056 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T06:36:15.056 INFO:teuthology.orchestra.run.smithi164.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T06:36:15.056 INFO:teuthology.orchestra.run.smithi164.stdout:Installing weak dependencies: 2024-07-26T06:36:15.056 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T06:36:15.056 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:36:15.056 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-07-26T06:36:15.056 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================= 2024-07-26T06:36:15.056 INFO:teuthology.orchestra.run.smithi164.stdout:Install 148 Packages 2024-07-26T06:36:15.056 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrade 12 Packages 2024-07-26T06:36:15.057 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:36:15.057 INFO:teuthology.orchestra.run.smithi163.stdout:(50/160): python3-urllib3-1.26.5-6.el9.noarch.r 274 kB/s | 215 kB 00:00 2024-07-26T06:36:15.057 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 233 M 2024-07-26T06:36:15.057 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-07-26T06:36:15.067 INFO:teuthology.orchestra.run.smithi042.stdout:(94/160): qatlib-24.02.0-1.el9.x86_64.rpm 1.5 MB/s | 221 kB 00:00 2024-07-26T06:36:15.097 INFO:teuthology.orchestra.run.smithi163.stdout:(51/160): bison-3.7.4-5.el9.x86_64.rpm 3.7 MB/s | 943 kB 00:00 2024-07-26T06:36:15.117 INFO:teuthology.orchestra.run.smithi042.stdout:(95/160): qatlib-service-24.02.0-1.el9.x86_64.r 719 kB/s | 36 kB 00:00 2024-07-26T06:36:15.122 INFO:teuthology.orchestra.run.smithi163.stdout:(52/160): elfutils-libelf-devel-0.191-4.el9.x86 200 kB/s | 22 kB 00:00 2024-07-26T06:36:15.176 INFO:teuthology.orchestra.run.smithi042.stdout:(96/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 811 kB/s | 47 kB 00:00 2024-07-26T06:36:15.198 INFO:teuthology.orchestra.run.smithi163.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 326 kB/s | 33 kB 00:00 2024-07-26T06:36:15.256 INFO:teuthology.orchestra.run.smithi163.stdout:(54/160): flex-2.6.4-9.el9.x86_64.rpm 1.5 MB/s | 313 kB 00:00 2024-07-26T06:36:15.298 INFO:teuthology.orchestra.run.smithi163.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 182 kB/s | 18 kB 00:00 2024-07-26T06:36:15.302 INFO:teuthology.orchestra.run.smithi042.stdout:(97/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.9 MB/s | 5.1 MB 00:02 2024-07-26T06:36:15.349 INFO:teuthology.orchestra.run.smithi163.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 121 kB/s | 11 kB 00:00 2024-07-26T06:36:15.352 INFO:teuthology.orchestra.run.smithi042.stdout:(98/160): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-07-26T06:36:15.411 INFO:teuthology.orchestra.run.smithi042.stdout:(99/160): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-07-26T06:36:15.432 INFO:teuthology.orchestra.run.smithi163.stdout:(57/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.7 MB/s | 3.0 MB 00:00 2024-07-26T06:36:15.469 INFO:teuthology.orchestra.run.smithi042.stdout:(100/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 817 kB/s | 96 kB 00:00 2024-07-26T06:36:15.499 INFO:teuthology.orchestra.run.smithi163.stdout:(58/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 799 kB/s | 160 kB 00:00 2024-07-26T06:36:15.520 INFO:teuthology.orchestra.run.smithi042.stdout:(101/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 586 kB/s | 64 kB 00:00 2024-07-26T06:36:15.524 INFO:teuthology.orchestra.run.smithi163.stdout:(59/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 258 kB/s | 45 kB 00:00 2024-07-26T06:36:15.545 INFO:teuthology.orchestra.run.smithi042.stdout:(102/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 594 kB/s | 45 kB 00:00 2024-07-26T06:36:15.649 INFO:teuthology.orchestra.run.smithi163.stdout:(60/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 203 kB/s | 25 kB 00:00 2024-07-26T06:36:15.675 INFO:teuthology.orchestra.run.smithi163.stdout:(61/160): librdkafka-1.6.1-102.el9.x86_64.rpm 2.7 MB/s | 662 kB 00:00 2024-07-26T06:36:15.695 INFO:teuthology.orchestra.run.smithi042.stdout:(103/160): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2024-07-26T06:36:15.734 INFO:teuthology.orchestra.run.smithi163.stdout:(62/160): libtool-2.4.6-46.el9.x86_64.rpm 2.4 MB/s | 581 kB 00:00 2024-07-26T06:36:15.779 INFO:teuthology.orchestra.run.smithi042.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 6.4 MB/s | 548 kB 00:00 2024-07-26T06:36:15.804 INFO:teuthology.orchestra.run.smithi042.stdout:(105/160): gperftools-libs-2.9.1-2.el9.x86_64.r 12 MB/s | 309 kB 00:00 2024-07-26T06:36:15.809 INFO:teuthology.orchestra.run.smithi163.stdout:(63/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 298 kB/s | 47 kB 00:00 2024-07-26T06:36:15.821 INFO:teuthology.orchestra.run.smithi042.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-07-26T06:36:15.842 INFO:teuthology.orchestra.run.smithi164.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-07-26T06:36:15.855 INFO:teuthology.orchestra.run.smithi042.stdout:(107/160): protobuf-compiler-3.14.0-13.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-07-26T06:36:15.875 INFO:teuthology.orchestra.run.smithi163.stdout:(64/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-07-26T06:36:15.909 INFO:teuthology.orchestra.run.smithi163.stdout:(65/160): openblas-0.3.21-2.el9.x86_64.rpm 352 kB/s | 35 kB 00:00 2024-07-26T06:36:15.914 INFO:teuthology.orchestra.run.smithi042.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-07-26T06:36:15.930 INFO:teuthology.orchestra.run.smithi042.stdout:(109/160): liboath-2.6.7-2.el9.x86_64.rpm 2.9 MB/s | 49 kB 00:00 2024-07-26T06:36:15.935 INFO:teuthology.orchestra.run.smithi163.stdout:(66/160): lua-5.4.4-4.el9.x86_64.rpm 939 kB/s | 188 kB 00:00 2024-07-26T06:36:15.956 INFO:teuthology.orchestra.run.smithi042.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-07-26T06:36:15.990 INFO:teuthology.orchestra.run.smithi042.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 4.4 MB/s | 151 kB 00:00 2024-07-26T06:36:16.065 INFO:teuthology.orchestra.run.smithi042.stdout:(112/160): libarrow-9.0.0-11.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-07-26T06:36:16.067 INFO:teuthology.orchestra.run.smithi164.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 3.8 MB/s | 872 kB 00:00 2024-07-26T06:36:16.107 INFO:teuthology.orchestra.run.smithi042.stdout:(113/160): python3-asyncssh-2.13.2-5.el9.noarch 13 MB/s | 548 kB 00:00 2024-07-26T06:36:16.140 INFO:teuthology.orchestra.run.smithi042.stdout:(114/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.4 MB/s | 839 kB 00:00 2024-07-26T06:36:16.142 INFO:teuthology.orchestra.run.smithi164.stdout:(3/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 12 MB/s | 5.5 MB 00:00 2024-07-26T06:36:16.166 INFO:teuthology.orchestra.run.smithi042.stdout:(115/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 742 kB/s | 43 kB 00:00 2024-07-26T06:36:16.167 INFO:teuthology.orchestra.run.smithi164.stdout:(4/160): ceph-immutable-object-cache-19.0.0-532 1.4 MB/s | 148 kB 00:00 2024-07-26T06:36:16.191 INFO:teuthology.orchestra.run.smithi042.stdout:(116/160): python3-cachetools-4.2.4-1.el9.noarc 639 kB/s | 32 kB 00:00 2024-07-26T06:36:16.216 INFO:teuthology.orchestra.run.smithi042.stdout:(117/160): python3-certifi-2023.05.07-4.el9.noa 281 kB/s | 14 kB 00:00 2024-07-26T06:36:16.234 INFO:teuthology.orchestra.run.smithi163.stdout:(67/160): openblas-openmp-0.3.21-2.el9.x86_64.r 13 MB/s | 4.7 MB 00:00 2024-07-26T06:36:16.242 INFO:teuthology.orchestra.run.smithi042.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-07-26T06:36:16.251 INFO:teuthology.orchestra.run.smithi164.stdout:(5/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 21 MB/s | 2.3 MB 00:00 2024-07-26T06:36:16.267 INFO:teuthology.orchestra.run.smithi042.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-07-26T06:36:16.292 INFO:teuthology.orchestra.run.smithi042.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-07-26T06:36:16.293 INFO:teuthology.orchestra.run.smithi163.stdout:(68/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-07-26T06:36:16.317 INFO:teuthology.orchestra.run.smithi042.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 4.2 MB/s | 216 kB 00:00 2024-07-26T06:36:16.343 INFO:teuthology.orchestra.run.smithi042.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 5.6 MB/s | 144 kB 00:00 2024-07-26T06:36:16.344 INFO:teuthology.orchestra.run.smithi163.stdout:(69/160): postgresql-13.14-1.el9.x86_64.rpm 3.9 MB/s | 1.6 MB 00:00 2024-07-26T06:36:16.360 INFO:teuthology.orchestra.run.smithi164.stdout:(6/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 7.9 MB/s | 1.5 MB 00:00 2024-07-26T06:36:16.368 INFO:teuthology.orchestra.run.smithi042.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 427 kB/s | 11 kB 00:00 2024-07-26T06:36:16.394 INFO:teuthology.orchestra.run.smithi042.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 705 kB/s | 18 kB 00:00 2024-07-26T06:36:16.419 INFO:teuthology.orchestra.run.smithi042.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 936 kB/s | 23 kB 00:00 2024-07-26T06:36:16.443 INFO:teuthology.orchestra.run.smithi164.stdout:(7/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 25 MB/s | 4.9 MB 00:00 2024-07-26T06:36:16.444 INFO:teuthology.orchestra.run.smithi042.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 774 kB/s | 19 kB 00:00 2024-07-26T06:36:16.469 INFO:teuthology.orchestra.run.smithi042.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 793 kB/s | 20 kB 00:00 2024-07-26T06:36:16.502 INFO:teuthology.orchestra.run.smithi163.stdout:(70/160): postgresql-private-libs-13.14-1.el9.x 639 kB/s | 133 kB 00:00 2024-07-26T06:36:16.513 INFO:teuthology.orchestra.run.smithi042.stdout:(128/160): python3-grpcio-1.46.7-10.el9.x86_64. 9.3 MB/s | 2.0 MB 00:00 2024-07-26T06:36:16.537 INFO:teuthology.orchestra.run.smithi042.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-07-26T06:36:16.562 INFO:teuthology.orchestra.run.smithi042.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 3.0 MB/s | 79 kB 00:00 2024-07-26T06:36:16.577 INFO:teuthology.orchestra.run.smithi163.stdout:(71/160): postgresql-contrib-13.14-1.el9.x86_64 2.6 MB/s | 911 kB 00:00 2024-07-26T06:36:16.579 INFO:teuthology.orchestra.run.smithi042.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 3.4 MB/s | 58 kB 00:00 2024-07-26T06:36:16.613 INFO:teuthology.orchestra.run.smithi042.stdout:(132/160): python3-kubernetes-26.1.0-2.el9.noar 7.1 MB/s | 1.0 MB 00:00 2024-07-26T06:36:16.630 INFO:teuthology.orchestra.run.smithi042.stdout:(133/160): python3-portend-3.1.0-2.el9.noarch.r 985 kB/s | 16 kB 00:00 2024-07-26T06:36:16.655 INFO:teuthology.orchestra.run.smithi042.stdout:(134/160): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-07-26T06:36:16.681 INFO:teuthology.orchestra.run.smithi042.stdout:(135/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-07-26T06:36:16.706 INFO:teuthology.orchestra.run.smithi042.stdout:(136/160): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-07-26T06:36:16.731 INFO:teuthology.orchestra.run.smithi042.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-07-26T06:36:16.757 INFO:teuthology.orchestra.run.smithi042.stdout:(138/160): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-07-26T06:36:16.844 INFO:teuthology.orchestra.run.smithi164.stdout:(8/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el9 26 MB/s | 10 MB 00:00 2024-07-26T06:36:16.845 INFO:teuthology.orchestra.run.smithi163.stdout:(72/160): python3-attrs-20.3.0-7.el9.noarch.rpm 328 kB/s | 87 kB 00:00 2024-07-26T06:36:16.849 INFO:teuthology.orchestra.run.smithi042.stdout:(139/160): python3-rsa-4.9-2.el9.noarch.rpm 505 kB/s | 59 kB 00:00 2024-07-26T06:36:16.874 INFO:teuthology.orchestra.run.smithi042.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 305 kB/s | 36 kB 00:00 2024-07-26T06:36:16.878 INFO:teuthology.orchestra.run.smithi163.stdout:(73/160): protobuf-3.14.0-13.el9.x86_64.rpm 2.7 MB/s | 1.0 MB 00:00 2024-07-26T06:36:16.894 INFO:teuthology.orchestra.run.smithi164.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 506 kB/s | 25 kB 00:00 2024-07-26T06:36:16.899 INFO:teuthology.orchestra.run.smithi042.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-07-26T06:36:16.925 INFO:teuthology.orchestra.run.smithi042.stdout:(142/160): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-07-26T06:36:16.950 INFO:teuthology.orchestra.run.smithi042.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-07-26T06:36:16.953 INFO:teuthology.orchestra.run.smithi163.stdout:(74/160): postgresql-server-13.14-1.el9.x86_64. 9.5 MB/s | 5.8 MB 00:00 2024-07-26T06:36:16.975 INFO:teuthology.orchestra.run.smithi042.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 805 kB/s | 20 kB 00:00 2024-07-26T06:36:17.000 INFO:teuthology.orchestra.run.smithi042.stdout:(145/160): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-07-26T06:36:17.026 INFO:teuthology.orchestra.run.smithi042.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-07-26T06:36:17.087 INFO:teuthology.orchestra.run.smithi163.stdout:(75/160): python3-colorama-0.4.6-3.el9.noarch.r 294 kB/s | 61 kB 00:00 2024-07-26T06:36:17.126 INFO:teuthology.orchestra.run.smithi042.stdout:(147/160): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-07-26T06:36:17.129 INFO:teuthology.orchestra.run.smithi163.stdout:(76/160): python3-devel-3.9.18-3.el9.x86_64.rpm 1.4 MB/s | 244 kB 00:00 2024-07-26T06:36:17.246 INFO:teuthology.orchestra.run.smithi163.stdout:(77/160): python3-jsonpatch-1.21-16.el9.noarch. 220 kB/s | 26 kB 00:00 2024-07-26T06:36:17.355 INFO:teuthology.orchestra.run.smithi163.stdout:(78/160): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-07-26T06:36:17.380 INFO:teuthology.orchestra.run.smithi163.stdout:(79/160): python3-jsonpointer-2.0-4.el9.noarch. 144 kB/s | 19 kB 00:00 2024-07-26T06:36:17.386 INFO:teuthology.orchestra.run.smithi164.stdout:(10/160): ceph-common-19.0.0-5320.ge8f88bb1.el9 12 MB/s | 20 MB 00:01 2024-07-26T06:36:17.397 INFO:teuthology.orchestra.run.smithi163.stdout:(80/160): python3-jinja2-2.11.3-6.el9.noarch.rp 803 kB/s | 248 kB 00:00 2024-07-26T06:36:17.443 INFO:teuthology.orchestra.run.smithi042.stdout:(148/160): librados2-19.0.0-5320.ge8f88bb1.el9. 8.1 MB/s | 3.4 MB 00:00 2024-07-26T06:36:17.444 INFO:teuthology.orchestra.run.smithi164.stdout:(11/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 3.8 MB/s | 225 kB 00:00 2024-07-26T06:36:17.514 INFO:teuthology.orchestra.run.smithi163.stdout:(81/160): python3-markupsafe-1.1.1-12.el9.x86_6 260 kB/s | 35 kB 00:00 2024-07-26T06:36:17.528 INFO:teuthology.orchestra.run.smithi164.stdout:(12/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 325 kB/s | 27 kB 00:00 2024-07-26T06:36:17.539 INFO:teuthology.orchestra.run.smithi163.stdout:(82/160): python3-mako-1.1.4-6.el9.noarch.rpm 937 kB/s | 172 kB 00:00 2024-07-26T06:36:17.568 INFO:teuthology.orchestra.run.smithi042.stdout:(149/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 7.1 MB/s | 3.1 MB 00:00 2024-07-26T06:36:17.579 INFO:teuthology.orchestra.run.smithi164.stdout:(13/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 637 kB/s | 32 kB 00:00 2024-07-26T06:36:17.662 INFO:teuthology.orchestra.run.smithi164.stdout:(14/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 9.0 MB/s | 765 kB 00:00 2024-07-26T06:36:17.681 INFO:teuthology.orchestra.run.smithi163.stdout:(83/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 250 kB 00:00 2024-07-26T06:36:17.729 INFO:teuthology.orchestra.run.smithi164.stdout:(15/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 2.3 MB/s | 158 kB 00:00 2024-07-26T06:36:17.748 INFO:teuthology.orchestra.run.smithi163.stdout:(84/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 5.1 MB 00:00 2024-07-26T06:36:17.773 INFO:teuthology.orchestra.run.smithi163.stdout:(85/160): python3-oauthlib-3.1.1-5.el9.noarch.r 950 kB/s | 222 kB 00:00 2024-07-26T06:36:17.849 INFO:teuthology.orchestra.run.smithi163.stdout:(86/160): python3-packaging-20.9-5.el9.noarch.r 461 kB/s | 77 kB 00:00 2024-07-26T06:36:17.915 INFO:teuthology.orchestra.run.smithi163.stdout:(87/160): python3-prettytable-0.7.2-27.el9.noar 253 kB/s | 42 kB 00:00 2024-07-26T06:36:17.921 INFO:teuthology.orchestra.run.smithi164.stdout:(16/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 11 MB/s | 18 MB 00:01 2024-07-26T06:36:17.946 INFO:teuthology.orchestra.run.smithi164.stdout:(17/160): librados-devel-19.0.0-5320.ge8f88bb1. 584 kB/s | 127 kB 00:00 2024-07-26T06:36:17.965 INFO:teuthology.orchestra.run.smithi163.stdout:(88/160): python3-protobuf-3.14.0-13.el9.noarch 1.4 MB/s | 269 kB 00:00 2024-07-26T06:36:17.971 INFO:teuthology.orchestra.run.smithi164.stdout:(18/160): libradosstriper1-19.0.0-5320.ge8f88bb 9.9 MB/s | 503 kB 00:00 2024-07-26T06:36:18.022 INFO:teuthology.orchestra.run.smithi164.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 901 kB/s | 45 kB 00:00 2024-07-26T06:36:18.091 INFO:teuthology.orchestra.run.smithi163.stdout:(89/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 656 kB/s | 159 kB 00:00 2024-07-26T06:36:18.114 INFO:teuthology.orchestra.run.smithi164.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 1.5 MB/s | 138 kB 00:00 2024-07-26T06:36:18.116 INFO:teuthology.orchestra.run.smithi163.stdout:(90/160): python3-pytz-2021.1-5.el9.noarch.rpm 339 kB/s | 51 kB 00:00 2024-07-26T06:36:18.174 INFO:teuthology.orchestra.run.smithi163.stdout:(91/160): python3-pyasn1-modules-0.4.8-6.el9.no 1.1 MB/s | 279 kB 00:00 2024-07-26T06:36:18.189 INFO:teuthology.orchestra.run.smithi164.stdout:(21/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 22 MB/s | 5.3 MB 00:00 2024-07-26T06:36:18.214 INFO:teuthology.orchestra.run.smithi164.stdout:(22/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 1.8 MB/s | 188 kB 00:00 2024-07-26T06:36:18.240 INFO:teuthology.orchestra.run.smithi164.stdout:(23/160): python3-rados-19.0.0-5320.ge8f88bb1.e 7.2 MB/s | 372 kB 00:00 2024-07-26T06:36:18.258 INFO:teuthology.orchestra.run.smithi163.stdout:(92/160): python3-requests-oauthlib-1.3.0-12.el 321 kB/s | 54 kB 00:00 2024-07-26T06:36:18.265 INFO:teuthology.orchestra.run.smithi164.stdout:(24/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 6.9 MB/s | 355 kB 00:00 2024-07-26T06:36:18.290 INFO:teuthology.orchestra.run.smithi164.stdout:(25/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 2.3 MB/s | 116 kB 00:00 2024-07-26T06:36:18.300 INFO:teuthology.orchestra.run.smithi163.stdout:(93/160): python3-toml-0.10.2-6.el9.noarch.rpm 332 kB/s | 42 kB 00:00 2024-07-26T06:36:18.315 INFO:teuthology.orchestra.run.smithi164.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.7 MB/s | 86 kB 00:00 2024-07-26T06:36:18.374 INFO:teuthology.orchestra.run.smithi164.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 2.8 MB/s | 170 kB 00:00 2024-07-26T06:36:18.434 INFO:teuthology.orchestra.run.smithi163.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 252 kB/s | 44 kB 00:00 2024-07-26T06:36:18.435 INFO:teuthology.orchestra.run.smithi042.stdout:(150/160): NetworkManager-1.48.2-2.el9.x86_64.r 2.3 MB/s | 2.3 MB 00:00 2024-07-26T06:36:18.441 INFO:teuthology.orchestra.run.smithi164.stdout:(28/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 20 MB/s | 3.1 MB 00:00 2024-07-26T06:36:18.466 INFO:teuthology.orchestra.run.smithi164.stdout:(29/160): ceph-grafana-dashboards-19.0.0-5320.g 374 kB/s | 34 kB 00:00 2024-07-26T06:36:18.492 INFO:teuthology.orchestra.run.smithi164.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 2.9 MB/s | 149 kB 00:00 2024-07-26T06:36:18.534 INFO:teuthology.orchestra.run.smithi163.stdout:(95/160): qatlib-24.02.0-1.el9.x86_64.rpm 946 kB/s | 221 kB 00:00 2024-07-26T06:36:18.535 INFO:teuthology.orchestra.run.smithi042.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 409 kB/s | 41 kB 00:00 2024-07-26T06:36:18.576 INFO:teuthology.orchestra.run.smithi163.stdout:(96/160): qatlib-service-24.02.0-1.el9.x86_64.r 253 kB/s | 36 kB 00:00 2024-07-26T06:36:18.586 INFO:teuthology.orchestra.run.smithi042.stdout:(152/160): NetworkManager-libnm-1.48.2-2.el9.x8 1.8 MB/s | 1.8 MB 00:01 2024-07-26T06:36:18.645 INFO:teuthology.orchestra.run.smithi042.stdout:(153/160): NetworkManager-tui-1.48.2-2.el9.x86_ 2.2 MB/s | 247 kB 00:00 2024-07-26T06:36:18.686 INFO:teuthology.orchestra.run.smithi042.stdout:(154/160): elfutils-debuginfod-client-0.191-4.e 366 kB/s | 37 kB 00:00 2024-07-26T06:36:18.754 INFO:teuthology.orchestra.run.smithi042.stdout:(155/160): elfutils-libelf-0.191-4.el9.x86_64.r 1.9 MB/s | 207 kB 00:00 2024-07-26T06:36:18.793 INFO:teuthology.orchestra.run.smithi163.stdout:(97/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-07-26T06:36:18.796 INFO:teuthology.orchestra.run.smithi042.stdout:(156/160): elfutils-libs-0.191-4.el9.x86_64.rpm 2.3 MB/s | 260 kB 00:00 2024-07-26T06:36:18.818 INFO:teuthology.orchestra.run.smithi163.stdout:(98/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 167 kB/s | 47 kB 00:00 2024-07-26T06:36:18.888 INFO:teuthology.orchestra.run.smithi042.stdout:(157/160): openssl-3.2.2-2.el9.x86_64.rpm 10 MB/s | 1.4 MB 00:00 2024-07-26T06:36:18.918 INFO:teuthology.orchestra.run.smithi163.stdout:(99/160): socat-1.7.4.1-6.el9.x86_64.rpm 889 kB/s | 304 kB 00:00 2024-07-26T06:36:18.977 INFO:teuthology.orchestra.run.smithi163.stdout:(100/160): uuid-1.6.2-55.el9.x86_64.rpm 314 kB/s | 58 kB 00:00 2024-07-26T06:36:19.003 INFO:teuthology.orchestra.run.smithi163.stdout:(101/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 519 kB/s | 96 kB 00:00 2024-07-26T06:36:19.035 INFO:teuthology.orchestra.run.smithi042.stdout:(158/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 7.4 MB/s | 1.1 MB 00:00 2024-07-26T06:36:19.086 INFO:teuthology.orchestra.run.smithi163.stdout:(102/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 381 kB/s | 64 kB 00:00 2024-07-26T06:36:19.109 INFO:teuthology.orchestra.run.smithi164.stdout:(31/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 20 MB/s | 44 MB 00:02 2024-07-26T06:36:19.136 INFO:teuthology.orchestra.run.smithi163.stdout:(103/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 282 kB/s | 45 kB 00:00 2024-07-26T06:36:19.201 INFO:teuthology.orchestra.run.smithi164.stdout:(32/160): ceph-mgr-diskprediction-local-19.0.0- 10 MB/s | 7.4 MB 00:00 2024-07-26T06:36:19.211 INFO:teuthology.orchestra.run.smithi163.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 7.2 MB/s | 548 kB 00:00 2024-07-26T06:36:19.226 INFO:teuthology.orchestra.run.smithi164.stdout:(33/160): ceph-mgr-modules-core-19.0.0-5320.ge8 2.4 MB/s | 284 kB 00:00 2024-07-26T06:36:19.251 INFO:teuthology.orchestra.run.smithi164.stdout:(34/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 989 kB/s | 49 kB 00:00 2024-07-26T06:36:19.270 INFO:teuthology.orchestra.run.smithi163.stdout:(105/160): gperftools-libs-2.9.1-2.el9.x86_64.r 9.8 MB/s | 309 kB 00:00 2024-07-26T06:36:19.276 INFO:teuthology.orchestra.run.smithi164.stdout:(35/160): ceph-prometheus-alerts-19.0.0-5320.ge 329 kB/s | 16 kB 00:00 2024-07-26T06:36:19.295 INFO:teuthology.orchestra.run.smithi163.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 778 kB/s | 19 kB 00:00 2024-07-26T06:36:19.302 INFO:teuthology.orchestra.run.smithi164.stdout:(36/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 5.3 MB/s | 274 kB 00:00 2024-07-26T06:36:19.352 INFO:teuthology.orchestra.run.smithi164.stdout:(37/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 10 MB/s | 771 kB 00:00 2024-07-26T06:36:19.377 INFO:teuthology.orchestra.run.smithi164.stdout:(38/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 243 kB/s | 18 kB 00:00 2024-07-26T06:36:19.404 INFO:teuthology.orchestra.run.smithi163.stdout:(107/160): libarrow-9.0.0-11.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-07-26T06:36:19.429 INFO:teuthology.orchestra.run.smithi163.stdout:(108/160): lua-devel-5.4.4-4.el9.x86_64.rpm 52 kB/s | 22 kB 00:00 2024-07-26T06:36:19.454 INFO:teuthology.orchestra.run.smithi163.stdout:(109/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 519 kB/s | 26 kB 00:00 2024-07-26T06:36:19.505 INFO:teuthology.orchestra.run.smithi163.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-07-26T06:36:19.539 INFO:teuthology.orchestra.run.smithi163.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 4.8 MB/s | 151 kB 00:00 2024-07-26T06:36:19.578 INFO:teuthology.orchestra.run.smithi163.stdout:(112/160): liboath-2.6.7-2.el9.x86_64.rpm 326 kB/s | 49 kB 00:00 2024-07-26T06:36:19.622 INFO:teuthology.orchestra.run.smithi163.stdout:(113/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 839 kB 00:00 2024-07-26T06:36:19.647 INFO:teuthology.orchestra.run.smithi163.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-07-26T06:36:19.653 INFO:teuthology.orchestra.run.smithi164.stdout:(39/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 12 MB/s | 14 MB 00:01 2024-07-26T06:36:19.673 INFO:teuthology.orchestra.run.smithi163.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-07-26T06:36:19.698 INFO:teuthology.orchestra.run.smithi163.stdout:(116/160): python3-asyncssh-2.13.2-5.el9.noarch 4.5 MB/s | 548 kB 00:00 2024-07-26T06:36:19.723 INFO:teuthology.orchestra.run.smithi163.stdout:(117/160): python3-certifi-2023.05.07-4.el9.noa 281 kB/s | 14 kB 00:00 2024-07-26T06:36:19.749 INFO:teuthology.orchestra.run.smithi163.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-07-26T06:36:19.774 INFO:teuthology.orchestra.run.smithi163.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-07-26T06:36:19.799 INFO:teuthology.orchestra.run.smithi163.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-07-26T06:36:19.824 INFO:teuthology.orchestra.run.smithi163.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 4.2 MB/s | 216 kB 00:00 2024-07-26T06:36:19.858 INFO:teuthology.orchestra.run.smithi163.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-07-26T06:36:19.875 INFO:teuthology.orchestra.run.smithi163.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 653 kB/s | 11 kB 00:00 2024-07-26T06:36:19.908 INFO:teuthology.orchestra.run.smithi163.stdout:(124/160): protobuf-compiler-3.14.0-13.el9.x86_ 1.1 MB/s | 863 kB 00:00 2024-07-26T06:36:19.925 INFO:teuthology.orchestra.run.smithi163.stdout:(125/160): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-07-26T06:36:19.967 INFO:teuthology.orchestra.run.smithi163.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 1.2 MB/s | 19 kB 00:00 2024-07-26T06:36:19.993 INFO:teuthology.orchestra.run.smithi163.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 868 kB/s | 20 kB 00:00 2024-07-26T06:36:20.018 INFO:teuthology.orchestra.run.smithi163.stdout:(128/160): python3-jaraco-collections-3.0.0-8.e 212 kB/s | 23 kB 00:00 2024-07-26T06:36:20.052 INFO:teuthology.orchestra.run.smithi163.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.3 MB/s | 46 kB 00:00 2024-07-26T06:36:20.086 INFO:teuthology.orchestra.run.smithi163.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-07-26T06:36:20.111 INFO:teuthology.orchestra.run.smithi163.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-07-26T06:36:20.153 INFO:teuthology.orchestra.run.smithi163.stdout:(132/160): python3-grpcio-1.46.7-10.el9.x86_64. 5.8 MB/s | 2.0 MB 00:00 2024-07-26T06:36:20.170 INFO:teuthology.orchestra.run.smithi164.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 98 kB/s | 80 kB 00:00 2024-07-26T06:36:20.178 INFO:teuthology.orchestra.run.smithi163.stdout:(133/160): python3-portend-3.1.0-2.el9.noarch.r 658 kB/s | 16 kB 00:00 2024-07-26T06:36:20.211 INFO:teuthology.orchestra.run.smithi163.stdout:(134/160): python3-kubernetes-26.1.0-2.el9.noar 4.7 MB/s | 1.0 MB 00:00 2024-07-26T06:36:20.237 INFO:teuthology.orchestra.run.smithi163.stdout:(135/160): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-07-26T06:36:20.262 INFO:teuthology.orchestra.run.smithi163.stdout:(136/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-07-26T06:36:20.287 INFO:teuthology.orchestra.run.smithi163.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-07-26T06:36:20.313 INFO:teuthology.orchestra.run.smithi163.stdout:(138/160): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-07-26T06:36:20.338 INFO:teuthology.orchestra.run.smithi163.stdout:(139/160): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-07-26T06:36:20.355 INFO:teuthology.orchestra.run.smithi042.stdout:(159/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 1.5 MB/s | 2.4 MB 00:01 2024-07-26T06:36:20.363 INFO:teuthology.orchestra.run.smithi163.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 715 kB/s | 36 kB 00:00 2024-07-26T06:36:20.389 INFO:teuthology.orchestra.run.smithi163.stdout:(141/160): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-07-26T06:36:20.395 INFO:teuthology.orchestra.run.smithi164.stdout:(41/160): libquadmath-11.4.1-3.el9.x86_64.rpm 263 kB/s | 195 kB 00:00 2024-07-26T06:36:20.413 INFO:teuthology.orchestra.run.smithi163.stdout:(142/160): python3-typing-extensions-4.4.0-2.el 678 kB/s | 51 kB 00:00 2024-07-26T06:36:20.439 INFO:teuthology.orchestra.run.smithi163.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-07-26T06:36:20.462 INFO:teuthology.orchestra.run.smithi164.stdout:(42/160): libgfortran-11.4.1-3.el9.x86_64.rpm 741 kB/s | 803 kB 00:01 2024-07-26T06:36:20.464 INFO:teuthology.orchestra.run.smithi163.stdout:(144/160): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-07-26T06:36:20.490 INFO:teuthology.orchestra.run.smithi163.stdout:(145/160): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-07-26T06:36:20.523 INFO:teuthology.orchestra.run.smithi163.stdout:(146/160): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-07-26T06:36:20.548 INFO:teuthology.orchestra.run.smithi163.stdout:(147/160): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-07-26T06:36:20.599 INFO:teuthology.orchestra.run.smithi163.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-07-26T06:36:20.807 INFO:teuthology.orchestra.run.smithi163.stdout:(149/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 12 MB/s | 3.1 MB 00:00 2024-07-26T06:36:20.883 INFO:teuthology.orchestra.run.smithi163.stdout:(150/160): librados2-19.0.0-5320.ge8f88bb1.el9. 9.4 MB/s | 3.4 MB 00:00 2024-07-26T06:36:20.962 INFO:teuthology.orchestra.run.smithi164.stdout:(43/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 320 kB/s | 253 kB 00:00 2024-07-26T06:36:21.154 INFO:teuthology.orchestra.run.smithi164.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 154 kB/s | 106 kB 00:00 2024-07-26T06:36:21.396 INFO:teuthology.orchestra.run.smithi164.stdout:(45/160): python3-cryptography-36.0.1-4.el9.x86 1.3 MB/s | 1.2 MB 00:00 2024-07-26T06:36:21.508 INFO:teuthology.orchestra.run.smithi163.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 65 kB/s | 41 kB 00:00 2024-07-26T06:36:21.650 INFO:teuthology.orchestra.run.smithi163.stdout:(152/160): NetworkManager-1.48.2-2.el9.x86_64.r 2.2 MB/s | 2.3 MB 00:01 2024-07-26T06:36:21.688 INFO:teuthology.orchestra.run.smithi164.stdout:(46/160): python3-pycparser-2.20-6.el9.noarch.r 186 kB/s | 135 kB 00:00 2024-07-26T06:36:21.896 INFO:teuthology.orchestra.run.smithi164.stdout:(47/160): python3-requests-2.25.1-8.el9.noarch. 168 kB/s | 125 kB 00:00 2024-07-26T06:36:22.033 INFO:teuthology.orchestra.run.smithi163.stdout:(153/160): NetworkManager-libnm-1.48.2-2.el9.x8 1.5 MB/s | 1.8 MB 00:01 2024-07-26T06:36:22.180 INFO:teuthology.orchestra.run.smithi164.stdout:(48/160): python3-urllib3-1.26.5-6.el9.noarch.r 284 kB/s | 215 kB 00:00 2024-07-26T06:36:22.425 INFO:teuthology.orchestra.run.smithi163.stdout:(154/160): NetworkManager-tui-1.48.2-2.el9.x86_ 270 kB/s | 247 kB 00:00 2024-07-26T06:36:22.438 INFO:teuthology.orchestra.run.smithi164.stdout:(49/160): automake-1.16.2-8.el9.noarch.rpm 919 kB/s | 689 kB 00:00 2024-07-26T06:36:22.450 INFO:teuthology.orchestra.run.smithi163.stdout:(155/160): elfutils-debuginfod-client-0.191-4.e 46 kB/s | 37 kB 00:00 2024-07-26T06:36:22.464 INFO:teuthology.orchestra.run.smithi164.stdout:(50/160): boost-program-options-1.75.0-8.el9.x8 377 kB/s | 107 kB 00:00 2024-07-26T06:36:22.547 INFO:teuthology.orchestra.run.smithi164.stdout:(51/160): bison-3.7.4-5.el9.x86_64.rpm 1.4 MB/s | 943 kB 00:00 2024-07-26T06:36:22.572 INFO:teuthology.orchestra.run.smithi164.stdout:(52/160): elfutils-libelf-devel-0.191-4.el9.x86 163 kB/s | 22 kB 00:00 2024-07-26T06:36:22.598 INFO:teuthology.orchestra.run.smithi164.stdout:(53/160): flex-2.6.4-9.el9.x86_64.rpm 2.3 MB/s | 313 kB 00:00 2024-07-26T06:36:22.673 INFO:teuthology.orchestra.run.smithi164.stdout:(54/160): flexiblas-3.0.4-8.el9.x86_64.rpm 328 kB/s | 33 kB 00:00 2024-07-26T06:36:22.713 INFO:teuthology.orchestra.run.smithi042.stdout:(160/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.0 MB/s | 16 MB 00:07 2024-07-26T06:36:22.718 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-07-26T06:36:22.719 INFO:teuthology.orchestra.run.smithi042.stdout:Total 12 MB/s | 233 MB 00:18 2024-07-26T06:36:22.798 INFO:teuthology.orchestra.run.smithi164.stdout:(55/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2024-07-26T06:36:22.823 INFO:teuthology.orchestra.run.smithi164.stdout:(56/160): flexiblas-openblas-openmp-3.0.4-8.el9 81 kB/s | 18 kB 00:00 2024-07-26T06:36:22.834 INFO:teuthology.orchestra.run.smithi163.stdout:(156/160): elfutils-libelf-0.191-4.el9.x86_64.r 267 kB/s | 207 kB 00:00 2024-07-26T06:36:22.848 INFO:teuthology.orchestra.run.smithi164.stdout:(57/160): libaio-devel-0.3.111-13.el9.x86_64.rp 64 kB/s | 11 kB 00:00 2024-07-26T06:36:22.916 INFO:teuthology.orchestra.run.smithi164.stdout:(58/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 160 kB 00:00 2024-07-26T06:36:22.941 INFO:teuthology.orchestra.run.smithi164.stdout:(59/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-07-26T06:36:22.966 INFO:teuthology.orchestra.run.smithi164.stdout:(60/160): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-07-26T06:36:23.025 INFO:teuthology.orchestra.run.smithi164.stdout:(61/160): libtool-2.4.6-46.el9.x86_64.rpm 5.2 MB/s | 581 kB 00:00 2024-07-26T06:36:23.050 INFO:teuthology.orchestra.run.smithi164.stdout:(62/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 236 kB/s | 25 kB 00:00 2024-07-26T06:36:23.075 INFO:teuthology.orchestra.run.smithi164.stdout:(63/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 436 kB/s | 47 kB 00:00 2024-07-26T06:36:23.134 INFO:teuthology.orchestra.run.smithi164.stdout:(64/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-07-26T06:36:23.151 INFO:teuthology.orchestra.run.smithi164.stdout:(65/160): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-07-26T06:36:23.176 INFO:teuthology.orchestra.run.smithi164.stdout:(66/160): openblas-0.3.21-2.el9.x86_64.rpm 350 kB/s | 35 kB 00:00 2024-07-26T06:36:23.226 INFO:teuthology.orchestra.run.smithi163.stdout:(157/160): elfutils-libs-0.191-4.el9.x86_64.rpm 324 kB/s | 260 kB 00:00 2024-07-26T06:36:23.376 INFO:teuthology.orchestra.run.smithi164.stdout:(67/160): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-07-26T06:36:23.434 INFO:teuthology.orchestra.run.smithi163.stdout:(158/160): openssl-3.2.2-2.el9.x86_64.rpm 1.4 MB/s | 1.4 MB 00:00 2024-07-26T06:36:23.519 INFO:teuthology.orchestra.run.smithi164.stdout:(68/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-07-26T06:36:23.560 INFO:teuthology.orchestra.run.smithi164.stdout:(69/160): postgresql-13.14-1.el9.x86_64.rpm 4.1 MB/s | 1.6 MB 00:00 2024-07-26T06:36:23.586 INFO:teuthology.orchestra.run.smithi164.stdout:(70/160): postgresql-contrib-13.14-1.el9.x86_64 4.3 MB/s | 911 kB 00:00 2024-07-26T06:36:23.628 INFO:teuthology.orchestra.run.smithi164.stdout:(71/160): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-07-26T06:36:23.778 INFO:teuthology.orchestra.run.smithi164.stdout:(72/160): python3-attrs-20.3.0-7.el9.noarch.rpm 583 kB/s | 87 kB 00:00 2024-07-26T06:36:23.845 INFO:teuthology.orchestra.run.smithi164.stdout:(73/160): postgresql-server-13.14-1.el9.x86_64. 20 MB/s | 5.8 MB 00:00 2024-07-26T06:36:23.876 INFO:teuthology.orchestra.run.smithi163.stdout:(159/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 2.3 MB/s | 2.4 MB 00:01 2024-07-26T06:36:23.878 INFO:teuthology.orchestra.run.smithi164.stdout:(74/160): protobuf-3.14.0-13.el9.x86_64.rpm 3.5 MB/s | 1.0 MB 00:00 2024-07-26T06:36:23.984 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-07-26T06:36:24.054 INFO:teuthology.orchestra.run.smithi164.stdout:(75/160): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 6.0 MB 00:00 2024-07-26T06:36:24.079 INFO:teuthology.orchestra.run.smithi164.stdout:(76/160): python3-colorama-0.4.6-3.el9.noarch.r 308 kB/s | 61 kB 00:00 2024-07-26T06:36:24.104 INFO:teuthology.orchestra.run.smithi164.stdout:(77/160): python3-devel-3.9.18-3.el9.x86_64.rpm 1.1 MB/s | 244 kB 00:00 2024-07-26T06:36:24.160 INFO:teuthology.orchestra.run.smithi163.stdout:(160/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 1.2 MB/s | 1.1 MB 00:00 2024-07-26T06:36:24.162 INFO:teuthology.orchestra.run.smithi164.stdout:(78/160): python3-jinja2-2.11.3-6.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-07-26T06:36:24.169 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-07-26T06:36:24.169 INFO:teuthology.orchestra.run.smithi163.stdout:Total 14 MB/s | 233 MB 00:16 2024-07-26T06:36:24.204 INFO:teuthology.orchestra.run.smithi164.stdout:(79/160): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-07-26T06:36:24.209 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-07-26T06:36:24.209 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-07-26T06:36:24.230 INFO:teuthology.orchestra.run.smithi164.stdout:(80/160): python3-jsonpointer-2.0-4.el9.noarch. 153 kB/s | 19 kB 00:00 2024-07-26T06:36:24.263 INFO:teuthology.orchestra.run.smithi164.stdout:(81/160): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-07-26T06:36:24.305 INFO:teuthology.orchestra.run.smithi164.stdout:(82/160): python3-markupsafe-1.1.1-12.el9.x86_6 346 kB/s | 35 kB 00:00 2024-07-26T06:36:24.489 INFO:teuthology.orchestra.run.smithi164.stdout:(83/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-07-26T06:36:24.514 INFO:teuthology.orchestra.run.smithi164.stdout:(84/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 250 kB 00:00 2024-07-26T06:36:24.539 INFO:teuthology.orchestra.run.smithi164.stdout:(85/160): python3-oauthlib-3.1.1-5.el9.noarch.r 950 kB/s | 222 kB 00:00 2024-07-26T06:36:24.590 INFO:teuthology.orchestra.run.smithi164.stdout:(86/160): python3-packaging-20.9-5.el9.noarch.r 773 kB/s | 77 kB 00:00 2024-07-26T06:36:24.640 INFO:teuthology.orchestra.run.smithi164.stdout:(87/160): python3-prettytable-0.7.2-27.el9.noar 421 kB/s | 42 kB 00:00 2024-07-26T06:36:24.665 INFO:teuthology.orchestra.run.smithi164.stdout:(88/160): python3-protobuf-3.14.0-13.el9.noarch 2.1 MB/s | 269 kB 00:00 2024-07-26T06:36:24.690 INFO:teuthology.orchestra.run.smithi164.stdout:(89/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-07-26T06:36:24.749 INFO:teuthology.orchestra.run.smithi164.stdout:(90/160): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-07-26T06:36:24.774 INFO:teuthology.orchestra.run.smithi164.stdout:(91/160): python3-pytz-2021.1-5.el9.noarch.rpm 469 kB/s | 51 kB 00:00 2024-07-26T06:36:24.800 INFO:teuthology.orchestra.run.smithi164.stdout:(92/160): python3-requests-oauthlib-1.3.0-12.el 492 kB/s | 54 kB 00:00 2024-07-26T06:36:24.942 INFO:teuthology.orchestra.run.smithi164.stdout:(93/160): python3-toml-0.10.2-6.el9.noarch.rpm 250 kB/s | 42 kB 00:00 2024-07-26T06:36:24.967 INFO:teuthology.orchestra.run.smithi164.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 264 kB/s | 44 kB 00:00 2024-07-26T06:36:25.125 INFO:teuthology.orchestra.run.smithi164.stdout:(95/160): qatlib-service-24.02.0-1.el9.x86_64.r 229 kB/s | 36 kB 00:00 2024-07-26T06:36:25.226 INFO:teuthology.orchestra.run.smithi164.stdout:(96/160): qatlib-24.02.0-1.el9.x86_64.rpm 778 kB/s | 221 kB 00:00 2024-07-26T06:36:25.251 INFO:teuthology.orchestra.run.smithi164.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 384 kB/s | 47 kB 00:00 2024-07-26T06:36:25.365 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-07-26T06:36:25.410 INFO:teuthology.orchestra.run.smithi164.stdout:(98/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-07-26T06:36:25.435 INFO:teuthology.orchestra.run.smithi164.stdout:(99/160): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-07-26T06:36:25.451 INFO:teuthology.orchestra.run.smithi164.stdout:(100/160): uuid-1.6.2-55.el9.x86_64.rpm 289 kB/s | 58 kB 00:00 2024-07-26T06:36:25.511 INFO:teuthology.orchestra.run.smithi164.stdout:(101/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 958 kB/s | 96 kB 00:00 2024-07-26T06:36:25.559 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-07-26T06:36:25.559 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-07-26T06:36:25.561 INFO:teuthology.orchestra.run.smithi164.stdout:(102/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 584 kB/s | 64 kB 00:00 2024-07-26T06:36:25.586 INFO:teuthology.orchestra.run.smithi164.stdout:(103/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 334 kB/s | 45 kB 00:00 2024-07-26T06:36:25.636 INFO:teuthology.orchestra.run.smithi164.stdout:(104/160): lua-devel-5.4.4-4.el9.x86_64.rpm 177 kB/s | 22 kB 00:00 2024-07-26T06:36:25.703 INFO:teuthology.orchestra.run.smithi164.stdout:(105/160): abseil-cpp-20211102.0-3.el9.x86_64.r 4.6 MB/s | 548 kB 00:00 2024-07-26T06:36:25.720 INFO:teuthology.orchestra.run.smithi164.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 1.3 MB/s | 19 kB 00:00 2024-07-26T06:36:25.754 INFO:teuthology.orchestra.run.smithi164.stdout:(107/160): gperftools-libs-2.9.1-2.el9.x86_64.r 2.6 MB/s | 309 kB 00:00 2024-07-26T06:36:25.771 INFO:teuthology.orchestra.run.smithi164.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.5 MB/s | 26 kB 00:00 2024-07-26T06:36:25.804 INFO:teuthology.orchestra.run.smithi164.stdout:(109/160): protobuf-compiler-3.14.0-13.el9.x86_ 3.5 MB/s | 863 kB 00:00 2024-07-26T06:36:25.821 INFO:teuthology.orchestra.run.smithi164.stdout:(110/160): liboath-2.6.7-2.el9.x86_64.rpm 964 kB/s | 49 kB 00:00 2024-07-26T06:36:25.880 INFO:teuthology.orchestra.run.smithi164.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 4.3 MB/s | 151 kB 00:00 2024-07-26T06:36:25.913 INFO:teuthology.orchestra.run.smithi164.stdout:(112/160): libunwind-1.6.2-1.el9.x86_64.rpm 620 kB/s | 67 kB 00:00 2024-07-26T06:36:25.956 INFO:teuthology.orchestra.run.smithi164.stdout:(113/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-07-26T06:36:25.981 INFO:teuthology.orchestra.run.smithi164.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-07-26T06:36:26.006 INFO:teuthology.orchestra.run.smithi164.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-07-26T06:36:26.031 INFO:teuthology.orchestra.run.smithi164.stdout:(116/160): python3-certifi-2023.05.07-4.el9.noa 563 kB/s | 14 kB 00:00 2024-07-26T06:36:26.056 INFO:teuthology.orchestra.run.smithi164.stdout:(117/160): python3-asyncssh-2.13.2-5.el9.noarch 3.7 MB/s | 548 kB 00:00 2024-07-26T06:36:26.115 INFO:teuthology.orchestra.run.smithi164.stdout:(118/160): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-07-26T06:36:26.140 INFO:teuthology.orchestra.run.smithi164.stdout:(119/160): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-07-26T06:36:26.166 INFO:teuthology.orchestra.run.smithi164.stdout:(120/160): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-07-26T06:36:26.191 INFO:teuthology.orchestra.run.smithi164.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 4.2 MB/s | 216 kB 00:00 2024-07-26T06:36:26.216 INFO:teuthology.orchestra.run.smithi164.stdout:(122/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.9 MB/s | 303 kB 00:00 2024-07-26T06:36:26.241 INFO:teuthology.orchestra.run.smithi164.stdout:(123/160): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-07-26T06:36:26.266 INFO:teuthology.orchestra.run.smithi164.stdout:(124/160): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-07-26T06:36:26.292 INFO:teuthology.orchestra.run.smithi164.stdout:(125/160): python3-jaraco-classes-3.2.1-5.el9.n 351 kB/s | 18 kB 00:00 2024-07-26T06:36:26.317 INFO:teuthology.orchestra.run.smithi164.stdout:(126/160): python3-jaraco-collections-3.0.0-8.e 459 kB/s | 23 kB 00:00 2024-07-26T06:36:26.342 INFO:teuthology.orchestra.run.smithi164.stdout:(127/160): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-07-26T06:36:26.368 INFO:teuthology.orchestra.run.smithi164.stdout:(128/160): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-07-26T06:36:26.393 INFO:teuthology.orchestra.run.smithi164.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-07-26T06:36:26.418 INFO:teuthology.orchestra.run.smithi164.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-07-26T06:36:26.444 INFO:teuthology.orchestra.run.smithi164.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-07-26T06:36:26.479 INFO:teuthology.orchestra.run.smithi164.stdout:(132/160): python3-kubernetes-26.1.0-2.el9.noar 7.5 MB/s | 1.0 MB 00:00 2024-07-26T06:36:26.505 INFO:teuthology.orchestra.run.smithi164.stdout:(133/160): python3-pecan-1.4.2-3.el9.noarch.rpm 4.4 MB/s | 272 kB 00:00 2024-07-26T06:36:26.530 INFO:teuthology.orchestra.run.smithi164.stdout:(134/160): python3-portend-3.1.0-2.el9.noarch.r 325 kB/s | 16 kB 00:00 2024-07-26T06:36:26.531 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-07-26T06:36:26.531 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-07-26T06:36:26.555 INFO:teuthology.orchestra.run.smithi164.stdout:(135/160): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-07-26T06:36:26.580 INFO:teuthology.orchestra.run.smithi164.stdout:(136/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-07-26T06:36:26.598 INFO:teuthology.orchestra.run.smithi164.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 735 kB/s | 31 kB 00:00 2024-07-26T06:36:26.614 INFO:teuthology.orchestra.run.smithi164.stdout:(138/160): python3-rsa-4.9-2.el9.noarch.rpm 3.5 MB/s | 59 kB 00:00 2024-07-26T06:36:26.640 INFO:teuthology.orchestra.run.smithi164.stdout:(139/160): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-07-26T06:36:26.657 INFO:teuthology.orchestra.run.smithi164.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 849 kB/s | 36 kB 00:00 2024-07-26T06:36:26.674 INFO:teuthology.orchestra.run.smithi164.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 1.5 MB/s | 51 kB 00:00 2024-07-26T06:36:26.699 INFO:teuthology.orchestra.run.smithi164.stdout:(142/160): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-07-26T06:36:26.724 INFO:teuthology.orchestra.run.smithi164.stdout:(143/160): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-07-26T06:36:26.749 INFO:teuthology.orchestra.run.smithi164.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 800 kB/s | 20 kB 00:00 2024-07-26T06:36:26.791 INFO:teuthology.orchestra.run.smithi164.stdout:(145/160): python3-grpcio-1.46.7-10.el9.x86_64. 3.4 MB/s | 2.0 MB 00:00 2024-07-26T06:36:26.817 INFO:teuthology.orchestra.run.smithi164.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-07-26T06:36:26.842 INFO:teuthology.orchestra.run.smithi164.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-07-26T06:36:26.893 INFO:teuthology.orchestra.run.smithi164.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-07-26T06:36:27.076 INFO:teuthology.orchestra.run.smithi164.stdout:(149/160): librados2-19.0.0-5320.ge8f88bb1.el9. 13 MB/s | 3.4 MB 00:00 2024-07-26T06:36:27.151 INFO:teuthology.orchestra.run.smithi164.stdout:(150/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 10 MB/s | 3.1 MB 00:00 2024-07-26T06:36:27.427 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-07-26T06:36:27.427 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-07-26T06:36:27.785 INFO:teuthology.orchestra.run.smithi164.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 67 kB/s | 41 kB 00:00 2024-07-26T06:36:27.993 INFO:teuthology.orchestra.run.smithi164.stdout:(152/160): NetworkManager-1.48.2-2.el9.x86_64.r 2.1 MB/s | 2.3 MB 00:01 2024-07-26T06:36:28.127 INFO:teuthology.orchestra.run.smithi164.stdout:(153/160): NetworkManager-libnm-1.48.2-2.el9.x8 1.7 MB/s | 1.8 MB 00:01 2024-07-26T06:36:28.561 INFO:teuthology.orchestra.run.smithi164.stdout:(154/160): NetworkManager-tui-1.48.2-2.el9.x86_ 319 kB/s | 247 kB 00:00 2024-07-26T06:36:28.586 INFO:teuthology.orchestra.run.smithi164.stdout:(155/160): elfutils-debuginfod-client-0.191-4.e 62 kB/s | 37 kB 00:00 2024-07-26T06:36:28.894 INFO:teuthology.orchestra.run.smithi164.stdout:(156/160): elfutils-libelf-0.191-4.el9.x86_64.r 270 kB/s | 207 kB 00:00 2024-07-26T06:36:29.387 INFO:teuthology.orchestra.run.smithi164.stdout:(157/160): elfutils-libs-0.191-4.el9.x86_64.rpm 315 kB/s | 260 kB 00:00 2024-07-26T06:36:29.553 INFO:teuthology.orchestra.run.smithi164.stdout:(158/160): openssl-3.2.2-2.el9.x86_64.rpm 1.4 MB/s | 1.4 MB 00:00 2024-07-26T06:36:29.920 INFO:teuthology.orchestra.run.smithi164.stdout:(159/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 2.3 MB/s | 2.4 MB 00:01 2024-07-26T06:36:30.345 INFO:teuthology.orchestra.run.smithi164.stdout:(160/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 1.1 MB/s | 1.1 MB 00:00 2024-07-26T06:36:30.352 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-07-26T06:36:30.352 INFO:teuthology.orchestra.run.smithi164.stdout:Total 15 MB/s | 233 MB 00:15 2024-07-26T06:36:31.581 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-07-26T06:36:31.777 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-07-26T06:36:31.777 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-07-26T06:36:33.171 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-07-26T06:36:33.261 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T06:36:33.304 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T06:36:33.362 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T06:36:33.402 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T06:36:33.428 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T06:36:33.481 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T06:36:33.718 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-07-26T06:36:33.814 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T06:36:33.838 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T06:36:33.851 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T06:36:33.874 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-07-26T06:36:33.875 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-07-26T06:36:33.905 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T06:36:33.939 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T06:36:33.945 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T06:36:33.965 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T06:36:34.007 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T06:36:34.074 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T06:36:34.105 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T06:36:34.185 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T06:36:34.218 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T06:36:34.260 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T06:36:34.400 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T06:36:34.465 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T06:36:34.475 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T06:36:34.533 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T06:36:34.560 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T06:36:34.590 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T06:36:34.617 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T06:36:34.668 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T06:36:34.695 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T06:36:34.712 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T06:36:34.737 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T06:36:34.757 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T06:36:34.770 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T06:36:34.779 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T06:36:34.804 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T06:36:34.941 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T06:36:34.962 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T06:36:35.027 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T06:36:35.034 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T06:36:35.055 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T06:36:35.143 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T06:36:35.173 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T06:36:35.235 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T06:36:35.257 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T06:36:35.276 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T06:36:35.278 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T06:36:35.430 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T06:36:35.465 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T06:36:35.473 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T06:36:35.557 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T06:36:35.567 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T06:36:35.649 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T06:36:35.735 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T06:36:35.873 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T06:36:35.891 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T06:36:35.959 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T06:36:35.986 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T06:36:36.017 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T06:36:36.052 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T06:36:36.061 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T06:36:36.127 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T06:36:36.160 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T06:36:36.179 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T06:36:36.219 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T06:36:36.262 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T06:36:36.295 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T06:36:36.322 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T06:36:36.367 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T06:36:36.393 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T06:36:36.454 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T06:36:36.477 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T06:36:36.483 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T06:36:36.505 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T06:36:36.528 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T06:36:36.559 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T06:36:36.605 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T06:36:36.631 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T06:36:36.646 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T06:36:36.675 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T06:36:36.695 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T06:36:36.715 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T06:36:36.726 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T06:36:36.759 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T06:36:36.791 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T06:36:36.799 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T06:36:36.819 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T06:36:36.850 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T06:36:36.889 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T06:36:36.928 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T06:36:36.954 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T06:36:36.989 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T06:36:37.024 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T06:36:37.082 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T06:36:37.100 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T06:36:37.135 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T06:36:37.136 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T06:36:37.180 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T06:36:37.207 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T06:36:37.278 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T06:36:37.322 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T06:36:37.396 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T06:36:37.539 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T06:36:37.599 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T06:36:37.699 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T06:36:37.783 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T06:36:37.819 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T06:36:37.863 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T06:36:37.913 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T06:36:37.950 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T06:36:37.994 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T06:36:38.123 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T06:36:38.154 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T06:36:38.203 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T06:36:38.211 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T06:36:38.232 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T06:36:38.267 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T06:36:38.307 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T06:36:38.317 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T06:36:38.355 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T06:36:38.370 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T06:36:38.396 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T06:36:38.442 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T06:36:38.480 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T06:36:38.516 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T06:36:38.561 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T06:36:38.564 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T06:36:38.599 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T06:36:38.632 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T06:36:38.642 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T06:36:38.684 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T06:36:38.725 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T06:36:38.797 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T06:36:38.827 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T06:36:38.869 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T06:36:38.870 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T06:36:38.901 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T06:36:38.937 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T06:36:38.952 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T06:36:38.986 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T06:36:39.038 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T06:36:39.054 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T06:36:39.080 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T06:36:39.085 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T06:36:39.122 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T06:36:39.126 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T06:36:39.167 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T06:36:39.192 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T06:36:39.259 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T06:36:39.274 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T06:36:39.363 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T06:36:39.415 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T06:36:39.449 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T06:36:39.477 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T06:36:39.512 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T06:36:39.542 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T06:36:39.552 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T06:36:39.591 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T06:36:39.629 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T06:36:39.644 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T06:36:39.664 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T06:36:39.682 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T06:36:39.697 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T06:36:39.722 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T06:36:39.739 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T06:36:39.755 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T06:36:39.773 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T06:36:39.780 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T06:36:39.857 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T06:36:39.862 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T06:36:39.913 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T06:36:39.943 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T06:36:39.982 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T06:36:40.020 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T06:36:40.048 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T06:36:40.050 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T06:36:40.083 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T06:36:40.111 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T06:36:40.122 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T06:36:40.153 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T06:36:40.183 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T06:36:40.217 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T06:36:40.251 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T06:36:40.293 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T06:36:40.327 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T06:36:40.414 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T06:36:40.434 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-07-26T06:36:40.457 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T06:36:40.488 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T06:36:40.508 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T06:36:40.546 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T06:36:40.593 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T06:36:40.656 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T06:36:40.660 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T06:36:40.700 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T06:36:40.725 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T06:36:40.769 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T06:36:40.870 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T06:36:40.924 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T06:36:40.958 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T06:36:41.060 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T06:36:41.106 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T06:36:41.122 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T06:36:41.167 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T06:36:41.226 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T06:36:41.265 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T06:36:41.302 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T06:36:41.352 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T06:36:41.378 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T06:36:41.384 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T06:36:41.427 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T06:36:41.461 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T06:36:41.465 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T06:36:41.487 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T06:36:41.490 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T06:36:41.517 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T06:36:41.523 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T06:36:41.560 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T06:36:41.597 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T06:36:41.616 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T06:36:41.661 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T06:36:41.698 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T06:36:41.747 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T06:36:41.748 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T06:36:41.775 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T06:36:41.797 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T06:36:41.880 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T06:36:41.909 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T06:36:41.972 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T06:36:41.976 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T06:36:41.993 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T06:36:42.003 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T06:36:42.011 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T06:36:42.081 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T06:36:42.202 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T06:36:42.282 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T06:36:42.293 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T06:36:42.402 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T06:36:42.474 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T06:36:42.634 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T06:36:42.686 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T06:36:42.733 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T06:36:42.776 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T06:36:42.852 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T06:36:42.896 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T06:36:43.092 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T06:36:43.194 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T06:36:43.217 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T06:36:43.252 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T06:36:43.326 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T06:36:43.370 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T06:36:43.423 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T06:36:43.470 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T06:36:43.503 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T06:36:43.531 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T06:36:43.600 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T06:36:43.653 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T06:36:43.684 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T06:36:43.711 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T06:36:43.719 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T06:36:43.795 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T06:36:43.796 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T06:36:43.823 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T06:36:43.830 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T06:36:43.857 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T06:36:43.875 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T06:36:43.902 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T06:36:43.973 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T06:36:44.017 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T06:36:44.090 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T06:36:44.092 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T06:36:44.189 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T06:36:44.201 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T06:36:44.217 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T06:36:44.223 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T06:36:44.236 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T06:36:44.250 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T06:36:44.286 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T06:36:44.332 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T06:36:44.358 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T06:36:44.394 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T06:36:44.579 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T06:36:44.638 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T06:36:44.734 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T06:36:44.764 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T06:36:44.800 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T06:36:44.812 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T06:36:44.890 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T06:36:44.919 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T06:36:44.921 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T06:36:44.956 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T06:36:44.956 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T06:36:45.004 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T06:36:45.028 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T06:36:45.043 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T06:36:45.057 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T06:36:45.085 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T06:36:45.234 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T06:36:45.296 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T06:36:45.300 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T06:36:45.336 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T06:36:45.388 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T06:36:45.400 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T06:36:45.429 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T06:36:45.443 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T06:36:45.516 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T06:36:45.555 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T06:36:45.592 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T06:36:45.639 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T06:36:45.674 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T06:36:45.726 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T06:36:45.768 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T06:36:46.116 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T06:36:46.154 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T06:36:46.195 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T06:36:46.301 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T06:36:46.336 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T06:36:46.376 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T06:36:46.402 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T06:36:46.427 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T06:36:46.500 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T06:36:46.628 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T06:36:46.676 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T06:36:46.703 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T06:36:46.738 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T06:36:46.777 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T06:36:46.809 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T06:36:46.838 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T06:36:46.873 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T06:36:46.906 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T06:36:46.947 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T06:36:46.982 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T06:36:47.060 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T06:36:47.112 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T06:36:47.144 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T06:36:47.247 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T06:36:47.295 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T06:36:47.473 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T06:36:47.512 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T06:36:47.662 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T06:36:47.693 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T06:36:48.021 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T06:36:48.066 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T06:36:48.100 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T06:36:48.108 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T06:36:48.158 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T06:36:48.199 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T06:36:48.247 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T06:36:48.250 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T06:36:48.250 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T06:36:48.251 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T06:36:48.251 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:36:48.257 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T06:36:48.284 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T06:36:48.292 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T06:36:48.305 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T06:36:48.380 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T06:36:48.399 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T06:36:48.399 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T06:36:48.399 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T06:36:48.399 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:36:48.428 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T06:36:48.432 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T06:36:48.605 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T06:36:48.634 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T06:36:48.712 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T06:36:49.360 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T06:36:49.465 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T06:36:50.698 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T06:36:50.779 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T06:36:50.798 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T06:36:50.823 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T06:36:51.149 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T06:36:51.194 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T06:36:51.290 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T06:36:51.324 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T06:36:51.360 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T06:36:51.859 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T06:36:51.889 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T06:36:51.962 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T06:36:51.991 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T06:36:54.235 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T06:36:54.270 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T06:36:54.830 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T06:36:54.865 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T06:36:54.943 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T06:36:54.944 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T06:36:54.944 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T06:36:54.944 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:36:54.975 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T06:37:00.122 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T06:37:00.122 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /sys 2024-07-26T06:37:00.122 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /proc 2024-07-26T06:37:00.122 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /mnt 2024-07-26T06:37:00.123 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /var/tmp 2024-07-26T06:37:00.123 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /home 2024-07-26T06:37:00.123 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /root 2024-07-26T06:37:00.123 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /tmp 2024-07-26T06:37:00.123 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:37:00.182 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T06:37:00.182 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /sys 2024-07-26T06:37:00.182 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /proc 2024-07-26T06:37:00.183 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /mnt 2024-07-26T06:37:00.183 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /var/tmp 2024-07-26T06:37:00.183 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /home 2024-07-26T06:37:00.183 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /root 2024-07-26T06:37:00.183 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /tmp 2024-07-26T06:37:00.183 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:37:00.293 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T06:37:00.308 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T06:37:00.308 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:00.308 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T06:37:00.308 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:37:00.366 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T06:37:00.383 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T06:37:00.383 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:00.383 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T06:37:00.383 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:37:01.913 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T06:37:01.939 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T06:37:01.939 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:01.939 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T06:37:01.939 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T06:37:01.939 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T06:37:01.939 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:37:02.036 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T06:37:02.064 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T06:37:02.064 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:02.064 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T06:37:02.064 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T06:37:02.064 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T06:37:02.064 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:37:02.163 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T06:37:02.186 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T06:37:02.186 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:02.186 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T06:37:02.186 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T06:37:02.186 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T06:37:02.187 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:37:02.303 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T06:37:02.327 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T06:37:02.328 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:02.328 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T06:37:02.328 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T06:37:02.328 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T06:37:02.328 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:37:02.647 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T06:37:02.670 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T06:37:02.670 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:02.670 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T06:37:02.670 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T06:37:02.671 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T06:37:02.671 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:37:02.778 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T06:37:02.812 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T06:37:02.834 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T06:37:02.837 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T06:37:02.837 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:02.837 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T06:37:02.838 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T06:37:02.838 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T06:37:02.838 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:37:02.890 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T06:37:02.951 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T06:37:03.011 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T06:37:03.074 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T06:37:03.075 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T06:37:03.124 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T06:37:03.201 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T06:37:03.269 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T06:37:03.290 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T06:37:03.314 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T06:37:03.396 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T06:37:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T06:37:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T06:37:03.493 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T06:37:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T06:37:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T06:37:03.635 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T06:37:03.681 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T06:37:03.712 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T06:37:03.740 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T06:37:03.979 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T06:37:04.010 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T06:37:04.057 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T06:37:04.121 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T06:37:04.148 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T06:37:04.161 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T06:37:04.188 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T06:37:04.221 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T06:37:04.287 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T06:37:04.311 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T06:37:06.539 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T06:37:06.539 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /sys 2024-07-26T06:37:06.539 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /proc 2024-07-26T06:37:06.539 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /mnt 2024-07-26T06:37:06.539 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /var/tmp 2024-07-26T06:37:06.539 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /home 2024-07-26T06:37:06.539 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /root 2024-07-26T06:37:06.539 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /tmp 2024-07-26T06:37:06.540 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:37:06.724 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T06:37:06.741 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T06:37:06.741 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:06.741 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T06:37:06.741 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:37:07.326 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T06:37:07.396 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T06:37:07.648 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T06:37:07.701 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T06:37:08.311 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T06:37:08.337 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T06:37:08.337 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:08.338 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T06:37:08.338 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T06:37:08.338 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T06:37:08.338 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:37:08.367 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T06:37:08.388 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T06:37:08.500 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T06:37:08.553 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T06:37:08.576 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T06:37:08.576 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:08.576 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T06:37:08.576 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T06:37:08.576 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T06:37:08.577 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:37:08.673 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T06:37:08.715 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T06:37:08.717 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T06:37:08.738 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T06:37:08.738 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:08.738 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T06:37:08.738 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T06:37:08.738 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T06:37:08.738 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:37:08.751 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T06:37:08.782 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T06:37:08.794 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T06:37:08.872 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T06:37:08.894 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T06:37:08.940 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T06:37:09.045 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T06:37:09.052 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T06:37:09.069 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T06:37:09.069 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:09.069 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T06:37:09.069 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T06:37:09.069 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T06:37:09.069 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:37:09.086 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T06:37:09.112 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T06:37:09.112 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:09.112 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T06:37:09.112 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T06:37:09.112 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T06:37:09.112 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:37:09.162 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T06:37:09.174 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T06:37:09.176 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T06:37:09.233 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T06:37:09.278 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T06:37:09.287 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T06:37:09.335 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T06:37:09.473 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T06:37:09.521 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T06:37:09.601 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T06:37:09.688 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T06:37:09.828 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T06:37:09.871 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T06:37:09.897 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T06:37:09.903 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T06:37:09.927 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T06:37:09.931 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T06:37:09.950 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T06:37:09.950 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:09.950 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T06:37:09.950 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T06:37:09.950 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T06:37:09.950 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:37:09.994 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T06:37:10.016 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T06:37:10.017 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:10.017 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T06:37:10.017 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:37:10.062 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T06:37:10.084 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T06:37:10.084 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:10.084 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T06:37:10.084 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T06:37:10.084 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T06:37:10.084 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:37:10.326 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T06:37:10.355 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T06:37:10.357 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T06:37:10.387 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T06:37:10.389 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T06:37:10.404 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T06:37:10.414 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T06:37:10.414 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:10.414 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T06:37:10.414 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T06:37:10.415 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T06:37:10.415 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:37:10.426 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T06:37:10.427 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:10.427 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T06:37:10.427 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T06:37:10.427 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T06:37:10.427 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:37:10.451 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T06:37:10.457 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T06:37:10.478 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T06:37:10.480 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T06:37:10.480 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:10.481 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T06:37:10.481 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:37:10.533 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T06:37:10.557 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T06:37:10.557 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:10.557 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T06:37:10.557 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T06:37:10.558 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T06:37:10.558 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:37:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T06:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T06:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T06:37:10.925 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T06:37:10.926 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T06:37:10.926 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:37:13.606 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T06:37:13.667 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T06:37:14.623 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T06:37:14.659 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T06:37:14.771 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T06:37:14.852 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T06:37:14.893 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T06:37:14.933 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T06:37:14.942 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T06:37:14.961 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T06:37:14.969 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T06:37:14.989 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T06:37:14.993 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T06:37:14.993 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:14.993 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T06:37:14.993 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T06:37:14.993 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T06:37:14.993 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:37:15.029 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T06:37:15.040 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T06:37:15.139 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T06:37:15.175 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T06:37:15.193 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T06:37:15.220 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T06:37:15.252 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T06:37:15.285 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T06:37:15.323 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T06:37:15.370 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T06:37:15.604 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T06:37:15.646 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T06:37:15.687 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T06:37:15.719 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T06:37:15.745 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T06:37:15.944 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T06:37:15.956 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T06:37:15.986 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T06:37:16.000 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T06:37:16.005 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T06:37:16.033 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T06:37:16.072 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T06:37:16.102 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T06:37:16.140 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T06:37:16.147 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T06:37:16.173 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T06:37:16.196 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T06:37:16.196 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:16.196 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T06:37:16.196 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T06:37:16.196 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T06:37:16.196 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:37:16.240 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T06:37:16.262 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T06:37:16.263 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:16.263 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T06:37:16.263 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:37:16.307 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T06:37:16.330 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T06:37:16.330 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:16.330 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T06:37:16.330 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T06:37:16.330 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T06:37:16.331 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:37:16.650 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T06:37:16.672 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T06:37:16.672 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T06:37:16.672 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T06:37:16.673 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T06:37:16.673 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T06:37:16.673 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:37:16.756 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T06:37:16.788 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T06:37:16.809 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T06:37:16.846 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T06:37:16.945 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T06:37:16.994 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T06:37:17.075 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T06:37:17.106 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T06:37:17.131 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T06:37:17.157 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T06:37:17.183 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T06:37:17.184 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T06:37:17.268 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T06:37:17.301 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T06:37:17.301 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T06:37:17.412 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T06:37:17.437 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T06:37:17.442 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T06:37:17.442 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T06:37:17.631 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T06:37:17.631 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T06:37:17.715 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T06:37:17.722 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T06:37:17.740 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T06:37:17.766 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T06:37:17.826 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T06:37:17.839 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T06:37:17.868 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T06:37:17.961 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T06:37:17.996 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T06:37:18.021 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T06:37:18.046 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T06:37:18.072 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T06:37:18.073 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T06:37:18.157 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T06:37:18.198 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T06:37:18.198 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T06:37:18.283 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T06:37:18.308 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T06:37:18.317 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T06:37:18.317 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T06:37:18.502 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T06:37:18.502 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T06:37:18.586 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T06:37:18.619 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T06:37:18.644 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T06:37:18.721 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T06:37:21.046 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T06:37:21.087 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T06:37:21.127 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T06:37:21.160 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T06:37:21.187 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T06:37:21.379 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T06:37:21.421 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T06:37:21.454 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T06:37:21.494 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T06:37:21.524 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T06:37:21.570 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T06:37:22.185 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T06:37:22.236 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T06:37:22.257 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T06:37:23.100 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T06:37:23.196 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T06:37:23.237 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T06:37:23.318 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T06:37:23.350 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T06:37:23.375 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T06:37:23.400 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T06:37:23.427 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T06:37:23.427 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T06:37:23.495 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T06:37:23.536 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T06:37:23.536 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T06:37:23.621 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T06:37:23.646 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T06:37:23.651 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T06:37:23.651 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T06:37:23.831 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T06:37:23.831 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T06:37:23.907 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T06:37:23.932 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T06:37:23.958 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T06:37:24.040 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T06:37:24.484 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T06:37:24.484 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T06:37:24.484 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T06:37:24.484 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T06:37:24.484 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T06:37:24.484 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T06:37:24.485 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T06:37:24.485 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T06:37:24.485 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T06:37:24.485 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T06:37:24.485 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T06:37:24.485 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T06:37:24.485 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T06:37:24.485 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T06:37:24.485 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T06:37:24.485 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T06:37:24.485 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T06:37:24.486 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T06:37:24.486 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T06:37:24.486 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T06:37:24.486 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T06:37:24.486 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T06:37:24.486 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T06:37:24.486 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T06:37:24.486 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T06:37:24.486 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T06:37:24.486 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T06:37:24.486 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T06:37:24.486 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T06:37:24.487 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T06:37:24.499 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T06:37:24.499 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T06:37:24.499 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T06:37:24.499 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T06:37:24.499 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T06:37:24.499 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T06:37:24.500 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T06:37:24.500 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T06:37:24.500 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T06:37:24.500 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T06:37:24.500 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T06:37:24.500 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T06:37:24.500 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T06:37:24.500 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T06:37:24.500 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T06:37:24.500 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T06:37:24.500 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T06:37:24.500 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T06:37:24.501 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T06:37:24.501 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T06:37:24.501 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T06:37:24.501 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T06:37:24.501 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T06:37:24.501 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T06:37:24.501 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T06:37:24.501 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T06:37:24.501 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T06:37:24.501 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T06:37:24.501 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T06:37:24.501 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T06:37:24.502 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T06:37:24.502 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T06:37:24.502 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T06:37:24.502 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T06:37:24.502 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T06:37:24.502 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T06:37:24.502 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T06:37:24.502 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T06:37:24.502 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T06:37:24.502 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T06:37:24.502 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T06:37:24.503 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T06:37:24.503 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T06:37:24.503 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T06:37:24.503 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T06:37:24.503 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T06:37:24.503 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T06:37:24.503 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T06:37:24.503 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T06:37:24.503 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T06:37:24.503 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T06:37:24.503 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T06:37:24.503 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T06:37:24.504 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T06:37:24.504 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T06:37:24.504 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T06:37:24.504 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T06:37:24.504 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T06:37:24.504 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T06:37:24.504 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T06:37:24.504 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T06:37:24.504 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T06:37:24.504 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T06:37:24.504 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T06:37:24.504 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T06:37:24.504 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T06:37:24.505 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T06:37:24.505 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T06:37:24.505 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T06:37:24.505 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T06:37:24.505 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T06:37:24.505 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T06:37:24.505 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T06:37:24.505 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T06:37:24.505 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T06:37:24.505 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T06:37:24.505 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T06:37:24.505 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T06:37:24.505 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T06:37:24.506 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T06:37:24.506 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T06:37:24.506 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T06:37:24.506 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T06:37:24.506 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T06:37:24.506 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T06:37:24.506 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T06:37:24.506 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T06:37:24.506 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T06:37:24.506 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T06:37:24.506 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T06:37:24.506 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T06:37:24.506 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T06:37:24.507 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T06:37:24.507 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T06:37:24.507 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T06:37:24.507 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T06:37:24.507 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T06:37:24.507 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T06:37:24.507 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T06:37:24.507 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T06:37:24.507 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T06:37:24.507 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T06:37:24.507 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T06:37:24.507 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T06:37:24.507 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T06:37:24.508 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T06:37:24.508 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T06:37:24.508 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T06:37:24.508 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T06:37:24.508 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T06:37:24.508 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T06:37:24.508 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T06:37:24.508 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T06:37:24.508 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T06:37:24.508 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T06:37:24.508 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T06:37:24.508 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T06:37:24.508 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T06:37:24.509 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T06:37:24.509 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T06:37:24.509 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T06:37:24.509 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T06:37:24.509 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T06:37:24.509 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T06:37:24.509 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T06:37:24.509 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T06:37:24.509 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T06:37:24.509 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T06:37:24.509 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T06:37:24.509 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T06:37:24.510 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T06:37:24.510 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T06:37:24.510 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T06:37:24.510 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T06:37:24.510 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T06:37:24.510 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T06:37:24.510 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T06:37:24.510 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T06:37:24.510 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T06:37:24.510 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T06:37:24.510 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T06:37:24.510 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T06:37:25.064 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T06:37:25.064 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T06:37:25.064 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T06:37:25.064 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T06:37:25.064 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T06:37:25.065 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T06:37:25.065 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T06:37:25.065 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T06:37:25.065 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T06:37:25.065 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T06:37:25.065 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T06:37:25.065 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T06:37:25.065 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T06:37:25.065 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T06:37:25.065 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T06:37:25.065 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T06:37:25.065 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T06:37:25.065 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T06:37:25.066 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T06:37:25.066 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T06:37:25.066 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T06:37:25.066 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T06:37:25.066 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T06:37:25.066 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T06:37:25.066 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T06:37:25.066 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T06:37:25.066 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T06:37:25.066 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T06:37:25.066 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T06:37:25.066 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T06:37:25.066 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T06:37:25.067 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T06:37:25.067 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T06:37:25.067 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T06:37:25.067 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T06:37:25.067 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T06:37:25.067 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T06:37:25.067 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T06:37:25.067 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T06:37:25.067 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T06:37:25.067 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T06:37:25.067 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T06:37:25.067 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T06:37:25.067 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T06:37:25.068 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T06:37:25.068 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T06:37:25.068 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T06:37:25.068 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T06:37:25.068 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T06:37:25.068 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T06:37:25.068 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T06:37:25.068 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T06:37:25.068 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T06:37:25.068 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T06:37:25.068 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T06:37:25.068 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T06:37:25.068 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T06:37:25.068 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T06:37:25.069 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T06:37:25.069 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T06:37:25.069 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T06:37:25.069 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T06:37:25.069 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T06:37:25.069 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T06:37:25.069 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T06:37:25.069 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T06:37:25.069 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T06:37:25.069 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T06:37:25.069 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T06:37:25.069 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T06:37:25.069 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T06:37:25.070 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T06:37:25.070 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T06:37:25.070 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T06:37:25.070 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T06:37:25.070 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T06:37:25.070 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T06:37:25.070 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T06:37:25.070 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T06:37:25.070 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T06:37:25.070 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T06:37:25.070 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T06:37:25.070 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T06:37:25.070 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T06:37:25.071 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T06:37:25.071 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T06:37:25.071 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T06:37:25.071 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T06:37:25.071 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T06:37:25.071 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T06:37:25.071 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T06:37:25.071 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T06:37:25.071 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T06:37:25.071 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T06:37:25.071 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T06:37:25.071 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T06:37:25.071 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T06:37:25.072 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T06:37:25.072 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T06:37:25.072 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T06:37:25.072 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T06:37:25.072 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T06:37:25.072 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T06:37:25.072 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T06:37:25.072 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T06:37:25.072 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T06:37:25.072 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T06:37:25.072 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T06:37:25.072 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T06:37:25.072 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T06:37:25.073 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T06:37:25.073 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T06:37:25.073 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T06:37:25.073 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T06:37:25.073 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T06:37:25.073 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T06:37:25.073 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T06:37:25.073 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T06:37:25.073 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T06:37:25.073 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T06:37:25.073 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T06:37:25.073 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T06:37:25.073 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T06:37:25.074 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T06:37:25.074 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T06:37:25.074 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T06:37:25.074 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T06:37:25.074 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T06:37:25.074 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T06:37:25.074 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T06:37:25.074 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T06:37:25.074 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T06:37:25.074 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T06:37:25.074 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T06:37:25.074 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T06:37:25.074 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T06:37:25.075 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T06:37:25.075 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T06:37:25.075 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T06:37:25.075 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T06:37:25.075 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T06:37:25.075 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T06:37:25.075 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T06:37:25.075 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T06:37:25.075 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T06:37:25.075 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T06:37:25.075 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T06:37:25.075 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T06:37:25.075 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T06:37:25.076 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T06:37:25.076 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T06:37:25.076 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T06:37:25.076 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T06:37:25.076 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T06:37:25.076 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T06:37:25.076 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T06:37:25.076 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T06:37:25.076 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T06:37:25.076 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T06:37:25.076 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T06:37:25.076 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T06:37:25.076 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T06:37:25.077 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T06:37:25.077 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T06:37:25.077 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T06:37:25.077 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T06:37:25.077 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T06:37:25.077 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T06:37:25.077 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T06:37:25.077 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T06:37:25.077 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T06:37:25.077 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T06:37:27.213 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T06:37:27.213 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:37:27.213 INFO:teuthology.orchestra.run.smithi163.stdout:Upgraded: 2024-07-26T06:37:27.213 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T06:37:27.213 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T06:37:27.213 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T06:37:27.214 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T06:37:27.214 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T06:37:27.214 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T06:37:27.214 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T06:37:27.214 INFO:teuthology.orchestra.run.smithi163.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.214 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.214 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T06:37:27.214 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T06:37:27.214 INFO:teuthology.orchestra.run.smithi163.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T06:37:27.214 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-07-26T06:37:27.214 INFO:teuthology.orchestra.run.smithi163.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T06:37:27.214 INFO:teuthology.orchestra.run.smithi163.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T06:37:27.215 INFO:teuthology.orchestra.run.smithi163.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T06:37:27.215 INFO:teuthology.orchestra.run.smithi163.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T06:37:27.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.216 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.216 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.216 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.216 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.216 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.216 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.216 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.216 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.216 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.216 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.216 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.216 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.217 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.217 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T06:37:27.217 INFO:teuthology.orchestra.run.smithi163.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T06:37:27.217 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T06:37:27.217 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T06:37:27.217 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T06:37:27.217 INFO:teuthology.orchestra.run.smithi163.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T06:37:27.217 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T06:37:27.217 INFO:teuthology.orchestra.run.smithi163.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T06:37:27.217 INFO:teuthology.orchestra.run.smithi163.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T06:37:27.217 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T06:37:27.217 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T06:37:27.218 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.218 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.218 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.218 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T06:37:27.218 INFO:teuthology.orchestra.run.smithi163.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T06:37:27.218 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T06:37:27.218 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T06:37:27.218 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T06:37:27.218 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.218 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.218 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T06:37:27.219 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.219 INFO:teuthology.orchestra.run.smithi163.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T06:37:27.219 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T06:37:27.219 INFO:teuthology.orchestra.run.smithi163.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T06:37:27.219 INFO:teuthology.orchestra.run.smithi163.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T06:37:27.219 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T06:37:27.219 INFO:teuthology.orchestra.run.smithi163.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T06:37:27.219 INFO:teuthology.orchestra.run.smithi163.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T06:37:27.219 INFO:teuthology.orchestra.run.smithi163.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T06:37:27.219 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T06:37:27.219 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T06:37:27.219 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T06:37:27.220 INFO:teuthology.orchestra.run.smithi163.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T06:37:27.220 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T06:37:27.220 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T06:37:27.220 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T06:37:27.220 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T06:37:27.220 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T06:37:27.220 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T06:37:27.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T06:37:27.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T06:37:27.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T06:37:27.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T06:37:27.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T06:37:27.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T06:37:27.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T06:37:27.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T06:37:27.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T06:37:27.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T06:37:27.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T06:37:27.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T06:37:27.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T06:37:27.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T06:37:27.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T06:37:27.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T06:37:27.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T06:37:27.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T06:37:27.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T06:37:27.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T06:37:27.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T06:37:27.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T06:37:27.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T06:37:27.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T06:37:27.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T06:37:27.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T06:37:27.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T06:37:27.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T06:37:27.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T06:37:27.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T06:37:27.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T06:37:27.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T06:37:27.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T06:37:27.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T06:37:27.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T06:37:27.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T06:37:27.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T06:37:27.224 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T06:37:27.224 INFO:teuthology.orchestra.run.smithi163.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T06:37:27.224 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T06:37:27.224 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T06:37:27.224 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T06:37:27.224 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T06:37:27.224 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T06:37:27.224 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T06:37:27.224 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.224 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.224 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T06:37:27.224 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T06:37:27.224 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T06:37:27.224 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.225 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T06:37:27.225 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T06:37:27.225 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T06:37:27.225 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T06:37:27.225 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T06:37:27.225 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T06:37:27.225 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T06:37:27.225 INFO:teuthology.orchestra.run.smithi163.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T06:37:27.225 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T06:37:27.225 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T06:37:27.225 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T06:37:27.225 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T06:37:27.225 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T06:37:27.225 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T06:37:27.226 INFO:teuthology.orchestra.run.smithi163.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T06:37:27.226 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.226 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.226 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.226 INFO:teuthology.orchestra.run.smithi163.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T06:37:27.226 INFO:teuthology.orchestra.run.smithi163.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T06:37:27.226 INFO:teuthology.orchestra.run.smithi163.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T06:37:27.226 INFO:teuthology.orchestra.run.smithi163.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T06:37:27.226 INFO:teuthology.orchestra.run.smithi163.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T06:37:27.226 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T06:37:27.227 INFO:teuthology.orchestra.run.smithi163.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T06:37:27.227 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-07-26T06:37:27.227 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-07-26T06:37:27.694 DEBUG:teuthology.parallel:result is None 2024-07-26T06:37:27.786 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T06:37:27.786 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:37:27.786 INFO:teuthology.orchestra.run.smithi042.stdout:Upgraded: 2024-07-26T06:37:27.786 INFO:teuthology.orchestra.run.smithi042.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T06:37:27.787 INFO:teuthology.orchestra.run.smithi042.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T06:37:27.787 INFO:teuthology.orchestra.run.smithi042.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T06:37:27.787 INFO:teuthology.orchestra.run.smithi042.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T06:37:27.787 INFO:teuthology.orchestra.run.smithi042.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T06:37:27.787 INFO:teuthology.orchestra.run.smithi042.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T06:37:27.787 INFO:teuthology.orchestra.run.smithi042.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T06:37:27.787 INFO:teuthology.orchestra.run.smithi042.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.787 INFO:teuthology.orchestra.run.smithi042.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.787 INFO:teuthology.orchestra.run.smithi042.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T06:37:27.787 INFO:teuthology.orchestra.run.smithi042.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T06:37:27.787 INFO:teuthology.orchestra.run.smithi042.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T06:37:27.787 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-07-26T06:37:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T06:37:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T06:37:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T06:37:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T06:37:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.788 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.789 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.789 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.789 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.789 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.789 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.789 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.789 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.789 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.789 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.789 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.789 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.789 INFO:teuthology.orchestra.run.smithi042.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.790 INFO:teuthology.orchestra.run.smithi042.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.790 INFO:teuthology.orchestra.run.smithi042.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.790 INFO:teuthology.orchestra.run.smithi042.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:27.790 INFO:teuthology.orchestra.run.smithi042.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T06:37:27.790 INFO:teuthology.orchestra.run.smithi042.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T06:37:27.790 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T06:37:27.790 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T06:37:27.790 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T06:37:27.790 INFO:teuthology.orchestra.run.smithi042.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T06:37:27.790 INFO:teuthology.orchestra.run.smithi042.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T06:37:27.790 INFO:teuthology.orchestra.run.smithi042.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T06:37:27.791 INFO:teuthology.orchestra.run.smithi042.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T06:37:27.791 INFO:teuthology.orchestra.run.smithi042.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T06:37:27.791 INFO:teuthology.orchestra.run.smithi042.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T06:37:27.791 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.791 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.791 INFO:teuthology.orchestra.run.smithi042.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.791 INFO:teuthology.orchestra.run.smithi042.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T06:37:27.791 INFO:teuthology.orchestra.run.smithi042.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T06:37:27.791 INFO:teuthology.orchestra.run.smithi042.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T06:37:27.791 INFO:teuthology.orchestra.run.smithi042.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T06:37:27.791 INFO:teuthology.orchestra.run.smithi042.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T06:37:27.791 INFO:teuthology.orchestra.run.smithi042.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.792 INFO:teuthology.orchestra.run.smithi042.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.792 INFO:teuthology.orchestra.run.smithi042.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T06:37:27.792 INFO:teuthology.orchestra.run.smithi042.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.792 INFO:teuthology.orchestra.run.smithi042.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T06:37:27.792 INFO:teuthology.orchestra.run.smithi042.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T06:37:27.792 INFO:teuthology.orchestra.run.smithi042.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T06:37:27.792 INFO:teuthology.orchestra.run.smithi042.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T06:37:27.792 INFO:teuthology.orchestra.run.smithi042.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T06:37:27.792 INFO:teuthology.orchestra.run.smithi042.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T06:37:27.792 INFO:teuthology.orchestra.run.smithi042.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T06:37:27.792 INFO:teuthology.orchestra.run.smithi042.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T06:37:27.792 INFO:teuthology.orchestra.run.smithi042.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T06:37:27.793 INFO:teuthology.orchestra.run.smithi042.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T06:37:27.793 INFO:teuthology.orchestra.run.smithi042.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T06:37:27.793 INFO:teuthology.orchestra.run.smithi042.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T06:37:27.793 INFO:teuthology.orchestra.run.smithi042.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T06:37:27.793 INFO:teuthology.orchestra.run.smithi042.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T06:37:27.793 INFO:teuthology.orchestra.run.smithi042.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T06:37:27.793 INFO:teuthology.orchestra.run.smithi042.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T06:37:27.793 INFO:teuthology.orchestra.run.smithi042.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T06:37:27.793 INFO:teuthology.orchestra.run.smithi042.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T06:37:27.793 INFO:teuthology.orchestra.run.smithi042.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T06:37:27.793 INFO:teuthology.orchestra.run.smithi042.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T06:37:27.793 INFO:teuthology.orchestra.run.smithi042.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T06:37:27.794 INFO:teuthology.orchestra.run.smithi042.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T06:37:27.794 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T06:37:27.794 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.794 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.794 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.794 INFO:teuthology.orchestra.run.smithi042.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T06:37:27.794 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T06:37:27.794 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T06:37:27.794 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T06:37:27.794 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T06:37:27.794 INFO:teuthology.orchestra.run.smithi042.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T06:37:27.794 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T06:37:27.794 INFO:teuthology.orchestra.run.smithi042.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T06:37:27.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T06:37:27.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T06:37:27.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T06:37:27.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T06:37:27.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T06:37:27.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T06:37:27.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T06:37:27.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T06:37:27.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T06:37:27.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T06:37:27.795 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T06:37:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T06:37:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T06:37:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T06:37:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T06:37:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T06:37:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T06:37:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T06:37:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T06:37:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T06:37:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T06:37:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T06:37:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T06:37:27.796 INFO:teuthology.orchestra.run.smithi042.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T06:37:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T06:37:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T06:37:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T06:37:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T06:37:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T06:37:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T06:37:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T06:37:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T06:37:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T06:37:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T06:37:27.797 INFO:teuthology.orchestra.run.smithi042.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T06:37:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T06:37:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T06:37:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T06:37:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T06:37:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T06:37:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T06:37:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T06:37:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T06:37:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T06:37:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T06:37:27.798 INFO:teuthology.orchestra.run.smithi042.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T06:37:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T06:37:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T06:37:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T06:37:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T06:37:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T06:37:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T06:37:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T06:37:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T06:37:27.799 INFO:teuthology.orchestra.run.smithi042.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T06:37:27.800 INFO:teuthology.orchestra.run.smithi042.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T06:37:27.800 INFO:teuthology.orchestra.run.smithi042.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T06:37:27.800 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:37:27.800 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-07-26T06:37:28.241 DEBUG:teuthology.parallel:result is None 2024-07-26T06:37:30.448 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T06:37:30.448 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T06:37:30.448 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T06:37:30.448 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T06:37:30.448 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T06:37:30.448 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T06:37:30.448 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T06:37:30.448 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T06:37:30.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T06:37:30.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T06:37:30.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T06:37:30.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T06:37:30.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T06:37:30.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T06:37:30.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T06:37:30.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T06:37:30.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T06:37:30.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T06:37:30.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T06:37:30.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T06:37:30.449 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T06:37:30.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T06:37:30.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T06:37:30.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T06:37:30.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T06:37:30.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T06:37:30.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T06:37:30.450 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T06:37:30.462 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T06:37:30.462 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T06:37:30.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T06:37:30.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T06:37:30.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T06:37:30.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T06:37:30.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T06:37:30.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T06:37:30.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T06:37:30.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T06:37:30.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T06:37:30.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T06:37:30.463 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T06:37:30.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T06:37:30.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T06:37:30.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T06:37:30.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T06:37:30.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T06:37:30.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T06:37:30.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T06:37:30.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T06:37:30.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T06:37:30.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T06:37:30.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T06:37:30.464 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T06:37:30.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T06:37:30.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T06:37:30.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T06:37:30.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T06:37:30.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T06:37:30.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T06:37:30.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T06:37:30.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T06:37:30.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T06:37:30.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T06:37:30.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T06:37:30.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T06:37:30.465 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T06:37:30.466 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T06:37:30.466 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T06:37:30.466 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T06:37:30.466 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T06:37:30.466 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T06:37:30.466 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T06:37:30.466 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T06:37:30.466 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T06:37:30.466 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T06:37:30.466 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T06:37:30.466 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T06:37:30.467 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T06:37:30.467 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T06:37:30.467 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T06:37:30.467 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T06:37:30.467 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T06:37:30.467 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T06:37:30.467 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T06:37:30.467 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T06:37:30.467 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T06:37:30.467 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T06:37:30.467 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T06:37:30.467 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T06:37:30.467 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T06:37:30.468 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T06:37:30.468 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T06:37:30.468 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T06:37:30.468 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T06:37:30.468 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T06:37:30.468 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T06:37:30.468 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T06:37:30.468 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T06:37:30.468 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T06:37:30.468 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T06:37:30.468 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T06:37:30.469 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T06:37:30.469 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T06:37:30.469 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T06:37:30.469 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T06:37:30.469 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T06:37:30.469 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T06:37:30.469 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T06:37:30.469 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T06:37:30.469 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T06:37:30.469 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T06:37:30.469 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T06:37:30.469 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T06:37:30.470 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T06:37:30.470 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T06:37:30.470 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T06:37:30.470 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T06:37:30.470 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T06:37:30.470 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T06:37:30.470 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T06:37:30.470 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T06:37:30.470 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T06:37:30.470 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T06:37:30.470 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T06:37:30.471 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T06:37:30.471 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T06:37:30.471 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T06:37:30.471 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T06:37:30.471 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T06:37:30.471 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T06:37:30.471 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T06:37:30.471 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T06:37:30.471 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T06:37:30.471 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T06:37:30.471 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T06:37:30.472 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T06:37:30.472 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T06:37:30.472 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T06:37:30.472 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T06:37:30.472 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T06:37:30.472 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T06:37:30.472 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T06:37:30.472 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T06:37:30.472 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T06:37:30.472 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T06:37:30.472 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T06:37:30.473 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T06:37:30.473 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T06:37:30.473 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T06:37:30.473 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T06:37:30.473 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T06:37:30.473 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T06:37:30.473 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T06:37:30.473 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T06:37:30.473 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T06:37:30.473 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T06:37:30.473 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T06:37:30.474 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T06:37:30.474 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T06:37:30.474 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T06:37:30.474 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T06:37:30.474 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T06:37:30.474 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T06:37:30.474 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T06:37:30.474 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T06:37:30.474 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T06:37:30.474 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T06:37:30.475 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T06:37:30.475 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T06:37:30.475 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T06:37:30.475 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T06:37:30.475 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T06:37:33.342 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T06:37:33.342 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:37:33.342 INFO:teuthology.orchestra.run.smithi164.stdout:Upgraded: 2024-07-26T06:37:33.343 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T06:37:33.343 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T06:37:33.343 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T06:37:33.343 INFO:teuthology.orchestra.run.smithi164.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T06:37:33.343 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T06:37:33.343 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T06:37:33.343 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T06:37:33.343 INFO:teuthology.orchestra.run.smithi164.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.343 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.343 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T06:37:33.343 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T06:37:33.343 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T06:37:33.344 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-07-26T06:37:33.344 INFO:teuthology.orchestra.run.smithi164.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T06:37:33.344 INFO:teuthology.orchestra.run.smithi164.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T06:37:33.344 INFO:teuthology.orchestra.run.smithi164.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T06:37:33.344 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T06:37:33.344 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.344 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.344 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.344 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.344 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:33.345 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.345 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.345 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.345 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:33.345 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:33.345 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:33.345 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:33.345 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:33.345 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.345 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.345 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:33.346 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.346 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.346 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.346 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:33.346 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:33.346 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.346 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.346 INFO:teuthology.orchestra.run.smithi164.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T06:37:33.346 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T06:37:33.346 INFO:teuthology.orchestra.run.smithi164.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T06:37:33.347 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T06:37:33.347 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T06:37:33.347 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T06:37:33.347 INFO:teuthology.orchestra.run.smithi164.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T06:37:33.347 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T06:37:33.347 INFO:teuthology.orchestra.run.smithi164.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T06:37:33.347 INFO:teuthology.orchestra.run.smithi164.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T06:37:33.347 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T06:37:33.347 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T06:37:33.347 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.347 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.348 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.348 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T06:37:33.348 INFO:teuthology.orchestra.run.smithi164.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T06:37:33.348 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T06:37:33.348 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T06:37:33.348 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T06:37:33.348 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.348 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.348 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T06:37:33.349 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.349 INFO:teuthology.orchestra.run.smithi164.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T06:37:33.349 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T06:37:33.349 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T06:37:33.349 INFO:teuthology.orchestra.run.smithi164.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T06:37:33.349 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T06:37:33.349 INFO:teuthology.orchestra.run.smithi164.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T06:37:33.349 INFO:teuthology.orchestra.run.smithi164.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T06:37:33.349 INFO:teuthology.orchestra.run.smithi164.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T06:37:33.349 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T06:37:33.349 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T06:37:33.350 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T06:37:33.350 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T06:37:33.350 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T06:37:33.350 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T06:37:33.350 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T06:37:33.350 INFO:teuthology.orchestra.run.smithi164.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T06:37:33.350 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T06:37:33.350 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T06:37:33.350 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T06:37:33.350 INFO:teuthology.orchestra.run.smithi164.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T06:37:33.351 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T06:37:33.351 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T06:37:33.351 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T06:37:33.351 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.351 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.351 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.351 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T06:37:33.351 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T06:37:33.351 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T06:37:33.351 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T06:37:33.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T06:37:33.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T06:37:33.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T06:37:33.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T06:37:33.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T06:37:33.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T06:37:33.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T06:37:33.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T06:37:33.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T06:37:33.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T06:37:33.352 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T06:37:33.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T06:37:33.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T06:37:33.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T06:37:33.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T06:37:33.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T06:37:33.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T06:37:33.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T06:37:33.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T06:37:33.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T06:37:33.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T06:37:33.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T06:37:33.353 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T06:37:33.354 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T06:37:33.354 INFO:teuthology.orchestra.run.smithi164.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T06:37:33.354 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T06:37:33.354 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T06:37:33.354 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T06:37:33.354 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T06:37:33.354 INFO:teuthology.orchestra.run.smithi164.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T06:37:33.354 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T06:37:33.354 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T06:37:33.354 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T06:37:33.354 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T06:37:33.355 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T06:37:33.355 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T06:37:33.355 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.355 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.355 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T06:37:33.355 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T06:37:33.355 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T06:37:33.355 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.355 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T06:37:33.355 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T06:37:33.355 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T06:37:33.355 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T06:37:33.356 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T06:37:33.356 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T06:37:33.356 INFO:teuthology.orchestra.run.smithi164.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T06:37:33.356 INFO:teuthology.orchestra.run.smithi164.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T06:37:33.356 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T06:37:33.356 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T06:37:33.356 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T06:37:33.356 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T06:37:33.356 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T06:37:33.356 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T06:37:33.356 INFO:teuthology.orchestra.run.smithi164.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T06:37:33.356 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.357 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.357 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T06:37:33.357 INFO:teuthology.orchestra.run.smithi164.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T06:37:33.357 INFO:teuthology.orchestra.run.smithi164.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T06:37:33.357 INFO:teuthology.orchestra.run.smithi164.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T06:37:33.357 INFO:teuthology.orchestra.run.smithi164.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T06:37:33.357 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T06:37:33.357 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T06:37:33.357 INFO:teuthology.orchestra.run.smithi164.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T06:37:33.357 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:37:33.357 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-07-26T06:37:33.789 DEBUG:teuthology.parallel:result is None 2024-07-26T06:37:33.790 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T06:37:33.951 DEBUG:teuthology.orchestra.run.smithi042:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T06:37:33.979 INFO:teuthology.orchestra.run.smithi042.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T06:37:33.980 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T06:37:33.980 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T06:37:33.982 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T06:37:34.128 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T06:37:34.152 INFO:teuthology.orchestra.run.smithi163.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T06:37:34.152 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T06:37:34.152 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T06:37:34.154 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T06:37:34.293 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T06:37:34.318 INFO:teuthology.orchestra.run.smithi164.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T06:37:34.318 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T06:37:34.318 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T06:37:34.320 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-07-26T06:37:34.320 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-07-26T06:37:34.320 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T06:37:34.354 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:37:34.354 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T06:37:34.384 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:37:34.384 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T06:37:34.414 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-07-26T06:37:34.414 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-07-26T06:37:34.414 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T06:37:34.444 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T06:37:34.513 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:37:34.513 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T06:37:34.542 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T06:37:34.611 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:37:34.611 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T06:37:34.639 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T06:37:34.705 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-07-26T06:37:34.705 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-07-26T06:37:34.705 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T06:37:34.734 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T06:37:34.801 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:37:34.801 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T06:37:34.829 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T06:37:34.896 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:37:34.896 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T06:37:34.924 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T06:37:34.991 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-07-26T06:37:34.992 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-07-26T06:37:34.992 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T06:37:35.018 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T06:37:35.085 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:37:35.085 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T06:37:35.113 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T06:37:35.180 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:37:35.180 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T06:37:35.209 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T06:37:35.276 INFO:teuthology.run_tasks:Running task cephadm... 2024-07-26T06:37:35.414 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'global': {'ms die on skipped message': False}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-07-26T06:37:35.414 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T06:37:35.414 INFO:tasks.cephadm:Cluster fsid is 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:37:35.414 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-07-26T06:37:35.415 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.42', 'mon.b': '172.21.15.163', 'mon.c': '172.21.15.164'} 2024-07-26T06:37:35.415 INFO:tasks.cephadm:First mon is mon.a on smithi042 2024-07-26T06:37:35.415 INFO:tasks.cephadm:First mgr is x 2024-07-26T06:37:35.415 INFO:tasks.cephadm:Normalizing hostnames... 2024-07-26T06:37:35.415 DEBUG:teuthology.orchestra.run.smithi042:> sudo hostname $(hostname -s) 2024-07-26T06:37:35.442 DEBUG:teuthology.orchestra.run.smithi163:> sudo hostname $(hostname -s) 2024-07-26T06:37:35.471 DEBUG:teuthology.orchestra.run.smithi164:> sudo hostname $(hostname -s) 2024-07-26T06:37:35.500 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-07-26T06:37:35.500 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T06:37:35.640 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/81524/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-5320-ge8f88bb1', 'node_name': '172.21.4.69+confusa07', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-5320.ge8f88bb1'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-07-25 16:18:49.953482', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240725.121331-debug', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'archs': ['arm64', 'x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/81524/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-5320-ge8f88bb1', 'node_name': '172.21.2.8+braggi08', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-5320.ge8f88bb1'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-07-25 15:10:42.298781', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240725.121331-debug', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-07-26T06:37:35.778 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-vshankar-testing-20240725.121331-debug, sha1 e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T06:37:35.779 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm 2024-07-26T06:37:35.779 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm 2024-07-26T06:37:35.779 DEBUG:teuthology.orchestra.run.smithi042:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T06:37:36.174 INFO:teuthology.orchestra.run.smithi042.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 06:37 /home/ubuntu/cephtest/cephadm 2024-07-26T06:37:36.175 DEBUG:teuthology.orchestra.run.smithi163:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T06:37:36.558 INFO:teuthology.orchestra.run.smithi163.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 06:37 /home/ubuntu/cephtest/cephadm 2024-07-26T06:37:36.558 DEBUG:teuthology.orchestra.run.smithi164:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T06:37:36.925 INFO:teuthology.orchestra.run.smithi164.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 06:37 /home/ubuntu/cephtest/cephadm 2024-07-26T06:37:36.926 DEBUG:teuthology.orchestra.run.smithi042:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T06:37:36.946 DEBUG:teuthology.orchestra.run.smithi163:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T06:37:36.965 DEBUG:teuthology.orchestra.run.smithi164:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T06:37:37.018 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 on all hosts... 2024-07-26T06:37:37.019 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T06:37:37.021 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T06:37:37.024 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T06:37:37.840 INFO:teuthology.orchestra.run.smithi164.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T06:37:37.908 INFO:teuthology.orchestra.run.smithi163.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T06:37:37.936 INFO:teuthology.orchestra.run.smithi042.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T06:38:10.383 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-07-26T06:38:10.383 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T06:38:10.384 INFO:teuthology.orchestra.run.smithi163.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T06:38:10.384 INFO:teuthology.orchestra.run.smithi163.stdout: "repo_digests": [ 2024-07-26T06:38:10.384 INFO:teuthology.orchestra.run.smithi163.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T06:38:10.384 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-07-26T06:38:10.384 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-07-26T06:38:17.911 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-07-26T06:38:17.911 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T06:38:17.911 INFO:teuthology.orchestra.run.smithi164.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T06:38:17.911 INFO:teuthology.orchestra.run.smithi164.stdout: "repo_digests": [ 2024-07-26T06:38:17.912 INFO:teuthology.orchestra.run.smithi164.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T06:38:17.912 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-07-26T06:38:17.912 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-07-26T06:38:20.541 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:38:20.542 INFO:teuthology.orchestra.run.smithi042.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T06:38:20.542 INFO:teuthology.orchestra.run.smithi042.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T06:38:20.542 INFO:teuthology.orchestra.run.smithi042.stdout: "repo_digests": [ 2024-07-26T06:38:20.542 INFO:teuthology.orchestra.run.smithi042.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T06:38:20.542 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-07-26T06:38:20.542 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:38:20.568 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /etc/ceph 2024-07-26T06:38:20.607 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /etc/ceph 2024-07-26T06:38:20.639 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph 2024-07-26T06:38:20.672 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 777 /etc/ceph 2024-07-26T06:38:20.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 777 /etc/ceph 2024-07-26T06:38:20.726 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 777 /etc/ceph 2024-07-26T06:38:20.753 INFO:tasks.cephadm:Writing seed config... 2024-07-26T06:38:20.767 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-07-26T06:38:20.767 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-07-26T06:38:20.767 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-07-26T06:38:20.767 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-07-26T06:38:20.767 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-07-26T06:38:20.767 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-07-26T06:38:20.767 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-07-26T06:38:20.768 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-07-26T06:38:20.768 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-07-26T06:38:20.768 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-07-26T06:38:20.768 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-07-26T06:38:20.768 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-07-26T06:38:20.768 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-07-26T06:38:20.768 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-07-26T06:38:20.768 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-07-26T06:38:20.768 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-07-26T06:38:20.768 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-07-26T06:38:20.768 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-07-26T06:38:20.768 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-07-26T06:38:20.768 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-07-26T06:38:20.769 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-07-26T06:38:20.769 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-07-26T06:38:20.769 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-07-26T06:38:20.769 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-07-26T06:38:20.769 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-07-26T06:38:20.770 DEBUG:teuthology.orchestra.run.smithi042:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-07-26T06:38:20.786 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 8b3e2750-4b19-11ef-bcb6-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-07-26T06:38:20.786 DEBUG:teuthology.orchestra.run.smithi042:mon.a> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mon.a.service 2024-07-26T06:38:20.829 DEBUG:teuthology.orchestra.run.smithi042:mgr.x> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mgr.x.service 2024-07-26T06:38:20.872 INFO:tasks.cephadm:Bootstrapping... 2024-07-26T06:38:20.872 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 -v bootstrap --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.42 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-07-26T06:38:21.088 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-07-26T06:38:21.088 INFO:teuthology.orchestra.run.smithi042.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', '-v', 'bootstrap', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.42', '--skip-admin-label'] 2024-07-26T06:38:21.088 INFO:teuthology.orchestra.run.smithi042.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-07-26T06:38:21.089 INFO:teuthology.orchestra.run.smithi042.stdout:Verifying podman|docker is present... 2024-07-26T06:38:21.115 INFO:teuthology.orchestra.run.smithi042.stdout:/bin/podman: stdout 5.1.2 2024-07-26T06:38:21.116 INFO:teuthology.orchestra.run.smithi042.stdout:Verifying lvm2 is present... 2024-07-26T06:38:21.116 INFO:teuthology.orchestra.run.smithi042.stdout:Verifying time synchronization is in place... 2024-07-26T06:38:21.123 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-07-26T06:38:21.124 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-07-26T06:38:21.130 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-07-26T06:38:21.130 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout inactive 2024-07-26T06:38:21.137 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout enabled 2024-07-26T06:38:21.143 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout active 2024-07-26T06:38:21.144 INFO:teuthology.orchestra.run.smithi042.stdout:Unit chronyd.service is enabled and running 2024-07-26T06:38:21.144 INFO:teuthology.orchestra.run.smithi042.stdout:Repeating the final host check... 2024-07-26T06:38:21.168 INFO:teuthology.orchestra.run.smithi042.stdout:/bin/podman: stdout 5.1.2 2024-07-26T06:38:21.168 INFO:teuthology.orchestra.run.smithi042.stdout:podman (/bin/podman) version 5.1.2 is present 2024-07-26T06:38:21.168 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl is present 2024-07-26T06:38:21.168 INFO:teuthology.orchestra.run.smithi042.stdout:lvcreate is present 2024-07-26T06:38:21.174 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-07-26T06:38:21.174 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-07-26T06:38:21.181 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-07-26T06:38:21.181 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout inactive 2024-07-26T06:38:21.188 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout enabled 2024-07-26T06:38:21.194 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout active 2024-07-26T06:38:21.194 INFO:teuthology.orchestra.run.smithi042.stdout:Unit chronyd.service is enabled and running 2024-07-26T06:38:21.195 INFO:teuthology.orchestra.run.smithi042.stdout:Host looks OK 2024-07-26T06:38:21.195 INFO:teuthology.orchestra.run.smithi042.stdout:Cluster fsid: 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:38:21.195 INFO:teuthology.orchestra.run.smithi042.stdout:Acquiring lock 139880637465408 on /run/cephadm/8b3e2750-4b19-11ef-bcb6-c7b262605968.lock 2024-07-26T06:38:21.195 INFO:teuthology.orchestra.run.smithi042.stdout:Lock 139880637465408 acquired on /run/cephadm/8b3e2750-4b19-11ef-bcb6-c7b262605968.lock 2024-07-26T06:38:21.196 INFO:teuthology.orchestra.run.smithi042.stdout:Verifying IP 172.21.15.42 port 3300 ... 2024-07-26T06:38:21.196 INFO:teuthology.orchestra.run.smithi042.stdout:Verifying IP 172.21.15.42 port 6789 ... 2024-07-26T06:38:21.196 INFO:teuthology.orchestra.run.smithi042.stdout:Base mon IP(s) is [172.21.15.42:3300, 172.21.15.42:6789], mon addrv is [v2:172.21.15.42:3300,v1:172.21.15.42:6789] 2024-07-26T06:38:21.199 INFO:teuthology.orchestra.run.smithi042.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.42 metric 100 2024-07-26T06:38:21.199 INFO:teuthology.orchestra.run.smithi042.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.42 metric 100 2024-07-26T06:38:21.201 INFO:teuthology.orchestra.run.smithi042.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-07-26T06:38:21.202 INFO:teuthology.orchestra.run.smithi042.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-07-26T06:38:21.202 INFO:teuthology.orchestra.run.smithi042.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-07-26T06:38:21.204 INFO:teuthology.orchestra.run.smithi042.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-07-26T06:38:21.204 INFO:teuthology.orchestra.run.smithi042.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-07-26T06:38:21.204 INFO:teuthology.orchestra.run.smithi042.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-07-26T06:38:21.205 INFO:teuthology.orchestra.run.smithi042.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-07-26T06:38:21.205 INFO:teuthology.orchestra.run.smithi042.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1440/64 scope link noprefixroute 2024-07-26T06:38:21.205 INFO:teuthology.orchestra.run.smithi042.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-07-26T06:38:21.205 INFO:teuthology.orchestra.run.smithi042.stdout:Mon IP `172.21.15.42` is in CIDR network `172.21.0.0/20` 2024-07-26T06:38:21.205 INFO:teuthology.orchestra.run.smithi042.stdout:Mon IP `172.21.15.42` is in CIDR network `172.21.0.0/20` 2024-07-26T06:38:21.206 INFO:teuthology.orchestra.run.smithi042.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-07-26T06:38:21.206 INFO:teuthology.orchestra.run.smithi042.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-07-26T06:38:21.206 INFO:teuthology.orchestra.run.smithi042.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T06:38:21.653 INFO:teuthology.orchestra.run.smithi042.stdout:/bin/podman: stdout ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 2024-07-26T06:38:21.654 INFO:teuthology.orchestra.run.smithi042.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T06:38:21.654 INFO:teuthology.orchestra.run.smithi042.stdout:/bin/podman: stderr Getting image source signatures 2024-07-26T06:38:21.654 INFO:teuthology.orchestra.run.smithi042.stdout:/bin/podman: stderr Copying blob sha256:43735fbce9214f061a6894341d7822563f5aafe16fc7532189f4415cff6d2d91 2024-07-26T06:38:21.654 INFO:teuthology.orchestra.run.smithi042.stdout:/bin/podman: stderr Copying blob sha256:27b44fafaf0d8ddf6712bf077b0e069fd1632e81c2016bcdf9194ad64e762303 2024-07-26T06:38:21.654 INFO:teuthology.orchestra.run.smithi042.stdout:/bin/podman: stderr Copying config sha256:ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 2024-07-26T06:38:21.654 INFO:teuthology.orchestra.run.smithi042.stdout:/bin/podman: stderr Writing manifest to image destination 2024-07-26T06:38:22.533 INFO:teuthology.orchestra.run.smithi042.stdout:ceph: stdout ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:38:22.533 INFO:teuthology.orchestra.run.smithi042.stdout:Ceph version: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:38:22.533 INFO:teuthology.orchestra.run.smithi042.stdout:Extracting ceph user uid/gid from container image... 2024-07-26T06:38:23.464 INFO:teuthology.orchestra.run.smithi042.stdout:stat: stdout 167 167 2024-07-26T06:38:23.464 INFO:teuthology.orchestra.run.smithi042.stdout:Creating initial keys... 2024-07-26T06:38:24.361 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-authtool: stdout AQBfRKNmrfvqLxAAlEHKfQ4abrW0EEPGhv1Eig== 2024-07-26T06:38:25.225 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-authtool: stdout AQBgRKNmvWTIJxAAEoPOOTGZEUoFz+mtK6qlaQ== 2024-07-26T06:38:26.143 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph-authtool: stdout AQBhRKNmUCaxIRAAoKuxHRFTm7yN4hN30lX1mQ== 2024-07-26T06:38:26.143 INFO:teuthology.orchestra.run.smithi042.stdout:Creating initial monmap... 2024-07-26T06:38:26.996 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-07-26T06:38:26.996 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-07-26T06:38:26.996 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:38:26.996 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-07-26T06:38:26.996 INFO:teuthology.orchestra.run.smithi042.stdout:monmaptool for a [v2:172.21.15.42:3300,v1:172.21.15.42:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-07-26T06:38:26.996 INFO:teuthology.orchestra.run.smithi042.stdout:setting min_mon_release = quincy 2024-07-26T06:38:26.996 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/monmaptool: set fsid to 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:38:26.996 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-07-26T06:38:26.996 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:38:26.996 INFO:teuthology.orchestra.run.smithi042.stdout:Creating mon... 2024-07-26T06:38:27.835 INFO:teuthology.orchestra.run.smithi042.stdout:create mon.a on 2024-07-26T06:38:28.054 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-07-26T06:38:28.213 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-07-26T06:38:28.403 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968.target → /etc/systemd/system/ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968.target. 2024-07-26T06:38:28.403 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968.target → /etc/systemd/system/ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968.target. 2024-07-26T06:38:28.660 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mon.a 2024-07-26T06:38:28.660 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mon.a.service: Unit ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mon.a.service not loaded. 2024-07-26T06:38:28.846 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968.target.wants/ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mon.a.service → /etc/systemd/system/ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@.service. 2024-07-26T06:38:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:28 smithi042 systemd[1]: Starting Ceph mon.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:38:29.158 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:29 smithi042 podman[32386]: 2024-07-26 06:38:29.05897352 +0000 UTC m=+0.086301084 container create 5a9d41068b7130f72b32dd0be92a43b8ec3423b04d28a2802ef2a8cd3167933f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=) 2024-07-26T06:38:29.158 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:29 smithi042 podman[32386]: 2024-07-26 06:38:28.990945212 +0000 UTC m=+0.018272794 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T06:38:29.351 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T06:38:29.351 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout disabled 2024-07-26T06:38:29.359 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T06:38:29.359 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout inactive 2024-07-26T06:38:29.359 INFO:teuthology.orchestra.run.smithi042.stdout:firewalld.service is not enabled 2024-07-26T06:38:29.359 INFO:teuthology.orchestra.run.smithi042.stdout:Not possible to enable service . firewalld.service is not available 2024-07-26T06:38:29.359 INFO:teuthology.orchestra.run.smithi042.stdout:Waiting for mon to start... 2024-07-26T06:38:29.360 INFO:teuthology.orchestra.run.smithi042.stdout:Waiting for mon... 2024-07-26T06:38:29.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:29 smithi042 podman[32386]: 2024-07-26 06:38:29.253428997 +0000 UTC m=+0.280756569 container init 5a9d41068b7130f72b32dd0be92a43b8ec3423b04d28a2802ef2a8cd3167933f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-07-26T06:38:29.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:29 smithi042 podman[32386]: 2024-07-26 06:38:29.257255374 +0000 UTC m=+0.284582947 container start 5a9d41068b7130f72b32dd0be92a43b8ec3423b04d28a2802ef2a8cd3167933f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mon-a, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T06:38:29.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:29 smithi042 ceph-mon[32404]: mkfs 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:38:29.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:29 smithi042 ceph-mon[32404]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T06:38:29.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:29 smithi042 bash[32386]: 5a9d41068b7130f72b32dd0be92a43b8ec3423b04d28a2802ef2a8cd3167933f 2024-07-26T06:38:29.467 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:29 smithi042 systemd[1]: Started Ceph mon.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:38:30.364 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout cluster: 2024-07-26T06:38:30.364 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout id: 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:38:30.364 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-07-26T06:38:30.364 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-07-26T06:38:30.365 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout services: 2024-07-26T06:38:30.365 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.578281s) 2024-07-26T06:38:30.365 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-07-26T06:38:30.365 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-07-26T06:38:30.365 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-07-26T06:38:30.365 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout data: 2024-07-26T06:38:30.365 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-07-26T06:38:30.365 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-07-26T06:38:30.365 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-07-26T06:38:30.365 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout pgs: 2024-07-26T06:38:30.365 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-07-26T06:38:30.365 INFO:teuthology.orchestra.run.smithi042.stdout:mon is available 2024-07-26T06:38:30.365 INFO:teuthology.orchestra.run.smithi042.stdout:Assimilating anything we can from ceph.conf... 2024-07-26T06:38:30.477 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:30 smithi042 ceph-mon[32404]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T06:38:30.477 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:30 smithi042 ceph-mon[32404]: monmap epoch 1 2024-07-26T06:38:30.477 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:30 smithi042 ceph-mon[32404]: fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:38:30.477 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:30 smithi042 ceph-mon[32404]: last_changed 2024-07-26T06:38:26.468434+0000 2024-07-26T06:38:30.478 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:30 smithi042 ceph-mon[32404]: created 2024-07-26T06:38:26.468434+0000 2024-07-26T06:38:30.478 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:30 smithi042 ceph-mon[32404]: min_mon_release 19 (squid) 2024-07-26T06:38:30.478 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:30 smithi042 ceph-mon[32404]: election_strategy: 1 2024-07-26T06:38:30.478 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:30 smithi042 ceph-mon[32404]: 0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-07-26T06:38:30.478 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:30 smithi042 ceph-mon[32404]: fsmap 2024-07-26T06:38:30.478 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:30 smithi042 ceph-mon[32404]: osdmap e1: 0 total, 0 up, 0 in 2024-07-26T06:38:30.478 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:30 smithi042 ceph-mon[32404]: mgrmap e1: no daemons active 2024-07-26T06:38:30.478 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:30 smithi042 ceph-mon[32404]: from='client.? 172.21.15.42:0/2875194817' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-07-26T06:38:31.399 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:31 smithi042 ceph-mon[32404]: from='client.? 172.21.15.42:0/1153449852' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-07-26T06:38:31.399 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:31 smithi042 ceph-mon[32404]: from='client.? 172.21.15.42:0/1153449852' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-07-26T06:38:31.404 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-07-26T06:38:31.404 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout [global] 2024-07-26T06:38:31.404 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout fsid = 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:38:31.404 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-07-26T06:38:31.404 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.42:3300,v1:172.21.15.42:6789] 2024-07-26T06:38:31.404 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-07-26T06:38:31.404 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-07-26T06:38:31.404 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-07-26T06:38:31.404 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-07-26T06:38:31.405 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-07-26T06:38:31.405 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout [mgr] 2024-07-26T06:38:31.405 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-07-26T06:38:31.405 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-07-26T06:38:31.405 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout [osd] 2024-07-26T06:38:31.405 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-07-26T06:38:31.405 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-07-26T06:38:31.405 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-07-26T06:38:31.405 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-07-26T06:38:31.405 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-07-26T06:38:31.405 INFO:teuthology.orchestra.run.smithi042.stdout:Generating new minimal ceph.conf... 2024-07-26T06:38:32.375 INFO:teuthology.orchestra.run.smithi042.stdout:Restarting the monitor... 2024-07-26T06:38:32.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:32 smithi042 ceph-mon[32404]: from='client.? 172.21.15.42:0/2818638299' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:38:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:32 smithi042 systemd[1]: Stopping Ceph mon.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:38:33.075 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mon-a[32400]: 2024-07-26T06:38:32.658+0000 7f9ac2cbd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-07-26T06:38:33.076 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mon-a[32400]: 2024-07-26T06:38:32.658+0000 7f9ac2cbd640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-07-26T06:38:33.076 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:32 smithi042 podman[32647]: 2024-07-26 06:38:32.832899968 +0000 UTC m=+0.279026551 container died 5a9d41068b7130f72b32dd0be92a43b8ec3423b04d28a2802ef2a8cd3167933f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mon-a, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-07-26T06:38:33.334 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:33 smithi042 podman[32647]: 2024-07-26 06:38:33.149809453 +0000 UTC m=+0.595936035 container cleanup 5a9d41068b7130f72b32dd0be92a43b8ec3423b04d28a2802ef2a8cd3167933f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mon-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T06:38:33.334 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:33 smithi042 bash[32647]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mon-a 2024-07-26T06:38:33.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:33 smithi042 podman[32662]: 2024-07-26 06:38:33.334057719 +0000 UTC m=+0.498083203 container remove 5a9d41068b7130f72b32dd0be92a43b8ec3423b04d28a2802ef2a8cd3167933f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mon-a, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-07-26T06:38:33.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:33 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mon.a.service: Deactivated successfully. 2024-07-26T06:38:33.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:33 smithi042 systemd[1]: Stopped Ceph mon.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:38:33.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:33 smithi042 systemd[1]: Starting Ceph mon.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:38:34.054 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:33 smithi042 podman[32756]: 2024-07-26 06:38:33.776865947 +0000 UTC m=+0.096515108 container create 33bc7f46d883f96e8551cf60ff295bd2e84044b6535668aa93363e0d4f9c1841 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T06:38:34.054 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:33 smithi042 podman[32756]: 2024-07-26 06:38:33.69805818 +0000 UTC m=+0.017707305 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T06:38:34.054 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:33 smithi042 podman[32756]: 2024-07-26 06:38:33.979231669 +0000 UTC m=+0.298880788 container init 33bc7f46d883f96e8551cf60ff295bd2e84044b6535668aa93363e0d4f9c1841 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T06:38:34.055 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:33 smithi042 podman[32756]: 2024-07-26 06:38:33.982603321 +0000 UTC m=+0.302252437 container start 33bc7f46d883f96e8551cf60ff295bd2e84044b6535668aa93363e0d4f9c1841 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-07-26T06:38:34.055 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: set uid:gid to 167:167 (ceph:ceph) 2024-07-26T06:38:34.055 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev), process ceph-mon, pid 2 2024-07-26T06:38:34.055 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: pidfile_write: ignore empty --pid-file 2024-07-26T06:38:34.055 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: load: jerasure load: lrc 2024-07-26T06:38:34.055 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: RocksDB version: 7.9.2 2024-07-26T06:38:34.055 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Git sha 0 2024-07-26T06:38:34.055 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Compile date 2024-07-25 13:49:38 2024-07-26T06:38:34.055 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: DB SUMMARY 2024-07-26T06:38:34.055 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: DB Session ID: 8M0QPN7OPPDRCAHQBDSY 2024-07-26T06:38:34.055 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: CURRENT file: CURRENT 2024-07-26T06:38:34.056 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: IDENTITY file: IDENTITY 2024-07-26T06:38:34.056 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-07-26T06:38:34.056 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-07-26T06:38:34.056 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 94421 ; 2024-07-26T06:38:34.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.error_if_exists: 0 2024-07-26T06:38:34.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.create_if_missing: 0 2024-07-26T06:38:34.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.paranoid_checks: 1 2024-07-26T06:38:34.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.flush_verify_memtable_count: 1 2024-07-26T06:38:34.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-07-26T06:38:34.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-07-26T06:38:34.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.env: 0x55a6cf771280 2024-07-26T06:38:34.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.fs: PosixFileSystem 2024-07-26T06:38:34.058 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.info_log: 0x55a6d4fd7080 2024-07-26T06:38:34.058 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_file_opening_threads: 16 2024-07-26T06:38:34.058 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.statistics: (nil) 2024-07-26T06:38:34.058 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.use_fsync: 0 2024-07-26T06:38:34.058 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_log_file_size: 0 2024-07-26T06:38:34.058 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-07-26T06:38:34.058 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.log_file_time_to_roll: 0 2024-07-26T06:38:34.058 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.keep_log_file_num: 1000 2024-07-26T06:38:34.058 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.recycle_log_file_num: 0 2024-07-26T06:38:34.059 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.allow_fallocate: 1 2024-07-26T06:38:34.059 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.allow_mmap_reads: 0 2024-07-26T06:38:34.059 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.allow_mmap_writes: 0 2024-07-26T06:38:34.059 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.use_direct_reads: 0 2024-07-26T06:38:34.059 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-07-26T06:38:34.059 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.create_missing_column_families: 0 2024-07-26T06:38:34.059 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.db_log_dir: 2024-07-26T06:38:34.059 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.wal_dir: 2024-07-26T06:38:34.059 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.table_cache_numshardbits: 6 2024-07-26T06:38:34.059 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.WAL_ttl_seconds: 0 2024-07-26T06:38:34.059 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.WAL_size_limit_MB: 0 2024-07-26T06:38:34.059 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-07-26T06:38:34.060 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-07-26T06:38:34.060 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.is_fd_close_on_exec: 1 2024-07-26T06:38:34.060 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.advise_random_on_open: 1 2024-07-26T06:38:34.060 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.db_write_buffer_size: 0 2024-07-26T06:38:34.060 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.write_buffer_manager: 0x55a6d4fdb5e0 2024-07-26T06:38:34.060 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-07-26T06:38:34.060 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-07-26T06:38:34.060 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.use_adaptive_mutex: 0 2024-07-26T06:38:34.060 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.rate_limiter: (nil) 2024-07-26T06:38:34.060 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-07-26T06:38:34.060 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.wal_recovery_mode: 2 2024-07-26T06:38:34.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.enable_thread_tracking: 0 2024-07-26T06:38:34.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.enable_pipelined_write: 0 2024-07-26T06:38:34.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.unordered_write: 0 2024-07-26T06:38:34.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-07-26T06:38:34.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-07-26T06:38:34.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-07-26T06:38:34.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-07-26T06:38:34.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.row_cache: None 2024-07-26T06:38:34.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.wal_filter: None 2024-07-26T06:38:34.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-07-26T06:38:34.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.allow_ingest_behind: 0 2024-07-26T06:38:34.061 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.two_write_queues: 0 2024-07-26T06:38:34.062 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.manual_wal_flush: 0 2024-07-26T06:38:34.062 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.wal_compression: 0 2024-07-26T06:38:34.062 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.atomic_flush: 0 2024-07-26T06:38:34.062 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-07-26T06:38:34.062 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.persist_stats_to_disk: 0 2024-07-26T06:38:34.062 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.write_dbid_to_manifest: 0 2024-07-26T06:38:34.062 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.log_readahead_size: 0 2024-07-26T06:38:34.062 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-07-26T06:38:34.062 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.best_efforts_recovery: 0 2024-07-26T06:38:34.062 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-07-26T06:38:34.062 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-07-26T06:38:34.062 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.allow_data_in_errors: 0 2024-07-26T06:38:34.062 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.db_host_id: __hostname__ 2024-07-26T06:38:34.063 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.enforce_single_del_contracts: true 2024-07-26T06:38:34.063 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_background_jobs: 2 2024-07-26T06:38:34.063 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_background_compactions: -1 2024-07-26T06:38:34.063 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_subcompactions: 1 2024-07-26T06:38:34.063 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-07-26T06:38:34.063 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-07-26T06:38:34.063 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.delayed_write_rate : 16777216 2024-07-26T06:38:34.063 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_total_wal_size: 0 2024-07-26T06:38:34.063 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-07-26T06:38:34.063 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.stats_dump_period_sec: 600 2024-07-26T06:38:34.063 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.stats_persist_period_sec: 600 2024-07-26T06:38:34.063 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-07-26T06:38:34.063 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_open_files: -1 2024-07-26T06:38:34.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.bytes_per_sync: 0 2024-07-26T06:38:34.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.wal_bytes_per_sync: 0 2024-07-26T06:38:34.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.strict_bytes_per_sync: 0 2024-07-26T06:38:34.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compaction_readahead_size: 0 2024-07-26T06:38:34.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_background_flushes: -1 2024-07-26T06:38:34.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Compression algorithms supported: 2024-07-26T06:38:34.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: kZSTD supported: 0 2024-07-26T06:38:34.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: kXpressCompression supported: 0 2024-07-26T06:38:34.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: kBZip2Compression supported: 0 2024-07-26T06:38:34.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-07-26T06:38:34.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: kLZ4Compression supported: 1 2024-07-26T06:38:34.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: kZlibCompression supported: 1 2024-07-26T06:38:34.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: kLZ4HCCompression supported: 1 2024-07-26T06:38:34.065 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: kSnappyCompression supported: 1 2024-07-26T06:38:34.065 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Fast CRC32 supported: Supported on x86 2024-07-26T06:38:34.065 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: DMutex implementation: pthread_mutex_t 2024-07-26T06:38:34.065 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-07-26T06:38:34.065 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-07-26T06:38:34.065 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-07-26T06:38:34.065 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.merge_operator: 2024-07-26T06:38:34.065 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compaction_filter: None 2024-07-26T06:38:34.065 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compaction_filter_factory: None 2024-07-26T06:38:34.065 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.sst_partitioner_factory: None 2024-07-26T06:38:34.065 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.memtable_factory: SkipListFactory 2024-07-26T06:38:34.065 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.table_factory: BlockBasedTable 2024-07-26T06:38:34.065 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a6d4fd6c80) 2024-07-26T06:38:34.066 INFO:journalctl@ceph.mon.a.smithi042.stdout: cache_index_and_filter_blocks: 1 2024-07-26T06:38:34.066 INFO:journalctl@ceph.mon.a.smithi042.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-07-26T06:38:34.066 INFO:journalctl@ceph.mon.a.smithi042.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-07-26T06:38:34.066 INFO:journalctl@ceph.mon.a.smithi042.stdout: pin_top_level_index_and_filter: 1 2024-07-26T06:38:34.066 INFO:journalctl@ceph.mon.a.smithi042.stdout: index_type: 0 2024-07-26T06:38:34.066 INFO:journalctl@ceph.mon.a.smithi042.stdout: data_block_index_type: 0 2024-07-26T06:38:34.066 INFO:journalctl@ceph.mon.a.smithi042.stdout: index_shortening: 1 2024-07-26T06:38:34.066 INFO:journalctl@ceph.mon.a.smithi042.stdout: data_block_hash_table_util_ratio: 0.750000 2024-07-26T06:38:34.066 INFO:journalctl@ceph.mon.a.smithi042.stdout: checksum: 4 2024-07-26T06:38:34.066 INFO:journalctl@ceph.mon.a.smithi042.stdout: no_block_cache: 0 2024-07-26T06:38:34.066 INFO:journalctl@ceph.mon.a.smithi042.stdout: block_cache: 0x55a6d0f9b610 2024-07-26T06:38:34.066 INFO:journalctl@ceph.mon.a.smithi042.stdout: block_cache_name: BinnedLRUCache 2024-07-26T06:38:34.067 INFO:journalctl@ceph.mon.a.smithi042.stdout: block_cache_options: 2024-07-26T06:38:34.067 INFO:journalctl@ceph.mon.a.smithi042.stdout: capacity : 536870912 2024-07-26T06:38:34.067 INFO:journalctl@ceph.mon.a.smithi042.stdout: num_shard_bits : 4 2024-07-26T06:38:34.067 INFO:journalctl@ceph.mon.a.smithi042.stdout: strict_capacity_limit : 0 2024-07-26T06:38:34.067 INFO:journalctl@ceph.mon.a.smithi042.stdout: high_pri_pool_ratio: 0.000 2024-07-26T06:38:34.067 INFO:journalctl@ceph.mon.a.smithi042.stdout: block_cache_compressed: (nil) 2024-07-26T06:38:34.067 INFO:journalctl@ceph.mon.a.smithi042.stdout: persistent_cache: (nil) 2024-07-26T06:38:34.067 INFO:journalctl@ceph.mon.a.smithi042.stdout: block_size: 4096 2024-07-26T06:38:34.067 INFO:journalctl@ceph.mon.a.smithi042.stdout: block_size_deviation: 10 2024-07-26T06:38:34.067 INFO:journalctl@ceph.mon.a.smithi042.stdout: block_restart_interval: 16 2024-07-26T06:38:34.067 INFO:journalctl@ceph.mon.a.smithi042.stdout: index_block_restart_interval: 1 2024-07-26T06:38:34.067 INFO:journalctl@ceph.mon.a.smithi042.stdout: metadata_block_size: 4096 2024-07-26T06:38:34.068 INFO:journalctl@ceph.mon.a.smithi042.stdout: partition_filters: 0 2024-07-26T06:38:34.068 INFO:journalctl@ceph.mon.a.smithi042.stdout: use_delta_encoding: 1 2024-07-26T06:38:34.068 INFO:journalctl@ceph.mon.a.smithi042.stdout: filter_policy: bloomfilter 2024-07-26T06:38:34.068 INFO:journalctl@ceph.mon.a.smithi042.stdout: whole_key_filtering: 1 2024-07-26T06:38:34.068 INFO:journalctl@ceph.mon.a.smithi042.stdout: verify_compression: 0 2024-07-26T06:38:34.068 INFO:journalctl@ceph.mon.a.smithi042.stdout: read_amp_bytes_per_bit: 0 2024-07-26T06:38:34.069 INFO:journalctl@ceph.mon.a.smithi042.stdout: format_version: 5 2024-07-26T06:38:34.069 INFO:journalctl@ceph.mon.a.smithi042.stdout: enable_index_compression: 1 2024-07-26T06:38:34.069 INFO:journalctl@ceph.mon.a.smithi042.stdout: block_align: 0 2024-07-26T06:38:34.069 INFO:journalctl@ceph.mon.a.smithi042.stdout: max_auto_readahead_size: 262144 2024-07-26T06:38:34.069 INFO:journalctl@ceph.mon.a.smithi042.stdout: prepopulate_block_cache: 0 2024-07-26T06:38:34.069 INFO:journalctl@ceph.mon.a.smithi042.stdout: initial_auto_readahead_size: 8192 2024-07-26T06:38:34.069 INFO:journalctl@ceph.mon.a.smithi042.stdout: num_file_reads_for_auto_readahead: 2 2024-07-26T06:38:34.069 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.write_buffer_size: 33554432 2024-07-26T06:38:34.069 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_write_buffer_number: 2 2024-07-26T06:38:34.070 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compression: NoCompression 2024-07-26T06:38:34.070 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.bottommost_compression: Disabled 2024-07-26T06:38:34.070 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.prefix_extractor: nullptr 2024-07-26T06:38:34.070 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-07-26T06:38:34.070 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.num_levels: 7 2024-07-26T06:38:34.070 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-07-26T06:38:34.070 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-07-26T06:38:34.070 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-07-26T06:38:34.070 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-07-26T06:38:34.070 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-07-26T06:38:34.070 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-07-26T06:38:34.070 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-07-26T06:38:34.071 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-07-26T06:38:34.071 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-07-26T06:38:34.071 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-07-26T06:38:34.071 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-07-26T06:38:34.071 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-07-26T06:38:34.071 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compression_opts.window_bits: -14 2024-07-26T06:38:34.071 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compression_opts.level: 32767 2024-07-26T06:38:34.071 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compression_opts.strategy: 0 2024-07-26T06:38:34.071 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-07-26T06:38:34.071 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-07-26T06:38:34.071 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-07-26T06:38:34.071 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-07-26T06:38:34.072 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compression_opts.enabled: false 2024-07-26T06:38:34.072 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-07-26T06:38:34.072 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-07-26T06:38:34.073 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-07-26T06:38:34.073 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-07-26T06:38:34.073 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.target_file_size_base: 67108864 2024-07-26T06:38:34.073 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.target_file_size_multiplier: 1 2024-07-26T06:38:34.073 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-07-26T06:38:34.073 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-07-26T06:38:34.073 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-07-26T06:38:34.073 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-07-26T06:38:34.073 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-07-26T06:38:34.073 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-07-26T06:38:34.073 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-07-26T06:38:34.074 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-07-26T06:38:34.074 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-07-26T06:38:34.074 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-07-26T06:38:34.074 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-07-26T06:38:34.074 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-07-26T06:38:34.074 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-07-26T06:38:34.074 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.arena_block_size: 1048576 2024-07-26T06:38:34.074 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-07-26T06:38:34.074 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-07-26T06:38:34.074 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.disable_auto_compactions: 0 2024-07-26T06:38:34.074 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-07-26T06:38:34.075 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-07-26T06:38:34.075 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-07-26T06:38:34.075 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-07-26T06:38:34.075 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-07-26T06:38:34.075 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-07-26T06:38:34.075 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-07-26T06:38:34.075 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-07-26T06:38:34.075 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-07-26T06:38:34.075 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-07-26T06:38:34.075 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-07-26T06:38:34.075 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.inplace_update_support: 0 2024-07-26T06:38:34.076 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.inplace_update_num_locks: 10000 2024-07-26T06:38:34.076 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-07-26T06:38:34.076 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-07-26T06:38:34.076 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.memtable_huge_page_size: 0 2024-07-26T06:38:34.076 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.bloom_locality: 0 2024-07-26T06:38:34.076 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.max_successive_merges: 0 2024-07-26T06:38:34.076 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.optimize_filters_for_hits: 0 2024-07-26T06:38:34.076 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.paranoid_file_checks: 0 2024-07-26T06:38:34.076 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.force_consistency_checks: 1 2024-07-26T06:38:34.076 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.report_bg_io_stats: 0 2024-07-26T06:38:34.076 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.ttl: 2592000 2024-07-26T06:38:34.076 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.periodic_compaction_seconds: 0 2024-07-26T06:38:34.076 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-07-26T06:38:34.076 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-07-26T06:38:34.077 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.enable_blob_files: false 2024-07-26T06:38:34.077 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.min_blob_size: 0 2024-07-26T06:38:34.077 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.blob_file_size: 268435456 2024-07-26T06:38:34.077 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.blob_compression_type: NoCompression 2024-07-26T06:38:34.077 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.enable_blob_garbage_collection: false 2024-07-26T06:38:34.077 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-07-26T06:38:34.077 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-07-26T06:38:34.077 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-07-26T06:38:34.077 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.blob_file_starting_level: 0 2024-07-26T06:38:34.077 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-07-26T06:38:34.077 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-07-26T06:38:34.077 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-07-26T06:38:34.077 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d0dab53f-1713-4db4-b154-ebd6022d1dc6 2024-07-26T06:38:34.077 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721975914019353, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-07-26T06:38:34.078 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-07-26T06:38:34.078 INFO:teuthology.orchestra.run.smithi042.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-07-26T06:38:34.078 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721975914021012, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 90651, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 280, "table_properties": {"data_size": 88692, "index_size": 279, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11486, "raw_average_key_size": 48, "raw_value_size": 82217, "raw_average_value_size": 344, "num_data_blocks": 12, "num_entries": 239, "num_filter_entries": 239, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1721975914, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d0dab53f-1713-4db4-b154-ebd6022d1dc6", "db_session_id": "8M0QPN7OPPDRCAHQBDSY", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-07-26T06:38:34.078 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721975914021148, "job": 1, "event": "recovery_finished"} 2024-07-26T06:38:34.079 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-07-26T06:38:34.079 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-07-26T06:38:34.079 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a6d4ff8a80 2024-07-26T06:38:34.079 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: DB pointer 0x55a6d5d98000 2024-07-26T06:38:34.079 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-07-26T06:38:34.079 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-07-26T06:38:34.079 INFO:journalctl@ceph.mon.a.smithi042.stdout: ** DB Stats ** 2024-07-26T06:38:34.079 INFO:journalctl@ceph.mon.a.smithi042.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T06:38:34.079 INFO:journalctl@ceph.mon.a.smithi042.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-07-26T06:38:34.079 INFO:journalctl@ceph.mon.a.smithi042.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T06:38:34.079 INFO:journalctl@ceph.mon.a.smithi042.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T06:38:34.079 INFO:journalctl@ceph.mon.a.smithi042.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-07-26T06:38:34.079 INFO:journalctl@ceph.mon.a.smithi042.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T06:38:34.080 INFO:journalctl@ceph.mon.a.smithi042.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T06:38:34.080 INFO:journalctl@ceph.mon.a.smithi042.stdout: 2024-07-26T06:38:34.080 INFO:journalctl@ceph.mon.a.smithi042.stdout: ** Compaction Stats [default] ** 2024-07-26T06:38:34.080 INFO:journalctl@ceph.mon.a.smithi042.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T06:38:34.080 INFO:journalctl@ceph.mon.a.smithi042.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-07-26T06:38:34.080 INFO:journalctl@ceph.mon.a.smithi042.stdout: L0 2/0 90.39 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 96.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T06:38:34.080 INFO:journalctl@ceph.mon.a.smithi042.stdout: Sum 2/0 90.39 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 96.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T06:38:34.080 INFO:journalctl@ceph.mon.a.smithi042.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 96.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T06:38:34.080 INFO:journalctl@ceph.mon.a.smithi042.stdout: 2024-07-26T06:38:34.080 INFO:journalctl@ceph.mon.a.smithi042.stdout: ** Compaction Stats [default] ** 2024-07-26T06:38:34.080 INFO:journalctl@ceph.mon.a.smithi042.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T06:38:34.080 INFO:journalctl@ceph.mon.a.smithi042.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-07-26T06:38:34.080 INFO:journalctl@ceph.mon.a.smithi042.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 96.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T06:38:34.081 INFO:journalctl@ceph.mon.a.smithi042.stdout: 2024-07-26T06:38:34.081 INFO:journalctl@ceph.mon.a.smithi042.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-07-26T06:38:34.081 INFO:journalctl@ceph.mon.a.smithi042.stdout: 2024-07-26T06:38:34.081 INFO:journalctl@ceph.mon.a.smithi042.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T06:38:34.081 INFO:journalctl@ceph.mon.a.smithi042.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-07-26T06:38:34.081 INFO:journalctl@ceph.mon.a.smithi042.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-07-26T06:38:34.081 INFO:journalctl@ceph.mon.a.smithi042.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-07-26T06:38:34.081 INFO:journalctl@ceph.mon.a.smithi042.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-07-26T06:38:34.081 INFO:journalctl@ceph.mon.a.smithi042.stdout: AddFile(Keys): cumulative 0, interval 0 2024-07-26T06:38:34.081 INFO:journalctl@ceph.mon.a.smithi042.stdout: Cumulative compaction: 0.00 GB write, 17.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T06:38:34.081 INFO:journalctl@ceph.mon.a.smithi042.stdout: Interval compaction: 0.00 GB write, 17.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T06:38:34.081 INFO:journalctl@ceph.mon.a.smithi042.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-07-26T06:38:34.081 INFO:journalctl@ceph.mon.a.smithi042.stdout: Block cache BinnedLRUCache@0x55a6d0f9b610#2 capacity: 512.00 MB usage: 1.31 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-07-26T06:38:34.082 INFO:journalctl@ceph.mon.a.smithi042.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.84 KB,0.000160933%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-07-26T06:38:34.082 INFO:journalctl@ceph.mon.a.smithi042.stdout: 2024-07-26T06:38:34.082 INFO:journalctl@ceph.mon.a.smithi042.stdout: ** File Read Latency Histogram By Level [default] ** 2024-07-26T06:38:34.082 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: starting mon.a rank 0 at public addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] at bind addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:38:34.082 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: mon.a@-1(???) e1 preinit fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:38:34.082 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: mon.a@-1(???).mds e1 new map 2024-07-26T06:38:34.082 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: mon.a@-1(???).mds e1 print_map 2024-07-26T06:38:34.082 INFO:journalctl@ceph.mon.a.smithi042.stdout: e1 2024-07-26T06:38:34.082 INFO:journalctl@ceph.mon.a.smithi042.stdout: btime 2024-07-26T06:38:29:306299+0000 2024-07-26T06:38:34.082 INFO:journalctl@ceph.mon.a.smithi042.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T06:38:34.082 INFO:journalctl@ceph.mon.a.smithi042.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T06:38:34.082 INFO:journalctl@ceph.mon.a.smithi042.stdout: legacy client fscid: -1 2024-07-26T06:38:34.082 INFO:journalctl@ceph.mon.a.smithi042.stdout: 2024-07-26T06:38:34.083 INFO:journalctl@ceph.mon.a.smithi042.stdout: No filesystems configured 2024-07-26T06:38:34.083 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-07-26T06:38:34.083 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T06:38:34.083 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T06:38:34.083 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T06:38:34.083 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-07-26T06:38:34.083 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 bash[32756]: 33bc7f46d883f96e8551cf60ff295bd2e84044b6535668aa93363e0d4f9c1841 2024-07-26T06:38:34.083 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 systemd[1]: Started Ceph mon.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:38:34.356 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T06:38:34.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: monmap epoch 1 2024-07-26T06:38:34.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:38:34.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: last_changed 2024-07-26T06:38:26.468434+0000 2024-07-26T06:38:34.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: created 2024-07-26T06:38:26.468434+0000 2024-07-26T06:38:34.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: min_mon_release 19 (squid) 2024-07-26T06:38:34.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: election_strategy: 1 2024-07-26T06:38:34.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: 0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-07-26T06:38:34.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: fsmap 2024-07-26T06:38:34.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: osdmap e1: 0 total, 0 up, 0 in 2024-07-26T06:38:34.357 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:34 smithi042 ceph-mon[32775]: mgrmap e1: no daemons active 2024-07-26T06:38:35.021 INFO:teuthology.orchestra.run.smithi042.stdout:Wrote config to /etc/ceph/ceph.conf 2024-07-26T06:38:35.043 INFO:teuthology.orchestra.run.smithi042.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-07-26T06:38:35.043 INFO:teuthology.orchestra.run.smithi042.stdout:Creating mgr... 2024-07-26T06:38:35.043 INFO:teuthology.orchestra.run.smithi042.stdout:Verifying port 0.0.0.0:9283 ... 2024-07-26T06:38:35.044 INFO:teuthology.orchestra.run.smithi042.stdout:Verifying port 0.0.0.0:8765 ... 2024-07-26T06:38:35.306 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mgr.x 2024-07-26T06:38:35.306 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mgr.x.service: Unit ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mgr.x.service not loaded. 2024-07-26T06:38:35.464 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968.target.wants/ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@.service. 2024-07-26T06:38:35.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:35 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3811588277' entity='client.admin' 2024-07-26T06:38:35.990 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:35 smithi042 podman[32988]: 2024-07-26 06:38:35.909223561 +0000 UTC m=+0.321935176 container start 3be6bbb38ec527be8d88fa6aac770f503b06ef652d1f20e80fcff1b0e2dd8c11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-07-26T06:38:35.990 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:35 smithi042 bash[32988]: 3be6bbb38ec527be8d88fa6aac770f503b06ef652d1f20e80fcff1b0e2dd8c11 2024-07-26T06:38:36.004 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T06:38:36.004 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout disabled 2024-07-26T06:38:36.011 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T06:38:36.011 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout inactive 2024-07-26T06:38:36.011 INFO:teuthology.orchestra.run.smithi042.stdout:firewalld.service is not enabled 2024-07-26T06:38:36.011 INFO:teuthology.orchestra.run.smithi042.stdout:Not possible to enable service . firewalld.service is not available 2024-07-26T06:38:36.020 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T06:38:36.021 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout disabled 2024-07-26T06:38:36.029 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T06:38:36.029 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout inactive 2024-07-26T06:38:36.029 INFO:teuthology.orchestra.run.smithi042.stdout:firewalld.service is not enabled 2024-07-26T06:38:36.029 INFO:teuthology.orchestra.run.smithi042.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-07-26T06:38:36.029 INFO:teuthology.orchestra.run.smithi042.stdout:Waiting for mgr to start... 2024-07-26T06:38:36.029 INFO:teuthology.orchestra.run.smithi042.stdout:Waiting for mgr... 2024-07-26T06:38:36.239 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:35 smithi042 systemd[1]: Started Ceph mgr.x for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:38:36.804 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:36 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2424092053' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-26T06:38:37.065 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-07-26T06:38:37.071 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout { 2024-07-26T06:38:37.071 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "fsid": "8b3e2750-4b19-11ef-bcb6-c7b262605968", 2024-07-26T06:38:37.071 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "health": { 2024-07-26T06:38:37.071 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-26T06:38:37.071 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-26T06:38:37.071 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-26T06:38:37.071 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-07-26T06:38:37.071 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-26T06:38:37.071 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-26T06:38:37.072 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 0 2024-07-26T06:38:37.072 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout ], 2024-07-26T06:38:37.072 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-26T06:38:37.072 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "a" 2024-07-26T06:38:37.072 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout ], 2024-07-26T06:38:37.072 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-07-26T06:38:37.072 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-26T06:38:37.072 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T06:38:37.072 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-26T06:38:37.072 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-26T06:38:37.072 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-07-26T06:38:37.072 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-26T06:38:37.072 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T06:38:37.073 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-26T06:38:37.073 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-26T06:38:37.073 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-26T06:38:37.073 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-26T06:38:37.073 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-26T06:38:37.073 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-26T06:38:37.073 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-07-26T06:38:37.073 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-26T06:38:37.073 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-26T06:38:37.073 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-26T06:38:37.073 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-26T06:38:37.073 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-26T06:38:37.074 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-26T06:38:37.074 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-26T06:38:37.074 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-26T06:38:37.074 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-26T06:38:37.074 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-07-26T06:38:37.074 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-26T06:38:37.074 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T06:38:37.074 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "btime": "2024-07-26T06:38:29:306299+0000", 2024-07-26T06:38:37.074 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-26T06:38:37.074 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-26T06:38:37.074 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-07-26T06:38:37.074 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-26T06:38:37.074 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "available": false, 2024-07-26T06:38:37.075 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-26T06:38:37.075 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-26T06:38:37.075 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-26T06:38:37.075 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-26T06:38:37.075 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "restful" 2024-07-26T06:38:37.075 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout ], 2024-07-26T06:38:37.075 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T06:38:37.075 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-07-26T06:38:37.075 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-26T06:38:37.075 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T06:38:37.075 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "modified": "2024-07-26T06:38:29.307093+0000", 2024-07-26T06:38:37.075 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T06:38:37.076 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-07-26T06:38:37.076 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-26T06:38:37.076 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout } 2024-07-26T06:38:37.076 INFO:teuthology.orchestra.run.smithi042.stdout:mgr not available, waiting (1/15)... 2024-07-26T06:38:41.465 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:41 smithi042 ceph-mon[32775]: Activating manager daemon x 2024-07-26T06:38:41.465 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:41 smithi042 ceph-mon[32775]: mgrmap e2: x(active, starting, since 0.00393406s) 2024-07-26T06:38:41.465 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:41 smithi042 ceph-mon[32775]: from='mgr.14100 172.21.15.42:0/1919864858' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T06:38:41.465 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:41 smithi042 ceph-mon[32775]: from='mgr.14100 172.21.15.42:0/1919864858' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T06:38:41.465 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:41 smithi042 ceph-mon[32775]: from='mgr.14100 172.21.15.42:0/1919864858' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T06:38:41.465 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:41 smithi042 ceph-mon[32775]: from='mgr.14100 172.21.15.42:0/1919864858' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:38:41.465 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:41 smithi042 ceph-mon[32775]: from='mgr.14100 172.21.15.42:0/1919864858' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T06:38:41.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:41 smithi042 ceph-mon[32775]: Manager daemon x is now available 2024-07-26T06:38:41.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:41 smithi042 ceph-mon[32775]: from='mgr.14100 172.21.15.42:0/1919864858' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T06:38:41.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:41 smithi042 ceph-mon[32775]: from='mgr.14100 172.21.15.42:0/1919864858' entity='mgr.x' 2024-07-26T06:38:41.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:41 smithi042 ceph-mon[32775]: from='mgr.14100 172.21.15.42:0/1919864858' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T06:38:41.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:41 smithi042 ceph-mon[32775]: from='mgr.14100 172.21.15.42:0/1919864858' entity='mgr.x' 2024-07-26T06:38:41.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:41 smithi042 ceph-mon[32775]: from='mgr.14100 172.21.15.42:0/1919864858' entity='mgr.x' 2024-07-26T06:38:43.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:43 smithi042 ceph-mon[32775]: mgrmap e3: x(active, since 1.00875s) 2024-07-26T06:38:43.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:43 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1549841643' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-26T06:38:44.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:44 smithi042 ceph-mon[32775]: mgrmap e4: x(active, since 2s) 2024-07-26T06:38:49.256 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-07-26T06:38:49.257 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout { 2024-07-26T06:38:49.257 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "fsid": "8b3e2750-4b19-11ef-bcb6-c7b262605968", 2024-07-26T06:38:49.257 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "health": { 2024-07-26T06:38:49.257 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-26T06:38:49.257 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-26T06:38:49.257 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-26T06:38:49.257 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-07-26T06:38:49.257 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-26T06:38:49.257 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-26T06:38:49.258 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 0 2024-07-26T06:38:49.258 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout ], 2024-07-26T06:38:49.258 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-26T06:38:49.258 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "a" 2024-07-26T06:38:49.258 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout ], 2024-07-26T06:38:49.258 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-07-26T06:38:49.258 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-26T06:38:49.258 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T06:38:49.258 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-26T06:38:49.258 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-26T06:38:49.258 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-07-26T06:38:49.258 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-26T06:38:49.259 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T06:38:49.259 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-26T06:38:49.259 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-26T06:38:49.259 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-26T06:38:49.259 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-26T06:38:49.259 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-26T06:38:49.259 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-26T06:38:49.262 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-07-26T06:38:49.263 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-26T06:38:49.263 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-26T06:38:49.263 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-26T06:38:49.263 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-26T06:38:49.263 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-26T06:38:49.263 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-26T06:38:49.263 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-26T06:38:49.263 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-26T06:38:49.263 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-26T06:38:49.263 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-07-26T06:38:49.263 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-26T06:38:49.263 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T06:38:49.263 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "btime": "2024-07-26T06:38:29:306299+0000", 2024-07-26T06:38:49.264 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-26T06:38:49.264 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-26T06:38:49.264 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-07-26T06:38:49.264 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-26T06:38:49.264 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T06:38:49.264 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-26T06:38:49.264 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-26T06:38:49.264 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-26T06:38:49.264 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-26T06:38:49.264 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "restful" 2024-07-26T06:38:49.264 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout ], 2024-07-26T06:38:49.265 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T06:38:49.265 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-07-26T06:38:49.265 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-26T06:38:49.265 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T06:38:49.265 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "modified": "2024-07-26T06:38:29.307093+0000", 2024-07-26T06:38:49.265 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T06:38:49.265 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout }, 2024-07-26T06:38:49.265 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-26T06:38:49.266 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout } 2024-07-26T06:38:49.266 INFO:teuthology.orchestra.run.smithi042.stdout:mgr is available 2024-07-26T06:38:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:50 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/4252180603' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-07-26T06:38:50.860 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-07-26T06:38:50.904 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout [global] 2024-07-26T06:38:50.904 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout fsid = 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:38:50.904 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-07-26T06:38:50.904 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.42:3300,v1:172.21.15.42:6789] 2024-07-26T06:38:50.904 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-07-26T06:38:50.905 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-07-26T06:38:50.905 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-07-26T06:38:50.905 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-07-26T06:38:50.905 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-07-26T06:38:50.905 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout [mgr] 2024-07-26T06:38:50.905 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-07-26T06:38:50.905 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 2024-07-26T06:38:50.905 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout [osd] 2024-07-26T06:38:50.905 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-07-26T06:38:50.905 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-07-26T06:38:50.905 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-07-26T06:38:50.905 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-07-26T06:38:50.906 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-07-26T06:38:50.906 INFO:teuthology.orchestra.run.smithi042.stdout:Enabling cephadm module... 2024-07-26T06:38:51.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:51 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2998698852' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-07-26T06:38:51.895 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:51 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: ignoring --setuser ceph since I am not root 2024-07-26T06:38:51.895 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:51 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: ignoring --setgroup ceph since I am not root 2024-07-26T06:38:52.852 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:52 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2998698852' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-07-26T06:38:52.852 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:52 smithi042 ceph-mon[32775]: mgrmap e5: x(active, since 10s) 2024-07-26T06:38:53.357 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout { 2024-07-26T06:38:53.357 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-07-26T06:38:53.357 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T06:38:53.357 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-07-26T06:38:53.357 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-07-26T06:38:53.357 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout } 2024-07-26T06:38:53.357 INFO:teuthology.orchestra.run.smithi042.stdout:Waiting for the mgr to restart... 2024-07-26T06:38:53.357 INFO:teuthology.orchestra.run.smithi042.stdout:Waiting for mgr epoch 5... 2024-07-26T06:38:53.765 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:53 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/187060421' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-07-26T06:38:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:56 smithi042 ceph-mon[32775]: Active manager daemon x restarted 2024-07-26T06:38:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:56 smithi042 ceph-mon[32775]: Activating manager daemon x 2024-07-26T06:38:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:56 smithi042 ceph-mon[32775]: osdmap e2: 0 total, 0 up, 0 in 2024-07-26T06:38:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:56 smithi042 ceph-mon[32775]: mgrmap e6: x(active, starting, since 0.00421686s) 2024-07-26T06:38:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:56 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:38:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:56 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T06:38:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:56 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T06:38:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:56 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T06:38:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:56 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T06:38:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:56 smithi042 ceph-mon[32775]: Manager daemon x is now available 2024-07-26T06:38:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:56 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' 2024-07-26T06:38:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:56 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' 2024-07-26T06:38:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:56 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:38:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:56 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:38:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:56 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T06:38:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:56 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T06:38:58.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:57 smithi042 ceph-mon[32775]: Found migration_current of "None". Setting to last migration. 2024-07-26T06:38:58.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:57 smithi042 ceph-mon[32775]: mgrmap e7: x(active, since 1.00664s) 2024-07-26T06:38:58.426 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout { 2024-07-26T06:38:58.426 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-07-26T06:38:58.426 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "initialized": true 2024-07-26T06:38:58.426 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout } 2024-07-26T06:38:58.426 INFO:teuthology.orchestra.run.smithi042.stdout:mgr epoch 5 is available 2024-07-26T06:38:58.426 INFO:teuthology.orchestra.run.smithi042.stdout:Setting orchestrator backend to cephadm... 2024-07-26T06:38:59.068 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:58 smithi042 ceph-mon[32775]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-07-26T06:38:59.068 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:58 smithi042 ceph-mon[32775]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-07-26T06:38:59.068 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:58 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' 2024-07-26T06:38:59.068 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:58 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' 2024-07-26T06:38:59.639 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: 2024-07-26T06:38:59.455+0000 7fb69ad45640 -1 log_channel(cephadm) log [ERR] : [26/Jul/2024:06:38:59] ENGINE Error in HTTPServer.serve 2024-07-26T06:38:59.639 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: Traceback (most recent call last): 2024-07-26T06:38:59.639 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T06:38:59.639 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: self._connections.run(self.expiration_interval) 2024-07-26T06:38:59.639 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T06:38:59.640 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: self._run(expiration_interval) 2024-07-26T06:38:59.640 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T06:38:59.640 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T06:38:59.640 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T06:38:59.640 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T06:38:59.641 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T06:38:59.641 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: s = self.context.wrap_socket( 2024-07-26T06:38:59.641 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T06:38:59.641 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: return self.sslsocket_class._create( 2024-07-26T06:38:59.641 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T06:38:59.641 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: self.do_handshake() 2024-07-26T06:38:59.642 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T06:38:59.642 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: self._sslobj.do_handshake() 2024-07-26T06:38:59.642 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T06:38:59.642 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: 2024-07-26T06:39:00.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-mon[32775]: mgrmap e8: x(active, since 2s) 2024-07-26T06:39:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-mon[32775]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:39:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' 2024-07-26T06:39:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:39:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-mon[32775]: [26/Jul/2024:06:38:59] ENGINE Bus STARTING 2024-07-26T06:39:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-mon[32775]: [26/Jul/2024:06:38:59] ENGINE Serving on http://172.21.15.42:8765 2024-07-26T06:39:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-mon[32775]: [26/Jul/2024:06:38:59] ENGINE Serving on https://172.21.15.42:7150 2024-07-26T06:39:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-mon[32775]: [26/Jul/2024:06:38:59] ENGINE Bus STARTED 2024-07-26T06:39:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:39:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:38:59 smithi042 ceph-mon[32775]: [26/Jul/2024:06:38:59] ENGINE Error in HTTPServer.serve 2024-07-26T06:39:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout: Traceback (most recent call last): 2024-07-26T06:39:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T06:39:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout: self._connections.run(self.expiration_interval) 2024-07-26T06:39:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T06:39:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout: self._run(expiration_interval) 2024-07-26T06:39:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T06:39:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T06:39:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T06:39:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T06:39:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T06:39:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout: s = self.context.wrap_socket( 2024-07-26T06:39:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T06:39:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout: return self.sslsocket_class._create( 2024-07-26T06:39:00.147 INFO:journalctl@ceph.mon.a.smithi042.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T06:39:00.147 INFO:journalctl@ceph.mon.a.smithi042.stdout: self.do_handshake() 2024-07-26T06:39:00.147 INFO:journalctl@ceph.mon.a.smithi042.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T06:39:00.147 INFO:journalctl@ceph.mon.a.smithi042.stdout: self._sslobj.do_handshake() 2024-07-26T06:39:00.147 INFO:journalctl@ceph.mon.a.smithi042.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T06:39:00.746 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout value unchanged 2024-07-26T06:39:00.746 INFO:teuthology.orchestra.run.smithi042.stdout:Generating ssh key... 2024-07-26T06:39:00.996 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:00 smithi042 ceph-mon[32775]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:39:01.866 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:01 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: Generating public/private rsa key pair. 2024-07-26T06:39:01.867 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:01 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: Your identification has been saved in /tmp/tmpqvudqdru/key 2024-07-26T06:39:01.867 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:01 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: Your public key has been saved in /tmp/tmpqvudqdru/key.pub 2024-07-26T06:39:01.867 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:01 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: The key fingerprint is: 2024-07-26T06:39:01.867 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:01 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: SHA256:6CyoONU9GV64syiO8aEckIS2SZBB2ooRjbsG5+euKWg ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:39:01.867 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:01 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: The key's randomart image is: 2024-07-26T06:39:01.867 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:01 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: +---[RSA 3072]----+ 2024-07-26T06:39:01.867 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:01 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: |=* | 2024-07-26T06:39:01.867 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:01 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: |*o. | 2024-07-26T06:39:01.867 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:01 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: |+=. . | 2024-07-26T06:39:01.867 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:01 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: |B++ o.. | 2024-07-26T06:39:01.868 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:01 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: |** . o.=S | 2024-07-26T06:39:01.868 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:01 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: |o.o.ooB | 2024-07-26T06:39:01.868 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:01 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: |+ooo..o+ | 2024-07-26T06:39:01.868 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:01 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: |*Eooo.. | 2024-07-26T06:39:01.868 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:01 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: |*==+. | 2024-07-26T06:39:01.868 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:01 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: +----[SHA256]-----+ 2024-07-26T06:39:02.758 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:02 smithi042 ceph-mon[32775]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:39:02.758 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:02 smithi042 ceph-mon[32775]: Generating ssh key... 2024-07-26T06:39:02.758 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:02 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' 2024-07-26T06:39:02.758 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:02 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' 2024-07-26T06:39:03.243 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCs3w2hE4YBJ7k0oCeNpjLrwc5pyJ1I/LeJDD7l7l6xBnLPFVRtg82WKMEIrmmYwRGuscINRZZjHAY1X/bCIW6OlSvkHHBFBBzJO/SSVL6dP6Wr12RfAtSjaCC7FYIw2OG147qUJayXPRBWfgBhkymTjRfwujKeyBhPDKTr0541+78QqB44FdztZmRpuXsCZUvMDvrcTKoosPALJficHqPN72i54ACFBS0q+NC3UptDY5ekHiN9twAKiusS7DARktmOHZVggA2lVEUIY9KD4xIxa7bRzO7kDjojdWduAEWQu81abY1MrgBnYUzYWRU+PguOkkxGMG36Z5i+L4EhwrcjcyyKpsuclFrYpcF+fqvrzCeCADqapc3R4hq6yCJaRIPI3Om5d+ILFhW8lV+TfbxKcVmlHT1VrAEaunLCjZ18dwo4CCedg0TKxglYNb5cRb/vU6J3VS4/GWSio5XO+eZwThIWAywslyvLjpLbnwdHNlbiERE5qcP7HjtXaLnMSec= ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:39:03.243 INFO:teuthology.orchestra.run.smithi042.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-07-26T06:39:03.243 INFO:teuthology.orchestra.run.smithi042.stdout:Adding key to root@localhost authorized_keys... 2024-07-26T06:39:03.244 INFO:teuthology.orchestra.run.smithi042.stdout:Adding host smithi042... 2024-07-26T06:39:03.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:03 smithi042 ceph-mon[32775]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:39:04.978 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:04 smithi042 ceph-mon[32775]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi042", "addr": "172.21.15.42", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:39:05.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:05 smithi042 ceph-mon[32775]: Deploying cephadm binary to smithi042 2024-07-26T06:39:08.451 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout Added host 'smithi042' with addr '172.21.15.42' 2024-07-26T06:39:08.451 INFO:teuthology.orchestra.run.smithi042.stdout:Deploying unmanaged mon service... 2024-07-26T06:39:09.078 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:08 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' 2024-07-26T06:39:09.079 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:08 smithi042 ceph-mon[32775]: Added host smithi042 2024-07-26T06:39:09.079 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:08 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:39:09.801 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-07-26T06:39:09.802 INFO:teuthology.orchestra.run.smithi042.stdout:Deploying unmanaged mgr service... 2024-07-26T06:39:10.236 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:10 smithi042 ceph-mon[32775]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:39:10.237 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:10 smithi042 ceph-mon[32775]: Saving service mon spec with placement count:5 2024-07-26T06:39:10.237 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:10 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' 2024-07-26T06:39:11.737 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:11 smithi042 ceph-mon[32775]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:39:11.737 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:11 smithi042 ceph-mon[32775]: Saving service mgr spec with placement count:2 2024-07-26T06:39:11.737 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:11 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' 2024-07-26T06:39:11.737 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:11 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' 2024-07-26T06:39:11.739 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-07-26T06:39:13.103 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:12 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' 2024-07-26T06:39:13.103 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:12 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2542011820' entity='client.admin' 2024-07-26T06:39:14.263 INFO:teuthology.orchestra.run.smithi042.stdout:Enabling the dashboard module... 2024-07-26T06:39:14.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:14 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1317885657' entity='client.admin' 2024-07-26T06:39:16.234 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:15 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' 2024-07-26T06:39:16.235 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:15 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1016849715' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-07-26T06:39:16.235 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:15 smithi042 ceph-mon[32775]: from='mgr.14116 172.21.15.42:0/299698448' entity='mgr.x' 2024-07-26T06:39:16.235 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:15 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: ignoring --setuser ceph since I am not root 2024-07-26T06:39:16.235 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:15 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: ignoring --setgroup ceph since I am not root 2024-07-26T06:39:17.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:16 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1016849715' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-07-26T06:39:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:16 smithi042 ceph-mon[32775]: mgrmap e9: x(active, since 19s) 2024-07-26T06:39:18.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:17 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1154772982' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-07-26T06:39:18.426 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout { 2024-07-26T06:39:18.426 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-07-26T06:39:18.426 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T06:39:18.426 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-07-26T06:39:18.426 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-07-26T06:39:18.426 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout } 2024-07-26T06:39:18.426 INFO:teuthology.orchestra.run.smithi042.stdout:Waiting for the mgr to restart... 2024-07-26T06:39:18.426 INFO:teuthology.orchestra.run.smithi042.stdout:Waiting for mgr epoch 9... 2024-07-26T06:39:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:21 smithi042 ceph-mon[32775]: Active manager daemon x restarted 2024-07-26T06:39:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:21 smithi042 ceph-mon[32775]: Activating manager daemon x 2024-07-26T06:39:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:21 smithi042 ceph-mon[32775]: osdmap e3: 0 total, 0 up, 0 in 2024-07-26T06:39:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:21 smithi042 ceph-mon[32775]: mgrmap e10: x(active, starting, since 0.00393855s) 2024-07-26T06:39:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:39:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T06:39:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T06:39:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T06:39:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T06:39:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:21 smithi042 ceph-mon[32775]: Manager daemon x is now available 2024-07-26T06:39:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:39:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T06:39:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T06:39:23.119 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout { 2024-07-26T06:39:23.119 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-07-26T06:39:23.119 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout "initialized": true 2024-07-26T06:39:23.119 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout } 2024-07-26T06:39:23.119 INFO:teuthology.orchestra.run.smithi042.stdout:mgr epoch 9 is available 2024-07-26T06:39:23.119 INFO:teuthology.orchestra.run.smithi042.stdout:Generating a dashboard self-signed certificate... 2024-07-26T06:39:23.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-mon[32775]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-07-26T06:39:23.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-mon[32775]: mgrmap e11: x(active, since 1.00658s) 2024-07-26T06:39:23.373 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-mon[32775]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-07-26T06:39:23.373 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:23.373 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: 2024-07-26T06:39:23.156+0000 7f813ba42640 -1 log_channel(cephadm) log [ERR] : [26/Jul/2024:06:39:23] ENGINE Error in HTTPServer.serve 2024-07-26T06:39:23.373 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: Traceback (most recent call last): 2024-07-26T06:39:23.373 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T06:39:23.373 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: self._connections.run(self.expiration_interval) 2024-07-26T06:39:23.373 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T06:39:23.373 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: self._run(expiration_interval) 2024-07-26T06:39:23.374 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T06:39:23.374 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T06:39:23.374 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T06:39:23.374 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T06:39:23.374 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T06:39:23.374 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: s = self.context.wrap_socket( 2024-07-26T06:39:23.374 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T06:39:23.374 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: return self.sslsocket_class._create( 2024-07-26T06:39:23.374 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T06:39:23.374 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: self.do_handshake() 2024-07-26T06:39:23.374 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T06:39:23.374 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: self._sslobj.do_handshake() 2024-07-26T06:39:23.375 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T06:39:23.375 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:39:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: 2024-07-26T06:39:24.417 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-07-26T06:39:24.417 INFO:teuthology.orchestra.run.smithi042.stdout:Creating initial admin user... 2024-07-26T06:39:24.800 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:24 smithi042 ceph-mon[32775]: [26/Jul/2024:06:39:23] ENGINE Bus STARTING 2024-07-26T06:39:24.801 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:24 smithi042 ceph-mon[32775]: [26/Jul/2024:06:39:23] ENGINE Serving on https://172.21.15.42:7150 2024-07-26T06:39:24.801 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:24 smithi042 ceph-mon[32775]: [26/Jul/2024:06:39:23] ENGINE Error in HTTPServer.serve 2024-07-26T06:39:24.801 INFO:journalctl@ceph.mon.a.smithi042.stdout: Traceback (most recent call last): 2024-07-26T06:39:24.801 INFO:journalctl@ceph.mon.a.smithi042.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T06:39:24.801 INFO:journalctl@ceph.mon.a.smithi042.stdout: self._connections.run(self.expiration_interval) 2024-07-26T06:39:24.801 INFO:journalctl@ceph.mon.a.smithi042.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T06:39:24.801 INFO:journalctl@ceph.mon.a.smithi042.stdout: self._run(expiration_interval) 2024-07-26T06:39:24.801 INFO:journalctl@ceph.mon.a.smithi042.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T06:39:24.801 INFO:journalctl@ceph.mon.a.smithi042.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T06:39:24.801 INFO:journalctl@ceph.mon.a.smithi042.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T06:39:24.801 INFO:journalctl@ceph.mon.a.smithi042.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T06:39:24.801 INFO:journalctl@ceph.mon.a.smithi042.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T06:39:24.802 INFO:journalctl@ceph.mon.a.smithi042.stdout: s = self.context.wrap_socket( 2024-07-26T06:39:24.802 INFO:journalctl@ceph.mon.a.smithi042.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T06:39:24.802 INFO:journalctl@ceph.mon.a.smithi042.stdout: return self.sslsocket_class._create( 2024-07-26T06:39:24.802 INFO:journalctl@ceph.mon.a.smithi042.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T06:39:24.802 INFO:journalctl@ceph.mon.a.smithi042.stdout: self.do_handshake() 2024-07-26T06:39:24.802 INFO:journalctl@ceph.mon.a.smithi042.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T06:39:24.802 INFO:journalctl@ceph.mon.a.smithi042.stdout: self._sslobj.do_handshake() 2024-07-26T06:39:24.802 INFO:journalctl@ceph.mon.a.smithi042.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T06:39:24.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:24 smithi042 ceph-mon[32775]: [26/Jul/2024:06:39:23] ENGINE Serving on http://172.21.15.42:8765 2024-07-26T06:39:24.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:24 smithi042 ceph-mon[32775]: [26/Jul/2024:06:39:23] ENGINE Bus STARTED 2024-07-26T06:39:24.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:24 smithi042 ceph-mon[32775]: mgrmap e12: x(active, since 2s) 2024-07-26T06:39:24.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:24 smithi042 ceph-mon[32775]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:39:24.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:24.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:24.803 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:26.413 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$kmkUOwfJL9UkqJKVsxDXaOYc6jlEx9pLJLeMYsSh6xBO9pMydo/I.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1721975965, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-07-26T06:39:26.413 INFO:teuthology.orchestra.run.smithi042.stdout:Fetching dashboard port number... 2024-07-26T06:39:26.535 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:26 smithi042 ceph-mon[32775]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:39:26.536 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:27.575 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:27 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2808539212' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-07-26T06:39:27.577 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stdout 8443 2024-07-26T06:39:27.588 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T06:39:27.588 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout disabled 2024-07-26T06:39:27.595 INFO:teuthology.orchestra.run.smithi042.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T06:39:27.595 INFO:teuthology.orchestra.run.smithi042.stdout:systemctl: stdout inactive 2024-07-26T06:39:27.596 INFO:teuthology.orchestra.run.smithi042.stdout:firewalld.service is not enabled 2024-07-26T06:39:27.596 INFO:teuthology.orchestra.run.smithi042.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-07-26T06:39:27.597 INFO:teuthology.orchestra.run.smithi042.stdout:Ceph Dashboard is now available at: 2024-07-26T06:39:27.598 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:39:27.598 INFO:teuthology.orchestra.run.smithi042.stdout: URL: https://smithi042.front.sepia.ceph.com:8443/ 2024-07-26T06:39:27.598 INFO:teuthology.orchestra.run.smithi042.stdout: User: admin 2024-07-26T06:39:27.598 INFO:teuthology.orchestra.run.smithi042.stdout: Password: 137lxldzug 2024-07-26T06:39:27.598 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:39:27.598 INFO:teuthology.orchestra.run.smithi042.stdout:Saving cluster configuration to /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config directory 2024-07-26T06:39:28.590 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:28 smithi042 ceph-mon[32775]: mgrmap e13: x(active, since 6s) 2024-07-26T06:39:28.590 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:28 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2625026897' entity='client.admin' 2024-07-26T06:39:28.852 INFO:teuthology.orchestra.run.smithi042.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-07-26T06:39:28.852 INFO:teuthology.orchestra.run.smithi042.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-07-26T06:39:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:39:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-07-26T06:39:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:39:28.852 INFO:teuthology.orchestra.run.smithi042.stdout:Or, if you are only running a single cluster on this host: 2024-07-26T06:39:28.852 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:39:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-07-26T06:39:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:39:28.853 INFO:teuthology.orchestra.run.smithi042.stdout:Please consider enabling telemetry to help improve Ceph: 2024-07-26T06:39:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:39:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: ceph telemetry on 2024-07-26T06:39:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:39:28.853 INFO:teuthology.orchestra.run.smithi042.stdout:For more information see: 2024-07-26T06:39:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:39:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-07-26T06:39:28.853 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:39:28.853 INFO:teuthology.orchestra.run.smithi042.stdout:Bootstrap complete. 2024-07-26T06:39:28.898 INFO:tasks.cephadm:Fetching config... 2024-07-26T06:39:28.899 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-07-26T06:39:28.899 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-07-26T06:39:28.917 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-07-26T06:39:28.917 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-07-26T06:39:28.917 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-07-26T06:39:28.973 INFO:tasks.cephadm:Fetching mon keyring... 2024-07-26T06:39:28.973 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-07-26T06:39:28.973 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/keyring of=/dev/stdout 2024-07-26T06:39:29.040 INFO:tasks.cephadm:Fetching pub ssh key... 2024-07-26T06:39:29.040 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-07-26T06:39:29.040 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-07-26T06:39:29.097 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-07-26T06:39:29.097 DEBUG:teuthology.orchestra.run.smithi042:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCs3w2hE4YBJ7k0oCeNpjLrwc5pyJ1I/LeJDD7l7l6xBnLPFVRtg82WKMEIrmmYwRGuscINRZZjHAY1X/bCIW6OlSvkHHBFBBzJO/SSVL6dP6Wr12RfAtSjaCC7FYIw2OG147qUJayXPRBWfgBhkymTjRfwujKeyBhPDKTr0541+78QqB44FdztZmRpuXsCZUvMDvrcTKoosPALJficHqPN72i54ACFBS0q+NC3UptDY5ekHiN9twAKiusS7DARktmOHZVggA2lVEUIY9KD4xIxa7bRzO7kDjojdWduAEWQu81abY1MrgBnYUzYWRU+PguOkkxGMG36Z5i+L4EhwrcjcyyKpsuclFrYpcF+fqvrzCeCADqapc3R4hq6yCJaRIPI3Om5d+ILFhW8lV+TfbxKcVmlHT1VrAEaunLCjZ18dwo4CCedg0TKxglYNb5cRb/vU6J3VS4/GWSio5XO+eZwThIWAywslyvLjpLbnwdHNlbiERE5qcP7HjtXaLnMSec= ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T06:39:29.172 INFO:teuthology.orchestra.run.smithi042.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCs3w2hE4YBJ7k0oCeNpjLrwc5pyJ1I/LeJDD7l7l6xBnLPFVRtg82WKMEIrmmYwRGuscINRZZjHAY1X/bCIW6OlSvkHHBFBBzJO/SSVL6dP6Wr12RfAtSjaCC7FYIw2OG147qUJayXPRBWfgBhkymTjRfwujKeyBhPDKTr0541+78QqB44FdztZmRpuXsCZUvMDvrcTKoosPALJficHqPN72i54ACFBS0q+NC3UptDY5ekHiN9twAKiusS7DARktmOHZVggA2lVEUIY9KD4xIxa7bRzO7kDjojdWduAEWQu81abY1MrgBnYUzYWRU+PguOkkxGMG36Z5i+L4EhwrcjcyyKpsuclFrYpcF+fqvrzCeCADqapc3R4hq6yCJaRIPI3Om5d+ILFhW8lV+TfbxKcVmlHT1VrAEaunLCjZ18dwo4CCedg0TKxglYNb5cRb/vU6J3VS4/GWSio5XO+eZwThIWAywslyvLjpLbnwdHNlbiERE5qcP7HjtXaLnMSec= ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:39:29.185 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCs3w2hE4YBJ7k0oCeNpjLrwc5pyJ1I/LeJDD7l7l6xBnLPFVRtg82WKMEIrmmYwRGuscINRZZjHAY1X/bCIW6OlSvkHHBFBBzJO/SSVL6dP6Wr12RfAtSjaCC7FYIw2OG147qUJayXPRBWfgBhkymTjRfwujKeyBhPDKTr0541+78QqB44FdztZmRpuXsCZUvMDvrcTKoosPALJficHqPN72i54ACFBS0q+NC3UptDY5ekHiN9twAKiusS7DARktmOHZVggA2lVEUIY9KD4xIxa7bRzO7kDjojdWduAEWQu81abY1MrgBnYUzYWRU+PguOkkxGMG36Z5i+L4EhwrcjcyyKpsuclFrYpcF+fqvrzCeCADqapc3R4hq6yCJaRIPI3Om5d+ILFhW8lV+TfbxKcVmlHT1VrAEaunLCjZ18dwo4CCedg0TKxglYNb5cRb/vU6J3VS4/GWSio5XO+eZwThIWAywslyvLjpLbnwdHNlbiERE5qcP7HjtXaLnMSec= ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T06:39:29.224 INFO:teuthology.orchestra.run.smithi163.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCs3w2hE4YBJ7k0oCeNpjLrwc5pyJ1I/LeJDD7l7l6xBnLPFVRtg82WKMEIrmmYwRGuscINRZZjHAY1X/bCIW6OlSvkHHBFBBzJO/SSVL6dP6Wr12RfAtSjaCC7FYIw2OG147qUJayXPRBWfgBhkymTjRfwujKeyBhPDKTr0541+78QqB44FdztZmRpuXsCZUvMDvrcTKoosPALJficHqPN72i54ACFBS0q+NC3UptDY5ekHiN9twAKiusS7DARktmOHZVggA2lVEUIY9KD4xIxa7bRzO7kDjojdWduAEWQu81abY1MrgBnYUzYWRU+PguOkkxGMG36Z5i+L4EhwrcjcyyKpsuclFrYpcF+fqvrzCeCADqapc3R4hq6yCJaRIPI3Om5d+ILFhW8lV+TfbxKcVmlHT1VrAEaunLCjZ18dwo4CCedg0TKxglYNb5cRb/vU6J3VS4/GWSio5XO+eZwThIWAywslyvLjpLbnwdHNlbiERE5qcP7HjtXaLnMSec= ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:39:29.235 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCs3w2hE4YBJ7k0oCeNpjLrwc5pyJ1I/LeJDD7l7l6xBnLPFVRtg82WKMEIrmmYwRGuscINRZZjHAY1X/bCIW6OlSvkHHBFBBzJO/SSVL6dP6Wr12RfAtSjaCC7FYIw2OG147qUJayXPRBWfgBhkymTjRfwujKeyBhPDKTr0541+78QqB44FdztZmRpuXsCZUvMDvrcTKoosPALJficHqPN72i54ACFBS0q+NC3UptDY5ekHiN9twAKiusS7DARktmOHZVggA2lVEUIY9KD4xIxa7bRzO7kDjojdWduAEWQu81abY1MrgBnYUzYWRU+PguOkkxGMG36Z5i+L4EhwrcjcyyKpsuclFrYpcF+fqvrzCeCADqapc3R4hq6yCJaRIPI3Om5d+ILFhW8lV+TfbxKcVmlHT1VrAEaunLCjZ18dwo4CCedg0TKxglYNb5cRb/vU6J3VS4/GWSio5XO+eZwThIWAywslyvLjpLbnwdHNlbiERE5qcP7HjtXaLnMSec= ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T06:39:29.275 INFO:teuthology.orchestra.run.smithi164.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCs3w2hE4YBJ7k0oCeNpjLrwc5pyJ1I/LeJDD7l7l6xBnLPFVRtg82WKMEIrmmYwRGuscINRZZjHAY1X/bCIW6OlSvkHHBFBBzJO/SSVL6dP6Wr12RfAtSjaCC7FYIw2OG147qUJayXPRBWfgBhkymTjRfwujKeyBhPDKTr0541+78QqB44FdztZmRpuXsCZUvMDvrcTKoosPALJficHqPN72i54ACFBS0q+NC3UptDY5ekHiN9twAKiusS7DARktmOHZVggA2lVEUIY9KD4xIxa7bRzO7kDjojdWduAEWQu81abY1MrgBnYUzYWRU+PguOkkxGMG36Z5i+L4EhwrcjcyyKpsuclFrYpcF+fqvrzCeCADqapc3R4hq6yCJaRIPI3Om5d+ILFhW8lV+TfbxKcVmlHT1VrAEaunLCjZ18dwo4CCedg0TKxglYNb5cRb/vU6J3VS4/GWSio5XO+eZwThIWAywslyvLjpLbnwdHNlbiERE5qcP7HjtXaLnMSec= ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:39:29.287 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-07-26T06:39:29.550 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:39:32.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi042", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:39:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:39:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:32.484 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-07-26T06:39:32.485 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-07-26T06:39:32.756 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:39:33.126 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:32 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1189088589' entity='client.admin' 2024-07-26T06:39:33.127 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:39:33.127 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:35.882 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi163 2024-07-26T06:39:35.895 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:39:35.896 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/etc/ceph/ceph.conf 2024-07-26T06:39:35.915 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:39:35.916 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T06:39:35.973 INFO:tasks.cephadm:Adding host smithi163 to orchestrator... 2024-07-26T06:39:35.973 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch host add smithi163 2024-07-26T06:39:36.234 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:39:36.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:36 smithi042 ceph-mon[32775]: from='client.14170 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:39:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:39:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:39:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:39:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:36 smithi042 ceph-mon[32775]: Updating smithi042:/etc/ceph/ceph.conf 2024-07-26T06:39:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:36 smithi042 ceph-mon[32775]: Updating smithi042:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T06:39:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:36 smithi042 ceph-mon[32775]: Updating smithi042:/etc/ceph/ceph.client.admin.keyring 2024-07-26T06:39:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:36 smithi042 ceph-mon[32775]: Updating smithi042:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T06:39:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:38.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:38 smithi042 ceph-mon[32775]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi163", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:39:40.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:40 smithi042 ceph-mon[32775]: Deploying cephadm binary to smithi163 2024-07-26T06:39:42.049 INFO:teuthology.orchestra.run.smithi042.stdout:Added host 'smithi163' with addr '172.21.15.163' 2024-07-26T06:39:42.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:42 smithi042 ceph-mon[32775]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:39:42.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:42.345 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:39:42.640 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch host ls --format=json 2024-07-26T06:39:42.902 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:39:43.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:43 smithi042 ceph-mon[32775]: Added host smithi163 2024-07-26T06:39:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:44.542 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:39:44.545 INFO:teuthology.orchestra.run.smithi042.stdout:[{"addr": "172.21.15.42", "hostname": "smithi042", "labels": [], "status": ""}, {"addr": "172.21.15.163", "hostname": "smithi163", "labels": [], "status": ""}] 2024-07-26T06:39:44.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:44 smithi042 ceph-mon[32775]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:39:44.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:45.118 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi164 2024-07-26T06:39:45.118 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:39:45.118 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/etc/ceph/ceph.conf 2024-07-26T06:39:45.138 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:39:45.138 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T06:39:45.195 INFO:tasks.cephadm:Adding host smithi164 to orchestrator... 2024-07-26T06:39:45.196 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch host add smithi164 2024-07-26T06:39:45.454 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:39:45.655 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:45 smithi042 ceph-mon[32775]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:39:46.707 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:46 smithi042 ceph-mon[32775]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:39:46.707 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:46.708 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:46.708 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:46.708 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:46.708 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi163", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:39:46.708 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:39:46.708 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:39:48.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:47 smithi042 ceph-mon[32775]: Updating smithi163:/etc/ceph/ceph.conf 2024-07-26T06:39:48.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:47 smithi042 ceph-mon[32775]: Updating smithi163:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T06:39:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:47 smithi042 ceph-mon[32775]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-07-26T06:39:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:47 smithi042 ceph-mon[32775]: Updating smithi163:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T06:39:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:49.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:48 smithi042 ceph-mon[32775]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:39:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:48 smithi042 ceph-mon[32775]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi164", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:39:50.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:49 smithi042 ceph-mon[32775]: Deploying cephadm binary to smithi164 2024-07-26T06:39:51.102 INFO:teuthology.orchestra.run.smithi042.stdout:Added host 'smithi164' with addr '172.21.15.164' 2024-07-26T06:39:51.114 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:50 smithi042 ceph-mon[32775]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:39:51.823 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch host ls --format=json 2024-07-26T06:39:52.084 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:39:52.390 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:52.390 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:52 smithi042 ceph-mon[32775]: Added host smithi164 2024-07-26T06:39:52.390 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:39:52.390 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:52 smithi042 ceph-mon[32775]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:39:52.391 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:52.391 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:53.641 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:39:53.641 INFO:teuthology.orchestra.run.smithi042.stdout:[{"addr": "172.21.15.42", "hostname": "smithi042", "labels": [], "status": ""}, {"addr": "172.21.15.163", "hostname": "smithi163", "labels": [], "status": ""}, {"addr": "172.21.15.164", "hostname": "smithi164", "labels": [], "status": ""}] 2024-07-26T06:39:53.642 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:54.189 INFO:tasks.cephadm:Setting crush tunables to default 2024-07-26T06:39:54.190 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd crush tunables default 2024-07-26T06:39:54.449 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:39:54.777 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:54 smithi042 ceph-mon[32775]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:39:54.777 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:54 smithi042 ceph-mon[32775]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:39:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:39:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:39:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:39:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:56 smithi042 ceph-mon[32775]: Updating smithi164:/etc/ceph/ceph.conf 2024-07-26T06:39:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:56 smithi042 ceph-mon[32775]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:39:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:56 smithi042 ceph-mon[32775]: Updating smithi164:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T06:39:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:56 smithi042 ceph-mon[32775]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-07-26T06:39:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:56 smithi042 ceph-mon[32775]: Updating smithi164:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T06:39:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:39:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:56 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/4180670100' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-07-26T06:39:57.044 INFO:teuthology.orchestra.run.smithi042.stderr:adjusted tunables profile to default 2024-07-26T06:39:57.670 INFO:tasks.cephadm:Adding mon.a on smithi042 2024-07-26T06:39:57.671 INFO:tasks.cephadm:Adding mon.b on smithi163 2024-07-26T06:39:57.671 INFO:tasks.cephadm:Adding mon.c on smithi164 2024-07-26T06:39:57.671 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch apply mon '3;smithi042:172.21.15.42=a;smithi163:172.21.15.163=b;smithi164:172.21.15.164=c' 2024-07-26T06:39:57.921 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T06:39:58.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:58 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/4180670100' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-07-26T06:39:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:58 smithi042 ceph-mon[32775]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T06:39:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:39:58 smithi042 ceph-mon[32775]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:39:59.658 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled mon update... 2024-07-26T06:40:00.211 DEBUG:teuthology.orchestra.run.smithi163:mon.b> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mon.b.service 2024-07-26T06:40:00.214 DEBUG:teuthology.orchestra.run.smithi164:mon.c> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mon.c.service 2024-07-26T06:40:00.217 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T06:40:00.217 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph mon dump -f json 2024-07-26T06:40:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:00 smithi042 ceph-mon[32775]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:00 smithi042 ceph-mon[32775]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi042:172.21.15.42=a;smithi163:172.21.15.163=b;smithi164:172.21.15.164=c", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:40:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:00 smithi042 ceph-mon[32775]: Saving service mon spec with placement smithi042:172.21.15.42=a;smithi163:172.21.15.163=b;smithi164:172.21.15.164=c;count:3 2024-07-26T06:40:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T06:40:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:00 smithi042 ceph-mon[32775]: Deploying daemon mon.c on smithi164 2024-07-26T06:40:01.314 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T06:40:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:02 smithi042 ceph-mon[32775]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:04.174 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:40:04.174 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":1,"fsid":"8b3e2750-4b19-11ef-bcb6-c7b262605968","modified":"2024-07-26T06:38:26.468434Z","created":"2024-07-26T06:38:26.468434Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-07-26T06:40:04.174 INFO:teuthology.orchestra.run.smithi164.stderr:dumped monmap epoch 1 2024-07-26T06:40:04.971 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:04 smithi164 podman[36951]: 2024-07-26 06:40:04.792325049 +0000 UTC m=+0.017071716 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:40:05.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:04 smithi042 ceph-mon[32775]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:05.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:04 smithi042 ceph-mon[32775]: from='client.? 172.21.15.164:0/3173483381' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T06:40:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 podman[36951]: 2024-07-26 06:40:05.051528434 +0000 UTC m=+0.276275090 container init e19612f16f406eaea736aebc997329c19285598ee6a8642be6aaa380e1317e85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mon-c, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-07-26T06:40:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 podman[36951]: 2024-07-26 06:40:05.055460832 +0000 UTC m=+0.280207490 container start e19612f16f406eaea736aebc997329c19285598ee6a8642be6aaa380e1317e85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-07-26T06:40:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: set uid:gid to 167:167 (ceph:ceph) 2024-07-26T06:40:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev), process ceph-mon, pid 2 2024-07-26T06:40:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: pidfile_write: ignore empty --pid-file 2024-07-26T06:40:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: load: jerasure load: lrc 2024-07-26T06:40:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: RocksDB version: 7.9.2 2024-07-26T06:40:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Git sha 0 2024-07-26T06:40:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Compile date 2024-07-25 13:49:38 2024-07-26T06:40:05.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: DB SUMMARY 2024-07-26T06:40:05.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: DB Session ID: Y50OCTUKIX70DOW0WSO1 2024-07-26T06:40:05.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: CURRENT file: CURRENT 2024-07-26T06:40:05.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: IDENTITY file: IDENTITY 2024-07-26T06:40:05.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-07-26T06:40:05.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-07-26T06:40:05.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-07-26T06:40:05.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.error_if_exists: 0 2024-07-26T06:40:05.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.create_if_missing: 0 2024-07-26T06:40:05.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.paranoid_checks: 1 2024-07-26T06:40:05.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.flush_verify_memtable_count: 1 2024-07-26T06:40:05.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-07-26T06:40:05.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-07-26T06:40:05.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.env: 0x55e6253e8280 2024-07-26T06:40:05.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.fs: PosixFileSystem 2024-07-26T06:40:05.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.info_log: 0x55e62b56b080 2024-07-26T06:40:05.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_file_opening_threads: 16 2024-07-26T06:40:05.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.statistics: (nil) 2024-07-26T06:40:05.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.use_fsync: 0 2024-07-26T06:40:05.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_log_file_size: 0 2024-07-26T06:40:05.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-07-26T06:40:05.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.log_file_time_to_roll: 0 2024-07-26T06:40:05.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.keep_log_file_num: 1000 2024-07-26T06:40:05.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.recycle_log_file_num: 0 2024-07-26T06:40:05.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.allow_fallocate: 1 2024-07-26T06:40:05.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.allow_mmap_reads: 0 2024-07-26T06:40:05.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.allow_mmap_writes: 0 2024-07-26T06:40:05.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.use_direct_reads: 0 2024-07-26T06:40:05.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-07-26T06:40:05.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.create_missing_column_families: 0 2024-07-26T06:40:05.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.db_log_dir: 2024-07-26T06:40:05.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.wal_dir: 2024-07-26T06:40:05.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.table_cache_numshardbits: 6 2024-07-26T06:40:05.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.WAL_ttl_seconds: 0 2024-07-26T06:40:05.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.WAL_size_limit_MB: 0 2024-07-26T06:40:05.228 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-07-26T06:40:05.228 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-07-26T06:40:05.228 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.is_fd_close_on_exec: 1 2024-07-26T06:40:05.228 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.advise_random_on_open: 1 2024-07-26T06:40:05.228 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.db_write_buffer_size: 0 2024-07-26T06:40:05.228 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.write_buffer_manager: 0x55e62b56f5e0 2024-07-26T06:40:05.228 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-07-26T06:40:05.228 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-07-26T06:40:05.228 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.use_adaptive_mutex: 0 2024-07-26T06:40:05.228 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.rate_limiter: (nil) 2024-07-26T06:40:05.229 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-07-26T06:40:05.229 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.wal_recovery_mode: 2 2024-07-26T06:40:05.229 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.enable_thread_tracking: 0 2024-07-26T06:40:05.229 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.enable_pipelined_write: 0 2024-07-26T06:40:05.229 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.unordered_write: 0 2024-07-26T06:40:05.229 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-07-26T06:40:05.229 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-07-26T06:40:05.229 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-07-26T06:40:05.229 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-07-26T06:40:05.229 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.row_cache: None 2024-07-26T06:40:05.229 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.wal_filter: None 2024-07-26T06:40:05.229 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-07-26T06:40:05.230 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.allow_ingest_behind: 0 2024-07-26T06:40:05.230 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.two_write_queues: 0 2024-07-26T06:40:05.230 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.manual_wal_flush: 0 2024-07-26T06:40:05.230 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.wal_compression: 0 2024-07-26T06:40:05.230 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.atomic_flush: 0 2024-07-26T06:40:05.230 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-07-26T06:40:05.230 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.persist_stats_to_disk: 0 2024-07-26T06:40:05.230 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.write_dbid_to_manifest: 0 2024-07-26T06:40:05.230 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.log_readahead_size: 0 2024-07-26T06:40:05.230 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-07-26T06:40:05.230 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.best_efforts_recovery: 0 2024-07-26T06:40:05.230 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-07-26T06:40:05.231 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-07-26T06:40:05.231 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.allow_data_in_errors: 0 2024-07-26T06:40:05.231 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.db_host_id: __hostname__ 2024-07-26T06:40:05.231 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.enforce_single_del_contracts: true 2024-07-26T06:40:05.231 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_background_jobs: 2 2024-07-26T06:40:05.231 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_background_compactions: -1 2024-07-26T06:40:05.231 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_subcompactions: 1 2024-07-26T06:40:05.231 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-07-26T06:40:05.231 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-07-26T06:40:05.231 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.delayed_write_rate : 16777216 2024-07-26T06:40:05.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_total_wal_size: 0 2024-07-26T06:40:05.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-07-26T06:40:05.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.stats_dump_period_sec: 600 2024-07-26T06:40:05.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.stats_persist_period_sec: 600 2024-07-26T06:40:05.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-07-26T06:40:05.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_open_files: -1 2024-07-26T06:40:05.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.bytes_per_sync: 0 2024-07-26T06:40:05.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.wal_bytes_per_sync: 0 2024-07-26T06:40:05.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.strict_bytes_per_sync: 0 2024-07-26T06:40:05.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compaction_readahead_size: 0 2024-07-26T06:40:05.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_background_flushes: -1 2024-07-26T06:40:05.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Compression algorithms supported: 2024-07-26T06:40:05.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: kZSTD supported: 0 2024-07-26T06:40:05.233 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: kXpressCompression supported: 0 2024-07-26T06:40:05.233 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: kBZip2Compression supported: 0 2024-07-26T06:40:05.233 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-07-26T06:40:05.233 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: kLZ4Compression supported: 1 2024-07-26T06:40:05.233 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: kZlibCompression supported: 1 2024-07-26T06:40:05.233 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: kLZ4HCCompression supported: 1 2024-07-26T06:40:05.233 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: kSnappyCompression supported: 1 2024-07-26T06:40:05.233 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Fast CRC32 supported: Supported on x86 2024-07-26T06:40:05.233 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: DMutex implementation: pthread_mutex_t 2024-07-26T06:40:05.233 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-07-26T06:40:05.233 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-07-26T06:40:05.234 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-07-26T06:40:05.234 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.merge_operator: 2024-07-26T06:40:05.234 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compaction_filter: None 2024-07-26T06:40:05.234 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compaction_filter_factory: None 2024-07-26T06:40:05.234 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.sst_partitioner_factory: None 2024-07-26T06:40:05.234 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.memtable_factory: SkipListFactory 2024-07-26T06:40:05.234 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.table_factory: BlockBasedTable 2024-07-26T06:40:05.234 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e62b56ac80) 2024-07-26T06:40:05.234 INFO:journalctl@ceph.mon.c.smithi164.stdout: cache_index_and_filter_blocks: 1 2024-07-26T06:40:05.234 INFO:journalctl@ceph.mon.c.smithi164.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-07-26T06:40:05.234 INFO:journalctl@ceph.mon.c.smithi164.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-07-26T06:40:05.234 INFO:journalctl@ceph.mon.c.smithi164.stdout: pin_top_level_index_and_filter: 1 2024-07-26T06:40:05.235 INFO:journalctl@ceph.mon.c.smithi164.stdout: index_type: 0 2024-07-26T06:40:05.235 INFO:journalctl@ceph.mon.c.smithi164.stdout: data_block_index_type: 0 2024-07-26T06:40:05.235 INFO:journalctl@ceph.mon.c.smithi164.stdout: index_shortening: 1 2024-07-26T06:40:05.235 INFO:journalctl@ceph.mon.c.smithi164.stdout: data_block_hash_table_util_ratio: 0.750000 2024-07-26T06:40:05.235 INFO:journalctl@ceph.mon.c.smithi164.stdout: checksum: 4 2024-07-26T06:40:05.235 INFO:journalctl@ceph.mon.c.smithi164.stdout: no_block_cache: 0 2024-07-26T06:40:05.235 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_cache: 0x55e62752f610 2024-07-26T06:40:05.235 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_cache_name: BinnedLRUCache 2024-07-26T06:40:05.235 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_cache_options: 2024-07-26T06:40:05.235 INFO:journalctl@ceph.mon.c.smithi164.stdout: capacity : 536870912 2024-07-26T06:40:05.235 INFO:journalctl@ceph.mon.c.smithi164.stdout: num_shard_bits : 4 2024-07-26T06:40:05.235 INFO:journalctl@ceph.mon.c.smithi164.stdout: strict_capacity_limit : 0 2024-07-26T06:40:05.235 INFO:journalctl@ceph.mon.c.smithi164.stdout: high_pri_pool_ratio: 0.000 2024-07-26T06:40:05.236 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_cache_compressed: (nil) 2024-07-26T06:40:05.236 INFO:journalctl@ceph.mon.c.smithi164.stdout: persistent_cache: (nil) 2024-07-26T06:40:05.236 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_size: 4096 2024-07-26T06:40:05.236 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_size_deviation: 10 2024-07-26T06:40:05.236 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_restart_interval: 16 2024-07-26T06:40:05.236 INFO:journalctl@ceph.mon.c.smithi164.stdout: index_block_restart_interval: 1 2024-07-26T06:40:05.236 INFO:journalctl@ceph.mon.c.smithi164.stdout: metadata_block_size: 4096 2024-07-26T06:40:05.236 INFO:journalctl@ceph.mon.c.smithi164.stdout: partition_filters: 0 2024-07-26T06:40:05.236 INFO:journalctl@ceph.mon.c.smithi164.stdout: use_delta_encoding: 1 2024-07-26T06:40:05.236 INFO:journalctl@ceph.mon.c.smithi164.stdout: filter_policy: bloomfilter 2024-07-26T06:40:05.236 INFO:journalctl@ceph.mon.c.smithi164.stdout: whole_key_filtering: 1 2024-07-26T06:40:05.236 INFO:journalctl@ceph.mon.c.smithi164.stdout: verify_compression: 0 2024-07-26T06:40:05.236 INFO:journalctl@ceph.mon.c.smithi164.stdout: read_amp_bytes_per_bit: 0 2024-07-26T06:40:05.236 INFO:journalctl@ceph.mon.c.smithi164.stdout: format_version: 5 2024-07-26T06:40:05.237 INFO:journalctl@ceph.mon.c.smithi164.stdout: enable_index_compression: 1 2024-07-26T06:40:05.237 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_align: 0 2024-07-26T06:40:05.237 INFO:journalctl@ceph.mon.c.smithi164.stdout: max_auto_readahead_size: 262144 2024-07-26T06:40:05.237 INFO:journalctl@ceph.mon.c.smithi164.stdout: prepopulate_block_cache: 0 2024-07-26T06:40:05.237 INFO:journalctl@ceph.mon.c.smithi164.stdout: initial_auto_readahead_size: 8192 2024-07-26T06:40:05.237 INFO:journalctl@ceph.mon.c.smithi164.stdout: num_file_reads_for_auto_readahead: 2 2024-07-26T06:40:05.237 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.write_buffer_size: 33554432 2024-07-26T06:40:05.237 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_write_buffer_number: 2 2024-07-26T06:40:05.237 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compression: NoCompression 2024-07-26T06:40:05.237 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.bottommost_compression: Disabled 2024-07-26T06:40:05.237 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.prefix_extractor: nullptr 2024-07-26T06:40:05.237 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-07-26T06:40:05.237 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.num_levels: 7 2024-07-26T06:40:05.237 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-07-26T06:40:05.238 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-07-26T06:40:05.238 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-07-26T06:40:05.238 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-07-26T06:40:05.238 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-07-26T06:40:05.238 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-07-26T06:40:05.238 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-07-26T06:40:05.238 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-07-26T06:40:05.238 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-07-26T06:40:05.238 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-07-26T06:40:05.238 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-07-26T06:40:05.238 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-07-26T06:40:05.238 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compression_opts.window_bits: -14 2024-07-26T06:40:05.239 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compression_opts.level: 32767 2024-07-26T06:40:05.239 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compression_opts.strategy: 0 2024-07-26T06:40:05.239 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-07-26T06:40:05.239 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-07-26T06:40:05.239 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-07-26T06:40:05.239 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-07-26T06:40:05.239 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compression_opts.enabled: false 2024-07-26T06:40:05.239 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-07-26T06:40:05.239 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-07-26T06:40:05.239 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-07-26T06:40:05.239 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-07-26T06:40:05.239 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.target_file_size_base: 67108864 2024-07-26T06:40:05.240 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.target_file_size_multiplier: 1 2024-07-26T06:40:05.240 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-07-26T06:40:05.240 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-07-26T06:40:05.240 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-07-26T06:40:05.240 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-07-26T06:40:05.240 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-07-26T06:40:05.240 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-07-26T06:40:05.240 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-07-26T06:40:05.240 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-07-26T06:40:05.240 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-07-26T06:40:05.240 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-07-26T06:40:05.240 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-07-26T06:40:05.240 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-07-26T06:40:05.241 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-07-26T06:40:05.241 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.arena_block_size: 1048576 2024-07-26T06:40:05.241 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-07-26T06:40:05.241 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-07-26T06:40:05.241 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.disable_auto_compactions: 0 2024-07-26T06:40:05.241 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-07-26T06:40:05.241 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-07-26T06:40:05.241 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-07-26T06:40:05.241 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-07-26T06:40:05.241 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-07-26T06:40:05.241 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-07-26T06:40:05.241 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-07-26T06:40:05.241 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-07-26T06:40:05.242 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-07-26T06:40:05.242 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-07-26T06:40:05.242 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-07-26T06:40:05.242 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.inplace_update_support: 0 2024-07-26T06:40:05.242 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.inplace_update_num_locks: 10000 2024-07-26T06:40:05.242 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-07-26T06:40:05.242 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-07-26T06:40:05.242 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.memtable_huge_page_size: 0 2024-07-26T06:40:05.242 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.bloom_locality: 0 2024-07-26T06:40:05.242 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.max_successive_merges: 0 2024-07-26T06:40:05.242 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.optimize_filters_for_hits: 0 2024-07-26T06:40:05.242 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.paranoid_file_checks: 0 2024-07-26T06:40:05.243 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.force_consistency_checks: 1 2024-07-26T06:40:05.243 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.report_bg_io_stats: 0 2024-07-26T06:40:05.243 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.ttl: 2592000 2024-07-26T06:40:05.243 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.periodic_compaction_seconds: 0 2024-07-26T06:40:05.243 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-07-26T06:40:05.244 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-07-26T06:40:05.244 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.enable_blob_files: false 2024-07-26T06:40:05.244 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.min_blob_size: 0 2024-07-26T06:40:05.244 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.blob_file_size: 268435456 2024-07-26T06:40:05.244 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.blob_compression_type: NoCompression 2024-07-26T06:40:05.244 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.enable_blob_garbage_collection: false 2024-07-26T06:40:05.244 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-07-26T06:40:05.244 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-07-26T06:40:05.244 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-07-26T06:40:05.245 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.blob_file_starting_level: 0 2024-07-26T06:40:05.245 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-07-26T06:40:05.245 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-07-26T06:40:05.245 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-07-26T06:40:05.245 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e7ef5e95-385f-48b5-a29a-5beaeb0172d0 2024-07-26T06:40:05.245 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721976005103024, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-07-26T06:40:05.245 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-07-26T06:40:05.245 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721976005103440, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1721976005, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e7ef5e95-385f-48b5-a29a-5beaeb0172d0", "db_session_id": "Y50OCTUKIX70DOW0WSO1", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-07-26T06:40:05.245 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721976005103535, "job": 1, "event": "recovery_finished"} 2024-07-26T06:40:05.245 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-07-26T06:40:05.245 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-07-26T06:40:05.245 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e62b58ca80 2024-07-26T06:40:05.245 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: DB pointer 0x55e62c32c000 2024-07-26T06:40:05.246 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-07-26T06:40:05.246 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-07-26T06:40:05.246 INFO:journalctl@ceph.mon.c.smithi164.stdout: ** DB Stats ** 2024-07-26T06:40:05.246 INFO:journalctl@ceph.mon.c.smithi164.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T06:40:05.246 INFO:journalctl@ceph.mon.c.smithi164.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-07-26T06:40:05.246 INFO:journalctl@ceph.mon.c.smithi164.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T06:40:05.246 INFO:journalctl@ceph.mon.c.smithi164.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T06:40:05.246 INFO:journalctl@ceph.mon.c.smithi164.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-07-26T06:40:05.246 INFO:journalctl@ceph.mon.c.smithi164.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T06:40:05.246 INFO:journalctl@ceph.mon.c.smithi164.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T06:40:05.246 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-07-26T06:40:05.246 INFO:journalctl@ceph.mon.c.smithi164.stdout: ** Compaction Stats [default] ** 2024-07-26T06:40:05.247 INFO:journalctl@ceph.mon.c.smithi164.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T06:40:05.247 INFO:journalctl@ceph.mon.c.smithi164.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-07-26T06:40:05.247 INFO:journalctl@ceph.mon.c.smithi164.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T06:40:05.247 INFO:journalctl@ceph.mon.c.smithi164.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T06:40:05.247 INFO:journalctl@ceph.mon.c.smithi164.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T06:40:05.247 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-07-26T06:40:05.247 INFO:journalctl@ceph.mon.c.smithi164.stdout: ** Compaction Stats [default] ** 2024-07-26T06:40:05.247 INFO:journalctl@ceph.mon.c.smithi164.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T06:40:05.247 INFO:journalctl@ceph.mon.c.smithi164.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-07-26T06:40:05.247 INFO:journalctl@ceph.mon.c.smithi164.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T06:40:05.247 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-07-26T06:40:05.247 INFO:journalctl@ceph.mon.c.smithi164.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-07-26T06:40:05.248 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-07-26T06:40:05.248 INFO:journalctl@ceph.mon.c.smithi164.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T06:40:05.248 INFO:journalctl@ceph.mon.c.smithi164.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-07-26T06:40:05.248 INFO:journalctl@ceph.mon.c.smithi164.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-07-26T06:40:05.248 INFO:journalctl@ceph.mon.c.smithi164.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-07-26T06:40:05.248 INFO:journalctl@ceph.mon.c.smithi164.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-07-26T06:40:05.248 INFO:journalctl@ceph.mon.c.smithi164.stdout: AddFile(Keys): cumulative 0, interval 0 2024-07-26T06:40:05.248 INFO:journalctl@ceph.mon.c.smithi164.stdout: Cumulative compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T06:40:05.248 INFO:journalctl@ceph.mon.c.smithi164.stdout: Interval compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T06:40:05.248 INFO:journalctl@ceph.mon.c.smithi164.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-07-26T06:40:05.248 INFO:journalctl@ceph.mon.c.smithi164.stdout: Block cache BinnedLRUCache@0x55e62752f610#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-07-26T06:40:05.248 INFO:journalctl@ceph.mon.c.smithi164.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-07-26T06:40:05.248 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-07-26T06:40:05.249 INFO:journalctl@ceph.mon.c.smithi164.stdout: ** File Read Latency Histogram By Level [default] ** 2024-07-26T06:40:05.249 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-07-26T06:40:05.249 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: using public_addr v2:172.21.15.164:0/0 -> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-07-26T06:40:05.249 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: starting mon.c rank -1 at public addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] at bind addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:40:05.249 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: mon.c@-1(???) e0 preinit fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:40:05.249 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: mon.c@-1(synchronizing).mds e1 new map 2024-07-26T06:40:05.249 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: mon.c@-1(synchronizing).mds e1 print_map 2024-07-26T06:40:05.249 INFO:journalctl@ceph.mon.c.smithi164.stdout: e1 2024-07-26T06:40:05.249 INFO:journalctl@ceph.mon.c.smithi164.stdout: btime 2024-07-26T06:38:29:306299+0000 2024-07-26T06:40:05.249 INFO:journalctl@ceph.mon.c.smithi164.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T06:40:05.249 INFO:journalctl@ceph.mon.c.smithi164.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T06:40:05.249 INFO:journalctl@ceph.mon.c.smithi164.stdout: legacy client fscid: -1 2024-07-26T06:40:05.249 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-07-26T06:40:05.250 INFO:journalctl@ceph.mon.c.smithi164.stdout: No filesystems configured 2024-07-26T06:40:05.250 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-07-26T06:40:05.250 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-07-26T06:40:05.250 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-07-26T06:40:05.250 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-07-26T06:40:05.250 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-07-26T06:40:05.250 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-07-26T06:40:05.250 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-07-26T06:40:05.250 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T06:40:05.250 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T06:40:05.250 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T06:40:05.251 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:05.251 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:05.251 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: Added host smithi164 2024-07-26T06:40:05.251 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:05.251 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:05.251 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:05.251 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:05.251 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:05.251 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:05.251 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:40:05.251 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:05.251 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:05.251 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:05.252 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:05.252 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:40:05.252 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:05.252 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:05.252 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: Updating smithi164:/etc/ceph/ceph.conf 2024-07-26T06:40:05.252 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:05.252 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: Updating smithi164:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T06:40:05.252 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-07-26T06:40:05.252 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: Updating smithi164:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T06:40:05.252 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:05.252 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:05.252 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:05.252 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/4180670100' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-07-26T06:40:05.253 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/4180670100' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-07-26T06:40:05.253 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T06:40:05.253 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:05.253 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:05.253 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi042:172.21.15.42=a;smithi163:172.21.15.163=b;smithi164:172.21.15.164=c", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:40:05.253 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: Saving service mon spec with placement smithi042:172.21.15.42=a;smithi163:172.21.15.163=b;smithi164:172.21.15.164=c;count:3 2024-07-26T06:40:05.253 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:05.253 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:05.253 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:05.253 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:05.253 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:05.253 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T06:40:05.253 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:05.254 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: Deploying daemon mon.c on smithi164 2024-07-26T06:40:05.254 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:05.254 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:05.254 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: from='client.? 172.21.15.164:0/3173483381' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T06:40:05.254 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 ceph-mon[36970]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-07-26T06:40:05.254 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 bash[36951]: e19612f16f406eaea736aebc997329c19285598ee6a8642be6aaa380e1317e85 2024-07-26T06:40:05.254 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:05 smithi164 systemd[1]: Started Ceph mon.c for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:40:05.737 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T06:40:05.737 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph mon dump -f json 2024-07-26T06:40:05.986 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T06:40:10.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:40:10.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: mon.a calling monitor election 2024-07-26T06:40:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: mon.c calling monitor election 2024-07-26T06:40:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:10.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:10.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T06:40:10.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: monmap epoch 2 2024-07-26T06:40:10.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:40:10.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: last_changed 2024-07-26T06:40:05.131205+0000 2024-07-26T06:40:10.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: created 2024-07-26T06:38:26.468434+0000 2024-07-26T06:40:10.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: min_mon_release 19 (squid) 2024-07-26T06:40:10.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: election_strategy: 1 2024-07-26T06:40:10.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: 0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-07-26T06:40:10.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: 1: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.c 2024-07-26T06:40:10.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: fsmap 2024-07-26T06:40:10.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T06:40:10.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: mgrmap e13: x(active, since 48s) 2024-07-26T06:40:10.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: overall HEALTH_OK 2024-07-26T06:40:10.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:10.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:10.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:10.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T06:40:10.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:40:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: mon.a calling monitor election 2024-07-26T06:40:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: mon.c calling monitor election 2024-07-26T06:40:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T06:40:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: monmap epoch 2 2024-07-26T06:40:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:40:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: last_changed 2024-07-26T06:40:05.131205+0000 2024-07-26T06:40:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: created 2024-07-26T06:38:26.468434+0000 2024-07-26T06:40:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: min_mon_release 19 (squid) 2024-07-26T06:40:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: election_strategy: 1 2024-07-26T06:40:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: 0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-07-26T06:40:10.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: 1: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.c 2024-07-26T06:40:10.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: fsmap 2024-07-26T06:40:10.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T06:40:10.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: mgrmap e13: x(active, since 48s) 2024-07-26T06:40:10.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: overall HEALTH_OK 2024-07-26T06:40:10.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:10.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:10.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:10.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T06:40:10.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:10.827 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:40:10.827 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":2,"fsid":"8b3e2750-4b19-11ef-bcb6-c7b262605968","modified":"2024-07-26T06:40:05.131205Z","created":"2024-07-26T06:38:26.468434Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-07-26T06:40:10.828 INFO:teuthology.orchestra.run.smithi164.stderr:dumped monmap epoch 2 2024-07-26T06:40:11.388 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:11 smithi164 ceph-mon[36970]: Deploying daemon mon.b on smithi163 2024-07-26T06:40:11.388 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:11 smithi164 ceph-mon[36970]: from='client.? 172.21.15.164:0/197110018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T06:40:11.388 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:11.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:11 smithi042 ceph-mon[32775]: Deploying daemon mon.b on smithi163 2024-07-26T06:40:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:11 smithi042 ceph-mon[32775]: from='client.? 172.21.15.164:0/197110018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T06:40:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:12.416 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T06:40:12.417 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph mon dump -f json 2024-07-26T06:40:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:12 smithi164 ceph-mon[36970]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:12.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:12 smithi042 ceph-mon[32775]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:12.667 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T06:40:13.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: kBZip2Compression supported: 0 2024-07-26T06:40:13.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-07-26T06:40:13.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: kLZ4Compression supported: 1 2024-07-26T06:40:13.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: kZlibCompression supported: 1 2024-07-26T06:40:13.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: kLZ4HCCompression supported: 1 2024-07-26T06:40:13.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: kSnappyCompression supported: 1 2024-07-26T06:40:13.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Fast CRC32 supported: Supported on x86 2024-07-26T06:40:13.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: DMutex implementation: pthread_mutex_t 2024-07-26T06:40:13.983 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-07-26T06:40:13.984 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-07-26T06:40:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-07-26T06:40:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.merge_operator: 2024-07-26T06:40:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compaction_filter: None 2024-07-26T06:40:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compaction_filter_factory: None 2024-07-26T06:40:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.sst_partitioner_factory: None 2024-07-26T06:40:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.memtable_factory: SkipListFactory 2024-07-26T06:40:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.table_factory: BlockBasedTable 2024-07-26T06:40:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55875801ac80) 2024-07-26T06:40:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout: cache_index_and_filter_blocks: 1 2024-07-26T06:40:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-07-26T06:40:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-07-26T06:40:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout: pin_top_level_index_and_filter: 1 2024-07-26T06:40:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout: index_type: 0 2024-07-26T06:40:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout: data_block_index_type: 0 2024-07-26T06:40:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout: index_shortening: 1 2024-07-26T06:40:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout: data_block_hash_table_util_ratio: 0.750000 2024-07-26T06:40:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout: checksum: 4 2024-07-26T06:40:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout: no_block_cache: 0 2024-07-26T06:40:14.259 INFO:journalctl@ceph.mon.b.smithi163.stdout: block_cache: 0x558753fdf610 2024-07-26T06:40:14.259 INFO:journalctl@ceph.mon.b.smithi163.stdout: block_cache_name: BinnedLRUCache 2024-07-26T06:40:14.259 INFO:journalctl@ceph.mon.b.smithi163.stdout: block_cache_options: 2024-07-26T06:40:14.259 INFO:journalctl@ceph.mon.b.smithi163.stdout: capacity : 536870912 2024-07-26T06:40:14.259 INFO:journalctl@ceph.mon.b.smithi163.stdout: num_shard_bits : 4 2024-07-26T06:40:14.259 INFO:journalctl@ceph.mon.b.smithi163.stdout: strict_capacity_limit : 0 2024-07-26T06:40:14.259 INFO:journalctl@ceph.mon.b.smithi163.stdout: high_pri_pool_ratio: 0.000 2024-07-26T06:40:14.259 INFO:journalctl@ceph.mon.b.smithi163.stdout: block_cache_compressed: (nil) 2024-07-26T06:40:14.259 INFO:journalctl@ceph.mon.b.smithi163.stdout: persistent_cache: (nil) 2024-07-26T06:40:14.259 INFO:journalctl@ceph.mon.b.smithi163.stdout: block_size: 4096 2024-07-26T06:40:14.259 INFO:journalctl@ceph.mon.b.smithi163.stdout: block_size_deviation: 10 2024-07-26T06:40:14.259 INFO:journalctl@ceph.mon.b.smithi163.stdout: block_restart_interval: 16 2024-07-26T06:40:14.259 INFO:journalctl@ceph.mon.b.smithi163.stdout: index_block_restart_interval: 1 2024-07-26T06:40:14.260 INFO:journalctl@ceph.mon.b.smithi163.stdout: metadata_block_size: 4096 2024-07-26T06:40:14.260 INFO:journalctl@ceph.mon.b.smithi163.stdout: partition_filters: 0 2024-07-26T06:40:14.260 INFO:journalctl@ceph.mon.b.smithi163.stdout: use_delta_encoding: 1 2024-07-26T06:40:14.260 INFO:journalctl@ceph.mon.b.smithi163.stdout: filter_policy: bloomfilter 2024-07-26T06:40:14.260 INFO:journalctl@ceph.mon.b.smithi163.stdout: whole_key_filtering: 1 2024-07-26T06:40:14.260 INFO:journalctl@ceph.mon.b.smithi163.stdout: verify_compression: 0 2024-07-26T06:40:14.260 INFO:journalctl@ceph.mon.b.smithi163.stdout: read_amp_bytes_per_bit: 0 2024-07-26T06:40:14.260 INFO:journalctl@ceph.mon.b.smithi163.stdout: format_version: 5 2024-07-26T06:40:14.260 INFO:journalctl@ceph.mon.b.smithi163.stdout: enable_index_compression: 1 2024-07-26T06:40:14.260 INFO:journalctl@ceph.mon.b.smithi163.stdout: block_align: 0 2024-07-26T06:40:14.260 INFO:journalctl@ceph.mon.b.smithi163.stdout: max_auto_readahead_size: 262144 2024-07-26T06:40:14.260 INFO:journalctl@ceph.mon.b.smithi163.stdout: prepopulate_block_cache: 0 2024-07-26T06:40:14.260 INFO:journalctl@ceph.mon.b.smithi163.stdout: initial_auto_readahead_size: 8192 2024-07-26T06:40:14.261 INFO:journalctl@ceph.mon.b.smithi163.stdout: num_file_reads_for_auto_readahead: 2 2024-07-26T06:40:14.261 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.write_buffer_size: 33554432 2024-07-26T06:40:14.261 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.max_write_buffer_number: 2 2024-07-26T06:40:14.261 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compression: NoCompression 2024-07-26T06:40:14.261 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.bottommost_compression: Disabled 2024-07-26T06:40:14.261 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.prefix_extractor: nullptr 2024-07-26T06:40:14.261 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-07-26T06:40:14.261 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.num_levels: 7 2024-07-26T06:40:14.261 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-07-26T06:40:14.261 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-07-26T06:40:14.261 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-07-26T06:40:14.261 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-07-26T06:40:14.261 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-07-26T06:40:14.262 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-07-26T06:40:14.262 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-07-26T06:40:14.262 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-07-26T06:40:14.262 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-07-26T06:40:14.262 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-07-26T06:40:14.262 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-07-26T06:40:14.262 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-07-26T06:40:14.262 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compression_opts.window_bits: -14 2024-07-26T06:40:14.262 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compression_opts.level: 32767 2024-07-26T06:40:14.262 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compression_opts.strategy: 0 2024-07-26T06:40:14.262 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-07-26T06:40:14.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-07-26T06:40:14.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-07-26T06:40:14.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-07-26T06:40:14.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compression_opts.enabled: false 2024-07-26T06:40:14.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-07-26T06:40:14.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-07-26T06:40:14.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-07-26T06:40:14.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-07-26T06:40:14.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.target_file_size_base: 67108864 2024-07-26T06:40:14.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.target_file_size_multiplier: 1 2024-07-26T06:40:14.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-07-26T06:40:14.263 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-07-26T06:40:14.264 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-07-26T06:40:14.264 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-07-26T06:40:14.264 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-07-26T06:40:14.264 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-07-26T06:40:14.264 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-07-26T06:40:14.264 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-07-26T06:40:14.264 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-07-26T06:40:14.264 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-07-26T06:40:14.264 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-07-26T06:40:14.265 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-07-26T06:40:14.265 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-07-26T06:40:14.265 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.arena_block_size: 1048576 2024-07-26T06:40:14.266 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-07-26T06:40:14.266 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-07-26T06:40:14.266 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.disable_auto_compactions: 0 2024-07-26T06:40:14.266 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-07-26T06:40:14.266 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-07-26T06:40:14.266 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-07-26T06:40:14.266 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-07-26T06:40:14.266 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-07-26T06:40:14.266 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-07-26T06:40:14.266 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-07-26T06:40:14.266 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-07-26T06:40:14.266 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-07-26T06:40:14.267 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-07-26T06:40:14.267 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-07-26T06:40:14.267 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.inplace_update_support: 0 2024-07-26T06:40:14.267 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.inplace_update_num_locks: 10000 2024-07-26T06:40:14.267 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-07-26T06:40:14.267 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-07-26T06:40:14.267 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.memtable_huge_page_size: 0 2024-07-26T06:40:14.267 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.bloom_locality: 0 2024-07-26T06:40:14.267 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.max_successive_merges: 0 2024-07-26T06:40:14.267 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.optimize_filters_for_hits: 0 2024-07-26T06:40:14.267 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.paranoid_file_checks: 0 2024-07-26T06:40:14.267 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.force_consistency_checks: 1 2024-07-26T06:40:14.268 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.report_bg_io_stats: 0 2024-07-26T06:40:14.268 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.ttl: 2592000 2024-07-26T06:40:14.268 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.periodic_compaction_seconds: 0 2024-07-26T06:40:14.268 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-07-26T06:40:14.268 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-07-26T06:40:14.268 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.enable_blob_files: false 2024-07-26T06:40:14.268 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.min_blob_size: 0 2024-07-26T06:40:14.268 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.blob_file_size: 268435456 2024-07-26T06:40:14.268 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.blob_compression_type: NoCompression 2024-07-26T06:40:14.268 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.enable_blob_garbage_collection: false 2024-07-26T06:40:14.268 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-07-26T06:40:14.268 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-07-26T06:40:14.268 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-07-26T06:40:14.269 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.blob_file_starting_level: 0 2024-07-26T06:40:14.269 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-07-26T06:40:14.269 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-07-26T06:40:14.269 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-07-26T06:40:14.269 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fcaa4651-fa26-4157-a68d-390ff283d3cc 2024-07-26T06:40:14.269 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721976013981288, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-07-26T06:40:14.269 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-07-26T06:40:14.269 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721976013981725, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1721976013, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fcaa4651-fa26-4157-a68d-390ff283d3cc", "db_session_id": "2XTVIYI0DLOJ8PAUZMDH", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-07-26T06:40:14.269 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721976013981820, "job": 1, "event": "recovery_finished"} 2024-07-26T06:40:14.269 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-07-26T06:40:14.269 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-07-26T06:40:14.269 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55875803ca80 2024-07-26T06:40:14.270 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: DB pointer 0x558758ddc000 2024-07-26T06:40:14.270 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-07-26T06:40:14.270 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-07-26T06:40:14.270 INFO:journalctl@ceph.mon.b.smithi163.stdout: ** DB Stats ** 2024-07-26T06:40:14.270 INFO:journalctl@ceph.mon.b.smithi163.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T06:40:14.270 INFO:journalctl@ceph.mon.b.smithi163.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-07-26T06:40:14.270 INFO:journalctl@ceph.mon.b.smithi163.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T06:40:14.270 INFO:journalctl@ceph.mon.b.smithi163.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T06:40:14.270 INFO:journalctl@ceph.mon.b.smithi163.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-07-26T06:40:14.270 INFO:journalctl@ceph.mon.b.smithi163.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T06:40:14.270 INFO:journalctl@ceph.mon.b.smithi163.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T06:40:14.271 INFO:journalctl@ceph.mon.b.smithi163.stdout: 2024-07-26T06:40:14.271 INFO:journalctl@ceph.mon.b.smithi163.stdout: ** Compaction Stats [default] ** 2024-07-26T06:40:14.271 INFO:journalctl@ceph.mon.b.smithi163.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T06:40:14.271 INFO:journalctl@ceph.mon.b.smithi163.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-07-26T06:40:14.271 INFO:journalctl@ceph.mon.b.smithi163.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T06:40:14.271 INFO:journalctl@ceph.mon.b.smithi163.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T06:40:14.271 INFO:journalctl@ceph.mon.b.smithi163.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T06:40:14.271 INFO:journalctl@ceph.mon.b.smithi163.stdout: 2024-07-26T06:40:14.271 INFO:journalctl@ceph.mon.b.smithi163.stdout: ** Compaction Stats [default] ** 2024-07-26T06:40:14.271 INFO:journalctl@ceph.mon.b.smithi163.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T06:40:14.271 INFO:journalctl@ceph.mon.b.smithi163.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-07-26T06:40:14.272 INFO:journalctl@ceph.mon.b.smithi163.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T06:40:14.272 INFO:journalctl@ceph.mon.b.smithi163.stdout: 2024-07-26T06:40:14.272 INFO:journalctl@ceph.mon.b.smithi163.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-07-26T06:40:14.272 INFO:journalctl@ceph.mon.b.smithi163.stdout: 2024-07-26T06:40:14.272 INFO:journalctl@ceph.mon.b.smithi163.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T06:40:14.272 INFO:journalctl@ceph.mon.b.smithi163.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-07-26T06:40:14.272 INFO:journalctl@ceph.mon.b.smithi163.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-07-26T06:40:14.272 INFO:journalctl@ceph.mon.b.smithi163.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-07-26T06:40:14.272 INFO:journalctl@ceph.mon.b.smithi163.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-07-26T06:40:14.272 INFO:journalctl@ceph.mon.b.smithi163.stdout: AddFile(Keys): cumulative 0, interval 0 2024-07-26T06:40:14.272 INFO:journalctl@ceph.mon.b.smithi163.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T06:40:14.272 INFO:journalctl@ceph.mon.b.smithi163.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T06:40:14.273 INFO:journalctl@ceph.mon.b.smithi163.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-07-26T06:40:14.273 INFO:journalctl@ceph.mon.b.smithi163.stdout: Block cache BinnedLRUCache@0x558753fdf610#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-07-26T06:40:14.273 INFO:journalctl@ceph.mon.b.smithi163.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-07-26T06:40:14.273 INFO:journalctl@ceph.mon.b.smithi163.stdout: 2024-07-26T06:40:14.273 INFO:journalctl@ceph.mon.b.smithi163.stdout: ** File Read Latency Histogram By Level [default] ** 2024-07-26T06:40:14.273 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-07-26T06:40:14.273 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: using public_addr v2:172.21.15.163:0/0 -> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-07-26T06:40:14.273 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: starting mon.b rank -1 at public addrs [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] at bind addrs [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:40:14.273 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:13 smithi163 ceph-mon[36873]: mon.b@-1(???) e0 preinit fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:40:14.273 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: mon.b@-1(synchronizing).mds e1 new map 2024-07-26T06:40:14.273 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: mon.b@-1(synchronizing).mds e1 print_map 2024-07-26T06:40:14.273 INFO:journalctl@ceph.mon.b.smithi163.stdout: e1 2024-07-26T06:40:14.274 INFO:journalctl@ceph.mon.b.smithi163.stdout: btime 2024-07-26T06:38:29:306299+0000 2024-07-26T06:40:14.274 INFO:journalctl@ceph.mon.b.smithi163.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T06:40:14.274 INFO:journalctl@ceph.mon.b.smithi163.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T06:40:14.274 INFO:journalctl@ceph.mon.b.smithi163.stdout: legacy client fscid: -1 2024-07-26T06:40:14.274 INFO:journalctl@ceph.mon.b.smithi163.stdout: 2024-07-26T06:40:14.274 INFO:journalctl@ceph.mon.b.smithi163.stdout: No filesystems configured 2024-07-26T06:40:14.274 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-07-26T06:40:14.274 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-07-26T06:40:14.274 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-07-26T06:40:14.274 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-07-26T06:40:14.274 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-07-26T06:40:14.274 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-07-26T06:40:14.274 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-07-26T06:40:14.275 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T06:40:14.275 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T06:40:14.275 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T06:40:14.275 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:14.275 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:14.275 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: Added host smithi164 2024-07-26T06:40:14.275 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:14.275 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:14.275 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:14.275 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:14.275 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:14.275 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:14.275 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:40:14.276 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:14.276 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:14.276 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:14.276 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:14.276 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:40:14.276 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:14.276 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:14.276 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: Updating smithi164:/etc/ceph/ceph.conf 2024-07-26T06:40:14.276 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:14.276 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: Updating smithi164:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T06:40:14.276 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-07-26T06:40:14.276 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: Updating smithi164:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T06:40:14.276 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:14.277 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:14.277 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:14.277 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/4180670100' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-07-26T06:40:14.277 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/4180670100' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-07-26T06:40:14.277 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T06:40:14.277 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:14.277 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:14.277 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi042:172.21.15.42=a;smithi163:172.21.15.163=b;smithi164:172.21.15.164=c", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:40:14.277 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: Saving service mon spec with placement smithi042:172.21.15.42=a;smithi163:172.21.15.163=b;smithi164:172.21.15.164=c;count:3 2024-07-26T06:40:14.277 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:14.277 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:14.277 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:14.277 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:14.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:14.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T06:40:14.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:14.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: Deploying daemon mon.c on smithi164 2024-07-26T06:40:14.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:14.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:14.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='client.? 172.21.15.164:0/3173483381' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T06:40:14.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:40:14.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:14.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: mon.a calling monitor election 2024-07-26T06:40:14.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:14.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:14.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:14.278 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: mon.c calling monitor election 2024-07-26T06:40:14.279 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:14.279 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:14.279 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:14.279 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:14.279 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:14.279 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T06:40:14.279 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: monmap epoch 2 2024-07-26T06:40:14.279 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:40:14.279 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: last_changed 2024-07-26T06:40:05.131205+0000 2024-07-26T06:40:14.279 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: created 2024-07-26T06:38:26.468434+0000 2024-07-26T06:40:14.279 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: min_mon_release 19 (squid) 2024-07-26T06:40:14.279 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: election_strategy: 1 2024-07-26T06:40:14.279 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: 0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-07-26T06:40:14.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: 1: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.c 2024-07-26T06:40:14.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: fsmap 2024-07-26T06:40:14.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T06:40:14.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: mgrmap e13: x(active, since 48s) 2024-07-26T06:40:14.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: overall HEALTH_OK 2024-07-26T06:40:14.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:14.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:14.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:14.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T06:40:14.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:14.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: Deploying daemon mon.b on smithi163 2024-07-26T06:40:14.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='client.? 172.21.15.164:0/197110018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T06:40:14.280 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:14.281 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:14.281 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 ceph-mon[36873]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-07-26T06:40:14.281 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 bash[36854]: 99821459e73a76fb088548ad0b3cf9b69623041aa22e564e7b2e201bafdce28c 2024-07-26T06:40:14.281 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:14 smithi163 systemd[1]: Started Ceph mon.b for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:40:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:40:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: mon.a calling monitor election 2024-07-26T06:40:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: mon.c calling monitor election 2024-07-26T06:40:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: mon.b calling monitor election 2024-07-26T06:40:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:19.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T06:40:19.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: monmap epoch 3 2024-07-26T06:40:19.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:40:19.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: last_changed 2024-07-26T06:40:14.009959+0000 2024-07-26T06:40:19.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: created 2024-07-26T06:38:26.468434+0000 2024-07-26T06:40:19.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: min_mon_release 19 (squid) 2024-07-26T06:40:19.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: election_strategy: 1 2024-07-26T06:40:19.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: 0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-07-26T06:40:19.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: 1: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.c 2024-07-26T06:40:19.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: 2: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-07-26T06:40:19.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: fsmap 2024-07-26T06:40:19.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T06:40:19.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: mgrmap e13: x(active, since 57s) 2024-07-26T06:40:19.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: overall HEALTH_OK 2024-07-26T06:40:19.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:19.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:19.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:19.399 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:40:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: mon.a calling monitor election 2024-07-26T06:40:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: mon.c calling monitor election 2024-07-26T06:40:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: mon.b calling monitor election 2024-07-26T06:40:19.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:19.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T06:40:19.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: monmap epoch 3 2024-07-26T06:40:19.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:40:19.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: last_changed 2024-07-26T06:40:14.009959+0000 2024-07-26T06:40:19.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: created 2024-07-26T06:38:26.468434+0000 2024-07-26T06:40:19.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: min_mon_release 19 (squid) 2024-07-26T06:40:19.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: election_strategy: 1 2024-07-26T06:40:19.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: 0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-07-26T06:40:19.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: 1: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.c 2024-07-26T06:40:19.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: 2: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-07-26T06:40:19.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: fsmap 2024-07-26T06:40:19.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T06:40:19.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: mgrmap e13: x(active, since 57s) 2024-07-26T06:40:19.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: overall HEALTH_OK 2024-07-26T06:40:19.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:19.476 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:19.476 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:19.476 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:40:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: mon.a calling monitor election 2024-07-26T06:40:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:40:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: mon.c calling monitor election 2024-07-26T06:40:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: mon.b calling monitor election 2024-07-26T06:40:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T06:40:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: monmap epoch 3 2024-07-26T06:40:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:40:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: last_changed 2024-07-26T06:40:14.009959+0000 2024-07-26T06:40:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: created 2024-07-26T06:38:26.468434+0000 2024-07-26T06:40:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: min_mon_release 19 (squid) 2024-07-26T06:40:19.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: election_strategy: 1 2024-07-26T06:40:19.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: 0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-07-26T06:40:19.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: 1: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.c 2024-07-26T06:40:19.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: 2: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.b 2024-07-26T06:40:19.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: fsmap 2024-07-26T06:40:19.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T06:40:19.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: mgrmap e13: x(active, since 57s) 2024-07-26T06:40:19.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: overall HEALTH_OK 2024-07-26T06:40:19.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:19.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:19.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:19.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:20 smithi163 ceph-mon[36873]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:20.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:20.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:20 smithi042 ceph-mon[32775]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:20.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:20 smithi164 ceph-mon[36970]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:40:20.623 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-07-26T06:40:20.624 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":3,"fsid":"8b3e2750-4b19-11ef-bcb6-c7b262605968","modified":"2024-07-26T06:40:14.009959Z","created":"2024-07-26T06:38:26.468434Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-07-26T06:40:20.624 INFO:teuthology.orchestra.run.smithi164.stderr:dumped monmap epoch 3 2024-07-26T06:40:21.172 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-07-26T06:40:21.172 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config generate-minimal-conf 2024-07-26T06:40:21.470 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.470 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.470 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:21.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:21.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: Updating smithi042:/etc/ceph/ceph.conf 2024-07-26T06:40:21.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: Updating smithi163:/etc/ceph/ceph.conf 2024-07-26T06:40:21.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: Updating smithi164:/etc/ceph/ceph.conf 2024-07-26T06:40:21.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: Updating smithi163:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T06:40:21.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: Updating smithi164:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T06:40:21.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: Updating smithi042:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T06:40:21.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='client.? 172.21.15.164:0/1782888992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T06:40:21.471 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.472 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.472 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.472 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.472 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.472 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.472 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.472 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.472 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.472 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.472 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.472 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T06:40:21.472 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T06:40:21.472 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T06:40:21.472 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:21.472 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:21 smithi042 ceph-mon[32775]: Reconfiguring daemon mon.a on smithi042 2024-07-26T06:40:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: Updating smithi042:/etc/ceph/ceph.conf 2024-07-26T06:40:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: Updating smithi163:/etc/ceph/ceph.conf 2024-07-26T06:40:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: Updating smithi164:/etc/ceph/ceph.conf 2024-07-26T06:40:21.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: Updating smithi163:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T06:40:21.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: Updating smithi164:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T06:40:21.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: Updating smithi042:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T06:40:21.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='client.? 172.21.15.164:0/1782888992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T06:40:21.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T06:40:21.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T06:40:21.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T06:40:21.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:21.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:21 smithi164 ceph-mon[36970]: Reconfiguring daemon mon.a on smithi042 2024-07-26T06:40:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: Updating smithi042:/etc/ceph/ceph.conf 2024-07-26T06:40:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: Updating smithi163:/etc/ceph/ceph.conf 2024-07-26T06:40:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: Updating smithi164:/etc/ceph/ceph.conf 2024-07-26T06:40:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: Updating smithi163:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T06:40:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: Updating smithi164:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T06:40:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: Updating smithi042:/var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/config/ceph.conf 2024-07-26T06:40:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='client.? 172.21.15.164:0/1782888992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T06:40:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:21.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T06:40:21.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T06:40:21.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T06:40:21.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:21.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:21 smithi163 ceph-mon[36873]: Reconfiguring daemon mon.a on smithi042 2024-07-26T06:40:22.261 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:22.543 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:22 smithi042 ceph-mon[32775]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:22.543 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:22.669 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:22 smithi163 ceph-mon[36873]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:22.669 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:22 smithi164 ceph-mon[36970]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:23.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:23.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:23.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:23 smithi042 ceph-mon[32775]: Reconfiguring mon.b (monmap changed)... 2024-07-26T06:40:23.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T06:40:23.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T06:40:23.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:23.372 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:23 smithi042 ceph-mon[32775]: Reconfiguring daemon mon.b on smithi163 2024-07-26T06:40:23.634 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:23.635 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:23.635 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:23 smithi163 ceph-mon[36873]: Reconfiguring mon.b (monmap changed)... 2024-07-26T06:40:23.635 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T06:40:23.635 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T06:40:23.635 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:23.635 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:23 smithi163 ceph-mon[36873]: Reconfiguring daemon mon.b on smithi163 2024-07-26T06:40:23.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:23.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:23.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:23 smithi164 ceph-mon[36970]: Reconfiguring mon.b (monmap changed)... 2024-07-26T06:40:23.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T06:40:23.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T06:40:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:23 smithi164 ceph-mon[36970]: Reconfiguring daemon mon.b on smithi163 2024-07-26T06:40:23.817 INFO:teuthology.orchestra.run.smithi042.stdout:# minimal ceph.conf for 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:40:23.817 INFO:teuthology.orchestra.run.smithi042.stdout:[global] 2024-07-26T06:40:23.817 INFO:teuthology.orchestra.run.smithi042.stdout: fsid = 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:40:23.817 INFO:teuthology.orchestra.run.smithi042.stdout: mon_host = [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-07-26T06:40:24.385 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-07-26T06:40:24.385 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-07-26T06:40:24.385 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T06:40:24.415 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-07-26T06:40:24.415 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T06:40:24.482 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:40:24.483 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T06:40:24.514 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:40:24.515 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T06:40:24.582 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:40:24.582 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T06:40:24.609 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:40:24.610 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T06:40:24.674 INFO:tasks.cephadm:Adding mgr.x on smithi042 2024-07-26T06:40:24.674 INFO:tasks.cephadm:Adding mgr.y on smithi163 2024-07-26T06:40:24.674 INFO:tasks.cephadm:Adding mgr.z on smithi164 2024-07-26T06:40:24.674 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch apply mgr '3;smithi042=x;smithi163=y;smithi164=z' 2024-07-26T06:40:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:24 smithi164 ceph-mon[36970]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:24 smithi164 ceph-mon[36970]: Reconfiguring mon.c (monmap changed)... 2024-07-26T06:40:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T06:40:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T06:40:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:24 smithi164 ceph-mon[36970]: Reconfiguring daemon mon.c on smithi164 2024-07-26T06:40:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:24 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2709296894' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:24 smithi163 ceph-mon[36873]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:24 smithi163 ceph-mon[36873]: Reconfiguring mon.c (monmap changed)... 2024-07-26T06:40:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T06:40:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T06:40:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:24 smithi163 ceph-mon[36873]: Reconfiguring daemon mon.c on smithi164 2024-07-26T06:40:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:24 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2709296894' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:25.063 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T06:40:25.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:24 smithi042 ceph-mon[32775]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:24 smithi042 ceph-mon[32775]: Reconfiguring mon.c (monmap changed)... 2024-07-26T06:40:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T06:40:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T06:40:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:24 smithi042 ceph-mon[32775]: Reconfiguring daemon mon.c on smithi164 2024-07-26T06:40:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:24 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2709296894' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:26.352 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:26.352 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:26.353 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:26.353 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:26 smithi164 ceph-mon[36970]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:26.353 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:26.353 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:26.353 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:26.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:26 smithi042 ceph-mon[32775]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:26 smithi163 ceph-mon[36873]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:26.735 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled mgr update... 2024-07-26T06:40:27.336 DEBUG:teuthology.orchestra.run.smithi163:mgr.y> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mgr.y.service 2024-07-26T06:40:27.339 DEBUG:teuthology.orchestra.run.smithi164:mgr.z> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mgr.z.service 2024-07-26T06:40:27.342 INFO:tasks.cephadm:setting config [client] client_mount_timeout = 600 2024-07-26T06:40:27.342 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set client client_mount_timeout 600 2024-07-26T06:40:27.345 INFO:tasks.cephadm:setting config [client] debug_client = 20 2024-07-26T06:40:27.345 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set client debug_client 20 2024-07-26T06:40:27.387 INFO:tasks.cephadm:setting config [client] debug_ms = 1 2024-07-26T06:40:27.387 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set client debug_ms 1 2024-07-26T06:40:27.431 INFO:tasks.cephadm:setting config [client] rados_mon_op_timeout = 900 2024-07-26T06:40:27.431 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set client rados_mon_op_timeout 900 2024-07-26T06:40:27.475 INFO:tasks.cephadm:setting config [client] rados_osd_op_timeout = 900 2024-07-26T06:40:27.476 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set client rados_osd_op_timeout 900 2024-07-26T06:40:27.519 INFO:tasks.cephadm:setting config [mds] debug_mds = 20 2024-07-26T06:40:27.519 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set mds debug_mds 20 2024-07-26T06:40:27.562 INFO:tasks.cephadm:setting config [mds] debug_mds_balancer = 20 2024-07-26T06:40:27.563 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set mds debug_mds_balancer 20 2024-07-26T06:40:27.605 INFO:tasks.cephadm:setting config [mds] debug_ms = 2 2024-07-26T06:40:27.606 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set mds debug_ms 2 2024-07-26T06:40:27.650 INFO:tasks.cephadm:setting config [mds] mds_bal_fragment_size_max = 10000 2024-07-26T06:40:27.650 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set mds mds_bal_fragment_size_max 10000 2024-07-26T06:40:27.683 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:27.687 INFO:tasks.cephadm:setting config [mds] mds_bal_merge_size = 5 2024-07-26T06:40:27.687 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set mds mds_bal_merge_size 5 2024-07-26T06:40:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:27.723 INFO:tasks.cephadm:setting config [mds] mds_bal_split_bits = 3 2024-07-26T06:40:27.723 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set mds mds_bal_split_bits 3 2024-07-26T06:40:27.767 INFO:tasks.cephadm:setting config [mds] mds_bal_split_size = 100 2024-07-26T06:40:27.767 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set mds mds_bal_split_size 100 2024-07-26T06:40:27.811 INFO:tasks.cephadm:setting config [mds] mds_debug_frag = True 2024-07-26T06:40:27.811 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set mds mds_debug_frag True 2024-07-26T06:40:27.861 INFO:tasks.cephadm:setting config [mds] mds_debug_scatterstat = True 2024-07-26T06:40:27.861 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set mds mds_debug_scatterstat True 2024-07-26T06:40:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:27 smithi042 ceph-mon[32775]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi042=x;smithi163=y;smithi164=z", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:40:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:27 smithi042 ceph-mon[32775]: Saving service mgr spec with placement smithi042=x;smithi163=y;smithi164=z;count:3 2024-07-26T06:40:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T06:40:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T06:40:27.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T06:40:27.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:27.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:27 smithi042 ceph-mon[32775]: Deploying daemon mgr.z on smithi164 2024-07-26T06:40:27.898 INFO:tasks.cephadm:setting config [mds] mds_op_complaint_time = 180 2024-07-26T06:40:27.898 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set mds mds_op_complaint_time 180 2024-07-26T06:40:27.943 INFO:tasks.cephadm:setting config [mds] mds_verify_scatter = True 2024-07-26T06:40:27.943 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set mds mds_verify_scatter True 2024-07-26T06:40:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:27 smithi164 ceph-mon[36970]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi042=x;smithi163=y;smithi164=z", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:40:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:27 smithi164 ceph-mon[36970]: Saving service mgr spec with placement smithi042=x;smithi163=y;smithi164=z;count:3 2024-07-26T06:40:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T06:40:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T06:40:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T06:40:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:27 smithi164 ceph-mon[36970]: Deploying daemon mgr.z on smithi164 2024-07-26T06:40:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:27 smithi163 ceph-mon[36873]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi042=x;smithi163=y;smithi164=z", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:40:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:27 smithi163 ceph-mon[36873]: Saving service mgr spec with placement smithi042=x;smithi163=y;smithi164=z;count:3 2024-07-26T06:40:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T06:40:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T06:40:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T06:40:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:27 smithi163 ceph-mon[36873]: Deploying daemon mgr.z on smithi164 2024-07-26T06:40:28.008 INFO:tasks.cephadm:setting config [mds] osd_op_complaint_time = 180 2024-07-26T06:40:28.008 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set mds osd_op_complaint_time 180 2024-07-26T06:40:28.052 INFO:tasks.cephadm:setting config [mds] rados_mon_op_timeout = 900 2024-07-26T06:40:28.052 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set mds rados_mon_op_timeout 900 2024-07-26T06:40:28.107 INFO:tasks.cephadm:setting config [mds] rados_osd_op_timeout = 900 2024-07-26T06:40:28.107 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set mds rados_osd_op_timeout 900 2024-07-26T06:40:28.163 INFO:tasks.cephadm:setting config [mon] mon_op_complaint_time = 120 2024-07-26T06:40:28.163 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set mon mon_op_complaint_time 120 2024-07-26T06:40:28.207 INFO:tasks.cephadm:setting config [mon] mon_warn_on_pool_no_app = False 2024-07-26T06:40:28.207 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set mon mon_warn_on_pool_no_app False 2024-07-26T06:40:28.251 INFO:tasks.cephadm:setting config [osd] osd_op_complaint_time = 180 2024-07-26T06:40:28.251 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config set osd osd_op_complaint_time 180 2024-07-26T06:40:28.304 DEBUG:tasks.cephadm:set 22 configs 2024-07-26T06:40:28.304 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi042') 2024-07-26T06:40:28.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:28 smithi042 ceph-mon[32775]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:29.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:28 smithi163 ceph-mon[36873]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:29.041 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:28 smithi164 ceph-mon[36970]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:29.041 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Jul 26 06:40:28 smithi164 systemd[1]: Starting Ceph mgr.z for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:40:29.291 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Jul 26 06:40:29 smithi164 podman[38584]: 2024-07-26 06:40:28.942911833 +0000 UTC m=+0.016922950 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:40:29.292 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Jul 26 06:40:29 smithi164 podman[38584]: 2024-07-26 06:40:29.19066394 +0000 UTC m=+0.264675100 container create d1a92d8c9a1a5e7cb57123355ee1fca55cdb55dbaec277346f7be6297a180afb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-z, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-07-26T06:40:29.667 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:29.722 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Jul 26 06:40:29 smithi164 podman[38584]: 2024-07-26 06:40:29.390494522 +0000 UTC m=+0.464505639 container init d1a92d8c9a1a5e7cb57123355ee1fca55cdb55dbaec277346f7be6297a180afb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-07-26T06:40:29.722 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Jul 26 06:40:29 smithi164 podman[38584]: 2024-07-26 06:40:29.392956441 +0000 UTC m=+0.466967558 container start d1a92d8c9a1a5e7cb57123355ee1fca55cdb55dbaec277346f7be6297a180afb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-z, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-07-26T06:40:29.723 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Jul 26 06:40:29 smithi164 ceph-mgr[38603]: -- 172.21.15.164:0/1568674820 <== mon.1 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56323e6c30e0 con 0x563243462000 2024-07-26T06:40:29.723 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Jul 26 06:40:29 smithi164 bash[38584]: d1a92d8c9a1a5e7cb57123355ee1fca55cdb55dbaec277346f7be6297a180afb 2024-07-26T06:40:29.723 INFO:journalctl@ceph.mgr.z.smithi164.stdout:Jul 26 06:40:29 smithi164 systemd[1]: Started Ceph mgr.z for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:40:30.099 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:30.101 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:30.103 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:30.104 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:30.105 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:30.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:30 smithi042 ceph-mon[32775]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:30.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:30.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:30.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:30.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T06:40:30.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T06:40:30.602 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T06:40:30.602 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:30.602 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:30 smithi042 ceph-mon[32775]: Deploying daemon mgr.y on smithi163 2024-07-26T06:40:30.808 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:30 smithi164 ceph-mon[36970]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:30.808 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:30.808 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:30.808 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:30.808 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T06:40:30.808 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T06:40:30.809 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T06:40:30.809 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:30.809 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:30 smithi164 ceph-mon[36970]: Deploying daemon mgr.y on smithi163 2024-07-26T06:40:30.815 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:30 smithi163 ceph-mon[36873]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:30.815 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:30.815 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:30.815 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:30.815 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T06:40:30.815 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T06:40:30.815 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T06:40:30.815 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:30.815 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:30 smithi163 ceph-mon[36873]: Deploying daemon mgr.y on smithi163 2024-07-26T06:40:31.507 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jul 26 06:40:31 smithi163 podman[38031]: 2024-07-26 06:40:31.44229778 +0000 UTC m=+0.121679476 container create b8a91b052f0c0a978377e6285534b11b45afa9a0639f882aceb229cf25a5c392 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-y, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-07-26T06:40:32.007 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jul 26 06:40:31 smithi163 podman[38031]: 2024-07-26 06:40:31.677675395 +0000 UTC m=+0.357057090 container init b8a91b052f0c0a978377e6285534b11b45afa9a0639f882aceb229cf25a5c392 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD) 2024-07-26T06:40:32.007 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jul 26 06:40:31 smithi163 podman[38031]: 2024-07-26 06:40:31.680239818 +0000 UTC m=+0.359621514 container start b8a91b052f0c0a978377e6285534b11b45afa9a0639f882aceb229cf25a5c392 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-y, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-07-26T06:40:32.007 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jul 26 06:40:31 smithi163 ceph-mgr[38051]: -- 172.21.15.163:0/1027531690 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dc9cda50e0 con 0x55dc9cdc3a80 2024-07-26T06:40:32.007 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jul 26 06:40:31 smithi163 bash[38031]: b8a91b052f0c0a978377e6285534b11b45afa9a0639f882aceb229cf25a5c392 2024-07-26T06:40:32.007 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Jul 26 06:40:31 smithi163 systemd[1]: Started Ceph mgr.y for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:40:32.671 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:32.671 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:32.672 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:32.678 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:32.681 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:33.003 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:32 smithi042 ceph-mon[32775]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:33.004 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:33.004 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:33.004 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:33.004 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:33.004 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:32 smithi164 ceph-mon[36970]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:33.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:32 smithi163 ceph-mon[36873]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:33.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:33.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:33.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:33.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:33.232 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T06:40:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T06:40:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T06:40:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T06:40:34.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:34.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T06:40:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T06:40:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:35.053 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:34 smithi042 ceph-mon[32775]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T06:40:35.053 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:34 smithi042 ceph-mon[32775]: Reconfiguring daemon mgr.x on smithi042 2024-07-26T06:40:35.053 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:34 smithi042 ceph-mon[32775]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:35.053 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:34 smithi042 ceph-mon[32775]: Standby manager daemon z started 2024-07-26T06:40:35.053 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:34 smithi042 ceph-mon[32775]: from='mgr.? 172.21.15.164:0/1065363416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T06:40:35.053 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:34 smithi042 ceph-mon[32775]: from='mgr.? 172.21.15.164:0/1065363416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T06:40:35.053 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:34 smithi042 ceph-mon[32775]: from='mgr.? 172.21.15.164:0/1065363416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T06:40:35.054 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:34 smithi042 ceph-mon[32775]: from='mgr.? 172.21.15.164:0/1065363416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T06:40:35.054 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:34 smithi042 ceph-mon[32775]: from='client.? ' entity='client.admin' 2024-07-26T06:40:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:34 smithi164 ceph-mon[36970]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T06:40:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:34 smithi164 ceph-mon[36970]: Reconfiguring daemon mgr.x on smithi042 2024-07-26T06:40:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:34 smithi164 ceph-mon[36970]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:34 smithi164 ceph-mon[36970]: Standby manager daemon z started 2024-07-26T06:40:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:34 smithi164 ceph-mon[36970]: from='mgr.? 172.21.15.164:0/1065363416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T06:40:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:34 smithi164 ceph-mon[36970]: from='mgr.? 172.21.15.164:0/1065363416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T06:40:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:34 smithi164 ceph-mon[36970]: from='mgr.? 172.21.15.164:0/1065363416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T06:40:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:34 smithi164 ceph-mon[36970]: from='mgr.? 172.21.15.164:0/1065363416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T06:40:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:34 smithi164 ceph-mon[36970]: from='client.? ' entity='client.admin' 2024-07-26T06:40:35.223 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:35.224 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:35.224 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:35.226 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:35.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:34 smithi163 ceph-mon[36873]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T06:40:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:34 smithi163 ceph-mon[36873]: Reconfiguring daemon mgr.x on smithi042 2024-07-26T06:40:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:34 smithi163 ceph-mon[36873]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:34 smithi163 ceph-mon[36873]: Standby manager daemon z started 2024-07-26T06:40:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:34 smithi163 ceph-mon[36873]: from='mgr.? 172.21.15.164:0/1065363416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T06:40:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:34 smithi163 ceph-mon[36873]: from='mgr.? 172.21.15.164:0/1065363416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T06:40:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:34 smithi163 ceph-mon[36873]: from='mgr.? 172.21.15.164:0/1065363416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T06:40:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:34 smithi163 ceph-mon[36873]: from='mgr.? 172.21.15.164:0/1065363416' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T06:40:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:34 smithi163 ceph-mon[36873]: from='client.? ' entity='client.admin' 2024-07-26T06:40:35.470 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:35.476 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:35.477 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:36 smithi042 ceph-mon[32775]: mgrmap e14: x(active, since 73s), standbys: z 2024-07-26T06:40:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T06:40:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:36 smithi042 ceph-mon[32775]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:36 smithi164 ceph-mon[36970]: mgrmap e14: x(active, since 73s), standbys: z 2024-07-26T06:40:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T06:40:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:36 smithi164 ceph-mon[36970]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:36 smithi163 ceph-mon[36873]: mgrmap e14: x(active, since 73s), standbys: z 2024-07-26T06:40:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T06:40:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:36 smithi163 ceph-mon[36873]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:36.753 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:37.460+0000 7fd4a1732640 1 -- 172.21.15.42:0/1634917464 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd484079450 msgr2=0x7fd48407b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:37.460+0000 7fd4a1732640 1 --2- 172.21.15.42:0/1634917464 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd484079450 0x7fd48407b960 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd48c009b50 tx=0x7fd48c005f90 comp rx=0 tx=0).stop 2024-07-26T06:40:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:37.460+0000 7fd4a1732640 1 -- 172.21.15.42:0/1634917464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd49c103100 msgr2=0x7fd49c199d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:37.460+0000 7fd4a1732640 1 --2- 172.21.15.42:0/1634917464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd49c103100 0x7fd49c199d90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd49000eab0 tx=0x7fd49000ef70 comp rx=0 tx=0).stop 2024-07-26T06:40:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:37.460+0000 7fd4a1732640 1 -- 172.21.15.42:0/1634917464 shutdown_connections 2024-07-26T06:40:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:37.460+0000 7fd4a1732640 1 --2- 172.21.15.42:0/1634917464 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd484079450 0x7fd48407b960 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:37.460+0000 7fd4a1732640 1 --2- 172.21.15.42:0/1634917464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd49c103aa0 0x7fd49c19a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:37.460+0000 7fd4a1732640 1 --2- 172.21.15.42:0/1634917464 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd49c103100 0x7fd49c199d90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:37.460+0000 7fd4a1732640 1 --2- 172.21.15.42:0/1634917464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd49c102070 0x7fd49c199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:37.460+0000 7fd4a1732640 1 -- 172.21.15.42:0/1634917464 >> 172.21.15.42:0/1634917464 conn(0x7fd49c0fd9e0 msgr2=0x7fd49c0ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:37.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:37.460+0000 7fd4a1732640 1 -- 172.21.15.42:0/1634917464 shutdown_connections 2024-07-26T06:40:37.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:37.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:37 smithi042 ceph-mon[32775]: Standby manager daemon y started 2024-07-26T06:40:37.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:37 smithi042 ceph-mon[32775]: from='mgr.? 172.21.15.163:0/2373800087' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T06:40:37.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:37 smithi042 ceph-mon[32775]: from='mgr.? 172.21.15.163:0/2373800087' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T06:40:37.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:37 smithi042 ceph-mon[32775]: from='mgr.? 172.21.15.163:0/2373800087' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T06:40:37.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:37 smithi042 ceph-mon[32775]: from='mgr.? 172.21.15.163:0/2373800087' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T06:40:37.462 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:37 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3101759755' entity='client.admin' 2024-07-26T06:40:37.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:37.460+0000 7fd4a1732640 1 -- 172.21.15.42:0/1634917464 wait complete. 2024-07-26T06:40:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:37 smithi164 ceph-mon[36970]: Standby manager daemon y started 2024-07-26T06:40:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:37 smithi164 ceph-mon[36970]: from='mgr.? 172.21.15.163:0/2373800087' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T06:40:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:37 smithi164 ceph-mon[36970]: from='mgr.? 172.21.15.163:0/2373800087' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T06:40:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:37 smithi164 ceph-mon[36970]: from='mgr.? 172.21.15.163:0/2373800087' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T06:40:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:37 smithi164 ceph-mon[36970]: from='mgr.? 172.21.15.163:0/2373800087' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T06:40:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:37 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3101759755' entity='client.admin' 2024-07-26T06:40:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:37 smithi163 ceph-mon[36873]: Standby manager daemon y started 2024-07-26T06:40:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:37 smithi163 ceph-mon[36873]: from='mgr.? 172.21.15.163:0/2373800087' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T06:40:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:37 smithi163 ceph-mon[36873]: from='mgr.? 172.21.15.163:0/2373800087' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T06:40:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:37 smithi163 ceph-mon[36873]: from='mgr.? 172.21.15.163:0/2373800087' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T06:40:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:37 smithi163 ceph-mon[36873]: from='mgr.? 172.21.15.163:0/2373800087' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T06:40:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:37 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3101759755' entity='client.admin' 2024-07-26T06:40:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:38 smithi042 ceph-mon[32775]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:38 smithi042 ceph-mon[32775]: mgrmap e15: x(active, since 76s), standbys: z, y 2024-07-26T06:40:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T06:40:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:38 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1634917464' entity='client.admin' 2024-07-26T06:40:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:38 smithi164 ceph-mon[36970]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:38 smithi164 ceph-mon[36970]: mgrmap e15: x(active, since 76s), standbys: z, y 2024-07-26T06:40:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T06:40:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:38 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1634917464' entity='client.admin' 2024-07-26T06:40:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:38 smithi163 ceph-mon[36873]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:38 smithi163 ceph-mon[36873]: mgrmap e15: x(active, since 76s), standbys: z, y 2024-07-26T06:40:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T06:40:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:38 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1634917464' entity='client.admin' 2024-07-26T06:40:40.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:40 smithi042 ceph-mon[32775]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:40 smithi164 ceph-mon[36970]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:40 smithi163 ceph-mon[36873]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:42.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.216+0000 7f2541fbb640 1 -- 172.21.15.42:0/2450804203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c102700 msgr2=0x7f253c104b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:42.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.216+0000 7f2541fbb640 1 --2- 172.21.15.42:0/2450804203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c102700 0x7f253c104b80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f252c009900 tx=0x7f252c02a410 comp rx=0 tx=0).stop 2024-07-26T06:40:42.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.216+0000 7f2541fbb640 1 -- 172.21.15.42:0/2450804203 shutdown_connections 2024-07-26T06:40:42.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.216+0000 7f2541fbb640 1 --2- 172.21.15.42:0/2450804203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f253c107de0 0x7f253c10a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:42.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.216+0000 7f2541fbb640 1 --2- 172.21.15.42:0/2450804203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f253c1051c0 0x7f253c1076a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.216+0000 7f2541fbb640 1 --2- 172.21.15.42:0/2450804203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c102700 0x7f253c104b80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.216+0000 7f2541fbb640 1 -- 172.21.15.42:0/2450804203 >> 172.21.15.42:0/2450804203 conn(0x7f253c0fd1a0 msgr2=0x7f253c0ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.216+0000 7f2541fbb640 1 -- 172.21.15.42:0/2450804203 shutdown_connections 2024-07-26T06:40:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.216+0000 7f2541fbb640 1 -- 172.21.15.42:0/2450804203 wait complete. 2024-07-26T06:40:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.217+0000 7f2541fbb640 1 Processor -- start 2024-07-26T06:40:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.218+0000 7f2541fbb640 1 -- start start 2024-07-26T06:40:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.218+0000 7f2541fbb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c102700 0x7f253c101c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.218+0000 7f2541fbb640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f253c1051c0 0x7f253c19e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.218+0000 7f2541fbb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f253c107de0 0x7f253c1a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.218+0000 7f2541fbb640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f253c10dac0 con 0x7f253c102700 2024-07-26T06:40:42.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.218+0000 7f2541fbb640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f253c10d940 con 0x7f253c1051c0 2024-07-26T06:40:42.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.218+0000 7f2541fbb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f253c10dc40 con 0x7f253c107de0 2024-07-26T06:40:42.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.218+0000 7f253b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c102700 0x7f253c101c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:42.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.218+0000 7f253b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c102700 0x7f253c101c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50050/0 (socket says 172.21.15.42:50050) 2024-07-26T06:40:42.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.218+0000 7f253b7fe640 1 -- 172.21.15.42:0/2602109909 learned_addr learned my addr 172.21.15.42:0/2602109909 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:40:42.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.218+0000 7f253bfff640 1 --2- 172.21.15.42:0/2602109909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f253c107de0 0x7f253c1a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:42.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.218+0000 7f253affd640 1 --2- 172.21.15.42:0/2602109909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f253c1051c0 0x7f253c19e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:42.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.218+0000 7f253b7fe640 1 -- 172.21.15.42:0/2602109909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f253c107de0 msgr2=0x7f253c1a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:42.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.218+0000 7f253b7fe640 1 --2- 172.21.15.42:0/2602109909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f253c107de0 0x7f253c1a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:42.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.218+0000 7f253b7fe640 1 -- 172.21.15.42:0/2602109909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f253c1051c0 msgr2=0x7f253c19e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:42.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.218+0000 7f253b7fe640 1 --2- 172.21.15.42:0/2602109909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f253c1051c0 0x7f253c19e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:42.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.218+0000 7f253b7fe640 1 -- 172.21.15.42:0/2602109909 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253c1a1350 con 0x7f253c102700 2024-07-26T06:40:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.219+0000 7f253b7fe640 1 --2- 172.21.15.42:0/2602109909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c102700 0x7f253c101c60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f252c006010 tx=0x7f252c003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.219+0000 7f2538ff9640 1 -- 172.21.15.42:0/2602109909 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c03f070 con 0x7f253c102700 2024-07-26T06:40:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.219+0000 7f2541fbb640 1 -- 172.21.15.42:0/2602109909 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253c1a15e0 con 0x7f253c102700 2024-07-26T06:40:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.219+0000 7f2538ff9640 1 -- 172.21.15.42:0/2602109909 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(24 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f252c031cf0 con 0x7f253c102700 2024-07-26T06:40:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.219+0000 7f2538ff9640 1 -- 172.21.15.42:0/2602109909 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c035030 con 0x7f253c102700 2024-07-26T06:40:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.219+0000 7f2541fbb640 1 -- 172.21.15.42:0/2602109909 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f253c1a1ac0 con 0x7f253c102700 2024-07-26T06:40:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.220+0000 7f2541fbb640 1 -- 172.21.15.42:0/2602109909 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f253c105690 con 0x7f253c102700 2024-07-26T06:40:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.222+0000 7f2538ff9640 1 -- 172.21.15.42:0/2602109909 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f252c042070 con 0x7f253c102700 2024-07-26T06:40:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.222+0000 7f2538ff9640 1 --2- 172.21.15.42:0/2602109909 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f25240b3b40 0x7f25240b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.222+0000 7f253affd640 1 --2- 172.21.15.42:0/2602109909 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f25240b3b40 0x7f25240b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.223+0000 7f2538ff9640 1 -- 172.21.15.42:0/2602109909 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f252c103450 con 0x7f253c102700 2024-07-26T06:40:42.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.223+0000 7f253affd640 1 --2- 172.21.15.42:0/2602109909 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f25240b3b40 0x7f25240b6050 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2530005fd0 tx=0x7f2530005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:42.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.225+0000 7f2538ff9640 1 -- 172.21.15.42:0/2602109909 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f252c0cc770 con 0x7f253c102700 2024-07-26T06:40:42.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:42 smithi042 ceph-mon[32775]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:42.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.413+0000 7f2541fbb640 1 -- 172.21.15.42:0/2602109909 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7f253c0fecc0 con 0x7f253c102700 2024-07-26T06:40:42.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.417+0000 7f2538ff9640 1 -- 172.21.15.42:0/2602109909 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v13)=0 v13) ==== 123+0+0 (secure 0 0 0) 0x7f252c0cc070 con 0x7f253c102700 2024-07-26T06:40:42.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.419+0000 7f2538ff9640 1 -- 172.21.15.42:0/2602109909 <== mon.0 v2:172.21.15.42:3300/0 8 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f252c0cbb80 con 0x7f253c102700 2024-07-26T06:40:42.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.420+0000 7f2541fbb640 1 -- 172.21.15.42:0/2602109909 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f25240b3b40 msgr2=0x7f25240b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:42.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.420+0000 7f2541fbb640 1 --2- 172.21.15.42:0/2602109909 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f25240b3b40 0x7f25240b6050 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2530005fd0 tx=0x7f2530005850 comp rx=0 tx=0).stop 2024-07-26T06:40:42.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.420+0000 7f2541fbb640 1 -- 172.21.15.42:0/2602109909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c102700 msgr2=0x7f253c101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:42.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.420+0000 7f2541fbb640 1 --2- 172.21.15.42:0/2602109909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c102700 0x7f253c101c60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f252c006010 tx=0x7f252c003d50 comp rx=0 tx=0).stop 2024-07-26T06:40:42.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.420+0000 7f2541fbb640 1 -- 172.21.15.42:0/2602109909 shutdown_connections 2024-07-26T06:40:42.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.420+0000 7f2541fbb640 1 --2- 172.21.15.42:0/2602109909 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f25240b3b40 0x7f25240b6050 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:42.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.420+0000 7f2541fbb640 1 --2- 172.21.15.42:0/2602109909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f253c107de0 0x7f253c1a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:42.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.420+0000 7f2541fbb640 1 --2- 172.21.15.42:0/2602109909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f253c1051c0 0x7f253c19e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:42.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.420+0000 7f2541fbb640 1 --2- 172.21.15.42:0/2602109909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f253c102700 0x7f253c101c60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:42.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.420+0000 7f2541fbb640 1 -- 172.21.15.42:0/2602109909 >> 172.21.15.42:0/2602109909 conn(0x7f253c0fd1a0 msgr2=0x7f253c108de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:42.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.420+0000 7f2541fbb640 1 -- 172.21.15.42:0/2602109909 shutdown_connections 2024-07-26T06:40:42.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:42.420+0000 7f2541fbb640 1 -- 172.21.15.42:0/2602109909 wait complete. 2024-07-26T06:40:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:42 smithi164 ceph-mon[36970]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:42.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:42 smithi163 ceph-mon[36873]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:43.053 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi042') 2024-07-26T06:40:43.673 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:43 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2602109909' entity='client.admin' 2024-07-26T06:40:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:43 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2602109909' entity='client.admin' 2024-07-26T06:40:43.741 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:43 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2602109909' entity='client.admin' 2024-07-26T06:40:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:44 smithi163 ceph-mon[36873]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:44.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:44 smithi042 ceph-mon[32775]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:44 smithi164 ceph-mon[36970]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:46.343 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:46.343 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:46.343 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:46.343 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:46 smithi042 ceph-mon[32775]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:46.344 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:46.344 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:46.344 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:46.344 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:46 smithi042 ceph-mon[32775]: from='client.? ' entity='client.admin' 2024-07-26T06:40:46.374 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi042') 2024-07-26T06:40:46.375 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi042') 2024-07-26T06:40:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:46 smithi164 ceph-mon[36970]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:46 smithi164 ceph-mon[36970]: from='client.? ' entity='client.admin' 2024-07-26T06:40:46.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:40:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:46 smithi163 ceph-mon[36873]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:40:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:40:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:40:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:46 smithi163 ceph-mon[36873]: from='client.? ' entity='client.admin' 2024-07-26T06:40:47.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.521+0000 7f9bb6960640 1 -- 172.21.15.42:0/1562365097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb0107de0 msgr2=0x7f9bb010a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:47.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.521+0000 7f9bb6960640 1 --2- 172.21.15.42:0/1562365097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb0107de0 0x7f9bb010a2f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009900 tx=0x7f9ba002a410 comp rx=0 tx=0).stop 2024-07-26T06:40:47.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.521+0000 7f9bb6960640 1 -- 172.21.15.42:0/1562365097 shutdown_connections 2024-07-26T06:40:47.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.521+0000 7f9bb6960640 1 --2- 172.21.15.42:0/1562365097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb0107de0 0x7f9bb010a2f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:47.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.521+0000 7f9bb6960640 1 --2- 172.21.15.42:0/1562365097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bb01051c0 0x7f9bb01076a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:47.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.521+0000 7f9bb6960640 1 --2- 172.21.15.42:0/1562365097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bb0102760 0x7f9bb0104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:47.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.521+0000 7f9bb6960640 1 -- 172.21.15.42:0/1562365097 >> 172.21.15.42:0/1562365097 conn(0x7f9bb00fd1a0 msgr2=0x7f9bb00ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:47.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.521+0000 7f9bb6960640 1 -- 172.21.15.42:0/1562365097 shutdown_connections 2024-07-26T06:40:47.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.521+0000 7f9bb6960640 1 -- 172.21.15.42:0/1562365097 wait complete. 2024-07-26T06:40:47.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.522+0000 7f9bb6960640 1 Processor -- start 2024-07-26T06:40:47.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.522+0000 7f9bb6960640 1 -- start start 2024-07-26T06:40:47.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.523+0000 7f9bb6960640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bb0102760 0x7f9bb01043c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:47.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.523+0000 7f9bb6960640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb01051c0 0x7f9bb0102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:47.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.523+0000 7f9bb6960640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bb0107de0 0x7f9bb01031d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:47.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.523+0000 7f9bb6960640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb010dbe0 con 0x7f9bb01051c0 2024-07-26T06:40:47.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.523+0000 7f9bb6960640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb010da60 con 0x7f9bb0107de0 2024-07-26T06:40:47.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.523+0000 7f9bb6960640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb010dd60 con 0x7f9bb0102760 2024-07-26T06:40:47.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.523+0000 7f9baf7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb01051c0 0x7f9bb0102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:47.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.523+0000 7f9baffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bb0102760 0x7f9bb01043c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:47.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.523+0000 7f9bb4efd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bb0107de0 0x7f9bb01031d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:47.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.523+0000 7f9baf7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb01051c0 0x7f9bb0102c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50068/0 (socket says 172.21.15.42:50068) 2024-07-26T06:40:47.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.523+0000 7f9baf7fe640 1 -- 172.21.15.42:0/1464777458 learned_addr learned my addr 172.21.15.42:0/1464777458 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:40:47.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.524+0000 7f9baffff640 1 -- 172.21.15.42:0/1464777458 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bb0107de0 msgr2=0x7f9bb01031d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:47.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.524+0000 7f9baffff640 1 --2- 172.21.15.42:0/1464777458 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bb0107de0 0x7f9bb01031d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:47.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.524+0000 7f9baffff640 1 -- 172.21.15.42:0/1464777458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb01051c0 msgr2=0x7f9bb0102c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:47.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.524+0000 7f9baffff640 1 --2- 172.21.15.42:0/1464777458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb01051c0 0x7f9bb0102c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:47.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.524+0000 7f9baffff640 1 -- 172.21.15.42:0/1464777458 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bb01038d0 con 0x7f9bb0102760 2024-07-26T06:40:47.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.524+0000 7f9bb4efd640 1 --2- 172.21.15.42:0/1464777458 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bb0107de0 0x7f9bb01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:40:47.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.524+0000 7f9baffff640 1 --2- 172.21.15.42:0/1464777458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bb0102760 0x7f9bb01043c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800e9f0 tx=0x7f9ba800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:47.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.524+0000 7f9bad7fa640 1 -- 172.21.15.42:0/1464777458 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba800cd80 con 0x7f9bb0102760 2024-07-26T06:40:47.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.524+0000 7f9bad7fa640 1 -- 172.21.15.42:0/1464777458 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9ba8011900 con 0x7f9bb0102760 2024-07-26T06:40:47.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.524+0000 7f9bad7fa640 1 -- 172.21.15.42:0/1464777458 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba80108c0 con 0x7f9bb0102760 2024-07-26T06:40:47.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.524+0000 7f9bb6960640 1 -- 172.21.15.42:0/1464777458 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bb01a3920 con 0x7f9bb0102760 2024-07-26T06:40:47.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.525+0000 7f9bb6960640 1 -- 172.21.15.42:0/1464777458 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bb01a3dd0 con 0x7f9bb0102760 2024-07-26T06:40:47.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.526+0000 7f9bb6960640 1 -- 172.21.15.42:0/1464777458 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bb0104d80 con 0x7f9bb0102760 2024-07-26T06:40:47.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.527+0000 7f9bad7fa640 1 -- 172.21.15.42:0/1464777458 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f9ba8011470 con 0x7f9bb0102760 2024-07-26T06:40:47.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.528+0000 7f9bad7fa640 1 --2- 172.21.15.42:0/1464777458 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9b980b3bc0 0x7f9b980b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:47.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.528+0000 7f9baf7fe640 1 --2- 172.21.15.42:0/1464777458 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9b980b3bc0 0x7f9b980b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:47.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.528+0000 7f9bad7fa640 1 -- 172.21.15.42:0/1464777458 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9ba80e23b0 con 0x7f9bb0102760 2024-07-26T06:40:47.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.529+0000 7f9baf7fe640 1 --2- 172.21.15.42:0/1464777458 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9b980b3bc0 0x7f9b980b60d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4009c90 tx=0x7f9ba4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:47.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.533+0000 7f9bad7fa640 1 -- 172.21.15.42:0/1464777458 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9ba8016030 con 0x7f9bb0102760 2024-07-26T06:40:47.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.721+0000 7f9bb6960640 1 -- 172.21.15.42:0/1464777458 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7f9bb01a40b0 con 0x7f9bb0102760 2024-07-26T06:40:47.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.730+0000 7f9bad7fa640 1 -- 172.21.15.42:0/1464777458 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v14)=0 v14) ==== 127+0+0 (secure 0 0 0) 0x7f9ba80ab630 con 0x7f9bb0102760 2024-07-26T06:40:47.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.732+0000 7f9bb6960640 1 -- 172.21.15.42:0/1464777458 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9b980b3bc0 msgr2=0x7f9b980b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:47.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.732+0000 7f9bb6960640 1 --2- 172.21.15.42:0/1464777458 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9b980b3bc0 0x7f9b980b60d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4009c90 tx=0x7f9ba4005f90 comp rx=0 tx=0).stop 2024-07-26T06:40:47.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.732+0000 7f9bb6960640 1 -- 172.21.15.42:0/1464777458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bb0102760 msgr2=0x7f9bb01043c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:47.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.732+0000 7f9bb6960640 1 --2- 172.21.15.42:0/1464777458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bb0102760 0x7f9bb01043c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9ba800e9f0 tx=0x7f9ba800eeb0 comp rx=0 tx=0).stop 2024-07-26T06:40:47.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.732+0000 7f9bb6960640 1 -- 172.21.15.42:0/1464777458 shutdown_connections 2024-07-26T06:40:47.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.732+0000 7f9bb6960640 1 --2- 172.21.15.42:0/1464777458 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9b980b3bc0 0x7f9b980b60d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:47.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.732+0000 7f9bb6960640 1 --2- 172.21.15.42:0/1464777458 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9bb0107de0 0x7f9bb01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:47.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.732+0000 7f9bb6960640 1 --2- 172.21.15.42:0/1464777458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bb01051c0 0x7f9bb0102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:47.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.732+0000 7f9bb6960640 1 --2- 172.21.15.42:0/1464777458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bb0102760 0x7f9bb01043c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:47.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.732+0000 7f9bb6960640 1 -- 172.21.15.42:0/1464777458 >> 172.21.15.42:0/1464777458 conn(0x7f9bb00fd1a0 msgr2=0x7f9bb01073e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:47.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.732+0000 7f9bb6960640 1 -- 172.21.15.42:0/1464777458 shutdown_connections 2024-07-26T06:40:47.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:45.732+0000 7f9bb6960640 1 -- 172.21.15.42:0/1464777458 wait complete. 2024-07-26T06:40:48.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:48 smithi042 ceph-mon[32775]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:48.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:48 smithi164 ceph-mon[36970]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:48 smithi163 ceph-mon[36873]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.023+0000 7f7aaeaa4640 1 -- 172.21.15.42:0/883149028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7aa8102590 msgr2=0x7f7aa81029a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.023+0000 7f7aaeaa4640 1 --2- 172.21.15.42:0/883149028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7aa8102590 0x7f7aa81029a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7a98009900 tx=0x7f7a9802a410 comp rx=0 tx=0).stop 2024-07-26T06:40:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.023+0000 7f7aaeaa4640 1 -- 172.21.15.42:0/883149028 shutdown_connections 2024-07-26T06:40:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.023+0000 7f7aaeaa4640 1 --2- 172.21.15.42:0/883149028 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7aa8103b80 0x7f7aa810a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.023+0000 7f7aaeaa4640 1 --2- 172.21.15.42:0/883149028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7aa8102f70 0x7f7aa8103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.023+0000 7f7aaeaa4640 1 --2- 172.21.15.42:0/883149028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7aa8102590 0x7f7aa81029a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.023+0000 7f7aaeaa4640 1 -- 172.21.15.42:0/883149028 >> 172.21.15.42:0/883149028 conn(0x7f7aa80fd170 msgr2=0x7f7aa80ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:49.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.023+0000 7f7aaeaa4640 1 -- 172.21.15.42:0/883149028 shutdown_connections 2024-07-26T06:40:49.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.023+0000 7f7aaeaa4640 1 -- 172.21.15.42:0/883149028 wait complete. 2024-07-26T06:40:49.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.024+0000 7f7aaeaa4640 1 Processor -- start 2024-07-26T06:40:49.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.024+0000 7f7aaeaa4640 1 -- start start 2024-07-26T06:40:49.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.024+0000 7f7aaeaa4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7aa8102590 0x7f7aa8101af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:49.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.024+0000 7f7aaeaa4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7aa8102f70 0x7f7aa819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:49.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.024+0000 7f7aaeaa4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7aa8103b80 0x7f7aa81a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:49.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.024+0000 7f7aaeaa4640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa810dbb0 con 0x7f7aa8102590 2024-07-26T06:40:49.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.024+0000 7f7aaeaa4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa810da30 con 0x7f7aa8103b80 2024-07-26T06:40:49.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.024+0000 7f7aaeaa4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa810dd30 con 0x7f7aa8102f70 2024-07-26T06:40:49.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.025+0000 7f7aac840640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7aa8102590 0x7f7aa8101af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:49.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.025+0000 7f7aac840640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7aa8102590 0x7f7aa8101af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46242/0 (socket says 172.21.15.42:46242) 2024-07-26T06:40:49.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.025+0000 7f7aac840640 1 -- 172.21.15.42:0/102483348 learned_addr learned my addr 172.21.15.42:0/102483348 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:40:49.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.025+0000 7f7aad041640 1 --2- 172.21.15.42:0/102483348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7aa8103b80 0x7f7aa81a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:49.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.025+0000 7f7aa7fff640 1 --2- 172.21.15.42:0/102483348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7aa8102f70 0x7f7aa819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:49.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.025+0000 7f7aac840640 1 -- 172.21.15.42:0/102483348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7aa8102f70 msgr2=0x7f7aa819e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:49.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.025+0000 7f7aac840640 1 --2- 172.21.15.42:0/102483348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7aa8102f70 0x7f7aa819e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:49.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.025+0000 7f7aac840640 1 -- 172.21.15.42:0/102483348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7aa8103b80 msgr2=0x7f7aa81a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:49.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.025+0000 7f7aac840640 1 --2- 172.21.15.42:0/102483348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7aa8103b80 0x7f7aa81a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.025+0000 7f7aac840640 1 -- 172.21.15.42:0/102483348 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa81a1410 con 0x7f7aa8102590 2024-07-26T06:40:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.025+0000 7f7aa7fff640 1 --2- 172.21.15.42:0/102483348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7aa8102f70 0x7f7aa819e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:40:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.025+0000 7f7aac840640 1 --2- 172.21.15.42:0/102483348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7aa8102590 0x7f7aa8101af0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7a98004770 tx=0x7f7a980042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.025+0000 7f7aa5ffb640 1 -- 172.21.15.42:0/102483348 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9803f030 con 0x7f7aa8102590 2024-07-26T06:40:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.025+0000 7f7aa5ffb640 1 -- 172.21.15.42:0/102483348 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7a98003c90 con 0x7f7aa8102590 2024-07-26T06:40:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.025+0000 7f7aaeaa4640 1 -- 172.21.15.42:0/102483348 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aa81a16a0 con 0x7f7aa8102590 2024-07-26T06:40:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.025+0000 7f7aa5ffb640 1 -- 172.21.15.42:0/102483348 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a98035070 con 0x7f7aa8102590 2024-07-26T06:40:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.026+0000 7f7aaeaa4640 1 -- 172.21.15.42:0/102483348 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aa81a1bd0 con 0x7f7aa8102590 2024-07-26T06:40:49.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.027+0000 7f7aaeaa4640 1 -- 172.21.15.42:0/102483348 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7aa81029a0 con 0x7f7aa8102590 2024-07-26T06:40:49.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.028+0000 7f7aa5ffb640 1 -- 172.21.15.42:0/102483348 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f7a98042030 con 0x7f7aa8102590 2024-07-26T06:40:49.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.029+0000 7f7aa5ffb640 1 --2- 172.21.15.42:0/102483348 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7a900b3bc0 0x7f7a900b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:49.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.029+0000 7f7aa7fff640 1 --2- 172.21.15.42:0/102483348 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7a900b3bc0 0x7f7a900b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:49.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.029+0000 7f7aa5ffb640 1 -- 172.21.15.42:0/102483348 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7a981025d0 con 0x7f7aa8102590 2024-07-26T06:40:49.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.029+0000 7f7aa7fff640 1 --2- 172.21.15.42:0/102483348 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7a900b3bc0 0x7f7a900b60d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c005d00 tx=0x7f7a9c005c70 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:49.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.033+0000 7f7aa5ffb640 1 -- 172.21.15.42:0/102483348 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7a980cb8f0 con 0x7f7aa8102590 2024-07-26T06:40:49.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.218+0000 7f7aaeaa4640 1 -- 172.21.15.42:0/102483348 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f7aa80639f0 con 0x7f7aa8102590 2024-07-26T06:40:49.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.222+0000 7f7aa5ffb640 1 -- 172.21.15.42:0/102483348 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v15)=0 v15) ==== 123+0+0 (secure 0 0 0) 0x7f7a980cb1f0 con 0x7f7aa8102590 2024-07-26T06:40:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.223+0000 7f7aa5ffb640 1 -- 172.21.15.42:0/102483348 <== mon.0 v2:172.21.15.42:3300/0 8 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a98003e40 con 0x7f7aa8102590 2024-07-26T06:40:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.224+0000 7f7aaeaa4640 1 -- 172.21.15.42:0/102483348 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7a900b3bc0 msgr2=0x7f7a900b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.224+0000 7f7aaeaa4640 1 --2- 172.21.15.42:0/102483348 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7a900b3bc0 0x7f7a900b60d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c005d00 tx=0x7f7a9c005c70 comp rx=0 tx=0).stop 2024-07-26T06:40:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.224+0000 7f7aaeaa4640 1 -- 172.21.15.42:0/102483348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7aa8102590 msgr2=0x7f7aa8101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.224+0000 7f7aaeaa4640 1 --2- 172.21.15.42:0/102483348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7aa8102590 0x7f7aa8101af0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7a98004770 tx=0x7f7a980042f0 comp rx=0 tx=0).stop 2024-07-26T06:40:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.225+0000 7f7aaeaa4640 1 -- 172.21.15.42:0/102483348 shutdown_connections 2024-07-26T06:40:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.225+0000 7f7aaeaa4640 1 --2- 172.21.15.42:0/102483348 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7a900b3bc0 0x7f7a900b60d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.225+0000 7f7aaeaa4640 1 --2- 172.21.15.42:0/102483348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7aa8103b80 0x7f7aa81a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.225+0000 7f7aaeaa4640 1 --2- 172.21.15.42:0/102483348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7aa8102f70 0x7f7aa819e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.225+0000 7f7aaeaa4640 1 --2- 172.21.15.42:0/102483348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7aa8102590 0x7f7aa8101af0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.225+0000 7f7aaeaa4640 1 -- 172.21.15.42:0/102483348 >> 172.21.15.42:0/102483348 conn(0x7f7aa80fd170 msgr2=0x7f7aa8106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.225+0000 7f7aaeaa4640 1 -- 172.21.15.42:0/102483348 shutdown_connections 2024-07-26T06:40:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:49.225+0000 7f7aaeaa4640 1 -- 172.21.15.42:0/102483348 wait complete. 2024-07-26T06:40:50.293 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:50 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/102483348' entity='client.admin' 2024-07-26T06:40:50.293 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:50 smithi042 ceph-mon[32775]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:50 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/102483348' entity='client.admin' 2024-07-26T06:40:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:50 smithi163 ceph-mon[36873]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:50.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:50 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/102483348' entity='client.admin' 2024-07-26T06:40:50.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:50 smithi164 ceph-mon[36970]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:50.931 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi042') 2024-07-26T06:40:50.931 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi042') 2024-07-26T06:40:52.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.957+0000 7fc85b296640 1 -- 172.21.15.42:0/3820928802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8541065d0 msgr2=0x7fc8541089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:52.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.957+0000 7fc85b296640 1 --2- 172.21.15.42:0/3820928802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8541065d0 0x7fc8541089f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc844009900 tx=0x7fc84402a410 comp rx=0 tx=0).stop 2024-07-26T06:40:52.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.957+0000 7fc85b296640 1 -- 172.21.15.42:0/3820928802 shutdown_connections 2024-07-26T06:40:52.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.957+0000 7fc85b296640 1 --2- 172.21.15.42:0/3820928802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc854108f30 0x7fc85410b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:52.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.957+0000 7fc85b296640 1 --2- 172.21.15.42:0/3820928802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc8541065d0 0x7fc8541089f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:52.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.957+0000 7fc85b296640 1 --2- 172.21.15.42:0/3820928802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc854101530 0x7fc854101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:52.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.957+0000 7fc85b296640 1 -- 172.21.15.42:0/3820928802 >> 172.21.15.42:0/3820928802 conn(0x7fc8540fd1a0 msgr2=0x7fc8540ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:52.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.957+0000 7fc85b296640 1 -- 172.21.15.42:0/3820928802 shutdown_connections 2024-07-26T06:40:52.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.957+0000 7fc85b296640 1 -- 172.21.15.42:0/3820928802 wait complete. 2024-07-26T06:40:52.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.958+0000 7fc85b296640 1 Processor -- start 2024-07-26T06:40:52.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.958+0000 7fc85b296640 1 -- start start 2024-07-26T06:40:52.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.958+0000 7fc85b296640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc854101530 0x7fc8541999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:52.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.958+0000 7fc85b296640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8541065d0 0x7fc85419a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:52.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.958+0000 7fc85b296640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc854108f30 0x7fc85419abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:52.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.958+0000 7fc85b296640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc85410edc0 con 0x7fc854101530 2024-07-26T06:40:52.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.958+0000 7fc85b296640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc85410ec40 con 0x7fc854108f30 2024-07-26T06:40:52.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.958+0000 7fc85b296640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc85410ef40 con 0x7fc8541065d0 2024-07-26T06:40:52.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.958+0000 7fc859032640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc854101530 0x7fc8541999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:52.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.958+0000 7fc859032640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc854101530 0x7fc8541999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46270/0 (socket says 172.21.15.42:46270) 2024-07-26T06:40:52.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.958+0000 7fc859032640 1 -- 172.21.15.42:0/2545653944 learned_addr learned my addr 172.21.15.42:0/2545653944 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:40:52.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.958+0000 7fc858831640 1 --2- 172.21.15.42:0/2545653944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8541065d0 0x7fc85419a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:52.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.959+0000 7fc859833640 1 --2- 172.21.15.42:0/2545653944 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc854108f30 0x7fc85419abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:52.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.959+0000 7fc859032640 1 -- 172.21.15.42:0/2545653944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8541065d0 msgr2=0x7fc85419a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:52.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.959+0000 7fc859032640 1 --2- 172.21.15.42:0/2545653944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8541065d0 0x7fc85419a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:52.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.959+0000 7fc859032640 1 -- 172.21.15.42:0/2545653944 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc854108f30 msgr2=0x7fc85419abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:52.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.959+0000 7fc859032640 1 --2- 172.21.15.42:0/2545653944 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc854108f30 0x7fc85419abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:52.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.959+0000 7fc859032640 1 -- 172.21.15.42:0/2545653944 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc85419b400 con 0x7fc854101530 2024-07-26T06:40:52.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.959+0000 7fc859833640 1 --2- 172.21.15.42:0/2545653944 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc854108f30 0x7fc85419abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:40:52.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.959+0000 7fc859032640 1 --2- 172.21.15.42:0/2545653944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc854101530 0x7fc8541999d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc85000e9f0 tx=0x7fc85000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:52.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.959+0000 7fc84a7fc640 1 -- 172.21.15.42:0/2545653944 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85000cd80 con 0x7fc854101530 2024-07-26T06:40:52.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.959+0000 7fc84a7fc640 1 -- 172.21.15.42:0/2545653944 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8500108e0 con 0x7fc854101530 2024-07-26T06:40:52.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.959+0000 7fc85b296640 1 -- 172.21.15.42:0/2545653944 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc85419b6f0 con 0x7fc854101530 2024-07-26T06:40:52.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.959+0000 7fc84a7fc640 1 -- 172.21.15.42:0/2545653944 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc850018980 con 0x7fc854101530 2024-07-26T06:40:52.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.959+0000 7fc85b296640 1 -- 172.21.15.42:0/2545653944 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8541a1df0 con 0x7fc854101530 2024-07-26T06:40:52.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.961+0000 7fc85b296640 1 -- 172.21.15.42:0/2545653944 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8540fecc0 con 0x7fc854101530 2024-07-26T06:40:52.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.962+0000 7fc84a7fc640 1 -- 172.21.15.42:0/2545653944 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7fc850010430 con 0x7fc854101530 2024-07-26T06:40:52.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.962+0000 7fc84a7fc640 1 --2- 172.21.15.42:0/2545653944 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc8400b3bc0 0x7fc8400b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:52.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.963+0000 7fc84a7fc640 1 -- 172.21.15.42:0/2545653944 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc8500e22d0 con 0x7fc854101530 2024-07-26T06:40:52.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.963+0000 7fc858831640 1 --2- 172.21.15.42:0/2545653944 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc8400b3bc0 0x7fc8400b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:52.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.963+0000 7fc858831640 1 --2- 172.21.15.42:0/2545653944 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc8400b3bc0 0x7fc8400b60d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc844031440 tx=0x7fc8440314a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:52.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.967+0000 7fc84a7fc640 1 -- 172.21.15.42:0/2545653944 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc8500ab5f0 con 0x7fc854101530 2024-07-26T06:40:52.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:52.151+0000 7fc85b296640 1 -- 172.21.15.42:0/2545653944 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7fc8540639f0 con 0x7fc854101530 2024-07-26T06:40:52.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:52.155+0000 7fc84a7fc640 1 -- 172.21.15.42:0/2545653944 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v17)=0 v17) ==== 108+0+0 (secure 0 0 0) 0x7fc8500aaef0 con 0x7fc854101530 2024-07-26T06:40:52.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:52.158+0000 7fc85b296640 1 -- 172.21.15.42:0/2545653944 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc8400b3bc0 msgr2=0x7fc8400b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:52.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:52.158+0000 7fc85b296640 1 --2- 172.21.15.42:0/2545653944 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc8400b3bc0 0x7fc8400b60d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc844031440 tx=0x7fc8440314a0 comp rx=0 tx=0).stop 2024-07-26T06:40:52.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:52.158+0000 7fc85b296640 1 -- 172.21.15.42:0/2545653944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc854101530 msgr2=0x7fc8541999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:52.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:52.158+0000 7fc85b296640 1 --2- 172.21.15.42:0/2545653944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc854101530 0x7fc8541999d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc85000e9f0 tx=0x7fc85000eeb0 comp rx=0 tx=0).stop 2024-07-26T06:40:52.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:52.158+0000 7fc85b296640 1 -- 172.21.15.42:0/2545653944 shutdown_connections 2024-07-26T06:40:52.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:52.158+0000 7fc85b296640 1 --2- 172.21.15.42:0/2545653944 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc8400b3bc0 0x7fc8400b60d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:52.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:52.158+0000 7fc85b296640 1 --2- 172.21.15.42:0/2545653944 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc854108f30 0x7fc85419abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:52.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:52.158+0000 7fc85b296640 1 --2- 172.21.15.42:0/2545653944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8541065d0 0x7fc85419a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:52.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:52.158+0000 7fc85b296640 1 --2- 172.21.15.42:0/2545653944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc854101530 0x7fc8541999d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:52.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:52.158+0000 7fc85b296640 1 -- 172.21.15.42:0/2545653944 >> 172.21.15.42:0/2545653944 conn(0x7fc8540fd1a0 msgr2=0x7fc854107480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:52.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:52.158+0000 7fc85b296640 1 -- 172.21.15.42:0/2545653944 shutdown_connections 2024-07-26T06:40:52.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:52.158+0000 7fc85b296640 1 -- 172.21.15.42:0/2545653944 wait complete. 2024-07-26T06:40:52.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:52 smithi042 ceph-mon[32775]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:52.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:52 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3832450296' entity='client.admin' 2024-07-26T06:40:52.834 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:52 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2545653944' entity='client.admin' 2024-07-26T06:40:52.869 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:52 smithi164 ceph-mon[36970]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:52.869 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:52 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3832450296' entity='client.admin' 2024-07-26T06:40:52.869 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:52 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2545653944' entity='client.admin' 2024-07-26T06:40:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:52 smithi163 ceph-mon[36873]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:52 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3832450296' entity='client.admin' 2024-07-26T06:40:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:52 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2545653944' entity='client.admin' 2024-07-26T06:40:54.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.375+0000 7f05cda56640 1 -- 172.21.15.42:0/2065283705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05c8103100 msgr2=0x7f05c8103510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:54.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.375+0000 7f05cda56640 1 --2- 172.21.15.42:0/2065283705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05c8103100 0x7f05c8103510 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009900 tx=0x7f05b002a480 comp rx=0 tx=0).stop 2024-07-26T06:40:54.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.375+0000 7f05cda56640 1 -- 172.21.15.42:0/2065283705 shutdown_connections 2024-07-26T06:40:54.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.375+0000 7f05cda56640 1 --2- 172.21.15.42:0/2065283705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c8103aa0 0x7f05c810a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:54.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.375+0000 7f05cda56640 1 --2- 172.21.15.42:0/2065283705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05c8103100 0x7f05c8103510 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:54.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.375+0000 7f05cda56640 1 --2- 172.21.15.42:0/2065283705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05c8102070 0x7f05c81024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:54.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.375+0000 7f05cda56640 1 -- 172.21.15.42:0/2065283705 >> 172.21.15.42:0/2065283705 conn(0x7f05c80fd9e0 msgr2=0x7f05c80ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:54.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.375+0000 7f05cda56640 1 -- 172.21.15.42:0/2065283705 shutdown_connections 2024-07-26T06:40:54.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.375+0000 7f05cda56640 1 -- 172.21.15.42:0/2065283705 wait complete. 2024-07-26T06:40:54.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.376+0000 7f05cda56640 1 Processor -- start 2024-07-26T06:40:54.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.376+0000 7f05cda56640 1 -- start start 2024-07-26T06:40:54.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.377+0000 7f05cda56640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c8102070 0x7f05c8199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:54.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.377+0000 7f05cda56640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05c8103100 0x7f05c819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:54.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.377+0000 7f05cda56640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05c8103aa0 0x7f05c819ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:54.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.377+0000 7f05cda56640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f05c810dcd0 con 0x7f05c8103100 2024-07-26T06:40:54.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.377+0000 7f05cda56640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f05c810db50 con 0x7f05c8102070 2024-07-26T06:40:54.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.377+0000 7f05cda56640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f05c810de50 con 0x7f05c8103aa0 2024-07-26T06:40:54.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.377+0000 7f05c67fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05c8103100 0x7f05c819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:54.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.377+0000 7f05c67fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05c8103100 0x7f05c819a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46260/0 (socket says 172.21.15.42:46260) 2024-07-26T06:40:54.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.377+0000 7f05c67fc640 1 -- 172.21.15.42:0/3832450296 learned_addr learned my addr 172.21.15.42:0/3832450296 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:40:54.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.377+0000 7f05c6ffd640 1 --2- 172.21.15.42:0/3832450296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c8102070 0x7f05c8199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:54.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.377+0000 7f05c77fe640 1 --2- 172.21.15.42:0/3832450296 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05c8103aa0 0x7f05c819ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:54.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.377+0000 7f05c67fc640 1 -- 172.21.15.42:0/3832450296 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05c8103aa0 msgr2=0x7f05c819ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:54.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.377+0000 7f05c67fc640 1 --2- 172.21.15.42:0/3832450296 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05c8103aa0 0x7f05c819ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:54.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.377+0000 7f05c67fc640 1 -- 172.21.15.42:0/3832450296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c8102070 msgr2=0x7f05c8199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:54.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.377+0000 7f05c67fc640 1 --2- 172.21.15.42:0/3832450296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c8102070 0x7f05c8199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:54.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.377+0000 7f05c67fc640 1 -- 172.21.15.42:0/3832450296 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c819b3f0 con 0x7f05c8103100 2024-07-26T06:40:54.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.377+0000 7f05c67fc640 1 --2- 172.21.15.42:0/3832450296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05c8103100 0x7f05c819a0c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f05b0006010 tx=0x7f05b0004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:54.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.378+0000 7f05cca54640 1 -- 172.21.15.42:0/3832450296 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b003f030 con 0x7f05c8103100 2024-07-26T06:40:54.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.378+0000 7f05cda56640 1 -- 172.21.15.42:0/3832450296 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05c819b680 con 0x7f05c8103100 2024-07-26T06:40:54.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.378+0000 7f05cca54640 1 -- 172.21.15.42:0/3832450296 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05b0003cb0 con 0x7f05c8103100 2024-07-26T06:40:54.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.378+0000 7f05cca54640 1 -- 172.21.15.42:0/3832450296 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b0035070 con 0x7f05c8103100 2024-07-26T06:40:54.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.378+0000 7f05cda56640 1 -- 172.21.15.42:0/3832450296 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05c81a1ca0 con 0x7f05c8103100 2024-07-26T06:40:54.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.379+0000 7f05cda56640 1 -- 172.21.15.42:0/3832450296 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05c81a1ed0 con 0x7f05c8103100 2024-07-26T06:40:54.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.380+0000 7f05cca54640 1 -- 172.21.15.42:0/3832450296 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f05b0042030 con 0x7f05c8103100 2024-07-26T06:40:54.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.381+0000 7f05cca54640 1 --2- 172.21.15.42:0/3832450296 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f05b40b3bc0 0x7f05b40b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:54.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.381+0000 7f05c6ffd640 1 --2- 172.21.15.42:0/3832450296 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f05b40b3bc0 0x7f05b40b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:54.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.381+0000 7f05cca54640 1 -- 172.21.15.42:0/3832450296 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f05b0102590 con 0x7f05c8103100 2024-07-26T06:40:54.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.382+0000 7f05c6ffd640 1 --2- 172.21.15.42:0/3832450296 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f05b40b3bc0 0x7f05b40b60d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f05c0005fd0 tx=0x7f05c0009680 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:54.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.386+0000 7f05cca54640 1 -- 172.21.15.42:0/3832450296 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f05b0003e60 con 0x7f05c8103100 2024-07-26T06:40:54.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.570+0000 7f05cda56640 1 -- 172.21.15.42:0/3832450296 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7f05c81a20a0 con 0x7f05c8103100 2024-07-26T06:40:54.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.574+0000 7f05cca54640 1 -- 172.21.15.42:0/3832450296 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v16)=0 v16) ==== 125+0+0 (secure 0 0 0) 0x7f05b00cb1c0 con 0x7f05c8103100 2024-07-26T06:40:54.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.576+0000 7f05cda56640 1 -- 172.21.15.42:0/3832450296 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f05b40b3bc0 msgr2=0x7f05b40b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:54.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.576+0000 7f05cda56640 1 --2- 172.21.15.42:0/3832450296 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f05b40b3bc0 0x7f05b40b60d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f05c0005fd0 tx=0x7f05c0009680 comp rx=0 tx=0).stop 2024-07-26T06:40:54.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.576+0000 7f05cda56640 1 -- 172.21.15.42:0/3832450296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05c8103100 msgr2=0x7f05c819a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:54.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.576+0000 7f05cda56640 1 --2- 172.21.15.42:0/3832450296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05c8103100 0x7f05c819a0c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f05b0006010 tx=0x7f05b0004340 comp rx=0 tx=0).stop 2024-07-26T06:40:54.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.576+0000 7f05cda56640 1 -- 172.21.15.42:0/3832450296 shutdown_connections 2024-07-26T06:40:54.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.576+0000 7f05cda56640 1 --2- 172.21.15.42:0/3832450296 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f05b40b3bc0 0x7f05b40b60d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:54.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.576+0000 7f05cda56640 1 --2- 172.21.15.42:0/3832450296 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05c8103aa0 0x7f05c819ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:54.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.576+0000 7f05cda56640 1 --2- 172.21.15.42:0/3832450296 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f05c8103100 0x7f05c819a0c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:54.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.576+0000 7f05cda56640 1 --2- 172.21.15.42:0/3832450296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05c8102070 0x7f05c8199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:54.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.576+0000 7f05cda56640 1 -- 172.21.15.42:0/3832450296 >> 172.21.15.42:0/3832450296 conn(0x7f05c80fd9e0 msgr2=0x7f05c80ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:54.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.576+0000 7f05cda56640 1 -- 172.21.15.42:0/3832450296 shutdown_connections 2024-07-26T06:40:54.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:51.576+0000 7f05cda56640 1 -- 172.21.15.42:0/3832450296 wait complete. 2024-07-26T06:40:54.600 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:54 smithi042 ceph-mon[32775]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:54 smithi164 ceph-mon[36970]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:55.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:54 smithi163 ceph-mon[36873]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:55.151 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi042') 2024-07-26T06:40:56.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.132+0000 7f9de4fbf640 1 -- 172.21.15.42:0/4054244598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9de0106310 msgr2=0x7f9de0108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:56.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.132+0000 7f9de4fbf640 1 --2- 172.21.15.42:0/4054244598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9de0106310 0x7f9de0108770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4009ca0 tx=0x7f9dd402a8c0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.132+0000 7f9de4fbf640 1 -- 172.21.15.42:0/4054244598 shutdown_connections 2024-07-26T06:40:56.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.132+0000 7f9de4fbf640 1 --2- 172.21.15.42:0/4054244598 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de0108eb0 0x7f9de010b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.132+0000 7f9de4fbf640 1 --2- 172.21.15.42:0/4054244598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9de0106310 0x7f9de0108770 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.132+0000 7f9de4fbf640 1 --2- 172.21.15.42:0/4054244598 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9de006c090 0x7f9de0105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.132+0000 7f9de4fbf640 1 -- 172.21.15.42:0/4054244598 >> 172.21.15.42:0/4054244598 conn(0x7f9de00fda10 msgr2=0x7f9de00ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:56.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.132+0000 7f9de4fbf640 1 -- 172.21.15.42:0/4054244598 shutdown_connections 2024-07-26T06:40:56.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.132+0000 7f9de4fbf640 1 -- 172.21.15.42:0/4054244598 wait complete. 2024-07-26T06:40:56.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.133+0000 7f9de4fbf640 1 Processor -- start 2024-07-26T06:40:56.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.133+0000 7f9de4fbf640 1 -- start start 2024-07-26T06:40:56.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.133+0000 7f9de4fbf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de006c090 0x7f9de0199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:56.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.133+0000 7f9de4fbf640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9de0106310 0x7f9de019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:56.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.133+0000 7f9de4fbf640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9de0108eb0 0x7f9de019ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:56.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.133+0000 7f9de4fbf640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9de010edd0 con 0x7f9de0106310 2024-07-26T06:40:56.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.133+0000 7f9de4fbf640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9de010ec50 con 0x7f9de0108eb0 2024-07-26T06:40:56.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.133+0000 7f9de4fbf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9de010ef50 con 0x7f9de006c090 2024-07-26T06:40:56.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.133+0000 7f9dddd9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9de0106310 0x7f9de019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:56.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.133+0000 7f9dddd9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9de0106310 0x7f9de019a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46296/0 (socket says 172.21.15.42:46296) 2024-07-26T06:40:56.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.133+0000 7f9dddd9b640 1 -- 172.21.15.42:0/848649320 learned_addr learned my addr 172.21.15.42:0/848649320 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:40:56.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.133+0000 7f9dded9d640 1 --2- 172.21.15.42:0/848649320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9de0108eb0 0x7f9de019ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:56.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.134+0000 7f9dde59c640 1 --2- 172.21.15.42:0/848649320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de006c090 0x7f9de0199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:56.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.134+0000 7f9dddd9b640 1 -- 172.21.15.42:0/848649320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de006c090 msgr2=0x7f9de0199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:56.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.134+0000 7f9dddd9b640 1 --2- 172.21.15.42:0/848649320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de006c090 0x7f9de0199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.134+0000 7f9dddd9b640 1 -- 172.21.15.42:0/848649320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9de0108eb0 msgr2=0x7f9de019ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:56.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.134+0000 7f9dddd9b640 1 --2- 172.21.15.42:0/848649320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9de0108eb0 0x7f9de019ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.134+0000 7f9dddd9b640 1 -- 172.21.15.42:0/848649320 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de019b430 con 0x7f9de0106310 2024-07-26T06:40:56.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.134+0000 7f9dde59c640 1 --2- 172.21.15.42:0/848649320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de006c090 0x7f9de0199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:40:56.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.134+0000 7f9dddd9b640 1 --2- 172.21.15.42:0/848649320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9de0106310 0x7f9de019a070 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9dd402ade0 tx=0x7f9dd4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:56.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.134+0000 7f9dcf7fe640 1 -- 172.21.15.42:0/848649320 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd40317c0 con 0x7f9de0106310 2024-07-26T06:40:56.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.134+0000 7f9dcf7fe640 1 -- 172.21.15.42:0/848649320 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dd40044f0 con 0x7f9de0106310 2024-07-26T06:40:56.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.134+0000 7f9de4fbf640 1 -- 172.21.15.42:0/848649320 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9de019b6c0 con 0x7f9de0106310 2024-07-26T06:40:56.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.134+0000 7f9dcf7fe640 1 -- 172.21.15.42:0/848649320 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd402fd10 con 0x7f9de0106310 2024-07-26T06:40:56.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.134+0000 7f9de4fbf640 1 -- 172.21.15.42:0/848649320 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9de01a1e20 con 0x7f9de0106310 2024-07-26T06:40:56.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.135+0000 7f9de4fbf640 1 -- 172.21.15.42:0/848649320 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9de00ff530 con 0x7f9de0106310 2024-07-26T06:40:56.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.137+0000 7f9dcf7fe640 1 -- 172.21.15.42:0/848649320 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f9dd4004040 con 0x7f9de0106310 2024-07-26T06:40:56.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.138+0000 7f9dcf7fe640 1 --2- 172.21.15.42:0/848649320 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9dc80b3bc0 0x7f9dc80b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:56.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.138+0000 7f9dde59c640 1 --2- 172.21.15.42:0/848649320 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9dc80b3bc0 0x7f9dc80b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:56.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.138+0000 7f9dcf7fe640 1 -- 172.21.15.42:0/848649320 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9dd41036a0 con 0x7f9de0106310 2024-07-26T06:40:56.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.138+0000 7f9dde59c640 1 --2- 172.21.15.42:0/848649320 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9dc80b3bc0 0x7f9dc80b60d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0005d00 tx=0x7f9dd0005c70 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:56.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.141+0000 7f9dcf7fe640 1 -- 172.21.15.42:0/848649320 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9dd40cc9c0 con 0x7f9de0106310 2024-07-26T06:40:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.322+0000 7f110f6e9640 1 -- 172.21.15.42:0/3505437920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1108106360 msgr2=0x7f1108108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.322+0000 7f110f6e9640 1 --2- 172.21.15.42:0/3505437920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1108106360 0x7f1108108780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009900 tx=0x7f10f802a410 comp rx=0 tx=0).stop 2024-07-26T06:40:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.322+0000 7f110f6e9640 1 -- 172.21.15.42:0/3505437920 shutdown_connections 2024-07-26T06:40:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.322+0000 7f110f6e9640 1 --2- 172.21.15.42:0/3505437920 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1108108ea0 0x7f110810b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.322+0000 7f110f6e9640 1 --2- 172.21.15.42:0/3505437920 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1108106360 0x7f1108108780 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.322+0000 7f110f6e9640 1 --2- 172.21.15.42:0/3505437920 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f110806b350 0x7f1108105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.322+0000 7f110f6e9640 1 -- 172.21.15.42:0/3505437920 >> 172.21.15.42:0/3505437920 conn(0x7f11080fd9e0 msgr2=0x7f11080ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.322+0000 7f110f6e9640 1 -- 172.21.15.42:0/3505437920 shutdown_connections 2024-07-26T06:40:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.323+0000 7f110f6e9640 1 -- 172.21.15.42:0/3505437920 wait complete. 2024-07-26T06:40:56.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.323+0000 7f110f6e9640 1 Processor -- start 2024-07-26T06:40:56.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.323+0000 7f110f6e9640 1 -- start start 2024-07-26T06:40:56.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.324+0000 7f110f6e9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f110806b350 0x7f1108199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:56.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.324+0000 7f110f6e9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1108106360 0x7f110819a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.324+0000 7f110f6e9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1108108ea0 0x7f110819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.324+0000 7f110f6e9640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f110810edc0 con 0x7f1108106360 2024-07-26T06:40:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.324+0000 7f110f6e9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f110810ec40 con 0x7f110806b350 2024-07-26T06:40:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.324+0000 7f110f6e9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f110810ef40 con 0x7f1108108ea0 2024-07-26T06:40:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.324+0000 7f110d485640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f110806b350 0x7f1108199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.324+0000 7f110cc84640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1108106360 0x7f110819a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.324+0000 7f110d485640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f110806b350 0x7f1108199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:47628/0 (socket says 172.21.15.42:47628) 2024-07-26T06:40:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.324+0000 7f110d485640 1 -- 172.21.15.42:0/3622911979 learned_addr learned my addr 172.21.15.42:0/3622911979 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:40:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.324+0000 7f110dc86640 1 --2- 172.21.15.42:0/3622911979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1108108ea0 0x7f110819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.324+0000 7f110cc84640 1 -- 172.21.15.42:0/3622911979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1108108ea0 msgr2=0x7f110819ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.324+0000 7f110cc84640 1 --2- 172.21.15.42:0/3622911979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1108108ea0 0x7f110819ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.324+0000 7f110cc84640 1 -- 172.21.15.42:0/3622911979 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f110806b350 msgr2=0x7f1108199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.324+0000 7f110cc84640 1 --2- 172.21.15.42:0/3622911979 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f110806b350 0x7f1108199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.324+0000 7f110cc84640 1 -- 172.21.15.42:0/3622911979 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110819b450 con 0x7f1108106360 2024-07-26T06:40:56.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.325+0000 7f110cc84640 1 --2- 172.21.15.42:0/3622911979 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1108106360 0x7f110819a090 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f10f8006010 tx=0x7f10f8031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:56.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.325+0000 7f10fe7fc640 1 -- 172.21.15.42:0/3622911979 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f803f070 con 0x7f1108106360 2024-07-26T06:40:56.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.325+0000 7f10fe7fc640 1 -- 172.21.15.42:0/3622911979 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10f8003ca0 con 0x7f1108106360 2024-07-26T06:40:56.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.325+0000 7f110f6e9640 1 -- 172.21.15.42:0/3622911979 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f110819b6e0 con 0x7f1108106360 2024-07-26T06:40:56.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.325+0000 7f10fe7fc640 1 -- 172.21.15.42:0/3622911979 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8035070 con 0x7f1108106360 2024-07-26T06:40:56.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.325+0000 7f110f6e9640 1 -- 172.21.15.42:0/3622911979 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1108101c00 con 0x7f1108106360 2024-07-26T06:40:56.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.326+0000 7f110f6e9640 1 -- 172.21.15.42:0/3622911979 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11080ff500 con 0x7f1108106360 2024-07-26T06:40:56.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.328+0000 7f9de4fbf640 1 -- 172.21.15.42:0/848649320 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7f9de002e0c0 con 0x7f9de0106310 2024-07-26T06:40:56.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.328+0000 7f10fe7fc640 1 -- 172.21.15.42:0/3622911979 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f10f803b580 con 0x7f1108106360 2024-07-26T06:40:56.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.329+0000 7f10fe7fc640 1 --2- 172.21.15.42:0/3622911979 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f10f40b3bc0 0x7f10f40b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:56.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.329+0000 7f110d485640 1 --2- 172.21.15.42:0/3622911979 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f10f40b3bc0 0x7f10f40b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:56.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.329+0000 7f10fe7fc640 1 -- 172.21.15.42:0/3622911979 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f10f81029e0 con 0x7f1108106360 2024-07-26T06:40:56.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.329+0000 7f110d485640 1 --2- 172.21.15.42:0/3622911979 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f10f40b3bc0 0x7f10f40b60d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f110400bca0 tx=0x7f110400b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:56.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.332+0000 7f10fe7fc640 1 -- 172.21.15.42:0/3622911979 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f10f803be20 con 0x7f1108106360 2024-07-26T06:40:56.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.335+0000 7f9dcf7fe640 1 -- 172.21.15.42:0/848649320 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v18)=0 v18) ==== 119+0+0 (secure 0 0 0) 0x7f9dd40cc2c0 con 0x7f9de0106310 2024-07-26T06:40:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.338+0000 7f9de4fbf640 1 -- 172.21.15.42:0/848649320 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9dc80b3bc0 msgr2=0x7f9dc80b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.338+0000 7f9de4fbf640 1 --2- 172.21.15.42:0/848649320 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9dc80b3bc0 0x7f9dc80b60d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0005d00 tx=0x7f9dd0005c70 comp rx=0 tx=0).stop 2024-07-26T06:40:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.338+0000 7f9de4fbf640 1 -- 172.21.15.42:0/848649320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9de0106310 msgr2=0x7f9de019a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.338+0000 7f9de4fbf640 1 --2- 172.21.15.42:0/848649320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9de0106310 0x7f9de019a070 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9dd402ade0 tx=0x7f9dd4003ba0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.338+0000 7f9de4fbf640 1 -- 172.21.15.42:0/848649320 shutdown_connections 2024-07-26T06:40:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.338+0000 7f9de4fbf640 1 --2- 172.21.15.42:0/848649320 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9dc80b3bc0 0x7f9dc80b60d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.338+0000 7f9de4fbf640 1 --2- 172.21.15.42:0/848649320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9de0108eb0 0x7f9de019ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.338+0000 7f9de4fbf640 1 --2- 172.21.15.42:0/848649320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9de0106310 0x7f9de019a070 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.338+0000 7f9de4fbf640 1 --2- 172.21.15.42:0/848649320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de006c090 0x7f9de0199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.338+0000 7f9de4fbf640 1 -- 172.21.15.42:0/848649320 >> 172.21.15.42:0/848649320 conn(0x7f9de00fda10 msgr2=0x7f9de0104860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.338+0000 7f9de4fbf640 1 -- 172.21.15.42:0/848649320 shutdown_connections 2024-07-26T06:40:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.338+0000 7f9de4fbf640 1 -- 172.21.15.42:0/848649320 wait complete. 2024-07-26T06:40:56.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.518+0000 7f110f6e9640 1 -- 172.21.15.42:0/3622911979 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7f110802e0c0 con 0x7f1108106360 2024-07-26T06:40:56.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.522+0000 7f10fe7fc640 1 -- 172.21.15.42:0/3622911979 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v19)=0 v19) ==== 115+0+0 (secure 0 0 0) 0x7f10f80cb6b0 con 0x7f1108106360 2024-07-26T06:40:56.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.524+0000 7f110f6e9640 1 -- 172.21.15.42:0/3622911979 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f10f40b3bc0 msgr2=0x7f10f40b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:56.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.524+0000 7f110f6e9640 1 --2- 172.21.15.42:0/3622911979 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f10f40b3bc0 0x7f10f40b60d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f110400bca0 tx=0x7f110400b410 comp rx=0 tx=0).stop 2024-07-26T06:40:56.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.524+0000 7f110f6e9640 1 -- 172.21.15.42:0/3622911979 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1108106360 msgr2=0x7f110819a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:56.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.524+0000 7f110f6e9640 1 --2- 172.21.15.42:0/3622911979 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1108106360 0x7f110819a090 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f10f8006010 tx=0x7f10f8031a50 comp rx=0 tx=0).stop 2024-07-26T06:40:56.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.524+0000 7f110f6e9640 1 -- 172.21.15.42:0/3622911979 shutdown_connections 2024-07-26T06:40:56.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.524+0000 7f110f6e9640 1 --2- 172.21.15.42:0/3622911979 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f10f40b3bc0 0x7f10f40b60d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.524+0000 7f110f6e9640 1 --2- 172.21.15.42:0/3622911979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1108108ea0 0x7f110819ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.524+0000 7f110f6e9640 1 --2- 172.21.15.42:0/3622911979 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1108106360 0x7f110819a090 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.524+0000 7f110f6e9640 1 --2- 172.21.15.42:0/3622911979 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f110806b350 0x7f1108199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:56.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.524+0000 7f110f6e9640 1 -- 172.21.15.42:0/3622911979 >> 172.21.15.42:0/3622911979 conn(0x7f11080fd9e0 msgr2=0x7f1108104810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:56.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.524+0000 7f110f6e9640 1 -- 172.21.15.42:0/3622911979 shutdown_connections 2024-07-26T06:40:56.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:56.524+0000 7f110f6e9640 1 -- 172.21.15.42:0/3622911979 wait complete. 2024-07-26T06:40:56.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:56 smithi042 ceph-mon[32775]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:56.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:56 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/848649320' entity='client.admin' 2024-07-26T06:40:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:56 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3622911979' entity='client.admin' 2024-07-26T06:40:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:56 smithi164 ceph-mon[36970]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:56 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/848649320' entity='client.admin' 2024-07-26T06:40:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:56 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3622911979' entity='client.admin' 2024-07-26T06:40:57.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:56 smithi163 ceph-mon[36873]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:56 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/848649320' entity='client.admin' 2024-07-26T06:40:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:56 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3622911979' entity='client.admin' 2024-07-26T06:40:57.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.086+0000 7fb42e24a640 1 -- 172.21.15.42:0/2930874051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb42806b3b0 msgr2=0x7fb428105dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:57.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.086+0000 7fb42e24a640 1 --2- 172.21.15.42:0/2930874051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb42806b3b0 0x7fb428105dc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb418009ca0 tx=0x7fb41802a8c0 comp rx=0 tx=0).stop 2024-07-26T06:40:57.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.086+0000 7fb42e24a640 1 -- 172.21.15.42:0/2930874051 shutdown_connections 2024-07-26T06:40:57.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.086+0000 7fb42e24a640 1 --2- 172.21.15.42:0/2930874051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb428108ea0 0x7fb42810b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:57.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.086+0000 7fb42e24a640 1 --2- 172.21.15.42:0/2930874051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb428106300 0x7fb428108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:57.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.086+0000 7fb42e24a640 1 --2- 172.21.15.42:0/2930874051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb42806b3b0 0x7fb428105dc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:57.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.086+0000 7fb42e24a640 1 -- 172.21.15.42:0/2930874051 >> 172.21.15.42:0/2930874051 conn(0x7fb4280fd9e0 msgr2=0x7fb4280ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:57.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.086+0000 7fb42e24a640 1 -- 172.21.15.42:0/2930874051 shutdown_connections 2024-07-26T06:40:57.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.086+0000 7fb42e24a640 1 -- 172.21.15.42:0/2930874051 wait complete. 2024-07-26T06:40:57.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.087+0000 7fb42e24a640 1 Processor -- start 2024-07-26T06:40:57.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.087+0000 7fb42e24a640 1 -- start start 2024-07-26T06:40:57.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.087+0000 7fb42e24a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb42806b3b0 0x7fb4281999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:57.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.087+0000 7fb42e24a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb428106300 0x7fb42819a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:57.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.087+0000 7fb42e24a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb428108ea0 0x7fb42819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:57.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.087+0000 7fb42e24a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb42810ee10 con 0x7fb42806b3b0 2024-07-26T06:40:57.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.087+0000 7fb42e24a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb42810ec90 con 0x7fb428108ea0 2024-07-26T06:40:57.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.087+0000 7fb42e24a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb42810ef90 con 0x7fb428106300 2024-07-26T06:40:57.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.087+0000 7fb4277fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb42806b3b0 0x7fb4281999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:57.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.087+0000 7fb4277fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb42806b3b0 0x7fb4281999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43700/0 (socket says 172.21.15.42:43700) 2024-07-26T06:40:57.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.087+0000 7fb4277fe640 1 -- 172.21.15.42:0/707982404 learned_addr learned my addr 172.21.15.42:0/707982404 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:40:57.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.087+0000 7fb426ffd640 1 --2- 172.21.15.42:0/707982404 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb428106300 0x7fb42819a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:57.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.088+0000 7fb427fff640 1 --2- 172.21.15.42:0/707982404 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb428108ea0 0x7fb42819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:57.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.088+0000 7fb4277fe640 1 -- 172.21.15.42:0/707982404 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb428106300 msgr2=0x7fb42819a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:57.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.088+0000 7fb4277fe640 1 --2- 172.21.15.42:0/707982404 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb428106300 0x7fb42819a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:57.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.088+0000 7fb4277fe640 1 -- 172.21.15.42:0/707982404 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb428108ea0 msgr2=0x7fb42819abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:57.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.088+0000 7fb4277fe640 1 --2- 172.21.15.42:0/707982404 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb428108ea0 0x7fb42819abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:57.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.088+0000 7fb4277fe640 1 -- 172.21.15.42:0/707982404 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb42819b3d0 con 0x7fb42806b3b0 2024-07-26T06:40:57.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.088+0000 7fb4277fe640 1 --2- 172.21.15.42:0/707982404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb42806b3b0 0x7fb4281999a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb41802aeb0 tx=0x7fb418003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:57.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.088+0000 7fb424ff9640 1 -- 172.21.15.42:0/707982404 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb418004240 con 0x7fb42806b3b0 2024-07-26T06:40:57.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.088+0000 7fb424ff9640 1 -- 172.21.15.42:0/707982404 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4180317c0 con 0x7fb42806b3b0 2024-07-26T06:40:57.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.088+0000 7fb42e24a640 1 -- 172.21.15.42:0/707982404 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb42819b660 con 0x7fb42806b3b0 2024-07-26T06:40:57.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.088+0000 7fb424ff9640 1 -- 172.21.15.42:0/707982404 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41802fc30 con 0x7fb42806b3b0 2024-07-26T06:40:57.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.088+0000 7fb42e24a640 1 -- 172.21.15.42:0/707982404 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4281a1dc0 con 0x7fb42806b3b0 2024-07-26T06:40:57.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.089+0000 7fb42e24a640 1 -- 172.21.15.42:0/707982404 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4280ff500 con 0x7fb42806b3b0 2024-07-26T06:40:57.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.091+0000 7fb424ff9640 1 -- 172.21.15.42:0/707982404 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7fb4180043e0 con 0x7fb42806b3b0 2024-07-26T06:40:57.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.091+0000 7fb424ff9640 1 --2- 172.21.15.42:0/707982404 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb4140b3bc0 0x7fb4140b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:57.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.092+0000 7fb426ffd640 1 --2- 172.21.15.42:0/707982404 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb4140b3bc0 0x7fb4140b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:57.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.092+0000 7fb424ff9640 1 -- 172.21.15.42:0/707982404 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb4181036f0 con 0x7fb42806b3b0 2024-07-26T06:40:57.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.092+0000 7fb426ffd640 1 --2- 172.21.15.42:0/707982404 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb4140b3bc0 0x7fb4140b60d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb410005d00 tx=0x7fb41000b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:57.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.096+0000 7fb424ff9640 1 -- 172.21.15.42:0/707982404 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb4180cca10 con 0x7fb42806b3b0 2024-07-26T06:40:57.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.289+0000 7fb42e24a640 1 -- 172.21.15.42:0/707982404 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7fb4280639f0 con 0x7fb42806b3b0 2024-07-26T06:40:57.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.293+0000 7fb424ff9640 1 -- 172.21.15.42:0/707982404 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v20)=0 v20) ==== 117+0+0 (secure 0 0 0) 0x7fb4180098b0 con 0x7fb42806b3b0 2024-07-26T06:40:57.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.295+0000 7fb42e24a640 1 -- 172.21.15.42:0/707982404 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb4140b3bc0 msgr2=0x7fb4140b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:57.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.295+0000 7fb42e24a640 1 --2- 172.21.15.42:0/707982404 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb4140b3bc0 0x7fb4140b60d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb410005d00 tx=0x7fb41000b410 comp rx=0 tx=0).stop 2024-07-26T06:40:57.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.295+0000 7fb42e24a640 1 -- 172.21.15.42:0/707982404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb42806b3b0 msgr2=0x7fb4281999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:57.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.295+0000 7fb42e24a640 1 --2- 172.21.15.42:0/707982404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb42806b3b0 0x7fb4281999a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb41802aeb0 tx=0x7fb418003c20 comp rx=0 tx=0).stop 2024-07-26T06:40:57.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.295+0000 7fb42e24a640 1 -- 172.21.15.42:0/707982404 shutdown_connections 2024-07-26T06:40:57.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.295+0000 7fb42e24a640 1 --2- 172.21.15.42:0/707982404 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb4140b3bc0 0x7fb4140b60d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:57.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.295+0000 7fb42e24a640 1 --2- 172.21.15.42:0/707982404 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb428108ea0 0x7fb42819abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:57.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.295+0000 7fb42e24a640 1 --2- 172.21.15.42:0/707982404 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb428106300 0x7fb42819a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:57.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.295+0000 7fb42e24a640 1 --2- 172.21.15.42:0/707982404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb42806b3b0 0x7fb4281999a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:57.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.295+0000 7fb42e24a640 1 -- 172.21.15.42:0/707982404 >> 172.21.15.42:0/707982404 conn(0x7fb4280fd9e0 msgr2=0x7fb428104810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:57.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.295+0000 7fb42e24a640 1 -- 172.21.15.42:0/707982404 shutdown_connections 2024-07-26T06:40:57.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:57.295+0000 7fb42e24a640 1 -- 172.21.15.42:0/707982404 wait complete. 2024-07-26T06:40:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.108+0000 7f8a95c4b640 1 -- 172.21.15.42:0/1372204697 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a901042a0 msgr2=0x7f8a901046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.108+0000 7f8a95c4b640 1 --2- 172.21.15.42:0/1372204697 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a901042a0 0x7f8a901046b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009900 tx=0x7f8a7802a480 comp rx=0 tx=0).stop 2024-07-26T06:40:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.108+0000 7f8a95c4b640 1 -- 172.21.15.42:0/1372204697 shutdown_connections 2024-07-26T06:40:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.108+0000 7f8a95c4b640 1 --2- 172.21.15.42:0/1372204697 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a90104c40 0x7f8a9010b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.108+0000 7f8a95c4b640 1 --2- 172.21.15.42:0/1372204697 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a901042a0 0x7f8a901046b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.108+0000 7f8a95c4b640 1 --2- 172.21.15.42:0/1372204697 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a90103210 0x7f8a90103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.108+0000 7f8a95c4b640 1 -- 172.21.15.42:0/1372204697 >> 172.21.15.42:0/1372204697 conn(0x7f8a900feb80 msgr2=0x7f8a90101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.108+0000 7f8a95c4b640 1 -- 172.21.15.42:0/1372204697 shutdown_connections 2024-07-26T06:40:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.108+0000 7f8a95c4b640 1 -- 172.21.15.42:0/1372204697 wait complete. 2024-07-26T06:40:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.109+0000 7f8a95c4b640 1 Processor -- start 2024-07-26T06:40:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.109+0000 7f8a95c4b640 1 -- start start 2024-07-26T06:40:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.110+0000 7f8a95c4b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a90103210 0x7f8a901999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.110+0000 7f8a95c4b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a901042a0 0x7f8a9019a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.110+0000 7f8a95c4b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a90104c40 0x7f8a9019ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.110+0000 7f8a95c4b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9010ee50 con 0x7f8a90104c40 2024-07-26T06:40:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.110+0000 7f8a95c4b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9010ecd0 con 0x7f8a901042a0 2024-07-26T06:40:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.110+0000 7f8a95c4b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9010efd0 con 0x7f8a90103210 2024-07-26T06:40:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.110+0000 7f8a8ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a90104c40 0x7f8a9019ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.110+0000 7f8a8ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a90104c40 0x7f8a9019ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43714/0 (socket says 172.21.15.42:43714) 2024-07-26T06:40:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.110+0000 7f8a8ffff640 1 -- 172.21.15.42:0/3517942698 learned_addr learned my addr 172.21.15.42:0/3517942698 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:40:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.110+0000 7f8a8effd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a901042a0 0x7f8a9019a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.110+0000 7f8a8f7fe640 1 --2- 172.21.15.42:0/3517942698 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a90103210 0x7f8a901999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.110+0000 7f8a8f7fe640 1 -- 172.21.15.42:0/3517942698 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a901042a0 msgr2=0x7f8a9019a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.110+0000 7f8a8f7fe640 1 --2- 172.21.15.42:0/3517942698 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a901042a0 0x7f8a9019a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.110+0000 7f8a8f7fe640 1 -- 172.21.15.42:0/3517942698 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a90104c40 msgr2=0x7f8a9019ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.110+0000 7f8a8f7fe640 1 --2- 172.21.15.42:0/3517942698 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a90104c40 0x7f8a9019ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.110+0000 7f8a8f7fe640 1 -- 172.21.15.42:0/3517942698 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a9019b420 con 0x7f8a90103210 2024-07-26T06:40:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.110+0000 7f8a8ffff640 1 --2- 172.21.15.42:0/3517942698 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a90104c40 0x7f8a9019ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:40:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.111+0000 7f8a8f7fe640 1 --2- 172.21.15.42:0/3517942698 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a90103210 0x7f8a901999f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8a8000e9f0 tx=0x7f8a8000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.111+0000 7f8a8cff9640 1 -- 172.21.15.42:0/3517942698 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8000cd80 con 0x7f8a90103210 2024-07-26T06:40:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.111+0000 7f8a8cff9640 1 -- 172.21.15.42:0/3517942698 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a80011920 con 0x7f8a90103210 2024-07-26T06:40:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.111+0000 7f8a95c4b640 1 -- 172.21.15.42:0/3517942698 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a9019b710 con 0x7f8a90103210 2024-07-26T06:40:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.111+0000 7f8a8cff9640 1 -- 172.21.15.42:0/3517942698 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a80010970 con 0x7f8a90103210 2024-07-26T06:40:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.111+0000 7f8a95c4b640 1 -- 172.21.15.42:0/3517942698 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a901a1e10 con 0x7f8a90103210 2024-07-26T06:40:58.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.112+0000 7f8a95c4b640 1 -- 172.21.15.42:0/3517942698 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a9010a3d0 con 0x7f8a90103210 2024-07-26T06:40:58.115 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:40:58.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.114+0000 7f8a8cff9640 1 -- 172.21.15.42:0/3517942698 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f8a80011470 con 0x7f8a90103210 2024-07-26T06:40:58.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.115+0000 7f8a8cff9640 1 --2- 172.21.15.42:0/3517942698 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f8a7c0b3bc0 0x7f8a7c0b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:58.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.115+0000 7f8a8effd640 1 --2- 172.21.15.42:0/3517942698 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f8a7c0b3bc0 0x7f8a7c0b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:58.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.115+0000 7f8a8cff9640 1 -- 172.21.15.42:0/3517942698 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8a800e25c0 con 0x7f8a90103210 2024-07-26T06:40:58.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.115+0000 7f8a8effd640 1 --2- 172.21.15.42:0/3517942698 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f8a7c0b3bc0 0x7f8a7c0b60d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8a78004770 tx=0x7f8a780314a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:58.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.117+0000 7f8a8cff9640 1 -- 172.21.15.42:0/3517942698 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8a80016030 con 0x7f8a90103210 2024-07-26T06:40:58.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.311+0000 7f8a95c4b640 1 -- 172.21.15.42:0/3517942698 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f8a9002e0c0 con 0x7f8a90103210 2024-07-26T06:40:58.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.316+0000 7f8a8cff9640 1 -- 172.21.15.42:0/3517942698 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v21)=0 v21) ==== 120+0+0 (secure 0 0 0) 0x7f8a800ab8c0 con 0x7f8a90103210 2024-07-26T06:40:58.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.318+0000 7f8a95c4b640 1 -- 172.21.15.42:0/3517942698 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f8a7c0b3bc0 msgr2=0x7f8a7c0b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:58.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.318+0000 7f8a95c4b640 1 --2- 172.21.15.42:0/3517942698 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f8a7c0b3bc0 0x7f8a7c0b60d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8a78004770 tx=0x7f8a780314a0 comp rx=0 tx=0).stop 2024-07-26T06:40:58.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.318+0000 7f8a95c4b640 1 -- 172.21.15.42:0/3517942698 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a90103210 msgr2=0x7f8a901999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:58.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.318+0000 7f8a95c4b640 1 --2- 172.21.15.42:0/3517942698 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a90103210 0x7f8a901999f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8a8000e9f0 tx=0x7f8a8000eeb0 comp rx=0 tx=0).stop 2024-07-26T06:40:58.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.318+0000 7f8a95c4b640 1 -- 172.21.15.42:0/3517942698 shutdown_connections 2024-07-26T06:40:58.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.318+0000 7f8a95c4b640 1 --2- 172.21.15.42:0/3517942698 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f8a7c0b3bc0 0x7f8a7c0b60d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:58.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.318+0000 7f8a95c4b640 1 --2- 172.21.15.42:0/3517942698 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a90104c40 0x7f8a9019ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:58.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.318+0000 7f8a95c4b640 1 --2- 172.21.15.42:0/3517942698 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a901042a0 0x7f8a9019a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:58.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.318+0000 7f8a95c4b640 1 --2- 172.21.15.42:0/3517942698 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a90103210 0x7f8a901999f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:58.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.319+0000 7f8a95c4b640 1 -- 172.21.15.42:0/3517942698 >> 172.21.15.42:0/3517942698 conn(0x7f8a900feb80 msgr2=0x7f8a901075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:58.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.319+0000 7f8a95c4b640 1 -- 172.21.15.42:0/3517942698 shutdown_connections 2024-07-26T06:40:58.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:58.319+0000 7f8a95c4b640 1 -- 172.21.15.42:0/3517942698 wait complete. 2024-07-26T06:40:58.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:58 smithi042 ceph-mon[32775]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:58.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:58 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/707982404' entity='client.admin' 2024-07-26T06:40:58.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:58 smithi164 ceph-mon[36970]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:58.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:58 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/707982404' entity='client.admin' 2024-07-26T06:40:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:58 smithi163 ceph-mon[36873]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:40:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:58 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/707982404' entity='client.admin' 2024-07-26T06:40:59.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.473+0000 7f4b8c1e7640 1 -- 172.21.15.42:0/3526049710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b84106300 msgr2=0x7f4b84108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:59.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.473+0000 7f4b8c1e7640 1 --2- 172.21.15.42:0/3526049710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b84106300 0x7f4b84108760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4b74009900 tx=0x7f4b7402a410 comp rx=0 tx=0).stop 2024-07-26T06:40:59.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.473+0000 7f4b8c1e7640 1 -- 172.21.15.42:0/3526049710 shutdown_connections 2024-07-26T06:40:59.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.473+0000 7f4b8c1e7640 1 --2- 172.21.15.42:0/3526049710 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b84108ea0 0x7f4b8410b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:59.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.473+0000 7f4b8c1e7640 1 --2- 172.21.15.42:0/3526049710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b84106300 0x7f4b84108760 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:59.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.473+0000 7f4b8c1e7640 1 --2- 172.21.15.42:0/3526049710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b8406b3b0 0x7f4b84105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:59.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.473+0000 7f4b8c1e7640 1 -- 172.21.15.42:0/3526049710 >> 172.21.15.42:0/3526049710 conn(0x7f4b840fd9e0 msgr2=0x7f4b840ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:59.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.473+0000 7f4b8c1e7640 1 -- 172.21.15.42:0/3526049710 shutdown_connections 2024-07-26T06:40:59.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.473+0000 7f4b8c1e7640 1 -- 172.21.15.42:0/3526049710 wait complete. 2024-07-26T06:40:59.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b8c1e7640 1 Processor -- start 2024-07-26T06:40:59.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b8c1e7640 1 -- start start 2024-07-26T06:40:59.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b8c1e7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8406b3b0 0x7f4b841999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b8c1e7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b84106300 0x7f4b8419a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b8c1e7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b84108ea0 0x7f4b8419abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b8c1e7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8410edc0 con 0x7f4b8406b3b0 2024-07-26T06:40:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b8c1e7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8410ec40 con 0x7f4b84106300 2024-07-26T06:40:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b8c1e7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8410ef40 con 0x7f4b84108ea0 2024-07-26T06:40:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b89f83640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8406b3b0 0x7f4b841999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b89f83640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8406b3b0 0x7f4b841999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43734/0 (socket says 172.21.15.42:43734) 2024-07-26T06:40:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b89f83640 1 -- 172.21.15.42:0/1956392435 learned_addr learned my addr 172.21.15.42:0/1956392435 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:40:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b89782640 1 --2- 172.21.15.42:0/1956392435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b84106300 0x7f4b8419a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b8a784640 1 --2- 172.21.15.42:0/1956392435 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b84108ea0 0x7f4b8419abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b89f83640 1 -- 172.21.15.42:0/1956392435 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b84108ea0 msgr2=0x7f4b8419abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b89f83640 1 --2- 172.21.15.42:0/1956392435 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b84108ea0 0x7f4b8419abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:59.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b89f83640 1 -- 172.21.15.42:0/1956392435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b84106300 msgr2=0x7f4b8419a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b89f83640 1 --2- 172.21.15.42:0/1956392435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b84106300 0x7f4b8419a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b89f83640 1 -- 172.21.15.42:0/1956392435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8419b3d0 con 0x7f4b8406b3b0 2024-07-26T06:40:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.474+0000 7f4b8a784640 1 --2- 172.21.15.42:0/1956392435 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b84108ea0 0x7f4b8419abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:40:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.475+0000 7f4b89f83640 1 --2- 172.21.15.42:0/1956392435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8406b3b0 0x7f4b841999a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000d950 tx=0x7f4b8000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.475+0000 7f4b7affd640 1 -- 172.21.15.42:0/1956392435 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b80017070 con 0x7f4b8406b3b0 2024-07-26T06:40:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.475+0000 7f4b7affd640 1 -- 172.21.15.42:0/1956392435 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b80009c30 con 0x7f4b8406b3b0 2024-07-26T06:40:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.475+0000 7f4b8c1e7640 1 -- 172.21.15.42:0/1956392435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b8419b6c0 con 0x7f4b8406b3b0 2024-07-26T06:40:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.475+0000 7f4b7affd640 1 -- 172.21.15.42:0/1956392435 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b8000f620 con 0x7f4b8406b3b0 2024-07-26T06:40:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.475+0000 7f4b8c1e7640 1 -- 172.21.15.42:0/1956392435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b841a1dc0 con 0x7f4b8406b3b0 2024-07-26T06:40:59.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.476+0000 7f4b8c1e7640 1 -- 172.21.15.42:0/1956392435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b840ff500 con 0x7f4b8406b3b0 2024-07-26T06:40:59.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.477+0000 7f4b7affd640 1 -- 172.21.15.42:0/1956392435 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f4b80010460 con 0x7f4b8406b3b0 2024-07-26T06:40:59.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.478+0000 7f4b7affd640 1 --2- 172.21.15.42:0/1956392435 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f4b700b3c40 0x7f4b700b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:40:59.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.479+0000 7f4b89782640 1 --2- 172.21.15.42:0/1956392435 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f4b700b3c40 0x7f4b700b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:40:59.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.479+0000 7f4b7affd640 1 -- 172.21.15.42:0/1956392435 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4b800e1c00 con 0x7f4b8406b3b0 2024-07-26T06:40:59.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.479+0000 7f4b89782640 1 --2- 172.21.15.42:0/1956392435 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f4b700b3c40 0x7f4b700b6150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4b74031440 tx=0x7f4b740314a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:40:59.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.483+0000 7f4b7affd640 1 -- 172.21.15.42:0/1956392435 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4b800aae80 con 0x7f4b8406b3b0 2024-07-26T06:40:59.484 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:40:59 smithi042 ceph-mon[32775]: from='client.? ' entity='client.admin' 2024-07-26T06:40:59.574 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:40:59 smithi163 ceph-mon[36873]: from='client.? ' entity='client.admin' 2024-07-26T06:40:59.623 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:40:59 smithi164 ceph-mon[36970]: from='client.? ' entity='client.admin' 2024-07-26T06:40:59.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.668+0000 7f4b8c1e7640 1 -- 172.21.15.42:0/1956392435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) -- 0x7f4b8402e0c0 con 0x7f4b8406b3b0 2024-07-26T06:40:59.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.672+0000 7f4b7affd640 1 -- 172.21.15.42:0/1956392435 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v22)=0 v22) ==== 116+0+0 (secure 0 0 0) 0x7f4b80009de0 con 0x7f4b8406b3b0 2024-07-26T06:40:59.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.674+0000 7f4b8c1e7640 1 -- 172.21.15.42:0/1956392435 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f4b700b3c40 msgr2=0x7f4b700b6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:59.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.674+0000 7f4b8c1e7640 1 --2- 172.21.15.42:0/1956392435 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f4b700b3c40 0x7f4b700b6150 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4b74031440 tx=0x7f4b740314a0 comp rx=0 tx=0).stop 2024-07-26T06:40:59.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.674+0000 7f4b8c1e7640 1 -- 172.21.15.42:0/1956392435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8406b3b0 msgr2=0x7f4b841999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:40:59.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.674+0000 7f4b8c1e7640 1 --2- 172.21.15.42:0/1956392435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8406b3b0 0x7f4b841999a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000d950 tx=0x7f4b8000de10 comp rx=0 tx=0).stop 2024-07-26T06:40:59.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.674+0000 7f4b8c1e7640 1 -- 172.21.15.42:0/1956392435 shutdown_connections 2024-07-26T06:40:59.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.674+0000 7f4b8c1e7640 1 --2- 172.21.15.42:0/1956392435 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f4b700b3c40 0x7f4b700b6150 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:59.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.674+0000 7f4b8c1e7640 1 --2- 172.21.15.42:0/1956392435 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b84108ea0 0x7f4b8419abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:59.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.674+0000 7f4b8c1e7640 1 --2- 172.21.15.42:0/1956392435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b84106300 0x7f4b8419a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:59.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.674+0000 7f4b8c1e7640 1 --2- 172.21.15.42:0/1956392435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8406b3b0 0x7f4b841999a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:40:59.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.674+0000 7f4b8c1e7640 1 -- 172.21.15.42:0/1956392435 >> 172.21.15.42:0/1956392435 conn(0x7f4b840fd9e0 msgr2=0x7f4b841047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:40:59.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.675+0000 7f4b8c1e7640 1 -- 172.21.15.42:0/1956392435 shutdown_connections 2024-07-26T06:40:59.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:40:59.675+0000 7f4b8c1e7640 1 -- 172.21.15.42:0/1956392435 wait complete. 2024-07-26T06:41:00.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.393+0000 7f1a347c5640 1 -- 172.21.15.42:0/2255169313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a2c102080 msgr2=0x7f1a2c1024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:00.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.393+0000 7f1a347c5640 1 --2- 172.21.15.42:0/2255169313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a2c102080 0x7f1a2c1024f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1a28009900 tx=0x7f1a2802a480 comp rx=0 tx=0).stop 2024-07-26T06:41:00.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.394+0000 7f1a347c5640 1 -- 172.21.15.42:0/2255169313 shutdown_connections 2024-07-26T06:41:00.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.394+0000 7f1a347c5640 1 --2- 172.21.15.42:0/2255169313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a2c103ab0 0x7f1a2c10a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:00.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.394+0000 7f1a347c5640 1 --2- 172.21.15.42:0/2255169313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a2c103110 0x7f1a2c103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:00.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.394+0000 7f1a347c5640 1 --2- 172.21.15.42:0/2255169313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a2c102080 0x7f1a2c1024f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:00.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.394+0000 7f1a347c5640 1 -- 172.21.15.42:0/2255169313 >> 172.21.15.42:0/2255169313 conn(0x7f1a2c0fda10 msgr2=0x7f1a2c0ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:00.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.394+0000 7f1a347c5640 1 -- 172.21.15.42:0/2255169313 shutdown_connections 2024-07-26T06:41:00.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.394+0000 7f1a347c5640 1 -- 172.21.15.42:0/2255169313 wait complete. 2024-07-26T06:41:00.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.394+0000 7f1a347c5640 1 Processor -- start 2024-07-26T06:41:00.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.394+0000 7f1a347c5640 1 -- start start 2024-07-26T06:41:00.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.395+0000 7f1a347c5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a2c102080 0x7f1a2c10c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:00.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.395+0000 7f1a347c5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a2c103110 0x7f1a2c10ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:00.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.395+0000 7f1a347c5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a2c103ab0 0x7f1a2c10b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:00.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.395+0000 7f1a347c5640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c10ede0 con 0x7f1a2c103110 2024-07-26T06:41:00.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.395+0000 7f1a347c5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c10ec60 con 0x7f1a2c102080 2024-07-26T06:41:00.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.395+0000 7f1a347c5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2c10ef60 con 0x7f1a2c103ab0 2024-07-26T06:41:00.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.395+0000 7f1a31d60640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a2c103110 0x7f1a2c10ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:00.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.395+0000 7f1a31d60640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a2c103110 0x7f1a2c10ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43758/0 (socket says 172.21.15.42:43758) 2024-07-26T06:41:00.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.395+0000 7f1a31d60640 1 -- 172.21.15.42:0/90319287 learned_addr learned my addr 172.21.15.42:0/90319287 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:41:00.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.395+0000 7f1a32d62640 1 --2- 172.21.15.42:0/90319287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a2c103ab0 0x7f1a2c10b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:00.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.395+0000 7f1a32561640 1 --2- 172.21.15.42:0/90319287 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a2c102080 0x7f1a2c10c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:00.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.395+0000 7f1a31d60640 1 -- 172.21.15.42:0/90319287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a2c103ab0 msgr2=0x7f1a2c10b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:00.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.395+0000 7f1a31d60640 1 --2- 172.21.15.42:0/90319287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a2c103ab0 0x7f1a2c10b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:00.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.395+0000 7f1a31d60640 1 -- 172.21.15.42:0/90319287 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a2c102080 msgr2=0x7f1a2c10c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:00.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.395+0000 7f1a31d60640 1 --2- 172.21.15.42:0/90319287 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a2c102080 0x7f1a2c10c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:00.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.395+0000 7f1a31d60640 1 -- 172.21.15.42:0/90319287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a2c10bad0 con 0x7f1a2c103110 2024-07-26T06:41:00.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.395+0000 7f1a31d60640 1 --2- 172.21.15.42:0/90319287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a2c103110 0x7f1a2c10ae90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c00d770 tx=0x7f1a1c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:00.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.396+0000 7f1a237fe640 1 -- 172.21.15.42:0/90319287 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c017070 con 0x7f1a2c103110 2024-07-26T06:41:00.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.396+0000 7f1a237fe640 1 -- 172.21.15.42:0/90319287 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a1c0054d0 con 0x7f1a2c103110 2024-07-26T06:41:00.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.396+0000 7f1a237fe640 1 -- 172.21.15.42:0/90319287 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1c004c30 con 0x7f1a2c103110 2024-07-26T06:41:00.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.396+0000 7f1a347c5640 1 -- 172.21.15.42:0/90319287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a2c1a3ab0 con 0x7f1a2c103110 2024-07-26T06:41:00.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.396+0000 7f1a347c5640 1 -- 172.21.15.42:0/90319287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a2c1a3f10 con 0x7f1a2c103110 2024-07-26T06:41:00.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.397+0000 7f1a347c5640 1 -- 172.21.15.42:0/90319287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a2c109260 con 0x7f1a2c103110 2024-07-26T06:41:00.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.398+0000 7f1a237fe640 1 -- 172.21.15.42:0/90319287 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f1a1c004dd0 con 0x7f1a2c103110 2024-07-26T06:41:00.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.399+0000 7f1a237fe640 1 --2- 172.21.15.42:0/90319287 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1a180b3bc0 0x7f1a180b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:00.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.399+0000 7f1a32561640 1 --2- 172.21.15.42:0/90319287 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1a180b3bc0 0x7f1a180b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:00.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.399+0000 7f1a237fe640 1 -- 172.21.15.42:0/90319287 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1a1c0e1cb0 con 0x7f1a2c103110 2024-07-26T06:41:00.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.400+0000 7f1a32561640 1 --2- 172.21.15.42:0/90319287 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1a180b3bc0 0x7f1a180b60d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1a28000c00 tx=0x7f1a28031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:00.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.402+0000 7f1a237fe640 1 -- 172.21.15.42:0/90319287 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1a1c0aaa00 con 0x7f1a2c103110 2024-07-26T06:41:00.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.591+0000 7f1a347c5640 1 -- 172.21.15.42:0/90319287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f1a2c02e0c0 con 0x7f1a2c103110 2024-07-26T06:41:00.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.597+0000 7f1a237fe640 1 -- 172.21.15.42:0/90319287 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v23)=0 v23) ==== 106+0+0 (secure 0 0 0) 0x7f1a1c0aef50 con 0x7f1a2c103110 2024-07-26T06:41:00.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.599+0000 7f1a347c5640 1 -- 172.21.15.42:0/90319287 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1a180b3bc0 msgr2=0x7f1a180b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:00.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.599+0000 7f1a347c5640 1 --2- 172.21.15.42:0/90319287 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1a180b3bc0 0x7f1a180b60d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1a28000c00 tx=0x7f1a28031400 comp rx=0 tx=0).stop 2024-07-26T06:41:00.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.599+0000 7f1a347c5640 1 -- 172.21.15.42:0/90319287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a2c103110 msgr2=0x7f1a2c10ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:00.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.599+0000 7f1a347c5640 1 --2- 172.21.15.42:0/90319287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a2c103110 0x7f1a2c10ae90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c00d770 tx=0x7f1a1c00dc30 comp rx=0 tx=0).stop 2024-07-26T06:41:00.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.599+0000 7f1a347c5640 1 -- 172.21.15.42:0/90319287 shutdown_connections 2024-07-26T06:41:00.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.599+0000 7f1a347c5640 1 --2- 172.21.15.42:0/90319287 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1a180b3bc0 0x7f1a180b60d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.599+0000 7f1a347c5640 1 --2- 172.21.15.42:0/90319287 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a2c103ab0 0x7f1a2c10b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.599+0000 7f1a347c5640 1 --2- 172.21.15.42:0/90319287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1a2c103110 0x7f1a2c10ae90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.599+0000 7f1a347c5640 1 --2- 172.21.15.42:0/90319287 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a2c102080 0x7f1a2c10c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.599+0000 7f1a347c5640 1 -- 172.21.15.42:0/90319287 >> 172.21.15.42:0/90319287 conn(0x7f1a2c0fda10 msgr2=0x7f1a2c106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.599+0000 7f1a347c5640 1 -- 172.21.15.42:0/90319287 shutdown_connections 2024-07-26T06:41:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.599+0000 7f1a347c5640 1 -- 172.21.15.42:0/90319287 wait complete. 2024-07-26T06:41:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:00 smithi164 ceph-mon[36970]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:00 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1956392435' entity='client.admin' 2024-07-26T06:41:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:00 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/90319287' entity='client.admin' 2024-07-26T06:41:01.001 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:00 smithi042 ceph-mon[32775]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:01.001 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:00 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1956392435' entity='client.admin' 2024-07-26T06:41:01.001 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:00 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/90319287' entity='client.admin' 2024-07-26T06:41:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:00 smithi163 ceph-mon[36873]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:00 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1956392435' entity='client.admin' 2024-07-26T06:41:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:00 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/90319287' entity='client.admin' 2024-07-26T06:41:01.942 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi042') 2024-07-26T06:41:01.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.574+0000 7fa73d55a640 1 -- 172.21.15.42:0/4834946 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa738106300 msgr2=0x7fa738108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:01.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.574+0000 7fa73d55a640 1 --2- 172.21.15.42:0/4834946 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa738106300 0x7fa738108760 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa72c009900 tx=0x7fa72c02a410 comp rx=0 tx=0).stop 2024-07-26T06:41:02.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.574+0000 7fa73d55a640 1 -- 172.21.15.42:0/4834946 shutdown_connections 2024-07-26T06:41:02.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.574+0000 7fa73d55a640 1 --2- 172.21.15.42:0/4834946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa738108ea0 0x7fa73810b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:02.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.574+0000 7fa73d55a640 1 --2- 172.21.15.42:0/4834946 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa738106300 0x7fa738108760 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:02.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.574+0000 7fa73d55a640 1 --2- 172.21.15.42:0/4834946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa73806b3b0 0x7fa738105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:02.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.574+0000 7fa73d55a640 1 -- 172.21.15.42:0/4834946 >> 172.21.15.42:0/4834946 conn(0x7fa7380fd9e0 msgr2=0x7fa7380ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:02.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.574+0000 7fa73d55a640 1 -- 172.21.15.42:0/4834946 shutdown_connections 2024-07-26T06:41:02.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.574+0000 7fa73d55a640 1 -- 172.21.15.42:0/4834946 wait complete. 2024-07-26T06:41:02.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.575+0000 7fa73d55a640 1 Processor -- start 2024-07-26T06:41:02.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.575+0000 7fa73d55a640 1 -- start start 2024-07-26T06:41:02.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.576+0000 7fa73d55a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa73806b3b0 0x7fa7381999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:02.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.576+0000 7fa73d55a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa738106300 0x7fa73819a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:02.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.576+0000 7fa73d55a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa738108ea0 0x7fa73819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:02.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.576+0000 7fa73d55a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa73810edc0 con 0x7fa738108ea0 2024-07-26T06:41:02.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.576+0000 7fa73d55a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa73810ec40 con 0x7fa73806b3b0 2024-07-26T06:41:02.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.576+0000 7fa73d55a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa73810ef40 con 0x7fa738106300 2024-07-26T06:41:02.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.576+0000 7fa7377fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa738108ea0 0x7fa73819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:02.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.576+0000 7fa7377fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa738108ea0 0x7fa73819abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43768/0 (socket says 172.21.15.42:43768) 2024-07-26T06:41:02.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.576+0000 7fa7377fe640 1 -- 172.21.15.42:0/2377815016 learned_addr learned my addr 172.21.15.42:0/2377815016 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:41:02.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.576+0000 7fa7367fc640 1 --2- 172.21.15.42:0/2377815016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa738106300 0x7fa73819a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:02.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.576+0000 7fa736ffd640 1 --2- 172.21.15.42:0/2377815016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa73806b3b0 0x7fa7381999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:02.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.576+0000 7fa7377fe640 1 -- 172.21.15.42:0/2377815016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa738106300 msgr2=0x7fa73819a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:02.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.576+0000 7fa7377fe640 1 --2- 172.21.15.42:0/2377815016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa738106300 0x7fa73819a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:02.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.576+0000 7fa7377fe640 1 -- 172.21.15.42:0/2377815016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa73806b3b0 msgr2=0x7fa7381999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:02.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.576+0000 7fa7377fe640 1 --2- 172.21.15.42:0/2377815016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa73806b3b0 0x7fa7381999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:02.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.576+0000 7fa7377fe640 1 -- 172.21.15.42:0/2377815016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa73819b3d0 con 0x7fa738108ea0 2024-07-26T06:41:02.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.577+0000 7fa7377fe640 1 --2- 172.21.15.42:0/2377815016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa738108ea0 0x7fa73819abb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa73000eb10 tx=0x7fa73000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:02.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.577+0000 7fa71ffff640 1 -- 172.21.15.42:0/2377815016 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa730004b20 con 0x7fa738108ea0 2024-07-26T06:41:02.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.577+0000 7fa71ffff640 1 -- 172.21.15.42:0/2377815016 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa730004cc0 con 0x7fa738108ea0 2024-07-26T06:41:02.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.577+0000 7fa73d55a640 1 -- 172.21.15.42:0/2377815016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa73819b6c0 con 0x7fa738108ea0 2024-07-26T06:41:02.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.577+0000 7fa71ffff640 1 -- 172.21.15.42:0/2377815016 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa730010670 con 0x7fa738108ea0 2024-07-26T06:41:02.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.577+0000 7fa73d55a640 1 -- 172.21.15.42:0/2377815016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7381a1dc0 con 0x7fa738108ea0 2024-07-26T06:41:02.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.578+0000 7fa73d55a640 1 -- 172.21.15.42:0/2377815016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7380ff500 con 0x7fa738108ea0 2024-07-26T06:41:02.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.580+0000 7fa71ffff640 1 -- 172.21.15.42:0/2377815016 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7fa730005000 con 0x7fa738108ea0 2024-07-26T06:41:02.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.580+0000 7fa71ffff640 1 --2- 172.21.15.42:0/2377815016 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa7200b3b40 0x7fa7200b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:02.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.580+0000 7fa736ffd640 1 --2- 172.21.15.42:0/2377815016 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa7200b3b40 0x7fa7200b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:02.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.580+0000 7fa71ffff640 1 -- 172.21.15.42:0/2377815016 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fa7300e2620 con 0x7fa738108ea0 2024-07-26T06:41:02.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.581+0000 7fa736ffd640 1 --2- 172.21.15.42:0/2377815016 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa7200b3b40 0x7fa7200b6050 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa72800bbe0 tx=0x7fa72800b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:02.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.583+0000 7fa71ffff640 1 -- 172.21.15.42:0/2377815016 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa730055600 con 0x7fa738108ea0 2024-07-26T06:41:02.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.800+0000 7fa73d55a640 1 -- 172.21.15.42:0/2377815016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7fa73802e0c0 con 0x7fa738108ea0 2024-07-26T06:41:02.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.804+0000 7fa71ffff640 1 -- 172.21.15.42:0/2377815016 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v24)=0 v24) ==== 121+0+0 (secure 0 0 0) 0x7fa7300ab940 con 0x7fa738108ea0 2024-07-26T06:41:02.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.807+0000 7fa73d55a640 1 -- 172.21.15.42:0/2377815016 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa7200b3b40 msgr2=0x7fa7200b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:02.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.807+0000 7fa73d55a640 1 --2- 172.21.15.42:0/2377815016 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa7200b3b40 0x7fa7200b6050 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa72800bbe0 tx=0x7fa72800b390 comp rx=0 tx=0).stop 2024-07-26T06:41:02.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.807+0000 7fa73d55a640 1 -- 172.21.15.42:0/2377815016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa738108ea0 msgr2=0x7fa73819abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:02.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.807+0000 7fa73d55a640 1 --2- 172.21.15.42:0/2377815016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa738108ea0 0x7fa73819abb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa73000eb10 tx=0x7fa73000c2d0 comp rx=0 tx=0).stop 2024-07-26T06:41:02.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.807+0000 7fa73d55a640 1 -- 172.21.15.42:0/2377815016 shutdown_connections 2024-07-26T06:41:02.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.807+0000 7fa73d55a640 1 --2- 172.21.15.42:0/2377815016 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa7200b3b40 0x7fa7200b6050 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:02.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.807+0000 7fa73d55a640 1 --2- 172.21.15.42:0/2377815016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa738108ea0 0x7fa73819abb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:02.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.807+0000 7fa73d55a640 1 --2- 172.21.15.42:0/2377815016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa738106300 0x7fa73819a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:02.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.807+0000 7fa73d55a640 1 --2- 172.21.15.42:0/2377815016 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa73806b3b0 0x7fa7381999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:02.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.807+0000 7fa73d55a640 1 -- 172.21.15.42:0/2377815016 >> 172.21.15.42:0/2377815016 conn(0x7fa7380fd9e0 msgr2=0x7fa7381047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:02.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.807+0000 7fa73d55a640 1 -- 172.21.15.42:0/2377815016 shutdown_connections 2024-07-26T06:41:02.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:00.807+0000 7fa73d55a640 1 -- 172.21.15.42:0/2377815016 wait complete. 2024-07-26T06:41:02.121 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:01 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2377815016' entity='client.admin' 2024-07-26T06:41:02.198 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:01 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2377815016' entity='client.admin' 2024-07-26T06:41:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:01 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2377815016' entity='client.admin' 2024-07-26T06:41:03.122 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:02 smithi042 ceph-mon[32775]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:02 smithi164 ceph-mon[36970]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:03.252 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'], check_status=True, hostname='smithi042') 2024-07-26T06:41:03.253 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'], check_status=True, hostname='smithi042') 2024-07-26T06:41:03.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:02 smithi163 ceph-mon[36873]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:04.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.748+0000 7fdb55e09640 1 -- 172.21.15.42:0/1371660002 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb50103150 msgr2=0x7fdb50103560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:04.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.748+0000 7fdb55e09640 1 --2- 172.21.15.42:0/1371660002 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb50103150 0x7fdb50103560 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdb44009900 tx=0x7fdb4402a480 comp rx=0 tx=0).stop 2024-07-26T06:41:04.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.748+0000 7fdb55e09640 1 -- 172.21.15.42:0/1371660002 shutdown_connections 2024-07-26T06:41:04.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.748+0000 7fdb55e09640 1 --2- 172.21.15.42:0/1371660002 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb50103af0 0x7fdb5010a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:04.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.748+0000 7fdb55e09640 1 --2- 172.21.15.42:0/1371660002 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb50103150 0x7fdb50103560 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:04.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.748+0000 7fdb55e09640 1 --2- 172.21.15.42:0/1371660002 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb501020c0 0x7fdb50102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:04.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.748+0000 7fdb55e09640 1 -- 172.21.15.42:0/1371660002 >> 172.21.15.42:0/1371660002 conn(0x7fdb500fda30 msgr2=0x7fdb500ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:04.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.748+0000 7fdb55e09640 1 -- 172.21.15.42:0/1371660002 shutdown_connections 2024-07-26T06:41:04.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.748+0000 7fdb55e09640 1 -- 172.21.15.42:0/1371660002 wait complete. 2024-07-26T06:41:04.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.749+0000 7fdb55e09640 1 Processor -- start 2024-07-26T06:41:04.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.749+0000 7fdb55e09640 1 -- start start 2024-07-26T06:41:04.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb55e09640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb501020c0 0x7fdb5006bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:04.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb55e09640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb50103150 0x7fdb5019e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:04.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb55e09640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb50103af0 0x7fdb501a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:04.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb55e09640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5010ddb0 con 0x7fdb50103af0 2024-07-26T06:41:04.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb55e09640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5010dc30 con 0x7fdb501020c0 2024-07-26T06:41:04.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb55e09640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5010df30 con 0x7fdb50103150 2024-07-26T06:41:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb4ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb50103af0 0x7fdb501a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb4ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb50103af0 0x7fdb501a0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43778/0 (socket says 172.21.15.42:43778) 2024-07-26T06:41:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb4ffff640 1 -- 172.21.15.42:0/3143943272 learned_addr learned my addr 172.21.15.42:0/3143943272 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:41:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb4effd640 1 --2- 172.21.15.42:0/3143943272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb50103150 0x7fdb5019e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb4f7fe640 1 --2- 172.21.15.42:0/3143943272 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb501020c0 0x7fdb5006bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb4ffff640 1 -- 172.21.15.42:0/3143943272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb50103150 msgr2=0x7fdb5019e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb4ffff640 1 --2- 172.21.15.42:0/3143943272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb50103150 0x7fdb5019e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb4ffff640 1 -- 172.21.15.42:0/3143943272 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb501020c0 msgr2=0x7fdb5006bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb4ffff640 1 --2- 172.21.15.42:0/3143943272 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb501020c0 0x7fdb5006bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb4ffff640 1 -- 172.21.15.42:0/3143943272 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb501a1540 con 0x7fdb50103af0 2024-07-26T06:41:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb4ffff640 1 --2- 172.21.15.42:0/3143943272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb50103af0 0x7fdb501a0d40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdb4800b970 tx=0x7fdb4800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:04.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.750+0000 7fdb4cff9640 1 -- 172.21.15.42:0/3143943272 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb48019070 con 0x7fdb50103af0 2024-07-26T06:41:04.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.751+0000 7fdb4cff9640 1 -- 172.21.15.42:0/3143943272 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb48014440 con 0x7fdb50103af0 2024-07-26T06:41:04.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.751+0000 7fdb55e09640 1 -- 172.21.15.42:0/3143943272 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb50198620 con 0x7fdb50103af0 2024-07-26T06:41:04.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.751+0000 7fdb4cff9640 1 -- 172.21.15.42:0/3143943272 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb480075d0 con 0x7fdb50103af0 2024-07-26T06:41:04.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.751+0000 7fdb55e09640 1 -- 172.21.15.42:0/3143943272 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb50198be0 con 0x7fdb50103af0 2024-07-26T06:41:04.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.752+0000 7fdb55e09640 1 -- 172.21.15.42:0/3143943272 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb5019a5b0 con 0x7fdb50103af0 2024-07-26T06:41:04.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.753+0000 7fdb4cff9640 1 -- 172.21.15.42:0/3143943272 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7fdb4801c070 con 0x7fdb50103af0 2024-07-26T06:41:04.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.754+0000 7fdb4cff9640 1 --2- 172.21.15.42:0/3143943272 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdb380b3bc0 0x7fdb380b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:04.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.754+0000 7fdb4f7fe640 1 --2- 172.21.15.42:0/3143943272 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdb380b3bc0 0x7fdb380b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:04.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.754+0000 7fdb4cff9640 1 -- 172.21.15.42:0/3143943272 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdb480e18b0 con 0x7fdb50103af0 2024-07-26T06:41:04.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.754+0000 7fdb4f7fe640 1 --2- 172.21.15.42:0/3143943272 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdb380b3bc0 0x7fdb380b60d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdb400058c0 tx=0x7fdb40010040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:04.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.757+0000 7fdb4cff9640 1 -- 172.21.15.42:0/3143943272 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdb480aa580 con 0x7fdb50103af0 2024-07-26T06:41:04.847 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:04 smithi042 ceph-mon[32775]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:04.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.945+0000 7fdb55e09640 1 -- 172.21.15.42:0/3143943272 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7fdb5002e0c0 con 0x7fdb50103af0 2024-07-26T06:41:04.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.949+0000 7fdb4cff9640 1 -- 172.21.15.42:0/3143943272 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v25)=0 v25) ==== 121+0+0 (secure 0 0 0) 0x7fdb480aead0 con 0x7fdb50103af0 2024-07-26T06:41:04.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.952+0000 7fdb55e09640 1 -- 172.21.15.42:0/3143943272 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdb380b3bc0 msgr2=0x7fdb380b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:04.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.952+0000 7fdb55e09640 1 --2- 172.21.15.42:0/3143943272 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdb380b3bc0 0x7fdb380b60d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdb400058c0 tx=0x7fdb40010040 comp rx=0 tx=0).stop 2024-07-26T06:41:04.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.952+0000 7fdb55e09640 1 -- 172.21.15.42:0/3143943272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb50103af0 msgr2=0x7fdb501a0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:04.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.952+0000 7fdb55e09640 1 --2- 172.21.15.42:0/3143943272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb50103af0 0x7fdb501a0d40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdb4800b970 tx=0x7fdb4800be30 comp rx=0 tx=0).stop 2024-07-26T06:41:04.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.952+0000 7fdb55e09640 1 -- 172.21.15.42:0/3143943272 shutdown_connections 2024-07-26T06:41:04.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.952+0000 7fdb55e09640 1 --2- 172.21.15.42:0/3143943272 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdb380b3bc0 0x7fdb380b60d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:04.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.952+0000 7fdb55e09640 1 --2- 172.21.15.42:0/3143943272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb50103af0 0x7fdb501a0d40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:04.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.952+0000 7fdb55e09640 1 --2- 172.21.15.42:0/3143943272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb50103150 0x7fdb5019e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:04.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.952+0000 7fdb55e09640 1 --2- 172.21.15.42:0/3143943272 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb501020c0 0x7fdb5006bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:04.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.952+0000 7fdb55e09640 1 -- 172.21.15.42:0/3143943272 >> 172.21.15.42:0/3143943272 conn(0x7fdb500fda30 msgr2=0x7fdb50106450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:04.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.952+0000 7fdb55e09640 1 -- 172.21.15.42:0/3143943272 shutdown_connections 2024-07-26T06:41:04.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:04.952+0000 7fdb55e09640 1 -- 172.21.15.42:0/3143943272 wait complete. 2024-07-26T06:41:05.168 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'], check_status=True, hostname='smithi042') 2024-07-26T06:41:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:04 smithi164 ceph-mon[36970]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:05.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:04 smithi163 ceph-mon[36873]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:05.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.505+0000 7f4dee07e640 1 -- 172.21.15.42:0/2542681065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4de8104230 msgr2=0x7f4de8104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:05.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.505+0000 7f4dee07e640 1 --2- 172.21.15.42:0/2542681065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4de8104230 0x7f4de8104640 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0009900 tx=0x7f4dd002a480 comp rx=0 tx=0).stop 2024-07-26T06:41:05.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.505+0000 7f4dee07e640 1 -- 172.21.15.42:0/2542681065 shutdown_connections 2024-07-26T06:41:05.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.505+0000 7f4dee07e640 1 --2- 172.21.15.42:0/2542681065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4de8104bd0 0x7f4de810b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:05.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.505+0000 7f4dee07e640 1 --2- 172.21.15.42:0/2542681065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4de8104230 0x7f4de8104640 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:05.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.505+0000 7f4dee07e640 1 --2- 172.21.15.42:0/2542681065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4de81031a0 0x7f4de8103610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:05.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.505+0000 7f4dee07e640 1 -- 172.21.15.42:0/2542681065 >> 172.21.15.42:0/2542681065 conn(0x7f4de80feaf0 msgr2=0x7f4de8100fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:05.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.505+0000 7f4dee07e640 1 -- 172.21.15.42:0/2542681065 shutdown_connections 2024-07-26T06:41:05.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.505+0000 7f4dee07e640 1 -- 172.21.15.42:0/2542681065 wait complete. 2024-07-26T06:41:05.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.505+0000 7f4dee07e640 1 Processor -- start 2024-07-26T06:41:05.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4dee07e640 1 -- start start 2024-07-26T06:41:05.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4dee07e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4de81031a0 0x7f4de8199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:05.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4dee07e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4de8104230 0x7f4de819a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:05.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4dee07e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4de8104bd0 0x7f4de819ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:05.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4dee07e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4de810ee00 con 0x7f4de81031a0 2024-07-26T06:41:05.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4dee07e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4de810ec80 con 0x7f4de8104bd0 2024-07-26T06:41:05.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4dee07e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4de810ef80 con 0x7f4de8104230 2024-07-26T06:41:05.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4de77fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4de81031a0 0x7f4de8199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:05.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4de77fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4de81031a0 0x7f4de8199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43810/0 (socket says 172.21.15.42:43810) 2024-07-26T06:41:05.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4de77fe640 1 -- 172.21.15.42:0/3822651283 learned_addr learned my addr 172.21.15.42:0/3822651283 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:41:05.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4de6ffd640 1 --2- 172.21.15.42:0/3822651283 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4de8104230 0x7f4de819a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:05.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4de7fff640 1 --2- 172.21.15.42:0/3822651283 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4de8104bd0 0x7f4de819ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:05.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4de77fe640 1 -- 172.21.15.42:0/3822651283 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4de8104230 msgr2=0x7f4de819a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:05.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4de77fe640 1 --2- 172.21.15.42:0/3822651283 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4de8104230 0x7f4de819a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:05.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4de77fe640 1 -- 172.21.15.42:0/3822651283 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4de8104bd0 msgr2=0x7f4de819ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:05.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4de77fe640 1 --2- 172.21.15.42:0/3822651283 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4de8104bd0 0x7f4de819ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:05.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4de77fe640 1 -- 172.21.15.42:0/3822651283 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4de819b430 con 0x7f4de81031a0 2024-07-26T06:41:05.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.506+0000 7f4de77fe640 1 --2- 172.21.15.42:0/3822651283 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4de81031a0 0x7f4de8199a00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4dd800e9f0 tx=0x7f4dd800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:05.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.507+0000 7f4de4ff9640 1 -- 172.21.15.42:0/3822651283 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd800cd80 con 0x7f4de81031a0 2024-07-26T06:41:05.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.507+0000 7f4de4ff9640 1 -- 172.21.15.42:0/3822651283 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4dd80054d0 con 0x7f4de81031a0 2024-07-26T06:41:05.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.507+0000 7f4de4ff9640 1 -- 172.21.15.42:0/3822651283 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8010690 con 0x7f4de81031a0 2024-07-26T06:41:05.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.507+0000 7f4dee07e640 1 -- 172.21.15.42:0/3822651283 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4de819b720 con 0x7f4de81031a0 2024-07-26T06:41:05.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.507+0000 7f4dee07e640 1 -- 172.21.15.42:0/3822651283 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4de81a1e20 con 0x7f4de81031a0 2024-07-26T06:41:05.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.508+0000 7f4dee07e640 1 -- 172.21.15.42:0/3822651283 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4de810a380 con 0x7f4de81031a0 2024-07-26T06:41:05.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.509+0000 7f4de4ff9640 1 -- 172.21.15.42:0/3822651283 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f4dd8004e30 con 0x7f4de81031a0 2024-07-26T06:41:05.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.510+0000 7f4de4ff9640 1 --2- 172.21.15.42:0/3822651283 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f4dd40b3b40 0x7f4dd40b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:05.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.510+0000 7f4de6ffd640 1 --2- 172.21.15.42:0/3822651283 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f4dd40b3b40 0x7f4dd40b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:05.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.510+0000 7f4de4ff9640 1 -- 172.21.15.42:0/3822651283 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4dd80e2670 con 0x7f4de81031a0 2024-07-26T06:41:05.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.511+0000 7f4de6ffd640 1 --2- 172.21.15.42:0/3822651283 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f4dd40b3b40 0x7f4dd40b6050 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0031440 tx=0x7f4dd00314a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:05.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.513+0000 7f4de4ff9640 1 -- 172.21.15.42:0/3822651283 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4dd8055600 con 0x7f4de81031a0 2024-07-26T06:41:05.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.696+0000 7f4dee07e640 1 -- 172.21.15.42:0/3822651283 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7f4de802e0c0 con 0x7f4de81031a0 2024-07-26T06:41:05.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.700+0000 7f4de4ff9640 1 -- 172.21.15.42:0/3822651283 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v26)=0 v26) ==== 121+0+0 (secure 0 0 0) 0x7f4dd80ab990 con 0x7f4de81031a0 2024-07-26T06:41:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.702+0000 7f4dee07e640 1 -- 172.21.15.42:0/3822651283 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f4dd40b3b40 msgr2=0x7f4dd40b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.702+0000 7f4dee07e640 1 --2- 172.21.15.42:0/3822651283 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f4dd40b3b40 0x7f4dd40b6050 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0031440 tx=0x7f4dd00314a0 comp rx=0 tx=0).stop 2024-07-26T06:41:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.702+0000 7f4dee07e640 1 -- 172.21.15.42:0/3822651283 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4de81031a0 msgr2=0x7f4de8199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:05.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.702+0000 7f4dee07e640 1 --2- 172.21.15.42:0/3822651283 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4de81031a0 0x7f4de8199a00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4dd800e9f0 tx=0x7f4dd800edf0 comp rx=0 tx=0).stop 2024-07-26T06:41:05.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.702+0000 7f4dee07e640 1 -- 172.21.15.42:0/3822651283 shutdown_connections 2024-07-26T06:41:05.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.702+0000 7f4dee07e640 1 --2- 172.21.15.42:0/3822651283 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f4dd40b3b40 0x7f4dd40b6050 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:05.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.702+0000 7f4dee07e640 1 --2- 172.21.15.42:0/3822651283 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4de8104bd0 0x7f4de819ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:05.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.702+0000 7f4dee07e640 1 --2- 172.21.15.42:0/3822651283 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4de8104230 0x7f4de819a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:05.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.702+0000 7f4dee07e640 1 --2- 172.21.15.42:0/3822651283 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4de81031a0 0x7f4de8199a00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:05.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.702+0000 7f4dee07e640 1 -- 172.21.15.42:0/3822651283 >> 172.21.15.42:0/3822651283 conn(0x7f4de80feaf0 msgr2=0x7f4de8107550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:05.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.703+0000 7f4dee07e640 1 -- 172.21.15.42:0/3822651283 shutdown_connections 2024-07-26T06:41:05.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:05.703+0000 7f4dee07e640 1 -- 172.21.15.42:0/3822651283 wait complete. 2024-07-26T06:41:06.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:05 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3143943272' entity='client.admin' 2024-07-26T06:41:06.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:05 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3822651283' entity='client.admin' 2024-07-26T06:41:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:05 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3143943272' entity='client.admin' 2024-07-26T06:41:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:05 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3822651283' entity='client.admin' 2024-07-26T06:41:06.265 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:05 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3143943272' entity='client.admin' 2024-07-26T06:41:06.265 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:05 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3822651283' entity='client.admin' 2024-07-26T06:41:06.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.445+0000 7f883feb7640 1 -- 172.21.15.42:0/1377165778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8838104210 msgr2=0x7f8838104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:06.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.445+0000 7f883feb7640 1 --2- 172.21.15.42:0/1377165778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8838104210 0x7f8838104620 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8834009900 tx=0x7f883402a480 comp rx=0 tx=0).stop 2024-07-26T06:41:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.445+0000 7f883feb7640 1 -- 172.21.15.42:0/1377165778 shutdown_connections 2024-07-26T06:41:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.445+0000 7f883feb7640 1 --2- 172.21.15.42:0/1377165778 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8838104bb0 0x7f883810b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.445+0000 7f883feb7640 1 --2- 172.21.15.42:0/1377165778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8838104210 0x7f8838104620 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.445+0000 7f883feb7640 1 --2- 172.21.15.42:0/1377165778 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8838103180 0x7f88381035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.445+0000 7f883feb7640 1 -- 172.21.15.42:0/1377165778 >> 172.21.15.42:0/1377165778 conn(0x7f88380feaf0 msgr2=0x7f8838100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.445+0000 7f883feb7640 1 -- 172.21.15.42:0/1377165778 shutdown_connections 2024-07-26T06:41:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.445+0000 7f883feb7640 1 -- 172.21.15.42:0/1377165778 wait complete. 2024-07-26T06:41:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.445+0000 7f883feb7640 1 Processor -- start 2024-07-26T06:41:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.445+0000 7f883feb7640 1 -- start start 2024-07-26T06:41:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883feb7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8838103180 0x7f88381999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883feb7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8838104210 0x7f883819a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883feb7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8838104bb0 0x7f883819abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883feb7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f883810ede0 con 0x7f8838103180 2024-07-26T06:41:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883feb7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f883810ec60 con 0x7f8838104210 2024-07-26T06:41:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883feb7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f883810ef60 con 0x7f8838104bb0 2024-07-26T06:41:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883dc53640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8838103180 0x7f88381999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883dc53640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8838103180 0x7f88381999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43832/0 (socket says 172.21.15.42:43832) 2024-07-26T06:41:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883dc53640 1 -- 172.21.15.42:0/2340768043 learned_addr learned my addr 172.21.15.42:0/2340768043 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:41:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883e454640 1 --2- 172.21.15.42:0/2340768043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8838104bb0 0x7f883819abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883d452640 1 --2- 172.21.15.42:0/2340768043 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8838104210 0x7f883819a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883dc53640 1 -- 172.21.15.42:0/2340768043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8838104bb0 msgr2=0x7f883819abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883dc53640 1 --2- 172.21.15.42:0/2340768043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8838104bb0 0x7f883819abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883dc53640 1 -- 172.21.15.42:0/2340768043 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8838104210 msgr2=0x7f883819a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883dc53640 1 --2- 172.21.15.42:0/2340768043 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8838104210 0x7f883819a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883dc53640 1 -- 172.21.15.42:0/2340768043 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f883819b3e0 con 0x7f8838103180 2024-07-26T06:41:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883d452640 1 --2- 172.21.15.42:0/2340768043 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8838104210 0x7f883819a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:41:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883dc53640 1 --2- 172.21.15.42:0/2340768043 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8838103180 0x7f88381999b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f882c00d950 tx=0x7f882c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f882affd640 1 -- 172.21.15.42:0/2340768043 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882c017070 con 0x7f8838103180 2024-07-26T06:41:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f882affd640 1 -- 172.21.15.42:0/2340768043 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f882c009c30 con 0x7f8838103180 2024-07-26T06:41:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f883feb7640 1 -- 172.21.15.42:0/2340768043 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f883819b6d0 con 0x7f8838103180 2024-07-26T06:41:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.446+0000 7f882affd640 1 -- 172.21.15.42:0/2340768043 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882c00f620 con 0x7f8838103180 2024-07-26T06:41:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.447+0000 7f883feb7640 1 -- 172.21.15.42:0/2340768043 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88381a1dd0 con 0x7f8838103180 2024-07-26T06:41:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.447+0000 7f883feb7640 1 -- 172.21.15.42:0/2340768043 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f883810a360 con 0x7f8838103180 2024-07-26T06:41:06.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.450+0000 7f882affd640 1 -- 172.21.15.42:0/2340768043 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f882c010460 con 0x7f8838103180 2024-07-26T06:41:06.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.451+0000 7f882affd640 1 --2- 172.21.15.42:0/2340768043 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f88240b3bc0 0x7f88240b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:06.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.451+0000 7f883d452640 1 --2- 172.21.15.42:0/2340768043 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f88240b3bc0 0x7f88240b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:06.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.451+0000 7f882affd640 1 -- 172.21.15.42:0/2340768043 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f882c0e1b40 con 0x7f8838103180 2024-07-26T06:41:06.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.452+0000 7f883d452640 1 --2- 172.21.15.42:0/2340768043 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f88240b3bc0 0x7f88240b60d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8834005f70 tx=0x7f8834000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:06.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.455+0000 7f882affd640 1 -- 172.21.15.42:0/2340768043 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f882c0aaee0 con 0x7f8838103180 2024-07-26T06:41:06.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.640+0000 7f883feb7640 1 -- 172.21.15.42:0/2340768043 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) -- 0x7f883802e0c0 con 0x7f8838103180 2024-07-26T06:41:06.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.644+0000 7f882affd640 1 -- 172.21.15.42:0/2340768043 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v27)=0 v27) ==== 116+0+0 (secure 0 0 0) 0x7f882c009de0 con 0x7f8838103180 2024-07-26T06:41:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.647+0000 7f883feb7640 1 -- 172.21.15.42:0/2340768043 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f88240b3bc0 msgr2=0x7f88240b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.647+0000 7f883feb7640 1 --2- 172.21.15.42:0/2340768043 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f88240b3bc0 0x7f88240b60d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8834005f70 tx=0x7f8834000bc0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.647+0000 7f883feb7640 1 -- 172.21.15.42:0/2340768043 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8838103180 msgr2=0x7f88381999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.647+0000 7f883feb7640 1 --2- 172.21.15.42:0/2340768043 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8838103180 0x7f88381999b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f882c00d950 tx=0x7f882c00de10 comp rx=0 tx=0).stop 2024-07-26T06:41:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.647+0000 7f883feb7640 1 -- 172.21.15.42:0/2340768043 shutdown_connections 2024-07-26T06:41:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.647+0000 7f883feb7640 1 --2- 172.21.15.42:0/2340768043 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f88240b3bc0 0x7f88240b60d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.647+0000 7f883feb7640 1 --2- 172.21.15.42:0/2340768043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8838104bb0 0x7f883819abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.647+0000 7f883feb7640 1 --2- 172.21.15.42:0/2340768043 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8838104210 0x7f883819a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.647+0000 7f883feb7640 1 --2- 172.21.15.42:0/2340768043 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8838103180 0x7f88381999b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.647+0000 7f883feb7640 1 -- 172.21.15.42:0/2340768043 >> 172.21.15.42:0/2340768043 conn(0x7f88380feaf0 msgr2=0x7f8838107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.647+0000 7f883feb7640 1 -- 172.21.15.42:0/2340768043 shutdown_connections 2024-07-26T06:41:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.647+0000 7f883feb7640 1 -- 172.21.15.42:0/2340768043 wait complete. 2024-07-26T06:41:06.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.656+0000 7f9fe5d7e640 1 -- 172.21.15.42:0/3016930037 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9fe0103100 msgr2=0x7f9fe0103510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:06.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.656+0000 7f9fe5d7e640 1 --2- 172.21.15.42:0/3016930037 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9fe0103100 0x7f9fe0103510 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009900 tx=0x7f9fc802a480 comp rx=0 tx=0).stop 2024-07-26T06:41:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.656+0000 7f9fe5d7e640 1 -- 172.21.15.42:0/3016930037 shutdown_connections 2024-07-26T06:41:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.656+0000 7f9fe5d7e640 1 --2- 172.21.15.42:0/3016930037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9fe0103aa0 0x7f9fe010a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.656+0000 7f9fe5d7e640 1 --2- 172.21.15.42:0/3016930037 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9fe0103100 0x7f9fe0103510 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.656+0000 7f9fe5d7e640 1 --2- 172.21.15.42:0/3016930037 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fe0102070 0x7f9fe01024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.656+0000 7f9fe5d7e640 1 -- 172.21.15.42:0/3016930037 >> 172.21.15.42:0/3016930037 conn(0x7f9fe00fd9e0 msgr2=0x7f9fe00ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.656+0000 7f9fe5d7e640 1 -- 172.21.15.42:0/3016930037 shutdown_connections 2024-07-26T06:41:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.656+0000 7f9fe5d7e640 1 -- 172.21.15.42:0/3016930037 wait complete. 2024-07-26T06:41:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fe5d7e640 1 Processor -- start 2024-07-26T06:41:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fe5d7e640 1 -- start start 2024-07-26T06:41:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fe5d7e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fe0102070 0x7f9fe006b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fe5d7e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9fe0103100 0x7f9fe019e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fe5d7e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9fe0103aa0 0x7f9fe01a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fe5d7e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe010dcd0 con 0x7f9fe0103aa0 2024-07-26T06:41:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fe5d7e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe010db50 con 0x7f9fe0103100 2024-07-26T06:41:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fe5d7e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe010de50 con 0x7f9fe0102070 2024-07-26T06:41:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fdf7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fe0102070 0x7f9fe006b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9fe0103aa0 0x7f9fe01a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fdf7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fe0102070 0x7f9fe006b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:55830/0 (socket says 172.21.15.42:55830) 2024-07-26T06:41:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fdffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9fe0103aa0 0x7f9fe01a0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43856/0 (socket says 172.21.15.42:43856) 2024-07-26T06:41:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fdf7fe640 1 -- 172.21.15.42:0/195861889 learned_addr learned my addr 172.21.15.42:0/195861889 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:41:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fd7fff640 1 --2- 172.21.15.42:0/195861889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9fe0103100 0x7f9fe019e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fdf7fe640 1 -- 172.21.15.42:0/195861889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9fe0103100 msgr2=0x7f9fe019e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fdf7fe640 1 --2- 172.21.15.42:0/195861889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9fe0103100 0x7f9fe019e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fdf7fe640 1 -- 172.21.15.42:0/195861889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9fe0103aa0 msgr2=0x7f9fe01a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fdf7fe640 1 --2- 172.21.15.42:0/195861889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9fe0103aa0 0x7f9fe01a0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fdf7fe640 1 -- 172.21.15.42:0/195861889 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe01a14e0 con 0x7f9fe0102070 2024-07-26T06:41:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.657+0000 7f9fdffff640 1 --2- 172.21.15.42:0/195861889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9fe0103aa0 0x7f9fe01a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:41:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.658+0000 7f9fdf7fe640 1 --2- 172.21.15.42:0/195861889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fe0102070 0x7f9fe006b0d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000e9f0 tx=0x7f9fd000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.658+0000 7f9fdd7fa640 1 -- 172.21.15.42:0/195861889 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd000cd80 con 0x7f9fe0102070 2024-07-26T06:41:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.658+0000 7f9fe5d7e640 1 -- 172.21.15.42:0/195861889 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe01a17d0 con 0x7f9fe0102070 2024-07-26T06:41:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.658+0000 7f9fdd7fa640 1 -- 172.21.15.42:0/195861889 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fd00054d0 con 0x7f9fe0102070 2024-07-26T06:41:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.658+0000 7f9fdd7fa640 1 -- 172.21.15.42:0/195861889 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0010690 con 0x7f9fe0102070 2024-07-26T06:41:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.658+0000 7f9fe5d7e640 1 -- 172.21.15.42:0/195861889 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe01a1d10 con 0x7f9fe0102070 2024-07-26T06:41:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.659+0000 7f9fe5d7e640 1 -- 172.21.15.42:0/195861889 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fe01024e0 con 0x7f9fe0102070 2024-07-26T06:41:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.661+0000 7f9fdd7fa640 1 -- 172.21.15.42:0/195861889 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f9fd0004e70 con 0x7f9fe0102070 2024-07-26T06:41:06.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.661+0000 7f9fdd7fa640 1 --2- 172.21.15.42:0/195861889 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9fcc0b3b40 0x7f9fcc0b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:06.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.662+0000 7f9fd7fff640 1 --2- 172.21.15.42:0/195861889 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9fcc0b3b40 0x7f9fcc0b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:06.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.662+0000 7f9fdd7fa640 1 -- 172.21.15.42:0/195861889 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9fd00e2480 con 0x7f9fe0102070 2024-07-26T06:41:06.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.662+0000 7f9fd7fff640 1 --2- 172.21.15.42:0/195861889 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9fcc0b3b40 0x7f9fcc0b6050 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8031440 tx=0x7f9fc80314a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:06.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.665+0000 7f9fdd7fa640 1 -- 172.21.15.42:0/195861889 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9fd00ab780 con 0x7f9fe0102070 2024-07-26T06:41:06.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.850+0000 7f9fe5d7e640 1 -- 172.21.15.42:0/195861889 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f9fe0109250 con 0x7f9fe0102070 2024-07-26T06:41:06.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.856+0000 7f9fdd7fa640 1 -- 172.21.15.42:0/195861889 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v28)=0 v28) ==== 120+0+0 (secure 0 0 0) 0x7f9fd00ab080 con 0x7f9fe0102070 2024-07-26T06:41:06.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.858+0000 7f9fe5d7e640 1 -- 172.21.15.42:0/195861889 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9fcc0b3b40 msgr2=0x7f9fcc0b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:06.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.858+0000 7f9fe5d7e640 1 --2- 172.21.15.42:0/195861889 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9fcc0b3b40 0x7f9fcc0b6050 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8031440 tx=0x7f9fc80314a0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.858+0000 7f9fe5d7e640 1 -- 172.21.15.42:0/195861889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fe0102070 msgr2=0x7f9fe006b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:06.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.858+0000 7f9fe5d7e640 1 --2- 172.21.15.42:0/195861889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fe0102070 0x7f9fe006b0d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000e9f0 tx=0x7f9fd000edf0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.858+0000 7f9fe5d7e640 1 -- 172.21.15.42:0/195861889 shutdown_connections 2024-07-26T06:41:06.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.858+0000 7f9fe5d7e640 1 --2- 172.21.15.42:0/195861889 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9fcc0b3b40 0x7f9fcc0b6050 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.858+0000 7f9fe5d7e640 1 --2- 172.21.15.42:0/195861889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9fe0103aa0 0x7f9fe01a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.858+0000 7f9fe5d7e640 1 --2- 172.21.15.42:0/195861889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9fe0103100 0x7f9fe019e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.858+0000 7f9fe5d7e640 1 --2- 172.21.15.42:0/195861889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fe0102070 0x7f9fe006b0d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:06.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.858+0000 7f9fe5d7e640 1 -- 172.21.15.42:0/195861889 >> 172.21.15.42:0/195861889 conn(0x7f9fe00fd9e0 msgr2=0x7f9fe0106420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:06.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.858+0000 7f9fe5d7e640 1 -- 172.21.15.42:0/195861889 shutdown_connections 2024-07-26T06:41:06.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:06.858+0000 7f9fe5d7e640 1 -- 172.21.15.42:0/195861889 wait complete. 2024-07-26T06:41:07.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:06 smithi042 ceph-mon[32775]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:07.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:06 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2340768043' entity='client.admin' 2024-07-26T06:41:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:06 smithi042 ceph-mon[32775]: from='client.? ' entity='client.admin' 2024-07-26T06:41:07.211 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'], check_status=True, hostname='smithi042') 2024-07-26T06:41:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:06 smithi164 ceph-mon[36970]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:06 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2340768043' entity='client.admin' 2024-07-26T06:41:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:06 smithi164 ceph-mon[36970]: from='client.? ' entity='client.admin' 2024-07-26T06:41:07.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:06 smithi163 ceph-mon[36873]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:07.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:06 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2340768043' entity='client.admin' 2024-07-26T06:41:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:06 smithi163 ceph-mon[36873]: from='client.? ' entity='client.admin' 2024-07-26T06:41:08.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.994+0000 7fdc0af72640 1 -- 172.21.15.42:0/3652531189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc04108ed0 msgr2=0x7fdc0410b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:08.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.994+0000 7fdc0af72640 1 --2- 172.21.15.42:0/3652531189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc04108ed0 0x7fdc0410b3e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009ca0 tx=0x7fdbfc02a8c0 comp rx=0 tx=0).stop 2024-07-26T06:41:08.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.994+0000 7fdc0af72640 1 -- 172.21.15.42:0/3652531189 shutdown_connections 2024-07-26T06:41:08.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.995+0000 7fdc0af72640 1 --2- 172.21.15.42:0/3652531189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc04108ed0 0x7fdc0410b3e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:08.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.995+0000 7fdc0af72640 1 --2- 172.21.15.42:0/3652531189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc04106330 0x7fdc04108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:08.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.995+0000 7fdc0af72640 1 --2- 172.21.15.42:0/3652531189 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc0406c090 0x7fdc04105df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:08.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.995+0000 7fdc0af72640 1 -- 172.21.15.42:0/3652531189 >> 172.21.15.42:0/3652531189 conn(0x7fdc040fda10 msgr2=0x7fdc040ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:08.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.995+0000 7fdc0af72640 1 -- 172.21.15.42:0/3652531189 shutdown_connections 2024-07-26T06:41:08.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.995+0000 7fdc0af72640 1 -- 172.21.15.42:0/3652531189 wait complete. 2024-07-26T06:41:08.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.995+0000 7fdc0af72640 1 Processor -- start 2024-07-26T06:41:08.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.996+0000 7fdc0af72640 1 -- start start 2024-07-26T06:41:08.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.996+0000 7fdc0af72640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc0406c090 0x7fdc041999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:08.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.996+0000 7fdc0af72640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc04106330 0x7fdc0419a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:08.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.996+0000 7fdc0af72640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc04108ed0 0x7fdc0419abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:08.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.996+0000 7fdc0af72640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0410edf0 con 0x7fdc04108ed0 2024-07-26T06:41:08.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.996+0000 7fdc0af72640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0410ec70 con 0x7fdc0406c090 2024-07-26T06:41:08.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.996+0000 7fdc0af72640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0410ef70 con 0x7fdc04106330 2024-07-26T06:41:08.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.996+0000 7fdc0950f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc04108ed0 0x7fdc0419abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:08.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.996+0000 7fdc0950f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc04108ed0 0x7fdc0419abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51054/0 (socket says 172.21.15.42:51054) 2024-07-26T06:41:08.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.996+0000 7fdc0950f640 1 -- 172.21.15.42:0/3918060172 learned_addr learned my addr 172.21.15.42:0/3918060172 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:41:08.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.996+0000 7fdc08d0e640 1 --2- 172.21.15.42:0/3918060172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc0406c090 0x7fdc041999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:08.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.996+0000 7fdbfbfff640 1 --2- 172.21.15.42:0/3918060172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc04106330 0x7fdc0419a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:08.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.996+0000 7fdc0950f640 1 -- 172.21.15.42:0/3918060172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc04106330 msgr2=0x7fdc0419a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:08.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.996+0000 7fdc0950f640 1 --2- 172.21.15.42:0/3918060172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc04106330 0x7fdc0419a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:08.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.996+0000 7fdc0950f640 1 -- 172.21.15.42:0/3918060172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc0406c090 msgr2=0x7fdc041999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:08.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.996+0000 7fdc0950f640 1 --2- 172.21.15.42:0/3918060172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc0406c090 0x7fdc041999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:08.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.996+0000 7fdc0950f640 1 -- 172.21.15.42:0/3918060172 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc0419b3f0 con 0x7fdc04108ed0 2024-07-26T06:41:08.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.997+0000 7fdbfbfff640 1 --2- 172.21.15.42:0/3918060172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc04106330 0x7fdc0419a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:41:09.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.997+0000 7fdc0950f640 1 --2- 172.21.15.42:0/3918060172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc04108ed0 0x7fdc0419abd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0097b0 tx=0x7fdbfc003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:09.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.997+0000 7fdbf9ffb640 1 -- 172.21.15.42:0/3918060172 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc004240 con 0x7fdc04108ed0 2024-07-26T06:41:09.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.997+0000 7fdbf9ffb640 1 -- 172.21.15.42:0/3918060172 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbfc02f380 con 0x7fdc04108ed0 2024-07-26T06:41:09.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.997+0000 7fdbf9ffb640 1 -- 172.21.15.42:0/3918060172 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc02fc30 con 0x7fdc04108ed0 2024-07-26T06:41:09.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.997+0000 7fdc0af72640 1 -- 172.21.15.42:0/3918060172 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc0419b680 con 0x7fdc04108ed0 2024-07-26T06:41:09.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.997+0000 7fdc0af72640 1 -- 172.21.15.42:0/3918060172 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc041a1de0 con 0x7fdc04108ed0 2024-07-26T06:41:09.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:08.999+0000 7fdc0af72640 1 -- 172.21.15.42:0/3918060172 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc04106780 con 0x7fdc04108ed0 2024-07-26T06:41:09.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.000+0000 7fdbf9ffb640 1 -- 172.21.15.42:0/3918060172 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7fdbfc0043e0 con 0x7fdc04108ed0 2024-07-26T06:41:09.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.001+0000 7fdbf9ffb640 1 --2- 172.21.15.42:0/3918060172 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdbec0b3b40 0x7fdbec0b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:09.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.001+0000 7fdc08d0e640 1 --2- 172.21.15.42:0/3918060172 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdbec0b3b40 0x7fdbec0b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:09.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.001+0000 7fdbf9ffb640 1 -- 172.21.15.42:0/3918060172 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fdbfc1037b0 con 0x7fdc04108ed0 2024-07-26T06:41:09.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.002+0000 7fdc08d0e640 1 --2- 172.21.15.42:0/3918060172 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdbec0b3b40 0x7fdbec0b6050 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdc00005fd0 tx=0x7fdc0000b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:09.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.004+0000 7fdbf9ffb640 1 -- 172.21.15.42:0/3918060172 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdbfc0ccad0 con 0x7fdc04108ed0 2024-07-26T06:41:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:08 smithi042 ceph-mon[32775]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:09.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.194+0000 7fdc0af72640 1 -- 172.21.15.42:0/3918060172 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7fdc040ff530 con 0x7fdc04108ed0 2024-07-26T06:41:09.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.198+0000 7fdbf9ffb640 1 -- 172.21.15.42:0/3918060172 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v29)=0 v29) ==== 121+0+0 (secure 0 0 0) 0x7fdc040ff530 con 0x7fdc04108ed0 2024-07-26T06:41:09.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.200+0000 7fdc0af72640 1 -- 172.21.15.42:0/3918060172 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdbec0b3b40 msgr2=0x7fdbec0b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:09.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.200+0000 7fdc0af72640 1 --2- 172.21.15.42:0/3918060172 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdbec0b3b40 0x7fdbec0b6050 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdc00005fd0 tx=0x7fdc0000b390 comp rx=0 tx=0).stop 2024-07-26T06:41:09.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.200+0000 7fdc0af72640 1 -- 172.21.15.42:0/3918060172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc04108ed0 msgr2=0x7fdc0419abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:09.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.200+0000 7fdc0af72640 1 --2- 172.21.15.42:0/3918060172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc04108ed0 0x7fdc0419abd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0097b0 tx=0x7fdbfc003c20 comp rx=0 tx=0).stop 2024-07-26T06:41:09.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.200+0000 7fdc0af72640 1 -- 172.21.15.42:0/3918060172 shutdown_connections 2024-07-26T06:41:09.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.200+0000 7fdc0af72640 1 --2- 172.21.15.42:0/3918060172 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdbec0b3b40 0x7fdbec0b6050 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:09.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.200+0000 7fdc0af72640 1 --2- 172.21.15.42:0/3918060172 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc04108ed0 0x7fdc0419abd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:09.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.200+0000 7fdc0af72640 1 --2- 172.21.15.42:0/3918060172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc04106330 0x7fdc0419a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:09.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.200+0000 7fdc0af72640 1 --2- 172.21.15.42:0/3918060172 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc0406c090 0x7fdc041999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:09.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.200+0000 7fdc0af72640 1 -- 172.21.15.42:0/3918060172 >> 172.21.15.42:0/3918060172 conn(0x7fdc040fda10 msgr2=0x7fdc04104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:09.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.200+0000 7fdc0af72640 1 -- 172.21.15.42:0/3918060172 shutdown_connections 2024-07-26T06:41:09.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:09.200+0000 7fdc0af72640 1 -- 172.21.15.42:0/3918060172 wait complete. 2024-07-26T06:41:09.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:08 smithi164 ceph-mon[36970]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:09.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:08 smithi163 ceph-mon[36873]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:10.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.338+0000 7f65290e1640 1 -- 172.21.15.42:0/60188295 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6524102070 msgr2=0x7f65241024e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:10.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.338+0000 7f65290e1640 1 --2- 172.21.15.42:0/60188295 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6524102070 0x7f65241024e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6514009900 tx=0x7f651402a480 comp rx=0 tx=0).stop 2024-07-26T06:41:10.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.338+0000 7f65290e1640 1 -- 172.21.15.42:0/60188295 shutdown_connections 2024-07-26T06:41:10.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.338+0000 7f65290e1640 1 --2- 172.21.15.42:0/60188295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6524103aa0 0x7f652410a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:10.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.338+0000 7f65290e1640 1 --2- 172.21.15.42:0/60188295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6524103100 0x7f6524103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:10.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.338+0000 7f65290e1640 1 --2- 172.21.15.42:0/60188295 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6524102070 0x7f65241024e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:10.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.338+0000 7f65290e1640 1 -- 172.21.15.42:0/60188295 >> 172.21.15.42:0/60188295 conn(0x7f65240fd9e0 msgr2=0x7f65240ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:10.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.338+0000 7f65290e1640 1 -- 172.21.15.42:0/60188295 shutdown_connections 2024-07-26T06:41:10.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.338+0000 7f65290e1640 1 -- 172.21.15.42:0/60188295 wait complete. 2024-07-26T06:41:10.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.339+0000 7f65290e1640 1 Processor -- start 2024-07-26T06:41:10.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.339+0000 7f65290e1640 1 -- start start 2024-07-26T06:41:10.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.339+0000 7f65290e1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6524102070 0x7f6524199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:10.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.339+0000 7f65290e1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6524103100 0x7f652419a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:10.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.339+0000 7f65290e1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6524103aa0 0x7f652419aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:10.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.339+0000 7f65290e1640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f652410dcd0 con 0x7f6524102070 2024-07-26T06:41:10.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.339+0000 7f65290e1640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f652410db50 con 0x7f6524103100 2024-07-26T06:41:10.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.339+0000 7f65290e1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f652410de50 con 0x7f6524103aa0 2024-07-26T06:41:10.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.339+0000 7f6522d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6524102070 0x7f6524199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:10.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.339+0000 7f6522d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6524102070 0x7f6524199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51070/0 (socket says 172.21.15.42:51070) 2024-07-26T06:41:10.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.339+0000 7f6522d9d640 1 -- 172.21.15.42:0/457361454 learned_addr learned my addr 172.21.15.42:0/457361454 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:41:10.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.340+0000 7f652359e640 1 --2- 172.21.15.42:0/457361454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6524103aa0 0x7f652419aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:10.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.340+0000 7f652259c640 1 --2- 172.21.15.42:0/457361454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6524103100 0x7f652419a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:10.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.340+0000 7f6522d9d640 1 -- 172.21.15.42:0/457361454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6524103aa0 msgr2=0x7f652419aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:10.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.340+0000 7f6522d9d640 1 --2- 172.21.15.42:0/457361454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6524103aa0 0x7f652419aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:10.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.340+0000 7f6522d9d640 1 -- 172.21.15.42:0/457361454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6524103100 msgr2=0x7f652419a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:10.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.340+0000 7f6522d9d640 1 --2- 172.21.15.42:0/457361454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6524103100 0x7f652419a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:10.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.340+0000 7f6522d9d640 1 -- 172.21.15.42:0/457361454 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f652419b330 con 0x7f6524102070 2024-07-26T06:41:10.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.340+0000 7f652259c640 1 --2- 172.21.15.42:0/457361454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6524103100 0x7f652419a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:41:10.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.340+0000 7f652359e640 1 --2- 172.21.15.42:0/457361454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6524103aa0 0x7f652419aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:41:10.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.340+0000 7f6522d9d640 1 --2- 172.21.15.42:0/457361454 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6524102070 0x7f6524199990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6514006010 tx=0x7f65140043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:10.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.340+0000 7f650bfff640 1 -- 172.21.15.42:0/457361454 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651403f030 con 0x7f6524102070 2024-07-26T06:41:10.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.340+0000 7f65290e1640 1 -- 172.21.15.42:0/457361454 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f652419b5c0 con 0x7f6524102070 2024-07-26T06:41:10.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.340+0000 7f650bfff640 1 -- 172.21.15.42:0/457361454 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6514003cb0 con 0x7f6524102070 2024-07-26T06:41:10.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.340+0000 7f650bfff640 1 -- 172.21.15.42:0/457361454 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514035070 con 0x7f6524102070 2024-07-26T06:41:10.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.340+0000 7f65290e1640 1 -- 172.21.15.42:0/457361454 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65241a1be0 con 0x7f6524102070 2024-07-26T06:41:10.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.342+0000 7f65290e1640 1 -- 172.21.15.42:0/457361454 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65241a1e70 con 0x7f6524102070 2024-07-26T06:41:10.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.343+0000 7f650bfff640 1 -- 172.21.15.42:0/457361454 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f6514042030 con 0x7f6524102070 2024-07-26T06:41:10.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.344+0000 7f650bfff640 1 --2- 172.21.15.42:0/457361454 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f650c0b3b40 0x7f650c0b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:10.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.344+0000 7f652259c640 1 --2- 172.21.15.42:0/457361454 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f650c0b3b40 0x7f650c0b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:10.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.344+0000 7f650bfff640 1 -- 172.21.15.42:0/457361454 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6514102e20 con 0x7f6524102070 2024-07-26T06:41:10.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.344+0000 7f652259c640 1 --2- 172.21.15.42:0/457361454 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f650c0b3b40 0x7f650c0b6050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f651800abe0 tx=0x7f651800a5c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:10.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.347+0000 7f650bfff640 1 -- 172.21.15.42:0/457361454 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f65140cc0a0 con 0x7f6524102070 2024-07-26T06:41:10.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:10 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3918060172' entity='client.admin' 2024-07-26T06:41:10.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:10 smithi042 ceph-mon[32775]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:10.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:10 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3918060172' entity='client.admin' 2024-07-26T06:41:10.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:10 smithi164 ceph-mon[36970]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:10.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:10 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3918060172' entity='client.admin' 2024-07-26T06:41:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:10 smithi163 ceph-mon[36873]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:10.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.534+0000 7f65290e1640 1 -- 172.21.15.42:0/457361454 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) -- 0x7f65241a2040 con 0x7f6524102070 2024-07-26T06:41:10.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.538+0000 7f650bfff640 1 -- 172.21.15.42:0/457361454 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v30)=0 v30) ==== 118+0+0 (secure 0 0 0) 0x7f65140cb9a0 con 0x7f6524102070 2024-07-26T06:41:10.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.541+0000 7f65290e1640 1 -- 172.21.15.42:0/457361454 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f650c0b3b40 msgr2=0x7f650c0b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:10.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.541+0000 7f65290e1640 1 --2- 172.21.15.42:0/457361454 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f650c0b3b40 0x7f650c0b6050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f651800abe0 tx=0x7f651800a5c0 comp rx=0 tx=0).stop 2024-07-26T06:41:10.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.541+0000 7f65290e1640 1 -- 172.21.15.42:0/457361454 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6524102070 msgr2=0x7f6524199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:10.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.541+0000 7f65290e1640 1 --2- 172.21.15.42:0/457361454 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6524102070 0x7f6524199990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6514006010 tx=0x7f65140043b0 comp rx=0 tx=0).stop 2024-07-26T06:41:10.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.541+0000 7f65290e1640 1 -- 172.21.15.42:0/457361454 shutdown_connections 2024-07-26T06:41:10.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.541+0000 7f65290e1640 1 --2- 172.21.15.42:0/457361454 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f650c0b3b40 0x7f650c0b6050 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:10.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.541+0000 7f65290e1640 1 --2- 172.21.15.42:0/457361454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6524103aa0 0x7f652419aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:10.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.541+0000 7f65290e1640 1 --2- 172.21.15.42:0/457361454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6524103100 0x7f652419a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:10.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.541+0000 7f65290e1640 1 --2- 172.21.15.42:0/457361454 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6524102070 0x7f6524199990 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:10.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.541+0000 7f65290e1640 1 -- 172.21.15.42:0/457361454 >> 172.21.15.42:0/457361454 conn(0x7f65240fd9e0 msgr2=0x7f65240ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:10.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.541+0000 7f65290e1640 1 -- 172.21.15.42:0/457361454 shutdown_connections 2024-07-26T06:41:10.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:10.541+0000 7f65290e1640 1 -- 172.21.15.42:0/457361454 wait complete. 2024-07-26T06:41:11.270 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi042') 2024-07-26T06:41:11.271 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi042') 2024-07-26T06:41:11.794 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:11 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/457361454' entity='client.admin' 2024-07-26T06:41:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:11 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/457361454' entity='client.admin' 2024-07-26T06:41:12.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:11 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/457361454' entity='client.admin' 2024-07-26T06:41:12.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.105+0000 7f3652289640 1 -- 172.21.15.42:0/1949491536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f364c108ea0 msgr2=0x7f364c10b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:12.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.105+0000 7f3652289640 1 --2- 172.21.15.42:0/1949491536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f364c108ea0 0x7f364c10b3b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3644009ca0 tx=0x7f364402a8c0 comp rx=0 tx=0).stop 2024-07-26T06:41:12.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.105+0000 7f3652289640 1 -- 172.21.15.42:0/1949491536 shutdown_connections 2024-07-26T06:41:12.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.105+0000 7f3652289640 1 --2- 172.21.15.42:0/1949491536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f364c108ea0 0x7f364c10b3b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:12.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.105+0000 7f3652289640 1 --2- 172.21.15.42:0/1949491536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f364c106300 0x7f364c108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:12.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.105+0000 7f3652289640 1 --2- 172.21.15.42:0/1949491536 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f364c06b3b0 0x7f364c105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:12.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.105+0000 7f3652289640 1 -- 172.21.15.42:0/1949491536 >> 172.21.15.42:0/1949491536 conn(0x7f364c0fd9e0 msgr2=0x7f364c0ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:12.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.105+0000 7f3652289640 1 -- 172.21.15.42:0/1949491536 shutdown_connections 2024-07-26T06:41:12.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.105+0000 7f3652289640 1 -- 172.21.15.42:0/1949491536 wait complete. 2024-07-26T06:41:12.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.106+0000 7f3652289640 1 Processor -- start 2024-07-26T06:41:12.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.106+0000 7f3652289640 1 -- start start 2024-07-26T06:41:12.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.106+0000 7f3652289640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f364c06b3b0 0x7f364c1998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:12.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.106+0000 7f3652289640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f364c106300 0x7f364c199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:12.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.106+0000 7f3652289640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f364c108ea0 0x7f364c19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:12.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.106+0000 7f3652289640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f364c10ee10 con 0x7f364c06b3b0 2024-07-26T06:41:12.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.106+0000 7f3652289640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f364c10ec90 con 0x7f364c106300 2024-07-26T06:41:12.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.106+0000 7f3652289640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f364c10ef90 con 0x7f364c108ea0 2024-07-26T06:41:12.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.106+0000 7f364bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f364c06b3b0 0x7f364c1998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:12.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.106+0000 7f364bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f364c06b3b0 0x7f364c1998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51084/0 (socket says 172.21.15.42:51084) 2024-07-26T06:41:12.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.106+0000 7f364bfff640 1 -- 172.21.15.42:0/4051298693 learned_addr learned my addr 172.21.15.42:0/4051298693 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:41:12.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.106+0000 7f3650826640 1 --2- 172.21.15.42:0/4051298693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f364c108ea0 0x7f364c19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:12.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.107+0000 7f3643fff640 1 --2- 172.21.15.42:0/4051298693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f364c106300 0x7f364c199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:12.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.107+0000 7f364bfff640 1 -- 172.21.15.42:0/4051298693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f364c108ea0 msgr2=0x7f364c19aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:12.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.107+0000 7f364bfff640 1 --2- 172.21.15.42:0/4051298693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f364c108ea0 0x7f364c19aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:12.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.107+0000 7f364bfff640 1 -- 172.21.15.42:0/4051298693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f364c106300 msgr2=0x7f364c199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:12.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.107+0000 7f364bfff640 1 --2- 172.21.15.42:0/4051298693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f364c106300 0x7f364c199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:12.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.107+0000 7f364bfff640 1 -- 172.21.15.42:0/4051298693 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f364c19b260 con 0x7f364c06b3b0 2024-07-26T06:41:12.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.107+0000 7f3650826640 1 --2- 172.21.15.42:0/4051298693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f364c108ea0 0x7f364c19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:41:12.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.107+0000 7f364bfff640 1 --2- 172.21.15.42:0/4051298693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f364c06b3b0 0x7f364c1998c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f363c00c920 tx=0x7f363c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:12.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.107+0000 7f3649ffb640 1 -- 172.21.15.42:0/4051298693 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c007c70 con 0x7f364c06b3b0 2024-07-26T06:41:12.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.107+0000 7f3649ffb640 1 -- 172.21.15.42:0/4051298693 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f363c007e10 con 0x7f364c06b3b0 2024-07-26T06:41:12.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.107+0000 7f3652289640 1 -- 172.21.15.42:0/4051298693 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f364c19b550 con 0x7f364c06b3b0 2024-07-26T06:41:12.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.107+0000 7f3649ffb640 1 -- 172.21.15.42:0/4051298693 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c018820 con 0x7f364c06b3b0 2024-07-26T06:41:12.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.107+0000 7f3652289640 1 -- 172.21.15.42:0/4051298693 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f364c1a1ce0 con 0x7f364c06b3b0 2024-07-26T06:41:12.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.108+0000 7f3652289640 1 -- 172.21.15.42:0/4051298693 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f364c1a1eb0 con 0x7f364c06b3b0 2024-07-26T06:41:12.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.110+0000 7f3649ffb640 1 -- 172.21.15.42:0/4051298693 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f363c00f450 con 0x7f364c06b3b0 2024-07-26T06:41:12.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.110+0000 7f3649ffb640 1 --2- 172.21.15.42:0/4051298693 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36380b3b40 0x7f36380b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:12.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.111+0000 7f3643fff640 1 --2- 172.21.15.42:0/4051298693 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36380b3b40 0x7f36380b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:12.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.111+0000 7f3649ffb640 1 -- 172.21.15.42:0/4051298693 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f363c0e2360 con 0x7f364c06b3b0 2024-07-26T06:41:12.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.111+0000 7f3643fff640 1 --2- 172.21.15.42:0/4051298693 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36380b3b40 0x7f36380b6050 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3634005e10 tx=0x7f3634005da0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:12.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.114+0000 7f3649ffb640 1 -- 172.21.15.42:0/4051298693 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f363c014030 con 0x7f364c06b3b0 2024-07-26T06:41:12.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.296+0000 7f3652289640 1 -- 172.21.15.42:0/4051298693 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7f364c1a2080 con 0x7f364c06b3b0 2024-07-26T06:41:12.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.300+0000 7f3649ffb640 1 -- 172.21.15.42:0/4051298693 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v31)=0 v31) ==== 122+0+0 (secure 0 0 0) 0x7f363c0ab710 con 0x7f364c06b3b0 2024-07-26T06:41:12.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.303+0000 7f3652289640 1 -- 172.21.15.42:0/4051298693 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36380b3b40 msgr2=0x7f36380b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:12.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.303+0000 7f3652289640 1 --2- 172.21.15.42:0/4051298693 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36380b3b40 0x7f36380b6050 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3634005e10 tx=0x7f3634005da0 comp rx=0 tx=0).stop 2024-07-26T06:41:12.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.303+0000 7f3652289640 1 -- 172.21.15.42:0/4051298693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f364c06b3b0 msgr2=0x7f364c1998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.303+0000 7f3652289640 1 --2- 172.21.15.42:0/4051298693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f364c06b3b0 0x7f364c1998c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f363c00c920 tx=0x7f363c00cde0 comp rx=0 tx=0).stop 2024-07-26T06:41:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.303+0000 7f3652289640 1 -- 172.21.15.42:0/4051298693 shutdown_connections 2024-07-26T06:41:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.303+0000 7f3652289640 1 --2- 172.21.15.42:0/4051298693 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36380b3b40 0x7f36380b6050 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.303+0000 7f3652289640 1 --2- 172.21.15.42:0/4051298693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f364c108ea0 0x7f364c19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.303+0000 7f3652289640 1 --2- 172.21.15.42:0/4051298693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f364c106300 0x7f364c199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.303+0000 7f3652289640 1 --2- 172.21.15.42:0/4051298693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f364c06b3b0 0x7f364c1998c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.303+0000 7f3652289640 1 -- 172.21.15.42:0/4051298693 >> 172.21.15.42:0/4051298693 conn(0x7f364c0fd9e0 msgr2=0x7f364c104770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.303+0000 7f3652289640 1 -- 172.21.15.42:0/4051298693 shutdown_connections 2024-07-26T06:41:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:12.303+0000 7f3652289640 1 -- 172.21.15.42:0/4051298693 wait complete. 2024-07-26T06:41:12.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:12 smithi042 ceph-mon[32775]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:12.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:12 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/4051298693' entity='client.admin' 2024-07-26T06:41:12.927 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi042') 2024-07-26T06:41:12.927 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi042') 2024-07-26T06:41:12.927 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi042') 2024-07-26T06:41:12.927 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi042') 2024-07-26T06:41:12.927 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi042') 2024-07-26T06:41:12.927 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi042') 2024-07-26T06:41:12.928 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi042') 2024-07-26T06:41:12.928 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '8b3e2750-4b19-11ef-bcb6-c7b262605968', '--', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi042') 2024-07-26T06:41:12.928 INFO:tasks.cephadm:Deploying OSDs... 2024-07-26T06:41:12.928 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-07-26T06:41:12.928 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T06:41:12.946 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T06:41:12.947 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_1 2024-07-26T06:41:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:12 smithi164 ceph-mon[36970]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:12 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/4051298693' entity='client.admin' 2024-07-26T06:41:13.003 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T06:41:13.003 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:41:13.003 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 674 Links: 1 2024-07-26T06:41:13.003 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:41:13.003 INFO:teuthology.orchestra.run.smithi042.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:41:13.003 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-07-26 06:39:32.980351633 +0000 2024-07-26T06:41:13.003 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-07-26 06:34:19.632899556 +0000 2024-07-26T06:41:13.004 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-07-26 06:34:19.632899556 +0000 2024-07-26T06:41:13.004 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: 2024-07-26 06:34:19.632899556 +0000 2024-07-26T06:41:13.004 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T06:41:13.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:12 smithi163 ceph-mon[36873]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:13.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:12 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/4051298693' entity='client.admin' 2024-07-26T06:41:13.072 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-07-26T06:41:13.072 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-07-26T06:41:13.072 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000112679 s, 4.5 MB/s 2024-07-26T06:41:13.073 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T06:41:13.134 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_2 2024-07-26T06:41:13.191 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T06:41:13.191 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:41:13.191 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 690 Links: 1 2024-07-26T06:41:13.192 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:41:13.192 INFO:teuthology.orchestra.run.smithi042.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:41:13.192 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-07-26 06:39:33.014351343 +0000 2024-07-26T06:41:13.192 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-07-26 06:34:19.969896641 +0000 2024-07-26T06:41:13.192 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-07-26 06:34:19.969896641 +0000 2024-07-26T06:41:13.192 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: 2024-07-26 06:34:19.969896641 +0000 2024-07-26T06:41:13.192 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T06:41:13.257 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-07-26T06:41:13.257 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-07-26T06:41:13.257 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000113168 s, 4.5 MB/s 2024-07-26T06:41:13.258 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T06:41:13.316 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_3 2024-07-26T06:41:13.373 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T06:41:13.373 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:41:13.373 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 712 Links: 1 2024-07-26T06:41:13.373 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:41:13.373 INFO:teuthology.orchestra.run.smithi042.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:41:13.373 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-07-26 06:39:33.045351079 +0000 2024-07-26T06:41:13.373 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-07-26 06:34:20.297893804 +0000 2024-07-26T06:41:13.373 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-07-26 06:34:20.297893804 +0000 2024-07-26T06:41:13.373 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: 2024-07-26 06:34:20.297893804 +0000 2024-07-26T06:41:13.374 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T06:41:13.437 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-07-26T06:41:13.438 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-07-26T06:41:13.438 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000110092 s, 4.7 MB/s 2024-07-26T06:41:13.439 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T06:41:13.496 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_4 2024-07-26T06:41:13.552 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T06:41:13.552 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:41:13.552 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 728 Links: 1 2024-07-26T06:41:13.552 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:41:13.552 INFO:teuthology.orchestra.run.smithi042.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:41:13.552 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-07-26 06:39:33.076350814 +0000 2024-07-26T06:41:13.552 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-07-26 06:34:20.630890923 +0000 2024-07-26T06:41:13.553 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-07-26 06:34:20.630890923 +0000 2024-07-26T06:41:13.553 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: 2024-07-26 06:34:20.630890923 +0000 2024-07-26T06:41:13.553 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T06:41:13.616 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-07-26T06:41:13.616 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-07-26T06:41:13.617 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 9.4556e-05 s, 5.4 MB/s 2024-07-26T06:41:13.618 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T06:41:13.675 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-07-26T06:41:13.675 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T06:41:13.695 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T06:41:13.695 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_1 2024-07-26T06:41:13.753 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T06:41:13.753 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:41:13.753 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-07-26T06:41:13.753 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:41:13.753 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:41:13.753 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-07-26 06:40:31.162212900 +0000 2024-07-26T06:41:13.753 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-07-26 06:34:49.170129470 +0000 2024-07-26T06:41:13.753 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-07-26 06:34:49.170129470 +0000 2024-07-26T06:41:13.753 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-07-26 06:34:49.170129470 +0000 2024-07-26T06:41:13.754 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T06:41:13.821 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-07-26T06:41:13.822 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-07-26T06:41:13.822 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.0002914 s, 1.8 MB/s 2024-07-26T06:41:13.823 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T06:41:13.882 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_2 2024-07-26T06:41:13.940 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T06:41:13.940 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:41:13.940 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-07-26T06:41:13.940 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:41:13.940 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:41:13.940 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-07-26 06:40:31.163212878 +0000 2024-07-26T06:41:13.940 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-07-26 06:34:49.503122005 +0000 2024-07-26T06:41:13.940 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-07-26 06:34:49.503122005 +0000 2024-07-26T06:41:13.941 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-07-26 06:34:49.503122005 +0000 2024-07-26T06:41:13.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T06:41:14.005 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-07-26T06:41:14.006 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-07-26T06:41:14.006 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000241503 s, 2.1 MB/s 2024-07-26T06:41:14.007 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T06:41:14.064 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_3 2024-07-26T06:41:14.120 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T06:41:14.120 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:41:14.120 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-07-26T06:41:14.120 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:41:14.120 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:41:14.121 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-07-26 06:40:31.163212878 +0000 2024-07-26T06:41:14.121 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-07-26 06:34:49.846114315 +0000 2024-07-26T06:41:14.121 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-07-26 06:34:49.846114315 +0000 2024-07-26T06:41:14.121 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-07-26 06:34:49.846114315 +0000 2024-07-26T06:41:14.121 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T06:41:14.184 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-07-26T06:41:14.184 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-07-26T06:41:14.184 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000240769 s, 2.1 MB/s 2024-07-26T06:41:14.185 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T06:41:14.242 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_4 2024-07-26T06:41:14.299 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T06:41:14.299 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:41:14.299 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-07-26T06:41:14.299 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:41:14.299 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:41:14.300 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-07-26 06:40:31.163212878 +0000 2024-07-26T06:41:14.300 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-07-26 06:34:50.170107052 +0000 2024-07-26T06:41:14.300 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-07-26 06:34:50.170107052 +0000 2024-07-26T06:41:14.300 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-07-26 06:34:50.170107052 +0000 2024-07-26T06:41:14.301 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T06:41:14.363 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-07-26T06:41:14.364 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-07-26T06:41:14.364 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000226297 s, 2.3 MB/s 2024-07-26T06:41:14.365 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T06:41:14.422 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-07-26T06:41:14.423 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T06:41:14.443 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T06:41:14.444 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 2024-07-26T06:41:14.502 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T06:41:14.502 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:41:14.502 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-07-26T06:41:14.502 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:41:14.502 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:41:14.502 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-07-26 06:40:28.765774873 +0000 2024-07-26T06:41:14.502 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-07-26 06:34:34.205331242 +0000 2024-07-26T06:41:14.502 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-07-26 06:34:34.205331242 +0000 2024-07-26T06:41:14.502 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-07-26 06:34:34.205331242 +0000 2024-07-26T06:41:14.503 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T06:41:14.573 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-07-26T06:41:14.573 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-07-26T06:41:14.573 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000254578 s, 2.0 MB/s 2024-07-26T06:41:14.574 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T06:41:14.633 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 2024-07-26T06:41:14.691 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T06:41:14.692 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:41:14.692 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-07-26T06:41:14.692 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:41:14.692 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:41:14.692 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-07-26 06:40:28.765774873 +0000 2024-07-26T06:41:14.692 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-07-26 06:34:34.550325450 +0000 2024-07-26T06:41:14.692 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-07-26 06:34:34.550325450 +0000 2024-07-26T06:41:14.692 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-07-26 06:34:34.550325450 +0000 2024-07-26T06:41:14.693 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T06:41:14.757 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-07-26T06:41:14.757 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-07-26T06:41:14.757 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000208607 s, 2.5 MB/s 2024-07-26T06:41:14.758 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T06:41:14.816 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 2024-07-26T06:41:14.873 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T06:41:14.873 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:41:14.873 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-07-26T06:41:14.874 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:41:14.874 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:41:14.874 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-07-26 06:40:28.766774857 +0000 2024-07-26T06:41:14.874 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-07-26 06:34:34.872320045 +0000 2024-07-26T06:41:14.874 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-07-26 06:34:34.872320045 +0000 2024-07-26T06:41:14.874 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-07-26 06:34:34.872320045 +0000 2024-07-26T06:41:14.874 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T06:41:14.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:14 smithi042 ceph-mon[32775]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:14.935 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:14 smithi164 ceph-mon[36970]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:14.938 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-07-26T06:41:14.938 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-07-26T06:41:14.938 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000214057 s, 2.4 MB/s 2024-07-26T06:41:14.939 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T06:41:14.997 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 2024-07-26T06:41:15.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:14 smithi163 ceph-mon[36873]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:15.053 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T06:41:15.054 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T06:41:15.054 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-07-26T06:41:15.054 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:41:15.054 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T06:41:15.054 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-07-26 06:40:28.766774857 +0000 2024-07-26T06:41:15.054 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-07-26 06:34:35.195314623 +0000 2024-07-26T06:41:15.054 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-07-26 06:34:35.195314623 +0000 2024-07-26T06:41:15.054 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-07-26 06:34:35.195314623 +0000 2024-07-26T06:41:15.054 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T06:41:15.117 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-07-26T06:41:15.117 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-07-26T06:41:15.118 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000206859 s, 2.5 MB/s 2024-07-26T06:41:15.119 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T06:41:15.176 INFO:tasks.cephadm:Deploying osd.0 on smithi042 with /dev/vg_nvme/lv_4... 2024-07-26T06:41:15.176 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T06:41:15.445 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:41:16.751 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:16 smithi042 ceph-mon[32775]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:16 smithi164 ceph-mon[36970]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:17.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:16 smithi163 ceph-mon[36873]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:17.130 INFO:teuthology.orchestra.run.smithi042.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T06:41:17.226 INFO:teuthology.orchestra.run.smithi042.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T06:41:17.237 INFO:teuthology.orchestra.run.smithi042.stderr: stderr: 10+0 records in 2024-07-26T06:41:17.238 INFO:teuthology.orchestra.run.smithi042.stderr:10+0 records out 2024-07-26T06:41:17.238 INFO:teuthology.orchestra.run.smithi042.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0086313 s, 1.2 GB/s 2024-07-26T06:41:17.238 INFO:teuthology.orchestra.run.smithi042.stderr:--> Zapping successful for: 2024-07-26T06:41:17.747 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi042:vg_nvme/lv_4 2024-07-26T06:41:18.007 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:41:18.632 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:18 smithi042 ceph-mon[32775]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:18 smithi164 ceph-mon[36970]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:19.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:18 smithi163 ceph-mon[36873]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:19.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.363+0000 7fb62e4c0640 1 -- 172.21.15.42:0/4206513295 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb628101fc0 msgr2=0x7fb6281087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:19.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.363+0000 7fb62e4c0640 1 --2- 172.21.15.42:0/4206513295 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb628101fc0 0x7fb6281087a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009900 tx=0x7fb61c02a410 comp rx=0 tx=0).stop 2024-07-26T06:41:19.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.363+0000 7fb62e4c0640 1 -- 172.21.15.42:0/4206513295 shutdown_connections 2024-07-26T06:41:19.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.363+0000 7fb62e4c0640 1 --2- 172.21.15.42:0/4206513295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb628108ee0 0x7fb62810b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.363+0000 7fb62e4c0640 1 --2- 172.21.15.42:0/4206513295 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb628101fc0 0x7fb6281087a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.363+0000 7fb62e4c0640 1 --2- 172.21.15.42:0/4206513295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb628101590 0x7fb6281019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.363+0000 7fb62e4c0640 1 -- 172.21.15.42:0/4206513295 >> 172.21.15.42:0/4206513295 conn(0x7fb6280fd1a0 msgr2=0x7fb6280ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.363+0000 7fb62e4c0640 1 -- 172.21.15.42:0/4206513295 shutdown_connections 2024-07-26T06:41:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.363+0000 7fb62e4c0640 1 -- 172.21.15.42:0/4206513295 wait complete. 2024-07-26T06:41:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.363+0000 7fb62e4c0640 1 Processor -- start 2024-07-26T06:41:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.363+0000 7fb62e4c0640 1 -- start start 2024-07-26T06:41:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.364+0000 7fb62e4c0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb628101590 0x7fb62819bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.364+0000 7fb62e4c0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb628101fc0 0x7fb62819c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:19.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.364+0000 7fb62e4c0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb628108ee0 0x7fb62819cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:19.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.364+0000 7fb62e4c0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb62810ee00 con 0x7fb628101590 2024-07-26T06:41:19.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.364+0000 7fb62e4c0640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb62810ec80 con 0x7fb628101fc0 2024-07-26T06:41:19.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.364+0000 7fb62e4c0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb62810ef80 con 0x7fb628108ee0 2024-07-26T06:41:19.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.364+0000 7fb627fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb628101590 0x7fb62819bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:19.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.364+0000 7fb627fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb628101590 0x7fb62819bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56690/0 (socket says 172.21.15.42:56690) 2024-07-26T06:41:19.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.364+0000 7fb627fff640 1 -- 172.21.15.42:0/3404492941 learned_addr learned my addr 172.21.15.42:0/3404492941 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:41:19.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.364+0000 7fb6277fe640 1 --2- 172.21.15.42:0/3404492941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb628101fc0 0x7fb62819c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:19.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.364+0000 7fb62ca5d640 1 --2- 172.21.15.42:0/3404492941 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb628108ee0 0x7fb62819cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:19.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.364+0000 7fb627fff640 1 -- 172.21.15.42:0/3404492941 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb628108ee0 msgr2=0x7fb62819cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:19.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.364+0000 7fb627fff640 1 --2- 172.21.15.42:0/3404492941 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb628108ee0 0x7fb62819cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:19.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.364+0000 7fb627fff640 1 -- 172.21.15.42:0/3404492941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb628101fc0 msgr2=0x7fb62819c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:19.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.364+0000 7fb627fff640 1 --2- 172.21.15.42:0/3404492941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb628101fc0 0x7fb62819c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:19.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.364+0000 7fb627fff640 1 -- 172.21.15.42:0/3404492941 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62819d610 con 0x7fb628101590 2024-07-26T06:41:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.364+0000 7fb627fff640 1 --2- 172.21.15.42:0/3404492941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb628101590 0x7fb62819bbe0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb61800c890 tx=0x7fb61800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.365+0000 7fb6257fa640 1 -- 172.21.15.42:0/3404492941 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb618004220 con 0x7fb628101590 2024-07-26T06:41:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.365+0000 7fb6257fa640 1 -- 172.21.15.42:0/3404492941 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6180043c0 con 0x7fb628101590 2024-07-26T06:41:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.365+0000 7fb62e4c0640 1 -- 172.21.15.42:0/3404492941 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb62819d900 con 0x7fb628101590 2024-07-26T06:41:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.365+0000 7fb6257fa640 1 -- 172.21.15.42:0/3404492941 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb618004a00 con 0x7fb628101590 2024-07-26T06:41:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.365+0000 7fb62e4c0640 1 -- 172.21.15.42:0/3404492941 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6281a4000 con 0x7fb628101590 2024-07-26T06:41:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.366+0000 7fb62e4c0640 1 -- 172.21.15.42:0/3404492941 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6281019a0 con 0x7fb628101590 2024-07-26T06:41:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.367+0000 7fb6257fa640 1 -- 172.21.15.42:0/3404492941 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7fb6180079b0 con 0x7fb628101590 2024-07-26T06:41:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.368+0000 7fb6257fa640 1 --2- 172.21.15.42:0/3404492941 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb6100b3b40 0x7fb6100b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.369+0000 7fb6277fe640 1 --2- 172.21.15.42:0/3404492941 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb6100b3b40 0x7fb6100b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:19.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.369+0000 7fb6257fa640 1 -- 172.21.15.42:0/3404492941 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb6180e1db0 con 0x7fb628101590 2024-07-26T06:41:19.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.369+0000 7fb6277fe640 1 --2- 172.21.15.42:0/3404492941 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb6100b3b40 0x7fb6100b6050 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb61c031440 tx=0x7fb61c0314a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:19.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.373+0000 7fb6257fa640 1 -- 172.21.15.42:0/3404492941 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb618014030 con 0x7fb628101590 2024-07-26T06:41:19.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:19.568+0000 7fb62e4c0640 1 -- 172.21.15.42:0/3404492941 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fb6281a41d0 con 0x7fb6100b3b40 2024-07-26T06:41:20.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:20 smithi042 ceph-mon[32775]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:20.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:20 smithi042 ceph-mon[32775]: from='client.14346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:41:20.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:41:20.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:41:20.860 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:20 smithi164 ceph-mon[36970]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:20 smithi164 ceph-mon[36970]: from='client.14346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:41:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:41:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:41:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:21.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:20 smithi163 ceph-mon[36873]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:20 smithi163 ceph-mon[36873]: from='client.14346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:41:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:41:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:41:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:22 smithi042 ceph-mon[32775]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:22 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2433560675' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bda88e0-09ee-447b-8c04-24f4e5d55d19"}]: dispatch 2024-07-26T06:41:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:22 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2433560675' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bda88e0-09ee-447b-8c04-24f4e5d55d19"}]': finished 2024-07-26T06:41:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:22 smithi042 ceph-mon[32775]: osdmap e5: 1 total, 0 up, 1 in 2024-07-26T06:41:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T06:41:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:22 smithi164 ceph-mon[36970]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:22 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2433560675' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bda88e0-09ee-447b-8c04-24f4e5d55d19"}]: dispatch 2024-07-26T06:41:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:22 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2433560675' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bda88e0-09ee-447b-8c04-24f4e5d55d19"}]': finished 2024-07-26T06:41:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:22 smithi164 ceph-mon[36970]: osdmap e5: 1 total, 0 up, 1 in 2024-07-26T06:41:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T06:41:23.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:22 smithi163 ceph-mon[36873]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:22 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2433560675' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bda88e0-09ee-447b-8c04-24f4e5d55d19"}]: dispatch 2024-07-26T06:41:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:22 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2433560675' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bda88e0-09ee-447b-8c04-24f4e5d55d19"}]': finished 2024-07-26T06:41:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:22 smithi163 ceph-mon[36873]: osdmap e5: 1 total, 0 up, 1 in 2024-07-26T06:41:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T06:41:23.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:23 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/76931742' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:41:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:23 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/76931742' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:41:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:23 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/76931742' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:41:24.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:24 smithi042 ceph-mon[32775]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:24 smithi164 ceph-mon[36970]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:25.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:24 smithi163 ceph-mon[36873]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:26.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:26 smithi042 ceph-mon[32775]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:26 smithi164 ceph-mon[36970]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:26 smithi163 ceph-mon[36873]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:28.733 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:28 smithi042 ceph-mon[32775]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:28 smithi164 ceph-mon[36970]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:29.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:28 smithi163 ceph-mon[36873]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:29.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T06:41:29.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:29.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:29 smithi042 ceph-mon[32775]: Deploying daemon osd.0 on smithi042 2024-07-26T06:41:29.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T06:41:29.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:29.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:29 smithi164 ceph-mon[36970]: Deploying daemon osd.0 on smithi042 2024-07-26T06:41:30.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T06:41:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:29 smithi163 ceph-mon[36873]: Deploying daemon osd.0 on smithi042 2024-07-26T06:41:30.878 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:30 smithi042 ceph-mon[32775]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:30.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:30 smithi163 ceph-mon[36873]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:30.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:30 smithi164 ceph-mon[36970]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:32 smithi042 ceph-mon[32775]: pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:32 smithi164 ceph-mon[36970]: pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:33.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:32 smithi163 ceph-mon[36873]: pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:34.850 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:34 smithi042 ceph-mon[32775]: pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:34.850 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:41:34.850 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:34.850 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:34.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:34 smithi164 ceph-mon[36970]: pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:34.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:41:34.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:34.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:35.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:34 smithi163 ceph-mon[36873]: pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:35.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:41:35.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:35.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:36.717+0000 7fb6257fa640 1 -- 172.21.15.42:0/3404492941 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb6281a41d0 con 0x7fb6100b3b40 2024-07-26T06:41:36.718 INFO:teuthology.orchestra.run.smithi042.stdout:Created osd(s) 0 on host 'smithi042' 2024-07-26T06:41:36.719 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:36 smithi042 ceph-mon[32775]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:36.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:36.719+0000 7fb62e4c0640 1 -- 172.21.15.42:0/3404492941 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb6100b3b40 msgr2=0x7fb6100b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:36.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:36.719+0000 7fb62e4c0640 1 --2- 172.21.15.42:0/3404492941 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb6100b3b40 0x7fb6100b6050 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb61c031440 tx=0x7fb61c0314a0 comp rx=0 tx=0).stop 2024-07-26T06:41:36.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:36.719+0000 7fb62e4c0640 1 -- 172.21.15.42:0/3404492941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb628101590 msgr2=0x7fb62819bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:36.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:36.719+0000 7fb62e4c0640 1 --2- 172.21.15.42:0/3404492941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb628101590 0x7fb62819bbe0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb61800c890 tx=0x7fb61800cd50 comp rx=0 tx=0).stop 2024-07-26T06:41:36.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:36.719+0000 7fb62e4c0640 1 -- 172.21.15.42:0/3404492941 shutdown_connections 2024-07-26T06:41:36.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:36.719+0000 7fb62e4c0640 1 --2- 172.21.15.42:0/3404492941 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb6100b3b40 0x7fb6100b6050 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:36.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:36.719+0000 7fb62e4c0640 1 --2- 172.21.15.42:0/3404492941 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb628108ee0 0x7fb62819cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:36.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:36.719+0000 7fb62e4c0640 1 --2- 172.21.15.42:0/3404492941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb628101fc0 0x7fb62819c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:36.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:36.719+0000 7fb62e4c0640 1 --2- 172.21.15.42:0/3404492941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb628101590 0x7fb62819bbe0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:36.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:36.719+0000 7fb62e4c0640 1 -- 172.21.15.42:0/3404492941 >> 172.21.15.42:0/3404492941 conn(0x7fb6280fd1a0 msgr2=0x7fb628102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:36.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:36.719+0000 7fb62e4c0640 1 -- 172.21.15.42:0/3404492941 shutdown_connections 2024-07-26T06:41:36.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:41:36.719+0000 7fb62e4c0640 1 -- 172.21.15.42:0/3404492941 wait complete. 2024-07-26T06:41:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:36 smithi164 ceph-mon[36970]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:37.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:36 smithi163 ceph-mon[36873]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:37.360 DEBUG:teuthology.orchestra.run.smithi042:osd.0> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@osd.0.service 2024-07-26T06:41:37.362 INFO:tasks.cephadm:Deploying osd.1 on smithi163 with /dev/vg_nvme/lv_4... 2024-07-26T06:41:37.362 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T06:41:37.621 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.b/config 2024-07-26T06:41:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:38.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:38.021 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:38.022 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:38.022 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:38.022 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:38.022 INFO:journalctl@ceph.osd.0.smithi042.stdout:Jul 26 06:41:37 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-0[50247]: 2024-07-26T06:41:37.718+0000 7f8c22329640 -1 osd.0 0 log_to_monitors true 2024-07-26T06:41:38.834 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:38 smithi163 ceph-mon[36873]: pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:38.834 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:38 smithi163 ceph-mon[36873]: from='osd.0 [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T06:41:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:38 smithi164 ceph-mon[36970]: pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:38 smithi164 ceph-mon[36970]: from='osd.0 [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T06:41:39.084 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:38 smithi042 ceph-mon[32775]: pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:39.084 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:38 smithi042 ceph-mon[32775]: from='osd.0 [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T06:41:39.196 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T06:41:39.285 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T06:41:39.298 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-07-26T06:41:39.298 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-07-26T06:41:39.299 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115782 s, 906 MB/s 2024-07-26T06:41:39.299 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-07-26T06:41:39.871 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:39 smithi163 ceph-mon[36873]: from='osd.0 [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T06:41:39.871 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:39 smithi163 ceph-mon[36873]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T06:41:39.871 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:39 smithi163 ceph-mon[36873]: from='osd.0 [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]: dispatch 2024-07-26T06:41:39.871 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T06:41:39.872 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi163:vg_nvme/lv_4 2024-07-26T06:41:39.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:39 smithi164 ceph-mon[36970]: from='osd.0 [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T06:41:39.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:39 smithi164 ceph-mon[36970]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T06:41:39.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:39 smithi164 ceph-mon[36970]: from='osd.0 [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]: dispatch 2024-07-26T06:41:39.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T06:41:40.119 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.b/config 2024-07-26T06:41:40.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:39 smithi042 ceph-mon[32775]: from='osd.0 [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T06:41:40.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:39 smithi042 ceph-mon[32775]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T06:41:40.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:39 smithi042 ceph-mon[32775]: from='osd.0 [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]: dispatch 2024-07-26T06:41:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T06:41:40.145 INFO:journalctl@ceph.osd.0.smithi042.stdout:Jul 26 06:41:39 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-0[50247]: 2024-07-26T06:41:39.729+0000 7f8c1e2ab640 -1 osd.0 0 waiting for initial osdmap 2024-07-26T06:41:40.145 INFO:journalctl@ceph.osd.0.smithi042.stdout:Jul 26 06:41:39 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-0[50247]: 2024-07-26T06:41:39.732+0000 7f8c1a0a9640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:41:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: purged_snaps scrub starts 2024-07-26T06:41:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: purged_snaps scrub ok 2024-07-26T06:41:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: from='osd.0 [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-07-26T06:41:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T06:41:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T06:41:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T06:41:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:41:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:41:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:41:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:41:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:40.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: purged_snaps scrub starts 2024-07-26T06:41:40.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: purged_snaps scrub ok 2024-07-26T06:41:40.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:40.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: from='osd.0 [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-07-26T06:41:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T06:41:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T06:41:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T06:41:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:41:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:41:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:41:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:41:40.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: purged_snaps scrub starts 2024-07-26T06:41:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: purged_snaps scrub ok 2024-07-26T06:41:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T06:41:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: from='osd.0 [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-07-26T06:41:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T06:41:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T06:41:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T06:41:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:41:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:41:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:41:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:41:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:41.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.538+0000 7ff51798b640 1 -- 172.21.15.163:0/1352619415 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff510104c50 msgr2=0x7ff51010b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:41.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.538+0000 7ff51798b640 1 --2- 172.21.15.163:0/1352619415 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff510104c50 0x7ff51010b5f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff508009ca0 tx=0x7ff50802a950 comp rx=0 tx=0).stop 2024-07-26T06:41:41.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.538+0000 7ff51798b640 1 -- 172.21.15.163:0/1352619415 shutdown_connections 2024-07-26T06:41:41.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.538+0000 7ff51798b640 1 --2- 172.21.15.163:0/1352619415 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff510104c50 0x7ff51010b5f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:41.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.538+0000 7ff51798b640 1 --2- 172.21.15.163:0/1352619415 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5101042b0 0x7ff5101046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:41.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.538+0000 7ff51798b640 1 --2- 172.21.15.163:0/1352619415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff510103220 0x7ff510103690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:41.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.538+0000 7ff51798b640 1 -- 172.21.15.163:0/1352619415 >> 172.21.15.163:0/1352619415 conn(0x7ff5100febb0 msgr2=0x7ff510101040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:41.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.539+0000 7ff51798b640 1 -- 172.21.15.163:0/1352619415 shutdown_connections 2024-07-26T06:41:41.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.539+0000 7ff51798b640 1 -- 172.21.15.163:0/1352619415 wait complete. 2024-07-26T06:41:41.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.539+0000 7ff51798b640 1 Processor -- start 2024-07-26T06:41:41.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.539+0000 7ff51798b640 1 -- start start 2024-07-26T06:41:41.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.539+0000 7ff51798b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff510103220 0x7ff51019ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:41.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.539+0000 7ff51798b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5101042b0 0x7ff5101a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:41.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.539+0000 7ff51798b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff510104c50 0x7ff5101a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:41.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.539+0000 7ff51798b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff51010ee50 con 0x7ff510103220 2024-07-26T06:41:41.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.539+0000 7ff51798b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff51010ecd0 con 0x7ff5101042b0 2024-07-26T06:41:41.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.539+0000 7ff51798b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff51010efd0 con 0x7ff510104c50 2024-07-26T06:41:41.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.539+0000 7ff514f26640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5101042b0 0x7ff5101a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:41.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.540+0000 7ff514f26640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5101042b0 0x7ff5101a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:39114/0 (socket says 172.21.15.163:39114) 2024-07-26T06:41:41.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.540+0000 7ff514f26640 1 -- 172.21.15.163:0/1216922757 learned_addr learned my addr 172.21.15.163:0/1216922757 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:41:41.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.540+0000 7ff515727640 1 --2- 172.21.15.163:0/1216922757 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff510103220 0x7ff51019ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:41.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.540+0000 7ff515f28640 1 --2- 172.21.15.163:0/1216922757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff510104c50 0x7ff5101a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:41.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.540+0000 7ff514f26640 1 -- 172.21.15.163:0/1216922757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff510104c50 msgr2=0x7ff5101a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:41.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.540+0000 7ff514f26640 1 --2- 172.21.15.163:0/1216922757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff510104c50 0x7ff5101a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:41.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.540+0000 7ff514f26640 1 -- 172.21.15.163:0/1216922757 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff510103220 msgr2=0x7ff51019ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:41.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.540+0000 7ff514f26640 1 --2- 172.21.15.163:0/1216922757 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff510103220 0x7ff51019ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:41.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.540+0000 7ff514f26640 1 -- 172.21.15.163:0/1216922757 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5101a1a00 con 0x7ff5101042b0 2024-07-26T06:41:41.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.540+0000 7ff515f28640 1 --2- 172.21.15.163:0/1216922757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff510104c50 0x7ff5101a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:41:41.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.540+0000 7ff514f26640 1 --2- 172.21.15.163:0/1216922757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5101042b0 0x7ff5101a0640 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff50000b4a0 tx=0x7ff50000b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:41.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.540+0000 7ff5067fc640 1 -- 172.21.15.163:0/1216922757 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff500014070 con 0x7ff5101042b0 2024-07-26T06:41:41.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.540+0000 7ff5067fc640 1 -- 172.21.15.163:0/1216922757 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff500010ac0 con 0x7ff5101042b0 2024-07-26T06:41:41.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.540+0000 7ff51798b640 1 -- 172.21.15.163:0/1216922757 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5101a1cf0 con 0x7ff5101042b0 2024-07-26T06:41:41.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.541+0000 7ff5067fc640 1 -- 172.21.15.163:0/1216922757 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff500017c30 con 0x7ff5101042b0 2024-07-26T06:41:41.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.541+0000 7ff51798b640 1 -- 172.21.15.163:0/1216922757 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5101a83f0 con 0x7ff5101042b0 2024-07-26T06:41:41.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.542+0000 7ff51798b640 1 -- 172.21.15.163:0/1216922757 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff510103690 con 0x7ff5101042b0 2024-07-26T06:41:41.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.544+0000 7ff5067fc640 1 -- 172.21.15.163:0/1216922757 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7ff500017070 con 0x7ff5101042b0 2024-07-26T06:41:41.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.545+0000 7ff5067fc640 1 --2- 172.21.15.163:0/1216922757 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff4fc0b3a90 0x7ff4fc0b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:41:41.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.545+0000 7ff5067fc640 1 -- 172.21.15.163:0/1216922757 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1767+0+0 (secure 0 0 0) 0x7ff5000e18b0 con 0x7ff5101042b0 2024-07-26T06:41:41.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.545+0000 7ff515727640 1 --2- 172.21.15.163:0/1216922757 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff4fc0b3a90 0x7ff4fc0b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:41:41.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.545+0000 7ff515727640 1 --2- 172.21.15.163:0/1216922757 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff4fc0b3a90 0x7ff4fc0b5fa0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0058c0 tx=0x7ff50c005c60 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:41:41.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.548+0000 7ff5067fc640 1 -- 172.21.15.163:0/1216922757 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff5000aa970 con 0x7ff5101042b0 2024-07-26T06:41:41.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:41.730+0000 7ff51798b640 1 -- 172.21.15.163:0/1216922757 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7ff510063230 con 0x7ff4fc0b3a90 2024-07-26T06:41:42.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:41 smithi163 ceph-mon[36873]: Detected new or changed devices on smithi042 2024-07-26T06:41:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:41 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 17092M 2024-07-26T06:41:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:41 smithi163 ceph-mon[36873]: osd.0 [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379] boot 2024-07-26T06:41:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:41 smithi163 ceph-mon[36873]: osdmap e8: 1 total, 1 up, 1 in 2024-07-26T06:41:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T06:41:42.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:41 smithi042 ceph-mon[32775]: Detected new or changed devices on smithi042 2024-07-26T06:41:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:41 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 17092M 2024-07-26T06:41:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:41 smithi042 ceph-mon[32775]: osd.0 [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379] boot 2024-07-26T06:41:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:41 smithi042 ceph-mon[32775]: osdmap e8: 1 total, 1 up, 1 in 2024-07-26T06:41:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T06:41:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:41 smithi164 ceph-mon[36970]: Detected new or changed devices on smithi042 2024-07-26T06:41:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:41 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 17092M 2024-07-26T06:41:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:41 smithi164 ceph-mon[36970]: osd.0 [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379] boot 2024-07-26T06:41:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:41 smithi164 ceph-mon[36970]: osdmap e8: 1 total, 1 up, 1 in 2024-07-26T06:41:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T06:41:42.933 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:42 smithi163 ceph-mon[36873]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:42.934 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:42 smithi163 ceph-mon[36873]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:41:42.934 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:41:42.934 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:42 smithi163 ceph-mon[36873]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T06:41:42.934 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:41:42.934 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:42 smithi042 ceph-mon[32775]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:42 smithi042 ceph-mon[32775]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:41:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:41:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:42 smithi042 ceph-mon[32775]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T06:41:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:41:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:43.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:42 smithi164 ceph-mon[36970]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:43.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:42 smithi164 ceph-mon[36970]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:41:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:41:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:42 smithi164 ceph-mon[36970]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T06:41:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:41:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:44 smithi163 ceph-mon[36873]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:44 smithi163 ceph-mon[36873]: from='client.? 172.21.15.163:0/3408627230' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d747138-8217-4836-a173-cf8ac65e9fe3"}]: dispatch 2024-07-26T06:41:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:44 smithi163 ceph-mon[36873]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d747138-8217-4836-a173-cf8ac65e9fe3"}]: dispatch 2024-07-26T06:41:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:44 smithi163 ceph-mon[36873]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d747138-8217-4836-a173-cf8ac65e9fe3"}]': finished 2024-07-26T06:41:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:44 smithi163 ceph-mon[36873]: osdmap e10: 2 total, 1 up, 2 in 2024-07-26T06:41:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:41:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:44 smithi163 ceph-mon[36873]: from='client.? 172.21.15.163:0/2807403757' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:41:45.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:44 smithi042 ceph-mon[32775]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:44 smithi042 ceph-mon[32775]: from='client.? 172.21.15.163:0/3408627230' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d747138-8217-4836-a173-cf8ac65e9fe3"}]: dispatch 2024-07-26T06:41:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:44 smithi042 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d747138-8217-4836-a173-cf8ac65e9fe3"}]: dispatch 2024-07-26T06:41:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:44 smithi042 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d747138-8217-4836-a173-cf8ac65e9fe3"}]': finished 2024-07-26T06:41:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:44 smithi042 ceph-mon[32775]: osdmap e10: 2 total, 1 up, 2 in 2024-07-26T06:41:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:41:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:44 smithi042 ceph-mon[32775]: from='client.? 172.21.15.163:0/2807403757' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:41:45.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:44 smithi164 ceph-mon[36970]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:45.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:44 smithi164 ceph-mon[36970]: from='client.? 172.21.15.163:0/3408627230' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d747138-8217-4836-a173-cf8ac65e9fe3"}]: dispatch 2024-07-26T06:41:45.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:44 smithi164 ceph-mon[36970]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d747138-8217-4836-a173-cf8ac65e9fe3"}]: dispatch 2024-07-26T06:41:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:44 smithi164 ceph-mon[36970]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d747138-8217-4836-a173-cf8ac65e9fe3"}]': finished 2024-07-26T06:41:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:44 smithi164 ceph-mon[36970]: osdmap e10: 2 total, 1 up, 2 in 2024-07-26T06:41:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:41:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:44 smithi164 ceph-mon[36970]: from='client.? 172.21.15.163:0/2807403757' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:41:47.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:46 smithi163 ceph-mon[36873]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:47.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:46 smithi042 ceph-mon[32775]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:46 smithi164 ceph-mon[36970]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:48.911 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:48 smithi163 ceph-mon[36873]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:49.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:48 smithi042 ceph-mon[32775]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:49.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:48 smithi164 ceph-mon[36970]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:50 smithi163 ceph-mon[36873]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T06:41:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:51.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:50 smithi042 ceph-mon[32775]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:51.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T06:41:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:50 smithi164 ceph-mon[36970]: pgmap v74: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T06:41:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:41:51.925 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:51 smithi163 ceph-mon[36873]: Deploying daemon osd.1 on smithi163 2024-07-26T06:41:52.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:51 smithi042 ceph-mon[32775]: Deploying daemon osd.1 on smithi163 2024-07-26T06:41:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:51 smithi164 ceph-mon[36970]: Deploying daemon osd.1 on smithi163 2024-07-26T06:41:52.856 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:52 smithi163 ceph-mon[36873]: pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:53.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:52 smithi042 ceph-mon[32775]: pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:53.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:52 smithi164 ceph-mon[36970]: pgmap v75: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:54 smithi163 ceph-mon[36873]: pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:55.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:54 smithi042 ceph-mon[32775]: pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:55.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:54 smithi164 ceph-mon[36970]: pgmap v76: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:56.096 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:41:56.096 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:56.096 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:56.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:41:56.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:56.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:56.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:41:56.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:56.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:57.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:56 smithi163 ceph-mon[36873]: pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:57.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:56 smithi042 ceph-mon[32775]: pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:56 smithi164 ceph-mon[36970]: pgmap v77: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:58.646 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 1 on host 'smithi163' 2024-07-26T06:41:58.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:58.645+0000 7ff5067fc640 1 -- 172.21.15.163:0/1216922757 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff510063230 con 0x7ff4fc0b3a90 2024-07-26T06:41:58.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:58.648+0000 7ff51798b640 1 -- 172.21.15.163:0/1216922757 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff4fc0b3a90 msgr2=0x7ff4fc0b5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:58.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:58.648+0000 7ff51798b640 1 --2- 172.21.15.163:0/1216922757 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff4fc0b3a90 0x7ff4fc0b5fa0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff50c0058c0 tx=0x7ff50c005c60 comp rx=0 tx=0).stop 2024-07-26T06:41:58.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:58.648+0000 7ff51798b640 1 -- 172.21.15.163:0/1216922757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5101042b0 msgr2=0x7ff5101a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:41:58.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:58.648+0000 7ff51798b640 1 --2- 172.21.15.163:0/1216922757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5101042b0 0x7ff5101a0640 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff50000b4a0 tx=0x7ff50000b960 comp rx=0 tx=0).stop 2024-07-26T06:41:58.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:58.648+0000 7ff51798b640 1 -- 172.21.15.163:0/1216922757 shutdown_connections 2024-07-26T06:41:58.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:58.648+0000 7ff51798b640 1 --2- 172.21.15.163:0/1216922757 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff4fc0b3a90 0x7ff4fc0b5fa0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:58.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:58.648+0000 7ff51798b640 1 --2- 172.21.15.163:0/1216922757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff510104c50 0x7ff5101a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:58.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:58.648+0000 7ff51798b640 1 --2- 172.21.15.163:0/1216922757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5101042b0 0x7ff5101a0640 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:58.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:58.648+0000 7ff51798b640 1 --2- 172.21.15.163:0/1216922757 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff510103220 0x7ff51019ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:41:58.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:58.648+0000 7ff51798b640 1 -- 172.21.15.163:0/1216922757 >> 172.21.15.163:0/1216922757 conn(0x7ff5100febb0 msgr2=0x7ff5101055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:41:58.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:58.648+0000 7ff51798b640 1 -- 172.21.15.163:0/1216922757 shutdown_connections 2024-07-26T06:41:58.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:41:58.648+0000 7ff51798b640 1 -- 172.21.15.163:0/1216922757 wait complete. 2024-07-26T06:41:58.993 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:58 smithi163 ceph-mon[36873]: pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:58.994 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:58.994 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:58.994 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:58.994 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:59.024 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:58 smithi164 ceph-mon[36970]: pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:59.024 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:59.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:59.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:59.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:59.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:58 smithi042 ceph-mon[32775]: pgmap v78: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:41:59.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:41:59.287 DEBUG:teuthology.orchestra.run.smithi163:osd.1> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@osd.1.service 2024-07-26T06:41:59.289 INFO:tasks.cephadm:Deploying osd.2 on smithi164 with /dev/vg_nvme/lv_4... 2024-07-26T06:41:59.290 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T06:41:59.546 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T06:41:59.786 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:59 smithi163 ceph-mon[36873]: from='osd.1 [v2:172.21.15.163:6800/360219078,v1:172.21.15.163:6801/360219078]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T06:41:59.786 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:41:59 smithi163 ceph-mon[36873]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T06:41:59.786 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jul 26 06:41:59 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-1[41416]: 2024-07-26T06:41:59.587+0000 7fe9d4949640 -1 osd.1 0 log_to_monitors true 2024-07-26T06:41:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:59 smithi164 ceph-mon[36970]: from='osd.1 [v2:172.21.15.163:6800/360219078,v1:172.21.15.163:6801/360219078]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T06:41:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:41:59 smithi164 ceph-mon[36970]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T06:42:00.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:59 smithi042 ceph-mon[32775]: from='osd.1 [v2:172.21.15.163:6800/360219078,v1:172.21.15.163:6801/360219078]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T06:42:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:41:59 smithi042 ceph-mon[32775]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T06:42:00.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:00 smithi164 ceph-mon[36970]: pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:42:00.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:00 smithi164 ceph-mon[36970]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T06:42:00.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:00 smithi164 ceph-mon[36970]: from='osd.1 [v2:172.21.15.163:6800/360219078,v1:172.21.15.163:6801/360219078]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:42:00.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:00 smithi164 ceph-mon[36970]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T06:42:00.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:42:00.894 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:00 smithi164 ceph-mon[36970]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:42:01.061 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:00 smithi163 ceph-mon[36873]: pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:42:01.061 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:00 smithi163 ceph-mon[36873]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T06:42:01.062 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:00 smithi163 ceph-mon[36873]: from='osd.1 [v2:172.21.15.163:6800/360219078,v1:172.21.15.163:6801/360219078]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:42:01.062 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:00 smithi163 ceph-mon[36873]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T06:42:01.062 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:42:01.062 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:00 smithi163 ceph-mon[36873]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:42:01.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:00 smithi042 ceph-mon[32775]: pgmap v79: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:42:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:00 smithi042 ceph-mon[32775]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T06:42:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:00 smithi042 ceph-mon[32775]: from='osd.1 [v2:172.21.15.163:6800/360219078,v1:172.21.15.163:6801/360219078]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:42:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:00 smithi042 ceph-mon[32775]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T06:42:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:42:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:00 smithi042 ceph-mon[32775]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:42:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T06:42:01.358 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T06:42:01.371 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2024-07-26T06:42:01.371 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2024-07-26T06:42:01.371 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117717 s, 891 MB/s 2024-07-26T06:42:01.372 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2024-07-26T06:42:01.507 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jul 26 06:42:01 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-1[41416]: 2024-07-26T06:42:01.238+0000 7fe9d08cb640 -1 osd.1 0 waiting for initial osdmap 2024-07-26T06:42:01.507 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jul 26 06:42:01 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-1[41416]: 2024-07-26T06:42:01.242+0000 7fe9c86c1640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:42:01.864 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:01 smithi163 ceph-mon[36873]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-07-26T06:42:01.864 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:01 smithi163 ceph-mon[36873]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T06:42:01.865 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:42:01.865 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:42:01.865 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:01 smithi163 ceph-mon[36873]: from='osd.1 ' entity='osd.1' 2024-07-26T06:42:01.994 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi164:vg_nvme/lv_4 2024-07-26T06:42:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:01 smithi042 ceph-mon[32775]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-07-26T06:42:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:01 smithi042 ceph-mon[32775]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T06:42:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:42:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:42:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:01 smithi042 ceph-mon[32775]: from='osd.1 ' entity='osd.1' 2024-07-26T06:42:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:01 smithi164 ceph-mon[36970]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-07-26T06:42:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:01 smithi164 ceph-mon[36970]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T06:42:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:42:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:42:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:01 smithi164 ceph-mon[36970]: from='osd.1 ' entity='osd.1' 2024-07-26T06:42:02.262 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T06:42:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: purged_snaps scrub starts 2024-07-26T06:42:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: purged_snaps scrub ok 2024-07-26T06:42:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: pgmap v82: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:42:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:42:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:42:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: osd.1 [v2:172.21.15.163:6800/360219078,v1:172.21.15.163:6801/360219078] boot 2024-07-26T06:42:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: osdmap e13: 2 total, 2 up, 2 in 2024-07-26T06:42:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:42:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:42:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: purged_snaps scrub starts 2024-07-26T06:42:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: purged_snaps scrub ok 2024-07-26T06:42:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: pgmap v82: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:42:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:42:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:42:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: osd.1 [v2:172.21.15.163:6800/360219078,v1:172.21.15.163:6801/360219078] boot 2024-07-26T06:42:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: osdmap e13: 2 total, 2 up, 2 in 2024-07-26T06:42:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:42:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:42:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: purged_snaps scrub starts 2024-07-26T06:42:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: purged_snaps scrub ok 2024-07-26T06:42:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: pgmap v82: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T06:42:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:42:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:42:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: osd.1 [v2:172.21.15.163:6800/360219078,v1:172.21.15.163:6801/360219078] boot 2024-07-26T06:42:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: osdmap e13: 2 total, 2 up, 2 in 2024-07-26T06:42:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T06:42:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:42:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:03.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.823+0000 7f156149f640 1 -- 172.21.15.164:0/1054596886 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f155c1015a0 msgr2=0x7f155c1019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:42:03.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.823+0000 7f156149f640 1 --2- 172.21.15.164:0/1054596886 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f155c1015a0 0x7f155c1019b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f154c009900 tx=0x7f154c02a410 comp rx=0 tx=0).stop 2024-07-26T06:42:03.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.823+0000 7f156149f640 1 -- 172.21.15.164:0/1054596886 shutdown_connections 2024-07-26T06:42:03.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.823+0000 7f156149f640 1 --2- 172.21.15.164:0/1054596886 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f155c108ed0 0x7f155c10b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:03.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.823+0000 7f156149f640 1 --2- 172.21.15.164:0/1054596886 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f155c101fd0 0x7f155c108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:03.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.823+0000 7f156149f640 1 --2- 172.21.15.164:0/1054596886 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f155c1015a0 0x7f155c1019b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:03.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.823+0000 7f156149f640 1 -- 172.21.15.164:0/1054596886 >> 172.21.15.164:0/1054596886 conn(0x7f155c0fd190 msgr2=0x7f155c0ff640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:42:03.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.823+0000 7f156149f640 1 -- 172.21.15.164:0/1054596886 shutdown_connections 2024-07-26T06:42:03.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.823+0000 7f156149f640 1 -- 172.21.15.164:0/1054596886 wait complete. 2024-07-26T06:42:03.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.824+0000 7f156149f640 1 Processor -- start 2024-07-26T06:42:03.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.824+0000 7f156149f640 1 -- start start 2024-07-26T06:42:03.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f156149f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f155c1015a0 0x7f155c1999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:42:03.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f156149f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f155c101fd0 0x7f155c19a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:42:03.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f156149f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f155c108ed0 0x7f155c19abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:42:03.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f156149f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f155c10edf0 con 0x7f155c108ed0 2024-07-26T06:42:03.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f156149f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f155c10ec70 con 0x7f155c101fd0 2024-07-26T06:42:03.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f156149f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f155c10ef70 con 0x7f155c1015a0 2024-07-26T06:42:03.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f155affd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f155c1015a0 0x7f155c1999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:42:03.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f155affd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f155c1015a0 0x7f155c1999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:53768/0 (socket says 172.21.15.164:53768) 2024-07-26T06:42:03.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f155affd640 1 -- 172.21.15.164:0/1069079412 learned_addr learned my addr 172.21.15.164:0/1069079412 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-07-26T06:42:03.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f15527fc640 1 --2- 172.21.15.164:0/1069079412 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f155c101fd0 0x7f155c19a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:42:03.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f155b7fe640 1 --2- 172.21.15.164:0/1069079412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f155c108ed0 0x7f155c19abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:42:03.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f155affd640 1 -- 172.21.15.164:0/1069079412 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f155c101fd0 msgr2=0x7f155c19a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:42:03.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f155affd640 1 --2- 172.21.15.164:0/1069079412 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f155c101fd0 0x7f155c19a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:03.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f155affd640 1 -- 172.21.15.164:0/1069079412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f155c108ed0 msgr2=0x7f155c19abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:42:03.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f155affd640 1 --2- 172.21.15.164:0/1069079412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f155c108ed0 0x7f155c19abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:03.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f155affd640 1 -- 172.21.15.164:0/1069079412 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f155c19b400 con 0x7f155c1015a0 2024-07-26T06:42:03.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f155b7fe640 1 --2- 172.21.15.164:0/1069079412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f155c108ed0 0x7f155c19abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:42:03.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f155affd640 1 --2- 172.21.15.164:0/1069079412 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f155c1015a0 0x7f155c1999d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f154c006010 tx=0x7f154c004340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:42:03.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f1558ff9640 1 -- 172.21.15.164:0/1069079412 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154c03f030 con 0x7f155c1015a0 2024-07-26T06:42:03.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f1558ff9640 1 -- 172.21.15.164:0/1069079412 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f154c003cb0 con 0x7f155c1015a0 2024-07-26T06:42:03.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.825+0000 7f1558ff9640 1 -- 172.21.15.164:0/1069079412 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154c035070 con 0x7f155c1015a0 2024-07-26T06:42:03.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.826+0000 7f156149f640 1 -- 172.21.15.164:0/1069079412 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f155c19b690 con 0x7f155c1015a0 2024-07-26T06:42:03.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.826+0000 7f156149f640 1 -- 172.21.15.164:0/1069079412 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f155c1a1c20 con 0x7f155c1015a0 2024-07-26T06:42:03.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.827+0000 7f156149f640 1 -- 172.21.15.164:0/1069079412 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f155c0fecb0 con 0x7f155c1015a0 2024-07-26T06:42:03.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.828+0000 7f1558ff9640 1 -- 172.21.15.164:0/1069079412 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7f154c042030 con 0x7f155c1015a0 2024-07-26T06:42:03.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.829+0000 7f1558ff9640 1 --2- 172.21.15.164:0/1069079412 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f15480b3b40 0x7f15480b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:42:03.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.829+0000 7f15527fc640 1 --2- 172.21.15.164:0/1069079412 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f15480b3b40 0x7f15480b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:42:03.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.829+0000 7f1558ff9640 1 -- 172.21.15.164:0/1069079412 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f154c1030f0 con 0x7f155c1015a0 2024-07-26T06:42:03.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.830+0000 7f15527fc640 1 --2- 172.21.15.164:0/1069079412 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f15480b3b40 0x7f15480b6050 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1544005fd0 tx=0x7f154400b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:42:03.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:03.833+0000 7f1558ff9640 1 -- 172.21.15.164:0/1069079412 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f154c0cbef0 con 0x7f155c1015a0 2024-07-26T06:42:04.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:04.014+0000 7f156149f640 1 -- 172.21.15.164:0/1069079412 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f155c063230 con 0x7f15480b3b40 2024-07-26T06:42:04.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:03 smithi042 ceph-mon[32775]: Detected new or changed devices on smithi163 2024-07-26T06:42:04.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:03 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 17088M 2024-07-26T06:42:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:03 smithi042 ceph-mon[32775]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T06:42:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:03 smithi164 ceph-mon[36970]: Detected new or changed devices on smithi163 2024-07-26T06:42:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:03 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 17088M 2024-07-26T06:42:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:03 smithi164 ceph-mon[36970]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T06:42:04.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:03 smithi163 ceph-mon[36873]: Detected new or changed devices on smithi163 2024-07-26T06:42:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:03 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 17088M 2024-07-26T06:42:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:03 smithi163 ceph-mon[36873]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T06:42:05.097 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:04 smithi164 ceph-mon[36970]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:05.098 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:04 smithi164 ceph-mon[36970]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:42:05.098 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:42:05.098 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:42:05.098 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:05.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:04 smithi042 ceph-mon[32775]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:05.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:04 smithi042 ceph-mon[32775]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:42:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:42:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:42:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:05.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:04 smithi163 ceph-mon[36873]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:04 smithi163 ceph-mon[36873]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:42:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:42:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:42:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:06.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:06 smithi164 ceph-mon[36970]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:06.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:06 smithi164 ceph-mon[36970]: from='client.? 172.21.15.164:0/4148082223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b7258c1-3d35-4930-ab10-0b04a2a2c8a2"}]: dispatch 2024-07-26T06:42:06.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:06 smithi164 ceph-mon[36970]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b7258c1-3d35-4930-ab10-0b04a2a2c8a2"}]: dispatch 2024-07-26T06:42:06.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:06 smithi164 ceph-mon[36970]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b7258c1-3d35-4930-ab10-0b04a2a2c8a2"}]': finished 2024-07-26T06:42:06.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:06 smithi164 ceph-mon[36970]: osdmap e15: 3 total, 2 up, 3 in 2024-07-26T06:42:06.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:07.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:06 smithi042 ceph-mon[32775]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:06 smithi042 ceph-mon[32775]: from='client.? 172.21.15.164:0/4148082223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b7258c1-3d35-4930-ab10-0b04a2a2c8a2"}]: dispatch 2024-07-26T06:42:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:06 smithi042 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b7258c1-3d35-4930-ab10-0b04a2a2c8a2"}]: dispatch 2024-07-26T06:42:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:06 smithi042 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b7258c1-3d35-4930-ab10-0b04a2a2c8a2"}]': finished 2024-07-26T06:42:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:06 smithi042 ceph-mon[32775]: osdmap e15: 3 total, 2 up, 3 in 2024-07-26T06:42:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:07.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:06 smithi163 ceph-mon[36873]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:07.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:06 smithi163 ceph-mon[36873]: from='client.? 172.21.15.164:0/4148082223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b7258c1-3d35-4930-ab10-0b04a2a2c8a2"}]: dispatch 2024-07-26T06:42:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:06 smithi163 ceph-mon[36873]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b7258c1-3d35-4930-ab10-0b04a2a2c8a2"}]: dispatch 2024-07-26T06:42:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:06 smithi163 ceph-mon[36873]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b7258c1-3d35-4930-ab10-0b04a2a2c8a2"}]': finished 2024-07-26T06:42:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:06 smithi163 ceph-mon[36873]: osdmap e15: 3 total, 2 up, 3 in 2024-07-26T06:42:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:08.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:07 smithi042 ceph-mon[32775]: from='client.? 172.21.15.164:0/2474090741' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:42:08.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:07 smithi164 ceph-mon[36970]: from='client.? 172.21.15.164:0/2474090741' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:42:08.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:07 smithi163 ceph-mon[36873]: from='client.? 172.21.15.164:0/2474090741' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:42:09.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:08 smithi042 ceph-mon[32775]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:09.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:08 smithi164 ceph-mon[36970]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:09.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:08 smithi163 ceph-mon[36873]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:10 smithi164 ceph-mon[36970]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:11.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:10 smithi042 ceph-mon[32775]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:11.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:10 smithi163 ceph-mon[36873]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:12 smithi042 ceph-mon[32775]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:13.191 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:12 smithi164 ceph-mon[36970]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:13.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:12 smithi163 ceph-mon[36873]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:13.919 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T06:42:13.919 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:14.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T06:42:14.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:14.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T06:42:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:15.079 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:14 smithi164 ceph-mon[36970]: Deploying daemon osd.2 on smithi164 2024-07-26T06:42:15.079 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:14 smithi164 ceph-mon[36970]: pgmap v91: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:15.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:14 smithi042 ceph-mon[32775]: Deploying daemon osd.2 on smithi164 2024-07-26T06:42:15.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:14 smithi042 ceph-mon[32775]: pgmap v91: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:15.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:14 smithi163 ceph-mon[36873]: Deploying daemon osd.2 on smithi164 2024-07-26T06:42:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:14 smithi163 ceph-mon[36873]: pgmap v91: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:16 smithi164 ceph-mon[36970]: pgmap v92: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:17.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:16 smithi042 ceph-mon[32775]: pgmap v92: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:17.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:16 smithi163 ceph-mon[36873]: pgmap v92: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:19.034 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:18 smithi164 ceph-mon[36970]: pgmap v93: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:19.034 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:42:19.034 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:19.035 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:19.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:18 smithi042 ceph-mon[32775]: pgmap v93: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:42:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:18 smithi163 ceph-mon[36873]: pgmap v93: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:42:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:20 smithi042 ceph-mon[32775]: pgmap v94: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:21.169 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:20 smithi164 ceph-mon[36970]: pgmap v94: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:21.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:21.209+0000 7f1558ff9640 1 -- 172.21.15.164:0/1069079412 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f155c063230 con 0x7f15480b3b40 2024-07-26T06:42:21.211 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 2 on host 'smithi164' 2024-07-26T06:42:21.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:21.211+0000 7f156149f640 1 -- 172.21.15.164:0/1069079412 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f15480b3b40 msgr2=0x7f15480b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:42:21.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:21.211+0000 7f156149f640 1 --2- 172.21.15.164:0/1069079412 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f15480b3b40 0x7f15480b6050 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1544005fd0 tx=0x7f154400b390 comp rx=0 tx=0).stop 2024-07-26T06:42:21.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:21.211+0000 7f156149f640 1 -- 172.21.15.164:0/1069079412 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f155c1015a0 msgr2=0x7f155c1999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:42:21.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:21.211+0000 7f156149f640 1 --2- 172.21.15.164:0/1069079412 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f155c1015a0 0x7f155c1999d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f154c006010 tx=0x7f154c004340 comp rx=0 tx=0).stop 2024-07-26T06:42:21.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:21.211+0000 7f156149f640 1 -- 172.21.15.164:0/1069079412 shutdown_connections 2024-07-26T06:42:21.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:21.211+0000 7f156149f640 1 --2- 172.21.15.164:0/1069079412 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f15480b3b40 0x7f15480b6050 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:21.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:21.211+0000 7f156149f640 1 --2- 172.21.15.164:0/1069079412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f155c108ed0 0x7f155c19abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:21.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:21.211+0000 7f156149f640 1 --2- 172.21.15.164:0/1069079412 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f155c101fd0 0x7f155c19a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:21.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:21.211+0000 7f156149f640 1 --2- 172.21.15.164:0/1069079412 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f155c1015a0 0x7f155c1999d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:21.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:21.211+0000 7f156149f640 1 -- 172.21.15.164:0/1069079412 >> 172.21.15.164:0/1069079412 conn(0x7f155c0fd190 msgr2=0x7f155c107240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:42:21.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:21.211+0000 7f156149f640 1 -- 172.21.15.164:0/1069079412 shutdown_connections 2024-07-26T06:42:21.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:42:21.211+0000 7f156149f640 1 -- 172.21.15.164:0/1069079412 wait complete. 2024-07-26T06:42:21.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:20 smithi163 ceph-mon[36873]: pgmap v94: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:22.135 DEBUG:teuthology.orchestra.run.smithi164:osd.2> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@osd.2.service 2024-07-26T06:42:22.138 INFO:tasks.cephadm:Deploying osd.3 on smithi042 with /dev/vg_nvme/lv_3... 2024-07-26T06:42:22.138 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T06:42:22.417 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:42:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:22 smithi164 ceph-mon[36970]: pgmap v95: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:22.472 INFO:journalctl@ceph.osd.2.smithi164.stdout:Jul 26 06:42:22 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-2[42059]: 2024-07-26T06:42:22.287+0000 7fcfd79df640 -1 osd.2 0 log_to_monitors true 2024-07-26T06:42:22.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:22 smithi163 ceph-mon[36873]: pgmap v95: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:22.540 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:22.540 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:22.540 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:22 smithi042 ceph-mon[32775]: pgmap v95: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:23 smithi163 ceph-mon[36873]: from='osd.2 [v2:172.21.15.164:6800/336113412,v1:172.21.15.164:6801/336113412]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T06:42:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:23 smithi163 ceph-mon[36873]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T06:42:23.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:23.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:23.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:23 smithi042 ceph-mon[32775]: from='osd.2 [v2:172.21.15.164:6800/336113412,v1:172.21.15.164:6801/336113412]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T06:42:23.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:23 smithi042 ceph-mon[32775]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T06:42:23.581 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:23.581 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:23.581 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:23 smithi164 ceph-mon[36970]: from='osd.2 [v2:172.21.15.164:6800/336113412,v1:172.21.15.164:6801/336113412]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T06:42:23.581 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:23 smithi164 ceph-mon[36970]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T06:42:24.149 INFO:teuthology.orchestra.run.smithi042.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T06:42:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T06:42:24.243 INFO:teuthology.orchestra.run.smithi042.stderr: stderr: 10+0 records in 2024-07-26T06:42:24.243 INFO:teuthology.orchestra.run.smithi042.stderr:10+0 records out 2024-07-26T06:42:24.243 INFO:teuthology.orchestra.run.smithi042.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00881009 s, 1.2 GB/s 2024-07-26T06:42:24.244 INFO:teuthology.orchestra.run.smithi042.stderr:--> Zapping successful for: 2024-07-26T06:42:24.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:24 smithi163 ceph-mon[36873]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T06:42:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:24 smithi163 ceph-mon[36873]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T06:42:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:24 smithi163 ceph-mon[36873]: from='osd.2 [v2:172.21.15.164:6800/336113412,v1:172.21.15.164:6801/336113412]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:42:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:24 smithi163 ceph-mon[36873]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:42:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:24 smithi163 ceph-mon[36873]: pgmap v97: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:24.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:24 smithi042 ceph-mon[32775]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T06:42:24.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:24 smithi042 ceph-mon[32775]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T06:42:24.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:24.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:24 smithi042 ceph-mon[32775]: from='osd.2 [v2:172.21.15.164:6800/336113412,v1:172.21.15.164:6801/336113412]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:42:24.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:24 smithi042 ceph-mon[32775]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:42:24.574 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:24 smithi042 ceph-mon[32775]: pgmap v97: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:24.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:24 smithi164 ceph-mon[36970]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T06:42:24.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:24 smithi164 ceph-mon[36970]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T06:42:24.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:24.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:24 smithi164 ceph-mon[36970]: from='osd.2 [v2:172.21.15.164:6800/336113412,v1:172.21.15.164:6801/336113412]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:42:24.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:24 smithi164 ceph-mon[36970]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:42:24.605 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:24 smithi164 ceph-mon[36970]: pgmap v97: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:24.838 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi042:vg_nvme/lv_3 2024-07-26T06:42:24.972 INFO:journalctl@ceph.osd.2.smithi164.stdout:Jul 26 06:42:24 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-2[42059]: 2024-07-26T06:42:24.720+0000 7fcfd3961640 -1 osd.2 0 waiting for initial osdmap 2024-07-26T06:42:24.972 INFO:journalctl@ceph.osd.2.smithi164.stdout:Jul 26 06:42:24 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-2[42059]: 2024-07-26T06:42:24.725+0000 7fcfcb757640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:42:25.100 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:42:25.288 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:25 smithi164 ceph-mon[36970]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-07-26T06:42:25.288 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:25 smithi164 ceph-mon[36970]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T06:42:25.288 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:25.288 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:25.289 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:25 smithi164 ceph-mon[36970]: from='osd.2 ' entity='osd.2' 2024-07-26T06:42:25.502 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:25 smithi042 ceph-mon[32775]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-07-26T06:42:25.502 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:25 smithi042 ceph-mon[32775]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T06:42:25.503 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:25.503 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:25.503 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:25 smithi042 ceph-mon[32775]: from='osd.2 ' entity='osd.2' 2024-07-26T06:42:25.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:25 smithi163 ceph-mon[36873]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-07-26T06:42:25.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:25 smithi163 ceph-mon[36873]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T06:42:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:25 smithi163 ceph-mon[36873]: from='osd.2 ' entity='osd.2' 2024-07-26T06:42:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: purged_snaps scrub starts 2024-07-26T06:42:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: purged_snaps scrub ok 2024-07-26T06:42:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: pgmap v99: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: Detected new or changed devices on smithi164 2024-07-26T06:42:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:42:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 17088M 2024-07-26T06:42:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:42:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: osd.2 [v2:172.21.15.164:6800/336113412,v1:172.21.15.164:6801/336113412] boot 2024-07-26T06:42:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: osdmap e18: 3 total, 3 up, 3 in 2024-07-26T06:42:26.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: purged_snaps scrub starts 2024-07-26T06:42:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: purged_snaps scrub ok 2024-07-26T06:42:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: pgmap v99: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: Detected new or changed devices on smithi164 2024-07-26T06:42:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:42:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 17088M 2024-07-26T06:42:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:42:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: osd.2 [v2:172.21.15.164:6800/336113412,v1:172.21.15.164:6801/336113412] boot 2024-07-26T06:42:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: osdmap e18: 3 total, 3 up, 3 in 2024-07-26T06:42:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: purged_snaps scrub starts 2024-07-26T06:42:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: purged_snaps scrub ok 2024-07-26T06:42:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: pgmap v99: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T06:42:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: Detected new or changed devices on smithi164 2024-07-26T06:42:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:42:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 17088M 2024-07-26T06:42:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:42:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: osd.2 [v2:172.21.15.164:6800/336113412,v1:172.21.15.164:6801/336113412] boot 2024-07-26T06:42:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: osdmap e18: 3 total, 3 up, 3 in 2024-07-26T06:42:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T06:42:27.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:26.999+0000 7effc2b84640 1 -- 172.21.15.42:0/1361053451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7effbc108ea0 msgr2=0x7effbc10b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:42:27.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:26.999+0000 7effc2b84640 1 --2- 172.21.15.42:0/1361053451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7effbc108ea0 0x7effbc10b3b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7effb4009ca0 tx=0x7effb402a8c0 comp rx=0 tx=0).stop 2024-07-26T06:42:27.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:26.999+0000 7effc2b84640 1 -- 172.21.15.42:0/1361053451 shutdown_connections 2024-07-26T06:42:27.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:26.999+0000 7effc2b84640 1 --2- 172.21.15.42:0/1361053451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7effbc108ea0 0x7effbc10b3b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:27.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:26.999+0000 7effc2b84640 1 --2- 172.21.15.42:0/1361053451 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effbc1063c0 0x7effbc1087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:27.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:26.999+0000 7effc2b84640 1 --2- 172.21.15.42:0/1361053451 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effbc06b2f0 0x7effbc105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:27.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:26.999+0000 7effc2b84640 1 -- 172.21.15.42:0/1361053451 >> 172.21.15.42:0/1361053451 conn(0x7effbc0fd9e0 msgr2=0x7effbc0ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:42:27.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:26.999+0000 7effc2b84640 1 -- 172.21.15.42:0/1361053451 shutdown_connections 2024-07-26T06:42:27.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:26.999+0000 7effc2b84640 1 -- 172.21.15.42:0/1361053451 wait complete. 2024-07-26T06:42:27.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.000+0000 7effc2b84640 1 Processor -- start 2024-07-26T06:42:27.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.000+0000 7effc2b84640 1 -- start start 2024-07-26T06:42:27.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.000+0000 7effc2b84640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effbc06b2f0 0x7effbc199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:42:27.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.000+0000 7effc2b84640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7effbc1063c0 0x7effbc199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:42:27.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.000+0000 7effc2b84640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effbc108ea0 0x7effbc19ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:42:27.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.000+0000 7effc2b84640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7effbc10edc0 con 0x7effbc1063c0 2024-07-26T06:42:27.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.000+0000 7effc2b84640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7effbc10ec40 con 0x7effbc06b2f0 2024-07-26T06:42:27.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.000+0000 7effc2b84640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7effbc10ef40 con 0x7effbc108ea0 2024-07-26T06:42:27.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.001+0000 7effbbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7effbc1063c0 0x7effbc199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:42:27.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.001+0000 7effbbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7effbc1063c0 0x7effbc199fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36414/0 (socket says 172.21.15.42:36414) 2024-07-26T06:42:27.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.001+0000 7effbbfff640 1 -- 172.21.15.42:0/3317105638 learned_addr learned my addr 172.21.15.42:0/3317105638 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:42:27.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.001+0000 7effc1121640 1 --2- 172.21.15.42:0/3317105638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effbc108ea0 0x7effbc19ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:42:27.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.001+0000 7effc0920640 1 --2- 172.21.15.42:0/3317105638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effbc06b2f0 0x7effbc199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:42:27.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.001+0000 7effbbfff640 1 -- 172.21.15.42:0/3317105638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effbc108ea0 msgr2=0x7effbc19ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:42:27.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.001+0000 7effbbfff640 1 --2- 172.21.15.42:0/3317105638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effbc108ea0 0x7effbc19ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:27.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.001+0000 7effbbfff640 1 -- 172.21.15.42:0/3317105638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effbc06b2f0 msgr2=0x7effbc199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:42:27.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.001+0000 7effbbfff640 1 --2- 172.21.15.42:0/3317105638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effbc06b2f0 0x7effbc199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:27.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.001+0000 7effbbfff640 1 -- 172.21.15.42:0/3317105638 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effbc19b3a0 con 0x7effbc1063c0 2024-07-26T06:42:27.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.001+0000 7effbbfff640 1 --2- 172.21.15.42:0/3317105638 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7effbc1063c0 0x7effbc199fe0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7effb000c890 tx=0x7effb000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:42:27.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.001+0000 7effb9ffb640 1 -- 172.21.15.42:0/3317105638 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb0004220 con 0x7effbc1063c0 2024-07-26T06:42:27.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.001+0000 7effb9ffb640 1 -- 172.21.15.42:0/3317105638 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effb00043c0 con 0x7effbc1063c0 2024-07-26T06:42:27.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.001+0000 7effc2b84640 1 -- 172.21.15.42:0/3317105638 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effbc19b690 con 0x7effbc1063c0 2024-07-26T06:42:27.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.001+0000 7effb9ffb640 1 -- 172.21.15.42:0/3317105638 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb0004a00 con 0x7effbc1063c0 2024-07-26T06:42:27.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.002+0000 7effc2b84640 1 -- 172.21.15.42:0/3317105638 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effbc1a1d90 con 0x7effbc1063c0 2024-07-26T06:42:27.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.003+0000 7effc2b84640 1 -- 172.21.15.42:0/3317105638 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effbc0ff500 con 0x7effbc1063c0 2024-07-26T06:42:27.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.004+0000 7effb9ffb640 1 -- 172.21.15.42:0/3317105638 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 15) ==== 151003+0+0 (secure 0 0 0) 0x7effb00079b0 con 0x7effbc1063c0 2024-07-26T06:42:27.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.005+0000 7effb9ffb640 1 --2- 172.21.15.42:0/3317105638 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7effa40b3bc0 0x7effa40b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:42:27.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.005+0000 7effc0920640 1 --2- 172.21.15.42:0/3317105638 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7effa40b3bc0 0x7effa40b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:42:27.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.005+0000 7effb9ffb640 1 -- 172.21.15.42:0/3317105638 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2851+0+0 (secure 0 0 0) 0x7effb00e2270 con 0x7effbc1063c0 2024-07-26T06:42:27.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.005+0000 7effc0920640 1 --2- 172.21.15.42:0/3317105638 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7effa40b3bc0 0x7effa40b60d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7effac005fd0 tx=0x7effac00b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:42:27.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.009+0000 7effb9ffb640 1 -- 172.21.15.42:0/3317105638 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7effb0014030 con 0x7effbc1063c0 2024-07-26T06:42:27.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:27.203+0000 7effc2b84640 1 -- 172.21.15.42:0/3317105638 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7effbc063230 con 0x7effa40b3bc0 2024-07-26T06:42:27.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:42:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:42:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:27.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:42:27.628 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:42:27.628 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:42:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:42:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:28.458 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:28 smithi042 ceph-mon[32775]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:42:28.458 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:28 smithi042 ceph-mon[32775]: pgmap v101: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:28.458 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:28 smithi042 ceph-mon[32775]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T06:42:28.458 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T06:42:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:28 smithi164 ceph-mon[36970]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:42:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:28 smithi164 ceph-mon[36970]: pgmap v101: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:28 smithi164 ceph-mon[36970]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T06:42:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T06:42:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:28 smithi163 ceph-mon[36873]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:42:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:28 smithi163 ceph-mon[36873]: pgmap v101: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:28 smithi163 ceph-mon[36873]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T06:42:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T06:42:29.536 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T06:42:29.536 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:29 smithi042 ceph-mon[32775]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T06:42:29.536 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T06:42:29.537 INFO:journalctl@ceph.osd.0.smithi042.stdout:Jul 26 06:42:29 smithi042 sudo[54811]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T06:42:29.537 INFO:journalctl@ceph.osd.0.smithi042.stdout:Jul 26 06:42:29 smithi042 sudo[54811]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T06:42:29.537 INFO:journalctl@ceph.osd.0.smithi042.stdout:Jul 26 06:42:29 smithi042 sudo[54811]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T06:42:29.537 INFO:journalctl@ceph.osd.0.smithi042.stdout:Jul 26 06:42:29 smithi042 sudo[54811]: pam_unix(sudo:session): session closed for user root 2024-07-26T06:42:29.537 INFO:journalctl@ceph.osd.0.smithi042.stdout:Jul 26 06:42:29 smithi042 sudo[54814]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T06:42:29.537 INFO:journalctl@ceph.osd.0.smithi042.stdout:Jul 26 06:42:29 smithi042 sudo[54814]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T06:42:29.537 INFO:journalctl@ceph.osd.0.smithi042.stdout:Jul 26 06:42:29 smithi042 sudo[54814]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T06:42:29.537 INFO:journalctl@ceph.osd.0.smithi042.stdout:Jul 26 06:42:29 smithi042 sudo[54814]: pam_unix(sudo:session): session closed for user root 2024-07-26T06:42:29.581 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T06:42:29.581 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:29 smithi164 ceph-mon[36970]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T06:42:29.581 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T06:42:29.581 INFO:journalctl@ceph.osd.2.smithi164.stdout:Jul 26 06:42:29 smithi164 sudo[45807]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T06:42:29.581 INFO:journalctl@ceph.osd.2.smithi164.stdout:Jul 26 06:42:29 smithi164 sudo[45807]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T06:42:29.581 INFO:journalctl@ceph.osd.2.smithi164.stdout:Jul 26 06:42:29 smithi164 sudo[45807]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T06:42:29.581 INFO:journalctl@ceph.osd.2.smithi164.stdout:Jul 26 06:42:29 smithi164 sudo[45807]: pam_unix(sudo:session): session closed for user root 2024-07-26T06:42:29.581 INFO:journalctl@ceph.osd.2.smithi164.stdout:Jul 26 06:42:29 smithi164 sudo[45810]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T06:42:29.581 INFO:journalctl@ceph.osd.2.smithi164.stdout:Jul 26 06:42:29 smithi164 sudo[45810]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T06:42:29.581 INFO:journalctl@ceph.osd.2.smithi164.stdout:Jul 26 06:42:29 smithi164 sudo[45810]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T06:42:29.582 INFO:journalctl@ceph.osd.2.smithi164.stdout:Jul 26 06:42:29 smithi164 sudo[45810]: pam_unix(sudo:session): session closed for user root 2024-07-26T06:42:29.622 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T06:42:29.622 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:29 smithi163 ceph-mon[36873]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T06:42:29.622 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T06:42:29.622 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jul 26 06:42:29 smithi163 sudo[45646]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T06:42:29.622 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jul 26 06:42:29 smithi163 sudo[45646]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T06:42:29.622 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jul 26 06:42:29 smithi163 sudo[45646]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T06:42:29.622 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jul 26 06:42:29 smithi163 sudo[45646]: pam_unix(sudo:session): session closed for user root 2024-07-26T06:42:29.623 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jul 26 06:42:29 smithi163 sudo[45649]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T06:42:29.623 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jul 26 06:42:29 smithi163 sudo[45649]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T06:42:29.623 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jul 26 06:42:29 smithi163 sudo[45649]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T06:42:29.623 INFO:journalctl@ceph.osd.1.smithi163.stdout:Jul 26 06:42:29 smithi163 sudo[45649]: pam_unix(sudo:session): session closed for user root 2024-07-26T06:42:29.845 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:29 smithi164 sudo[45813]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T06:42:29.845 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:29 smithi164 sudo[45813]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T06:42:29.845 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:29 smithi164 sudo[45813]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T06:42:29.846 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:29 smithi164 sudo[45813]: pam_unix(sudo:session): session closed for user root 2024-07-26T06:42:29.846 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:29 smithi164 sudo[45817]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T06:42:29.846 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:29 smithi164 sudo[45817]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T06:42:29.846 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:29 smithi164 sudo[45817]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T06:42:29.846 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:29 smithi164 sudo[45817]: pam_unix(sudo:session): session closed for user root 2024-07-26T06:42:29.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:29 smithi042 sudo[54817]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T06:42:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:29 smithi042 sudo[54817]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T06:42:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:29 smithi042 sudo[54817]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T06:42:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:29 smithi042 sudo[54817]: pam_unix(sudo:session): session closed for user root 2024-07-26T06:42:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:29 smithi042 sudo[54821]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T06:42:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:29 smithi042 sudo[54821]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T06:42:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:29 smithi042 sudo[54821]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T06:42:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:29 smithi042 sudo[54821]: pam_unix(sudo:session): session closed for user root 2024-07-26T06:42:30.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:29 smithi163 sudo[45652]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T06:42:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:29 smithi163 sudo[45652]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T06:42:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:29 smithi163 sudo[45652]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T06:42:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:29 smithi163 sudo[45652]: pam_unix(sudo:session): session closed for user root 2024-07-26T06:42:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:29 smithi163 sudo[45655]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T06:42:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:29 smithi163 sudo[45655]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T06:42:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:29 smithi163 sudo[45655]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T06:42:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:29 smithi163 sudo[45655]: pam_unix(sudo:session): session closed for user root 2024-07-26T06:42:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: pgmap v104: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T06:42:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T06:42:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T06:42:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T06:42:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T06:42:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:42:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1968967720' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96608387-0de0-4d90-a952-d6eba61f7fbe"}]: dispatch 2024-07-26T06:42:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:42:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:42:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1968967720' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96608387-0de0-4d90-a952-d6eba61f7fbe"}]': finished 2024-07-26T06:42:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T06:42:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: osdmap e22: 4 total, 3 up, 4 in 2024-07-26T06:42:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T06:42:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:42:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:42:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:42:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T06:42:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T06:42:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:42:30.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:42:30.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:42:30.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T06:42:30.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:30 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3490958634' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:42:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: pgmap v104: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T06:42:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T06:42:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T06:42:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T06:42:30.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T06:42:30.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:42:30.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1968967720' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96608387-0de0-4d90-a952-d6eba61f7fbe"}]: dispatch 2024-07-26T06:42:30.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:42:30.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:42:30.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1968967720' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96608387-0de0-4d90-a952-d6eba61f7fbe"}]': finished 2024-07-26T06:42:30.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T06:42:30.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: osdmap e22: 4 total, 3 up, 4 in 2024-07-26T06:42:30.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T06:42:30.727 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:42:30.727 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:42:30.727 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:42:30.727 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T06:42:30.727 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T06:42:30.728 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:42:30.728 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:42:30.728 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:42:30.728 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T06:42:30.728 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:30 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3490958634' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:42:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: pgmap v104: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T06:42:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T06:42:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T06:42:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T06:42:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T06:42:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:42:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1968967720' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96608387-0de0-4d90-a952-d6eba61f7fbe"}]: dispatch 2024-07-26T06:42:30.760 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:42:30.760 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:42:30.760 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1968967720' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96608387-0de0-4d90-a952-d6eba61f7fbe"}]': finished 2024-07-26T06:42:30.760 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T06:42:30.760 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: osdmap e22: 4 total, 3 up, 4 in 2024-07-26T06:42:30.760 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T06:42:30.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:42:30.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:42:30.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:42:30.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T06:42:30.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T06:42:30.762 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T06:42:30.762 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T06:42:30.762 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T06:42:30.762 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T06:42:30.762 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:30 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3490958634' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:42:31.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:31 smithi163 ceph-mon[36873]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T06:42:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:31 smithi163 ceph-mon[36873]: Cluster is now healthy 2024-07-26T06:42:31.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:31 smithi042 ceph-mon[32775]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T06:42:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:31 smithi042 ceph-mon[32775]: Cluster is now healthy 2024-07-26T06:42:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:31 smithi164 ceph-mon[36970]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T06:42:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:31 smithi164 ceph-mon[36970]: Cluster is now healthy 2024-07-26T06:42:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:32.282+0000 7effb9ffb640 1 -- 172.21.15.42:0/3317105638 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7effb00a6a90 con 0x7effbc1063c0 2024-07-26T06:42:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:32 smithi042 ceph-mon[32775]: pgmap v107: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:32 smithi042 ceph-mon[32775]: mgrmap e16: x(active, since 3m), standbys: z, y 2024-07-26T06:42:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:32 smithi164 ceph-mon[36970]: pgmap v107: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:32 smithi164 ceph-mon[36970]: mgrmap e16: x(active, since 3m), standbys: z, y 2024-07-26T06:42:32.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:32 smithi163 ceph-mon[36873]: pgmap v107: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:32 smithi163 ceph-mon[36873]: mgrmap e16: x(active, since 3m), standbys: z, y 2024-07-26T06:42:34.596 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:34 smithi042 ceph-mon[32775]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:34 smithi164 ceph-mon[36970]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:34.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:34 smithi163 ceph-mon[36873]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:36.598 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:36 smithi042 ceph-mon[32775]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:36 smithi164 ceph-mon[36970]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:36.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:36 smithi163 ceph-mon[36873]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:37.633 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T06:42:37.633 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:37.633 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:37 smithi042 ceph-mon[32775]: Deploying daemon osd.3 on smithi042 2024-07-26T06:42:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T06:42:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:37 smithi164 ceph-mon[36970]: Deploying daemon osd.3 on smithi042 2024-07-26T06:42:37.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T06:42:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:37 smithi163 ceph-mon[36873]: Deploying daemon osd.3 on smithi042 2024-07-26T06:42:38.597 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:38 smithi042 ceph-mon[32775]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:38 smithi164 ceph-mon[36970]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:38.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:38 smithi163 ceph-mon[36873]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:40.499 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:40 smithi042 ceph-mon[32775]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:40.701 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:40 smithi163 ceph-mon[36873]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:40 smithi164 ceph-mon[36970]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:42.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:42 smithi042 ceph-mon[32775]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:42.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:42:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:42 smithi164 ceph-mon[36970]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:42:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:42.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:42 smithi163 ceph-mon[36873]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:42:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:44.475 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:44 smithi042 ceph-mon[32775]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:44 smithi164 ceph-mon[36970]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:44.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:44 smithi163 ceph-mon[36873]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:45.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:45.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:45.980 INFO:teuthology.orchestra.run.smithi042.stdout:Created osd(s) 3 on host 'smithi042' 2024-07-26T06:42:45.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:45.979+0000 7effb9ffb640 1 -- 172.21.15.42:0/3317105638 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7effbc063230 con 0x7effa40b3bc0 2024-07-26T06:42:45.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:45.982+0000 7effc2b84640 1 -- 172.21.15.42:0/3317105638 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7effa40b3bc0 msgr2=0x7effa40b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:42:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:45.982+0000 7effc2b84640 1 --2- 172.21.15.42:0/3317105638 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7effa40b3bc0 0x7effa40b60d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7effac005fd0 tx=0x7effac00b390 comp rx=0 tx=0).stop 2024-07-26T06:42:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:45.982+0000 7effc2b84640 1 -- 172.21.15.42:0/3317105638 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7effbc1063c0 msgr2=0x7effbc199fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:42:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:45.982+0000 7effc2b84640 1 --2- 172.21.15.42:0/3317105638 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7effbc1063c0 0x7effbc199fe0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7effb000c890 tx=0x7effb000cd50 comp rx=0 tx=0).stop 2024-07-26T06:42:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:45.982+0000 7effc2b84640 1 -- 172.21.15.42:0/3317105638 shutdown_connections 2024-07-26T06:42:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:45.982+0000 7effc2b84640 1 --2- 172.21.15.42:0/3317105638 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7effa40b3bc0 0x7effa40b60d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:45.982+0000 7effc2b84640 1 --2- 172.21.15.42:0/3317105638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effbc108ea0 0x7effbc19ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:45.982+0000 7effc2b84640 1 --2- 172.21.15.42:0/3317105638 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7effbc1063c0 0x7effbc199fe0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:45.982+0000 7effc2b84640 1 --2- 172.21.15.42:0/3317105638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effbc06b2f0 0x7effbc199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:45.982+0000 7effc2b84640 1 -- 172.21.15.42:0/3317105638 >> 172.21.15.42:0/3317105638 conn(0x7effbc0fd9e0 msgr2=0x7effbc1048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:42:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:45.982+0000 7effc2b84640 1 -- 172.21.15.42:0/3317105638 shutdown_connections 2024-07-26T06:42:45.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:42:45.982+0000 7effc2b84640 1 -- 172.21.15.42:0/3317105638 wait complete. 2024-07-26T06:42:46.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:46.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:46.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:46.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:46.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:46.726 DEBUG:teuthology.orchestra.run.smithi042:osd.3> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@osd.3.service 2024-07-26T06:42:46.729 INFO:tasks.cephadm:Deploying osd.4 on smithi163 with /dev/vg_nvme/lv_3... 2024-07-26T06:42:46.729 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T06:42:46.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:46 smithi042 ceph-mon[32775]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:46.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:42:46.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:46.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:46.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:46 smithi042 ceph-mon[32775]: from='osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T06:42:46.992 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.b/config 2024-07-26T06:42:47.114 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:46 smithi163 ceph-mon[36873]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:47.115 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:42:47.115 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:47.115 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:47.115 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:46 smithi163 ceph-mon[36873]: from='osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T06:42:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:46 smithi164 ceph-mon[36970]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:42:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:46 smithi164 ceph-mon[36970]: from='osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T06:42:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:47 smithi042 ceph-mon[32775]: from='osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T06:42:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:47 smithi042 ceph-mon[32775]: osdmap e23: 4 total, 3 up, 4 in 2024-07-26T06:42:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:47 smithi042 ceph-mon[32775]: from='osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]: dispatch 2024-07-26T06:42:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T06:42:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:47 smithi163 ceph-mon[36873]: from='osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T06:42:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:47 smithi163 ceph-mon[36873]: osdmap e23: 4 total, 3 up, 4 in 2024-07-26T06:42:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:47 smithi163 ceph-mon[36873]: from='osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]: dispatch 2024-07-26T06:42:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T06:42:48.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:47 smithi164 ceph-mon[36970]: from='osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T06:42:48.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:47 smithi164 ceph-mon[36970]: osdmap e23: 4 total, 3 up, 4 in 2024-07-26T06:42:48.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:47 smithi164 ceph-mon[36970]: from='osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]: dispatch 2024-07-26T06:42:48.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T06:42:48.645 INFO:journalctl@ceph.osd.3.smithi042.stdout:Jul 26 06:42:48 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-3[56828]: 2024-07-26T06:42:48.438+0000 7f3888852640 -1 osd.3 0 waiting for initial osdmap 2024-07-26T06:42:48.645 INFO:journalctl@ceph.osd.3.smithi042.stdout:Jul 26 06:42:48 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-3[56828]: 2024-07-26T06:42:48.443+0000 7f3880648640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:42:48.795 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T06:42:48.876 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T06:42:48.888 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-07-26T06:42:48.889 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-07-26T06:42:48.889 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011286 s, 929 MB/s 2024-07-26T06:42:48.889 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-07-26T06:42:49.216 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:48 smithi163 ceph-mon[36873]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:49.216 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:48 smithi163 ceph-mon[36873]: from='osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-07-26T06:42:49.216 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:48 smithi163 ceph-mon[36873]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T06:42:49.216 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T06:42:49.216 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T06:42:49.217 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:48 smithi163 ceph-mon[36873]: from='osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064]' entity='osd.3' 2024-07-26T06:42:49.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:48 smithi042 ceph-mon[32775]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:49.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:48 smithi042 ceph-mon[32775]: from='osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-07-26T06:42:49.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:48 smithi042 ceph-mon[32775]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T06:42:49.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T06:42:49.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T06:42:49.276 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:48 smithi042 ceph-mon[32775]: from='osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064]' entity='osd.3' 2024-07-26T06:42:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:48 smithi164 ceph-mon[36970]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T06:42:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:48 smithi164 ceph-mon[36970]: from='osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-07-26T06:42:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:48 smithi164 ceph-mon[36970]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T06:42:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T06:42:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T06:42:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:48 smithi164 ceph-mon[36970]: from='osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064]' entity='osd.3' 2024-07-26T06:42:49.544 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi163:vg_nvme/lv_3 2024-07-26T06:42:49.805 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.b/config 2024-07-26T06:42:50.083 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:49 smithi163 ceph-mon[36873]: purged_snaps scrub starts 2024-07-26T06:42:50.083 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:49 smithi163 ceph-mon[36873]: purged_snaps scrub ok 2024-07-26T06:42:50.083 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:49 smithi163 ceph-mon[36873]: osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064] boot 2024-07-26T06:42:50.083 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:49 smithi163 ceph-mon[36873]: osdmap e25: 4 total, 4 up, 4 in 2024-07-26T06:42:50.084 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T06:42:50.084 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:49 smithi163 ceph-mon[36873]: osdmap e26: 4 total, 4 up, 4 in 2024-07-26T06:42:50.084 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:50.084 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:50.084 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:42:50.084 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:42:50.084 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:50.084 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:50.084 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:50.084 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:49 smithi042 ceph-mon[32775]: purged_snaps scrub starts 2024-07-26T06:42:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:49 smithi042 ceph-mon[32775]: purged_snaps scrub ok 2024-07-26T06:42:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:49 smithi042 ceph-mon[32775]: osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064] boot 2024-07-26T06:42:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:49 smithi042 ceph-mon[32775]: osdmap e25: 4 total, 4 up, 4 in 2024-07-26T06:42:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T06:42:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:49 smithi042 ceph-mon[32775]: osdmap e26: 4 total, 4 up, 4 in 2024-07-26T06:42:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:42:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:42:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:50.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:49 smithi164 ceph-mon[36970]: purged_snaps scrub starts 2024-07-26T06:42:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:49 smithi164 ceph-mon[36970]: purged_snaps scrub ok 2024-07-26T06:42:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:49 smithi164 ceph-mon[36970]: osd.3 [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064] boot 2024-07-26T06:42:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:49 smithi164 ceph-mon[36970]: osdmap e25: 4 total, 4 up, 4 in 2024-07-26T06:42:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T06:42:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:49 smithi164 ceph-mon[36970]: osdmap e26: 4 total, 4 up, 4 in 2024-07-26T06:42:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:42:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:42:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:50.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:42:50.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:42:51.146 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:50 smithi163 ceph-mon[36873]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:42:51.146 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:50 smithi163 ceph-mon[36873]: Detected new or changed devices on smithi042 2024-07-26T06:42:51.146 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:50 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 8546M 2024-07-26T06:42:51.146 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:50 smithi163 ceph-mon[36873]: osdmap e27: 4 total, 4 up, 4 in 2024-07-26T06:42:51.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.358+0000 7f0cfd008640 1 -- 172.21.15.163:0/2595582837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0cf81051c0 msgr2=0x7f0cf81076a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:42:51.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.358+0000 7f0cfd008640 1 --2- 172.21.15.163:0/2595582837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0cf81051c0 0x7f0cf81076a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0009900 tx=0x7f0ce002a410 comp rx=0 tx=0).stop 2024-07-26T06:42:51.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.358+0000 7f0cfd008640 1 -- 172.21.15.163:0/2595582837 shutdown_connections 2024-07-26T06:42:51.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.358+0000 7f0cfd008640 1 --2- 172.21.15.163:0/2595582837 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cf8107de0 0x7f0cf810a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:51.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.358+0000 7f0cfd008640 1 --2- 172.21.15.163:0/2595582837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0cf81051c0 0x7f0cf81076a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:51.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.358+0000 7f0cfd008640 1 --2- 172.21.15.163:0/2595582837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0cf8102760 0x7f0cf8104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:51.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.358+0000 7f0cfd008640 1 -- 172.21.15.163:0/2595582837 >> 172.21.15.163:0/2595582837 conn(0x7f0cf80fd1a0 msgr2=0x7f0cf80ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:42:51.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.358+0000 7f0cfd008640 1 -- 172.21.15.163:0/2595582837 shutdown_connections 2024-07-26T06:42:51.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.358+0000 7f0cfd008640 1 -- 172.21.15.163:0/2595582837 wait complete. 2024-07-26T06:42:51.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.359+0000 7f0cfd008640 1 Processor -- start 2024-07-26T06:42:51.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.359+0000 7f0cfd008640 1 -- start start 2024-07-26T06:42:51.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.359+0000 7f0cfd008640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0cf8102760 0x7f0cf8101b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:42:51.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.359+0000 7f0cfd008640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0cf81051c0 0x7f0cf81a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:42:51.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.359+0000 7f0cfd008640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cf8107de0 0x7f0cf81a3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:42:51.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.359+0000 7f0cfd008640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf810dd00 con 0x7f0cf8107de0 2024-07-26T06:42:51.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.359+0000 7f0cfd008640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf810db80 con 0x7f0cf8102760 2024-07-26T06:42:51.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0cfd008640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf810de80 con 0x7f0cf81051c0 2024-07-26T06:42:51.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0cf6d9d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0cf8102760 0x7f0cf8101b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:42:51.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0cf6d9d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0cf8102760 0x7f0cf8101b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:58508/0 (socket says 172.21.15.163:58508) 2024-07-26T06:42:51.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0cf6d9d640 1 -- 172.21.15.163:0/3844942417 learned_addr learned my addr 172.21.15.163:0/3844942417 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:42:51.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0ceffff640 1 --2- 172.21.15.163:0/3844942417 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0cf81051c0 0x7f0cf81a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:42:51.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0cf759e640 1 --2- 172.21.15.163:0/3844942417 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cf8107de0 0x7f0cf81a3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:42:51.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0cf6d9d640 1 -- 172.21.15.163:0/3844942417 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0cf81051c0 msgr2=0x7f0cf81a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:42:51.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0cf6d9d640 1 --2- 172.21.15.163:0/3844942417 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0cf81051c0 0x7f0cf81a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:51.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0cf6d9d640 1 -- 172.21.15.163:0/3844942417 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cf8107de0 msgr2=0x7f0cf81a3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:42:51.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0cf6d9d640 1 --2- 172.21.15.163:0/3844942417 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cf8107de0 0x7f0cf81a3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:42:51.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0cf6d9d640 1 -- 172.21.15.163:0/3844942417 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cf81a3680 con 0x7f0cf8102760 2024-07-26T06:42:51.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0cf759e640 1 --2- 172.21.15.163:0/3844942417 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cf8107de0 0x7f0cf81a3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:42:51.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0cf6d9d640 1 --2- 172.21.15.163:0/3844942417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0cf8102760 0x7f0cf8101b60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0ce800d770 tx=0x7f0ce800dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:42:51.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0cf4d99640 1 -- 172.21.15.163:0/3844942417 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8017070 con 0x7f0cf8102760 2024-07-26T06:42:51.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0cfd008640 1 -- 172.21.15.163:0/3844942417 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cf81a3970 con 0x7f0cf8102760 2024-07-26T06:42:51.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0cf4d99640 1 -- 172.21.15.163:0/3844942417 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ce80054d0 con 0x7f0cf8102760 2024-07-26T06:42:51.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0cf4d99640 1 -- 172.21.15.163:0/3844942417 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8004c30 con 0x7f0cf8102760 2024-07-26T06:42:51.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.360+0000 7f0cfd008640 1 -- 172.21.15.163:0/3844942417 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cf81a3e60 con 0x7f0cf8102760 2024-07-26T06:42:51.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.362+0000 7f0cfd008640 1 -- 172.21.15.163:0/3844942417 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cf80fecc0 con 0x7f0cf8102760 2024-07-26T06:42:51.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.363+0000 7f0cf4d99640 1 -- 172.21.15.163:0/3844942417 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f0ce8004dd0 con 0x7f0cf8102760 2024-07-26T06:42:51.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.364+0000 7f0cf4d99640 1 --2- 172.21.15.163:0/3844942417 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0ce40b3bd0 0x7f0ce40b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:42:51.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.364+0000 7f0cf4d99640 1 -- 172.21.15.163:0/3844942417 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3669+0+0 (secure 0 0 0) 0x7f0ce80e2890 con 0x7f0cf8102760 2024-07-26T06:42:51.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.364+0000 7f0ceffff640 1 --2- 172.21.15.163:0/3844942417 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0ce40b3bd0 0x7f0ce40b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:42:51.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.364+0000 7f0ceffff640 1 --2- 172.21.15.163:0/3844942417 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0ce40b3bd0 0x7f0ce40b60e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0000c00 tx=0x7f0ce0031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:42:51.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.367+0000 7f0cf4d99640 1 -- 172.21.15.163:0/3844942417 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0ce80ab100 con 0x7f0cf8102760 2024-07-26T06:42:51.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:50 smithi042 ceph-mon[32775]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:42:51.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:50 smithi042 ceph-mon[32775]: Detected new or changed devices on smithi042 2024-07-26T06:42:51.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:50 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 8546M 2024-07-26T06:42:51.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:50 smithi042 ceph-mon[32775]: osdmap e27: 4 total, 4 up, 4 in 2024-07-26T06:42:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:50 smithi164 ceph-mon[36970]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:42:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:50 smithi164 ceph-mon[36970]: Detected new or changed devices on smithi042 2024-07-26T06:42:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:50 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 8546M 2024-07-26T06:42:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:50 smithi164 ceph-mon[36970]: osdmap e27: 4 total, 4 up, 4 in 2024-07-26T06:42:51.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:42:51.550+0000 7f0cfd008640 1 -- 172.21.15.163:0/3844942417 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0cf8063230 con 0x7f0ce40b3bd0 2024-07-26T06:42:52.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:42:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:42:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:52.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:42:52.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:42:52.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:42:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:42:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:42:53.135 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:53 smithi163 ceph-mon[36873]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:42:53.135 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:53 smithi163 ceph-mon[36873]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:42:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:53 smithi042 ceph-mon[32775]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:42:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:53 smithi042 ceph-mon[32775]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:42:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:53 smithi164 ceph-mon[36970]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:42:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:53 smithi164 ceph-mon[36970]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:42:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:54 smithi163 ceph-mon[36873]: from='client.? 172.21.15.163:0/1176410973' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fabf6b02-fe34-4560-85b5-01bf37fc9be4"}]: dispatch 2024-07-26T06:42:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:54 smithi163 ceph-mon[36873]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fabf6b02-fe34-4560-85b5-01bf37fc9be4"}]: dispatch 2024-07-26T06:42:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:54 smithi163 ceph-mon[36873]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fabf6b02-fe34-4560-85b5-01bf37fc9be4"}]': finished 2024-07-26T06:42:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:54 smithi163 ceph-mon[36873]: osdmap e28: 5 total, 4 up, 5 in 2024-07-26T06:42:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:42:54.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:54 smithi042 ceph-mon[32775]: from='client.? 172.21.15.163:0/1176410973' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fabf6b02-fe34-4560-85b5-01bf37fc9be4"}]: dispatch 2024-07-26T06:42:54.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:54 smithi042 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fabf6b02-fe34-4560-85b5-01bf37fc9be4"}]: dispatch 2024-07-26T06:42:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:54 smithi042 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fabf6b02-fe34-4560-85b5-01bf37fc9be4"}]': finished 2024-07-26T06:42:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:54 smithi042 ceph-mon[32775]: osdmap e28: 5 total, 4 up, 5 in 2024-07-26T06:42:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:42:54.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:54 smithi164 ceph-mon[36970]: from='client.? 172.21.15.163:0/1176410973' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fabf6b02-fe34-4560-85b5-01bf37fc9be4"}]: dispatch 2024-07-26T06:42:54.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:54 smithi164 ceph-mon[36970]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fabf6b02-fe34-4560-85b5-01bf37fc9be4"}]: dispatch 2024-07-26T06:42:54.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:54 smithi164 ceph-mon[36970]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fabf6b02-fe34-4560-85b5-01bf37fc9be4"}]': finished 2024-07-26T06:42:54.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:54 smithi164 ceph-mon[36970]: osdmap e28: 5 total, 4 up, 5 in 2024-07-26T06:42:54.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:42:55.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:55 smithi042 ceph-mon[32775]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 357 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-07-26T06:42:55.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:55 smithi042 ceph-mon[32775]: from='client.? 172.21.15.163:0/128617519' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:42:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:55 smithi164 ceph-mon[36970]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 357 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-07-26T06:42:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:55 smithi164 ceph-mon[36970]: from='client.? 172.21.15.163:0/128617519' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:42:55.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:55 smithi163 ceph-mon[36873]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 357 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-07-26T06:42:55.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:55 smithi163 ceph-mon[36873]: from='client.? 172.21.15.163:0/128617519' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:42:57.272 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:57 smithi163 ceph-mon[36873]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-07-26T06:42:57.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:57 smithi042 ceph-mon[32775]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-07-26T06:42:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:57 smithi164 ceph-mon[36970]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-07-26T06:42:59.274 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:42:59 smithi164 ceph-mon[36970]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T06:42:59.288 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:42:59 smithi163 ceph-mon[36873]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T06:42:59.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:42:59 smithi042 ceph-mon[32775]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T06:43:01.098 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:01 smithi163 ceph-mon[36873]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-07-26T06:43:01.098 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T06:43:01.098 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:01.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:01 smithi042 ceph-mon[32775]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-07-26T06:43:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T06:43:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:01 smithi164 ceph-mon[36970]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-07-26T06:43:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T06:43:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:02.116 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:02 smithi163 ceph-mon[36873]: Deploying daemon osd.4 on smithi163 2024-07-26T06:43:02.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:02 smithi042 ceph-mon[32775]: Deploying daemon osd.4 on smithi163 2024-07-26T06:43:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:02 smithi164 ceph-mon[36970]: Deploying daemon osd.4 on smithi163 2024-07-26T06:43:03.223 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:03 smithi163 ceph-mon[36873]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:43:03.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:03 smithi042 ceph-mon[32775]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:43:03.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:03 smithi164 ceph-mon[36970]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:43:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:05 smithi163 ceph-mon[36873]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:43:05.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:05 smithi042 ceph-mon[32775]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:43:05.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:05 smithi164 ceph-mon[36970]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:43:06.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:43:06.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:06.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:06.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:43:06.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:43:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:07.202 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:07 smithi163 ceph-mon[36873]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:43:07.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:07 smithi042 ceph-mon[32775]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:43:07.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:07 smithi164 ceph-mon[36970]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:43:08.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:08 smithi163 ceph-mon[36873]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:43:08.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:08 smithi042 ceph-mon[32775]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:43:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:08 smithi164 ceph-mon[36970]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:43:09.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:43:09.059+0000 7f0cf4d99640 1 -- 172.21.15.163:0/3844942417 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0cf8063230 con 0x7f0ce40b3bd0 2024-07-26T06:43:09.061 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 4 on host 'smithi163' 2024-07-26T06:43:09.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:43:09.061+0000 7f0cfd008640 1 -- 172.21.15.163:0/3844942417 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0ce40b3bd0 msgr2=0x7f0ce40b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:43:09.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:43:09.061+0000 7f0cfd008640 1 --2- 172.21.15.163:0/3844942417 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0ce40b3bd0 0x7f0ce40b60e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0000c00 tx=0x7f0ce0031400 comp rx=0 tx=0).stop 2024-07-26T06:43:09.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:43:09.061+0000 7f0cfd008640 1 -- 172.21.15.163:0/3844942417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0cf8102760 msgr2=0x7f0cf8101b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:43:09.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:43:09.061+0000 7f0cfd008640 1 --2- 172.21.15.163:0/3844942417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0cf8102760 0x7f0cf8101b60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0ce800d770 tx=0x7f0ce800dc30 comp rx=0 tx=0).stop 2024-07-26T06:43:09.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:43:09.061+0000 7f0cfd008640 1 -- 172.21.15.163:0/3844942417 shutdown_connections 2024-07-26T06:43:09.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:43:09.061+0000 7f0cfd008640 1 --2- 172.21.15.163:0/3844942417 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0ce40b3bd0 0x7f0ce40b60e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:09.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:43:09.061+0000 7f0cfd008640 1 --2- 172.21.15.163:0/3844942417 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cf8107de0 0x7f0cf81a3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:09.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:43:09.061+0000 7f0cfd008640 1 --2- 172.21.15.163:0/3844942417 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0cf81051c0 0x7f0cf81a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:09.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:43:09.061+0000 7f0cfd008640 1 --2- 172.21.15.163:0/3844942417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0cf8102760 0x7f0cf8101b60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:09.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:43:09.061+0000 7f0cfd008640 1 -- 172.21.15.163:0/3844942417 >> 172.21.15.163:0/3844942417 conn(0x7f0cf80fd1a0 msgr2=0x7f0cf8103530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:43:09.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:43:09.061+0000 7f0cfd008640 1 -- 172.21.15.163:0/3844942417 shutdown_connections 2024-07-26T06:43:09.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:43:09.062+0000 7f0cfd008640 1 -- 172.21.15.163:0/3844942417 wait complete. 2024-07-26T06:43:09.364 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:09.364 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:09.364 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:09.364 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:43:09.364 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:09.364 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:43:09.364 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:09.364 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:43:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:43:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:09.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:43:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:43:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:09.785 DEBUG:teuthology.orchestra.run.smithi163:osd.4> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@osd.4.service 2024-07-26T06:43:09.787 INFO:tasks.cephadm:Deploying osd.5 on smithi164 with /dev/vg_nvme/lv_3... 2024-07-26T06:43:09.787 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T06:43:10.046 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T06:43:10.352 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:10 smithi163 ceph-mon[36873]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:43:10.353 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:10 smithi163 ceph-mon[36873]: from='osd.4 [v2:172.21.15.163:6808/431934374,v1:172.21.15.163:6809/431934374]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T06:43:10.353 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:10 smithi163 ceph-mon[36873]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T06:43:10.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:10 smithi164 ceph-mon[36970]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:43:10.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:10 smithi164 ceph-mon[36970]: from='osd.4 [v2:172.21.15.163:6808/431934374,v1:172.21.15.163:6809/431934374]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T06:43:10.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:10 smithi164 ceph-mon[36970]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T06:43:10.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:10 smithi042 ceph-mon[32775]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:43:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:10 smithi042 ceph-mon[32775]: from='osd.4 [v2:172.21.15.163:6808/431934374,v1:172.21.15.163:6809/431934374]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T06:43:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:10 smithi042 ceph-mon[32775]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T06:43:11.249 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:11 smithi164 ceph-mon[36970]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T06:43:11.250 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:11 smithi164 ceph-mon[36970]: osdmap e29: 5 total, 4 up, 5 in 2024-07-26T06:43:11.250 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:43:11.250 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:11 smithi164 ceph-mon[36970]: from='osd.4 [v2:172.21.15.163:6808/431934374,v1:172.21.15.163:6809/431934374]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:43:11.250 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:11 smithi164 ceph-mon[36970]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:43:11.339 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:11 smithi163 ceph-mon[36873]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T06:43:11.339 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:11 smithi163 ceph-mon[36873]: osdmap e29: 5 total, 4 up, 5 in 2024-07-26T06:43:11.339 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:43:11.339 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:11 smithi163 ceph-mon[36873]: from='osd.4 [v2:172.21.15.163:6808/431934374,v1:172.21.15.163:6809/431934374]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:43:11.339 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:11 smithi163 ceph-mon[36873]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:43:11.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:11 smithi042 ceph-mon[32775]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T06:43:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:11 smithi042 ceph-mon[32775]: osdmap e29: 5 total, 4 up, 5 in 2024-07-26T06:43:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:43:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:11 smithi042 ceph-mon[32775]: from='osd.4 [v2:172.21.15.163:6808/431934374,v1:172.21.15.163:6809/431934374]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:43:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:11 smithi042 ceph-mon[32775]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:43:11.901 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T06:43:11.991 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T06:43:12.004 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2024-07-26T06:43:12.004 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2024-07-26T06:43:12.004 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114818 s, 913 MB/s 2024-07-26T06:43:12.004 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2024-07-26T06:43:12.007 INFO:journalctl@ceph.osd.4.smithi163.stdout:Jul 26 06:43:11 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-4[48351]: 2024-07-26T06:43:11.669+0000 7efe34738640 -1 osd.4 0 waiting for initial osdmap 2024-07-26T06:43:12.007 INFO:journalctl@ceph.osd.4.smithi163.stdout:Jul 26 06:43:11 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-4[48351]: 2024-07-26T06:43:11.675+0000 7efe2bd19640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:43:12.342 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:12 smithi163 ceph-mon[36873]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-07-26T06:43:12.342 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:12 smithi163 ceph-mon[36873]: osdmap e30: 5 total, 4 up, 5 in 2024-07-26T06:43:12.343 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:43:12.343 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:43:12.343 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:12 smithi163 ceph-mon[36873]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:43:12.343 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:12 smithi163 ceph-mon[36873]: from='osd.4 ' entity='osd.4' 2024-07-26T06:43:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:12 smithi164 ceph-mon[36970]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-07-26T06:43:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:12 smithi164 ceph-mon[36970]: osdmap e30: 5 total, 4 up, 5 in 2024-07-26T06:43:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:43:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:43:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:12 smithi164 ceph-mon[36970]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:43:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:12 smithi164 ceph-mon[36970]: from='osd.4 ' entity='osd.4' 2024-07-26T06:43:12.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:12 smithi042 ceph-mon[32775]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-07-26T06:43:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:12 smithi042 ceph-mon[32775]: osdmap e30: 5 total, 4 up, 5 in 2024-07-26T06:43:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:43:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:43:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:12 smithi042 ceph-mon[32775]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-07-26T06:43:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:12 smithi042 ceph-mon[32775]: from='osd.4 ' entity='osd.4' 2024-07-26T06:43:12.654 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi164:vg_nvme/lv_3 2024-07-26T06:43:12.916 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T06:43:13.377 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:13 smithi164 ceph-mon[36970]: purged_snaps scrub starts 2024-07-26T06:43:13.377 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:13 smithi164 ceph-mon[36970]: purged_snaps scrub ok 2024-07-26T06:43:13.377 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:43:13.377 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:13 smithi164 ceph-mon[36970]: Detected new or changed devices on smithi163 2024-07-26T06:43:13.378 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:13.378 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:13 smithi164 ceph-mon[36970]: osd.4 [v2:172.21.15.163:6808/431934374,v1:172.21.15.163:6809/431934374] boot 2024-07-26T06:43:13.378 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:13 smithi164 ceph-mon[36970]: osdmap e31: 5 total, 5 up, 5 in 2024-07-26T06:43:13.378 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:43:13.378 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:13.378 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:43:13.378 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:43:13.378 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:13 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 8544M 2024-07-26T06:43:13.378 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:13.378 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:13.378 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:43:13.378 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:13 smithi163 ceph-mon[36873]: purged_snaps scrub starts 2024-07-26T06:43:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:13 smithi163 ceph-mon[36873]: purged_snaps scrub ok 2024-07-26T06:43:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:43:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:13 smithi163 ceph-mon[36873]: Detected new or changed devices on smithi163 2024-07-26T06:43:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:13 smithi163 ceph-mon[36873]: osd.4 [v2:172.21.15.163:6808/431934374,v1:172.21.15.163:6809/431934374] boot 2024-07-26T06:43:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:13 smithi163 ceph-mon[36873]: osdmap e31: 5 total, 5 up, 5 in 2024-07-26T06:43:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:43:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:43:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:43:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:13 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 8544M 2024-07-26T06:43:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:43:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:13 smithi042 ceph-mon[32775]: purged_snaps scrub starts 2024-07-26T06:43:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:13 smithi042 ceph-mon[32775]: purged_snaps scrub ok 2024-07-26T06:43:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:43:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:13 smithi042 ceph-mon[32775]: Detected new or changed devices on smithi163 2024-07-26T06:43:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:13 smithi042 ceph-mon[32775]: osd.4 [v2:172.21.15.163:6808/431934374,v1:172.21.15.163:6809/431934374] boot 2024-07-26T06:43:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:13 smithi042 ceph-mon[32775]: osdmap e31: 5 total, 5 up, 5 in 2024-07-26T06:43:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T06:43:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:43:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:43:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:13 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 8544M 2024-07-26T06:43:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:43:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:14.325 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:14 smithi164 ceph-mon[36970]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:14.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:14 smithi163 ceph-mon[36873]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:14.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.516+0000 7f6c66140640 1 -- 172.21.15.164:0/2135100179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c6006b3b0 msgr2=0x7f6c60105dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:43:14.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.516+0000 7f6c66140640 1 --2- 172.21.15.164:0/2135100179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c6006b3b0 0x7f6c60105dc0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6c50009900 tx=0x7f6c5002a410 comp rx=0 tx=0).stop 2024-07-26T06:43:14.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.516+0000 7f6c66140640 1 -- 172.21.15.164:0/2135100179 shutdown_connections 2024-07-26T06:43:14.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.516+0000 7f6c66140640 1 --2- 172.21.15.164:0/2135100179 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c60108ea0 0x7f6c6010b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:14.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.516+0000 7f6c66140640 1 --2- 172.21.15.164:0/2135100179 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c60106300 0x7f6c60108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:14.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.516+0000 7f6c66140640 1 --2- 172.21.15.164:0/2135100179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c6006b3b0 0x7f6c60105dc0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:14.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.516+0000 7f6c66140640 1 -- 172.21.15.164:0/2135100179 >> 172.21.15.164:0/2135100179 conn(0x7f6c600fd9e0 msgr2=0x7f6c600ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:43:14.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.516+0000 7f6c66140640 1 -- 172.21.15.164:0/2135100179 shutdown_connections 2024-07-26T06:43:14.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.516+0000 7f6c66140640 1 -- 172.21.15.164:0/2135100179 wait complete. 2024-07-26T06:43:14.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.517+0000 7f6c66140640 1 Processor -- start 2024-07-26T06:43:14.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.517+0000 7f6c66140640 1 -- start start 2024-07-26T06:43:14.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.517+0000 7f6c66140640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c6006b3b0 0x7f6c601033f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:43:14.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.517+0000 7f6c66140640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c60106300 0x7f6c60101cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:43:14.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.517+0000 7f6c66140640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c60108ea0 0x7f6c60102200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:43:14.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.517+0000 7f6c66140640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6010edc0 con 0x7f6c60106300 2024-07-26T06:43:14.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.517+0000 7f6c66140640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6010ec40 con 0x7f6c60108ea0 2024-07-26T06:43:14.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.517+0000 7f6c66140640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6010ef40 con 0x7f6c6006b3b0 2024-07-26T06:43:14.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.517+0000 7f6c5f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c6006b3b0 0x7f6c601033f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:43:14.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.517+0000 7f6c5f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c6006b3b0 0x7f6c601033f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:49400/0 (socket says 172.21.15.164:49400) 2024-07-26T06:43:14.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.517+0000 7f6c5f7fe640 1 -- 172.21.15.164:0/2873093220 learned_addr learned my addr 172.21.15.164:0/2873093220 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-07-26T06:43:14.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.517+0000 7f6c56ffd640 1 --2- 172.21.15.164:0/2873093220 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c60106300 0x7f6c60101cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:43:14.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.517+0000 7f6c5ffff640 1 --2- 172.21.15.164:0/2873093220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c60108ea0 0x7f6c60102200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:43:14.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.518+0000 7f6c5f7fe640 1 -- 172.21.15.164:0/2873093220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c60108ea0 msgr2=0x7f6c60102200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:43:14.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.518+0000 7f6c5f7fe640 1 --2- 172.21.15.164:0/2873093220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c60108ea0 0x7f6c60102200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:14.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.518+0000 7f6c5f7fe640 1 -- 172.21.15.164:0/2873093220 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c60106300 msgr2=0x7f6c60101cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:43:14.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.518+0000 7f6c5f7fe640 1 --2- 172.21.15.164:0/2873093220 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c60106300 0x7f6c60101cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:14.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.518+0000 7f6c5f7fe640 1 -- 172.21.15.164:0/2873093220 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c60102900 con 0x7f6c6006b3b0 2024-07-26T06:43:14.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.518+0000 7f6c5f7fe640 1 --2- 172.21.15.164:0/2873093220 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c6006b3b0 0x7f6c601033f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6c50006010 tx=0x7f6c50004340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:43:14.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.518+0000 7f6c5d7fa640 1 -- 172.21.15.164:0/2873093220 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5003f030 con 0x7f6c6006b3b0 2024-07-26T06:43:14.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.518+0000 7f6c5d7fa640 1 -- 172.21.15.164:0/2873093220 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c50003cb0 con 0x7f6c6006b3b0 2024-07-26T06:43:14.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.518+0000 7f6c66140640 1 -- 172.21.15.164:0/2873093220 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c601a3a10 con 0x7f6c6006b3b0 2024-07-26T06:43:14.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.518+0000 7f6c5d7fa640 1 -- 172.21.15.164:0/2873093220 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c50035070 con 0x7f6c6006b3b0 2024-07-26T06:43:14.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.518+0000 7f6c66140640 1 -- 172.21.15.164:0/2873093220 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c601a3db0 con 0x7f6c6006b3b0 2024-07-26T06:43:14.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.519+0000 7f6c66140640 1 -- 172.21.15.164:0/2873093220 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c600ff500 con 0x7f6c6006b3b0 2024-07-26T06:43:14.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.520+0000 7f6c5d7fa640 1 -- 172.21.15.164:0/2873093220 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f6c50042030 con 0x7f6c6006b3b0 2024-07-26T06:43:14.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.521+0000 7f6c5d7fa640 1 --2- 172.21.15.164:0/2873093220 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6c4c0b3bd0 0x7f6c4c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:43:14.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.521+0000 7f6c5d7fa640 1 -- 172.21.15.164:0/2873093220 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4101+0+0 (secure 0 0 0) 0x7f6c50103810 con 0x7f6c6006b3b0 2024-07-26T06:43:14.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.521+0000 7f6c56ffd640 1 --2- 172.21.15.164:0/2873093220 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6c4c0b3bd0 0x7f6c4c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:43:14.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.522+0000 7f6c56ffd640 1 --2- 172.21.15.164:0/2873093220 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6c4c0b3bd0 0x7f6c4c0b60e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6c480058c0 tx=0x7f6c4800a2d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:43:14.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.524+0000 7f6c5d7fa640 1 -- 172.21.15.164:0/2873093220 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6c50003e60 con 0x7f6c6006b3b0 2024-07-26T06:43:14.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:14 smithi042 ceph-mon[32775]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:14.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:14.706+0000 7f6c66140640 1 -- 172.21.15.164:0/2873093220 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f6c60063230 con 0x7f6c4c0b3bd0 2024-07-26T06:43:15.445 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:15 smithi164 ceph-mon[36970]: osdmap e32: 5 total, 5 up, 5 in 2024-07-26T06:43:15.445 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:15 smithi164 ceph-mon[36970]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:43:15.445 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:43:15.445 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:43:15.445 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:15 smithi163 ceph-mon[36873]: osdmap e32: 5 total, 5 up, 5 in 2024-07-26T06:43:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:15 smithi163 ceph-mon[36873]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:43:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:43:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:43:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:15.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:15 smithi042 ceph-mon[32775]: osdmap e32: 5 total, 5 up, 5 in 2024-07-26T06:43:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:15 smithi042 ceph-mon[32775]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:43:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:43:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:43:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:16.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:16 smithi164 ceph-mon[36970]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:16.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:16 smithi163 ceph-mon[36873]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:16 smithi042 ceph-mon[32775]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:17 smithi163 ceph-mon[36873]: from='client.? 172.21.15.164:0/838233407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e226fb4f-15b2-40df-af34-900040ba93ad"}]: dispatch 2024-07-26T06:43:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:17 smithi163 ceph-mon[36873]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e226fb4f-15b2-40df-af34-900040ba93ad"}]: dispatch 2024-07-26T06:43:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:17 smithi163 ceph-mon[36873]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e226fb4f-15b2-40df-af34-900040ba93ad"}]': finished 2024-07-26T06:43:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:17 smithi163 ceph-mon[36873]: osdmap e33: 6 total, 5 up, 6 in 2024-07-26T06:43:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:17.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:17 smithi042 ceph-mon[32775]: from='client.? 172.21.15.164:0/838233407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e226fb4f-15b2-40df-af34-900040ba93ad"}]: dispatch 2024-07-26T06:43:17.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:17 smithi042 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e226fb4f-15b2-40df-af34-900040ba93ad"}]: dispatch 2024-07-26T06:43:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:17 smithi042 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e226fb4f-15b2-40df-af34-900040ba93ad"}]': finished 2024-07-26T06:43:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:17 smithi042 ceph-mon[32775]: osdmap e33: 6 total, 5 up, 6 in 2024-07-26T06:43:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:17.695 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:17 smithi164 ceph-mon[36970]: from='client.? 172.21.15.164:0/838233407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e226fb4f-15b2-40df-af34-900040ba93ad"}]: dispatch 2024-07-26T06:43:17.696 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:17 smithi164 ceph-mon[36970]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e226fb4f-15b2-40df-af34-900040ba93ad"}]: dispatch 2024-07-26T06:43:17.696 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:17 smithi164 ceph-mon[36970]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e226fb4f-15b2-40df-af34-900040ba93ad"}]': finished 2024-07-26T06:43:17.696 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:17 smithi164 ceph-mon[36970]: osdmap e33: 6 total, 5 up, 6 in 2024-07-26T06:43:17.696 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:18.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:18 smithi163 ceph-mon[36873]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:18 smithi163 ceph-mon[36873]: from='client.? 172.21.15.164:0/1717622521' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:43:18.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:18 smithi042 ceph-mon[32775]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:18 smithi042 ceph-mon[32775]: from='client.? 172.21.15.164:0/1717622521' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:43:18.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:18 smithi164 ceph-mon[36970]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:18.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:18 smithi164 ceph-mon[36970]: from='client.? 172.21.15.164:0/1717622521' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:43:20.626 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:20 smithi164 ceph-mon[36970]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:20.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:20 smithi042 ceph-mon[32775]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:20.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:20 smithi163 ceph-mon[36873]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:22.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:22 smithi042 ceph-mon[32775]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:22 smithi164 ceph-mon[36970]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:22.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:22 smithi163 ceph-mon[36873]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:24.583 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:24 smithi164 ceph-mon[36970]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:24.583 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T06:43:24.584 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:24.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:24 smithi042 ceph-mon[32775]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:24.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T06:43:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:24 smithi163 ceph-mon[36873]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T06:43:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:25.555 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:25 smithi164 ceph-mon[36970]: Deploying daemon osd.5 on smithi164 2024-07-26T06:43:25.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:25 smithi042 ceph-mon[32775]: Deploying daemon osd.5 on smithi164 2024-07-26T06:43:25.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:25 smithi163 ceph-mon[36873]: Deploying daemon osd.5 on smithi164 2024-07-26T06:43:26.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:26 smithi042 ceph-mon[32775]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:26 smithi164 ceph-mon[36970]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:26.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:26 smithi163 ceph-mon[36873]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:28.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:28 smithi042 ceph-mon[32775]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:28.661 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:28 smithi164 ceph-mon[36970]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:28.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:28 smithi163 ceph-mon[36873]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:30.616 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:30 smithi164 ceph-mon[36970]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:30.616 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:43:30.616 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:30.616 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:30.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:30 smithi042 ceph-mon[32775]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:30.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:43:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:30.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:30 smithi163 ceph-mon[36873]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:43:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:32.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:32 smithi042 ceph-mon[32775]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:32.663 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:32 smithi164 ceph-mon[36970]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:32 smithi163 ceph-mon[36873]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:32.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:32.977+0000 7f6c5d7fa640 1 -- 172.21.15.164:0/2873093220 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6c60063230 con 0x7f6c4c0b3bd0 2024-07-26T06:43:32.978 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 5 on host 'smithi164' 2024-07-26T06:43:32.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:32.979+0000 7f6c66140640 1 -- 172.21.15.164:0/2873093220 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6c4c0b3bd0 msgr2=0x7f6c4c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:43:32.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:32.979+0000 7f6c66140640 1 --2- 172.21.15.164:0/2873093220 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6c4c0b3bd0 0x7f6c4c0b60e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6c480058c0 tx=0x7f6c4800a2d0 comp rx=0 tx=0).stop 2024-07-26T06:43:32.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:32.979+0000 7f6c66140640 1 -- 172.21.15.164:0/2873093220 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c6006b3b0 msgr2=0x7f6c601033f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:43:32.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:32.979+0000 7f6c66140640 1 --2- 172.21.15.164:0/2873093220 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c6006b3b0 0x7f6c601033f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6c50006010 tx=0x7f6c50004340 comp rx=0 tx=0).stop 2024-07-26T06:43:32.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:32.980+0000 7f6c66140640 1 -- 172.21.15.164:0/2873093220 shutdown_connections 2024-07-26T06:43:32.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:32.980+0000 7f6c66140640 1 --2- 172.21.15.164:0/2873093220 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6c4c0b3bd0 0x7f6c4c0b60e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:32.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:32.980+0000 7f6c66140640 1 --2- 172.21.15.164:0/2873093220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c60108ea0 0x7f6c60102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:32.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:32.980+0000 7f6c66140640 1 --2- 172.21.15.164:0/2873093220 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c60106300 0x7f6c60101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:32.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:32.980+0000 7f6c66140640 1 --2- 172.21.15.164:0/2873093220 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6c6006b3b0 0x7f6c601033f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:32.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:32.980+0000 7f6c66140640 1 -- 172.21.15.164:0/2873093220 >> 172.21.15.164:0/2873093220 conn(0x7f6c600fd9e0 msgr2=0x7f6c60104810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:43:32.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:32.980+0000 7f6c66140640 1 -- 172.21.15.164:0/2873093220 shutdown_connections 2024-07-26T06:43:32.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:43:32.980+0000 7f6c66140640 1 -- 172.21.15.164:0/2873093220 wait complete. 2024-07-26T06:43:33.663 DEBUG:teuthology.orchestra.run.smithi164:osd.5> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@osd.5.service 2024-07-26T06:43:33.665 INFO:tasks.cephadm:Deploying osd.6 on smithi042 with /dev/vg_nvme/lv_2... 2024-07-26T06:43:33.665 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T06:43:33.947 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:43:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:33 smithi164 ceph-mon[36970]: from='osd.5 [v2:172.21.15.164:6808/261549362,v1:172.21.15.164:6809/261549362]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T06:43:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:33 smithi164 ceph-mon[36970]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T06:43:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:33 smithi163 ceph-mon[36873]: from='osd.5 [v2:172.21.15.164:6808/261549362,v1:172.21.15.164:6809/261549362]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T06:43:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:33 smithi163 ceph-mon[36873]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T06:43:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:34.271 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:34.271 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:34.271 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:33 smithi042 ceph-mon[32775]: from='osd.5 [v2:172.21.15.164:6808/261549362,v1:172.21.15.164:6809/261549362]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T06:43:34.271 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:33 smithi042 ceph-mon[32775]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T06:43:34.271 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:34.271 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:35.183 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:34 smithi164 ceph-mon[36970]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:35.184 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:34 smithi164 ceph-mon[36970]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T06:43:35.184 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:34 smithi164 ceph-mon[36970]: osdmap e34: 6 total, 5 up, 6 in 2024-07-26T06:43:35.184 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:35.184 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:34 smithi164 ceph-mon[36970]: from='osd.5 [v2:172.21.15.164:6808/261549362,v1:172.21.15.164:6809/261549362]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:43:35.184 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:34 smithi164 ceph-mon[36970]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:43:35.184 INFO:journalctl@ceph.osd.5.smithi164.stdout:Jul 26 06:43:34 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-5[48670]: 2024-07-26T06:43:34.993+0000 7ffab614b640 -1 osd.5 0 waiting for initial osdmap 2024-07-26T06:43:35.184 INFO:journalctl@ceph.osd.5.smithi164.stdout:Jul 26 06:43:35 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-5[48670]: 2024-07-26T06:43:35.000+0000 7ffab1f49640 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:43:35.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:34 smithi163 ceph-mon[36873]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:34 smithi163 ceph-mon[36873]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T06:43:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:34 smithi163 ceph-mon[36873]: osdmap e34: 6 total, 5 up, 6 in 2024-07-26T06:43:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:34 smithi163 ceph-mon[36873]: from='osd.5 [v2:172.21.15.164:6808/261549362,v1:172.21.15.164:6809/261549362]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:43:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:34 smithi163 ceph-mon[36873]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:43:35.270 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:34 smithi042 ceph-mon[32775]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:35.270 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:34 smithi042 ceph-mon[32775]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T06:43:35.270 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:34 smithi042 ceph-mon[32775]: osdmap e34: 6 total, 5 up, 6 in 2024-07-26T06:43:35.270 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:35.270 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:34 smithi042 ceph-mon[32775]: from='osd.5 [v2:172.21.15.164:6808/261549362,v1:172.21.15.164:6809/261549362]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:43:35.270 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:34 smithi042 ceph-mon[32775]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:43:35.920 INFO:teuthology.orchestra.run.smithi042.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T06:43:36.011 INFO:teuthology.orchestra.run.smithi042.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T06:43:36.021 INFO:teuthology.orchestra.run.smithi042.stderr: stderr: 10+0 records in 2024-07-26T06:43:36.021 INFO:teuthology.orchestra.run.smithi042.stderr:10+0 records out 2024-07-26T06:43:36.021 INFO:teuthology.orchestra.run.smithi042.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00855015 s, 1.2 GB/s 2024-07-26T06:43:36.021 INFO:teuthology.orchestra.run.smithi042.stderr:--> Zapping successful for: 2024-07-26T06:43:36.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:35 smithi163 ceph-mon[36873]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-07-26T06:43:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:35 smithi163 ceph-mon[36873]: osdmap e35: 6 total, 5 up, 6 in 2024-07-26T06:43:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:36.324 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:35 smithi042 ceph-mon[32775]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-07-26T06:43:36.324 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:35 smithi042 ceph-mon[32775]: osdmap e35: 6 total, 5 up, 6 in 2024-07-26T06:43:36.324 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:36.324 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:36.465 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:35 smithi164 ceph-mon[36970]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-07-26T06:43:36.465 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:35 smithi164 ceph-mon[36970]: osdmap e35: 6 total, 5 up, 6 in 2024-07-26T06:43:36.465 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:36.465 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:36.759 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi042:vg_nvme/lv_2 2024-07-26T06:43:37.049 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:43:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:37 smithi164 ceph-mon[36970]: purged_snaps scrub starts 2024-07-26T06:43:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:37 smithi164 ceph-mon[36970]: purged_snaps scrub ok 2024-07-26T06:43:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:37 smithi164 ceph-mon[36970]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:37 smithi164 ceph-mon[36970]: osd.5 [v2:172.21.15.164:6808/261549362,v1:172.21.15.164:6809/261549362] boot 2024-07-26T06:43:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:37 smithi164 ceph-mon[36970]: osdmap e36: 6 total, 6 up, 6 in 2024-07-26T06:43:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:43:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:43:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:43:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:37 smithi163 ceph-mon[36873]: purged_snaps scrub starts 2024-07-26T06:43:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:37 smithi163 ceph-mon[36873]: purged_snaps scrub ok 2024-07-26T06:43:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:37 smithi163 ceph-mon[36873]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:37 smithi163 ceph-mon[36873]: osd.5 [v2:172.21.15.164:6808/261549362,v1:172.21.15.164:6809/261549362] boot 2024-07-26T06:43:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:37 smithi163 ceph-mon[36873]: osdmap e36: 6 total, 6 up, 6 in 2024-07-26T06:43:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:43:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:43:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:43:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:37.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:37 smithi042 ceph-mon[32775]: purged_snaps scrub starts 2024-07-26T06:43:37.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:37 smithi042 ceph-mon[32775]: purged_snaps scrub ok 2024-07-26T06:43:37.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:37 smithi042 ceph-mon[32775]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-07-26T06:43:37.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:37.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:37 smithi042 ceph-mon[32775]: osd.5 [v2:172.21.15.164:6808/261549362,v1:172.21.15.164:6809/261549362] boot 2024-07-26T06:43:37.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:37 smithi042 ceph-mon[32775]: osdmap e36: 6 total, 6 up, 6 in 2024-07-26T06:43:37.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T06:43:37.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:37.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:37.278 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:43:37.279 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:43:37.279 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:37.279 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:37.279 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:43:37.279 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:38.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:38 smithi042 ceph-mon[32775]: Detected new or changed devices on smithi164 2024-07-26T06:43:38.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:38 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 8544M 2024-07-26T06:43:38.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:38 smithi042 ceph-mon[32775]: osdmap e37: 6 total, 6 up, 6 in 2024-07-26T06:43:38.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:43:38.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:38.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:43:38.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:38 smithi164 ceph-mon[36970]: Detected new or changed devices on smithi164 2024-07-26T06:43:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:38 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 8544M 2024-07-26T06:43:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:38 smithi164 ceph-mon[36970]: osdmap e37: 6 total, 6 up, 6 in 2024-07-26T06:43:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:43:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:43:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:38.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:38 smithi163 ceph-mon[36873]: Detected new or changed devices on smithi164 2024-07-26T06:43:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:38 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 8544M 2024-07-26T06:43:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:38 smithi163 ceph-mon[36873]: osdmap e37: 6 total, 6 up, 6 in 2024-07-26T06:43:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:43:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:43:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:38.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.676+0000 7ff3d594f640 1 -- 172.21.15.42:0/607388885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3d0103110 msgr2=0x7ff3d0103520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:43:38.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.676+0000 7ff3d594f640 1 --2- 172.21.15.42:0/607388885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3d0103110 0x7ff3d0103520 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4009900 tx=0x7ff3c402a480 comp rx=0 tx=0).stop 2024-07-26T06:43:38.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.676+0000 7ff3d594f640 1 -- 172.21.15.42:0/607388885 shutdown_connections 2024-07-26T06:43:38.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.676+0000 7ff3d594f640 1 --2- 172.21.15.42:0/607388885 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3d0103ab0 0x7ff3d010a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:38.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.676+0000 7ff3d594f640 1 --2- 172.21.15.42:0/607388885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3d0103110 0x7ff3d0103520 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:38.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.676+0000 7ff3d594f640 1 --2- 172.21.15.42:0/607388885 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3d0102080 0x7ff3d01024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.676+0000 7ff3d594f640 1 -- 172.21.15.42:0/607388885 >> 172.21.15.42:0/607388885 conn(0x7ff3d00fda10 msgr2=0x7ff3d00ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:43:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.676+0000 7ff3d594f640 1 -- 172.21.15.42:0/607388885 shutdown_connections 2024-07-26T06:43:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.676+0000 7ff3d594f640 1 -- 172.21.15.42:0/607388885 wait complete. 2024-07-26T06:43:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3d594f640 1 Processor -- start 2024-07-26T06:43:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3d594f640 1 -- start start 2024-07-26T06:43:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3d594f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3d0102080 0x7ff3d006be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:43:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3d594f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3d0103110 0x7ff3d019e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:43:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3d594f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3d0103ab0 0x7ff3d01a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:43:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3d594f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d010dce0 con 0x7ff3d0103110 2024-07-26T06:43:38.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3d594f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d010db60 con 0x7ff3d0102080 2024-07-26T06:43:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3d594f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d010de60 con 0x7ff3d0103ab0 2024-07-26T06:43:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3ce7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3d0103110 0x7ff3d019e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:43:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3ce7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3d0103110 0x7ff3d019e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44322/0 (socket says 172.21.15.42:44322) 2024-07-26T06:43:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3ce7fc640 1 -- 172.21.15.42:0/842890903 learned_addr learned my addr 172.21.15.42:0/842890903 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:43:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3ceffd640 1 --2- 172.21.15.42:0/842890903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3d0102080 0x7ff3d006be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:43:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3cf7fe640 1 --2- 172.21.15.42:0/842890903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3d0103ab0 0x7ff3d01a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:43:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3ce7fc640 1 -- 172.21.15.42:0/842890903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3d0103ab0 msgr2=0x7ff3d01a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:43:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3ce7fc640 1 --2- 172.21.15.42:0/842890903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3d0103ab0 0x7ff3d01a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3ce7fc640 1 -- 172.21.15.42:0/842890903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3d0102080 msgr2=0x7ff3d006be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:43:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3ce7fc640 1 --2- 172.21.15.42:0/842890903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3d0102080 0x7ff3d006be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:38.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3ce7fc640 1 -- 172.21.15.42:0/842890903 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d01a14c0 con 0x7ff3d0103110 2024-07-26T06:43:38.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.677+0000 7ff3ceffd640 1 --2- 172.21.15.42:0/842890903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3d0102080 0x7ff3d006be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:43:38.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.678+0000 7ff3ce7fc640 1 --2- 172.21.15.42:0/842890903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3d0103110 0x7ff3d019e710 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4006010 tx=0x7ff3c4004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:43:38.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.678+0000 7ff3d494d640 1 -- 172.21.15.42:0/842890903 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c403f030 con 0x7ff3d0103110 2024-07-26T06:43:38.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.678+0000 7ff3d594f640 1 -- 172.21.15.42:0/842890903 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3d01a1750 con 0x7ff3d0103110 2024-07-26T06:43:38.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.678+0000 7ff3d494d640 1 -- 172.21.15.42:0/842890903 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3c4003cb0 con 0x7ff3d0103110 2024-07-26T06:43:38.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.678+0000 7ff3d494d640 1 -- 172.21.15.42:0/842890903 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4035070 con 0x7ff3d0103110 2024-07-26T06:43:38.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.678+0000 7ff3d594f640 1 -- 172.21.15.42:0/842890903 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3d01a1c80 con 0x7ff3d0103110 2024-07-26T06:43:38.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.679+0000 7ff3d594f640 1 -- 172.21.15.42:0/842890903 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3d0109260 con 0x7ff3d0103110 2024-07-26T06:43:38.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.681+0000 7ff3d494d640 1 -- 172.21.15.42:0/842890903 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7ff3c4042030 con 0x7ff3d0103110 2024-07-26T06:43:38.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.682+0000 7ff3d494d640 1 --2- 172.21.15.42:0/842890903 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff3b80b3bd0 0x7ff3b80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:43:38.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.682+0000 7ff3ceffd640 1 --2- 172.21.15.42:0/842890903 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff3b80b3bd0 0x7ff3b80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:43:38.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.682+0000 7ff3d494d640 1 -- 172.21.15.42:0/842890903 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 4533+0+0 (secure 0 0 0) 0x7ff3c4103b20 con 0x7ff3d0103110 2024-07-26T06:43:38.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.682+0000 7ff3ceffd640 1 --2- 172.21.15.42:0/842890903 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff3b80b3bd0 0x7ff3b80b60e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff3c00058c0 tx=0x7ff3c0005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:43:38.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.685+0000 7ff3d494d640 1 -- 172.21.15.42:0/842890903 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff3c4103e20 con 0x7ff3d0103110 2024-07-26T06:43:38.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:38.881+0000 7ff3d594f640 1 -- 172.21.15.42:0/842890903 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff3d0063230 con 0x7ff3b80b3bd0 2024-07-26T06:43:39.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:39 smithi042 ceph-mon[32775]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:43:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:39 smithi042 ceph-mon[32775]: osdmap e38: 6 total, 6 up, 6 in 2024-07-26T06:43:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:43:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:43:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:39.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:39 smithi164 ceph-mon[36970]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:43:39.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:39 smithi164 ceph-mon[36970]: osdmap e38: 6 total, 6 up, 6 in 2024-07-26T06:43:39.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:43:39.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:43:39.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:39.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:39 smithi163 ceph-mon[36873]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:43:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:39 smithi163 ceph-mon[36873]: osdmap e38: 6 total, 6 up, 6 in 2024-07-26T06:43:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:43:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:43:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:40.221 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:40 smithi042 ceph-mon[32775]: from='client.14481 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:43:40.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:40 smithi164 ceph-mon[36970]: from='client.14481 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:43:40.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:40 smithi163 ceph-mon[36873]: from='client.14481 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:43:41.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:41 smithi164 ceph-mon[36970]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:43:41.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:41 smithi042 ceph-mon[32775]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:43:41.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:41 smithi163 ceph-mon[36873]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:43:42.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:42 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2812119941' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "beb8ec5d-7ea5-4e4e-a46c-a84f7768cec0"}]: dispatch 2024-07-26T06:43:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:42 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2812119941' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "beb8ec5d-7ea5-4e4e-a46c-a84f7768cec0"}]': finished 2024-07-26T06:43:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:42 smithi042 ceph-mon[32775]: osdmap e39: 7 total, 6 up, 7 in 2024-07-26T06:43:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:43:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:42 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2956799268' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:43:42.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:42 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2812119941' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "beb8ec5d-7ea5-4e4e-a46c-a84f7768cec0"}]: dispatch 2024-07-26T06:43:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:42 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2812119941' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "beb8ec5d-7ea5-4e4e-a46c-a84f7768cec0"}]': finished 2024-07-26T06:43:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:42 smithi163 ceph-mon[36873]: osdmap e39: 7 total, 6 up, 7 in 2024-07-26T06:43:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:43:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:42 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2956799268' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:43:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:42 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2812119941' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "beb8ec5d-7ea5-4e4e-a46c-a84f7768cec0"}]: dispatch 2024-07-26T06:43:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:42 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2812119941' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "beb8ec5d-7ea5-4e4e-a46c-a84f7768cec0"}]': finished 2024-07-26T06:43:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:42 smithi164 ceph-mon[36970]: osdmap e39: 7 total, 6 up, 7 in 2024-07-26T06:43:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:43:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:42 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2956799268' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:43:43.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:43 smithi042 ceph-mon[32775]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:43:43.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:43 smithi164 ceph-mon[36970]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:43:43.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:43 smithi163 ceph-mon[36873]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:43:45.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:45 smithi042 ceph-mon[32775]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-07-26T06:43:45.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:45 smithi164 ceph-mon[36970]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-07-26T06:43:45.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:45 smithi163 ceph-mon[36873]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-07-26T06:43:47.097 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:47 smithi042 ceph-mon[32775]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T06:43:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:47 smithi164 ceph-mon[36970]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T06:43:47.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:47 smithi163 ceph-mon[36873]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T06:43:49.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:49 smithi042 ceph-mon[32775]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-07-26T06:43:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T06:43:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:49 smithi164 ceph-mon[36970]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-07-26T06:43:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T06:43:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:49.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:49 smithi163 ceph-mon[36873]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-07-26T06:43:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T06:43:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:43:50.261 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:50 smithi042 ceph-mon[32775]: Deploying daemon osd.6 on smithi042 2024-07-26T06:43:50.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:50 smithi164 ceph-mon[36970]: Deploying daemon osd.6 on smithi042 2024-07-26T06:43:50.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:50 smithi163 ceph-mon[36873]: Deploying daemon osd.6 on smithi042 2024-07-26T06:43:51.315 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:51 smithi042 ceph-mon[32775]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:43:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:51 smithi164 ceph-mon[36970]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:43:51.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:51 smithi163 ceph-mon[36873]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:43:52.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:52 smithi042 ceph-mon[32775]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:43:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:52 smithi164 ceph-mon[36970]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:43:52.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:52 smithi163 ceph-mon[36873]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:43:54.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:54 smithi042 ceph-mon[32775]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-07-26T06:43:54.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:43:54.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:54.506 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:54 smithi164 ceph-mon[36970]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-07-26T06:43:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:43:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:54 smithi163 ceph-mon[36873]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-07-26T06:43:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:43:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:56.382 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:56 smithi042 ceph-mon[32775]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:43:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:56 smithi164 ceph-mon[36970]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:43:56.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:56 smithi163 ceph-mon[36873]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:43:57.494 INFO:teuthology.orchestra.run.smithi042.stdout:Created osd(s) 6 on host 'smithi042' 2024-07-26T06:43:57.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:57.493+0000 7ff3d494d640 1 -- 172.21.15.42:0/842890903 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff3d0063230 con 0x7ff3b80b3bd0 2024-07-26T06:43:57.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:57.495+0000 7ff3d594f640 1 -- 172.21.15.42:0/842890903 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff3b80b3bd0 msgr2=0x7ff3b80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:43:57.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:57.495+0000 7ff3d594f640 1 --2- 172.21.15.42:0/842890903 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff3b80b3bd0 0x7ff3b80b60e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff3c00058c0 tx=0x7ff3c0005850 comp rx=0 tx=0).stop 2024-07-26T06:43:57.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:57.495+0000 7ff3d594f640 1 -- 172.21.15.42:0/842890903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3d0103110 msgr2=0x7ff3d019e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:43:57.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:57.495+0000 7ff3d594f640 1 --2- 172.21.15.42:0/842890903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3d0103110 0x7ff3d019e710 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4006010 tx=0x7ff3c4004340 comp rx=0 tx=0).stop 2024-07-26T06:43:57.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:57.496+0000 7ff3d594f640 1 -- 172.21.15.42:0/842890903 shutdown_connections 2024-07-26T06:43:57.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:57.496+0000 7ff3d594f640 1 --2- 172.21.15.42:0/842890903 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff3b80b3bd0 0x7ff3b80b60e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:57.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:57.496+0000 7ff3d594f640 1 --2- 172.21.15.42:0/842890903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3d0103ab0 0x7ff3d01a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:57.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:57.496+0000 7ff3d594f640 1 --2- 172.21.15.42:0/842890903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3d0103110 0x7ff3d019e710 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:57.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:57.496+0000 7ff3d594f640 1 --2- 172.21.15.42:0/842890903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3d0102080 0x7ff3d006be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:43:57.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:57.496+0000 7ff3d594f640 1 -- 172.21.15.42:0/842890903 >> 172.21.15.42:0/842890903 conn(0x7ff3d00fda10 msgr2=0x7ff3d0106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:43:57.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:57.496+0000 7ff3d594f640 1 -- 172.21.15.42:0/842890903 shutdown_connections 2024-07-26T06:43:57.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:43:57.496+0000 7ff3d594f640 1 -- 172.21.15.42:0/842890903 wait complete. 2024-07-26T06:43:57.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:57.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:57.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:57.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:58.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:43:58.128 DEBUG:teuthology.orchestra.run.smithi042:osd.6> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@osd.6.service 2024-07-26T06:43:58.130 INFO:tasks.cephadm:Deploying osd.7 on smithi163 with /dev/vg_nvme/lv_2... 2024-07-26T06:43:58.130 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T06:43:58.392 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.b/config 2024-07-26T06:43:58.619 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:58 smithi042 ceph-mon[32775]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:43:58.620 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:58 smithi042 ceph-mon[32775]: from='osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T06:43:58.792 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:58 smithi163 ceph-mon[36873]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:43:58.792 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:58 smithi163 ceph-mon[36873]: from='osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T06:43:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:58 smithi164 ceph-mon[36970]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:43:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:58 smithi164 ceph-mon[36970]: from='osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T06:43:59.716 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:59 smithi163 ceph-mon[36873]: from='osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T06:43:59.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:59 smithi163 ceph-mon[36873]: osdmap e40: 7 total, 6 up, 7 in 2024-07-26T06:43:59.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:59 smithi163 ceph-mon[36873]: from='osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]: dispatch 2024-07-26T06:43:59.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:43:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:43:59.728 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:59 smithi042 ceph-mon[32775]: from='osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T06:43:59.729 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:59 smithi042 ceph-mon[32775]: osdmap e40: 7 total, 6 up, 7 in 2024-07-26T06:43:59.729 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:59 smithi042 ceph-mon[32775]: from='osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]: dispatch 2024-07-26T06:43:59.729 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:43:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:43:59.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:59 smithi164 ceph-mon[36970]: from='osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T06:43:59.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:59 smithi164 ceph-mon[36970]: osdmap e40: 7 total, 6 up, 7 in 2024-07-26T06:43:59.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:59 smithi164 ceph-mon[36970]: from='osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]: dispatch 2024-07-26T06:43:59.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:43:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:44:00.091 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T06:44:00.200 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T06:44:00.213 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-07-26T06:44:00.213 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-07-26T06:44:00.213 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113424 s, 924 MB/s 2024-07-26T06:44:00.213 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-07-26T06:44:00.395 INFO:journalctl@ceph.osd.6.smithi042.stdout:Jul 26 06:44:00 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-6[63627]: 2024-07-26T06:44:00.072+0000 7f6f907f3640 -1 osd.6 0 waiting for initial osdmap 2024-07-26T06:44:00.395 INFO:journalctl@ceph.osd.6.smithi042.stdout:Jul 26 06:44:00 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-6[63627]: 2024-07-26T06:44:00.082+0000 7f6f885e9640 -1 osd.6 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:44:00.862 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:00 smithi163 ceph-mon[36873]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:44:00.862 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:00 smithi163 ceph-mon[36873]: from='osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-07-26T06:44:00.862 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:00 smithi163 ceph-mon[36873]: osdmap e41: 7 total, 6 up, 7 in 2024-07-26T06:44:00.862 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:44:00.862 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:44:00.862 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:00 smithi163 ceph-mon[36873]: from='osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300]' entity='osd.6' 2024-07-26T06:44:00.889 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:00 smithi164 ceph-mon[36970]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:44:00.889 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:00 smithi164 ceph-mon[36970]: from='osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-07-26T06:44:00.889 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:00 smithi164 ceph-mon[36970]: osdmap e41: 7 total, 6 up, 7 in 2024-07-26T06:44:00.889 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:44:00.889 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:44:00.889 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:00 smithi164 ceph-mon[36970]: from='osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300]' entity='osd.6' 2024-07-26T06:44:00.890 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi163:vg_nvme/lv_2 2024-07-26T06:44:00.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:00 smithi042 ceph-mon[32775]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:44:00.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:00 smithi042 ceph-mon[32775]: from='osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-07-26T06:44:00.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:00 smithi042 ceph-mon[32775]: osdmap e41: 7 total, 6 up, 7 in 2024-07-26T06:44:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:44:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:44:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:00 smithi042 ceph-mon[32775]: from='osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300]' entity='osd.6' 2024-07-26T06:44:01.150 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.b/config 2024-07-26T06:44:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: purged_snaps scrub starts 2024-07-26T06:44:01.968 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: purged_snaps scrub ok 2024-07-26T06:44:01.969 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:44:01.969 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300] boot 2024-07-26T06:44:01.969 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: osdmap e42: 7 total, 7 up, 7 in 2024-07-26T06:44:01.969 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:44:01.969 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:01.969 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:01.969 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:01.969 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:01.969 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:01.969 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:01.969 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:01.970 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:01.970 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:01.970 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:44:01.970 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:01.970 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:01.970 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:01.970 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: purged_snaps scrub starts 2024-07-26T06:44:01.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: purged_snaps scrub ok 2024-07-26T06:44:01.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:44:01.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300] boot 2024-07-26T06:44:01.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: osdmap e42: 7 total, 7 up, 7 in 2024-07-26T06:44:01.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:44:01.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:01.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:01.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:01.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:01.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:01.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:01.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:01.971 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:01.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:01.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:44:01.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:01.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:01.972 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: purged_snaps scrub starts 2024-07-26T06:44:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: purged_snaps scrub ok 2024-07-26T06:44:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:44:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: osd.6 [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300] boot 2024-07-26T06:44:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: osdmap e42: 7 total, 7 up, 7 in 2024-07-26T06:44:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T06:44:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:44:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:02.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.763+0000 7faabff52640 1 -- 172.21.15.163:0/1099345757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faab8104be0 msgr2=0x7faab810b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:44:02.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.763+0000 7faabff52640 1 --2- 172.21.15.163:0/1099345757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faab8104be0 0x7faab810b580 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7faab0009ca0 tx=0x7faab002a950 comp rx=0 tx=0).stop 2024-07-26T06:44:02.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.763+0000 7faabff52640 1 -- 172.21.15.163:0/1099345757 shutdown_connections 2024-07-26T06:44:02.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.763+0000 7faabff52640 1 --2- 172.21.15.163:0/1099345757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faab8104be0 0x7faab810b580 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:02.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.763+0000 7faabff52640 1 --2- 172.21.15.163:0/1099345757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faab8104240 0x7faab8104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:02.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.763+0000 7faabff52640 1 --2- 172.21.15.163:0/1099345757 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faab81031b0 0x7faab8103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:02.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.763+0000 7faabff52640 1 -- 172.21.15.163:0/1099345757 >> 172.21.15.163:0/1099345757 conn(0x7faab80feb20 msgr2=0x7faab8100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:44:02.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.763+0000 7faabff52640 1 -- 172.21.15.163:0/1099345757 shutdown_connections 2024-07-26T06:44:02.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.763+0000 7faabff52640 1 -- 172.21.15.163:0/1099345757 wait complete. 2024-07-26T06:44:02.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.764+0000 7faabff52640 1 Processor -- start 2024-07-26T06:44:02.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.764+0000 7faabff52640 1 -- start start 2024-07-26T06:44:02.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.764+0000 7faabff52640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faab81031b0 0x7faab8076870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:44:02.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.764+0000 7faabff52640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faab8104240 0x7faab8075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:44:02.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.764+0000 7faabff52640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faab8104be0 0x7faab8075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:44:02.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.764+0000 7faabff52640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faab810ee60 con 0x7faab8104240 2024-07-26T06:44:02.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.764+0000 7faabff52640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7faab810ece0 con 0x7faab81031b0 2024-07-26T06:44:02.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.764+0000 7faabff52640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faab810efe0 con 0x7faab8104be0 2024-07-26T06:44:02.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.764+0000 7faabdcee640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faab81031b0 0x7faab8076870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:44:02.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.764+0000 7faabdcee640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faab81031b0 0x7faab8076870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:59802/0 (socket says 172.21.15.163:59802) 2024-07-26T06:44:02.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.764+0000 7faabdcee640 1 -- 172.21.15.163:0/308595754 learned_addr learned my addr 172.21.15.163:0/308595754 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:44:02.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.765+0000 7faabd4ed640 1 --2- 172.21.15.163:0/308595754 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faab8104240 0x7faab8075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:44:02.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.765+0000 7faabe4ef640 1 --2- 172.21.15.163:0/308595754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faab8104be0 0x7faab8075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:44:02.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.765+0000 7faabdcee640 1 -- 172.21.15.163:0/308595754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faab8104be0 msgr2=0x7faab8075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:44:02.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.765+0000 7faabdcee640 1 --2- 172.21.15.163:0/308595754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faab8104be0 0x7faab8075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:02.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.765+0000 7faabdcee640 1 -- 172.21.15.163:0/308595754 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faab8104240 msgr2=0x7faab8075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:44:02.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.765+0000 7faabdcee640 1 --2- 172.21.15.163:0/308595754 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faab8104240 0x7faab8075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:02.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.765+0000 7faabdcee640 1 -- 172.21.15.163:0/308595754 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab8075d80 con 0x7faab81031b0 2024-07-26T06:44:02.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.765+0000 7faabdcee640 1 --2- 172.21.15.163:0/308595754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faab81031b0 0x7faab8076870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7faab400d950 tx=0x7faab400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:44:02.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.765+0000 7faaaeffd640 1 -- 172.21.15.163:0/308595754 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab4017070 con 0x7faab81031b0 2024-07-26T06:44:02.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.765+0000 7faaaeffd640 1 -- 172.21.15.163:0/308595754 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faab4009c30 con 0x7faab81031b0 2024-07-26T06:44:02.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.765+0000 7faabff52640 1 -- 172.21.15.163:0/308595754 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab81a3a30 con 0x7faab81031b0 2024-07-26T06:44:02.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.765+0000 7faaaeffd640 1 -- 172.21.15.163:0/308595754 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab400f620 con 0x7faab81031b0 2024-07-26T06:44:02.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.765+0000 7faabff52640 1 -- 172.21.15.163:0/308595754 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab81a3e90 con 0x7faab81031b0 2024-07-26T06:44:02.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.766+0000 7faabff52640 1 -- 172.21.15.163:0/308595754 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faab810a390 con 0x7faab81031b0 2024-07-26T06:44:02.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.768+0000 7faaaeffd640 1 -- 172.21.15.163:0/308595754 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7faab4010460 con 0x7faab81031b0 2024-07-26T06:44:02.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.768+0000 7faaaeffd640 1 --2- 172.21.15.163:0/308595754 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faaa40b3bd0 0x7faaa40b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:44:02.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.768+0000 7faaaeffd640 1 -- 172.21.15.163:0/308595754 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 4965+0+0 (secure 0 0 0) 0x7faab40e2bd0 con 0x7faab81031b0 2024-07-26T06:44:02.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.768+0000 7faabd4ed640 1 --2- 172.21.15.163:0/308595754 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faaa40b3bd0 0x7faaa40b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:44:02.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.769+0000 7faabd4ed640 1 --2- 172.21.15.163:0/308595754 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faaa40b3bd0 0x7faaa40b60e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7faaa8009aa0 tx=0x7faaa8005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:44:02.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.772+0000 7faaaeffd640 1 -- 172.21.15.163:0/308595754 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7faab40aaf30 con 0x7faab81031b0 2024-07-26T06:44:02.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:02 smithi042 ceph-mon[32775]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:44:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:02 smithi042 ceph-mon[32775]: Detected new or changed devices on smithi042 2024-07-26T06:44:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:02 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 5697M 2024-07-26T06:44:02.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:02.955+0000 7faabff52640 1 -- 172.21.15.163:0/308595754 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7faab8063230 con 0x7faaa40b3bd0 2024-07-26T06:44:02.956 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:02 smithi163 ceph-mon[36873]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:44:02.956 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:02 smithi163 ceph-mon[36873]: Detected new or changed devices on smithi042 2024-07-26T06:44:02.956 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:02 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 5697M 2024-07-26T06:44:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:02 smithi164 ceph-mon[36970]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-07-26T06:44:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:02 smithi164 ceph-mon[36970]: Detected new or changed devices on smithi042 2024-07-26T06:44:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:02 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 5697M 2024-07-26T06:44:03.673 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:03 smithi163 ceph-mon[36873]: osdmap e43: 7 total, 7 up, 7 in 2024-07-26T06:44:03.673 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:03 smithi163 ceph-mon[36873]: from='client.24416 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:44:03.673 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:44:03.673 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:44:03.674 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:03.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:03 smithi042 ceph-mon[32775]: osdmap e43: 7 total, 7 up, 7 in 2024-07-26T06:44:03.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:03 smithi042 ceph-mon[32775]: from='client.24416 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:44:03.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:44:03.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:44:03.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:03.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:03 smithi164 ceph-mon[36970]: osdmap e43: 7 total, 7 up, 7 in 2024-07-26T06:44:03.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:03 smithi164 ceph-mon[36970]: from='client.24416 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:44:03.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:44:03.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:44:03.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:04.670 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:04 smithi163 ceph-mon[36873]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:04.867 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:04 smithi042 ceph-mon[32775]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:04.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:04 smithi164 ceph-mon[36970]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:05 smithi042 ceph-mon[32775]: from='client.? 172.21.15.163:0/1274449842' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f699004c-f7c2-4f1c-ae4b-7178f3b06e2d"}]: dispatch 2024-07-26T06:44:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:05 smithi042 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f699004c-f7c2-4f1c-ae4b-7178f3b06e2d"}]: dispatch 2024-07-26T06:44:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:05 smithi042 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f699004c-f7c2-4f1c-ae4b-7178f3b06e2d"}]': finished 2024-07-26T06:44:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:05 smithi042 ceph-mon[32775]: osdmap e44: 8 total, 7 up, 8 in 2024-07-26T06:44:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T06:44:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:05 smithi164 ceph-mon[36970]: from='client.? 172.21.15.163:0/1274449842' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f699004c-f7c2-4f1c-ae4b-7178f3b06e2d"}]: dispatch 2024-07-26T06:44:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:05 smithi164 ceph-mon[36970]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f699004c-f7c2-4f1c-ae4b-7178f3b06e2d"}]: dispatch 2024-07-26T06:44:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:05 smithi164 ceph-mon[36970]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f699004c-f7c2-4f1c-ae4b-7178f3b06e2d"}]': finished 2024-07-26T06:44:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:05 smithi164 ceph-mon[36970]: osdmap e44: 8 total, 7 up, 8 in 2024-07-26T06:44:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T06:44:06.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:05 smithi163 ceph-mon[36873]: from='client.? 172.21.15.163:0/1274449842' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f699004c-f7c2-4f1c-ae4b-7178f3b06e2d"}]: dispatch 2024-07-26T06:44:06.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:05 smithi163 ceph-mon[36873]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f699004c-f7c2-4f1c-ae4b-7178f3b06e2d"}]: dispatch 2024-07-26T06:44:06.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:05 smithi163 ceph-mon[36873]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f699004c-f7c2-4f1c-ae4b-7178f3b06e2d"}]': finished 2024-07-26T06:44:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:05 smithi163 ceph-mon[36873]: osdmap e44: 8 total, 7 up, 8 in 2024-07-26T06:44:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T06:44:06.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:06 smithi042 ceph-mon[32775]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:06 smithi042 ceph-mon[32775]: from='client.? 172.21.15.163:0/844608076' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:44:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:06 smithi164 ceph-mon[36970]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:06 smithi164 ceph-mon[36970]: from='client.? 172.21.15.163:0/844608076' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:44:07.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:06 smithi163 ceph-mon[36873]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:06 smithi163 ceph-mon[36873]: from='client.? 172.21.15.163:0/844608076' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:44:08.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:08 smithi042 ceph-mon[32775]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:08.943 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:08 smithi163 ceph-mon[36873]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:08.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:08 smithi164 ceph-mon[36970]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:10.717 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:10 smithi163 ceph-mon[36873]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:10.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:10 smithi042 ceph-mon[32775]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:10 smithi164 ceph-mon[36970]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:12 smithi042 ceph-mon[32775]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T06:44:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:12.953 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:12 smithi163 ceph-mon[36873]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:12.953 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T06:44:12.953 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:12 smithi164 ceph-mon[36970]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T06:44:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:13.741 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:13 smithi163 ceph-mon[36873]: Deploying daemon osd.7 on smithi163 2024-07-26T06:44:13.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:13 smithi042 ceph-mon[32775]: Deploying daemon osd.7 on smithi163 2024-07-26T06:44:13.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:13 smithi164 ceph-mon[36970]: Deploying daemon osd.7 on smithi163 2024-07-26T06:44:14.890 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:14 smithi163 ceph-mon[36873]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:14.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:14 smithi042 ceph-mon[32775]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:14 smithi164 ceph-mon[36970]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:16 smithi042 ceph-mon[32775]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:16.906 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:16 smithi163 ceph-mon[36873]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:16 smithi164 ceph-mon[36970]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:17.923 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:44:17.923 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:17.923 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:17.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:44:17.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:17.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:18.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:44:18.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:18.958 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:18 smithi163 ceph-mon[36873]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:18 smithi164 ceph-mon[36970]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:19.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:18 smithi042 ceph-mon[32775]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:20 smithi164 ceph-mon[36970]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:21.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:20 smithi163 ceph-mon[36873]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:21.114 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 7 on host 'smithi163' 2024-07-26T06:44:21.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:21.113+0000 7faaaeffd640 1 -- 172.21.15.163:0/308595754 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7faab8063230 con 0x7faaa40b3bd0 2024-07-26T06:44:21.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:21.115+0000 7faabff52640 1 -- 172.21.15.163:0/308595754 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faaa40b3bd0 msgr2=0x7faaa40b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:44:21.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:21.115+0000 7faabff52640 1 --2- 172.21.15.163:0/308595754 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faaa40b3bd0 0x7faaa40b60e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7faaa8009aa0 tx=0x7faaa8005f90 comp rx=0 tx=0).stop 2024-07-26T06:44:21.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:21.115+0000 7faabff52640 1 -- 172.21.15.163:0/308595754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faab81031b0 msgr2=0x7faab8076870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:44:21.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:21.115+0000 7faabff52640 1 --2- 172.21.15.163:0/308595754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faab81031b0 0x7faab8076870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7faab400d950 tx=0x7faab400de10 comp rx=0 tx=0).stop 2024-07-26T06:44:21.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:21.116+0000 7faabff52640 1 -- 172.21.15.163:0/308595754 shutdown_connections 2024-07-26T06:44:21.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:21.116+0000 7faabff52640 1 --2- 172.21.15.163:0/308595754 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faaa40b3bd0 0x7faaa40b60e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:21.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:21.116+0000 7faabff52640 1 --2- 172.21.15.163:0/308595754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faab8104be0 0x7faab8075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:21.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:21.116+0000 7faabff52640 1 --2- 172.21.15.163:0/308595754 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faab8104240 0x7faab8075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:21.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:21.116+0000 7faabff52640 1 --2- 172.21.15.163:0/308595754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faab81031b0 0x7faab8076870 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:21.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:21.116+0000 7faabff52640 1 -- 172.21.15.163:0/308595754 >> 172.21.15.163:0/308595754 conn(0x7faab80feb20 msgr2=0x7faab8107560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:44:21.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:21.116+0000 7faabff52640 1 -- 172.21.15.163:0/308595754 shutdown_connections 2024-07-26T06:44:21.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:44:21.116+0000 7faabff52640 1 -- 172.21.15.163:0/308595754 wait complete. 2024-07-26T06:44:21.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:20 smithi042 ceph-mon[32775]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:21.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:21.828 DEBUG:teuthology.orchestra.run.smithi163:osd.7> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@osd.7.service 2024-07-26T06:44:21.830 INFO:tasks.cephadm:Deploying osd.8 on smithi164 with /dev/vg_nvme/lv_2... 2024-07-26T06:44:21.830 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T06:44:21.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:21.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:21.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:21.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:44:21.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:21.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:21.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:21 smithi163 ceph-mon[36873]: from='osd.7 [v2:172.21.15.163:6816/812214111,v1:172.21.15.163:6817/812214111]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T06:44:21.969 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:21 smithi163 ceph-mon[36873]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T06:44:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:44:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:21 smithi164 ceph-mon[36970]: from='osd.7 [v2:172.21.15.163:6816/812214111,v1:172.21.15.163:6817/812214111]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T06:44:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:21 smithi164 ceph-mon[36970]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T06:44:22.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:22.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:22.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:22.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:44:22.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:22.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:22.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:21 smithi042 ceph-mon[32775]: from='osd.7 [v2:172.21.15.163:6816/812214111,v1:172.21.15.163:6817/812214111]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T06:44:22.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:21 smithi042 ceph-mon[32775]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T06:44:22.096 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T06:44:22.969 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:22 smithi164 ceph-mon[36970]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:22.969 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:22 smithi164 ceph-mon[36970]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T06:44:22.969 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:22 smithi164 ceph-mon[36970]: osdmap e45: 8 total, 7 up, 8 in 2024-07-26T06:44:22.969 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T06:44:22.969 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:22 smithi164 ceph-mon[36970]: from='osd.7 [v2:172.21.15.163:6816/812214111,v1:172.21.15.163:6817/812214111]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:44:22.970 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:22 smithi164 ceph-mon[36970]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:44:23.000 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:22 smithi163 ceph-mon[36873]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:23.000 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:22 smithi163 ceph-mon[36873]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T06:44:23.000 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:22 smithi163 ceph-mon[36873]: osdmap e45: 8 total, 7 up, 8 in 2024-07-26T06:44:23.000 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T06:44:23.000 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:22 smithi163 ceph-mon[36873]: from='osd.7 [v2:172.21.15.163:6816/812214111,v1:172.21.15.163:6817/812214111]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:44:23.000 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:22 smithi163 ceph-mon[36873]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:44:23.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:22 smithi042 ceph-mon[32775]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:23.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:22 smithi042 ceph-mon[32775]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T06:44:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:22 smithi042 ceph-mon[32775]: osdmap e45: 8 total, 7 up, 8 in 2024-07-26T06:44:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T06:44:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:22 smithi042 ceph-mon[32775]: from='osd.7 [v2:172.21.15.163:6816/812214111,v1:172.21.15.163:6817/812214111]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:44:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:22 smithi042 ceph-mon[32775]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:44:23.983 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T06:44:24.007 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jul 26 06:44:23 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-7[54852]: 2024-07-26T06:44:23.596+0000 7fd256908640 -1 osd.7 0 waiting for initial osdmap 2024-07-26T06:44:24.007 INFO:journalctl@ceph.osd.7.smithi163.stdout:Jul 26 06:44:23 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-7[54852]: 2024-07-26T06:44:23.606+0000 7fd24e6fe640 -1 osd.7 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:44:24.069 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T06:44:24.083 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2024-07-26T06:44:24.083 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2024-07-26T06:44:24.083 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113191 s, 926 MB/s 2024-07-26T06:44:24.083 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2024-07-26T06:44:24.373 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:24 smithi163 ceph-mon[36873]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-07-26T06:44:24.374 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:24 smithi163 ceph-mon[36873]: osdmap e46: 8 total, 7 up, 8 in 2024-07-26T06:44:24.374 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T06:44:24.374 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T06:44:24.374 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:24 smithi163 ceph-mon[36873]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:24.374 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:24 smithi163 ceph-mon[36873]: from='osd.7 ' entity='osd.7' 2024-07-26T06:44:24.374 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:24 smithi163 ceph-mon[36873]: osd.7 [v2:172.21.15.163:6816/812214111,v1:172.21.15.163:6817/812214111] boot 2024-07-26T06:44:24.374 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:24 smithi163 ceph-mon[36873]: osdmap e47: 8 total, 8 up, 8 in 2024-07-26T06:44:24.374 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T06:44:24.382 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:24 smithi164 ceph-mon[36970]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-07-26T06:44:24.382 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:24 smithi164 ceph-mon[36970]: osdmap e46: 8 total, 7 up, 8 in 2024-07-26T06:44:24.383 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T06:44:24.383 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T06:44:24.383 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:24 smithi164 ceph-mon[36970]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:24.383 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:24 smithi164 ceph-mon[36970]: from='osd.7 ' entity='osd.7' 2024-07-26T06:44:24.383 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:24 smithi164 ceph-mon[36970]: osd.7 [v2:172.21.15.163:6816/812214111,v1:172.21.15.163:6817/812214111] boot 2024-07-26T06:44:24.383 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:24 smithi164 ceph-mon[36970]: osdmap e47: 8 total, 8 up, 8 in 2024-07-26T06:44:24.383 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T06:44:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:24 smithi042 ceph-mon[32775]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-07-26T06:44:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:24 smithi042 ceph-mon[32775]: osdmap e46: 8 total, 7 up, 8 in 2024-07-26T06:44:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T06:44:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T06:44:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:24 smithi042 ceph-mon[32775]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-07-26T06:44:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:24 smithi042 ceph-mon[32775]: from='osd.7 ' entity='osd.7' 2024-07-26T06:44:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:24 smithi042 ceph-mon[32775]: osd.7 [v2:172.21.15.163:6816/812214111,v1:172.21.15.163:6817/812214111] boot 2024-07-26T06:44:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:24 smithi042 ceph-mon[32775]: osdmap e47: 8 total, 8 up, 8 in 2024-07-26T06:44:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T06:44:24.808 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi164:vg_nvme/lv_2 2024-07-26T06:44:25.066 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T06:44:25.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:25 smithi042 ceph-mon[32775]: purged_snaps scrub starts 2024-07-26T06:44:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:25 smithi042 ceph-mon[32775]: purged_snaps scrub ok 2024-07-26T06:44:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:25 smithi042 ceph-mon[32775]: Detected new or changed devices on smithi163 2024-07-26T06:44:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:25 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 5696M 2024-07-26T06:44:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:25 smithi042 ceph-mon[32775]: osdmap e48: 8 total, 8 up, 8 in 2024-07-26T06:44:25.431 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:25 smithi164 ceph-mon[36970]: purged_snaps scrub starts 2024-07-26T06:44:25.431 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:25 smithi164 ceph-mon[36970]: purged_snaps scrub ok 2024-07-26T06:44:25.431 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:25 smithi164 ceph-mon[36970]: Detected new or changed devices on smithi163 2024-07-26T06:44:25.431 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:25.431 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:25.431 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:25.431 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:25.431 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:25.431 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:25 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 5696M 2024-07-26T06:44:25.431 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:25.431 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:25.431 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:25.431 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:25.432 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:25 smithi164 ceph-mon[36970]: osdmap e48: 8 total, 8 up, 8 in 2024-07-26T06:44:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:25 smithi163 ceph-mon[36873]: purged_snaps scrub starts 2024-07-26T06:44:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:25 smithi163 ceph-mon[36873]: purged_snaps scrub ok 2024-07-26T06:44:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:25 smithi163 ceph-mon[36873]: Detected new or changed devices on smithi163 2024-07-26T06:44:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:25 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 5696M 2024-07-26T06:44:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:25 smithi163 ceph-mon[36873]: osdmap e48: 8 total, 8 up, 8 in 2024-07-26T06:44:26.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:26 smithi164 ceph-mon[36970]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:26.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:26 smithi042 ceph-mon[32775]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:26 smithi163 ceph-mon[36873]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:26.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.770+0000 7f7cb22a7640 1 -- 172.21.15.164:0/1935327447 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cac103180 msgr2=0x7f7cac1035f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:44:26.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.770+0000 7f7cb22a7640 1 --2- 172.21.15.164:0/1935327447 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cac103180 0x7f7cac1035f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009900 tx=0x7f7c9c02a480 comp rx=0 tx=0).stop 2024-07-26T06:44:26.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.770+0000 7f7cb22a7640 1 -- 172.21.15.164:0/1935327447 shutdown_connections 2024-07-26T06:44:26.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.770+0000 7f7cb22a7640 1 --2- 172.21.15.164:0/1935327447 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac104bb0 0x7f7cac10b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.770+0000 7f7cb22a7640 1 --2- 172.21.15.164:0/1935327447 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7cac104210 0x7f7cac104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.770+0000 7f7cb22a7640 1 --2- 172.21.15.164:0/1935327447 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cac103180 0x7f7cac1035f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.770+0000 7f7cb22a7640 1 -- 172.21.15.164:0/1935327447 >> 172.21.15.164:0/1935327447 conn(0x7f7cac0feaf0 msgr2=0x7f7cac100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:44:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.770+0000 7f7cb22a7640 1 -- 172.21.15.164:0/1935327447 shutdown_connections 2024-07-26T06:44:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.770+0000 7f7cb22a7640 1 -- 172.21.15.164:0/1935327447 wait complete. 2024-07-26T06:44:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.770+0000 7f7cb22a7640 1 Processor -- start 2024-07-26T06:44:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.770+0000 7f7cb22a7640 1 -- start start 2024-07-26T06:44:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cb22a7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7cac103180 0x7f7cac1999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:44:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cb22a7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cac104210 0x7f7cac19a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:44:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cb22a7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac104bb0 0x7f7cac19ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:44:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cb22a7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7cac10ede0 con 0x7f7cac104bb0 2024-07-26T06:44:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cb22a7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7cac10ec60 con 0x7f7cac103180 2024-07-26T06:44:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cb22a7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7cac10ef60 con 0x7f7cac104210 2024-07-26T06:44:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cab7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cac104210 0x7f7cac19a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:44:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cab7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cac104210 0x7f7cac19a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:59330/0 (socket says 172.21.15.164:59330) 2024-07-26T06:44:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cab7fe640 1 -- 172.21.15.164:0/1261158898 learned_addr learned my addr 172.21.15.164:0/1261158898 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-07-26T06:44:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cabfff640 1 --2- 172.21.15.164:0/1261158898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7cac103180 0x7f7cac1999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:44:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cb0844640 1 --2- 172.21.15.164:0/1261158898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac104bb0 0x7f7cac19ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:44:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cab7fe640 1 -- 172.21.15.164:0/1261158898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7cac103180 msgr2=0x7f7cac1999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:44:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cab7fe640 1 --2- 172.21.15.164:0/1261158898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7cac103180 0x7f7cac1999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cab7fe640 1 -- 172.21.15.164:0/1261158898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac104bb0 msgr2=0x7f7cac19ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:44:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cab7fe640 1 --2- 172.21.15.164:0/1261158898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac104bb0 0x7f7cac19ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cab7fe640 1 -- 172.21.15.164:0/1261158898 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cac19b390 con 0x7f7cac104210 2024-07-26T06:44:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cb0844640 1 --2- 172.21.15.164:0/1261158898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac104bb0 0x7f7cac19ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:44:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.771+0000 7f7cab7fe640 1 --2- 172.21.15.164:0/1261158898 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cac104210 0x7f7cac19a060 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7ca000e9f0 tx=0x7f7ca000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:44:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.772+0000 7f7ca97fa640 1 -- 172.21.15.164:0/1261158898 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca000cd80 con 0x7f7cac104210 2024-07-26T06:44:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.772+0000 7f7cb22a7640 1 -- 172.21.15.164:0/1261158898 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cac19b680 con 0x7f7cac104210 2024-07-26T06:44:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.772+0000 7f7ca97fa640 1 -- 172.21.15.164:0/1261158898 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ca00054d0 con 0x7f7cac104210 2024-07-26T06:44:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.772+0000 7f7ca97fa640 1 -- 172.21.15.164:0/1261158898 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0010690 con 0x7f7cac104210 2024-07-26T06:44:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.772+0000 7f7cb22a7640 1 -- 172.21.15.164:0/1261158898 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cac1a1e10 con 0x7f7cac104210 2024-07-26T06:44:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.773+0000 7f7cb22a7640 1 -- 172.21.15.164:0/1261158898 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cac10a360 con 0x7f7cac104210 2024-07-26T06:44:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.774+0000 7f7ca97fa640 1 -- 172.21.15.164:0/1261158898 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f7ca0004e30 con 0x7f7cac104210 2024-07-26T06:44:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.775+0000 7f7ca97fa640 1 --2- 172.21.15.164:0/1261158898 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7c940b3bd0 0x7f7c940b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:44:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.775+0000 7f7ca97fa640 1 -- 172.21.15.164:0/1261158898 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5397+0+0 (secure 0 0 0) 0x7f7ca00e3820 con 0x7f7cac104210 2024-07-26T06:44:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.775+0000 7f7cabfff640 1 --2- 172.21.15.164:0/1261158898 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7c940b3bd0 0x7f7c940b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:44:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.776+0000 7f7cabfff640 1 --2- 172.21.15.164:0/1261158898 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7c940b3bd0 0x7f7c940b60e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c031440 tx=0x7f7c9c0314a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:44:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.778+0000 7f7ca97fa640 1 -- 172.21.15.164:0/1261158898 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7ca0055600 con 0x7f7cac104210 2024-07-26T06:44:26.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:26.959+0000 7f7cb22a7640 1 -- 172.21.15.164:0/1261158898 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7cac063230 con 0x7f7c940b3bd0 2024-07-26T06:44:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:27 smithi164 ceph-mon[36970]: osdmap e49: 8 total, 8 up, 8 in 2024-07-26T06:44:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:27 smithi164 ceph-mon[36970]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:44:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:44:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:44:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:27.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:27 smithi042 ceph-mon[32775]: osdmap e49: 8 total, 8 up, 8 in 2024-07-26T06:44:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:27 smithi042 ceph-mon[32775]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:44:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:44:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:44:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:27 smithi163 ceph-mon[36873]: osdmap e49: 8 total, 8 up, 8 in 2024-07-26T06:44:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:27 smithi163 ceph-mon[36873]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:44:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:44:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:44:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:28.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:28 smithi042 ceph-mon[32775]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:28.437 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:28 smithi164 ceph-mon[36970]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:28.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:28 smithi163 ceph-mon[36873]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:30 smithi042 ceph-mon[32775]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T06:44:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:30 smithi042 ceph-mon[32775]: from='client.? 172.21.15.164:0/2278989379' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3476163e-efea-4652-a8d4-b7c4a908f984"}]: dispatch 2024-07-26T06:44:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:30 smithi042 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3476163e-efea-4652-a8d4-b7c4a908f984"}]: dispatch 2024-07-26T06:44:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:30 smithi042 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3476163e-efea-4652-a8d4-b7c4a908f984"}]': finished 2024-07-26T06:44:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:30 smithi042 ceph-mon[32775]: osdmap e50: 9 total, 8 up, 9 in 2024-07-26T06:44:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T06:44:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:30 smithi042 ceph-mon[32775]: from='client.? 172.21.15.164:0/2894450498' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:44:30.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:30 smithi164 ceph-mon[36970]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T06:44:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:30 smithi164 ceph-mon[36970]: from='client.? 172.21.15.164:0/2278989379' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3476163e-efea-4652-a8d4-b7c4a908f984"}]: dispatch 2024-07-26T06:44:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:30 smithi164 ceph-mon[36970]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3476163e-efea-4652-a8d4-b7c4a908f984"}]: dispatch 2024-07-26T06:44:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:30 smithi164 ceph-mon[36970]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3476163e-efea-4652-a8d4-b7c4a908f984"}]': finished 2024-07-26T06:44:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:30 smithi164 ceph-mon[36970]: osdmap e50: 9 total, 8 up, 9 in 2024-07-26T06:44:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T06:44:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:30 smithi164 ceph-mon[36970]: from='client.? 172.21.15.164:0/2894450498' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:44:30.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:30 smithi163 ceph-mon[36873]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T06:44:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:30 smithi163 ceph-mon[36873]: from='client.? 172.21.15.164:0/2278989379' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3476163e-efea-4652-a8d4-b7c4a908f984"}]: dispatch 2024-07-26T06:44:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:30 smithi163 ceph-mon[36873]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3476163e-efea-4652-a8d4-b7c4a908f984"}]: dispatch 2024-07-26T06:44:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:30 smithi163 ceph-mon[36873]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3476163e-efea-4652-a8d4-b7c4a908f984"}]': finished 2024-07-26T06:44:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:30 smithi163 ceph-mon[36873]: osdmap e50: 9 total, 8 up, 9 in 2024-07-26T06:44:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T06:44:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:30 smithi163 ceph-mon[36873]: from='client.? 172.21.15.164:0/2894450498' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:44:32.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:32 smithi042 ceph-mon[32775]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-07-26T06:44:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:32 smithi164 ceph-mon[36970]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-07-26T06:44:32.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:32 smithi163 ceph-mon[36873]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-07-26T06:44:34.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:34 smithi164 ceph-mon[36970]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T06:44:34.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:34 smithi042 ceph-mon[32775]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T06:44:34.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:34 smithi163 ceph-mon[36873]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T06:44:36.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:36 smithi042 ceph-mon[32775]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-07-26T06:44:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:36 smithi164 ceph-mon[36970]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-07-26T06:44:36.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:36 smithi163 ceph-mon[36873]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-07-26T06:44:37.633 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T06:44:37.633 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:37.633 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:37 smithi164 ceph-mon[36970]: Deploying daemon osd.8 on smithi164 2024-07-26T06:44:37.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T06:44:37.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:37.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:37 smithi042 ceph-mon[32775]: Deploying daemon osd.8 on smithi164 2024-07-26T06:44:37.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T06:44:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:37 smithi163 ceph-mon[36873]: Deploying daemon osd.8 on smithi164 2024-07-26T06:44:38.552 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:38 smithi164 ceph-mon[36970]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:44:38.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:38 smithi042 ceph-mon[32775]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:44:38.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:38 smithi163 ceph-mon[36873]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:44:40.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:40 smithi042 ceph-mon[32775]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:40.662 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:40 smithi164 ceph-mon[36970]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:40.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:40 smithi163 ceph-mon[36873]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:42.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:42 smithi042 ceph-mon[32775]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:44:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:42.657 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:42 smithi164 ceph-mon[36970]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:42.657 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:44:42.657 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:42.657 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:42.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:42 smithi163 ceph-mon[36873]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:44:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:44.608 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:44 smithi164 ceph-mon[36970]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:44.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:44 smithi042 ceph-mon[32775]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:44 smithi163 ceph-mon[36873]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:45.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:45.498+0000 7f7ca97fa640 1 -- 172.21.15.164:0/1261158898 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7cac063230 con 0x7f7c940b3bd0 2024-07-26T06:44:45.499 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 8 on host 'smithi164' 2024-07-26T06:44:45.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:45.500+0000 7f7cb22a7640 1 -- 172.21.15.164:0/1261158898 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7c940b3bd0 msgr2=0x7f7c940b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:44:45.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:45.500+0000 7f7cb22a7640 1 --2- 172.21.15.164:0/1261158898 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7c940b3bd0 0x7f7c940b60e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c031440 tx=0x7f7c9c0314a0 comp rx=0 tx=0).stop 2024-07-26T06:44:45.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:45.500+0000 7f7cb22a7640 1 -- 172.21.15.164:0/1261158898 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cac104210 msgr2=0x7f7cac19a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:44:45.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:45.500+0000 7f7cb22a7640 1 --2- 172.21.15.164:0/1261158898 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cac104210 0x7f7cac19a060 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7ca000e9f0 tx=0x7f7ca000edf0 comp rx=0 tx=0).stop 2024-07-26T06:44:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:45.500+0000 7f7cb22a7640 1 -- 172.21.15.164:0/1261158898 shutdown_connections 2024-07-26T06:44:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:45.500+0000 7f7cb22a7640 1 --2- 172.21.15.164:0/1261158898 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7c940b3bd0 0x7f7c940b60e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:45.500+0000 7f7cb22a7640 1 --2- 172.21.15.164:0/1261158898 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac104bb0 0x7f7cac19ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:45.500+0000 7f7cb22a7640 1 --2- 172.21.15.164:0/1261158898 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cac104210 0x7f7cac19a060 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:45.500+0000 7f7cb22a7640 1 --2- 172.21.15.164:0/1261158898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7cac103180 0x7f7cac1999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:45.500+0000 7f7cb22a7640 1 -- 172.21.15.164:0/1261158898 >> 172.21.15.164:0/1261158898 conn(0x7f7cac0feaf0 msgr2=0x7f7cac107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:44:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:45.500+0000 7f7cb22a7640 1 -- 172.21.15.164:0/1261158898 shutdown_connections 2024-07-26T06:44:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:44:45.500+0000 7f7cb22a7640 1 -- 172.21.15.164:0/1261158898 wait complete. 2024-07-26T06:44:46.141 DEBUG:teuthology.orchestra.run.smithi164:osd.8> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@osd.8.service 2024-07-26T06:44:46.144 INFO:tasks.cephadm:Deploying osd.9 on smithi042 with /dev/vg_nvme/lv_1... 2024-07-26T06:44:46.144 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T06:44:46.430 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:44:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:46 smithi163 ceph-mon[36873]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:46 smithi163 ceph-mon[36873]: from='osd.8 [v2:172.21.15.164:6816/962845839,v1:172.21.15.164:6817/962845839]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T06:44:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:46 smithi163 ceph-mon[36873]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T06:44:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:46.820 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:46 smithi164 ceph-mon[36970]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:46.820 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:46.820 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:46.820 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:46 smithi164 ceph-mon[36970]: from='osd.8 [v2:172.21.15.164:6816/962845839,v1:172.21.15.164:6817/962845839]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T06:44:46.820 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:46 smithi164 ceph-mon[36970]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T06:44:46.820 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:46.820 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:46.877 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:46 smithi042 ceph-mon[32775]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:46.877 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:46.877 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:46.877 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:46 smithi042 ceph-mon[32775]: from='osd.8 [v2:172.21.15.164:6816/962845839,v1:172.21.15.164:6817/962845839]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T06:44:46.878 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:46 smithi042 ceph-mon[32775]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T06:44:46.878 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:46.878 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:47.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:47 smithi042 ceph-mon[32775]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T06:44:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:47 smithi042 ceph-mon[32775]: osdmap e51: 9 total, 8 up, 9 in 2024-07-26T06:44:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T06:44:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:47 smithi042 ceph-mon[32775]: from='osd.8 [v2:172.21.15.164:6816/962845839,v1:172.21.15.164:6817/962845839]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:44:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:47 smithi042 ceph-mon[32775]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:44:47.696 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:47 smithi164 ceph-mon[36970]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T06:44:47.696 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:47 smithi164 ceph-mon[36970]: osdmap e51: 9 total, 8 up, 9 in 2024-07-26T06:44:47.696 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T06:44:47.696 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:47 smithi164 ceph-mon[36970]: from='osd.8 [v2:172.21.15.164:6816/962845839,v1:172.21.15.164:6817/962845839]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:44:47.696 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:47 smithi164 ceph-mon[36970]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:44:47.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:47 smithi163 ceph-mon[36873]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T06:44:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:47 smithi163 ceph-mon[36873]: osdmap e51: 9 total, 8 up, 9 in 2024-07-26T06:44:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T06:44:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:47 smithi163 ceph-mon[36873]: from='osd.8 [v2:172.21.15.164:6816/962845839,v1:172.21.15.164:6817/962845839]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:44:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:47 smithi163 ceph-mon[36873]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:44:48.222 INFO:journalctl@ceph.osd.8.smithi164.stdout:Jul 26 06:44:47 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-8[55173]: 2024-07-26T06:44:47.955+0000 7f0f21a54640 -1 osd.8 0 waiting for initial osdmap 2024-07-26T06:44:48.222 INFO:journalctl@ceph.osd.8.smithi164.stdout:Jul 26 06:44:47 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-8[55173]: 2024-07-26T06:44:47.966+0000 7f0f1984a640 -1 osd.8 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:44:48.418 INFO:teuthology.orchestra.run.smithi042.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T06:44:48.514 INFO:teuthology.orchestra.run.smithi042.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T06:44:48.525 INFO:teuthology.orchestra.run.smithi042.stderr: stderr: 10+0 records in 2024-07-26T06:44:48.525 INFO:teuthology.orchestra.run.smithi042.stderr:10+0 records out 2024-07-26T06:44:48.525 INFO:teuthology.orchestra.run.smithi042.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00858555 s, 1.2 GB/s 2024-07-26T06:44:48.526 INFO:teuthology.orchestra.run.smithi042.stderr:--> Zapping successful for: 2024-07-26T06:44:48.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:48 smithi042 ceph-mon[32775]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:48.682 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:48 smithi042 ceph-mon[32775]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-07-26T06:44:48.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:48 smithi042 ceph-mon[32775]: osdmap e52: 9 total, 8 up, 9 in 2024-07-26T06:44:48.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T06:44:48.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T06:44:48.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:48 smithi042 ceph-mon[32775]: from='osd.8 ' entity='osd.8' 2024-07-26T06:44:48.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:48 smithi164 ceph-mon[36970]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:48.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:48 smithi164 ceph-mon[36970]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-07-26T06:44:48.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:48 smithi164 ceph-mon[36970]: osdmap e52: 9 total, 8 up, 9 in 2024-07-26T06:44:48.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T06:44:48.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T06:44:48.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:48 smithi164 ceph-mon[36970]: from='osd.8 ' entity='osd.8' 2024-07-26T06:44:49.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:48 smithi163 ceph-mon[36873]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-07-26T06:44:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:48 smithi163 ceph-mon[36873]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-07-26T06:44:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:48 smithi163 ceph-mon[36873]: osdmap e52: 9 total, 8 up, 9 in 2024-07-26T06:44:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T06:44:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T06:44:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:48 smithi163 ceph-mon[36873]: from='osd.8 ' entity='osd.8' 2024-07-26T06:44:49.296 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi042:vg_nvme/lv_1 2024-07-26T06:44:49.579 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:44:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:49 smithi042 ceph-mon[32775]: purged_snaps scrub starts 2024-07-26T06:44:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:49 smithi042 ceph-mon[32775]: purged_snaps scrub ok 2024-07-26T06:44:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:49 smithi042 ceph-mon[32775]: osd.8 [v2:172.21.15.164:6816/962845839,v1:172.21.15.164:6817/962845839] boot 2024-07-26T06:44:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:49 smithi042 ceph-mon[32775]: osdmap e53: 9 total, 9 up, 9 in 2024-07-26T06:44:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T06:44:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:49 smithi042 ceph-mon[32775]: osdmap e54: 9 total, 9 up, 9 in 2024-07-26T06:44:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:49 smithi164 ceph-mon[36970]: purged_snaps scrub starts 2024-07-26T06:44:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:49 smithi164 ceph-mon[36970]: purged_snaps scrub ok 2024-07-26T06:44:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:49 smithi164 ceph-mon[36970]: osd.8 [v2:172.21.15.164:6816/962845839,v1:172.21.15.164:6817/962845839] boot 2024-07-26T06:44:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:49 smithi164 ceph-mon[36970]: osdmap e53: 9 total, 9 up, 9 in 2024-07-26T06:44:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T06:44:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:49 smithi164 ceph-mon[36970]: osdmap e54: 9 total, 9 up, 9 in 2024-07-26T06:44:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:49 smithi163 ceph-mon[36873]: purged_snaps scrub starts 2024-07-26T06:44:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:49 smithi163 ceph-mon[36873]: purged_snaps scrub ok 2024-07-26T06:44:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:49 smithi163 ceph-mon[36873]: osd.8 [v2:172.21.15.164:6816/962845839,v1:172.21.15.164:6817/962845839] boot 2024-07-26T06:44:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:49 smithi163 ceph-mon[36873]: osdmap e53: 9 total, 9 up, 9 in 2024-07-26T06:44:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T06:44:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:49 smithi163 ceph-mon[36873]: osdmap e54: 9 total, 9 up, 9 in 2024-07-26T06:44:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:44:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:50.764 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:50 smithi042 ceph-mon[32775]: pgmap v208: 1 pgs: 1 peering; 577 KiB data, 314 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:44:50.764 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:50 smithi042 ceph-mon[32775]: Detected new or changed devices on smithi164 2024-07-26T06:44:50.764 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:50 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 5696M 2024-07-26T06:44:50.764 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:44:50.764 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:50.764 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:50.764 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:50.764 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:50 smithi042 ceph-mon[32775]: osdmap e55: 9 total, 9 up, 9 in 2024-07-26T06:44:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:50 smithi164 ceph-mon[36970]: pgmap v208: 1 pgs: 1 peering; 577 KiB data, 314 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:44:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:50 smithi164 ceph-mon[36970]: Detected new or changed devices on smithi164 2024-07-26T06:44:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:50 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 5696M 2024-07-26T06:44:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:44:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:50 smithi164 ceph-mon[36970]: osdmap e55: 9 total, 9 up, 9 in 2024-07-26T06:44:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:50 smithi163 ceph-mon[36873]: pgmap v208: 1 pgs: 1 peering; 577 KiB data, 314 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:44:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:50 smithi163 ceph-mon[36873]: Detected new or changed devices on smithi164 2024-07-26T06:44:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:50 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 5696M 2024-07-26T06:44:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:44:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:44:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:44:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:50 smithi163 ceph-mon[36873]: osdmap e55: 9 total, 9 up, 9 in 2024-07-26T06:44:51.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.147+0000 7f548718a640 1 -- 172.21.15.42:0/1741472968 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5480103180 msgr2=0x7f54801035f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:44:51.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.147+0000 7f548718a640 1 --2- 172.21.15.42:0/1741472968 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5480103180 0x7f54801035f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5474009900 tx=0x7f547402a480 comp rx=0 tx=0).stop 2024-07-26T06:44:51.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.147+0000 7f548718a640 1 -- 172.21.15.42:0/1741472968 shutdown_connections 2024-07-26T06:44:51.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.147+0000 7f548718a640 1 --2- 172.21.15.42:0/1741472968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5480104bb0 0x7f548010b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:51.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.147+0000 7f548718a640 1 --2- 172.21.15.42:0/1741472968 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5480104210 0x7f5480104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.147+0000 7f548718a640 1 --2- 172.21.15.42:0/1741472968 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5480103180 0x7f54801035f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.147+0000 7f548718a640 1 -- 172.21.15.42:0/1741472968 >> 172.21.15.42:0/1741472968 conn(0x7f54800feaf0 msgr2=0x7f5480100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:44:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.147+0000 7f548718a640 1 -- 172.21.15.42:0/1741472968 shutdown_connections 2024-07-26T06:44:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.147+0000 7f548718a640 1 -- 172.21.15.42:0/1741472968 wait complete. 2024-07-26T06:44:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.147+0000 7f548718a640 1 Processor -- start 2024-07-26T06:44:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.147+0000 7f548718a640 1 -- start start 2024-07-26T06:44:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f548718a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5480103180 0x7f5480199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:44:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f548718a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5480104210 0x7f548019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:44:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f548718a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5480104bb0 0x7f548019abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:44:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f548718a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f548010ede0 con 0x7f5480104bb0 2024-07-26T06:44:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f548718a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f548010ec60 con 0x7f5480104210 2024-07-26T06:44:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f548718a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f548010ef60 con 0x7f5480103180 2024-07-26T06:44:51.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f5485727640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5480104bb0 0x7f548019abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:44:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f5485727640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5480104bb0 0x7f548019abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40010/0 (socket says 172.21.15.42:40010) 2024-07-26T06:44:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f5485727640 1 -- 172.21.15.42:0/1172416246 learned_addr learned my addr 172.21.15.42:0/1172416246 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:44:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f5484f26640 1 --2- 172.21.15.42:0/1172416246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5480103180 0x7f5480199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:44:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f547ffff640 1 --2- 172.21.15.42:0/1172416246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5480104210 0x7f548019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:44:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f5485727640 1 -- 172.21.15.42:0/1172416246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5480103180 msgr2=0x7f5480199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:44:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f5485727640 1 --2- 172.21.15.42:0/1172416246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5480103180 0x7f5480199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f5485727640 1 -- 172.21.15.42:0/1172416246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5480104210 msgr2=0x7f548019a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:44:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f5485727640 1 --2- 172.21.15.42:0/1172416246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5480104210 0x7f548019a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:44:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f5485727640 1 -- 172.21.15.42:0/1172416246 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f548019b400 con 0x7f5480104bb0 2024-07-26T06:44:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f547ffff640 1 --2- 172.21.15.42:0/1172416246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5480104210 0x7f548019a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:44:51.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f5485727640 1 --2- 172.21.15.42:0/1172416246 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5480104bb0 0x7f548019abe0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f547800cad0 tx=0x7f547800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:44:51.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.148+0000 7f547dffb640 1 -- 172.21.15.42:0/1172416246 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547800ae20 con 0x7f5480104bb0 2024-07-26T06:44:51.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.149+0000 7f547dffb640 1 -- 172.21.15.42:0/1172416246 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5478005020 con 0x7f5480104bb0 2024-07-26T06:44:51.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.149+0000 7f547dffb640 1 -- 172.21.15.42:0/1172416246 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547800f6b0 con 0x7f5480104bb0 2024-07-26T06:44:51.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.149+0000 7f548718a640 1 -- 172.21.15.42:0/1172416246 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f548019b6f0 con 0x7f5480104bb0 2024-07-26T06:44:51.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.149+0000 7f548718a640 1 -- 172.21.15.42:0/1172416246 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54801a1df0 con 0x7f5480104bb0 2024-07-26T06:44:51.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.150+0000 7f548718a640 1 -- 172.21.15.42:0/1172416246 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f548010a360 con 0x7f5480104bb0 2024-07-26T06:44:51.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.151+0000 7f547dffb640 1 -- 172.21.15.42:0/1172416246 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f54780051d0 con 0x7f5480104bb0 2024-07-26T06:44:51.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.152+0000 7f547dffb640 1 --2- 172.21.15.42:0/1172416246 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f54700b3c50 0x7f54700b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:44:51.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.152+0000 7f5484f26640 1 --2- 172.21.15.42:0/1172416246 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f54700b3c50 0x7f54700b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:44:51.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.152+0000 7f547dffb640 1 -- 172.21.15.42:0/1172416246 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5829+0+0 (secure 0 0 0) 0x7f54780e31b0 con 0x7f5480104bb0 2024-07-26T06:44:51.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.153+0000 7f5484f26640 1 --2- 172.21.15.42:0/1172416246 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f54700b3c50 0x7f54700b6160 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5474000c00 tx=0x7f5474031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:44:51.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.156+0000 7f547dffb640 1 -- 172.21.15.42:0/1172416246 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5478014030 con 0x7f5480104bb0 2024-07-26T06:44:51.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:44:51.350+0000 7f548718a640 1 -- 172.21.15.42:0/1172416246 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f5480063230 con 0x7f54700b3c50 2024-07-26T06:44:51.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:44:51.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:44:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:44:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:44:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:52.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:44:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:44:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:44:52.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:52 smithi042 ceph-mon[32775]: pgmap v210: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:44:52.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:52 smithi042 ceph-mon[32775]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:44:52.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:52 smithi164 ceph-mon[36970]: pgmap v210: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:44:52.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:52 smithi164 ceph-mon[36970]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:44:53.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:52 smithi163 ceph-mon[36873]: pgmap v210: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:44:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:52 smithi163 ceph-mon[36873]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi042:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:44:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:54 smithi042 ceph-mon[32775]: pgmap v211: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:44:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:54 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3907712339' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f7ceac6-fb61-4c6f-9da8-851b937c4586"}]: dispatch 2024-07-26T06:44:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:54 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3907712339' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f7ceac6-fb61-4c6f-9da8-851b937c4586"}]': finished 2024-07-26T06:44:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:54 smithi042 ceph-mon[32775]: osdmap e56: 10 total, 9 up, 10 in 2024-07-26T06:44:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T06:44:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:54 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3918005847' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:44:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:54 smithi164 ceph-mon[36970]: pgmap v211: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:44:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:54 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3907712339' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f7ceac6-fb61-4c6f-9da8-851b937c4586"}]: dispatch 2024-07-26T06:44:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:54 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3907712339' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f7ceac6-fb61-4c6f-9da8-851b937c4586"}]': finished 2024-07-26T06:44:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:54 smithi164 ceph-mon[36970]: osdmap e56: 10 total, 9 up, 10 in 2024-07-26T06:44:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T06:44:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:54 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3918005847' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:44:55.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:54 smithi163 ceph-mon[36873]: pgmap v211: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:44:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:54 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3907712339' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f7ceac6-fb61-4c6f-9da8-851b937c4586"}]: dispatch 2024-07-26T06:44:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:54 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3907712339' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f7ceac6-fb61-4c6f-9da8-851b937c4586"}]': finished 2024-07-26T06:44:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:54 smithi163 ceph-mon[36873]: osdmap e56: 10 total, 9 up, 10 in 2024-07-26T06:44:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T06:44:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:54 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3918005847' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:44:56.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:56 smithi042 ceph-mon[32775]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-07-26T06:44:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:56 smithi164 ceph-mon[36970]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-07-26T06:44:57.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:56 smithi163 ceph-mon[36873]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-07-26T06:44:58.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:44:58 smithi042 ceph-mon[32775]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T06:44:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:44:58 smithi164 ceph-mon[36970]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T06:44:59.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:44:58 smithi163 ceph-mon[36873]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T06:45:00.789 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:00 smithi164 ceph-mon[36970]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-07-26T06:45:00.809 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:00 smithi163 ceph-mon[36873]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-07-26T06:45:00.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:00 smithi042 ceph-mon[32775]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-07-26T06:45:01.814 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T06:45:01.814 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T06:45:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:02.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T06:45:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:02.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:02 smithi042 ceph-mon[32775]: Deploying daemon osd.9 on smithi042 2024-07-26T06:45:02.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:02 smithi042 ceph-mon[32775]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:45:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:02 smithi164 ceph-mon[36970]: Deploying daemon osd.9 on smithi042 2024-07-26T06:45:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:02 smithi164 ceph-mon[36970]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:45:03.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:02 smithi163 ceph-mon[36873]: Deploying daemon osd.9 on smithi042 2024-07-26T06:45:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:02 smithi163 ceph-mon[36873]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:45:04.831 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:04 smithi042 ceph-mon[32775]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:45:04.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:04 smithi164 ceph-mon[36970]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:45:05.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:04 smithi163 ceph-mon[36873]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:45:06.847 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:06 smithi042 ceph-mon[32775]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-07-26T06:45:06.847 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:45:06.847 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:06.847 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:06 smithi164 ceph-mon[36970]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-07-26T06:45:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:45:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:07.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:06 smithi163 ceph-mon[36873]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-07-26T06:45:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:45:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:08.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:08 smithi042 ceph-mon[32775]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:45:08.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:08 smithi164 ceph-mon[36970]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:45:09.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:08 smithi163 ceph-mon[36873]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:45:09.760 INFO:teuthology.orchestra.run.smithi042.stdout:Created osd(s) 9 on host 'smithi042' 2024-07-26T06:45:09.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:45:09.759+0000 7f547dffb640 1 -- 172.21.15.42:0/1172416246 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5480063230 con 0x7f54700b3c50 2024-07-26T06:45:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:45:09.761+0000 7f548718a640 1 -- 172.21.15.42:0/1172416246 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f54700b3c50 msgr2=0x7f54700b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:45:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:45:09.761+0000 7f548718a640 1 --2- 172.21.15.42:0/1172416246 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f54700b3c50 0x7f54700b6160 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5474000c00 tx=0x7f5474031400 comp rx=0 tx=0).stop 2024-07-26T06:45:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:45:09.761+0000 7f548718a640 1 -- 172.21.15.42:0/1172416246 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5480104bb0 msgr2=0x7f548019abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:45:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:45:09.761+0000 7f548718a640 1 --2- 172.21.15.42:0/1172416246 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5480104bb0 0x7f548019abe0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f547800cad0 tx=0x7f547800cf90 comp rx=0 tx=0).stop 2024-07-26T06:45:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:45:09.761+0000 7f548718a640 1 -- 172.21.15.42:0/1172416246 shutdown_connections 2024-07-26T06:45:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:45:09.761+0000 7f548718a640 1 --2- 172.21.15.42:0/1172416246 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f54700b3c50 0x7f54700b6160 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:45:09.761+0000 7f548718a640 1 --2- 172.21.15.42:0/1172416246 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5480104bb0 0x7f548019abe0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:45:09.761+0000 7f548718a640 1 --2- 172.21.15.42:0/1172416246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5480104210 0x7f548019a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:09.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:45:09.761+0000 7f548718a640 1 --2- 172.21.15.42:0/1172416246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5480103180 0x7f5480199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:09.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:45:09.761+0000 7f548718a640 1 -- 172.21.15.42:0/1172416246 >> 172.21.15.42:0/1172416246 conn(0x7f54800feaf0 msgr2=0x7f5480107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:45:09.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:45:09.761+0000 7f548718a640 1 -- 172.21.15.42:0/1172416246 shutdown_connections 2024-07-26T06:45:09.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:45:09.761+0000 7f548718a640 1 -- 172.21.15.42:0/1172416246 wait complete. 2024-07-26T06:45:10.396 DEBUG:teuthology.orchestra.run.smithi042:osd.9> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@osd.9.service 2024-07-26T06:45:10.399 INFO:tasks.cephadm:Deploying osd.10 on smithi163 with /dev/vg_nvme/lv_1... 2024-07-26T06:45:10.399 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T06:45:10.663 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.b/config 2024-07-26T06:45:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:10 smithi163 ceph-mon[36873]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:45:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:10 smithi163 ceph-mon[36873]: from='osd.9 [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T06:45:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:11.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:10 smithi042 ceph-mon[32775]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:45:11.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:10 smithi042 ceph-mon[32775]: from='osd.9 [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T06:45:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:10 smithi164 ceph-mon[36970]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:45:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:10 smithi164 ceph-mon[36970]: from='osd.9 [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T06:45:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:12.043 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:11 smithi164 ceph-mon[36970]: from='osd.9 [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T06:45:12.043 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:11 smithi164 ceph-mon[36970]: osdmap e57: 10 total, 9 up, 10 in 2024-07-26T06:45:12.043 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:11 smithi164 ceph-mon[36970]: from='osd.9 [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]: dispatch 2024-07-26T06:45:12.043 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T06:45:12.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:11 smithi042 ceph-mon[32775]: from='osd.9 [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T06:45:12.052 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:11 smithi042 ceph-mon[32775]: osdmap e57: 10 total, 9 up, 10 in 2024-07-26T06:45:12.052 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:11 smithi042 ceph-mon[32775]: from='osd.9 [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]: dispatch 2024-07-26T06:45:12.052 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T06:45:12.052 INFO:journalctl@ceph.osd.9.smithi042.stdout:Jul 26 06:45:11 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-9[70160]: 2024-07-26T06:45:11.767+0000 7fcf5ee27640 -1 osd.9 0 waiting for initial osdmap 2024-07-26T06:45:12.052 INFO:journalctl@ceph.osd.9.smithi042.stdout:Jul 26 06:45:11 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-9[70160]: 2024-07-26T06:45:11.779+0000 7fcf5ac25640 -1 osd.9 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:45:12.117 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:11 smithi163 ceph-mon[36873]: from='osd.9 [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T06:45:12.117 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:11 smithi163 ceph-mon[36873]: osdmap e57: 10 total, 9 up, 10 in 2024-07-26T06:45:12.117 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:11 smithi163 ceph-mon[36873]: from='osd.9 [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]: dispatch 2024-07-26T06:45:12.117 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T06:45:12.931 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T06:45:13.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:12 smithi163 ceph-mon[36873]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:45:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:12 smithi163 ceph-mon[36873]: from='osd.9 [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-07-26T06:45:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:12 smithi163 ceph-mon[36873]: osdmap e58: 10 total, 9 up, 10 in 2024-07-26T06:45:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T06:45:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T06:45:13.011 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T06:45:13.024 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-07-26T06:45:13.024 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-07-26T06:45:13.024 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114472 s, 916 MB/s 2024-07-26T06:45:13.024 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-07-26T06:45:13.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:12 smithi042 ceph-mon[32775]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:45:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:12 smithi042 ceph-mon[32775]: from='osd.9 [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-07-26T06:45:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:12 smithi042 ceph-mon[32775]: osdmap e58: 10 total, 9 up, 10 in 2024-07-26T06:45:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T06:45:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T06:45:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:12 smithi164 ceph-mon[36970]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:45:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:12 smithi164 ceph-mon[36970]: from='osd.9 [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi042", "root=default"]}]': finished 2024-07-26T06:45:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:12 smithi164 ceph-mon[36970]: osdmap e58: 10 total, 9 up, 10 in 2024-07-26T06:45:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T06:45:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T06:45:13.828 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi163:vg_nvme/lv_1 2024-07-26T06:45:14.002 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:13 smithi042 ceph-mon[32775]: purged_snaps scrub starts 2024-07-26T06:45:14.002 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:13 smithi042 ceph-mon[32775]: purged_snaps scrub ok 2024-07-26T06:45:14.003 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:13 smithi042 ceph-mon[32775]: osd.9 [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418] boot 2024-07-26T06:45:14.003 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:13 smithi042 ceph-mon[32775]: osdmap e59: 10 total, 10 up, 10 in 2024-07-26T06:45:14.003 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T06:45:14.086 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.b/config 2024-07-26T06:45:14.209 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:13 smithi163 ceph-mon[36873]: purged_snaps scrub starts 2024-07-26T06:45:14.209 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:13 smithi163 ceph-mon[36873]: purged_snaps scrub ok 2024-07-26T06:45:14.209 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:13 smithi163 ceph-mon[36873]: osd.9 [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418] boot 2024-07-26T06:45:14.209 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:13 smithi163 ceph-mon[36873]: osdmap e59: 10 total, 10 up, 10 in 2024-07-26T06:45:14.209 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T06:45:14.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:13 smithi164 ceph-mon[36970]: purged_snaps scrub starts 2024-07-26T06:45:14.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:13 smithi164 ceph-mon[36970]: purged_snaps scrub ok 2024-07-26T06:45:14.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:13 smithi164 ceph-mon[36970]: osd.9 [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418] boot 2024-07-26T06:45:14.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:13 smithi164 ceph-mon[36970]: osdmap e59: 10 total, 10 up, 10 in 2024-07-26T06:45:14.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T06:45:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:45:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: osdmap e60: 10 total, 10 up, 10 in 2024-07-26T06:45:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: Detected new or changed devices on smithi042 2024-07-26T06:45:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 4273M 2024-07-26T06:45:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:45:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:45:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:45:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:15.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:45:15.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: osdmap e60: 10 total, 10 up, 10 in 2024-07-26T06:45:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: Detected new or changed devices on smithi042 2024-07-26T06:45:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 4273M 2024-07-26T06:45:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:45:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:45:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:45:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:15.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 279 MiB used, 804 GiB / 805 GiB avail 2024-07-26T06:45:15.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: osdmap e60: 10 total, 10 up, 10 in 2024-07-26T06:45:15.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: Detected new or changed devices on smithi042 2024-07-26T06:45:15.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:15.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:15.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:15.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:15.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:15.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:15.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 4273M 2024-07-26T06:45:15.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:15.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:15.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:45:15.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:15.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:45:15.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:15.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:45:15.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:15.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.739+0000 7fdd05ade640 1 -- 172.21.15.163:0/850214130 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd00104bb0 msgr2=0x7fdd0010b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:45:15.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.739+0000 7fdd05ade640 1 --2- 172.21.15.163:0/850214130 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd00104bb0 0x7fdd0010b550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8009ca0 tx=0x7fdcf802a950 comp rx=0 tx=0).stop 2024-07-26T06:45:15.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.739+0000 7fdd05ade640 1 -- 172.21.15.163:0/850214130 shutdown_connections 2024-07-26T06:45:15.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.739+0000 7fdd05ade640 1 --2- 172.21.15.163:0/850214130 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd00104bb0 0x7fdd0010b550 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:15.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.739+0000 7fdd05ade640 1 --2- 172.21.15.163:0/850214130 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd00104210 0x7fdd00104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:15.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.739+0000 7fdd05ade640 1 --2- 172.21.15.163:0/850214130 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd00103180 0x7fdd001035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:15.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.739+0000 7fdd05ade640 1 -- 172.21.15.163:0/850214130 >> 172.21.15.163:0/850214130 conn(0x7fdd000feaf0 msgr2=0x7fdd00100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:45:15.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.739+0000 7fdd05ade640 1 -- 172.21.15.163:0/850214130 shutdown_connections 2024-07-26T06:45:15.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.739+0000 7fdd05ade640 1 -- 172.21.15.163:0/850214130 wait complete. 2024-07-26T06:45:15.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.739+0000 7fdd05ade640 1 Processor -- start 2024-07-26T06:45:15.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.739+0000 7fdd05ade640 1 -- start start 2024-07-26T06:45:15.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdd05ade640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd00103180 0x7fdd00199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:45:15.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdd05ade640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd00104210 0x7fdd0019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:45:15.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdd05ade640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd00104bb0 0x7fdd0019ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:45:15.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdd05ade640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0010ee40 con 0x7fdd00104bb0 2024-07-26T06:45:15.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdd05ade640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0010ecc0 con 0x7fdd00104210 2024-07-26T06:45:15.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdd05ade640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0010efc0 con 0x7fdd00103180 2024-07-26T06:45:15.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdcfeffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd00104210 0x7fdd0019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:45:15.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdcfeffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd00104210 0x7fdd0019a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:39460/0 (socket says 172.21.15.163:39460) 2024-07-26T06:45:15.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdcfeffd640 1 -- 172.21.15.163:0/166143607 learned_addr learned my addr 172.21.15.163:0/166143607 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-07-26T06:45:15.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdcff7fe640 1 --2- 172.21.15.163:0/166143607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd00103180 0x7fdd00199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:45:15.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdcfffff640 1 --2- 172.21.15.163:0/166143607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd00104bb0 0x7fdd0019ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:45:15.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdcfeffd640 1 -- 172.21.15.163:0/166143607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd00103180 msgr2=0x7fdd00199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:45:15.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdcfeffd640 1 --2- 172.21.15.163:0/166143607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd00103180 0x7fdd00199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:15.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdcfeffd640 1 -- 172.21.15.163:0/166143607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd00104bb0 msgr2=0x7fdd0019ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:45:15.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdcfeffd640 1 --2- 172.21.15.163:0/166143607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd00104bb0 0x7fdd0019ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:15.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdcfeffd640 1 -- 172.21.15.163:0/166143607 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd0019b430 con 0x7fdd00104210 2024-07-26T06:45:15.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdcfffff640 1 --2- 172.21.15.163:0/166143607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd00104bb0 0x7fdd0019ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:45:15.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdcfeffd640 1 --2- 172.21.15.163:0/166143607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd00104210 0x7fdd0019a070 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdcf400d770 tx=0x7fdcf400db70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:45:15.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdcfcff9640 1 -- 172.21.15.163:0/166143607 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4017070 con 0x7fdd00104210 2024-07-26T06:45:15.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdd05ade640 1 -- 172.21.15.163:0/166143607 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd0019b720 con 0x7fdd00104210 2024-07-26T06:45:15.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdcfcff9640 1 -- 172.21.15.163:0/166143607 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcf4009820 con 0x7fdd00104210 2024-07-26T06:45:15.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.740+0000 7fdcfcff9640 1 -- 172.21.15.163:0/166143607 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4004bc0 con 0x7fdd00104210 2024-07-26T06:45:15.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.741+0000 7fdd05ade640 1 -- 172.21.15.163:0/166143607 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd001a1e20 con 0x7fdd00104210 2024-07-26T06:45:15.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.741+0000 7fdd05ade640 1 -- 172.21.15.163:0/166143607 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd001035f0 con 0x7fdd00104210 2024-07-26T06:45:15.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.743+0000 7fdcfcff9640 1 -- 172.21.15.163:0/166143607 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fdcf40099d0 con 0x7fdd00104210 2024-07-26T06:45:15.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.744+0000 7fdcfcff9640 1 --2- 172.21.15.163:0/166143607 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdce80b3bd0 0x7fdce80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:45:15.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.744+0000 7fdcfcff9640 1 -- 172.21.15.163:0/166143607 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6261+0+0 (secure 0 0 0) 0x7fdcf40e3320 con 0x7fdd00104210 2024-07-26T06:45:15.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.744+0000 7fdcff7fe640 1 --2- 172.21.15.163:0/166143607 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdce80b3bd0 0x7fdce80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:45:15.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.744+0000 7fdcff7fe640 1 --2- 172.21.15.163:0/166143607 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdce80b3bd0 0x7fdce80b60e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0009990 tx=0x7fdcf0009110 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:45:15.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.747+0000 7fdcfcff9640 1 -- 172.21.15.163:0/166143607 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdcf40ab170 con 0x7fdd00104210 2024-07-26T06:45:15.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:15.930+0000 7fdd05ade640 1 -- 172.21.15.163:0/166143607 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fdd00063230 con 0x7fdce80b3bd0 2024-07-26T06:45:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:16 smithi163 ceph-mon[36873]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:16 smithi163 ceph-mon[36873]: from='client.24485 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:45:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:45:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:45:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:17.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:16 smithi042 ceph-mon[32775]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:16 smithi042 ceph-mon[32775]: from='client.24485 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:45:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:45:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:45:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:16 smithi164 ceph-mon[36970]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:16 smithi164 ceph-mon[36970]: from='client.24485 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:45:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:45:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:45:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:18.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:18 smithi163 ceph-mon[36873]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:18.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:18 smithi163 ceph-mon[36873]: from='client.? 172.21.15.163:0/1716453220' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "553132c2-a6a1-410c-a7d8-ecdf9a67dafb"}]: dispatch 2024-07-26T06:45:18.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:18 smithi163 ceph-mon[36873]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "553132c2-a6a1-410c-a7d8-ecdf9a67dafb"}]: dispatch 2024-07-26T06:45:18.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:18 smithi163 ceph-mon[36873]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "553132c2-a6a1-410c-a7d8-ecdf9a67dafb"}]': finished 2024-07-26T06:45:18.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:18 smithi163 ceph-mon[36873]: osdmap e61: 11 total, 10 up, 11 in 2024-07-26T06:45:18.974 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:19.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:18 smithi042 ceph-mon[32775]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:18 smithi042 ceph-mon[32775]: from='client.? 172.21.15.163:0/1716453220' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "553132c2-a6a1-410c-a7d8-ecdf9a67dafb"}]: dispatch 2024-07-26T06:45:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:18 smithi042 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "553132c2-a6a1-410c-a7d8-ecdf9a67dafb"}]: dispatch 2024-07-26T06:45:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:18 smithi042 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "553132c2-a6a1-410c-a7d8-ecdf9a67dafb"}]': finished 2024-07-26T06:45:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:18 smithi042 ceph-mon[32775]: osdmap e61: 11 total, 10 up, 11 in 2024-07-26T06:45:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:18 smithi164 ceph-mon[36970]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:18 smithi164 ceph-mon[36970]: from='client.? 172.21.15.163:0/1716453220' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "553132c2-a6a1-410c-a7d8-ecdf9a67dafb"}]: dispatch 2024-07-26T06:45:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:18 smithi164 ceph-mon[36970]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "553132c2-a6a1-410c-a7d8-ecdf9a67dafb"}]: dispatch 2024-07-26T06:45:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:18 smithi164 ceph-mon[36970]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "553132c2-a6a1-410c-a7d8-ecdf9a67dafb"}]': finished 2024-07-26T06:45:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:18 smithi164 ceph-mon[36970]: osdmap e61: 11 total, 10 up, 11 in 2024-07-26T06:45:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:20.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:19 smithi042 ceph-mon[32775]: from='client.? 172.21.15.163:0/4275242277' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:45:20.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:19 smithi164 ceph-mon[36970]: from='client.? 172.21.15.163:0/4275242277' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:45:20.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:19 smithi163 ceph-mon[36873]: from='client.? 172.21.15.163:0/4275242277' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:45:21.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:20 smithi042 ceph-mon[32775]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:20 smithi164 ceph-mon[36970]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:21.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:20 smithi163 ceph-mon[36873]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:23.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:22 smithi163 ceph-mon[36873]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:23.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:22 smithi042 ceph-mon[32775]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:23.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:22 smithi164 ceph-mon[36970]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 306 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:24.929 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:24 smithi163 ceph-mon[36873]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:24 smithi042 ceph-mon[32775]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:25.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:24 smithi164 ceph-mon[36970]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T06:45:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:26.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T06:45:26.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:26.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T06:45:26.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:27.103 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:26 smithi163 ceph-mon[36873]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:27.103 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:26 smithi163 ceph-mon[36873]: Deploying daemon osd.10 on smithi163 2024-07-26T06:45:27.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:26 smithi042 ceph-mon[32775]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:27.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:26 smithi042 ceph-mon[32775]: Deploying daemon osd.10 on smithi163 2024-07-26T06:45:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:26 smithi164 ceph-mon[36970]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:26 smithi164 ceph-mon[36970]: Deploying daemon osd.10 on smithi163 2024-07-26T06:45:29.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:28 smithi163 ceph-mon[36873]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:29.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:28 smithi042 ceph-mon[32775]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:29.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:28 smithi164 ceph-mon[36970]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:30.958 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:30 smithi163 ceph-mon[36873]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:30 smithi042 ceph-mon[32775]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:31.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:30 smithi164 ceph-mon[36970]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:45:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:32.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:45:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:45:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:33.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:32 smithi042 ceph-mon[32775]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:32 smithi164 ceph-mon[36970]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:32 smithi163 ceph-mon[36873]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:34.862 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 10 on host 'smithi163' 2024-07-26T06:45:34.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:34.861+0000 7fdcfcff9640 1 -- 172.21.15.163:0/166143607 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fdd00063230 con 0x7fdce80b3bd0 2024-07-26T06:45:34.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:34.864+0000 7fdd05ade640 1 -- 172.21.15.163:0/166143607 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdce80b3bd0 msgr2=0x7fdce80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:45:34.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:34.864+0000 7fdd05ade640 1 --2- 172.21.15.163:0/166143607 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdce80b3bd0 0x7fdce80b60e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0009990 tx=0x7fdcf0009110 comp rx=0 tx=0).stop 2024-07-26T06:45:34.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:34.864+0000 7fdd05ade640 1 -- 172.21.15.163:0/166143607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd00104210 msgr2=0x7fdd0019a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:45:34.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:34.864+0000 7fdd05ade640 1 --2- 172.21.15.163:0/166143607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd00104210 0x7fdd0019a070 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdcf400d770 tx=0x7fdcf400db70 comp rx=0 tx=0).stop 2024-07-26T06:45:34.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:34.864+0000 7fdd05ade640 1 -- 172.21.15.163:0/166143607 shutdown_connections 2024-07-26T06:45:34.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:34.864+0000 7fdd05ade640 1 --2- 172.21.15.163:0/166143607 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdce80b3bd0 0x7fdce80b60e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:34.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:34.864+0000 7fdd05ade640 1 --2- 172.21.15.163:0/166143607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd00104bb0 0x7fdd0019ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:34.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:34.864+0000 7fdd05ade640 1 --2- 172.21.15.163:0/166143607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd00104210 0x7fdd0019a070 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:34.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:34.864+0000 7fdd05ade640 1 --2- 172.21.15.163:0/166143607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd00103180 0x7fdd00199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:34.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:34.864+0000 7fdd05ade640 1 -- 172.21.15.163:0/166143607 >> 172.21.15.163:0/166143607 conn(0x7fdd000feaf0 msgr2=0x7fdd00107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:45:34.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:34.865+0000 7fdd05ade640 1 -- 172.21.15.163:0/166143607 shutdown_connections 2024-07-26T06:45:34.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-07-26T06:45:34.865+0000 7fdd05ade640 1 -- 172.21.15.163:0/166143607 wait complete. 2024-07-26T06:45:35.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:34 smithi163 ceph-mon[36873]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:35.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:35.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:35.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:35.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:45:35.161 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:35.161 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:45:35.161 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:35.161 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:34 smithi164 ceph-mon[36970]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:45:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:45:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:34 smithi042 ceph-mon[32775]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:45:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:45:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:35.552 DEBUG:teuthology.orchestra.run.smithi163:osd.10> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@osd.10.service 2024-07-26T06:45:35.554 INFO:tasks.cephadm:Deploying osd.11 on smithi164 with /dev/vg_nvme/lv_1... 2024-07-26T06:45:35.554 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T06:45:35.816 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T06:45:36.189 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:35 smithi164 ceph-mon[36970]: from='osd.10 [v2:172.21.15.163:6824/3268512227,v1:172.21.15.163:6825/3268512227]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T06:45:36.189 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:35 smithi164 ceph-mon[36970]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T06:45:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:35 smithi163 ceph-mon[36873]: from='osd.10 [v2:172.21.15.163:6824/3268512227,v1:172.21.15.163:6825/3268512227]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T06:45:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:35 smithi163 ceph-mon[36873]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T06:45:36.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:35 smithi042 ceph-mon[32775]: from='osd.10 [v2:172.21.15.163:6824/3268512227,v1:172.21.15.163:6825/3268512227]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T06:45:36.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:35 smithi042 ceph-mon[32775]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T06:45:37.213 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:36 smithi164 ceph-mon[36970]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:37.213 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:36 smithi164 ceph-mon[36970]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T06:45:37.213 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:36 smithi164 ceph-mon[36970]: osdmap e62: 11 total, 10 up, 11 in 2024-07-26T06:45:37.213 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:37.213 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:36 smithi164 ceph-mon[36970]: from='osd.10 [v2:172.21.15.163:6824/3268512227,v1:172.21.15.163:6825/3268512227]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:45:37.213 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:36 smithi164 ceph-mon[36970]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:45:37.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:36 smithi163 ceph-mon[36873]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:37.220 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:36 smithi163 ceph-mon[36873]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T06:45:37.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:36 smithi163 ceph-mon[36873]: osdmap e62: 11 total, 10 up, 11 in 2024-07-26T06:45:37.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:37.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:36 smithi163 ceph-mon[36873]: from='osd.10 [v2:172.21.15.163:6824/3268512227,v1:172.21.15.163:6825/3268512227]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:45:37.221 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:36 smithi163 ceph-mon[36873]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:45:37.221 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jul 26 06:45:36 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-10[61851]: 2024-07-26T06:45:36.963+0000 7f3538374640 -1 osd.10 0 waiting for initial osdmap 2024-07-26T06:45:37.221 INFO:journalctl@ceph.osd.10.smithi163.stdout:Jul 26 06:45:36 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-10[61851]: 2024-07-26T06:45:36.975+0000 7f3534172640 -1 osd.10 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:45:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:36 smithi042 ceph-mon[32775]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:36 smithi042 ceph-mon[32775]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T06:45:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:36 smithi042 ceph-mon[32775]: osdmap e62: 11 total, 10 up, 11 in 2024-07-26T06:45:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:36 smithi042 ceph-mon[32775]: from='osd.10 [v2:172.21.15.163:6824/3268512227,v1:172.21.15.163:6825/3268512227]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:45:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:36 smithi042 ceph-mon[32775]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-07-26T06:45:37.697 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T06:45:37.772 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T06:45:37.786 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2024-07-26T06:45:37.786 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2024-07-26T06:45:37.786 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0118848 s, 882 MB/s 2024-07-26T06:45:37.786 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2024-07-26T06:45:38.155 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:37 smithi164 ceph-mon[36970]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-07-26T06:45:38.155 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:37 smithi164 ceph-mon[36970]: osdmap e63: 11 total, 10 up, 11 in 2024-07-26T06:45:38.156 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:38.156 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:37 smithi163 ceph-mon[36873]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-07-26T06:45:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:37 smithi163 ceph-mon[36873]: osdmap e63: 11 total, 10 up, 11 in 2024-07-26T06:45:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:38.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:37 smithi042 ceph-mon[32775]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-07-26T06:45:38.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:37 smithi042 ceph-mon[32775]: osdmap e63: 11 total, 10 up, 11 in 2024-07-26T06:45:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:38.487 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch daemon add osd smithi164:vg_nvme/lv_1 2024-07-26T06:45:38.744 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T06:45:39.016 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:38 smithi164 ceph-mon[36970]: purged_snaps scrub starts 2024-07-26T06:45:39.017 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:38 smithi164 ceph-mon[36970]: purged_snaps scrub ok 2024-07-26T06:45:39.017 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:38 smithi164 ceph-mon[36970]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:39.017 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:39.017 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:38 smithi164 ceph-mon[36970]: osd.10 [v2:172.21.15.163:6824/3268512227,v1:172.21.15.163:6825/3268512227] boot 2024-07-26T06:45:39.017 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:38 smithi164 ceph-mon[36970]: osdmap e64: 11 total, 11 up, 11 in 2024-07-26T06:45:39.017 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:39.017 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:39.017 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:39.017 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:39.017 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:39.017 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:39.017 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:39.018 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:39.018 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:39.018 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:45:39.018 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:38 smithi163 ceph-mon[36873]: purged_snaps scrub starts 2024-07-26T06:45:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:38 smithi163 ceph-mon[36873]: purged_snaps scrub ok 2024-07-26T06:45:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:38 smithi163 ceph-mon[36873]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:38 smithi163 ceph-mon[36873]: osd.10 [v2:172.21.15.163:6824/3268512227,v1:172.21.15.163:6825/3268512227] boot 2024-07-26T06:45:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:38 smithi163 ceph-mon[36873]: osdmap e64: 11 total, 11 up, 11 in 2024-07-26T06:45:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:45:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:38 smithi042 ceph-mon[32775]: purged_snaps scrub starts 2024-07-26T06:45:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:38 smithi042 ceph-mon[32775]: purged_snaps scrub ok 2024-07-26T06:45:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:38 smithi042 ceph-mon[32775]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-07-26T06:45:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:38 smithi042 ceph-mon[32775]: osd.10 [v2:172.21.15.163:6824/3268512227,v1:172.21.15.163:6825/3268512227] boot 2024-07-26T06:45:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:38 smithi042 ceph-mon[32775]: osdmap e64: 11 total, 11 up, 11 in 2024-07-26T06:45:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T06:45:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:45:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:45:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:40.177 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:39 smithi164 ceph-mon[36970]: Detected new or changed devices on smithi163 2024-07-26T06:45:40.177 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:39 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 4272M 2024-07-26T06:45:40.177 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:39 smithi164 ceph-mon[36970]: osdmap e65: 11 total, 11 up, 11 in 2024-07-26T06:45:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:39 smithi163 ceph-mon[36873]: Detected new or changed devices on smithi163 2024-07-26T06:45:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:39 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 4272M 2024-07-26T06:45:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:39 smithi163 ceph-mon[36873]: osdmap e65: 11 total, 11 up, 11 in 2024-07-26T06:45:40.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:39 smithi042 ceph-mon[32775]: Detected new or changed devices on smithi163 2024-07-26T06:45:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:39 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 4272M 2024-07-26T06:45:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:39 smithi042 ceph-mon[32775]: osdmap e65: 11 total, 11 up, 11 in 2024-07-26T06:45:40.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.400+0000 7f7541df4640 1 -- 172.21.15.164:0/1396156352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f753c103130 msgr2=0x7f753c103540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:45:40.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.400+0000 7f7541df4640 1 --2- 172.21.15.164:0/1396156352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f753c103130 0x7f753c103540 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7530009900 tx=0x7f753002a480 comp rx=0 tx=0).stop 2024-07-26T06:45:40.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.400+0000 7f7541df4640 1 -- 172.21.15.164:0/1396156352 shutdown_connections 2024-07-26T06:45:40.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.400+0000 7f7541df4640 1 --2- 172.21.15.164:0/1396156352 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f753c103ad0 0x7f753c10a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:40.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.400+0000 7f7541df4640 1 --2- 172.21.15.164:0/1396156352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f753c103130 0x7f753c103540 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:40.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.400+0000 7f7541df4640 1 --2- 172.21.15.164:0/1396156352 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f753c1020a0 0x7f753c102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:40.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.400+0000 7f7541df4640 1 -- 172.21.15.164:0/1396156352 >> 172.21.15.164:0/1396156352 conn(0x7f753c0fda10 msgr2=0x7f753c0ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:45:40.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.400+0000 7f7541df4640 1 -- 172.21.15.164:0/1396156352 shutdown_connections 2024-07-26T06:45:40.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.400+0000 7f7541df4640 1 -- 172.21.15.164:0/1396156352 wait complete. 2024-07-26T06:45:40.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.400+0000 7f7541df4640 1 Processor -- start 2024-07-26T06:45:40.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f7541df4640 1 -- start start 2024-07-26T06:45:40.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f7541df4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f753c1020a0 0x7f753c199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:45:40.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f7541df4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f753c103130 0x7f753c19a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:45:40.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f7541df4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f753c103ad0 0x7f753c19ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:45:40.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f7541df4640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f753c10ee50 con 0x7f753c103130 2024-07-26T06:45:40.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f7541df4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f753c10ecd0 con 0x7f753c103ad0 2024-07-26T06:45:40.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f7541df4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f753c10efd0 con 0x7f753c1020a0 2024-07-26T06:45:40.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f753b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f753c1020a0 0x7f753c199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:45:40.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f753b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f753c1020a0 0x7f753c199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56780/0 (socket says 172.21.15.164:56780) 2024-07-26T06:45:40.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f753b7fe640 1 -- 172.21.15.164:0/1356241163 learned_addr learned my addr 172.21.15.164:0/1356241163 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-07-26T06:45:40.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f753affd640 1 --2- 172.21.15.164:0/1356241163 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f753c103130 0x7f753c19a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:45:40.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f753bfff640 1 --2- 172.21.15.164:0/1356241163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f753c103ad0 0x7f753c19ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:45:40.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f753b7fe640 1 -- 172.21.15.164:0/1356241163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f753c103ad0 msgr2=0x7f753c19ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:45:40.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f753b7fe640 1 --2- 172.21.15.164:0/1356241163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f753c103ad0 0x7f753c19ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:40.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f753b7fe640 1 -- 172.21.15.164:0/1356241163 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f753c103130 msgr2=0x7f753c19a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:45:40.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f753b7fe640 1 --2- 172.21.15.164:0/1356241163 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f753c103130 0x7f753c19a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:40.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f753b7fe640 1 -- 172.21.15.164:0/1356241163 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753c19b4b0 con 0x7f753c1020a0 2024-07-26T06:45:40.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f753affd640 1 --2- 172.21.15.164:0/1356241163 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f753c103130 0x7f753c19a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:45:40.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f753b7fe640 1 --2- 172.21.15.164:0/1356241163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f753c1020a0 0x7f753c199a80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f752c00b4a0 tx=0x7f752c00b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:45:40.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.401+0000 7f7538ff9640 1 -- 172.21.15.164:0/1356241163 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752c014070 con 0x7f753c1020a0 2024-07-26T06:45:40.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.402+0000 7f7538ff9640 1 -- 172.21.15.164:0/1356241163 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f752c010ac0 con 0x7f753c1020a0 2024-07-26T06:45:40.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.402+0000 7f7541df4640 1 -- 172.21.15.164:0/1356241163 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f753c19b7a0 con 0x7f753c1020a0 2024-07-26T06:45:40.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.402+0000 7f7538ff9640 1 -- 172.21.15.164:0/1356241163 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752c017c30 con 0x7f753c1020a0 2024-07-26T06:45:40.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.402+0000 7f7541df4640 1 -- 172.21.15.164:0/1356241163 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f753c1a1ea0 con 0x7f753c1020a0 2024-07-26T06:45:40.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.403+0000 7f7541df4640 1 -- 172.21.15.164:0/1356241163 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f753c102510 con 0x7f753c1020a0 2024-07-26T06:45:40.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.404+0000 7f7538ff9640 1 -- 172.21.15.164:0/1356241163 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f752c017070 con 0x7f753c1020a0 2024-07-26T06:45:40.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.405+0000 7f7538ff9640 1 --2- 172.21.15.164:0/1356241163 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f75240b3bd0 0x7f75240b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:45:40.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.405+0000 7f7538ff9640 1 -- 172.21.15.164:0/1356241163 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 6694+0+0 (secure 0 0 0) 0x7f752c0e2d40 con 0x7f753c1020a0 2024-07-26T06:45:40.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.405+0000 7f753affd640 1 --2- 172.21.15.164:0/1356241163 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f75240b3bd0 0x7f75240b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:45:40.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.406+0000 7f753affd640 1 --2- 172.21.15.164:0/1356241163 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f75240b3bd0 0x7f75240b60e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7530031440 tx=0x7f7530005d50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:45:40.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.408+0000 7f7538ff9640 1 -- 172.21.15.164:0/1356241163 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f752c0aa9e0 con 0x7f753c1020a0 2024-07-26T06:45:40.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:40.591+0000 7f7541df4640 1 -- 172.21.15.164:0/1356241163 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f753c063230 con 0x7f75240b3bd0 2024-07-26T06:45:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:40 smithi163 ceph-mon[36873]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:45:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:40 smithi163 ceph-mon[36873]: osdmap e66: 11 total, 11 up, 11 in 2024-07-26T06:45:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:45:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:45:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:41.262 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:40 smithi164 ceph-mon[36970]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:45:41.262 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:40 smithi164 ceph-mon[36970]: osdmap e66: 11 total, 11 up, 11 in 2024-07-26T06:45:41.262 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:45:41.262 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:45:41.263 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:41.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:40 smithi042 ceph-mon[32775]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:45:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:40 smithi042 ceph-mon[32775]: osdmap e66: 11 total, 11 up, 11 in 2024-07-26T06:45:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T06:45:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T06:45:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:42.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:41 smithi163 ceph-mon[36873]: from='client.24484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:45:42.315 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:41 smithi164 ceph-mon[36970]: from='client.24484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:45:42.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:41 smithi042 ceph-mon[32775]: from='client.24484 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:45:43.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:42 smithi163 ceph-mon[36873]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:45:43.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:42 smithi164 ceph-mon[36970]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:45:43.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:42 smithi042 ceph-mon[32775]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:45:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:43 smithi163 ceph-mon[36873]: from='client.? 172.21.15.164:0/2046228308' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09abc8e9-7884-400f-ab6d-8e5aa7bf31ff"}]: dispatch 2024-07-26T06:45:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:43 smithi163 ceph-mon[36873]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09abc8e9-7884-400f-ab6d-8e5aa7bf31ff"}]: dispatch 2024-07-26T06:45:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:43 smithi163 ceph-mon[36873]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09abc8e9-7884-400f-ab6d-8e5aa7bf31ff"}]': finished 2024-07-26T06:45:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:43 smithi163 ceph-mon[36873]: osdmap e67: 12 total, 11 up, 12 in 2024-07-26T06:45:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T06:45:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:43 smithi163 ceph-mon[36873]: from='client.? 172.21.15.164:0/163769354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:45:44.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:43 smithi042 ceph-mon[32775]: from='client.? 172.21.15.164:0/2046228308' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09abc8e9-7884-400f-ab6d-8e5aa7bf31ff"}]: dispatch 2024-07-26T06:45:44.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:43 smithi042 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09abc8e9-7884-400f-ab6d-8e5aa7bf31ff"}]: dispatch 2024-07-26T06:45:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:43 smithi042 ceph-mon[32775]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09abc8e9-7884-400f-ab6d-8e5aa7bf31ff"}]': finished 2024-07-26T06:45:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:43 smithi042 ceph-mon[32775]: osdmap e67: 12 total, 11 up, 12 in 2024-07-26T06:45:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T06:45:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:43 smithi042 ceph-mon[32775]: from='client.? 172.21.15.164:0/163769354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:45:44.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:43 smithi164 ceph-mon[36970]: from='client.? 172.21.15.164:0/2046228308' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09abc8e9-7884-400f-ab6d-8e5aa7bf31ff"}]: dispatch 2024-07-26T06:45:44.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:43 smithi164 ceph-mon[36970]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09abc8e9-7884-400f-ab6d-8e5aa7bf31ff"}]: dispatch 2024-07-26T06:45:44.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:43 smithi164 ceph-mon[36970]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09abc8e9-7884-400f-ab6d-8e5aa7bf31ff"}]': finished 2024-07-26T06:45:44.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:43 smithi164 ceph-mon[36970]: osdmap e67: 12 total, 11 up, 12 in 2024-07-26T06:45:44.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T06:45:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:43 smithi164 ceph-mon[36970]: from='client.? 172.21.15.164:0/163769354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T06:45:45.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:45 smithi042 ceph-mon[32775]: pgmap v248: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:45:45.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:45 smithi164 ceph-mon[36970]: pgmap v248: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:45:45.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:45 smithi163 ceph-mon[36873]: pgmap v248: 1 pgs: 1 peering; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:45:47.368 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:47 smithi164 ceph-mon[36970]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-07-26T06:45:47.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:47 smithi042 ceph-mon[32775]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-07-26T06:45:47.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:47 smithi163 ceph-mon[36873]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-07-26T06:45:49.321 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:49 smithi164 ceph-mon[36970]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T06:45:49.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:49 smithi042 ceph-mon[32775]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T06:45:49.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:49 smithi163 ceph-mon[36873]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T06:45:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:51 smithi164 ceph-mon[36970]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-07-26T06:45:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T06:45:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:51.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:51 smithi042 ceph-mon[32775]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-07-26T06:45:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T06:45:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:51.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:51 smithi163 ceph-mon[36873]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-07-26T06:45:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T06:45:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:52.183 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:52 smithi164 ceph-mon[36970]: Deploying daemon osd.11 on smithi164 2024-07-26T06:45:52.323 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:52 smithi042 ceph-mon[32775]: Deploying daemon osd.11 on smithi164 2024-07-26T06:45:52.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:52 smithi163 ceph-mon[36873]: Deploying daemon osd.11 on smithi164 2024-07-26T06:45:53.191 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:53 smithi164 ceph-mon[36970]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:45:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:53 smithi042 ceph-mon[32775]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:45:53.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:53 smithi163 ceph-mon[36873]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T06:45:55.318 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:55 smithi164 ceph-mon[36970]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-07-26T06:45:55.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:55 smithi042 ceph-mon[32775]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-07-26T06:45:55.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:55 smithi163 ceph-mon[36873]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-07-26T06:45:56.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:45:56.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:56.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:45:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:56.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:45:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:57 smithi164 ceph-mon[36970]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-07-26T06:45:57.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:57 smithi042 ceph-mon[32775]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-07-26T06:45:57.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:57 smithi163 ceph-mon[36873]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-07-26T06:45:59.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:59.140+0000 7f7538ff9640 1 -- 172.21.15.164:0/1356241163 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f753c063230 con 0x7f75240b3bd0 2024-07-26T06:45:59.142 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 11 on host 'smithi164' 2024-07-26T06:45:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:59.142+0000 7f7541df4640 1 -- 172.21.15.164:0/1356241163 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f75240b3bd0 msgr2=0x7f75240b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:45:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:59.142+0000 7f7541df4640 1 --2- 172.21.15.164:0/1356241163 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f75240b3bd0 0x7f75240b60e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7530031440 tx=0x7f7530005d50 comp rx=0 tx=0).stop 2024-07-26T06:45:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:59.142+0000 7f7541df4640 1 -- 172.21.15.164:0/1356241163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f753c1020a0 msgr2=0x7f753c199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:45:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:59.142+0000 7f7541df4640 1 --2- 172.21.15.164:0/1356241163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f753c1020a0 0x7f753c199a80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f752c00b4a0 tx=0x7f752c00b960 comp rx=0 tx=0).stop 2024-07-26T06:45:59.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:59.142+0000 7f7541df4640 1 -- 172.21.15.164:0/1356241163 shutdown_connections 2024-07-26T06:45:59.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:59.142+0000 7f7541df4640 1 --2- 172.21.15.164:0/1356241163 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f75240b3bd0 0x7f75240b60e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:59.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:59.142+0000 7f7541df4640 1 --2- 172.21.15.164:0/1356241163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f753c103ad0 0x7f753c19ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:59.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:59.142+0000 7f7541df4640 1 --2- 172.21.15.164:0/1356241163 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f753c103130 0x7f753c19a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:59.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:59.142+0000 7f7541df4640 1 --2- 172.21.15.164:0/1356241163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f753c1020a0 0x7f753c199a80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:45:59.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:59.142+0000 7f7541df4640 1 -- 172.21.15.164:0/1356241163 >> 172.21.15.164:0/1356241163 conn(0x7f753c0fda10 msgr2=0x7f753c106450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:45:59.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:59.143+0000 7f7541df4640 1 -- 172.21.15.164:0/1356241163 shutdown_connections 2024-07-26T06:45:59.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:45:59.143+0000 7f7541df4640 1 -- 172.21.15.164:0/1356241163 wait complete. 2024-07-26T06:45:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:59 smithi042 ceph-mon[32775]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:45:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:45:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:45:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:59.462 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:59 smithi164 ceph-mon[36970]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:45:59.462 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:59.462 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:59.462 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:59.462 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:45:59.462 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:45:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:59.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:59 smithi163 ceph-mon[36873]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:45:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:45:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:45:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:45:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:45:59.889 DEBUG:teuthology.orchestra.run.smithi164:osd.11> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@osd.11.service 2024-07-26T06:45:59.892 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-07-26T06:45:59.892 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd stat -f json 2024-07-26T06:46:00.174 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:46:00.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:46:00.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:00.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:00.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:00 smithi164 ceph-mon[36970]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:46:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:00 smithi164 ceph-mon[36970]: from='osd.11 [v2:172.21.15.164:6824/1861076336,v1:172.21.15.164:6825/1861076336]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T06:46:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:00 smithi164 ceph-mon[36970]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T06:46:00.286 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:46:00.287 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:00.287 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:00.287 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:00 smithi042 ceph-mon[32775]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:46:00.287 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:00 smithi042 ceph-mon[32775]: from='osd.11 [v2:172.21.15.164:6824/1861076336,v1:172.21.15.164:6825/1861076336]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T06:46:00.287 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:00 smithi042 ceph-mon[32775]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T06:46:00.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:46:00.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:00.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:00.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:00 smithi163 ceph-mon[36873]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:46:00.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:00 smithi163 ceph-mon[36873]: from='osd.11 [v2:172.21.15.164:6824/1861076336,v1:172.21.15.164:6825/1861076336]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T06:46:00.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:00 smithi163 ceph-mon[36873]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T06:46:01.364 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:01 smithi164 ceph-mon[36970]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T06:46:01.364 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:01 smithi164 ceph-mon[36970]: osdmap e68: 12 total, 11 up, 12 in 2024-07-26T06:46:01.364 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T06:46:01.364 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:01 smithi164 ceph-mon[36970]: from='osd.11 [v2:172.21.15.164:6824/1861076336,v1:172.21.15.164:6825/1861076336]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:46:01.364 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:01 smithi164 ceph-mon[36970]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:46:01.365 INFO:journalctl@ceph.osd.11.smithi164.stdout:Jul 26 06:46:01 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-11[61755]: 2024-07-26T06:46:01.153+0000 7fb303b15640 -1 osd.11 0 waiting for initial osdmap 2024-07-26T06:46:01.365 INFO:journalctl@ceph.osd.11.smithi164.stdout:Jul 26 06:46:01 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-osd-11[61755]: 2024-07-26T06:46:01.165+0000 7fb2ff913640 -1 osd.11 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T06:46:01.411 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:01 smithi042 ceph-mon[32775]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T06:46:01.412 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:01 smithi042 ceph-mon[32775]: osdmap e68: 12 total, 11 up, 12 in 2024-07-26T06:46:01.412 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T06:46:01.412 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:01 smithi042 ceph-mon[32775]: from='osd.11 [v2:172.21.15.164:6824/1861076336,v1:172.21.15.164:6825/1861076336]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:46:01.412 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:01 smithi042 ceph-mon[32775]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:46:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:01 smithi163 ceph-mon[36873]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T06:46:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:01 smithi163 ceph-mon[36873]: osdmap e68: 12 total, 11 up, 12 in 2024-07-26T06:46:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T06:46:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:01 smithi163 ceph-mon[36873]: from='osd.11 [v2:172.21.15.164:6824/1861076336,v1:172.21.15.164:6825/1861076336]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:46:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:01 smithi163 ceph-mon[36873]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-07-26T06:46:01.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.934+0000 7f6bf7b2d640 1 -- 172.21.15.42:0/1635194034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bf0103150 msgr2=0x7f6bf0103560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:01.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.934+0000 7f6bf7b2d640 1 --2- 172.21.15.42:0/1635194034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bf0103150 0x7f6bf0103560 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6be4009900 tx=0x7f6be402a480 comp rx=0 tx=0).stop 2024-07-26T06:46:01.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.934+0000 7f6bf7b2d640 1 -- 172.21.15.42:0/1635194034 shutdown_connections 2024-07-26T06:46:01.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.934+0000 7f6bf7b2d640 1 --2- 172.21.15.42:0/1635194034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6bf0103af0 0x7f6bf010a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:01.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.934+0000 7f6bf7b2d640 1 --2- 172.21.15.42:0/1635194034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bf0103150 0x7f6bf0103560 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:01.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.934+0000 7f6bf7b2d640 1 --2- 172.21.15.42:0/1635194034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bf01020c0 0x7f6bf0102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:01.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.934+0000 7f6bf7b2d640 1 -- 172.21.15.42:0/1635194034 >> 172.21.15.42:0/1635194034 conn(0x7f6bf00fda30 msgr2=0x7f6bf00ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:01.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.934+0000 7f6bf7b2d640 1 -- 172.21.15.42:0/1635194034 shutdown_connections 2024-07-26T06:46:01.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.934+0000 7f6bf7b2d640 1 -- 172.21.15.42:0/1635194034 wait complete. 2024-07-26T06:46:01.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.935+0000 7f6bf7b2d640 1 Processor -- start 2024-07-26T06:46:01.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.935+0000 7f6bf7b2d640 1 -- start start 2024-07-26T06:46:01.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.935+0000 7f6bf7b2d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6bf01020c0 0x7f6bf0199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:01.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.935+0000 7f6bf7b2d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bf0103150 0x7f6bf019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:01.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.935+0000 7f6bf7b2d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bf0103af0 0x7f6bf019ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:01.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.935+0000 7f6bf7b2d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf010dd20 con 0x7f6bf0103150 2024-07-26T06:46:01.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.935+0000 7f6bf7b2d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf010dba0 con 0x7f6bf0103af0 2024-07-26T06:46:01.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.935+0000 7f6bf7b2d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf010dea0 con 0x7f6bf01020c0 2024-07-26T06:46:01.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.935+0000 7f6bf58c9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6bf01020c0 0x7f6bf0199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:01.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.935+0000 7f6bf50c8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bf0103150 0x7f6bf019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:01.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.935+0000 7f6bf58c9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6bf01020c0 0x7f6bf0199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:36436/0 (socket says 172.21.15.42:36436) 2024-07-26T06:46:01.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.935+0000 7f6bf58c9640 1 -- 172.21.15.42:0/239483558 learned_addr learned my addr 172.21.15.42:0/239483558 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:46:01.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.935+0000 7f6bf50c8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bf0103150 0x7f6bf019a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46734/0 (socket says 172.21.15.42:46734) 2024-07-26T06:46:01.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.935+0000 7f6bf60ca640 1 --2- 172.21.15.42:0/239483558 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bf0103af0 0x7f6bf019ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:01.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.936+0000 7f6bf58c9640 1 -- 172.21.15.42:0/239483558 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bf0103af0 msgr2=0x7f6bf019ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:01.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.936+0000 7f6bf58c9640 1 --2- 172.21.15.42:0/239483558 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bf0103af0 0x7f6bf019ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:01.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.936+0000 7f6bf58c9640 1 -- 172.21.15.42:0/239483558 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bf0103150 msgr2=0x7f6bf019a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:01.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.936+0000 7f6bf58c9640 1 --2- 172.21.15.42:0/239483558 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bf0103150 0x7f6bf019a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:01.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.936+0000 7f6bf58c9640 1 -- 172.21.15.42:0/239483558 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf019b440 con 0x7f6bf01020c0 2024-07-26T06:46:01.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.936+0000 7f6bf58c9640 1 --2- 172.21.15.42:0/239483558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6bf01020c0 0x7f6bf0199a10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6be800d950 tx=0x7f6be800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:01.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.936+0000 7f6beeffd640 1 -- 172.21.15.42:0/239483558 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8017070 con 0x7f6bf01020c0 2024-07-26T06:46:01.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.936+0000 7f6bf7b2d640 1 -- 172.21.15.42:0/239483558 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf019b730 con 0x7f6bf01020c0 2024-07-26T06:46:01.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.936+0000 7f6beeffd640 1 -- 172.21.15.42:0/239483558 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6be8009c30 con 0x7f6bf01020c0 2024-07-26T06:46:01.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.936+0000 7f6beeffd640 1 -- 172.21.15.42:0/239483558 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be800f620 con 0x7f6bf01020c0 2024-07-26T06:46:01.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.936+0000 7f6bf7b2d640 1 -- 172.21.15.42:0/239483558 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf01a1e30 con 0x7f6bf01020c0 2024-07-26T06:46:01.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.937+0000 7f6bf7b2d640 1 -- 172.21.15.42:0/239483558 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bf0102530 con 0x7f6bf01020c0 2024-07-26T06:46:01.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.939+0000 7f6beeffd640 1 -- 172.21.15.42:0/239483558 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f6be8010460 con 0x7f6bf01020c0 2024-07-26T06:46:01.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.940+0000 7f6beeffd640 1 --2- 172.21.15.42:0/239483558 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6bd80b3bd0 0x7f6bd80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:01.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.940+0000 7f6bf50c8640 1 --2- 172.21.15.42:0/239483558 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6bd80b3bd0 0x7f6bd80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:01.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.940+0000 7f6beeffd640 1 -- 172.21.15.42:0/239483558 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6847+0+0 (secure 0 0 0) 0x7f6be80e3250 con 0x7f6bf01020c0 2024-07-26T06:46:01.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.940+0000 7f6bf50c8640 1 --2- 172.21.15.42:0/239483558 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6bd80b3bd0 0x7f6bd80b60e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6be4000c00 tx=0x7f6be4031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:01.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:01.942+0000 7f6beeffd640 1 -- 172.21.15.42:0/239483558 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6be8009de0 con 0x7f6bf01020c0 2024-07-26T06:46:02.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:02.125+0000 7f6bf7b2d640 1 -- 172.21.15.42:0/239483558 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f6bf01a2120 con 0x7f6bf01020c0 2024-07-26T06:46:02.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:02.125+0000 7f6beeffd640 1 -- 172.21.15.42:0/239483558 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v69) ==== 74+0+133 (secure 0 0 0) 0x7f6be80aa770 con 0x7f6bf01020c0 2024-07-26T06:46:02.126 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:46:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:02.128+0000 7f6bf7b2d640 1 -- 172.21.15.42:0/239483558 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6bd80b3bd0 msgr2=0x7f6bd80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:02.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:02.128+0000 7f6bf7b2d640 1 --2- 172.21.15.42:0/239483558 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6bd80b3bd0 0x7f6bd80b60e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6be4000c00 tx=0x7f6be4031400 comp rx=0 tx=0).stop 2024-07-26T06:46:02.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:02.128+0000 7f6bf7b2d640 1 -- 172.21.15.42:0/239483558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6bf01020c0 msgr2=0x7f6bf0199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:02.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:02.128+0000 7f6bf7b2d640 1 --2- 172.21.15.42:0/239483558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6bf01020c0 0x7f6bf0199a10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6be800d950 tx=0x7f6be800de10 comp rx=0 tx=0).stop 2024-07-26T06:46:02.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:02.128+0000 7f6bf7b2d640 1 -- 172.21.15.42:0/239483558 shutdown_connections 2024-07-26T06:46:02.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:02.128+0000 7f6bf7b2d640 1 --2- 172.21.15.42:0/239483558 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6bd80b3bd0 0x7f6bd80b60e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:02.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:02.128+0000 7f6bf7b2d640 1 --2- 172.21.15.42:0/239483558 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6bf0103af0 0x7f6bf019ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:02.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:02.128+0000 7f6bf7b2d640 1 --2- 172.21.15.42:0/239483558 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bf0103150 0x7f6bf019a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:02.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:02.128+0000 7f6bf7b2d640 1 --2- 172.21.15.42:0/239483558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6bf01020c0 0x7f6bf0199a10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:02.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:02.128+0000 7f6bf7b2d640 1 -- 172.21.15.42:0/239483558 >> 172.21.15.42:0/239483558 conn(0x7f6bf00fda30 msgr2=0x7f6bf006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:02.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:02.128+0000 7f6bf7b2d640 1 -- 172.21.15.42:0/239483558 shutdown_connections 2024-07-26T06:46:02.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:02.128+0000 7f6bf7b2d640 1 -- 172.21.15.42:0/239483558 wait complete. 2024-07-26T06:46:02.334 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:02 smithi164 ceph-mon[36970]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-07-26T06:46:02.334 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:02 smithi164 ceph-mon[36970]: osdmap e69: 12 total, 11 up, 12 in 2024-07-26T06:46:02.334 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T06:46:02.334 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T06:46:02.334 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:02 smithi164 ceph-mon[36970]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:46:02.334 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:02 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/239483558' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T06:46:02.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:02 smithi042 ceph-mon[32775]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-07-26T06:46:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:02 smithi042 ceph-mon[32775]: osdmap e69: 12 total, 11 up, 12 in 2024-07-26T06:46:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T06:46:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T06:46:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:02 smithi042 ceph-mon[32775]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:46:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:02 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/239483558' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T06:46:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:02 smithi163 ceph-mon[36873]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-07-26T06:46:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:02 smithi163 ceph-mon[36873]: osdmap e69: 12 total, 11 up, 12 in 2024-07-26T06:46:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T06:46:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T06:46:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:02 smithi163 ceph-mon[36873]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-07-26T06:46:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:02 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/239483558' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T06:46:02.859 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":69,"num_osds":12,"num_up_osds":11,"osd_up_since":1721976337,"num_in_osds":12,"osd_in_since":1721976343,"num_remapped_pgs":0} 2024-07-26T06:46:03.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:03 smithi164 ceph-mon[36970]: purged_snaps scrub starts 2024-07-26T06:46:03.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:03 smithi164 ceph-mon[36970]: purged_snaps scrub ok 2024-07-26T06:46:03.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:03 smithi164 ceph-mon[36970]: osd.11 [v2:172.21.15.164:6824/1861076336,v1:172.21.15.164:6825/1861076336] boot 2024-07-26T06:46:03.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:03 smithi164 ceph-mon[36970]: osdmap e70: 12 total, 12 up, 12 in 2024-07-26T06:46:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T06:46:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:03 smithi164 ceph-mon[36970]: Detected new or changed devices on smithi164 2024-07-26T06:46:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:46:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:46:03.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:46:03.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:46:03.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:03 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 4272M 2024-07-26T06:46:03.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:03.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:03.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:46:03.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:03 smithi163 ceph-mon[36873]: purged_snaps scrub starts 2024-07-26T06:46:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:03 smithi163 ceph-mon[36873]: purged_snaps scrub ok 2024-07-26T06:46:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:03 smithi163 ceph-mon[36873]: osd.11 [v2:172.21.15.164:6824/1861076336,v1:172.21.15.164:6825/1861076336] boot 2024-07-26T06:46:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:03 smithi163 ceph-mon[36873]: osdmap e70: 12 total, 12 up, 12 in 2024-07-26T06:46:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T06:46:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:03 smithi163 ceph-mon[36873]: Detected new or changed devices on smithi164 2024-07-26T06:46:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:46:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:46:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:46:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:46:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:03 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 4272M 2024-07-26T06:46:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:46:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:03 smithi042 ceph-mon[32775]: purged_snaps scrub starts 2024-07-26T06:46:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:03 smithi042 ceph-mon[32775]: purged_snaps scrub ok 2024-07-26T06:46:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:03 smithi042 ceph-mon[32775]: osd.11 [v2:172.21.15.164:6824/1861076336,v1:172.21.15.164:6825/1861076336] boot 2024-07-26T06:46:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:03 smithi042 ceph-mon[32775]: osdmap e70: 12 total, 12 up, 12 in 2024-07-26T06:46:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T06:46:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:03 smithi042 ceph-mon[32775]: Detected new or changed devices on smithi164 2024-07-26T06:46:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:46:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:46:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:46:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:46:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:03 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 4272M 2024-07-26T06:46:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:46:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:03.860 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd stat -f json 2024-07-26T06:46:04.136 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:46:04.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:04 smithi042 ceph-mon[32775]: osdmap e71: 12 total, 12 up, 12 in 2024-07-26T06:46:04.466 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:04 smithi042 ceph-mon[32775]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:04.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:04 smithi164 ceph-mon[36970]: osdmap e71: 12 total, 12 up, 12 in 2024-07-26T06:46:04.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:04 smithi164 ceph-mon[36970]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:04.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:04 smithi163 ceph-mon[36873]: osdmap e71: 12 total, 12 up, 12 in 2024-07-26T06:46:04.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:04 smithi163 ceph-mon[36873]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:05.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.722+0000 7f1b5a9d3640 1 -- 172.21.15.42:0/2811585498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b541051b0 msgr2=0x7f1b54107690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:05.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.722+0000 7f1b5a9d3640 1 --2- 172.21.15.42:0/2811585498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b541051b0 0x7f1b54107690 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009900 tx=0x7f1b4802a410 comp rx=0 tx=0).stop 2024-07-26T06:46:05.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.722+0000 7f1b5a9d3640 1 -- 172.21.15.42:0/2811585498 shutdown_connections 2024-07-26T06:46:05.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.722+0000 7f1b5a9d3640 1 --2- 172.21.15.42:0/2811585498 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b54107dd0 0x7f1b5410a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:05.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.722+0000 7f1b5a9d3640 1 --2- 172.21.15.42:0/2811585498 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b541051b0 0x7f1b54107690 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:05.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.722+0000 7f1b5a9d3640 1 --2- 172.21.15.42:0/2811585498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b541026f0 0x7f1b54104b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:05.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.722+0000 7f1b5a9d3640 1 -- 172.21.15.42:0/2811585498 >> 172.21.15.42:0/2811585498 conn(0x7f1b540fd190 msgr2=0x7f1b540ff640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:05.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.722+0000 7f1b5a9d3640 1 -- 172.21.15.42:0/2811585498 shutdown_connections 2024-07-26T06:46:05.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.722+0000 7f1b5a9d3640 1 -- 172.21.15.42:0/2811585498 wait complete. 2024-07-26T06:46:05.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.722+0000 7f1b5a9d3640 1 Processor -- start 2024-07-26T06:46:05.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.722+0000 7f1b5a9d3640 1 -- start start 2024-07-26T06:46:05.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b5a9d3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b541026f0 0x7f1b541043b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:05.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b5a9d3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b541051b0 0x7f1b54102c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:05.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b5a9d3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b54107dd0 0x7f1b541031c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:05.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b5a9d3640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5410dcf0 con 0x7f1b541051b0 2024-07-26T06:46:05.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b5a9d3640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5410db70 con 0x7f1b54107dd0 2024-07-26T06:46:05.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b5a9d3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5410de70 con 0x7f1b541026f0 2024-07-26T06:46:05.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b541051b0 0x7f1b54102c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:05.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b541051b0 0x7f1b54102c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46750/0 (socket says 172.21.15.42:46750) 2024-07-26T06:46:05.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b537fe640 1 -- 172.21.15.42:0/3257085106 learned_addr learned my addr 172.21.15.42:0/3257085106 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:46:05.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b53fff640 1 --2- 172.21.15.42:0/3257085106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b541026f0 0x7f1b541043b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:05.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b537fe640 1 -- 172.21.15.42:0/3257085106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b541026f0 msgr2=0x7f1b541043b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:05.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b537fe640 1 --2- 172.21.15.42:0/3257085106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b541026f0 0x7f1b541043b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:05.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b537fe640 1 -- 172.21.15.42:0/3257085106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b54107dd0 msgr2=0x7f1b541031c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:05.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b58f70640 1 --2- 172.21.15.42:0/3257085106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b54107dd0 0x7f1b541031c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:05.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b537fe640 1 --2- 172.21.15.42:0/3257085106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b54107dd0 0x7f1b541031c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:05.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b537fe640 1 -- 172.21.15.42:0/3257085106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b541038c0 con 0x7f1b541051b0 2024-07-26T06:46:05.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b58f70640 1 --2- 172.21.15.42:0/3257085106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b54107dd0 0x7f1b541031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:46:05.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.723+0000 7f1b537fe640 1 --2- 172.21.15.42:0/3257085106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b541051b0 0x7f1b54102c80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1b48006010 tx=0x7f1b48031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:05.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.724+0000 7f1b517fa640 1 -- 172.21.15.42:0/3257085106 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4803f030 con 0x7f1b541051b0 2024-07-26T06:46:05.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.724+0000 7f1b5a9d3640 1 -- 172.21.15.42:0/3257085106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b541a3ab0 con 0x7f1b541051b0 2024-07-26T06:46:05.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.724+0000 7f1b517fa640 1 -- 172.21.15.42:0/3257085106 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b48003aa0 con 0x7f1b541051b0 2024-07-26T06:46:05.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.724+0000 7f1b5a9d3640 1 -- 172.21.15.42:0/3257085106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b541a3ea0 con 0x7f1b541051b0 2024-07-26T06:46:05.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.724+0000 7f1b517fa640 1 -- 172.21.15.42:0/3257085106 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b48035070 con 0x7f1b541051b0 2024-07-26T06:46:05.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.725+0000 7f1b5a9d3640 1 -- 172.21.15.42:0/3257085106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b54105680 con 0x7f1b541051b0 2024-07-26T06:46:05.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.727+0000 7f1b517fa640 1 -- 172.21.15.42:0/3257085106 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f1b48042030 con 0x7f1b541051b0 2024-07-26T06:46:05.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.727+0000 7f1b517fa640 1 --2- 172.21.15.42:0/3257085106 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1b3c0b3bd0 0x7f1b3c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:05.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.728+0000 7f1b53fff640 1 --2- 172.21.15.42:0/3257085106 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1b3c0b3bd0 0x7f1b3c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:05.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.728+0000 7f1b517fa640 1 -- 172.21.15.42:0/3257085106 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f1b481043f0 con 0x7f1b541051b0 2024-07-26T06:46:05.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.728+0000 7f1b53fff640 1 --2- 172.21.15.42:0/3257085106 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1b3c0b3bd0 0x7f1b3c0b60e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009bd0 tx=0x7f1b44005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:05.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.732+0000 7f1b517fa640 1 -- 172.21.15.42:0/3257085106 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1b480cbee0 con 0x7f1b541051b0 2024-07-26T06:46:05.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.916+0000 7f1b5a9d3640 1 -- 172.21.15.42:0/3257085106 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f1b541a4290 con 0x7f1b541051b0 2024-07-26T06:46:05.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.917+0000 7f1b517fa640 1 -- 172.21.15.42:0/3257085106 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v71) ==== 74+0+133 (secure 0 0 0) 0x7f1b480d41b0 con 0x7f1b541051b0 2024-07-26T06:46:05.917 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:46:05.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.919+0000 7f1b5a9d3640 1 -- 172.21.15.42:0/3257085106 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1b3c0b3bd0 msgr2=0x7f1b3c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:05.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.919+0000 7f1b5a9d3640 1 --2- 172.21.15.42:0/3257085106 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1b3c0b3bd0 0x7f1b3c0b60e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009bd0 tx=0x7f1b44005f90 comp rx=0 tx=0).stop 2024-07-26T06:46:05.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.919+0000 7f1b5a9d3640 1 -- 172.21.15.42:0/3257085106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b541051b0 msgr2=0x7f1b54102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:05.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.919+0000 7f1b5a9d3640 1 --2- 172.21.15.42:0/3257085106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b541051b0 0x7f1b54102c80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1b48006010 tx=0x7f1b48031a50 comp rx=0 tx=0).stop 2024-07-26T06:46:05.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.919+0000 7f1b5a9d3640 1 -- 172.21.15.42:0/3257085106 shutdown_connections 2024-07-26T06:46:05.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.919+0000 7f1b5a9d3640 1 --2- 172.21.15.42:0/3257085106 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1b3c0b3bd0 0x7f1b3c0b60e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:05.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.919+0000 7f1b5a9d3640 1 --2- 172.21.15.42:0/3257085106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b54107dd0 0x7f1b541031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:05.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.919+0000 7f1b5a9d3640 1 --2- 172.21.15.42:0/3257085106 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b541051b0 0x7f1b54102c80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:05.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.919+0000 7f1b5a9d3640 1 --2- 172.21.15.42:0/3257085106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b541026f0 0x7f1b541043b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:05.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.919+0000 7f1b5a9d3640 1 -- 172.21.15.42:0/3257085106 >> 172.21.15.42:0/3257085106 conn(0x7f1b540fd190 msgr2=0x7f1b54107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:05.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.919+0000 7f1b5a9d3640 1 -- 172.21.15.42:0/3257085106 shutdown_connections 2024-07-26T06:46:05.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:05.919+0000 7f1b5a9d3640 1 -- 172.21.15.42:0/3257085106 wait complete. 2024-07-26T06:46:06.580 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:06 smithi042 ceph-mon[32775]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:06.580 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:06 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3257085106' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T06:46:06.609 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":71,"num_osds":12,"num_up_osds":12,"osd_up_since":1721976362,"num_in_osds":12,"osd_in_since":1721976343,"num_remapped_pgs":0} 2024-07-26T06:46:06.610 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T06:46:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:06 smithi164 ceph-mon[36970]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:06 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3257085106' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T06:46:06.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:06 smithi163 ceph-mon[36873]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:06 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3257085106' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T06:46:06.872 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:46:08.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.585+0000 7f5c6ae32640 1 -- 172.21.15.42:0/1683707246 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c64107e00 msgr2=0x7f5c64108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.585+0000 7f5c6ae32640 1 --2- 172.21.15.42:0/1683707246 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c64107e00 0x7f5c64108210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5c58009900 tx=0x7f5c5802a410 comp rx=0 tx=0).stop 2024-07-26T06:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.585+0000 7f5c6ae32640 1 -- 172.21.15.42:0/1683707246 shutdown_connections 2024-07-26T06:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.585+0000 7f5c6ae32640 1 --2- 172.21.15.42:0/1683707246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c64108750 0x7f5c6410ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.585+0000 7f5c6ae32640 1 --2- 172.21.15.42:0/1683707246 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c64107e00 0x7f5c64108210 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.585+0000 7f5c6ae32640 1 --2- 172.21.15.42:0/1683707246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c64102510 0x7f5c64102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.585+0000 7f5c6ae32640 1 -- 172.21.15.42:0/1683707246 >> 172.21.15.42:0/1683707246 conn(0x7f5c64079190 msgr2=0x7f5c640795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.585+0000 7f5c6ae32640 1 -- 172.21.15.42:0/1683707246 shutdown_connections 2024-07-26T06:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.585+0000 7f5c6ae32640 1 -- 172.21.15.42:0/1683707246 wait complete. 2024-07-26T06:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c6ae32640 1 Processor -- start 2024-07-26T06:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c6ae32640 1 -- start start 2024-07-26T06:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c6ae32640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c64102510 0x7f5c640ff0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:08.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c6ae32640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c64107e00 0x7f5c640ff5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:08.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c6ae32640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c64108750 0x7f5c640ffb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:08.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c6ae32640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5c64112380 con 0x7f5c64102510 2024-07-26T06:46:08.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c6ae32640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5c64112200 con 0x7f5c64108750 2024-07-26T06:46:08.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c6ae32640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5c64112500 con 0x7f5c64107e00 2024-07-26T06:46:08.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c68bce640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c64102510 0x7f5c640ff0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:08.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c68bce640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c64102510 0x7f5c640ff0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41846/0 (socket says 172.21.15.42:41846) 2024-07-26T06:46:08.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c68bce640 1 -- 172.21.15.42:0/1780534754 learned_addr learned my addr 172.21.15.42:0/1780534754 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:46:08.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c63fff640 1 --2- 172.21.15.42:0/1780534754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c64107e00 0x7f5c640ff5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:08.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c68bce640 1 -- 172.21.15.42:0/1780534754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c64107e00 msgr2=0x7f5c640ff5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:08.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c68bce640 1 --2- 172.21.15.42:0/1780534754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c64107e00 0x7f5c640ff5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:08.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c68bce640 1 -- 172.21.15.42:0/1780534754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c64108750 msgr2=0x7f5c640ffb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:08.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c68bce640 1 --2- 172.21.15.42:0/1780534754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c64108750 0x7f5c640ffb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:08.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c68bce640 1 -- 172.21.15.42:0/1780534754 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c641a80d0 con 0x7f5c64102510 2024-07-26T06:46:08.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.586+0000 7f5c63fff640 1 --2- 172.21.15.42:0/1780534754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c64107e00 0x7f5c640ff5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:46:08.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.587+0000 7f5c68bce640 1 --2- 172.21.15.42:0/1780534754 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c64102510 0x7f5c640ff0a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400c890 tx=0x7f5c5400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:08.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.587+0000 7f5c61ffb640 1 -- 172.21.15.42:0/1780534754 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c54004220 con 0x7f5c64102510 2024-07-26T06:46:08.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.587+0000 7f5c6ae32640 1 -- 172.21.15.42:0/1780534754 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c641a83c0 con 0x7f5c64102510 2024-07-26T06:46:08.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.587+0000 7f5c61ffb640 1 -- 172.21.15.42:0/1780534754 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c540043c0 con 0x7f5c64102510 2024-07-26T06:46:08.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.587+0000 7f5c61ffb640 1 -- 172.21.15.42:0/1780534754 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c54004a00 con 0x7f5c64102510 2024-07-26T06:46:08.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.587+0000 7f5c6ae32640 1 -- 172.21.15.42:0/1780534754 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c641a8900 con 0x7f5c64102510 2024-07-26T06:46:08.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.588+0000 7f5c6ae32640 1 -- 172.21.15.42:0/1780534754 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c64103740 con 0x7f5c64102510 2024-07-26T06:46:08.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.589+0000 7f5c61ffb640 1 -- 172.21.15.42:0/1780534754 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f5c540079b0 con 0x7f5c64102510 2024-07-26T06:46:08.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.590+0000 7f5c61ffb640 1 --2- 172.21.15.42:0/1780534754 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5c4c0b3bd0 0x7f5c4c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:08.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.590+0000 7f5c63fff640 1 --2- 172.21.15.42:0/1780534754 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5c4c0b3bd0 0x7f5c4c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:08.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.590+0000 7f5c61ffb640 1 -- 172.21.15.42:0/1780534754 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f5c540e3620 con 0x7f5c64102510 2024-07-26T06:46:08.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.591+0000 7f5c63fff640 1 --2- 172.21.15.42:0/1780534754 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5c4c0b3bd0 0x7f5c4c0b60e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5c580096c0 tx=0x7f5c58031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:08.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.595+0000 7f5c61ffb640 1 -- 172.21.15.42:0/1780534754 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5c54014030 con 0x7f5c64102510 2024-07-26T06:46:08.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:08 smithi042 ceph-mon[32775]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:08 smithi164 ceph-mon[36970]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:08.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:08 smithi163 ceph-mon[36873]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.778+0000 7f5c6ae32640 1 -- 172.21.15.42:0/1780534754 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5c6402e0c0 con 0x7f5c64102510 2024-07-26T06:46:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.779+0000 7f5c61ffb640 1 -- 172.21.15.42:0/1780534754 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+18407 (secure 0 0 0) 0x7f5c540ab070 con 0x7f5c64102510 2024-07-26T06:46:08.781 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:46:08.781 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":71,"fsid":"8b3e2750-4b19-11ef-bcb6-c7b262605968","created":"2024-07-26T06:38:29.306470+0000","modified":"2024-07-26T06:46:03.156061+0000","last_up_change":"2024-07-26T06:46:02.149453+0000","last_in_change":"2024-07-26T06:45:43.027764+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:42:27.302195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bda88e0-09ee-447b-8c04-24f4e5d55d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6803","nonce":3587066379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6805","nonce":3587066379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6809","nonce":3587066379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6807","nonce":3587066379}]},"public_addr":"172.21.15.42:6803/3587066379","cluster_addr":"172.21.15.42:6805/3587066379","heartbeat_back_addr":"172.21.15.42:6809/3587066379","heartbeat_front_addr":"172.21.15.42:6807/3587066379","state":["exists","up"]},{"osd":1,"uuid":"7d747138-8217-4836-a173-cf8ac65e9fe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6801","nonce":360219078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6803","nonce":360219078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6807","nonce":360219078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6805","nonce":360219078}]},"public_addr":"172.21.15.163:6801/360219078","cluster_addr":"172.21.15.163:6803/360219078","heartbeat_back_addr":"172.21.15.163:6807/360219078","heartbeat_front_addr":"172.21.15.163:6805/360219078","state":["exists","up"]},{"osd":2,"uuid":"4b7258c1-3d35-4930-ab10-0b04a2a2c8a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6801","nonce":336113412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6803","nonce":336113412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6807","nonce":336113412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6805","nonce":336113412}]},"public_addr":"172.21.15.164:6801/336113412","cluster_addr":"172.21.15.164:6803/336113412","heartbeat_back_addr":"172.21.15.164:6807/336113412","heartbeat_front_addr":"172.21.15.164:6805/336113412","state":["exists","up"]},{"osd":3,"uuid":"96608387-0de0-4d90-a952-d6eba61f7fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6811","nonce":2819018064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6813","nonce":2819018064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6817","nonce":2819018064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6815","nonce":2819018064}]},"public_addr":"172.21.15.42:6811/2819018064","cluster_addr":"172.21.15.42:6813/2819018064","heartbeat_back_addr":"172.21.15.42:6817/2819018064","heartbeat_front_addr":"172.21.15.42:6815/2819018064","state":["exists","up"]},{"osd":4,"uuid":"fabf6b02-fe34-4560-85b5-01bf37fc9be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6809","nonce":431934374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6811","nonce":431934374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6815","nonce":431934374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6813","nonce":431934374}]},"public_addr":"172.21.15.163:6809/431934374","cluster_addr":"172.21.15.163:6811/431934374","heartbeat_back_addr":"172.21.15.163:6815/431934374","heartbeat_front_addr":"172.21.15.163:6813/431934374","state":["exists","up"]},{"osd":5,"uuid":"e226fb4f-15b2-40df-af34-900040ba93ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6809","nonce":261549362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6811","nonce":261549362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6815","nonce":261549362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6813","nonce":261549362}]},"public_addr":"172.21.15.164:6809/261549362","cluster_addr":"172.21.15.164:6811/261549362","heartbeat_back_addr":"172.21.15.164:6815/261549362","heartbeat_front_addr":"172.21.15.164:6813/261549362","state":["exists","up"]},{"osd":6,"uuid":"beb8ec5d-7ea5-4e4e-a46c-a84f7768cec0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6819","nonce":2160811300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6821","nonce":2160811300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6825","nonce":2160811300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6823","nonce":2160811300}]},"public_addr":"172.21.15.42:6819/2160811300","cluster_addr":"172.21.15.42:6821/2160811300","heartbeat_back_addr":"172.21.15.42:6825/2160811300","heartbeat_front_addr":"172.21.15.42:6823/2160811300","state":["exists","up"]},{"osd":7,"uuid":"f699004c-f7c2-4f1c-ae4b-7178f3b06e2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6817","nonce":812214111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6819","nonce":812214111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6823","nonce":812214111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6821","nonce":812214111}]},"public_addr":"172.21.15.163:6817/812214111","cluster_addr":"172.21.15.163:6819/812214111","heartbeat_back_addr":"172.21.15.163:6823/812214111","heartbeat_front_addr":"172.21.15.163:6821/812214111","state":["exists","up"]},{"osd":8,"uuid":"3476163e-efea-4652-a8d4-b7c4a908f984","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6817","nonce":962845839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6819","nonce":962845839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6823","nonce":962845839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6821","nonce":962845839}]},"public_addr":"172.21.15.164:6817/962845839","cluster_addr":"172.21.15.164:6819/962845839","heartbeat_back_addr":"172.21.15.164:6823/962845839","heartbeat_front_addr":"172.21.15.164:6821/962845839","state":["exists","up"]},{"osd":9,"uuid":"6f7ceac6-fb61-4c6f-9da8-851b937c4586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6827","nonce":994024418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6829","nonce":994024418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6833","nonce":994024418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6831","nonce":994024418}]},"public_addr":"172.21.15.42:6827/994024418","cluster_addr":"172.21.15.42:6829/994024418","heartbeat_back_addr":"172.21.15.42:6833/994024418","heartbeat_front_addr":"172.21.15.42:6831/994024418","state":["exists","up"]},{"osd":10,"uuid":"553132c2-a6a1-410c-a7d8-ecdf9a67dafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6825","nonce":3268512227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6827","nonce":3268512227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6831","nonce":3268512227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6829","nonce":3268512227}]},"public_addr":"172.21.15.163:6825/3268512227","cluster_addr":"172.21.15.163:6827/3268512227","heartbeat_back_addr":"172.21.15.163:6831/3268512227","heartbeat_front_addr":"172.21.15.163:6829/3268512227","state":["exists","up"]},{"osd":11,"uuid":"09abc8e9-7884-400f-ab6d-8e5aa7bf31ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6825","nonce":1861076336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6827","nonce":1861076336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6831","nonce":1861076336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6829","nonce":1861076336}]},"public_addr":"172.21.15.164:6825/1861076336","cluster_addr":"172.21.15.164:6827/1861076336","heartbeat_back_addr":"172.21.15.164:6831/1861076336","heartbeat_front_addr":"172.21.15.164:6829/1861076336","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:41:38.761674+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:00.621745+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:23.248620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:47.232764+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:10.379922+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:34.502519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:58.694642+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:22.350247+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:46.988007+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:11.390474+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:36.150054+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:46:00.474470+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.42:6801/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6800/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1842521399":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/834172950":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1730109419":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6801/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:6800/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2687772889":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/3371433307":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2861719816":"2024-07-27T06:38:56.887264+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:46:08.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.782+0000 7f5c6ae32640 1 -- 172.21.15.42:0/1780534754 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5c4c0b3bd0 msgr2=0x7f5c4c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:08.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.782+0000 7f5c6ae32640 1 --2- 172.21.15.42:0/1780534754 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5c4c0b3bd0 0x7f5c4c0b60e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5c580096c0 tx=0x7f5c58031400 comp rx=0 tx=0).stop 2024-07-26T06:46:08.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.782+0000 7f5c6ae32640 1 -- 172.21.15.42:0/1780534754 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c64102510 msgr2=0x7f5c640ff0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:08.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.782+0000 7f5c6ae32640 1 --2- 172.21.15.42:0/1780534754 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c64102510 0x7f5c640ff0a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400c890 tx=0x7f5c5400cd50 comp rx=0 tx=0).stop 2024-07-26T06:46:08.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.782+0000 7f5c6ae32640 1 -- 172.21.15.42:0/1780534754 shutdown_connections 2024-07-26T06:46:08.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.782+0000 7f5c6ae32640 1 --2- 172.21.15.42:0/1780534754 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5c4c0b3bd0 0x7f5c4c0b60e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:08.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.782+0000 7f5c6ae32640 1 --2- 172.21.15.42:0/1780534754 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c64108750 0x7f5c640ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:08.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.782+0000 7f5c6ae32640 1 --2- 172.21.15.42:0/1780534754 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c64107e00 0x7f5c640ff5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:08.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.782+0000 7f5c6ae32640 1 --2- 172.21.15.42:0/1780534754 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c64102510 0x7f5c640ff0a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:08.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.782+0000 7f5c6ae32640 1 -- 172.21.15.42:0/1780534754 >> 172.21.15.42:0/1780534754 conn(0x7f5c64079190 msgr2=0x7f5c64101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:08.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.782+0000 7f5c6ae32640 1 -- 172.21.15.42:0/1780534754 shutdown_connections 2024-07-26T06:46:08.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:08.782+0000 7f5c6ae32640 1 -- 172.21.15.42:0/1780534754 wait complete. 2024-07-26T06:46:09.487 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-26T06:42:27.302195+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-07-26T06:46:09.488 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-07-26T06:46:09.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:09 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1780534754' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:46:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:09 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1780534754' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:46:09.749 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:46:09.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:09 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1780534754' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:46:10.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:10 smithi042 ceph-mon[32775]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:10 smithi164 ceph-mon[36970]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:10.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:10 smithi163 ceph-mon[36873]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:11.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.502+0000 7fad56580640 1 -- 172.21.15.42:0/3798489957 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad501087a0 msgr2=0x7fad5010efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:11.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.502+0000 7fad56580640 1 --2- 172.21.15.42:0/3798489957 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad501087a0 0x7fad5010efd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fad48009ca0 tx=0x7fad4802a8c0 comp rx=0 tx=0).stop 2024-07-26T06:46:11.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.502+0000 7fad56580640 1 -- 172.21.15.42:0/3798489957 shutdown_connections 2024-07-26T06:46:11.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.502+0000 7fad56580640 1 --2- 172.21.15.42:0/3798489957 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad501087a0 0x7fad5010efd0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:11.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.502+0000 7fad56580640 1 --2- 172.21.15.42:0/3798489957 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad50107e50 0x7fad50108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:11.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.502+0000 7fad56580640 1 --2- 172.21.15.42:0/3798489957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad50102510 0x7fad50102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:11.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.502+0000 7fad56580640 1 -- 172.21.15.42:0/3798489957 >> 172.21.15.42:0/3798489957 conn(0x7fad500791a0 msgr2=0x7fad500795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:11.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.502+0000 7fad56580640 1 -- 172.21.15.42:0/3798489957 shutdown_connections 2024-07-26T06:46:11.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.502+0000 7fad56580640 1 -- 172.21.15.42:0/3798489957 wait complete. 2024-07-26T06:46:11.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.503+0000 7fad56580640 1 Processor -- start 2024-07-26T06:46:11.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.503+0000 7fad56580640 1 -- start start 2024-07-26T06:46:11.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.503+0000 7fad56580640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad50102510 0x7fad50198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:11.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.503+0000 7fad56580640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad50107e50 0x7fad50199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:11.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.503+0000 7fad56580640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad501087a0 0x7fad5019e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:11.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.503+0000 7fad56580640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fad50112450 con 0x7fad501087a0 2024-07-26T06:46:11.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.503+0000 7fad56580640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fad501122d0 con 0x7fad50102510 2024-07-26T06:46:11.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.503+0000 7fad56580640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fad501125d0 con 0x7fad50107e50 2024-07-26T06:46:11.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.503+0000 7fad54b1d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad501087a0 0x7fad5019e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:11.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.504+0000 7fad54b1d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad501087a0 0x7fad5019e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41874/0 (socket says 172.21.15.42:41874) 2024-07-26T06:46:11.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.504+0000 7fad54b1d640 1 -- 172.21.15.42:0/3225358014 learned_addr learned my addr 172.21.15.42:0/3225358014 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:46:11.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.504+0000 7fad4ffff640 1 --2- 172.21.15.42:0/3225358014 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad50102510 0x7fad50198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:11.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.504+0000 7fad4f7fe640 1 --2- 172.21.15.42:0/3225358014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad50107e50 0x7fad50199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:11.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.504+0000 7fad54b1d640 1 -- 172.21.15.42:0/3225358014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad50107e50 msgr2=0x7fad50199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:11.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.504+0000 7fad54b1d640 1 --2- 172.21.15.42:0/3225358014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad50107e50 0x7fad50199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:11.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.504+0000 7fad54b1d640 1 -- 172.21.15.42:0/3225358014 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad50102510 msgr2=0x7fad50198da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:11.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.504+0000 7fad54b1d640 1 --2- 172.21.15.42:0/3225358014 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad50102510 0x7fad50198da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:11.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.504+0000 7fad54b1d640 1 -- 172.21.15.42:0/3225358014 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad5019ed10 con 0x7fad501087a0 2024-07-26T06:46:11.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.504+0000 7fad54b1d640 1 --2- 172.21.15.42:0/3225358014 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad501087a0 0x7fad5019e5e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fad48005cc0 tx=0x7fad4802f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:11.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.504+0000 7fad4d7fa640 1 -- 172.21.15.42:0/3225358014 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad48003bf0 con 0x7fad501087a0 2024-07-26T06:46:11.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.504+0000 7fad56580640 1 -- 172.21.15.42:0/3225358014 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad5019efa0 con 0x7fad501087a0 2024-07-26T06:46:11.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.504+0000 7fad4d7fa640 1 -- 172.21.15.42:0/3225358014 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad48003d90 con 0x7fad501087a0 2024-07-26T06:46:11.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.504+0000 7fad4d7fa640 1 -- 172.21.15.42:0/3225358014 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4802fbc0 con 0x7fad501087a0 2024-07-26T06:46:11.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.504+0000 7fad56580640 1 -- 172.21.15.42:0/3225358014 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad5006b590 con 0x7fad501087a0 2024-07-26T06:46:11.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.505+0000 7fad56580640 1 -- 172.21.15.42:0/3225358014 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad50103790 con 0x7fad501087a0 2024-07-26T06:46:11.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.507+0000 7fad4d7fa640 1 -- 172.21.15.42:0/3225358014 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fad4802fd60 con 0x7fad501087a0 2024-07-26T06:46:11.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.508+0000 7fad4d7fa640 1 --2- 172.21.15.42:0/3225358014 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fad380b5de0 0x7fad380b82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:11.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.508+0000 7fad4ffff640 1 --2- 172.21.15.42:0/3225358014 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fad380b5de0 0x7fad380b82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:11.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.508+0000 7fad4d7fa640 1 -- 172.21.15.42:0/3225358014 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fad48105070 con 0x7fad501087a0 2024-07-26T06:46:11.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.508+0000 7fad4ffff640 1 --2- 172.21.15.42:0/3225358014 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fad380b5de0 0x7fad380b82f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fad4000bbe0 tx=0x7fad4000b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:11.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.511+0000 7fad4d7fa640 1 -- 172.21.15.42:0/3225358014 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fad480ccac0 con 0x7fad501087a0 2024-07-26T06:46:11.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.692+0000 7fad56580640 1 -- 172.21.15.42:0/3225358014 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fad5006bfb0 con 0x7fad501087a0 2024-07-26T06:46:11.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.693+0000 7fad4d7fa640 1 -- 172.21.15.42:0/3225358014 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) ==== 93+0+10 (secure 0 0 0) 0x7fad480cc3c0 con 0x7fad501087a0 2024-07-26T06:46:11.695 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 1 2024-07-26T06:46:11.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.695+0000 7fad56580640 1 -- 172.21.15.42:0/3225358014 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fad380b5de0 msgr2=0x7fad380b82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:11.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.695+0000 7fad56580640 1 --2- 172.21.15.42:0/3225358014 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fad380b5de0 0x7fad380b82f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fad4000bbe0 tx=0x7fad4000b390 comp rx=0 tx=0).stop 2024-07-26T06:46:11.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.695+0000 7fad56580640 1 -- 172.21.15.42:0/3225358014 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad501087a0 msgr2=0x7fad5019e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:11.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.695+0000 7fad56580640 1 --2- 172.21.15.42:0/3225358014 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad501087a0 0x7fad5019e5e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fad48005cc0 tx=0x7fad4802f380 comp rx=0 tx=0).stop 2024-07-26T06:46:11.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.695+0000 7fad56580640 1 -- 172.21.15.42:0/3225358014 shutdown_connections 2024-07-26T06:46:11.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.695+0000 7fad56580640 1 --2- 172.21.15.42:0/3225358014 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fad380b5de0 0x7fad380b82f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:11.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.695+0000 7fad56580640 1 --2- 172.21.15.42:0/3225358014 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad501087a0 0x7fad5019e5e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:11.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.695+0000 7fad56580640 1 --2- 172.21.15.42:0/3225358014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad50107e50 0x7fad50199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:11.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.695+0000 7fad56580640 1 --2- 172.21.15.42:0/3225358014 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad50102510 0x7fad50198da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:11.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.695+0000 7fad56580640 1 -- 172.21.15.42:0/3225358014 >> 172.21.15.42:0/3225358014 conn(0x7fad500791a0 msgr2=0x7fad50105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:11.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.695+0000 7fad56580640 1 -- 172.21.15.42:0/3225358014 shutdown_connections 2024-07-26T06:46:11.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:11.695+0000 7fad56580640 1 -- 172.21.15.42:0/3225358014 wait complete. 2024-07-26T06:46:12.403 INFO:tasks.cephadm:Adding mds.a on smithi042 2024-07-26T06:46:12.403 INFO:tasks.cephadm:Adding mds.d on smithi042 2024-07-26T06:46:12.403 INFO:tasks.cephadm:Adding mds.g on smithi042 2024-07-26T06:46:12.404 INFO:tasks.cephadm:Adding mds.j on smithi042 2024-07-26T06:46:12.404 INFO:tasks.cephadm:Adding mds.b on smithi163 2024-07-26T06:46:12.404 INFO:tasks.cephadm:Adding mds.e on smithi163 2024-07-26T06:46:12.404 INFO:tasks.cephadm:Adding mds.h on smithi163 2024-07-26T06:46:12.404 INFO:tasks.cephadm:Adding mds.k on smithi163 2024-07-26T06:46:12.404 INFO:tasks.cephadm:Adding mds.c on smithi164 2024-07-26T06:46:12.404 INFO:tasks.cephadm:Adding mds.f on smithi164 2024-07-26T06:46:12.404 INFO:tasks.cephadm:Adding mds.i on smithi164 2024-07-26T06:46:12.405 INFO:tasks.cephadm:Adding mds.l on smithi164 2024-07-26T06:46:12.405 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph orch apply mds all '12;smithi042=a;smithi042=d;smithi042=g;smithi042=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi164=c;smithi164=f;smithi164=i;smithi164=l' 2024-07-26T06:46:12.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:12 smithi042 ceph-mon[32775]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:12 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3225358014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T06:46:12.670 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.c/config 2024-07-26T06:46:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:12 smithi164 ceph-mon[36970]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:12 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3225358014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T06:46:12.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:12 smithi163 ceph-mon[36873]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:12 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3225358014' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T06:46:14.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.425+0000 7fb7eb910640 1 -- 172.21.15.164:0/4140928193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7e4104210 msgr2=0x7fb7e4104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:14.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.425+0000 7fb7eb910640 1 --2- 172.21.15.164:0/4140928193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7e4104210 0x7fb7e4104620 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009900 tx=0x7fb7d402a480 comp rx=0 tx=0).stop 2024-07-26T06:46:14.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.426+0000 7fb7eb910640 1 -- 172.21.15.164:0/4140928193 shutdown_connections 2024-07-26T06:46:14.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.426+0000 7fb7eb910640 1 --2- 172.21.15.164:0/4140928193 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7e4104bb0 0x7fb7e410b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:14.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.426+0000 7fb7eb910640 1 --2- 172.21.15.164:0/4140928193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7e4104210 0x7fb7e4104620 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:14.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.426+0000 7fb7eb910640 1 --2- 172.21.15.164:0/4140928193 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7e4103180 0x7fb7e41035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:14.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.426+0000 7fb7eb910640 1 -- 172.21.15.164:0/4140928193 >> 172.21.15.164:0/4140928193 conn(0x7fb7e40feaf0 msgr2=0x7fb7e4100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:14.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.426+0000 7fb7eb910640 1 -- 172.21.15.164:0/4140928193 shutdown_connections 2024-07-26T06:46:14.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.426+0000 7fb7eb910640 1 -- 172.21.15.164:0/4140928193 wait complete. 2024-07-26T06:46:14.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.426+0000 7fb7eb910640 1 Processor -- start 2024-07-26T06:46:14.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.426+0000 7fb7eb910640 1 -- start start 2024-07-26T06:46:14.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.426+0000 7fb7eb910640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7e4103180 0x7fb7e4076870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:14.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.426+0000 7fb7eb910640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7e4104210 0x7fb7e4075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:14.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.426+0000 7fb7eb910640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7e4104bb0 0x7fb7e4075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:14.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.426+0000 7fb7eb910640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e410ede0 con 0x7fb7e4104bb0 2024-07-26T06:46:14.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.426+0000 7fb7eb910640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e410ec60 con 0x7fb7e4103180 2024-07-26T06:46:14.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.426+0000 7fb7eb910640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb7e410ef60 con 0x7fb7e4104210 2024-07-26T06:46:14.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.427+0000 7fb7e8eab640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7e4104210 0x7fb7e4075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:14.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.427+0000 7fb7e8eab640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7e4104210 0x7fb7e4075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54858/0 (socket says 172.21.15.164:54858) 2024-07-26T06:46:14.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.427+0000 7fb7e8eab640 1 -- 172.21.15.164:0/791512180 learned_addr learned my addr 172.21.15.164:0/791512180 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-07-26T06:46:14.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.427+0000 7fb7e96ac640 1 --2- 172.21.15.164:0/791512180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7e4103180 0x7fb7e4076870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:14.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.427+0000 7fb7e9ead640 1 --2- 172.21.15.164:0/791512180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7e4104bb0 0x7fb7e4075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:14.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.427+0000 7fb7e8eab640 1 -- 172.21.15.164:0/791512180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7e4103180 msgr2=0x7fb7e4076870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:14.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.427+0000 7fb7e8eab640 1 --2- 172.21.15.164:0/791512180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7e4103180 0x7fb7e4076870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:14.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.427+0000 7fb7e8eab640 1 -- 172.21.15.164:0/791512180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7e4104bb0 msgr2=0x7fb7e4075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:14.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.427+0000 7fb7e8eab640 1 --2- 172.21.15.164:0/791512180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7e4104bb0 0x7fb7e4075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:14.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.427+0000 7fb7e8eab640 1 -- 172.21.15.164:0/791512180 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7e4075d80 con 0x7fb7e4104210 2024-07-26T06:46:14.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.427+0000 7fb7e9ead640 1 --2- 172.21.15.164:0/791512180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7e4104bb0 0x7fb7e4075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:46:14.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.427+0000 7fb7e8eab640 1 --2- 172.21.15.164:0/791512180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7e4104210 0x7fb7e4075140 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4006010 tx=0x7fb7d4004340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:14.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.427+0000 7fb7da7fc640 1 -- 172.21.15.164:0/791512180 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d403f030 con 0x7fb7e4104210 2024-07-26T06:46:14.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.427+0000 7fb7da7fc640 1 -- 172.21.15.164:0/791512180 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7d4003cb0 con 0x7fb7e4104210 2024-07-26T06:46:14.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.427+0000 7fb7eb910640 1 -- 172.21.15.164:0/791512180 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7e41a3a70 con 0x7fb7e4104210 2024-07-26T06:46:14.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.427+0000 7fb7da7fc640 1 -- 172.21.15.164:0/791512180 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4035070 con 0x7fb7e4104210 2024-07-26T06:46:14.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.427+0000 7fb7eb910640 1 -- 172.21.15.164:0/791512180 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7e41a3e60 con 0x7fb7e4104210 2024-07-26T06:46:14.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.428+0000 7fb7eb910640 1 -- 172.21.15.164:0/791512180 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7e41035f0 con 0x7fb7e4104210 2024-07-26T06:46:14.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.430+0000 7fb7da7fc640 1 -- 172.21.15.164:0/791512180 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fb7d4042030 con 0x7fb7e4104210 2024-07-26T06:46:14.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.431+0000 7fb7da7fc640 1 --2- 172.21.15.164:0/791512180 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb7d00b3c50 0x7fb7d00b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:14.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.431+0000 7fb7da7fc640 1 -- 172.21.15.164:0/791512180 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fb7d41043f0 con 0x7fb7e4104210 2024-07-26T06:46:14.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.431+0000 7fb7e96ac640 1 --2- 172.21.15.164:0/791512180 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb7d00b3c50 0x7fb7d00b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:14.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.431+0000 7fb7e96ac640 1 --2- 172.21.15.164:0/791512180 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb7d00b3c50 0x7fb7d00b6160 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000bbe0 tx=0x7fb7e000b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:14.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.435+0000 7fb7da7fc640 1 -- 172.21.15.164:0/791512180 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb7d40cbee0 con 0x7fb7e4104210 2024-07-26T06:46:14.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:14 smithi164 ceph-mon[36970]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:14.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.619+0000 7fb7eb910640 1 -- 172.21.15.164:0/791512180 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi042=a;smithi042=d;smithi042=g;smithi042=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi164=c;smithi164=f;smithi164=i;smithi164=l", "target": ["mon-mgr", ""]}) -- 0x7fb7e4063330 con 0x7fb7d00b3c50 2024-07-26T06:46:14.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.628+0000 7fb7da7fc640 1 -- 172.21.15.164:0/791512180 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fb7e4063330 con 0x7fb7d00b3c50 2024-07-26T06:46:14.629 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled mds.all update... 2024-07-26T06:46:14.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.630+0000 7fb7eb910640 1 -- 172.21.15.164:0/791512180 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb7d00b3c50 msgr2=0x7fb7d00b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:14.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.630+0000 7fb7eb910640 1 --2- 172.21.15.164:0/791512180 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb7d00b3c50 0x7fb7d00b6160 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000bbe0 tx=0x7fb7e000b390 comp rx=0 tx=0).stop 2024-07-26T06:46:14.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.630+0000 7fb7eb910640 1 -- 172.21.15.164:0/791512180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7e4104210 msgr2=0x7fb7e4075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:14.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.630+0000 7fb7eb910640 1 --2- 172.21.15.164:0/791512180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7e4104210 0x7fb7e4075140 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4006010 tx=0x7fb7d4004340 comp rx=0 tx=0).stop 2024-07-26T06:46:14.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.630+0000 7fb7eb910640 1 -- 172.21.15.164:0/791512180 shutdown_connections 2024-07-26T06:46:14.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.630+0000 7fb7eb910640 1 --2- 172.21.15.164:0/791512180 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb7d00b3c50 0x7fb7d00b6160 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:14.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.630+0000 7fb7eb910640 1 --2- 172.21.15.164:0/791512180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7e4104bb0 0x7fb7e4075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:14.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.630+0000 7fb7eb910640 1 --2- 172.21.15.164:0/791512180 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7e4104210 0x7fb7e4075140 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:14.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.630+0000 7fb7eb910640 1 --2- 172.21.15.164:0/791512180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7e4103180 0x7fb7e4076870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:14.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.630+0000 7fb7eb910640 1 -- 172.21.15.164:0/791512180 >> 172.21.15.164:0/791512180 conn(0x7fb7e40feaf0 msgr2=0x7fb7e4107510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:14.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.630+0000 7fb7eb910640 1 -- 172.21.15.164:0/791512180 shutdown_connections 2024-07-26T06:46:14.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-07-26T06:46:14.630+0000 7fb7eb910640 1 -- 172.21.15.164:0/791512180 wait complete. 2024-07-26T06:46:14.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:14 smithi042 ceph-mon[32775]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:14 smithi163 ceph-mon[36873]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:15.267 DEBUG:teuthology.orchestra.run.smithi042:mds.a> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service 2024-07-26T06:46:15.270 DEBUG:teuthology.orchestra.run.smithi042:mds.d> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.d.service 2024-07-26T06:46:15.313 DEBUG:teuthology.orchestra.run.smithi042:mds.g> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service 2024-07-26T06:46:15.357 DEBUG:teuthology.orchestra.run.smithi042:mds.j> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.j.service 2024-07-26T06:46:15.399 DEBUG:teuthology.orchestra.run.smithi163:mds.b> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service 2024-07-26T06:46:15.402 DEBUG:teuthology.orchestra.run.smithi163:mds.e> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service 2024-07-26T06:46:15.445 DEBUG:teuthology.orchestra.run.smithi163:mds.h> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service 2024-07-26T06:46:15.488 DEBUG:teuthology.orchestra.run.smithi163:mds.k> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.k.service 2024-07-26T06:46:15.531 DEBUG:teuthology.orchestra.run.smithi164:mds.c> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.c.service 2024-07-26T06:46:15.534 DEBUG:teuthology.orchestra.run.smithi164:mds.f> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.f.service 2024-07-26T06:46:15.577 DEBUG:teuthology.orchestra.run.smithi164:mds.i> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.i.service 2024-07-26T06:46:15.620 DEBUG:teuthology.orchestra.run.smithi164:mds.l> sudo journalctl -f -n 0 -u ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service 2024-07-26T06:46:15.665 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-07-26T06:46:15.665 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:46:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:15 smithi163 ceph-mon[36873]: from='client.24529 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi042=a;smithi042=d;smithi042=g;smithi042=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi164=c;smithi164=f;smithi164=i;smithi164=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:46:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:15 smithi163 ceph-mon[36873]: Saving service mds.all spec with placement smithi042=a;smithi042=d;smithi042=g;smithi042=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi164=c;smithi164=f;smithi164=i;smithi164=l;count:12 2024-07-26T06:46:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:46:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:46:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:15.884 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:15 smithi164 ceph-mon[36970]: from='client.24529 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi042=a;smithi042=d;smithi042=g;smithi042=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi164=c;smithi164=f;smithi164=i;smithi164=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:46:15.884 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:15 smithi164 ceph-mon[36970]: Saving service mds.all spec with placement smithi042=a;smithi042=d;smithi042=g;smithi042=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi164=c;smithi164=f;smithi164=i;smithi164=l;count:12 2024-07-26T06:46:15.884 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:15.884 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:46:15.884 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:15.885 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:46:15.885 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:15.885 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:15.885 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:15.885 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:15.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:15 smithi042 ceph-mon[32775]: from='client.24529 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi042=a;smithi042=d;smithi042=g;smithi042=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi164=c;smithi164=f;smithi164=i;smithi164=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:46:15.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:15 smithi042 ceph-mon[32775]: Saving service mds.all spec with placement smithi042=a;smithi042=d;smithi042=g;smithi042=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi164=c;smithi164=f;smithi164=i;smithi164=l;count:12 2024-07-26T06:46:15.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:46:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:46:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:15.926 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:46:16.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:16 smithi042 ceph-mon[32775]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:16 smithi042 ceph-mon[32775]: Deploying daemon mds.l on smithi164 2024-07-26T06:46:16.940 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:16 smithi164 ceph-mon[36970]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:16.941 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:16 smithi164 ceph-mon[36970]: Deploying daemon mds.l on smithi164 2024-07-26T06:46:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:16 smithi163 ceph-mon[36873]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:16 smithi163 ceph-mon[36873]: Deploying daemon mds.l on smithi164 2024-07-26T06:46:17.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.580+0000 7f829d56c640 1 -- 172.21.15.42:0/2745872050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8298101b70 msgr2=0x7f8298101f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:17.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.580+0000 7f829d56c640 1 --2- 172.21.15.42:0/2745872050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8298101b70 0x7f8298101f80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8288009900 tx=0x7f828802a410 comp rx=0 tx=0).stop 2024-07-26T06:46:17.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.580+0000 7f829d56c640 1 -- 172.21.15.42:0/2745872050 shutdown_connections 2024-07-26T06:46:17.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.580+0000 7f829d56c640 1 --2- 172.21.15.42:0/2745872050 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82981066b0 0x7f8298110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:17.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.580+0000 7f829d56c640 1 --2- 172.21.15.42:0/2745872050 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8298105d60 0x7f8298106170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:17.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.580+0000 7f829d56c640 1 --2- 172.21.15.42:0/2745872050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8298101b70 0x7f8298101f80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:17.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.580+0000 7f829d56c640 1 -- 172.21.15.42:0/2745872050 >> 172.21.15.42:0/2745872050 conn(0x7f82980798d0 msgr2=0x7f8298079ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:17.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.580+0000 7f829d56c640 1 -- 172.21.15.42:0/2745872050 shutdown_connections 2024-07-26T06:46:17.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.580+0000 7f829d56c640 1 -- 172.21.15.42:0/2745872050 wait complete. 2024-07-26T06:46:17.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.580+0000 7f829d56c640 1 Processor -- start 2024-07-26T06:46:17.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.581+0000 7f829d56c640 1 -- start start 2024-07-26T06:46:17.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.581+0000 7f829d56c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8298101b70 0x7f8298199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:17.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.581+0000 7f829d56c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8298105d60 0x7f829819a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:17.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.581+0000 7f829d56c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82981066b0 0x7f829819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:17.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.581+0000 7f829d56c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8298113570 con 0x7f8298105d60 2024-07-26T06:46:17.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.581+0000 7f829d56c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f82981133f0 con 0x7f8298101b70 2024-07-26T06:46:17.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.581+0000 7f829d56c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f82981136f0 con 0x7f82981066b0 2024-07-26T06:46:17.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.581+0000 7f828ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8298105d60 0x7f829819a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:17.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.581+0000 7f828ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8298105d60 0x7f829819a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36886/0 (socket says 172.21.15.42:36886) 2024-07-26T06:46:17.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.581+0000 7f828ffff640 1 -- 172.21.15.42:0/1674544238 learned_addr learned my addr 172.21.15.42:0/1674544238 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:46:17.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.581+0000 7f8296ffd640 1 --2- 172.21.15.42:0/1674544238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8298101b70 0x7f8298199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:17.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.581+0000 7f82977fe640 1 --2- 172.21.15.42:0/1674544238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82981066b0 0x7f829819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:17.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.581+0000 7f828ffff640 1 -- 172.21.15.42:0/1674544238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82981066b0 msgr2=0x7f829819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:17.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.581+0000 7f828ffff640 1 --2- 172.21.15.42:0/1674544238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82981066b0 0x7f829819f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:17.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.581+0000 7f828ffff640 1 -- 172.21.15.42:0/1674544238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8298101b70 msgr2=0x7f8298199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:17.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.581+0000 7f828ffff640 1 --2- 172.21.15.42:0/1674544238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8298101b70 0x7f8298199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:17.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.581+0000 7f828ffff640 1 -- 172.21.15.42:0/1674544238 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829819feb0 con 0x7f8298105d60 2024-07-26T06:46:17.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.582+0000 7f828ffff640 1 --2- 172.21.15.42:0/1674544238 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8298105d60 0x7f829819a4e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f828000d770 tx=0x7f828000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:17.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.582+0000 7f8294ff9640 1 -- 172.21.15.42:0/1674544238 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8280017070 con 0x7f8298105d60 2024-07-26T06:46:17.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.582+0000 7f829d56c640 1 -- 172.21.15.42:0/1674544238 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82981a01a0 con 0x7f8298105d60 2024-07-26T06:46:17.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.582+0000 7f8294ff9640 1 -- 172.21.15.42:0/1674544238 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8280009c30 con 0x7f8298105d60 2024-07-26T06:46:17.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.582+0000 7f8294ff9640 1 -- 172.21.15.42:0/1674544238 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828000f620 con 0x7f8298105d60 2024-07-26T06:46:17.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.582+0000 7f829d56c640 1 -- 172.21.15.42:0/1674544238 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82981a6900 con 0x7f8298105d60 2024-07-26T06:46:17.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.583+0000 7f829d56c640 1 -- 172.21.15.42:0/1674544238 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8298101f80 con 0x7f8298105d60 2024-07-26T06:46:17.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.585+0000 7f8294ff9640 1 -- 172.21.15.42:0/1674544238 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f828001a070 con 0x7f8298105d60 2024-07-26T06:46:17.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.585+0000 7f8294ff9640 1 --2- 172.21.15.42:0/1674544238 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82840b3bd0 0x7f82840b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:17.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.585+0000 7f8296ffd640 1 --2- 172.21.15.42:0/1674544238 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82840b3bd0 0x7f82840b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:17.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.585+0000 7f8294ff9640 1 -- 172.21.15.42:0/1674544238 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f82800e3270 con 0x7f8298105d60 2024-07-26T06:46:17.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.586+0000 7f8296ffd640 1 --2- 172.21.15.42:0/1674544238 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82840b3bd0 0x7f82840b60e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8288031440 tx=0x7f8288009560 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:17.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.590+0000 7f8294ff9640 1 -- 172.21.15.42:0/1674544238 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8280009de0 con 0x7f8298105d60 2024-07-26T06:46:17.632 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:46:17 smithi164 systemd[1]: Starting Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:46:17.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.810+0000 7f829d56c640 1 -- 172.21.15.42:0/1674544238 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f829810b630 con 0x7f8298105d60 2024-07-26T06:46:17.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.811+0000 7f8294ff9640 1 -- 172.21.15.42:0/1674544238 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+611 (secure 0 0 0) 0x7f82800aa670 con 0x7f8298105d60 2024-07-26T06:46:17.812 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:46:17.813 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":1,"btime":"2024-07-26T06:38:29:306299+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-07-26T06:46:17.813 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 1 2024-07-26T06:46:17.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.814+0000 7f829d56c640 1 -- 172.21.15.42:0/1674544238 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82840b3bd0 msgr2=0x7f82840b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:17.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.814+0000 7f829d56c640 1 --2- 172.21.15.42:0/1674544238 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82840b3bd0 0x7f82840b60e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8288031440 tx=0x7f8288009560 comp rx=0 tx=0).stop 2024-07-26T06:46:17.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.814+0000 7f829d56c640 1 -- 172.21.15.42:0/1674544238 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8298105d60 msgr2=0x7f829819a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:17.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.814+0000 7f829d56c640 1 --2- 172.21.15.42:0/1674544238 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8298105d60 0x7f829819a4e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f828000d770 tx=0x7f828000dc30 comp rx=0 tx=0).stop 2024-07-26T06:46:17.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.814+0000 7f829d56c640 1 -- 172.21.15.42:0/1674544238 shutdown_connections 2024-07-26T06:46:17.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.814+0000 7f829d56c640 1 --2- 172.21.15.42:0/1674544238 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82840b3bd0 0x7f82840b60e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:17.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.814+0000 7f829d56c640 1 --2- 172.21.15.42:0/1674544238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82981066b0 0x7f829819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:17.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.814+0000 7f829d56c640 1 --2- 172.21.15.42:0/1674544238 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8298105d60 0x7f829819a4e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:17.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.814+0000 7f829d56c640 1 --2- 172.21.15.42:0/1674544238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8298101b70 0x7f8298199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:17.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.814+0000 7f829d56c640 1 -- 172.21.15.42:0/1674544238 >> 172.21.15.42:0/1674544238 conn(0x7f82980798d0 msgr2=0x7f8298107b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:17.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.814+0000 7f829d56c640 1 -- 172.21.15.42:0/1674544238 shutdown_connections 2024-07-26T06:46:17.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:17.814+0000 7f829d56c640 1 -- 172.21.15.42:0/1674544238 wait complete. 2024-07-26T06:46:17.905 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:46:17 smithi164 podman[66239]: 2024-07-26 06:46:17.534633968 +0000 UTC m=+0.017566508 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:46:17.905 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:46:17 smithi164 podman[66239]: 2024-07-26 06:46:17.652382828 +0000 UTC m=+0.135315365 container create e9a5e5169b93b9e5b0a686baa294eee7a03693fbe18bc4733061f1edb7041fe9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-07-26T06:46:17.905 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:46:17 smithi164 podman[66239]: 2024-07-26 06:46:17.905060763 +0000 UTC m=+0.387993293 container init e9a5e5169b93b9e5b0a686baa294eee7a03693fbe18bc4733061f1edb7041fe9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-07-26T06:46:18.222 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:46:17 smithi164 podman[66239]: 2024-07-26 06:46:17.907687395 +0000 UTC m=+0.390619924 container start e9a5e5169b93b9e5b0a686baa294eee7a03693fbe18bc4733061f1edb7041fe9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-07-26T06:46:18.222 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:46:17 smithi164 ceph-mds[66258]: -- 172.21.15.164:0/491201925 <== mon.1 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55934934dc20 con 0x55934d4efa80 2024-07-26T06:46:18.222 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:46:17 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: starting mds.l at 2024-07-26T06:46:18.222 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:46:18 smithi164 bash[66239]: e9a5e5169b93b9e5b0a686baa294eee7a03693fbe18bc4733061f1edb7041fe9 2024-07-26T06:46:18.222 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:46:18 smithi164 systemd[1]: Started Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:46:18.750 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:18 smithi163 ceph-mon[36873]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:18.751 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:18 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1674544238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:18.751 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:18 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/4144543280,v1:172.21.15.164:6833/4144543280] up:boot 2024-07-26T06:46:18.751 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:18 smithi163 ceph-mon[36873]: fsmap 1 up:standby 2024-07-26T06:46:18.751 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:46:18.751 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:18.751 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:18.751 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:18.751 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:18.751 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:18.751 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:18 smithi042 ceph-mon[32775]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:18 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1674544238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:18 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/4144543280,v1:172.21.15.164:6833/4144543280] up:boot 2024-07-26T06:46:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:18 smithi042 ceph-mon[32775]: fsmap 1 up:standby 2024-07-26T06:46:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:46:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:18 smithi164 ceph-mon[36970]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:18 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1674544238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:18 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/4144543280,v1:172.21.15.164:6833/4144543280] up:boot 2024-07-26T06:46:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:18 smithi164 ceph-mon[36970]: fsmap 1 up:standby 2024-07-26T06:46:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:46:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:19.795 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:19 smithi163 ceph-mon[36873]: Deploying daemon mds.k on smithi163 2024-07-26T06:46:19.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:19 smithi042 ceph-mon[32775]: Deploying daemon mds.k on smithi163 2024-07-26T06:46:19.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:19 smithi164 ceph-mon[36970]: Deploying daemon mds.k on smithi163 2024-07-26T06:46:20.180 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jul 26 06:46:19 smithi163 systemd[1]: Starting Ceph mds.k for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:46:20.468 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jul 26 06:46:20 smithi163 podman[66097]: 2024-07-26 06:46:20.082386375 +0000 UTC m=+0.017237406 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:46:20.469 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jul 26 06:46:20 smithi163 podman[66097]: 2024-07-26 06:46:20.199583961 +0000 UTC m=+0.134435063 container create 2488c0aef30a85cd93eb9999d31c927d21fd8aa2fd29ff29e1ee2a1c045622b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240724, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T06:46:20.469 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jul 26 06:46:20 smithi163 podman[66097]: 2024-07-26 06:46:20.468649116 +0000 UTC m=+0.403500152 container init 2488c0aef30a85cd93eb9999d31c927d21fd8aa2fd29ff29e1ee2a1c045622b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-k, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD) 2024-07-26T06:46:20.576 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:46:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:20 smithi163 ceph-mon[36873]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:20.757 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jul 26 06:46:20 smithi163 podman[66097]: 2024-07-26 06:46:20.471322294 +0000 UTC m=+0.406173319 container start 2488c0aef30a85cd93eb9999d31c927d21fd8aa2fd29ff29e1ee2a1c045622b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-k, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-07-26T06:46:20.757 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jul 26 06:46:20 smithi163 ceph-mds[66116]: -- 172.21.15.163:0/733941788 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565120757c20 con 0x5651248f9600 2024-07-26T06:46:20.758 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jul 26 06:46:20 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-k[66112]: starting mds.k at 2024-07-26T06:46:20.758 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jul 26 06:46:20 smithi163 bash[66097]: 2488c0aef30a85cd93eb9999d31c927d21fd8aa2fd29ff29e1ee2a1c045622b6 2024-07-26T06:46:20.758 INFO:journalctl@ceph.mds.k.smithi163.stdout:Jul 26 06:46:20 smithi163 systemd[1]: Started Ceph mds.k for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:46:20.869 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:46:20.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:20 smithi042 ceph-mon[32775]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:20 smithi164 ceph-mon[36970]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:21.879 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:21.880 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:21 smithi163 ceph-mon[36873]: Deploying daemon mds.j on smithi042 2024-07-26T06:46:21.880 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:21 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6832/845444723,v1:172.21.15.163:6833/845444723] up:boot 2024-07-26T06:46:21.880 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:21 smithi163 ceph-mon[36873]: fsmap 2 up:standby 2024-07-26T06:46:21.880 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T06:46:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:21 smithi164 ceph-mon[36970]: Deploying daemon mds.j on smithi042 2024-07-26T06:46:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:21 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6832/845444723,v1:172.21.15.163:6833/845444723] up:boot 2024-07-26T06:46:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:21 smithi164 ceph-mon[36970]: fsmap 2 up:standby 2024-07-26T06:46:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T06:46:22.045 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:22.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:21 smithi042 ceph-mon[32775]: Deploying daemon mds.j on smithi042 2024-07-26T06:46:22.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:21 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6832/845444723,v1:172.21.15.163:6833/845444723] up:boot 2024-07-26T06:46:22.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:21 smithi042 ceph-mon[32775]: fsmap 2 up:standby 2024-07-26T06:46:22.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T06:46:22.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.738+0000 7fe9f9c80640 1 -- 172.21.15.42:0/1959968020 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9f410d820 msgr2=0x7fe9f410fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:22.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.738+0000 7fe9f9c80640 1 --2- 172.21.15.42:0/1959968020 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9f410d820 0x7fe9f410fd30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009900 tx=0x7fe9ec02a410 comp rx=0 tx=0).stop 2024-07-26T06:46:22.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.738+0000 7fe9f9c80640 1 -- 172.21.15.42:0/1959968020 shutdown_connections 2024-07-26T06:46:22.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.738+0000 7fe9f9c80640 1 --2- 172.21.15.42:0/1959968020 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9f410d820 0x7fe9f410fd30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:22.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.738+0000 7fe9f9c80640 1 --2- 172.21.15.42:0/1959968020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9f4104ca0 0x7fe9f410d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:22.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.738+0000 7fe9f9c80640 1 --2- 172.21.15.42:0/1959968020 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9f4104270 0x7fe9f4104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:22.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.738+0000 7fe9f9c80640 1 -- 172.21.15.42:0/1959968020 >> 172.21.15.42:0/1959968020 conn(0x7fe9f4079160 msgr2=0x7fe9f4079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:22.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.738+0000 7fe9f9c80640 1 -- 172.21.15.42:0/1959968020 shutdown_connections 2024-07-26T06:46:22.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.738+0000 7fe9f9c80640 1 -- 172.21.15.42:0/1959968020 wait complete. 2024-07-26T06:46:22.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.739+0000 7fe9f9c80640 1 Processor -- start 2024-07-26T06:46:22.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.739+0000 7fe9f9c80640 1 -- start start 2024-07-26T06:46:22.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f9c80640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9f4104270 0x7fe9f4199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:22.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f9c80640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9f4104ca0 0x7fe9f419a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:22.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f9c80640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9f410d820 0x7fe9f419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:22.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f9c80640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f41134e0 con 0x7fe9f4104270 2024-07-26T06:46:22.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f9c80640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f4113360 con 0x7fe9f4104ca0 2024-07-26T06:46:22.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f9c80640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f4113660 con 0x7fe9f410d820 2024-07-26T06:46:22.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9f4104270 0x7fe9f4199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:22.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9f4104270 0x7fe9f4199e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36910/0 (socket says 172.21.15.42:36910) 2024-07-26T06:46:22.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f37fe640 1 -- 172.21.15.42:0/4149803393 learned_addr learned my addr 172.21.15.42:0/4149803393 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:46:22.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f2ffd640 1 --2- 172.21.15.42:0/4149803393 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9f4104ca0 0x7fe9f419a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:22.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f3fff640 1 --2- 172.21.15.42:0/4149803393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9f410d820 0x7fe9f419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:22.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f37fe640 1 -- 172.21.15.42:0/4149803393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9f410d820 msgr2=0x7fe9f419f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:22.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f37fe640 1 --2- 172.21.15.42:0/4149803393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9f410d820 0x7fe9f419f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:22.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f37fe640 1 -- 172.21.15.42:0/4149803393 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9f4104ca0 msgr2=0x7fe9f419a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:22.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f37fe640 1 --2- 172.21.15.42:0/4149803393 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9f4104ca0 0x7fe9f419a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:22.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f37fe640 1 -- 172.21.15.42:0/4149803393 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f419fde0 con 0x7fe9f4104270 2024-07-26T06:46:22.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f37fe640 1 --2- 172.21.15.42:0/4149803393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9f4104270 0x7fe9f4199e70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400eaa0 tx=0x7fe9e400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:22.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f0ff9640 1 -- 172.21.15.42:0/4149803393 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4016070 con 0x7fe9f4104270 2024-07-26T06:46:22.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f0ff9640 1 -- 172.21.15.42:0/4149803393 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe9e4003e20 con 0x7fe9f4104270 2024-07-26T06:46:22.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f9c80640 1 -- 172.21.15.42:0/4149803393 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9f41a00d0 con 0x7fe9f4104270 2024-07-26T06:46:22.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.740+0000 7fe9f0ff9640 1 -- 172.21.15.42:0/4149803393 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e40116d0 con 0x7fe9f4104270 2024-07-26T06:46:22.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.741+0000 7fe9f9c80640 1 -- 172.21.15.42:0/4149803393 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9f41a6830 con 0x7fe9f4104270 2024-07-26T06:46:22.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.741+0000 7fe9f9c80640 1 -- 172.21.15.42:0/4149803393 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9f4104680 con 0x7fe9f4104270 2024-07-26T06:46:22.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.743+0000 7fe9f0ff9640 1 -- 172.21.15.42:0/4149803393 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fe9e4019070 con 0x7fe9f4104270 2024-07-26T06:46:22.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.744+0000 7fe9f0ff9640 1 --2- 172.21.15.42:0/4149803393 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe9dc0b3bd0 0x7fe9dc0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:22.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.744+0000 7fe9f2ffd640 1 --2- 172.21.15.42:0/4149803393 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe9dc0b3bd0 0x7fe9dc0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:22.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.744+0000 7fe9f0ff9640 1 -- 172.21.15.42:0/4149803393 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fe9e40e3b70 con 0x7fe9f4104270 2024-07-26T06:46:22.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.744+0000 7fe9f2ffd640 1 --2- 172.21.15.42:0/4149803393 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe9dc0b3bd0 0x7fe9dc0b60e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe9f410f4a0 tx=0x7fe9e8005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:22.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.747+0000 7fe9f0ff9640 1 -- 172.21.15.42:0/4149803393 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe9e40ab010 con 0x7fe9f4104270 2024-07-26T06:46:22.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:22 smithi042 ceph-mon[32775]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:22 smithi164 ceph-mon[36970]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:22.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.979+0000 7fe9f9c80640 1 -- 172.21.15.42:0/4149803393 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9f419afa0 con 0x7fe9f4104270 2024-07-26T06:46:22.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.980+0000 7fe9f0ff9640 1 -- 172.21.15.42:0/4149803393 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2263 (secure 0 0 0) 0x7fe9e40af560 con 0x7fe9f4104270 2024-07-26T06:46:22.983 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:46:22.983 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":3,"btime":"2024-07-26T06:46:20:640878+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24541,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24557,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-07-26T06:46:22.984 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 3 2024-07-26T06:46:22.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.982+0000 7fe9f9c80640 1 -- 172.21.15.42:0/4149803393 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe9dc0b3bd0 msgr2=0x7fe9dc0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:22.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.982+0000 7fe9f9c80640 1 --2- 172.21.15.42:0/4149803393 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe9dc0b3bd0 0x7fe9dc0b60e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe9f410f4a0 tx=0x7fe9e8005f90 comp rx=0 tx=0).stop 2024-07-26T06:46:22.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.982+0000 7fe9f9c80640 1 -- 172.21.15.42:0/4149803393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9f4104270 msgr2=0x7fe9f4199e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:22.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.982+0000 7fe9f9c80640 1 --2- 172.21.15.42:0/4149803393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9f4104270 0x7fe9f4199e70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400eaa0 tx=0x7fe9e400ef60 comp rx=0 tx=0).stop 2024-07-26T06:46:22.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.982+0000 7fe9f9c80640 1 -- 172.21.15.42:0/4149803393 shutdown_connections 2024-07-26T06:46:22.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.982+0000 7fe9f9c80640 1 --2- 172.21.15.42:0/4149803393 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe9dc0b3bd0 0x7fe9dc0b60e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:22.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.982+0000 7fe9f9c80640 1 --2- 172.21.15.42:0/4149803393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9f410d820 0x7fe9f419f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:22.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.982+0000 7fe9f9c80640 1 --2- 172.21.15.42:0/4149803393 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe9f4104ca0 0x7fe9f419a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:22.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.982+0000 7fe9f9c80640 1 --2- 172.21.15.42:0/4149803393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9f4104270 0x7fe9f4199e70 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:22.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.982+0000 7fe9f9c80640 1 -- 172.21.15.42:0/4149803393 >> 172.21.15.42:0/4149803393 conn(0x7fe9f4079160 msgr2=0x7fe9f40fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:22.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.982+0000 7fe9f9c80640 1 -- 172.21.15.42:0/4149803393 shutdown_connections 2024-07-26T06:46:22.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:22.983+0000 7fe9f9c80640 1 -- 172.21.15.42:0/4149803393 wait complete. 2024-07-26T06:46:23.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:22 smithi163 ceph-mon[36873]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:23 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/4149803393' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:24.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:23 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/4149803393' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:24.063 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:23 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/4149803393' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:24.645 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:46:24 smithi042 systemd[1]: Starting Ceph mds.j for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:46:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:24 smithi164 ceph-mon[36970]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:25.001 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:24 smithi042 ceph-mon[32775]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:25.002 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:46:24 smithi042 podman[75774]: 2024-07-26 06:46:24.648483545 +0000 UTC m=+0.018398076 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:46:25.002 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:46:24 smithi042 podman[75774]: 2024-07-26 06:46:24.766128044 +0000 UTC m=+0.136042608 container create 68440941ad331dda7900f6e75f9ac7502b50e8b1dc84d6ff3e1771b98b526003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-07-26T06:46:25.002 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:46:25 smithi042 podman[75774]: 2024-07-26 06:46:25.002277932 +0000 UTC m=+0.372192454 container init 68440941ad331dda7900f6e75f9ac7502b50e8b1dc84d6ff3e1771b98b526003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_BRANCH=HEAD) 2024-07-26T06:46:25.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:24 smithi163 ceph-mon[36873]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:25.394 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:46:25 smithi042 podman[75774]: 2024-07-26 06:46:25.005391594 +0000 UTC m=+0.375306120 container start 68440941ad331dda7900f6e75f9ac7502b50e8b1dc84d6ff3e1771b98b526003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724) 2024-07-26T06:46:25.395 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:46:25 smithi042 ceph-mds[75793]: -- 172.21.15.42:0/1374551490 <== mon.0 v2:172.21.15.42:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5654c4ea1c20 con 0x5654c9043a80 2024-07-26T06:46:25.395 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:46:25 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: starting mds.j at 2024-07-26T06:46:25.395 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:46:25 smithi042 bash[75774]: 68440941ad331dda7900f6e75f9ac7502b50e8b1dc84d6ff3e1771b98b526003 2024-07-26T06:46:25.395 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:46:25 smithi042 systemd[1]: Started Ceph mds.j for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:46:25.717 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:46:26.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:26 smithi042 ceph-mon[32775]: Deploying daemon mds.d on smithi042 2024-07-26T06:46:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:26 smithi042 ceph-mon[32775]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:26 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6834/3548166563,v1:172.21.15.42:6835/3548166563] up:boot 2024-07-26T06:46:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:26 smithi042 ceph-mon[32775]: fsmap 3 up:standby 2024-07-26T06:46:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T06:46:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:26 smithi164 ceph-mon[36970]: Deploying daemon mds.d on smithi042 2024-07-26T06:46:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:26 smithi164 ceph-mon[36970]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:26 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6834/3548166563,v1:172.21.15.42:6835/3548166563] up:boot 2024-07-26T06:46:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:26 smithi164 ceph-mon[36970]: fsmap 3 up:standby 2024-07-26T06:46:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T06:46:26.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:26 smithi163 ceph-mon[36873]: Deploying daemon mds.d on smithi042 2024-07-26T06:46:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:26 smithi163 ceph-mon[36873]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:26 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6834/3548166563,v1:172.21.15.42:6835/3548166563] up:boot 2024-07-26T06:46:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:26 smithi163 ceph-mon[36873]: fsmap 3 up:standby 2024-07-26T06:46:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T06:46:26.754 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:46:27.395 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:46:27 smithi042 systemd[1]: Starting Ceph mds.d for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:46:27.935 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:46:27 smithi042 podman[76172]: 2024-07-26 06:46:27.568670174 +0000 UTC m=+0.017427811 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:46:27.935 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:46:27 smithi042 podman[76172]: 2024-07-26 06:46:27.669652276 +0000 UTC m=+0.118409899 container create 4e393b9ca4095f4d7dfbcbdf5d466c8c74c17d726e5c27b040584902c7f6d270 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T06:46:27.935 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:46:27 smithi042 podman[76172]: 2024-07-26 06:46:27.897478057 +0000 UTC m=+0.346235694 container init 4e393b9ca4095f4d7dfbcbdf5d466c8c74c17d726e5c27b040584902c7f6d270 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240724) 2024-07-26T06:46:27.935 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:46:27 smithi042 podman[76172]: 2024-07-26 06:46:27.900697012 +0000 UTC m=+0.349454639 container start 4e393b9ca4095f4d7dfbcbdf5d466c8c74c17d726e5c27b040584902c7f6d270 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0) 2024-07-26T06:46:28.394 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:46:27 smithi042 ceph-mds[76191]: -- 172.21.15.42:0/3544806490 <== mon.0 v2:172.21.15.42:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5559f6c73860 con 0x5559fae15600 2024-07-26T06:46:28.395 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:46:27 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: starting mds.d at 2024-07-26T06:46:28.395 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:46:27 smithi042 bash[76172]: 4e393b9ca4095f4d7dfbcbdf5d466c8c74c17d726e5c27b040584902c7f6d270 2024-07-26T06:46:28.395 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:46:28 smithi042 systemd[1]: Started Ceph mds.d for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:46:29.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.041+0000 7f9985933640 1 -- 172.21.15.42:0/2925305291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9980108720 msgr2=0x7f998010ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:29.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.041+0000 7f9985933640 1 --2- 172.21.15.42:0/2925305291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9980108720 0x7f998010ef50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9978009ca0 tx=0x7f997802a8c0 comp rx=0 tx=0).stop 2024-07-26T06:46:29.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.041+0000 7f9985933640 1 -- 172.21.15.42:0/2925305291 shutdown_connections 2024-07-26T06:46:29.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.041+0000 7f9985933640 1 --2- 172.21.15.42:0/2925305291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9980108720 0x7f998010ef50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:29.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.041+0000 7f9985933640 1 --2- 172.21.15.42:0/2925305291 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9980107dd0 0x7f99801081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:29.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.041+0000 7f9985933640 1 --2- 172.21.15.42:0/2925305291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9980102490 0x7f99801028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:29.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.041+0000 7f9985933640 1 -- 172.21.15.42:0/2925305291 >> 172.21.15.42:0/2925305291 conn(0x7f9980079160 msgr2=0x7f9980079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:29.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.041+0000 7f9985933640 1 -- 172.21.15.42:0/2925305291 shutdown_connections 2024-07-26T06:46:29.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.041+0000 7f9985933640 1 -- 172.21.15.42:0/2925305291 wait complete. 2024-07-26T06:46:29.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f9985933640 1 Processor -- start 2024-07-26T06:46:29.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f9985933640 1 -- start start 2024-07-26T06:46:29.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f9985933640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9980102490 0x7f99800ff0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:29.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f9985933640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9980107dd0 0x7f99800ff5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:29.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f9985933640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9980108720 0x7f99800ffb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:29.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f9985933640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f99801123a0 con 0x7f9980107dd0 2024-07-26T06:46:29.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f9985933640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9980112220 con 0x7f9980102490 2024-07-26T06:46:29.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f9985933640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9980112520 con 0x7f9980108720 2024-07-26T06:46:29.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f9977fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9980107dd0 0x7f99800ff5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:29.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f9977fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9980107dd0 0x7f99800ff5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39312/0 (socket says 172.21.15.42:39312) 2024-07-26T06:46:29.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f9977fff640 1 -- 172.21.15.42:0/1268756116 learned_addr learned my addr 172.21.15.42:0/1268756116 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:46:29.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f997effd640 1 --2- 172.21.15.42:0/1268756116 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9980102490 0x7f99800ff0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:29.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f997f7fe640 1 --2- 172.21.15.42:0/1268756116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9980108720 0x7f99800ffb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:29.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f9977fff640 1 -- 172.21.15.42:0/1268756116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9980108720 msgr2=0x7f99800ffb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:29.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f9977fff640 1 --2- 172.21.15.42:0/1268756116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9980108720 0x7f99800ffb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:29.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f9977fff640 1 -- 172.21.15.42:0/1268756116 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9980102490 msgr2=0x7f99800ff0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:29.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f9977fff640 1 --2- 172.21.15.42:0/1268756116 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9980102490 0x7f99800ff0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:29.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.042+0000 7f9977fff640 1 -- 172.21.15.42:0/1268756116 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99801a8070 con 0x7f9980107dd0 2024-07-26T06:46:29.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.043+0000 7f9977fff640 1 --2- 172.21.15.42:0/1268756116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9980107dd0 0x7f99800ff5e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f996800c970 tx=0x7f996800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:29.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.043+0000 7f9984931640 1 -- 172.21.15.42:0/1268756116 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9968014070 con 0x7f9980107dd0 2024-07-26T06:46:29.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.043+0000 7f9984931640 1 -- 172.21.15.42:0/1268756116 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9968004440 con 0x7f9980107dd0 2024-07-26T06:46:29.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.043+0000 7f9984931640 1 -- 172.21.15.42:0/1268756116 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9968003e50 con 0x7f9980107dd0 2024-07-26T06:46:29.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.043+0000 7f9985933640 1 -- 172.21.15.42:0/1268756116 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99801a8360 con 0x7f9980107dd0 2024-07-26T06:46:29.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.043+0000 7f9985933640 1 -- 172.21.15.42:0/1268756116 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99801a88a0 con 0x7f9980107dd0 2024-07-26T06:46:29.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.044+0000 7f9985933640 1 -- 172.21.15.42:0/1268756116 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99801028a0 con 0x7f9980107dd0 2024-07-26T06:46:29.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.047+0000 7f9984931640 1 -- 172.21.15.42:0/1268756116 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f9968017070 con 0x7f9980107dd0 2024-07-26T06:46:29.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.047+0000 7f9984931640 1 --2- 172.21.15.42:0/1268756116 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f996c0b3d70 0x7f996c0b6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:29.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.048+0000 7f997effd640 1 --2- 172.21.15.42:0/1268756116 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f996c0b3d70 0x7f996c0b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:29.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.048+0000 7f9984931640 1 -- 172.21.15.42:0/1268756116 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f99680e3a10 con 0x7f9980107dd0 2024-07-26T06:46:29.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.048+0000 7f997effd640 1 --2- 172.21.15.42:0/1268756116 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f996c0b3d70 0x7f996c0b6280 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9980100af0 tx=0x7f997000b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:29.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.050+0000 7f9984931640 1 -- 172.21.15.42:0/1268756116 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f996801e020 con 0x7f9980107dd0 2024-07-26T06:46:29.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:29 smithi042 ceph-mon[32775]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:29.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:29 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6836/1420377891,v1:172.21.15.42:6837/1420377891] up:boot 2024-07-26T06:46:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:29 smithi042 ceph-mon[32775]: fsmap 4 up:standby 2024-07-26T06:46:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T06:46:29.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.276+0000 7f9985933640 1 -- 172.21.15.42:0/1268756116 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99801003a0 con 0x7f9980107dd0 2024-07-26T06:46:29.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.277+0000 7f9984931640 1 -- 172.21.15.42:0/1268756116 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3913 (secure 0 0 0) 0x7f99680ab500 con 0x7f9980107dd0 2024-07-26T06:46:29.278 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:46:29.278 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":5,"btime":"2024-07-26T06:46:28:357535+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24541,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24557,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-07-26T06:46:29.278 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 5 2024-07-26T06:46:29.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.279+0000 7f9985933640 1 -- 172.21.15.42:0/1268756116 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f996c0b3d70 msgr2=0x7f996c0b6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:29.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.279+0000 7f9985933640 1 --2- 172.21.15.42:0/1268756116 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f996c0b3d70 0x7f996c0b6280 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9980100af0 tx=0x7f997000b390 comp rx=0 tx=0).stop 2024-07-26T06:46:29.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.279+0000 7f9985933640 1 -- 172.21.15.42:0/1268756116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9980107dd0 msgr2=0x7f99800ff5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:29.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.279+0000 7f9985933640 1 --2- 172.21.15.42:0/1268756116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9980107dd0 0x7f99800ff5e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f996800c970 tx=0x7f996800ce30 comp rx=0 tx=0).stop 2024-07-26T06:46:29.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.279+0000 7f9985933640 1 -- 172.21.15.42:0/1268756116 shutdown_connections 2024-07-26T06:46:29.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.279+0000 7f9985933640 1 --2- 172.21.15.42:0/1268756116 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f996c0b3d70 0x7f996c0b6280 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:29.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.279+0000 7f9985933640 1 --2- 172.21.15.42:0/1268756116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9980108720 0x7f99800ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:29.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.279+0000 7f9985933640 1 --2- 172.21.15.42:0/1268756116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9980107dd0 0x7f99800ff5e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:29.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.279+0000 7f9985933640 1 --2- 172.21.15.42:0/1268756116 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9980102490 0x7f99800ff0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:29.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.279+0000 7f9985933640 1 -- 172.21.15.42:0/1268756116 >> 172.21.15.42:0/1268756116 conn(0x7f9980079160 msgr2=0x7f998006ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:29.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.279+0000 7f9985933640 1 -- 172.21.15.42:0/1268756116 shutdown_connections 2024-07-26T06:46:29.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:29.279+0000 7f9985933640 1 -- 172.21.15.42:0/1268756116 wait complete. 2024-07-26T06:46:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:29 smithi164 ceph-mon[36970]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:29 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6836/1420377891,v1:172.21.15.42:6837/1420377891] up:boot 2024-07-26T06:46:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:29 smithi164 ceph-mon[36970]: fsmap 4 up:standby 2024-07-26T06:46:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T06:46:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:29 smithi163 ceph-mon[36873]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:29 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6836/1420377891,v1:172.21.15.42:6837/1420377891] up:boot 2024-07-26T06:46:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:29 smithi163 ceph-mon[36873]: fsmap 4 up:standby 2024-07-26T06:46:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T06:46:30.058 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:46:29 smithi163 systemd[1]: Starting Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:46:30.355 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:30 smithi163 ceph-mon[36873]: Deploying daemon mds.e on smithi163 2024-07-26T06:46:30.355 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:30 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1268756116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:30.359 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:46:30 smithi163 podman[66411]: 2024-07-26 06:46:30.104750651 +0000 UTC m=+0.017404299 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:46:30.359 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:46:30 smithi163 podman[66411]: 2024-07-26 06:46:30.237254701 +0000 UTC m=+0.149908380 container create c47b4288f0737602408393f380d6895893ae90b6b7388dd5840f1a27ba7c98a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-07-26T06:46:30.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:30 smithi042 ceph-mon[32775]: Deploying daemon mds.e on smithi163 2024-07-26T06:46:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:30 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1268756116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:30 smithi164 ceph-mon[36970]: Deploying daemon mds.e on smithi163 2024-07-26T06:46:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:30 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1268756116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:30.757 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:46:30 smithi163 podman[66411]: 2024-07-26 06:46:30.48194108 +0000 UTC m=+0.394594714 container init c47b4288f0737602408393f380d6895893ae90b6b7388dd5840f1a27ba7c98a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, ceph=True) 2024-07-26T06:46:30.757 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:46:30 smithi163 podman[66411]: 2024-07-26 06:46:30.484723929 +0000 UTC m=+0.397377571 container start c47b4288f0737602408393f380d6895893ae90b6b7388dd5840f1a27ba7c98a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-07-26T06:46:30.757 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:46:30 smithi163 ceph-mds[66430]: -- 172.21.15.163:0/3480166631 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d7e9f95860 con 0x55d7ee137600 2024-07-26T06:46:30.757 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:46:30 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: starting mds.e at 2024-07-26T06:46:30.757 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:46:30 smithi163 bash[66411]: c47b4288f0737602408393f380d6895893ae90b6b7388dd5840f1a27ba7c98a4 2024-07-26T06:46:30.757 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:46:30 smithi163 systemd[1]: Started Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:46:31.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:31 smithi164 ceph-mon[36970]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:31.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:31.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:31.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:31.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:31.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:31.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:31.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:31 smithi042 ceph-mon[32775]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:31.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:31.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:31 smithi163 ceph-mon[36873]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:31.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:32.001 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:46:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:46:32.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:32 smithi164 ceph-mon[36970]: Deploying daemon mds.f on smithi164 2024-07-26T06:46:32.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:32 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] up:boot 2024-07-26T06:46:32.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:32 smithi164 ceph-mon[36970]: fsmap 5 up:standby 2024-07-26T06:46:32.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:46:32.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:32 smithi042 ceph-mon[32775]: Deploying daemon mds.f on smithi164 2024-07-26T06:46:32.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:32 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] up:boot 2024-07-26T06:46:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:32 smithi042 ceph-mon[32775]: fsmap 5 up:standby 2024-07-26T06:46:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:46:32.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:32 smithi163 ceph-mon[36873]: Deploying daemon mds.f on smithi164 2024-07-26T06:46:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:32 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] up:boot 2024-07-26T06:46:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:32 smithi163 ceph-mon[36873]: fsmap 5 up:standby 2024-07-26T06:46:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:46:32.722 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:46:32 smithi164 systemd[1]: Starting Ceph mds.f for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:46:33.043 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:46:32 smithi164 podman[66544]: 2024-07-26 06:46:32.670831056 +0000 UTC m=+0.017438061 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:46:33.043 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:46:32 smithi164 podman[66544]: 2024-07-26 06:46:32.823110921 +0000 UTC m=+0.169717912 container create 20df7fdbf85ab4ff19c27a596b99a13f8ec104012f9225145dc8d105dc18121d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T06:46:33.043 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:46:33 smithi164 podman[66544]: 2024-07-26 06:46:33.042240135 +0000 UTC m=+0.388847127 container init 20df7fdbf85ab4ff19c27a596b99a13f8ec104012f9225145dc8d105dc18121d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T06:46:33.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:33 smithi163 ceph-mon[36873]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:33.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:33 smithi042 ceph-mon[32775]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:33.422 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:33 smithi164 ceph-mon[36970]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:33.423 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:46:33 smithi164 podman[66544]: 2024-07-26 06:46:33.045391549 +0000 UTC m=+0.391998551 container start 20df7fdbf85ab4ff19c27a596b99a13f8ec104012f9225145dc8d105dc18121d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-f, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T06:46:33.423 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:46:33 smithi164 ceph-mds[66563]: -- 172.21.15.164:0/4012281541 <== mon.1 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5626708c3860 con 0x562674a65180 2024-07-26T06:46:33.423 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:46:33 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-f[66559]: starting mds.f at 2024-07-26T06:46:33.423 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:46:33 smithi164 bash[66544]: 20df7fdbf85ab4ff19c27a596b99a13f8ec104012f9225145dc8d105dc18121d 2024-07-26T06:46:33.423 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:46:33 smithi164 systemd[1]: Started Ceph mds.f for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:46:34.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.041+0000 7f2c2bc9a640 1 -- 172.21.15.42:0/2371159670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c24104d40 msgr2=0x7f2c2410d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.041+0000 7f2c2bc9a640 1 --2- 172.21.15.42:0/2371159670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c24104d40 0x7f2c2410d290 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2c14009900 tx=0x7f2c1402a410 comp rx=0 tx=0).stop 2024-07-26T06:46:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.041+0000 7f2c2bc9a640 1 -- 172.21.15.42:0/2371159670 shutdown_connections 2024-07-26T06:46:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.041+0000 7f2c2bc9a640 1 --2- 172.21.15.42:0/2371159670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c2410d850 0x7f2c2410fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.041+0000 7f2c2bc9a640 1 --2- 172.21.15.42:0/2371159670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c24104d40 0x7f2c2410d290 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.041+0000 7f2c2bc9a640 1 --2- 172.21.15.42:0/2371159670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2c241043f0 0x7f2c24104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.041+0000 7f2c2bc9a640 1 -- 172.21.15.42:0/2371159670 >> 172.21.15.42:0/2371159670 conn(0x7f2c24079140 msgr2=0x7f2c24079550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.041+0000 7f2c2bc9a640 1 -- 172.21.15.42:0/2371159670 shutdown_connections 2024-07-26T06:46:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.041+0000 7f2c2bc9a640 1 -- 172.21.15.42:0/2371159670 wait complete. 2024-07-26T06:46:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c2bc9a640 1 Processor -- start 2024-07-26T06:46:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c2bc9a640 1 -- start start 2024-07-26T06:46:34.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c2bc9a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c241043f0 0x7f2c24199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c2bc9a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c24104d40 0x7f2c2419a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c2bc9a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2c2410d850 0x7f2c2419f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c2bc9a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2c24113480 con 0x7f2c24104d40 2024-07-26T06:46:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c2bc9a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2c24113300 con 0x7f2c2410d850 2024-07-26T06:46:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c2bc9a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2c24113600 con 0x7f2c241043f0 2024-07-26T06:46:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c29235640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c24104d40 0x7f2c2419a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c29235640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c24104d40 0x7f2c2419a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39332/0 (socket says 172.21.15.42:39332) 2024-07-26T06:46:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c29235640 1 -- 172.21.15.42:0/197866637 learned_addr learned my addr 172.21.15.42:0/197866637 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:46:34.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c29a36640 1 --2- 172.21.15.42:0/197866637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c241043f0 0x7f2c24199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:34.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c2a237640 1 --2- 172.21.15.42:0/197866637 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2c2410d850 0x7f2c2419f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:34.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c29235640 1 -- 172.21.15.42:0/197866637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c241043f0 msgr2=0x7f2c24199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:34.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c29235640 1 --2- 172.21.15.42:0/197866637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c241043f0 0x7f2c24199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:34.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c29235640 1 -- 172.21.15.42:0/197866637 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2c2410d850 msgr2=0x7f2c2419f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:34.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c29235640 1 --2- 172.21.15.42:0/197866637 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2c2410d850 0x7f2c2419f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:34.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.042+0000 7f2c29235640 1 -- 172.21.15.42:0/197866637 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c2419fec0 con 0x7f2c24104d40 2024-07-26T06:46:34.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.043+0000 7f2c29235640 1 --2- 172.21.15.42:0/197866637 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c24104d40 0x7f2c2419a4f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2c1402a930 tx=0x7f2c14004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:34.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.043+0000 7f2c1affd640 1 -- 172.21.15.42:0/197866637 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1403f030 con 0x7f2c24104d40 2024-07-26T06:46:34.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.043+0000 7f2c2bc9a640 1 -- 172.21.15.42:0/197866637 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c241a0150 con 0x7f2c24104d40 2024-07-26T06:46:34.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.043+0000 7f2c1affd640 1 -- 172.21.15.42:0/197866637 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c14003aa0 con 0x7f2c24104d40 2024-07-26T06:46:34.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.043+0000 7f2c1affd640 1 -- 172.21.15.42:0/197866637 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c14035070 con 0x7f2c24104d40 2024-07-26T06:46:34.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.043+0000 7f2c2bc9a640 1 -- 172.21.15.42:0/197866637 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c241a6910 con 0x7f2c24104d40 2024-07-26T06:46:34.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.044+0000 7f2c2bc9a640 1 -- 172.21.15.42:0/197866637 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c2406bcb0 con 0x7f2c24104d40 2024-07-26T06:46:34.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.046+0000 7f2c1affd640 1 -- 172.21.15.42:0/197866637 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f2c14042030 con 0x7f2c24104d40 2024-07-26T06:46:34.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.046+0000 7f2c1affd640 1 --2- 172.21.15.42:0/197866637 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f2c100b3bd0 0x7f2c100b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:34.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.046+0000 7f2c29a36640 1 --2- 172.21.15.42:0/197866637 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f2c100b3bd0 0x7f2c100b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:34.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.046+0000 7f2c1affd640 1 -- 172.21.15.42:0/197866637 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f2c14104250 con 0x7f2c24104d40 2024-07-26T06:46:34.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.047+0000 7f2c29a36640 1 --2- 172.21.15.42:0/197866637 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f2c100b3bd0 0x7f2c100b60e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000bbe0 tx=0x7f2c2000b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:34.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.050+0000 7f2c1affd640 1 -- 172.21.15.42:0/197866637 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2c140cbd40 con 0x7f2c24104d40 2024-07-26T06:46:34.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.270+0000 7f2c2bc9a640 1 -- 172.21.15.42:0/197866637 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c241089b0 con 0x7f2c24104d40 2024-07-26T06:46:34.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.271+0000 7f2c1affd640 1 -- 172.21.15.42:0/197866637 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5566 (secure 0 0 0) 0x7f2c140cb640 con 0x7f2c24104d40 2024-07-26T06:46:34.272 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:46:34.272 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":7,"btime":"2024-07-26T06:46:34:072683+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24541,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24557,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24571,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24578,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-07-26T06:46:34.272 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 7 2024-07-26T06:46:34.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.273+0000 7f2c2bc9a640 1 -- 172.21.15.42:0/197866637 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f2c100b3bd0 msgr2=0x7f2c100b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:34.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.273+0000 7f2c2bc9a640 1 --2- 172.21.15.42:0/197866637 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f2c100b3bd0 0x7f2c100b60e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000bbe0 tx=0x7f2c2000b390 comp rx=0 tx=0).stop 2024-07-26T06:46:34.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.273+0000 7f2c2bc9a640 1 -- 172.21.15.42:0/197866637 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c24104d40 msgr2=0x7f2c2419a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:34.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.273+0000 7f2c2bc9a640 1 --2- 172.21.15.42:0/197866637 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c24104d40 0x7f2c2419a4f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2c1402a930 tx=0x7f2c14004270 comp rx=0 tx=0).stop 2024-07-26T06:46:34.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.273+0000 7f2c2bc9a640 1 -- 172.21.15.42:0/197866637 shutdown_connections 2024-07-26T06:46:34.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.273+0000 7f2c2bc9a640 1 --2- 172.21.15.42:0/197866637 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f2c100b3bd0 0x7f2c100b60e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:34.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.273+0000 7f2c2bc9a640 1 --2- 172.21.15.42:0/197866637 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2c2410d850 0x7f2c2419f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:34.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.273+0000 7f2c2bc9a640 1 --2- 172.21.15.42:0/197866637 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c24104d40 0x7f2c2419a4f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:34.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.273+0000 7f2c2bc9a640 1 --2- 172.21.15.42:0/197866637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c241043f0 0x7f2c24199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:34.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.273+0000 7f2c2bc9a640 1 -- 172.21.15.42:0/197866637 >> 172.21.15.42:0/197866637 conn(0x7f2c24079140 msgr2=0x7f2c240ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:34.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.273+0000 7f2c2bc9a640 1 -- 172.21.15.42:0/197866637 shutdown_connections 2024-07-26T06:46:34.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:34.273+0000 7f2c2bc9a640 1 -- 172.21.15.42:0/197866637 wait complete. 2024-07-26T06:46:34.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:34.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:34.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:34.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:34.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:34 smithi164 ceph-mon[36970]: Deploying daemon mds.c on smithi164 2024-07-26T06:46:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:34 smithi164 ceph-mon[36970]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:34 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6834/2604534474,v1:172.21.15.164:6835/2604534474] up:boot 2024-07-26T06:46:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:34 smithi164 ceph-mon[36970]: fsmap 6 up:standby 2024-07-26T06:46:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T06:46:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:34 smithi163 ceph-mon[36873]: Deploying daemon mds.c on smithi164 2024-07-26T06:46:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:34 smithi163 ceph-mon[36873]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:34 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6834/2604534474,v1:172.21.15.164:6835/2604534474] up:boot 2024-07-26T06:46:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:34 smithi163 ceph-mon[36873]: fsmap 6 up:standby 2024-07-26T06:46:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T06:46:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:34 smithi042 ceph-mon[32775]: Deploying daemon mds.c on smithi164 2024-07-26T06:46:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:34 smithi042 ceph-mon[32775]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:34 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6834/2604534474,v1:172.21.15.164:6835/2604534474] up:boot 2024-07-26T06:46:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:34 smithi042 ceph-mon[32775]: fsmap 6 up:standby 2024-07-26T06:46:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T06:46:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:35 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/197866637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:35.430 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:35 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/197866637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:35.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:35 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/197866637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:35.972 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:46:35 smithi164 systemd[1]: Started Ceph mds.c for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:46:37.050 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:46:37.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:36 smithi042 ceph-mon[32775]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:36 smithi042 ceph-mon[32775]: Deploying daemon mds.h on smithi163 2024-07-26T06:46:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:36 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] up:boot 2024-07-26T06:46:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:36 smithi042 ceph-mon[32775]: fsmap 7 up:standby 2024-07-26T06:46:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T06:46:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:36 smithi164 ceph-mon[36970]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:36 smithi164 ceph-mon[36970]: Deploying daemon mds.h on smithi163 2024-07-26T06:46:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:36 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] up:boot 2024-07-26T06:46:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:36 smithi164 ceph-mon[36970]: fsmap 7 up:standby 2024-07-26T06:46:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T06:46:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:36 smithi163 ceph-mon[36873]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:36 smithi163 ceph-mon[36873]: Deploying daemon mds.h on smithi163 2024-07-26T06:46:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:36 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] up:boot 2024-07-26T06:46:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:36 smithi163 ceph-mon[36873]: fsmap 7 up:standby 2024-07-26T06:46:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T06:46:37.329 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:46:38.007 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:46:37 smithi163 systemd[1]: Starting Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:46:38.311 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:46:38 smithi163 podman[66711]: 2024-07-26 06:46:37.932971627 +0000 UTC m=+0.016707849 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:46:38.311 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:46:38 smithi163 podman[66711]: 2024-07-26 06:46:38.067231045 +0000 UTC m=+0.150967303 container create eff753a144e58fd48fb44fbae126322c40d5550dddd5afda0cef03fff94cb86e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T06:46:38.311 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:46:38 smithi163 podman[66711]: 2024-07-26 06:46:38.311371571 +0000 UTC m=+0.395107784 container init eff753a144e58fd48fb44fbae126322c40d5550dddd5afda0cef03fff94cb86e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_CLEAN=True) 2024-07-26T06:46:38.757 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:46:38 smithi163 podman[66711]: 2024-07-26 06:46:38.314090876 +0000 UTC m=+0.397827092 container start eff753a144e58fd48fb44fbae126322c40d5550dddd5afda0cef03fff94cb86e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T06:46:38.757 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:46:38 smithi163 ceph-mds[66730]: -- 172.21.15.163:0/3192994936 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ba6fa3fc20 con 0x55ba73be1180 2024-07-26T06:46:38.757 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:46:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: starting mds.h at 2024-07-26T06:46:38.757 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:46:38 smithi163 bash[66711]: eff753a144e58fd48fb44fbae126322c40d5550dddd5afda0cef03fff94cb86e 2024-07-26T06:46:38.757 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:46:38 smithi163 systemd[1]: Started Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:46:38.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.902+0000 7fa849d39640 1 -- 172.21.15.42:0/3872948628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa844107dd0 msgr2=0x7fa8441081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:38.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.902+0000 7fa849d39640 1 --2- 172.21.15.42:0/3872948628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa844107dd0 0x7fa8441081e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa838009900 tx=0x7fa83802a410 comp rx=0 tx=0).stop 2024-07-26T06:46:38.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.902+0000 7fa849d39640 1 -- 172.21.15.42:0/3872948628 shutdown_connections 2024-07-26T06:46:38.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.902+0000 7fa849d39640 1 --2- 172.21.15.42:0/3872948628 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa844108720 0x7fa84410ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:38.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.902+0000 7fa849d39640 1 --2- 172.21.15.42:0/3872948628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa844107dd0 0x7fa8441081e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:38.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.902+0000 7fa849d39640 1 --2- 172.21.15.42:0/3872948628 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa844102490 0x7fa8441028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:38.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.902+0000 7fa849d39640 1 -- 172.21.15.42:0/3872948628 >> 172.21.15.42:0/3872948628 conn(0x7fa844079160 msgr2=0x7fa844079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:38.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.902+0000 7fa849d39640 1 -- 172.21.15.42:0/3872948628 shutdown_connections 2024-07-26T06:46:38.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.902+0000 7fa849d39640 1 -- 172.21.15.42:0/3872948628 wait complete. 2024-07-26T06:46:38.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.903+0000 7fa849d39640 1 Processor -- start 2024-07-26T06:46:38.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.903+0000 7fa849d39640 1 -- start start 2024-07-26T06:46:38.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.903+0000 7fa849d39640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa844102490 0x7fa844199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:38.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.903+0000 7fa849d39640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa844107dd0 0x7fa84419a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:38.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.903+0000 7fa849d39640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa844108720 0x7fa84419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:38.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.904+0000 7fa849d39640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa844112350 con 0x7fa844102490 2024-07-26T06:46:38.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.904+0000 7fa849d39640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa8441121d0 con 0x7fa844108720 2024-07-26T06:46:38.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.904+0000 7fa849d39640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa8441124d0 con 0x7fa844107dd0 2024-07-26T06:46:38.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.904+0000 7fa8437fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa844102490 0x7fa844199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:38.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.904+0000 7fa8437fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa844102490 0x7fa844199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40998/0 (socket says 172.21.15.42:40998) 2024-07-26T06:46:38.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.904+0000 7fa8437fe640 1 -- 172.21.15.42:0/554362610 learned_addr learned my addr 172.21.15.42:0/554362610 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:46:38.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.904+0000 7fa842ffd640 1 --2- 172.21.15.42:0/554362610 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa844107dd0 0x7fa84419a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:38.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.904+0000 7fa843fff640 1 --2- 172.21.15.42:0/554362610 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa844108720 0x7fa84419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:38.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.904+0000 7fa8437fe640 1 -- 172.21.15.42:0/554362610 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa844107dd0 msgr2=0x7fa84419a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:38.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.904+0000 7fa8437fe640 1 --2- 172.21.15.42:0/554362610 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa844107dd0 0x7fa84419a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:38.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.904+0000 7fa8437fe640 1 -- 172.21.15.42:0/554362610 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa844108720 msgr2=0x7fa84419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:38.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.904+0000 7fa8437fe640 1 --2- 172.21.15.42:0/554362610 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa844108720 0x7fa84419f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:38.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.904+0000 7fa8437fe640 1 -- 172.21.15.42:0/554362610 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa84419fe80 con 0x7fa844102490 2024-07-26T06:46:38.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.904+0000 7fa8437fe640 1 --2- 172.21.15.42:0/554362610 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa844102490 0x7fa844199f10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa83400d950 tx=0x7fa83400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:38.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.904+0000 7fa840ff9640 1 -- 172.21.15.42:0/554362610 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa834017070 con 0x7fa844102490 2024-07-26T06:46:38.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.905+0000 7fa840ff9640 1 -- 172.21.15.42:0/554362610 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa834009c30 con 0x7fa844102490 2024-07-26T06:46:38.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.905+0000 7fa849d39640 1 -- 172.21.15.42:0/554362610 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8441a0170 con 0x7fa844102490 2024-07-26T06:46:38.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.905+0000 7fa840ff9640 1 -- 172.21.15.42:0/554362610 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83400f620 con 0x7fa844102490 2024-07-26T06:46:38.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.905+0000 7fa849d39640 1 -- 172.21.15.42:0/554362610 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8441a68d0 con 0x7fa844102490 2024-07-26T06:46:38.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.906+0000 7fa849d39640 1 -- 172.21.15.42:0/554362610 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8441028a0 con 0x7fa844102490 2024-07-26T06:46:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.907+0000 7fa840ff9640 1 -- 172.21.15.42:0/554362610 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fa834010460 con 0x7fa844102490 2024-07-26T06:46:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.908+0000 7fa840ff9640 1 --2- 172.21.15.42:0/554362610 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa82c0b3bd0 0x7fa82c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.908+0000 7fa842ffd640 1 --2- 172.21.15.42:0/554362610 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa82c0b3bd0 0x7fa82c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.908+0000 7fa840ff9640 1 -- 172.21.15.42:0/554362610 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fa8340e32e0 con 0x7fa844102490 2024-07-26T06:46:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.908+0000 7fa842ffd640 1 --2- 172.21.15.42:0/554362610 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa82c0b3bd0 0x7fa82c0b60e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa838000c00 tx=0x7fa838031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:38.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:38.911+0000 7fa840ff9640 1 -- 172.21.15.42:0/554362610 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa834009de0 con 0x7fa844102490 2024-07-26T06:46:39.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:39.130+0000 7fa849d39640 1 -- 172.21.15.42:0/554362610 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8441a6bb0 con 0x7fa844102490 2024-07-26T06:46:39.131 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:38 smithi042 ceph-mon[32775]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:39.131 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:39.131 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:39.131 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:39.132 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:39.132 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:39.132 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:39.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:39.131+0000 7fa840ff9640 1 -- 172.21.15.42:0/554362610 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7219 (secure 0 0 0) 0x7fa8340aa780 con 0x7fa844102490 2024-07-26T06:46:39.133 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:46:39.133 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":9,"btime":"2024-07-26T06:46:38:874911+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24541,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24557,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24571,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24578,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24583,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24593,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-07-26T06:46:39.134 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 9 2024-07-26T06:46:39.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:39.134+0000 7fa849d39640 1 -- 172.21.15.42:0/554362610 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa82c0b3bd0 msgr2=0x7fa82c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:39.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:39.134+0000 7fa849d39640 1 --2- 172.21.15.42:0/554362610 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa82c0b3bd0 0x7fa82c0b60e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa838000c00 tx=0x7fa838031400 comp rx=0 tx=0).stop 2024-07-26T06:46:39.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:39.134+0000 7fa849d39640 1 -- 172.21.15.42:0/554362610 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa844102490 msgr2=0x7fa844199f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:39.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:39.134+0000 7fa849d39640 1 --2- 172.21.15.42:0/554362610 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa844102490 0x7fa844199f10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa83400d950 tx=0x7fa83400de10 comp rx=0 tx=0).stop 2024-07-26T06:46:39.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:39.134+0000 7fa849d39640 1 -- 172.21.15.42:0/554362610 shutdown_connections 2024-07-26T06:46:39.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:39.134+0000 7fa849d39640 1 --2- 172.21.15.42:0/554362610 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa82c0b3bd0 0x7fa82c0b60e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:39.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:39.134+0000 7fa849d39640 1 --2- 172.21.15.42:0/554362610 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa844108720 0x7fa84419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:39.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:39.134+0000 7fa849d39640 1 --2- 172.21.15.42:0/554362610 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa844107dd0 0x7fa84419a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:39.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:39.134+0000 7fa849d39640 1 --2- 172.21.15.42:0/554362610 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa844102490 0x7fa844199f10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:39.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:39.134+0000 7fa849d39640 1 -- 172.21.15.42:0/554362610 >> 172.21.15.42:0/554362610 conn(0x7fa844079160 msgr2=0x7fa8440ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:39.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:39.134+0000 7fa849d39640 1 -- 172.21.15.42:0/554362610 shutdown_connections 2024-07-26T06:46:39.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:39.134+0000 7fa849d39640 1 -- 172.21.15.42:0/554362610 wait complete. 2024-07-26T06:46:39.189 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:38 smithi164 ceph-mon[36970]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:39.189 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:39.189 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:39.189 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:39.190 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:39.190 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:39.190 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:38 smithi163 ceph-mon[36873]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:40.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:39 smithi042 ceph-mon[32775]: Deploying daemon mds.i on smithi164 2024-07-26T06:46:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:39 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] up:boot 2024-07-26T06:46:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:39 smithi042 ceph-mon[32775]: fsmap 8 up:standby 2024-07-26T06:46:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:46:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:39 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/554362610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:40.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:39 smithi164 ceph-mon[36970]: Deploying daemon mds.i on smithi164 2024-07-26T06:46:40.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:39 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] up:boot 2024-07-26T06:46:40.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:39 smithi164 ceph-mon[36970]: fsmap 8 up:standby 2024-07-26T06:46:40.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:46:40.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:39 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/554362610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:40.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:39 smithi163 ceph-mon[36873]: Deploying daemon mds.i on smithi164 2024-07-26T06:46:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:39 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] up:boot 2024-07-26T06:46:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:39 smithi163 ceph-mon[36873]: fsmap 8 up:standby 2024-07-26T06:46:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:46:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:39 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/554362610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:40.684 INFO:journalctl@ceph.mds.i.smithi164.stdout:Jul 26 06:46:40 smithi164 systemd[1]: Starting Ceph mds.i for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:46:40.936 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:40 smithi164 ceph-mon[36970]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:40.937 INFO:journalctl@ceph.mds.i.smithi164.stdout:Jul 26 06:46:40 smithi164 podman[67165]: 2024-07-26 06:46:40.684038108 +0000 UTC m=+0.109852479 container create 40b3fac68d8c0e30311b7341e0f3658b98f3eb03ccd98a79b56b2ac80b3b663d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-i, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-07-26T06:46:40.937 INFO:journalctl@ceph.mds.i.smithi164.stdout:Jul 26 06:46:40 smithi164 podman[67165]: 2024-07-26 06:46:40.591470973 +0000 UTC m=+0.017285354 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:46:40.937 INFO:journalctl@ceph.mds.i.smithi164.stdout:Jul 26 06:46:40 smithi164 podman[67165]: 2024-07-26 06:46:40.936811986 +0000 UTC m=+0.362626374 container init 40b3fac68d8c0e30311b7341e0f3658b98f3eb03ccd98a79b56b2ac80b3b663d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-i, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, ceph=True, GIT_BRANCH=HEAD) 2024-07-26T06:46:41.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:40 smithi042 ceph-mon[32775]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:41.222 INFO:journalctl@ceph.mds.i.smithi164.stdout:Jul 26 06:46:40 smithi164 podman[67165]: 2024-07-26 06:46:40.939609374 +0000 UTC m=+0.365423759 container start 40b3fac68d8c0e30311b7341e0f3658b98f3eb03ccd98a79b56b2ac80b3b663d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-i, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-07-26T06:46:41.222 INFO:journalctl@ceph.mds.i.smithi164.stdout:Jul 26 06:46:40 smithi164 ceph-mds[67185]: -- 172.21.15.164:0/535957335 <== mon.1 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dfe1525860 con 0x55dfe56c7600 2024-07-26T06:46:41.222 INFO:journalctl@ceph.mds.i.smithi164.stdout:Jul 26 06:46:40 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-i[67181]: starting mds.i at 2024-07-26T06:46:41.222 INFO:journalctl@ceph.mds.i.smithi164.stdout:Jul 26 06:46:41 smithi164 bash[67165]: 40b3fac68d8c0e30311b7341e0f3658b98f3eb03ccd98a79b56b2ac80b3b663d 2024-07-26T06:46:41.222 INFO:journalctl@ceph.mds.i.smithi164.stdout:Jul 26 06:46:41 smithi164 systemd[1]: Started Ceph mds.i for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:46:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:40 smithi163 ceph-mon[36873]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:41.853 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:46:42.140 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:46:42.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:42.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:42.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:42 smithi042 ceph-mon[32775]: Deploying daemon mds.g on smithi042 2024-07-26T06:46:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:42 smithi042 ceph-mon[32775]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:42 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] up:boot 2024-07-26T06:46:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:42 smithi042 ceph-mon[32775]: fsmap 9 up:standby 2024-07-26T06:46:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T06:46:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:42 smithi164 ceph-mon[36970]: Deploying daemon mds.g on smithi042 2024-07-26T06:46:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:42 smithi164 ceph-mon[36970]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:42 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] up:boot 2024-07-26T06:46:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:42 smithi164 ceph-mon[36970]: fsmap 9 up:standby 2024-07-26T06:46:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T06:46:42.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:42 smithi163 ceph-mon[36873]: Deploying daemon mds.g on smithi042 2024-07-26T06:46:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:42 smithi163 ceph-mon[36873]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:42 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] up:boot 2024-07-26T06:46:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:42 smithi163 ceph-mon[36873]: fsmap 9 up:standby 2024-07-26T06:46:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T06:46:43.145 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:46:43 smithi042 systemd[1]: Starting Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:46:44.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.208+0000 7f97c88ca640 1 -- 172.21.15.42:0/40324029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97c4103590 msgr2=0x7f97c41039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.208+0000 7f97c88ca640 1 --2- 172.21.15.42:0/40324029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97c4103590 0x7f97c41039a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009900 tx=0x7f97b402a410 comp rx=0 tx=0).stop 2024-07-26T06:46:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.209+0000 7f97c88ca640 1 -- 172.21.15.42:0/40324029 shutdown_connections 2024-07-26T06:46:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.209+0000 7f97c88ca640 1 --2- 172.21.15.42:0/40324029 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97c4109820 0x7f97c4110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.209+0000 7f97c88ca640 1 --2- 172.21.15.42:0/40324029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f97c4108ed0 0x7f97c41092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.209+0000 7f97c88ca640 1 --2- 172.21.15.42:0/40324029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97c4103590 0x7f97c41039a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.209+0000 7f97c88ca640 1 -- 172.21.15.42:0/40324029 >> 172.21.15.42:0/40324029 conn(0x7f97c407b280 msgr2=0x7f97c407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.209+0000 7f97c88ca640 1 -- 172.21.15.42:0/40324029 shutdown_connections 2024-07-26T06:46:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.209+0000 7f97c88ca640 1 -- 172.21.15.42:0/40324029 wait complete. 2024-07-26T06:46:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.209+0000 7f97c88ca640 1 Processor -- start 2024-07-26T06:46:44.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c88ca640 1 -- start start 2024-07-26T06:46:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c88ca640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97c4103590 0x7f97c4199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c88ca640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f97c4108ed0 0x7f97c419a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c88ca640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97c4109820 0x7f97c419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c88ca640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f97c4113520 con 0x7f97c4103590 2024-07-26T06:46:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c88ca640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41133a0 con 0x7f97c4108ed0 2024-07-26T06:46:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c88ca640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f97c41136a0 con 0x7f97c4109820 2024-07-26T06:46:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97c4103590 0x7f97c4199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97c4103590 0x7f97c4199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41014/0 (socket says 172.21.15.42:41014) 2024-07-26T06:46:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c259c640 1 -- 172.21.15.42:0/2939247223 learned_addr learned my addr 172.21.15.42:0/2939247223 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:46:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c2d9d640 1 --2- 172.21.15.42:0/2939247223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97c4109820 0x7f97c419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97bbfff640 1 --2- 172.21.15.42:0/2939247223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f97c4108ed0 0x7f97c419a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:44.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c259c640 1 -- 172.21.15.42:0/2939247223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97c4109820 msgr2=0x7f97c419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c259c640 1 --2- 172.21.15.42:0/2939247223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97c4109820 0x7f97c419f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c259c640 1 -- 172.21.15.42:0/2939247223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f97c4108ed0 msgr2=0x7f97c419a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c259c640 1 --2- 172.21.15.42:0/2939247223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f97c4108ed0 0x7f97c419a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c259c640 1 -- 172.21.15.42:0/2939247223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c419fe80 con 0x7f97c4103590 2024-07-26T06:46:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97bbfff640 1 --2- 172.21.15.42:0/2939247223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f97c4108ed0 0x7f97c419a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:46:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c2d9d640 1 --2- 172.21.15.42:0/2939247223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97c4109820 0x7f97c419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:46:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.210+0000 7f97c259c640 1 --2- 172.21.15.42:0/2939247223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97c4103590 0x7f97c4199f10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f97b4005940 tx=0x7f97b40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.211+0000 7f97bb7fe640 1 -- 172.21.15.42:0/2939247223 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b403f030 con 0x7f97c4103590 2024-07-26T06:46:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.211+0000 7f97c88ca640 1 -- 172.21.15.42:0/2939247223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c41a0110 con 0x7f97c4103590 2024-07-26T06:46:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.211+0000 7f97bb7fe640 1 -- 172.21.15.42:0/2939247223 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97b4005d10 con 0x7f97c4103590 2024-07-26T06:46:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.211+0000 7f97bb7fe640 1 -- 172.21.15.42:0/2939247223 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4035070 con 0x7f97c4103590 2024-07-26T06:46:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.211+0000 7f97c88ca640 1 -- 172.21.15.42:0/2939247223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c41a68d0 con 0x7f97c4103590 2024-07-26T06:46:44.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.212+0000 7f97c88ca640 1 -- 172.21.15.42:0/2939247223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97c41039a0 con 0x7f97c4103590 2024-07-26T06:46:44.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.213+0000 7f97bb7fe640 1 -- 172.21.15.42:0/2939247223 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f97b4042030 con 0x7f97c4103590 2024-07-26T06:46:44.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.214+0000 7f97bb7fe640 1 --2- 172.21.15.42:0/2939247223 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f97b00b3bd0 0x7f97b00b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:44.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.215+0000 7f97bbfff640 1 --2- 172.21.15.42:0/2939247223 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f97b00b3bd0 0x7f97b00b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:44.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.215+0000 7f97bb7fe640 1 -- 172.21.15.42:0/2939247223 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f97b4103fd0 con 0x7f97c4103590 2024-07-26T06:46:44.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.215+0000 7f97bbfff640 1 --2- 172.21.15.42:0/2939247223 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f97b00b3bd0 0x7f97b00b60e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f97ac006fd0 tx=0x7f97ac008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:44.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.218+0000 7f97bb7fe640 1 -- 172.21.15.42:0/2939247223 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f97b40cbac0 con 0x7f97c4103590 2024-07-26T06:46:44.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.455+0000 7f97c88ca640 1 -- 172.21.15.42:0/2939247223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97c419b040 con 0x7f97c4103590 2024-07-26T06:46:44.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.455+0000 7f97bb7fe640 1 -- 172.21.15.42:0/2939247223 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8049 (secure 0 0 0) 0x7f97b40cb3c0 con 0x7f97c4103590 2024-07-26T06:46:44.456 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:46:44.457 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":10,"btime":"2024-07-26T06:46:41:888245+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24541,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24557,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24571,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24578,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24583,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24593,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24601,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-07-26T06:46:44.457 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 10 2024-07-26T06:46:44.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.457+0000 7f97c88ca640 1 -- 172.21.15.42:0/2939247223 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f97b00b3bd0 msgr2=0x7f97b00b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:44.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.457+0000 7f97c88ca640 1 --2- 172.21.15.42:0/2939247223 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f97b00b3bd0 0x7f97b00b60e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f97ac006fd0 tx=0x7f97ac008040 comp rx=0 tx=0).stop 2024-07-26T06:46:44.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.457+0000 7f97c88ca640 1 -- 172.21.15.42:0/2939247223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97c4103590 msgr2=0x7f97c4199f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:44.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.457+0000 7f97c88ca640 1 --2- 172.21.15.42:0/2939247223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97c4103590 0x7f97c4199f10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f97b4005940 tx=0x7f97b40042f0 comp rx=0 tx=0).stop 2024-07-26T06:46:44.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.458+0000 7f97c88ca640 1 -- 172.21.15.42:0/2939247223 shutdown_connections 2024-07-26T06:46:44.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.458+0000 7f97c88ca640 1 --2- 172.21.15.42:0/2939247223 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f97b00b3bd0 0x7f97b00b60e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:44.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.458+0000 7f97c88ca640 1 --2- 172.21.15.42:0/2939247223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97c4109820 0x7f97c419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:44.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.458+0000 7f97c88ca640 1 --2- 172.21.15.42:0/2939247223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f97c4108ed0 0x7f97c419a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:44.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.458+0000 7f97c88ca640 1 --2- 172.21.15.42:0/2939247223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97c4103590 0x7f97c4199f10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:44.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.458+0000 7f97c88ca640 1 -- 172.21.15.42:0/2939247223 >> 172.21.15.42:0/2939247223 conn(0x7f97c407b280 msgr2=0x7f97c4106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:44.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.458+0000 7f97c88ca640 1 -- 172.21.15.42:0/2939247223 shutdown_connections 2024-07-26T06:46:44.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:44.458+0000 7f97c88ca640 1 -- 172.21.15.42:0/2939247223 wait complete. 2024-07-26T06:46:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:44 smithi042 ceph-mon[32775]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:44.645 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:46:44 smithi042 podman[76999]: 2024-07-26 06:46:44.284528248 +0000 UTC m=+0.018347038 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:46:44.645 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:46:44 smithi042 podman[76999]: 2024-07-26 06:46:44.39869486 +0000 UTC m=+0.132513635 container create 3e3da20f6e4dca345cc00478eaf1049f852fda21c2ba2f4022d9c69642d83050 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.36.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T06:46:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:44 smithi164 ceph-mon[36970]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:44.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:44 smithi163 ceph-mon[36873]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:45.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:45 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2939247223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:45.645 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:46:45 smithi042 podman[76999]: 2024-07-26 06:46:45.148518351 +0000 UTC m=+0.882337141 container init 3e3da20f6e4dca345cc00478eaf1049f852fda21c2ba2f4022d9c69642d83050 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-07-26T06:46:45.645 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:46:45 smithi042 podman[76999]: 2024-07-26 06:46:45.1518331 +0000 UTC m=+0.885651882 container start 3e3da20f6e4dca345cc00478eaf1049f852fda21c2ba2f4022d9c69642d83050 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724) 2024-07-26T06:46:45.646 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:46:45 smithi042 ceph-mds[77034]: -- 172.21.15.42:0/393748712 <== mon.0 v2:172.21.15.42:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560287d87860 con 0x56028bf29180 2024-07-26T06:46:45.646 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:46:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: starting mds.g at 2024-07-26T06:46:45.646 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:46:45 smithi042 bash[76999]: 3e3da20f6e4dca345cc00478eaf1049f852fda21c2ba2f4022d9c69642d83050 2024-07-26T06:46:45.646 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:46:45 smithi042 systemd[1]: Started Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:46:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:45 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2939247223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:45.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:45 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2939247223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:46 smithi042 ceph-mon[32775]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:46 smithi042 ceph-mon[32775]: Deploying daemon mds.a on smithi042 2024-07-26T06:46:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:46 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/4044632348,v1:172.21.15.42:6839/4044632348] up:boot 2024-07-26T06:46:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:46 smithi042 ceph-mon[32775]: fsmap 10 up:standby 2024-07-26T06:46:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:46:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:46 smithi164 ceph-mon[36970]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:46 smithi164 ceph-mon[36970]: Deploying daemon mds.a on smithi042 2024-07-26T06:46:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:46 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/4044632348,v1:172.21.15.42:6839/4044632348] up:boot 2024-07-26T06:46:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:46 smithi164 ceph-mon[36970]: fsmap 10 up:standby 2024-07-26T06:46:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:46:46.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:46 smithi163 ceph-mon[36873]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:46 smithi163 ceph-mon[36873]: Deploying daemon mds.a on smithi042 2024-07-26T06:46:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:46 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/4044632348,v1:172.21.15.42:6839/4044632348] up:boot 2024-07-26T06:46:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:46 smithi163 ceph-mon[36873]: fsmap 10 up:standby 2024-07-26T06:46:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:46:47.053 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:46:47.381 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:46:47 smithi042 systemd[1]: Starting Ceph mds.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:46:47.644 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:46:47 smithi042 podman[77337]: 2024-07-26 06:46:47.28351325 +0000 UTC m=+0.017943503 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:46:47.645 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:46:47 smithi042 podman[77337]: 2024-07-26 06:46:47.385605093 +0000 UTC m=+0.120035327 container create 84e1094b936b61dbe1abfb18a38ee674f356d345b9102572498214a2967d416a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True) 2024-07-26T06:46:47.645 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:46:47 smithi042 podman[77337]: 2024-07-26 06:46:47.622435676 +0000 UTC m=+0.356865929 container init 84e1094b936b61dbe1abfb18a38ee674f356d345b9102572498214a2967d416a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-07-26T06:46:47.645 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:46:47 smithi042 podman[77337]: 2024-07-26 06:46:47.625635043 +0000 UTC m=+0.360065279 container start 84e1094b936b61dbe1abfb18a38ee674f356d345b9102572498214a2967d416a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724) 2024-07-26T06:46:47.763 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:46:47.982 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:46:47 smithi042 ceph-mds[77380]: -- 172.21.15.42:0/1353825250 <== mon.0 v2:172.21.15.42:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55925e5bd860 con 0x55926275f180 2024-07-26T06:46:47.982 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:46:47 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: starting mds.a at 2024-07-26T06:46:47.982 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:46:47 smithi042 bash[77337]: 84e1094b936b61dbe1abfb18a38ee674f356d345b9102572498214a2967d416a 2024-07-26T06:46:47.982 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:46:47 smithi042 systemd[1]: Started Ceph mds.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:46:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:48 smithi163 ceph-mon[36873]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:48 smithi163 ceph-mon[36873]: Deploying daemon mds.b on smithi163 2024-07-26T06:46:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:48 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/1430155791,v1:172.21.15.42:6841/1430155791] up:boot 2024-07-26T06:46:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:48 smithi163 ceph-mon[36873]: fsmap 11 up:standby 2024-07-26T06:46:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T06:46:49.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:48 smithi042 ceph-mon[32775]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:49.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:49.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:49.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:49.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:49.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:49.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:49.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:48 smithi042 ceph-mon[32775]: Deploying daemon mds.b on smithi163 2024-07-26T06:46:49.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:48 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/1430155791,v1:172.21.15.42:6841/1430155791] up:boot 2024-07-26T06:46:49.051 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:48 smithi042 ceph-mon[32775]: fsmap 11 up:standby 2024-07-26T06:46:49.052 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T06:46:49.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:48 smithi164 ceph-mon[36970]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:49.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:49.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T06:46:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T06:46:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:48 smithi164 ceph-mon[36970]: Deploying daemon mds.b on smithi163 2024-07-26T06:46:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:48 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/1430155791,v1:172.21.15.42:6841/1430155791] up:boot 2024-07-26T06:46:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:48 smithi164 ceph-mon[36970]: fsmap 11 up:standby 2024-07-26T06:46:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T06:46:49.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.368+0000 7f7e5e493640 1 -- 172.21.15.42:0/1078630791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e58109820 msgr2=0x7f7e58110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:49.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.368+0000 7f7e5e493640 1 --2- 172.21.15.42:0/1078630791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e58109820 0x7f7e58110050 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7e50009ca0 tx=0x7f7e5002a8c0 comp rx=0 tx=0).stop 2024-07-26T06:46:49.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.369+0000 7f7e5e493640 1 -- 172.21.15.42:0/1078630791 shutdown_connections 2024-07-26T06:46:49.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.369+0000 7f7e5e493640 1 --2- 172.21.15.42:0/1078630791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e58109820 0x7f7e58110050 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:49.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.369+0000 7f7e5e493640 1 --2- 172.21.15.42:0/1078630791 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e58108ed0 0x7f7e581092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:49.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.369+0000 7f7e5e493640 1 --2- 172.21.15.42:0/1078630791 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e58103590 0x7f7e581039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:49.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.369+0000 7f7e5e493640 1 -- 172.21.15.42:0/1078630791 >> 172.21.15.42:0/1078630791 conn(0x7f7e5807b280 msgr2=0x7f7e5807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:49.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.369+0000 7f7e5e493640 1 -- 172.21.15.42:0/1078630791 shutdown_connections 2024-07-26T06:46:49.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.369+0000 7f7e5e493640 1 -- 172.21.15.42:0/1078630791 wait complete. 2024-07-26T06:46:49.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.369+0000 7f7e5e493640 1 Processor -- start 2024-07-26T06:46:49.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.369+0000 7f7e5e493640 1 -- start start 2024-07-26T06:46:49.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e5e493640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e58103590 0x7f7e58075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:49.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e5e493640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e58108ed0 0x7f7e58075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:49.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e5e493640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e58109820 0x7f7e58075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:49.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e5e493640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7e581134d0 con 0x7f7e58109820 2024-07-26T06:46:49.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e5e493640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7e58113350 con 0x7f7e58108ed0 2024-07-26T06:46:49.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e5e493640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7e58113650 con 0x7f7e58103590 2024-07-26T06:46:49.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e5ca30640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e58109820 0x7f7e58075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:49.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e5ca30640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e58109820 0x7f7e58075bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59878/0 (socket says 172.21.15.42:59878) 2024-07-26T06:46:49.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e5ca30640 1 -- 172.21.15.42:0/3451936573 learned_addr learned my addr 172.21.15.42:0/3451936573 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:46:49.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e57fff640 1 --2- 172.21.15.42:0/3451936573 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e58103590 0x7f7e58075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:49.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e577fe640 1 --2- 172.21.15.42:0/3451936573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e58108ed0 0x7f7e58075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:49.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e5ca30640 1 -- 172.21.15.42:0/3451936573 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e58103590 msgr2=0x7f7e58075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:49.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e5ca30640 1 --2- 172.21.15.42:0/3451936573 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e58103590 0x7f7e58075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:49.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e5ca30640 1 -- 172.21.15.42:0/3451936573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e58108ed0 msgr2=0x7f7e58075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:49.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e5ca30640 1 --2- 172.21.15.42:0/3451936573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e58108ed0 0x7f7e58075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:49.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e5ca30640 1 -- 172.21.15.42:0/3451936573 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e581a81e0 con 0x7f7e58109820 2024-07-26T06:46:49.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e57fff640 1 --2- 172.21.15.42:0/3451936573 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e58103590 0x7f7e58075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:46:49.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.370+0000 7f7e577fe640 1 --2- 172.21.15.42:0/3451936573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e58108ed0 0x7f7e58075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:46:49.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.371+0000 7f7e5ca30640 1 --2- 172.21.15.42:0/3451936573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e58109820 0x7f7e58075bc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7e50005cc0 tx=0x7f7e5002f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:49.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.371+0000 7f7e557fa640 1 -- 172.21.15.42:0/3451936573 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e50003bf0 con 0x7f7e58109820 2024-07-26T06:46:49.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.371+0000 7f7e5e493640 1 -- 172.21.15.42:0/3451936573 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e581a8470 con 0x7f7e58109820 2024-07-26T06:46:49.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.371+0000 7f7e557fa640 1 -- 172.21.15.42:0/3451936573 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e50003d90 con 0x7f7e58109820 2024-07-26T06:46:49.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.371+0000 7f7e557fa640 1 -- 172.21.15.42:0/3451936573 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5002fbc0 con 0x7f7e58109820 2024-07-26T06:46:49.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.371+0000 7f7e5e493640 1 -- 172.21.15.42:0/3451936573 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e581a8a00 con 0x7f7e58109820 2024-07-26T06:46:49.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.372+0000 7f7e5e493640 1 -- 172.21.15.42:0/3451936573 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e581039a0 con 0x7f7e58109820 2024-07-26T06:46:49.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.374+0000 7f7e557fa640 1 -- 172.21.15.42:0/3451936573 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f7e5002fd60 con 0x7f7e58109820 2024-07-26T06:46:49.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.374+0000 7f7e557fa640 1 --2- 172.21.15.42:0/3451936573 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7e400b3bd0 0x7f7e400b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:49.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.375+0000 7f7e557fa640 1 -- 172.21.15.42:0/3451936573 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f7e50105100 con 0x7f7e58109820 2024-07-26T06:46:49.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.375+0000 7f7e57fff640 1 --2- 172.21.15.42:0/3451936573 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7e400b3bd0 0x7f7e400b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:49.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.375+0000 7f7e57fff640 1 --2- 172.21.15.42:0/3451936573 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7e400b3bd0 0x7f7e400b60e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7e48005fd0 tx=0x7f7e48005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:49.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.378+0000 7f7e557fa640 1 -- 172.21.15.42:0/3451936573 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7e500ccbf0 con 0x7f7e58109820 2024-07-26T06:46:49.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.599+0000 7f7e5e493640 1 -- 172.21.15.42:0/3451936573 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e58076440 con 0x7f7e58109820 2024-07-26T06:46:49.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.599+0000 7f7e557fa640 1 -- 172.21.15.42:0/3451936573 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9701 (secure 0 0 0) 0x7f7e500098b0 con 0x7f7e58109820 2024-07-26T06:46:49.600 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:46:49.600 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":12,"btime":"2024-07-26T06:46:48:372236+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14703,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14709,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24541,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24557,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24571,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24578,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24583,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24593,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24601,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-07-26T06:46:49.600 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 12 2024-07-26T06:46:49.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.601+0000 7f7e5e493640 1 -- 172.21.15.42:0/3451936573 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7e400b3bd0 msgr2=0x7f7e400b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:49.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.601+0000 7f7e5e493640 1 --2- 172.21.15.42:0/3451936573 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7e400b3bd0 0x7f7e400b60e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7e48005fd0 tx=0x7f7e48005850 comp rx=0 tx=0).stop 2024-07-26T06:46:49.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.601+0000 7f7e5e493640 1 -- 172.21.15.42:0/3451936573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e58109820 msgr2=0x7f7e58075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:49.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.601+0000 7f7e5e493640 1 --2- 172.21.15.42:0/3451936573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e58109820 0x7f7e58075bc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7e50005cc0 tx=0x7f7e5002f380 comp rx=0 tx=0).stop 2024-07-26T06:46:49.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.602+0000 7f7e5e493640 1 -- 172.21.15.42:0/3451936573 shutdown_connections 2024-07-26T06:46:49.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.602+0000 7f7e5e493640 1 --2- 172.21.15.42:0/3451936573 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7e400b3bd0 0x7f7e400b60e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:49.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.602+0000 7f7e5e493640 1 --2- 172.21.15.42:0/3451936573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e58109820 0x7f7e58075bc0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:49.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.602+0000 7f7e5e493640 1 --2- 172.21.15.42:0/3451936573 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e58108ed0 0x7f7e58075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:49.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.602+0000 7f7e5e493640 1 --2- 172.21.15.42:0/3451936573 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e58103590 0x7f7e58075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:49.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.602+0000 7f7e5e493640 1 -- 172.21.15.42:0/3451936573 >> 172.21.15.42:0/3451936573 conn(0x7f7e5807b280 msgr2=0x7f7e58106f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:49.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.602+0000 7f7e5e493640 1 -- 172.21.15.42:0/3451936573 shutdown_connections 2024-07-26T06:46:49.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:49.602+0000 7f7e5e493640 1 -- 172.21.15.42:0/3451936573 wait complete. 2024-07-26T06:46:49.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:49 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3451936573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:50.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:49 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3451936573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:50.007 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:46:49 smithi163 systemd[1]: Starting Ceph mds.b for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:46:50.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:49 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3451936573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:50.418 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:46:50 smithi163 podman[67013]: 2024-07-26 06:46:50.055478132 +0000 UTC m=+0.017366860 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:46:50.419 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:46:50 smithi163 podman[67013]: 2024-07-26 06:46:50.173978579 +0000 UTC m=+0.135867338 container create 464ee1df1f53533c9a64fd32aeb32517814d1cba132245c46a16d7bc4aaf372f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T06:46:50.419 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:46:50 smithi163 podman[67013]: 2024-07-26 06:46:50.41873448 +0000 UTC m=+0.380623217 container init 464ee1df1f53533c9a64fd32aeb32517814d1cba132245c46a16d7bc4aaf372f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-07-26T06:46:50.757 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:46:50 smithi163 podman[67013]: 2024-07-26 06:46:50.421923791 +0000 UTC m=+0.383812517 container start 464ee1df1f53533c9a64fd32aeb32517814d1cba132245c46a16d7bc4aaf372f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T06:46:50.757 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:46:50 smithi163 ceph-mds[67032]: -- 172.21.15.163:0/1336029684 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555646839860 con 0x55564a9dba80 2024-07-26T06:46:50.757 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:46:50 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: starting mds.b at 2024-07-26T06:46:50.757 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:46:50 smithi163 bash[67013]: 464ee1df1f53533c9a64fd32aeb32517814d1cba132245c46a16d7bc4aaf372f 2024-07-26T06:46:50.757 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:46:50 smithi163 systemd[1]: Started Ceph mds.b for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:46:51.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:50 smithi042 ceph-mon[32775]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:51.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:46:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:50 smithi164 ceph-mon[36970]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:46:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:50 smithi163 ceph-mon[36873]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:51.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:46:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:51 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] up:boot 2024-07-26T06:46:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:51 smithi163 ceph-mon[36873]: fsmap 12 up:standby 2024-07-26T06:46:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T06:46:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:52.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:51 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] up:boot 2024-07-26T06:46:52.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:51 smithi042 ceph-mon[32775]: fsmap 12 up:standby 2024-07-26T06:46:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T06:46:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:52.211 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:46:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:51 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] up:boot 2024-07-26T06:46:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:51 smithi164 ceph-mon[36970]: fsmap 12 up:standby 2024-07-26T06:46:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T06:46:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:52.491 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:46:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:53 smithi042 ceph-mon[32775]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:46:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:46:53.416 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:53 smithi164 ceph-mon[36970]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:53.416 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.416 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.416 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.416 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.417 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.417 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.417 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:53.417 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:46:53.417 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.417 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:46:53.440 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:53 smithi163 ceph-mon[36873]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:53.440 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.440 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.440 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.440 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.440 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.441 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.441 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:53.441 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:46:53.441 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:53.441 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:46:54.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.067+0000 7f6268f6c640 1 -- 172.21.15.42:0/1629179734 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6264107e00 msgr2=0x7f6264108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:54.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.067+0000 7f6268f6c640 1 --2- 172.21.15.42:0/1629179734 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6264107e00 0x7f6264108210 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6258009900 tx=0x7f625802a410 comp rx=0 tx=0).stop 2024-07-26T06:46:54.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.067+0000 7f6268f6c640 1 -- 172.21.15.42:0/1629179734 shutdown_connections 2024-07-26T06:46:54.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.067+0000 7f6268f6c640 1 --2- 172.21.15.42:0/1629179734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6264108750 0x7f626410ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:54.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.067+0000 7f6268f6c640 1 --2- 172.21.15.42:0/1629179734 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6264107e00 0x7f6264108210 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:54.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.067+0000 7f6268f6c640 1 --2- 172.21.15.42:0/1629179734 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6264102510 0x7f6264102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:54.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.067+0000 7f6268f6c640 1 -- 172.21.15.42:0/1629179734 >> 172.21.15.42:0/1629179734 conn(0x7f6264079190 msgr2=0x7f62640795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:54.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.067+0000 7f6268f6c640 1 -- 172.21.15.42:0/1629179734 shutdown_connections 2024-07-26T06:46:54.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.067+0000 7f6268f6c640 1 -- 172.21.15.42:0/1629179734 wait complete. 2024-07-26T06:46:54.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.068+0000 7f6268f6c640 1 Processor -- start 2024-07-26T06:46:54.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.068+0000 7f6268f6c640 1 -- start start 2024-07-26T06:46:54.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f6268f6c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6264102510 0x7f626419ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:54.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f6268f6c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6264107e00 0x7f626419b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:54.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f6268f6c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6264108750 0x7f626419b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:54.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f6268f6c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f62641123a0 con 0x7f6264108750 2024-07-26T06:46:54.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f6268f6c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6264112220 con 0x7f6264107e00 2024-07-26T06:46:54.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f6268f6c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6264112520 con 0x7f6264102510 2024-07-26T06:46:54.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f6262d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6264108750 0x7f626419b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:54.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f6262d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6264108750 0x7f626419b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59908/0 (socket says 172.21.15.42:59908) 2024-07-26T06:46:54.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f6262d9d640 1 -- 172.21.15.42:0/524445612 learned_addr learned my addr 172.21.15.42:0/524445612 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:46:54.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f626259c640 1 --2- 172.21.15.42:0/524445612 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6264102510 0x7f626419ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:54.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f6261d9b640 1 --2- 172.21.15.42:0/524445612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6264107e00 0x7f626419b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:54.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f6262d9d640 1 -- 172.21.15.42:0/524445612 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6264102510 msgr2=0x7f626419ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:54.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f6262d9d640 1 --2- 172.21.15.42:0/524445612 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6264102510 0x7f626419ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:54.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f6262d9d640 1 -- 172.21.15.42:0/524445612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6264107e00 msgr2=0x7f626419b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:54.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f6262d9d640 1 --2- 172.21.15.42:0/524445612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6264107e00 0x7f626419b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:54.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f6262d9d640 1 -- 172.21.15.42:0/524445612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62641a5f60 con 0x7f6264108750 2024-07-26T06:46:54.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f6262d9d640 1 --2- 172.21.15.42:0/524445612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6264108750 0x7f626419b6d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f625c00eb10 tx=0x7f625c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:54.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f62537fe640 1 -- 172.21.15.42:0/524445612 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c004b20 con 0x7f6264108750 2024-07-26T06:46:54.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.069+0000 7f6268f6c640 1 -- 172.21.15.42:0/524445612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62641a6250 con 0x7f6264108750 2024-07-26T06:46:54.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.070+0000 7f62537fe640 1 -- 172.21.15.42:0/524445612 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f625c004cc0 con 0x7f6264108750 2024-07-26T06:46:54.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.070+0000 7f62537fe640 1 -- 172.21.15.42:0/524445612 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c010690 con 0x7f6264108750 2024-07-26T06:46:54.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.070+0000 7f6268f6c640 1 -- 172.21.15.42:0/524445612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62641a6760 con 0x7f6264108750 2024-07-26T06:46:54.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.071+0000 7f6268f6c640 1 -- 172.21.15.42:0/524445612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6264103740 con 0x7f6264108750 2024-07-26T06:46:54.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.072+0000 7f62537fe640 1 -- 172.21.15.42:0/524445612 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f625c005000 con 0x7f6264108750 2024-07-26T06:46:54.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.073+0000 7f62537fe640 1 --2- 172.21.15.42:0/524445612 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f624c0b3bd0 0x7f624c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:54.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.073+0000 7f62537fe640 1 -- 172.21.15.42:0/524445612 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f625c0e3f80 con 0x7f6264108750 2024-07-26T06:46:54.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.073+0000 7f626259c640 1 --2- 172.21.15.42:0/524445612 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f624c0b3bd0 0x7f624c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:54.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.074+0000 7f626259c640 1 --2- 172.21.15.42:0/524445612 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f624c0b3bd0 0x7f624c0b60e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f626406c130 tx=0x7f6254005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:54.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.078+0000 7f62537fe640 1 -- 172.21.15.42:0/524445612 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f625c055600 con 0x7f6264108750 2024-07-26T06:46:54.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.319+0000 7f6268f6c640 1 -- 172.21.15.42:0/524445612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f626406b9e0 con 0x7f6264108750 2024-07-26T06:46:54.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.320+0000 7f62537fe640 1 -- 172.21.15.42:0/524445612 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10527 (secure 0 0 0) 0x7f626406b9e0 con 0x7f6264108750 2024-07-26T06:46:54.321 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:46:54.321 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":13,"btime":"2024-07-26T06:46:50:760736+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14703,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14709,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24541,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24557,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24571,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24578,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24583,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24593,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24601,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24620,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-07-26T06:46:54.321 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 13 2024-07-26T06:46:54.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.322+0000 7f6268f6c640 1 -- 172.21.15.42:0/524445612 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f624c0b3bd0 msgr2=0x7f624c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:54.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.322+0000 7f6268f6c640 1 --2- 172.21.15.42:0/524445612 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f624c0b3bd0 0x7f624c0b60e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f626406c130 tx=0x7f6254005f90 comp rx=0 tx=0).stop 2024-07-26T06:46:54.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.322+0000 7f6268f6c640 1 -- 172.21.15.42:0/524445612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6264108750 msgr2=0x7f626419b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:54.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.322+0000 7f6268f6c640 1 --2- 172.21.15.42:0/524445612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6264108750 0x7f626419b6d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f625c00eb10 tx=0x7f625c00c2d0 comp rx=0 tx=0).stop 2024-07-26T06:46:54.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.322+0000 7f6268f6c640 1 -- 172.21.15.42:0/524445612 shutdown_connections 2024-07-26T06:46:54.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.322+0000 7f6268f6c640 1 --2- 172.21.15.42:0/524445612 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f624c0b3bd0 0x7f624c0b60e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:54.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.322+0000 7f6268f6c640 1 --2- 172.21.15.42:0/524445612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6264108750 0x7f626419b6d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:54.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.322+0000 7f6268f6c640 1 --2- 172.21.15.42:0/524445612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6264107e00 0x7f626419b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:54.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.322+0000 7f6268f6c640 1 --2- 172.21.15.42:0/524445612 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6264102510 0x7f626419ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:54.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.322+0000 7f6268f6c640 1 -- 172.21.15.42:0/524445612 >> 172.21.15.42:0/524445612 conn(0x7f6264079190 msgr2=0x7f6264104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:54.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.322+0000 7f6268f6c640 1 -- 172.21.15.42:0/524445612 shutdown_connections 2024-07-26T06:46:54.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:54.322+0000 7f6268f6c640 1 -- 172.21.15.42:0/524445612 wait complete. 2024-07-26T06:46:54.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:54 smithi042 ceph-mon[32775]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:54.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:54 smithi164 ceph-mon[36970]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:54.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:54 smithi163 ceph-mon[36873]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:55.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:55.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:55.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:55.380 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:55 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/524445612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:55.382 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-07-26T06:46:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:55 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/524445612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:55 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/524445612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:46:55.679 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:46:57.214 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:56 smithi042 ceph-mon[32775]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:57.214 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:57.214 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:57.214 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:57.214 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:46:57.214 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:56 smithi164 ceph-mon[36970]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:46:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:57.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:56 smithi163 ceph-mon[36873]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:46:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:46:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:46:57.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.309+0000 7f20d2260640 1 -- 172.21.15.42:0/3602200593 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20cc1035c0 msgr2=0x7f20cc1039d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:57.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.309+0000 7f20d2260640 1 --2- 172.21.15.42:0/3602200593 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20cc1035c0 0x7f20cc1039d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f20bc009ca0 tx=0x7f20bc02a8c0 comp rx=0 tx=0).stop 2024-07-26T06:46:57.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.309+0000 7f20d2260640 1 -- 172.21.15.42:0/3602200593 shutdown_connections 2024-07-26T06:46:57.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.309+0000 7f20d2260640 1 --2- 172.21.15.42:0/3602200593 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20cc109850 0x7f20cc110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:57.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.309+0000 7f20d2260640 1 --2- 172.21.15.42:0/3602200593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20cc108f00 0x7f20cc109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:57.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.309+0000 7f20d2260640 1 --2- 172.21.15.42:0/3602200593 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20cc1035c0 0x7f20cc1039d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:57.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.309+0000 7f20d2260640 1 -- 172.21.15.42:0/3602200593 >> 172.21.15.42:0/3602200593 conn(0x7f20cc0ff030 msgr2=0x7f20cc101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:57.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.309+0000 7f20d2260640 1 -- 172.21.15.42:0/3602200593 shutdown_connections 2024-07-26T06:46:57.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.309+0000 7f20d2260640 1 -- 172.21.15.42:0/3602200593 wait complete. 2024-07-26T06:46:57.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.310+0000 7f20d2260640 1 Processor -- start 2024-07-26T06:46:57.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.310+0000 7f20d2260640 1 -- start start 2024-07-26T06:46:57.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.310+0000 7f20d2260640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20cc1035c0 0x7f20cc199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:57.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.310+0000 7f20d2260640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20cc108f00 0x7f20cc19a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:57.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.310+0000 7f20d2260640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20cc109850 0x7f20cc19f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:57.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.310+0000 7f20d2260640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f20cc113570 con 0x7f20cc108f00 2024-07-26T06:46:57.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.310+0000 7f20d2260640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f20cc1133f0 con 0x7f20cc109850 2024-07-26T06:46:57.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.311+0000 7f20d2260640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f20cc1136f0 con 0x7f20cc1035c0 2024-07-26T06:46:57.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.311+0000 7f20caffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20cc108f00 0x7f20cc19a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:57.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.311+0000 7f20caffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20cc108f00 0x7f20cc19a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49204/0 (socket says 172.21.15.42:49204) 2024-07-26T06:46:57.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.311+0000 7f20caffd640 1 -- 172.21.15.42:0/3522006709 learned_addr learned my addr 172.21.15.42:0/3522006709 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:46:57.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.311+0000 7f20cbfff640 1 --2- 172.21.15.42:0/3522006709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20cc109850 0x7f20cc19f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:57.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.311+0000 7f20cb7fe640 1 --2- 172.21.15.42:0/3522006709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20cc1035c0 0x7f20cc199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:57.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.311+0000 7f20caffd640 1 -- 172.21.15.42:0/3522006709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20cc1035c0 msgr2=0x7f20cc199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:57.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.311+0000 7f20caffd640 1 --2- 172.21.15.42:0/3522006709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20cc1035c0 0x7f20cc199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:57.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.311+0000 7f20caffd640 1 -- 172.21.15.42:0/3522006709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20cc109850 msgr2=0x7f20cc19f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:57.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.311+0000 7f20caffd640 1 --2- 172.21.15.42:0/3522006709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20cc109850 0x7f20cc19f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:57.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.311+0000 7f20caffd640 1 -- 172.21.15.42:0/3522006709 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20cc19ff10 con 0x7f20cc108f00 2024-07-26T06:46:57.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.311+0000 7f20caffd640 1 --2- 172.21.15.42:0/3522006709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20cc108f00 0x7f20cc19a540 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f20b400d740 tx=0x7f20b400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:57.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.311+0000 7f20c8ff9640 1 -- 172.21.15.42:0/3522006709 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b4017070 con 0x7f20cc108f00 2024-07-26T06:46:57.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.312+0000 7f20c8ff9640 1 -- 172.21.15.42:0/3522006709 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20b400de50 con 0x7f20cc108f00 2024-07-26T06:46:57.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.312+0000 7f20d2260640 1 -- 172.21.15.42:0/3522006709 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20cc1a0200 con 0x7f20cc108f00 2024-07-26T06:46:57.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.312+0000 7f20c8ff9640 1 -- 172.21.15.42:0/3522006709 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b400f620 con 0x7f20cc108f00 2024-07-26T06:46:57.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.312+0000 7f20d2260640 1 -- 172.21.15.42:0/3522006709 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20cc1a6960 con 0x7f20cc108f00 2024-07-26T06:46:57.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.313+0000 7f20d2260640 1 -- 172.21.15.42:0/3522006709 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20cc1039d0 con 0x7f20cc108f00 2024-07-26T06:46:57.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.314+0000 7f20c8ff9640 1 -- 172.21.15.42:0/3522006709 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f20b401a070 con 0x7f20cc108f00 2024-07-26T06:46:57.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.315+0000 7f20c8ff9640 1 --2- 172.21.15.42:0/3522006709 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f20b80b3bd0 0x7f20b80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:57.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.316+0000 7f20cb7fe640 1 --2- 172.21.15.42:0/3522006709 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f20b80b3bd0 0x7f20b80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:57.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.316+0000 7f20c8ff9640 1 -- 172.21.15.42:0/3522006709 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f20b40e30f0 con 0x7f20cc108f00 2024-07-26T06:46:57.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.316+0000 7f20cb7fe640 1 --2- 172.21.15.42:0/3522006709 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f20b80b3bd0 0x7f20b80b60e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f20bc005e30 tx=0x7f20bc009360 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:57.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.319+0000 7f20c8ff9640 1 -- 172.21.15.42:0/3522006709 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f20b40aa4f0 con 0x7f20cc108f00 2024-07-26T06:46:57.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.504+0000 7f20d2260640 1 -- 172.21.15.42:0/3522006709 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f20cc02e0c0 con 0x7f20cc108f00 2024-07-26T06:46:57.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.504+0000 7f20c8ff9640 1 -- 172.21.15.42:0/3522006709 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v71) ==== 84+0+65 (secure 0 0 0) 0x7f20b40aea40 con 0x7f20cc108f00 2024-07-26T06:46:57.505 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:46:57.505 INFO:teuthology.orchestra.run.smithi042.stdout:[ 2024-07-26T06:46:57.505 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-07-26T06:46:57.505 INFO:teuthology.orchestra.run.smithi042.stdout: "poolnum": 1, 2024-07-26T06:46:57.506 INFO:teuthology.orchestra.run.smithi042.stdout: "poolname": ".mgr" 2024-07-26T06:46:57.506 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:46:57.506 INFO:teuthology.orchestra.run.smithi042.stdout:] 2024-07-26T06:46:57.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.506+0000 7f20d2260640 1 -- 172.21.15.42:0/3522006709 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f20b80b3bd0 msgr2=0x7f20b80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:57.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.506+0000 7f20d2260640 1 --2- 172.21.15.42:0/3522006709 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f20b80b3bd0 0x7f20b80b60e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f20bc005e30 tx=0x7f20bc009360 comp rx=0 tx=0).stop 2024-07-26T06:46:57.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.506+0000 7f20d2260640 1 -- 172.21.15.42:0/3522006709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20cc108f00 msgr2=0x7f20cc19a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:57.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.506+0000 7f20d2260640 1 --2- 172.21.15.42:0/3522006709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20cc108f00 0x7f20cc19a540 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f20b400d740 tx=0x7f20b400dc00 comp rx=0 tx=0).stop 2024-07-26T06:46:57.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.506+0000 7f20d2260640 1 -- 172.21.15.42:0/3522006709 shutdown_connections 2024-07-26T06:46:57.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.506+0000 7f20d2260640 1 --2- 172.21.15.42:0/3522006709 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f20b80b3bd0 0x7f20b80b60e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:57.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.506+0000 7f20d2260640 1 --2- 172.21.15.42:0/3522006709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20cc109850 0x7f20cc19f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:57.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.506+0000 7f20d2260640 1 --2- 172.21.15.42:0/3522006709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20cc108f00 0x7f20cc19a540 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:57.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.506+0000 7f20d2260640 1 --2- 172.21.15.42:0/3522006709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20cc1035c0 0x7f20cc199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:57.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.506+0000 7f20d2260640 1 -- 172.21.15.42:0/3522006709 >> 172.21.15.42:0/3522006709 conn(0x7f20cc0ff030 msgr2=0x7f20cc101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:57.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.507+0000 7f20d2260640 1 -- 172.21.15.42:0/3522006709 shutdown_connections 2024-07-26T06:46:57.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:57.507+0000 7f20d2260640 1 -- 172.21.15.42:0/3522006709 wait complete. 2024-07-26T06:46:58.074 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-07-26T06:46:58.074 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-07-26T06:46:58.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:57 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3522006709' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T06:46:58.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:57 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3522006709' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T06:46:58.341 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:46:58.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:57 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3522006709' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T06:46:59.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:46:58 smithi164 ceph-mon[36970]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:59.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:46:58 smithi163 ceph-mon[36873]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:59.328 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:46:58 smithi042 ceph-mon[32775]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:46:59.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.843+0000 7f92dd7ab640 1 -- 172.21.15.42:0/987572617 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92d8102510 msgr2=0x7f92d8102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:59.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.843+0000 7f92dd7ab640 1 --2- 172.21.15.42:0/987572617 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92d8102510 0x7f92d8102920 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f92c8009900 tx=0x7f92c802a410 comp rx=0 tx=0).stop 2024-07-26T06:46:59.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.843+0000 7f92dd7ab640 1 -- 172.21.15.42:0/987572617 shutdown_connections 2024-07-26T06:46:59.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.843+0000 7f92dd7ab640 1 --2- 172.21.15.42:0/987572617 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d8108750 0x7f92d810ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:59.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.843+0000 7f92dd7ab640 1 --2- 172.21.15.42:0/987572617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92d8107e00 0x7f92d8108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:59.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.843+0000 7f92dd7ab640 1 --2- 172.21.15.42:0/987572617 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92d8102510 0x7f92d8102920 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:59.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.843+0000 7f92dd7ab640 1 -- 172.21.15.42:0/987572617 >> 172.21.15.42:0/987572617 conn(0x7f92d8079190 msgr2=0x7f92d80795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:46:59.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.843+0000 7f92dd7ab640 1 -- 172.21.15.42:0/987572617 shutdown_connections 2024-07-26T06:46:59.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.843+0000 7f92dd7ab640 1 -- 172.21.15.42:0/987572617 wait complete. 2024-07-26T06:46:59.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.844+0000 7f92dd7ab640 1 Processor -- start 2024-07-26T06:46:59.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.845+0000 7f92dd7ab640 1 -- start start 2024-07-26T06:46:59.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.845+0000 7f92dd7ab640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92d8102510 0x7f92d8199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:59.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.845+0000 7f92dd7ab640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d8107e00 0x7f92d819a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:59.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.845+0000 7f92dd7ab640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92d8108750 0x7f92d819f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:59.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.845+0000 7f92dd7ab640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f92d81123d0 con 0x7f92d8102510 2024-07-26T06:46:59.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.845+0000 7f92dd7ab640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f92d8112250 con 0x7f92d8108750 2024-07-26T06:46:59.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.845+0000 7f92dd7ab640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f92d8112550 con 0x7f92d8107e00 2024-07-26T06:46:59.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.845+0000 7f92d6ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92d8102510 0x7f92d8199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:59.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.845+0000 7f92d6ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92d8102510 0x7f92d8199e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49224/0 (socket says 172.21.15.42:49224) 2024-07-26T06:46:59.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.845+0000 7f92d6ffd640 1 -- 172.21.15.42:0/268859201 learned_addr learned my addr 172.21.15.42:0/268859201 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:46:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.845+0000 7f92d77fe640 1 --2- 172.21.15.42:0/268859201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92d8108750 0x7f92d819f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.845+0000 7f92d67fc640 1 --2- 172.21.15.42:0/268859201 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d8107e00 0x7f92d819a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.845+0000 7f92d6ffd640 1 -- 172.21.15.42:0/268859201 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d8107e00 msgr2=0x7f92d819a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.845+0000 7f92d6ffd640 1 --2- 172.21.15.42:0/268859201 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d8107e00 0x7f92d819a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.845+0000 7f92d6ffd640 1 -- 172.21.15.42:0/268859201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92d8108750 msgr2=0x7f92d819f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:46:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.845+0000 7f92d6ffd640 1 --2- 172.21.15.42:0/268859201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92d8108750 0x7f92d819f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:46:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.845+0000 7f92d6ffd640 1 -- 172.21.15.42:0/268859201 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92d819fde0 con 0x7f92d8102510 2024-07-26T06:46:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.846+0000 7f92d6ffd640 1 --2- 172.21.15.42:0/268859201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92d8102510 0x7f92d8199e70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f92c8005940 tx=0x7f92c80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.846+0000 7f92bffff640 1 -- 172.21.15.42:0/268859201 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c803f030 con 0x7f92d8102510 2024-07-26T06:46:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.846+0000 7f92dd7ab640 1 -- 172.21.15.42:0/268859201 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92d81a0070 con 0x7f92d8102510 2024-07-26T06:46:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.846+0000 7f92bffff640 1 -- 172.21.15.42:0/268859201 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92c8003cb0 con 0x7f92d8102510 2024-07-26T06:46:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.846+0000 7f92bffff640 1 -- 172.21.15.42:0/268859201 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c8035070 con 0x7f92d8102510 2024-07-26T06:46:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.846+0000 7f92dd7ab640 1 -- 172.21.15.42:0/268859201 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92d81a6830 con 0x7f92d8102510 2024-07-26T06:46:59.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.847+0000 7f92dd7ab640 1 -- 172.21.15.42:0/268859201 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92d8103740 con 0x7f92d8102510 2024-07-26T06:46:59.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.850+0000 7f92bffff640 1 -- 172.21.15.42:0/268859201 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f92c8042030 con 0x7f92d8102510 2024-07-26T06:46:59.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.850+0000 7f92bffff640 1 --2- 172.21.15.42:0/268859201 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f92c00b3bd0 0x7f92c00b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:46:59.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.851+0000 7f92d67fc640 1 --2- 172.21.15.42:0/268859201 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f92c00b3bd0 0x7f92c00b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:46:59.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.851+0000 7f92bffff640 1 -- 172.21.15.42:0/268859201 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f92c8104310 con 0x7f92d8102510 2024-07-26T06:46:59.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.851+0000 7f92d67fc640 1 --2- 172.21.15.42:0/268859201 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f92c00b3bd0 0x7f92c00b60e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f92d8100750 tx=0x7f92cc005c60 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:46:59.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:46:59.854+0000 7f92bffff640 1 -- 172.21.15.42:0/268859201 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f92c80cbe00 con 0x7f92d8102510 2024-07-26T06:47:00.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:00.037+0000 7f92dd7ab640 1 -- 172.21.15.42:0/268859201 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f92d8102920 con 0x7f92d8102510 2024-07-26T06:47:00.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:00.977+0000 7f92bffff640 1 -- 172.21.15.42:0/268859201 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v72) ==== 138+0+0 (secure 0 0 0) 0x7f92c80c5e20 con 0x7f92d8102510 2024-07-26T06:47:00.978 INFO:teuthology.orchestra.run.smithi042.stderr:pool 'cephfs_metadata' created 2024-07-26T06:47:00.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:00.981+0000 7f92dd7ab640 1 -- 172.21.15.42:0/268859201 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f92c00b3bd0 msgr2=0x7f92c00b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:00.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:00.981+0000 7f92dd7ab640 1 --2- 172.21.15.42:0/268859201 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f92c00b3bd0 0x7f92c00b60e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f92d8100750 tx=0x7f92cc005c60 comp rx=0 tx=0).stop 2024-07-26T06:47:00.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:00.981+0000 7f92dd7ab640 1 -- 172.21.15.42:0/268859201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92d8102510 msgr2=0x7f92d8199e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:00.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:00.981+0000 7f92dd7ab640 1 --2- 172.21.15.42:0/268859201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92d8102510 0x7f92d8199e70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f92c8005940 tx=0x7f92c80042f0 comp rx=0 tx=0).stop 2024-07-26T06:47:00.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:00.981+0000 7f92dd7ab640 1 -- 172.21.15.42:0/268859201 shutdown_connections 2024-07-26T06:47:00.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:00.981+0000 7f92dd7ab640 1 --2- 172.21.15.42:0/268859201 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f92c00b3bd0 0x7f92c00b60e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:00.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:00.981+0000 7f92dd7ab640 1 --2- 172.21.15.42:0/268859201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92d8108750 0x7f92d819f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:00.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:00.981+0000 7f92dd7ab640 1 --2- 172.21.15.42:0/268859201 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f92d8107e00 0x7f92d819a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:00.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:00.981+0000 7f92dd7ab640 1 --2- 172.21.15.42:0/268859201 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92d8102510 0x7f92d8199e70 secure :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f92c8005940 tx=0x7f92c80042f0 comp rx=0 tx=0).stop 2024-07-26T06:47:00.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:00.981+0000 7f92dd7ab640 1 -- 172.21.15.42:0/268859201 >> 172.21.15.42:0/268859201 conn(0x7f92d8079190 msgr2=0x7f92d8106130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:00.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:00.981+0000 7f92dd7ab640 1 -- 172.21.15.42:0/268859201 shutdown_connections 2024-07-26T06:47:00.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:00.982+0000 7f92dd7ab640 1 -- 172.21.15.42:0/268859201 wait complete. 2024-07-26T06:47:01.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:00 smithi163 ceph-mon[36873]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:47:01.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:00 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/268859201' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T06:47:01.293 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:00 smithi164 ceph-mon[36970]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:47:01.293 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:00 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/268859201' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T06:47:01.337 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:00 smithi042 ceph-mon[32775]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:47:01.337 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:00 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/268859201' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T06:47:01.632 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-07-26T06:47:01.907 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:02.238 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:01 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/268859201' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T06:47:02.238 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:01 smithi164 ceph-mon[36970]: osdmap e72: 12 total, 12 up, 12 in 2024-07-26T06:47:02.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:01 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/268859201' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T06:47:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:01 smithi163 ceph-mon[36873]: osdmap e72: 12 total, 12 up, 12 in 2024-07-26T06:47:02.307 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:01 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/268859201' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T06:47:02.308 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:01 smithi042 ceph-mon[32775]: osdmap e72: 12 total, 12 up, 12 in 2024-07-26T06:47:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:02 smithi163 ceph-mon[36873]: pgmap v292: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:47:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:02 smithi163 ceph-mon[36873]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T06:47:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:02 smithi163 ceph-mon[36873]: osdmap e73: 12 total, 12 up, 12 in 2024-07-26T06:47:03.260 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:02 smithi042 ceph-mon[32775]: pgmap v292: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:47:03.261 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:02 smithi042 ceph-mon[32775]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T06:47:03.261 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:02 smithi042 ceph-mon[32775]: osdmap e73: 12 total, 12 up, 12 in 2024-07-26T06:47:03.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:02 smithi164 ceph-mon[36970]: pgmap v292: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:47:03.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:02 smithi164 ceph-mon[36970]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T06:47:03.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:02 smithi164 ceph-mon[36970]: osdmap e73: 12 total, 12 up, 12 in 2024-07-26T06:47:03.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.474+0000 7f5dbb744640 1 -- 172.21.15.42:0/2681727214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5db4108750 msgr2=0x7f5db410ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:03.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.474+0000 7f5dbb744640 1 --2- 172.21.15.42:0/2681727214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5db4108750 0x7f5db410ef80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009ca0 tx=0x7f5dac02a8c0 comp rx=0 tx=0).stop 2024-07-26T06:47:03.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.474+0000 7f5dbb744640 1 -- 172.21.15.42:0/2681727214 shutdown_connections 2024-07-26T06:47:03.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.474+0000 7f5dbb744640 1 --2- 172.21.15.42:0/2681727214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5db4108750 0x7f5db410ef80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:03.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.474+0000 7f5dbb744640 1 --2- 172.21.15.42:0/2681727214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5db4107e00 0x7f5db4108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:03.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.474+0000 7f5dbb744640 1 --2- 172.21.15.42:0/2681727214 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5db4102510 0x7f5db4102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:03.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.474+0000 7f5dbb744640 1 -- 172.21.15.42:0/2681727214 >> 172.21.15.42:0/2681727214 conn(0x7f5db4079190 msgr2=0x7f5db40795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:03.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.474+0000 7f5dbb744640 1 -- 172.21.15.42:0/2681727214 shutdown_connections 2024-07-26T06:47:03.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.474+0000 7f5dbb744640 1 -- 172.21.15.42:0/2681727214 wait complete. 2024-07-26T06:47:03.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.475+0000 7f5dbb744640 1 Processor -- start 2024-07-26T06:47:03.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.475+0000 7f5dbb744640 1 -- start start 2024-07-26T06:47:03.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.475+0000 7f5dbb744640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5db4102510 0x7f5db4199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:03.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.475+0000 7f5dbb744640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5db4107e00 0x7f5db419a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:03.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.475+0000 7f5dbb744640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5db4108750 0x7f5db419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:03.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.475+0000 7f5dbb744640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5db41123d0 con 0x7f5db4108750 2024-07-26T06:47:03.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.475+0000 7f5dbb744640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5db4112250 con 0x7f5db4107e00 2024-07-26T06:47:03.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.475+0000 7f5dbb744640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5db4112550 con 0x7f5db4102510 2024-07-26T06:47:03.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.475+0000 7f5db9ce1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5db4108750 0x7f5db419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:03.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.475+0000 7f5db9ce1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5db4108750 0x7f5db419f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49244/0 (socket says 172.21.15.42:49244) 2024-07-26T06:47:03.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.475+0000 7f5db9ce1640 1 -- 172.21.15.42:0/213051238 learned_addr learned my addr 172.21.15.42:0/213051238 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:03.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.475+0000 7f5db94e0640 1 --2- 172.21.15.42:0/213051238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5db4102510 0x7f5db4199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:03.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.475+0000 7f5db8cdf640 1 --2- 172.21.15.42:0/213051238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5db4107e00 0x7f5db419a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:03.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.476+0000 7f5db9ce1640 1 -- 172.21.15.42:0/213051238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5db4102510 msgr2=0x7f5db4199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:03.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.476+0000 7f5db9ce1640 1 --2- 172.21.15.42:0/213051238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5db4102510 0x7f5db4199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:03.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.476+0000 7f5db9ce1640 1 -- 172.21.15.42:0/213051238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5db4107e00 msgr2=0x7f5db419a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:03.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.476+0000 7f5db9ce1640 1 --2- 172.21.15.42:0/213051238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5db4107e00 0x7f5db419a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:03.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.476+0000 7f5db9ce1640 1 -- 172.21.15.42:0/213051238 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db419fda0 con 0x7f5db4108750 2024-07-26T06:47:03.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.476+0000 7f5db8cdf640 1 --2- 172.21.15.42:0/213051238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5db4107e00 0x7f5db419a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:47:03.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.476+0000 7f5db9ce1640 1 --2- 172.21.15.42:0/213051238 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5db4108750 0x7f5db419f730 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0098d0 tx=0x7f5dac0317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:03.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.476+0000 7f5daa7fc640 1 -- 172.21.15.42:0/213051238 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dac003b70 con 0x7f5db4108750 2024-07-26T06:47:03.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.476+0000 7f5dbb744640 1 -- 172.21.15.42:0/213051238 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db41a0030 con 0x7f5db4108750 2024-07-26T06:47:03.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.476+0000 7f5daa7fc640 1 -- 172.21.15.42:0/213051238 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5dac003d10 con 0x7f5db4108750 2024-07-26T06:47:03.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.476+0000 7f5daa7fc640 1 -- 172.21.15.42:0/213051238 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dac02fa20 con 0x7f5db4108750 2024-07-26T06:47:03.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.476+0000 7f5dbb744640 1 -- 172.21.15.42:0/213051238 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db41a0570 con 0x7f5db4108750 2024-07-26T06:47:03.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.477+0000 7f5dbb744640 1 -- 172.21.15.42:0/213051238 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5db4103740 con 0x7f5db4108750 2024-07-26T06:47:03.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.479+0000 7f5daa7fc640 1 -- 172.21.15.42:0/213051238 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f5dac02fbc0 con 0x7f5db4108750 2024-07-26T06:47:03.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.479+0000 7f5daa7fc640 1 --2- 172.21.15.42:0/213051238 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5da00b3c00 0x7f5da00b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:03.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.479+0000 7f5db94e0640 1 --2- 172.21.15.42:0/213051238 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5da00b3c00 0x7f5da00b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:03.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.479+0000 7f5daa7fc640 1 -- 172.21.15.42:0/213051238 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7497+0+0 (secure 0 0 0) 0x7f5dac104fe0 con 0x7f5db4108750 2024-07-26T06:47:03.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.480+0000 7f5db94e0640 1 --2- 172.21.15.42:0/213051238 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5da00b3c00 0x7f5da00b6110 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5db0005e10 tx=0x7f5db0005da0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:03.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.483+0000 7f5daa7fc640 1 -- 172.21.15.42:0/213051238 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5dac0cc960 con 0x7f5db4108750 2024-07-26T06:47:03.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.669+0000 7f5dbb744640 1 -- 172.21.15.42:0/213051238 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f5db4102920 con 0x7f5db4108750 2024-07-26T06:47:03.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:03.998+0000 7f5daa7fc640 1 -- 172.21.15.42:0/213051238 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v75) ==== 170+0+0 (secure 0 0 0) 0x7f5db4102920 con 0x7f5db4108750 2024-07-26T06:47:03.999 INFO:teuthology.orchestra.run.smithi042.stderr:pool 'cephfs_data' created 2024-07-26T06:47:04.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:04.001+0000 7f5dbb744640 1 -- 172.21.15.42:0/213051238 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5da00b3c00 msgr2=0x7f5da00b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:04.001+0000 7f5dbb744640 1 --2- 172.21.15.42:0/213051238 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5da00b3c00 0x7f5da00b6110 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5db0005e10 tx=0x7f5db0005da0 comp rx=0 tx=0).stop 2024-07-26T06:47:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:04.001+0000 7f5dbb744640 1 -- 172.21.15.42:0/213051238 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5db4108750 msgr2=0x7f5db419f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:04.001+0000 7f5dbb744640 1 --2- 172.21.15.42:0/213051238 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5db4108750 0x7f5db419f730 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0098d0 tx=0x7f5dac0317b0 comp rx=0 tx=0).stop 2024-07-26T06:47:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:04.001+0000 7f5dbb744640 1 -- 172.21.15.42:0/213051238 shutdown_connections 2024-07-26T06:47:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:04.001+0000 7f5dbb744640 1 --2- 172.21.15.42:0/213051238 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5da00b3c00 0x7f5da00b6110 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:04.001+0000 7f5dbb744640 1 --2- 172.21.15.42:0/213051238 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5db4108750 0x7f5db419f730 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:04.001+0000 7f5dbb744640 1 --2- 172.21.15.42:0/213051238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5db4107e00 0x7f5db419a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:04.001+0000 7f5dbb744640 1 --2- 172.21.15.42:0/213051238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5db4102510 0x7f5db4199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:04.001+0000 7f5dbb744640 1 -- 172.21.15.42:0/213051238 >> 172.21.15.42:0/213051238 conn(0x7f5db4079190 msgr2=0x7f5db406b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:04.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:04.002+0000 7f5dbb744640 1 -- 172.21.15.42:0/213051238 shutdown_connections 2024-07-26T06:47:04.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:04.002+0000 7f5dbb744640 1 -- 172.21.15.42:0/213051238 wait complete. 2024-07-26T06:47:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:04 smithi163 ceph-mon[36873]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T06:47:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:04 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/213051238' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T06:47:04.308 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:03 smithi042 ceph-mon[32775]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T06:47:04.309 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:03 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/213051238' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T06:47:04.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:04 smithi164 ceph-mon[36970]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T06:47:04.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:04 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/213051238' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T06:47:04.686 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-07-26T06:47:04.960 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:05 smithi163 ceph-mon[36873]: pgmap v295: 33 pgs: 1 creating+activating, 6 creating+peering, 17 unknown, 9 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:47:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:05 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/213051238' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T06:47:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:05 smithi163 ceph-mon[36873]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T06:47:05.293 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:05 smithi042 ceph-mon[32775]: pgmap v295: 33 pgs: 1 creating+activating, 6 creating+peering, 17 unknown, 9 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:47:05.293 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:05 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/213051238' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T06:47:05.293 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:05 smithi042 ceph-mon[32775]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T06:47:05.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:05 smithi164 ceph-mon[36970]: pgmap v295: 33 pgs: 1 creating+activating, 6 creating+peering, 17 unknown, 9 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:47:05.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:05 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/213051238' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T06:47:05.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:05 smithi164 ceph-mon[36970]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T06:47:06.335 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:06 smithi042 ceph-mon[32775]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T06:47:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:06 smithi164 ceph-mon[36970]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T06:47:06.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:06 smithi163 ceph-mon[36873]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T06:47:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.543+0000 7fb517430640 1 -- 172.21.15.42:0/2796853971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb510075800 msgr2=0x7fb510075c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.543+0000 7fb517430640 1 --2- 172.21.15.42:0/2796853971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb510075800 0x7fb510075c10 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb500009900 tx=0x7fb50002a410 comp rx=0 tx=0).stop 2024-07-26T06:47:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.543+0000 7fb517430640 1 -- 172.21.15.42:0/2796853971 shutdown_connections 2024-07-26T06:47:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.543+0000 7fb517430640 1 --2- 172.21.15.42:0/2796853971 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb510113eb0 0x7fb510116380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.543+0000 7fb517430640 1 --2- 172.21.15.42:0/2796853971 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb510075800 0x7fb510075c10 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.543+0000 7fb517430640 1 --2- 172.21.15.42:0/2796853971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb510076db0 0x7fb5100751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.543+0000 7fb517430640 1 -- 172.21.15.42:0/2796853971 >> 172.21.15.42:0/2796853971 conn(0x7fb5100ff030 msgr2=0x7fb510101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.543+0000 7fb517430640 1 -- 172.21.15.42:0/2796853971 shutdown_connections 2024-07-26T06:47:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.543+0000 7fb517430640 1 -- 172.21.15.42:0/2796853971 wait complete. 2024-07-26T06:47:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.544+0000 7fb517430640 1 Processor -- start 2024-07-26T06:47:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.544+0000 7fb517430640 1 -- start start 2024-07-26T06:47:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.544+0000 7fb517430640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb510075800 0x7fb5101a0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.544+0000 7fb517430640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb510076db0 0x7fb5101a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.544+0000 7fb517430640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb510113eb0 0x7fb5101a5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.544+0000 7fb517430640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb510119af0 con 0x7fb510076db0 2024-07-26T06:47:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.544+0000 7fb517430640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb510119970 con 0x7fb510113eb0 2024-07-26T06:47:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.544+0000 7fb517430640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb510119c70 con 0x7fb510075800 2024-07-26T06:47:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.544+0000 7fb5149cb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb510076db0 0x7fb5101a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.544+0000 7fb5149cb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb510076db0 0x7fb5101a0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49264/0 (socket says 172.21.15.42:49264) 2024-07-26T06:47:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.544+0000 7fb5149cb640 1 -- 172.21.15.42:0/2987984728 learned_addr learned my addr 172.21.15.42:0/2987984728 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.544+0000 7fb5151cc640 1 --2- 172.21.15.42:0/2987984728 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb510075800 0x7fb5101a0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.544+0000 7fb5159cd640 1 --2- 172.21.15.42:0/2987984728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb510113eb0 0x7fb5101a5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.545+0000 7fb5149cb640 1 -- 172.21.15.42:0/2987984728 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb510075800 msgr2=0x7fb5101a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.545+0000 7fb5149cb640 1 --2- 172.21.15.42:0/2987984728 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb510075800 0x7fb5101a0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.545+0000 7fb5149cb640 1 -- 172.21.15.42:0/2987984728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb510113eb0 msgr2=0x7fb5101a5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.545+0000 7fb5149cb640 1 --2- 172.21.15.42:0/2987984728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb510113eb0 0x7fb5101a5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.545+0000 7fb5149cb640 1 -- 172.21.15.42:0/2987984728 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5101a64d0 con 0x7fb510076db0 2024-07-26T06:47:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.545+0000 7fb5151cc640 1 --2- 172.21.15.42:0/2987984728 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb510075800 0x7fb5101a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:47:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.545+0000 7fb5149cb640 1 --2- 172.21.15.42:0/2987984728 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb510076db0 0x7fb5101a0b00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb500032fd0 tx=0x7fb500004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.545+0000 7fb5067fc640 1 -- 172.21.15.42:0/2987984728 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb50003f030 con 0x7fb510076db0 2024-07-26T06:47:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.545+0000 7fb5067fc640 1 -- 172.21.15.42:0/2987984728 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb500004490 con 0x7fb510076db0 2024-07-26T06:47:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.545+0000 7fb5067fc640 1 -- 172.21.15.42:0/2987984728 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb500035070 con 0x7fb510076db0 2024-07-26T06:47:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.545+0000 7fb517430640 1 -- 172.21.15.42:0/2987984728 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5101a6760 con 0x7fb510076db0 2024-07-26T06:47:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.545+0000 7fb517430640 1 -- 172.21.15.42:0/2987984728 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5101acf20 con 0x7fb510076db0 2024-07-26T06:47:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.546+0000 7fb517430640 1 -- 172.21.15.42:0/2987984728 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5100766f0 con 0x7fb510076db0 2024-07-26T06:47:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.548+0000 7fb5067fc640 1 -- 172.21.15.42:0/2987984728 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fb500042030 con 0x7fb510076db0 2024-07-26T06:47:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.549+0000 7fb5067fc640 1 --2- 172.21.15.42:0/2987984728 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb4fc0b3c50 0x7fb4fc0b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.549+0000 7fb5151cc640 1 --2- 172.21.15.42:0/2987984728 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb4fc0b3c50 0x7fb4fc0b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.549+0000 7fb5067fc640 1 -- 172.21.15.42:0/2987984728 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7879+0+0 (secure 0 0 0) 0x7fb500104350 con 0x7fb510076db0 2024-07-26T06:47:06.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.549+0000 7fb5151cc640 1 --2- 172.21.15.42:0/2987984728 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb4fc0b3c50 0x7fb4fc0b6160 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb50c0058c0 tx=0x7fb50c005d30 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:06.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.552+0000 7fb5067fc640 1 -- 172.21.15.42:0/2987984728 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb5000cbb50 con 0x7fb510076db0 2024-07-26T06:47:06.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:06.779+0000 7fb517430640 1 -- 172.21.15.42:0/2987984728 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb510112c40 con 0x7fb510076db0 2024-07-26T06:47:07.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:07.012+0000 7fb5067fc640 1 -- 172.21.15.42:0/2987984728 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fb5000cb450 con 0x7fb510076db0 2024-07-26T06:47:07.013 INFO:teuthology.orchestra.run.smithi042.stderr:new fs with metadata pool 2 and data pool 3 2024-07-26T06:47:07.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:07.015+0000 7fb517430640 1 -- 172.21.15.42:0/2987984728 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb4fc0b3c50 msgr2=0x7fb4fc0b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:07.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:07.015+0000 7fb517430640 1 --2- 172.21.15.42:0/2987984728 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb4fc0b3c50 0x7fb4fc0b6160 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb50c0058c0 tx=0x7fb50c005d30 comp rx=0 tx=0).stop 2024-07-26T06:47:07.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:07.015+0000 7fb517430640 1 -- 172.21.15.42:0/2987984728 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb510076db0 msgr2=0x7fb5101a0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:07.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:07.015+0000 7fb517430640 1 --2- 172.21.15.42:0/2987984728 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb510076db0 0x7fb5101a0b00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb500032fd0 tx=0x7fb500004270 comp rx=0 tx=0).stop 2024-07-26T06:47:07.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:07.015+0000 7fb517430640 1 -- 172.21.15.42:0/2987984728 shutdown_connections 2024-07-26T06:47:07.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:07.015+0000 7fb517430640 1 --2- 172.21.15.42:0/2987984728 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb4fc0b3c50 0x7fb4fc0b6160 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:07.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:07.015+0000 7fb517430640 1 --2- 172.21.15.42:0/2987984728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb510113eb0 0x7fb5101a5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:07.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:07.015+0000 7fb517430640 1 --2- 172.21.15.42:0/2987984728 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb510076db0 0x7fb5101a0b00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:07.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:07.015+0000 7fb517430640 1 --2- 172.21.15.42:0/2987984728 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb510075800 0x7fb5101a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:07.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:07.015+0000 7fb517430640 1 -- 172.21.15.42:0/2987984728 >> 172.21.15.42:0/2987984728 conn(0x7fb5100ff030 msgr2=0x7fb51010fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:07.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:07.016+0000 7fb517430640 1 -- 172.21.15.42:0/2987984728 shutdown_connections 2024-07-26T06:47:07.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:07.016+0000 7fb517430640 1 -- 172.21.15.42:0/2987984728 wait complete. 2024-07-26T06:47:07.030 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:07 smithi042 ceph-mon[32775]: pgmap v298: 97 pgs: 1 creating+activating, 22 creating+peering, 48 unknown, 26 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:47:07.031 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:07 smithi042 ceph-mon[32775]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T06:47:07.031 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:07 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2987984728' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T06:47:07.031 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:07 smithi042 ceph-mon[32775]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T06:47:07.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:07 smithi164 ceph-mon[36970]: pgmap v298: 97 pgs: 1 creating+activating, 22 creating+peering, 48 unknown, 26 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:47:07.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:07 smithi164 ceph-mon[36970]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T06:47:07.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:07 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2987984728' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T06:47:07.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:07 smithi164 ceph-mon[36970]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T06:47:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:07 smithi163 ceph-mon[36873]: pgmap v298: 97 pgs: 1 creating+activating, 22 creating+peering, 48 unknown, 26 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:47:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:07 smithi163 ceph-mon[36873]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T06:47:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:07 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2987984728' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T06:47:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:07 smithi163 ceph-mon[36873]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T06:47:07.909 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T06:47:08.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:08 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2987984728' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T06:47:08.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:08 smithi042 ceph-mon[32775]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T06:47:08.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:08 smithi042 ceph-mon[32775]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-07-26T06:47:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:08 smithi042 ceph-mon[32775]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T06:47:08.187 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:08 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2987984728' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T06:47:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:08 smithi164 ceph-mon[36970]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T06:47:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:08 smithi164 ceph-mon[36970]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-07-26T06:47:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:08 smithi164 ceph-mon[36970]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T06:47:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:08 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2987984728' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T06:47:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:08 smithi163 ceph-mon[36873]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T06:47:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:08 smithi163 ceph-mon[36873]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-07-26T06:47:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:08 smithi163 ceph-mon[36873]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T06:47:09.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:09 smithi042 ceph-mon[32775]: pgmap v301: 97 pgs: 16 creating+peering, 25 unknown, 56 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:47:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:09 smithi042 ceph-mon[32775]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T06:47:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:09 smithi042 ceph-mon[32775]: Cluster is now healthy 2024-07-26T06:47:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:09 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] up:active 2024-07-26T06:47:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:09 smithi042 ceph-mon[32775]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T06:47:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:09 smithi042 ceph-mon[32775]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T06:47:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:09 smithi164 ceph-mon[36970]: pgmap v301: 97 pgs: 16 creating+peering, 25 unknown, 56 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:47:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:09 smithi164 ceph-mon[36970]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T06:47:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:09 smithi164 ceph-mon[36970]: Cluster is now healthy 2024-07-26T06:47:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:09 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] up:active 2024-07-26T06:47:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:09 smithi164 ceph-mon[36970]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T06:47:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:09 smithi164 ceph-mon[36970]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T06:47:09.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:09 smithi163 ceph-mon[36873]: pgmap v301: 97 pgs: 16 creating+peering, 25 unknown, 56 active+clean; 577 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T06:47:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:09 smithi163 ceph-mon[36873]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T06:47:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:09 smithi163 ceph-mon[36873]: Cluster is now healthy 2024-07-26T06:47:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:09 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] up:active 2024-07-26T06:47:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:09 smithi163 ceph-mon[36873]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T06:47:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:09 smithi163 ceph-mon[36873]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T06:47:09.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.754+0000 7f53c4b9b640 1 -- 172.21.15.42:0/4120839442 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c0107e00 msgr2=0x7f53c0108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:09.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.754+0000 7f53c4b9b640 1 --2- 172.21.15.42:0/4120839442 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c0107e00 0x7f53c0108210 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f53a8009900 tx=0x7f53a802a410 comp rx=0 tx=0).stop 2024-07-26T06:47:09.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.754+0000 7f53c4b9b640 1 -- 172.21.15.42:0/4120839442 shutdown_connections 2024-07-26T06:47:09.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.754+0000 7f53c4b9b640 1 --2- 172.21.15.42:0/4120839442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53c0108750 0x7f53c010ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:09.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.754+0000 7f53c4b9b640 1 --2- 172.21.15.42:0/4120839442 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c0107e00 0x7f53c0108210 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:09.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.754+0000 7f53c4b9b640 1 --2- 172.21.15.42:0/4120839442 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53c01024c0 0x7f53c01028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:09.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.754+0000 7f53c4b9b640 1 -- 172.21.15.42:0/4120839442 >> 172.21.15.42:0/4120839442 conn(0x7f53c0079180 msgr2=0x7f53c0079590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:09.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.754+0000 7f53c4b9b640 1 -- 172.21.15.42:0/4120839442 shutdown_connections 2024-07-26T06:47:09.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.754+0000 7f53c4b9b640 1 -- 172.21.15.42:0/4120839442 wait complete. 2024-07-26T06:47:09.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.755+0000 7f53c4b9b640 1 Processor -- start 2024-07-26T06:47:09.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.755+0000 7f53c4b9b640 1 -- start start 2024-07-26T06:47:09.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53c4b9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53c01024c0 0x7f53c0199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:09.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53c4b9b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53c0107e00 0x7f53c019a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:09.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53c4b9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c0108750 0x7f53c019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:09.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53c4b9b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f53c01134e0 con 0x7f53c0108750 2024-07-26T06:47:09.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53c4b9b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f53c0113360 con 0x7f53c0107e00 2024-07-26T06:47:09.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53c4b9b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f53c0113660 con 0x7f53c01024c0 2024-07-26T06:47:09.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53bed9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c0108750 0x7f53c019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:09.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53bed9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c0108750 0x7f53c019f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48460/0 (socket says 172.21.15.42:48460) 2024-07-26T06:47:09.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53bed9d640 1 -- 172.21.15.42:0/4192599138 learned_addr learned my addr 172.21.15.42:0/4192599138 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:09.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53be59c640 1 --2- 172.21.15.42:0/4192599138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53c01024c0 0x7f53c0199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:09.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53bdd9b640 1 --2- 172.21.15.42:0/4192599138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53c0107e00 0x7f53c019a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:09.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53bed9d640 1 -- 172.21.15.42:0/4192599138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53c01024c0 msgr2=0x7f53c0199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:09.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53bed9d640 1 --2- 172.21.15.42:0/4192599138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53c01024c0 0x7f53c0199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:09.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53bed9d640 1 -- 172.21.15.42:0/4192599138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53c0107e00 msgr2=0x7f53c019a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:09.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53bed9d640 1 --2- 172.21.15.42:0/4192599138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53c0107e00 0x7f53c019a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:09.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53bed9d640 1 -- 172.21.15.42:0/4192599138 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c019fda0 con 0x7f53c0108750 2024-07-26T06:47:09.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53bed9d640 1 --2- 172.21.15.42:0/4192599138 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c0108750 0x7f53c019f730 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f53b800cce0 tx=0x7f53b800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:09.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.756+0000 7f53b77fe640 1 -- 172.21.15.42:0/4192599138 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b800f450 con 0x7f53c0108750 2024-07-26T06:47:09.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.757+0000 7f53b77fe640 1 -- 172.21.15.42:0/4192599138 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53b800faa0 con 0x7f53c0108750 2024-07-26T06:47:09.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.757+0000 7f53b77fe640 1 -- 172.21.15.42:0/4192599138 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b8018b70 con 0x7f53c0108750 2024-07-26T06:47:09.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.757+0000 7f53c4b9b640 1 -- 172.21.15.42:0/4192599138 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53c01a0090 con 0x7f53c0108750 2024-07-26T06:47:09.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.757+0000 7f53c4b9b640 1 -- 172.21.15.42:0/4192599138 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53c01a0550 con 0x7f53c0108750 2024-07-26T06:47:09.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.758+0000 7f53c4b9b640 1 -- 172.21.15.42:0/4192599138 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53c0103740 con 0x7f53c0108750 2024-07-26T06:47:09.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.759+0000 7f53b77fe640 1 -- 172.21.15.42:0/4192599138 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f53b8018d10 con 0x7f53c0108750 2024-07-26T06:47:09.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.760+0000 7f53b77fe640 1 --2- 172.21.15.42:0/4192599138 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f53ac0b3c80 0x7f53ac0b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:09.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.760+0000 7f53be59c640 1 --2- 172.21.15.42:0/4192599138 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f53ac0b3c80 0x7f53ac0b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:09.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.760+0000 7f53b77fe640 1 -- 172.21.15.42:0/4192599138 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7f53b80e3de0 con 0x7f53c0108750 2024-07-26T06:47:09.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.761+0000 7f53be59c640 1 --2- 172.21.15.42:0/4192599138 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f53ac0b3c80 0x7f53ac0b6190 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f53b00058c0 tx=0x7f53b000a2d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:09.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.763+0000 7f53b77fe640 1 -- 172.21.15.42:0/4192599138 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f53b8014030 con 0x7f53c0108750 2024-07-26T06:47:09.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.946+0000 7f53c4b9b640 1 -- 172.21.15.42:0/4192599138 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f53c01028d0 con 0x7f53c0108750 2024-07-26T06:47:09.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.948+0000 7f53b77fe640 1 -- 172.21.15.42:0/4192599138 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22170 (secure 0 0 0) 0x7f53c01028d0 con 0x7f53c0108750 2024-07-26T06:47:09.949 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:47:09.949 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":78,"fsid":"8b3e2750-4b19-11ef-bcb6-c7b262605968","created":"2024-07-26T06:38:29.306470+0000","modified":"2024-07-26T06:47:06.780610+0000","last_up_change":"2024-07-26T06:46:02.149453+0000","last_in_change":"2024-07-26T06:45:43.027764+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:42:27.302195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:47:00.042262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T06:47:03.674160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bda88e0-09ee-447b-8c04-24f4e5d55d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6803","nonce":3587066379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6805","nonce":3587066379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6809","nonce":3587066379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6807","nonce":3587066379}]},"public_addr":"172.21.15.42:6803/3587066379","cluster_addr":"172.21.15.42:6805/3587066379","heartbeat_back_addr":"172.21.15.42:6809/3587066379","heartbeat_front_addr":"172.21.15.42:6807/3587066379","state":["exists","up"]},{"osd":1,"uuid":"7d747138-8217-4836-a173-cf8ac65e9fe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6801","nonce":360219078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6803","nonce":360219078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6807","nonce":360219078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6805","nonce":360219078}]},"public_addr":"172.21.15.163:6801/360219078","cluster_addr":"172.21.15.163:6803/360219078","heartbeat_back_addr":"172.21.15.163:6807/360219078","heartbeat_front_addr":"172.21.15.163:6805/360219078","state":["exists","up"]},{"osd":2,"uuid":"4b7258c1-3d35-4930-ab10-0b04a2a2c8a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6801","nonce":336113412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6803","nonce":336113412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6807","nonce":336113412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6805","nonce":336113412}]},"public_addr":"172.21.15.164:6801/336113412","cluster_addr":"172.21.15.164:6803/336113412","heartbeat_back_addr":"172.21.15.164:6807/336113412","heartbeat_front_addr":"172.21.15.164:6805/336113412","state":["exists","up"]},{"osd":3,"uuid":"96608387-0de0-4d90-a952-d6eba61f7fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6811","nonce":2819018064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6813","nonce":2819018064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6817","nonce":2819018064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6815","nonce":2819018064}]},"public_addr":"172.21.15.42:6811/2819018064","cluster_addr":"172.21.15.42:6813/2819018064","heartbeat_back_addr":"172.21.15.42:6817/2819018064","heartbeat_front_addr":"172.21.15.42:6815/2819018064","state":["exists","up"]},{"osd":4,"uuid":"fabf6b02-fe34-4560-85b5-01bf37fc9be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6809","nonce":431934374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6811","nonce":431934374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6815","nonce":431934374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6813","nonce":431934374}]},"public_addr":"172.21.15.163:6809/431934374","cluster_addr":"172.21.15.163:6811/431934374","heartbeat_back_addr":"172.21.15.163:6815/431934374","heartbeat_front_addr":"172.21.15.163:6813/431934374","state":["exists","up"]},{"osd":5,"uuid":"e226fb4f-15b2-40df-af34-900040ba93ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6809","nonce":261549362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6811","nonce":261549362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6815","nonce":261549362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6813","nonce":261549362}]},"public_addr":"172.21.15.164:6809/261549362","cluster_addr":"172.21.15.164:6811/261549362","heartbeat_back_addr":"172.21.15.164:6815/261549362","heartbeat_front_addr":"172.21.15.164:6813/261549362","state":["exists","up"]},{"osd":6,"uuid":"beb8ec5d-7ea5-4e4e-a46c-a84f7768cec0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6819","nonce":2160811300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6821","nonce":2160811300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6825","nonce":2160811300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6823","nonce":2160811300}]},"public_addr":"172.21.15.42:6819/2160811300","cluster_addr":"172.21.15.42:6821/2160811300","heartbeat_back_addr":"172.21.15.42:6825/2160811300","heartbeat_front_addr":"172.21.15.42:6823/2160811300","state":["exists","up"]},{"osd":7,"uuid":"f699004c-f7c2-4f1c-ae4b-7178f3b06e2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6817","nonce":812214111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6819","nonce":812214111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6823","nonce":812214111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6821","nonce":812214111}]},"public_addr":"172.21.15.163:6817/812214111","cluster_addr":"172.21.15.163:6819/812214111","heartbeat_back_addr":"172.21.15.163:6823/812214111","heartbeat_front_addr":"172.21.15.163:6821/812214111","state":["exists","up"]},{"osd":8,"uuid":"3476163e-efea-4652-a8d4-b7c4a908f984","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6817","nonce":962845839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6819","nonce":962845839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6823","nonce":962845839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6821","nonce":962845839}]},"public_addr":"172.21.15.164:6817/962845839","cluster_addr":"172.21.15.164:6819/962845839","heartbeat_back_addr":"172.21.15.164:6823/962845839","heartbeat_front_addr":"172.21.15.164:6821/962845839","state":["exists","up"]},{"osd":9,"uuid":"6f7ceac6-fb61-4c6f-9da8-851b937c4586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6827","nonce":994024418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6829","nonce":994024418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6833","nonce":994024418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6831","nonce":994024418}]},"public_addr":"172.21.15.42:6827/994024418","cluster_addr":"172.21.15.42:6829/994024418","heartbeat_back_addr":"172.21.15.42:6833/994024418","heartbeat_front_addr":"172.21.15.42:6831/994024418","state":["exists","up"]},{"osd":10,"uuid":"553132c2-a6a1-410c-a7d8-ecdf9a67dafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6825","nonce":3268512227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6827","nonce":3268512227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6831","nonce":3268512227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6829","nonce":3268512227}]},"public_addr":"172.21.15.163:6825/3268512227","cluster_addr":"172.21.15.163:6827/3268512227","heartbeat_back_addr":"172.21.15.163:6831/3268512227","heartbeat_front_addr":"172.21.15.163:6829/3268512227","state":["exists","up"]},{"osd":11,"uuid":"09abc8e9-7884-400f-ab6d-8e5aa7bf31ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6825","nonce":1861076336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6827","nonce":1861076336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6831","nonce":1861076336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6829","nonce":1861076336}]},"public_addr":"172.21.15.164:6825/1861076336","cluster_addr":"172.21.15.164:6827/1861076336","heartbeat_back_addr":"172.21.15.164:6831/1861076336","heartbeat_front_addr":"172.21.15.164:6829/1861076336","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:41:38.761674+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:00.621745+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:23.248620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:47.232764+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:10.379922+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:34.502519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:58.694642+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:22.350247+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:46.988007+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:11.390474+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:36.150054+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:46:00.474470+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.42:6801/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6800/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1842521399":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/834172950":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1730109419":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6801/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:6800/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2687772889":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/3371433307":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2861719816":"2024-07-27T06:38:56.887264+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:47:09.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.950+0000 7f53c4b9b640 1 -- 172.21.15.42:0/4192599138 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f53ac0b3c80 msgr2=0x7f53ac0b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:09.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.950+0000 7f53c4b9b640 1 --2- 172.21.15.42:0/4192599138 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f53ac0b3c80 0x7f53ac0b6190 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f53b00058c0 tx=0x7f53b000a2d0 comp rx=0 tx=0).stop 2024-07-26T06:47:09.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.950+0000 7f53c4b9b640 1 -- 172.21.15.42:0/4192599138 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c0108750 msgr2=0x7f53c019f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:09.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.950+0000 7f53c4b9b640 1 --2- 172.21.15.42:0/4192599138 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c0108750 0x7f53c019f730 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f53b800cce0 tx=0x7f53b800a620 comp rx=0 tx=0).stop 2024-07-26T06:47:09.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.950+0000 7f53c4b9b640 1 -- 172.21.15.42:0/4192599138 shutdown_connections 2024-07-26T06:47:09.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.950+0000 7f53c4b9b640 1 --2- 172.21.15.42:0/4192599138 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f53ac0b3c80 0x7f53ac0b6190 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:09.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.950+0000 7f53c4b9b640 1 --2- 172.21.15.42:0/4192599138 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53c0108750 0x7f53c019f730 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:09.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.950+0000 7f53c4b9b640 1 --2- 172.21.15.42:0/4192599138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53c0107e00 0x7f53c019a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:09.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.950+0000 7f53c4b9b640 1 --2- 172.21.15.42:0/4192599138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53c01024c0 0x7f53c0199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:09.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.950+0000 7f53c4b9b640 1 -- 172.21.15.42:0/4192599138 >> 172.21.15.42:0/4192599138 conn(0x7f53c0079180 msgr2=0x7f53c0103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:09.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.950+0000 7f53c4b9b640 1 -- 172.21.15.42:0/4192599138 shutdown_connections 2024-07-26T06:47:09.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:09.950+0000 7f53c4b9b640 1 -- 172.21.15.42:0/4192599138 wait complete. 2024-07-26T06:47:10.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:10 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/4192599138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:47:10.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:10 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/4192599138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:47:10.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:10 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/4192599138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:47:10.563 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T06:47:10.827 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:11.037 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:47:10 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 2024-07-26T06:47:10.777+0000 7fe5373cb640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T06:47:11.288 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:11 smithi042 ceph-mon[32775]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 963 B/s wr, 6 op/s 2024-07-26T06:47:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:11 smithi164 ceph-mon[36970]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 963 B/s wr, 6 op/s 2024-07-26T06:47:11.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:11 smithi163 ceph-mon[36873]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 963 B/s wr, 6 op/s 2024-07-26T06:47:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.505+0000 7fdbb91e4640 1 -- 172.21.15.42:0/2490456173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbb4101b10 msgr2=0x7fdbb4101f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.505+0000 7fdbb91e4640 1 --2- 172.21.15.42:0/2490456173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbb4101b10 0x7fdbb4101f20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdba4009900 tx=0x7fdba402a410 comp rx=0 tx=0).stop 2024-07-26T06:47:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.506+0000 7fdbb91e4640 1 -- 172.21.15.42:0/2490456173 shutdown_connections 2024-07-26T06:47:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.506+0000 7fdbb91e4640 1 --2- 172.21.15.42:0/2490456173 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbb4106650 0x7fdbb41100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.506+0000 7fdbb91e4640 1 --2- 172.21.15.42:0/2490456173 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbb4105d00 0x7fdbb4106110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.506+0000 7fdbb91e4640 1 --2- 172.21.15.42:0/2490456173 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbb4101b10 0x7fdbb4101f20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.506+0000 7fdbb91e4640 1 -- 172.21.15.42:0/2490456173 >> 172.21.15.42:0/2490456173 conn(0x7fdbb4079870 msgr2=0x7fdbb4079c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.506+0000 7fdbb91e4640 1 -- 172.21.15.42:0/2490456173 shutdown_connections 2024-07-26T06:47:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.506+0000 7fdbb91e4640 1 -- 172.21.15.42:0/2490456173 wait complete. 2024-07-26T06:47:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.507+0000 7fdbb91e4640 1 Processor -- start 2024-07-26T06:47:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.507+0000 7fdbb91e4640 1 -- start start 2024-07-26T06:47:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.507+0000 7fdbb91e4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbb4101b10 0x7fdbb4199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.507+0000 7fdbb91e4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbb4105d00 0x7fdbb419a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.507+0000 7fdbb91e4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbb4106650 0x7fdbb419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.508+0000 7fdbb91e4640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb41134f0 con 0x7fdbb4106650 2024-07-26T06:47:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.508+0000 7fdbb91e4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb4113370 con 0x7fdbb4101b10 2024-07-26T06:47:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.508+0000 7fdbb91e4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb4113670 con 0x7fdbb4105d00 2024-07-26T06:47:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.508+0000 7fdbb2d9d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbb4101b10 0x7fdbb4199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.508+0000 7fdbb359e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbb4106650 0x7fdbb419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.508+0000 7fdbb359e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbb4106650 0x7fdbb419f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48482/0 (socket says 172.21.15.42:48482) 2024-07-26T06:47:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.508+0000 7fdbb359e640 1 -- 172.21.15.42:0/3127211204 learned_addr learned my addr 172.21.15.42:0/3127211204 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.508+0000 7fdbb259c640 1 --2- 172.21.15.42:0/3127211204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbb4105d00 0x7fdbb419a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.508+0000 7fdbb359e640 1 -- 172.21.15.42:0/3127211204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbb4105d00 msgr2=0x7fdbb419a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.508+0000 7fdbb359e640 1 --2- 172.21.15.42:0/3127211204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbb4105d00 0x7fdbb419a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.508+0000 7fdbb359e640 1 -- 172.21.15.42:0/3127211204 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbb4101b10 msgr2=0x7fdbb4199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.508+0000 7fdbb359e640 1 --2- 172.21.15.42:0/3127211204 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbb4101b10 0x7fdbb4199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.508+0000 7fdbb359e640 1 -- 172.21.15.42:0/3127211204 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb419fe10 con 0x7fdbb4106650 2024-07-26T06:47:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.508+0000 7fdbb359e640 1 --2- 172.21.15.42:0/3127211204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbb4106650 0x7fdbb419f6e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00eaa0 tx=0x7fdbac00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.509+0000 7fdbabfff640 1 -- 172.21.15.42:0/3127211204 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac00cdb0 con 0x7fdbb4106650 2024-07-26T06:47:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.509+0000 7fdbabfff640 1 -- 172.21.15.42:0/3127211204 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbac005020 con 0x7fdbb4106650 2024-07-26T06:47:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.509+0000 7fdbabfff640 1 -- 172.21.15.42:0/3127211204 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbac0116d0 con 0x7fdbb4106650 2024-07-26T06:47:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.509+0000 7fdbb91e4640 1 -- 172.21.15.42:0/3127211204 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbb41a0100 con 0x7fdbb4106650 2024-07-26T06:47:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.509+0000 7fdbb91e4640 1 -- 172.21.15.42:0/3127211204 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbb41a6860 con 0x7fdbb4106650 2024-07-26T06:47:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.510+0000 7fdbb91e4640 1 -- 172.21.15.42:0/3127211204 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbb4101f20 con 0x7fdbb4106650 2024-07-26T06:47:12.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.512+0000 7fdbabfff640 1 -- 172.21.15.42:0/3127211204 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fdbac0051d0 con 0x7fdbb4106650 2024-07-26T06:47:12.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.512+0000 7fdbabfff640 1 --2- 172.21.15.42:0/3127211204 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdba00b3c50 0x7fdba00b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:12.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.512+0000 7fdbabfff640 1 -- 172.21.15.42:0/3127211204 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7fdbac0e3ea0 con 0x7fdbb4106650 2024-07-26T06:47:12.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.512+0000 7fdbb2d9d640 1 --2- 172.21.15.42:0/3127211204 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdba00b3c50 0x7fdba00b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:12.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.513+0000 7fdbb2d9d640 1 --2- 172.21.15.42:0/3127211204 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdba00b3c50 0x7fdba00b6160 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdba4031440 tx=0x7fdba4009560 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:12.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.515+0000 7fdbabfff640 1 -- 172.21.15.42:0/3127211204 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdbac016030 con 0x7fdbb4106650 2024-07-26T06:47:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.700+0000 7fdbb91e4640 1 -- 172.21.15.42:0/3127211204 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdbb419afd0 con 0x7fdbb4106650 2024-07-26T06:47:12.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.701+0000 7fdbabfff640 1 -- 172.21.15.42:0/3127211204 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+22170 (secure 0 0 0) 0x7fdbac0ab070 con 0x7fdbb4106650 2024-07-26T06:47:12.702 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:47:12.702 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":78,"fsid":"8b3e2750-4b19-11ef-bcb6-c7b262605968","created":"2024-07-26T06:38:29.306470+0000","modified":"2024-07-26T06:47:06.780610+0000","last_up_change":"2024-07-26T06:46:02.149453+0000","last_in_change":"2024-07-26T06:45:43.027764+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:42:27.302195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:47:00.042262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T06:47:03.674160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bda88e0-09ee-447b-8c04-24f4e5d55d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6803","nonce":3587066379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6805","nonce":3587066379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6809","nonce":3587066379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6807","nonce":3587066379}]},"public_addr":"172.21.15.42:6803/3587066379","cluster_addr":"172.21.15.42:6805/3587066379","heartbeat_back_addr":"172.21.15.42:6809/3587066379","heartbeat_front_addr":"172.21.15.42:6807/3587066379","state":["exists","up"]},{"osd":1,"uuid":"7d747138-8217-4836-a173-cf8ac65e9fe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6801","nonce":360219078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6803","nonce":360219078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6807","nonce":360219078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6805","nonce":360219078}]},"public_addr":"172.21.15.163:6801/360219078","cluster_addr":"172.21.15.163:6803/360219078","heartbeat_back_addr":"172.21.15.163:6807/360219078","heartbeat_front_addr":"172.21.15.163:6805/360219078","state":["exists","up"]},{"osd":2,"uuid":"4b7258c1-3d35-4930-ab10-0b04a2a2c8a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6801","nonce":336113412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6803","nonce":336113412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6807","nonce":336113412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6805","nonce":336113412}]},"public_addr":"172.21.15.164:6801/336113412","cluster_addr":"172.21.15.164:6803/336113412","heartbeat_back_addr":"172.21.15.164:6807/336113412","heartbeat_front_addr":"172.21.15.164:6805/336113412","state":["exists","up"]},{"osd":3,"uuid":"96608387-0de0-4d90-a952-d6eba61f7fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6811","nonce":2819018064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6813","nonce":2819018064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6817","nonce":2819018064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6815","nonce":2819018064}]},"public_addr":"172.21.15.42:6811/2819018064","cluster_addr":"172.21.15.42:6813/2819018064","heartbeat_back_addr":"172.21.15.42:6817/2819018064","heartbeat_front_addr":"172.21.15.42:6815/2819018064","state":["exists","up"]},{"osd":4,"uuid":"fabf6b02-fe34-4560-85b5-01bf37fc9be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6809","nonce":431934374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6811","nonce":431934374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6815","nonce":431934374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6813","nonce":431934374}]},"public_addr":"172.21.15.163:6809/431934374","cluster_addr":"172.21.15.163:6811/431934374","heartbeat_back_addr":"172.21.15.163:6815/431934374","heartbeat_front_addr":"172.21.15.163:6813/431934374","state":["exists","up"]},{"osd":5,"uuid":"e226fb4f-15b2-40df-af34-900040ba93ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6809","nonce":261549362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6811","nonce":261549362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6815","nonce":261549362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6813","nonce":261549362}]},"public_addr":"172.21.15.164:6809/261549362","cluster_addr":"172.21.15.164:6811/261549362","heartbeat_back_addr":"172.21.15.164:6815/261549362","heartbeat_front_addr":"172.21.15.164:6813/261549362","state":["exists","up"]},{"osd":6,"uuid":"beb8ec5d-7ea5-4e4e-a46c-a84f7768cec0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6819","nonce":2160811300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6821","nonce":2160811300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6825","nonce":2160811300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6823","nonce":2160811300}]},"public_addr":"172.21.15.42:6819/2160811300","cluster_addr":"172.21.15.42:6821/2160811300","heartbeat_back_addr":"172.21.15.42:6825/2160811300","heartbeat_front_addr":"172.21.15.42:6823/2160811300","state":["exists","up"]},{"osd":7,"uuid":"f699004c-f7c2-4f1c-ae4b-7178f3b06e2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6817","nonce":812214111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6819","nonce":812214111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6823","nonce":812214111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6821","nonce":812214111}]},"public_addr":"172.21.15.163:6817/812214111","cluster_addr":"172.21.15.163:6819/812214111","heartbeat_back_addr":"172.21.15.163:6823/812214111","heartbeat_front_addr":"172.21.15.163:6821/812214111","state":["exists","up"]},{"osd":8,"uuid":"3476163e-efea-4652-a8d4-b7c4a908f984","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6817","nonce":962845839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6819","nonce":962845839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6823","nonce":962845839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6821","nonce":962845839}]},"public_addr":"172.21.15.164:6817/962845839","cluster_addr":"172.21.15.164:6819/962845839","heartbeat_back_addr":"172.21.15.164:6823/962845839","heartbeat_front_addr":"172.21.15.164:6821/962845839","state":["exists","up"]},{"osd":9,"uuid":"6f7ceac6-fb61-4c6f-9da8-851b937c4586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6827","nonce":994024418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6829","nonce":994024418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6833","nonce":994024418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6831","nonce":994024418}]},"public_addr":"172.21.15.42:6827/994024418","cluster_addr":"172.21.15.42:6829/994024418","heartbeat_back_addr":"172.21.15.42:6833/994024418","heartbeat_front_addr":"172.21.15.42:6831/994024418","state":["exists","up"]},{"osd":10,"uuid":"553132c2-a6a1-410c-a7d8-ecdf9a67dafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6825","nonce":3268512227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6827","nonce":3268512227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6831","nonce":3268512227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6829","nonce":3268512227}]},"public_addr":"172.21.15.163:6825/3268512227","cluster_addr":"172.21.15.163:6827/3268512227","heartbeat_back_addr":"172.21.15.163:6831/3268512227","heartbeat_front_addr":"172.21.15.163:6829/3268512227","state":["exists","up"]},{"osd":11,"uuid":"09abc8e9-7884-400f-ab6d-8e5aa7bf31ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6825","nonce":1861076336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6827","nonce":1861076336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6831","nonce":1861076336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6829","nonce":1861076336}]},"public_addr":"172.21.15.164:6825/1861076336","cluster_addr":"172.21.15.164:6827/1861076336","heartbeat_back_addr":"172.21.15.164:6831/1861076336","heartbeat_front_addr":"172.21.15.164:6829/1861076336","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:41:38.761674+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:00.621745+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:23.248620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:47.232764+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:10.379922+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:34.502519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:58.694642+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:22.350247+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:46.988007+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:11.390474+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:36.150054+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:46:00.474470+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.42:6801/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6800/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1842521399":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/834172950":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1730109419":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6801/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:6800/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2687772889":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/3371433307":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2861719816":"2024-07-27T06:38:56.887264+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:47:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.703+0000 7fdbb91e4640 1 -- 172.21.15.42:0/3127211204 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdba00b3c50 msgr2=0x7fdba00b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.703+0000 7fdbb91e4640 1 --2- 172.21.15.42:0/3127211204 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdba00b3c50 0x7fdba00b6160 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdba4031440 tx=0x7fdba4009560 comp rx=0 tx=0).stop 2024-07-26T06:47:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.703+0000 7fdbb91e4640 1 -- 172.21.15.42:0/3127211204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbb4106650 msgr2=0x7fdbb419f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.703+0000 7fdbb91e4640 1 --2- 172.21.15.42:0/3127211204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbb4106650 0x7fdbb419f6e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00eaa0 tx=0x7fdbac00ef60 comp rx=0 tx=0).stop 2024-07-26T06:47:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.703+0000 7fdbb91e4640 1 -- 172.21.15.42:0/3127211204 shutdown_connections 2024-07-26T06:47:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.703+0000 7fdbb91e4640 1 --2- 172.21.15.42:0/3127211204 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdba00b3c50 0x7fdba00b6160 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.703+0000 7fdbb91e4640 1 --2- 172.21.15.42:0/3127211204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbb4106650 0x7fdbb419f6e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.703+0000 7fdbb91e4640 1 --2- 172.21.15.42:0/3127211204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbb4105d00 0x7fdbb419a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.703+0000 7fdbb91e4640 1 --2- 172.21.15.42:0/3127211204 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdbb4101b10 0x7fdbb4199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.703+0000 7fdbb91e4640 1 -- 172.21.15.42:0/3127211204 >> 172.21.15.42:0/3127211204 conn(0x7fdbb4079870 msgr2=0x7fdbb4107b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.704+0000 7fdbb91e4640 1 -- 172.21.15.42:0/3127211204 shutdown_connections 2024-07-26T06:47:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:12.704+0000 7fdbb91e4640 1 -- 172.21.15.42:0/3127211204 wait complete. 2024-07-26T06:47:13.286 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:13 smithi042 ceph-mon[32775]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s 2024-07-26T06:47:13.286 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:13 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3127211204' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:47:13.316 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-07-26T06:47:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:13 smithi164 ceph-mon[36970]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s 2024-07-26T06:47:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:13 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3127211204' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:47:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:13 smithi163 ceph-mon[36873]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s 2024-07-26T06:47:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:13 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3127211204' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:47:13.581 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.094+0000 7f995eda0640 1 -- 172.21.15.42:0/581602957 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99581022b0 msgr2=0x7f99581026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.094+0000 7f995eda0640 1 --2- 172.21.15.42:0/581602957 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99581022b0 0x7f99581026c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9954009900 tx=0x7f995402a410 comp rx=0 tx=0).stop 2024-07-26T06:47:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.094+0000 7f995eda0640 1 -- 172.21.15.42:0/581602957 shutdown_connections 2024-07-26T06:47:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.094+0000 7f995eda0640 1 --2- 172.21.15.42:0/581602957 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9958108540 0x7f995810ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.094+0000 7f995eda0640 1 --2- 172.21.15.42:0/581602957 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9958107bf0 0x7f9958108000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.094+0000 7f995eda0640 1 --2- 172.21.15.42:0/581602957 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99581022b0 0x7f99581026c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.094+0000 7f995eda0640 1 -- 172.21.15.42:0/581602957 >> 172.21.15.42:0/581602957 conn(0x7f99580fdd00 msgr2=0x7f9958100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.095+0000 7f995eda0640 1 -- 172.21.15.42:0/581602957 shutdown_connections 2024-07-26T06:47:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.095+0000 7f995eda0640 1 -- 172.21.15.42:0/581602957 wait complete. 2024-07-26T06:47:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.095+0000 7f995eda0640 1 Processor -- start 2024-07-26T06:47:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995eda0640 1 -- start start 2024-07-26T06:47:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995eda0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99581022b0 0x7f995819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995eda0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9958107bf0 0x7f995819b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995eda0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9958108540 0x7f995819b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995eda0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f99581122c0 con 0x7f9958108540 2024-07-26T06:47:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995eda0640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9958112140 con 0x7f9958107bf0 2024-07-26T06:47:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995eda0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9958112440 con 0x7f99581022b0 2024-07-26T06:47:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995e59f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9958108540 0x7f995819b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995e59f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9958108540 0x7f995819b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48492/0 (socket says 172.21.15.42:48492) 2024-07-26T06:47:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995e59f640 1 -- 172.21.15.42:0/4060247049 learned_addr learned my addr 172.21.15.42:0/4060247049 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995dd9e640 1 --2- 172.21.15.42:0/4060247049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99581022b0 0x7f995819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:15.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995d59d640 1 --2- 172.21.15.42:0/4060247049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9958107bf0 0x7f995819b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:15.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995e59f640 1 -- 172.21.15.42:0/4060247049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99581022b0 msgr2=0x7f995819ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:15.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995e59f640 1 --2- 172.21.15.42:0/4060247049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99581022b0 0x7f995819ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:15.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995e59f640 1 -- 172.21.15.42:0/4060247049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9958107bf0 msgr2=0x7f995819b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:15.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995e59f640 1 --2- 172.21.15.42:0/4060247049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9958107bf0 0x7f995819b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:15.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995e59f640 1 -- 172.21.15.42:0/4060247049 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99581a5ef0 con 0x7f9958108540 2024-07-26T06:47:15.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.096+0000 7f995d59d640 1 --2- 172.21.15.42:0/4060247049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9958107bf0 0x7f995819b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:47:15.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.097+0000 7f995e59f640 1 --2- 172.21.15.42:0/4060247049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9958108540 0x7f995819b6b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f995000b970 tx=0x7f995000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:15.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.097+0000 7f994effd640 1 -- 172.21.15.42:0/4060247049 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9950019070 con 0x7f9958108540 2024-07-26T06:47:15.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.097+0000 7f995eda0640 1 -- 172.21.15.42:0/4060247049 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99581a61e0 con 0x7f9958108540 2024-07-26T06:47:15.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.097+0000 7f994effd640 1 -- 172.21.15.42:0/4060247049 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9950014440 con 0x7f9958108540 2024-07-26T06:47:15.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.097+0000 7f994effd640 1 -- 172.21.15.42:0/4060247049 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99500075d0 con 0x7f9958108540 2024-07-26T06:47:15.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.097+0000 7f995eda0640 1 -- 172.21.15.42:0/4060247049 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99581a6750 con 0x7f9958108540 2024-07-26T06:47:15.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.098+0000 7f995eda0640 1 -- 172.21.15.42:0/4060247049 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9958102740 con 0x7f9958108540 2024-07-26T06:47:15.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.099+0000 7f994effd640 1 -- 172.21.15.42:0/4060247049 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f995001c070 con 0x7f9958108540 2024-07-26T06:47:15.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.100+0000 7f994effd640 1 --2- 172.21.15.42:0/4060247049 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f99440b3bd0 0x7f99440b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:15.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.100+0000 7f995dd9e640 1 --2- 172.21.15.42:0/4060247049 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f99440b3bd0 0x7f99440b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:15.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.100+0000 7f994effd640 1 -- 172.21.15.42:0/4060247049 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7f99500e35c0 con 0x7f9958108540 2024-07-26T06:47:15.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.101+0000 7f995dd9e640 1 --2- 172.21.15.42:0/4060247049 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f99440b3bd0 0x7f99440b60e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9954000c00 tx=0x7f9954031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:15.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.103+0000 7f994effd640 1 -- 172.21.15.42:0/4060247049 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f99500aa670 con 0x7f9958108540 2024-07-26T06:47:15.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:15 smithi042 ceph-mon[32775]: pgmap v304: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T06:47:15.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:15.328+0000 7f995eda0640 1 -- 172.21.15.42:0/4060247049 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f995802e0c0 con 0x7f9958108540 2024-07-26T06:47:15.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:15 smithi164 ceph-mon[36970]: pgmap v304: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T06:47:15.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:15 smithi163 ceph-mon[36873]: pgmap v304: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T06:47:16.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:16.047+0000 7f994effd640 1 -- 172.21.15.42:0/4060247049 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f99500aebc0 con 0x7f9958108540 2024-07-26T06:47:16.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:16.051+0000 7f995eda0640 1 -- 172.21.15.42:0/4060247049 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f99440b3bd0 msgr2=0x7f99440b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:16.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:16.051+0000 7f995eda0640 1 --2- 172.21.15.42:0/4060247049 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f99440b3bd0 0x7f99440b60e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9954000c00 tx=0x7f9954031400 comp rx=0 tx=0).stop 2024-07-26T06:47:16.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:16.051+0000 7f995eda0640 1 -- 172.21.15.42:0/4060247049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9958108540 msgr2=0x7f995819b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:16.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:16.051+0000 7f995eda0640 1 --2- 172.21.15.42:0/4060247049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9958108540 0x7f995819b6b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f995000b970 tx=0x7f995000be30 comp rx=0 tx=0).stop 2024-07-26T06:47:16.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:16.051+0000 7f995eda0640 1 -- 172.21.15.42:0/4060247049 shutdown_connections 2024-07-26T06:47:16.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:16.051+0000 7f995eda0640 1 --2- 172.21.15.42:0/4060247049 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f99440b3bd0 0x7f99440b60e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:16.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:16.051+0000 7f995eda0640 1 --2- 172.21.15.42:0/4060247049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9958108540 0x7f995819b6b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:16.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:16.051+0000 7f995eda0640 1 --2- 172.21.15.42:0/4060247049 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9958107bf0 0x7f995819b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:16.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:16.051+0000 7f995eda0640 1 --2- 172.21.15.42:0/4060247049 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f99581022b0 0x7f995819ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:16.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:16.051+0000 7f995eda0640 1 -- 172.21.15.42:0/4060247049 >> 172.21.15.42:0/4060247049 conn(0x7f99580fdd00 msgr2=0x7f9958100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:16.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:16.051+0000 7f995eda0640 1 -- 172.21.15.42:0/4060247049 shutdown_connections 2024-07-26T06:47:16.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:16.051+0000 7f995eda0640 1 -- 172.21.15.42:0/4060247049 wait complete. 2024-07-26T06:47:16.379 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:16 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/4060247049' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T06:47:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:16 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/4060247049' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T06:47:16.514 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:16 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/4060247049' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T06:47:16.666 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-07-26T06:47:16.666 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-07-26T06:47:16.941 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:17.316 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:17 smithi042 ceph-mon[32775]: pgmap v305: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-07-26T06:47:17.316 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:17 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/4060247049' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T06:47:17.316 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:17 smithi042 ceph-mon[32775]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T06:47:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:17 smithi164 ceph-mon[36970]: pgmap v305: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-07-26T06:47:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:17 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/4060247049' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T06:47:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:17 smithi164 ceph-mon[36970]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T06:47:17.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:17 smithi163 ceph-mon[36873]: pgmap v305: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-07-26T06:47:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:17 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/4060247049' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T06:47:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:17 smithi163 ceph-mon[36873]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T06:47:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.620+0000 7f4060481640 1 -- 172.21.15.42:0/2737018827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4058075ab0 msgr2=0x7f405810f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.620+0000 7f4060481640 1 --2- 172.21.15.42:0/2737018827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4058075ab0 0x7f405810f690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f404c009900 tx=0x7f404c02a410 comp rx=0 tx=0).stop 2024-07-26T06:47:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.620+0000 7f4060481640 1 -- 172.21.15.42:0/2737018827 shutdown_connections 2024-07-26T06:47:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.620+0000 7f4060481640 1 --2- 172.21.15.42:0/2737018827 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f405810fbd0 0x7f4058111ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.620+0000 7f4060481640 1 --2- 172.21.15.42:0/2737018827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4058075ab0 0x7f405810f690 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.620+0000 7f4060481640 1 --2- 172.21.15.42:0/2737018827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4058075160 0x7f4058075570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:18.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.620+0000 7f4060481640 1 -- 172.21.15.42:0/2737018827 >> 172.21.15.42:0/2737018827 conn(0x7f405807b340 msgr2=0x7f405807b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:18.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.620+0000 7f4060481640 1 -- 172.21.15.42:0/2737018827 shutdown_connections 2024-07-26T06:47:18.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.620+0000 7f4060481640 1 -- 172.21.15.42:0/2737018827 wait complete. 2024-07-26T06:47:18.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.621+0000 7f4060481640 1 Processor -- start 2024-07-26T06:47:18.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.621+0000 7f4060481640 1 -- start start 2024-07-26T06:47:18.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.621+0000 7f4060481640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4058075160 0x7f405819c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:18.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.621+0000 7f4060481640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4058075ab0 0x7f405819c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:18.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.621+0000 7f4060481640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405810fbd0 0x7f40581a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:18.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.621+0000 7f4060481640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4058115690 con 0x7f405810fbd0 2024-07-26T06:47:18.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.621+0000 7f4060481640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4058115510 con 0x7f4058075160 2024-07-26T06:47:18.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.621+0000 7f4060481640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4058115810 con 0x7f4058075ab0 2024-07-26T06:47:18.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.621+0000 7f405ea1e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405810fbd0 0x7f40581a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:18.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.621+0000 7f405ea1e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405810fbd0 0x7f40581a1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33358/0 (socket says 172.21.15.42:33358) 2024-07-26T06:47:18.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.621+0000 7f405ea1e640 1 -- 172.21.15.42:0/460371864 learned_addr learned my addr 172.21.15.42:0/460371864 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:18.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.621+0000 7f405e21d640 1 --2- 172.21.15.42:0/460371864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4058075160 0x7f405819c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:18.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.622+0000 7f405da1c640 1 --2- 172.21.15.42:0/460371864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4058075ab0 0x7f405819c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:18.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.622+0000 7f405ea1e640 1 -- 172.21.15.42:0/460371864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4058075ab0 msgr2=0x7f405819c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:18.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.622+0000 7f405ea1e640 1 --2- 172.21.15.42:0/460371864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4058075ab0 0x7f405819c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:18.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.622+0000 7f405ea1e640 1 -- 172.21.15.42:0/460371864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4058075160 msgr2=0x7f405819c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:18.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.622+0000 7f405ea1e640 1 --2- 172.21.15.42:0/460371864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4058075160 0x7f405819c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:18.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.622+0000 7f405ea1e640 1 -- 172.21.15.42:0/460371864 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40581a2130 con 0x7f405810fbd0 2024-07-26T06:47:18.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.622+0000 7f405e21d640 1 --2- 172.21.15.42:0/460371864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4058075160 0x7f405819c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:47:18.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.622+0000 7f405ea1e640 1 --2- 172.21.15.42:0/460371864 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405810fbd0 0x7f40581a1a00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f404800cad0 tx=0x7f404800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:18.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.622+0000 7f40577fe640 1 -- 172.21.15.42:0/460371864 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404800ae20 con 0x7f405810fbd0 2024-07-26T06:47:18.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.622+0000 7f40577fe640 1 -- 172.21.15.42:0/460371864 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4048005020 con 0x7f405810fbd0 2024-07-26T06:47:18.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.622+0000 7f4060481640 1 -- 172.21.15.42:0/460371864 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40581a2420 con 0x7f405810fbd0 2024-07-26T06:47:18.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.622+0000 7f40577fe640 1 -- 172.21.15.42:0/460371864 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404800f6b0 con 0x7f405810fbd0 2024-07-26T06:47:18.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.622+0000 7f4060481640 1 -- 172.21.15.42:0/460371864 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40581a8b80 con 0x7f405810fbd0 2024-07-26T06:47:18.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.623+0000 7f4060481640 1 -- 172.21.15.42:0/460371864 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4058076620 con 0x7f405810fbd0 2024-07-26T06:47:18.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.625+0000 7f40577fe640 1 -- 172.21.15.42:0/460371864 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f40480051d0 con 0x7f405810fbd0 2024-07-26T06:47:18.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.627+0000 7f40577fe640 1 --2- 172.21.15.42:0/460371864 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f40400b3bd0 0x7f40400b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:18.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.627+0000 7f405e21d640 1 --2- 172.21.15.42:0/460371864 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f40400b3bd0 0x7f40400b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:18.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.627+0000 7f40577fe640 1 -- 172.21.15.42:0/460371864 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7f40480e3ef0 con 0x7f405810fbd0 2024-07-26T06:47:18.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.627+0000 7f405e21d640 1 --2- 172.21.15.42:0/460371864 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f40400b3bd0 0x7f40400b60e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f40500058c0 tx=0x7f4050005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:18.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.629+0000 7f40577fe640 1 -- 172.21.15.42:0/460371864 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4048014030 con 0x7f405810fbd0 2024-07-26T06:47:18.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:18.852+0000 7f4060481640 1 -- 172.21.15.42:0/460371864 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f40581a8e20 con 0x7f405810fbd0 2024-07-26T06:47:19.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:19.056+0000 7f40577fe640 1 -- 172.21.15.42:0/460371864 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f40480ab690 con 0x7f405810fbd0 2024-07-26T06:47:19.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:19.059+0000 7f4060481640 1 -- 172.21.15.42:0/460371864 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f40400b3bd0 msgr2=0x7f40400b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:19.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:19.059+0000 7f4060481640 1 --2- 172.21.15.42:0/460371864 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f40400b3bd0 0x7f40400b60e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f40500058c0 tx=0x7f4050005850 comp rx=0 tx=0).stop 2024-07-26T06:47:19.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:19.059+0000 7f4060481640 1 -- 172.21.15.42:0/460371864 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405810fbd0 msgr2=0x7f40581a1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:19.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:19.059+0000 7f4060481640 1 --2- 172.21.15.42:0/460371864 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405810fbd0 0x7f40581a1a00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f404800cad0 tx=0x7f404800cf90 comp rx=0 tx=0).stop 2024-07-26T06:47:19.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:19.059+0000 7f4060481640 1 -- 172.21.15.42:0/460371864 shutdown_connections 2024-07-26T06:47:19.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:19.059+0000 7f4060481640 1 --2- 172.21.15.42:0/460371864 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f40400b3bd0 0x7f40400b60e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:19.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:19.059+0000 7f4060481640 1 --2- 172.21.15.42:0/460371864 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405810fbd0 0x7f40581a1a00 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:19.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:19.059+0000 7f4060481640 1 --2- 172.21.15.42:0/460371864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4058075ab0 0x7f405819c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:19.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:19.059+0000 7f4060481640 1 --2- 172.21.15.42:0/460371864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4058075160 0x7f405819c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:19.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:19.059+0000 7f4060481640 1 -- 172.21.15.42:0/460371864 >> 172.21.15.42:0/460371864 conn(0x7f405807b340 msgr2=0x7f4058108680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:19.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:19.059+0000 7f4060481640 1 -- 172.21.15.42:0/460371864 shutdown_connections 2024-07-26T06:47:19.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:19.059+0000 7f4060481640 1 -- 172.21.15.42:0/460371864 wait complete. 2024-07-26T06:47:19.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:19 smithi042 ceph-mon[32775]: pgmap v306: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-07-26T06:47:19.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:19 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/460371864' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T06:47:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:19 smithi164 ceph-mon[36970]: pgmap v306: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-07-26T06:47:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:19 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/460371864' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T06:47:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:19 smithi163 ceph-mon[36873]: pgmap v306: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-07-26T06:47:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:19 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/460371864' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T06:47:19.736 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-07-26T06:47:20.003 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:20.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:20 smithi042 ceph-mon[32775]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T06:47:20.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:20 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/460371864' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-26T06:47:20.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:20 smithi042 ceph-mon[32775]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T06:47:20.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:20 smithi042 ceph-mon[32775]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T06:47:20.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:20 smithi042 ceph-mon[32775]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-07-26T06:47:20.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:20 smithi042 ceph-mon[32775]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T06:47:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:20 smithi164 ceph-mon[36970]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T06:47:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:20 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/460371864' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-26T06:47:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:20 smithi164 ceph-mon[36970]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T06:47:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:20 smithi164 ceph-mon[36970]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T06:47:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:20 smithi164 ceph-mon[36970]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-07-26T06:47:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:20 smithi164 ceph-mon[36970]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T06:47:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:20 smithi163 ceph-mon[36873]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T06:47:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:20 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/460371864' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-26T06:47:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:20 smithi163 ceph-mon[36873]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T06:47:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:20 smithi163 ceph-mon[36873]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T06:47:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:20 smithi163 ceph-mon[36873]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-07-26T06:47:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:20 smithi163 ceph-mon[36873]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-07-26T06:47:21.069 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:47:20 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 2024-07-26T06:47:20.778+0000 7fe5373cb640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T06:47:21.264 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:21 smithi042 ceph-mon[32775]: pgmap v307: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-07-26T06:47:21.264 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:21 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] up:active 2024-07-26T06:47:21.264 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:21 smithi042 ceph-mon[32775]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T06:47:21.264 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:21 smithi042 ceph-mon[32775]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-07-26T06:47:21.264 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:21 smithi042 ceph-mon[32775]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-07-26T06:47:21.264 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:21 smithi042 ceph-mon[32775]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-07-26T06:47:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:21 smithi164 ceph-mon[36970]: pgmap v307: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-07-26T06:47:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:21 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] up:active 2024-07-26T06:47:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:21 smithi164 ceph-mon[36970]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T06:47:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:21 smithi164 ceph-mon[36970]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-07-26T06:47:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:21 smithi164 ceph-mon[36970]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-07-26T06:47:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:21 smithi164 ceph-mon[36970]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-07-26T06:47:21.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.473+0000 7fe57bef9640 1 -- 172.21.15.42:0/1582456474 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe574107dd0 msgr2=0x7fe5741081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:21.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.473+0000 7fe57bef9640 1 --2- 172.21.15.42:0/1582456474 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe574107dd0 0x7fe5741081e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe568009900 tx=0x7fe56802a410 comp rx=0 tx=0).stop 2024-07-26T06:47:21.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.473+0000 7fe57bef9640 1 -- 172.21.15.42:0/1582456474 shutdown_connections 2024-07-26T06:47:21.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.473+0000 7fe57bef9640 1 --2- 172.21.15.42:0/1582456474 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe574108720 0x7fe57410ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:21.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.473+0000 7fe57bef9640 1 --2- 172.21.15.42:0/1582456474 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe574107dd0 0x7fe5741081e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:21.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.473+0000 7fe57bef9640 1 --2- 172.21.15.42:0/1582456474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe574102490 0x7fe5741028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:21.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.473+0000 7fe57bef9640 1 -- 172.21.15.42:0/1582456474 >> 172.21.15.42:0/1582456474 conn(0x7fe574079160 msgr2=0x7fe574079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:21.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.473+0000 7fe57bef9640 1 -- 172.21.15.42:0/1582456474 shutdown_connections 2024-07-26T06:47:21.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.473+0000 7fe57bef9640 1 -- 172.21.15.42:0/1582456474 wait complete. 2024-07-26T06:47:21.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.474+0000 7fe57bef9640 1 Processor -- start 2024-07-26T06:47:21.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.474+0000 7fe57bef9640 1 -- start start 2024-07-26T06:47:21.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.474+0000 7fe57bef9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe574102490 0x7fe574199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:21.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.474+0000 7fe57bef9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe574107dd0 0x7fe57419a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:21.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.474+0000 7fe57bef9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe574108720 0x7fe57419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:21.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.474+0000 7fe57bef9640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe5741134b0 con 0x7fe574108720 2024-07-26T06:47:21.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.474+0000 7fe57bef9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe574113330 con 0x7fe574107dd0 2024-07-26T06:47:21.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.474+0000 7fe57bef9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe574113630 con 0x7fe574102490 2024-07-26T06:47:21.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.475+0000 7fe57a496640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe574108720 0x7fe57419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:21.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.475+0000 7fe57a496640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe574108720 0x7fe57419f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33372/0 (socket says 172.21.15.42:33372) 2024-07-26T06:47:21.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.475+0000 7fe57a496640 1 -- 172.21.15.42:0/827158258 learned_addr learned my addr 172.21.15.42:0/827158258 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:21.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.475+0000 7fe579494640 1 --2- 172.21.15.42:0/827158258 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe574107dd0 0x7fe57419a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:21.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.475+0000 7fe579c95640 1 --2- 172.21.15.42:0/827158258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe574102490 0x7fe574199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:21.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.475+0000 7fe57a496640 1 -- 172.21.15.42:0/827158258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe574102490 msgr2=0x7fe574199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:21.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.475+0000 7fe57a496640 1 --2- 172.21.15.42:0/827158258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe574102490 0x7fe574199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:21.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.475+0000 7fe57a496640 1 -- 172.21.15.42:0/827158258 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe574107dd0 msgr2=0x7fe57419a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:21.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.475+0000 7fe57a496640 1 --2- 172.21.15.42:0/827158258 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe574107dd0 0x7fe57419a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:21.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.475+0000 7fe57a496640 1 -- 172.21.15.42:0/827158258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe57419fed0 con 0x7fe574108720 2024-07-26T06:47:21.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.475+0000 7fe579c95640 1 --2- 172.21.15.42:0/827158258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe574102490 0x7fe574199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:47:21.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.475+0000 7fe57a496640 1 --2- 172.21.15.42:0/827158258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe574108720 0x7fe57419f7a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe564009fd0 tx=0x7fe56400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:21.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.475+0000 7fe572ffd640 1 -- 172.21.15.42:0/827158258 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe564017070 con 0x7fe574108720 2024-07-26T06:47:21.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.475+0000 7fe57bef9640 1 -- 172.21.15.42:0/827158258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5741a01c0 con 0x7fe574108720 2024-07-26T06:47:21.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.475+0000 7fe572ffd640 1 -- 172.21.15.42:0/827158258 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe564003e20 con 0x7fe574108720 2024-07-26T06:47:21.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.475+0000 7fe572ffd640 1 -- 172.21.15.42:0/827158258 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe564010460 con 0x7fe574108720 2024-07-26T06:47:21.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.476+0000 7fe57bef9640 1 -- 172.21.15.42:0/827158258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5741a6920 con 0x7fe574108720 2024-07-26T06:47:21.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.477+0000 7fe57bef9640 1 -- 172.21.15.42:0/827158258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5741028a0 con 0x7fe574108720 2024-07-26T06:47:21.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.478+0000 7fe572ffd640 1 -- 172.21.15.42:0/827158258 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7fe564003970 con 0x7fe574108720 2024-07-26T06:47:21.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.479+0000 7fe572ffd640 1 --2- 172.21.15.42:0/827158258 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe55c0b3bd0 0x7fe55c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:21.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.479+0000 7fe579c95640 1 --2- 172.21.15.42:0/827158258 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe55c0b3bd0 0x7fe55c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:21.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.479+0000 7fe572ffd640 1 -- 172.21.15.42:0/827158258 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7979+0+0 (secure 0 0 0) 0x7fe5640e3810 con 0x7fe574108720 2024-07-26T06:47:21.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.480+0000 7fe579c95640 1 --2- 172.21.15.42:0/827158258 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe55c0b3bd0 0x7fe55c0b60e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0058c0 tx=0x7fe56c005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:21.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.482+0000 7fe572ffd640 1 -- 172.21.15.42:0/827158258 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe5640aaf10 con 0x7fe574108720 2024-07-26T06:47:21.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:21 smithi163 ceph-mon[36873]: pgmap v307: 97 pgs: 97 active+clean; 579 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-07-26T06:47:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:21 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] up:active 2024-07-26T06:47:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:21 smithi163 ceph-mon[36873]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T06:47:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:21 smithi163 ceph-mon[36873]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-07-26T06:47:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:21 smithi163 ceph-mon[36873]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-07-26T06:47:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:21 smithi163 ceph-mon[36873]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-07-26T06:47:21.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:21.709+0000 7fe57bef9640 1 -- 172.21.15.42:0/827158258 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fe57419b090 con 0x7fe574108720 2024-07-26T06:47:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:22.085+0000 7fe572ffd640 1 -- 172.21.15.42:0/827158258 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v24) ==== 121+0+0 (secure 0 0 0) 0x7fe57419b090 con 0x7fe574108720 2024-07-26T06:47:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:22.089+0000 7fe57bef9640 1 -- 172.21.15.42:0/827158258 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe55c0b3bd0 msgr2=0x7fe55c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:22.089+0000 7fe57bef9640 1 --2- 172.21.15.42:0/827158258 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe55c0b3bd0 0x7fe55c0b60e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0058c0 tx=0x7fe56c005850 comp rx=0 tx=0).stop 2024-07-26T06:47:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:22.089+0000 7fe57bef9640 1 -- 172.21.15.42:0/827158258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe574108720 msgr2=0x7fe57419f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:22.089+0000 7fe57bef9640 1 --2- 172.21.15.42:0/827158258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe574108720 0x7fe57419f7a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe564009fd0 tx=0x7fe56400b260 comp rx=0 tx=0).stop 2024-07-26T06:47:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:22.089+0000 7fe57bef9640 1 -- 172.21.15.42:0/827158258 shutdown_connections 2024-07-26T06:47:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:22.089+0000 7fe57bef9640 1 --2- 172.21.15.42:0/827158258 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe55c0b3bd0 0x7fe55c0b60e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:22.089+0000 7fe57bef9640 1 --2- 172.21.15.42:0/827158258 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe574108720 0x7fe57419f7a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:22.089+0000 7fe57bef9640 1 --2- 172.21.15.42:0/827158258 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe574107dd0 0x7fe57419a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:22.089+0000 7fe57bef9640 1 --2- 172.21.15.42:0/827158258 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe574102490 0x7fe574199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:22.089+0000 7fe57bef9640 1 -- 172.21.15.42:0/827158258 >> 172.21.15.42:0/827158258 conn(0x7fe574079160 msgr2=0x7fe574103f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:22.089+0000 7fe57bef9640 1 -- 172.21.15.42:0/827158258 shutdown_connections 2024-07-26T06:47:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:22.090+0000 7fe57bef9640 1 -- 172.21.15.42:0/827158258 wait complete. 2024-07-26T06:47:22.368 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:22 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] up:active 2024-07-26T06:47:22.368 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:22 smithi042 ceph-mon[32775]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-26T06:47:22.368 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:22 smithi042 ceph-mon[32775]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-07-26T06:47:22.368 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:22 smithi042 ceph-mon[32775]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-07-26T06:47:22.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:22 smithi042 ceph-mon[32775]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-07-26T06:47:22.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-07-26T06:47:22.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-07-26T06:47:22.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-07-26T06:47:22.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-07-26T06:47:22.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-07-26T06:47:22.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-07-26T06:47:22.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-07-26T06:47:22.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-07-26T06:47:22.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]: dispatch 2024-07-26T06:47:22.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-07-26T06:47:22.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:22 smithi042 ceph-mon[32775]: pgmap v308: 97 pgs: 97 active+clean; 580 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-07-26T06:47:22.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:22 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/827158258' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T06:47:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:22 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] up:active 2024-07-26T06:47:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:22 smithi164 ceph-mon[36970]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-26T06:47:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:22 smithi164 ceph-mon[36970]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-07-26T06:47:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:22 smithi164 ceph-mon[36970]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-07-26T06:47:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:22 smithi164 ceph-mon[36970]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-07-26T06:47:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-07-26T06:47:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-07-26T06:47:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-07-26T06:47:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-07-26T06:47:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-07-26T06:47:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-07-26T06:47:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-07-26T06:47:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-07-26T06:47:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]: dispatch 2024-07-26T06:47:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-07-26T06:47:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:22 smithi164 ceph-mon[36970]: pgmap v308: 97 pgs: 97 active+clean; 580 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-07-26T06:47:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:22 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/827158258' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T06:47:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:22 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] up:active 2024-07-26T06:47:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:22 smithi163 ceph-mon[36873]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-26T06:47:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:22 smithi163 ceph-mon[36873]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-07-26T06:47:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:22 smithi163 ceph-mon[36873]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-07-26T06:47:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:22 smithi163 ceph-mon[36873]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-07-26T06:47:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-07-26T06:47:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-07-26T06:47:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-07-26T06:47:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-07-26T06:47:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-07-26T06:47:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-07-26T06:47:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-07-26T06:47:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-07-26T06:47:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]: dispatch 2024-07-26T06:47:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-07-26T06:47:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:22 smithi163 ceph-mon[36873]: pgmap v308: 97 pgs: 97 active+clean; 580 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 2 op/s 2024-07-26T06:47:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:22 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/827158258' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T06:47:22.888 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-07-26T06:47:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/827158258' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T06:47:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-07-26T06:47:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-07-26T06:47:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-07-26T06:47:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-07-26T06:47:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-07-26T06:47:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-07-26T06:47:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-07-26T06:47:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-07-26T06:47:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]': finished 2024-07-26T06:47:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-07-26T06:47:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] up:active 2024-07-26T06:47:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-26T06:47:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T06:47:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: Cluster is now healthy 2024-07-26T06:47:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-07-26T06:47:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T06:47:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:23 smithi042 ceph-mon[32775]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-26T06:47:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/827158258' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T06:47:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-07-26T06:47:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-07-26T06:47:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-07-26T06:47:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-07-26T06:47:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-07-26T06:47:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-07-26T06:47:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-07-26T06:47:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-07-26T06:47:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]': finished 2024-07-26T06:47:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-07-26T06:47:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] up:active 2024-07-26T06:47:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-26T06:47:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T06:47:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: Cluster is now healthy 2024-07-26T06:47:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-07-26T06:47:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T06:47:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:23 smithi164 ceph-mon[36970]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-26T06:47:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/827158258' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T06:47:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-07-26T06:47:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-07-26T06:47:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-07-26T06:47:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-07-26T06:47:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-07-26T06:47:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-07-26T06:47:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-07-26T06:47:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-07-26T06:47:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 8]}]': finished 2024-07-26T06:47:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-07-26T06:47:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] up:active 2024-07-26T06:47:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-26T06:47:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T06:47:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: Cluster is now healthy 2024-07-26T06:47:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2024-07-26T06:47:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T06:47:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:23 smithi163 ceph-mon[36873]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-26T06:47:24.360 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:24 smithi042 ceph-mon[32775]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T06:47:24.361 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:24 smithi042 ceph-mon[32775]: pgmap v311: 97 pgs: 97 active+clean; 583 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 8 op/s; 2 B/s, 0 objects/s recovering 2024-07-26T06:47:24.361 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:24 smithi042 ceph-mon[32775]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-07-26T06:47:24.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:24 smithi164 ceph-mon[36970]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T06:47:24.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:24 smithi164 ceph-mon[36970]: pgmap v311: 97 pgs: 97 active+clean; 583 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 8 op/s; 2 B/s, 0 objects/s recovering 2024-07-26T06:47:24.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:24 smithi164 ceph-mon[36970]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-07-26T06:47:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:24 smithi163 ceph-mon[36873]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T06:47:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:24 smithi163 ceph-mon[36873]: pgmap v311: 97 pgs: 97 active+clean; 583 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 8 op/s; 2 B/s, 0 objects/s recovering 2024-07-26T06:47:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:24 smithi163 ceph-mon[36873]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-07-26T06:47:24.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.892+0000 7f125908c640 1 -- 172.21.15.42:0/456299941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1254103310 msgr2=0x7f125410eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:24.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.892+0000 7f125908c640 1 --2- 172.21.15.42:0/456299941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1254103310 0x7f125410eec0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f124c009ca0 tx=0x7f124c02a8c0 comp rx=0 tx=0).stop 2024-07-26T06:47:24.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.892+0000 7f125908c640 1 -- 172.21.15.42:0/456299941 shutdown_connections 2024-07-26T06:47:24.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.892+0000 7f125908c640 1 --2- 172.21.15.42:0/456299941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1254103310 0x7f125410eec0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:24.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.892+0000 7f125908c640 1 --2- 172.21.15.42:0/456299941 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12541029c0 0x7f1254102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:24.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.892+0000 7f125908c640 1 --2- 172.21.15.42:0/456299941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1254107050 0x7f1254107460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:24.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.892+0000 7f125908c640 1 -- 172.21.15.42:0/456299941 >> 172.21.15.42:0/456299941 conn(0x7f1254079860 msgr2=0x7f1254079c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:24.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.892+0000 7f125908c640 1 -- 172.21.15.42:0/456299941 shutdown_connections 2024-07-26T06:47:24.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.892+0000 7f125908c640 1 -- 172.21.15.42:0/456299941 wait complete. 2024-07-26T06:47:24.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.893+0000 7f125908c640 1 Processor -- start 2024-07-26T06:47:24.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.893+0000 7f125908c640 1 -- start start 2024-07-26T06:47:24.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.893+0000 7f125908c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12541029c0 0x7f1254198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:24.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.893+0000 7f125908c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1254103310 0x7f12541993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:24.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.893+0000 7f125908c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1254107050 0x7f125419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:24.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.893+0000 7f125908c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1254112400 con 0x7f1254103310 2024-07-26T06:47:24.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.893+0000 7f125908c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1254112280 con 0x7f1254107050 2024-07-26T06:47:24.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.893+0000 7f125908c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1254112580 con 0x7f12541029c0 2024-07-26T06:47:24.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.893+0000 7f125259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1254103310 0x7f12541993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:24.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.893+0000 7f125259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1254103310 0x7f12541993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33384/0 (socket says 172.21.15.42:33384) 2024-07-26T06:47:24.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.893+0000 7f125259c640 1 -- 172.21.15.42:0/914117239 learned_addr learned my addr 172.21.15.42:0/914117239 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:24.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.893+0000 7f1252d9d640 1 --2- 172.21.15.42:0/914117239 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12541029c0 0x7f1254198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:24.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.893+0000 7f125359e640 1 --2- 172.21.15.42:0/914117239 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1254107050 0x7f125419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:24.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.894+0000 7f125259c640 1 -- 172.21.15.42:0/914117239 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12541029c0 msgr2=0x7f1254198e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:24.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.894+0000 7f125259c640 1 --2- 172.21.15.42:0/914117239 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12541029c0 0x7f1254198e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:24.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.894+0000 7f125259c640 1 -- 172.21.15.42:0/914117239 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1254107050 msgr2=0x7f125419e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:24.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.894+0000 7f125259c640 1 --2- 172.21.15.42:0/914117239 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1254107050 0x7f125419e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:24.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.894+0000 7f125259c640 1 -- 172.21.15.42:0/914117239 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125419edc0 con 0x7f1254103310 2024-07-26T06:47:24.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.894+0000 7f125259c640 1 --2- 172.21.15.42:0/914117239 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1254103310 0x7f12541993f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1248005d00 tx=0x7f124800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:24.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.894+0000 7f123bfff640 1 -- 172.21.15.42:0/914117239 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1248017070 con 0x7f1254103310 2024-07-26T06:47:24.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.894+0000 7f125908c640 1 -- 172.21.15.42:0/914117239 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f125419f0b0 con 0x7f1254103310 2024-07-26T06:47:24.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.894+0000 7f123bfff640 1 -- 172.21.15.42:0/914117239 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1248009c30 con 0x7f1254103310 2024-07-26T06:47:24.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.894+0000 7f123bfff640 1 -- 172.21.15.42:0/914117239 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124800f620 con 0x7f1254103310 2024-07-26T06:47:24.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.894+0000 7f125908c640 1 -- 172.21.15.42:0/914117239 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12541a5810 con 0x7f1254103310 2024-07-26T06:47:24.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.895+0000 7f125908c640 1 -- 172.21.15.42:0/914117239 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1254102dd0 con 0x7f1254103310 2024-07-26T06:47:24.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.897+0000 7f123bfff640 1 -- 172.21.15.42:0/914117239 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f124801a070 con 0x7f1254103310 2024-07-26T06:47:24.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.898+0000 7f123bfff640 1 --2- 172.21.15.42:0/914117239 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f123c0b3bd0 0x7f123c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:24.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.898+0000 7f1252d9d640 1 --2- 172.21.15.42:0/914117239 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f123c0b3bd0 0x7f123c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:24.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.898+0000 7f123bfff640 1 -- 172.21.15.42:0/914117239 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f12480e3710 con 0x7f1254103310 2024-07-26T06:47:24.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.899+0000 7f1252d9d640 1 --2- 172.21.15.42:0/914117239 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f123c0b3bd0 0x7f123c0b60e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1244005d90 tx=0x7f1244005d20 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:24.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:24.901+0000 7f123bfff640 1 -- 172.21.15.42:0/914117239 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1248009de0 con 0x7f1254103310 2024-07-26T06:47:25.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:25.123+0000 7f125908c640 1 -- 172.21.15.42:0/914117239 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f1254199f80 con 0x7f1254103310 2024-07-26T06:47:25.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:25 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] up:active 2024-07-26T06:47:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:25 smithi042 ceph-mon[32775]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-26T06:47:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:25 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:47:25.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:25 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] up:active 2024-07-26T06:47:25.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:25 smithi164 ceph-mon[36970]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-26T06:47:25.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:25 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:47:25.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:25 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] up:active 2024-07-26T06:47:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:25 smithi163 ceph-mon[36873]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-26T06:47:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:25 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:47:26.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:26.112+0000 7f123bfff640 1 -- 172.21.15.42:0/914117239 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v28) ==== 115+0+0 (secure 0 0 0) 0x7f12480aad40 con 0x7f1254103310 2024-07-26T06:47:26.114 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:47:25 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 2024-07-26T06:47:25.779+0000 7fe5373cb640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-07-26T06:47:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:26.116+0000 7f125908c640 1 -- 172.21.15.42:0/914117239 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f123c0b3bd0 msgr2=0x7f123c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:26.116+0000 7f125908c640 1 --2- 172.21.15.42:0/914117239 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f123c0b3bd0 0x7f123c0b60e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1244005d90 tx=0x7f1244005d20 comp rx=0 tx=0).stop 2024-07-26T06:47:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:26.116+0000 7f125908c640 1 -- 172.21.15.42:0/914117239 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1254103310 msgr2=0x7f12541993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:26.116+0000 7f125908c640 1 --2- 172.21.15.42:0/914117239 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1254103310 0x7f12541993f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1248005d00 tx=0x7f124800dde0 comp rx=0 tx=0).stop 2024-07-26T06:47:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:26.116+0000 7f125908c640 1 -- 172.21.15.42:0/914117239 shutdown_connections 2024-07-26T06:47:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:26.116+0000 7f125908c640 1 --2- 172.21.15.42:0/914117239 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f123c0b3bd0 0x7f123c0b60e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:26.116+0000 7f125908c640 1 --2- 172.21.15.42:0/914117239 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1254107050 0x7f125419e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:26.116+0000 7f125908c640 1 --2- 172.21.15.42:0/914117239 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1254103310 0x7f12541993f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:26.116+0000 7f125908c640 1 --2- 172.21.15.42:0/914117239 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12541029c0 0x7f1254198e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:26.116+0000 7f125908c640 1 -- 172.21.15.42:0/914117239 >> 172.21.15.42:0/914117239 conn(0x7f1254079860 msgr2=0x7f1254104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:26.116+0000 7f125908c640 1 -- 172.21.15.42:0/914117239 shutdown_connections 2024-07-26T06:47:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:26.116+0000 7f125908c640 1 -- 172.21.15.42:0/914117239 wait complete. 2024-07-26T06:47:26.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:26 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/914117239' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T06:47:26.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:26 smithi042 ceph-mon[32775]: pgmap v312: 97 pgs: 97 active+clean; 584 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 5.1 KiB/s wr, 18 op/s; 204 B/s, 1 objects/s recovering 2024-07-26T06:47:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:26 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/914117239' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T06:47:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:26 smithi164 ceph-mon[36970]: pgmap v312: 97 pgs: 97 active+clean; 584 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 5.1 KiB/s wr, 18 op/s; 204 B/s, 1 objects/s recovering 2024-07-26T06:47:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:26 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/914117239' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T06:47:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:26 smithi163 ceph-mon[36873]: pgmap v312: 97 pgs: 97 active+clean; 584 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 5.1 KiB/s wr, 18 op/s; 204 B/s, 1 objects/s recovering 2024-07-26T06:47:26.808 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-07-26T06:47:26.808 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-07-26T06:47:27.081 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:27.365 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:27 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/914117239' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T06:47:27.365 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:27 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:47:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:27 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/914117239' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T06:47:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:27 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:47:27.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:27 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/914117239' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T06:47:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:27 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:47:28.333 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:28 smithi042 ceph-mon[32775]: pgmap v313: 97 pgs: 97 active+clean; 584 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 6.0 KiB/s wr, 22 op/s; 204 B/s, 1 objects/s recovering 2024-07-26T06:47:28.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:28 smithi164 ceph-mon[36970]: pgmap v313: 97 pgs: 97 active+clean; 584 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 6.0 KiB/s wr, 22 op/s; 204 B/s, 1 objects/s recovering 2024-07-26T06:47:28.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:28 smithi163 ceph-mon[36873]: pgmap v313: 97 pgs: 97 active+clean; 584 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 6.0 KiB/s wr, 22 op/s; 204 B/s, 1 objects/s recovering 2024-07-26T06:47:28.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.875+0000 7f077ad2c640 1 -- 172.21.15.42:0/2712187115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07741043e0 msgr2=0x7f07741047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:28.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.875+0000 7f077ad2c640 1 --2- 172.21.15.42:0/2712187115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07741043e0 0x7f07741047f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0764009900 tx=0x7f076402a410 comp rx=0 tx=0).stop 2024-07-26T06:47:28.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.875+0000 7f077ad2c640 1 -- 172.21.15.42:0/2712187115 shutdown_connections 2024-07-26T06:47:28.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.875+0000 7f077ad2c640 1 --2- 172.21.15.42:0/2712187115 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f077410d840 0x7f077410fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.875+0000 7f077ad2c640 1 --2- 172.21.15.42:0/2712187115 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0774104d30 0x7f077410d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.875+0000 7f077ad2c640 1 --2- 172.21.15.42:0/2712187115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07741043e0 0x7f07741047f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.875+0000 7f077ad2c640 1 -- 172.21.15.42:0/2712187115 >> 172.21.15.42:0/2712187115 conn(0x7f0774079160 msgr2=0x7f0774079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.875+0000 7f077ad2c640 1 -- 172.21.15.42:0/2712187115 shutdown_connections 2024-07-26T06:47:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.875+0000 7f077ad2c640 1 -- 172.21.15.42:0/2712187115 wait complete. 2024-07-26T06:47:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.876+0000 7f077ad2c640 1 Processor -- start 2024-07-26T06:47:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.876+0000 7f077ad2c640 1 -- start start 2024-07-26T06:47:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.876+0000 7f077ad2c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07741043e0 0x7f0774199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.876+0000 7f077ad2c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0774104d30 0x7f077419a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.876+0000 7f077ad2c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f077410d840 0x7f077419f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.876+0000 7f077ad2c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f07741134b0 con 0x7f077410d840 2024-07-26T06:47:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.876+0000 7f077ad2c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0774113330 con 0x7f0774104d30 2024-07-26T06:47:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.876+0000 7f077ad2c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0774113630 con 0x7f07741043e0 2024-07-26T06:47:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.876+0000 7f07792c9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f077410d840 0x7f077419f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.876+0000 7f07792c9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f077410d840 0x7f077419f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47642/0 (socket says 172.21.15.42:47642) 2024-07-26T06:47:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.877+0000 7f07792c9640 1 -- 172.21.15.42:0/3470283483 learned_addr learned my addr 172.21.15.42:0/3470283483 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.877+0000 7f0778ac8640 1 --2- 172.21.15.42:0/3470283483 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07741043e0 0x7f0774199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.877+0000 7f0773fff640 1 --2- 172.21.15.42:0/3470283483 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0774104d30 0x7f077419a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.877+0000 7f07792c9640 1 -- 172.21.15.42:0/3470283483 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07741043e0 msgr2=0x7f0774199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.877+0000 7f07792c9640 1 --2- 172.21.15.42:0/3470283483 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07741043e0 0x7f0774199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.877+0000 7f07792c9640 1 -- 172.21.15.42:0/3470283483 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0774104d30 msgr2=0x7f077419a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.877+0000 7f07792c9640 1 --2- 172.21.15.42:0/3470283483 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0774104d30 0x7f077419a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.877+0000 7f07792c9640 1 -- 172.21.15.42:0/3470283483 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f077419fe50 con 0x7f077410d840 2024-07-26T06:47:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.877+0000 7f07792c9640 1 --2- 172.21.15.42:0/3470283483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f077410d840 0x7f077419f720 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f076c00eaa0 tx=0x7f076c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.877+0000 7f0771ffb640 1 -- 172.21.15.42:0/3470283483 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f076c00cdb0 con 0x7f077410d840 2024-07-26T06:47:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.877+0000 7f0771ffb640 1 -- 172.21.15.42:0/3470283483 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f076c005020 con 0x7f077410d840 2024-07-26T06:47:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.877+0000 7f0771ffb640 1 -- 172.21.15.42:0/3470283483 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f076c0116d0 con 0x7f077410d840 2024-07-26T06:47:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.877+0000 7f077ad2c640 1 -- 172.21.15.42:0/3470283483 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07741a0140 con 0x7f077410d840 2024-07-26T06:47:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.877+0000 7f077ad2c640 1 -- 172.21.15.42:0/3470283483 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07741a68a0 con 0x7f077410d840 2024-07-26T06:47:28.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.878+0000 7f077ad2c640 1 -- 172.21.15.42:0/3470283483 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07741047f0 con 0x7f077410d840 2024-07-26T06:47:28.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.880+0000 7f0771ffb640 1 -- 172.21.15.42:0/3470283483 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 16) ==== 151059+0+0 (secure 0 0 0) 0x7f076c0051d0 con 0x7f077410d840 2024-07-26T06:47:28.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.881+0000 7f0771ffb640 1 --2- 172.21.15.42:0/3470283483 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f075c0b3c50 0x7f075c0b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.881+0000 7f0778ac8640 1 --2- 172.21.15.42:0/3470283483 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f075c0b3c50 0x7f075c0b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.881+0000 7f0771ffb640 1 -- 172.21.15.42:0/3470283483 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f076c0e3e90 con 0x7f077410d840 2024-07-26T06:47:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.881+0000 7f0778ac8640 1 --2- 172.21.15.42:0/3470283483 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f075c0b3c50 0x7f075c0b6160 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0764000c00 tx=0x7f0764031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:28.886+0000 7f0771ffb640 1 -- 172.21.15.42:0/3470283483 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f076c016030 con 0x7f077410d840 2024-07-26T06:47:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:29.094+0000 7f077ad2c640 1 -- 172.21.15.42:0/3470283483 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f0774063230 con 0x7f075c0b3c50 2024-07-26T06:47:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:29.126+0000 7f0771ffb640 1 -- 172.21.15.42:0/3470283483 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0774063230 con 0x7f075c0b3c50 2024-07-26T06:47:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:29.129+0000 7f077ad2c640 1 -- 172.21.15.42:0/3470283483 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f075c0b3c50 msgr2=0x7f075c0b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:29.129+0000 7f077ad2c640 1 --2- 172.21.15.42:0/3470283483 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f075c0b3c50 0x7f075c0b6160 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0764000c00 tx=0x7f0764031400 comp rx=0 tx=0).stop 2024-07-26T06:47:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:29.129+0000 7f077ad2c640 1 -- 172.21.15.42:0/3470283483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f077410d840 msgr2=0x7f077419f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:29.129+0000 7f077ad2c640 1 --2- 172.21.15.42:0/3470283483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f077410d840 0x7f077419f720 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f076c00eaa0 tx=0x7f076c00ef60 comp rx=0 tx=0).stop 2024-07-26T06:47:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:29.129+0000 7f077ad2c640 1 -- 172.21.15.42:0/3470283483 shutdown_connections 2024-07-26T06:47:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:29.129+0000 7f077ad2c640 1 --2- 172.21.15.42:0/3470283483 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f075c0b3c50 0x7f075c0b6160 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:29.129+0000 7f077ad2c640 1 --2- 172.21.15.42:0/3470283483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f077410d840 0x7f077419f720 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:29.129+0000 7f077ad2c640 1 --2- 172.21.15.42:0/3470283483 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0774104d30 0x7f077419a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:29.129+0000 7f077ad2c640 1 --2- 172.21.15.42:0/3470283483 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07741043e0 0x7f0774199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:29.129+0000 7f077ad2c640 1 -- 172.21.15.42:0/3470283483 >> 172.21.15.42:0/3470283483 conn(0x7f0774079160 msgr2=0x7f07741001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:29.129+0000 7f077ad2c640 1 -- 172.21.15.42:0/3470283483 shutdown_connections 2024-07-26T06:47:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:29.129+0000 7f077ad2c640 1 -- 172.21.15.42:0/3470283483 wait complete. 2024-07-26T06:47:29.395 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:47:29 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: 2024-07-26T06:47:29.100+0000 7f813c2c3640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:47:29.395 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:47:29 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: 2024-07-26T06:47:29.100+0000 7f813c2c3640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:47:29.395 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:47:29 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: 2024-07-26T06:47:29.100+0000 7f813c2c3640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:47:29.395 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:47:29 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: 2024-07-26T06:47:29.100+0000 7f813c2c3640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:47:29.395 INFO:journalctl@ceph.mgr.x.smithi042.stdout:Jul 26 06:47:29 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mgr-x[33004]: 2024-07-26T06:47:29.100+0000 7f813c2c3640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T06:47:29.714 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-07-26T06:47:29.979 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:30.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:30 smithi042 ceph-mon[32775]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:47:30.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:30 smithi042 ceph-mon[32775]: pgmap v314: 97 pgs: 97 active+clean; 584 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 5.6 KiB/s wr, 27 op/s; 204 B/s, 1 objects/s recovering 2024-07-26T06:47:30.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:30 smithi164 ceph-mon[36970]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:47:30.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:30 smithi164 ceph-mon[36970]: pgmap v314: 97 pgs: 97 active+clean; 584 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 5.6 KiB/s wr, 27 op/s; 204 B/s, 1 objects/s recovering 2024-07-26T06:47:30.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:30 smithi163 ceph-mon[36873]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:47:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:30 smithi163 ceph-mon[36873]: pgmap v314: 97 pgs: 97 active+clean; 584 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 5.6 KiB/s wr, 27 op/s; 204 B/s, 1 objects/s recovering 2024-07-26T06:47:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.613+0000 7f10a384f640 1 -- 172.21.15.42:0/1227230988 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f109c107df0 msgr2=0x7f109c108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.613+0000 7f10a384f640 1 --2- 172.21.15.42:0/1227230988 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f109c107df0 0x7f109c108200 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1090009900 tx=0x7f109002a3f0 comp rx=0 tx=0).stop 2024-07-26T06:47:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.613+0000 7f10a384f640 1 -- 172.21.15.42:0/1227230988 shutdown_connections 2024-07-26T06:47:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.613+0000 7f10a384f640 1 --2- 172.21.15.42:0/1227230988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f109c108740 0x7f109c10ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.613+0000 7f10a384f640 1 --2- 172.21.15.42:0/1227230988 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f109c107df0 0x7f109c108200 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.613+0000 7f10a384f640 1 --2- 172.21.15.42:0/1227230988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f109c1024b0 0x7f109c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.613+0000 7f10a384f640 1 -- 172.21.15.42:0/1227230988 >> 172.21.15.42:0/1227230988 conn(0x7f109c079170 msgr2=0x7f109c079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.613+0000 7f10a384f640 1 -- 172.21.15.42:0/1227230988 shutdown_connections 2024-07-26T06:47:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.613+0000 7f10a384f640 1 -- 172.21.15.42:0/1227230988 wait complete. 2024-07-26T06:47:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.614+0000 7f10a384f640 1 Processor -- start 2024-07-26T06:47:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.614+0000 7f10a384f640 1 -- start start 2024-07-26T06:47:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.615+0000 7f10a384f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f109c1024b0 0x7f109c19ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.615+0000 7f10a384f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f109c107df0 0x7f109c19b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.615+0000 7f10a384f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f109c108740 0x7f109c19b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.615+0000 7f10a384f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f109c112390 con 0x7f109c108740 2024-07-26T06:47:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.615+0000 7f10a384f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f109c112210 con 0x7f109c1024b0 2024-07-26T06:47:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.615+0000 7f10a384f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f109c112510 con 0x7f109c107df0 2024-07-26T06:47:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.615+0000 7f10a1dec640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f109c108740 0x7f109c19b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.615+0000 7f10a1dec640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f109c108740 0x7f109c19b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47672/0 (socket says 172.21.15.42:47672) 2024-07-26T06:47:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.615+0000 7f10a1dec640 1 -- 172.21.15.42:0/585403557 learned_addr learned my addr 172.21.15.42:0/585403557 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.615+0000 7f10a0dea640 1 --2- 172.21.15.42:0/585403557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f109c107df0 0x7f109c19b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.615+0000 7f10a15eb640 1 --2- 172.21.15.42:0/585403557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f109c1024b0 0x7f109c19ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.615+0000 7f10a1dec640 1 -- 172.21.15.42:0/585403557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f109c107df0 msgr2=0x7f109c19b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.615+0000 7f10a1dec640 1 --2- 172.21.15.42:0/585403557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f109c107df0 0x7f109c19b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.615+0000 7f10a1dec640 1 -- 172.21.15.42:0/585403557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f109c1024b0 msgr2=0x7f109c19ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.615+0000 7f10a1dec640 1 --2- 172.21.15.42:0/585403557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f109c1024b0 0x7f109c19ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.615+0000 7f10a1dec640 1 -- 172.21.15.42:0/585403557 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f109c1a5f90 con 0x7f109c108740 2024-07-26T06:47:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.616+0000 7f10a1dec640 1 --2- 172.21.15.42:0/585403557 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f109c108740 0x7f109c19b7e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f109400c970 tx=0x7f109400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.616+0000 7f109a7fc640 1 -- 172.21.15.42:0/585403557 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094007bb0 con 0x7f109c108740 2024-07-26T06:47:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.616+0000 7f10a384f640 1 -- 172.21.15.42:0/585403557 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f109c1a6280 con 0x7f109c108740 2024-07-26T06:47:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.616+0000 7f109a7fc640 1 -- 172.21.15.42:0/585403557 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1094007d50 con 0x7f109c108740 2024-07-26T06:47:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.616+0000 7f109a7fc640 1 -- 172.21.15.42:0/585403557 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094018770 con 0x7f109c108740 2024-07-26T06:47:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.616+0000 7f10a384f640 1 -- 172.21.15.42:0/585403557 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f109c1a6740 con 0x7f109c108740 2024-07-26T06:47:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.617+0000 7f10a384f640 1 -- 172.21.15.42:0/585403557 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f109c103730 con 0x7f109c108740 2024-07-26T06:47:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.618+0000 7f109a7fc640 1 -- 172.21.15.42:0/585403557 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f109400f990 con 0x7f109c108740 2024-07-26T06:47:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.619+0000 7f109a7fc640 1 --2- 172.21.15.42:0/585403557 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f10840b3c60 0x7f10840b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.619+0000 7f10a15eb640 1 --2- 172.21.15.42:0/585403557 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f10840b3c60 0x7f10840b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.619+0000 7f109a7fc640 1 -- 172.21.15.42:0/585403557 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f10940e3e80 con 0x7f109c108740 2024-07-26T06:47:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.620+0000 7f10a15eb640 1 --2- 172.21.15.42:0/585403557 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f10840b3c60 0x7f10840b6170 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f109c06c0f0 tx=0x7f108c009680 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:31.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.622+0000 7f109a7fc640 1 -- 172.21.15.42:0/585403557 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1094014030 con 0x7f109c108740 2024-07-26T06:47:31.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:31 smithi163 ceph-mon[36873]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-07-26T06:47:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:31 smithi164 ceph-mon[36970]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-07-26T06:47:31.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.832+0000 7f10a384f640 1 -- 172.21.15.42:0/585403557 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f109c063230 con 0x7f10840b3c60 2024-07-26T06:47:31.833 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:31 smithi042 ceph-mon[32775]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-07-26T06:47:31.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.858+0000 7f109a7fc640 1 -- 172.21.15.42:0/585403557 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f109c063230 con 0x7f10840b3c60 2024-07-26T06:47:31.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.860+0000 7f10a384f640 1 -- 172.21.15.42:0/585403557 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f10840b3c60 msgr2=0x7f10840b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:31.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.860+0000 7f10a384f640 1 --2- 172.21.15.42:0/585403557 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f10840b3c60 0x7f10840b6170 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f109c06c0f0 tx=0x7f108c009680 comp rx=0 tx=0).stop 2024-07-26T06:47:31.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.860+0000 7f10a384f640 1 -- 172.21.15.42:0/585403557 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f109c108740 msgr2=0x7f109c19b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:31.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.860+0000 7f10a384f640 1 --2- 172.21.15.42:0/585403557 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f109c108740 0x7f109c19b7e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f109400c970 tx=0x7f109400ce30 comp rx=0 tx=0).stop 2024-07-26T06:47:31.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.860+0000 7f10a384f640 1 -- 172.21.15.42:0/585403557 shutdown_connections 2024-07-26T06:47:31.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.860+0000 7f10a384f640 1 --2- 172.21.15.42:0/585403557 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f10840b3c60 0x7f10840b6170 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:31.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.860+0000 7f10a384f640 1 --2- 172.21.15.42:0/585403557 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f109c108740 0x7f109c19b7e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:31.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.860+0000 7f10a384f640 1 --2- 172.21.15.42:0/585403557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f109c107df0 0x7f109c19b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:31.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.860+0000 7f10a384f640 1 --2- 172.21.15.42:0/585403557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f109c1024b0 0x7f109c19ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:31.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.860+0000 7f10a384f640 1 -- 172.21.15.42:0/585403557 >> 172.21.15.42:0/585403557 conn(0x7f109c079170 msgr2=0x7f109c103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:31.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.860+0000 7f10a384f640 1 -- 172.21.15.42:0/585403557 shutdown_connections 2024-07-26T06:47:31.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:31.860+0000 7f10a384f640 1 -- 172.21.15.42:0/585403557 wait complete. 2024-07-26T06:47:32.463 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:32 smithi042 ceph-mon[32775]: pgmap v315: 97 pgs: 97 active+clean; 584 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 3.6 KiB/s wr, 22 op/s; 176 B/s, 1 objects/s recovering 2024-07-26T06:47:32.464 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:32 smithi042 ceph-mon[32775]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:47:32.495 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-07-26T06:47:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:32 smithi164 ceph-mon[36970]: pgmap v315: 97 pgs: 97 active+clean; 584 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 3.6 KiB/s wr, 22 op/s; 176 B/s, 1 objects/s recovering 2024-07-26T06:47:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:32 smithi164 ceph-mon[36970]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:47:32.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:32 smithi163 ceph-mon[36873]: pgmap v315: 97 pgs: 97 active+clean; 584 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 3.6 KiB/s wr, 22 op/s; 176 B/s, 1 objects/s recovering 2024-07-26T06:47:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:32 smithi163 ceph-mon[36873]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:47:32.764 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:34.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.507+0000 7fef0fa0e640 1 -- 172.21.15.42:0/3790073203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef081043e0 msgr2=0x7fef081047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:34.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.507+0000 7fef0fa0e640 1 --2- 172.21.15.42:0/3790073203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef081043e0 0x7fef081047f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fef04009900 tx=0x7fef0402a440 comp rx=0 tx=0).stop 2024-07-26T06:47:34.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.508+0000 7fef0fa0e640 1 -- 172.21.15.42:0/3790073203 shutdown_connections 2024-07-26T06:47:34.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.508+0000 7fef0fa0e640 1 --2- 172.21.15.42:0/3790073203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef0810d840 0x7fef0810fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:34.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.508+0000 7fef0fa0e640 1 --2- 172.21.15.42:0/3790073203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef08104d30 0x7fef0810d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:34.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.508+0000 7fef0fa0e640 1 --2- 172.21.15.42:0/3790073203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef081043e0 0x7fef081047f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:34.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.508+0000 7fef0fa0e640 1 -- 172.21.15.42:0/3790073203 >> 172.21.15.42:0/3790073203 conn(0x7fef08079190 msgr2=0x7fef080795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:34.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.508+0000 7fef0fa0e640 1 -- 172.21.15.42:0/3790073203 shutdown_connections 2024-07-26T06:47:34.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.508+0000 7fef0fa0e640 1 -- 172.21.15.42:0/3790073203 wait complete. 2024-07-26T06:47:34.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.508+0000 7fef0fa0e640 1 Processor -- start 2024-07-26T06:47:34.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.508+0000 7fef0fa0e640 1 -- start start 2024-07-26T06:47:34.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0fa0e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef081043e0 0x7fef08102120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:34.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0fa0e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef08104d30 0x7fef08102660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:34.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0fa0e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef0810d840 0x7fef08102ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0fa0e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fef081135a0 con 0x7fef08104d30 2024-07-26T06:47:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0fa0e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fef08113420 con 0x7fef0810d840 2024-07-26T06:47:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0fa0e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fef08113720 con 0x7fef081043e0 2024-07-26T06:47:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0cfa9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef08104d30 0x7fef08102660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0cfa9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef08104d30 0x7fef08102660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47696/0 (socket says 172.21.15.42:47696) 2024-07-26T06:47:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0cfa9640 1 -- 172.21.15.42:0/776308346 learned_addr learned my addr 172.21.15.42:0/776308346 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0dfab640 1 --2- 172.21.15.42:0/776308346 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef0810d840 0x7fef08102ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0d7aa640 1 --2- 172.21.15.42:0/776308346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef081043e0 0x7fef08102120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0cfa9640 1 -- 172.21.15.42:0/776308346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef081043e0 msgr2=0x7fef08102120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0cfa9640 1 --2- 172.21.15.42:0/776308346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef081043e0 0x7fef08102120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:34.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0cfa9640 1 -- 172.21.15.42:0/776308346 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef0810d840 msgr2=0x7fef08102ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0cfa9640 1 --2- 172.21.15.42:0/776308346 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef0810d840 0x7fef08102ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0cfa9640 1 -- 172.21.15.42:0/776308346 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef081a82f0 con 0x7fef08104d30 2024-07-26T06:47:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0dfab640 1 --2- 172.21.15.42:0/776308346 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef0810d840 0x7fef08102ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:47:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.509+0000 7fef0cfa9640 1 --2- 172.21.15.42:0/776308346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef08104d30 0x7fef08102660 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7feef800b450 tx=0x7feef800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.510+0000 7feefe7fc640 1 -- 172.21.15.42:0/776308346 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef8015020 con 0x7fef08104d30 2024-07-26T06:47:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.510+0000 7feefe7fc640 1 -- 172.21.15.42:0/776308346 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feef8011d40 con 0x7fef08104d30 2024-07-26T06:47:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.510+0000 7fef0fa0e640 1 -- 172.21.15.42:0/776308346 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef081a85e0 con 0x7fef08104d30 2024-07-26T06:47:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.510+0000 7feefe7fc640 1 -- 172.21.15.42:0/776308346 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef800f340 con 0x7fef08104d30 2024-07-26T06:47:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.510+0000 7fef0fa0e640 1 -- 172.21.15.42:0/776308346 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef081a8bf0 con 0x7fef08104d30 2024-07-26T06:47:34.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.511+0000 7fef0fa0e640 1 -- 172.21.15.42:0/776308346 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef0806bcd0 con 0x7fef08104d30 2024-07-26T06:47:34.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.512+0000 7feefe7fc640 1 -- 172.21.15.42:0/776308346 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7feef8018020 con 0x7fef08104d30 2024-07-26T06:47:34.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.513+0000 7feefe7fc640 1 --2- 172.21.15.42:0/776308346 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7feef40b3ce0 0x7feef40b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:34.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.513+0000 7feefe7fc640 1 -- 172.21.15.42:0/776308346 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7feef80e3c90 con 0x7fef08104d30 2024-07-26T06:47:34.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.513+0000 7fef0d7aa640 1 --2- 172.21.15.42:0/776308346 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7feef40b3ce0 0x7feef40b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:34.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.514+0000 7fef0d7aa640 1 --2- 172.21.15.42:0/776308346 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7feef40b3ce0 0x7feef40b61f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fef04000c00 tx=0x7fef04031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:34.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.516+0000 7feefe7fc640 1 -- 172.21.15.42:0/776308346 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7feef801f020 con 0x7fef08104d30 2024-07-26T06:47:34.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:34 smithi042 ceph-mon[32775]: pgmap v316: 97 pgs: 97 active+clean; 585 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.4 KiB/s wr, 21 op/s; 159 B/s, 0 objects/s recovering 2024-07-26T06:47:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:34 smithi164 ceph-mon[36970]: pgmap v316: 97 pgs: 97 active+clean; 585 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.4 KiB/s wr, 21 op/s; 159 B/s, 0 objects/s recovering 2024-07-26T06:47:34.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.728+0000 7fef0fa0e640 1 -- 172.21.15.42:0/776308346 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fef08063230 con 0x7feef40b3ce0 2024-07-26T06:47:34.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.733+0000 7feefe7fc640 1 -- 172.21.15.42:0/776308346 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fef08063230 con 0x7feef40b3ce0 2024-07-26T06:47:34.734 INFO:teuthology.orchestra.run.smithi042.stdout:/volumes/qa/sv_0/fcfa1750-6a13-4c07-ace3-21952ffac61c 2024-07-26T06:47:34.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.735+0000 7fef0fa0e640 1 -- 172.21.15.42:0/776308346 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7feef40b3ce0 msgr2=0x7feef40b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:34.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.735+0000 7fef0fa0e640 1 --2- 172.21.15.42:0/776308346 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7feef40b3ce0 0x7feef40b61f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fef04000c00 tx=0x7fef04031400 comp rx=0 tx=0).stop 2024-07-26T06:47:34.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.735+0000 7fef0fa0e640 1 -- 172.21.15.42:0/776308346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef08104d30 msgr2=0x7fef08102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:34.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.735+0000 7fef0fa0e640 1 --2- 172.21.15.42:0/776308346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef08104d30 0x7fef08102660 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7feef800b450 tx=0x7feef800b910 comp rx=0 tx=0).stop 2024-07-26T06:47:34.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.735+0000 7fef0fa0e640 1 -- 172.21.15.42:0/776308346 shutdown_connections 2024-07-26T06:47:34.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.735+0000 7fef0fa0e640 1 --2- 172.21.15.42:0/776308346 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7feef40b3ce0 0x7feef40b61f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:34.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.735+0000 7fef0fa0e640 1 --2- 172.21.15.42:0/776308346 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef0810d840 0x7fef08102ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:34.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.735+0000 7fef0fa0e640 1 --2- 172.21.15.42:0/776308346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef08104d30 0x7fef08102660 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:34.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.735+0000 7fef0fa0e640 1 --2- 172.21.15.42:0/776308346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef081043e0 0x7fef08102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:34.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.735+0000 7fef0fa0e640 1 -- 172.21.15.42:0/776308346 >> 172.21.15.42:0/776308346 conn(0x7fef08079190 msgr2=0x7fef08100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:34.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.736+0000 7fef0fa0e640 1 -- 172.21.15.42:0/776308346 shutdown_connections 2024-07-26T06:47:34.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:34.736+0000 7fef0fa0e640 1 -- 172.21.15.42:0/776308346 wait complete. 2024-07-26T06:47:34.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:34 smithi163 ceph-mon[36873]: pgmap v316: 97 pgs: 97 active+clean; 585 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.4 KiB/s wr, 21 op/s; 159 B/s, 0 objects/s recovering 2024-07-26T06:47:35.398 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-07-26T06:47:35.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:35 smithi042 ceph-mon[32775]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:47:35.673 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:35.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:35 smithi164 ceph-mon[36970]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:47:35.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:35 smithi163 ceph-mon[36873]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:47:36.695 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:36 smithi042 ceph-mon[32775]: pgmap v317: 97 pgs: 97 active+clean; 605 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.0 KiB/s wr, 20 op/s; 134 B/s, 0 objects/s recovering 2024-07-26T06:47:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:36 smithi164 ceph-mon[36970]: pgmap v317: 97 pgs: 97 active+clean; 605 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.0 KiB/s wr, 20 op/s; 134 B/s, 0 objects/s recovering 2024-07-26T06:47:36.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:36 smithi163 ceph-mon[36873]: pgmap v317: 97 pgs: 97 active+clean; 605 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 4.0 KiB/s wr, 20 op/s; 134 B/s, 0 objects/s recovering 2024-07-26T06:47:37.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.432+0000 7f49dfcc7640 1 -- 172.21.15.42:0/1884303763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49d8102970 msgr2=0x7f49d810f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:37.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.432+0000 7f49dfcc7640 1 --2- 172.21.15.42:0/1884303763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49d8102970 0x7f49d810f690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f49c8009900 tx=0x7f49c802a480 comp rx=0 tx=0).stop 2024-07-26T06:47:37.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.432+0000 7f49dfcc7640 1 -- 172.21.15.42:0/1884303763 shutdown_connections 2024-07-26T06:47:37.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.432+0000 7f49dfcc7640 1 --2- 172.21.15.42:0/1884303763 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49d810fbd0 0x7f49d8111ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:37.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.432+0000 7f49dfcc7640 1 --2- 172.21.15.42:0/1884303763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49d8102970 0x7f49d810f690 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:37.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.432+0000 7f49dfcc7640 1 --2- 172.21.15.42:0/1884303763 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49d8102020 0x7f49d8102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:37.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.432+0000 7f49dfcc7640 1 -- 172.21.15.42:0/1884303763 >> 172.21.15.42:0/1884303763 conn(0x7f49d8079170 msgr2=0x7f49d8079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.432+0000 7f49dfcc7640 1 -- 172.21.15.42:0/1884303763 shutdown_connections 2024-07-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.432+0000 7f49dfcc7640 1 -- 172.21.15.42:0/1884303763 wait complete. 2024-07-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.433+0000 7f49dfcc7640 1 Processor -- start 2024-07-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.433+0000 7f49dfcc7640 1 -- start start 2024-07-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.433+0000 7f49dfcc7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49d8102020 0x7f49d819c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.433+0000 7f49dfcc7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49d8102970 0x7f49d819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.433+0000 7f49dfcc7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49d810fbd0 0x7f49d81a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.433+0000 7f49dfcc7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f49d8115760 con 0x7f49d8102970 2024-07-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.433+0000 7f49dfcc7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f49d81155e0 con 0x7f49d810fbd0 2024-07-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.433+0000 7f49dfcc7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f49d81158e0 con 0x7f49d8102020 2024-07-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.433+0000 7f49dd262640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49d8102970 0x7f49d819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.433+0000 7f49dd262640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49d8102970 0x7f49d819c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54520/0 (socket says 172.21.15.42:54520) 2024-07-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.433+0000 7f49dd262640 1 -- 172.21.15.42:0/3004061852 learned_addr learned my addr 172.21.15.42:0/3004061852 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.433+0000 7f49dda63640 1 --2- 172.21.15.42:0/3004061852 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49d8102020 0x7f49d819c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.433+0000 7f49de264640 1 --2- 172.21.15.42:0/3004061852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49d810fbd0 0x7f49d81a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.434+0000 7f49dd262640 1 -- 172.21.15.42:0/3004061852 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49d8102020 msgr2=0x7f49d819c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.434+0000 7f49dd262640 1 --2- 172.21.15.42:0/3004061852 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49d8102020 0x7f49d819c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.434+0000 7f49dd262640 1 -- 172.21.15.42:0/3004061852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49d810fbd0 msgr2=0x7f49d81a1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.434+0000 7f49dd262640 1 --2- 172.21.15.42:0/3004061852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49d810fbd0 0x7f49d81a1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.434+0000 7f49dd262640 1 -- 172.21.15.42:0/3004061852 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d81a2180 con 0x7f49d8102970 2024-07-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.434+0000 7f49dd262640 1 --2- 172.21.15.42:0/3004061852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49d8102970 0x7f49d819c7b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f49c8006010 tx=0x7f49c80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.434+0000 7f49ceffd640 1 -- 172.21.15.42:0/3004061852 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c803f030 con 0x7f49d8102970 2024-07-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.434+0000 7f49ceffd640 1 -- 172.21.15.42:0/3004061852 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49c8003800 con 0x7f49d8102970 2024-07-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.434+0000 7f49dfcc7640 1 -- 172.21.15.42:0/3004061852 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49d81a2410 con 0x7f49d8102970 2024-07-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.434+0000 7f49ceffd640 1 -- 172.21.15.42:0/3004061852 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c8035070 con 0x7f49d8102970 2024-07-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.434+0000 7f49dfcc7640 1 -- 172.21.15.42:0/3004061852 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49d81a8bd0 con 0x7f49d8102970 2024-07-26T06:47:37.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.435+0000 7f49dfcc7640 1 -- 172.21.15.42:0/3004061852 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49d806bc70 con 0x7f49d8102970 2024-07-26T06:47:37.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.437+0000 7f49ceffd640 1 -- 172.21.15.42:0/3004061852 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f49c8003cb0 con 0x7f49d8102970 2024-07-26T06:47:37.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.437+0000 7f49ceffd640 1 --2- 172.21.15.42:0/3004061852 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49c40b3ce0 0x7f49c40b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:37.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.438+0000 7f49ceffd640 1 -- 172.21.15.42:0/3004061852 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f49c81049c0 con 0x7f49d8102970 2024-07-26T06:47:37.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.438+0000 7f49dda63640 1 --2- 172.21.15.42:0/3004061852 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49c40b3ce0 0x7f49c40b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:37.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.438+0000 7f49dda63640 1 --2- 172.21.15.42:0/3004061852 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49c40b3ce0 0x7f49c40b61f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f49d4009890 tx=0x7f49d4008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:37.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.441+0000 7f49ceffd640 1 -- 172.21.15.42:0/3004061852 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f49c80cbe90 con 0x7f49d8102970 2024-07-26T06:47:37.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.653+0000 7f49dfcc7640 1 -- 172.21.15.42:0/3004061852 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f49d8063230 con 0x7f49c40b3ce0 2024-07-26T06:47:37.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.680+0000 7f49ceffd640 1 -- 172.21.15.42:0/3004061852 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f49d8063230 con 0x7f49c40b3ce0 2024-07-26T06:47:37.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.682+0000 7f49dfcc7640 1 -- 172.21.15.42:0/3004061852 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49c40b3ce0 msgr2=0x7f49c40b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:37.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.682+0000 7f49dfcc7640 1 --2- 172.21.15.42:0/3004061852 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49c40b3ce0 0x7f49c40b61f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f49d4009890 tx=0x7f49d4008040 comp rx=0 tx=0).stop 2024-07-26T06:47:37.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.682+0000 7f49dfcc7640 1 -- 172.21.15.42:0/3004061852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49d8102970 msgr2=0x7f49d819c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:37.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.682+0000 7f49dfcc7640 1 --2- 172.21.15.42:0/3004061852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49d8102970 0x7f49d819c7b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f49c8006010 tx=0x7f49c80042f0 comp rx=0 tx=0).stop 2024-07-26T06:47:37.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.682+0000 7f49dfcc7640 1 -- 172.21.15.42:0/3004061852 shutdown_connections 2024-07-26T06:47:37.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.682+0000 7f49dfcc7640 1 --2- 172.21.15.42:0/3004061852 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49c40b3ce0 0x7f49c40b61f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:37.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.682+0000 7f49dfcc7640 1 --2- 172.21.15.42:0/3004061852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49d810fbd0 0x7f49d81a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:37.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.682+0000 7f49dfcc7640 1 --2- 172.21.15.42:0/3004061852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49d8102970 0x7f49d819c7b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:37.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.682+0000 7f49dfcc7640 1 --2- 172.21.15.42:0/3004061852 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49d8102020 0x7f49d819c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:37.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.682+0000 7f49dfcc7640 1 -- 172.21.15.42:0/3004061852 >> 172.21.15.42:0/3004061852 conn(0x7f49d8079170 msgr2=0x7f49d81110e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:37.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.682+0000 7f49dfcc7640 1 -- 172.21.15.42:0/3004061852 shutdown_connections 2024-07-26T06:47:37.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:37.682+0000 7f49dfcc7640 1 -- 172.21.15.42:0/3004061852 wait complete. 2024-07-26T06:47:38.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:38 smithi042 ceph-mon[32775]: pgmap v318: 97 pgs: 97 active+clean; 605 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 2.9 KiB/s wr, 14 op/s 2024-07-26T06:47:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:38 smithi042 ceph-mon[32775]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:47:38.693 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-07-26T06:47:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:38 smithi164 ceph-mon[36970]: pgmap v318: 97 pgs: 97 active+clean; 605 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 2.9 KiB/s wr, 14 op/s 2024-07-26T06:47:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:38 smithi164 ceph-mon[36970]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:47:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:38 smithi163 ceph-mon[36873]: pgmap v318: 97 pgs: 97 active+clean; 605 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 2.9 KiB/s wr, 14 op/s 2024-07-26T06:47:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:38 smithi163 ceph-mon[36873]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:47:38.968 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:40.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.366+0000 7f6da988c640 1 -- 172.21.15.42:0/497592295 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da4103590 msgr2=0x7f6da41039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:40.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.366+0000 7f6da988c640 1 --2- 172.21.15.42:0/497592295 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da4103590 0x7f6da41039a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6d94009900 tx=0x7f6d9402a410 comp rx=0 tx=0).stop 2024-07-26T06:47:40.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.366+0000 7f6da988c640 1 -- 172.21.15.42:0/497592295 shutdown_connections 2024-07-26T06:47:40.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.366+0000 7f6da988c640 1 --2- 172.21.15.42:0/497592295 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6da4109820 0x7f6da4110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:40.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.366+0000 7f6da988c640 1 --2- 172.21.15.42:0/497592295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6da4108ed0 0x7f6da41092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:40.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.366+0000 7f6da988c640 1 --2- 172.21.15.42:0/497592295 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da4103590 0x7f6da41039a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:40.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.366+0000 7f6da988c640 1 -- 172.21.15.42:0/497592295 >> 172.21.15.42:0/497592295 conn(0x7f6da407b280 msgr2=0x7f6da407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:40.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.366+0000 7f6da988c640 1 -- 172.21.15.42:0/497592295 shutdown_connections 2024-07-26T06:47:40.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.366+0000 7f6da988c640 1 -- 172.21.15.42:0/497592295 wait complete. 2024-07-26T06:47:40.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.366+0000 7f6da988c640 1 Processor -- start 2024-07-26T06:47:40.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.366+0000 7f6da988c640 1 -- start start 2024-07-26T06:47:40.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6da988c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6da4103590 0x7f6da4199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:40.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6da988c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da4108ed0 0x7f6da419a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:40.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6da988c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6da4109820 0x7f6da419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:40.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6da988c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6da4113540 con 0x7f6da4108ed0 2024-07-26T06:47:40.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6da988c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6da41133c0 con 0x7f6da4103590 2024-07-26T06:47:40.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6da988c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6da41136c0 con 0x7f6da4109820 2024-07-26T06:47:40.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6d9bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da4108ed0 0x7f6da419a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:40.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6d9bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da4108ed0 0x7f6da419a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54548/0 (socket says 172.21.15.42:54548) 2024-07-26T06:47:40.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6d9bfff640 1 -- 172.21.15.42:0/2401799521 learned_addr learned my addr 172.21.15.42:0/2401799521 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:40.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6da37fe640 1 --2- 172.21.15.42:0/2401799521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6da4109820 0x7f6da419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:40.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6da2ffd640 1 --2- 172.21.15.42:0/2401799521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6da4103590 0x7f6da4199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:40.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6d9bfff640 1 -- 172.21.15.42:0/2401799521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6da4109820 msgr2=0x7f6da419f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:40.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6d9bfff640 1 --2- 172.21.15.42:0/2401799521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6da4109820 0x7f6da419f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:40.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6d9bfff640 1 -- 172.21.15.42:0/2401799521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6da4103590 msgr2=0x7f6da4199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:40.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6d9bfff640 1 --2- 172.21.15.42:0/2401799521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6da4103590 0x7f6da4199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:40.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6d9bfff640 1 -- 172.21.15.42:0/2401799521 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da419fee0 con 0x7f6da4108ed0 2024-07-26T06:47:40.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6da2ffd640 1 --2- 172.21.15.42:0/2401799521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6da4103590 0x7f6da4199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:47:40.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.367+0000 7f6d9bfff640 1 --2- 172.21.15.42:0/2401799521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da4108ed0 0x7f6da419a510 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c006fd0 tx=0x7f6d8c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:40.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.368+0000 7f6da888a640 1 -- 172.21.15.42:0/2401799521 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c011830 con 0x7f6da4108ed0 2024-07-26T06:47:40.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.368+0000 7f6da888a640 1 -- 172.21.15.42:0/2401799521 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d8c016070 con 0x7f6da4108ed0 2024-07-26T06:47:40.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.368+0000 7f6da988c640 1 -- 172.21.15.42:0/2401799521 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da41a01d0 con 0x7f6da4108ed0 2024-07-26T06:47:40.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.368+0000 7f6da888a640 1 -- 172.21.15.42:0/2401799521 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c010e00 con 0x7f6da4108ed0 2024-07-26T06:47:40.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.368+0000 7f6da988c640 1 -- 172.21.15.42:0/2401799521 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da41a6930 con 0x7f6da4108ed0 2024-07-26T06:47:40.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.369+0000 7f6da988c640 1 -- 172.21.15.42:0/2401799521 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6da41039a0 con 0x7f6da4108ed0 2024-07-26T06:47:40.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.371+0000 7f6da888a640 1 -- 172.21.15.42:0/2401799521 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f6d8c01b480 con 0x7f6da4108ed0 2024-07-26T06:47:40.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.371+0000 7f6da888a640 1 --2- 172.21.15.42:0/2401799521 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6d900b3c60 0x7f6d900b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:40.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.372+0000 7f6da2ffd640 1 --2- 172.21.15.42:0/2401799521 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6d900b3c60 0x7f6d900b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:40.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.372+0000 7f6da888a640 1 -- 172.21.15.42:0/2401799521 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f6d8c0e33a0 con 0x7f6da4108ed0 2024-07-26T06:47:40.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.372+0000 7f6da2ffd640 1 --2- 172.21.15.42:0/2401799521 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6d900b3c60 0x7f6d900b6170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6d94031440 tx=0x7f6d94005da0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:40.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.376+0000 7f6da888a640 1 -- 172.21.15.42:0/2401799521 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6d8c021250 con 0x7f6da4108ed0 2024-07-26T06:47:40.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.588+0000 7f6da988c640 1 -- 172.21.15.42:0/2401799521 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f6da4063230 con 0x7f6d900b3c60 2024-07-26T06:47:40.589 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:40 smithi042 ceph-mon[32775]: pgmap v319: 97 pgs: 97 active+clean; 625 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.3 KiB/s wr, 15 op/s 2024-07-26T06:47:40.594 INFO:teuthology.orchestra.run.smithi042.stdout:/volumes/qa/sv_1/dcf68d2d-a95d-4cd5-9339-221de9d6bf68 2024-07-26T06:47:40.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.593+0000 7f6da888a640 1 -- 172.21.15.42:0/2401799521 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f6da4063230 con 0x7f6d900b3c60 2024-07-26T06:47:40.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.595+0000 7f6da988c640 1 -- 172.21.15.42:0/2401799521 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6d900b3c60 msgr2=0x7f6d900b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:40.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.595+0000 7f6da988c640 1 --2- 172.21.15.42:0/2401799521 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6d900b3c60 0x7f6d900b6170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6d94031440 tx=0x7f6d94005da0 comp rx=0 tx=0).stop 2024-07-26T06:47:40.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.595+0000 7f6da988c640 1 -- 172.21.15.42:0/2401799521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da4108ed0 msgr2=0x7f6da419a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:40.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.595+0000 7f6da988c640 1 --2- 172.21.15.42:0/2401799521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da4108ed0 0x7f6da419a510 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c006fd0 tx=0x7f6d8c00b8e0 comp rx=0 tx=0).stop 2024-07-26T06:47:40.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.595+0000 7f6da988c640 1 -- 172.21.15.42:0/2401799521 shutdown_connections 2024-07-26T06:47:40.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.595+0000 7f6da988c640 1 --2- 172.21.15.42:0/2401799521 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6d900b3c60 0x7f6d900b6170 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:40.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.595+0000 7f6da988c640 1 --2- 172.21.15.42:0/2401799521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6da4109820 0x7f6da419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:40.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.595+0000 7f6da988c640 1 --2- 172.21.15.42:0/2401799521 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6da4108ed0 0x7f6da419a510 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:40.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.595+0000 7f6da988c640 1 --2- 172.21.15.42:0/2401799521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6da4103590 0x7f6da4199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:40.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.595+0000 7f6da988c640 1 -- 172.21.15.42:0/2401799521 >> 172.21.15.42:0/2401799521 conn(0x7f6da407b280 msgr2=0x7f6da4107010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:40.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.595+0000 7f6da988c640 1 -- 172.21.15.42:0/2401799521 shutdown_connections 2024-07-26T06:47:40.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:40.595+0000 7f6da988c640 1 -- 172.21.15.42:0/2401799521 wait complete. 2024-07-26T06:47:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:40 smithi164 ceph-mon[36970]: pgmap v319: 97 pgs: 97 active+clean; 625 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.3 KiB/s wr, 15 op/s 2024-07-26T06:47:40.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:40 smithi163 ceph-mon[36873]: pgmap v319: 97 pgs: 97 active+clean; 625 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.3 KiB/s wr, 15 op/s 2024-07-26T06:47:41.387 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:47:41.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:41 smithi042 ceph-mon[32775]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:47:41.660 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:41 smithi164 ceph-mon[36970]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:47:41.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:41 smithi163 ceph-mon[36873]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:47:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:42 smithi164 ceph-mon[36970]: pgmap v320: 97 pgs: 97 active+clean; 625 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-07-26T06:47:42.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:42 smithi163 ceph-mon[36873]: pgmap v320: 97 pgs: 97 active+clean; 625 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-07-26T06:47:42.785 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:42 smithi042 ceph-mon[32775]: pgmap v320: 97 pgs: 97 active+clean; 625 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-07-26T06:47:43.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.334+0000 7f402016d640 1 -- 172.21.15.42:0/2024487247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40181024b0 msgr2=0x7f40181028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:43.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.334+0000 7f402016d640 1 --2- 172.21.15.42:0/2024487247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40181024b0 0x7f40181028c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4014009900 tx=0x7f401402a410 comp rx=0 tx=0).stop 2024-07-26T06:47:43.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.335+0000 7f402016d640 1 -- 172.21.15.42:0/2024487247 shutdown_connections 2024-07-26T06:47:43.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.335+0000 7f402016d640 1 --2- 172.21.15.42:0/2024487247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4018108740 0x7f401810ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:43.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.335+0000 7f402016d640 1 --2- 172.21.15.42:0/2024487247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4018107df0 0x7f4018108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:43.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.335+0000 7f402016d640 1 --2- 172.21.15.42:0/2024487247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40181024b0 0x7f40181028c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:43.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.335+0000 7f402016d640 1 -- 172.21.15.42:0/2024487247 >> 172.21.15.42:0/2024487247 conn(0x7f4018079170 msgr2=0x7f4018079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:43.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.335+0000 7f402016d640 1 -- 172.21.15.42:0/2024487247 shutdown_connections 2024-07-26T06:47:43.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.335+0000 7f402016d640 1 -- 172.21.15.42:0/2024487247 wait complete. 2024-07-26T06:47:43.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.335+0000 7f402016d640 1 Processor -- start 2024-07-26T06:47:43.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.335+0000 7f402016d640 1 -- start start 2024-07-26T06:47:43.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.336+0000 7f402016d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40181024b0 0x7f4018103ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:43.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.336+0000 7f402016d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4018107df0 0x7f40181041e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:43.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.336+0000 7f402016d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4018108740 0x7f4018104720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:43.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.336+0000 7f402016d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4018112430 con 0x7f4018108740 2024-07-26T06:47:43.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.336+0000 7f402016d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f40181122b0 con 0x7f40181024b0 2024-07-26T06:47:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.336+0000 7f402016d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f40181125b0 con 0x7f4018107df0 2024-07-26T06:47:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.336+0000 7f401e70a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4018108740 0x7f4018104720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.336+0000 7f401e70a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4018108740 0x7f4018104720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54560/0 (socket says 172.21.15.42:54560) 2024-07-26T06:47:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.336+0000 7f401e70a640 1 -- 172.21.15.42:0/33369500 learned_addr learned my addr 172.21.15.42:0/33369500 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.336+0000 7f401d708640 1 --2- 172.21.15.42:0/33369500 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4018107df0 0x7f40181041e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.336+0000 7f401df09640 1 --2- 172.21.15.42:0/33369500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40181024b0 0x7f4018103ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.336+0000 7f401e70a640 1 -- 172.21.15.42:0/33369500 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4018107df0 msgr2=0x7f40181041e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.336+0000 7f401e70a640 1 --2- 172.21.15.42:0/33369500 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4018107df0 0x7f40181041e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.336+0000 7f401e70a640 1 -- 172.21.15.42:0/33369500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40181024b0 msgr2=0x7f4018103ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.336+0000 7f401e70a640 1 --2- 172.21.15.42:0/33369500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40181024b0 0x7f4018103ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.336+0000 7f401e70a640 1 -- 172.21.15.42:0/33369500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40181a8210 con 0x7f4018108740 2024-07-26T06:47:43.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.336+0000 7f401e70a640 1 --2- 172.21.15.42:0/33369500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4018108740 0x7f4018104720 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f401000eaa0 tx=0x7f401000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:43.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.337+0000 7f400effd640 1 -- 172.21.15.42:0/33369500 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4010004d20 con 0x7f4018108740 2024-07-26T06:47:43.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.337+0000 7f402016d640 1 -- 172.21.15.42:0/33369500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40181a8500 con 0x7f4018108740 2024-07-26T06:47:43.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.337+0000 7f400effd640 1 -- 172.21.15.42:0/33369500 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4010005020 con 0x7f4018108740 2024-07-26T06:47:43.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.337+0000 7f402016d640 1 -- 172.21.15.42:0/33369500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40181a8b10 con 0x7f4018108740 2024-07-26T06:47:43.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.337+0000 7f400effd640 1 -- 172.21.15.42:0/33369500 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40100116d0 con 0x7f4018108740 2024-07-26T06:47:43.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.338+0000 7f402016d640 1 -- 172.21.15.42:0/33369500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4018103730 con 0x7f4018108740 2024-07-26T06:47:43.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.340+0000 7f400effd640 1 -- 172.21.15.42:0/33369500 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f401000caf0 con 0x7f4018108740 2024-07-26T06:47:43.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.341+0000 7f400effd640 1 --2- 172.21.15.42:0/33369500 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f40040b3c60 0x7f40040b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:43.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.341+0000 7f401df09640 1 --2- 172.21.15.42:0/33369500 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f40040b3c60 0x7f40040b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:43.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.341+0000 7f400effd640 1 -- 172.21.15.42:0/33369500 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f40100e4330 con 0x7f4018108740 2024-07-26T06:47:43.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.342+0000 7f401df09640 1 --2- 172.21.15.42:0/33369500 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f40040b3c60 0x7f40040b6170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4014000c00 tx=0x7f4014031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:43.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.344+0000 7f400effd640 1 -- 172.21.15.42:0/33369500 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4010016030 con 0x7f4018108740 2024-07-26T06:47:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.563+0000 7f402016d640 1 -- 172.21.15.42:0/33369500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40181028c0 con 0x7f4018108740 2024-07-26T06:47:43.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.564+0000 7f400effd640 1 -- 172.21.15.42:0/33369500 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11978 (secure 0 0 0) 0x7f40100ab7a0 con 0x7f4018108740 2024-07-26T06:47:43.564 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:47:43.565 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":28,"btime":"2024-07-26T06:47:26:107129+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:47:25.125103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24578,24583,24593,24601,24620]},"id":1}]} 2024-07-26T06:47:43.565 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 28 2024-07-26T06:47:43.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.566+0000 7f402016d640 1 -- 172.21.15.42:0/33369500 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f40040b3c60 msgr2=0x7f40040b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:43.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.566+0000 7f402016d640 1 --2- 172.21.15.42:0/33369500 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f40040b3c60 0x7f40040b6170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4014000c00 tx=0x7f4014031400 comp rx=0 tx=0).stop 2024-07-26T06:47:43.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.566+0000 7f402016d640 1 -- 172.21.15.42:0/33369500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4018108740 msgr2=0x7f4018104720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:43.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.566+0000 7f402016d640 1 --2- 172.21.15.42:0/33369500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4018108740 0x7f4018104720 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f401000eaa0 tx=0x7f401000eea0 comp rx=0 tx=0).stop 2024-07-26T06:47:43.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.566+0000 7f402016d640 1 -- 172.21.15.42:0/33369500 shutdown_connections 2024-07-26T06:47:43.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.566+0000 7f402016d640 1 --2- 172.21.15.42:0/33369500 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f40040b3c60 0x7f40040b6170 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:43.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.566+0000 7f402016d640 1 --2- 172.21.15.42:0/33369500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4018108740 0x7f4018104720 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:43.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.566+0000 7f402016d640 1 --2- 172.21.15.42:0/33369500 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4018107df0 0x7f40181041e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:43.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.566+0000 7f402016d640 1 --2- 172.21.15.42:0/33369500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40181024b0 0x7f4018103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:43.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.566+0000 7f402016d640 1 -- 172.21.15.42:0/33369500 >> 172.21.15.42:0/33369500 conn(0x7f4018079170 msgr2=0x7f40181014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:43.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.566+0000 7f402016d640 1 -- 172.21.15.42:0/33369500 shutdown_connections 2024-07-26T06:47:43.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:43.566+0000 7f402016d640 1 -- 172.21.15.42:0/33369500 wait complete. 2024-07-26T06:47:44.321 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T06:47:44.592 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:44 smithi042 ceph-mon[32775]: pgmap v321: 97 pgs: 97 active+clean; 625 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.3 KiB/s wr, 12 op/s 2024-07-26T06:47:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:44 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/33369500' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:47:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:44 smithi164 ceph-mon[36970]: pgmap v321: 97 pgs: 97 active+clean; 625 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.3 KiB/s wr, 12 op/s 2024-07-26T06:47:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:44 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/33369500' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:47:44.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:44 smithi163 ceph-mon[36873]: pgmap v321: 97 pgs: 97 active+clean; 625 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.3 KiB/s wr, 12 op/s 2024-07-26T06:47:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:44 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/33369500' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:47:46.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.410+0000 7f659d7a8640 1 -- 172.21.15.42:0/1148708223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65981035b0 msgr2=0x7f65981039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:46.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.410+0000 7f659d7a8640 1 --2- 172.21.15.42:0/1148708223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65981035b0 0x7f65981039c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6588009ca0 tx=0x7f658802a8c0 comp rx=0 tx=0).stop 2024-07-26T06:47:46.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.410+0000 7f659d7a8640 1 -- 172.21.15.42:0/1148708223 shutdown_connections 2024-07-26T06:47:46.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.410+0000 7f659d7a8640 1 --2- 172.21.15.42:0/1148708223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6598109840 0x7f6598110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:46.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.410+0000 7f659d7a8640 1 --2- 172.21.15.42:0/1148708223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6598108ef0 0x7f6598109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:46.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.410+0000 7f659d7a8640 1 --2- 172.21.15.42:0/1148708223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65981035b0 0x7f65981039c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:46.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.410+0000 7f659d7a8640 1 -- 172.21.15.42:0/1148708223 >> 172.21.15.42:0/1148708223 conn(0x7f65980ff040 msgr2=0x7f6598101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:46.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.410+0000 7f659d7a8640 1 -- 172.21.15.42:0/1148708223 shutdown_connections 2024-07-26T06:47:46.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.410+0000 7f659d7a8640 1 -- 172.21.15.42:0/1148708223 wait complete. 2024-07-26T06:47:46.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.411+0000 7f659d7a8640 1 Processor -- start 2024-07-26T06:47:46.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.411+0000 7f659d7a8640 1 -- start start 2024-07-26T06:47:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.411+0000 7f659d7a8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65981035b0 0x7f6598199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.411+0000 7f659d7a8640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6598108ef0 0x7f659819a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.411+0000 7f659d7a8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6598109840 0x7f659819f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.411+0000 7f659d7a8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6598113510 con 0x7f6598109840 2024-07-26T06:47:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.411+0000 7f659d7a8640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6598113390 con 0x7f6598108ef0 2024-07-26T06:47:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.411+0000 7f659d7a8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6598113690 con 0x7f65981035b0 2024-07-26T06:47:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.411+0000 7f65977fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6598109840 0x7f659819f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.412+0000 7f65977fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6598109840 0x7f659819f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54582/0 (socket says 172.21.15.42:54582) 2024-07-26T06:47:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.412+0000 7f65977fe640 1 -- 172.21.15.42:0/218207330 learned_addr learned my addr 172.21.15.42:0/218207330 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.412+0000 7f65967fc640 1 --2- 172.21.15.42:0/218207330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6598108ef0 0x7f659819a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:46.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.412+0000 7f6596ffd640 1 --2- 172.21.15.42:0/218207330 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65981035b0 0x7f6598199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.412+0000 7f65977fe640 1 -- 172.21.15.42:0/218207330 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65981035b0 msgr2=0x7f6598199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.412+0000 7f65977fe640 1 --2- 172.21.15.42:0/218207330 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65981035b0 0x7f6598199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.412+0000 7f65977fe640 1 -- 172.21.15.42:0/218207330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6598108ef0 msgr2=0x7f659819a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.412+0000 7f65977fe640 1 --2- 172.21.15.42:0/218207330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6598108ef0 0x7f659819a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.412+0000 7f65977fe640 1 -- 172.21.15.42:0/218207330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f659819fe70 con 0x7f6598109840 2024-07-26T06:47:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.412+0000 7f6596ffd640 1 --2- 172.21.15.42:0/218207330 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65981035b0 0x7f6598199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:47:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.412+0000 7f65977fe640 1 --2- 172.21.15.42:0/218207330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6598109840 0x7f659819f740 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f659000d750 tx=0x7f659000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.412+0000 7f657ffff640 1 -- 172.21.15.42:0/218207330 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6590017070 con 0x7f6598109840 2024-07-26T06:47:46.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.412+0000 7f657ffff640 1 -- 172.21.15.42:0/218207330 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65900054d0 con 0x7f6598109840 2024-07-26T06:47:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.412+0000 7f659d7a8640 1 -- 172.21.15.42:0/218207330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65981a0160 con 0x7f6598109840 2024-07-26T06:47:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.412+0000 7f657ffff640 1 -- 172.21.15.42:0/218207330 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6590004c60 con 0x7f6598109840 2024-07-26T06:47:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.412+0000 7f659d7a8640 1 -- 172.21.15.42:0/218207330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65981a68c0 con 0x7f6598109840 2024-07-26T06:47:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.414+0000 7f659d7a8640 1 -- 172.21.15.42:0/218207330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65981039c0 con 0x7f6598109840 2024-07-26T06:47:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.415+0000 7f657ffff640 1 -- 172.21.15.42:0/218207330 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f6590009bd0 con 0x7f6598109840 2024-07-26T06:47:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.416+0000 7f657ffff640 1 --2- 172.21.15.42:0/218207330 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f65800b3ce0 0x7f65800b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.416+0000 7f6596ffd640 1 --2- 172.21.15.42:0/218207330 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f65800b3ce0 0x7f65800b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.416+0000 7f657ffff640 1 -- 172.21.15.42:0/218207330 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f65900e3730 con 0x7f6598109840 2024-07-26T06:47:46.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.416+0000 7f6596ffd640 1 --2- 172.21.15.42:0/218207330 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f65800b3ce0 0x7f65800b61f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6588006010 tx=0x7f6588000f40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:46.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.420+0000 7f657ffff640 1 -- 172.21.15.42:0/218207330 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f65900aac00 con 0x7f6598109840 2024-07-26T06:47:46.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.605+0000 7f659d7a8640 1 -- 172.21.15.42:0/218207330 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f659802e0c0 con 0x7f6598109840 2024-07-26T06:47:46.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.606+0000 7f657ffff640 1 -- 172.21.15.42:0/218207330 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22697 (secure 0 0 0) 0x7f65900aa500 con 0x7f6598109840 2024-07-26T06:47:46.607 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:47:46.607 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":80,"fsid":"8b3e2750-4b19-11ef-bcb6-c7b262605968","created":"2024-07-26T06:38:29.306470+0000","modified":"2024-07-26T06:47:23.086609+0000","last_up_change":"2024-07-26T06:46:02.149453+0000","last_in_change":"2024-07-26T06:45:43.027764+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:42:27.302195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:47:00.042262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T06:47:03.674160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bda88e0-09ee-447b-8c04-24f4e5d55d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6803","nonce":3587066379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6805","nonce":3587066379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6809","nonce":3587066379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6807","nonce":3587066379}]},"public_addr":"172.21.15.42:6803/3587066379","cluster_addr":"172.21.15.42:6805/3587066379","heartbeat_back_addr":"172.21.15.42:6809/3587066379","heartbeat_front_addr":"172.21.15.42:6807/3587066379","state":["exists","up"]},{"osd":1,"uuid":"7d747138-8217-4836-a173-cf8ac65e9fe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6801","nonce":360219078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6803","nonce":360219078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6807","nonce":360219078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6805","nonce":360219078}]},"public_addr":"172.21.15.163:6801/360219078","cluster_addr":"172.21.15.163:6803/360219078","heartbeat_back_addr":"172.21.15.163:6807/360219078","heartbeat_front_addr":"172.21.15.163:6805/360219078","state":["exists","up"]},{"osd":2,"uuid":"4b7258c1-3d35-4930-ab10-0b04a2a2c8a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6801","nonce":336113412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6803","nonce":336113412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6807","nonce":336113412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6805","nonce":336113412}]},"public_addr":"172.21.15.164:6801/336113412","cluster_addr":"172.21.15.164:6803/336113412","heartbeat_back_addr":"172.21.15.164:6807/336113412","heartbeat_front_addr":"172.21.15.164:6805/336113412","state":["exists","up"]},{"osd":3,"uuid":"96608387-0de0-4d90-a952-d6eba61f7fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6811","nonce":2819018064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6813","nonce":2819018064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6817","nonce":2819018064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6815","nonce":2819018064}]},"public_addr":"172.21.15.42:6811/2819018064","cluster_addr":"172.21.15.42:6813/2819018064","heartbeat_back_addr":"172.21.15.42:6817/2819018064","heartbeat_front_addr":"172.21.15.42:6815/2819018064","state":["exists","up"]},{"osd":4,"uuid":"fabf6b02-fe34-4560-85b5-01bf37fc9be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6809","nonce":431934374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6811","nonce":431934374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6815","nonce":431934374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6813","nonce":431934374}]},"public_addr":"172.21.15.163:6809/431934374","cluster_addr":"172.21.15.163:6811/431934374","heartbeat_back_addr":"172.21.15.163:6815/431934374","heartbeat_front_addr":"172.21.15.163:6813/431934374","state":["exists","up"]},{"osd":5,"uuid":"e226fb4f-15b2-40df-af34-900040ba93ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6809","nonce":261549362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6811","nonce":261549362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6815","nonce":261549362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6813","nonce":261549362}]},"public_addr":"172.21.15.164:6809/261549362","cluster_addr":"172.21.15.164:6811/261549362","heartbeat_back_addr":"172.21.15.164:6815/261549362","heartbeat_front_addr":"172.21.15.164:6813/261549362","state":["exists","up"]},{"osd":6,"uuid":"beb8ec5d-7ea5-4e4e-a46c-a84f7768cec0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6819","nonce":2160811300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6821","nonce":2160811300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6825","nonce":2160811300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6823","nonce":2160811300}]},"public_addr":"172.21.15.42:6819/2160811300","cluster_addr":"172.21.15.42:6821/2160811300","heartbeat_back_addr":"172.21.15.42:6825/2160811300","heartbeat_front_addr":"172.21.15.42:6823/2160811300","state":["exists","up"]},{"osd":7,"uuid":"f699004c-f7c2-4f1c-ae4b-7178f3b06e2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6817","nonce":812214111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6819","nonce":812214111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6823","nonce":812214111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6821","nonce":812214111}]},"public_addr":"172.21.15.163:6817/812214111","cluster_addr":"172.21.15.163:6819/812214111","heartbeat_back_addr":"172.21.15.163:6823/812214111","heartbeat_front_addr":"172.21.15.163:6821/812214111","state":["exists","up"]},{"osd":8,"uuid":"3476163e-efea-4652-a8d4-b7c4a908f984","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6817","nonce":962845839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6819","nonce":962845839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6823","nonce":962845839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6821","nonce":962845839}]},"public_addr":"172.21.15.164:6817/962845839","cluster_addr":"172.21.15.164:6819/962845839","heartbeat_back_addr":"172.21.15.164:6823/962845839","heartbeat_front_addr":"172.21.15.164:6821/962845839","state":["exists","up"]},{"osd":9,"uuid":"6f7ceac6-fb61-4c6f-9da8-851b937c4586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6827","nonce":994024418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6829","nonce":994024418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6833","nonce":994024418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6831","nonce":994024418}]},"public_addr":"172.21.15.42:6827/994024418","cluster_addr":"172.21.15.42:6829/994024418","heartbeat_back_addr":"172.21.15.42:6833/994024418","heartbeat_front_addr":"172.21.15.42:6831/994024418","state":["exists","up"]},{"osd":10,"uuid":"553132c2-a6a1-410c-a7d8-ecdf9a67dafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6825","nonce":3268512227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6827","nonce":3268512227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6831","nonce":3268512227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6829","nonce":3268512227}]},"public_addr":"172.21.15.163:6825/3268512227","cluster_addr":"172.21.15.163:6827/3268512227","heartbeat_back_addr":"172.21.15.163:6831/3268512227","heartbeat_front_addr":"172.21.15.163:6829/3268512227","state":["exists","up"]},{"osd":11,"uuid":"09abc8e9-7884-400f-ab6d-8e5aa7bf31ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6825","nonce":1861076336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6827","nonce":1861076336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6831","nonce":1861076336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6829","nonce":1861076336}]},"public_addr":"172.21.15.164:6825/1861076336","cluster_addr":"172.21.15.164:6827/1861076336","heartbeat_back_addr":"172.21.15.164:6831/1861076336","heartbeat_front_addr":"172.21.15.164:6829/1861076336","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:41:38.761674+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:00.621745+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:23.248620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:47.232764+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:10.379922+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:34.502519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:58.694642+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:22.350247+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:46.988007+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:11.390474+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:36.150054+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:46:00.474470+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.42:6801/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6800/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1842521399":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/834172950":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1730109419":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6801/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:6800/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2687772889":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/3371433307":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2861719816":"2024-07-27T06:38:56.887264+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:47:46.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.608+0000 7f659d7a8640 1 -- 172.21.15.42:0/218207330 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f65800b3ce0 msgr2=0x7f65800b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:46.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.608+0000 7f659d7a8640 1 --2- 172.21.15.42:0/218207330 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f65800b3ce0 0x7f65800b61f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6588006010 tx=0x7f6588000f40 comp rx=0 tx=0).stop 2024-07-26T06:47:46.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.608+0000 7f659d7a8640 1 -- 172.21.15.42:0/218207330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6598109840 msgr2=0x7f659819f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:46.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.608+0000 7f659d7a8640 1 --2- 172.21.15.42:0/218207330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6598109840 0x7f659819f740 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f659000d750 tx=0x7f659000dc10 comp rx=0 tx=0).stop 2024-07-26T06:47:46.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.609+0000 7f659d7a8640 1 -- 172.21.15.42:0/218207330 shutdown_connections 2024-07-26T06:47:46.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.609+0000 7f659d7a8640 1 --2- 172.21.15.42:0/218207330 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f65800b3ce0 0x7f65800b61f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:46.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.609+0000 7f659d7a8640 1 --2- 172.21.15.42:0/218207330 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6598109840 0x7f659819f740 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:46.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.609+0000 7f659d7a8640 1 --2- 172.21.15.42:0/218207330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6598108ef0 0x7f659819a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:46.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.609+0000 7f659d7a8640 1 --2- 172.21.15.42:0/218207330 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65981035b0 0x7f6598199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:46.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.609+0000 7f659d7a8640 1 -- 172.21.15.42:0/218207330 >> 172.21.15.42:0/218207330 conn(0x7f65980ff040 msgr2=0x7f6598101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:46.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.609+0000 7f659d7a8640 1 -- 172.21.15.42:0/218207330 shutdown_connections 2024-07-26T06:47:46.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:46.609+0000 7f659d7a8640 1 -- 172.21.15.42:0/218207330 wait complete. 2024-07-26T06:47:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:46 smithi164 ceph-mon[36970]: pgmap v322: 97 pgs: 97 active+clean; 637 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.3 KiB/s wr, 13 op/s 2024-07-26T06:47:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:46 smithi163 ceph-mon[36873]: pgmap v322: 97 pgs: 97 active+clean; 637 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.3 KiB/s wr, 13 op/s 2024-07-26T06:47:46.757 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:46 smithi042 ceph-mon[32775]: pgmap v322: 97 pgs: 97 active+clean; 637 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.3 KiB/s wr, 13 op/s 2024-07-26T06:47:47.347 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-07-26T06:47:47.348 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph pg dump --format=json 2024-07-26T06:47:47.615 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:47.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:47 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/218207330' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:47:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:47 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/218207330' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:47:47.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:47 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/218207330' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:47:48.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:48 smithi164 ceph-mon[36970]: pgmap v323: 97 pgs: 97 active+clean; 637 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-07-26T06:47:48.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:48 smithi163 ceph-mon[36873]: pgmap v323: 97 pgs: 97 active+clean; 637 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-07-26T06:47:48.819 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:48 smithi042 ceph-mon[32775]: pgmap v323: 97 pgs: 97 active+clean; 637 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-07-26T06:47:49.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.335+0000 7f60ab66e640 1 -- 172.21.15.42:0/3418176516 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60a4107df0 msgr2=0x7f60a4108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:49.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.335+0000 7f60ab66e640 1 --2- 172.21.15.42:0/3418176516 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60a4107df0 0x7f60a4108200 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6098009900 tx=0x7f609802a410 comp rx=0 tx=0).stop 2024-07-26T06:47:49.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.335+0000 7f60ab66e640 1 -- 172.21.15.42:0/3418176516 shutdown_connections 2024-07-26T06:47:49.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.335+0000 7f60ab66e640 1 --2- 172.21.15.42:0/3418176516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60a4108740 0x7f60a410ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:49.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.335+0000 7f60ab66e640 1 --2- 172.21.15.42:0/3418176516 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60a4107df0 0x7f60a4108200 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:49.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.335+0000 7f60ab66e640 1 --2- 172.21.15.42:0/3418176516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60a41024b0 0x7f60a41028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:49.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.335+0000 7f60ab66e640 1 -- 172.21.15.42:0/3418176516 >> 172.21.15.42:0/3418176516 conn(0x7f60a4079170 msgr2=0x7f60a4079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:49.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.335+0000 7f60ab66e640 1 -- 172.21.15.42:0/3418176516 shutdown_connections 2024-07-26T06:47:49.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.335+0000 7f60ab66e640 1 -- 172.21.15.42:0/3418176516 wait complete. 2024-07-26T06:47:49.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60ab66e640 1 Processor -- start 2024-07-26T06:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60ab66e640 1 -- start start 2024-07-26T06:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60ab66e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60a41024b0 0x7f60a419ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60ab66e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60a4107df0 0x7f60a419b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60ab66e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60a4108740 0x7f60a419b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60ab66e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f60a4112370 con 0x7f60a41024b0 2024-07-26T06:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60ab66e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f60a41121f0 con 0x7f60a4107df0 2024-07-26T06:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60ab66e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f60a41124f0 con 0x7f60a4108740 2024-07-26T06:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60a940a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60a41024b0 0x7f60a419ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60a940a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60a41024b0 0x7f60a419ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56460/0 (socket says 172.21.15.42:56460) 2024-07-26T06:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60a940a640 1 -- 172.21.15.42:0/1437213874 learned_addr learned my addr 172.21.15.42:0/1437213874 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60a8c09640 1 --2- 172.21.15.42:0/1437213874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60a4107df0 0x7f60a419b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60a9c0b640 1 --2- 172.21.15.42:0/1437213874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60a4108740 0x7f60a419b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60a940a640 1 -- 172.21.15.42:0/1437213874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60a4108740 msgr2=0x7f60a419b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60a940a640 1 --2- 172.21.15.42:0/1437213874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60a4108740 0x7f60a419b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60a940a640 1 -- 172.21.15.42:0/1437213874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60a4107df0 msgr2=0x7f60a419b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60a940a640 1 --2- 172.21.15.42:0/1437213874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60a4107df0 0x7f60a419b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.336+0000 7f60a940a640 1 -- 172.21.15.42:0/1437213874 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60a41a60b0 con 0x7f60a41024b0 2024-07-26T06:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.337+0000 7f60a940a640 1 --2- 172.21.15.42:0/1437213874 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60a41024b0 0x7f60a419ae20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f609c00c890 tx=0x7f609c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.337+0000 7f60a27fc640 1 -- 172.21.15.42:0/1437213874 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c004220 con 0x7f60a41024b0 2024-07-26T06:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.337+0000 7f60ab66e640 1 -- 172.21.15.42:0/1437213874 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60a41a63a0 con 0x7f60a41024b0 2024-07-26T06:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.337+0000 7f60a27fc640 1 -- 172.21.15.42:0/1437213874 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f609c0043c0 con 0x7f60a41024b0 2024-07-26T06:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.337+0000 7f60a27fc640 1 -- 172.21.15.42:0/1437213874 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c004a00 con 0x7f60a41024b0 2024-07-26T06:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.337+0000 7f60ab66e640 1 -- 172.21.15.42:0/1437213874 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60a41a68b0 con 0x7f60a41024b0 2024-07-26T06:47:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.338+0000 7f60ab66e640 1 -- 172.21.15.42:0/1437213874 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60a4103730 con 0x7f60a41024b0 2024-07-26T06:47:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.340+0000 7f60a27fc640 1 -- 172.21.15.42:0/1437213874 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f609c0079b0 con 0x7f60a41024b0 2024-07-26T06:47:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.340+0000 7f60a27fc640 1 --2- 172.21.15.42:0/1437213874 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f608c0b3c90 0x7f608c0b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.340+0000 7f60a8c09640 1 --2- 172.21.15.42:0/1437213874 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f608c0b3c90 0x7f608c0b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.341+0000 7f60a27fc640 1 -- 172.21.15.42:0/1437213874 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f609c0e3ef0 con 0x7f60a41024b0 2024-07-26T06:47:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.341+0000 7f60a8c09640 1 --2- 172.21.15.42:0/1437213874 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f608c0b3c90 0x7f608c0b61a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6098000c00 tx=0x7f6098031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:49.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.343+0000 7f60a27fc640 1 -- 172.21.15.42:0/1437213874 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f609c014030 con 0x7f60a41024b0 2024-07-26T06:47:49.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.527+0000 7f60ab66e640 1 -- 172.21.15.42:0/1437213874 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f60a4063230 con 0x7f608c0b3c90 2024-07-26T06:47:49.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.532+0000 7f60a27fc640 1 -- 172.21.15.42:0/1437213874 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258989 (secure 0 0 0) 0x7f60a4063230 con 0x7f608c0b3c90 2024-07-26T06:47:49.536 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:47:49.538 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-07-26T06:47:49.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.535+0000 7f60ab66e640 1 -- 172.21.15.42:0/1437213874 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f608c0b3c90 msgr2=0x7f608c0b61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:49.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.535+0000 7f60ab66e640 1 --2- 172.21.15.42:0/1437213874 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f608c0b3c90 0x7f608c0b61a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6098000c00 tx=0x7f6098031400 comp rx=0 tx=0).stop 2024-07-26T06:47:49.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.535+0000 7f60ab66e640 1 -- 172.21.15.42:0/1437213874 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60a41024b0 msgr2=0x7f60a419ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:49.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.535+0000 7f60ab66e640 1 --2- 172.21.15.42:0/1437213874 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60a41024b0 0x7f60a419ae20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f609c00c890 tx=0x7f609c00cd50 comp rx=0 tx=0).stop 2024-07-26T06:47:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.535+0000 7f60ab66e640 1 -- 172.21.15.42:0/1437213874 shutdown_connections 2024-07-26T06:47:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.535+0000 7f60ab66e640 1 --2- 172.21.15.42:0/1437213874 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f608c0b3c90 0x7f608c0b61a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.535+0000 7f60ab66e640 1 --2- 172.21.15.42:0/1437213874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60a4108740 0x7f60a419b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.535+0000 7f60ab66e640 1 --2- 172.21.15.42:0/1437213874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60a4107df0 0x7f60a419b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.535+0000 7f60ab66e640 1 --2- 172.21.15.42:0/1437213874 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60a41024b0 0x7f60a419ae20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.535+0000 7f60ab66e640 1 -- 172.21.15.42:0/1437213874 >> 172.21.15.42:0/1437213874 conn(0x7f60a4079170 msgr2=0x7f60a4103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.535+0000 7f60ab66e640 1 -- 172.21.15.42:0/1437213874 shutdown_connections 2024-07-26T06:47:49.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:49.535+0000 7f60ab66e640 1 -- 172.21.15.42:0/1437213874 wait complete. 2024-07-26T06:47:50.182 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-07-26T06:47:49.326076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653387,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":360,"num_read_kb":276,"num_write":367,"num_write_kb":2281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2953474,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380760,"kb_used_data":10788,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124561064,"statfs":{"total":1151940427776,"available":1151550529536,"internally_reserved":0,"allocated":11046912,"data_stored":7052193,"data_compressed":177858,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33368,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":104,"num_write":24,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003830"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096639+0000","last_change":"2024-07-26T06:47:05.026186+0000","last_active":"2024-07-26T06:47:23.096639+0000","last_peered":"2024-07-26T06:47:23.096639+0000","last_clean":"2024-07-26T06:47:23.096639+0000","last_became_active":"2024-07-26T06:47:05.025589+0000","last_became_peered":"2024-07-26T06:47:05.025589+0000","last_unstale":"2024-07-26T06:47:23.096639+0000","last_undegraded":"2024-07-26T06:47:23.096639+0000","last_fullsized":"2024-07-26T06:47:23.096639+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:48:23.100966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093146+0000","last_change":"2024-07-26T06:47:05.033211+0000","last_active":"2024-07-26T06:47:23.093146+0000","last_peered":"2024-07-26T06:47:23.093146+0000","last_clean":"2024-07-26T06:47:23.093146+0000","last_became_active":"2024-07-26T06:47:05.032630+0000","last_became_peered":"2024-07-26T06:47:05.032630+0000","last_unstale":"2024-07-26T06:47:23.093146+0000","last_undegraded":"2024-07-26T06:47:23.093146+0000","last_fullsized":"2024-07-26T06:47:23.093146+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:03:19.612963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092844+0000","last_change":"2024-07-26T06:47:05.034767+0000","last_active":"2024-07-26T06:47:23.092844+0000","last_peered":"2024-07-26T06:47:23.092844+0000","last_clean":"2024-07-26T06:47:23.092844+0000","last_became_active":"2024-07-26T06:47:05.034376+0000","last_became_peered":"2024-07-26T06:47:05.034376+0000","last_unstale":"2024-07-26T06:47:23.092844+0000","last_undegraded":"2024-07-26T06:47:23.092844+0000","last_fullsized":"2024-07-26T06:47:23.092844+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:55:55.729930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.380468+0000","last_change":"2024-07-26T06:47:05.030995+0000","last_active":"2024-07-26T06:47:23.380468+0000","last_peered":"2024-07-26T06:47:23.380468+0000","last_clean":"2024-07-26T06:47:23.380468+0000","last_became_active":"2024-07-26T06:47:05.025423+0000","last_became_peered":"2024-07-26T06:47:05.025423+0000","last_unstale":"2024-07-26T06:47:23.380468+0000","last_undegraded":"2024-07-26T06:47:23.380468+0000","last_fullsized":"2024-07-26T06:47:23.380468+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:59:39.511383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092282+0000","last_change":"2024-07-26T06:47:05.035359+0000","last_active":"2024-07-26T06:47:23.092282+0000","last_peered":"2024-07-26T06:47:23.092282+0000","last_clean":"2024-07-26T06:47:23.092282+0000","last_became_active":"2024-07-26T06:47:05.034836+0000","last_became_peered":"2024-07-26T06:47:05.034836+0000","last_unstale":"2024-07-26T06:47:23.092282+0000","last_undegraded":"2024-07-26T06:47:23.092282+0000","last_fullsized":"2024-07-26T06:47:23.092282+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:54.482774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093596+0000","last_change":"2024-07-26T06:47:05.023660+0000","last_active":"2024-07-26T06:47:23.093596+0000","last_peered":"2024-07-26T06:47:23.093596+0000","last_clean":"2024-07-26T06:47:23.093596+0000","last_became_active":"2024-07-26T06:47:05.022446+0000","last_became_peered":"2024-07-26T06:47:05.022446+0000","last_unstale":"2024-07-26T06:47:23.093596+0000","last_undegraded":"2024-07-26T06:47:23.093596+0000","last_fullsized":"2024-07-26T06:47:23.093596+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:35:00.519226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097130+0000","last_change":"2024-07-26T06:47:05.025132+0000","last_active":"2024-07-26T06:47:23.097130+0000","last_peered":"2024-07-26T06:47:23.097130+0000","last_clean":"2024-07-26T06:47:23.097130+0000","last_became_active":"2024-07-26T06:47:05.024770+0000","last_became_peered":"2024-07-26T06:47:05.024770+0000","last_unstale":"2024-07-26T06:47:23.097130+0000","last_undegraded":"2024-07-26T06:47:23.097130+0000","last_fullsized":"2024-07-26T06:47:23.097130+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:21:39.920152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090188+0000","last_change":"2024-07-26T06:47:05.022613+0000","last_active":"2024-07-26T06:47:23.090188+0000","last_peered":"2024-07-26T06:47:23.090188+0000","last_clean":"2024-07-26T06:47:23.090188+0000","last_became_active":"2024-07-26T06:47:05.022202+0000","last_became_peered":"2024-07-26T06:47:05.022202+0000","last_unstale":"2024-07-26T06:47:23.090188+0000","last_undegraded":"2024-07-26T06:47:23.090188+0000","last_fullsized":"2024-07-26T06:47:23.090188+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:07:23.980843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091579+0000","last_change":"2024-07-26T06:47:05.026352+0000","last_active":"2024-07-26T06:47:23.091579+0000","last_peered":"2024-07-26T06:47:23.091579+0000","last_clean":"2024-07-26T06:47:23.091579+0000","last_became_active":"2024-07-26T06:47:05.025969+0000","last_became_peered":"2024-07-26T06:47:05.025969+0000","last_unstale":"2024-07-26T06:47:23.091579+0000","last_undegraded":"2024-07-26T06:47:23.091579+0000","last_fullsized":"2024-07-26T06:47:23.091579+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:27.062013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.100482+0000","last_change":"2024-07-26T06:47:23.100422+0000","last_active":"2024-07-26T06:47:23.100482+0000","last_peered":"2024-07-26T06:47:23.100482+0000","last_clean":"2024-07-26T06:47:23.100482+0000","last_became_active":"2024-07-26T06:47:23.100023+0000","last_became_peered":"2024-07-26T06:47:23.100023+0000","last_unstale":"2024-07-26T06:47:23.100482+0000","last_undegraded":"2024-07-26T06:47:23.100482+0000","last_fullsized":"2024-07-26T06:47:23.100482+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:28.158112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097318+0000","last_change":"2024-07-26T06:47:05.037142+0000","last_active":"2024-07-26T06:47:23.097318+0000","last_peered":"2024-07-26T06:47:23.097318+0000","last_clean":"2024-07-26T06:47:23.097318+0000","last_became_active":"2024-07-26T06:47:05.036745+0000","last_became_peered":"2024-07-26T06:47:05.036745+0000","last_unstale":"2024-07-26T06:47:23.097318+0000","last_undegraded":"2024-07-26T06:47:23.097318+0000","last_fullsized":"2024-07-26T06:47:23.097318+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:31:10.034560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.380519+0000","last_change":"2024-07-26T06:47:05.030640+0000","last_active":"2024-07-26T06:47:23.380519+0000","last_peered":"2024-07-26T06:47:23.380519+0000","last_clean":"2024-07-26T06:47:23.380519+0000","last_became_active":"2024-07-26T06:47:05.023583+0000","last_became_peered":"2024-07-26T06:47:05.023583+0000","last_unstale":"2024-07-26T06:47:23.380519+0000","last_undegraded":"2024-07-26T06:47:23.380519+0000","last_fullsized":"2024-07-26T06:47:23.380519+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:59:12.312930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095476+0000","last_change":"2024-07-26T06:47:05.021768+0000","last_active":"2024-07-26T06:47:23.095476+0000","last_peered":"2024-07-26T06:47:23.095476+0000","last_clean":"2024-07-26T06:47:23.095476+0000","last_became_active":"2024-07-26T06:47:05.021410+0000","last_became_peered":"2024-07-26T06:47:05.021410+0000","last_unstale":"2024-07-26T06:47:23.095476+0000","last_undegraded":"2024-07-26T06:47:23.095476+0000","last_fullsized":"2024-07-26T06:47:23.095476+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:05.576122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092848+0000","last_change":"2024-07-26T06:47:05.035814+0000","last_active":"2024-07-26T06:47:23.092848+0000","last_peered":"2024-07-26T06:47:23.092848+0000","last_clean":"2024-07-26T06:47:23.092848+0000","last_became_active":"2024-07-26T06:47:05.035451+0000","last_became_peered":"2024-07-26T06:47:05.035451+0000","last_unstale":"2024-07-26T06:47:23.092848+0000","last_undegraded":"2024-07-26T06:47:23.092848+0000","last_fullsized":"2024-07-26T06:47:23.092848+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:33.533726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093301+0000","last_change":"2024-07-26T06:47:05.035720+0000","last_active":"2024-07-26T06:47:23.093301+0000","last_peered":"2024-07-26T06:47:23.093301+0000","last_clean":"2024-07-26T06:47:23.093301+0000","last_became_active":"2024-07-26T06:47:05.035359+0000","last_became_peered":"2024-07-26T06:47:05.035359+0000","last_unstale":"2024-07-26T06:47:23.093301+0000","last_undegraded":"2024-07-26T06:47:23.093301+0000","last_fullsized":"2024-07-26T06:47:23.093301+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:38.561268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090922+0000","last_change":"2024-07-26T06:47:05.027109+0000","last_active":"2024-07-26T06:47:23.090922+0000","last_peered":"2024-07-26T06:47:23.090922+0000","last_clean":"2024-07-26T06:47:23.090922+0000","last_became_active":"2024-07-26T06:47:05.026694+0000","last_became_peered":"2024-07-26T06:47:05.026694+0000","last_unstale":"2024-07-26T06:47:23.090922+0000","last_undegraded":"2024-07-26T06:47:23.090922+0000","last_fullsized":"2024-07-26T06:47:23.090922+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:25:01.215646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096756+0000","last_change":"2024-07-26T06:47:05.028212+0000","last_active":"2024-07-26T06:47:23.096756+0000","last_peered":"2024-07-26T06:47:23.096756+0000","last_clean":"2024-07-26T06:47:23.096756+0000","last_became_active":"2024-07-26T06:47:05.027598+0000","last_became_peered":"2024-07-26T06:47:05.027598+0000","last_unstale":"2024-07-26T06:47:23.096756+0000","last_undegraded":"2024-07-26T06:47:23.096756+0000","last_fullsized":"2024-07-26T06:47:23.096756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:55:31.536194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092258+0000","last_change":"2024-07-26T06:47:05.027724+0000","last_active":"2024-07-26T06:47:23.092258+0000","last_peered":"2024-07-26T06:47:23.092258+0000","last_clean":"2024-07-26T06:47:23.092258+0000","last_became_active":"2024-07-26T06:47:05.027348+0000","last_became_peered":"2024-07-26T06:47:05.027348+0000","last_unstale":"2024-07-26T06:47:23.092258+0000","last_undegraded":"2024-07-26T06:47:23.092258+0000","last_fullsized":"2024-07-26T06:47:23.092258+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:04.180467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097071+0000","last_change":"2024-07-26T06:47:05.025474+0000","last_active":"2024-07-26T06:47:23.097071+0000","last_peered":"2024-07-26T06:47:23.097071+0000","last_clean":"2024-07-26T06:47:23.097071+0000","last_became_active":"2024-07-26T06:47:05.025016+0000","last_became_peered":"2024-07-26T06:47:05.025016+0000","last_unstale":"2024-07-26T06:47:23.097071+0000","last_undegraded":"2024-07-26T06:47:23.097071+0000","last_fullsized":"2024-07-26T06:47:23.097071+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:40:03.034532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095635+0000","last_change":"2024-07-26T06:47:05.027533+0000","last_active":"2024-07-26T06:47:23.095635+0000","last_peered":"2024-07-26T06:47:23.095635+0000","last_clean":"2024-07-26T06:47:23.095635+0000","last_became_active":"2024-07-26T06:47:05.027143+0000","last_became_peered":"2024-07-26T06:47:05.027143+0000","last_unstale":"2024-07-26T06:47:23.095635+0000","last_undegraded":"2024-07-26T06:47:23.095635+0000","last_fullsized":"2024-07-26T06:47:23.095635+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:13.406430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095577+0000","last_change":"2024-07-26T06:47:05.025962+0000","last_active":"2024-07-26T06:47:23.095577+0000","last_peered":"2024-07-26T06:47:23.095577+0000","last_clean":"2024-07-26T06:47:23.095577+0000","last_became_active":"2024-07-26T06:47:05.025621+0000","last_became_peered":"2024-07-26T06:47:05.025621+0000","last_unstale":"2024-07-26T06:47:23.095577+0000","last_undegraded":"2024-07-26T06:47:23.095577+0000","last_fullsized":"2024-07-26T06:47:23.095577+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:56:43.669247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091492+0000","last_change":"2024-07-26T06:47:05.026252+0000","last_active":"2024-07-26T06:47:23.091492+0000","last_peered":"2024-07-26T06:47:23.091492+0000","last_clean":"2024-07-26T06:47:23.091492+0000","last_became_active":"2024-07-26T06:47:05.025875+0000","last_became_peered":"2024-07-26T06:47:05.025875+0000","last_unstale":"2024-07-26T06:47:23.091492+0000","last_undegraded":"2024-07-26T06:47:23.091492+0000","last_fullsized":"2024-07-26T06:47:23.091492+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:27:32.955862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090380+0000","last_change":"2024-07-26T06:47:05.022513+0000","last_active":"2024-07-26T06:47:23.090380+0000","last_peered":"2024-07-26T06:47:23.090380+0000","last_clean":"2024-07-26T06:47:23.090380+0000","last_became_active":"2024-07-26T06:47:05.022159+0000","last_became_peered":"2024-07-26T06:47:05.022159+0000","last_unstale":"2024-07-26T06:47:23.090380+0000","last_undegraded":"2024-07-26T06:47:23.090380+0000","last_fullsized":"2024-07-26T06:47:23.090380+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:53:06.402985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.098283+0000","last_change":"2024-07-26T06:47:23.097604+0000","last_active":"2024-07-26T06:47:23.098283+0000","last_peered":"2024-07-26T06:47:23.098283+0000","last_clean":"2024-07-26T06:47:23.098283+0000","last_became_active":"2024-07-26T06:47:23.097170+0000","last_became_peered":"2024-07-26T06:47:23.097170+0000","last_unstale":"2024-07-26T06:47:23.098283+0000","last_undegraded":"2024-07-26T06:47:23.098283+0000","last_fullsized":"2024-07-26T06:47:23.098283+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:38:24.530396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090964+0000","last_change":"2024-07-26T06:47:05.026349+0000","last_active":"2024-07-26T06:47:23.090964+0000","last_peered":"2024-07-26T06:47:23.090964+0000","last_clean":"2024-07-26T06:47:23.090964+0000","last_became_active":"2024-07-26T06:47:05.026016+0000","last_became_peered":"2024-07-26T06:47:05.026016+0000","last_unstale":"2024-07-26T06:47:23.090964+0000","last_undegraded":"2024-07-26T06:47:23.090964+0000","last_fullsized":"2024-07-26T06:47:23.090964+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:19:02.264183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097162+0000","last_change":"2024-07-26T06:47:05.032439+0000","last_active":"2024-07-26T06:47:23.097162+0000","last_peered":"2024-07-26T06:47:23.097162+0000","last_clean":"2024-07-26T06:47:23.097162+0000","last_became_active":"2024-07-26T06:47:05.032105+0000","last_became_peered":"2024-07-26T06:47:05.032105+0000","last_unstale":"2024-07-26T06:47:23.097162+0000","last_undegraded":"2024-07-26T06:47:23.097162+0000","last_fullsized":"2024-07-26T06:47:23.097162+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:21:50.423939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096506+0000","last_change":"2024-07-26T06:47:05.026927+0000","last_active":"2024-07-26T06:47:23.096506+0000","last_peered":"2024-07-26T06:47:23.096506+0000","last_clean":"2024-07-26T06:47:23.096506+0000","last_became_active":"2024-07-26T06:47:05.026569+0000","last_became_peered":"2024-07-26T06:47:05.026569+0000","last_unstale":"2024-07-26T06:47:23.096506+0000","last_undegraded":"2024-07-26T06:47:23.096506+0000","last_fullsized":"2024-07-26T06:47:23.096506+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:12:14.687633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.099776+0000","last_change":"2024-07-26T06:47:23.099689+0000","last_active":"2024-07-26T06:47:23.099776+0000","last_peered":"2024-07-26T06:47:23.099776+0000","last_clean":"2024-07-26T06:47:23.099776+0000","last_became_active":"2024-07-26T06:47:23.099313+0000","last_became_peered":"2024-07-26T06:47:23.099313+0000","last_unstale":"2024-07-26T06:47:23.099776+0000","last_undegraded":"2024-07-26T06:47:23.099776+0000","last_fullsized":"2024-07-26T06:47:23.099776+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:49:01.833297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095052+0000","last_change":"2024-07-26T06:47:05.034864+0000","last_active":"2024-07-26T06:47:23.095052+0000","last_peered":"2024-07-26T06:47:23.095052+0000","last_clean":"2024-07-26T06:47:23.095052+0000","last_became_active":"2024-07-26T06:47:05.033414+0000","last_became_peered":"2024-07-26T06:47:05.033414+0000","last_unstale":"2024-07-26T06:47:23.095052+0000","last_undegraded":"2024-07-26T06:47:23.095052+0000","last_fullsized":"2024-07-26T06:47:23.095052+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:21:31.420212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096728+0000","last_change":"2024-07-26T06:47:05.025462+0000","last_active":"2024-07-26T06:47:23.096728+0000","last_peered":"2024-07-26T06:47:23.096728+0000","last_clean":"2024-07-26T06:47:23.096728+0000","last_became_active":"2024-07-26T06:47:05.025049+0000","last_became_peered":"2024-07-26T06:47:05.025049+0000","last_unstale":"2024-07-26T06:47:23.096728+0000","last_undegraded":"2024-07-26T06:47:23.096728+0000","last_fullsized":"2024-07-26T06:47:23.096728+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:59:28.903338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097665+0000","last_change":"2024-07-26T06:47:05.026394+0000","last_active":"2024-07-26T06:47:23.097665+0000","last_peered":"2024-07-26T06:47:23.097665+0000","last_clean":"2024-07-26T06:47:23.097665+0000","last_became_active":"2024-07-26T06:47:05.025820+0000","last_became_peered":"2024-07-26T06:47:05.025820+0000","last_unstale":"2024-07-26T06:47:23.097665+0000","last_undegraded":"2024-07-26T06:47:23.097665+0000","last_fullsized":"2024-07-26T06:47:23.097665+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:44:57.441638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095529+0000","last_change":"2024-07-26T06:47:05.024373+0000","last_active":"2024-07-26T06:47:23.095529+0000","last_peered":"2024-07-26T06:47:23.095529+0000","last_clean":"2024-07-26T06:47:23.095529+0000","last_became_active":"2024-07-26T06:47:05.023962+0000","last_became_peered":"2024-07-26T06:47:05.023962+0000","last_unstale":"2024-07-26T06:47:23.095529+0000","last_undegraded":"2024-07-26T06:47:23.095529+0000","last_fullsized":"2024-07-26T06:47:23.095529+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:43.804614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092705+0000","last_change":"2024-07-26T06:47:05.031146+0000","last_active":"2024-07-26T06:47:23.092705+0000","last_peered":"2024-07-26T06:47:23.092705+0000","last_clean":"2024-07-26T06:47:23.092705+0000","last_became_active":"2024-07-26T06:47:05.029393+0000","last_became_peered":"2024-07-26T06:47:05.029393+0000","last_unstale":"2024-07-26T06:47:23.092705+0000","last_undegraded":"2024-07-26T06:47:23.092705+0000","last_fullsized":"2024-07-26T06:47:23.092705+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:46.853933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"80'6","reported_seq":58,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:45.156905+0000","last_change":"2024-07-26T06:47:02.001800+0000","last_active":"2024-07-26T06:47:45.156905+0000","last_peered":"2024-07-26T06:47:45.156905+0000","last_clean":"2024-07-26T06:47:45.156905+0000","last_became_active":"2024-07-26T06:47:02.001435+0000","last_became_peered":"2024-07-26T06:47:02.001435+0000","last_unstale":"2024-07-26T06:47:45.156905+0000","last_undegraded":"2024-07-26T06:47:45.156905+0000","last_fullsized":"2024-07-26T06:47:45.156905+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:23.121378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.094853+0000","last_change":"2024-07-26T06:47:05.028301+0000","last_active":"2024-07-26T06:47:23.094853+0000","last_peered":"2024-07-26T06:47:23.094853+0000","last_clean":"2024-07-26T06:47:23.094853+0000","last_became_active":"2024-07-26T06:47:05.027761+0000","last_became_peered":"2024-07-26T06:47:05.027761+0000","last_unstale":"2024-07-26T06:47:23.094853+0000","last_undegraded":"2024-07-26T06:47:23.094853+0000","last_fullsized":"2024-07-26T06:47:23.094853+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:19.109395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093760+0000","last_change":"2024-07-26T06:47:02.004488+0000","last_active":"2024-07-26T06:47:23.093760+0000","last_peered":"2024-07-26T06:47:23.093760+0000","last_clean":"2024-07-26T06:47:23.093760+0000","last_became_active":"2024-07-26T06:47:02.004163+0000","last_became_peered":"2024-07-26T06:47:02.004163+0000","last_unstale":"2024-07-26T06:47:23.093760+0000","last_undegraded":"2024-07-26T06:47:23.093760+0000","last_fullsized":"2024-07-26T06:47:23.093760+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:16.221618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096069+0000","last_change":"2024-07-26T06:47:01.992708+0000","last_active":"2024-07-26T06:47:23.096069+0000","last_peered":"2024-07-26T06:47:23.096069+0000","last_clean":"2024-07-26T06:47:23.096069+0000","last_became_active":"2024-07-26T06:47:01.992330+0000","last_became_peered":"2024-07-26T06:47:01.992330+0000","last_unstale":"2024-07-26T06:47:23.096069+0000","last_undegraded":"2024-07-26T06:47:23.096069+0000","last_fullsized":"2024-07-26T06:47:23.096069+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:25.951946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090325+0000","last_change":"2024-07-26T06:47:05.013946+0000","last_active":"2024-07-26T06:47:23.090325+0000","last_peered":"2024-07-26T06:47:23.090325+0000","last_clean":"2024-07-26T06:47:23.090325+0000","last_became_active":"2024-07-26T06:47:05.013509+0000","last_became_peered":"2024-07-26T06:47:05.013509+0000","last_unstale":"2024-07-26T06:47:23.090325+0000","last_undegraded":"2024-07-26T06:47:23.090325+0000","last_fullsized":"2024-07-26T06:47:23.090325+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:09.789287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093848+0000","last_change":"2024-07-26T06:47:01.997617+0000","last_active":"2024-07-26T06:47:23.093848+0000","last_peered":"2024-07-26T06:47:23.093848+0000","last_clean":"2024-07-26T06:47:23.093848+0000","last_became_active":"2024-07-26T06:47:01.997238+0000","last_became_peered":"2024-07-26T06:47:01.997238+0000","last_unstale":"2024-07-26T06:47:23.093848+0000","last_undegraded":"2024-07-26T06:47:23.093848+0000","last_fullsized":"2024-07-26T06:47:23.093848+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:45.596438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.099897+0000","last_change":"2024-07-26T06:47:23.099813+0000","last_active":"2024-07-26T06:47:23.099897+0000","last_peered":"2024-07-26T06:47:23.099897+0000","last_clean":"2024-07-26T06:47:23.099897+0000","last_became_active":"2024-07-26T06:47:23.098701+0000","last_became_peered":"2024-07-26T06:47:23.098701+0000","last_unstale":"2024-07-26T06:47:23.099897+0000","last_undegraded":"2024-07-26T06:47:23.099897+0000","last_fullsized":"2024-07-26T06:47:23.099897+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:42:04.059637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"78'4","reported_seq":80,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:48.160574+0000","last_change":"2024-07-26T06:47:01.998508+0000","last_active":"2024-07-26T06:47:48.160574+0000","last_peered":"2024-07-26T06:47:48.160574+0000","last_clean":"2024-07-26T06:47:48.160574+0000","last_became_active":"2024-07-26T06:47:01.998132+0000","last_became_peered":"2024-07-26T06:47:01.998132+0000","last_unstale":"2024-07-26T06:47:48.160574+0000","last_undegraded":"2024-07-26T06:47:48.160574+0000","last_fullsized":"2024-07-26T06:47:48.160574+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:21.588085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":50,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091351+0000","last_change":"2024-07-26T06:47:05.032704+0000","last_active":"2024-07-26T06:47:23.091351+0000","last_peered":"2024-07-26T06:47:23.091351+0000","last_clean":"2024-07-26T06:47:23.091351+0000","last_became_active":"2024-07-26T06:47:05.031843+0000","last_became_peered":"2024-07-26T06:47:05.031843+0000","last_unstale":"2024-07-26T06:47:23.091351+0000","last_undegraded":"2024-07-26T06:47:23.091351+0000","last_fullsized":"2024-07-26T06:47:23.091351+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:58:24.436233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097398+0000","last_change":"2024-07-26T06:47:02.006266+0000","last_active":"2024-07-26T06:47:23.097398+0000","last_peered":"2024-07-26T06:47:23.097398+0000","last_clean":"2024-07-26T06:47:23.097398+0000","last_became_active":"2024-07-26T06:47:02.005830+0000","last_became_peered":"2024-07-26T06:47:02.005830+0000","last_unstale":"2024-07-26T06:47:23.097398+0000","last_undegraded":"2024-07-26T06:47:23.097398+0000","last_fullsized":"2024-07-26T06:47:23.097398+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:06:09.097781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.094830+0000","last_change":"2024-07-26T06:47:05.024820+0000","last_active":"2024-07-26T06:47:23.094830+0000","last_peered":"2024-07-26T06:47:23.094830+0000","last_clean":"2024-07-26T06:47:23.094830+0000","last_became_active":"2024-07-26T06:47:05.023941+0000","last_became_peered":"2024-07-26T06:47:05.023941+0000","last_unstale":"2024-07-26T06:47:23.094830+0000","last_undegraded":"2024-07-26T06:47:23.094830+0000","last_fullsized":"2024-07-26T06:47:23.094830+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:07.885159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095379+0000","last_change":"2024-07-26T06:47:02.023308+0000","last_active":"2024-07-26T06:47:23.095379+0000","last_peered":"2024-07-26T06:47:23.095379+0000","last_clean":"2024-07-26T06:47:23.095379+0000","last_became_active":"2024-07-26T06:47:02.022933+0000","last_became_peered":"2024-07-26T06:47:02.022933+0000","last_unstale":"2024-07-26T06:47:23.095379+0000","last_undegraded":"2024-07-26T06:47:23.095379+0000","last_fullsized":"2024-07-26T06:47:23.095379+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:59:01.233386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091075+0000","last_change":"2024-07-26T06:47:05.026005+0000","last_active":"2024-07-26T06:47:23.091075+0000","last_peered":"2024-07-26T06:47:23.091075+0000","last_clean":"2024-07-26T06:47:23.091075+0000","last_became_active":"2024-07-26T06:47:05.025599+0000","last_became_peered":"2024-07-26T06:47:05.025599+0000","last_unstale":"2024-07-26T06:47:23.091075+0000","last_undegraded":"2024-07-26T06:47:23.091075+0000","last_fullsized":"2024-07-26T06:47:23.091075+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:44:19.924157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'29","reported_seq":132,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:48.163072+0000","last_change":"2024-07-26T06:47:23.393050+0000","last_active":"2024-07-26T06:47:48.163072+0000","last_peered":"2024-07-26T06:47:48.163072+0000","last_clean":"2024-07-26T06:47:48.163072+0000","last_became_active":"2024-07-26T06:47:23.381611+0000","last_became_peered":"2024-07-26T06:47:23.381611+0000","last_unstale":"2024-07-26T06:47:48.163072+0000","last_undegraded":"2024-07-26T06:47:48.163072+0000","last_fullsized":"2024-07-26T06:47:48.163072+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:23.866523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55581,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":58,"num_read_kb":82,"num_write":31,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092605+0000","last_change":"2024-07-26T06:47:05.034190+0000","last_active":"2024-07-26T06:47:23.092605+0000","last_peered":"2024-07-26T06:47:23.092605+0000","last_clean":"2024-07-26T06:47:23.092605+0000","last_became_active":"2024-07-26T06:47:05.033569+0000","last_became_peered":"2024-07-26T06:47:05.033569+0000","last_unstale":"2024-07-26T06:47:23.092605+0000","last_undegraded":"2024-07-26T06:47:23.092605+0000","last_fullsized":"2024-07-26T06:47:23.092605+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:39.306996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.120074+0000","last_change":"2024-07-26T06:47:02.001718+0000","last_active":"2024-07-26T06:47:24.120074+0000","last_peered":"2024-07-26T06:47:24.120074+0000","last_clean":"2024-07-26T06:47:24.120074+0000","last_became_active":"2024-07-26T06:47:02.001359+0000","last_became_peered":"2024-07-26T06:47:02.001359+0000","last_unstale":"2024-07-26T06:47:24.120074+0000","last_undegraded":"2024-07-26T06:47:24.120074+0000","last_fullsized":"2024-07-26T06:47:24.120074+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:03:23.390279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095009+0000","last_change":"2024-07-26T06:47:05.027445+0000","last_active":"2024-07-26T06:47:23.095009+0000","last_peered":"2024-07-26T06:47:23.095009+0000","last_clean":"2024-07-26T06:47:23.095009+0000","last_became_active":"2024-07-26T06:47:05.027114+0000","last_became_peered":"2024-07-26T06:47:05.027114+0000","last_unstale":"2024-07-26T06:47:23.095009+0000","last_undegraded":"2024-07-26T06:47:23.095009+0000","last_fullsized":"2024-07-26T06:47:23.095009+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:29:10.519271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.094571+0000","last_change":"2024-07-26T06:47:01.998832+0000","last_active":"2024-07-26T06:47:23.094571+0000","last_peered":"2024-07-26T06:47:23.094571+0000","last_clean":"2024-07-26T06:47:23.094571+0000","last_became_active":"2024-07-26T06:47:01.998512+0000","last_became_peered":"2024-07-26T06:47:01.998512+0000","last_unstale":"2024-07-26T06:47:23.094571+0000","last_undegraded":"2024-07-26T06:47:23.094571+0000","last_fullsized":"2024-07-26T06:47:23.094571+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:27.722329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097582+0000","last_change":"2024-07-26T06:47:05.027803+0000","last_active":"2024-07-26T06:47:23.097582+0000","last_peered":"2024-07-26T06:47:23.097582+0000","last_clean":"2024-07-26T06:47:23.097582+0000","last_became_active":"2024-07-26T06:47:05.027320+0000","last_became_peered":"2024-07-26T06:47:05.027320+0000","last_unstale":"2024-07-26T06:47:23.097582+0000","last_undegraded":"2024-07-26T06:47:23.097582+0000","last_fullsized":"2024-07-26T06:47:23.097582+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:43:06.242945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'6","reported_seq":61,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:47.160230+0000","last_change":"2024-07-26T06:47:01.998356+0000","last_active":"2024-07-26T06:47:47.160230+0000","last_peered":"2024-07-26T06:47:47.160230+0000","last_clean":"2024-07-26T06:47:47.160230+0000","last_became_active":"2024-07-26T06:47:01.997922+0000","last_became_peered":"2024-07-26T06:47:01.997922+0000","last_unstale":"2024-07-26T06:47:47.160230+0000","last_undegraded":"2024-07-26T06:47:47.160230+0000","last_fullsized":"2024-07-26T06:47:47.160230+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:50:58.066682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096946+0000","last_change":"2024-07-26T06:47:05.025285+0000","last_active":"2024-07-26T06:47:23.096946+0000","last_peered":"2024-07-26T06:47:23.096946+0000","last_clean":"2024-07-26T06:47:23.096946+0000","last_became_active":"2024-07-26T06:47:05.024945+0000","last_became_peered":"2024-07-26T06:47:05.024945+0000","last_unstale":"2024-07-26T06:47:23.096946+0000","last_undegraded":"2024-07-26T06:47:23.096946+0000","last_fullsized":"2024-07-26T06:47:23.096946+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:21:04.680435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093595+0000","last_change":"2024-07-26T06:47:02.024895+0000","last_active":"2024-07-26T06:47:23.093595+0000","last_peered":"2024-07-26T06:47:23.093595+0000","last_clean":"2024-07-26T06:47:23.093595+0000","last_became_active":"2024-07-26T06:47:02.024451+0000","last_became_peered":"2024-07-26T06:47:02.024451+0000","last_unstale":"2024-07-26T06:47:23.093595+0000","last_undegraded":"2024-07-26T06:47:23.093595+0000","last_fullsized":"2024-07-26T06:47:23.093595+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:05:13.396499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.381402+0000","last_change":"2024-07-26T06:47:23.381060+0000","last_active":"2024-07-26T06:47:23.381402+0000","last_peered":"2024-07-26T06:47:23.381402+0000","last_clean":"2024-07-26T06:47:23.381402+0000","last_became_active":"2024-07-26T06:47:23.380685+0000","last_became_peered":"2024-07-26T06:47:23.380685+0000","last_unstale":"2024-07-26T06:47:23.381402+0000","last_undegraded":"2024-07-26T06:47:23.381402+0000","last_fullsized":"2024-07-26T06:47:23.381402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:21:38.062900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093109+0000","last_change":"2024-07-26T06:47:02.002098+0000","last_active":"2024-07-26T06:47:23.093109+0000","last_peered":"2024-07-26T06:47:23.093109+0000","last_clean":"2024-07-26T06:47:23.093109+0000","last_became_active":"2024-07-26T06:47:02.001428+0000","last_became_peered":"2024-07-26T06:47:02.001428+0000","last_unstale":"2024-07-26T06:47:23.093109+0000","last_undegraded":"2024-07-26T06:47:23.093109+0000","last_fullsized":"2024-07-26T06:47:23.093109+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:52:29.112500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096258+0000","last_change":"2024-07-26T06:47:05.023686+0000","last_active":"2024-07-26T06:47:23.096258+0000","last_peered":"2024-07-26T06:47:23.096258+0000","last_clean":"2024-07-26T06:47:23.096258+0000","last_became_active":"2024-07-26T06:47:05.022583+0000","last_became_peered":"2024-07-26T06:47:05.022583+0000","last_unstale":"2024-07-26T06:47:23.096258+0000","last_undegraded":"2024-07-26T06:47:23.096258+0000","last_fullsized":"2024-07-26T06:47:23.096258+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:29:03.752429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096434+0000","last_change":"2024-07-26T06:47:02.006040+0000","last_active":"2024-07-26T06:47:23.096434+0000","last_peered":"2024-07-26T06:47:23.096434+0000","last_clean":"2024-07-26T06:47:23.096434+0000","last_became_active":"2024-07-26T06:47:02.005586+0000","last_became_peered":"2024-07-26T06:47:02.005586+0000","last_unstale":"2024-07-26T06:47:23.096434+0000","last_undegraded":"2024-07-26T06:47:23.096434+0000","last_fullsized":"2024-07-26T06:47:23.096434+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:09:27.180873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097475+0000","last_change":"2024-07-26T06:47:05.035524+0000","last_active":"2024-07-26T06:47:23.097475+0000","last_peered":"2024-07-26T06:47:23.097475+0000","last_clean":"2024-07-26T06:47:23.097475+0000","last_became_active":"2024-07-26T06:47:05.035078+0000","last_became_peered":"2024-07-26T06:47:05.035078+0000","last_unstale":"2024-07-26T06:47:23.097475+0000","last_undegraded":"2024-07-26T06:47:23.097475+0000","last_fullsized":"2024-07-26T06:47:23.097475+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:03.758261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095730+0000","last_change":"2024-07-26T06:47:01.999810+0000","last_active":"2024-07-26T06:47:23.095730+0000","last_peered":"2024-07-26T06:47:23.095730+0000","last_clean":"2024-07-26T06:47:23.095730+0000","last_became_active":"2024-07-26T06:47:01.999460+0000","last_became_peered":"2024-07-26T06:47:01.999460+0000","last_unstale":"2024-07-26T06:47:23.095730+0000","last_undegraded":"2024-07-26T06:47:23.095730+0000","last_fullsized":"2024-07-26T06:47:23.095730+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:25:05.691900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'1","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097358+0000","last_change":"2024-07-26T06:47:05.033631+0000","last_active":"2024-07-26T06:47:23.097358+0000","last_peered":"2024-07-26T06:47:23.097358+0000","last_clean":"2024-07-26T06:47:23.097358+0000","last_became_active":"2024-07-26T06:47:05.033266+0000","last_became_peered":"2024-07-26T06:47:05.033266+0000","last_unstale":"2024-07-26T06:47:23.097358+0000","last_undegraded":"2024-07-26T06:47:23.097358+0000","last_fullsized":"2024-07-26T06:47:23.097358+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:55.895271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"78'5","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:45.156995+0000","last_change":"2024-07-26T06:47:02.113525+0000","last_active":"2024-07-26T06:47:45.156995+0000","last_peered":"2024-07-26T06:47:45.156995+0000","last_clean":"2024-07-26T06:47:45.156995+0000","last_became_active":"2024-07-26T06:47:02.113112+0000","last_became_peered":"2024-07-26T06:47:02.113112+0000","last_unstale":"2024-07-26T06:47:45.156995+0000","last_undegraded":"2024-07-26T06:47:45.156995+0000","last_fullsized":"2024-07-26T06:47:45.156995+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:05:15.071264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097372+0000","last_change":"2024-07-26T06:47:05.026423+0000","last_active":"2024-07-26T06:47:23.097372+0000","last_peered":"2024-07-26T06:47:23.097372+0000","last_clean":"2024-07-26T06:47:23.097372+0000","last_became_active":"2024-07-26T06:47:05.026053+0000","last_became_peered":"2024-07-26T06:47:05.026053+0000","last_unstale":"2024-07-26T06:47:23.097372+0000","last_undegraded":"2024-07-26T06:47:23.097372+0000","last_fullsized":"2024-07-26T06:47:23.097372+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:07:20.174538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":101,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097187+0000","last_change":"2024-07-26T06:45:40.184988+0000","last_active":"2024-07-26T06:47:23.097187+0000","last_peered":"2024-07-26T06:47:23.097187+0000","last_clean":"2024-07-26T06:47:23.097187+0000","last_became_active":"2024-07-26T06:45:40.178827+0000","last_became_peered":"2024-07-26T06:45:40.178827+0000","last_unstale":"2024-07-26T06:47:23.097187+0000","last_undegraded":"2024-07-26T06:47:23.097187+0000","last_fullsized":"2024-07-26T06:47:23.097187+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:42:28.257602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:42:28.257602+0000","last_clean_scrub_stamp":"2024-07-26T06:42:28.257602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:52:14.123696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.120218+0000","last_change":"2024-07-26T06:47:01.996610+0000","last_active":"2024-07-26T06:47:24.120218+0000","last_peered":"2024-07-26T06:47:24.120218+0000","last_clean":"2024-07-26T06:47:24.120218+0000","last_became_active":"2024-07-26T06:47:01.996155+0000","last_became_peered":"2024-07-26T06:47:01.996155+0000","last_unstale":"2024-07-26T06:47:24.120218+0000","last_undegraded":"2024-07-26T06:47:24.120218+0000","last_fullsized":"2024-07-26T06:47:24.120218+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:51:30.420254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090885+0000","last_change":"2024-07-26T06:47:05.027699+0000","last_active":"2024-07-26T06:47:23.090885+0000","last_peered":"2024-07-26T06:47:23.090885+0000","last_clean":"2024-07-26T06:47:23.090885+0000","last_became_active":"2024-07-26T06:47:05.027348+0000","last_became_peered":"2024-07-26T06:47:05.027348+0000","last_unstale":"2024-07-26T06:47:23.090885+0000","last_undegraded":"2024-07-26T06:47:23.090885+0000","last_fullsized":"2024-07-26T06:47:23.090885+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:28:42.664351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097120+0000","last_change":"2024-07-26T06:47:02.003205+0000","last_active":"2024-07-26T06:47:23.097120+0000","last_peered":"2024-07-26T06:47:23.097120+0000","last_clean":"2024-07-26T06:47:23.097120+0000","last_became_active":"2024-07-26T06:47:02.002543+0000","last_became_peered":"2024-07-26T06:47:02.002543+0000","last_unstale":"2024-07-26T06:47:23.097120+0000","last_undegraded":"2024-07-26T06:47:23.097120+0000","last_fullsized":"2024-07-26T06:47:23.097120+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:43:10.911943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095958+0000","last_change":"2024-07-26T06:47:05.023258+0000","last_active":"2024-07-26T06:47:23.095958+0000","last_peered":"2024-07-26T06:47:23.095958+0000","last_clean":"2024-07-26T06:47:23.095958+0000","last_became_active":"2024-07-26T06:47:05.022903+0000","last_became_peered":"2024-07-26T06:47:05.022903+0000","last_unstale":"2024-07-26T06:47:23.095958+0000","last_undegraded":"2024-07-26T06:47:23.095958+0000","last_fullsized":"2024-07-26T06:47:23.095958+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:08.155850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":55,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:45.157023+0000","last_change":"2024-07-26T06:47:02.022316+0000","last_active":"2024-07-26T06:47:45.157023+0000","last_peered":"2024-07-26T06:47:45.157023+0000","last_clean":"2024-07-26T06:47:45.157023+0000","last_became_active":"2024-07-26T06:47:02.021923+0000","last_became_peered":"2024-07-26T06:47:02.021923+0000","last_unstale":"2024-07-26T06:47:45.157023+0000","last_undegraded":"2024-07-26T06:47:45.157023+0000","last_fullsized":"2024-07-26T06:47:45.157023+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:29:31.533449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093615+0000","last_change":"2024-07-26T06:47:05.021510+0000","last_active":"2024-07-26T06:47:23.093615+0000","last_peered":"2024-07-26T06:47:23.093615+0000","last_clean":"2024-07-26T06:47:23.093615+0000","last_became_active":"2024-07-26T06:47:05.021140+0000","last_became_peered":"2024-07-26T06:47:05.021140+0000","last_unstale":"2024-07-26T06:47:23.093615+0000","last_undegraded":"2024-07-26T06:47:23.093615+0000","last_fullsized":"2024-07-26T06:47:23.093615+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:37:01.340126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"79'5","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:47.158885+0000","last_change":"2024-07-26T06:47:02.001698+0000","last_active":"2024-07-26T06:47:47.158885+0000","last_peered":"2024-07-26T06:47:47.158885+0000","last_clean":"2024-07-26T06:47:47.158885+0000","last_became_active":"2024-07-26T06:47:02.001244+0000","last_became_peered":"2024-07-26T06:47:02.001244+0000","last_unstale":"2024-07-26T06:47:47.158885+0000","last_undegraded":"2024-07-26T06:47:47.158885+0000","last_fullsized":"2024-07-26T06:47:47.158885+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:32:07.920065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093652+0000","last_change":"2024-07-26T06:47:05.029524+0000","last_active":"2024-07-26T06:47:23.093652+0000","last_peered":"2024-07-26T06:47:23.093652+0000","last_clean":"2024-07-26T06:47:23.093652+0000","last_became_active":"2024-07-26T06:47:05.028981+0000","last_became_peered":"2024-07-26T06:47:05.028981+0000","last_unstale":"2024-07-26T06:47:23.093652+0000","last_undegraded":"2024-07-26T06:47:23.093652+0000","last_fullsized":"2024-07-26T06:47:23.093652+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:49:57.019486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092276+0000","last_change":"2024-07-26T06:47:02.025117+0000","last_active":"2024-07-26T06:47:23.092276+0000","last_peered":"2024-07-26T06:47:23.092276+0000","last_clean":"2024-07-26T06:47:23.092276+0000","last_became_active":"2024-07-26T06:47:02.024612+0000","last_became_peered":"2024-07-26T06:47:02.024612+0000","last_unstale":"2024-07-26T06:47:23.092276+0000","last_undegraded":"2024-07-26T06:47:23.092276+0000","last_fullsized":"2024-07-26T06:47:23.092276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:15:44.999658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'1","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.094137+0000","last_change":"2024-07-26T06:47:05.015085+0000","last_active":"2024-07-26T06:47:23.094137+0000","last_peered":"2024-07-26T06:47:23.094137+0000","last_clean":"2024-07-26T06:47:23.094137+0000","last_became_active":"2024-07-26T06:47:05.014686+0000","last_became_peered":"2024-07-26T06:47:05.014686+0000","last_unstale":"2024-07-26T06:47:23.094137+0000","last_undegraded":"2024-07-26T06:47:23.094137+0000","last_fullsized":"2024-07-26T06:47:23.094137+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:39:30.392347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"80'4","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.159121+0000","last_change":"2024-07-26T06:47:23.111918+0000","last_active":"2024-07-26T06:47:23.159121+0000","last_peered":"2024-07-26T06:47:23.159121+0000","last_clean":"2024-07-26T06:47:23.159121+0000","last_became_active":"2024-07-26T06:47:23.099518+0000","last_became_peered":"2024-07-26T06:47:23.099518+0000","last_unstale":"2024-07-26T06:47:23.159121+0000","last_undegraded":"2024-07-26T06:47:23.159121+0000","last_fullsized":"2024-07-26T06:47:23.159121+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:41.012096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096626+0000","last_change":"2024-07-26T06:47:05.019415+0000","last_active":"2024-07-26T06:47:23.096626+0000","last_peered":"2024-07-26T06:47:23.096626+0000","last_clean":"2024-07-26T06:47:23.096626+0000","last_became_active":"2024-07-26T06:47:05.019044+0000","last_became_peered":"2024-07-26T06:47:05.019044+0000","last_unstale":"2024-07-26T06:47:23.096626+0000","last_undegraded":"2024-07-26T06:47:23.096626+0000","last_fullsized":"2024-07-26T06:47:23.096626+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:02:37.228174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'14","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:44.026778+0000","last_change":"2024-07-26T06:47:02.006585+0000","last_active":"2024-07-26T06:47:44.026778+0000","last_peered":"2024-07-26T06:47:44.026778+0000","last_clean":"2024-07-26T06:47:44.026778+0000","last_became_active":"2024-07-26T06:47:02.006185+0000","last_became_peered":"2024-07-26T06:47:02.006185+0000","last_unstale":"2024-07-26T06:47:44.026778+0000","last_undegraded":"2024-07-26T06:47:44.026778+0000","last_fullsized":"2024-07-26T06:47:44.026778+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:02:45.480712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091090+0000","last_change":"2024-07-26T06:47:05.028840+0000","last_active":"2024-07-26T06:47:23.091090+0000","last_peered":"2024-07-26T06:47:23.091090+0000","last_clean":"2024-07-26T06:47:23.091090+0000","last_became_active":"2024-07-26T06:47:05.028487+0000","last_became_peered":"2024-07-26T06:47:05.028487+0000","last_unstale":"2024-07-26T06:47:23.091090+0000","last_undegraded":"2024-07-26T06:47:23.091090+0000","last_fullsized":"2024-07-26T06:47:23.091090+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:47:48.185227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.380389+0000","last_change":"2024-07-26T06:47:02.113691+0000","last_active":"2024-07-26T06:47:23.380389+0000","last_peered":"2024-07-26T06:47:23.380389+0000","last_clean":"2024-07-26T06:47:23.380389+0000","last_became_active":"2024-07-26T06:47:02.113342+0000","last_became_peered":"2024-07-26T06:47:02.113342+0000","last_unstale":"2024-07-26T06:47:23.380389+0000","last_undegraded":"2024-07-26T06:47:23.380389+0000","last_fullsized":"2024-07-26T06:47:23.380389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:14:03.586564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.098503+0000","last_change":"2024-07-26T06:47:23.098420+0000","last_active":"2024-07-26T06:47:23.098503+0000","last_peered":"2024-07-26T06:47:23.098503+0000","last_clean":"2024-07-26T06:47:23.098503+0000","last_became_active":"2024-07-26T06:47:23.097981+0000","last_became_peered":"2024-07-26T06:47:23.097981+0000","last_unstale":"2024-07-26T06:47:23.098503+0000","last_undegraded":"2024-07-26T06:47:23.098503+0000","last_fullsized":"2024-07-26T06:47:23.098503+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:38:40.155514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091072+0000","last_change":"2024-07-26T06:47:02.024781+0000","last_active":"2024-07-26T06:47:23.091072+0000","last_peered":"2024-07-26T06:47:23.091072+0000","last_clean":"2024-07-26T06:47:23.091072+0000","last_became_active":"2024-07-26T06:47:02.024492+0000","last_became_peered":"2024-07-26T06:47:02.024492+0000","last_unstale":"2024-07-26T06:47:23.091072+0000","last_undegraded":"2024-07-26T06:47:23.091072+0000","last_fullsized":"2024-07-26T06:47:23.091072+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:23:41.323963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092602+0000","last_change":"2024-07-26T06:47:05.033052+0000","last_active":"2024-07-26T06:47:23.092602+0000","last_peered":"2024-07-26T06:47:23.092602+0000","last_clean":"2024-07-26T06:47:23.092602+0000","last_became_active":"2024-07-26T06:47:05.032052+0000","last_became_peered":"2024-07-26T06:47:05.032052+0000","last_unstale":"2024-07-26T06:47:23.092602+0000","last_undegraded":"2024-07-26T06:47:23.092602+0000","last_fullsized":"2024-07-26T06:47:23.092602+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:42:04.683586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090564+0000","last_change":"2024-07-26T06:47:01.996097+0000","last_active":"2024-07-26T06:47:23.090564+0000","last_peered":"2024-07-26T06:47:23.090564+0000","last_clean":"2024-07-26T06:47:23.090564+0000","last_became_active":"2024-07-26T06:47:01.995594+0000","last_became_peered":"2024-07-26T06:47:01.995594+0000","last_unstale":"2024-07-26T06:47:23.090564+0000","last_undegraded":"2024-07-26T06:47:23.090564+0000","last_fullsized":"2024-07-26T06:47:23.090564+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:34.969493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095563+0000","last_change":"2024-07-26T06:47:05.024171+0000","last_active":"2024-07-26T06:47:23.095563+0000","last_peered":"2024-07-26T06:47:23.095563+0000","last_clean":"2024-07-26T06:47:23.095563+0000","last_became_active":"2024-07-26T06:47:05.022888+0000","last_became_peered":"2024-07-26T06:47:05.022888+0000","last_unstale":"2024-07-26T06:47:23.095563+0000","last_undegraded":"2024-07-26T06:47:23.095563+0000","last_fullsized":"2024-07-26T06:47:23.095563+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:31:52.691068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096894+0000","last_change":"2024-07-26T06:47:02.004578+0000","last_active":"2024-07-26T06:47:23.096894+0000","last_peered":"2024-07-26T06:47:23.096894+0000","last_clean":"2024-07-26T06:47:23.096894+0000","last_became_active":"2024-07-26T06:47:02.004231+0000","last_became_peered":"2024-07-26T06:47:02.004231+0000","last_unstale":"2024-07-26T06:47:23.096894+0000","last_undegraded":"2024-07-26T06:47:23.096894+0000","last_fullsized":"2024-07-26T06:47:23.096894+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:43:39.803817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091223+0000","last_change":"2024-07-26T06:47:05.024232+0000","last_active":"2024-07-26T06:47:23.091223+0000","last_peered":"2024-07-26T06:47:23.091223+0000","last_clean":"2024-07-26T06:47:23.091223+0000","last_became_active":"2024-07-26T06:47:05.023869+0000","last_became_peered":"2024-07-26T06:47:05.023869+0000","last_unstale":"2024-07-26T06:47:23.091223+0000","last_undegraded":"2024-07-26T06:47:23.091223+0000","last_fullsized":"2024-07-26T06:47:23.091223+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:27:01.357311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.105594+0000","last_change":"2024-07-26T06:47:23.105051+0000","last_active":"2024-07-26T06:47:23.105594+0000","last_peered":"2024-07-26T06:47:23.105594+0000","last_clean":"2024-07-26T06:47:23.105594+0000","last_became_active":"2024-07-26T06:47:23.098116+0000","last_became_peered":"2024-07-26T06:47:23.098116+0000","last_unstale":"2024-07-26T06:47:23.105594+0000","last_undegraded":"2024-07-26T06:47:23.105594+0000","last_fullsized":"2024-07-26T06:47:23.105594+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:15:32.991870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097110+0000","last_change":"2024-07-26T06:47:05.026056+0000","last_active":"2024-07-26T06:47:23.097110+0000","last_peered":"2024-07-26T06:47:23.097110+0000","last_clean":"2024-07-26T06:47:23.097110+0000","last_became_active":"2024-07-26T06:47:05.025654+0000","last_became_peered":"2024-07-26T06:47:05.025654+0000","last_unstale":"2024-07-26T06:47:23.097110+0000","last_undegraded":"2024-07-26T06:47:23.097110+0000","last_fullsized":"2024-07-26T06:47:23.097110+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:36:20.886768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.094905+0000","last_change":"2024-07-26T06:47:05.026693+0000","last_active":"2024-07-26T06:47:23.094905+0000","last_peered":"2024-07-26T06:47:23.094905+0000","last_clean":"2024-07-26T06:47:23.094905+0000","last_became_active":"2024-07-26T06:47:05.026320+0000","last_became_peered":"2024-07-26T06:47:05.026320+0000","last_unstale":"2024-07-26T06:47:23.094905+0000","last_undegraded":"2024-07-26T06:47:23.094905+0000","last_fullsized":"2024-07-26T06:47:23.094905+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:52:30.776335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"80'5","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.159101+0000","last_change":"2024-07-26T06:47:23.107707+0000","last_active":"2024-07-26T06:47:23.159101+0000","last_peered":"2024-07-26T06:47:23.159101+0000","last_clean":"2024-07-26T06:47:23.159101+0000","last_became_active":"2024-07-26T06:47:23.099429+0000","last_became_peered":"2024-07-26T06:47:23.099429+0000","last_unstale":"2024-07-26T06:47:23.159101+0000","last_undegraded":"2024-07-26T06:47:23.159101+0000","last_fullsized":"2024-07-26T06:47:23.159101+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:40:52.856756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091011+0000","last_change":"2024-07-26T06:47:05.032319+0000","last_active":"2024-07-26T06:47:23.091011+0000","last_peered":"2024-07-26T06:47:23.091011+0000","last_clean":"2024-07-26T06:47:23.091011+0000","last_became_active":"2024-07-26T06:47:05.030305+0000","last_became_peered":"2024-07-26T06:47:05.030305+0000","last_unstale":"2024-07-26T06:47:23.091011+0000","last_undegraded":"2024-07-26T06:47:23.091011+0000","last_fullsized":"2024-07-26T06:47:23.091011+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:34:05.351917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"80'12","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.119914+0000","last_change":"2024-07-26T06:47:02.007812+0000","last_active":"2024-07-26T06:47:24.119914+0000","last_peered":"2024-07-26T06:47:24.119914+0000","last_clean":"2024-07-26T06:47:24.119914+0000","last_became_active":"2024-07-26T06:47:02.006965+0000","last_became_peered":"2024-07-26T06:47:02.006965+0000","last_unstale":"2024-07-26T06:47:24.119914+0000","last_undegraded":"2024-07-26T06:47:24.119914+0000","last_fullsized":"2024-07-26T06:47:24.119914+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:53.173273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.094716+0000","last_change":"2024-07-26T06:47:05.027900+0000","last_active":"2024-07-26T06:47:23.094716+0000","last_peered":"2024-07-26T06:47:23.094716+0000","last_clean":"2024-07-26T06:47:23.094716+0000","last_became_active":"2024-07-26T06:47:05.024467+0000","last_became_peered":"2024-07-26T06:47:05.024467+0000","last_unstale":"2024-07-26T06:47:23.094716+0000","last_undegraded":"2024-07-26T06:47:23.094716+0000","last_fullsized":"2024-07-26T06:47:23.094716+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:05:00.793487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090822+0000","last_change":"2024-07-26T06:47:02.022863+0000","last_active":"2024-07-26T06:47:23.090822+0000","last_peered":"2024-07-26T06:47:23.090822+0000","last_clean":"2024-07-26T06:47:23.090822+0000","last_became_active":"2024-07-26T06:47:02.022446+0000","last_became_peered":"2024-07-26T06:47:02.022446+0000","last_unstale":"2024-07-26T06:47:23.090822+0000","last_undegraded":"2024-07-26T06:47:23.090822+0000","last_fullsized":"2024-07-26T06:47:23.090822+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:14:48.649099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090672+0000","last_change":"2024-07-26T06:47:05.023669+0000","last_active":"2024-07-26T06:47:23.090672+0000","last_peered":"2024-07-26T06:47:23.090672+0000","last_clean":"2024-07-26T06:47:23.090672+0000","last_became_active":"2024-07-26T06:47:05.023265+0000","last_became_peered":"2024-07-26T06:47:05.023265+0000","last_unstale":"2024-07-26T06:47:23.090672+0000","last_undegraded":"2024-07-26T06:47:23.090672+0000","last_fullsized":"2024-07-26T06:47:23.090672+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:56:45.278253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:47.158283+0000","last_change":"2024-07-26T06:47:02.006668+0000","last_active":"2024-07-26T06:47:47.158283+0000","last_peered":"2024-07-26T06:47:47.158283+0000","last_clean":"2024-07-26T06:47:47.158283+0000","last_became_active":"2024-07-26T06:47:02.006287+0000","last_became_peered":"2024-07-26T06:47:02.006287+0000","last_unstale":"2024-07-26T06:47:47.158283+0000","last_undegraded":"2024-07-26T06:47:47.158283+0000","last_fullsized":"2024-07-26T06:47:47.158283+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:28:05.722539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62785,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":284,"num_read_kb":212,"num_write":242,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1634,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":188355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250626,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710744,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":884736,"data_stored":426222,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906973,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":1072529,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070497,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":425458,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266726,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":880640,"data_stored":425142,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462955,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":1018745,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626480,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36196,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708956,"statfs":{"total":95995035648,"available":95957970944,"internally_reserved":0,"allocated":880640,"data_stored":425175,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822709,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":1018032,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986234,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":481008,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182462,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":897024,"data_stored":425000,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411395,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":892928,"data_stored":426699,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":937984,"data_stored":481669,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738443,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":901120,"data_stored":426514,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":57333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":56672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":57708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:47:50.190 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph pg dump --format=json 2024-07-26T06:47:50.461 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:50.713 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:50 smithi042 ceph-mon[32775]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.4 KiB/s wr, 13 op/s 2024-07-26T06:47:50.714 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:50 smithi042 ceph-mon[32775]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:47:50.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:50 smithi164 ceph-mon[36970]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.4 KiB/s wr, 13 op/s 2024-07-26T06:47:50.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:50 smithi164 ceph-mon[36970]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:47:50.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:50 smithi163 ceph-mon[36873]: pgmap v324: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.4 KiB/s wr, 13 op/s 2024-07-26T06:47:50.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:50 smithi163 ceph-mon[36873]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:47:52.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.088+0000 7f6e8a0dd640 1 -- 172.21.15.42:0/3355566748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e84105560 msgr2=0x7f6e8410efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:52.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.088+0000 7f6e8a0dd640 1 --2- 172.21.15.42:0/3355566748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e84105560 0x7f6e8410efb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c009ca0 tx=0x7f6e7c02a8c0 comp rx=0 tx=0).stop 2024-07-26T06:47:52.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.088+0000 7f6e8a0dd640 1 -- 172.21.15.42:0/3355566748 shutdown_connections 2024-07-26T06:47:52.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.088+0000 7f6e8a0dd640 1 --2- 172.21.15.42:0/3355566748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e84105560 0x7f6e8410efb0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:52.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.088+0000 7f6e8a0dd640 1 --2- 172.21.15.42:0/3355566748 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e84104c10 0x7f6e84105020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:52.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.088+0000 7f6e8a0dd640 1 --2- 172.21.15.42:0/3355566748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e84102c40 0x7f6e84103050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:52.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.088+0000 7f6e8a0dd640 1 -- 172.21.15.42:0/3355566748 >> 172.21.15.42:0/3355566748 conn(0x7f6e840799c0 msgr2=0x7f6e84079dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:52.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.088+0000 7f6e8a0dd640 1 -- 172.21.15.42:0/3355566748 shutdown_connections 2024-07-26T06:47:52.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.088+0000 7f6e8a0dd640 1 -- 172.21.15.42:0/3355566748 wait complete. 2024-07-26T06:47:52.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.089+0000 7f6e8a0dd640 1 Processor -- start 2024-07-26T06:47:52.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.089+0000 7f6e8a0dd640 1 -- start start 2024-07-26T06:47:52.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.089+0000 7f6e8a0dd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e84102c40 0x7f6e8419c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:52.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.089+0000 7f6e8a0dd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e84104c10 0x7f6e8419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:52.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.089+0000 7f6e8a0dd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e84105560 0x7f6e841a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:52.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.089+0000 7f6e8a0dd640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6e84113560 con 0x7f6e84104c10 2024-07-26T06:47:52.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.089+0000 7f6e8a0dd640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6e841133e0 con 0x7f6e84102c40 2024-07-26T06:47:52.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.089+0000 7f6e8a0dd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6e841136e0 con 0x7f6e84105560 2024-07-26T06:47:52.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.089+0000 7f6e82ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e84104c10 0x7f6e8419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:52.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.089+0000 7f6e82ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e84104c10 0x7f6e8419c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56470/0 (socket says 172.21.15.42:56470) 2024-07-26T06:47:52.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.089+0000 7f6e82ffd640 1 -- 172.21.15.42:0/3703630382 learned_addr learned my addr 172.21.15.42:0/3703630382 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:52.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.089+0000 7f6e837fe640 1 --2- 172.21.15.42:0/3703630382 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e84102c40 0x7f6e8419c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:52.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.089+0000 7f6e83fff640 1 --2- 172.21.15.42:0/3703630382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e84105560 0x7f6e841a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:52.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.089+0000 7f6e82ffd640 1 -- 172.21.15.42:0/3703630382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e84105560 msgr2=0x7f6e841a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:52.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.089+0000 7f6e82ffd640 1 --2- 172.21.15.42:0/3703630382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e84105560 0x7f6e841a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:52.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.090+0000 7f6e82ffd640 1 -- 172.21.15.42:0/3703630382 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e84102c40 msgr2=0x7f6e8419c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:52.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.090+0000 7f6e82ffd640 1 --2- 172.21.15.42:0/3703630382 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e84102c40 0x7f6e8419c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:52.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.090+0000 7f6e82ffd640 1 -- 172.21.15.42:0/3703630382 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e841a20e0 con 0x7f6e84104c10 2024-07-26T06:47:52.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.090+0000 7f6e82ffd640 1 --2- 172.21.15.42:0/3703630382 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e84104c10 0x7f6e8419c710 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800d770 tx=0x7f6e7800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:52.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.090+0000 7f6e80ff9640 1 -- 172.21.15.42:0/3703630382 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e78017070 con 0x7f6e84104c10 2024-07-26T06:47:52.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.090+0000 7f6e80ff9640 1 -- 172.21.15.42:0/3703630382 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e78009c30 con 0x7f6e84104c10 2024-07-26T06:47:52.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.090+0000 7f6e8a0dd640 1 -- 172.21.15.42:0/3703630382 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e841a23d0 con 0x7f6e84104c10 2024-07-26T06:47:52.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.090+0000 7f6e80ff9640 1 -- 172.21.15.42:0/3703630382 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7800f620 con 0x7f6e84104c10 2024-07-26T06:47:52.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.090+0000 7f6e8a0dd640 1 -- 172.21.15.42:0/3703630382 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e841a8b30 con 0x7f6e84104c10 2024-07-26T06:47:52.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.091+0000 7f6e8a0dd640 1 -- 172.21.15.42:0/3703630382 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e84103050 con 0x7f6e84104c10 2024-07-26T06:47:52.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.093+0000 7f6e80ff9640 1 -- 172.21.15.42:0/3703630382 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f6e7801a070 con 0x7f6e84104c10 2024-07-26T06:47:52.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.093+0000 7f6e80ff9640 1 --2- 172.21.15.42:0/3703630382 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e6c0b3c60 0x7f6e6c0b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:52.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.094+0000 7f6e837fe640 1 --2- 172.21.15.42:0/3703630382 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e6c0b3c60 0x7f6e6c0b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:52.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.094+0000 7f6e80ff9640 1 -- 172.21.15.42:0/3703630382 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f6e780e38f0 con 0x7f6e84104c10 2024-07-26T06:47:52.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.094+0000 7f6e837fe640 1 --2- 172.21.15.42:0/3703630382 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e6c0b3c60 0x7f6e6c0b6170 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6e74009890 tx=0x7f6e74008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:52.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.097+0000 7f6e80ff9640 1 -- 172.21.15.42:0/3703630382 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6e78009de0 con 0x7f6e84104c10 2024-07-26T06:47:52.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.280+0000 7f6e8a0dd640 1 -- 172.21.15.42:0/3703630382 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6e84106350 con 0x7f6e6c0b3c60 2024-07-26T06:47:52.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.285+0000 7f6e80ff9640 1 -- 172.21.15.42:0/3703630382 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258988 (secure 0 0 0) 0x7f6e84106350 con 0x7f6e6c0b3c60 2024-07-26T06:47:52.290 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:47:52.291 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-07-26T06:47:52.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.288+0000 7f6e8a0dd640 1 -- 172.21.15.42:0/3703630382 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e6c0b3c60 msgr2=0x7f6e6c0b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:52.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.288+0000 7f6e8a0dd640 1 --2- 172.21.15.42:0/3703630382 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e6c0b3c60 0x7f6e6c0b6170 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6e74009890 tx=0x7f6e74008040 comp rx=0 tx=0).stop 2024-07-26T06:47:52.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.288+0000 7f6e8a0dd640 1 -- 172.21.15.42:0/3703630382 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e84104c10 msgr2=0x7f6e8419c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:52.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.288+0000 7f6e8a0dd640 1 --2- 172.21.15.42:0/3703630382 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e84104c10 0x7f6e8419c710 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800d770 tx=0x7f6e7800dc30 comp rx=0 tx=0).stop 2024-07-26T06:47:52.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.288+0000 7f6e8a0dd640 1 -- 172.21.15.42:0/3703630382 shutdown_connections 2024-07-26T06:47:52.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.288+0000 7f6e8a0dd640 1 --2- 172.21.15.42:0/3703630382 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e6c0b3c60 0x7f6e6c0b6170 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:52.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.288+0000 7f6e8a0dd640 1 --2- 172.21.15.42:0/3703630382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e84105560 0x7f6e841a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:52.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.288+0000 7f6e8a0dd640 1 --2- 172.21.15.42:0/3703630382 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e84104c10 0x7f6e8419c710 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:52.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.288+0000 7f6e8a0dd640 1 --2- 172.21.15.42:0/3703630382 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e84102c40 0x7f6e8419c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:52.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.288+0000 7f6e8a0dd640 1 -- 172.21.15.42:0/3703630382 >> 172.21.15.42:0/3703630382 conn(0x7f6e840799c0 msgr2=0x7f6e84105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:52.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.288+0000 7f6e8a0dd640 1 -- 172.21.15.42:0/3703630382 shutdown_connections 2024-07-26T06:47:52.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:52.288+0000 7f6e8a0dd640 1 -- 172.21.15.42:0/3703630382 wait complete. 2024-07-26T06:47:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:52 smithi164 ceph-mon[36970]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-07-26T06:47:52.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:52 smithi163 ceph-mon[36873]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-07-26T06:47:52.842 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:52 smithi042 ceph-mon[32775]: pgmap v325: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-07-26T06:47:52.870 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-07-26T06:47:51.326832+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653387,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":365,"num_read_kb":276,"num_write":367,"num_write_kb":2281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2953474,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380760,"kb_used_data":10788,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124561064,"statfs":{"total":1151940427776,"available":1151550529536,"internally_reserved":0,"allocated":11046912,"data_stored":7052193,"data_compressed":177858,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13380,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":64,"num_write":12,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003964"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096639+0000","last_change":"2024-07-26T06:47:05.026186+0000","last_active":"2024-07-26T06:47:23.096639+0000","last_peered":"2024-07-26T06:47:23.096639+0000","last_clean":"2024-07-26T06:47:23.096639+0000","last_became_active":"2024-07-26T06:47:05.025589+0000","last_became_peered":"2024-07-26T06:47:05.025589+0000","last_unstale":"2024-07-26T06:47:23.096639+0000","last_undegraded":"2024-07-26T06:47:23.096639+0000","last_fullsized":"2024-07-26T06:47:23.096639+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:48:23.100966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093146+0000","last_change":"2024-07-26T06:47:05.033211+0000","last_active":"2024-07-26T06:47:23.093146+0000","last_peered":"2024-07-26T06:47:23.093146+0000","last_clean":"2024-07-26T06:47:23.093146+0000","last_became_active":"2024-07-26T06:47:05.032630+0000","last_became_peered":"2024-07-26T06:47:05.032630+0000","last_unstale":"2024-07-26T06:47:23.093146+0000","last_undegraded":"2024-07-26T06:47:23.093146+0000","last_fullsized":"2024-07-26T06:47:23.093146+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:03:19.612963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092844+0000","last_change":"2024-07-26T06:47:05.034767+0000","last_active":"2024-07-26T06:47:23.092844+0000","last_peered":"2024-07-26T06:47:23.092844+0000","last_clean":"2024-07-26T06:47:23.092844+0000","last_became_active":"2024-07-26T06:47:05.034376+0000","last_became_peered":"2024-07-26T06:47:05.034376+0000","last_unstale":"2024-07-26T06:47:23.092844+0000","last_undegraded":"2024-07-26T06:47:23.092844+0000","last_fullsized":"2024-07-26T06:47:23.092844+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:55:55.729930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.380468+0000","last_change":"2024-07-26T06:47:05.030995+0000","last_active":"2024-07-26T06:47:23.380468+0000","last_peered":"2024-07-26T06:47:23.380468+0000","last_clean":"2024-07-26T06:47:23.380468+0000","last_became_active":"2024-07-26T06:47:05.025423+0000","last_became_peered":"2024-07-26T06:47:05.025423+0000","last_unstale":"2024-07-26T06:47:23.380468+0000","last_undegraded":"2024-07-26T06:47:23.380468+0000","last_fullsized":"2024-07-26T06:47:23.380468+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:59:39.511383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092282+0000","last_change":"2024-07-26T06:47:05.035359+0000","last_active":"2024-07-26T06:47:23.092282+0000","last_peered":"2024-07-26T06:47:23.092282+0000","last_clean":"2024-07-26T06:47:23.092282+0000","last_became_active":"2024-07-26T06:47:05.034836+0000","last_became_peered":"2024-07-26T06:47:05.034836+0000","last_unstale":"2024-07-26T06:47:23.092282+0000","last_undegraded":"2024-07-26T06:47:23.092282+0000","last_fullsized":"2024-07-26T06:47:23.092282+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:54.482774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093596+0000","last_change":"2024-07-26T06:47:05.023660+0000","last_active":"2024-07-26T06:47:23.093596+0000","last_peered":"2024-07-26T06:47:23.093596+0000","last_clean":"2024-07-26T06:47:23.093596+0000","last_became_active":"2024-07-26T06:47:05.022446+0000","last_became_peered":"2024-07-26T06:47:05.022446+0000","last_unstale":"2024-07-26T06:47:23.093596+0000","last_undegraded":"2024-07-26T06:47:23.093596+0000","last_fullsized":"2024-07-26T06:47:23.093596+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:35:00.519226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097130+0000","last_change":"2024-07-26T06:47:05.025132+0000","last_active":"2024-07-26T06:47:23.097130+0000","last_peered":"2024-07-26T06:47:23.097130+0000","last_clean":"2024-07-26T06:47:23.097130+0000","last_became_active":"2024-07-26T06:47:05.024770+0000","last_became_peered":"2024-07-26T06:47:05.024770+0000","last_unstale":"2024-07-26T06:47:23.097130+0000","last_undegraded":"2024-07-26T06:47:23.097130+0000","last_fullsized":"2024-07-26T06:47:23.097130+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:21:39.920152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090188+0000","last_change":"2024-07-26T06:47:05.022613+0000","last_active":"2024-07-26T06:47:23.090188+0000","last_peered":"2024-07-26T06:47:23.090188+0000","last_clean":"2024-07-26T06:47:23.090188+0000","last_became_active":"2024-07-26T06:47:05.022202+0000","last_became_peered":"2024-07-26T06:47:05.022202+0000","last_unstale":"2024-07-26T06:47:23.090188+0000","last_undegraded":"2024-07-26T06:47:23.090188+0000","last_fullsized":"2024-07-26T06:47:23.090188+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:07:23.980843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091579+0000","last_change":"2024-07-26T06:47:05.026352+0000","last_active":"2024-07-26T06:47:23.091579+0000","last_peered":"2024-07-26T06:47:23.091579+0000","last_clean":"2024-07-26T06:47:23.091579+0000","last_became_active":"2024-07-26T06:47:05.025969+0000","last_became_peered":"2024-07-26T06:47:05.025969+0000","last_unstale":"2024-07-26T06:47:23.091579+0000","last_undegraded":"2024-07-26T06:47:23.091579+0000","last_fullsized":"2024-07-26T06:47:23.091579+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:27.062013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.100482+0000","last_change":"2024-07-26T06:47:23.100422+0000","last_active":"2024-07-26T06:47:23.100482+0000","last_peered":"2024-07-26T06:47:23.100482+0000","last_clean":"2024-07-26T06:47:23.100482+0000","last_became_active":"2024-07-26T06:47:23.100023+0000","last_became_peered":"2024-07-26T06:47:23.100023+0000","last_unstale":"2024-07-26T06:47:23.100482+0000","last_undegraded":"2024-07-26T06:47:23.100482+0000","last_fullsized":"2024-07-26T06:47:23.100482+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:28.158112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097318+0000","last_change":"2024-07-26T06:47:05.037142+0000","last_active":"2024-07-26T06:47:23.097318+0000","last_peered":"2024-07-26T06:47:23.097318+0000","last_clean":"2024-07-26T06:47:23.097318+0000","last_became_active":"2024-07-26T06:47:05.036745+0000","last_became_peered":"2024-07-26T06:47:05.036745+0000","last_unstale":"2024-07-26T06:47:23.097318+0000","last_undegraded":"2024-07-26T06:47:23.097318+0000","last_fullsized":"2024-07-26T06:47:23.097318+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:31:10.034560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.380519+0000","last_change":"2024-07-26T06:47:05.030640+0000","last_active":"2024-07-26T06:47:23.380519+0000","last_peered":"2024-07-26T06:47:23.380519+0000","last_clean":"2024-07-26T06:47:23.380519+0000","last_became_active":"2024-07-26T06:47:05.023583+0000","last_became_peered":"2024-07-26T06:47:05.023583+0000","last_unstale":"2024-07-26T06:47:23.380519+0000","last_undegraded":"2024-07-26T06:47:23.380519+0000","last_fullsized":"2024-07-26T06:47:23.380519+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:59:12.312930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095476+0000","last_change":"2024-07-26T06:47:05.021768+0000","last_active":"2024-07-26T06:47:23.095476+0000","last_peered":"2024-07-26T06:47:23.095476+0000","last_clean":"2024-07-26T06:47:23.095476+0000","last_became_active":"2024-07-26T06:47:05.021410+0000","last_became_peered":"2024-07-26T06:47:05.021410+0000","last_unstale":"2024-07-26T06:47:23.095476+0000","last_undegraded":"2024-07-26T06:47:23.095476+0000","last_fullsized":"2024-07-26T06:47:23.095476+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:05.576122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092848+0000","last_change":"2024-07-26T06:47:05.035814+0000","last_active":"2024-07-26T06:47:23.092848+0000","last_peered":"2024-07-26T06:47:23.092848+0000","last_clean":"2024-07-26T06:47:23.092848+0000","last_became_active":"2024-07-26T06:47:05.035451+0000","last_became_peered":"2024-07-26T06:47:05.035451+0000","last_unstale":"2024-07-26T06:47:23.092848+0000","last_undegraded":"2024-07-26T06:47:23.092848+0000","last_fullsized":"2024-07-26T06:47:23.092848+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:33.533726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093301+0000","last_change":"2024-07-26T06:47:05.035720+0000","last_active":"2024-07-26T06:47:23.093301+0000","last_peered":"2024-07-26T06:47:23.093301+0000","last_clean":"2024-07-26T06:47:23.093301+0000","last_became_active":"2024-07-26T06:47:05.035359+0000","last_became_peered":"2024-07-26T06:47:05.035359+0000","last_unstale":"2024-07-26T06:47:23.093301+0000","last_undegraded":"2024-07-26T06:47:23.093301+0000","last_fullsized":"2024-07-26T06:47:23.093301+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:38.561268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090922+0000","last_change":"2024-07-26T06:47:05.027109+0000","last_active":"2024-07-26T06:47:23.090922+0000","last_peered":"2024-07-26T06:47:23.090922+0000","last_clean":"2024-07-26T06:47:23.090922+0000","last_became_active":"2024-07-26T06:47:05.026694+0000","last_became_peered":"2024-07-26T06:47:05.026694+0000","last_unstale":"2024-07-26T06:47:23.090922+0000","last_undegraded":"2024-07-26T06:47:23.090922+0000","last_fullsized":"2024-07-26T06:47:23.090922+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:25:01.215646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096756+0000","last_change":"2024-07-26T06:47:05.028212+0000","last_active":"2024-07-26T06:47:23.096756+0000","last_peered":"2024-07-26T06:47:23.096756+0000","last_clean":"2024-07-26T06:47:23.096756+0000","last_became_active":"2024-07-26T06:47:05.027598+0000","last_became_peered":"2024-07-26T06:47:05.027598+0000","last_unstale":"2024-07-26T06:47:23.096756+0000","last_undegraded":"2024-07-26T06:47:23.096756+0000","last_fullsized":"2024-07-26T06:47:23.096756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:55:31.536194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092258+0000","last_change":"2024-07-26T06:47:05.027724+0000","last_active":"2024-07-26T06:47:23.092258+0000","last_peered":"2024-07-26T06:47:23.092258+0000","last_clean":"2024-07-26T06:47:23.092258+0000","last_became_active":"2024-07-26T06:47:05.027348+0000","last_became_peered":"2024-07-26T06:47:05.027348+0000","last_unstale":"2024-07-26T06:47:23.092258+0000","last_undegraded":"2024-07-26T06:47:23.092258+0000","last_fullsized":"2024-07-26T06:47:23.092258+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:04.180467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097071+0000","last_change":"2024-07-26T06:47:05.025474+0000","last_active":"2024-07-26T06:47:23.097071+0000","last_peered":"2024-07-26T06:47:23.097071+0000","last_clean":"2024-07-26T06:47:23.097071+0000","last_became_active":"2024-07-26T06:47:05.025016+0000","last_became_peered":"2024-07-26T06:47:05.025016+0000","last_unstale":"2024-07-26T06:47:23.097071+0000","last_undegraded":"2024-07-26T06:47:23.097071+0000","last_fullsized":"2024-07-26T06:47:23.097071+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:40:03.034532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095635+0000","last_change":"2024-07-26T06:47:05.027533+0000","last_active":"2024-07-26T06:47:23.095635+0000","last_peered":"2024-07-26T06:47:23.095635+0000","last_clean":"2024-07-26T06:47:23.095635+0000","last_became_active":"2024-07-26T06:47:05.027143+0000","last_became_peered":"2024-07-26T06:47:05.027143+0000","last_unstale":"2024-07-26T06:47:23.095635+0000","last_undegraded":"2024-07-26T06:47:23.095635+0000","last_fullsized":"2024-07-26T06:47:23.095635+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:13.406430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095577+0000","last_change":"2024-07-26T06:47:05.025962+0000","last_active":"2024-07-26T06:47:23.095577+0000","last_peered":"2024-07-26T06:47:23.095577+0000","last_clean":"2024-07-26T06:47:23.095577+0000","last_became_active":"2024-07-26T06:47:05.025621+0000","last_became_peered":"2024-07-26T06:47:05.025621+0000","last_unstale":"2024-07-26T06:47:23.095577+0000","last_undegraded":"2024-07-26T06:47:23.095577+0000","last_fullsized":"2024-07-26T06:47:23.095577+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:56:43.669247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091492+0000","last_change":"2024-07-26T06:47:05.026252+0000","last_active":"2024-07-26T06:47:23.091492+0000","last_peered":"2024-07-26T06:47:23.091492+0000","last_clean":"2024-07-26T06:47:23.091492+0000","last_became_active":"2024-07-26T06:47:05.025875+0000","last_became_peered":"2024-07-26T06:47:05.025875+0000","last_unstale":"2024-07-26T06:47:23.091492+0000","last_undegraded":"2024-07-26T06:47:23.091492+0000","last_fullsized":"2024-07-26T06:47:23.091492+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:27:32.955862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090380+0000","last_change":"2024-07-26T06:47:05.022513+0000","last_active":"2024-07-26T06:47:23.090380+0000","last_peered":"2024-07-26T06:47:23.090380+0000","last_clean":"2024-07-26T06:47:23.090380+0000","last_became_active":"2024-07-26T06:47:05.022159+0000","last_became_peered":"2024-07-26T06:47:05.022159+0000","last_unstale":"2024-07-26T06:47:23.090380+0000","last_undegraded":"2024-07-26T06:47:23.090380+0000","last_fullsized":"2024-07-26T06:47:23.090380+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:53:06.402985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.098283+0000","last_change":"2024-07-26T06:47:23.097604+0000","last_active":"2024-07-26T06:47:23.098283+0000","last_peered":"2024-07-26T06:47:23.098283+0000","last_clean":"2024-07-26T06:47:23.098283+0000","last_became_active":"2024-07-26T06:47:23.097170+0000","last_became_peered":"2024-07-26T06:47:23.097170+0000","last_unstale":"2024-07-26T06:47:23.098283+0000","last_undegraded":"2024-07-26T06:47:23.098283+0000","last_fullsized":"2024-07-26T06:47:23.098283+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:38:24.530396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090964+0000","last_change":"2024-07-26T06:47:05.026349+0000","last_active":"2024-07-26T06:47:23.090964+0000","last_peered":"2024-07-26T06:47:23.090964+0000","last_clean":"2024-07-26T06:47:23.090964+0000","last_became_active":"2024-07-26T06:47:05.026016+0000","last_became_peered":"2024-07-26T06:47:05.026016+0000","last_unstale":"2024-07-26T06:47:23.090964+0000","last_undegraded":"2024-07-26T06:47:23.090964+0000","last_fullsized":"2024-07-26T06:47:23.090964+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:19:02.264183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097162+0000","last_change":"2024-07-26T06:47:05.032439+0000","last_active":"2024-07-26T06:47:23.097162+0000","last_peered":"2024-07-26T06:47:23.097162+0000","last_clean":"2024-07-26T06:47:23.097162+0000","last_became_active":"2024-07-26T06:47:05.032105+0000","last_became_peered":"2024-07-26T06:47:05.032105+0000","last_unstale":"2024-07-26T06:47:23.097162+0000","last_undegraded":"2024-07-26T06:47:23.097162+0000","last_fullsized":"2024-07-26T06:47:23.097162+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:21:50.423939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096506+0000","last_change":"2024-07-26T06:47:05.026927+0000","last_active":"2024-07-26T06:47:23.096506+0000","last_peered":"2024-07-26T06:47:23.096506+0000","last_clean":"2024-07-26T06:47:23.096506+0000","last_became_active":"2024-07-26T06:47:05.026569+0000","last_became_peered":"2024-07-26T06:47:05.026569+0000","last_unstale":"2024-07-26T06:47:23.096506+0000","last_undegraded":"2024-07-26T06:47:23.096506+0000","last_fullsized":"2024-07-26T06:47:23.096506+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:12:14.687633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.099776+0000","last_change":"2024-07-26T06:47:23.099689+0000","last_active":"2024-07-26T06:47:23.099776+0000","last_peered":"2024-07-26T06:47:23.099776+0000","last_clean":"2024-07-26T06:47:23.099776+0000","last_became_active":"2024-07-26T06:47:23.099313+0000","last_became_peered":"2024-07-26T06:47:23.099313+0000","last_unstale":"2024-07-26T06:47:23.099776+0000","last_undegraded":"2024-07-26T06:47:23.099776+0000","last_fullsized":"2024-07-26T06:47:23.099776+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:49:01.833297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095052+0000","last_change":"2024-07-26T06:47:05.034864+0000","last_active":"2024-07-26T06:47:23.095052+0000","last_peered":"2024-07-26T06:47:23.095052+0000","last_clean":"2024-07-26T06:47:23.095052+0000","last_became_active":"2024-07-26T06:47:05.033414+0000","last_became_peered":"2024-07-26T06:47:05.033414+0000","last_unstale":"2024-07-26T06:47:23.095052+0000","last_undegraded":"2024-07-26T06:47:23.095052+0000","last_fullsized":"2024-07-26T06:47:23.095052+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:21:31.420212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096728+0000","last_change":"2024-07-26T06:47:05.025462+0000","last_active":"2024-07-26T06:47:23.096728+0000","last_peered":"2024-07-26T06:47:23.096728+0000","last_clean":"2024-07-26T06:47:23.096728+0000","last_became_active":"2024-07-26T06:47:05.025049+0000","last_became_peered":"2024-07-26T06:47:05.025049+0000","last_unstale":"2024-07-26T06:47:23.096728+0000","last_undegraded":"2024-07-26T06:47:23.096728+0000","last_fullsized":"2024-07-26T06:47:23.096728+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:59:28.903338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097665+0000","last_change":"2024-07-26T06:47:05.026394+0000","last_active":"2024-07-26T06:47:23.097665+0000","last_peered":"2024-07-26T06:47:23.097665+0000","last_clean":"2024-07-26T06:47:23.097665+0000","last_became_active":"2024-07-26T06:47:05.025820+0000","last_became_peered":"2024-07-26T06:47:05.025820+0000","last_unstale":"2024-07-26T06:47:23.097665+0000","last_undegraded":"2024-07-26T06:47:23.097665+0000","last_fullsized":"2024-07-26T06:47:23.097665+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:44:57.441638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095529+0000","last_change":"2024-07-26T06:47:05.024373+0000","last_active":"2024-07-26T06:47:23.095529+0000","last_peered":"2024-07-26T06:47:23.095529+0000","last_clean":"2024-07-26T06:47:23.095529+0000","last_became_active":"2024-07-26T06:47:05.023962+0000","last_became_peered":"2024-07-26T06:47:05.023962+0000","last_unstale":"2024-07-26T06:47:23.095529+0000","last_undegraded":"2024-07-26T06:47:23.095529+0000","last_fullsized":"2024-07-26T06:47:23.095529+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:43.804614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092705+0000","last_change":"2024-07-26T06:47:05.031146+0000","last_active":"2024-07-26T06:47:23.092705+0000","last_peered":"2024-07-26T06:47:23.092705+0000","last_clean":"2024-07-26T06:47:23.092705+0000","last_became_active":"2024-07-26T06:47:05.029393+0000","last_became_peered":"2024-07-26T06:47:05.029393+0000","last_unstale":"2024-07-26T06:47:23.092705+0000","last_undegraded":"2024-07-26T06:47:23.092705+0000","last_fullsized":"2024-07-26T06:47:23.092705+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:46.853933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"80'6","reported_seq":58,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:45.156905+0000","last_change":"2024-07-26T06:47:02.001800+0000","last_active":"2024-07-26T06:47:45.156905+0000","last_peered":"2024-07-26T06:47:45.156905+0000","last_clean":"2024-07-26T06:47:45.156905+0000","last_became_active":"2024-07-26T06:47:02.001435+0000","last_became_peered":"2024-07-26T06:47:02.001435+0000","last_unstale":"2024-07-26T06:47:45.156905+0000","last_undegraded":"2024-07-26T06:47:45.156905+0000","last_fullsized":"2024-07-26T06:47:45.156905+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:23.121378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.094853+0000","last_change":"2024-07-26T06:47:05.028301+0000","last_active":"2024-07-26T06:47:23.094853+0000","last_peered":"2024-07-26T06:47:23.094853+0000","last_clean":"2024-07-26T06:47:23.094853+0000","last_became_active":"2024-07-26T06:47:05.027761+0000","last_became_peered":"2024-07-26T06:47:05.027761+0000","last_unstale":"2024-07-26T06:47:23.094853+0000","last_undegraded":"2024-07-26T06:47:23.094853+0000","last_fullsized":"2024-07-26T06:47:23.094853+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:19.109395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093760+0000","last_change":"2024-07-26T06:47:02.004488+0000","last_active":"2024-07-26T06:47:23.093760+0000","last_peered":"2024-07-26T06:47:23.093760+0000","last_clean":"2024-07-26T06:47:23.093760+0000","last_became_active":"2024-07-26T06:47:02.004163+0000","last_became_peered":"2024-07-26T06:47:02.004163+0000","last_unstale":"2024-07-26T06:47:23.093760+0000","last_undegraded":"2024-07-26T06:47:23.093760+0000","last_fullsized":"2024-07-26T06:47:23.093760+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:16.221618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096069+0000","last_change":"2024-07-26T06:47:01.992708+0000","last_active":"2024-07-26T06:47:23.096069+0000","last_peered":"2024-07-26T06:47:23.096069+0000","last_clean":"2024-07-26T06:47:23.096069+0000","last_became_active":"2024-07-26T06:47:01.992330+0000","last_became_peered":"2024-07-26T06:47:01.992330+0000","last_unstale":"2024-07-26T06:47:23.096069+0000","last_undegraded":"2024-07-26T06:47:23.096069+0000","last_fullsized":"2024-07-26T06:47:23.096069+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:25.951946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090325+0000","last_change":"2024-07-26T06:47:05.013946+0000","last_active":"2024-07-26T06:47:23.090325+0000","last_peered":"2024-07-26T06:47:23.090325+0000","last_clean":"2024-07-26T06:47:23.090325+0000","last_became_active":"2024-07-26T06:47:05.013509+0000","last_became_peered":"2024-07-26T06:47:05.013509+0000","last_unstale":"2024-07-26T06:47:23.090325+0000","last_undegraded":"2024-07-26T06:47:23.090325+0000","last_fullsized":"2024-07-26T06:47:23.090325+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:09.789287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093848+0000","last_change":"2024-07-26T06:47:01.997617+0000","last_active":"2024-07-26T06:47:23.093848+0000","last_peered":"2024-07-26T06:47:23.093848+0000","last_clean":"2024-07-26T06:47:23.093848+0000","last_became_active":"2024-07-26T06:47:01.997238+0000","last_became_peered":"2024-07-26T06:47:01.997238+0000","last_unstale":"2024-07-26T06:47:23.093848+0000","last_undegraded":"2024-07-26T06:47:23.093848+0000","last_fullsized":"2024-07-26T06:47:23.093848+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:45.596438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.099897+0000","last_change":"2024-07-26T06:47:23.099813+0000","last_active":"2024-07-26T06:47:23.099897+0000","last_peered":"2024-07-26T06:47:23.099897+0000","last_clean":"2024-07-26T06:47:23.099897+0000","last_became_active":"2024-07-26T06:47:23.098701+0000","last_became_peered":"2024-07-26T06:47:23.098701+0000","last_unstale":"2024-07-26T06:47:23.099897+0000","last_undegraded":"2024-07-26T06:47:23.099897+0000","last_fullsized":"2024-07-26T06:47:23.099897+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:42:04.059637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"78'4","reported_seq":80,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:48.160574+0000","last_change":"2024-07-26T06:47:01.998508+0000","last_active":"2024-07-26T06:47:48.160574+0000","last_peered":"2024-07-26T06:47:48.160574+0000","last_clean":"2024-07-26T06:47:48.160574+0000","last_became_active":"2024-07-26T06:47:01.998132+0000","last_became_peered":"2024-07-26T06:47:01.998132+0000","last_unstale":"2024-07-26T06:47:48.160574+0000","last_undegraded":"2024-07-26T06:47:48.160574+0000","last_fullsized":"2024-07-26T06:47:48.160574+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:21.588085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":50,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091351+0000","last_change":"2024-07-26T06:47:05.032704+0000","last_active":"2024-07-26T06:47:23.091351+0000","last_peered":"2024-07-26T06:47:23.091351+0000","last_clean":"2024-07-26T06:47:23.091351+0000","last_became_active":"2024-07-26T06:47:05.031843+0000","last_became_peered":"2024-07-26T06:47:05.031843+0000","last_unstale":"2024-07-26T06:47:23.091351+0000","last_undegraded":"2024-07-26T06:47:23.091351+0000","last_fullsized":"2024-07-26T06:47:23.091351+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:58:24.436233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097398+0000","last_change":"2024-07-26T06:47:02.006266+0000","last_active":"2024-07-26T06:47:23.097398+0000","last_peered":"2024-07-26T06:47:23.097398+0000","last_clean":"2024-07-26T06:47:23.097398+0000","last_became_active":"2024-07-26T06:47:02.005830+0000","last_became_peered":"2024-07-26T06:47:02.005830+0000","last_unstale":"2024-07-26T06:47:23.097398+0000","last_undegraded":"2024-07-26T06:47:23.097398+0000","last_fullsized":"2024-07-26T06:47:23.097398+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:06:09.097781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.094830+0000","last_change":"2024-07-26T06:47:05.024820+0000","last_active":"2024-07-26T06:47:23.094830+0000","last_peered":"2024-07-26T06:47:23.094830+0000","last_clean":"2024-07-26T06:47:23.094830+0000","last_became_active":"2024-07-26T06:47:05.023941+0000","last_became_peered":"2024-07-26T06:47:05.023941+0000","last_unstale":"2024-07-26T06:47:23.094830+0000","last_undegraded":"2024-07-26T06:47:23.094830+0000","last_fullsized":"2024-07-26T06:47:23.094830+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:07.885159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095379+0000","last_change":"2024-07-26T06:47:02.023308+0000","last_active":"2024-07-26T06:47:23.095379+0000","last_peered":"2024-07-26T06:47:23.095379+0000","last_clean":"2024-07-26T06:47:23.095379+0000","last_became_active":"2024-07-26T06:47:02.022933+0000","last_became_peered":"2024-07-26T06:47:02.022933+0000","last_unstale":"2024-07-26T06:47:23.095379+0000","last_undegraded":"2024-07-26T06:47:23.095379+0000","last_fullsized":"2024-07-26T06:47:23.095379+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:59:01.233386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091075+0000","last_change":"2024-07-26T06:47:05.026005+0000","last_active":"2024-07-26T06:47:23.091075+0000","last_peered":"2024-07-26T06:47:23.091075+0000","last_clean":"2024-07-26T06:47:23.091075+0000","last_became_active":"2024-07-26T06:47:05.025599+0000","last_became_peered":"2024-07-26T06:47:05.025599+0000","last_unstale":"2024-07-26T06:47:23.091075+0000","last_undegraded":"2024-07-26T06:47:23.091075+0000","last_fullsized":"2024-07-26T06:47:23.091075+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:44:19.924157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'29","reported_seq":132,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:48.163072+0000","last_change":"2024-07-26T06:47:23.393050+0000","last_active":"2024-07-26T06:47:48.163072+0000","last_peered":"2024-07-26T06:47:48.163072+0000","last_clean":"2024-07-26T06:47:48.163072+0000","last_became_active":"2024-07-26T06:47:23.381611+0000","last_became_peered":"2024-07-26T06:47:23.381611+0000","last_unstale":"2024-07-26T06:47:48.163072+0000","last_undegraded":"2024-07-26T06:47:48.163072+0000","last_fullsized":"2024-07-26T06:47:48.163072+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:23.866523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55581,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":58,"num_read_kb":82,"num_write":31,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092605+0000","last_change":"2024-07-26T06:47:05.034190+0000","last_active":"2024-07-26T06:47:23.092605+0000","last_peered":"2024-07-26T06:47:23.092605+0000","last_clean":"2024-07-26T06:47:23.092605+0000","last_became_active":"2024-07-26T06:47:05.033569+0000","last_became_peered":"2024-07-26T06:47:05.033569+0000","last_unstale":"2024-07-26T06:47:23.092605+0000","last_undegraded":"2024-07-26T06:47:23.092605+0000","last_fullsized":"2024-07-26T06:47:23.092605+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:39.306996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.120074+0000","last_change":"2024-07-26T06:47:02.001718+0000","last_active":"2024-07-26T06:47:24.120074+0000","last_peered":"2024-07-26T06:47:24.120074+0000","last_clean":"2024-07-26T06:47:24.120074+0000","last_became_active":"2024-07-26T06:47:02.001359+0000","last_became_peered":"2024-07-26T06:47:02.001359+0000","last_unstale":"2024-07-26T06:47:24.120074+0000","last_undegraded":"2024-07-26T06:47:24.120074+0000","last_fullsized":"2024-07-26T06:47:24.120074+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:03:23.390279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095009+0000","last_change":"2024-07-26T06:47:05.027445+0000","last_active":"2024-07-26T06:47:23.095009+0000","last_peered":"2024-07-26T06:47:23.095009+0000","last_clean":"2024-07-26T06:47:23.095009+0000","last_became_active":"2024-07-26T06:47:05.027114+0000","last_became_peered":"2024-07-26T06:47:05.027114+0000","last_unstale":"2024-07-26T06:47:23.095009+0000","last_undegraded":"2024-07-26T06:47:23.095009+0000","last_fullsized":"2024-07-26T06:47:23.095009+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:29:10.519271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.094571+0000","last_change":"2024-07-26T06:47:01.998832+0000","last_active":"2024-07-26T06:47:23.094571+0000","last_peered":"2024-07-26T06:47:23.094571+0000","last_clean":"2024-07-26T06:47:23.094571+0000","last_became_active":"2024-07-26T06:47:01.998512+0000","last_became_peered":"2024-07-26T06:47:01.998512+0000","last_unstale":"2024-07-26T06:47:23.094571+0000","last_undegraded":"2024-07-26T06:47:23.094571+0000","last_fullsized":"2024-07-26T06:47:23.094571+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:27.722329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097582+0000","last_change":"2024-07-26T06:47:05.027803+0000","last_active":"2024-07-26T06:47:23.097582+0000","last_peered":"2024-07-26T06:47:23.097582+0000","last_clean":"2024-07-26T06:47:23.097582+0000","last_became_active":"2024-07-26T06:47:05.027320+0000","last_became_peered":"2024-07-26T06:47:05.027320+0000","last_unstale":"2024-07-26T06:47:23.097582+0000","last_undegraded":"2024-07-26T06:47:23.097582+0000","last_fullsized":"2024-07-26T06:47:23.097582+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:43:06.242945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'6","reported_seq":61,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:47.160230+0000","last_change":"2024-07-26T06:47:01.998356+0000","last_active":"2024-07-26T06:47:47.160230+0000","last_peered":"2024-07-26T06:47:47.160230+0000","last_clean":"2024-07-26T06:47:47.160230+0000","last_became_active":"2024-07-26T06:47:01.997922+0000","last_became_peered":"2024-07-26T06:47:01.997922+0000","last_unstale":"2024-07-26T06:47:47.160230+0000","last_undegraded":"2024-07-26T06:47:47.160230+0000","last_fullsized":"2024-07-26T06:47:47.160230+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:50:58.066682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096946+0000","last_change":"2024-07-26T06:47:05.025285+0000","last_active":"2024-07-26T06:47:23.096946+0000","last_peered":"2024-07-26T06:47:23.096946+0000","last_clean":"2024-07-26T06:47:23.096946+0000","last_became_active":"2024-07-26T06:47:05.024945+0000","last_became_peered":"2024-07-26T06:47:05.024945+0000","last_unstale":"2024-07-26T06:47:23.096946+0000","last_undegraded":"2024-07-26T06:47:23.096946+0000","last_fullsized":"2024-07-26T06:47:23.096946+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:21:04.680435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093595+0000","last_change":"2024-07-26T06:47:02.024895+0000","last_active":"2024-07-26T06:47:23.093595+0000","last_peered":"2024-07-26T06:47:23.093595+0000","last_clean":"2024-07-26T06:47:23.093595+0000","last_became_active":"2024-07-26T06:47:02.024451+0000","last_became_peered":"2024-07-26T06:47:02.024451+0000","last_unstale":"2024-07-26T06:47:23.093595+0000","last_undegraded":"2024-07-26T06:47:23.093595+0000","last_fullsized":"2024-07-26T06:47:23.093595+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:05:13.396499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.381402+0000","last_change":"2024-07-26T06:47:23.381060+0000","last_active":"2024-07-26T06:47:23.381402+0000","last_peered":"2024-07-26T06:47:23.381402+0000","last_clean":"2024-07-26T06:47:23.381402+0000","last_became_active":"2024-07-26T06:47:23.380685+0000","last_became_peered":"2024-07-26T06:47:23.380685+0000","last_unstale":"2024-07-26T06:47:23.381402+0000","last_undegraded":"2024-07-26T06:47:23.381402+0000","last_fullsized":"2024-07-26T06:47:23.381402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:21:38.062900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093109+0000","last_change":"2024-07-26T06:47:02.002098+0000","last_active":"2024-07-26T06:47:23.093109+0000","last_peered":"2024-07-26T06:47:23.093109+0000","last_clean":"2024-07-26T06:47:23.093109+0000","last_became_active":"2024-07-26T06:47:02.001428+0000","last_became_peered":"2024-07-26T06:47:02.001428+0000","last_unstale":"2024-07-26T06:47:23.093109+0000","last_undegraded":"2024-07-26T06:47:23.093109+0000","last_fullsized":"2024-07-26T06:47:23.093109+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:52:29.112500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096258+0000","last_change":"2024-07-26T06:47:05.023686+0000","last_active":"2024-07-26T06:47:23.096258+0000","last_peered":"2024-07-26T06:47:23.096258+0000","last_clean":"2024-07-26T06:47:23.096258+0000","last_became_active":"2024-07-26T06:47:05.022583+0000","last_became_peered":"2024-07-26T06:47:05.022583+0000","last_unstale":"2024-07-26T06:47:23.096258+0000","last_undegraded":"2024-07-26T06:47:23.096258+0000","last_fullsized":"2024-07-26T06:47:23.096258+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:29:03.752429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096434+0000","last_change":"2024-07-26T06:47:02.006040+0000","last_active":"2024-07-26T06:47:23.096434+0000","last_peered":"2024-07-26T06:47:23.096434+0000","last_clean":"2024-07-26T06:47:23.096434+0000","last_became_active":"2024-07-26T06:47:02.005586+0000","last_became_peered":"2024-07-26T06:47:02.005586+0000","last_unstale":"2024-07-26T06:47:23.096434+0000","last_undegraded":"2024-07-26T06:47:23.096434+0000","last_fullsized":"2024-07-26T06:47:23.096434+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:09:27.180873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097475+0000","last_change":"2024-07-26T06:47:05.035524+0000","last_active":"2024-07-26T06:47:23.097475+0000","last_peered":"2024-07-26T06:47:23.097475+0000","last_clean":"2024-07-26T06:47:23.097475+0000","last_became_active":"2024-07-26T06:47:05.035078+0000","last_became_peered":"2024-07-26T06:47:05.035078+0000","last_unstale":"2024-07-26T06:47:23.097475+0000","last_undegraded":"2024-07-26T06:47:23.097475+0000","last_fullsized":"2024-07-26T06:47:23.097475+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:03.758261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095730+0000","last_change":"2024-07-26T06:47:01.999810+0000","last_active":"2024-07-26T06:47:23.095730+0000","last_peered":"2024-07-26T06:47:23.095730+0000","last_clean":"2024-07-26T06:47:23.095730+0000","last_became_active":"2024-07-26T06:47:01.999460+0000","last_became_peered":"2024-07-26T06:47:01.999460+0000","last_unstale":"2024-07-26T06:47:23.095730+0000","last_undegraded":"2024-07-26T06:47:23.095730+0000","last_fullsized":"2024-07-26T06:47:23.095730+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:25:05.691900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'1","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097358+0000","last_change":"2024-07-26T06:47:05.033631+0000","last_active":"2024-07-26T06:47:23.097358+0000","last_peered":"2024-07-26T06:47:23.097358+0000","last_clean":"2024-07-26T06:47:23.097358+0000","last_became_active":"2024-07-26T06:47:05.033266+0000","last_became_peered":"2024-07-26T06:47:05.033266+0000","last_unstale":"2024-07-26T06:47:23.097358+0000","last_undegraded":"2024-07-26T06:47:23.097358+0000","last_fullsized":"2024-07-26T06:47:23.097358+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:55.895271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"78'5","reported_seq":62,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:50.164406+0000","last_change":"2024-07-26T06:47:02.113525+0000","last_active":"2024-07-26T06:47:50.164406+0000","last_peered":"2024-07-26T06:47:50.164406+0000","last_clean":"2024-07-26T06:47:50.164406+0000","last_became_active":"2024-07-26T06:47:02.113112+0000","last_became_peered":"2024-07-26T06:47:02.113112+0000","last_unstale":"2024-07-26T06:47:50.164406+0000","last_undegraded":"2024-07-26T06:47:50.164406+0000","last_fullsized":"2024-07-26T06:47:50.164406+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:05:15.071264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097372+0000","last_change":"2024-07-26T06:47:05.026423+0000","last_active":"2024-07-26T06:47:23.097372+0000","last_peered":"2024-07-26T06:47:23.097372+0000","last_clean":"2024-07-26T06:47:23.097372+0000","last_became_active":"2024-07-26T06:47:05.026053+0000","last_became_peered":"2024-07-26T06:47:05.026053+0000","last_unstale":"2024-07-26T06:47:23.097372+0000","last_undegraded":"2024-07-26T06:47:23.097372+0000","last_fullsized":"2024-07-26T06:47:23.097372+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:07:20.174538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":101,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097187+0000","last_change":"2024-07-26T06:45:40.184988+0000","last_active":"2024-07-26T06:47:23.097187+0000","last_peered":"2024-07-26T06:47:23.097187+0000","last_clean":"2024-07-26T06:47:23.097187+0000","last_became_active":"2024-07-26T06:45:40.178827+0000","last_became_peered":"2024-07-26T06:45:40.178827+0000","last_unstale":"2024-07-26T06:47:23.097187+0000","last_undegraded":"2024-07-26T06:47:23.097187+0000","last_fullsized":"2024-07-26T06:47:23.097187+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:42:28.257602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:42:28.257602+0000","last_clean_scrub_stamp":"2024-07-26T06:42:28.257602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:52:14.123696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.120218+0000","last_change":"2024-07-26T06:47:01.996610+0000","last_active":"2024-07-26T06:47:24.120218+0000","last_peered":"2024-07-26T06:47:24.120218+0000","last_clean":"2024-07-26T06:47:24.120218+0000","last_became_active":"2024-07-26T06:47:01.996155+0000","last_became_peered":"2024-07-26T06:47:01.996155+0000","last_unstale":"2024-07-26T06:47:24.120218+0000","last_undegraded":"2024-07-26T06:47:24.120218+0000","last_fullsized":"2024-07-26T06:47:24.120218+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:51:30.420254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090885+0000","last_change":"2024-07-26T06:47:05.027699+0000","last_active":"2024-07-26T06:47:23.090885+0000","last_peered":"2024-07-26T06:47:23.090885+0000","last_clean":"2024-07-26T06:47:23.090885+0000","last_became_active":"2024-07-26T06:47:05.027348+0000","last_became_peered":"2024-07-26T06:47:05.027348+0000","last_unstale":"2024-07-26T06:47:23.090885+0000","last_undegraded":"2024-07-26T06:47:23.090885+0000","last_fullsized":"2024-07-26T06:47:23.090885+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:28:42.664351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097120+0000","last_change":"2024-07-26T06:47:02.003205+0000","last_active":"2024-07-26T06:47:23.097120+0000","last_peered":"2024-07-26T06:47:23.097120+0000","last_clean":"2024-07-26T06:47:23.097120+0000","last_became_active":"2024-07-26T06:47:02.002543+0000","last_became_peered":"2024-07-26T06:47:02.002543+0000","last_unstale":"2024-07-26T06:47:23.097120+0000","last_undegraded":"2024-07-26T06:47:23.097120+0000","last_fullsized":"2024-07-26T06:47:23.097120+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:43:10.911943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095958+0000","last_change":"2024-07-26T06:47:05.023258+0000","last_active":"2024-07-26T06:47:23.095958+0000","last_peered":"2024-07-26T06:47:23.095958+0000","last_clean":"2024-07-26T06:47:23.095958+0000","last_became_active":"2024-07-26T06:47:05.022903+0000","last_became_peered":"2024-07-26T06:47:05.022903+0000","last_unstale":"2024-07-26T06:47:23.095958+0000","last_undegraded":"2024-07-26T06:47:23.095958+0000","last_fullsized":"2024-07-26T06:47:23.095958+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:08.155850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":55,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:45.157023+0000","last_change":"2024-07-26T06:47:02.022316+0000","last_active":"2024-07-26T06:47:45.157023+0000","last_peered":"2024-07-26T06:47:45.157023+0000","last_clean":"2024-07-26T06:47:45.157023+0000","last_became_active":"2024-07-26T06:47:02.021923+0000","last_became_peered":"2024-07-26T06:47:02.021923+0000","last_unstale":"2024-07-26T06:47:45.157023+0000","last_undegraded":"2024-07-26T06:47:45.157023+0000","last_fullsized":"2024-07-26T06:47:45.157023+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:29:31.533449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093615+0000","last_change":"2024-07-26T06:47:05.021510+0000","last_active":"2024-07-26T06:47:23.093615+0000","last_peered":"2024-07-26T06:47:23.093615+0000","last_clean":"2024-07-26T06:47:23.093615+0000","last_became_active":"2024-07-26T06:47:05.021140+0000","last_became_peered":"2024-07-26T06:47:05.021140+0000","last_unstale":"2024-07-26T06:47:23.093615+0000","last_undegraded":"2024-07-26T06:47:23.093615+0000","last_fullsized":"2024-07-26T06:47:23.093615+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:37:01.340126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"79'5","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:47.158885+0000","last_change":"2024-07-26T06:47:02.001698+0000","last_active":"2024-07-26T06:47:47.158885+0000","last_peered":"2024-07-26T06:47:47.158885+0000","last_clean":"2024-07-26T06:47:47.158885+0000","last_became_active":"2024-07-26T06:47:02.001244+0000","last_became_peered":"2024-07-26T06:47:02.001244+0000","last_unstale":"2024-07-26T06:47:47.158885+0000","last_undegraded":"2024-07-26T06:47:47.158885+0000","last_fullsized":"2024-07-26T06:47:47.158885+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:32:07.920065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093652+0000","last_change":"2024-07-26T06:47:05.029524+0000","last_active":"2024-07-26T06:47:23.093652+0000","last_peered":"2024-07-26T06:47:23.093652+0000","last_clean":"2024-07-26T06:47:23.093652+0000","last_became_active":"2024-07-26T06:47:05.028981+0000","last_became_peered":"2024-07-26T06:47:05.028981+0000","last_unstale":"2024-07-26T06:47:23.093652+0000","last_undegraded":"2024-07-26T06:47:23.093652+0000","last_fullsized":"2024-07-26T06:47:23.093652+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:49:57.019486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092276+0000","last_change":"2024-07-26T06:47:02.025117+0000","last_active":"2024-07-26T06:47:23.092276+0000","last_peered":"2024-07-26T06:47:23.092276+0000","last_clean":"2024-07-26T06:47:23.092276+0000","last_became_active":"2024-07-26T06:47:02.024612+0000","last_became_peered":"2024-07-26T06:47:02.024612+0000","last_unstale":"2024-07-26T06:47:23.092276+0000","last_undegraded":"2024-07-26T06:47:23.092276+0000","last_fullsized":"2024-07-26T06:47:23.092276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:15:44.999658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'1","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.094137+0000","last_change":"2024-07-26T06:47:05.015085+0000","last_active":"2024-07-26T06:47:23.094137+0000","last_peered":"2024-07-26T06:47:23.094137+0000","last_clean":"2024-07-26T06:47:23.094137+0000","last_became_active":"2024-07-26T06:47:05.014686+0000","last_became_peered":"2024-07-26T06:47:05.014686+0000","last_unstale":"2024-07-26T06:47:23.094137+0000","last_undegraded":"2024-07-26T06:47:23.094137+0000","last_fullsized":"2024-07-26T06:47:23.094137+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:39:30.392347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"80'4","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.159121+0000","last_change":"2024-07-26T06:47:23.111918+0000","last_active":"2024-07-26T06:47:23.159121+0000","last_peered":"2024-07-26T06:47:23.159121+0000","last_clean":"2024-07-26T06:47:23.159121+0000","last_became_active":"2024-07-26T06:47:23.099518+0000","last_became_peered":"2024-07-26T06:47:23.099518+0000","last_unstale":"2024-07-26T06:47:23.159121+0000","last_undegraded":"2024-07-26T06:47:23.159121+0000","last_fullsized":"2024-07-26T06:47:23.159121+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:41.012096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096626+0000","last_change":"2024-07-26T06:47:05.019415+0000","last_active":"2024-07-26T06:47:23.096626+0000","last_peered":"2024-07-26T06:47:23.096626+0000","last_clean":"2024-07-26T06:47:23.096626+0000","last_became_active":"2024-07-26T06:47:05.019044+0000","last_became_peered":"2024-07-26T06:47:05.019044+0000","last_unstale":"2024-07-26T06:47:23.096626+0000","last_undegraded":"2024-07-26T06:47:23.096626+0000","last_fullsized":"2024-07-26T06:47:23.096626+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:02:37.228174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'14","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:44.026778+0000","last_change":"2024-07-26T06:47:02.006585+0000","last_active":"2024-07-26T06:47:44.026778+0000","last_peered":"2024-07-26T06:47:44.026778+0000","last_clean":"2024-07-26T06:47:44.026778+0000","last_became_active":"2024-07-26T06:47:02.006185+0000","last_became_peered":"2024-07-26T06:47:02.006185+0000","last_unstale":"2024-07-26T06:47:44.026778+0000","last_undegraded":"2024-07-26T06:47:44.026778+0000","last_fullsized":"2024-07-26T06:47:44.026778+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:02:45.480712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091090+0000","last_change":"2024-07-26T06:47:05.028840+0000","last_active":"2024-07-26T06:47:23.091090+0000","last_peered":"2024-07-26T06:47:23.091090+0000","last_clean":"2024-07-26T06:47:23.091090+0000","last_became_active":"2024-07-26T06:47:05.028487+0000","last_became_peered":"2024-07-26T06:47:05.028487+0000","last_unstale":"2024-07-26T06:47:23.091090+0000","last_undegraded":"2024-07-26T06:47:23.091090+0000","last_fullsized":"2024-07-26T06:47:23.091090+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:47:48.185227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.380389+0000","last_change":"2024-07-26T06:47:02.113691+0000","last_active":"2024-07-26T06:47:23.380389+0000","last_peered":"2024-07-26T06:47:23.380389+0000","last_clean":"2024-07-26T06:47:23.380389+0000","last_became_active":"2024-07-26T06:47:02.113342+0000","last_became_peered":"2024-07-26T06:47:02.113342+0000","last_unstale":"2024-07-26T06:47:23.380389+0000","last_undegraded":"2024-07-26T06:47:23.380389+0000","last_fullsized":"2024-07-26T06:47:23.380389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:14:03.586564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.098503+0000","last_change":"2024-07-26T06:47:23.098420+0000","last_active":"2024-07-26T06:47:23.098503+0000","last_peered":"2024-07-26T06:47:23.098503+0000","last_clean":"2024-07-26T06:47:23.098503+0000","last_became_active":"2024-07-26T06:47:23.097981+0000","last_became_peered":"2024-07-26T06:47:23.097981+0000","last_unstale":"2024-07-26T06:47:23.098503+0000","last_undegraded":"2024-07-26T06:47:23.098503+0000","last_fullsized":"2024-07-26T06:47:23.098503+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:38:40.155514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091072+0000","last_change":"2024-07-26T06:47:02.024781+0000","last_active":"2024-07-26T06:47:23.091072+0000","last_peered":"2024-07-26T06:47:23.091072+0000","last_clean":"2024-07-26T06:47:23.091072+0000","last_became_active":"2024-07-26T06:47:02.024492+0000","last_became_peered":"2024-07-26T06:47:02.024492+0000","last_unstale":"2024-07-26T06:47:23.091072+0000","last_undegraded":"2024-07-26T06:47:23.091072+0000","last_fullsized":"2024-07-26T06:47:23.091072+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:23:41.323963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092602+0000","last_change":"2024-07-26T06:47:05.033052+0000","last_active":"2024-07-26T06:47:23.092602+0000","last_peered":"2024-07-26T06:47:23.092602+0000","last_clean":"2024-07-26T06:47:23.092602+0000","last_became_active":"2024-07-26T06:47:05.032052+0000","last_became_peered":"2024-07-26T06:47:05.032052+0000","last_unstale":"2024-07-26T06:47:23.092602+0000","last_undegraded":"2024-07-26T06:47:23.092602+0000","last_fullsized":"2024-07-26T06:47:23.092602+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:42:04.683586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090564+0000","last_change":"2024-07-26T06:47:01.996097+0000","last_active":"2024-07-26T06:47:23.090564+0000","last_peered":"2024-07-26T06:47:23.090564+0000","last_clean":"2024-07-26T06:47:23.090564+0000","last_became_active":"2024-07-26T06:47:01.995594+0000","last_became_peered":"2024-07-26T06:47:01.995594+0000","last_unstale":"2024-07-26T06:47:23.090564+0000","last_undegraded":"2024-07-26T06:47:23.090564+0000","last_fullsized":"2024-07-26T06:47:23.090564+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:34.969493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095563+0000","last_change":"2024-07-26T06:47:05.024171+0000","last_active":"2024-07-26T06:47:23.095563+0000","last_peered":"2024-07-26T06:47:23.095563+0000","last_clean":"2024-07-26T06:47:23.095563+0000","last_became_active":"2024-07-26T06:47:05.022888+0000","last_became_peered":"2024-07-26T06:47:05.022888+0000","last_unstale":"2024-07-26T06:47:23.095563+0000","last_undegraded":"2024-07-26T06:47:23.095563+0000","last_fullsized":"2024-07-26T06:47:23.095563+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:31:52.691068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096894+0000","last_change":"2024-07-26T06:47:02.004578+0000","last_active":"2024-07-26T06:47:23.096894+0000","last_peered":"2024-07-26T06:47:23.096894+0000","last_clean":"2024-07-26T06:47:23.096894+0000","last_became_active":"2024-07-26T06:47:02.004231+0000","last_became_peered":"2024-07-26T06:47:02.004231+0000","last_unstale":"2024-07-26T06:47:23.096894+0000","last_undegraded":"2024-07-26T06:47:23.096894+0000","last_fullsized":"2024-07-26T06:47:23.096894+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:43:39.803817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091223+0000","last_change":"2024-07-26T06:47:05.024232+0000","last_active":"2024-07-26T06:47:23.091223+0000","last_peered":"2024-07-26T06:47:23.091223+0000","last_clean":"2024-07-26T06:47:23.091223+0000","last_became_active":"2024-07-26T06:47:05.023869+0000","last_became_peered":"2024-07-26T06:47:05.023869+0000","last_unstale":"2024-07-26T06:47:23.091223+0000","last_undegraded":"2024-07-26T06:47:23.091223+0000","last_fullsized":"2024-07-26T06:47:23.091223+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:27:01.357311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.105594+0000","last_change":"2024-07-26T06:47:23.105051+0000","last_active":"2024-07-26T06:47:23.105594+0000","last_peered":"2024-07-26T06:47:23.105594+0000","last_clean":"2024-07-26T06:47:23.105594+0000","last_became_active":"2024-07-26T06:47:23.098116+0000","last_became_peered":"2024-07-26T06:47:23.098116+0000","last_unstale":"2024-07-26T06:47:23.105594+0000","last_undegraded":"2024-07-26T06:47:23.105594+0000","last_fullsized":"2024-07-26T06:47:23.105594+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:15:32.991870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097110+0000","last_change":"2024-07-26T06:47:05.026056+0000","last_active":"2024-07-26T06:47:23.097110+0000","last_peered":"2024-07-26T06:47:23.097110+0000","last_clean":"2024-07-26T06:47:23.097110+0000","last_became_active":"2024-07-26T06:47:05.025654+0000","last_became_peered":"2024-07-26T06:47:05.025654+0000","last_unstale":"2024-07-26T06:47:23.097110+0000","last_undegraded":"2024-07-26T06:47:23.097110+0000","last_fullsized":"2024-07-26T06:47:23.097110+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:36:20.886768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.094905+0000","last_change":"2024-07-26T06:47:05.026693+0000","last_active":"2024-07-26T06:47:23.094905+0000","last_peered":"2024-07-26T06:47:23.094905+0000","last_clean":"2024-07-26T06:47:23.094905+0000","last_became_active":"2024-07-26T06:47:05.026320+0000","last_became_peered":"2024-07-26T06:47:05.026320+0000","last_unstale":"2024-07-26T06:47:23.094905+0000","last_undegraded":"2024-07-26T06:47:23.094905+0000","last_fullsized":"2024-07-26T06:47:23.094905+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:52:30.776335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"80'5","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.159101+0000","last_change":"2024-07-26T06:47:23.107707+0000","last_active":"2024-07-26T06:47:23.159101+0000","last_peered":"2024-07-26T06:47:23.159101+0000","last_clean":"2024-07-26T06:47:23.159101+0000","last_became_active":"2024-07-26T06:47:23.099429+0000","last_became_peered":"2024-07-26T06:47:23.099429+0000","last_unstale":"2024-07-26T06:47:23.159101+0000","last_undegraded":"2024-07-26T06:47:23.159101+0000","last_fullsized":"2024-07-26T06:47:23.159101+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:40:52.856756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091011+0000","last_change":"2024-07-26T06:47:05.032319+0000","last_active":"2024-07-26T06:47:23.091011+0000","last_peered":"2024-07-26T06:47:23.091011+0000","last_clean":"2024-07-26T06:47:23.091011+0000","last_became_active":"2024-07-26T06:47:05.030305+0000","last_became_peered":"2024-07-26T06:47:05.030305+0000","last_unstale":"2024-07-26T06:47:23.091011+0000","last_undegraded":"2024-07-26T06:47:23.091011+0000","last_fullsized":"2024-07-26T06:47:23.091011+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:34:05.351917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"80'12","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.119914+0000","last_change":"2024-07-26T06:47:02.007812+0000","last_active":"2024-07-26T06:47:24.119914+0000","last_peered":"2024-07-26T06:47:24.119914+0000","last_clean":"2024-07-26T06:47:24.119914+0000","last_became_active":"2024-07-26T06:47:02.006965+0000","last_became_peered":"2024-07-26T06:47:02.006965+0000","last_unstale":"2024-07-26T06:47:24.119914+0000","last_undegraded":"2024-07-26T06:47:24.119914+0000","last_fullsized":"2024-07-26T06:47:24.119914+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:53.173273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.094716+0000","last_change":"2024-07-26T06:47:05.027900+0000","last_active":"2024-07-26T06:47:23.094716+0000","last_peered":"2024-07-26T06:47:23.094716+0000","last_clean":"2024-07-26T06:47:23.094716+0000","last_became_active":"2024-07-26T06:47:05.024467+0000","last_became_peered":"2024-07-26T06:47:05.024467+0000","last_unstale":"2024-07-26T06:47:23.094716+0000","last_undegraded":"2024-07-26T06:47:23.094716+0000","last_fullsized":"2024-07-26T06:47:23.094716+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:05:00.793487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090822+0000","last_change":"2024-07-26T06:47:02.022863+0000","last_active":"2024-07-26T06:47:23.090822+0000","last_peered":"2024-07-26T06:47:23.090822+0000","last_clean":"2024-07-26T06:47:23.090822+0000","last_became_active":"2024-07-26T06:47:02.022446+0000","last_became_peered":"2024-07-26T06:47:02.022446+0000","last_unstale":"2024-07-26T06:47:23.090822+0000","last_undegraded":"2024-07-26T06:47:23.090822+0000","last_fullsized":"2024-07-26T06:47:23.090822+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:14:48.649099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090672+0000","last_change":"2024-07-26T06:47:05.023669+0000","last_active":"2024-07-26T06:47:23.090672+0000","last_peered":"2024-07-26T06:47:23.090672+0000","last_clean":"2024-07-26T06:47:23.090672+0000","last_became_active":"2024-07-26T06:47:05.023265+0000","last_became_peered":"2024-07-26T06:47:05.023265+0000","last_unstale":"2024-07-26T06:47:23.090672+0000","last_undegraded":"2024-07-26T06:47:23.090672+0000","last_fullsized":"2024-07-26T06:47:23.090672+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:56:45.278253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":66,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:47.158283+0000","last_change":"2024-07-26T06:47:02.006668+0000","last_active":"2024-07-26T06:47:47.158283+0000","last_peered":"2024-07-26T06:47:47.158283+0000","last_clean":"2024-07-26T06:47:47.158283+0000","last_became_active":"2024-07-26T06:47:02.006287+0000","last_became_peered":"2024-07-26T06:47:02.006287+0000","last_unstale":"2024-07-26T06:47:47.158283+0000","last_undegraded":"2024-07-26T06:47:47.158283+0000","last_fullsized":"2024-07-26T06:47:47.158283+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:28:05.722539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62785,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":289,"num_read_kb":212,"num_write":242,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1634,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":188355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250626,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710744,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":884736,"data_stored":426222,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906973,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":1072529,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070497,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":425458,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266726,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":880640,"data_stored":425142,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462955,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":1018745,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626480,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36196,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708956,"statfs":{"total":95995035648,"available":95957970944,"internally_reserved":0,"allocated":880640,"data_stored":425175,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822709,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":1018032,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986234,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":481008,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182462,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":897024,"data_stored":425000,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411396,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":892928,"data_stored":426699,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":937984,"data_stored":481669,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":901120,"data_stored":426514,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":57333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":56672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":57708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:47:52.875 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-07-26T06:47:52.875 INFO:tasks.cephadm:Setting up client nodes... 2024-07-26T06:47:52.876 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-07-26T06:47:53.148 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:53.555 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:53 smithi042 ceph-mon[32775]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:47:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:53 smithi164 ceph-mon[36970]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:47:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:53 smithi163 ceph-mon[36873]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:47:54.637 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:54 smithi042 ceph-mon[32775]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-07-26T06:47:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:54 smithi164 ceph-mon[36970]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-07-26T06:47:54.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:54 smithi163 ceph-mon[36873]: pgmap v326: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-07-26T06:47:54.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.881+0000 7f25d39e2640 1 -- 172.21.15.42:0/997307950 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25cc103220 msgr2=0x7f25cc109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:54.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.881+0000 7f25d39e2640 1 --2- 172.21.15.42:0/997307950 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25cc103220 0x7f25cc109bc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f25c4009ca0 tx=0x7f25c402a950 comp rx=0 tx=0).stop 2024-07-26T06:47:54.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.881+0000 7f25d39e2640 1 -- 172.21.15.42:0/997307950 shutdown_connections 2024-07-26T06:47:54.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.881+0000 7f25d39e2640 1 --2- 172.21.15.42:0/997307950 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25cc103220 0x7f25cc109bc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:54.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.881+0000 7f25d39e2640 1 --2- 172.21.15.42:0/997307950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25cc102880 0x7f25cc102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:54.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.881+0000 7f25d39e2640 1 --2- 172.21.15.42:0/997307950 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25cc1017f0 0x7f25cc101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:54.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.881+0000 7f25d39e2640 1 -- 172.21.15.42:0/997307950 >> 172.21.15.42:0/997307950 conn(0x7f25cc0fd160 msgr2=0x7f25cc0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:54.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.881+0000 7f25d39e2640 1 -- 172.21.15.42:0/997307950 shutdown_connections 2024-07-26T06:47:54.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.881+0000 7f25d39e2640 1 -- 172.21.15.42:0/997307950 wait complete. 2024-07-26T06:47:54.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d39e2640 1 Processor -- start 2024-07-26T06:47:54.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d39e2640 1 -- start start 2024-07-26T06:47:54.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d39e2640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25cc1017f0 0x7f25cc0feef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:54.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d39e2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25cc102880 0x7f25cc19e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:54.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d39e2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25cc103220 0x7f25cc1a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:54.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d39e2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f25cc10d490 con 0x7f25cc102880 2024-07-26T06:47:54.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d39e2640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f25cc10d310 con 0x7f25cc1017f0 2024-07-26T06:47:54.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d39e2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f25cc10d610 con 0x7f25cc103220 2024-07-26T06:47:54.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d0f7d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25cc102880 0x7f25cc19e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:54.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d0f7d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25cc102880 0x7f25cc19e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56478/0 (socket says 172.21.15.42:56478) 2024-07-26T06:47:54.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d0f7d640 1 -- 172.21.15.42:0/759749993 learned_addr learned my addr 172.21.15.42:0/759749993 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:54.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d1f7f640 1 --2- 172.21.15.42:0/759749993 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25cc103220 0x7f25cc1a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:54.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d177e640 1 --2- 172.21.15.42:0/759749993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25cc1017f0 0x7f25cc0feef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:54.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d0f7d640 1 -- 172.21.15.42:0/759749993 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25cc103220 msgr2=0x7f25cc1a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:54.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d0f7d640 1 --2- 172.21.15.42:0/759749993 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25cc103220 0x7f25cc1a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:54.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d0f7d640 1 -- 172.21.15.42:0/759749993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25cc1017f0 msgr2=0x7f25cc0feef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:54.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d0f7d640 1 --2- 172.21.15.42:0/759749993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25cc1017f0 0x7f25cc0feef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d0f7d640 1 -- 172.21.15.42:0/759749993 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25cc1a13e0 con 0x7f25cc102880 2024-07-26T06:47:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.882+0000 7f25d177e640 1 --2- 172.21.15.42:0/759749993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25cc1017f0 0x7f25cc0feef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:47:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.883+0000 7f25d0f7d640 1 --2- 172.21.15.42:0/759749993 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25cc102880 0x7f25cc19e720 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00c920 tx=0x7f25bc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.883+0000 7f25c27fc640 1 -- 172.21.15.42:0/759749993 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25bc007c70 con 0x7f25cc102880 2024-07-26T06:47:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.883+0000 7f25d39e2640 1 -- 172.21.15.42:0/759749993 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25cc1a16d0 con 0x7f25cc102880 2024-07-26T06:47:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.883+0000 7f25c27fc640 1 -- 172.21.15.42:0/759749993 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25bc007e10 con 0x7f25cc102880 2024-07-26T06:47:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.883+0000 7f25c27fc640 1 -- 172.21.15.42:0/759749993 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25bc018820 con 0x7f25cc102880 2024-07-26T06:47:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.883+0000 7f25d39e2640 1 -- 172.21.15.42:0/759749993 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25cc1a1c90 con 0x7f25cc102880 2024-07-26T06:47:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.884+0000 7f25d39e2640 1 -- 172.21.15.42:0/759749993 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25cc109390 con 0x7f25cc102880 2024-07-26T06:47:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.885+0000 7f25c27fc640 1 -- 172.21.15.42:0/759749993 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f25bc00f450 con 0x7f25cc102880 2024-07-26T06:47:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.886+0000 7f25c27fc640 1 --2- 172.21.15.42:0/759749993 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f25b80b3c60 0x7f25b80b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.886+0000 7f25d177e640 1 --2- 172.21.15.42:0/759749993 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f25b80b3c60 0x7f25b80b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.886+0000 7f25c27fc640 1 -- 172.21.15.42:0/759749993 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f25bc0e4180 con 0x7f25cc102880 2024-07-26T06:47:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.887+0000 7f25d177e640 1 --2- 172.21.15.42:0/759749993 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f25b80b3c60 0x7f25b80b6170 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f25c8006fd0 tx=0x7f25c8008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:54.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:54.891+0000 7f25c27fc640 1 -- 172.21.15.42:0/759749993 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f25bc014030 con 0x7f25cc102880 2024-07-26T06:47:55.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:55.145+0000 7f25d39e2640 1 -- 172.21.15.42:0/759749993 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f25cc02e0c0 con 0x7f25cc102880 2024-07-26T06:47:55.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:55.148+0000 7f25c27fc640 1 -- 172.21.15.42:0/759749993 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f25bc0ab6e0 con 0x7f25cc102880 2024-07-26T06:47:55.151 INFO:teuthology.orchestra.run.smithi042.stdout:[client.0] 2024-07-26T06:47:55.151 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQCbRqNm6oO+CBAACS7ef+Lhp6yXXsPQvSYv9w== 2024-07-26T06:47:55.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:55.151+0000 7f25d39e2640 1 -- 172.21.15.42:0/759749993 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f25b80b3c60 msgr2=0x7f25b80b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:55.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:55.151+0000 7f25d39e2640 1 --2- 172.21.15.42:0/759749993 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f25b80b3c60 0x7f25b80b6170 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f25c8006fd0 tx=0x7f25c8008040 comp rx=0 tx=0).stop 2024-07-26T06:47:55.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:55.151+0000 7f25d39e2640 1 -- 172.21.15.42:0/759749993 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25cc102880 msgr2=0x7f25cc19e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:55.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:55.151+0000 7f25d39e2640 1 --2- 172.21.15.42:0/759749993 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25cc102880 0x7f25cc19e720 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00c920 tx=0x7f25bc00cde0 comp rx=0 tx=0).stop 2024-07-26T06:47:55.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:55.151+0000 7f25d39e2640 1 -- 172.21.15.42:0/759749993 shutdown_connections 2024-07-26T06:47:55.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:55.151+0000 7f25d39e2640 1 --2- 172.21.15.42:0/759749993 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f25b80b3c60 0x7f25b80b6170 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:55.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:55.151+0000 7f25d39e2640 1 --2- 172.21.15.42:0/759749993 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25cc103220 0x7f25cc1a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:55.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:55.151+0000 7f25d39e2640 1 --2- 172.21.15.42:0/759749993 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25cc102880 0x7f25cc19e720 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:55.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:55.151+0000 7f25d39e2640 1 --2- 172.21.15.42:0/759749993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25cc1017f0 0x7f25cc0feef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:55.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:55.151+0000 7f25d39e2640 1 -- 172.21.15.42:0/759749993 >> 172.21.15.42:0/759749993 conn(0x7f25cc0fd160 msgr2=0x7f25cc106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:55.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:55.151+0000 7f25d39e2640 1 -- 172.21.15.42:0/759749993 shutdown_connections 2024-07-26T06:47:55.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:55.151+0000 7f25d39e2640 1 -- 172.21.15.42:0/759749993 wait complete. 2024-07-26T06:47:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:55 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/759749993' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T06:47:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:55 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/759749993' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T06:47:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:55 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/759749993' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T06:47:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:55 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/759749993' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T06:47:55.804 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:55 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/759749993' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T06:47:55.804 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:55 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/759749993' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T06:47:55.838 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-07-26T06:47:55.838 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-07-26T06:47:55.838 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-07-26T06:47:55.880 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph config log 1 --format=json 2024-07-26T06:47:56.196 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:47:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:56 smithi042 ceph-mon[32775]: pgmap v327: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.0 KiB/s wr, 12 op/s 2024-07-26T06:47:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:47:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:56 smithi164 ceph-mon[36970]: pgmap v327: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.0 KiB/s wr, 12 op/s 2024-07-26T06:47:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:47:56.753 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:56 smithi163 ceph-mon[36873]: pgmap v327: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.0 KiB/s wr, 12 op/s 2024-07-26T06:47:56.753 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:47:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.852+0000 7f7db0e70640 1 -- 172.21.15.42:0/1777316658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7dac107060 msgr2=0x7f7dac107470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.852+0000 7f7db0e70640 1 --2- 172.21.15.42:0/1777316658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7dac107060 0x7f7dac107470 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c009900 tx=0x7f7d9c02a410 comp rx=0 tx=0).stop 2024-07-26T06:47:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.852+0000 7f7db0e70640 1 -- 172.21.15.42:0/1777316658 shutdown_connections 2024-07-26T06:47:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.852+0000 7f7db0e70640 1 --2- 172.21.15.42:0/1777316658 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dac103320 0x7f7dac10eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.852+0000 7f7db0e70640 1 --2- 172.21.15.42:0/1777316658 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7dac1029d0 0x7f7dac102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.852+0000 7f7db0e70640 1 --2- 172.21.15.42:0/1777316658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7dac107060 0x7f7dac107470 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:57.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.852+0000 7f7db0e70640 1 -- 172.21.15.42:0/1777316658 >> 172.21.15.42:0/1777316658 conn(0x7f7dac079870 msgr2=0x7f7dac079c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:57.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.852+0000 7f7db0e70640 1 -- 172.21.15.42:0/1777316658 shutdown_connections 2024-07-26T06:47:57.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.852+0000 7f7db0e70640 1 -- 172.21.15.42:0/1777316658 wait complete. 2024-07-26T06:47:57.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.853+0000 7f7db0e70640 1 Processor -- start 2024-07-26T06:47:57.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.853+0000 7f7db0e70640 1 -- start start 2024-07-26T06:47:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.853+0000 7f7db0e70640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7dac1029d0 0x7f7dac19ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.853+0000 7f7db0e70640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7dac103320 0x7f7dac19b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.853+0000 7f7db0e70640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dac107060 0x7f7dac19b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.853+0000 7f7db0e70640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7dac112400 con 0x7f7dac103320 2024-07-26T06:47:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.853+0000 7f7db0e70640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7dac112280 con 0x7f7dac1029d0 2024-07-26T06:47:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.853+0000 7f7db0e70640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7dac112580 con 0x7f7dac107060 2024-07-26T06:47:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.854+0000 7f7daa59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7dac1029d0 0x7f7dac19ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.854+0000 7f7daa59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7dac1029d0 0x7f7dac19ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:41998/0 (socket says 172.21.15.42:41998) 2024-07-26T06:47:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.854+0000 7f7daa59c640 1 -- 172.21.15.42:0/1024473723 learned_addr learned my addr 172.21.15.42:0/1024473723 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:47:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.854+0000 7f7da9d9b640 1 --2- 172.21.15.42:0/1024473723 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7dac103320 0x7f7dac19b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.854+0000 7f7daad9d640 1 --2- 172.21.15.42:0/1024473723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dac107060 0x7f7dac19b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.854+0000 7f7daa59c640 1 -- 172.21.15.42:0/1024473723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dac107060 msgr2=0x7f7dac19b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.854+0000 7f7daa59c640 1 --2- 172.21.15.42:0/1024473723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dac107060 0x7f7dac19b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.854+0000 7f7daa59c640 1 -- 172.21.15.42:0/1024473723 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7dac103320 msgr2=0x7f7dac19b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.854+0000 7f7daa59c640 1 --2- 172.21.15.42:0/1024473723 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7dac103320 0x7f7dac19b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.854+0000 7f7daa59c640 1 -- 172.21.15.42:0/1024473723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dac1a5fa0 con 0x7f7dac1029d0 2024-07-26T06:47:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.854+0000 7f7daad9d640 1 --2- 172.21.15.42:0/1024473723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dac107060 0x7f7dac19b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:47:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.854+0000 7f7daa59c640 1 --2- 172.21.15.42:0/1024473723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7dac1029d0 0x7f7dac19ad10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c005940 tx=0x7f7d9c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.854+0000 7f7d9b7fe640 1 -- 172.21.15.42:0/1024473723 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c03f030 con 0x7f7dac1029d0 2024-07-26T06:47:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.855+0000 7f7db0e70640 1 -- 172.21.15.42:0/1024473723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dac1a6230 con 0x7f7dac1029d0 2024-07-26T06:47:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.855+0000 7f7d9b7fe640 1 -- 172.21.15.42:0/1024473723 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d9c003aa0 con 0x7f7dac1029d0 2024-07-26T06:47:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.855+0000 7f7d9b7fe640 1 -- 172.21.15.42:0/1024473723 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c035070 con 0x7f7dac1029d0 2024-07-26T06:47:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.855+0000 7f7db0e70640 1 -- 172.21.15.42:0/1024473723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dac1a6710 con 0x7f7dac1029d0 2024-07-26T06:47:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.856+0000 7f7db0e70640 1 -- 172.21.15.42:0/1024473723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dac102de0 con 0x7f7dac1029d0 2024-07-26T06:47:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.857+0000 7f7d9b7fe640 1 -- 172.21.15.42:0/1024473723 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f7d9c042030 con 0x7f7dac1029d0 2024-07-26T06:47:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.858+0000 7f7d9b7fe640 1 --2- 172.21.15.42:0/1024473723 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7d940b3c90 0x7f7d940b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:47:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.858+0000 7f7da9d9b640 1 --2- 172.21.15.42:0/1024473723 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7d940b3c90 0x7f7d940b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:47:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.858+0000 7f7d9b7fe640 1 -- 172.21.15.42:0/1024473723 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f7d9c104650 con 0x7f7dac1029d0 2024-07-26T06:47:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.858+0000 7f7da9d9b640 1 --2- 172.21.15.42:0/1024473723 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7d940b3c90 0x7f7d940b61a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7dac104a40 tx=0x7f7da000b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:47:57.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:57.861+0000 7f7d9b7fe640 1 -- 172.21.15.42:0/1024473723 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7d9c0cb470 con 0x7f7dac1029d0 2024-07-26T06:47:58.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:58.050+0000 7f7db0e70640 1 -- 172.21.15.42:0/1024473723 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f7dac109f50 con 0x7f7dac1029d0 2024-07-26T06:47:58.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:58.051+0000 7f7d9b7fe640 1 -- 172.21.15.42:0/1024473723 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v51) ==== 86+0+133 (secure 0 0 0) 0x7f7d9c0cf9c0 con 0x7f7dac1029d0 2024-07-26T06:47:58.052 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:47:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:58.054+0000 7f7db0e70640 1 -- 172.21.15.42:0/1024473723 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7d940b3c90 msgr2=0x7f7d940b61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:58.054+0000 7f7db0e70640 1 --2- 172.21.15.42:0/1024473723 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7d940b3c90 0x7f7d940b61a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7dac104a40 tx=0x7f7da000b410 comp rx=0 tx=0).stop 2024-07-26T06:47:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:58.054+0000 7f7db0e70640 1 -- 172.21.15.42:0/1024473723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7dac1029d0 msgr2=0x7f7dac19ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:47:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:58.054+0000 7f7db0e70640 1 --2- 172.21.15.42:0/1024473723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7dac1029d0 0x7f7dac19ad10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c005940 tx=0x7f7d9c004270 comp rx=0 tx=0).stop 2024-07-26T06:47:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:58.054+0000 7f7db0e70640 1 -- 172.21.15.42:0/1024473723 shutdown_connections 2024-07-26T06:47:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:58.054+0000 7f7db0e70640 1 --2- 172.21.15.42:0/1024473723 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f7d940b3c90 0x7f7d940b61a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:58.054+0000 7f7db0e70640 1 --2- 172.21.15.42:0/1024473723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7dac107060 0x7f7dac19b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:58.054+0000 7f7db0e70640 1 --2- 172.21.15.42:0/1024473723 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7dac103320 0x7f7dac19b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:58.054+0000 7f7db0e70640 1 --2- 172.21.15.42:0/1024473723 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7dac1029d0 0x7f7dac19ad10 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:47:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:58.054+0000 7f7db0e70640 1 -- 172.21.15.42:0/1024473723 >> 172.21.15.42:0/1024473723 conn(0x7f7dac079870 msgr2=0x7f7dac105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:47:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:58.054+0000 7f7db0e70640 1 -- 172.21.15.42:0/1024473723 shutdown_connections 2024-07-26T06:47:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:47:58.054+0000 7f7db0e70640 1 -- 172.21.15.42:0/1024473723 wait complete. 2024-07-26T06:47:58.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:58 smithi042 ceph-mon[32775]: pgmap v328: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 10 op/s 2024-07-26T06:47:58.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:47:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:47:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:47:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:47:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:47:58 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1024473723' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T06:47:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:58 smithi164 ceph-mon[36970]: pgmap v328: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 10 op/s 2024-07-26T06:47:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:47:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:47:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:47:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:47:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:47:58 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1024473723' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T06:47:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:58 smithi163 ceph-mon[36873]: pgmap v328: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 10 op/s 2024-07-26T06:47:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:47:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:47:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:47:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:47:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:47:58 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1024473723' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T06:47:59.170 INFO:teuthology.orchestra.run.smithi042.stdout:[{"version":51,"timestamp":"2024-07-26T06:46:50.564278+0000","name":"","changes":[{"name":"mds.all/mds_join_fs","new_value":"all"}]}] 2024-07-26T06:47:59.171 INFO:tasks.ceph_manager:config epoch is 51 2024-07-26T06:47:59.171 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-07-26T06:47:59.171 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-07-26T06:47:59.171 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph mgr dump --format=json 2024-07-26T06:47:59.457 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:00.746 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:00 smithi042 ceph-mon[32775]: pgmap v329: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 853 B/s wr, 12 op/s 2024-07-26T06:48:00.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.960+0000 7f36210e6640 1 -- 172.21.15.42:0/2130951587 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f361c107dd0 msgr2=0x7f361c1081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:00.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.960+0000 7f36210e6640 1 --2- 172.21.15.42:0/2130951587 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f361c107dd0 0x7f361c1081e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3610009900 tx=0x7f361002a3f0 comp rx=0 tx=0).stop 2024-07-26T06:48:00.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.960+0000 7f36210e6640 1 -- 172.21.15.42:0/2130951587 shutdown_connections 2024-07-26T06:48:00.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.960+0000 7f36210e6640 1 --2- 172.21.15.42:0/2130951587 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f361c108720 0x7f361c10ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:00.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.960+0000 7f36210e6640 1 --2- 172.21.15.42:0/2130951587 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f361c107dd0 0x7f361c1081e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:00.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.960+0000 7f36210e6640 1 --2- 172.21.15.42:0/2130951587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361c102490 0x7f361c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:00.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.960+0000 7f36210e6640 1 -- 172.21.15.42:0/2130951587 >> 172.21.15.42:0/2130951587 conn(0x7f361c079160 msgr2=0x7f361c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:00.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.960+0000 7f36210e6640 1 -- 172.21.15.42:0/2130951587 shutdown_connections 2024-07-26T06:48:00.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.960+0000 7f36210e6640 1 -- 172.21.15.42:0/2130951587 wait complete. 2024-07-26T06:48:00.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.961+0000 7f36210e6640 1 Processor -- start 2024-07-26T06:48:00.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.961+0000 7f36210e6640 1 -- start start 2024-07-26T06:48:00.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.962+0000 7f36210e6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f361c102490 0x7f361c19aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:00.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.962+0000 7f36210e6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361c107dd0 0x7f361c19b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:00.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.962+0000 7f36210e6640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f361c108720 0x7f361c19b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:00.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.962+0000 7f36210e6640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f361c112350 con 0x7f361c102490 2024-07-26T06:48:00.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.962+0000 7f36210e6640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f361c1121d0 con 0x7f361c108720 2024-07-26T06:48:00.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.962+0000 7f36210e6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f361c1124d0 con 0x7f361c107dd0 2024-07-26T06:48:00.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.962+0000 7f361ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f361c102490 0x7f361c19aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:00.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.962+0000 7f361ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f361c102490 0x7f361c19aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35158/0 (socket says 172.21.15.42:35158) 2024-07-26T06:48:00.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.962+0000 7f361ad9d640 1 -- 172.21.15.42:0/4173678379 learned_addr learned my addr 172.21.15.42:0/4173678379 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:00.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.962+0000 7f361a59c640 1 --2- 172.21.15.42:0/4173678379 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361c107dd0 0x7f361c19b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:00.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.962+0000 7f361b59e640 1 --2- 172.21.15.42:0/4173678379 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f361c108720 0x7f361c19b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:00.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.962+0000 7f361ad9d640 1 -- 172.21.15.42:0/4173678379 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361c107dd0 msgr2=0x7f361c19b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:00.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.962+0000 7f361ad9d640 1 --2- 172.21.15.42:0/4173678379 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361c107dd0 0x7f361c19b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:00.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.962+0000 7f361ad9d640 1 -- 172.21.15.42:0/4173678379 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f361c108720 msgr2=0x7f361c19b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:00.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.962+0000 7f361ad9d640 1 --2- 172.21.15.42:0/4173678379 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f361c108720 0x7f361c19b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:00.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.962+0000 7f361ad9d640 1 -- 172.21.15.42:0/4173678379 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f361c1a5e20 con 0x7f361c102490 2024-07-26T06:48:00.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.963+0000 7f361ad9d640 1 --2- 172.21.15.42:0/4173678379 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f361c102490 0x7f361c19aba0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f360c00c920 tx=0x7f360c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:00.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.963+0000 7f3603fff640 1 -- 172.21.15.42:0/4173678379 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c007c70 con 0x7f361c102490 2024-07-26T06:48:00.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.963+0000 7f3603fff640 1 -- 172.21.15.42:0/4173678379 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f360c007e10 con 0x7f361c102490 2024-07-26T06:48:00.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.963+0000 7f36210e6640 1 -- 172.21.15.42:0/4173678379 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f361c1a6110 con 0x7f361c102490 2024-07-26T06:48:00.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.963+0000 7f36210e6640 1 -- 172.21.15.42:0/4173678379 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f361c1a6670 con 0x7f361c102490 2024-07-26T06:48:00.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.963+0000 7f3603fff640 1 -- 172.21.15.42:0/4173678379 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c018820 con 0x7f361c102490 2024-07-26T06:48:00.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.964+0000 7f36210e6640 1 -- 172.21.15.42:0/4173678379 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36080052c0 con 0x7f361c102490 2024-07-26T06:48:00.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.967+0000 7f3603fff640 1 -- 172.21.15.42:0/4173678379 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f360c0189c0 con 0x7f361c102490 2024-07-26T06:48:00.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.967+0000 7f3603fff640 1 --2- 172.21.15.42:0/4173678379 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36040b3ce0 0x7f36040b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:00.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.968+0000 7f361a59c640 1 --2- 172.21.15.42:0/4173678379 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36040b3ce0 0x7f36040b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:00.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.968+0000 7f3603fff640 1 -- 172.21.15.42:0/4173678379 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f360c0e3e60 con 0x7f361c102490 2024-07-26T06:48:00.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.968+0000 7f361a59c640 1 --2- 172.21.15.42:0/4173678379 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36040b3ce0 0x7f36040b61f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3610000c00 tx=0x7f3610031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:00.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:00.971+0000 7f3603fff640 1 -- 172.21.15.42:0/4173678379 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f360c014030 con 0x7f361c102490 2024-07-26T06:48:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:00 smithi164 ceph-mon[36970]: pgmap v329: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 853 B/s wr, 12 op/s 2024-07-26T06:48:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:00 smithi163 ceph-mon[36873]: pgmap v329: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 853 B/s wr, 12 op/s 2024-07-26T06:48:01.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:01.277+0000 7f36210e6640 1 -- 172.21.15.42:0/4173678379 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f3608005880 con 0x7f361c102490 2024-07-26T06:48:01.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:01.282+0000 7f3603fff640 1 -- 172.21.15.42:0/4173678379 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+291018 (secure 0 0 0) 0x7f360c0ab440 con 0x7f361c102490 2024-07-26T06:48:01.286 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:48:01.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:01.286+0000 7f36210e6640 1 -- 172.21.15.42:0/4173678379 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36040b3ce0 msgr2=0x7f36040b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:01.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:01.286+0000 7f36210e6640 1 --2- 172.21.15.42:0/4173678379 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36040b3ce0 0x7f36040b61f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3610000c00 tx=0x7f3610031400 comp rx=0 tx=0).stop 2024-07-26T06:48:01.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:01.286+0000 7f36210e6640 1 -- 172.21.15.42:0/4173678379 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f361c102490 msgr2=0x7f361c19aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:01.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:01.286+0000 7f36210e6640 1 --2- 172.21.15.42:0/4173678379 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f361c102490 0x7f361c19aba0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f360c00c920 tx=0x7f360c00cde0 comp rx=0 tx=0).stop 2024-07-26T06:48:01.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:01.286+0000 7f36210e6640 1 -- 172.21.15.42:0/4173678379 shutdown_connections 2024-07-26T06:48:01.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:01.286+0000 7f36210e6640 1 --2- 172.21.15.42:0/4173678379 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36040b3ce0 0x7f36040b61f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:01.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:01.286+0000 7f36210e6640 1 --2- 172.21.15.42:0/4173678379 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f361c108720 0x7f361c19b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:01.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:01.286+0000 7f36210e6640 1 --2- 172.21.15.42:0/4173678379 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361c107dd0 0x7f361c19b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:01.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:01.286+0000 7f36210e6640 1 --2- 172.21.15.42:0/4173678379 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f361c102490 0x7f361c19aba0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:01.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:01.286+0000 7f36210e6640 1 -- 172.21.15.42:0/4173678379 >> 172.21.15.42:0/4173678379 conn(0x7f361c079160 msgr2=0x7f361c104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:01.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:01.287+0000 7f36210e6640 1 -- 172.21.15.42:0/4173678379 shutdown_connections 2024-07-26T06:48:01.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:01.287+0000 7f36210e6640 1 -- 172.21.15.42:0/4173678379 wait complete. 2024-07-26T06:48:01.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:48:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:48:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:48:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:48:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:48:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:01 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/4173678379' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T06:48:01.962 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":17,"flags":0,"active_gid":14148,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6800","nonce":2719631589},{"type":"v1","addr":"172.21.15.42:6801","nonce":2719631589}]},"active_addr":"172.21.15.42:6801/2719631589","active_change":"2024-07-26T06:39:21.217249+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24125,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.42:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":4107465271}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":392418010}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":3782872716}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":3080336070}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":139470006}]}]} 2024-07-26T06:48:01.968 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-07-26T06:48:01.968 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-07-26T06:48:01.969 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T06:48:02.183 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:48:02.183 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:48:02.183 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:48:02.183 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:48:02.183 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:48:02.183 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:01 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/4173678379' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T06:48:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:48:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:48:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:48:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:48:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:48:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:01 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/4173678379' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T06:48:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:03.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:02 smithi042 ceph-mon[32775]: pgmap v330: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-07-26T06:48:03.112 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:02 smithi163 ceph-mon[36873]: pgmap v330: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-07-26T06:48:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:02 smithi164 ceph-mon[36970]: pgmap v330: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-07-26T06:48:03.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.897+0000 7fc943c67640 1 -- 172.21.15.42:0/3117000736 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc93c108720 msgr2=0x7fc93c10ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:03.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.897+0000 7fc943c67640 1 --2- 172.21.15.42:0/3117000736 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc93c108720 0x7fc93c10ef50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc934009ca0 tx=0x7fc93402a8c0 comp rx=0 tx=0).stop 2024-07-26T06:48:03.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.897+0000 7fc943c67640 1 -- 172.21.15.42:0/3117000736 shutdown_connections 2024-07-26T06:48:03.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.897+0000 7fc943c67640 1 --2- 172.21.15.42:0/3117000736 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc93c108720 0x7fc93c10ef50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:03.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.897+0000 7fc943c67640 1 --2- 172.21.15.42:0/3117000736 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc93c107dd0 0x7fc93c1081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:03.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.897+0000 7fc943c67640 1 --2- 172.21.15.42:0/3117000736 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc93c102490 0x7fc93c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:03.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.897+0000 7fc943c67640 1 -- 172.21.15.42:0/3117000736 >> 172.21.15.42:0/3117000736 conn(0x7fc93c079160 msgr2=0x7fc93c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:03.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.897+0000 7fc943c67640 1 -- 172.21.15.42:0/3117000736 shutdown_connections 2024-07-26T06:48:03.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.897+0000 7fc943c67640 1 -- 172.21.15.42:0/3117000736 wait complete. 2024-07-26T06:48:03.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.898+0000 7fc943c67640 1 Processor -- start 2024-07-26T06:48:03.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.898+0000 7fc943c67640 1 -- start start 2024-07-26T06:48:03.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc943c67640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc93c102490 0x7fc93c104b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:03.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc943c67640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc93c107dd0 0x7fc93c105050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:03.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc943c67640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc93c108720 0x7fc93c105590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:03.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc943c67640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc93c1123d0 con 0x7fc93c108720 2024-07-26T06:48:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc943c67640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc93c112250 con 0x7fc93c107dd0 2024-07-26T06:48:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc943c67640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc93c112550 con 0x7fc93c102490 2024-07-26T06:48:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc942204640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc93c108720 0x7fc93c105590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc942204640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc93c108720 0x7fc93c105590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35170/0 (socket says 172.21.15.42:35170) 2024-07-26T06:48:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc942204640 1 -- 172.21.15.42:0/139336743 learned_addr learned my addr 172.21.15.42:0/139336743 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc941a03640 1 --2- 172.21.15.42:0/139336743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc93c102490 0x7fc93c104b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc941202640 1 --2- 172.21.15.42:0/139336743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc93c107dd0 0x7fc93c105050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc942204640 1 -- 172.21.15.42:0/139336743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc93c102490 msgr2=0x7fc93c104b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc942204640 1 --2- 172.21.15.42:0/139336743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc93c102490 0x7fc93c104b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc942204640 1 -- 172.21.15.42:0/139336743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc93c107dd0 msgr2=0x7fc93c105050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc942204640 1 --2- 172.21.15.42:0/139336743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc93c107dd0 0x7fc93c105050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc942204640 1 -- 172.21.15.42:0/139336743 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc93c1a80c0 con 0x7fc93c108720 2024-07-26T06:48:03.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc941a03640 1 --2- 172.21.15.42:0/139336743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc93c102490 0x7fc93c104b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:48:03.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc942204640 1 --2- 172.21.15.42:0/139336743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc93c108720 0x7fc93c105590 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc934009430 tx=0x7fc9340317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:03.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.899+0000 7fc92effd640 1 -- 172.21.15.42:0/139336743 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc934003b70 con 0x7fc93c108720 2024-07-26T06:48:03.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.900+0000 7fc92effd640 1 -- 172.21.15.42:0/139336743 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc934003d10 con 0x7fc93c108720 2024-07-26T06:48:03.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.900+0000 7fc943c67640 1 -- 172.21.15.42:0/139336743 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc93c1a8350 con 0x7fc93c108720 2024-07-26T06:48:03.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.900+0000 7fc92effd640 1 -- 172.21.15.42:0/139336743 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93402fa30 con 0x7fc93c108720 2024-07-26T06:48:03.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.900+0000 7fc943c67640 1 -- 172.21.15.42:0/139336743 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc93c1a88e0 con 0x7fc93c108720 2024-07-26T06:48:03.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.901+0000 7fc943c67640 1 -- 172.21.15.42:0/139336743 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc93c103710 con 0x7fc93c108720 2024-07-26T06:48:03.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.902+0000 7fc92effd640 1 -- 172.21.15.42:0/139336743 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7fc93402fbd0 con 0x7fc93c108720 2024-07-26T06:48:03.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.903+0000 7fc92effd640 1 --2- 172.21.15.42:0/139336743 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc9280b3ce0 0x7fc9280b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:03.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.903+0000 7fc941a03640 1 --2- 172.21.15.42:0/139336743 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc9280b3ce0 0x7fc9280b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:03.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.904+0000 7fc92effd640 1 -- 172.21.15.42:0/139336743 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fc9341052d0 con 0x7fc93c108720 2024-07-26T06:48:03.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.904+0000 7fc941a03640 1 --2- 172.21.15.42:0/139336743 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc9280b3ce0 0x7fc9280b61f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc930005d00 tx=0x7fc930005c90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:03.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:03.907+0000 7fc92effd640 1 -- 172.21.15.42:0/139336743 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc9340cc8b0 con 0x7fc93c108720 2024-07-26T06:48:04.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:04.090+0000 7fc943c67640 1 -- 172.21.15.42:0/139336743 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc93c105e10 con 0x7fc93c108720 2024-07-26T06:48:04.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:04.091+0000 7fc92effd640 1 -- 172.21.15.42:0/139336743 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22697 (secure 0 0 0) 0x7fc93c105e10 con 0x7fc93c108720 2024-07-26T06:48:04.092 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:48:04.092 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":80,"fsid":"8b3e2750-4b19-11ef-bcb6-c7b262605968","created":"2024-07-26T06:38:29.306470+0000","modified":"2024-07-26T06:47:23.086609+0000","last_up_change":"2024-07-26T06:46:02.149453+0000","last_in_change":"2024-07-26T06:45:43.027764+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:42:27.302195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:47:00.042262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T06:47:03.674160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bda88e0-09ee-447b-8c04-24f4e5d55d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6803","nonce":3587066379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6805","nonce":3587066379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6809","nonce":3587066379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6807","nonce":3587066379}]},"public_addr":"172.21.15.42:6803/3587066379","cluster_addr":"172.21.15.42:6805/3587066379","heartbeat_back_addr":"172.21.15.42:6809/3587066379","heartbeat_front_addr":"172.21.15.42:6807/3587066379","state":["exists","up"]},{"osd":1,"uuid":"7d747138-8217-4836-a173-cf8ac65e9fe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6801","nonce":360219078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6803","nonce":360219078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6807","nonce":360219078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6805","nonce":360219078}]},"public_addr":"172.21.15.163:6801/360219078","cluster_addr":"172.21.15.163:6803/360219078","heartbeat_back_addr":"172.21.15.163:6807/360219078","heartbeat_front_addr":"172.21.15.163:6805/360219078","state":["exists","up"]},{"osd":2,"uuid":"4b7258c1-3d35-4930-ab10-0b04a2a2c8a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6801","nonce":336113412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6803","nonce":336113412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6807","nonce":336113412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6805","nonce":336113412}]},"public_addr":"172.21.15.164:6801/336113412","cluster_addr":"172.21.15.164:6803/336113412","heartbeat_back_addr":"172.21.15.164:6807/336113412","heartbeat_front_addr":"172.21.15.164:6805/336113412","state":["exists","up"]},{"osd":3,"uuid":"96608387-0de0-4d90-a952-d6eba61f7fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6811","nonce":2819018064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6813","nonce":2819018064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6817","nonce":2819018064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6815","nonce":2819018064}]},"public_addr":"172.21.15.42:6811/2819018064","cluster_addr":"172.21.15.42:6813/2819018064","heartbeat_back_addr":"172.21.15.42:6817/2819018064","heartbeat_front_addr":"172.21.15.42:6815/2819018064","state":["exists","up"]},{"osd":4,"uuid":"fabf6b02-fe34-4560-85b5-01bf37fc9be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6809","nonce":431934374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6811","nonce":431934374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6815","nonce":431934374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6813","nonce":431934374}]},"public_addr":"172.21.15.163:6809/431934374","cluster_addr":"172.21.15.163:6811/431934374","heartbeat_back_addr":"172.21.15.163:6815/431934374","heartbeat_front_addr":"172.21.15.163:6813/431934374","state":["exists","up"]},{"osd":5,"uuid":"e226fb4f-15b2-40df-af34-900040ba93ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6809","nonce":261549362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6811","nonce":261549362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6815","nonce":261549362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6813","nonce":261549362}]},"public_addr":"172.21.15.164:6809/261549362","cluster_addr":"172.21.15.164:6811/261549362","heartbeat_back_addr":"172.21.15.164:6815/261549362","heartbeat_front_addr":"172.21.15.164:6813/261549362","state":["exists","up"]},{"osd":6,"uuid":"beb8ec5d-7ea5-4e4e-a46c-a84f7768cec0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6819","nonce":2160811300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6821","nonce":2160811300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6825","nonce":2160811300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6823","nonce":2160811300}]},"public_addr":"172.21.15.42:6819/2160811300","cluster_addr":"172.21.15.42:6821/2160811300","heartbeat_back_addr":"172.21.15.42:6825/2160811300","heartbeat_front_addr":"172.21.15.42:6823/2160811300","state":["exists","up"]},{"osd":7,"uuid":"f699004c-f7c2-4f1c-ae4b-7178f3b06e2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6817","nonce":812214111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6819","nonce":812214111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6823","nonce":812214111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6821","nonce":812214111}]},"public_addr":"172.21.15.163:6817/812214111","cluster_addr":"172.21.15.163:6819/812214111","heartbeat_back_addr":"172.21.15.163:6823/812214111","heartbeat_front_addr":"172.21.15.163:6821/812214111","state":["exists","up"]},{"osd":8,"uuid":"3476163e-efea-4652-a8d4-b7c4a908f984","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6817","nonce":962845839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6819","nonce":962845839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6823","nonce":962845839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6821","nonce":962845839}]},"public_addr":"172.21.15.164:6817/962845839","cluster_addr":"172.21.15.164:6819/962845839","heartbeat_back_addr":"172.21.15.164:6823/962845839","heartbeat_front_addr":"172.21.15.164:6821/962845839","state":["exists","up"]},{"osd":9,"uuid":"6f7ceac6-fb61-4c6f-9da8-851b937c4586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6827","nonce":994024418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6829","nonce":994024418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6833","nonce":994024418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6831","nonce":994024418}]},"public_addr":"172.21.15.42:6827/994024418","cluster_addr":"172.21.15.42:6829/994024418","heartbeat_back_addr":"172.21.15.42:6833/994024418","heartbeat_front_addr":"172.21.15.42:6831/994024418","state":["exists","up"]},{"osd":10,"uuid":"553132c2-a6a1-410c-a7d8-ecdf9a67dafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6825","nonce":3268512227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6827","nonce":3268512227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6831","nonce":3268512227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6829","nonce":3268512227}]},"public_addr":"172.21.15.163:6825/3268512227","cluster_addr":"172.21.15.163:6827/3268512227","heartbeat_back_addr":"172.21.15.163:6831/3268512227","heartbeat_front_addr":"172.21.15.163:6829/3268512227","state":["exists","up"]},{"osd":11,"uuid":"09abc8e9-7884-400f-ab6d-8e5aa7bf31ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6825","nonce":1861076336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6827","nonce":1861076336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6831","nonce":1861076336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6829","nonce":1861076336}]},"public_addr":"172.21.15.164:6825/1861076336","cluster_addr":"172.21.15.164:6827/1861076336","heartbeat_back_addr":"172.21.15.164:6831/1861076336","heartbeat_front_addr":"172.21.15.164:6829/1861076336","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:41:38.761674+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:00.621745+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:23.248620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:47.232764+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:10.379922+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:34.502519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:58.694642+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:22.350247+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:46.988007+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:11.390474+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:36.150054+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:46:00.474470+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.42:6801/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6800/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1842521399":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/834172950":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1730109419":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6801/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:6800/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2687772889":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/3371433307":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2861719816":"2024-07-27T06:38:56.887264+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:48:04.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:04.093+0000 7fc943c67640 1 -- 172.21.15.42:0/139336743 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc9280b3ce0 msgr2=0x7fc9280b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:04.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:04.093+0000 7fc943c67640 1 --2- 172.21.15.42:0/139336743 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc9280b3ce0 0x7fc9280b61f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc930005d00 tx=0x7fc930005c90 comp rx=0 tx=0).stop 2024-07-26T06:48:04.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:04.093+0000 7fc943c67640 1 -- 172.21.15.42:0/139336743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc93c108720 msgr2=0x7fc93c105590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:04.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:04.093+0000 7fc943c67640 1 --2- 172.21.15.42:0/139336743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc93c108720 0x7fc93c105590 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc934009430 tx=0x7fc9340317c0 comp rx=0 tx=0).stop 2024-07-26T06:48:04.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:04.093+0000 7fc943c67640 1 -- 172.21.15.42:0/139336743 shutdown_connections 2024-07-26T06:48:04.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:04.093+0000 7fc943c67640 1 --2- 172.21.15.42:0/139336743 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc9280b3ce0 0x7fc9280b61f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:04.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:04.093+0000 7fc943c67640 1 --2- 172.21.15.42:0/139336743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc93c108720 0x7fc93c105590 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:04.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:04.093+0000 7fc943c67640 1 --2- 172.21.15.42:0/139336743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc93c107dd0 0x7fc93c105050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:04.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:04.094+0000 7fc943c67640 1 --2- 172.21.15.42:0/139336743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc93c102490 0x7fc93c104b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:04.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:04.094+0000 7fc943c67640 1 -- 172.21.15.42:0/139336743 >> 172.21.15.42:0/139336743 conn(0x7fc93c079160 msgr2=0x7fc93c06b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:04.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:04.094+0000 7fc943c67640 1 -- 172.21.15.42:0/139336743 shutdown_connections 2024-07-26T06:48:04.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:04.094+0000 7fc943c67640 1 -- 172.21.15.42:0/139336743 wait complete. 2024-07-26T06:48:04.757 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-07-26T06:48:04.758 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T06:48:05.042 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:05.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:04 smithi042 ceph-mon[32775]: pgmap v331: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 9 op/s 2024-07-26T06:48:05.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:04 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/139336743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:48:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:04 smithi164 ceph-mon[36970]: pgmap v331: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 9 op/s 2024-07-26T06:48:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:04 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/139336743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:48:05.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:04 smithi163 ceph-mon[36873]: pgmap v331: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 9 op/s 2024-07-26T06:48:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:04 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/139336743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:48:06.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.767+0000 7f522b1c2640 1 -- 172.21.15.42:0/3450326533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5224109830 msgr2=0x7f5224110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:06.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.767+0000 7f522b1c2640 1 --2- 172.21.15.42:0/3450326533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5224109830 0x7f5224110060 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f521c009ca0 tx=0x7f521c02a8e0 comp rx=0 tx=0).stop 2024-07-26T06:48:06.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.767+0000 7f522b1c2640 1 -- 172.21.15.42:0/3450326533 shutdown_connections 2024-07-26T06:48:06.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.767+0000 7f522b1c2640 1 --2- 172.21.15.42:0/3450326533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5224109830 0x7f5224110060 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:06.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.767+0000 7f522b1c2640 1 --2- 172.21.15.42:0/3450326533 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5224108ee0 0x7f52241092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:06.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.767+0000 7f522b1c2640 1 --2- 172.21.15.42:0/3450326533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52241035f0 0x7f5224103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:06.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.767+0000 7f522b1c2640 1 -- 172.21.15.42:0/3450326533 >> 172.21.15.42:0/3450326533 conn(0x7f522407b340 msgr2=0x7f522407b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:06.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.767+0000 7f522b1c2640 1 -- 172.21.15.42:0/3450326533 shutdown_connections 2024-07-26T06:48:06.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.767+0000 7f522b1c2640 1 -- 172.21.15.42:0/3450326533 wait complete. 2024-07-26T06:48:06.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.768+0000 7f522b1c2640 1 Processor -- start 2024-07-26T06:48:06.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.768+0000 7f522b1c2640 1 -- start start 2024-07-26T06:48:06.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.768+0000 7f522b1c2640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52241035f0 0x7f522419c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:06.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.768+0000 7f522b1c2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5224108ee0 0x7f522419c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:06.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.768+0000 7f522b1c2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5224109830 0x7f52241a1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:06.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.768+0000 7f522b1c2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5224113600 con 0x7f5224109830 2024-07-26T06:48:06.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.768+0000 7f522b1c2640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5224113480 con 0x7f52241035f0 2024-07-26T06:48:06.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.768+0000 7f522b1c2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5224113780 con 0x7f5224108ee0 2024-07-26T06:48:06.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.768+0000 7f522975f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5224109830 0x7f52241a1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:06.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.768+0000 7f522975f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5224109830 0x7f52241a1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35182/0 (socket says 172.21.15.42:35182) 2024-07-26T06:48:06.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.768+0000 7f522975f640 1 -- 172.21.15.42:0/194955534 learned_addr learned my addr 172.21.15.42:0/194955534 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:06.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.768+0000 7f5228f5e640 1 --2- 172.21.15.42:0/194955534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52241035f0 0x7f522419c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:06.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.769+0000 7f5223fff640 1 --2- 172.21.15.42:0/194955534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5224108ee0 0x7f522419c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:06.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.769+0000 7f522975f640 1 -- 172.21.15.42:0/194955534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5224108ee0 msgr2=0x7f522419c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:06.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.769+0000 7f522975f640 1 --2- 172.21.15.42:0/194955534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5224108ee0 0x7f522419c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:06.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.769+0000 7f522975f640 1 -- 172.21.15.42:0/194955534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52241035f0 msgr2=0x7f522419c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:06.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.769+0000 7f522975f640 1 --2- 172.21.15.42:0/194955534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52241035f0 0x7f522419c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:06.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.769+0000 7f522975f640 1 -- 172.21.15.42:0/194955534 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52241a21a0 con 0x7f5224109830 2024-07-26T06:48:06.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.769+0000 7f522975f640 1 --2- 172.21.15.42:0/194955534 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5224109830 0x7f52241a1a70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f521c003bd0 tx=0x7f521c003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:06.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.769+0000 7f5221ffb640 1 -- 172.21.15.42:0/194955534 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c004240 con 0x7f5224109830 2024-07-26T06:48:06.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.769+0000 7f5221ffb640 1 -- 172.21.15.42:0/194955534 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f521c003e20 con 0x7f5224109830 2024-07-26T06:48:06.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.769+0000 7f522b1c2640 1 -- 172.21.15.42:0/194955534 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52241a2430 con 0x7f5224109830 2024-07-26T06:48:06.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.769+0000 7f5221ffb640 1 -- 172.21.15.42:0/194955534 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c030eb0 con 0x7f5224109830 2024-07-26T06:48:06.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.769+0000 7f522b1c2640 1 -- 172.21.15.42:0/194955534 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52241a8bf0 con 0x7f5224109830 2024-07-26T06:48:06.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.770+0000 7f522b1c2640 1 -- 172.21.15.42:0/194955534 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5224103a80 con 0x7f5224109830 2024-07-26T06:48:06.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.772+0000 7f5221ffb640 1 -- 172.21.15.42:0/194955534 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f521c0305e0 con 0x7f5224109830 2024-07-26T06:48:06.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.773+0000 7f5221ffb640 1 --2- 172.21.15.42:0/194955534 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f52140b3ce0 0x7f52140b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:06.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.773+0000 7f5228f5e640 1 --2- 172.21.15.42:0/194955534 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f52140b3ce0 0x7f52140b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:06.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.773+0000 7f5221ffb640 1 -- 172.21.15.42:0/194955534 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f521c105240 con 0x7f5224109830 2024-07-26T06:48:06.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.774+0000 7f5228f5e640 1 --2- 172.21.15.42:0/194955534 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f52140b3ce0 0x7f52140b61f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5218009990 tx=0x7f5218009110 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:06.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.776+0000 7f5221ffb640 1 -- 172.21.15.42:0/194955534 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f521c0cc7c0 con 0x7f5224109830 2024-07-26T06:48:06.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.960+0000 7f522b1c2640 1 -- 172.21.15.42:0/194955534 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f522419d360 con 0x7f5224109830 2024-07-26T06:48:06.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.962+0000 7f5221ffb640 1 -- 172.21.15.42:0/194955534 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22697 (secure 0 0 0) 0x7f522419d360 con 0x7f5224109830 2024-07-26T06:48:06.964 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:48:06.964 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":80,"fsid":"8b3e2750-4b19-11ef-bcb6-c7b262605968","created":"2024-07-26T06:38:29.306470+0000","modified":"2024-07-26T06:47:23.086609+0000","last_up_change":"2024-07-26T06:46:02.149453+0000","last_in_change":"2024-07-26T06:45:43.027764+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:42:27.302195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:47:00.042262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T06:47:03.674160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bda88e0-09ee-447b-8c04-24f4e5d55d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6803","nonce":3587066379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6805","nonce":3587066379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6809","nonce":3587066379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6807","nonce":3587066379}]},"public_addr":"172.21.15.42:6803/3587066379","cluster_addr":"172.21.15.42:6805/3587066379","heartbeat_back_addr":"172.21.15.42:6809/3587066379","heartbeat_front_addr":"172.21.15.42:6807/3587066379","state":["exists","up"]},{"osd":1,"uuid":"7d747138-8217-4836-a173-cf8ac65e9fe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6801","nonce":360219078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6803","nonce":360219078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6807","nonce":360219078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6805","nonce":360219078}]},"public_addr":"172.21.15.163:6801/360219078","cluster_addr":"172.21.15.163:6803/360219078","heartbeat_back_addr":"172.21.15.163:6807/360219078","heartbeat_front_addr":"172.21.15.163:6805/360219078","state":["exists","up"]},{"osd":2,"uuid":"4b7258c1-3d35-4930-ab10-0b04a2a2c8a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6801","nonce":336113412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6803","nonce":336113412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6807","nonce":336113412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6805","nonce":336113412}]},"public_addr":"172.21.15.164:6801/336113412","cluster_addr":"172.21.15.164:6803/336113412","heartbeat_back_addr":"172.21.15.164:6807/336113412","heartbeat_front_addr":"172.21.15.164:6805/336113412","state":["exists","up"]},{"osd":3,"uuid":"96608387-0de0-4d90-a952-d6eba61f7fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6811","nonce":2819018064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6813","nonce":2819018064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6817","nonce":2819018064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6815","nonce":2819018064}]},"public_addr":"172.21.15.42:6811/2819018064","cluster_addr":"172.21.15.42:6813/2819018064","heartbeat_back_addr":"172.21.15.42:6817/2819018064","heartbeat_front_addr":"172.21.15.42:6815/2819018064","state":["exists","up"]},{"osd":4,"uuid":"fabf6b02-fe34-4560-85b5-01bf37fc9be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6809","nonce":431934374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6811","nonce":431934374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6815","nonce":431934374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6813","nonce":431934374}]},"public_addr":"172.21.15.163:6809/431934374","cluster_addr":"172.21.15.163:6811/431934374","heartbeat_back_addr":"172.21.15.163:6815/431934374","heartbeat_front_addr":"172.21.15.163:6813/431934374","state":["exists","up"]},{"osd":5,"uuid":"e226fb4f-15b2-40df-af34-900040ba93ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6809","nonce":261549362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6811","nonce":261549362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6815","nonce":261549362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6813","nonce":261549362}]},"public_addr":"172.21.15.164:6809/261549362","cluster_addr":"172.21.15.164:6811/261549362","heartbeat_back_addr":"172.21.15.164:6815/261549362","heartbeat_front_addr":"172.21.15.164:6813/261549362","state":["exists","up"]},{"osd":6,"uuid":"beb8ec5d-7ea5-4e4e-a46c-a84f7768cec0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6819","nonce":2160811300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6821","nonce":2160811300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6825","nonce":2160811300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6823","nonce":2160811300}]},"public_addr":"172.21.15.42:6819/2160811300","cluster_addr":"172.21.15.42:6821/2160811300","heartbeat_back_addr":"172.21.15.42:6825/2160811300","heartbeat_front_addr":"172.21.15.42:6823/2160811300","state":["exists","up"]},{"osd":7,"uuid":"f699004c-f7c2-4f1c-ae4b-7178f3b06e2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6817","nonce":812214111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6819","nonce":812214111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6823","nonce":812214111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6821","nonce":812214111}]},"public_addr":"172.21.15.163:6817/812214111","cluster_addr":"172.21.15.163:6819/812214111","heartbeat_back_addr":"172.21.15.163:6823/812214111","heartbeat_front_addr":"172.21.15.163:6821/812214111","state":["exists","up"]},{"osd":8,"uuid":"3476163e-efea-4652-a8d4-b7c4a908f984","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6817","nonce":962845839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6819","nonce":962845839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6823","nonce":962845839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6821","nonce":962845839}]},"public_addr":"172.21.15.164:6817/962845839","cluster_addr":"172.21.15.164:6819/962845839","heartbeat_back_addr":"172.21.15.164:6823/962845839","heartbeat_front_addr":"172.21.15.164:6821/962845839","state":["exists","up"]},{"osd":9,"uuid":"6f7ceac6-fb61-4c6f-9da8-851b937c4586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6827","nonce":994024418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6829","nonce":994024418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6833","nonce":994024418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6831","nonce":994024418}]},"public_addr":"172.21.15.42:6827/994024418","cluster_addr":"172.21.15.42:6829/994024418","heartbeat_back_addr":"172.21.15.42:6833/994024418","heartbeat_front_addr":"172.21.15.42:6831/994024418","state":["exists","up"]},{"osd":10,"uuid":"553132c2-a6a1-410c-a7d8-ecdf9a67dafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6825","nonce":3268512227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6827","nonce":3268512227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6831","nonce":3268512227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6829","nonce":3268512227}]},"public_addr":"172.21.15.163:6825/3268512227","cluster_addr":"172.21.15.163:6827/3268512227","heartbeat_back_addr":"172.21.15.163:6831/3268512227","heartbeat_front_addr":"172.21.15.163:6829/3268512227","state":["exists","up"]},{"osd":11,"uuid":"09abc8e9-7884-400f-ab6d-8e5aa7bf31ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6825","nonce":1861076336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6827","nonce":1861076336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6831","nonce":1861076336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6829","nonce":1861076336}]},"public_addr":"172.21.15.164:6825/1861076336","cluster_addr":"172.21.15.164:6827/1861076336","heartbeat_back_addr":"172.21.15.164:6831/1861076336","heartbeat_front_addr":"172.21.15.164:6829/1861076336","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:41:38.761674+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:00.621745+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:23.248620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:47.232764+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:10.379922+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:34.502519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:58.694642+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:22.350247+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:46.988007+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:11.390474+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:36.150054+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:46:00.474470+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.42:6801/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6800/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1842521399":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/834172950":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1730109419":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6801/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:6800/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2687772889":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/3371433307":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2861719816":"2024-07-27T06:38:56.887264+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:48:06.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.964+0000 7f522b1c2640 1 -- 172.21.15.42:0/194955534 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f52140b3ce0 msgr2=0x7f52140b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:06.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.964+0000 7f522b1c2640 1 --2- 172.21.15.42:0/194955534 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f52140b3ce0 0x7f52140b61f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5218009990 tx=0x7f5218009110 comp rx=0 tx=0).stop 2024-07-26T06:48:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.964+0000 7f522b1c2640 1 -- 172.21.15.42:0/194955534 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5224109830 msgr2=0x7f52241a1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.964+0000 7f522b1c2640 1 --2- 172.21.15.42:0/194955534 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5224109830 0x7f52241a1a70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f521c003bd0 tx=0x7f521c003c00 comp rx=0 tx=0).stop 2024-07-26T06:48:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.964+0000 7f522b1c2640 1 -- 172.21.15.42:0/194955534 shutdown_connections 2024-07-26T06:48:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.964+0000 7f522b1c2640 1 --2- 172.21.15.42:0/194955534 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f52140b3ce0 0x7f52140b61f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.964+0000 7f522b1c2640 1 --2- 172.21.15.42:0/194955534 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5224109830 0x7f52241a1a70 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.964+0000 7f522b1c2640 1 --2- 172.21.15.42:0/194955534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5224108ee0 0x7f522419c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.964+0000 7f522b1c2640 1 --2- 172.21.15.42:0/194955534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f52241035f0 0x7f522419c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.964+0000 7f522b1c2640 1 -- 172.21.15.42:0/194955534 >> 172.21.15.42:0/194955534 conn(0x7f522407b340 msgr2=0x7f5224106cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.964+0000 7f522b1c2640 1 -- 172.21.15.42:0/194955534 shutdown_connections 2024-07-26T06:48:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:06.964+0000 7f522b1c2640 1 -- 172.21.15.42:0/194955534 wait complete. 2024-07-26T06:48:07.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:06 smithi042 ceph-mon[32775]: pgmap v332: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T06:48:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:06 smithi164 ceph-mon[36970]: pgmap v332: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T06:48:07.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:06 smithi163 ceph-mon[36873]: pgmap v332: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T06:48:07.678 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-07-26T06:48:07.678 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-07-26T06:48:07.679 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-07-26T06:48:07.679 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-07-26T06:48:07.680 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-07-26T06:48:07.680 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-07-26T06:48:07.680 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-07-26T06:48:07.681 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-07-26T06:48:07.681 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-07-26T06:48:07.682 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-07-26T06:48:07.682 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-07-26T06:48:07.683 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-07-26T06:48:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:07 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/194955534' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:48:08.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:07 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/194955534' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:48:08.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:07 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/194955534' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:48:08.407 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:08.423 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:09.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:08 smithi042 ceph-mon[32775]: pgmap v333: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:09.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:08 smithi164 ceph-mon[36970]: pgmap v333: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:09.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:08 smithi163 ceph-mon[36873]: pgmap v333: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:09.349 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:09.349 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:10.082 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:10.085 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:10.085 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:10.085 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:10.086 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:10.088 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:10.089 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:10.089 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:11.117 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:10 smithi042 ceph-mon[32775]: pgmap v334: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:48:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:10 smithi164 ceph-mon[36970]: pgmap v334: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:48:11.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:10 smithi163 ceph-mon[36873]: pgmap v334: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:48:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:12 smithi163 ceph-mon[36873]: pgmap v335: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:48:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:12 smithi042 ceph-mon[32775]: pgmap v335: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:48:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:12 smithi164 ceph-mon[36970]: pgmap v335: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:48:14.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.826+0000 7f4d42b61640 1 -- 172.21.15.42:0/945350852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d3c107e00 msgr2=0x7f4d3c108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:14.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.826+0000 7f4d42b61640 1 --2- 172.21.15.42:0/945350852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d3c107e00 0x7f4d3c108210 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c009ca0 tx=0x7f4d2c02a8c0 comp rx=0 tx=0).stop 2024-07-26T06:48:14.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.827+0000 7f4d42b61640 1 -- 172.21.15.42:0/945350852 shutdown_connections 2024-07-26T06:48:14.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.827+0000 7f4d42b61640 1 --2- 172.21.15.42:0/945350852 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d3c108750 0x7f4d3c10ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:14.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.827+0000 7f4d42b61640 1 --2- 172.21.15.42:0/945350852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d3c107e00 0x7f4d3c108210 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:14.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.827+0000 7f4d42b61640 1 --2- 172.21.15.42:0/945350852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d3c102510 0x7f4d3c102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:14.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.827+0000 7f4d42b61640 1 -- 172.21.15.42:0/945350852 >> 172.21.15.42:0/945350852 conn(0x7f4d3c0791e0 msgr2=0x7f4d3c0795f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:14.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.827+0000 7f4d42b61640 1 -- 172.21.15.42:0/945350852 shutdown_connections 2024-07-26T06:48:14.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.827+0000 7f4d42b61640 1 -- 172.21.15.42:0/945350852 wait complete. 2024-07-26T06:48:14.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.828+0000 7f4d42b61640 1 Processor -- start 2024-07-26T06:48:14.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.828+0000 7f4d42b61640 1 -- start start 2024-07-26T06:48:14.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.828+0000 7f4d42b61640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d3c102510 0x7f4d3c19acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:14.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.828+0000 7f4d42b61640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d3c107e00 0x7f4d3c19b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:14.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.828+0000 7f4d42b61640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d3c108750 0x7f4d3c19b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:14.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.828+0000 7f4d42b61640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c1123e0 con 0x7f4d3c102510 2024-07-26T06:48:14.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.828+0000 7f4d42b61640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c112260 con 0x7f4d3c108750 2024-07-26T06:48:14.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.828+0000 7f4d42b61640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c112560 con 0x7f4d3c107e00 2024-07-26T06:48:14.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.828+0000 7f4d408fd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d3c102510 0x7f4d3c19acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:14.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.829+0000 7f4d408fd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d3c102510 0x7f4d3c19acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46562/0 (socket says 172.21.15.42:46562) 2024-07-26T06:48:14.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.829+0000 7f4d408fd640 1 -- 172.21.15.42:0/3328452153 learned_addr learned my addr 172.21.15.42:0/3328452153 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:14.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.829+0000 7f4d410fe640 1 --2- 172.21.15.42:0/3328452153 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d3c108750 0x7f4d3c19b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:14.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.829+0000 7f4d3bfff640 1 --2- 172.21.15.42:0/3328452153 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d3c107e00 0x7f4d3c19b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:14.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.829+0000 7f4d410fe640 1 -- 172.21.15.42:0/3328452153 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d3c107e00 msgr2=0x7f4d3c19b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:14.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.829+0000 7f4d410fe640 1 --2- 172.21.15.42:0/3328452153 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4d3c107e00 0x7f4d3c19b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:14.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.829+0000 7f4d410fe640 1 -- 172.21.15.42:0/3328452153 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d3c102510 msgr2=0x7f4d3c19acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:14.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.829+0000 7f4d410fe640 1 --2- 172.21.15.42:0/3328452153 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d3c102510 0x7f4d3c19acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:14.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.829+0000 7f4d410fe640 1 -- 172.21.15.42:0/3328452153 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3c1a5fc0 con 0x7f4d3c108750 2024-07-26T06:48:14.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.829+0000 7f4d408fd640 1 --2- 172.21.15.42:0/3328452153 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d3c102510 0x7f4d3c19acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:48:14.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.829+0000 7f4d410fe640 1 --2- 172.21.15.42:0/3328452153 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d3c108750 0x7f4d3c19b730 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400d680 tx=0x7f4d3400db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:14.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.833+0000 7f4d39ffb640 1 -- 172.21.15.42:0/3328452153 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d34017070 con 0x7f4d3c108750 2024-07-26T06:48:14.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.833+0000 7f4d39ffb640 1 -- 172.21.15.42:0/3328452153 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d34009820 con 0x7f4d3c108750 2024-07-26T06:48:14.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.833+0000 7f4d42b61640 1 -- 172.21.15.42:0/3328452153 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d3c1a62b0 con 0x7f4d3c108750 2024-07-26T06:48:14.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.833+0000 7f4d39ffb640 1 -- 172.21.15.42:0/3328452153 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d34004ce0 con 0x7f4d3c108750 2024-07-26T06:48:14.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.833+0000 7f4d42b61640 1 -- 172.21.15.42:0/3328452153 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d3c1a67c0 con 0x7f4d3c108750 2024-07-26T06:48:14.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.834+0000 7f4d42b61640 1 -- 172.21.15.42:0/3328452153 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4d3c063540 con 0x7f4d3c108750 2024-07-26T06:48:14.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.836+0000 7f4d39ffb640 1 -- 172.21.15.42:0/3328452153 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f4d34005220 con 0x7f4d3c108750 2024-07-26T06:48:14.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.836+0000 7f4d39ffb640 1 --2- 172.21.15.42:0/3328452153 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f4d280b3ce0 0x7f4d280b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:14.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.837+0000 7f4d408fd640 1 --2- 172.21.15.42:0/3328452153 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f4d280b3ce0 0x7f4d280b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:14.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.837+0000 7f4d39ffb640 1 -- 172.21.15.42:0/3328452153 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f4d340e41e0 con 0x7f4d3c108750 2024-07-26T06:48:14.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.837+0000 7f4d39ffb640 1 --2- 172.21.15.42:0/3328452153 >> [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300] conn(0x7f4d280bd9e0 0x7f4d280bfed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:14.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.837+0000 7f4d39ffb640 1 -- 172.21.15.42:0/3328452153 --> [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d34016ec0 con 0x7f4d280bd9e0 2024-07-26T06:48:14.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.837+0000 7f4d39ffb640 1 -- 172.21.15.42:0/3328452153 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f4d340e4620 con 0x7f4d3c108750 2024-07-26T06:48:14.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.837+0000 7f4d3bfff640 1 --2- 172.21.15.42:0/3328452153 >> [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300] conn(0x7f4d280bd9e0 0x7f4d280bfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:14.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.837+0000 7f4d408fd640 1 --2- 172.21.15.42:0/3328452153 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f4d280b3ce0 0x7f4d280b61f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4d24005fd0 tx=0x7f4d2400b480 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:14.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.837+0000 7f4d3bfff640 1 --2- 172.21.15.42:0/3328452153 >> [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300] conn(0x7f4d280bd9e0 0x7f4d280bfed0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:14.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.838+0000 7f4d39ffb640 1 -- 172.21.15.42:0/3328452153 <== osd.6 v2:172.21.15.42:6818/2160811300 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d34016ec0 con 0x7f4d280bd9e0 2024-07-26T06:48:14.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.853+0000 7f4d42b61640 1 -- 172.21.15.42:0/3328452153 --> [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4d3c063970 con 0x7f4d280bd9e0 2024-07-26T06:48:14.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.853+0000 7f4d39ffb640 1 -- 172.21.15.42:0/3328452153 <== osd.6 v2:172.21.15.42:6818/2160811300 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4d3c063970 con 0x7f4d280bd9e0 2024-07-26T06:48:14.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.853+0000 7f4d42b61640 1 -- 172.21.15.42:0/3328452153 >> [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300] conn(0x7f4d280bd9e0 msgr2=0x7f4d280bfed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:14.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.853+0000 7f4d42b61640 1 --2- 172.21.15.42:0/3328452153 >> [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300] conn(0x7f4d280bd9e0 0x7f4d280bfed0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:14.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.854+0000 7f4d42b61640 1 -- 172.21.15.42:0/3328452153 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f4d280b3ce0 msgr2=0x7f4d280b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:14.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.854+0000 7f4d42b61640 1 --2- 172.21.15.42:0/3328452153 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f4d280b3ce0 0x7f4d280b61f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4d24005fd0 tx=0x7f4d2400b480 comp rx=0 tx=0).stop 2024-07-26T06:48:14.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.854+0000 7f4d42b61640 1 -- 172.21.15.42:0/3328452153 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d3c108750 msgr2=0x7f4d3c19b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:14.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.854+0000 7f4d42b61640 1 --2- 172.21.15.42:0/3328452153 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4d3c108750 0x7f4d3c19b730 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400d680 tx=0x7f4d3400db40 comp rx=0 tx=0).stop 2024-07-26T06:48:14.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.854+0000 7f4d410fe640 1 -- 172.21.15.42:0/3328452153 reap_dead start 2024-07-26T06:48:14.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.854+0000 7f4d42b61640 1 -- 172.21.15.42:0/3328452153 shutdown_connections 2024-07-26T06:48:14.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.854+0000 7f4d42b61640 1 -- 172.21.15.42:0/3328452153 >> 172.21.15.42:0/3328452153 conn(0x7f4d3c0791e0 msgr2=0x7f4d3c105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:14.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.854+0000 7f4d42b61640 1 -- 172.21.15.42:0/3328452153 shutdown_connections 2024-07-26T06:48:14.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:14.854+0000 7f4d42b61640 1 -- 172.21.15.42:0/3328452153 wait complete. 2024-07-26T06:48:15.030 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:14 smithi042 ceph-mon[32775]: pgmap v336: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:15.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:14 smithi164 ceph-mon[36970]: pgmap v336: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:14 smithi163 ceph-mon[36873]: pgmap v336: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:17.089 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:16 smithi042 ceph-mon[32775]: pgmap v337: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s 2024-07-26T06:48:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:16 smithi164 ceph-mon[36970]: pgmap v337: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s 2024-07-26T06:48:17.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:16 smithi163 ceph-mon[36873]: pgmap v337: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s 2024-07-26T06:48:19.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:18 smithi042 ceph-mon[32775]: pgmap v338: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:18 smithi164 ceph-mon[36970]: pgmap v338: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:19.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:18 smithi163 ceph-mon[36873]: pgmap v338: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:20.317 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:20 smithi042 ceph-mon[32775]: pgmap v339: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:48:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:20 smithi164 ceph-mon[36970]: pgmap v339: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:48:20.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:20 smithi163 ceph-mon[36873]: pgmap v339: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:48:20.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.293+0000 7fc0ff313640 1 -- 172.21.15.42:0/3123295772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0f8109820 msgr2=0x7fc0f8110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:20.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.294+0000 7fc0ff313640 1 --2- 172.21.15.42:0/3123295772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0f8109820 0x7fc0f8110050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0009ca0 tx=0x7fc0f002a8c0 comp rx=0 tx=0).stop 2024-07-26T06:48:20.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.294+0000 7fc0ff313640 1 -- 172.21.15.42:0/3123295772 shutdown_connections 2024-07-26T06:48:20.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.294+0000 7fc0ff313640 1 --2- 172.21.15.42:0/3123295772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0f8109820 0x7fc0f8110050 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:20.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.294+0000 7fc0ff313640 1 --2- 172.21.15.42:0/3123295772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0f8108ed0 0x7fc0f81092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:20.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.294+0000 7fc0ff313640 1 --2- 172.21.15.42:0/3123295772 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0f8103590 0x7fc0f81039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:20.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.294+0000 7fc0ff313640 1 -- 172.21.15.42:0/3123295772 >> 172.21.15.42:0/3123295772 conn(0x7fc0f807b280 msgr2=0x7fc0f807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:20.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.294+0000 7fc0ff313640 1 -- 172.21.15.42:0/3123295772 shutdown_connections 2024-07-26T06:48:20.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.294+0000 7fc0ff313640 1 -- 172.21.15.42:0/3123295772 wait complete. 2024-07-26T06:48:20.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.294+0000 7fc0ff313640 1 Processor -- start 2024-07-26T06:48:20.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.294+0000 7fc0ff313640 1 -- start start 2024-07-26T06:48:20.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.295+0000 7fc0ff313640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0f8103590 0x7fc0f8199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:20.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.295+0000 7fc0ff313640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0f8108ed0 0x7fc0f819a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:20.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.295+0000 7fc0ff313640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0f8109820 0x7fc0f819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:20.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.295+0000 7fc0ff313640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81134f0 con 0x7fc0f8108ed0 2024-07-26T06:48:20.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.295+0000 7fc0ff313640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f8113370 con 0x7fc0f8103590 2024-07-26T06:48:20.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.295+0000 7fc0ff313640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f8113670 con 0x7fc0f8109820 2024-07-26T06:48:20.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.295+0000 7fc0fc8ae640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0f8108ed0 0x7fc0f819a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:20.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.295+0000 7fc0fc8ae640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0f8108ed0 0x7fc0f819a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43690/0 (socket says 172.21.15.42:43690) 2024-07-26T06:48:20.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.295+0000 7fc0fc8ae640 1 -- 172.21.15.42:0/3867159554 learned_addr learned my addr 172.21.15.42:0/3867159554 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:20.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.295+0000 7fc0fd8b0640 1 --2- 172.21.15.42:0/3867159554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0f8109820 0x7fc0f819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:20.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.295+0000 7fc0fd0af640 1 --2- 172.21.15.42:0/3867159554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0f8103590 0x7fc0f8199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:20.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.295+0000 7fc0fc8ae640 1 -- 172.21.15.42:0/3867159554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0f8109820 msgr2=0x7fc0f819f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:20.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.295+0000 7fc0fc8ae640 1 --2- 172.21.15.42:0/3867159554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc0f8109820 0x7fc0f819f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:20.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.295+0000 7fc0fc8ae640 1 -- 172.21.15.42:0/3867159554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0f8103590 msgr2=0x7fc0f8199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:20.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.295+0000 7fc0fc8ae640 1 --2- 172.21.15.42:0/3867159554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0f8103590 0x7fc0f8199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:20.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.295+0000 7fc0fc8ae640 1 -- 172.21.15.42:0/3867159554 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f819fdf0 con 0x7fc0f8108ed0 2024-07-26T06:48:20.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.295+0000 7fc0fc8ae640 1 --2- 172.21.15.42:0/3867159554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0f8108ed0 0x7fc0f819a420 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc0e800d750 tx=0x7fc0e800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:20.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.296+0000 7fc0ee7fc640 1 -- 172.21.15.42:0/3867159554 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8017070 con 0x7fc0f8108ed0 2024-07-26T06:48:20.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.296+0000 7fc0ff313640 1 -- 172.21.15.42:0/3867159554 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f81a00e0 con 0x7fc0f8108ed0 2024-07-26T06:48:20.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.296+0000 7fc0ff313640 1 -- 172.21.15.42:0/3867159554 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f81a6840 con 0x7fc0f8108ed0 2024-07-26T06:48:20.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.296+0000 7fc0ee7fc640 1 -- 172.21.15.42:0/3867159554 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0e80054d0 con 0x7fc0f8108ed0 2024-07-26T06:48:20.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.296+0000 7fc0ee7fc640 1 -- 172.21.15.42:0/3867159554 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8020060 con 0x7fc0f8108ed0 2024-07-26T06:48:20.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.297+0000 7fc0ff313640 1 -- 172.21.15.42:0/3867159554 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc0f8063540 con 0x7fc0f8108ed0 2024-07-26T06:48:20.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.299+0000 7fc0ee7fc640 1 -- 172.21.15.42:0/3867159554 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7fc0e8004c10 con 0x7fc0f8108ed0 2024-07-26T06:48:20.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.300+0000 7fc0ee7fc640 1 --2- 172.21.15.42:0/3867159554 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc0e40b3ce0 0x7fc0e40b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:20.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.300+0000 7fc0fd0af640 1 --2- 172.21.15.42:0/3867159554 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc0e40b3ce0 0x7fc0e40b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:20.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.300+0000 7fc0ee7fc640 1 -- 172.21.15.42:0/3867159554 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fc0e80e3b10 con 0x7fc0f8108ed0 2024-07-26T06:48:20.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.300+0000 7fc0ee7fc640 1 --2- 172.21.15.42:0/3867159554 >> [v2:172.21.15.164:6808/261549362,v1:172.21.15.164:6809/261549362] conn(0x7fc0e40bd9e0 0x7fc0e40bfed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:20.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.300+0000 7fc0ee7fc640 1 -- 172.21.15.42:0/3867159554 --> [v2:172.21.15.164:6808/261549362,v1:172.21.15.164:6809/261549362] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc0e801dec0 con 0x7fc0e40bd9e0 2024-07-26T06:48:20.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.300+0000 7fc0ee7fc640 1 -- 172.21.15.42:0/3867159554 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7fc0e80e3f00 con 0x7fc0f8108ed0 2024-07-26T06:48:20.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.301+0000 7fc0fd0af640 1 --2- 172.21.15.42:0/3867159554 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc0e40b3ce0 0x7fc0e40b61f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4009bd0 tx=0x7fc0f4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:20.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.301+0000 7fc0fd8b0640 1 --2- 172.21.15.42:0/3867159554 >> [v2:172.21.15.164:6808/261549362,v1:172.21.15.164:6809/261549362] conn(0x7fc0e40bd9e0 0x7fc0e40bfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.301+0000 7fc0fd8b0640 1 --2- 172.21.15.42:0/3867159554 >> [v2:172.21.15.164:6808/261549362,v1:172.21.15.164:6809/261549362] conn(0x7fc0e40bd9e0 0x7fc0e40bfed0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.302+0000 7fc0ee7fc640 1 -- 172.21.15.42:0/3867159554 <== osd.5 v2:172.21.15.164:6808/261549362 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc0e801dec0 con 0x7fc0e40bd9e0 2024-07-26T06:48:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.316+0000 7fc0ff313640 1 -- 172.21.15.42:0/3867159554 --> [v2:172.21.15.164:6808/261549362,v1:172.21.15.164:6809/261549362] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc0f8063940 con 0x7fc0e40bd9e0 2024-07-26T06:48:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.317+0000 7fc0ee7fc640 1 -- 172.21.15.42:0/3867159554 <== osd.5 v2:172.21.15.164:6808/261549362 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc0f8063940 con 0x7fc0e40bd9e0 2024-07-26T06:48:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.317+0000 7fc0ff313640 1 -- 172.21.15.42:0/3867159554 >> [v2:172.21.15.164:6808/261549362,v1:172.21.15.164:6809/261549362] conn(0x7fc0e40bd9e0 msgr2=0x7fc0e40bfed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.317+0000 7fc0ff313640 1 --2- 172.21.15.42:0/3867159554 >> [v2:172.21.15.164:6808/261549362,v1:172.21.15.164:6809/261549362] conn(0x7fc0e40bd9e0 0x7fc0e40bfed0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.317+0000 7fc0ff313640 1 -- 172.21.15.42:0/3867159554 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc0e40b3ce0 msgr2=0x7fc0e40b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.317+0000 7fc0ff313640 1 --2- 172.21.15.42:0/3867159554 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc0e40b3ce0 0x7fc0e40b61f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4009bd0 tx=0x7fc0f4005f90 comp rx=0 tx=0).stop 2024-07-26T06:48:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.317+0000 7fc0ff313640 1 -- 172.21.15.42:0/3867159554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0f8108ed0 msgr2=0x7fc0f819a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.317+0000 7fc0ff313640 1 --2- 172.21.15.42:0/3867159554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0f8108ed0 0x7fc0f819a420 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc0e800d750 tx=0x7fc0e800dc10 comp rx=0 tx=0).stop 2024-07-26T06:48:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.317+0000 7fc0fd8b0640 1 -- 172.21.15.42:0/3867159554 reap_dead start 2024-07-26T06:48:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.317+0000 7fc0ff313640 1 -- 172.21.15.42:0/3867159554 shutdown_connections 2024-07-26T06:48:20.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.317+0000 7fc0ff313640 1 -- 172.21.15.42:0/3867159554 >> 172.21.15.42:0/3867159554 conn(0x7fc0f807b280 msgr2=0x7fc0f8106f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:20.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.317+0000 7fc0ff313640 1 -- 172.21.15.42:0/3867159554 shutdown_connections 2024-07-26T06:48:20.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:20.317+0000 7fc0ff313640 1 -- 172.21.15.42:0/3867159554 wait complete. 2024-07-26T06:48:21.125 INFO:teuthology.orchestra.run.smithi042.stdout:154618822716 2024-07-26T06:48:21.126 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-07-26T06:48:21.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.288+0000 7f4770ad3640 1 -- 172.21.15.42:0/4161864449 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f476c1024b0 msgr2=0x7f476c1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:21.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.288+0000 7f4770ad3640 1 --2- 172.21.15.42:0/4161864449 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f476c1024b0 0x7f476c1028c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f475c009900 tx=0x7f475c02a3f0 comp rx=0 tx=0).stop 2024-07-26T06:48:21.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.288+0000 7f4770ad3640 1 -- 172.21.15.42:0/4161864449 shutdown_connections 2024-07-26T06:48:21.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.288+0000 7f4770ad3640 1 --2- 172.21.15.42:0/4161864449 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f476c108740 0x7f476c10ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:21.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.288+0000 7f4770ad3640 1 --2- 172.21.15.42:0/4161864449 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f476c107df0 0x7f476c108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:21.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.288+0000 7f4770ad3640 1 --2- 172.21.15.42:0/4161864449 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f476c1024b0 0x7f476c1028c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:21.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.288+0000 7f4770ad3640 1 -- 172.21.15.42:0/4161864449 >> 172.21.15.42:0/4161864449 conn(0x7f476c079170 msgr2=0x7f476c079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:21.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.288+0000 7f4770ad3640 1 -- 172.21.15.42:0/4161864449 shutdown_connections 2024-07-26T06:48:21.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.288+0000 7f4770ad3640 1 -- 172.21.15.42:0/4161864449 wait complete. 2024-07-26T06:48:21.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.289+0000 7f4770ad3640 1 Processor -- start 2024-07-26T06:48:21.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.289+0000 7f4770ad3640 1 -- start start 2024-07-26T06:48:21.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.290+0000 7f4770ad3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f476c1024b0 0x7f476c111410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:21.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.290+0000 7f4770ad3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f476c107df0 0x7f476c10fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:21.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.290+0000 7f476a59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f476c1024b0 0x7f476c111410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:21.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.290+0000 7f476a59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f476c1024b0 0x7f476c111410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:35516/0 (socket says 172.21.15.42:35516) 2024-07-26T06:48:21.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.290+0000 7f476a59c640 1 -- 172.21.15.42:0/1780294614 learned_addr learned my addr 172.21.15.42:0/1780294614 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:21.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.290+0000 7f4770ad3640 1 --2- 172.21.15.42:0/1780294614 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f476c108740 0x7f476c10ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:21.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.290+0000 7f4770ad3640 1 -- 172.21.15.42:0/1780294614 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f476c113500 con 0x7f476c108740 2024-07-26T06:48:21.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.290+0000 7f4770ad3640 1 -- 172.21.15.42:0/1780294614 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f476c113380 con 0x7f476c107df0 2024-07-26T06:48:21.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.290+0000 7f4770ad3640 1 -- 172.21.15.42:0/1780294614 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f476c113680 con 0x7f476c1024b0 2024-07-26T06:48:21.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.290+0000 7f4769d9b640 1 --2- 172.21.15.42:0/1780294614 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f476c107df0 0x7f476c10fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:21.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.290+0000 7f476a59c640 1 -- 172.21.15.42:0/1780294614 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f476c107df0 msgr2=0x7f476c10fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:21.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.290+0000 7f476a59c640 1 --2- 172.21.15.42:0/1780294614 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f476c107df0 0x7f476c10fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:21.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.290+0000 7f476a59c640 1 -- 172.21.15.42:0/1780294614 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f476c108740 msgr2=0x7f476c10ff40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-07-26T06:48:21.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.290+0000 7f476a59c640 1 --2- 172.21.15.42:0/1780294614 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f476c108740 0x7f476c10ff40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:21.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.291+0000 7f476a59c640 1 -- 172.21.15.42:0/1780294614 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476c1106d0 con 0x7f476c1024b0 2024-07-26T06:48:21.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.291+0000 7f4769d9b640 1 --2- 172.21.15.42:0/1780294614 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f476c107df0 0x7f476c10fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:48:21.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.291+0000 7f476a59c640 1 --2- 172.21.15.42:0/1780294614 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f476c1024b0 0x7f476c111410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f475c005ab0 tx=0x7f475c003cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:21.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.291+0000 7f475b7fe640 1 -- 172.21.15.42:0/1780294614 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c03f070 con 0x7f476c1024b0 2024-07-26T06:48:21.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.291+0000 7f475b7fe640 1 -- 172.21.15.42:0/1780294614 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f475c031d60 con 0x7f476c1024b0 2024-07-26T06:48:21.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.291+0000 7f4770ad3640 1 -- 172.21.15.42:0/1780294614 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f476c1ac7f0 con 0x7f476c1024b0 2024-07-26T06:48:21.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.292+0000 7f4770ad3640 1 -- 172.21.15.42:0/1780294614 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f476c1acc40 con 0x7f476c1024b0 2024-07-26T06:48:21.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.292+0000 7f475b7fe640 1 -- 172.21.15.42:0/1780294614 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c035030 con 0x7f476c1024b0 2024-07-26T06:48:21.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.293+0000 7f4770ad3640 1 -- 172.21.15.42:0/1780294614 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f474c001030 con 0x7f476c1024b0 2024-07-26T06:48:21.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.295+0000 7f475b7fe640 1 -- 172.21.15.42:0/1780294614 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f475c042070 con 0x7f476c1024b0 2024-07-26T06:48:21.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.296+0000 7f475b7fe640 1 --2- 172.21.15.42:0/1780294614 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f47540b3db0 0x7f47540b62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:21.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.296+0000 7f4769d9b640 1 --2- 172.21.15.42:0/1780294614 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f47540b3db0 0x7f47540b62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:21.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.296+0000 7f475b7fe640 1 -- 172.21.15.42:0/1780294614 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f475c1048a0 con 0x7f476c1024b0 2024-07-26T06:48:21.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.296+0000 7f475b7fe640 1 --2- 172.21.15.42:0/1780294614 >> [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379] conn(0x7f47540bdab0 0x7f47540bffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:21.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.296+0000 7f475b7fe640 1 -- 172.21.15.42:0/1780294614 --> [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f475c036e90 con 0x7f47540bdab0 2024-07-26T06:48:21.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.296+0000 7f475b7fe640 1 -- 172.21.15.42:0/1780294614 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f475c104c50 con 0x7f476c1024b0 2024-07-26T06:48:21.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.297+0000 7f476ad9d640 1 --2- 172.21.15.42:0/1780294614 >> [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379] conn(0x7f47540bdab0 0x7f47540bffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:21.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.297+0000 7f4769d9b640 1 --2- 172.21.15.42:0/1780294614 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f47540b3db0 0x7f47540b62c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f476c107bf0 tx=0x7f476000a2d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:21.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.297+0000 7f476ad9d640 1 --2- 172.21.15.42:0/1780294614 >> [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379] conn(0x7f47540bdab0 0x7f47540bffa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:21.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.298+0000 7f475b7fe640 1 -- 172.21.15.42:0/1780294614 <== osd.0 v2:172.21.15.42:6802/3587066379 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f475c036e90 con 0x7f47540bdab0 2024-07-26T06:48:21.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.312+0000 7f4770ad3640 1 -- 172.21.15.42:0/1780294614 --> [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f474c002dd0 con 0x7f47540bdab0 2024-07-26T06:48:21.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.312+0000 7f475b7fe640 1 -- 172.21.15.42:0/1780294614 <== osd.0 v2:172.21.15.42:6802/3587066379 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f474c002dd0 con 0x7f47540bdab0 2024-07-26T06:48:21.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.313+0000 7f4770ad3640 1 -- 172.21.15.42:0/1780294614 >> [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379] conn(0x7f47540bdab0 msgr2=0x7f47540bffa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:21.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.313+0000 7f4770ad3640 1 --2- 172.21.15.42:0/1780294614 >> [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379] conn(0x7f47540bdab0 0x7f47540bffa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:21.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.313+0000 7f4770ad3640 1 -- 172.21.15.42:0/1780294614 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f47540b3db0 msgr2=0x7f47540b62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:21.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.313+0000 7f4770ad3640 1 --2- 172.21.15.42:0/1780294614 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f47540b3db0 0x7f47540b62c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f476c107bf0 tx=0x7f476000a2d0 comp rx=0 tx=0).stop 2024-07-26T06:48:21.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.313+0000 7f4770ad3640 1 -- 172.21.15.42:0/1780294614 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f476c1024b0 msgr2=0x7f476c111410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:21.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.313+0000 7f4770ad3640 1 --2- 172.21.15.42:0/1780294614 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f476c1024b0 0x7f476c111410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f475c005ab0 tx=0x7f475c003cd0 comp rx=0 tx=0).stop 2024-07-26T06:48:21.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.313+0000 7f476ad9d640 1 -- 172.21.15.42:0/1780294614 reap_dead start 2024-07-26T06:48:21.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.313+0000 7f4770ad3640 1 -- 172.21.15.42:0/1780294614 shutdown_connections 2024-07-26T06:48:21.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.313+0000 7f4770ad3640 1 -- 172.21.15.42:0/1780294614 >> 172.21.15.42:0/1780294614 conn(0x7f476c079170 msgr2=0x7f476c100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:21.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.313+0000 7f4770ad3640 1 -- 172.21.15.42:0/1780294614 shutdown_connections 2024-07-26T06:48:21.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:21.313+0000 7f4770ad3640 1 -- 172.21.15.42:0/1780294614 wait complete. 2024-07-26T06:48:22.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:22 smithi042 ceph-mon[32775]: pgmap v340: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T06:48:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:22 smithi164 ceph-mon[36970]: pgmap v340: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T06:48:22.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:22 smithi163 ceph-mon[36873]: pgmap v340: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T06:48:23.030 INFO:teuthology.orchestra.run.smithi042.stdout:180388626486 2024-07-26T06:48:23.031 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-07-26T06:48:24.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.850+0000 7fcb51624640 1 -- 172.21.15.42:0/3151337775 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb4c10d860 msgr2=0x7fcb4c10fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:24.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.850+0000 7fcb51624640 1 --2- 172.21.15.42:0/3151337775 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb4c10d860 0x7fcb4c10fd30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009ca0 tx=0x7fcb4402a8c0 comp rx=0 tx=0).stop 2024-07-26T06:48:24.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.850+0000 7fcb51624640 1 -- 172.21.15.42:0/3151337775 shutdown_connections 2024-07-26T06:48:24.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.850+0000 7fcb51624640 1 --2- 172.21.15.42:0/3151337775 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb4c10d860 0x7fcb4c10fd30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:24.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.850+0000 7fcb51624640 1 --2- 172.21.15.42:0/3151337775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcb4c104ce0 0x7fcb4c10d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:24.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.850+0000 7fcb51624640 1 --2- 172.21.15.42:0/3151337775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb4c1042b0 0x7fcb4c1046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:24.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.850+0000 7fcb51624640 1 -- 172.21.15.42:0/3151337775 >> 172.21.15.42:0/3151337775 conn(0x7fcb4c079160 msgr2=0x7fcb4c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:24.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.850+0000 7fcb51624640 1 -- 172.21.15.42:0/3151337775 shutdown_connections 2024-07-26T06:48:24.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.850+0000 7fcb51624640 1 -- 172.21.15.42:0/3151337775 wait complete. 2024-07-26T06:48:24.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.851+0000 7fcb51624640 1 Processor -- start 2024-07-26T06:48:24.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.851+0000 7fcb51624640 1 -- start start 2024-07-26T06:48:24.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.851+0000 7fcb51624640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcb4c1042b0 0x7fcb4c199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:24.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.851+0000 7fcb51624640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb4c104ce0 0x7fcb4c19a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:24.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.851+0000 7fcb51624640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb4c10d860 0x7fcb4c19f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:24.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.851+0000 7fcb51624640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4c1134a0 con 0x7fcb4c10d860 2024-07-26T06:48:24.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.851+0000 7fcb51624640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4c113320 con 0x7fcb4c1042b0 2024-07-26T06:48:24.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.851+0000 7fcb51624640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4c113620 con 0x7fcb4c104ce0 2024-07-26T06:48:24.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.851+0000 7fcb4b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb4c10d860 0x7fcb4c19f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:24.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.851+0000 7fcb4b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb4c10d860 0x7fcb4c19f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43722/0 (socket says 172.21.15.42:43722) 2024-07-26T06:48:24.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.851+0000 7fcb4b7fe640 1 -- 172.21.15.42:0/3248274284 learned_addr learned my addr 172.21.15.42:0/3248274284 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:24.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.852+0000 7fcb4affd640 1 --2- 172.21.15.42:0/3248274284 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcb4c1042b0 0x7fcb4c199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:24.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.852+0000 7fcb4a7fc640 1 --2- 172.21.15.42:0/3248274284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb4c104ce0 0x7fcb4c19a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:24.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.852+0000 7fcb4b7fe640 1 -- 172.21.15.42:0/3248274284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb4c104ce0 msgr2=0x7fcb4c19a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:24.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.852+0000 7fcb4b7fe640 1 --2- 172.21.15.42:0/3248274284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb4c104ce0 0x7fcb4c19a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:24.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.852+0000 7fcb4b7fe640 1 -- 172.21.15.42:0/3248274284 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcb4c1042b0 msgr2=0x7fcb4c199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:24.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.852+0000 7fcb4b7fe640 1 --2- 172.21.15.42:0/3248274284 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcb4c1042b0 0x7fcb4c199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:24.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.852+0000 7fcb4b7fe640 1 -- 172.21.15.42:0/3248274284 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb4c19fdf0 con 0x7fcb4c10d860 2024-07-26T06:48:24.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.852+0000 7fcb4a7fc640 1 --2- 172.21.15.42:0/3248274284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb4c104ce0 0x7fcb4c19a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:48:24.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.852+0000 7fcb4b7fe640 1 --2- 172.21.15.42:0/3248274284 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb4c10d860 0x7fcb4c19f6c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcb440098d0 tx=0x7fcb440317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:24.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.852+0000 7fcb33fff640 1 -- 172.21.15.42:0/3248274284 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb44003b70 con 0x7fcb4c10d860 2024-07-26T06:48:24.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.852+0000 7fcb51624640 1 -- 172.21.15.42:0/3248274284 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb4c1a0080 con 0x7fcb4c10d860 2024-07-26T06:48:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.852+0000 7fcb33fff640 1 -- 172.21.15.42:0/3248274284 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb44003d10 con 0x7fcb4c10d860 2024-07-26T06:48:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.852+0000 7fcb33fff640 1 -- 172.21.15.42:0/3248274284 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4402fa20 con 0x7fcb4c10d860 2024-07-26T06:48:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.852+0000 7fcb51624640 1 -- 172.21.15.42:0/3248274284 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb4c1a6840 con 0x7fcb4c10d860 2024-07-26T06:48:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.853+0000 7fcb51624640 1 -- 172.21.15.42:0/3248274284 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcb4c107df0 con 0x7fcb4c10d860 2024-07-26T06:48:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.855+0000 7fcb33fff640 1 -- 172.21.15.42:0/3248274284 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7fcb4402fbc0 con 0x7fcb4c10d860 2024-07-26T06:48:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.856+0000 7fcb33fff640 1 --2- 172.21.15.42:0/3248274284 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fcb340b3c60 0x7fcb340b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.856+0000 7fcb4affd640 1 --2- 172.21.15.42:0/3248274284 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fcb340b3c60 0x7fcb340b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.856+0000 7fcb33fff640 1 -- 172.21.15.42:0/3248274284 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fcb44105440 con 0x7fcb4c10d860 2024-07-26T06:48:24.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.857+0000 7fcb33fff640 1 --2- 172.21.15.42:0/3248274284 >> [v2:172.21.15.164:6800/336113412,v1:172.21.15.164:6801/336113412] conn(0x7fcb340bd960 0x7fcb340bfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.857+0000 7fcb33fff640 1 -- 172.21.15.42:0/3248274284 --> [v2:172.21.15.164:6800/336113412,v1:172.21.15.164:6801/336113412] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb44047ec0 con 0x7fcb340bd960 2024-07-26T06:48:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.857+0000 7fcb33fff640 1 -- 172.21.15.42:0/3248274284 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7fcb44105890 con 0x7fcb4c10d860 2024-07-26T06:48:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.857+0000 7fcb4affd640 1 --2- 172.21.15.42:0/3248274284 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fcb340b3c60 0x7fcb340b6170 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0058c0 tx=0x7fcb3c00a2d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.857+0000 7fcb4a7fc640 1 --2- 172.21.15.42:0/3248274284 >> [v2:172.21.15.164:6800/336113412,v1:172.21.15.164:6801/336113412] conn(0x7fcb340bd960 0x7fcb340bfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.858+0000 7fcb4a7fc640 1 --2- 172.21.15.42:0/3248274284 >> [v2:172.21.15.164:6800/336113412,v1:172.21.15.164:6801/336113412] conn(0x7fcb340bd960 0x7fcb340bfe50 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.858+0000 7fcb33fff640 1 -- 172.21.15.42:0/3248274284 <== osd.2 v2:172.21.15.164:6800/336113412 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcb44047ec0 con 0x7fcb340bd960 2024-07-26T06:48:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.886+0000 7fcb51624640 1 -- 172.21.15.42:0/3248274284 --> [v2:172.21.15.164:6800/336113412,v1:172.21.15.164:6801/336113412] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcb4c06aef0 con 0x7fcb340bd960 2024-07-26T06:48:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.886+0000 7fcb33fff640 1 -- 172.21.15.42:0/3248274284 <== osd.2 v2:172.21.15.164:6800/336113412 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcb4c06aef0 con 0x7fcb340bd960 2024-07-26T06:48:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.887+0000 7fcb51624640 1 -- 172.21.15.42:0/3248274284 >> [v2:172.21.15.164:6800/336113412,v1:172.21.15.164:6801/336113412] conn(0x7fcb340bd960 msgr2=0x7fcb340bfe50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.887+0000 7fcb51624640 1 --2- 172.21.15.42:0/3248274284 >> [v2:172.21.15.164:6800/336113412,v1:172.21.15.164:6801/336113412] conn(0x7fcb340bd960 0x7fcb340bfe50 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.887+0000 7fcb51624640 1 -- 172.21.15.42:0/3248274284 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fcb340b3c60 msgr2=0x7fcb340b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.887+0000 7fcb51624640 1 --2- 172.21.15.42:0/3248274284 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fcb340b3c60 0x7fcb340b6170 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0058c0 tx=0x7fcb3c00a2d0 comp rx=0 tx=0).stop 2024-07-26T06:48:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.887+0000 7fcb51624640 1 -- 172.21.15.42:0/3248274284 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb4c10d860 msgr2=0x7fcb4c19f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.887+0000 7fcb51624640 1 --2- 172.21.15.42:0/3248274284 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb4c10d860 0x7fcb4c19f6c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcb440098d0 tx=0x7fcb440317b0 comp rx=0 tx=0).stop 2024-07-26T06:48:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.887+0000 7fcb4b7fe640 1 -- 172.21.15.42:0/3248274284 reap_dead start 2024-07-26T06:48:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.887+0000 7fcb51624640 1 -- 172.21.15.42:0/3248274284 shutdown_connections 2024-07-26T06:48:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.887+0000 7fcb51624640 1 -- 172.21.15.42:0/3248274284 >> 172.21.15.42:0/3248274284 conn(0x7fcb4c079160 msgr2=0x7fcb4c107390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.887+0000 7fcb51624640 1 -- 172.21.15.42:0/3248274284 shutdown_connections 2024-07-26T06:48:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:23.887+0000 7fcb51624640 1 -- 172.21.15.42:0/3248274284 wait complete. 2024-07-26T06:48:24.622 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:24 smithi042 ceph-mon[32775]: pgmap v341: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:24.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:24 smithi164 ceph-mon[36970]: pgmap v341: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:24.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:24 smithi163 ceph-mon[36873]: pgmap v341: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:25.508 INFO:teuthology.orchestra.run.smithi042.stdout:34359738451 2024-07-26T06:48:25.509 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-07-26T06:48:26.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.190+0000 7f9eca039640 1 -- 172.21.15.42:0/3191012025 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec4108720 msgr2=0x7f9ec410ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.190+0000 7f9eca039640 1 --2- 172.21.15.42:0/3191012025 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec4108720 0x7f9ec410ef50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc009900 tx=0x7f9ebc02a410 comp rx=0 tx=0).stop 2024-07-26T06:48:26.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.190+0000 7f9eca039640 1 -- 172.21.15.42:0/3191012025 shutdown_connections 2024-07-26T06:48:26.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.190+0000 7f9eca039640 1 --2- 172.21.15.42:0/3191012025 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec4108720 0x7f9ec410ef50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.190+0000 7f9eca039640 1 --2- 172.21.15.42:0/3191012025 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ec4107dd0 0x7f9ec41081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.190+0000 7f9eca039640 1 --2- 172.21.15.42:0/3191012025 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ec4102490 0x7f9ec41028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.190+0000 7f9eca039640 1 -- 172.21.15.42:0/3191012025 >> 172.21.15.42:0/3191012025 conn(0x7f9ec4079160 msgr2=0x7f9ec4079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.190+0000 7f9eca039640 1 -- 172.21.15.42:0/3191012025 shutdown_connections 2024-07-26T06:48:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.190+0000 7f9eca039640 1 -- 172.21.15.42:0/3191012025 wait complete. 2024-07-26T06:48:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.191+0000 7f9eca039640 1 Processor -- start 2024-07-26T06:48:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.191+0000 7f9eca039640 1 -- start start 2024-07-26T06:48:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9eca039640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ec4102490 0x7f9ec4199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9eca039640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ec4107dd0 0x7f9ec419a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9eca039640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec4108720 0x7f9ec419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9eca039640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec41134a0 con 0x7f9ec4108720 2024-07-26T06:48:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9eca039640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec4113320 con 0x7f9ec4107dd0 2024-07-26T06:48:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9eca039640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec4113620 con 0x7f9ec4102490 2024-07-26T06:48:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9ec3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec4108720 0x7f9ec419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9ec3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec4108720 0x7f9ec419f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43740/0 (socket says 172.21.15.42:43740) 2024-07-26T06:48:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9ec3fff640 1 -- 172.21.15.42:0/4175289310 learned_addr learned my addr 172.21.15.42:0/4175289310 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9ec2ffd640 1 --2- 172.21.15.42:0/4175289310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ec4107dd0 0x7f9ec419a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9ec37fe640 1 --2- 172.21.15.42:0/4175289310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ec4102490 0x7f9ec4199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9ec3fff640 1 -- 172.21.15.42:0/4175289310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ec4102490 msgr2=0x7f9ec4199e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9ec3fff640 1 --2- 172.21.15.42:0/4175289310 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ec4102490 0x7f9ec4199e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9ec3fff640 1 -- 172.21.15.42:0/4175289310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ec4107dd0 msgr2=0x7f9ec419a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9ec3fff640 1 --2- 172.21.15.42:0/4175289310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9ec4107dd0 0x7f9ec419a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9ec3fff640 1 -- 172.21.15.42:0/4175289310 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ec419fda0 con 0x7f9ec4108720 2024-07-26T06:48:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9ec3fff640 1 --2- 172.21.15.42:0/4175289310 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec4108720 0x7f9ec419f670 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc004770 tx=0x7f9ebc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9ec0ff9640 1 -- 172.21.15.42:0/4175289310 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc03f030 con 0x7f9ec4108720 2024-07-26T06:48:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9eca039640 1 -- 172.21.15.42:0/4175289310 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ec41a0030 con 0x7f9ec4108720 2024-07-26T06:48:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9ec0ff9640 1 -- 172.21.15.42:0/4175289310 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ebc003aa0 con 0x7f9ec4108720 2024-07-26T06:48:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.192+0000 7f9ec0ff9640 1 -- 172.21.15.42:0/4175289310 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc035070 con 0x7f9ec4108720 2024-07-26T06:48:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.193+0000 7f9eca039640 1 -- 172.21.15.42:0/4175289310 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ec41a67f0 con 0x7f9ec4108720 2024-07-26T06:48:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.193+0000 7f9eca039640 1 -- 172.21.15.42:0/4175289310 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9ec4063540 con 0x7f9ec4108720 2024-07-26T06:48:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.195+0000 7f9ec0ff9640 1 -- 172.21.15.42:0/4175289310 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f9ebc042030 con 0x7f9ec4108720 2024-07-26T06:48:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.196+0000 7f9ec0ff9640 1 --2- 172.21.15.42:0/4175289310 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9eac0b3c60 0x7f9eac0b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.196+0000 7f9ec37fe640 1 --2- 172.21.15.42:0/4175289310 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9eac0b3c60 0x7f9eac0b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.196+0000 7f9ec0ff9640 1 -- 172.21.15.42:0/4175289310 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f9ebc104b40 con 0x7f9ec4108720 2024-07-26T06:48:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.196+0000 7f9ec0ff9640 1 --2- 172.21.15.42:0/4175289310 >> [v2:172.21.15.163:6824/3268512227,v1:172.21.15.163:6825/3268512227] conn(0x7f9eac0bd960 0x7f9eac0bfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.196+0000 7f9ec0ff9640 1 -- 172.21.15.42:0/4175289310 --> [v2:172.21.15.163:6824/3268512227,v1:172.21.15.163:6825/3268512227] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9ebc03ce50 con 0x7f9eac0bd960 2024-07-26T06:48:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.196+0000 7f9ec0ff9640 1 -- 172.21.15.42:0/4175289310 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f9ebc104f90 con 0x7f9ec4108720 2024-07-26T06:48:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.197+0000 7f9ec37fe640 1 --2- 172.21.15.42:0/4175289310 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9eac0b3c60 0x7f9eac0b6170 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4005fd0 tx=0x7f9eb4009680 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.197+0000 7f9ec2ffd640 1 --2- 172.21.15.42:0/4175289310 >> [v2:172.21.15.163:6824/3268512227,v1:172.21.15.163:6825/3268512227] conn(0x7f9eac0bd960 0x7f9eac0bfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:26.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.197+0000 7f9ec2ffd640 1 --2- 172.21.15.42:0/4175289310 >> [v2:172.21.15.163:6824/3268512227,v1:172.21.15.163:6825/3268512227] conn(0x7f9eac0bd960 0x7f9eac0bfe50 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:26.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.198+0000 7f9ec0ff9640 1 -- 172.21.15.42:0/4175289310 <== osd.10 v2:172.21.15.163:6824/3268512227 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9ebc03ce50 con 0x7f9eac0bd960 2024-07-26T06:48:26.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.214+0000 7f9eca039640 1 -- 172.21.15.42:0/4175289310 --> [v2:172.21.15.163:6824/3268512227,v1:172.21.15.163:6825/3268512227] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9ec4063970 con 0x7f9eac0bd960 2024-07-26T06:48:26.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.214+0000 7f9ec0ff9640 1 -- 172.21.15.42:0/4175289310 <== osd.10 v2:172.21.15.163:6824/3268512227 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9ec4063970 con 0x7f9eac0bd960 2024-07-26T06:48:26.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.215+0000 7f9eca039640 1 -- 172.21.15.42:0/4175289310 >> [v2:172.21.15.163:6824/3268512227,v1:172.21.15.163:6825/3268512227] conn(0x7f9eac0bd960 msgr2=0x7f9eac0bfe50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.215+0000 7f9eca039640 1 --2- 172.21.15.42:0/4175289310 >> [v2:172.21.15.163:6824/3268512227,v1:172.21.15.163:6825/3268512227] conn(0x7f9eac0bd960 0x7f9eac0bfe50 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.215+0000 7f9eca039640 1 -- 172.21.15.42:0/4175289310 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9eac0b3c60 msgr2=0x7f9eac0b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.215+0000 7f9eca039640 1 --2- 172.21.15.42:0/4175289310 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9eac0b3c60 0x7f9eac0b6170 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4005fd0 tx=0x7f9eb4009680 comp rx=0 tx=0).stop 2024-07-26T06:48:26.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.215+0000 7f9eca039640 1 -- 172.21.15.42:0/4175289310 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec4108720 msgr2=0x7f9ec419f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.215+0000 7f9eca039640 1 --2- 172.21.15.42:0/4175289310 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ec4108720 0x7f9ec419f670 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc004770 tx=0x7f9ebc004270 comp rx=0 tx=0).stop 2024-07-26T06:48:26.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.215+0000 7f9ec3fff640 1 -- 172.21.15.42:0/4175289310 reap_dead start 2024-07-26T06:48:26.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.215+0000 7f9eca039640 1 -- 172.21.15.42:0/4175289310 shutdown_connections 2024-07-26T06:48:26.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.215+0000 7f9eca039640 1 -- 172.21.15.42:0/4175289310 >> 172.21.15.42:0/4175289310 conn(0x7f9ec4079160 msgr2=0x7f9ec4103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:26.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.215+0000 7f9eca039640 1 -- 172.21.15.42:0/4175289310 shutdown_connections 2024-07-26T06:48:26.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.215+0000 7f9eca039640 1 -- 172.21.15.42:0/4175289310 wait complete. 2024-07-26T06:48:26.617 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:26 smithi042 ceph-mon[32775]: pgmap v342: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:48:26.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.710+0000 7fafd7249640 1 -- 172.21.15.42:0/3632198997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafd01024b0 msgr2=0x7fafd01028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.710+0000 7fafd7249640 1 --2- 172.21.15.42:0/3632198997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafd01024b0 0x7fafd01028c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fafc0009900 tx=0x7fafc002a3f0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.710+0000 7fafd7249640 1 -- 172.21.15.42:0/3632198997 shutdown_connections 2024-07-26T06:48:26.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.710+0000 7fafd7249640 1 --2- 172.21.15.42:0/3632198997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafd0108740 0x7fafd010ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.710+0000 7fafd7249640 1 --2- 172.21.15.42:0/3632198997 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd0107df0 0x7fafd0108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.710+0000 7fafd7249640 1 --2- 172.21.15.42:0/3632198997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafd01024b0 0x7fafd01028c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.710+0000 7fafd7249640 1 -- 172.21.15.42:0/3632198997 >> 172.21.15.42:0/3632198997 conn(0x7fafd0079170 msgr2=0x7fafd0079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:26.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.710+0000 7fafd7249640 1 -- 172.21.15.42:0/3632198997 shutdown_connections 2024-07-26T06:48:26.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.710+0000 7fafd7249640 1 -- 172.21.15.42:0/3632198997 wait complete. 2024-07-26T06:48:26.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.711+0000 7fafd7249640 1 Processor -- start 2024-07-26T06:48:26.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.711+0000 7fafd7249640 1 -- start start 2024-07-26T06:48:26.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.711+0000 7fafd7249640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd01024b0 0x7fafd0199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:26.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.711+0000 7fafd7249640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafd0107df0 0x7fafd019a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:26.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.711+0000 7fafd7249640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafd0108740 0x7fafd019f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:26.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.711+0000 7fafd7249640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fafd0113520 con 0x7fafd0108740 2024-07-26T06:48:26.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.711+0000 7fafd7249640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fafd01133a0 con 0x7fafd0107df0 2024-07-26T06:48:26.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.711+0000 7fafd7249640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fafd01136a0 con 0x7fafd01024b0 2024-07-26T06:48:26.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.711+0000 7fafd57e6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafd0108740 0x7fafd019f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:26.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.711+0000 7fafd57e6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafd0108740 0x7fafd019f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43764/0 (socket says 172.21.15.42:43764) 2024-07-26T06:48:26.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.711+0000 7fafd57e6640 1 -- 172.21.15.42:0/676406620 learned_addr learned my addr 172.21.15.42:0/676406620 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:26.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.711+0000 7fafd4fe5640 1 --2- 172.21.15.42:0/676406620 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd01024b0 0x7fafd0199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:26.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.711+0000 7fafcffff640 1 --2- 172.21.15.42:0/676406620 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafd0107df0 0x7fafd019a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:26.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.712+0000 7fafd57e6640 1 -- 172.21.15.42:0/676406620 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd01024b0 msgr2=0x7fafd0199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.712+0000 7fafd57e6640 1 --2- 172.21.15.42:0/676406620 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd01024b0 0x7fafd0199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.712+0000 7fafd57e6640 1 -- 172.21.15.42:0/676406620 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafd0107df0 msgr2=0x7fafd019a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.712+0000 7fafd57e6640 1 --2- 172.21.15.42:0/676406620 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafd0107df0 0x7fafd019a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.712+0000 7fafd57e6640 1 -- 172.21.15.42:0/676406620 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd019fea0 con 0x7fafd0108740 2024-07-26T06:48:26.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.712+0000 7fafd4fe5640 1 --2- 172.21.15.42:0/676406620 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd01024b0 0x7fafd0199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:48:26.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.712+0000 7fafcffff640 1 --2- 172.21.15.42:0/676406620 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafd0107df0 0x7fafd019a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:48:26.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.712+0000 7fafd57e6640 1 --2- 172.21.15.42:0/676406620 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafd0108740 0x7fafd019f830 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fafc800c970 tx=0x7fafc800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:26.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.712+0000 7fafcdffb640 1 -- 172.21.15.42:0/676406620 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8007bb0 con 0x7fafd0108740 2024-07-26T06:48:26.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.712+0000 7fafcdffb640 1 -- 172.21.15.42:0/676406620 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafc8007d50 con 0x7fafd0108740 2024-07-26T06:48:26.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.712+0000 7fafd7249640 1 -- 172.21.15.42:0/676406620 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd01a0190 con 0x7fafd0108740 2024-07-26T06:48:26.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.712+0000 7fafcdffb640 1 -- 172.21.15.42:0/676406620 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8018770 con 0x7fafd0108740 2024-07-26T06:48:26.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.712+0000 7fafd7249640 1 -- 172.21.15.42:0/676406620 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd01a69b0 con 0x7fafd0108740 2024-07-26T06:48:26.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.713+0000 7fafd7249640 1 -- 172.21.15.42:0/676406620 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fafd0063580 con 0x7fafd0108740 2024-07-26T06:48:26.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.715+0000 7fafcdffb640 1 -- 172.21.15.42:0/676406620 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7fafc800f990 con 0x7fafd0108740 2024-07-26T06:48:26.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.715+0000 7fafcdffb640 1 --2- 172.21.15.42:0/676406620 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fafbc0b3c90 0x7fafbc0b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:26.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.715+0000 7fafd4fe5640 1 --2- 172.21.15.42:0/676406620 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fafbc0b3c90 0x7fafbc0b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:26.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.716+0000 7fafcdffb640 1 -- 172.21.15.42:0/676406620 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fafc80e4190 con 0x7fafd0108740 2024-07-26T06:48:26.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.716+0000 7fafcdffb640 1 --2- 172.21.15.42:0/676406620 >> [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064] conn(0x7fafbc0bd990 0x7fafbc0bfe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:26.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.716+0000 7fafcdffb640 1 -- 172.21.15.42:0/676406620 --> [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fafc800eec0 con 0x7fafbc0bd990 2024-07-26T06:48:26.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.716+0000 7fafcdffb640 1 -- 172.21.15.42:0/676406620 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7fafc80e45e0 con 0x7fafd0108740 2024-07-26T06:48:26.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.716+0000 7fafcffff640 1 --2- 172.21.15.42:0/676406620 >> [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064] conn(0x7fafbc0bd990 0x7fafbc0bfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:26.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.716+0000 7fafd4fe5640 1 --2- 172.21.15.42:0/676406620 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fafbc0b3c90 0x7fafbc0b61a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fafc0031440 tx=0x7fafc0005d50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:26.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.716+0000 7fafcffff640 1 --2- 172.21.15.42:0/676406620 >> [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064] conn(0x7fafbc0bd990 0x7fafbc0bfe80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:26.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.717+0000 7fafcdffb640 1 -- 172.21.15.42:0/676406620 <== osd.3 v2:172.21.15.42:6810/2819018064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fafc800eec0 con 0x7fafbc0bd990 2024-07-26T06:48:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:26 smithi164 ceph-mon[36970]: pgmap v342: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:48:26.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.731+0000 7fafd7249640 1 -- 172.21.15.42:0/676406620 --> [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fafd0063a00 con 0x7fafbc0bd990 2024-07-26T06:48:26.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.731+0000 7fafcdffb640 1 -- 172.21.15.42:0/676406620 <== osd.3 v2:172.21.15.42:6810/2819018064 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fafd0063a00 con 0x7fafbc0bd990 2024-07-26T06:48:26.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.731+0000 7fafd7249640 1 -- 172.21.15.42:0/676406620 >> [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064] conn(0x7fafbc0bd990 msgr2=0x7fafbc0bfe80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.731+0000 7fafd7249640 1 --2- 172.21.15.42:0/676406620 >> [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064] conn(0x7fafbc0bd990 0x7fafbc0bfe80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.732+0000 7fafd7249640 1 -- 172.21.15.42:0/676406620 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fafbc0b3c90 msgr2=0x7fafbc0b61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.732+0000 7fafd7249640 1 --2- 172.21.15.42:0/676406620 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fafbc0b3c90 0x7fafbc0b61a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fafc0031440 tx=0x7fafc0005d50 comp rx=0 tx=0).stop 2024-07-26T06:48:26.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.732+0000 7fafd7249640 1 -- 172.21.15.42:0/676406620 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafd0108740 msgr2=0x7fafd019f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.732+0000 7fafd7249640 1 --2- 172.21.15.42:0/676406620 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fafd0108740 0x7fafd019f830 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fafc800c970 tx=0x7fafc800ce30 comp rx=0 tx=0).stop 2024-07-26T06:48:26.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.732+0000 7fafd57e6640 1 -- 172.21.15.42:0/676406620 reap_dead start 2024-07-26T06:48:26.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.732+0000 7fafd7249640 1 -- 172.21.15.42:0/676406620 shutdown_connections 2024-07-26T06:48:26.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.732+0000 7fafd7249640 1 -- 172.21.15.42:0/676406620 >> 172.21.15.42:0/676406620 conn(0x7fafd0079170 msgr2=0x7fafd0100fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:26.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.732+0000 7fafd7249640 1 -- 172.21.15.42:0/676406620 shutdown_connections 2024-07-26T06:48:26.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.732+0000 7fafd7249640 1 -- 172.21.15.42:0/676406620 wait complete. 2024-07-26T06:48:26.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:26 smithi163 ceph-mon[36873]: pgmap v342: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:48:26.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.901+0000 7fc132fb2640 1 -- 172.21.15.42:0/3582923692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c101860 msgr2=0x7fc12c101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.901+0000 7fc132fb2640 1 --2- 172.21.15.42:0/3582923692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c101860 0x7fc12c101c70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc11c009ca0 tx=0x7fc11c02a8c0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.901+0000 7fc132fb2640 1 -- 172.21.15.42:0/3582923692 shutdown_connections 2024-07-26T06:48:26.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.901+0000 7fc132fb2640 1 --2- 172.21.15.42:0/3582923692 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc12c1021b0 0x7fc12c10ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.901+0000 7fc132fb2640 1 --2- 172.21.15.42:0/3582923692 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c101860 0x7fc12c101c70 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.901+0000 7fc132fb2640 1 --2- 172.21.15.42:0/3582923692 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc12c108170 0x7fc12c108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.901+0000 7fc132fb2640 1 -- 172.21.15.42:0/3582923692 >> 172.21.15.42:0/3582923692 conn(0x7fc12c0799c0 msgr2=0x7fc12c079dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:26.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.901+0000 7fc132fb2640 1 -- 172.21.15.42:0/3582923692 shutdown_connections 2024-07-26T06:48:26.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.901+0000 7fc132fb2640 1 -- 172.21.15.42:0/3582923692 wait complete. 2024-07-26T06:48:26.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.902+0000 7fc132fb2640 1 Processor -- start 2024-07-26T06:48:26.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.902+0000 7fc132fb2640 1 -- start start 2024-07-26T06:48:26.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.902+0000 7fc132fb2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c101860 0x7fc12c102a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:26.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.902+0000 7fc132fb2640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc12c1021b0 0x7fc12c102fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:26.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc132fb2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc12c108170 0x7fc12c103510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:26.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc132fb2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc12c1135b0 con 0x7fc12c101860 2024-07-26T06:48:26.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc132fb2640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc12c113430 con 0x7fc12c1021b0 2024-07-26T06:48:26.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc132fb2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc12c113730 con 0x7fc12c108170 2024-07-26T06:48:26.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc130d4e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c101860 0x7fc12c102a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:26.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc130d4e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c101860 0x7fc12c102a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43440/0 (socket says 172.21.15.42:43440) 2024-07-26T06:48:26.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc130d4e640 1 -- 172.21.15.42:0/820810033 learned_addr learned my addr 172.21.15.42:0/820810033 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:26.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc12bfff640 1 --2- 172.21.15.42:0/820810033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc12c1021b0 0x7fc12c102fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:26.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc13154f640 1 --2- 172.21.15.42:0/820810033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc12c108170 0x7fc12c103510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:26.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc130d4e640 1 -- 172.21.15.42:0/820810033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc12c108170 msgr2=0x7fc12c103510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc130d4e640 1 --2- 172.21.15.42:0/820810033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc12c108170 0x7fc12c103510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc130d4e640 1 -- 172.21.15.42:0/820810033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc12c1021b0 msgr2=0x7fc12c102fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc130d4e640 1 --2- 172.21.15.42:0/820810033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc12c1021b0 0x7fc12c102fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc130d4e640 1 -- 172.21.15.42:0/820810033 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc12c1a8220 con 0x7fc12c101860 2024-07-26T06:48:26.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc13154f640 1 --2- 172.21.15.42:0/820810033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc12c108170 0x7fc12c103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:48:26.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc130d4e640 1 --2- 172.21.15.42:0/820810033 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c101860 0x7fc12c102a90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc11400d950 tx=0x7fc11400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:26.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc129ffb640 1 -- 172.21.15.42:0/820810033 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc114017070 con 0x7fc12c101860 2024-07-26T06:48:26.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc129ffb640 1 -- 172.21.15.42:0/820810033 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc114009c30 con 0x7fc12c101860 2024-07-26T06:48:26.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc129ffb640 1 -- 172.21.15.42:0/820810033 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11400f620 con 0x7fc12c101860 2024-07-26T06:48:26.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.903+0000 7fc132fb2640 1 -- 172.21.15.42:0/820810033 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc12c1a8510 con 0x7fc12c101860 2024-07-26T06:48:26.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.904+0000 7fc132fb2640 1 -- 172.21.15.42:0/820810033 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc12c1a8af0 con 0x7fc12c101860 2024-07-26T06:48:26.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.904+0000 7fc132fb2640 1 -- 172.21.15.42:0/820810033 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc12c063580 con 0x7fc12c101860 2024-07-26T06:48:26.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.906+0000 7fc129ffb640 1 -- 172.21.15.42:0/820810033 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7fc114010460 con 0x7fc12c101860 2024-07-26T06:48:26.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.907+0000 7fc129ffb640 1 --2- 172.21.15.42:0/820810033 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc1180b3ce0 0x7fc1180b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:26.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.907+0000 7fc12bfff640 1 --2- 172.21.15.42:0/820810033 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc1180b3ce0 0x7fc1180b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:26.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.907+0000 7fc129ffb640 1 -- 172.21.15.42:0/820810033 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fc1140e3d60 con 0x7fc12c101860 2024-07-26T06:48:26.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.907+0000 7fc129ffb640 1 --2- 172.21.15.42:0/820810033 >> [v2:172.21.15.164:6816/962845839,v1:172.21.15.164:6817/962845839] conn(0x7fc1180bd9e0 0x7fc1180bfed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:26.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.907+0000 7fc129ffb640 1 -- 172.21.15.42:0/820810033 --> [v2:172.21.15.164:6816/962845839,v1:172.21.15.164:6817/962845839] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc11400d130 con 0x7fc1180bd9e0 2024-07-26T06:48:26.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.907+0000 7fc129ffb640 1 -- 172.21.15.42:0/820810033 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7fc1140e41b0 con 0x7fc12c101860 2024-07-26T06:48:26.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.908+0000 7fc12bfff640 1 --2- 172.21.15.42:0/820810033 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc1180b3ce0 0x7fc1180b61f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc11c000f80 tx=0x7fc11c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:26.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.908+0000 7fc13154f640 1 --2- 172.21.15.42:0/820810033 >> [v2:172.21.15.164:6816/962845839,v1:172.21.15.164:6817/962845839] conn(0x7fc1180bd9e0 0x7fc1180bfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:26.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.908+0000 7fc13154f640 1 --2- 172.21.15.42:0/820810033 >> [v2:172.21.15.164:6816/962845839,v1:172.21.15.164:6817/962845839] conn(0x7fc1180bd9e0 0x7fc1180bfed0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:26.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.909+0000 7fc129ffb640 1 -- 172.21.15.42:0/820810033 <== osd.8 v2:172.21.15.164:6816/962845839 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc11400d130 con 0x7fc1180bd9e0 2024-07-26T06:48:26.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.923+0000 7fc132fb2640 1 -- 172.21.15.42:0/820810033 --> [v2:172.21.15.164:6816/962845839,v1:172.21.15.164:6817/962845839] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc12c063a00 con 0x7fc1180bd9e0 2024-07-26T06:48:26.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.923+0000 7fc129ffb640 1 -- 172.21.15.42:0/820810033 <== osd.8 v2:172.21.15.164:6816/962845839 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc12c063a00 con 0x7fc1180bd9e0 2024-07-26T06:48:26.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.924+0000 7fc132fb2640 1 -- 172.21.15.42:0/820810033 >> [v2:172.21.15.164:6816/962845839,v1:172.21.15.164:6817/962845839] conn(0x7fc1180bd9e0 msgr2=0x7fc1180bfed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.924+0000 7fc132fb2640 1 --2- 172.21.15.42:0/820810033 >> [v2:172.21.15.164:6816/962845839,v1:172.21.15.164:6817/962845839] conn(0x7fc1180bd9e0 0x7fc1180bfed0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.924+0000 7fc132fb2640 1 -- 172.21.15.42:0/820810033 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc1180b3ce0 msgr2=0x7fc1180b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.924+0000 7fc132fb2640 1 --2- 172.21.15.42:0/820810033 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc1180b3ce0 0x7fc1180b61f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc11c000f80 tx=0x7fc11c005fd0 comp rx=0 tx=0).stop 2024-07-26T06:48:26.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.924+0000 7fc132fb2640 1 -- 172.21.15.42:0/820810033 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c101860 msgr2=0x7fc12c102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:26.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.924+0000 7fc132fb2640 1 --2- 172.21.15.42:0/820810033 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc12c101860 0x7fc12c102a90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc11400d950 tx=0x7fc11400de10 comp rx=0 tx=0).stop 2024-07-26T06:48:26.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.924+0000 7fc13154f640 1 -- 172.21.15.42:0/820810033 reap_dead start 2024-07-26T06:48:26.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.924+0000 7fc132fb2640 1 -- 172.21.15.42:0/820810033 shutdown_connections 2024-07-26T06:48:26.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.924+0000 7fc132fb2640 1 -- 172.21.15.42:0/820810033 >> 172.21.15.42:0/820810033 conn(0x7fc12c0799c0 msgr2=0x7fc12c105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:26.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.924+0000 7fc132fb2640 1 -- 172.21.15.42:0/820810033 shutdown_connections 2024-07-26T06:48:26.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:26.924+0000 7fc132fb2640 1 -- 172.21.15.42:0/820810033 wait complete. 2024-07-26T06:48:27.509 INFO:teuthology.orchestra.run.smithi042.stdout:77309411403 2024-07-26T06:48:27.510 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-07-26T06:48:27.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.146+0000 7f54f9f10640 1 -- 172.21.15.42:0/1752214003 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54f4107dd0 msgr2=0x7f54f41081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:27.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.146+0000 7f54f9f10640 1 --2- 172.21.15.42:0/1752214003 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54f4107dd0 0x7f54f41081e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f54e8009900 tx=0x7f54e802a3f0 comp rx=0 tx=0).stop 2024-07-26T06:48:27.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.146+0000 7f54f9f10640 1 -- 172.21.15.42:0/1752214003 shutdown_connections 2024-07-26T06:48:27.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.146+0000 7f54f9f10640 1 --2- 172.21.15.42:0/1752214003 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54f4108720 0x7f54f410ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:27.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.146+0000 7f54f9f10640 1 --2- 172.21.15.42:0/1752214003 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54f4107dd0 0x7f54f41081e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:27.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.146+0000 7f54f9f10640 1 --2- 172.21.15.42:0/1752214003 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54f4102490 0x7f54f41028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:27.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.146+0000 7f54f9f10640 1 -- 172.21.15.42:0/1752214003 >> 172.21.15.42:0/1752214003 conn(0x7f54f4079160 msgr2=0x7f54f4079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:27.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.146+0000 7f54f9f10640 1 -- 172.21.15.42:0/1752214003 shutdown_connections 2024-07-26T06:48:27.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.146+0000 7f54f9f10640 1 -- 172.21.15.42:0/1752214003 wait complete. 2024-07-26T06:48:27.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.147+0000 7f54f9f10640 1 Processor -- start 2024-07-26T06:48:27.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.147+0000 7f54f9f10640 1 -- start start 2024-07-26T06:48:27.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.147+0000 7f54f9f10640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54f4102490 0x7f54f410f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:27.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.147+0000 7f54f9f10640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54f4107dd0 0x7f54f410ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:27.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.147+0000 7f54f9f10640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54f4108720 0x7f54f4110460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:27.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.147+0000 7f54f9f10640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f54f41134e0 con 0x7f54f4102490 2024-07-26T06:48:27.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.147+0000 7f54f9f10640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f54f4113360 con 0x7f54f4107dd0 2024-07-26T06:48:27.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.147+0000 7f54f9f10640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f54f4113660 con 0x7f54f4108720 2024-07-26T06:48:27.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.147+0000 7f54f37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54f4102490 0x7f54f410f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:27.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.148+0000 7f54f37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54f4102490 0x7f54f410f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43458/0 (socket says 172.21.15.42:43458) 2024-07-26T06:48:27.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.148+0000 7f54f37fe640 1 -- 172.21.15.42:0/3774915449 learned_addr learned my addr 172.21.15.42:0/3774915449 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:27.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.148+0000 7f54f3fff640 1 --2- 172.21.15.42:0/3774915449 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54f4108720 0x7f54f4110460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:27.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.148+0000 7f54f2ffd640 1 --2- 172.21.15.42:0/3774915449 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54f4107dd0 0x7f54f410ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:27.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.148+0000 7f54f37fe640 1 -- 172.21.15.42:0/3774915449 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54f4108720 msgr2=0x7f54f4110460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:27.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.148+0000 7f54f37fe640 1 --2- 172.21.15.42:0/3774915449 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f54f4108720 0x7f54f4110460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:27.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.148+0000 7f54f37fe640 1 -- 172.21.15.42:0/3774915449 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54f4107dd0 msgr2=0x7f54f410ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:27.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.148+0000 7f54f37fe640 1 --2- 172.21.15.42:0/3774915449 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54f4107dd0 0x7f54f410ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:27.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.148+0000 7f54f37fe640 1 -- 172.21.15.42:0/3774915449 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54f41a80e0 con 0x7f54f4102490 2024-07-26T06:48:27.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.148+0000 7f54f2ffd640 1 --2- 172.21.15.42:0/3774915449 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54f4107dd0 0x7f54f410ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:48:27.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.148+0000 7f54f37fe640 1 --2- 172.21.15.42:0/3774915449 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54f4102490 0x7f54f410f9e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f54e400e9f0 tx=0x7f54e400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:27.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.148+0000 7f54f0ff9640 1 -- 172.21.15.42:0/3774915449 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e400cd80 con 0x7f54f4102490 2024-07-26T06:48:27.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.148+0000 7f54f0ff9640 1 -- 172.21.15.42:0/3774915449 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54e40108e0 con 0x7f54f4102490 2024-07-26T06:48:27.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.148+0000 7f54f9f10640 1 -- 172.21.15.42:0/3774915449 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54f41a83d0 con 0x7f54f4102490 2024-07-26T06:48:27.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.148+0000 7f54f0ff9640 1 -- 172.21.15.42:0/3774915449 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e4018980 con 0x7f54f4102490 2024-07-26T06:48:27.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.148+0000 7f54f9f10640 1 -- 172.21.15.42:0/3774915449 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54f41a8930 con 0x7f54f4102490 2024-07-26T06:48:27.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.149+0000 7f54f9f10640 1 -- 172.21.15.42:0/3774915449 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f54f4063580 con 0x7f54f4102490 2024-07-26T06:48:27.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.151+0000 7f54f0ff9640 1 -- 172.21.15.42:0/3774915449 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f54e4010430 con 0x7f54f4102490 2024-07-26T06:48:27.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.152+0000 7f54f0ff9640 1 --2- 172.21.15.42:0/3774915449 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f54dc0b3ce0 0x7f54dc0b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:27.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.152+0000 7f54f2ffd640 1 --2- 172.21.15.42:0/3774915449 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f54dc0b3ce0 0x7f54dc0b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:27.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.152+0000 7f54f0ff9640 1 -- 172.21.15.42:0/3774915449 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f54e40e4c80 con 0x7f54f4102490 2024-07-26T06:48:27.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.152+0000 7f54f0ff9640 1 --2- 172.21.15.42:0/3774915449 >> [v2:172.21.15.164:6824/1861076336,v1:172.21.15.164:6825/1861076336] conn(0x7f54dc0bd9e0 0x7f54dc0bfed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:27.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.152+0000 7f54f0ff9640 1 -- 172.21.15.42:0/3774915449 --> [v2:172.21.15.164:6824/1861076336,v1:172.21.15.164:6825/1861076336] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f54e4015e80 con 0x7f54dc0bd9e0 2024-07-26T06:48:27.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.152+0000 7f54f0ff9640 1 -- 172.21.15.42:0/3774915449 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f54e40e5030 con 0x7f54f4102490 2024-07-26T06:48:27.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.152+0000 7f54f2ffd640 1 --2- 172.21.15.42:0/3774915449 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f54dc0b3ce0 0x7f54dc0b61f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f54e8006010 tx=0x7f54e8000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:27.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.153+0000 7f54f3fff640 1 --2- 172.21.15.42:0/3774915449 >> [v2:172.21.15.164:6824/1861076336,v1:172.21.15.164:6825/1861076336] conn(0x7f54dc0bd9e0 0x7f54dc0bfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:27.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.153+0000 7f54f3fff640 1 --2- 172.21.15.42:0/3774915449 >> [v2:172.21.15.164:6824/1861076336,v1:172.21.15.164:6825/1861076336] conn(0x7f54dc0bd9e0 0x7f54dc0bfed0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:27.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.154+0000 7f54f0ff9640 1 -- 172.21.15.42:0/3774915449 <== osd.11 v2:172.21.15.164:6824/1861076336 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f54e4015e80 con 0x7f54dc0bd9e0 2024-07-26T06:48:27.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.168+0000 7f54f9f10640 1 -- 172.21.15.42:0/3774915449 --> [v2:172.21.15.164:6824/1861076336,v1:172.21.15.164:6825/1861076336] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f54f4063980 con 0x7f54dc0bd9e0 2024-07-26T06:48:27.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.169+0000 7f54f0ff9640 1 -- 172.21.15.42:0/3774915449 <== osd.11 v2:172.21.15.164:6824/1861076336 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f54f4063980 con 0x7f54dc0bd9e0 2024-07-26T06:48:27.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.169+0000 7f54f9f10640 1 -- 172.21.15.42:0/3774915449 >> [v2:172.21.15.164:6824/1861076336,v1:172.21.15.164:6825/1861076336] conn(0x7f54dc0bd9e0 msgr2=0x7f54dc0bfed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:27.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.169+0000 7f54f9f10640 1 --2- 172.21.15.42:0/3774915449 >> [v2:172.21.15.164:6824/1861076336,v1:172.21.15.164:6825/1861076336] conn(0x7f54dc0bd9e0 0x7f54dc0bfed0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:27.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.169+0000 7f54f9f10640 1 -- 172.21.15.42:0/3774915449 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f54dc0b3ce0 msgr2=0x7f54dc0b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:27.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.169+0000 7f54f9f10640 1 --2- 172.21.15.42:0/3774915449 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f54dc0b3ce0 0x7f54dc0b61f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f54e8006010 tx=0x7f54e8000bc0 comp rx=0 tx=0).stop 2024-07-26T06:48:27.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.169+0000 7f54f9f10640 1 -- 172.21.15.42:0/3774915449 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54f4102490 msgr2=0x7f54f410f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:27.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.169+0000 7f54f9f10640 1 --2- 172.21.15.42:0/3774915449 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54f4102490 0x7f54f410f9e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f54e400e9f0 tx=0x7f54e400eeb0 comp rx=0 tx=0).stop 2024-07-26T06:48:27.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.169+0000 7f54f3fff640 1 -- 172.21.15.42:0/3774915449 reap_dead start 2024-07-26T06:48:27.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.169+0000 7f54f9f10640 1 -- 172.21.15.42:0/3774915449 shutdown_connections 2024-07-26T06:48:27.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.169+0000 7f54f9f10640 1 -- 172.21.15.42:0/3774915449 >> 172.21.15.42:0/3774915449 conn(0x7f54f4079160 msgr2=0x7f54f4103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:27.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.169+0000 7f54f9f10640 1 -- 172.21.15.42:0/3774915449 shutdown_connections 2024-07-26T06:48:27.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:27.170+0000 7f54f9f10640 1 -- 172.21.15.42:0/3774915449 wait complete. 2024-07-26T06:48:28.235 INFO:teuthology.orchestra.run.smithi042.stdout:274877906981 2024-07-26T06:48:28.235 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-07-26T06:48:28.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:28 smithi042 ceph-mon[32775]: pgmap v343: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T06:48:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:28 smithi164 ceph-mon[36970]: pgmap v343: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T06:48:28.754 INFO:teuthology.orchestra.run.smithi042.stdout:300647710752 2024-07-26T06:48:28.754 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-07-26T06:48:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:28 smithi163 ceph-mon[36873]: pgmap v343: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T06:48:29.966 INFO:teuthology.orchestra.run.smithi042.stdout:107374182470 2024-07-26T06:48:29.967 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-07-26T06:48:30.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:30 smithi042 ceph-mon[32775]: pgmap v344: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:48:30.695 INFO:teuthology.orchestra.run.smithi042.stdout:227633266734 2024-07-26T06:48:30.696 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-07-26T06:48:30.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:30 smithi164 ceph-mon[36970]: pgmap v344: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:48:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:30 smithi163 ceph-mon[36873]: pgmap v344: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:48:30.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.853+0000 7f50bedbf640 1 -- 172.21.15.42:0/3273589726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50b8102490 msgr2=0x7f50b81028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:30.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.853+0000 7f50bedbf640 1 --2- 172.21.15.42:0/3273589726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50b8102490 0x7f50b81028a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f50a8009900 tx=0x7f50a802a410 comp rx=0 tx=0).stop 2024-07-26T06:48:30.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.854+0000 7f50bedbf640 1 -- 172.21.15.42:0/3273589726 shutdown_connections 2024-07-26T06:48:30.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.854+0000 7f50bedbf640 1 --2- 172.21.15.42:0/3273589726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50b8108720 0x7f50b810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:30.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.854+0000 7f50bedbf640 1 --2- 172.21.15.42:0/3273589726 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50b8107dd0 0x7f50b81081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:30.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.854+0000 7f50bedbf640 1 --2- 172.21.15.42:0/3273589726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50b8102490 0x7f50b81028a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:30.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.854+0000 7f50bedbf640 1 -- 172.21.15.42:0/3273589726 >> 172.21.15.42:0/3273589726 conn(0x7f50b8079160 msgr2=0x7f50b8079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:30.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.854+0000 7f50bedbf640 1 -- 172.21.15.42:0/3273589726 shutdown_connections 2024-07-26T06:48:30.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.854+0000 7f50bedbf640 1 -- 172.21.15.42:0/3273589726 wait complete. 2024-07-26T06:48:30.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.854+0000 7f50bedbf640 1 Processor -- start 2024-07-26T06:48:30.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.854+0000 7f50bedbf640 1 -- start start 2024-07-26T06:48:30.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50bedbf640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50b8102490 0x7f50b8103c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:30.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50bedbf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50b8107dd0 0x7f50b81041c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:30.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50bedbf640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50b8108720 0x7f50b8104700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:30.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50bedbf640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f50b81123a0 con 0x7f50b8108720 2024-07-26T06:48:30.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50bedbf640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f50b8112220 con 0x7f50b8102490 2024-07-26T06:48:30.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50bedbf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f50b8112520 con 0x7f50b8107dd0 2024-07-26T06:48:30.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50bd35c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50b8108720 0x7f50b8104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:30.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50bd35c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50b8108720 0x7f50b8104700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43490/0 (socket says 172.21.15.42:43490) 2024-07-26T06:48:30.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50bd35c640 1 -- 172.21.15.42:0/819833185 learned_addr learned my addr 172.21.15.42:0/819833185 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:30.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50b7fff640 1 --2- 172.21.15.42:0/819833185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50b8107dd0 0x7f50b81041c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:30.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50bcb5b640 1 --2- 172.21.15.42:0/819833185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50b8102490 0x7f50b8103c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:30.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50bd35c640 1 -- 172.21.15.42:0/819833185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50b8107dd0 msgr2=0x7f50b81041c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:30.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50bd35c640 1 --2- 172.21.15.42:0/819833185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50b8107dd0 0x7f50b81041c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:30.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50bd35c640 1 -- 172.21.15.42:0/819833185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50b8102490 msgr2=0x7f50b8103c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:30.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50bd35c640 1 --2- 172.21.15.42:0/819833185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50b8102490 0x7f50b8103c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:30.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50bd35c640 1 -- 172.21.15.42:0/819833185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50b81a8160 con 0x7f50b8108720 2024-07-26T06:48:30.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50bcb5b640 1 --2- 172.21.15.42:0/819833185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50b8102490 0x7f50b8103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:48:30.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50bd35c640 1 --2- 172.21.15.42:0/819833185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50b8108720 0x7f50b8104700 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f50b000cad0 tx=0x7f50b000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:30.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50b5ffb640 1 -- 172.21.15.42:0/819833185 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b000ae20 con 0x7f50b8108720 2024-07-26T06:48:30.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.855+0000 7f50b5ffb640 1 -- 172.21.15.42:0/819833185 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50b0005020 con 0x7f50b8108720 2024-07-26T06:48:30.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.856+0000 7f50bedbf640 1 -- 172.21.15.42:0/819833185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50b81a8450 con 0x7f50b8108720 2024-07-26T06:48:30.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.856+0000 7f50b5ffb640 1 -- 172.21.15.42:0/819833185 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b000f6b0 con 0x7f50b8108720 2024-07-26T06:48:30.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.856+0000 7f50bedbf640 1 -- 172.21.15.42:0/819833185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50b81a8990 con 0x7f50b8108720 2024-07-26T06:48:30.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.857+0000 7f50bedbf640 1 -- 172.21.15.42:0/819833185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f50b81060a0 con 0x7f50b8108720 2024-07-26T06:48:30.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.858+0000 7f50b5ffb640 1 -- 172.21.15.42:0/819833185 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f50b00051d0 con 0x7f50b8108720 2024-07-26T06:48:30.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.859+0000 7f50b5ffb640 1 --2- 172.21.15.42:0/819833185 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f50a00b3c60 0x7f50a00b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:30.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.859+0000 7f50bcb5b640 1 --2- 172.21.15.42:0/819833185 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f50a00b3c60 0x7f50a00b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:30.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.859+0000 7f50b5ffb640 1 -- 172.21.15.42:0/819833185 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f50b00e3e90 con 0x7f50b8108720 2024-07-26T06:48:30.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.859+0000 7f50b5ffb640 1 --2- 172.21.15.42:0/819833185 >> [v2:172.21.15.163:6808/431934374,v1:172.21.15.163:6809/431934374] conn(0x7f50a00bd960 0x7f50a00bfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:30.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.859+0000 7f50b5ffb640 1 -- 172.21.15.42:0/819833185 --> [v2:172.21.15.163:6808/431934374,v1:172.21.15.163:6809/431934374] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f50a00c0520 con 0x7f50a00bd960 2024-07-26T06:48:30.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.859+0000 7f50b5ffb640 1 -- 172.21.15.42:0/819833185 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f50b00e64b0 con 0x7f50b8108720 2024-07-26T06:48:30.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.860+0000 7f50bcb5b640 1 --2- 172.21.15.42:0/819833185 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f50a00b3c60 0x7f50a00b6170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f50a8000c00 tx=0x7f50a8031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:30.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.860+0000 7f50b7fff640 1 --2- 172.21.15.42:0/819833185 >> [v2:172.21.15.163:6808/431934374,v1:172.21.15.163:6809/431934374] conn(0x7f50a00bd960 0x7f50a00bfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:30.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.860+0000 7f50b7fff640 1 --2- 172.21.15.42:0/819833185 >> [v2:172.21.15.163:6808/431934374,v1:172.21.15.163:6809/431934374] conn(0x7f50a00bd960 0x7f50a00bfe50 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:30.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.861+0000 7f50b5ffb640 1 -- 172.21.15.42:0/819833185 <== osd.4 v2:172.21.15.163:6808/431934374 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f50a00c0520 con 0x7f50a00bd960 2024-07-26T06:48:30.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.891+0000 7f50bedbf640 1 -- 172.21.15.42:0/819833185 --> [v2:172.21.15.163:6808/431934374,v1:172.21.15.163:6809/431934374] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f50b8103710 con 0x7f50a00bd960 2024-07-26T06:48:30.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.892+0000 7f50b5ffb640 1 -- 172.21.15.42:0/819833185 <== osd.4 v2:172.21.15.163:6808/431934374 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f50b8103710 con 0x7f50a00bd960 2024-07-26T06:48:30.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.892+0000 7f50bedbf640 1 -- 172.21.15.42:0/819833185 >> [v2:172.21.15.163:6808/431934374,v1:172.21.15.163:6809/431934374] conn(0x7f50a00bd960 msgr2=0x7f50a00bfe50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:30.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.892+0000 7f50bedbf640 1 --2- 172.21.15.42:0/819833185 >> [v2:172.21.15.163:6808/431934374,v1:172.21.15.163:6809/431934374] conn(0x7f50a00bd960 0x7f50a00bfe50 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:30.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.893+0000 7f50bedbf640 1 -- 172.21.15.42:0/819833185 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f50a00b3c60 msgr2=0x7f50a00b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:30.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.893+0000 7f50bedbf640 1 --2- 172.21.15.42:0/819833185 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f50a00b3c60 0x7f50a00b6170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f50a8000c00 tx=0x7f50a8031400 comp rx=0 tx=0).stop 2024-07-26T06:48:30.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.893+0000 7f50bedbf640 1 -- 172.21.15.42:0/819833185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50b8108720 msgr2=0x7f50b8104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:30.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.893+0000 7f50bedbf640 1 --2- 172.21.15.42:0/819833185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50b8108720 0x7f50b8104700 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f50b000cad0 tx=0x7f50b000cf90 comp rx=0 tx=0).stop 2024-07-26T06:48:30.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.893+0000 7f50bd35c640 1 -- 172.21.15.42:0/819833185 reap_dead start 2024-07-26T06:48:30.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.893+0000 7f50bedbf640 1 -- 172.21.15.42:0/819833185 shutdown_connections 2024-07-26T06:48:30.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.893+0000 7f50bedbf640 1 -- 172.21.15.42:0/819833185 >> 172.21.15.42:0/819833185 conn(0x7f50b8079160 msgr2=0x7f50b81013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:30.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.893+0000 7f50bedbf640 1 -- 172.21.15.42:0/819833185 shutdown_connections 2024-07-26T06:48:30.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:30.893+0000 7f50bedbf640 1 -- 172.21.15.42:0/819833185 wait complete. 2024-07-26T06:48:31.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.264+0000 7fd5affb5640 1 -- 172.21.15.42:0/645676540 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5a8108ed0 msgr2=0x7fd5a81092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:31.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.264+0000 7fd5affb5640 1 --2- 172.21.15.42:0/645676540 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5a8108ed0 0x7fd5a81092e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd598009900 tx=0x7fd59802a3f0 comp rx=0 tx=0).stop 2024-07-26T06:48:31.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.264+0000 7fd5affb5640 1 -- 172.21.15.42:0/645676540 shutdown_connections 2024-07-26T06:48:31.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.264+0000 7fd5affb5640 1 --2- 172.21.15.42:0/645676540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd5a8109820 0x7fd5a8110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:31.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.264+0000 7fd5affb5640 1 --2- 172.21.15.42:0/645676540 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5a8108ed0 0x7fd5a81092e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:31.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.264+0000 7fd5affb5640 1 --2- 172.21.15.42:0/645676540 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5a8103590 0x7fd5a81039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:31.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.264+0000 7fd5affb5640 1 -- 172.21.15.42:0/645676540 >> 172.21.15.42:0/645676540 conn(0x7fd5a807b280 msgr2=0x7fd5a807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:31.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.264+0000 7fd5affb5640 1 -- 172.21.15.42:0/645676540 shutdown_connections 2024-07-26T06:48:31.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.264+0000 7fd5affb5640 1 -- 172.21.15.42:0/645676540 wait complete. 2024-07-26T06:48:31.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.265+0000 7fd5affb5640 1 Processor -- start 2024-07-26T06:48:31.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.265+0000 7fd5affb5640 1 -- start start 2024-07-26T06:48:31.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5affb5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5a8103590 0x7fd5a8199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:31.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5affb5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd5a8108ed0 0x7fd5a819a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:31.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5affb5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5a8109820 0x7fd5a819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:31.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5affb5640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a8113490 con 0x7fd5a8103590 2024-07-26T06:48:31.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5affb5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a8113310 con 0x7fd5a8108ed0 2024-07-26T06:48:31.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5affb5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a8113610 con 0x7fd5a8109820 2024-07-26T06:48:31.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5add51640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5a8103590 0x7fd5a8199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:31.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5add51640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5a8103590 0x7fd5a8199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43476/0 (socket says 172.21.15.42:43476) 2024-07-26T06:48:31.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5add51640 1 -- 172.21.15.42:0/3895131171 learned_addr learned my addr 172.21.15.42:0/3895131171 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:31.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5ad550640 1 --2- 172.21.15.42:0/3895131171 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd5a8108ed0 0x7fd5a819a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:31.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5ae552640 1 --2- 172.21.15.42:0/3895131171 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5a8109820 0x7fd5a819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:31.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5add51640 1 -- 172.21.15.42:0/3895131171 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5a8109820 msgr2=0x7fd5a819f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:31.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5add51640 1 --2- 172.21.15.42:0/3895131171 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5a8109820 0x7fd5a819f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:31.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5add51640 1 -- 172.21.15.42:0/3895131171 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd5a8108ed0 msgr2=0x7fd5a819a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:31.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5add51640 1 --2- 172.21.15.42:0/3895131171 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd5a8108ed0 0x7fd5a819a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:31.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5add51640 1 -- 172.21.15.42:0/3895131171 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a819fcf0 con 0x7fd5a8103590 2024-07-26T06:48:31.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5ae552640 1 --2- 172.21.15.42:0/3895131171 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd5a8109820 0x7fd5a819f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:48:31.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.266+0000 7fd5add51640 1 --2- 172.21.15.42:0/3895131171 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5a8103590 0x7fd5a8199e40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd5a400c920 tx=0x7fd5a400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:31.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.267+0000 7fd59effd640 1 -- 172.21.15.42:0/3895131171 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4007c70 con 0x7fd5a8103590 2024-07-26T06:48:31.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.267+0000 7fd59effd640 1 -- 172.21.15.42:0/3895131171 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5a4007e10 con 0x7fd5a8103590 2024-07-26T06:48:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.267+0000 7fd59effd640 1 -- 172.21.15.42:0/3895131171 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4018820 con 0x7fd5a8103590 2024-07-26T06:48:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.267+0000 7fd5affb5640 1 -- 172.21.15.42:0/3895131171 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a819ffe0 con 0x7fd5a8103590 2024-07-26T06:48:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.267+0000 7fd5affb5640 1 -- 172.21.15.42:0/3895131171 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a81a6800 con 0x7fd5a8103590 2024-07-26T06:48:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.267+0000 7fd5affb5640 1 -- 172.21.15.42:0/3895131171 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd5a8063540 con 0x7fd5a8103590 2024-07-26T06:48:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.269+0000 7fd59effd640 1 -- 172.21.15.42:0/3895131171 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7fd5a400f450 con 0x7fd5a8103590 2024-07-26T06:48:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.270+0000 7fd59effd640 1 --2- 172.21.15.42:0/3895131171 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd5940b3d10 0x7fd5940b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.270+0000 7fd5ad550640 1 --2- 172.21.15.42:0/3895131171 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd5940b3d10 0x7fd5940b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.270+0000 7fd59effd640 1 -- 172.21.15.42:0/3895131171 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fd5a40e43a0 con 0x7fd5a8103590 2024-07-26T06:48:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.270+0000 7fd59effd640 1 --2- 172.21.15.42:0/3895131171 >> [v2:172.21.15.163:6800/360219078,v1:172.21.15.163:6801/360219078] conn(0x7fd5940bda10 0x7fd5940bff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.270+0000 7fd59effd640 1 -- 172.21.15.42:0/3895131171 --> [v2:172.21.15.163:6800/360219078,v1:172.21.15.163:6801/360219078] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd5a401ee50 con 0x7fd5940bda10 2024-07-26T06:48:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.270+0000 7fd59effd640 1 -- 172.21.15.42:0/3895131171 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7fd5a40e4750 con 0x7fd5a8103590 2024-07-26T06:48:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.271+0000 7fd5ad550640 1 --2- 172.21.15.42:0/3895131171 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd5940b3d10 0x7fd5940b6220 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd598009230 tx=0x7fd598031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.271+0000 7fd5ae552640 1 --2- 172.21.15.42:0/3895131171 >> [v2:172.21.15.163:6800/360219078,v1:172.21.15.163:6801/360219078] conn(0x7fd5940bda10 0x7fd5940bff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.271+0000 7fd5ae552640 1 --2- 172.21.15.42:0/3895131171 >> [v2:172.21.15.163:6800/360219078,v1:172.21.15.163:6801/360219078] conn(0x7fd5940bda10 0x7fd5940bff00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.272+0000 7fd59effd640 1 -- 172.21.15.42:0/3895131171 <== osd.1 v2:172.21.15.163:6800/360219078 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd5a401ee50 con 0x7fd5940bda10 2024-07-26T06:48:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.286+0000 7fd5affb5640 1 -- 172.21.15.42:0/3895131171 --> [v2:172.21.15.163:6800/360219078,v1:172.21.15.163:6801/360219078] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd5a80639c0 con 0x7fd5940bda10 2024-07-26T06:48:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.287+0000 7fd59effd640 1 -- 172.21.15.42:0/3895131171 <== osd.1 v2:172.21.15.163:6800/360219078 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd5a80639c0 con 0x7fd5940bda10 2024-07-26T06:48:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.287+0000 7fd5affb5640 1 -- 172.21.15.42:0/3895131171 >> [v2:172.21.15.163:6800/360219078,v1:172.21.15.163:6801/360219078] conn(0x7fd5940bda10 msgr2=0x7fd5940bff00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.287+0000 7fd5affb5640 1 --2- 172.21.15.42:0/3895131171 >> [v2:172.21.15.163:6800/360219078,v1:172.21.15.163:6801/360219078] conn(0x7fd5940bda10 0x7fd5940bff00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.287+0000 7fd5affb5640 1 -- 172.21.15.42:0/3895131171 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd5940b3d10 msgr2=0x7fd5940b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.287+0000 7fd5affb5640 1 --2- 172.21.15.42:0/3895131171 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd5940b3d10 0x7fd5940b6220 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd598009230 tx=0x7fd598031400 comp rx=0 tx=0).stop 2024-07-26T06:48:31.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.287+0000 7fd5affb5640 1 -- 172.21.15.42:0/3895131171 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5a8103590 msgr2=0x7fd5a8199e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:31.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.287+0000 7fd5affb5640 1 --2- 172.21.15.42:0/3895131171 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5a8103590 0x7fd5a8199e40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd5a400c920 tx=0x7fd5a400cde0 comp rx=0 tx=0).stop 2024-07-26T06:48:31.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.287+0000 7fd5ae552640 1 -- 172.21.15.42:0/3895131171 reap_dead start 2024-07-26T06:48:31.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.287+0000 7fd5affb5640 1 -- 172.21.15.42:0/3895131171 shutdown_connections 2024-07-26T06:48:31.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.287+0000 7fd5affb5640 1 -- 172.21.15.42:0/3895131171 >> 172.21.15.42:0/3895131171 conn(0x7fd5a807b280 msgr2=0x7fd5a8106f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:31.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.287+0000 7fd5affb5640 1 -- 172.21.15.42:0/3895131171 shutdown_connections 2024-07-26T06:48:31.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:29.287+0000 7fd5affb5640 1 -- 172.21.15.42:0/3895131171 wait complete. 2024-07-26T06:48:32.291 INFO:teuthology.orchestra.run.smithi042.stdout:133143986243 2024-07-26T06:48:32.291 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-07-26T06:48:32.507 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:32 smithi042 ceph-mon[32775]: pgmap v345: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:48:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:32 smithi164 ceph-mon[36970]: pgmap v345: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:48:32.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:32 smithi163 ceph-mon[36873]: pgmap v345: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:48:32.849 INFO:teuthology.orchestra.run.smithi042.stdout:55834574929 2024-07-26T06:48:32.849 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-07-26T06:48:33.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.396+0000 7faa59654640 1 -- 172.21.15.42:0/1019837916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa54103590 msgr2=0x7faa541039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:33.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.396+0000 7faa59654640 1 --2- 172.21.15.42:0/1019837916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa54103590 0x7faa541039a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7faa44009900 tx=0x7faa4402a410 comp rx=0 tx=0).stop 2024-07-26T06:48:33.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.396+0000 7faa59654640 1 -- 172.21.15.42:0/1019837916 shutdown_connections 2024-07-26T06:48:33.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.396+0000 7faa59654640 1 --2- 172.21.15.42:0/1019837916 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa54109820 0x7faa54110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:33.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.396+0000 7faa59654640 1 --2- 172.21.15.42:0/1019837916 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa54108ed0 0x7faa541092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:33.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.396+0000 7faa59654640 1 --2- 172.21.15.42:0/1019837916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa54103590 0x7faa541039a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:33.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.396+0000 7faa59654640 1 -- 172.21.15.42:0/1019837916 >> 172.21.15.42:0/1019837916 conn(0x7faa5407b280 msgr2=0x7faa5407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:33.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.396+0000 7faa59654640 1 -- 172.21.15.42:0/1019837916 shutdown_connections 2024-07-26T06:48:33.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.396+0000 7faa59654640 1 -- 172.21.15.42:0/1019837916 wait complete. 2024-07-26T06:48:33.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.397+0000 7faa59654640 1 Processor -- start 2024-07-26T06:48:33.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.397+0000 7faa59654640 1 -- start start 2024-07-26T06:48:33.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa59654640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa54103590 0x7faa54075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:33.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa59654640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa54108ed0 0x7faa54075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:33.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa59654640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa54109820 0x7faa54075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:33.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa59654640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faa541134f0 con 0x7faa54103590 2024-07-26T06:48:33.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa59654640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7faa54113370 con 0x7faa54108ed0 2024-07-26T06:48:33.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa59654640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faa54113670 con 0x7faa54109820 2024-07-26T06:48:33.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa52ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa54103590 0x7faa54075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:33.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa52ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa54103590 0x7faa54075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43512/0 (socket says 172.21.15.42:43512) 2024-07-26T06:48:33.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa52ffd640 1 -- 172.21.15.42:0/2615780034 learned_addr learned my addr 172.21.15.42:0/2615780034 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:33.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa537fe640 1 --2- 172.21.15.42:0/2615780034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa54109820 0x7faa54075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:33.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa4a7fc640 1 --2- 172.21.15.42:0/2615780034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa54108ed0 0x7faa54075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:33.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa52ffd640 1 -- 172.21.15.42:0/2615780034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa54109820 msgr2=0x7faa54075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:33.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa52ffd640 1 --2- 172.21.15.42:0/2615780034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa54109820 0x7faa54075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:33.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa52ffd640 1 -- 172.21.15.42:0/2615780034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa54108ed0 msgr2=0x7faa54075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:33.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa52ffd640 1 --2- 172.21.15.42:0/2615780034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa54108ed0 0x7faa54075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:33.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa52ffd640 1 -- 172.21.15.42:0/2615780034 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa541a8160 con 0x7faa54103590 2024-07-26T06:48:33.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa4a7fc640 1 --2- 172.21.15.42:0/2615780034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa54108ed0 0x7faa54075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:48:33.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.398+0000 7faa52ffd640 1 --2- 172.21.15.42:0/2615780034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa54103590 0x7faa54075140 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faa44005940 tx=0x7faa440042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:33.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.399+0000 7faa50ff9640 1 -- 172.21.15.42:0/2615780034 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4403f030 con 0x7faa54103590 2024-07-26T06:48:33.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.399+0000 7faa50ff9640 1 -- 172.21.15.42:0/2615780034 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa44003cb0 con 0x7faa54103590 2024-07-26T06:48:33.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.399+0000 7faa59654640 1 -- 172.21.15.42:0/2615780034 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa541a83f0 con 0x7faa54103590 2024-07-26T06:48:33.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.399+0000 7faa50ff9640 1 -- 172.21.15.42:0/2615780034 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa44035070 con 0x7faa54103590 2024-07-26T06:48:33.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.399+0000 7faa59654640 1 -- 172.21.15.42:0/2615780034 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa541a88d0 con 0x7faa54103590 2024-07-26T06:48:33.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.400+0000 7faa59654640 1 -- 172.21.15.42:0/2615780034 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faa54063540 con 0x7faa54103590 2024-07-26T06:48:33.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.401+0000 7faa50ff9640 1 -- 172.21.15.42:0/2615780034 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7faa44042030 con 0x7faa54103590 2024-07-26T06:48:33.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.402+0000 7faa50ff9640 1 --2- 172.21.15.42:0/2615780034 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faa400b3c60 0x7faa400b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:33.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.402+0000 7faa4a7fc640 1 --2- 172.21.15.42:0/2615780034 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faa400b3c60 0x7faa400b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:33.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.402+0000 7faa50ff9640 1 -- 172.21.15.42:0/2615780034 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7faa44104a00 con 0x7faa54103590 2024-07-26T06:48:33.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.403+0000 7faa50ff9640 1 --2- 172.21.15.42:0/2615780034 >> [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418] conn(0x7faa400bd960 0x7faa400bfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:33.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.403+0000 7faa50ff9640 1 -- 172.21.15.42:0/2615780034 --> [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa4403ce50 con 0x7faa400bd960 2024-07-26T06:48:33.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.403+0000 7faa50ff9640 1 -- 172.21.15.42:0/2615780034 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7faa44104e50 con 0x7faa54103590 2024-07-26T06:48:33.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.403+0000 7faa537fe640 1 --2- 172.21.15.42:0/2615780034 >> [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418] conn(0x7faa400bd960 0x7faa400bfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:33.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.403+0000 7faa4a7fc640 1 --2- 172.21.15.42:0/2615780034 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faa400b3c60 0x7faa400b6170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0058c0 tx=0x7faa3c005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:33.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.403+0000 7faa537fe640 1 --2- 172.21.15.42:0/2615780034 >> [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418] conn(0x7faa400bd960 0x7faa400bfe50 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:33.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.404+0000 7faa50ff9640 1 -- 172.21.15.42:0/2615780034 <== osd.9 v2:172.21.15.42:6826/994024418 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faa4403ce50 con 0x7faa400bd960 2024-07-26T06:48:33.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.418+0000 7faa59654640 1 -- 172.21.15.42:0/2615780034 --> [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faa54063940 con 0x7faa400bd960 2024-07-26T06:48:33.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.418+0000 7faa50ff9640 1 -- 172.21.15.42:0/2615780034 <== osd.9 v2:172.21.15.42:6826/994024418 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7faa54063940 con 0x7faa400bd960 2024-07-26T06:48:33.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.419+0000 7faa59654640 1 -- 172.21.15.42:0/2615780034 >> [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418] conn(0x7faa400bd960 msgr2=0x7faa400bfe50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:33.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.419+0000 7faa59654640 1 --2- 172.21.15.42:0/2615780034 >> [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418] conn(0x7faa400bd960 0x7faa400bfe50 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:33.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.419+0000 7faa59654640 1 -- 172.21.15.42:0/2615780034 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faa400b3c60 msgr2=0x7faa400b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:33.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.419+0000 7faa59654640 1 --2- 172.21.15.42:0/2615780034 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faa400b3c60 0x7faa400b6170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0058c0 tx=0x7faa3c005850 comp rx=0 tx=0).stop 2024-07-26T06:48:33.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.419+0000 7faa59654640 1 -- 172.21.15.42:0/2615780034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa54103590 msgr2=0x7faa54075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:33.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.419+0000 7faa59654640 1 --2- 172.21.15.42:0/2615780034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa54103590 0x7faa54075140 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faa44005940 tx=0x7faa440042f0 comp rx=0 tx=0).stop 2024-07-26T06:48:33.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.419+0000 7faa537fe640 1 -- 172.21.15.42:0/2615780034 reap_dead start 2024-07-26T06:48:33.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.419+0000 7faa59654640 1 -- 172.21.15.42:0/2615780034 shutdown_connections 2024-07-26T06:48:33.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.419+0000 7faa59654640 1 -- 172.21.15.42:0/2615780034 >> 172.21.15.42:0/2615780034 conn(0x7faa5407b280 msgr2=0x7faa54106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:33.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.419+0000 7faa59654640 1 -- 172.21.15.42:0/2615780034 shutdown_connections 2024-07-26T06:48:33.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.419+0000 7faa59654640 1 -- 172.21.15.42:0/2615780034 wait complete. 2024-07-26T06:48:33.641 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:33.641 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:33.642 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:33.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.950+0000 7f04e48bc640 1 -- 172.21.15.42:0/555994635 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04e0108740 msgr2=0x7f04e010ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:33.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.950+0000 7f04e48bc640 1 --2- 172.21.15.42:0/555994635 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04e0108740 0x7f04e010ef70 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f04d8009ca0 tx=0x7f04d802a8c0 comp rx=0 tx=0).stop 2024-07-26T06:48:33.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.950+0000 7f04e48bc640 1 -- 172.21.15.42:0/555994635 shutdown_connections 2024-07-26T06:48:33.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.950+0000 7f04e48bc640 1 --2- 172.21.15.42:0/555994635 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04e0108740 0x7f04e010ef70 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:33.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.950+0000 7f04e48bc640 1 --2- 172.21.15.42:0/555994635 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04e0107df0 0x7f04e0108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:33.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.950+0000 7f04e48bc640 1 --2- 172.21.15.42:0/555994635 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04e01024b0 0x7f04e01028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:33.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.950+0000 7f04e48bc640 1 -- 172.21.15.42:0/555994635 >> 172.21.15.42:0/555994635 conn(0x7f04e0079170 msgr2=0x7f04e0079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:33.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.950+0000 7f04e48bc640 1 -- 172.21.15.42:0/555994635 shutdown_connections 2024-07-26T06:48:33.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.950+0000 7f04e48bc640 1 -- 172.21.15.42:0/555994635 wait complete. 2024-07-26T06:48:33.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.951+0000 7f04e48bc640 1 Processor -- start 2024-07-26T06:48:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.951+0000 7f04e48bc640 1 -- start start 2024-07-26T06:48:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04e48bc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04e01024b0 0x7f04e0199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04e48bc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04e0107df0 0x7f04e019a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04e48bc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04e0108740 0x7f04e019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04e48bc640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f04e01134d0 con 0x7f04e01024b0 2024-07-26T06:48:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04e48bc640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04e0113350 con 0x7f04e0107df0 2024-07-26T06:48:33.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04e48bc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f04e0113650 con 0x7f04e0108740 2024-07-26T06:48:33.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04de59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04e01024b0 0x7f04e0199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:33.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04de59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04e01024b0 0x7f04e0199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43530/0 (socket says 172.21.15.42:43530) 2024-07-26T06:48:33.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04de59c640 1 -- 172.21.15.42:0/1684170428 learned_addr learned my addr 172.21.15.42:0/1684170428 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:33.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04ded9d640 1 --2- 172.21.15.42:0/1684170428 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04e0108740 0x7f04e019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:33.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04ddd9b640 1 --2- 172.21.15.42:0/1684170428 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04e0107df0 0x7f04e019a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:33.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04de59c640 1 -- 172.21.15.42:0/1684170428 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04e0108740 msgr2=0x7f04e019f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:33.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04de59c640 1 --2- 172.21.15.42:0/1684170428 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04e0108740 0x7f04e019f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:33.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04de59c640 1 -- 172.21.15.42:0/1684170428 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04e0107df0 msgr2=0x7f04e019a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:33.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04de59c640 1 --2- 172.21.15.42:0/1684170428 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04e0107df0 0x7f04e019a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:33.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04de59c640 1 -- 172.21.15.42:0/1684170428 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04e019feb0 con 0x7f04e01024b0 2024-07-26T06:48:33.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04ddd9b640 1 --2- 172.21.15.42:0/1684170428 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04e0107df0 0x7f04e019a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:48:33.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04de59c640 1 --2- 172.21.15.42:0/1684170428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04e01024b0 0x7f04e0199f40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f04d000c890 tx=0x7f04d000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:33.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04cf7fe640 1 -- 172.21.15.42:0/1684170428 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d0004220 con 0x7f04e01024b0 2024-07-26T06:48:33.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04cf7fe640 1 -- 172.21.15.42:0/1684170428 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04d00043c0 con 0x7f04e01024b0 2024-07-26T06:48:33.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04e48bc640 1 -- 172.21.15.42:0/1684170428 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04e01a01a0 con 0x7f04e01024b0 2024-07-26T06:48:33.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.952+0000 7f04cf7fe640 1 -- 172.21.15.42:0/1684170428 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d0004a00 con 0x7f04e01024b0 2024-07-26T06:48:33.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.953+0000 7f04e48bc640 1 -- 172.21.15.42:0/1684170428 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04e01a6900 con 0x7f04e01024b0 2024-07-26T06:48:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.953+0000 7f04e48bc640 1 -- 172.21.15.42:0/1684170428 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f04e0063580 con 0x7f04e01024b0 2024-07-26T06:48:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.955+0000 7f04cf7fe640 1 -- 172.21.15.42:0/1684170428 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f04d00079b0 con 0x7f04e01024b0 2024-07-26T06:48:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.956+0000 7f04cf7fe640 1 --2- 172.21.15.42:0/1684170428 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f04c80b3c60 0x7f04c80b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.956+0000 7f04ddd9b640 1 --2- 172.21.15.42:0/1684170428 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f04c80b3c60 0x7f04c80b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.956+0000 7f04cf7fe640 1 -- 172.21.15.42:0/1684170428 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f04d00e3fa0 con 0x7f04e01024b0 2024-07-26T06:48:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.956+0000 7f04cf7fe640 1 --2- 172.21.15.42:0/1684170428 >> [v2:172.21.15.163:6816/812214111,v1:172.21.15.163:6817/812214111] conn(0x7f04c80bd960 0x7f04c80bfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.956+0000 7f04cf7fe640 1 -- 172.21.15.42:0/1684170428 --> [v2:172.21.15.163:6816/812214111,v1:172.21.15.163:6817/812214111] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f04d0012ec0 con 0x7f04c80bd960 2024-07-26T06:48:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.956+0000 7f04cf7fe640 1 -- 172.21.15.42:0/1684170428 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f04d00e4350 con 0x7f04e01024b0 2024-07-26T06:48:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.957+0000 7f04ddd9b640 1 --2- 172.21.15.42:0/1684170428 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f04c80b3c60 0x7f04c80b6170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f04e0107590 tx=0x7f04d4005cb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.957+0000 7f04ded9d640 1 --2- 172.21.15.42:0/1684170428 >> [v2:172.21.15.163:6816/812214111,v1:172.21.15.163:6817/812214111] conn(0x7f04c80bd960 0x7f04c80bfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.957+0000 7f04ded9d640 1 --2- 172.21.15.42:0/1684170428 >> [v2:172.21.15.163:6816/812214111,v1:172.21.15.163:6817/812214111] conn(0x7f04c80bd960 0x7f04c80bfe50 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.958+0000 7f04cf7fe640 1 -- 172.21.15.42:0/1684170428 <== osd.7 v2:172.21.15.163:6816/812214111 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f04d0012ec0 con 0x7f04c80bd960 2024-07-26T06:48:33.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.972+0000 7f04e48bc640 1 -- 172.21.15.42:0/1684170428 --> [v2:172.21.15.163:6816/812214111,v1:172.21.15.163:6817/812214111] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f04e0063a00 con 0x7f04c80bd960 2024-07-26T06:48:33.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.973+0000 7f04cf7fe640 1 -- 172.21.15.42:0/1684170428 <== osd.7 v2:172.21.15.163:6816/812214111 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f04e0063a00 con 0x7f04c80bd960 2024-07-26T06:48:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.973+0000 7f04e48bc640 1 -- 172.21.15.42:0/1684170428 >> [v2:172.21.15.163:6816/812214111,v1:172.21.15.163:6817/812214111] conn(0x7f04c80bd960 msgr2=0x7f04c80bfe50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.973+0000 7f04e48bc640 1 --2- 172.21.15.42:0/1684170428 >> [v2:172.21.15.163:6816/812214111,v1:172.21.15.163:6817/812214111] conn(0x7f04c80bd960 0x7f04c80bfe50 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.973+0000 7f04e48bc640 1 -- 172.21.15.42:0/1684170428 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f04c80b3c60 msgr2=0x7f04c80b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.973+0000 7f04e48bc640 1 --2- 172.21.15.42:0/1684170428 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f04c80b3c60 0x7f04c80b6170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f04e0107590 tx=0x7f04d4005cb0 comp rx=0 tx=0).stop 2024-07-26T06:48:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.973+0000 7f04e48bc640 1 -- 172.21.15.42:0/1684170428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04e01024b0 msgr2=0x7f04e0199f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.973+0000 7f04e48bc640 1 --2- 172.21.15.42:0/1684170428 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04e01024b0 0x7f04e0199f40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f04d000c890 tx=0x7f04d000cd50 comp rx=0 tx=0).stop 2024-07-26T06:48:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.973+0000 7f04ded9d640 1 -- 172.21.15.42:0/1684170428 reap_dead start 2024-07-26T06:48:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.973+0000 7f04e48bc640 1 -- 172.21.15.42:0/1684170428 shutdown_connections 2024-07-26T06:48:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.973+0000 7f04e48bc640 1 -- 172.21.15.42:0/1684170428 >> 172.21.15.42:0/1684170428 conn(0x7f04e0079170 msgr2=0x7f04e0104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.974+0000 7f04e48bc640 1 -- 172.21.15.42:0/1684170428 shutdown_connections 2024-07-26T06:48:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:33.974+0000 7f04e48bc640 1 -- 172.21.15.42:0/1684170428 wait complete. 2024-07-26T06:48:34.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:34 smithi042 ceph-mon[32775]: pgmap v346: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:34.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:34 smithi164 ceph-mon[36970]: pgmap v346: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:34.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:34 smithi163 ceph-mon[36873]: pgmap v346: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:34.911 INFO:teuthology.orchestra.run.smithi042.stdout:253403070506 2024-07-26T06:48:34.911 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-07-26T06:48:35.674 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:36.603 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:36 smithi042 ceph-mon[32775]: pgmap v347: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:48:36.637 INFO:teuthology.orchestra.run.smithi042.stdout:201863462964 2024-07-26T06:48:36.638 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-07-26T06:48:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:36 smithi164 ceph-mon[36970]: pgmap v347: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:48:36.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:36 smithi163 ceph-mon[36873]: pgmap v347: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:48:37.999 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:38.000 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:38.442 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:38.442 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:38.447 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:38.688 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:38 smithi042 ceph-mon[32775]: pgmap v348: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:38 smithi164 ceph-mon[36970]: pgmap v348: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:38.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:38 smithi163 ceph-mon[36873]: pgmap v348: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:40.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.475+0000 7f16c6962640 1 -- 172.21.15.42:0/937279677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c0103590 msgr2=0x7f16c01039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:40.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.475+0000 7f16c6962640 1 --2- 172.21.15.42:0/937279677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c0103590 0x7f16c01039a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009900 tx=0x7f16b002a410 comp rx=0 tx=0).stop 2024-07-26T06:48:40.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.475+0000 7f16c6962640 1 -- 172.21.15.42:0/937279677 shutdown_connections 2024-07-26T06:48:40.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.475+0000 7f16c6962640 1 --2- 172.21.15.42:0/937279677 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16c0109820 0x7f16c0110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:40.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.475+0000 7f16c6962640 1 --2- 172.21.15.42:0/937279677 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16c0108ed0 0x7f16c01092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:40.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.475+0000 7f16c6962640 1 --2- 172.21.15.42:0/937279677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c0103590 0x7f16c01039a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:40.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.475+0000 7f16c6962640 1 -- 172.21.15.42:0/937279677 >> 172.21.15.42:0/937279677 conn(0x7f16c007b280 msgr2=0x7f16c007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:40.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.475+0000 7f16c6962640 1 -- 172.21.15.42:0/937279677 shutdown_connections 2024-07-26T06:48:40.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.475+0000 7f16c6962640 1 -- 172.21.15.42:0/937279677 wait complete. 2024-07-26T06:48:40.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.476+0000 7f16c6962640 1 Processor -- start 2024-07-26T06:48:40.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.476+0000 7f16c6962640 1 -- start start 2024-07-26T06:48:40.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.476+0000 7f16c6962640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16c0103590 0x7f16c0199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:40.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.476+0000 7f16c6962640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16c0108ed0 0x7f16c019a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:40.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.476+0000 7f16c6962640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c0109820 0x7f16c019f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:40.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.477+0000 7f16c6962640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f16c0113520 con 0x7f16c0109820 2024-07-26T06:48:40.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.477+0000 7f16c6962640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f16c01133a0 con 0x7f16c0108ed0 2024-07-26T06:48:40.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.477+0000 7f16c6962640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f16c01136a0 con 0x7f16c0103590 2024-07-26T06:48:40.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.477+0000 7f16c4eff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c0109820 0x7f16c019f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:40.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.477+0000 7f16c4eff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c0109820 0x7f16c019f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59546/0 (socket says 172.21.15.42:59546) 2024-07-26T06:48:40.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.477+0000 7f16c4eff640 1 -- 172.21.15.42:0/3898178960 learned_addr learned my addr 172.21.15.42:0/3898178960 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:40.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.477+0000 7f16bffff640 1 --2- 172.21.15.42:0/3898178960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16c0103590 0x7f16c0199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:40.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.477+0000 7f16b7fff640 1 --2- 172.21.15.42:0/3898178960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16c0108ed0 0x7f16c019a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:40.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.477+0000 7f16c4eff640 1 -- 172.21.15.42:0/3898178960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16c0103590 msgr2=0x7f16c0199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:40.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.477+0000 7f16c4eff640 1 --2- 172.21.15.42:0/3898178960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16c0103590 0x7f16c0199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:40.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.477+0000 7f16c4eff640 1 -- 172.21.15.42:0/3898178960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16c0108ed0 msgr2=0x7f16c019a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:40.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.477+0000 7f16c4eff640 1 --2- 172.21.15.42:0/3898178960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16c0108ed0 0x7f16c019a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.477+0000 7f16c4eff640 1 -- 172.21.15.42:0/3898178960 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c019fe40 con 0x7f16c0109820 2024-07-26T06:48:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.477+0000 7f16c4eff640 1 --2- 172.21.15.42:0/3898178960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c0109820 0x7f16c019f710 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f16b800eaa0 tx=0x7f16b800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.477+0000 7f16bdffb640 1 -- 172.21.15.42:0/3898178960 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b8004d20 con 0x7f16c0109820 2024-07-26T06:48:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.478+0000 7f16c6962640 1 -- 172.21.15.42:0/3898178960 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c01a0130 con 0x7f16c0109820 2024-07-26T06:48:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.478+0000 7f16bdffb640 1 -- 172.21.15.42:0/3898178960 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16b8005020 con 0x7f16c0109820 2024-07-26T06:48:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.478+0000 7f16bdffb640 1 -- 172.21.15.42:0/3898178960 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b80116b0 con 0x7f16c0109820 2024-07-26T06:48:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.478+0000 7f16c6962640 1 -- 172.21.15.42:0/3898178960 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c01a6890 con 0x7f16c0109820 2024-07-26T06:48:40.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.479+0000 7f16c6962640 1 -- 172.21.15.42:0/3898178960 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16c01039a0 con 0x7f16c0109820 2024-07-26T06:48:40.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.481+0000 7f16bdffb640 1 -- 172.21.15.42:0/3898178960 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f16b80051d0 con 0x7f16c0109820 2024-07-26T06:48:40.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.482+0000 7f16bdffb640 1 --2- 172.21.15.42:0/3898178960 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f16ac0b3c60 0x7f16ac0b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:40.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.483+0000 7f16bffff640 1 --2- 172.21.15.42:0/3898178960 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f16ac0b3c60 0x7f16ac0b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:40.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.483+0000 7f16bdffb640 1 -- 172.21.15.42:0/3898178960 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f16b80e4380 con 0x7f16c0109820 2024-07-26T06:48:40.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.483+0000 7f16bffff640 1 --2- 172.21.15.42:0/3898178960 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f16ac0b3c60 0x7f16ac0b6170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f16b0031440 tx=0x7f16b0009560 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:40.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.486+0000 7f16bdffb640 1 -- 172.21.15.42:0/3898178960 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f16b8016030 con 0x7f16c0109820 2024-07-26T06:48:40.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:40 smithi042 ceph-mon[32775]: pgmap v349: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T06:48:40.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.678+0000 7f16c6962640 1 -- 172.21.15.42:0/3898178960 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f16c019b000 con 0x7f16c0109820 2024-07-26T06:48:40.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.678+0000 7f16bdffb640 1 -- 172.21.15.42:0/3898178960 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f16c019b000 con 0x7f16c0109820 2024-07-26T06:48:40.679 INFO:teuthology.orchestra.run.smithi042.stdout:180388626491 2024-07-26T06:48:40.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.681+0000 7f16c6962640 1 -- 172.21.15.42:0/3898178960 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f16ac0b3c60 msgr2=0x7f16ac0b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:40.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.681+0000 7f16c6962640 1 --2- 172.21.15.42:0/3898178960 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f16ac0b3c60 0x7f16ac0b6170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f16b0031440 tx=0x7f16b0009560 comp rx=0 tx=0).stop 2024-07-26T06:48:40.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.681+0000 7f16c6962640 1 -- 172.21.15.42:0/3898178960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c0109820 msgr2=0x7f16c019f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:40.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.681+0000 7f16c6962640 1 --2- 172.21.15.42:0/3898178960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c0109820 0x7f16c019f710 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f16b800eaa0 tx=0x7f16b800eea0 comp rx=0 tx=0).stop 2024-07-26T06:48:40.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.681+0000 7f16c6962640 1 -- 172.21.15.42:0/3898178960 shutdown_connections 2024-07-26T06:48:40.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.681+0000 7f16c6962640 1 --2- 172.21.15.42:0/3898178960 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f16ac0b3c60 0x7f16ac0b6170 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:40.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.681+0000 7f16c6962640 1 --2- 172.21.15.42:0/3898178960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16c0109820 0x7f16c019f710 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:40.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.681+0000 7f16c6962640 1 --2- 172.21.15.42:0/3898178960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f16c0108ed0 0x7f16c019a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:40.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.681+0000 7f16c6962640 1 --2- 172.21.15.42:0/3898178960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16c0103590 0x7f16c0199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:40.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.681+0000 7f16c6962640 1 -- 172.21.15.42:0/3898178960 >> 172.21.15.42:0/3898178960 conn(0x7f16c007b280 msgr2=0x7f16c0106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:40.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.681+0000 7f16c6962640 1 -- 172.21.15.42:0/3898178960 shutdown_connections 2024-07-26T06:48:40.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:40.681+0000 7f16c6962640 1 -- 172.21.15.42:0/3898178960 wait complete. 2024-07-26T06:48:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:40 smithi164 ceph-mon[36970]: pgmap v349: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T06:48:40.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:40 smithi163 ceph-mon[36873]: pgmap v349: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T06:48:41.486 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626486 got 180388626491 for osd.6 2024-07-26T06:48:41.486 DEBUG:teuthology.parallel:result is None 2024-07-26T06:48:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:41 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3898178960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T06:48:41.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:41 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3898178960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T06:48:41.876 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:41 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3898178960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T06:48:42.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.089+0000 7f65b6a34640 1 -- 172.21.15.42:0/279765657 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65b01024a0 msgr2=0x7f65b01028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:42.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.089+0000 7f65b6a34640 1 --2- 172.21.15.42:0/279765657 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65b01024a0 0x7f65b01028b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f65a0009900 tx=0x7f65a002a410 comp rx=0 tx=0).stop 2024-07-26T06:48:42.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.089+0000 7f65b6a34640 1 -- 172.21.15.42:0/279765657 shutdown_connections 2024-07-26T06:48:42.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.089+0000 7f65b6a34640 1 --2- 172.21.15.42:0/279765657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65b0108730 0x7f65b010ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:42.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.089+0000 7f65b6a34640 1 --2- 172.21.15.42:0/279765657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65b0107de0 0x7f65b01081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:42.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.089+0000 7f65b6a34640 1 --2- 172.21.15.42:0/279765657 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65b01024a0 0x7f65b01028b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.089+0000 7f65b6a34640 1 -- 172.21.15.42:0/279765657 >> 172.21.15.42:0/279765657 conn(0x7f65b0079160 msgr2=0x7f65b0079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.089+0000 7f65b6a34640 1 -- 172.21.15.42:0/279765657 shutdown_connections 2024-07-26T06:48:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.089+0000 7f65b6a34640 1 -- 172.21.15.42:0/279765657 wait complete. 2024-07-26T06:48:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.090+0000 7f65b6a34640 1 Processor -- start 2024-07-26T06:48:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.090+0000 7f65b6a34640 1 -- start start 2024-07-26T06:48:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.090+0000 7f65b6a34640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65b01024a0 0x7f65b0103c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.090+0000 7f65b6a34640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65b0107de0 0x7f65b01041d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.090+0000 7f65b6a34640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65b0108730 0x7f65b0104710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.090+0000 7f65b6a34640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f65b01123b0 con 0x7f65b0107de0 2024-07-26T06:48:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.090+0000 7f65b6a34640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f65b0112230 con 0x7f65b0108730 2024-07-26T06:48:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.090+0000 7f65b6a34640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f65b0112530 con 0x7f65b01024a0 2024-07-26T06:48:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.090+0000 7f65af7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65b0107de0 0x7f65b01041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.090+0000 7f65af7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65b0107de0 0x7f65b01041d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59576/0 (socket says 172.21.15.42:59576) 2024-07-26T06:48:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.090+0000 7f65af7fe640 1 -- 172.21.15.42:0/3523988095 learned_addr learned my addr 172.21.15.42:0/3523988095 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.090+0000 7f65b4fd1640 1 --2- 172.21.15.42:0/3523988095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65b0108730 0x7f65b0104710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.090+0000 7f65af7fe640 1 -- 172.21.15.42:0/3523988095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65b01024a0 msgr2=0x7f65b0103c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.090+0000 7f65affff640 1 --2- 172.21.15.42:0/3523988095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65b01024a0 0x7f65b0103c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.090+0000 7f65af7fe640 1 --2- 172.21.15.42:0/3523988095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65b01024a0 0x7f65b0103c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.091+0000 7f65af7fe640 1 -- 172.21.15.42:0/3523988095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65b0108730 msgr2=0x7f65b0104710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.091+0000 7f65af7fe640 1 --2- 172.21.15.42:0/3523988095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65b0108730 0x7f65b0104710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.091+0000 7f65af7fe640 1 -- 172.21.15.42:0/3523988095 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b01a8210 con 0x7f65b0107de0 2024-07-26T06:48:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.091+0000 7f65b4fd1640 1 --2- 172.21.15.42:0/3523988095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65b0108730 0x7f65b0104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:48:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.091+0000 7f65af7fe640 1 --2- 172.21.15.42:0/3523988095 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65b0107de0 0x7f65b01041d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f65a400d770 tx=0x7f65a400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.091+0000 7f65ad7fa640 1 -- 172.21.15.42:0/3523988095 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a4017070 con 0x7f65b0107de0 2024-07-26T06:48:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.091+0000 7f65ad7fa640 1 -- 172.21.15.42:0/3523988095 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65a4009c30 con 0x7f65b0107de0 2024-07-26T06:48:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.091+0000 7f65b6a34640 1 -- 172.21.15.42:0/3523988095 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65b01a8500 con 0x7f65b0107de0 2024-07-26T06:48:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.091+0000 7f65ad7fa640 1 -- 172.21.15.42:0/3523988095 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a400f620 con 0x7f65b0107de0 2024-07-26T06:48:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.091+0000 7f65b6a34640 1 -- 172.21.15.42:0/3523988095 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65b01a8a40 con 0x7f65b0107de0 2024-07-26T06:48:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.092+0000 7f65b6a34640 1 -- 172.21.15.42:0/3523988095 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65b0103720 con 0x7f65b0107de0 2024-07-26T06:48:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.094+0000 7f65ad7fa640 1 -- 172.21.15.42:0/3523988095 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7f65a401a070 con 0x7f65b0107de0 2024-07-26T06:48:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.094+0000 7f65ad7fa640 1 --2- 172.21.15.42:0/3523988095 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f65980b3c60 0x7f65980b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.094+0000 7f65affff640 1 --2- 172.21.15.42:0/3523988095 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f65980b3c60 0x7f65980b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.094+0000 7f65ad7fa640 1 -- 172.21.15.42:0/3523988095 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f65a40e3940 con 0x7f65b0107de0 2024-07-26T06:48:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.095+0000 7f65affff640 1 --2- 172.21.15.42:0/3523988095 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f65980b3c60 0x7f65980b6170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f65a0006010 tx=0x7f65a0000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:42.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.098+0000 7f65ad7fa640 1 -- 172.21.15.42:0/3523988095 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f65a4009de0 con 0x7f65b0107de0 2024-07-26T06:48:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.286+0000 7f65b6a34640 1 -- 172.21.15.42:0/3523988095 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f65b0104f90 con 0x7f65b0107de0 2024-07-26T06:48:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.286+0000 7f65ad7fa640 1 -- 172.21.15.42:0/3523988095 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f65a40aae10 con 0x7f65b0107de0 2024-07-26T06:48:42.287 INFO:teuthology.orchestra.run.smithi042.stdout:77309411407 2024-07-26T06:48:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.288+0000 7f65b6a34640 1 -- 172.21.15.42:0/3523988095 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f65980b3c60 msgr2=0x7f65980b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.288+0000 7f65b6a34640 1 --2- 172.21.15.42:0/3523988095 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f65980b3c60 0x7f65980b6170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f65a0006010 tx=0x7f65a0000bc0 comp rx=0 tx=0).stop 2024-07-26T06:48:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.288+0000 7f65b6a34640 1 -- 172.21.15.42:0/3523988095 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65b0107de0 msgr2=0x7f65b01041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.288+0000 7f65b6a34640 1 --2- 172.21.15.42:0/3523988095 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65b0107de0 0x7f65b01041d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f65a400d770 tx=0x7f65a400dc30 comp rx=0 tx=0).stop 2024-07-26T06:48:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.289+0000 7f65b6a34640 1 -- 172.21.15.42:0/3523988095 shutdown_connections 2024-07-26T06:48:42.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.289+0000 7f65b6a34640 1 --2- 172.21.15.42:0/3523988095 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f65980b3c60 0x7f65980b6170 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:42.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.289+0000 7f65b6a34640 1 --2- 172.21.15.42:0/3523988095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65b0108730 0x7f65b0104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:42.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.289+0000 7f65b6a34640 1 --2- 172.21.15.42:0/3523988095 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65b0107de0 0x7f65b01041d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:42.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.289+0000 7f65b6a34640 1 --2- 172.21.15.42:0/3523988095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f65b01024a0 0x7f65b0103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:42.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.289+0000 7f65b6a34640 1 -- 172.21.15.42:0/3523988095 >> 172.21.15.42:0/3523988095 conn(0x7f65b0079160 msgr2=0x7f65b01014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:42.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.289+0000 7f65b6a34640 1 -- 172.21.15.42:0/3523988095 shutdown_connections 2024-07-26T06:48:42.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:42.289+0000 7f65b6a34640 1 -- 172.21.15.42:0/3523988095 wait complete. 2024-07-26T06:48:42.701 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:42 smithi042 ceph-mon[32775]: pgmap v350: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:48:42.702 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:42 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3523988095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T06:48:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:42 smithi164 ceph-mon[36970]: pgmap v350: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:48:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:42 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3523988095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T06:48:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:42 smithi163 ceph-mon[36873]: pgmap v350: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:48:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:42 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3523988095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T06:48:43.517 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411403 got 77309411407 for osd.2 2024-07-26T06:48:43.517 DEBUG:teuthology.parallel:result is None 2024-07-26T06:48:44.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:44 smithi042 ceph-mon[32775]: pgmap v351: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:44 smithi164 ceph-mon[36970]: pgmap v351: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:44 smithi163 ceph-mon[36873]: pgmap v351: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:46.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.194+0000 7fbca3a99640 1 -- 172.21.15.42:0/2296052151 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9c107e00 msgr2=0x7fbc9c108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:46.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.194+0000 7fbca3a99640 1 --2- 172.21.15.42:0/2296052151 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9c107e00 0x7fbc9c108210 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c009900 tx=0x7fbc8c02a410 comp rx=0 tx=0).stop 2024-07-26T06:48:46.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.194+0000 7fbca3a99640 1 -- 172.21.15.42:0/2296052151 shutdown_connections 2024-07-26T06:48:46.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.194+0000 7fbca3a99640 1 --2- 172.21.15.42:0/2296052151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbc9c108750 0x7fbc9c10ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:46.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.194+0000 7fbca3a99640 1 --2- 172.21.15.42:0/2296052151 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9c107e00 0x7fbc9c108210 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:46.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.194+0000 7fbca3a99640 1 --2- 172.21.15.42:0/2296052151 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc9c102510 0x7fbc9c102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:46.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.194+0000 7fbca3a99640 1 -- 172.21.15.42:0/2296052151 >> 172.21.15.42:0/2296052151 conn(0x7fbc9c079190 msgr2=0x7fbc9c0795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:46.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.195+0000 7fbca3a99640 1 -- 172.21.15.42:0/2296052151 shutdown_connections 2024-07-26T06:48:46.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.195+0000 7fbca3a99640 1 -- 172.21.15.42:0/2296052151 wait complete. 2024-07-26T06:48:46.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.195+0000 7fbca3a99640 1 Processor -- start 2024-07-26T06:48:46.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.195+0000 7fbca3a99640 1 -- start start 2024-07-26T06:48:46.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.196+0000 7fbca3a99640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbc9c102510 0x7fbc9c199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:46.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.196+0000 7fbca3a99640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9c107e00 0x7fbc9c19a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:46.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.196+0000 7fbca3a99640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc9c108750 0x7fbc9c19f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:46.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.196+0000 7fbca3a99640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c1134e0 con 0x7fbc9c107e00 2024-07-26T06:48:46.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.196+0000 7fbca3a99640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c113360 con 0x7fbc9c102510 2024-07-26T06:48:46.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.196+0000 7fbca3a99640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c113660 con 0x7fbc9c108750 2024-07-26T06:48:46.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.196+0000 7fbca1034640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9c107e00 0x7fbc9c19a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:46.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.196+0000 7fbca1034640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9c107e00 0x7fbc9c19a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59594/0 (socket says 172.21.15.42:59594) 2024-07-26T06:48:46.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.196+0000 7fbca1034640 1 -- 172.21.15.42:0/4292549780 learned_addr learned my addr 172.21.15.42:0/4292549780 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:46.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.196+0000 7fbca1835640 1 --2- 172.21.15.42:0/4292549780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbc9c102510 0x7fbc9c199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:46.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.196+0000 7fbca2036640 1 --2- 172.21.15.42:0/4292549780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc9c108750 0x7fbc9c19f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:46.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.196+0000 7fbca1034640 1 -- 172.21.15.42:0/4292549780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc9c108750 msgr2=0x7fbc9c19f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:46.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.196+0000 7fbca1034640 1 --2- 172.21.15.42:0/4292549780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc9c108750 0x7fbc9c19f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:46.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.196+0000 7fbca1034640 1 -- 172.21.15.42:0/4292549780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbc9c102510 msgr2=0x7fbc9c199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:46.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.196+0000 7fbca1034640 1 --2- 172.21.15.42:0/4292549780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbc9c102510 0x7fbc9c199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:46.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.196+0000 7fbca1034640 1 -- 172.21.15.42:0/4292549780 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc9c19fe40 con 0x7fbc9c107e00 2024-07-26T06:48:46.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.196+0000 7fbca1034640 1 --2- 172.21.15.42:0/4292549780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9c107e00 0x7fbc9c19a470 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c032fd0 tx=0x7fbc8c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:46.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.197+0000 7fbc92ffd640 1 -- 172.21.15.42:0/4292549780 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c03f030 con 0x7fbc9c107e00 2024-07-26T06:48:46.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.197+0000 7fbca3a99640 1 -- 172.21.15.42:0/4292549780 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc9c1a00d0 con 0x7fbc9c107e00 2024-07-26T06:48:46.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.197+0000 7fbc92ffd640 1 -- 172.21.15.42:0/4292549780 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc8c004490 con 0x7fbc9c107e00 2024-07-26T06:48:46.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.197+0000 7fbc92ffd640 1 -- 172.21.15.42:0/4292549780 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c035070 con 0x7fbc9c107e00 2024-07-26T06:48:46.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.197+0000 7fbca3a99640 1 -- 172.21.15.42:0/4292549780 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc9c1a6890 con 0x7fbc9c107e00 2024-07-26T06:48:46.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.198+0000 7fbca3a99640 1 -- 172.21.15.42:0/4292549780 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc9c103740 con 0x7fbc9c107e00 2024-07-26T06:48:46.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.199+0000 7fbc92ffd640 1 -- 172.21.15.42:0/4292549780 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7fbc8c042030 con 0x7fbc9c107e00 2024-07-26T06:48:46.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.200+0000 7fbc92ffd640 1 --2- 172.21.15.42:0/4292549780 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbc880b3c60 0x7fbc880b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:46.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.200+0000 7fbca1835640 1 --2- 172.21.15.42:0/4292549780 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbc880b3c60 0x7fbc880b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:46.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.200+0000 7fbc92ffd640 1 -- 172.21.15.42:0/4292549780 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fbc8c1042e0 con 0x7fbc9c107e00 2024-07-26T06:48:46.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.201+0000 7fbca1835640 1 --2- 172.21.15.42:0/4292549780 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbc880b3c60 0x7fbc880b6170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbc98005fd0 tx=0x7fbc9800b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:46.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.205+0000 7fbc92ffd640 1 -- 172.21.15.42:0/4292549780 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbc8c0cb7b0 con 0x7fbc9c107e00 2024-07-26T06:48:46.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.393+0000 7fbca3a99640 1 -- 172.21.15.42:0/4292549780 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fbc9c10a520 con 0x7fbc9c107e00 2024-07-26T06:48:46.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.394+0000 7fbc92ffd640 1 -- 172.21.15.42:0/4292549780 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbc8c0cb0b0 con 0x7fbc9c107e00 2024-07-26T06:48:46.394 INFO:teuthology.orchestra.run.smithi042.stdout:133143986246 2024-07-26T06:48:46.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.396+0000 7fbca3a99640 1 -- 172.21.15.42:0/4292549780 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbc880b3c60 msgr2=0x7fbc880b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:46.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.396+0000 7fbca3a99640 1 --2- 172.21.15.42:0/4292549780 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbc880b3c60 0x7fbc880b6170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbc98005fd0 tx=0x7fbc9800b390 comp rx=0 tx=0).stop 2024-07-26T06:48:46.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.396+0000 7fbca3a99640 1 -- 172.21.15.42:0/4292549780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9c107e00 msgr2=0x7fbc9c19a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:46.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.396+0000 7fbca3a99640 1 --2- 172.21.15.42:0/4292549780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9c107e00 0x7fbc9c19a470 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c032fd0 tx=0x7fbc8c004270 comp rx=0 tx=0).stop 2024-07-26T06:48:46.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.396+0000 7fbca3a99640 1 -- 172.21.15.42:0/4292549780 shutdown_connections 2024-07-26T06:48:46.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.396+0000 7fbca3a99640 1 --2- 172.21.15.42:0/4292549780 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbc880b3c60 0x7fbc880b6170 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:46.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.396+0000 7fbca3a99640 1 --2- 172.21.15.42:0/4292549780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc9c108750 0x7fbc9c19f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:46.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.396+0000 7fbca3a99640 1 --2- 172.21.15.42:0/4292549780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc9c107e00 0x7fbc9c19a470 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:46.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.396+0000 7fbca3a99640 1 --2- 172.21.15.42:0/4292549780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbc9c102510 0x7fbc9c199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:46.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.396+0000 7fbca3a99640 1 -- 172.21.15.42:0/4292549780 >> 172.21.15.42:0/4292549780 conn(0x7fbc9c079190 msgr2=0x7fbc9c103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:46.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.397+0000 7fbca3a99640 1 -- 172.21.15.42:0/4292549780 shutdown_connections 2024-07-26T06:48:46.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:46.397+0000 7fbca3a99640 1 -- 172.21.15.42:0/4292549780 wait complete. 2024-07-26T06:48:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:46 smithi042 ceph-mon[32775]: pgmap v352: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:48:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:46 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/4292549780' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T06:48:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:46 smithi164 ceph-mon[36970]: pgmap v352: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:48:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:46 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/4292549780' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T06:48:46.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:46 smithi163 ceph-mon[36873]: pgmap v352: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:48:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:46 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/4292549780' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T06:48:47.462 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:48.720 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:48 smithi042 ceph-mon[32775]: pgmap v353: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:48.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:48 smithi164 ceph-mon[36970]: pgmap v353: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:48.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.723+0000 7faf7eebe640 1 -- 172.21.15.42:0/1668411634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf78104ba0 msgr2=0x7faf78104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:48.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.723+0000 7faf7eebe640 1 --2- 172.21.15.42:0/1668411634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf78104ba0 0x7faf78104fb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7faf64009900 tx=0x7faf6402a3f0 comp rx=0 tx=0).stop 2024-07-26T06:48:48.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.724+0000 7faf7eebe640 1 -- 172.21.15.42:0/1668411634 shutdown_connections 2024-07-26T06:48:48.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.724+0000 7faf7eebe640 1 --2- 172.21.15.42:0/1668411634 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf781054f0 0x7faf7810ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:48.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.724+0000 7faf7eebe640 1 --2- 172.21.15.42:0/1668411634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf78104ba0 0x7faf78104fb0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:48.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.724+0000 7faf7eebe640 1 --2- 172.21.15.42:0/1668411634 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf78102b90 0x7faf78102fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:48.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.724+0000 7faf7eebe640 1 -- 172.21.15.42:0/1668411634 >> 172.21.15.42:0/1668411634 conn(0x7faf78077690 msgr2=0x7faf78077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:48.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.724+0000 7faf7eebe640 1 -- 172.21.15.42:0/1668411634 shutdown_connections 2024-07-26T06:48:48.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.724+0000 7faf7eebe640 1 -- 172.21.15.42:0/1668411634 wait complete. 2024-07-26T06:48:48.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.724+0000 7faf7eebe640 1 Processor -- start 2024-07-26T06:48:48.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.724+0000 7faf7eebe640 1 -- start start 2024-07-26T06:48:48.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.725+0000 7faf7eebe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf78102b90 0x7faf780fe810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:48.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.725+0000 7faf7eebe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf78104ba0 0x7faf780fed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:48.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.725+0000 7faf7eebe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf781054f0 0x7faf780ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:48.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.725+0000 7faf7eebe640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faf781123a0 con 0x7faf781054f0 2024-07-26T06:48:48.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.725+0000 7faf7eebe640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7faf78112220 con 0x7faf78102b90 2024-07-26T06:48:48.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.725+0000 7faf7eebe640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faf78112520 con 0x7faf78104ba0 2024-07-26T06:48:48.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.725+0000 7faf7d45b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf781054f0 0x7faf780ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:48.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.725+0000 7faf7d45b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf781054f0 0x7faf780ff290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34380/0 (socket says 172.21.15.42:34380) 2024-07-26T06:48:48.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.725+0000 7faf7d45b640 1 -- 172.21.15.42:0/1517108359 learned_addr learned my addr 172.21.15.42:0/1517108359 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:48.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.725+0000 7faf7cc5a640 1 --2- 172.21.15.42:0/1517108359 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf78102b90 0x7faf780fe810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:48.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.725+0000 7faf77fff640 1 --2- 172.21.15.42:0/1517108359 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf78104ba0 0x7faf780fed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:48.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.725+0000 7faf7d45b640 1 -- 172.21.15.42:0/1517108359 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf78104ba0 msgr2=0x7faf780fed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:48.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.725+0000 7faf7d45b640 1 --2- 172.21.15.42:0/1517108359 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf78104ba0 0x7faf780fed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:48.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.725+0000 7faf7d45b640 1 -- 172.21.15.42:0/1517108359 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf78102b90 msgr2=0x7faf780fe810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:48.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.725+0000 7faf7d45b640 1 --2- 172.21.15.42:0/1517108359 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf78102b90 0x7faf780fe810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:48.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.725+0000 7faf7d45b640 1 -- 172.21.15.42:0/1517108359 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf781a80f0 con 0x7faf781054f0 2024-07-26T06:48:48.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.725+0000 7faf7cc5a640 1 --2- 172.21.15.42:0/1517108359 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf78102b90 0x7faf780fe810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:48:48.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.726+0000 7faf7d45b640 1 --2- 172.21.15.42:0/1517108359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf781054f0 0x7faf780ff290 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7faf7000cce0 tx=0x7faf7000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:48.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.726+0000 7faf75ffb640 1 -- 172.21.15.42:0/1517108359 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf7000f450 con 0x7faf781054f0 2024-07-26T06:48:48.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.726+0000 7faf7eebe640 1 -- 172.21.15.42:0/1517108359 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf781a83e0 con 0x7faf781054f0 2024-07-26T06:48:48.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.726+0000 7faf75ffb640 1 -- 172.21.15.42:0/1517108359 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf7000faa0 con 0x7faf781054f0 2024-07-26T06:48:48.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.726+0000 7faf75ffb640 1 -- 172.21.15.42:0/1517108359 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf70018b70 con 0x7faf781054f0 2024-07-26T06:48:48.729 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:48.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.726+0000 7faf7eebe640 1 -- 172.21.15.42:0/1517108359 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf781a88f0 con 0x7faf781054f0 2024-07-26T06:48:48.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.727+0000 7faf7eebe640 1 -- 172.21.15.42:0/1517108359 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf78102fa0 con 0x7faf781054f0 2024-07-26T06:48:48.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.729+0000 7faf75ffb640 1 -- 172.21.15.42:0/1517108359 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 17) ==== 151110+0+0 (secure 0 0 0) 0x7faf70018d10 con 0x7faf781054f0 2024-07-26T06:48:48.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.729+0000 7faf75ffb640 1 --2- 172.21.15.42:0/1517108359 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faf680b3c60 0x7faf680b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:48.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.730+0000 7faf7cc5a640 1 --2- 172.21.15.42:0/1517108359 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faf680b3c60 0x7faf680b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:48.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.730+0000 7faf75ffb640 1 -- 172.21.15.42:0/1517108359 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7faf700e4000 con 0x7faf781054f0 2024-07-26T06:48:48.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.730+0000 7faf7cc5a640 1 --2- 172.21.15.42:0/1517108359 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faf680b3c60 0x7faf680b6170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faf78100260 tx=0x7faf6c00b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:48.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.734+0000 7faf75ffb640 1 -- 172.21.15.42:0/1517108359 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7faf70014030 con 0x7faf781054f0 2024-07-26T06:48:48.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:48 smithi163 ceph-mon[36873]: pgmap v353: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:48:48.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.929+0000 7faf7eebe640 1 -- 172.21.15.42:0/1517108359 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7faf781a8c40 con 0x7faf781054f0 2024-07-26T06:48:48.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.930+0000 7faf75ffb640 1 -- 172.21.15.42:0/1517108359 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7faf700ab570 con 0x7faf781054f0 2024-07-26T06:48:48.930 INFO:teuthology.orchestra.run.smithi042.stdout:300647710756 2024-07-26T06:48:48.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.932+0000 7faf7eebe640 1 -- 172.21.15.42:0/1517108359 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faf680b3c60 msgr2=0x7faf680b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:48.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.932+0000 7faf7eebe640 1 --2- 172.21.15.42:0/1517108359 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faf680b3c60 0x7faf680b6170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faf78100260 tx=0x7faf6c00b410 comp rx=0 tx=0).stop 2024-07-26T06:48:48.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.932+0000 7faf7eebe640 1 -- 172.21.15.42:0/1517108359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf781054f0 msgr2=0x7faf780ff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:48.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.932+0000 7faf7eebe640 1 --2- 172.21.15.42:0/1517108359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf781054f0 0x7faf780ff290 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7faf7000cce0 tx=0x7faf7000a620 comp rx=0 tx=0).stop 2024-07-26T06:48:48.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.932+0000 7faf7eebe640 1 -- 172.21.15.42:0/1517108359 shutdown_connections 2024-07-26T06:48:48.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.932+0000 7faf7eebe640 1 --2- 172.21.15.42:0/1517108359 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faf680b3c60 0x7faf680b6170 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:48.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.932+0000 7faf7eebe640 1 --2- 172.21.15.42:0/1517108359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf781054f0 0x7faf780ff290 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:48.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.932+0000 7faf7eebe640 1 --2- 172.21.15.42:0/1517108359 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf78104ba0 0x7faf780fed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:48.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.932+0000 7faf7eebe640 1 --2- 172.21.15.42:0/1517108359 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf78102b90 0x7faf780fe810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:48.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.932+0000 7faf7eebe640 1 -- 172.21.15.42:0/1517108359 >> 172.21.15.42:0/1517108359 conn(0x7faf78077690 msgr2=0x7faf78105f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:48.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.932+0000 7faf7eebe640 1 -- 172.21.15.42:0/1517108359 shutdown_connections 2024-07-26T06:48:48.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:48.932+0000 7faf7eebe640 1 -- 172.21.15.42:0/1517108359 wait complete. 2024-07-26T06:48:48.969 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:48:49.555 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986243 got 133143986246 for osd.4 2024-07-26T06:48:49.556 DEBUG:teuthology.parallel:result is None 2024-07-26T06:48:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:49 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1517108359' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T06:48:49.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:49 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1517108359' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T06:48:49.764 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:49 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1517108359' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T06:48:50.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:50 smithi164 ceph-mon[36970]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:48:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:50 smithi163 ceph-mon[36873]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:48:50.763 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:50 smithi042 ceph-mon[32775]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:48:51.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.672+0000 7fe83da73640 1 -- 172.21.15.42:0/3595703577 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe838102bc0 msgr2=0x7fe838102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:51.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.672+0000 7fe83da73640 1 --2- 172.21.15.42:0/3595703577 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe838102bc0 0x7fe838102fd0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe828009ca0 tx=0x7fe82802a8c0 comp rx=0 tx=0).stop 2024-07-26T06:48:51.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.672+0000 7fe83da73640 1 -- 172.21.15.42:0/3595703577 shutdown_connections 2024-07-26T06:48:51.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.672+0000 7fe83da73640 1 --2- 172.21.15.42:0/3595703577 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe838105520 0x7fe83810ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:51.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.672+0000 7fe83da73640 1 --2- 172.21.15.42:0/3595703577 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe838104bd0 0x7fe838104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:51.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.672+0000 7fe83da73640 1 --2- 172.21.15.42:0/3595703577 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe838102bc0 0x7fe838102fd0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:51.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.672+0000 7fe83da73640 1 -- 172.21.15.42:0/3595703577 >> 172.21.15.42:0/3595703577 conn(0x7fe8380776f0 msgr2=0x7fe838077b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:51.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.672+0000 7fe83da73640 1 -- 172.21.15.42:0/3595703577 shutdown_connections 2024-07-26T06:48:51.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.672+0000 7fe83da73640 1 -- 172.21.15.42:0/3595703577 wait complete. 2024-07-26T06:48:51.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.673+0000 7fe83da73640 1 Processor -- start 2024-07-26T06:48:51.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.673+0000 7fe83da73640 1 -- start start 2024-07-26T06:48:51.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.673+0000 7fe83da73640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe838102bc0 0x7fe83819c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:51.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.673+0000 7fe83da73640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe838104bd0 0x7fe83819c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:51.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.673+0000 7fe83da73640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe838105520 0x7fe8381a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:51.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.673+0000 7fe83da73640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe838112430 con 0x7fe838104bd0 2024-07-26T06:48:51.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.673+0000 7fe83da73640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe8381122b0 con 0x7fe838102bc0 2024-07-26T06:48:51.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.673+0000 7fe83da73640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe8381125b0 con 0x7fe838105520 2024-07-26T06:48:51.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.673+0000 7fe836ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe838104bd0 0x7fe83819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:51.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.673+0000 7fe836ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe838104bd0 0x7fe83819c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34398/0 (socket says 172.21.15.42:34398) 2024-07-26T06:48:51.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.673+0000 7fe836ffd640 1 -- 172.21.15.42:0/3328819777 learned_addr learned my addr 172.21.15.42:0/3328819777 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:51.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.674+0000 7fe8377fe640 1 --2- 172.21.15.42:0/3328819777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe838102bc0 0x7fe83819c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:51.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.674+0000 7fe837fff640 1 --2- 172.21.15.42:0/3328819777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe838105520 0x7fe8381a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:51.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.674+0000 7fe836ffd640 1 -- 172.21.15.42:0/3328819777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe838105520 msgr2=0x7fe8381a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:51.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.674+0000 7fe836ffd640 1 --2- 172.21.15.42:0/3328819777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe838105520 0x7fe8381a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:51.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.674+0000 7fe836ffd640 1 -- 172.21.15.42:0/3328819777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe838102bc0 msgr2=0x7fe83819c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:51.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.674+0000 7fe836ffd640 1 --2- 172.21.15.42:0/3328819777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe838102bc0 0x7fe83819c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:51.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.674+0000 7fe836ffd640 1 -- 172.21.15.42:0/3328819777 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8381a2040 con 0x7fe838104bd0 2024-07-26T06:48:51.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.674+0000 7fe8377fe640 1 --2- 172.21.15.42:0/3328819777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe838102bc0 0x7fe83819c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:48:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.674+0000 7fe836ffd640 1 --2- 172.21.15.42:0/3328819777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe838104bd0 0x7fe83819c670 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe82000d740 tx=0x7fe82000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.674+0000 7fe834ff9640 1 -- 172.21.15.42:0/3328819777 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe820017070 con 0x7fe838104bd0 2024-07-26T06:48:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.674+0000 7fe83da73640 1 -- 172.21.15.42:0/3328819777 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8381a2330 con 0x7fe838104bd0 2024-07-26T06:48:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.674+0000 7fe834ff9640 1 -- 172.21.15.42:0/3328819777 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe82000de50 con 0x7fe838104bd0 2024-07-26T06:48:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.674+0000 7fe834ff9640 1 -- 172.21.15.42:0/3328819777 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82000f620 con 0x7fe838104bd0 2024-07-26T06:48:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.674+0000 7fe83da73640 1 -- 172.21.15.42:0/3328819777 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8381a8a90 con 0x7fe838104bd0 2024-07-26T06:48:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.675+0000 7fe83da73640 1 -- 172.21.15.42:0/3328819777 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe838103940 con 0x7fe838104bd0 2024-07-26T06:48:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.677+0000 7fe834ff9640 1 -- 172.21.15.42:0/3328819777 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fe82001a070 con 0x7fe838104bd0 2024-07-26T06:48:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.678+0000 7fe834ff9640 1 --2- 172.21.15.42:0/3328819777 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8240b3bd0 0x7fe8240b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.678+0000 7fe8377fe640 1 --2- 172.21.15.42:0/3328819777 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8240b3bd0 0x7fe8240b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.678+0000 7fe834ff9640 1 -- 172.21.15.42:0/3328819777 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fe8200e3670 con 0x7fe838104bd0 2024-07-26T06:48:51.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.679+0000 7fe8377fe640 1 --2- 172.21.15.42:0/3328819777 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8240b3bd0 0x7fe8240b60e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe8280094e0 tx=0x7fe8280093f0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:51.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.682+0000 7fe834ff9640 1 -- 172.21.15.42:0/3328819777 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe8200aaca0 con 0x7fe838104bd0 2024-07-26T06:48:51.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.868+0000 7fe83da73640 1 -- 172.21.15.42:0/3328819777 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fe8381a8cf0 con 0x7fe838104bd0 2024-07-26T06:48:51.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.868+0000 7fe834ff9640 1 -- 172.21.15.42:0/3328819777 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe8200aa5a0 con 0x7fe838104bd0 2024-07-26T06:48:51.871 INFO:teuthology.orchestra.run.smithi042.stdout:34359738457 2024-07-26T06:48:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.870+0000 7fe83da73640 1 -- 172.21.15.42:0/3328819777 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8240b3bd0 msgr2=0x7fe8240b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.870+0000 7fe83da73640 1 --2- 172.21.15.42:0/3328819777 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8240b3bd0 0x7fe8240b60e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe8280094e0 tx=0x7fe8280093f0 comp rx=0 tx=0).stop 2024-07-26T06:48:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.870+0000 7fe83da73640 1 -- 172.21.15.42:0/3328819777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe838104bd0 msgr2=0x7fe83819c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.870+0000 7fe83da73640 1 --2- 172.21.15.42:0/3328819777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe838104bd0 0x7fe83819c670 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe82000d740 tx=0x7fe82000dc00 comp rx=0 tx=0).stop 2024-07-26T06:48:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.870+0000 7fe83da73640 1 -- 172.21.15.42:0/3328819777 shutdown_connections 2024-07-26T06:48:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.870+0000 7fe83da73640 1 --2- 172.21.15.42:0/3328819777 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8240b3bd0 0x7fe8240b60e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.870+0000 7fe83da73640 1 --2- 172.21.15.42:0/3328819777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe838105520 0x7fe8381a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.870+0000 7fe83da73640 1 --2- 172.21.15.42:0/3328819777 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe838104bd0 0x7fe83819c670 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.870+0000 7fe83da73640 1 --2- 172.21.15.42:0/3328819777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe838102bc0 0x7fe83819c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.870+0000 7fe83da73640 1 -- 172.21.15.42:0/3328819777 >> 172.21.15.42:0/3328819777 conn(0x7fe8380776f0 msgr2=0x7fe83810e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.870+0000 7fe83da73640 1 -- 172.21.15.42:0/3328819777 shutdown_connections 2024-07-26T06:48:51.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:51.870+0000 7fe83da73640 1 -- 172.21.15.42:0/3328819777 wait complete. 2024-07-26T06:48:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:52 smithi164 ceph-mon[36970]: pgmap v355: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:48:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:52 smithi164 ceph-mon[36970]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-07-26T06:48:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:52 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3328819777' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T06:48:52.732 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:52 smithi042 ceph-mon[32775]: pgmap v355: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:48:52.733 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:52 smithi042 ceph-mon[32775]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-07-26T06:48:52.733 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:52 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3328819777' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T06:48:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:52 smithi163 ceph-mon[36873]: pgmap v355: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:48:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:52 smithi163 ceph-mon[36873]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-07-26T06:48:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:52 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3328819777' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T06:48:53.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.052+0000 7f41f178b640 1 -- 172.21.15.42:0/2294752531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41ec102510 msgr2=0x7f41ec102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:53.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.052+0000 7f41f178b640 1 --2- 172.21.15.42:0/2294752531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41ec102510 0x7f41ec102920 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009900 tx=0x7f41dc02a410 comp rx=0 tx=0).stop 2024-07-26T06:48:53.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.053+0000 7f41f178b640 1 -- 172.21.15.42:0/2294752531 shutdown_connections 2024-07-26T06:48:53.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.053+0000 7f41f178b640 1 --2- 172.21.15.42:0/2294752531 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41ec108750 0x7f41ec10ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.053+0000 7f41f178b640 1 --2- 172.21.15.42:0/2294752531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41ec107e00 0x7f41ec108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.053+0000 7f41f178b640 1 --2- 172.21.15.42:0/2294752531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41ec102510 0x7f41ec102920 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.053+0000 7f41f178b640 1 -- 172.21.15.42:0/2294752531 >> 172.21.15.42:0/2294752531 conn(0x7f41ec0791e0 msgr2=0x7f41ec0795f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:53.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.053+0000 7f41f178b640 1 -- 172.21.15.42:0/2294752531 shutdown_connections 2024-07-26T06:48:53.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.053+0000 7f41f178b640 1 -- 172.21.15.42:0/2294752531 wait complete. 2024-07-26T06:48:53.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.054+0000 7f41f178b640 1 Processor -- start 2024-07-26T06:48:53.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.054+0000 7f41f178b640 1 -- start start 2024-07-26T06:48:53.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.054+0000 7f41f178b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41ec102510 0x7f41ec19ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:53.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.054+0000 7f41f178b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41ec107e00 0x7f41ec19b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:53.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.054+0000 7f41f178b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41ec108750 0x7f41ec19b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.054+0000 7f41f178b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec112400 con 0x7f41ec108750 2024-07-26T06:48:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.054+0000 7f41f178b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec112280 con 0x7f41ec102510 2024-07-26T06:48:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.054+0000 7f41f178b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec112580 con 0x7f41ec107e00 2024-07-26T06:48:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.055+0000 7f41eb7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41ec108750 0x7f41ec19b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.055+0000 7f41eb7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41ec108750 0x7f41ec19b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34424/0 (socket says 172.21.15.42:34424) 2024-07-26T06:48:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.055+0000 7f41eb7fe640 1 -- 172.21.15.42:0/2598547865 learned_addr learned my addr 172.21.15.42:0/2598547865 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.055+0000 7f41eaffd640 1 --2- 172.21.15.42:0/2598547865 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41ec102510 0x7f41ec19ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.055+0000 7f41ea7fc640 1 --2- 172.21.15.42:0/2598547865 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41ec107e00 0x7f41ec19b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.055+0000 7f41eb7fe640 1 -- 172.21.15.42:0/2598547865 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41ec107e00 msgr2=0x7f41ec19b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.055+0000 7f41eb7fe640 1 --2- 172.21.15.42:0/2598547865 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41ec107e00 0x7f41ec19b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.055+0000 7f41eb7fe640 1 -- 172.21.15.42:0/2598547865 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41ec102510 msgr2=0x7f41ec19ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:53.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.055+0000 7f41eb7fe640 1 --2- 172.21.15.42:0/2598547865 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41ec102510 0x7f41ec19ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.055+0000 7f41eb7fe640 1 -- 172.21.15.42:0/2598547865 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ec1a5fc0 con 0x7f41ec108750 2024-07-26T06:48:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.055+0000 7f41eaffd640 1 --2- 172.21.15.42:0/2598547865 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41ec102510 0x7f41ec19ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:48:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.055+0000 7f41eb7fe640 1 --2- 172.21.15.42:0/2598547865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41ec108750 0x7f41ec19b7b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f41e4009fd0 tx=0x7f41e400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.055+0000 7f41d3fff640 1 -- 172.21.15.42:0/2598547865 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e4017070 con 0x7f41ec108750 2024-07-26T06:48:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.056+0000 7f41d3fff640 1 -- 172.21.15.42:0/2598547865 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41e4003e20 con 0x7f41ec108750 2024-07-26T06:48:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.056+0000 7f41f178b640 1 -- 172.21.15.42:0/2598547865 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41ec1a62b0 con 0x7f41ec108750 2024-07-26T06:48:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.056+0000 7f41d3fff640 1 -- 172.21.15.42:0/2598547865 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e4010460 con 0x7f41ec108750 2024-07-26T06:48:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.056+0000 7f41f178b640 1 -- 172.21.15.42:0/2598547865 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41ec1a67a0 con 0x7f41ec108750 2024-07-26T06:48:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.057+0000 7f41f178b640 1 -- 172.21.15.42:0/2598547865 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41ec102920 con 0x7f41ec108750 2024-07-26T06:48:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.060+0000 7f41d3fff640 1 -- 172.21.15.42:0/2598547865 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f41e4003970 con 0x7f41ec108750 2024-07-26T06:48:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.060+0000 7f41d3fff640 1 --2- 172.21.15.42:0/2598547865 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f41d40b3c00 0x7f41d40b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:53.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.061+0000 7f41eaffd640 1 --2- 172.21.15.42:0/2598547865 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f41d40b3c00 0x7f41d40b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:53.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.061+0000 7f41d3fff640 1 -- 172.21.15.42:0/2598547865 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f41e40e3d00 con 0x7f41ec108750 2024-07-26T06:48:53.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.061+0000 7f41eaffd640 1 --2- 172.21.15.42:0/2598547865 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f41d40b3c00 0x7f41d40b6110 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f41dc000c00 tx=0x7f41dc009680 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:53.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.064+0000 7f41d3fff640 1 -- 172.21.15.42:0/2598547865 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f41e40ab330 con 0x7f41ec108750 2024-07-26T06:48:53.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.285+0000 7f41f178b640 1 -- 172.21.15.42:0/2598547865 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f41ec06b660 con 0x7f41ec108750 2024-07-26T06:48:53.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.286+0000 7f41d3fff640 1 -- 172.21.15.42:0/2598547865 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f41e40aac30 con 0x7f41ec108750 2024-07-26T06:48:53.286 INFO:teuthology.orchestra.run.smithi042.stdout:274877906986 2024-07-26T06:48:53.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.288+0000 7f41f178b640 1 -- 172.21.15.42:0/2598547865 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f41d40b3c00 msgr2=0x7f41d40b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:53.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.288+0000 7f41f178b640 1 --2- 172.21.15.42:0/2598547865 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f41d40b3c00 0x7f41d40b6110 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f41dc000c00 tx=0x7f41dc009680 comp rx=0 tx=0).stop 2024-07-26T06:48:53.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.288+0000 7f41f178b640 1 -- 172.21.15.42:0/2598547865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41ec108750 msgr2=0x7f41ec19b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:53.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.288+0000 7f41f178b640 1 --2- 172.21.15.42:0/2598547865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41ec108750 0x7f41ec19b7b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f41e4009fd0 tx=0x7f41e400b260 comp rx=0 tx=0).stop 2024-07-26T06:48:53.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.288+0000 7f41f178b640 1 -- 172.21.15.42:0/2598547865 shutdown_connections 2024-07-26T06:48:53.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.288+0000 7f41f178b640 1 --2- 172.21.15.42:0/2598547865 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f41d40b3c00 0x7f41d40b6110 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.288+0000 7f41f178b640 1 --2- 172.21.15.42:0/2598547865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41ec108750 0x7f41ec19b7b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.288+0000 7f41f178b640 1 --2- 172.21.15.42:0/2598547865 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f41ec107e00 0x7f41ec19b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.288+0000 7f41f178b640 1 --2- 172.21.15.42:0/2598547865 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41ec102510 0x7f41ec19ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.288+0000 7f41f178b640 1 -- 172.21.15.42:0/2598547865 >> 172.21.15.42:0/2598547865 conn(0x7f41ec0791e0 msgr2=0x7f41ec105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:53.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.289+0000 7f41f178b640 1 -- 172.21.15.42:0/2598547865 shutdown_connections 2024-07-26T06:48:53.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.289+0000 7f41f178b640 1 -- 172.21.15.42:0/2598547865 wait complete. 2024-07-26T06:48:53.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.334+0000 7fe470afb640 1 -- 172.21.15.42:0/3271869630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe46c107dd0 msgr2=0x7fe46c1081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:53.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.334+0000 7fe470afb640 1 --2- 172.21.15.42:0/3271869630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe46c107dd0 0x7fe46c1081e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe460009900 tx=0x7fe46002a410 comp rx=0 tx=0).stop 2024-07-26T06:48:53.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.334+0000 7fe470afb640 1 -- 172.21.15.42:0/3271869630 shutdown_connections 2024-07-26T06:48:53.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.334+0000 7fe470afb640 1 --2- 172.21.15.42:0/3271869630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe46c108720 0x7fe46c10ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.334+0000 7fe470afb640 1 --2- 172.21.15.42:0/3271869630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe46c107dd0 0x7fe46c1081e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.334+0000 7fe470afb640 1 --2- 172.21.15.42:0/3271869630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe46c102490 0x7fe46c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.334+0000 7fe470afb640 1 -- 172.21.15.42:0/3271869630 >> 172.21.15.42:0/3271869630 conn(0x7fe46c079160 msgr2=0x7fe46c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:53.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.334+0000 7fe470afb640 1 -- 172.21.15.42:0/3271869630 shutdown_connections 2024-07-26T06:48:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.334+0000 7fe470afb640 1 -- 172.21.15.42:0/3271869630 wait complete. 2024-07-26T06:48:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.335+0000 7fe470afb640 1 Processor -- start 2024-07-26T06:48:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.335+0000 7fe470afb640 1 -- start start 2024-07-26T06:48:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.336+0000 7fe470afb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe46c102490 0x7fe46c06b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.336+0000 7fe470afb640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe46c107dd0 0x7fe46c198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.336+0000 7fe470afb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe46c108720 0x7fe46c199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.336+0000 7fe470afb640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe46c112370 con 0x7fe46c108720 2024-07-26T06:48:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.336+0000 7fe470afb640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe46c1121f0 con 0x7fe46c107dd0 2024-07-26T06:48:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.336+0000 7fe470afb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe46c1124f0 con 0x7fe46c102490 2024-07-26T06:48:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.336+0000 7fe46ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe46c108720 0x7fe46c199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.336+0000 7fe46ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe46c108720 0x7fe46c199050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34448/0 (socket says 172.21.15.42:34448) 2024-07-26T06:48:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.336+0000 7fe46ad9d640 1 -- 172.21.15.42:0/1432493781 learned_addr learned my addr 172.21.15.42:0/1432493781 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:53.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.336+0000 7fe46a59c640 1 --2- 172.21.15.42:0/1432493781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe46c102490 0x7fe46c06b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:53.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.336+0000 7fe469d9b640 1 --2- 172.21.15.42:0/1432493781 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe46c107dd0 0x7fe46c198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:53.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.336+0000 7fe46ad9d640 1 -- 172.21.15.42:0/1432493781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe46c102490 msgr2=0x7fe46c06b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:53.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.336+0000 7fe46ad9d640 1 --2- 172.21.15.42:0/1432493781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe46c102490 0x7fe46c06b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.336+0000 7fe46ad9d640 1 -- 172.21.15.42:0/1432493781 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe46c107dd0 msgr2=0x7fe46c198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:53.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.336+0000 7fe46ad9d640 1 --2- 172.21.15.42:0/1432493781 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe46c107dd0 0x7fe46c198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.336+0000 7fe46ad9d640 1 -- 172.21.15.42:0/1432493781 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe46c199740 con 0x7fe46c108720 2024-07-26T06:48:53.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.336+0000 7fe46ad9d640 1 --2- 172.21.15.42:0/1432493781 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe46c108720 0x7fe46c199050 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe46400cad0 tx=0x7fe46400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:53.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.337+0000 7fe45b7fe640 1 -- 172.21.15.42:0/1432493781 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46400ad60 con 0x7fe46c108720 2024-07-26T06:48:53.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.337+0000 7fe45b7fe640 1 -- 172.21.15.42:0/1432493781 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe464005020 con 0x7fe46c108720 2024-07-26T06:48:53.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.337+0000 7fe470afb640 1 -- 172.21.15.42:0/1432493781 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe46c19db60 con 0x7fe46c108720 2024-07-26T06:48:53.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.337+0000 7fe45b7fe640 1 -- 172.21.15.42:0/1432493781 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46400f6b0 con 0x7fe46c108720 2024-07-26T06:48:53.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.337+0000 7fe470afb640 1 -- 172.21.15.42:0/1432493781 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe46c19e010 con 0x7fe46c108720 2024-07-26T06:48:53.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.338+0000 7fe470afb640 1 -- 172.21.15.42:0/1432493781 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe46c103710 con 0x7fe46c108720 2024-07-26T06:48:53.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.340+0000 7fe45b7fe640 1 -- 172.21.15.42:0/1432493781 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fe4640051d0 con 0x7fe46c108720 2024-07-26T06:48:53.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.341+0000 7fe45b7fe640 1 --2- 172.21.15.42:0/1432493781 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe4540b3c00 0x7fe4540b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:53.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.341+0000 7fe46a59c640 1 --2- 172.21.15.42:0/1432493781 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe4540b3c00 0x7fe4540b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:53.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.341+0000 7fe45b7fe640 1 -- 172.21.15.42:0/1432493781 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fe4640e3a90 con 0x7fe46c108720 2024-07-26T06:48:53.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.341+0000 7fe46a59c640 1 --2- 172.21.15.42:0/1432493781 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe4540b3c00 0x7fe4540b6110 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe46c19ebd0 tx=0x7fe45c00a2d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:53.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.345+0000 7fe45b7fe640 1 -- 172.21.15.42:0/1432493781 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe464014030 con 0x7fe46c108720 2024-07-26T06:48:53.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.532+0000 7fe470afb640 1 -- 172.21.15.42:0/1432493781 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fe46c109f80 con 0x7fe46c108720 2024-07-26T06:48:53.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.532+0000 7fe45b7fe640 1 -- 172.21.15.42:0/1432493781 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe4640ab0c0 con 0x7fe46c108720 2024-07-26T06:48:53.559 INFO:teuthology.orchestra.run.smithi042.stdout:154618822723 2024-07-26T06:48:53.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.534+0000 7fe470afb640 1 -- 172.21.15.42:0/1432493781 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe4540b3c00 msgr2=0x7fe4540b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:53.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.534+0000 7fe470afb640 1 --2- 172.21.15.42:0/1432493781 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe4540b3c00 0x7fe4540b6110 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe46c19ebd0 tx=0x7fe45c00a2d0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.534+0000 7fe470afb640 1 -- 172.21.15.42:0/1432493781 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe46c108720 msgr2=0x7fe46c199050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:53.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.534+0000 7fe470afb640 1 --2- 172.21.15.42:0/1432493781 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe46c108720 0x7fe46c199050 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe46400cad0 tx=0x7fe46400ced0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.535+0000 7fe470afb640 1 -- 172.21.15.42:0/1432493781 shutdown_connections 2024-07-26T06:48:53.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.535+0000 7fe470afb640 1 --2- 172.21.15.42:0/1432493781 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe4540b3c00 0x7fe4540b6110 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.535+0000 7fe470afb640 1 --2- 172.21.15.42:0/1432493781 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe46c108720 0x7fe46c199050 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.535+0000 7fe470afb640 1 --2- 172.21.15.42:0/1432493781 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe46c107dd0 0x7fe46c198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.535+0000 7fe470afb640 1 --2- 172.21.15.42:0/1432493781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe46c102490 0x7fe46c06b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:53.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.535+0000 7fe470afb640 1 -- 172.21.15.42:0/1432493781 >> 172.21.15.42:0/1432493781 conn(0x7fe46c079160 msgr2=0x7fe46c104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:53.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.535+0000 7fe470afb640 1 -- 172.21.15.42:0/1432493781 shutdown_connections 2024-07-26T06:48:53.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.535+0000 7fe470afb640 1 -- 172.21.15.42:0/1432493781 wait complete. 2024-07-26T06:48:53.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:53 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3535793544' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T06:48:53.601 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:53 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2598547865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T06:48:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:53 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3535793544' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T06:48:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:53 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2598547865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T06:48:53.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:53 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3535793544' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T06:48:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:53 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2598547865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T06:48:54.343 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822716 got 154618822723 for osd.5 2024-07-26T06:48:54.344 DEBUG:teuthology.parallel:result is None 2024-07-26T06:48:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:54 smithi042 ceph-mon[32775]: pgmap v356: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T06:48:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:54 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1432493781' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T06:48:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:54 smithi164 ceph-mon[36970]: pgmap v356: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T06:48:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:54 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1432493781' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T06:48:54.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:54 smithi163 ceph-mon[36873]: pgmap v356: 97 pgs: 97 active+clean; 644 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T06:48:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:54 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1432493781' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T06:48:54.954 INFO:tasks.cephadm.ceph_manager.ceph:need seq 274877906981 got 274877906986 for osd.10 2024-07-26T06:48:54.954 DEBUG:teuthology.parallel:result is None 2024-07-26T06:48:55.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.189+0000 7fb07117e640 1 -- 172.21.15.42:0/1614541542 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb06c106650 msgr2=0x7fb06c1100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:55.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.189+0000 7fb07117e640 1 --2- 172.21.15.42:0/1614541542 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb06c106650 0x7fb06c1100a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb064009ca0 tx=0x7fb06402a8c0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.190+0000 7fb07117e640 1 -- 172.21.15.42:0/1614541542 shutdown_connections 2024-07-26T06:48:55.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.190+0000 7fb07117e640 1 --2- 172.21.15.42:0/1614541542 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb06c106650 0x7fb06c1100a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.190+0000 7fb07117e640 1 --2- 172.21.15.42:0/1614541542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb06c105d00 0x7fb06c106110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.190+0000 7fb07117e640 1 --2- 172.21.15.42:0/1614541542 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb06c101a20 0x7fb06c101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.190+0000 7fb07117e640 1 -- 172.21.15.42:0/1614541542 >> 172.21.15.42:0/1614541542 conn(0x7fb06c079860 msgr2=0x7fb06c079c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:55.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.190+0000 7fb07117e640 1 -- 172.21.15.42:0/1614541542 shutdown_connections 2024-07-26T06:48:55.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.190+0000 7fb07117e640 1 -- 172.21.15.42:0/1614541542 wait complete. 2024-07-26T06:48:55.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.190+0000 7fb07117e640 1 Processor -- start 2024-07-26T06:48:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.191+0000 7fb07117e640 1 -- start start 2024-07-26T06:48:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.191+0000 7fb07117e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb06c101a20 0x7fb06c1029c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.191+0000 7fb07117e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb06c105d00 0x7fb06c102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.191+0000 7fb07117e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb06c106650 0x7fb06c103440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.191+0000 7fb07117e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c113500 con 0x7fb06c101a20 2024-07-26T06:48:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.191+0000 7fb07117e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c113380 con 0x7fb06c105d00 2024-07-26T06:48:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.191+0000 7fb07117e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb06c113680 con 0x7fb06c106650 2024-07-26T06:48:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.191+0000 7fb06ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb06c101a20 0x7fb06c1029c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.191+0000 7fb06ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb06c101a20 0x7fb06c1029c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34466/0 (socket says 172.21.15.42:34466) 2024-07-26T06:48:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.191+0000 7fb06ad9d640 1 -- 172.21.15.42:0/1087794624 learned_addr learned my addr 172.21.15.42:0/1087794624 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.191+0000 7fb06a59c640 1 --2- 172.21.15.42:0/1087794624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb06c105d00 0x7fb06c102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:55.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.191+0000 7fb06b59e640 1 --2- 172.21.15.42:0/1087794624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb06c106650 0x7fb06c103440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:55.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.192+0000 7fb06ad9d640 1 -- 172.21.15.42:0/1087794624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb06c106650 msgr2=0x7fb06c103440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:55.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.192+0000 7fb06ad9d640 1 --2- 172.21.15.42:0/1087794624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb06c106650 0x7fb06c103440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.192+0000 7fb06ad9d640 1 -- 172.21.15.42:0/1087794624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb06c105d00 msgr2=0x7fb06c102f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:55.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.192+0000 7fb06ad9d640 1 --2- 172.21.15.42:0/1087794624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb06c105d00 0x7fb06c102f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.192+0000 7fb06ad9d640 1 -- 172.21.15.42:0/1087794624 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb06c1a8190 con 0x7fb06c101a20 2024-07-26T06:48:55.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.192+0000 7fb06a59c640 1 --2- 172.21.15.42:0/1087794624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb06c105d00 0x7fb06c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:48:55.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.192+0000 7fb06ad9d640 1 --2- 172.21.15.42:0/1087794624 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb06c101a20 0x7fb06c1029c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00d770 tx=0x7fb05c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:55.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.192+0000 7fb053fff640 1 -- 172.21.15.42:0/1087794624 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c017070 con 0x7fb06c101a20 2024-07-26T06:48:55.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.192+0000 7fb053fff640 1 -- 172.21.15.42:0/1087794624 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb05c0054d0 con 0x7fb06c101a20 2024-07-26T06:48:55.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.192+0000 7fb07117e640 1 -- 172.21.15.42:0/1087794624 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb06c1a8480 con 0x7fb06c101a20 2024-07-26T06:48:55.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.192+0000 7fb053fff640 1 -- 172.21.15.42:0/1087794624 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c004c30 con 0x7fb06c101a20 2024-07-26T06:48:55.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.192+0000 7fb07117e640 1 -- 172.21.15.42:0/1087794624 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb06c1a8970 con 0x7fb06c101a20 2024-07-26T06:48:55.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.193+0000 7fb07117e640 1 -- 172.21.15.42:0/1087794624 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb06c101e30 con 0x7fb06c101a20 2024-07-26T06:48:55.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.195+0000 7fb053fff640 1 -- 172.21.15.42:0/1087794624 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fb05c004dd0 con 0x7fb06c101a20 2024-07-26T06:48:55.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.195+0000 7fb053fff640 1 --2- 172.21.15.42:0/1087794624 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb0540b3bd0 0x7fb0540b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:55.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.196+0000 7fb06a59c640 1 --2- 172.21.15.42:0/1087794624 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb0540b3bd0 0x7fb0540b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:55.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.196+0000 7fb053fff640 1 -- 172.21.15.42:0/1087794624 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fb05c0e3b80 con 0x7fb06c101a20 2024-07-26T06:48:55.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.196+0000 7fb06a59c640 1 --2- 172.21.15.42:0/1087794624 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb0540b3bd0 0x7fb0540b60e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb060005fd0 tx=0x7fb060005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:55.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.198+0000 7fb053fff640 1 -- 172.21.15.42:0/1087794624 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb05c0ab1b0 con 0x7fb06c101a20 2024-07-26T06:48:55.379 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647710752 got 300647710756 for osd.11 2024-07-26T06:48:55.379 DEBUG:teuthology.parallel:result is None 2024-07-26T06:48:55.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.403+0000 7fb07117e640 1 -- 172.21.15.42:0/1087794624 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fb06c02e0c0 con 0x7fb06c101a20 2024-07-26T06:48:55.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.403+0000 7fb053fff640 1 -- 172.21.15.42:0/1087794624 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb05c0aaab0 con 0x7fb06c101a20 2024-07-26T06:48:55.404 INFO:teuthology.orchestra.run.smithi042.stdout:227633266739 2024-07-26T06:48:55.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.405+0000 7fb07117e640 1 -- 172.21.15.42:0/1087794624 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb0540b3bd0 msgr2=0x7fb0540b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.405+0000 7fb07117e640 1 --2- 172.21.15.42:0/1087794624 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb0540b3bd0 0x7fb0540b60e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb060005fd0 tx=0x7fb060005850 comp rx=0 tx=0).stop 2024-07-26T06:48:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.405+0000 7fb07117e640 1 -- 172.21.15.42:0/1087794624 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb06c101a20 msgr2=0x7fb06c1029c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.405+0000 7fb07117e640 1 --2- 172.21.15.42:0/1087794624 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb06c101a20 0x7fb06c1029c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00d770 tx=0x7fb05c00dc30 comp rx=0 tx=0).stop 2024-07-26T06:48:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.406+0000 7fb07117e640 1 -- 172.21.15.42:0/1087794624 shutdown_connections 2024-07-26T06:48:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.406+0000 7fb07117e640 1 --2- 172.21.15.42:0/1087794624 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb0540b3bd0 0x7fb0540b60e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.406+0000 7fb07117e640 1 --2- 172.21.15.42:0/1087794624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb06c106650 0x7fb06c103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.406+0000 7fb07117e640 1 --2- 172.21.15.42:0/1087794624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb06c105d00 0x7fb06c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.406+0000 7fb07117e640 1 --2- 172.21.15.42:0/1087794624 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb06c101a20 0x7fb06c1029c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.406+0000 7fb07117e640 1 -- 172.21.15.42:0/1087794624 >> 172.21.15.42:0/1087794624 conn(0x7fb06c079860 msgr2=0x7fb06c106f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:55.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.406+0000 7fb07117e640 1 -- 172.21.15.42:0/1087794624 shutdown_connections 2024-07-26T06:48:55.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:55.406+0000 7fb07117e640 1 -- 172.21.15.42:0/1087794624 wait complete. 2024-07-26T06:48:55.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.935+0000 7f823fea4640 1 -- 172.21.15.42:0/2968504013 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8238102510 msgr2=0x7f8238102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.935+0000 7f823fea4640 1 --2- 172.21.15.42:0/2968504013 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8238102510 0x7f8238102920 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8234009900 tx=0x7f823402a410 comp rx=0 tx=0).stop 2024-07-26T06:48:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.935+0000 7f823fea4640 1 -- 172.21.15.42:0/2968504013 shutdown_connections 2024-07-26T06:48:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.935+0000 7f823fea4640 1 --2- 172.21.15.42:0/2968504013 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8238108750 0x7f823810ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.935+0000 7f823fea4640 1 --2- 172.21.15.42:0/2968504013 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8238107e00 0x7f8238108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.935+0000 7f823fea4640 1 --2- 172.21.15.42:0/2968504013 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8238102510 0x7f8238102920 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.935+0000 7f823fea4640 1 -- 172.21.15.42:0/2968504013 >> 172.21.15.42:0/2968504013 conn(0x7f8238079190 msgr2=0x7f82380795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.936+0000 7f823fea4640 1 -- 172.21.15.42:0/2968504013 shutdown_connections 2024-07-26T06:48:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.936+0000 7f823fea4640 1 -- 172.21.15.42:0/2968504013 wait complete. 2024-07-26T06:48:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.936+0000 7f823fea4640 1 Processor -- start 2024-07-26T06:48:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.936+0000 7f823fea4640 1 -- start start 2024-07-26T06:48:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.936+0000 7f823fea4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8238102510 0x7f823819aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.936+0000 7f823fea4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8238107e00 0x7f823819b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f823fea4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8238108750 0x7f823819b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f823fea4640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f82381123d0 con 0x7f8238107e00 2024-07-26T06:48:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f823fea4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8238112250 con 0x7f8238108750 2024-07-26T06:48:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f823fea4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8238112550 con 0x7f8238102510 2024-07-26T06:48:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f823d43f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8238107e00 0x7f823819b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f823d43f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8238107e00 0x7f823819b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34410/0 (socket says 172.21.15.42:34410) 2024-07-26T06:48:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f823d43f640 1 -- 172.21.15.42:0/3535793544 learned_addr learned my addr 172.21.15.42:0/3535793544 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f823e441640 1 --2- 172.21.15.42:0/3535793544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8238108750 0x7f823819b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f823dc40640 1 --2- 172.21.15.42:0/3535793544 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8238102510 0x7f823819aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:55.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f823d43f640 1 -- 172.21.15.42:0/3535793544 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8238102510 msgr2=0x7f823819aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:55.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f823d43f640 1 --2- 172.21.15.42:0/3535793544 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8238102510 0x7f823819aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f823d43f640 1 -- 172.21.15.42:0/3535793544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8238108750 msgr2=0x7f823819b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:55.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f823d43f640 1 --2- 172.21.15.42:0/3535793544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8238108750 0x7f823819b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f823d43f640 1 -- 172.21.15.42:0/3535793544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82381a5ea0 con 0x7f8238107e00 2024-07-26T06:48:55.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f823d43f640 1 --2- 172.21.15.42:0/3535793544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8238107e00 0x7f823819b1e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f822800c970 tx=0x7f822800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:55.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f822effd640 1 -- 172.21.15.42:0/3535793544 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8228014070 con 0x7f8238107e00 2024-07-26T06:48:55.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f822effd640 1 -- 172.21.15.42:0/3535793544 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8228004440 con 0x7f8238107e00 2024-07-26T06:48:55.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f823fea4640 1 -- 172.21.15.42:0/3535793544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82381a6190 con 0x7f8238107e00 2024-07-26T06:48:55.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.937+0000 7f822effd640 1 -- 172.21.15.42:0/3535793544 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8228003e50 con 0x7f8238107e00 2024-07-26T06:48:55.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.938+0000 7f823fea4640 1 -- 172.21.15.42:0/3535793544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82381a6720 con 0x7f8238107e00 2024-07-26T06:48:55.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.938+0000 7f823fea4640 1 -- 172.21.15.42:0/3535793544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8238103740 con 0x7f8238107e00 2024-07-26T06:48:55.498 INFO:teuthology.orchestra.run.smithi042.stdout:55834574933 2024-07-26T06:48:55.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.940+0000 7f822effd640 1 -- 172.21.15.42:0/3535793544 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f8228017070 con 0x7f8238107e00 2024-07-26T06:48:55.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.941+0000 7f822effd640 1 --2- 172.21.15.42:0/3535793544 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82240b3e20 0x7f82240b6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:55.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.941+0000 7f823dc40640 1 --2- 172.21.15.42:0/3535793544 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82240b3e20 0x7f82240b6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:55.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.941+0000 7f822effd640 1 -- 172.21.15.42:0/3535793544 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f82280e3d80 con 0x7f8238107e00 2024-07-26T06:48:55.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.942+0000 7f823dc40640 1 --2- 172.21.15.42:0/3535793544 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82240b3e20 0x7f82240b6330 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8234000c00 tx=0x7f8234031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:55.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:52.945+0000 7f822effd640 1 -- 172.21.15.42:0/3535793544 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f822801e030 con 0x7f8238107e00 2024-07-26T06:48:55.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.170+0000 7f823fea4640 1 -- 172.21.15.42:0/3535793544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f8238102920 con 0x7f8238107e00 2024-07-26T06:48:55.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.170+0000 7f822effd640 1 -- 172.21.15.42:0/3535793544 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f82280ab430 con 0x7f8238107e00 2024-07-26T06:48:55.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.172+0000 7f823fea4640 1 -- 172.21.15.42:0/3535793544 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82240b3e20 msgr2=0x7f82240b6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:55.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.172+0000 7f823fea4640 1 --2- 172.21.15.42:0/3535793544 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82240b3e20 0x7f82240b6330 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8234000c00 tx=0x7f8234031400 comp rx=0 tx=0).stop 2024-07-26T06:48:55.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.172+0000 7f823fea4640 1 -- 172.21.15.42:0/3535793544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8238107e00 msgr2=0x7f823819b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:55.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.172+0000 7f823fea4640 1 --2- 172.21.15.42:0/3535793544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8238107e00 0x7f823819b1e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f822800c970 tx=0x7f822800ce30 comp rx=0 tx=0).stop 2024-07-26T06:48:55.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.172+0000 7f823fea4640 1 -- 172.21.15.42:0/3535793544 shutdown_connections 2024-07-26T06:48:55.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.172+0000 7f823fea4640 1 --2- 172.21.15.42:0/3535793544 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82240b3e20 0x7f82240b6330 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.172+0000 7f823fea4640 1 --2- 172.21.15.42:0/3535793544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8238108750 0x7f823819b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.172+0000 7f823fea4640 1 --2- 172.21.15.42:0/3535793544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8238107e00 0x7f823819b1e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.172+0000 7f823fea4640 1 --2- 172.21.15.42:0/3535793544 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8238102510 0x7f823819aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:55.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.172+0000 7f823fea4640 1 -- 172.21.15.42:0/3535793544 >> 172.21.15.42:0/3535793544 conn(0x7f8238079190 msgr2=0x7f8238100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:55.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.172+0000 7f823fea4640 1 -- 172.21.15.42:0/3535793544 shutdown_connections 2024-07-26T06:48:55.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:53.172+0000 7f823fea4640 1 -- 172.21.15.42:0/3535793544 wait complete. 2024-07-26T06:48:55.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:55 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1087794624' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T06:48:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:55 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1087794624' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T06:48:55.800 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:55 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1087794624' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T06:48:56.695 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738451 got 34359738457 for osd.0 2024-07-26T06:48:56.695 DEBUG:teuthology.parallel:result is None 2024-07-26T06:48:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:56 smithi164 ceph-mon[36970]: pgmap v357: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T06:48:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:56 smithi163 ceph-mon[36873]: pgmap v357: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T06:48:56.815 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:56 smithi042 ceph-mon[32775]: pgmap v357: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T06:48:57.126 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266734 got 227633266739 for osd.8 2024-07-26T06:48:57.127 DEBUG:teuthology.parallel:result is None 2024-07-26T06:48:57.838 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574929 got 55834574933 for osd.1 2024-07-26T06:48:57.839 DEBUG:teuthology.parallel:result is None 2024-07-26T06:48:58.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:48:58 smithi164 ceph-mon[36970]: pgmap v358: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T06:48:58.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:48:58 smithi163 ceph-mon[36873]: pgmap v358: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T06:48:58.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:48:58 smithi042 ceph-mon[32775]: pgmap v358: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T06:48:59.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.669+0000 7fb768493640 1 -- 172.21.15.42:0/1457186058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7601024b0 msgr2=0x7fb7601028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:59.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.669+0000 7fb768493640 1 --2- 172.21.15.42:0/1457186058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7601024b0 0x7fb7601028c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb75c009900 tx=0x7fb75c02a410 comp rx=0 tx=0).stop 2024-07-26T06:48:59.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.669+0000 7fb768493640 1 -- 172.21.15.42:0/1457186058 shutdown_connections 2024-07-26T06:48:59.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.669+0000 7fb768493640 1 --2- 172.21.15.42:0/1457186058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb760108740 0x7fb76010ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:59.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.669+0000 7fb768493640 1 --2- 172.21.15.42:0/1457186058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb760107df0 0x7fb760108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:59.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.669+0000 7fb768493640 1 --2- 172.21.15.42:0/1457186058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7601024b0 0x7fb7601028c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:59.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.669+0000 7fb768493640 1 -- 172.21.15.42:0/1457186058 >> 172.21.15.42:0/1457186058 conn(0x7fb760079170 msgr2=0x7fb760079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:59.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.669+0000 7fb768493640 1 -- 172.21.15.42:0/1457186058 shutdown_connections 2024-07-26T06:48:59.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.669+0000 7fb768493640 1 -- 172.21.15.42:0/1457186058 wait complete. 2024-07-26T06:48:59.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb768493640 1 Processor -- start 2024-07-26T06:48:59.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb768493640 1 -- start start 2024-07-26T06:48:59.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb768493640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7601024b0 0x7fb76019c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:59.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb768493640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb760107df0 0x7fb76019c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:59.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb768493640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb760108740 0x7fb7601a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:59.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb768493640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb7601134d0 con 0x7fb760107df0 2024-07-26T06:48:59.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb768493640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb760113350 con 0x7fb7601024b0 2024-07-26T06:48:59.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb768493640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb760113650 con 0x7fb760108740 2024-07-26T06:48:59.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb765a2e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb760107df0 0x7fb76019c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:59.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb765a2e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb760107df0 0x7fb76019c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36454/0 (socket says 172.21.15.42:36454) 2024-07-26T06:48:59.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb765a2e640 1 -- 172.21.15.42:0/2966241336 learned_addr learned my addr 172.21.15.42:0/2966241336 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:48:59.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb76622f640 1 --2- 172.21.15.42:0/2966241336 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7601024b0 0x7fb76019c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:59.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb766a30640 1 --2- 172.21.15.42:0/2966241336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb760108740 0x7fb7601a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:59.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb765a2e640 1 -- 172.21.15.42:0/2966241336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb760108740 msgr2=0x7fb7601a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:59.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb765a2e640 1 --2- 172.21.15.42:0/2966241336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb760108740 0x7fb7601a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:59.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb765a2e640 1 -- 172.21.15.42:0/2966241336 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7601024b0 msgr2=0x7fb76019c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:59.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb765a2e640 1 --2- 172.21.15.42:0/2966241336 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7601024b0 0x7fb76019c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:59.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb765a2e640 1 -- 172.21.15.42:0/2966241336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7601a1fd0 con 0x7fb760107df0 2024-07-26T06:48:59.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb76622f640 1 --2- 172.21.15.42:0/2966241336 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7601024b0 0x7fb76019c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:48:59.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.670+0000 7fb766a30640 1 --2- 172.21.15.42:0/2966241336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb760108740 0x7fb7601a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:48:59.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.671+0000 7fb765a2e640 1 --2- 172.21.15.42:0/2966241336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb760107df0 0x7fb76019c6c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb75000d770 tx=0x7fb75000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:59.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.671+0000 7fb7577fe640 1 -- 172.21.15.42:0/2966241336 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb750017070 con 0x7fb760107df0 2024-07-26T06:48:59.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.671+0000 7fb7577fe640 1 -- 172.21.15.42:0/2966241336 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb750009c30 con 0x7fb760107df0 2024-07-26T06:48:59.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.671+0000 7fb768493640 1 -- 172.21.15.42:0/2966241336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7601a22c0 con 0x7fb760107df0 2024-07-26T06:48:59.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.671+0000 7fb7577fe640 1 -- 172.21.15.42:0/2966241336 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75000f620 con 0x7fb760107df0 2024-07-26T06:48:59.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.671+0000 7fb768493640 1 -- 172.21.15.42:0/2966241336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7601a8ae0 con 0x7fb760107df0 2024-07-26T06:48:59.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.672+0000 7fb768493640 1 -- 172.21.15.42:0/2966241336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb760103730 con 0x7fb760107df0 2024-07-26T06:48:59.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.674+0000 7fb7577fe640 1 -- 172.21.15.42:0/2966241336 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fb75001a070 con 0x7fb760107df0 2024-07-26T06:48:59.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.675+0000 7fb7577fe640 1 --2- 172.21.15.42:0/2966241336 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb74c0b3c00 0x7fb74c0b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:48:59.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.675+0000 7fb76622f640 1 --2- 172.21.15.42:0/2966241336 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb74c0b3c00 0x7fb74c0b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:48:59.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.675+0000 7fb7577fe640 1 -- 172.21.15.42:0/2966241336 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fb7500e37f0 con 0x7fb760107df0 2024-07-26T06:48:59.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.676+0000 7fb76622f640 1 --2- 172.21.15.42:0/2966241336 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb74c0b3c00 0x7fb74c0b6110 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb75c0096c0 tx=0x7fb75c031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:48:59.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.678+0000 7fb7577fe640 1 -- 172.21.15.42:0/2966241336 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb7500aadc0 con 0x7fb760107df0 2024-07-26T06:48:59.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.866+0000 7fb768493640 1 -- 172.21.15.42:0/2966241336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb76019d250 con 0x7fb760107df0 2024-07-26T06:48:59.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.866+0000 7fb7577fe640 1 -- 172.21.15.42:0/2966241336 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb76019d250 con 0x7fb760107df0 2024-07-26T06:48:59.867 INFO:teuthology.orchestra.run.smithi042.stdout:107374182477 2024-07-26T06:48:59.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.868+0000 7fb768493640 1 -- 172.21.15.42:0/2966241336 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb74c0b3c00 msgr2=0x7fb74c0b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:59.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.868+0000 7fb768493640 1 --2- 172.21.15.42:0/2966241336 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb74c0b3c00 0x7fb74c0b6110 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb75c0096c0 tx=0x7fb75c031400 comp rx=0 tx=0).stop 2024-07-26T06:48:59.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.868+0000 7fb768493640 1 -- 172.21.15.42:0/2966241336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb760107df0 msgr2=0x7fb76019c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:48:59.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.868+0000 7fb768493640 1 --2- 172.21.15.42:0/2966241336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb760107df0 0x7fb76019c6c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb75000d770 tx=0x7fb75000dc30 comp rx=0 tx=0).stop 2024-07-26T06:48:59.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.868+0000 7fb768493640 1 -- 172.21.15.42:0/2966241336 shutdown_connections 2024-07-26T06:48:59.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.868+0000 7fb768493640 1 --2- 172.21.15.42:0/2966241336 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb74c0b3c00 0x7fb74c0b6110 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:59.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.868+0000 7fb768493640 1 --2- 172.21.15.42:0/2966241336 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb760108740 0x7fb7601a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:59.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.868+0000 7fb768493640 1 --2- 172.21.15.42:0/2966241336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb760107df0 0x7fb76019c6c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:59.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.868+0000 7fb768493640 1 --2- 172.21.15.42:0/2966241336 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb7601024b0 0x7fb76019c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:48:59.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.868+0000 7fb768493640 1 -- 172.21.15.42:0/2966241336 >> 172.21.15.42:0/2966241336 conn(0x7fb760079170 msgr2=0x7fb76010e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:48:59.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.868+0000 7fb768493640 1 -- 172.21.15.42:0/2966241336 shutdown_connections 2024-07-26T06:48:59.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:48:59.868+0000 7fb768493640 1 -- 172.21.15.42:0/2966241336 wait complete. 2024-07-26T06:49:00.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.335+0000 7fc77835d640 1 -- 172.21.15.42:0/3420157735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc770107e30 msgr2=0x7fc770108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:00.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.335+0000 7fc77835d640 1 --2- 172.21.15.42:0/3420157735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc770107e30 0x7fc770108240 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc764009900 tx=0x7fc76402a410 comp rx=0 tx=0).stop 2024-07-26T06:49:00.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.335+0000 7fc77835d640 1 -- 172.21.15.42:0/3420157735 shutdown_connections 2024-07-26T06:49:00.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.335+0000 7fc77835d640 1 --2- 172.21.15.42:0/3420157735 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc770108780 0x7fc77010efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:00.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.335+0000 7fc77835d640 1 --2- 172.21.15.42:0/3420157735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc770107e30 0x7fc770108240 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:00.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.335+0000 7fc77835d640 1 --2- 172.21.15.42:0/3420157735 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc770102bc0 0x7fc770102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:00.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.335+0000 7fc77835d640 1 -- 172.21.15.42:0/3420157735 >> 172.21.15.42:0/3420157735 conn(0x7fc7700799c0 msgr2=0x7fc770079dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:00.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.336+0000 7fc77835d640 1 -- 172.21.15.42:0/3420157735 shutdown_connections 2024-07-26T06:49:00.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.336+0000 7fc77835d640 1 -- 172.21.15.42:0/3420157735 wait complete. 2024-07-26T06:49:00.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.336+0000 7fc77835d640 1 Processor -- start 2024-07-26T06:49:00.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.336+0000 7fc77835d640 1 -- start start 2024-07-26T06:49:00.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.336+0000 7fc77835d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc770102bc0 0x7fc77019ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:00.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.336+0000 7fc77835d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc770107e30 0x7fc77019b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:00.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.336+0000 7fc77835d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc770108780 0x7fc77019b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:00.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.336+0000 7fc77835d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc770112410 con 0x7fc770102bc0 2024-07-26T06:49:00.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.336+0000 7fc77835d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc770112290 con 0x7fc770108780 2024-07-26T06:49:00.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.336+0000 7fc77835d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc770112590 con 0x7fc770107e30 2024-07-26T06:49:00.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.337+0000 7fc7760f9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc770102bc0 0x7fc77019ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:00.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.337+0000 7fc7760f9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc770102bc0 0x7fc77019ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36474/0 (socket says 172.21.15.42:36474) 2024-07-26T06:49:00.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.337+0000 7fc7760f9640 1 -- 172.21.15.42:0/1828645 learned_addr learned my addr 172.21.15.42:0/1828645 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:00.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.337+0000 7fc7768fa640 1 --2- 172.21.15.42:0/1828645 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc770108780 0x7fc77019b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:00.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.337+0000 7fc7758f8640 1 --2- 172.21.15.42:0/1828645 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc770107e30 0x7fc77019b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:00.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.337+0000 7fc7760f9640 1 -- 172.21.15.42:0/1828645 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc770107e30 msgr2=0x7fc77019b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:00.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.337+0000 7fc7760f9640 1 --2- 172.21.15.42:0/1828645 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc770107e30 0x7fc77019b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:00.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.337+0000 7fc7760f9640 1 -- 172.21.15.42:0/1828645 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc770108780 msgr2=0x7fc77019b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:00.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.337+0000 7fc7760f9640 1 --2- 172.21.15.42:0/1828645 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc770108780 0x7fc77019b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:00.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.337+0000 7fc7760f9640 1 -- 172.21.15.42:0/1828645 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7701a5fe0 con 0x7fc770102bc0 2024-07-26T06:49:00.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.337+0000 7fc7760f9640 1 --2- 172.21.15.42:0/1828645 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc770102bc0 0x7fc77019ad20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc76c00d950 tx=0x7fc76c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:00.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.337+0000 7fc7637fe640 1 -- 172.21.15.42:0/1828645 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76c017070 con 0x7fc770102bc0 2024-07-26T06:49:00.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.337+0000 7fc7637fe640 1 -- 172.21.15.42:0/1828645 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc76c009c30 con 0x7fc770102bc0 2024-07-26T06:49:00.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.337+0000 7fc77835d640 1 -- 172.21.15.42:0/1828645 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7701a62d0 con 0x7fc770102bc0 2024-07-26T06:49:00.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.337+0000 7fc7637fe640 1 -- 172.21.15.42:0/1828645 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76c00f620 con 0x7fc770102bc0 2024-07-26T06:49:00.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.337+0000 7fc77835d640 1 -- 172.21.15.42:0/1828645 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7701a67e0 con 0x7fc770102bc0 2024-07-26T06:49:00.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.338+0000 7fc77835d640 1 -- 172.21.15.42:0/1828645 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc770102fd0 con 0x7fc770102bc0 2024-07-26T06:49:00.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.340+0000 7fc7637fe640 1 -- 172.21.15.42:0/1828645 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fc76c010460 con 0x7fc770102bc0 2024-07-26T06:49:00.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.341+0000 7fc7637fe640 1 --2- 172.21.15.42:0/1828645 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc75c0b3bd0 0x7fc75c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:00.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.341+0000 7fc7637fe640 1 -- 172.21.15.42:0/1828645 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fc76c0e3740 con 0x7fc770102bc0 2024-07-26T06:49:00.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.341+0000 7fc7758f8640 1 --2- 172.21.15.42:0/1828645 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc75c0b3bd0 0x7fc75c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:00.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.342+0000 7fc7758f8640 1 --2- 172.21.15.42:0/1828645 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc75c0b3bd0 0x7fc75c0b60e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc764009230 tx=0x7fc764031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:00.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.344+0000 7fc7637fe640 1 -- 172.21.15.42:0/1828645 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc76c009de0 con 0x7fc770102bc0 2024-07-26T06:49:00.528 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:00 smithi042 ceph-mon[32775]: pgmap v359: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-07-26T06:49:00.528 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:00 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2966241336' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T06:49:00.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.531+0000 7fc77835d640 1 -- 172.21.15.42:0/1828645 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fc77002e0c0 con 0x7fc770102bc0 2024-07-26T06:49:00.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.531+0000 7fc7637fe640 1 -- 172.21.15.42:0/1828645 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc76c0aa720 con 0x7fc770102bc0 2024-07-26T06:49:00.532 INFO:teuthology.orchestra.run.smithi042.stdout:201863462970 2024-07-26T06:49:00.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.533+0000 7fc77835d640 1 -- 172.21.15.42:0/1828645 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc75c0b3bd0 msgr2=0x7fc75c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.533+0000 7fc77835d640 1 --2- 172.21.15.42:0/1828645 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc75c0b3bd0 0x7fc75c0b60e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc764009230 tx=0x7fc764031400 comp rx=0 tx=0).stop 2024-07-26T06:49:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.534+0000 7fc77835d640 1 -- 172.21.15.42:0/1828645 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc770102bc0 msgr2=0x7fc77019ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.534+0000 7fc77835d640 1 --2- 172.21.15.42:0/1828645 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc770102bc0 0x7fc77019ad20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc76c00d950 tx=0x7fc76c00de10 comp rx=0 tx=0).stop 2024-07-26T06:49:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.534+0000 7fc77835d640 1 -- 172.21.15.42:0/1828645 shutdown_connections 2024-07-26T06:49:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.534+0000 7fc77835d640 1 --2- 172.21.15.42:0/1828645 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc75c0b3bd0 0x7fc75c0b60e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.534+0000 7fc77835d640 1 --2- 172.21.15.42:0/1828645 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc770108780 0x7fc77019b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.534+0000 7fc77835d640 1 --2- 172.21.15.42:0/1828645 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc770107e30 0x7fc77019b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.534+0000 7fc77835d640 1 --2- 172.21.15.42:0/1828645 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc770102bc0 0x7fc77019ad20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.534+0000 7fc77835d640 1 -- 172.21.15.42:0/1828645 >> 172.21.15.42:0/1828645 conn(0x7fc7700799c0 msgr2=0x7fc770105e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.534+0000 7fc77835d640 1 -- 172.21.15.42:0/1828645 shutdown_connections 2024-07-26T06:49:00.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:00.534+0000 7fc77835d640 1 -- 172.21.15.42:0/1828645 wait complete. 2024-07-26T06:49:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:00 smithi164 ceph-mon[36970]: pgmap v359: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-07-26T06:49:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:00 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2966241336' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T06:49:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:00 smithi163 ceph-mon[36873]: pgmap v359: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-07-26T06:49:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:00 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2966241336' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T06:49:01.548 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182470 got 107374182477 for osd.3 2024-07-26T06:49:01.548 DEBUG:teuthology.parallel:result is None 2024-07-26T06:49:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:01 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1828645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T06:49:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:49:01.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:01 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1828645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T06:49:01.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:49:01.880 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:01 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1828645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T06:49:01.881 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:49:01.911 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462964 got 201863462970 for osd.7 2024-07-26T06:49:01.911 DEBUG:teuthology.parallel:result is None 2024-07-26T06:49:02.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:02 smithi042 ceph-mon[32775]: pgmap v360: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-07-26T06:49:02.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:02 smithi163 ceph-mon[36873]: pgmap v360: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-07-26T06:49:02.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.893+0000 7f39a0100640 1 -- 172.21.15.42:0/1868050346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3998107df0 msgr2=0x7f3998108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:02.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.893+0000 7f39a0100640 1 --2- 172.21.15.42:0/1868050346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3998107df0 0x7f3998108200 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3988009900 tx=0x7f398802a410 comp rx=0 tx=0).stop 2024-07-26T06:49:02.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.893+0000 7f39a0100640 1 -- 172.21.15.42:0/1868050346 shutdown_connections 2024-07-26T06:49:02.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.893+0000 7f39a0100640 1 --2- 172.21.15.42:0/1868050346 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3998108740 0x7f399810ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:02.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.893+0000 7f39a0100640 1 --2- 172.21.15.42:0/1868050346 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3998107df0 0x7f3998108200 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:02.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.893+0000 7f39a0100640 1 --2- 172.21.15.42:0/1868050346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39981024b0 0x7f39981028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:02.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.893+0000 7f39a0100640 1 -- 172.21.15.42:0/1868050346 >> 172.21.15.42:0/1868050346 conn(0x7f3998079170 msgr2=0x7f3998079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:02.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.893+0000 7f39a0100640 1 -- 172.21.15.42:0/1868050346 shutdown_connections 2024-07-26T06:49:02.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.893+0000 7f39a0100640 1 -- 172.21.15.42:0/1868050346 wait complete. 2024-07-26T06:49:02.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.894+0000 7f39a0100640 1 Processor -- start 2024-07-26T06:49:02.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.894+0000 7f39a0100640 1 -- start start 2024-07-26T06:49:02.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.894+0000 7f39a0100640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39981024b0 0x7f3998199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:02.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.894+0000 7f39a0100640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3998107df0 0x7f399819a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:02.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.894+0000 7f39a0100640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3998108740 0x7f399819f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:02.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.894+0000 7f39a0100640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3998112390 con 0x7f39981024b0 2024-07-26T06:49:02.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.894+0000 7f39a0100640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3998112210 con 0x7f3998108740 2024-07-26T06:49:02.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.894+0000 7f39a0100640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3998112510 con 0x7f3998107df0 2024-07-26T06:49:02.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.894+0000 7f399de9c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39981024b0 0x7f3998199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:02.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.894+0000 7f399de9c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39981024b0 0x7f3998199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36498/0 (socket says 172.21.15.42:36498) 2024-07-26T06:49:02.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.894+0000 7f399de9c640 1 -- 172.21.15.42:0/2120422766 learned_addr learned my addr 172.21.15.42:0/2120422766 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:02.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.894+0000 7f399e69d640 1 --2- 172.21.15.42:0/2120422766 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3998108740 0x7f399819f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:02.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.894+0000 7f399d69b640 1 --2- 172.21.15.42:0/2120422766 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3998107df0 0x7f399819a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:02.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.895+0000 7f399de9c640 1 -- 172.21.15.42:0/2120422766 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3998107df0 msgr2=0x7f399819a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:02.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.895+0000 7f399de9c640 1 --2- 172.21.15.42:0/2120422766 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3998107df0 0x7f399819a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:02.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.895+0000 7f399de9c640 1 -- 172.21.15.42:0/2120422766 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3998108740 msgr2=0x7f399819f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:02.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.895+0000 7f399de9c640 1 --2- 172.21.15.42:0/2120422766 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3998108740 0x7f399819f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:02.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.895+0000 7f399de9c640 1 -- 172.21.15.42:0/2120422766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399819fe20 con 0x7f39981024b0 2024-07-26T06:49:02.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.895+0000 7f399d69b640 1 --2- 172.21.15.42:0/2120422766 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3998107df0 0x7f399819a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:49:02.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.895+0000 7f399de9c640 1 --2- 172.21.15.42:0/2120422766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39981024b0 0x7f3998199eb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f399400e9f0 tx=0x7f399400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:02.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.895+0000 7f398effd640 1 -- 172.21.15.42:0/2120422766 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399400cd80 con 0x7f39981024b0 2024-07-26T06:49:02.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.895+0000 7f398effd640 1 -- 172.21.15.42:0/2120422766 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39940054d0 con 0x7f39981024b0 2024-07-26T06:49:02.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.895+0000 7f39a0100640 1 -- 172.21.15.42:0/2120422766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39981a0110 con 0x7f39981024b0 2024-07-26T06:49:02.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.895+0000 7f39a0100640 1 -- 172.21.15.42:0/2120422766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39981a6870 con 0x7f39981024b0 2024-07-26T06:49:02.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.895+0000 7f398effd640 1 -- 172.21.15.42:0/2120422766 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3994010690 con 0x7f39981024b0 2024-07-26T06:49:02.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.896+0000 7f39a0100640 1 -- 172.21.15.42:0/2120422766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3998103730 con 0x7f39981024b0 2024-07-26T06:49:02.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.899+0000 7f398effd640 1 -- 172.21.15.42:0/2120422766 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f3994004e30 con 0x7f39981024b0 2024-07-26T06:49:02.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.899+0000 7f398effd640 1 --2- 172.21.15.42:0/2120422766 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f39840b3c50 0x7f39840b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:02.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.900+0000 7f399d69b640 1 --2- 172.21.15.42:0/2120422766 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f39840b3c50 0x7f39840b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:02.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.900+0000 7f398effd640 1 -- 172.21.15.42:0/2120422766 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f39940e4250 con 0x7f39981024b0 2024-07-26T06:49:02.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.900+0000 7f399d69b640 1 --2- 172.21.15.42:0/2120422766 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f39840b3c50 0x7f39840b6160 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3988000c00 tx=0x7f3988032cc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:02.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:02.903+0000 7f398effd640 1 -- 172.21.15.42:0/2120422766 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3994055600 con 0x7f39981024b0 2024-07-26T06:49:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:02 smithi164 ceph-mon[36970]: pgmap v360: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-07-26T06:49:03.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:03.093+0000 7f39a0100640 1 -- 172.21.15.42:0/2120422766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f399819afe0 con 0x7f39981024b0 2024-07-26T06:49:03.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:03.093+0000 7f398effd640 1 -- 172.21.15.42:0/2120422766 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f399819afe0 con 0x7f39981024b0 2024-07-26T06:49:03.094 INFO:teuthology.orchestra.run.smithi042.stdout:253403070512 2024-07-26T06:49:03.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:03.095+0000 7f39a0100640 1 -- 172.21.15.42:0/2120422766 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f39840b3c50 msgr2=0x7f39840b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:03.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:03.095+0000 7f39a0100640 1 --2- 172.21.15.42:0/2120422766 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f39840b3c50 0x7f39840b6160 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3988000c00 tx=0x7f3988032cc0 comp rx=0 tx=0).stop 2024-07-26T06:49:03.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:03.095+0000 7f39a0100640 1 -- 172.21.15.42:0/2120422766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39981024b0 msgr2=0x7f3998199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:03.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:03.095+0000 7f39a0100640 1 --2- 172.21.15.42:0/2120422766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39981024b0 0x7f3998199eb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f399400e9f0 tx=0x7f399400edf0 comp rx=0 tx=0).stop 2024-07-26T06:49:03.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:03.096+0000 7f39a0100640 1 -- 172.21.15.42:0/2120422766 shutdown_connections 2024-07-26T06:49:03.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:03.096+0000 7f39a0100640 1 --2- 172.21.15.42:0/2120422766 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f39840b3c50 0x7f39840b6160 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:03.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:03.096+0000 7f39a0100640 1 --2- 172.21.15.42:0/2120422766 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3998108740 0x7f399819f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:03.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:03.096+0000 7f39a0100640 1 --2- 172.21.15.42:0/2120422766 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3998107df0 0x7f399819a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:03.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:03.096+0000 7f39a0100640 1 --2- 172.21.15.42:0/2120422766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39981024b0 0x7f3998199eb0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:03.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:03.096+0000 7f39a0100640 1 -- 172.21.15.42:0/2120422766 >> 172.21.15.42:0/2120422766 conn(0x7f3998079170 msgr2=0x7f39981013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:03.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:03.096+0000 7f39a0100640 1 -- 172.21.15.42:0/2120422766 shutdown_connections 2024-07-26T06:49:03.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:03.096+0000 7f39a0100640 1 -- 172.21.15.42:0/2120422766 wait complete. 2024-07-26T06:49:03.913 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070506 got 253403070512 for osd.9 2024-07-26T06:49:03.913 DEBUG:teuthology.parallel:result is None 2024-07-26T06:49:03.914 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-07-26T06:49:03.914 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph pg dump --format=json 2024-07-26T06:49:04.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:03 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2120422766' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T06:49:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:03 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2120422766' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T06:49:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:03 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2120422766' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T06:49:04.322 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:05.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:04 smithi042 ceph-mon[32775]: pgmap v361: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T06:49:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:04 smithi164 ceph-mon[36970]: pgmap v361: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T06:49:05.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:04 smithi163 ceph-mon[36873]: pgmap v361: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T06:49:05.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.976+0000 7f45d5ad1640 1 -- 172.21.15.42:0/2204562083 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45d0108740 msgr2=0x7f45d010ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:05.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.976+0000 7f45d5ad1640 1 --2- 172.21.15.42:0/2204562083 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45d0108740 0x7f45d010ef70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009ca0 tx=0x7f45c802a8c0 comp rx=0 tx=0).stop 2024-07-26T06:49:05.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.976+0000 7f45d5ad1640 1 -- 172.21.15.42:0/2204562083 shutdown_connections 2024-07-26T06:49:05.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.976+0000 7f45d5ad1640 1 --2- 172.21.15.42:0/2204562083 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45d0108740 0x7f45d010ef70 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:05.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.976+0000 7f45d5ad1640 1 --2- 172.21.15.42:0/2204562083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45d0107df0 0x7f45d0108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:05.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.976+0000 7f45d5ad1640 1 --2- 172.21.15.42:0/2204562083 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d01024b0 0x7f45d01028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:05.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.976+0000 7f45d5ad1640 1 -- 172.21.15.42:0/2204562083 >> 172.21.15.42:0/2204562083 conn(0x7f45d0079170 msgr2=0x7f45d0079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:05.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.976+0000 7f45d5ad1640 1 -- 172.21.15.42:0/2204562083 shutdown_connections 2024-07-26T06:49:05.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.976+0000 7f45d5ad1640 1 -- 172.21.15.42:0/2204562083 wait complete. 2024-07-26T06:49:05.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.977+0000 7f45d5ad1640 1 Processor -- start 2024-07-26T06:49:05.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.977+0000 7f45d5ad1640 1 -- start start 2024-07-26T06:49:05.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.977+0000 7f45d5ad1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45d01024b0 0x7f45d0199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:05.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.977+0000 7f45d5ad1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d0107df0 0x7f45d019a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:05.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.977+0000 7f45d5ad1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45d0108740 0x7f45d019f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:05.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.977+0000 7f45d5ad1640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01134d0 con 0x7f45d01024b0 2024-07-26T06:49:05.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.977+0000 7f45d5ad1640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f45d0113350 con 0x7f45d0107df0 2024-07-26T06:49:05.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.977+0000 7f45d5ad1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f45d0113650 con 0x7f45d0108740 2024-07-26T06:49:05.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.977+0000 7f45cf7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45d01024b0 0x7f45d0199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:05.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.977+0000 7f45cf7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45d01024b0 0x7f45d0199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36522/0 (socket says 172.21.15.42:36522) 2024-07-26T06:49:05.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.977+0000 7f45cf7fe640 1 -- 172.21.15.42:0/1053419437 learned_addr learned my addr 172.21.15.42:0/1053419437 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:05.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.978+0000 7f45ceffd640 1 --2- 172.21.15.42:0/1053419437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d0107df0 0x7f45d019a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:05.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.978+0000 7f45cffff640 1 --2- 172.21.15.42:0/1053419437 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45d0108740 0x7f45d019f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:05.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.978+0000 7f45cf7fe640 1 -- 172.21.15.42:0/1053419437 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45d0108740 msgr2=0x7f45d019f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:05.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.978+0000 7f45cf7fe640 1 --2- 172.21.15.42:0/1053419437 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45d0108740 0x7f45d019f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:05.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.978+0000 7f45cf7fe640 1 -- 172.21.15.42:0/1053419437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d0107df0 msgr2=0x7f45d019a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:05.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.978+0000 7f45cf7fe640 1 --2- 172.21.15.42:0/1053419437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d0107df0 0x7f45d019a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:05.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.978+0000 7f45cf7fe640 1 -- 172.21.15.42:0/1053419437 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d019fd10 con 0x7f45d01024b0 2024-07-26T06:49:05.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.978+0000 7f45ceffd640 1 --2- 172.21.15.42:0/1053419437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d0107df0 0x7f45d019a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:49:05.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.978+0000 7f45cf7fe640 1 --2- 172.21.15.42:0/1053419437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45d01024b0 0x7f45d0199e60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f45c000c920 tx=0x7f45c000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:05.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.978+0000 7f45ccff9640 1 -- 172.21.15.42:0/1053419437 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c0007c70 con 0x7f45d01024b0 2024-07-26T06:49:05.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.978+0000 7f45d5ad1640 1 -- 172.21.15.42:0/1053419437 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d01a0000 con 0x7f45d01024b0 2024-07-26T06:49:05.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.978+0000 7f45ccff9640 1 -- 172.21.15.42:0/1053419437 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45c0007e10 con 0x7f45d01024b0 2024-07-26T06:49:05.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.978+0000 7f45ccff9640 1 -- 172.21.15.42:0/1053419437 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c0018820 con 0x7f45d01024b0 2024-07-26T06:49:05.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.978+0000 7f45d5ad1640 1 -- 172.21.15.42:0/1053419437 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d01a6820 con 0x7f45d01024b0 2024-07-26T06:49:05.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.979+0000 7f45d5ad1640 1 -- 172.21.15.42:0/1053419437 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45d0103730 con 0x7f45d01024b0 2024-07-26T06:49:05.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.981+0000 7f45ccff9640 1 -- 172.21.15.42:0/1053419437 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f45c000f450 con 0x7f45d01024b0 2024-07-26T06:49:05.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.982+0000 7f45ccff9640 1 --2- 172.21.15.42:0/1053419437 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f45b80b3c00 0x7f45b80b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.982+0000 7f45ceffd640 1 --2- 172.21.15.42:0/1053419437 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f45b80b3c00 0x7f45b80b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.982+0000 7f45ccff9640 1 -- 172.21.15.42:0/1053419437 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f45c00e4050 con 0x7f45d01024b0 2024-07-26T06:49:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.983+0000 7f45ceffd640 1 --2- 172.21.15.42:0/1053419437 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f45b80b3c00 0x7f45b80b6110 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f45c40058c0 tx=0x7f45c4005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:05.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:05.986+0000 7f45ccff9640 1 -- 172.21.15.42:0/1053419437 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f45c0014030 con 0x7f45d01024b0 2024-07-26T06:49:06.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:06.183+0000 7f45d5ad1640 1 -- 172.21.15.42:0/1053419437 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f45d0063230 con 0x7f45b80b3c00 2024-07-26T06:49:06.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:06.188+0000 7f45ccff9640 1 -- 172.21.15.42:0/1053419437 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259109 (secure 0 0 0) 0x7f45d0063230 con 0x7f45b80b3c00 2024-07-26T06:49:06.191 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:49:06.193 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-07-26T06:49:06.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:06.191+0000 7f45d5ad1640 1 -- 172.21.15.42:0/1053419437 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f45b80b3c00 msgr2=0x7f45b80b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:06.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:06.191+0000 7f45d5ad1640 1 --2- 172.21.15.42:0/1053419437 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f45b80b3c00 0x7f45b80b6110 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f45c40058c0 tx=0x7f45c4005850 comp rx=0 tx=0).stop 2024-07-26T06:49:06.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:06.191+0000 7f45d5ad1640 1 -- 172.21.15.42:0/1053419437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45d01024b0 msgr2=0x7f45d0199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:06.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:06.191+0000 7f45d5ad1640 1 --2- 172.21.15.42:0/1053419437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45d01024b0 0x7f45d0199e60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f45c000c920 tx=0x7f45c000cde0 comp rx=0 tx=0).stop 2024-07-26T06:49:06.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:06.191+0000 7f45d5ad1640 1 -- 172.21.15.42:0/1053419437 shutdown_connections 2024-07-26T06:49:06.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:06.191+0000 7f45d5ad1640 1 --2- 172.21.15.42:0/1053419437 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f45b80b3c00 0x7f45b80b6110 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:06.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:06.191+0000 7f45d5ad1640 1 --2- 172.21.15.42:0/1053419437 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45d0108740 0x7f45d019f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:06.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:06.191+0000 7f45d5ad1640 1 --2- 172.21.15.42:0/1053419437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45d0107df0 0x7f45d019a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:06.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:06.191+0000 7f45d5ad1640 1 --2- 172.21.15.42:0/1053419437 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45d01024b0 0x7f45d0199e60 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:06.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:06.191+0000 7f45d5ad1640 1 -- 172.21.15.42:0/1053419437 >> 172.21.15.42:0/1053419437 conn(0x7f45d0079170 msgr2=0x7f45d0104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:06.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:06.191+0000 7f45d5ad1640 1 -- 172.21.15.42:0/1053419437 shutdown_connections 2024-07-26T06:49:06.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:06.191+0000 7f45d5ad1640 1 -- 172.21.15.42:0/1053419437 wait complete. 2024-07-26T06:49:06.977 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-07-26T06:49:05.348753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660163,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1113,"num_read_kb":661,"num_write":378,"num_write_kb":2295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2953474,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":811,"ondisk_log_size":811,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384912,"kb_used_data":10800,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124556912,"statfs":{"total":1151940427776,"available":1151546277888,"internally_reserved":0,"allocated":11059200,"data_stored":7072521,"data_compressed":177858,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":132,"num_read_kb":72,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003827"},"pg_stats":[{"pgid":"3.3f","version":"80'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335928+0000","last_change":"2024-07-26T06:47:05.026186+0000","last_active":"2024-07-26T06:48:51.335928+0000","last_peered":"2024-07-26T06:48:51.335928+0000","last_clean":"2024-07-26T06:48:51.335928+0000","last_became_active":"2024-07-26T06:47:05.025589+0000","last_became_peered":"2024-07-26T06:47:05.025589+0000","last_unstale":"2024-07-26T06:48:51.335928+0000","last_undegraded":"2024-07-26T06:48:51.335928+0000","last_fullsized":"2024-07-26T06:48:51.335928+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:48:23.100966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"80'4","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.325860+0000","last_change":"2024-07-26T06:47:05.033211+0000","last_active":"2024-07-26T06:48:51.325860+0000","last_peered":"2024-07-26T06:48:51.325860+0000","last_clean":"2024-07-26T06:48:51.325860+0000","last_became_active":"2024-07-26T06:47:05.032630+0000","last_became_peered":"2024-07-26T06:47:05.032630+0000","last_unstale":"2024-07-26T06:48:51.325860+0000","last_undegraded":"2024-07-26T06:48:51.325860+0000","last_fullsized":"2024-07-26T06:48:51.325860+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:03:19.612963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"80'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335834+0000","last_change":"2024-07-26T06:47:05.034767+0000","last_active":"2024-07-26T06:48:51.335834+0000","last_peered":"2024-07-26T06:48:51.335834+0000","last_clean":"2024-07-26T06:48:51.335834+0000","last_became_active":"2024-07-26T06:47:05.034376+0000","last_became_peered":"2024-07-26T06:47:05.034376+0000","last_unstale":"2024-07-26T06:48:51.335834+0000","last_undegraded":"2024-07-26T06:48:51.335834+0000","last_fullsized":"2024-07-26T06:48:51.335834+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:55:55.729930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"80'8","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326864+0000","last_change":"2024-07-26T06:47:05.030995+0000","last_active":"2024-07-26T06:48:51.326864+0000","last_peered":"2024-07-26T06:48:51.326864+0000","last_clean":"2024-07-26T06:48:51.326864+0000","last_became_active":"2024-07-26T06:47:05.025423+0000","last_became_peered":"2024-07-26T06:47:05.025423+0000","last_unstale":"2024-07-26T06:48:51.326864+0000","last_undegraded":"2024-07-26T06:48:51.326864+0000","last_fullsized":"2024-07-26T06:48:51.326864+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:59:39.511383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"80'4","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.316860+0000","last_change":"2024-07-26T06:47:05.035359+0000","last_active":"2024-07-26T06:48:51.316860+0000","last_peered":"2024-07-26T06:48:51.316860+0000","last_clean":"2024-07-26T06:48:51.316860+0000","last_became_active":"2024-07-26T06:47:05.034836+0000","last_became_peered":"2024-07-26T06:47:05.034836+0000","last_unstale":"2024-07-26T06:48:51.316860+0000","last_undegraded":"2024-07-26T06:48:51.316860+0000","last_fullsized":"2024-07-26T06:48:51.316860+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:54.482774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"80'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326138+0000","last_change":"2024-07-26T06:47:05.023660+0000","last_active":"2024-07-26T06:48:51.326138+0000","last_peered":"2024-07-26T06:48:51.326138+0000","last_clean":"2024-07-26T06:48:51.326138+0000","last_became_active":"2024-07-26T06:47:05.022446+0000","last_became_peered":"2024-07-26T06:47:05.022446+0000","last_unstale":"2024-07-26T06:48:51.326138+0000","last_undegraded":"2024-07-26T06:48:51.326138+0000","last_fullsized":"2024-07-26T06:48:51.326138+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:35:00.519226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"80'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339395+0000","last_change":"2024-07-26T06:47:05.025132+0000","last_active":"2024-07-26T06:48:51.339395+0000","last_peered":"2024-07-26T06:48:51.339395+0000","last_clean":"2024-07-26T06:48:51.339395+0000","last_became_active":"2024-07-26T06:47:05.024770+0000","last_became_peered":"2024-07-26T06:47:05.024770+0000","last_unstale":"2024-07-26T06:48:51.339395+0000","last_undegraded":"2024-07-26T06:48:51.339395+0000","last_fullsized":"2024-07-26T06:48:51.339395+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:21:39.920152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"80'9","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.332397+0000","last_change":"2024-07-26T06:47:05.022613+0000","last_active":"2024-07-26T06:48:51.332397+0000","last_peered":"2024-07-26T06:48:51.332397+0000","last_clean":"2024-07-26T06:48:51.332397+0000","last_became_active":"2024-07-26T06:47:05.022202+0000","last_became_peered":"2024-07-26T06:47:05.022202+0000","last_unstale":"2024-07-26T06:48:51.332397+0000","last_undegraded":"2024-07-26T06:48:51.332397+0000","last_fullsized":"2024-07-26T06:48:51.332397+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:07:23.980843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"80'11","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.336833+0000","last_change":"2024-07-26T06:47:05.026352+0000","last_active":"2024-07-26T06:48:51.336833+0000","last_peered":"2024-07-26T06:48:51.336833+0000","last_clean":"2024-07-26T06:48:51.336833+0000","last_became_active":"2024-07-26T06:47:05.025969+0000","last_became_peered":"2024-07-26T06:47:05.025969+0000","last_unstale":"2024-07-26T06:48:51.336833+0000","last_undegraded":"2024-07-26T06:48:51.336833+0000","last_fullsized":"2024-07-26T06:48:51.336833+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:27.062013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"80'16","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328270+0000","last_change":"2024-07-26T06:47:23.100422+0000","last_active":"2024-07-26T06:48:51.328270+0000","last_peered":"2024-07-26T06:48:51.328270+0000","last_clean":"2024-07-26T06:48:51.328270+0000","last_became_active":"2024-07-26T06:47:23.100023+0000","last_became_peered":"2024-07-26T06:47:23.100023+0000","last_unstale":"2024-07-26T06:48:51.328270+0000","last_undegraded":"2024-07-26T06:48:51.328270+0000","last_fullsized":"2024-07-26T06:48:51.328270+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:28.158112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"80'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338442+0000","last_change":"2024-07-26T06:47:05.037142+0000","last_active":"2024-07-26T06:48:51.338442+0000","last_peered":"2024-07-26T06:48:51.338442+0000","last_clean":"2024-07-26T06:48:51.338442+0000","last_became_active":"2024-07-26T06:47:05.036745+0000","last_became_peered":"2024-07-26T06:47:05.036745+0000","last_unstale":"2024-07-26T06:48:51.338442+0000","last_undegraded":"2024-07-26T06:48:51.338442+0000","last_fullsized":"2024-07-26T06:48:51.338442+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:31:10.034560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"80'7","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.315576+0000","last_change":"2024-07-26T06:47:05.030640+0000","last_active":"2024-07-26T06:48:51.315576+0000","last_peered":"2024-07-26T06:48:51.315576+0000","last_clean":"2024-07-26T06:48:51.315576+0000","last_became_active":"2024-07-26T06:47:05.023583+0000","last_became_peered":"2024-07-26T06:47:05.023583+0000","last_unstale":"2024-07-26T06:48:51.315576+0000","last_undegraded":"2024-07-26T06:48:51.315576+0000","last_fullsized":"2024-07-26T06:48:51.315576+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:59:12.312930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"80'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333051+0000","last_change":"2024-07-26T06:47:05.021768+0000","last_active":"2024-07-26T06:48:51.333051+0000","last_peered":"2024-07-26T06:48:51.333051+0000","last_clean":"2024-07-26T06:48:51.333051+0000","last_became_active":"2024-07-26T06:47:05.021410+0000","last_became_peered":"2024-07-26T06:47:05.021410+0000","last_unstale":"2024-07-26T06:48:51.333051+0000","last_undegraded":"2024-07-26T06:48:51.333051+0000","last_fullsized":"2024-07-26T06:48:51.333051+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:05.576122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"80'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.324272+0000","last_change":"2024-07-26T06:47:05.035814+0000","last_active":"2024-07-26T06:48:51.324272+0000","last_peered":"2024-07-26T06:48:51.324272+0000","last_clean":"2024-07-26T06:48:51.324272+0000","last_became_active":"2024-07-26T06:47:05.035451+0000","last_became_peered":"2024-07-26T06:47:05.035451+0000","last_unstale":"2024-07-26T06:48:51.324272+0000","last_undegraded":"2024-07-26T06:48:51.324272+0000","last_fullsized":"2024-07-26T06:48:51.324272+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:33.533726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"80'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.331253+0000","last_change":"2024-07-26T06:47:05.035720+0000","last_active":"2024-07-26T06:48:51.331253+0000","last_peered":"2024-07-26T06:48:51.331253+0000","last_clean":"2024-07-26T06:48:51.331253+0000","last_became_active":"2024-07-26T06:47:05.035359+0000","last_became_peered":"2024-07-26T06:47:05.035359+0000","last_unstale":"2024-07-26T06:48:51.331253+0000","last_undegraded":"2024-07-26T06:48:51.331253+0000","last_fullsized":"2024-07-26T06:48:51.331253+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:38.561268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"80'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338926+0000","last_change":"2024-07-26T06:47:05.027109+0000","last_active":"2024-07-26T06:48:51.338926+0000","last_peered":"2024-07-26T06:48:51.338926+0000","last_clean":"2024-07-26T06:48:51.338926+0000","last_became_active":"2024-07-26T06:47:05.026694+0000","last_became_peered":"2024-07-26T06:47:05.026694+0000","last_unstale":"2024-07-26T06:48:51.338926+0000","last_undegraded":"2024-07-26T06:48:51.338926+0000","last_fullsized":"2024-07-26T06:48:51.338926+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:25:01.215646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"80'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337344+0000","last_change":"2024-07-26T06:47:05.028212+0000","last_active":"2024-07-26T06:48:51.337344+0000","last_peered":"2024-07-26T06:48:51.337344+0000","last_clean":"2024-07-26T06:48:51.337344+0000","last_became_active":"2024-07-26T06:47:05.027598+0000","last_became_peered":"2024-07-26T06:47:05.027598+0000","last_unstale":"2024-07-26T06:48:51.337344+0000","last_undegraded":"2024-07-26T06:48:51.337344+0000","last_fullsized":"2024-07-26T06:48:51.337344+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:55:31.536194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"80'10","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335959+0000","last_change":"2024-07-26T06:47:05.027724+0000","last_active":"2024-07-26T06:48:51.335959+0000","last_peered":"2024-07-26T06:48:51.335959+0000","last_clean":"2024-07-26T06:48:51.335959+0000","last_became_active":"2024-07-26T06:47:05.027348+0000","last_became_peered":"2024-07-26T06:47:05.027348+0000","last_unstale":"2024-07-26T06:48:51.335959+0000","last_undegraded":"2024-07-26T06:48:51.335959+0000","last_fullsized":"2024-07-26T06:48:51.335959+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:04.180467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"80'7","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.327359+0000","last_change":"2024-07-26T06:47:05.025474+0000","last_active":"2024-07-26T06:48:51.327359+0000","last_peered":"2024-07-26T06:48:51.327359+0000","last_clean":"2024-07-26T06:48:51.327359+0000","last_became_active":"2024-07-26T06:47:05.025016+0000","last_became_peered":"2024-07-26T06:47:05.025016+0000","last_unstale":"2024-07-26T06:48:51.327359+0000","last_undegraded":"2024-07-26T06:48:51.327359+0000","last_fullsized":"2024-07-26T06:48:51.327359+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:40:03.034532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"80'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.330259+0000","last_change":"2024-07-26T06:47:05.027533+0000","last_active":"2024-07-26T06:48:51.330259+0000","last_peered":"2024-07-26T06:48:51.330259+0000","last_clean":"2024-07-26T06:48:51.330259+0000","last_became_active":"2024-07-26T06:47:05.027143+0000","last_became_peered":"2024-07-26T06:47:05.027143+0000","last_unstale":"2024-07-26T06:48:51.330259+0000","last_undegraded":"2024-07-26T06:48:51.330259+0000","last_fullsized":"2024-07-26T06:48:51.330259+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:13.406430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"80'11","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338113+0000","last_change":"2024-07-26T06:47:05.025962+0000","last_active":"2024-07-26T06:48:51.338113+0000","last_peered":"2024-07-26T06:48:51.338113+0000","last_clean":"2024-07-26T06:48:51.338113+0000","last_became_active":"2024-07-26T06:47:05.025621+0000","last_became_peered":"2024-07-26T06:47:05.025621+0000","last_unstale":"2024-07-26T06:48:51.338113+0000","last_undegraded":"2024-07-26T06:48:51.338113+0000","last_fullsized":"2024-07-26T06:48:51.338113+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:56:43.669247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"80'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337312+0000","last_change":"2024-07-26T06:47:05.026252+0000","last_active":"2024-07-26T06:48:51.337312+0000","last_peered":"2024-07-26T06:48:51.337312+0000","last_clean":"2024-07-26T06:48:51.337312+0000","last_became_active":"2024-07-26T06:47:05.025875+0000","last_became_peered":"2024-07-26T06:47:05.025875+0000","last_unstale":"2024-07-26T06:48:51.337312+0000","last_undegraded":"2024-07-26T06:48:51.337312+0000","last_fullsized":"2024-07-26T06:48:51.337312+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:27:32.955862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"80'8","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326328+0000","last_change":"2024-07-26T06:47:05.022513+0000","last_active":"2024-07-26T06:48:51.326328+0000","last_peered":"2024-07-26T06:48:51.326328+0000","last_clean":"2024-07-26T06:48:51.326328+0000","last_became_active":"2024-07-26T06:47:05.022159+0000","last_became_peered":"2024-07-26T06:47:05.022159+0000","last_unstale":"2024-07-26T06:48:51.326328+0000","last_undegraded":"2024-07-26T06:48:51.326328+0000","last_fullsized":"2024-07-26T06:48:51.326328+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:53:06.402985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"80'6","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.325210+0000","last_change":"2024-07-26T06:47:23.097604+0000","last_active":"2024-07-26T06:48:51.325210+0000","last_peered":"2024-07-26T06:48:51.325210+0000","last_clean":"2024-07-26T06:48:51.325210+0000","last_became_active":"2024-07-26T06:47:23.097170+0000","last_became_peered":"2024-07-26T06:47:23.097170+0000","last_unstale":"2024-07-26T06:48:51.325210+0000","last_undegraded":"2024-07-26T06:48:51.325210+0000","last_fullsized":"2024-07-26T06:48:51.325210+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:38:24.530396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"80'2","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.320538+0000","last_change":"2024-07-26T06:47:05.026349+0000","last_active":"2024-07-26T06:48:51.320538+0000","last_peered":"2024-07-26T06:48:51.320538+0000","last_clean":"2024-07-26T06:48:51.320538+0000","last_became_active":"2024-07-26T06:47:05.026016+0000","last_became_peered":"2024-07-26T06:47:05.026016+0000","last_unstale":"2024-07-26T06:48:51.320538+0000","last_undegraded":"2024-07-26T06:48:51.320538+0000","last_fullsized":"2024-07-26T06:48:51.320538+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:19:02.264183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"80'15","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.323233+0000","last_change":"2024-07-26T06:47:05.032439+0000","last_active":"2024-07-26T06:48:51.323233+0000","last_peered":"2024-07-26T06:48:51.323233+0000","last_clean":"2024-07-26T06:48:51.323233+0000","last_became_active":"2024-07-26T06:47:05.032105+0000","last_became_peered":"2024-07-26T06:47:05.032105+0000","last_unstale":"2024-07-26T06:48:51.323233+0000","last_undegraded":"2024-07-26T06:48:51.323233+0000","last_fullsized":"2024-07-26T06:48:51.323233+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:21:50.423939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"80'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.319169+0000","last_change":"2024-07-26T06:47:05.026927+0000","last_active":"2024-07-26T06:48:51.319169+0000","last_peered":"2024-07-26T06:48:51.319169+0000","last_clean":"2024-07-26T06:48:51.319169+0000","last_became_active":"2024-07-26T06:47:05.026569+0000","last_became_peered":"2024-07-26T06:47:05.026569+0000","last_unstale":"2024-07-26T06:48:51.319169+0000","last_undegraded":"2024-07-26T06:48:51.319169+0000","last_fullsized":"2024-07-26T06:48:51.319169+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:12:14.687633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"80'4","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333414+0000","last_change":"2024-07-26T06:47:23.099689+0000","last_active":"2024-07-26T06:48:51.333414+0000","last_peered":"2024-07-26T06:48:51.333414+0000","last_clean":"2024-07-26T06:48:51.333414+0000","last_became_active":"2024-07-26T06:47:23.099313+0000","last_became_peered":"2024-07-26T06:47:23.099313+0000","last_unstale":"2024-07-26T06:48:51.333414+0000","last_undegraded":"2024-07-26T06:48:51.333414+0000","last_fullsized":"2024-07-26T06:48:51.333414+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:49:01.833297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"80'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333205+0000","last_change":"2024-07-26T06:47:05.034864+0000","last_active":"2024-07-26T06:48:51.333205+0000","last_peered":"2024-07-26T06:48:51.333205+0000","last_clean":"2024-07-26T06:48:51.333205+0000","last_became_active":"2024-07-26T06:47:05.033414+0000","last_became_peered":"2024-07-26T06:47:05.033414+0000","last_unstale":"2024-07-26T06:48:51.333205+0000","last_undegraded":"2024-07-26T06:48:51.333205+0000","last_fullsized":"2024-07-26T06:48:51.333205+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:21:31.420212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"80'8","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.324215+0000","last_change":"2024-07-26T06:47:05.025462+0000","last_active":"2024-07-26T06:48:51.324215+0000","last_peered":"2024-07-26T06:48:51.324215+0000","last_clean":"2024-07-26T06:48:51.324215+0000","last_became_active":"2024-07-26T06:47:05.025049+0000","last_became_peered":"2024-07-26T06:47:05.025049+0000","last_unstale":"2024-07-26T06:48:51.324215+0000","last_undegraded":"2024-07-26T06:48:51.324215+0000","last_fullsized":"2024-07-26T06:48:51.324215+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:59:28.903338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"80'12","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.330035+0000","last_change":"2024-07-26T06:47:05.026394+0000","last_active":"2024-07-26T06:48:51.330035+0000","last_peered":"2024-07-26T06:48:51.330035+0000","last_clean":"2024-07-26T06:48:51.330035+0000","last_became_active":"2024-07-26T06:47:05.025820+0000","last_became_peered":"2024-07-26T06:47:05.025820+0000","last_unstale":"2024-07-26T06:48:51.330035+0000","last_undegraded":"2024-07-26T06:48:51.330035+0000","last_fullsized":"2024-07-26T06:48:51.330035+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:44:57.441638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"80'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328916+0000","last_change":"2024-07-26T06:47:05.024373+0000","last_active":"2024-07-26T06:48:51.328916+0000","last_peered":"2024-07-26T06:48:51.328916+0000","last_clean":"2024-07-26T06:48:51.328916+0000","last_became_active":"2024-07-26T06:47:05.023962+0000","last_became_peered":"2024-07-26T06:47:05.023962+0000","last_unstale":"2024-07-26T06:48:51.328916+0000","last_undegraded":"2024-07-26T06:48:51.328916+0000","last_fullsized":"2024-07-26T06:48:51.328916+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:43.804614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"80'7","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.325261+0000","last_change":"2024-07-26T06:47:05.031146+0000","last_active":"2024-07-26T06:48:51.325261+0000","last_peered":"2024-07-26T06:48:51.325261+0000","last_clean":"2024-07-26T06:48:51.325261+0000","last_became_active":"2024-07-26T06:47:05.029393+0000","last_became_peered":"2024-07-26T06:47:05.029393+0000","last_unstale":"2024-07-26T06:48:51.325261+0000","last_undegraded":"2024-07-26T06:48:51.325261+0000","last_fullsized":"2024-07-26T06:48:51.325261+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:46.853933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"80'7","reported_seq":135,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:00.273892+0000","last_change":"2024-07-26T06:47:02.001800+0000","last_active":"2024-07-26T06:49:00.273892+0000","last_peered":"2024-07-26T06:49:00.273892+0000","last_clean":"2024-07-26T06:49:00.273892+0000","last_became_active":"2024-07-26T06:47:02.001435+0000","last_became_peered":"2024-07-26T06:47:02.001435+0000","last_unstale":"2024-07-26T06:49:00.273892+0000","last_undegraded":"2024-07-26T06:49:00.273892+0000","last_fullsized":"2024-07-26T06:49:00.273892+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:23.121378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"80'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326709+0000","last_change":"2024-07-26T06:47:05.028301+0000","last_active":"2024-07-26T06:48:51.326709+0000","last_peered":"2024-07-26T06:48:51.326709+0000","last_clean":"2024-07-26T06:48:51.326709+0000","last_became_active":"2024-07-26T06:47:05.027761+0000","last_became_peered":"2024-07-26T06:47:05.027761+0000","last_unstale":"2024-07-26T06:48:51.326709+0000","last_undegraded":"2024-07-26T06:48:51.326709+0000","last_fullsized":"2024-07-26T06:48:51.326709+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:19.109395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093760+0000","last_change":"2024-07-26T06:47:02.004488+0000","last_active":"2024-07-26T06:47:23.093760+0000","last_peered":"2024-07-26T06:47:23.093760+0000","last_clean":"2024-07-26T06:47:23.093760+0000","last_became_active":"2024-07-26T06:47:02.004163+0000","last_became_peered":"2024-07-26T06:47:02.004163+0000","last_unstale":"2024-07-26T06:47:23.093760+0000","last_undegraded":"2024-07-26T06:47:23.093760+0000","last_fullsized":"2024-07-26T06:47:23.093760+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:16.221618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096069+0000","last_change":"2024-07-26T06:47:01.992708+0000","last_active":"2024-07-26T06:47:23.096069+0000","last_peered":"2024-07-26T06:47:23.096069+0000","last_clean":"2024-07-26T06:47:23.096069+0000","last_became_active":"2024-07-26T06:47:01.992330+0000","last_became_peered":"2024-07-26T06:47:01.992330+0000","last_unstale":"2024-07-26T06:47:23.096069+0000","last_undegraded":"2024-07-26T06:47:23.096069+0000","last_fullsized":"2024-07-26T06:47:23.096069+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:25.951946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"80'8","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.330585+0000","last_change":"2024-07-26T06:47:05.013946+0000","last_active":"2024-07-26T06:48:51.330585+0000","last_peered":"2024-07-26T06:48:51.330585+0000","last_clean":"2024-07-26T06:48:51.330585+0000","last_became_active":"2024-07-26T06:47:05.013509+0000","last_became_peered":"2024-07-26T06:47:05.013509+0000","last_unstale":"2024-07-26T06:48:51.330585+0000","last_undegraded":"2024-07-26T06:48:51.330585+0000","last_fullsized":"2024-07-26T06:48:51.330585+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:09.789287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093848+0000","last_change":"2024-07-26T06:47:01.997617+0000","last_active":"2024-07-26T06:47:23.093848+0000","last_peered":"2024-07-26T06:47:23.093848+0000","last_clean":"2024-07-26T06:47:23.093848+0000","last_became_active":"2024-07-26T06:47:01.997238+0000","last_became_peered":"2024-07-26T06:47:01.997238+0000","last_unstale":"2024-07-26T06:47:23.093848+0000","last_undegraded":"2024-07-26T06:47:23.093848+0000","last_fullsized":"2024-07-26T06:47:23.093848+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:45.596438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"80'11","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328388+0000","last_change":"2024-07-26T06:47:23.099813+0000","last_active":"2024-07-26T06:48:51.328388+0000","last_peered":"2024-07-26T06:48:51.328388+0000","last_clean":"2024-07-26T06:48:51.328388+0000","last_became_active":"2024-07-26T06:47:23.098701+0000","last_became_peered":"2024-07-26T06:47:23.098701+0000","last_unstale":"2024-07-26T06:48:51.328388+0000","last_undegraded":"2024-07-26T06:48:51.328388+0000","last_fullsized":"2024-07-26T06:48:51.328388+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:42:04.059637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'5","reported_seq":231,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:03.277803+0000","last_change":"2024-07-26T06:47:01.998508+0000","last_active":"2024-07-26T06:49:03.277803+0000","last_peered":"2024-07-26T06:49:03.277803+0000","last_clean":"2024-07-26T06:49:03.277803+0000","last_became_active":"2024-07-26T06:47:01.998132+0000","last_became_peered":"2024-07-26T06:47:01.998132+0000","last_unstale":"2024-07-26T06:49:03.277803+0000","last_undegraded":"2024-07-26T06:49:03.277803+0000","last_fullsized":"2024-07-26T06:49:03.277803+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:21.588085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":200,"num_read_kb":200,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"80'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338987+0000","last_change":"2024-07-26T06:47:05.032704+0000","last_active":"2024-07-26T06:48:51.338987+0000","last_peered":"2024-07-26T06:48:51.338987+0000","last_clean":"2024-07-26T06:48:51.338987+0000","last_became_active":"2024-07-26T06:47:05.031843+0000","last_became_peered":"2024-07-26T06:47:05.031843+0000","last_unstale":"2024-07-26T06:48:51.338987+0000","last_undegraded":"2024-07-26T06:48:51.338987+0000","last_fullsized":"2024-07-26T06:48:51.338987+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:58:24.436233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097398+0000","last_change":"2024-07-26T06:47:02.006266+0000","last_active":"2024-07-26T06:47:23.097398+0000","last_peered":"2024-07-26T06:47:23.097398+0000","last_clean":"2024-07-26T06:47:23.097398+0000","last_became_active":"2024-07-26T06:47:02.005830+0000","last_became_peered":"2024-07-26T06:47:02.005830+0000","last_unstale":"2024-07-26T06:47:23.097398+0000","last_undegraded":"2024-07-26T06:47:23.097398+0000","last_fullsized":"2024-07-26T06:47:23.097398+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:06:09.097781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"80'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.331339+0000","last_change":"2024-07-26T06:47:05.024820+0000","last_active":"2024-07-26T06:48:51.331339+0000","last_peered":"2024-07-26T06:48:51.331339+0000","last_clean":"2024-07-26T06:48:51.331339+0000","last_became_active":"2024-07-26T06:47:05.023941+0000","last_became_peered":"2024-07-26T06:47:05.023941+0000","last_unstale":"2024-07-26T06:48:51.331339+0000","last_undegraded":"2024-07-26T06:48:51.331339+0000","last_fullsized":"2024-07-26T06:48:51.331339+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:07.885159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095379+0000","last_change":"2024-07-26T06:47:02.023308+0000","last_active":"2024-07-26T06:47:23.095379+0000","last_peered":"2024-07-26T06:47:23.095379+0000","last_clean":"2024-07-26T06:47:23.095379+0000","last_became_active":"2024-07-26T06:47:02.022933+0000","last_became_peered":"2024-07-26T06:47:02.022933+0000","last_unstale":"2024-07-26T06:47:23.095379+0000","last_undegraded":"2024-07-26T06:47:23.095379+0000","last_fullsized":"2024-07-26T06:47:23.095379+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:59:01.233386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"80'11","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.340434+0000","last_change":"2024-07-26T06:47:05.026005+0000","last_active":"2024-07-26T06:48:51.340434+0000","last_peered":"2024-07-26T06:48:51.340434+0000","last_clean":"2024-07-26T06:48:51.340434+0000","last_became_active":"2024-07-26T06:47:05.025599+0000","last_became_peered":"2024-07-26T06:47:05.025599+0000","last_unstale":"2024-07-26T06:48:51.340434+0000","last_undegraded":"2024-07-26T06:48:51.340434+0000","last_fullsized":"2024-07-26T06:48:51.340434+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:44:19.924157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'34","reported_seq":289,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:03.277912+0000","last_change":"2024-07-26T06:47:23.393050+0000","last_active":"2024-07-26T06:49:03.277912+0000","last_peered":"2024-07-26T06:49:03.277912+0000","last_clean":"2024-07-26T06:49:03.277912+0000","last_became_active":"2024-07-26T06:47:23.381611+0000","last_became_peered":"2024-07-26T06:47:23.381611+0000","last_unstale":"2024-07-26T06:49:03.277912+0000","last_undegraded":"2024-07-26T06:49:03.277912+0000","last_fullsized":"2024-07-26T06:49:03.277912+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:23.866523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62075,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":210,"num_read_kb":165,"num_write":36,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"80'6","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335552+0000","last_change":"2024-07-26T06:47:05.034190+0000","last_active":"2024-07-26T06:48:51.335552+0000","last_peered":"2024-07-26T06:48:51.335552+0000","last_clean":"2024-07-26T06:48:51.335552+0000","last_became_active":"2024-07-26T06:47:05.033569+0000","last_became_peered":"2024-07-26T06:47:05.033569+0000","last_unstale":"2024-07-26T06:48:51.335552+0000","last_undegraded":"2024-07-26T06:48:51.335552+0000","last_fullsized":"2024-07-26T06:48:51.335552+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:39.306996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.120074+0000","last_change":"2024-07-26T06:47:02.001718+0000","last_active":"2024-07-26T06:47:24.120074+0000","last_peered":"2024-07-26T06:47:24.120074+0000","last_clean":"2024-07-26T06:47:24.120074+0000","last_became_active":"2024-07-26T06:47:02.001359+0000","last_became_peered":"2024-07-26T06:47:02.001359+0000","last_unstale":"2024-07-26T06:47:24.120074+0000","last_undegraded":"2024-07-26T06:47:24.120074+0000","last_fullsized":"2024-07-26T06:47:24.120074+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:03:23.390279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"80'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337734+0000","last_change":"2024-07-26T06:47:05.027445+0000","last_active":"2024-07-26T06:48:51.337734+0000","last_peered":"2024-07-26T06:48:51.337734+0000","last_clean":"2024-07-26T06:48:51.337734+0000","last_became_active":"2024-07-26T06:47:05.027114+0000","last_became_peered":"2024-07-26T06:47:05.027114+0000","last_unstale":"2024-07-26T06:48:51.337734+0000","last_undegraded":"2024-07-26T06:48:51.337734+0000","last_fullsized":"2024-07-26T06:48:51.337734+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:29:10.519271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.094571+0000","last_change":"2024-07-26T06:47:01.998832+0000","last_active":"2024-07-26T06:47:23.094571+0000","last_peered":"2024-07-26T06:47:23.094571+0000","last_clean":"2024-07-26T06:47:23.094571+0000","last_became_active":"2024-07-26T06:47:01.998512+0000","last_became_peered":"2024-07-26T06:47:01.998512+0000","last_unstale":"2024-07-26T06:47:23.094571+0000","last_undegraded":"2024-07-26T06:47:23.094571+0000","last_fullsized":"2024-07-26T06:47:23.094571+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:27.722329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"80'15","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.340035+0000","last_change":"2024-07-26T06:47:05.027803+0000","last_active":"2024-07-26T06:48:51.340035+0000","last_peered":"2024-07-26T06:48:51.340035+0000","last_clean":"2024-07-26T06:48:51.340035+0000","last_became_active":"2024-07-26T06:47:05.027320+0000","last_became_peered":"2024-07-26T06:47:05.027320+0000","last_unstale":"2024-07-26T06:48:51.340035+0000","last_undegraded":"2024-07-26T06:48:51.340035+0000","last_fullsized":"2024-07-26T06:48:51.340035+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:43:06.242945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'7","reported_seq":137,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:01.275653+0000","last_change":"2024-07-26T06:47:01.998356+0000","last_active":"2024-07-26T06:49:01.275653+0000","last_peered":"2024-07-26T06:49:01.275653+0000","last_clean":"2024-07-26T06:49:01.275653+0000","last_became_active":"2024-07-26T06:47:01.997922+0000","last_became_peered":"2024-07-26T06:47:01.997922+0000","last_unstale":"2024-07-26T06:49:01.275653+0000","last_undegraded":"2024-07-26T06:49:01.275653+0000","last_fullsized":"2024-07-26T06:49:01.275653+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:50:58.066682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"80'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339316+0000","last_change":"2024-07-26T06:47:05.025285+0000","last_active":"2024-07-26T06:48:51.339316+0000","last_peered":"2024-07-26T06:48:51.339316+0000","last_clean":"2024-07-26T06:48:51.339316+0000","last_became_active":"2024-07-26T06:47:05.024945+0000","last_became_peered":"2024-07-26T06:47:05.024945+0000","last_unstale":"2024-07-26T06:48:51.339316+0000","last_undegraded":"2024-07-26T06:48:51.339316+0000","last_fullsized":"2024-07-26T06:48:51.339316+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:21:04.680435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093595+0000","last_change":"2024-07-26T06:47:02.024895+0000","last_active":"2024-07-26T06:47:23.093595+0000","last_peered":"2024-07-26T06:47:23.093595+0000","last_clean":"2024-07-26T06:47:23.093595+0000","last_became_active":"2024-07-26T06:47:02.024451+0000","last_became_peered":"2024-07-26T06:47:02.024451+0000","last_unstale":"2024-07-26T06:47:23.093595+0000","last_undegraded":"2024-07-26T06:47:23.093595+0000","last_fullsized":"2024-07-26T06:47:23.093595+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:05:13.396499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"80'6","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335375+0000","last_change":"2024-07-26T06:47:23.381060+0000","last_active":"2024-07-26T06:48:51.335375+0000","last_peered":"2024-07-26T06:48:51.335375+0000","last_clean":"2024-07-26T06:48:51.335375+0000","last_became_active":"2024-07-26T06:47:23.380685+0000","last_became_peered":"2024-07-26T06:47:23.380685+0000","last_unstale":"2024-07-26T06:48:51.335375+0000","last_undegraded":"2024-07-26T06:48:51.335375+0000","last_fullsized":"2024-07-26T06:48:51.335375+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:21:38.062900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093109+0000","last_change":"2024-07-26T06:47:02.002098+0000","last_active":"2024-07-26T06:47:23.093109+0000","last_peered":"2024-07-26T06:47:23.093109+0000","last_clean":"2024-07-26T06:47:23.093109+0000","last_became_active":"2024-07-26T06:47:02.001428+0000","last_became_peered":"2024-07-26T06:47:02.001428+0000","last_unstale":"2024-07-26T06:47:23.093109+0000","last_undegraded":"2024-07-26T06:47:23.093109+0000","last_fullsized":"2024-07-26T06:47:23.093109+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:52:29.112500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"80'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.329946+0000","last_change":"2024-07-26T06:47:05.023686+0000","last_active":"2024-07-26T06:48:51.329946+0000","last_peered":"2024-07-26T06:48:51.329946+0000","last_clean":"2024-07-26T06:48:51.329946+0000","last_became_active":"2024-07-26T06:47:05.022583+0000","last_became_peered":"2024-07-26T06:47:05.022583+0000","last_unstale":"2024-07-26T06:48:51.329946+0000","last_undegraded":"2024-07-26T06:48:51.329946+0000","last_fullsized":"2024-07-26T06:48:51.329946+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:29:03.752429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096434+0000","last_change":"2024-07-26T06:47:02.006040+0000","last_active":"2024-07-26T06:47:23.096434+0000","last_peered":"2024-07-26T06:47:23.096434+0000","last_clean":"2024-07-26T06:47:23.096434+0000","last_became_active":"2024-07-26T06:47:02.005586+0000","last_became_peered":"2024-07-26T06:47:02.005586+0000","last_unstale":"2024-07-26T06:47:23.096434+0000","last_undegraded":"2024-07-26T06:47:23.096434+0000","last_fullsized":"2024-07-26T06:47:23.096434+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:09:27.180873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"80'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335167+0000","last_change":"2024-07-26T06:47:05.035524+0000","last_active":"2024-07-26T06:48:51.335167+0000","last_peered":"2024-07-26T06:48:51.335167+0000","last_clean":"2024-07-26T06:48:51.335167+0000","last_became_active":"2024-07-26T06:47:05.035078+0000","last_became_peered":"2024-07-26T06:47:05.035078+0000","last_unstale":"2024-07-26T06:48:51.335167+0000","last_undegraded":"2024-07-26T06:48:51.335167+0000","last_fullsized":"2024-07-26T06:48:51.335167+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:03.758261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095730+0000","last_change":"2024-07-26T06:47:01.999810+0000","last_active":"2024-07-26T06:47:23.095730+0000","last_peered":"2024-07-26T06:47:23.095730+0000","last_clean":"2024-07-26T06:47:23.095730+0000","last_became_active":"2024-07-26T06:47:01.999460+0000","last_became_peered":"2024-07-26T06:47:01.999460+0000","last_unstale":"2024-07-26T06:47:23.095730+0000","last_undegraded":"2024-07-26T06:47:23.095730+0000","last_fullsized":"2024-07-26T06:47:23.095730+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:25:05.691900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339575+0000","last_change":"2024-07-26T06:47:05.033631+0000","last_active":"2024-07-26T06:48:51.339575+0000","last_peered":"2024-07-26T06:48:51.339575+0000","last_clean":"2024-07-26T06:48:51.339575+0000","last_became_active":"2024-07-26T06:47:05.033266+0000","last_became_peered":"2024-07-26T06:47:05.033266+0000","last_unstale":"2024-07-26T06:48:51.339575+0000","last_undegraded":"2024-07-26T06:48:51.339575+0000","last_fullsized":"2024-07-26T06:48:51.339575+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:55.895271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"78'5","reported_seq":132,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:00.291845+0000","last_change":"2024-07-26T06:47:02.113525+0000","last_active":"2024-07-26T06:49:00.291845+0000","last_peered":"2024-07-26T06:49:00.291845+0000","last_clean":"2024-07-26T06:49:00.291845+0000","last_became_active":"2024-07-26T06:47:02.113112+0000","last_became_peered":"2024-07-26T06:47:02.113112+0000","last_unstale":"2024-07-26T06:49:00.291845+0000","last_undegraded":"2024-07-26T06:49:00.291845+0000","last_fullsized":"2024-07-26T06:49:00.291845+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:05:15.071264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"80'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338314+0000","last_change":"2024-07-26T06:47:05.026423+0000","last_active":"2024-07-26T06:48:51.338314+0000","last_peered":"2024-07-26T06:48:51.338314+0000","last_clean":"2024-07-26T06:48:51.338314+0000","last_became_active":"2024-07-26T06:47:05.026053+0000","last_became_peered":"2024-07-26T06:47:05.026053+0000","last_unstale":"2024-07-26T06:48:51.338314+0000","last_undegraded":"2024-07-26T06:48:51.338314+0000","last_fullsized":"2024-07-26T06:48:51.338314+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:07:20.174538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":102,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097268+0000","last_change":"2024-07-26T06:45:40.184988+0000","last_active":"2024-07-26T06:47:23.097268+0000","last_peered":"2024-07-26T06:47:23.097268+0000","last_clean":"2024-07-26T06:47:23.097268+0000","last_became_active":"2024-07-26T06:45:40.178827+0000","last_became_peered":"2024-07-26T06:45:40.178827+0000","last_unstale":"2024-07-26T06:47:23.097268+0000","last_undegraded":"2024-07-26T06:47:23.097268+0000","last_fullsized":"2024-07-26T06:47:23.097268+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:42:28.257602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:42:28.257602+0000","last_clean_scrub_stamp":"2024-07-26T06:42:28.257602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:52:14.123696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.120218+0000","last_change":"2024-07-26T06:47:01.996610+0000","last_active":"2024-07-26T06:47:24.120218+0000","last_peered":"2024-07-26T06:47:24.120218+0000","last_clean":"2024-07-26T06:47:24.120218+0000","last_became_active":"2024-07-26T06:47:01.996155+0000","last_became_peered":"2024-07-26T06:47:01.996155+0000","last_unstale":"2024-07-26T06:47:24.120218+0000","last_undegraded":"2024-07-26T06:47:24.120218+0000","last_fullsized":"2024-07-26T06:47:24.120218+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:51:30.420254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"80'8","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333651+0000","last_change":"2024-07-26T06:47:05.027699+0000","last_active":"2024-07-26T06:48:51.333651+0000","last_peered":"2024-07-26T06:48:51.333651+0000","last_clean":"2024-07-26T06:48:51.333651+0000","last_became_active":"2024-07-26T06:47:05.027348+0000","last_became_peered":"2024-07-26T06:47:05.027348+0000","last_unstale":"2024-07-26T06:48:51.333651+0000","last_undegraded":"2024-07-26T06:48:51.333651+0000","last_fullsized":"2024-07-26T06:48:51.333651+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:28:42.664351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097188+0000","last_change":"2024-07-26T06:47:02.003205+0000","last_active":"2024-07-26T06:47:23.097188+0000","last_peered":"2024-07-26T06:47:23.097188+0000","last_clean":"2024-07-26T06:47:23.097188+0000","last_became_active":"2024-07-26T06:47:02.002543+0000","last_became_peered":"2024-07-26T06:47:02.002543+0000","last_unstale":"2024-07-26T06:47:23.097188+0000","last_undegraded":"2024-07-26T06:47:23.097188+0000","last_fullsized":"2024-07-26T06:47:23.097188+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:43:10.911943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"80'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.334496+0000","last_change":"2024-07-26T06:47:05.023258+0000","last_active":"2024-07-26T06:48:51.334496+0000","last_peered":"2024-07-26T06:48:51.334496+0000","last_clean":"2024-07-26T06:48:51.334496+0000","last_became_active":"2024-07-26T06:47:05.022903+0000","last_became_peered":"2024-07-26T06:47:05.022903+0000","last_unstale":"2024-07-26T06:48:51.334496+0000","last_undegraded":"2024-07-26T06:48:51.334496+0000","last_fullsized":"2024-07-26T06:48:51.334496+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:08.155850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"80'6","reported_seq":131,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:00.273361+0000","last_change":"2024-07-26T06:47:02.022316+0000","last_active":"2024-07-26T06:49:00.273361+0000","last_peered":"2024-07-26T06:49:00.273361+0000","last_clean":"2024-07-26T06:49:00.273361+0000","last_became_active":"2024-07-26T06:47:02.021923+0000","last_became_peered":"2024-07-26T06:47:02.021923+0000","last_unstale":"2024-07-26T06:49:00.273361+0000","last_undegraded":"2024-07-26T06:49:00.273361+0000","last_fullsized":"2024-07-26T06:49:00.273361+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:29:31.533449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":99,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"80'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.327946+0000","last_change":"2024-07-26T06:47:05.021510+0000","last_active":"2024-07-26T06:48:51.327946+0000","last_peered":"2024-07-26T06:48:51.327946+0000","last_clean":"2024-07-26T06:48:51.327946+0000","last_became_active":"2024-07-26T06:47:05.021140+0000","last_became_peered":"2024-07-26T06:47:05.021140+0000","last_unstale":"2024-07-26T06:48:51.327946+0000","last_undegraded":"2024-07-26T06:48:51.327946+0000","last_fullsized":"2024-07-26T06:48:51.327946+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:37:01.340126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"79'5","reported_seq":132,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:02.308916+0000","last_change":"2024-07-26T06:47:02.001698+0000","last_active":"2024-07-26T06:49:02.308916+0000","last_peered":"2024-07-26T06:49:02.308916+0000","last_clean":"2024-07-26T06:49:02.308916+0000","last_became_active":"2024-07-26T06:47:02.001244+0000","last_became_peered":"2024-07-26T06:47:02.001244+0000","last_unstale":"2024-07-26T06:49:02.308916+0000","last_undegraded":"2024-07-26T06:49:02.308916+0000","last_fullsized":"2024-07-26T06:49:02.308916+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:32:07.920065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"80'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.321186+0000","last_change":"2024-07-26T06:47:05.029524+0000","last_active":"2024-07-26T06:48:51.321186+0000","last_peered":"2024-07-26T06:48:51.321186+0000","last_clean":"2024-07-26T06:48:51.321186+0000","last_became_active":"2024-07-26T06:47:05.028981+0000","last_became_peered":"2024-07-26T06:47:05.028981+0000","last_unstale":"2024-07-26T06:48:51.321186+0000","last_undegraded":"2024-07-26T06:48:51.321186+0000","last_fullsized":"2024-07-26T06:48:51.321186+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:49:57.019486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092276+0000","last_change":"2024-07-26T06:47:02.025117+0000","last_active":"2024-07-26T06:47:23.092276+0000","last_peered":"2024-07-26T06:47:23.092276+0000","last_clean":"2024-07-26T06:47:23.092276+0000","last_became_active":"2024-07-26T06:47:02.024612+0000","last_became_peered":"2024-07-26T06:47:02.024612+0000","last_unstale":"2024-07-26T06:47:23.092276+0000","last_undegraded":"2024-07-26T06:47:23.092276+0000","last_fullsized":"2024-07-26T06:47:23.092276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:15:44.999658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.324434+0000","last_change":"2024-07-26T06:47:05.015085+0000","last_active":"2024-07-26T06:48:51.324434+0000","last_peered":"2024-07-26T06:48:51.324434+0000","last_clean":"2024-07-26T06:48:51.324434+0000","last_became_active":"2024-07-26T06:47:05.014686+0000","last_became_peered":"2024-07-26T06:47:05.014686+0000","last_unstale":"2024-07-26T06:48:51.324434+0000","last_undegraded":"2024-07-26T06:48:51.324434+0000","last_fullsized":"2024-07-26T06:48:51.324434+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:39:30.392347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"80'4","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.159121+0000","last_change":"2024-07-26T06:47:23.111918+0000","last_active":"2024-07-26T06:47:23.159121+0000","last_peered":"2024-07-26T06:47:23.159121+0000","last_clean":"2024-07-26T06:47:23.159121+0000","last_became_active":"2024-07-26T06:47:23.099518+0000","last_became_peered":"2024-07-26T06:47:23.099518+0000","last_unstale":"2024-07-26T06:47:23.159121+0000","last_undegraded":"2024-07-26T06:47:23.159121+0000","last_fullsized":"2024-07-26T06:47:23.159121+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:41.012096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"80'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.331791+0000","last_change":"2024-07-26T06:47:05.019415+0000","last_active":"2024-07-26T06:48:51.331791+0000","last_peered":"2024-07-26T06:48:51.331791+0000","last_clean":"2024-07-26T06:48:51.331791+0000","last_became_active":"2024-07-26T06:47:05.019044+0000","last_became_peered":"2024-07-26T06:47:05.019044+0000","last_unstale":"2024-07-26T06:48:51.331791+0000","last_undegraded":"2024-07-26T06:48:51.331791+0000","last_fullsized":"2024-07-26T06:48:51.331791+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:02:37.228174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'15","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:47.027153+0000","last_change":"2024-07-26T06:47:02.006585+0000","last_active":"2024-07-26T06:47:47.027153+0000","last_peered":"2024-07-26T06:47:47.027153+0000","last_clean":"2024-07-26T06:47:47.027153+0000","last_became_active":"2024-07-26T06:47:02.006185+0000","last_became_peered":"2024-07-26T06:47:02.006185+0000","last_unstale":"2024-07-26T06:47:47.027153+0000","last_undegraded":"2024-07-26T06:47:47.027153+0000","last_fullsized":"2024-07-26T06:47:47.027153+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:02:45.480712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"80'5","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326811+0000","last_change":"2024-07-26T06:47:05.028840+0000","last_active":"2024-07-26T06:48:51.326811+0000","last_peered":"2024-07-26T06:48:51.326811+0000","last_clean":"2024-07-26T06:48:51.326811+0000","last_became_active":"2024-07-26T06:47:05.028487+0000","last_became_peered":"2024-07-26T06:47:05.028487+0000","last_unstale":"2024-07-26T06:48:51.326811+0000","last_undegraded":"2024-07-26T06:48:51.326811+0000","last_fullsized":"2024-07-26T06:48:51.326811+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:47:48.185227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.380389+0000","last_change":"2024-07-26T06:47:02.113691+0000","last_active":"2024-07-26T06:47:23.380389+0000","last_peered":"2024-07-26T06:47:23.380389+0000","last_clean":"2024-07-26T06:47:23.380389+0000","last_became_active":"2024-07-26T06:47:02.113342+0000","last_became_peered":"2024-07-26T06:47:02.113342+0000","last_unstale":"2024-07-26T06:47:23.380389+0000","last_undegraded":"2024-07-26T06:47:23.380389+0000","last_fullsized":"2024-07-26T06:47:23.380389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:14:03.586564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"80'7","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.332485+0000","last_change":"2024-07-26T06:47:23.098420+0000","last_active":"2024-07-26T06:48:51.332485+0000","last_peered":"2024-07-26T06:48:51.332485+0000","last_clean":"2024-07-26T06:48:51.332485+0000","last_became_active":"2024-07-26T06:47:23.097981+0000","last_became_peered":"2024-07-26T06:47:23.097981+0000","last_unstale":"2024-07-26T06:48:51.332485+0000","last_undegraded":"2024-07-26T06:48:51.332485+0000","last_fullsized":"2024-07-26T06:48:51.332485+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:38:40.155514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091072+0000","last_change":"2024-07-26T06:47:02.024781+0000","last_active":"2024-07-26T06:47:23.091072+0000","last_peered":"2024-07-26T06:47:23.091072+0000","last_clean":"2024-07-26T06:47:23.091072+0000","last_became_active":"2024-07-26T06:47:02.024492+0000","last_became_peered":"2024-07-26T06:47:02.024492+0000","last_unstale":"2024-07-26T06:47:23.091072+0000","last_undegraded":"2024-07-26T06:47:23.091072+0000","last_fullsized":"2024-07-26T06:47:23.091072+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:23:41.323963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"80'2","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.283302+0000","last_change":"2024-07-26T06:47:05.033052+0000","last_active":"2024-07-26T06:48:51.283302+0000","last_peered":"2024-07-26T06:48:51.283302+0000","last_clean":"2024-07-26T06:48:51.283302+0000","last_became_active":"2024-07-26T06:47:05.032052+0000","last_became_peered":"2024-07-26T06:47:05.032052+0000","last_unstale":"2024-07-26T06:48:51.283302+0000","last_undegraded":"2024-07-26T06:48:51.283302+0000","last_fullsized":"2024-07-26T06:48:51.283302+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:42:04.683586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090564+0000","last_change":"2024-07-26T06:47:01.996097+0000","last_active":"2024-07-26T06:47:23.090564+0000","last_peered":"2024-07-26T06:47:23.090564+0000","last_clean":"2024-07-26T06:47:23.090564+0000","last_became_active":"2024-07-26T06:47:01.995594+0000","last_became_peered":"2024-07-26T06:47:01.995594+0000","last_unstale":"2024-07-26T06:47:23.090564+0000","last_undegraded":"2024-07-26T06:47:23.090564+0000","last_fullsized":"2024-07-26T06:47:23.090564+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:34.969493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"80'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328345+0000","last_change":"2024-07-26T06:47:05.024171+0000","last_active":"2024-07-26T06:48:51.328345+0000","last_peered":"2024-07-26T06:48:51.328345+0000","last_clean":"2024-07-26T06:48:51.328345+0000","last_became_active":"2024-07-26T06:47:05.022888+0000","last_became_peered":"2024-07-26T06:47:05.022888+0000","last_unstale":"2024-07-26T06:48:51.328345+0000","last_undegraded":"2024-07-26T06:48:51.328345+0000","last_fullsized":"2024-07-26T06:48:51.328345+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:31:52.691068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096894+0000","last_change":"2024-07-26T06:47:02.004578+0000","last_active":"2024-07-26T06:47:23.096894+0000","last_peered":"2024-07-26T06:47:23.096894+0000","last_clean":"2024-07-26T06:47:23.096894+0000","last_became_active":"2024-07-26T06:47:02.004231+0000","last_became_peered":"2024-07-26T06:47:02.004231+0000","last_unstale":"2024-07-26T06:47:23.096894+0000","last_undegraded":"2024-07-26T06:47:23.096894+0000","last_fullsized":"2024-07-26T06:47:23.096894+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:43:39.803817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"80'8","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.327728+0000","last_change":"2024-07-26T06:47:05.024232+0000","last_active":"2024-07-26T06:48:51.327728+0000","last_peered":"2024-07-26T06:48:51.327728+0000","last_clean":"2024-07-26T06:48:51.327728+0000","last_became_active":"2024-07-26T06:47:05.023869+0000","last_became_peered":"2024-07-26T06:47:05.023869+0000","last_unstale":"2024-07-26T06:48:51.327728+0000","last_undegraded":"2024-07-26T06:48:51.327728+0000","last_fullsized":"2024-07-26T06:48:51.327728+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:27:01.357311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.105594+0000","last_change":"2024-07-26T06:47:23.105051+0000","last_active":"2024-07-26T06:47:23.105594+0000","last_peered":"2024-07-26T06:47:23.105594+0000","last_clean":"2024-07-26T06:47:23.105594+0000","last_became_active":"2024-07-26T06:47:23.098116+0000","last_became_peered":"2024-07-26T06:47:23.098116+0000","last_unstale":"2024-07-26T06:47:23.105594+0000","last_undegraded":"2024-07-26T06:47:23.105594+0000","last_fullsized":"2024-07-26T06:47:23.105594+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:15:32.991870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"80'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338310+0000","last_change":"2024-07-26T06:47:05.026056+0000","last_active":"2024-07-26T06:48:51.338310+0000","last_peered":"2024-07-26T06:48:51.338310+0000","last_clean":"2024-07-26T06:48:51.338310+0000","last_became_active":"2024-07-26T06:47:05.025654+0000","last_became_peered":"2024-07-26T06:47:05.025654+0000","last_unstale":"2024-07-26T06:48:51.338310+0000","last_undegraded":"2024-07-26T06:48:51.338310+0000","last_fullsized":"2024-07-26T06:48:51.338310+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:36:20.886768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"80'15","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338605+0000","last_change":"2024-07-26T06:47:05.026693+0000","last_active":"2024-07-26T06:48:51.338605+0000","last_peered":"2024-07-26T06:48:51.338605+0000","last_clean":"2024-07-26T06:48:51.338605+0000","last_became_active":"2024-07-26T06:47:05.026320+0000","last_became_peered":"2024-07-26T06:47:05.026320+0000","last_unstale":"2024-07-26T06:48:51.338605+0000","last_undegraded":"2024-07-26T06:48:51.338605+0000","last_fullsized":"2024-07-26T06:48:51.338605+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:52:30.776335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"80'5","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.159101+0000","last_change":"2024-07-26T06:47:23.107707+0000","last_active":"2024-07-26T06:47:23.159101+0000","last_peered":"2024-07-26T06:47:23.159101+0000","last_clean":"2024-07-26T06:47:23.159101+0000","last_became_active":"2024-07-26T06:47:23.099429+0000","last_became_peered":"2024-07-26T06:47:23.099429+0000","last_unstale":"2024-07-26T06:47:23.159101+0000","last_undegraded":"2024-07-26T06:47:23.159101+0000","last_fullsized":"2024-07-26T06:47:23.159101+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:40:52.856756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"80'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.334591+0000","last_change":"2024-07-26T06:47:05.032319+0000","last_active":"2024-07-26T06:48:51.334591+0000","last_peered":"2024-07-26T06:48:51.334591+0000","last_clean":"2024-07-26T06:48:51.334591+0000","last_became_active":"2024-07-26T06:47:05.030305+0000","last_became_peered":"2024-07-26T06:47:05.030305+0000","last_unstale":"2024-07-26T06:48:51.334591+0000","last_undegraded":"2024-07-26T06:48:51.334591+0000","last_fullsized":"2024-07-26T06:48:51.334591+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:34:05.351917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"80'12","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.119914+0000","last_change":"2024-07-26T06:47:02.007812+0000","last_active":"2024-07-26T06:47:24.119914+0000","last_peered":"2024-07-26T06:47:24.119914+0000","last_clean":"2024-07-26T06:47:24.119914+0000","last_became_active":"2024-07-26T06:47:02.006965+0000","last_became_peered":"2024-07-26T06:47:02.006965+0000","last_unstale":"2024-07-26T06:47:24.119914+0000","last_undegraded":"2024-07-26T06:47:24.119914+0000","last_fullsized":"2024-07-26T06:47:24.119914+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:53.173273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"80'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335704+0000","last_change":"2024-07-26T06:47:05.027900+0000","last_active":"2024-07-26T06:48:51.335704+0000","last_peered":"2024-07-26T06:48:51.335704+0000","last_clean":"2024-07-26T06:48:51.335704+0000","last_became_active":"2024-07-26T06:47:05.024467+0000","last_became_peered":"2024-07-26T06:47:05.024467+0000","last_unstale":"2024-07-26T06:48:51.335704+0000","last_undegraded":"2024-07-26T06:48:51.335704+0000","last_fullsized":"2024-07-26T06:48:51.335704+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:05:00.793487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090879+0000","last_change":"2024-07-26T06:47:02.022863+0000","last_active":"2024-07-26T06:47:23.090879+0000","last_peered":"2024-07-26T06:47:23.090879+0000","last_clean":"2024-07-26T06:47:23.090879+0000","last_became_active":"2024-07-26T06:47:02.022446+0000","last_became_peered":"2024-07-26T06:47:02.022446+0000","last_unstale":"2024-07-26T06:47:23.090879+0000","last_undegraded":"2024-07-26T06:47:23.090879+0000","last_fullsized":"2024-07-26T06:47:23.090879+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:14:48.649099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"80'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333567+0000","last_change":"2024-07-26T06:47:05.023669+0000","last_active":"2024-07-26T06:48:51.333567+0000","last_peered":"2024-07-26T06:48:51.333567+0000","last_clean":"2024-07-26T06:48:51.333567+0000","last_became_active":"2024-07-26T06:47:05.023265+0000","last_became_peered":"2024-07-26T06:47:05.023265+0000","last_unstale":"2024-07-26T06:48:51.333567+0000","last_undegraded":"2024-07-26T06:48:51.333567+0000","last_fullsized":"2024-07-26T06:48:51.333567+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:56:45.278253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":141,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:02.308393+0000","last_change":"2024-07-26T06:47:02.006668+0000","last_active":"2024-07-26T06:49:02.308393+0000","last_peered":"2024-07-26T06:49:02.308393+0000","last_clean":"2024-07-26T06:49:02.308393+0000","last_became_active":"2024-07-26T06:47:02.006287+0000","last_became_peered":"2024-07-26T06:47:02.006287+0000","last_unstale":"2024-07-26T06:49:02.308393+0000","last_undegraded":"2024-07-26T06:49:02.308393+0000","last_fullsized":"2024-07-26T06:49:02.308393+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:28:05.722539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":101,"num_read_kb":101,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69561,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1037,"num_read_kb":597,"num_write":253,"num_write_kb":143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1634,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":208683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":217,"ondisk_log_size":217,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250626,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710760,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":426222,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906989,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":1079164,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070513,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":425458,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266742,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":880640,"data_stored":425142,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462971,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":946176,"data_stored":1019027,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626496,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36196,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708956,"statfs":{"total":95995035648,"available":95957970944,"internally_reserved":0,"allocated":880640,"data_stored":425175,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822725,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":1018173,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986250,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":487643,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":897024,"data_stored":425000,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36208,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708944,"statfs":{"total":95995035648,"available":95957958656,"internally_reserved":0,"allocated":892928,"data_stored":426840,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36256,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708896,"statfs":{"total":95995035648,"available":95957909504,"internally_reserved":0,"allocated":942080,"data_stored":488163,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":426514,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":63307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":64343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:49:06.982 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph pg dump --format=json 2024-07-26T06:49:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:06 smithi164 ceph-mon[36970]: pgmap v362: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-26T06:49:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:49:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:49:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:07.247 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:07.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:06 smithi163 ceph-mon[36873]: pgmap v362: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-26T06:49:07.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:49:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:49:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:07.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:06 smithi042 ceph-mon[32775]: pgmap v362: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-26T06:49:07.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:07.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:07.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:49:07.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:49:07.370 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:49:08.073 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:07 smithi042 ceph-mon[32775]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:49:08.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:07 smithi164 ceph-mon[36970]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:49:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:07 smithi163 ceph-mon[36873]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:49:08.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.886+0000 7f1f9048c640 1 -- 172.21.15.42:0/198205086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f88107df0 msgr2=0x7f1f88108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:08.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.886+0000 7f1f9048c640 1 --2- 172.21.15.42:0/198205086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f88107df0 0x7f1f88108200 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1f78009900 tx=0x7f1f7802a410 comp rx=0 tx=0).stop 2024-07-26T06:49:08.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.886+0000 7f1f9048c640 1 -- 172.21.15.42:0/198205086 shutdown_connections 2024-07-26T06:49:08.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.886+0000 7f1f9048c640 1 --2- 172.21.15.42:0/198205086 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f88108740 0x7f1f8810ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:08.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.886+0000 7f1f9048c640 1 --2- 172.21.15.42:0/198205086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f88107df0 0x7f1f88108200 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:08.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.886+0000 7f1f9048c640 1 --2- 172.21.15.42:0/198205086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f881024b0 0x7f1f881028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:08.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.886+0000 7f1f9048c640 1 -- 172.21.15.42:0/198205086 >> 172.21.15.42:0/198205086 conn(0x7f1f88079170 msgr2=0x7f1f88079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:08.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.886+0000 7f1f9048c640 1 -- 172.21.15.42:0/198205086 shutdown_connections 2024-07-26T06:49:08.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.886+0000 7f1f9048c640 1 -- 172.21.15.42:0/198205086 wait complete. 2024-07-26T06:49:08.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.887+0000 7f1f9048c640 1 Processor -- start 2024-07-26T06:49:08.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.887+0000 7f1f9048c640 1 -- start start 2024-07-26T06:49:08.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.887+0000 7f1f9048c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f881024b0 0x7f1f88199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:08.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.887+0000 7f1f9048c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f88107df0 0x7f1f8819a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:08.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.887+0000 7f1f9048c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f88108740 0x7f1f8819f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:08.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.887+0000 7f1f9048c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1f881123f0 con 0x7f1f88108740 2024-07-26T06:49:08.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.887+0000 7f1f9048c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1f88112270 con 0x7f1f88107df0 2024-07-26T06:49:08.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.887+0000 7f1f9048c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1f88112570 con 0x7f1f881024b0 2024-07-26T06:49:08.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.888+0000 7f1f8ea29640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f88108740 0x7f1f8819f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:08.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.888+0000 7f1f8ea29640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f88108740 0x7f1f8819f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60690/0 (socket says 172.21.15.42:60690) 2024-07-26T06:49:08.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.888+0000 7f1f8ea29640 1 -- 172.21.15.42:0/765607189 learned_addr learned my addr 172.21.15.42:0/765607189 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:08.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.888+0000 7f1f8da27640 1 --2- 172.21.15.42:0/765607189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f88107df0 0x7f1f8819a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:08.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.888+0000 7f1f8e228640 1 --2- 172.21.15.42:0/765607189 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f881024b0 0x7f1f88199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:08.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.888+0000 7f1f8ea29640 1 -- 172.21.15.42:0/765607189 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f881024b0 msgr2=0x7f1f88199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:08.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.888+0000 7f1f8ea29640 1 --2- 172.21.15.42:0/765607189 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f881024b0 0x7f1f88199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:08.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.888+0000 7f1f8ea29640 1 -- 172.21.15.42:0/765607189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f88107df0 msgr2=0x7f1f8819a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:08.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.888+0000 7f1f8ea29640 1 --2- 172.21.15.42:0/765607189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f88107df0 0x7f1f8819a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:08.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.888+0000 7f1f8ea29640 1 -- 172.21.15.42:0/765607189 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f8819ff40 con 0x7f1f88108740 2024-07-26T06:49:08.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.888+0000 7f1f8e228640 1 --2- 172.21.15.42:0/765607189 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f881024b0 0x7f1f88199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:49:08.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.888+0000 7f1f8ea29640 1 --2- 172.21.15.42:0/765607189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f88108740 0x7f1f8819f810 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009fd0 tx=0x7f1f8000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:08.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.888+0000 7f1f7f7fe640 1 -- 172.21.15.42:0/765607189 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f80017070 con 0x7f1f88108740 2024-07-26T06:49:08.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.888+0000 7f1f9048c640 1 -- 172.21.15.42:0/765607189 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f881a0230 con 0x7f1f88108740 2024-07-26T06:49:08.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.888+0000 7f1f7f7fe640 1 -- 172.21.15.42:0/765607189 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f80003e20 con 0x7f1f88108740 2024-07-26T06:49:08.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.888+0000 7f1f7f7fe640 1 -- 172.21.15.42:0/765607189 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f80010460 con 0x7f1f88108740 2024-07-26T06:49:08.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.888+0000 7f1f9048c640 1 -- 172.21.15.42:0/765607189 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f881a6990 con 0x7f1f88108740 2024-07-26T06:49:08.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.889+0000 7f1f9048c640 1 -- 172.21.15.42:0/765607189 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f88103730 con 0x7f1f88108740 2024-07-26T06:49:08.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.891+0000 7f1f7f7fe640 1 -- 172.21.15.42:0/765607189 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f1f80003970 con 0x7f1f88108740 2024-07-26T06:49:08.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.892+0000 7f1f7f7fe640 1 --2- 172.21.15.42:0/765607189 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1f740b3bd0 0x7f1f740b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:08.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.892+0000 7f1f8e228640 1 --2- 172.21.15.42:0/765607189 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1f740b3bd0 0x7f1f740b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:08.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.892+0000 7f1f7f7fe640 1 -- 172.21.15.42:0/765607189 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f1f800e3930 con 0x7f1f88108740 2024-07-26T06:49:08.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.892+0000 7f1f8e228640 1 --2- 172.21.15.42:0/765607189 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1f740b3bd0 0x7f1f740b60e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1f84009930 tx=0x7f1f84008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:08.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:08.895+0000 7f1f7f7fe640 1 -- 172.21.15.42:0/765607189 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1f800aaf60 con 0x7f1f88108740 2024-07-26T06:49:09.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:09.078+0000 7f1f9048c640 1 -- 172.21.15.42:0/765607189 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1f88104a10 con 0x7f1f740b3bd0 2024-07-26T06:49:09.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:09.082+0000 7f1f7f7fe640 1 -- 172.21.15.42:0/765607189 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259109 (secure 0 0 0) 0x7f1f88104a10 con 0x7f1f740b3bd0 2024-07-26T06:49:09.085 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:49:09.086 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-07-26T06:49:09.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:09.085+0000 7f1f9048c640 1 -- 172.21.15.42:0/765607189 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1f740b3bd0 msgr2=0x7f1f740b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:09.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:09.085+0000 7f1f9048c640 1 --2- 172.21.15.42:0/765607189 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1f740b3bd0 0x7f1f740b60e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1f84009930 tx=0x7f1f84008040 comp rx=0 tx=0).stop 2024-07-26T06:49:09.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:09.085+0000 7f1f9048c640 1 -- 172.21.15.42:0/765607189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f88108740 msgr2=0x7f1f8819f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:09.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:09.085+0000 7f1f9048c640 1 --2- 172.21.15.42:0/765607189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f88108740 0x7f1f8819f810 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009fd0 tx=0x7f1f8000b260 comp rx=0 tx=0).stop 2024-07-26T06:49:09.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:09.085+0000 7f1f9048c640 1 -- 172.21.15.42:0/765607189 shutdown_connections 2024-07-26T06:49:09.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:09.085+0000 7f1f9048c640 1 --2- 172.21.15.42:0/765607189 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1f740b3bd0 0x7f1f740b60e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:09.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:09.085+0000 7f1f9048c640 1 --2- 172.21.15.42:0/765607189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f88108740 0x7f1f8819f810 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:09.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:09.085+0000 7f1f9048c640 1 --2- 172.21.15.42:0/765607189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f88107df0 0x7f1f8819a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:09.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:09.085+0000 7f1f9048c640 1 --2- 172.21.15.42:0/765607189 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f881024b0 0x7f1f88199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:09.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:09.085+0000 7f1f9048c640 1 -- 172.21.15.42:0/765607189 >> 172.21.15.42:0/765607189 conn(0x7f1f88079170 msgr2=0x7f1f88101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:09.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:09.086+0000 7f1f9048c640 1 -- 172.21.15.42:0/765607189 shutdown_connections 2024-07-26T06:49:09.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:09.086+0000 7f1f9048c640 1 -- 172.21.15.42:0/765607189 wait complete. 2024-07-26T06:49:09.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:08 smithi164 ceph-mon[36970]: pgmap v363: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T06:49:09.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:08 smithi163 ceph-mon[36873]: pgmap v363: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T06:49:09.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:08 smithi042 ceph-mon[32775]: pgmap v363: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T06:49:09.764 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-07-26T06:49:07.349427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660163,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1128,"num_read_kb":666,"num_write":378,"num_write_kb":2295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2953474,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":811,"ondisk_log_size":811,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384912,"kb_used_data":10800,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124556912,"statfs":{"total":1151940427776,"available":1151546277888,"internally_reserved":0,"allocated":11059200,"data_stored":7072521,"data_compressed":177858,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":116,"num_read_kb":56,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003717"},"pg_stats":[{"pgid":"3.3f","version":"80'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335928+0000","last_change":"2024-07-26T06:47:05.026186+0000","last_active":"2024-07-26T06:48:51.335928+0000","last_peered":"2024-07-26T06:48:51.335928+0000","last_clean":"2024-07-26T06:48:51.335928+0000","last_became_active":"2024-07-26T06:47:05.025589+0000","last_became_peered":"2024-07-26T06:47:05.025589+0000","last_unstale":"2024-07-26T06:48:51.335928+0000","last_undegraded":"2024-07-26T06:48:51.335928+0000","last_fullsized":"2024-07-26T06:48:51.335928+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:48:23.100966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"80'4","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.325860+0000","last_change":"2024-07-26T06:47:05.033211+0000","last_active":"2024-07-26T06:48:51.325860+0000","last_peered":"2024-07-26T06:48:51.325860+0000","last_clean":"2024-07-26T06:48:51.325860+0000","last_became_active":"2024-07-26T06:47:05.032630+0000","last_became_peered":"2024-07-26T06:47:05.032630+0000","last_unstale":"2024-07-26T06:48:51.325860+0000","last_undegraded":"2024-07-26T06:48:51.325860+0000","last_fullsized":"2024-07-26T06:48:51.325860+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:03:19.612963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"80'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335834+0000","last_change":"2024-07-26T06:47:05.034767+0000","last_active":"2024-07-26T06:48:51.335834+0000","last_peered":"2024-07-26T06:48:51.335834+0000","last_clean":"2024-07-26T06:48:51.335834+0000","last_became_active":"2024-07-26T06:47:05.034376+0000","last_became_peered":"2024-07-26T06:47:05.034376+0000","last_unstale":"2024-07-26T06:48:51.335834+0000","last_undegraded":"2024-07-26T06:48:51.335834+0000","last_fullsized":"2024-07-26T06:48:51.335834+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:55:55.729930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"80'8","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326864+0000","last_change":"2024-07-26T06:47:05.030995+0000","last_active":"2024-07-26T06:48:51.326864+0000","last_peered":"2024-07-26T06:48:51.326864+0000","last_clean":"2024-07-26T06:48:51.326864+0000","last_became_active":"2024-07-26T06:47:05.025423+0000","last_became_peered":"2024-07-26T06:47:05.025423+0000","last_unstale":"2024-07-26T06:48:51.326864+0000","last_undegraded":"2024-07-26T06:48:51.326864+0000","last_fullsized":"2024-07-26T06:48:51.326864+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:59:39.511383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"80'4","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.316860+0000","last_change":"2024-07-26T06:47:05.035359+0000","last_active":"2024-07-26T06:48:51.316860+0000","last_peered":"2024-07-26T06:48:51.316860+0000","last_clean":"2024-07-26T06:48:51.316860+0000","last_became_active":"2024-07-26T06:47:05.034836+0000","last_became_peered":"2024-07-26T06:47:05.034836+0000","last_unstale":"2024-07-26T06:48:51.316860+0000","last_undegraded":"2024-07-26T06:48:51.316860+0000","last_fullsized":"2024-07-26T06:48:51.316860+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:54.482774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"80'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326138+0000","last_change":"2024-07-26T06:47:05.023660+0000","last_active":"2024-07-26T06:48:51.326138+0000","last_peered":"2024-07-26T06:48:51.326138+0000","last_clean":"2024-07-26T06:48:51.326138+0000","last_became_active":"2024-07-26T06:47:05.022446+0000","last_became_peered":"2024-07-26T06:47:05.022446+0000","last_unstale":"2024-07-26T06:48:51.326138+0000","last_undegraded":"2024-07-26T06:48:51.326138+0000","last_fullsized":"2024-07-26T06:48:51.326138+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:35:00.519226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"80'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339395+0000","last_change":"2024-07-26T06:47:05.025132+0000","last_active":"2024-07-26T06:48:51.339395+0000","last_peered":"2024-07-26T06:48:51.339395+0000","last_clean":"2024-07-26T06:48:51.339395+0000","last_became_active":"2024-07-26T06:47:05.024770+0000","last_became_peered":"2024-07-26T06:47:05.024770+0000","last_unstale":"2024-07-26T06:48:51.339395+0000","last_undegraded":"2024-07-26T06:48:51.339395+0000","last_fullsized":"2024-07-26T06:48:51.339395+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:21:39.920152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"80'9","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.332397+0000","last_change":"2024-07-26T06:47:05.022613+0000","last_active":"2024-07-26T06:48:51.332397+0000","last_peered":"2024-07-26T06:48:51.332397+0000","last_clean":"2024-07-26T06:48:51.332397+0000","last_became_active":"2024-07-26T06:47:05.022202+0000","last_became_peered":"2024-07-26T06:47:05.022202+0000","last_unstale":"2024-07-26T06:48:51.332397+0000","last_undegraded":"2024-07-26T06:48:51.332397+0000","last_fullsized":"2024-07-26T06:48:51.332397+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:07:23.980843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"80'11","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.336833+0000","last_change":"2024-07-26T06:47:05.026352+0000","last_active":"2024-07-26T06:48:51.336833+0000","last_peered":"2024-07-26T06:48:51.336833+0000","last_clean":"2024-07-26T06:48:51.336833+0000","last_became_active":"2024-07-26T06:47:05.025969+0000","last_became_peered":"2024-07-26T06:47:05.025969+0000","last_unstale":"2024-07-26T06:48:51.336833+0000","last_undegraded":"2024-07-26T06:48:51.336833+0000","last_fullsized":"2024-07-26T06:48:51.336833+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:27.062013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"80'16","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328270+0000","last_change":"2024-07-26T06:47:23.100422+0000","last_active":"2024-07-26T06:48:51.328270+0000","last_peered":"2024-07-26T06:48:51.328270+0000","last_clean":"2024-07-26T06:48:51.328270+0000","last_became_active":"2024-07-26T06:47:23.100023+0000","last_became_peered":"2024-07-26T06:47:23.100023+0000","last_unstale":"2024-07-26T06:48:51.328270+0000","last_undegraded":"2024-07-26T06:48:51.328270+0000","last_fullsized":"2024-07-26T06:48:51.328270+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:28.158112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"80'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338442+0000","last_change":"2024-07-26T06:47:05.037142+0000","last_active":"2024-07-26T06:48:51.338442+0000","last_peered":"2024-07-26T06:48:51.338442+0000","last_clean":"2024-07-26T06:48:51.338442+0000","last_became_active":"2024-07-26T06:47:05.036745+0000","last_became_peered":"2024-07-26T06:47:05.036745+0000","last_unstale":"2024-07-26T06:48:51.338442+0000","last_undegraded":"2024-07-26T06:48:51.338442+0000","last_fullsized":"2024-07-26T06:48:51.338442+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:31:10.034560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"80'7","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.315576+0000","last_change":"2024-07-26T06:47:05.030640+0000","last_active":"2024-07-26T06:48:51.315576+0000","last_peered":"2024-07-26T06:48:51.315576+0000","last_clean":"2024-07-26T06:48:51.315576+0000","last_became_active":"2024-07-26T06:47:05.023583+0000","last_became_peered":"2024-07-26T06:47:05.023583+0000","last_unstale":"2024-07-26T06:48:51.315576+0000","last_undegraded":"2024-07-26T06:48:51.315576+0000","last_fullsized":"2024-07-26T06:48:51.315576+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:59:12.312930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"80'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333051+0000","last_change":"2024-07-26T06:47:05.021768+0000","last_active":"2024-07-26T06:48:51.333051+0000","last_peered":"2024-07-26T06:48:51.333051+0000","last_clean":"2024-07-26T06:48:51.333051+0000","last_became_active":"2024-07-26T06:47:05.021410+0000","last_became_peered":"2024-07-26T06:47:05.021410+0000","last_unstale":"2024-07-26T06:48:51.333051+0000","last_undegraded":"2024-07-26T06:48:51.333051+0000","last_fullsized":"2024-07-26T06:48:51.333051+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:05.576122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"80'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.324272+0000","last_change":"2024-07-26T06:47:05.035814+0000","last_active":"2024-07-26T06:48:51.324272+0000","last_peered":"2024-07-26T06:48:51.324272+0000","last_clean":"2024-07-26T06:48:51.324272+0000","last_became_active":"2024-07-26T06:47:05.035451+0000","last_became_peered":"2024-07-26T06:47:05.035451+0000","last_unstale":"2024-07-26T06:48:51.324272+0000","last_undegraded":"2024-07-26T06:48:51.324272+0000","last_fullsized":"2024-07-26T06:48:51.324272+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:33.533726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"80'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.331253+0000","last_change":"2024-07-26T06:47:05.035720+0000","last_active":"2024-07-26T06:48:51.331253+0000","last_peered":"2024-07-26T06:48:51.331253+0000","last_clean":"2024-07-26T06:48:51.331253+0000","last_became_active":"2024-07-26T06:47:05.035359+0000","last_became_peered":"2024-07-26T06:47:05.035359+0000","last_unstale":"2024-07-26T06:48:51.331253+0000","last_undegraded":"2024-07-26T06:48:51.331253+0000","last_fullsized":"2024-07-26T06:48:51.331253+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:38.561268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"80'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338926+0000","last_change":"2024-07-26T06:47:05.027109+0000","last_active":"2024-07-26T06:48:51.338926+0000","last_peered":"2024-07-26T06:48:51.338926+0000","last_clean":"2024-07-26T06:48:51.338926+0000","last_became_active":"2024-07-26T06:47:05.026694+0000","last_became_peered":"2024-07-26T06:47:05.026694+0000","last_unstale":"2024-07-26T06:48:51.338926+0000","last_undegraded":"2024-07-26T06:48:51.338926+0000","last_fullsized":"2024-07-26T06:48:51.338926+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:25:01.215646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"80'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337344+0000","last_change":"2024-07-26T06:47:05.028212+0000","last_active":"2024-07-26T06:48:51.337344+0000","last_peered":"2024-07-26T06:48:51.337344+0000","last_clean":"2024-07-26T06:48:51.337344+0000","last_became_active":"2024-07-26T06:47:05.027598+0000","last_became_peered":"2024-07-26T06:47:05.027598+0000","last_unstale":"2024-07-26T06:48:51.337344+0000","last_undegraded":"2024-07-26T06:48:51.337344+0000","last_fullsized":"2024-07-26T06:48:51.337344+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:55:31.536194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"80'10","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335959+0000","last_change":"2024-07-26T06:47:05.027724+0000","last_active":"2024-07-26T06:48:51.335959+0000","last_peered":"2024-07-26T06:48:51.335959+0000","last_clean":"2024-07-26T06:48:51.335959+0000","last_became_active":"2024-07-26T06:47:05.027348+0000","last_became_peered":"2024-07-26T06:47:05.027348+0000","last_unstale":"2024-07-26T06:48:51.335959+0000","last_undegraded":"2024-07-26T06:48:51.335959+0000","last_fullsized":"2024-07-26T06:48:51.335959+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:04.180467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"80'7","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.327359+0000","last_change":"2024-07-26T06:47:05.025474+0000","last_active":"2024-07-26T06:48:51.327359+0000","last_peered":"2024-07-26T06:48:51.327359+0000","last_clean":"2024-07-26T06:48:51.327359+0000","last_became_active":"2024-07-26T06:47:05.025016+0000","last_became_peered":"2024-07-26T06:47:05.025016+0000","last_unstale":"2024-07-26T06:48:51.327359+0000","last_undegraded":"2024-07-26T06:48:51.327359+0000","last_fullsized":"2024-07-26T06:48:51.327359+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:40:03.034532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"80'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.330259+0000","last_change":"2024-07-26T06:47:05.027533+0000","last_active":"2024-07-26T06:48:51.330259+0000","last_peered":"2024-07-26T06:48:51.330259+0000","last_clean":"2024-07-26T06:48:51.330259+0000","last_became_active":"2024-07-26T06:47:05.027143+0000","last_became_peered":"2024-07-26T06:47:05.027143+0000","last_unstale":"2024-07-26T06:48:51.330259+0000","last_undegraded":"2024-07-26T06:48:51.330259+0000","last_fullsized":"2024-07-26T06:48:51.330259+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:13.406430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"80'11","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338113+0000","last_change":"2024-07-26T06:47:05.025962+0000","last_active":"2024-07-26T06:48:51.338113+0000","last_peered":"2024-07-26T06:48:51.338113+0000","last_clean":"2024-07-26T06:48:51.338113+0000","last_became_active":"2024-07-26T06:47:05.025621+0000","last_became_peered":"2024-07-26T06:47:05.025621+0000","last_unstale":"2024-07-26T06:48:51.338113+0000","last_undegraded":"2024-07-26T06:48:51.338113+0000","last_fullsized":"2024-07-26T06:48:51.338113+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:56:43.669247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"80'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337312+0000","last_change":"2024-07-26T06:47:05.026252+0000","last_active":"2024-07-26T06:48:51.337312+0000","last_peered":"2024-07-26T06:48:51.337312+0000","last_clean":"2024-07-26T06:48:51.337312+0000","last_became_active":"2024-07-26T06:47:05.025875+0000","last_became_peered":"2024-07-26T06:47:05.025875+0000","last_unstale":"2024-07-26T06:48:51.337312+0000","last_undegraded":"2024-07-26T06:48:51.337312+0000","last_fullsized":"2024-07-26T06:48:51.337312+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:27:32.955862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"80'8","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326328+0000","last_change":"2024-07-26T06:47:05.022513+0000","last_active":"2024-07-26T06:48:51.326328+0000","last_peered":"2024-07-26T06:48:51.326328+0000","last_clean":"2024-07-26T06:48:51.326328+0000","last_became_active":"2024-07-26T06:47:05.022159+0000","last_became_peered":"2024-07-26T06:47:05.022159+0000","last_unstale":"2024-07-26T06:48:51.326328+0000","last_undegraded":"2024-07-26T06:48:51.326328+0000","last_fullsized":"2024-07-26T06:48:51.326328+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:53:06.402985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"80'6","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.325210+0000","last_change":"2024-07-26T06:47:23.097604+0000","last_active":"2024-07-26T06:48:51.325210+0000","last_peered":"2024-07-26T06:48:51.325210+0000","last_clean":"2024-07-26T06:48:51.325210+0000","last_became_active":"2024-07-26T06:47:23.097170+0000","last_became_peered":"2024-07-26T06:47:23.097170+0000","last_unstale":"2024-07-26T06:48:51.325210+0000","last_undegraded":"2024-07-26T06:48:51.325210+0000","last_fullsized":"2024-07-26T06:48:51.325210+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:38:24.530396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"80'2","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.320538+0000","last_change":"2024-07-26T06:47:05.026349+0000","last_active":"2024-07-26T06:48:51.320538+0000","last_peered":"2024-07-26T06:48:51.320538+0000","last_clean":"2024-07-26T06:48:51.320538+0000","last_became_active":"2024-07-26T06:47:05.026016+0000","last_became_peered":"2024-07-26T06:47:05.026016+0000","last_unstale":"2024-07-26T06:48:51.320538+0000","last_undegraded":"2024-07-26T06:48:51.320538+0000","last_fullsized":"2024-07-26T06:48:51.320538+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:19:02.264183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"80'15","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.323233+0000","last_change":"2024-07-26T06:47:05.032439+0000","last_active":"2024-07-26T06:48:51.323233+0000","last_peered":"2024-07-26T06:48:51.323233+0000","last_clean":"2024-07-26T06:48:51.323233+0000","last_became_active":"2024-07-26T06:47:05.032105+0000","last_became_peered":"2024-07-26T06:47:05.032105+0000","last_unstale":"2024-07-26T06:48:51.323233+0000","last_undegraded":"2024-07-26T06:48:51.323233+0000","last_fullsized":"2024-07-26T06:48:51.323233+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:21:50.423939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"80'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.319169+0000","last_change":"2024-07-26T06:47:05.026927+0000","last_active":"2024-07-26T06:48:51.319169+0000","last_peered":"2024-07-26T06:48:51.319169+0000","last_clean":"2024-07-26T06:48:51.319169+0000","last_became_active":"2024-07-26T06:47:05.026569+0000","last_became_peered":"2024-07-26T06:47:05.026569+0000","last_unstale":"2024-07-26T06:48:51.319169+0000","last_undegraded":"2024-07-26T06:48:51.319169+0000","last_fullsized":"2024-07-26T06:48:51.319169+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:12:14.687633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"80'4","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333414+0000","last_change":"2024-07-26T06:47:23.099689+0000","last_active":"2024-07-26T06:48:51.333414+0000","last_peered":"2024-07-26T06:48:51.333414+0000","last_clean":"2024-07-26T06:48:51.333414+0000","last_became_active":"2024-07-26T06:47:23.099313+0000","last_became_peered":"2024-07-26T06:47:23.099313+0000","last_unstale":"2024-07-26T06:48:51.333414+0000","last_undegraded":"2024-07-26T06:48:51.333414+0000","last_fullsized":"2024-07-26T06:48:51.333414+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:49:01.833297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"80'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333205+0000","last_change":"2024-07-26T06:47:05.034864+0000","last_active":"2024-07-26T06:48:51.333205+0000","last_peered":"2024-07-26T06:48:51.333205+0000","last_clean":"2024-07-26T06:48:51.333205+0000","last_became_active":"2024-07-26T06:47:05.033414+0000","last_became_peered":"2024-07-26T06:47:05.033414+0000","last_unstale":"2024-07-26T06:48:51.333205+0000","last_undegraded":"2024-07-26T06:48:51.333205+0000","last_fullsized":"2024-07-26T06:48:51.333205+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:21:31.420212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"80'8","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.324215+0000","last_change":"2024-07-26T06:47:05.025462+0000","last_active":"2024-07-26T06:48:51.324215+0000","last_peered":"2024-07-26T06:48:51.324215+0000","last_clean":"2024-07-26T06:48:51.324215+0000","last_became_active":"2024-07-26T06:47:05.025049+0000","last_became_peered":"2024-07-26T06:47:05.025049+0000","last_unstale":"2024-07-26T06:48:51.324215+0000","last_undegraded":"2024-07-26T06:48:51.324215+0000","last_fullsized":"2024-07-26T06:48:51.324215+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:59:28.903338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"80'12","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.330035+0000","last_change":"2024-07-26T06:47:05.026394+0000","last_active":"2024-07-26T06:48:51.330035+0000","last_peered":"2024-07-26T06:48:51.330035+0000","last_clean":"2024-07-26T06:48:51.330035+0000","last_became_active":"2024-07-26T06:47:05.025820+0000","last_became_peered":"2024-07-26T06:47:05.025820+0000","last_unstale":"2024-07-26T06:48:51.330035+0000","last_undegraded":"2024-07-26T06:48:51.330035+0000","last_fullsized":"2024-07-26T06:48:51.330035+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:44:57.441638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"80'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328916+0000","last_change":"2024-07-26T06:47:05.024373+0000","last_active":"2024-07-26T06:48:51.328916+0000","last_peered":"2024-07-26T06:48:51.328916+0000","last_clean":"2024-07-26T06:48:51.328916+0000","last_became_active":"2024-07-26T06:47:05.023962+0000","last_became_peered":"2024-07-26T06:47:05.023962+0000","last_unstale":"2024-07-26T06:48:51.328916+0000","last_undegraded":"2024-07-26T06:48:51.328916+0000","last_fullsized":"2024-07-26T06:48:51.328916+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:43.804614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"80'7","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.325261+0000","last_change":"2024-07-26T06:47:05.031146+0000","last_active":"2024-07-26T06:48:51.325261+0000","last_peered":"2024-07-26T06:48:51.325261+0000","last_clean":"2024-07-26T06:48:51.325261+0000","last_became_active":"2024-07-26T06:47:05.029393+0000","last_became_peered":"2024-07-26T06:47:05.029393+0000","last_unstale":"2024-07-26T06:48:51.325261+0000","last_undegraded":"2024-07-26T06:48:51.325261+0000","last_fullsized":"2024-07-26T06:48:51.325261+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:46.853933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"80'7","reported_seq":140,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:05.281666+0000","last_change":"2024-07-26T06:47:02.001800+0000","last_active":"2024-07-26T06:49:05.281666+0000","last_peered":"2024-07-26T06:49:05.281666+0000","last_clean":"2024-07-26T06:49:05.281666+0000","last_became_active":"2024-07-26T06:47:02.001435+0000","last_became_peered":"2024-07-26T06:47:02.001435+0000","last_unstale":"2024-07-26T06:49:05.281666+0000","last_undegraded":"2024-07-26T06:49:05.281666+0000","last_fullsized":"2024-07-26T06:49:05.281666+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:23.121378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"80'8","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326709+0000","last_change":"2024-07-26T06:47:05.028301+0000","last_active":"2024-07-26T06:48:51.326709+0000","last_peered":"2024-07-26T06:48:51.326709+0000","last_clean":"2024-07-26T06:48:51.326709+0000","last_became_active":"2024-07-26T06:47:05.027761+0000","last_became_peered":"2024-07-26T06:47:05.027761+0000","last_unstale":"2024-07-26T06:48:51.326709+0000","last_undegraded":"2024-07-26T06:48:51.326709+0000","last_fullsized":"2024-07-26T06:48:51.326709+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:19.109395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093760+0000","last_change":"2024-07-26T06:47:02.004488+0000","last_active":"2024-07-26T06:47:23.093760+0000","last_peered":"2024-07-26T06:47:23.093760+0000","last_clean":"2024-07-26T06:47:23.093760+0000","last_became_active":"2024-07-26T06:47:02.004163+0000","last_became_peered":"2024-07-26T06:47:02.004163+0000","last_unstale":"2024-07-26T06:47:23.093760+0000","last_undegraded":"2024-07-26T06:47:23.093760+0000","last_fullsized":"2024-07-26T06:47:23.093760+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:16.221618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096069+0000","last_change":"2024-07-26T06:47:01.992708+0000","last_active":"2024-07-26T06:47:23.096069+0000","last_peered":"2024-07-26T06:47:23.096069+0000","last_clean":"2024-07-26T06:47:23.096069+0000","last_became_active":"2024-07-26T06:47:01.992330+0000","last_became_peered":"2024-07-26T06:47:01.992330+0000","last_unstale":"2024-07-26T06:47:23.096069+0000","last_undegraded":"2024-07-26T06:47:23.096069+0000","last_fullsized":"2024-07-26T06:47:23.096069+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:25.951946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"80'8","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.330585+0000","last_change":"2024-07-26T06:47:05.013946+0000","last_active":"2024-07-26T06:48:51.330585+0000","last_peered":"2024-07-26T06:48:51.330585+0000","last_clean":"2024-07-26T06:48:51.330585+0000","last_became_active":"2024-07-26T06:47:05.013509+0000","last_became_peered":"2024-07-26T06:47:05.013509+0000","last_unstale":"2024-07-26T06:48:51.330585+0000","last_undegraded":"2024-07-26T06:48:51.330585+0000","last_fullsized":"2024-07-26T06:48:51.330585+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:09.789287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093848+0000","last_change":"2024-07-26T06:47:01.997617+0000","last_active":"2024-07-26T06:47:23.093848+0000","last_peered":"2024-07-26T06:47:23.093848+0000","last_clean":"2024-07-26T06:47:23.093848+0000","last_became_active":"2024-07-26T06:47:01.997238+0000","last_became_peered":"2024-07-26T06:47:01.997238+0000","last_unstale":"2024-07-26T06:47:23.093848+0000","last_undegraded":"2024-07-26T06:47:23.093848+0000","last_fullsized":"2024-07-26T06:47:23.093848+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:45.596438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"80'11","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328388+0000","last_change":"2024-07-26T06:47:23.099813+0000","last_active":"2024-07-26T06:48:51.328388+0000","last_peered":"2024-07-26T06:48:51.328388+0000","last_clean":"2024-07-26T06:48:51.328388+0000","last_became_active":"2024-07-26T06:47:23.098701+0000","last_became_peered":"2024-07-26T06:47:23.098701+0000","last_unstale":"2024-07-26T06:48:51.328388+0000","last_undegraded":"2024-07-26T06:48:51.328388+0000","last_fullsized":"2024-07-26T06:48:51.328388+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:42:04.059637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'5","reported_seq":231,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:03.277803+0000","last_change":"2024-07-26T06:47:01.998508+0000","last_active":"2024-07-26T06:49:03.277803+0000","last_peered":"2024-07-26T06:49:03.277803+0000","last_clean":"2024-07-26T06:49:03.277803+0000","last_became_active":"2024-07-26T06:47:01.998132+0000","last_became_peered":"2024-07-26T06:47:01.998132+0000","last_unstale":"2024-07-26T06:49:03.277803+0000","last_undegraded":"2024-07-26T06:49:03.277803+0000","last_fullsized":"2024-07-26T06:49:03.277803+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:21.588085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":200,"num_read_kb":200,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"80'10","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338987+0000","last_change":"2024-07-26T06:47:05.032704+0000","last_active":"2024-07-26T06:48:51.338987+0000","last_peered":"2024-07-26T06:48:51.338987+0000","last_clean":"2024-07-26T06:48:51.338987+0000","last_became_active":"2024-07-26T06:47:05.031843+0000","last_became_peered":"2024-07-26T06:47:05.031843+0000","last_unstale":"2024-07-26T06:48:51.338987+0000","last_undegraded":"2024-07-26T06:48:51.338987+0000","last_fullsized":"2024-07-26T06:48:51.338987+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:58:24.436233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097398+0000","last_change":"2024-07-26T06:47:02.006266+0000","last_active":"2024-07-26T06:47:23.097398+0000","last_peered":"2024-07-26T06:47:23.097398+0000","last_clean":"2024-07-26T06:47:23.097398+0000","last_became_active":"2024-07-26T06:47:02.005830+0000","last_became_peered":"2024-07-26T06:47:02.005830+0000","last_unstale":"2024-07-26T06:47:23.097398+0000","last_undegraded":"2024-07-26T06:47:23.097398+0000","last_fullsized":"2024-07-26T06:47:23.097398+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:06:09.097781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"80'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.331339+0000","last_change":"2024-07-26T06:47:05.024820+0000","last_active":"2024-07-26T06:48:51.331339+0000","last_peered":"2024-07-26T06:48:51.331339+0000","last_clean":"2024-07-26T06:48:51.331339+0000","last_became_active":"2024-07-26T06:47:05.023941+0000","last_became_peered":"2024-07-26T06:47:05.023941+0000","last_unstale":"2024-07-26T06:48:51.331339+0000","last_undegraded":"2024-07-26T06:48:51.331339+0000","last_fullsized":"2024-07-26T06:48:51.331339+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:07.885159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095379+0000","last_change":"2024-07-26T06:47:02.023308+0000","last_active":"2024-07-26T06:47:23.095379+0000","last_peered":"2024-07-26T06:47:23.095379+0000","last_clean":"2024-07-26T06:47:23.095379+0000","last_became_active":"2024-07-26T06:47:02.022933+0000","last_became_peered":"2024-07-26T06:47:02.022933+0000","last_unstale":"2024-07-26T06:47:23.095379+0000","last_undegraded":"2024-07-26T06:47:23.095379+0000","last_fullsized":"2024-07-26T06:47:23.095379+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:59:01.233386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"80'11","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.340434+0000","last_change":"2024-07-26T06:47:05.026005+0000","last_active":"2024-07-26T06:48:51.340434+0000","last_peered":"2024-07-26T06:48:51.340434+0000","last_clean":"2024-07-26T06:48:51.340434+0000","last_became_active":"2024-07-26T06:47:05.025599+0000","last_became_peered":"2024-07-26T06:47:05.025599+0000","last_unstale":"2024-07-26T06:48:51.340434+0000","last_undegraded":"2024-07-26T06:48:51.340434+0000","last_fullsized":"2024-07-26T06:48:51.340434+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:44:19.924157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'34","reported_seq":289,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:03.277912+0000","last_change":"2024-07-26T06:47:23.393050+0000","last_active":"2024-07-26T06:49:03.277912+0000","last_peered":"2024-07-26T06:49:03.277912+0000","last_clean":"2024-07-26T06:49:03.277912+0000","last_became_active":"2024-07-26T06:47:23.381611+0000","last_became_peered":"2024-07-26T06:47:23.381611+0000","last_unstale":"2024-07-26T06:49:03.277912+0000","last_undegraded":"2024-07-26T06:49:03.277912+0000","last_fullsized":"2024-07-26T06:49:03.277912+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:23.866523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62075,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":210,"num_read_kb":165,"num_write":36,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"80'6","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335552+0000","last_change":"2024-07-26T06:47:05.034190+0000","last_active":"2024-07-26T06:48:51.335552+0000","last_peered":"2024-07-26T06:48:51.335552+0000","last_clean":"2024-07-26T06:48:51.335552+0000","last_became_active":"2024-07-26T06:47:05.033569+0000","last_became_peered":"2024-07-26T06:47:05.033569+0000","last_unstale":"2024-07-26T06:48:51.335552+0000","last_undegraded":"2024-07-26T06:48:51.335552+0000","last_fullsized":"2024-07-26T06:48:51.335552+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:39.306996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.120074+0000","last_change":"2024-07-26T06:47:02.001718+0000","last_active":"2024-07-26T06:47:24.120074+0000","last_peered":"2024-07-26T06:47:24.120074+0000","last_clean":"2024-07-26T06:47:24.120074+0000","last_became_active":"2024-07-26T06:47:02.001359+0000","last_became_peered":"2024-07-26T06:47:02.001359+0000","last_unstale":"2024-07-26T06:47:24.120074+0000","last_undegraded":"2024-07-26T06:47:24.120074+0000","last_fullsized":"2024-07-26T06:47:24.120074+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:03:23.390279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"80'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337734+0000","last_change":"2024-07-26T06:47:05.027445+0000","last_active":"2024-07-26T06:48:51.337734+0000","last_peered":"2024-07-26T06:48:51.337734+0000","last_clean":"2024-07-26T06:48:51.337734+0000","last_became_active":"2024-07-26T06:47:05.027114+0000","last_became_peered":"2024-07-26T06:47:05.027114+0000","last_unstale":"2024-07-26T06:48:51.337734+0000","last_undegraded":"2024-07-26T06:48:51.337734+0000","last_fullsized":"2024-07-26T06:48:51.337734+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:29:10.519271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.094571+0000","last_change":"2024-07-26T06:47:01.998832+0000","last_active":"2024-07-26T06:47:23.094571+0000","last_peered":"2024-07-26T06:47:23.094571+0000","last_clean":"2024-07-26T06:47:23.094571+0000","last_became_active":"2024-07-26T06:47:01.998512+0000","last_became_peered":"2024-07-26T06:47:01.998512+0000","last_unstale":"2024-07-26T06:47:23.094571+0000","last_undegraded":"2024-07-26T06:47:23.094571+0000","last_fullsized":"2024-07-26T06:47:23.094571+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:27.722329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"80'15","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.340035+0000","last_change":"2024-07-26T06:47:05.027803+0000","last_active":"2024-07-26T06:48:51.340035+0000","last_peered":"2024-07-26T06:48:51.340035+0000","last_clean":"2024-07-26T06:48:51.340035+0000","last_became_active":"2024-07-26T06:47:05.027320+0000","last_became_peered":"2024-07-26T06:47:05.027320+0000","last_unstale":"2024-07-26T06:48:51.340035+0000","last_undegraded":"2024-07-26T06:48:51.340035+0000","last_fullsized":"2024-07-26T06:48:51.340035+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:43:06.242945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'7","reported_seq":137,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:01.275653+0000","last_change":"2024-07-26T06:47:01.998356+0000","last_active":"2024-07-26T06:49:01.275653+0000","last_peered":"2024-07-26T06:49:01.275653+0000","last_clean":"2024-07-26T06:49:01.275653+0000","last_became_active":"2024-07-26T06:47:01.997922+0000","last_became_peered":"2024-07-26T06:47:01.997922+0000","last_unstale":"2024-07-26T06:49:01.275653+0000","last_undegraded":"2024-07-26T06:49:01.275653+0000","last_fullsized":"2024-07-26T06:49:01.275653+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:50:58.066682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"80'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339316+0000","last_change":"2024-07-26T06:47:05.025285+0000","last_active":"2024-07-26T06:48:51.339316+0000","last_peered":"2024-07-26T06:48:51.339316+0000","last_clean":"2024-07-26T06:48:51.339316+0000","last_became_active":"2024-07-26T06:47:05.024945+0000","last_became_peered":"2024-07-26T06:47:05.024945+0000","last_unstale":"2024-07-26T06:48:51.339316+0000","last_undegraded":"2024-07-26T06:48:51.339316+0000","last_fullsized":"2024-07-26T06:48:51.339316+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:21:04.680435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093595+0000","last_change":"2024-07-26T06:47:02.024895+0000","last_active":"2024-07-26T06:47:23.093595+0000","last_peered":"2024-07-26T06:47:23.093595+0000","last_clean":"2024-07-26T06:47:23.093595+0000","last_became_active":"2024-07-26T06:47:02.024451+0000","last_became_peered":"2024-07-26T06:47:02.024451+0000","last_unstale":"2024-07-26T06:47:23.093595+0000","last_undegraded":"2024-07-26T06:47:23.093595+0000","last_fullsized":"2024-07-26T06:47:23.093595+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:05:13.396499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"80'6","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335375+0000","last_change":"2024-07-26T06:47:23.381060+0000","last_active":"2024-07-26T06:48:51.335375+0000","last_peered":"2024-07-26T06:48:51.335375+0000","last_clean":"2024-07-26T06:48:51.335375+0000","last_became_active":"2024-07-26T06:47:23.380685+0000","last_became_peered":"2024-07-26T06:47:23.380685+0000","last_unstale":"2024-07-26T06:48:51.335375+0000","last_undegraded":"2024-07-26T06:48:51.335375+0000","last_fullsized":"2024-07-26T06:48:51.335375+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:21:38.062900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093109+0000","last_change":"2024-07-26T06:47:02.002098+0000","last_active":"2024-07-26T06:47:23.093109+0000","last_peered":"2024-07-26T06:47:23.093109+0000","last_clean":"2024-07-26T06:47:23.093109+0000","last_became_active":"2024-07-26T06:47:02.001428+0000","last_became_peered":"2024-07-26T06:47:02.001428+0000","last_unstale":"2024-07-26T06:47:23.093109+0000","last_undegraded":"2024-07-26T06:47:23.093109+0000","last_fullsized":"2024-07-26T06:47:23.093109+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:52:29.112500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"80'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.329946+0000","last_change":"2024-07-26T06:47:05.023686+0000","last_active":"2024-07-26T06:48:51.329946+0000","last_peered":"2024-07-26T06:48:51.329946+0000","last_clean":"2024-07-26T06:48:51.329946+0000","last_became_active":"2024-07-26T06:47:05.022583+0000","last_became_peered":"2024-07-26T06:47:05.022583+0000","last_unstale":"2024-07-26T06:48:51.329946+0000","last_undegraded":"2024-07-26T06:48:51.329946+0000","last_fullsized":"2024-07-26T06:48:51.329946+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:29:03.752429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096434+0000","last_change":"2024-07-26T06:47:02.006040+0000","last_active":"2024-07-26T06:47:23.096434+0000","last_peered":"2024-07-26T06:47:23.096434+0000","last_clean":"2024-07-26T06:47:23.096434+0000","last_became_active":"2024-07-26T06:47:02.005586+0000","last_became_peered":"2024-07-26T06:47:02.005586+0000","last_unstale":"2024-07-26T06:47:23.096434+0000","last_undegraded":"2024-07-26T06:47:23.096434+0000","last_fullsized":"2024-07-26T06:47:23.096434+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:09:27.180873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"80'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335167+0000","last_change":"2024-07-26T06:47:05.035524+0000","last_active":"2024-07-26T06:48:51.335167+0000","last_peered":"2024-07-26T06:48:51.335167+0000","last_clean":"2024-07-26T06:48:51.335167+0000","last_became_active":"2024-07-26T06:47:05.035078+0000","last_became_peered":"2024-07-26T06:47:05.035078+0000","last_unstale":"2024-07-26T06:48:51.335167+0000","last_undegraded":"2024-07-26T06:48:51.335167+0000","last_fullsized":"2024-07-26T06:48:51.335167+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:03.758261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.095730+0000","last_change":"2024-07-26T06:47:01.999810+0000","last_active":"2024-07-26T06:47:23.095730+0000","last_peered":"2024-07-26T06:47:23.095730+0000","last_clean":"2024-07-26T06:47:23.095730+0000","last_became_active":"2024-07-26T06:47:01.999460+0000","last_became_peered":"2024-07-26T06:47:01.999460+0000","last_unstale":"2024-07-26T06:47:23.095730+0000","last_undegraded":"2024-07-26T06:47:23.095730+0000","last_fullsized":"2024-07-26T06:47:23.095730+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:25:05.691900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339575+0000","last_change":"2024-07-26T06:47:05.033631+0000","last_active":"2024-07-26T06:48:51.339575+0000","last_peered":"2024-07-26T06:48:51.339575+0000","last_clean":"2024-07-26T06:48:51.339575+0000","last_became_active":"2024-07-26T06:47:05.033266+0000","last_became_peered":"2024-07-26T06:47:05.033266+0000","last_unstale":"2024-07-26T06:48:51.339575+0000","last_undegraded":"2024-07-26T06:48:51.339575+0000","last_fullsized":"2024-07-26T06:48:51.339575+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:55.895271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"78'5","reported_seq":137,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:05.301099+0000","last_change":"2024-07-26T06:47:02.113525+0000","last_active":"2024-07-26T06:49:05.301099+0000","last_peered":"2024-07-26T06:49:05.301099+0000","last_clean":"2024-07-26T06:49:05.301099+0000","last_became_active":"2024-07-26T06:47:02.113112+0000","last_became_peered":"2024-07-26T06:47:02.113112+0000","last_unstale":"2024-07-26T06:49:05.301099+0000","last_undegraded":"2024-07-26T06:49:05.301099+0000","last_fullsized":"2024-07-26T06:49:05.301099+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:05:15.071264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":105,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"80'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338314+0000","last_change":"2024-07-26T06:47:05.026423+0000","last_active":"2024-07-26T06:48:51.338314+0000","last_peered":"2024-07-26T06:48:51.338314+0000","last_clean":"2024-07-26T06:48:51.338314+0000","last_became_active":"2024-07-26T06:47:05.026053+0000","last_became_peered":"2024-07-26T06:47:05.026053+0000","last_unstale":"2024-07-26T06:48:51.338314+0000","last_undegraded":"2024-07-26T06:48:51.338314+0000","last_fullsized":"2024-07-26T06:48:51.338314+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:07:20.174538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":102,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097268+0000","last_change":"2024-07-26T06:45:40.184988+0000","last_active":"2024-07-26T06:47:23.097268+0000","last_peered":"2024-07-26T06:47:23.097268+0000","last_clean":"2024-07-26T06:47:23.097268+0000","last_became_active":"2024-07-26T06:45:40.178827+0000","last_became_peered":"2024-07-26T06:45:40.178827+0000","last_unstale":"2024-07-26T06:47:23.097268+0000","last_undegraded":"2024-07-26T06:47:23.097268+0000","last_fullsized":"2024-07-26T06:47:23.097268+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:42:28.257602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:42:28.257602+0000","last_clean_scrub_stamp":"2024-07-26T06:42:28.257602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:52:14.123696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.120218+0000","last_change":"2024-07-26T06:47:01.996610+0000","last_active":"2024-07-26T06:47:24.120218+0000","last_peered":"2024-07-26T06:47:24.120218+0000","last_clean":"2024-07-26T06:47:24.120218+0000","last_became_active":"2024-07-26T06:47:01.996155+0000","last_became_peered":"2024-07-26T06:47:01.996155+0000","last_unstale":"2024-07-26T06:47:24.120218+0000","last_undegraded":"2024-07-26T06:47:24.120218+0000","last_fullsized":"2024-07-26T06:47:24.120218+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:51:30.420254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"80'8","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333651+0000","last_change":"2024-07-26T06:47:05.027699+0000","last_active":"2024-07-26T06:48:51.333651+0000","last_peered":"2024-07-26T06:48:51.333651+0000","last_clean":"2024-07-26T06:48:51.333651+0000","last_became_active":"2024-07-26T06:47:05.027348+0000","last_became_peered":"2024-07-26T06:47:05.027348+0000","last_unstale":"2024-07-26T06:48:51.333651+0000","last_undegraded":"2024-07-26T06:48:51.333651+0000","last_fullsized":"2024-07-26T06:48:51.333651+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:28:42.664351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097188+0000","last_change":"2024-07-26T06:47:02.003205+0000","last_active":"2024-07-26T06:47:23.097188+0000","last_peered":"2024-07-26T06:47:23.097188+0000","last_clean":"2024-07-26T06:47:23.097188+0000","last_became_active":"2024-07-26T06:47:02.002543+0000","last_became_peered":"2024-07-26T06:47:02.002543+0000","last_unstale":"2024-07-26T06:47:23.097188+0000","last_undegraded":"2024-07-26T06:47:23.097188+0000","last_fullsized":"2024-07-26T06:47:23.097188+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:43:10.911943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"80'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.334496+0000","last_change":"2024-07-26T06:47:05.023258+0000","last_active":"2024-07-26T06:48:51.334496+0000","last_peered":"2024-07-26T06:48:51.334496+0000","last_clean":"2024-07-26T06:48:51.334496+0000","last_became_active":"2024-07-26T06:47:05.022903+0000","last_became_peered":"2024-07-26T06:47:05.022903+0000","last_unstale":"2024-07-26T06:48:51.334496+0000","last_undegraded":"2024-07-26T06:48:51.334496+0000","last_fullsized":"2024-07-26T06:48:51.334496+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:08.155850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"80'6","reported_seq":137,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:06.282714+0000","last_change":"2024-07-26T06:47:02.022316+0000","last_active":"2024-07-26T06:49:06.282714+0000","last_peered":"2024-07-26T06:49:06.282714+0000","last_clean":"2024-07-26T06:49:06.282714+0000","last_became_active":"2024-07-26T06:47:02.021923+0000","last_became_peered":"2024-07-26T06:47:02.021923+0000","last_unstale":"2024-07-26T06:49:06.282714+0000","last_undegraded":"2024-07-26T06:49:06.282714+0000","last_fullsized":"2024-07-26T06:49:06.282714+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:29:31.533449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"80'8","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328077+0000","last_change":"2024-07-26T06:47:05.021510+0000","last_active":"2024-07-26T06:48:51.328077+0000","last_peered":"2024-07-26T06:48:51.328077+0000","last_clean":"2024-07-26T06:48:51.328077+0000","last_became_active":"2024-07-26T06:47:05.021140+0000","last_became_peered":"2024-07-26T06:47:05.021140+0000","last_unstale":"2024-07-26T06:48:51.328077+0000","last_undegraded":"2024-07-26T06:48:51.328077+0000","last_fullsized":"2024-07-26T06:48:51.328077+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:37:01.340126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"79'5","reported_seq":132,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:02.308916+0000","last_change":"2024-07-26T06:47:02.001698+0000","last_active":"2024-07-26T06:49:02.308916+0000","last_peered":"2024-07-26T06:49:02.308916+0000","last_clean":"2024-07-26T06:49:02.308916+0000","last_became_active":"2024-07-26T06:47:02.001244+0000","last_became_peered":"2024-07-26T06:47:02.001244+0000","last_unstale":"2024-07-26T06:49:02.308916+0000","last_undegraded":"2024-07-26T06:49:02.308916+0000","last_fullsized":"2024-07-26T06:49:02.308916+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:32:07.920065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"80'7","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.321186+0000","last_change":"2024-07-26T06:47:05.029524+0000","last_active":"2024-07-26T06:48:51.321186+0000","last_peered":"2024-07-26T06:48:51.321186+0000","last_clean":"2024-07-26T06:48:51.321186+0000","last_became_active":"2024-07-26T06:47:05.028981+0000","last_became_peered":"2024-07-26T06:47:05.028981+0000","last_unstale":"2024-07-26T06:48:51.321186+0000","last_undegraded":"2024-07-26T06:48:51.321186+0000","last_fullsized":"2024-07-26T06:48:51.321186+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:49:57.019486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092276+0000","last_change":"2024-07-26T06:47:02.025117+0000","last_active":"2024-07-26T06:47:23.092276+0000","last_peered":"2024-07-26T06:47:23.092276+0000","last_clean":"2024-07-26T06:47:23.092276+0000","last_became_active":"2024-07-26T06:47:02.024612+0000","last_became_peered":"2024-07-26T06:47:02.024612+0000","last_unstale":"2024-07-26T06:47:23.092276+0000","last_undegraded":"2024-07-26T06:47:23.092276+0000","last_fullsized":"2024-07-26T06:47:23.092276+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:15:44.999658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.324434+0000","last_change":"2024-07-26T06:47:05.015085+0000","last_active":"2024-07-26T06:48:51.324434+0000","last_peered":"2024-07-26T06:48:51.324434+0000","last_clean":"2024-07-26T06:48:51.324434+0000","last_became_active":"2024-07-26T06:47:05.014686+0000","last_became_peered":"2024-07-26T06:47:05.014686+0000","last_unstale":"2024-07-26T06:48:51.324434+0000","last_undegraded":"2024-07-26T06:48:51.324434+0000","last_fullsized":"2024-07-26T06:48:51.324434+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:39:30.392347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"80'4","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.159121+0000","last_change":"2024-07-26T06:47:23.111918+0000","last_active":"2024-07-26T06:47:23.159121+0000","last_peered":"2024-07-26T06:47:23.159121+0000","last_clean":"2024-07-26T06:47:23.159121+0000","last_became_active":"2024-07-26T06:47:23.099518+0000","last_became_peered":"2024-07-26T06:47:23.099518+0000","last_unstale":"2024-07-26T06:47:23.159121+0000","last_undegraded":"2024-07-26T06:47:23.159121+0000","last_fullsized":"2024-07-26T06:47:23.159121+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:41.012096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"80'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.331791+0000","last_change":"2024-07-26T06:47:05.019415+0000","last_active":"2024-07-26T06:48:51.331791+0000","last_peered":"2024-07-26T06:48:51.331791+0000","last_clean":"2024-07-26T06:48:51.331791+0000","last_became_active":"2024-07-26T06:47:05.019044+0000","last_became_peered":"2024-07-26T06:47:05.019044+0000","last_unstale":"2024-07-26T06:48:51.331791+0000","last_undegraded":"2024-07-26T06:48:51.331791+0000","last_fullsized":"2024-07-26T06:48:51.331791+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:02:37.228174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'15","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:47.027153+0000","last_change":"2024-07-26T06:47:02.006585+0000","last_active":"2024-07-26T06:47:47.027153+0000","last_peered":"2024-07-26T06:47:47.027153+0000","last_clean":"2024-07-26T06:47:47.027153+0000","last_became_active":"2024-07-26T06:47:02.006185+0000","last_became_peered":"2024-07-26T06:47:02.006185+0000","last_unstale":"2024-07-26T06:47:47.027153+0000","last_undegraded":"2024-07-26T06:47:47.027153+0000","last_fullsized":"2024-07-26T06:47:47.027153+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:02:45.480712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"80'5","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326811+0000","last_change":"2024-07-26T06:47:05.028840+0000","last_active":"2024-07-26T06:48:51.326811+0000","last_peered":"2024-07-26T06:48:51.326811+0000","last_clean":"2024-07-26T06:48:51.326811+0000","last_became_active":"2024-07-26T06:47:05.028487+0000","last_became_peered":"2024-07-26T06:47:05.028487+0000","last_unstale":"2024-07-26T06:48:51.326811+0000","last_undegraded":"2024-07-26T06:48:51.326811+0000","last_fullsized":"2024-07-26T06:48:51.326811+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:47:48.185227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.380389+0000","last_change":"2024-07-26T06:47:02.113691+0000","last_active":"2024-07-26T06:47:23.380389+0000","last_peered":"2024-07-26T06:47:23.380389+0000","last_clean":"2024-07-26T06:47:23.380389+0000","last_became_active":"2024-07-26T06:47:02.113342+0000","last_became_peered":"2024-07-26T06:47:02.113342+0000","last_unstale":"2024-07-26T06:47:23.380389+0000","last_undegraded":"2024-07-26T06:47:23.380389+0000","last_fullsized":"2024-07-26T06:47:23.380389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:14:03.586564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"80'7","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.332485+0000","last_change":"2024-07-26T06:47:23.098420+0000","last_active":"2024-07-26T06:48:51.332485+0000","last_peered":"2024-07-26T06:48:51.332485+0000","last_clean":"2024-07-26T06:48:51.332485+0000","last_became_active":"2024-07-26T06:47:23.097981+0000","last_became_peered":"2024-07-26T06:47:23.097981+0000","last_unstale":"2024-07-26T06:48:51.332485+0000","last_undegraded":"2024-07-26T06:48:51.332485+0000","last_fullsized":"2024-07-26T06:48:51.332485+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:38:40.155514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091072+0000","last_change":"2024-07-26T06:47:02.024781+0000","last_active":"2024-07-26T06:47:23.091072+0000","last_peered":"2024-07-26T06:47:23.091072+0000","last_clean":"2024-07-26T06:47:23.091072+0000","last_became_active":"2024-07-26T06:47:02.024492+0000","last_became_peered":"2024-07-26T06:47:02.024492+0000","last_unstale":"2024-07-26T06:47:23.091072+0000","last_undegraded":"2024-07-26T06:47:23.091072+0000","last_fullsized":"2024-07-26T06:47:23.091072+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:23:41.323963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"80'2","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.283302+0000","last_change":"2024-07-26T06:47:05.033052+0000","last_active":"2024-07-26T06:48:51.283302+0000","last_peered":"2024-07-26T06:48:51.283302+0000","last_clean":"2024-07-26T06:48:51.283302+0000","last_became_active":"2024-07-26T06:47:05.032052+0000","last_became_peered":"2024-07-26T06:47:05.032052+0000","last_unstale":"2024-07-26T06:48:51.283302+0000","last_undegraded":"2024-07-26T06:48:51.283302+0000","last_fullsized":"2024-07-26T06:48:51.283302+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:42:04.683586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090564+0000","last_change":"2024-07-26T06:47:01.996097+0000","last_active":"2024-07-26T06:47:23.090564+0000","last_peered":"2024-07-26T06:47:23.090564+0000","last_clean":"2024-07-26T06:47:23.090564+0000","last_became_active":"2024-07-26T06:47:01.995594+0000","last_became_peered":"2024-07-26T06:47:01.995594+0000","last_unstale":"2024-07-26T06:47:23.090564+0000","last_undegraded":"2024-07-26T06:47:23.090564+0000","last_fullsized":"2024-07-26T06:47:23.090564+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:34.969493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"80'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328345+0000","last_change":"2024-07-26T06:47:05.024171+0000","last_active":"2024-07-26T06:48:51.328345+0000","last_peered":"2024-07-26T06:48:51.328345+0000","last_clean":"2024-07-26T06:48:51.328345+0000","last_became_active":"2024-07-26T06:47:05.022888+0000","last_became_peered":"2024-07-26T06:47:05.022888+0000","last_unstale":"2024-07-26T06:48:51.328345+0000","last_undegraded":"2024-07-26T06:48:51.328345+0000","last_fullsized":"2024-07-26T06:48:51.328345+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:31:52.691068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096894+0000","last_change":"2024-07-26T06:47:02.004578+0000","last_active":"2024-07-26T06:47:23.096894+0000","last_peered":"2024-07-26T06:47:23.096894+0000","last_clean":"2024-07-26T06:47:23.096894+0000","last_became_active":"2024-07-26T06:47:02.004231+0000","last_became_peered":"2024-07-26T06:47:02.004231+0000","last_unstale":"2024-07-26T06:47:23.096894+0000","last_undegraded":"2024-07-26T06:47:23.096894+0000","last_fullsized":"2024-07-26T06:47:23.096894+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:43:39.803817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"80'8","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.327728+0000","last_change":"2024-07-26T06:47:05.024232+0000","last_active":"2024-07-26T06:48:51.327728+0000","last_peered":"2024-07-26T06:48:51.327728+0000","last_clean":"2024-07-26T06:48:51.327728+0000","last_became_active":"2024-07-26T06:47:05.023869+0000","last_became_peered":"2024-07-26T06:47:05.023869+0000","last_unstale":"2024-07-26T06:48:51.327728+0000","last_undegraded":"2024-07-26T06:48:51.327728+0000","last_fullsized":"2024-07-26T06:48:51.327728+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:27:01.357311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.105594+0000","last_change":"2024-07-26T06:47:23.105051+0000","last_active":"2024-07-26T06:47:23.105594+0000","last_peered":"2024-07-26T06:47:23.105594+0000","last_clean":"2024-07-26T06:47:23.105594+0000","last_became_active":"2024-07-26T06:47:23.098116+0000","last_became_peered":"2024-07-26T06:47:23.098116+0000","last_unstale":"2024-07-26T06:47:23.105594+0000","last_undegraded":"2024-07-26T06:47:23.105594+0000","last_fullsized":"2024-07-26T06:47:23.105594+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:15:32.991870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"80'5","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338310+0000","last_change":"2024-07-26T06:47:05.026056+0000","last_active":"2024-07-26T06:48:51.338310+0000","last_peered":"2024-07-26T06:48:51.338310+0000","last_clean":"2024-07-26T06:48:51.338310+0000","last_became_active":"2024-07-26T06:47:05.025654+0000","last_became_peered":"2024-07-26T06:47:05.025654+0000","last_unstale":"2024-07-26T06:48:51.338310+0000","last_undegraded":"2024-07-26T06:48:51.338310+0000","last_fullsized":"2024-07-26T06:48:51.338310+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:36:20.886768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"80'15","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338605+0000","last_change":"2024-07-26T06:47:05.026693+0000","last_active":"2024-07-26T06:48:51.338605+0000","last_peered":"2024-07-26T06:48:51.338605+0000","last_clean":"2024-07-26T06:48:51.338605+0000","last_became_active":"2024-07-26T06:47:05.026320+0000","last_became_peered":"2024-07-26T06:47:05.026320+0000","last_unstale":"2024-07-26T06:48:51.338605+0000","last_undegraded":"2024-07-26T06:48:51.338605+0000","last_fullsized":"2024-07-26T06:48:51.338605+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:52:30.776335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"80'5","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.159101+0000","last_change":"2024-07-26T06:47:23.107707+0000","last_active":"2024-07-26T06:47:23.159101+0000","last_peered":"2024-07-26T06:47:23.159101+0000","last_clean":"2024-07-26T06:47:23.159101+0000","last_became_active":"2024-07-26T06:47:23.099429+0000","last_became_peered":"2024-07-26T06:47:23.099429+0000","last_unstale":"2024-07-26T06:47:23.159101+0000","last_undegraded":"2024-07-26T06:47:23.159101+0000","last_fullsized":"2024-07-26T06:47:23.159101+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:40:52.856756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"80'9","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.334591+0000","last_change":"2024-07-26T06:47:05.032319+0000","last_active":"2024-07-26T06:48:51.334591+0000","last_peered":"2024-07-26T06:48:51.334591+0000","last_clean":"2024-07-26T06:48:51.334591+0000","last_became_active":"2024-07-26T06:47:05.030305+0000","last_became_peered":"2024-07-26T06:47:05.030305+0000","last_unstale":"2024-07-26T06:48:51.334591+0000","last_undegraded":"2024-07-26T06:48:51.334591+0000","last_fullsized":"2024-07-26T06:48:51.334591+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:34:05.351917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"80'12","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.119914+0000","last_change":"2024-07-26T06:47:02.007812+0000","last_active":"2024-07-26T06:47:24.119914+0000","last_peered":"2024-07-26T06:47:24.119914+0000","last_clean":"2024-07-26T06:47:24.119914+0000","last_became_active":"2024-07-26T06:47:02.006965+0000","last_became_peered":"2024-07-26T06:47:02.006965+0000","last_unstale":"2024-07-26T06:47:24.119914+0000","last_undegraded":"2024-07-26T06:47:24.119914+0000","last_fullsized":"2024-07-26T06:47:24.119914+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:53.173273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"80'6","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335704+0000","last_change":"2024-07-26T06:47:05.027900+0000","last_active":"2024-07-26T06:48:51.335704+0000","last_peered":"2024-07-26T06:48:51.335704+0000","last_clean":"2024-07-26T06:48:51.335704+0000","last_became_active":"2024-07-26T06:47:05.024467+0000","last_became_peered":"2024-07-26T06:47:05.024467+0000","last_unstale":"2024-07-26T06:48:51.335704+0000","last_undegraded":"2024-07-26T06:48:51.335704+0000","last_fullsized":"2024-07-26T06:48:51.335704+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:05:00.793487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090879+0000","last_change":"2024-07-26T06:47:02.022863+0000","last_active":"2024-07-26T06:47:23.090879+0000","last_peered":"2024-07-26T06:47:23.090879+0000","last_clean":"2024-07-26T06:47:23.090879+0000","last_became_active":"2024-07-26T06:47:02.022446+0000","last_became_peered":"2024-07-26T06:47:02.022446+0000","last_unstale":"2024-07-26T06:47:23.090879+0000","last_undegraded":"2024-07-26T06:47:23.090879+0000","last_fullsized":"2024-07-26T06:47:23.090879+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:14:48.649099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"80'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333567+0000","last_change":"2024-07-26T06:47:05.023669+0000","last_active":"2024-07-26T06:48:51.333567+0000","last_peered":"2024-07-26T06:48:51.333567+0000","last_clean":"2024-07-26T06:48:51.333567+0000","last_became_active":"2024-07-26T06:47:05.023265+0000","last_became_peered":"2024-07-26T06:47:05.023265+0000","last_unstale":"2024-07-26T06:48:51.333567+0000","last_undegraded":"2024-07-26T06:48:51.333567+0000","last_fullsized":"2024-07-26T06:48:51.333567+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:56:45.278253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":141,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:02.308393+0000","last_change":"2024-07-26T06:47:02.006668+0000","last_active":"2024-07-26T06:49:02.308393+0000","last_peered":"2024-07-26T06:49:02.308393+0000","last_clean":"2024-07-26T06:49:02.308393+0000","last_became_active":"2024-07-26T06:47:02.006287+0000","last_became_peered":"2024-07-26T06:47:02.006287+0000","last_unstale":"2024-07-26T06:49:02.308393+0000","last_undegraded":"2024-07-26T06:49:02.308393+0000","last_fullsized":"2024-07-26T06:49:02.308393+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:28:05.722539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":101,"num_read_kb":101,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69561,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1052,"num_read_kb":602,"num_write":253,"num_write_kb":143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1634,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":208683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":217,"ondisk_log_size":217,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250626,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710760,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":426222,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906989,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":1079164,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070513,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":425458,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266742,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":880640,"data_stored":425142,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462971,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":946176,"data_stored":1019027,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626497,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36196,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708956,"statfs":{"total":95995035648,"available":95957970944,"internally_reserved":0,"allocated":880640,"data_stored":425175,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822726,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":1018173,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986250,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":487643,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":897024,"data_stored":425000,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36208,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708944,"statfs":{"total":95995035648,"available":95957958656,"internally_reserved":0,"allocated":892928,"data_stored":426840,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36256,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708896,"statfs":{"total":95995035648,"available":95957909504,"internally_reserved":0,"allocated":942080,"data_stored":488163,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":426514,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":63307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":64343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:49:09.771 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-07-26T06:49:09.771 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-07-26T06:49:09.771 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-07-26T06:49:09.771 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph health --format=json 2024-07-26T06:49:10.040 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:10.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:09 smithi042 ceph-mon[32775]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:49:10.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:09 smithi164 ceph-mon[36970]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:49:10.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:09 smithi163 ceph-mon[36873]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:49:11.192 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:10 smithi042 ceph-mon[32775]: pgmap v364: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:49:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:10 smithi164 ceph-mon[36970]: pgmap v364: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:49:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:10 smithi163 ceph-mon[36873]: pgmap v364: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:49:11.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.754+0000 7fe4cb915640 1 -- 172.21.15.42:0/653027062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4c4108740 msgr2=0x7fe4c410ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.754+0000 7fe4cb915640 1 --2- 172.21.15.42:0/653027062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4c4108740 0x7fe4c410ef70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc009ca0 tx=0x7fe4bc02a8c0 comp rx=0 tx=0).stop 2024-07-26T06:49:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.754+0000 7fe4cb915640 1 -- 172.21.15.42:0/653027062 shutdown_connections 2024-07-26T06:49:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.754+0000 7fe4cb915640 1 --2- 172.21.15.42:0/653027062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4c4108740 0x7fe4c410ef70 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.754+0000 7fe4cb915640 1 --2- 172.21.15.42:0/653027062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4c4107df0 0x7fe4c4108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.754+0000 7fe4cb915640 1 --2- 172.21.15.42:0/653027062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4c41024b0 0x7fe4c41028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.754+0000 7fe4cb915640 1 -- 172.21.15.42:0/653027062 >> 172.21.15.42:0/653027062 conn(0x7fe4c4079170 msgr2=0x7fe4c4079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.754+0000 7fe4cb915640 1 -- 172.21.15.42:0/653027062 shutdown_connections 2024-07-26T06:49:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.754+0000 7fe4cb915640 1 -- 172.21.15.42:0/653027062 wait complete. 2024-07-26T06:49:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.755+0000 7fe4cb915640 1 Processor -- start 2024-07-26T06:49:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.755+0000 7fe4cb915640 1 -- start start 2024-07-26T06:49:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.755+0000 7fe4cb915640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4c41024b0 0x7fe4c4198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.755+0000 7fe4cb915640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4c4107df0 0x7fe4c4199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.755+0000 7fe4cb915640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4c4108740 0x7fe4c4199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.755+0000 7fe4cb915640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c4112490 con 0x7fe4c41024b0 2024-07-26T06:49:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.755+0000 7fe4cb915640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c4112310 con 0x7fe4c4107df0 2024-07-26T06:49:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.755+0000 7fe4cb915640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c4112610 con 0x7fe4c4108740 2024-07-26T06:49:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.755+0000 7fe4c96b1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4c41024b0 0x7fe4c4198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.755+0000 7fe4c96b1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4c41024b0 0x7fe4c4198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60708/0 (socket says 172.21.15.42:60708) 2024-07-26T06:49:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.755+0000 7fe4c96b1640 1 -- 172.21.15.42:0/1732072974 learned_addr learned my addr 172.21.15.42:0/1732072974 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.756+0000 7fe4c8eb0640 1 --2- 172.21.15.42:0/1732072974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4c4107df0 0x7fe4c4199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.756+0000 7fe4c9eb2640 1 --2- 172.21.15.42:0/1732072974 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4c4108740 0x7fe4c4199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.756+0000 7fe4c96b1640 1 -- 172.21.15.42:0/1732072974 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4c4108740 msgr2=0x7fe4c4199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.756+0000 7fe4c96b1640 1 --2- 172.21.15.42:0/1732072974 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4c4108740 0x7fe4c4199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.756+0000 7fe4c96b1640 1 -- 172.21.15.42:0/1732072974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4c4107df0 msgr2=0x7fe4c4199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.756+0000 7fe4c96b1640 1 --2- 172.21.15.42:0/1732072974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4c4107df0 0x7fe4c4199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.756+0000 7fe4c96b1640 1 -- 172.21.15.42:0/1732072974 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c41a8280 con 0x7fe4c41024b0 2024-07-26T06:49:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.756+0000 7fe4c8eb0640 1 --2- 172.21.15.42:0/1732072974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4c4107df0 0x7fe4c4199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:49:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.756+0000 7fe4c96b1640 1 --2- 172.21.15.42:0/1732072974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4c41024b0 0x7fe4c4198bd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000b4a0 tx=0x7fe4c000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.756+0000 7fe4ba7fc640 1 -- 172.21.15.42:0/1732072974 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0014070 con 0x7fe4c41024b0 2024-07-26T06:49:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.756+0000 7fe4cb915640 1 -- 172.21.15.42:0/1732072974 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c41a8570 con 0x7fe4c41024b0 2024-07-26T06:49:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.756+0000 7fe4ba7fc640 1 -- 172.21.15.42:0/1732072974 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4c0010ac0 con 0x7fe4c41024b0 2024-07-26T06:49:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.756+0000 7fe4ba7fc640 1 -- 172.21.15.42:0/1732072974 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0017c30 con 0x7fe4c41024b0 2024-07-26T06:49:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.756+0000 7fe4cb915640 1 -- 172.21.15.42:0/1732072974 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c41a8b30 con 0x7fe4c41024b0 2024-07-26T06:49:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.757+0000 7fe4cb915640 1 -- 172.21.15.42:0/1732072974 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c4103730 con 0x7fe4c41024b0 2024-07-26T06:49:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.759+0000 7fe4ba7fc640 1 -- 172.21.15.42:0/1732072974 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fe4c0017070 con 0x7fe4c41024b0 2024-07-26T06:49:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.760+0000 7fe4ba7fc640 1 --2- 172.21.15.42:0/1732072974 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe4b00b3bd0 0x7fe4b00b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.760+0000 7fe4c8eb0640 1 --2- 172.21.15.42:0/1732072974 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe4b00b3bd0 0x7fe4b00b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.760+0000 7fe4ba7fc640 1 -- 172.21.15.42:0/1732072974 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fe4c00e34d0 con 0x7fe4c41024b0 2024-07-26T06:49:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.761+0000 7fe4c8eb0640 1 --2- 172.21.15.42:0/1732072974 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe4b00b3bd0 0x7fe4b00b60e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4107450 tx=0x7fe4b4008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.764+0000 7fe4ba7fc640 1 -- 172.21.15.42:0/1732072974 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe4c00aaae0 con 0x7fe4c41024b0 2024-07-26T06:49:11.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.987+0000 7fe4cb915640 1 -- 172.21.15.42:0/1732072974 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe4c410a560 con 0x7fe4c41024b0 2024-07-26T06:49:11.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.987+0000 7fe4ba7fc640 1 -- 172.21.15.42:0/1732072974 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fe4c00aa3e0 con 0x7fe4c41024b0 2024-07-26T06:49:11.988 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:49:11.988 INFO:teuthology.orchestra.run.smithi042.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-07-26T06:49:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.989+0000 7fe4cb915640 1 -- 172.21.15.42:0/1732072974 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe4b00b3bd0 msgr2=0x7fe4b00b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.989+0000 7fe4cb915640 1 --2- 172.21.15.42:0/1732072974 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe4b00b3bd0 0x7fe4b00b60e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe4c4107450 tx=0x7fe4b4008040 comp rx=0 tx=0).stop 2024-07-26T06:49:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.989+0000 7fe4cb915640 1 -- 172.21.15.42:0/1732072974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4c41024b0 msgr2=0x7fe4c4198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.989+0000 7fe4cb915640 1 --2- 172.21.15.42:0/1732072974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4c41024b0 0x7fe4c4198bd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000b4a0 tx=0x7fe4c000b960 comp rx=0 tx=0).stop 2024-07-26T06:49:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.989+0000 7fe4cb915640 1 -- 172.21.15.42:0/1732072974 shutdown_connections 2024-07-26T06:49:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.989+0000 7fe4cb915640 1 --2- 172.21.15.42:0/1732072974 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe4b00b3bd0 0x7fe4b00b60e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.989+0000 7fe4cb915640 1 --2- 172.21.15.42:0/1732072974 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4c4108740 0x7fe4c4199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.989+0000 7fe4cb915640 1 --2- 172.21.15.42:0/1732072974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4c4107df0 0x7fe4c4199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.989+0000 7fe4cb915640 1 --2- 172.21.15.42:0/1732072974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4c41024b0 0x7fe4c4198bd0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:11.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.989+0000 7fe4cb915640 1 -- 172.21.15.42:0/1732072974 >> 172.21.15.42:0/1732072974 conn(0x7fe4c4079170 msgr2=0x7fe4c4104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:11.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.989+0000 7fe4cb915640 1 -- 172.21.15.42:0/1732072974 shutdown_connections 2024-07-26T06:49:11.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:11.989+0000 7fe4cb915640 1 -- 172.21.15.42:0/1732072974 wait complete. 2024-07-26T06:49:12.684 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-07-26T06:49:12.684 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:49:12.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:12 smithi042 ceph-mon[32775]: pgmap v365: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:49:12.940 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:12 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1732072974' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T06:49:12.960 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:12 smithi164 ceph-mon[36970]: pgmap v365: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:49:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:12 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1732072974' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T06:49:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:12 smithi163 ceph-mon[36873]: pgmap v365: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:49:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:12 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1732072974' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T06:49:14.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.637+0000 7f07a8119640 1 -- 172.21.15.42:0/3295298460 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a0108740 msgr2=0x7f07a010ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:14.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.637+0000 7f07a8119640 1 --2- 172.21.15.42:0/3295298460 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a0108740 0x7f07a010ef70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0798009ca0 tx=0x7f079802a8c0 comp rx=0 tx=0).stop 2024-07-26T06:49:14.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.637+0000 7f07a8119640 1 -- 172.21.15.42:0/3295298460 shutdown_connections 2024-07-26T06:49:14.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.637+0000 7f07a8119640 1 --2- 172.21.15.42:0/3295298460 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a0108740 0x7f07a010ef70 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:14.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.637+0000 7f07a8119640 1 --2- 172.21.15.42:0/3295298460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07a0107df0 0x7f07a0108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:14.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.637+0000 7f07a8119640 1 --2- 172.21.15.42:0/3295298460 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07a01024b0 0x7f07a01028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:14.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.637+0000 7f07a8119640 1 -- 172.21.15.42:0/3295298460 >> 172.21.15.42:0/3295298460 conn(0x7f07a0079170 msgr2=0x7f07a0079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:14.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.638+0000 7f07a8119640 1 -- 172.21.15.42:0/3295298460 shutdown_connections 2024-07-26T06:49:14.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.638+0000 7f07a8119640 1 -- 172.21.15.42:0/3295298460 wait complete. 2024-07-26T06:49:14.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.638+0000 7f07a8119640 1 Processor -- start 2024-07-26T06:49:14.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.638+0000 7f07a8119640 1 -- start start 2024-07-26T06:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.638+0000 7f07a8119640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07a01024b0 0x7f07a0198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.638+0000 7f07a8119640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a0107df0 0x7f07a0199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f07a8119640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07a0108740 0x7f07a019e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f07a8119640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f07a01123c0 con 0x7f07a0107df0 2024-07-26T06:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f07a8119640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f07a0112240 con 0x7f07a01024b0 2024-07-26T06:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f07a8119640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f07a0112540 con 0x7f07a0108740 2024-07-26T06:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f07a56b4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a0107df0 0x7f07a0199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f07a56b4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a0107df0 0x7f07a0199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60716/0 (socket says 172.21.15.42:60716) 2024-07-26T06:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f07a56b4640 1 -- 172.21.15.42:0/661123367 learned_addr learned my addr 172.21.15.42:0/661123367 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f07a66b6640 1 --2- 172.21.15.42:0/661123367 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07a0108740 0x7f07a019e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f07a5eb5640 1 --2- 172.21.15.42:0/661123367 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07a01024b0 0x7f07a0198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:14.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f07a56b4640 1 -- 172.21.15.42:0/661123367 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07a0108740 msgr2=0x7f07a019e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:14.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f07a56b4640 1 --2- 172.21.15.42:0/661123367 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07a0108740 0x7f07a019e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:14.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f07a56b4640 1 -- 172.21.15.42:0/661123367 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07a01024b0 msgr2=0x7f07a0198d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:14.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f07a56b4640 1 --2- 172.21.15.42:0/661123367 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07a01024b0 0x7f07a0198d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:14.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f07a56b4640 1 -- 172.21.15.42:0/661123367 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a019ece0 con 0x7f07a0107df0 2024-07-26T06:49:14.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f07a5eb5640 1 --2- 172.21.15.42:0/661123367 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07a01024b0 0x7f07a0198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:49:14.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f07a56b4640 1 --2- 172.21.15.42:0/661123367 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a0107df0 0x7f07a0199310 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f079000d740 tx=0x7f079000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:14.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f0796ffd640 1 -- 172.21.15.42:0/661123367 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0790017070 con 0x7f07a0107df0 2024-07-26T06:49:14.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.639+0000 7f07a8119640 1 -- 172.21.15.42:0/661123367 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07a019efd0 con 0x7f07a0107df0 2024-07-26T06:49:14.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.640+0000 7f0796ffd640 1 -- 172.21.15.42:0/661123367 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f079000de50 con 0x7f07a0107df0 2024-07-26T06:49:14.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.640+0000 7f0796ffd640 1 -- 172.21.15.42:0/661123367 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0790004ce0 con 0x7f07a0107df0 2024-07-26T06:49:14.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.640+0000 7f07a8119640 1 -- 172.21.15.42:0/661123367 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07a01a5730 con 0x7f07a0107df0 2024-07-26T06:49:14.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.640+0000 7f07a8119640 1 -- 172.21.15.42:0/661123367 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07a0103730 con 0x7f07a0107df0 2024-07-26T06:49:14.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.643+0000 7f0796ffd640 1 -- 172.21.15.42:0/661123367 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f0790004e80 con 0x7f07a0107df0 2024-07-26T06:49:14.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.644+0000 7f0796ffd640 1 --2- 172.21.15.42:0/661123367 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f078c0b3bd0 0x7f078c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:14.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.644+0000 7f07a5eb5640 1 --2- 172.21.15.42:0/661123367 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f078c0b3bd0 0x7f078c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:14.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.644+0000 7f0796ffd640 1 -- 172.21.15.42:0/661123367 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f07900e39b0 con 0x7f07a0107df0 2024-07-26T06:49:14.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.644+0000 7f07a5eb5640 1 --2- 172.21.15.42:0/661123367 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f078c0b3bd0 0x7f078c0b60e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f079c0058c0 tx=0x7f079c005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:14.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.647+0000 7f0796ffd640 1 -- 172.21.15.42:0/661123367 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f07900aafe0 con 0x7f07a0107df0 2024-07-26T06:49:14.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.867+0000 7f07a8119640 1 -- 172.21.15.42:0/661123367 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07a0199ea0 con 0x7f07a0107df0 2024-07-26T06:49:14.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.867+0000 7f0796ffd640 1 -- 172.21.15.42:0/661123367 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11978 (secure 0 0 0) 0x7f07900aa8e0 con 0x7f07a0107df0 2024-07-26T06:49:14.868 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:49:14.869 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":28,"btime":"2024-07-26T06:47:26:107129+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:47:25.125103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24578,24583,24593,24601,24620]},"id":1}]} 2024-07-26T06:49:14.869 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 28 2024-07-26T06:49:14.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.870+0000 7f07a8119640 1 -- 172.21.15.42:0/661123367 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f078c0b3bd0 msgr2=0x7f078c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:14.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.870+0000 7f07a8119640 1 --2- 172.21.15.42:0/661123367 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f078c0b3bd0 0x7f078c0b60e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f079c0058c0 tx=0x7f079c005850 comp rx=0 tx=0).stop 2024-07-26T06:49:14.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.870+0000 7f07a8119640 1 -- 172.21.15.42:0/661123367 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a0107df0 msgr2=0x7f07a0199310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:14.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.870+0000 7f07a8119640 1 --2- 172.21.15.42:0/661123367 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a0107df0 0x7f07a0199310 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f079000d740 tx=0x7f079000dc00 comp rx=0 tx=0).stop 2024-07-26T06:49:14.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.870+0000 7f07a8119640 1 -- 172.21.15.42:0/661123367 shutdown_connections 2024-07-26T06:49:14.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.870+0000 7f07a8119640 1 --2- 172.21.15.42:0/661123367 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f078c0b3bd0 0x7f078c0b60e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:14.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.870+0000 7f07a8119640 1 --2- 172.21.15.42:0/661123367 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07a0108740 0x7f07a019e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:14.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.870+0000 7f07a8119640 1 --2- 172.21.15.42:0/661123367 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a0107df0 0x7f07a0199310 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:14.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.870+0000 7f07a8119640 1 --2- 172.21.15.42:0/661123367 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f07a01024b0 0x7f07a0198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:14.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.870+0000 7f07a8119640 1 -- 172.21.15.42:0/661123367 >> 172.21.15.42:0/661123367 conn(0x7f07a0079170 msgr2=0x7f07a0104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:14.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.870+0000 7f07a8119640 1 -- 172.21.15.42:0/661123367 shutdown_connections 2024-07-26T06:49:14.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:14.870+0000 7f07a8119640 1 -- 172.21.15.42:0/661123367 wait complete. 2024-07-26T06:49:15.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:14 smithi042 ceph-mon[32775]: pgmap v366: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:15.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:14 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/661123367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:49:15.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:14 smithi164 ceph-mon[36970]: pgmap v366: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:15.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:14 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/661123367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:49:15.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:14 smithi163 ceph-mon[36873]: pgmap v366: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:14 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/661123367' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:49:15.656 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:49:15.929 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:17.005 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:16 smithi042 ceph-mon[32775]: pgmap v367: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:49:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:16 smithi164 ceph-mon[36970]: pgmap v367: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:49:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:16 smithi163 ceph-mon[36873]: pgmap v367: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:49:17.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.560+0000 7fd8cf8b7640 1 -- 172.21.15.42:0/3153941549 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8c8108720 msgr2=0x7fd8c810ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:17.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.560+0000 7fd8cf8b7640 1 --2- 172.21.15.42:0/3153941549 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8c8108720 0x7fd8c810ef50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0009ca0 tx=0x7fd8c002a8c0 comp rx=0 tx=0).stop 2024-07-26T06:49:17.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.560+0000 7fd8cf8b7640 1 -- 172.21.15.42:0/3153941549 shutdown_connections 2024-07-26T06:49:17.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.560+0000 7fd8cf8b7640 1 --2- 172.21.15.42:0/3153941549 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8c8108720 0x7fd8c810ef50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:17.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.560+0000 7fd8cf8b7640 1 --2- 172.21.15.42:0/3153941549 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8c8107dd0 0x7fd8c81081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:17.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.560+0000 7fd8cf8b7640 1 --2- 172.21.15.42:0/3153941549 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8c8102490 0x7fd8c81028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:17.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.560+0000 7fd8cf8b7640 1 -- 172.21.15.42:0/3153941549 >> 172.21.15.42:0/3153941549 conn(0x7fd8c8079160 msgr2=0x7fd8c8079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:17.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.560+0000 7fd8cf8b7640 1 -- 172.21.15.42:0/3153941549 shutdown_connections 2024-07-26T06:49:17.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.560+0000 7fd8cf8b7640 1 -- 172.21.15.42:0/3153941549 wait complete. 2024-07-26T06:49:17.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.561+0000 7fd8cf8b7640 1 Processor -- start 2024-07-26T06:49:17.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.561+0000 7fd8cf8b7640 1 -- start start 2024-07-26T06:49:17.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.561+0000 7fd8cf8b7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8c8102490 0x7fd8c8199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:17.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.561+0000 7fd8cf8b7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8c8107dd0 0x7fd8c819a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:17.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.561+0000 7fd8cf8b7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8c8108720 0x7fd8c819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:17.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.561+0000 7fd8cf8b7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c8112400 con 0x7fd8c8102490 2024-07-26T06:49:17.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.561+0000 7fd8cf8b7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c8112280 con 0x7fd8c8107dd0 2024-07-26T06:49:17.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.561+0000 7fd8cf8b7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c8112580 con 0x7fd8c8108720 2024-07-26T06:49:17.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.561+0000 7fd8cd653640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8c8102490 0x7fd8c8199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:17.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.561+0000 7fd8cd653640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8c8102490 0x7fd8c8199ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42058/0 (socket says 172.21.15.42:42058) 2024-07-26T06:49:17.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.561+0000 7fd8cd653640 1 -- 172.21.15.42:0/2730750089 learned_addr learned my addr 172.21.15.42:0/2730750089 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:17.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.561+0000 7fd8cd653640 1 -- 172.21.15.42:0/2730750089 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8c8108720 msgr2=0x7fd8c819f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:17.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.561+0000 7fd8cd653640 1 --2- 172.21.15.42:0/2730750089 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8c8108720 0x7fd8c819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:17.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.561+0000 7fd8cd653640 1 -- 172.21.15.42:0/2730750089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8c8107dd0 msgr2=0x7fd8c819a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:17.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.561+0000 7fd8cd653640 1 --2- 172.21.15.42:0/2730750089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8c8107dd0 0x7fd8c819a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:17.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.561+0000 7fd8cd653640 1 -- 172.21.15.42:0/2730750089 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c819fe50 con 0x7fd8c8102490 2024-07-26T06:49:17.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.562+0000 7fd8cd653640 1 --2- 172.21.15.42:0/2730750089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8c8102490 0x7fd8c8199ee0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd8c400b4a0 tx=0x7fd8c400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:17.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.562+0000 7fd8be7fc640 1 -- 172.21.15.42:0/2730750089 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4014070 con 0x7fd8c8102490 2024-07-26T06:49:17.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.562+0000 7fd8be7fc640 1 -- 172.21.15.42:0/2730750089 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8c4010ac0 con 0x7fd8c8102490 2024-07-26T06:49:17.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.562+0000 7fd8be7fc640 1 -- 172.21.15.42:0/2730750089 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c4017c30 con 0x7fd8c8102490 2024-07-26T06:49:17.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.562+0000 7fd8cf8b7640 1 -- 172.21.15.42:0/2730750089 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c81a0140 con 0x7fd8c8102490 2024-07-26T06:49:17.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.562+0000 7fd8cf8b7640 1 -- 172.21.15.42:0/2730750089 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c81a6920 con 0x7fd8c8102490 2024-07-26T06:49:17.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.563+0000 7fd8cf8b7640 1 -- 172.21.15.42:0/2730750089 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c8103710 con 0x7fd8c8102490 2024-07-26T06:49:17.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.564+0000 7fd8be7fc640 1 -- 172.21.15.42:0/2730750089 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fd8c4017070 con 0x7fd8c8102490 2024-07-26T06:49:17.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.565+0000 7fd8be7fc640 1 --2- 172.21.15.42:0/2730750089 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd8b40b3c50 0x7fd8b40b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:17.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.565+0000 7fd8cce52640 1 --2- 172.21.15.42:0/2730750089 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd8b40b3c50 0x7fd8b40b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:17.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.565+0000 7fd8be7fc640 1 -- 172.21.15.42:0/2730750089 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fd8c40e3380 con 0x7fd8c8102490 2024-07-26T06:49:17.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.566+0000 7fd8cce52640 1 --2- 172.21.15.42:0/2730750089 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd8b40b3c50 0x7fd8b40b6160 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd8c81073c0 tx=0x7fd8b8005bf0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:17.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.568+0000 7fd8be7fc640 1 -- 172.21.15.42:0/2730750089 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd8c40aa950 con 0x7fd8c8102490 2024-07-26T06:49:17.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.788+0000 7fd8cf8b7640 1 -- 172.21.15.42:0/2730750089 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8c81028a0 con 0x7fd8c8102490 2024-07-26T06:49:17.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.789+0000 7fd8be7fc640 1 -- 172.21.15.42:0/2730750089 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11978 (secure 0 0 0) 0x7fd8c40aa250 con 0x7fd8c8102490 2024-07-26T06:49:17.789 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:49:17.789 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":28,"btime":"2024-07-26T06:47:26:107129+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:47:25.125103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24578,24583,24593,24601,24620]},"id":1}]} 2024-07-26T06:49:17.790 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 28 2024-07-26T06:49:17.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.791+0000 7fd8cf8b7640 1 -- 172.21.15.42:0/2730750089 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd8b40b3c50 msgr2=0x7fd8b40b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:17.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.791+0000 7fd8cf8b7640 1 --2- 172.21.15.42:0/2730750089 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd8b40b3c50 0x7fd8b40b6160 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd8c81073c0 tx=0x7fd8b8005bf0 comp rx=0 tx=0).stop 2024-07-26T06:49:17.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.791+0000 7fd8cf8b7640 1 -- 172.21.15.42:0/2730750089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8c8102490 msgr2=0x7fd8c8199ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:17.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.791+0000 7fd8cf8b7640 1 --2- 172.21.15.42:0/2730750089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8c8102490 0x7fd8c8199ee0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd8c400b4a0 tx=0x7fd8c400b960 comp rx=0 tx=0).stop 2024-07-26T06:49:17.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.791+0000 7fd8cf8b7640 1 -- 172.21.15.42:0/2730750089 shutdown_connections 2024-07-26T06:49:17.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.791+0000 7fd8cf8b7640 1 --2- 172.21.15.42:0/2730750089 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd8b40b3c50 0x7fd8b40b6160 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:17.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.791+0000 7fd8cf8b7640 1 --2- 172.21.15.42:0/2730750089 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd8c8108720 0x7fd8c819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:17.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.791+0000 7fd8cf8b7640 1 --2- 172.21.15.42:0/2730750089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8c8107dd0 0x7fd8c819a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:17.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.791+0000 7fd8cf8b7640 1 --2- 172.21.15.42:0/2730750089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8c8102490 0x7fd8c8199ee0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:17.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.791+0000 7fd8cf8b7640 1 -- 172.21.15.42:0/2730750089 >> 172.21.15.42:0/2730750089 conn(0x7fd8c8079160 msgr2=0x7fd8c806b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:17.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.791+0000 7fd8cf8b7640 1 -- 172.21.15.42:0/2730750089 shutdown_connections 2024-07-26T06:49:17.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:17.791+0000 7fd8cf8b7640 1 -- 172.21.15.42:0/2730750089 wait complete. 2024-07-26T06:49:18.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:17 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2730750089' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:49:18.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:17 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2730750089' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:49:18.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:17 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2730750089' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:49:18.494 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T06:49:18.763 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:19.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:18 smithi042 ceph-mon[32775]: pgmap v368: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:18 smithi164 ceph-mon[36970]: pgmap v368: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:19.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:18 smithi163 ceph-mon[36873]: pgmap v368: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:20.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.471+0000 7f03791c7640 1 -- 172.21.15.42:0/3822206969 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0374109840 msgr2=0x7f0374110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:20.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.471+0000 7f03791c7640 1 --2- 172.21.15.42:0/3822206969 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0374109840 0x7f0374110070 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f036c009ca0 tx=0x7f036c02a8c0 comp rx=0 tx=0).stop 2024-07-26T06:49:20.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.471+0000 7f03791c7640 1 -- 172.21.15.42:0/3822206969 shutdown_connections 2024-07-26T06:49:20.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.471+0000 7f03791c7640 1 --2- 172.21.15.42:0/3822206969 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0374109840 0x7f0374110070 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:20.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.471+0000 7f03791c7640 1 --2- 172.21.15.42:0/3822206969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0374108ef0 0x7f0374109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:20.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.471+0000 7f03791c7640 1 --2- 172.21.15.42:0/3822206969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0374101a10 0x7f0374101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:20.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.471+0000 7f03791c7640 1 -- 172.21.15.42:0/3822206969 >> 172.21.15.42:0/3822206969 conn(0x7f03740775f0 msgr2=0x7f0374077a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:20.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.471+0000 7f03791c7640 1 -- 172.21.15.42:0/3822206969 shutdown_connections 2024-07-26T06:49:20.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.471+0000 7f03791c7640 1 -- 172.21.15.42:0/3822206969 wait complete. 2024-07-26T06:49:20.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.472+0000 7f03791c7640 1 Processor -- start 2024-07-26T06:49:20.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.472+0000 7f03791c7640 1 -- start start 2024-07-26T06:49:20.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f03791c7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0374101a10 0x7f037419c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:20.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f03791c7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0374108ef0 0x7f037419c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:20.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f03791c7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0374109840 0x7f03741a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:20.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f03791c7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0374113490 con 0x7f0374109840 2024-07-26T06:49:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f03791c7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0374113310 con 0x7f0374101a10 2024-07-26T06:49:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f03791c7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0374113610 con 0x7f0374108ef0 2024-07-26T06:49:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f037359e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0374109840 0x7f03741a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f037359e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0374109840 0x7f03741a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42072/0 (socket says 172.21.15.42:42072) 2024-07-26T06:49:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f037359e640 1 -- 172.21.15.42:0/2138223348 learned_addr learned my addr 172.21.15.42:0/2138223348 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f0372d9d640 1 --2- 172.21.15.42:0/2138223348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0374101a10 0x7f037419c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f037259c640 1 --2- 172.21.15.42:0/2138223348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0374108ef0 0x7f037419c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f037359e640 1 -- 172.21.15.42:0/2138223348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0374108ef0 msgr2=0x7f037419c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f037359e640 1 --2- 172.21.15.42:0/2138223348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0374108ef0 0x7f037419c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f037359e640 1 -- 172.21.15.42:0/2138223348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0374101a10 msgr2=0x7f037419c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f037359e640 1 --2- 172.21.15.42:0/2138223348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0374101a10 0x7f037419c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f037359e640 1 -- 172.21.15.42:0/2138223348 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03741a1f70 con 0x7f0374109840 2024-07-26T06:49:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f037259c640 1 --2- 172.21.15.42:0/2138223348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0374108ef0 0x7f037419c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:49:20.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.473+0000 7f037359e640 1 --2- 172.21.15.42:0/2138223348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0374109840 0x7f03741a1900 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f036c005cc0 tx=0x7f036c02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:20.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.474+0000 7f035bfff640 1 -- 172.21.15.42:0/2138223348 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c003bf0 con 0x7f0374109840 2024-07-26T06:49:20.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.474+0000 7f03791c7640 1 -- 172.21.15.42:0/2138223348 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03741a2200 con 0x7f0374109840 2024-07-26T06:49:20.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.474+0000 7f035bfff640 1 -- 172.21.15.42:0/2138223348 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f036c003d90 con 0x7f0374109840 2024-07-26T06:49:20.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.474+0000 7f035bfff640 1 -- 172.21.15.42:0/2138223348 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c02fb20 con 0x7f0374109840 2024-07-26T06:49:20.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.474+0000 7f03791c7640 1 -- 172.21.15.42:0/2138223348 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03741a2740 con 0x7f0374109840 2024-07-26T06:49:20.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.475+0000 7f03791c7640 1 -- 172.21.15.42:0/2138223348 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0374101e20 con 0x7f0374109840 2024-07-26T06:49:20.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.476+0000 7f035bfff640 1 -- 172.21.15.42:0/2138223348 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f036c004580 con 0x7f0374109840 2024-07-26T06:49:20.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.477+0000 7f035bfff640 1 --2- 172.21.15.42:0/2138223348 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f035c0b3c00 0x7f035c0b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:20.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.477+0000 7f0372d9d640 1 --2- 172.21.15.42:0/2138223348 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f035c0b3c00 0x7f035c0b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:20.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.478+0000 7f035bfff640 1 -- 172.21.15.42:0/2138223348 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f036c105380 con 0x7f0374109840 2024-07-26T06:49:20.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.478+0000 7f0372d9d640 1 --2- 172.21.15.42:0/2138223348 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f035c0b3c00 0x7f035c0b6110 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f036400abe0 tx=0x7f036400a2d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:20.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.480+0000 7f035bfff640 1 -- 172.21.15.42:0/2138223348 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f036c0cc950 con 0x7f0374109840 2024-07-26T06:49:20.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.664+0000 7f03791c7640 1 -- 172.21.15.42:0/2138223348 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f03741a8c20 con 0x7f0374109840 2024-07-26T06:49:20.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.666+0000 7f035bfff640 1 -- 172.21.15.42:0/2138223348 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22697 (secure 0 0 0) 0x7f036c0098b0 con 0x7f0374109840 2024-07-26T06:49:20.667 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:49:20.667 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":80,"fsid":"8b3e2750-4b19-11ef-bcb6-c7b262605968","created":"2024-07-26T06:38:29.306470+0000","modified":"2024-07-26T06:47:23.086609+0000","last_up_change":"2024-07-26T06:46:02.149453+0000","last_in_change":"2024-07-26T06:45:43.027764+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:42:27.302195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:47:00.042262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T06:47:03.674160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bda88e0-09ee-447b-8c04-24f4e5d55d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6803","nonce":3587066379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6805","nonce":3587066379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6809","nonce":3587066379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6807","nonce":3587066379}]},"public_addr":"172.21.15.42:6803/3587066379","cluster_addr":"172.21.15.42:6805/3587066379","heartbeat_back_addr":"172.21.15.42:6809/3587066379","heartbeat_front_addr":"172.21.15.42:6807/3587066379","state":["exists","up"]},{"osd":1,"uuid":"7d747138-8217-4836-a173-cf8ac65e9fe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6801","nonce":360219078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6803","nonce":360219078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6807","nonce":360219078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6805","nonce":360219078}]},"public_addr":"172.21.15.163:6801/360219078","cluster_addr":"172.21.15.163:6803/360219078","heartbeat_back_addr":"172.21.15.163:6807/360219078","heartbeat_front_addr":"172.21.15.163:6805/360219078","state":["exists","up"]},{"osd":2,"uuid":"4b7258c1-3d35-4930-ab10-0b04a2a2c8a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6801","nonce":336113412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6803","nonce":336113412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6807","nonce":336113412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6805","nonce":336113412}]},"public_addr":"172.21.15.164:6801/336113412","cluster_addr":"172.21.15.164:6803/336113412","heartbeat_back_addr":"172.21.15.164:6807/336113412","heartbeat_front_addr":"172.21.15.164:6805/336113412","state":["exists","up"]},{"osd":3,"uuid":"96608387-0de0-4d90-a952-d6eba61f7fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6811","nonce":2819018064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6813","nonce":2819018064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6817","nonce":2819018064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6815","nonce":2819018064}]},"public_addr":"172.21.15.42:6811/2819018064","cluster_addr":"172.21.15.42:6813/2819018064","heartbeat_back_addr":"172.21.15.42:6817/2819018064","heartbeat_front_addr":"172.21.15.42:6815/2819018064","state":["exists","up"]},{"osd":4,"uuid":"fabf6b02-fe34-4560-85b5-01bf37fc9be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6809","nonce":431934374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6811","nonce":431934374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6815","nonce":431934374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6813","nonce":431934374}]},"public_addr":"172.21.15.163:6809/431934374","cluster_addr":"172.21.15.163:6811/431934374","heartbeat_back_addr":"172.21.15.163:6815/431934374","heartbeat_front_addr":"172.21.15.163:6813/431934374","state":["exists","up"]},{"osd":5,"uuid":"e226fb4f-15b2-40df-af34-900040ba93ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6809","nonce":261549362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6811","nonce":261549362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6815","nonce":261549362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6813","nonce":261549362}]},"public_addr":"172.21.15.164:6809/261549362","cluster_addr":"172.21.15.164:6811/261549362","heartbeat_back_addr":"172.21.15.164:6815/261549362","heartbeat_front_addr":"172.21.15.164:6813/261549362","state":["exists","up"]},{"osd":6,"uuid":"beb8ec5d-7ea5-4e4e-a46c-a84f7768cec0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6819","nonce":2160811300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6821","nonce":2160811300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6825","nonce":2160811300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6823","nonce":2160811300}]},"public_addr":"172.21.15.42:6819/2160811300","cluster_addr":"172.21.15.42:6821/2160811300","heartbeat_back_addr":"172.21.15.42:6825/2160811300","heartbeat_front_addr":"172.21.15.42:6823/2160811300","state":["exists","up"]},{"osd":7,"uuid":"f699004c-f7c2-4f1c-ae4b-7178f3b06e2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6817","nonce":812214111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6819","nonce":812214111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6823","nonce":812214111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6821","nonce":812214111}]},"public_addr":"172.21.15.163:6817/812214111","cluster_addr":"172.21.15.163:6819/812214111","heartbeat_back_addr":"172.21.15.163:6823/812214111","heartbeat_front_addr":"172.21.15.163:6821/812214111","state":["exists","up"]},{"osd":8,"uuid":"3476163e-efea-4652-a8d4-b7c4a908f984","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6817","nonce":962845839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6819","nonce":962845839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6823","nonce":962845839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6821","nonce":962845839}]},"public_addr":"172.21.15.164:6817/962845839","cluster_addr":"172.21.15.164:6819/962845839","heartbeat_back_addr":"172.21.15.164:6823/962845839","heartbeat_front_addr":"172.21.15.164:6821/962845839","state":["exists","up"]},{"osd":9,"uuid":"6f7ceac6-fb61-4c6f-9da8-851b937c4586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6827","nonce":994024418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6829","nonce":994024418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6833","nonce":994024418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6831","nonce":994024418}]},"public_addr":"172.21.15.42:6827/994024418","cluster_addr":"172.21.15.42:6829/994024418","heartbeat_back_addr":"172.21.15.42:6833/994024418","heartbeat_front_addr":"172.21.15.42:6831/994024418","state":["exists","up"]},{"osd":10,"uuid":"553132c2-a6a1-410c-a7d8-ecdf9a67dafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6825","nonce":3268512227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6827","nonce":3268512227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6831","nonce":3268512227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6829","nonce":3268512227}]},"public_addr":"172.21.15.163:6825/3268512227","cluster_addr":"172.21.15.163:6827/3268512227","heartbeat_back_addr":"172.21.15.163:6831/3268512227","heartbeat_front_addr":"172.21.15.163:6829/3268512227","state":["exists","up"]},{"osd":11,"uuid":"09abc8e9-7884-400f-ab6d-8e5aa7bf31ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6825","nonce":1861076336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6827","nonce":1861076336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6831","nonce":1861076336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6829","nonce":1861076336}]},"public_addr":"172.21.15.164:6825/1861076336","cluster_addr":"172.21.15.164:6827/1861076336","heartbeat_back_addr":"172.21.15.164:6831/1861076336","heartbeat_front_addr":"172.21.15.164:6829/1861076336","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:41:38.761674+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:00.621745+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:23.248620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:47.232764+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:10.379922+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:34.502519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:58.694642+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:22.350247+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:46.988007+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:11.390474+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:36.150054+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:46:00.474470+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.42:6801/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6800/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1842521399":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/834172950":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1730109419":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6801/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:6800/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2687772889":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/3371433307":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2861719816":"2024-07-27T06:38:56.887264+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:49:20.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.668+0000 7f03791c7640 1 -- 172.21.15.42:0/2138223348 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f035c0b3c00 msgr2=0x7f035c0b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:20.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.668+0000 7f03791c7640 1 --2- 172.21.15.42:0/2138223348 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f035c0b3c00 0x7f035c0b6110 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f036400abe0 tx=0x7f036400a2d0 comp rx=0 tx=0).stop 2024-07-26T06:49:20.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.668+0000 7f03791c7640 1 -- 172.21.15.42:0/2138223348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0374109840 msgr2=0x7f03741a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:20.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.668+0000 7f03791c7640 1 --2- 172.21.15.42:0/2138223348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0374109840 0x7f03741a1900 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f036c005cc0 tx=0x7f036c02f380 comp rx=0 tx=0).stop 2024-07-26T06:49:20.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.668+0000 7f03791c7640 1 -- 172.21.15.42:0/2138223348 shutdown_connections 2024-07-26T06:49:20.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.668+0000 7f03791c7640 1 --2- 172.21.15.42:0/2138223348 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f035c0b3c00 0x7f035c0b6110 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:20.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.668+0000 7f03791c7640 1 --2- 172.21.15.42:0/2138223348 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0374109840 0x7f03741a1900 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:20.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.668+0000 7f03791c7640 1 --2- 172.21.15.42:0/2138223348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0374108ef0 0x7f037419c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:20.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.668+0000 7f03791c7640 1 --2- 172.21.15.42:0/2138223348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0374101a10 0x7f037419c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:20.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.668+0000 7f03791c7640 1 -- 172.21.15.42:0/2138223348 >> 172.21.15.42:0/2138223348 conn(0x7f03740775f0 msgr2=0x7f0374106d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:20.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.668+0000 7f03791c7640 1 -- 172.21.15.42:0/2138223348 shutdown_connections 2024-07-26T06:49:20.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:20.668+0000 7f03791c7640 1 -- 172.21.15.42:0/2138223348 wait complete. 2024-07-26T06:49:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:20 smithi164 ceph-mon[36970]: pgmap v369: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:49:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:20 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2138223348' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:49:21.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:20 smithi163 ceph-mon[36873]: pgmap v369: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:49:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:20 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2138223348' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:49:21.358 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:20 smithi042 ceph-mon[32775]: pgmap v369: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:49:21.359 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:20 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2138223348' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:49:21.392 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:49:21.661 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:23.071 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:22 smithi042 ceph-mon[32775]: pgmap v370: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:49:23.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:22 smithi164 ceph-mon[36970]: pgmap v370: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:49:23.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:22 smithi163 ceph-mon[36873]: pgmap v370: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:49:23.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.305+0000 7f49604f3640 1 -- 172.21.15.42:0/43704980 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4958101a20 msgr2=0x7f4958101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:23.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.305+0000 7f49604f3640 1 --2- 172.21.15.42:0/43704980 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4958101a20 0x7f4958101e30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4954009900 tx=0x7f495402a410 comp rx=0 tx=0).stop 2024-07-26T06:49:23.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.305+0000 7f49604f3640 1 -- 172.21.15.42:0/43704980 shutdown_connections 2024-07-26T06:49:23.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.305+0000 7f49604f3640 1 --2- 172.21.15.42:0/43704980 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4958106650 0x7f49581100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.305+0000 7f49604f3640 1 --2- 172.21.15.42:0/43704980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4958105d00 0x7f4958106110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.305+0000 7f49604f3640 1 --2- 172.21.15.42:0/43704980 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4958101a20 0x7f4958101e30 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.305+0000 7f49604f3640 1 -- 172.21.15.42:0/43704980 >> 172.21.15.42:0/43704980 conn(0x7f4958079860 msgr2=0x7f4958079c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.305+0000 7f49604f3640 1 -- 172.21.15.42:0/43704980 shutdown_connections 2024-07-26T06:49:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.305+0000 7f49604f3640 1 -- 172.21.15.42:0/43704980 wait complete. 2024-07-26T06:49:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.306+0000 7f49604f3640 1 Processor -- start 2024-07-26T06:49:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.306+0000 7f49604f3640 1 -- start start 2024-07-26T06:49:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.306+0000 7f49604f3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4958101a20 0x7f4958199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.306+0000 7f49604f3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4958105d00 0x7f495819a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.306+0000 7f49604f3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4958106650 0x7f495819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.306+0000 7f49604f3640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4958113560 con 0x7f4958105d00 2024-07-26T06:49:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.306+0000 7f49604f3640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f49581133e0 con 0x7f4958101a20 2024-07-26T06:49:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.306+0000 7f49604f3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f49581136e0 con 0x7f4958106650 2024-07-26T06:49:23.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.306+0000 7f495da8e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4958105d00 0x7f495819a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:23.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.307+0000 7f495da8e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4958105d00 0x7f495819a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42086/0 (socket says 172.21.15.42:42086) 2024-07-26T06:49:23.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.307+0000 7f495da8e640 1 -- 172.21.15.42:0/4114477681 learned_addr learned my addr 172.21.15.42:0/4114477681 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:23.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.307+0000 7f495e28f640 1 --2- 172.21.15.42:0/4114477681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4958101a20 0x7f4958199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:23.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.307+0000 7f495ea90640 1 --2- 172.21.15.42:0/4114477681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4958106650 0x7f495819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:23.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.307+0000 7f495da8e640 1 -- 172.21.15.42:0/4114477681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4958106650 msgr2=0x7f495819f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:23.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.307+0000 7f495da8e640 1 --2- 172.21.15.42:0/4114477681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4958106650 0x7f495819f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:23.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.307+0000 7f495da8e640 1 -- 172.21.15.42:0/4114477681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4958101a20 msgr2=0x7f4958199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:23.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.307+0000 7f495da8e640 1 --2- 172.21.15.42:0/4114477681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4958101a20 0x7f4958199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:23.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.307+0000 7f495da8e640 1 -- 172.21.15.42:0/4114477681 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495819fdc0 con 0x7f4958105d00 2024-07-26T06:49:23.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.307+0000 7f495ea90640 1 --2- 172.21.15.42:0/4114477681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4958106650 0x7f495819f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:49:23.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.307+0000 7f495da8e640 1 --2- 172.21.15.42:0/4114477681 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4958105d00 0x7f495819a4b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f494800c970 tx=0x7f494800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:23.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.307+0000 7f494f7fe640 1 -- 172.21.15.42:0/4114477681 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4948014070 con 0x7f4958105d00 2024-07-26T06:49:23.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.307+0000 7f494f7fe640 1 -- 172.21.15.42:0/4114477681 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4948004440 con 0x7f4958105d00 2024-07-26T06:49:23.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.307+0000 7f49604f3640 1 -- 172.21.15.42:0/4114477681 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49581a00b0 con 0x7f4958105d00 2024-07-26T06:49:23.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.307+0000 7f494f7fe640 1 -- 172.21.15.42:0/4114477681 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4948003e50 con 0x7f4958105d00 2024-07-26T06:49:23.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.307+0000 7f49604f3640 1 -- 172.21.15.42:0/4114477681 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49581a68d0 con 0x7f4958105d00 2024-07-26T06:49:23.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.308+0000 7f49604f3640 1 -- 172.21.15.42:0/4114477681 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f495819b040 con 0x7f4958105d00 2024-07-26T06:49:23.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.310+0000 7f494f7fe640 1 -- 172.21.15.42:0/4114477681 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f4948017070 con 0x7f4958105d00 2024-07-26T06:49:23.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.311+0000 7f494f7fe640 1 --2- 172.21.15.42:0/4114477681 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49440b3e20 0x7f49440b6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:23.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.311+0000 7f495e28f640 1 --2- 172.21.15.42:0/4114477681 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49440b3e20 0x7f49440b6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:23.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.311+0000 7f494f7fe640 1 -- 172.21.15.42:0/4114477681 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f49480e3ee0 con 0x7f4958105d00 2024-07-26T06:49:23.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.311+0000 7f495e28f640 1 --2- 172.21.15.42:0/4114477681 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49440b3e20 0x7f49440b6330 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4954031440 tx=0x7f4954005e80 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:23.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.314+0000 7f494f7fe640 1 -- 172.21.15.42:0/4114477681 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f494801e030 con 0x7f4958105d00 2024-07-26T06:49:23.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.559+0000 7f49604f3640 1 -- 172.21.15.42:0/4114477681 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f495810b5d0 con 0x7f4958105d00 2024-07-26T06:49:23.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.560+0000 7f494f7fe640 1 -- 172.21.15.42:0/4114477681 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11978 (secure 0 0 0) 0x7f495810b5d0 con 0x7f4958105d00 2024-07-26T06:49:23.561 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:49:23.561 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":28,"btime":"2024-07-26T06:47:26:107129+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:47:25.125103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24578,24583,24593,24601,24620]},"id":1}]} 2024-07-26T06:49:23.561 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 28 2024-07-26T06:49:23.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.562+0000 7f49604f3640 1 -- 172.21.15.42:0/4114477681 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49440b3e20 msgr2=0x7f49440b6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:23.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.562+0000 7f49604f3640 1 --2- 172.21.15.42:0/4114477681 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49440b3e20 0x7f49440b6330 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4954031440 tx=0x7f4954005e80 comp rx=0 tx=0).stop 2024-07-26T06:49:23.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.562+0000 7f49604f3640 1 -- 172.21.15.42:0/4114477681 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4958105d00 msgr2=0x7f495819a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:23.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.562+0000 7f49604f3640 1 --2- 172.21.15.42:0/4114477681 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4958105d00 0x7f495819a4b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f494800c970 tx=0x7f494800ce30 comp rx=0 tx=0).stop 2024-07-26T06:49:23.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.562+0000 7f49604f3640 1 -- 172.21.15.42:0/4114477681 shutdown_connections 2024-07-26T06:49:23.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.562+0000 7f49604f3640 1 --2- 172.21.15.42:0/4114477681 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49440b3e20 0x7f49440b6330 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:23.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.562+0000 7f49604f3640 1 --2- 172.21.15.42:0/4114477681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4958106650 0x7f495819f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:23.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.562+0000 7f49604f3640 1 --2- 172.21.15.42:0/4114477681 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4958105d00 0x7f495819a4b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:23.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.562+0000 7f49604f3640 1 --2- 172.21.15.42:0/4114477681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4958101a20 0x7f4958199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:23.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.562+0000 7f49604f3640 1 -- 172.21.15.42:0/4114477681 >> 172.21.15.42:0/4114477681 conn(0x7f4958079860 msgr2=0x7f4958107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:23.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.562+0000 7f49604f3640 1 -- 172.21.15.42:0/4114477681 shutdown_connections 2024-07-26T06:49:23.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:23.562+0000 7f49604f3640 1 -- 172.21.15.42:0/4114477681 wait complete. 2024-07-26T06:49:24.216 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:23 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/4114477681' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:49:24.218 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-07-26T06:47:06.780568+0000', 'modified': '2024-07-26T06:47:25.125103+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24620, 'mds_1': 24601, 'mds_2': 24593, 'mds_3': 24583, 'mds_4': 24578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14703': {'gid': 14703, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.42:6839/4044632348', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.42:6838', 'nonce': 4044632348}, {'type': 'v1', 'addr': '172.21.15.42:6839', 'nonce': 4044632348}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14709': {'gid': 14709, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.42:6841/1430155791', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.42:6840', 'nonce': 1430155791}, {'type': 'v1', 'addr': '172.21.15.42:6841', 'nonce': 1430155791}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24541': {'gid': 24541, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.164:6833/4144543280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6832', 'nonce': 4144543280}, {'type': 'v1', 'addr': '172.21.15.164:6833', 'nonce': 4144543280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24557': {'gid': 24557, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.163:6833/845444723', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6832', 'nonce': 845444723}, {'type': 'v1', 'addr': '172.21.15.163:6833', 'nonce': 845444723}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24571': {'gid': 24571, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.164:6835/2604534474', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6834', 'nonce': 2604534474}, {'type': 'v1', 'addr': '172.21.15.164:6835', 'nonce': 2604534474}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24578': {'gid': 24578, 'name': 'e', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.163:6835/375978170', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6834', 'nonce': 375978170}, {'type': 'v1', 'addr': '172.21.15.163:6835', 'nonce': 375978170}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24583': {'gid': 24583, 'name': 'c', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.164:6837/3642457979', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6836', 'nonce': 3642457979}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 3642457979}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24593': {'gid': 24593, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.163:6837/154041066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6836', 'nonce': 154041066}, {'type': 'v1', 'addr': '172.21.15.163:6837', 'nonce': 154041066}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24601': {'gid': 24601, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.164:6839/3593178314', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6838', 'nonce': 3593178314}, {'type': 'v1', 'addr': '172.21.15.164:6839', 'nonce': 3593178314}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24620': {'gid': 24620, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.163:6839/563658453', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6838', 'nonce': 563658453}, {'type': 'v1', 'addr': '172.21.15.163:6839', 'nonce': 563658453}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24620, 'qdb_cluster': [24578, 24583, 24593, 24601, 24620]} 2024-07-26T06:49:24.218 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-07-26T06:49:24.218 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell mds.e status 2024-07-26T06:49:24.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:23 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/4114477681' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:49:24.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:23 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/4114477681' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:49:24.493 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:25.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:24 smithi042 ceph-mon[32775]: pgmap v371: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:25.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:24 smithi164 ceph-mon[36970]: pgmap v371: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:25.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:24 smithi163 ceph-mon[36873]: pgmap v371: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:26.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.158+0000 7f1cf5d41640 1 -- 172.21.15.42:0/812183350 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1cf0108f00 msgr2=0x7f1cf0109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:26.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.158+0000 7f1cf5d41640 1 --2- 172.21.15.42:0/812183350 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1cf0108f00 0x7f1cf0109310 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4009900 tx=0x7f1ce402a410 comp rx=0 tx=0).stop 2024-07-26T06:49:26.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.158+0000 7f1cf5d41640 1 -- 172.21.15.42:0/812183350 shutdown_connections 2024-07-26T06:49:26.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.158+0000 7f1cf5d41640 1 --2- 172.21.15.42:0/812183350 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cf0109850 0x7f1cf0110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.158+0000 7f1cf5d41640 1 --2- 172.21.15.42:0/812183350 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1cf0108f00 0x7f1cf0109310 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.158+0000 7f1cf5d41640 1 --2- 172.21.15.42:0/812183350 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1cf01035c0 0x7f1cf01039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.158+0000 7f1cf5d41640 1 -- 172.21.15.42:0/812183350 >> 172.21.15.42:0/812183350 conn(0x7f1cf00ff030 msgr2=0x7f1cf0101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:26.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.158+0000 7f1cf5d41640 1 -- 172.21.15.42:0/812183350 shutdown_connections 2024-07-26T06:49:26.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.158+0000 7f1cf5d41640 1 -- 172.21.15.42:0/812183350 wait complete. 2024-07-26T06:49:26.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.159+0000 7f1cf5d41640 1 Processor -- start 2024-07-26T06:49:26.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.159+0000 7f1cf5d41640 1 -- start start 2024-07-26T06:49:26.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.159+0000 7f1cf5d41640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1cf01035c0 0x7f1cf0199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.159+0000 7f1cf5d41640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1cf0108f00 0x7f1cf019a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.159+0000 7f1cf5d41640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cf0109850 0x7f1cf019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.159+0000 7f1cf5d41640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf01134e0 con 0x7f1cf0108f00 2024-07-26T06:49:26.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.159+0000 7f1cf5d41640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf0113360 con 0x7f1cf0109850 2024-07-26T06:49:26.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.159+0000 7f1cf5d41640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf0113660 con 0x7f1cf01035c0 2024-07-26T06:49:26.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.159+0000 7f1ceeffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1cf0108f00 0x7f1cf019a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:26.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.159+0000 7f1ceeffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1cf0108f00 0x7f1cf019a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42092/0 (socket says 172.21.15.42:42092) 2024-07-26T06:49:26.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.159+0000 7f1ceeffd640 1 -- 172.21.15.42:0/3124712197 learned_addr learned my addr 172.21.15.42:0/3124712197 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:26.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.160+0000 7f1ceffff640 1 --2- 172.21.15.42:0/3124712197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cf0109850 0x7f1cf019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:26.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.160+0000 7f1cef7fe640 1 --2- 172.21.15.42:0/3124712197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1cf01035c0 0x7f1cf0199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:26.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.160+0000 7f1ceeffd640 1 -- 172.21.15.42:0/3124712197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1cf01035c0 msgr2=0x7f1cf0199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:26.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.160+0000 7f1ceeffd640 1 --2- 172.21.15.42:0/3124712197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1cf01035c0 0x7f1cf0199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.160+0000 7f1ceeffd640 1 -- 172.21.15.42:0/3124712197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cf0109850 msgr2=0x7f1cf019f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:26.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.160+0000 7f1ceeffd640 1 --2- 172.21.15.42:0/3124712197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cf0109850 0x7f1cf019f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.160+0000 7f1ceeffd640 1 -- 172.21.15.42:0/3124712197 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf019fe80 con 0x7f1cf0108f00 2024-07-26T06:49:26.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.160+0000 7f1ceeffd640 1 --2- 172.21.15.42:0/3124712197 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1cf0108f00 0x7f1cf019a4b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1ce402a930 tx=0x7f1ce4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:26.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.160+0000 7f1cecff9640 1 -- 172.21.15.42:0/3124712197 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce403f030 con 0x7f1cf0108f00 2024-07-26T06:49:26.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.160+0000 7f1cf5d41640 1 -- 172.21.15.42:0/3124712197 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf01a0110 con 0x7f1cf0108f00 2024-07-26T06:49:26.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.160+0000 7f1cecff9640 1 -- 172.21.15.42:0/3124712197 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ce4003cb0 con 0x7f1cf0108f00 2024-07-26T06:49:26.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.160+0000 7f1cecff9640 1 -- 172.21.15.42:0/3124712197 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4035070 con 0x7f1cf0108f00 2024-07-26T06:49:26.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.160+0000 7f1cf5d41640 1 -- 172.21.15.42:0/3124712197 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cf01a68d0 con 0x7f1cf0108f00 2024-07-26T06:49:26.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.163+0000 7f1cfc2c5740 1 Processor -- start 2024-07-26T06:49:26.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.163+0000 7f1cfc2c5740 1 -- start start 2024-07-26T06:49:26.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.163+0000 7f1cecff9640 1 -- 172.21.15.42:0/3124712197 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f1ce4042030 con 0x7f1cf0108f00 2024-07-26T06:49:26.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.163+0000 7f1cfc2c5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42357af60 0x55a4239203a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.163+0000 7f1cfc2c5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a4235520a0 0x55a4239208e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.163+0000 7f1cfc2c5740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a423920e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.163+0000 7f1cfc2c5740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55a423896940 con 0x55a42355a280 2024-07-26T06:49:26.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.163+0000 7f1cfc2c5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a423885e40 con 0x55a42357af60 2024-07-26T06:49:26.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.163+0000 7f1cfc2c5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55a423889e80 con 0x55a4235520a0 2024-07-26T06:49:26.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.163+0000 7f1ceeffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a423920e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:26.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.163+0000 7f1ceeffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a423920e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42094/0 (socket says 172.21.15.42:42094) 2024-07-26T06:49:26.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.163+0000 7f1ceeffd640 1 -- 172.21.15.42:0/3935157325 learned_addr learned my addr 172.21.15.42:0/3935157325 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:26.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.163+0000 7f1ceffff640 1 --2- 172.21.15.42:0/3935157325 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42357af60 0x55a4239203a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:26.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.164+0000 7f1cef7fe640 1 --2- 172.21.15.42:0/3935157325 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a4235520a0 0x55a4239208e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:26.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.164+0000 7f1ceffff640 1 -- 172.21.15.42:0/3935157325 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a4235520a0 msgr2=0x55a4239208e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:26.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.164+0000 7f1ceffff640 1 --2- 172.21.15.42:0/3935157325 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a4235520a0 0x55a4239208e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.164+0000 7f1ceffff640 1 -- 172.21.15.42:0/3935157325 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 msgr2=0x55a423920e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:26.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.164+0000 7f1ceffff640 1 --2- 172.21.15.42:0/3935157325 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a423920e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.164+0000 7f1ceffff640 1 -- 172.21.15.42:0/3935157325 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a42382ca60 con 0x55a42357af60 2024-07-26T06:49:26.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.164+0000 7f1ceeffd640 1 --2- 172.21.15.42:0/3935157325 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a423920e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:49:26.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.164+0000 7f1cecff9640 1 --2- 172.21.15.42:0/3124712197 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1cd80b3bd0 0x7f1cd80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.164+0000 7f1cef7fe640 1 --2- 172.21.15.42:0/3124712197 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1cd80b3bd0 0x7f1cd80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:26.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.164+0000 7f1ceffff640 1 --2- 172.21.15.42:0/3935157325 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42357af60 0x55a4239203a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1ce800add0 tx=0x7f1ce800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a882677370c5d0a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:26.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.164+0000 7f1cecff9640 1 -- 172.21.15.42:0/3124712197 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f1ce4104260 con 0x7f1cf0108f00 2024-07-26T06:49:26.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.165+0000 7f1cd5ffb640 1 -- 172.21.15.42:0/3935157325 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8003b70 con 0x55a42357af60 2024-07-26T06:49:26.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.165+0000 7f1cd5ffb640 1 -- 172.21.15.42:0/3935157325 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ce8003d10 con 0x55a42357af60 2024-07-26T06:49:26.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.165+0000 7f1cd5ffb640 1 -- 172.21.15.42:0/3935157325 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce8010b60 con 0x55a42357af60 2024-07-26T06:49:26.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.166+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/3935157325 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42357af60 msgr2=0x55a4239203a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:26.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.166+0000 7f1cef7fe640 1 --2- 172.21.15.42:0/3124712197 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1cd80b3bd0 0x7f1cd80b60e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x55a42391f020 tx=0x7f1ce000c430 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:26.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.166+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/3935157325 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42357af60 0x55a4239203a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1ce800add0 tx=0x7f1ce800f790 comp rx=0 tx=0).stop 2024-07-26T06:49:26.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.166+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/3935157325 shutdown_connections 2024-07-26T06:49:26.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.166+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/3935157325 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a423920e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.166+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/3935157325 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a4235520a0 0x55a4239208e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.166+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/3935157325 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42357af60 0x55a4239203a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.166+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/3935157325 >> 172.21.15.42:0/3935157325 conn(0x55a42356f350 msgr2=0x55a42391e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:26.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.166+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/3935157325 shutdown_connections 2024-07-26T06:49:26.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.166+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/3935157325 wait complete. 2024-07-26T06:49:26.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.167+0000 7f1cfc2c5740 1 -- start start 2024-07-26T06:49:26.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.167+0000 7f1cfc2c5740 1 Processor -- start 2024-07-26T06:49:26.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.167+0000 7f1cfc2c5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a4235520a0 0x55a42392bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.167+0000 7f1cfc2c5740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a42392c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.167+0000 7f1cfc2c5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42357af60 0x55a423937ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.167+0000 7f1cfc2c5740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55a4236e95f0 con 0x55a42355a280 2024-07-26T06:49:26.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.167+0000 7f1cfc2c5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a4238288b0 con 0x55a42357af60 2024-07-26T06:49:26.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.167+0000 7f1cfc2c5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55a42392eb80 con 0x55a4235520a0 2024-07-26T06:49:26.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.167+0000 7f1cfc2c5740 10 client.0 _ll_register_callbacks cb 0x55a4238327f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:49:26.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.167+0000 7f1ceeffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a42392c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:26.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.167+0000 7f1ceeffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a42392c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42102/0 (socket says 172.21.15.42:42102) 2024-07-26T06:49:26.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.167+0000 7f1ceeffd640 1 -- 172.21.15.42:0/2055388816 learned_addr learned my addr 172.21.15.42:0/2055388816 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:26.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.168+0000 7f1cef7fe640 1 --2- 172.21.15.42:0/2055388816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a4235520a0 0x55a42392bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:26.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.168+0000 7f1ceffff640 1 --2- 172.21.15.42:0/2055388816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42357af60 0x55a423937ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:26.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.168+0000 7f1ceeffd640 1 -- 172.21.15.42:0/2055388816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a4235520a0 msgr2=0x55a42392bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:26.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.168+0000 7f1ceeffd640 1 --2- 172.21.15.42:0/2055388816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a4235520a0 0x55a42392bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.168+0000 7f1ceeffd640 1 -- 172.21.15.42:0/2055388816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42357af60 msgr2=0x55a423937ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:26.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.168+0000 7f1ceeffd640 1 --2- 172.21.15.42:0/2055388816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42357af60 0x55a423937ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.168+0000 7f1ceeffd640 1 -- 172.21.15.42:0/2055388816 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4235e6540 con 0x55a42355a280 2024-07-26T06:49:26.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.168+0000 7f1ceeffd640 1 --2- 172.21.15.42:0/2055388816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a42392c530 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40cae90 tx=0x7f1ce4005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d654c09e531435a8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:26.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.168+0000 7f1cceffd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:49:26.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.168+0000 7f1cceffd640 1 -- 172.21.15.42:0/2055388816 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4049070 con 0x55a42355a280 2024-07-26T06:49:26.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.168+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/2055388816 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ce8003d10 con 0x55a42355a280 2024-07-26T06:49:26.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.168+0000 7f1cceffd640 1 -- 172.21.15.42:0/2055388816 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f1ce40c7420 con 0x55a42355a280 2024-07-26T06:49:26.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.168+0000 7f1cceffd640 1 -- 172.21.15.42:0/2055388816 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1ce40c7050 con 0x55a42355a280 2024-07-26T06:49:26.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.168+0000 7f1cfc2c5740 10 client.15072 fetch_fsmap learned FSMap version 28 2024-07-26T06:49:26.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.168+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/2055388816 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a42393ac40 con 0x55a42355a280 2024-07-26T06:49:26.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.169+0000 7f1cceffd640 1 -- 172.21.15.42:0/2055388816 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ce8003d10 con 0x55a42355a280 2024-07-26T06:49:26.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.169+0000 7f1cceffd640 1 -- 172.21.15.42:0/2055388816 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1ce403bbe0 con 0x55a42355a280 2024-07-26T06:49:26.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.169+0000 7f1cfc2c5740 10 client.15072 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:49:26.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.169+0000 7f1cfc2c5740 10 client.15072 resolve_mds: resolved name 'e' to daemon mds.e 2024-07-26T06:49:26.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.169+0000 7f1cfc2c5740 20 client.15072 populate_metadata read hostname 'smithi042' 2024-07-26T06:49:26.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.169+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/2055388816 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x55a4235398e0 0x55a42393e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.169+0000 7f1cfc2c5740 4 client.15072 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:49:26.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.169+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/2055388816 --> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a423898890 con 0x55a4235398e0 2024-07-26T06:49:26.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.169+0000 7f1cef7fe640 1 --2- 172.21.15.42:0/2055388816 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x55a4235398e0 0x55a42393e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:26.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.170+0000 7f1cef7fe640 1 --2- 172.21.15.42:0/2055388816 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x55a4235398e0 0x55a42393e750 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e4242534f1809c79 server_cookie=dbbc90458258b679 in_seq=0 out_seq=0 2024-07-26T06:49:26.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.170+0000 7f1cceffd640 10 client.15072 ms_handle_connect on v2:172.21.15.163:6834/375978170 2024-07-26T06:49:26.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.171+0000 7f1cceffd640 1 -- 172.21.15.42:0/2055388816 <== mds.4 v2:172.21.15.163:6834/375978170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a423898890 con 0x55a4235398e0 2024-07-26T06:49:26.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.171+0000 7f1cceffd640 10 client.15072 handle_command_reply: tid=0 2024-07-26T06:49:26.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.171+0000 7f1cfc2c5740 1 client.15072 shutdown 2024-07-26T06:49:26.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.171+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/2055388816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 msgr2=0x55a42392c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:26.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.171+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/2055388816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a42392c530 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40cae90 tx=0x7f1ce4005de0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.171+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/2055388816 shutdown_connections 2024-07-26T06:49:26.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.171+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/2055388816 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x55a4235398e0 0x55a42393e750 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.171+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/2055388816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42357af60 0x55a423937ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.171+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/2055388816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a42392c530 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.171+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/2055388816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a4235520a0 0x55a42392bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.171+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/2055388816 >> 172.21.15.42:0/2055388816 conn(0x55a42356f350 msgr2=0x55a423923530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:26.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.171+0000 7f1cceffd640 0 client.15072 ms_handle_reset on v2:172.21.15.163:6834/375978170 2024-07-26T06:49:26.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.171+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/2055388816 shutdown_connections 2024-07-26T06:49:26.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.171+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/2055388816 wait complete. 2024-07-26T06:49:26.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.171+0000 7f1cfc2c5740 20 client.15072 trim_cache size 0 max 16384 2024-07-26T06:49:26.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.189+0000 7f1cfc2c5740 1 Processor -- start 2024-07-26T06:49:26.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.189+0000 7f1cfc2c5740 1 -- start start 2024-07-26T06:49:26.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.189+0000 7f1cfc2c5740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a4235520a0 0x55a423925620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.189+0000 7f1cfc2c5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42355a280 0x55a423925b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.189+0000 7f1cfc2c5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a42357af60 0x55a42392a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.189+0000 7f1cfc2c5740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55a42393ca90 con 0x55a4235520a0 2024-07-26T06:49:26.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.189+0000 7f1cfc2c5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a42393d160 con 0x55a42355a280 2024-07-26T06:49:26.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.189+0000 7f1cfc2c5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55a42393d830 con 0x55a42357af60 2024-07-26T06:49:26.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.189+0000 7f1cef7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a4235520a0 0x55a423925620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:26.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.189+0000 7f1cef7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a4235520a0 0x55a423925620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42110/0 (socket says 172.21.15.42:42110) 2024-07-26T06:49:26.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.189+0000 7f1cef7fe640 1 -- 172.21.15.42:0/607591641 learned_addr learned my addr 172.21.15.42:0/607591641 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:26.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.190+0000 7f1cef7fe640 1 -- 172.21.15.42:0/607591641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a42357af60 msgr2=0x55a42392a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:26.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.190+0000 7f1cef7fe640 1 --2- 172.21.15.42:0/607591641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a42357af60 0x55a42392a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.190+0000 7f1cef7fe640 1 -- 172.21.15.42:0/607591641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42355a280 msgr2=0x55a423925b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:26.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.190+0000 7f1cef7fe640 1 --2- 172.21.15.42:0/607591641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42355a280 0x55a423925b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.190+0000 7f1cef7fe640 1 -- 172.21.15.42:0/607591641 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a423922690 con 0x55a4235520a0 2024-07-26T06:49:26.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.190+0000 7f1cef7fe640 1 --2- 172.21.15.42:0/607591641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a4235520a0 0x55a423925620 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1ce001b4a0 tx=0x7f1ce0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b44067970fbd24e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:26.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.190+0000 7f1cd57fa640 1 -- 172.21.15.42:0/607591641 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0027670 con 0x55a4235520a0 2024-07-26T06:49:26.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.190+0000 7f1cd57fa640 1 -- 172.21.15.42:0/607591641 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ce0027cc0 con 0x55a4235520a0 2024-07-26T06:49:26.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.190+0000 7f1cd57fa640 1 -- 172.21.15.42:0/607591641 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce001a3e0 con 0x55a4235520a0 2024-07-26T06:49:26.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.191+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/607591641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a4235520a0 msgr2=0x55a423925620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:26.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.191+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/607591641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a4235520a0 0x55a423925620 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1ce001b4a0 tx=0x7f1ce0014000 comp rx=0 tx=0).stop 2024-07-26T06:49:26.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.191+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/607591641 shutdown_connections 2024-07-26T06:49:26.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.191+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/607591641 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a42357af60 0x55a42392a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.191+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/607591641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42355a280 0x55a423925b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.191+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/607591641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a4235520a0 0x55a423925620 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.191+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/607591641 >> 172.21.15.42:0/607591641 conn(0x55a42356f350 msgr2=0x55a423926ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.191+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/607591641 shutdown_connections 2024-07-26T06:49:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.191+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/607591641 wait complete. 2024-07-26T06:49:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.191+0000 7f1cfc2c5740 1 -- start start 2024-07-26T06:49:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.191+0000 7f1cfc2c5740 1 Processor -- start 2024-07-26T06:49:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1cfc2c5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a4235520a0 0x55a42391c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1cfc2c5740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a42391cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1cfc2c5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42357af60 0x55a42391d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1cfc2c5740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55a42393e5d0 con 0x55a42355a280 2024-07-26T06:49:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1cfc2c5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a42393df00 con 0x55a42357af60 2024-07-26T06:49:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1cfc2c5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55a42393eca0 con 0x55a4235520a0 2024-07-26T06:49:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1cfc2c5740 10 client.0 _ll_register_callbacks cb 0x7f1ce4004520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:49:26.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1ceeffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a42391cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1ceeffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a42391cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42112/0 (socket says 172.21.15.42:42112) 2024-07-26T06:49:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1ceeffd640 1 -- 172.21.15.42:0/176822635 learned_addr learned my addr 172.21.15.42:0/176822635 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1cef7fe640 1 --2- 172.21.15.42:0/176822635 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a4235520a0 0x55a42391c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1ceffff640 1 --2- 172.21.15.42:0/176822635 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42357af60 0x55a42391d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1ceeffd640 1 -- 172.21.15.42:0/176822635 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a4235520a0 msgr2=0x55a42391c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1ceeffd640 1 --2- 172.21.15.42:0/176822635 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a4235520a0 0x55a42391c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1ceeffd640 1 -- 172.21.15.42:0/176822635 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42357af60 msgr2=0x55a42391d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1ceeffd640 1 --2- 172.21.15.42:0/176822635 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42357af60 0x55a42391d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1ceeffd640 1 -- 172.21.15.42:0/176822635 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4239459f0 con 0x55a42355a280 2024-07-26T06:49:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1ceffff640 1 --2- 172.21.15.42:0/176822635 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42357af60 0x55a42391d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:49:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1cef7fe640 1 --2- 172.21.15.42:0/176822635 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a4235520a0 0x55a42391c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:49:26.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1ceeffd640 1 --2- 172.21.15.42:0/176822635 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a42391cd90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40c8400 tx=0x7f1ce40c7650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f304bf292d350843 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.192+0000 7f1cceffd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:49:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.193+0000 7f1cceffd640 1 -- 172.21.15.42:0/176822635 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4049070 con 0x55a42355a280 2024-07-26T06:49:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.193+0000 7f1cceffd640 1 -- 172.21.15.42:0/176822635 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f1ce40d3fc0 con 0x55a42355a280 2024-07-26T06:49:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.193+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/176822635 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a423894330 con 0x55a42355a280 2024-07-26T06:49:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.193+0000 7f1cceffd640 1 -- 172.21.15.42:0/176822635 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1ce40eb070 con 0x55a42355a280 2024-07-26T06:49:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.193+0000 7f1cfc2c5740 10 client.15078 fetch_fsmap learned FSMap version 28 2024-07-26T06:49:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.193+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/176822635 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a423946c70 con 0x55a42355a280 2024-07-26T06:49:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.193+0000 7f1cceffd640 1 -- 172.21.15.42:0/176822635 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a423894330 con 0x55a42355a280 2024-07-26T06:49:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.193+0000 7f1cceffd640 1 -- 172.21.15.42:0/176822635 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1ce40ec290 con 0x55a42355a280 2024-07-26T06:49:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.193+0000 7f1cfc2c5740 10 client.15078 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:49:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.193+0000 7f1cfc2c5740 10 client.15078 resolve_mds: resolved name 'e' to daemon mds.e 2024-07-26T06:49:26.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.193+0000 7f1cfc2c5740 20 client.15078 populate_metadata read hostname 'smithi042' 2024-07-26T06:49:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.193+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/176822635 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x55a42350ec90 0x55a4239b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.193+0000 7f1cfc2c5740 4 client.15078 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:49:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.193+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/176822635 --> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] -- command(tid 0: {"prefix": "status"}) -- 0x55a423891a70 con 0x55a42350ec90 2024-07-26T06:49:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.194+0000 7f1cef7fe640 1 --2- 172.21.15.42:0/176822635 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x55a42350ec90 0x55a4239b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.194+0000 7f1cceffd640 10 client.15078 ms_handle_connect on v2:172.21.15.163:6834/375978170 2024-07-26T06:49:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.194+0000 7f1cef7fe640 1 --2- 172.21.15.42:0/176822635 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x55a42350ec90 0x55a4239b2ce0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f0ccb5a7dc2543b3 server_cookie=fe3a8d2bf32e29a1 in_seq=0 out_seq=0 2024-07-26T06:49:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.194+0000 7f1cceffd640 1 -- 172.21.15.42:0/176822635 <== mds.4 v2:172.21.15.163:6834/375978170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a423891a70 con 0x55a42350ec90 2024-07-26T06:49:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.194+0000 7f1cceffd640 10 client.15078 handle_command_reply: tid=0 2024-07-26T06:49:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.194+0000 7f1cfc2c5740 1 client.15078 shutdown 2024-07-26T06:49:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.194+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/176822635 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 msgr2=0x55a42391cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/176822635 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a42391cd90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40c8400 tx=0x7f1ce40c7650 comp rx=0 tx=0).stop 2024-07-26T06:49:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/176822635 shutdown_connections 2024-07-26T06:49:26.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/176822635 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x55a42350ec90 0x55a4239b2ce0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/176822635 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a42357af60 0x55a42391d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/176822635 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a42355a280 0x55a42391cd90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cfc2c5740 1 --2- 172.21.15.42:0/176822635 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a4235520a0 0x55a42391c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cceffd640 0 client.15078 ms_handle_reset on v2:172.21.15.163:6834/375978170 2024-07-26T06:49:26.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/176822635 >> 172.21.15.42:0/176822635 conn(0x55a42356f350 msgr2=0x55a4239234a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:26.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/176822635 shutdown_connections 2024-07-26T06:49:26.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cfc2c5740 1 -- 172.21.15.42:0/176822635 wait complete. 2024-07-26T06:49:26.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cfc2c5740 20 client.15078 trim_cache size 0 max 16384 2024-07-26T06:49:26.201 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:49:26.201 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_fsid": "8b3e2750-4b19-11ef-bcb6-c7b262605968", 2024-07-26T06:49:26.202 INFO:teuthology.orchestra.run.smithi042.stdout: "whoami": 4, 2024-07-26T06:49:26.202 INFO:teuthology.orchestra.run.smithi042.stdout: "id": 24578, 2024-07-26T06:49:26.202 INFO:teuthology.orchestra.run.smithi042.stdout: "want_state": "up:active", 2024-07-26T06:49:26.202 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "up:active", 2024-07-26T06:49:26.202 INFO:teuthology.orchestra.run.smithi042.stdout: "fs_name": "cephfs", 2024-07-26T06:49:26.202 INFO:teuthology.orchestra.run.smithi042.stdout: "rank_uptime": 124.095399124, 2024-07-26T06:49:26.202 INFO:teuthology.orchestra.run.smithi042.stdout: "mdsmap_epoch": 28, 2024-07-26T06:49:26.202 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch": 80, 2024-07-26T06:49:26.202 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch_barrier": 80, 2024-07-26T06:49:26.202 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 175.67568306999999 2024-07-26T06:49:26.202 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:49:26.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cf5d41640 1 -- 172.21.15.42:0/3124712197 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1cd80b3bd0 msgr2=0x7f1cd80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:26.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cf5d41640 1 --2- 172.21.15.42:0/3124712197 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1cd80b3bd0 0x7f1cd80b60e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x55a42391f020 tx=0x7f1ce000c430 comp rx=0 tx=0).stop 2024-07-26T06:49:26.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cf5d41640 1 -- 172.21.15.42:0/3124712197 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1cf0108f00 msgr2=0x7f1cf019a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:26.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cf5d41640 1 --2- 172.21.15.42:0/3124712197 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1cf0108f00 0x7f1cf019a4b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1ce402a930 tx=0x7f1ce4004270 comp rx=0 tx=0).stop 2024-07-26T06:49:26.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cf5d41640 1 -- 172.21.15.42:0/3124712197 shutdown_connections 2024-07-26T06:49:26.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cf5d41640 1 --2- 172.21.15.42:0/3124712197 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1cd80b3bd0 0x7f1cd80b60e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cf5d41640 1 --2- 172.21.15.42:0/3124712197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1cf0109850 0x7f1cf019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cf5d41640 1 --2- 172.21.15.42:0/3124712197 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1cf0108f00 0x7f1cf019a4b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cf5d41640 1 --2- 172.21.15.42:0/3124712197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1cf01035c0 0x7f1cf0199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:26.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.195+0000 7f1cf5d41640 1 -- 172.21.15.42:0/3124712197 >> 172.21.15.42:0/3124712197 conn(0x7f1cf00ff030 msgr2=0x7f1cf0101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:26.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.196+0000 7f1cf5d41640 1 -- 172.21.15.42:0/3124712197 shutdown_connections 2024-07-26T06:49:26.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:26.196+0000 7f1cf5d41640 1 -- 172.21.15.42:0/3124712197 wait complete. 2024-07-26T06:49:26.919 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell mds.c status 2024-07-26T06:49:27.190 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:26 smithi164 ceph-mon[36970]: pgmap v372: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:49:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:26 smithi163 ceph-mon[36873]: pgmap v372: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:49:27.313 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:26 smithi042 ceph-mon[32775]: pgmap v372: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:49:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.793+0000 7f19e585d640 1 -- 172.21.15.42:0/714916795 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19e0108ed0 msgr2=0x7f19e01092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.793+0000 7f19e585d640 1 --2- 172.21.15.42:0/714916795 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19e0108ed0 0x7f19e01092e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f19c8009900 tx=0x7f19c802a410 comp rx=0 tx=0).stop 2024-07-26T06:49:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.793+0000 7f19e585d640 1 -- 172.21.15.42:0/714916795 shutdown_connections 2024-07-26T06:49:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.793+0000 7f19e585d640 1 --2- 172.21.15.42:0/714916795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19e0109820 0x7f19e0110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.793+0000 7f19e585d640 1 --2- 172.21.15.42:0/714916795 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19e0108ed0 0x7f19e01092e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.793+0000 7f19e585d640 1 --2- 172.21.15.42:0/714916795 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19e0103590 0x7f19e01039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.793+0000 7f19e585d640 1 -- 172.21.15.42:0/714916795 >> 172.21.15.42:0/714916795 conn(0x7f19e007b280 msgr2=0x7f19e007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:28.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.793+0000 7f19e585d640 1 -- 172.21.15.42:0/714916795 shutdown_connections 2024-07-26T06:49:28.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.793+0000 7f19e585d640 1 -- 172.21.15.42:0/714916795 wait complete. 2024-07-26T06:49:28.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.794+0000 7f19e585d640 1 Processor -- start 2024-07-26T06:49:28.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.794+0000 7f19e585d640 1 -- start start 2024-07-26T06:49:28.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.795+0000 7f19e585d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19e0103590 0x7f19e0199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.795+0000 7f19e585d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19e0108ed0 0x7f19e019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.795+0000 7f19e585d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19e0109820 0x7f19e019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.795+0000 7f19e585d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f19e01134b0 con 0x7f19e0109820 2024-07-26T06:49:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.795+0000 7f19e585d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f19e0113330 con 0x7f19e0108ed0 2024-07-26T06:49:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.795+0000 7f19e585d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f19e0113630 con 0x7f19e0103590 2024-07-26T06:49:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.795+0000 7f19df7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19e0109820 0x7f19e019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.795+0000 7f19df7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19e0109820 0x7f19e019f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51350/0 (socket says 172.21.15.42:51350) 2024-07-26T06:49:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.795+0000 7f19df7fe640 1 -- 172.21.15.42:0/4217223572 learned_addr learned my addr 172.21.15.42:0/4217223572 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.795+0000 7f19d67fc640 1 --2- 172.21.15.42:0/4217223572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19e0108ed0 0x7f19e019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.795+0000 7f19deffd640 1 --2- 172.21.15.42:0/4217223572 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19e0103590 0x7f19e0199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.796+0000 7f19df7fe640 1 -- 172.21.15.42:0/4217223572 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19e0103590 msgr2=0x7f19e0199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.796+0000 7f19df7fe640 1 --2- 172.21.15.42:0/4217223572 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19e0103590 0x7f19e0199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.796+0000 7f19df7fe640 1 -- 172.21.15.42:0/4217223572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19e0108ed0 msgr2=0x7f19e019a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.796+0000 7f19df7fe640 1 --2- 172.21.15.42:0/4217223572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19e0108ed0 0x7f19e019a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.796+0000 7f19df7fe640 1 -- 172.21.15.42:0/4217223572 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e019fe50 con 0x7f19e0109820 2024-07-26T06:49:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.796+0000 7f19d67fc640 1 --2- 172.21.15.42:0/4217223572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19e0108ed0 0x7f19e019a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:49:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.796+0000 7f19deffd640 1 --2- 172.21.15.42:0/4217223572 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19e0103590 0x7f19e0199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:49:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.796+0000 7f19df7fe640 1 --2- 172.21.15.42:0/4217223572 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19e0109820 0x7f19e019f720 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f19d800eaa0 tx=0x7f19d800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.796+0000 7f19e485b640 1 -- 172.21.15.42:0/4217223572 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d800cdb0 con 0x7f19e0109820 2024-07-26T06:49:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.796+0000 7f19e485b640 1 -- 172.21.15.42:0/4217223572 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d8005020 con 0x7f19e0109820 2024-07-26T06:49:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.796+0000 7f19e485b640 1 -- 172.21.15.42:0/4217223572 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d80116d0 con 0x7f19e0109820 2024-07-26T06:49:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.796+0000 7f19e585d640 1 -- 172.21.15.42:0/4217223572 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e01a0140 con 0x7f19e0109820 2024-07-26T06:49:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.796+0000 7f19e585d640 1 -- 172.21.15.42:0/4217223572 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e01a68a0 con 0x7f19e0109820 2024-07-26T06:49:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.798+0000 7f19ebde1740 1 Processor -- start 2024-07-26T06:49:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.799+0000 7f19e485b640 1 -- 172.21.15.42:0/4217223572 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f19d80051d0 con 0x7f19e0109820 2024-07-26T06:49:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.799+0000 7f19ebde1740 1 -- start start 2024-07-26T06:49:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.799+0000 7f19ebde1740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1488f60 0x5579c182e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.799+0000 7f19ebde1740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c14600a0 0x5579c182e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.799+0000 7f19ebde1740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c1468280 0x5579c182ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.799+0000 7f19ebde1740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5579c17a4940 con 0x5579c1468280 2024-07-26T06:49:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.799+0000 7f19ebde1740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5579c1793e40 con 0x5579c1488f60 2024-07-26T06:49:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.799+0000 7f19ebde1740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5579c1797e80 con 0x5579c14600a0 2024-07-26T06:49:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.799+0000 7f19d67fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c1468280 0x5579c182ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.799+0000 7f19d67fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c1468280 0x5579c182ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51352/0 (socket says 172.21.15.42:51352) 2024-07-26T06:49:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.799+0000 7f19d67fc640 1 -- 172.21.15.42:0/3608143403 learned_addr learned my addr 172.21.15.42:0/3608143403 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.799+0000 7f19deffd640 1 --2- 172.21.15.42:0/3608143403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c14600a0 0x5579c182e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.799+0000 7f19d67fc640 1 --2- 172.21.15.42:0/3608143403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1488f60 0x5579c182e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.799+0000 7f19d67fc640 1 -- 172.21.15.42:0/3608143403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c14600a0 msgr2=0x5579c182e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.799+0000 7f19d67fc640 1 --2- 172.21.15.42:0/3608143403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c14600a0 0x5579c182e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.800+0000 7f19d67fc640 1 -- 172.21.15.42:0/3608143403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1488f60 msgr2=0x5579c182e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.800+0000 7f19d67fc640 1 --2- 172.21.15.42:0/3608143403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1488f60 0x5579c182e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.800+0000 7f19d67fc640 1 -- 172.21.15.42:0/3608143403 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579c173aa60 con 0x5579c1468280 2024-07-26T06:49:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.800+0000 7f19e485b640 1 --2- 172.21.15.42:0/4217223572 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f19cc0b3c50 0x7f19cc0b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.800+0000 7f19deffd640 1 --2- 172.21.15.42:0/3608143403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c14600a0 0x5579c182e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:49:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.800+0000 7f19deffd640 1 --2- 172.21.15.42:0/4217223572 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f19cc0b3c50 0x7f19cc0b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.800+0000 7f19d67fc640 1 --2- 172.21.15.42:0/3608143403 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c1468280 0x5579c182ee20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f19c8031440 tx=0x7f19c8009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989123db5f5017fd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:28.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.800+0000 7f19e485b640 1 -- 172.21.15.42:0/4217223572 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f19d80e3e10 con 0x7f19e0109820 2024-07-26T06:49:28.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.800+0000 7f19d5ffb640 1 -- 172.21.15.42:0/3608143403 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c8035050 con 0x5579c1468280 2024-07-26T06:49:28.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.800+0000 7f19d5ffb640 1 -- 172.21.15.42:0/3608143403 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19c8041020 con 0x5579c1468280 2024-07-26T06:49:28.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.800+0000 7f19d5ffb640 1 -- 172.21.15.42:0/3608143403 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c80053a0 con 0x5579c1468280 2024-07-26T06:49:28.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.801+0000 7f19ebde1740 1 -- 172.21.15.42:0/3608143403 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c1468280 msgr2=0x5579c182ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:28.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.801+0000 7f19deffd640 1 --2- 172.21.15.42:0/4217223572 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f19cc0b3c50 0x7f19cc0b6160 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f19d0005fd0 tx=0x7f19d000b2c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:28.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.801+0000 7f19ebde1740 1 --2- 172.21.15.42:0/3608143403 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c1468280 0x5579c182ee20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f19c8031440 tx=0x7f19c8009560 comp rx=0 tx=0).stop 2024-07-26T06:49:28.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.801+0000 7f19ebde1740 1 -- 172.21.15.42:0/3608143403 shutdown_connections 2024-07-26T06:49:28.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.801+0000 7f19ebde1740 1 --2- 172.21.15.42:0/3608143403 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c1468280 0x5579c182ee20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.801+0000 7f19ebde1740 1 --2- 172.21.15.42:0/3608143403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c14600a0 0x5579c182e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.801+0000 7f19ebde1740 1 --2- 172.21.15.42:0/3608143403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1488f60 0x5579c182e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.801+0000 7f19ebde1740 1 -- 172.21.15.42:0/3608143403 >> 172.21.15.42:0/3608143403 conn(0x5579c147d350 msgr2=0x5579c182c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.801+0000 7f19ebde1740 1 -- 172.21.15.42:0/3608143403 shutdown_connections 2024-07-26T06:49:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.801+0000 7f19ebde1740 1 -- 172.21.15.42:0/3608143403 wait complete. 2024-07-26T06:49:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.802+0000 7f19ebde1740 1 -- start start 2024-07-26T06:49:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.802+0000 7f19ebde1740 1 Processor -- start 2024-07-26T06:49:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.802+0000 7f19ebde1740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c14600a0 0x5579c1839ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.802+0000 7f19ebde1740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1468280 0x5579c183a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.802+0000 7f19ebde1740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c1488f60 0x5579c1845e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.802+0000 7f19ebde1740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5579c15f75f0 con 0x5579c14600a0 2024-07-26T06:49:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.802+0000 7f19ebde1740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5579c17368b0 con 0x5579c1468280 2024-07-26T06:49:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.802+0000 7f19ebde1740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5579c183cb20 con 0x5579c1488f60 2024-07-26T06:49:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.802+0000 7f19deffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c14600a0 0x5579c1839ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.802+0000 7f19ebde1740 10 client.0 _ll_register_callbacks cb 0x5579c17407f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:49:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.802+0000 7f19deffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c14600a0 0x5579c1839ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51356/0 (socket says 172.21.15.42:51356) 2024-07-26T06:49:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.802+0000 7f19deffd640 1 -- 172.21.15.42:0/2133837241 learned_addr learned my addr 172.21.15.42:0/2133837241 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.803+0000 7f19d67fc640 1 --2- 172.21.15.42:0/2133837241 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1468280 0x5579c183a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.803+0000 7f19deffd640 1 -- 172.21.15.42:0/2133837241 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c1488f60 msgr2=0x5579c1845e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.803+0000 7f19deffd640 1 --2- 172.21.15.42:0/2133837241 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c1488f60 0x5579c1845e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.803+0000 7f19deffd640 1 -- 172.21.15.42:0/2133837241 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1468280 msgr2=0x5579c183a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.803+0000 7f19deffd640 1 --2- 172.21.15.42:0/2133837241 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1468280 0x5579c183a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.803+0000 7f19deffd640 1 -- 172.21.15.42:0/2133837241 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579c14f4540 con 0x5579c14600a0 2024-07-26T06:49:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.803+0000 7f19deffd640 1 --2- 172.21.15.42:0/2133837241 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c14600a0 0x5579c1839ee0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f19d0004630 tx=0x7f19d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52cf835420594f0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.803+0000 7f19beffd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:49:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.803+0000 7f19beffd640 1 -- 172.21.15.42:0/2133837241 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0023070 con 0x5579c14600a0 2024-07-26T06:49:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.803+0000 7f19beffd640 1 -- 172.21.15.42:0/2133837241 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f19d00119b0 con 0x5579c14600a0 2024-07-26T06:49:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.803+0000 7f19ebde1740 1 -- 172.21.15.42:0/2133837241 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19c8041020 con 0x5579c14600a0 2024-07-26T06:49:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.803+0000 7f19beffd640 1 -- 172.21.15.42:0/2133837241 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f19d002f050 con 0x5579c14600a0 2024-07-26T06:49:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.803+0000 7f19ebde1740 10 client.15090 fetch_fsmap learned FSMap version 28 2024-07-26T06:49:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.803+0000 7f19ebde1740 1 -- 172.21.15.42:0/2133837241 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5579c1848ae0 con 0x5579c14600a0 2024-07-26T06:49:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.803+0000 7f19beffd640 1 -- 172.21.15.42:0/2133837241 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19c8041020 con 0x5579c14600a0 2024-07-26T06:49:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.804+0000 7f19beffd640 1 -- 172.21.15.42:0/2133837241 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f19d000f040 con 0x5579c14600a0 2024-07-26T06:49:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.804+0000 7f19ebde1740 10 client.15090 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:49:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.804+0000 7f19ebde1740 10 client.15090 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T06:49:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.804+0000 7f19ebde1740 20 client.15090 populate_metadata read hostname 'smithi042' 2024-07-26T06:49:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.804+0000 7f19ebde1740 1 --2- 172.21.15.42:0/2133837241 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x5579c14478e0 0x5579c184c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.804+0000 7f19ebde1740 4 client.15090 mds_command: new command op to 24583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:49:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.804+0000 7f19ebde1740 1 -- 172.21.15.42:0/2133837241 --> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579c17a6890 con 0x5579c14478e0 2024-07-26T06:49:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.804+0000 7f19d67fc640 1 --2- 172.21.15.42:0/2133837241 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x5579c14478e0 0x5579c184c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.805+0000 7f19d67fc640 1 --2- 172.21.15.42:0/2133837241 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x5579c14478e0 0x5579c184c640 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ed20475c0fdbb51 server_cookie=6bcea83569d1b6cb in_seq=0 out_seq=0 2024-07-26T06:49:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.805+0000 7f19beffd640 10 client.15090 ms_handle_connect on v2:172.21.15.164:6836/3642457979 2024-07-26T06:49:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.806+0000 7f19beffd640 1 -- 172.21.15.42:0/2133837241 <== mds.3 v2:172.21.15.164:6836/3642457979 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5579c17a6890 con 0x5579c14478e0 2024-07-26T06:49:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.806+0000 7f19beffd640 10 client.15090 handle_command_reply: tid=0 2024-07-26T06:49:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.806+0000 7f19ebde1740 1 client.15090 shutdown 2024-07-26T06:49:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.806+0000 7f19ebde1740 1 -- 172.21.15.42:0/2133837241 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c14600a0 msgr2=0x5579c1839ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.806+0000 7f19ebde1740 1 --2- 172.21.15.42:0/2133837241 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c14600a0 0x5579c1839ee0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f19d0004630 tx=0x7f19d0004730 comp rx=0 tx=0).stop 2024-07-26T06:49:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.806+0000 7f19ebde1740 1 -- 172.21.15.42:0/2133837241 shutdown_connections 2024-07-26T06:49:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.806+0000 7f19ebde1740 1 --2- 172.21.15.42:0/2133837241 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x5579c14478e0 0x5579c184c640 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.806+0000 7f19ebde1740 1 --2- 172.21.15.42:0/2133837241 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c1488f60 0x5579c1845e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.806+0000 7f19ebde1740 1 --2- 172.21.15.42:0/2133837241 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1468280 0x5579c183a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.806+0000 7f19ebde1740 1 --2- 172.21.15.42:0/2133837241 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c14600a0 0x5579c1839ee0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.806+0000 7f19ebde1740 1 -- 172.21.15.42:0/2133837241 >> 172.21.15.42:0/2133837241 conn(0x5579c147d350 msgr2=0x5579c1831530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:28.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.806+0000 7f19beffd640 0 client.15090 ms_handle_reset on v2:172.21.15.164:6836/3642457979 2024-07-26T06:49:28.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.806+0000 7f19ebde1740 1 -- 172.21.15.42:0/2133837241 shutdown_connections 2024-07-26T06:49:28.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.806+0000 7f19ebde1740 1 -- 172.21.15.42:0/2133837241 wait complete. 2024-07-26T06:49:28.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.806+0000 7f19ebde1740 20 client.15090 trim_cache size 0 max 16384 2024-07-26T06:49:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.824+0000 7f19ebde1740 1 Processor -- start 2024-07-26T06:49:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.824+0000 7f19ebde1740 1 -- start start 2024-07-26T06:49:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.824+0000 7f19ebde1740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c14600a0 0x5579c182c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.824+0000 7f19ebde1740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c1468280 0x5579c18344c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.824+0000 7f19ebde1740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1488f60 0x5579c1854d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.824+0000 7f19ebde1740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5579c184a9f0 con 0x5579c14600a0 2024-07-26T06:49:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.824+0000 7f19ebde1740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5579c184b050 con 0x5579c1488f60 2024-07-26T06:49:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.824+0000 7f19ebde1740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5579c184b720 con 0x5579c1468280 2024-07-26T06:49:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.825+0000 7f19deffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c14600a0 0x5579c182c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.825+0000 7f19deffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c14600a0 0x5579c182c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51358/0 (socket says 172.21.15.42:51358) 2024-07-26T06:49:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.825+0000 7f19deffd640 1 -- 172.21.15.42:0/1044439682 learned_addr learned my addr 172.21.15.42:0/1044439682 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.825+0000 7f19d67fc640 1 --2- 172.21.15.42:0/1044439682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c1468280 0x5579c18344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.825+0000 7f19df7fe640 1 --2- 172.21.15.42:0/1044439682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1488f60 0x5579c1854d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.825+0000 7f19deffd640 1 -- 172.21.15.42:0/1044439682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c1468280 msgr2=0x5579c18344c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.825+0000 7f19deffd640 1 --2- 172.21.15.42:0/1044439682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c1468280 0x5579c18344c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.825+0000 7f19deffd640 1 -- 172.21.15.42:0/1044439682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1488f60 msgr2=0x5579c1854d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.825+0000 7f19deffd640 1 --2- 172.21.15.42:0/1044439682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1488f60 0x5579c1854d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.825+0000 7f19deffd640 1 -- 172.21.15.42:0/1044439682 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579c182cb80 con 0x5579c14600a0 2024-07-26T06:49:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.825+0000 7f19df7fe640 1 --2- 172.21.15.42:0/1044439682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1488f60 0x5579c1854d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:49:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.825+0000 7f19deffd640 1 --2- 172.21.15.42:0/1044439682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c14600a0 0x5579c182c300 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f19d000bf80 tx=0x7f19d00276c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80325426cfa1698 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.825+0000 7f19d57fa640 1 -- 172.21.15.42:0/1044439682 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0037030 con 0x5579c14600a0 2024-07-26T06:49:28.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.825+0000 7f19d57fa640 1 -- 172.21.15.42:0/1044439682 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d0027dd0 con 0x5579c14600a0 2024-07-26T06:49:28.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.825+0000 7f19d57fa640 1 -- 172.21.15.42:0/1044439682 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d0023050 con 0x5579c14600a0 2024-07-26T06:49:28.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.826+0000 7f19ebde1740 1 -- 172.21.15.42:0/1044439682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c14600a0 msgr2=0x5579c182c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:28.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.826+0000 7f19ebde1740 1 --2- 172.21.15.42:0/1044439682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c14600a0 0x5579c182c300 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f19d000bf80 tx=0x7f19d00276c0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.826+0000 7f19ebde1740 1 -- 172.21.15.42:0/1044439682 shutdown_connections 2024-07-26T06:49:28.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.826+0000 7f19ebde1740 1 --2- 172.21.15.42:0/1044439682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1488f60 0x5579c1854d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.826+0000 7f19ebde1740 1 --2- 172.21.15.42:0/1044439682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c1468280 0x5579c18344c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.826+0000 7f19ebde1740 1 --2- 172.21.15.42:0/1044439682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c14600a0 0x5579c182c300 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.826+0000 7f19ebde1740 1 -- 172.21.15.42:0/1044439682 >> 172.21.15.42:0/1044439682 conn(0x5579c147d350 msgr2=0x5579c182bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:28.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.826+0000 7f19ebde1740 1 -- 172.21.15.42:0/1044439682 shutdown_connections 2024-07-26T06:49:28.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.826+0000 7f19ebde1740 1 -- 172.21.15.42:0/1044439682 wait complete. 2024-07-26T06:49:28.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.826+0000 7f19ebde1740 1 -- start start 2024-07-26T06:49:28.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.827+0000 7f19ebde1740 1 Processor -- start 2024-07-26T06:49:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.827+0000 7f19ebde1740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c14600a0 0x5579c184f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.827+0000 7f19ebde1740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1468280 0x5579c1845980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.827+0000 7f19ebde1740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c1488f60 0x5579c1845f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.827+0000 7f19ebde1740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5579c184c4c0 con 0x5579c1488f60 2024-07-26T06:49:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.827+0000 7f19ebde1740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5579c184bdf0 con 0x5579c1468280 2024-07-26T06:49:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.827+0000 7f19ebde1740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5579c184cb90 con 0x5579c14600a0 2024-07-26T06:49:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.827+0000 7f19ebde1740 10 client.0 _ll_register_callbacks cb 0x7f19d000beb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:49:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.827+0000 7f19df7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c1488f60 0x5579c1845f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.827+0000 7f19df7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c1488f60 0x5579c1845f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51372/0 (socket says 172.21.15.42:51372) 2024-07-26T06:49:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.827+0000 7f19deffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c14600a0 0x5579c184f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.827+0000 7f19df7fe640 1 -- 172.21.15.42:0/1305710554 learned_addr learned my addr 172.21.15.42:0/1305710554 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.827+0000 7f19d67fc640 1 --2- 172.21.15.42:0/1305710554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1468280 0x5579c1845980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.828+0000 7f19df7fe640 1 -- 172.21.15.42:0/1305710554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c14600a0 msgr2=0x5579c184f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.828+0000 7f19df7fe640 1 --2- 172.21.15.42:0/1305710554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c14600a0 0x5579c184f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.828+0000 7f19df7fe640 1 -- 172.21.15.42:0/1305710554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1468280 msgr2=0x5579c1845980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.828+0000 7f19df7fe640 1 --2- 172.21.15.42:0/1305710554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1468280 0x5579c1845980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.828+0000 7f19df7fe640 1 -- 172.21.15.42:0/1305710554 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579c18466e0 con 0x5579c1488f60 2024-07-26T06:49:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.828+0000 7f19df7fe640 1 --2- 172.21.15.42:0/1305710554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c1488f60 0x5579c1845f80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f19d80e1940 tx=0x7f19d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a23d9a30ebb81720 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.828+0000 7f19beffd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:49:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.828+0000 7f19beffd640 1 -- 172.21.15.42:0/1305710554 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d80af020 con 0x5579c1488f60 2024-07-26T06:49:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.828+0000 7f19beffd640 1 -- 172.21.15.42:0/1305710554 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f19d80aa1b0 con 0x5579c1488f60 2024-07-26T06:49:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.828+0000 7f19ebde1740 1 -- 172.21.15.42:0/1305710554 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19d0027dd0 con 0x5579c1488f60 2024-07-26T06:49:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.828+0000 7f19beffd640 1 -- 172.21.15.42:0/1305710554 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f19d801f040 con 0x5579c1488f60 2024-07-26T06:49:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.828+0000 7f19ebde1740 10 client.15096 fetch_fsmap learned FSMap version 28 2024-07-26T06:49:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.828+0000 7f19ebde1740 1 -- 172.21.15.42:0/1305710554 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5579c18b98b0 con 0x5579c1488f60 2024-07-26T06:49:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.829+0000 7f19beffd640 1 -- 172.21.15.42:0/1305710554 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19d0027dd0 con 0x5579c1488f60 2024-07-26T06:49:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.829+0000 7f19beffd640 1 -- 172.21.15.42:0/1305710554 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f19d800f040 con 0x5579c1488f60 2024-07-26T06:49:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.829+0000 7f19ebde1740 10 client.15096 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:49:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.829+0000 7f19ebde1740 10 client.15096 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T06:49:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.829+0000 7f19ebde1740 20 client.15096 populate_metadata read hostname 'smithi042' 2024-07-26T06:49:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.829+0000 7f19ebde1740 1 --2- 172.21.15.42:0/1305710554 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x5579c141cc90 0x5579c18bd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.829+0000 7f19ebde1740 4 client.15096 mds_command: new command op to 24583 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:49:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.829+0000 7f19ebde1740 1 -- 172.21.15.42:0/1305710554 --> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] -- command(tid 0: {"prefix": "status"}) -- 0x5579c17a2330 con 0x5579c141cc90 2024-07-26T06:49:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.829+0000 7f19deffd640 1 --2- 172.21.15.42:0/1305710554 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x5579c141cc90 0x5579c18bd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.830+0000 7f19beffd640 10 client.15096 ms_handle_connect on v2:172.21.15.164:6836/3642457979 2024-07-26T06:49:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.830+0000 7f19deffd640 1 --2- 172.21.15.42:0/1305710554 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x5579c141cc90 0x5579c18bd1b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2583485505f71159 server_cookie=c5591d7325d7b729 in_seq=0 out_seq=0 2024-07-26T06:49:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.830+0000 7f19beffd640 1 -- 172.21.15.42:0/1305710554 <== mds.3 v2:172.21.15.164:6836/3642457979 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x5579c17a2330 con 0x5579c141cc90 2024-07-26T06:49:28.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.830+0000 7f19beffd640 10 client.15096 handle_command_reply: tid=0 2024-07-26T06:49:28.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.830+0000 7f19ebde1740 1 client.15096 shutdown 2024-07-26T06:49:28.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.831+0000 7f19ebde1740 1 -- 172.21.15.42:0/1305710554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c1488f60 msgr2=0x5579c1845f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:28.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.831+0000 7f19ebde1740 1 --2- 172.21.15.42:0/1305710554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c1488f60 0x5579c1845f80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f19d80e1940 tx=0x7f19d80057f0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.831+0000 7f19ebde1740 1 -- 172.21.15.42:0/1305710554 shutdown_connections 2024-07-26T06:49:28.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.831+0000 7f19ebde1740 1 --2- 172.21.15.42:0/1305710554 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x5579c141cc90 0x5579c18bd1b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.831+0000 7f19ebde1740 1 --2- 172.21.15.42:0/1305710554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5579c1488f60 0x5579c1845f80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.831+0000 7f19beffd640 0 client.15096 ms_handle_reset on v2:172.21.15.164:6836/3642457979 2024-07-26T06:49:28.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.831+0000 7f19ebde1740 1 --2- 172.21.15.42:0/1305710554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5579c1468280 0x5579c1845980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.831+0000 7f19ebde1740 1 --2- 172.21.15.42:0/1305710554 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5579c14600a0 0x5579c184f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.831+0000 7f19ebde1740 1 -- 172.21.15.42:0/1305710554 >> 172.21.15.42:0/1305710554 conn(0x5579c147d350 msgr2=0x5579c182a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:28.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.831+0000 7f19ebde1740 1 -- 172.21.15.42:0/1305710554 shutdown_connections 2024-07-26T06:49:28.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.831+0000 7f19ebde1740 1 -- 172.21.15.42:0/1305710554 wait complete. 2024-07-26T06:49:28.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.831+0000 7f19ebde1740 20 client.15096 trim_cache size 0 max 16384 2024-07-26T06:49:28.837 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:49:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_fsid": "8b3e2750-4b19-11ef-bcb6-c7b262605968", 2024-07-26T06:49:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: "whoami": 3, 2024-07-26T06:49:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: "id": 24583, 2024-07-26T06:49:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: "want_state": "up:active", 2024-07-26T06:49:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "up:active", 2024-07-26T06:49:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: "fs_name": "cephfs", 2024-07-26T06:49:28.837 INFO:teuthology.orchestra.run.smithi042.stdout: "rank_uptime": 127.74803442, 2024-07-26T06:49:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: "mdsmap_epoch": 28, 2024-07-26T06:49:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch": 80, 2024-07-26T06:49:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch_barrier": 80, 2024-07-26T06:49:28.838 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 173.09610734699999 2024-07-26T06:49:28.838 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:49:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.832+0000 7f19e585d640 1 -- 172.21.15.42:0/4217223572 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f19cc0b3c50 msgr2=0x7f19cc0b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.832+0000 7f19e585d640 1 --2- 172.21.15.42:0/4217223572 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f19cc0b3c50 0x7f19cc0b6160 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f19d0005fd0 tx=0x7f19d000b2c0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.832+0000 7f19e585d640 1 -- 172.21.15.42:0/4217223572 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19e0109820 msgr2=0x7f19e019f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.832+0000 7f19e585d640 1 --2- 172.21.15.42:0/4217223572 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19e0109820 0x7f19e019f720 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f19d800eaa0 tx=0x7f19d800ef60 comp rx=0 tx=0).stop 2024-07-26T06:49:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.832+0000 7f19e585d640 1 -- 172.21.15.42:0/4217223572 shutdown_connections 2024-07-26T06:49:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.832+0000 7f19e585d640 1 --2- 172.21.15.42:0/4217223572 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f19cc0b3c50 0x7f19cc0b6160 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.832+0000 7f19e585d640 1 --2- 172.21.15.42:0/4217223572 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f19e0109820 0x7f19e019f720 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.832+0000 7f19e585d640 1 --2- 172.21.15.42:0/4217223572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19e0108ed0 0x7f19e019a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.832+0000 7f19e585d640 1 --2- 172.21.15.42:0/4217223572 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f19e0103590 0x7f19e0199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.832+0000 7f19e585d640 1 -- 172.21.15.42:0/4217223572 >> 172.21.15.42:0/4217223572 conn(0x7f19e007b280 msgr2=0x7f19e0106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.832+0000 7f19e585d640 1 -- 172.21.15.42:0/4217223572 shutdown_connections 2024-07-26T06:49:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:28.832+0000 7f19e585d640 1 -- 172.21.15.42:0/4217223572 wait complete. 2024-07-26T06:49:29.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:28 smithi042 ceph-mon[32775]: pgmap v373: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:29.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:28 smithi164 ceph-mon[36970]: pgmap v373: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:29.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:28 smithi163 ceph-mon[36873]: pgmap v373: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:29.544 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell mds.h status 2024-07-26T06:49:29.813 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:31.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:30 smithi042 ceph-mon[32775]: pgmap v374: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:49:31.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:30 smithi163 ceph-mon[36873]: pgmap v374: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:49:31.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.420+0000 7f08eb3a3640 1 -- 172.21.15.42:0/3233897750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08e4101cc0 msgr2=0x7f08e41020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:31.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.420+0000 7f08eb3a3640 1 --2- 172.21.15.42:0/3233897750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08e4101cc0 0x7f08e41020d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009900 tx=0x7f08e002a410 comp rx=0 tx=0).stop 2024-07-26T06:49:31.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.420+0000 7f08eb3a3640 1 -- 172.21.15.42:0/3233897750 shutdown_connections 2024-07-26T06:49:31.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.420+0000 7f08eb3a3640 1 --2- 172.21.15.42:0/3233897750 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08e4107f50 0x7f08e410e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.420+0000 7f08eb3a3640 1 --2- 172.21.15.42:0/3233897750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08e4107600 0x7f08e4107a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.420+0000 7f08eb3a3640 1 --2- 172.21.15.42:0/3233897750 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08e4101cc0 0x7f08e41020d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.420+0000 7f08eb3a3640 1 -- 172.21.15.42:0/3233897750 >> 172.21.15.42:0/3233897750 conn(0x7f08e40798d0 msgr2=0x7f08e4079ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:31.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.420+0000 7f08eb3a3640 1 -- 172.21.15.42:0/3233897750 shutdown_connections 2024-07-26T06:49:31.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.420+0000 7f08eb3a3640 1 -- 172.21.15.42:0/3233897750 wait complete. 2024-07-26T06:49:31.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.421+0000 7f08eb3a3640 1 Processor -- start 2024-07-26T06:49:31.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.421+0000 7f08eb3a3640 1 -- start start 2024-07-26T06:49:31.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.421+0000 7f08eb3a3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08e4101cc0 0x7f08e4199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.421+0000 7f08eb3a3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08e4107600 0x7f08e419a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.421+0000 7f08eb3a3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08e4107f50 0x7f08e419f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.421+0000 7f08eb3a3640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f08e41135e0 con 0x7f08e4107600 2024-07-26T06:49:31.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.421+0000 7f08eb3a3640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f08e4113460 con 0x7f08e4107f50 2024-07-26T06:49:31.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.421+0000 7f08eb3a3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f08e4113760 con 0x7f08e4101cc0 2024-07-26T06:49:31.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.421+0000 7f08e893e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08e4107600 0x7f08e419a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.421+0000 7f08e893e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08e4107600 0x7f08e419a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51380/0 (socket says 172.21.15.42:51380) 2024-07-26T06:49:31.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.421+0000 7f08e893e640 1 -- 172.21.15.42:0/2721461250 learned_addr learned my addr 172.21.15.42:0/2721461250 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:31.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.421+0000 7f08e913f640 1 --2- 172.21.15.42:0/2721461250 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08e4101cc0 0x7f08e4199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.422+0000 7f08e9940640 1 --2- 172.21.15.42:0/2721461250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08e4107f50 0x7f08e419f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.422+0000 7f08e893e640 1 -- 172.21.15.42:0/2721461250 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08e4101cc0 msgr2=0x7f08e4199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:31.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.422+0000 7f08e893e640 1 --2- 172.21.15.42:0/2721461250 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08e4101cc0 0x7f08e4199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.422+0000 7f08e893e640 1 -- 172.21.15.42:0/2721461250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08e4107f50 msgr2=0x7f08e419f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:31.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.422+0000 7f08e893e640 1 --2- 172.21.15.42:0/2721461250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08e4107f50 0x7f08e419f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.422+0000 7f08e893e640 1 -- 172.21.15.42:0/2721461250 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e419ff40 con 0x7f08e4107600 2024-07-26T06:49:31.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.422+0000 7f08e893e640 1 --2- 172.21.15.42:0/2721461250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08e4107600 0x7f08e419a570 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f08d4006fd0 tx=0x7f08d400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:31.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.422+0000 7f08da7fc640 1 -- 172.21.15.42:0/2721461250 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d4011830 con 0x7f08e4107600 2024-07-26T06:49:31.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.422+0000 7f08da7fc640 1 -- 172.21.15.42:0/2721461250 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08d4016070 con 0x7f08e4107600 2024-07-26T06:49:31.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.422+0000 7f08da7fc640 1 -- 172.21.15.42:0/2721461250 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d4010e00 con 0x7f08e4107600 2024-07-26T06:49:31.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.422+0000 7f08eb3a3640 1 -- 172.21.15.42:0/2721461250 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08e41a0230 con 0x7f08e4107600 2024-07-26T06:49:31.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.422+0000 7f08eb3a3640 1 -- 172.21.15.42:0/2721461250 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08e41a6990 con 0x7f08e4107600 2024-07-26T06:49:31.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.425+0000 7f08f1927740 1 Processor -- start 2024-07-26T06:49:31.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.425+0000 7f08f1927740 1 -- start start 2024-07-26T06:49:31.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.425+0000 7f08da7fc640 1 -- 172.21.15.42:0/2721461250 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f08d401b460 con 0x7f08e4107600 2024-07-26T06:49:31.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08f1927740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b542489f60 0x55b54282f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08f1927740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b5424610a0 0x55b54282f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08f1927740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b542469280 0x55b54282fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08f1927740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55b5427a5940 con 0x55b542489f60 2024-07-26T06:49:31.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08f1927740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b542794e40 con 0x55b542469280 2024-07-26T06:49:31.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08f1927740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55b542798e80 con 0x55b5424610a0 2024-07-26T06:49:31.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08e9940640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b542489f60 0x55b54282f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08e9940640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b542489f60 0x55b54282f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51392/0 (socket says 172.21.15.42:51392) 2024-07-26T06:49:31.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08e9940640 1 -- 172.21.15.42:0/3794303559 learned_addr learned my addr 172.21.15.42:0/3794303559 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:31.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08e913f640 1 --2- 172.21.15.42:0/3794303559 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b5424610a0 0x55b54282f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08e893e640 1 --2- 172.21.15.42:0/3794303559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b542469280 0x55b54282fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08e9940640 1 -- 172.21.15.42:0/3794303559 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b5424610a0 msgr2=0x55b54282f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:31.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08e9940640 1 --2- 172.21.15.42:0/3794303559 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b5424610a0 0x55b54282f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08e9940640 1 -- 172.21.15.42:0/3794303559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b542469280 msgr2=0x55b54282fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:31.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08e9940640 1 --2- 172.21.15.42:0/3794303559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b542469280 0x55b54282fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08e9940640 1 -- 172.21.15.42:0/3794303559 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b54273ba60 con 0x55b542489f60 2024-07-26T06:49:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08da7fc640 1 --2- 172.21.15.42:0/2721461250 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f08d00b3bd0 0x7f08d00b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.426+0000 7f08e9940640 1 --2- 172.21.15.42:0/3794303559 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b542489f60 0x55b54282f3a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f08dc009ee0 tx=0x7f08dc010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a49b0f62edd6280 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.427+0000 7f08e913f640 1 --2- 172.21.15.42:0/2721461250 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f08d00b3bd0 0x7f08d00b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.427+0000 7f08da7fc640 1 -- 172.21.15.42:0/2721461250 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f08d40e32b0 con 0x7f08e4107600 2024-07-26T06:49:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.427+0000 7f08cf7fe640 1 -- 172.21.15.42:0/3794303559 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc016070 con 0x55b542489f60 2024-07-26T06:49:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.427+0000 7f08cf7fe640 1 -- 172.21.15.42:0/3794303559 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08dc004d20 con 0x55b542489f60 2024-07-26T06:49:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.427+0000 7f08cf7fe640 1 -- 172.21.15.42:0/3794303559 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc0126b0 con 0x55b542489f60 2024-07-26T06:49:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.428+0000 7f08f1927740 1 -- 172.21.15.42:0/3794303559 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b542489f60 msgr2=0x55b54282f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.428+0000 7f08e913f640 1 --2- 172.21.15.42:0/2721461250 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f08d00b3bd0 0x7f08d00b60e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f08e0031440 tx=0x7f08e0032b60 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.428+0000 7f08f1927740 1 --2- 172.21.15.42:0/3794303559 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b542489f60 0x55b54282f3a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f08dc009ee0 tx=0x7f08dc010740 comp rx=0 tx=0).stop 2024-07-26T06:49:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.428+0000 7f08f1927740 1 -- 172.21.15.42:0/3794303559 shutdown_connections 2024-07-26T06:49:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.428+0000 7f08f1927740 1 --2- 172.21.15.42:0/3794303559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b542469280 0x55b54282fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.428+0000 7f08f1927740 1 --2- 172.21.15.42:0/3794303559 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b5424610a0 0x55b54282f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.428+0000 7f08f1927740 1 --2- 172.21.15.42:0/3794303559 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b542489f60 0x55b54282f3a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.428+0000 7f08f1927740 1 -- 172.21.15.42:0/3794303559 >> 172.21.15.42:0/3794303559 conn(0x55b54247e350 msgr2=0x55b54282d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.428+0000 7f08f1927740 1 -- 172.21.15.42:0/3794303559 shutdown_connections 2024-07-26T06:49:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.428+0000 7f08f1927740 1 -- 172.21.15.42:0/3794303559 wait complete. 2024-07-26T06:49:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.428+0000 7f08f1927740 1 -- start start 2024-07-26T06:49:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.428+0000 7f08f1927740 1 Processor -- start 2024-07-26T06:49:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.429+0000 7f08f1927740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b5424610a0 0x55b54283afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.429+0000 7f08f1927740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b542469280 0x55b54283b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.429+0000 7f08f1927740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b542489f60 0x55b542846e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.429+0000 7f08f1927740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55b5425f85f0 con 0x55b5424610a0 2024-07-26T06:49:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.429+0000 7f08f1927740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b5427378b0 con 0x55b542489f60 2024-07-26T06:49:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.429+0000 7f08f1927740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55b54283db80 con 0x55b542469280 2024-07-26T06:49:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.429+0000 7f08e913f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b5424610a0 0x55b54283afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.429+0000 7f08f1927740 10 client.0 _ll_register_callbacks cb 0x55b5427417f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:49:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.429+0000 7f08e913f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b5424610a0 0x55b54283afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51400/0 (socket says 172.21.15.42:51400) 2024-07-26T06:49:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.429+0000 7f08e913f640 1 -- 172.21.15.42:0/4159977118 learned_addr learned my addr 172.21.15.42:0/4159977118 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.429+0000 7f08e893e640 1 --2- 172.21.15.42:0/4159977118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b542469280 0x55b54283b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.429+0000 7f08e9940640 1 --2- 172.21.15.42:0/4159977118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b542489f60 0x55b542846e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.429+0000 7f08e913f640 1 -- 172.21.15.42:0/4159977118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b542469280 msgr2=0x55b54283b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.429+0000 7f08e913f640 1 --2- 172.21.15.42:0/4159977118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b542469280 0x55b54283b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.429+0000 7f08e913f640 1 -- 172.21.15.42:0/4159977118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b542489f60 msgr2=0x55b542846e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.429+0000 7f08e913f640 1 --2- 172.21.15.42:0/4159977118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b542489f60 0x55b542846e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.429+0000 7f08e913f640 1 -- 172.21.15.42:0/4159977118 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5424f5540 con 0x55b5424610a0 2024-07-26T06:49:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.430+0000 7f08e913f640 1 --2- 172.21.15.42:0/4159977118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b5424610a0 0x55b54283afe0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009900 tx=0x7f08e0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ece81cdd41eb345 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.430+0000 7f08ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:49:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.430+0000 7f08ccff9640 1 -- 172.21.15.42:0/4159977118 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e004d070 con 0x55b5424610a0 2024-07-26T06:49:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.430+0000 7f08ccff9640 1 -- 172.21.15.42:0/4159977118 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f08e0005030 con 0x55b5424610a0 2024-07-26T06:49:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.430+0000 7f08f1927740 1 -- 172.21.15.42:0/4159977118 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08dc004d20 con 0x55b5424610a0 2024-07-26T06:49:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.430+0000 7f08ccff9640 1 -- 172.21.15.42:0/4159977118 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f08e0059050 con 0x55b5424610a0 2024-07-26T06:49:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.430+0000 7f08f1927740 10 client.15108 fetch_fsmap learned FSMap version 28 2024-07-26T06:49:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.430+0000 7f08f1927740 1 -- 172.21.15.42:0/4159977118 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b542849c20 con 0x55b5424610a0 2024-07-26T06:49:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.430+0000 7f08ccff9640 1 -- 172.21.15.42:0/4159977118 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08dc004d20 con 0x55b5424610a0 2024-07-26T06:49:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.430+0000 7f08ccff9640 1 -- 172.21.15.42:0/4159977118 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f08e00035f0 con 0x55b5424610a0 2024-07-26T06:49:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.430+0000 7f08f1927740 10 client.15108 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:49:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.430+0000 7f08f1927740 10 client.15108 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T06:49:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.431+0000 7f08f1927740 20 client.15108 populate_metadata read hostname 'smithi042' 2024-07-26T06:49:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.431+0000 7f08f1927740 1 --2- 172.21.15.42:0/4159977118 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55b5424488e0 0x55b54284d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.431+0000 7f08f1927740 4 client.15108 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:49:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.431+0000 7f08f1927740 1 -- 172.21.15.42:0/4159977118 --> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5427a7890 con 0x55b5424488e0 2024-07-26T06:49:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.431+0000 7f08e893e640 1 --2- 172.21.15.42:0/4159977118 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55b5424488e0 0x55b54284d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.431+0000 7f08ccff9640 10 client.15108 ms_handle_connect on v2:172.21.15.163:6836/154041066 2024-07-26T06:49:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.431+0000 7f08e893e640 1 --2- 172.21.15.42:0/4159977118 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55b5424488e0 0x55b54284d6e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ae4b46f9e717323d server_cookie=3eddd2fd2f177990 in_seq=0 out_seq=0 2024-07-26T06:49:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.432+0000 7f08ccff9640 1 -- 172.21.15.42:0/4159977118 <== mds.2 v2:172.21.15.163:6836/154041066 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b5427a7890 con 0x55b5424488e0 2024-07-26T06:49:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.432+0000 7f08ccff9640 10 client.15108 handle_command_reply: tid=0 2024-07-26T06:49:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.432+0000 7f08f1927740 1 client.15108 shutdown 2024-07-26T06:49:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.432+0000 7f08f1927740 1 -- 172.21.15.42:0/4159977118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b5424610a0 msgr2=0x55b54283afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.432+0000 7f08f1927740 1 --2- 172.21.15.42:0/4159977118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b5424610a0 0x55b54283afe0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009900 tx=0x7f08e0009260 comp rx=0 tx=0).stop 2024-07-26T06:49:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.432+0000 7f08f1927740 1 -- 172.21.15.42:0/4159977118 shutdown_connections 2024-07-26T06:49:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.432+0000 7f08f1927740 1 --2- 172.21.15.42:0/4159977118 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55b5424488e0 0x55b54284d6e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.432+0000 7f08f1927740 1 --2- 172.21.15.42:0/4159977118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b542489f60 0x55b542846e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.432+0000 7f08f1927740 1 --2- 172.21.15.42:0/4159977118 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b542469280 0x55b54283b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.432+0000 7f08f1927740 1 --2- 172.21.15.42:0/4159977118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b5424610a0 0x55b54283afe0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.432+0000 7f08f1927740 1 -- 172.21.15.42:0/4159977118 >> 172.21.15.42:0/4159977118 conn(0x55b54247e350 msgr2=0x55b542832580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.433+0000 7f08ccff9640 0 client.15108 ms_handle_reset on v2:172.21.15.163:6836/154041066 2024-07-26T06:49:31.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.433+0000 7f08f1927740 1 -- 172.21.15.42:0/4159977118 shutdown_connections 2024-07-26T06:49:31.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.433+0000 7f08f1927740 1 -- 172.21.15.42:0/4159977118 wait complete. 2024-07-26T06:49:31.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.433+0000 7f08f1927740 20 client.15108 trim_cache size 0 max 16384 2024-07-26T06:49:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.451+0000 7f08f1927740 1 Processor -- start 2024-07-26T06:49:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.451+0000 7f08f1927740 1 -- start start 2024-07-26T06:49:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.451+0000 7f08f1927740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5424610a0 0x55b542846de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.451+0000 7f08f1927740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b542469280 0x55b54283a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.451+0000 7f08f1927740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b542489f60 0x55b542834620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.451+0000 7f08f1927740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55b54284ba20 con 0x55b542489f60 2024-07-26T06:49:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.451+0000 7f08f1927740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b54284c0f0 con 0x55b5424610a0 2024-07-26T06:49:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.451+0000 7f08f1927740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55b54284c7c0 con 0x55b542469280 2024-07-26T06:49:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.451+0000 7f08e9940640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b542489f60 0x55b542834620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.451+0000 7f08e9940640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b542489f60 0x55b542834620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51416/0 (socket says 172.21.15.42:51416) 2024-07-26T06:49:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.451+0000 7f08e9940640 1 -- 172.21.15.42:0/3943252893 learned_addr learned my addr 172.21.15.42:0/3943252893 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.451+0000 7f08e913f640 1 --2- 172.21.15.42:0/3943252893 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5424610a0 0x55b542846de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.452+0000 7f08e893e640 1 --2- 172.21.15.42:0/3943252893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b542469280 0x55b54283a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.452+0000 7f08e9940640 1 -- 172.21.15.42:0/3943252893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b542469280 msgr2=0x55b54283a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.452+0000 7f08e9940640 1 --2- 172.21.15.42:0/3943252893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b542469280 0x55b54283a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.452+0000 7f08e9940640 1 -- 172.21.15.42:0/3943252893 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5424610a0 msgr2=0x55b542846de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.452+0000 7f08e9940640 1 --2- 172.21.15.42:0/3943252893 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5424610a0 0x55b542846de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.452+0000 7f08e9940640 1 -- 172.21.15.42:0/3943252893 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b54283c0a0 con 0x55b542489f60 2024-07-26T06:49:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.452+0000 7f08e893e640 1 --2- 172.21.15.42:0/3943252893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b542469280 0x55b54283a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:49:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.452+0000 7f08e913f640 1 --2- 172.21.15.42:0/3943252893 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5424610a0 0x55b542846de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:49:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.452+0000 7f08e9940640 1 --2- 172.21.15.42:0/3943252893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b542489f60 0x55b542834620 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f08dc006010 tx=0x7f08dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ced76564058bb7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.452+0000 7f08ceffd640 1 -- 172.21.15.42:0/3943252893 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc016050 con 0x55b542489f60 2024-07-26T06:49:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.452+0000 7f08ceffd640 1 -- 172.21.15.42:0/3943252893 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08dc012b00 con 0x55b542489f60 2024-07-26T06:49:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.452+0000 7f08ceffd640 1 -- 172.21.15.42:0/3943252893 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc01aa60 con 0x55b542489f60 2024-07-26T06:49:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.453+0000 7f08f1927740 1 -- 172.21.15.42:0/3943252893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b542489f60 msgr2=0x55b542834620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.453+0000 7f08f1927740 1 --2- 172.21.15.42:0/3943252893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b542489f60 0x55b542834620 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f08dc006010 tx=0x7f08dc009bb0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.453+0000 7f08f1927740 1 -- 172.21.15.42:0/3943252893 shutdown_connections 2024-07-26T06:49:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.453+0000 7f08f1927740 1 --2- 172.21.15.42:0/3943252893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b542489f60 0x55b542834620 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.453+0000 7f08f1927740 1 --2- 172.21.15.42:0/3943252893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b542469280 0x55b54283a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.453+0000 7f08f1927740 1 --2- 172.21.15.42:0/3943252893 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5424610a0 0x55b542846de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.453+0000 7f08f1927740 1 -- 172.21.15.42:0/3943252893 >> 172.21.15.42:0/3943252893 conn(0x55b54247e350 msgr2=0x55b542835f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.454+0000 7f08f1927740 1 -- 172.21.15.42:0/3943252893 shutdown_connections 2024-07-26T06:49:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.454+0000 7f08f1927740 1 -- 172.21.15.42:0/3943252893 wait complete. 2024-07-26T06:49:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.454+0000 7f08f1927740 1 -- start start 2024-07-26T06:49:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.454+0000 7f08f1927740 1 Processor -- start 2024-07-26T06:49:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08f1927740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b5424610a0 0x55b542833d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08f1927740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b542469280 0x55b542834260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08f1927740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b542489f60 0x55b5428347a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08f1927740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55b54284d560 con 0x55b5424610a0 2024-07-26T06:49:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08f1927740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b54284ce90 con 0x55b542489f60 2024-07-26T06:49:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08f1927740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55b54284dc30 con 0x55b542469280 2024-07-26T06:49:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08f1927740 10 client.0 _ll_register_callbacks cb 0x7f08e0031620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:49:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08e913f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b5424610a0 0x55b542833d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08e913f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b5424610a0 0x55b542833d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51422/0 (socket says 172.21.15.42:51422) 2024-07-26T06:49:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08e913f640 1 -- 172.21.15.42:0/1432198316 learned_addr learned my addr 172.21.15.42:0/1432198316 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08e893e640 1 --2- 172.21.15.42:0/1432198316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b542469280 0x55b542834260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08e9940640 1 --2- 172.21.15.42:0/1432198316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b542489f60 0x55b5428347a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08e913f640 1 -- 172.21.15.42:0/1432198316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b542469280 msgr2=0x55b542834260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08e913f640 1 --2- 172.21.15.42:0/1432198316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b542469280 0x55b542834260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08e913f640 1 -- 172.21.15.42:0/1432198316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b542489f60 msgr2=0x55b5428347a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08e913f640 1 --2- 172.21.15.42:0/1432198316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b542489f60 0x55b5428347a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08e913f640 1 -- 172.21.15.42:0/1432198316 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b542854a30 con 0x55b5424610a0 2024-07-26T06:49:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08e9940640 1 --2- 172.21.15.42:0/1432198316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b542489f60 0x55b5428347a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:49:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08e893e640 1 --2- 172.21.15.42:0/1432198316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b542469280 0x55b542834260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:49:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08e913f640 1 --2- 172.21.15.42:0/1432198316 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b5424610a0 0x55b542833d20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f08e0003bb0 tx=0x7f08e0003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b07339d277e83088 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.455+0000 7f08ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:49:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.456+0000 7f08ccff9640 1 -- 172.21.15.42:0/1432198316 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e0061020 con 0x55b5424610a0 2024-07-26T06:49:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.456+0000 7f08ccff9640 1 -- 172.21.15.42:0/1432198316 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f08e0005040 con 0x55b5424610a0 2024-07-26T06:49:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.456+0000 7f08f1927740 1 -- 172.21.15.42:0/1432198316 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08dc012b00 con 0x55b5424610a0 2024-07-26T06:49:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.456+0000 7f08ccff9640 1 -- 172.21.15.42:0/1432198316 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f08e006c050 con 0x55b5424610a0 2024-07-26T06:49:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.456+0000 7f08f1927740 10 client.15114 fetch_fsmap learned FSMap version 28 2024-07-26T06:49:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.456+0000 7f08f1927740 1 -- 172.21.15.42:0/1432198316 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b542855cb0 con 0x55b5424610a0 2024-07-26T06:49:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.456+0000 7f08ccff9640 1 -- 172.21.15.42:0/1432198316 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08dc012b00 con 0x55b5424610a0 2024-07-26T06:49:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.457+0000 7f08ccff9640 1 -- 172.21.15.42:0/1432198316 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f08e003d450 con 0x55b5424610a0 2024-07-26T06:49:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.457+0000 7f08f1927740 10 client.15114 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:49:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.457+0000 7f08f1927740 10 client.15114 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T06:49:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.457+0000 7f08f1927740 20 client.15114 populate_metadata read hostname 'smithi042' 2024-07-26T06:49:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.457+0000 7f08f1927740 1 --2- 172.21.15.42:0/1432198316 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55b54241dc90 0x55b5428c1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.457+0000 7f08f1927740 4 client.15114 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:49:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.457+0000 7f08f1927740 1 -- 172.21.15.42:0/1432198316 --> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] -- command(tid 0: {"prefix": "status"}) -- 0x55b5427a3330 con 0x55b54241dc90 2024-07-26T06:49:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.457+0000 7f08e893e640 1 --2- 172.21.15.42:0/1432198316 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55b54241dc90 0x55b5428c1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.457+0000 7f08ccff9640 10 client.15114 ms_handle_connect on v2:172.21.15.163:6836/154041066 2024-07-26T06:49:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.457+0000 7f08e893e640 1 --2- 172.21.15.42:0/1432198316 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55b54241dc90 0x55b5428c1f20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d4c66583074ecb9c server_cookie=738ec01b73945fe in_seq=0 out_seq=0 2024-07-26T06:49:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.458+0000 7f08ccff9640 1 -- 172.21.15.42:0/1432198316 <== mds.2 v2:172.21.15.163:6836/154041066 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b5427a3330 con 0x55b54241dc90 2024-07-26T06:49:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.458+0000 7f08ccff9640 10 client.15114 handle_command_reply: tid=0 2024-07-26T06:49:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.458+0000 7f08f1927740 1 client.15114 shutdown 2024-07-26T06:49:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.458+0000 7f08f1927740 1 -- 172.21.15.42:0/1432198316 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b5424610a0 msgr2=0x55b542833d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.458+0000 7f08f1927740 1 --2- 172.21.15.42:0/1432198316 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b5424610a0 0x55b542833d20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f08e0003bb0 tx=0x7f08e0003820 comp rx=0 tx=0).stop 2024-07-26T06:49:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.458+0000 7f08f1927740 1 -- 172.21.15.42:0/1432198316 shutdown_connections 2024-07-26T06:49:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.458+0000 7f08f1927740 1 --2- 172.21.15.42:0/1432198316 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55b54241dc90 0x55b5428c1f20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.458+0000 7f08f1927740 1 --2- 172.21.15.42:0/1432198316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b542489f60 0x55b5428347a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.458+0000 7f08ccff9640 0 client.15114 ms_handle_reset on v2:172.21.15.163:6836/154041066 2024-07-26T06:49:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.458+0000 7f08f1927740 1 --2- 172.21.15.42:0/1432198316 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b542469280 0x55b542834260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.458+0000 7f08f1927740 1 --2- 172.21.15.42:0/1432198316 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b5424610a0 0x55b542833d20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.458+0000 7f08f1927740 1 -- 172.21.15.42:0/1432198316 >> 172.21.15.42:0/1432198316 conn(0x55b54247e350 msgr2=0x55b5428324a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.458+0000 7f08f1927740 1 -- 172.21.15.42:0/1432198316 shutdown_connections 2024-07-26T06:49:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.458+0000 7f08f1927740 1 -- 172.21.15.42:0/1432198316 wait complete. 2024-07-26T06:49:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.458+0000 7f08f1927740 20 client.15114 trim_cache size 0 max 16384 2024-07-26T06:49:31.468 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:49:31.468 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_fsid": "8b3e2750-4b19-11ef-bcb6-c7b262605968", 2024-07-26T06:49:31.468 INFO:teuthology.orchestra.run.smithi042.stdout: "whoami": 2, 2024-07-26T06:49:31.468 INFO:teuthology.orchestra.run.smithi042.stdout: "id": 24593, 2024-07-26T06:49:31.468 INFO:teuthology.orchestra.run.smithi042.stdout: "want_state": "up:active", 2024-07-26T06:49:31.468 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "up:active", 2024-07-26T06:49:31.468 INFO:teuthology.orchestra.run.smithi042.stdout: "fs_name": "cephfs", 2024-07-26T06:49:31.469 INFO:teuthology.orchestra.run.smithi042.stdout: "rank_uptime": 131.38619404799999, 2024-07-26T06:49:31.469 INFO:teuthology.orchestra.run.smithi042.stdout: "mdsmap_epoch": 28, 2024-07-26T06:49:31.469 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch": 80, 2024-07-26T06:49:31.469 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch_barrier": 80, 2024-07-26T06:49:31.469 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 173.108546362 2024-07-26T06:49:31.469 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:49:31.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.459+0000 7f08eb3a3640 1 -- 172.21.15.42:0/2721461250 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f08d00b3bd0 msgr2=0x7f08d00b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:31.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.459+0000 7f08eb3a3640 1 --2- 172.21.15.42:0/2721461250 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f08d00b3bd0 0x7f08d00b60e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f08e0031440 tx=0x7f08e0032b60 comp rx=0 tx=0).stop 2024-07-26T06:49:31.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.459+0000 7f08eb3a3640 1 -- 172.21.15.42:0/2721461250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08e4107600 msgr2=0x7f08e419a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:31.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.459+0000 7f08eb3a3640 1 --2- 172.21.15.42:0/2721461250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08e4107600 0x7f08e419a570 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f08d4006fd0 tx=0x7f08d400b8e0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.459+0000 7f08eb3a3640 1 -- 172.21.15.42:0/2721461250 shutdown_connections 2024-07-26T06:49:31.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.459+0000 7f08eb3a3640 1 --2- 172.21.15.42:0/2721461250 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f08d00b3bd0 0x7f08d00b60e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.459+0000 7f08eb3a3640 1 --2- 172.21.15.42:0/2721461250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08e4107f50 0x7f08e419f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.459+0000 7f08eb3a3640 1 --2- 172.21.15.42:0/2721461250 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08e4107600 0x7f08e419a570 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.459+0000 7f08eb3a3640 1 --2- 172.21.15.42:0/2721461250 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f08e4101cc0 0x7f08e4199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:31.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.459+0000 7f08eb3a3640 1 -- 172.21.15.42:0/2721461250 >> 172.21.15.42:0/2721461250 conn(0x7f08e40798d0 msgr2=0x7f08e4105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:31.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.459+0000 7f08eb3a3640 1 -- 172.21.15.42:0/2721461250 shutdown_connections 2024-07-26T06:49:31.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:31.459+0000 7f08eb3a3640 1 -- 172.21.15.42:0/2721461250 wait complete. 2024-07-26T06:49:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:30 smithi164 ceph-mon[36970]: pgmap v374: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:49:32.252 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell mds.i status 2024-07-26T06:49:32.525 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:33.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:32 smithi163 ceph-mon[36873]: pgmap v375: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:49:33.274 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:32 smithi164 ceph-mon[36970]: pgmap v375: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:49:33.348 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:32 smithi042 ceph-mon[32775]: pgmap v375: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:49:34.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.163+0000 7f17deed6640 1 -- 172.21.15.42:0/4248537300 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17d8103590 msgr2=0x7f17d81039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:34.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.163+0000 7f17deed6640 1 --2- 172.21.15.42:0/4248537300 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17d8103590 0x7f17d81039a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009900 tx=0x7f17cc02a410 comp rx=0 tx=0).stop 2024-07-26T06:49:34.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.163+0000 7f17deed6640 1 -- 172.21.15.42:0/4248537300 shutdown_connections 2024-07-26T06:49:34.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.163+0000 7f17deed6640 1 --2- 172.21.15.42:0/4248537300 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17d8109820 0x7f17d8110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.163+0000 7f17deed6640 1 --2- 172.21.15.42:0/4248537300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d8108ed0 0x7f17d81092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.163+0000 7f17deed6640 1 --2- 172.21.15.42:0/4248537300 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17d8103590 0x7f17d81039a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.163+0000 7f17deed6640 1 -- 172.21.15.42:0/4248537300 >> 172.21.15.42:0/4248537300 conn(0x7f17d807b280 msgr2=0x7f17d807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:34.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.163+0000 7f17deed6640 1 -- 172.21.15.42:0/4248537300 shutdown_connections 2024-07-26T06:49:34.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.163+0000 7f17deed6640 1 -- 172.21.15.42:0/4248537300 wait complete. 2024-07-26T06:49:34.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.164+0000 7f17deed6640 1 Processor -- start 2024-07-26T06:49:34.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.164+0000 7f17deed6640 1 -- start start 2024-07-26T06:49:34.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.164+0000 7f17deed6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17d8103590 0x7f17d8075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.164+0000 7f17deed6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d8108ed0 0x7f17d8075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.164+0000 7f17deed6640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17d8109820 0x7f17d8075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.164+0000 7f17deed6640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8113520 con 0x7f17d8103590 2024-07-26T06:49:34.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.164+0000 7f17deed6640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f17d81133a0 con 0x7f17d8109820 2024-07-26T06:49:34.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.164+0000 7f17deed6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f17d81136a0 con 0x7f17d8108ed0 2024-07-26T06:49:34.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.164+0000 7f17dcc72640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17d8103590 0x7f17d8075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:34.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.164+0000 7f17dcc72640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17d8103590 0x7f17d8075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51440/0 (socket says 172.21.15.42:51440) 2024-07-26T06:49:34.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.164+0000 7f17dcc72640 1 -- 172.21.15.42:0/337800420 learned_addr learned my addr 172.21.15.42:0/337800420 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:34.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.164+0000 7f17dd473640 1 --2- 172.21.15.42:0/337800420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17d8109820 0x7f17d8075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:34.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.164+0000 7f17d7fff640 1 --2- 172.21.15.42:0/337800420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d8108ed0 0x7f17d8075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:34.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.165+0000 7f17dcc72640 1 -- 172.21.15.42:0/337800420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d8108ed0 msgr2=0x7f17d8075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:34.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.165+0000 7f17dcc72640 1 --2- 172.21.15.42:0/337800420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d8108ed0 0x7f17d8075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.165+0000 7f17dcc72640 1 -- 172.21.15.42:0/337800420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17d8109820 msgr2=0x7f17d8075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:34.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.165+0000 7f17dcc72640 1 --2- 172.21.15.42:0/337800420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17d8109820 0x7f17d8075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.165+0000 7f17dcc72640 1 -- 172.21.15.42:0/337800420 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d81a81c0 con 0x7f17d8103590 2024-07-26T06:49:34.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.165+0000 7f17dd473640 1 --2- 172.21.15.42:0/337800420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17d8109820 0x7f17d8075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:49:34.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.165+0000 7f17dcc72640 1 --2- 172.21.15.42:0/337800420 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17d8103590 0x7f17d8075140 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f17cc006010 tx=0x7f17cc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:34.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.165+0000 7f17d5ffb640 1 -- 172.21.15.42:0/337800420 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc03f030 con 0x7f17d8103590 2024-07-26T06:49:34.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.165+0000 7f17d5ffb640 1 -- 172.21.15.42:0/337800420 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17cc003800 con 0x7f17d8103590 2024-07-26T06:49:34.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.165+0000 7f17deed6640 1 -- 172.21.15.42:0/337800420 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17d81a8450 con 0x7f17d8103590 2024-07-26T06:49:34.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.165+0000 7f17d5ffb640 1 -- 172.21.15.42:0/337800420 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc035070 con 0x7f17d8103590 2024-07-26T06:49:34.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.165+0000 7f17deed6640 1 -- 172.21.15.42:0/337800420 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17d81a8930 con 0x7f17d8103590 2024-07-26T06:49:34.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.169+0000 7f17e545a740 1 Processor -- start 2024-07-26T06:49:34.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.169+0000 7f17e545a740 1 -- start start 2024-07-26T06:49:34.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.169+0000 7f17d5ffb640 1 -- 172.21.15.42:0/337800420 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f17cc0039b0 con 0x7f17d8103590 2024-07-26T06:49:34.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.169+0000 7f17e545a740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f94821f60 0x557f94bc7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.169+0000 7f17e545a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f947f90a0 0x557f94bc78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.169+0000 7f17e545a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f94801280 0x557f94bc7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.169+0000 7f17e545a740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x557f94b3d920 con 0x557f94821f60 2024-07-26T06:49:34.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.169+0000 7f17e545a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557f94b2ce20 con 0x557f947f90a0 2024-07-26T06:49:34.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.169+0000 7f17e545a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x557f94b30e60 con 0x557f94801280 2024-07-26T06:49:34.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.170+0000 7f17dd473640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f94821f60 0x557f94bc7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:34.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.170+0000 7f17dd473640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f94821f60 0x557f94bc7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51446/0 (socket says 172.21.15.42:51446) 2024-07-26T06:49:34.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.170+0000 7f17dd473640 1 -- 172.21.15.42:0/4154216264 learned_addr learned my addr 172.21.15.42:0/4154216264 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:34.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.170+0000 7f17dcc72640 1 --2- 172.21.15.42:0/4154216264 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f947f90a0 0x557f94bc78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:34.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.170+0000 7f17d7fff640 1 --2- 172.21.15.42:0/4154216264 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f94801280 0x557f94bc7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:34.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.170+0000 7f17dd473640 1 -- 172.21.15.42:0/4154216264 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f94801280 msgr2=0x557f94bc7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:34.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.170+0000 7f17dd473640 1 --2- 172.21.15.42:0/4154216264 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f94801280 0x557f94bc7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.170+0000 7f17dd473640 1 -- 172.21.15.42:0/4154216264 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f947f90a0 msgr2=0x557f94bc78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:34.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.170+0000 7f17dd473640 1 --2- 172.21.15.42:0/4154216264 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f947f90a0 0x557f94bc78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.170+0000 7f17dd473640 1 -- 172.21.15.42:0/4154216264 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f94ad3a40 con 0x557f94821f60 2024-07-26T06:49:34.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.170+0000 7f17d5ffb640 1 --2- 172.21.15.42:0/337800420 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f17c80b3bd0 0x7f17c80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.170+0000 7f17dcc72640 1 --2- 172.21.15.42:0/337800420 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f17c80b3bd0 0x7f17c80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:34.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.170+0000 7f17dd473640 1 --2- 172.21.15.42:0/4154216264 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f94821f60 0x557f94bc7380 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f17d0007f40 tx=0x7f17d000e7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=722c92c0cffcec26 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:34.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.170+0000 7f17d5ffb640 1 -- 172.21.15.42:0/337800420 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f17cc104630 con 0x7f17d8103590 2024-07-26T06:49:34.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.171+0000 7f17c2ffd640 1 -- 172.21.15.42:0/4154216264 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d0014070 con 0x557f94821f60 2024-07-26T06:49:34.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.171+0000 7f17c2ffd640 1 -- 172.21.15.42:0/4154216264 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17d0004d20 con 0x557f94821f60 2024-07-26T06:49:34.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.171+0000 7f17c2ffd640 1 -- 172.21.15.42:0/4154216264 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d0005520 con 0x557f94821f60 2024-07-26T06:49:34.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.172+0000 7f17e545a740 1 -- 172.21.15.42:0/4154216264 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f94821f60 msgr2=0x557f94bc7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:34.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.172+0000 7f17e545a740 1 --2- 172.21.15.42:0/4154216264 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f94821f60 0x557f94bc7380 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f17d0007f40 tx=0x7f17d000e7e0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.172+0000 7f17dcc72640 1 --2- 172.21.15.42:0/337800420 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f17c80b3bd0 0x7f17c80b60e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x557f94bc6000 tx=0x7f17cc02a410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:34.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.172+0000 7f17e545a740 1 -- 172.21.15.42:0/4154216264 shutdown_connections 2024-07-26T06:49:34.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.172+0000 7f17e545a740 1 --2- 172.21.15.42:0/4154216264 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f94801280 0x557f94bc7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.172+0000 7f17e545a740 1 --2- 172.21.15.42:0/4154216264 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f947f90a0 0x557f94bc78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.172+0000 7f17e545a740 1 --2- 172.21.15.42:0/4154216264 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f94821f60 0x557f94bc7380 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.172+0000 7f17e545a740 1 -- 172.21.15.42:0/4154216264 >> 172.21.15.42:0/4154216264 conn(0x557f94816350 msgr2=0x557f94bc5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:34.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.172+0000 7f17e545a740 1 -- 172.21.15.42:0/4154216264 shutdown_connections 2024-07-26T06:49:34.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.172+0000 7f17e545a740 1 -- 172.21.15.42:0/4154216264 wait complete. 2024-07-26T06:49:34.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.172+0000 7f17e545a740 1 -- start start 2024-07-26T06:49:34.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.172+0000 7f17e545a740 1 Processor -- start 2024-07-26T06:49:34.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17e545a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f947f90a0 0x557f94bd2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17e545a740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f94801280 0x557f94bd3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17e545a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f94821f60 0x557f94bdeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17e545a740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x557f949905f0 con 0x557f94801280 2024-07-26T06:49:34.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17e545a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557f94acf890 con 0x557f94821f60 2024-07-26T06:49:34.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17e545a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x557f94bd5c00 con 0x557f947f90a0 2024-07-26T06:49:34.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17e545a740 10 client.0 _ll_register_callbacks cb 0x557f94ad97d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:49:34.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17d7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f94801280 0x557f94bd3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:34.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17d7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f94801280 0x557f94bd3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51450/0 (socket says 172.21.15.42:51450) 2024-07-26T06:49:34.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17d7fff640 1 -- 172.21.15.42:0/678408701 learned_addr learned my addr 172.21.15.42:0/678408701 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:34.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17dd473640 1 --2- 172.21.15.42:0/678408701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f947f90a0 0x557f94bd2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:34.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17dd473640 1 --2- 172.21.15.42:0/678408701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f94821f60 0x557f94bdeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:34.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17d7fff640 1 -- 172.21.15.42:0/678408701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f947f90a0 msgr2=0x557f94bd2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:34.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17d7fff640 1 --2- 172.21.15.42:0/678408701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f947f90a0 0x557f94bd2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17d7fff640 1 -- 172.21.15.42:0/678408701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f94821f60 msgr2=0x557f94bdeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:34.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17d7fff640 1 --2- 172.21.15.42:0/678408701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f94821f60 0x557f94bdeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17d7fff640 1 -- 172.21.15.42:0/678408701 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f9488d540 con 0x557f94801280 2024-07-26T06:49:34.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.173+0000 7f17dd473640 1 --2- 172.21.15.42:0/678408701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f94821f60 0x557f94bdeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:49:34.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.174+0000 7f17d7fff640 1 --2- 172.21.15.42:0/678408701 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f94801280 0x557f94bd3480 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f17c40104d0 tx=0x7f17c4010990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2602410a23ef952d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:34.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.174+0000 7f17b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:49:34.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.174+0000 7f17b7fff640 1 -- 172.21.15.42:0/678408701 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c401a020 con 0x557f94801280 2024-07-26T06:49:34.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.174+0000 7f17b7fff640 1 -- 172.21.15.42:0/678408701 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f17c4016690 con 0x557f94801280 2024-07-26T06:49:34.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.174+0000 7f17e545a740 1 -- 172.21.15.42:0/678408701 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17d0004d20 con 0x557f94801280 2024-07-26T06:49:34.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.174+0000 7f17b7fff640 1 -- 172.21.15.42:0/678408701 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f17c4025050 con 0x557f94801280 2024-07-26T06:49:34.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.174+0000 7f17e545a740 10 client.15126 fetch_fsmap learned FSMap version 28 2024-07-26T06:49:34.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.174+0000 7f17e545a740 1 -- 172.21.15.42:0/678408701 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557f94be1c40 con 0x557f94801280 2024-07-26T06:49:34.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.174+0000 7f17b7fff640 1 -- 172.21.15.42:0/678408701 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17d0004d20 con 0x557f94801280 2024-07-26T06:49:34.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.174+0000 7f17b7fff640 1 -- 172.21.15.42:0/678408701 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f17c4014a40 con 0x557f94801280 2024-07-26T06:49:34.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.174+0000 7f17e545a740 10 client.15126 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:49:34.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.175+0000 7f17e545a740 10 client.15126 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T06:49:34.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.175+0000 7f17e545a740 20 client.15126 populate_metadata read hostname 'smithi042' 2024-07-26T06:49:34.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.175+0000 7f17e545a740 1 --2- 172.21.15.42:0/678408701 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x557f947e08e0 0x557f94be5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.175+0000 7f17e545a740 4 client.15126 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:49:34.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.175+0000 7f17e545a740 1 -- 172.21.15.42:0/678408701 --> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f94b3f870 con 0x557f947e08e0 2024-07-26T06:49:34.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.175+0000 7f17dd473640 1 --2- 172.21.15.42:0/678408701 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x557f947e08e0 0x557f94be5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:34.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.175+0000 7f17b7fff640 10 client.15126 ms_handle_connect on v2:172.21.15.164:6838/3593178314 2024-07-26T06:49:34.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.175+0000 7f17dd473640 1 --2- 172.21.15.42:0/678408701 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x557f947e08e0 0x557f94be5730 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9362010f40e19fb4 server_cookie=7aed827e126ff8c2 in_seq=0 out_seq=0 2024-07-26T06:49:34.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.176+0000 7f17b7fff640 1 -- 172.21.15.42:0/678408701 <== mds.1 v2:172.21.15.164:6838/3593178314 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557f94b3f870 con 0x557f947e08e0 2024-07-26T06:49:34.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.176+0000 7f17b7fff640 10 client.15126 handle_command_reply: tid=0 2024-07-26T06:49:34.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.176+0000 7f17e545a740 1 client.15126 shutdown 2024-07-26T06:49:34.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.176+0000 7f17e545a740 1 -- 172.21.15.42:0/678408701 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f94801280 msgr2=0x557f94bd3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:34.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.176+0000 7f17e545a740 1 --2- 172.21.15.42:0/678408701 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f94801280 0x557f94bd3480 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f17c40104d0 tx=0x7f17c4010990 comp rx=0 tx=0).stop 2024-07-26T06:49:34.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.176+0000 7f17e545a740 1 -- 172.21.15.42:0/678408701 shutdown_connections 2024-07-26T06:49:34.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.176+0000 7f17e545a740 1 --2- 172.21.15.42:0/678408701 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x557f947e08e0 0x557f94be5730 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.176+0000 7f17e545a740 1 --2- 172.21.15.42:0/678408701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f94821f60 0x557f94bdeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.176+0000 7f17b7fff640 0 client.15126 ms_handle_reset on v2:172.21.15.164:6838/3593178314 2024-07-26T06:49:34.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.176+0000 7f17e545a740 1 --2- 172.21.15.42:0/678408701 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f94801280 0x557f94bd3480 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.176+0000 7f17e545a740 1 --2- 172.21.15.42:0/678408701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f947f90a0 0x557f94bd2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.176+0000 7f17e545a740 1 -- 172.21.15.42:0/678408701 >> 172.21.15.42:0/678408701 conn(0x557f94816350 msgr2=0x557f94bc83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:34.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.177+0000 7f17e545a740 1 -- 172.21.15.42:0/678408701 shutdown_connections 2024-07-26T06:49:34.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.177+0000 7f17e545a740 1 -- 172.21.15.42:0/678408701 wait complete. 2024-07-26T06:49:34.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.177+0000 7f17e545a740 20 client.15126 trim_cache size 0 max 16384 2024-07-26T06:49:34.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.195+0000 7f17e545a740 1 Processor -- start 2024-07-26T06:49:34.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.195+0000 7f17e545a740 1 -- start start 2024-07-26T06:49:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.195+0000 7f17e545a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f947e08e0 0x557f94bcd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.195+0000 7f17e545a740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f947f90a0 0x557f94bd2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.195+0000 7f17e545a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f94801280 0x557f94bd15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.195+0000 7f17e545a740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x557f94be3a70 con 0x557f947f90a0 2024-07-26T06:49:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.195+0000 7f17e545a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557f94be4140 con 0x557f94801280 2024-07-26T06:49:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.195+0000 7f17e545a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x557f94be4810 con 0x557f947e08e0 2024-07-26T06:49:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.195+0000 7f17d7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f947f90a0 0x557f94bd2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.195+0000 7f17d7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f947f90a0 0x557f94bd2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51452/0 (socket says 172.21.15.42:51452) 2024-07-26T06:49:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.195+0000 7f17d7fff640 1 -- 172.21.15.42:0/1262308757 learned_addr learned my addr 172.21.15.42:0/1262308757 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.196+0000 7f17dd473640 1 --2- 172.21.15.42:0/1262308757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f947e08e0 0x557f94bcd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.196+0000 7f17d7fff640 1 -- 172.21.15.42:0/1262308757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f947e08e0 msgr2=0x557f94bcd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:34.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.196+0000 7f17d7fff640 1 --2- 172.21.15.42:0/1262308757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f947e08e0 0x557f94bcd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.196+0000 7f17d7fff640 1 -- 172.21.15.42:0/1262308757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f94801280 msgr2=0x557f94bd15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:34.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.196+0000 7f17d7fff640 1 --2- 172.21.15.42:0/1262308757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f94801280 0x557f94bd15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.196+0000 7f17d7fff640 1 -- 172.21.15.42:0/1262308757 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f94bd3600 con 0x557f947f90a0 2024-07-26T06:49:34.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.196+0000 7f17dd473640 1 --2- 172.21.15.42:0/1262308757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f947e08e0 0x557f94bcd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:49:34.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.196+0000 7f17d7fff640 1 --2- 172.21.15.42:0/1262308757 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f947f90a0 0x557f94bd2230 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f17c4014eb0 tx=0x7f17c4006a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fbab5dfb70ca223 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:34.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.196+0000 7f17c27fc640 1 -- 172.21.15.42:0/1262308757 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c402e070 con 0x557f947f90a0 2024-07-26T06:49:34.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.196+0000 7f17c27fc640 1 -- 172.21.15.42:0/1262308757 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17c4012d10 con 0x557f947f90a0 2024-07-26T06:49:34.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.196+0000 7f17c27fc640 1 -- 172.21.15.42:0/1262308757 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c401a040 con 0x557f947f90a0 2024-07-26T06:49:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.197+0000 7f17e545a740 1 -- 172.21.15.42:0/1262308757 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f947f90a0 msgr2=0x557f94bd2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.197+0000 7f17e545a740 1 --2- 172.21.15.42:0/1262308757 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f947f90a0 0x557f94bd2230 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f17c4014eb0 tx=0x7f17c4006a30 comp rx=0 tx=0).stop 2024-07-26T06:49:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.197+0000 7f17e545a740 1 -- 172.21.15.42:0/1262308757 shutdown_connections 2024-07-26T06:49:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.197+0000 7f17e545a740 1 --2- 172.21.15.42:0/1262308757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f94801280 0x557f94bd15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.197+0000 7f17e545a740 1 --2- 172.21.15.42:0/1262308757 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f947f90a0 0x557f94bd2230 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.197+0000 7f17e545a740 1 --2- 172.21.15.42:0/1262308757 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f947e08e0 0x557f94bcd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.197+0000 7f17e545a740 1 -- 172.21.15.42:0/1262308757 >> 172.21.15.42:0/1262308757 conn(0x557f94816350 msgr2=0x557f94bcde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.197+0000 7f17e545a740 1 -- 172.21.15.42:0/1262308757 shutdown_connections 2024-07-26T06:49:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.197+0000 7f17e545a740 1 -- 172.21.15.42:0/1262308757 wait complete. 2024-07-26T06:49:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.197+0000 7f17e545a740 1 -- start start 2024-07-26T06:49:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.198+0000 7f17e545a740 1 Processor -- start 2024-07-26T06:49:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.198+0000 7f17e545a740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f947e08e0 0x557f94be6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.198+0000 7f17e545a740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f947f90a0 0x557f94be7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.198+0000 7f17e545a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f94801280 0x557f94bec450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.198+0000 7f17e545a740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x557f94be55b0 con 0x557f947e08e0 2024-07-26T06:49:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.198+0000 7f17e545a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557f94be4ee0 con 0x557f94801280 2024-07-26T06:49:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.198+0000 7f17e545a740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x557f94be5c80 con 0x557f947f90a0 2024-07-26T06:49:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.198+0000 7f17e545a740 10 client.0 _ll_register_callbacks cb 0x7f17c400bdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:49:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.198+0000 7f17dd473640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f947e08e0 0x557f94be6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.198+0000 7f17dd473640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f947e08e0 0x557f94be6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51454/0 (socket says 172.21.15.42:51454) 2024-07-26T06:49:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.198+0000 7f17dd473640 1 -- 172.21.15.42:0/1569678178 learned_addr learned my addr 172.21.15.42:0/1569678178 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.198+0000 7f17d7fff640 1 --2- 172.21.15.42:0/1569678178 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f947f90a0 0x557f94be7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.199+0000 7f17dd473640 1 -- 172.21.15.42:0/1569678178 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f947f90a0 msgr2=0x557f94be7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.199+0000 7f17dd473640 1 --2- 172.21.15.42:0/1569678178 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f947f90a0 0x557f94be7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.199+0000 7f17dd473640 1 -- 172.21.15.42:0/1569678178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f94801280 msgr2=0x557f94bec450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.199+0000 7f17dd473640 1 --2- 172.21.15.42:0/1569678178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f94801280 0x557f94bec450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.199+0000 7f17dd473640 1 -- 172.21.15.42:0/1569678178 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f94becac0 con 0x557f947e08e0 2024-07-26T06:49:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.199+0000 7f17dd473640 1 --2- 172.21.15.42:0/1569678178 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f947e08e0 0x557f94be6f60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f17d0007f10 tx=0x7f17d0005130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab6c96379e448f92 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.199+0000 7f17b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:49:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.199+0000 7f17b7fff640 1 -- 172.21.15.42:0/1569678178 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d0014040 con 0x557f947e08e0 2024-07-26T06:49:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.199+0000 7f17b7fff640 1 -- 172.21.15.42:0/1569678178 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f17d0004c00 con 0x557f947e08e0 2024-07-26T06:49:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.199+0000 7f17e545a740 1 -- 172.21.15.42:0/1569678178 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17c4012d10 con 0x557f947e08e0 2024-07-26T06:49:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.199+0000 7f17b7fff640 1 -- 172.21.15.42:0/1569678178 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f17d0021040 con 0x557f947e08e0 2024-07-26T06:49:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.199+0000 7f17e545a740 10 client.15132 fetch_fsmap learned FSMap version 28 2024-07-26T06:49:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.199+0000 7f17e545a740 1 -- 172.21.15.42:0/1569678178 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557f94beddf0 con 0x557f947e08e0 2024-07-26T06:49:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.200+0000 7f17b7fff640 1 -- 172.21.15.42:0/1569678178 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17c4012d10 con 0x557f947e08e0 2024-07-26T06:49:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.200+0000 7f17b7fff640 1 -- 172.21.15.42:0/1569678178 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f17d002f020 con 0x557f947e08e0 2024-07-26T06:49:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.200+0000 7f17e545a740 10 client.15132 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:49:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.200+0000 7f17e545a740 10 client.15132 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T06:49:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.200+0000 7f17e545a740 20 client.15132 populate_metadata read hostname 'smithi042' 2024-07-26T06:49:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.200+0000 7f17e545a740 1 --2- 172.21.15.42:0/1569678178 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x557f94821f60 0x557f94c516a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.200+0000 7f17e545a740 4 client.15132 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:49:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.200+0000 7f17e545a740 1 -- 172.21.15.42:0/1569678178 --> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] -- command(tid 0: {"prefix": "status"}) -- 0x557f94b3b310 con 0x557f94821f60 2024-07-26T06:49:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.200+0000 7f17d7fff640 1 --2- 172.21.15.42:0/1569678178 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x557f94821f60 0x557f94c516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:34.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.201+0000 7f17b7fff640 10 client.15132 ms_handle_connect on v2:172.21.15.164:6838/3593178314 2024-07-26T06:49:34.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.201+0000 7f17d7fff640 1 --2- 172.21.15.42:0/1569678178 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x557f94821f60 0x557f94c516a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=25cac4e3af5dfe53 server_cookie=d2fc8d373947a174 in_seq=0 out_seq=0 2024-07-26T06:49:34.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.201+0000 7f17b7fff640 1 -- 172.21.15.42:0/1569678178 <== mds.1 v2:172.21.15.164:6838/3593178314 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557f94b3b310 con 0x557f94821f60 2024-07-26T06:49:34.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.201+0000 7f17b7fff640 10 client.15132 handle_command_reply: tid=0 2024-07-26T06:49:34.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.201+0000 7f17e545a740 1 client.15132 shutdown 2024-07-26T06:49:34.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.201+0000 7f17e545a740 1 -- 172.21.15.42:0/1569678178 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f947e08e0 msgr2=0x557f94be6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:34.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.201+0000 7f17e545a740 1 --2- 172.21.15.42:0/1569678178 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f947e08e0 0x557f94be6f60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f17d0007f10 tx=0x7f17d0005130 comp rx=0 tx=0).stop 2024-07-26T06:49:34.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.201+0000 7f17e545a740 1 -- 172.21.15.42:0/1569678178 shutdown_connections 2024-07-26T06:49:34.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.201+0000 7f17e545a740 1 --2- 172.21.15.42:0/1569678178 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x557f94821f60 0x557f94c516a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.201+0000 7f17e545a740 1 --2- 172.21.15.42:0/1569678178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557f94801280 0x557f94bec450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.201+0000 7f17e545a740 1 --2- 172.21.15.42:0/1569678178 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557f947f90a0 0x557f94be7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.201+0000 7f17e545a740 1 --2- 172.21.15.42:0/1569678178 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557f947e08e0 0x557f94be6f60 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.201+0000 7f17e545a740 1 -- 172.21.15.42:0/1569678178 >> 172.21.15.42:0/1569678178 conn(0x557f94816350 msgr2=0x557f94bd1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:34.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.201+0000 7f17b7fff640 0 client.15132 ms_handle_reset on v2:172.21.15.164:6838/3593178314 2024-07-26T06:49:34.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.201+0000 7f17e545a740 1 -- 172.21.15.42:0/1569678178 shutdown_connections 2024-07-26T06:49:34.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.201+0000 7f17e545a740 1 -- 172.21.15.42:0/1569678178 wait complete. 2024-07-26T06:49:34.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.201+0000 7f17e545a740 20 client.15132 trim_cache size 0 max 16384 2024-07-26T06:49:34.207 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:49:34.207 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_fsid": "8b3e2750-4b19-11ef-bcb6-c7b262605968", 2024-07-26T06:49:34.207 INFO:teuthology.orchestra.run.smithi042.stdout: "whoami": 1, 2024-07-26T06:49:34.207 INFO:teuthology.orchestra.run.smithi042.stdout: "id": 24601, 2024-07-26T06:49:34.207 INFO:teuthology.orchestra.run.smithi042.stdout: "want_state": "up:active", 2024-07-26T06:49:34.208 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "up:active", 2024-07-26T06:49:34.208 INFO:teuthology.orchestra.run.smithi042.stdout: "fs_name": "cephfs", 2024-07-26T06:49:34.208 INFO:teuthology.orchestra.run.smithi042.stdout: "rank_uptime": 135.137516917, 2024-07-26T06:49:34.208 INFO:teuthology.orchestra.run.smithi042.stdout: "mdsmap_epoch": 28, 2024-07-26T06:49:34.208 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch": 80, 2024-07-26T06:49:34.208 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch_barrier": 80, 2024-07-26T06:49:34.208 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 173.22791760999999 2024-07-26T06:49:34.208 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:49:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.202+0000 7f17deed6640 1 -- 172.21.15.42:0/337800420 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f17c80b3bd0 msgr2=0x7f17c80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.202+0000 7f17deed6640 1 --2- 172.21.15.42:0/337800420 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f17c80b3bd0 0x7f17c80b60e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x557f94bc6000 tx=0x7f17cc02a410 comp rx=0 tx=0).stop 2024-07-26T06:49:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.202+0000 7f17deed6640 1 -- 172.21.15.42:0/337800420 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17d8103590 msgr2=0x7f17d8075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.202+0000 7f17deed6640 1 --2- 172.21.15.42:0/337800420 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17d8103590 0x7f17d8075140 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f17cc006010 tx=0x7f17cc0042f0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.202+0000 7f17deed6640 1 -- 172.21.15.42:0/337800420 shutdown_connections 2024-07-26T06:49:34.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.202+0000 7f17deed6640 1 --2- 172.21.15.42:0/337800420 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f17c80b3bd0 0x7f17c80b60e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.202+0000 7f17deed6640 1 --2- 172.21.15.42:0/337800420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17d8109820 0x7f17d8075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.202+0000 7f17deed6640 1 --2- 172.21.15.42:0/337800420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d8108ed0 0x7f17d8075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.202+0000 7f17deed6640 1 --2- 172.21.15.42:0/337800420 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f17d8103590 0x7f17d8075140 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:34.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.202+0000 7f17deed6640 1 -- 172.21.15.42:0/337800420 >> 172.21.15.42:0/337800420 conn(0x7f17d807b280 msgr2=0x7f17d8107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:34.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.202+0000 7f17deed6640 1 -- 172.21.15.42:0/337800420 shutdown_connections 2024-07-26T06:49:34.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:34.202+0000 7f17deed6640 1 -- 172.21.15.42:0/337800420 wait complete. 2024-07-26T06:49:34.932 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell mds.b status 2024-07-26T06:49:35.204 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:35.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:34 smithi163 ceph-mon[36873]: pgmap v376: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:35.327 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:34 smithi042 ceph-mon[32775]: pgmap v376: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:35.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:34 smithi164 ceph-mon[36970]: pgmap v376: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:36.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.898+0000 7f8960fd8640 1 -- 172.21.15.42:0/1687041756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f895c108720 msgr2=0x7f895c10ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:36.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.898+0000 7f8960fd8640 1 --2- 172.21.15.42:0/1687041756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f895c108720 0x7f895c10ef50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f894c009ca0 tx=0x7f894c02a8c0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.898+0000 7f8960fd8640 1 -- 172.21.15.42:0/1687041756 shutdown_connections 2024-07-26T06:49:36.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.898+0000 7f8960fd8640 1 --2- 172.21.15.42:0/1687041756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f895c108720 0x7f895c10ef50 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.898+0000 7f8960fd8640 1 --2- 172.21.15.42:0/1687041756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f895c107dd0 0x7f895c1081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.898+0000 7f8960fd8640 1 --2- 172.21.15.42:0/1687041756 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f895c102490 0x7f895c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.898+0000 7f8960fd8640 1 -- 172.21.15.42:0/1687041756 >> 172.21.15.42:0/1687041756 conn(0x7f895c079160 msgr2=0x7f895c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:36.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.898+0000 7f8960fd8640 1 -- 172.21.15.42:0/1687041756 shutdown_connections 2024-07-26T06:49:36.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.898+0000 7f8960fd8640 1 -- 172.21.15.42:0/1687041756 wait complete. 2024-07-26T06:49:36.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.899+0000 7f8960fd8640 1 Processor -- start 2024-07-26T06:49:36.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.899+0000 7f8960fd8640 1 -- start start 2024-07-26T06:49:36.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.900+0000 7f8960fd8640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f895c102490 0x7f895c104b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.900+0000 7f8960fd8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f895c107dd0 0x7f895c105050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.900+0000 7f8960fd8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f895c108720 0x7f895c105590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.900+0000 7f8960fd8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f895c112400 con 0x7f895c107dd0 2024-07-26T06:49:36.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.900+0000 7f8960fd8640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f895c112280 con 0x7f895c102490 2024-07-26T06:49:36.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.900+0000 7f8960fd8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f895c112580 con 0x7f895c108720 2024-07-26T06:49:36.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.900+0000 7f8959d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f895c107dd0 0x7f895c105050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:36.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.900+0000 7f8959d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f895c107dd0 0x7f895c105050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41188/0 (socket says 172.21.15.42:41188) 2024-07-26T06:49:36.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.900+0000 7f8959d9b640 1 -- 172.21.15.42:0/3321065073 learned_addr learned my addr 172.21.15.42:0/3321065073 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:36.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.900+0000 7f895a59c640 1 --2- 172.21.15.42:0/3321065073 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f895c102490 0x7f895c104b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:36.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.900+0000 7f895ad9d640 1 --2- 172.21.15.42:0/3321065073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f895c108720 0x7f895c105590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:36.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.900+0000 7f8959d9b640 1 -- 172.21.15.42:0/3321065073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f895c108720 msgr2=0x7f895c105590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:36.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.900+0000 7f8959d9b640 1 --2- 172.21.15.42:0/3321065073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f895c108720 0x7f895c105590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.900+0000 7f8959d9b640 1 -- 172.21.15.42:0/3321065073 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f895c102490 msgr2=0x7f895c104b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:36.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.900+0000 7f8959d9b640 1 --2- 172.21.15.42:0/3321065073 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f895c102490 0x7f895c104b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.900+0000 7f8959d9b640 1 -- 172.21.15.42:0/3321065073 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895c1a8130 con 0x7f895c107dd0 2024-07-26T06:49:36.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.900+0000 7f8959d9b640 1 --2- 172.21.15.42:0/3321065073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f895c107dd0 0x7f895c105050 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f894400d770 tx=0x7f894400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:36.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.901+0000 7f89537fe640 1 -- 172.21.15.42:0/3321065073 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944017070 con 0x7f895c107dd0 2024-07-26T06:49:36.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.901+0000 7f8960fd8640 1 -- 172.21.15.42:0/3321065073 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f895c1a8420 con 0x7f895c107dd0 2024-07-26T06:49:36.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.901+0000 7f89537fe640 1 -- 172.21.15.42:0/3321065073 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8944009c30 con 0x7f895c107dd0 2024-07-26T06:49:36.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.901+0000 7f89537fe640 1 -- 172.21.15.42:0/3321065073 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894400f620 con 0x7f895c107dd0 2024-07-26T06:49:36.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.901+0000 7f8960fd8640 1 -- 172.21.15.42:0/3321065073 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f895c1a89e0 con 0x7f895c107dd0 2024-07-26T06:49:36.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.903+0000 7f896755c740 1 Processor -- start 2024-07-26T06:49:36.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.903+0000 7f896755c740 1 -- start start 2024-07-26T06:49:36.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.903+0000 7f89537fe640 1 -- 172.21.15.42:0/3321065073 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f894401a070 con 0x7f895c107dd0 2024-07-26T06:49:36.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.904+0000 7f896755c740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6fcf60 0x55b24daa23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.904+0000 7f896755c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6d40a0 0x55b24daa28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.904+0000 7f896755c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6dc280 0x55b24daa2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.904+0000 7f896755c740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55b24da18940 con 0x55b24d6fcf60 2024-07-26T06:49:36.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.904+0000 7f896755c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b24da07e40 con 0x55b24d6dc280 2024-07-26T06:49:36.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.904+0000 7f896755c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55b24da0be80 con 0x55b24d6d40a0 2024-07-26T06:49:36.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.904+0000 7f895ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6fcf60 0x55b24daa23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:36.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.904+0000 7f895ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6fcf60 0x55b24daa23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41198/0 (socket says 172.21.15.42:41198) 2024-07-26T06:49:36.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.904+0000 7f895ad9d640 1 -- 172.21.15.42:0/3099271364 learned_addr learned my addr 172.21.15.42:0/3099271364 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:36.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.904+0000 7f8959d9b640 1 --2- 172.21.15.42:0/3099271364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6dc280 0x55b24daa2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:36.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.904+0000 7f895a59c640 1 --2- 172.21.15.42:0/3099271364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6d40a0 0x55b24daa28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:36.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.904+0000 7f895ad9d640 1 -- 172.21.15.42:0/3099271364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6d40a0 msgr2=0x55b24daa28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:36.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.904+0000 7f895ad9d640 1 --2- 172.21.15.42:0/3099271364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6d40a0 0x55b24daa28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.904+0000 7f895ad9d640 1 -- 172.21.15.42:0/3099271364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6dc280 msgr2=0x55b24daa2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:36.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.904+0000 7f895ad9d640 1 --2- 172.21.15.42:0/3099271364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6dc280 0x55b24daa2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.904+0000 7f895ad9d640 1 -- 172.21.15.42:0/3099271364 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b24d9aea60 con 0x55b24d6fcf60 2024-07-26T06:49:36.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.904+0000 7f895ad9d640 1 --2- 172.21.15.42:0/3099271364 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6fcf60 0x55b24daa23a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f894c005cc0 tx=0x7f894c009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9479ed9d4fd828a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:36.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.905+0000 7f89537fe640 1 --2- 172.21.15.42:0/3321065073 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f89480b3bd0 0x7f89480b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.905+0000 7f8950ff9640 1 -- 172.21.15.42:0/3099271364 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c031cf0 con 0x55b24d6fcf60 2024-07-26T06:49:36.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.905+0000 7f8950ff9640 1 -- 172.21.15.42:0/3099271364 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f894c036070 con 0x55b24d6fcf60 2024-07-26T06:49:36.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.905+0000 7f895a59c640 1 --2- 172.21.15.42:0/3321065073 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f89480b3bd0 0x7f89480b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:36.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.905+0000 7f8950ff9640 1 -- 172.21.15.42:0/3099271364 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c0042a0 con 0x55b24d6fcf60 2024-07-26T06:49:36.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.905+0000 7f89537fe640 1 -- 172.21.15.42:0/3321065073 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f89440e3860 con 0x7f895c107dd0 2024-07-26T06:49:36.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.906+0000 7f895a59c640 1 --2- 172.21.15.42:0/3321065073 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f89480b3bd0 0x7f89480b60e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x55b24daa1020 tx=0x7f8954008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:36.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.906+0000 7f896755c740 1 -- 172.21.15.42:0/3099271364 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6fcf60 msgr2=0x55b24daa23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:36.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.906+0000 7f896755c740 1 --2- 172.21.15.42:0/3099271364 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6fcf60 0x55b24daa23a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f894c005cc0 tx=0x7f894c009800 comp rx=0 tx=0).stop 2024-07-26T06:49:36.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.906+0000 7f896755c740 1 -- 172.21.15.42:0/3099271364 shutdown_connections 2024-07-26T06:49:36.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.906+0000 7f896755c740 1 --2- 172.21.15.42:0/3099271364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6dc280 0x55b24daa2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.906+0000 7f896755c740 1 --2- 172.21.15.42:0/3099271364 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6d40a0 0x55b24daa28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.906+0000 7f896755c740 1 --2- 172.21.15.42:0/3099271364 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6fcf60 0x55b24daa23a0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.906+0000 7f896755c740 1 -- 172.21.15.42:0/3099271364 >> 172.21.15.42:0/3099271364 conn(0x55b24d6f1350 msgr2=0x55b24daa0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:36.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.906+0000 7f896755c740 1 -- 172.21.15.42:0/3099271364 shutdown_connections 2024-07-26T06:49:36.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.906+0000 7f896755c740 1 -- 172.21.15.42:0/3099271364 wait complete. 2024-07-26T06:49:36.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.906+0000 7f896755c740 1 -- start start 2024-07-26T06:49:36.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.906+0000 7f896755c740 1 Processor -- start 2024-07-26T06:49:36.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f896755c740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6d40a0 0x55b24daadf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f896755c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6dc280 0x55b24daae450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f896755c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6fcf60 0x55b24dab9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f896755c740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55b24d86b5f0 con 0x55b24d6d40a0 2024-07-26T06:49:36.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f896755c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b24d9aa8b0 con 0x55b24d6dc280 2024-07-26T06:49:36.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f896755c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55b24dab0b20 con 0x55b24d6fcf60 2024-07-26T06:49:36.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f896755c740 10 client.0 _ll_register_callbacks cb 0x55b24d9b47f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:49:36.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f895a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6d40a0 0x55b24daadf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:36.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f895a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6d40a0 0x55b24daadf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41210/0 (socket says 172.21.15.42:41210) 2024-07-26T06:49:36.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f895a59c640 1 -- 172.21.15.42:0/2166825790 learned_addr learned my addr 172.21.15.42:0/2166825790 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:36.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f8959d9b640 1 --2- 172.21.15.42:0/2166825790 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6dc280 0x55b24daae450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:36.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f895a59c640 1 -- 172.21.15.42:0/2166825790 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6fcf60 msgr2=0x55b24dab9e70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T06:49:36.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f895a59c640 1 --2- 172.21.15.42:0/2166825790 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6fcf60 0x55b24dab9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f895a59c640 1 -- 172.21.15.42:0/2166825790 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6dc280 msgr2=0x55b24daae450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:36.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f895a59c640 1 --2- 172.21.15.42:0/2166825790 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6dc280 0x55b24daae450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f895a59c640 1 -- 172.21.15.42:0/2166825790 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b24d768540 con 0x55b24d6d40a0 2024-07-26T06:49:36.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f895a59c640 1 --2- 172.21.15.42:0/2166825790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6d40a0 0x55b24daadf10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f89540146e0 tx=0x7f89540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c2bad491e61a6a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:36.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.907+0000 7f8941ffb640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:49:36.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.908+0000 7f8941ffb640 1 -- 172.21.15.42:0/2166825790 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895401b050 con 0x55b24d6d40a0 2024-07-26T06:49:36.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.908+0000 7f8941ffb640 1 -- 172.21.15.42:0/2166825790 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f89540158e0 con 0x55b24d6d40a0 2024-07-26T06:49:36.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.908+0000 7f896755c740 1 -- 172.21.15.42:0/2166825790 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f894c036070 con 0x55b24d6d40a0 2024-07-26T06:49:36.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.908+0000 7f8941ffb640 1 -- 172.21.15.42:0/2166825790 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8954015ae0 con 0x55b24d6d40a0 2024-07-26T06:49:36.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.908+0000 7f896755c740 10 client.15144 fetch_fsmap learned FSMap version 28 2024-07-26T06:49:36.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.908+0000 7f896755c740 1 -- 172.21.15.42:0/2166825790 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b24dabcb40 con 0x55b24d6d40a0 2024-07-26T06:49:36.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.908+0000 7f8941ffb640 1 -- 172.21.15.42:0/2166825790 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f894c036070 con 0x55b24d6d40a0 2024-07-26T06:49:36.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.908+0000 7f8941ffb640 1 -- 172.21.15.42:0/2166825790 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8954029400 con 0x55b24d6d40a0 2024-07-26T06:49:36.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.908+0000 7f896755c740 10 client.15144 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:49:36.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.908+0000 7f896755c740 10 client.15144 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T06:49:36.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.908+0000 7f896755c740 20 client.15144 populate_metadata read hostname 'smithi042' 2024-07-26T06:49:36.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.909+0000 7f896755c740 1 --2- 172.21.15.42:0/2166825790 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55b24d6bb8e0 0x55b24dac06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.909+0000 7f896755c740 4 client.15144 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:49:36.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.909+0000 7f896755c740 1 -- 172.21.15.42:0/2166825790 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b24da1a890 con 0x55b24d6bb8e0 2024-07-26T06:49:36.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.909+0000 7f8959d9b640 1 --2- 172.21.15.42:0/2166825790 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55b24d6bb8e0 0x55b24dac06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:36.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.909+0000 7f8959d9b640 1 --2- 172.21.15.42:0/2166825790 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55b24d6bb8e0 0x55b24dac06c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffef5032f13b2067 server_cookie=4f475b70ee6c9995 in_seq=0 out_seq=0 2024-07-26T06:49:36.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.909+0000 7f8941ffb640 10 client.15144 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:49:36.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.910+0000 7f8941ffb640 1 -- 172.21.15.42:0/2166825790 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b24da1a890 con 0x55b24d6bb8e0 2024-07-26T06:49:36.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.910+0000 7f8941ffb640 10 client.15144 handle_command_reply: tid=0 2024-07-26T06:49:36.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.910+0000 7f896755c740 1 client.15144 shutdown 2024-07-26T06:49:36.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.910+0000 7f896755c740 1 -- 172.21.15.42:0/2166825790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6d40a0 msgr2=0x55b24daadf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:36.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.910+0000 7f896755c740 1 --2- 172.21.15.42:0/2166825790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6d40a0 0x55b24daadf10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f89540146e0 tx=0x7f89540046c0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.910+0000 7f896755c740 1 -- 172.21.15.42:0/2166825790 shutdown_connections 2024-07-26T06:49:36.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.910+0000 7f896755c740 1 --2- 172.21.15.42:0/2166825790 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55b24d6bb8e0 0x55b24dac06c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.910+0000 7f8941ffb640 0 client.15144 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:49:36.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.910+0000 7f896755c740 1 --2- 172.21.15.42:0/2166825790 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6fcf60 0x55b24dab9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.910+0000 7f896755c740 1 --2- 172.21.15.42:0/2166825790 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6dc280 0x55b24daae450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.910+0000 7f896755c740 1 --2- 172.21.15.42:0/2166825790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6d40a0 0x55b24daadf10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.910+0000 7f896755c740 1 -- 172.21.15.42:0/2166825790 >> 172.21.15.42:0/2166825790 conn(0x55b24d6f1350 msgr2=0x55b24daa5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:36.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.910+0000 7f896755c740 1 -- 172.21.15.42:0/2166825790 shutdown_connections 2024-07-26T06:49:36.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.911+0000 7f896755c740 1 -- 172.21.15.42:0/2166825790 wait complete. 2024-07-26T06:49:36.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.911+0000 7f896755c740 20 client.15144 trim_cache size 0 max 16384 2024-07-26T06:49:36.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f896755c740 1 Processor -- start 2024-07-26T06:49:36.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f896755c740 1 -- start start 2024-07-26T06:49:36.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f896755c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6bb8e0 0x55b24dab9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f896755c740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6d40a0 0x55b24daba320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f896755c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6dc280 0x55b24daadef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f896755c740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55b24dabea00 con 0x55b24d6d40a0 2024-07-26T06:49:36.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f896755c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b24dabf0d0 con 0x55b24d6bb8e0 2024-07-26T06:49:36.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f896755c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55b24dabf7a0 con 0x55b24d6dc280 2024-07-26T06:49:36.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f8959d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6d40a0 0x55b24daba320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:36.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f8959d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6d40a0 0x55b24daba320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41222/0 (socket says 172.21.15.42:41222) 2024-07-26T06:49:36.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f8959d9b640 1 -- 172.21.15.42:0/1893648644 learned_addr learned my addr 172.21.15.42:0/1893648644 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:36.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f895ad9d640 1 --2- 172.21.15.42:0/1893648644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6dc280 0x55b24daadef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:36.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f895a59c640 1 --2- 172.21.15.42:0/1893648644 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6bb8e0 0x55b24dab9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:36.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f8959d9b640 1 -- 172.21.15.42:0/1893648644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6dc280 msgr2=0x55b24daadef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:36.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f8959d9b640 1 --2- 172.21.15.42:0/1893648644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6dc280 0x55b24daadef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f8959d9b640 1 -- 172.21.15.42:0/1893648644 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6bb8e0 msgr2=0x55b24dab9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:36.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f8959d9b640 1 --2- 172.21.15.42:0/1893648644 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6bb8e0 0x55b24dab9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f8959d9b640 1 -- 172.21.15.42:0/1893648644 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b24daa4690 con 0x55b24d6d40a0 2024-07-26T06:49:36.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.929+0000 7f895a59c640 1 --2- 172.21.15.42:0/1893648644 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6bb8e0 0x55b24dab9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:49:36.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.930+0000 7f8959d9b640 1 --2- 172.21.15.42:0/1893648644 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6d40a0 0x55b24daba320 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8944000c00 tx=0x7f89440e1210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4a3d755ae905968 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:36.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.930+0000 7f8943fff640 1 -- 172.21.15.42:0/1893648644 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944021070 con 0x55b24d6d40a0 2024-07-26T06:49:36.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.930+0000 7f8943fff640 1 -- 172.21.15.42:0/1893648644 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89440e6e50 con 0x55b24d6d40a0 2024-07-26T06:49:36.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.930+0000 7f8943fff640 1 -- 172.21.15.42:0/1893648644 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89440aa790 con 0x55b24d6d40a0 2024-07-26T06:49:36.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.931+0000 7f896755c740 1 -- 172.21.15.42:0/1893648644 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6d40a0 msgr2=0x55b24daba320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:36.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.931+0000 7f896755c740 1 --2- 172.21.15.42:0/1893648644 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6d40a0 0x55b24daba320 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8944000c00 tx=0x7f89440e1210 comp rx=0 tx=0).stop 2024-07-26T06:49:36.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.931+0000 7f896755c740 1 -- 172.21.15.42:0/1893648644 shutdown_connections 2024-07-26T06:49:36.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.931+0000 7f896755c740 1 --2- 172.21.15.42:0/1893648644 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6dc280 0x55b24daadef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.931+0000 7f896755c740 1 --2- 172.21.15.42:0/1893648644 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6d40a0 0x55b24daba320 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.931+0000 7f896755c740 1 --2- 172.21.15.42:0/1893648644 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6bb8e0 0x55b24dab9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.931+0000 7f896755c740 1 -- 172.21.15.42:0/1893648644 >> 172.21.15.42:0/1893648644 conn(0x55b24d6f1350 msgr2=0x55b24daa8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:36.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.931+0000 7f896755c740 1 -- 172.21.15.42:0/1893648644 shutdown_connections 2024-07-26T06:49:36.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.931+0000 7f896755c740 1 -- 172.21.15.42:0/1893648644 wait complete. 2024-07-26T06:49:36.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.931+0000 7f896755c740 1 -- start start 2024-07-26T06:49:36.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.931+0000 7f896755c740 1 Processor -- start 2024-07-26T06:49:36.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f896755c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6bb8e0 0x55b24dac1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f896755c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6d40a0 0x55b24dac2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f896755c740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6dc280 0x55b24dac70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f896755c740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55b24dac0540 con 0x55b24d6dc280 2024-07-26T06:49:36.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f896755c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b24dabfe70 con 0x55b24d6bb8e0 2024-07-26T06:49:36.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f896755c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55b24dac0c10 con 0x55b24d6d40a0 2024-07-26T06:49:36.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f896755c740 10 client.0 _ll_register_callbacks cb 0x7f895400fdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:49:36.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f895ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6dc280 0x55b24dac70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:36.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f895ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6dc280 0x55b24dac70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41228/0 (socket says 172.21.15.42:41228) 2024-07-26T06:49:36.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f895ad9d640 1 -- 172.21.15.42:0/1040607267 learned_addr learned my addr 172.21.15.42:0/1040607267 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:36.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f8959d9b640 1 --2- 172.21.15.42:0/1040607267 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6d40a0 0x55b24dac2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:36.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f895ad9d640 1 -- 172.21.15.42:0/1040607267 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6d40a0 msgr2=0x55b24dac2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:36.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f895ad9d640 1 --2- 172.21.15.42:0/1040607267 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6d40a0 0x55b24dac2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f895ad9d640 1 -- 172.21.15.42:0/1040607267 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6bb8e0 msgr2=0x55b24dac1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:49:36.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f895ad9d640 1 --2- 172.21.15.42:0/1040607267 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6bb8e0 0x55b24dac1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f895ad9d640 1 -- 172.21.15.42:0/1040607267 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b24dac7730 con 0x55b24d6dc280 2024-07-26T06:49:36.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f895ad9d640 1 --2- 172.21.15.42:0/1040607267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6dc280 0x55b24dac70c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f894c044000 tx=0x7f894c009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d22cf9d5c4e4da0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:36.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.932+0000 7f8941ffb640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:49:36.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.933+0000 7f8941ffb640 1 -- 172.21.15.42:0/1040607267 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c052060 con 0x55b24d6dc280 2024-07-26T06:49:36.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.933+0000 7f896755c740 1 -- 172.21.15.42:0/1040607267 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89440e6e50 con 0x55b24d6dc280 2024-07-26T06:49:36.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.933+0000 7f8941ffb640 1 -- 172.21.15.42:0/1040607267 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f894c031d00 con 0x55b24d6dc280 2024-07-26T06:49:36.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.933+0000 7f8941ffb640 1 -- 172.21.15.42:0/1040607267 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f894c031450 con 0x55b24d6dc280 2024-07-26T06:49:36.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.933+0000 7f896755c740 10 client.15150 fetch_fsmap learned FSMap version 28 2024-07-26T06:49:36.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.933+0000 7f896755c740 1 -- 172.21.15.42:0/1040607267 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b24dac8a60 con 0x55b24d6dc280 2024-07-26T06:49:36.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.933+0000 7f8941ffb640 1 -- 172.21.15.42:0/1040607267 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89440e6e50 con 0x55b24d6dc280 2024-07-26T06:49:36.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.933+0000 7f8941ffb640 1 -- 172.21.15.42:0/1040607267 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f894c004160 con 0x55b24d6dc280 2024-07-26T06:49:36.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.933+0000 7f896755c740 10 client.15150 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:49:36.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.933+0000 7f896755c740 10 client.15150 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T06:49:36.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.933+0000 7f896755c740 20 client.15150 populate_metadata read hostname 'smithi042' 2024-07-26T06:49:36.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.934+0000 7f896755c740 1 --2- 172.21.15.42:0/1040607267 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55b24d6fcf60 0x55b24db348c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:36.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.934+0000 7f896755c740 4 client.15150 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:49:36.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.934+0000 7f896755c740 1 -- 172.21.15.42:0/1040607267 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "status"}) -- 0x55b24da16330 con 0x55b24d6fcf60 2024-07-26T06:49:36.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.934+0000 7f895a59c640 1 --2- 172.21.15.42:0/1040607267 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55b24d6fcf60 0x55b24db348c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:36.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.934+0000 7f895a59c640 1 --2- 172.21.15.42:0/1040607267 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55b24d6fcf60 0x55b24db348c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa76cc1530c9d1c4 server_cookie=7f7a9c44176a044f in_seq=0 out_seq=0 2024-07-26T06:49:36.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.934+0000 7f8941ffb640 10 client.15150 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:49:36.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.935+0000 7f8941ffb640 1 -- 172.21.15.42:0/1040607267 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b24da16330 con 0x55b24d6fcf60 2024-07-26T06:49:36.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.935+0000 7f8941ffb640 10 client.15150 handle_command_reply: tid=0 2024-07-26T06:49:36.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.935+0000 7f896755c740 1 client.15150 shutdown 2024-07-26T06:49:36.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.935+0000 7f896755c740 1 -- 172.21.15.42:0/1040607267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6dc280 msgr2=0x55b24dac70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:36.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.935+0000 7f896755c740 1 --2- 172.21.15.42:0/1040607267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6dc280 0x55b24dac70c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f894c044000 tx=0x7f894c009540 comp rx=0 tx=0).stop 2024-07-26T06:49:36.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.935+0000 7f896755c740 1 -- 172.21.15.42:0/1040607267 shutdown_connections 2024-07-26T06:49:36.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.935+0000 7f896755c740 1 --2- 172.21.15.42:0/1040607267 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55b24d6fcf60 0x55b24db348c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.935+0000 7f896755c740 1 --2- 172.21.15.42:0/1040607267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b24d6dc280 0x55b24dac70c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.935+0000 7f896755c740 1 --2- 172.21.15.42:0/1040607267 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b24d6d40a0 0x55b24dac2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.935+0000 7f8941ffb640 0 client.15150 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:49:36.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.935+0000 7f896755c740 1 --2- 172.21.15.42:0/1040607267 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b24d6bb8e0 0x55b24dac1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.935+0000 7f896755c740 1 -- 172.21.15.42:0/1040607267 >> 172.21.15.42:0/1040607267 conn(0x55b24d6f1350 msgr2=0x55b24daac4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:36.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.935+0000 7f896755c740 1 -- 172.21.15.42:0/1040607267 shutdown_connections 2024-07-26T06:49:36.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.935+0000 7f896755c740 1 -- 172.21.15.42:0/1040607267 wait complete. 2024-07-26T06:49:36.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.935+0000 7f896755c740 20 client.15150 trim_cache size 0 max 16384 2024-07-26T06:49:36.941 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:49:36.941 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_fsid": "8b3e2750-4b19-11ef-bcb6-c7b262605968", 2024-07-26T06:49:36.941 INFO:teuthology.orchestra.run.smithi042.stdout: "whoami": 0, 2024-07-26T06:49:36.941 INFO:teuthology.orchestra.run.smithi042.stdout: "id": 24620, 2024-07-26T06:49:36.941 INFO:teuthology.orchestra.run.smithi042.stdout: "want_state": "up:active", 2024-07-26T06:49:36.941 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "up:active", 2024-07-26T06:49:36.941 INFO:teuthology.orchestra.run.smithi042.stdout: "fs_name": "cephfs", 2024-07-26T06:49:36.942 INFO:teuthology.orchestra.run.smithi042.stdout: "rank_uptime": 149.92054181099999, 2024-07-26T06:49:36.942 INFO:teuthology.orchestra.run.smithi042.stdout: "mdsmap_epoch": 28, 2024-07-26T06:49:36.942 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch": 80, 2024-07-26T06:49:36.942 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch_barrier": 80, 2024-07-26T06:49:36.942 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 166.47817006400001 2024-07-26T06:49:36.942 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:49:36.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.936+0000 7f8960fd8640 1 -- 172.21.15.42:0/3321065073 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f89480b3bd0 msgr2=0x7f89480b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:36.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.936+0000 7f8960fd8640 1 --2- 172.21.15.42:0/3321065073 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f89480b3bd0 0x7f89480b60e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x55b24daa1020 tx=0x7f8954008040 comp rx=0 tx=0).stop 2024-07-26T06:49:36.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.936+0000 7f8960fd8640 1 -- 172.21.15.42:0/3321065073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f895c107dd0 msgr2=0x7f895c105050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:36.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.936+0000 7f8960fd8640 1 --2- 172.21.15.42:0/3321065073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f895c107dd0 0x7f895c105050 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f894400d770 tx=0x7f894400dc30 comp rx=0 tx=0).stop 2024-07-26T06:49:36.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.936+0000 7f8960fd8640 1 -- 172.21.15.42:0/3321065073 shutdown_connections 2024-07-26T06:49:36.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.936+0000 7f8960fd8640 1 --2- 172.21.15.42:0/3321065073 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f89480b3bd0 0x7f89480b60e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.936+0000 7f8960fd8640 1 --2- 172.21.15.42:0/3321065073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f895c108720 0x7f895c105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.936+0000 7f8960fd8640 1 --2- 172.21.15.42:0/3321065073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f895c107dd0 0x7f895c105050 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.936+0000 7f8960fd8640 1 --2- 172.21.15.42:0/3321065073 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f895c102490 0x7f895c104b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:36.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.936+0000 7f8960fd8640 1 -- 172.21.15.42:0/3321065073 >> 172.21.15.42:0/3321065073 conn(0x7f895c079160 msgr2=0x7f895c06abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:36.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.936+0000 7f8960fd8640 1 -- 172.21.15.42:0/3321065073 shutdown_connections 2024-07-26T06:49:36.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:36.936+0000 7f8960fd8640 1 -- 172.21.15.42:0/3321065073 wait complete. 2024-07-26T06:49:37.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:36 smithi163 ceph-mon[36873]: pgmap v377: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:49:37.337 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:36 smithi042 ceph-mon[32775]: pgmap v377: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:49:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:36 smithi164 ceph-mon[36970]: pgmap v377: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:49:37.617 INFO:tasks.cephadm:Setup complete, yielding 2024-07-26T06:49:37.617 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-07-26T06:49:37.629 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi042.front.sepia.ceph.com 2024-07-26T06:49:37.629 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- bash -c 'ceph orch status' 2024-07-26T06:49:37.897 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:39.097 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:39 smithi042 ceph-mon[32775]: pgmap v378: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:39 smithi163 ceph-mon[36873]: pgmap v378: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:39.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:39 smithi164 ceph-mon[36970]: pgmap v378: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:39.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.635+0000 7fbe1eefb640 1 -- 172.21.15.42:0/3891778399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe18104bb0 msgr2=0x7fbe1810b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:39.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.635+0000 7fbe1eefb640 1 --2- 172.21.15.42:0/3891778399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe18104bb0 0x7fbe1810b550 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbe10009ca0 tx=0x7fbe1002a950 comp rx=0 tx=0).stop 2024-07-26T06:49:39.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.635+0000 7fbe1eefb640 1 -- 172.21.15.42:0/3891778399 shutdown_connections 2024-07-26T06:49:39.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.635+0000 7fbe1eefb640 1 --2- 172.21.15.42:0/3891778399 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe18104bb0 0x7fbe1810b550 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:39.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.635+0000 7fbe1eefb640 1 --2- 172.21.15.42:0/3891778399 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe18104210 0x7fbe18104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:39.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.635+0000 7fbe1eefb640 1 --2- 172.21.15.42:0/3891778399 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe18103180 0x7fbe181035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:39.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.635+0000 7fbe1eefb640 1 -- 172.21.15.42:0/3891778399 >> 172.21.15.42:0/3891778399 conn(0x7fbe180feaf0 msgr2=0x7fbe18100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:39.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.635+0000 7fbe1eefb640 1 -- 172.21.15.42:0/3891778399 shutdown_connections 2024-07-26T06:49:39.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.635+0000 7fbe1eefb640 1 -- 172.21.15.42:0/3891778399 wait complete. 2024-07-26T06:49:39.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.636+0000 7fbe1eefb640 1 Processor -- start 2024-07-26T06:49:39.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.636+0000 7fbe1eefb640 1 -- start start 2024-07-26T06:49:39.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.636+0000 7fbe1eefb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe18103180 0x7fbe18199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:39.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.636+0000 7fbe1eefb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe18104210 0x7fbe1819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:39.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.636+0000 7fbe1eefb640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe18104bb0 0x7fbe1819acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:39.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.636+0000 7fbe1eefb640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1810eea0 con 0x7fbe18104210 2024-07-26T06:49:39.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.636+0000 7fbe1eefb640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1810ed20 con 0x7fbe18104bb0 2024-07-26T06:49:39.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.636+0000 7fbe1eefb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1810f020 con 0x7fbe18103180 2024-07-26T06:49:39.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.636+0000 7fbe17fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe18104210 0x7fbe1819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:39.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.636+0000 7fbe17fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe18104210 0x7fbe1819a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41246/0 (socket says 172.21.15.42:41246) 2024-07-26T06:49:39.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.636+0000 7fbe17fff640 1 -- 172.21.15.42:0/2133411058 learned_addr learned my addr 172.21.15.42:0/2133411058 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:39.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.637+0000 7fbe1cc97640 1 --2- 172.21.15.42:0/2133411058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe18103180 0x7fbe18199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:39.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.637+0000 7fbe1d498640 1 --2- 172.21.15.42:0/2133411058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe18104bb0 0x7fbe1819acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:39.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.637+0000 7fbe17fff640 1 -- 172.21.15.42:0/2133411058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe18103180 msgr2=0x7fbe18199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:39.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.637+0000 7fbe17fff640 1 --2- 172.21.15.42:0/2133411058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe18103180 0x7fbe18199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:39.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.637+0000 7fbe17fff640 1 -- 172.21.15.42:0/2133411058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe18104bb0 msgr2=0x7fbe1819acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:39.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.637+0000 7fbe17fff640 1 --2- 172.21.15.42:0/2133411058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe18104bb0 0x7fbe1819acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:39.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.637+0000 7fbe17fff640 1 -- 172.21.15.42:0/2133411058 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe1819b4e0 con 0x7fbe18104210 2024-07-26T06:49:39.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.637+0000 7fbe1d498640 1 --2- 172.21.15.42:0/2133411058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe18104bb0 0x7fbe1819acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:49:39.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.637+0000 7fbe17fff640 1 --2- 172.21.15.42:0/2133411058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe18104210 0x7fbe1819a120 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00b4a0 tx=0x7fbe0c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:39.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.637+0000 7fbe15ffb640 1 -- 172.21.15.42:0/2133411058 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c015020 con 0x7fbe18104210 2024-07-26T06:49:39.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.637+0000 7fbe1eefb640 1 -- 172.21.15.42:0/2133411058 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe1819b7d0 con 0x7fbe18104210 2024-07-26T06:49:39.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.637+0000 7fbe15ffb640 1 -- 172.21.15.42:0/2133411058 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe0c011ce0 con 0x7fbe18104210 2024-07-26T06:49:39.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.637+0000 7fbe15ffb640 1 -- 172.21.15.42:0/2133411058 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c010dc0 con 0x7fbe18104210 2024-07-26T06:49:39.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.637+0000 7fbe1eefb640 1 -- 172.21.15.42:0/2133411058 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe181a1ed0 con 0x7fbe18104210 2024-07-26T06:49:39.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.638+0000 7fbe1eefb640 1 -- 172.21.15.42:0/2133411058 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe1810a360 con 0x7fbe18104210 2024-07-26T06:49:39.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.640+0000 7fbe15ffb640 1 -- 172.21.15.42:0/2133411058 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fbe0c018020 con 0x7fbe18104210 2024-07-26T06:49:39.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.641+0000 7fbe15ffb640 1 --2- 172.21.15.42:0/2133411058 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbe000b3bd0 0x7fbe000b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:39.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.641+0000 7fbe1cc97640 1 --2- 172.21.15.42:0/2133411058 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbe000b3bd0 0x7fbe000b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:39.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.641+0000 7fbe15ffb640 1 -- 172.21.15.42:0/2133411058 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fbe0c0e3540 con 0x7fbe18104210 2024-07-26T06:49:39.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.642+0000 7fbe1cc97640 1 --2- 172.21.15.42:0/2133411058 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbe000b3bd0 0x7fbe000b60e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbe08009990 tx=0x7fbe08009110 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:39.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.646+0000 7fbe15ffb640 1 -- 172.21.15.42:0/2133411058 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbe0c01f020 con 0x7fbe18104210 2024-07-26T06:49:39.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.845+0000 7fbe1eefb640 1 -- 172.21.15.42:0/2133411058 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fbe18063230 con 0x7fbe000b3bd0 2024-07-26T06:49:39.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.846+0000 7fbe15ffb640 1 -- 172.21.15.42:0/2133411058 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fbe18063230 con 0x7fbe000b3bd0 2024-07-26T06:49:39.846 INFO:teuthology.orchestra.run.smithi042.stdout:Backend: cephadm 2024-07-26T06:49:39.846 INFO:teuthology.orchestra.run.smithi042.stdout:Available: Yes 2024-07-26T06:49:39.847 INFO:teuthology.orchestra.run.smithi042.stdout:Paused: No 2024-07-26T06:49:39.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.848+0000 7fbe1eefb640 1 -- 172.21.15.42:0/2133411058 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbe000b3bd0 msgr2=0x7fbe000b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:39.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.848+0000 7fbe1eefb640 1 --2- 172.21.15.42:0/2133411058 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbe000b3bd0 0x7fbe000b60e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbe08009990 tx=0x7fbe08009110 comp rx=0 tx=0).stop 2024-07-26T06:49:39.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.848+0000 7fbe1eefb640 1 -- 172.21.15.42:0/2133411058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe18104210 msgr2=0x7fbe1819a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:39.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.848+0000 7fbe1eefb640 1 --2- 172.21.15.42:0/2133411058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe18104210 0x7fbe1819a120 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00b4a0 tx=0x7fbe0c00b8a0 comp rx=0 tx=0).stop 2024-07-26T06:49:39.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.848+0000 7fbe1eefb640 1 -- 172.21.15.42:0/2133411058 shutdown_connections 2024-07-26T06:49:39.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.848+0000 7fbe1eefb640 1 --2- 172.21.15.42:0/2133411058 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbe000b3bd0 0x7fbe000b60e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:39.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.848+0000 7fbe1eefb640 1 --2- 172.21.15.42:0/2133411058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe18104bb0 0x7fbe1819acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:39.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.848+0000 7fbe1eefb640 1 --2- 172.21.15.42:0/2133411058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe18104210 0x7fbe1819a120 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:39.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.848+0000 7fbe1eefb640 1 --2- 172.21.15.42:0/2133411058 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbe18103180 0x7fbe18199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:39.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.848+0000 7fbe1eefb640 1 -- 172.21.15.42:0/2133411058 >> 172.21.15.42:0/2133411058 conn(0x7fbe180feaf0 msgr2=0x7fbe18107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:39.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.848+0000 7fbe1eefb640 1 -- 172.21.15.42:0/2133411058 shutdown_connections 2024-07-26T06:49:39.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:39.848+0000 7fbe1eefb640 1 -- 172.21.15.42:0/2133411058 wait complete. 2024-07-26T06:49:40.534 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- bash -c 'ceph orch ps' 2024-07-26T06:49:40.809 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:41.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:41 smithi163 ceph-mon[36873]: pgmap v379: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:49:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:41 smithi163 ceph-mon[36873]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:49:41.268 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:41 smithi042 ceph-mon[32775]: pgmap v379: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:49:41.268 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:41 smithi042 ceph-mon[32775]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:49:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:41 smithi164 ceph-mon[36970]: pgmap v379: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:49:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:41 smithi164 ceph-mon[36970]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:49:42.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.477+0000 7fe5f37ad640 1 -- 172.21.15.42:0/1407050217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe5ec06bec0 msgr2=0x7fe5ec1081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:42.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.477+0000 7fe5f37ad640 1 --2- 172.21.15.42:0/1407050217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe5ec06bec0 0x7fe5ec1081f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009900 tx=0x7fe5e802a410 comp rx=0 tx=0).stop 2024-07-26T06:49:42.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.477+0000 7fe5f37ad640 1 -- 172.21.15.42:0/1407050217 shutdown_connections 2024-07-26T06:49:42.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.477+0000 7fe5f37ad640 1 --2- 172.21.15.42:0/1407050217 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5ec10b0e0 0x7fe5ec10d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:42.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.477+0000 7fe5f37ad640 1 --2- 172.21.15.42:0/1407050217 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5ec108730 0x7fe5ec10ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:42.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.477+0000 7fe5f37ad640 1 --2- 172.21.15.42:0/1407050217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe5ec06bec0 0x7fe5ec1081f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:42.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.477+0000 7fe5f37ad640 1 -- 172.21.15.42:0/1407050217 >> 172.21.15.42:0/1407050217 conn(0x7fe5ec0fda10 msgr2=0x7fe5ec0ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:42.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.477+0000 7fe5f37ad640 1 -- 172.21.15.42:0/1407050217 shutdown_connections 2024-07-26T06:49:42.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.477+0000 7fe5f37ad640 1 -- 172.21.15.42:0/1407050217 wait complete. 2024-07-26T06:49:42.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f37ad640 1 Processor -- start 2024-07-26T06:49:42.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f37ad640 1 -- start start 2024-07-26T06:49:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f37ad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5ec06bec0 0x7fe5ec19bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f37ad640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe5ec108730 0x7fe5ec19c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f37ad640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5ec10b0e0 0x7fe5ec19cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f37ad640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec111000 con 0x7fe5ec108730 2024-07-26T06:49:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f37ad640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec110e80 con 0x7fe5ec10b0e0 2024-07-26T06:49:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f37ad640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec111180 con 0x7fe5ec06bec0 2024-07-26T06:49:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f0d48640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe5ec108730 0x7fe5ec19c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f0d48640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe5ec108730 0x7fe5ec19c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41268/0 (socket says 172.21.15.42:41268) 2024-07-26T06:49:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f0d48640 1 -- 172.21.15.42:0/4257175683 learned_addr learned my addr 172.21.15.42:0/4257175683 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f1549640 1 --2- 172.21.15.42:0/4257175683 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5ec06bec0 0x7fe5ec19bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f1d4a640 1 --2- 172.21.15.42:0/4257175683 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5ec10b0e0 0x7fe5ec19cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:42.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f0d48640 1 -- 172.21.15.42:0/4257175683 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5ec06bec0 msgr2=0x7fe5ec19bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:42.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f0d48640 1 --2- 172.21.15.42:0/4257175683 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5ec06bec0 0x7fe5ec19bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:42.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f0d48640 1 -- 172.21.15.42:0/4257175683 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5ec10b0e0 msgr2=0x7fe5ec19cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:42.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f0d48640 1 --2- 172.21.15.42:0/4257175683 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5ec10b0e0 0x7fe5ec19cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:42.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.478+0000 7fe5f0d48640 1 -- 172.21.15.42:0/4257175683 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5ec19d580 con 0x7fe5ec108730 2024-07-26T06:49:42.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.479+0000 7fe5f0d48640 1 --2- 172.21.15.42:0/4257175683 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe5ec108730 0x7fe5ec19c250 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc00c920 tx=0x7fe5dc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:42.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.479+0000 7fe5e27fc640 1 -- 172.21.15.42:0/4257175683 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc007c70 con 0x7fe5ec108730 2024-07-26T06:49:42.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.479+0000 7fe5f37ad640 1 -- 172.21.15.42:0/4257175683 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5ec19d870 con 0x7fe5ec108730 2024-07-26T06:49:42.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.479+0000 7fe5e27fc640 1 -- 172.21.15.42:0/4257175683 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5dc007e10 con 0x7fe5ec108730 2024-07-26T06:49:42.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.479+0000 7fe5e27fc640 1 -- 172.21.15.42:0/4257175683 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc018820 con 0x7fe5ec108730 2024-07-26T06:49:42.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.479+0000 7fe5f37ad640 1 -- 172.21.15.42:0/4257175683 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5ec1a4000 con 0x7fe5ec108730 2024-07-26T06:49:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.480+0000 7fe5f37ad640 1 -- 172.21.15.42:0/4257175683 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5ec0ff530 con 0x7fe5ec108730 2024-07-26T06:49:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.482+0000 7fe5e27fc640 1 -- 172.21.15.42:0/4257175683 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fe5dc00f450 con 0x7fe5ec108730 2024-07-26T06:49:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.482+0000 7fe5e27fc640 1 --2- 172.21.15.42:0/4257175683 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe5d80b3bd0 0x7fe5d80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:42.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.483+0000 7fe5f1549640 1 --2- 172.21.15.42:0/4257175683 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe5d80b3bd0 0x7fe5d80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:42.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.483+0000 7fe5e27fc640 1 -- 172.21.15.42:0/4257175683 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fe5dc0e3e10 con 0x7fe5ec108730 2024-07-26T06:49:42.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.483+0000 7fe5f1549640 1 --2- 172.21.15.42:0/4257175683 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe5d80b3bd0 0x7fe5d80b60e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8031440 tx=0x7fe5e80314a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.487+0000 7fe5e27fc640 1 -- 172.21.15.42:0/4257175683 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe5dc014030 con 0x7fe5ec108730 2024-07-26T06:49:42.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.676+0000 7fe5f37ad640 1 -- 172.21.15.42:0/4257175683 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fe5ec063230 con 0x7fe5d80b3bd0 2024-07-26T06:49:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.688+0000 7fe5e27fc640 1 -- 172.21.15.42:0/4257175683 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fe5ec063230 con 0x7fe5d80b3bd0 2024-07-26T06:49:42.689 INFO:teuthology.orchestra.run.smithi042.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-07-26T06:49:42.689 INFO:teuthology.orchestra.run.smithi042.stdout:mds.a smithi042 running (2m) 36s ago 2m 87.6M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 84e1094b936b 2024-07-26T06:49:42.689 INFO:teuthology.orchestra.run.smithi042.stdout:mds.b smithi163 running (2m) 39s ago 2m 95.4M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 464ee1df1f53 2024-07-26T06:49:42.689 INFO:teuthology.orchestra.run.smithi042.stdout:mds.c smithi164 running (3m) 39s ago 3m 87.5M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 016ec20fcc4a 2024-07-26T06:49:42.689 INFO:teuthology.orchestra.run.smithi042.stdout:mds.d smithi042 running (3m) 36s ago 3m 82.4M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 4e393b9ca409 2024-07-26T06:49:42.689 INFO:teuthology.orchestra.run.smithi042.stdout:mds.e smithi163 running (3m) 39s ago 3m 87.5M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa c47b4288f073 2024-07-26T06:49:42.689 INFO:teuthology.orchestra.run.smithi042.stdout:mds.f smithi164 running (3m) 39s ago 3m 88.3M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 20df7fdbf85a 2024-07-26T06:49:42.690 INFO:teuthology.orchestra.run.smithi042.stdout:mds.g smithi042 running (2m) 36s ago 2m 87.7M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 3e3da20f6e4d 2024-07-26T06:49:42.690 INFO:teuthology.orchestra.run.smithi042.stdout:mds.h smithi163 running (3m) 39s ago 3m 87.5M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa eff753a144e5 2024-07-26T06:49:42.690 INFO:teuthology.orchestra.run.smithi042.stdout:mds.i smithi164 running (3m) 39s ago 3m 87.3M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 40b3fac68d8c 2024-07-26T06:49:42.690 INFO:teuthology.orchestra.run.smithi042.stdout:mds.j smithi042 running (3m) 36s ago 3m 82.2M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 68440941ad33 2024-07-26T06:49:42.690 INFO:teuthology.orchestra.run.smithi042.stdout:mds.k smithi163 running (3m) 39s ago 3m 87.8M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 2488c0aef30a 2024-07-26T06:49:42.690 INFO:teuthology.orchestra.run.smithi042.stdout:mds.l smithi164 running (3m) 39s ago 3m 87.8M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa e9a5e5169b93 2024-07-26T06:49:42.690 INFO:teuthology.orchestra.run.smithi042.stdout:mgr.x smithi042 *:9283,8765 running (11m) 36s ago 11m 590M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 3be6bbb38ec5 2024-07-26T06:49:42.690 INFO:teuthology.orchestra.run.smithi042.stdout:mgr.y smithi163 *:8443,8765 running (9m) 39s ago 9m 523M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa b8a91b052f0c 2024-07-26T06:49:42.690 INFO:teuthology.orchestra.run.smithi042.stdout:mgr.z smithi164 *:8443,8765 running (9m) 39s ago 9m 522M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa d1a92d8c9a1a 2024-07-26T06:49:42.690 INFO:teuthology.orchestra.run.smithi042.stdout:mon.a smithi042 running (11m) 36s ago 11m 121M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 33bc7f46d883 2024-07-26T06:49:42.690 INFO:teuthology.orchestra.run.smithi042.stdout:mon.b smithi163 running (9m) 39s ago 9m 110M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 99821459e73a 2024-07-26T06:49:42.690 INFO:teuthology.orchestra.run.smithi042.stdout:mon.c smithi164 running (9m) 39s ago 9m 109M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa e19612f16f40 2024-07-26T06:49:42.691 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0 smithi042 running (8m) 36s ago 8m 114M 4273M 19.0.0-5320-ge8f88bb1 ae51f432d4fa e60c456ef988 2024-07-26T06:49:42.691 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1 smithi163 running (7m) 39s ago 7m 126M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa c67dcae16d32 2024-07-26T06:49:42.691 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2 smithi164 running (7m) 39s ago 7m 125M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa ec00f9a99465 2024-07-26T06:49:42.691 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3 smithi042 running (7m) 36s ago 7m 121M 4273M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 83ae64cee350 2024-07-26T06:49:42.691 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4 smithi163 running (6m) 39s ago 6m 122M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 4545cb4e6004 2024-07-26T06:49:42.691 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5 smithi164 running (6m) 39s ago 6m 112M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 089f413be0ee 2024-07-26T06:49:42.691 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6 smithi042 running (5m) 36s ago 5m 121M 4273M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 27a370104450 2024-07-26T06:49:42.691 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7 smithi163 running (5m) 39s ago 5m 124M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 067a98f0ba0b 2024-07-26T06:49:42.691 INFO:teuthology.orchestra.run.smithi042.stdout:osd.8 smithi164 running (5m) 39s ago 5m 123M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 139afb9bbdd6 2024-07-26T06:49:42.691 INFO:teuthology.orchestra.run.smithi042.stdout:osd.9 smithi042 running (4m) 36s ago 4m 109M 4273M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 39203993dcea 2024-07-26T06:49:42.692 INFO:teuthology.orchestra.run.smithi042.stdout:osd.10 smithi163 running (4m) 39s ago 4m 112M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 15022d57c12a 2024-07-26T06:49:42.692 INFO:teuthology.orchestra.run.smithi042.stdout:osd.11 smithi164 running (3m) 39s ago 3m 113M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa cdfd3369a83b 2024-07-26T06:49:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.690+0000 7fe5f37ad640 1 -- 172.21.15.42:0/4257175683 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe5d80b3bd0 msgr2=0x7fe5d80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.690+0000 7fe5f37ad640 1 --2- 172.21.15.42:0/4257175683 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe5d80b3bd0 0x7fe5d80b60e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8031440 tx=0x7fe5e80314a0 comp rx=0 tx=0).stop 2024-07-26T06:49:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.690+0000 7fe5f37ad640 1 -- 172.21.15.42:0/4257175683 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe5ec108730 msgr2=0x7fe5ec19c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.690+0000 7fe5f37ad640 1 --2- 172.21.15.42:0/4257175683 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe5ec108730 0x7fe5ec19c250 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc00c920 tx=0x7fe5dc00cde0 comp rx=0 tx=0).stop 2024-07-26T06:49:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.690+0000 7fe5f37ad640 1 -- 172.21.15.42:0/4257175683 shutdown_connections 2024-07-26T06:49:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.690+0000 7fe5f37ad640 1 --2- 172.21.15.42:0/4257175683 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe5d80b3bd0 0x7fe5d80b60e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.690+0000 7fe5f37ad640 1 --2- 172.21.15.42:0/4257175683 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe5ec10b0e0 0x7fe5ec19cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.690+0000 7fe5f37ad640 1 --2- 172.21.15.42:0/4257175683 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe5ec108730 0x7fe5ec19c250 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.690+0000 7fe5f37ad640 1 --2- 172.21.15.42:0/4257175683 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe5ec06bec0 0x7fe5ec19bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.690+0000 7fe5f37ad640 1 -- 172.21.15.42:0/4257175683 >> 172.21.15.42:0/4257175683 conn(0x7fe5ec0fda10 msgr2=0x7fe5ec1095e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.690+0000 7fe5f37ad640 1 -- 172.21.15.42:0/4257175683 shutdown_connections 2024-07-26T06:49:42.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:42.691+0000 7fe5f37ad640 1 -- 172.21.15.42:0/4257175683 wait complete. 2024-07-26T06:49:43.317 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:43 smithi042 ceph-mon[32775]: pgmap v380: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:49:43.348 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- bash -c 'ceph orch ls' 2024-07-26T06:49:43.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:43 smithi164 ceph-mon[36970]: pgmap v380: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:49:43.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:43 smithi163 ceph-mon[36873]: pgmap v380: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:49:43.621 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:44.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:44 smithi042 ceph-mon[32775]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:49:44.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:44 smithi164 ceph-mon[36970]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:49:44.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:44 smithi163 ceph-mon[36873]: from='client.15162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:49:45.142 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:45 smithi042 ceph-mon[32775]: pgmap v381: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.365+0000 7fdc9d1d8640 1 -- 172.21.15.42:0/1489577102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc98103ac0 msgr2=0x7fdc9810a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.365+0000 7fdc9d1d8640 1 --2- 172.21.15.42:0/1489577102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc98103ac0 0x7fdc9810a460 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fdc90009ca0 tx=0x7fdc9002a950 comp rx=0 tx=0).stop 2024-07-26T06:49:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.365+0000 7fdc9d1d8640 1 -- 172.21.15.42:0/1489577102 shutdown_connections 2024-07-26T06:49:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.365+0000 7fdc9d1d8640 1 --2- 172.21.15.42:0/1489577102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc98103ac0 0x7fdc9810a460 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.365+0000 7fdc9d1d8640 1 --2- 172.21.15.42:0/1489577102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc98103120 0x7fdc98103530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.365+0000 7fdc9d1d8640 1 --2- 172.21.15.42:0/1489577102 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc98102090 0x7fdc98102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.365+0000 7fdc9d1d8640 1 -- 172.21.15.42:0/1489577102 >> 172.21.15.42:0/1489577102 conn(0x7fdc980fda00 msgr2=0x7fdc980ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.366+0000 7fdc9d1d8640 1 -- 172.21.15.42:0/1489577102 shutdown_connections 2024-07-26T06:49:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.366+0000 7fdc9d1d8640 1 -- 172.21.15.42:0/1489577102 wait complete. 2024-07-26T06:49:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.366+0000 7fdc9d1d8640 1 Processor -- start 2024-07-26T06:49:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.367+0000 7fdc9d1d8640 1 -- start start 2024-07-26T06:49:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.367+0000 7fdc9d1d8640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc98102090 0x7fdc9806bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.367+0000 7fdc9d1d8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc98103120 0x7fdc9819e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.367+0000 7fdc9d1d8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc98103ac0 0x7fdc981a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.367+0000 7fdc9d1d8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9810dcf0 con 0x7fdc98103ac0 2024-07-26T06:49:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.367+0000 7fdc9d1d8640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9810db70 con 0x7fdc98102090 2024-07-26T06:49:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.367+0000 7fdc9d1d8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9810de70 con 0x7fdc98103120 2024-07-26T06:49:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.367+0000 7fdc9759e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc98103ac0 0x7fdc981a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.367+0000 7fdc9759e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc98103ac0 0x7fdc981a0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41296/0 (socket says 172.21.15.42:41296) 2024-07-26T06:49:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.367+0000 7fdc9759e640 1 -- 172.21.15.42:0/1770037432 learned_addr learned my addr 172.21.15.42:0/1770037432 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.367+0000 7fdc96d9d640 1 --2- 172.21.15.42:0/1770037432 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc98102090 0x7fdc9806bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.367+0000 7fdc9659c640 1 --2- 172.21.15.42:0/1770037432 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc98103120 0x7fdc9819e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.367+0000 7fdc9759e640 1 -- 172.21.15.42:0/1770037432 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc98103120 msgr2=0x7fdc9819e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.367+0000 7fdc9759e640 1 --2- 172.21.15.42:0/1770037432 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc98103120 0x7fdc9819e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.367+0000 7fdc9759e640 1 -- 172.21.15.42:0/1770037432 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc98102090 msgr2=0x7fdc9806bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.367+0000 7fdc9759e640 1 --2- 172.21.15.42:0/1770037432 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc98102090 0x7fdc9806bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.367+0000 7fdc9759e640 1 -- 172.21.15.42:0/1770037432 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc981a14b0 con 0x7fdc98103ac0 2024-07-26T06:49:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.368+0000 7fdc9759e640 1 --2- 172.21.15.42:0/1770037432 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc98103ac0 0x7fdc981a0cb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fdc90006010 tx=0x7fdc90031d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.368+0000 7fdc7ffff640 1 -- 172.21.15.42:0/1770037432 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc90034040 con 0x7fdc98103ac0 2024-07-26T06:49:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.368+0000 7fdc7ffff640 1 -- 172.21.15.42:0/1770037432 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc900308f0 con 0x7fdc98103ac0 2024-07-26T06:49:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.368+0000 7fdc7ffff640 1 -- 172.21.15.42:0/1770037432 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc900047f0 con 0x7fdc98103ac0 2024-07-26T06:49:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.368+0000 7fdc9d1d8640 1 -- 172.21.15.42:0/1770037432 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc981a1740 con 0x7fdc98103ac0 2024-07-26T06:49:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.368+0000 7fdc9d1d8640 1 -- 172.21.15.42:0/1770037432 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc981a1bd0 con 0x7fdc98103ac0 2024-07-26T06:49:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.369+0000 7fdc9d1d8640 1 -- 172.21.15.42:0/1770037432 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc98109270 con 0x7fdc98103ac0 2024-07-26T06:49:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.371+0000 7fdc7ffff640 1 -- 172.21.15.42:0/1770037432 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fdc90037020 con 0x7fdc98103ac0 2024-07-26T06:49:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.371+0000 7fdc7ffff640 1 --2- 172.21.15.42:0/1770037432 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdc800b3bd0 0x7fdc800b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.372+0000 7fdc96d9d640 1 --2- 172.21.15.42:0/1770037432 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdc800b3bd0 0x7fdc800b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.372+0000 7fdc7ffff640 1 -- 172.21.15.42:0/1770037432 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fdc90104c80 con 0x7fdc98103ac0 2024-07-26T06:49:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.372+0000 7fdc96d9d640 1 --2- 172.21.15.42:0/1770037432 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdc800b3bd0 0x7fdc800b60e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdc880058c0 tx=0x7fdc8800a2d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.375+0000 7fdc7ffff640 1 -- 172.21.15.42:0/1770037432 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdc900cc340 con 0x7fdc98103ac0 2024-07-26T06:49:45.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:45 smithi164 ceph-mon[36970]: pgmap v381: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:45.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:45 smithi163 ceph-mon[36873]: pgmap v381: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.569+0000 7fdc9d1d8640 1 -- 172.21.15.42:0/1770037432 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fdc98063230 con 0x7fdc800b3bd0 2024-07-26T06:49:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.575+0000 7fdc7ffff640 1 -- 172.21.15.42:0/1770037432 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fdc98063230 con 0x7fdc800b3bd0 2024-07-26T06:49:45.575 INFO:teuthology.orchestra.run.smithi042.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-07-26T06:49:45.575 INFO:teuthology.orchestra.run.smithi042.stdout:mds.all 12/12 42s ago 3m smithi042=a;smithi042=d;smithi042=g;smithi042=j;smithi163=b;smithi163=e;smithi163=h;smithi163=k;smithi164=c;smithi164=f;smithi164=i;smithi164=l;count:12 2024-07-26T06:49:45.575 INFO:teuthology.orchestra.run.smithi042.stdout:mgr 3/3 42s ago 9m smithi042=x;smithi163=y;smithi164=z;count:3 2024-07-26T06:49:45.575 INFO:teuthology.orchestra.run.smithi042.stdout:mon 3/3 42s ago 9m smithi042:172.21.15.42=a;smithi163:172.21.15.163=b;smithi164:172.21.15.164=c;count:3 2024-07-26T06:49:45.575 INFO:teuthology.orchestra.run.smithi042.stdout:osd 12 42s ago - 2024-07-26T06:49:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.577+0000 7fdc9d1d8640 1 -- 172.21.15.42:0/1770037432 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdc800b3bd0 msgr2=0x7fdc800b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.577+0000 7fdc9d1d8640 1 --2- 172.21.15.42:0/1770037432 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdc800b3bd0 0x7fdc800b60e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdc880058c0 tx=0x7fdc8800a2d0 comp rx=0 tx=0).stop 2024-07-26T06:49:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.577+0000 7fdc9d1d8640 1 -- 172.21.15.42:0/1770037432 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc98103ac0 msgr2=0x7fdc981a0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.577+0000 7fdc9d1d8640 1 --2- 172.21.15.42:0/1770037432 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc98103ac0 0x7fdc981a0cb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fdc90006010 tx=0x7fdc90031d30 comp rx=0 tx=0).stop 2024-07-26T06:49:45.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.577+0000 7fdc9d1d8640 1 -- 172.21.15.42:0/1770037432 shutdown_connections 2024-07-26T06:49:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.577+0000 7fdc9d1d8640 1 --2- 172.21.15.42:0/1770037432 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdc800b3bd0 0x7fdc800b60e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.577+0000 7fdc9d1d8640 1 --2- 172.21.15.42:0/1770037432 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdc98103ac0 0x7fdc981a0cb0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.577+0000 7fdc9d1d8640 1 --2- 172.21.15.42:0/1770037432 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdc98103120 0x7fdc9819e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.577+0000 7fdc9d1d8640 1 --2- 172.21.15.42:0/1770037432 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdc98102090 0x7fdc9806bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.577+0000 7fdc9d1d8640 1 -- 172.21.15.42:0/1770037432 >> 172.21.15.42:0/1770037432 conn(0x7fdc980fda00 msgr2=0x7fdc98106440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.577+0000 7fdc9d1d8640 1 -- 172.21.15.42:0/1770037432 shutdown_connections 2024-07-26T06:49:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:45.577+0000 7fdc9d1d8640 1 -- 172.21.15.42:0/1770037432 wait complete. 2024-07-26T06:49:46.400 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- bash -c 'ceph orch host ls' 2024-07-26T06:49:46.673 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:47.108 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:47 smithi042 ceph-mon[32775]: pgmap v382: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:49:47.109 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:47 smithi042 ceph-mon[32775]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:49:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:47 smithi164 ceph-mon[36970]: pgmap v382: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:49:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:47 smithi164 ceph-mon[36970]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:49:47.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:47 smithi163 ceph-mon[36873]: pgmap v382: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:49:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:47 smithi163 ceph-mon[36873]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:49:48.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.307+0000 7fdd97dda640 1 -- 172.21.15.42:0/2560731136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd90104bb0 msgr2=0x7fdd9010b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:48.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.307+0000 7fdd97dda640 1 --2- 172.21.15.42:0/2560731136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd90104bb0 0x7fdd9010b550 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fdd88009ca0 tx=0x7fdd8802a950 comp rx=0 tx=0).stop 2024-07-26T06:49:48.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.307+0000 7fdd97dda640 1 -- 172.21.15.42:0/2560731136 shutdown_connections 2024-07-26T06:49:48.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.307+0000 7fdd97dda640 1 --2- 172.21.15.42:0/2560731136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd90104bb0 0x7fdd9010b550 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:48.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.307+0000 7fdd97dda640 1 --2- 172.21.15.42:0/2560731136 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd90104210 0x7fdd90104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:48.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.307+0000 7fdd97dda640 1 --2- 172.21.15.42:0/2560731136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd90103180 0x7fdd901035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:48.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.307+0000 7fdd97dda640 1 -- 172.21.15.42:0/2560731136 >> 172.21.15.42:0/2560731136 conn(0x7fdd900feaf0 msgr2=0x7fdd90100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:48.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.307+0000 7fdd97dda640 1 -- 172.21.15.42:0/2560731136 shutdown_connections 2024-07-26T06:49:48.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.307+0000 7fdd97dda640 1 -- 172.21.15.42:0/2560731136 wait complete. 2024-07-26T06:49:48.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd97dda640 1 Processor -- start 2024-07-26T06:49:48.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd97dda640 1 -- start start 2024-07-26T06:49:48.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd97dda640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd90103180 0x7fdd90199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:48.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd97dda640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd90104210 0x7fdd90199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:48.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd97dda640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd90104bb0 0x7fdd9019ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:48.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd97dda640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9010ede0 con 0x7fdd90104bb0 2024-07-26T06:49:48.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd97dda640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9010ec60 con 0x7fdd90103180 2024-07-26T06:49:48.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd97dda640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9010ef60 con 0x7fdd90104210 2024-07-26T06:49:48.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd96377640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd90104bb0 0x7fdd9019ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd96377640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd90104bb0 0x7fdd9019ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44562/0 (socket says 172.21.15.42:44562) 2024-07-26T06:49:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd96377640 1 -- 172.21.15.42:0/3732441733 learned_addr learned my addr 172.21.15.42:0/3732441733 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd95b76640 1 --2- 172.21.15.42:0/3732441733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd90103180 0x7fdd90199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd95375640 1 --2- 172.21.15.42:0/3732441733 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd90104210 0x7fdd90199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd96377640 1 -- 172.21.15.42:0/3732441733 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd90104210 msgr2=0x7fdd90199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd96377640 1 --2- 172.21.15.42:0/3732441733 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd90104210 0x7fdd90199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd96377640 1 -- 172.21.15.42:0/3732441733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd90103180 msgr2=0x7fdd90199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd96377640 1 --2- 172.21.15.42:0/3732441733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd90103180 0x7fdd90199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.308+0000 7fdd96377640 1 -- 172.21.15.42:0/3732441733 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd9019b370 con 0x7fdd90104bb0 2024-07-26T06:49:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.309+0000 7fdd96377640 1 --2- 172.21.15.42:0/3732441733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd90104bb0 0x7fdd9019ab50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdd88006010 tx=0x7fdd8802fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.309+0000 7fdd86ffd640 1 -- 172.21.15.42:0/3732441733 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd88034040 con 0x7fdd90104bb0 2024-07-26T06:49:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.309+0000 7fdd86ffd640 1 -- 172.21.15.42:0/3732441733 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd880308f0 con 0x7fdd90104bb0 2024-07-26T06:49:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.309+0000 7fdd97dda640 1 -- 172.21.15.42:0/3732441733 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd9019b600 con 0x7fdd90104bb0 2024-07-26T06:49:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.309+0000 7fdd86ffd640 1 -- 172.21.15.42:0/3732441733 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd88039450 con 0x7fdd90104bb0 2024-07-26T06:49:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.309+0000 7fdd97dda640 1 -- 172.21.15.42:0/3732441733 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd901a1d60 con 0x7fdd90104bb0 2024-07-26T06:49:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.310+0000 7fdd97dda640 1 -- 172.21.15.42:0/3732441733 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd901035f0 con 0x7fdd90104bb0 2024-07-26T06:49:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.311+0000 7fdd86ffd640 1 -- 172.21.15.42:0/3732441733 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fdd88037020 con 0x7fdd90104bb0 2024-07-26T06:49:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.313+0000 7fdd86ffd640 1 --2- 172.21.15.42:0/3732441733 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdd7c0b3c50 0x7fdd7c0b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.313+0000 7fdd95b76640 1 --2- 172.21.15.42:0/3732441733 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdd7c0b3c50 0x7fdd7c0b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.313+0000 7fdd86ffd640 1 -- 172.21.15.42:0/3732441733 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fdd88104f70 con 0x7fdd90104bb0 2024-07-26T06:49:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.313+0000 7fdd95b76640 1 --2- 172.21.15.42:0/3732441733 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdd7c0b3c50 0x7fdd7c0b6160 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c005d00 tx=0x7fdd8c005c50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.316+0000 7fdd86ffd640 1 -- 172.21.15.42:0/3732441733 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdd880cc5a0 con 0x7fdd90104bb0 2024-07-26T06:49:48.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.507+0000 7fdd97dda640 1 -- 172.21.15.42:0/3732441733 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fdd90063230 con 0x7fdd7c0b3c50 2024-07-26T06:49:48.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.508+0000 7fdd86ffd640 1 -- 172.21.15.42:0/3732441733 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fdd90063230 con 0x7fdd7c0b3c50 2024-07-26T06:49:48.509 INFO:teuthology.orchestra.run.smithi042.stdout:HOST ADDR LABELS STATUS 2024-07-26T06:49:48.509 INFO:teuthology.orchestra.run.smithi042.stdout:smithi042 172.21.15.42 2024-07-26T06:49:48.509 INFO:teuthology.orchestra.run.smithi042.stdout:smithi163 172.21.15.163 2024-07-26T06:49:48.509 INFO:teuthology.orchestra.run.smithi042.stdout:smithi164 172.21.15.164 2024-07-26T06:49:48.509 INFO:teuthology.orchestra.run.smithi042.stdout:3 hosts in cluster 2024-07-26T06:49:48.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.511+0000 7fdd97dda640 1 -- 172.21.15.42:0/3732441733 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdd7c0b3c50 msgr2=0x7fdd7c0b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:48.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.511+0000 7fdd97dda640 1 --2- 172.21.15.42:0/3732441733 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdd7c0b3c50 0x7fdd7c0b6160 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c005d00 tx=0x7fdd8c005c50 comp rx=0 tx=0).stop 2024-07-26T06:49:48.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.511+0000 7fdd97dda640 1 -- 172.21.15.42:0/3732441733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd90104bb0 msgr2=0x7fdd9019ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:48.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.511+0000 7fdd97dda640 1 --2- 172.21.15.42:0/3732441733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd90104bb0 0x7fdd9019ab50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdd88006010 tx=0x7fdd8802fc00 comp rx=0 tx=0).stop 2024-07-26T06:49:48.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.511+0000 7fdd97dda640 1 -- 172.21.15.42:0/3732441733 shutdown_connections 2024-07-26T06:49:48.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.511+0000 7fdd97dda640 1 --2- 172.21.15.42:0/3732441733 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdd7c0b3c50 0x7fdd7c0b6160 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:48.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.511+0000 7fdd97dda640 1 --2- 172.21.15.42:0/3732441733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd90104bb0 0x7fdd9019ab50 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:48.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.511+0000 7fdd97dda640 1 --2- 172.21.15.42:0/3732441733 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd90104210 0x7fdd90199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:48.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.511+0000 7fdd97dda640 1 --2- 172.21.15.42:0/3732441733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd90103180 0x7fdd90199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:48.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.511+0000 7fdd97dda640 1 -- 172.21.15.42:0/3732441733 >> 172.21.15.42:0/3732441733 conn(0x7fdd900feaf0 msgr2=0x7fdd90107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:48.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.511+0000 7fdd97dda640 1 -- 172.21.15.42:0/3732441733 shutdown_connections 2024-07-26T06:49:48.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:48.511+0000 7fdd97dda640 1 -- 172.21.15.42:0/3732441733 wait complete. 2024-07-26T06:49:49.293 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:49 smithi042 ceph-mon[32775]: pgmap v383: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:49.324 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- bash -c 'ceph orch device ls' 2024-07-26T06:49:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:49 smithi164 ceph-mon[36970]: pgmap v383: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:49.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:49 smithi163 ceph-mon[36873]: pgmap v383: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:49.593 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:50.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:50 smithi042 ceph-mon[32775]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:49:50.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:50 smithi164 ceph-mon[36970]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:49:50.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:50 smithi163 ceph-mon[36873]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:49:51.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:51 smithi042 ceph-mon[32775]: pgmap v384: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:49:51.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.334+0000 7f009f6c0640 1 -- 172.21.15.42:0/2601488775 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0098103ad0 msgr2=0x7f009810a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:51.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.334+0000 7f009f6c0640 1 --2- 172.21.15.42:0/2601488775 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0098103ad0 0x7f009810a470 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0090009ca0 tx=0x7f009002a950 comp rx=0 tx=0).stop 2024-07-26T06:49:51.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.334+0000 7f009f6c0640 1 -- 172.21.15.42:0/2601488775 shutdown_connections 2024-07-26T06:49:51.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.334+0000 7f009f6c0640 1 --2- 172.21.15.42:0/2601488775 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0098103ad0 0x7f009810a470 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:51.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.334+0000 7f009f6c0640 1 --2- 172.21.15.42:0/2601488775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0098103130 0x7f0098103540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:51.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.334+0000 7f009f6c0640 1 --2- 172.21.15.42:0/2601488775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f00981020a0 0x7f0098102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:51.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.334+0000 7f009f6c0640 1 -- 172.21.15.42:0/2601488775 >> 172.21.15.42:0/2601488775 conn(0x7f00980fda10 msgr2=0x7f00980ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:51.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.334+0000 7f009f6c0640 1 -- 172.21.15.42:0/2601488775 shutdown_connections 2024-07-26T06:49:51.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.334+0000 7f009f6c0640 1 -- 172.21.15.42:0/2601488775 wait complete. 2024-07-26T06:49:51.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.334+0000 7f009f6c0640 1 Processor -- start 2024-07-26T06:49:51.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.334+0000 7f009f6c0640 1 -- start start 2024-07-26T06:49:51.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009f6c0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00981020a0 0x7f0098199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:51.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009f6c0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0098103130 0x7f009819a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:51.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009f6c0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0098103ad0 0x7f009819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:51.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009f6c0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f009810dd60 con 0x7f00981020a0 2024-07-26T06:49:51.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009f6c0640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f009810dbe0 con 0x7f0098103130 2024-07-26T06:49:51.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009f6c0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f009810dee0 con 0x7f0098103ad0 2024-07-26T06:49:51.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009d45c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00981020a0 0x7f0098199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:51.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009d45c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00981020a0 0x7f0098199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44574/0 (socket says 172.21.15.42:44574) 2024-07-26T06:49:51.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009d45c640 1 -- 172.21.15.42:0/3097021020 learned_addr learned my addr 172.21.15.42:0/3097021020 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:51.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009cc5b640 1 --2- 172.21.15.42:0/3097021020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0098103130 0x7f009819a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:51.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009dc5d640 1 --2- 172.21.15.42:0/3097021020 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0098103ad0 0x7f009819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:51.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009d45c640 1 -- 172.21.15.42:0/3097021020 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0098103ad0 msgr2=0x7f009819ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:51.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009d45c640 1 --2- 172.21.15.42:0/3097021020 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0098103ad0 0x7f009819ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:51.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009d45c640 1 -- 172.21.15.42:0/3097021020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0098103130 msgr2=0x7f009819a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:51.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009d45c640 1 --2- 172.21.15.42:0/3097021020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0098103130 0x7f009819a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:51.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009d45c640 1 -- 172.21.15.42:0/3097021020 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009819b450 con 0x7f00981020a0 2024-07-26T06:49:51.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009dc5d640 1 --2- 172.21.15.42:0/3097021020 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0098103ad0 0x7f009819ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:49:51.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.335+0000 7f009d45c640 1 --2- 172.21.15.42:0/3097021020 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00981020a0 0x7f0098199a20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f009400b4a0 tx=0x7f009400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:51.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.336+0000 7f008e7fc640 1 -- 172.21.15.42:0/3097021020 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0094014070 con 0x7f00981020a0 2024-07-26T06:49:51.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.336+0000 7f008e7fc640 1 -- 172.21.15.42:0/3097021020 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0094010ac0 con 0x7f00981020a0 2024-07-26T06:49:51.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.336+0000 7f009f6c0640 1 -- 172.21.15.42:0/3097021020 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f009819b740 con 0x7f00981020a0 2024-07-26T06:49:51.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.336+0000 7f008e7fc640 1 -- 172.21.15.42:0/3097021020 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0094017c30 con 0x7f00981020a0 2024-07-26T06:49:51.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.336+0000 7f009f6c0640 1 -- 172.21.15.42:0/3097021020 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00981a1e40 con 0x7f00981020a0 2024-07-26T06:49:51.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.337+0000 7f009f6c0640 1 -- 172.21.15.42:0/3097021020 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00981a2010 con 0x7f00981020a0 2024-07-26T06:49:51.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.338+0000 7f008e7fc640 1 -- 172.21.15.42:0/3097021020 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f0094017070 con 0x7f00981020a0 2024-07-26T06:49:51.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.339+0000 7f008e7fc640 1 --2- 172.21.15.42:0/3097021020 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f00840b3bd0 0x7f00840b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:51.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.339+0000 7f009cc5b640 1 --2- 172.21.15.42:0/3097021020 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f00840b3bd0 0x7f00840b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:51.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.339+0000 7f008e7fc640 1 -- 172.21.15.42:0/3097021020 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f00940e33b0 con 0x7f00981020a0 2024-07-26T06:49:51.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.340+0000 7f009cc5b640 1 --2- 172.21.15.42:0/3097021020 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f00840b3bd0 0x7f00840b60e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0088005fd0 tx=0x7f0088005e20 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:51.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.342+0000 7f008e7fc640 1 -- 172.21.15.42:0/3097021020 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f00940aa9e0 con 0x7f00981020a0 2024-07-26T06:49:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:51 smithi164 ceph-mon[36970]: pgmap v384: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:49:51.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:51 smithi163 ceph-mon[36873]: pgmap v384: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:49:51.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.534+0000 7f009f6c0640 1 -- 172.21.15.42:0/3097021020 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f00980ff970 con 0x7f00840b3bd0 2024-07-26T06:49:51.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.535+0000 7f008e7fc640 1 -- 172.21.15.42:0/3097021020 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f00980ff970 con 0x7f00840b3bd0 2024-07-26T06:49:51.536 INFO:teuthology.orchestra.run.smithi042.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-07-26T06:49:51.536 INFO:teuthology.orchestra.run.smithi042.stdout:smithi042 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB75130021480DGN 447G No 4m ago Has a FileSystem, LVM detected 2024-07-26T06:49:51.536 INFO:teuthology.orchestra.run.smithi042.stdout:smithi163 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GU400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-07-26T06:49:51.536 INFO:teuthology.orchestra.run.smithi042.stdout:smithi164 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300H0400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-07-26T06:49:51.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.537+0000 7f009f6c0640 1 -- 172.21.15.42:0/3097021020 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f00840b3bd0 msgr2=0x7f00840b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:51.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.537+0000 7f009f6c0640 1 --2- 172.21.15.42:0/3097021020 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f00840b3bd0 0x7f00840b60e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0088005fd0 tx=0x7f0088005e20 comp rx=0 tx=0).stop 2024-07-26T06:49:51.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.537+0000 7f009f6c0640 1 -- 172.21.15.42:0/3097021020 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00981020a0 msgr2=0x7f0098199a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:51.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.537+0000 7f009f6c0640 1 --2- 172.21.15.42:0/3097021020 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00981020a0 0x7f0098199a20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f009400b4a0 tx=0x7f009400b960 comp rx=0 tx=0).stop 2024-07-26T06:49:51.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.538+0000 7f009f6c0640 1 -- 172.21.15.42:0/3097021020 shutdown_connections 2024-07-26T06:49:51.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.538+0000 7f009f6c0640 1 --2- 172.21.15.42:0/3097021020 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f00840b3bd0 0x7f00840b60e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:51.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.538+0000 7f009f6c0640 1 --2- 172.21.15.42:0/3097021020 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0098103ad0 0x7f009819ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:51.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.538+0000 7f009f6c0640 1 --2- 172.21.15.42:0/3097021020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0098103130 0x7f009819a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:51.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.538+0000 7f009f6c0640 1 --2- 172.21.15.42:0/3097021020 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00981020a0 0x7f0098199a20 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:51.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.538+0000 7f009f6c0640 1 -- 172.21.15.42:0/3097021020 >> 172.21.15.42:0/3097021020 conn(0x7f00980fda10 msgr2=0x7f00980ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:51.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.538+0000 7f009f6c0640 1 -- 172.21.15.42:0/3097021020 shutdown_connections 2024-07-26T06:49:51.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:51.538+0000 7f009f6c0640 1 -- 172.21.15.42:0/3097021020 wait complete. 2024-07-26T06:49:52.184 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-07-26T06:49:52.193 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi042.front.sepia.ceph.com 2024-07-26T06:49:52.194 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- bash -c 'ceph fs dump' 2024-07-26T06:49:52.458 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:53.313 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:53 smithi042 ceph-mon[32775]: pgmap v385: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:49:53.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:53 smithi042 ceph-mon[32775]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:49:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:53 smithi164 ceph-mon[36970]: pgmap v385: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:49:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:53 smithi164 ceph-mon[36970]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:49:53.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:53 smithi163 ceph-mon[36873]: pgmap v385: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:49:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:53 smithi163 ceph-mon[36873]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T06:49:54.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.278+0000 7fa7afe2d640 1 -- 172.21.15.42:0/1499020443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7a8103180 msgr2=0x7fa7a81035f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:54.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.278+0000 7fa7afe2d640 1 --2- 172.21.15.42:0/1499020443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7a8103180 0x7fa7a81035f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4009900 tx=0x7fa7a402a480 comp rx=0 tx=0).stop 2024-07-26T06:49:54.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.278+0000 7fa7afe2d640 1 -- 172.21.15.42:0/1499020443 shutdown_connections 2024-07-26T06:49:54.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.278+0000 7fa7afe2d640 1 --2- 172.21.15.42:0/1499020443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7a8104bb0 0x7fa7a810b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:54.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.278+0000 7fa7afe2d640 1 --2- 172.21.15.42:0/1499020443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a8104210 0x7fa7a8104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:54.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.278+0000 7fa7afe2d640 1 --2- 172.21.15.42:0/1499020443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7a8103180 0x7fa7a81035f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:54.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.278+0000 7fa7afe2d640 1 -- 172.21.15.42:0/1499020443 >> 172.21.15.42:0/1499020443 conn(0x7fa7a80feaf0 msgr2=0x7fa7a8100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:54.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.278+0000 7fa7afe2d640 1 -- 172.21.15.42:0/1499020443 shutdown_connections 2024-07-26T06:49:54.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.278+0000 7fa7afe2d640 1 -- 172.21.15.42:0/1499020443 wait complete. 2024-07-26T06:49:54.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.279+0000 7fa7afe2d640 1 Processor -- start 2024-07-26T06:49:54.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.279+0000 7fa7afe2d640 1 -- start start 2024-07-26T06:49:54.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.280+0000 7fa7afe2d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a8103180 0x7fa7a81999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:54.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.280+0000 7fa7afe2d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7a8104210 0x7fa7a819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:54.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.280+0000 7fa7afe2d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7a8104bb0 0x7fa7a819ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:54.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.280+0000 7fa7afe2d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a810ede0 con 0x7fa7a8104210 2024-07-26T06:49:54.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.280+0000 7fa7afe2d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a810ec60 con 0x7fa7a8104bb0 2024-07-26T06:49:54.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.280+0000 7fa7afe2d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a810ef60 con 0x7fa7a8103180 2024-07-26T06:49:54.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.280+0000 7fa7ad3c8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7a8104210 0x7fa7a819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:54.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.280+0000 7fa7ad3c8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7a8104210 0x7fa7a819a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44592/0 (socket says 172.21.15.42:44592) 2024-07-26T06:49:54.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.280+0000 7fa7ad3c8640 1 -- 172.21.15.42:0/1070086597 learned_addr learned my addr 172.21.15.42:0/1070086597 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:54.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.280+0000 7fa7adbc9640 1 --2- 172.21.15.42:0/1070086597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a8103180 0x7fa7a81999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:54.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.280+0000 7fa7ae3ca640 1 --2- 172.21.15.42:0/1070086597 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7a8104bb0 0x7fa7a819ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:54.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.280+0000 7fa7ad3c8640 1 -- 172.21.15.42:0/1070086597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a8103180 msgr2=0x7fa7a81999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:54.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.280+0000 7fa7ad3c8640 1 --2- 172.21.15.42:0/1070086597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a8103180 0x7fa7a81999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:54.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.280+0000 7fa7ad3c8640 1 -- 172.21.15.42:0/1070086597 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7a8104bb0 msgr2=0x7fa7a819ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:54.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.280+0000 7fa7ad3c8640 1 --2- 172.21.15.42:0/1070086597 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7a8104bb0 0x7fa7a819ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:54.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.280+0000 7fa7ad3c8640 1 -- 172.21.15.42:0/1070086597 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a819b390 con 0x7fa7a8104210 2024-07-26T06:49:54.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.280+0000 7fa7ad3c8640 1 --2- 172.21.15.42:0/1070086597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7a8104210 0x7fa7a819a060 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa79800d770 tx=0x7fa79800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:54.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.281+0000 7fa79effd640 1 -- 172.21.15.42:0/1070086597 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa798017070 con 0x7fa7a8104210 2024-07-26T06:49:54.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.281+0000 7fa79effd640 1 -- 172.21.15.42:0/1070086597 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7980054d0 con 0x7fa7a8104210 2024-07-26T06:49:54.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.281+0000 7fa7afe2d640 1 -- 172.21.15.42:0/1070086597 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a819b680 con 0x7fa7a8104210 2024-07-26T06:49:54.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.281+0000 7fa79effd640 1 -- 172.21.15.42:0/1070086597 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa798004c30 con 0x7fa7a8104210 2024-07-26T06:49:54.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.281+0000 7fa7afe2d640 1 -- 172.21.15.42:0/1070086597 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a81a1c40 con 0x7fa7a8104210 2024-07-26T06:49:54.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.282+0000 7fa7afe2d640 1 -- 172.21.15.42:0/1070086597 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7a810a360 con 0x7fa7a8104210 2024-07-26T06:49:54.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.283+0000 7fa79effd640 1 -- 172.21.15.42:0/1070086597 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fa798004e10 con 0x7fa7a8104210 2024-07-26T06:49:54.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.284+0000 7fa79effd640 1 --2- 172.21.15.42:0/1070086597 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa7940b3bd0 0x7fa7940b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:54.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.284+0000 7fa7adbc9640 1 --2- 172.21.15.42:0/1070086597 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa7940b3bd0 0x7fa7940b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:54.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.284+0000 7fa79effd640 1 -- 172.21.15.42:0/1070086597 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fa7980e3800 con 0x7fa7a8104210 2024-07-26T06:49:54.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.284+0000 7fa7adbc9640 1 --2- 172.21.15.42:0/1070086597 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa7940b3bd0 0x7fa7940b60e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4031440 tx=0x7fa7a40314a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:54.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.288+0000 7fa79effd640 1 -- 172.21.15.42:0/1070086597 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa7980aadd0 con 0x7fa7a8104210 2024-07-26T06:49:54.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.508+0000 7fa7afe2d640 1 -- 172.21.15.42:0/1070086597 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fa7a802e0c0 con 0x7fa7a8104210 2024-07-26T06:49:54.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.508+0000 7fa79effd640 1 -- 172.21.15.42:0/1070086597 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3132 (secure 0 0 0) 0x7fa7980aa6d0 con 0x7fa7a8104210 2024-07-26T06:49:54.509 INFO:teuthology.orchestra.run.smithi042.stdout:e28 2024-07-26T06:49:54.509 INFO:teuthology.orchestra.run.smithi042.stdout:btime 2024-07-26T06:47:26:107129+0000 2024-07-26T06:49:54.510 INFO:teuthology.orchestra.run.smithi042.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T06:49:54.510 INFO:teuthology.orchestra.run.smithi042.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T06:49:54.510 INFO:teuthology.orchestra.run.smithi042.stdout:legacy client fscid: 1 2024-07-26T06:49:54.510 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:49:54.510 INFO:teuthology.orchestra.run.smithi042.stdout:Filesystem 'cephfs' (1) 2024-07-26T06:49:54.510 INFO:teuthology.orchestra.run.smithi042.stdout:fs_name cephfs 2024-07-26T06:49:54.510 INFO:teuthology.orchestra.run.smithi042.stdout:epoch 28 2024-07-26T06:49:54.510 INFO:teuthology.orchestra.run.smithi042.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-07-26T06:49:54.510 INFO:teuthology.orchestra.run.smithi042.stdout:created 2024-07-26T06:47:06.780568+0000 2024-07-26T06:49:54.510 INFO:teuthology.orchestra.run.smithi042.stdout:modified 2024-07-26T06:47:25.125103+0000 2024-07-26T06:49:54.510 INFO:teuthology.orchestra.run.smithi042.stdout:tableserver 0 2024-07-26T06:49:54.511 INFO:teuthology.orchestra.run.smithi042.stdout:root 0 2024-07-26T06:49:54.511 INFO:teuthology.orchestra.run.smithi042.stdout:session_timeout 300 2024-07-26T06:49:54.511 INFO:teuthology.orchestra.run.smithi042.stdout:session_autoclose 300 2024-07-26T06:49:54.511 INFO:teuthology.orchestra.run.smithi042.stdout:max_file_size 1099511627776 2024-07-26T06:49:54.511 INFO:teuthology.orchestra.run.smithi042.stdout:max_xattr_size 65536 2024-07-26T06:49:54.511 INFO:teuthology.orchestra.run.smithi042.stdout:required_client_features {} 2024-07-26T06:49:54.511 INFO:teuthology.orchestra.run.smithi042.stdout:last_failure 0 2024-07-26T06:49:54.511 INFO:teuthology.orchestra.run.smithi042.stdout:last_failure_osd_epoch 0 2024-07-26T06:49:54.511 INFO:teuthology.orchestra.run.smithi042.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T06:49:54.511 INFO:teuthology.orchestra.run.smithi042.stdout:max_mds 5 2024-07-26T06:49:54.511 INFO:teuthology.orchestra.run.smithi042.stdout:in 0,1,2,3,4 2024-07-26T06:49:54.511 INFO:teuthology.orchestra.run.smithi042.stdout:up {0=24620,1=24601,2=24593,3=24583,4=24578} 2024-07-26T06:49:54.511 INFO:teuthology.orchestra.run.smithi042.stdout:failed 2024-07-26T06:49:54.512 INFO:teuthology.orchestra.run.smithi042.stdout:damaged 2024-07-26T06:49:54.512 INFO:teuthology.orchestra.run.smithi042.stdout:stopped 2024-07-26T06:49:54.512 INFO:teuthology.orchestra.run.smithi042.stdout:data_pools [3] 2024-07-26T06:49:54.512 INFO:teuthology.orchestra.run.smithi042.stdout:metadata_pool 2 2024-07-26T06:49:54.512 INFO:teuthology.orchestra.run.smithi042.stdout:inline_data disabled 2024-07-26T06:49:54.512 INFO:teuthology.orchestra.run.smithi042.stdout:balancer 2024-07-26T06:49:54.512 INFO:teuthology.orchestra.run.smithi042.stdout:bal_rank_mask -1 2024-07-26T06:49:54.512 INFO:teuthology.orchestra.run.smithi042.stdout:standby_count_wanted 0 2024-07-26T06:49:54.512 INFO:teuthology.orchestra.run.smithi042.stdout:qdb_cluster leader: 24620 members: 24578,24583,24593,24601,24620 2024-07-26T06:49:54.512 INFO:teuthology.orchestra.run.smithi042.stdout:[mds.b{0:24620} state up:active seq 6 addr [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T06:49:54.512 INFO:teuthology.orchestra.run.smithi042.stdout:[mds.f{0:24571} state up:standby-replay seq 1 addr [v2:172.21.15.164:6834/2604534474,v1:172.21.15.164:6835/2604534474] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T06:49:54.512 INFO:teuthology.orchestra.run.smithi042.stdout:[mds.i{1:24601} state up:active seq 11 addr [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T06:49:54.512 INFO:teuthology.orchestra.run.smithi042.stdout:[mds.k{1:24557} state up:standby-replay seq 1 addr [v2:172.21.15.163:6832/845444723,v1:172.21.15.163:6833/845444723] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T06:49:54.513 INFO:teuthology.orchestra.run.smithi042.stdout:[mds.h{2:24593} state up:active seq 12 addr [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T06:49:54.513 INFO:teuthology.orchestra.run.smithi042.stdout:[mds.l{2:24541} state up:standby-replay seq 1 addr [v2:172.21.15.164:6832/4144543280,v1:172.21.15.164:6833/4144543280] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T06:49:54.513 INFO:teuthology.orchestra.run.smithi042.stdout:[mds.c{3:24583} state up:active seq 13 addr [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T06:49:54.513 INFO:teuthology.orchestra.run.smithi042.stdout:[mds.a{3:14709} state up:standby-replay seq 1 addr [v2:172.21.15.42:6840/1430155791,v1:172.21.15.42:6841/1430155791] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T06:49:54.513 INFO:teuthology.orchestra.run.smithi042.stdout:[mds.e{4:24578} state up:active seq 15 addr [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T06:49:54.513 INFO:teuthology.orchestra.run.smithi042.stdout:[mds.g{4:14703} state up:standby-replay seq 1 addr [v2:172.21.15.42:6838/4044632348,v1:172.21.15.42:6839/4044632348] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T06:49:54.513 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:49:54.513 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:49:54.513 INFO:teuthology.orchestra.run.smithi042.stdout:Standby daemons: 2024-07-26T06:49:54.513 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:49:54.513 INFO:teuthology.orchestra.run.smithi042.stdout:[mds.j{-1:14658} state up:standby seq 1 addr [v2:172.21.15.42:6834/3548166563,v1:172.21.15.42:6835/3548166563] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T06:49:54.513 INFO:teuthology.orchestra.run.smithi042.stdout:[mds.d{-1:14664} state up:standby seq 1 addr [v2:172.21.15.42:6836/1420377891,v1:172.21.15.42:6837/1420377891] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T06:49:54.514 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 28 2024-07-26T06:49:54.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.510+0000 7fa7afe2d640 1 -- 172.21.15.42:0/1070086597 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa7940b3bd0 msgr2=0x7fa7940b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:54.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.510+0000 7fa7afe2d640 1 --2- 172.21.15.42:0/1070086597 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa7940b3bd0 0x7fa7940b60e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4031440 tx=0x7fa7a40314a0 comp rx=0 tx=0).stop 2024-07-26T06:49:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.510+0000 7fa7afe2d640 1 -- 172.21.15.42:0/1070086597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7a8104210 msgr2=0x7fa7a819a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.510+0000 7fa7afe2d640 1 --2- 172.21.15.42:0/1070086597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7a8104210 0x7fa7a819a060 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa79800d770 tx=0x7fa79800dc30 comp rx=0 tx=0).stop 2024-07-26T06:49:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.511+0000 7fa7afe2d640 1 -- 172.21.15.42:0/1070086597 shutdown_connections 2024-07-26T06:49:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.511+0000 7fa7afe2d640 1 --2- 172.21.15.42:0/1070086597 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa7940b3bd0 0x7fa7940b60e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.511+0000 7fa7afe2d640 1 --2- 172.21.15.42:0/1070086597 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7a8104bb0 0x7fa7a819ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.511+0000 7fa7afe2d640 1 --2- 172.21.15.42:0/1070086597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa7a8104210 0x7fa7a819a060 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.511+0000 7fa7afe2d640 1 --2- 172.21.15.42:0/1070086597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a8103180 0x7fa7a81999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.511+0000 7fa7afe2d640 1 -- 172.21.15.42:0/1070086597 >> 172.21.15.42:0/1070086597 conn(0x7fa7a80feaf0 msgr2=0x7fa7a8107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.511+0000 7fa7afe2d640 1 -- 172.21.15.42:0/1070086597 shutdown_connections 2024-07-26T06:49:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:54.511+0000 7fa7afe2d640 1 -- 172.21.15.42:0/1070086597 wait complete. 2024-07-26T06:49:55.116 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:55 smithi042 ceph-mon[32775]: pgmap v386: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:55.116 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:55 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1070086597' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T06:49:55.145 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- bash -c 'ceph osd dump' 2024-07-26T06:49:55.411 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:55 smithi164 ceph-mon[36970]: pgmap v386: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:55 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1070086597' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T06:49:55.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:55 smithi163 ceph-mon[36873]: pgmap v386: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:55.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:55 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1070086597' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T06:49:57.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.089+0000 7f49be8e5640 1 -- 172.21.15.42:0/2506178961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49b8103af0 msgr2=0x7f49b810a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:57.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.089+0000 7f49be8e5640 1 --2- 172.21.15.42:0/2506178961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49b8103af0 0x7f49b810a490 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f49b0009ca0 tx=0x7f49b002a950 comp rx=0 tx=0).stop 2024-07-26T06:49:57.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.089+0000 7f49be8e5640 1 -- 172.21.15.42:0/2506178961 shutdown_connections 2024-07-26T06:49:57.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.089+0000 7f49be8e5640 1 --2- 172.21.15.42:0/2506178961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49b8103af0 0x7f49b810a490 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:57.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.089+0000 7f49be8e5640 1 --2- 172.21.15.42:0/2506178961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49b8103150 0x7f49b8103560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:57.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.089+0000 7f49be8e5640 1 --2- 172.21.15.42:0/2506178961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49b81020c0 0x7f49b8102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:57.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.089+0000 7f49be8e5640 1 -- 172.21.15.42:0/2506178961 >> 172.21.15.42:0/2506178961 conn(0x7f49b80fda30 msgr2=0x7f49b80ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:57.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.089+0000 7f49be8e5640 1 -- 172.21.15.42:0/2506178961 shutdown_connections 2024-07-26T06:49:57.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.089+0000 7f49be8e5640 1 -- 172.21.15.42:0/2506178961 wait complete. 2024-07-26T06:49:57.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.090+0000 7f49be8e5640 1 Processor -- start 2024-07-26T06:49:57.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.090+0000 7f49be8e5640 1 -- start start 2024-07-26T06:49:57.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.090+0000 7f49be8e5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49b81020c0 0x7f49b806bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:57.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.090+0000 7f49be8e5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49b8103150 0x7f49b819e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:57.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.090+0000 7f49be8e5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49b8103af0 0x7f49b81a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:57.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.090+0000 7f49be8e5640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f49b810dd80 con 0x7f49b8103af0 2024-07-26T06:49:57.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.090+0000 7f49be8e5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f49b810dc00 con 0x7f49b8103150 2024-07-26T06:49:57.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.090+0000 7f49be8e5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f49b810df00 con 0x7f49b81020c0 2024-07-26T06:49:57.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.091+0000 7f49bce82640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49b8103af0 0x7f49b81a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:57.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.091+0000 7f49bce82640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49b8103af0 0x7f49b81a0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33774/0 (socket says 172.21.15.42:33774) 2024-07-26T06:49:57.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.091+0000 7f49b7fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49b81020c0 0x7f49b806bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:57.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.091+0000 7f49bce82640 1 -- 172.21.15.42:0/396851147 learned_addr learned my addr 172.21.15.42:0/396851147 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:49:57.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.091+0000 7f49affff640 1 --2- 172.21.15.42:0/396851147 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49b8103150 0x7f49b819e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:57.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.091+0000 7f49bce82640 1 -- 172.21.15.42:0/396851147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49b81020c0 msgr2=0x7f49b806bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:57.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.091+0000 7f49bce82640 1 --2- 172.21.15.42:0/396851147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49b81020c0 0x7f49b806bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:57.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.091+0000 7f49bce82640 1 -- 172.21.15.42:0/396851147 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49b8103150 msgr2=0x7f49b819e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:57.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.091+0000 7f49bce82640 1 --2- 172.21.15.42:0/396851147 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49b8103150 0x7f49b819e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:57.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.091+0000 7f49bce82640 1 -- 172.21.15.42:0/396851147 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b81a14b0 con 0x7f49b8103af0 2024-07-26T06:49:57.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.091+0000 7f49bce82640 1 --2- 172.21.15.42:0/396851147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49b8103af0 0x7f49b81a0cb0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f49b0006010 tx=0x7f49b002fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:57.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.091+0000 7f49b5ffb640 1 -- 172.21.15.42:0/396851147 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b0034040 con 0x7f49b8103af0 2024-07-26T06:49:57.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.091+0000 7f49b5ffb640 1 -- 172.21.15.42:0/396851147 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49b00308f0 con 0x7f49b8103af0 2024-07-26T06:49:57.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.091+0000 7f49b5ffb640 1 -- 172.21.15.42:0/396851147 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b0039450 con 0x7f49b8103af0 2024-07-26T06:49:57.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.091+0000 7f49be8e5640 1 -- 172.21.15.42:0/396851147 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b81a1740 con 0x7f49b8103af0 2024-07-26T06:49:57.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.091+0000 7f49be8e5640 1 -- 172.21.15.42:0/396851147 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b81a1c50 con 0x7f49b8103af0 2024-07-26T06:49:57.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.092+0000 7f49be8e5640 1 -- 172.21.15.42:0/396851147 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49b81092a0 con 0x7f49b8103af0 2024-07-26T06:49:57.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.094+0000 7f49b5ffb640 1 -- 172.21.15.42:0/396851147 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f49b0037020 con 0x7f49b8103af0 2024-07-26T06:49:57.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.095+0000 7f49b5ffb640 1 --2- 172.21.15.42:0/396851147 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49a40b3c50 0x7f49a40b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:49:57.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.095+0000 7f49b7fff640 1 --2- 172.21.15.42:0/396851147 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49a40b3c50 0x7f49a40b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:49:57.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.095+0000 7f49b5ffb640 1 -- 172.21.15.42:0/396851147 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f49b0104e40 con 0x7f49b8103af0 2024-07-26T06:49:57.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.095+0000 7f49b7fff640 1 --2- 172.21.15.42:0/396851147 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49a40b3c50 0x7f49a40b6160 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f49a8005fd0 tx=0x7f49a800b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:49:57.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.098+0000 7f49b5ffb640 1 -- 172.21.15.42:0/396851147 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f49b00cc410 con 0x7f49b8103af0 2024-07-26T06:49:57.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:57 smithi042 ceph-mon[32775]: pgmap v387: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:49:57.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.283+0000 7f49be8e5640 1 -- 172.21.15.42:0/396851147 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f49b802e0c0 con 0x7f49b8103af0 2024-07-26T06:49:57.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.284+0000 7f49b5ffb640 1 -- 172.21.15.42:0/396851147 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v80) ==== 56+0+5318 (secure 0 0 0) 0x7f49b00cbd10 con 0x7f49b8103af0 2024-07-26T06:49:57.284 INFO:teuthology.orchestra.run.smithi042.stdout:epoch 80 2024-07-26T06:49:57.285 INFO:teuthology.orchestra.run.smithi042.stdout:fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 2024-07-26T06:49:57.285 INFO:teuthology.orchestra.run.smithi042.stdout:created 2024-07-26T06:38:29.306470+0000 2024-07-26T06:49:57.285 INFO:teuthology.orchestra.run.smithi042.stdout:modified 2024-07-26T06:47:23.086609+0000 2024-07-26T06:49:57.285 INFO:teuthology.orchestra.run.smithi042.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-07-26T06:49:57.285 INFO:teuthology.orchestra.run.smithi042.stdout:crush_version 26 2024-07-26T06:49:57.285 INFO:teuthology.orchestra.run.smithi042.stdout:full_ratio 0.95 2024-07-26T06:49:57.285 INFO:teuthology.orchestra.run.smithi042.stdout:backfillfull_ratio 0.9 2024-07-26T06:49:57.285 INFO:teuthology.orchestra.run.smithi042.stdout:nearfull_ratio 0.85 2024-07-26T06:49:57.285 INFO:teuthology.orchestra.run.smithi042.stdout:require_min_compat_client luminous 2024-07-26T06:49:57.285 INFO:teuthology.orchestra.run.smithi042.stdout:min_compat_client luminous 2024-07-26T06:49:57.285 INFO:teuthology.orchestra.run.smithi042.stdout:require_osd_release squid 2024-07-26T06:49:57.285 INFO:teuthology.orchestra.run.smithi042.stdout:stretch_mode_enabled false 2024-07-26T06:49:57.286 INFO:teuthology.orchestra.run.smithi042.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-07-26T06:49:57.286 INFO:teuthology.orchestra.run.smithi042.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 78 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-07-26T06:49:57.286 INFO:teuthology.orchestra.run.smithi042.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 78 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-07-26T06:49:57.286 INFO:teuthology.orchestra.run.smithi042.stdout:max_osd 12 2024-07-26T06:49:57.286 INFO:teuthology.orchestra.run.smithi042.stdout:osd.0 up in weight 1 up_from 8 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.42:6802/3587066379,v1:172.21.15.42:6803/3587066379] [v2:172.21.15.42:6804/3587066379,v1:172.21.15.42:6805/3587066379] exists,up 1bda88e0-09ee-447b-8c04-24f4e5d55d19 2024-07-26T06:49:57.286 INFO:teuthology.orchestra.run.smithi042.stdout:osd.1 up in weight 1 up_from 13 up_thru 75 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6800/360219078,v1:172.21.15.163:6801/360219078] [v2:172.21.15.163:6802/360219078,v1:172.21.15.163:6803/360219078] exists,up 7d747138-8217-4836-a173-cf8ac65e9fe3 2024-07-26T06:49:57.286 INFO:teuthology.orchestra.run.smithi042.stdout:osd.2 up in weight 1 up_from 18 up_thru 75 down_at 0 last_clean_interval [0,0) [v2:172.21.15.164:6800/336113412,v1:172.21.15.164:6801/336113412] [v2:172.21.15.164:6802/336113412,v1:172.21.15.164:6803/336113412] exists,up 4b7258c1-3d35-4930-ab10-0b04a2a2c8a2 2024-07-26T06:49:57.286 INFO:teuthology.orchestra.run.smithi042.stdout:osd.3 up in weight 1 up_from 25 up_thru 75 down_at 0 last_clean_interval [0,0) [v2:172.21.15.42:6810/2819018064,v1:172.21.15.42:6811/2819018064] [v2:172.21.15.42:6812/2819018064,v1:172.21.15.42:6813/2819018064] exists,up 96608387-0de0-4d90-a952-d6eba61f7fbe 2024-07-26T06:49:57.286 INFO:teuthology.orchestra.run.smithi042.stdout:osd.4 up in weight 1 up_from 31 up_thru 75 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6808/431934374,v1:172.21.15.163:6809/431934374] [v2:172.21.15.163:6810/431934374,v1:172.21.15.163:6811/431934374] exists,up fabf6b02-fe34-4560-85b5-01bf37fc9be4 2024-07-26T06:49:57.286 INFO:teuthology.orchestra.run.smithi042.stdout:osd.5 up in weight 1 up_from 36 up_thru 75 down_at 0 last_clean_interval [0,0) [v2:172.21.15.164:6808/261549362,v1:172.21.15.164:6809/261549362] [v2:172.21.15.164:6810/261549362,v1:172.21.15.164:6811/261549362] exists,up e226fb4f-15b2-40df-af34-900040ba93ad 2024-07-26T06:49:57.286 INFO:teuthology.orchestra.run.smithi042.stdout:osd.6 up in weight 1 up_from 42 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.42:6818/2160811300,v1:172.21.15.42:6819/2160811300] [v2:172.21.15.42:6820/2160811300,v1:172.21.15.42:6821/2160811300] exists,up beb8ec5d-7ea5-4e4e-a46c-a84f7768cec0 2024-07-26T06:49:57.287 INFO:teuthology.orchestra.run.smithi042.stdout:osd.7 up in weight 1 up_from 47 up_thru 75 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6816/812214111,v1:172.21.15.163:6817/812214111] [v2:172.21.15.163:6818/812214111,v1:172.21.15.163:6819/812214111] exists,up f699004c-f7c2-4f1c-ae4b-7178f3b06e2d 2024-07-26T06:49:57.287 INFO:teuthology.orchestra.run.smithi042.stdout:osd.8 up in weight 1 up_from 53 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.164:6816/962845839,v1:172.21.15.164:6817/962845839] [v2:172.21.15.164:6818/962845839,v1:172.21.15.164:6819/962845839] exists,up 3476163e-efea-4652-a8d4-b7c4a908f984 2024-07-26T06:49:57.287 INFO:teuthology.orchestra.run.smithi042.stdout:osd.9 up in weight 1 up_from 59 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.42:6826/994024418,v1:172.21.15.42:6827/994024418] [v2:172.21.15.42:6828/994024418,v1:172.21.15.42:6829/994024418] exists,up 6f7ceac6-fb61-4c6f-9da8-851b937c4586 2024-07-26T06:49:57.287 INFO:teuthology.orchestra.run.smithi042.stdout:osd.10 up in weight 1 up_from 64 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6824/3268512227,v1:172.21.15.163:6825/3268512227] [v2:172.21.15.163:6826/3268512227,v1:172.21.15.163:6827/3268512227] exists,up 553132c2-a6a1-410c-a7d8-ecdf9a67dafb 2024-07-26T06:49:57.287 INFO:teuthology.orchestra.run.smithi042.stdout:osd.11 up in weight 1 up_from 70 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.164:6824/1861076336,v1:172.21.15.164:6825/1861076336] [v2:172.21.15.164:6826/1861076336,v1:172.21.15.164:6827/1861076336] exists,up 09abc8e9-7884-400f-ab6d-8e5aa7bf31ff 2024-07-26T06:49:57.287 INFO:teuthology.orchestra.run.smithi042.stdout:pg_upmap_items 2.5 [8,11] 2024-07-26T06:49:57.287 INFO:teuthology.orchestra.run.smithi042.stdout:pg_upmap_items 2.b [2,1] 2024-07-26T06:49:57.287 INFO:teuthology.orchestra.run.smithi042.stdout:pg_upmap_items 2.14 [2,4] 2024-07-26T06:49:57.287 INFO:teuthology.orchestra.run.smithi042.stdout:pg_upmap_items 2.1c [8,11] 2024-07-26T06:49:57.287 INFO:teuthology.orchestra.run.smithi042.stdout:pg_upmap_items 3.6 [4,6] 2024-07-26T06:49:57.287 INFO:teuthology.orchestra.run.smithi042.stdout:pg_upmap_items 3.11 [5,9] 2024-07-26T06:49:57.287 INFO:teuthology.orchestra.run.smithi042.stdout:pg_upmap_items 3.19 [3,7] 2024-07-26T06:49:57.287 INFO:teuthology.orchestra.run.smithi042.stdout:pg_upmap_items 3.24 [4,9] 2024-07-26T06:49:57.288 INFO:teuthology.orchestra.run.smithi042.stdout:pg_upmap_items 3.28 [5,8] 2024-07-26T06:49:57.288 INFO:teuthology.orchestra.run.smithi042.stdout:pg_upmap_items 3.36 [4,9] 2024-07-26T06:49:57.288 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:6801/118189018 expires 2024-07-27T06:39:21.217122+0000 2024-07-26T06:49:57.288 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:6800/118189018 expires 2024-07-27T06:39:21.217122+0000 2024-07-26T06:49:57.288 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:0/1842521399 expires 2024-07-27T06:39:21.217122+0000 2024-07-26T06:49:57.288 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:0/834172950 expires 2024-07-27T06:39:21.217122+0000 2024-07-26T06:49:57.288 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:0/1730109419 expires 2024-07-27T06:39:21.217122+0000 2024-07-26T06:49:57.288 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:6801/2946137308 expires 2024-07-27T06:38:56.887264+0000 2024-07-26T06:49:57.288 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:6800/2946137308 expires 2024-07-27T06:38:56.887264+0000 2024-07-26T06:49:57.288 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:0/2687772889 expires 2024-07-27T06:38:56.887264+0000 2024-07-26T06:49:57.288 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:0/3371433307 expires 2024-07-27T06:38:56.887264+0000 2024-07-26T06:49:57.288 INFO:teuthology.orchestra.run.smithi042.stdout:blocklist 172.21.15.42:0/2861719816 expires 2024-07-27T06:38:56.887264+0000 2024-07-26T06:49:57.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.286+0000 7f49be8e5640 1 -- 172.21.15.42:0/396851147 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49a40b3c50 msgr2=0x7f49a40b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:57.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.286+0000 7f49be8e5640 1 --2- 172.21.15.42:0/396851147 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49a40b3c50 0x7f49a40b6160 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f49a8005fd0 tx=0x7f49a800b390 comp rx=0 tx=0).stop 2024-07-26T06:49:57.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.286+0000 7f49be8e5640 1 -- 172.21.15.42:0/396851147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49b8103af0 msgr2=0x7f49b81a0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:49:57.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.286+0000 7f49be8e5640 1 --2- 172.21.15.42:0/396851147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49b8103af0 0x7f49b81a0cb0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f49b0006010 tx=0x7f49b002fc00 comp rx=0 tx=0).stop 2024-07-26T06:49:57.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.286+0000 7f49be8e5640 1 -- 172.21.15.42:0/396851147 shutdown_connections 2024-07-26T06:49:57.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.286+0000 7f49be8e5640 1 --2- 172.21.15.42:0/396851147 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f49a40b3c50 0x7f49a40b6160 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:57.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.286+0000 7f49be8e5640 1 --2- 172.21.15.42:0/396851147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49b8103af0 0x7f49b81a0cb0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:57.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.286+0000 7f49be8e5640 1 --2- 172.21.15.42:0/396851147 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f49b8103150 0x7f49b819e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:57.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.286+0000 7f49be8e5640 1 --2- 172.21.15.42:0/396851147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f49b81020c0 0x7f49b806bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:49:57.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.286+0000 7f49be8e5640 1 -- 172.21.15.42:0/396851147 >> 172.21.15.42:0/396851147 conn(0x7f49b80fda30 msgr2=0x7f49b8106470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:49:57.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.286+0000 7f49be8e5640 1 -- 172.21.15.42:0/396851147 shutdown_connections 2024-07-26T06:49:57.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:49:57.286+0000 7f49be8e5640 1 -- 172.21.15.42:0/396851147 wait complete. 2024-07-26T06:49:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:57 smithi164 ceph-mon[36970]: pgmap v387: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:49:57.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:57 smithi163 ceph-mon[36873]: pgmap v387: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:49:58.056 INFO:teuthology.run_tasks:Running task fs.ready... 2024-07-26T06:49:58.130 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:49:58.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:58 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/396851147' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T06:49:58.401 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:49:58.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:58 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/396851147' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T06:49:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:58 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/396851147' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T06:49:59.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:49:59 smithi042 ceph-mon[32775]: pgmap v388: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:59.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:49:59 smithi164 ceph-mon[36970]: pgmap v388: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:49:59.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:49:59 smithi163 ceph-mon[36873]: pgmap v388: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:50:00.092 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:00 smithi042 ceph-mon[32775]: overall HEALTH_OK 2024-07-26T06:50:00.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.185+0000 7f25a9b4b640 1 -- 172.21.15.42:0/227928806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25a4109820 msgr2=0x7f25a4110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:00.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.185+0000 7f25a9b4b640 1 --2- 172.21.15.42:0/227928806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25a4109820 0x7f25a4110050 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f259c009ca0 tx=0x7f259c02a8c0 comp rx=0 tx=0).stop 2024-07-26T06:50:00.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.185+0000 7f25a9b4b640 1 -- 172.21.15.42:0/227928806 shutdown_connections 2024-07-26T06:50:00.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.185+0000 7f25a9b4b640 1 --2- 172.21.15.42:0/227928806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25a4109820 0x7f25a4110050 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:00.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.185+0000 7f25a9b4b640 1 --2- 172.21.15.42:0/227928806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25a4108ed0 0x7f25a41092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:00.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.185+0000 7f25a9b4b640 1 --2- 172.21.15.42:0/227928806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25a4103590 0x7f25a41039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:00.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.185+0000 7f25a9b4b640 1 -- 172.21.15.42:0/227928806 >> 172.21.15.42:0/227928806 conn(0x7f25a407b280 msgr2=0x7f25a407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:00.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.185+0000 7f25a9b4b640 1 -- 172.21.15.42:0/227928806 shutdown_connections 2024-07-26T06:50:00.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.185+0000 7f25a9b4b640 1 -- 172.21.15.42:0/227928806 wait complete. 2024-07-26T06:50:00.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.186+0000 7f25a9b4b640 1 Processor -- start 2024-07-26T06:50:00.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a9b4b640 1 -- start start 2024-07-26T06:50:00.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a9b4b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25a4103590 0x7f25a4199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:00.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a9b4b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25a4108ed0 0x7f25a419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:00.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a9b4b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25a4109820 0x7f25a419f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:00.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a9b4b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f25a41134d0 con 0x7f25a4109820 2024-07-26T06:50:00.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a9b4b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f25a4113350 con 0x7f25a4108ed0 2024-07-26T06:50:00.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a9b4b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f25a4113650 con 0x7f25a4103590 2024-07-26T06:50:00.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25a4109820 0x7f25a419f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:00.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25a4109820 0x7f25a419f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33802/0 (socket says 172.21.15.42:33802) 2024-07-26T06:50:00.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a3fff640 1 -- 172.21.15.42:0/728224601 learned_addr learned my addr 172.21.15.42:0/728224601 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:00.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a37fe640 1 --2- 172.21.15.42:0/728224601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25a4103590 0x7f25a4199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:00.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a2ffd640 1 --2- 172.21.15.42:0/728224601 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25a4108ed0 0x7f25a419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:00.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a3fff640 1 -- 172.21.15.42:0/728224601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25a4103590 msgr2=0x7f25a4199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:00.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a3fff640 1 --2- 172.21.15.42:0/728224601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25a4103590 0x7f25a4199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:00.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a3fff640 1 -- 172.21.15.42:0/728224601 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25a4108ed0 msgr2=0x7f25a419a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:00.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a3fff640 1 --2- 172.21.15.42:0/728224601 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25a4108ed0 0x7f25a419a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:00.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a3fff640 1 -- 172.21.15.42:0/728224601 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a419fdd0 con 0x7f25a4109820 2024-07-26T06:50:00.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a37fe640 1 --2- 172.21.15.42:0/728224601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25a4103590 0x7f25a4199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:50:00.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.187+0000 7f25a2ffd640 1 --2- 172.21.15.42:0/728224601 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25a4108ed0 0x7f25a419a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:50:00.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.188+0000 7f25a3fff640 1 --2- 172.21.15.42:0/728224601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25a4109820 0x7f25a419f760 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f259c005cc0 tx=0x7f259c02f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:00.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.188+0000 7f25a0ff9640 1 -- 172.21.15.42:0/728224601 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c003bf0 con 0x7f25a4109820 2024-07-26T06:50:00.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.188+0000 7f25a0ff9640 1 -- 172.21.15.42:0/728224601 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f259c003d90 con 0x7f25a4109820 2024-07-26T06:50:00.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.188+0000 7f25a9b4b640 1 -- 172.21.15.42:0/728224601 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a41a0060 con 0x7f25a4109820 2024-07-26T06:50:00.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.188+0000 7f25a0ff9640 1 -- 172.21.15.42:0/728224601 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c02fbc0 con 0x7f25a4109820 2024-07-26T06:50:00.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.188+0000 7f25a9b4b640 1 -- 172.21.15.42:0/728224601 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a41a68e0 con 0x7f25a4109820 2024-07-26T06:50:00.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.189+0000 7f25a9b4b640 1 -- 172.21.15.42:0/728224601 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a41039a0 con 0x7f25a4109820 2024-07-26T06:50:00.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.191+0000 7f25a0ff9640 1 -- 172.21.15.42:0/728224601 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f259c02fd60 con 0x7f25a4109820 2024-07-26T06:50:00.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.191+0000 7f25a0ff9640 1 --2- 172.21.15.42:0/728224601 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f258c0b3c80 0x7f258c0b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:00.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.192+0000 7f25a37fe640 1 --2- 172.21.15.42:0/728224601 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f258c0b3c80 0x7f258c0b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:00.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.192+0000 7f25a0ff9640 1 -- 172.21.15.42:0/728224601 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f259c1051e0 con 0x7f25a4109820 2024-07-26T06:50:00.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.192+0000 7f25a37fe640 1 --2- 172.21.15.42:0/728224601 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f258c0b3c80 0x7f258c0b6190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2594009b40 tx=0x7f2594005e50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:00.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.196+0000 7f25a0ff9640 1 -- 172.21.15.42:0/728224601 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f259c0cc810 con 0x7f25a4109820 2024-07-26T06:50:00.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.417+0000 7f25a9b4b640 1 -- 172.21.15.42:0/728224601 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25a419b050 con 0x7f25a4109820 2024-07-26T06:50:00.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.418+0000 7f25a0ff9640 1 -- 172.21.15.42:0/728224601 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11978 (secure 0 0 0) 0x7f259c0cc110 con 0x7f25a4109820 2024-07-26T06:50:00.419 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:50:00.420 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":28,"btime":"2024-07-26T06:47:26:107129+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:47:25.125103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24578,24583,24593,24601,24620]},"id":1}]} 2024-07-26T06:50:00.420 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 28 2024-07-26T06:50:00.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.420+0000 7f25a9b4b640 1 -- 172.21.15.42:0/728224601 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f258c0b3c80 msgr2=0x7f258c0b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:00.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.420+0000 7f25a9b4b640 1 --2- 172.21.15.42:0/728224601 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f258c0b3c80 0x7f258c0b6190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2594009b40 tx=0x7f2594005e50 comp rx=0 tx=0).stop 2024-07-26T06:50:00.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.420+0000 7f25a9b4b640 1 -- 172.21.15.42:0/728224601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25a4109820 msgr2=0x7f25a419f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:00.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.420+0000 7f25a9b4b640 1 --2- 172.21.15.42:0/728224601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25a4109820 0x7f25a419f760 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f259c005cc0 tx=0x7f259c02f380 comp rx=0 tx=0).stop 2024-07-26T06:50:00.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.420+0000 7f25a9b4b640 1 -- 172.21.15.42:0/728224601 shutdown_connections 2024-07-26T06:50:00.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.420+0000 7f25a9b4b640 1 --2- 172.21.15.42:0/728224601 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f258c0b3c80 0x7f258c0b6190 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:00.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.420+0000 7f25a9b4b640 1 --2- 172.21.15.42:0/728224601 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25a4109820 0x7f25a419f760 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:00.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.420+0000 7f25a9b4b640 1 --2- 172.21.15.42:0/728224601 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25a4108ed0 0x7f25a419a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:00.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.420+0000 7f25a9b4b640 1 --2- 172.21.15.42:0/728224601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25a4103590 0x7f25a4199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:00.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.420+0000 7f25a9b4b640 1 -- 172.21.15.42:0/728224601 >> 172.21.15.42:0/728224601 conn(0x7f25a407b280 msgr2=0x7f25a4106f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:00.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.420+0000 7f25a9b4b640 1 -- 172.21.15.42:0/728224601 shutdown_connections 2024-07-26T06:50:00.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:00.420+0000 7f25a9b4b640 1 -- 172.21.15.42:0/728224601 wait complete. 2024-07-26T06:50:00.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:00 smithi164 ceph-mon[36970]: overall HEALTH_OK 2024-07-26T06:50:00.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:00 smithi163 ceph-mon[36873]: overall HEALTH_OK 2024-07-26T06:50:01.241 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:50:01.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:01 smithi042 ceph-mon[32775]: pgmap v389: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:50:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:01 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/728224601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:50:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:01 smithi164 ceph-mon[36970]: pgmap v389: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:50:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:01 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/728224601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:50:01.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:01 smithi163 ceph-mon[36873]: pgmap v389: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:50:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:01 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/728224601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:50:01.513 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:03.309 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:03 smithi163 ceph-mon[36873]: pgmap v390: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:50:03.314 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:03 smithi042 ceph-mon[32775]: pgmap v390: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:50:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.317+0000 7f8b6097e640 1 -- 172.21.15.42:0/2423771966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b5c102490 msgr2=0x7f8b5c1028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.317+0000 7f8b6097e640 1 --2- 172.21.15.42:0/2423771966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b5c102490 0x7f8b5c1028a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009ca0 tx=0x7f8b4c02a8c0 comp rx=0 tx=0).stop 2024-07-26T06:50:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.317+0000 7f8b6097e640 1 -- 172.21.15.42:0/2423771966 shutdown_connections 2024-07-26T06:50:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.317+0000 7f8b6097e640 1 --2- 172.21.15.42:0/2423771966 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8b5c108720 0x7f8b5c10ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.317+0000 7f8b6097e640 1 --2- 172.21.15.42:0/2423771966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b5c107dd0 0x7f8b5c1081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.317+0000 7f8b6097e640 1 --2- 172.21.15.42:0/2423771966 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b5c102490 0x7f8b5c1028a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.317+0000 7f8b6097e640 1 -- 172.21.15.42:0/2423771966 >> 172.21.15.42:0/2423771966 conn(0x7f8b5c079160 msgr2=0x7f8b5c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:03.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.317+0000 7f8b6097e640 1 -- 172.21.15.42:0/2423771966 shutdown_connections 2024-07-26T06:50:03.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.317+0000 7f8b6097e640 1 -- 172.21.15.42:0/2423771966 wait complete. 2024-07-26T06:50:03.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.318+0000 7f8b6097e640 1 Processor -- start 2024-07-26T06:50:03.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.318+0000 7f8b6097e640 1 -- start start 2024-07-26T06:50:03.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b6097e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b5c102490 0x7f8b5c199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:03.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b6097e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b5c107dd0 0x7f8b5c19a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:03.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b6097e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8b5c108720 0x7f8b5c19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:03.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b6097e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5c113570 con 0x7f8b5c102490 2024-07-26T06:50:03.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b6097e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5c1133f0 con 0x7f8b5c108720 2024-07-26T06:50:03.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b6097e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5c1136f0 con 0x7f8b5c107dd0 2024-07-26T06:50:03.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b5a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b5c102490 0x7f8b5c199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:03.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b5a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b5c102490 0x7f8b5c199fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33820/0 (socket says 172.21.15.42:33820) 2024-07-26T06:50:03.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b5a59c640 1 -- 172.21.15.42:0/1542418557 learned_addr learned my addr 172.21.15.42:0/1542418557 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:03.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b5ad9d640 1 --2- 172.21.15.42:0/1542418557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8b5c108720 0x7f8b5c19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:03.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b59d9b640 1 --2- 172.21.15.42:0/1542418557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b5c107dd0 0x7f8b5c19a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:03.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b5a59c640 1 -- 172.21.15.42:0/1542418557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b5c107dd0 msgr2=0x7f8b5c19a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:03.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b5a59c640 1 --2- 172.21.15.42:0/1542418557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b5c107dd0 0x7f8b5c19a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:03.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b5a59c640 1 -- 172.21.15.42:0/1542418557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8b5c108720 msgr2=0x7f8b5c19f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:03.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b5a59c640 1 --2- 172.21.15.42:0/1542418557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8b5c108720 0x7f8b5c19f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:03.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b5a59c640 1 -- 172.21.15.42:0/1542418557 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b5c19ff50 con 0x7f8b5c102490 2024-07-26T06:50:03.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b5ad9d640 1 --2- 172.21.15.42:0/1542418557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8b5c108720 0x7f8b5c19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:03.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b5a59c640 1 --2- 172.21.15.42:0/1542418557 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b5c102490 0x7f8b5c199fe0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c02ade0 tx=0x7f8b4c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:03.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.319+0000 7f8b4b7fe640 1 -- 172.21.15.42:0/1542418557 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c003d60 con 0x7f8b5c102490 2024-07-26T06:50:03.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.320+0000 7f8b6097e640 1 -- 172.21.15.42:0/1542418557 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b5c1a01e0 con 0x7f8b5c102490 2024-07-26T06:50:03.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.320+0000 7f8b4b7fe640 1 -- 172.21.15.42:0/1542418557 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b4c0044f0 con 0x7f8b5c102490 2024-07-26T06:50:03.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.320+0000 7f8b6097e640 1 -- 172.21.15.42:0/1542418557 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b5c1a69a0 con 0x7f8b5c102490 2024-07-26T06:50:03.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.320+0000 7f8b4b7fe640 1 -- 172.21.15.42:0/1542418557 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c02fd10 con 0x7f8b5c102490 2024-07-26T06:50:03.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.321+0000 7f8b6097e640 1 -- 172.21.15.42:0/1542418557 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b5c102920 con 0x7f8b5c102490 2024-07-26T06:50:03.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.322+0000 7f8b4b7fe640 1 -- 172.21.15.42:0/1542418557 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f8b4c0046a0 con 0x7f8b5c102490 2024-07-26T06:50:03.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.323+0000 7f8b4b7fe640 1 --2- 172.21.15.42:0/1542418557 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f8b440b3bd0 0x7f8b440b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:03.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.323+0000 7f8b59d9b640 1 --2- 172.21.15.42:0/1542418557 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f8b440b3bd0 0x7f8b440b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:03.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.323+0000 7f8b4b7fe640 1 -- 172.21.15.42:0/1542418557 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f8b4c1055a0 con 0x7f8b5c102490 2024-07-26T06:50:03.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.323+0000 7f8b59d9b640 1 --2- 172.21.15.42:0/1542418557 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f8b440b3bd0 0x7f8b440b60e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c079860 tx=0x7f8b50008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:03.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.326+0000 7f8b4b7fe640 1 -- 172.21.15.42:0/1542418557 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8b4c03b410 con 0x7f8b5c102490 2024-07-26T06:50:03.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:03 smithi164 ceph-mon[36970]: pgmap v390: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:50:03.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.544+0000 7f8b6097e640 1 -- 172.21.15.42:0/1542418557 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b5c10a4f0 con 0x7f8b5c102490 2024-07-26T06:50:03.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.545+0000 7f8b4b7fe640 1 -- 172.21.15.42:0/1542418557 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11978 (secure 0 0 0) 0x7f8b4c0ccbd0 con 0x7f8b5c102490 2024-07-26T06:50:03.547 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:50:03.547 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":28,"btime":"2024-07-26T06:47:26:107129+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:47:25.125103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24578,24583,24593,24601,24620]},"id":1}]} 2024-07-26T06:50:03.547 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 28 2024-07-26T06:50:03.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.547+0000 7f8b6097e640 1 -- 172.21.15.42:0/1542418557 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f8b440b3bd0 msgr2=0x7f8b440b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:03.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.547+0000 7f8b6097e640 1 --2- 172.21.15.42:0/1542418557 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f8b440b3bd0 0x7f8b440b60e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c079860 tx=0x7f8b50008040 comp rx=0 tx=0).stop 2024-07-26T06:50:03.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.547+0000 7f8b6097e640 1 -- 172.21.15.42:0/1542418557 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b5c102490 msgr2=0x7f8b5c199fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:03.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.547+0000 7f8b6097e640 1 --2- 172.21.15.42:0/1542418557 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b5c102490 0x7f8b5c199fe0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c02ade0 tx=0x7f8b4c003970 comp rx=0 tx=0).stop 2024-07-26T06:50:03.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.547+0000 7f8b6097e640 1 -- 172.21.15.42:0/1542418557 shutdown_connections 2024-07-26T06:50:03.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.547+0000 7f8b6097e640 1 --2- 172.21.15.42:0/1542418557 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f8b440b3bd0 0x7f8b440b60e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:03.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.547+0000 7f8b6097e640 1 --2- 172.21.15.42:0/1542418557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8b5c108720 0x7f8b5c19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:03.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.547+0000 7f8b6097e640 1 --2- 172.21.15.42:0/1542418557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b5c107dd0 0x7f8b5c19a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:03.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.547+0000 7f8b6097e640 1 --2- 172.21.15.42:0/1542418557 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b5c102490 0x7f8b5c199fe0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:03.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.547+0000 7f8b6097e640 1 -- 172.21.15.42:0/1542418557 >> 172.21.15.42:0/1542418557 conn(0x7f8b5c079160 msgr2=0x7f8b5c100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:03.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.547+0000 7f8b6097e640 1 -- 172.21.15.42:0/1542418557 shutdown_connections 2024-07-26T06:50:03.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:03.547+0000 7f8b6097e640 1 -- 172.21.15.42:0/1542418557 wait complete. 2024-07-26T06:50:04.371 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:04 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1542418557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:50:04.403 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T06:50:04.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:04 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1542418557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:50:04.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:04 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1542418557' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:50:04.671 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:05.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:05 smithi042 ceph-mon[32775]: pgmap v391: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:50:05.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:05 smithi164 ceph-mon[36970]: pgmap v391: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:50:05.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:05 smithi163 ceph-mon[36873]: pgmap v391: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:50:06.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.410+0000 7fde91d0c640 1 -- 172.21.15.42:0/3423552779 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde8c107ef0 msgr2=0x7fde8c10e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:06.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.410+0000 7fde91d0c640 1 --2- 172.21.15.42:0/3423552779 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde8c107ef0 0x7fde8c10e720 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fde84009ca0 tx=0x7fde8402a8c0 comp rx=0 tx=0).stop 2024-07-26T06:50:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.410+0000 7fde91d0c640 1 -- 172.21.15.42:0/3423552779 shutdown_connections 2024-07-26T06:50:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.410+0000 7fde91d0c640 1 --2- 172.21.15.42:0/3423552779 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde8c107ef0 0x7fde8c10e720 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.410+0000 7fde91d0c640 1 --2- 172.21.15.42:0/3423552779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde8c1075a0 0x7fde8c1079b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.410+0000 7fde91d0c640 1 --2- 172.21.15.42:0/3423552779 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde8c101c60 0x7fde8c102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.410+0000 7fde91d0c640 1 -- 172.21.15.42:0/3423552779 >> 172.21.15.42:0/3423552779 conn(0x7fde8c079870 msgr2=0x7fde8c079c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.410+0000 7fde91d0c640 1 -- 172.21.15.42:0/3423552779 shutdown_connections 2024-07-26T06:50:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.410+0000 7fde91d0c640 1 -- 172.21.15.42:0/3423552779 wait complete. 2024-07-26T06:50:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.411+0000 7fde91d0c640 1 Processor -- start 2024-07-26T06:50:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.411+0000 7fde91d0c640 1 -- start start 2024-07-26T06:50:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.411+0000 7fde91d0c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde8c101c60 0x7fde8c0fe8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.411+0000 7fde91d0c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde8c1075a0 0x7fde8c0fede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.411+0000 7fde91d0c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde8c107ef0 0x7fde8c0ff320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.411+0000 7fde91d0c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fde8c111330 con 0x7fde8c107ef0 2024-07-26T06:50:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.411+0000 7fde91d0c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fde8c1111b0 con 0x7fde8c1075a0 2024-07-26T06:50:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.411+0000 7fde91d0c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fde8c1114b0 con 0x7fde8c101c60 2024-07-26T06:50:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.412+0000 7fde8bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde8c107ef0 0x7fde8c0ff320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.412+0000 7fde8bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde8c107ef0 0x7fde8c0ff320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33834/0 (socket says 172.21.15.42:33834) 2024-07-26T06:50:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.412+0000 7fde8bfff640 1 -- 172.21.15.42:0/2923112705 learned_addr learned my addr 172.21.15.42:0/2923112705 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.412+0000 7fde82ffd640 1 --2- 172.21.15.42:0/2923112705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde8c1075a0 0x7fde8c0fede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.412+0000 7fde8b7fe640 1 --2- 172.21.15.42:0/2923112705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde8c101c60 0x7fde8c0fe8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.412+0000 7fde8bfff640 1 -- 172.21.15.42:0/2923112705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde8c101c60 msgr2=0x7fde8c0fe8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.412+0000 7fde8bfff640 1 --2- 172.21.15.42:0/2923112705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde8c101c60 0x7fde8c0fe8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.412+0000 7fde8bfff640 1 -- 172.21.15.42:0/2923112705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde8c1075a0 msgr2=0x7fde8c0fede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.412+0000 7fde8bfff640 1 --2- 172.21.15.42:0/2923112705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde8c1075a0 0x7fde8c0fede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.412+0000 7fde8bfff640 1 -- 172.21.15.42:0/2923112705 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8c1a80b0 con 0x7fde8c107ef0 2024-07-26T06:50:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.412+0000 7fde82ffd640 1 --2- 172.21.15.42:0/2923112705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde8c1075a0 0x7fde8c0fede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.412+0000 7fde8bfff640 1 --2- 172.21.15.42:0/2923112705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde8c107ef0 0x7fde8c0ff320 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fde84005cc0 tx=0x7fde8402f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.412+0000 7fde897fa640 1 -- 172.21.15.42:0/2923112705 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84003bf0 con 0x7fde8c107ef0 2024-07-26T06:50:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.412+0000 7fde897fa640 1 -- 172.21.15.42:0/2923112705 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde84003d90 con 0x7fde8c107ef0 2024-07-26T06:50:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.412+0000 7fde91d0c640 1 -- 172.21.15.42:0/2923112705 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde8c1a8340 con 0x7fde8c107ef0 2024-07-26T06:50:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.412+0000 7fde897fa640 1 -- 172.21.15.42:0/2923112705 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8402fbc0 con 0x7fde8c107ef0 2024-07-26T06:50:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.413+0000 7fde91d0c640 1 -- 172.21.15.42:0/2923112705 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde8c1a8880 con 0x7fde8c107ef0 2024-07-26T06:50:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.413+0000 7fde91d0c640 1 -- 172.21.15.42:0/2923112705 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde8c102070 con 0x7fde8c107ef0 2024-07-26T06:50:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.415+0000 7fde897fa640 1 -- 172.21.15.42:0/2923112705 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fde8402fd60 con 0x7fde8c107ef0 2024-07-26T06:50:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.416+0000 7fde897fa640 1 --2- 172.21.15.42:0/2923112705 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fde780b3bc0 0x7fde780b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.416+0000 7fde8b7fe640 1 --2- 172.21.15.42:0/2923112705 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fde780b3bc0 0x7fde780b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.416+0000 7fde897fa640 1 -- 172.21.15.42:0/2923112705 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fde841055d0 con 0x7fde8c107ef0 2024-07-26T06:50:06.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.416+0000 7fde8b7fe640 1 --2- 172.21.15.42:0/2923112705 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fde780b3bc0 0x7fde780b60d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fde7c005fd0 tx=0x7fde7c005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:06.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.419+0000 7fde897fa640 1 -- 172.21.15.42:0/2923112705 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fde840ccc00 con 0x7fde8c107ef0 2024-07-26T06:50:06.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.601+0000 7fde91d0c640 1 -- 172.21.15.42:0/2923112705 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fde8c02e0c0 con 0x7fde8c107ef0 2024-07-26T06:50:06.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.603+0000 7fde897fa640 1 -- 172.21.15.42:0/2923112705 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22697 (secure 0 0 0) 0x7fde840cc500 con 0x7fde8c107ef0 2024-07-26T06:50:06.605 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:50:06.605 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":80,"fsid":"8b3e2750-4b19-11ef-bcb6-c7b262605968","created":"2024-07-26T06:38:29.306470+0000","modified":"2024-07-26T06:47:23.086609+0000","last_up_change":"2024-07-26T06:46:02.149453+0000","last_in_change":"2024-07-26T06:45:43.027764+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:42:27.302195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:47:00.042262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T06:47:03.674160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bda88e0-09ee-447b-8c04-24f4e5d55d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6803","nonce":3587066379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6805","nonce":3587066379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6809","nonce":3587066379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6807","nonce":3587066379}]},"public_addr":"172.21.15.42:6803/3587066379","cluster_addr":"172.21.15.42:6805/3587066379","heartbeat_back_addr":"172.21.15.42:6809/3587066379","heartbeat_front_addr":"172.21.15.42:6807/3587066379","state":["exists","up"]},{"osd":1,"uuid":"7d747138-8217-4836-a173-cf8ac65e9fe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6801","nonce":360219078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6803","nonce":360219078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6807","nonce":360219078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6805","nonce":360219078}]},"public_addr":"172.21.15.163:6801/360219078","cluster_addr":"172.21.15.163:6803/360219078","heartbeat_back_addr":"172.21.15.163:6807/360219078","heartbeat_front_addr":"172.21.15.163:6805/360219078","state":["exists","up"]},{"osd":2,"uuid":"4b7258c1-3d35-4930-ab10-0b04a2a2c8a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6801","nonce":336113412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6803","nonce":336113412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6807","nonce":336113412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6805","nonce":336113412}]},"public_addr":"172.21.15.164:6801/336113412","cluster_addr":"172.21.15.164:6803/336113412","heartbeat_back_addr":"172.21.15.164:6807/336113412","heartbeat_front_addr":"172.21.15.164:6805/336113412","state":["exists","up"]},{"osd":3,"uuid":"96608387-0de0-4d90-a952-d6eba61f7fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6811","nonce":2819018064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6813","nonce":2819018064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6817","nonce":2819018064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6815","nonce":2819018064}]},"public_addr":"172.21.15.42:6811/2819018064","cluster_addr":"172.21.15.42:6813/2819018064","heartbeat_back_addr":"172.21.15.42:6817/2819018064","heartbeat_front_addr":"172.21.15.42:6815/2819018064","state":["exists","up"]},{"osd":4,"uuid":"fabf6b02-fe34-4560-85b5-01bf37fc9be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6809","nonce":431934374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6811","nonce":431934374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6815","nonce":431934374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6813","nonce":431934374}]},"public_addr":"172.21.15.163:6809/431934374","cluster_addr":"172.21.15.163:6811/431934374","heartbeat_back_addr":"172.21.15.163:6815/431934374","heartbeat_front_addr":"172.21.15.163:6813/431934374","state":["exists","up"]},{"osd":5,"uuid":"e226fb4f-15b2-40df-af34-900040ba93ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6809","nonce":261549362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6811","nonce":261549362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6815","nonce":261549362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6813","nonce":261549362}]},"public_addr":"172.21.15.164:6809/261549362","cluster_addr":"172.21.15.164:6811/261549362","heartbeat_back_addr":"172.21.15.164:6815/261549362","heartbeat_front_addr":"172.21.15.164:6813/261549362","state":["exists","up"]},{"osd":6,"uuid":"beb8ec5d-7ea5-4e4e-a46c-a84f7768cec0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6819","nonce":2160811300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6821","nonce":2160811300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6825","nonce":2160811300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6823","nonce":2160811300}]},"public_addr":"172.21.15.42:6819/2160811300","cluster_addr":"172.21.15.42:6821/2160811300","heartbeat_back_addr":"172.21.15.42:6825/2160811300","heartbeat_front_addr":"172.21.15.42:6823/2160811300","state":["exists","up"]},{"osd":7,"uuid":"f699004c-f7c2-4f1c-ae4b-7178f3b06e2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6817","nonce":812214111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6819","nonce":812214111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6823","nonce":812214111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6821","nonce":812214111}]},"public_addr":"172.21.15.163:6817/812214111","cluster_addr":"172.21.15.163:6819/812214111","heartbeat_back_addr":"172.21.15.163:6823/812214111","heartbeat_front_addr":"172.21.15.163:6821/812214111","state":["exists","up"]},{"osd":8,"uuid":"3476163e-efea-4652-a8d4-b7c4a908f984","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6817","nonce":962845839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6819","nonce":962845839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6823","nonce":962845839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6821","nonce":962845839}]},"public_addr":"172.21.15.164:6817/962845839","cluster_addr":"172.21.15.164:6819/962845839","heartbeat_back_addr":"172.21.15.164:6823/962845839","heartbeat_front_addr":"172.21.15.164:6821/962845839","state":["exists","up"]},{"osd":9,"uuid":"6f7ceac6-fb61-4c6f-9da8-851b937c4586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6827","nonce":994024418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6829","nonce":994024418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6833","nonce":994024418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6831","nonce":994024418}]},"public_addr":"172.21.15.42:6827/994024418","cluster_addr":"172.21.15.42:6829/994024418","heartbeat_back_addr":"172.21.15.42:6833/994024418","heartbeat_front_addr":"172.21.15.42:6831/994024418","state":["exists","up"]},{"osd":10,"uuid":"553132c2-a6a1-410c-a7d8-ecdf9a67dafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6825","nonce":3268512227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6827","nonce":3268512227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6831","nonce":3268512227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6829","nonce":3268512227}]},"public_addr":"172.21.15.163:6825/3268512227","cluster_addr":"172.21.15.163:6827/3268512227","heartbeat_back_addr":"172.21.15.163:6831/3268512227","heartbeat_front_addr":"172.21.15.163:6829/3268512227","state":["exists","up"]},{"osd":11,"uuid":"09abc8e9-7884-400f-ab6d-8e5aa7bf31ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6825","nonce":1861076336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6827","nonce":1861076336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6831","nonce":1861076336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6829","nonce":1861076336}]},"public_addr":"172.21.15.164:6825/1861076336","cluster_addr":"172.21.15.164:6827/1861076336","heartbeat_back_addr":"172.21.15.164:6831/1861076336","heartbeat_front_addr":"172.21.15.164:6829/1861076336","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:41:38.761674+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:00.621745+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:23.248620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:47.232764+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:10.379922+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:34.502519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:58.694642+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:22.350247+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:46.988007+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:11.390474+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:36.150054+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:46:00.474470+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.42:6801/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6800/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1842521399":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/834172950":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1730109419":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6801/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:6800/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2687772889":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/3371433307":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2861719816":"2024-07-27T06:38:56.887264+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:50:06.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.606+0000 7fde91d0c640 1 -- 172.21.15.42:0/2923112705 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fde780b3bc0 msgr2=0x7fde780b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:06.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.606+0000 7fde91d0c640 1 --2- 172.21.15.42:0/2923112705 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fde780b3bc0 0x7fde780b60d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fde7c005fd0 tx=0x7fde7c005850 comp rx=0 tx=0).stop 2024-07-26T06:50:06.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.606+0000 7fde91d0c640 1 -- 172.21.15.42:0/2923112705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde8c107ef0 msgr2=0x7fde8c0ff320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:06.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.606+0000 7fde91d0c640 1 --2- 172.21.15.42:0/2923112705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde8c107ef0 0x7fde8c0ff320 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fde84005cc0 tx=0x7fde8402f380 comp rx=0 tx=0).stop 2024-07-26T06:50:06.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.606+0000 7fde91d0c640 1 -- 172.21.15.42:0/2923112705 shutdown_connections 2024-07-26T06:50:06.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.606+0000 7fde91d0c640 1 --2- 172.21.15.42:0/2923112705 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fde780b3bc0 0x7fde780b60d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:06.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.606+0000 7fde91d0c640 1 --2- 172.21.15.42:0/2923112705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde8c107ef0 0x7fde8c0ff320 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:06.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.606+0000 7fde91d0c640 1 --2- 172.21.15.42:0/2923112705 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde8c1075a0 0x7fde8c0fede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:06.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.606+0000 7fde91d0c640 1 --2- 172.21.15.42:0/2923112705 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde8c101c60 0x7fde8c0fe8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:06.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.606+0000 7fde91d0c640 1 -- 172.21.15.42:0/2923112705 >> 172.21.15.42:0/2923112705 conn(0x7fde8c079870 msgr2=0x7fde8c103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:06.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.606+0000 7fde91d0c640 1 -- 172.21.15.42:0/2923112705 shutdown_connections 2024-07-26T06:50:06.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:06.606+0000 7fde91d0c640 1 -- 172.21.15.42:0/2923112705 wait complete. 2024-07-26T06:50:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:07 smithi164 ceph-mon[36970]: pgmap v392: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:50:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:07 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2923112705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:50:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:07 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:50:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:07 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:07 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:07 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:07 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:07.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:07 smithi163 ceph-mon[36873]: pgmap v392: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:50:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:07 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2923112705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:50:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:07 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:50:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:07 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:07 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:07 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:07 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:07.317 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:07 smithi042 ceph-mon[32775]: pgmap v392: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:50:07.317 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:07 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2923112705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:50:07.317 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:07 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:50:07.317 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:07 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:07.317 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:07 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:07.317 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:07 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:07.317 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:07 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:07.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-07-26T06:47:06.780568+0000', 'modified': '2024-07-26T06:47:25.125103+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24620, 'mds_1': 24601, 'mds_2': 24593, 'mds_3': 24583, 'mds_4': 24578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14703': {'gid': 14703, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.42:6839/4044632348', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.42:6838', 'nonce': 4044632348}, {'type': 'v1', 'addr': '172.21.15.42:6839', 'nonce': 4044632348}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14709': {'gid': 14709, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.42:6841/1430155791', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.42:6840', 'nonce': 1430155791}, {'type': 'v1', 'addr': '172.21.15.42:6841', 'nonce': 1430155791}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24541': {'gid': 24541, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.164:6833/4144543280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6832', 'nonce': 4144543280}, {'type': 'v1', 'addr': '172.21.15.164:6833', 'nonce': 4144543280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24557': {'gid': 24557, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.163:6833/845444723', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6832', 'nonce': 845444723}, {'type': 'v1', 'addr': '172.21.15.163:6833', 'nonce': 845444723}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24571': {'gid': 24571, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.164:6835/2604534474', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6834', 'nonce': 2604534474}, {'type': 'v1', 'addr': '172.21.15.164:6835', 'nonce': 2604534474}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24578': {'gid': 24578, 'name': 'e', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.163:6835/375978170', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6834', 'nonce': 375978170}, {'type': 'v1', 'addr': '172.21.15.163:6835', 'nonce': 375978170}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24583': {'gid': 24583, 'name': 'c', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.164:6837/3642457979', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6836', 'nonce': 3642457979}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 3642457979}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24593': {'gid': 24593, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.163:6837/154041066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6836', 'nonce': 154041066}, {'type': 'v1', 'addr': '172.21.15.163:6837', 'nonce': 154041066}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24601': {'gid': 24601, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.164:6839/3593178314', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6838', 'nonce': 3593178314}, {'type': 'v1', 'addr': '172.21.15.164:6839', 'nonce': 3593178314}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24620': {'gid': 24620, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.163:6839/563658453', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6838', 'nonce': 563658453}, {'type': 'v1', 'addr': '172.21.15.163:6839', 'nonce': 563658453}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24620, 'qdb_cluster': [24578, 24583, 24593, 24601, 24620]} 2024-07-26T06:50:07.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-07-26T06:50:07.348 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell mds.e status 2024-07-26T06:50:07.643 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:08 smithi164 ceph-mon[36970]: pgmap v393: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:50:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:08 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:08 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:08.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:08 smithi163 ceph-mon[36873]: pgmap v393: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:50:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:08 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:08 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:08.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:08 smithi042 ceph-mon[32775]: pgmap v393: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:50:08.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:08 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:08.861 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:08 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:09.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.399+0000 7f771e7be640 1 -- 172.21.15.42:0/197556617 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7718107df0 msgr2=0x7f7718108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:09.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.399+0000 7f771e7be640 1 --2- 172.21.15.42:0/197556617 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7718107df0 0x7f7718108200 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f770c009900 tx=0x7f770c02a410 comp rx=0 tx=0).stop 2024-07-26T06:50:09.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.399+0000 7f771e7be640 1 -- 172.21.15.42:0/197556617 shutdown_connections 2024-07-26T06:50:09.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.399+0000 7f771e7be640 1 --2- 172.21.15.42:0/197556617 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7718108740 0x7f771810ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.399+0000 7f771e7be640 1 --2- 172.21.15.42:0/197556617 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7718107df0 0x7f7718108200 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.399+0000 7f771e7be640 1 --2- 172.21.15.42:0/197556617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77181024b0 0x7f77181028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.399+0000 7f771e7be640 1 -- 172.21.15.42:0/197556617 >> 172.21.15.42:0/197556617 conn(0x7f7718079170 msgr2=0x7f7718079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:09.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.399+0000 7f771e7be640 1 -- 172.21.15.42:0/197556617 shutdown_connections 2024-07-26T06:50:09.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.399+0000 7f771e7be640 1 -- 172.21.15.42:0/197556617 wait complete. 2024-07-26T06:50:09.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.400+0000 7f771e7be640 1 Processor -- start 2024-07-26T06:50:09.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.400+0000 7f771e7be640 1 -- start start 2024-07-26T06:50:09.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.400+0000 7f771e7be640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77181024b0 0x7f7718198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.400+0000 7f771e7be640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7718107df0 0x7f7718199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.400+0000 7f771e7be640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7718108740 0x7f771819e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.400+0000 7f771e7be640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f77181123f0 con 0x7f77181024b0 2024-07-26T06:50:09.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.400+0000 7f771e7be640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7718112270 con 0x7f7718107df0 2024-07-26T06:50:09.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.401+0000 7f771e7be640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7718112570 con 0x7f7718108740 2024-07-26T06:50:09.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.401+0000 7f7717fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77181024b0 0x7f7718198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.401+0000 7f7717fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77181024b0 0x7f7718198fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44930/0 (socket says 172.21.15.42:44930) 2024-07-26T06:50:09.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.401+0000 7f7717fff640 1 -- 172.21.15.42:0/3187811616 learned_addr learned my addr 172.21.15.42:0/3187811616 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:09.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.401+0000 7f771cd5b640 1 --2- 172.21.15.42:0/3187811616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7718108740 0x7f771819e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.401+0000 7f77177fe640 1 --2- 172.21.15.42:0/3187811616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7718107df0 0x7f7718199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.401+0000 7f7717fff640 1 -- 172.21.15.42:0/3187811616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7718108740 msgr2=0x7f771819e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:09.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.401+0000 7f7717fff640 1 --2- 172.21.15.42:0/3187811616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7718108740 0x7f771819e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.401+0000 7f7717fff640 1 -- 172.21.15.42:0/3187811616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7718107df0 msgr2=0x7f7718199580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:09.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.401+0000 7f7717fff640 1 --2- 172.21.15.42:0/3187811616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7718107df0 0x7f7718199580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.401+0000 7f7717fff640 1 -- 172.21.15.42:0/3187811616 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f771819ef50 con 0x7f77181024b0 2024-07-26T06:50:09.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.401+0000 7f771cd5b640 1 --2- 172.21.15.42:0/3187811616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7718108740 0x7f771819e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:09.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.401+0000 7f7717fff640 1 --2- 172.21.15.42:0/3187811616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77181024b0 0x7f7718198fe0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f770800c890 tx=0x7f770800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:09.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.401+0000 7f77157fa640 1 -- 172.21.15.42:0/3187811616 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7708004220 con 0x7f77181024b0 2024-07-26T06:50:09.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.401+0000 7f77157fa640 1 -- 172.21.15.42:0/3187811616 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77080043c0 con 0x7f77181024b0 2024-07-26T06:50:09.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.401+0000 7f771e7be640 1 -- 172.21.15.42:0/3187811616 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f771819f240 con 0x7f77181024b0 2024-07-26T06:50:09.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.401+0000 7f77157fa640 1 -- 172.21.15.42:0/3187811616 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7708004a00 con 0x7f77181024b0 2024-07-26T06:50:09.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.402+0000 7f771e7be640 1 -- 172.21.15.42:0/3187811616 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77181a59a0 con 0x7f77181024b0 2024-07-26T06:50:09.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.404+0000 7f7724d42740 1 Processor -- start 2024-07-26T06:50:09.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.404+0000 7f7724d42740 1 -- start start 2024-07-26T06:50:09.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.404+0000 7f77157fa640 1 -- 172.21.15.42:0/3187811616 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f7708004ba0 con 0x7f77181024b0 2024-07-26T06:50:09.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.404+0000 7f7724d42740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b1304b3f60 0x55b13085b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.404+0000 7f7724d42740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b13048b0a0 0x55b13085baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.404+0000 7f7724d42740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b130493280 0x55b13085c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.404+0000 7f7724d42740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55b1307cf940 con 0x55b1304b3f60 2024-07-26T06:50:09.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.404+0000 7f7724d42740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b1307bee40 con 0x55b13048b0a0 2024-07-26T06:50:09.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.404+0000 7f7724d42740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55b1307c2e80 con 0x55b130493280 2024-07-26T06:50:09.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.405+0000 7f771cd5b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b1304b3f60 0x55b13085b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.405+0000 7f771cd5b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b1304b3f60 0x55b13085b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44932/0 (socket says 172.21.15.42:44932) 2024-07-26T06:50:09.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.405+0000 7f771cd5b640 1 -- 172.21.15.42:0/1930007502 learned_addr learned my addr 172.21.15.42:0/1930007502 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:09.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.405+0000 7f7717fff640 1 --2- 172.21.15.42:0/1930007502 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b13048b0a0 0x55b13085baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.405+0000 7f77177fe640 1 --2- 172.21.15.42:0/1930007502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b130493280 0x55b13085c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.405+0000 7f771cd5b640 1 -- 172.21.15.42:0/1930007502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b130493280 msgr2=0x55b13085c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:09.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.405+0000 7f771cd5b640 1 --2- 172.21.15.42:0/1930007502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b130493280 0x55b13085c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.405+0000 7f771cd5b640 1 -- 172.21.15.42:0/1930007502 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b13048b0a0 msgr2=0x55b13085baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:09.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.405+0000 7f771cd5b640 1 --2- 172.21.15.42:0/1930007502 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b13048b0a0 0x55b13085baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.405+0000 7f771cd5b640 1 -- 172.21.15.42:0/1930007502 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b130765a60 con 0x55b1304b3f60 2024-07-26T06:50:09.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.405+0000 7f771cd5b640 1 --2- 172.21.15.42:0/1930007502 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b1304b3f60 0x55b13085b5b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f771000c450 tx=0x7f771000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b085a01de79264ee server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:09.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.405+0000 7f76fe7fc640 1 -- 172.21.15.42:0/1930007502 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7710007430 con 0x55b1304b3f60 2024-07-26T06:50:09.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.405+0000 7f77157fa640 1 --2- 172.21.15.42:0/3187811616 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f77000b3c50 0x7f77000b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.406+0000 7f76fe7fc640 1 -- 172.21.15.42:0/1930007502 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7710007a80 con 0x55b1304b3f60 2024-07-26T06:50:09.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.406+0000 7f76fe7fc640 1 -- 172.21.15.42:0/1930007502 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771000a9f0 con 0x55b1304b3f60 2024-07-26T06:50:09.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.406+0000 7f7717fff640 1 --2- 172.21.15.42:0/3187811616 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f77000b3c50 0x7f77000b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.406+0000 7f77157fa640 1 -- 172.21.15.42:0/3187811616 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f77080e3a00 con 0x7f77181024b0 2024-07-26T06:50:09.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.406+0000 7f7717fff640 1 --2- 172.21.15.42:0/3187811616 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f77000b3c50 0x7f77000b6160 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x55b13085a230 tx=0x7f7708092040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:09.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.406+0000 7f7724d42740 1 -- 172.21.15.42:0/1930007502 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b1304b3f60 msgr2=0x55b13085b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:09.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.407+0000 7f7724d42740 1 --2- 172.21.15.42:0/1930007502 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b1304b3f60 0x55b13085b5b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f771000c450 tx=0x7f771000c910 comp rx=0 tx=0).stop 2024-07-26T06:50:09.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.407+0000 7f7724d42740 1 -- 172.21.15.42:0/1930007502 shutdown_connections 2024-07-26T06:50:09.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.407+0000 7f7724d42740 1 --2- 172.21.15.42:0/1930007502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b130493280 0x55b13085c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.407+0000 7f7724d42740 1 --2- 172.21.15.42:0/1930007502 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b13048b0a0 0x55b13085baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.407+0000 7f7724d42740 1 --2- 172.21.15.42:0/1930007502 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b1304b3f60 0x55b13085b5b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.407+0000 7f7724d42740 1 -- 172.21.15.42:0/1930007502 >> 172.21.15.42:0/1930007502 conn(0x55b1304a8350 msgr2=0x55b1308594a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:09.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.407+0000 7f7724d42740 1 -- 172.21.15.42:0/1930007502 shutdown_connections 2024-07-26T06:50:09.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.407+0000 7f7724d42740 1 -- 172.21.15.42:0/1930007502 wait complete. 2024-07-26T06:50:09.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.407+0000 7f7724d42740 1 -- start start 2024-07-26T06:50:09.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.407+0000 7f7724d42740 1 Processor -- start 2024-07-26T06:50:09.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f7724d42740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b13048b0a0 0x55b130867210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f7724d42740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b130493280 0x55b130867790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f7724d42740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b1304b3f60 0x55b130873080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f7724d42740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55b1306225f0 con 0x55b13048b0a0 2024-07-26T06:50:09.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f7724d42740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b1307618b0 con 0x55b130493280 2024-07-26T06:50:09.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f7724d42740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55b130869d90 con 0x55b1304b3f60 2024-07-26T06:50:09.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f7724d42740 10 client.0 _ll_register_callbacks cb 0x55b13076b7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:09.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f771cd5b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b13048b0a0 0x55b130867210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f771cd5b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b13048b0a0 0x55b130867210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44938/0 (socket says 172.21.15.42:44938) 2024-07-26T06:50:09.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f771cd5b640 1 -- 172.21.15.42:0/2310440492 learned_addr learned my addr 172.21.15.42:0/2310440492 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:09.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f771cd5b640 1 --2- 172.21.15.42:0/2310440492 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b1304b3f60 0x55b130873080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f77177fe640 1 --2- 172.21.15.42:0/2310440492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b130493280 0x55b130867790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f771cd5b640 1 -- 172.21.15.42:0/2310440492 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b1304b3f60 msgr2=0x55b130873080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:09.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f771cd5b640 1 --2- 172.21.15.42:0/2310440492 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b1304b3f60 0x55b130873080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f771cd5b640 1 -- 172.21.15.42:0/2310440492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b130493280 msgr2=0x55b130867790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:09.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f771cd5b640 1 --2- 172.21.15.42:0/2310440492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b130493280 0x55b130867790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f771cd5b640 1 -- 172.21.15.42:0/2310440492 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b13051f540 con 0x55b13048b0a0 2024-07-26T06:50:09.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f771cd5b640 1 --2- 172.21.15.42:0/2310440492 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b13048b0a0 0x55b130867210 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f771000a570 tx=0x7f7710009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815ef9daff519a8d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:09.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.408+0000 7f76f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:09.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.409+0000 7f76f77fe640 1 -- 172.21.15.42:0/2310440492 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7710028060 con 0x55b13048b0a0 2024-07-26T06:50:09.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.409+0000 7f76f77fe640 1 -- 172.21.15.42:0/2310440492 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f7710007880 con 0x55b13048b0a0 2024-07-26T06:50:09.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.409+0000 7f7724d42740 1 -- 172.21.15.42:0/2310440492 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7710007a80 con 0x55b13048b0a0 2024-07-26T06:50:09.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.409+0000 7f76f77fe640 1 -- 172.21.15.42:0/2310440492 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f771001b040 con 0x55b13048b0a0 2024-07-26T06:50:09.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.409+0000 7f7724d42740 10 client.15222 fetch_fsmap learned FSMap version 28 2024-07-26T06:50:09.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.409+0000 7f7724d42740 1 -- 172.21.15.42:0/2310440492 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b130875df0 con 0x55b13048b0a0 2024-07-26T06:50:09.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.409+0000 7f76f77fe640 1 -- 172.21.15.42:0/2310440492 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7710007a80 con 0x55b13048b0a0 2024-07-26T06:50:09.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.409+0000 7f76f77fe640 1 -- 172.21.15.42:0/2310440492 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f7710021040 con 0x55b13048b0a0 2024-07-26T06:50:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.409+0000 7f7724d42740 10 client.15222 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:50:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.409+0000 7f7724d42740 10 client.15222 resolve_mds: resolved name 'e' to daemon mds.e 2024-07-26T06:50:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.409+0000 7f7724d42740 20 client.15222 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.409+0000 7f7724d42740 1 --2- 172.21.15.42:0/2310440492 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x55b1304728e0 0x55b130877250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.409+0000 7f7724d42740 4 client.15222 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:50:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.409+0000 7f7724d42740 1 -- 172.21.15.42:0/2310440492 --> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1307d1890 con 0x55b1304728e0 2024-07-26T06:50:09.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.410+0000 7f77177fe640 1 --2- 172.21.15.42:0/2310440492 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x55b1304728e0 0x55b130877250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.410+0000 7f76f77fe640 10 client.15222 ms_handle_connect on v2:172.21.15.163:6834/375978170 2024-07-26T06:50:09.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.410+0000 7f77177fe640 1 --2- 172.21.15.42:0/2310440492 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x55b1304728e0 0x55b130877250 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3f0909e158ac03a4 server_cookie=94fa29db13bd8ea9 in_seq=0 out_seq=0 2024-07-26T06:50:09.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.411+0000 7f76f77fe640 1 -- 172.21.15.42:0/2310440492 <== mds.4 v2:172.21.15.163:6834/375978170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b1307d1890 con 0x55b1304728e0 2024-07-26T06:50:09.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.411+0000 7f76f77fe640 10 client.15222 handle_command_reply: tid=0 2024-07-26T06:50:09.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.411+0000 7f7724d42740 1 client.15222 shutdown 2024-07-26T06:50:09.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.411+0000 7f7724d42740 1 -- 172.21.15.42:0/2310440492 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b13048b0a0 msgr2=0x55b130867210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:09.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.411+0000 7f7724d42740 1 --2- 172.21.15.42:0/2310440492 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b13048b0a0 0x55b130867210 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f771000a570 tx=0x7f7710009b20 comp rx=0 tx=0).stop 2024-07-26T06:50:09.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.411+0000 7f7724d42740 1 -- 172.21.15.42:0/2310440492 shutdown_connections 2024-07-26T06:50:09.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.411+0000 7f7724d42740 1 --2- 172.21.15.42:0/2310440492 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x55b1304728e0 0x55b130877250 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.411+0000 7f7724d42740 1 --2- 172.21.15.42:0/2310440492 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b1304b3f60 0x55b130873080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.411+0000 7f7724d42740 1 --2- 172.21.15.42:0/2310440492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b130493280 0x55b130867790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.411+0000 7f7724d42740 1 --2- 172.21.15.42:0/2310440492 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b13048b0a0 0x55b130867210 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.411+0000 7f7724d42740 1 -- 172.21.15.42:0/2310440492 >> 172.21.15.42:0/2310440492 conn(0x55b1304a8350 msgr2=0x55b13085e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:09.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.411+0000 7f76f77fe640 0 client.15222 ms_handle_reset on v2:172.21.15.163:6834/375978170 2024-07-26T06:50:09.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.411+0000 7f7724d42740 1 -- 172.21.15.42:0/2310440492 shutdown_connections 2024-07-26T06:50:09.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.412+0000 7f7724d42740 1 -- 172.21.15.42:0/2310440492 wait complete. 2024-07-26T06:50:09.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.412+0000 7f7724d42740 20 client.15222 trim_cache size 0 max 16384 2024-07-26T06:50:09.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.430+0000 7f7724d42740 1 Processor -- start 2024-07-26T06:50:09.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.430+0000 7f7724d42740 1 -- start start 2024-07-26T06:50:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:50:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:50:09.431 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:09.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f7724d42740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b13048b0a0 0x55b130866500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f7724d42740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b130493280 0x55b130860830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f7724d42740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b1304b3f60 0x55b130860d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f7724d42740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55b130877790 con 0x55b130493280 2024-07-26T06:50:09.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f7724d42740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b130877e60 con 0x55b1304b3f60 2024-07-26T06:50:09.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f7724d42740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55b130878530 con 0x55b13048b0a0 2024-07-26T06:50:09.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f77177fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b130493280 0x55b130860830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f77177fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b130493280 0x55b130860830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44954/0 (socket says 172.21.15.42:44954) 2024-07-26T06:50:09.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f77177fe640 1 -- 172.21.15.42:0/1796325323 learned_addr learned my addr 172.21.15.42:0/1796325323 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:09.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f771cd5b640 1 --2- 172.21.15.42:0/1796325323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b13048b0a0 0x55b130866500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f771cd5b640 1 --2- 172.21.15.42:0/1796325323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b1304b3f60 0x55b130860d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f77177fe640 1 -- 172.21.15.42:0/1796325323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b13048b0a0 msgr2=0x55b130866500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:09.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f77177fe640 1 --2- 172.21.15.42:0/1796325323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b13048b0a0 0x55b130866500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f77177fe640 1 -- 172.21.15.42:0/1796325323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b1304b3f60 msgr2=0x55b130860d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:09.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f77177fe640 1 --2- 172.21.15.42:0/1796325323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b1304b3f60 0x55b130860d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f77177fe640 1 -- 172.21.15.42:0/1796325323 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b130867bb0 con 0x55b130493280 2024-07-26T06:50:09.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f771cd5b640 1 --2- 172.21.15.42:0/1796325323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b1304b3f60 0x55b130860d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:09.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.431+0000 7f77177fe640 1 --2- 172.21.15.42:0/1796325323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b130493280 0x55b130860830 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f770c005f70 tx=0x7f770c009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c096f441eaa37578 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:09.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.432+0000 7f76fdffb640 1 -- 172.21.15.42:0/1796325323 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c048030 con 0x55b130493280 2024-07-26T06:50:09.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.432+0000 7f76fdffb640 1 -- 172.21.15.42:0/1796325323 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f770c031950 con 0x55b130493280 2024-07-26T06:50:09.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.432+0000 7f76fdffb640 1 -- 172.21.15.42:0/1796325323 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c035040 con 0x55b130493280 2024-07-26T06:50:09.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.433+0000 7f7724d42740 1 -- 172.21.15.42:0/1796325323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b130493280 msgr2=0x55b130860830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:09.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.433+0000 7f7724d42740 1 --2- 172.21.15.42:0/1796325323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b130493280 0x55b130860830 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f770c005f70 tx=0x7f770c009640 comp rx=0 tx=0).stop 2024-07-26T06:50:09.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.433+0000 7f7724d42740 1 -- 172.21.15.42:0/1796325323 shutdown_connections 2024-07-26T06:50:09.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.433+0000 7f7724d42740 1 --2- 172.21.15.42:0/1796325323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b1304b3f60 0x55b130860d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.433+0000 7f7724d42740 1 --2- 172.21.15.42:0/1796325323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b130493280 0x55b130860830 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.433+0000 7f7724d42740 1 --2- 172.21.15.42:0/1796325323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b13048b0a0 0x55b130866500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.433+0000 7f7724d42740 1 -- 172.21.15.42:0/1796325323 >> 172.21.15.42:0/1796325323 conn(0x55b1304a8350 msgr2=0x55b1308621c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:09.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.433+0000 7f7724d42740 1 -- 172.21.15.42:0/1796325323 shutdown_connections 2024-07-26T06:50:09.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.433+0000 7f7724d42740 1 -- 172.21.15.42:0/1796325323 wait complete. 2024-07-26T06:50:09.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.433+0000 7f7724d42740 1 -- start start 2024-07-26T06:50:09.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.434+0000 7f7724d42740 1 Processor -- start 2024-07-26T06:50:09.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.434+0000 7f7724d42740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b13048b0a0 0x55b130873730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.434+0000 7f7724d42740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b130493280 0x55b130871fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.434+0000 7f7724d42740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b1304b3f60 0x55b1308725e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.434+0000 7f7724d42740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55b1308792d0 con 0x55b130493280 2024-07-26T06:50:09.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.434+0000 7f7724d42740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b130878c00 con 0x55b1304b3f60 2024-07-26T06:50:09.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.434+0000 7f7724d42740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55b1308799a0 con 0x55b13048b0a0 2024-07-26T06:50:09.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.434+0000 7f7724d42740 10 client.0 _ll_register_callbacks cb 0x7f77100074c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:09.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.434+0000 7f77177fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b130493280 0x55b130871fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.434+0000 7f77177fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b130493280 0x55b130871fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44964/0 (socket says 172.21.15.42:44964) 2024-07-26T06:50:09.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.434+0000 7f77177fe640 1 -- 172.21.15.42:0/1876981503 learned_addr learned my addr 172.21.15.42:0/1876981503 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.434+0000 7f771cd5b640 1 --2- 172.21.15.42:0/1876981503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b13048b0a0 0x55b130873730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.434+0000 7f771cd5b640 1 --2- 172.21.15.42:0/1876981503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b1304b3f60 0x55b1308725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.435+0000 7f77177fe640 1 -- 172.21.15.42:0/1876981503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b13048b0a0 msgr2=0x55b130873730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.435+0000 7f77177fe640 1 --2- 172.21.15.42:0/1876981503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b13048b0a0 0x55b130873730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.435+0000 7f77177fe640 1 -- 172.21.15.42:0/1876981503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b1304b3f60 msgr2=0x55b1308725e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.435+0000 7f77177fe640 1 --2- 172.21.15.42:0/1876981503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b1304b3f60 0x55b1308725e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.435+0000 7f77177fe640 1 -- 172.21.15.42:0/1876981503 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b130872d70 con 0x55b130493280 2024-07-26T06:50:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.435+0000 7f77177fe640 1 --2- 172.21.15.42:0/1876981503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b130493280 0x55b130871fe0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f770c006010 tx=0x7f770c005590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe5b2290294951f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.435+0000 7f76f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.435+0000 7f76f77fe640 1 -- 172.21.15.42:0/1876981503 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c048030 con 0x55b130493280 2024-07-26T06:50:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.435+0000 7f76f77fe640 1 -- 172.21.15.42:0/1876981503 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f770c0314a0 con 0x55b130493280 2024-07-26T06:50:09.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.435+0000 7f7724d42740 1 -- 172.21.15.42:0/1876981503 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b1307cd330 con 0x55b130493280 2024-07-26T06:50:09.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.435+0000 7f76f77fe640 1 -- 172.21.15.42:0/1876981503 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f770c035040 con 0x55b130493280 2024-07-26T06:50:09.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.435+0000 7f7724d42740 10 client.15228 fetch_fsmap learned FSMap version 28 2024-07-26T06:50:09.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.435+0000 7f7724d42740 1 -- 172.21.15.42:0/1876981503 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b1308e16f0 con 0x55b130493280 2024-07-26T06:50:09.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.436+0000 7f76f77fe640 1 -- 172.21.15.42:0/1876981503 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b1307cd330 con 0x55b130493280 2024-07-26T06:50:09.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.436+0000 7f76f77fe640 1 -- 172.21.15.42:0/1876981503 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f770c0443d0 con 0x55b130493280 2024-07-26T06:50:09.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.436+0000 7f7724d42740 10 client.15228 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:50:09.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.436+0000 7f7724d42740 10 client.15228 resolve_mds: resolved name 'e' to daemon mds.e 2024-07-26T06:50:09.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.436+0000 7f7724d42740 20 client.15228 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:09.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.436+0000 7f7724d42740 1 --2- 172.21.15.42:0/1876981503 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x55b130447c90 0x55b1308e4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:09.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.436+0000 7f7724d42740 4 client.15228 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:50:09.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.436+0000 7f7724d42740 1 -- 172.21.15.42:0/1876981503 --> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] -- command(tid 0: {"prefix": "status"}) -- 0x55b1307caa70 con 0x55b130447c90 2024-07-26T06:50:09.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.436+0000 7f771cd5b640 1 --2- 172.21.15.42:0/1876981503 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x55b130447c90 0x55b1308e4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:09.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.437+0000 7f76f77fe640 10 client.15228 ms_handle_connect on v2:172.21.15.163:6834/375978170 2024-07-26T06:50:09.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.437+0000 7f771cd5b640 1 --2- 172.21.15.42:0/1876981503 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x55b130447c90 0x55b1308e4a90 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e931ab77531d36bb server_cookie=eacde1cfba143777 in_seq=0 out_seq=0 2024-07-26T06:50:09.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.437+0000 7f76f77fe640 1 -- 172.21.15.42:0/1876981503 <== mds.4 v2:172.21.15.163:6834/375978170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b1307caa70 con 0x55b130447c90 2024-07-26T06:50:09.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.437+0000 7f76f77fe640 10 client.15228 handle_command_reply: tid=0 2024-07-26T06:50:09.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.437+0000 7f7724d42740 1 client.15228 shutdown 2024-07-26T06:50:09.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.437+0000 7f7724d42740 1 -- 172.21.15.42:0/1876981503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b130493280 msgr2=0x55b130871fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:09.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.437+0000 7f7724d42740 1 --2- 172.21.15.42:0/1876981503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b130493280 0x55b130871fe0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f770c006010 tx=0x7f770c005590 comp rx=0 tx=0).stop 2024-07-26T06:50:09.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.437+0000 7f7724d42740 1 -- 172.21.15.42:0/1876981503 shutdown_connections 2024-07-26T06:50:09.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.437+0000 7f7724d42740 1 --2- 172.21.15.42:0/1876981503 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x55b130447c90 0x55b1308e4a90 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.437+0000 7f7724d42740 1 --2- 172.21.15.42:0/1876981503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b1304b3f60 0x55b1308725e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.437+0000 7f7724d42740 1 --2- 172.21.15.42:0/1876981503 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55b130493280 0x55b130871fe0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.437+0000 7f7724d42740 1 --2- 172.21.15.42:0/1876981503 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55b13048b0a0 0x55b130873730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.437+0000 7f7724d42740 1 -- 172.21.15.42:0/1876981503 >> 172.21.15.42:0/1876981503 conn(0x55b1304a8350 msgr2=0x55b1308657c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:09.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.437+0000 7f76f77fe640 0 client.15228 ms_handle_reset on v2:172.21.15.163:6834/375978170 2024-07-26T06:50:09.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.437+0000 7f7724d42740 1 -- 172.21.15.42:0/1876981503 shutdown_connections 2024-07-26T06:50:09.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.438+0000 7f7724d42740 1 -- 172.21.15.42:0/1876981503 wait complete. 2024-07-26T06:50:09.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.438+0000 7f7724d42740 20 client.15228 trim_cache size 0 max 16384 2024-07-26T06:50:09.444 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:50:09.444 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_fsid": "8b3e2750-4b19-11ef-bcb6-c7b262605968", 2024-07-26T06:50:09.444 INFO:teuthology.orchestra.run.smithi042.stdout: "whoami": 4, 2024-07-26T06:50:09.444 INFO:teuthology.orchestra.run.smithi042.stdout: "id": 24578, 2024-07-26T06:50:09.445 INFO:teuthology.orchestra.run.smithi042.stdout: "want_state": "up:active", 2024-07-26T06:50:09.445 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "up:active", 2024-07-26T06:50:09.445 INFO:teuthology.orchestra.run.smithi042.stdout: "fs_name": "cephfs", 2024-07-26T06:50:09.445 INFO:teuthology.orchestra.run.smithi042.stdout: "rank_uptime": 167.33807046000001, 2024-07-26T06:50:09.445 INFO:teuthology.orchestra.run.smithi042.stdout: "mdsmap_epoch": 28, 2024-07-26T06:50:09.445 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch": 80, 2024-07-26T06:50:09.445 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch_barrier": 80, 2024-07-26T06:50:09.445 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 218.918354064 2024-07-26T06:50:09.445 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:50:09.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.438+0000 7f771e7be640 1 -- 172.21.15.42:0/3187811616 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f77000b3c50 msgr2=0x7f77000b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:09.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.438+0000 7f771e7be640 1 --2- 172.21.15.42:0/3187811616 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f77000b3c50 0x7f77000b6160 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x55b13085a230 tx=0x7f7708092040 comp rx=0 tx=0).stop 2024-07-26T06:50:09.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.438+0000 7f771e7be640 1 -- 172.21.15.42:0/3187811616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77181024b0 msgr2=0x7f7718198fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:09.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.438+0000 7f771e7be640 1 --2- 172.21.15.42:0/3187811616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77181024b0 0x7f7718198fe0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f770800c890 tx=0x7f770800cd50 comp rx=0 tx=0).stop 2024-07-26T06:50:09.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.439+0000 7f771e7be640 1 -- 172.21.15.42:0/3187811616 shutdown_connections 2024-07-26T06:50:09.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.439+0000 7f771e7be640 1 --2- 172.21.15.42:0/3187811616 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f77000b3c50 0x7f77000b6160 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.439+0000 7f771e7be640 1 --2- 172.21.15.42:0/3187811616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7718108740 0x7f771819e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.439+0000 7f771e7be640 1 --2- 172.21.15.42:0/3187811616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7718107df0 0x7f7718199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.439+0000 7f771e7be640 1 --2- 172.21.15.42:0/3187811616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77181024b0 0x7f7718198fe0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:09.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.439+0000 7f771e7be640 1 -- 172.21.15.42:0/3187811616 >> 172.21.15.42:0/3187811616 conn(0x7f7718079170 msgr2=0x7f7718104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:09.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.439+0000 7f771e7be640 1 -- 172.21.15.42:0/3187811616 shutdown_connections 2024-07-26T06:50:09.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:09.439+0000 7f771e7be640 1 -- 172.21.15.42:0/3187811616 wait complete. 2024-07-26T06:50:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:50:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:50:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:09.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:50:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:50:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:50:10.233 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell mds.c status 2024-07-26T06:50:10.508 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:10 smithi164 ceph-mon[36970]: pgmap v394: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:50:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:10 smithi163 ceph-mon[36873]: pgmap v394: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:50:10.759 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:10 smithi042 ceph-mon[32775]: pgmap v394: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:50:12.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.390+0000 7f9cf1a5c640 1 -- 172.21.15.42:0/862701388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cec108ed0 msgr2=0x7f9cec1092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:12.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.390+0000 7f9cf1a5c640 1 --2- 172.21.15.42:0/862701388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cec108ed0 0x7f9cec1092e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0009900 tx=0x7f9ce002a3f0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.390+0000 7f9cf1a5c640 1 -- 172.21.15.42:0/862701388 shutdown_connections 2024-07-26T06:50:12.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.390+0000 7f9cf1a5c640 1 --2- 172.21.15.42:0/862701388 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9cec109820 0x7f9cec110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.390+0000 7f9cf1a5c640 1 --2- 172.21.15.42:0/862701388 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cec108ed0 0x7f9cec1092e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.390+0000 7f9cf1a5c640 1 --2- 172.21.15.42:0/862701388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cec103590 0x7f9cec1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.390+0000 7f9cf1a5c640 1 -- 172.21.15.42:0/862701388 >> 172.21.15.42:0/862701388 conn(0x7f9cec07b280 msgr2=0x7f9cec07b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:12.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.390+0000 7f9cf1a5c640 1 -- 172.21.15.42:0/862701388 shutdown_connections 2024-07-26T06:50:12.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.390+0000 7f9cf1a5c640 1 -- 172.21.15.42:0/862701388 wait complete. 2024-07-26T06:50:12.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.391+0000 7f9cf1a5c640 1 Processor -- start 2024-07-26T06:50:12.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.391+0000 7f9cf1a5c640 1 -- start start 2024-07-26T06:50:12.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.391+0000 7f9cf1a5c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cec103590 0x7f9cec199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.391+0000 7f9cf1a5c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9cec108ed0 0x7f9cec19a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.391+0000 7f9cf1a5c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cec109820 0x7f9cec19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.391+0000 7f9cf1a5c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec113500 con 0x7f9cec103590 2024-07-26T06:50:12.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.391+0000 7f9cf1a5c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec113380 con 0x7f9cec109820 2024-07-26T06:50:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.391+0000 7f9cf1a5c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec113680 con 0x7f9cec108ed0 2024-07-26T06:50:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.391+0000 7f9ceaffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cec103590 0x7f9cec199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.391+0000 7f9ceaffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cec103590 0x7f9cec199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44980/0 (socket says 172.21.15.42:44980) 2024-07-26T06:50:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.391+0000 7f9ceaffd640 1 -- 172.21.15.42:0/3038631775 learned_addr learned my addr 172.21.15.42:0/3038631775 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.392+0000 7f9cea7fc640 1 --2- 172.21.15.42:0/3038631775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9cec108ed0 0x7f9cec19a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.392+0000 7f9ceb7fe640 1 --2- 172.21.15.42:0/3038631775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cec109820 0x7f9cec19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.392+0000 7f9ceaffd640 1 -- 172.21.15.42:0/3038631775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9cec108ed0 msgr2=0x7f9cec19a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.392+0000 7f9ceaffd640 1 --2- 172.21.15.42:0/3038631775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9cec108ed0 0x7f9cec19a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.392+0000 7f9ceaffd640 1 -- 172.21.15.42:0/3038631775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cec109820 msgr2=0x7f9cec19f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.392+0000 7f9ceaffd640 1 --2- 172.21.15.42:0/3038631775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cec109820 0x7f9cec19f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.392+0000 7f9ceaffd640 1 -- 172.21.15.42:0/3038631775 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cec19fda0 con 0x7f9cec103590 2024-07-26T06:50:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.392+0000 7f9ceb7fe640 1 --2- 172.21.15.42:0/3038631775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cec109820 0x7f9cec19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:50:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.392+0000 7f9ceaffd640 1 --2- 172.21.15.42:0/3038631775 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cec103590 0x7f9cec199ef0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c920 tx=0x7f9cdc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.392+0000 7f9cf0a5a640 1 -- 172.21.15.42:0/3038631775 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc007c70 con 0x7f9cec103590 2024-07-26T06:50:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.392+0000 7f9cf0a5a640 1 -- 172.21.15.42:0/3038631775 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9cdc007e10 con 0x7f9cec103590 2024-07-26T06:50:12.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.392+0000 7f9cf1a5c640 1 -- 172.21.15.42:0/3038631775 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cec1a0090 con 0x7f9cec103590 2024-07-26T06:50:12.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.392+0000 7f9cf0a5a640 1 -- 172.21.15.42:0/3038631775 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc018820 con 0x7f9cec103590 2024-07-26T06:50:12.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.392+0000 7f9cf1a5c640 1 -- 172.21.15.42:0/3038631775 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cec1a68b0 con 0x7f9cec103590 2024-07-26T06:50:12.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.395+0000 7f9cf7fe0740 1 Processor -- start 2024-07-26T06:50:12.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.395+0000 7f9cf7fe0740 1 -- start start 2024-07-26T06:50:12.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.395+0000 7f9cf7fe0740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500bd0f60 0x561500f76380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.395+0000 7f9cf7fe0740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500ba80a0 0x561500f768c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.395+0000 7f9cf7fe0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500bb0280 0x561500f76e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.395+0000 7f9cf7fe0740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x561500eec920 con 0x561500ba80a0 2024-07-26T06:50:12.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.395+0000 7f9cf7fe0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561500edbe20 con 0x561500bb0280 2024-07-26T06:50:12.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.395+0000 7f9cf7fe0740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x561500edfe60 con 0x561500bd0f60 2024-07-26T06:50:12.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.395+0000 7f9cf0a5a640 1 -- 172.21.15.42:0/3038631775 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f9cdc00f450 con 0x7f9cec103590 2024-07-26T06:50:12.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.395+0000 7f9ceb7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500bd0f60 0x561500f76380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.395+0000 7f9cea7fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500bb0280 0x561500f76e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.396+0000 7f9ceb7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500bd0f60 0x561500f76380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:45066/0 (socket says 172.21.15.42:45066) 2024-07-26T06:50:12.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.396+0000 7f9ceb7fe640 1 -- 172.21.15.42:0/3848543646 learned_addr learned my addr 172.21.15.42:0/3848543646 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:12.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.396+0000 7f9ceaffd640 1 --2- 172.21.15.42:0/3848543646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500ba80a0 0x561500f768c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.396+0000 7f9cea7fc640 1 -- 172.21.15.42:0/3848543646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500bd0f60 msgr2=0x561500f76380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:12.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.396+0000 7f9cea7fc640 1 --2- 172.21.15.42:0/3848543646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500bd0f60 0x561500f76380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.396+0000 7f9cea7fc640 1 -- 172.21.15.42:0/3848543646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500ba80a0 msgr2=0x561500f768c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:12.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.396+0000 7f9cea7fc640 1 --2- 172.21.15.42:0/3848543646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500ba80a0 0x561500f768c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.396+0000 7f9cea7fc640 1 -- 172.21.15.42:0/3848543646 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561500e82a40 con 0x561500bb0280 2024-07-26T06:50:12.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.396+0000 7f9ceaffd640 1 --2- 172.21.15.42:0/3848543646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500ba80a0 0x561500f768c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:12.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.396+0000 7f9cea7fc640 1 --2- 172.21.15.42:0/3848543646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500bb0280 0x561500f76e00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9cec105ef0 tx=0x7f9ce0004170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37af48dbcaf46003 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:12.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.396+0000 7f9cf0a5a640 1 --2- 172.21.15.42:0/3038631775 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9cd40b3c00 0x7f9cd40b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.397+0000 7f9cd1ffb640 1 -- 172.21.15.42:0/3848543646 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce003f070 con 0x561500bb0280 2024-07-26T06:50:12.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.397+0000 7f9ceb7fe640 1 --2- 172.21.15.42:0/3038631775 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9cd40b3c00 0x7f9cd40b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.397+0000 7f9cd1ffb640 1 -- 172.21.15.42:0/3848543646 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ce00054f0 con 0x561500bb0280 2024-07-26T06:50:12.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.397+0000 7f9cd1ffb640 1 -- 172.21.15.42:0/3848543646 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0035030 con 0x561500bb0280 2024-07-26T06:50:12.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.397+0000 7f9cf0a5a640 1 -- 172.21.15.42:0/3038631775 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f9cdc0e3df0 con 0x7f9cec103590 2024-07-26T06:50:12.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.398+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/3848543646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500bb0280 msgr2=0x561500f76e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:12.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.398+0000 7f9ceb7fe640 1 --2- 172.21.15.42:0/3038631775 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9cd40b3c00 0x7f9cd40b6110 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4012560 tx=0x7f9ce4009790 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:12.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.398+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/3848543646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500bb0280 0x561500f76e00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9cec105ef0 tx=0x7f9ce0004170 comp rx=0 tx=0).stop 2024-07-26T06:50:12.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.398+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/3848543646 shutdown_connections 2024-07-26T06:50:12.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.398+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/3848543646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500bb0280 0x561500f76e00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.398+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/3848543646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500ba80a0 0x561500f768c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.398+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/3848543646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500bd0f60 0x561500f76380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.398+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/3848543646 >> 172.21.15.42:0/3848543646 conn(0x561500bc5350 msgr2=0x561500f74270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:12.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.398+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/3848543646 shutdown_connections 2024-07-26T06:50:12.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.398+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/3848543646 wait complete. 2024-07-26T06:50:12.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.398+0000 7f9cf7fe0740 1 -- start start 2024-07-26T06:50:12.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.398+0000 7f9cf7fe0740 1 Processor -- start 2024-07-26T06:50:12.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9cf7fe0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500ba80a0 0x561500f81fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9cf7fe0740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500bb0280 0x561500f82500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9cf7fe0740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500bd0f60 0x561500f8de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9cf7fe0740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x561500d3f5f0 con 0x561500bd0f60 2024-07-26T06:50:12.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9cf7fe0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561500e7e890 con 0x561500ba80a0 2024-07-26T06:50:12.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9cf7fe0740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x561500f84b00 con 0x561500bb0280 2024-07-26T06:50:12.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9cf7fe0740 10 client.0 _ll_register_callbacks cb 0x561500e887d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:12.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9ceb7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500bd0f60 0x561500f8de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9ceb7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500bd0f60 0x561500f8de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45002/0 (socket says 172.21.15.42:45002) 2024-07-26T06:50:12.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9ceb7fe640 1 -- 172.21.15.42:0/1591877285 learned_addr learned my addr 172.21.15.42:0/1591877285 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:12.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9ceaffd640 1 --2- 172.21.15.42:0/1591877285 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500ba80a0 0x561500f81fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9cea7fc640 1 --2- 172.21.15.42:0/1591877285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500bb0280 0x561500f82500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9ceb7fe640 1 -- 172.21.15.42:0/1591877285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500bb0280 msgr2=0x561500f82500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:12.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9ceb7fe640 1 --2- 172.21.15.42:0/1591877285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500bb0280 0x561500f82500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9ceb7fe640 1 -- 172.21.15.42:0/1591877285 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500ba80a0 msgr2=0x561500f81fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:12.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9ceb7fe640 1 --2- 172.21.15.42:0/1591877285 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500ba80a0 0x561500f81fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9ceb7fe640 1 -- 172.21.15.42:0/1591877285 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561500c3c540 con 0x561500bd0f60 2024-07-26T06:50:12.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.399+0000 7f9ceaffd640 1 --2- 172.21.15.42:0/1591877285 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500ba80a0 0x561500f81fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:12.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.400+0000 7f9ceb7fe640 1 --2- 172.21.15.42:0/1591877285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500bd0f60 0x561500f8de80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4013fd0 tx=0x7f9ce4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253a273404df9a84 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:12.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.400+0000 7f9ccaffd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:12.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.400+0000 7f9ccaffd640 1 -- 172.21.15.42:0/1591877285 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce400f700 con 0x561500bd0f60 2024-07-26T06:50:12.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.400+0000 7f9ccaffd640 1 -- 172.21.15.42:0/1591877285 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f9ce400f8a0 con 0x561500bd0f60 2024-07-26T06:50:12.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.400+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/1591877285 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ce00054f0 con 0x561500bd0f60 2024-07-26T06:50:12.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.400+0000 7f9ccaffd640 1 -- 172.21.15.42:0/1591877285 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9ce400faa0 con 0x561500bd0f60 2024-07-26T06:50:12.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.400+0000 7f9cf7fe0740 10 client.15240 fetch_fsmap learned FSMap version 28 2024-07-26T06:50:12.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.400+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/1591877285 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561500f94fe0 con 0x561500bd0f60 2024-07-26T06:50:12.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.400+0000 7f9ccaffd640 1 -- 172.21.15.42:0/1591877285 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ce00054f0 con 0x561500bd0f60 2024-07-26T06:50:12.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.400+0000 7f9ccaffd640 1 -- 172.21.15.42:0/1591877285 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9ce4031420 con 0x561500bd0f60 2024-07-26T06:50:12.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.400+0000 7f9cf7fe0740 10 client.15240 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:50:12.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.400+0000 7f9cf7fe0740 10 client.15240 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T06:50:12.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.400+0000 7f9cf7fe0740 20 client.15240 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:12.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.400+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/1591877285 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x561500b8f8e0 0x561500f98aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.401+0000 7f9cf7fe0740 4 client.15240 mds_command: new command op to 24583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:50:12.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.401+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/1591877285 --> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561500eee870 con 0x561500b8f8e0 2024-07-26T06:50:12.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.401+0000 7f9ceaffd640 1 --2- 172.21.15.42:0/1591877285 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x561500b8f8e0 0x561500f98aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.401+0000 7f9ceaffd640 1 --2- 172.21.15.42:0/1591877285 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x561500b8f8e0 0x561500f98aa0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8f128dba59e5c5b6 server_cookie=d86dbd9cb7c394e0 in_seq=0 out_seq=0 2024-07-26T06:50:12.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.401+0000 7f9ccaffd640 10 client.15240 ms_handle_connect on v2:172.21.15.164:6836/3642457979 2024-07-26T06:50:12.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.402+0000 7f9ccaffd640 1 -- 172.21.15.42:0/1591877285 <== mds.3 v2:172.21.15.164:6836/3642457979 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561500eee870 con 0x561500b8f8e0 2024-07-26T06:50:12.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.402+0000 7f9ccaffd640 10 client.15240 handle_command_reply: tid=0 2024-07-26T06:50:12.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.402+0000 7f9cf7fe0740 1 client.15240 shutdown 2024-07-26T06:50:12.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.402+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/1591877285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500bd0f60 msgr2=0x561500f8de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:12.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.402+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/1591877285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500bd0f60 0x561500f8de80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4013fd0 tx=0x7f9ce4004a40 comp rx=0 tx=0).stop 2024-07-26T06:50:12.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.403+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/1591877285 shutdown_connections 2024-07-26T06:50:12.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.403+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/1591877285 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x561500b8f8e0 0x561500f98aa0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.403+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/1591877285 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500bd0f60 0x561500f8de80 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.403+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/1591877285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500bb0280 0x561500f82500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.403+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/1591877285 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500ba80a0 0x561500f81fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.403+0000 7f9ccaffd640 0 client.15240 ms_handle_reset on v2:172.21.15.164:6836/3642457979 2024-07-26T06:50:12.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.403+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/1591877285 >> 172.21.15.42:0/1591877285 conn(0x561500bc5350 msgr2=0x561500f79510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:12.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.403+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/1591877285 shutdown_connections 2024-07-26T06:50:12.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.403+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/1591877285 wait complete. 2024-07-26T06:50:12.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.403+0000 7f9cf7fe0740 20 client.15240 trim_cache size 0 max 16384 2024-07-26T06:50:12.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.421+0000 7f9cf7fe0740 1 Processor -- start 2024-07-26T06:50:12.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.421+0000 7f9cf7fe0740 1 -- start start 2024-07-26T06:50:12.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.421+0000 7f9cf7fe0740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500ba80a0 0x561500f8ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.421+0000 7f9cf7fe0740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500bb0280 0x561500f812b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.421+0000 7f9cf7fe0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500bd0f60 0x561500f7cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.421+0000 7f9cf7fe0740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x561500f96de0 con 0x561500ba80a0 2024-07-26T06:50:12.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.421+0000 7f9cf7fe0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561500f974b0 con 0x561500bd0f60 2024-07-26T06:50:12.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.421+0000 7f9cf7fe0740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x561500f97b80 con 0x561500bb0280 2024-07-26T06:50:12.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.421+0000 7f9ceaffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500ba80a0 0x561500f8ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.421+0000 7f9ceaffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500ba80a0 0x561500f8ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45018/0 (socket says 172.21.15.42:45018) 2024-07-26T06:50:12.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.421+0000 7f9ceaffd640 1 -- 172.21.15.42:0/1203801526 learned_addr learned my addr 172.21.15.42:0/1203801526 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:12.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.422+0000 7f9ceb7fe640 1 --2- 172.21.15.42:0/1203801526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500bd0f60 0x561500f7cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.422+0000 7f9cea7fc640 1 --2- 172.21.15.42:0/1203801526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500bb0280 0x561500f812b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.422+0000 7f9ceaffd640 1 -- 172.21.15.42:0/1203801526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500bb0280 msgr2=0x561500f812b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.422+0000 7f9ceaffd640 1 --2- 172.21.15.42:0/1203801526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500bb0280 0x561500f812b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.422+0000 7f9ceaffd640 1 -- 172.21.15.42:0/1203801526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500bd0f60 msgr2=0x561500f7cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.422+0000 7f9ceaffd640 1 --2- 172.21.15.42:0/1203801526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500bd0f60 0x561500f7cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.422+0000 7f9ceaffd640 1 -- 172.21.15.42:0/1203801526 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561500f78670 con 0x561500ba80a0 2024-07-26T06:50:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.422+0000 7f9ceaffd640 1 --2- 172.21.15.42:0/1203801526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500ba80a0 0x561500f8ddf0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c8f0 tx=0x7f9cdc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cfa40d8cf871f49 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.422+0000 7f9cd17fa640 1 -- 172.21.15.42:0/1203801526 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc0aad20 con 0x561500ba80a0 2024-07-26T06:50:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.422+0000 7f9cd17fa640 1 -- 172.21.15.42:0/1203801526 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9cdc0af320 con 0x561500ba80a0 2024-07-26T06:50:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.422+0000 7f9cd17fa640 1 -- 172.21.15.42:0/1203801526 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc0a6fb0 con 0x561500ba80a0 2024-07-26T06:50:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.423+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/1203801526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500ba80a0 msgr2=0x561500f8ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:12.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.423+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/1203801526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500ba80a0 0x561500f8ddf0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c8f0 tx=0x7f9cdc000bc0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.423+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/1203801526 shutdown_connections 2024-07-26T06:50:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.423+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/1203801526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500bd0f60 0x561500f7cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.423+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/1203801526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500bb0280 0x561500f812b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.423+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/1203801526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500ba80a0 0x561500f8ddf0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.423+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/1203801526 >> 172.21.15.42:0/1203801526 conn(0x561500bc5350 msgr2=0x561500f73bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.423+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/1203801526 shutdown_connections 2024-07-26T06:50:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.423+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/1203801526 wait complete. 2024-07-26T06:50:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.424+0000 7f9cf7fe0740 1 -- start start 2024-07-26T06:50:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.424+0000 7f9cf7fe0740 1 Processor -- start 2024-07-26T06:50:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.424+0000 7f9cf7fe0740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500bb0280 0x561500f9a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.424+0000 7f9cf7fe0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500bd0f60 0x561500f9a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.424+0000 7f9cf7fe0740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500b64c90 0x561500f9f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.424+0000 7f9cf7fe0740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x561500f98920 con 0x561500bb0280 2024-07-26T06:50:12.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.424+0000 7f9cf7fe0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561500f98250 con 0x561500bd0f60 2024-07-26T06:50:12.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.424+0000 7f9cf7fe0740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x561500f98ff0 con 0x561500b64c90 2024-07-26T06:50:12.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.424+0000 7f9cf7fe0740 10 client.0 _ll_register_callbacks cb 0x7f9ce40106d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:12.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.424+0000 7f9ceaffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500bb0280 0x561500f9a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.424+0000 7f9ceaffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500bb0280 0x561500f9a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45026/0 (socket says 172.21.15.42:45026) 2024-07-26T06:50:12.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.424+0000 7f9ceaffd640 1 -- 172.21.15.42:0/3008392719 learned_addr learned my addr 172.21.15.42:0/3008392719 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:12.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.425+0000 7f9cea7fc640 1 --2- 172.21.15.42:0/3008392719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500bd0f60 0x561500f9a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.425+0000 7f9ceb7fe640 1 --2- 172.21.15.42:0/3008392719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500b64c90 0x561500f9f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.425+0000 7f9ceaffd640 1 -- 172.21.15.42:0/3008392719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500b64c90 msgr2=0x561500f9f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.425+0000 7f9ceaffd640 1 --2- 172.21.15.42:0/3008392719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500b64c90 0x561500f9f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.425+0000 7f9ceaffd640 1 -- 172.21.15.42:0/3008392719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500bd0f60 msgr2=0x561500f9a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.425+0000 7f9ceaffd640 1 --2- 172.21.15.42:0/3008392719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500bd0f60 0x561500f9a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.425+0000 7f9ceaffd640 1 -- 172.21.15.42:0/3008392719 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561500f9fee0 con 0x561500bb0280 2024-07-26T06:50:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.425+0000 7f9cea7fc640 1 --2- 172.21.15.42:0/3008392719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500bd0f60 0x561500f9a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.425+0000 7f9ceb7fe640 1 --2- 172.21.15.42:0/3008392719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500b64c90 0x561500f9f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.425+0000 7f9ceaffd640 1 --2- 172.21.15.42:0/3008392719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500bb0280 0x561500f9a2d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0e1920 tx=0x7f9cdc0aac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6bced9080316c7f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.425+0000 7f9ccaffd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.425+0000 7f9ccaffd640 1 -- 172.21.15.42:0/3008392719 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc0aad20 con 0x561500bb0280 2024-07-26T06:50:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.425+0000 7f9ccaffd640 1 -- 172.21.15.42:0/3008392719 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f9cdc00f740 con 0x561500bb0280 2024-07-26T06:50:12.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.425+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/3008392719 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cdc0af320 con 0x561500bb0280 2024-07-26T06:50:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.425+0000 7f9ccaffd640 1 -- 172.21.15.42:0/3008392719 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9cdc0a6fb0 con 0x561500bb0280 2024-07-26T06:50:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.425+0000 7f9cf7fe0740 10 client.15246 fetch_fsmap learned FSMap version 28 2024-07-26T06:50:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.425+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/3008392719 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561500fa12a0 con 0x561500bb0280 2024-07-26T06:50:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.426+0000 7f9ccaffd640 1 -- 172.21.15.42:0/3008392719 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9cdc0af320 con 0x561500bb0280 2024-07-26T06:50:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.426+0000 7f9ccaffd640 1 -- 172.21.15.42:0/3008392719 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9cdc0b39d0 con 0x561500bb0280 2024-07-26T06:50:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.426+0000 7f9cf7fe0740 10 client.15246 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:50:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.426+0000 7f9cf7fe0740 10 client.15246 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T06:50:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.426+0000 7f9cf7fe0740 20 client.15246 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.426+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/3008392719 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x561501003380 0x5615010047f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.426+0000 7f9cf7fe0740 4 client.15246 mds_command: new command op to 24583 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:50:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.426+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/3008392719 --> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] -- command(tid 0: {"prefix": "status"}) -- 0x561500eea310 con 0x561501003380 2024-07-26T06:50:12.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.426+0000 7f9cea7fc640 1 --2- 172.21.15.42:0/3008392719 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x561501003380 0x5615010047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.427+0000 7f9cea7fc640 1 --2- 172.21.15.42:0/3008392719 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x561501003380 0x5615010047f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=98979dc2319089ca server_cookie=a2c575af184f45bd in_seq=0 out_seq=0 2024-07-26T06:50:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.427+0000 7f9ccaffd640 10 client.15246 ms_handle_connect on v2:172.21.15.164:6836/3642457979 2024-07-26T06:50:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.427+0000 7f9ccaffd640 1 -- 172.21.15.42:0/3008392719 <== mds.3 v2:172.21.15.164:6836/3642457979 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561500eea310 con 0x561501003380 2024-07-26T06:50:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.427+0000 7f9ccaffd640 10 client.15246 handle_command_reply: tid=0 2024-07-26T06:50:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.427+0000 7f9cf7fe0740 1 client.15246 shutdown 2024-07-26T06:50:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.427+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/3008392719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500bb0280 msgr2=0x561500f9a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.427+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/3008392719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500bb0280 0x561500f9a2d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc0e1920 tx=0x7f9cdc0aac00 comp rx=0 tx=0).stop 2024-07-26T06:50:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.427+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/3008392719 shutdown_connections 2024-07-26T06:50:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.427+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/3008392719 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x561501003380 0x5615010047f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.427+0000 7f9ccaffd640 0 client.15246 ms_handle_reset on v2:172.21.15.164:6836/3642457979 2024-07-26T06:50:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.427+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/3008392719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561500b64c90 0x561500f9f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.427+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/3008392719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561500bd0f60 0x561500f9a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.427+0000 7f9cf7fe0740 1 --2- 172.21.15.42:0/3008392719 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561500bb0280 0x561500f9a2d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.427+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/3008392719 >> 172.21.15.42:0/3008392719 conn(0x561500bc5350 msgr2=0x561500f804d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:12.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.427+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/3008392719 shutdown_connections 2024-07-26T06:50:12.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.427+0000 7f9cf7fe0740 1 -- 172.21.15.42:0/3008392719 wait complete. 2024-07-26T06:50:12.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.427+0000 7f9cf7fe0740 20 client.15246 trim_cache size 0 max 16384 2024-07-26T06:50:12.436 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:50:12.436 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_fsid": "8b3e2750-4b19-11ef-bcb6-c7b262605968", 2024-07-26T06:50:12.436 INFO:teuthology.orchestra.run.smithi042.stdout: "whoami": 3, 2024-07-26T06:50:12.436 INFO:teuthology.orchestra.run.smithi042.stdout: "id": 24583, 2024-07-26T06:50:12.436 INFO:teuthology.orchestra.run.smithi042.stdout: "want_state": "up:active", 2024-07-26T06:50:12.436 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "up:active", 2024-07-26T06:50:12.437 INFO:teuthology.orchestra.run.smithi042.stdout: "fs_name": "cephfs", 2024-07-26T06:50:12.437 INFO:teuthology.orchestra.run.smithi042.stdout: "rank_uptime": 171.34476007500001, 2024-07-26T06:50:12.437 INFO:teuthology.orchestra.run.smithi042.stdout: "mdsmap_epoch": 28, 2024-07-26T06:50:12.437 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch": 80, 2024-07-26T06:50:12.437 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch_barrier": 80, 2024-07-26T06:50:12.437 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 216.692823065 2024-07-26T06:50:12.437 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:50:12.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.428+0000 7f9cf1a5c640 1 -- 172.21.15.42:0/3038631775 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9cd40b3c00 msgr2=0x7f9cd40b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:12.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.428+0000 7f9cf1a5c640 1 --2- 172.21.15.42:0/3038631775 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9cd40b3c00 0x7f9cd40b6110 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4012560 tx=0x7f9ce4009790 comp rx=0 tx=0).stop 2024-07-26T06:50:12.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.428+0000 7f9cf1a5c640 1 -- 172.21.15.42:0/3038631775 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cec103590 msgr2=0x7f9cec199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:12.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.428+0000 7f9cf1a5c640 1 --2- 172.21.15.42:0/3038631775 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cec103590 0x7f9cec199ef0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c920 tx=0x7f9cdc00cde0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.428+0000 7f9cf1a5c640 1 -- 172.21.15.42:0/3038631775 shutdown_connections 2024-07-26T06:50:12.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.428+0000 7f9cf1a5c640 1 --2- 172.21.15.42:0/3038631775 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f9cd40b3c00 0x7f9cd40b6110 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.428+0000 7f9cf1a5c640 1 --2- 172.21.15.42:0/3038631775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cec109820 0x7f9cec19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.428+0000 7f9cf1a5c640 1 --2- 172.21.15.42:0/3038631775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9cec108ed0 0x7f9cec19a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.428+0000 7f9cf1a5c640 1 --2- 172.21.15.42:0/3038631775 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cec103590 0x7f9cec199ef0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:12.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.428+0000 7f9cf1a5c640 1 -- 172.21.15.42:0/3038631775 >> 172.21.15.42:0/3038631775 conn(0x7f9cec07b280 msgr2=0x7f9cec106f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:12.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.428+0000 7f9cf1a5c640 1 -- 172.21.15.42:0/3038631775 shutdown_connections 2024-07-26T06:50:12.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:12.428+0000 7f9cf1a5c640 1 -- 172.21.15.42:0/3038631775 wait complete. 2024-07-26T06:50:12.738 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:12 smithi042 ceph-mon[32775]: pgmap v395: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:50:12.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:12 smithi163 ceph-mon[36873]: pgmap v395: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:50:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:12 smithi164 ceph-mon[36970]: pgmap v395: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:50:13.221 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell mds.h status 2024-07-26T06:50:13.500 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:14.728 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:14 smithi042 ceph-mon[32775]: pgmap v396: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:50:14.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:14 smithi163 ceph-mon[36873]: pgmap v396: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:50:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:14 smithi164 ceph-mon[36970]: pgmap v396: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:50:15.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.295+0000 7f1f123be640 1 -- 172.21.15.42:0/4093808094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f0c107670 msgr2=0x7f1f0c10ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:15.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.295+0000 7f1f123be640 1 --2- 172.21.15.42:0/4093808094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f0c107670 0x7f1f0c10ef50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1f04009ca0 tx=0x7f1f0402a8c0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.295+0000 7f1f123be640 1 -- 172.21.15.42:0/4093808094 shutdown_connections 2024-07-26T06:50:15.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.295+0000 7f1f123be640 1 --2- 172.21.15.42:0/4093808094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f0c107670 0x7f1f0c10ef50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.295+0000 7f1f123be640 1 --2- 172.21.15.42:0/4093808094 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f0c106d20 0x7f1f0c107130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.295+0000 7f1f123be640 1 --2- 172.21.15.42:0/4093808094 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f0c108030 0x7f1f0c108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.295+0000 7f1f123be640 1 -- 172.21.15.42:0/4093808094 >> 172.21.15.42:0/4093808094 conn(0x7f1f0c077230 msgr2=0x7f1f0c077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:15.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.295+0000 7f1f123be640 1 -- 172.21.15.42:0/4093808094 shutdown_connections 2024-07-26T06:50:15.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.295+0000 7f1f123be640 1 -- 172.21.15.42:0/4093808094 wait complete. 2024-07-26T06:50:15.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.295+0000 7f1f123be640 1 Processor -- start 2024-07-26T06:50:15.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.296+0000 7f1f123be640 1 -- start start 2024-07-26T06:50:15.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.296+0000 7f1f123be640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f0c106d20 0x7f1f0c10fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.296+0000 7f1f123be640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f0c107670 0x7f1f0c10ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.296+0000 7f1f123be640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f0c108030 0x7f1f0c1104b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.296+0000 7f1f123be640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0c113500 con 0x7f1f0c108030 2024-07-26T06:50:15.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.296+0000 7f1f123be640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0c113380 con 0x7f1f0c106d20 2024-07-26T06:50:15.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.296+0000 7f1f123be640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0c113680 con 0x7f1f0c107670 2024-07-26T06:50:15.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.296+0000 7f1f1095b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f0c108030 0x7f1f0c1104b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:15.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.296+0000 7f1f1095b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f0c108030 0x7f1f0c1104b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45038/0 (socket says 172.21.15.42:45038) 2024-07-26T06:50:15.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.296+0000 7f1f1095b640 1 -- 172.21.15.42:0/2443098222 learned_addr learned my addr 172.21.15.42:0/2443098222 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:15.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.296+0000 7f1f0bfff640 1 --2- 172.21.15.42:0/2443098222 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f0c106d20 0x7f1f0c10fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:15.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.296+0000 7f1f03fff640 1 --2- 172.21.15.42:0/2443098222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f0c107670 0x7f1f0c10ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:15.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.296+0000 7f1f1095b640 1 -- 172.21.15.42:0/2443098222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f0c107670 msgr2=0x7f1f0c10ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:15.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.296+0000 7f1f1095b640 1 --2- 172.21.15.42:0/2443098222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f0c107670 0x7f1f0c10ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.296+0000 7f1f1095b640 1 -- 172.21.15.42:0/2443098222 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f0c106d20 msgr2=0x7f1f0c10fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:15.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.296+0000 7f1f1095b640 1 --2- 172.21.15.42:0/2443098222 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f0c106d20 0x7f1f0c10fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.296+0000 7f1f1095b640 1 -- 172.21.15.42:0/2443098222 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f0c1a80f0 con 0x7f1f0c108030 2024-07-26T06:50:15.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.297+0000 7f1f1095b640 1 --2- 172.21.15.42:0/2443098222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f0c108030 0x7f1f0c1104b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1f04005cc0 tx=0x7f1f0402f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:15.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.297+0000 7f1f09ffb640 1 -- 172.21.15.42:0/2443098222 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04003bf0 con 0x7f1f0c108030 2024-07-26T06:50:15.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.297+0000 7f1f123be640 1 -- 172.21.15.42:0/2443098222 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f0c1a8380 con 0x7f1f0c108030 2024-07-26T06:50:15.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.297+0000 7f1f09ffb640 1 -- 172.21.15.42:0/2443098222 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f04003d90 con 0x7f1f0c108030 2024-07-26T06:50:15.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.297+0000 7f1f09ffb640 1 -- 172.21.15.42:0/2443098222 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0402fbc0 con 0x7f1f0c108030 2024-07-26T06:50:15.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.297+0000 7f1f123be640 1 -- 172.21.15.42:0/2443098222 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f0c1a88c0 con 0x7f1f0c108030 2024-07-26T06:50:15.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.299+0000 7f1f09ffb640 1 -- 172.21.15.42:0/2443098222 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f1f0402fd60 con 0x7f1f0c108030 2024-07-26T06:50:15.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.300+0000 7f1f18942740 1 Processor -- start 2024-07-26T06:50:15.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.300+0000 7f1f18942740 1 -- start start 2024-07-26T06:50:15.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.300+0000 7f1f18942740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f789f60 0x55754fb2f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.300+0000 7f1f18942740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f7610a0 0x55754fb2f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.300+0000 7f1f18942740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f769280 0x55754fb2fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.300+0000 7f1f18942740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55754faa5940 con 0x55754f789f60 2024-07-26T06:50:15.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.300+0000 7f1f18942740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55754fa94e40 con 0x55754f7610a0 2024-07-26T06:50:15.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.300+0000 7f1f18942740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55754fa98e80 con 0x55754f769280 2024-07-26T06:50:15.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.300+0000 7f1f03fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f789f60 0x55754fb2f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:15.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.300+0000 7f1f03fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f789f60 0x55754fb2f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45042/0 (socket says 172.21.15.42:45042) 2024-07-26T06:50:15.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.300+0000 7f1f03fff640 1 -- 172.21.15.42:0/1847836097 learned_addr learned my addr 172.21.15.42:0/1847836097 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:15.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.300+0000 7f1f0bfff640 1 --2- 172.21.15.42:0/1847836097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f7610a0 0x55754fb2f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:15.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.301+0000 7f1f03fff640 1 --2- 172.21.15.42:0/1847836097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f769280 0x55754fb2fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:15.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.301+0000 7f1f09ffb640 1 --2- 172.21.15.42:0/2443098222 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1ef80b3bd0 0x7f1ef80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.301+0000 7f1f03fff640 1 -- 172.21.15.42:0/1847836097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f769280 msgr2=0x55754fb2fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:15.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.301+0000 7f1f03fff640 1 --2- 172.21.15.42:0/1847836097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f769280 0x55754fb2fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.301+0000 7f1f03fff640 1 -- 172.21.15.42:0/1847836097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f7610a0 msgr2=0x55754fb2f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:15.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.301+0000 7f1f03fff640 1 --2- 172.21.15.42:0/1847836097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f7610a0 0x55754fb2f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.301+0000 7f1f03fff640 1 -- 172.21.15.42:0/1847836097 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55754fa3ba60 con 0x55754f789f60 2024-07-26T06:50:15.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.301+0000 7f1f1095b640 1 --2- 172.21.15.42:0/2443098222 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1ef80b3bd0 0x7f1ef80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:15.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.301+0000 7f1f03fff640 1 --2- 172.21.15.42:0/1847836097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f789f60 0x55754fb2f3a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1ef400bfa0 tx=0x7f1ef400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d768af68d62ab67 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:15.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.301+0000 7f1f09ffb640 1 -- 172.21.15.42:0/2443098222 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f1f041052c0 con 0x7f1f0c108030 2024-07-26T06:50:15.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.301+0000 7f1f027fc640 1 -- 172.21.15.42:0/1847836097 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef401a070 con 0x55754f789f60 2024-07-26T06:50:15.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.301+0000 7f1f027fc640 1 -- 172.21.15.42:0/1847836097 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ef4005020 con 0x55754f789f60 2024-07-26T06:50:15.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.301+0000 7f1f027fc640 1 -- 172.21.15.42:0/1847836097 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4004a00 con 0x55754f789f60 2024-07-26T06:50:15.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.302+0000 7f1f18942740 1 -- 172.21.15.42:0/1847836097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f789f60 msgr2=0x55754fb2f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:15.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.302+0000 7f1f1095b640 1 --2- 172.21.15.42:0/2443098222 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1ef80b3bd0 0x7f1ef80b60e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1f040c9460 tx=0x7f1f040c8930 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:15.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.302+0000 7f1f18942740 1 --2- 172.21.15.42:0/1847836097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f789f60 0x55754fb2f3a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1ef400bfa0 tx=0x7f1ef400b8e0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.302+0000 7f1f18942740 1 -- 172.21.15.42:0/1847836097 shutdown_connections 2024-07-26T06:50:15.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.302+0000 7f1f18942740 1 --2- 172.21.15.42:0/1847836097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f769280 0x55754fb2fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.302+0000 7f1f18942740 1 --2- 172.21.15.42:0/1847836097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f7610a0 0x55754fb2f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.302+0000 7f1f18942740 1 --2- 172.21.15.42:0/1847836097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f789f60 0x55754fb2f3a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.302+0000 7f1f18942740 1 -- 172.21.15.42:0/1847836097 >> 172.21.15.42:0/1847836097 conn(0x55754f77e350 msgr2=0x55754fb2d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:15.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.302+0000 7f1f18942740 1 -- 172.21.15.42:0/1847836097 shutdown_connections 2024-07-26T06:50:15.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.302+0000 7f1f18942740 1 -- 172.21.15.42:0/1847836097 wait complete. 2024-07-26T06:50:15.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.303+0000 7f1f18942740 1 -- start start 2024-07-26T06:50:15.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.303+0000 7f1f18942740 1 Processor -- start 2024-07-26T06:50:15.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.303+0000 7f1f18942740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f7610a0 0x55754fb3b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1f18942740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f769280 0x55754fb3b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1f18942740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f789f60 0x55754fb40e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1f18942740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55754f8f85f0 con 0x55754f789f60 2024-07-26T06:50:15.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1f18942740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55754fa378b0 con 0x55754f7610a0 2024-07-26T06:50:15.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1f18942740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55754fb3db80 con 0x55754f769280 2024-07-26T06:50:15.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1f18942740 10 client.0 _ll_register_callbacks cb 0x55754fa417f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:15.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1f03fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f789f60 0x55754fb40e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:15.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1f03fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f7610a0 0x55754fb3b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:15.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1f03fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f7610a0 0x55754fb3b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:40844/0 (socket says 172.21.15.42:40844) 2024-07-26T06:50:15.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1f03fff640 1 -- 172.21.15.42:0/1892458059 learned_addr learned my addr 172.21.15.42:0/1892458059 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:15.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1f0bfff640 1 --2- 172.21.15.42:0/1892458059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f769280 0x55754fb3b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:15.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1f03fff640 1 -- 172.21.15.42:0/1892458059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f769280 msgr2=0x55754fb3b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:15.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1f03fff640 1 --2- 172.21.15.42:0/1892458059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f769280 0x55754fb3b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1f03fff640 1 -- 172.21.15.42:0/1892458059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f789f60 msgr2=0x55754fb40e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:15.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1f03fff640 1 --2- 172.21.15.42:0/1892458059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f789f60 0x55754fb40e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1f03fff640 1 -- 172.21.15.42:0/1892458059 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55754f7f5540 con 0x55754f7610a0 2024-07-26T06:50:15.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1f03fff640 1 --2- 172.21.15.42:0/1892458059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f7610a0 0x55754fb3b0c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4014440 tx=0x7f1ef400b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2a34ea1947519cb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:15.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.304+0000 7f1eeb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:50:15.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.305+0000 7f1eeb7fe640 1 -- 172.21.15.42:0/1892458059 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4022020 con 0x55754f7610a0 2024-07-26T06:50:15.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.305+0000 7f1eeb7fe640 1 -- 172.21.15.42:0/1892458059 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f1ef40048c0 con 0x55754f7610a0 2024-07-26T06:50:15.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.305+0000 7f1f18942740 1 -- 172.21.15.42:0/1892458059 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ef4005020 con 0x55754f7610a0 2024-07-26T06:50:15.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.305+0000 7f1eeb7fe640 1 -- 172.21.15.42:0/1892458059 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1ef4028040 con 0x55754f7610a0 2024-07-26T06:50:15.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.305+0000 7f1f18942740 10 client.24968 fetch_fsmap learned FSMap version 28 2024-07-26T06:50:15.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.305+0000 7f1f18942740 1 -- 172.21.15.42:0/1892458059 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55754fb49ca0 con 0x55754f7610a0 2024-07-26T06:50:15.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.305+0000 7f1eeb7fe640 1 -- 172.21.15.42:0/1892458059 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ef4005020 con 0x55754f7610a0 2024-07-26T06:50:15.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.305+0000 7f1eeb7fe640 1 -- 172.21.15.42:0/1892458059 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1ef4010990 con 0x55754f7610a0 2024-07-26T06:50:15.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.305+0000 7f1f18942740 10 client.24968 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:50:15.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.305+0000 7f1f18942740 10 client.24968 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T06:50:15.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.305+0000 7f1f18942740 20 client.24968 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:15.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.305+0000 7f1f18942740 1 --2- 172.21.15.42:0/1892458059 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55754f7488e0 0x55754fb4d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.305+0000 7f1f18942740 4 client.24968 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:50:15.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.305+0000 7f1f18942740 1 -- 172.21.15.42:0/1892458059 --> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55754faa7890 con 0x55754f7488e0 2024-07-26T06:50:15.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.306+0000 7f1f0bfff640 1 --2- 172.21.15.42:0/1892458059 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55754f7488e0 0x55754fb4d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:15.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.306+0000 7f1eeb7fe640 10 client.24968 ms_handle_connect on v2:172.21.15.163:6836/154041066 2024-07-26T06:50:15.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.306+0000 7f1f0bfff640 1 --2- 172.21.15.42:0/1892458059 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55754f7488e0 0x55754fb4d230 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bec69dac481baa3e server_cookie=98b473d0fb6fd598 in_seq=0 out_seq=0 2024-07-26T06:50:15.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.307+0000 7f1eeb7fe640 1 -- 172.21.15.42:0/1892458059 <== mds.2 v2:172.21.15.163:6836/154041066 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55754faa7890 con 0x55754f7488e0 2024-07-26T06:50:15.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.307+0000 7f1eeb7fe640 10 client.24968 handle_command_reply: tid=0 2024-07-26T06:50:15.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.307+0000 7f1f18942740 1 client.24968 shutdown 2024-07-26T06:50:15.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.307+0000 7f1f18942740 1 -- 172.21.15.42:0/1892458059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f7610a0 msgr2=0x55754fb3b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:15.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.307+0000 7f1f18942740 1 --2- 172.21.15.42:0/1892458059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f7610a0 0x55754fb3b0c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4014440 tx=0x7f1ef400b390 comp rx=0 tx=0).stop 2024-07-26T06:50:15.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.307+0000 7f1f18942740 1 -- 172.21.15.42:0/1892458059 shutdown_connections 2024-07-26T06:50:15.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.307+0000 7f1f18942740 1 --2- 172.21.15.42:0/1892458059 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55754f7488e0 0x55754fb4d230 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.307+0000 7f1f18942740 1 --2- 172.21.15.42:0/1892458059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f789f60 0x55754fb40e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.307+0000 7f1f18942740 1 --2- 172.21.15.42:0/1892458059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f769280 0x55754fb3b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.307+0000 7f1f18942740 1 --2- 172.21.15.42:0/1892458059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f7610a0 0x55754fb3b0c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.307+0000 7f1f18942740 1 -- 172.21.15.42:0/1892458059 >> 172.21.15.42:0/1892458059 conn(0x55754f77e350 msgr2=0x55754fb32580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:15.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.307+0000 7f1eeb7fe640 0 client.24968 ms_handle_reset on v2:172.21.15.163:6836/154041066 2024-07-26T06:50:15.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.307+0000 7f1f18942740 1 -- 172.21.15.42:0/1892458059 shutdown_connections 2024-07-26T06:50:15.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.307+0000 7f1f18942740 1 -- 172.21.15.42:0/1892458059 wait complete. 2024-07-26T06:50:15.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.308+0000 7f1f18942740 20 client.24968 trim_cache size 0 max 16384 2024-07-26T06:50:15.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.325+0000 7f1f18942740 1 Processor -- start 2024-07-26T06:50:15.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.326+0000 7f1f18942740 1 -- start start 2024-07-26T06:50:15.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.326+0000 7f1f18942740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f7610a0 0x55754fb4c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.326+0000 7f1f18942740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f769280 0x55754fb47c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.326+0000 7f1f18942740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f789f60 0x55754fb40db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.326+0000 7f1f18942740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55754fb4b570 con 0x55754f7610a0 2024-07-26T06:50:15.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.326+0000 7f1f18942740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55754fb4bc40 con 0x55754f789f60 2024-07-26T06:50:15.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.326+0000 7f1f18942740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55754fb4c310 con 0x55754f769280 2024-07-26T06:50:15.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.326+0000 7f1f03fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f7610a0 0x55754fb4c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:15.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.326+0000 7f1f03fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f7610a0 0x55754fb4c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45048/0 (socket says 172.21.15.42:45048) 2024-07-26T06:50:15.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.326+0000 7f1f03fff640 1 -- 172.21.15.42:0/165790836 learned_addr learned my addr 172.21.15.42:0/165790836 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:15.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.326+0000 7f1f03fff640 1 -- 172.21.15.42:0/165790836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f769280 msgr2=0x55754fb47c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:15.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.326+0000 7f1f0bfff640 1 --2- 172.21.15.42:0/165790836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f769280 0x55754fb47c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:15.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.326+0000 7f1f03fff640 1 --2- 172.21.15.42:0/165790836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f769280 0x55754fb47c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.326+0000 7f1f03fff640 1 -- 172.21.15.42:0/165790836 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f789f60 msgr2=0x55754fb40db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:15.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.326+0000 7f1f03fff640 1 --2- 172.21.15.42:0/165790836 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f789f60 0x55754fb40db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.326+0000 7f1f03fff640 1 -- 172.21.15.42:0/165790836 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55754fb2de30 con 0x55754f7610a0 2024-07-26T06:50:15.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.326+0000 7f1f03fff640 1 --2- 172.21.15.42:0/165790836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f7610a0 0x55754fb4c490 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40058b0 tx=0x7f1ef4015ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0a2b6425266cf2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:15.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.327+0000 7f1f01ffb640 1 -- 172.21.15.42:0/165790836 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4022070 con 0x55754f7610a0 2024-07-26T06:50:15.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.327+0000 7f1f01ffb640 1 -- 172.21.15.42:0/165790836 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ef40141f0 con 0x55754f7610a0 2024-07-26T06:50:15.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.327+0000 7f1f01ffb640 1 -- 172.21.15.42:0/165790836 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef401a040 con 0x55754f7610a0 2024-07-26T06:50:15.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.327+0000 7f1f18942740 1 -- 172.21.15.42:0/165790836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f7610a0 msgr2=0x55754fb4c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:15.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.327+0000 7f1f18942740 1 --2- 172.21.15.42:0/165790836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f7610a0 0x55754fb4c490 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40058b0 tx=0x7f1ef4015ec0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.328+0000 7f1f18942740 1 -- 172.21.15.42:0/165790836 shutdown_connections 2024-07-26T06:50:15.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.328+0000 7f1f18942740 1 --2- 172.21.15.42:0/165790836 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f789f60 0x55754fb40db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.328+0000 7f1f18942740 1 --2- 172.21.15.42:0/165790836 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f769280 0x55754fb47c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.328+0000 7f1f18942740 1 --2- 172.21.15.42:0/165790836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f7610a0 0x55754fb4c490 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.328+0000 7f1f18942740 1 -- 172.21.15.42:0/165790836 >> 172.21.15.42:0/165790836 conn(0x55754f77e350 msgr2=0x55754fb2cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:15.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.328+0000 7f1f18942740 1 -- 172.21.15.42:0/165790836 shutdown_connections 2024-07-26T06:50:15.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.328+0000 7f1f18942740 1 -- 172.21.15.42:0/165790836 wait complete. 2024-07-26T06:50:15.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.328+0000 7f1f18942740 1 -- start start 2024-07-26T06:50:15.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.328+0000 7f1f18942740 1 Processor -- start 2024-07-26T06:50:15.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f18942740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f7610a0 0x55754fb335c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f18942740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f769280 0x55754fb33b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f18942740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f789f60 0x55754fb34040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f18942740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55754fb4d0b0 con 0x55754f769280 2024-07-26T06:50:15.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f18942740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55754fb4c9e0 con 0x55754f7610a0 2024-07-26T06:50:15.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f18942740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55754fb4d780 con 0x55754f789f60 2024-07-26T06:50:15.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f18942740 10 client.0 _ll_register_callbacks cb 0x7f1ef4005310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:15.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f0bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f769280 0x55754fb33b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:15.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f0bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f769280 0x55754fb33b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45060/0 (socket says 172.21.15.42:45060) 2024-07-26T06:50:15.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f0bfff640 1 -- 172.21.15.42:0/1018528746 learned_addr learned my addr 172.21.15.42:0/1018528746 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:15.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f03fff640 1 --2- 172.21.15.42:0/1018528746 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f7610a0 0x55754fb335c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:15.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f03fff640 1 --2- 172.21.15.42:0/1018528746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f789f60 0x55754fb34040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:15.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f0bfff640 1 -- 172.21.15.42:0/1018528746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f789f60 msgr2=0x55754fb34040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:15.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f0bfff640 1 --2- 172.21.15.42:0/1018528746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f789f60 0x55754fb34040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f0bfff640 1 -- 172.21.15.42:0/1018528746 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f7610a0 msgr2=0x55754fb335c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:15.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f0bfff640 1 --2- 172.21.15.42:0/1018528746 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f7610a0 0x55754fb335c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f0bfff640 1 -- 172.21.15.42:0/1018528746 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55754fb46ac0 con 0x55754f769280 2024-07-26T06:50:15.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f03fff640 1 --2- 172.21.15.42:0/1018528746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f789f60 0x55754fb34040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:15.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1f0bfff640 1 --2- 172.21.15.42:0/1018528746 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f769280 0x55754fb33b00 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x55754fb3b630 tx=0x7f1efc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=575c28c3e98174dd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:15.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.329+0000 7f1eeb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:15.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.330+0000 7f1eeb7fe640 1 -- 172.21.15.42:0/1018528746 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1efc011440 con 0x55754f769280 2024-07-26T06:50:15.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.330+0000 7f1eeb7fe640 1 -- 172.21.15.42:0/1018528746 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f1efc0115e0 con 0x55754f769280 2024-07-26T06:50:15.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.330+0000 7f1f18942740 1 -- 172.21.15.42:0/1018528746 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ef40141f0 con 0x55754f769280 2024-07-26T06:50:15.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.330+0000 7f1eeb7fe640 1 -- 172.21.15.42:0/1018528746 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1efc0117e0 con 0x55754f769280 2024-07-26T06:50:15.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.330+0000 7f1f18942740 10 client.15264 fetch_fsmap learned FSMap version 28 2024-07-26T06:50:15.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.330+0000 7f1f18942740 1 -- 172.21.15.42:0/1018528746 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55754fb47dd0 con 0x55754f769280 2024-07-26T06:50:15.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.330+0000 7f1eeb7fe640 1 -- 172.21.15.42:0/1018528746 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ef40141f0 con 0x55754f769280 2024-07-26T06:50:15.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.330+0000 7f1eeb7fe640 1 -- 172.21.15.42:0/1018528746 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1efc026ad0 con 0x55754f769280 2024-07-26T06:50:15.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.330+0000 7f1f18942740 10 client.15264 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:50:15.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.330+0000 7f1f18942740 10 client.15264 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T06:50:15.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.330+0000 7f1f18942740 20 client.15264 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:15.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.330+0000 7f1f18942740 1 --2- 172.21.15.42:0/1018528746 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55754f71dc90 0x55754fbb99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:15.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.330+0000 7f1f18942740 4 client.15264 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:50:15.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.330+0000 7f1f18942740 1 -- 172.21.15.42:0/1018528746 --> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] -- command(tid 0: {"prefix": "status"}) -- 0x55754faa3330 con 0x55754f71dc90 2024-07-26T06:50:15.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.331+0000 7f1f03fff640 1 --2- 172.21.15.42:0/1018528746 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55754f71dc90 0x55754fbb99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:15.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.331+0000 7f1f03fff640 1 --2- 172.21.15.42:0/1018528746 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55754f71dc90 0x55754fbb99f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9abe25fc88699ac2 server_cookie=5f12ec30aa56c149 in_seq=0 out_seq=0 2024-07-26T06:50:15.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.331+0000 7f1eeb7fe640 10 client.15264 ms_handle_connect on v2:172.21.15.163:6836/154041066 2024-07-26T06:50:15.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.331+0000 7f1eeb7fe640 1 -- 172.21.15.42:0/1018528746 <== mds.2 v2:172.21.15.163:6836/154041066 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55754faa3330 con 0x55754f71dc90 2024-07-26T06:50:15.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.331+0000 7f1eeb7fe640 10 client.15264 handle_command_reply: tid=0 2024-07-26T06:50:15.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.331+0000 7f1f18942740 1 client.15264 shutdown 2024-07-26T06:50:15.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.331+0000 7f1f18942740 1 -- 172.21.15.42:0/1018528746 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f769280 msgr2=0x55754fb33b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:15.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.331+0000 7f1f18942740 1 --2- 172.21.15.42:0/1018528746 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f769280 0x55754fb33b00 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x55754fb3b630 tx=0x7f1efc00be30 comp rx=0 tx=0).stop 2024-07-26T06:50:15.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f18942740 1 -- 172.21.15.42:0/1018528746 shutdown_connections 2024-07-26T06:50:15.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f18942740 1 --2- 172.21.15.42:0/1018528746 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55754f71dc90 0x55754fbb99f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f18942740 1 --2- 172.21.15.42:0/1018528746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55754f789f60 0x55754fb34040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f18942740 1 --2- 172.21.15.42:0/1018528746 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55754f769280 0x55754fb33b00 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f18942740 1 --2- 172.21.15.42:0/1018528746 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55754f7610a0 0x55754fb335c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1eeb7fe640 0 client.15264 ms_handle_reset on v2:172.21.15.163:6836/154041066 2024-07-26T06:50:15.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f18942740 1 -- 172.21.15.42:0/1018528746 >> 172.21.15.42:0/1018528746 conn(0x55754f77e350 msgr2=0x55754fb39650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:15.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f18942740 1 -- 172.21.15.42:0/1018528746 shutdown_connections 2024-07-26T06:50:15.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f18942740 1 -- 172.21.15.42:0/1018528746 wait complete. 2024-07-26T06:50:15.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f18942740 20 client.15264 trim_cache size 0 max 16384 2024-07-26T06:50:15.337 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:50:15.338 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_fsid": "8b3e2750-4b19-11ef-bcb6-c7b262605968", 2024-07-26T06:50:15.338 INFO:teuthology.orchestra.run.smithi042.stdout: "whoami": 2, 2024-07-26T06:50:15.338 INFO:teuthology.orchestra.run.smithi042.stdout: "id": 24593, 2024-07-26T06:50:15.338 INFO:teuthology.orchestra.run.smithi042.stdout: "want_state": "up:active", 2024-07-26T06:50:15.338 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "up:active", 2024-07-26T06:50:15.338 INFO:teuthology.orchestra.run.smithi042.stdout: "fs_name": "cephfs", 2024-07-26T06:50:15.338 INFO:teuthology.orchestra.run.smithi042.stdout: "rank_uptime": 175.25973943700001, 2024-07-26T06:50:15.338 INFO:teuthology.orchestra.run.smithi042.stdout: "mdsmap_epoch": 28, 2024-07-26T06:50:15.338 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch": 80, 2024-07-26T06:50:15.338 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch_barrier": 80, 2024-07-26T06:50:15.338 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 216.982091492 2024-07-26T06:50:15.338 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:50:15.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f123be640 1 -- 172.21.15.42:0/2443098222 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1ef80b3bd0 msgr2=0x7f1ef80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:15.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f123be640 1 --2- 172.21.15.42:0/2443098222 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1ef80b3bd0 0x7f1ef80b60e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1f040c9460 tx=0x7f1f040c8930 comp rx=0 tx=0).stop 2024-07-26T06:50:15.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f123be640 1 -- 172.21.15.42:0/2443098222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f0c108030 msgr2=0x7f1f0c1104b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:15.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f123be640 1 --2- 172.21.15.42:0/2443098222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f0c108030 0x7f1f0c1104b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1f04005cc0 tx=0x7f1f0402f380 comp rx=0 tx=0).stop 2024-07-26T06:50:15.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f123be640 1 -- 172.21.15.42:0/2443098222 shutdown_connections 2024-07-26T06:50:15.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f123be640 1 --2- 172.21.15.42:0/2443098222 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1ef80b3bd0 0x7f1ef80b60e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f123be640 1 --2- 172.21.15.42:0/2443098222 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1f0c108030 0x7f1f0c1104b0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f123be640 1 --2- 172.21.15.42:0/2443098222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f0c107670 0x7f1f0c10ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f123be640 1 --2- 172.21.15.42:0/2443098222 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1f0c106d20 0x7f1f0c10fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:15.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f123be640 1 -- 172.21.15.42:0/2443098222 >> 172.21.15.42:0/2443098222 conn(0x7f1f0c077230 msgr2=0x7f1f0c0fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:15.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.332+0000 7f1f123be640 1 -- 172.21.15.42:0/2443098222 shutdown_connections 2024-07-26T06:50:15.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:15.333+0000 7f1f123be640 1 -- 172.21.15.42:0/2443098222 wait complete. 2024-07-26T06:50:16.010 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell mds.i status 2024-07-26T06:50:16.277 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:16.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:16 smithi042 ceph-mon[32775]: pgmap v397: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:50:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:16 smithi163 ceph-mon[36873]: pgmap v397: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:50:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:16 smithi164 ceph-mon[36970]: pgmap v397: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:50:17.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.924+0000 7f053dae7640 1 -- 172.21.15.42:0/2277560789 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0538107dd0 msgr2=0x7f05381081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:17.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.924+0000 7f053dae7640 1 --2- 172.21.15.42:0/2277560789 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0538107dd0 0x7f05381081e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f052c009900 tx=0x7f052c02a410 comp rx=0 tx=0).stop 2024-07-26T06:50:17.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.924+0000 7f053dae7640 1 -- 172.21.15.42:0/2277560789 shutdown_connections 2024-07-26T06:50:17.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.924+0000 7f053dae7640 1 --2- 172.21.15.42:0/2277560789 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0538108720 0x7f053810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.924+0000 7f053dae7640 1 --2- 172.21.15.42:0/2277560789 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0538107dd0 0x7f05381081e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.924+0000 7f053dae7640 1 --2- 172.21.15.42:0/2277560789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0538102490 0x7f05381028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.924+0000 7f053dae7640 1 -- 172.21.15.42:0/2277560789 >> 172.21.15.42:0/2277560789 conn(0x7f0538079160 msgr2=0x7f0538079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:17.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.924+0000 7f053dae7640 1 -- 172.21.15.42:0/2277560789 shutdown_connections 2024-07-26T06:50:17.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.924+0000 7f053dae7640 1 -- 172.21.15.42:0/2277560789 wait complete. 2024-07-26T06:50:17.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.925+0000 7f053dae7640 1 Processor -- start 2024-07-26T06:50:17.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.925+0000 7f053dae7640 1 -- start start 2024-07-26T06:50:17.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.925+0000 7f053dae7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0538102490 0x7f053819ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.925+0000 7f053dae7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0538107dd0 0x7f053819b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.925+0000 7f053dae7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0538108720 0x7f053819b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.925+0000 7f053dae7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0538112370 con 0x7f0538102490 2024-07-26T06:50:17.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.925+0000 7f053dae7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f05381121f0 con 0x7f0538107dd0 2024-07-26T06:50:17.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.925+0000 7f053dae7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f05381124f0 con 0x7f0538108720 2024-07-26T06:50:17.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.925+0000 7f05377fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0538102490 0x7f053819ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.925+0000 7f05377fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0538102490 0x7f053819ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43494/0 (socket says 172.21.15.42:43494) 2024-07-26T06:50:17.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.925+0000 7f05377fe640 1 -- 172.21.15.42:0/2652283767 learned_addr learned my addr 172.21.15.42:0/2652283767 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:17.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.925+0000 7f0536ffd640 1 --2- 172.21.15.42:0/2652283767 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0538107dd0 0x7f053819b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.926+0000 7f0537fff640 1 --2- 172.21.15.42:0/2652283767 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0538108720 0x7f053819b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.926+0000 7f05377fe640 1 -- 172.21.15.42:0/2652283767 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0538108720 msgr2=0x7f053819b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:17.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.926+0000 7f05377fe640 1 --2- 172.21.15.42:0/2652283767 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0538108720 0x7f053819b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.926+0000 7f05377fe640 1 -- 172.21.15.42:0/2652283767 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0538107dd0 msgr2=0x7f053819b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:17.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.926+0000 7f05377fe640 1 --2- 172.21.15.42:0/2652283767 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0538107dd0 0x7f053819b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.926+0000 7f05377fe640 1 -- 172.21.15.42:0/2652283767 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05381a5f40 con 0x7f0538102490 2024-07-26T06:50:17.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.926+0000 7f0536ffd640 1 --2- 172.21.15.42:0/2652283767 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0538107dd0 0x7f053819b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:17.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.926+0000 7f05377fe640 1 --2- 172.21.15.42:0/2652283767 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0538102490 0x7f053819ad40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f052800c920 tx=0x7f052800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:17.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.926+0000 7f0534ff9640 1 -- 172.21.15.42:0/2652283767 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0528007c70 con 0x7f0538102490 2024-07-26T06:50:17.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.926+0000 7f0534ff9640 1 -- 172.21.15.42:0/2652283767 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0528007e10 con 0x7f0538102490 2024-07-26T06:50:17.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.926+0000 7f053dae7640 1 -- 172.21.15.42:0/2652283767 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05381a6230 con 0x7f0538102490 2024-07-26T06:50:17.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.926+0000 7f0534ff9640 1 -- 172.21.15.42:0/2652283767 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0528018820 con 0x7f0538102490 2024-07-26T06:50:17.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.926+0000 7f053dae7640 1 -- 172.21.15.42:0/2652283767 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05381a6740 con 0x7f0538102490 2024-07-26T06:50:17.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.929+0000 7f054406b740 1 Processor -- start 2024-07-26T06:50:17.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.929+0000 7f054406b740 1 -- start start 2024-07-26T06:50:17.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.930+0000 7f054406b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c83f60 0x55ae720293a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.930+0000 7f054406b740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c5b0a0 0x55ae720298e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.930+0000 7f054406b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c63280 0x55ae72029e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.930+0000 7f054406b740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55ae71f9f940 con 0x55ae71c5b0a0 2024-07-26T06:50:17.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.930+0000 7f054406b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ae71f8ee40 con 0x55ae71c83f60 2024-07-26T06:50:17.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.930+0000 7f054406b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55ae71f92e80 con 0x55ae71c63280 2024-07-26T06:50:17.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.930+0000 7f0534ff9640 1 -- 172.21.15.42:0/2652283767 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f052800f450 con 0x7f0538102490 2024-07-26T06:50:17.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.930+0000 7f0537fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c83f60 0x55ae720293a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.930+0000 7f05377fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c5b0a0 0x55ae720298e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.930+0000 7f0537fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c83f60 0x55ae720293a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:41642/0 (socket says 172.21.15.42:41642) 2024-07-26T06:50:17.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.930+0000 7f0537fff640 1 -- 172.21.15.42:0/982736759 learned_addr learned my addr 172.21.15.42:0/982736759 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:17.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.930+0000 7f0536ffd640 1 --2- 172.21.15.42:0/982736759 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c63280 0x55ae72029e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.931+0000 7f05377fe640 1 -- 172.21.15.42:0/982736759 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c63280 msgr2=0x55ae72029e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:17.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.931+0000 7f05377fe640 1 --2- 172.21.15.42:0/982736759 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c63280 0x55ae72029e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.931+0000 7f05377fe640 1 -- 172.21.15.42:0/982736759 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c83f60 msgr2=0x55ae720293a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:17.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.931+0000 7f05377fe640 1 --2- 172.21.15.42:0/982736759 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c83f60 0x55ae720293a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.931+0000 7f05377fe640 1 -- 172.21.15.42:0/982736759 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae71f35a60 con 0x55ae71c5b0a0 2024-07-26T06:50:17.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.931+0000 7f05377fe640 1 --2- 172.21.15.42:0/982736759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c5b0a0 0x55ae720298e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f05280e18a0 tx=0x7f05280ea470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420fa6c31add47ba server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:17.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.931+0000 7f0534ff9640 1 --2- 172.21.15.42:0/2652283767 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f05200b3c00 0x7f05200b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.931+0000 7f051dffb640 1 -- 172.21.15.42:0/982736759 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05280aa670 con 0x55ae71c5b0a0 2024-07-26T06:50:17.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.931+0000 7f0537fff640 1 --2- 172.21.15.42:0/2652283767 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f05200b3c00 0x7f05200b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.931+0000 7f051dffb640 1 -- 172.21.15.42:0/982736759 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05280d07f0 con 0x55ae71c5b0a0 2024-07-26T06:50:17.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.931+0000 7f051dffb640 1 -- 172.21.15.42:0/982736759 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05280aa860 con 0x55ae71c5b0a0 2024-07-26T06:50:17.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.931+0000 7f0534ff9640 1 -- 172.21.15.42:0/2652283767 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f05280e3df0 con 0x7f0538102490 2024-07-26T06:50:17.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.932+0000 7f054406b740 1 -- 172.21.15.42:0/982736759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c5b0a0 msgr2=0x55ae720298e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:17.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.932+0000 7f0537fff640 1 --2- 172.21.15.42:0/2652283767 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f05200b3c00 0x7f05200b6110 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0530008e40 tx=0x7f0530008590 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:17.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.932+0000 7f054406b740 1 --2- 172.21.15.42:0/982736759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c5b0a0 0x55ae720298e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f05280e18a0 tx=0x7f05280ea470 comp rx=0 tx=0).stop 2024-07-26T06:50:17.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.932+0000 7f054406b740 1 -- 172.21.15.42:0/982736759 shutdown_connections 2024-07-26T06:50:17.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.932+0000 7f054406b740 1 --2- 172.21.15.42:0/982736759 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c63280 0x55ae72029e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.932+0000 7f054406b740 1 --2- 172.21.15.42:0/982736759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c5b0a0 0x55ae720298e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.932+0000 7f054406b740 1 --2- 172.21.15.42:0/982736759 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c83f60 0x55ae720293a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.932+0000 7f054406b740 1 -- 172.21.15.42:0/982736759 >> 172.21.15.42:0/982736759 conn(0x55ae71c78350 msgr2=0x55ae72027290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:17.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.932+0000 7f054406b740 1 -- 172.21.15.42:0/982736759 shutdown_connections 2024-07-26T06:50:17.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.932+0000 7f054406b740 1 -- 172.21.15.42:0/982736759 wait complete. 2024-07-26T06:50:17.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.933+0000 7f054406b740 1 -- start start 2024-07-26T06:50:17.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.933+0000 7f054406b740 1 Processor -- start 2024-07-26T06:50:17.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.933+0000 7f054406b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c5b0a0 0x55ae72034f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.933+0000 7f054406b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c63280 0x55ae72035480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.933+0000 7f054406b740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c83f60 0x55ae72040ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.933+0000 7f054406b740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55ae71df25f0 con 0x55ae71c83f60 2024-07-26T06:50:17.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.933+0000 7f054406b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ae71f318b0 con 0x55ae71c5b0a0 2024-07-26T06:50:17.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.933+0000 7f054406b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55ae72037b80 con 0x55ae71c63280 2024-07-26T06:50:17.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.933+0000 7f054406b740 10 client.0 _ll_register_callbacks cb 0x55ae71f3b7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:17.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.934+0000 7f0537fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c83f60 0x55ae72040ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.934+0000 7f0537fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c83f60 0x55ae72040ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43508/0 (socket says 172.21.15.42:43508) 2024-07-26T06:50:17.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.934+0000 7f0537fff640 1 -- 172.21.15.42:0/3471729062 learned_addr learned my addr 172.21.15.42:0/3471729062 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:17.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.934+0000 7f05377fe640 1 --2- 172.21.15.42:0/3471729062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c5b0a0 0x55ae72034f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.934+0000 7f0536ffd640 1 --2- 172.21.15.42:0/3471729062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c63280 0x55ae72035480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.934+0000 7f0537fff640 1 -- 172.21.15.42:0/3471729062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c63280 msgr2=0x55ae72035480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:17.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.934+0000 7f0537fff640 1 --2- 172.21.15.42:0/3471729062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c63280 0x55ae72035480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.934+0000 7f0537fff640 1 -- 172.21.15.42:0/3471729062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c5b0a0 msgr2=0x55ae72034f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:17.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.934+0000 7f0537fff640 1 --2- 172.21.15.42:0/3471729062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c5b0a0 0x55ae72034f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.934+0000 7f0537fff640 1 -- 172.21.15.42:0/3471729062 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae71cef540 con 0x55ae71c83f60 2024-07-26T06:50:17.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.934+0000 7f0537fff640 1 --2- 172.21.15.42:0/3471729062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c83f60 0x55ae72040ea0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0530005cc0 tx=0x7f0530004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=479bf2510014ce77 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:17.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.934+0000 7f0516ffd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:17.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.934+0000 7f0516ffd640 1 -- 172.21.15.42:0/3471729062 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0530025400 con 0x55ae71c83f60 2024-07-26T06:50:17.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.934+0000 7f0516ffd640 1 -- 172.21.15.42:0/3471729062 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f05300255a0 con 0x55ae71c83f60 2024-07-26T06:50:17.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.934+0000 7f054406b740 1 -- 172.21.15.42:0/3471729062 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05280d07f0 con 0x55ae71c83f60 2024-07-26T06:50:17.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.935+0000 7f0516ffd640 1 -- 172.21.15.42:0/3471729062 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f053002b070 con 0x55ae71c83f60 2024-07-26T06:50:17.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.935+0000 7f054406b740 10 client.15276 fetch_fsmap learned FSMap version 28 2024-07-26T06:50:17.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.935+0000 7f054406b740 1 -- 172.21.15.42:0/3471729062 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ae72043ba0 con 0x55ae71c83f60 2024-07-26T06:50:17.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.935+0000 7f0516ffd640 1 -- 172.21.15.42:0/3471729062 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05280d07f0 con 0x55ae71c83f60 2024-07-26T06:50:17.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.935+0000 7f0516ffd640 1 -- 172.21.15.42:0/3471729062 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f053001d800 con 0x55ae71c83f60 2024-07-26T06:50:17.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.935+0000 7f054406b740 10 client.15276 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:50:17.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.935+0000 7f054406b740 10 client.15276 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T06:50:17.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.935+0000 7f054406b740 20 client.15276 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.935+0000 7f054406b740 1 --2- 172.21.15.42:0/3471729062 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x55ae71c428e0 0x55ae72047650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.935+0000 7f054406b740 4 client.15276 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:50:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.935+0000 7f054406b740 1 -- 172.21.15.42:0/3471729062 --> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae71fa1890 con 0x55ae71c428e0 2024-07-26T06:50:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.935+0000 7f05377fe640 1 --2- 172.21.15.42:0/3471729062 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x55ae71c428e0 0x55ae72047650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.936+0000 7f0516ffd640 10 client.15276 ms_handle_connect on v2:172.21.15.164:6838/3593178314 2024-07-26T06:50:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.936+0000 7f05377fe640 1 --2- 172.21.15.42:0/3471729062 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x55ae71c428e0 0x55ae72047650 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ce04bbacf96b2ad2 server_cookie=ca40a3198a2956d in_seq=0 out_seq=0 2024-07-26T06:50:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.937+0000 7f0516ffd640 1 -- 172.21.15.42:0/3471729062 <== mds.1 v2:172.21.15.164:6838/3593178314 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ae71fa1890 con 0x55ae71c428e0 2024-07-26T06:50:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.937+0000 7f0516ffd640 10 client.15276 handle_command_reply: tid=0 2024-07-26T06:50:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.937+0000 7f054406b740 1 client.15276 shutdown 2024-07-26T06:50:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.937+0000 7f054406b740 1 -- 172.21.15.42:0/3471729062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c83f60 msgr2=0x55ae72040ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.937+0000 7f054406b740 1 --2- 172.21.15.42:0/3471729062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c83f60 0x55ae72040ea0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0530005cc0 tx=0x7f0530004a40 comp rx=0 tx=0).stop 2024-07-26T06:50:17.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.937+0000 7f054406b740 1 -- 172.21.15.42:0/3471729062 shutdown_connections 2024-07-26T06:50:17.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.937+0000 7f054406b740 1 --2- 172.21.15.42:0/3471729062 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x55ae71c428e0 0x55ae72047650 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.937+0000 7f054406b740 1 --2- 172.21.15.42:0/3471729062 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c83f60 0x55ae72040ea0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.937+0000 7f054406b740 1 --2- 172.21.15.42:0/3471729062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c63280 0x55ae72035480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.937+0000 7f054406b740 1 --2- 172.21.15.42:0/3471729062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c5b0a0 0x55ae72034f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.937+0000 7f054406b740 1 -- 172.21.15.42:0/3471729062 >> 172.21.15.42:0/3471729062 conn(0x55ae71c78350 msgr2=0x55ae7202c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:17.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.937+0000 7f0516ffd640 0 client.15276 ms_handle_reset on v2:172.21.15.164:6838/3593178314 2024-07-26T06:50:17.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.937+0000 7f054406b740 1 -- 172.21.15.42:0/3471729062 shutdown_connections 2024-07-26T06:50:17.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.938+0000 7f054406b740 1 -- 172.21.15.42:0/3471729062 wait complete. 2024-07-26T06:50:17.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.938+0000 7f054406b740 20 client.15276 trim_cache size 0 max 16384 2024-07-26T06:50:17.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.956+0000 7f054406b740 1 Processor -- start 2024-07-26T06:50:17.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.956+0000 7f054406b740 1 -- start start 2024-07-26T06:50:17.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.956+0000 7f054406b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c5b0a0 0x55ae72040e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.956+0000 7f054406b740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c63280 0x55ae72041350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.956+0000 7f054406b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c83f60 0x55ae7202e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.956+0000 7f054406b740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55ae72045990 con 0x55ae71c63280 2024-07-26T06:50:17.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.956+0000 7f054406b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ae72046060 con 0x55ae71c5b0a0 2024-07-26T06:50:17.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.956+0000 7f054406b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55ae72046730 con 0x55ae71c83f60 2024-07-26T06:50:17.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.956+0000 7f0536ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c63280 0x55ae72041350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.956+0000 7f0536ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c63280 0x55ae72041350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43520/0 (socket says 172.21.15.42:43520) 2024-07-26T06:50:17.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.956+0000 7f0536ffd640 1 -- 172.21.15.42:0/2479725021 learned_addr learned my addr 172.21.15.42:0/2479725021 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:17.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.957+0000 7f05377fe640 1 --2- 172.21.15.42:0/2479725021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c5b0a0 0x55ae72040e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.957+0000 7f0537fff640 1 --2- 172.21.15.42:0/2479725021 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c83f60 0x55ae7202e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.957+0000 7f0536ffd640 1 -- 172.21.15.42:0/2479725021 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c83f60 msgr2=0x55ae7202e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:17.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.957+0000 7f0536ffd640 1 --2- 172.21.15.42:0/2479725021 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c83f60 0x55ae7202e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.957+0000 7f0536ffd640 1 -- 172.21.15.42:0/2479725021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c5b0a0 msgr2=0x55ae72040e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:17.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.957+0000 7f0536ffd640 1 --2- 172.21.15.42:0/2479725021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c5b0a0 0x55ae72040e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.957+0000 7f0536ffd640 1 -- 172.21.15.42:0/2479725021 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae72036000 con 0x55ae71c63280 2024-07-26T06:50:17.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.957+0000 7f0536ffd640 1 --2- 172.21.15.42:0/2479725021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c63280 0x55ae72041350 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f053806a4d0 tx=0x7f052c0092f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df6ea1fbb2e23c85 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:17.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.957+0000 7f051d7fa640 1 -- 172.21.15.42:0/2479725021 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c035040 con 0x55ae71c63280 2024-07-26T06:50:17.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.957+0000 7f051d7fa640 1 -- 172.21.15.42:0/2479725021 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f052c0317e0 con 0x55ae71c63280 2024-07-26T06:50:17.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.957+0000 7f051d7fa640 1 -- 172.21.15.42:0/2479725021 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c004a00 con 0x55ae71c63280 2024-07-26T06:50:17.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.958+0000 7f054406b740 1 -- 172.21.15.42:0/2479725021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c63280 msgr2=0x55ae72041350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:17.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.958+0000 7f054406b740 1 --2- 172.21.15.42:0/2479725021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c63280 0x55ae72041350 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f053806a4d0 tx=0x7f052c0092f0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.958+0000 7f054406b740 1 -- 172.21.15.42:0/2479725021 shutdown_connections 2024-07-26T06:50:17.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.958+0000 7f054406b740 1 --2- 172.21.15.42:0/2479725021 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c83f60 0x55ae7202e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.958+0000 7f054406b740 1 --2- 172.21.15.42:0/2479725021 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c63280 0x55ae72041350 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.958+0000 7f054406b740 1 --2- 172.21.15.42:0/2479725021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c5b0a0 0x55ae72040e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.958+0000 7f054406b740 1 -- 172.21.15.42:0/2479725021 >> 172.21.15.42:0/2479725021 conn(0x55ae71c78350 msgr2=0x55ae720335c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:17.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.958+0000 7f054406b740 1 -- 172.21.15.42:0/2479725021 shutdown_connections 2024-07-26T06:50:17.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.958+0000 7f054406b740 1 -- 172.21.15.42:0/2479725021 wait complete. 2024-07-26T06:50:17.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.958+0000 7f054406b740 1 -- start start 2024-07-26T06:50:17.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.959+0000 7f054406b740 1 Processor -- start 2024-07-26T06:50:17.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.959+0000 7f054406b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c5b0a0 0x55ae720498a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.959+0000 7f054406b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c63280 0x55ae72049e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.959+0000 7f054406b740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c83f60 0x55ae7204e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.959+0000 7f054406b740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55ae720474d0 con 0x55ae71c83f60 2024-07-26T06:50:17.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.959+0000 7f054406b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ae72046e00 con 0x55ae71c63280 2024-07-26T06:50:17.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.959+0000 7f054406b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55ae72047ba0 con 0x55ae71c5b0a0 2024-07-26T06:50:17.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.959+0000 7f054406b740 10 client.0 _ll_register_callbacks cb 0x7f053000ea50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:17.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.959+0000 7f0537fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c83f60 0x55ae7204e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.959+0000 7f0537fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c83f60 0x55ae7204e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43530/0 (socket says 172.21.15.42:43530) 2024-07-26T06:50:17.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.959+0000 7f0537fff640 1 -- 172.21.15.42:0/340490589 learned_addr learned my addr 172.21.15.42:0/340490589 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:17.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.959+0000 7f05377fe640 1 --2- 172.21.15.42:0/340490589 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c5b0a0 0x55ae720498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.959+0000 7f0536ffd640 1 --2- 172.21.15.42:0/340490589 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c63280 0x55ae72049e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.960+0000 7f0537fff640 1 -- 172.21.15.42:0/340490589 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c5b0a0 msgr2=0x55ae720498a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:17.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.960+0000 7f0537fff640 1 --2- 172.21.15.42:0/340490589 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c5b0a0 0x55ae720498a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.960+0000 7f0537fff640 1 -- 172.21.15.42:0/340490589 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c63280 msgr2=0x55ae72049e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:17.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.960+0000 7f0537fff640 1 --2- 172.21.15.42:0/340490589 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c63280 0x55ae72049e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.960+0000 7f0537fff640 1 -- 172.21.15.42:0/340490589 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae7204ee30 con 0x55ae71c83f60 2024-07-26T06:50:17.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.960+0000 7f05377fe640 1 --2- 172.21.15.42:0/340490589 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c5b0a0 0x55ae720498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:50:17.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.960+0000 7f0537fff640 1 --2- 172.21.15.42:0/340490589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c83f60 0x55ae7204e8f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0530008700 tx=0x7f053000dea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfcff1887ab8109c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:17.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.960+0000 7f0516ffd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:17.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.960+0000 7f0516ffd640 1 -- 172.21.15.42:0/340490589 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0530004e00 con 0x55ae71c83f60 2024-07-26T06:50:17.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.960+0000 7f0516ffd640 1 -- 172.21.15.42:0/340490589 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f053001d3f0 con 0x55ae71c83f60 2024-07-26T06:50:17.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.960+0000 7f054406b740 1 -- 172.21.15.42:0/340490589 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f052c0317e0 con 0x55ae71c83f60 2024-07-26T06:50:17.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.960+0000 7f0516ffd640 1 -- 172.21.15.42:0/340490589 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0530040070 con 0x55ae71c83f60 2024-07-26T06:50:17.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.960+0000 7f054406b740 10 client.15282 fetch_fsmap learned FSMap version 28 2024-07-26T06:50:17.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.960+0000 7f054406b740 1 -- 172.21.15.42:0/340490589 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ae720500b0 con 0x55ae71c83f60 2024-07-26T06:50:17.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.961+0000 7f0516ffd640 1 -- 172.21.15.42:0/340490589 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f052c0317e0 con 0x55ae71c83f60 2024-07-26T06:50:17.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.961+0000 7f0516ffd640 1 -- 172.21.15.42:0/340490589 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0530030480 con 0x55ae71c83f60 2024-07-26T06:50:17.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.961+0000 7f054406b740 10 client.15282 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:50:17.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.961+0000 7f054406b740 10 client.15282 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T06:50:17.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.961+0000 7f054406b740 20 client.15282 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:17.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.961+0000 7f054406b740 1 --2- 172.21.15.42:0/340490589 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x55ae71c17c90 0x55ae720b3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:17.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.961+0000 7f054406b740 4 client.15282 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:50:17.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.961+0000 7f054406b740 1 -- 172.21.15.42:0/340490589 --> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] -- command(tid 0: {"prefix": "status"}) -- 0x55ae71f9d330 con 0x55ae71c17c90 2024-07-26T06:50:17.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.961+0000 7f05377fe640 1 --2- 172.21.15.42:0/340490589 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x55ae71c17c90 0x55ae720b3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:17.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.962+0000 7f05377fe640 1 --2- 172.21.15.42:0/340490589 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x55ae71c17c90 0x55ae720b3990 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=86d048758d4b684b server_cookie=70e4f9b64dfa4e4d in_seq=0 out_seq=0 2024-07-26T06:50:17.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.962+0000 7f0516ffd640 10 client.15282 ms_handle_connect on v2:172.21.15.164:6838/3593178314 2024-07-26T06:50:17.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.962+0000 7f0516ffd640 1 -- 172.21.15.42:0/340490589 <== mds.1 v2:172.21.15.164:6838/3593178314 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55ae71f9d330 con 0x55ae71c17c90 2024-07-26T06:50:17.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.962+0000 7f0516ffd640 10 client.15282 handle_command_reply: tid=0 2024-07-26T06:50:17.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.962+0000 7f054406b740 1 client.15282 shutdown 2024-07-26T06:50:17.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.962+0000 7f054406b740 1 -- 172.21.15.42:0/340490589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c83f60 msgr2=0x55ae7204e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:17.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.962+0000 7f054406b740 1 --2- 172.21.15.42:0/340490589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c83f60 0x55ae7204e8f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0530008700 tx=0x7f053000dea0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.962+0000 7f054406b740 1 -- 172.21.15.42:0/340490589 shutdown_connections 2024-07-26T06:50:17.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.962+0000 7f054406b740 1 --2- 172.21.15.42:0/340490589 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x55ae71c17c90 0x55ae720b3990 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.962+0000 7f054406b740 1 --2- 172.21.15.42:0/340490589 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55ae71c83f60 0x55ae7204e8f0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.962+0000 7f054406b740 1 --2- 172.21.15.42:0/340490589 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae71c63280 0x55ae72049e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.962+0000 7f054406b740 1 --2- 172.21.15.42:0/340490589 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55ae71c5b0a0 0x55ae720498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.962+0000 7f054406b740 1 -- 172.21.15.42:0/340490589 >> 172.21.15.42:0/340490589 conn(0x55ae71c78350 msgr2=0x55ae720361d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:17.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.962+0000 7f0516ffd640 0 client.15282 ms_handle_reset on v2:172.21.15.164:6838/3593178314 2024-07-26T06:50:17.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.963+0000 7f054406b740 1 -- 172.21.15.42:0/340490589 shutdown_connections 2024-07-26T06:50:17.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.963+0000 7f054406b740 1 -- 172.21.15.42:0/340490589 wait complete. 2024-07-26T06:50:17.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.963+0000 7f054406b740 20 client.15282 trim_cache size 0 max 16384 2024-07-26T06:50:17.969 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:50:17.969 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_fsid": "8b3e2750-4b19-11ef-bcb6-c7b262605968", 2024-07-26T06:50:17.969 INFO:teuthology.orchestra.run.smithi042.stdout: "whoami": 1, 2024-07-26T06:50:17.969 INFO:teuthology.orchestra.run.smithi042.stdout: "id": 24601, 2024-07-26T06:50:17.969 INFO:teuthology.orchestra.run.smithi042.stdout: "want_state": "up:active", 2024-07-26T06:50:17.969 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "up:active", 2024-07-26T06:50:17.969 INFO:teuthology.orchestra.run.smithi042.stdout: "fs_name": "cephfs", 2024-07-26T06:50:17.969 INFO:teuthology.orchestra.run.smithi042.stdout: "rank_uptime": 178.89873692, 2024-07-26T06:50:17.969 INFO:teuthology.orchestra.run.smithi042.stdout: "mdsmap_epoch": 28, 2024-07-26T06:50:17.969 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch": 80, 2024-07-26T06:50:17.969 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch_barrier": 80, 2024-07-26T06:50:17.970 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 216.98913817600001 2024-07-26T06:50:17.970 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:50:17.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.963+0000 7f053dae7640 1 -- 172.21.15.42:0/2652283767 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f05200b3c00 msgr2=0x7f05200b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:17.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.963+0000 7f053dae7640 1 --2- 172.21.15.42:0/2652283767 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f05200b3c00 0x7f05200b6110 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0530008e40 tx=0x7f0530008590 comp rx=0 tx=0).stop 2024-07-26T06:50:17.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.963+0000 7f053dae7640 1 -- 172.21.15.42:0/2652283767 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0538102490 msgr2=0x7f053819ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:17.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.963+0000 7f053dae7640 1 --2- 172.21.15.42:0/2652283767 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0538102490 0x7f053819ad40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f052800c920 tx=0x7f052800cde0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.963+0000 7f053dae7640 1 -- 172.21.15.42:0/2652283767 shutdown_connections 2024-07-26T06:50:17.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.963+0000 7f053dae7640 1 --2- 172.21.15.42:0/2652283767 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f05200b3c00 0x7f05200b6110 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.963+0000 7f053dae7640 1 --2- 172.21.15.42:0/2652283767 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0538108720 0x7f053819b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.963+0000 7f053dae7640 1 --2- 172.21.15.42:0/2652283767 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0538107dd0 0x7f053819b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.963+0000 7f053dae7640 1 --2- 172.21.15.42:0/2652283767 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0538102490 0x7f053819ad40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:17.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.963+0000 7f053dae7640 1 -- 172.21.15.42:0/2652283767 >> 172.21.15.42:0/2652283767 conn(0x7f0538079160 msgr2=0x7f0538104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:17.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.964+0000 7f053dae7640 1 -- 172.21.15.42:0/2652283767 shutdown_connections 2024-07-26T06:50:17.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:17.964+0000 7f053dae7640 1 -- 172.21.15.42:0/2652283767 wait complete. 2024-07-26T06:50:18.643 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:18 smithi042 ceph-mon[32775]: pgmap v398: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:50:18.675 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell mds.b status 2024-07-26T06:50:18.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:18 smithi163 ceph-mon[36873]: pgmap v398: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:50:18.944 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:18 smithi164 ceph-mon[36970]: pgmap v398: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:50:20.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:20 smithi042 ceph-mon[32775]: pgmap v399: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:50:20.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.668+0000 7fd04acc8640 1 -- 172.21.15.42:0/615307483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd04410fc30 msgr2=0x7fd044112050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:20.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.668+0000 7fd04acc8640 1 --2- 172.21.15.42:0/615307483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd04410fc30 0x7fd044112050 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd03c009ca0 tx=0x7fd03c02a950 comp rx=0 tx=0).stop 2024-07-26T06:50:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.668+0000 7fd04acc8640 1 -- 172.21.15.42:0/615307483 shutdown_connections 2024-07-26T06:50:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.668+0000 7fd04acc8640 1 --2- 172.21.15.42:0/615307483 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd04410fc30 0x7fd044112050 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.668+0000 7fd04acc8640 1 --2- 172.21.15.42:0/615307483 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd0441029d0 0x7fd04410f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.668+0000 7fd04acc8640 1 --2- 172.21.15.42:0/615307483 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd044102080 0x7fd044102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.668+0000 7fd04acc8640 1 -- 172.21.15.42:0/615307483 >> 172.21.15.42:0/615307483 conn(0x7fd0440791a0 msgr2=0x7fd0440795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.668+0000 7fd04acc8640 1 -- 172.21.15.42:0/615307483 shutdown_connections 2024-07-26T06:50:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.668+0000 7fd04acc8640 1 -- 172.21.15.42:0/615307483 wait complete. 2024-07-26T06:50:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.669+0000 7fd04acc8640 1 Processor -- start 2024-07-26T06:50:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.669+0000 7fd04acc8640 1 -- start start 2024-07-26T06:50:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.669+0000 7fd04acc8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd044102080 0x7fd04419c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.669+0000 7fd04acc8640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd0441029d0 0x7fd04419c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.669+0000 7fd04acc8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd04410fc30 0x7fd0441a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.669+0000 7fd04acc8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd0441156f0 con 0x7fd04410fc30 2024-07-26T06:50:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.669+0000 7fd04acc8640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd044115570 con 0x7fd0441029d0 2024-07-26T06:50:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.669+0000 7fd04acc8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd044115870 con 0x7fd044102080 2024-07-26T06:50:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.669+0000 7fd049265640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd04410fc30 0x7fd0441a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.669+0000 7fd049265640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd04410fc30 0x7fd0441a1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43548/0 (socket says 172.21.15.42:43548) 2024-07-26T06:50:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.669+0000 7fd049265640 1 -- 172.21.15.42:0/827160693 learned_addr learned my addr 172.21.15.42:0/827160693 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.670+0000 7fd043fff640 1 --2- 172.21.15.42:0/827160693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd0441029d0 0x7fd04419c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.670+0000 7fd048a64640 1 --2- 172.21.15.42:0/827160693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd044102080 0x7fd04419c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.670+0000 7fd049265640 1 -- 172.21.15.42:0/827160693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd044102080 msgr2=0x7fd04419c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.670+0000 7fd049265640 1 --2- 172.21.15.42:0/827160693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd044102080 0x7fd04419c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.670+0000 7fd049265640 1 -- 172.21.15.42:0/827160693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd0441029d0 msgr2=0x7fd04419c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.670+0000 7fd049265640 1 --2- 172.21.15.42:0/827160693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd0441029d0 0x7fd04419c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.670+0000 7fd049265640 1 -- 172.21.15.42:0/827160693 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0441a2090 con 0x7fd04410fc30 2024-07-26T06:50:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.670+0000 7fd049265640 1 --2- 172.21.15.42:0/827160693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd04410fc30 0x7fd0441a1960 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd03c02ae70 tx=0x7fd03c02f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.670+0000 7fd041ffb640 1 -- 172.21.15.42:0/827160693 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c034040 con 0x7fd04410fc30 2024-07-26T06:50:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.670+0000 7fd04acc8640 1 -- 172.21.15.42:0/827160693 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0441a2320 con 0x7fd04410fc30 2024-07-26T06:50:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.670+0000 7fd041ffb640 1 -- 172.21.15.42:0/827160693 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd03c0308f0 con 0x7fd04410fc30 2024-07-26T06:50:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.670+0000 7fd041ffb640 1 -- 172.21.15.42:0/827160693 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c0047f0 con 0x7fd04410fc30 2024-07-26T06:50:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.670+0000 7fd04acc8640 1 -- 172.21.15.42:0/827160693 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0441a8ae0 con 0x7fd04410fc30 2024-07-26T06:50:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.674+0000 7fd041ffb640 1 -- 172.21.15.42:0/827160693 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fd03c037020 con 0x7fd04410fc30 2024-07-26T06:50:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.674+0000 7fd05124c740 1 Processor -- start 2024-07-26T06:50:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.675+0000 7fd05124c740 1 -- start start 2024-07-26T06:50:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.675+0000 7fd041ffb640 1 --2- 172.21.15.42:0/827160693 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd02c0b3bd0 0x7fd02c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.675+0000 7fd043fff640 1 --2- 172.21.15.42:0/827160693 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd02c0b3bd0 0x7fd02c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.675+0000 7fd05124c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2223f60 0x5618a25c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.675+0000 7fd041ffb640 1 -- 172.21.15.42:0/827160693 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fd03c104c90 con 0x7fd04410fc30 2024-07-26T06:50:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.675+0000 7fd05124c740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21fb0a0 0x5618a25c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.675+0000 7fd043fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21fb0a0 0x5618a25c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.675+0000 7fd05124c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a2203280 0x5618a25c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.675+0000 7fd05124c740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5618a253f940 con 0x5618a21fb0a0 2024-07-26T06:50:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.675+0000 7fd043fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21fb0a0 0x5618a25c98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43564/0 (socket says 172.21.15.42:43564) 2024-07-26T06:50:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.675+0000 7fd043fff640 1 -- 172.21.15.42:0/4099746804 learned_addr learned my addr 172.21.15.42:0/4099746804 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.675+0000 7fd05124c740 1 -- 172.21.15.42:0/4099746804 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5618a252ee40 con 0x5618a2223f60 2024-07-26T06:50:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.675+0000 7fd05124c740 1 -- 172.21.15.42:0/4099746804 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5618a2532e80 con 0x5618a2203280 2024-07-26T06:50:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.675+0000 7fd048a64640 1 --2- 172.21.15.42:0/4099746804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2223f60 0x5618a25c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.676+0000 7fd049265640 1 --2- 172.21.15.42:0/4099746804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a2203280 0x5618a25c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.676+0000 7fd043fff640 1 --2- 172.21.15.42:0/827160693 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd02c0b3bd0 0x7fd02c0b60e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd038013510 tx=0x7fd038013910 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.676+0000 7fd043fff640 1 -- 172.21.15.42:0/4099746804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a2203280 msgr2=0x5618a25c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.676+0000 7fd043fff640 1 --2- 172.21.15.42:0/4099746804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a2203280 0x5618a25c9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.676+0000 7fd043fff640 1 -- 172.21.15.42:0/4099746804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2223f60 msgr2=0x5618a25c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.676+0000 7fd043fff640 1 --2- 172.21.15.42:0/4099746804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2223f60 0x5618a25c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.676+0000 7fd043fff640 1 -- 172.21.15.42:0/4099746804 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618a24d5a60 con 0x5618a21fb0a0 2024-07-26T06:50:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.676+0000 7fd049265640 1 --2- 172.21.15.42:0/4099746804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a2203280 0x5618a25c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.676+0000 7fd048a64640 1 --2- 172.21.15.42:0/4099746804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2223f60 0x5618a25c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:50:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.676+0000 7fd043fff640 1 --2- 172.21.15.42:0/4099746804 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21fb0a0 0x5618a25c98e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd03800ba40 tx=0x7fd03801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27613c49a3f5b570 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.676+0000 7fd02affd640 1 -- 172.21.15.42:0/4099746804 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03801f7c0 con 0x5618a21fb0a0 2024-07-26T06:50:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.676+0000 7fd02affd640 1 -- 172.21.15.42:0/4099746804 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd03801fe10 con 0x5618a21fb0a0 2024-07-26T06:50:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.676+0000 7fd02affd640 1 -- 172.21.15.42:0/4099746804 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03801ede0 con 0x5618a21fb0a0 2024-07-26T06:50:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.677+0000 7fd05124c740 1 -- 172.21.15.42:0/4099746804 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21fb0a0 msgr2=0x5618a25c98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.677+0000 7fd05124c740 1 --2- 172.21.15.42:0/4099746804 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21fb0a0 0x5618a25c98e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd03800ba40 tx=0x7fd03801d000 comp rx=0 tx=0).stop 2024-07-26T06:50:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.678+0000 7fd05124c740 1 -- 172.21.15.42:0/4099746804 shutdown_connections 2024-07-26T06:50:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.678+0000 7fd05124c740 1 --2- 172.21.15.42:0/4099746804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a2203280 0x5618a25c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.678+0000 7fd05124c740 1 --2- 172.21.15.42:0/4099746804 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21fb0a0 0x5618a25c98e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.678+0000 7fd05124c740 1 --2- 172.21.15.42:0/4099746804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2223f60 0x5618a25c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.678+0000 7fd05124c740 1 -- 172.21.15.42:0/4099746804 >> 172.21.15.42:0/4099746804 conn(0x5618a2218350 msgr2=0x5618a25c7290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.678+0000 7fd05124c740 1 -- 172.21.15.42:0/4099746804 shutdown_connections 2024-07-26T06:50:20.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.678+0000 7fd05124c740 1 -- 172.21.15.42:0/4099746804 wait complete. 2024-07-26T06:50:20.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.678+0000 7fd05124c740 1 -- start start 2024-07-26T06:50:20.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.678+0000 7fd05124c740 1 Processor -- start 2024-07-26T06:50:20.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd05124c740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21fb0a0 0x5618a25d4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd05124c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a2203280 0x5618a25d54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd05124c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2223f60 0x5618a25e0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd05124c740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5618a23925f0 con 0x5618a21fb0a0 2024-07-26T06:50:20.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd05124c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5618a24d18b0 con 0x5618a2223f60 2024-07-26T06:50:20.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd05124c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5618a25d7b20 con 0x5618a2203280 2024-07-26T06:50:20.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd05124c740 10 client.0 _ll_register_callbacks cb 0x5618a24db7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:20.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd048a64640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21fb0a0 0x5618a25d4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd048a64640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21fb0a0 0x5618a25d4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43574/0 (socket says 172.21.15.42:43574) 2024-07-26T06:50:20.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd048a64640 1 -- 172.21.15.42:0/3146579982 learned_addr learned my addr 172.21.15.42:0/3146579982 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:20.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd043fff640 1 --2- 172.21.15.42:0/3146579982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a2203280 0x5618a25d54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd049265640 1 --2- 172.21.15.42:0/3146579982 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2223f60 0x5618a25e0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd048a64640 1 -- 172.21.15.42:0/3146579982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a2203280 msgr2=0x5618a25d54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:20.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd048a64640 1 --2- 172.21.15.42:0/3146579982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a2203280 0x5618a25d54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd048a64640 1 -- 172.21.15.42:0/3146579982 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2223f60 msgr2=0x5618a25e0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:20.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd048a64640 1 --2- 172.21.15.42:0/3146579982 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2223f60 0x5618a25e0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd048a64640 1 -- 172.21.15.42:0/3146579982 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618a228f540 con 0x5618a21fb0a0 2024-07-26T06:50:20.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd049265640 1 --2- 172.21.15.42:0/3146579982 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2223f60 0x5618a25e0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:50:20.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd043fff640 1 --2- 172.21.15.42:0/3146579982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a2203280 0x5618a25d54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:20.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd048a64640 1 --2- 172.21.15.42:0/3146579982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21fb0a0 0x5618a25d4fb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd03400bf90 tx=0x7fd0340128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a641e2aac60b7c83 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:20.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.679+0000 7fd023fff640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:20.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.680+0000 7fd023fff640 1 -- 172.21.15.42:0/3146579982 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03401f060 con 0x5618a21fb0a0 2024-07-26T06:50:20.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.680+0000 7fd05124c740 1 -- 172.21.15.42:0/3146579982 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd03801fe10 con 0x5618a21fb0a0 2024-07-26T06:50:20.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.680+0000 7fd023fff640 1 -- 172.21.15.42:0/3146579982 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fd034010900 con 0x5618a21fb0a0 2024-07-26T06:50:20.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.680+0000 7fd023fff640 1 -- 172.21.15.42:0/3146579982 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd034010b00 con 0x5618a21fb0a0 2024-07-26T06:50:20.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.680+0000 7fd05124c740 10 client.15294 fetch_fsmap learned FSMap version 28 2024-07-26T06:50:20.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.680+0000 7fd05124c740 1 -- 172.21.15.42:0/3146579982 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5618a25e3bf0 con 0x5618a21fb0a0 2024-07-26T06:50:20.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.680+0000 7fd023fff640 1 -- 172.21.15.42:0/3146579982 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd03801fe10 con 0x5618a21fb0a0 2024-07-26T06:50:20.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.680+0000 7fd023fff640 1 -- 172.21.15.42:0/3146579982 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd034019030 con 0x5618a21fb0a0 2024-07-26T06:50:20.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.680+0000 7fd05124c740 10 client.15294 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:50:20.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.680+0000 7fd05124c740 10 client.15294 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T06:50:20.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.680+0000 7fd05124c740 20 client.15294 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:20.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.681+0000 7fd05124c740 1 --2- 172.21.15.42:0/3146579982 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5618a21e28e0 0x5618a25e7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.681+0000 7fd05124c740 4 client.15294 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:50:20.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.681+0000 7fd05124c740 1 -- 172.21.15.42:0/3146579982 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618a2541890 con 0x5618a21e28e0 2024-07-26T06:50:20.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.681+0000 7fd043fff640 1 --2- 172.21.15.42:0/3146579982 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5618a21e28e0 0x5618a25e7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.681+0000 7fd023fff640 10 client.15294 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:50:20.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.681+0000 7fd043fff640 1 --2- 172.21.15.42:0/3146579982 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5618a21e28e0 0x5618a25e7690 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcf118893d42e75e server_cookie=e7020ece542f70a3 in_seq=0 out_seq=0 2024-07-26T06:50:20.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.682+0000 7fd023fff640 1 -- 172.21.15.42:0/3146579982 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5618a2541890 con 0x5618a21e28e0 2024-07-26T06:50:20.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.682+0000 7fd023fff640 10 client.15294 handle_command_reply: tid=0 2024-07-26T06:50:20.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.682+0000 7fd05124c740 1 client.15294 shutdown 2024-07-26T06:50:20.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.682+0000 7fd05124c740 1 -- 172.21.15.42:0/3146579982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21fb0a0 msgr2=0x5618a25d4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.682+0000 7fd05124c740 1 --2- 172.21.15.42:0/3146579982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21fb0a0 0x5618a25d4fb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd03400bf90 tx=0x7fd0340128e0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.682+0000 7fd05124c740 1 -- 172.21.15.42:0/3146579982 shutdown_connections 2024-07-26T06:50:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.682+0000 7fd05124c740 1 --2- 172.21.15.42:0/3146579982 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5618a21e28e0 0x5618a25e7690 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.682+0000 7fd05124c740 1 --2- 172.21.15.42:0/3146579982 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2223f60 0x5618a25e0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.682+0000 7fd05124c740 1 --2- 172.21.15.42:0/3146579982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a2203280 0x5618a25d54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.682+0000 7fd023fff640 0 client.15294 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:50:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.682+0000 7fd05124c740 1 --2- 172.21.15.42:0/3146579982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21fb0a0 0x5618a25d4fb0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.683+0000 7fd05124c740 1 -- 172.21.15.42:0/3146579982 >> 172.21.15.42:0/3146579982 conn(0x5618a2218350 msgr2=0x5618a25cc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.683+0000 7fd05124c740 1 -- 172.21.15.42:0/3146579982 shutdown_connections 2024-07-26T06:50:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.683+0000 7fd05124c740 1 -- 172.21.15.42:0/3146579982 wait complete. 2024-07-26T06:50:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.683+0000 7fd05124c740 20 client.15294 trim_cache size 0 max 16384 2024-07-26T06:50:20.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.701+0000 7fd05124c740 1 Processor -- start 2024-07-26T06:50:20.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.701+0000 7fd05124c740 1 -- start start 2024-07-26T06:50:20.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.701+0000 7fd05124c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a21e28e0 0x5618a25e6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.701+0000 7fd05124c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a21fb0a0 0x5618a25e0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.701+0000 7fd05124c740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a2203280 0x5618a25d4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.701+0000 7fd05124c740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5618a25e59d0 con 0x5618a2203280 2024-07-26T06:50:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.701+0000 7fd05124c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5618a25e60a0 con 0x5618a21e28e0 2024-07-26T06:50:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.701+0000 7fd05124c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5618a25e6770 con 0x5618a21fb0a0 2024-07-26T06:50:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.701+0000 7fd049265640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a2203280 0x5618a25d4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.701+0000 7fd049265640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a2203280 0x5618a25d4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43576/0 (socket says 172.21.15.42:43576) 2024-07-26T06:50:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.701+0000 7fd049265640 1 -- 172.21.15.42:0/1765907164 learned_addr learned my addr 172.21.15.42:0/1765907164 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.701+0000 7fd043fff640 1 --2- 172.21.15.42:0/1765907164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a21fb0a0 0x5618a25e0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.701+0000 7fd049265640 1 -- 172.21.15.42:0/1765907164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a21fb0a0 msgr2=0x5618a25e0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.701+0000 7fd049265640 1 --2- 172.21.15.42:0/1765907164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a21fb0a0 0x5618a25e0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.701+0000 7fd049265640 1 -- 172.21.15.42:0/1765907164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a21e28e0 msgr2=0x5618a25e6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.701+0000 7fd048a64640 1 --2- 172.21.15.42:0/1765907164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a21e28e0 0x5618a25e6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.701+0000 7fd049265640 1 --2- 172.21.15.42:0/1765907164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a21e28e0 0x5618a25e6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.702+0000 7fd049265640 1 -- 172.21.15.42:0/1765907164 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618a25c7b80 con 0x5618a2203280 2024-07-26T06:50:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.702+0000 7fd049265640 1 --2- 172.21.15.42:0/1765907164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a2203280 0x5618a25d4f90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd03c02fd10 tx=0x7fd03c02a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c59d0fc28304f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.702+0000 7fd02a7fc640 1 -- 172.21.15.42:0/1765907164 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c02c040 con 0x5618a2203280 2024-07-26T06:50:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.702+0000 7fd02a7fc640 1 -- 172.21.15.42:0/1765907164 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd03c0cb6d0 con 0x5618a2203280 2024-07-26T06:50:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.702+0000 7fd02a7fc640 1 -- 172.21.15.42:0/1765907164 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c0d4c70 con 0x5618a2203280 2024-07-26T06:50:20.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.703+0000 7fd05124c740 1 -- 172.21.15.42:0/1765907164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a2203280 msgr2=0x5618a25d4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:20.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.703+0000 7fd05124c740 1 --2- 172.21.15.42:0/1765907164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a2203280 0x5618a25d4f90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd03c02fd10 tx=0x7fd03c02a950 comp rx=0 tx=0).stop 2024-07-26T06:50:20.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.703+0000 7fd05124c740 1 -- 172.21.15.42:0/1765907164 shutdown_connections 2024-07-26T06:50:20.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.703+0000 7fd05124c740 1 --2- 172.21.15.42:0/1765907164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a2203280 0x5618a25d4f90 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.703+0000 7fd05124c740 1 --2- 172.21.15.42:0/1765907164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a21fb0a0 0x5618a25e0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.703+0000 7fd05124c740 1 --2- 172.21.15.42:0/1765907164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a21e28e0 0x5618a25e6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.703+0000 7fd05124c740 1 -- 172.21.15.42:0/1765907164 >> 172.21.15.42:0/1765907164 conn(0x5618a2218350 msgr2=0x5618a25d3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:20.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.703+0000 7fd05124c740 1 -- 172.21.15.42:0/1765907164 shutdown_connections 2024-07-26T06:50:20.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.703+0000 7fd05124c740 1 -- 172.21.15.42:0/1765907164 wait complete. 2024-07-26T06:50:20.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.703+0000 7fd05124c740 1 -- start start 2024-07-26T06:50:20.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.703+0000 7fd05124c740 1 Processor -- start 2024-07-26T06:50:20.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd05124c740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21e28e0 0x5618a25e0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd05124c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a21fb0a0 0x5618a25e9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd05124c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2203280 0x5618a25ea490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd05124c740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5618a25e7510 con 0x5618a21e28e0 2024-07-26T06:50:20.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd05124c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5618a25e6e40 con 0x5618a2203280 2024-07-26T06:50:20.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd05124c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5618a25e7be0 con 0x5618a21fb0a0 2024-07-26T06:50:20.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd05124c740 10 client.0 _ll_register_callbacks cb 0x7fd03400bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:20.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd048a64640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21e28e0 0x5618a25e0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd048a64640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21e28e0 0x5618a25e0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43580/0 (socket says 172.21.15.42:43580) 2024-07-26T06:50:20.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd048a64640 1 -- 172.21.15.42:0/1786485479 learned_addr learned my addr 172.21.15.42:0/1786485479 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:20.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd049265640 1 --2- 172.21.15.42:0/1786485479 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2203280 0x5618a25ea490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd043fff640 1 --2- 172.21.15.42:0/1786485479 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a21fb0a0 0x5618a25e9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd048a64640 1 -- 172.21.15.42:0/1786485479 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a21fb0a0 msgr2=0x5618a25e9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:20.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd048a64640 1 --2- 172.21.15.42:0/1786485479 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a21fb0a0 0x5618a25e9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd048a64640 1 -- 172.21.15.42:0/1786485479 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2203280 msgr2=0x5618a25ea490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:20.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd048a64640 1 --2- 172.21.15.42:0/1786485479 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2203280 0x5618a25ea490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd048a64640 1 -- 172.21.15.42:0/1786485479 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618a25eed70 con 0x5618a21e28e0 2024-07-26T06:50:20.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd049265640 1 --2- 172.21.15.42:0/1786485479 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2203280 0x5618a25ea490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:20.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd048a64640 1 --2- 172.21.15.42:0/1786485479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21e28e0 0x5618a25e0e30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd0340109b0 tx=0x7fd0340109e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=890002289b1db147 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:20.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.704+0000 7fd023fff640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:20.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.705+0000 7fd023fff640 1 -- 172.21.15.42:0/1786485479 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd034005510 con 0x5618a21e28e0 2024-07-26T06:50:20.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.705+0000 7fd023fff640 1 -- 172.21.15.42:0/1786485479 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fd034004680 con 0x5618a21e28e0 2024-07-26T06:50:20.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.705+0000 7fd05124c740 1 -- 172.21.15.42:0/1786485479 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd03c0cb6d0 con 0x5618a21e28e0 2024-07-26T06:50:20.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.705+0000 7fd023fff640 1 -- 172.21.15.42:0/1786485479 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd03401f030 con 0x5618a21e28e0 2024-07-26T06:50:20.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.705+0000 7fd05124c740 10 client.15300 fetch_fsmap learned FSMap version 28 2024-07-26T06:50:20.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.705+0000 7fd05124c740 1 -- 172.21.15.42:0/1786485479 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5618a25efff0 con 0x5618a21e28e0 2024-07-26T06:50:20.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.705+0000 7fd023fff640 1 -- 172.21.15.42:0/1786485479 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd03c0cb6d0 con 0x5618a21e28e0 2024-07-26T06:50:20.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.705+0000 7fd023fff640 1 -- 172.21.15.42:0/1786485479 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd034019040 con 0x5618a21e28e0 2024-07-26T06:50:20.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.705+0000 7fd05124c740 10 client.15300 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T06:50:20.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.705+0000 7fd05124c740 10 client.15300 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T06:50:20.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.705+0000 7fd05124c740 20 client.15300 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:20.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.705+0000 7fd05124c740 1 --2- 172.21.15.42:0/1786485479 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5618a2223f60 0x5618a2653b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:20.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.705+0000 7fd05124c740 4 client.15300 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:50:20.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.705+0000 7fd05124c740 1 -- 172.21.15.42:0/1786485479 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "status"}) -- 0x5618a253d330 con 0x5618a2223f60 2024-07-26T06:50:20.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.706+0000 7fd043fff640 1 --2- 172.21.15.42:0/1786485479 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5618a2223f60 0x5618a2653b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:20.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.706+0000 7fd043fff640 1 --2- 172.21.15.42:0/1786485479 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5618a2223f60 0x5618a2653b00 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9acec3b08e23be2 server_cookie=ca72334475423f73 in_seq=0 out_seq=0 2024-07-26T06:50:20.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.706+0000 7fd023fff640 10 client.15300 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:50:20.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.706+0000 7fd023fff640 1 -- 172.21.15.42:0/1786485479 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5618a253d330 con 0x5618a2223f60 2024-07-26T06:50:20.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.706+0000 7fd023fff640 10 client.15300 handle_command_reply: tid=0 2024-07-26T06:50:20.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.706+0000 7fd05124c740 1 client.15300 shutdown 2024-07-26T06:50:20.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.706+0000 7fd05124c740 1 -- 172.21.15.42:0/1786485479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21e28e0 msgr2=0x5618a25e0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:20.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.706+0000 7fd05124c740 1 --2- 172.21.15.42:0/1786485479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21e28e0 0x5618a25e0e30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd0340109b0 tx=0x7fd0340109e0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.706+0000 7fd05124c740 1 -- 172.21.15.42:0/1786485479 shutdown_connections 2024-07-26T06:50:20.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.706+0000 7fd05124c740 1 --2- 172.21.15.42:0/1786485479 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5618a2223f60 0x5618a2653b00 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.706+0000 7fd05124c740 1 --2- 172.21.15.42:0/1786485479 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5618a2203280 0x5618a25ea490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.706+0000 7fd05124c740 1 --2- 172.21.15.42:0/1786485479 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5618a21fb0a0 0x5618a25e9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.706+0000 7fd05124c740 1 --2- 172.21.15.42:0/1786485479 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5618a21e28e0 0x5618a25e0e30 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.706+0000 7fd05124c740 1 -- 172.21.15.42:0/1786485479 >> 172.21.15.42:0/1786485479 conn(0x5618a2218350 msgr2=0x5618a25d3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:20.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.706+0000 7fd023fff640 0 client.15300 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:50:20.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.707+0000 7fd05124c740 1 -- 172.21.15.42:0/1786485479 shutdown_connections 2024-07-26T06:50:20.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.707+0000 7fd05124c740 1 -- 172.21.15.42:0/1786485479 wait complete. 2024-07-26T06:50:20.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.707+0000 7fd05124c740 20 client.15300 trim_cache size 0 max 16384 2024-07-26T06:50:20.714 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:50:20.714 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_fsid": "8b3e2750-4b19-11ef-bcb6-c7b262605968", 2024-07-26T06:50:20.714 INFO:teuthology.orchestra.run.smithi042.stdout: "whoami": 0, 2024-07-26T06:50:20.714 INFO:teuthology.orchestra.run.smithi042.stdout: "id": 24620, 2024-07-26T06:50:20.714 INFO:teuthology.orchestra.run.smithi042.stdout: "want_state": "up:active", 2024-07-26T06:50:20.714 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "up:active", 2024-07-26T06:50:20.714 INFO:teuthology.orchestra.run.smithi042.stdout: "fs_name": "cephfs", 2024-07-26T06:50:20.715 INFO:teuthology.orchestra.run.smithi042.stdout: "rank_uptime": 193.692143902, 2024-07-26T06:50:20.715 INFO:teuthology.orchestra.run.smithi042.stdout: "mdsmap_epoch": 28, 2024-07-26T06:50:20.715 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch": 80, 2024-07-26T06:50:20.715 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch_barrier": 80, 2024-07-26T06:50:20.715 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 210.24977436899999 2024-07-26T06:50:20.715 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:50:20.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.707+0000 7fd04acc8640 1 -- 172.21.15.42:0/827160693 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd02c0b3bd0 msgr2=0x7fd02c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:20.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.707+0000 7fd04acc8640 1 --2- 172.21.15.42:0/827160693 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd02c0b3bd0 0x7fd02c0b60e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd038013510 tx=0x7fd038013910 comp rx=0 tx=0).stop 2024-07-26T06:50:20.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.707+0000 7fd04acc8640 1 -- 172.21.15.42:0/827160693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd04410fc30 msgr2=0x7fd0441a1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:20.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.707+0000 7fd04acc8640 1 --2- 172.21.15.42:0/827160693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd04410fc30 0x7fd0441a1960 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd03c02ae70 tx=0x7fd03c02f5d0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.707+0000 7fd04acc8640 1 -- 172.21.15.42:0/827160693 shutdown_connections 2024-07-26T06:50:20.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.707+0000 7fd04acc8640 1 --2- 172.21.15.42:0/827160693 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd02c0b3bd0 0x7fd02c0b60e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.707+0000 7fd04acc8640 1 --2- 172.21.15.42:0/827160693 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd04410fc30 0x7fd0441a1960 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.707+0000 7fd04acc8640 1 --2- 172.21.15.42:0/827160693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd0441029d0 0x7fd04419c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.707+0000 7fd04acc8640 1 --2- 172.21.15.42:0/827160693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd044102080 0x7fd04419c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:20.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.707+0000 7fd04acc8640 1 -- 172.21.15.42:0/827160693 >> 172.21.15.42:0/827160693 conn(0x7fd0440791a0 msgr2=0x7fd0441097b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:20.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.708+0000 7fd04acc8640 1 -- 172.21.15.42:0/827160693 shutdown_connections 2024-07-26T06:50:20.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:20.708+0000 7fd04acc8640 1 -- 172.21.15.42:0/827160693 wait complete. 2024-07-26T06:50:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:20 smithi163 ceph-mon[36873]: pgmap v399: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:50:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:20 smithi164 ceph-mon[36970]: pgmap v399: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T06:50:21.394 INFO:teuthology.run_tasks:Running task sequential... 2024-07-26T06:50:21.404 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-07-26T06:50:21.404 INFO:teuthology.task.sequential:In sequential, running task print... 2024-07-26T06:50:21.405 INFO:teuthology.task.print:Enabling mgr modules 2024-07-26T06:50:21.405 INFO:teuthology.run_tasks:Running task kclient... 2024-07-26T06:50:21.482 INFO:tasks.kclient:Mounting kernel clients... 2024-07-26T06:50:21.482 INFO:tasks.kclient:config is {'client.0': {}} 2024-07-26T06:50:21.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:50:21.483 DEBUG:teuthology.orchestra.run.smithi042:> ip netns list 2024-07-26T06:50:21.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:50:21.503 DEBUG:teuthology.orchestra.run.smithi042:> sudo ip link delete ceph-brx 2024-07-26T06:50:21.574 INFO:teuthology.orchestra.run.smithi042.stderr:Cannot find device "ceph-brx" 2024-07-26T06:50:21.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T06:50:21.576 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-07-26T06:50:21.576 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-07-26T06:50:21.576 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T06:50:21.576 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs ls 2024-07-26T06:50:21.882 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:22.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:22 smithi042 ceph-mon[32775]: pgmap v400: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:50:22.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:22 smithi163 ceph-mon[36873]: pgmap v400: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:50:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:22 smithi164 ceph-mon[36970]: pgmap v400: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:50:23.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.583+0000 7fdd2fcb2640 1 -- 172.21.15.42:0/865055396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd28108ed0 msgr2=0x7fdd281092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:23.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.583+0000 7fdd2fcb2640 1 --2- 172.21.15.42:0/865055396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd28108ed0 0x7fdd281092e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009900 tx=0x7fdd1802a410 comp rx=0 tx=0).stop 2024-07-26T06:50:23.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.583+0000 7fdd2fcb2640 1 -- 172.21.15.42:0/865055396 shutdown_connections 2024-07-26T06:50:23.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.583+0000 7fdd2fcb2640 1 --2- 172.21.15.42:0/865055396 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd28109820 0x7fdd28110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:23.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.583+0000 7fdd2fcb2640 1 --2- 172.21.15.42:0/865055396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd28108ed0 0x7fdd281092e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:23.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.583+0000 7fdd2fcb2640 1 --2- 172.21.15.42:0/865055396 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd28103590 0x7fdd281039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:23.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.583+0000 7fdd2fcb2640 1 -- 172.21.15.42:0/865055396 >> 172.21.15.42:0/865055396 conn(0x7fdd2807b280 msgr2=0x7fdd2807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:23.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.583+0000 7fdd2fcb2640 1 -- 172.21.15.42:0/865055396 shutdown_connections 2024-07-26T06:50:23.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.583+0000 7fdd2fcb2640 1 -- 172.21.15.42:0/865055396 wait complete. 2024-07-26T06:50:23.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2fcb2640 1 Processor -- start 2024-07-26T06:50:23.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2fcb2640 1 -- start start 2024-07-26T06:50:23.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2fcb2640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd28103590 0x7fdd28075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:23.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2fcb2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd28108ed0 0x7fdd28075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:23.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2fcb2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd28109820 0x7fdd28075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:23.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2fcb2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdd28113500 con 0x7fdd28108ed0 2024-07-26T06:50:23.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2fcb2640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdd28113380 con 0x7fdd28103590 2024-07-26T06:50:23.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2fcb2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdd28113680 con 0x7fdd28109820 2024-07-26T06:50:23.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2d24d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd28108ed0 0x7fdd28075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:23.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2d24d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd28108ed0 0x7fdd28075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43598/0 (socket says 172.21.15.42:43598) 2024-07-26T06:50:23.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2d24d640 1 -- 172.21.15.42:0/3596971659 learned_addr learned my addr 172.21.15.42:0/3596971659 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:23.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2da4e640 1 --2- 172.21.15.42:0/3596971659 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd28103590 0x7fdd28075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:23.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2e24f640 1 --2- 172.21.15.42:0/3596971659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd28109820 0x7fdd28075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:23.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2d24d640 1 -- 172.21.15.42:0/3596971659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd28109820 msgr2=0x7fdd28075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:23.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2d24d640 1 --2- 172.21.15.42:0/3596971659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd28109820 0x7fdd28075bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:23.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2d24d640 1 -- 172.21.15.42:0/3596971659 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd28103590 msgr2=0x7fdd28075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:23.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2d24d640 1 --2- 172.21.15.42:0/3596971659 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd28103590 0x7fdd28075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:23.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2d24d640 1 -- 172.21.15.42:0/3596971659 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd281a81b0 con 0x7fdd28108ed0 2024-07-26T06:50:23.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.584+0000 7fdd2da4e640 1 --2- 172.21.15.42:0/3596971659 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd28103590 0x7fdd28075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:50:23.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.585+0000 7fdd2d24d640 1 --2- 172.21.15.42:0/3596971659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd28108ed0 0x7fdd28075680 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fdd18005ab0 tx=0x7fdd18003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:23.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.585+0000 7fdd1effd640 1 -- 172.21.15.42:0/3596971659 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1803f070 con 0x7fdd28108ed0 2024-07-26T06:50:23.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.585+0000 7fdd2fcb2640 1 -- 172.21.15.42:0/3596971659 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd281a8440 con 0x7fdd28108ed0 2024-07-26T06:50:23.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.585+0000 7fdd1effd640 1 -- 172.21.15.42:0/3596971659 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdd18031d60 con 0x7fdd28108ed0 2024-07-26T06:50:23.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.585+0000 7fdd1effd640 1 -- 172.21.15.42:0/3596971659 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18035030 con 0x7fdd28108ed0 2024-07-26T06:50:23.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.585+0000 7fdd2fcb2640 1 -- 172.21.15.42:0/3596971659 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd281a8970 con 0x7fdd28108ed0 2024-07-26T06:50:23.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.586+0000 7fdd2fcb2640 1 -- 172.21.15.42:0/3596971659 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd281039a0 con 0x7fdd28108ed0 2024-07-26T06:50:23.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.588+0000 7fdd1effd640 1 -- 172.21.15.42:0/3596971659 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fdd18042070 con 0x7fdd28108ed0 2024-07-26T06:50:23.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.589+0000 7fdd1effd640 1 --2- 172.21.15.42:0/3596971659 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdd140b3bd0 0x7fdd140b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:23.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.589+0000 7fdd2da4e640 1 --2- 172.21.15.42:0/3596971659 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdd140b3bd0 0x7fdd140b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:23.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.589+0000 7fdd1effd640 1 -- 172.21.15.42:0/3596971659 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fdd181047c0 con 0x7fdd28108ed0 2024-07-26T06:50:23.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.589+0000 7fdd2da4e640 1 --2- 172.21.15.42:0/3596971659 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdd140b3bd0 0x7fdd140b60e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdd28076b90 tx=0x7fdd24005e70 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:23.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.592+0000 7fdd1effd640 1 -- 172.21.15.42:0/3596971659 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdd180cbd90 con 0x7fdd28108ed0 2024-07-26T06:50:23.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.814+0000 7fdd2fcb2640 1 -- 172.21.15.42:0/3596971659 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fdd2810b660 con 0x7fdd28108ed0 2024-07-26T06:50:23.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.814+0000 7fdd1effd640 1 -- 172.21.15.42:0/3596971659 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+73 (secure 0 0 0) 0x7fdd2810b660 con 0x7fdd28108ed0 2024-07-26T06:50:23.815 INFO:teuthology.orchestra.run.smithi042.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-07-26T06:50:23.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.816+0000 7fdd2fcb2640 1 -- 172.21.15.42:0/3596971659 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdd140b3bd0 msgr2=0x7fdd140b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:23.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.816+0000 7fdd2fcb2640 1 --2- 172.21.15.42:0/3596971659 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdd140b3bd0 0x7fdd140b60e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdd28076b90 tx=0x7fdd24005e70 comp rx=0 tx=0).stop 2024-07-26T06:50:23.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.816+0000 7fdd2fcb2640 1 -- 172.21.15.42:0/3596971659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd28108ed0 msgr2=0x7fdd28075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:23.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.816+0000 7fdd2fcb2640 1 --2- 172.21.15.42:0/3596971659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd28108ed0 0x7fdd28075680 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fdd18005ab0 tx=0x7fdd18003cd0 comp rx=0 tx=0).stop 2024-07-26T06:50:23.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.816+0000 7fdd2fcb2640 1 -- 172.21.15.42:0/3596971659 shutdown_connections 2024-07-26T06:50:23.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.816+0000 7fdd2fcb2640 1 --2- 172.21.15.42:0/3596971659 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fdd140b3bd0 0x7fdd140b60e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:23.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.816+0000 7fdd2fcb2640 1 --2- 172.21.15.42:0/3596971659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd28109820 0x7fdd28075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:23.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.816+0000 7fdd2fcb2640 1 --2- 172.21.15.42:0/3596971659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdd28108ed0 0x7fdd28075680 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:23.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.816+0000 7fdd2fcb2640 1 --2- 172.21.15.42:0/3596971659 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd28103590 0x7fdd28075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:23.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.816+0000 7fdd2fcb2640 1 -- 172.21.15.42:0/3596971659 >> 172.21.15.42:0/3596971659 conn(0x7fdd2807b280 msgr2=0x7fdd28106f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:23.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.816+0000 7fdd2fcb2640 1 -- 172.21.15.42:0/3596971659 shutdown_connections 2024-07-26T06:50:23.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:23.817+0000 7fdd2fcb2640 1 -- 172.21.15.42:0/3596971659 wait complete. 2024-07-26T06:50:24.517 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-07-26T06:50:24.518 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T06:50:24.518 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi042.front.sepia.ceph.com 2024-07-26T06:50:24.518 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-07-26T06:50:24.518 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T06:50:24.518 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-07-26T06:50:24.518 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-07-26T06:50:24.518 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T06:50:24.518 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:50:24.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:50:24.522 DEBUG:teuthology.orchestra.run.smithi042:> ip addr 2024-07-26T06:50:24.541 INFO:teuthology.orchestra.run.smithi042.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T06:50:24.541 INFO:teuthology.orchestra.run.smithi042.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T06:50:24.541 INFO:teuthology.orchestra.run.smithi042.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T06:50:24.541 INFO:teuthology.orchestra.run.smithi042.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:50:24.541 INFO:teuthology.orchestra.run.smithi042.stdout: inet6 ::1/128 scope host 2024-07-26T06:50:24.541 INFO:teuthology.orchestra.run.smithi042.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:50:24.541 INFO:teuthology.orchestra.run.smithi042.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:50:24.541 INFO:teuthology.orchestra.run.smithi042.stdout: link/ether 0c:c4:7a:6b:fd:0e brd ff:ff:ff:ff:ff:ff 2024-07-26T06:50:24.542 INFO:teuthology.orchestra.run.smithi042.stdout: altname enp7s0f0 2024-07-26T06:50:24.542 INFO:teuthology.orchestra.run.smithi042.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:50:24.542 INFO:teuthology.orchestra.run.smithi042.stdout: link/ether 0c:c4:7a:6b:fd:0f brd ff:ff:ff:ff:ff:ff 2024-07-26T06:50:24.542 INFO:teuthology.orchestra.run.smithi042.stdout: altname enp7s0f1 2024-07-26T06:50:24.542 INFO:teuthology.orchestra.run.smithi042.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T06:50:24.542 INFO:teuthology.orchestra.run.smithi042.stdout: link/ether 0c:c4:7a:bd:14:40 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:50:24.542 INFO:teuthology.orchestra.run.smithi042.stdout: altname enp1s0f0 2024-07-26T06:50:24.542 INFO:teuthology.orchestra.run.smithi042.stdout: inet 172.21.15.42/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-07-26T06:50:24.542 INFO:teuthology.orchestra.run.smithi042.stdout: valid_lft 41977sec preferred_lft 41977sec 2024-07-26T06:50:24.542 INFO:teuthology.orchestra.run.smithi042.stdout: inet6 fe80::ec4:7aff:febd:1440/64 scope link noprefixroute 2024-07-26T06:50:24.542 INFO:teuthology.orchestra.run.smithi042.stdout: valid_lft forever preferred_lft forever 2024-07-26T06:50:24.542 INFO:teuthology.orchestra.run.smithi042.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T06:50:24.543 INFO:teuthology.orchestra.run.smithi042.stdout: link/ether 0c:c4:7a:bd:14:41 brd ff:ff:ff:ff:ff:ff 2024-07-26T06:50:24.543 INFO:teuthology.orchestra.run.smithi042.stdout: altname enp1s0f1 2024-07-26T06:50:24.543 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-07-26T06:50:24.543 DEBUG:teuthology.orchestra.run.smithi042:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:50:24.543 DEBUG:teuthology.orchestra.run.smithi042:> set -e 2024-07-26T06:50:24.543 DEBUG:teuthology.orchestra.run.smithi042:> sudo ip link add name ceph-brx type bridge 2024-07-26T06:50:24.543 DEBUG:teuthology.orchestra.run.smithi042:> sudo ip addr flush dev ceph-brx 2024-07-26T06:50:24.544 DEBUG:teuthology.orchestra.run.smithi042:> sudo ip link set ceph-brx up 2024-07-26T06:50:24.544 DEBUG:teuthology.orchestra.run.smithi042:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-07-26T06:50:24.544 DEBUG:teuthology.orchestra.run.smithi042:> ') 2024-07-26T06:50:24.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:50:24.742 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:24 smithi042 ceph-mon[32775]: pgmap v401: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:50:24.743 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:24 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/3596971659' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T06:50:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:24 smithi163 ceph-mon[36873]: pgmap v401: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:50:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:24 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/3596971659' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T06:50:24.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:50:24.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:50:24.763 DEBUG:teuthology.orchestra.run.smithi042:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T06:50:24.789 INFO:teuthology.orchestra.run.smithi042.stdout:1 2024-07-26T06:50:24.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:50:24.791 DEBUG:teuthology.orchestra.run.smithi042:> route 2024-07-26T06:50:24.860 INFO:teuthology.orchestra.run.smithi042.stdout:Kernel IP routing table 2024-07-26T06:50:24.860 INFO:teuthology.orchestra.run.smithi042.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T06:50:24.860 INFO:teuthology.orchestra.run.smithi042.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-07-26T06:50:24.860 INFO:teuthology.orchestra.run.smithi042.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-07-26T06:50:24.861 INFO:teuthology.orchestra.run.smithi042.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T06:50:24.861 DEBUG:teuthology.orchestra.run.smithi042:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:50:24.861 DEBUG:teuthology.orchestra.run.smithi042:> set -e 2024-07-26T06:50:24.861 DEBUG:teuthology.orchestra.run.smithi042:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-07-26T06:50:24.861 DEBUG:teuthology.orchestra.run.smithi042:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-07-26T06:50:24.861 DEBUG:teuthology.orchestra.run.smithi042:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-07-26T06:50:24.861 DEBUG:teuthology.orchestra.run.smithi042:> ') 2024-07-26T06:50:24.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:50:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:24 smithi164 ceph-mon[36970]: pgmap v401: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:50:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:24 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/3596971659' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T06:50:25.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:50:25.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:50:25.218 DEBUG:teuthology.orchestra.run.smithi042:> ip netns list 2024-07-26T06:50:25.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:50:25.235 DEBUG:teuthology.orchestra.run.smithi042:> ip netns list-id 2024-07-26T06:50:25.292 DEBUG:teuthology.orchestra.run.smithi042:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:50:25.292 DEBUG:teuthology.orchestra.run.smithi042:> set -e 2024-07-26T06:50:25.292 DEBUG:teuthology.orchestra.run.smithi042:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T06:50:25.292 DEBUG:teuthology.orchestra.run.smithi042:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-07-26T06:50:25.292 DEBUG:teuthology.orchestra.run.smithi042:> ') 2024-07-26T06:50:25.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:50:25.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:50:25.413 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-07-26T06:50:25.413 DEBUG:teuthology.orchestra.run.smithi042:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:50:25.413 DEBUG:teuthology.orchestra.run.smithi042:> set -e 2024-07-26T06:50:25.413 DEBUG:teuthology.orchestra.run.smithi042:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-07-26T06:50:25.413 DEBUG:teuthology.orchestra.run.smithi042:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-07-26T06:50:25.413 DEBUG:teuthology.orchestra.run.smithi042:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-07-26T06:50:25.414 DEBUG:teuthology.orchestra.run.smithi042:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-07-26T06:50:25.414 DEBUG:teuthology.orchestra.run.smithi042:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-07-26T06:50:25.414 DEBUG:teuthology.orchestra.run.smithi042:> ') 2024-07-26T06:50:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:50:25.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:50:25.547 DEBUG:teuthology.orchestra.run.smithi042:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T06:50:25.547 DEBUG:teuthology.orchestra.run.smithi042:> set -e 2024-07-26T06:50:25.547 DEBUG:teuthology.orchestra.run.smithi042:> sudo ip link set brx.0 up 2024-07-26T06:50:25.547 DEBUG:teuthology.orchestra.run.smithi042:> sudo ip link set dev brx.0 master ceph-brx 2024-07-26T06:50:25.548 DEBUG:teuthology.orchestra.run.smithi042:> ') 2024-07-26T06:50:25.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:50:25.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:50:25.629 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:50:25.629 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-07-26T06:50:25.644 INFO:teuthology.orchestra.run.smithi042.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-07-26T06:50:25.645 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T06:50:25.645 DEBUG:teuthology.orchestra.run.smithi042:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-07-26T06:50:25.701 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-07-26T06:50:25.701 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-07-26T06:50:25.701 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/dcf68d2d-a95d-4cd5-9339-221de9d6bf68 2024-07-26T06:50:25.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:50:25.702 DEBUG:teuthology.orchestra.run.smithi042:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/dcf68d2d-a95d-4cd5-9339-221de9d6bf68 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-07-26T06:50:25.810 INFO:teuthology.orchestra.run.smithi042.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-07-26T06:50:25.811 INFO:teuthology.orchestra.run.smithi042.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-07-26T06:50:25.811 INFO:teuthology.orchestra.run.smithi042.stdout:invalid new device string format 2024-07-26T06:50:25.945 INFO:teuthology.orchestra.run.smithi042.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-07-26T06:50:25.945 INFO:teuthology.orchestra.run.smithi042.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-07-26T06:50:25.945 INFO:teuthology.orchestra.run.smithi042.stdout:invalid new device string format 2024-07-26T06:50:25.945 INFO:teuthology.orchestra.run.smithi042.stdout:mount.ceph: resolved to: "172.21.15.42:6789,172.21.15.163:6789,172.21.15.164:6789" 2024-07-26T06:50:25.945 INFO:teuthology.orchestra.run.smithi042.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.42:6789,172.21.15.163:6789,172.21.15.164:6789:/volumes/qa/sv_1/dcf68d2d-a95d-4cd5-9339-221de9d6bf68 2024-07-26T06:50:25.945 INFO:teuthology.orchestra.run.smithi042.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,key=0,fsid=8b3e2750-4b19-11ef-bcb6-c7b262605968" will pass to kernel 2024-07-26T06:50:25.946 INFO:teuthology.orchestra.run.smithi042.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-07-26T06:50:25.946 INFO:teuthology.orchestra.run.smithi042.stdout: You just mounted a file system that supports labels which does not 2024-07-26T06:50:25.946 INFO:teuthology.orchestra.run.smithi042.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T06:50:25.947 INFO:teuthology.orchestra.run.smithi042.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T06:50:25.947 INFO:teuthology.orchestra.run.smithi042.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T06:50:25.948 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T06:50:25.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T06:50:25.948 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-07-26T06:50:25.980 DEBUG:teuthology.orchestra.run.smithi042:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:50:26.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:50:26.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:50:26.080 INFO:teuthology.orchestra.run.smithi042.stdout:client25018 2024-07-26T06:50:26.080 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-07-26T06:50:26.081 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T06:50:26.098 DEBUG:teuthology.orchestra.run.smithi042:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:50:26.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:50:26.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:50:26.196 INFO:teuthology.orchestra.run.smithi042.stdout:client25018 2024-07-26T06:50:26.197 DEBUG:teuthology.orchestra.run.smithi042:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T06:50:26.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T06:50:26.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:50:26.255 INFO:teuthology.orchestra.run.smithi042.stdout:client25018 2024-07-26T06:50:26.255 DEBUG:teuthology.orchestra.run.smithi042:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968.client25018/status') 2024-07-26T06:50:26.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T06:50:26.317 INFO:teuthology.orchestra.run.smithi042.stdout:instance: client.25018 (3)172.21.15.42:0/110110209 2024-07-26T06:50:26.317 INFO:teuthology.orchestra.run.smithi042.stdout:blocklisted: false 2024-07-26T06:50:26.317 INFO:teuthology.orchestra.run.smithi042.stderr:0+1 records in 2024-07-26T06:50:26.317 INFO:teuthology.orchestra.run.smithi042.stderr:0+1 records out 2024-07-26T06:50:26.317 INFO:teuthology.orchestra.run.smithi042.stderr:70 bytes copied, 2.4997e-05 s, 2.8 MB/s 2024-07-26T06:50:26.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T06:50:26.324 INFO:teuthology.run_tasks:Running task exec... 2024-07-26T06:50:26.335 INFO:teuthology.task.exec:Executing custom commands... 2024-07-26T06:50:26.335 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi042.front.sepia.ceph.com 2024-07-26T06:50:26.335 DEBUG:teuthology.orchestra.run.smithi042:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-07-26T06:50:26.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.451+0000 7f210270e640 1 -- 172.21.15.42:0/1442875764 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20fc108f70 msgr2=0x7f20fc10b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:26.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.451+0000 7f210270e640 1 --2- 172.21.15.42:0/1442875764 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20fc108f70 0x7f20fc10b390 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009ca0 tx=0x7f20f402a8c0 comp rx=0 tx=0).stop 2024-07-26T06:50:26.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.451+0000 7f210270e640 1 -- 172.21.15.42:0/1442875764 shutdown_connections 2024-07-26T06:50:26.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.451+0000 7f210270e640 1 --2- 172.21.15.42:0/1442875764 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20fc108f70 0x7f20fc10b390 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:26.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.451+0000 7f210270e640 1 --2- 172.21.15.42:0/1442875764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20fc106610 0x7f20fc108a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:26.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.451+0000 7f210270e640 1 --2- 172.21.15.42:0/1442875764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20fc1014c0 0x7f20fc1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:26.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.451+0000 7f210270e640 1 -- 172.21.15.42:0/1442875764 >> 172.21.15.42:0/1442875764 conn(0x7f20fc0fd100 msgr2=0x7f20fc0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:26.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.451+0000 7f210270e640 1 -- 172.21.15.42:0/1442875764 shutdown_connections 2024-07-26T06:50:26.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.451+0000 7f210270e640 1 -- 172.21.15.42:0/1442875764 wait complete. 2024-07-26T06:50:26.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.452+0000 7f210270e640 1 Processor -- start 2024-07-26T06:50:26.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.452+0000 7f210270e640 1 -- start start 2024-07-26T06:50:26.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.452+0000 7f210270e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20fc1014c0 0x7f20fc1998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:26.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.452+0000 7f210270e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20fc106610 0x7f20fc199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:26.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.452+0000 7f210270e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20fc108f70 0x7f20fc19aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:26.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.452+0000 7f210270e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc10ed50 con 0x7f20fc108f70 2024-07-26T06:50:26.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.452+0000 7f210270e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc10ebd0 con 0x7f20fc106610 2024-07-26T06:50:26.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.452+0000 7f210270e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f20fc10eed0 con 0x7f20fc1014c0 2024-07-26T06:50:26.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.452+0000 7f2100cab640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20fc108f70 0x7f20fc19aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.452+0000 7f2100cab640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20fc108f70 0x7f20fc19aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43612/0 (socket says 172.21.15.42:43612) 2024-07-26T06:50:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.452+0000 7f2100cab640 1 -- 172.21.15.42:0/452141726 learned_addr learned my addr 172.21.15.42:0/452141726 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.452+0000 7f20fb7fe640 1 --2- 172.21.15.42:0/452141726 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20fc106610 0x7f20fc199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.453+0000 7f20fbfff640 1 --2- 172.21.15.42:0/452141726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20fc1014c0 0x7f20fc1998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.453+0000 7f2100cab640 1 -- 172.21.15.42:0/452141726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20fc1014c0 msgr2=0x7f20fc1998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.453+0000 7f2100cab640 1 --2- 172.21.15.42:0/452141726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20fc1014c0 0x7f20fc1998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.453+0000 7f2100cab640 1 -- 172.21.15.42:0/452141726 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20fc106610 msgr2=0x7f20fc199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.453+0000 7f2100cab640 1 --2- 172.21.15.42:0/452141726 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20fc106610 0x7f20fc199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.453+0000 7f2100cab640 1 -- 172.21.15.42:0/452141726 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20fc19b310 con 0x7f20fc108f70 2024-07-26T06:50:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.453+0000 7f2100cab640 1 --2- 172.21.15.42:0/452141726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20fc108f70 0x7f20fc19aaf0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f20f40097b0 tx=0x7f20f4003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.453+0000 7f20f97fa640 1 -- 172.21.15.42:0/452141726 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f4004240 con 0x7f20fc108f70 2024-07-26T06:50:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.453+0000 7f20f97fa640 1 -- 172.21.15.42:0/452141726 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20f40317c0 con 0x7f20fc108f70 2024-07-26T06:50:26.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.453+0000 7f210270e640 1 -- 172.21.15.42:0/452141726 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20fc19b5a0 con 0x7f20fc108f70 2024-07-26T06:50:26.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.453+0000 7f20f97fa640 1 -- 172.21.15.42:0/452141726 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f402fde0 con 0x7f20fc108f70 2024-07-26T06:50:26.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.453+0000 7f210270e640 1 -- 172.21.15.42:0/452141726 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20fc1a1cf0 con 0x7f20fc108f70 2024-07-26T06:50:26.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.454+0000 7f210270e640 1 -- 172.21.15.42:0/452141726 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20fc0fec20 con 0x7f20fc108f70 2024-07-26T06:50:26.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.456+0000 7f20f97fa640 1 -- 172.21.15.42:0/452141726 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f20f40043e0 con 0x7f20fc108f70 2024-07-26T06:50:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.456+0000 7f20f97fa640 1 --2- 172.21.15.42:0/452141726 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f20e40b3c50 0x7f20e40b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.457+0000 7f20fbfff640 1 --2- 172.21.15.42:0/452141726 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f20e40b3c50 0x7f20e40b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.457+0000 7f20f97fa640 1 -- 172.21.15.42:0/452141726 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f20f4105880 con 0x7f20fc108f70 2024-07-26T06:50:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.457+0000 7f20fbfff640 1 --2- 172.21.15.42:0/452141726 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f20e40b3c50 0x7f20e40b6160 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0058c0 tx=0x7f20ec00a4c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:26.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.459+0000 7f20f97fa640 1 -- 172.21.15.42:0/452141726 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f20f40cceb0 con 0x7f20fc108f70 2024-07-26T06:50:26.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:26 smithi042 ceph-mon[32775]: pgmap v402: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:50:26.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:26.695+0000 7f210270e640 1 -- 172.21.15.42:0/452141726 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f20fc02e0c0 con 0x7f20fc108f70 2024-07-26T06:50:26.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:26 smithi163 ceph-mon[36873]: pgmap v402: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:50:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:26 smithi164 ceph-mon[36970]: pgmap v402: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T06:50:27.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:27.498+0000 7f20f97fa640 1 -- 172.21.15.42:0/452141726 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v29) ==== 117+0+0 (secure 0 0 0) 0x7f20f40cc7b0 con 0x7f20fc108f70 2024-07-26T06:50:27.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:27.501+0000 7f210270e640 1 -- 172.21.15.42:0/452141726 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f20e40b3c50 msgr2=0x7f20e40b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:27.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:27.501+0000 7f210270e640 1 --2- 172.21.15.42:0/452141726 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f20e40b3c50 0x7f20e40b6160 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0058c0 tx=0x7f20ec00a4c0 comp rx=0 tx=0).stop 2024-07-26T06:50:27.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:27.502+0000 7f210270e640 1 -- 172.21.15.42:0/452141726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20fc108f70 msgr2=0x7f20fc19aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:27.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:27.502+0000 7f210270e640 1 --2- 172.21.15.42:0/452141726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20fc108f70 0x7f20fc19aaf0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f20f40097b0 tx=0x7f20f4003c20 comp rx=0 tx=0).stop 2024-07-26T06:50:27.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:27.502+0000 7f210270e640 1 -- 172.21.15.42:0/452141726 shutdown_connections 2024-07-26T06:50:27.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:27.502+0000 7f210270e640 1 --2- 172.21.15.42:0/452141726 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f20e40b3c50 0x7f20e40b6160 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:27.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:27.502+0000 7f210270e640 1 --2- 172.21.15.42:0/452141726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20fc108f70 0x7f20fc19aaf0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:27.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:27.502+0000 7f210270e640 1 --2- 172.21.15.42:0/452141726 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20fc106610 0x7f20fc199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:27.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:27.502+0000 7f210270e640 1 --2- 172.21.15.42:0/452141726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20fc1014c0 0x7f20fc1998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:27.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:27.502+0000 7f210270e640 1 -- 172.21.15.42:0/452141726 >> 172.21.15.42:0/452141726 conn(0x7f20fc0fd100 msgr2=0x7f20fc1073e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:27.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:27.502+0000 7f210270e640 1 -- 172.21.15.42:0/452141726 shutdown_connections 2024-07-26T06:50:27.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:27.502+0000 7f210270e640 1 -- 172.21.15.42:0/452141726 wait complete. 2024-07-26T06:50:27.516 INFO:teuthology.run_tasks:Running task check-counter... 2024-07-26T06:50:27.552 INFO:teuthology.run_tasks:Running task background_exec... 2024-07-26T06:50:27.620 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi042.front.sepia.ceph.com 2024-07-26T06:50:27.620 DEBUG:teuthology.orchestra.run.smithi042:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-07-26T06:50:27.623 INFO:teuthology.run_tasks:Running task quiescer... 2024-07-26T06:50:27.655 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-07-26T06:50:27.656 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph pg dump --format=json 2024-07-26T06:50:27.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:27 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/452141726' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T06:50:27.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:27 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/452141726' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T06:50:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:27 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/452141726' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T06:50:28.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:28 smithi042 ceph-mon[32775]: pgmap v403: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T06:50:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:28 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/452141726' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-26T06:50:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:28 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:50:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:28 smithi163 ceph-mon[36873]: pgmap v403: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T06:50:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:28 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/452141726' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-26T06:50:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:28 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:50:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:28 smithi164 ceph-mon[36970]: pgmap v403: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T06:50:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:28 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/452141726' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-26T06:50:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:28 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:50:29.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.619+0000 7fe06e4b0640 1 -- 172.21.15.42:0/1499849242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068102490 msgr2=0x7fe0681028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:29.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.619+0000 7fe06e4b0640 1 --2- 172.21.15.42:0/1499849242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068102490 0x7fe0681028a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe058009900 tx=0x7fe05802a3f0 comp rx=0 tx=0).stop 2024-07-26T06:50:29.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.619+0000 7fe06e4b0640 1 -- 172.21.15.42:0/1499849242 shutdown_connections 2024-07-26T06:50:29.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.619+0000 7fe06e4b0640 1 --2- 172.21.15.42:0/1499849242 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe068108720 0x7fe06810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:29.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.619+0000 7fe06e4b0640 1 --2- 172.21.15.42:0/1499849242 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe068107dd0 0x7fe0681081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:29.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.619+0000 7fe06e4b0640 1 --2- 172.21.15.42:0/1499849242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068102490 0x7fe0681028a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:29.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.619+0000 7fe06e4b0640 1 -- 172.21.15.42:0/1499849242 >> 172.21.15.42:0/1499849242 conn(0x7fe068079160 msgr2=0x7fe068079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:29.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.619+0000 7fe06e4b0640 1 -- 172.21.15.42:0/1499849242 shutdown_connections 2024-07-26T06:50:29.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.619+0000 7fe06e4b0640 1 -- 172.21.15.42:0/1499849242 wait complete. 2024-07-26T06:50:29.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.620+0000 7fe06e4b0640 1 Processor -- start 2024-07-26T06:50:29.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.620+0000 7fe06e4b0640 1 -- start start 2024-07-26T06:50:29.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe06e4b0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe068102490 0x7fe06819a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:29.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe06e4b0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe068107dd0 0x7fe06819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:29.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe06e4b0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068108720 0x7fe06819f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:29.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe06e4b0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe0681123f0 con 0x7fe068108720 2024-07-26T06:50:29.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe06e4b0640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe068112270 con 0x7fe068102490 2024-07-26T06:50:29.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe06e4b0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe068112570 con 0x7fe068107dd0 2024-07-26T06:50:29.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe06ca4d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068108720 0x7fe06819f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:29.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe06ca4d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068108720 0x7fe06819f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44836/0 (socket says 172.21.15.42:44836) 2024-07-26T06:50:29.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe06ca4d640 1 -- 172.21.15.42:0/883274059 learned_addr learned my addr 172.21.15.42:0/883274059 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:29.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe067fff640 1 --2- 172.21.15.42:0/883274059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe068102490 0x7fe06819a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:29.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe0677fe640 1 --2- 172.21.15.42:0/883274059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe068107dd0 0x7fe06819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:29.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe06ca4d640 1 -- 172.21.15.42:0/883274059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe068107dd0 msgr2=0x7fe06819a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:29.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe06ca4d640 1 --2- 172.21.15.42:0/883274059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe068107dd0 0x7fe06819a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:29.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe06ca4d640 1 -- 172.21.15.42:0/883274059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe068102490 msgr2=0x7fe06819a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:29.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe06ca4d640 1 --2- 172.21.15.42:0/883274059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe068102490 0x7fe06819a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:29.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe06ca4d640 1 -- 172.21.15.42:0/883274059 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe06819fec0 con 0x7fe068108720 2024-07-26T06:50:29.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe0677fe640 1 --2- 172.21.15.42:0/883274059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe068107dd0 0x7fe06819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:29.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.621+0000 7fe06ca4d640 1 --2- 172.21.15.42:0/883274059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068108720 0x7fe06819f850 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe06000cce0 tx=0x7fe06000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:29.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.622+0000 7fe0657fa640 1 -- 172.21.15.42:0/883274059 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06000f450 con 0x7fe068108720 2024-07-26T06:50:29.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.622+0000 7fe06e4b0640 1 -- 172.21.15.42:0/883274059 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0681a01b0 con 0x7fe068108720 2024-07-26T06:50:29.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.622+0000 7fe0657fa640 1 -- 172.21.15.42:0/883274059 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe06000faa0 con 0x7fe068108720 2024-07-26T06:50:29.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.622+0000 7fe0657fa640 1 -- 172.21.15.42:0/883274059 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe060018b70 con 0x7fe068108720 2024-07-26T06:50:29.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.622+0000 7fe06e4b0640 1 -- 172.21.15.42:0/883274059 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0681a69d0 con 0x7fe068108720 2024-07-26T06:50:29.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.623+0000 7fe06e4b0640 1 -- 172.21.15.42:0/883274059 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0681028a0 con 0x7fe068108720 2024-07-26T06:50:29.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.624+0000 7fe0657fa640 1 -- 172.21.15.42:0/883274059 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fe060018d10 con 0x7fe068108720 2024-07-26T06:50:29.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.626+0000 7fe0657fa640 1 --2- 172.21.15.42:0/883274059 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe0500b3c00 0x7fe0500b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:29.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.626+0000 7fe067fff640 1 --2- 172.21.15.42:0/883274059 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe0500b3c00 0x7fe0500b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:29.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.626+0000 7fe0657fa640 1 -- 172.21.15.42:0/883274059 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fe0600e41c0 con 0x7fe068108720 2024-07-26T06:50:29.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.626+0000 7fe067fff640 1 --2- 172.21.15.42:0/883274059 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe0500b3c00 0x7fe0500b6110 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe058031440 tx=0x7fe058005e80 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:29.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.629+0000 7fe0657fa640 1 -- 172.21.15.42:0/883274059 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe060014030 con 0x7fe068108720 2024-07-26T06:50:29.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.812+0000 7fe06e4b0640 1 -- 172.21.15.42:0/883274059 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe068105d70 con 0x7fe0500b3c00 2024-07-26T06:50:29.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.816+0000 7fe0657fa640 1 -- 172.21.15.42:0/883274059 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259113 (secure 0 0 0) 0x7fe068105d70 con 0x7fe0500b3c00 2024-07-26T06:50:29.819 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:50:29.820 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-07-26T06:50:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.819+0000 7fe06e4b0640 1 -- 172.21.15.42:0/883274059 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe0500b3c00 msgr2=0x7fe0500b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.819+0000 7fe06e4b0640 1 --2- 172.21.15.42:0/883274059 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe0500b3c00 0x7fe0500b6110 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe058031440 tx=0x7fe058005e80 comp rx=0 tx=0).stop 2024-07-26T06:50:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.819+0000 7fe06e4b0640 1 -- 172.21.15.42:0/883274059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068108720 msgr2=0x7fe06819f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.819+0000 7fe06e4b0640 1 --2- 172.21.15.42:0/883274059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068108720 0x7fe06819f850 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe06000cce0 tx=0x7fe06000a620 comp rx=0 tx=0).stop 2024-07-26T06:50:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.819+0000 7fe06e4b0640 1 -- 172.21.15.42:0/883274059 shutdown_connections 2024-07-26T06:50:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.819+0000 7fe06e4b0640 1 --2- 172.21.15.42:0/883274059 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe0500b3c00 0x7fe0500b6110 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.819+0000 7fe06e4b0640 1 --2- 172.21.15.42:0/883274059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068108720 0x7fe06819f850 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.819+0000 7fe06e4b0640 1 --2- 172.21.15.42:0/883274059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe068107dd0 0x7fe06819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.819+0000 7fe06e4b0640 1 --2- 172.21.15.42:0/883274059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe068102490 0x7fe06819a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:29.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.819+0000 7fe06e4b0640 1 -- 172.21.15.42:0/883274059 >> 172.21.15.42:0/883274059 conn(0x7fe068079160 msgr2=0x7fe068101580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:29.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.820+0000 7fe06e4b0640 1 -- 172.21.15.42:0/883274059 shutdown_connections 2024-07-26T06:50:29.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:29.820+0000 7fe06e4b0640 1 -- 172.21.15.42:0/883274059 wait complete. 2024-07-26T06:50:30.555 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-07-26T06:50:29.372041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660779,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1964,"num_read_kb":1088,"num_write":386,"num_write_kb":2301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2953474,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":817,"ondisk_log_size":817,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384924,"kb_used_data":10812,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124556900,"statfs":{"total":1151940427776,"available":1151546265600,"internally_reserved":0,"allocated":11071488,"data_stored":7074061,"data_compressed":177858,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":137,"num_read_kb":72,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003297"},"pg_stats":[{"pgid":"3.3f","version":"80'8","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337456+0000","last_change":"2024-07-26T06:47:05.026186+0000","last_active":"2024-07-26T06:48:51.337456+0000","last_peered":"2024-07-26T06:48:51.337456+0000","last_clean":"2024-07-26T06:48:51.337456+0000","last_became_active":"2024-07-26T06:47:05.025589+0000","last_became_peered":"2024-07-26T06:47:05.025589+0000","last_unstale":"2024-07-26T06:48:51.337456+0000","last_undegraded":"2024-07-26T06:48:51.337456+0000","last_fullsized":"2024-07-26T06:48:51.337456+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:48:23.100966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"80'4","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326008+0000","last_change":"2024-07-26T06:47:05.033211+0000","last_active":"2024-07-26T06:48:51.326008+0000","last_peered":"2024-07-26T06:48:51.326008+0000","last_clean":"2024-07-26T06:48:51.326008+0000","last_became_active":"2024-07-26T06:47:05.032630+0000","last_became_peered":"2024-07-26T06:47:05.032630+0000","last_unstale":"2024-07-26T06:48:51.326008+0000","last_undegraded":"2024-07-26T06:48:51.326008+0000","last_fullsized":"2024-07-26T06:48:51.326008+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:03:19.612963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335969+0000","last_change":"2024-07-26T06:47:05.034767+0000","last_active":"2024-07-26T06:48:51.335969+0000","last_peered":"2024-07-26T06:48:51.335969+0000","last_clean":"2024-07-26T06:48:51.335969+0000","last_became_active":"2024-07-26T06:47:05.034376+0000","last_became_peered":"2024-07-26T06:47:05.034376+0000","last_unstale":"2024-07-26T06:48:51.335969+0000","last_undegraded":"2024-07-26T06:48:51.335969+0000","last_fullsized":"2024-07-26T06:48:51.335969+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:55:55.729930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328614+0000","last_change":"2024-07-26T06:47:05.030995+0000","last_active":"2024-07-26T06:48:51.328614+0000","last_peered":"2024-07-26T06:48:51.328614+0000","last_clean":"2024-07-26T06:48:51.328614+0000","last_became_active":"2024-07-26T06:47:05.025423+0000","last_became_peered":"2024-07-26T06:47:05.025423+0000","last_unstale":"2024-07-26T06:48:51.328614+0000","last_undegraded":"2024-07-26T06:48:51.328614+0000","last_fullsized":"2024-07-26T06:48:51.328614+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:59:39.511383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"80'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.317041+0000","last_change":"2024-07-26T06:47:05.035359+0000","last_active":"2024-07-26T06:48:51.317041+0000","last_peered":"2024-07-26T06:48:51.317041+0000","last_clean":"2024-07-26T06:48:51.317041+0000","last_became_active":"2024-07-26T06:47:05.034836+0000","last_became_peered":"2024-07-26T06:47:05.034836+0000","last_unstale":"2024-07-26T06:48:51.317041+0000","last_undegraded":"2024-07-26T06:48:51.317041+0000","last_fullsized":"2024-07-26T06:48:51.317041+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:54.482774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"80'6","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326306+0000","last_change":"2024-07-26T06:47:05.023660+0000","last_active":"2024-07-26T06:48:51.326306+0000","last_peered":"2024-07-26T06:48:51.326306+0000","last_clean":"2024-07-26T06:48:51.326306+0000","last_became_active":"2024-07-26T06:47:05.022446+0000","last_became_peered":"2024-07-26T06:47:05.022446+0000","last_unstale":"2024-07-26T06:48:51.326306+0000","last_undegraded":"2024-07-26T06:48:51.326306+0000","last_fullsized":"2024-07-26T06:48:51.326306+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:35:00.519226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"80'5","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339509+0000","last_change":"2024-07-26T06:47:05.025132+0000","last_active":"2024-07-26T06:48:51.339509+0000","last_peered":"2024-07-26T06:48:51.339509+0000","last_clean":"2024-07-26T06:48:51.339509+0000","last_became_active":"2024-07-26T06:47:05.024770+0000","last_became_peered":"2024-07-26T06:47:05.024770+0000","last_unstale":"2024-07-26T06:48:51.339509+0000","last_undegraded":"2024-07-26T06:48:51.339509+0000","last_fullsized":"2024-07-26T06:48:51.339509+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:21:39.920152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"80'9","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333625+0000","last_change":"2024-07-26T06:47:05.022613+0000","last_active":"2024-07-26T06:48:51.333625+0000","last_peered":"2024-07-26T06:48:51.333625+0000","last_clean":"2024-07-26T06:48:51.333625+0000","last_became_active":"2024-07-26T06:47:05.022202+0000","last_became_peered":"2024-07-26T06:47:05.022202+0000","last_unstale":"2024-07-26T06:48:51.333625+0000","last_undegraded":"2024-07-26T06:48:51.333625+0000","last_fullsized":"2024-07-26T06:48:51.333625+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:07:23.980843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"80'11","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.336947+0000","last_change":"2024-07-26T06:47:05.026352+0000","last_active":"2024-07-26T06:48:51.336947+0000","last_peered":"2024-07-26T06:48:51.336947+0000","last_clean":"2024-07-26T06:48:51.336947+0000","last_became_active":"2024-07-26T06:47:05.025969+0000","last_became_peered":"2024-07-26T06:47:05.025969+0000","last_unstale":"2024-07-26T06:48:51.336947+0000","last_undegraded":"2024-07-26T06:48:51.336947+0000","last_fullsized":"2024-07-26T06:48:51.336947+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:27.062013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"80'16","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.329437+0000","last_change":"2024-07-26T06:47:23.100422+0000","last_active":"2024-07-26T06:48:51.329437+0000","last_peered":"2024-07-26T06:48:51.329437+0000","last_clean":"2024-07-26T06:48:51.329437+0000","last_became_active":"2024-07-26T06:47:23.100023+0000","last_became_peered":"2024-07-26T06:47:23.100023+0000","last_unstale":"2024-07-26T06:48:51.329437+0000","last_undegraded":"2024-07-26T06:48:51.329437+0000","last_fullsized":"2024-07-26T06:48:51.329437+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:28.158112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"80'6","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338557+0000","last_change":"2024-07-26T06:47:05.037142+0000","last_active":"2024-07-26T06:48:51.338557+0000","last_peered":"2024-07-26T06:48:51.338557+0000","last_clean":"2024-07-26T06:48:51.338557+0000","last_became_active":"2024-07-26T06:47:05.036745+0000","last_became_peered":"2024-07-26T06:47:05.036745+0000","last_unstale":"2024-07-26T06:48:51.338557+0000","last_undegraded":"2024-07-26T06:48:51.338557+0000","last_fullsized":"2024-07-26T06:48:51.338557+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:31:10.034560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"80'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.315686+0000","last_change":"2024-07-26T06:47:05.030640+0000","last_active":"2024-07-26T06:48:51.315686+0000","last_peered":"2024-07-26T06:48:51.315686+0000","last_clean":"2024-07-26T06:48:51.315686+0000","last_became_active":"2024-07-26T06:47:05.023583+0000","last_became_peered":"2024-07-26T06:47:05.023583+0000","last_unstale":"2024-07-26T06:48:51.315686+0000","last_undegraded":"2024-07-26T06:48:51.315686+0000","last_fullsized":"2024-07-26T06:48:51.315686+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:59:12.312930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"80'10","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333194+0000","last_change":"2024-07-26T06:47:05.021768+0000","last_active":"2024-07-26T06:48:51.333194+0000","last_peered":"2024-07-26T06:48:51.333194+0000","last_clean":"2024-07-26T06:48:51.333194+0000","last_became_active":"2024-07-26T06:47:05.021410+0000","last_became_peered":"2024-07-26T06:47:05.021410+0000","last_unstale":"2024-07-26T06:48:51.333194+0000","last_undegraded":"2024-07-26T06:48:51.333194+0000","last_fullsized":"2024-07-26T06:48:51.333194+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:05.576122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.324480+0000","last_change":"2024-07-26T06:47:05.035814+0000","last_active":"2024-07-26T06:48:51.324480+0000","last_peered":"2024-07-26T06:48:51.324480+0000","last_clean":"2024-07-26T06:48:51.324480+0000","last_became_active":"2024-07-26T06:47:05.035451+0000","last_became_peered":"2024-07-26T06:47:05.035451+0000","last_unstale":"2024-07-26T06:48:51.324480+0000","last_undegraded":"2024-07-26T06:48:51.324480+0000","last_fullsized":"2024-07-26T06:48:51.324480+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:33.533726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.331515+0000","last_change":"2024-07-26T06:47:05.035720+0000","last_active":"2024-07-26T06:48:51.331515+0000","last_peered":"2024-07-26T06:48:51.331515+0000","last_clean":"2024-07-26T06:48:51.331515+0000","last_became_active":"2024-07-26T06:47:05.035359+0000","last_became_peered":"2024-07-26T06:47:05.035359+0000","last_unstale":"2024-07-26T06:48:51.331515+0000","last_undegraded":"2024-07-26T06:48:51.331515+0000","last_fullsized":"2024-07-26T06:48:51.331515+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:38.561268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339962+0000","last_change":"2024-07-26T06:47:05.027109+0000","last_active":"2024-07-26T06:48:51.339962+0000","last_peered":"2024-07-26T06:48:51.339962+0000","last_clean":"2024-07-26T06:48:51.339962+0000","last_became_active":"2024-07-26T06:47:05.026694+0000","last_became_peered":"2024-07-26T06:47:05.026694+0000","last_unstale":"2024-07-26T06:48:51.339962+0000","last_undegraded":"2024-07-26T06:48:51.339962+0000","last_fullsized":"2024-07-26T06:48:51.339962+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:25:01.215646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"80'8","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337458+0000","last_change":"2024-07-26T06:47:05.028212+0000","last_active":"2024-07-26T06:48:51.337458+0000","last_peered":"2024-07-26T06:48:51.337458+0000","last_clean":"2024-07-26T06:48:51.337458+0000","last_became_active":"2024-07-26T06:47:05.027598+0000","last_became_peered":"2024-07-26T06:47:05.027598+0000","last_unstale":"2024-07-26T06:48:51.337458+0000","last_undegraded":"2024-07-26T06:48:51.337458+0000","last_fullsized":"2024-07-26T06:48:51.337458+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:55:31.536194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"80'10","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.336116+0000","last_change":"2024-07-26T06:47:05.027724+0000","last_active":"2024-07-26T06:48:51.336116+0000","last_peered":"2024-07-26T06:48:51.336116+0000","last_clean":"2024-07-26T06:48:51.336116+0000","last_became_active":"2024-07-26T06:47:05.027348+0000","last_became_peered":"2024-07-26T06:47:05.027348+0000","last_unstale":"2024-07-26T06:48:51.336116+0000","last_undegraded":"2024-07-26T06:48:51.336116+0000","last_fullsized":"2024-07-26T06:48:51.336116+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:04.180467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"80'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.327682+0000","last_change":"2024-07-26T06:47:05.025474+0000","last_active":"2024-07-26T06:48:51.327682+0000","last_peered":"2024-07-26T06:48:51.327682+0000","last_clean":"2024-07-26T06:48:51.327682+0000","last_became_active":"2024-07-26T06:47:05.025016+0000","last_became_peered":"2024-07-26T06:47:05.025016+0000","last_unstale":"2024-07-26T06:48:51.327682+0000","last_undegraded":"2024-07-26T06:48:51.327682+0000","last_fullsized":"2024-07-26T06:48:51.327682+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:40:03.034532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"80'8","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.330348+0000","last_change":"2024-07-26T06:47:05.027533+0000","last_active":"2024-07-26T06:48:51.330348+0000","last_peered":"2024-07-26T06:48:51.330348+0000","last_clean":"2024-07-26T06:48:51.330348+0000","last_became_active":"2024-07-26T06:47:05.027143+0000","last_became_peered":"2024-07-26T06:47:05.027143+0000","last_unstale":"2024-07-26T06:48:51.330348+0000","last_undegraded":"2024-07-26T06:48:51.330348+0000","last_fullsized":"2024-07-26T06:48:51.330348+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:13.406430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"80'11","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338271+0000","last_change":"2024-07-26T06:47:05.025962+0000","last_active":"2024-07-26T06:48:51.338271+0000","last_peered":"2024-07-26T06:48:51.338271+0000","last_clean":"2024-07-26T06:48:51.338271+0000","last_became_active":"2024-07-26T06:47:05.025621+0000","last_became_peered":"2024-07-26T06:47:05.025621+0000","last_unstale":"2024-07-26T06:48:51.338271+0000","last_undegraded":"2024-07-26T06:48:51.338271+0000","last_fullsized":"2024-07-26T06:48:51.338271+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:56:43.669247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"80'10","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337425+0000","last_change":"2024-07-26T06:47:05.026252+0000","last_active":"2024-07-26T06:48:51.337425+0000","last_peered":"2024-07-26T06:48:51.337425+0000","last_clean":"2024-07-26T06:48:51.337425+0000","last_became_active":"2024-07-26T06:47:05.025875+0000","last_became_peered":"2024-07-26T06:47:05.025875+0000","last_unstale":"2024-07-26T06:48:51.337425+0000","last_undegraded":"2024-07-26T06:48:51.337425+0000","last_fullsized":"2024-07-26T06:48:51.337425+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:27:32.955862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.327073+0000","last_change":"2024-07-26T06:47:05.022513+0000","last_active":"2024-07-26T06:48:51.327073+0000","last_peered":"2024-07-26T06:48:51.327073+0000","last_clean":"2024-07-26T06:48:51.327073+0000","last_became_active":"2024-07-26T06:47:05.022159+0000","last_became_peered":"2024-07-26T06:47:05.022159+0000","last_unstale":"2024-07-26T06:48:51.327073+0000","last_undegraded":"2024-07-26T06:48:51.327073+0000","last_fullsized":"2024-07-26T06:48:51.327073+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:53:06.402985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"80'6","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.325351+0000","last_change":"2024-07-26T06:47:23.097604+0000","last_active":"2024-07-26T06:48:51.325351+0000","last_peered":"2024-07-26T06:48:51.325351+0000","last_clean":"2024-07-26T06:48:51.325351+0000","last_became_active":"2024-07-26T06:47:23.097170+0000","last_became_peered":"2024-07-26T06:47:23.097170+0000","last_unstale":"2024-07-26T06:48:51.325351+0000","last_undegraded":"2024-07-26T06:48:51.325351+0000","last_fullsized":"2024-07-26T06:48:51.325351+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:38:24.530396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"80'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.323758+0000","last_change":"2024-07-26T06:47:05.026349+0000","last_active":"2024-07-26T06:48:51.323758+0000","last_peered":"2024-07-26T06:48:51.323758+0000","last_clean":"2024-07-26T06:48:51.323758+0000","last_became_active":"2024-07-26T06:47:05.026016+0000","last_became_peered":"2024-07-26T06:47:05.026016+0000","last_unstale":"2024-07-26T06:48:51.323758+0000","last_undegraded":"2024-07-26T06:48:51.323758+0000","last_fullsized":"2024-07-26T06:48:51.323758+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:19:02.264183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"80'15","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.323355+0000","last_change":"2024-07-26T06:47:05.032439+0000","last_active":"2024-07-26T06:48:51.323355+0000","last_peered":"2024-07-26T06:48:51.323355+0000","last_clean":"2024-07-26T06:48:51.323355+0000","last_became_active":"2024-07-26T06:47:05.032105+0000","last_became_peered":"2024-07-26T06:47:05.032105+0000","last_unstale":"2024-07-26T06:48:51.323355+0000","last_undegraded":"2024-07-26T06:48:51.323355+0000","last_fullsized":"2024-07-26T06:48:51.323355+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:21:50.423939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"80'6","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.319567+0000","last_change":"2024-07-26T06:47:05.026927+0000","last_active":"2024-07-26T06:48:51.319567+0000","last_peered":"2024-07-26T06:48:51.319567+0000","last_clean":"2024-07-26T06:48:51.319567+0000","last_became_active":"2024-07-26T06:47:05.026569+0000","last_became_peered":"2024-07-26T06:47:05.026569+0000","last_unstale":"2024-07-26T06:48:51.319567+0000","last_undegraded":"2024-07-26T06:48:51.319567+0000","last_fullsized":"2024-07-26T06:48:51.319567+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:12:14.687633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"80'4","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333564+0000","last_change":"2024-07-26T06:47:23.099689+0000","last_active":"2024-07-26T06:48:51.333564+0000","last_peered":"2024-07-26T06:48:51.333564+0000","last_clean":"2024-07-26T06:48:51.333564+0000","last_became_active":"2024-07-26T06:47:23.099313+0000","last_became_peered":"2024-07-26T06:47:23.099313+0000","last_unstale":"2024-07-26T06:48:51.333564+0000","last_undegraded":"2024-07-26T06:48:51.333564+0000","last_fullsized":"2024-07-26T06:48:51.333564+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:49:01.833297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333361+0000","last_change":"2024-07-26T06:47:05.034864+0000","last_active":"2024-07-26T06:48:51.333361+0000","last_peered":"2024-07-26T06:48:51.333361+0000","last_clean":"2024-07-26T06:48:51.333361+0000","last_became_active":"2024-07-26T06:47:05.033414+0000","last_became_peered":"2024-07-26T06:47:05.033414+0000","last_unstale":"2024-07-26T06:48:51.333361+0000","last_undegraded":"2024-07-26T06:48:51.333361+0000","last_fullsized":"2024-07-26T06:48:51.333361+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:21:31.420212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.324409+0000","last_change":"2024-07-26T06:47:05.025462+0000","last_active":"2024-07-26T06:48:51.324409+0000","last_peered":"2024-07-26T06:48:51.324409+0000","last_clean":"2024-07-26T06:48:51.324409+0000","last_became_active":"2024-07-26T06:47:05.025049+0000","last_became_peered":"2024-07-26T06:47:05.025049+0000","last_unstale":"2024-07-26T06:48:51.324409+0000","last_undegraded":"2024-07-26T06:48:51.324409+0000","last_fullsized":"2024-07-26T06:48:51.324409+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:59:28.903338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"80'12","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335500+0000","last_change":"2024-07-26T06:47:05.026394+0000","last_active":"2024-07-26T06:48:51.335500+0000","last_peered":"2024-07-26T06:48:51.335500+0000","last_clean":"2024-07-26T06:48:51.335500+0000","last_became_active":"2024-07-26T06:47:05.025820+0000","last_became_peered":"2024-07-26T06:47:05.025820+0000","last_unstale":"2024-07-26T06:48:51.335500+0000","last_undegraded":"2024-07-26T06:48:51.335500+0000","last_fullsized":"2024-07-26T06:48:51.335500+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:44:57.441638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.329050+0000","last_change":"2024-07-26T06:47:05.024373+0000","last_active":"2024-07-26T06:48:51.329050+0000","last_peered":"2024-07-26T06:48:51.329050+0000","last_clean":"2024-07-26T06:48:51.329050+0000","last_became_active":"2024-07-26T06:47:05.023962+0000","last_became_peered":"2024-07-26T06:47:05.023962+0000","last_unstale":"2024-07-26T06:48:51.329050+0000","last_undegraded":"2024-07-26T06:48:51.329050+0000","last_fullsized":"2024-07-26T06:48:51.329050+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:43.804614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"80'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.325450+0000","last_change":"2024-07-26T06:47:05.031146+0000","last_active":"2024-07-26T06:48:51.325450+0000","last_peered":"2024-07-26T06:48:51.325450+0000","last_clean":"2024-07-26T06:48:51.325450+0000","last_became_active":"2024-07-26T06:47:05.029393+0000","last_became_peered":"2024-07-26T06:47:05.029393+0000","last_unstale":"2024-07-26T06:48:51.325450+0000","last_undegraded":"2024-07-26T06:48:51.325450+0000","last_fullsized":"2024-07-26T06:48:51.325450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:46.853933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"80'8","reported_seq":223,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:26.418121+0000","last_change":"2024-07-26T06:47:02.001800+0000","last_active":"2024-07-26T06:50:26.418121+0000","last_peered":"2024-07-26T06:50:26.418121+0000","last_clean":"2024-07-26T06:50:26.418121+0000","last_became_active":"2024-07-26T06:47:02.001435+0000","last_became_peered":"2024-07-26T06:47:02.001435+0000","last_unstale":"2024-07-26T06:50:26.418121+0000","last_undegraded":"2024-07-26T06:50:26.418121+0000","last_fullsized":"2024-07-26T06:50:26.418121+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:23.121378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1462,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"80'8","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.334283+0000","last_change":"2024-07-26T06:47:05.028301+0000","last_active":"2024-07-26T06:48:51.334283+0000","last_peered":"2024-07-26T06:48:51.334283+0000","last_clean":"2024-07-26T06:48:51.334283+0000","last_became_active":"2024-07-26T06:47:05.027761+0000","last_became_peered":"2024-07-26T06:47:05.027761+0000","last_unstale":"2024-07-26T06:48:51.334283+0000","last_undegraded":"2024-07-26T06:48:51.334283+0000","last_fullsized":"2024-07-26T06:48:51.334283+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:19.109395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093824+0000","last_change":"2024-07-26T06:47:02.004488+0000","last_active":"2024-07-26T06:47:23.093824+0000","last_peered":"2024-07-26T06:47:23.093824+0000","last_clean":"2024-07-26T06:47:23.093824+0000","last_became_active":"2024-07-26T06:47:02.004163+0000","last_became_peered":"2024-07-26T06:47:02.004163+0000","last_unstale":"2024-07-26T06:47:23.093824+0000","last_undegraded":"2024-07-26T06:47:23.093824+0000","last_fullsized":"2024-07-26T06:47:23.093824+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:16.221618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096130+0000","last_change":"2024-07-26T06:47:01.992708+0000","last_active":"2024-07-26T06:47:23.096130+0000","last_peered":"2024-07-26T06:47:23.096130+0000","last_clean":"2024-07-26T06:47:23.096130+0000","last_became_active":"2024-07-26T06:47:01.992330+0000","last_became_peered":"2024-07-26T06:47:01.992330+0000","last_unstale":"2024-07-26T06:47:23.096130+0000","last_undegraded":"2024-07-26T06:47:23.096130+0000","last_fullsized":"2024-07-26T06:47:23.096130+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:25.951946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.330722+0000","last_change":"2024-07-26T06:47:05.013946+0000","last_active":"2024-07-26T06:48:51.330722+0000","last_peered":"2024-07-26T06:48:51.330722+0000","last_clean":"2024-07-26T06:48:51.330722+0000","last_became_active":"2024-07-26T06:47:05.013509+0000","last_became_peered":"2024-07-26T06:47:05.013509+0000","last_unstale":"2024-07-26T06:48:51.330722+0000","last_undegraded":"2024-07-26T06:48:51.330722+0000","last_fullsized":"2024-07-26T06:48:51.330722+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:09.789287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.124184+0000","last_change":"2024-07-26T06:47:01.997617+0000","last_active":"2024-07-26T06:47:24.124184+0000","last_peered":"2024-07-26T06:47:24.124184+0000","last_clean":"2024-07-26T06:47:24.124184+0000","last_became_active":"2024-07-26T06:47:01.997238+0000","last_became_peered":"2024-07-26T06:47:01.997238+0000","last_unstale":"2024-07-26T06:47:24.124184+0000","last_undegraded":"2024-07-26T06:47:24.124184+0000","last_fullsized":"2024-07-26T06:47:24.124184+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:45.596438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"80'11","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328533+0000","last_change":"2024-07-26T06:47:23.099813+0000","last_active":"2024-07-26T06:48:51.328533+0000","last_peered":"2024-07-26T06:48:51.328533+0000","last_clean":"2024-07-26T06:48:51.328533+0000","last_became_active":"2024-07-26T06:47:23.098701+0000","last_became_peered":"2024-07-26T06:47:23.098701+0000","last_unstale":"2024-07-26T06:48:51.328533+0000","last_undegraded":"2024-07-26T06:48:51.328533+0000","last_fullsized":"2024-07-26T06:48:51.328533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:42:04.059637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'6","reported_seq":403,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:28.421232+0000","last_change":"2024-07-26T06:47:01.998508+0000","last_active":"2024-07-26T06:50:28.421232+0000","last_peered":"2024-07-26T06:50:28.421232+0000","last_clean":"2024-07-26T06:50:28.421232+0000","last_became_active":"2024-07-26T06:47:01.998132+0000","last_became_peered":"2024-07-26T06:47:01.998132+0000","last_unstale":"2024-07-26T06:50:28.421232+0000","last_undegraded":"2024-07-26T06:50:28.421232+0000","last_fullsized":"2024-07-26T06:50:28.421232+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:21.588085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":370,"num_read_kb":370,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"80'10","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339164+0000","last_change":"2024-07-26T06:47:05.032704+0000","last_active":"2024-07-26T06:48:51.339164+0000","last_peered":"2024-07-26T06:48:51.339164+0000","last_clean":"2024-07-26T06:48:51.339164+0000","last_became_active":"2024-07-26T06:47:05.031843+0000","last_became_peered":"2024-07-26T06:47:05.031843+0000","last_unstale":"2024-07-26T06:48:51.339164+0000","last_undegraded":"2024-07-26T06:48:51.339164+0000","last_fullsized":"2024-07-26T06:48:51.339164+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:58:24.436233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097471+0000","last_change":"2024-07-26T06:47:02.006266+0000","last_active":"2024-07-26T06:47:23.097471+0000","last_peered":"2024-07-26T06:47:23.097471+0000","last_clean":"2024-07-26T06:47:23.097471+0000","last_became_active":"2024-07-26T06:47:02.005830+0000","last_became_peered":"2024-07-26T06:47:02.005830+0000","last_unstale":"2024-07-26T06:47:23.097471+0000","last_undegraded":"2024-07-26T06:47:23.097471+0000","last_fullsized":"2024-07-26T06:47:23.097471+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:06:09.097781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.331471+0000","last_change":"2024-07-26T06:47:05.024820+0000","last_active":"2024-07-26T06:48:51.331471+0000","last_peered":"2024-07-26T06:48:51.331471+0000","last_clean":"2024-07-26T06:48:51.331471+0000","last_became_active":"2024-07-26T06:47:05.023941+0000","last_became_peered":"2024-07-26T06:47:05.023941+0000","last_unstale":"2024-07-26T06:48:51.331471+0000","last_undegraded":"2024-07-26T06:48:51.331471+0000","last_fullsized":"2024-07-26T06:48:51.331471+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:07.885159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.123256+0000","last_change":"2024-07-26T06:47:02.023308+0000","last_active":"2024-07-26T06:47:24.123256+0000","last_peered":"2024-07-26T06:47:24.123256+0000","last_clean":"2024-07-26T06:47:24.123256+0000","last_became_active":"2024-07-26T06:47:02.022933+0000","last_became_peered":"2024-07-26T06:47:02.022933+0000","last_unstale":"2024-07-26T06:47:24.123256+0000","last_undegraded":"2024-07-26T06:47:24.123256+0000","last_fullsized":"2024-07-26T06:47:24.123256+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:59:01.233386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"80'11","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.340511+0000","last_change":"2024-07-26T06:47:05.026005+0000","last_active":"2024-07-26T06:48:51.340511+0000","last_peered":"2024-07-26T06:48:51.340511+0000","last_clean":"2024-07-26T06:48:51.340511+0000","last_became_active":"2024-07-26T06:47:05.025599+0000","last_became_peered":"2024-07-26T06:47:05.025599+0000","last_unstale":"2024-07-26T06:48:51.340511+0000","last_undegraded":"2024-07-26T06:48:51.340511+0000","last_fullsized":"2024-07-26T06:48:51.340511+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:44:19.924157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'36","reported_seq":463,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:28.421272+0000","last_change":"2024-07-26T06:47:23.393050+0000","last_active":"2024-07-26T06:50:28.421272+0000","last_peered":"2024-07-26T06:50:28.421272+0000","last_clean":"2024-07-26T06:50:28.421272+0000","last_became_active":"2024-07-26T06:47:23.381611+0000","last_became_peered":"2024-07-26T06:47:23.381611+0000","last_unstale":"2024-07-26T06:50:28.421272+0000","last_undegraded":"2024-07-26T06:50:28.421272+0000","last_fullsized":"2024-07-26T06:50:28.421272+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:23.866523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62383,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":381,"num_read_kb":251,"num_write":38,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"80'6","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335684+0000","last_change":"2024-07-26T06:47:05.034190+0000","last_active":"2024-07-26T06:48:51.335684+0000","last_peered":"2024-07-26T06:48:51.335684+0000","last_clean":"2024-07-26T06:48:51.335684+0000","last_became_active":"2024-07-26T06:47:05.033569+0000","last_became_peered":"2024-07-26T06:47:05.033569+0000","last_unstale":"2024-07-26T06:48:51.335684+0000","last_undegraded":"2024-07-26T06:48:51.335684+0000","last_fullsized":"2024-07-26T06:48:51.335684+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:39.306996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.123635+0000","last_change":"2024-07-26T06:47:02.001718+0000","last_active":"2024-07-26T06:47:24.123635+0000","last_peered":"2024-07-26T06:47:24.123635+0000","last_clean":"2024-07-26T06:47:24.123635+0000","last_became_active":"2024-07-26T06:47:02.001359+0000","last_became_peered":"2024-07-26T06:47:02.001359+0000","last_unstale":"2024-07-26T06:47:24.123635+0000","last_undegraded":"2024-07-26T06:47:24.123635+0000","last_fullsized":"2024-07-26T06:47:24.123635+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:03:23.390279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337843+0000","last_change":"2024-07-26T06:47:05.027445+0000","last_active":"2024-07-26T06:48:51.337843+0000","last_peered":"2024-07-26T06:48:51.337843+0000","last_clean":"2024-07-26T06:48:51.337843+0000","last_became_active":"2024-07-26T06:47:05.027114+0000","last_became_peered":"2024-07-26T06:47:05.027114+0000","last_unstale":"2024-07-26T06:48:51.337843+0000","last_undegraded":"2024-07-26T06:48:51.337843+0000","last_fullsized":"2024-07-26T06:48:51.337843+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:29:10.519271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.119909+0000","last_change":"2024-07-26T06:47:01.998832+0000","last_active":"2024-07-26T06:47:24.119909+0000","last_peered":"2024-07-26T06:47:24.119909+0000","last_clean":"2024-07-26T06:47:24.119909+0000","last_became_active":"2024-07-26T06:47:01.998512+0000","last_became_peered":"2024-07-26T06:47:01.998512+0000","last_unstale":"2024-07-26T06:47:24.119909+0000","last_undegraded":"2024-07-26T06:47:24.119909+0000","last_fullsized":"2024-07-26T06:47:24.119909+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:27.722329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"80'15","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.340182+0000","last_change":"2024-07-26T06:47:05.027803+0000","last_active":"2024-07-26T06:48:51.340182+0000","last_peered":"2024-07-26T06:48:51.340182+0000","last_clean":"2024-07-26T06:48:51.340182+0000","last_became_active":"2024-07-26T06:47:05.027320+0000","last_became_peered":"2024-07-26T06:47:05.027320+0000","last_unstale":"2024-07-26T06:48:51.340182+0000","last_undegraded":"2024-07-26T06:48:51.340182+0000","last_fullsized":"2024-07-26T06:48:51.340182+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:43:06.242945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'7","reported_seq":223,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:26.417956+0000","last_change":"2024-07-26T06:47:01.998356+0000","last_active":"2024-07-26T06:50:26.417956+0000","last_peered":"2024-07-26T06:50:26.417956+0000","last_clean":"2024-07-26T06:50:26.417956+0000","last_became_active":"2024-07-26T06:47:01.997922+0000","last_became_peered":"2024-07-26T06:47:01.997922+0000","last_unstale":"2024-07-26T06:50:26.417956+0000","last_undegraded":"2024-07-26T06:50:26.417956+0000","last_fullsized":"2024-07-26T06:50:26.417956+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:50:58.066682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":189,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"80'6","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339434+0000","last_change":"2024-07-26T06:47:05.025285+0000","last_active":"2024-07-26T06:48:51.339434+0000","last_peered":"2024-07-26T06:48:51.339434+0000","last_clean":"2024-07-26T06:48:51.339434+0000","last_became_active":"2024-07-26T06:47:05.024945+0000","last_became_peered":"2024-07-26T06:47:05.024945+0000","last_unstale":"2024-07-26T06:48:51.339434+0000","last_undegraded":"2024-07-26T06:48:51.339434+0000","last_fullsized":"2024-07-26T06:48:51.339434+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:21:04.680435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.124278+0000","last_change":"2024-07-26T06:47:02.024895+0000","last_active":"2024-07-26T06:47:24.124278+0000","last_peered":"2024-07-26T06:47:24.124278+0000","last_clean":"2024-07-26T06:47:24.124278+0000","last_became_active":"2024-07-26T06:47:02.024451+0000","last_became_peered":"2024-07-26T06:47:02.024451+0000","last_unstale":"2024-07-26T06:47:24.124278+0000","last_undegraded":"2024-07-26T06:47:24.124278+0000","last_fullsized":"2024-07-26T06:47:24.124278+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:05:13.396499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"80'6","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335788+0000","last_change":"2024-07-26T06:47:23.381060+0000","last_active":"2024-07-26T06:48:51.335788+0000","last_peered":"2024-07-26T06:48:51.335788+0000","last_clean":"2024-07-26T06:48:51.335788+0000","last_became_active":"2024-07-26T06:47:23.380685+0000","last_became_peered":"2024-07-26T06:47:23.380685+0000","last_unstale":"2024-07-26T06:48:51.335788+0000","last_undegraded":"2024-07-26T06:48:51.335788+0000","last_fullsized":"2024-07-26T06:48:51.335788+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:21:38.062900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.124080+0000","last_change":"2024-07-26T06:47:02.002098+0000","last_active":"2024-07-26T06:47:24.124080+0000","last_peered":"2024-07-26T06:47:24.124080+0000","last_clean":"2024-07-26T06:47:24.124080+0000","last_became_active":"2024-07-26T06:47:02.001428+0000","last_became_peered":"2024-07-26T06:47:02.001428+0000","last_unstale":"2024-07-26T06:47:24.124080+0000","last_undegraded":"2024-07-26T06:47:24.124080+0000","last_fullsized":"2024-07-26T06:47:24.124080+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:52:29.112500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"80'5","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.330074+0000","last_change":"2024-07-26T06:47:05.023686+0000","last_active":"2024-07-26T06:48:51.330074+0000","last_peered":"2024-07-26T06:48:51.330074+0000","last_clean":"2024-07-26T06:48:51.330074+0000","last_became_active":"2024-07-26T06:47:05.022583+0000","last_became_peered":"2024-07-26T06:47:05.022583+0000","last_unstale":"2024-07-26T06:48:51.330074+0000","last_undegraded":"2024-07-26T06:48:51.330074+0000","last_fullsized":"2024-07-26T06:48:51.330074+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:29:03.752429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096491+0000","last_change":"2024-07-26T06:47:02.006040+0000","last_active":"2024-07-26T06:47:23.096491+0000","last_peered":"2024-07-26T06:47:23.096491+0000","last_clean":"2024-07-26T06:47:23.096491+0000","last_became_active":"2024-07-26T06:47:02.005586+0000","last_became_peered":"2024-07-26T06:47:02.005586+0000","last_unstale":"2024-07-26T06:47:23.096491+0000","last_undegraded":"2024-07-26T06:47:23.096491+0000","last_fullsized":"2024-07-26T06:47:23.096491+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:09:27.180873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335338+0000","last_change":"2024-07-26T06:47:05.035524+0000","last_active":"2024-07-26T06:48:51.335338+0000","last_peered":"2024-07-26T06:48:51.335338+0000","last_clean":"2024-07-26T06:48:51.335338+0000","last_became_active":"2024-07-26T06:47:05.035078+0000","last_became_peered":"2024-07-26T06:47:05.035078+0000","last_unstale":"2024-07-26T06:48:51.335338+0000","last_undegraded":"2024-07-26T06:48:51.335338+0000","last_fullsized":"2024-07-26T06:48:51.335338+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:03.758261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.123534+0000","last_change":"2024-07-26T06:47:01.999810+0000","last_active":"2024-07-26T06:47:24.123534+0000","last_peered":"2024-07-26T06:47:24.123534+0000","last_clean":"2024-07-26T06:47:24.123534+0000","last_became_active":"2024-07-26T06:47:01.999460+0000","last_became_peered":"2024-07-26T06:47:01.999460+0000","last_unstale":"2024-07-26T06:47:24.123534+0000","last_undegraded":"2024-07-26T06:47:24.123534+0000","last_fullsized":"2024-07-26T06:47:24.123534+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:25:05.691900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339690+0000","last_change":"2024-07-26T06:47:05.033631+0000","last_active":"2024-07-26T06:48:51.339690+0000","last_peered":"2024-07-26T06:48:51.339690+0000","last_clean":"2024-07-26T06:48:51.339690+0000","last_became_active":"2024-07-26T06:47:05.033266+0000","last_became_peered":"2024-07-26T06:47:05.033266+0000","last_unstale":"2024-07-26T06:48:51.339690+0000","last_undegraded":"2024-07-26T06:48:51.339690+0000","last_fullsized":"2024-07-26T06:48:51.339690+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:55.895271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"78'5","reported_seq":217,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:24.431606+0000","last_change":"2024-07-26T06:47:02.113525+0000","last_active":"2024-07-26T06:50:24.431606+0000","last_peered":"2024-07-26T06:50:24.431606+0000","last_clean":"2024-07-26T06:50:24.431606+0000","last_became_active":"2024-07-26T06:47:02.113112+0000","last_became_peered":"2024-07-26T06:47:02.113112+0000","last_unstale":"2024-07-26T06:50:24.431606+0000","last_undegraded":"2024-07-26T06:50:24.431606+0000","last_fullsized":"2024-07-26T06:50:24.431606+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:05:15.071264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":184,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338494+0000","last_change":"2024-07-26T06:47:05.026423+0000","last_active":"2024-07-26T06:48:51.338494+0000","last_peered":"2024-07-26T06:48:51.338494+0000","last_clean":"2024-07-26T06:48:51.338494+0000","last_became_active":"2024-07-26T06:47:05.026053+0000","last_became_peered":"2024-07-26T06:47:05.026053+0000","last_unstale":"2024-07-26T06:48:51.338494+0000","last_undegraded":"2024-07-26T06:48:51.338494+0000","last_fullsized":"2024-07-26T06:48:51.338494+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:07:20.174538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":103,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:29.151093+0000","last_change":"2024-07-26T06:45:40.184988+0000","last_active":"2024-07-26T06:48:29.151093+0000","last_peered":"2024-07-26T06:48:29.151093+0000","last_clean":"2024-07-26T06:48:29.151093+0000","last_became_active":"2024-07-26T06:45:40.178827+0000","last_became_peered":"2024-07-26T06:45:40.178827+0000","last_unstale":"2024-07-26T06:48:29.151093+0000","last_undegraded":"2024-07-26T06:48:29.151093+0000","last_fullsized":"2024-07-26T06:48:29.151093+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:42:28.257602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:42:28.257602+0000","last_clean_scrub_stamp":"2024-07-26T06:42:28.257602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:52:14.123696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.123395+0000","last_change":"2024-07-26T06:47:01.996610+0000","last_active":"2024-07-26T06:47:24.123395+0000","last_peered":"2024-07-26T06:47:24.123395+0000","last_clean":"2024-07-26T06:47:24.123395+0000","last_became_active":"2024-07-26T06:47:01.996155+0000","last_became_peered":"2024-07-26T06:47:01.996155+0000","last_unstale":"2024-07-26T06:47:24.123395+0000","last_undegraded":"2024-07-26T06:47:24.123395+0000","last_fullsized":"2024-07-26T06:47:24.123395+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:51:30.420254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333771+0000","last_change":"2024-07-26T06:47:05.027699+0000","last_active":"2024-07-26T06:48:51.333771+0000","last_peered":"2024-07-26T06:48:51.333771+0000","last_clean":"2024-07-26T06:48:51.333771+0000","last_became_active":"2024-07-26T06:47:05.027348+0000","last_became_peered":"2024-07-26T06:47:05.027348+0000","last_unstale":"2024-07-26T06:48:51.333771+0000","last_undegraded":"2024-07-26T06:48:51.333771+0000","last_fullsized":"2024-07-26T06:48:51.333771+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:28:42.664351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097188+0000","last_change":"2024-07-26T06:47:02.003205+0000","last_active":"2024-07-26T06:47:23.097188+0000","last_peered":"2024-07-26T06:47:23.097188+0000","last_clean":"2024-07-26T06:47:23.097188+0000","last_became_active":"2024-07-26T06:47:02.002543+0000","last_became_peered":"2024-07-26T06:47:02.002543+0000","last_unstale":"2024-07-26T06:47:23.097188+0000","last_undegraded":"2024-07-26T06:47:23.097188+0000","last_fullsized":"2024-07-26T06:47:23.097188+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:43:10.911943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"80'5","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.334595+0000","last_change":"2024-07-26T06:47:05.023258+0000","last_active":"2024-07-26T06:48:51.334595+0000","last_peered":"2024-07-26T06:48:51.334595+0000","last_clean":"2024-07-26T06:48:51.334595+0000","last_became_active":"2024-07-26T06:47:05.022903+0000","last_became_peered":"2024-07-26T06:47:05.022903+0000","last_unstale":"2024-07-26T06:48:51.334595+0000","last_undegraded":"2024-07-26T06:48:51.334595+0000","last_fullsized":"2024-07-26T06:48:51.334595+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:08.155850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"80'6","reported_seq":217,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:25.415768+0000","last_change":"2024-07-26T06:47:02.022316+0000","last_active":"2024-07-26T06:50:25.415768+0000","last_peered":"2024-07-26T06:50:25.415768+0000","last_clean":"2024-07-26T06:50:25.415768+0000","last_became_active":"2024-07-26T06:47:02.021923+0000","last_became_peered":"2024-07-26T06:47:02.021923+0000","last_unstale":"2024-07-26T06:50:25.415768+0000","last_undegraded":"2024-07-26T06:50:25.415768+0000","last_fullsized":"2024-07-26T06:50:25.415768+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:29:31.533449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":184,"num_read_kb":184,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"80'8","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328077+0000","last_change":"2024-07-26T06:47:05.021510+0000","last_active":"2024-07-26T06:48:51.328077+0000","last_peered":"2024-07-26T06:48:51.328077+0000","last_clean":"2024-07-26T06:48:51.328077+0000","last_became_active":"2024-07-26T06:47:05.021140+0000","last_became_peered":"2024-07-26T06:47:05.021140+0000","last_unstale":"2024-07-26T06:48:51.328077+0000","last_undegraded":"2024-07-26T06:48:51.328077+0000","last_fullsized":"2024-07-26T06:48:51.328077+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:37:01.340126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"79'5","reported_seq":218,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:27.429487+0000","last_change":"2024-07-26T06:47:02.001698+0000","last_active":"2024-07-26T06:50:27.429487+0000","last_peered":"2024-07-26T06:50:27.429487+0000","last_clean":"2024-07-26T06:50:27.429487+0000","last_became_active":"2024-07-26T06:47:02.001244+0000","last_became_peered":"2024-07-26T06:47:02.001244+0000","last_unstale":"2024-07-26T06:50:27.429487+0000","last_undegraded":"2024-07-26T06:50:27.429487+0000","last_fullsized":"2024-07-26T06:50:27.429487+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:32:07.920065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.323756+0000","last_change":"2024-07-26T06:47:05.029524+0000","last_active":"2024-07-26T06:48:51.323756+0000","last_peered":"2024-07-26T06:48:51.323756+0000","last_clean":"2024-07-26T06:48:51.323756+0000","last_became_active":"2024-07-26T06:47:05.028981+0000","last_became_peered":"2024-07-26T06:47:05.028981+0000","last_unstale":"2024-07-26T06:48:51.323756+0000","last_undegraded":"2024-07-26T06:48:51.323756+0000","last_fullsized":"2024-07-26T06:48:51.323756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:49:57.019486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092333+0000","last_change":"2024-07-26T06:47:02.025117+0000","last_active":"2024-07-26T06:47:23.092333+0000","last_peered":"2024-07-26T06:47:23.092333+0000","last_clean":"2024-07-26T06:47:23.092333+0000","last_became_active":"2024-07-26T06:47:02.024612+0000","last_became_peered":"2024-07-26T06:47:02.024612+0000","last_unstale":"2024-07-26T06:47:23.092333+0000","last_undegraded":"2024-07-26T06:47:23.092333+0000","last_fullsized":"2024-07-26T06:47:23.092333+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:15:44.999658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'5","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.324532+0000","last_change":"2024-07-26T06:47:05.015085+0000","last_active":"2024-07-26T06:48:51.324532+0000","last_peered":"2024-07-26T06:48:51.324532+0000","last_clean":"2024-07-26T06:48:51.324532+0000","last_became_active":"2024-07-26T06:47:05.014686+0000","last_became_peered":"2024-07-26T06:47:05.014686+0000","last_unstale":"2024-07-26T06:48:51.324532+0000","last_undegraded":"2024-07-26T06:48:51.324532+0000","last_fullsized":"2024-07-26T06:48:51.324532+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:39:30.392347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.120649+0000","last_change":"2024-07-26T06:47:23.111918+0000","last_active":"2024-07-26T06:47:24.120649+0000","last_peered":"2024-07-26T06:47:24.120649+0000","last_clean":"2024-07-26T06:47:24.120649+0000","last_became_active":"2024-07-26T06:47:23.099518+0000","last_became_peered":"2024-07-26T06:47:23.099518+0000","last_unstale":"2024-07-26T06:47:24.120649+0000","last_undegraded":"2024-07-26T06:47:24.120649+0000","last_fullsized":"2024-07-26T06:47:24.120649+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:41.012096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.332524+0000","last_change":"2024-07-26T06:47:05.019415+0000","last_active":"2024-07-26T06:48:51.332524+0000","last_peered":"2024-07-26T06:48:51.332524+0000","last_clean":"2024-07-26T06:48:51.332524+0000","last_became_active":"2024-07-26T06:47:05.019044+0000","last_became_peered":"2024-07-26T06:47:05.019044+0000","last_unstale":"2024-07-26T06:48:51.332524+0000","last_undegraded":"2024-07-26T06:48:51.332524+0000","last_fullsized":"2024-07-26T06:48:51.332524+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:02:37.228174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'16","reported_seq":44,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:13.791758+0000","last_change":"2024-07-26T06:47:02.006585+0000","last_active":"2024-07-26T06:49:13.791758+0000","last_peered":"2024-07-26T06:49:13.791758+0000","last_clean":"2024-07-26T06:49:13.791758+0000","last_became_active":"2024-07-26T06:47:02.006185+0000","last_became_peered":"2024-07-26T06:47:02.006185+0000","last_unstale":"2024-07-26T06:49:13.791758+0000","last_undegraded":"2024-07-26T06:49:13.791758+0000","last_fullsized":"2024-07-26T06:49:13.791758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:02:45.480712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"80'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326954+0000","last_change":"2024-07-26T06:47:05.028840+0000","last_active":"2024-07-26T06:48:51.326954+0000","last_peered":"2024-07-26T06:48:51.326954+0000","last_clean":"2024-07-26T06:48:51.326954+0000","last_became_active":"2024-07-26T06:47:05.028487+0000","last_became_peered":"2024-07-26T06:47:05.028487+0000","last_unstale":"2024-07-26T06:48:51.326954+0000","last_undegraded":"2024-07-26T06:48:51.326954+0000","last_fullsized":"2024-07-26T06:48:51.326954+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:47:48.185227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.119901+0000","last_change":"2024-07-26T06:47:02.113691+0000","last_active":"2024-07-26T06:47:24.119901+0000","last_peered":"2024-07-26T06:47:24.119901+0000","last_clean":"2024-07-26T06:47:24.119901+0000","last_became_active":"2024-07-26T06:47:02.113342+0000","last_became_peered":"2024-07-26T06:47:02.113342+0000","last_unstale":"2024-07-26T06:47:24.119901+0000","last_undegraded":"2024-07-26T06:47:24.119901+0000","last_fullsized":"2024-07-26T06:47:24.119901+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:14:03.586564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"80'7","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.332634+0000","last_change":"2024-07-26T06:47:23.098420+0000","last_active":"2024-07-26T06:48:51.332634+0000","last_peered":"2024-07-26T06:48:51.332634+0000","last_clean":"2024-07-26T06:48:51.332634+0000","last_became_active":"2024-07-26T06:47:23.097981+0000","last_became_peered":"2024-07-26T06:47:23.097981+0000","last_unstale":"2024-07-26T06:48:51.332634+0000","last_undegraded":"2024-07-26T06:48:51.332634+0000","last_fullsized":"2024-07-26T06:48:51.332634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:38:40.155514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091145+0000","last_change":"2024-07-26T06:47:02.024781+0000","last_active":"2024-07-26T06:47:23.091145+0000","last_peered":"2024-07-26T06:47:23.091145+0000","last_clean":"2024-07-26T06:47:23.091145+0000","last_became_active":"2024-07-26T06:47:02.024492+0000","last_became_peered":"2024-07-26T06:47:02.024492+0000","last_unstale":"2024-07-26T06:47:23.091145+0000","last_undegraded":"2024-07-26T06:47:23.091145+0000","last_fullsized":"2024-07-26T06:47:23.091145+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:23:41.323963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"80'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.310434+0000","last_change":"2024-07-26T06:47:05.033052+0000","last_active":"2024-07-26T06:48:51.310434+0000","last_peered":"2024-07-26T06:48:51.310434+0000","last_clean":"2024-07-26T06:48:51.310434+0000","last_became_active":"2024-07-26T06:47:05.032052+0000","last_became_peered":"2024-07-26T06:47:05.032052+0000","last_unstale":"2024-07-26T06:48:51.310434+0000","last_undegraded":"2024-07-26T06:48:51.310434+0000","last_fullsized":"2024-07-26T06:48:51.310434+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:42:04.683586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090616+0000","last_change":"2024-07-26T06:47:01.996097+0000","last_active":"2024-07-26T06:47:23.090616+0000","last_peered":"2024-07-26T06:47:23.090616+0000","last_clean":"2024-07-26T06:47:23.090616+0000","last_became_active":"2024-07-26T06:47:01.995594+0000","last_became_peered":"2024-07-26T06:47:01.995594+0000","last_unstale":"2024-07-26T06:47:23.090616+0000","last_undegraded":"2024-07-26T06:47:23.090616+0000","last_fullsized":"2024-07-26T06:47:23.090616+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:34.969493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.329013+0000","last_change":"2024-07-26T06:47:05.024171+0000","last_active":"2024-07-26T06:48:51.329013+0000","last_peered":"2024-07-26T06:48:51.329013+0000","last_clean":"2024-07-26T06:48:51.329013+0000","last_became_active":"2024-07-26T06:47:05.022888+0000","last_became_peered":"2024-07-26T06:47:05.022888+0000","last_unstale":"2024-07-26T06:48:51.329013+0000","last_undegraded":"2024-07-26T06:48:51.329013+0000","last_fullsized":"2024-07-26T06:48:51.329013+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:31:52.691068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"80'4","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:11.305169+0000","last_change":"2024-07-26T06:47:02.004578+0000","last_active":"2024-07-26T06:49:11.305169+0000","last_peered":"2024-07-26T06:49:11.305169+0000","last_clean":"2024-07-26T06:49:11.305169+0000","last_became_active":"2024-07-26T06:47:02.004231+0000","last_became_peered":"2024-07-26T06:47:02.004231+0000","last_unstale":"2024-07-26T06:49:11.305169+0000","last_undegraded":"2024-07-26T06:49:11.305169+0000","last_fullsized":"2024-07-26T06:49:11.305169+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:43:39.803817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.327862+0000","last_change":"2024-07-26T06:47:05.024232+0000","last_active":"2024-07-26T06:48:51.327862+0000","last_peered":"2024-07-26T06:48:51.327862+0000","last_clean":"2024-07-26T06:48:51.327862+0000","last_became_active":"2024-07-26T06:47:05.023869+0000","last_became_peered":"2024-07-26T06:47:05.023869+0000","last_unstale":"2024-07-26T06:48:51.327862+0000","last_undegraded":"2024-07-26T06:48:51.327862+0000","last_fullsized":"2024-07-26T06:48:51.327862+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:27:01.357311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.123564+0000","last_change":"2024-07-26T06:47:23.105051+0000","last_active":"2024-07-26T06:47:24.123564+0000","last_peered":"2024-07-26T06:47:24.123564+0000","last_clean":"2024-07-26T06:47:24.123564+0000","last_became_active":"2024-07-26T06:47:23.098116+0000","last_became_peered":"2024-07-26T06:47:23.098116+0000","last_unstale":"2024-07-26T06:47:24.123564+0000","last_undegraded":"2024-07-26T06:47:24.123564+0000","last_fullsized":"2024-07-26T06:47:24.123564+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:15:32.991870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"80'5","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338438+0000","last_change":"2024-07-26T06:47:05.026056+0000","last_active":"2024-07-26T06:48:51.338438+0000","last_peered":"2024-07-26T06:48:51.338438+0000","last_clean":"2024-07-26T06:48:51.338438+0000","last_became_active":"2024-07-26T06:47:05.025654+0000","last_became_peered":"2024-07-26T06:47:05.025654+0000","last_unstale":"2024-07-26T06:48:51.338438+0000","last_undegraded":"2024-07-26T06:48:51.338438+0000","last_fullsized":"2024-07-26T06:48:51.338438+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:36:20.886768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"80'15","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338740+0000","last_change":"2024-07-26T06:47:05.026693+0000","last_active":"2024-07-26T06:48:51.338740+0000","last_peered":"2024-07-26T06:48:51.338740+0000","last_clean":"2024-07-26T06:48:51.338740+0000","last_became_active":"2024-07-26T06:47:05.026320+0000","last_became_peered":"2024-07-26T06:47:05.026320+0000","last_unstale":"2024-07-26T06:48:51.338740+0000","last_undegraded":"2024-07-26T06:48:51.338740+0000","last_fullsized":"2024-07-26T06:48:51.338740+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:52:30.776335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"80'5","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.124104+0000","last_change":"2024-07-26T06:47:23.107707+0000","last_active":"2024-07-26T06:47:24.124104+0000","last_peered":"2024-07-26T06:47:24.124104+0000","last_clean":"2024-07-26T06:47:24.124104+0000","last_became_active":"2024-07-26T06:47:23.099429+0000","last_became_peered":"2024-07-26T06:47:23.099429+0000","last_unstale":"2024-07-26T06:47:24.124104+0000","last_undegraded":"2024-07-26T06:47:24.124104+0000","last_fullsized":"2024-07-26T06:47:24.124104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:40:52.856756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.334900+0000","last_change":"2024-07-26T06:47:05.032319+0000","last_active":"2024-07-26T06:48:51.334900+0000","last_peered":"2024-07-26T06:48:51.334900+0000","last_clean":"2024-07-26T06:48:51.334900+0000","last_became_active":"2024-07-26T06:47:05.030305+0000","last_became_peered":"2024-07-26T06:47:05.030305+0000","last_unstale":"2024-07-26T06:48:51.334900+0000","last_undegraded":"2024-07-26T06:48:51.334900+0000","last_fullsized":"2024-07-26T06:48:51.334900+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:34:05.351917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"80'12","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:25.105155+0000","last_change":"2024-07-26T06:47:02.007812+0000","last_active":"2024-07-26T06:47:25.105155+0000","last_peered":"2024-07-26T06:47:25.105155+0000","last_clean":"2024-07-26T06:47:25.105155+0000","last_became_active":"2024-07-26T06:47:02.006965+0000","last_became_peered":"2024-07-26T06:47:02.006965+0000","last_unstale":"2024-07-26T06:47:25.105155+0000","last_undegraded":"2024-07-26T06:47:25.105155+0000","last_fullsized":"2024-07-26T06:47:25.105155+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:53.173273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"80'6","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335825+0000","last_change":"2024-07-26T06:47:05.027900+0000","last_active":"2024-07-26T06:48:51.335825+0000","last_peered":"2024-07-26T06:48:51.335825+0000","last_clean":"2024-07-26T06:48:51.335825+0000","last_became_active":"2024-07-26T06:47:05.024467+0000","last_became_peered":"2024-07-26T06:47:05.024467+0000","last_unstale":"2024-07-26T06:48:51.335825+0000","last_undegraded":"2024-07-26T06:48:51.335825+0000","last_fullsized":"2024-07-26T06:48:51.335825+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:05:00.793487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090879+0000","last_change":"2024-07-26T06:47:02.022863+0000","last_active":"2024-07-26T06:47:23.090879+0000","last_peered":"2024-07-26T06:47:23.090879+0000","last_clean":"2024-07-26T06:47:23.090879+0000","last_became_active":"2024-07-26T06:47:02.022446+0000","last_became_peered":"2024-07-26T06:47:02.022446+0000","last_unstale":"2024-07-26T06:47:23.090879+0000","last_undegraded":"2024-07-26T06:47:23.090879+0000","last_fullsized":"2024-07-26T06:47:23.090879+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:14:48.649099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"80'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333567+0000","last_change":"2024-07-26T06:47:05.023669+0000","last_active":"2024-07-26T06:48:51.333567+0000","last_peered":"2024-07-26T06:48:51.333567+0000","last_clean":"2024-07-26T06:48:51.333567+0000","last_became_active":"2024-07-26T06:47:05.023265+0000","last_became_peered":"2024-07-26T06:47:05.023265+0000","last_unstale":"2024-07-26T06:48:51.333567+0000","last_undegraded":"2024-07-26T06:48:51.333567+0000","last_fullsized":"2024-07-26T06:48:51.333567+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:56:45.278253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":227,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:27.428838+0000","last_change":"2024-07-26T06:47:02.006668+0000","last_active":"2024-07-26T06:50:27.428838+0000","last_peered":"2024-07-26T06:50:27.428838+0000","last_clean":"2024-07-26T06:50:27.428838+0000","last_became_active":"2024-07-26T06:47:02.006287+0000","last_became_peered":"2024-07-26T06:47:02.006287+0000","last_unstale":"2024-07-26T06:50:27.428838+0000","last_undegraded":"2024-07-26T06:50:27.428838+0000","last_fullsized":"2024-07-26T06:50:27.428838+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:28:05.722539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":186,"num_read_kb":186,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70177,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1888,"num_read_kb":1024,"num_write":261,"num_write_kb":149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1634,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":210531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250626,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710777,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":426222,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877907006,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":1079472,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070530,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":425458,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266759,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":880640,"data_stored":425142,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462988,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":946176,"data_stored":1019335,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626513,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36196,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708956,"statfs":{"total":95995035648,"available":95957970944,"internally_reserved":0,"allocated":880640,"data_stored":425175,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822742,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":1018481,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986267,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":487951,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182495,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":897024,"data_stored":425000,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411428,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36208,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708944,"statfs":{"total":95995035648,"available":95957958656,"internally_reserved":0,"allocated":892928,"data_stored":426840,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574953,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":946176,"data_stored":488471,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738476,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":426514,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:50:30.560 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph pg dump --format=json 2024-07-26T06:50:30.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:30 smithi163 ceph-mon[36873]: pgmap v404: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 12 op/s 2024-07-26T06:50:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:30 smithi163 ceph-mon[36873]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:50:30.832 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:30.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:30 smithi042 ceph-mon[32775]: pgmap v404: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 12 op/s 2024-07-26T06:50:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:30 smithi042 ceph-mon[32775]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:50:30.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:30 smithi164 ceph-mon[36970]: pgmap v404: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 12 op/s 2024-07-26T06:50:30.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:30 smithi164 ceph-mon[36970]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:50:32.531 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:32 smithi042 ceph-mon[32775]: pgmap v405: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 511 B/s wr, 9 op/s 2024-07-26T06:50:32.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.530+0000 7fe2ec18b640 1 -- 172.21.15.42:0/1593591578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2e410fc30 msgr2=0x7fe2e4112050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:32.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.530+0000 7fe2ec18b640 1 --2- 172.21.15.42:0/1593591578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2e410fc30 0x7fe2e4112050 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc009ca0 tx=0x7fe2dc02a950 comp rx=0 tx=0).stop 2024-07-26T06:50:32.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.531+0000 7fe2ec18b640 1 -- 172.21.15.42:0/1593591578 shutdown_connections 2024-07-26T06:50:32.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.531+0000 7fe2ec18b640 1 --2- 172.21.15.42:0/1593591578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2e410fc30 0x7fe2e4112050 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:32.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.531+0000 7fe2ec18b640 1 --2- 172.21.15.42:0/1593591578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2e41029d0 0x7fe2e410f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:32.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.531+0000 7fe2ec18b640 1 --2- 172.21.15.42:0/1593591578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe2e4102080 0x7fe2e4102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:32.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.531+0000 7fe2ec18b640 1 -- 172.21.15.42:0/1593591578 >> 172.21.15.42:0/1593591578 conn(0x7fe2e40791a0 msgr2=0x7fe2e40795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:32.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.531+0000 7fe2ec18b640 1 -- 172.21.15.42:0/1593591578 shutdown_connections 2024-07-26T06:50:32.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.531+0000 7fe2ec18b640 1 -- 172.21.15.42:0/1593591578 wait complete. 2024-07-26T06:50:32.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.532+0000 7fe2ec18b640 1 Processor -- start 2024-07-26T06:50:32.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.532+0000 7fe2ec18b640 1 -- start start 2024-07-26T06:50:32.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.532+0000 7fe2ec18b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2e4102080 0x7fe2e419c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:32.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.532+0000 7fe2ec18b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2e41029d0 0x7fe2e419c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:32.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.532+0000 7fe2ec18b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe2e410fc30 0x7fe2e41a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:32.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.532+0000 7fe2ec18b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e41156f0 con 0x7fe2e4102080 2024-07-26T06:50:32.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.532+0000 7fe2ec18b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e4115570 con 0x7fe2e410fc30 2024-07-26T06:50:32.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.532+0000 7fe2ec18b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e4115870 con 0x7fe2e41029d0 2024-07-26T06:50:32.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.532+0000 7fe2e9f27640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2e4102080 0x7fe2e419c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:32.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.532+0000 7fe2e9f27640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2e4102080 0x7fe2e419c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44856/0 (socket says 172.21.15.42:44856) 2024-07-26T06:50:32.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.532+0000 7fe2e9f27640 1 -- 172.21.15.42:0/2021927857 learned_addr learned my addr 172.21.15.42:0/2021927857 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:32.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.533+0000 7fe2e9f27640 1 -- 172.21.15.42:0/2021927857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2e41029d0 msgr2=0x7fe2e419c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:32.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.533+0000 7fe2e9f27640 1 --2- 172.21.15.42:0/2021927857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2e41029d0 0x7fe2e419c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:32.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.533+0000 7fe2e9f27640 1 -- 172.21.15.42:0/2021927857 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe2e410fc30 msgr2=0x7fe2e41a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:32.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.533+0000 7fe2e9f27640 1 --2- 172.21.15.42:0/2021927857 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe2e410fc30 0x7fe2e41a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:32.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.533+0000 7fe2e9f27640 1 -- 172.21.15.42:0/2021927857 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e41a2030 con 0x7fe2e4102080 2024-07-26T06:50:32.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.533+0000 7fe2e9f27640 1 --2- 172.21.15.42:0/2021927857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2e4102080 0x7fe2e419c0c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000d950 tx=0x7fe2e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:32.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.533+0000 7fe2daffd640 1 -- 172.21.15.42:0/2021927857 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0017030 con 0x7fe2e4102080 2024-07-26T06:50:32.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.533+0000 7fe2ec18b640 1 -- 172.21.15.42:0/2021927857 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2e41a2320 con 0x7fe2e4102080 2024-07-26T06:50:32.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.533+0000 7fe2daffd640 1 -- 172.21.15.42:0/2021927857 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2e0004020 con 0x7fe2e4102080 2024-07-26T06:50:32.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.533+0000 7fe2daffd640 1 -- 172.21.15.42:0/2021927857 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e00047a0 con 0x7fe2e4102080 2024-07-26T06:50:32.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.533+0000 7fe2ec18b640 1 -- 172.21.15.42:0/2021927857 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2e41a8a80 con 0x7fe2e4102080 2024-07-26T06:50:32.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.534+0000 7fe2ec18b640 1 -- 172.21.15.42:0/2021927857 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2e406bc90 con 0x7fe2e4102080 2024-07-26T06:50:32.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.536+0000 7fe2daffd640 1 -- 172.21.15.42:0/2021927857 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fe2e0009a30 con 0x7fe2e4102080 2024-07-26T06:50:32.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.537+0000 7fe2daffd640 1 --2- 172.21.15.42:0/2021927857 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe2d00b3bd0 0x7fe2d00b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:32.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.537+0000 7fe2e9726640 1 --2- 172.21.15.42:0/2021927857 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe2d00b3bd0 0x7fe2d00b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:32.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.537+0000 7fe2daffd640 1 -- 172.21.15.42:0/2021927857 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fe2e00e3e20 con 0x7fe2e4102080 2024-07-26T06:50:32.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.538+0000 7fe2e9726640 1 --2- 172.21.15.42:0/2021927857 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe2d00b3bd0 0x7fe2d00b60e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe2e41107f0 tx=0x7fe2d4009590 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:32.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.540+0000 7fe2daffd640 1 -- 172.21.15.42:0/2021927857 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe2e00ab450 con 0x7fe2e4102080 2024-07-26T06:50:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.722+0000 7fe2ec18b640 1 -- 172.21.15.42:0/2021927857 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe2e4109e50 con 0x7fe2d00b3bd0 2024-07-26T06:50:32.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.727+0000 7fe2daffd640 1 -- 172.21.15.42:0/2021927857 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259113 (secure 0 0 0) 0x7fe2e4109e50 con 0x7fe2d00b3bd0 2024-07-26T06:50:32.730 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:50:32.731 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-07-26T06:50:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.729+0000 7fe2ec18b640 1 -- 172.21.15.42:0/2021927857 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe2d00b3bd0 msgr2=0x7fe2d00b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.729+0000 7fe2ec18b640 1 --2- 172.21.15.42:0/2021927857 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe2d00b3bd0 0x7fe2d00b60e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe2e41107f0 tx=0x7fe2d4009590 comp rx=0 tx=0).stop 2024-07-26T06:50:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.729+0000 7fe2ec18b640 1 -- 172.21.15.42:0/2021927857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2e4102080 msgr2=0x7fe2e419c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.729+0000 7fe2ec18b640 1 --2- 172.21.15.42:0/2021927857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2e4102080 0x7fe2e419c0c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000d950 tx=0x7fe2e000de10 comp rx=0 tx=0).stop 2024-07-26T06:50:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.729+0000 7fe2ec18b640 1 -- 172.21.15.42:0/2021927857 shutdown_connections 2024-07-26T06:50:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.729+0000 7fe2ec18b640 1 --2- 172.21.15.42:0/2021927857 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe2d00b3bd0 0x7fe2d00b60e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.729+0000 7fe2ec18b640 1 --2- 172.21.15.42:0/2021927857 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe2e410fc30 0x7fe2e41a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.729+0000 7fe2ec18b640 1 --2- 172.21.15.42:0/2021927857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe2e41029d0 0x7fe2e419c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.729+0000 7fe2ec18b640 1 --2- 172.21.15.42:0/2021927857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2e4102080 0x7fe2e419c0c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.729+0000 7fe2ec18b640 1 -- 172.21.15.42:0/2021927857 >> 172.21.15.42:0/2021927857 conn(0x7fe2e40791a0 msgr2=0x7fe2e4109740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.730+0000 7fe2ec18b640 1 -- 172.21.15.42:0/2021927857 shutdown_connections 2024-07-26T06:50:32.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:32.730+0000 7fe2ec18b640 1 -- 172.21.15.42:0/2021927857 wait complete. 2024-07-26T06:50:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:32 smithi163 ceph-mon[36873]: pgmap v405: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 511 B/s wr, 9 op/s 2024-07-26T06:50:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:32 smithi164 ceph-mon[36970]: pgmap v405: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 511 B/s wr, 9 op/s 2024-07-26T06:50:33.415 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-07-26T06:50:31.372719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660779,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1969,"num_read_kb":1088,"num_write":386,"num_write_kb":2301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2953474,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":817,"ondisk_log_size":817,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384924,"kb_used_data":10812,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124556900,"statfs":{"total":1151940427776,"available":1151546265600,"internally_reserved":0,"allocated":11071488,"data_stored":7074369,"data_compressed":177858,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":52,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003352"},"pg_stats":[{"pgid":"3.3f","version":"80'8","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337456+0000","last_change":"2024-07-26T06:47:05.026186+0000","last_active":"2024-07-26T06:48:51.337456+0000","last_peered":"2024-07-26T06:48:51.337456+0000","last_clean":"2024-07-26T06:48:51.337456+0000","last_became_active":"2024-07-26T06:47:05.025589+0000","last_became_peered":"2024-07-26T06:47:05.025589+0000","last_unstale":"2024-07-26T06:48:51.337456+0000","last_undegraded":"2024-07-26T06:48:51.337456+0000","last_fullsized":"2024-07-26T06:48:51.337456+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:48:23.100966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"80'4","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326008+0000","last_change":"2024-07-26T06:47:05.033211+0000","last_active":"2024-07-26T06:48:51.326008+0000","last_peered":"2024-07-26T06:48:51.326008+0000","last_clean":"2024-07-26T06:48:51.326008+0000","last_became_active":"2024-07-26T06:47:05.032630+0000","last_became_peered":"2024-07-26T06:47:05.032630+0000","last_unstale":"2024-07-26T06:48:51.326008+0000","last_undegraded":"2024-07-26T06:48:51.326008+0000","last_fullsized":"2024-07-26T06:48:51.326008+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:03:19.612963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335969+0000","last_change":"2024-07-26T06:47:05.034767+0000","last_active":"2024-07-26T06:48:51.335969+0000","last_peered":"2024-07-26T06:48:51.335969+0000","last_clean":"2024-07-26T06:48:51.335969+0000","last_became_active":"2024-07-26T06:47:05.034376+0000","last_became_peered":"2024-07-26T06:47:05.034376+0000","last_unstale":"2024-07-26T06:48:51.335969+0000","last_undegraded":"2024-07-26T06:48:51.335969+0000","last_fullsized":"2024-07-26T06:48:51.335969+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:55:55.729930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328614+0000","last_change":"2024-07-26T06:47:05.030995+0000","last_active":"2024-07-26T06:48:51.328614+0000","last_peered":"2024-07-26T06:48:51.328614+0000","last_clean":"2024-07-26T06:48:51.328614+0000","last_became_active":"2024-07-26T06:47:05.025423+0000","last_became_peered":"2024-07-26T06:47:05.025423+0000","last_unstale":"2024-07-26T06:48:51.328614+0000","last_undegraded":"2024-07-26T06:48:51.328614+0000","last_fullsized":"2024-07-26T06:48:51.328614+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:59:39.511383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"80'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.317041+0000","last_change":"2024-07-26T06:47:05.035359+0000","last_active":"2024-07-26T06:48:51.317041+0000","last_peered":"2024-07-26T06:48:51.317041+0000","last_clean":"2024-07-26T06:48:51.317041+0000","last_became_active":"2024-07-26T06:47:05.034836+0000","last_became_peered":"2024-07-26T06:47:05.034836+0000","last_unstale":"2024-07-26T06:48:51.317041+0000","last_undegraded":"2024-07-26T06:48:51.317041+0000","last_fullsized":"2024-07-26T06:48:51.317041+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:54.482774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"80'6","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326306+0000","last_change":"2024-07-26T06:47:05.023660+0000","last_active":"2024-07-26T06:48:51.326306+0000","last_peered":"2024-07-26T06:48:51.326306+0000","last_clean":"2024-07-26T06:48:51.326306+0000","last_became_active":"2024-07-26T06:47:05.022446+0000","last_became_peered":"2024-07-26T06:47:05.022446+0000","last_unstale":"2024-07-26T06:48:51.326306+0000","last_undegraded":"2024-07-26T06:48:51.326306+0000","last_fullsized":"2024-07-26T06:48:51.326306+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:35:00.519226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"80'5","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339509+0000","last_change":"2024-07-26T06:47:05.025132+0000","last_active":"2024-07-26T06:48:51.339509+0000","last_peered":"2024-07-26T06:48:51.339509+0000","last_clean":"2024-07-26T06:48:51.339509+0000","last_became_active":"2024-07-26T06:47:05.024770+0000","last_became_peered":"2024-07-26T06:47:05.024770+0000","last_unstale":"2024-07-26T06:48:51.339509+0000","last_undegraded":"2024-07-26T06:48:51.339509+0000","last_fullsized":"2024-07-26T06:48:51.339509+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:21:39.920152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"80'9","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333625+0000","last_change":"2024-07-26T06:47:05.022613+0000","last_active":"2024-07-26T06:48:51.333625+0000","last_peered":"2024-07-26T06:48:51.333625+0000","last_clean":"2024-07-26T06:48:51.333625+0000","last_became_active":"2024-07-26T06:47:05.022202+0000","last_became_peered":"2024-07-26T06:47:05.022202+0000","last_unstale":"2024-07-26T06:48:51.333625+0000","last_undegraded":"2024-07-26T06:48:51.333625+0000","last_fullsized":"2024-07-26T06:48:51.333625+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:07:23.980843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"80'11","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.336947+0000","last_change":"2024-07-26T06:47:05.026352+0000","last_active":"2024-07-26T06:48:51.336947+0000","last_peered":"2024-07-26T06:48:51.336947+0000","last_clean":"2024-07-26T06:48:51.336947+0000","last_became_active":"2024-07-26T06:47:05.025969+0000","last_became_peered":"2024-07-26T06:47:05.025969+0000","last_unstale":"2024-07-26T06:48:51.336947+0000","last_undegraded":"2024-07-26T06:48:51.336947+0000","last_fullsized":"2024-07-26T06:48:51.336947+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:27.062013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"80'16","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.329437+0000","last_change":"2024-07-26T06:47:23.100422+0000","last_active":"2024-07-26T06:48:51.329437+0000","last_peered":"2024-07-26T06:48:51.329437+0000","last_clean":"2024-07-26T06:48:51.329437+0000","last_became_active":"2024-07-26T06:47:23.100023+0000","last_became_peered":"2024-07-26T06:47:23.100023+0000","last_unstale":"2024-07-26T06:48:51.329437+0000","last_undegraded":"2024-07-26T06:48:51.329437+0000","last_fullsized":"2024-07-26T06:48:51.329437+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:28.158112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"80'6","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338557+0000","last_change":"2024-07-26T06:47:05.037142+0000","last_active":"2024-07-26T06:48:51.338557+0000","last_peered":"2024-07-26T06:48:51.338557+0000","last_clean":"2024-07-26T06:48:51.338557+0000","last_became_active":"2024-07-26T06:47:05.036745+0000","last_became_peered":"2024-07-26T06:47:05.036745+0000","last_unstale":"2024-07-26T06:48:51.338557+0000","last_undegraded":"2024-07-26T06:48:51.338557+0000","last_fullsized":"2024-07-26T06:48:51.338557+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:31:10.034560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"80'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.315686+0000","last_change":"2024-07-26T06:47:05.030640+0000","last_active":"2024-07-26T06:48:51.315686+0000","last_peered":"2024-07-26T06:48:51.315686+0000","last_clean":"2024-07-26T06:48:51.315686+0000","last_became_active":"2024-07-26T06:47:05.023583+0000","last_became_peered":"2024-07-26T06:47:05.023583+0000","last_unstale":"2024-07-26T06:48:51.315686+0000","last_undegraded":"2024-07-26T06:48:51.315686+0000","last_fullsized":"2024-07-26T06:48:51.315686+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:59:12.312930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"80'10","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333194+0000","last_change":"2024-07-26T06:47:05.021768+0000","last_active":"2024-07-26T06:48:51.333194+0000","last_peered":"2024-07-26T06:48:51.333194+0000","last_clean":"2024-07-26T06:48:51.333194+0000","last_became_active":"2024-07-26T06:47:05.021410+0000","last_became_peered":"2024-07-26T06:47:05.021410+0000","last_unstale":"2024-07-26T06:48:51.333194+0000","last_undegraded":"2024-07-26T06:48:51.333194+0000","last_fullsized":"2024-07-26T06:48:51.333194+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:05.576122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.324480+0000","last_change":"2024-07-26T06:47:05.035814+0000","last_active":"2024-07-26T06:48:51.324480+0000","last_peered":"2024-07-26T06:48:51.324480+0000","last_clean":"2024-07-26T06:48:51.324480+0000","last_became_active":"2024-07-26T06:47:05.035451+0000","last_became_peered":"2024-07-26T06:47:05.035451+0000","last_unstale":"2024-07-26T06:48:51.324480+0000","last_undegraded":"2024-07-26T06:48:51.324480+0000","last_fullsized":"2024-07-26T06:48:51.324480+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:33.533726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.331515+0000","last_change":"2024-07-26T06:47:05.035720+0000","last_active":"2024-07-26T06:48:51.331515+0000","last_peered":"2024-07-26T06:48:51.331515+0000","last_clean":"2024-07-26T06:48:51.331515+0000","last_became_active":"2024-07-26T06:47:05.035359+0000","last_became_peered":"2024-07-26T06:47:05.035359+0000","last_unstale":"2024-07-26T06:48:51.331515+0000","last_undegraded":"2024-07-26T06:48:51.331515+0000","last_fullsized":"2024-07-26T06:48:51.331515+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:38.561268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339962+0000","last_change":"2024-07-26T06:47:05.027109+0000","last_active":"2024-07-26T06:48:51.339962+0000","last_peered":"2024-07-26T06:48:51.339962+0000","last_clean":"2024-07-26T06:48:51.339962+0000","last_became_active":"2024-07-26T06:47:05.026694+0000","last_became_peered":"2024-07-26T06:47:05.026694+0000","last_unstale":"2024-07-26T06:48:51.339962+0000","last_undegraded":"2024-07-26T06:48:51.339962+0000","last_fullsized":"2024-07-26T06:48:51.339962+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:25:01.215646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"80'8","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337458+0000","last_change":"2024-07-26T06:47:05.028212+0000","last_active":"2024-07-26T06:48:51.337458+0000","last_peered":"2024-07-26T06:48:51.337458+0000","last_clean":"2024-07-26T06:48:51.337458+0000","last_became_active":"2024-07-26T06:47:05.027598+0000","last_became_peered":"2024-07-26T06:47:05.027598+0000","last_unstale":"2024-07-26T06:48:51.337458+0000","last_undegraded":"2024-07-26T06:48:51.337458+0000","last_fullsized":"2024-07-26T06:48:51.337458+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:55:31.536194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"80'10","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.336116+0000","last_change":"2024-07-26T06:47:05.027724+0000","last_active":"2024-07-26T06:48:51.336116+0000","last_peered":"2024-07-26T06:48:51.336116+0000","last_clean":"2024-07-26T06:48:51.336116+0000","last_became_active":"2024-07-26T06:47:05.027348+0000","last_became_peered":"2024-07-26T06:47:05.027348+0000","last_unstale":"2024-07-26T06:48:51.336116+0000","last_undegraded":"2024-07-26T06:48:51.336116+0000","last_fullsized":"2024-07-26T06:48:51.336116+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:04.180467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"80'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.327682+0000","last_change":"2024-07-26T06:47:05.025474+0000","last_active":"2024-07-26T06:48:51.327682+0000","last_peered":"2024-07-26T06:48:51.327682+0000","last_clean":"2024-07-26T06:48:51.327682+0000","last_became_active":"2024-07-26T06:47:05.025016+0000","last_became_peered":"2024-07-26T06:47:05.025016+0000","last_unstale":"2024-07-26T06:48:51.327682+0000","last_undegraded":"2024-07-26T06:48:51.327682+0000","last_fullsized":"2024-07-26T06:48:51.327682+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:40:03.034532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"80'8","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.330348+0000","last_change":"2024-07-26T06:47:05.027533+0000","last_active":"2024-07-26T06:48:51.330348+0000","last_peered":"2024-07-26T06:48:51.330348+0000","last_clean":"2024-07-26T06:48:51.330348+0000","last_became_active":"2024-07-26T06:47:05.027143+0000","last_became_peered":"2024-07-26T06:47:05.027143+0000","last_unstale":"2024-07-26T06:48:51.330348+0000","last_undegraded":"2024-07-26T06:48:51.330348+0000","last_fullsized":"2024-07-26T06:48:51.330348+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:13.406430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"80'11","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338271+0000","last_change":"2024-07-26T06:47:05.025962+0000","last_active":"2024-07-26T06:48:51.338271+0000","last_peered":"2024-07-26T06:48:51.338271+0000","last_clean":"2024-07-26T06:48:51.338271+0000","last_became_active":"2024-07-26T06:47:05.025621+0000","last_became_peered":"2024-07-26T06:47:05.025621+0000","last_unstale":"2024-07-26T06:48:51.338271+0000","last_undegraded":"2024-07-26T06:48:51.338271+0000","last_fullsized":"2024-07-26T06:48:51.338271+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:56:43.669247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"80'10","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337425+0000","last_change":"2024-07-26T06:47:05.026252+0000","last_active":"2024-07-26T06:48:51.337425+0000","last_peered":"2024-07-26T06:48:51.337425+0000","last_clean":"2024-07-26T06:48:51.337425+0000","last_became_active":"2024-07-26T06:47:05.025875+0000","last_became_peered":"2024-07-26T06:47:05.025875+0000","last_unstale":"2024-07-26T06:48:51.337425+0000","last_undegraded":"2024-07-26T06:48:51.337425+0000","last_fullsized":"2024-07-26T06:48:51.337425+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:27:32.955862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.327073+0000","last_change":"2024-07-26T06:47:05.022513+0000","last_active":"2024-07-26T06:48:51.327073+0000","last_peered":"2024-07-26T06:48:51.327073+0000","last_clean":"2024-07-26T06:48:51.327073+0000","last_became_active":"2024-07-26T06:47:05.022159+0000","last_became_peered":"2024-07-26T06:47:05.022159+0000","last_unstale":"2024-07-26T06:48:51.327073+0000","last_undegraded":"2024-07-26T06:48:51.327073+0000","last_fullsized":"2024-07-26T06:48:51.327073+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:53:06.402985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"80'6","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.325351+0000","last_change":"2024-07-26T06:47:23.097604+0000","last_active":"2024-07-26T06:48:51.325351+0000","last_peered":"2024-07-26T06:48:51.325351+0000","last_clean":"2024-07-26T06:48:51.325351+0000","last_became_active":"2024-07-26T06:47:23.097170+0000","last_became_peered":"2024-07-26T06:47:23.097170+0000","last_unstale":"2024-07-26T06:48:51.325351+0000","last_undegraded":"2024-07-26T06:48:51.325351+0000","last_fullsized":"2024-07-26T06:48:51.325351+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:38:24.530396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"80'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.323758+0000","last_change":"2024-07-26T06:47:05.026349+0000","last_active":"2024-07-26T06:48:51.323758+0000","last_peered":"2024-07-26T06:48:51.323758+0000","last_clean":"2024-07-26T06:48:51.323758+0000","last_became_active":"2024-07-26T06:47:05.026016+0000","last_became_peered":"2024-07-26T06:47:05.026016+0000","last_unstale":"2024-07-26T06:48:51.323758+0000","last_undegraded":"2024-07-26T06:48:51.323758+0000","last_fullsized":"2024-07-26T06:48:51.323758+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:19:02.264183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"80'15","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.323355+0000","last_change":"2024-07-26T06:47:05.032439+0000","last_active":"2024-07-26T06:48:51.323355+0000","last_peered":"2024-07-26T06:48:51.323355+0000","last_clean":"2024-07-26T06:48:51.323355+0000","last_became_active":"2024-07-26T06:47:05.032105+0000","last_became_peered":"2024-07-26T06:47:05.032105+0000","last_unstale":"2024-07-26T06:48:51.323355+0000","last_undegraded":"2024-07-26T06:48:51.323355+0000","last_fullsized":"2024-07-26T06:48:51.323355+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:21:50.423939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"80'6","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.319567+0000","last_change":"2024-07-26T06:47:05.026927+0000","last_active":"2024-07-26T06:48:51.319567+0000","last_peered":"2024-07-26T06:48:51.319567+0000","last_clean":"2024-07-26T06:48:51.319567+0000","last_became_active":"2024-07-26T06:47:05.026569+0000","last_became_peered":"2024-07-26T06:47:05.026569+0000","last_unstale":"2024-07-26T06:48:51.319567+0000","last_undegraded":"2024-07-26T06:48:51.319567+0000","last_fullsized":"2024-07-26T06:48:51.319567+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:12:14.687633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"80'4","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333564+0000","last_change":"2024-07-26T06:47:23.099689+0000","last_active":"2024-07-26T06:48:51.333564+0000","last_peered":"2024-07-26T06:48:51.333564+0000","last_clean":"2024-07-26T06:48:51.333564+0000","last_became_active":"2024-07-26T06:47:23.099313+0000","last_became_peered":"2024-07-26T06:47:23.099313+0000","last_unstale":"2024-07-26T06:48:51.333564+0000","last_undegraded":"2024-07-26T06:48:51.333564+0000","last_fullsized":"2024-07-26T06:48:51.333564+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:49:01.833297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333361+0000","last_change":"2024-07-26T06:47:05.034864+0000","last_active":"2024-07-26T06:48:51.333361+0000","last_peered":"2024-07-26T06:48:51.333361+0000","last_clean":"2024-07-26T06:48:51.333361+0000","last_became_active":"2024-07-26T06:47:05.033414+0000","last_became_peered":"2024-07-26T06:47:05.033414+0000","last_unstale":"2024-07-26T06:48:51.333361+0000","last_undegraded":"2024-07-26T06:48:51.333361+0000","last_fullsized":"2024-07-26T06:48:51.333361+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:21:31.420212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.324409+0000","last_change":"2024-07-26T06:47:05.025462+0000","last_active":"2024-07-26T06:48:51.324409+0000","last_peered":"2024-07-26T06:48:51.324409+0000","last_clean":"2024-07-26T06:48:51.324409+0000","last_became_active":"2024-07-26T06:47:05.025049+0000","last_became_peered":"2024-07-26T06:47:05.025049+0000","last_unstale":"2024-07-26T06:48:51.324409+0000","last_undegraded":"2024-07-26T06:48:51.324409+0000","last_fullsized":"2024-07-26T06:48:51.324409+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:59:28.903338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"80'12","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335500+0000","last_change":"2024-07-26T06:47:05.026394+0000","last_active":"2024-07-26T06:48:51.335500+0000","last_peered":"2024-07-26T06:48:51.335500+0000","last_clean":"2024-07-26T06:48:51.335500+0000","last_became_active":"2024-07-26T06:47:05.025820+0000","last_became_peered":"2024-07-26T06:47:05.025820+0000","last_unstale":"2024-07-26T06:48:51.335500+0000","last_undegraded":"2024-07-26T06:48:51.335500+0000","last_fullsized":"2024-07-26T06:48:51.335500+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:44:57.441638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.329050+0000","last_change":"2024-07-26T06:47:05.024373+0000","last_active":"2024-07-26T06:48:51.329050+0000","last_peered":"2024-07-26T06:48:51.329050+0000","last_clean":"2024-07-26T06:48:51.329050+0000","last_became_active":"2024-07-26T06:47:05.023962+0000","last_became_peered":"2024-07-26T06:47:05.023962+0000","last_unstale":"2024-07-26T06:48:51.329050+0000","last_undegraded":"2024-07-26T06:48:51.329050+0000","last_fullsized":"2024-07-26T06:48:51.329050+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:43.804614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"80'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.325450+0000","last_change":"2024-07-26T06:47:05.031146+0000","last_active":"2024-07-26T06:48:51.325450+0000","last_peered":"2024-07-26T06:48:51.325450+0000","last_clean":"2024-07-26T06:48:51.325450+0000","last_became_active":"2024-07-26T06:47:05.029393+0000","last_became_peered":"2024-07-26T06:47:05.029393+0000","last_unstale":"2024-07-26T06:48:51.325450+0000","last_undegraded":"2024-07-26T06:48:51.325450+0000","last_fullsized":"2024-07-26T06:48:51.325450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:46.853933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"80'8","reported_seq":223,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:26.418121+0000","last_change":"2024-07-26T06:47:02.001800+0000","last_active":"2024-07-26T06:50:26.418121+0000","last_peered":"2024-07-26T06:50:26.418121+0000","last_clean":"2024-07-26T06:50:26.418121+0000","last_became_active":"2024-07-26T06:47:02.001435+0000","last_became_peered":"2024-07-26T06:47:02.001435+0000","last_unstale":"2024-07-26T06:50:26.418121+0000","last_undegraded":"2024-07-26T06:50:26.418121+0000","last_fullsized":"2024-07-26T06:50:26.418121+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:23.121378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1462,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"80'8","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.334283+0000","last_change":"2024-07-26T06:47:05.028301+0000","last_active":"2024-07-26T06:48:51.334283+0000","last_peered":"2024-07-26T06:48:51.334283+0000","last_clean":"2024-07-26T06:48:51.334283+0000","last_became_active":"2024-07-26T06:47:05.027761+0000","last_became_peered":"2024-07-26T06:47:05.027761+0000","last_unstale":"2024-07-26T06:48:51.334283+0000","last_undegraded":"2024-07-26T06:48:51.334283+0000","last_fullsized":"2024-07-26T06:48:51.334283+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:19.109395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093824+0000","last_change":"2024-07-26T06:47:02.004488+0000","last_active":"2024-07-26T06:47:23.093824+0000","last_peered":"2024-07-26T06:47:23.093824+0000","last_clean":"2024-07-26T06:47:23.093824+0000","last_became_active":"2024-07-26T06:47:02.004163+0000","last_became_peered":"2024-07-26T06:47:02.004163+0000","last_unstale":"2024-07-26T06:47:23.093824+0000","last_undegraded":"2024-07-26T06:47:23.093824+0000","last_fullsized":"2024-07-26T06:47:23.093824+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:16.221618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096130+0000","last_change":"2024-07-26T06:47:01.992708+0000","last_active":"2024-07-26T06:47:23.096130+0000","last_peered":"2024-07-26T06:47:23.096130+0000","last_clean":"2024-07-26T06:47:23.096130+0000","last_became_active":"2024-07-26T06:47:01.992330+0000","last_became_peered":"2024-07-26T06:47:01.992330+0000","last_unstale":"2024-07-26T06:47:23.096130+0000","last_undegraded":"2024-07-26T06:47:23.096130+0000","last_fullsized":"2024-07-26T06:47:23.096130+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:25.951946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.330722+0000","last_change":"2024-07-26T06:47:05.013946+0000","last_active":"2024-07-26T06:48:51.330722+0000","last_peered":"2024-07-26T06:48:51.330722+0000","last_clean":"2024-07-26T06:48:51.330722+0000","last_became_active":"2024-07-26T06:47:05.013509+0000","last_became_peered":"2024-07-26T06:47:05.013509+0000","last_unstale":"2024-07-26T06:48:51.330722+0000","last_undegraded":"2024-07-26T06:48:51.330722+0000","last_fullsized":"2024-07-26T06:48:51.330722+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:09.789287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.124184+0000","last_change":"2024-07-26T06:47:01.997617+0000","last_active":"2024-07-26T06:47:24.124184+0000","last_peered":"2024-07-26T06:47:24.124184+0000","last_clean":"2024-07-26T06:47:24.124184+0000","last_became_active":"2024-07-26T06:47:01.997238+0000","last_became_peered":"2024-07-26T06:47:01.997238+0000","last_unstale":"2024-07-26T06:47:24.124184+0000","last_undegraded":"2024-07-26T06:47:24.124184+0000","last_fullsized":"2024-07-26T06:47:24.124184+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:45.596438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"80'11","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328533+0000","last_change":"2024-07-26T06:47:23.099813+0000","last_active":"2024-07-26T06:48:51.328533+0000","last_peered":"2024-07-26T06:48:51.328533+0000","last_clean":"2024-07-26T06:48:51.328533+0000","last_became_active":"2024-07-26T06:47:23.098701+0000","last_became_peered":"2024-07-26T06:47:23.098701+0000","last_unstale":"2024-07-26T06:48:51.328533+0000","last_undegraded":"2024-07-26T06:48:51.328533+0000","last_fullsized":"2024-07-26T06:48:51.328533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:42:04.059637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'6","reported_seq":403,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:28.421232+0000","last_change":"2024-07-26T06:47:01.998508+0000","last_active":"2024-07-26T06:50:28.421232+0000","last_peered":"2024-07-26T06:50:28.421232+0000","last_clean":"2024-07-26T06:50:28.421232+0000","last_became_active":"2024-07-26T06:47:01.998132+0000","last_became_peered":"2024-07-26T06:47:01.998132+0000","last_unstale":"2024-07-26T06:50:28.421232+0000","last_undegraded":"2024-07-26T06:50:28.421232+0000","last_fullsized":"2024-07-26T06:50:28.421232+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:21.588085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":370,"num_read_kb":370,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"80'10","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339164+0000","last_change":"2024-07-26T06:47:05.032704+0000","last_active":"2024-07-26T06:48:51.339164+0000","last_peered":"2024-07-26T06:48:51.339164+0000","last_clean":"2024-07-26T06:48:51.339164+0000","last_became_active":"2024-07-26T06:47:05.031843+0000","last_became_peered":"2024-07-26T06:47:05.031843+0000","last_unstale":"2024-07-26T06:48:51.339164+0000","last_undegraded":"2024-07-26T06:48:51.339164+0000","last_fullsized":"2024-07-26T06:48:51.339164+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:58:24.436233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097471+0000","last_change":"2024-07-26T06:47:02.006266+0000","last_active":"2024-07-26T06:47:23.097471+0000","last_peered":"2024-07-26T06:47:23.097471+0000","last_clean":"2024-07-26T06:47:23.097471+0000","last_became_active":"2024-07-26T06:47:02.005830+0000","last_became_peered":"2024-07-26T06:47:02.005830+0000","last_unstale":"2024-07-26T06:47:23.097471+0000","last_undegraded":"2024-07-26T06:47:23.097471+0000","last_fullsized":"2024-07-26T06:47:23.097471+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:06:09.097781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.331471+0000","last_change":"2024-07-26T06:47:05.024820+0000","last_active":"2024-07-26T06:48:51.331471+0000","last_peered":"2024-07-26T06:48:51.331471+0000","last_clean":"2024-07-26T06:48:51.331471+0000","last_became_active":"2024-07-26T06:47:05.023941+0000","last_became_peered":"2024-07-26T06:47:05.023941+0000","last_unstale":"2024-07-26T06:48:51.331471+0000","last_undegraded":"2024-07-26T06:48:51.331471+0000","last_fullsized":"2024-07-26T06:48:51.331471+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:07.885159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.123256+0000","last_change":"2024-07-26T06:47:02.023308+0000","last_active":"2024-07-26T06:47:24.123256+0000","last_peered":"2024-07-26T06:47:24.123256+0000","last_clean":"2024-07-26T06:47:24.123256+0000","last_became_active":"2024-07-26T06:47:02.022933+0000","last_became_peered":"2024-07-26T06:47:02.022933+0000","last_unstale":"2024-07-26T06:47:24.123256+0000","last_undegraded":"2024-07-26T06:47:24.123256+0000","last_fullsized":"2024-07-26T06:47:24.123256+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:59:01.233386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"80'11","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.340511+0000","last_change":"2024-07-26T06:47:05.026005+0000","last_active":"2024-07-26T06:48:51.340511+0000","last_peered":"2024-07-26T06:48:51.340511+0000","last_clean":"2024-07-26T06:48:51.340511+0000","last_became_active":"2024-07-26T06:47:05.025599+0000","last_became_peered":"2024-07-26T06:47:05.025599+0000","last_unstale":"2024-07-26T06:48:51.340511+0000","last_undegraded":"2024-07-26T06:48:51.340511+0000","last_fullsized":"2024-07-26T06:48:51.340511+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:44:19.924157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'36","reported_seq":463,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:28.421272+0000","last_change":"2024-07-26T06:47:23.393050+0000","last_active":"2024-07-26T06:50:28.421272+0000","last_peered":"2024-07-26T06:50:28.421272+0000","last_clean":"2024-07-26T06:50:28.421272+0000","last_became_active":"2024-07-26T06:47:23.381611+0000","last_became_peered":"2024-07-26T06:47:23.381611+0000","last_unstale":"2024-07-26T06:50:28.421272+0000","last_undegraded":"2024-07-26T06:50:28.421272+0000","last_fullsized":"2024-07-26T06:50:28.421272+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:23.866523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62383,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":381,"num_read_kb":251,"num_write":38,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"80'6","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335684+0000","last_change":"2024-07-26T06:47:05.034190+0000","last_active":"2024-07-26T06:48:51.335684+0000","last_peered":"2024-07-26T06:48:51.335684+0000","last_clean":"2024-07-26T06:48:51.335684+0000","last_became_active":"2024-07-26T06:47:05.033569+0000","last_became_peered":"2024-07-26T06:47:05.033569+0000","last_unstale":"2024-07-26T06:48:51.335684+0000","last_undegraded":"2024-07-26T06:48:51.335684+0000","last_fullsized":"2024-07-26T06:48:51.335684+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:39.306996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.123635+0000","last_change":"2024-07-26T06:47:02.001718+0000","last_active":"2024-07-26T06:47:24.123635+0000","last_peered":"2024-07-26T06:47:24.123635+0000","last_clean":"2024-07-26T06:47:24.123635+0000","last_became_active":"2024-07-26T06:47:02.001359+0000","last_became_peered":"2024-07-26T06:47:02.001359+0000","last_unstale":"2024-07-26T06:47:24.123635+0000","last_undegraded":"2024-07-26T06:47:24.123635+0000","last_fullsized":"2024-07-26T06:47:24.123635+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:03:23.390279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337843+0000","last_change":"2024-07-26T06:47:05.027445+0000","last_active":"2024-07-26T06:48:51.337843+0000","last_peered":"2024-07-26T06:48:51.337843+0000","last_clean":"2024-07-26T06:48:51.337843+0000","last_became_active":"2024-07-26T06:47:05.027114+0000","last_became_peered":"2024-07-26T06:47:05.027114+0000","last_unstale":"2024-07-26T06:48:51.337843+0000","last_undegraded":"2024-07-26T06:48:51.337843+0000","last_fullsized":"2024-07-26T06:48:51.337843+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:29:10.519271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.119909+0000","last_change":"2024-07-26T06:47:01.998832+0000","last_active":"2024-07-26T06:47:24.119909+0000","last_peered":"2024-07-26T06:47:24.119909+0000","last_clean":"2024-07-26T06:47:24.119909+0000","last_became_active":"2024-07-26T06:47:01.998512+0000","last_became_peered":"2024-07-26T06:47:01.998512+0000","last_unstale":"2024-07-26T06:47:24.119909+0000","last_undegraded":"2024-07-26T06:47:24.119909+0000","last_fullsized":"2024-07-26T06:47:24.119909+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:27.722329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"80'15","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.340182+0000","last_change":"2024-07-26T06:47:05.027803+0000","last_active":"2024-07-26T06:48:51.340182+0000","last_peered":"2024-07-26T06:48:51.340182+0000","last_clean":"2024-07-26T06:48:51.340182+0000","last_became_active":"2024-07-26T06:47:05.027320+0000","last_became_peered":"2024-07-26T06:47:05.027320+0000","last_unstale":"2024-07-26T06:48:51.340182+0000","last_undegraded":"2024-07-26T06:48:51.340182+0000","last_fullsized":"2024-07-26T06:48:51.340182+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:43:06.242945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'7","reported_seq":223,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:26.417956+0000","last_change":"2024-07-26T06:47:01.998356+0000","last_active":"2024-07-26T06:50:26.417956+0000","last_peered":"2024-07-26T06:50:26.417956+0000","last_clean":"2024-07-26T06:50:26.417956+0000","last_became_active":"2024-07-26T06:47:01.997922+0000","last_became_peered":"2024-07-26T06:47:01.997922+0000","last_unstale":"2024-07-26T06:50:26.417956+0000","last_undegraded":"2024-07-26T06:50:26.417956+0000","last_fullsized":"2024-07-26T06:50:26.417956+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:50:58.066682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":189,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"80'6","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339434+0000","last_change":"2024-07-26T06:47:05.025285+0000","last_active":"2024-07-26T06:48:51.339434+0000","last_peered":"2024-07-26T06:48:51.339434+0000","last_clean":"2024-07-26T06:48:51.339434+0000","last_became_active":"2024-07-26T06:47:05.024945+0000","last_became_peered":"2024-07-26T06:47:05.024945+0000","last_unstale":"2024-07-26T06:48:51.339434+0000","last_undegraded":"2024-07-26T06:48:51.339434+0000","last_fullsized":"2024-07-26T06:48:51.339434+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:21:04.680435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.124278+0000","last_change":"2024-07-26T06:47:02.024895+0000","last_active":"2024-07-26T06:47:24.124278+0000","last_peered":"2024-07-26T06:47:24.124278+0000","last_clean":"2024-07-26T06:47:24.124278+0000","last_became_active":"2024-07-26T06:47:02.024451+0000","last_became_peered":"2024-07-26T06:47:02.024451+0000","last_unstale":"2024-07-26T06:47:24.124278+0000","last_undegraded":"2024-07-26T06:47:24.124278+0000","last_fullsized":"2024-07-26T06:47:24.124278+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:05:13.396499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"80'6","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335788+0000","last_change":"2024-07-26T06:47:23.381060+0000","last_active":"2024-07-26T06:48:51.335788+0000","last_peered":"2024-07-26T06:48:51.335788+0000","last_clean":"2024-07-26T06:48:51.335788+0000","last_became_active":"2024-07-26T06:47:23.380685+0000","last_became_peered":"2024-07-26T06:47:23.380685+0000","last_unstale":"2024-07-26T06:48:51.335788+0000","last_undegraded":"2024-07-26T06:48:51.335788+0000","last_fullsized":"2024-07-26T06:48:51.335788+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:21:38.062900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.124080+0000","last_change":"2024-07-26T06:47:02.002098+0000","last_active":"2024-07-26T06:47:24.124080+0000","last_peered":"2024-07-26T06:47:24.124080+0000","last_clean":"2024-07-26T06:47:24.124080+0000","last_became_active":"2024-07-26T06:47:02.001428+0000","last_became_peered":"2024-07-26T06:47:02.001428+0000","last_unstale":"2024-07-26T06:47:24.124080+0000","last_undegraded":"2024-07-26T06:47:24.124080+0000","last_fullsized":"2024-07-26T06:47:24.124080+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:52:29.112500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"80'5","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.330074+0000","last_change":"2024-07-26T06:47:05.023686+0000","last_active":"2024-07-26T06:48:51.330074+0000","last_peered":"2024-07-26T06:48:51.330074+0000","last_clean":"2024-07-26T06:48:51.330074+0000","last_became_active":"2024-07-26T06:47:05.022583+0000","last_became_peered":"2024-07-26T06:47:05.022583+0000","last_unstale":"2024-07-26T06:48:51.330074+0000","last_undegraded":"2024-07-26T06:48:51.330074+0000","last_fullsized":"2024-07-26T06:48:51.330074+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:29:03.752429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096491+0000","last_change":"2024-07-26T06:47:02.006040+0000","last_active":"2024-07-26T06:47:23.096491+0000","last_peered":"2024-07-26T06:47:23.096491+0000","last_clean":"2024-07-26T06:47:23.096491+0000","last_became_active":"2024-07-26T06:47:02.005586+0000","last_became_peered":"2024-07-26T06:47:02.005586+0000","last_unstale":"2024-07-26T06:47:23.096491+0000","last_undegraded":"2024-07-26T06:47:23.096491+0000","last_fullsized":"2024-07-26T06:47:23.096491+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:09:27.180873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335338+0000","last_change":"2024-07-26T06:47:05.035524+0000","last_active":"2024-07-26T06:48:51.335338+0000","last_peered":"2024-07-26T06:48:51.335338+0000","last_clean":"2024-07-26T06:48:51.335338+0000","last_became_active":"2024-07-26T06:47:05.035078+0000","last_became_peered":"2024-07-26T06:47:05.035078+0000","last_unstale":"2024-07-26T06:48:51.335338+0000","last_undegraded":"2024-07-26T06:48:51.335338+0000","last_fullsized":"2024-07-26T06:48:51.335338+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:03.758261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.123534+0000","last_change":"2024-07-26T06:47:01.999810+0000","last_active":"2024-07-26T06:47:24.123534+0000","last_peered":"2024-07-26T06:47:24.123534+0000","last_clean":"2024-07-26T06:47:24.123534+0000","last_became_active":"2024-07-26T06:47:01.999460+0000","last_became_peered":"2024-07-26T06:47:01.999460+0000","last_unstale":"2024-07-26T06:47:24.123534+0000","last_undegraded":"2024-07-26T06:47:24.123534+0000","last_fullsized":"2024-07-26T06:47:24.123534+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:25:05.691900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339690+0000","last_change":"2024-07-26T06:47:05.033631+0000","last_active":"2024-07-26T06:48:51.339690+0000","last_peered":"2024-07-26T06:48:51.339690+0000","last_clean":"2024-07-26T06:48:51.339690+0000","last_became_active":"2024-07-26T06:47:05.033266+0000","last_became_peered":"2024-07-26T06:47:05.033266+0000","last_unstale":"2024-07-26T06:48:51.339690+0000","last_undegraded":"2024-07-26T06:48:51.339690+0000","last_fullsized":"2024-07-26T06:48:51.339690+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:55.895271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"78'5","reported_seq":222,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:29.444901+0000","last_change":"2024-07-26T06:47:02.113525+0000","last_active":"2024-07-26T06:50:29.444901+0000","last_peered":"2024-07-26T06:50:29.444901+0000","last_clean":"2024-07-26T06:50:29.444901+0000","last_became_active":"2024-07-26T06:47:02.113112+0000","last_became_peered":"2024-07-26T06:47:02.113112+0000","last_unstale":"2024-07-26T06:50:29.444901+0000","last_undegraded":"2024-07-26T06:50:29.444901+0000","last_fullsized":"2024-07-26T06:50:29.444901+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:05:15.071264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":189,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338494+0000","last_change":"2024-07-26T06:47:05.026423+0000","last_active":"2024-07-26T06:48:51.338494+0000","last_peered":"2024-07-26T06:48:51.338494+0000","last_clean":"2024-07-26T06:48:51.338494+0000","last_became_active":"2024-07-26T06:47:05.026053+0000","last_became_peered":"2024-07-26T06:47:05.026053+0000","last_unstale":"2024-07-26T06:48:51.338494+0000","last_undegraded":"2024-07-26T06:48:51.338494+0000","last_fullsized":"2024-07-26T06:48:51.338494+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:07:20.174538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":103,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:29.151093+0000","last_change":"2024-07-26T06:45:40.184988+0000","last_active":"2024-07-26T06:48:29.151093+0000","last_peered":"2024-07-26T06:48:29.151093+0000","last_clean":"2024-07-26T06:48:29.151093+0000","last_became_active":"2024-07-26T06:45:40.178827+0000","last_became_peered":"2024-07-26T06:45:40.178827+0000","last_unstale":"2024-07-26T06:48:29.151093+0000","last_undegraded":"2024-07-26T06:48:29.151093+0000","last_fullsized":"2024-07-26T06:48:29.151093+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:42:28.257602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:42:28.257602+0000","last_clean_scrub_stamp":"2024-07-26T06:42:28.257602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:52:14.123696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.123395+0000","last_change":"2024-07-26T06:47:01.996610+0000","last_active":"2024-07-26T06:47:24.123395+0000","last_peered":"2024-07-26T06:47:24.123395+0000","last_clean":"2024-07-26T06:47:24.123395+0000","last_became_active":"2024-07-26T06:47:01.996155+0000","last_became_peered":"2024-07-26T06:47:01.996155+0000","last_unstale":"2024-07-26T06:47:24.123395+0000","last_undegraded":"2024-07-26T06:47:24.123395+0000","last_fullsized":"2024-07-26T06:47:24.123395+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:51:30.420254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333771+0000","last_change":"2024-07-26T06:47:05.027699+0000","last_active":"2024-07-26T06:48:51.333771+0000","last_peered":"2024-07-26T06:48:51.333771+0000","last_clean":"2024-07-26T06:48:51.333771+0000","last_became_active":"2024-07-26T06:47:05.027348+0000","last_became_peered":"2024-07-26T06:47:05.027348+0000","last_unstale":"2024-07-26T06:48:51.333771+0000","last_undegraded":"2024-07-26T06:48:51.333771+0000","last_fullsized":"2024-07-26T06:48:51.333771+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:28:42.664351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097188+0000","last_change":"2024-07-26T06:47:02.003205+0000","last_active":"2024-07-26T06:47:23.097188+0000","last_peered":"2024-07-26T06:47:23.097188+0000","last_clean":"2024-07-26T06:47:23.097188+0000","last_became_active":"2024-07-26T06:47:02.002543+0000","last_became_peered":"2024-07-26T06:47:02.002543+0000","last_unstale":"2024-07-26T06:47:23.097188+0000","last_undegraded":"2024-07-26T06:47:23.097188+0000","last_fullsized":"2024-07-26T06:47:23.097188+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:43:10.911943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"80'5","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.334595+0000","last_change":"2024-07-26T06:47:05.023258+0000","last_active":"2024-07-26T06:48:51.334595+0000","last_peered":"2024-07-26T06:48:51.334595+0000","last_clean":"2024-07-26T06:48:51.334595+0000","last_became_active":"2024-07-26T06:47:05.022903+0000","last_became_peered":"2024-07-26T06:47:05.022903+0000","last_unstale":"2024-07-26T06:48:51.334595+0000","last_undegraded":"2024-07-26T06:48:51.334595+0000","last_fullsized":"2024-07-26T06:48:51.334595+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:08.155850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"80'6","reported_seq":217,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:25.415768+0000","last_change":"2024-07-26T06:47:02.022316+0000","last_active":"2024-07-26T06:50:25.415768+0000","last_peered":"2024-07-26T06:50:25.415768+0000","last_clean":"2024-07-26T06:50:25.415768+0000","last_became_active":"2024-07-26T06:47:02.021923+0000","last_became_peered":"2024-07-26T06:47:02.021923+0000","last_unstale":"2024-07-26T06:50:25.415768+0000","last_undegraded":"2024-07-26T06:50:25.415768+0000","last_fullsized":"2024-07-26T06:50:25.415768+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:29:31.533449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":184,"num_read_kb":184,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"80'8","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328077+0000","last_change":"2024-07-26T06:47:05.021510+0000","last_active":"2024-07-26T06:48:51.328077+0000","last_peered":"2024-07-26T06:48:51.328077+0000","last_clean":"2024-07-26T06:48:51.328077+0000","last_became_active":"2024-07-26T06:47:05.021140+0000","last_became_peered":"2024-07-26T06:47:05.021140+0000","last_unstale":"2024-07-26T06:48:51.328077+0000","last_undegraded":"2024-07-26T06:48:51.328077+0000","last_fullsized":"2024-07-26T06:48:51.328077+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:37:01.340126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"79'5","reported_seq":218,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:27.429487+0000","last_change":"2024-07-26T06:47:02.001698+0000","last_active":"2024-07-26T06:50:27.429487+0000","last_peered":"2024-07-26T06:50:27.429487+0000","last_clean":"2024-07-26T06:50:27.429487+0000","last_became_active":"2024-07-26T06:47:02.001244+0000","last_became_peered":"2024-07-26T06:47:02.001244+0000","last_unstale":"2024-07-26T06:50:27.429487+0000","last_undegraded":"2024-07-26T06:50:27.429487+0000","last_fullsized":"2024-07-26T06:50:27.429487+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:32:07.920065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.323756+0000","last_change":"2024-07-26T06:47:05.029524+0000","last_active":"2024-07-26T06:48:51.323756+0000","last_peered":"2024-07-26T06:48:51.323756+0000","last_clean":"2024-07-26T06:48:51.323756+0000","last_became_active":"2024-07-26T06:47:05.028981+0000","last_became_peered":"2024-07-26T06:47:05.028981+0000","last_unstale":"2024-07-26T06:48:51.323756+0000","last_undegraded":"2024-07-26T06:48:51.323756+0000","last_fullsized":"2024-07-26T06:48:51.323756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:49:57.019486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092333+0000","last_change":"2024-07-26T06:47:02.025117+0000","last_active":"2024-07-26T06:47:23.092333+0000","last_peered":"2024-07-26T06:47:23.092333+0000","last_clean":"2024-07-26T06:47:23.092333+0000","last_became_active":"2024-07-26T06:47:02.024612+0000","last_became_peered":"2024-07-26T06:47:02.024612+0000","last_unstale":"2024-07-26T06:47:23.092333+0000","last_undegraded":"2024-07-26T06:47:23.092333+0000","last_fullsized":"2024-07-26T06:47:23.092333+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:15:44.999658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'5","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.324532+0000","last_change":"2024-07-26T06:47:05.015085+0000","last_active":"2024-07-26T06:48:51.324532+0000","last_peered":"2024-07-26T06:48:51.324532+0000","last_clean":"2024-07-26T06:48:51.324532+0000","last_became_active":"2024-07-26T06:47:05.014686+0000","last_became_peered":"2024-07-26T06:47:05.014686+0000","last_unstale":"2024-07-26T06:48:51.324532+0000","last_undegraded":"2024-07-26T06:48:51.324532+0000","last_fullsized":"2024-07-26T06:48:51.324532+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:39:30.392347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.120649+0000","last_change":"2024-07-26T06:47:23.111918+0000","last_active":"2024-07-26T06:47:24.120649+0000","last_peered":"2024-07-26T06:47:24.120649+0000","last_clean":"2024-07-26T06:47:24.120649+0000","last_became_active":"2024-07-26T06:47:23.099518+0000","last_became_peered":"2024-07-26T06:47:23.099518+0000","last_unstale":"2024-07-26T06:47:24.120649+0000","last_undegraded":"2024-07-26T06:47:24.120649+0000","last_fullsized":"2024-07-26T06:47:24.120649+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:41.012096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.332524+0000","last_change":"2024-07-26T06:47:05.019415+0000","last_active":"2024-07-26T06:48:51.332524+0000","last_peered":"2024-07-26T06:48:51.332524+0000","last_clean":"2024-07-26T06:48:51.332524+0000","last_became_active":"2024-07-26T06:47:05.019044+0000","last_became_peered":"2024-07-26T06:47:05.019044+0000","last_unstale":"2024-07-26T06:48:51.332524+0000","last_undegraded":"2024-07-26T06:48:51.332524+0000","last_fullsized":"2024-07-26T06:48:51.332524+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:02:37.228174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'16","reported_seq":44,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:13.791758+0000","last_change":"2024-07-26T06:47:02.006585+0000","last_active":"2024-07-26T06:49:13.791758+0000","last_peered":"2024-07-26T06:49:13.791758+0000","last_clean":"2024-07-26T06:49:13.791758+0000","last_became_active":"2024-07-26T06:47:02.006185+0000","last_became_peered":"2024-07-26T06:47:02.006185+0000","last_unstale":"2024-07-26T06:49:13.791758+0000","last_undegraded":"2024-07-26T06:49:13.791758+0000","last_fullsized":"2024-07-26T06:49:13.791758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:02:45.480712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"80'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326954+0000","last_change":"2024-07-26T06:47:05.028840+0000","last_active":"2024-07-26T06:48:51.326954+0000","last_peered":"2024-07-26T06:48:51.326954+0000","last_clean":"2024-07-26T06:48:51.326954+0000","last_became_active":"2024-07-26T06:47:05.028487+0000","last_became_peered":"2024-07-26T06:47:05.028487+0000","last_unstale":"2024-07-26T06:48:51.326954+0000","last_undegraded":"2024-07-26T06:48:51.326954+0000","last_fullsized":"2024-07-26T06:48:51.326954+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:47:48.185227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.119901+0000","last_change":"2024-07-26T06:47:02.113691+0000","last_active":"2024-07-26T06:47:24.119901+0000","last_peered":"2024-07-26T06:47:24.119901+0000","last_clean":"2024-07-26T06:47:24.119901+0000","last_became_active":"2024-07-26T06:47:02.113342+0000","last_became_peered":"2024-07-26T06:47:02.113342+0000","last_unstale":"2024-07-26T06:47:24.119901+0000","last_undegraded":"2024-07-26T06:47:24.119901+0000","last_fullsized":"2024-07-26T06:47:24.119901+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:14:03.586564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"80'7","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.332634+0000","last_change":"2024-07-26T06:47:23.098420+0000","last_active":"2024-07-26T06:48:51.332634+0000","last_peered":"2024-07-26T06:48:51.332634+0000","last_clean":"2024-07-26T06:48:51.332634+0000","last_became_active":"2024-07-26T06:47:23.097981+0000","last_became_peered":"2024-07-26T06:47:23.097981+0000","last_unstale":"2024-07-26T06:48:51.332634+0000","last_undegraded":"2024-07-26T06:48:51.332634+0000","last_fullsized":"2024-07-26T06:48:51.332634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:38:40.155514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091145+0000","last_change":"2024-07-26T06:47:02.024781+0000","last_active":"2024-07-26T06:47:23.091145+0000","last_peered":"2024-07-26T06:47:23.091145+0000","last_clean":"2024-07-26T06:47:23.091145+0000","last_became_active":"2024-07-26T06:47:02.024492+0000","last_became_peered":"2024-07-26T06:47:02.024492+0000","last_unstale":"2024-07-26T06:47:23.091145+0000","last_undegraded":"2024-07-26T06:47:23.091145+0000","last_fullsized":"2024-07-26T06:47:23.091145+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:23:41.323963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"80'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.310434+0000","last_change":"2024-07-26T06:47:05.033052+0000","last_active":"2024-07-26T06:48:51.310434+0000","last_peered":"2024-07-26T06:48:51.310434+0000","last_clean":"2024-07-26T06:48:51.310434+0000","last_became_active":"2024-07-26T06:47:05.032052+0000","last_became_peered":"2024-07-26T06:47:05.032052+0000","last_unstale":"2024-07-26T06:48:51.310434+0000","last_undegraded":"2024-07-26T06:48:51.310434+0000","last_fullsized":"2024-07-26T06:48:51.310434+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:42:04.683586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090616+0000","last_change":"2024-07-26T06:47:01.996097+0000","last_active":"2024-07-26T06:47:23.090616+0000","last_peered":"2024-07-26T06:47:23.090616+0000","last_clean":"2024-07-26T06:47:23.090616+0000","last_became_active":"2024-07-26T06:47:01.995594+0000","last_became_peered":"2024-07-26T06:47:01.995594+0000","last_unstale":"2024-07-26T06:47:23.090616+0000","last_undegraded":"2024-07-26T06:47:23.090616+0000","last_fullsized":"2024-07-26T06:47:23.090616+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:34.969493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.329013+0000","last_change":"2024-07-26T06:47:05.024171+0000","last_active":"2024-07-26T06:48:51.329013+0000","last_peered":"2024-07-26T06:48:51.329013+0000","last_clean":"2024-07-26T06:48:51.329013+0000","last_became_active":"2024-07-26T06:47:05.022888+0000","last_became_peered":"2024-07-26T06:47:05.022888+0000","last_unstale":"2024-07-26T06:48:51.329013+0000","last_undegraded":"2024-07-26T06:48:51.329013+0000","last_fullsized":"2024-07-26T06:48:51.329013+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:31:52.691068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"80'4","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:11.305169+0000","last_change":"2024-07-26T06:47:02.004578+0000","last_active":"2024-07-26T06:49:11.305169+0000","last_peered":"2024-07-26T06:49:11.305169+0000","last_clean":"2024-07-26T06:49:11.305169+0000","last_became_active":"2024-07-26T06:47:02.004231+0000","last_became_peered":"2024-07-26T06:47:02.004231+0000","last_unstale":"2024-07-26T06:49:11.305169+0000","last_undegraded":"2024-07-26T06:49:11.305169+0000","last_fullsized":"2024-07-26T06:49:11.305169+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:43:39.803817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.327862+0000","last_change":"2024-07-26T06:47:05.024232+0000","last_active":"2024-07-26T06:48:51.327862+0000","last_peered":"2024-07-26T06:48:51.327862+0000","last_clean":"2024-07-26T06:48:51.327862+0000","last_became_active":"2024-07-26T06:47:05.023869+0000","last_became_peered":"2024-07-26T06:47:05.023869+0000","last_unstale":"2024-07-26T06:48:51.327862+0000","last_undegraded":"2024-07-26T06:48:51.327862+0000","last_fullsized":"2024-07-26T06:48:51.327862+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:27:01.357311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.123564+0000","last_change":"2024-07-26T06:47:23.105051+0000","last_active":"2024-07-26T06:47:24.123564+0000","last_peered":"2024-07-26T06:47:24.123564+0000","last_clean":"2024-07-26T06:47:24.123564+0000","last_became_active":"2024-07-26T06:47:23.098116+0000","last_became_peered":"2024-07-26T06:47:23.098116+0000","last_unstale":"2024-07-26T06:47:24.123564+0000","last_undegraded":"2024-07-26T06:47:24.123564+0000","last_fullsized":"2024-07-26T06:47:24.123564+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:15:32.991870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"80'5","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338438+0000","last_change":"2024-07-26T06:47:05.026056+0000","last_active":"2024-07-26T06:48:51.338438+0000","last_peered":"2024-07-26T06:48:51.338438+0000","last_clean":"2024-07-26T06:48:51.338438+0000","last_became_active":"2024-07-26T06:47:05.025654+0000","last_became_peered":"2024-07-26T06:47:05.025654+0000","last_unstale":"2024-07-26T06:48:51.338438+0000","last_undegraded":"2024-07-26T06:48:51.338438+0000","last_fullsized":"2024-07-26T06:48:51.338438+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:36:20.886768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"80'15","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338740+0000","last_change":"2024-07-26T06:47:05.026693+0000","last_active":"2024-07-26T06:48:51.338740+0000","last_peered":"2024-07-26T06:48:51.338740+0000","last_clean":"2024-07-26T06:48:51.338740+0000","last_became_active":"2024-07-26T06:47:05.026320+0000","last_became_peered":"2024-07-26T06:47:05.026320+0000","last_unstale":"2024-07-26T06:48:51.338740+0000","last_undegraded":"2024-07-26T06:48:51.338740+0000","last_fullsized":"2024-07-26T06:48:51.338740+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:52:30.776335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"80'5","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.124104+0000","last_change":"2024-07-26T06:47:23.107707+0000","last_active":"2024-07-26T06:47:24.124104+0000","last_peered":"2024-07-26T06:47:24.124104+0000","last_clean":"2024-07-26T06:47:24.124104+0000","last_became_active":"2024-07-26T06:47:23.099429+0000","last_became_peered":"2024-07-26T06:47:23.099429+0000","last_unstale":"2024-07-26T06:47:24.124104+0000","last_undegraded":"2024-07-26T06:47:24.124104+0000","last_fullsized":"2024-07-26T06:47:24.124104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:40:52.856756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.334900+0000","last_change":"2024-07-26T06:47:05.032319+0000","last_active":"2024-07-26T06:48:51.334900+0000","last_peered":"2024-07-26T06:48:51.334900+0000","last_clean":"2024-07-26T06:48:51.334900+0000","last_became_active":"2024-07-26T06:47:05.030305+0000","last_became_peered":"2024-07-26T06:47:05.030305+0000","last_unstale":"2024-07-26T06:48:51.334900+0000","last_undegraded":"2024-07-26T06:48:51.334900+0000","last_fullsized":"2024-07-26T06:48:51.334900+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:34:05.351917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"80'12","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:25.105155+0000","last_change":"2024-07-26T06:47:02.007812+0000","last_active":"2024-07-26T06:47:25.105155+0000","last_peered":"2024-07-26T06:47:25.105155+0000","last_clean":"2024-07-26T06:47:25.105155+0000","last_became_active":"2024-07-26T06:47:02.006965+0000","last_became_peered":"2024-07-26T06:47:02.006965+0000","last_unstale":"2024-07-26T06:47:25.105155+0000","last_undegraded":"2024-07-26T06:47:25.105155+0000","last_fullsized":"2024-07-26T06:47:25.105155+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:53.173273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"80'6","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335825+0000","last_change":"2024-07-26T06:47:05.027900+0000","last_active":"2024-07-26T06:48:51.335825+0000","last_peered":"2024-07-26T06:48:51.335825+0000","last_clean":"2024-07-26T06:48:51.335825+0000","last_became_active":"2024-07-26T06:47:05.024467+0000","last_became_peered":"2024-07-26T06:47:05.024467+0000","last_unstale":"2024-07-26T06:48:51.335825+0000","last_undegraded":"2024-07-26T06:48:51.335825+0000","last_fullsized":"2024-07-26T06:48:51.335825+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:05:00.793487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090879+0000","last_change":"2024-07-26T06:47:02.022863+0000","last_active":"2024-07-26T06:47:23.090879+0000","last_peered":"2024-07-26T06:47:23.090879+0000","last_clean":"2024-07-26T06:47:23.090879+0000","last_became_active":"2024-07-26T06:47:02.022446+0000","last_became_peered":"2024-07-26T06:47:02.022446+0000","last_unstale":"2024-07-26T06:47:23.090879+0000","last_undegraded":"2024-07-26T06:47:23.090879+0000","last_fullsized":"2024-07-26T06:47:23.090879+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:14:48.649099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"80'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333567+0000","last_change":"2024-07-26T06:47:05.023669+0000","last_active":"2024-07-26T06:48:51.333567+0000","last_peered":"2024-07-26T06:48:51.333567+0000","last_clean":"2024-07-26T06:48:51.333567+0000","last_became_active":"2024-07-26T06:47:05.023265+0000","last_became_peered":"2024-07-26T06:47:05.023265+0000","last_unstale":"2024-07-26T06:48:51.333567+0000","last_undegraded":"2024-07-26T06:48:51.333567+0000","last_fullsized":"2024-07-26T06:48:51.333567+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:56:45.278253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":227,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:27.428838+0000","last_change":"2024-07-26T06:47:02.006668+0000","last_active":"2024-07-26T06:50:27.428838+0000","last_peered":"2024-07-26T06:50:27.428838+0000","last_clean":"2024-07-26T06:50:27.428838+0000","last_became_active":"2024-07-26T06:47:02.006287+0000","last_became_peered":"2024-07-26T06:47:02.006287+0000","last_unstale":"2024-07-26T06:50:27.428838+0000","last_undegraded":"2024-07-26T06:50:27.428838+0000","last_fullsized":"2024-07-26T06:50:27.428838+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:28:05.722539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":186,"num_read_kb":186,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70177,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1893,"num_read_kb":1024,"num_write":261,"num_write_kb":149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1634,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":210531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250626,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710777,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":426222,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877907006,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":1079472,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070530,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":425458,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266759,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":880640,"data_stored":425142,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462988,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":946176,"data_stored":1019335,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626513,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36196,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708956,"statfs":{"total":95995035648,"available":95957970944,"internally_reserved":0,"allocated":880640,"data_stored":425175,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822742,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":1018481,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986267,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":487951,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182495,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":897024,"data_stored":425000,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411429,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36208,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708944,"statfs":{"total":95995035648,"available":95957958656,"internally_reserved":0,"allocated":892928,"data_stored":427148,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574953,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":946176,"data_stored":488471,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738477,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":426514,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:50:33.419 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-07-26T06:50:33.419 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph pg dump --format=json 2024-07-26T06:50:33.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:33 smithi042 ceph-mon[32775]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:50:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:33.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:33 smithi164 ceph-mon[36970]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:50:33.933 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:33 smithi163 ceph-mon[36873]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:50:34.802 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:34 smithi042 ceph-mon[32775]: pgmap v406: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T06:50:34.864 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:34 smithi163 ceph-mon[36873]: pgmap v406: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T06:50:34.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:34 smithi164 ceph-mon[36970]: pgmap v406: 97 pgs: 97 active+clean; 645 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T06:50:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.344+0000 7fd1740d2640 1 -- 172.21.15.42:0/1285657452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16c102490 msgr2=0x7fd16c1028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.344+0000 7fd1740d2640 1 --2- 172.21.15.42:0/1285657452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16c102490 0x7fd16c1028a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd160009900 tx=0x7fd16002a410 comp rx=0 tx=0).stop 2024-07-26T06:50:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.344+0000 7fd1740d2640 1 -- 172.21.15.42:0/1285657452 shutdown_connections 2024-07-26T06:50:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.344+0000 7fd1740d2640 1 --2- 172.21.15.42:0/1285657452 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd16c108720 0x7fd16c10ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.344+0000 7fd1740d2640 1 --2- 172.21.15.42:0/1285657452 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd16c107dd0 0x7fd16c1081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.344+0000 7fd1740d2640 1 --2- 172.21.15.42:0/1285657452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16c102490 0x7fd16c1028a0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.344+0000 7fd1740d2640 1 -- 172.21.15.42:0/1285657452 >> 172.21.15.42:0/1285657452 conn(0x7fd16c079160 msgr2=0x7fd16c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.345+0000 7fd1740d2640 1 -- 172.21.15.42:0/1285657452 shutdown_connections 2024-07-26T06:50:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.345+0000 7fd1740d2640 1 -- 172.21.15.42:0/1285657452 wait complete. 2024-07-26T06:50:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.345+0000 7fd1740d2640 1 Processor -- start 2024-07-26T06:50:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.345+0000 7fd1740d2640 1 -- start start 2024-07-26T06:50:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.346+0000 7fd1740d2640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd16c102490 0x7fd16c19c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.346+0000 7fd1740d2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd16c107dd0 0x7fd16c19c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.346+0000 7fd1740d2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16c108720 0x7fd16c1a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.346+0000 7fd1740d2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c1134b0 con 0x7fd16c108720 2024-07-26T06:50:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.346+0000 7fd1740d2640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c113330 con 0x7fd16c102490 2024-07-26T06:50:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.346+0000 7fd1740d2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c113630 con 0x7fd16c107dd0 2024-07-26T06:50:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.346+0000 7fd17266f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16c108720 0x7fd16c1a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.346+0000 7fd17266f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16c108720 0x7fd16c1a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44870/0 (socket says 172.21.15.42:44870) 2024-07-26T06:50:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.346+0000 7fd17266f640 1 -- 172.21.15.42:0/1588482540 learned_addr learned my addr 172.21.15.42:0/1588482540 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.346+0000 7fd17166d640 1 --2- 172.21.15.42:0/1588482540 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd16c107dd0 0x7fd16c19c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.346+0000 7fd171e6e640 1 --2- 172.21.15.42:0/1588482540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd16c102490 0x7fd16c19c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.346+0000 7fd17266f640 1 -- 172.21.15.42:0/1588482540 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd16c107dd0 msgr2=0x7fd16c19c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.346+0000 7fd17266f640 1 --2- 172.21.15.42:0/1588482540 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd16c107dd0 0x7fd16c19c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.346+0000 7fd17266f640 1 -- 172.21.15.42:0/1588482540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd16c102490 msgr2=0x7fd16c19c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.346+0000 7fd17266f640 1 --2- 172.21.15.42:0/1588482540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd16c102490 0x7fd16c19c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.346+0000 7fd17266f640 1 -- 172.21.15.42:0/1588482540 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16c1a2030 con 0x7fd16c108720 2024-07-26T06:50:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.346+0000 7fd17166d640 1 --2- 172.21.15.42:0/1588482540 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd16c107dd0 0x7fd16c19c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:50:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.347+0000 7fd17266f640 1 --2- 172.21.15.42:0/1588482540 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16c108720 0x7fd16c1a1900 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd164009fd0 tx=0x7fd16400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.347+0000 7fd15effd640 1 -- 172.21.15.42:0/1588482540 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164017070 con 0x7fd16c108720 2024-07-26T06:50:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.347+0000 7fd15effd640 1 -- 172.21.15.42:0/1588482540 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd164003e20 con 0x7fd16c108720 2024-07-26T06:50:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.347+0000 7fd1740d2640 1 -- 172.21.15.42:0/1588482540 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd16c1a2320 con 0x7fd16c108720 2024-07-26T06:50:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.347+0000 7fd15effd640 1 -- 172.21.15.42:0/1588482540 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164010460 con 0x7fd16c108720 2024-07-26T06:50:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.347+0000 7fd1740d2640 1 -- 172.21.15.42:0/1588482540 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd16c1a8a80 con 0x7fd16c108720 2024-07-26T06:50:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.348+0000 7fd1740d2640 1 -- 172.21.15.42:0/1588482540 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd16c1028a0 con 0x7fd16c108720 2024-07-26T06:50:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.349+0000 7fd15effd640 1 -- 172.21.15.42:0/1588482540 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fd164003970 con 0x7fd16c108720 2024-07-26T06:50:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.350+0000 7fd15effd640 1 --2- 172.21.15.42:0/1588482540 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd1580b3bd0 0x7fd1580b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.350+0000 7fd15effd640 1 -- 172.21.15.42:0/1588482540 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fd1640e38f0 con 0x7fd16c108720 2024-07-26T06:50:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.350+0000 7fd171e6e640 1 --2- 172.21.15.42:0/1588482540 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd1580b3bd0 0x7fd1580b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.351+0000 7fd171e6e640 1 --2- 172.21.15.42:0/1588482540 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd1580b3bd0 0x7fd1580b60e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd160031440 tx=0x7fd160009560 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.353+0000 7fd15effd640 1 -- 172.21.15.42:0/1588482540 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd1640aa8d0 con 0x7fd16c108720 2024-07-26T06:50:35.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.547+0000 7fd1740d2640 1 -- 172.21.15.42:0/1588482540 --> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd16c111150 con 0x7fd1580b3bd0 2024-07-26T06:50:35.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.552+0000 7fd15effd640 1 -- 172.21.15.42:0/1588482540 <== mgr.14148 v2:172.21.15.42:6800/2719631589 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259115 (secure 0 0 0) 0x7fd16c111150 con 0x7fd1580b3bd0 2024-07-26T06:50:35.555 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:50:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-07-26T06:50:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.555+0000 7fd1740d2640 1 -- 172.21.15.42:0/1588482540 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd1580b3bd0 msgr2=0x7fd1580b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.555+0000 7fd1740d2640 1 --2- 172.21.15.42:0/1588482540 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd1580b3bd0 0x7fd1580b60e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd160031440 tx=0x7fd160009560 comp rx=0 tx=0).stop 2024-07-26T06:50:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.555+0000 7fd1740d2640 1 -- 172.21.15.42:0/1588482540 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16c108720 msgr2=0x7fd16c1a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.555+0000 7fd1740d2640 1 --2- 172.21.15.42:0/1588482540 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16c108720 0x7fd16c1a1900 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd164009fd0 tx=0x7fd16400b260 comp rx=0 tx=0).stop 2024-07-26T06:50:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.555+0000 7fd1740d2640 1 -- 172.21.15.42:0/1588482540 shutdown_connections 2024-07-26T06:50:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.555+0000 7fd1740d2640 1 --2- 172.21.15.42:0/1588482540 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fd1580b3bd0 0x7fd1580b60e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:35.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.555+0000 7fd1740d2640 1 --2- 172.21.15.42:0/1588482540 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16c108720 0x7fd16c1a1900 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:35.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.555+0000 7fd1740d2640 1 --2- 172.21.15.42:0/1588482540 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd16c107dd0 0x7fd16c19c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:35.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.555+0000 7fd1740d2640 1 --2- 172.21.15.42:0/1588482540 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd16c102490 0x7fd16c19c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:35.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.555+0000 7fd1740d2640 1 -- 172.21.15.42:0/1588482540 >> 172.21.15.42:0/1588482540 conn(0x7fd16c079160 msgr2=0x7fd16c110a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:35.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.555+0000 7fd1740d2640 1 -- 172.21.15.42:0/1588482540 shutdown_connections 2024-07-26T06:50:35.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:35.555+0000 7fd1740d2640 1 -- 172.21.15.42:0/1588482540 wait complete. 2024-07-26T06:50:36.247 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-07-26T06:50:35.373903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":665093,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":2015,"num_read_kb":1118,"num_write":387,"num_write_kb":2306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2953474,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":818,"ondisk_log_size":818,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384936,"kb_used_data":10824,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124556888,"statfs":{"total":1151940427776,"available":1151546253312,"internally_reserved":0,"allocated":11083776,"data_stored":7087311,"data_compressed":177858,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4930,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":133,"num_read_kb":77,"num_write":9,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003516"},"pg_stats":[{"pgid":"3.3f","version":"80'8","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337456+0000","last_change":"2024-07-26T06:47:05.026186+0000","last_active":"2024-07-26T06:48:51.337456+0000","last_peered":"2024-07-26T06:48:51.337456+0000","last_clean":"2024-07-26T06:48:51.337456+0000","last_became_active":"2024-07-26T06:47:05.025589+0000","last_became_peered":"2024-07-26T06:47:05.025589+0000","last_unstale":"2024-07-26T06:48:51.337456+0000","last_undegraded":"2024-07-26T06:48:51.337456+0000","last_fullsized":"2024-07-26T06:48:51.337456+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:48:23.100966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"80'4","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326008+0000","last_change":"2024-07-26T06:47:05.033211+0000","last_active":"2024-07-26T06:48:51.326008+0000","last_peered":"2024-07-26T06:48:51.326008+0000","last_clean":"2024-07-26T06:48:51.326008+0000","last_became_active":"2024-07-26T06:47:05.032630+0000","last_became_peered":"2024-07-26T06:47:05.032630+0000","last_unstale":"2024-07-26T06:48:51.326008+0000","last_undegraded":"2024-07-26T06:48:51.326008+0000","last_fullsized":"2024-07-26T06:48:51.326008+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:03:19.612963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335969+0000","last_change":"2024-07-26T06:47:05.034767+0000","last_active":"2024-07-26T06:48:51.335969+0000","last_peered":"2024-07-26T06:48:51.335969+0000","last_clean":"2024-07-26T06:48:51.335969+0000","last_became_active":"2024-07-26T06:47:05.034376+0000","last_became_peered":"2024-07-26T06:47:05.034376+0000","last_unstale":"2024-07-26T06:48:51.335969+0000","last_undegraded":"2024-07-26T06:48:51.335969+0000","last_fullsized":"2024-07-26T06:48:51.335969+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:55:55.729930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328614+0000","last_change":"2024-07-26T06:47:05.030995+0000","last_active":"2024-07-26T06:48:51.328614+0000","last_peered":"2024-07-26T06:48:51.328614+0000","last_clean":"2024-07-26T06:48:51.328614+0000","last_became_active":"2024-07-26T06:47:05.025423+0000","last_became_peered":"2024-07-26T06:47:05.025423+0000","last_unstale":"2024-07-26T06:48:51.328614+0000","last_undegraded":"2024-07-26T06:48:51.328614+0000","last_fullsized":"2024-07-26T06:48:51.328614+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:59:39.511383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"80'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.317041+0000","last_change":"2024-07-26T06:47:05.035359+0000","last_active":"2024-07-26T06:48:51.317041+0000","last_peered":"2024-07-26T06:48:51.317041+0000","last_clean":"2024-07-26T06:48:51.317041+0000","last_became_active":"2024-07-26T06:47:05.034836+0000","last_became_peered":"2024-07-26T06:47:05.034836+0000","last_unstale":"2024-07-26T06:48:51.317041+0000","last_undegraded":"2024-07-26T06:48:51.317041+0000","last_fullsized":"2024-07-26T06:48:51.317041+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:54.482774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"80'6","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326306+0000","last_change":"2024-07-26T06:47:05.023660+0000","last_active":"2024-07-26T06:48:51.326306+0000","last_peered":"2024-07-26T06:48:51.326306+0000","last_clean":"2024-07-26T06:48:51.326306+0000","last_became_active":"2024-07-26T06:47:05.022446+0000","last_became_peered":"2024-07-26T06:47:05.022446+0000","last_unstale":"2024-07-26T06:48:51.326306+0000","last_undegraded":"2024-07-26T06:48:51.326306+0000","last_fullsized":"2024-07-26T06:48:51.326306+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:35:00.519226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"80'5","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339509+0000","last_change":"2024-07-26T06:47:05.025132+0000","last_active":"2024-07-26T06:48:51.339509+0000","last_peered":"2024-07-26T06:48:51.339509+0000","last_clean":"2024-07-26T06:48:51.339509+0000","last_became_active":"2024-07-26T06:47:05.024770+0000","last_became_peered":"2024-07-26T06:47:05.024770+0000","last_unstale":"2024-07-26T06:48:51.339509+0000","last_undegraded":"2024-07-26T06:48:51.339509+0000","last_fullsized":"2024-07-26T06:48:51.339509+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:21:39.920152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"80'9","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333625+0000","last_change":"2024-07-26T06:47:05.022613+0000","last_active":"2024-07-26T06:48:51.333625+0000","last_peered":"2024-07-26T06:48:51.333625+0000","last_clean":"2024-07-26T06:48:51.333625+0000","last_became_active":"2024-07-26T06:47:05.022202+0000","last_became_peered":"2024-07-26T06:47:05.022202+0000","last_unstale":"2024-07-26T06:48:51.333625+0000","last_undegraded":"2024-07-26T06:48:51.333625+0000","last_fullsized":"2024-07-26T06:48:51.333625+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:07:23.980843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"80'11","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.336947+0000","last_change":"2024-07-26T06:47:05.026352+0000","last_active":"2024-07-26T06:48:51.336947+0000","last_peered":"2024-07-26T06:48:51.336947+0000","last_clean":"2024-07-26T06:48:51.336947+0000","last_became_active":"2024-07-26T06:47:05.025969+0000","last_became_peered":"2024-07-26T06:47:05.025969+0000","last_unstale":"2024-07-26T06:48:51.336947+0000","last_undegraded":"2024-07-26T06:48:51.336947+0000","last_fullsized":"2024-07-26T06:48:51.336947+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:08:27.062013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"80'16","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.329437+0000","last_change":"2024-07-26T06:47:23.100422+0000","last_active":"2024-07-26T06:48:51.329437+0000","last_peered":"2024-07-26T06:48:51.329437+0000","last_clean":"2024-07-26T06:48:51.329437+0000","last_became_active":"2024-07-26T06:47:23.100023+0000","last_became_peered":"2024-07-26T06:47:23.100023+0000","last_unstale":"2024-07-26T06:48:51.329437+0000","last_undegraded":"2024-07-26T06:48:51.329437+0000","last_fullsized":"2024-07-26T06:48:51.329437+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:28.158112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"80'6","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338557+0000","last_change":"2024-07-26T06:47:05.037142+0000","last_active":"2024-07-26T06:48:51.338557+0000","last_peered":"2024-07-26T06:48:51.338557+0000","last_clean":"2024-07-26T06:48:51.338557+0000","last_became_active":"2024-07-26T06:47:05.036745+0000","last_became_peered":"2024-07-26T06:47:05.036745+0000","last_unstale":"2024-07-26T06:48:51.338557+0000","last_undegraded":"2024-07-26T06:48:51.338557+0000","last_fullsized":"2024-07-26T06:48:51.338557+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:31:10.034560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"80'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.315686+0000","last_change":"2024-07-26T06:47:05.030640+0000","last_active":"2024-07-26T06:48:51.315686+0000","last_peered":"2024-07-26T06:48:51.315686+0000","last_clean":"2024-07-26T06:48:51.315686+0000","last_became_active":"2024-07-26T06:47:05.023583+0000","last_became_peered":"2024-07-26T06:47:05.023583+0000","last_unstale":"2024-07-26T06:48:51.315686+0000","last_undegraded":"2024-07-26T06:48:51.315686+0000","last_fullsized":"2024-07-26T06:48:51.315686+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:59:12.312930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"80'10","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333194+0000","last_change":"2024-07-26T06:47:05.021768+0000","last_active":"2024-07-26T06:48:51.333194+0000","last_peered":"2024-07-26T06:48:51.333194+0000","last_clean":"2024-07-26T06:48:51.333194+0000","last_became_active":"2024-07-26T06:47:05.021410+0000","last_became_peered":"2024-07-26T06:47:05.021410+0000","last_unstale":"2024-07-26T06:48:51.333194+0000","last_undegraded":"2024-07-26T06:48:51.333194+0000","last_fullsized":"2024-07-26T06:48:51.333194+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:05.576122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.324480+0000","last_change":"2024-07-26T06:47:05.035814+0000","last_active":"2024-07-26T06:48:51.324480+0000","last_peered":"2024-07-26T06:48:51.324480+0000","last_clean":"2024-07-26T06:48:51.324480+0000","last_became_active":"2024-07-26T06:47:05.035451+0000","last_became_peered":"2024-07-26T06:47:05.035451+0000","last_unstale":"2024-07-26T06:48:51.324480+0000","last_undegraded":"2024-07-26T06:48:51.324480+0000","last_fullsized":"2024-07-26T06:48:51.324480+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:33.533726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.331515+0000","last_change":"2024-07-26T06:47:05.035720+0000","last_active":"2024-07-26T06:48:51.331515+0000","last_peered":"2024-07-26T06:48:51.331515+0000","last_clean":"2024-07-26T06:48:51.331515+0000","last_became_active":"2024-07-26T06:47:05.035359+0000","last_became_peered":"2024-07-26T06:47:05.035359+0000","last_unstale":"2024-07-26T06:48:51.331515+0000","last_undegraded":"2024-07-26T06:48:51.331515+0000","last_fullsized":"2024-07-26T06:48:51.331515+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:38.561268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339962+0000","last_change":"2024-07-26T06:47:05.027109+0000","last_active":"2024-07-26T06:48:51.339962+0000","last_peered":"2024-07-26T06:48:51.339962+0000","last_clean":"2024-07-26T06:48:51.339962+0000","last_became_active":"2024-07-26T06:47:05.026694+0000","last_became_peered":"2024-07-26T06:47:05.026694+0000","last_unstale":"2024-07-26T06:48:51.339962+0000","last_undegraded":"2024-07-26T06:48:51.339962+0000","last_fullsized":"2024-07-26T06:48:51.339962+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:25:01.215646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"80'8","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337458+0000","last_change":"2024-07-26T06:47:05.028212+0000","last_active":"2024-07-26T06:48:51.337458+0000","last_peered":"2024-07-26T06:48:51.337458+0000","last_clean":"2024-07-26T06:48:51.337458+0000","last_became_active":"2024-07-26T06:47:05.027598+0000","last_became_peered":"2024-07-26T06:47:05.027598+0000","last_unstale":"2024-07-26T06:48:51.337458+0000","last_undegraded":"2024-07-26T06:48:51.337458+0000","last_fullsized":"2024-07-26T06:48:51.337458+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:55:31.536194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"80'10","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.336116+0000","last_change":"2024-07-26T06:47:05.027724+0000","last_active":"2024-07-26T06:48:51.336116+0000","last_peered":"2024-07-26T06:48:51.336116+0000","last_clean":"2024-07-26T06:48:51.336116+0000","last_became_active":"2024-07-26T06:47:05.027348+0000","last_became_peered":"2024-07-26T06:47:05.027348+0000","last_unstale":"2024-07-26T06:48:51.336116+0000","last_undegraded":"2024-07-26T06:48:51.336116+0000","last_fullsized":"2024-07-26T06:48:51.336116+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:04.180467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"80'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.327682+0000","last_change":"2024-07-26T06:47:05.025474+0000","last_active":"2024-07-26T06:48:51.327682+0000","last_peered":"2024-07-26T06:48:51.327682+0000","last_clean":"2024-07-26T06:48:51.327682+0000","last_became_active":"2024-07-26T06:47:05.025016+0000","last_became_peered":"2024-07-26T06:47:05.025016+0000","last_unstale":"2024-07-26T06:48:51.327682+0000","last_undegraded":"2024-07-26T06:48:51.327682+0000","last_fullsized":"2024-07-26T06:48:51.327682+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:40:03.034532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"80'8","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.330348+0000","last_change":"2024-07-26T06:47:05.027533+0000","last_active":"2024-07-26T06:48:51.330348+0000","last_peered":"2024-07-26T06:48:51.330348+0000","last_clean":"2024-07-26T06:48:51.330348+0000","last_became_active":"2024-07-26T06:47:05.027143+0000","last_became_peered":"2024-07-26T06:47:05.027143+0000","last_unstale":"2024-07-26T06:48:51.330348+0000","last_undegraded":"2024-07-26T06:48:51.330348+0000","last_fullsized":"2024-07-26T06:48:51.330348+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:41:13.406430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2b","version":"80'11","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338271+0000","last_change":"2024-07-26T06:47:05.025962+0000","last_active":"2024-07-26T06:48:51.338271+0000","last_peered":"2024-07-26T06:48:51.338271+0000","last_clean":"2024-07-26T06:48:51.338271+0000","last_became_active":"2024-07-26T06:47:05.025621+0000","last_became_peered":"2024-07-26T06:47:05.025621+0000","last_unstale":"2024-07-26T06:48:51.338271+0000","last_undegraded":"2024-07-26T06:48:51.338271+0000","last_fullsized":"2024-07-26T06:48:51.338271+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:56:43.669247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"80'10","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337425+0000","last_change":"2024-07-26T06:47:05.026252+0000","last_active":"2024-07-26T06:48:51.337425+0000","last_peered":"2024-07-26T06:48:51.337425+0000","last_clean":"2024-07-26T06:48:51.337425+0000","last_became_active":"2024-07-26T06:47:05.025875+0000","last_became_peered":"2024-07-26T06:47:05.025875+0000","last_unstale":"2024-07-26T06:48:51.337425+0000","last_undegraded":"2024-07-26T06:48:51.337425+0000","last_fullsized":"2024-07-26T06:48:51.337425+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:27:32.955862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.327073+0000","last_change":"2024-07-26T06:47:05.022513+0000","last_active":"2024-07-26T06:48:51.327073+0000","last_peered":"2024-07-26T06:48:51.327073+0000","last_clean":"2024-07-26T06:48:51.327073+0000","last_became_active":"2024-07-26T06:47:05.022159+0000","last_became_peered":"2024-07-26T06:47:05.022159+0000","last_unstale":"2024-07-26T06:48:51.327073+0000","last_undegraded":"2024-07-26T06:48:51.327073+0000","last_fullsized":"2024-07-26T06:48:51.327073+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:53:06.402985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"80'6","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.325351+0000","last_change":"2024-07-26T06:47:23.097604+0000","last_active":"2024-07-26T06:48:51.325351+0000","last_peered":"2024-07-26T06:48:51.325351+0000","last_clean":"2024-07-26T06:48:51.325351+0000","last_became_active":"2024-07-26T06:47:23.097170+0000","last_became_peered":"2024-07-26T06:47:23.097170+0000","last_unstale":"2024-07-26T06:48:51.325351+0000","last_undegraded":"2024-07-26T06:48:51.325351+0000","last_fullsized":"2024-07-26T06:48:51.325351+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:38:24.530396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.27","version":"80'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.323758+0000","last_change":"2024-07-26T06:47:05.026349+0000","last_active":"2024-07-26T06:48:51.323758+0000","last_peered":"2024-07-26T06:48:51.323758+0000","last_clean":"2024-07-26T06:48:51.323758+0000","last_became_active":"2024-07-26T06:47:05.026016+0000","last_became_peered":"2024-07-26T06:47:05.026016+0000","last_unstale":"2024-07-26T06:48:51.323758+0000","last_undegraded":"2024-07-26T06:48:51.323758+0000","last_fullsized":"2024-07-26T06:48:51.323758+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:19:02.264183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"80'15","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.323355+0000","last_change":"2024-07-26T06:47:05.032439+0000","last_active":"2024-07-26T06:48:51.323355+0000","last_peered":"2024-07-26T06:48:51.323355+0000","last_clean":"2024-07-26T06:48:51.323355+0000","last_became_active":"2024-07-26T06:47:05.032105+0000","last_became_peered":"2024-07-26T06:47:05.032105+0000","last_unstale":"2024-07-26T06:48:51.323355+0000","last_undegraded":"2024-07-26T06:48:51.323355+0000","last_fullsized":"2024-07-26T06:48:51.323355+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:21:50.423939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"80'6","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.319567+0000","last_change":"2024-07-26T06:47:05.026927+0000","last_active":"2024-07-26T06:48:51.319567+0000","last_peered":"2024-07-26T06:48:51.319567+0000","last_clean":"2024-07-26T06:48:51.319567+0000","last_became_active":"2024-07-26T06:47:05.026569+0000","last_became_peered":"2024-07-26T06:47:05.026569+0000","last_unstale":"2024-07-26T06:48:51.319567+0000","last_undegraded":"2024-07-26T06:48:51.319567+0000","last_fullsized":"2024-07-26T06:48:51.319567+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:12:14.687633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.24","version":"80'4","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333564+0000","last_change":"2024-07-26T06:47:23.099689+0000","last_active":"2024-07-26T06:48:51.333564+0000","last_peered":"2024-07-26T06:48:51.333564+0000","last_clean":"2024-07-26T06:48:51.333564+0000","last_became_active":"2024-07-26T06:47:23.099313+0000","last_became_peered":"2024-07-26T06:47:23.099313+0000","last_unstale":"2024-07-26T06:48:51.333564+0000","last_undegraded":"2024-07-26T06:48:51.333564+0000","last_fullsized":"2024-07-26T06:48:51.333564+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:49:01.833297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333361+0000","last_change":"2024-07-26T06:47:05.034864+0000","last_active":"2024-07-26T06:48:51.333361+0000","last_peered":"2024-07-26T06:48:51.333361+0000","last_clean":"2024-07-26T06:48:51.333361+0000","last_became_active":"2024-07-26T06:47:05.033414+0000","last_became_peered":"2024-07-26T06:47:05.033414+0000","last_unstale":"2024-07-26T06:48:51.333361+0000","last_undegraded":"2024-07-26T06:48:51.333361+0000","last_fullsized":"2024-07-26T06:48:51.333361+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:21:31.420212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.324409+0000","last_change":"2024-07-26T06:47:05.025462+0000","last_active":"2024-07-26T06:48:51.324409+0000","last_peered":"2024-07-26T06:48:51.324409+0000","last_clean":"2024-07-26T06:48:51.324409+0000","last_became_active":"2024-07-26T06:47:05.025049+0000","last_became_peered":"2024-07-26T06:47:05.025049+0000","last_unstale":"2024-07-26T06:48:51.324409+0000","last_undegraded":"2024-07-26T06:48:51.324409+0000","last_fullsized":"2024-07-26T06:48:51.324409+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:59:28.903338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.21","version":"80'12","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335500+0000","last_change":"2024-07-26T06:47:05.026394+0000","last_active":"2024-07-26T06:48:51.335500+0000","last_peered":"2024-07-26T06:48:51.335500+0000","last_clean":"2024-07-26T06:48:51.335500+0000","last_became_active":"2024-07-26T06:47:05.025820+0000","last_became_peered":"2024-07-26T06:47:05.025820+0000","last_unstale":"2024-07-26T06:48:51.335500+0000","last_undegraded":"2024-07-26T06:48:51.335500+0000","last_fullsized":"2024-07-26T06:48:51.335500+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:44:57.441638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.329050+0000","last_change":"2024-07-26T06:47:05.024373+0000","last_active":"2024-07-26T06:48:51.329050+0000","last_peered":"2024-07-26T06:48:51.329050+0000","last_clean":"2024-07-26T06:48:51.329050+0000","last_became_active":"2024-07-26T06:47:05.023962+0000","last_became_peered":"2024-07-26T06:47:05.023962+0000","last_unstale":"2024-07-26T06:48:51.329050+0000","last_undegraded":"2024-07-26T06:48:51.329050+0000","last_fullsized":"2024-07-26T06:48:51.329050+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:43.804614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"80'7","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.325450+0000","last_change":"2024-07-26T06:47:05.031146+0000","last_active":"2024-07-26T06:48:51.325450+0000","last_peered":"2024-07-26T06:48:51.325450+0000","last_clean":"2024-07-26T06:48:51.325450+0000","last_became_active":"2024-07-26T06:47:05.029393+0000","last_became_peered":"2024-07-26T06:47:05.029393+0000","last_unstale":"2024-07-26T06:48:51.325450+0000","last_undegraded":"2024-07-26T06:48:51.325450+0000","last_fullsized":"2024-07-26T06:48:51.325450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:27:46.853933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1b","version":"80'8","reported_seq":228,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:30.425104+0000","last_change":"2024-07-26T06:47:02.001800+0000","last_active":"2024-07-26T06:50:30.425104+0000","last_peered":"2024-07-26T06:50:30.425104+0000","last_clean":"2024-07-26T06:50:30.425104+0000","last_became_active":"2024-07-26T06:47:02.001435+0000","last_became_peered":"2024-07-26T06:47:02.001435+0000","last_unstale":"2024-07-26T06:50:30.425104+0000","last_undegraded":"2024-07-26T06:50:30.425104+0000","last_fullsized":"2024-07-26T06:50:30.425104+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:23.121378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1462,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":193,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"80'8","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.334283+0000","last_change":"2024-07-26T06:47:05.028301+0000","last_active":"2024-07-26T06:48:51.334283+0000","last_peered":"2024-07-26T06:48:51.334283+0000","last_clean":"2024-07-26T06:48:51.334283+0000","last_became_active":"2024-07-26T06:47:05.027761+0000","last_became_peered":"2024-07-26T06:47:05.027761+0000","last_unstale":"2024-07-26T06:48:51.334283+0000","last_undegraded":"2024-07-26T06:48:51.334283+0000","last_fullsized":"2024-07-26T06:48:51.334283+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:58:19.109395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1a","version":"78'7","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.093824+0000","last_change":"2024-07-26T06:47:02.004488+0000","last_active":"2024-07-26T06:47:23.093824+0000","last_peered":"2024-07-26T06:47:23.093824+0000","last_clean":"2024-07-26T06:47:23.093824+0000","last_became_active":"2024-07-26T06:47:02.004163+0000","last_became_peered":"2024-07-26T06:47:02.004163+0000","last_unstale":"2024-07-26T06:47:23.093824+0000","last_undegraded":"2024-07-26T06:47:23.093824+0000","last_fullsized":"2024-07-26T06:47:23.093824+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:05:16.221618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096130+0000","last_change":"2024-07-26T06:47:01.992708+0000","last_active":"2024-07-26T06:47:23.096130+0000","last_peered":"2024-07-26T06:47:23.096130+0000","last_clean":"2024-07-26T06:47:23.096130+0000","last_became_active":"2024-07-26T06:47:01.992330+0000","last_became_peered":"2024-07-26T06:47:01.992330+0000","last_unstale":"2024-07-26T06:47:23.096130+0000","last_undegraded":"2024-07-26T06:47:23.096130+0000","last_fullsized":"2024-07-26T06:47:23.096130+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:25.951946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.330722+0000","last_change":"2024-07-26T06:47:05.013946+0000","last_active":"2024-07-26T06:48:51.330722+0000","last_peered":"2024-07-26T06:48:51.330722+0000","last_clean":"2024-07-26T06:48:51.330722+0000","last_became_active":"2024-07-26T06:47:05.013509+0000","last_became_peered":"2024-07-26T06:47:05.013509+0000","last_unstale":"2024-07-26T06:48:51.330722+0000","last_undegraded":"2024-07-26T06:48:51.330722+0000","last_fullsized":"2024-07-26T06:48:51.330722+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:09.789287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.124184+0000","last_change":"2024-07-26T06:47:01.997617+0000","last_active":"2024-07-26T06:47:24.124184+0000","last_peered":"2024-07-26T06:47:24.124184+0000","last_clean":"2024-07-26T06:47:24.124184+0000","last_became_active":"2024-07-26T06:47:01.997238+0000","last_became_peered":"2024-07-26T06:47:01.997238+0000","last_unstale":"2024-07-26T06:47:24.124184+0000","last_undegraded":"2024-07-26T06:47:24.124184+0000","last_fullsized":"2024-07-26T06:47:24.124184+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:45.596438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"80'11","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328533+0000","last_change":"2024-07-26T06:47:23.099813+0000","last_active":"2024-07-26T06:48:51.328533+0000","last_peered":"2024-07-26T06:48:51.328533+0000","last_clean":"2024-07-26T06:48:51.328533+0000","last_became_active":"2024-07-26T06:47:23.098701+0000","last_became_peered":"2024-07-26T06:47:23.098701+0000","last_unstale":"2024-07-26T06:48:51.328533+0000","last_undegraded":"2024-07-26T06:48:51.328533+0000","last_fullsized":"2024-07-26T06:48:51.328533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:42:04.059637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"80'6","reported_seq":413,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:33.429806+0000","last_change":"2024-07-26T06:47:01.998508+0000","last_active":"2024-07-26T06:50:33.429806+0000","last_peered":"2024-07-26T06:50:33.429806+0000","last_clean":"2024-07-26T06:50:33.429806+0000","last_became_active":"2024-07-26T06:47:01.998132+0000","last_became_peered":"2024-07-26T06:47:01.998132+0000","last_unstale":"2024-07-26T06:50:33.429806+0000","last_undegraded":"2024-07-26T06:50:33.429806+0000","last_fullsized":"2024-07-26T06:50:33.429806+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:21.588085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":380,"num_read_kb":380,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"80'10","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339164+0000","last_change":"2024-07-26T06:47:05.032704+0000","last_active":"2024-07-26T06:48:51.339164+0000","last_peered":"2024-07-26T06:48:51.339164+0000","last_clean":"2024-07-26T06:48:51.339164+0000","last_became_active":"2024-07-26T06:47:05.031843+0000","last_became_peered":"2024-07-26T06:47:05.031843+0000","last_unstale":"2024-07-26T06:48:51.339164+0000","last_undegraded":"2024-07-26T06:48:51.339164+0000","last_fullsized":"2024-07-26T06:48:51.339164+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:58:24.436233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097471+0000","last_change":"2024-07-26T06:47:02.006266+0000","last_active":"2024-07-26T06:47:23.097471+0000","last_peered":"2024-07-26T06:47:23.097471+0000","last_clean":"2024-07-26T06:47:23.097471+0000","last_became_active":"2024-07-26T06:47:02.005830+0000","last_became_peered":"2024-07-26T06:47:02.005830+0000","last_unstale":"2024-07-26T06:47:23.097471+0000","last_undegraded":"2024-07-26T06:47:23.097471+0000","last_fullsized":"2024-07-26T06:47:23.097471+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:06:09.097781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.331471+0000","last_change":"2024-07-26T06:47:05.024820+0000","last_active":"2024-07-26T06:48:51.331471+0000","last_peered":"2024-07-26T06:48:51.331471+0000","last_clean":"2024-07-26T06:48:51.331471+0000","last_became_active":"2024-07-26T06:47:05.023941+0000","last_became_peered":"2024-07-26T06:47:05.023941+0000","last_unstale":"2024-07-26T06:48:51.331471+0000","last_undegraded":"2024-07-26T06:48:51.331471+0000","last_fullsized":"2024-07-26T06:48:51.331471+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:51:07.885159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"78'5","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.123256+0000","last_change":"2024-07-26T06:47:02.023308+0000","last_active":"2024-07-26T06:47:24.123256+0000","last_peered":"2024-07-26T06:47:24.123256+0000","last_clean":"2024-07-26T06:47:24.123256+0000","last_became_active":"2024-07-26T06:47:02.022933+0000","last_became_peered":"2024-07-26T06:47:02.022933+0000","last_unstale":"2024-07-26T06:47:24.123256+0000","last_undegraded":"2024-07-26T06:47:24.123256+0000","last_fullsized":"2024-07-26T06:47:24.123256+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:59:01.233386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"80'11","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.340511+0000","last_change":"2024-07-26T06:47:05.026005+0000","last_active":"2024-07-26T06:48:51.340511+0000","last_peered":"2024-07-26T06:48:51.340511+0000","last_clean":"2024-07-26T06:48:51.340511+0000","last_became_active":"2024-07-26T06:47:05.025599+0000","last_became_peered":"2024-07-26T06:47:05.025599+0000","last_unstale":"2024-07-26T06:48:51.340511+0000","last_undegraded":"2024-07-26T06:48:51.340511+0000","last_fullsized":"2024-07-26T06:48:51.340511+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:44:19.924157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"80'37","reported_seq":475,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:33.430166+0000","last_change":"2024-07-26T06:47:23.393050+0000","last_active":"2024-07-26T06:50:33.430166+0000","last_peered":"2024-07-26T06:50:33.430166+0000","last_clean":"2024-07-26T06:50:33.430166+0000","last_became_active":"2024-07-26T06:47:23.381611+0000","last_became_peered":"2024-07-26T06:47:23.381611+0000","last_unstale":"2024-07-26T06:50:33.430166+0000","last_undegraded":"2024-07-26T06:50:33.430166+0000","last_fullsized":"2024-07-26T06:50:33.430166+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:51:23.866523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66697,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":392,"num_read_kb":261,"num_write":39,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"80'6","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335684+0000","last_change":"2024-07-26T06:47:05.034190+0000","last_active":"2024-07-26T06:48:51.335684+0000","last_peered":"2024-07-26T06:48:51.335684+0000","last_clean":"2024-07-26T06:48:51.335684+0000","last_became_active":"2024-07-26T06:47:05.033569+0000","last_became_peered":"2024-07-26T06:47:05.033569+0000","last_unstale":"2024-07-26T06:48:51.335684+0000","last_undegraded":"2024-07-26T06:48:51.335684+0000","last_fullsized":"2024-07-26T06:48:51.335684+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:19:39.306996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.123635+0000","last_change":"2024-07-26T06:47:02.001718+0000","last_active":"2024-07-26T06:47:24.123635+0000","last_peered":"2024-07-26T06:47:24.123635+0000","last_clean":"2024-07-26T06:47:24.123635+0000","last_became_active":"2024-07-26T06:47:02.001359+0000","last_became_peered":"2024-07-26T06:47:02.001359+0000","last_unstale":"2024-07-26T06:47:24.123635+0000","last_undegraded":"2024-07-26T06:47:24.123635+0000","last_fullsized":"2024-07-26T06:47:24.123635+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:03:23.390279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.337843+0000","last_change":"2024-07-26T06:47:05.027445+0000","last_active":"2024-07-26T06:48:51.337843+0000","last_peered":"2024-07-26T06:48:51.337843+0000","last_clean":"2024-07-26T06:48:51.337843+0000","last_became_active":"2024-07-26T06:47:05.027114+0000","last_became_peered":"2024-07-26T06:47:05.027114+0000","last_unstale":"2024-07-26T06:48:51.337843+0000","last_undegraded":"2024-07-26T06:48:51.337843+0000","last_fullsized":"2024-07-26T06:48:51.337843+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:29:10.519271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":33,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.119909+0000","last_change":"2024-07-26T06:47:01.998832+0000","last_active":"2024-07-26T06:47:24.119909+0000","last_peered":"2024-07-26T06:47:24.119909+0000","last_clean":"2024-07-26T06:47:24.119909+0000","last_became_active":"2024-07-26T06:47:01.998512+0000","last_became_peered":"2024-07-26T06:47:01.998512+0000","last_unstale":"2024-07-26T06:47:24.119909+0000","last_undegraded":"2024-07-26T06:47:24.119909+0000","last_fullsized":"2024-07-26T06:47:24.119909+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:10:27.722329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"80'15","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.340182+0000","last_change":"2024-07-26T06:47:05.027803+0000","last_active":"2024-07-26T06:48:51.340182+0000","last_peered":"2024-07-26T06:48:51.340182+0000","last_clean":"2024-07-26T06:48:51.340182+0000","last_became_active":"2024-07-26T06:47:05.027320+0000","last_became_peered":"2024-07-26T06:47:05.027320+0000","last_unstale":"2024-07-26T06:48:51.340182+0000","last_undegraded":"2024-07-26T06:48:51.340182+0000","last_fullsized":"2024-07-26T06:48:51.340182+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:43:06.242945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"80'7","reported_seq":228,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:31.427119+0000","last_change":"2024-07-26T06:47:01.998356+0000","last_active":"2024-07-26T06:50:31.427119+0000","last_peered":"2024-07-26T06:50:31.427119+0000","last_clean":"2024-07-26T06:50:31.427119+0000","last_became_active":"2024-07-26T06:47:01.997922+0000","last_became_peered":"2024-07-26T06:47:01.997922+0000","last_unstale":"2024-07-26T06:50:31.427119+0000","last_undegraded":"2024-07-26T06:50:31.427119+0000","last_fullsized":"2024-07-26T06:50:31.427119+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:50:58.066682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":194,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"80'6","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339434+0000","last_change":"2024-07-26T06:47:05.025285+0000","last_active":"2024-07-26T06:48:51.339434+0000","last_peered":"2024-07-26T06:48:51.339434+0000","last_clean":"2024-07-26T06:48:51.339434+0000","last_became_active":"2024-07-26T06:47:05.024945+0000","last_became_peered":"2024-07-26T06:47:05.024945+0000","last_unstale":"2024-07-26T06:48:51.339434+0000","last_undegraded":"2024-07-26T06:48:51.339434+0000","last_fullsized":"2024-07-26T06:48:51.339434+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:21:04.680435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.124278+0000","last_change":"2024-07-26T06:47:02.024895+0000","last_active":"2024-07-26T06:47:24.124278+0000","last_peered":"2024-07-26T06:47:24.124278+0000","last_clean":"2024-07-26T06:47:24.124278+0000","last_became_active":"2024-07-26T06:47:02.024451+0000","last_became_peered":"2024-07-26T06:47:02.024451+0000","last_unstale":"2024-07-26T06:47:24.124278+0000","last_undegraded":"2024-07-26T06:47:24.124278+0000","last_fullsized":"2024-07-26T06:47:24.124278+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:05:13.396499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"80'6","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335788+0000","last_change":"2024-07-26T06:47:23.381060+0000","last_active":"2024-07-26T06:48:51.335788+0000","last_peered":"2024-07-26T06:48:51.335788+0000","last_clean":"2024-07-26T06:48:51.335788+0000","last_became_active":"2024-07-26T06:47:23.380685+0000","last_became_peered":"2024-07-26T06:47:23.380685+0000","last_unstale":"2024-07-26T06:48:51.335788+0000","last_undegraded":"2024-07-26T06:48:51.335788+0000","last_fullsized":"2024-07-26T06:48:51.335788+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:21:38.062900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"78'2","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.124080+0000","last_change":"2024-07-26T06:47:02.002098+0000","last_active":"2024-07-26T06:47:24.124080+0000","last_peered":"2024-07-26T06:47:24.124080+0000","last_clean":"2024-07-26T06:47:24.124080+0000","last_became_active":"2024-07-26T06:47:02.001428+0000","last_became_peered":"2024-07-26T06:47:02.001428+0000","last_unstale":"2024-07-26T06:47:24.124080+0000","last_undegraded":"2024-07-26T06:47:24.124080+0000","last_fullsized":"2024-07-26T06:47:24.124080+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:52:29.112500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"80'5","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.330074+0000","last_change":"2024-07-26T06:47:05.023686+0000","last_active":"2024-07-26T06:48:51.330074+0000","last_peered":"2024-07-26T06:48:51.330074+0000","last_clean":"2024-07-26T06:48:51.330074+0000","last_became_active":"2024-07-26T06:47:05.022583+0000","last_became_peered":"2024-07-26T06:47:05.022583+0000","last_unstale":"2024-07-26T06:48:51.330074+0000","last_undegraded":"2024-07-26T06:48:51.330074+0000","last_fullsized":"2024-07-26T06:48:51.330074+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:29:03.752429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.096491+0000","last_change":"2024-07-26T06:47:02.006040+0000","last_active":"2024-07-26T06:47:23.096491+0000","last_peered":"2024-07-26T06:47:23.096491+0000","last_clean":"2024-07-26T06:47:23.096491+0000","last_became_active":"2024-07-26T06:47:02.005586+0000","last_became_peered":"2024-07-26T06:47:02.005586+0000","last_unstale":"2024-07-26T06:47:23.096491+0000","last_undegraded":"2024-07-26T06:47:23.096491+0000","last_fullsized":"2024-07-26T06:47:23.096491+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:09:27.180873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335338+0000","last_change":"2024-07-26T06:47:05.035524+0000","last_active":"2024-07-26T06:48:51.335338+0000","last_peered":"2024-07-26T06:48:51.335338+0000","last_clean":"2024-07-26T06:48:51.335338+0000","last_became_active":"2024-07-26T06:47:05.035078+0000","last_became_peered":"2024-07-26T06:47:05.035078+0000","last_unstale":"2024-07-26T06:48:51.335338+0000","last_undegraded":"2024-07-26T06:48:51.335338+0000","last_fullsized":"2024-07-26T06:48:51.335338+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:15:03.758261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.123534+0000","last_change":"2024-07-26T06:47:01.999810+0000","last_active":"2024-07-26T06:47:24.123534+0000","last_peered":"2024-07-26T06:47:24.123534+0000","last_clean":"2024-07-26T06:47:24.123534+0000","last_became_active":"2024-07-26T06:47:01.999460+0000","last_became_peered":"2024-07-26T06:47:01.999460+0000","last_unstale":"2024-07-26T06:47:24.123534+0000","last_undegraded":"2024-07-26T06:47:24.123534+0000","last_fullsized":"2024-07-26T06:47:24.123534+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:25:05.691900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.339690+0000","last_change":"2024-07-26T06:47:05.033631+0000","last_active":"2024-07-26T06:48:51.339690+0000","last_peered":"2024-07-26T06:48:51.339690+0000","last_clean":"2024-07-26T06:48:51.339690+0000","last_became_active":"2024-07-26T06:47:05.033266+0000","last_became_peered":"2024-07-26T06:47:05.033266+0000","last_unstale":"2024-07-26T06:48:51.339690+0000","last_undegraded":"2024-07-26T06:48:51.339690+0000","last_fullsized":"2024-07-26T06:48:51.339690+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:58:55.895271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"78'5","reported_seq":222,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:29.444901+0000","last_change":"2024-07-26T06:47:02.113525+0000","last_active":"2024-07-26T06:50:29.444901+0000","last_peered":"2024-07-26T06:50:29.444901+0000","last_clean":"2024-07-26T06:50:29.444901+0000","last_became_active":"2024-07-26T06:47:02.113112+0000","last_became_peered":"2024-07-26T06:47:02.113112+0000","last_unstale":"2024-07-26T06:50:29.444901+0000","last_undegraded":"2024-07-26T06:50:29.444901+0000","last_fullsized":"2024-07-26T06:50:29.444901+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:05:15.071264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":189,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338494+0000","last_change":"2024-07-26T06:47:05.026423+0000","last_active":"2024-07-26T06:48:51.338494+0000","last_peered":"2024-07-26T06:48:51.338494+0000","last_clean":"2024-07-26T06:48:51.338494+0000","last_became_active":"2024-07-26T06:47:05.026053+0000","last_became_peered":"2024-07-26T06:47:05.026053+0000","last_unstale":"2024-07-26T06:48:51.338494+0000","last_undegraded":"2024-07-26T06:48:51.338494+0000","last_fullsized":"2024-07-26T06:48:51.338494+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:07:20.174538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":103,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:29.151093+0000","last_change":"2024-07-26T06:45:40.184988+0000","last_active":"2024-07-26T06:48:29.151093+0000","last_peered":"2024-07-26T06:48:29.151093+0000","last_clean":"2024-07-26T06:48:29.151093+0000","last_became_active":"2024-07-26T06:45:40.178827+0000","last_became_peered":"2024-07-26T06:45:40.178827+0000","last_unstale":"2024-07-26T06:48:29.151093+0000","last_undegraded":"2024-07-26T06:48:29.151093+0000","last_fullsized":"2024-07-26T06:48:29.151093+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:42:28.257602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:42:28.257602+0000","last_clean_scrub_stamp":"2024-07-26T06:42:28.257602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:52:14.123696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.123395+0000","last_change":"2024-07-26T06:47:01.996610+0000","last_active":"2024-07-26T06:47:24.123395+0000","last_peered":"2024-07-26T06:47:24.123395+0000","last_clean":"2024-07-26T06:47:24.123395+0000","last_became_active":"2024-07-26T06:47:01.996155+0000","last_became_peered":"2024-07-26T06:47:01.996155+0000","last_unstale":"2024-07-26T06:47:24.123395+0000","last_undegraded":"2024-07-26T06:47:24.123395+0000","last_fullsized":"2024-07-26T06:47:24.123395+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:51:30.420254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333771+0000","last_change":"2024-07-26T06:47:05.027699+0000","last_active":"2024-07-26T06:48:51.333771+0000","last_peered":"2024-07-26T06:48:51.333771+0000","last_clean":"2024-07-26T06:48:51.333771+0000","last_became_active":"2024-07-26T06:47:05.027348+0000","last_became_peered":"2024-07-26T06:47:05.027348+0000","last_unstale":"2024-07-26T06:48:51.333771+0000","last_undegraded":"2024-07-26T06:48:51.333771+0000","last_fullsized":"2024-07-26T06:48:51.333771+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:28:42.664351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.097188+0000","last_change":"2024-07-26T06:47:02.003205+0000","last_active":"2024-07-26T06:47:23.097188+0000","last_peered":"2024-07-26T06:47:23.097188+0000","last_clean":"2024-07-26T06:47:23.097188+0000","last_became_active":"2024-07-26T06:47:02.002543+0000","last_became_peered":"2024-07-26T06:47:02.002543+0000","last_unstale":"2024-07-26T06:47:23.097188+0000","last_undegraded":"2024-07-26T06:47:23.097188+0000","last_fullsized":"2024-07-26T06:47:23.097188+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:43:10.911943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"80'5","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.334595+0000","last_change":"2024-07-26T06:47:05.023258+0000","last_active":"2024-07-26T06:48:51.334595+0000","last_peered":"2024-07-26T06:48:51.334595+0000","last_clean":"2024-07-26T06:48:51.334595+0000","last_became_active":"2024-07-26T06:47:05.022903+0000","last_became_peered":"2024-07-26T06:47:05.022903+0000","last_unstale":"2024-07-26T06:48:51.334595+0000","last_undegraded":"2024-07-26T06:48:51.334595+0000","last_fullsized":"2024-07-26T06:48:51.334595+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:08:08.155850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"80'6","reported_seq":222,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:30.424321+0000","last_change":"2024-07-26T06:47:02.022316+0000","last_active":"2024-07-26T06:50:30.424321+0000","last_peered":"2024-07-26T06:50:30.424321+0000","last_clean":"2024-07-26T06:50:30.424321+0000","last_became_active":"2024-07-26T06:47:02.021923+0000","last_became_peered":"2024-07-26T06:47:02.021923+0000","last_unstale":"2024-07-26T06:50:30.424321+0000","last_undegraded":"2024-07-26T06:50:30.424321+0000","last_fullsized":"2024-07-26T06:50:30.424321+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:29:31.533449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":189,"num_read_kb":189,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"80'8","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.328077+0000","last_change":"2024-07-26T06:47:05.021510+0000","last_active":"2024-07-26T06:48:51.328077+0000","last_peered":"2024-07-26T06:48:51.328077+0000","last_clean":"2024-07-26T06:48:51.328077+0000","last_became_active":"2024-07-26T06:47:05.021140+0000","last_became_peered":"2024-07-26T06:47:05.021140+0000","last_unstale":"2024-07-26T06:48:51.328077+0000","last_undegraded":"2024-07-26T06:48:51.328077+0000","last_fullsized":"2024-07-26T06:48:51.328077+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:37:01.340126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"79'5","reported_seq":223,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:32.437024+0000","last_change":"2024-07-26T06:47:02.001698+0000","last_active":"2024-07-26T06:50:32.437024+0000","last_peered":"2024-07-26T06:50:32.437024+0000","last_clean":"2024-07-26T06:50:32.437024+0000","last_became_active":"2024-07-26T06:47:02.001244+0000","last_became_peered":"2024-07-26T06:47:02.001244+0000","last_unstale":"2024-07-26T06:50:32.437024+0000","last_undegraded":"2024-07-26T06:50:32.437024+0000","last_fullsized":"2024-07-26T06:50:32.437024+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:32:07.920065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":191,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"80'7","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.323756+0000","last_change":"2024-07-26T06:47:05.029524+0000","last_active":"2024-07-26T06:48:51.323756+0000","last_peered":"2024-07-26T06:48:51.323756+0000","last_clean":"2024-07-26T06:48:51.323756+0000","last_became_active":"2024-07-26T06:47:05.028981+0000","last_became_peered":"2024-07-26T06:47:05.028981+0000","last_unstale":"2024-07-26T06:48:51.323756+0000","last_undegraded":"2024-07-26T06:48:51.323756+0000","last_fullsized":"2024-07-26T06:48:51.323756+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:49:57.019486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"78'1","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.092333+0000","last_change":"2024-07-26T06:47:02.025117+0000","last_active":"2024-07-26T06:47:23.092333+0000","last_peered":"2024-07-26T06:47:23.092333+0000","last_clean":"2024-07-26T06:47:23.092333+0000","last_became_active":"2024-07-26T06:47:02.024612+0000","last_became_peered":"2024-07-26T06:47:02.024612+0000","last_unstale":"2024-07-26T06:47:23.092333+0000","last_undegraded":"2024-07-26T06:47:23.092333+0000","last_fullsized":"2024-07-26T06:47:23.092333+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:15:44.999658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"80'5","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.324532+0000","last_change":"2024-07-26T06:47:05.015085+0000","last_active":"2024-07-26T06:48:51.324532+0000","last_peered":"2024-07-26T06:48:51.324532+0000","last_clean":"2024-07-26T06:48:51.324532+0000","last_became_active":"2024-07-26T06:47:05.014686+0000","last_became_peered":"2024-07-26T06:47:05.014686+0000","last_unstale":"2024-07-26T06:48:51.324532+0000","last_undegraded":"2024-07-26T06:48:51.324532+0000","last_fullsized":"2024-07-26T06:48:51.324532+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:39:30.392347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"80'4","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.120649+0000","last_change":"2024-07-26T06:47:23.111918+0000","last_active":"2024-07-26T06:47:24.120649+0000","last_peered":"2024-07-26T06:47:24.120649+0000","last_clean":"2024-07-26T06:47:24.120649+0000","last_became_active":"2024-07-26T06:47:23.099518+0000","last_became_peered":"2024-07-26T06:47:23.099518+0000","last_unstale":"2024-07-26T06:47:24.120649+0000","last_undegraded":"2024-07-26T06:47:24.120649+0000","last_fullsized":"2024-07-26T06:47:24.120649+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:41.012096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.332524+0000","last_change":"2024-07-26T06:47:05.019415+0000","last_active":"2024-07-26T06:48:51.332524+0000","last_peered":"2024-07-26T06:48:51.332524+0000","last_clean":"2024-07-26T06:48:51.332524+0000","last_became_active":"2024-07-26T06:47:05.019044+0000","last_became_peered":"2024-07-26T06:47:05.019044+0000","last_unstale":"2024-07-26T06:48:51.332524+0000","last_undegraded":"2024-07-26T06:48:51.332524+0000","last_fullsized":"2024-07-26T06:48:51.332524+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:02:37.228174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"80'16","reported_seq":44,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:13.791758+0000","last_change":"2024-07-26T06:47:02.006585+0000","last_active":"2024-07-26T06:49:13.791758+0000","last_peered":"2024-07-26T06:49:13.791758+0000","last_clean":"2024-07-26T06:49:13.791758+0000","last_became_active":"2024-07-26T06:47:02.006185+0000","last_became_peered":"2024-07-26T06:47:02.006185+0000","last_unstale":"2024-07-26T06:49:13.791758+0000","last_undegraded":"2024-07-26T06:49:13.791758+0000","last_fullsized":"2024-07-26T06:49:13.791758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:02:45.480712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"80'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.326954+0000","last_change":"2024-07-26T06:47:05.028840+0000","last_active":"2024-07-26T06:48:51.326954+0000","last_peered":"2024-07-26T06:48:51.326954+0000","last_clean":"2024-07-26T06:48:51.326954+0000","last_became_active":"2024-07-26T06:47:05.028487+0000","last_became_peered":"2024-07-26T06:47:05.028487+0000","last_unstale":"2024-07-26T06:48:51.326954+0000","last_undegraded":"2024-07-26T06:48:51.326954+0000","last_fullsized":"2024-07-26T06:48:51.326954+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T06:47:48.185227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.119901+0000","last_change":"2024-07-26T06:47:02.113691+0000","last_active":"2024-07-26T06:47:24.119901+0000","last_peered":"2024-07-26T06:47:24.119901+0000","last_clean":"2024-07-26T06:47:24.119901+0000","last_became_active":"2024-07-26T06:47:02.113342+0000","last_became_peered":"2024-07-26T06:47:02.113342+0000","last_unstale":"2024-07-26T06:47:24.119901+0000","last_undegraded":"2024-07-26T06:47:24.119901+0000","last_fullsized":"2024-07-26T06:47:24.119901+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:14:03.586564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"80'7","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.332634+0000","last_change":"2024-07-26T06:47:23.098420+0000","last_active":"2024-07-26T06:48:51.332634+0000","last_peered":"2024-07-26T06:48:51.332634+0000","last_clean":"2024-07-26T06:48:51.332634+0000","last_became_active":"2024-07-26T06:47:23.097981+0000","last_became_peered":"2024-07-26T06:47:23.097981+0000","last_unstale":"2024-07-26T06:48:51.332634+0000","last_undegraded":"2024-07-26T06:48:51.332634+0000","last_fullsized":"2024-07-26T06:48:51.332634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T09:38:40.155514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"78'2","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.091145+0000","last_change":"2024-07-26T06:47:02.024781+0000","last_active":"2024-07-26T06:47:23.091145+0000","last_peered":"2024-07-26T06:47:23.091145+0000","last_clean":"2024-07-26T06:47:23.091145+0000","last_became_active":"2024-07-26T06:47:02.024492+0000","last_became_peered":"2024-07-26T06:47:02.024492+0000","last_unstale":"2024-07-26T06:47:23.091145+0000","last_undegraded":"2024-07-26T06:47:23.091145+0000","last_fullsized":"2024-07-26T06:47:23.091145+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:23:41.323963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"80'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.310434+0000","last_change":"2024-07-26T06:47:05.033052+0000","last_active":"2024-07-26T06:48:51.310434+0000","last_peered":"2024-07-26T06:48:51.310434+0000","last_clean":"2024-07-26T06:48:51.310434+0000","last_became_active":"2024-07-26T06:47:05.032052+0000","last_became_peered":"2024-07-26T06:47:05.032052+0000","last_unstale":"2024-07-26T06:48:51.310434+0000","last_undegraded":"2024-07-26T06:48:51.310434+0000","last_fullsized":"2024-07-26T06:48:51.310434+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T08:42:04.683586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090616+0000","last_change":"2024-07-26T06:47:01.996097+0000","last_active":"2024-07-26T06:47:23.090616+0000","last_peered":"2024-07-26T06:47:23.090616+0000","last_clean":"2024-07-26T06:47:23.090616+0000","last_became_active":"2024-07-26T06:47:01.995594+0000","last_became_peered":"2024-07-26T06:47:01.995594+0000","last_unstale":"2024-07-26T06:47:23.090616+0000","last_undegraded":"2024-07-26T06:47:23.090616+0000","last_fullsized":"2024-07-26T06:47:23.090616+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:34.969493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.329013+0000","last_change":"2024-07-26T06:47:05.024171+0000","last_active":"2024-07-26T06:48:51.329013+0000","last_peered":"2024-07-26T06:48:51.329013+0000","last_clean":"2024-07-26T06:48:51.329013+0000","last_became_active":"2024-07-26T06:47:05.022888+0000","last_became_peered":"2024-07-26T06:47:05.022888+0000","last_unstale":"2024-07-26T06:48:51.329013+0000","last_undegraded":"2024-07-26T06:48:51.329013+0000","last_fullsized":"2024-07-26T06:48:51.329013+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:31:52.691068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"80'4","reported_seq":32,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:49:11.305169+0000","last_change":"2024-07-26T06:47:02.004578+0000","last_active":"2024-07-26T06:49:11.305169+0000","last_peered":"2024-07-26T06:49:11.305169+0000","last_clean":"2024-07-26T06:49:11.305169+0000","last_became_active":"2024-07-26T06:47:02.004231+0000","last_became_peered":"2024-07-26T06:47:02.004231+0000","last_unstale":"2024-07-26T06:49:11.305169+0000","last_undegraded":"2024-07-26T06:49:11.305169+0000","last_fullsized":"2024-07-26T06:49:11.305169+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:43:39.803817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"80'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.327862+0000","last_change":"2024-07-26T06:47:05.024232+0000","last_active":"2024-07-26T06:48:51.327862+0000","last_peered":"2024-07-26T06:48:51.327862+0000","last_clean":"2024-07-26T06:48:51.327862+0000","last_became_active":"2024-07-26T06:47:05.023869+0000","last_became_peered":"2024-07-26T06:47:05.023869+0000","last_unstale":"2024-07-26T06:48:51.327862+0000","last_undegraded":"2024-07-26T06:48:51.327862+0000","last_fullsized":"2024-07-26T06:48:51.327862+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:27:01.357311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"78'4","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.123564+0000","last_change":"2024-07-26T06:47:23.105051+0000","last_active":"2024-07-26T06:47:24.123564+0000","last_peered":"2024-07-26T06:47:24.123564+0000","last_clean":"2024-07-26T06:47:24.123564+0000","last_became_active":"2024-07-26T06:47:23.098116+0000","last_became_peered":"2024-07-26T06:47:23.098116+0000","last_unstale":"2024-07-26T06:47:24.123564+0000","last_undegraded":"2024-07-26T06:47:24.123564+0000","last_fullsized":"2024-07-26T06:47:24.123564+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:15:32.991870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"80'5","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338438+0000","last_change":"2024-07-26T06:47:05.026056+0000","last_active":"2024-07-26T06:48:51.338438+0000","last_peered":"2024-07-26T06:48:51.338438+0000","last_clean":"2024-07-26T06:48:51.338438+0000","last_became_active":"2024-07-26T06:47:05.025654+0000","last_became_peered":"2024-07-26T06:47:05.025654+0000","last_unstale":"2024-07-26T06:48:51.338438+0000","last_undegraded":"2024-07-26T06:48:51.338438+0000","last_fullsized":"2024-07-26T06:48:51.338438+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:36:20.886768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"80'15","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.338740+0000","last_change":"2024-07-26T06:47:05.026693+0000","last_active":"2024-07-26T06:48:51.338740+0000","last_peered":"2024-07-26T06:48:51.338740+0000","last_clean":"2024-07-26T06:48:51.338740+0000","last_became_active":"2024-07-26T06:47:05.026320+0000","last_became_peered":"2024-07-26T06:47:05.026320+0000","last_unstale":"2024-07-26T06:48:51.338740+0000","last_undegraded":"2024-07-26T06:48:51.338740+0000","last_fullsized":"2024-07-26T06:48:51.338740+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:52:30.776335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"80'5","reported_seq":21,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:24.124104+0000","last_change":"2024-07-26T06:47:23.107707+0000","last_active":"2024-07-26T06:47:24.124104+0000","last_peered":"2024-07-26T06:47:24.124104+0000","last_clean":"2024-07-26T06:47:24.124104+0000","last_became_active":"2024-07-26T06:47:23.099429+0000","last_became_peered":"2024-07-26T06:47:23.099429+0000","last_unstale":"2024-07-26T06:47:24.124104+0000","last_undegraded":"2024-07-26T06:47:24.124104+0000","last_fullsized":"2024-07-26T06:47:24.124104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T10:40:52.856756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"80'9","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.334900+0000","last_change":"2024-07-26T06:47:05.032319+0000","last_active":"2024-07-26T06:48:51.334900+0000","last_peered":"2024-07-26T06:48:51.334900+0000","last_clean":"2024-07-26T06:48:51.334900+0000","last_became_active":"2024-07-26T06:47:05.030305+0000","last_became_peered":"2024-07-26T06:47:05.030305+0000","last_unstale":"2024-07-26T06:48:51.334900+0000","last_undegraded":"2024-07-26T06:48:51.334900+0000","last_fullsized":"2024-07-26T06:48:51.334900+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:34:05.351917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"80'12","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:25.105155+0000","last_change":"2024-07-26T06:47:02.007812+0000","last_active":"2024-07-26T06:47:25.105155+0000","last_peered":"2024-07-26T06:47:25.105155+0000","last_clean":"2024-07-26T06:47:25.105155+0000","last_became_active":"2024-07-26T06:47:02.006965+0000","last_became_peered":"2024-07-26T06:47:02.006965+0000","last_unstale":"2024-07-26T06:47:25.105155+0000","last_undegraded":"2024-07-26T06:47:25.105155+0000","last_fullsized":"2024-07-26T06:47:25.105155+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:53.173273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1f","version":"80'6","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.335825+0000","last_change":"2024-07-26T06:47:05.027900+0000","last_active":"2024-07-26T06:48:51.335825+0000","last_peered":"2024-07-26T06:48:51.335825+0000","last_clean":"2024-07-26T06:48:51.335825+0000","last_became_active":"2024-07-26T06:47:05.024467+0000","last_became_peered":"2024-07-26T06:47:05.024467+0000","last_unstale":"2024-07-26T06:48:51.335825+0000","last_undegraded":"2024-07-26T06:48:51.335825+0000","last_fullsized":"2024-07-26T06:48:51.335825+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T07:05:00.793487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"78'2","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:47:23.090879+0000","last_change":"2024-07-26T06:47:02.022863+0000","last_active":"2024-07-26T06:47:23.090879+0000","last_peered":"2024-07-26T06:47:23.090879+0000","last_clean":"2024-07-26T06:47:23.090879+0000","last_became_active":"2024-07-26T06:47:02.022446+0000","last_became_peered":"2024-07-26T06:47:02.022446+0000","last_unstale":"2024-07-26T06:47:23.090879+0000","last_undegraded":"2024-07-26T06:47:23.090879+0000","last_fullsized":"2024-07-26T06:47:23.090879+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:14:48.649099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"80'5","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:48:51.333567+0000","last_change":"2024-07-26T06:47:05.023669+0000","last_active":"2024-07-26T06:48:51.333567+0000","last_peered":"2024-07-26T06:48:51.333567+0000","last_clean":"2024-07-26T06:48:51.333567+0000","last_became_active":"2024-07-26T06:47:05.023265+0000","last_became_peered":"2024-07-26T06:47:05.023265+0000","last_unstale":"2024-07-26T06:48:51.333567+0000","last_undegraded":"2024-07-26T06:48:51.333567+0000","last_fullsized":"2024-07-26T06:48:51.333567+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:03.992869+0000","last_clean_scrub_stamp":"2024-07-26T06:47:03.992869+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:56:45.278253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":232,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-07-26T06:50:32.436228+0000","last_change":"2024-07-26T06:47:02.006668+0000","last_active":"2024-07-26T06:50:32.436228+0000","last_peered":"2024-07-26T06:50:32.436228+0000","last_clean":"2024-07-26T06:50:32.436228+0000","last_became_active":"2024-07-26T06:47:02.006287+0000","last_became_peered":"2024-07-26T06:47:02.006287+0000","last_unstale":"2024-07-26T06:50:32.436228+0000","last_undegraded":"2024-07-26T06:50:32.436228+0000","last_fullsized":"2024-07-26T06:50:32.436228+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T06:47:00.971463+0000","last_clean_scrub_stamp":"2024-07-26T06:47:00.971463+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:28:05.722539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":191,"num_read_kb":191,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74491,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1939,"num_read_kb":1054,"num_write":262,"num_write_kb":154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1634,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":223473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250626,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710778,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":426222,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877907007,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1083786,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070531,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":425458,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266760,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":880640,"data_stored":425142,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462989,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":946176,"data_stored":1019335,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626514,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36196,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708956,"statfs":{"total":95995035648,"available":95957970944,"internally_reserved":0,"allocated":880640,"data_stored":425175,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822743,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":1018481,"data_compressed":42021,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986268,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":492265,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182496,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":897024,"data_stored":425000,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411429,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36208,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708944,"statfs":{"total":95995035648,"available":95957958656,"internally_reserved":0,"allocated":892928,"data_stored":427148,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":950272,"data_stored":492785,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738477,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":426514,"data_compressed":5755,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36266,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":67929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":68965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T06:50:36.251 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:50:36.525 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:36.772 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:36 smithi042 ceph-mon[32775]: pgmap v407: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-26T06:50:36.773 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:36 smithi042 ceph-mon[32775]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:50:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:36 smithi164 ceph-mon[36970]: pgmap v407: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-26T06:50:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:36 smithi164 ceph-mon[36970]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:50:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:36 smithi163 ceph-mon[36873]: pgmap v407: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-26T06:50:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:36 smithi163 ceph-mon[36873]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T06:50:38.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.194+0000 7faf39d6f640 1 -- 172.21.15.42:0/3746851059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf34102510 msgr2=0x7faf34102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:38.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.194+0000 7faf39d6f640 1 --2- 172.21.15.42:0/3746851059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf34102510 0x7faf34102920 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7faf24009900 tx=0x7faf2402a410 comp rx=0 tx=0).stop 2024-07-26T06:50:38.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.194+0000 7faf39d6f640 1 -- 172.21.15.42:0/3746851059 shutdown_connections 2024-07-26T06:50:38.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.194+0000 7faf39d6f640 1 --2- 172.21.15.42:0/3746851059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf34108750 0x7faf3410ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:38.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.194+0000 7faf39d6f640 1 --2- 172.21.15.42:0/3746851059 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf34107e00 0x7faf34108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:38.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.194+0000 7faf39d6f640 1 --2- 172.21.15.42:0/3746851059 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf34102510 0x7faf34102920 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:38.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.194+0000 7faf39d6f640 1 -- 172.21.15.42:0/3746851059 >> 172.21.15.42:0/3746851059 conn(0x7faf34079190 msgr2=0x7faf340795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:38.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.194+0000 7faf39d6f640 1 -- 172.21.15.42:0/3746851059 shutdown_connections 2024-07-26T06:50:38.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.194+0000 7faf39d6f640 1 -- 172.21.15.42:0/3746851059 wait complete. 2024-07-26T06:50:38.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.195+0000 7faf39d6f640 1 Processor -- start 2024-07-26T06:50:38.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.195+0000 7faf39d6f640 1 -- start start 2024-07-26T06:50:38.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.195+0000 7faf39d6f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf34102510 0x7faf34199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:38.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.195+0000 7faf39d6f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf34107e00 0x7faf3419a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:38.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.195+0000 7faf39d6f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf34108750 0x7faf3419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:38.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.195+0000 7faf39d6f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faf341134e0 con 0x7faf34102510 2024-07-26T06:50:38.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.195+0000 7faf39d6f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7faf34113360 con 0x7faf34107e00 2024-07-26T06:50:38.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.195+0000 7faf39d6f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faf34113660 con 0x7faf34108750 2024-07-26T06:50:38.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.195+0000 7faf337fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf34102510 0x7faf34199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:38.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.195+0000 7faf337fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf34102510 0x7faf34199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34656/0 (socket says 172.21.15.42:34656) 2024-07-26T06:50:38.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.195+0000 7faf337fe640 1 -- 172.21.15.42:0/828242537 learned_addr learned my addr 172.21.15.42:0/828242537 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:38.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.195+0000 7faf2bfff640 1 --2- 172.21.15.42:0/828242537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf34107e00 0x7faf3419a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:38.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.196+0000 7faf33fff640 1 --2- 172.21.15.42:0/828242537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf34108750 0x7faf3419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:38.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.196+0000 7faf337fe640 1 -- 172.21.15.42:0/828242537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf34108750 msgr2=0x7faf3419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:38.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.196+0000 7faf337fe640 1 --2- 172.21.15.42:0/828242537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf34108750 0x7faf3419f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:38.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.196+0000 7faf337fe640 1 -- 172.21.15.42:0/828242537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf34107e00 msgr2=0x7faf3419a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:38.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.196+0000 7faf337fe640 1 --2- 172.21.15.42:0/828242537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf34107e00 0x7faf3419a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:38.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.196+0000 7faf337fe640 1 -- 172.21.15.42:0/828242537 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf3419fe80 con 0x7faf34102510 2024-07-26T06:50:38.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.196+0000 7faf33fff640 1 --2- 172.21.15.42:0/828242537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf34108750 0x7faf3419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:38.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.196+0000 7faf337fe640 1 --2- 172.21.15.42:0/828242537 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf34102510 0x7faf34199f10 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7faf24005940 tx=0x7faf240042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:38.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.196+0000 7faf317fa640 1 -- 172.21.15.42:0/828242537 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf2403f030 con 0x7faf34102510 2024-07-26T06:50:38.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.196+0000 7faf39d6f640 1 -- 172.21.15.42:0/828242537 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf341a0110 con 0x7faf34102510 2024-07-26T06:50:38.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.196+0000 7faf317fa640 1 -- 172.21.15.42:0/828242537 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf24003cb0 con 0x7faf34102510 2024-07-26T06:50:38.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.196+0000 7faf317fa640 1 -- 172.21.15.42:0/828242537 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf24035070 con 0x7faf34102510 2024-07-26T06:50:38.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.196+0000 7faf39d6f640 1 -- 172.21.15.42:0/828242537 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf341a68d0 con 0x7faf34102510 2024-07-26T06:50:38.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.197+0000 7faf39d6f640 1 -- 172.21.15.42:0/828242537 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf34103740 con 0x7faf34102510 2024-07-26T06:50:38.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.199+0000 7faf317fa640 1 -- 172.21.15.42:0/828242537 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7faf24042030 con 0x7faf34102510 2024-07-26T06:50:38.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.200+0000 7faf317fa640 1 --2- 172.21.15.42:0/828242537 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faf200b3bd0 0x7faf200b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:38.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.200+0000 7faf2bfff640 1 --2- 172.21.15.42:0/828242537 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faf200b3bd0 0x7faf200b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:38.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.200+0000 7faf317fa640 1 -- 172.21.15.42:0/828242537 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7faf24104740 con 0x7faf34102510 2024-07-26T06:50:38.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.200+0000 7faf2bfff640 1 --2- 172.21.15.42:0/828242537 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faf200b3bd0 0x7faf200b60e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faf34100680 tx=0x7faf1c00b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:38.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.203+0000 7faf317fa640 1 -- 172.21.15.42:0/828242537 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7faf240cbd70 con 0x7faf34102510 2024-07-26T06:50:38.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.423+0000 7faf39d6f640 1 -- 172.21.15.42:0/828242537 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf3410a550 con 0x7faf34102510 2024-07-26T06:50:38.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.423+0000 7faf317fa640 1 -- 172.21.15.42:0/828242537 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11978 (secure 0 0 0) 0x7faf240cb670 con 0x7faf34102510 2024-07-26T06:50:38.424 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:50:38.424 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":29,"btime":"2024-07-26T06:50:27:493008+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:50:26.696932+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24620,24601,24593,24583,24578]},"id":1}]} 2024-07-26T06:50:38.424 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 29 2024-07-26T06:50:38.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.425+0000 7faf39d6f640 1 -- 172.21.15.42:0/828242537 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faf200b3bd0 msgr2=0x7faf200b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:38.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.425+0000 7faf39d6f640 1 --2- 172.21.15.42:0/828242537 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faf200b3bd0 0x7faf200b60e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faf34100680 tx=0x7faf1c00b410 comp rx=0 tx=0).stop 2024-07-26T06:50:38.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.425+0000 7faf39d6f640 1 -- 172.21.15.42:0/828242537 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf34102510 msgr2=0x7faf34199f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:38.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.425+0000 7faf39d6f640 1 --2- 172.21.15.42:0/828242537 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf34102510 0x7faf34199f10 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7faf24005940 tx=0x7faf240042f0 comp rx=0 tx=0).stop 2024-07-26T06:50:38.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.425+0000 7faf39d6f640 1 -- 172.21.15.42:0/828242537 shutdown_connections 2024-07-26T06:50:38.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.425+0000 7faf39d6f640 1 --2- 172.21.15.42:0/828242537 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faf200b3bd0 0x7faf200b60e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:38.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.425+0000 7faf39d6f640 1 --2- 172.21.15.42:0/828242537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf34108750 0x7faf3419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:38.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.425+0000 7faf39d6f640 1 --2- 172.21.15.42:0/828242537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faf34107e00 0x7faf3419a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:38.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.425+0000 7faf39d6f640 1 --2- 172.21.15.42:0/828242537 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf34102510 0x7faf34199f10 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:38.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.425+0000 7faf39d6f640 1 -- 172.21.15.42:0/828242537 >> 172.21.15.42:0/828242537 conn(0x7faf34079190 msgr2=0x7faf34100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:38.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.426+0000 7faf39d6f640 1 -- 172.21.15.42:0/828242537 shutdown_connections 2024-07-26T06:50:38.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:38.426+0000 7faf39d6f640 1 -- 172.21.15.42:0/828242537 wait complete. 2024-07-26T06:50:38.805 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:38 smithi042 ceph-mon[32775]: pgmap v408: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-26T06:50:38.806 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:38 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/828242537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:50:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:38 smithi164 ceph-mon[36970]: pgmap v408: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-26T06:50:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:38 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/828242537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:50:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:38 smithi163 ceph-mon[36873]: pgmap v408: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-07-26T06:50:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:38 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/828242537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:50:39.127 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:50:39.392 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.781+0000 7f5c09d90640 1 -- 172.21.15.42:0/2398761593 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c04101fd0 msgr2=0x7f5c04102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.781+0000 7f5c09d90640 1 --2- 172.21.15.42:0/2398761593 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c04101fd0 0x7f5c04102440 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4009900 tx=0x7f5bf402a410 comp rx=0 tx=0).stop 2024-07-26T06:50:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.781+0000 7f5c09d90640 1 -- 172.21.15.42:0/2398761593 shutdown_connections 2024-07-26T06:50:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.781+0000 7f5c09d90640 1 --2- 172.21.15.42:0/2398761593 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c041039f0 0x7f5c0410a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.781+0000 7f5c09d90640 1 --2- 172.21.15.42:0/2398761593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c04103050 0x7f5c04103460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.781+0000 7f5c09d90640 1 --2- 172.21.15.42:0/2398761593 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c04101fd0 0x7f5c04102440 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.781+0000 7f5c09d90640 1 -- 172.21.15.42:0/2398761593 >> 172.21.15.42:0/2398761593 conn(0x7f5c040fd910 msgr2=0x7f5c040ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.781+0000 7f5c09d90640 1 -- 172.21.15.42:0/2398761593 shutdown_connections 2024-07-26T06:50:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.781+0000 7f5c09d90640 1 -- 172.21.15.42:0/2398761593 wait complete. 2024-07-26T06:50:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.782+0000 7f5c09d90640 1 Processor -- start 2024-07-26T06:50:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.782+0000 7f5c09d90640 1 -- start start 2024-07-26T06:50:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.782+0000 7f5c09d90640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c04101fd0 0x7f5c0406ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.782+0000 7f5c09d90640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c04103050 0x7f5c0419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.782+0000 7f5c09d90640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c041039f0 0x7f5c041a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.782+0000 7f5c09d90640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0410dc50 con 0x7f5c04101fd0 2024-07-26T06:50:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.782+0000 7f5c09d90640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0410dad0 con 0x7f5c041039f0 2024-07-26T06:50:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.782+0000 7f5c09d90640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0410ddd0 con 0x7f5c04103050 2024-07-26T06:50:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.783+0000 7f5c037fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c04101fd0 0x7f5c0406ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.783+0000 7f5c037fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c04101fd0 0x7f5c0406ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34666/0 (socket says 172.21.15.42:34666) 2024-07-26T06:50:40.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.783+0000 7f5c037fe640 1 -- 172.21.15.42:0/101983497 learned_addr learned my addr 172.21.15.42:0/101983497 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.783+0000 7f5c03fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c041039f0 0x7f5c041a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.783+0000 7f5bfaffd640 1 --2- 172.21.15.42:0/101983497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c04103050 0x7f5c0419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.783+0000 7f5c037fe640 1 -- 172.21.15.42:0/101983497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c04103050 msgr2=0x7f5c0419e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.783+0000 7f5c037fe640 1 --2- 172.21.15.42:0/101983497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c04103050 0x7f5c0419e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.783+0000 7f5c037fe640 1 -- 172.21.15.42:0/101983497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c041039f0 msgr2=0x7f5c041a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.783+0000 7f5c037fe640 1 --2- 172.21.15.42:0/101983497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c041039f0 0x7f5c041a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.783+0000 7f5c037fe640 1 -- 172.21.15.42:0/101983497 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c041a1450 con 0x7f5c04101fd0 2024-07-26T06:50:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.783+0000 7f5c037fe640 1 --2- 172.21.15.42:0/101983497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c04101fd0 0x7f5c0406ae60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4006010 tx=0x7f5bf4004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.783+0000 7f5c017fa640 1 -- 172.21.15.42:0/101983497 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf403f030 con 0x7f5c04101fd0 2024-07-26T06:50:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.783+0000 7f5c09d90640 1 -- 172.21.15.42:0/101983497 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c041a16e0 con 0x7f5c04101fd0 2024-07-26T06:50:40.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.783+0000 7f5c017fa640 1 -- 172.21.15.42:0/101983497 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bf4003cb0 con 0x7f5c04101fd0 2024-07-26T06:50:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.783+0000 7f5c017fa640 1 -- 172.21.15.42:0/101983497 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4035070 con 0x7f5c04101fd0 2024-07-26T06:50:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.784+0000 7f5c09d90640 1 -- 172.21.15.42:0/101983497 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c041a1bd0 con 0x7f5c04101fd0 2024-07-26T06:50:40.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.786+0000 7f5c017fa640 1 -- 172.21.15.42:0/101983497 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f5bf4042030 con 0x7f5c04101fd0 2024-07-26T06:50:40.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.786+0000 7f5c10d7d740 1 Processor -- start 2024-07-26T06:50:40.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.786+0000 7f5c10d7d740 1 -- start start 2024-07-26T06:50:40.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5c10d7d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42f0f1b0 0x562c432b8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5c10d7d740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5c10d7d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42eee4d0 0x562c432b9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5c10d7d740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x562c432273b0 con 0x562c42ee62f0 2024-07-26T06:50:40.796 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5c10d7d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562c432168b0 con 0x562c42eee4d0 2024-07-26T06:50:40.796 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5c10d7d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x562c4321a8f0 con 0x562c42f0f1b0 2024-07-26T06:50:40.796 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5c037fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.796 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5c037fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432b8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34672/0 (socket says 172.21.15.42:34672) 2024-07-26T06:50:40.796 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5c037fe640 1 -- 172.21.15.42:0/105396860 learned_addr learned my addr 172.21.15.42:0/105396860 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:40.796 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5c03fff640 1 --2- 172.21.15.42:0/105396860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42f0f1b0 0x562c432b8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.796 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5bfaffd640 1 --2- 172.21.15.42:0/105396860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42eee4d0 0x562c432b9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5c017fa640 1 --2- 172.21.15.42:0/101983497 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5bf00b3bd0 0x7f5bf00b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5c037fe640 1 -- 172.21.15.42:0/105396860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42f0f1b0 msgr2=0x562c432b8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5c037fe640 1 --2- 172.21.15.42:0/105396860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42f0f1b0 0x562c432b8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5c037fe640 1 -- 172.21.15.42:0/105396860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42eee4d0 msgr2=0x562c432b9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5c037fe640 1 --2- 172.21.15.42:0/105396860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42eee4d0 0x562c432b9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5c037fe640 1 -- 172.21.15.42:0/105396860 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c431bd9a0 con 0x562c42ee62f0 2024-07-26T06:50:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5bfaffd640 1 --2- 172.21.15.42:0/105396860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42eee4d0 0x562c432b9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.787+0000 7f5c03fff640 1 --2- 172.21.15.42:0/101983497 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5bf00b3bd0 0x7f5bf00b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.788+0000 7f5c037fe640 1 --2- 172.21.15.42:0/105396860 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432b8c00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5bf40ce9a0 tx=0x7f5bf40c70c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa07531a05a8261e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.788+0000 7f5c017fa640 1 -- 172.21.15.42:0/101983497 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f5bf4104280 con 0x7f5c04101fd0 2024-07-26T06:50:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.788+0000 7f5bf9ffb640 1 -- 172.21.15.42:0/105396860 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4049020 con 0x562c42ee62f0 2024-07-26T06:50:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.788+0000 7f5bf9ffb640 1 -- 172.21.15.42:0/105396860 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bf40cad10 con 0x562c42ee62f0 2024-07-26T06:50:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.788+0000 7f5bf9ffb640 1 -- 172.21.15.42:0/105396860 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf40ca970 con 0x562c42ee62f0 2024-07-26T06:50:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.789+0000 7f5c10d7d740 1 -- 172.21.15.42:0/105396860 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 msgr2=0x562c432b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.789+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/105396860 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432b8c00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5bf40ce9a0 tx=0x7f5bf40c70c0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.789+0000 7f5c03fff640 1 --2- 172.21.15.42:0/101983497 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5bf00b3bd0 0x7f5bf00b60e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc013520 tx=0x7f5bfc004a90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.789+0000 7f5c10d7d740 1 -- 172.21.15.42:0/105396860 shutdown_connections 2024-07-26T06:50:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.789+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/105396860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42eee4d0 0x562c432b9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.789+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/105396860 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432b8c00 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.789+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/105396860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42f0f1b0 0x562c432b8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.789+0000 7f5c10d7d740 1 -- 172.21.15.42:0/105396860 >> 172.21.15.42:0/105396860 conn(0x562c42f03300 msgr2=0x562c432b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:40.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.789+0000 7f5c10d7d740 1 -- 172.21.15.42:0/105396860 shutdown_connections 2024-07-26T06:50:40.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.789+0000 7f5c10d7d740 1 -- 172.21.15.42:0/105396860 wait complete. 2024-07-26T06:50:40.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.789+0000 7f5c10d7d740 1 -- start start 2024-07-26T06:50:40.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.789+0000 7f5c10d7d740 1 Processor -- start 2024-07-26T06:50:40.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c10d7d740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432c1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c10d7d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42eee4d0 0x562c432c22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c10d7d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42f0f1b0 0x562c432c2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c10d7d740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x562c43150ab0 con 0x562c42ee62f0 2024-07-26T06:50:40.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c10d7d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562c4319ecf0 con 0x562c42f0f1b0 2024-07-26T06:50:40.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c10d7d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x562c432ba030 con 0x562c42eee4d0 2024-07-26T06:50:40.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c10d7d740 10 client.0 _ll_register_callbacks cb 0x562c432109f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:40.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c037fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432c1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c037fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432c1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34676/0 (socket says 172.21.15.42:34676) 2024-07-26T06:50:40.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c037fe640 1 -- 172.21.15.42:0/1771701675 learned_addr learned my addr 172.21.15.42:0/1771701675 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:40.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5bfaffd640 1 --2- 172.21.15.42:0/1771701675 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42eee4d0 0x562c432c22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c03fff640 1 --2- 172.21.15.42:0/1771701675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42f0f1b0 0x562c432c2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c037fe640 1 -- 172.21.15.42:0/1771701675 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42eee4d0 msgr2=0x562c432c22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:40.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c037fe640 1 --2- 172.21.15.42:0/1771701675 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42eee4d0 0x562c432c22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c037fe640 1 -- 172.21.15.42:0/1771701675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42f0f1b0 msgr2=0x562c432c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:40.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c037fe640 1 --2- 172.21.15.42:0/1771701675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42f0f1b0 0x562c432c2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.802 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c037fe640 1 -- 172.21.15.42:0/1771701675 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c42f7a790 con 0x562c42ee62f0 2024-07-26T06:50:40.802 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c03fff640 1 --2- 172.21.15.42:0/1771701675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42f0f1b0 0x562c432c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:50:40.802 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5c037fe640 1 --2- 172.21.15.42:0/1771701675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432c1c10 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4038020 tx=0x7f5bf403b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dfdc17d255d979b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:40.802 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.790+0000 7f5be2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:40.802 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.791+0000 7f5be2ffd640 1 -- 172.21.15.42:0/1771701675 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4049040 con 0x562c42ee62f0 2024-07-26T06:50:40.802 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.791+0000 7f5be2ffd640 1 -- 172.21.15.42:0/1771701675 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f5bf40c6b70 con 0x562c42ee62f0 2024-07-26T06:50:40.802 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.791+0000 7f5c10d7d740 1 -- 172.21.15.42:0/1771701675 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bf40cad10 con 0x562c42ee62f0 2024-07-26T06:50:40.802 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.791+0000 7f5be2ffd640 1 -- 172.21.15.42:0/1771701675 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5bf4107db0 con 0x562c42ee62f0 2024-07-26T06:50:40.802 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.791+0000 7f5c10d7d740 10 client.15348 fetch_fsmap learned FSMap version 29 2024-07-26T06:50:40.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.791+0000 7f5c10d7d740 1 -- 172.21.15.42:0/1771701675 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562c42ff3620 con 0x562c42ee62f0 2024-07-26T06:50:40.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.791+0000 7f5be2ffd640 1 -- 172.21.15.42:0/1771701675 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bf40cad10 con 0x562c42ee62f0 2024-07-26T06:50:40.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.791+0000 7f5be2ffd640 1 -- 172.21.15.42:0/1771701675 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5bf40b2070 con 0x562c42ee62f0 2024-07-26T06:50:40.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.791+0000 7f5c10d7d740 10 client.15348 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:50:40.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.791+0000 7f5c10d7d740 10 client.15348 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:50:40.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.791+0000 7f5c10d7d740 20 client.15348 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:40.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.791+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/1771701675 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x562c42ecdd80 0x562c432cf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.791+0000 7f5c10d7d740 4 client.15348 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:50:40.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.791+0000 7f5c10d7d740 1 -- 172.21.15.42:0/1771701675 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c43229300 con 0x562c42ecdd80 2024-07-26T06:50:40.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.792+0000 7f5bfaffd640 1 --2- 172.21.15.42:0/1771701675 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x562c42ecdd80 0x562c432cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.792+0000 7f5bfaffd640 1 --2- 172.21.15.42:0/1771701675 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x562c42ecdd80 0x562c432cf070 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b22d5fbeb939565 server_cookie=27c5826a90d4de61 in_seq=0 out_seq=0 2024-07-26T06:50:40.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.792+0000 7f5be2ffd640 10 client.15348 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:50:40.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.793+0000 7f5be2ffd640 1 -- 172.21.15.42:0/1771701675 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562c43229300 con 0x562c42ecdd80 2024-07-26T06:50:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.793+0000 7f5be2ffd640 10 client.15348 handle_command_reply: tid=0 2024-07-26T06:50:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.793+0000 7f5c10d7d740 1 client.15348 shutdown 2024-07-26T06:50:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.793+0000 7f5c10d7d740 1 -- 172.21.15.42:0/1771701675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 msgr2=0x562c432c1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.793+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/1771701675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432c1c10 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4038020 tx=0x7f5bf403b6b0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.793+0000 7f5c10d7d740 1 -- 172.21.15.42:0/1771701675 shutdown_connections 2024-07-26T06:50:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.793+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/1771701675 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x562c42ecdd80 0x562c432cf070 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.793+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/1771701675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42f0f1b0 0x562c432c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.793+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/1771701675 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42eee4d0 0x562c432c22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.793+0000 7f5be2ffd640 0 client.15348 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:50:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.793+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/1771701675 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432c1c10 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.793+0000 7f5c10d7d740 1 -- 172.21.15.42:0/1771701675 >> 172.21.15.42:0/1771701675 conn(0x562c42f03300 msgr2=0x562c432ae2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.793+0000 7f5c10d7d740 1 -- 172.21.15.42:0/1771701675 shutdown_connections 2024-07-26T06:50:40.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.793+0000 7f5c10d7d740 1 -- 172.21.15.42:0/1771701675 wait complete. 2024-07-26T06:50:40.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.793+0000 7f5c10d7d740 20 client.15348 trim_cache size 0 max 16384 2024-07-26T06:50:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.810+0000 7f5c10d7d740 1 Processor -- start 2024-07-26T06:50:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.810+0000 7f5c10d7d740 1 -- start start 2024-07-26T06:50:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.810+0000 7f5c10d7d740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ecdd80 0x562c432bc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.810+0000 7f5c10d7d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42ee62f0 0x562c432bcc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.810+0000 7f5c10d7d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42eee4d0 0x562c432c0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.810+0000 7f5c10d7d740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5c041a8880 con 0x562c42ecdd80 2024-07-26T06:50:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.810+0000 7f5c10d7d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5c041aa3c0 con 0x562c42eee4d0 2024-07-26T06:50:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.810+0000 7f5c10d7d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5c041a8f50 con 0x562c42ee62f0 2024-07-26T06:50:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.811+0000 7f5c037fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ecdd80 0x562c432bc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.811+0000 7f5c037fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ecdd80 0x562c432bc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34682/0 (socket says 172.21.15.42:34682) 2024-07-26T06:50:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.811+0000 7f5c037fe640 1 -- 172.21.15.42:0/2992261923 learned_addr learned my addr 172.21.15.42:0/2992261923 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.811+0000 7f5bfaffd640 1 --2- 172.21.15.42:0/2992261923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42ee62f0 0x562c432bcc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.811+0000 7f5c03fff640 1 --2- 172.21.15.42:0/2992261923 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42eee4d0 0x562c432c0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.811+0000 7f5c037fe640 1 -- 172.21.15.42:0/2992261923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42ee62f0 msgr2=0x562c432bcc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.811+0000 7f5c037fe640 1 --2- 172.21.15.42:0/2992261923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42ee62f0 0x562c432bcc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.811+0000 7f5c037fe640 1 -- 172.21.15.42:0/2992261923 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42eee4d0 msgr2=0x562c432c0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.811+0000 7f5c037fe640 1 --2- 172.21.15.42:0/2992261923 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42eee4d0 0x562c432c0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.811+0000 7f5c037fe640 1 -- 172.21.15.42:0/2992261923 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c432aec70 con 0x562c42ecdd80 2024-07-26T06:50:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.811+0000 7f5c037fe640 1 --2- 172.21.15.42:0/2992261923 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ecdd80 0x562c432bc6e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4005ef0 tx=0x7f5bf40cafd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a00a6f3f99f45835 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.812+0000 7f5bf97fa640 1 -- 172.21.15.42:0/2992261923 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4049070 con 0x562c42ecdd80 2024-07-26T06:50:40.814 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.812+0000 7f5bf97fa640 1 -- 172.21.15.42:0/2992261923 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bf40044b0 con 0x562c42ecdd80 2024-07-26T06:50:40.814 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.812+0000 7f5bf97fa640 1 -- 172.21.15.42:0/2992261923 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf40ce770 con 0x562c42ecdd80 2024-07-26T06:50:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.813+0000 7f5c10d7d740 1 -- 172.21.15.42:0/2992261923 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ecdd80 msgr2=0x562c432bc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.813+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/2992261923 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ecdd80 0x562c432bc6e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4005ef0 tx=0x7f5bf40cafd0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.813+0000 7f5c10d7d740 1 -- 172.21.15.42:0/2992261923 shutdown_connections 2024-07-26T06:50:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.813+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/2992261923 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42eee4d0 0x562c432c0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.813+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/2992261923 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42ee62f0 0x562c432bcc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.813+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/2992261923 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ecdd80 0x562c432bc6e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.813+0000 7f5c10d7d740 1 -- 172.21.15.42:0/2992261923 >> 172.21.15.42:0/2992261923 conn(0x562c42f03300 msgr2=0x562c432c01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.813+0000 7f5c10d7d740 1 -- 172.21.15.42:0/2992261923 shutdown_connections 2024-07-26T06:50:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.813+0000 7f5c10d7d740 1 -- 172.21.15.42:0/2992261923 wait complete. 2024-07-26T06:50:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.813+0000 7f5c10d7d740 1 -- start start 2024-07-26T06:50:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.813+0000 7f5c10d7d740 1 Processor -- start 2024-07-26T06:50:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.814+0000 7f5c10d7d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42ecdd80 0x562c432c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.814+0000 7f5c10d7d740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.814+0000 7f5c10d7d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42eee4d0 0x562c432c7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.814+0000 7f5c10d7d740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5c041a9cf0 con 0x562c42ee62f0 2024-07-26T06:50:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.814+0000 7f5c10d7d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5c041a9620 con 0x562c42eee4d0 2024-07-26T06:50:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.814+0000 7f5c10d7d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5c041a6860 con 0x562c42ecdd80 2024-07-26T06:50:40.819 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.814+0000 7f5c10d7d740 10 client.0 _ll_register_callbacks cb 0x7f5bfc009940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:40.819 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.814+0000 7f5bfaffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.819 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.814+0000 7f5bfaffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432c94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34696/0 (socket says 172.21.15.42:34696) 2024-07-26T06:50:40.819 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.814+0000 7f5bfaffd640 1 -- 172.21.15.42:0/1009092711 learned_addr learned my addr 172.21.15.42:0/1009092711 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:40.819 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.814+0000 7f5c037fe640 1 --2- 172.21.15.42:0/1009092711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42ecdd80 0x562c432c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.819 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.814+0000 7f5c03fff640 1 --2- 172.21.15.42:0/1009092711 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42eee4d0 0x562c432c7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.815+0000 7f5bfaffd640 1 -- 172.21.15.42:0/1009092711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42ecdd80 msgr2=0x562c432c8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.815+0000 7f5bfaffd640 1 --2- 172.21.15.42:0/1009092711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42ecdd80 0x562c432c8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.815+0000 7f5bfaffd640 1 -- 172.21.15.42:0/1009092711 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42eee4d0 msgr2=0x562c432c7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.815+0000 7f5bfaffd640 1 --2- 172.21.15.42:0/1009092711 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42eee4d0 0x562c432c7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.815+0000 7f5bfaffd640 1 -- 172.21.15.42:0/1009092711 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c432c7fb0 con 0x562c42ee62f0 2024-07-26T06:50:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.815+0000 7f5c03fff640 1 --2- 172.21.15.42:0/1009092711 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42eee4d0 0x562c432c7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.815+0000 7f5bfaffd640 1 --2- 172.21.15.42:0/1009092711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432c94b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x562c432cf750 tx=0x7f5bec00bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5563dc94c8f6411 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.815+0000 7f5be2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.815+0000 7f5be2ffd640 1 -- 172.21.15.42:0/1009092711 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec005250 con 0x562c42ee62f0 2024-07-26T06:50:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.815+0000 7f5be2ffd640 1 -- 172.21.15.42:0/1009092711 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f5bec0053f0 con 0x562c42ee62f0 2024-07-26T06:50:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.815+0000 7f5c10d7d740 1 -- 172.21.15.42:0/1009092711 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bf40044b0 con 0x562c42ee62f0 2024-07-26T06:50:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.815+0000 7f5be2ffd640 1 -- 172.21.15.42:0/1009092711 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5bec02d050 con 0x562c42ee62f0 2024-07-26T06:50:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.815+0000 7f5c10d7d740 10 client.15354 fetch_fsmap learned FSMap version 29 2024-07-26T06:50:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.815+0000 7f5c10d7d740 1 -- 172.21.15.42:0/1009092711 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562c43335fd0 con 0x562c42ee62f0 2024-07-26T06:50:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.815+0000 7f5be2ffd640 1 -- 172.21.15.42:0/1009092711 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5bf40044b0 con 0x562c42ee62f0 2024-07-26T06:50:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.816+0000 7f5be2ffd640 1 -- 172.21.15.42:0/1009092711 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5bec01d640 con 0x562c42ee62f0 2024-07-26T06:50:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.816+0000 7f5c10d7d740 10 client.15354 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:50:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.816+0000 7f5c10d7d740 10 client.15354 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:50:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.816+0000 7f5c10d7d740 20 client.15354 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.816+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/1009092711 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x562c42f0f1b0 0x562c433389d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.816+0000 7f5c10d7d740 4 client.15354 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:50:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.816+0000 7f5c10d7d740 1 -- 172.21.15.42:0/1009092711 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562c43224da0 con 0x562c42f0f1b0 2024-07-26T06:50:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.816+0000 7f5c037fe640 1 --2- 172.21.15.42:0/1009092711 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x562c42f0f1b0 0x562c433389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.817+0000 7f5c037fe640 1 --2- 172.21.15.42:0/1009092711 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x562c42f0f1b0 0x562c433389d0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b86421b88a37bec server_cookie=fc4af2f94803bd8c in_seq=0 out_seq=0 2024-07-26T06:50:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.817+0000 7f5be2ffd640 10 client.15354 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:50:40.826 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:40 smithi042 ceph-mon[32775]: pgmap v409: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T06:50:40.829 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.828+0000 7f5be2ffd640 1 -- 172.21.15.42:0/1009092711 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562c43224da0 con 0x562c42f0f1b0 2024-07-26T06:50:40.829 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.828+0000 7f5be2ffd640 10 client.15354 handle_command_reply: tid=0 2024-07-26T06:50:40.829 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.828+0000 7f5c10d7d740 1 client.15354 shutdown 2024-07-26T06:50:40.829 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.828+0000 7f5c10d7d740 1 -- 172.21.15.42:0/1009092711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 msgr2=0x562c432c94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:40.829 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.828+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/1009092711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432c94b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x562c432cf750 tx=0x7f5bec00bf60 comp rx=0 tx=0).stop 2024-07-26T06:50:40.829 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.828+0000 7f5c10d7d740 1 -- 172.21.15.42:0/1009092711 shutdown_connections 2024-07-26T06:50:40.829 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.828+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/1009092711 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x562c42f0f1b0 0x562c433389d0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.828+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/1009092711 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c42eee4d0 0x562c432c7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.828+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/1009092711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562c42ee62f0 0x562c432c94b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.828+0000 7f5be2ffd640 0 client.15354 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:50:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.828+0000 7f5c10d7d740 1 --2- 172.21.15.42:0/1009092711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562c42ecdd80 0x562c432c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.828+0000 7f5c10d7d740 1 -- 172.21.15.42:0/1009092711 >> 172.21.15.42:0/1009092711 conn(0x562c42f03300 msgr2=0x562c432bc6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.828+0000 7f5c10d7d740 1 -- 172.21.15.42:0/1009092711 shutdown_connections 2024-07-26T06:50:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.828+0000 7f5c10d7d740 1 -- 172.21.15.42:0/1009092711 wait complete. 2024-07-26T06:50:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.828+0000 7f5c10d7d740 20 client.15354 trim_cache size 0 max 16384 2024-07-26T06:50:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:50:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:50:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:50:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:50:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.829+0000 7f5c09d90640 1 -- 172.21.15.42:0/101983497 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5bf00b3bd0 msgr2=0x7f5bf00b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.829+0000 7f5c09d90640 1 --2- 172.21.15.42:0/101983497 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5bf00b3bd0 0x7f5bf00b60e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc013520 tx=0x7f5bfc004a90 comp rx=0 tx=0).stop 2024-07-26T06:50:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.829+0000 7f5c09d90640 1 -- 172.21.15.42:0/101983497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c04101fd0 msgr2=0x7f5c0406ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.829+0000 7f5c09d90640 1 --2- 172.21.15.42:0/101983497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c04101fd0 0x7f5c0406ae60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4006010 tx=0x7f5bf4004340 comp rx=0 tx=0).stop 2024-07-26T06:50:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.829+0000 7f5c09d90640 1 -- 172.21.15.42:0/101983497 shutdown_connections 2024-07-26T06:50:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.829+0000 7f5c09d90640 1 --2- 172.21.15.42:0/101983497 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5bf00b3bd0 0x7f5bf00b60e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.829+0000 7f5c09d90640 1 --2- 172.21.15.42:0/101983497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c041039f0 0x7f5c041a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.829+0000 7f5c09d90640 1 --2- 172.21.15.42:0/101983497 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c04103050 0x7f5c0419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.829+0000 7f5c09d90640 1 --2- 172.21.15.42:0/101983497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c04101fd0 0x7f5c0406ae60 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:40.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.829+0000 7f5c09d90640 1 -- 172.21.15.42:0/101983497 >> 172.21.15.42:0/101983497 conn(0x7f5c040fd910 msgr2=0x7f5c04106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:40.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.829+0000 7f5c09d90640 1 -- 172.21.15.42:0/101983497 shutdown_connections 2024-07-26T06:50:40.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:40.830+0000 7f5c09d90640 1 -- 172.21.15.42:0/101983497 wait complete. 2024-07-26T06:50:40.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:40 smithi164 ceph-mon[36970]: pgmap v409: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T06:50:41.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:40 smithi163 ceph-mon[36873]: pgmap v409: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T06:50:41.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.064+0000 7f77c64d8640 1 -- 172.21.15.42:0/1364904695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c0104bf0 msgr2=0x7f77c0105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:41.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.064+0000 7f77c64d8640 1 --2- 172.21.15.42:0/1364904695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c0104bf0 0x7f77c0105000 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f77b4009900 tx=0x7f77b402a410 comp rx=0 tx=0).stop 2024-07-26T06:50:41.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.064+0000 7f77c64d8640 1 -- 172.21.15.42:0/1364904695 shutdown_connections 2024-07-26T06:50:41.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.064+0000 7f77c64d8640 1 --2- 172.21.15.42:0/1364904695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77c0105540 0x7f77c010efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:41.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.064+0000 7f77c64d8640 1 --2- 172.21.15.42:0/1364904695 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c0104bf0 0x7f77c0105000 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:41.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.064+0000 7f77c64d8640 1 --2- 172.21.15.42:0/1364904695 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77c0102be0 0x7f77c0102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:41.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.064+0000 7f77c64d8640 1 -- 172.21.15.42:0/1364904695 >> 172.21.15.42:0/1364904695 conn(0x7f77c0079870 msgr2=0x7f77c0079c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:41.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.064+0000 7f77c64d8640 1 -- 172.21.15.42:0/1364904695 shutdown_connections 2024-07-26T06:50:41.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.064+0000 7f77c64d8640 1 -- 172.21.15.42:0/1364904695 wait complete. 2024-07-26T06:50:41.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.064+0000 7f77c64d8640 1 Processor -- start 2024-07-26T06:50:41.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.064+0000 7f77c64d8640 1 -- start start 2024-07-26T06:50:41.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.065+0000 7f77c64d8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77c0102be0 0x7f77c019acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:41.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.065+0000 7f77c64d8640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77c0104bf0 0x7f77c019b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:41.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.065+0000 7f77c64d8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c0105540 0x7f77c019b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:41.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.065+0000 7f77c64d8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f77c01123d0 con 0x7f77c0105540 2024-07-26T06:50:41.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.065+0000 7f77c64d8640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f77c0112250 con 0x7f77c0104bf0 2024-07-26T06:50:41.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.065+0000 7f77c64d8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f77c0112550 con 0x7f77c0102be0 2024-07-26T06:50:41.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.065+0000 7f77c4a75640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c0105540 0x7f77c019b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:41.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.065+0000 7f77c4a75640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c0105540 0x7f77c019b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34722/0 (socket says 172.21.15.42:34722) 2024-07-26T06:50:41.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.065+0000 7f77c4a75640 1 -- 172.21.15.42:0/1332545702 learned_addr learned my addr 172.21.15.42:0/1332545702 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:41.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.065+0000 7f77bffff640 1 --2- 172.21.15.42:0/1332545702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77c0102be0 0x7f77c019acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:41.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.065+0000 7f77bf7fe640 1 --2- 172.21.15.42:0/1332545702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77c0104bf0 0x7f77c019b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:41.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.065+0000 7f77c4a75640 1 -- 172.21.15.42:0/1332545702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77c0102be0 msgr2=0x7f77c019acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:41.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.065+0000 7f77c4a75640 1 --2- 172.21.15.42:0/1332545702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77c0102be0 0x7f77c019acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:41.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.065+0000 7f77c4a75640 1 -- 172.21.15.42:0/1332545702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77c0104bf0 msgr2=0x7f77c019b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:41.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.065+0000 7f77c4a75640 1 --2- 172.21.15.42:0/1332545702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77c0104bf0 0x7f77c019b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:41.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.065+0000 7f77c4a75640 1 -- 172.21.15.42:0/1332545702 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c01a5fd0 con 0x7f77c0105540 2024-07-26T06:50:41.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.065+0000 7f77bf7fe640 1 --2- 172.21.15.42:0/1332545702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77c0104bf0 0x7f77c019b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:50:41.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.066+0000 7f77c4a75640 1 --2- 172.21.15.42:0/1332545702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c0105540 0x7f77c019b740 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f77b800eaa0 tx=0x7f77b800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:41.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.066+0000 7f77bd7fa640 1 -- 172.21.15.42:0/1332545702 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b800cdb0 con 0x7f77c0105540 2024-07-26T06:50:41.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.066+0000 7f77bd7fa640 1 -- 172.21.15.42:0/1332545702 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77b8005020 con 0x7f77c0105540 2024-07-26T06:50:41.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.066+0000 7f77c64d8640 1 -- 172.21.15.42:0/1332545702 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77c01a62c0 con 0x7f77c0105540 2024-07-26T06:50:41.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.066+0000 7f77bd7fa640 1 -- 172.21.15.42:0/1332545702 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b80116d0 con 0x7f77c0105540 2024-07-26T06:50:41.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.066+0000 7f77c64d8640 1 -- 172.21.15.42:0/1332545702 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77c01a67b0 con 0x7f77c0105540 2024-07-26T06:50:41.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.067+0000 7f77c64d8640 1 -- 172.21.15.42:0/1332545702 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77c0102ff0 con 0x7f77c0105540 2024-07-26T06:50:41.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.069+0000 7f77bd7fa640 1 -- 172.21.15.42:0/1332545702 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f77b80051d0 con 0x7f77c0105540 2024-07-26T06:50:41.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.069+0000 7f77bd7fa640 1 --2- 172.21.15.42:0/1332545702 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f77a80b3bd0 0x7f77a80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:41.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.070+0000 7f77bd7fa640 1 -- 172.21.15.42:0/1332545702 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f77b80e4240 con 0x7f77c0105540 2024-07-26T06:50:41.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.070+0000 7f77bffff640 1 --2- 172.21.15.42:0/1332545702 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f77a80b3bd0 0x7f77a80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:41.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.070+0000 7f77bffff640 1 --2- 172.21.15.42:0/1332545702 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f77a80b3bd0 0x7f77a80b60e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f77b0005db0 tx=0x7f77b0005d40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:41.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.072+0000 7f77bd7fa640 1 -- 172.21.15.42:0/1332545702 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f77b8016030 con 0x7f77c0105540 2024-07-26T06:50:41.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.295+0000 7f77c64d8640 1 -- 172.21.15.42:0/1332545702 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77c0101cb0 con 0x7f77c0105540 2024-07-26T06:50:41.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.296+0000 7f77bd7fa640 1 -- 172.21.15.42:0/1332545702 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11978 (secure 0 0 0) 0x7f77b80ab870 con 0x7f77c0105540 2024-07-26T06:50:41.297 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:50:41.297 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":29,"btime":"2024-07-26T06:50:27:493008+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:50:26.696932+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24620,24601,24593,24583,24578]},"id":1}]} 2024-07-26T06:50:41.297 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 29 2024-07-26T06:50:41.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.298+0000 7f77c64d8640 1 -- 172.21.15.42:0/1332545702 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f77a80b3bd0 msgr2=0x7f77a80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:41.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.298+0000 7f77c64d8640 1 --2- 172.21.15.42:0/1332545702 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f77a80b3bd0 0x7f77a80b60e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f77b0005db0 tx=0x7f77b0005d40 comp rx=0 tx=0).stop 2024-07-26T06:50:41.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.298+0000 7f77c64d8640 1 -- 172.21.15.42:0/1332545702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c0105540 msgr2=0x7f77c019b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:41.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.298+0000 7f77c64d8640 1 --2- 172.21.15.42:0/1332545702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c0105540 0x7f77c019b740 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f77b800eaa0 tx=0x7f77b800ef60 comp rx=0 tx=0).stop 2024-07-26T06:50:41.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.298+0000 7f77c64d8640 1 -- 172.21.15.42:0/1332545702 shutdown_connections 2024-07-26T06:50:41.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.298+0000 7f77c64d8640 1 --2- 172.21.15.42:0/1332545702 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f77a80b3bd0 0x7f77a80b60e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:41.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.298+0000 7f77c64d8640 1 --2- 172.21.15.42:0/1332545702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c0105540 0x7f77c019b740 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:41.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.298+0000 7f77c64d8640 1 --2- 172.21.15.42:0/1332545702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77c0104bf0 0x7f77c019b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:41.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.298+0000 7f77c64d8640 1 --2- 172.21.15.42:0/1332545702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f77c0102be0 0x7f77c019acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:41.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.298+0000 7f77c64d8640 1 -- 172.21.15.42:0/1332545702 >> 172.21.15.42:0/1332545702 conn(0x7f77c0079870 msgr2=0x7f77c0105e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:41.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.298+0000 7f77c64d8640 1 -- 172.21.15.42:0/1332545702 shutdown_connections 2024-07-26T06:50:41.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:41.298+0000 7f77c64d8640 1 -- 172.21.15.42:0/1332545702 wait complete. 2024-07-26T06:50:41.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:41 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1332545702' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:50:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:41 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1332545702' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:50:42.005 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph osd dump --format=json 2024-07-26T06:50:42.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:41 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1332545702' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:50:42.271 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:42 smithi042 ceph-mon[32775]: pgmap v410: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 9 op/s 2024-07-26T06:50:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:42 smithi164 ceph-mon[36970]: pgmap v410: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 9 op/s 2024-07-26T06:50:43.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:42 smithi163 ceph-mon[36873]: pgmap v410: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 9 op/s 2024-07-26T06:50:44.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.001+0000 7f6ea9240640 1 -- 172.21.15.42:0/208089992 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ea4102490 msgr2=0x7f6ea41028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:44.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.001+0000 7f6ea9240640 1 --2- 172.21.15.42:0/208089992 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ea4102490 0x7f6ea41028a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6e94009ca0 tx=0x7f6e9402a8c0 comp rx=0 tx=0).stop 2024-07-26T06:50:44.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.001+0000 7f6ea9240640 1 -- 172.21.15.42:0/208089992 shutdown_connections 2024-07-26T06:50:44.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.001+0000 7f6ea9240640 1 --2- 172.21.15.42:0/208089992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6ea4108720 0x7f6ea410ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:44.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.001+0000 7f6ea9240640 1 --2- 172.21.15.42:0/208089992 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ea4107dd0 0x7f6ea41081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:44.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.001+0000 7f6ea9240640 1 --2- 172.21.15.42:0/208089992 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ea4102490 0x7f6ea41028a0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:44.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.001+0000 7f6ea9240640 1 -- 172.21.15.42:0/208089992 >> 172.21.15.42:0/208089992 conn(0x7f6ea4079160 msgr2=0x7f6ea4079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:44.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.001+0000 7f6ea9240640 1 -- 172.21.15.42:0/208089992 shutdown_connections 2024-07-26T06:50:44.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.001+0000 7f6ea9240640 1 -- 172.21.15.42:0/208089992 wait complete. 2024-07-26T06:50:44.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.002+0000 7f6ea9240640 1 Processor -- start 2024-07-26T06:50:44.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.002+0000 7f6ea9240640 1 -- start start 2024-07-26T06:50:44.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea9240640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ea4102490 0x7f6ea4199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:44.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea9240640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ea4107dd0 0x7f6ea419a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:44.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea9240640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6ea4108720 0x7f6ea419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:44.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea9240640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea41123a0 con 0x7f6ea4102490 2024-07-26T06:50:44.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea9240640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea4112220 con 0x7f6ea4108720 2024-07-26T06:50:44.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea9240640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea4112520 con 0x7f6ea4107dd0 2024-07-26T06:50:44.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea2d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ea4102490 0x7f6ea4199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:44.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea2d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ea4102490 0x7f6ea4199ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34740/0 (socket says 172.21.15.42:34740) 2024-07-26T06:50:44.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea2d9d640 1 -- 172.21.15.42:0/2017585036 learned_addr learned my addr 172.21.15.42:0/2017585036 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:44.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea259c640 1 --2- 172.21.15.42:0/2017585036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ea4107dd0 0x7f6ea419a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:44.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea359e640 1 --2- 172.21.15.42:0/2017585036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6ea4108720 0x7f6ea419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:44.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea2d9d640 1 -- 172.21.15.42:0/2017585036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ea4107dd0 msgr2=0x7f6ea419a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:44.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea2d9d640 1 --2- 172.21.15.42:0/2017585036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ea4107dd0 0x7f6ea419a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:44.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea2d9d640 1 -- 172.21.15.42:0/2017585036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6ea4108720 msgr2=0x7f6ea419f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:44.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea2d9d640 1 --2- 172.21.15.42:0/2017585036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6ea4108720 0x7f6ea419f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:44.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea2d9d640 1 -- 172.21.15.42:0/2017585036 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea419fe10 con 0x7f6ea4102490 2024-07-26T06:50:44.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea359e640 1 --2- 172.21.15.42:0/2017585036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6ea4108720 0x7f6ea419f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:44.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea259c640 1 --2- 172.21.15.42:0/2017585036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ea4107dd0 0x7f6ea419a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:44.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.003+0000 7f6ea2d9d640 1 --2- 172.21.15.42:0/2017585036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ea4102490 0x7f6ea4199ea0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6e9402ade0 tx=0x7f6e94031770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:44.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.004+0000 7f6e9bfff640 1 -- 172.21.15.42:0/2017585036 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e94003b70 con 0x7f6ea4102490 2024-07-26T06:50:44.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.004+0000 7f6ea9240640 1 -- 172.21.15.42:0/2017585036 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea41a00a0 con 0x7f6ea4102490 2024-07-26T06:50:44.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.004+0000 7f6e9bfff640 1 -- 172.21.15.42:0/2017585036 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e94003d10 con 0x7f6ea4102490 2024-07-26T06:50:44.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.004+0000 7f6e9bfff640 1 -- 172.21.15.42:0/2017585036 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9402fa50 con 0x7f6ea4102490 2024-07-26T06:50:44.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.004+0000 7f6ea9240640 1 -- 172.21.15.42:0/2017585036 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea41a6860 con 0x7f6ea4102490 2024-07-26T06:50:44.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.005+0000 7f6ea9240640 1 -- 172.21.15.42:0/2017585036 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ea41028a0 con 0x7f6ea4102490 2024-07-26T06:50:44.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.006+0000 7f6e9bfff640 1 -- 172.21.15.42:0/2017585036 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f6e94004580 con 0x7f6ea4102490 2024-07-26T06:50:44.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.007+0000 7f6e9bfff640 1 --2- 172.21.15.42:0/2017585036 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e900b3bd0 0x7f6e900b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:44.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.007+0000 7f6ea259c640 1 --2- 172.21.15.42:0/2017585036 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e900b3bd0 0x7f6e900b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:44.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.007+0000 7f6e9bfff640 1 -- 172.21.15.42:0/2017585036 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f6e941053c0 con 0x7f6ea4102490 2024-07-26T06:50:44.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.008+0000 7f6ea259c640 1 --2- 172.21.15.42:0/2017585036 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e900b3bd0 0x7f6e900b60e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c0058c0 tx=0x7f6e8c005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:44.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.010+0000 7f6e9bfff640 1 -- 172.21.15.42:0/2017585036 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6e940cca20 con 0x7f6ea4102490 2024-07-26T06:50:44.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.197+0000 7f6ea9240640 1 -- 172.21.15.42:0/2017585036 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6ea419afd0 con 0x7f6ea4102490 2024-07-26T06:50:44.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.199+0000 7f6e9bfff640 1 -- 172.21.15.42:0/2017585036 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+22697 (secure 0 0 0) 0x7f6e940cc320 con 0x7f6ea4102490 2024-07-26T06:50:44.200 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:50:44.200 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":80,"fsid":"8b3e2750-4b19-11ef-bcb6-c7b262605968","created":"2024-07-26T06:38:29.306470+0000","modified":"2024-07-26T06:47:23.086609+0000","last_up_change":"2024-07-26T06:46:02.149453+0000","last_in_change":"2024-07-26T06:45:43.027764+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T06:42:27.302195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T06:47:00.042262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T06:47:03.674160+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1bda88e0-09ee-447b-8c04-24f4e5d55d19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6803","nonce":3587066379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6805","nonce":3587066379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6809","nonce":3587066379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":3587066379},{"type":"v1","addr":"172.21.15.42:6807","nonce":3587066379}]},"public_addr":"172.21.15.42:6803/3587066379","cluster_addr":"172.21.15.42:6805/3587066379","heartbeat_back_addr":"172.21.15.42:6809/3587066379","heartbeat_front_addr":"172.21.15.42:6807/3587066379","state":["exists","up"]},{"osd":1,"uuid":"7d747138-8217-4836-a173-cf8ac65e9fe3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6801","nonce":360219078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6803","nonce":360219078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6807","nonce":360219078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":360219078},{"type":"v1","addr":"172.21.15.163:6805","nonce":360219078}]},"public_addr":"172.21.15.163:6801/360219078","cluster_addr":"172.21.15.163:6803/360219078","heartbeat_back_addr":"172.21.15.163:6807/360219078","heartbeat_front_addr":"172.21.15.163:6805/360219078","state":["exists","up"]},{"osd":2,"uuid":"4b7258c1-3d35-4930-ab10-0b04a2a2c8a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6801","nonce":336113412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6803","nonce":336113412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6807","nonce":336113412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":336113412},{"type":"v1","addr":"172.21.15.164:6805","nonce":336113412}]},"public_addr":"172.21.15.164:6801/336113412","cluster_addr":"172.21.15.164:6803/336113412","heartbeat_back_addr":"172.21.15.164:6807/336113412","heartbeat_front_addr":"172.21.15.164:6805/336113412","state":["exists","up"]},{"osd":3,"uuid":"96608387-0de0-4d90-a952-d6eba61f7fbe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6811","nonce":2819018064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6813","nonce":2819018064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6817","nonce":2819018064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":2819018064},{"type":"v1","addr":"172.21.15.42:6815","nonce":2819018064}]},"public_addr":"172.21.15.42:6811/2819018064","cluster_addr":"172.21.15.42:6813/2819018064","heartbeat_back_addr":"172.21.15.42:6817/2819018064","heartbeat_front_addr":"172.21.15.42:6815/2819018064","state":["exists","up"]},{"osd":4,"uuid":"fabf6b02-fe34-4560-85b5-01bf37fc9be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6809","nonce":431934374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6811","nonce":431934374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6815","nonce":431934374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":431934374},{"type":"v1","addr":"172.21.15.163:6813","nonce":431934374}]},"public_addr":"172.21.15.163:6809/431934374","cluster_addr":"172.21.15.163:6811/431934374","heartbeat_back_addr":"172.21.15.163:6815/431934374","heartbeat_front_addr":"172.21.15.163:6813/431934374","state":["exists","up"]},{"osd":5,"uuid":"e226fb4f-15b2-40df-af34-900040ba93ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6809","nonce":261549362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6811","nonce":261549362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6815","nonce":261549362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":261549362},{"type":"v1","addr":"172.21.15.164:6813","nonce":261549362}]},"public_addr":"172.21.15.164:6809/261549362","cluster_addr":"172.21.15.164:6811/261549362","heartbeat_back_addr":"172.21.15.164:6815/261549362","heartbeat_front_addr":"172.21.15.164:6813/261549362","state":["exists","up"]},{"osd":6,"uuid":"beb8ec5d-7ea5-4e4e-a46c-a84f7768cec0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6819","nonce":2160811300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6821","nonce":2160811300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6825","nonce":2160811300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":2160811300},{"type":"v1","addr":"172.21.15.42:6823","nonce":2160811300}]},"public_addr":"172.21.15.42:6819/2160811300","cluster_addr":"172.21.15.42:6821/2160811300","heartbeat_back_addr":"172.21.15.42:6825/2160811300","heartbeat_front_addr":"172.21.15.42:6823/2160811300","state":["exists","up"]},{"osd":7,"uuid":"f699004c-f7c2-4f1c-ae4b-7178f3b06e2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6817","nonce":812214111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6819","nonce":812214111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6823","nonce":812214111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":812214111},{"type":"v1","addr":"172.21.15.163:6821","nonce":812214111}]},"public_addr":"172.21.15.163:6817/812214111","cluster_addr":"172.21.15.163:6819/812214111","heartbeat_back_addr":"172.21.15.163:6823/812214111","heartbeat_front_addr":"172.21.15.163:6821/812214111","state":["exists","up"]},{"osd":8,"uuid":"3476163e-efea-4652-a8d4-b7c4a908f984","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6817","nonce":962845839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6819","nonce":962845839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6823","nonce":962845839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":962845839},{"type":"v1","addr":"172.21.15.164:6821","nonce":962845839}]},"public_addr":"172.21.15.164:6817/962845839","cluster_addr":"172.21.15.164:6819/962845839","heartbeat_back_addr":"172.21.15.164:6823/962845839","heartbeat_front_addr":"172.21.15.164:6821/962845839","state":["exists","up"]},{"osd":9,"uuid":"6f7ceac6-fb61-4c6f-9da8-851b937c4586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6827","nonce":994024418}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6828","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6829","nonce":994024418}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6832","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6833","nonce":994024418}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6830","nonce":994024418},{"type":"v1","addr":"172.21.15.42:6831","nonce":994024418}]},"public_addr":"172.21.15.42:6827/994024418","cluster_addr":"172.21.15.42:6829/994024418","heartbeat_back_addr":"172.21.15.42:6833/994024418","heartbeat_front_addr":"172.21.15.42:6831/994024418","state":["exists","up"]},{"osd":10,"uuid":"553132c2-a6a1-410c-a7d8-ecdf9a67dafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6825","nonce":3268512227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6827","nonce":3268512227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6831","nonce":3268512227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":3268512227},{"type":"v1","addr":"172.21.15.163:6829","nonce":3268512227}]},"public_addr":"172.21.15.163:6825/3268512227","cluster_addr":"172.21.15.163:6827/3268512227","heartbeat_back_addr":"172.21.15.163:6831/3268512227","heartbeat_front_addr":"172.21.15.163:6829/3268512227","state":["exists","up"]},{"osd":11,"uuid":"09abc8e9-7884-400f-ab6d-8e5aa7bf31ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6825","nonce":1861076336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6827","nonce":1861076336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6831","nonce":1861076336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":1861076336},{"type":"v1","addr":"172.21.15.164:6829","nonce":1861076336}]},"public_addr":"172.21.15.164:6825/1861076336","cluster_addr":"172.21.15.164:6827/1861076336","heartbeat_back_addr":"172.21.15.164:6831/1861076336","heartbeat_front_addr":"172.21.15.164:6829/1861076336","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:41:38.761674+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:00.621745+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:23.248620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:42:47.232764+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:10.379922+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:34.502519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:43:58.694642+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:22.350247+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:44:46.988007+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:11.390474+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:45:36.150054+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T06:46:00.474470+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.42:6801/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6800/118189018":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1842521399":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/834172950":"2024-07-27T06:39:21.217122+0000","172.21.15.42:0/1730109419":"2024-07-27T06:39:21.217122+0000","172.21.15.42:6801/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:6800/2946137308":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2687772889":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/3371433307":"2024-07-27T06:38:56.887264+0000","172.21.15.42:0/2861719816":"2024-07-27T06:38:56.887264+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T06:50:44.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.202+0000 7f6ea9240640 1 -- 172.21.15.42:0/2017585036 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e900b3bd0 msgr2=0x7f6e900b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:44.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.202+0000 7f6ea9240640 1 --2- 172.21.15.42:0/2017585036 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e900b3bd0 0x7f6e900b60e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c0058c0 tx=0x7f6e8c005850 comp rx=0 tx=0).stop 2024-07-26T06:50:44.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.202+0000 7f6ea9240640 1 -- 172.21.15.42:0/2017585036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ea4102490 msgr2=0x7f6ea4199ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:44.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.202+0000 7f6ea9240640 1 --2- 172.21.15.42:0/2017585036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ea4102490 0x7f6ea4199ea0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6e9402ade0 tx=0x7f6e94031770 comp rx=0 tx=0).stop 2024-07-26T06:50:44.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.202+0000 7f6ea9240640 1 -- 172.21.15.42:0/2017585036 shutdown_connections 2024-07-26T06:50:44.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.202+0000 7f6ea9240640 1 --2- 172.21.15.42:0/2017585036 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e900b3bd0 0x7f6e900b60e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:44.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.202+0000 7f6ea9240640 1 --2- 172.21.15.42:0/2017585036 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6ea4108720 0x7f6ea419f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:44.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.202+0000 7f6ea9240640 1 --2- 172.21.15.42:0/2017585036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ea4107dd0 0x7f6ea419a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:44.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.202+0000 7f6ea9240640 1 --2- 172.21.15.42:0/2017585036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ea4102490 0x7f6ea4199ea0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:44.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.202+0000 7f6ea9240640 1 -- 172.21.15.42:0/2017585036 >> 172.21.15.42:0/2017585036 conn(0x7f6ea4079160 msgr2=0x7f6ea41013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:44.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.202+0000 7f6ea9240640 1 -- 172.21.15.42:0/2017585036 shutdown_connections 2024-07-26T06:50:44.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:44.202+0000 7f6ea9240640 1 -- 172.21.15.42:0/2017585036 wait complete. 2024-07-26T06:50:44.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:44 smithi163 ceph-mon[36873]: pgmap v411: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 853 B/s wr, 11 op/s 2024-07-26T06:50:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:44 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2017585036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:50:44.837 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:44 smithi042 ceph-mon[32775]: pgmap v411: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 853 B/s wr, 11 op/s 2024-07-26T06:50:44.837 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:44 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2017585036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:50:44.868 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-07-26T06:50:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:44 smithi164 ceph-mon[36970]: pgmap v411: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 853 B/s wr, 11 op/s 2024-07-26T06:50:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:44 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2017585036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T06:50:45.137 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:46.633 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:46 smithi042 ceph-mon[32775]: pgmap v412: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-07-26T06:50:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:46 smithi164 ceph-mon[36970]: pgmap v412: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-07-26T06:50:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:46 smithi163 ceph-mon[36873]: pgmap v412: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-07-26T06:50:47.609 INFO:teuthology.orchestra.run.smithi042.stdout:{"run_dir":"/var/run/ceph"} 2024-07-26T06:50:47.610 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-07-26T06:50:47.610 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 5532 2024-07-26T06:50:47.610 DEBUG:tasks.quiescer:Yielding 2024-07-26T06:50:47.610 INFO:teuthology.run_tasks:Running task workunit... 2024-07-26T06:50:47.625 INFO:tasks.workunit:Pulling workunits from ref e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T06:50:47.625 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-07-26T06:50:47.625 INFO:tasks.workunit:timeout=3h 2024-07-26T06:50:47.626 INFO:tasks.workunit:cleanup=True 2024-07-26T06:50:47.626 DEBUG:teuthology.orchestra.run.smithi042:> stat -- /home/ubuntu/cephtest/mnt.0 2024-07-26T06:50:47.627 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:50:47.648 INFO:teuthology.orchestra.run.smithi042.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-07-26T06:50:47.648 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-07-26T06:50:47.648 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-07-26T06:50:47.649 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T06:50:47.649 INFO:teuthology.orchestra.run.smithi042.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-07-26T06:50:47.649 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-07-26 06:47:37.658738442 +0000 2024-07-26T06:50:47.650 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-07-26 06:47:37.658738442 +0000 2024-07-26T06:50:47.650 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-07-26 06:50:25.973756530 +0000 2024-07-26T06:50:47.650 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: 2024-07-26 06:47:37.658738442 +0000 2024-07-26T06:50:47.650 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-07-26T06:50:47.650 DEBUG:teuthology.orchestra.run.smithi042:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-07-26T06:50:47.727 DEBUG:teuthology.orchestra.run.smithi042:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T06:50:47.784 INFO:tasks.workunit.client.0.smithi042.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-07-26T06:50:47.912 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:48.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:48 smithi042 ceph-mon[32775]: pgmap v413: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.3 KiB/s wr, 13 op/s 2024-07-26T06:50:48.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:48 smithi164 ceph-mon[36970]: pgmap v413: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.3 KiB/s wr, 13 op/s 2024-07-26T06:50:49.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:48 smithi163 ceph-mon[36873]: pgmap v413: 97 pgs: 97 active+clean; 650 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.3 KiB/s wr, 13 op/s 2024-07-26T06:50:49.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.572+0000 7fe6ee338640 1 -- 172.21.15.42:0/2523227856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e81035f0 msgr2=0x7fe6e8103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:49.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.572+0000 7fe6ee338640 1 --2- 172.21.15.42:0/2523227856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e81035f0 0x7fe6e8103a00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8009900 tx=0x7fe6d802a410 comp rx=0 tx=0).stop 2024-07-26T06:50:49.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.572+0000 7fe6ee338640 1 -- 172.21.15.42:0/2523227856 shutdown_connections 2024-07-26T06:50:49.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.572+0000 7fe6ee338640 1 --2- 172.21.15.42:0/2523227856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6e8109830 0x7fe6e8110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:49.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.572+0000 7fe6ee338640 1 --2- 172.21.15.42:0/2523227856 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe6e8108ee0 0x7fe6e81092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:49.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.572+0000 7fe6ee338640 1 --2- 172.21.15.42:0/2523227856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e81035f0 0x7fe6e8103a00 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:49.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.572+0000 7fe6ee338640 1 -- 172.21.15.42:0/2523227856 >> 172.21.15.42:0/2523227856 conn(0x7fe6e807b340 msgr2=0x7fe6e807b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:49.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.572+0000 7fe6ee338640 1 -- 172.21.15.42:0/2523227856 shutdown_connections 2024-07-26T06:50:49.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.572+0000 7fe6ee338640 1 -- 172.21.15.42:0/2523227856 wait complete. 2024-07-26T06:50:49.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.573+0000 7fe6ee338640 1 Processor -- start 2024-07-26T06:50:49.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.573+0000 7fe6ee338640 1 -- start start 2024-07-26T06:50:49.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6ee338640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e81035f0 0x7fe6e8199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:49.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6ee338640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe6e8108ee0 0x7fe6e819a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:49.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6ee338640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6e8109830 0x7fe6e819f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:49.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6ee338640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e81134e0 con 0x7fe6e81035f0 2024-07-26T06:50:49.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6ee338640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e8113360 con 0x7fe6e8108ee0 2024-07-26T06:50:49.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6ee338640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e8113660 con 0x7fe6e8109830 2024-07-26T06:50:49.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6e7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e81035f0 0x7fe6e8199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:49.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6e7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e81035f0 0x7fe6e8199e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54442/0 (socket says 172.21.15.42:54442) 2024-07-26T06:50:49.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6e7fff640 1 -- 172.21.15.42:0/1654170307 learned_addr learned my addr 172.21.15.42:0/1654170307 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:49.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6ec8d5640 1 --2- 172.21.15.42:0/1654170307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6e8109830 0x7fe6e819f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:49.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6e77fe640 1 --2- 172.21.15.42:0/1654170307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe6e8108ee0 0x7fe6e819a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:49.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6e7fff640 1 -- 172.21.15.42:0/1654170307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6e8109830 msgr2=0x7fe6e819f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:49.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6e7fff640 1 --2- 172.21.15.42:0/1654170307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6e8109830 0x7fe6e819f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:49.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6e7fff640 1 -- 172.21.15.42:0/1654170307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe6e8108ee0 msgr2=0x7fe6e819a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:49.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6e7fff640 1 --2- 172.21.15.42:0/1654170307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe6e8108ee0 0x7fe6e819a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:49.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6e7fff640 1 -- 172.21.15.42:0/1654170307 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6e819fd20 con 0x7fe6e81035f0 2024-07-26T06:50:49.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6e77fe640 1 --2- 172.21.15.42:0/1654170307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe6e8108ee0 0x7fe6e819a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:50:49.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.574+0000 7fe6e7fff640 1 --2- 172.21.15.42:0/1654170307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e81035f0 0x7fe6e8199e70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8006010 tx=0x7fe6d80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:49.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.575+0000 7fe6e57fa640 1 -- 172.21.15.42:0/1654170307 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d803f030 con 0x7fe6e81035f0 2024-07-26T06:50:49.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.575+0000 7fe6ee338640 1 -- 172.21.15.42:0/1654170307 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6e819ffb0 con 0x7fe6e81035f0 2024-07-26T06:50:49.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.575+0000 7fe6e57fa640 1 -- 172.21.15.42:0/1654170307 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6d8003800 con 0x7fe6e81035f0 2024-07-26T06:50:49.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.575+0000 7fe6e57fa640 1 -- 172.21.15.42:0/1654170307 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8035070 con 0x7fe6e81035f0 2024-07-26T06:50:49.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.575+0000 7fe6ee338640 1 -- 172.21.15.42:0/1654170307 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6e81a04e0 con 0x7fe6e81035f0 2024-07-26T06:50:49.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.576+0000 7fe6ee338640 1 -- 172.21.15.42:0/1654170307 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6e8103a00 con 0x7fe6e81035f0 2024-07-26T06:50:49.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.578+0000 7fe6e57fa640 1 -- 172.21.15.42:0/1654170307 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fe6d8003cb0 con 0x7fe6e81035f0 2024-07-26T06:50:49.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.579+0000 7fe6e57fa640 1 --2- 172.21.15.42:0/1654170307 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe6d00b3c00 0x7fe6d00b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:49.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.579+0000 7fe6e77fe640 1 --2- 172.21.15.42:0/1654170307 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe6d00b3c00 0x7fe6d00b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:49.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.579+0000 7fe6e57fa640 1 -- 172.21.15.42:0/1654170307 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fe6d8104890 con 0x7fe6e81035f0 2024-07-26T06:50:49.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.579+0000 7fe6e77fe640 1 --2- 172.21.15.42:0/1654170307 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe6d00b3c00 0x7fe6d00b6110 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8108a70 tx=0x7fe6dc005d90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:49.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.582+0000 7fe6e57fa640 1 -- 172.21.15.42:0/1654170307 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe6e81a04e0 con 0x7fe6e81035f0 2024-07-26T06:50:49.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.802+0000 7fe6ee338640 1 -- 172.21.15.42:0/1654170307 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe6e819afa0 con 0x7fe6e81035f0 2024-07-26T06:50:49.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.803+0000 7fe6e57fa640 1 -- 172.21.15.42:0/1654170307 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11978 (secure 0 0 0) 0x7fe6d80cbec0 con 0x7fe6e81035f0 2024-07-26T06:50:49.804 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:50:49.804 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":29,"btime":"2024-07-26T06:50:27:493008+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:50:26.696932+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24620,24601,24593,24583,24578]},"id":1}]} 2024-07-26T06:50:49.805 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 29 2024-07-26T06:50:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.805+0000 7fe6ee338640 1 -- 172.21.15.42:0/1654170307 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe6d00b3c00 msgr2=0x7fe6d00b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.805+0000 7fe6ee338640 1 --2- 172.21.15.42:0/1654170307 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe6d00b3c00 0x7fe6d00b6110 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8108a70 tx=0x7fe6dc005d90 comp rx=0 tx=0).stop 2024-07-26T06:50:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.805+0000 7fe6ee338640 1 -- 172.21.15.42:0/1654170307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e81035f0 msgr2=0x7fe6e8199e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.805+0000 7fe6ee338640 1 --2- 172.21.15.42:0/1654170307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e81035f0 0x7fe6e8199e70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8006010 tx=0x7fe6d80042f0 comp rx=0 tx=0).stop 2024-07-26T06:50:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.805+0000 7fe6ee338640 1 -- 172.21.15.42:0/1654170307 shutdown_connections 2024-07-26T06:50:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.805+0000 7fe6ee338640 1 --2- 172.21.15.42:0/1654170307 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe6d00b3c00 0x7fe6d00b6110 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.805+0000 7fe6ee338640 1 --2- 172.21.15.42:0/1654170307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6e8109830 0x7fe6e819f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.805+0000 7fe6ee338640 1 --2- 172.21.15.42:0/1654170307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe6e8108ee0 0x7fe6e819a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:49.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.805+0000 7fe6ee338640 1 --2- 172.21.15.42:0/1654170307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6e81035f0 0x7fe6e8199e70 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:49.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.805+0000 7fe6ee338640 1 -- 172.21.15.42:0/1654170307 >> 172.21.15.42:0/1654170307 conn(0x7fe6e807b340 msgr2=0x7fe6e8106f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:49.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.805+0000 7fe6ee338640 1 -- 172.21.15.42:0/1654170307 shutdown_connections 2024-07-26T06:50:49.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:49.805+0000 7fe6ee338640 1 -- 172.21.15.42:0/1654170307 wait complete. 2024-07-26T06:50:50.458 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-07-26T06:47:06.780568+0000', 'modified': '2024-07-26T06:50:26.696932+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24620, 'mds_1': 24601, 'mds_2': 24593, 'mds_3': 24583, 'mds_4': 24578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14703': {'gid': 14703, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.42:6839/4044632348', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.42:6838', 'nonce': 4044632348}, {'type': 'v1', 'addr': '172.21.15.42:6839', 'nonce': 4044632348}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14709': {'gid': 14709, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.42:6841/1430155791', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.42:6840', 'nonce': 1430155791}, {'type': 'v1', 'addr': '172.21.15.42:6841', 'nonce': 1430155791}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24541': {'gid': 24541, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.164:6833/4144543280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6832', 'nonce': 4144543280}, {'type': 'v1', 'addr': '172.21.15.164:6833', 'nonce': 4144543280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24557': {'gid': 24557, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.163:6833/845444723', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6832', 'nonce': 845444723}, {'type': 'v1', 'addr': '172.21.15.163:6833', 'nonce': 845444723}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24571': {'gid': 24571, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.164:6835/2604534474', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6834', 'nonce': 2604534474}, {'type': 'v1', 'addr': '172.21.15.164:6835', 'nonce': 2604534474}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24578': {'gid': 24578, 'name': 'e', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.163:6835/375978170', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6834', 'nonce': 375978170}, {'type': 'v1', 'addr': '172.21.15.163:6835', 'nonce': 375978170}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24583': {'gid': 24583, 'name': 'c', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.164:6837/3642457979', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6836', 'nonce': 3642457979}, {'type': 'v1', 'addr': '172.21.15.164:6837', 'nonce': 3642457979}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24593': {'gid': 24593, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.163:6837/154041066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6836', 'nonce': 154041066}, {'type': 'v1', 'addr': '172.21.15.163:6837', 'nonce': 154041066}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24601': {'gid': 24601, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.164:6839/3593178314', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.164:6838', 'nonce': 3593178314}, {'type': 'v1', 'addr': '172.21.15.164:6839', 'nonce': 3593178314}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24620': {'gid': 24620, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.163:6839/563658453', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6838', 'nonce': 563658453}, {'type': 'v1', 'addr': '172.21.15.163:6839', 'nonce': 563658453}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24620, 'qdb_cluster': [24620, 24601, 24593, 24583, 24578]} 2024-07-26T06:50:50.458 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-07-26T06:50:50.458 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell mds.e status 2024-07-26T06:50:50.765 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:50.887 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:50 smithi042 ceph-mon[32775]: pgmap v414: 97 pgs: 97 active+clean; 660 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.5 KiB/s wr, 15 op/s 2024-07-26T06:50:50.887 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:50 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1654170307' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:50:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:50 smithi164 ceph-mon[36970]: pgmap v414: 97 pgs: 97 active+clean; 660 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.5 KiB/s wr, 15 op/s 2024-07-26T06:50:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:50 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1654170307' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:50:51.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:50 smithi163 ceph-mon[36873]: pgmap v414: 97 pgs: 97 active+clean; 660 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.5 KiB/s wr, 15 op/s 2024-07-26T06:50:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:50 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1654170307' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:50:52.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.412+0000 7fe8a2354640 1 -- 172.21.15.42:0/3841126608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe89c103590 msgr2=0x7fe89c1039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:52.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.412+0000 7fe8a2354640 1 --2- 172.21.15.42:0/3841126608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe89c103590 0x7fe89c1039a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009900 tx=0x7fe88c02a410 comp rx=0 tx=0).stop 2024-07-26T06:50:52.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.412+0000 7fe8a2354640 1 -- 172.21.15.42:0/3841126608 shutdown_connections 2024-07-26T06:50:52.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.412+0000 7fe8a2354640 1 --2- 172.21.15.42:0/3841126608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe89c109820 0x7fe89c110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.412+0000 7fe8a2354640 1 --2- 172.21.15.42:0/3841126608 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe89c108ed0 0x7fe89c1092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.412+0000 7fe8a2354640 1 --2- 172.21.15.42:0/3841126608 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe89c103590 0x7fe89c1039a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.412+0000 7fe8a2354640 1 -- 172.21.15.42:0/3841126608 >> 172.21.15.42:0/3841126608 conn(0x7fe89c07b280 msgr2=0x7fe89c07b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:52.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.412+0000 7fe8a2354640 1 -- 172.21.15.42:0/3841126608 shutdown_connections 2024-07-26T06:50:52.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.412+0000 7fe8a2354640 1 -- 172.21.15.42:0/3841126608 wait complete. 2024-07-26T06:50:52.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.413+0000 7fe8a2354640 1 Processor -- start 2024-07-26T06:50:52.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.413+0000 7fe8a2354640 1 -- start start 2024-07-26T06:50:52.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.414+0000 7fe8a2354640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe89c103590 0x7fe89c199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.414+0000 7fe8a2354640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe89c108ed0 0x7fe89c19a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.414+0000 7fe8a2354640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe89c109820 0x7fe89c19f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.414+0000 7fe8a2354640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c1134f0 con 0x7fe89c108ed0 2024-07-26T06:50:52.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.414+0000 7fe8a2354640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c113370 con 0x7fe89c103590 2024-07-26T06:50:52.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.414+0000 7fe8a2354640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c113670 con 0x7fe89c109820 2024-07-26T06:50:52.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.414+0000 7fe89b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe89c108ed0 0x7fe89c19a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.414+0000 7fe89b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe89c108ed0 0x7fe89c19a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54468/0 (socket says 172.21.15.42:54468) 2024-07-26T06:50:52.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.414+0000 7fe89b7fe640 1 -- 172.21.15.42:0/1398614405 learned_addr learned my addr 172.21.15.42:0/1398614405 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:52.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.414+0000 7fe8a08f1640 1 --2- 172.21.15.42:0/1398614405 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe89c109820 0x7fe89c19f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.414+0000 7fe89bfff640 1 --2- 172.21.15.42:0/1398614405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe89c103590 0x7fe89c199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.414+0000 7fe89b7fe640 1 -- 172.21.15.42:0/1398614405 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe89c109820 msgr2=0x7fe89c19f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:52.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.414+0000 7fe89b7fe640 1 --2- 172.21.15.42:0/1398614405 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe89c109820 0x7fe89c19f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.414+0000 7fe89b7fe640 1 -- 172.21.15.42:0/1398614405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe89c103590 msgr2=0x7fe89c199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:52.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.414+0000 7fe89b7fe640 1 --2- 172.21.15.42:0/1398614405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe89c103590 0x7fe89c199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.414+0000 7fe89b7fe640 1 -- 172.21.15.42:0/1398614405 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe89c19fef0 con 0x7fe89c108ed0 2024-07-26T06:50:52.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.415+0000 7fe89b7fe640 1 --2- 172.21.15.42:0/1398614405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe89c108ed0 0x7fe89c19a520 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe89000d740 tx=0x7fe89000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:52.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.415+0000 7fe8997fa640 1 -- 172.21.15.42:0/1398614405 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe890017070 con 0x7fe89c108ed0 2024-07-26T06:50:52.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.415+0000 7fe8a2354640 1 -- 172.21.15.42:0/1398614405 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe89c1a01e0 con 0x7fe89c108ed0 2024-07-26T06:50:52.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.415+0000 7fe8997fa640 1 -- 172.21.15.42:0/1398614405 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe89000de50 con 0x7fe89c108ed0 2024-07-26T06:50:52.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.415+0000 7fe8997fa640 1 -- 172.21.15.42:0/1398614405 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe89000f620 con 0x7fe89c108ed0 2024-07-26T06:50:52.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.415+0000 7fe8a2354640 1 -- 172.21.15.42:0/1398614405 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe89c1a6940 con 0x7fe89c108ed0 2024-07-26T06:50:52.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.418+0000 7fe8a88d8740 1 Processor -- start 2024-07-26T06:50:52.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.418+0000 7fe8997fa640 1 -- 172.21.15.42:0/1398614405 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fe89001a070 con 0x7fe89c108ed0 2024-07-26T06:50:52.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.418+0000 7fe8a88d8740 1 -- start start 2024-07-26T06:50:52.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.418+0000 7fe8a88d8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444024f60 0x5644443ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.418+0000 7fe8a88d8740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564443ffc0a0 0x5644443ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.418+0000 7fe8a88d8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564444004280 0x5644443cae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.418+0000 7fe8a88d8740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x564444340940 con 0x564443ffc0a0 2024-07-26T06:50:52.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.418+0000 7fe8a88d8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56444432fe40 con 0x564444024f60 2024-07-26T06:50:52.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.418+0000 7fe8a88d8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x564444333e80 con 0x564444004280 2024-07-26T06:50:52.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.418+0000 7fe89bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564443ffc0a0 0x5644443ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.418+0000 7fe89bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564443ffc0a0 0x5644443ca8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54476/0 (socket says 172.21.15.42:54476) 2024-07-26T06:50:52.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.418+0000 7fe89bfff640 1 -- 172.21.15.42:0/2703877894 learned_addr learned my addr 172.21.15.42:0/2703877894 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:52.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.419+0000 7fe89b7fe640 1 --2- 172.21.15.42:0/2703877894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564444004280 0x5644443cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.419+0000 7fe8a08f1640 1 --2- 172.21.15.42:0/2703877894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444024f60 0x5644443ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.419+0000 7fe89bfff640 1 -- 172.21.15.42:0/2703877894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564444004280 msgr2=0x5644443cae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:52.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.419+0000 7fe89bfff640 1 --2- 172.21.15.42:0/2703877894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564444004280 0x5644443cae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.419+0000 7fe89bfff640 1 -- 172.21.15.42:0/2703877894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444024f60 msgr2=0x5644443ca3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:52.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.419+0000 7fe89bfff640 1 --2- 172.21.15.42:0/2703877894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444024f60 0x5644443ca3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.419+0000 7fe89bfff640 1 -- 172.21.15.42:0/2703877894 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644442d6a60 con 0x564443ffc0a0 2024-07-26T06:50:52.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.419+0000 7fe8a08f1640 1 --2- 172.21.15.42:0/2703877894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444024f60 0x5644443ca3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:52.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.419+0000 7fe8997fa640 1 --2- 172.21.15.42:0/1398614405 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8840b3bd0 0x7fe8840b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.419+0000 7fe8a08f1640 1 --2- 172.21.15.42:0/1398614405 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8840b3bd0 0x7fe8840b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.419+0000 7fe89bfff640 1 --2- 172.21.15.42:0/2703877894 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564443ffc0a0 0x5644443ca8e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe88c02a930 tx=0x7fe88c032c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d5ac836f0cdd51 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:52.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.419+0000 7fe8997fa640 1 -- 172.21.15.42:0/1398614405 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fe8900e3630 con 0x7fe89c108ed0 2024-07-26T06:50:52.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.420+0000 7fe8827fc640 1 -- 172.21.15.42:0/2703877894 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c035050 con 0x564443ffc0a0 2024-07-26T06:50:52.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.420+0000 7fe8827fc640 1 -- 172.21.15.42:0/2703877894 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe88c032e40 con 0x564443ffc0a0 2024-07-26T06:50:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.420+0000 7fe8827fc640 1 -- 172.21.15.42:0/2703877894 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c03c750 con 0x564443ffc0a0 2024-07-26T06:50:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.420+0000 7fe8a08f1640 1 --2- 172.21.15.42:0/1398614405 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8840b3bd0 0x7fe8840b60e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe894005fd0 tx=0x7fe894009600 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.421+0000 7fe8a88d8740 1 -- 172.21.15.42:0/2703877894 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564443ffc0a0 msgr2=0x5644443ca8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.421+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/2703877894 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564443ffc0a0 0x5644443ca8e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe88c02a930 tx=0x7fe88c032c50 comp rx=0 tx=0).stop 2024-07-26T06:50:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.421+0000 7fe8a88d8740 1 -- 172.21.15.42:0/2703877894 shutdown_connections 2024-07-26T06:50:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.421+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/2703877894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564444004280 0x5644443cae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.421+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/2703877894 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564443ffc0a0 0x5644443ca8e0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.421+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/2703877894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444024f60 0x5644443ca3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.421+0000 7fe8a88d8740 1 -- 172.21.15.42:0/2703877894 >> 172.21.15.42:0/2703877894 conn(0x564444019350 msgr2=0x5644443c8290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.421+0000 7fe8a88d8740 1 -- 172.21.15.42:0/2703877894 shutdown_connections 2024-07-26T06:50:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.421+0000 7fe8a88d8740 1 -- 172.21.15.42:0/2703877894 wait complete. 2024-07-26T06:50:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.421+0000 7fe8a88d8740 1 -- start start 2024-07-26T06:50:52.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.421+0000 7fe8a88d8740 1 Processor -- start 2024-07-26T06:50:52.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe8a88d8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564443ffc0a0 0x5644443d5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe8a88d8740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564444004280 0x5644443d6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe8a88d8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564444024f60 0x5644443e1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe8a88d8740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5644441935f0 con 0x564444004280 2024-07-26T06:50:52.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe8a88d8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5644442d28b0 con 0x564443ffc0a0 2024-07-26T06:50:52.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe8a88d8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5644443d8b20 con 0x564444024f60 2024-07-26T06:50:52.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe8a88d8740 10 client.0 _ll_register_callbacks cb 0x5644442dc7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:52.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe89b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564444004280 0x5644443d6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe89b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564444004280 0x5644443d6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54484/0 (socket says 172.21.15.42:54484) 2024-07-26T06:50:52.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe89b7fe640 1 -- 172.21.15.42:0/1585414101 learned_addr learned my addr 172.21.15.42:0/1585414101 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:52.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe8a08f1640 1 --2- 172.21.15.42:0/1585414101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564444024f60 0x5644443e1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe89bfff640 1 --2- 172.21.15.42:0/1585414101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564443ffc0a0 0x5644443d5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe89b7fe640 1 -- 172.21.15.42:0/1585414101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564444024f60 msgr2=0x5644443e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:52.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe89b7fe640 1 --2- 172.21.15.42:0/1585414101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564444024f60 0x5644443e1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe89b7fe640 1 -- 172.21.15.42:0/1585414101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564443ffc0a0 msgr2=0x5644443d5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:52.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe89b7fe640 1 --2- 172.21.15.42:0/1585414101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564443ffc0a0 0x5644443d5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe89b7fe640 1 -- 172.21.15.42:0/1585414101 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564444090540 con 0x564444004280 2024-07-26T06:50:52.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.422+0000 7fe89bfff640 1 --2- 172.21.15.42:0/1585414101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564443ffc0a0 0x5644443d5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:52.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.423+0000 7fe89b7fe640 1 --2- 172.21.15.42:0/1585414101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564444004280 0x5644443d6410 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe890000c00 tx=0x7fe8900a6420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8c6d3b66807971 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:52.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.423+0000 7fe87b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:52.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.423+0000 7fe87b7fe640 1 -- 172.21.15.42:0/1585414101 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe890021070 con 0x564444004280 2024-07-26T06:50:52.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.423+0000 7fe87b7fe640 1 -- 172.21.15.42:0/1585414101 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fe8900e5bb0 con 0x564444004280 2024-07-26T06:50:52.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.423+0000 7fe8a88d8740 1 -- 172.21.15.42:0/1585414101 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564444342890 con 0x564444004280 2024-07-26T06:50:52.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.423+0000 7fe87b7fe640 1 -- 172.21.15.42:0/1585414101 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe8900a9cd0 con 0x564444004280 2024-07-26T06:50:52.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.423+0000 7fe8a88d8740 10 client.15384 fetch_fsmap learned FSMap version 29 2024-07-26T06:50:52.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.423+0000 7fe8a88d8740 1 -- 172.21.15.42:0/1585414101 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5644443e4ad0 con 0x564444004280 2024-07-26T06:50:52.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.423+0000 7fe87b7fe640 1 -- 172.21.15.42:0/1585414101 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564444342890 con 0x564444004280 2024-07-26T06:50:52.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.423+0000 7fe87b7fe640 1 -- 172.21.15.42:0/1585414101 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe8900e3ae0 con 0x564444004280 2024-07-26T06:50:52.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.423+0000 7fe8a88d8740 10 client.15384 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:50:52.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.423+0000 7fe8a88d8740 10 client.15384 resolve_mds: resolved name 'e' to daemon mds.e 2024-07-26T06:50:52.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.424+0000 7fe8a88d8740 20 client.15384 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:52.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.424+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/1585414101 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x564443fe38e0 0x5644443e85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.424+0000 7fe8a88d8740 4 client.15384 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:50:52.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.424+0000 7fe8a88d8740 1 -- 172.21.15.42:0/1585414101 --> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56444433e330 con 0x564443fe38e0 2024-07-26T06:50:52.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.424+0000 7fe89bfff640 1 --2- 172.21.15.42:0/1585414101 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x564443fe38e0 0x5644443e85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.424+0000 7fe89bfff640 1 --2- 172.21.15.42:0/1585414101 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x564443fe38e0 0x5644443e85d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e9cb655b29ac9f41 server_cookie=afaed2353e1ca3b7 in_seq=0 out_seq=0 2024-07-26T06:50:52.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.424+0000 7fe87b7fe640 10 client.15384 ms_handle_connect on v2:172.21.15.163:6834/375978170 2024-07-26T06:50:52.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.425+0000 7fe87b7fe640 1 -- 172.21.15.42:0/1585414101 <== mds.4 v2:172.21.15.163:6834/375978170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56444433e330 con 0x564443fe38e0 2024-07-26T06:50:52.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.425+0000 7fe87b7fe640 10 client.15384 handle_command_reply: tid=0 2024-07-26T06:50:52.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.425+0000 7fe8a88d8740 1 client.15384 shutdown 2024-07-26T06:50:52.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.426+0000 7fe8a88d8740 1 -- 172.21.15.42:0/1585414101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564444004280 msgr2=0x5644443d6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:52.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.426+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/1585414101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564444004280 0x5644443d6410 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe890000c00 tx=0x7fe8900a6420 comp rx=0 tx=0).stop 2024-07-26T06:50:52.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.426+0000 7fe8a88d8740 1 -- 172.21.15.42:0/1585414101 shutdown_connections 2024-07-26T06:50:52.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.426+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/1585414101 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x564443fe38e0 0x5644443e85d0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.426+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/1585414101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564444024f60 0x5644443e1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.426+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/1585414101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564444004280 0x5644443d6410 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.426+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/1585414101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564443ffc0a0 0x5644443d5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.426+0000 7fe87b7fe640 0 client.15384 ms_handle_reset on v2:172.21.15.163:6834/375978170 2024-07-26T06:50:52.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.426+0000 7fe8a88d8740 1 -- 172.21.15.42:0/1585414101 >> 172.21.15.42:0/1585414101 conn(0x564444019350 msgr2=0x5644443cd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:52.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.426+0000 7fe8a88d8740 1 -- 172.21.15.42:0/1585414101 shutdown_connections 2024-07-26T06:50:52.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.426+0000 7fe8a88d8740 1 -- 172.21.15.42:0/1585414101 wait complete. 2024-07-26T06:50:52.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.426+0000 7fe8a88d8740 20 client.15384 trim_cache size 0 max 16384 2024-07-26T06:50:52.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.444+0000 7fe8a88d8740 1 Processor -- start 2024-07-26T06:50:52.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.444+0000 7fe8a88d8740 1 -- start start 2024-07-26T06:50:52.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.444+0000 7fe8a88d8740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564443ffc0a0 0x5644443e1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.444+0000 7fe8a88d8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564444004280 0x5644443e2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.444+0000 7fe8a88d8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444024f60 0x5644443d4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.444+0000 7fe8a88d8740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5644443e6910 con 0x564443ffc0a0 2024-07-26T06:50:52.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.444+0000 7fe8a88d8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5644443e6fe0 con 0x564444024f60 2024-07-26T06:50:52.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.444+0000 7fe8a88d8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5644443e76b0 con 0x564444004280 2024-07-26T06:50:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.445+0000 7fe89bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564443ffc0a0 0x5644443e1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.445+0000 7fe89bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564443ffc0a0 0x5644443e1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54498/0 (socket says 172.21.15.42:54498) 2024-07-26T06:50:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.445+0000 7fe89bfff640 1 -- 172.21.15.42:0/616317947 learned_addr learned my addr 172.21.15.42:0/616317947 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.445+0000 7fe89b7fe640 1 --2- 172.21.15.42:0/616317947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564444004280 0x5644443e2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.445+0000 7fe8a08f1640 1 --2- 172.21.15.42:0/616317947 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444024f60 0x5644443d4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.445+0000 7fe89bfff640 1 -- 172.21.15.42:0/616317947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564444004280 msgr2=0x5644443e2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.445+0000 7fe89bfff640 1 --2- 172.21.15.42:0/616317947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564444004280 0x5644443e2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.445+0000 7fe89bfff640 1 -- 172.21.15.42:0/616317947 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444024f60 msgr2=0x5644443d4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.445+0000 7fe89bfff640 1 --2- 172.21.15.42:0/616317947 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444024f60 0x5644443d4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.445+0000 7fe89bfff640 1 -- 172.21.15.42:0/616317947 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644443d6f90 con 0x564443ffc0a0 2024-07-26T06:50:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.445+0000 7fe89b7fe640 1 --2- 172.21.15.42:0/616317947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564444004280 0x5644443e2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.445+0000 7fe8a08f1640 1 --2- 172.21.15.42:0/616317947 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444024f60 0x5644443d4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:50:52.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.445+0000 7fe89bfff640 1 --2- 172.21.15.42:0/616317947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564443ffc0a0 0x5644443e1e10 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009900 tx=0x7fe88c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcaf63b04ab8f141 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.445+0000 7fe881ffb640 1 -- 172.21.15.42:0/616317947 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c035040 con 0x564443ffc0a0 2024-07-26T06:50:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.445+0000 7fe881ffb640 1 -- 172.21.15.42:0/616317947 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe88c0316a0 con 0x564443ffc0a0 2024-07-26T06:50:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.445+0000 7fe881ffb640 1 -- 172.21.15.42:0/616317947 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c03ca90 con 0x564443ffc0a0 2024-07-26T06:50:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.446+0000 7fe8a88d8740 1 -- 172.21.15.42:0/616317947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564443ffc0a0 msgr2=0x5644443e1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.446+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/616317947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564443ffc0a0 0x5644443e1e10 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe88c009900 tx=0x7fe88c005fd0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.446+0000 7fe8a88d8740 1 -- 172.21.15.42:0/616317947 shutdown_connections 2024-07-26T06:50:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.446+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/616317947 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444024f60 0x5644443d4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.446+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/616317947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564444004280 0x5644443e2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.446+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/616317947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564443ffc0a0 0x5644443e1e10 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.446+0000 7fe8a88d8740 1 -- 172.21.15.42:0/616317947 >> 172.21.15.42:0/616317947 conn(0x564444019350 msgr2=0x5644443d0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.446+0000 7fe8a88d8740 1 -- 172.21.15.42:0/616317947 shutdown_connections 2024-07-26T06:50:52.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.446+0000 7fe8a88d8740 1 -- 172.21.15.42:0/616317947 wait complete. 2024-07-26T06:50:52.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.447+0000 7fe8a88d8740 1 -- start start 2024-07-26T06:50:52.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.447+0000 7fe8a88d8740 1 Processor -- start 2024-07-26T06:50:52.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.447+0000 7fe8a88d8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564443ffc0a0 0x5644443e99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.447+0000 7fe8a88d8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444004280 0x5644443e9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.447+0000 7fe8a88d8740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564444024f60 0x5644443eef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.447+0000 7fe8a88d8740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5644443e8450 con 0x564444024f60 2024-07-26T06:50:52.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.447+0000 7fe8a88d8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5644443e7d80 con 0x564444004280 2024-07-26T06:50:52.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.447+0000 7fe8a88d8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5644443e8b20 con 0x564443ffc0a0 2024-07-26T06:50:52.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.447+0000 7fe8a88d8740 10 client.0 _ll_register_callbacks cb 0x7fe890010b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:52.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.447+0000 7fe8a08f1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564444024f60 0x5644443eef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.447+0000 7fe8a08f1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564444024f60 0x5644443eef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54508/0 (socket says 172.21.15.42:54508) 2024-07-26T06:50:52.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.447+0000 7fe8a08f1640 1 -- 172.21.15.42:0/299552072 learned_addr learned my addr 172.21.15.42:0/299552072 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:52.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.447+0000 7fe89bfff640 1 --2- 172.21.15.42:0/299552072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564443ffc0a0 0x5644443e99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.447+0000 7fe89b7fe640 1 --2- 172.21.15.42:0/299552072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444004280 0x5644443e9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.448+0000 7fe8a08f1640 1 -- 172.21.15.42:0/299552072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564443ffc0a0 msgr2=0x5644443e99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:52.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.448+0000 7fe8a08f1640 1 --2- 172.21.15.42:0/299552072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564443ffc0a0 0x5644443e99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.448+0000 7fe8a08f1640 1 -- 172.21.15.42:0/299552072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444004280 msgr2=0x5644443e9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:52.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.448+0000 7fe8a08f1640 1 --2- 172.21.15.42:0/299552072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444004280 0x5644443e9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.448+0000 7fe8a08f1640 1 -- 172.21.15.42:0/299552072 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644443ef5b0 con 0x564444024f60 2024-07-26T06:50:52.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.448+0000 7fe89b7fe640 1 --2- 172.21.15.42:0/299552072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444004280 0x5644443e9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:50:52.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.448+0000 7fe8a08f1640 1 --2- 172.21.15.42:0/299552072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564444024f60 0x5644443eef40 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe8940107b0 tx=0x7fe894014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad46e7635f2652e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:52.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.448+0000 7fe87b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:52.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.448+0000 7fe87b7fe640 1 -- 172.21.15.42:0/299552072 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe894004d20 con 0x564444024f60 2024-07-26T06:50:52.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.448+0000 7fe87b7fe640 1 -- 172.21.15.42:0/299552072 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fe89401c940 con 0x564444024f60 2024-07-26T06:50:52.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.448+0000 7fe8a88d8740 1 -- 172.21.15.42:0/299552072 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe88c0316a0 con 0x564444024f60 2024-07-26T06:50:52.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.448+0000 7fe87b7fe640 1 -- 172.21.15.42:0/299552072 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe89401cb40 con 0x564444024f60 2024-07-26T06:50:52.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.448+0000 7fe8a88d8740 10 client.15390 fetch_fsmap learned FSMap version 29 2024-07-26T06:50:52.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.448+0000 7fe8a88d8740 1 -- 172.21.15.42:0/299552072 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5644443f0970 con 0x564444024f60 2024-07-26T06:50:52.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.448+0000 7fe87b7fe640 1 -- 172.21.15.42:0/299552072 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe88c0316a0 con 0x564444024f60 2024-07-26T06:50:52.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.449+0000 7fe87b7fe640 1 -- 172.21.15.42:0/299552072 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe89400f040 con 0x564444024f60 2024-07-26T06:50:52.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.449+0000 7fe8a88d8740 10 client.15390 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:50:52.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.449+0000 7fe8a88d8740 10 client.15390 resolve_mds: resolved name 'e' to daemon mds.e 2024-07-26T06:50:52.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.449+0000 7fe8a88d8740 20 client.15390 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:52.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.449+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/299552072 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x564443fb8c90 0x5644444544c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:52.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.449+0000 7fe8a88d8740 4 client.15390 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:50:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.449+0000 7fe8a88d8740 1 -- 172.21.15.42:0/299552072 --> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] -- command(tid 0: {"prefix": "status"}) -- 0x56444433ba70 con 0x564443fb8c90 2024-07-26T06:50:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.449+0000 7fe89bfff640 1 --2- 172.21.15.42:0/299552072 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x564443fb8c90 0x5644444544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.449+0000 7fe87b7fe640 10 client.15390 ms_handle_connect on v2:172.21.15.163:6834/375978170 2024-07-26T06:50:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.449+0000 7fe89bfff640 1 --2- 172.21.15.42:0/299552072 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x564443fb8c90 0x5644444544c0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4c4296529e2d002b server_cookie=8becb5259320e869 in_seq=0 out_seq=0 2024-07-26T06:50:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.450+0000 7fe87b7fe640 1 -- 172.21.15.42:0/299552072 <== mds.4 v2:172.21.15.163:6834/375978170 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56444433ba70 con 0x564443fb8c90 2024-07-26T06:50:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.450+0000 7fe87b7fe640 10 client.15390 handle_command_reply: tid=0 2024-07-26T06:50:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.450+0000 7fe8a88d8740 1 client.15390 shutdown 2024-07-26T06:50:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.450+0000 7fe8a88d8740 1 -- 172.21.15.42:0/299552072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564444024f60 msgr2=0x5644443eef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.450+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/299552072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564444024f60 0x5644443eef40 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe8940107b0 tx=0x7fe894014040 comp rx=0 tx=0).stop 2024-07-26T06:50:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.450+0000 7fe8a88d8740 1 -- 172.21.15.42:0/299552072 shutdown_connections 2024-07-26T06:50:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.450+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/299552072 >> [v2:172.21.15.163:6834/375978170,v1:172.21.15.163:6835/375978170] conn(0x564443fb8c90 0x5644444544c0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.450+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/299552072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564444024f60 0x5644443eef40 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.450+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/299552072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564444004280 0x5644443e9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.450+0000 7fe8a88d8740 1 --2- 172.21.15.42:0/299552072 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564443ffc0a0 0x5644443e99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.450+0000 7fe8a88d8740 1 -- 172.21.15.42:0/299552072 >> 172.21.15.42:0/299552072 conn(0x564444019350 msgr2=0x5644443c7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:52.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.450+0000 7fe87b7fe640 0 client.15390 ms_handle_reset on v2:172.21.15.163:6834/375978170 2024-07-26T06:50:52.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.450+0000 7fe8a88d8740 1 -- 172.21.15.42:0/299552072 shutdown_connections 2024-07-26T06:50:52.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.450+0000 7fe8a88d8740 1 -- 172.21.15.42:0/299552072 wait complete. 2024-07-26T06:50:52.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.450+0000 7fe8a88d8740 20 client.15390 trim_cache size 0 max 16384 2024-07-26T06:50:52.457 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:50:52.457 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_fsid": "8b3e2750-4b19-11ef-bcb6-c7b262605968", 2024-07-26T06:50:52.458 INFO:teuthology.orchestra.run.smithi042.stdout: "whoami": 4, 2024-07-26T06:50:52.458 INFO:teuthology.orchestra.run.smithi042.stdout: "id": 24578, 2024-07-26T06:50:52.458 INFO:teuthology.orchestra.run.smithi042.stdout: "want_state": "up:active", 2024-07-26T06:50:52.458 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "up:active", 2024-07-26T06:50:52.458 INFO:teuthology.orchestra.run.smithi042.stdout: "fs_name": "cephfs", 2024-07-26T06:50:52.458 INFO:teuthology.orchestra.run.smithi042.stdout: "rank_uptime": 210.35082788400001, 2024-07-26T06:50:52.458 INFO:teuthology.orchestra.run.smithi042.stdout: "mdsmap_epoch": 29, 2024-07-26T06:50:52.458 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch": 80, 2024-07-26T06:50:52.458 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch_barrier": 80, 2024-07-26T06:50:52.458 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 261.931110827 2024-07-26T06:50:52.458 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:50:52.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.451+0000 7fe8a2354640 1 -- 172.21.15.42:0/1398614405 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8840b3bd0 msgr2=0x7fe8840b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:52.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.451+0000 7fe8a2354640 1 --2- 172.21.15.42:0/1398614405 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8840b3bd0 0x7fe8840b60e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe894005fd0 tx=0x7fe894009600 comp rx=0 tx=0).stop 2024-07-26T06:50:52.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.451+0000 7fe8a2354640 1 -- 172.21.15.42:0/1398614405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe89c108ed0 msgr2=0x7fe89c19a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:52.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.451+0000 7fe8a2354640 1 --2- 172.21.15.42:0/1398614405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe89c108ed0 0x7fe89c19a520 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe89000d740 tx=0x7fe89000dc00 comp rx=0 tx=0).stop 2024-07-26T06:50:52.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.451+0000 7fe8a2354640 1 -- 172.21.15.42:0/1398614405 shutdown_connections 2024-07-26T06:50:52.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.451+0000 7fe8a2354640 1 --2- 172.21.15.42:0/1398614405 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8840b3bd0 0x7fe8840b60e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.451+0000 7fe8a2354640 1 --2- 172.21.15.42:0/1398614405 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe89c109820 0x7fe89c19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.451+0000 7fe8a2354640 1 --2- 172.21.15.42:0/1398614405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe89c108ed0 0x7fe89c19a520 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.451+0000 7fe8a2354640 1 --2- 172.21.15.42:0/1398614405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe89c103590 0x7fe89c199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:52.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.451+0000 7fe8a2354640 1 -- 172.21.15.42:0/1398614405 >> 172.21.15.42:0/1398614405 conn(0x7fe89c07b280 msgr2=0x7fe89c107080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:52.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.451+0000 7fe8a2354640 1 -- 172.21.15.42:0/1398614405 shutdown_connections 2024-07-26T06:50:52.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:52.451+0000 7fe8a2354640 1 -- 172.21.15.42:0/1398614405 wait complete. 2024-07-26T06:50:52.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:52 smithi042 ceph-mon[32775]: pgmap v415: 97 pgs: 97 active+clean; 660 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.6 KiB/s wr, 13 op/s 2024-07-26T06:50:52.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:52 smithi164 ceph-mon[36970]: pgmap v415: 97 pgs: 97 active+clean; 660 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.6 KiB/s wr, 13 op/s 2024-07-26T06:50:53.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:52 smithi163 ceph-mon[36873]: pgmap v415: 97 pgs: 97 active+clean; 660 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.6 KiB/s wr, 13 op/s 2024-07-26T06:50:53.144 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell mds.c status 2024-07-26T06:50:53.412 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.932+0000 7fce4ae2c640 1 -- 172.21.15.42:0/3856679704 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce44103a20 msgr2=0x7fce4410a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.932+0000 7fce4ae2c640 1 --2- 172.21.15.42:0/3856679704 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce44103a20 0x7fce4410a3c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009ca0 tx=0x7fce3c02a8c0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.932+0000 7fce4ae2c640 1 -- 172.21.15.42:0/3856679704 shutdown_connections 2024-07-26T06:50:53.933 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.932+0000 7fce4ae2c640 1 --2- 172.21.15.42:0/3856679704 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce44103a20 0x7fce4410a3c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.934 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.932+0000 7fce4ae2c640 1 --2- 172.21.15.42:0/3856679704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce44103080 0x7fce44103490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.934 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.932+0000 7fce4ae2c640 1 --2- 172.21.15.42:0/3856679704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce44102000 0x7fce44102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.934 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.932+0000 7fce4ae2c640 1 -- 172.21.15.42:0/3856679704 >> 172.21.15.42:0/3856679704 conn(0x7fce440fd960 msgr2=0x7fce440ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:53.934 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.932+0000 7fce4ae2c640 1 -- 172.21.15.42:0/3856679704 shutdown_connections 2024-07-26T06:50:53.934 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.932+0000 7fce4ae2c640 1 -- 172.21.15.42:0/3856679704 wait complete. 2024-07-26T06:50:53.934 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.933+0000 7fce4ae2c640 1 Processor -- start 2024-07-26T06:50:53.934 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.933+0000 7fce4ae2c640 1 -- start start 2024-07-26T06:50:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.933+0000 7fce4ae2c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce44102000 0x7fce44199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.933+0000 7fce4ae2c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce44103080 0x7fce4419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.933+0000 7fce4ae2c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce44103a20 0x7fce4419acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.933+0000 7fce4ae2c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fce4410dd70 con 0x7fce44103080 2024-07-26T06:50:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.933+0000 7fce4ae2c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fce4410dbf0 con 0x7fce44102000 2024-07-26T06:50:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.933+0000 7fce4ae2c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fce4410def0 con 0x7fce44103a20 2024-07-26T06:50:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.933+0000 7fce3bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce44103080 0x7fce4419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.933+0000 7fce3bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce44103080 0x7fce4419a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54530/0 (socket says 172.21.15.42:54530) 2024-07-26T06:50:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.933+0000 7fce3bfff640 1 -- 172.21.15.42:0/165677099 learned_addr learned my addr 172.21.15.42:0/165677099 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.934+0000 7fce48bc8640 1 --2- 172.21.15.42:0/165677099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce44102000 0x7fce44199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.934+0000 7fce493c9640 1 --2- 172.21.15.42:0/165677099 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce44103a20 0x7fce4419acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.934+0000 7fce3bfff640 1 -- 172.21.15.42:0/165677099 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce44103a20 msgr2=0x7fce4419acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.934+0000 7fce3bfff640 1 --2- 172.21.15.42:0/165677099 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce44103a20 0x7fce4419acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.934+0000 7fce3bfff640 1 -- 172.21.15.42:0/165677099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce44102000 msgr2=0x7fce44199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.934+0000 7fce3bfff640 1 --2- 172.21.15.42:0/165677099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce44102000 0x7fce44199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.934+0000 7fce3bfff640 1 -- 172.21.15.42:0/165677099 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4419b4e0 con 0x7fce44103080 2024-07-26T06:50:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.934+0000 7fce493c9640 1 --2- 172.21.15.42:0/165677099 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce44103a20 0x7fce4419acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:53.938 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.934+0000 7fce48bc8640 1 --2- 172.21.15.42:0/165677099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce44102000 0x7fce44199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:53.938 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.934+0000 7fce3bfff640 1 --2- 172.21.15.42:0/165677099 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce44103080 0x7fce4419a110 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fce3400b4a0 tx=0x7fce3400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:53.938 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.934+0000 7fce39ffb640 1 -- 172.21.15.42:0/165677099 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce34014070 con 0x7fce44103080 2024-07-26T06:50:53.938 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.934+0000 7fce39ffb640 1 -- 172.21.15.42:0/165677099 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce34010ac0 con 0x7fce44103080 2024-07-26T06:50:53.938 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.934+0000 7fce4ae2c640 1 -- 172.21.15.42:0/165677099 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce4419b7d0 con 0x7fce44103080 2024-07-26T06:50:53.938 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.934+0000 7fce39ffb640 1 -- 172.21.15.42:0/165677099 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce34017c30 con 0x7fce44103080 2024-07-26T06:50:53.938 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.934+0000 7fce4ae2c640 1 -- 172.21.15.42:0/165677099 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce441a1ec0 con 0x7fce44103080 2024-07-26T06:50:53.945 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.937+0000 7fce51e29740 1 Processor -- start 2024-07-26T06:50:53.945 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.937+0000 7fce51e29740 1 -- start start 2024-07-26T06:50:53.945 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.937+0000 7fce39ffb640 1 -- 172.21.15.42:0/165677099 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fce34017070 con 0x7fce44103080 2024-07-26T06:50:53.945 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.937+0000 7fce51e29740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154c1e1b0 0x55f154fc3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.945 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.937+0000 7fce51e29740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154bf52f0 0x55f154fc37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.937+0000 7fce51e29740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fc4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.937+0000 7fce51e29740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55f154f363b0 con 0x55f154bfd4d0 2024-07-26T06:50:53.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.937+0000 7fce51e29740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f154f258b0 con 0x55f154bf52f0 2024-07-26T06:50:53.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.937+0000 7fce51e29740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55f154f298f0 con 0x55f154c1e1b0 2024-07-26T06:50:53.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.937+0000 7fce3bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fc4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:53.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.937+0000 7fce3bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fc4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54534/0 (socket says 172.21.15.42:54534) 2024-07-26T06:50:53.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.937+0000 7fce3bfff640 1 -- 172.21.15.42:0/21860570 learned_addr learned my addr 172.21.15.42:0/21860570 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:53.947 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.938+0000 7fce48bc8640 1 --2- 172.21.15.42:0/21860570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154bf52f0 0x55f154fc37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:53.947 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.938+0000 7fce493c9640 1 --2- 172.21.15.42:0/21860570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154c1e1b0 0x55f154fc3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:53.947 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.938+0000 7fce3bfff640 1 -- 172.21.15.42:0/21860570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154c1e1b0 msgr2=0x55f154fc3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:53.947 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.938+0000 7fce3bfff640 1 --2- 172.21.15.42:0/21860570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154c1e1b0 0x55f154fc3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.947 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.938+0000 7fce3bfff640 1 -- 172.21.15.42:0/21860570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154bf52f0 msgr2=0x55f154fc37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:53.947 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.938+0000 7fce3bfff640 1 --2- 172.21.15.42:0/21860570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154bf52f0 0x55f154fc37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.947 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.938+0000 7fce3bfff640 1 -- 172.21.15.42:0/21860570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f154ecc9a0 con 0x55f154bfd4d0 2024-07-26T06:50:53.947 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.938+0000 7fce39ffb640 1 --2- 172.21.15.42:0/165677099 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fce2c0b3bd0 0x7fce2c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.938+0000 7fce3bfff640 1 --2- 172.21.15.42:0/21860570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fc4340 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fce340eb580 tx=0x7fce340eb670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8732ee192491f473 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.938+0000 7fce493c9640 1 --2- 172.21.15.42:0/165677099 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fce2c0b3bd0 0x7fce2c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.938+0000 7fce39ffb640 1 -- 172.21.15.42:0/165677099 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fce340e33f0 con 0x7fce44103080 2024-07-26T06:50:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.938+0000 7fce2affd640 1 -- 172.21.15.42:0/21860570 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce340a61e0 con 0x55f154bfd4d0 2024-07-26T06:50:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.938+0000 7fce2affd640 1 -- 172.21.15.42:0/21860570 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce34017360 con 0x55f154bfd4d0 2024-07-26T06:50:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.938+0000 7fce2affd640 1 -- 172.21.15.42:0/21860570 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce340a9a90 con 0x55f154bfd4d0 2024-07-26T06:50:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.939+0000 7fce51e29740 1 -- 172.21.15.42:0/21860570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 msgr2=0x55f154fc4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:53.948 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.939+0000 7fce493c9640 1 --2- 172.21.15.42:0/165677099 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fce2c0b3bd0 0x7fce2c0b60e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fce3c02f540 tx=0x7fce3c031af0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:53.949 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.939+0000 7fce51e29740 1 --2- 172.21.15.42:0/21860570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fc4340 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fce340eb580 tx=0x7fce340eb670 comp rx=0 tx=0).stop 2024-07-26T06:50:53.949 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.939+0000 7fce51e29740 1 -- 172.21.15.42:0/21860570 shutdown_connections 2024-07-26T06:50:53.949 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.939+0000 7fce51e29740 1 --2- 172.21.15.42:0/21860570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fc4340 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.949 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.939+0000 7fce51e29740 1 --2- 172.21.15.42:0/21860570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154bf52f0 0x55f154fc37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.949 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.939+0000 7fce51e29740 1 --2- 172.21.15.42:0/21860570 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154c1e1b0 0x55f154fc3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.949 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.939+0000 7fce51e29740 1 -- 172.21.15.42:0/21860570 >> 172.21.15.42:0/21860570 conn(0x55f154c12300 msgr2=0x55f154fbec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:53.949 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.939+0000 7fce51e29740 1 -- 172.21.15.42:0/21860570 shutdown_connections 2024-07-26T06:50:53.949 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.939+0000 7fce51e29740 1 -- 172.21.15.42:0/21860570 wait complete. 2024-07-26T06:50:53.949 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.940+0000 7fce51e29740 1 -- start start 2024-07-26T06:50:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.940+0000 7fce51e29740 1 Processor -- start 2024-07-26T06:50:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.940+0000 7fce51e29740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154bf52f0 0x55f154fccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.940+0000 7fce51e29740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fcd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.940+0000 7fce51e29740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154c1e1b0 0x55f154fcdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.940+0000 7fce51e29740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55f154e5fab0 con 0x55f154bfd4d0 2024-07-26T06:50:53.950 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.940+0000 7fce51e29740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f154eadcf0 con 0x55f154bf52f0 2024-07-26T06:50:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.940+0000 7fce51e29740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55f154fc4e60 con 0x55f154c1e1b0 2024-07-26T06:50:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.940+0000 7fce51e29740 10 client.0 _ll_register_callbacks cb 0x55f154f1f9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.940+0000 7fce3bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fcd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.940+0000 7fce3bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fcd1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54546/0 (socket says 172.21.15.42:54546) 2024-07-26T06:50:53.951 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.940+0000 7fce3bfff640 1 -- 172.21.15.42:0/3251934280 learned_addr learned my addr 172.21.15.42:0/3251934280 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:53.952 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.940+0000 7fce493c9640 1 --2- 172.21.15.42:0/3251934280 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154c1e1b0 0x55f154fcdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:53.952 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.941+0000 7fce48bc8640 1 --2- 172.21.15.42:0/3251934280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154bf52f0 0x55f154fccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:53.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.941+0000 7fce3bfff640 1 -- 172.21.15.42:0/3251934280 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154c1e1b0 msgr2=0x55f154fcdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:53.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.941+0000 7fce3bfff640 1 --2- 172.21.15.42:0/3251934280 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154c1e1b0 0x55f154fcdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.941+0000 7fce3bfff640 1 -- 172.21.15.42:0/3251934280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154bf52f0 msgr2=0x55f154fccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:53.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.941+0000 7fce3bfff640 1 --2- 172.21.15.42:0/3251934280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154bf52f0 0x55f154fccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.941+0000 7fce3bfff640 1 -- 172.21.15.42:0/3251934280 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f154c89790 con 0x55f154bfd4d0 2024-07-26T06:50:53.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.941+0000 7fce493c9640 1 --2- 172.21.15.42:0/3251934280 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154c1e1b0 0x55f154fcdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.941+0000 7fce48bc8640 1 --2- 172.21.15.42:0/3251934280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154bf52f0 0x55f154fccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:50:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.941+0000 7fce3bfff640 1 --2- 172.21.15.42:0/3251934280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fcd1d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fce34004810 tx=0x7fce340a67c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd2de467eb28b16d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.941+0000 7fce23fff640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.941+0000 7fce23fff640 1 -- 172.21.15.42:0/3251934280 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3401e070 con 0x55f154bfd4d0 2024-07-26T06:50:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.941+0000 7fce23fff640 1 -- 172.21.15.42:0/3251934280 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fce34017dd0 con 0x55f154bfd4d0 2024-07-26T06:50:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.941+0000 7fce51e29740 1 -- 172.21.15.42:0/3251934280 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f154f38300 con 0x55f154bfd4d0 2024-07-26T06:50:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.941+0000 7fce23fff640 1 -- 172.21.15.42:0/3251934280 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fce340a9c70 con 0x55f154bfd4d0 2024-07-26T06:50:53.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.941+0000 7fce51e29740 10 client.15402 fetch_fsmap learned FSMap version 29 2024-07-26T06:50:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.941+0000 7fce51e29740 1 -- 172.21.15.42:0/3251934280 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f154d02620 con 0x55f154bfd4d0 2024-07-26T06:50:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.941+0000 7fce23fff640 1 -- 172.21.15.42:0/3251934280 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55f154f38300 con 0x55f154bfd4d0 2024-07-26T06:50:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.942+0000 7fce23fff640 1 -- 172.21.15.42:0/3251934280 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fce340aa580 con 0x55f154bfd4d0 2024-07-26T06:50:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.942+0000 7fce51e29740 10 client.15402 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:50:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.942+0000 7fce51e29740 10 client.15402 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:50:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.942+0000 7fce51e29740 20 client.15402 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.942+0000 7fce51e29740 1 --2- 172.21.15.42:0/3251934280 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55f154bdcd80 0x55f154fd9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.942+0000 7fce51e29740 4 client.15402 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:50:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.942+0000 7fce51e29740 1 -- 172.21.15.42:0/3251934280 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f154f33da0 con 0x55f154bdcd80 2024-07-26T06:50:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.942+0000 7fce48bc8640 1 --2- 172.21.15.42:0/3251934280 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55f154bdcd80 0x55f154fd9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.942+0000 7fce23fff640 10 client.15402 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:50:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.942+0000 7fce48bc8640 1 --2- 172.21.15.42:0/3251934280 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55f154bdcd80 0x55f154fd9e20 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd1ddcb9f298d189 server_cookie=c9b68c39388912df in_seq=0 out_seq=0 2024-07-26T06:50:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.943+0000 7fce23fff640 1 -- 172.21.15.42:0/3251934280 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f154f33da0 con 0x55f154bdcd80 2024-07-26T06:50:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.943+0000 7fce23fff640 10 client.15402 handle_command_reply: tid=0 2024-07-26T06:50:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.943+0000 7fce51e29740 1 client.15402 shutdown 2024-07-26T06:50:53.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.943+0000 7fce51e29740 1 -- 172.21.15.42:0/3251934280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 msgr2=0x55f154fcd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.943+0000 7fce51e29740 1 --2- 172.21.15.42:0/3251934280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fcd1d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fce34004810 tx=0x7fce340a67c0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.943+0000 7fce51e29740 1 -- 172.21.15.42:0/3251934280 shutdown_connections 2024-07-26T06:50:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.943+0000 7fce51e29740 1 --2- 172.21.15.42:0/3251934280 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55f154bdcd80 0x55f154fd9e20 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.943+0000 7fce51e29740 1 --2- 172.21.15.42:0/3251934280 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154c1e1b0 0x55f154fcdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.943+0000 7fce51e29740 1 --2- 172.21.15.42:0/3251934280 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fcd1d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.943+0000 7fce23fff640 0 client.15402 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:50:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.943+0000 7fce51e29740 1 --2- 172.21.15.42:0/3251934280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154bf52f0 0x55f154fccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.943+0000 7fce51e29740 1 -- 172.21.15.42:0/3251934280 >> 172.21.15.42:0/3251934280 conn(0x55f154c12300 msgr2=0x55f154fbd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.944+0000 7fce51e29740 1 -- 172.21.15.42:0/3251934280 shutdown_connections 2024-07-26T06:50:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.944+0000 7fce51e29740 1 -- 172.21.15.42:0/3251934280 wait complete. 2024-07-26T06:50:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.944+0000 7fce51e29740 20 client.15402 trim_cache size 0 max 16384 2024-07-26T06:50:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.959+0000 7fce51e29740 1 Processor -- start 2024-07-26T06:50:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.959+0000 7fce51e29740 1 -- start start 2024-07-26T06:50:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.959+0000 7fce51e29740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154bf52f0 0x55f154fc4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.959+0000 7fce51e29740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fd4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.959+0000 7fce51e29740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154c1e1b0 0x55f154fc2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.959+0000 7fce51e29740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fce441aaad0 con 0x55f154bfd4d0 2024-07-26T06:50:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.959+0000 7fce51e29740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fce441ac610 con 0x55f154c1e1b0 2024-07-26T06:50:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.959+0000 7fce51e29740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fce441ab1a0 con 0x55f154bf52f0 2024-07-26T06:50:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.959+0000 7fce3bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fd4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.959+0000 7fce3bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fd4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54560/0 (socket says 172.21.15.42:54560) 2024-07-26T06:50:53.962 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.959+0000 7fce3bfff640 1 -- 172.21.15.42:0/706972089 learned_addr learned my addr 172.21.15.42:0/706972089 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:53.962 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.960+0000 7fce493c9640 1 --2- 172.21.15.42:0/706972089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154c1e1b0 0x55f154fc2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:53.962 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.960+0000 7fce3bfff640 1 -- 172.21.15.42:0/706972089 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154bf52f0 msgr2=0x55f154fc4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:53.962 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.960+0000 7fce3bfff640 1 --2- 172.21.15.42:0/706972089 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154bf52f0 0x55f154fc4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.962 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.960+0000 7fce3bfff640 1 -- 172.21.15.42:0/706972089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154c1e1b0 msgr2=0x55f154fc2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:53.962 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.960+0000 7fce3bfff640 1 --2- 172.21.15.42:0/706972089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154c1e1b0 0x55f154fc2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.962 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.960+0000 7fce3bfff640 1 -- 172.21.15.42:0/706972089 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f154fcc890 con 0x55f154bfd4d0 2024-07-26T06:50:53.962 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.960+0000 7fce3bfff640 1 --2- 172.21.15.42:0/706972089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fd4c20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fce34000c00 tx=0x7fce340a6830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3813fa46715fa5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:53.962 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.960+0000 7fce2a7fc640 1 -- 172.21.15.42:0/706972089 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3401e070 con 0x55f154bfd4d0 2024-07-26T06:50:53.962 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.960+0000 7fce2a7fc640 1 -- 172.21.15.42:0/706972089 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce340e6ca0 con 0x55f154bfd4d0 2024-07-26T06:50:53.962 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.960+0000 7fce2a7fc640 1 -- 172.21.15.42:0/706972089 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce340aa0f0 con 0x55f154bfd4d0 2024-07-26T06:50:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.961+0000 7fce51e29740 1 -- 172.21.15.42:0/706972089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 msgr2=0x55f154fd4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.961+0000 7fce51e29740 1 --2- 172.21.15.42:0/706972089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fd4c20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fce34000c00 tx=0x7fce340a6830 comp rx=0 tx=0).stop 2024-07-26T06:50:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.961+0000 7fce51e29740 1 -- 172.21.15.42:0/706972089 shutdown_connections 2024-07-26T06:50:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.961+0000 7fce51e29740 1 --2- 172.21.15.42:0/706972089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154c1e1b0 0x55f154fc2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.961+0000 7fce51e29740 1 --2- 172.21.15.42:0/706972089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154bfd4d0 0x55f154fd4c20 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.961+0000 7fce51e29740 1 --2- 172.21.15.42:0/706972089 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154bf52f0 0x55f154fc4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.961+0000 7fce51e29740 1 -- 172.21.15.42:0/706972089 >> 172.21.15.42:0/706972089 conn(0x55f154c12300 msgr2=0x55f154fc7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.961+0000 7fce51e29740 1 -- 172.21.15.42:0/706972089 shutdown_connections 2024-07-26T06:50:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.961+0000 7fce51e29740 1 -- 172.21.15.42:0/706972089 wait complete. 2024-07-26T06:50:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.961+0000 7fce51e29740 1 -- start start 2024-07-26T06:50:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.961+0000 7fce51e29740 1 Processor -- start 2024-07-26T06:50:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce51e29740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154bf52f0 0x55f154fd3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce51e29740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154bfd4d0 0x55f154fd3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce51e29740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154c1e1b0 0x55f154fd4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce51e29740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fce441abf40 con 0x55f154c1e1b0 2024-07-26T06:50:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce51e29740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fce441ab870 con 0x55f154bfd4d0 2024-07-26T06:50:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce51e29740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fce441a8ab0 con 0x55f154bf52f0 2024-07-26T06:50:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce51e29740 10 client.0 _ll_register_callbacks cb 0x7fce400071a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce493c9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154c1e1b0 0x55f154fd4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce493c9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154c1e1b0 0x55f154fd4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54576/0 (socket says 172.21.15.42:54576) 2024-07-26T06:50:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce493c9640 1 -- 172.21.15.42:0/2405619862 learned_addr learned my addr 172.21.15.42:0/2405619862 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce48bc8640 1 --2- 172.21.15.42:0/2405619862 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154bf52f0 0x55f154fd3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce3bfff640 1 --2- 172.21.15.42:0/2405619862 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154bfd4d0 0x55f154fd3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce493c9640 1 -- 172.21.15.42:0/2405619862 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154bf52f0 msgr2=0x55f154fd3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce493c9640 1 --2- 172.21.15.42:0/2405619862 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154bf52f0 0x55f154fd3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce493c9640 1 -- 172.21.15.42:0/2405619862 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154bfd4d0 msgr2=0x55f154fd3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce493c9640 1 --2- 172.21.15.42:0/2405619862 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154bfd4d0 0x55f154fd3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce493c9640 1 -- 172.21.15.42:0/2405619862 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1550405a0 con 0x55f154c1e1b0 2024-07-26T06:50:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.962+0000 7fce3bfff640 1 --2- 172.21.15.42:0/2405619862 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154bfd4d0 0x55f154fd3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:50:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.963+0000 7fce493c9640 1 --2- 172.21.15.42:0/2405619862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154c1e1b0 0x55f154fd4080 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fce3c035000 tx=0x7fce3c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3c1e012abc8729 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.963+0000 7fce23fff640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.963+0000 7fce23fff640 1 -- 172.21.15.42:0/2405619862 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c003e50 con 0x55f154c1e1b0 2024-07-26T06:50:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.963+0000 7fce23fff640 1 -- 172.21.15.42:0/2405619862 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fce3c004750 con 0x55f154c1e1b0 2024-07-26T06:50:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.963+0000 7fce51e29740 1 -- 172.21.15.42:0/2405619862 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce340e6ca0 con 0x55f154c1e1b0 2024-07-26T06:50:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.963+0000 7fce23fff640 1 -- 172.21.15.42:0/2405619862 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fce3c02f610 con 0x55f154c1e1b0 2024-07-26T06:50:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.963+0000 7fce51e29740 10 client.15408 fetch_fsmap learned FSMap version 29 2024-07-26T06:50:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.963+0000 7fce51e29740 1 -- 172.21.15.42:0/2405619862 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f1550418d0 con 0x55f154c1e1b0 2024-07-26T06:50:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.963+0000 7fce23fff640 1 -- 172.21.15.42:0/2405619862 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce340e6ca0 con 0x55f154c1e1b0 2024-07-26T06:50:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.963+0000 7fce23fff640 1 -- 172.21.15.42:0/2405619862 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fce3c044040 con 0x55f154c1e1b0 2024-07-26T06:50:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.963+0000 7fce51e29740 10 client.15408 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:50:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.963+0000 7fce51e29740 10 client.15408 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:50:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.963+0000 7fce51e29740 20 client.15408 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.963+0000 7fce51e29740 1 --2- 172.21.15.42:0/2405619862 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55f154bb21c0 0x55f155044320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.963+0000 7fce51e29740 4 client.15408 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:50:53.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.963+0000 7fce51e29740 1 -- 172.21.15.42:0/2405619862 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f154f314e0 con 0x55f154bb21c0 2024-07-26T06:50:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.964+0000 7fce48bc8640 1 --2- 172.21.15.42:0/2405619862 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55f154bb21c0 0x55f155044320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.964+0000 7fce48bc8640 1 --2- 172.21.15.42:0/2405619862 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55f154bb21c0 0x55f155044320 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3389eeee7b24165 server_cookie=2e84091e02d5afaf in_seq=0 out_seq=0 2024-07-26T06:50:53.971 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.964+0000 7fce23fff640 10 client.15408 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:50:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.973+0000 7fce23fff640 1 -- 172.21.15.42:0/2405619862 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f154f314e0 con 0x55f154bb21c0 2024-07-26T06:50:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.973+0000 7fce23fff640 10 client.15408 handle_command_reply: tid=0 2024-07-26T06:50:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.973+0000 7fce51e29740 1 client.15408 shutdown 2024-07-26T06:50:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.973+0000 7fce51e29740 1 -- 172.21.15.42:0/2405619862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154c1e1b0 msgr2=0x55f154fd4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.973+0000 7fce51e29740 1 --2- 172.21.15.42:0/2405619862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154c1e1b0 0x55f154fd4080 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fce3c035000 tx=0x7fce3c0095c0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.973+0000 7fce51e29740 1 -- 172.21.15.42:0/2405619862 shutdown_connections 2024-07-26T06:50:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.973+0000 7fce51e29740 1 --2- 172.21.15.42:0/2405619862 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55f154bb21c0 0x55f155044320 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.973+0000 7fce51e29740 1 --2- 172.21.15.42:0/2405619862 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f154c1e1b0 0x55f154fd4080 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.973+0000 7fce51e29740 1 --2- 172.21.15.42:0/2405619862 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f154bfd4d0 0x55f154fd3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.973+0000 7fce23fff640 0 client.15408 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:50:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.973+0000 7fce51e29740 1 --2- 172.21.15.42:0/2405619862 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f154bf52f0 0x55f154fd3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.973+0000 7fce51e29740 1 -- 172.21.15.42:0/2405619862 >> 172.21.15.42:0/2405619862 conn(0x55f154c12300 msgr2=0x55f154fbf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.973+0000 7fce51e29740 1 -- 172.21.15.42:0/2405619862 shutdown_connections 2024-07-26T06:50:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.973+0000 7fce51e29740 1 -- 172.21.15.42:0/2405619862 wait complete. 2024-07-26T06:50:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.973+0000 7fce51e29740 20 client.15408 trim_cache size 0 max 16384 2024-07-26T06:50:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:50:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:50:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:50:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:50:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.974+0000 7fce4ae2c640 1 -- 172.21.15.42:0/165677099 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fce2c0b3bd0 msgr2=0x7fce2c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.974+0000 7fce4ae2c640 1 --2- 172.21.15.42:0/165677099 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fce2c0b3bd0 0x7fce2c0b60e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fce3c02f540 tx=0x7fce3c031af0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.974+0000 7fce4ae2c640 1 -- 172.21.15.42:0/165677099 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce44103080 msgr2=0x7fce4419a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.974+0000 7fce4ae2c640 1 --2- 172.21.15.42:0/165677099 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce44103080 0x7fce4419a110 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fce3400b4a0 tx=0x7fce3400b960 comp rx=0 tx=0).stop 2024-07-26T06:50:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.974+0000 7fce4ae2c640 1 -- 172.21.15.42:0/165677099 shutdown_connections 2024-07-26T06:50:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.974+0000 7fce4ae2c640 1 --2- 172.21.15.42:0/165677099 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fce2c0b3bd0 0x7fce2c0b60e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.974+0000 7fce4ae2c640 1 --2- 172.21.15.42:0/165677099 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce44103a20 0x7fce4419acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.974+0000 7fce4ae2c640 1 --2- 172.21.15.42:0/165677099 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fce44103080 0x7fce4419a110 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.974+0000 7fce4ae2c640 1 --2- 172.21.15.42:0/165677099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce44102000 0x7fce44199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.974+0000 7fce4ae2c640 1 -- 172.21.15.42:0/165677099 >> 172.21.15.42:0/165677099 conn(0x7fce440fd960 msgr2=0x7fce440ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.974+0000 7fce4ae2c640 1 -- 172.21.15.42:0/165677099 shutdown_connections 2024-07-26T06:50:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:50:53.974+0000 7fce4ae2c640 1 -- 172.21.15.42:0/165677099 wait complete. 2024-07-26T06:50:54.856 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:54 smithi042 ceph-mon[32775]: pgmap v416: 97 pgs: 97 active+clean; 660 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.6 KiB/s wr, 14 op/s 2024-07-26T06:50:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:54 smithi164 ceph-mon[36970]: pgmap v416: 97 pgs: 97 active+clean; 660 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.6 KiB/s wr, 14 op/s 2024-07-26T06:50:55.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:54 smithi163 ceph-mon[36873]: pgmap v416: 97 pgs: 97 active+clean; 660 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.6 KiB/s wr, 14 op/s 2024-07-26T06:50:55.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.086+0000 7fa415374640 1 -- 172.21.15.42:0/3054668152 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa410109820 msgr2=0x7fa410110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:55.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.086+0000 7fa415374640 1 --2- 172.21.15.42:0/3054668152 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa410109820 0x7fa410110050 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa408009ca0 tx=0x7fa40802a8c0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.087+0000 7fa415374640 1 -- 172.21.15.42:0/3054668152 shutdown_connections 2024-07-26T06:50:55.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.087+0000 7fa415374640 1 --2- 172.21.15.42:0/3054668152 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa410109820 0x7fa410110050 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.087+0000 7fa415374640 1 --2- 172.21.15.42:0/3054668152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa410108ed0 0x7fa4101092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.087+0000 7fa415374640 1 --2- 172.21.15.42:0/3054668152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa410103590 0x7fa4101039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.087+0000 7fa415374640 1 -- 172.21.15.42:0/3054668152 >> 172.21.15.42:0/3054668152 conn(0x7fa41007b280 msgr2=0x7fa41007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:55.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.087+0000 7fa415374640 1 -- 172.21.15.42:0/3054668152 shutdown_connections 2024-07-26T06:50:55.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.087+0000 7fa415374640 1 -- 172.21.15.42:0/3054668152 wait complete. 2024-07-26T06:50:55.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.087+0000 7fa415374640 1 Processor -- start 2024-07-26T06:50:55.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa415374640 1 -- start start 2024-07-26T06:50:55.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa415374640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa410103590 0x7fa410075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa415374640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa410108ed0 0x7fa410075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa415374640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa410109820 0x7fa410075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa415374640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa4101134f0 con 0x7fa410108ed0 2024-07-26T06:50:55.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa415374640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa410113370 con 0x7fa410109820 2024-07-26T06:50:55.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa415374640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa410113670 con 0x7fa410103590 2024-07-26T06:50:55.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa4067fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa410108ed0 0x7fa410075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa4067fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa410108ed0 0x7fa410075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54592/0 (socket says 172.21.15.42:54592) 2024-07-26T06:50:55.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa4067fc640 1 -- 172.21.15.42:0/4083718214 learned_addr learned my addr 172.21.15.42:0/4083718214 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:55.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa40effd640 1 --2- 172.21.15.42:0/4083718214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa410103590 0x7fa410075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa40f7fe640 1 --2- 172.21.15.42:0/4083718214 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa410109820 0x7fa410075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa4067fc640 1 -- 172.21.15.42:0/4083718214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa410103590 msgr2=0x7fa410075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:55.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa4067fc640 1 --2- 172.21.15.42:0/4083718214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa410103590 0x7fa410075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa4067fc640 1 -- 172.21.15.42:0/4083718214 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa410109820 msgr2=0x7fa410075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:55.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa4067fc640 1 --2- 172.21.15.42:0/4083718214 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa410109820 0x7fa410075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa4067fc640 1 -- 172.21.15.42:0/4083718214 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4101a8140 con 0x7fa410108ed0 2024-07-26T06:50:55.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa40effd640 1 --2- 172.21.15.42:0/4083718214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa410103590 0x7fa410075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:55.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.088+0000 7fa40f7fe640 1 --2- 172.21.15.42:0/4083718214 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa410109820 0x7fa410075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:50:55.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.089+0000 7fa4067fc640 1 --2- 172.21.15.42:0/4083718214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa410108ed0 0x7fa410075680 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800e9c0 tx=0x7fa3f800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:55.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.089+0000 7fa40cff9640 1 -- 172.21.15.42:0/4083718214 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f800ccc0 con 0x7fa410108ed0 2024-07-26T06:50:55.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.089+0000 7fa415374640 1 -- 172.21.15.42:0/4083718214 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4101a8430 con 0x7fa410108ed0 2024-07-26T06:50:55.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.089+0000 7fa40cff9640 1 -- 172.21.15.42:0/4083718214 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3f80054d0 con 0x7fa410108ed0 2024-07-26T06:50:55.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.089+0000 7fa40cff9640 1 -- 172.21.15.42:0/4083718214 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8010430 con 0x7fa410108ed0 2024-07-26T06:50:55.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.089+0000 7fa415374640 1 -- 172.21.15.42:0/4083718214 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4101a8970 con 0x7fa410108ed0 2024-07-26T06:50:55.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.092+0000 7fa40cff9640 1 -- 172.21.15.42:0/4083718214 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fa3f8005020 con 0x7fa410108ed0 2024-07-26T06:50:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.092+0000 7fa41b8f8740 1 Processor -- start 2024-07-26T06:50:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.092+0000 7fa41b8f8740 1 -- start start 2024-07-26T06:50:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.092+0000 7fa41b8f8740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2def60 0x55926c6843a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.092+0000 7fa41b8f8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2b60a0 0x55926c6848e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.092+0000 7fa41b8f8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2be280 0x55926c684e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.092+0000 7fa41b8f8740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55926c5fa940 con 0x55926c2def60 2024-07-26T06:50:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.092+0000 7fa41b8f8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55926c5e9e40 con 0x55926c2b60a0 2024-07-26T06:50:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.092+0000 7fa41b8f8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55926c5ede80 con 0x55926c2be280 2024-07-26T06:50:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.092+0000 7fa40f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2def60 0x55926c6843a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.093+0000 7fa40f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2def60 0x55926c6843a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54594/0 (socket says 172.21.15.42:54594) 2024-07-26T06:50:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.093+0000 7fa40f7fe640 1 -- 172.21.15.42:0/4274840083 learned_addr learned my addr 172.21.15.42:0/4274840083 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.093+0000 7fa40effd640 1 --2- 172.21.15.42:0/4274840083 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2b60a0 0x55926c6848e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.093+0000 7fa4067fc640 1 --2- 172.21.15.42:0/4274840083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2be280 0x55926c684e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.093+0000 7fa40cff9640 1 --2- 172.21.15.42:0/4083718214 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa3fc0b3bd0 0x7fa3fc0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.093+0000 7fa40f7fe640 1 -- 172.21.15.42:0/4274840083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2be280 msgr2=0x55926c684e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.093+0000 7fa40f7fe640 1 --2- 172.21.15.42:0/4274840083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2be280 0x55926c684e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.093+0000 7fa40f7fe640 1 -- 172.21.15.42:0/4274840083 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2b60a0 msgr2=0x55926c6848e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.093+0000 7fa40f7fe640 1 --2- 172.21.15.42:0/4274840083 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2b60a0 0x55926c6848e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.093+0000 7fa40f7fe640 1 -- 172.21.15.42:0/4274840083 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55926c590a60 con 0x55926c2def60 2024-07-26T06:50:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.093+0000 7fa4067fc640 1 --2- 172.21.15.42:0/4274840083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2be280 0x55926c684e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.093+0000 7fa40effd640 1 --2- 172.21.15.42:0/4274840083 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2b60a0 0x55926c6848e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:50:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.093+0000 7fa40f7fe640 1 --2- 172.21.15.42:0/4083718214 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa3fc0b3bd0 0x7fa3fc0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.093+0000 7fa40cff9640 1 -- 172.21.15.42:0/4083718214 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fa3f80e4000 con 0x7fa410108ed0 2024-07-26T06:50:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.093+0000 7fa40f7fe640 1 --2- 172.21.15.42:0/4274840083 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2def60 0x55926c6843a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa408006010 tx=0x7fa408009950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=560d66edbb8ce9aa server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:55.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.094+0000 7fa4057fa640 1 -- 172.21.15.42:0/4274840083 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40804c070 con 0x55926c2def60 2024-07-26T06:50:55.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.094+0000 7fa4057fa640 1 -- 172.21.15.42:0/4274840083 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa408036070 con 0x55926c2def60 2024-07-26T06:50:55.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.094+0000 7fa4057fa640 1 -- 172.21.15.42:0/4274840083 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40802f660 con 0x55926c2def60 2024-07-26T06:50:55.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.095+0000 7fa41b8f8740 1 -- 172.21.15.42:0/4274840083 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2def60 msgr2=0x55926c6843a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:55.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.095+0000 7fa40f7fe640 1 --2- 172.21.15.42:0/4083718214 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa3fc0b3bd0 0x7fa3fc0b60e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa408049000 tx=0x7fa408046000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:55.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.095+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/4274840083 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2def60 0x55926c6843a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa408006010 tx=0x7fa408009950 comp rx=0 tx=0).stop 2024-07-26T06:50:55.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.095+0000 7fa41b8f8740 1 -- 172.21.15.42:0/4274840083 shutdown_connections 2024-07-26T06:50:55.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.095+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/4274840083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2be280 0x55926c684e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.095+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/4274840083 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2b60a0 0x55926c6848e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.095+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/4274840083 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2def60 0x55926c6843a0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.095+0000 7fa41b8f8740 1 -- 172.21.15.42:0/4274840083 >> 172.21.15.42:0/4274840083 conn(0x55926c2d3350 msgr2=0x55926c682290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:55.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.095+0000 7fa41b8f8740 1 -- 172.21.15.42:0/4274840083 shutdown_connections 2024-07-26T06:50:55.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.095+0000 7fa41b8f8740 1 -- 172.21.15.42:0/4274840083 wait complete. 2024-07-26T06:50:55.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.095+0000 7fa41b8f8740 1 -- start start 2024-07-26T06:50:55.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.095+0000 7fa41b8f8740 1 Processor -- start 2024-07-26T06:50:55.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa41b8f8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2b60a0 0x55926c68ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa41b8f8740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2be280 0x55926c6904f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa41b8f8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2def60 0x55926c69bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa41b8f8740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55926c44d5f0 con 0x55926c2be280 2024-07-26T06:50:55.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa41b8f8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55926c58c8b0 con 0x55926c2b60a0 2024-07-26T06:50:55.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa41b8f8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55926c692a00 con 0x55926c2def60 2024-07-26T06:50:55.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa41b8f8740 10 client.0 _ll_register_callbacks cb 0x55926c5967f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:55.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa4067fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2be280 0x55926c6904f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa4067fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2be280 0x55926c6904f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54604/0 (socket says 172.21.15.42:54604) 2024-07-26T06:50:55.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa4067fc640 1 -- 172.21.15.42:0/483261772 learned_addr learned my addr 172.21.15.42:0/483261772 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:55.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa40effd640 1 --2- 172.21.15.42:0/483261772 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2b60a0 0x55926c68ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa40f7fe640 1 --2- 172.21.15.42:0/483261772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2def60 0x55926c69bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa4067fc640 1 -- 172.21.15.42:0/483261772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2def60 msgr2=0x55926c69bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:55.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa4067fc640 1 --2- 172.21.15.42:0/483261772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2def60 0x55926c69bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa4067fc640 1 -- 172.21.15.42:0/483261772 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2b60a0 msgr2=0x55926c68ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:55.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa4067fc640 1 --2- 172.21.15.42:0/483261772 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2b60a0 0x55926c68ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa4067fc640 1 -- 172.21.15.42:0/483261772 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55926c34a540 con 0x55926c2be280 2024-07-26T06:50:55.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa40f7fe640 1 --2- 172.21.15.42:0/483261772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2def60 0x55926c69bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:55.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa4067fc640 1 --2- 172.21.15.42:0/483261772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2be280 0x55926c6904f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80aaf60 tx=0x7fa3f80aa800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9707fd483eedb6ec server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:55.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.096+0000 7fa3f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:55.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.097+0000 7fa3f67fc640 1 -- 172.21.15.42:0/483261772 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f80ab430 con 0x55926c2be280 2024-07-26T06:50:55.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.097+0000 7fa3f67fc640 1 -- 172.21.15.42:0/483261772 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fa3f80b3bf0 con 0x55926c2be280 2024-07-26T06:50:55.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.097+0000 7fa41b8f8740 1 -- 172.21.15.42:0/483261772 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa408036070 con 0x55926c2be280 2024-07-26T06:50:55.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.097+0000 7fa3f67fc640 1 -- 172.21.15.42:0/483261772 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa3f80aaae0 con 0x55926c2be280 2024-07-26T06:50:55.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.097+0000 7fa41b8f8740 10 client.15420 fetch_fsmap learned FSMap version 29 2024-07-26T06:50:55.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.097+0000 7fa41b8f8740 1 -- 172.21.15.42:0/483261772 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55926c69ea70 con 0x55926c2be280 2024-07-26T06:50:55.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.097+0000 7fa3f67fc640 1 -- 172.21.15.42:0/483261772 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa408036070 con 0x55926c2be280 2024-07-26T06:50:55.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.097+0000 7fa3f67fc640 1 -- 172.21.15.42:0/483261772 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa3f80e60f0 con 0x55926c2be280 2024-07-26T06:50:55.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.097+0000 7fa41b8f8740 10 client.15420 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:50:55.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.097+0000 7fa41b8f8740 10 client.15420 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T06:50:55.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.097+0000 7fa41b8f8740 20 client.15420 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:55.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.097+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/483261772 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x55926c29d8e0 0x55926c6a24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.097+0000 7fa41b8f8740 4 client.15420 mds_command: new command op to 24583 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:50:55.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.097+0000 7fa41b8f8740 1 -- 172.21.15.42:0/483261772 --> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55926c5fc890 con 0x55926c29d8e0 2024-07-26T06:50:55.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.098+0000 7fa40effd640 1 --2- 172.21.15.42:0/483261772 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x55926c29d8e0 0x55926c6a24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.098+0000 7fa40effd640 1 --2- 172.21.15.42:0/483261772 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x55926c29d8e0 0x55926c6a24b0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=622a251d712e3e51 server_cookie=8aacffa41cd52b32 in_seq=0 out_seq=0 2024-07-26T06:50:55.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.098+0000 7fa3f67fc640 10 client.15420 ms_handle_connect on v2:172.21.15.164:6836/3642457979 2024-07-26T06:50:55.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.099+0000 7fa3f67fc640 1 -- 172.21.15.42:0/483261772 <== mds.3 v2:172.21.15.164:6836/3642457979 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55926c5fc890 con 0x55926c29d8e0 2024-07-26T06:50:55.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.099+0000 7fa3f67fc640 10 client.15420 handle_command_reply: tid=0 2024-07-26T06:50:55.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.099+0000 7fa41b8f8740 1 client.15420 shutdown 2024-07-26T06:50:55.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.099+0000 7fa41b8f8740 1 -- 172.21.15.42:0/483261772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2be280 msgr2=0x55926c6904f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:55.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.099+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/483261772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2be280 0x55926c6904f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80aaf60 tx=0x7fa3f80aa800 comp rx=0 tx=0).stop 2024-07-26T06:50:55.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.099+0000 7fa41b8f8740 1 -- 172.21.15.42:0/483261772 shutdown_connections 2024-07-26T06:50:55.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.099+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/483261772 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x55926c29d8e0 0x55926c6a24b0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.099+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/483261772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2def60 0x55926c69bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.099+0000 7fa3f67fc640 0 client.15420 ms_handle_reset on v2:172.21.15.164:6836/3642457979 2024-07-26T06:50:55.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.099+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/483261772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2be280 0x55926c6904f0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.099+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/483261772 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2b60a0 0x55926c68ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.099+0000 7fa41b8f8740 1 -- 172.21.15.42:0/483261772 >> 172.21.15.42:0/483261772 conn(0x55926c2d3350 msgr2=0x55926c687580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:55.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.099+0000 7fa41b8f8740 1 -- 172.21.15.42:0/483261772 shutdown_connections 2024-07-26T06:50:55.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.099+0000 7fa41b8f8740 1 -- 172.21.15.42:0/483261772 wait complete. 2024-07-26T06:50:55.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.099+0000 7fa41b8f8740 20 client.15420 trim_cache size 0 max 16384 2024-07-26T06:50:55.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.117+0000 7fa41b8f8740 1 Processor -- start 2024-07-26T06:50:55.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.117+0000 7fa41b8f8740 1 -- start start 2024-07-26T06:50:55.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.118+0000 7fa41b8f8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2b60a0 0x55926c681bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.118+0000 7fa41b8f8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2be280 0x55926c682110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.118+0000 7fa41b8f8740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2def60 0x55926c689620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.118+0000 7fa41b8f8740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55926c6a07f0 con 0x55926c2def60 2024-07-26T06:50:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.118+0000 7fa41b8f8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55926c6a0ec0 con 0x55926c2be280 2024-07-26T06:50:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.118+0000 7fa41b8f8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55926c6a1590 con 0x55926c2b60a0 2024-07-26T06:50:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.118+0000 7fa40f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2def60 0x55926c689620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.118+0000 7fa40f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2def60 0x55926c689620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54620/0 (socket says 172.21.15.42:54620) 2024-07-26T06:50:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.118+0000 7fa40f7fe640 1 -- 172.21.15.42:0/817223763 learned_addr learned my addr 172.21.15.42:0/817223763 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.118+0000 7fa40effd640 1 --2- 172.21.15.42:0/817223763 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2b60a0 0x55926c681bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.118+0000 7fa4067fc640 1 --2- 172.21.15.42:0/817223763 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2be280 0x55926c682110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.118+0000 7fa40f7fe640 1 -- 172.21.15.42:0/817223763 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2b60a0 msgr2=0x55926c681bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:55.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.118+0000 7fa40f7fe640 1 --2- 172.21.15.42:0/817223763 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2b60a0 0x55926c681bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.118+0000 7fa40f7fe640 1 -- 172.21.15.42:0/817223763 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2be280 msgr2=0x55926c682110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:55.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.118+0000 7fa40f7fe640 1 --2- 172.21.15.42:0/817223763 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2be280 0x55926c682110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.118+0000 7fa40f7fe640 1 -- 172.21.15.42:0/817223763 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55926c68a9e0 con 0x55926c2def60 2024-07-26T06:50:55.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.118+0000 7fa4067fc640 1 --2- 172.21.15.42:0/817223763 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2be280 0x55926c682110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:55.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.119+0000 7fa40f7fe640 1 --2- 172.21.15.42:0/817223763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2def60 0x55926c689620 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa408031fd0 tx=0x7fa40803d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e09855b3dfe758e6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:55.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.119+0000 7fa404ff9640 1 -- 172.21.15.42:0/817223763 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa408058070 con 0x55926c2def60 2024-07-26T06:50:55.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.119+0000 7fa404ff9640 1 -- 172.21.15.42:0/817223763 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa408036050 con 0x55926c2def60 2024-07-26T06:50:55.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.119+0000 7fa404ff9640 1 -- 172.21.15.42:0/817223763 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40804c040 con 0x55926c2def60 2024-07-26T06:50:55.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.120+0000 7fa41b8f8740 1 -- 172.21.15.42:0/817223763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2def60 msgr2=0x55926c689620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:55.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.120+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/817223763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2def60 0x55926c689620 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa408031fd0 tx=0x7fa40803d000 comp rx=0 tx=0).stop 2024-07-26T06:50:55.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.120+0000 7fa41b8f8740 1 -- 172.21.15.42:0/817223763 shutdown_connections 2024-07-26T06:50:55.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.120+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/817223763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2def60 0x55926c689620 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.120+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/817223763 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2be280 0x55926c682110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.120+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/817223763 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2b60a0 0x55926c681bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.120+0000 7fa41b8f8740 1 -- 172.21.15.42:0/817223763 >> 172.21.15.42:0/817223763 conn(0x55926c2d3350 msgr2=0x55926c67ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:55.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.120+0000 7fa41b8f8740 1 -- 172.21.15.42:0/817223763 shutdown_connections 2024-07-26T06:50:55.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.120+0000 7fa41b8f8740 1 -- 172.21.15.42:0/817223763 wait complete. 2024-07-26T06:50:55.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.120+0000 7fa41b8f8740 1 -- start start 2024-07-26T06:50:55.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.121+0000 7fa41b8f8740 1 Processor -- start 2024-07-26T06:50:55.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.121+0000 7fa41b8f8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2b60a0 0x55926c681f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.121+0000 7fa41b8f8740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2be280 0x55926c680850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.121+0000 7fa41b8f8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2def60 0x55926c680df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.121+0000 7fa41b8f8740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55926c6a2330 con 0x55926c2be280 2024-07-26T06:50:55.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.121+0000 7fa41b8f8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55926c6a1c60 con 0x55926c2def60 2024-07-26T06:50:55.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.121+0000 7fa41b8f8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55926c6a2a00 con 0x55926c2b60a0 2024-07-26T06:50:55.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.121+0000 7fa41b8f8740 10 client.0 _ll_register_callbacks cb 0x7fa3f80aae30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:55.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.121+0000 7fa40f7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2def60 0x55926c680df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.121+0000 7fa40effd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2b60a0 0x55926c681f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.121+0000 7fa40f7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2def60 0x55926c680df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:51232/0 (socket says 172.21.15.42:51232) 2024-07-26T06:50:55.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.121+0000 7fa40f7fe640 1 -- 172.21.15.42:0/410167776 learned_addr learned my addr 172.21.15.42:0/410167776 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:55.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.121+0000 7fa4067fc640 1 --2- 172.21.15.42:0/410167776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2be280 0x55926c680850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.122+0000 7fa40f7fe640 1 -- 172.21.15.42:0/410167776 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2b60a0 msgr2=0x55926c681f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.122+0000 7fa40f7fe640 1 --2- 172.21.15.42:0/410167776 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2b60a0 0x55926c681f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.122+0000 7fa40f7fe640 1 -- 172.21.15.42:0/410167776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2be280 msgr2=0x55926c680850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.122+0000 7fa40f7fe640 1 --2- 172.21.15.42:0/410167776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2be280 0x55926c680850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.122+0000 7fa40f7fe640 1 -- 172.21.15.42:0/410167776 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55926c681580 con 0x55926c2def60 2024-07-26T06:50:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.122+0000 7fa40f7fe640 1 --2- 172.21.15.42:0/410167776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2def60 0x55926c680df0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa408009bf0 tx=0x7fa408009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f1dca363d17bf6b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.122+0000 7fa3f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:50:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.122+0000 7fa3f67fc640 1 -- 172.21.15.42:0/410167776 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40804c030 con 0x55926c2def60 2024-07-26T06:50:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.122+0000 7fa3f67fc640 1 -- 172.21.15.42:0/410167776 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fa408004040 con 0x55926c2def60 2024-07-26T06:50:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.122+0000 7fa41b8f8740 1 -- 172.21.15.42:0/410167776 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa408036050 con 0x55926c2def60 2024-07-26T06:50:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.122+0000 7fa3f67fc640 1 -- 172.21.15.42:0/410167776 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa408058040 con 0x55926c2def60 2024-07-26T06:50:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.122+0000 7fa41b8f8740 10 client.25076 fetch_fsmap learned FSMap version 29 2024-07-26T06:50:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.123+0000 7fa41b8f8740 1 -- 172.21.15.42:0/410167776 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55926c6aa6b0 con 0x55926c2def60 2024-07-26T06:50:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.123+0000 7fa3f67fc640 1 -- 172.21.15.42:0/410167776 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa408036050 con 0x55926c2def60 2024-07-26T06:50:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.123+0000 7fa3f67fc640 1 -- 172.21.15.42:0/410167776 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa4080305c0 con 0x55926c2def60 2024-07-26T06:50:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.123+0000 7fa41b8f8740 10 client.25076 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:50:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.123+0000 7fa41b8f8740 10 client.25076 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T06:50:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.123+0000 7fa41b8f8740 20 client.25076 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.123+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/410167776 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x55926c272c90 0x55926c70dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.123+0000 7fa41b8f8740 4 client.25076 mds_command: new command op to 24583 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:50:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.123+0000 7fa41b8f8740 1 -- 172.21.15.42:0/410167776 --> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] -- command(tid 0: {"prefix": "status"}) -- 0x55926c5f8330 con 0x55926c272c90 2024-07-26T06:50:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.123+0000 7fa40effd640 1 --2- 172.21.15.42:0/410167776 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x55926c272c90 0x55926c70dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:55.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.124+0000 7fa3f67fc640 10 client.25076 ms_handle_connect on v2:172.21.15.164:6836/3642457979 2024-07-26T06:50:55.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.124+0000 7fa40effd640 1 --2- 172.21.15.42:0/410167776 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x55926c272c90 0x55926c70dcb0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c26dd0eb65a9fcda server_cookie=4d8bd467f81df434 in_seq=0 out_seq=0 2024-07-26T06:50:55.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.124+0000 7fa3f67fc640 1 -- 172.21.15.42:0/410167776 <== mds.3 v2:172.21.15.164:6836/3642457979 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55926c5f8330 con 0x55926c272c90 2024-07-26T06:50:55.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.124+0000 7fa3f67fc640 10 client.25076 handle_command_reply: tid=0 2024-07-26T06:50:55.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.124+0000 7fa41b8f8740 1 client.25076 shutdown 2024-07-26T06:50:55.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.124+0000 7fa41b8f8740 1 -- 172.21.15.42:0/410167776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2def60 msgr2=0x55926c680df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:55.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.124+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/410167776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2def60 0x55926c680df0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa408009bf0 tx=0x7fa408009800 comp rx=0 tx=0).stop 2024-07-26T06:50:55.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.124+0000 7fa41b8f8740 1 -- 172.21.15.42:0/410167776 shutdown_connections 2024-07-26T06:50:55.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.124+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/410167776 >> [v2:172.21.15.164:6836/3642457979,v1:172.21.15.164:6837/3642457979] conn(0x55926c272c90 0x55926c70dcb0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.124+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/410167776 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55926c2def60 0x55926c680df0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.124+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/410167776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55926c2be280 0x55926c680850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.124+0000 7fa3f67fc640 0 client.25076 ms_handle_reset on v2:172.21.15.164:6836/3642457979 2024-07-26T06:50:55.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.124+0000 7fa41b8f8740 1 --2- 172.21.15.42:0/410167776 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55926c2b60a0 0x55926c681f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.124+0000 7fa41b8f8740 1 -- 172.21.15.42:0/410167776 >> 172.21.15.42:0/410167776 conn(0x55926c2d3350 msgr2=0x55926c67ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:55.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.124+0000 7fa41b8f8740 1 -- 172.21.15.42:0/410167776 shutdown_connections 2024-07-26T06:50:55.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.124+0000 7fa41b8f8740 1 -- 172.21.15.42:0/410167776 wait complete. 2024-07-26T06:50:55.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.125+0000 7fa41b8f8740 20 client.25076 trim_cache size 0 max 16384 2024-07-26T06:50:55.130 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:50:55.130 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_fsid": "8b3e2750-4b19-11ef-bcb6-c7b262605968", 2024-07-26T06:50:55.130 INFO:teuthology.orchestra.run.smithi042.stdout: "whoami": 3, 2024-07-26T06:50:55.130 INFO:teuthology.orchestra.run.smithi042.stdout: "id": 24583, 2024-07-26T06:50:55.130 INFO:teuthology.orchestra.run.smithi042.stdout: "want_state": "up:active", 2024-07-26T06:50:55.131 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "up:active", 2024-07-26T06:50:55.131 INFO:teuthology.orchestra.run.smithi042.stdout: "fs_name": "cephfs", 2024-07-26T06:50:55.131 INFO:teuthology.orchestra.run.smithi042.stdout: "rank_uptime": 214.041892354, 2024-07-26T06:50:55.131 INFO:teuthology.orchestra.run.smithi042.stdout: "mdsmap_epoch": 29, 2024-07-26T06:50:55.131 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch": 80, 2024-07-26T06:50:55.131 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch_barrier": 80, 2024-07-26T06:50:55.131 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 259.389955757 2024-07-26T06:50:55.131 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:50:55.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.125+0000 7fa415374640 1 -- 172.21.15.42:0/4083718214 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa3fc0b3bd0 msgr2=0x7fa3fc0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:55.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.125+0000 7fa415374640 1 --2- 172.21.15.42:0/4083718214 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa3fc0b3bd0 0x7fa3fc0b60e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa408049000 tx=0x7fa408046000 comp rx=0 tx=0).stop 2024-07-26T06:50:55.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.125+0000 7fa415374640 1 -- 172.21.15.42:0/4083718214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa410108ed0 msgr2=0x7fa410075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:55.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.125+0000 7fa415374640 1 --2- 172.21.15.42:0/4083718214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa410108ed0 0x7fa410075680 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800e9c0 tx=0x7fa3f800ee80 comp rx=0 tx=0).stop 2024-07-26T06:50:55.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.125+0000 7fa415374640 1 -- 172.21.15.42:0/4083718214 shutdown_connections 2024-07-26T06:50:55.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.125+0000 7fa415374640 1 --2- 172.21.15.42:0/4083718214 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa3fc0b3bd0 0x7fa3fc0b60e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.125+0000 7fa415374640 1 --2- 172.21.15.42:0/4083718214 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa410109820 0x7fa410075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.125+0000 7fa415374640 1 --2- 172.21.15.42:0/4083718214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa410108ed0 0x7fa410075680 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.125+0000 7fa415374640 1 --2- 172.21.15.42:0/4083718214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa410103590 0x7fa410075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:55.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.125+0000 7fa415374640 1 -- 172.21.15.42:0/4083718214 >> 172.21.15.42:0/4083718214 conn(0x7fa41007b280 msgr2=0x7fa410106f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:55.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.125+0000 7fa415374640 1 -- 172.21.15.42:0/4083718214 shutdown_connections 2024-07-26T06:50:55.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:55.126+0000 7fa415374640 1 -- 172.21.15.42:0/4083718214 wait complete. 2024-07-26T06:50:55.812 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell mds.h status 2024-07-26T06:50:56.088 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:56 smithi042 ceph-mon[32775]: pgmap v417: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.6 KiB/s wr, 15 op/s 2024-07-26T06:50:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:56 smithi164 ceph-mon[36970]: pgmap v417: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.6 KiB/s wr, 15 op/s 2024-07-26T06:50:57.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:56 smithi163 ceph-mon[36873]: pgmap v417: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.6 KiB/s wr, 15 op/s 2024-07-26T06:50:57.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.769+0000 7f5b4120f640 1 -- 172.21.15.42:0/1321792607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b3c102490 msgr2=0x7f5b3c1028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:57.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.769+0000 7f5b4120f640 1 --2- 172.21.15.42:0/1321792607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b3c102490 0x7f5b3c1028a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c009900 tx=0x7f5b2c02a3f0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.769+0000 7f5b4120f640 1 -- 172.21.15.42:0/1321792607 shutdown_connections 2024-07-26T06:50:57.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.769+0000 7f5b4120f640 1 --2- 172.21.15.42:0/1321792607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b3c108720 0x7f5b3c10ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.769+0000 7f5b4120f640 1 --2- 172.21.15.42:0/1321792607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b3c107dd0 0x7f5b3c1081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.769+0000 7f5b4120f640 1 --2- 172.21.15.42:0/1321792607 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b3c102490 0x7f5b3c1028a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.769+0000 7f5b4120f640 1 -- 172.21.15.42:0/1321792607 >> 172.21.15.42:0/1321792607 conn(0x7f5b3c079160 msgr2=0x7f5b3c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:57.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.769+0000 7f5b4120f640 1 -- 172.21.15.42:0/1321792607 shutdown_connections 2024-07-26T06:50:57.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.769+0000 7f5b4120f640 1 -- 172.21.15.42:0/1321792607 wait complete. 2024-07-26T06:50:57.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.770+0000 7f5b4120f640 1 Processor -- start 2024-07-26T06:50:57.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.770+0000 7f5b4120f640 1 -- start start 2024-07-26T06:50:57.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.770+0000 7f5b4120f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b3c102490 0x7f5b3c199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.770+0000 7f5b4120f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b3c107dd0 0x7f5b3c19a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.770+0000 7f5b4120f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b3c108720 0x7f5b3c19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.770+0000 7f5b4120f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3c1134b0 con 0x7f5b3c102490 2024-07-26T06:50:57.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.770+0000 7f5b4120f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3c113330 con 0x7f5b3c107dd0 2024-07-26T06:50:57.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.770+0000 7f5b4120f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3c113630 con 0x7f5b3c108720 2024-07-26T06:50:57.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.771+0000 7f5b3ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b3c102490 0x7f5b3c199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:57.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.771+0000 7f5b3ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b3c102490 0x7f5b3c199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55968/0 (socket says 172.21.15.42:55968) 2024-07-26T06:50:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.771+0000 7f5b3ad9d640 1 -- 172.21.15.42:0/2381072797 learned_addr learned my addr 172.21.15.42:0/2381072797 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.771+0000 7f5b3b59e640 1 --2- 172.21.15.42:0/2381072797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b3c108720 0x7f5b3c19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.771+0000 7f5b3a59c640 1 --2- 172.21.15.42:0/2381072797 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b3c107dd0 0x7f5b3c19a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.771+0000 7f5b3ad9d640 1 -- 172.21.15.42:0/2381072797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b3c108720 msgr2=0x7f5b3c19f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.771+0000 7f5b3ad9d640 1 --2- 172.21.15.42:0/2381072797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b3c108720 0x7f5b3c19f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.771+0000 7f5b3ad9d640 1 -- 172.21.15.42:0/2381072797 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b3c107dd0 msgr2=0x7f5b3c19a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.771+0000 7f5b3ad9d640 1 --2- 172.21.15.42:0/2381072797 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b3c107dd0 0x7f5b3c19a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.771+0000 7f5b3ad9d640 1 -- 172.21.15.42:0/2381072797 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b3c19fd70 con 0x7f5b3c102490 2024-07-26T06:50:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.771+0000 7f5b3ad9d640 1 --2- 172.21.15.42:0/2381072797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b3c102490 0x7f5b3c199ec0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c005940 tx=0x7f5b2c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.771+0000 7f5b23fff640 1 -- 172.21.15.42:0/2381072797 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c03f030 con 0x7f5b3c102490 2024-07-26T06:50:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.771+0000 7f5b23fff640 1 -- 172.21.15.42:0/2381072797 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b2c003cb0 con 0x7f5b3c102490 2024-07-26T06:50:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.771+0000 7f5b4120f640 1 -- 172.21.15.42:0/2381072797 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b3c1a0000 con 0x7f5b3c102490 2024-07-26T06:50:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.772+0000 7f5b23fff640 1 -- 172.21.15.42:0/2381072797 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c035070 con 0x7f5b3c102490 2024-07-26T06:50:57.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.772+0000 7f5b4120f640 1 -- 172.21.15.42:0/2381072797 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b3c1a0530 con 0x7f5b3c102490 2024-07-26T06:50:57.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.775+0000 7f5b47793740 1 Processor -- start 2024-07-26T06:50:57.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.775+0000 7f5b47793740 1 -- start start 2024-07-26T06:50:57.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.775+0000 7f5b47793740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e43c2f60 0x55f3e4765de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.775+0000 7f5b47793740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f3e439a0a0 0x55f3e4766360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.775+0000 7f5b47793740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e4778730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.775+0000 7f5b47793740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55f3e46de920 con 0x55f3e43c2f60 2024-07-26T06:50:57.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.775+0000 7f5b47793740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f3e46cde20 con 0x55f3e439a0a0 2024-07-26T06:50:57.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.775+0000 7f5b47793740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55f3e46d1e60 con 0x55f3e43a2280 2024-07-26T06:50:57.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.775+0000 7f5b3b59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e43c2f60 0x55f3e4765de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:57.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.775+0000 7f5b3b59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e43c2f60 0x55f3e4765de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55976/0 (socket says 172.21.15.42:55976) 2024-07-26T06:50:57.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.775+0000 7f5b3b59e640 1 -- 172.21.15.42:0/4050492382 learned_addr learned my addr 172.21.15.42:0/4050492382 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:57.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.775+0000 7f5b23fff640 1 -- 172.21.15.42:0/2381072797 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f5b2c042030 con 0x7f5b3c102490 2024-07-26T06:50:57.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.775+0000 7f5b3a59c640 1 --2- 172.21.15.42:0/4050492382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e4778730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:57.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.775+0000 7f5b3b59e640 1 -- 172.21.15.42:0/4050492382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 msgr2=0x55f3e4778730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:57.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.776+0000 7f5b3b59e640 1 --2- 172.21.15.42:0/4050492382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e4778730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.776+0000 7f5b3b59e640 1 -- 172.21.15.42:0/4050492382 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f3e439a0a0 msgr2=0x55f3e4766360 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T06:50:57.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.776+0000 7f5b3b59e640 1 --2- 172.21.15.42:0/4050492382 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f3e439a0a0 0x55f3e4766360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.776+0000 7f5b3b59e640 1 -- 172.21.15.42:0/4050492382 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3e4674a40 con 0x55f3e43c2f60 2024-07-26T06:50:57.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.776+0000 7f5b3a59c640 1 --2- 172.21.15.42:0/4050492382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e4778730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:57.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.776+0000 7f5b3b59e640 1 --2- 172.21.15.42:0/4050492382 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e43c2f60 0x55f3e4765de0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5b3400b6b0 tx=0x7f5b34004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7cd6e0580b20bf0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:57.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.776+0000 7f5b217fa640 1 -- 172.21.15.42:0/4050492382 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b34003b20 con 0x55f3e43c2f60 2024-07-26T06:50:57.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.776+0000 7f5b217fa640 1 -- 172.21.15.42:0/4050492382 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b34015070 con 0x55f3e43c2f60 2024-07-26T06:50:57.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.776+0000 7f5b217fa640 1 -- 172.21.15.42:0/4050492382 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b340116a0 con 0x55f3e43c2f60 2024-07-26T06:50:57.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.776+0000 7f5b23fff640 1 --2- 172.21.15.42:0/2381072797 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5b240b3c80 0x7f5b240b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.777+0000 7f5b3ad9d640 1 --2- 172.21.15.42:0/2381072797 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5b240b3c80 0x7f5b240b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:57.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.777+0000 7f5b23fff640 1 -- 172.21.15.42:0/2381072797 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f5b2c103ff0 con 0x7f5b3c102490 2024-07-26T06:50:57.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.777+0000 7f5b47793740 1 -- 172.21.15.42:0/4050492382 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e43c2f60 msgr2=0x55f3e4765de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:57.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.777+0000 7f5b47793740 1 --2- 172.21.15.42:0/4050492382 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e43c2f60 0x55f3e4765de0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5b3400b6b0 tx=0x7f5b34004610 comp rx=0 tx=0).stop 2024-07-26T06:50:57.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.777+0000 7f5b3ad9d640 1 --2- 172.21.15.42:0/2381072797 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5b240b3c80 0x7f5b240b6190 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c006010 tx=0x7f5b2c095370 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:57.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.777+0000 7f5b47793740 1 -- 172.21.15.42:0/4050492382 shutdown_connections 2024-07-26T06:50:57.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.777+0000 7f5b47793740 1 --2- 172.21.15.42:0/4050492382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e4778730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.777+0000 7f5b47793740 1 --2- 172.21.15.42:0/4050492382 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f3e439a0a0 0x55f3e4766360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.777+0000 7f5b47793740 1 --2- 172.21.15.42:0/4050492382 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e43c2f60 0x55f3e4765de0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.777+0000 7f5b47793740 1 -- 172.21.15.42:0/4050492382 >> 172.21.15.42:0/4050492382 conn(0x55f3e43b7350 msgr2=0x55f3e4768300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:57.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.777+0000 7f5b47793740 1 -- 172.21.15.42:0/4050492382 shutdown_connections 2024-07-26T06:50:57.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.777+0000 7f5b47793740 1 -- 172.21.15.42:0/4050492382 wait complete. 2024-07-26T06:50:57.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.778+0000 7f5b47793740 1 -- start start 2024-07-26T06:50:57.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.778+0000 7f5b47793740 1 Processor -- start 2024-07-26T06:50:57.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.778+0000 7f5b47793740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f3e439a0a0 0x55f3e477ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.778+0000 7f5b47793740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e477d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.778+0000 7f5b47793740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e43c2f60 0x55f3e4782420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.778+0000 7f5b47793740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55f3e45315f0 con 0x55f3e43c2f60 2024-07-26T06:50:57.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.778+0000 7f5b47793740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f3e4670890 con 0x55f3e439a0a0 2024-07-26T06:50:57.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.778+0000 7f5b47793740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55f3e4779230 con 0x55f3e43a2280 2024-07-26T06:50:57.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.778+0000 7f5b47793740 10 client.0 _ll_register_callbacks cb 0x55f3e467a7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:57.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.778+0000 7f5b3b59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e43c2f60 0x55f3e4782420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:57.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.778+0000 7f5b3b59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e43c2f60 0x55f3e4782420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55982/0 (socket says 172.21.15.42:55982) 2024-07-26T06:50:57.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.778+0000 7f5b3b59e640 1 -- 172.21.15.42:0/33410074 learned_addr learned my addr 172.21.15.42:0/33410074 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:57.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.778+0000 7f5b3b59e640 1 --2- 172.21.15.42:0/33410074 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f3e439a0a0 0x55f3e477ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:57.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.778+0000 7f5b3a59c640 1 --2- 172.21.15.42:0/33410074 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e477d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:57.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.779+0000 7f5b3b59e640 1 -- 172.21.15.42:0/33410074 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 msgr2=0x55f3e477d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:57.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.779+0000 7f5b3b59e640 1 --2- 172.21.15.42:0/33410074 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e477d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.779+0000 7f5b3b59e640 1 -- 172.21.15.42:0/33410074 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f3e439a0a0 msgr2=0x55f3e477ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:57.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.779+0000 7f5b3b59e640 1 --2- 172.21.15.42:0/33410074 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f3e439a0a0 0x55f3e477ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.779+0000 7f5b3b59e640 1 -- 172.21.15.42:0/33410074 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3e442e540 con 0x55f3e43c2f60 2024-07-26T06:50:57.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.779+0000 7f5b3a59c640 1 --2- 172.21.15.42:0/33410074 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e477d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:50:57.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.779+0000 7f5b3b59e640 1 --2- 172.21.15.42:0/33410074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e43c2f60 0x55f3e4782420 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5b34011840 tx=0x7f5b3401ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b91fa623352cfbd8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:57.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.779+0000 7f5b1eff5640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:57.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.779+0000 7f5b1eff5640 1 -- 172.21.15.42:0/33410074 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b34005230 con 0x55f3e43c2f60 2024-07-26T06:50:57.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.779+0000 7f5b47793740 1 -- 172.21.15.42:0/33410074 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b34015070 con 0x55f3e43c2f60 2024-07-26T06:50:57.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.779+0000 7f5b1eff5640 1 -- 172.21.15.42:0/33410074 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f5b340053d0 con 0x55f3e43c2f60 2024-07-26T06:50:57.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.779+0000 7f5b1eff5640 1 -- 172.21.15.42:0/33410074 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5b3401a470 con 0x55f3e43c2f60 2024-07-26T06:50:57.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.779+0000 7f5b47793740 10 client.15438 fetch_fsmap learned FSMap version 29 2024-07-26T06:50:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.779+0000 7f5b47793740 1 -- 172.21.15.42:0/33410074 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f3e478b370 con 0x55f3e43c2f60 2024-07-26T06:50:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.779+0000 7f5b1eff5640 1 -- 172.21.15.42:0/33410074 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b34015070 con 0x55f3e43c2f60 2024-07-26T06:50:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.780+0000 7f5b1eff5640 1 -- 172.21.15.42:0/33410074 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5b34003c60 con 0x55f3e43c2f60 2024-07-26T06:50:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.780+0000 7f5b47793740 10 client.15438 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:50:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.780+0000 7f5b47793740 10 client.15438 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T06:50:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.780+0000 7f5b47793740 20 client.15438 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.780+0000 7f5b47793740 1 --2- 172.21.15.42:0/33410074 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55f3e43818e0 0x55f3e478e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.780+0000 7f5b47793740 4 client.15438 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:50:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.780+0000 7f5b47793740 1 -- 172.21.15.42:0/33410074 --> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3e46e0870 con 0x55f3e43818e0 2024-07-26T06:50:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.780+0000 7f5b3a59c640 1 --2- 172.21.15.42:0/33410074 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55f3e43818e0 0x55f3e478e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.780+0000 7f5b3a59c640 1 --2- 172.21.15.42:0/33410074 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55f3e43818e0 0x55f3e478e9c0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2b6242a4d282627f server_cookie=eb476a1f0df64d6f in_seq=0 out_seq=0 2024-07-26T06:50:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.781+0000 7f5b1eff5640 10 client.15438 ms_handle_connect on v2:172.21.15.163:6836/154041066 2024-07-26T06:50:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.781+0000 7f5b1eff5640 1 -- 172.21.15.42:0/33410074 <== mds.2 v2:172.21.15.163:6836/154041066 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f3e46e0870 con 0x55f3e43818e0 2024-07-26T06:50:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.781+0000 7f5b1eff5640 10 client.15438 handle_command_reply: tid=0 2024-07-26T06:50:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.781+0000 7f5b47793740 1 client.15438 shutdown 2024-07-26T06:50:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.781+0000 7f5b47793740 1 -- 172.21.15.42:0/33410074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e43c2f60 msgr2=0x55f3e4782420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.781+0000 7f5b47793740 1 --2- 172.21.15.42:0/33410074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e43c2f60 0x55f3e4782420 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5b34011840 tx=0x7f5b3401ad30 comp rx=0 tx=0).stop 2024-07-26T06:50:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.782+0000 7f5b47793740 1 -- 172.21.15.42:0/33410074 shutdown_connections 2024-07-26T06:50:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.782+0000 7f5b47793740 1 --2- 172.21.15.42:0/33410074 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55f3e43818e0 0x55f3e478e9c0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.782+0000 7f5b47793740 1 --2- 172.21.15.42:0/33410074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e43c2f60 0x55f3e4782420 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.782+0000 7f5b47793740 1 --2- 172.21.15.42:0/33410074 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e477d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.782+0000 7f5b47793740 1 --2- 172.21.15.42:0/33410074 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f3e439a0a0 0x55f3e477ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.782+0000 7f5b47793740 1 -- 172.21.15.42:0/33410074 >> 172.21.15.42:0/33410074 conn(0x55f3e43b7350 msgr2=0x55f3e47672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.782+0000 7f5b1eff5640 0 client.15438 ms_handle_reset on v2:172.21.15.163:6836/154041066 2024-07-26T06:50:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.782+0000 7f5b47793740 1 -- 172.21.15.42:0/33410074 shutdown_connections 2024-07-26T06:50:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.782+0000 7f5b47793740 1 -- 172.21.15.42:0/33410074 wait complete. 2024-07-26T06:50:57.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.782+0000 7f5b47793740 20 client.15438 trim_cache size 0 max 16384 2024-07-26T06:50:57.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.800+0000 7f5b47793740 1 Processor -- start 2024-07-26T06:50:57.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.800+0000 7f5b47793740 1 -- start start 2024-07-26T06:50:57.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.800+0000 7f5b47793740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e439a0a0 0x55f3e478d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.800+0000 7f5b47793740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e4778650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.800+0000 7f5b47793740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f3e43c2f60 0x55f3e4768250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.800+0000 7f5b47793740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55f3e478cd00 con 0x55f3e439a0a0 2024-07-26T06:50:57.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.800+0000 7f5b47793740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f3e478d3d0 con 0x55f3e43c2f60 2024-07-26T06:50:57.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.800+0000 7f5b47793740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55f3e478daa0 con 0x55f3e43a2280 2024-07-26T06:50:57.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.800+0000 7f5b3b59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e439a0a0 0x55f3e478d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.800+0000 7f5b3b59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e439a0a0 0x55f3e478d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55992/0 (socket says 172.21.15.42:55992) 2024-07-26T06:50:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.800+0000 7f5b3b59e640 1 -- 172.21.15.42:0/4094093211 learned_addr learned my addr 172.21.15.42:0/4094093211 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.800+0000 7f5b3a59c640 1 --2- 172.21.15.42:0/4094093211 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e4778650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.801+0000 7f5b3b59e640 1 -- 172.21.15.42:0/4094093211 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 msgr2=0x55f3e4778650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.801+0000 7f5b3b59e640 1 --2- 172.21.15.42:0/4094093211 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e4778650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.801+0000 7f5b3b59e640 1 -- 172.21.15.42:0/4094093211 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f3e43c2f60 msgr2=0x55f3e4768250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.801+0000 7f5b3b59e640 1 --2- 172.21.15.42:0/4094093211 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f3e43c2f60 0x55f3e4768250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.801+0000 7f5b3b59e640 1 -- 172.21.15.42:0/4094093211 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3e4766fb0 con 0x55f3e439a0a0 2024-07-26T06:50:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.801+0000 7f5b3b59e640 1 --2- 172.21.15.42:0/4094093211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e439a0a0 0x55f3e478d550 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f5b34004070 tx=0x7f5b3400b2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ad48189bfd1f7b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.801+0000 7f5b20ff9640 1 -- 172.21.15.42:0/4094093211 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b34038040 con 0x55f3e439a0a0 2024-07-26T06:50:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.801+0000 7f5b20ff9640 1 -- 172.21.15.42:0/4094093211 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b34015040 con 0x55f3e439a0a0 2024-07-26T06:50:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.801+0000 7f5b20ff9640 1 -- 172.21.15.42:0/4094093211 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b34004fb0 con 0x55f3e439a0a0 2024-07-26T06:50:57.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.802+0000 7f5b47793740 1 -- 172.21.15.42:0/4094093211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e439a0a0 msgr2=0x55f3e478d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:57.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.802+0000 7f5b47793740 1 --2- 172.21.15.42:0/4094093211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e439a0a0 0x55f3e478d550 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f5b34004070 tx=0x7f5b3400b2e0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.802+0000 7f5b47793740 1 -- 172.21.15.42:0/4094093211 shutdown_connections 2024-07-26T06:50:57.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.802+0000 7f5b47793740 1 --2- 172.21.15.42:0/4094093211 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f3e43c2f60 0x55f3e4768250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.802+0000 7f5b47793740 1 --2- 172.21.15.42:0/4094093211 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e4778650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.802+0000 7f5b47793740 1 --2- 172.21.15.42:0/4094093211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e439a0a0 0x55f3e478d550 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.802+0000 7f5b47793740 1 -- 172.21.15.42:0/4094093211 >> 172.21.15.42:0/4094093211 conn(0x55f3e43b7350 msgr2=0x55f3e47690d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.802+0000 7f5b47793740 1 -- 172.21.15.42:0/4094093211 shutdown_connections 2024-07-26T06:50:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.802+0000 7f5b47793740 1 -- 172.21.15.42:0/4094093211 wait complete. 2024-07-26T06:50:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.803+0000 7f5b47793740 1 -- start start 2024-07-26T06:50:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.803+0000 7f5b47793740 1 Processor -- start 2024-07-26T06:50:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.803+0000 7f5b47793740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e439a0a0 0x55f3e4796ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.803+0000 7f5b47793740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e4797220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.803+0000 7f5b47793740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f3e43c2f60 0x55f3e4779470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.803+0000 7f5b47793740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55f3e478e840 con 0x55f3e439a0a0 2024-07-26T06:50:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.803+0000 7f5b47793740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f3e478e170 con 0x55f3e43c2f60 2024-07-26T06:50:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.803+0000 7f5b47793740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55f3e478ef10 con 0x55f3e43a2280 2024-07-26T06:50:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.803+0000 7f5b47793740 10 client.0 _ll_register_callbacks cb 0x7f5b340118c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:50:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.803+0000 7f5b3b59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e439a0a0 0x55f3e4796ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.803+0000 7f5b3b59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e439a0a0 0x55f3e4796ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56000/0 (socket says 172.21.15.42:56000) 2024-07-26T06:50:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.803+0000 7f5b3b59e640 1 -- 172.21.15.42:0/1657281565 learned_addr learned my addr 172.21.15.42:0/1657281565 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:50:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.803+0000 7f5b3a59c640 1 --2- 172.21.15.42:0/1657281565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e4797220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.804+0000 7f5b3b59e640 1 -- 172.21.15.42:0/1657281565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 msgr2=0x55f3e4797220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.804+0000 7f5b3b59e640 1 --2- 172.21.15.42:0/1657281565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e4797220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.804+0000 7f5b3b59e640 1 -- 172.21.15.42:0/1657281565 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f3e43c2f60 msgr2=0x55f3e4779470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:50:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.804+0000 7f5b3b59e640 1 --2- 172.21.15.42:0/1657281565 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f3e43c2f60 0x55f3e4779470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.804+0000 7f5b3b59e640 1 -- 172.21.15.42:0/1657281565 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3e4779a60 con 0x55f3e439a0a0 2024-07-26T06:50:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.804+0000 7f5b3b59e640 1 --2- 172.21.15.42:0/1657281565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e439a0a0 0x55f3e4796ca0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5b34005d70 tx=0x7f5b34011750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cbd6cb1dd62393d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:50:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.804+0000 7f5b1eff5640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:50:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.804+0000 7f5b1eff5640 1 -- 172.21.15.42:0/1657281565 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3404b060 con 0x55f3e439a0a0 2024-07-26T06:50:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.804+0000 7f5b1eff5640 1 -- 172.21.15.42:0/1657281565 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f5b340048b0 con 0x55f3e439a0a0 2024-07-26T06:50:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.804+0000 7f5b47793740 1 -- 172.21.15.42:0/1657281565 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b34015040 con 0x55f3e439a0a0 2024-07-26T06:50:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.804+0000 7f5b1eff5640 1 -- 172.21.15.42:0/1657281565 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5b34038070 con 0x55f3e439a0a0 2024-07-26T06:50:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.804+0000 7f5b47793740 10 client.15444 fetch_fsmap learned FSMap version 29 2024-07-26T06:50:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.804+0000 7f5b47793740 1 -- 172.21.15.42:0/1657281565 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f3e4788d00 con 0x55f3e439a0a0 2024-07-26T06:50:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.805+0000 7f5b1eff5640 1 -- 172.21.15.42:0/1657281565 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b34015040 con 0x55f3e439a0a0 2024-07-26T06:50:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.805+0000 7f5b1eff5640 1 -- 172.21.15.42:0/1657281565 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5b34045030 con 0x55f3e439a0a0 2024-07-26T06:50:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.805+0000 7f5b47793740 10 client.15444 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:50:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.805+0000 7f5b47793740 10 client.15444 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T06:50:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.805+0000 7f5b47793740 20 client.15444 populate_metadata read hostname 'smithi042' 2024-07-26T06:50:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.805+0000 7f5b47793740 1 --2- 172.21.15.42:0/1657281565 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55f3e4356c90 0x55f3e47fa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:50:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.805+0000 7f5b47793740 4 client.15444 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:50:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.805+0000 7f5b47793740 1 -- 172.21.15.42:0/1657281565 --> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] -- command(tid 0: {"prefix": "status"}) -- 0x55f3e46dc310 con 0x55f3e4356c90 2024-07-26T06:50:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.805+0000 7f5b3a59c640 1 --2- 172.21.15.42:0/1657281565 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55f3e4356c90 0x55f3e47fa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:50:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.806+0000 7f5b3a59c640 1 --2- 172.21.15.42:0/1657281565 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55f3e4356c90 0x55f3e47fa5a0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7fc79ca614de46f2 server_cookie=6cb82c5aabf0d96c in_seq=0 out_seq=0 2024-07-26T06:50:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.806+0000 7f5b1eff5640 10 client.15444 ms_handle_connect on v2:172.21.15.163:6836/154041066 2024-07-26T06:50:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.806+0000 7f5b1eff5640 1 -- 172.21.15.42:0/1657281565 <== mds.2 v2:172.21.15.163:6836/154041066 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55f3e46dc310 con 0x55f3e4356c90 2024-07-26T06:50:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.806+0000 7f5b1eff5640 10 client.15444 handle_command_reply: tid=0 2024-07-26T06:50:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.806+0000 7f5b47793740 1 client.15444 shutdown 2024-07-26T06:50:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.806+0000 7f5b47793740 1 -- 172.21.15.42:0/1657281565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e439a0a0 msgr2=0x55f3e4796ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.806+0000 7f5b47793740 1 --2- 172.21.15.42:0/1657281565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e439a0a0 0x55f3e4796ca0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5b34005d70 tx=0x7f5b34011750 comp rx=0 tx=0).stop 2024-07-26T06:50:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.806+0000 7f5b47793740 1 -- 172.21.15.42:0/1657281565 shutdown_connections 2024-07-26T06:50:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.806+0000 7f5b47793740 1 --2- 172.21.15.42:0/1657281565 >> [v2:172.21.15.163:6836/154041066,v1:172.21.15.163:6837/154041066] conn(0x55f3e4356c90 0x55f3e47fa5a0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.806+0000 7f5b47793740 1 --2- 172.21.15.42:0/1657281565 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f3e43c2f60 0x55f3e4779470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.806+0000 7f5b47793740 1 --2- 172.21.15.42:0/1657281565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55f3e43a2280 0x55f3e4797220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.806+0000 7f5b47793740 1 --2- 172.21.15.42:0/1657281565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55f3e439a0a0 0x55f3e4796ca0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.806+0000 7f5b1eff5640 0 client.15444 ms_handle_reset on v2:172.21.15.163:6836/154041066 2024-07-26T06:50:57.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.806+0000 7f5b47793740 1 -- 172.21.15.42:0/1657281565 >> 172.21.15.42:0/1657281565 conn(0x55f3e43b7350 msgr2=0x55f3e4777620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:57.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.806+0000 7f5b47793740 1 -- 172.21.15.42:0/1657281565 shutdown_connections 2024-07-26T06:50:57.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.806+0000 7f5b47793740 1 -- 172.21.15.42:0/1657281565 wait complete. 2024-07-26T06:50:57.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.807+0000 7f5b47793740 20 client.15444 trim_cache size 0 max 16384 2024-07-26T06:50:57.815 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:50:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_fsid": "8b3e2750-4b19-11ef-bcb6-c7b262605968", 2024-07-26T06:50:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "whoami": 2, 2024-07-26T06:50:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "id": 24593, 2024-07-26T06:50:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "want_state": "up:active", 2024-07-26T06:50:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "up:active", 2024-07-26T06:50:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "fs_name": "cephfs", 2024-07-26T06:50:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "rank_uptime": 217.73443780700001, 2024-07-26T06:50:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "mdsmap_epoch": 29, 2024-07-26T06:50:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch": 80, 2024-07-26T06:50:57.816 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch_barrier": 80, 2024-07-26T06:50:57.816 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 259.45679043000001 2024-07-26T06:50:57.816 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:50:57.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.807+0000 7f5b4120f640 1 -- 172.21.15.42:0/2381072797 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5b240b3c80 msgr2=0x7f5b240b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:57.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.807+0000 7f5b4120f640 1 --2- 172.21.15.42:0/2381072797 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5b240b3c80 0x7f5b240b6190 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c006010 tx=0x7f5b2c095370 comp rx=0 tx=0).stop 2024-07-26T06:50:57.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.807+0000 7f5b4120f640 1 -- 172.21.15.42:0/2381072797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b3c102490 msgr2=0x7f5b3c199ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:50:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.807+0000 7f5b4120f640 1 --2- 172.21.15.42:0/2381072797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b3c102490 0x7f5b3c199ec0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c005940 tx=0x7f5b2c0042f0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.807+0000 7f5b4120f640 1 -- 172.21.15.42:0/2381072797 shutdown_connections 2024-07-26T06:50:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.807+0000 7f5b4120f640 1 --2- 172.21.15.42:0/2381072797 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f5b240b3c80 0x7f5b240b6190 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.807+0000 7f5b4120f640 1 --2- 172.21.15.42:0/2381072797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b3c108720 0x7f5b3c19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.807+0000 7f5b4120f640 1 --2- 172.21.15.42:0/2381072797 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b3c107dd0 0x7f5b3c19a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.807+0000 7f5b4120f640 1 --2- 172.21.15.42:0/2381072797 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b3c102490 0x7f5b3c199ec0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:50:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.807+0000 7f5b4120f640 1 -- 172.21.15.42:0/2381072797 >> 172.21.15.42:0/2381072797 conn(0x7f5b3c079160 msgr2=0x7f5b3c100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:50:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.807+0000 7f5b4120f640 1 -- 172.21.15.42:0/2381072797 shutdown_connections 2024-07-26T06:50:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:50:57.808+0000 7f5b4120f640 1 -- 172.21.15.42:0/2381072797 wait complete. 2024-07-26T06:50:58.443 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell mds.i status 2024-07-26T06:50:58.717 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:50:58.840 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:50:58 smithi042 ceph-mon[32775]: pgmap v418: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-07-26T06:50:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:50:58 smithi164 ceph-mon[36970]: pgmap v418: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-07-26T06:50:59.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:50:58 smithi163 ceph-mon[36873]: pgmap v418: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-07-26T06:51:00.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.431+0000 7efed3d87640 1 -- 172.21.15.42:0/3052382412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efecc109820 msgr2=0x7efecc110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:00.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.431+0000 7efed3d87640 1 --2- 172.21.15.42:0/3052382412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efecc109820 0x7efecc110050 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7efec4009ca0 tx=0x7efec402a8c0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.431+0000 7efed3d87640 1 -- 172.21.15.42:0/3052382412 shutdown_connections 2024-07-26T06:51:00.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.431+0000 7efed3d87640 1 --2- 172.21.15.42:0/3052382412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efecc109820 0x7efecc110050 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.431+0000 7efed3d87640 1 --2- 172.21.15.42:0/3052382412 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efecc108ed0 0x7efecc1092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.431+0000 7efed3d87640 1 --2- 172.21.15.42:0/3052382412 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efecc103590 0x7efecc1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.431+0000 7efed3d87640 1 -- 172.21.15.42:0/3052382412 >> 172.21.15.42:0/3052382412 conn(0x7efecc07b280 msgr2=0x7efecc07b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:00.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.431+0000 7efed3d87640 1 -- 172.21.15.42:0/3052382412 shutdown_connections 2024-07-26T06:51:00.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.431+0000 7efed3d87640 1 -- 172.21.15.42:0/3052382412 wait complete. 2024-07-26T06:51:00.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.432+0000 7efed3d87640 1 Processor -- start 2024-07-26T06:51:00.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.432+0000 7efed3d87640 1 -- start start 2024-07-26T06:51:00.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.432+0000 7efed3d87640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efecc103590 0x7efecc075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.432+0000 7efed3d87640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efecc108ed0 0x7efecc075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.432+0000 7efed3d87640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efecc109820 0x7efecc075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.432+0000 7efed3d87640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7efecc1134d0 con 0x7efecc109820 2024-07-26T06:51:00.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.432+0000 7efed3d87640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efecc113350 con 0x7efecc108ed0 2024-07-26T06:51:00.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.432+0000 7efed3d87640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7efecc113650 con 0x7efecc103590 2024-07-26T06:51:00.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.432+0000 7efed2324640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efecc109820 0x7efecc075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.432+0000 7efed2324640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efecc109820 0x7efecc075bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56024/0 (socket says 172.21.15.42:56024) 2024-07-26T06:51:00.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.432+0000 7efed2324640 1 -- 172.21.15.42:0/699348320 learned_addr learned my addr 172.21.15.42:0/699348320 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:00.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.432+0000 7efed1322640 1 --2- 172.21.15.42:0/699348320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efecc108ed0 0x7efecc075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.433+0000 7efed1b23640 1 --2- 172.21.15.42:0/699348320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efecc103590 0x7efecc075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.433+0000 7efed2324640 1 -- 172.21.15.42:0/699348320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efecc103590 msgr2=0x7efecc075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:00.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.433+0000 7efed2324640 1 --2- 172.21.15.42:0/699348320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efecc103590 0x7efecc075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.433+0000 7efed2324640 1 -- 172.21.15.42:0/699348320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efecc108ed0 msgr2=0x7efecc075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:00.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.433+0000 7efed2324640 1 --2- 172.21.15.42:0/699348320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efecc108ed0 0x7efecc075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.433+0000 7efed2324640 1 -- 172.21.15.42:0/699348320 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efecc1a8160 con 0x7efecc109820 2024-07-26T06:51:00.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.433+0000 7efed1b23640 1 --2- 172.21.15.42:0/699348320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efecc103590 0x7efecc075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:51:00.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.433+0000 7efed2324640 1 --2- 172.21.15.42:0/699348320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efecc109820 0x7efecc075bc0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7efec4005cc0 tx=0x7efec402f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:00.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.433+0000 7efec2ffd640 1 -- 172.21.15.42:0/699348320 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec4003bf0 con 0x7efecc109820 2024-07-26T06:51:00.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.433+0000 7efed3d87640 1 -- 172.21.15.42:0/699348320 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efecc1a83f0 con 0x7efecc109820 2024-07-26T06:51:00.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.433+0000 7efec2ffd640 1 -- 172.21.15.42:0/699348320 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efec4003d90 con 0x7efecc109820 2024-07-26T06:51:00.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.433+0000 7efec2ffd640 1 -- 172.21.15.42:0/699348320 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec402fbc0 con 0x7efecc109820 2024-07-26T06:51:00.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.433+0000 7efed3d87640 1 -- 172.21.15.42:0/699348320 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efecc1a8930 con 0x7efecc109820 2024-07-26T06:51:00.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.436+0000 7efeda30b740 1 Processor -- start 2024-07-26T06:51:00.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.436+0000 7efec2ffd640 1 -- 172.21.15.42:0/699348320 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7efec402fd60 con 0x7efecc109820 2024-07-26T06:51:00.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.436+0000 7efeda30b740 1 -- start start 2024-07-26T06:51:00.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efeda30b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565415b0f60 0x5565419563a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efeda30b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5565415880a0 0x5565419568e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efeda30b740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556541590280 0x556541956e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efeda30b740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5565418cc940 con 0x556541590280 2024-07-26T06:51:00.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efeda30b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5565418bbe40 con 0x5565415b0f60 2024-07-26T06:51:00.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efeda30b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5565418bfe80 con 0x5565415880a0 2024-07-26T06:51:00.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efed1322640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556541590280 0x556541956e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efed1322640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556541590280 0x556541956e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56036/0 (socket says 172.21.15.42:56036) 2024-07-26T06:51:00.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efed1322640 1 -- 172.21.15.42:0/1816047147 learned_addr learned my addr 172.21.15.42:0/1816047147 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:00.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efed1b23640 1 --2- 172.21.15.42:0/1816047147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5565415880a0 0x5565419568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efed1322640 1 --2- 172.21.15.42:0/1816047147 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565415b0f60 0x5565419563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efed1322640 1 -- 172.21.15.42:0/1816047147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5565415880a0 msgr2=0x5565419568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:00.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efed1322640 1 --2- 172.21.15.42:0/1816047147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5565415880a0 0x5565419568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efec2ffd640 1 --2- 172.21.15.42:0/699348320 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7efeb80b3bd0 0x7efeb80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efed1322640 1 -- 172.21.15.42:0/1816047147 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565415b0f60 msgr2=0x5565419563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:00.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efed1322640 1 --2- 172.21.15.42:0/1816047147 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565415b0f60 0x5565419563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efed1322640 1 -- 172.21.15.42:0/1816047147 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556541862a60 con 0x556541590280 2024-07-26T06:51:00.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.437+0000 7efed2324640 1 --2- 172.21.15.42:0/699348320 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7efeb80b3bd0 0x7efeb80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.438+0000 7efed1322640 1 --2- 172.21.15.42:0/1816047147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556541590280 0x556541956e20 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7efebc012530 tx=0x7efebc00b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab620916b514234 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:00.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.438+0000 7efec2ffd640 1 -- 172.21.15.42:0/699348320 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7efec4105440 con 0x7efecc109820 2024-07-26T06:51:00.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.438+0000 7efeb7fff640 1 -- 172.21.15.42:0/1816047147 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc01d5f0 con 0x556541590280 2024-07-26T06:51:00.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.438+0000 7efeb7fff640 1 -- 172.21.15.42:0/1816047147 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efebc0108e0 con 0x556541590280 2024-07-26T06:51:00.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.438+0000 7efeb7fff640 1 -- 172.21.15.42:0/1816047147 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc01c790 con 0x556541590280 2024-07-26T06:51:00.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.439+0000 7efeda30b740 1 -- 172.21.15.42:0/1816047147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556541590280 msgr2=0x556541956e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:00.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.439+0000 7efeda30b740 1 --2- 172.21.15.42:0/1816047147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556541590280 0x556541956e20 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7efebc012530 tx=0x7efebc00b9b0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.439+0000 7efed2324640 1 --2- 172.21.15.42:0/699348320 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7efeb80b3bd0 0x7efeb80b60e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7efec40cc070 tx=0x7efec4098870 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:00.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.439+0000 7efeda30b740 1 -- 172.21.15.42:0/1816047147 shutdown_connections 2024-07-26T06:51:00.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.439+0000 7efeda30b740 1 --2- 172.21.15.42:0/1816047147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556541590280 0x556541956e20 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.439+0000 7efeda30b740 1 --2- 172.21.15.42:0/1816047147 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5565415880a0 0x5565419568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.439+0000 7efeda30b740 1 --2- 172.21.15.42:0/1816047147 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565415b0f60 0x5565419563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.439+0000 7efeda30b740 1 -- 172.21.15.42:0/1816047147 >> 172.21.15.42:0/1816047147 conn(0x5565415a5350 msgr2=0x556541954290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:00.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.439+0000 7efeda30b740 1 -- 172.21.15.42:0/1816047147 shutdown_connections 2024-07-26T06:51:00.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.439+0000 7efeda30b740 1 -- 172.21.15.42:0/1816047147 wait complete. 2024-07-26T06:51:00.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.439+0000 7efeda30b740 1 -- start start 2024-07-26T06:51:00.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.440+0000 7efeda30b740 1 Processor -- start 2024-07-26T06:51:00.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.440+0000 7efeda30b740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5565415880a0 0x556541961f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.440+0000 7efeda30b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556541590280 0x556541962490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.440+0000 7efeda30b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5565415b0f60 0x55654196de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.440+0000 7efeda30b740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55654171f5f0 con 0x5565415880a0 2024-07-26T06:51:00.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.440+0000 7efeda30b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55654185e8b0 con 0x556541590280 2024-07-26T06:51:00.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.440+0000 7efeda30b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x556541964b80 con 0x5565415b0f60 2024-07-26T06:51:00.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.440+0000 7efeda30b740 10 client.0 _ll_register_callbacks cb 0x5565418687f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:00.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.440+0000 7efed1322640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5565415880a0 0x556541961f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.440+0000 7efed1322640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5565415880a0 0x556541961f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56050/0 (socket says 172.21.15.42:56050) 2024-07-26T06:51:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.440+0000 7efed1322640 1 -- 172.21.15.42:0/207605712 learned_addr learned my addr 172.21.15.42:0/207605712 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.440+0000 7efed1b23640 1 --2- 172.21.15.42:0/207605712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556541590280 0x556541962490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.440+0000 7efed1322640 1 --2- 172.21.15.42:0/207605712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5565415b0f60 0x55654196de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.441+0000 7efed1322640 1 -- 172.21.15.42:0/207605712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5565415b0f60 msgr2=0x55654196de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.441+0000 7efed1322640 1 --2- 172.21.15.42:0/207605712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5565415b0f60 0x55654196de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.441+0000 7efed1322640 1 -- 172.21.15.42:0/207605712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556541590280 msgr2=0x556541962490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.441+0000 7efed1322640 1 --2- 172.21.15.42:0/207605712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556541590280 0x556541962490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.441+0000 7efed1322640 1 -- 172.21.15.42:0/207605712 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55654161c540 con 0x5565415880a0 2024-07-26T06:51:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.441+0000 7efed1322640 1 --2- 172.21.15.42:0/207605712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5565415880a0 0x556541961f50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7efebc005890 tx=0x7efebc01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af58c0ec9a638aea server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.441+0000 7efeb57fa640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:51:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.441+0000 7efeb57fa640 1 -- 172.21.15.42:0/207605712 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc030060 con 0x5565415880a0 2024-07-26T06:51:00.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.441+0000 7efeb57fa640 1 -- 172.21.15.42:0/207605712 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7efebc010ca0 con 0x5565415880a0 2024-07-26T06:51:00.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.441+0000 7efeda30b740 1 -- 172.21.15.42:0/207605712 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efebc0108e0 con 0x5565415880a0 2024-07-26T06:51:00.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.441+0000 7efeb57fa640 1 -- 172.21.15.42:0/207605712 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efebc005020 con 0x5565415880a0 2024-07-26T06:51:00.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.441+0000 7efeda30b740 10 client.15456 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:00.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.441+0000 7efeda30b740 1 -- 172.21.15.42:0/207605712 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556541970b90 con 0x5565415880a0 2024-07-26T06:51:00.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.441+0000 7efeb57fa640 1 -- 172.21.15.42:0/207605712 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efebc0108e0 con 0x5565415880a0 2024-07-26T06:51:00.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.442+0000 7efeb57fa640 1 -- 172.21.15.42:0/207605712 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7efebc028030 con 0x5565415880a0 2024-07-26T06:51:00.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.442+0000 7efeda30b740 10 client.15456 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:00.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.442+0000 7efeda30b740 10 client.15456 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T06:51:00.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.442+0000 7efeda30b740 20 client.15456 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:00.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.442+0000 7efeda30b740 1 --2- 172.21.15.42:0/207605712 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x55654156f8e0 0x5565419743b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.442+0000 7efeda30b740 4 client.15456 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:51:00.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.442+0000 7efeda30b740 1 -- 172.21.15.42:0/207605712 --> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565418ce890 con 0x55654156f8e0 2024-07-26T06:51:00.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.442+0000 7efed1b23640 1 --2- 172.21.15.42:0/207605712 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x55654156f8e0 0x5565419743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.442+0000 7efeb57fa640 10 client.15456 ms_handle_connect on v2:172.21.15.164:6838/3593178314 2024-07-26T06:51:00.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.442+0000 7efed1b23640 1 --2- 172.21.15.42:0/207605712 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x55654156f8e0 0x5565419743b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f8fb27b47deea934 server_cookie=5f3064f947ac93c in_seq=0 out_seq=0 2024-07-26T06:51:00.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.443+0000 7efeb57fa640 1 -- 172.21.15.42:0/207605712 <== mds.1 v2:172.21.15.164:6838/3593178314 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5565418ce890 con 0x55654156f8e0 2024-07-26T06:51:00.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.443+0000 7efeb57fa640 10 client.15456 handle_command_reply: tid=0 2024-07-26T06:51:00.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.443+0000 7efeda30b740 1 client.15456 shutdown 2024-07-26T06:51:00.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.444+0000 7efeda30b740 1 -- 172.21.15.42:0/207605712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5565415880a0 msgr2=0x556541961f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:00.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.444+0000 7efeda30b740 1 --2- 172.21.15.42:0/207605712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5565415880a0 0x556541961f50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7efebc005890 tx=0x7efebc01c430 comp rx=0 tx=0).stop 2024-07-26T06:51:00.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.444+0000 7efeda30b740 1 -- 172.21.15.42:0/207605712 shutdown_connections 2024-07-26T06:51:00.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.444+0000 7efeda30b740 1 --2- 172.21.15.42:0/207605712 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x55654156f8e0 0x5565419743b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.444+0000 7efeda30b740 1 --2- 172.21.15.42:0/207605712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5565415b0f60 0x55654196de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.444+0000 7efeda30b740 1 --2- 172.21.15.42:0/207605712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556541590280 0x556541962490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.444+0000 7efeb57fa640 0 client.15456 ms_handle_reset on v2:172.21.15.164:6838/3593178314 2024-07-26T06:51:00.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.444+0000 7efeda30b740 1 --2- 172.21.15.42:0/207605712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5565415880a0 0x556541961f50 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.444+0000 7efeda30b740 1 -- 172.21.15.42:0/207605712 >> 172.21.15.42:0/207605712 conn(0x5565415a5350 msgr2=0x556541959580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:00.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.444+0000 7efeda30b740 1 -- 172.21.15.42:0/207605712 shutdown_connections 2024-07-26T06:51:00.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.444+0000 7efeda30b740 1 -- 172.21.15.42:0/207605712 wait complete. 2024-07-26T06:51:00.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.444+0000 7efeda30b740 20 client.15456 trim_cache size 0 max 16384 2024-07-26T06:51:00.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.463+0000 7efeda30b740 1 Processor -- start 2024-07-26T06:51:00.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.463+0000 7efeda30b740 1 -- start start 2024-07-26T06:51:00.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.463+0000 7efeda30b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55654156f8e0 0x556541957eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.463+0000 7efeda30b740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5565415880a0 0x55654196dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.463+0000 7efeda30b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556541590280 0x556541959bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.463+0000 7efeda30b740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5565419726f0 con 0x5565415880a0 2024-07-26T06:51:00.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.463+0000 7efeda30b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556541972dc0 con 0x55654156f8e0 2024-07-26T06:51:00.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.463+0000 7efeda30b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x556541973490 con 0x556541590280 2024-07-26T06:51:00.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.464+0000 7efed1b23640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5565415880a0 0x55654196dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.464+0000 7efed1b23640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5565415880a0 0x55654196dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56056/0 (socket says 172.21.15.42:56056) 2024-07-26T06:51:00.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.464+0000 7efed1b23640 1 -- 172.21.15.42:0/701829366 learned_addr learned my addr 172.21.15.42:0/701829366 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:00.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.464+0000 7efed1322640 1 --2- 172.21.15.42:0/701829366 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55654156f8e0 0x556541957eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.464+0000 7efed1322640 1 --2- 172.21.15.42:0/701829366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556541590280 0x556541959bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.464+0000 7efed1b23640 1 -- 172.21.15.42:0/701829366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556541590280 msgr2=0x556541959bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:00.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.464+0000 7efed1b23640 1 --2- 172.21.15.42:0/701829366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556541590280 0x556541959bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.464+0000 7efed1b23640 1 -- 172.21.15.42:0/701829366 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55654156f8e0 msgr2=0x556541957eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:00.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.464+0000 7efed1b23640 1 --2- 172.21.15.42:0/701829366 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55654156f8e0 0x556541957eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.464+0000 7efed1b23640 1 -- 172.21.15.42:0/701829366 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556541963010 con 0x5565415880a0 2024-07-26T06:51:00.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.464+0000 7efed1322640 1 --2- 172.21.15.42:0/701829366 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55654156f8e0 0x556541957eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:51:00.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.464+0000 7efed1b23640 1 --2- 172.21.15.42:0/701829366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5565415880a0 0x55654196dde0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7efec8004390 tx=0x7efec80039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20ebb356a0e7bf92 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:00.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.464+0000 7efeb77fe640 1 -- 172.21.15.42:0/701829366 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec801d620 con 0x5565415880a0 2024-07-26T06:51:00.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.464+0000 7efeb77fe640 1 -- 172.21.15.42:0/701829366 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efec801dc70 con 0x5565415880a0 2024-07-26T06:51:00.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.464+0000 7efeb77fe640 1 -- 172.21.15.42:0/701829366 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec801c3a0 con 0x5565415880a0 2024-07-26T06:51:00.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.465+0000 7efeda30b740 1 -- 172.21.15.42:0/701829366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5565415880a0 msgr2=0x55654196dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:00.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.465+0000 7efeda30b740 1 --2- 172.21.15.42:0/701829366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5565415880a0 0x55654196dde0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7efec8004390 tx=0x7efec80039c0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.465+0000 7efeda30b740 1 -- 172.21.15.42:0/701829366 shutdown_connections 2024-07-26T06:51:00.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.465+0000 7efeda30b740 1 --2- 172.21.15.42:0/701829366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556541590280 0x556541959bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.465+0000 7efeda30b740 1 --2- 172.21.15.42:0/701829366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5565415880a0 0x55654196dde0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.465+0000 7efeda30b740 1 --2- 172.21.15.42:0/701829366 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55654156f8e0 0x556541957eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.465+0000 7efeda30b740 1 -- 172.21.15.42:0/701829366 >> 172.21.15.42:0/701829366 conn(0x5565415a5350 msgr2=0x5565419594a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:00.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.465+0000 7efeda30b740 1 -- 172.21.15.42:0/701829366 shutdown_connections 2024-07-26T06:51:00.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.465+0000 7efeda30b740 1 -- 172.21.15.42:0/701829366 wait complete. 2024-07-26T06:51:00.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.466+0000 7efeda30b740 1 -- start start 2024-07-26T06:51:00.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.466+0000 7efeda30b740 1 Processor -- start 2024-07-26T06:51:00.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.466+0000 7efeda30b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55654156f8e0 0x556541975b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.466+0000 7efeda30b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565415880a0 0x556541976100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.466+0000 7efeda30b740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556541590280 0x55654197b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.466+0000 7efeda30b740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x556541974230 con 0x556541590280 2024-07-26T06:51:00.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.466+0000 7efeda30b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556541973b60 con 0x5565415880a0 2024-07-26T06:51:00.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.466+0000 7efeda30b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x556541974900 con 0x55654156f8e0 2024-07-26T06:51:00.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.466+0000 7efeda30b740 10 client.0 _ll_register_callbacks cb 0x7efebc0104c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:00.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.466+0000 7efed1322640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556541590280 0x55654197b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.466+0000 7efed1322640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556541590280 0x55654197b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56072/0 (socket says 172.21.15.42:56072) 2024-07-26T06:51:00.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.466+0000 7efed1322640 1 -- 172.21.15.42:0/3674677936 learned_addr learned my addr 172.21.15.42:0/3674677936 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:00.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.467+0000 7efed1b23640 1 --2- 172.21.15.42:0/3674677936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565415880a0 0x556541976100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.467+0000 7efed1322640 1 --2- 172.21.15.42:0/3674677936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55654156f8e0 0x556541975b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.467+0000 7efed1322640 1 -- 172.21.15.42:0/3674677936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55654156f8e0 msgr2=0x556541975b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:00.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.467+0000 7efed1322640 1 --2- 172.21.15.42:0/3674677936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55654156f8e0 0x556541975b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.467+0000 7efed1322640 1 -- 172.21.15.42:0/3674677936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565415880a0 msgr2=0x556541976100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:00.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.467+0000 7efed1322640 1 --2- 172.21.15.42:0/3674677936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565415880a0 0x556541976100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.467+0000 7efed1322640 1 -- 172.21.15.42:0/3674677936 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55654197b670 con 0x556541590280 2024-07-26T06:51:00.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.467+0000 7efed1322640 1 --2- 172.21.15.42:0/3674677936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556541590280 0x55654197b000 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7efebc00c010 tx=0x7efebc00bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbab56f08d07946c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:00.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.467+0000 7efeb57fa640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:51:00.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.467+0000 7efeb57fa640 1 -- 172.21.15.42:0/3674677936 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc03a040 con 0x556541590280 2024-07-26T06:51:00.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.467+0000 7efeda30b740 1 -- 172.21.15.42:0/3674677936 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efec801dc70 con 0x556541590280 2024-07-26T06:51:00.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.467+0000 7efeb57fa640 1 -- 172.21.15.42:0/3674677936 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7efebc00b780 con 0x556541590280 2024-07-26T06:51:00.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.467+0000 7efeb57fa640 1 -- 172.21.15.42:0/3674677936 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efebc030040 con 0x556541590280 2024-07-26T06:51:00.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.467+0000 7efeda30b740 10 client.15462 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:00.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.467+0000 7efeda30b740 1 -- 172.21.15.42:0/3674677936 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55654197ca30 con 0x556541590280 2024-07-26T06:51:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.468+0000 7efeb57fa640 1 -- 172.21.15.42:0/3674677936 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efec801dc70 con 0x556541590280 2024-07-26T06:51:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.468+0000 7efeb57fa640 1 -- 172.21.15.42:0/3674677936 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7efebc028040 con 0x556541590280 2024-07-26T06:51:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.468+0000 7efeda30b740 10 client.15462 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.468+0000 7efeda30b740 10 client.15462 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T06:51:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.468+0000 7efeda30b740 20 client.15462 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.468+0000 7efeda30b740 1 --2- 172.21.15.42:0/3674677936 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x5565415b0f60 0x5565419e0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.468+0000 7efeda30b740 4 client.15462 mds_command: new command op to 24601 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:51:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.468+0000 7efeda30b740 1 -- 172.21.15.42:0/3674677936 --> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] -- command(tid 0: {"prefix": "status"}) -- 0x5565418ca330 con 0x5565415b0f60 2024-07-26T06:51:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.468+0000 7efed1b23640 1 --2- 172.21.15.42:0/3674677936 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x5565415b0f60 0x5565419e0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.469+0000 7efed1b23640 1 --2- 172.21.15.42:0/3674677936 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x5565415b0f60 0x5565419e0320 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7c1ef1e4b6bbe2c2 server_cookie=daece80103370457 in_seq=0 out_seq=0 2024-07-26T06:51:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.469+0000 7efeb57fa640 10 client.15462 ms_handle_connect on v2:172.21.15.164:6838/3593178314 2024-07-26T06:51:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.469+0000 7efeb57fa640 1 -- 172.21.15.42:0/3674677936 <== mds.1 v2:172.21.15.164:6838/3593178314 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5565418ca330 con 0x5565415b0f60 2024-07-26T06:51:00.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.469+0000 7efeb57fa640 10 client.15462 handle_command_reply: tid=0 2024-07-26T06:51:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.469+0000 7efeda30b740 1 client.15462 shutdown 2024-07-26T06:51:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.469+0000 7efeda30b740 1 -- 172.21.15.42:0/3674677936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556541590280 msgr2=0x55654197b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.469+0000 7efeda30b740 1 --2- 172.21.15.42:0/3674677936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556541590280 0x55654197b000 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7efebc00c010 tx=0x7efebc00bdf0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.469+0000 7efeda30b740 1 -- 172.21.15.42:0/3674677936 shutdown_connections 2024-07-26T06:51:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.469+0000 7efeda30b740 1 --2- 172.21.15.42:0/3674677936 >> [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] conn(0x5565415b0f60 0x5565419e0320 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.469+0000 7efeda30b740 1 --2- 172.21.15.42:0/3674677936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556541590280 0x55654197b000 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.469+0000 7efeda30b740 1 --2- 172.21.15.42:0/3674677936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5565415880a0 0x556541976100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.469+0000 7efeb57fa640 0 client.15462 ms_handle_reset on v2:172.21.15.164:6838/3593178314 2024-07-26T06:51:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.469+0000 7efeda30b740 1 --2- 172.21.15.42:0/3674677936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55654156f8e0 0x556541975b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.469+0000 7efeda30b740 1 -- 172.21.15.42:0/3674677936 >> 172.21.15.42:0/3674677936 conn(0x5565415a5350 msgr2=0x5565419631e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.469+0000 7efeda30b740 1 -- 172.21.15.42:0/3674677936 shutdown_connections 2024-07-26T06:51:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.469+0000 7efeda30b740 1 -- 172.21.15.42:0/3674677936 wait complete. 2024-07-26T06:51:00.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.469+0000 7efeda30b740 20 client.15462 trim_cache size 0 max 16384 2024-07-26T06:51:00.477 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:51:00.477 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_fsid": "8b3e2750-4b19-11ef-bcb6-c7b262605968", 2024-07-26T06:51:00.477 INFO:teuthology.orchestra.run.smithi042.stdout: "whoami": 1, 2024-07-26T06:51:00.477 INFO:teuthology.orchestra.run.smithi042.stdout: "id": 24601, 2024-07-26T06:51:00.477 INFO:teuthology.orchestra.run.smithi042.stdout: "want_state": "up:active", 2024-07-26T06:51:00.477 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "up:active", 2024-07-26T06:51:00.477 INFO:teuthology.orchestra.run.smithi042.stdout: "fs_name": "cephfs", 2024-07-26T06:51:00.477 INFO:teuthology.orchestra.run.smithi042.stdout: "rank_uptime": 221.40560227700001, 2024-07-26T06:51:00.477 INFO:teuthology.orchestra.run.smithi042.stdout: "mdsmap_epoch": 29, 2024-07-26T06:51:00.477 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch": 80, 2024-07-26T06:51:00.478 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch_barrier": 80, 2024-07-26T06:51:00.478 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 259.49600352499999 2024-07-26T06:51:00.478 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:51:00.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.470+0000 7efed3d87640 1 -- 172.21.15.42:0/699348320 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7efeb80b3bd0 msgr2=0x7efeb80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:00.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.470+0000 7efed3d87640 1 --2- 172.21.15.42:0/699348320 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7efeb80b3bd0 0x7efeb80b60e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7efec40cc070 tx=0x7efec4098870 comp rx=0 tx=0).stop 2024-07-26T06:51:00.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.470+0000 7efed3d87640 1 -- 172.21.15.42:0/699348320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efecc109820 msgr2=0x7efecc075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:00.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.470+0000 7efed3d87640 1 --2- 172.21.15.42:0/699348320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efecc109820 0x7efecc075bc0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7efec4005cc0 tx=0x7efec402f380 comp rx=0 tx=0).stop 2024-07-26T06:51:00.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.470+0000 7efed3d87640 1 -- 172.21.15.42:0/699348320 shutdown_connections 2024-07-26T06:51:00.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.470+0000 7efed3d87640 1 --2- 172.21.15.42:0/699348320 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7efeb80b3bd0 0x7efeb80b60e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.470+0000 7efed3d87640 1 --2- 172.21.15.42:0/699348320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efecc109820 0x7efecc075bc0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.470+0000 7efed3d87640 1 --2- 172.21.15.42:0/699348320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efecc108ed0 0x7efecc075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.470+0000 7efed3d87640 1 --2- 172.21.15.42:0/699348320 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efecc103590 0x7efecc075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:00.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.470+0000 7efed3d87640 1 -- 172.21.15.42:0/699348320 >> 172.21.15.42:0/699348320 conn(0x7efecc07b280 msgr2=0x7efecc106f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:00.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.470+0000 7efed3d87640 1 -- 172.21.15.42:0/699348320 shutdown_connections 2024-07-26T06:51:00.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:00.470+0000 7efed3d87640 1 -- 172.21.15.42:0/699348320 wait complete. 2024-07-26T06:51:00.780 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:00 smithi042 ceph-mon[32775]: pgmap v419: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2024-07-26T06:51:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:00 smithi164 ceph-mon[36970]: pgmap v419: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2024-07-26T06:51:01.041 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:00 smithi163 ceph-mon[36873]: pgmap v419: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.2 KiB/s wr, 14 op/s 2024-07-26T06:51:01.246 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph tell mds.b status 2024-07-26T06:51:01.520 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:51:02.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:02 smithi042 ceph-mon[32775]: pgmap v420: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-07-26T06:51:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:02 smithi164 ceph-mon[36970]: pgmap v420: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-07-26T06:51:03.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:02 smithi163 ceph-mon[36873]: pgmap v420: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-07-26T06:51:03.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.339+0000 7fede1634640 1 -- 172.21.15.42:0/2075081214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feddc102190 msgr2=0x7feddc10ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:03.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.339+0000 7fede1634640 1 --2- 172.21.15.42:0/2075081214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feddc102190 0x7feddc10ffc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fedd4009ca0 tx=0x7fedd402a8c0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.339+0000 7fede1634640 1 -- 172.21.15.42:0/2075081214 shutdown_connections 2024-07-26T06:51:03.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.339+0000 7fede1634640 1 --2- 172.21.15.42:0/2075081214 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feddc102190 0x7feddc10ffc0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.339+0000 7fede1634640 1 --2- 172.21.15.42:0/2075081214 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feddc101840 0x7feddc101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.339+0000 7fede1634640 1 --2- 172.21.15.42:0/2075081214 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feddc108150 0x7feddc108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.339+0000 7fede1634640 1 -- 172.21.15.42:0/2075081214 >> 172.21.15.42:0/2075081214 conn(0x7feddc0776f0 msgr2=0x7feddc077b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:03.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.339+0000 7fede1634640 1 -- 172.21.15.42:0/2075081214 shutdown_connections 2024-07-26T06:51:03.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.339+0000 7fede1634640 1 -- 172.21.15.42:0/2075081214 wait complete. 2024-07-26T06:51:03.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.340+0000 7fede1634640 1 Processor -- start 2024-07-26T06:51:03.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.340+0000 7fede1634640 1 -- start start 2024-07-26T06:51:03.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.340+0000 7fede1634640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feddc101840 0x7feddc199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.340+0000 7fede1634640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feddc102190 0x7feddc19a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.340+0000 7fede1634640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feddc108150 0x7feddc19f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.341+0000 7fede1634640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7feddc1134f0 con 0x7feddc108150 2024-07-26T06:51:03.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.341+0000 7fede1634640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7feddc113370 con 0x7feddc102190 2024-07-26T06:51:03.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.341+0000 7fede1634640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7feddc113670 con 0x7feddc101840 2024-07-26T06:51:03.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.341+0000 7feddb7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feddc108150 0x7feddc19f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.341+0000 7feddb7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feddc108150 0x7feddc19f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56088/0 (socket says 172.21.15.42:56088) 2024-07-26T06:51:03.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.341+0000 7feddb7fe640 1 -- 172.21.15.42:0/3713595104 learned_addr learned my addr 172.21.15.42:0/3713595104 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:03.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.341+0000 7fedda7fc640 1 --2- 172.21.15.42:0/3713595104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feddc102190 0x7feddc19a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.341+0000 7feddaffd640 1 --2- 172.21.15.42:0/3713595104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feddc101840 0x7feddc199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.341+0000 7feddb7fe640 1 -- 172.21.15.42:0/3713595104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feddc101840 msgr2=0x7feddc199e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:03.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.341+0000 7feddb7fe640 1 --2- 172.21.15.42:0/3713595104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feddc101840 0x7feddc199e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.341+0000 7feddb7fe640 1 -- 172.21.15.42:0/3713595104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feddc102190 msgr2=0x7feddc19a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:03.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.341+0000 7feddb7fe640 1 --2- 172.21.15.42:0/3713595104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feddc102190 0x7feddc19a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.341+0000 7feddb7fe640 1 -- 172.21.15.42:0/3713595104 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feddc19fd90 con 0x7feddc108150 2024-07-26T06:51:03.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.341+0000 7feddb7fe640 1 --2- 172.21.15.42:0/3713595104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feddc108150 0x7feddc19f660 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fedd4005cc0 tx=0x7fedd402f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:03.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.341+0000 7fedc3fff640 1 -- 172.21.15.42:0/3713595104 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd4003bf0 con 0x7feddc108150 2024-07-26T06:51:03.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.341+0000 7fedc3fff640 1 -- 172.21.15.42:0/3713595104 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd4003d90 con 0x7feddc108150 2024-07-26T06:51:03.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.341+0000 7fede1634640 1 -- 172.21.15.42:0/3713595104 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feddc1a0020 con 0x7feddc108150 2024-07-26T06:51:03.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.342+0000 7fedc3fff640 1 -- 172.21.15.42:0/3713595104 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd402fbc0 con 0x7feddc108150 2024-07-26T06:51:03.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.342+0000 7fede1634640 1 -- 172.21.15.42:0/3713595104 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feddc1a67e0 con 0x7feddc108150 2024-07-26T06:51:03.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.345+0000 7fede7bb8740 1 Processor -- start 2024-07-26T06:51:03.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.345+0000 7fedc3fff640 1 -- 172.21.15.42:0/3713595104 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fedd402fd60 con 0x7feddc108150 2024-07-26T06:51:03.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.345+0000 7fede7bb8740 1 -- start start 2024-07-26T06:51:03.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7fede7bb8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c83200f60 0x556c835a6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7fede7bb8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c831d80a0 0x556c835a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7fede7bb8740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831e0280 0x556c835a6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7fede7bb8740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x556c8351c920 con 0x556c831e0280 2024-07-26T06:51:03.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7fede7bb8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556c8350be20 con 0x556c831d80a0 2024-07-26T06:51:03.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7fede7bb8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x556c8350fe60 con 0x556c83200f60 2024-07-26T06:51:03.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7fedda7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831e0280 0x556c835a6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7fedda7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831e0280 0x556c835a6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56098/0 (socket says 172.21.15.42:56098) 2024-07-26T06:51:03.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7fedda7fc640 1 -- 172.21.15.42:0/2572322065 learned_addr learned my addr 172.21.15.42:0/2572322065 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:03.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7fedda7fc640 1 --2- 172.21.15.42:0/2572322065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c83200f60 0x556c835a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7feddaffd640 1 --2- 172.21.15.42:0/2572322065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c831d80a0 0x556c835a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7fedc3fff640 1 --2- 172.21.15.42:0/3713595104 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fedc40b3bd0 0x7fedc40b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7fedda7fc640 1 -- 172.21.15.42:0/2572322065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c83200f60 msgr2=0x556c835a6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:03.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7fedda7fc640 1 --2- 172.21.15.42:0/2572322065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c83200f60 0x556c835a6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7feddb7fe640 1 --2- 172.21.15.42:0/3713595104 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fedc40b3bd0 0x7fedc40b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7fedda7fc640 1 -- 172.21.15.42:0/2572322065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c831d80a0 msgr2=0x556c835a68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:03.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7fedda7fc640 1 --2- 172.21.15.42:0/2572322065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c831d80a0 0x556c835a68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7fedda7fc640 1 -- 172.21.15.42:0/2572322065 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c834b2a40 con 0x556c831e0280 2024-07-26T06:51:03.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.346+0000 7fedc3fff640 1 -- 172.21.15.42:0/3713595104 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fedd41052c0 con 0x7feddc108150 2024-07-26T06:51:03.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.347+0000 7fedda7fc640 1 --2- 172.21.15.42:0/2572322065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831e0280 0x556c835a6e00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fedd0005d00 tx=0x7fedd0005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ffde2bf0a6da774 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:03.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.347+0000 7fedc17fa640 1 -- 172.21.15.42:0/2572322065 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd0012ca0 con 0x556c831e0280 2024-07-26T06:51:03.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.347+0000 7fedc17fa640 1 -- 172.21.15.42:0/2572322065 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd0012e40 con 0x556c831e0280 2024-07-26T06:51:03.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.347+0000 7fedc17fa640 1 -- 172.21.15.42:0/2572322065 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd0004a00 con 0x556c831e0280 2024-07-26T06:51:03.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.348+0000 7fede7bb8740 1 -- 172.21.15.42:0/2572322065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831e0280 msgr2=0x556c835a6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:03.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.348+0000 7feddb7fe640 1 --2- 172.21.15.42:0/3713595104 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fedc40b3bd0 0x7fedc40b60e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fedd40cbef0 tx=0x7fedd40c0ac0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:03.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.348+0000 7fede7bb8740 1 --2- 172.21.15.42:0/2572322065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831e0280 0x556c835a6e00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fedd0005d00 tx=0x7fedd0005c90 comp rx=0 tx=0).stop 2024-07-26T06:51:03.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.348+0000 7fede7bb8740 1 -- 172.21.15.42:0/2572322065 shutdown_connections 2024-07-26T06:51:03.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.348+0000 7fede7bb8740 1 --2- 172.21.15.42:0/2572322065 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831e0280 0x556c835a6e00 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.348+0000 7fede7bb8740 1 --2- 172.21.15.42:0/2572322065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c831d80a0 0x556c835a68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.348+0000 7fede7bb8740 1 --2- 172.21.15.42:0/2572322065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c83200f60 0x556c835a6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.348+0000 7fede7bb8740 1 -- 172.21.15.42:0/2572322065 >> 172.21.15.42:0/2572322065 conn(0x556c831f5350 msgr2=0x556c835a4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:03.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.348+0000 7fede7bb8740 1 -- 172.21.15.42:0/2572322065 shutdown_connections 2024-07-26T06:51:03.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.348+0000 7fede7bb8740 1 -- 172.21.15.42:0/2572322065 wait complete. 2024-07-26T06:51:03.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.349+0000 7fede7bb8740 1 -- start start 2024-07-26T06:51:03.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.349+0000 7fede7bb8740 1 Processor -- start 2024-07-26T06:51:03.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.349+0000 7fede7bb8740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831d80a0 0x556c835b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.349+0000 7fede7bb8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c831e0280 0x556c835b2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.349+0000 7fede7bb8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c83200f60 0x556c835bdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.349+0000 7fede7bb8740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x556c8336f5f0 con 0x556c831d80a0 2024-07-26T06:51:03.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.349+0000 7fede7bb8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556c834ae890 con 0x556c831e0280 2024-07-26T06:51:03.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.349+0000 7fede7bb8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x556c835b4b60 con 0x556c83200f60 2024-07-26T06:51:03.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.349+0000 7fede7bb8740 10 client.0 _ll_register_callbacks cb 0x556c834b87d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:03.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.349+0000 7fedda7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831d80a0 0x556c835b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.349+0000 7fedda7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831d80a0 0x556c835b1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56106/0 (socket says 172.21.15.42:56106) 2024-07-26T06:51:03.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.349+0000 7fedda7fc640 1 -- 172.21.15.42:0/3100916525 learned_addr learned my addr 172.21.15.42:0/3100916525 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:03.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.349+0000 7feddaffd640 1 --2- 172.21.15.42:0/3100916525 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c831e0280 0x556c835b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.349+0000 7fedda7fc640 1 --2- 172.21.15.42:0/3100916525 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c83200f60 0x556c835bdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.350+0000 7fedda7fc640 1 -- 172.21.15.42:0/3100916525 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c83200f60 msgr2=0x556c835bdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:03.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.350+0000 7fedda7fc640 1 --2- 172.21.15.42:0/3100916525 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c83200f60 0x556c835bdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.350+0000 7fedda7fc640 1 -- 172.21.15.42:0/3100916525 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c831e0280 msgr2=0x556c835b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.350+0000 7fedda7fc640 1 --2- 172.21.15.42:0/3100916525 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c831e0280 0x556c835b2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.350+0000 7fedda7fc640 1 -- 172.21.15.42:0/3100916525 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c8326c540 con 0x556c831d80a0 2024-07-26T06:51:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.350+0000 7fedda7fc640 1 --2- 172.21.15.42:0/3100916525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831d80a0 0x556c835b1ec0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fedd0005890 tx=0x7fedd000bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b9c8f4898952aac server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.350+0000 7fedbeff5640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:51:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.350+0000 7fedbeff5640 1 -- 172.21.15.42:0/3100916525 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd0025030 con 0x556c831d80a0 2024-07-26T06:51:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.350+0000 7fedbeff5640 1 -- 172.21.15.42:0/3100916525 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fedd0010980 con 0x556c831d80a0 2024-07-26T06:51:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.350+0000 7fede7bb8740 1 -- 172.21.15.42:0/3100916525 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556c8351e870 con 0x556c831d80a0 2024-07-26T06:51:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.350+0000 7fedbeff5640 1 -- 172.21.15.42:0/3100916525 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fedd0030070 con 0x556c831d80a0 2024-07-26T06:51:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.350+0000 7fede7bb8740 10 client.15474 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.350+0000 7fede7bb8740 1 -- 172.21.15.42:0/3100916525 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556c835c0b50 con 0x556c831d80a0 2024-07-26T06:51:03.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.351+0000 7fedbeff5640 1 -- 172.21.15.42:0/3100916525 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x556c8351e870 con 0x556c831d80a0 2024-07-26T06:51:03.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.351+0000 7fedbeff5640 1 -- 172.21.15.42:0/3100916525 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fedd002c4d0 con 0x556c831d80a0 2024-07-26T06:51:03.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.351+0000 7fede7bb8740 10 client.15474 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:03.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.351+0000 7fede7bb8740 10 client.15474 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T06:51:03.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.351+0000 7fede7bb8740 20 client.15474 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:03.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.351+0000 7fede7bb8740 1 --2- 172.21.15.42:0/3100916525 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x556c831bf8e0 0x556c835c4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.351+0000 7fede7bb8740 4 client.15474 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:51:03.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.351+0000 7fede7bb8740 1 -- 172.21.15.42:0/3100916525 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c8351a310 con 0x556c831bf8e0 2024-07-26T06:51:03.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.351+0000 7feddaffd640 1 --2- 172.21.15.42:0/3100916525 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x556c831bf8e0 0x556c835c4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.352+0000 7feddaffd640 1 --2- 172.21.15.42:0/3100916525 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x556c831bf8e0 0x556c835c4280 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4d39d044e881b99 server_cookie=2c5b55b28f41cde5 in_seq=0 out_seq=0 2024-07-26T06:51:03.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.352+0000 7fedbeff5640 10 client.15474 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:03.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.352+0000 7fedbeff5640 1 -- 172.21.15.42:0/3100916525 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556c8351a310 con 0x556c831bf8e0 2024-07-26T06:51:03.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.352+0000 7fedbeff5640 10 client.15474 handle_command_reply: tid=0 2024-07-26T06:51:03.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.352+0000 7fede7bb8740 1 client.15474 shutdown 2024-07-26T06:51:03.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.353+0000 7fede7bb8740 1 -- 172.21.15.42:0/3100916525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831d80a0 msgr2=0x556c835b1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:03.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.353+0000 7fede7bb8740 1 --2- 172.21.15.42:0/3100916525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831d80a0 0x556c835b1ec0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fedd0005890 tx=0x7fedd000bf50 comp rx=0 tx=0).stop 2024-07-26T06:51:03.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.353+0000 7fede7bb8740 1 -- 172.21.15.42:0/3100916525 shutdown_connections 2024-07-26T06:51:03.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.353+0000 7fede7bb8740 1 --2- 172.21.15.42:0/3100916525 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x556c831bf8e0 0x556c835c4280 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.353+0000 7fede7bb8740 1 --2- 172.21.15.42:0/3100916525 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c83200f60 0x556c835bdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.353+0000 7fede7bb8740 1 --2- 172.21.15.42:0/3100916525 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c831e0280 0x556c835b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.353+0000 7fedbeff5640 0 client.15474 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:03.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.353+0000 7fede7bb8740 1 --2- 172.21.15.42:0/3100916525 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831d80a0 0x556c835b1ec0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.353+0000 7fede7bb8740 1 -- 172.21.15.42:0/3100916525 >> 172.21.15.42:0/3100916525 conn(0x556c831f5350 msgr2=0x556c835a9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:03.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.353+0000 7fede7bb8740 1 -- 172.21.15.42:0/3100916525 shutdown_connections 2024-07-26T06:51:03.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.353+0000 7fede7bb8740 1 -- 172.21.15.42:0/3100916525 wait complete. 2024-07-26T06:51:03.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.353+0000 7fede7bb8740 20 client.15474 trim_cache size 0 max 16384 2024-07-26T06:51:03.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.371+0000 7fede7bb8740 1 Processor -- start 2024-07-26T06:51:03.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.372+0000 7fede7bb8740 1 -- start start 2024-07-26T06:51:03.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.372+0000 7fede7bb8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c831d80a0 0x556c835a31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.372+0000 7fede7bb8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c831e0280 0x556c835bde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.372+0000 7fede7bb8740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c83200f60 0x556c835be360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.372+0000 7fede7bb8740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x556c835c25c0 con 0x556c83200f60 2024-07-26T06:51:03.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.372+0000 7fede7bb8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556c835c2c90 con 0x556c831e0280 2024-07-26T06:51:03.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.372+0000 7fede7bb8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x556c835c3360 con 0x556c831d80a0 2024-07-26T06:51:03.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.372+0000 7fedda7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c83200f60 0x556c835be360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.372+0000 7fedda7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c83200f60 0x556c835be360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56118/0 (socket says 172.21.15.42:56118) 2024-07-26T06:51:03.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.372+0000 7fedda7fc640 1 -- 172.21.15.42:0/1895890164 learned_addr learned my addr 172.21.15.42:0/1895890164 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:03.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.372+0000 7feddaffd640 1 --2- 172.21.15.42:0/1895890164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c831e0280 0x556c835bde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.372+0000 7fedda7fc640 1 --2- 172.21.15.42:0/1895890164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c831d80a0 0x556c835a31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.372+0000 7fedda7fc640 1 -- 172.21.15.42:0/1895890164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c831d80a0 msgr2=0x556c835a31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:03.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.372+0000 7fedda7fc640 1 --2- 172.21.15.42:0/1895890164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c831d80a0 0x556c835a31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.372+0000 7fedda7fc640 1 -- 172.21.15.42:0/1895890164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c831e0280 msgr2=0x556c835bde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:03.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.372+0000 7fedda7fc640 1 --2- 172.21.15.42:0/1895890164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c831e0280 0x556c835bde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.372+0000 7fedda7fc640 1 -- 172.21.15.42:0/1895890164 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c835a8670 con 0x556c83200f60 2024-07-26T06:51:03.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.373+0000 7fedda7fc640 1 --2- 172.21.15.42:0/1895890164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c83200f60 0x556c835be360 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fedd0012720 tx=0x7fedd0014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff01366b55369548 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:03.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.373+0000 7fedc0ff9640 1 -- 172.21.15.42:0/1895890164 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd0025020 con 0x556c83200f60 2024-07-26T06:51:03.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.373+0000 7fedc0ff9640 1 -- 172.21.15.42:0/1895890164 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd0016050 con 0x556c83200f60 2024-07-26T06:51:03.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.373+0000 7fedc0ff9640 1 -- 172.21.15.42:0/1895890164 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd002c400 con 0x556c83200f60 2024-07-26T06:51:03.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.374+0000 7fede7bb8740 1 -- 172.21.15.42:0/1895890164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c83200f60 msgr2=0x556c835be360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:03.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.374+0000 7fede7bb8740 1 --2- 172.21.15.42:0/1895890164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c83200f60 0x556c835be360 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fedd0012720 tx=0x7fedd0014040 comp rx=0 tx=0).stop 2024-07-26T06:51:03.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.374+0000 7fede7bb8740 1 -- 172.21.15.42:0/1895890164 shutdown_connections 2024-07-26T06:51:03.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.374+0000 7fede7bb8740 1 --2- 172.21.15.42:0/1895890164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c83200f60 0x556c835be360 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.374+0000 7fede7bb8740 1 --2- 172.21.15.42:0/1895890164 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c831e0280 0x556c835bde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.374+0000 7fede7bb8740 1 --2- 172.21.15.42:0/1895890164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c831d80a0 0x556c835a31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.374+0000 7fede7bb8740 1 -- 172.21.15.42:0/1895890164 >> 172.21.15.42:0/1895890164 conn(0x556c831f5350 msgr2=0x556c835a40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.374+0000 7fede7bb8740 1 -- 172.21.15.42:0/1895890164 shutdown_connections 2024-07-26T06:51:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.374+0000 7fede7bb8740 1 -- 172.21.15.42:0/1895890164 wait complete. 2024-07-26T06:51:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.374+0000 7fede7bb8740 1 -- start start 2024-07-26T06:51:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.374+0000 7fede7bb8740 1 Processor -- start 2024-07-26T06:51:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7fede7bb8740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c831d80a0 0x556c835ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7fede7bb8740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831e0280 0x556c835ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7fede7bb8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c83200f60 0x556c835c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7fede7bb8740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x556c835c4100 con 0x556c831e0280 2024-07-26T06:51:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7fede7bb8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556c835c3a30 con 0x556c83200f60 2024-07-26T06:51:03.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7fede7bb8740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x556c835c47d0 con 0x556c831d80a0 2024-07-26T06:51:03.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7fede7bb8740 10 client.0 _ll_register_callbacks cb 0x7fedd0012650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:03.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7feddaffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831e0280 0x556c835ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7feddaffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831e0280 0x556c835ca9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56124/0 (socket says 172.21.15.42:56124) 2024-07-26T06:51:03.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7feddaffd640 1 -- 172.21.15.42:0/3612029812 learned_addr learned my addr 172.21.15.42:0/3612029812 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:03.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7fedda7fc640 1 --2- 172.21.15.42:0/3612029812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c83200f60 0x556c835c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7fedda7fc640 1 --2- 172.21.15.42:0/3612029812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c831d80a0 0x556c835ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7feddaffd640 1 -- 172.21.15.42:0/3612029812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c831d80a0 msgr2=0x556c835ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:03.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7feddaffd640 1 --2- 172.21.15.42:0/3612029812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c831d80a0 0x556c835ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7feddaffd640 1 -- 172.21.15.42:0/3612029812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c83200f60 msgr2=0x556c835c5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:03.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7feddaffd640 1 --2- 172.21.15.42:0/3612029812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c83200f60 0x556c835c5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7feddaffd640 1 -- 172.21.15.42:0/3612029812 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c835c61a0 con 0x556c831e0280 2024-07-26T06:51:03.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.375+0000 7fedda7fc640 1 --2- 172.21.15.42:0/3612029812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c83200f60 0x556c835c5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:51:03.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.376+0000 7feddaffd640 1 --2- 172.21.15.42:0/3612029812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831e0280 0x556c835ca9b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x556c835b4620 tx=0x7fedcc0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ecdddce40043109 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:03.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.376+0000 7fedbeff5640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:51:03.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.376+0000 7fedbeff5640 1 -- 172.21.15.42:0/3612029812 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc005220 con 0x556c831e0280 2024-07-26T06:51:03.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.376+0000 7fedbeff5640 1 -- 172.21.15.42:0/3612029812 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fedcc0053c0 con 0x556c831e0280 2024-07-26T06:51:03.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.376+0000 7fede7bb8740 1 -- 172.21.15.42:0/3612029812 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedd0016050 con 0x556c831e0280 2024-07-26T06:51:03.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.376+0000 7fedbeff5640 1 -- 172.21.15.42:0/3612029812 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fedcc00be90 con 0x556c831e0280 2024-07-26T06:51:03.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.376+0000 7fede7bb8740 10 client.15480 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:03.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.376+0000 7fede7bb8740 1 -- 172.21.15.42:0/3612029812 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556c835cbbb0 con 0x556c831e0280 2024-07-26T06:51:03.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.376+0000 7fedbeff5640 1 -- 172.21.15.42:0/3612029812 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd0016050 con 0x556c831e0280 2024-07-26T06:51:03.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.377+0000 7fedbeff5640 1 -- 172.21.15.42:0/3612029812 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fedcc0118a0 con 0x556c831e0280 2024-07-26T06:51:03.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.377+0000 7fede7bb8740 10 client.15480 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:03.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.377+0000 7fede7bb8740 10 client.15480 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T06:51:03.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.377+0000 7fede7bb8740 20 client.15480 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:03.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.377+0000 7fede7bb8740 1 --2- 172.21.15.42:0/3612029812 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x556c83194c90 0x556c83630730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:03.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.377+0000 7fede7bb8740 4 client.15480 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T06:51:03.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.377+0000 7fede7bb8740 1 -- 172.21.15.42:0/3612029812 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "status"}) -- 0x556c83517a50 con 0x556c83194c90 2024-07-26T06:51:03.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.377+0000 7fedda7fc640 1 --2- 172.21.15.42:0/3612029812 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x556c83194c90 0x556c83630730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:03.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.377+0000 7fedda7fc640 1 --2- 172.21.15.42:0/3612029812 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x556c83194c90 0x556c83630730 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=410e6b4d2f1537f server_cookie=6bd4e5e59d309bb1 in_seq=0 out_seq=0 2024-07-26T06:51:03.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.377+0000 7fedbeff5640 10 client.15480 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:03.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.378+0000 7fedbeff5640 1 -- 172.21.15.42:0/3612029812 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x556c83517a50 con 0x556c83194c90 2024-07-26T06:51:03.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.378+0000 7fedbeff5640 10 client.15480 handle_command_reply: tid=0 2024-07-26T06:51:03.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.378+0000 7fede7bb8740 1 client.15480 shutdown 2024-07-26T06:51:03.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.378+0000 7fede7bb8740 1 -- 172.21.15.42:0/3612029812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831e0280 msgr2=0x556c835ca9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:03.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.378+0000 7fede7bb8740 1 --2- 172.21.15.42:0/3612029812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831e0280 0x556c835ca9b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x556c835b4620 tx=0x7fedcc0039c0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.378+0000 7fede7bb8740 1 -- 172.21.15.42:0/3612029812 shutdown_connections 2024-07-26T06:51:03.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.378+0000 7fede7bb8740 1 --2- 172.21.15.42:0/3612029812 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x556c83194c90 0x556c83630730 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.378+0000 7fede7bb8740 1 --2- 172.21.15.42:0/3612029812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556c83200f60 0x556c835c5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.378+0000 7fede7bb8740 1 --2- 172.21.15.42:0/3612029812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x556c831e0280 0x556c835ca9b0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.378+0000 7fede7bb8740 1 --2- 172.21.15.42:0/3612029812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x556c831d80a0 0x556c835ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.378+0000 7fede7bb8740 1 -- 172.21.15.42:0/3612029812 >> 172.21.15.42:0/3612029812 conn(0x556c831f5350 msgr2=0x556c835a9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:03.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.378+0000 7fedbeff5640 0 client.15480 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:03.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.378+0000 7fede7bb8740 1 -- 172.21.15.42:0/3612029812 shutdown_connections 2024-07-26T06:51:03.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.378+0000 7fede7bb8740 1 -- 172.21.15.42:0/3612029812 wait complete. 2024-07-26T06:51:03.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.378+0000 7fede7bb8740 20 client.15480 trim_cache size 0 max 16384 2024-07-26T06:51:03.384 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:51:03.384 INFO:teuthology.orchestra.run.smithi042.stdout: "cluster_fsid": "8b3e2750-4b19-11ef-bcb6-c7b262605968", 2024-07-26T06:51:03.385 INFO:teuthology.orchestra.run.smithi042.stdout: "whoami": 0, 2024-07-26T06:51:03.385 INFO:teuthology.orchestra.run.smithi042.stdout: "id": 24620, 2024-07-26T06:51:03.385 INFO:teuthology.orchestra.run.smithi042.stdout: "want_state": "up:active", 2024-07-26T06:51:03.385 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "up:active", 2024-07-26T06:51:03.385 INFO:teuthology.orchestra.run.smithi042.stdout: "fs_name": "cephfs", 2024-07-26T06:51:03.385 INFO:teuthology.orchestra.run.smithi042.stdout: "rank_uptime": 236.36356934599999, 2024-07-26T06:51:03.385 INFO:teuthology.orchestra.run.smithi042.stdout: "mdsmap_epoch": 29, 2024-07-26T06:51:03.385 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch": 80, 2024-07-26T06:51:03.385 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap_epoch_barrier": 80, 2024-07-26T06:51:03.385 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 252.92119757899999 2024-07-26T06:51:03.385 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:51:03.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.379+0000 7fede1634640 1 -- 172.21.15.42:0/3713595104 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fedc40b3bd0 msgr2=0x7fedc40b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:03.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.379+0000 7fede1634640 1 --2- 172.21.15.42:0/3713595104 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fedc40b3bd0 0x7fedc40b60e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fedd40cbef0 tx=0x7fedd40c0ac0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.379+0000 7fede1634640 1 -- 172.21.15.42:0/3713595104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feddc108150 msgr2=0x7feddc19f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:03.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.379+0000 7fede1634640 1 --2- 172.21.15.42:0/3713595104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feddc108150 0x7feddc19f660 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fedd4005cc0 tx=0x7fedd402f380 comp rx=0 tx=0).stop 2024-07-26T06:51:03.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.379+0000 7fede1634640 1 -- 172.21.15.42:0/3713595104 shutdown_connections 2024-07-26T06:51:03.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.379+0000 7fede1634640 1 --2- 172.21.15.42:0/3713595104 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fedc40b3bd0 0x7fedc40b60e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.379+0000 7fede1634640 1 --2- 172.21.15.42:0/3713595104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feddc108150 0x7feddc19f660 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.379+0000 7fede1634640 1 --2- 172.21.15.42:0/3713595104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feddc102190 0x7feddc19a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.379+0000 7fede1634640 1 --2- 172.21.15.42:0/3713595104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feddc101840 0x7feddc199e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:03.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.379+0000 7fede1634640 1 -- 172.21.15.42:0/3713595104 >> 172.21.15.42:0/3713595104 conn(0x7feddc0776f0 msgr2=0x7feddc104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:03.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.379+0000 7fede1634640 1 -- 172.21.15.42:0/3713595104 shutdown_connections 2024-07-26T06:51:03.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:03.379+0000 7fede1634640 1 -- 172.21.15.42:0/3713595104 wait complete. 2024-07-26T06:51:04.140 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-07-26T06:51:04.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:04 smithi042 ceph-mon[32775]: pgmap v421: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 11 op/s 2024-07-26T06:51:04.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:04 smithi164 ceph-mon[36970]: pgmap v421: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 11 op/s 2024-07-26T06:51:05.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:04 smithi163 ceph-mon[36873]: pgmap v421: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 11 op/s 2024-07-26T06:51:06.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:06 smithi042 ceph-mon[32775]: pgmap v422: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 938 B/s wr, 13 op/s 2024-07-26T06:51:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:06 smithi164 ceph-mon[36970]: pgmap v422: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 938 B/s wr, 13 op/s 2024-07-26T06:51:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:06 smithi163 ceph-mon[36873]: pgmap v422: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 938 B/s wr, 13 op/s 2024-07-26T06:51:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.075+0000 7f6cc9594640 1 -- 172.21.15.42:0/1854559074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc4101fd0 msgr2=0x7f6cc4102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.075+0000 7f6cc9594640 1 --2- 172.21.15.42:0/1854559074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc4101fd0 0x7f6cc4102440 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4009900 tx=0x7f6cb402a410 comp rx=0 tx=0).stop 2024-07-26T06:51:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.075+0000 7f6cc9594640 1 -- 172.21.15.42:0/1854559074 shutdown_connections 2024-07-26T06:51:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.075+0000 7f6cc9594640 1 --2- 172.21.15.42:0/1854559074 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cc41039f0 0x7f6cc410a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.075+0000 7f6cc9594640 1 --2- 172.21.15.42:0/1854559074 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc4103050 0x7f6cc4103460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.075+0000 7f6cc9594640 1 --2- 172.21.15.42:0/1854559074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc4101fd0 0x7f6cc4102440 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.076+0000 7f6cc9594640 1 -- 172.21.15.42:0/1854559074 >> 172.21.15.42:0/1854559074 conn(0x7f6cc40fd910 msgr2=0x7f6cc40ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.076+0000 7f6cc9594640 1 -- 172.21.15.42:0/1854559074 shutdown_connections 2024-07-26T06:51:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.076+0000 7f6cc9594640 1 -- 172.21.15.42:0/1854559074 wait complete. 2024-07-26T06:51:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.076+0000 7f6cc9594640 1 Processor -- start 2024-07-26T06:51:07.080 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.076+0000 7f6cc9594640 1 -- start start 2024-07-26T06:51:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.077+0000 7f6cc9594640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc4101fd0 0x7f6cc4105bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.077+0000 7f6cc9594640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cc4103050 0x7f6cc4104480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.077+0000 7f6cc9594640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc41039f0 0x7f6cc41049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.077+0000 7f6cc9594640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc410dc50 con 0x7f6cc4101fd0 2024-07-26T06:51:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.077+0000 7f6cc9594640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc410dad0 con 0x7f6cc4103050 2024-07-26T06:51:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.077+0000 7f6cc9594640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc410ddd0 con 0x7f6cc41039f0 2024-07-26T06:51:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.077+0000 7f6cc2ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc4101fd0 0x7f6cc4105bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.077+0000 7f6cc2ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc4101fd0 0x7f6cc4105bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35102/0 (socket says 172.21.15.42:35102) 2024-07-26T06:51:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.077+0000 7f6cc2ffd640 1 -- 172.21.15.42:0/2278518726 learned_addr learned my addr 172.21.15.42:0/2278518726 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.077+0000 7f6cc27fc640 1 --2- 172.21.15.42:0/2278518726 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cc4103050 0x7f6cc4104480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.077+0000 7f6cc37fe640 1 --2- 172.21.15.42:0/2278518726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc41039f0 0x7f6cc41049c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.077+0000 7f6cc2ffd640 1 -- 172.21.15.42:0/2278518726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc41039f0 msgr2=0x7f6cc41049c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.077+0000 7f6cc2ffd640 1 --2- 172.21.15.42:0/2278518726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc41039f0 0x7f6cc41049c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.077+0000 7f6cc2ffd640 1 -- 172.21.15.42:0/2278518726 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cc4103050 msgr2=0x7f6cc4104480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.077+0000 7f6cc2ffd640 1 --2- 172.21.15.42:0/2278518726 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cc4103050 0x7f6cc4104480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.077+0000 7f6cc2ffd640 1 -- 172.21.15.42:0/2278518726 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc4105060 con 0x7f6cc4101fd0 2024-07-26T06:51:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.077+0000 7f6cc2ffd640 1 --2- 172.21.15.42:0/2278518726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc4101fd0 0x7f6cc4105bc0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4006010 tx=0x7f6cb4004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.078+0000 7f6cabfff640 1 -- 172.21.15.42:0/2278518726 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb403f030 con 0x7f6cc4101fd0 2024-07-26T06:51:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.078+0000 7f6cc9594640 1 -- 172.21.15.42:0/2278518726 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cc41a3a10 con 0x7f6cc4101fd0 2024-07-26T06:51:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.078+0000 7f6cabfff640 1 -- 172.21.15.42:0/2278518726 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cb4003cb0 con 0x7f6cc4101fd0 2024-07-26T06:51:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.078+0000 7f6cabfff640 1 -- 172.21.15.42:0/2278518726 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4035070 con 0x7f6cc4101fd0 2024-07-26T06:51:07.082 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.078+0000 7f6cc9594640 1 -- 172.21.15.42:0/2278518726 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cc41a3dc0 con 0x7f6cc4101fd0 2024-07-26T06:51:07.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.081+0000 7f6cd0582740 1 Processor -- start 2024-07-26T06:51:07.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.081+0000 7f6cd0582740 1 -- start start 2024-07-26T06:51:07.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.081+0000 7f6cd0582740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d391b0 0x56232a0de180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.081+0000 7f6cd0582740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329d102f0 0x56232a0de7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.081+0000 7f6cd0582740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d184d0 0x56232a0df340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.081+0000 7f6cd0582740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x56232a0513b0 con 0x562329d184d0 2024-07-26T06:51:07.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.081+0000 7f6cd0582740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56232a0408b0 con 0x562329d391b0 2024-07-26T06:51:07.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.081+0000 7f6cd0582740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x56232a0448f0 con 0x562329d102f0 2024-07-26T06:51:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.081+0000 7f6cabfff640 1 -- 172.21.15.42:0/2278518726 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f6cb4042030 con 0x7f6cc4101fd0 2024-07-26T06:51:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.081+0000 7f6cc27fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d184d0 0x56232a0df340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.081+0000 7f6cc27fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d184d0 0x56232a0df340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35112/0 (socket says 172.21.15.42:35112) 2024-07-26T06:51:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.081+0000 7f6cc27fc640 1 -- 172.21.15.42:0/856325455 learned_addr learned my addr 172.21.15.42:0/856325455 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.082+0000 7f6cc37fe640 1 --2- 172.21.15.42:0/856325455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d391b0 0x56232a0de180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.082+0000 7f6cc27fc640 1 -- 172.21.15.42:0/856325455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329d102f0 msgr2=0x56232a0de7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.082+0000 7f6cc2ffd640 1 --2- 172.21.15.42:0/856325455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329d102f0 0x56232a0de7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.082+0000 7f6cc27fc640 1 --2- 172.21.15.42:0/856325455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329d102f0 0x56232a0de7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.082+0000 7f6cc27fc640 1 -- 172.21.15.42:0/856325455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d391b0 msgr2=0x56232a0de180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.082+0000 7f6cc27fc640 1 --2- 172.21.15.42:0/856325455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d391b0 0x56232a0de180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.082+0000 7f6cc27fc640 1 -- 172.21.15.42:0/856325455 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562329fe79a0 con 0x562329d184d0 2024-07-26T06:51:07.090 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.082+0000 7f6cc2ffd640 1 --2- 172.21.15.42:0/856325455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329d102f0 0x56232a0de7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:51:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.082+0000 7f6cc27fc640 1 --2- 172.21.15.42:0/856325455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d184d0 0x56232a0df340 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800b760 tx=0x7f6cb800bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b64425c98f1d82b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.082+0000 7f6ca97fa640 1 -- 172.21.15.42:0/856325455 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8003dd0 con 0x562329d184d0 2024-07-26T06:51:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.082+0000 7f6ca97fa640 1 -- 172.21.15.42:0/856325455 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cb8015070 con 0x562329d184d0 2024-07-26T06:51:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.082+0000 7f6ca97fa640 1 -- 172.21.15.42:0/856325455 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8004d90 con 0x562329d184d0 2024-07-26T06:51:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.082+0000 7f6cabfff640 1 --2- 172.21.15.42:0/2278518726 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6cac0b3bd0 0x7f6cac0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.083+0000 7f6cc37fe640 1 --2- 172.21.15.42:0/2278518726 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6cac0b3bd0 0x7f6cac0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.083+0000 7f6cabfff640 1 -- 172.21.15.42:0/2278518726 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f6cb41042d0 con 0x7f6cc4101fd0 2024-07-26T06:51:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.083+0000 7f6cd0582740 1 -- 172.21.15.42:0/856325455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d184d0 msgr2=0x56232a0df340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.083+0000 7f6cc37fe640 1 --2- 172.21.15.42:0/2278518726 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6cac0b3bd0 0x7f6cac0b60e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc005fd0 tx=0x7f6cbc0076a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.083+0000 7f6cd0582740 1 --2- 172.21.15.42:0/856325455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d184d0 0x56232a0df340 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800b760 tx=0x7f6cb800bc20 comp rx=0 tx=0).stop 2024-07-26T06:51:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.083+0000 7f6cd0582740 1 -- 172.21.15.42:0/856325455 shutdown_connections 2024-07-26T06:51:07.091 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.083+0000 7f6cd0582740 1 --2- 172.21.15.42:0/856325455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d184d0 0x56232a0df340 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.083+0000 7f6cd0582740 1 --2- 172.21.15.42:0/856325455 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329d102f0 0x56232a0de7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.083+0000 7f6cd0582740 1 --2- 172.21.15.42:0/856325455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d391b0 0x56232a0de180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.083+0000 7f6cd0582740 1 -- 172.21.15.42:0/856325455 >> 172.21.15.42:0/856325455 conn(0x562329d2d300 msgr2=0x56232a0d9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.083+0000 7f6cd0582740 1 -- 172.21.15.42:0/856325455 shutdown_connections 2024-07-26T06:51:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.083+0000 7f6cd0582740 1 -- 172.21.15.42:0/856325455 wait complete. 2024-07-26T06:51:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.084+0000 7f6cd0582740 1 -- start start 2024-07-26T06:51:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.084+0000 7f6cd0582740 1 Processor -- start 2024-07-26T06:51:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.084+0000 7f6cd0582740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 0x56232a0e7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.084+0000 7f6cd0582740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329d184d0 0x56232a0e8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.084+0000 7f6cd0582740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d391b0 0x56232a0e8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.084+0000 7f6cd0582740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x562329f7aab0 con 0x562329d102f0 2024-07-26T06:51:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.084+0000 7f6cd0582740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562329fc8cf0 con 0x562329d391b0 2024-07-26T06:51:07.092 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.084+0000 7f6cd0582740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x56232a0dfe60 con 0x562329d184d0 2024-07-26T06:51:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.084+0000 7f6cd0582740 10 client.0 _ll_register_callbacks cb 0x56232a03a9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.084+0000 7f6cc2ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 0x56232a0e7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.084+0000 7f6cc2ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 0x56232a0e7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35128/0 (socket says 172.21.15.42:35128) 2024-07-26T06:51:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.084+0000 7f6cc2ffd640 1 -- 172.21.15.42:0/1756368433 learned_addr learned my addr 172.21.15.42:0/1756368433 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.084+0000 7f6cc27fc640 1 --2- 172.21.15.42:0/1756368433 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329d184d0 0x56232a0e8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.084+0000 7f6cc37fe640 1 --2- 172.21.15.42:0/1756368433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d391b0 0x56232a0e8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.085+0000 7f6cc2ffd640 1 -- 172.21.15.42:0/1756368433 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329d184d0 msgr2=0x56232a0e8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.085+0000 7f6cc2ffd640 1 --2- 172.21.15.42:0/1756368433 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329d184d0 0x56232a0e8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.085+0000 7f6cc2ffd640 1 -- 172.21.15.42:0/1756368433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d391b0 msgr2=0x56232a0e8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.085+0000 7f6cc2ffd640 1 --2- 172.21.15.42:0/1756368433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d391b0 0x56232a0e8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.085+0000 7f6cc2ffd640 1 -- 172.21.15.42:0/1756368433 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562329da4790 con 0x562329d102f0 2024-07-26T06:51:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.085+0000 7f6cc27fc640 1 --2- 172.21.15.42:0/1756368433 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329d184d0 0x56232a0e8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:51:07.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.085+0000 7f6cc37fe640 1 --2- 172.21.15.42:0/1756368433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d391b0 0x56232a0e8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:51:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.085+0000 7f6cc2ffd640 1 --2- 172.21.15.42:0/1756368433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 0x56232a0e7b40 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6cb40cb190 tx=0x7f6cb40ca970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3db57f87b92230 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.085+0000 7f6ca6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:51:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.085+0000 7f6ca6ff5640 1 -- 172.21.15.42:0/1756368433 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4049040 con 0x562329d102f0 2024-07-26T06:51:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.085+0000 7f6ca6ff5640 1 -- 172.21.15.42:0/1756368433 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f6cb403bc20 con 0x562329d102f0 2024-07-26T06:51:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.085+0000 7f6cd0582740 1 -- 172.21.15.42:0/1756368433 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cb8015070 con 0x562329d102f0 2024-07-26T06:51:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.085+0000 7f6ca6ff5640 1 -- 172.21.15.42:0/1756368433 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6cb4003e60 con 0x562329d102f0 2024-07-26T06:51:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.085+0000 7f6cd0582740 10 client.15492 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.085+0000 7f6cd0582740 1 -- 172.21.15.42:0/1756368433 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562329e1d620 con 0x562329d102f0 2024-07-26T06:51:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.085+0000 7f6ca6ff5640 1 -- 172.21.15.42:0/1756368433 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cb8015070 con 0x562329d102f0 2024-07-26T06:51:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.085+0000 7f6ca6ff5640 1 -- 172.21.15.42:0/1756368433 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6cb40b2040 con 0x562329d102f0 2024-07-26T06:51:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.086+0000 7f6cd0582740 10 client.15492 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:07.094 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.086+0000 7f6cd0582740 10 client.15492 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:51:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.086+0000 7f6cd0582740 20 client.15492 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.086+0000 7f6cd0582740 1 --2- 172.21.15.42:0/1756368433 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x562329cf7d80 0x56232a0f5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.086+0000 7f6cd0582740 4 client.15492 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:51:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.086+0000 7f6cd0582740 1 -- 172.21.15.42:0/1756368433 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56232a053300 con 0x562329cf7d80 2024-07-26T06:51:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.086+0000 7f6cc27fc640 1 --2- 172.21.15.42:0/1756368433 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x562329cf7d80 0x56232a0f5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.086+0000 7f6ca6ff5640 10 client.15492 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.086+0000 7f6cc27fc640 1 --2- 172.21.15.42:0/1756368433 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x562329cf7d80 0x56232a0f5240 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f18a102178540f6 server_cookie=85c355299f0d2f3f in_seq=0 out_seq=0 2024-07-26T06:51:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.087+0000 7f6ca6ff5640 1 -- 172.21.15.42:0/1756368433 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56232a053300 con 0x562329cf7d80 2024-07-26T06:51:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.087+0000 7f6ca6ff5640 10 client.15492 handle_command_reply: tid=0 2024-07-26T06:51:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.087+0000 7f6cd0582740 1 client.15492 shutdown 2024-07-26T06:51:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.087+0000 7f6cd0582740 1 -- 172.21.15.42:0/1756368433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 msgr2=0x56232a0e7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.087+0000 7f6cd0582740 1 --2- 172.21.15.42:0/1756368433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 0x56232a0e7b40 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6cb40cb190 tx=0x7f6cb40ca970 comp rx=0 tx=0).stop 2024-07-26T06:51:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.087+0000 7f6cd0582740 1 -- 172.21.15.42:0/1756368433 shutdown_connections 2024-07-26T06:51:07.096 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.087+0000 7f6cd0582740 1 --2- 172.21.15.42:0/1756368433 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x562329cf7d80 0x56232a0f5240 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.096 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.087+0000 7f6ca6ff5640 0 client.15492 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:07.096 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.087+0000 7f6cd0582740 1 --2- 172.21.15.42:0/1756368433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d391b0 0x56232a0e8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.096 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.087+0000 7f6cd0582740 1 --2- 172.21.15.42:0/1756368433 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329d184d0 0x56232a0e8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.096 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.087+0000 7f6cd0582740 1 --2- 172.21.15.42:0/1756368433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 0x56232a0e7b40 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.096 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.087+0000 7f6cd0582740 1 -- 172.21.15.42:0/1756368433 >> 172.21.15.42:0/1756368433 conn(0x562329d2d300 msgr2=0x56232a0d82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:07.096 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.087+0000 7f6cd0582740 1 -- 172.21.15.42:0/1756368433 shutdown_connections 2024-07-26T06:51:07.096 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.087+0000 7f6cd0582740 1 -- 172.21.15.42:0/1756368433 wait complete. 2024-07-26T06:51:07.096 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.088+0000 7f6cd0582740 20 client.15492 trim_cache size 0 max 16384 2024-07-26T06:51:07.104 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cd0582740 1 Processor -- start 2024-07-26T06:51:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cd0582740 1 -- start start 2024-07-26T06:51:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cd0582740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329cf7d80 0x56232a0df010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cd0582740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 0x56232a0e6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cd0582740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d184d0 0x56232a0e6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cd0582740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc41acb00 con 0x562329d102f0 2024-07-26T06:51:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cd0582740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc41ae640 con 0x562329d184d0 2024-07-26T06:51:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cd0582740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc41ad1d0 con 0x562329cf7d80 2024-07-26T06:51:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cc27fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 0x56232a0e6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cc27fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 0x56232a0e6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35140/0 (socket says 172.21.15.42:35140) 2024-07-26T06:51:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cc27fc640 1 -- 172.21.15.42:0/3051388237 learned_addr learned my addr 172.21.15.42:0/3051388237 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cc2ffd640 1 --2- 172.21.15.42:0/3051388237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329cf7d80 0x56232a0df010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cc27fc640 1 -- 172.21.15.42:0/3051388237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329cf7d80 msgr2=0x56232a0df010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cc27fc640 1 --2- 172.21.15.42:0/3051388237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329cf7d80 0x56232a0df010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cc27fc640 1 -- 172.21.15.42:0/3051388237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d184d0 msgr2=0x56232a0e6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cc37fe640 1 --2- 172.21.15.42:0/3051388237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d184d0 0x56232a0e6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cc27fc640 1 --2- 172.21.15.42:0/3051388237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d184d0 0x56232a0e6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.103+0000 7f6cc27fc640 1 -- 172.21.15.42:0/3051388237 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56232a0e5cd0 con 0x562329d102f0 2024-07-26T06:51:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.104+0000 7f6cc27fc640 1 --2- 172.21.15.42:0/3051388237 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 0x56232a0e6050 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8006010 tx=0x7f6cb800bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9556d0fb180b3e48 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.104+0000 7f6ca8ff9640 1 -- 172.21.15.42:0/3051388237 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8004340 con 0x562329d102f0 2024-07-26T06:51:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.104+0000 7f6ca8ff9640 1 -- 172.21.15.42:0/3051388237 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cb8015030 con 0x562329d102f0 2024-07-26T06:51:07.106 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.104+0000 7f6ca8ff9640 1 -- 172.21.15.42:0/3051388237 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8019680 con 0x562329d102f0 2024-07-26T06:51:07.110 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.105+0000 7f6cd0582740 1 -- 172.21.15.42:0/3051388237 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 msgr2=0x56232a0e6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:07.110 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.105+0000 7f6cd0582740 1 --2- 172.21.15.42:0/3051388237 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 0x56232a0e6050 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8006010 tx=0x7f6cb800bef0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.110 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.105+0000 7f6cd0582740 1 -- 172.21.15.42:0/3051388237 shutdown_connections 2024-07-26T06:51:07.110 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.105+0000 7f6cd0582740 1 --2- 172.21.15.42:0/3051388237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d184d0 0x56232a0e6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.110 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.105+0000 7f6cd0582740 1 --2- 172.21.15.42:0/3051388237 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 0x56232a0e6050 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.110 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.105+0000 7f6cd0582740 1 --2- 172.21.15.42:0/3051388237 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329cf7d80 0x56232a0df010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.110 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.105+0000 7f6cd0582740 1 -- 172.21.15.42:0/3051388237 >> 172.21.15.42:0/3051388237 conn(0x562329d2d300 msgr2=0x56232a0e2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:07.110 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.105+0000 7f6cd0582740 1 -- 172.21.15.42:0/3051388237 shutdown_connections 2024-07-26T06:51:07.110 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.105+0000 7f6cd0582740 1 -- 172.21.15.42:0/3051388237 wait complete. 2024-07-26T06:51:07.110 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.106+0000 7f6cd0582740 1 -- start start 2024-07-26T06:51:07.111 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.106+0000 7f6cd0582740 1 Processor -- start 2024-07-26T06:51:07.111 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.106+0000 7f6cd0582740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329cf7d80 0x56232a0ef8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.111 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.106+0000 7f6cd0582740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 0x56232a0ee160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.111 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.106+0000 7f6cd0582740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d184d0 0x56232a0ee7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.111 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.106+0000 7f6cd0582740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc41adf70 con 0x562329d102f0 2024-07-26T06:51:07.111 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.106+0000 7f6cd0582740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc41ad8a0 con 0x562329d184d0 2024-07-26T06:51:07.111 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.106+0000 7f6cd0582740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc41aaae0 con 0x562329cf7d80 2024-07-26T06:51:07.111 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.106+0000 7f6cd0582740 10 client.0 _ll_register_callbacks cb 0x7f6cb4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:07.111 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.106+0000 7f6cc2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329cf7d80 0x56232a0ef8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.111 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.106+0000 7f6cc27fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 0x56232a0ee160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.111 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.106+0000 7f6cc2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329cf7d80 0x56232a0ef8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:51284/0 (socket says 172.21.15.42:51284) 2024-07-26T06:51:07.111 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.106+0000 7f6cc2ffd640 1 -- 172.21.15.42:0/964883227 learned_addr learned my addr 172.21.15.42:0/964883227 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:07.111 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.106+0000 7f6cc37fe640 1 --2- 172.21.15.42:0/964883227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d184d0 0x56232a0ee7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.112 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.106+0000 7f6cc2ffd640 1 -- 172.21.15.42:0/964883227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d184d0 msgr2=0x56232a0ee7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:07.112 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.107+0000 7f6cc2ffd640 1 --2- 172.21.15.42:0/964883227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d184d0 0x56232a0ee7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.112 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.107+0000 7f6cc2ffd640 1 -- 172.21.15.42:0/964883227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 msgr2=0x56232a0ee160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:07.112 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.107+0000 7f6cc2ffd640 1 --2- 172.21.15.42:0/964883227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 0x56232a0ee160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.112 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.107+0000 7f6cc2ffd640 1 -- 172.21.15.42:0/964883227 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56232a0eef40 con 0x562329cf7d80 2024-07-26T06:51:07.112 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.107+0000 7f6cc27fc640 1 --2- 172.21.15.42:0/964883227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 0x56232a0ee160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:51:07.112 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.107+0000 7f6cc2ffd640 1 --2- 172.21.15.42:0/964883227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329cf7d80 0x56232a0ef8d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6cb403d040 tx=0x7f6cb40cf870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28f8ef52ba6d12ac server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:07.112 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.107+0000 7f6ca6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-07-26T06:51:07.112 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.107+0000 7f6ca6ff5640 1 -- 172.21.15.42:0/964883227 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4049040 con 0x562329cf7d80 2024-07-26T06:51:07.112 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.107+0000 7f6ca6ff5640 1 -- 172.21.15.42:0/964883227 <== mon.1 v2:172.21.15.164:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f6cb40d4030 con 0x562329cf7d80 2024-07-26T06:51:07.112 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.107+0000 7f6cd0582740 1 -- 172.21.15.42:0/964883227 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cb8015030 con 0x562329cf7d80 2024-07-26T06:51:07.112 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.108+0000 7f6ca6ff5640 1 -- 172.21.15.42:0/964883227 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6cb40cb900 con 0x562329cf7d80 2024-07-26T06:51:07.112 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.108+0000 7f6cd0582740 10 client.25189 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:07.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.108+0000 7f6cd0582740 1 -- 172.21.15.42:0/964883227 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56232a15cb00 con 0x562329cf7d80 2024-07-26T06:51:07.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.108+0000 7f6ca6ff5640 1 -- 172.21.15.42:0/964883227 <== mon.1 v2:172.21.15.164:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6cb8015030 con 0x562329cf7d80 2024-07-26T06:51:07.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.108+0000 7f6ca6ff5640 1 -- 172.21.15.42:0/964883227 <== mon.1 v2:172.21.15.164:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6cb40b2040 con 0x562329cf7d80 2024-07-26T06:51:07.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.108+0000 7f6cd0582740 10 client.25189 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:07.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.108+0000 7f6cd0582740 10 client.25189 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:51:07.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.108+0000 7f6cd0582740 20 client.25189 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:07.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.108+0000 7f6cd0582740 1 --2- 172.21.15.42:0/964883227 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x562329d391b0 0x56232a15f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:07.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.108+0000 7f6cd0582740 4 client.25189 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:51:07.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.108+0000 7f6cd0582740 1 -- 172.21.15.42:0/964883227 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56232a04eda0 con 0x562329d391b0 2024-07-26T06:51:07.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.109+0000 7f6cc27fc640 1 --2- 172.21.15.42:0/964883227 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x562329d391b0 0x56232a15f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:07.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.109+0000 7f6ca6ff5640 10 client.25189 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:07.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.109+0000 7f6cc27fc640 1 --2- 172.21.15.42:0/964883227 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x562329d391b0 0x56232a15f340 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7675fcdb9e812bb4 server_cookie=80ae0ebe43e121f in_seq=0 out_seq=0 2024-07-26T06:51:07.115 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.113+0000 7f6ca6ff5640 1 -- 172.21.15.42:0/964883227 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56232a04eda0 con 0x562329d391b0 2024-07-26T06:51:07.115 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.113+0000 7f6ca6ff5640 10 client.25189 handle_command_reply: tid=0 2024-07-26T06:51:07.115 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.113+0000 7f6cd0582740 1 client.25189 shutdown 2024-07-26T06:51:07.115 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.113+0000 7f6cd0582740 1 -- 172.21.15.42:0/964883227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329cf7d80 msgr2=0x56232a0ef8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:07.115 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.113+0000 7f6cd0582740 1 --2- 172.21.15.42:0/964883227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329cf7d80 0x56232a0ef8d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6cb403d040 tx=0x7f6cb40cf870 comp rx=0 tx=0).stop 2024-07-26T06:51:07.115 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.113+0000 7f6cd0582740 1 -- 172.21.15.42:0/964883227 shutdown_connections 2024-07-26T06:51:07.115 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.113+0000 7f6cd0582740 1 --2- 172.21.15.42:0/964883227 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x562329d391b0 0x56232a15f340 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.113+0000 7f6cd0582740 1 --2- 172.21.15.42:0/964883227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562329d184d0 0x56232a0ee7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.113+0000 7f6cd0582740 1 --2- 172.21.15.42:0/964883227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x562329d102f0 0x56232a0ee160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.113+0000 7f6ca6ff5640 0 client.25189 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.113+0000 7f6cd0582740 1 --2- 172.21.15.42:0/964883227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x562329cf7d80 0x56232a0ef8d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.113+0000 7f6cd0582740 1 -- 172.21.15.42:0/964883227 >> 172.21.15.42:0/964883227 conn(0x562329d2d300 msgr2=0x56232a0e5ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.113+0000 7f6cd0582740 1 -- 172.21.15.42:0/964883227 shutdown_connections 2024-07-26T06:51:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.113+0000 7f6cd0582740 1 -- 172.21.15.42:0/964883227 wait complete. 2024-07-26T06:51:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.113+0000 7f6cd0582740 20 client.25189 trim_cache size 0 max 16384 2024-07-26T06:51:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:51:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:51:07.116 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:51:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:51:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.114+0000 7f6cc9594640 1 -- 172.21.15.42:0/2278518726 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6cac0b3bd0 msgr2=0x7f6cac0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.114+0000 7f6cc9594640 1 --2- 172.21.15.42:0/2278518726 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6cac0b3bd0 0x7f6cac0b60e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc005fd0 tx=0x7f6cbc0076a0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.117 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.114+0000 7f6cc9594640 1 -- 172.21.15.42:0/2278518726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc4101fd0 msgr2=0x7f6cc4105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.114+0000 7f6cc9594640 1 --2- 172.21.15.42:0/2278518726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc4101fd0 0x7f6cc4105bc0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4006010 tx=0x7f6cb4004340 comp rx=0 tx=0).stop 2024-07-26T06:51:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.114+0000 7f6cc9594640 1 -- 172.21.15.42:0/2278518726 shutdown_connections 2024-07-26T06:51:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.114+0000 7f6cc9594640 1 --2- 172.21.15.42:0/2278518726 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6cac0b3bd0 0x7f6cac0b60e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.114+0000 7f6cc9594640 1 --2- 172.21.15.42:0/2278518726 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc41039f0 0x7f6cc41049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.114+0000 7f6cc9594640 1 --2- 172.21.15.42:0/2278518726 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6cc4103050 0x7f6cc4104480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.114+0000 7f6cc9594640 1 --2- 172.21.15.42:0/2278518726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6cc4101fd0 0x7f6cc4105bc0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.114+0000 7f6cc9594640 1 -- 172.21.15.42:0/2278518726 >> 172.21.15.42:0/2278518726 conn(0x7f6cc40fd910 msgr2=0x7f6cc40ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.114+0000 7f6cc9594640 1 -- 172.21.15.42:0/2278518726 shutdown_connections 2024-07-26T06:51:07.118 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:07.114+0000 7f6cc9594640 1 -- 172.21.15.42:0/2278518726 wait complete. 2024-07-26T06:51:08.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:08 smithi042 ceph-mon[32775]: pgmap v423: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T06:51:08.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:08 smithi164 ceph-mon[36970]: pgmap v423: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T06:51:09.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:08 smithi163 ceph-mon[36873]: pgmap v423: 97 pgs: 97 active+clean; 661 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T06:51:09.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:51:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:51:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:51:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:51:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:51:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:51:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:51:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:51:10.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:51:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:51:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:51:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:51:11.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:11 smithi042 ceph-mon[32775]: pgmap v424: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 682 B/s wr, 12 op/s 2024-07-26T06:51:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:11 smithi164 ceph-mon[36970]: pgmap v424: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 682 B/s wr, 12 op/s 2024-07-26T06:51:11.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:11 smithi163 ceph-mon[36873]: pgmap v424: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 682 B/s wr, 12 op/s 2024-07-26T06:51:13.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:13 smithi042 ceph-mon[32775]: pgmap v425: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T06:51:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:13 smithi164 ceph-mon[36970]: pgmap v425: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T06:51:13.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:13 smithi163 ceph-mon[36873]: pgmap v425: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T06:51:15.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:15 smithi042 ceph-mon[32775]: pgmap v426: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T06:51:15.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:15 smithi164 ceph-mon[36970]: pgmap v426: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T06:51:15.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:15 smithi163 ceph-mon[36873]: pgmap v426: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T06:51:17.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:17 smithi042 ceph-mon[32775]: pgmap v427: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T06:51:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:17 smithi164 ceph-mon[36970]: pgmap v427: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T06:51:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:17 smithi163 ceph-mon[36873]: pgmap v427: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T06:51:19.141 INFO:tasks.quiescer.fs.[cephfs]:Total duration (45.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-07-26T06:51:19.141 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.55 2024-07-26T06:51:19.142 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:51:19.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:19 smithi042 ceph-mon[32775]: pgmap v428: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T06:51:19.431 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:51:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:19 smithi164 ceph-mon[36970]: pgmap v428: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T06:51:19.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:19 smithi163 ceph-mon[36873]: pgmap v428: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T06:51:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.253+0000 7f0dfb9eb640 1 -- 172.21.15.42:0/1579004228 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0df4101580 msgr2=0x7f0df41039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.253+0000 7f0dfb9eb640 1 --2- 172.21.15.42:0/1579004228 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0df4101580 0x7f0df41039c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0df0009900 tx=0x7f0df002a410 comp rx=0 tx=0).stop 2024-07-26T06:51:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.253+0000 7f0dfb9eb640 1 -- 172.21.15.42:0/1579004228 shutdown_connections 2024-07-26T06:51:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.253+0000 7f0dfb9eb640 1 --2- 172.21.15.42:0/1579004228 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0df410b040 0x7f0df410d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.253+0000 7f0dfb9eb640 1 --2- 172.21.15.42:0/1579004228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0df4104000 0x7f0df410aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.253+0000 7f0dfb9eb640 1 --2- 172.21.15.42:0/1579004228 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0df4101580 0x7f0df41039c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.253+0000 7f0dfb9eb640 1 -- 172.21.15.42:0/1579004228 >> 172.21.15.42:0/1579004228 conn(0x7f0df40fd0c0 msgr2=0x7f0df40ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.254+0000 7f0dfb9eb640 1 -- 172.21.15.42:0/1579004228 shutdown_connections 2024-07-26T06:51:20.255 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.254+0000 7f0dfb9eb640 1 -- 172.21.15.42:0/1579004228 wait complete. 2024-07-26T06:51:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.254+0000 7f0dfb9eb640 1 Processor -- start 2024-07-26T06:51:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.254+0000 7f0dfb9eb640 1 -- start start 2024-07-26T06:51:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.255+0000 7f0dfb9eb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0df4101580 0x7f0df419bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.255+0000 7f0dfb9eb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0df4104000 0x7f0df419c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.255+0000 7f0dfb9eb640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0df410b040 0x7f0df419cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.255+0000 7f0dfb9eb640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0df4110f60 con 0x7f0df4104000 2024-07-26T06:51:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.255+0000 7f0dfb9eb640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0df4110de0 con 0x7f0df410b040 2024-07-26T06:51:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.255+0000 7f0dfb9eb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0df41110e0 con 0x7f0df4101580 2024-07-26T06:51:20.257 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.255+0000 7f0df8f86640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0df4104000 0x7f0df419c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:20.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.255+0000 7f0df8f86640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0df4104000 0x7f0df419c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48670/0 (socket says 172.21.15.42:48670) 2024-07-26T06:51:20.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.255+0000 7f0df8f86640 1 -- 172.21.15.42:0/1957705742 learned_addr learned my addr 172.21.15.42:0/1957705742 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:20.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.255+0000 7f0df9787640 1 --2- 172.21.15.42:0/1957705742 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0df4101580 0x7f0df419bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:20.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.255+0000 7f0df9f88640 1 --2- 172.21.15.42:0/1957705742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0df410b040 0x7f0df419cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:20.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.255+0000 7f0df8f86640 1 -- 172.21.15.42:0/1957705742 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0df4101580 msgr2=0x7f0df419bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:20.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.255+0000 7f0df8f86640 1 --2- 172.21.15.42:0/1957705742 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0df4101580 0x7f0df419bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.255+0000 7f0df8f86640 1 -- 172.21.15.42:0/1957705742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0df410b040 msgr2=0x7f0df419cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:20.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.255+0000 7f0df8f86640 1 --2- 172.21.15.42:0/1957705742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0df410b040 0x7f0df419cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.255+0000 7f0df8f86640 1 -- 172.21.15.42:0/1957705742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0df419d590 con 0x7f0df4104000 2024-07-26T06:51:20.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.255+0000 7f0df9f88640 1 --2- 172.21.15.42:0/1957705742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0df410b040 0x7f0df419cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:51:20.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.256+0000 7f0df8f86640 1 --2- 172.21.15.42:0/1957705742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0df4104000 0x7f0df419c1c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0de400d950 tx=0x7f0de400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:20.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.256+0000 7f0dea7fc640 1 -- 172.21.15.42:0/1957705742 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de4017030 con 0x7f0df4104000 2024-07-26T06:51:20.259 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.256+0000 7f0dea7fc640 1 -- 172.21.15.42:0/1957705742 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0de4004020 con 0x7f0df4104000 2024-07-26T06:51:20.259 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.256+0000 7f0dfb9eb640 1 -- 172.21.15.42:0/1957705742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0df419d880 con 0x7f0df4104000 2024-07-26T06:51:20.259 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.256+0000 7f0dea7fc640 1 -- 172.21.15.42:0/1957705742 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de40047a0 con 0x7f0df4104000 2024-07-26T06:51:20.259 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.256+0000 7f0dfb9eb640 1 -- 172.21.15.42:0/1957705742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0df41a3f70 con 0x7f0df4104000 2024-07-26T06:51:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.259+0000 7f0e029e4740 1 Processor -- start 2024-07-26T06:51:20.260 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.259+0000 7f0e029e4740 1 -- start start 2024-07-26T06:51:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.260+0000 7f0e029e4740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b5861b0 0x564a4b92ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.260+0000 7f0e029e4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b55d2f0 0x564a4b92b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.260+0000 7f0e029e4740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b5654d0 0x564a4b92bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.260+0000 7f0e029e4740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x564a4b89e3b0 con 0x564a4b5861b0 2024-07-26T06:51:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.260+0000 7f0e029e4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564a4b88d8b0 con 0x564a4b55d2f0 2024-07-26T06:51:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.260+0000 7f0e029e4740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x564a4b8918f0 con 0x564a4b5654d0 2024-07-26T06:51:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.260+0000 7f0df9f88640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b5861b0 0x564a4b92ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.260+0000 7f0df9f88640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b5861b0 0x564a4b92ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48676/0 (socket says 172.21.15.42:48676) 2024-07-26T06:51:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.260+0000 7f0df9f88640 1 -- 172.21.15.42:0/1572640835 learned_addr learned my addr 172.21.15.42:0/1572640835 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:20.261 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.260+0000 7f0df9787640 1 --2- 172.21.15.42:0/1572640835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b55d2f0 0x564a4b92b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.260+0000 7f0df9f88640 1 -- 172.21.15.42:0/1572640835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b5654d0 msgr2=0x564a4b92bd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T06:51:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.260+0000 7f0df9f88640 1 --2- 172.21.15.42:0/1572640835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b5654d0 0x564a4b92bd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.260+0000 7f0df9f88640 1 -- 172.21.15.42:0/1572640835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b55d2f0 msgr2=0x564a4b92b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.260+0000 7f0df9f88640 1 --2- 172.21.15.42:0/1572640835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b55d2f0 0x564a4b92b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.263 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.260+0000 7f0df9f88640 1 -- 172.21.15.42:0/1572640835 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a4b8349a0 con 0x564a4b5861b0 2024-07-26T06:51:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.261+0000 7f0dea7fc640 1 -- 172.21.15.42:0/1957705742 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f0de4009a30 con 0x7f0df4104000 2024-07-26T06:51:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.261+0000 7f0df9f88640 1 --2- 172.21.15.42:0/1572640835 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b5861b0 0x564a4b92ab50 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009f30 tx=0x7f0dec010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba2998b71969913 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.261+0000 7f0ddf7fe640 1 -- 172.21.15.42:0/1572640835 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec016070 con 0x564a4b5861b0 2024-07-26T06:51:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.261+0000 7f0ddf7fe640 1 -- 172.21.15.42:0/1572640835 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dec01bb10 con 0x564a4b5861b0 2024-07-26T06:51:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.261+0000 7f0ddf7fe640 1 -- 172.21.15.42:0/1572640835 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec012bc0 con 0x564a4b5861b0 2024-07-26T06:51:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.262+0000 7f0dea7fc640 1 --2- 172.21.15.42:0/1957705742 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0de00b3cd0 0x7f0de00b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.262+0000 7f0df9787640 1 --2- 172.21.15.42:0/1957705742 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0de00b3cd0 0x7f0de00b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.262+0000 7f0dea7fc640 1 -- 172.21.15.42:0/1957705742 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f0de40e3720 con 0x7f0df4104000 2024-07-26T06:51:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.262+0000 7f0e029e4740 1 -- 172.21.15.42:0/1572640835 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b5861b0 msgr2=0x564a4b92ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.262+0000 7f0e029e4740 1 --2- 172.21.15.42:0/1572640835 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b5861b0 0x564a4b92ab50 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009f30 tx=0x7f0dec010800 comp rx=0 tx=0).stop 2024-07-26T06:51:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.262+0000 7f0e029e4740 1 -- 172.21.15.42:0/1572640835 shutdown_connections 2024-07-26T06:51:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.262+0000 7f0e029e4740 1 --2- 172.21.15.42:0/1572640835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b5654d0 0x564a4b92bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.262+0000 7f0e029e4740 1 --2- 172.21.15.42:0/1572640835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b55d2f0 0x564a4b92b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.262+0000 7f0e029e4740 1 --2- 172.21.15.42:0/1572640835 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b5861b0 0x564a4b92ab50 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.262+0000 7f0e029e4740 1 -- 172.21.15.42:0/1572640835 >> 172.21.15.42:0/1572640835 conn(0x564a4b57a300 msgr2=0x564a4b9265e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.263+0000 7f0e029e4740 1 -- 172.21.15.42:0/1572640835 shutdown_connections 2024-07-26T06:51:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.263+0000 7f0df9787640 1 --2- 172.21.15.42:0/1957705742 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0de00b3cd0 0x7f0de00b61e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0df0000c00 tx=0x7f0df0031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.263+0000 7f0e029e4740 1 -- 172.21.15.42:0/1572640835 wait complete. 2024-07-26T06:51:20.265 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.263+0000 7f0e029e4740 1 -- start start 2024-07-26T06:51:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.263+0000 7f0e029e4740 1 Processor -- start 2024-07-26T06:51:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0e029e4740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b55d2f0 0x564a4b9345d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0e029e4740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b5654d0 0x564a4b934c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0e029e4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b5861b0 0x564a4b935800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0e029e4740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x564a4b7c7ab0 con 0x564a4b55d2f0 2024-07-26T06:51:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0e029e4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564a4b815cf0 con 0x564a4b5861b0 2024-07-26T06:51:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0e029e4740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x564a4b92c830 con 0x564a4b5654d0 2024-07-26T06:51:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0e029e4740 10 client.0 _ll_register_callbacks cb 0x564a4b8873e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:20.269 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0df9787640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b55d2f0 0x564a4b9345d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:20.270 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0df9787640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b55d2f0 0x564a4b9345d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48688/0 (socket says 172.21.15.42:48688) 2024-07-26T06:51:20.270 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0df9787640 1 -- 172.21.15.42:0/2826153858 learned_addr learned my addr 172.21.15.42:0/2826153858 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:20.270 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0df9f88640 1 --2- 172.21.15.42:0/2826153858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b5861b0 0x564a4b935800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:20.270 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0df9787640 1 -- 172.21.15.42:0/2826153858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b5654d0 msgr2=0x564a4b934c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:20.270 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0df9787640 1 --2- 172.21.15.42:0/2826153858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b5654d0 0x564a4b934c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.270 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0df9787640 1 -- 172.21.15.42:0/2826153858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b5861b0 msgr2=0x564a4b935800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:20.270 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0df9787640 1 --2- 172.21.15.42:0/2826153858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b5861b0 0x564a4b935800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.270 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0df9787640 1 -- 172.21.15.42:0/2826153858 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a4b5f1790 con 0x564a4b55d2f0 2024-07-26T06:51:20.270 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0df9787640 1 --2- 172.21.15.42:0/2826153858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b55d2f0 0x564a4b9345d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0df0009900 tx=0x7f0df0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc35946d128ecc5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:20.270 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.264+0000 7f0ddcff9640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:51:20.270 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.265+0000 7f0ddcff9640 1 -- 172.21.15.42:0/2826153858 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df0035040 con 0x564a4b55d2f0 2024-07-26T06:51:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.265+0000 7f0ddcff9640 1 -- 172.21.15.42:0/2826153858 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f0df00035f0 con 0x564a4b55d2f0 2024-07-26T06:51:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.265+0000 7f0e029e4740 1 -- 172.21.15.42:0/2826153858 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0dec01bb10 con 0x564a4b55d2f0 2024-07-26T06:51:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.265+0000 7f0ddcff9640 1 -- 172.21.15.42:0/2826153858 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0df00037f0 con 0x564a4b55d2f0 2024-07-26T06:51:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.265+0000 7f0e029e4740 10 client.15510 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.265+0000 7f0e029e4740 1 -- 172.21.15.42:0/2826153858 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564a4b66a620 con 0x564a4b55d2f0 2024-07-26T06:51:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.265+0000 7f0ddcff9640 1 -- 172.21.15.42:0/2826153858 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dec01bb10 con 0x564a4b55d2f0 2024-07-26T06:51:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.265+0000 7f0ddcff9640 1 -- 172.21.15.42:0/2826153858 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0df0044070 con 0x564a4b55d2f0 2024-07-26T06:51:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.266+0000 7f0e029e4740 10 client.15510 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.266+0000 7f0e029e4740 10 client.15510 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:51:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.266+0000 7f0e029e4740 20 client.15510 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.266+0000 7f0e029e4740 1 --2- 172.21.15.42:0/2826153858 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x564a4b544d80 0x564a4b941e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.271 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.266+0000 7f0e029e4740 4 client.15510 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:51:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.266+0000 7f0e029e4740 1 -- 172.21.15.42:0/2826153858 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a4b8a0300 con 0x564a4b544d80 2024-07-26T06:51:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.266+0000 7f0df8f86640 1 --2- 172.21.15.42:0/2826153858 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x564a4b544d80 0x564a4b941e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.266+0000 7f0df8f86640 1 --2- 172.21.15.42:0/2826153858 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x564a4b544d80 0x564a4b941e30 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a887f21b483286dd server_cookie=5f7ac190ea8a8f73 in_seq=0 out_seq=0 2024-07-26T06:51:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.266+0000 7f0ddcff9640 10 client.15510 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.267+0000 7f0ddcff9640 1 -- 172.21.15.42:0/2826153858 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564a4b8a0300 con 0x564a4b544d80 2024-07-26T06:51:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.267+0000 7f0ddcff9640 10 client.15510 handle_command_reply: tid=0 2024-07-26T06:51:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.267+0000 7f0e029e4740 1 client.15510 shutdown 2024-07-26T06:51:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.267+0000 7f0e029e4740 1 -- 172.21.15.42:0/2826153858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b55d2f0 msgr2=0x564a4b9345d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.267+0000 7f0e029e4740 1 --2- 172.21.15.42:0/2826153858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b55d2f0 0x564a4b9345d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0df0009900 tx=0x7f0df0009260 comp rx=0 tx=0).stop 2024-07-26T06:51:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.268+0000 7f0e029e4740 1 -- 172.21.15.42:0/2826153858 shutdown_connections 2024-07-26T06:51:20.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.268+0000 7f0e029e4740 1 --2- 172.21.15.42:0/2826153858 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x564a4b544d80 0x564a4b941e30 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.268+0000 7f0ddcff9640 0 client.15510 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.268+0000 7f0e029e4740 1 --2- 172.21.15.42:0/2826153858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b5861b0 0x564a4b935800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.268+0000 7f0e029e4740 1 --2- 172.21.15.42:0/2826153858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b5654d0 0x564a4b934c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.268+0000 7f0e029e4740 1 --2- 172.21.15.42:0/2826153858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b55d2f0 0x564a4b9345d0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.268+0000 7f0e029e4740 1 -- 172.21.15.42:0/2826153858 >> 172.21.15.42:0/2826153858 conn(0x564a4b57a300 msgr2=0x564a4b924c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.268+0000 7f0e029e4740 1 -- 172.21.15.42:0/2826153858 shutdown_connections 2024-07-26T06:51:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.268+0000 7f0e029e4740 1 -- 172.21.15.42:0/2826153858 wait complete. 2024-07-26T06:51:20.273 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.268+0000 7f0e029e4740 20 client.15510 trim_cache size 0 max 16384 2024-07-26T06:51:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.283+0000 7f0e029e4740 1 Processor -- start 2024-07-26T06:51:20.285 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0e029e4740 1 -- start start 2024-07-26T06:51:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0e029e4740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b544d80 0x564a4b92f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0e029e4740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b55d2f0 0x564a4b932ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0e029e4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b5654d0 0x564a4b933020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0e029e4740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0df41acb70 con 0x564a4b544d80 2024-07-26T06:51:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0e029e4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0df41ae6b0 con 0x564a4b5654d0 2024-07-26T06:51:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0e029e4740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0df41ad240 con 0x564a4b55d2f0 2024-07-26T06:51:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0df9787640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b544d80 0x564a4b92f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0df9787640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b544d80 0x564a4b92f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48692/0 (socket says 172.21.15.42:48692) 2024-07-26T06:51:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0df9787640 1 -- 172.21.15.42:0/2534052646 learned_addr learned my addr 172.21.15.42:0/2534052646 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0df8f86640 1 --2- 172.21.15.42:0/2534052646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b55d2f0 0x564a4b932ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:20.286 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0df9f88640 1 --2- 172.21.15.42:0/2534052646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b5654d0 0x564a4b933020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0df9787640 1 -- 172.21.15.42:0/2534052646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b55d2f0 msgr2=0x564a4b932ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0df9787640 1 --2- 172.21.15.42:0/2534052646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b55d2f0 0x564a4b932ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0df9787640 1 -- 172.21.15.42:0/2534052646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b5654d0 msgr2=0x564a4b933020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0df9787640 1 --2- 172.21.15.42:0/2534052646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b5654d0 0x564a4b933020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0df9787640 1 -- 172.21.15.42:0/2534052646 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a4b925c90 con 0x564a4b544d80 2024-07-26T06:51:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.284+0000 7f0df9f88640 1 --2- 172.21.15.42:0/2534052646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b5654d0 0x564a4b933020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:51:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.285+0000 7f0df9787640 1 --2- 172.21.15.42:0/2534052646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b544d80 0x564a4b92f5c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0df0033000 tx=0x7f0df0031a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e8d00d0bc5321f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.285+0000 7f0ddeffd640 1 -- 172.21.15.42:0/2534052646 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df005f070 con 0x564a4b544d80 2024-07-26T06:51:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.285+0000 7f0ddeffd640 1 -- 172.21.15.42:0/2534052646 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0df003d450 con 0x564a4b544d80 2024-07-26T06:51:20.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.285+0000 7f0ddeffd640 1 -- 172.21.15.42:0/2534052646 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df0035040 con 0x564a4b544d80 2024-07-26T06:51:20.291 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.286+0000 7f0e029e4740 1 -- 172.21.15.42:0/2534052646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b544d80 msgr2=0x564a4b92f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:20.291 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.286+0000 7f0e029e4740 1 --2- 172.21.15.42:0/2534052646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b544d80 0x564a4b92f5c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0df0033000 tx=0x7f0df0031a10 comp rx=0 tx=0).stop 2024-07-26T06:51:20.291 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.286+0000 7f0e029e4740 1 -- 172.21.15.42:0/2534052646 shutdown_connections 2024-07-26T06:51:20.291 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.286+0000 7f0e029e4740 1 --2- 172.21.15.42:0/2534052646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b5654d0 0x564a4b933020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.291 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.286+0000 7f0e029e4740 1 --2- 172.21.15.42:0/2534052646 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b55d2f0 0x564a4b932ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.286+0000 7f0e029e4740 1 --2- 172.21.15.42:0/2534052646 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b544d80 0x564a4b92f5c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.286+0000 7f0e029e4740 1 -- 172.21.15.42:0/2534052646 >> 172.21.15.42:0/2534052646 conn(0x564a4b57a300 msgr2=0x564a4b92b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.286+0000 7f0e029e4740 1 -- 172.21.15.42:0/2534052646 shutdown_connections 2024-07-26T06:51:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.286+0000 7f0e029e4740 1 -- 172.21.15.42:0/2534052646 wait complete. 2024-07-26T06:51:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.287+0000 7f0e029e4740 1 -- start start 2024-07-26T06:51:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.287+0000 7f0e029e4740 1 Processor -- start 2024-07-26T06:51:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.287+0000 7f0e029e4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b544d80 0x564a4b93b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.287+0000 7f0e029e4740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b55d2f0 0x564a4b93bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.287+0000 7f0e029e4740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b5654d0 0x564a4b9a85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0df9f88640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b5654d0 0x564a4b9a85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0df9787640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b544d80 0x564a4b93b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:20.292 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0df9787640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b544d80 0x564a4b93b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:35644/0 (socket says 172.21.15.42:35644) 2024-07-26T06:51:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0df9787640 1 -- 172.21.15.42:0/1135580410 learned_addr learned my addr 172.21.15.42:0/1135580410 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0df9f88640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b5654d0 0x564a4b9a85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48708/0 (socket says 172.21.15.42:48708) 2024-07-26T06:51:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0e029e4740 1 -- 172.21.15.42:0/1135580410 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0df41adfe0 con 0x564a4b5654d0 2024-07-26T06:51:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0e029e4740 1 -- 172.21.15.42:0/1135580410 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0df41ad910 con 0x564a4b544d80 2024-07-26T06:51:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0e029e4740 1 -- 172.21.15.42:0/1135580410 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0df41aab50 con 0x564a4b55d2f0 2024-07-26T06:51:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0df8f86640 1 --2- 172.21.15.42:0/1135580410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b55d2f0 0x564a4b93bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0e029e4740 10 client.0 _ll_register_callbacks cb 0x7f0df00318d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0df9787640 1 -- 172.21.15.42:0/1135580410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b55d2f0 msgr2=0x564a4b93bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0df9787640 1 --2- 172.21.15.42:0/1135580410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b55d2f0 0x564a4b93bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0df9787640 1 -- 172.21.15.42:0/1135580410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b5654d0 msgr2=0x564a4b9a85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0df9787640 1 --2- 172.21.15.42:0/1135580410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b5654d0 0x564a4b9a85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0df9787640 1 -- 172.21.15.42:0/1135580410 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a4b9a8cc0 con 0x564a4b544d80 2024-07-26T06:51:20.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0e029e4740 1 -- 172.21.15.42:0/1135580410 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0df003d450 con 0x564a4b544d80 2024-07-26T06:51:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0df9787640 1 --2- 172.21.15.42:0/1135580410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b544d80 0x564a4b93b8e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0df0006010 tx=0x7f0df0009230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7eb6a9f399c6c6c1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.288+0000 7f0ddcff9640 10 client.25145 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:51:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.289+0000 7f0ddcff9640 1 -- 172.21.15.42:0/1135580410 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df0035070 con 0x564a4b544d80 2024-07-26T06:51:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.289+0000 7f0ddcff9640 1 -- 172.21.15.42:0/1135580410 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f0df0031ad0 con 0x564a4b544d80 2024-07-26T06:51:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.289+0000 7f0ddcff9640 1 -- 172.21.15.42:0/1135580410 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0df005f040 con 0x564a4b544d80 2024-07-26T06:51:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.289+0000 7f0e029e4740 10 client.25145 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.289+0000 7f0e029e4740 1 -- 172.21.15.42:0/1135580410 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({fsmap=29}) -- 0x564a4b9a9e30 con 0x564a4b544d80 2024-07-26T06:51:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.289+0000 7f0ddcff9640 1 -- 172.21.15.42:0/1135580410 <== mon.2 v2:172.21.15.163:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0df0062070 con 0x564a4b544d80 2024-07-26T06:51:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.289+0000 7f0e029e4740 10 client.25145 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.289+0000 7f0e029e4740 10 client.25145 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:51:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.289+0000 7f0e029e4740 20 client.25145 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:20.294 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.289+0000 7f0e029e4740 1 --2- 172.21.15.42:0/1135580410 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x564a4b5861b0 0x564a4b9ac440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.289+0000 7f0e029e4740 4 client.25145 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:51:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.289+0000 7f0e029e4740 1 -- 172.21.15.42:0/1135580410 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564a4b89bda0 con 0x564a4b5861b0 2024-07-26T06:51:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.290+0000 7f0df8f86640 1 --2- 172.21.15.42:0/1135580410 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x564a4b5861b0 0x564a4b9ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.290+0000 7f0ddcff9640 10 client.25145 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.290+0000 7f0df8f86640 1 --2- 172.21.15.42:0/1135580410 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x564a4b5861b0 0x564a4b9ac440 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0f09614625af42d server_cookie=8b7ae5c835206d5 in_seq=0 out_seq=0 2024-07-26T06:51:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.294+0000 7f0ddcff9640 1 -- 172.21.15.42:0/1135580410 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564a4b89bda0 con 0x564a4b5861b0 2024-07-26T06:51:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.294+0000 7f0ddcff9640 10 client.25145 handle_command_reply: tid=0 2024-07-26T06:51:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.294+0000 7f0e029e4740 1 client.25145 shutdown 2024-07-26T06:51:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.294+0000 7f0e029e4740 1 -- 172.21.15.42:0/1135580410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b544d80 msgr2=0x564a4b93b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.294+0000 7f0e029e4740 1 --2- 172.21.15.42:0/1135580410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b544d80 0x564a4b93b8e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0df0006010 tx=0x7f0df0009230 comp rx=0 tx=0).stop 2024-07-26T06:51:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.295+0000 7f0e029e4740 1 -- 172.21.15.42:0/1135580410 shutdown_connections 2024-07-26T06:51:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.295+0000 7f0e029e4740 1 --2- 172.21.15.42:0/1135580410 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x564a4b5861b0 0x564a4b9ac440 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.295+0000 7f0e029e4740 1 --2- 172.21.15.42:0/1135580410 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564a4b5654d0 0x564a4b9a85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.295+0000 7f0e029e4740 1 --2- 172.21.15.42:0/1135580410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564a4b55d2f0 0x564a4b93bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.295+0000 7f0e029e4740 1 --2- 172.21.15.42:0/1135580410 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564a4b544d80 0x564a4b93b8e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.295+0000 7f0e029e4740 1 -- 172.21.15.42:0/1135580410 >> 172.21.15.42:0/1135580410 conn(0x564a4b57a300 msgr2=0x564a4b92b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.295+0000 7f0ddcff9640 0 client.25145 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.295+0000 7f0e029e4740 1 -- 172.21.15.42:0/1135580410 shutdown_connections 2024-07-26T06:51:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.295+0000 7f0e029e4740 1 -- 172.21.15.42:0/1135580410 wait complete. 2024-07-26T06:51:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.295+0000 7f0e029e4740 20 client.25145 trim_cache size 0 max 16384 2024-07-26T06:51:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:51:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:51:20.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:51:20.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:51:20.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.295+0000 7f0dfb9eb640 1 -- 172.21.15.42:0/1957705742 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0de00b3cd0 msgr2=0x7f0de00b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.295+0000 7f0dfb9eb640 1 --2- 172.21.15.42:0/1957705742 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0de00b3cd0 0x7f0de00b61e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0df0000c00 tx=0x7f0df0031400 comp rx=0 tx=0).stop 2024-07-26T06:51:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.295+0000 7f0dfb9eb640 1 -- 172.21.15.42:0/1957705742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0df4104000 msgr2=0x7f0df419c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.295+0000 7f0dfb9eb640 1 --2- 172.21.15.42:0/1957705742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0df4104000 0x7f0df419c1c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0de400d950 tx=0x7f0de400de10 comp rx=0 tx=0).stop 2024-07-26T06:51:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.296+0000 7f0dfb9eb640 1 -- 172.21.15.42:0/1957705742 shutdown_connections 2024-07-26T06:51:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.296+0000 7f0dfb9eb640 1 --2- 172.21.15.42:0/1957705742 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0de00b3cd0 0x7f0de00b61e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.296+0000 7f0dfb9eb640 1 --2- 172.21.15.42:0/1957705742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0df410b040 0x7f0df419cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.296+0000 7f0dfb9eb640 1 --2- 172.21.15.42:0/1957705742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0df4104000 0x7f0df419c1c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.296+0000 7f0dfb9eb640 1 --2- 172.21.15.42:0/1957705742 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0df4101580 0x7f0df419bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.296+0000 7f0dfb9eb640 1 -- 172.21.15.42:0/1957705742 >> 172.21.15.42:0/1957705742 conn(0x7f0df40fd0c0 msgr2=0x7f0df41023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.296+0000 7f0dfb9eb640 1 -- 172.21.15.42:0/1957705742 shutdown_connections 2024-07-26T06:51:20.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:20.296+0000 7f0dfb9eb640 1 -- 172.21.15.42:0/1957705742 wait complete. 2024-07-26T06:51:21.343 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:21 smithi042 ceph-mon[32775]: pgmap v429: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T06:51:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:21 smithi164 ceph-mon[36970]: pgmap v429: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T06:51:21.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:21 smithi163 ceph-mon[36873]: pgmap v429: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T06:51:22.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.850+0000 7ff2910cc640 1 -- 172.21.15.42:0/3570127876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff28c102340 msgr2=0x7ff28c111fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:22.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.850+0000 7ff2910cc640 1 --2- 172.21.15.42:0/3570127876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff28c102340 0x7ff28c111fd0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff284009ca0 tx=0x7ff28402a950 comp rx=0 tx=0).stop 2024-07-26T06:51:22.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.850+0000 7ff2910cc640 1 -- 172.21.15.42:0/3570127876 shutdown_connections 2024-07-26T06:51:22.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.850+0000 7ff2910cc640 1 --2- 172.21.15.42:0/3570127876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff28c102340 0x7ff28c111fd0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.850+0000 7ff2910cc640 1 --2- 172.21.15.42:0/3570127876 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff28c101840 0x7ff28c101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.850+0000 7ff2910cc640 1 --2- 172.21.15.42:0/3570127876 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff28c104110 0x7ff28c104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.850+0000 7ff2910cc640 1 -- 172.21.15.42:0/3570127876 >> 172.21.15.42:0/3570127876 conn(0x7ff28c0799c0 msgr2=0x7ff28c079dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.850+0000 7ff2910cc640 1 -- 172.21.15.42:0/3570127876 shutdown_connections 2024-07-26T06:51:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.850+0000 7ff2910cc640 1 -- 172.21.15.42:0/3570127876 wait complete. 2024-07-26T06:51:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.851+0000 7ff2910cc640 1 Processor -- start 2024-07-26T06:51:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.851+0000 7ff2910cc640 1 -- start start 2024-07-26T06:51:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.852+0000 7ff2910cc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff28c101840 0x7ff28c1089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.852+0000 7ff2910cc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff28c102340 0x7ff28c104c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.852+0000 7ff2910cc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff28c104110 0x7ff28c1053a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.852+0000 7ff2910cc640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c1156c0 con 0x7ff28c104110 2024-07-26T06:51:22.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.852+0000 7ff2910cc640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c115540 con 0x7ff28c102340 2024-07-26T06:51:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.852+0000 7ff2910cc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c115840 con 0x7ff28c101840 2024-07-26T06:51:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.852+0000 7ff28b59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff28c104110 0x7ff28c1053a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.852+0000 7ff28b59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff28c104110 0x7ff28c1053a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48728/0 (socket says 172.21.15.42:48728) 2024-07-26T06:51:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.852+0000 7ff28b59e640 1 -- 172.21.15.42:0/668916660 learned_addr learned my addr 172.21.15.42:0/668916660 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.852+0000 7ff28ad9d640 1 --2- 172.21.15.42:0/668916660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff28c101840 0x7ff28c1089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.852+0000 7ff28a59c640 1 --2- 172.21.15.42:0/668916660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff28c102340 0x7ff28c104c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.853+0000 7ff28b59e640 1 -- 172.21.15.42:0/668916660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff28c101840 msgr2=0x7ff28c1089a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.853+0000 7ff28b59e640 1 --2- 172.21.15.42:0/668916660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff28c101840 0x7ff28c1089a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.853+0000 7ff28b59e640 1 -- 172.21.15.42:0/668916660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff28c102340 msgr2=0x7ff28c104c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.853+0000 7ff28b59e640 1 --2- 172.21.15.42:0/668916660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff28c102340 0x7ff28c104c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:22.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.853+0000 7ff28b59e640 1 -- 172.21.15.42:0/668916660 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff28c1ac5a0 con 0x7ff28c104110 2024-07-26T06:51:22.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.853+0000 7ff28b59e640 1 --2- 172.21.15.42:0/668916660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff28c104110 0x7ff28c1053a0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff28402ae70 tx=0x7ff28402f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:22.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.853+0000 7ff273fff640 1 -- 172.21.15.42:0/668916660 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff284034040 con 0x7ff28c104110 2024-07-26T06:51:22.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.854+0000 7ff2910cc640 1 -- 172.21.15.42:0/668916660 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff28c1ac770 con 0x7ff28c104110 2024-07-26T06:51:22.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.854+0000 7ff273fff640 1 -- 172.21.15.42:0/668916660 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2840308f0 con 0x7ff28c104110 2024-07-26T06:51:22.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.854+0000 7ff273fff640 1 -- 172.21.15.42:0/668916660 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2840047f0 con 0x7ff28c104110 2024-07-26T06:51:22.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.854+0000 7ff2910cc640 1 -- 172.21.15.42:0/668916660 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff28c1acd00 con 0x7ff28c104110 2024-07-26T06:51:22.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.855+0000 7ff2910cc640 1 -- 172.21.15.42:0/668916660 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff28c101c50 con 0x7ff28c104110 2024-07-26T06:51:22.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.857+0000 7ff273fff640 1 -- 172.21.15.42:0/668916660 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7ff284037020 con 0x7ff28c104110 2024-07-26T06:51:22.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.858+0000 7ff273fff640 1 --2- 172.21.15.42:0/668916660 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff2740b3bd0 0x7ff2740b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:22.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.858+0000 7ff273fff640 1 -- 172.21.15.42:0/668916660 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7ff284104da0 con 0x7ff28c104110 2024-07-26T06:51:22.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.858+0000 7ff28ad9d640 1 --2- 172.21.15.42:0/668916660 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff2740b3bd0 0x7ff2740b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:22.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.859+0000 7ff28ad9d640 1 --2- 172.21.15.42:0/668916660 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff2740b3bd0 0x7ff2740b60e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff27c005fd0 tx=0x7ff27c005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:22.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:22.865+0000 7ff273fff640 1 -- 172.21.15.42:0/668916660 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff2840cc3d0 con 0x7ff28c104110 2024-07-26T06:51:23.058 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:23 smithi042 ceph-mon[32775]: pgmap v430: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:51:23.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:23.091+0000 7ff2910cc640 1 -- 172.21.15.42:0/668916660 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff28c1acfd0 con 0x7ff28c104110 2024-07-26T06:51:23.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:23.092+0000 7ff273fff640 1 -- 172.21.15.42:0/668916660 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11978 (secure 0 0 0) 0x7ff2840cbcd0 con 0x7ff28c104110 2024-07-26T06:51:23.093 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:51:23.093 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":29,"btime":"2024-07-26T06:50:27:493008+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:50:26.696932+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24620,24601,24593,24583,24578]},"id":1}]} 2024-07-26T06:51:23.093 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 29 2024-07-26T06:51:23.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:23.094+0000 7ff2910cc640 1 -- 172.21.15.42:0/668916660 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff2740b3bd0 msgr2=0x7ff2740b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:23.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:23.094+0000 7ff2910cc640 1 --2- 172.21.15.42:0/668916660 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff2740b3bd0 0x7ff2740b60e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff27c005fd0 tx=0x7ff27c005850 comp rx=0 tx=0).stop 2024-07-26T06:51:23.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:23.095+0000 7ff2910cc640 1 -- 172.21.15.42:0/668916660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff28c104110 msgr2=0x7ff28c1053a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:23.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:23.095+0000 7ff2910cc640 1 --2- 172.21.15.42:0/668916660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff28c104110 0x7ff28c1053a0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff28402ae70 tx=0x7ff28402f5d0 comp rx=0 tx=0).stop 2024-07-26T06:51:23.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:23.095+0000 7ff2910cc640 1 -- 172.21.15.42:0/668916660 shutdown_connections 2024-07-26T06:51:23.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:23.095+0000 7ff2910cc640 1 --2- 172.21.15.42:0/668916660 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff2740b3bd0 0x7ff2740b60e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:23.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:23.095+0000 7ff2910cc640 1 --2- 172.21.15.42:0/668916660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff28c104110 0x7ff28c1053a0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:23.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:23.095+0000 7ff2910cc640 1 --2- 172.21.15.42:0/668916660 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff28c102340 0x7ff28c104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:23.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:23.095+0000 7ff2910cc640 1 --2- 172.21.15.42:0/668916660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff28c101840 0x7ff28c1089a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:23.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:23.095+0000 7ff2910cc640 1 -- 172.21.15.42:0/668916660 >> 172.21.15.42:0/668916660 conn(0x7ff28c0799c0 msgr2=0x7ff28c0ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:23.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:23.095+0000 7ff2910cc640 1 -- 172.21.15.42:0/668916660 shutdown_connections 2024-07-26T06:51:23.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:23.095+0000 7ff2910cc640 1 -- 172.21.15.42:0/668916660 wait complete. 2024-07-26T06:51:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:23 smithi164 ceph-mon[36970]: pgmap v430: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:51:23.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:23 smithi163 ceph-mon[36873]: pgmap v430: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T06:51:23.952 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24620 quiesce db / --timeout 90 --expiration 142.55 --await-for 45.0' 2024-07-26T06:51:23.952 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.24620 quiesce db / --timeout 90 --expiration 142.55 --await-for 45.0 2024-07-26T06:51:24.381 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:51:24.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:24 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/668916660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:51:24.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:24 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/668916660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:51:24.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:24 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/668916660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:51:25.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:25 smithi042 ceph-mon[32775]: pgmap v431: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:51:25.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:25 smithi164 ceph-mon[36970]: pgmap v431: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:51:25.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:25 smithi163 ceph-mon[36873]: pgmap v431: 97 pgs: 97 active+clean; 662 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T06:51:26.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.384+0000 7f229c5af640 1 -- 172.21.15.42:0/1878679323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2294106610 msgr2=0x7f22941100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:26.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.384+0000 7f229c5af640 1 --2- 172.21.15.42:0/1878679323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2294106610 0x7f22941100a0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f228c009ca0 tx=0x7f228c02a8c0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.384+0000 7f229c5af640 1 -- 172.21.15.42:0/1878679323 shutdown_connections 2024-07-26T06:51:26.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.384+0000 7f229c5af640 1 --2- 172.21.15.42:0/1878679323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2294106610 0x7f22941100a0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.384+0000 7f229c5af640 1 --2- 172.21.15.42:0/1878679323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2294105cc0 0x7f22941060d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.384+0000 7f229c5af640 1 --2- 172.21.15.42:0/1878679323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2294101ad0 0x7f2294101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.384+0000 7f229c5af640 1 -- 172.21.15.42:0/1878679323 >> 172.21.15.42:0/1878679323 conn(0x7f2294079980 msgr2=0x7f2294079d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:26.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.384+0000 7f229c5af640 1 -- 172.21.15.42:0/1878679323 shutdown_connections 2024-07-26T06:51:26.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.384+0000 7f229c5af640 1 -- 172.21.15.42:0/1878679323 wait complete. 2024-07-26T06:51:26.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.385+0000 7f229c5af640 1 Processor -- start 2024-07-26T06:51:26.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.385+0000 7f229c5af640 1 -- start start 2024-07-26T06:51:26.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.386+0000 7f229c5af640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2294101ad0 0x7f2294199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.386+0000 7f229c5af640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2294105cc0 0x7f229419a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.386+0000 7f229c5af640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2294106610 0x7f229419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.386+0000 7f229c5af640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f22941134e0 con 0x7f2294105cc0 2024-07-26T06:51:26.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.386+0000 7f229c5af640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2294113360 con 0x7f2294101ad0 2024-07-26T06:51:26.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.386+0000 7f229c5af640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2294113660 con 0x7f2294106610 2024-07-26T06:51:26.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.386+0000 7f2299b4a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2294105cc0 0x7f229419a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:26.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.386+0000 7f2299b4a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2294105cc0 0x7f229419a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48754/0 (socket says 172.21.15.42:48754) 2024-07-26T06:51:26.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.386+0000 7f2299b4a640 1 -- 172.21.15.42:0/1385881272 learned_addr learned my addr 172.21.15.42:0/1385881272 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:26.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.386+0000 7f229a34b640 1 --2- 172.21.15.42:0/1385881272 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2294101ad0 0x7f2294199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:26.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.386+0000 7f229ab4c640 1 --2- 172.21.15.42:0/1385881272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2294106610 0x7f229419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:26.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.386+0000 7f2299b4a640 1 -- 172.21.15.42:0/1385881272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2294106610 msgr2=0x7f229419f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:26.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.386+0000 7f2299b4a640 1 --2- 172.21.15.42:0/1385881272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2294106610 0x7f229419f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.386+0000 7f2299b4a640 1 -- 172.21.15.42:0/1385881272 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2294101ad0 msgr2=0x7f2294199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:26.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.386+0000 7f2299b4a640 1 --2- 172.21.15.42:0/1385881272 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2294101ad0 0x7f2294199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.386+0000 7f2299b4a640 1 -- 172.21.15.42:0/1385881272 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f229419fde0 con 0x7f2294105cc0 2024-07-26T06:51:26.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.387+0000 7f2299b4a640 1 --2- 172.21.15.42:0/1385881272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2294105cc0 0x7f229419a410 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f228400d740 tx=0x7f228400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:26.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.387+0000 7f228b7fe640 1 -- 172.21.15.42:0/1385881272 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2284017070 con 0x7f2294105cc0 2024-07-26T06:51:26.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.387+0000 7f229c5af640 1 -- 172.21.15.42:0/1385881272 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22941a00d0 con 0x7f2294105cc0 2024-07-26T06:51:26.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.387+0000 7f229c5af640 1 -- 172.21.15.42:0/1385881272 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22941a6830 con 0x7f2294105cc0 2024-07-26T06:51:26.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.387+0000 7f228b7fe640 1 -- 172.21.15.42:0/1385881272 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f228400de50 con 0x7f2294105cc0 2024-07-26T06:51:26.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.387+0000 7f228b7fe640 1 -- 172.21.15.42:0/1385881272 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228400f620 con 0x7f2294105cc0 2024-07-26T06:51:26.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.390+0000 7f22a2b33740 1 Processor -- start 2024-07-26T06:51:26.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.390+0000 7f22a2b33740 1 -- start start 2024-07-26T06:51:26.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.391+0000 7f22a2b33740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e505850 0x55fe4e8ab540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.391+0000 7f22a2b33740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e4dc990 0x55fe4e8aba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.391+0000 7f22a2b33740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.391+0000 7f22a2b33740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55fe4e8206c0 con 0x55fe4e4dc990 2024-07-26T06:51:26.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.391+0000 7f22a2b33740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fe4e80fbc0 con 0x55fe4e505850 2024-07-26T06:51:26.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.391+0000 7f22a2b33740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55fe4e813c00 con 0x55fe4e4e4b70 2024-07-26T06:51:26.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.391+0000 7f229a34b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e4dc990 0x55fe4e8aba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:26.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.391+0000 7f229ab4c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e505850 0x55fe4e8ab540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:26.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.391+0000 7f229ab4c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e505850 0x55fe4e8ab540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:35676/0 (socket says 172.21.15.42:35676) 2024-07-26T06:51:26.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.391+0000 7f229a34b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e4dc990 0x55fe4e8aba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48770/0 (socket says 172.21.15.42:48770) 2024-07-26T06:51:26.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.391+0000 7f229ab4c640 1 -- 172.21.15.42:0/1876485104 learned_addr learned my addr 172.21.15.42:0/1876485104 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:26.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.391+0000 7f229ab4c640 1 -- 172.21.15.42:0/1876485104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4e4b70 msgr2=0x55fe4e8abfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T06:51:26.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.391+0000 7f229ab4c640 1 --2- 172.21.15.42:0/1876485104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8abfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.391+0000 7f229ab4c640 1 -- 172.21.15.42:0/1876485104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e4dc990 msgr2=0x55fe4e8aba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:26.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.392+0000 7f229ab4c640 1 --2- 172.21.15.42:0/1876485104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e4dc990 0x55fe4e8aba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.392+0000 7f229ab4c640 1 -- 172.21.15.42:0/1876485104 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe4e79ba40 con 0x55fe4e505850 2024-07-26T06:51:26.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.392+0000 7f229a34b640 1 --2- 172.21.15.42:0/1876485104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e4dc990 0x55fe4e8aba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:51:26.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.392+0000 7f228b7fe640 1 -- 172.21.15.42:0/1385881272 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f228401a070 con 0x7f2294105cc0 2024-07-26T06:51:26.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.392+0000 7f229ab4c640 1 --2- 172.21.15.42:0/1876485104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e505850 0x55fe4e8ab540 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f228c035000 tx=0x7f228c02f410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65879a1e3d11970f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:26.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.393+0000 7f2288ff9640 1 -- 172.21.15.42:0/1876485104 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c031ae0 con 0x55fe4e505850 2024-07-26T06:51:26.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.393+0000 7f2288ff9640 1 -- 172.21.15.42:0/1876485104 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f228c036040 con 0x55fe4e505850 2024-07-26T06:51:26.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.393+0000 7f2288ff9640 1 -- 172.21.15.42:0/1876485104 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c004040 con 0x55fe4e505850 2024-07-26T06:51:26.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.393+0000 7f228b7fe640 1 --2- 172.21.15.42:0/1385881272 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f22800b3c50 0x7f22800b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.393+0000 7f229a34b640 1 --2- 172.21.15.42:0/1385881272 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f22800b3c50 0x7f22800b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:26.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.393+0000 7f228b7fe640 1 -- 172.21.15.42:0/1385881272 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f22840e3450 con 0x7f2294105cc0 2024-07-26T06:51:26.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.394+0000 7f229a34b640 1 --2- 172.21.15.42:0/1385881272 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f22800b3c50 0x7f22800b6160 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x55fe4e8aa1c0 tx=0x7f2290005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:26.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.394+0000 7f22a2b33740 1 -- 172.21.15.42:0/1876485104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e505850 msgr2=0x55fe4e8ab540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:26.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.394+0000 7f22a2b33740 1 --2- 172.21.15.42:0/1876485104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e505850 0x55fe4e8ab540 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f228c035000 tx=0x7f228c02f410 comp rx=0 tx=0).stop 2024-07-26T06:51:26.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.394+0000 7f22a2b33740 1 -- 172.21.15.42:0/1876485104 shutdown_connections 2024-07-26T06:51:26.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.394+0000 7f22a2b33740 1 --2- 172.21.15.42:0/1876485104 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8abfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.394+0000 7f22a2b33740 1 --2- 172.21.15.42:0/1876485104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e4dc990 0x55fe4e8aba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.394+0000 7f22a2b33740 1 --2- 172.21.15.42:0/1876485104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e505850 0x55fe4e8ab540 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.394+0000 7f22a2b33740 1 -- 172.21.15.42:0/1876485104 >> 172.21.15.42:0/1876485104 conn(0x55fe4e4f9c40 msgr2=0x55fe4e8a9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:26.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.394+0000 7f22a2b33740 1 -- 172.21.15.42:0/1876485104 shutdown_connections 2024-07-26T06:51:26.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.394+0000 7f22a2b33740 1 -- 172.21.15.42:0/1876485104 wait complete. 2024-07-26T06:51:26.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.395+0000 7f22a2b33740 1 -- start start 2024-07-26T06:51:26.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.395+0000 7f22a2b33740 1 Processor -- start 2024-07-26T06:51:26.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.396+0000 7f22a2b33740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e4dc990 0x55fe4e8bd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.396+0000 7f22a2b33740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8b6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.396+0000 7f22a2b33740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e505850 0x55fe4e8b7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.396+0000 7f22a2b33740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55fe4e6737b0 con 0x55fe4e505850 2024-07-26T06:51:26.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.396+0000 7f22a2b33740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fe4e7b2240 con 0x55fe4e4dc990 2024-07-26T06:51:26.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.396+0000 7f22a2b33740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55fe4e8bdfc0 con 0x55fe4e4e4b70 2024-07-26T06:51:26.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.396+0000 7f22a2b33740 10 client.0 _ll_register_callbacks cb 0x55fe4e7932f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:26.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.396+0000 7f229a34b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e4dc990 0x55fe4e8bd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:26.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.396+0000 7f229ab4c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e505850 0x55fe4e8b7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:26.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.396+0000 7f229a34b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e4dc990 0x55fe4e8bd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:35680/0 (socket says 172.21.15.42:35680) 2024-07-26T06:51:26.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.396+0000 7f229a34b640 1 -- 172.21.15.42:0/2867501571 learned_addr learned my addr 172.21.15.42:0/2867501571 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:26.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.396+0000 7f2299b4a640 1 --2- 172.21.15.42:0/2867501571 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8b6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:26.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.396+0000 7f229ab4c640 1 -- 172.21.15.42:0/2867501571 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4e4b70 msgr2=0x55fe4e8b6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:26.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.396+0000 7f229ab4c640 1 --2- 172.21.15.42:0/2867501571 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8b6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.396+0000 7f229ab4c640 1 -- 172.21.15.42:0/2867501571 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e4dc990 msgr2=0x55fe4e8bd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:26.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.396+0000 7f229ab4c640 1 --2- 172.21.15.42:0/2867501571 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e4dc990 0x55fe4e8bd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.396+0000 7f229ab4c640 1 -- 172.21.15.42:0/2867501571 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe4e7c3740 con 0x55fe4e505850 2024-07-26T06:51:26.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.397+0000 7f2299b4a640 1 --2- 172.21.15.42:0/2867501571 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8b6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:51:26.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.397+0000 7f229ab4c640 1 --2- 172.21.15.42:0/2867501571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e505850 0x55fe4e8b7290 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f228c005e30 tx=0x7f228c02a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff558cda96039b4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:26.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.397+0000 7f227dffb640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:51:26.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.397+0000 7f227dffb640 1 -- 172.21.15.42:0/2867501571 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c031ae0 con 0x55fe4e505850 2024-07-26T06:51:26.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.397+0000 7f227dffb640 1 -- 172.21.15.42:0/2867501571 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f228c031c80 con 0x55fe4e505850 2024-07-26T06:51:26.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.397+0000 7f22a2b33740 1 -- 172.21.15.42:0/2867501571 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fe4e822610 con 0x55fe4e505850 2024-07-26T06:51:26.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.398+0000 7f227dffb640 1 -- 172.21.15.42:0/2867501571 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f228c031450 con 0x55fe4e505850 2024-07-26T06:51:26.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.398+0000 7f22a2b33740 10 client.15534 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:26.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.398+0000 7f22a2b33740 1 -- 172.21.15.42:0/2867501571 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fe4e57fba0 con 0x55fe4e505850 2024-07-26T06:51:26.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.398+0000 7f227dffb640 1 -- 172.21.15.42:0/2867501571 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55fe4e822610 con 0x55fe4e505850 2024-07-26T06:51:26.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.398+0000 7f227dffb640 1 -- 172.21.15.42:0/2867501571 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f228c0305c0 con 0x55fe4e505850 2024-07-26T06:51:26.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.398+0000 7f22a2b33740 10 client.15534 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:26.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.398+0000 7f22a2b33740 10 client.15534 resolve_mds: validated gid 24620 aka daemon mds.b 2024-07-26T06:51:26.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.398+0000 7f22a2b33740 20 client.15534 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:26.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.398+0000 7f22a2b33740 1 --2- 172.21.15.42:0/2867501571 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fe4e4c41d0 0x55fe4e8d1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.398+0000 7f22a2b33740 4 client.15534 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:51:26.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.398+0000 7f22a2b33740 1 -- 172.21.15.42:0/2867501571 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe4e81e0b0 con 0x55fe4e4c41d0 2024-07-26T06:51:26.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.399+0000 7f229a34b640 1 --2- 172.21.15.42:0/2867501571 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fe4e4c41d0 0x55fe4e8d1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:26.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.399+0000 7f227dffb640 10 client.15534 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:26.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.399+0000 7f229a34b640 1 --2- 172.21.15.42:0/2867501571 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fe4e4c41d0 0x55fe4e8d1340 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f0685a88e48e1cd server_cookie=97ea8df6f074d5c5 in_seq=0 out_seq=0 2024-07-26T06:51:26.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.400+0000 7f227dffb640 1 -- 172.21.15.42:0/2867501571 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fe4e81e0b0 con 0x55fe4e4c41d0 2024-07-26T06:51:26.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.400+0000 7f227dffb640 10 client.15534 handle_command_reply: tid=0 2024-07-26T06:51:26.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.400+0000 7f22a2b33740 1 client.15534 shutdown 2024-07-26T06:51:26.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.400+0000 7f22a2b33740 1 -- 172.21.15.42:0/2867501571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e505850 msgr2=0x55fe4e8b7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:26.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.400+0000 7f22a2b33740 1 --2- 172.21.15.42:0/2867501571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e505850 0x55fe4e8b7290 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f228c005e30 tx=0x7f228c02a8c0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.401+0000 7f22a2b33740 1 -- 172.21.15.42:0/2867501571 shutdown_connections 2024-07-26T06:51:26.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.401+0000 7f22a2b33740 1 --2- 172.21.15.42:0/2867501571 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fe4e4c41d0 0x55fe4e8d1340 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.401+0000 7f22a2b33740 1 --2- 172.21.15.42:0/2867501571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e505850 0x55fe4e8b7290 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.401+0000 7f22a2b33740 1 --2- 172.21.15.42:0/2867501571 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8b6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.401+0000 7f22a2b33740 1 --2- 172.21.15.42:0/2867501571 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e4dc990 0x55fe4e8bd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.401+0000 7f22a2b33740 1 -- 172.21.15.42:0/2867501571 >> 172.21.15.42:0/2867501571 conn(0x55fe4e4f9c40 msgr2=0x55fe4e8acf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:26.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.401+0000 7f227dffb640 0 client.15534 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:26.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.401+0000 7f22a2b33740 1 -- 172.21.15.42:0/2867501571 shutdown_connections 2024-07-26T06:51:26.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.401+0000 7f22a2b33740 1 -- 172.21.15.42:0/2867501571 wait complete. 2024-07-26T06:51:26.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.401+0000 7f22a2b33740 20 client.15534 trim_cache size 0 max 16384 2024-07-26T06:51:26.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.434+0000 7f22a2b33740 1 Processor -- start 2024-07-26T06:51:26.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.434+0000 7f22a2b33740 1 -- start start 2024-07-26T06:51:26.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.434+0000 7f22a2b33740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e4dc990 0x55fe4e8b2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.434+0000 7f22a2b33740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8a8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.435+0000 7f22a2b33740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e505850 0x55fe4e8a86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.435+0000 7f22a2b33740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f227800e500 con 0x55fe4e505850 2024-07-26T06:51:26.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.435+0000 7f22a2b33740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2278010040 con 0x55fe4e4dc990 2024-07-26T06:51:26.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.435+0000 7f22a2b33740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f227800ebd0 con 0x55fe4e4e4b70 2024-07-26T06:51:26.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.435+0000 7f2299b4a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8a8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:26.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.435+0000 7f2299b4a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8a8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:43820/0 (socket says 172.21.15.42:43820) 2024-07-26T06:51:26.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.435+0000 7f2299b4a640 1 -- 172.21.15.42:0/676256902 learned_addr learned my addr 172.21.15.42:0/676256902 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:26.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.435+0000 7f229ab4c640 1 --2- 172.21.15.42:0/676256902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e505850 0x55fe4e8a86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:26.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.435+0000 7f229a34b640 1 --2- 172.21.15.42:0/676256902 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e4dc990 0x55fe4e8b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:26.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.436+0000 7f2299b4a640 1 -- 172.21.15.42:0/676256902 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e4dc990 msgr2=0x55fe4e8b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:26.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.436+0000 7f2299b4a640 1 --2- 172.21.15.42:0/676256902 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e4dc990 0x55fe4e8b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.436+0000 7f2299b4a640 1 -- 172.21.15.42:0/676256902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e505850 msgr2=0x55fe4e8a86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:26.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.436+0000 7f2299b4a640 1 --2- 172.21.15.42:0/676256902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e505850 0x55fe4e8a86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.436+0000 7f2299b4a640 1 -- 172.21.15.42:0/676256902 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe4e8ada60 con 0x55fe4e4e4b70 2024-07-26T06:51:26.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.436+0000 7f229a34b640 1 --2- 172.21.15.42:0/676256902 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e4dc990 0x55fe4e8b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:51:26.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.436+0000 7f2299b4a640 1 --2- 172.21.15.42:0/676256902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8a8170 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f22840e12a0 tx=0x7f22840aabc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5708699576776596 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:26.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.437+0000 7f227ffff640 1 -- 172.21.15.42:0/676256902 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2284021070 con 0x55fe4e4e4b70 2024-07-26T06:51:26.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.437+0000 7f227ffff640 1 -- 172.21.15.42:0/676256902 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2284010b40 con 0x55fe4e4e4b70 2024-07-26T06:51:26.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.437+0000 7f227ffff640 1 -- 172.21.15.42:0/676256902 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22840a5cf0 con 0x55fe4e4e4b70 2024-07-26T06:51:26.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.438+0000 7f22a2b33740 1 -- 172.21.15.42:0/676256902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4e4b70 msgr2=0x55fe4e8a8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:26.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.438+0000 7f22a2b33740 1 --2- 172.21.15.42:0/676256902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8a8170 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f22840e12a0 tx=0x7f22840aabc0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.438+0000 7f22a2b33740 1 -- 172.21.15.42:0/676256902 shutdown_connections 2024-07-26T06:51:26.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.438+0000 7f22a2b33740 1 --2- 172.21.15.42:0/676256902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e505850 0x55fe4e8a86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.438+0000 7f22a2b33740 1 --2- 172.21.15.42:0/676256902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8a8170 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.438+0000 7f22a2b33740 1 --2- 172.21.15.42:0/676256902 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e4dc990 0x55fe4e8b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.438+0000 7f22a2b33740 1 -- 172.21.15.42:0/676256902 >> 172.21.15.42:0/676256902 conn(0x55fe4e4f9c40 msgr2=0x55fe4e8a7a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:26.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.439+0000 7f22a2b33740 1 -- 172.21.15.42:0/676256902 shutdown_connections 2024-07-26T06:51:26.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.439+0000 7f22a2b33740 1 -- 172.21.15.42:0/676256902 wait complete. 2024-07-26T06:51:26.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.439+0000 7f22a2b33740 1 -- start start 2024-07-26T06:51:26.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.439+0000 7f22a2b33740 1 Processor -- start 2024-07-26T06:51:26.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.440+0000 7f22a2b33740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4dc990 0x55fe4e8c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.440+0000 7f22a2b33740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8ca040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.440+0000 7f22a2b33740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e505850 0x55fe4e8ca580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.440+0000 7f22a2b33740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f227800f970 con 0x55fe4e4e4b70 2024-07-26T06:51:26.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.440+0000 7f22a2b33740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f227800f2a0 con 0x55fe4e505850 2024-07-26T06:51:26.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.440+0000 7f22a2b33740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f227800c4e0 con 0x55fe4e4dc990 2024-07-26T06:51:26.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.440+0000 7f229a34b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4dc990 0x55fe4e8c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:26.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.440+0000 7f229a34b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4dc990 0x55fe4e8c9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:43834/0 (socket says 172.21.15.42:43834) 2024-07-26T06:51:26.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.440+0000 7f229a34b640 1 -- 172.21.15.42:0/3793039796 learned_addr learned my addr 172.21.15.42:0/3793039796 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:26.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.440+0000 7f229ab4c640 1 --2- 172.21.15.42:0/3793039796 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e505850 0x55fe4e8ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:26.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.440+0000 7f22a2b33740 10 client.0 _ll_register_callbacks cb 0x7f22900097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:26.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.441+0000 7f229a34b640 1 -- 172.21.15.42:0/3793039796 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e505850 msgr2=0x55fe4e8ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:26.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.441+0000 7f229a34b640 1 --2- 172.21.15.42:0/3793039796 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e505850 0x55fe4e8ca580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.441+0000 7f229a34b640 1 -- 172.21.15.42:0/3793039796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e4e4b70 msgr2=0x55fe4e8ca040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T06:51:26.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.441+0000 7f2299b4a640 1 --2- 172.21.15.42:0/3793039796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:26.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.441+0000 7f229a34b640 1 --2- 172.21.15.42:0/3793039796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8ca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.441+0000 7f229a34b640 1 -- 172.21.15.42:0/3793039796 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe4e938890 con 0x55fe4e4dc990 2024-07-26T06:51:26.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.441+0000 7f2299b4a640 1 --2- 172.21.15.42:0/3793039796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:51:26.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.442+0000 7f229a34b640 1 --2- 172.21.15.42:0/3793039796 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4dc990 0x55fe4e8c9b00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2290013010 tx=0x7f2290017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db06c10a44bf9db1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:26.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.442+0000 7f227dffb640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-07-26T06:51:26.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.442+0000 7f227dffb640 1 -- 172.21.15.42:0/3793039796 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229001dd60 con 0x55fe4e4dc990 2024-07-26T06:51:26.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.442+0000 7f227dffb640 1 -- 172.21.15.42:0/3793039796 <== mon.1 v2:172.21.15.164:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f2290018030 con 0x55fe4e4dc990 2024-07-26T06:51:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.442+0000 7f22a2b33740 1 -- 172.21.15.42:0/3793039796 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2284010b40 con 0x55fe4e4dc990 2024-07-26T06:51:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.442+0000 7f227dffb640 1 -- 172.21.15.42:0/3793039796 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f229002b430 con 0x55fe4e4dc990 2024-07-26T06:51:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.442+0000 7f22a2b33740 10 client.25219 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.442+0000 7f22a2b33740 1 -- 172.21.15.42:0/3793039796 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fe4e9399f0 con 0x55fe4e4dc990 2024-07-26T06:51:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.442+0000 7f227dffb640 1 -- 172.21.15.42:0/3793039796 <== mon.1 v2:172.21.15.164:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f229002ba80 con 0x55fe4e4dc990 2024-07-26T06:51:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.444+0000 7f227dffb640 1 -- 172.21.15.42:0/3793039796 <== mon.1 v2:172.21.15.164:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f229002b5d0 con 0x55fe4e4dc990 2024-07-26T06:51:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.444+0000 7f22a2b33740 10 client.25219 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.444+0000 7f22a2b33740 10 client.25219 resolve_mds: validated gid 24620 aka daemon mds.b 2024-07-26T06:51:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.444+0000 7f22a2b33740 20 client.25219 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.444+0000 7f22a2b33740 1 --2- 172.21.15.42:0/3793039796 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fe4e499580 0x55fe4e93c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.444+0000 7f22a2b33740 4 client.25219 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.55, "await_for": 45.0}] 2024-07-26T06:51:26.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.444+0000 7f22a2b33740 1 -- 172.21.15.42:0/3793039796 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.55, "await_for": 45.0}) -- 0x7f2284010b40 con 0x55fe4e499580 2024-07-26T06:51:26.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.444+0000 7f2299b4a640 1 --2- 172.21.15.42:0/3793039796 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fe4e499580 0x55fe4e93c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:26.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.445+0000 7f2299b4a640 1 --2- 172.21.15.42:0/3793039796 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fe4e499580 0x55fe4e93c250 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba22fe25b62db51b server_cookie=7e12104b2f9121e3 in_seq=0 out_seq=0 2024-07-26T06:51:26.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.445+0000 7f227dffb640 10 client.25219 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:26.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.452+0000 7f227dffb640 1 -- 172.21.15.42:0/3793039796 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x7f2284010b40 con 0x55fe4e499580 2024-07-26T06:51:26.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.452+0000 7f227dffb640 10 client.25219 handle_command_reply: tid=0 2024-07-26T06:51:26.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.452+0000 7f22a2b33740 1 client.25219 shutdown 2024-07-26T06:51:26.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.452+0000 7f22a2b33740 1 -- 172.21.15.42:0/3793039796 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4dc990 msgr2=0x55fe4e8c9b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:26.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.452+0000 7f22a2b33740 1 --2- 172.21.15.42:0/3793039796 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4dc990 0x55fe4e8c9b00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2290013010 tx=0x7f2290017000 comp rx=0 tx=0).stop 2024-07-26T06:51:26.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.452+0000 7f22a2b33740 1 -- 172.21.15.42:0/3793039796 shutdown_connections 2024-07-26T06:51:26.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.452+0000 7f22a2b33740 1 --2- 172.21.15.42:0/3793039796 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fe4e499580 0x55fe4e93c250 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.452+0000 7f22a2b33740 1 --2- 172.21.15.42:0/3793039796 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fe4e505850 0x55fe4e8ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.452+0000 7f22a2b33740 1 --2- 172.21.15.42:0/3793039796 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fe4e4e4b70 0x55fe4e8ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.452+0000 7f22a2b33740 1 --2- 172.21.15.42:0/3793039796 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fe4e4dc990 0x55fe4e8c9b00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.452+0000 7f227dffb640 0 client.25219 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:26.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.452+0000 7f22a2b33740 1 -- 172.21.15.42:0/3793039796 >> 172.21.15.42:0/3793039796 conn(0x55fe4e4f9c40 msgr2=0x55fe4e8a8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:26.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.453+0000 7f22a2b33740 1 -- 172.21.15.42:0/3793039796 shutdown_connections 2024-07-26T06:51:26.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.453+0000 7f22a2b33740 1 -- 172.21.15.42:0/3793039796 wait complete. 2024-07-26T06:51:26.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.453+0000 7f22a2b33740 20 client.25219 trim_cache size 0 max 16384 2024-07-26T06:51:26.455 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:51:26.455 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 29, 2024-07-26T06:51:26.455 INFO:teuthology.orchestra.run.smithi042.stdout: "leader": 24620, 2024-07-26T06:51:26.455 INFO:teuthology.orchestra.run.smithi042.stdout: "set_version": 2, 2024-07-26T06:51:26.455 INFO:teuthology.orchestra.run.smithi042.stdout: "sets": { 2024-07-26T06:51:26.456 INFO:teuthology.orchestra.run.smithi042.stdout: "213c08d8": { 2024-07-26T06:51:26.456 INFO:teuthology.orchestra.run.smithi042.stdout: "version": 2, 2024-07-26T06:51:26.456 INFO:teuthology.orchestra.run.smithi042.stdout: "age_ref": 0.0, 2024-07-26T06:51:26.456 INFO:teuthology.orchestra.run.smithi042.stdout: "state": { 2024-07-26T06:51:26.456 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "QUIESCED", 2024-07-26T06:51:26.456 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 0.0 2024-07-26T06:51:26.456 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-07-26T06:51:26.456 INFO:teuthology.orchestra.run.smithi042.stdout: "timeout": 90.0, 2024-07-26T06:51:26.456 INFO:teuthology.orchestra.run.smithi042.stdout: "expiration": 142.6, 2024-07-26T06:51:26.456 INFO:teuthology.orchestra.run.smithi042.stdout: "members": { 2024-07-26T06:51:26.456 INFO:teuthology.orchestra.run.smithi042.stdout: "file:/": { 2024-07-26T06:51:26.456 INFO:teuthology.orchestra.run.smithi042.stdout: "excluded": false, 2024-07-26T06:51:26.456 INFO:teuthology.orchestra.run.smithi042.stdout: "state": { 2024-07-26T06:51:26.457 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "QUIESCED", 2024-07-26T06:51:26.457 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 0.0 2024-07-26T06:51:26.457 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:51:26.457 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:51:26.457 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:51:26.457 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:51:26.457 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:51:26.457 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:51:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.453+0000 7f229c5af640 1 -- 172.21.15.42:0/1385881272 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f22800b3c50 msgr2=0x7f22800b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.453+0000 7f229c5af640 1 --2- 172.21.15.42:0/1385881272 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f22800b3c50 0x7f22800b6160 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x55fe4e8aa1c0 tx=0x7f2290005f90 comp rx=0 tx=0).stop 2024-07-26T06:51:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.453+0000 7f229c5af640 1 -- 172.21.15.42:0/1385881272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2294105cc0 msgr2=0x7f229419a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.453+0000 7f229c5af640 1 --2- 172.21.15.42:0/1385881272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2294105cc0 0x7f229419a410 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f228400d740 tx=0x7f228400dc00 comp rx=0 tx=0).stop 2024-07-26T06:51:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.454+0000 7f229c5af640 1 -- 172.21.15.42:0/1385881272 shutdown_connections 2024-07-26T06:51:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.454+0000 7f229c5af640 1 --2- 172.21.15.42:0/1385881272 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f22800b3c50 0x7f22800b6160 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.454+0000 7f229c5af640 1 --2- 172.21.15.42:0/1385881272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2294106610 0x7f229419f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.454+0000 7f229c5af640 1 --2- 172.21.15.42:0/1385881272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2294105cc0 0x7f229419a410 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.454+0000 7f229c5af640 1 --2- 172.21.15.42:0/1385881272 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2294101ad0 0x7f2294199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:26.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.454+0000 7f229c5af640 1 -- 172.21.15.42:0/1385881272 >> 172.21.15.42:0/1385881272 conn(0x7f2294079980 msgr2=0x7f2294106eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:26.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.454+0000 7f229c5af640 1 -- 172.21.15.42:0/1385881272 shutdown_connections 2024-07-26T06:51:26.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:26.454+0000 7f229c5af640 1 -- 172.21.15.42:0/1385881272 wait complete. 2024-07-26T06:51:27.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:27 smithi042 ceph-mon[32775]: pgmap v432: 97 pgs: 97 active+clean; 672 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T06:51:27.197 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '213c08d8', quiesce took 8.1 seconds. Will release after: 14.450000000000001 2024-07-26T06:51:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:27 smithi164 ceph-mon[36970]: pgmap v432: 97 pgs: 97 active+clean; 672 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T06:51:27.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:27 smithi163 ceph-mon[36873]: pgmap v432: 97 pgs: 97 active+clean; 672 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T06:51:29.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:29 smithi042 ceph-mon[32775]: pgmap v433: 97 pgs: 97 active+clean; 672 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-26T06:51:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:29 smithi164 ceph-mon[36970]: pgmap v433: 97 pgs: 97 active+clean; 672 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-26T06:51:29.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:29 smithi163 ceph-mon[36873]: pgmap v433: 97 pgs: 97 active+clean; 672 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-26T06:51:31.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:31 smithi042 ceph-mon[32775]: pgmap v434: 97 pgs: 97 active+clean; 672 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T06:51:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:31 smithi164 ceph-mon[36970]: pgmap v434: 97 pgs: 97 active+clean; 672 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T06:51:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:31 smithi163 ceph-mon[36873]: pgmap v434: 97 pgs: 97 active+clean; 672 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T06:51:33.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:33 smithi042 ceph-mon[32775]: pgmap v435: 97 pgs: 97 active+clean; 672 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-07-26T06:51:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.449+0000 7fde5660d640 1 -- 172.21.15.42:0/879539659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde501039f0 msgr2=0x7fde5010a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.449+0000 7fde5660d640 1 --2- 172.21.15.42:0/879539659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde501039f0 0x7fde5010a390 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fde48009c90 tx=0x7fde4802a860 comp rx=0 tx=0).stop 2024-07-26T06:51:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.449+0000 7fde5660d640 1 -- 172.21.15.42:0/879539659 shutdown_connections 2024-07-26T06:51:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.449+0000 7fde5660d640 1 --2- 172.21.15.42:0/879539659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde501039f0 0x7fde5010a390 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.449+0000 7fde5660d640 1 --2- 172.21.15.42:0/879539659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde50103050 0x7fde50103460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.449+0000 7fde5660d640 1 --2- 172.21.15.42:0/879539659 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde50101fd0 0x7fde50102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.449+0000 7fde5660d640 1 -- 172.21.15.42:0/879539659 >> 172.21.15.42:0/879539659 conn(0x7fde500fd910 msgr2=0x7fde500ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.449+0000 7fde5660d640 1 -- 172.21.15.42:0/879539659 shutdown_connections 2024-07-26T06:51:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.449+0000 7fde5660d640 1 -- 172.21.15.42:0/879539659 wait complete. 2024-07-26T06:51:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.450+0000 7fde5660d640 1 Processor -- start 2024-07-26T06:51:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.450+0000 7fde5660d640 1 -- start start 2024-07-26T06:51:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.450+0000 7fde5660d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde50101fd0 0x7fde50197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.450+0000 7fde5660d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde50103050 0x7fde50197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.450+0000 7fde5660d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde501039f0 0x7fde50198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.451+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde50103050 0x7fde50197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.451+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde50103050 0x7fde50197dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35062/0 (socket says 172.21.15.42:35062) 2024-07-26T06:51:33.453 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.451+0000 7fde4f7fe640 1 -- 172.21.15.42:0/188817134 learned_addr learned my addr 172.21.15.42:0/188817134 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.451+0000 7fde4ffff640 1 --2- 172.21.15.42:0/188817134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde50101fd0 0x7fde50197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.451+0000 7fde5660d640 1 -- 172.21.15.42:0/188817134 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fde5010cb90 con 0x7fde50103050 2024-07-26T06:51:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.451+0000 7fde5660d640 1 -- 172.21.15.42:0/188817134 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fde5010ca10 con 0x7fde50101fd0 2024-07-26T06:51:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.451+0000 7fde5660d640 1 -- 172.21.15.42:0/188817134 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fde5010cd10 con 0x7fde501039f0 2024-07-26T06:51:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.451+0000 7fde54baa640 1 --2- 172.21.15.42:0/188817134 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde501039f0 0x7fde50198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.451+0000 7fde4f7fe640 1 -- 172.21.15.42:0/188817134 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde501039f0 msgr2=0x7fde50198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.451+0000 7fde4f7fe640 1 --2- 172.21.15.42:0/188817134 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde501039f0 0x7fde50198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.451+0000 7fde4f7fe640 1 -- 172.21.15.42:0/188817134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde50101fd0 msgr2=0x7fde50197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.451+0000 7fde4f7fe640 1 --2- 172.21.15.42:0/188817134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde50101fd0 0x7fde50197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.451+0000 7fde4f7fe640 1 -- 172.21.15.42:0/188817134 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde501991a0 con 0x7fde50103050 2024-07-26T06:51:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.452+0000 7fde4f7fe640 1 --2- 172.21.15.42:0/188817134 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde50103050 0x7fde50197dd0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fde4400d950 tx=0x7fde4400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:33.454 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.452+0000 7fde4d7fa640 1 -- 172.21.15.42:0/188817134 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde44017030 con 0x7fde50103050 2024-07-26T06:51:33.455 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.452+0000 7fde5660d640 1 -- 172.21.15.42:0/188817134 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde50199490 con 0x7fde50103050 2024-07-26T06:51:33.455 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.452+0000 7fde4d7fa640 1 -- 172.21.15.42:0/188817134 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde44004020 con 0x7fde50103050 2024-07-26T06:51:33.455 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.452+0000 7fde4d7fa640 1 -- 172.21.15.42:0/188817134 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde440047a0 con 0x7fde50103050 2024-07-26T06:51:33.455 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.452+0000 7fde5660d640 1 -- 172.21.15.42:0/188817134 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde5019fb80 con 0x7fde50103050 2024-07-26T06:51:33.457 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.456+0000 7fde5d5fe740 1 Processor -- start 2024-07-26T06:51:33.457 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.456+0000 7fde5d5fe740 1 -- start start 2024-07-26T06:51:33.457 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.456+0000 7fde5d5fe740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854ad241b0 0x55854b0c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.457 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.456+0000 7fde5d5fe740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854acfb2f0 0x55854b0c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.457 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.456+0000 7fde5d5fe740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854ad034d0 0x55854b0c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.456+0000 7fde5d5fe740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55854b03c3b0 con 0x55854acfb2f0 2024-07-26T06:51:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.456+0000 7fde5d5fe740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55854b02b8b0 con 0x55854ad034d0 2024-07-26T06:51:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.456+0000 7fde5d5fe740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55854b02f8f0 con 0x55854ad241b0 2024-07-26T06:51:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.456+0000 7fde4ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854acfb2f0 0x55854b0c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.456+0000 7fde4ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854acfb2f0 0x55854b0c91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35068/0 (socket says 172.21.15.42:35068) 2024-07-26T06:51:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.456+0000 7fde4ffff640 1 -- 172.21.15.42:0/1077571097 learned_addr learned my addr 172.21.15.42:0/1077571097 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.457+0000 7fde54baa640 1 --2- 172.21.15.42:0/1077571097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854ad241b0 0x55854b0c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.457+0000 7fde4ffff640 1 -- 172.21.15.42:0/1077571097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854ad241b0 msgr2=0x55854b0c8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.457+0000 7fde4ffff640 1 --2- 172.21.15.42:0/1077571097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854ad241b0 0x55854b0c8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.457+0000 7fde4ffff640 1 -- 172.21.15.42:0/1077571097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854ad034d0 msgr2=0x55854b0c9d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T06:51:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.457+0000 7fde4ffff640 1 --2- 172.21.15.42:0/1077571097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854ad034d0 0x55854b0c9d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.457+0000 7fde4ffff640 1 -- 172.21.15.42:0/1077571097 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55854afd29a0 con 0x55854acfb2f0 2024-07-26T06:51:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.457+0000 7fde4ffff640 1 --2- 172.21.15.42:0/1077571097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854acfb2f0 0x55854b0c91c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fde4000bd10 tx=0x7fde400107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74385dec7093cc2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.457+0000 7fde4d7fa640 1 -- 172.21.15.42:0/188817134 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fde44009a30 con 0x7fde50103050 2024-07-26T06:51:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.458+0000 7fde367fc640 1 -- 172.21.15.42:0/1077571097 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde40004030 con 0x55854acfb2f0 2024-07-26T06:51:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.458+0000 7fde367fc640 1 -- 172.21.15.42:0/1077571097 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde400035f0 con 0x55854acfb2f0 2024-07-26T06:51:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.458+0000 7fde367fc640 1 -- 172.21.15.42:0/1077571097 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde40011770 con 0x55854acfb2f0 2024-07-26T06:51:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.458+0000 7fde4d7fa640 1 --2- 172.21.15.42:0/188817134 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fde380b3c30 0x7fde380b6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.459+0000 7fde54baa640 1 --2- 172.21.15.42:0/188817134 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fde380b3c30 0x7fde380b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.459+0000 7fde4d7fa640 1 -- 172.21.15.42:0/188817134 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fde440e3590 con 0x7fde50103050 2024-07-26T06:51:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.459+0000 7fde5d5fe740 1 -- 172.21.15.42:0/1077571097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854acfb2f0 msgr2=0x55854b0c91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.459+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/1077571097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854acfb2f0 0x55854b0c91c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fde4000bd10 tx=0x7fde400107a0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.459+0000 7fde5d5fe740 1 -- 172.21.15.42:0/1077571097 shutdown_connections 2024-07-26T06:51:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.459+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/1077571097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854ad034d0 0x55854b0c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.459+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/1077571097 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854acfb2f0 0x55854b0c91c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.459+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/1077571097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854ad241b0 0x55854b0c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.459+0000 7fde5d5fe740 1 -- 172.21.15.42:0/1077571097 >> 172.21.15.42:0/1077571097 conn(0x55854ad18300 msgr2=0x55854b0c45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.459+0000 7fde5d5fe740 1 -- 172.21.15.42:0/1077571097 shutdown_connections 2024-07-26T06:51:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.459+0000 7fde54baa640 1 --2- 172.21.15.42:0/188817134 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fde380b3c30 0x7fde380b6140 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fde4802ae50 tx=0x7fde48031000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.460+0000 7fde5d5fe740 1 -- 172.21.15.42:0/1077571097 wait complete. 2024-07-26T06:51:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.460+0000 7fde5d5fe740 1 -- start start 2024-07-26T06:51:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.460+0000 7fde5d5fe740 1 Processor -- start 2024-07-26T06:51:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.461+0000 7fde5d5fe740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854acfb2f0 0x55854b0d2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.461+0000 7fde5d5fe740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854ad034d0 0x55854b0d2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.461+0000 7fde5d5fe740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad241b0 0x55854b0d3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.461+0000 7fde54baa640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad241b0 0x55854b0d3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.461+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854ad034d0 0x55854b0d2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.461+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854ad034d0 0x55854b0d2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:48634/0 (socket says 172.21.15.42:48634) 2024-07-26T06:51:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.461+0000 7fde54baa640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad241b0 0x55854b0d3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35072/0 (socket says 172.21.15.42:35072) 2024-07-26T06:51:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.461+0000 7fde4ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854acfb2f0 0x55854b0d2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:33.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.461+0000 7fde4f7fe640 1 -- 172.21.15.42:0/2536468281 learned_addr learned my addr 172.21.15.42:0/2536468281 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.461+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2536468281 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55854af65ab0 con 0x55854ad241b0 2024-07-26T06:51:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.461+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2536468281 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55854afb3cf0 con 0x55854ad034d0 2024-07-26T06:51:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.461+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2536468281 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55854b0ca830 con 0x55854acfb2f0 2024-07-26T06:51:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.462+0000 7fde4f7fe640 1 -- 172.21.15.42:0/2536468281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854acfb2f0 msgr2=0x55854b0d2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.462+0000 7fde4f7fe640 1 --2- 172.21.15.42:0/2536468281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854acfb2f0 0x55854b0d2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.462+0000 7fde4f7fe640 1 -- 172.21.15.42:0/2536468281 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad241b0 msgr2=0x55854b0d3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.462+0000 7fde4f7fe640 1 --2- 172.21.15.42:0/2536468281 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad241b0 0x55854b0d3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.462+0000 7fde4f7fe640 1 -- 172.21.15.42:0/2536468281 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55854ad8f790 con 0x55854ad034d0 2024-07-26T06:51:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.462+0000 7fde4ffff640 1 --2- 172.21.15.42:0/2536468281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854acfb2f0 0x55854b0d2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:51:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.462+0000 7fde54baa640 1 --2- 172.21.15.42:0/2536468281 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad241b0 0x55854b0d3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:51:33.468 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.462+0000 7fde4f7fe640 1 --2- 172.21.15.42:0/2536468281 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854ad034d0 0x55854b0d2cc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fde440a6540 tx=0x7fde440aa110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ea92f00eff680fc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.462+0000 7fde2f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:51:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.462+0000 7fde5d5fe740 10 client.0 _ll_register_callbacks cb 0x55854b0253e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.462+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2536468281 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde400035f0 con 0x55854ad034d0 2024-07-26T06:51:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.462+0000 7fde2f7fe640 1 -- 172.21.15.42:0/2536468281 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde440aeac0 con 0x55854ad034d0 2024-07-26T06:51:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.462+0000 7fde2f7fe640 1 -- 172.21.15.42:0/2536468281 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fde440041d0 con 0x55854ad034d0 2024-07-26T06:51:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.463+0000 7fde2f7fe640 1 -- 172.21.15.42:0/2536468281 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fde440b3570 con 0x55854ad034d0 2024-07-26T06:51:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.463+0000 7fde5d5fe740 10 client.25175 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.463+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2536468281 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55854ae08620 con 0x55854ad034d0 2024-07-26T06:51:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.463+0000 7fde2f7fe640 1 -- 172.21.15.42:0/2536468281 <== mon.2 v2:172.21.15.163:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fde440a6750 con 0x55854ad034d0 2024-07-26T06:51:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.463+0000 7fde5d5fe740 10 client.25175 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.463+0000 7fde5d5fe740 10 client.25175 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:51:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.463+0000 7fde5d5fe740 20 client.25175 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:33.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.463+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/2536468281 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55854ace2d80 0x55854b0dfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.463+0000 7fde5d5fe740 4 client.25175 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:51:33.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.463+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2536468281 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55854b03e300 con 0x55854ace2d80 2024-07-26T06:51:33.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.464+0000 7fde4ffff640 1 --2- 172.21.15.42:0/2536468281 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55854ace2d80 0x55854b0dfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:33.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.464+0000 7fde2f7fe640 10 client.25175 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:33.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.464+0000 7fde4ffff640 1 --2- 172.21.15.42:0/2536468281 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55854ace2d80 0x55854b0dfc20 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8537558fd304795 server_cookie=4748ae684c42f1c7 in_seq=0 out_seq=0 2024-07-26T06:51:33.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.465+0000 7fde2f7fe640 1 -- 172.21.15.42:0/2536468281 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55854b03e300 con 0x55854ace2d80 2024-07-26T06:51:33.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.465+0000 7fde2f7fe640 10 client.25175 handle_command_reply: tid=0 2024-07-26T06:51:33.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.465+0000 7fde5d5fe740 1 client.25175 shutdown 2024-07-26T06:51:33.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.465+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2536468281 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854ad034d0 msgr2=0x55854b0d2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:33.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.465+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/2536468281 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854ad034d0 0x55854b0d2cc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fde440a6540 tx=0x7fde440aa110 comp rx=0 tx=0).stop 2024-07-26T06:51:33.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.465+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2536468281 shutdown_connections 2024-07-26T06:51:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.465+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/2536468281 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55854ace2d80 0x55854b0dfc20 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.465+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/2536468281 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad241b0 0x55854b0d3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.465+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/2536468281 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854ad034d0 0x55854b0d2cc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.465+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/2536468281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854acfb2f0 0x55854b0d2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.465+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2536468281 >> 172.21.15.42:0/2536468281 conn(0x55854ad18300 msgr2=0x55854b0c2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.465+0000 7fde2f7fe640 0 client.25175 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.465+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2536468281 shutdown_connections 2024-07-26T06:51:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.465+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2536468281 wait complete. 2024-07-26T06:51:33.471 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.466+0000 7fde5d5fe740 20 client.25175 trim_cache size 0 max 16384 2024-07-26T06:51:33.471 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:33 smithi164 ceph-mon[36970]: pgmap v435: 97 pgs: 97 active+clean; 672 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-07-26T06:51:33.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.496+0000 7fde5d5fe740 1 Processor -- start 2024-07-26T06:51:33.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.496+0000 7fde5d5fe740 1 -- start start 2024-07-26T06:51:33.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.497+0000 7fde5d5fe740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854acfb2f0 0x55854b0c8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.497+0000 7fde5d5fe740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad034d0 0x55854b0c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.497+0000 7fde4ffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854acfb2f0 0x55854b0c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:33.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.497+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad034d0 0x55854b0c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:33.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.497+0000 7fde4ffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854acfb2f0 0x55854b0c8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:48638/0 (socket says 172.21.15.42:48638) 2024-07-26T06:51:33.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.497+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad034d0 0x55854b0c8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35088/0 (socket says 172.21.15.42:35088) 2024-07-26T06:51:33.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.497+0000 7fde4ffff640 1 -- 172.21.15.42:0/2986781531 learned_addr learned my addr 172.21.15.42:0/2986781531 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:33.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.497+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/2986781531 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854ad241b0 0x55854b0c2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.497+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2986781531 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fde501a8800 con 0x55854ad034d0 2024-07-26T06:51:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.497+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2986781531 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fde501aa340 con 0x55854acfb2f0 2024-07-26T06:51:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.497+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2986781531 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fde501a8ed0 con 0x55854ad241b0 2024-07-26T06:51:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.497+0000 7fde4ffff640 1 -- 172.21.15.42:0/2986781531 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854ad241b0 msgr2=0x55854b0c2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.498+0000 7fde4ffff640 1 --2- 172.21.15.42:0/2986781531 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854ad241b0 0x55854b0c2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.498+0000 7fde4ffff640 1 -- 172.21.15.42:0/2986781531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad034d0 msgr2=0x55854b0c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.498+0000 7fde4ffff640 1 --2- 172.21.15.42:0/2986781531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad034d0 0x55854b0c8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.498+0000 7fde4ffff640 1 -- 172.21.15.42:0/2986781531 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55854ace2fb0 con 0x55854acfb2f0 2024-07-26T06:51:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.498+0000 7fde4f7fe640 1 --2- 172.21.15.42:0/2986781531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad034d0 0x55854b0c8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:51:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.498+0000 7fde4ffff640 1 --2- 172.21.15.42:0/2986781531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854acfb2f0 0x55854b0c8810 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fde4000b3d0 tx=0x7fde40005870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fbcb8d9bf2665d1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.498+0000 7fde35ffb640 1 -- 172.21.15.42:0/2986781531 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde40003860 con 0x55854acfb2f0 2024-07-26T06:51:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.498+0000 7fde35ffb640 1 -- 172.21.15.42:0/2986781531 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde400054f0 con 0x55854acfb2f0 2024-07-26T06:51:33.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.498+0000 7fde35ffb640 1 -- 172.21.15.42:0/2986781531 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde40026880 con 0x55854acfb2f0 2024-07-26T06:51:33.505 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.499+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2986781531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854acfb2f0 msgr2=0x55854b0c8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:33.505 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.499+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/2986781531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854acfb2f0 0x55854b0c8810 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fde4000b3d0 tx=0x7fde40005870 comp rx=0 tx=0).stop 2024-07-26T06:51:33.505 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.500+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2986781531 shutdown_connections 2024-07-26T06:51:33.505 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.500+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/2986781531 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854ad241b0 0x55854b0c2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.505 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.500+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/2986781531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad034d0 0x55854b0c8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.505 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.500+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/2986781531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854acfb2f0 0x55854b0c8810 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.500+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2986781531 >> 172.21.15.42:0/2986781531 conn(0x55854ad18300 msgr2=0x55854b0df8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.500+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2986781531 shutdown_connections 2024-07-26T06:51:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.500+0000 7fde5d5fe740 1 -- 172.21.15.42:0/2986781531 wait complete. 2024-07-26T06:51:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.500+0000 7fde5d5fe740 1 -- start start 2024-07-26T06:51:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.500+0000 7fde5d5fe740 1 Processor -- start 2024-07-26T06:51:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.501+0000 7fde5d5fe740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854acfb2f0 0x55854b0d9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.501+0000 7fde5d5fe740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad034d0 0x55854b0d9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.501+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad034d0 0x55854b0d9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.501+0000 7fde4ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854acfb2f0 0x55854b0d9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.501+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad034d0 0x55854b0d9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35096/0 (socket says 172.21.15.42:35096) 2024-07-26T06:51:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.501+0000 7fde4f7fe640 1 -- 172.21.15.42:0/1172996338 learned_addr learned my addr 172.21.15.42:0/1172996338 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:33.506 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.501+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/1172996338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854ad241b0 0x55854b0d9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.501+0000 7fde5d5fe740 1 -- 172.21.15.42:0/1172996338 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fde501a9c70 con 0x55854ad034d0 2024-07-26T06:51:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.501+0000 7fde5d5fe740 1 -- 172.21.15.42:0/1172996338 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fde501a95a0 con 0x55854ad241b0 2024-07-26T06:51:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.501+0000 7fde5d5fe740 1 -- 172.21.15.42:0/1172996338 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fde501a67e0 con 0x55854acfb2f0 2024-07-26T06:51:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.501+0000 7fde5d5fe740 10 client.0 _ll_register_callbacks cb 0x7fde440044d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.502+0000 7fde4ffff640 1 -- 172.21.15.42:0/1172996338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854ad241b0 msgr2=0x55854b0d9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.502+0000 7fde4ffff640 1 --2- 172.21.15.42:0/1172996338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854ad241b0 0x55854b0d9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.502+0000 7fde4ffff640 1 -- 172.21.15.42:0/1172996338 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad034d0 msgr2=0x55854b0d9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.502+0000 7fde4ffff640 1 --2- 172.21.15.42:0/1172996338 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad034d0 0x55854b0d9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.502+0000 7fde4ffff640 1 -- 172.21.15.42:0/1172996338 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55854b0d7ff0 con 0x55854acfb2f0 2024-07-26T06:51:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.502+0000 7fde4f7fe640 1 --2- 172.21.15.42:0/1172996338 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad034d0 0x55854b0d9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:51:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.502+0000 7fde4ffff640 1 --2- 172.21.15.42:0/1172996338 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854acfb2f0 0x55854b0d9330 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55854b0dc700 tx=0x7fde400052e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49132b0aee9263bc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.502+0000 7fde2f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-07-26T06:51:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.502+0000 7fde2f7fe640 1 -- 172.21.15.42:0/1172996338 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde40036070 con 0x55854acfb2f0 2024-07-26T06:51:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.502+0000 7fde5d5fe740 1 -- 172.21.15.42:0/1172996338 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde400054f0 con 0x55854acfb2f0 2024-07-26T06:51:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.502+0000 7fde2f7fe640 1 -- 172.21.15.42:0/1172996338 <== mon.1 v2:172.21.15.164:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fde40003c70 con 0x55854acfb2f0 2024-07-26T06:51:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.502+0000 7fde2f7fe640 1 -- 172.21.15.42:0/1172996338 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fde40042050 con 0x55854acfb2f0 2024-07-26T06:51:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.502+0000 7fde5d5fe740 10 client.25231 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.503+0000 7fde5d5fe740 1 -- 172.21.15.42:0/1172996338 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55854b14ef20 con 0x55854acfb2f0 2024-07-26T06:51:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.503+0000 7fde2f7fe640 1 -- 172.21.15.42:0/1172996338 <== mon.1 v2:172.21.15.164:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde400054f0 con 0x55854acfb2f0 2024-07-26T06:51:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.503+0000 7fde2f7fe640 1 -- 172.21.15.42:0/1172996338 <== mon.1 v2:172.21.15.164:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fde400115d0 con 0x55854acfb2f0 2024-07-26T06:51:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.503+0000 7fde5d5fe740 10 client.25231 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.503+0000 7fde5d5fe740 10 client.25231 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:51:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.503+0000 7fde5d5fe740 20 client.25231 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:33.508 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.503+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/1172996338 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55854acb81c0 0x55854b1516f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.503+0000 7fde5d5fe740 4 client.25231 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:51:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.503+0000 7fde5d5fe740 1 -- 172.21.15.42:0/1172996338 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55854b039da0 con 0x55854acb81c0 2024-07-26T06:51:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.503+0000 7fde4f7fe640 1 --2- 172.21.15.42:0/1172996338 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55854acb81c0 0x55854b1516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.504+0000 7fde2f7fe640 10 client.25231 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:33.509 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.504+0000 7fde4f7fe640 1 --2- 172.21.15.42:0/1172996338 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55854acb81c0 0x55854b1516f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddcee6bffde66782 server_cookie=256ddcd489c33c9e in_seq=0 out_seq=0 2024-07-26T06:51:33.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:33 smithi163 ceph-mon[36873]: pgmap v435: 97 pgs: 97 active+clean; 672 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-07-26T06:51:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.512+0000 7fde2f7fe640 1 -- 172.21.15.42:0/1172996338 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55854b039da0 con 0x55854acb81c0 2024-07-26T06:51:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.512+0000 7fde2f7fe640 10 client.25231 handle_command_reply: tid=0 2024-07-26T06:51:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.512+0000 7fde5d5fe740 1 client.25231 shutdown 2024-07-26T06:51:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.512+0000 7fde5d5fe740 1 -- 172.21.15.42:0/1172996338 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854acfb2f0 msgr2=0x55854b0d9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.512+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/1172996338 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854acfb2f0 0x55854b0d9330 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55854b0dc700 tx=0x7fde400052e0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.512+0000 7fde5d5fe740 1 -- 172.21.15.42:0/1172996338 shutdown_connections 2024-07-26T06:51:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.512+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/1172996338 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55854acb81c0 0x55854b1516f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.512+0000 7fde2f7fe640 0 client.25231 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:33.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.512+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/1172996338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55854ad241b0 0x55854b0d9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.512+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/1172996338 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55854ad034d0 0x55854b0d9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.512+0000 7fde5d5fe740 1 --2- 172.21.15.42:0/1172996338 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55854acfb2f0 0x55854b0d9330 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.512+0000 7fde5d5fe740 1 -- 172.21.15.42:0/1172996338 >> 172.21.15.42:0/1172996338 conn(0x55854ad18300 msgr2=0x55854b0c8740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:33.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.512+0000 7fde5d5fe740 1 -- 172.21.15.42:0/1172996338 shutdown_connections 2024-07-26T06:51:33.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.512+0000 7fde5d5fe740 1 -- 172.21.15.42:0/1172996338 wait complete. 2024-07-26T06:51:33.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.512+0000 7fde5d5fe740 20 client.25231 trim_cache size 0 max 16384 2024-07-26T06:51:33.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:51:33.515 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:51:33.515 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:51:33.515 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:51:33.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.513+0000 7fde5660d640 1 -- 172.21.15.42:0/188817134 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fde380b3c30 msgr2=0x7fde380b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:33.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.513+0000 7fde5660d640 1 --2- 172.21.15.42:0/188817134 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fde380b3c30 0x7fde380b6140 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fde4802ae50 tx=0x7fde48031000 comp rx=0 tx=0).stop 2024-07-26T06:51:33.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.513+0000 7fde5660d640 1 -- 172.21.15.42:0/188817134 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde50103050 msgr2=0x7fde50197dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:33.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.513+0000 7fde5660d640 1 --2- 172.21.15.42:0/188817134 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde50103050 0x7fde50197dd0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fde4400d950 tx=0x7fde4400de10 comp rx=0 tx=0).stop 2024-07-26T06:51:33.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.513+0000 7fde5660d640 1 -- 172.21.15.42:0/188817134 shutdown_connections 2024-07-26T06:51:33.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.513+0000 7fde5660d640 1 --2- 172.21.15.42:0/188817134 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fde380b3c30 0x7fde380b6140 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.513+0000 7fde5660d640 1 --2- 172.21.15.42:0/188817134 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde501039f0 0x7fde50198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.514+0000 7fde5660d640 1 --2- 172.21.15.42:0/188817134 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fde50103050 0x7fde50197dd0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.514+0000 7fde5660d640 1 --2- 172.21.15.42:0/188817134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fde50101fd0 0x7fde50197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:33.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.514+0000 7fde5660d640 1 -- 172.21.15.42:0/188817134 >> 172.21.15.42:0/188817134 conn(0x7fde500fd910 msgr2=0x7fde50106350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:33.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.514+0000 7fde5660d640 1 -- 172.21.15.42:0/188817134 shutdown_connections 2024-07-26T06:51:33.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:33.514+0000 7fde5660d640 1 -- 172.21.15.42:0/188817134 wait complete. 2024-07-26T06:51:35.083 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:35 smithi042 ceph-mon[32775]: pgmap v436: 97 pgs: 97 active+clean; 672 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-26T06:51:35.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:35 smithi164 ceph-mon[36970]: pgmap v436: 97 pgs: 97 active+clean; 672 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-26T06:51:35.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:35 smithi163 ceph-mon[36873]: pgmap v436: 97 pgs: 97 active+clean; 672 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-26T06:51:37.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:37 smithi042 ceph-mon[32775]: pgmap v437: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-07-26T06:51:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:37 smithi164 ceph-mon[36970]: pgmap v437: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-07-26T06:51:37.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:37 smithi163 ceph-mon[36873]: pgmap v437: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-07-26T06:51:39.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:39 smithi042 ceph-mon[32775]: pgmap v438: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 11 op/s 2024-07-26T06:51:39.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:39 smithi164 ceph-mon[36970]: pgmap v438: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 11 op/s 2024-07-26T06:51:39.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:39 smithi163 ceph-mon[36873]: pgmap v438: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 11 op/s 2024-07-26T06:51:41.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:41 smithi042 ceph-mon[32775]: pgmap v439: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 853 B/s wr, 13 op/s 2024-07-26T06:51:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:41 smithi164 ceph-mon[36970]: pgmap v439: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 853 B/s wr, 13 op/s 2024-07-26T06:51:41.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:41 smithi163 ceph-mon[36873]: pgmap v439: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 853 B/s wr, 13 op/s 2024-07-26T06:51:41.649 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:51:42.058 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:51:43.346 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:43 smithi042 ceph-mon[32775]: pgmap v440: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 10 op/s 2024-07-26T06:51:43.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:43 smithi164 ceph-mon[36970]: pgmap v440: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 10 op/s 2024-07-26T06:51:43.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:43 smithi163 ceph-mon[36873]: pgmap v440: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 10 op/s 2024-07-26T06:51:44.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.262+0000 7f5841644640 1 -- 172.21.15.42:0/3201931702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f583c107e10 msgr2=0x7f583c108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:44.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.262+0000 7f5841644640 1 --2- 172.21.15.42:0/3201931702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f583c107e10 0x7f583c108220 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f582c009900 tx=0x7f582c02a410 comp rx=0 tx=0).stop 2024-07-26T06:51:44.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.262+0000 7f5841644640 1 -- 172.21.15.42:0/3201931702 shutdown_connections 2024-07-26T06:51:44.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.262+0000 7f5841644640 1 --2- 172.21.15.42:0/3201931702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f583c108760 0x7f583c10ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:44.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.262+0000 7f5841644640 1 --2- 172.21.15.42:0/3201931702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f583c107e10 0x7f583c108220 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:44.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.262+0000 7f5841644640 1 --2- 172.21.15.42:0/3201931702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f583c1024d0 0x7f583c1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:44.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.262+0000 7f5841644640 1 -- 172.21.15.42:0/3201931702 >> 172.21.15.42:0/3201931702 conn(0x7f583c079210 msgr2=0x7f583c079620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:44.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.262+0000 7f5841644640 1 -- 172.21.15.42:0/3201931702 shutdown_connections 2024-07-26T06:51:44.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.262+0000 7f5841644640 1 -- 172.21.15.42:0/3201931702 wait complete. 2024-07-26T06:51:44.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.263+0000 7f5841644640 1 Processor -- start 2024-07-26T06:51:44.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.263+0000 7f5841644640 1 -- start start 2024-07-26T06:51:44.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.263+0000 7f5841644640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f583c1024d0 0x7f583c19c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:44.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.263+0000 7f5841644640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f583c107e10 0x7f583c19c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:44.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.263+0000 7f5841644640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f583c108760 0x7f583c1a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:44.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.263+0000 7f5841644640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f583c112390 con 0x7f583c108760 2024-07-26T06:51:44.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.263+0000 7f5841644640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f583c112210 con 0x7f583c1024d0 2024-07-26T06:51:44.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.263+0000 7f5841644640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f583c112510 con 0x7f583c107e10 2024-07-26T06:51:44.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.263+0000 7f583b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f583c108760 0x7f583c1a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:44.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.263+0000 7f583b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f583c108760 0x7f583c1a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49022/0 (socket says 172.21.15.42:49022) 2024-07-26T06:51:44.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.263+0000 7f583b7fe640 1 -- 172.21.15.42:0/100854138 learned_addr learned my addr 172.21.15.42:0/100854138 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:44.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.263+0000 7f583a7fc640 1 --2- 172.21.15.42:0/100854138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f583c107e10 0x7f583c19c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:44.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.264+0000 7f583affd640 1 --2- 172.21.15.42:0/100854138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f583c1024d0 0x7f583c19c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:44.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.264+0000 7f583b7fe640 1 -- 172.21.15.42:0/100854138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f583c107e10 msgr2=0x7f583c19c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:44.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.264+0000 7f583b7fe640 1 --2- 172.21.15.42:0/100854138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f583c107e10 0x7f583c19c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:44.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.264+0000 7f583b7fe640 1 -- 172.21.15.42:0/100854138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f583c1024d0 msgr2=0x7f583c19c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:44.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.264+0000 7f583b7fe640 1 --2- 172.21.15.42:0/100854138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f583c1024d0 0x7f583c19c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:44.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.264+0000 7f583b7fe640 1 -- 172.21.15.42:0/100854138 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f583c1a2060 con 0x7f583c108760 2024-07-26T06:51:44.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.264+0000 7f583b7fe640 1 --2- 172.21.15.42:0/100854138 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f583c108760 0x7f583c1a1930 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5834009fd0 tx=0x7f583400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:44.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.264+0000 7f5833fff640 1 -- 172.21.15.42:0/100854138 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5834017070 con 0x7f583c108760 2024-07-26T06:51:44.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.264+0000 7f5841644640 1 -- 172.21.15.42:0/100854138 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f583c1a2350 con 0x7f583c108760 2024-07-26T06:51:44.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.264+0000 7f5833fff640 1 -- 172.21.15.42:0/100854138 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5834003e20 con 0x7f583c108760 2024-07-26T06:51:44.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.264+0000 7f5833fff640 1 -- 172.21.15.42:0/100854138 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5834010460 con 0x7f583c108760 2024-07-26T06:51:44.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.264+0000 7f5841644640 1 -- 172.21.15.42:0/100854138 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f583c1a8ab0 con 0x7f583c108760 2024-07-26T06:51:44.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.265+0000 7f5841644640 1 -- 172.21.15.42:0/100854138 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f583c103750 con 0x7f583c108760 2024-07-26T06:51:44.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.267+0000 7f5833fff640 1 -- 172.21.15.42:0/100854138 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f5834003970 con 0x7f583c108760 2024-07-26T06:51:44.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.268+0000 7f5833fff640 1 --2- 172.21.15.42:0/100854138 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f58280b3bd0 0x7f58280b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:44.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.268+0000 7f583affd640 1 --2- 172.21.15.42:0/100854138 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f58280b3bd0 0x7f58280b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:44.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.268+0000 7f5833fff640 1 -- 172.21.15.42:0/100854138 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f58340e3cf0 con 0x7f583c108760 2024-07-26T06:51:44.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.269+0000 7f583affd640 1 --2- 172.21.15.42:0/100854138 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f58280b3bd0 0x7f58280b60e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f582400bbe0 tx=0x7f582400b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:44.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.274+0000 7f5833fff640 1 -- 172.21.15.42:0/100854138 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f58340aacd0 con 0x7f583c108760 2024-07-26T06:51:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.495+0000 7f5841644640 1 -- 172.21.15.42:0/100854138 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f583c1028e0 con 0x7f583c108760 2024-07-26T06:51:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.495+0000 7f5833fff640 1 -- 172.21.15.42:0/100854138 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11978 (secure 0 0 0) 0x7f58340af220 con 0x7f583c108760 2024-07-26T06:51:44.496 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:51:44.496 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":29,"btime":"2024-07-26T06:50:27:493008+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:50:26.696932+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24620,24601,24593,24583,24578]},"id":1}]} 2024-07-26T06:51:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 29 2024-07-26T06:51:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.497+0000 7f5841644640 1 -- 172.21.15.42:0/100854138 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f58280b3bd0 msgr2=0x7f58280b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.497+0000 7f5841644640 1 --2- 172.21.15.42:0/100854138 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f58280b3bd0 0x7f58280b60e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f582400bbe0 tx=0x7f582400b390 comp rx=0 tx=0).stop 2024-07-26T06:51:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.497+0000 7f5841644640 1 -- 172.21.15.42:0/100854138 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f583c108760 msgr2=0x7f583c1a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.497+0000 7f5841644640 1 --2- 172.21.15.42:0/100854138 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f583c108760 0x7f583c1a1930 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5834009fd0 tx=0x7f583400b260 comp rx=0 tx=0).stop 2024-07-26T06:51:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.498+0000 7f5841644640 1 -- 172.21.15.42:0/100854138 shutdown_connections 2024-07-26T06:51:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.498+0000 7f5841644640 1 --2- 172.21.15.42:0/100854138 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f58280b3bd0 0x7f58280b60e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.498+0000 7f5841644640 1 --2- 172.21.15.42:0/100854138 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f583c108760 0x7f583c1a1930 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.498+0000 7f5841644640 1 --2- 172.21.15.42:0/100854138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f583c107e10 0x7f583c19c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.498+0000 7f5841644640 1 --2- 172.21.15.42:0/100854138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f583c1024d0 0x7f583c19c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.498+0000 7f5841644640 1 -- 172.21.15.42:0/100854138 >> 172.21.15.42:0/100854138 conn(0x7f583c079210 msgr2=0x7f583c104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.498+0000 7f5841644640 1 -- 172.21.15.42:0/100854138 shutdown_connections 2024-07-26T06:51:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:44.498+0000 7f5841644640 1 -- 172.21.15.42:0/100854138 wait complete. 2024-07-26T06:51:45.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:45 smithi042 ceph-mon[32775]: pgmap v441: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 11 op/s 2024-07-26T06:51:45.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:45 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/100854138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:51:45.399 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24620 quiesce db --set-id 213c08d8 --release --await' 2024-07-26T06:51:45.399 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.24620 quiesce db --set-id 213c08d8 --release --await 2024-07-26T06:51:45.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:45 smithi164 ceph-mon[36970]: pgmap v441: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 11 op/s 2024-07-26T06:51:45.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:45 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/100854138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:51:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:45 smithi163 ceph-mon[36873]: pgmap v441: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 11 op/s 2024-07-26T06:51:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:45 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/100854138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:51:45.680 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:51:46.113 INFO:tasks.workunit.client.0.smithi042.stderr:Updating files: 87% (11155/12697) Updating files: 88% (11174/12697) Updating files: 89% (11301/12697) Updating files: 90% (11428/12697) Updating files: 91% (11555/12697) Updating files: 92% (11682/12697) Updating files: 93% (11809/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-07-26T06:51:46.363 INFO:tasks.workunit.client.0.smithi042.stderr:Note: switching to 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'. 2024-07-26T06:51:46.363 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-07-26T06:51:46.363 INFO:tasks.workunit.client.0.smithi042.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-07-26T06:51:46.363 INFO:tasks.workunit.client.0.smithi042.stderr:changes and commit them, and you can discard any commits you make in this 2024-07-26T06:51:46.363 INFO:tasks.workunit.client.0.smithi042.stderr:state without impacting any branches by switching back to a branch. 2024-07-26T06:51:46.363 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-07-26T06:51:46.363 INFO:tasks.workunit.client.0.smithi042.stderr:If you want to create a new branch to retain commits you create, you may 2024-07-26T06:51:46.363 INFO:tasks.workunit.client.0.smithi042.stderr:do so (now or later) by using -c with the switch command. Example: 2024-07-26T06:51:46.364 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-07-26T06:51:46.364 INFO:tasks.workunit.client.0.smithi042.stderr: git switch -c 2024-07-26T06:51:46.364 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-07-26T06:51:46.364 INFO:tasks.workunit.client.0.smithi042.stderr:Or undo this operation with: 2024-07-26T06:51:46.364 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-07-26T06:51:46.364 INFO:tasks.workunit.client.0.smithi042.stderr: git switch - 2024-07-26T06:51:46.364 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-07-26T06:51:46.364 INFO:tasks.workunit.client.0.smithi042.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-07-26T06:51:46.364 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-07-26T06:51:46.364 INFO:tasks.workunit.client.0.smithi042.stderr:HEAD is now at e8f88bb1fe0 Merge PR #54435 into wip-vshankar-testing-20240725.121331-debug 2024-07-26T06:51:46.366 DEBUG:teuthology.orchestra.run.smithi042:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-07-26T06:51:46.445 INFO:tasks.workunit.client.0.smithi042.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-07-26T06:51:46.448 INFO:tasks.workunit.client.0.smithi042.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-26T06:51:46.448 INFO:tasks.workunit.client.0.smithi042.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-07-26T06:51:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.619+0000 7f0fc89c4640 1 -- 172.21.15.42:0/2979136921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0fc4103a20 msgr2=0x7f0fc410a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.619+0000 7f0fc89c4640 1 --2- 172.21.15.42:0/2979136921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0fc4103a20 0x7f0fc410a3c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009ca0 tx=0x7f0fbc02a8c0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.619+0000 7f0fc89c4640 1 -- 172.21.15.42:0/2979136921 shutdown_connections 2024-07-26T06:51:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.619+0000 7f0fc89c4640 1 --2- 172.21.15.42:0/2979136921 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0fc4103a20 0x7f0fc410a3c0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.619+0000 7f0fc89c4640 1 --2- 172.21.15.42:0/2979136921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fc4103080 0x7f0fc4103490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.619+0000 7f0fc89c4640 1 --2- 172.21.15.42:0/2979136921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fc4102000 0x7f0fc4102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.619+0000 7f0fc89c4640 1 -- 172.21.15.42:0/2979136921 >> 172.21.15.42:0/2979136921 conn(0x7f0fc40fd960 msgr2=0x7f0fc40ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.619+0000 7f0fc89c4640 1 -- 172.21.15.42:0/2979136921 shutdown_connections 2024-07-26T06:51:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.619+0000 7f0fc89c4640 1 -- 172.21.15.42:0/2979136921 wait complete. 2024-07-26T06:51:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.620+0000 7f0fc89c4640 1 Processor -- start 2024-07-26T06:51:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.620+0000 7f0fc89c4640 1 -- start start 2024-07-26T06:51:46.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.620+0000 7f0fc89c4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fc4102000 0x7f0fc406bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.620+0000 7f0fc89c4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fc4103080 0x7f0fc419e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.620+0000 7f0fc89c4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0fc4103a20 0x7f0fc41a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.620+0000 7f0fc89c4640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc410dc80 con 0x7f0fc4103a20 2024-07-26T06:51:46.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.620+0000 7f0fc89c4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc410db00 con 0x7f0fc4102000 2024-07-26T06:51:46.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.620+0000 7f0fc89c4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc410de00 con 0x7f0fc4103080 2024-07-26T06:51:46.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.621+0000 7f0fc2d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0fc4103a20 0x7f0fc41a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:46.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.621+0000 7f0fc2d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0fc4103a20 0x7f0fc41a0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49054/0 (socket says 172.21.15.42:49054) 2024-07-26T06:51:46.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.621+0000 7f0fc2d9d640 1 -- 172.21.15.42:0/3696344254 learned_addr learned my addr 172.21.15.42:0/3696344254 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:46.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.621+0000 7f0fc259c640 1 --2- 172.21.15.42:0/3696344254 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fc4102000 0x7f0fc406bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:46.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.621+0000 7f0fc1d9b640 1 --2- 172.21.15.42:0/3696344254 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fc4103080 0x7f0fc419e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:46.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.621+0000 7f0fc2d9d640 1 -- 172.21.15.42:0/3696344254 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fc4103080 msgr2=0x7f0fc419e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:46.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.621+0000 7f0fc2d9d640 1 --2- 172.21.15.42:0/3696344254 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fc4103080 0x7f0fc419e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.621+0000 7f0fc2d9d640 1 -- 172.21.15.42:0/3696344254 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fc4102000 msgr2=0x7f0fc406bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.621+0000 7f0fc2d9d640 1 --2- 172.21.15.42:0/3696344254 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fc4102000 0x7f0fc406bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.621+0000 7f0fc2d9d640 1 -- 172.21.15.42:0/3696344254 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc41a1460 con 0x7f0fc4103a20 2024-07-26T06:51:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.621+0000 7f0fc2d9d640 1 --2- 172.21.15.42:0/3696344254 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0fc4103a20 0x7f0fc41a0c40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0097b0 tx=0x7f0fbc003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.621+0000 7f0fbb7fe640 1 -- 172.21.15.42:0/3696344254 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc004240 con 0x7f0fc4103a20 2024-07-26T06:51:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.621+0000 7f0fbb7fe640 1 -- 172.21.15.42:0/3696344254 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fbc0317c0 con 0x7f0fc4103a20 2024-07-26T06:51:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.621+0000 7f0fc89c4640 1 -- 172.21.15.42:0/3696344254 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc41a16f0 con 0x7f0fc4103a20 2024-07-26T06:51:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.621+0000 7f0fbb7fe640 1 -- 172.21.15.42:0/3696344254 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc02fc30 con 0x7f0fc4103a20 2024-07-26T06:51:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.621+0000 7f0fc89c4640 1 -- 172.21.15.42:0/3696344254 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc41a1c40 con 0x7f0fc4103a20 2024-07-26T06:51:46.631 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.624+0000 7f0fbb7fe640 1 -- 172.21.15.42:0/3696344254 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f0fbc0043e0 con 0x7f0fc4103a20 2024-07-26T06:51:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.624+0000 7f0fcf9b2740 1 Processor -- start 2024-07-26T06:51:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.624+0000 7f0fcf9b2740 1 -- start start 2024-07-26T06:51:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fcf9b2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314d11b0 0x561a31876180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fcf9b2740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314a82f0 0x561a318767f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fcf9b2740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314b04d0 0x561a31877340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fcf9b2740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x561a317e93b0 con 0x561a314a82f0 2024-07-26T06:51:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fcf9b2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561a317d88b0 con 0x561a314d11b0 2024-07-26T06:51:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fcf9b2740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x561a317dc8f0 con 0x561a314b04d0 2024-07-26T06:51:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fc259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314a82f0 0x561a318767f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fc259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314a82f0 0x561a318767f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49064/0 (socket says 172.21.15.42:49064) 2024-07-26T06:51:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fc259c640 1 -- 172.21.15.42:0/1663785748 learned_addr learned my addr 172.21.15.42:0/1663785748 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:46.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fc1d9b640 1 --2- 172.21.15.42:0/1663785748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314b04d0 0x561a31877340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fc1d9b640 1 --2- 172.21.15.42:0/1663785748 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314d11b0 0x561a31876180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fc259c640 1 -- 172.21.15.42:0/1663785748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314b04d0 msgr2=0x561a31877340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fbb7fe640 1 --2- 172.21.15.42:0/3696344254 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0fb00b3bd0 0x7f0fb00b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fc259c640 1 --2- 172.21.15.42:0/1663785748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314b04d0 0x561a31877340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fc259c640 1 -- 172.21.15.42:0/1663785748 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314d11b0 msgr2=0x561a31876180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fc259c640 1 --2- 172.21.15.42:0/1663785748 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314d11b0 0x561a31876180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fc259c640 1 -- 172.21.15.42:0/1663785748 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a3177f9a0 con 0x561a314a82f0 2024-07-26T06:51:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fc2d9d640 1 --2- 172.21.15.42:0/3696344254 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0fb00b3bd0 0x7f0fb00b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:46.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fc259c640 1 --2- 172.21.15.42:0/1663785748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314a82f0 0x561a318767f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0fb400bd10 tx=0x7f0fb40107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3d82a34da4d0c7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.625+0000 7f0fbb7fe640 1 -- 172.21.15.42:0/3696344254 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f0fbc1053f0 con 0x7f0fc4103a20 2024-07-26T06:51:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.626+0000 7f0fb8ff9640 1 -- 172.21.15.42:0/1663785748 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4004030 con 0x561a314a82f0 2024-07-26T06:51:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.626+0000 7f0fb8ff9640 1 -- 172.21.15.42:0/1663785748 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fb40035f0 con 0x561a314a82f0 2024-07-26T06:51:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.626+0000 7f0fb8ff9640 1 -- 172.21.15.42:0/1663785748 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb40116a0 con 0x561a314a82f0 2024-07-26T06:51:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.626+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/1663785748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314a82f0 msgr2=0x561a318767f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.626+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/1663785748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314a82f0 0x561a318767f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0fb400bd10 tx=0x7f0fb40107a0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.626+0000 7f0fc2d9d640 1 --2- 172.21.15.42:0/3696344254 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0fb00b3bd0 0x7f0fb00b60e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0c82b0 tx=0x7f0fbc0c0b00 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.627+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/1663785748 shutdown_connections 2024-07-26T06:51:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.627+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/1663785748 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314b04d0 0x561a31877340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.627+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/1663785748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314a82f0 0x561a318767f0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.627+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/1663785748 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314d11b0 0x561a31876180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.627+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/1663785748 >> 172.21.15.42:0/1663785748 conn(0x561a314c5300 msgr2=0x561a31871c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.627+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/1663785748 shutdown_connections 2024-07-26T06:51:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.627+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/1663785748 wait complete. 2024-07-26T06:51:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.627+0000 7f0fcf9b2740 1 -- start start 2024-07-26T06:51:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.627+0000 7f0fcf9b2740 1 Processor -- start 2024-07-26T06:51:46.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.627+0000 7f0fcf9b2740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314a82f0 0x561a3187fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.627+0000 7f0fcf9b2740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314b04d0 0x561a31880200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.627+0000 7f0fcf9b2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314d11b0 0x561a31880d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.627+0000 7f0fcf9b2740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x561a31712ab0 con 0x561a314b04d0 2024-07-26T06:51:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.627+0000 7f0fcf9b2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561a31760cf0 con 0x561a314d11b0 2024-07-26T06:51:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.627+0000 7f0fcf9b2740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x561a31877e60 con 0x561a314a82f0 2024-07-26T06:51:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.627+0000 7f0fcf9b2740 10 client.0 _ll_register_callbacks cb 0x561a317d29f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fc259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314b04d0 0x561a31880200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fc259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314b04d0 0x561a31880200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49066/0 (socket says 172.21.15.42:49066) 2024-07-26T06:51:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fc259c640 1 -- 172.21.15.42:0/1818842486 learned_addr learned my addr 172.21.15.42:0/1818842486 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fc1d9b640 1 --2- 172.21.15.42:0/1818842486 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314d11b0 0x561a31880d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fc1d9b640 1 --2- 172.21.15.42:0/1818842486 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314a82f0 0x561a3187fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:46.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fc259c640 1 -- 172.21.15.42:0/1818842486 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314a82f0 msgr2=0x561a3187fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:46.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fc259c640 1 --2- 172.21.15.42:0/1818842486 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314a82f0 0x561a3187fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fc259c640 1 -- 172.21.15.42:0/1818842486 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314d11b0 msgr2=0x561a31880d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:46.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fc259c640 1 --2- 172.21.15.42:0/1818842486 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314d11b0 0x561a31880d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fc259c640 1 -- 172.21.15.42:0/1818842486 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a3153c790 con 0x561a314b04d0 2024-07-26T06:51:46.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fc1d9b640 1 --2- 172.21.15.42:0/1818842486 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314d11b0 0x561a31880d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:51:46.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fc259c640 1 --2- 172.21.15.42:0/1818842486 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314b04d0 0x561a31880200 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4006010 tx=0x7f0fb4004230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a43502e397cff5e3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:46.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fa9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:51:46.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fa9ffb640 1 -- 172.21.15.42:0/1818842486 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4029060 con 0x561a314b04d0 2024-07-26T06:51:46.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fa9ffb640 1 -- 172.21.15.42:0/1818842486 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f0fb40037a0 con 0x561a314b04d0 2024-07-26T06:51:46.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/1818842486 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fb40035f0 con 0x561a314b04d0 2024-07-26T06:51:46.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fa9ffb640 1 -- 172.21.15.42:0/1818842486 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0fb40039a0 con 0x561a314b04d0 2024-07-26T06:51:46.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fcf9b2740 10 client.15573 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.628+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/1818842486 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561a315b5620 con 0x561a314b04d0 2024-07-26T06:51:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.629+0000 7f0fa9ffb640 1 -- 172.21.15.42:0/1818842486 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fb40035f0 con 0x561a314b04d0 2024-07-26T06:51:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.629+0000 7f0fa9ffb640 1 -- 172.21.15.42:0/1818842486 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0fb40048d0 con 0x561a314b04d0 2024-07-26T06:51:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.629+0000 7f0fcf9b2740 10 client.15573 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.629+0000 7f0fcf9b2740 10 client.15573 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:51:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.629+0000 7f0fcf9b2740 20 client.15573 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.629+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/1818842486 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561a3148fd80 0x561a3188cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.629+0000 7f0fcf9b2740 4 client.15573 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:51:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.629+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/1818842486 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a317eb300 con 0x561a3148fd80 2024-07-26T06:51:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.629+0000 7f0fc1d9b640 1 --2- 172.21.15.42:0/1818842486 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561a3148fd80 0x561a3188cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.630+0000 7f0fc1d9b640 1 --2- 172.21.15.42:0/1818842486 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561a3148fd80 0x561a3188cf60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4cc1822135c1ca7 server_cookie=53663e33ed63f32d in_seq=0 out_seq=0 2024-07-26T06:51:46.638 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.630+0000 7f0fa9ffb640 10 client.15573 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.630+0000 7f0fa9ffb640 1 -- 172.21.15.42:0/1818842486 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561a317eb300 con 0x561a3148fd80 2024-07-26T06:51:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.630+0000 7f0fa9ffb640 10 client.15573 handle_command_reply: tid=0 2024-07-26T06:51:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.630+0000 7f0fcf9b2740 1 client.15573 shutdown 2024-07-26T06:51:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.630+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/1818842486 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314b04d0 msgr2=0x561a31880200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.630+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/1818842486 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314b04d0 0x561a31880200 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4006010 tx=0x7f0fb4004230 comp rx=0 tx=0).stop 2024-07-26T06:51:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.631+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/1818842486 shutdown_connections 2024-07-26T06:51:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.631+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/1818842486 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561a3148fd80 0x561a3188cf60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.631+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/1818842486 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314d11b0 0x561a31880d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.631+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/1818842486 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314b04d0 0x561a31880200 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.631+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/1818842486 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314a82f0 0x561a3187fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.631+0000 7f0fa9ffb640 0 client.15573 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.631+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/1818842486 >> 172.21.15.42:0/1818842486 conn(0x561a314c5300 msgr2=0x561a318771f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.631+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/1818842486 shutdown_connections 2024-07-26T06:51:46.639 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.631+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/1818842486 wait complete. 2024-07-26T06:51:46.640 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.631+0000 7f0fcf9b2740 20 client.15573 trim_cache size 0 max 16384 2024-07-26T06:51:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.646+0000 7f0fcf9b2740 1 Processor -- start 2024-07-26T06:51:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.646+0000 7f0fcf9b2740 1 -- start start 2024-07-26T06:51:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.646+0000 7f0fcf9b2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314a82f0 0x561a3187fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.646+0000 7f0fcf9b2740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314b04d0 0x561a31887c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.646+0000 7f0fcf9b2740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314d11b0 0x561a31880080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.646+0000 7f0fcf9b2740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc41aaa60 con 0x561a314b04d0 2024-07-26T06:51:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.646+0000 7f0fcf9b2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc41ac5a0 con 0x561a314a82f0 2024-07-26T06:51:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.646+0000 7f0fcf9b2740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc41ab130 con 0x561a314d11b0 2024-07-26T06:51:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.646+0000 7f0fc259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314b04d0 0x561a31887c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.646+0000 7f0fc259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314b04d0 0x561a31887c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49074/0 (socket says 172.21.15.42:49074) 2024-07-26T06:51:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.646+0000 7f0fc259c640 1 -- 172.21.15.42:0/2838899044 learned_addr learned my addr 172.21.15.42:0/2838899044 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.646+0000 7f0fc1d9b640 1 --2- 172.21.15.42:0/2838899044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314a82f0 0x561a3187fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.646+0000 7f0fc259c640 1 -- 172.21.15.42:0/2838899044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314d11b0 msgr2=0x561a31880080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.646+0000 7f0fc259c640 1 --2- 172.21.15.42:0/2838899044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314d11b0 0x561a31880080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.646+0000 7f0fc259c640 1 -- 172.21.15.42:0/2838899044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314a82f0 msgr2=0x561a3187fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.646+0000 7f0fc259c640 1 --2- 172.21.15.42:0/2838899044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314a82f0 0x561a3187fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.646+0000 7f0fc259c640 1 -- 172.21.15.42:0/2838899044 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a3187dcc0 con 0x561a314b04d0 2024-07-26T06:51:46.649 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.647+0000 7f0fc259c640 1 --2- 172.21.15.42:0/2838899044 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314b04d0 0x561a31887c50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40042a0 tx=0x7f0fb4005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f544cb9e9a18c509 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.647+0000 7f0fabfff640 1 -- 172.21.15.42:0/2838899044 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4029070 con 0x561a314b04d0 2024-07-26T06:51:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.647+0000 7f0fabfff640 1 -- 172.21.15.42:0/2838899044 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fb4011440 con 0x561a314b04d0 2024-07-26T06:51:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.647+0000 7f0fabfff640 1 -- 172.21.15.42:0/2838899044 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4011830 con 0x561a314b04d0 2024-07-26T06:51:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.647+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/2838899044 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314b04d0 msgr2=0x561a31887c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/2838899044 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314b04d0 0x561a31887c50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40042a0 tx=0x7f0fb4005850 comp rx=0 tx=0).stop 2024-07-26T06:51:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/2838899044 shutdown_connections 2024-07-26T06:51:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/2838899044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314d11b0 0x561a31880080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/2838899044 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314b04d0 0x561a31887c50 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/2838899044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314a82f0 0x561a3187fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/2838899044 >> 172.21.15.42:0/2838899044 conn(0x561a314c5300 msgr2=0x561a3187a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/2838899044 shutdown_connections 2024-07-26T06:51:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/2838899044 wait complete. 2024-07-26T06:51:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fcf9b2740 1 -- start start 2024-07-26T06:51:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fcf9b2740 1 Processor -- start 2024-07-26T06:51:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fcf9b2740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314a82f0 0x561a318842c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fcf9b2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314b04d0 0x561a31884980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fcf9b2740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314d11b0 0x561a31885410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fcf9b2740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc41abed0 con 0x561a314a82f0 2024-07-26T06:51:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fcf9b2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc41ab800 con 0x561a314b04d0 2024-07-26T06:51:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fcf9b2740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc41a8a40 con 0x561a314d11b0 2024-07-26T06:51:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fcf9b2740 10 client.0 _ll_register_callbacks cb 0x7f0fac00c950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fc1d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314a82f0 0x561a318842c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fc1d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314a82f0 0x561a318842c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49088/0 (socket says 172.21.15.42:49088) 2024-07-26T06:51:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fc1d9b640 1 -- 172.21.15.42:0/222240138 learned_addr learned my addr 172.21.15.42:0/222240138 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.648+0000 7f0fc259c640 1 --2- 172.21.15.42:0/222240138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314b04d0 0x561a31884980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.649+0000 7f0fc1d9b640 1 --2- 172.21.15.42:0/222240138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314d11b0 0x561a31885410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.649+0000 7f0fc1d9b640 1 -- 172.21.15.42:0/222240138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314d11b0 msgr2=0x561a31885410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.649+0000 7f0fc1d9b640 1 --2- 172.21.15.42:0/222240138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314d11b0 0x561a31885410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.649+0000 7f0fc1d9b640 1 -- 172.21.15.42:0/222240138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314b04d0 msgr2=0x561a31884980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.649+0000 7f0fc1d9b640 1 --2- 172.21.15.42:0/222240138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314b04d0 0x561a31884980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.649+0000 7f0fc1d9b640 1 -- 172.21.15.42:0/222240138 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a31885b70 con 0x561a314a82f0 2024-07-26T06:51:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.649+0000 7f0fc1d9b640 1 --2- 172.21.15.42:0/222240138 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314a82f0 0x561a318842c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x561a3188d610 tx=0x7f0fac00cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10a67c0601cf678 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.649+0000 7f0fa9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:51:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.649+0000 7f0fa9ffb640 1 -- 172.21.15.42:0/222240138 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac026020 con 0x561a314a82f0 2024-07-26T06:51:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.649+0000 7f0fa9ffb640 1 -- 172.21.15.42:0/222240138 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f0fac013230 con 0x561a314a82f0 2024-07-26T06:51:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.649+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/222240138 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fb4011440 con 0x561a314a82f0 2024-07-26T06:51:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.649+0000 7f0fa9ffb640 1 -- 172.21.15.42:0/222240138 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0fac01f040 con 0x561a314a82f0 2024-07-26T06:51:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.649+0000 7f0fcf9b2740 10 client.15579 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.649+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/222240138 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561a31886f60 con 0x561a314a82f0 2024-07-26T06:51:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.649+0000 7f0fa9ffb640 1 -- 172.21.15.42:0/222240138 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fb4011440 con 0x561a314a82f0 2024-07-26T06:51:46.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.650+0000 7f0fa9ffb640 1 -- 172.21.15.42:0/222240138 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0fac012900 con 0x561a314a82f0 2024-07-26T06:51:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.650+0000 7f0fcf9b2740 10 client.15579 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.650+0000 7f0fcf9b2740 10 client.15579 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:51:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.650+0000 7f0fcf9b2740 20 client.15579 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.650+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/222240138 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561a314651c0 0x561a31887970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.650+0000 7f0fcf9b2740 4 client.15579 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:51:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.650+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/222240138 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561a317e6da0 con 0x561a314651c0 2024-07-26T06:51:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.650+0000 7f0fc259c640 1 --2- 172.21.15.42:0/222240138 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561a314651c0 0x561a31887970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.650+0000 7f0fc259c640 1 --2- 172.21.15.42:0/222240138 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561a314651c0 0x561a31887970 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1eb4ef1d76a9982d server_cookie=5b155f37c3e89237 in_seq=0 out_seq=0 2024-07-26T06:51:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.650+0000 7f0fa9ffb640 10 client.15579 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:46.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.654+0000 7f0fa9ffb640 1 -- 172.21.15.42:0/222240138 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561a317e6da0 con 0x561a314651c0 2024-07-26T06:51:46.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.654+0000 7f0fa9ffb640 10 client.15579 handle_command_reply: tid=0 2024-07-26T06:51:46.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.654+0000 7f0fcf9b2740 1 client.15579 shutdown 2024-07-26T06:51:46.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.654+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/222240138 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314a82f0 msgr2=0x561a318842c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:46.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.654+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/222240138 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314a82f0 0x561a318842c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x561a3188d610 tx=0x7f0fac00cab0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.654+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/222240138 shutdown_connections 2024-07-26T06:51:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.654+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/222240138 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561a314651c0 0x561a31887970 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.654+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/222240138 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561a314d11b0 0x561a31885410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.654+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/222240138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561a314b04d0 0x561a31884980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.654+0000 7f0fa9ffb640 0 client.15579 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.654+0000 7f0fcf9b2740 1 --2- 172.21.15.42:0/222240138 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561a314a82f0 0x561a318842c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.654+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/222240138 >> 172.21.15.42:0/222240138 conn(0x561a314c5300 msgr2=0x561a318702b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.654+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/222240138 shutdown_connections 2024-07-26T06:51:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.654+0000 7f0fcf9b2740 1 -- 172.21.15.42:0/222240138 wait complete. 2024-07-26T06:51:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.654+0000 7f0fcf9b2740 20 client.15579 trim_cache size 0 max 16384 2024-07-26T06:51:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:51:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:51:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:51:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:51:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.655+0000 7f0fc89c4640 1 -- 172.21.15.42:0/3696344254 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0fb00b3bd0 msgr2=0x7f0fb00b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.655+0000 7f0fc89c4640 1 --2- 172.21.15.42:0/3696344254 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0fb00b3bd0 0x7f0fb00b60e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0c82b0 tx=0x7f0fbc0c0b00 comp rx=0 tx=0).stop 2024-07-26T06:51:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.655+0000 7f0fc89c4640 1 -- 172.21.15.42:0/3696344254 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0fc4103a20 msgr2=0x7f0fc41a0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.655+0000 7f0fc89c4640 1 --2- 172.21.15.42:0/3696344254 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0fc4103a20 0x7f0fc41a0c40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0097b0 tx=0x7f0fbc003c20 comp rx=0 tx=0).stop 2024-07-26T06:51:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.655+0000 7f0fc89c4640 1 -- 172.21.15.42:0/3696344254 shutdown_connections 2024-07-26T06:51:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.655+0000 7f0fc89c4640 1 --2- 172.21.15.42:0/3696344254 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0fb00b3bd0 0x7f0fb00b60e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.655+0000 7f0fc89c4640 1 --2- 172.21.15.42:0/3696344254 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0fc4103a20 0x7f0fc41a0c40 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.655+0000 7f0fc89c4640 1 --2- 172.21.15.42:0/3696344254 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0fc4103080 0x7f0fc419e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.655+0000 7f0fc89c4640 1 --2- 172.21.15.42:0/3696344254 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0fc4102000 0x7f0fc406bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.655+0000 7f0fc89c4640 1 -- 172.21.15.42:0/3696344254 >> 172.21.15.42:0/3696344254 conn(0x7f0fc40fd960 msgr2=0x7f0fc4106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.655+0000 7f0fc89c4640 1 -- 172.21.15.42:0/3696344254 shutdown_connections 2024-07-26T06:51:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:46.655+0000 7f0fc89c4640 1 -- 172.21.15.42:0/3696344254 wait complete. 2024-07-26T06:51:47.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:47 smithi042 ceph-mon[32775]: pgmap v442: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 853 B/s wr, 12 op/s 2024-07-26T06:51:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:47 smithi164 ceph-mon[36970]: pgmap v442: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 853 B/s wr, 12 op/s 2024-07-26T06:51:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:47 smithi163 ceph-mon[36873]: pgmap v442: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 853 B/s wr, 12 op/s 2024-07-26T06:51:47.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.921+0000 7f4386ec0640 1 -- 172.21.15.42:0/436326044 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43801024a0 msgr2=0x7f43801028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:47.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.921+0000 7f4386ec0640 1 --2- 172.21.15.42:0/436326044 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43801024a0 0x7f43801028b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f4370009900 tx=0x7f437002a410 comp rx=0 tx=0).stop 2024-07-26T06:51:47.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.921+0000 7f4386ec0640 1 -- 172.21.15.42:0/436326044 shutdown_connections 2024-07-26T06:51:47.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.921+0000 7f4386ec0640 1 --2- 172.21.15.42:0/436326044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4380108730 0x7f438010ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.921+0000 7f4386ec0640 1 --2- 172.21.15.42:0/436326044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4380107de0 0x7f43801081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.921+0000 7f4386ec0640 1 --2- 172.21.15.42:0/436326044 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43801024a0 0x7f43801028b0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.921+0000 7f4386ec0640 1 -- 172.21.15.42:0/436326044 >> 172.21.15.42:0/436326044 conn(0x7f4380079140 msgr2=0x7f4380079550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:47.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.921+0000 7f4386ec0640 1 -- 172.21.15.42:0/436326044 shutdown_connections 2024-07-26T06:51:47.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.921+0000 7f4386ec0640 1 -- 172.21.15.42:0/436326044 wait complete. 2024-07-26T06:51:47.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.921+0000 7f4386ec0640 1 Processor -- start 2024-07-26T06:51:47.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f4386ec0640 1 -- start start 2024-07-26T06:51:47.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f4386ec0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43801024a0 0x7f438010fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f4386ec0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4380107de0 0x7f438010ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f4386ec0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4380108730 0x7f43801104c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f4386ec0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4380113510 con 0x7f43801024a0 2024-07-26T06:51:47.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f4386ec0640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4380113390 con 0x7f4380107de0 2024-07-26T06:51:47.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f4386ec0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4380113690 con 0x7f4380108730 2024-07-26T06:51:47.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f4384c5c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43801024a0 0x7f438010fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:47.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f4384c5c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43801024a0 0x7f438010fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60278/0 (socket says 172.21.15.42:60278) 2024-07-26T06:51:47.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f4384c5c640 1 -- 172.21.15.42:0/2653935588 learned_addr learned my addr 172.21.15.42:0/2653935588 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:47.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f438545d640 1 --2- 172.21.15.42:0/2653935588 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4380108730 0x7f43801104c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:47.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f437ffff640 1 --2- 172.21.15.42:0/2653935588 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4380107de0 0x7f438010ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f4384c5c640 1 -- 172.21.15.42:0/2653935588 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4380108730 msgr2=0x7f43801104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f4384c5c640 1 --2- 172.21.15.42:0/2653935588 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4380108730 0x7f43801104c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f4384c5c640 1 -- 172.21.15.42:0/2653935588 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4380107de0 msgr2=0x7f438010ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f4384c5c640 1 --2- 172.21.15.42:0/2653935588 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4380107de0 0x7f438010ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f4384c5c640 1 -- 172.21.15.42:0/2653935588 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43801a81e0 con 0x7f43801024a0 2024-07-26T06:51:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.922+0000 7f4384c5c640 1 --2- 172.21.15.42:0/2653935588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43801024a0 0x7f438010fa40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4370005940 tx=0x7f43700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.923+0000 7f437dffb640 1 -- 172.21.15.42:0/2653935588 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437003f030 con 0x7f43801024a0 2024-07-26T06:51:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.923+0000 7f437dffb640 1 -- 172.21.15.42:0/2653935588 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4370003cb0 con 0x7f43801024a0 2024-07-26T06:51:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.923+0000 7f4386ec0640 1 -- 172.21.15.42:0/2653935588 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43801a8470 con 0x7f43801024a0 2024-07-26T06:51:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.923+0000 7f437dffb640 1 -- 172.21.15.42:0/2653935588 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4370035070 con 0x7f43801024a0 2024-07-26T06:51:47.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.923+0000 7f4386ec0640 1 -- 172.21.15.42:0/2653935588 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43801a8950 con 0x7f43801024a0 2024-07-26T06:51:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.926+0000 7f438d444740 1 Processor -- start 2024-07-26T06:51:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.926+0000 7f438d444740 1 -- start start 2024-07-26T06:51:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.926+0000 7f438d444740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee316520 0x563dee6bc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.926+0000 7f438d444740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee2ed660 0x563dee6bc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.926+0000 7f438d444740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee2f5840 0x563dee6bcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.926+0000 7f438d444740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x563dee631e70 con 0x563dee316520 2024-07-26T06:51:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.926+0000 7f438d444740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563dee621370 con 0x563dee2ed660 2024-07-26T06:51:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.926+0000 7f438d444740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x563dee6253b0 con 0x563dee2f5840 2024-07-26T06:51:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.927+0000 7f438545d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee316520 0x563dee6bc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.927+0000 7f438545d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee316520 0x563dee6bc070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60290/0 (socket says 172.21.15.42:60290) 2024-07-26T06:51:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.927+0000 7f438545d640 1 -- 172.21.15.42:0/3638970169 learned_addr learned my addr 172.21.15.42:0/3638970169 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.927+0000 7f437ffff640 1 --2- 172.21.15.42:0/3638970169 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee2f5840 0x563dee6bcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:47.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.927+0000 7f437dffb640 1 -- 172.21.15.42:0/2653935588 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f4370042030 con 0x7f43801024a0 2024-07-26T06:51:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.927+0000 7f438545d640 1 -- 172.21.15.42:0/3638970169 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee2f5840 msgr2=0x563dee6bcaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.927+0000 7f438545d640 1 --2- 172.21.15.42:0/3638970169 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee2f5840 0x563dee6bcaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.927+0000 7f438545d640 1 -- 172.21.15.42:0/3638970169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee2ed660 msgr2=0x563dee6bc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.927+0000 7f438545d640 1 --2- 172.21.15.42:0/3638970169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee2ed660 0x563dee6bc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.927+0000 7f438545d640 1 -- 172.21.15.42:0/3638970169 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dee5ee040 con 0x563dee316520 2024-07-26T06:51:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.927+0000 7f438545d640 1 --2- 172.21.15.42:0/3638970169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee316520 0x563dee6bc070 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4378009f70 tx=0x7f4378010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c12f499d0e0173 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.927+0000 7f4366ffd640 1 -- 172.21.15.42:0/3638970169 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4378016070 con 0x563dee316520 2024-07-26T06:51:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.927+0000 7f4366ffd640 1 -- 172.21.15.42:0/3638970169 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4378004d20 con 0x563dee316520 2024-07-26T06:51:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.927+0000 7f4366ffd640 1 -- 172.21.15.42:0/3638970169 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4378012790 con 0x563dee316520 2024-07-26T06:51:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.928+0000 7f437dffb640 1 --2- 172.21.15.42:0/2653935588 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f43680b3bd0 0x7f43680b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.928+0000 7f4384c5c640 1 --2- 172.21.15.42:0/2653935588 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f43680b3bd0 0x7f43680b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:47.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.928+0000 7f437dffb640 1 -- 172.21.15.42:0/2653935588 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f4370104500 con 0x7f43801024a0 2024-07-26T06:51:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.928+0000 7f438d444740 1 -- 172.21.15.42:0/3638970169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee316520 msgr2=0x563dee6bc070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.928+0000 7f438d444740 1 --2- 172.21.15.42:0/3638970169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee316520 0x563dee6bc070 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4378009f70 tx=0x7f4378010800 comp rx=0 tx=0).stop 2024-07-26T06:51:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.929+0000 7f438d444740 1 -- 172.21.15.42:0/3638970169 shutdown_connections 2024-07-26T06:51:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.929+0000 7f438d444740 1 --2- 172.21.15.42:0/3638970169 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee2f5840 0x563dee6bcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.929+0000 7f4384c5c640 1 --2- 172.21.15.42:0/2653935588 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f43680b3bd0 0x7f43680b60e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f43700cb130 tx=0x7f4370098bd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.929+0000 7f438d444740 1 --2- 172.21.15.42:0/3638970169 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee2ed660 0x563dee6bc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.929+0000 7f438d444740 1 --2- 172.21.15.42:0/3638970169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee316520 0x563dee6bc070 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.929+0000 7f438d444740 1 -- 172.21.15.42:0/3638970169 >> 172.21.15.42:0/3638970169 conn(0x563dee30a910 msgr2=0x563dee6b9f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.929+0000 7f438d444740 1 -- 172.21.15.42:0/3638970169 shutdown_connections 2024-07-26T06:51:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.929+0000 7f438d444740 1 -- 172.21.15.42:0/3638970169 wait complete. 2024-07-26T06:51:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.929+0000 7f438d444740 1 -- start start 2024-07-26T06:51:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.929+0000 7f438d444740 1 Processor -- start 2024-07-26T06:51:47.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f438d444740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee2ed660 0x563dee6c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f438d444740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee2f5840 0x563dee6c8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f438d444740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee316520 0x563dee6d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f438d444740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x563dee484bb0 con 0x563dee2f5840 2024-07-26T06:51:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f438d444740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563dee5c3ac0 con 0x563dee316520 2024-07-26T06:51:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f438d444740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x563dee6ca850 con 0x563dee2ed660 2024-07-26T06:51:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f438d444740 10 client.0 _ll_register_callbacks cb 0x563dee5cc3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f437ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee2f5840 0x563dee6c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f438545d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee2ed660 0x563dee6c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f437ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee2f5840 0x563dee6c8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60294/0 (socket says 172.21.15.42:60294) 2024-07-26T06:51:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f438545d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee2ed660 0x563dee6c7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:49816/0 (socket says 172.21.15.42:49816) 2024-07-26T06:51:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f437ffff640 1 -- 172.21.15.42:0/2014545073 learned_addr learned my addr 172.21.15.42:0/2014545073 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:47.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f438545d640 1 --2- 172.21.15.42:0/2014545073 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee316520 0x563dee6d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f437ffff640 1 -- 172.21.15.42:0/2014545073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee2ed660 msgr2=0x563dee6c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f437ffff640 1 --2- 172.21.15.42:0/2014545073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee2ed660 0x563dee6c7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f437ffff640 1 -- 172.21.15.42:0/2014545073 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee316520 msgr2=0x563dee6d3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f437ffff640 1 --2- 172.21.15.42:0/2014545073 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee316520 0x563dee6d3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f437ffff640 1 -- 172.21.15.42:0/2014545073 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dee381b00 con 0x563dee2f5840 2024-07-26T06:51:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.930+0000 7f438545d640 1 --2- 172.21.15.42:0/2014545073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee2ed660 0x563dee6c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:51:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.931+0000 7f437ffff640 1 --2- 172.21.15.42:0/2014545073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee2f5840 0x563dee6c8140 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4374009d00 tx=0x7f43740117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b09d13d1ee8ef30e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.931+0000 7f435ffff640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:51:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.931+0000 7f435ffff640 1 -- 172.21.15.42:0/2014545073 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4374012860 con 0x563dee2f5840 2024-07-26T06:51:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.931+0000 7f435ffff640 1 -- 172.21.15.42:0/2014545073 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f4374012a00 con 0x563dee2f5840 2024-07-26T06:51:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.931+0000 7f438d444740 1 -- 172.21.15.42:0/2014545073 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4378004d20 con 0x563dee2f5840 2024-07-26T06:51:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.931+0000 7f435ffff640 1 -- 172.21.15.42:0/2014545073 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4374012c00 con 0x563dee2f5840 2024-07-26T06:51:47.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.931+0000 7f438d444740 10 client.15591 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:47.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.931+0000 7f438d444740 1 -- 172.21.15.42:0/2014545073 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563dee3f8960 con 0x563dee2f5840 2024-07-26T06:51:47.980 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:51:47.980 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 29, 2024-07-26T06:51:47.980 INFO:teuthology.orchestra.run.smithi042.stdout: "leader": 24620, 2024-07-26T06:51:47.980 INFO:teuthology.orchestra.run.smithi042.stdout: "set_version": 4, 2024-07-26T06:51:47.980 INFO:teuthology.orchestra.run.smithi042.stdout: "sets": { 2024-07-26T06:51:47.980 INFO:teuthology.orchestra.run.smithi042.stdout: "213c08d8": { 2024-07-26T06:51:47.980 INFO:teuthology.orchestra.run.smithi042.stdout: "version": 4, 2024-07-26T06:51:47.981 INFO:teuthology.orchestra.run.smithi042.stdout: "age_ref": 0.0, 2024-07-26T06:51:47.981 INFO:teuthology.orchestra.run.smithi042.stdout: "state": { 2024-07-26T06:51:47.981 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "RELEASED", 2024-07-26T06:51:47.981 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 0.0 2024-07-26T06:51:47.981 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-07-26T06:51:47.981 INFO:teuthology.orchestra.run.smithi042.stdout: "timeout": 90.0, 2024-07-26T06:51:47.981 INFO:teuthology.orchestra.run.smithi042.stdout: "expiration": 142.6, 2024-07-26T06:51:47.981 INFO:teuthology.orchestra.run.smithi042.stdout: "members": { 2024-07-26T06:51:47.981 INFO:teuthology.orchestra.run.smithi042.stdout: "file:/": { 2024-07-26T06:51:47.981 INFO:teuthology.orchestra.run.smithi042.stdout: "excluded": false, 2024-07-26T06:51:47.981 INFO:teuthology.orchestra.run.smithi042.stdout: "state": { 2024-07-26T06:51:47.981 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "RELEASED", 2024-07-26T06:51:47.981 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 0.0 2024-07-26T06:51:47.982 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:51:47.982 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:51:47.982 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:51:47.982 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:51:47.982 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:51:47.982 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:51:47.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.931+0000 7f435ffff640 1 -- 172.21.15.42:0/2014545073 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4378004d20 con 0x563dee2f5840 2024-07-26T06:51:47.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.931+0000 7f435ffff640 1 -- 172.21.15.42:0/2014545073 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f437400f040 con 0x563dee2f5840 2024-07-26T06:51:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.932+0000 7f438d444740 10 client.15591 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.932+0000 7f438d444740 10 client.15591 resolve_mds: validated gid 24620 aka daemon mds.b 2024-07-26T06:51:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.932+0000 7f438d444740 20 client.15591 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.932+0000 7f438d444740 1 --2- 172.21.15.42:0/2014545073 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x563dee2d4ea0 0x563dee6d9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.932+0000 7f438d444740 4 client.15591 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:51:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.932+0000 7f438d444740 1 -- 172.21.15.42:0/2014545073 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563dee633dc0 con 0x563dee2d4ea0 2024-07-26T06:51:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.932+0000 7f438545d640 1 --2- 172.21.15.42:0/2014545073 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x563dee2d4ea0 0x563dee6d9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.932+0000 7f435ffff640 10 client.15591 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.932+0000 7f438545d640 1 --2- 172.21.15.42:0/2014545073 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x563dee2d4ea0 0x563dee6d9cb0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=467757d7f0743b51 server_cookie=a8616b1e05fc0a94 in_seq=0 out_seq=0 2024-07-26T06:51:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.933+0000 7f435ffff640 1 -- 172.21.15.42:0/2014545073 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563dee633dc0 con 0x563dee2d4ea0 2024-07-26T06:51:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.933+0000 7f435ffff640 10 client.15591 handle_command_reply: tid=0 2024-07-26T06:51:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.933+0000 7f438d444740 1 client.15591 shutdown 2024-07-26T06:51:47.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.933+0000 7f438d444740 1 -- 172.21.15.42:0/2014545073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee2f5840 msgr2=0x563dee6c8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.933+0000 7f438d444740 1 --2- 172.21.15.42:0/2014545073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee2f5840 0x563dee6c8140 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4374009d00 tx=0x7f43740117b0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.933+0000 7f438d444740 1 -- 172.21.15.42:0/2014545073 shutdown_connections 2024-07-26T06:51:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.933+0000 7f438d444740 1 --2- 172.21.15.42:0/2014545073 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x563dee2d4ea0 0x563dee6d9cb0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.933+0000 7f438d444740 1 --2- 172.21.15.42:0/2014545073 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee316520 0x563dee6d3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.933+0000 7f438d444740 1 --2- 172.21.15.42:0/2014545073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee2f5840 0x563dee6c8140 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.933+0000 7f435ffff640 0 client.15591 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.933+0000 7f438d444740 1 --2- 172.21.15.42:0/2014545073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee2ed660 0x563dee6c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.933+0000 7f438d444740 1 -- 172.21.15.42:0/2014545073 >> 172.21.15.42:0/2014545073 conn(0x563dee30a910 msgr2=0x563dee6bf250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.934+0000 7f438d444740 1 -- 172.21.15.42:0/2014545073 shutdown_connections 2024-07-26T06:51:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.934+0000 7f438d444740 1 -- 172.21.15.42:0/2014545073 wait complete. 2024-07-26T06:51:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.934+0000 7f438d444740 20 client.15591 trim_cache size 0 max 16384 2024-07-26T06:51:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.951+0000 7f438d444740 1 Processor -- start 2024-07-26T06:51:47.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.951+0000 7f438d444740 1 -- start start 2024-07-26T06:51:47.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.951+0000 7f438d444740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee2ed660 0x563dee6b8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.951+0000 7f438d444740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee2f5840 0x563dee6bda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.951+0000 7f438d444740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee316520 0x563dee6c9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.951+0000 7f438d444740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f43801b1110 con 0x563dee2ed660 2024-07-26T06:51:47.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.951+0000 7f438d444740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f43801b2c50 con 0x563dee2f5840 2024-07-26T06:51:47.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.951+0000 7f438d444740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f43801b17e0 con 0x563dee316520 2024-07-26T06:51:47.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.951+0000 7f438545d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee2ed660 0x563dee6b8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:47.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.951+0000 7f438545d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee2ed660 0x563dee6b8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60298/0 (socket says 172.21.15.42:60298) 2024-07-26T06:51:47.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.951+0000 7f438545d640 1 -- 172.21.15.42:0/1393551946 learned_addr learned my addr 172.21.15.42:0/1393551946 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:47.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.951+0000 7f438545d640 1 -- 172.21.15.42:0/1393551946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee316520 msgr2=0x563dee6c9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:47.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.951+0000 7f438545d640 1 --2- 172.21.15.42:0/1393551946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee316520 0x563dee6c9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.951+0000 7f438545d640 1 -- 172.21.15.42:0/1393551946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee2f5840 msgr2=0x563dee6bda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:47.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.951+0000 7f438545d640 1 --2- 172.21.15.42:0/1393551946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee2f5840 0x563dee6bda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.951+0000 7f438545d640 1 -- 172.21.15.42:0/1393551946 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dee6c8aa0 con 0x563dee2ed660 2024-07-26T06:51:47.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.951+0000 7f438545d640 1 --2- 172.21.15.42:0/1393551946 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee2ed660 0x563dee6b8be0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4378000f80 tx=0x7f4378010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeca137e6e672524 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:47.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.952+0000 7f43667fc640 1 -- 172.21.15.42:0/1393551946 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437802e070 con 0x563dee2ed660 2024-07-26T06:51:47.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.952+0000 7f43667fc640 1 -- 172.21.15.42:0/1393551946 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4378004c60 con 0x563dee2ed660 2024-07-26T06:51:47.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.952+0000 7f43667fc640 1 -- 172.21.15.42:0/1393551946 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4378016070 con 0x563dee2ed660 2024-07-26T06:51:47.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.952+0000 7f438d444740 1 -- 172.21.15.42:0/1393551946 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee2ed660 msgr2=0x563dee6b8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:47.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.952+0000 7f438d444740 1 --2- 172.21.15.42:0/1393551946 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee2ed660 0x563dee6b8be0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4378000f80 tx=0x7f4378010800 comp rx=0 tx=0).stop 2024-07-26T06:51:47.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.952+0000 7f438d444740 1 -- 172.21.15.42:0/1393551946 shutdown_connections 2024-07-26T06:51:47.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.952+0000 7f438d444740 1 --2- 172.21.15.42:0/1393551946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee316520 0x563dee6c9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.952+0000 7f438d444740 1 --2- 172.21.15.42:0/1393551946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee2f5840 0x563dee6bda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.952+0000 7f438d444740 1 --2- 172.21.15.42:0/1393551946 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee2ed660 0x563dee6b8be0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.952+0000 7f438d444740 1 -- 172.21.15.42:0/1393551946 >> 172.21.15.42:0/1393551946 conn(0x563dee30a910 msgr2=0x563dee6c8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:47.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.953+0000 7f438d444740 1 -- 172.21.15.42:0/1393551946 shutdown_connections 2024-07-26T06:51:47.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.953+0000 7f438d444740 1 -- 172.21.15.42:0/1393551946 wait complete. 2024-07-26T06:51:47.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.953+0000 7f438d444740 1 -- start start 2024-07-26T06:51:47.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.953+0000 7f438d444740 1 Processor -- start 2024-07-26T06:51:47.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.953+0000 7f438d444740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee2ed660 0x563dee6cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.953+0000 7f438d444740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee2f5840 0x563dee6cc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.953+0000 7f438d444740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee316520 0x563dee741440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.953+0000 7f438d444740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f43801b2580 con 0x563dee316520 2024-07-26T06:51:47.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.953+0000 7f438d444740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f43801b1eb0 con 0x563dee2f5840 2024-07-26T06:51:47.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.953+0000 7f438d444740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f43801af0f0 con 0x563dee2ed660 2024-07-26T06:51:47.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.954+0000 7f438d444740 10 client.0 _ll_register_callbacks cb 0x7f4374009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:47.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.954+0000 7f438545d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee316520 0x563dee741440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:47.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.954+0000 7f438545d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee316520 0x563dee741440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60310/0 (socket says 172.21.15.42:60310) 2024-07-26T06:51:47.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.954+0000 7f438545d640 1 -- 172.21.15.42:0/3429085405 learned_addr learned my addr 172.21.15.42:0/3429085405 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:47.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.954+0000 7f437ffff640 1 --2- 172.21.15.42:0/3429085405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee2f5840 0x563dee6cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:47.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.954+0000 7f438545d640 1 --2- 172.21.15.42:0/3429085405 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee2ed660 0x563dee6cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:47.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.954+0000 7f438545d640 1 -- 172.21.15.42:0/3429085405 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee2ed660 msgr2=0x563dee6cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:47.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.954+0000 7f438545d640 1 --2- 172.21.15.42:0/3429085405 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee2ed660 0x563dee6cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.954+0000 7f438545d640 1 -- 172.21.15.42:0/3429085405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee2f5840 msgr2=0x563dee6cc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:47.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.954+0000 7f438545d640 1 --2- 172.21.15.42:0/3429085405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee2f5840 0x563dee6cc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.954+0000 7f438545d640 1 -- 172.21.15.42:0/3429085405 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dee6ccb30 con 0x563dee316520 2024-07-26T06:51:47.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.954+0000 7f438545d640 1 --2- 172.21.15.42:0/3429085405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee316520 0x563dee741440 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4378005b40 tx=0x7f4378009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bdea501bccc794a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:47.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.954+0000 7f435ffff640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:51:47.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.954+0000 7f435ffff640 1 -- 172.21.15.42:0/3429085405 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4378037020 con 0x563dee316520 2024-07-26T06:51:47.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.954+0000 7f435ffff640 1 -- 172.21.15.42:0/3429085405 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f437802bbb0 con 0x563dee316520 2024-07-26T06:51:47.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.954+0000 7f438d444740 1 -- 172.21.15.42:0/3429085405 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4378004c60 con 0x563dee316520 2024-07-26T06:51:47.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.955+0000 7f435ffff640 1 -- 172.21.15.42:0/3429085405 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4378042050 con 0x563dee316520 2024-07-26T06:51:47.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.955+0000 7f438d444740 10 client.15597 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:47.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.955+0000 7f438d444740 1 -- 172.21.15.42:0/3429085405 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563dee742a70 con 0x563dee316520 2024-07-26T06:51:47.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.955+0000 7f435ffff640 1 -- 172.21.15.42:0/3429085405 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4378004c60 con 0x563dee316520 2024-07-26T06:51:47.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.955+0000 7f435ffff640 1 -- 172.21.15.42:0/3429085405 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f437800f040 con 0x563dee316520 2024-07-26T06:51:47.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.955+0000 7f438d444740 10 client.15597 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:47.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.955+0000 7f438d444740 10 client.15597 resolve_mds: validated gid 24620 aka daemon mds.b 2024-07-26T06:51:47.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.955+0000 7f438d444740 20 client.15597 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:47.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.955+0000 7f438d444740 1 --2- 172.21.15.42:0/3429085405 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x563dee2aa250 0x563dee744fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:47.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.955+0000 7f438d444740 4 client.15597 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "213c08d8", "release": true, "await": true}] 2024-07-26T06:51:47.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.955+0000 7f438d444740 1 -- 172.21.15.42:0/3429085405 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "quiesce db", "set_id": "213c08d8", "release": true, "await": true}) -- 0x563dee62f860 con 0x563dee2aa250 2024-07-26T06:51:47.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.955+0000 7f437ffff640 1 --2- 172.21.15.42:0/3429085405 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x563dee2aa250 0x563dee744fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:47.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.956+0000 7f435ffff640 10 client.15597 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.956+0000 7f437ffff640 1 --2- 172.21.15.42:0/3429085405 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x563dee2aa250 0x563dee744fe0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bdfc47510380da8 server_cookie=831756cfd62d987d in_seq=0 out_seq=0 2024-07-26T06:51:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.958+0000 7f435ffff640 1 -- 172.21.15.42:0/3429085405 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x563dee62f860 con 0x563dee2aa250 2024-07-26T06:51:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.958+0000 7f435ffff640 10 client.15597 handle_command_reply: tid=0 2024-07-26T06:51:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.958+0000 7f438d444740 1 client.15597 shutdown 2024-07-26T06:51:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.958+0000 7f438d444740 1 -- 172.21.15.42:0/3429085405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee316520 msgr2=0x563dee741440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.958+0000 7f438d444740 1 --2- 172.21.15.42:0/3429085405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee316520 0x563dee741440 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4378005b40 tx=0x7f4378009800 comp rx=0 tx=0).stop 2024-07-26T06:51:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.958+0000 7f438d444740 1 -- 172.21.15.42:0/3429085405 shutdown_connections 2024-07-26T06:51:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.958+0000 7f438d444740 1 --2- 172.21.15.42:0/3429085405 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x563dee2aa250 0x563dee744fe0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.958+0000 7f438d444740 1 --2- 172.21.15.42:0/3429085405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563dee316520 0x563dee741440 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.958+0000 7f438d444740 1 --2- 172.21.15.42:0/3429085405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563dee2f5840 0x563dee6cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.958+0000 7f435ffff640 0 client.15597 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.958+0000 7f438d444740 1 --2- 172.21.15.42:0/3429085405 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563dee2ed660 0x563dee6cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.958+0000 7f438d444740 1 -- 172.21.15.42:0/3429085405 >> 172.21.15.42:0/3429085405 conn(0x563dee30a910 msgr2=0x563dee6bd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.958+0000 7f438d444740 1 -- 172.21.15.42:0/3429085405 shutdown_connections 2024-07-26T06:51:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.958+0000 7f438d444740 1 -- 172.21.15.42:0/3429085405 wait complete. 2024-07-26T06:51:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.958+0000 7f438d444740 20 client.15597 trim_cache size 0 max 16384 2024-07-26T06:51:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.959+0000 7f4386ec0640 1 -- 172.21.15.42:0/2653935588 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f43680b3bd0 msgr2=0x7f43680b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.959+0000 7f4386ec0640 1 --2- 172.21.15.42:0/2653935588 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f43680b3bd0 0x7f43680b60e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f43700cb130 tx=0x7f4370098bd0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.959+0000 7f4386ec0640 1 -- 172.21.15.42:0/2653935588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43801024a0 msgr2=0x7f438010fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.959+0000 7f4386ec0640 1 --2- 172.21.15.42:0/2653935588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43801024a0 0x7f438010fa40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4370005940 tx=0x7f43700042f0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.959+0000 7f4386ec0640 1 -- 172.21.15.42:0/2653935588 shutdown_connections 2024-07-26T06:51:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.959+0000 7f4386ec0640 1 --2- 172.21.15.42:0/2653935588 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f43680b3bd0 0x7f43680b60e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.959+0000 7f4386ec0640 1 --2- 172.21.15.42:0/2653935588 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4380108730 0x7f43801104c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.959+0000 7f4386ec0640 1 --2- 172.21.15.42:0/2653935588 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4380107de0 0x7f438010ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.959+0000 7f4386ec0640 1 --2- 172.21.15.42:0/2653935588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43801024a0 0x7f438010fa40 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.959+0000 7f4386ec0640 1 -- 172.21.15.42:0/2653935588 >> 172.21.15.42:0/2653935588 conn(0x7f4380079140 msgr2=0x7f4380105ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:47.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.959+0000 7f4386ec0640 1 -- 172.21.15.42:0/2653935588 shutdown_connections 2024-07-26T06:51:47.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:47.959+0000 7f4386ec0640 1 -- 172.21.15.42:0/2653935588 wait complete. 2024-07-26T06:51:48.833 INFO:tasks.workunit.client.0.smithi042.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-07-26T06:51:48.933 INFO:tasks.workunit.client.0.smithi042.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-07-26T06:51:48.974 INFO:tasks.workunit.client.0.smithi042.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-26T06:51:48.976 INFO:tasks.workunit.client.0.smithi042.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-26T06:51:48.976 INFO:tasks.workunit.client.0.smithi042.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-07-26T06:51:49.016 INFO:tasks.workunit.client.0.smithi042.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-26T06:51:49.019 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-07-26T06:51:49.020 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-07-26T06:51:49.036 INFO:tasks.workunit:Running workunits matching suites/fsx.sh on client.0... 2024-07-26T06:51:49.038 INFO:tasks.workunit:Running workunit suites/fsx.sh... 2024-07-26T06:51:49.038 DEBUG:teuthology.orchestra.run.smithi042:workunit test suites/fsx.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-07-26T06:51:49.099 INFO:tasks.workunit.client.0.smithi042.stderr:+ set -e 2024-07-26T06:51:49.099 INFO:tasks.workunit.client.0.smithi042.stderr:+ git clone https://git.ceph.com/xfstests-dev.git 2024-07-26T06:51:49.101 INFO:tasks.workunit.client.0.smithi042.stderr:Cloning into 'xfstests-dev'... 2024-07-26T06:51:49.134 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '213c08d8', total seconds elapsed: 30.0 2024-07-26T06:51:49.134 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.55 2024-07-26T06:51:49.135 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:51:49.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:49 smithi042 ceph-mon[32775]: pgmap v443: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T06:51:49.440 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:51:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:49 smithi164 ceph-mon[36970]: pgmap v443: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T06:51:49.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:49 smithi163 ceph-mon[36873]: pgmap v443: 97 pgs: 97 active+clean; 673 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-07-26T06:51:51.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.245+0000 7f705871a640 1 -- 172.21.15.42:0/2675122537 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7050107eb0 msgr2=0x7f705010e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:51.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.245+0000 7f705871a640 1 --2- 172.21.15.42:0/2675122537 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7050107eb0 0x7f705010e6e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f704800adb0 tx=0x7f704802b7a0 comp rx=0 tx=0).stop 2024-07-26T06:51:51.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.245+0000 7f705871a640 1 -- 172.21.15.42:0/2675122537 shutdown_connections 2024-07-26T06:51:51.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.245+0000 7f705871a640 1 --2- 172.21.15.42:0/2675122537 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7050107eb0 0x7f705010e6e0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:51.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.245+0000 7f705871a640 1 --2- 172.21.15.42:0/2675122537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7050107560 0x7f7050107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:51.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.245+0000 7f705871a640 1 --2- 172.21.15.42:0/2675122537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7050101c20 0x7f7050102030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:51.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.245+0000 7f705871a640 1 -- 172.21.15.42:0/2675122537 >> 172.21.15.42:0/2675122537 conn(0x7f7050077690 msgr2=0x7f7050077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:51.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.245+0000 7f705871a640 1 -- 172.21.15.42:0/2675122537 shutdown_connections 2024-07-26T06:51:51.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.245+0000 7f705871a640 1 -- 172.21.15.42:0/2675122537 wait complete. 2024-07-26T06:51:51.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.246+0000 7f705871a640 1 Processor -- start 2024-07-26T06:51:51.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.246+0000 7f705871a640 1 -- start start 2024-07-26T06:51:51.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.247+0000 7f705871a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7050101c20 0x7f7050199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:51.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.247+0000 7f705871a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7050107560 0x7f705019a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:51.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.247+0000 7f705871a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7050107eb0 0x7f705019f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:51.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.247+0000 7f705871a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f70501112f0 con 0x7f7050101c20 2024-07-26T06:51:51.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.247+0000 7f705871a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7050111170 con 0x7f7050107560 2024-07-26T06:51:51.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.247+0000 7f705871a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7050111470 con 0x7f7050107eb0 2024-07-26T06:51:51.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.247+0000 7f70564b6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7050101c20 0x7f7050199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:51.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.247+0000 7f70564b6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7050101c20 0x7f7050199e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60322/0 (socket says 172.21.15.42:60322) 2024-07-26T06:51:51.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.247+0000 7f70564b6640 1 -- 172.21.15.42:0/1986479228 learned_addr learned my addr 172.21.15.42:0/1986479228 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:51.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.247+0000 7f7056cb7640 1 --2- 172.21.15.42:0/1986479228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7050107eb0 0x7f705019f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:51.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.247+0000 7f7055cb5640 1 --2- 172.21.15.42:0/1986479228 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7050107560 0x7f705019a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:51.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.248+0000 7f70564b6640 1 -- 172.21.15.42:0/1986479228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7050107eb0 msgr2=0x7f705019f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:51.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.248+0000 7f70564b6640 1 --2- 172.21.15.42:0/1986479228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7050107eb0 0x7f705019f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:51.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.248+0000 7f70564b6640 1 -- 172.21.15.42:0/1986479228 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7050107560 msgr2=0x7f705019a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:51.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.248+0000 7f70564b6640 1 --2- 172.21.15.42:0/1986479228 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7050107560 0x7f705019a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:51.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.248+0000 7f70564b6640 1 -- 172.21.15.42:0/1986479228 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f705019fe00 con 0x7f7050101c20 2024-07-26T06:51:51.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.248+0000 7f70564b6640 1 --2- 172.21.15.42:0/1986479228 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7050101c20 0x7f7050199e90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f704c00d950 tx=0x7f704c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:51.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.248+0000 7f70477fe640 1 -- 172.21.15.42:0/1986479228 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704c017030 con 0x7f7050101c20 2024-07-26T06:51:51.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.248+0000 7f705871a640 1 -- 172.21.15.42:0/1986479228 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70501a00f0 con 0x7f7050101c20 2024-07-26T06:51:51.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.248+0000 7f70477fe640 1 -- 172.21.15.42:0/1986479228 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f704c004020 con 0x7f7050101c20 2024-07-26T06:51:51.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.248+0000 7f70477fe640 1 -- 172.21.15.42:0/1986479228 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704c0047a0 con 0x7f7050101c20 2024-07-26T06:51:51.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.248+0000 7f705871a640 1 -- 172.21.15.42:0/1986479228 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70501a6850 con 0x7f7050101c20 2024-07-26T06:51:51.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.250+0000 7f705871a640 1 -- 172.21.15.42:0/1986479228 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7050102030 con 0x7f7050101c20 2024-07-26T06:51:51.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.253+0000 7f70477fe640 1 -- 172.21.15.42:0/1986479228 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f704c009a30 con 0x7f7050101c20 2024-07-26T06:51:51.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.253+0000 7f70477fe640 1 --2- 172.21.15.42:0/1986479228 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f703c0b3bd0 0x7f703c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:51.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.253+0000 7f70477fe640 1 -- 172.21.15.42:0/1986479228 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f704c0e3d20 con 0x7f7050101c20 2024-07-26T06:51:51.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.253+0000 7f7055cb5640 1 --2- 172.21.15.42:0/1986479228 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f703c0b3bd0 0x7f703c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:51.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.254+0000 7f7055cb5640 1 --2- 172.21.15.42:0/1986479228 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f703c0b3bd0 0x7f703c0b60e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7050107210 tx=0x7f704000b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:51.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.257+0000 7f70477fe640 1 -- 172.21.15.42:0/1986479228 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f704c0ab2f0 con 0x7f7050101c20 2024-07-26T06:51:51.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:51 smithi042 ceph-mon[32775]: pgmap v444: 97 pgs: 97 active+clean; 674 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 12 op/s 2024-07-26T06:51:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:51 smithi164 ceph-mon[36970]: pgmap v444: 97 pgs: 97 active+clean; 674 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 12 op/s 2024-07-26T06:51:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.497+0000 7f705871a640 1 -- 172.21.15.42:0/1986479228 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f705019afc0 con 0x7f7050101c20 2024-07-26T06:51:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.498+0000 7f70477fe640 1 -- 172.21.15.42:0/1986479228 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11978 (secure 0 0 0) 0x7f704c0aabf0 con 0x7f7050101c20 2024-07-26T06:51:51.499 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:51:51.499 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":29,"btime":"2024-07-26T06:50:27:493008+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:50:26.696932+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24620,24601,24593,24583,24578]},"id":1}]} 2024-07-26T06:51:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 29 2024-07-26T06:51:51.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.500+0000 7f705871a640 1 -- 172.21.15.42:0/1986479228 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f703c0b3bd0 msgr2=0x7f703c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:51.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.500+0000 7f705871a640 1 --2- 172.21.15.42:0/1986479228 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f703c0b3bd0 0x7f703c0b60e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7050107210 tx=0x7f704000b410 comp rx=0 tx=0).stop 2024-07-26T06:51:51.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.500+0000 7f705871a640 1 -- 172.21.15.42:0/1986479228 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7050101c20 msgr2=0x7f7050199e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:51.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.500+0000 7f705871a640 1 --2- 172.21.15.42:0/1986479228 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7050101c20 0x7f7050199e90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f704c00d950 tx=0x7f704c00de10 comp rx=0 tx=0).stop 2024-07-26T06:51:51.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.500+0000 7f705871a640 1 -- 172.21.15.42:0/1986479228 shutdown_connections 2024-07-26T06:51:51.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.500+0000 7f705871a640 1 --2- 172.21.15.42:0/1986479228 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f703c0b3bd0 0x7f703c0b60e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:51.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.500+0000 7f705871a640 1 --2- 172.21.15.42:0/1986479228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7050107eb0 0x7f705019f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:51.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.500+0000 7f705871a640 1 --2- 172.21.15.42:0/1986479228 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7050107560 0x7f705019a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:51.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.501+0000 7f705871a640 1 --2- 172.21.15.42:0/1986479228 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7050101c20 0x7f7050199e90 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:51.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.501+0000 7f705871a640 1 -- 172.21.15.42:0/1986479228 >> 172.21.15.42:0/1986479228 conn(0x7f7050077690 msgr2=0x7f70501055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:51.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.501+0000 7f705871a640 1 -- 172.21.15.42:0/1986479228 shutdown_connections 2024-07-26T06:51:51.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:51.501+0000 7f705871a640 1 -- 172.21.15.42:0/1986479228 wait complete. 2024-07-26T06:51:51.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:51 smithi163 ceph-mon[36873]: pgmap v444: 97 pgs: 97 active+clean; 674 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 12 op/s 2024-07-26T06:51:52.369 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:52 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/1986479228' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:51:52.397 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24620 quiesce db / --timeout 90 --expiration 142.55 --await-for 45.0' 2024-07-26T06:51:52.397 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.24620 quiesce db / --timeout 90 --expiration 142.55 --await-for 45.0 2024-07-26T06:51:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:52 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/1986479228' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:51:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:52 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/1986479228' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:51:52.664 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:51:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:53 smithi042 ceph-mon[32775]: pgmap v445: 97 pgs: 97 active+clean; 674 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T06:51:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:53 smithi164 ceph-mon[36970]: pgmap v445: 97 pgs: 97 active+clean; 674 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T06:51:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:53 smithi163 ceph-mon[36873]: pgmap v445: 97 pgs: 97 active+clean; 674 KiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T06:51:54.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.361+0000 7ff7d3825640 1 -- 172.21.15.42:0/2470579151 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7cc107e00 msgr2=0x7ff7cc108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:54.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.361+0000 7ff7d3825640 1 --2- 172.21.15.42:0/2470579151 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7cc107e00 0x7ff7cc108210 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009900 tx=0x7ff7bc02a410 comp rx=0 tx=0).stop 2024-07-26T06:51:54.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.361+0000 7ff7d3825640 1 -- 172.21.15.42:0/2470579151 shutdown_connections 2024-07-26T06:51:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.361+0000 7ff7d3825640 1 --2- 172.21.15.42:0/2470579151 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7cc108750 0x7ff7cc10ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.361+0000 7ff7d3825640 1 --2- 172.21.15.42:0/2470579151 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7cc107e00 0x7ff7cc108210 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.361+0000 7ff7d3825640 1 --2- 172.21.15.42:0/2470579151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7cc1024c0 0x7ff7cc1028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.361+0000 7ff7d3825640 1 -- 172.21.15.42:0/2470579151 >> 172.21.15.42:0/2470579151 conn(0x7ff7cc079250 msgr2=0x7ff7cc079660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.361+0000 7ff7d3825640 1 -- 172.21.15.42:0/2470579151 shutdown_connections 2024-07-26T06:51:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.361+0000 7ff7d3825640 1 -- 172.21.15.42:0/2470579151 wait complete. 2024-07-26T06:51:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.362+0000 7ff7d3825640 1 Processor -- start 2024-07-26T06:51:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.362+0000 7ff7d3825640 1 -- start start 2024-07-26T06:51:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d3825640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7cc1024c0 0x7ff7cc199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d3825640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7cc107e00 0x7ff7cc19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d3825640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7cc108750 0x7ff7cc19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d3825640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc1123e0 con 0x7ff7cc107e00 2024-07-26T06:51:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d3825640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc112260 con 0x7ff7cc108750 2024-07-26T06:51:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d3825640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc112560 con 0x7ff7cc1024c0 2024-07-26T06:51:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d0dc0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7cc107e00 0x7ff7cc19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d0dc0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7cc107e00 0x7ff7cc19a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60338/0 (socket says 172.21.15.42:60338) 2024-07-26T06:51:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d15c1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7cc1024c0 0x7ff7cc199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d0dc0640 1 -- 172.21.15.42:0/1811688989 learned_addr learned my addr 172.21.15.42:0/1811688989 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d1dc2640 1 --2- 172.21.15.42:0/1811688989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7cc108750 0x7ff7cc19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d0dc0640 1 -- 172.21.15.42:0/1811688989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7cc1024c0 msgr2=0x7ff7cc199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d0dc0640 1 --2- 172.21.15.42:0/1811688989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7cc1024c0 0x7ff7cc199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d0dc0640 1 -- 172.21.15.42:0/1811688989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7cc108750 msgr2=0x7ff7cc19f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d0dc0640 1 --2- 172.21.15.42:0/1811688989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7cc108750 0x7ff7cc19f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d0dc0640 1 -- 172.21.15.42:0/1811688989 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7cc19fe50 con 0x7ff7cc107e00 2024-07-26T06:51:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d1dc2640 1 --2- 172.21.15.42:0/1811688989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7cc108750 0x7ff7cc19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:51:54.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.363+0000 7ff7d0dc0640 1 --2- 172.21.15.42:0/1811688989 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7cc107e00 0x7ff7cc19a480 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc02a930 tx=0x7ff7bc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:54.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.364+0000 7ff7c27fc640 1 -- 172.21.15.42:0/1811688989 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc03f030 con 0x7ff7cc107e00 2024-07-26T06:51:54.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.364+0000 7ff7d3825640 1 -- 172.21.15.42:0/1811688989 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7cc1a00e0 con 0x7ff7cc107e00 2024-07-26T06:51:54.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.364+0000 7ff7c27fc640 1 -- 172.21.15.42:0/1811688989 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7bc003aa0 con 0x7ff7cc107e00 2024-07-26T06:51:54.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.364+0000 7ff7c27fc640 1 -- 172.21.15.42:0/1811688989 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc035070 con 0x7ff7cc107e00 2024-07-26T06:51:54.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.364+0000 7ff7d3825640 1 -- 172.21.15.42:0/1811688989 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7cc1a68a0 con 0x7ff7cc107e00 2024-07-26T06:51:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.367+0000 7ff7d9da9740 1 Processor -- start 2024-07-26T06:51:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.367+0000 7ff7d9da9740 1 -- start start 2024-07-26T06:51:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.367+0000 7ff7d9da9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a93850 0x557e69e39540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.367+0000 7ff7d9da9740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a6a990 0x557e69e39a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.367+0000 7ff7d9da9740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a72b70 0x557e69e39fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.367+0000 7ff7d9da9740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x557e69dae6c0 con 0x557e69a72b70 2024-07-26T06:51:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.367+0000 7ff7d9da9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557e69d9dbc0 con 0x557e69a93850 2024-07-26T06:51:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.367+0000 7ff7d9da9740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x557e69da1c00 con 0x557e69a6a990 2024-07-26T06:51:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.367+0000 7ff7d15c1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a6a990 0x557e69e39a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.367+0000 7ff7d1dc2640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a93850 0x557e69e39540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.367+0000 7ff7d15c1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a6a990 0x557e69e39a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:49918/0 (socket says 172.21.15.42:49918) 2024-07-26T06:51:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.367+0000 7ff7d15c1640 1 -- 172.21.15.42:0/3682326034 learned_addr learned my addr 172.21.15.42:0/3682326034 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.368+0000 7ff7d15c1640 1 -- 172.21.15.42:0/3682326034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a93850 msgr2=0x557e69e39540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:54.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.368+0000 7ff7d15c1640 1 --2- 172.21.15.42:0/3682326034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a93850 0x557e69e39540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.368+0000 7ff7d15c1640 1 -- 172.21.15.42:0/3682326034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a72b70 msgr2=0x557e69e39fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T06:51:54.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.368+0000 7ff7d15c1640 1 --2- 172.21.15.42:0/3682326034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a72b70 0x557e69e39fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.368+0000 7ff7d15c1640 1 -- 172.21.15.42:0/3682326034 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e69d29a40 con 0x557e69a6a990 2024-07-26T06:51:54.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.368+0000 7ff7c27fc640 1 -- 172.21.15.42:0/1811688989 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7ff7bc042030 con 0x7ff7cc107e00 2024-07-26T06:51:54.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.368+0000 7ff7d15c1640 1 --2- 172.21.15.42:0/3682326034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a6a990 0x557e69e39a80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800ac20 tx=0x7ff7c800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5eb4c894e2e7a1d0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:54.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.368+0000 7ff7b77fe640 1 -- 172.21.15.42:0/3682326034 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c801d070 con 0x557e69a6a990 2024-07-26T06:51:54.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.368+0000 7ff7b77fe640 1 -- 172.21.15.42:0/3682326034 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c8018af0 con 0x557e69a6a990 2024-07-26T06:51:54.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.369+0000 7ff7b77fe640 1 -- 172.21.15.42:0/3682326034 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8010ad0 con 0x557e69a6a990 2024-07-26T06:51:54.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.369+0000 7ff7c27fc640 1 --2- 172.21.15.42:0/1811688989 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff7b80b3bd0 0x7ff7b80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.369+0000 7ff7d1dc2640 1 --2- 172.21.15.42:0/1811688989 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff7b80b3bd0 0x7ff7b80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:54.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.369+0000 7ff7c27fc640 1 -- 172.21.15.42:0/1811688989 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7ff7bc104140 con 0x7ff7cc107e00 2024-07-26T06:51:54.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.370+0000 7ff7d9da9740 1 -- 172.21.15.42:0/3682326034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a6a990 msgr2=0x557e69e39a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:54.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.370+0000 7ff7d1dc2640 1 --2- 172.21.15.42:0/1811688989 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff7b80b3bd0 0x7ff7b80b60e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4005fd0 tx=0x7ff7c4009800 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:54.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.370+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/3682326034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a6a990 0x557e69e39a80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800ac20 tx=0x7ff7c800f790 comp rx=0 tx=0).stop 2024-07-26T06:51:54.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.370+0000 7ff7d9da9740 1 -- 172.21.15.42:0/3682326034 shutdown_connections 2024-07-26T06:51:54.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.370+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/3682326034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a72b70 0x557e69e39fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.370+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/3682326034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a6a990 0x557e69e39a80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.370+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/3682326034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a93850 0x557e69e39540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.370+0000 7ff7d9da9740 1 -- 172.21.15.42:0/3682326034 >> 172.21.15.42:0/3682326034 conn(0x557e69a87c40 msgr2=0x557e69e37430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:54.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.370+0000 7ff7d9da9740 1 -- 172.21.15.42:0/3682326034 shutdown_connections 2024-07-26T06:51:54.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.370+0000 7ff7d9da9740 1 -- 172.21.15.42:0/3682326034 wait complete. 2024-07-26T06:51:54.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.370+0000 7ff7d9da9740 1 -- start start 2024-07-26T06:51:54.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.370+0000 7ff7d9da9740 1 Processor -- start 2024-07-26T06:51:54.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d9da9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a6a990 0x557e69e49040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d9da9740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a72b70 0x557e69e49580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d9da9740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a93850 0x557e69e553e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d9da9740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x557e69c017b0 con 0x557e69a72b70 2024-07-26T06:51:54.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d9da9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557e69d40240 con 0x557e69a6a990 2024-07-26T06:51:54.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d9da9740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x557e69e4c080 con 0x557e69a93850 2024-07-26T06:51:54.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d9da9740 10 client.0 _ll_register_callbacks cb 0x557e69d212f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:54.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d0dc0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a72b70 0x557e69e49580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:54.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d1dc2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a93850 0x557e69e553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:54.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d0dc0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a72b70 0x557e69e49580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60346/0 (socket says 172.21.15.42:60346) 2024-07-26T06:51:54.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d1dc2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a93850 0x557e69e553e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:49930/0 (socket says 172.21.15.42:49930) 2024-07-26T06:51:54.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d0dc0640 1 -- 172.21.15.42:0/2321242816 learned_addr learned my addr 172.21.15.42:0/2321242816 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:54.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d15c1640 1 --2- 172.21.15.42:0/2321242816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a6a990 0x557e69e49040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:54.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d1dc2640 1 -- 172.21.15.42:0/2321242816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a6a990 msgr2=0x557e69e49040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:54.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d1dc2640 1 --2- 172.21.15.42:0/2321242816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a6a990 0x557e69e49040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d1dc2640 1 -- 172.21.15.42:0/2321242816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a72b70 msgr2=0x557e69e49580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:54.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d1dc2640 1 --2- 172.21.15.42:0/2321242816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a72b70 0x557e69e49580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.371+0000 7ff7d1dc2640 1 -- 172.21.15.42:0/2321242816 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e69d51740 con 0x557e69a93850 2024-07-26T06:51:54.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.372+0000 7ff7d1dc2640 1 --2- 172.21.15.42:0/2321242816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a93850 0x557e69e553e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4011cc0 tx=0x7ff7c4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eac25545cdd2f987 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:54.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.372+0000 7ff7b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-07-26T06:51:54.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.372+0000 7ff7b4ff9640 1 -- 172.21.15.42:0/2321242816 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c401ed30 con 0x557e69a93850 2024-07-26T06:51:54.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.372+0000 7ff7d9da9740 1 -- 172.21.15.42:0/2321242816 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557e69db0610 con 0x557e69a93850 2024-07-26T06:51:54.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.372+0000 7ff7b4ff9640 1 -- 172.21.15.42:0/2321242816 <== mon.1 v2:172.21.15.164:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7ff7c402a480 con 0x557e69a93850 2024-07-26T06:51:54.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.372+0000 7ff7b4ff9640 1 -- 172.21.15.42:0/2321242816 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff7c4011e70 con 0x557e69a93850 2024-07-26T06:51:54.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.372+0000 7ff7d9da9740 10 client.25285 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:54.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.372+0000 7ff7d9da9740 1 -- 172.21.15.42:0/2321242816 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557e69b0dba0 con 0x557e69a93850 2024-07-26T06:51:54.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.372+0000 7ff7b4ff9640 1 -- 172.21.15.42:0/2321242816 <== mon.1 v2:172.21.15.164:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x557e69db0610 con 0x557e69a93850 2024-07-26T06:51:54.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.372+0000 7ff7b4ff9640 1 -- 172.21.15.42:0/2321242816 <== mon.1 v2:172.21.15.164:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff7c400f040 con 0x557e69a93850 2024-07-26T06:51:54.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.373+0000 7ff7d9da9740 10 client.25285 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:54.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.373+0000 7ff7d9da9740 10 client.25285 resolve_mds: validated gid 24620 aka daemon mds.b 2024-07-26T06:51:54.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.373+0000 7ff7d9da9740 20 client.25285 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:54.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.373+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/2321242816 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x557e69a521d0 0x557e69e5b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.373+0000 7ff7d9da9740 4 client.25285 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:51:54.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.373+0000 7ff7d9da9740 1 -- 172.21.15.42:0/2321242816 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e69dac0b0 con 0x557e69a521d0 2024-07-26T06:51:54.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.373+0000 7ff7d15c1640 1 --2- 172.21.15.42:0/2321242816 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x557e69a521d0 0x557e69e5b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:54.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.373+0000 7ff7b4ff9640 10 client.25285 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:54.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.373+0000 7ff7d15c1640 1 --2- 172.21.15.42:0/2321242816 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x557e69a521d0 0x557e69e5b1e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c53ea30e1198658 server_cookie=2100fa5c8a665964 in_seq=0 out_seq=0 2024-07-26T06:51:54.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.374+0000 7ff7b4ff9640 1 -- 172.21.15.42:0/2321242816 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557e69dac0b0 con 0x557e69a521d0 2024-07-26T06:51:54.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.374+0000 7ff7b4ff9640 10 client.25285 handle_command_reply: tid=0 2024-07-26T06:51:54.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.374+0000 7ff7d9da9740 1 client.25285 shutdown 2024-07-26T06:51:54.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.374+0000 7ff7d9da9740 1 -- 172.21.15.42:0/2321242816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a93850 msgr2=0x557e69e553e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:54.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.374+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/2321242816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a93850 0x557e69e553e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4011cc0 tx=0x7ff7c4004a40 comp rx=0 tx=0).stop 2024-07-26T06:51:54.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.374+0000 7ff7d9da9740 1 -- 172.21.15.42:0/2321242816 shutdown_connections 2024-07-26T06:51:54.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.374+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/2321242816 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x557e69a521d0 0x557e69e5b1e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.374+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/2321242816 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a93850 0x557e69e553e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.374+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/2321242816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a72b70 0x557e69e49580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.374+0000 7ff7b4ff9640 0 client.25285 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:54.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.374+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/2321242816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a6a990 0x557e69e49040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.374+0000 7ff7d9da9740 1 -- 172.21.15.42:0/2321242816 >> 172.21.15.42:0/2321242816 conn(0x557e69a87c40 msgr2=0x557e69e3af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:54.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.375+0000 7ff7d9da9740 1 -- 172.21.15.42:0/2321242816 shutdown_connections 2024-07-26T06:51:54.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.375+0000 7ff7d9da9740 1 -- 172.21.15.42:0/2321242816 wait complete. 2024-07-26T06:51:54.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.375+0000 7ff7d9da9740 20 client.25285 trim_cache size 0 max 16384 2024-07-26T06:51:54.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.392+0000 7ff7d9da9740 1 Processor -- start 2024-07-26T06:51:54.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.392+0000 7ff7d9da9740 1 -- start start 2024-07-26T06:51:54.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.392+0000 7ff7d9da9740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a6a990 0x557e69e3ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.392+0000 7ff7d9da9740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a72b70 0x557e69e551d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.392+0000 7ff7d9da9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a93850 0x557e69e55710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.392+0000 7ff7d9da9740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc1acd00 con 0x557e69a6a990 2024-07-26T06:51:54.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.392+0000 7ff7d9da9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc1ae840 con 0x557e69a93850 2024-07-26T06:51:54.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.392+0000 7ff7d9da9740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc1ad3d0 con 0x557e69a72b70 2024-07-26T06:51:54.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.392+0000 7ff7d15c1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a6a990 0x557e69e3ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:54.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.392+0000 7ff7d15c1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a6a990 0x557e69e3ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60356/0 (socket says 172.21.15.42:60356) 2024-07-26T06:51:54.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.392+0000 7ff7d15c1640 1 -- 172.21.15.42:0/1165023626 learned_addr learned my addr 172.21.15.42:0/1165023626 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:54.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.392+0000 7ff7d0dc0640 1 --2- 172.21.15.42:0/1165023626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a72b70 0x557e69e551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:54.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.393+0000 7ff7d15c1640 1 -- 172.21.15.42:0/1165023626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a72b70 msgr2=0x557e69e551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:54.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.393+0000 7ff7d1dc2640 1 --2- 172.21.15.42:0/1165023626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a93850 0x557e69e55710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:54.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.393+0000 7ff7d15c1640 1 --2- 172.21.15.42:0/1165023626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a72b70 0x557e69e551d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.393+0000 7ff7d15c1640 1 -- 172.21.15.42:0/1165023626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a93850 msgr2=0x557e69e55710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:54.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.393+0000 7ff7d15c1640 1 --2- 172.21.15.42:0/1165023626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a93850 0x557e69e55710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.393+0000 7ff7d15c1640 1 -- 172.21.15.42:0/1165023626 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e69e32910 con 0x557e69a6a990 2024-07-26T06:51:54.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.393+0000 7ff7d1dc2640 1 --2- 172.21.15.42:0/1165023626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a93850 0x557e69e55710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:51:54.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.393+0000 7ff7d15c1640 1 --2- 172.21.15.42:0/1165023626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a6a990 0x557e69e3ac80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800ab70 tx=0x7ff7c800a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e64e4d0f2414b1b5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:54.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.393+0000 7ff7b6ffd640 1 -- 172.21.15.42:0/1165023626 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c801d040 con 0x557e69a6a990 2024-07-26T06:51:54.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.393+0000 7ff7b6ffd640 1 -- 172.21.15.42:0/1165023626 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c8029070 con 0x557e69a6a990 2024-07-26T06:51:54.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.393+0000 7ff7b6ffd640 1 -- 172.21.15.42:0/1165023626 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8010d50 con 0x557e69a6a990 2024-07-26T06:51:54.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.394+0000 7ff7d9da9740 1 -- 172.21.15.42:0/1165023626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a6a990 msgr2=0x557e69e3ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:54.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.394+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/1165023626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a6a990 0x557e69e3ac80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800ab70 tx=0x7ff7c800a2d0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.394+0000 7ff7d9da9740 1 -- 172.21.15.42:0/1165023626 shutdown_connections 2024-07-26T06:51:54.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.394+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/1165023626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a93850 0x557e69e55710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.394+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/1165023626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a72b70 0x557e69e551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.394+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/1165023626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a6a990 0x557e69e3ac80 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.394+0000 7ff7d9da9740 1 -- 172.21.15.42:0/1165023626 >> 172.21.15.42:0/1165023626 conn(0x557e69a87c40 msgr2=0x557e69e3a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:54.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.394+0000 7ff7d9da9740 1 -- 172.21.15.42:0/1165023626 shutdown_connections 2024-07-26T06:51:54.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.394+0000 7ff7d9da9740 1 -- 172.21.15.42:0/1165023626 wait complete. 2024-07-26T06:51:54.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.394+0000 7ff7d9da9740 1 -- start start 2024-07-26T06:51:54.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.394+0000 7ff7d9da9740 1 Processor -- start 2024-07-26T06:51:54.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d9da9740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a6a990 0x557e69e4d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d9da9740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a72b70 0x557e69e4da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d9da9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a93850 0x557e69ec2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d9da9740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc1ae170 con 0x557e69a6a990 2024-07-26T06:51:54.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d9da9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc1adaa0 con 0x557e69a93850 2024-07-26T06:51:54.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d9da9740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc1aace0 con 0x557e69a72b70 2024-07-26T06:51:54.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d9da9740 10 client.0 _ll_register_callbacks cb 0x7ff7c4009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:54.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d15c1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a6a990 0x557e69e4d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:54.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d15c1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a6a990 0x557e69e4d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60360/0 (socket says 172.21.15.42:60360) 2024-07-26T06:51:54.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d15c1640 1 -- 172.21.15.42:0/4245354725 learned_addr learned my addr 172.21.15.42:0/4245354725 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:54.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d1dc2640 1 --2- 172.21.15.42:0/4245354725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a93850 0x557e69ec2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:54.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d0dc0640 1 --2- 172.21.15.42:0/4245354725 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a72b70 0x557e69e4da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:54.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d15c1640 1 -- 172.21.15.42:0/4245354725 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a72b70 msgr2=0x557e69e4da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:54.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d15c1640 1 --2- 172.21.15.42:0/4245354725 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a72b70 0x557e69e4da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d15c1640 1 -- 172.21.15.42:0/4245354725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a93850 msgr2=0x557e69ec2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:54.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d15c1640 1 --2- 172.21.15.42:0/4245354725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a93850 0x557e69ec2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:54.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d15c1640 1 -- 172.21.15.42:0/4245354725 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e69e4e2f0 con 0x557e69a6a990 2024-07-26T06:51:54.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.395+0000 7ff7d0dc0640 1 --2- 172.21.15.42:0/4245354725 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a72b70 0x557e69e4da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:51:54.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.396+0000 7ff7d15c1640 1 --2- 172.21.15.42:0/4245354725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a6a990 0x557e69e4d530 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8005c50 tx=0x7ff7c8010760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99b8232ca93c9b3b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:54.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.396+0000 7ff7b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:51:54.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.396+0000 7ff7b4ff9640 1 -- 172.21.15.42:0/4245354725 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c803b060 con 0x557e69a6a990 2024-07-26T06:51:54.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.396+0000 7ff7b4ff9640 1 -- 172.21.15.42:0/4245354725 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7ff7c8011d00 con 0x557e69a6a990 2024-07-26T06:51:54.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.396+0000 7ff7d9da9740 1 -- 172.21.15.42:0/4245354725 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7c8029070 con 0x557e69a6a990 2024-07-26T06:51:54.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.396+0000 7ff7b4ff9640 1 -- 172.21.15.42:0/4245354725 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff7c801d040 con 0x557e69a6a990 2024-07-26T06:51:54.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.396+0000 7ff7d9da9740 10 client.15618 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:54.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.396+0000 7ff7d9da9740 1 -- 172.21.15.42:0/4245354725 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557e69ec3dd0 con 0x557e69a6a990 2024-07-26T06:51:54.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.396+0000 7ff7b4ff9640 1 -- 172.21.15.42:0/4245354725 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c8029070 con 0x557e69a6a990 2024-07-26T06:51:54.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.396+0000 7ff7b4ff9640 1 -- 172.21.15.42:0/4245354725 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff7c8004b20 con 0x557e69a6a990 2024-07-26T06:51:54.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.397+0000 7ff7d9da9740 10 client.15618 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:54.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.397+0000 7ff7d9da9740 10 client.15618 resolve_mds: validated gid 24620 aka daemon mds.b 2024-07-26T06:51:54.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.397+0000 7ff7d9da9740 20 client.15618 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:54.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.397+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/4245354725 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x557e69a27580 0x557e69ec6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:54.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.397+0000 7ff7d9da9740 4 client.15618 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.55, "await_for": 45.0}] 2024-07-26T06:51:54.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.397+0000 7ff7d9da9740 1 -- 172.21.15.42:0/4245354725 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.55, "await_for": 45.0}) -- 0x557e69da97f0 con 0x557e69a27580 2024-07-26T06:51:54.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.397+0000 7ff7d0dc0640 1 --2- 172.21.15.42:0/4245354725 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x557e69a27580 0x557e69ec6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:54.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.397+0000 7ff7d0dc0640 1 --2- 172.21.15.42:0/4245354725 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x557e69a27580 0x557e69ec6400 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d622ba31ab65c17 server_cookie=c3d1a2f8e1019eae in_seq=0 out_seq=0 2024-07-26T06:51:54.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:54.397+0000 7ff7b4ff9640 10 client.15618 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:55.377 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:55 smithi042 ceph-mon[32775]: pgmap v446: 97 pgs: 97 active+clean; 13 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1020 KiB/s wr, 10 op/s 2024-07-26T06:51:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:55 smithi164 ceph-mon[36970]: pgmap v446: 97 pgs: 97 active+clean; 13 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1020 KiB/s wr, 10 op/s 2024-07-26T06:51:55.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:55 smithi163 ceph-mon[36873]: pgmap v446: 97 pgs: 97 active+clean; 13 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1020 KiB/s wr, 10 op/s 2024-07-26T06:51:56.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7b4ff9640 1 -- 172.21.15.42:0/4245354725 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x557e69da97f0 con 0x557e69a27580 2024-07-26T06:51:56.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7b4ff9640 10 client.15618 handle_command_reply: tid=0 2024-07-26T06:51:56.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7d9da9740 1 client.15618 shutdown 2024-07-26T06:51:56.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7d9da9740 1 -- 172.21.15.42:0/4245354725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a6a990 msgr2=0x557e69e4d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:56.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/4245354725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a6a990 0x557e69e4d530 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8005c50 tx=0x7ff7c8010760 comp rx=0 tx=0).stop 2024-07-26T06:51:56.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7d9da9740 1 -- 172.21.15.42:0/4245354725 shutdown_connections 2024-07-26T06:51:56.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/4245354725 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x557e69a27580 0x557e69ec6400 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:56.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/4245354725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557e69a93850 0x557e69ec2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:56.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/4245354725 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557e69a72b70 0x557e69e4da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:56.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7d9da9740 1 --2- 172.21.15.42:0/4245354725 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x557e69a6a990 0x557e69e4d530 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:56.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7d9da9740 1 -- 172.21.15.42:0/4245354725 >> 172.21.15.42:0/4245354725 conn(0x557e69a87c40 msgr2=0x557e69e3a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:56.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7b4ff9640 0 client.15618 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:56.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7d9da9740 1 -- 172.21.15.42:0/4245354725 shutdown_connections 2024-07-26T06:51:56.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7d9da9740 1 -- 172.21.15.42:0/4245354725 wait complete. 2024-07-26T06:51:56.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7d9da9740 20 client.15618 trim_cache size 0 max 16384 2024-07-26T06:51:56.211 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:51:56.211 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 29, 2024-07-26T06:51:56.212 INFO:teuthology.orchestra.run.smithi042.stdout: "leader": 24620, 2024-07-26T06:51:56.212 INFO:teuthology.orchestra.run.smithi042.stdout: "set_version": 6, 2024-07-26T06:51:56.212 INFO:teuthology.orchestra.run.smithi042.stdout: "sets": { 2024-07-26T06:51:56.212 INFO:teuthology.orchestra.run.smithi042.stdout: "c6bd3542": { 2024-07-26T06:51:56.212 INFO:teuthology.orchestra.run.smithi042.stdout: "version": 6, 2024-07-26T06:51:56.212 INFO:teuthology.orchestra.run.smithi042.stdout: "age_ref": 0.0, 2024-07-26T06:51:56.212 INFO:teuthology.orchestra.run.smithi042.stdout: "state": { 2024-07-26T06:51:56.212 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "QUIESCED", 2024-07-26T06:51:56.212 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 0.0 2024-07-26T06:51:56.213 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-07-26T06:51:56.213 INFO:teuthology.orchestra.run.smithi042.stdout: "timeout": 90.0, 2024-07-26T06:51:56.213 INFO:teuthology.orchestra.run.smithi042.stdout: "expiration": 142.6, 2024-07-26T06:51:56.213 INFO:teuthology.orchestra.run.smithi042.stdout: "members": { 2024-07-26T06:51:56.213 INFO:teuthology.orchestra.run.smithi042.stdout: "file:/": { 2024-07-26T06:51:56.213 INFO:teuthology.orchestra.run.smithi042.stdout: "excluded": false, 2024-07-26T06:51:56.213 INFO:teuthology.orchestra.run.smithi042.stdout: "state": { 2024-07-26T06:51:56.213 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "QUIESCED", 2024-07-26T06:51:56.213 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 0.0 2024-07-26T06:51:56.213 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:51:56.214 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:51:56.214 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:51:56.214 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:51:56.214 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:51:56.214 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:51:56.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7d3825640 1 -- 172.21.15.42:0/1811688989 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff7b80b3bd0 msgr2=0x7ff7b80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:56.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7d3825640 1 --2- 172.21.15.42:0/1811688989 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff7b80b3bd0 0x7ff7b80b60e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4005fd0 tx=0x7ff7c4009800 comp rx=0 tx=0).stop 2024-07-26T06:51:56.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7d3825640 1 -- 172.21.15.42:0/1811688989 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7cc107e00 msgr2=0x7ff7cc19a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:56.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.208+0000 7ff7d3825640 1 --2- 172.21.15.42:0/1811688989 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7cc107e00 0x7ff7cc19a480 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc02a930 tx=0x7ff7bc004270 comp rx=0 tx=0).stop 2024-07-26T06:51:56.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.209+0000 7ff7d3825640 1 -- 172.21.15.42:0/1811688989 shutdown_connections 2024-07-26T06:51:56.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.209+0000 7ff7d3825640 1 --2- 172.21.15.42:0/1811688989 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff7b80b3bd0 0x7ff7b80b60e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:56.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.209+0000 7ff7d3825640 1 --2- 172.21.15.42:0/1811688989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff7cc108750 0x7ff7cc19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:56.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.209+0000 7ff7d3825640 1 --2- 172.21.15.42:0/1811688989 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff7cc107e00 0x7ff7cc19a480 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:56.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.209+0000 7ff7d3825640 1 --2- 172.21.15.42:0/1811688989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7cc1024c0 0x7ff7cc199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:56.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.209+0000 7ff7d3825640 1 -- 172.21.15.42:0/1811688989 >> 172.21.15.42:0/1811688989 conn(0x7ff7cc079250 msgr2=0x7ff7cc06b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:56.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.209+0000 7ff7d3825640 1 -- 172.21.15.42:0/1811688989 shutdown_connections 2024-07-26T06:51:56.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:51:56.209+0000 7ff7d3825640 1 -- 172.21.15.42:0/1811688989 wait complete. 2024-07-26T06:51:56.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:56 smithi042 ceph-mon[32775]: pgmap v447: 97 pgs: 97 active+clean; 13 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 1.0 MiB/s wr, 13 op/s 2024-07-26T06:51:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:56 smithi164 ceph-mon[36970]: pgmap v447: 97 pgs: 97 active+clean; 13 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 1.0 MiB/s wr, 13 op/s 2024-07-26T06:51:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:56 smithi163 ceph-mon[36873]: pgmap v447: 97 pgs: 97 active+clean; 13 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 1.0 MiB/s wr, 13 op/s 2024-07-26T06:51:56.967 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c6bd3542', quiesce took 7.8 seconds. Will release after: 14.75 2024-07-26T06:51:58.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:51:58 smithi164 ceph-mon[36970]: pgmap v448: 97 pgs: 97 active+clean; 14 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-07-26T06:51:58.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:51:58 smithi163 ceph-mon[36873]: pgmap v448: 97 pgs: 97 active+clean; 14 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-07-26T06:51:58.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:51:58 smithi042 ceph-mon[32775]: pgmap v448: 97 pgs: 97 active+clean; 14 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-07-26T06:51:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.768+0000 7fb9d7062640 1 -- 172.21.15.42:0/4247318067 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9d010daf0 msgr2=0x7fb9d010df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d7062640 1 --2- 172.21.15.42:0/4247318067 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9d010daf0 0x7fb9d010df60 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc009900 tx=0x7fb9cc02a410 comp rx=0 tx=0).stop 2024-07-26T06:51:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d7062640 1 -- 172.21.15.42:0/4247318067 shutdown_connections 2024-07-26T06:51:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d7062640 1 --2- 172.21.15.42:0/4247318067 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9d010f510 0x7fb9d0115eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d7062640 1 --2- 172.21.15.42:0/4247318067 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9d010eb70 0x7fb9d010ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d7062640 1 --2- 172.21.15.42:0/4247318067 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9d010daf0 0x7fb9d010df60 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d7062640 1 -- 172.21.15.42:0/4247318067 >> 172.21.15.42:0/4247318067 conn(0x7fb9d006dd50 msgr2=0x7fb9d010be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d7062640 1 -- 172.21.15.42:0/4247318067 shutdown_connections 2024-07-26T06:51:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d7062640 1 -- 172.21.15.42:0/4247318067 wait complete. 2024-07-26T06:51:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d7062640 1 Processor -- start 2024-07-26T06:51:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d7062640 1 -- start start 2024-07-26T06:51:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d7062640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9d010daf0 0x7fb9d01a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d7062640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9d010eb70 0x7fb9d01a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d7062640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9d010f510 0x7fb9d01a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d7062640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d0119760 con 0x7fb9d010eb70 2024-07-26T06:51:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d7062640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d01195e0 con 0x7fb9d010daf0 2024-07-26T06:51:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d7062640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d01198e0 con 0x7fb9d010f510 2024-07-26T06:51:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9d010eb70 0x7fb9d01a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9d010eb70 0x7fb9d01a4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53990/0 (socket says 172.21.15.42:53990) 2024-07-26T06:51:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9c7fff640 1 -- 172.21.15.42:0/3130815202 learned_addr learned my addr 172.21.15.42:0/3130815202 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d55ff640 1 --2- 172.21.15.42:0/3130815202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9d010f510 0x7fb9d01a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d4dfe640 1 --2- 172.21.15.42:0/3130815202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9d010daf0 0x7fb9d01a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9c7fff640 1 -- 172.21.15.42:0/3130815202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9d010f510 msgr2=0x7fb9d01a55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9c7fff640 1 --2- 172.21.15.42:0/3130815202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9d010f510 0x7fb9d01a55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9c7fff640 1 -- 172.21.15.42:0/3130815202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9d010daf0 msgr2=0x7fb9d01a43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9c7fff640 1 --2- 172.21.15.42:0/3130815202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9d010daf0 0x7fb9d01a43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9c7fff640 1 -- 172.21.15.42:0/3130815202 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d01a5df0 con 0x7fb9d010eb70 2024-07-26T06:51:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d55ff640 1 --2- 172.21.15.42:0/3130815202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9d010f510 0x7fb9d01a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:51:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9d4dfe640 1 --2- 172.21.15.42:0/3130815202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9d010daf0 0x7fb9d01a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:51:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9c7fff640 1 --2- 172.21.15.42:0/3130815202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9d010eb70 0x7fb9d01a4a20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb9c000d950 tx=0x7fb9c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.770+0000 7fb9c5ffb640 1 -- 172.21.15.42:0/3130815202 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0017070 con 0x7fb9d010eb70 2024-07-26T06:51:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.771+0000 7fb9c5ffb640 1 -- 172.21.15.42:0/3130815202 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9c0009c30 con 0x7fb9d010eb70 2024-07-26T06:51:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.771+0000 7fb9c5ffb640 1 -- 172.21.15.42:0/3130815202 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c000f620 con 0x7fb9d010eb70 2024-07-26T06:51:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.771+0000 7fb9d7062640 1 -- 172.21.15.42:0/3130815202 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9d01a60e0 con 0x7fb9d010eb70 2024-07-26T06:51:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.771+0000 7fb9d7062640 1 -- 172.21.15.42:0/3130815202 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9d01ac7d0 con 0x7fb9d010eb70 2024-07-26T06:51:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.773+0000 7fb9ddf7d740 1 Processor -- start 2024-07-26T06:51:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.773+0000 7fb9ddf7d740 1 -- start start 2024-07-26T06:51:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9c5ffb640 1 -- 172.21.15.42:0/3130815202 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fb9c0010460 con 0x7fb9d010eb70 2024-07-26T06:51:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9ddf7d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 0x55dc767a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9ddf7d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763d72f0 0x55dc767a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9ddf7d740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9ddf7d740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55dc767183b0 con 0x55dc763df4d0 2024-07-26T06:51:59.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9ddf7d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55dc767078b0 con 0x55dc763d72f0 2024-07-26T06:51:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9ddf7d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55dc7670b8f0 con 0x55dc764001b0 2024-07-26T06:51:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53992/0 (socket says 172.21.15.42:53992) 2024-07-26T06:51:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9c7fff640 1 -- 172.21.15.42:0/2754055578 learned_addr learned my addr 172.21.15.42:0/2754055578 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9d55ff640 1 --2- 172.21.15.42:0/2754055578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 0x55dc767a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9d4dfe640 1 --2- 172.21.15.42:0/2754055578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763d72f0 0x55dc767a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9d55ff640 1 -- 172.21.15.42:0/2754055578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763d72f0 msgr2=0x55dc767a51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9d55ff640 1 --2- 172.21.15.42:0/2754055578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763d72f0 0x55dc767a51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9d55ff640 1 -- 172.21.15.42:0/2754055578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 msgr2=0x55dc767a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9d55ff640 1 --2- 172.21.15.42:0/2754055578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9d55ff640 1 -- 172.21.15.42:0/2754055578 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc766ae9a0 con 0x55dc764001b0 2024-07-26T06:51:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.774+0000 7fb9c7fff640 1 --2- 172.21.15.42:0/2754055578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:51:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.775+0000 7fb9d55ff640 1 --2- 172.21.15.42:0/2754055578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 0x55dc767a4b50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009ee0 tx=0x7fb9c8010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cc5470d04643ab server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.775+0000 7fb9c5ffb640 1 --2- 172.21.15.42:0/3130815202 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb9b80b3c50 0x7fb9b80b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.775+0000 7fb9b6ffd640 1 -- 172.21.15.42:0/2754055578 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8016070 con 0x55dc764001b0 2024-07-26T06:51:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.775+0000 7fb9d4dfe640 1 --2- 172.21.15.42:0/3130815202 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb9b80b3c50 0x7fb9b80b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.775+0000 7fb9b6ffd640 1 -- 172.21.15.42:0/2754055578 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9c801bb10 con 0x55dc764001b0 2024-07-26T06:51:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.775+0000 7fb9b6ffd640 1 -- 172.21.15.42:0/2754055578 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8012c00 con 0x55dc764001b0 2024-07-26T06:51:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.775+0000 7fb9c5ffb640 1 -- 172.21.15.42:0/3130815202 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fb9c00e3890 con 0x7fb9d010eb70 2024-07-26T06:51:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.776+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/2754055578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 msgr2=0x55dc767a4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.776+0000 7fb9d4dfe640 1 --2- 172.21.15.42:0/3130815202 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb9b80b3c50 0x7fb9b80b6160 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc000c00 tx=0x7fb9cc005a20 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.776+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/2754055578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 0x55dc767a4b50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009ee0 tx=0x7fb9c8010740 comp rx=0 tx=0).stop 2024-07-26T06:51:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.776+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/2754055578 shutdown_connections 2024-07-26T06:51:59.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.776+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/2754055578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.776+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/2754055578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763d72f0 0x55dc767a51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.776+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/2754055578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 0x55dc767a4b50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.776+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/2754055578 >> 172.21.15.42:0/2754055578 conn(0x55dc763f4300 msgr2=0x55dc767a05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.776+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/2754055578 shutdown_connections 2024-07-26T06:51:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.776+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/2754055578 wait complete. 2024-07-26T06:51:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.776+0000 7fb9ddf7d740 1 -- start start 2024-07-26T06:51:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.776+0000 7fb9ddf7d740 1 Processor -- start 2024-07-26T06:51:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.777+0000 7fb9ddf7d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc763d72f0 0x55dc767ae520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.777+0000 7fb9ddf7d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763df4d0 0x55dc767aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.777+0000 7fb9ddf7d740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc764001b0 0x55dc767af750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.777+0000 7fb9ddf7d740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55dc76641ab0 con 0x55dc764001b0 2024-07-26T06:51:59.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.777+0000 7fb9ddf7d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55dc7668fcf0 con 0x55dc763df4d0 2024-07-26T06:51:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.777+0000 7fb9ddf7d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55dc767a6830 con 0x55dc763d72f0 2024-07-26T06:51:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.777+0000 7fb9ddf7d740 10 client.0 _ll_register_callbacks cb 0x55dc767013e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.777+0000 7fb9d55ff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc764001b0 0x55dc767af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.777+0000 7fb9d55ff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc764001b0 0x55dc767af750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53994/0 (socket says 172.21.15.42:53994) 2024-07-26T06:51:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.777+0000 7fb9d55ff640 1 -- 172.21.15.42:0/2198247221 learned_addr learned my addr 172.21.15.42:0/2198247221 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.777+0000 7fb9d4dfe640 1 --2- 172.21.15.42:0/2198247221 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc763d72f0 0x55dc767ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.777+0000 7fb9c7fff640 1 --2- 172.21.15.42:0/2198247221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763df4d0 0x55dc767aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.777+0000 7fb9d55ff640 1 -- 172.21.15.42:0/2198247221 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc763d72f0 msgr2=0x55dc767ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.777+0000 7fb9d55ff640 1 --2- 172.21.15.42:0/2198247221 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc763d72f0 0x55dc767ae520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.777+0000 7fb9d55ff640 1 -- 172.21.15.42:0/2198247221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763df4d0 msgr2=0x55dc767aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:59.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.777+0000 7fb9d55ff640 1 --2- 172.21.15.42:0/2198247221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763df4d0 0x55dc767aebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.777+0000 7fb9d55ff640 1 -- 172.21.15.42:0/2198247221 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc7646b790 con 0x55dc764001b0 2024-07-26T06:51:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.778+0000 7fb9d55ff640 1 --2- 172.21.15.42:0/2198247221 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc764001b0 0x55dc767af750 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009ca0 tx=0x7fb9c8012680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0751e85f07d2d85 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.778+0000 7fb9affff640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:51:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.778+0000 7fb9affff640 1 -- 172.21.15.42:0/2198247221 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8029060 con 0x55dc764001b0 2024-07-26T06:51:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.778+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/2198247221 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9c801bb10 con 0x55dc764001b0 2024-07-26T06:51:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.778+0000 7fb9affff640 1 -- 172.21.15.42:0/2198247221 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fb9c8004d80 con 0x55dc764001b0 2024-07-26T06:51:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.778+0000 7fb9affff640 1 -- 172.21.15.42:0/2198247221 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb9c8016070 con 0x55dc764001b0 2024-07-26T06:51:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.778+0000 7fb9ddf7d740 10 client.15630 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.778+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/2198247221 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dc764e4620 con 0x55dc764001b0 2024-07-26T06:51:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.778+0000 7fb9affff640 1 -- 172.21.15.42:0/2198247221 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9c801bb10 con 0x55dc764001b0 2024-07-26T06:51:59.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.778+0000 7fb9affff640 1 -- 172.21.15.42:0/2198247221 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb9c800f040 con 0x55dc764001b0 2024-07-26T06:51:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.778+0000 7fb9ddf7d740 10 client.15630 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.778+0000 7fb9ddf7d740 10 client.15630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:51:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.778+0000 7fb9ddf7d740 20 client.15630 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.779+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/2198247221 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55dc763bed80 0x55dc767bb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.779+0000 7fb9ddf7d740 4 client.15630 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:51:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.779+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/2198247221 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc7671a300 con 0x55dc763bed80 2024-07-26T06:51:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.779+0000 7fb9d4dfe640 1 --2- 172.21.15.42:0/2198247221 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55dc763bed80 0x55dc767bb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.779+0000 7fb9affff640 10 client.15630 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:59.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.779+0000 7fb9d4dfe640 1 --2- 172.21.15.42:0/2198247221 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55dc763bed80 0x55dc767bb940 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f52ca1da35bd6649 server_cookie=3d623343fe9f5feb in_seq=0 out_seq=0 2024-07-26T06:51:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.780+0000 7fb9affff640 1 -- 172.21.15.42:0/2198247221 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dc7671a300 con 0x55dc763bed80 2024-07-26T06:51:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.780+0000 7fb9affff640 10 client.15630 handle_command_reply: tid=0 2024-07-26T06:51:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.780+0000 7fb9ddf7d740 1 client.15630 shutdown 2024-07-26T06:51:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.780+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/2198247221 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc764001b0 msgr2=0x55dc767af750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:59.788 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.780+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/2198247221 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc764001b0 0x55dc767af750 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009ca0 tx=0x7fb9c8012680 comp rx=0 tx=0).stop 2024-07-26T06:51:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.780+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/2198247221 shutdown_connections 2024-07-26T06:51:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.780+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/2198247221 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55dc763bed80 0x55dc767bb940 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.780+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/2198247221 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc764001b0 0x55dc767af750 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.780+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/2198247221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763df4d0 0x55dc767aebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.780+0000 7fb9affff640 0 client.15630 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.780+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/2198247221 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc763d72f0 0x55dc767ae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.780+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/2198247221 >> 172.21.15.42:0/2198247221 conn(0x55dc763f4300 msgr2=0x55dc7679ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.780+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/2198247221 shutdown_connections 2024-07-26T06:51:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.781+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/2198247221 wait complete. 2024-07-26T06:51:59.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.781+0000 7fb9ddf7d740 20 client.15630 trim_cache size 0 max 16384 2024-07-26T06:51:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9ddf7d740 1 Processor -- start 2024-07-26T06:51:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9ddf7d740 1 -- start start 2024-07-26T06:51:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9ddf7d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763d72f0 0x55dc767ade50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9ddf7d740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767b6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9ddf7d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 0x55dc767a4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9ddf7d740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d01b5510 con 0x55dc763df4d0 2024-07-26T06:51:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9ddf7d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d01b7050 con 0x55dc763d72f0 2024-07-26T06:51:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9ddf7d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d01b5be0 con 0x55dc764001b0 2024-07-26T06:51:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767b6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767b6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53998/0 (socket says 172.21.15.42:53998) 2024-07-26T06:51:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9c7fff640 1 -- 172.21.15.42:0/379741194 learned_addr learned my addr 172.21.15.42:0/379741194 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9d4dfe640 1 --2- 172.21.15.42:0/379741194 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763d72f0 0x55dc767ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9d55ff640 1 --2- 172.21.15.42:0/379741194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 0x55dc767a4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9c7fff640 1 -- 172.21.15.42:0/379741194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 msgr2=0x55dc767a4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9c7fff640 1 --2- 172.21.15.42:0/379741194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 0x55dc767a4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9c7fff640 1 -- 172.21.15.42:0/379741194 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763d72f0 msgr2=0x55dc767ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9c7fff640 1 --2- 172.21.15.42:0/379741194 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763d72f0 0x55dc767ade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9c7fff640 1 -- 172.21.15.42:0/379741194 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc767a0f60 con 0x55dc763df4d0 2024-07-26T06:51:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9d55ff640 1 --2- 172.21.15.42:0/379741194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 0x55dc767a4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:51:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9d4dfe640 1 --2- 172.21.15.42:0/379741194 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763d72f0 0x55dc767ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:51:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9c7fff640 1 --2- 172.21.15.42:0/379741194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767b6630 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00a7a30 tx=0x7fb9c00e9550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a09e527a71faf735 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9b67fc640 1 -- 172.21.15.42:0/379741194 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0020070 con 0x55dc763df4d0 2024-07-26T06:51:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9b67fc640 1 -- 172.21.15.42:0/379741194 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9c00b31b0 con 0x55dc763df4d0 2024-07-26T06:51:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.797+0000 7fb9b67fc640 1 -- 172.21.15.42:0/379741194 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c00aa7c0 con 0x55dc763df4d0 2024-07-26T06:51:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.798+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/379741194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 msgr2=0x55dc767b6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.798+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/379741194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767b6630 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00a7a30 tx=0x7fb9c00e9550 comp rx=0 tx=0).stop 2024-07-26T06:51:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.798+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/379741194 shutdown_connections 2024-07-26T06:51:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.798+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/379741194 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 0x55dc767a4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.798+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/379741194 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767b6630 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.798+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/379741194 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763d72f0 0x55dc767ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.798+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/379741194 >> 172.21.15.42:0/379741194 conn(0x55dc763f4300 msgr2=0x55dc767ad740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.798+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/379741194 shutdown_connections 2024-07-26T06:51:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.798+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/379741194 wait complete. 2024-07-26T06:51:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.798+0000 7fb9ddf7d740 1 -- start start 2024-07-26T06:51:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.798+0000 7fb9ddf7d740 1 Processor -- start 2024-07-26T06:51:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9ddf7d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763d72f0 0x55dc767b5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9ddf7d740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9ddf7d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 0x55dc76822000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9ddf7d740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d01b6980 con 0x55dc763df4d0 2024-07-26T06:51:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9ddf7d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d01b62b0 con 0x55dc763d72f0 2024-07-26T06:51:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9ddf7d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d01b34f0 con 0x55dc764001b0 2024-07-26T06:51:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9ddf7d740 10 client.0 _ll_register_callbacks cb 0x7fb9c00e9420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:51:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767b5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54000/0 (socket says 172.21.15.42:54000) 2024-07-26T06:51:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9c7fff640 1 -- 172.21.15.42:0/4151095611 learned_addr learned my addr 172.21.15.42:0/4151095611 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:51:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9d4dfe640 1 --2- 172.21.15.42:0/4151095611 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763d72f0 0x55dc767b5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9d55ff640 1 --2- 172.21.15.42:0/4151095611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 0x55dc76822000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9c7fff640 1 -- 172.21.15.42:0/4151095611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 msgr2=0x55dc76822000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9c7fff640 1 --2- 172.21.15.42:0/4151095611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 0x55dc76822000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9c7fff640 1 -- 172.21.15.42:0/4151095611 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763d72f0 msgr2=0x55dc767b5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:51:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9c7fff640 1 --2- 172.21.15.42:0/4151095611 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763d72f0 0x55dc767b5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9c7fff640 1 -- 172.21.15.42:0/4151095611 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc767b6080 con 0x55dc763df4d0 2024-07-26T06:51:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.799+0000 7fb9d55ff640 1 --2- 172.21.15.42:0/4151095611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 0x55dc76822000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:51:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.800+0000 7fb9c7fff640 1 --2- 172.21.15.42:0/4151095611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767b5790 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0012040 tx=0x7fb9c00a6c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9bc0e336ba447b0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:51:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.800+0000 7fb9affff640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:51:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.800+0000 7fb9affff640 1 -- 172.21.15.42:0/4151095611 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c00bfe60 con 0x55dc763df4d0 2024-07-26T06:51:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.800+0000 7fb9affff640 1 -- 172.21.15.42:0/4151095611 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fb9c00cd2d0 con 0x55dc763df4d0 2024-07-26T06:51:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.800+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/4151095611 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dc76715da0 con 0x55dc763df4d0 2024-07-26T06:51:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.800+0000 7fb9affff640 1 -- 172.21.15.42:0/4151095611 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb9c0020050 con 0x55dc763df4d0 2024-07-26T06:51:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.800+0000 7fb9ddf7d740 10 client.15636 fetch_fsmap learned FSMap version 29 2024-07-26T06:51:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.800+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/4151095611 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dc768235a0 con 0x55dc763df4d0 2024-07-26T06:51:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.800+0000 7fb9affff640 1 -- 172.21.15.42:0/4151095611 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55dc76715da0 con 0x55dc763df4d0 2024-07-26T06:51:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.800+0000 7fb9affff640 1 -- 172.21.15.42:0/4151095611 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb9c00abaf0 con 0x55dc763df4d0 2024-07-26T06:51:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.800+0000 7fb9ddf7d740 10 client.15636 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T06:51:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.800+0000 7fb9ddf7d740 10 client.15636 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:51:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.800+0000 7fb9ddf7d740 20 client.15636 populate_metadata read hostname 'smithi042' 2024-07-26T06:51:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.800+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/4151095611 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55dc763941c0 0x55dc76825fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:51:59.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.800+0000 7fb9ddf7d740 4 client.15636 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:51:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.800+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/4151095611 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dc767134e0 con 0x55dc763941c0 2024-07-26T06:51:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.801+0000 7fb9d4dfe640 1 --2- 172.21.15.42:0/4151095611 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55dc763941c0 0x55dc76825fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:51:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.801+0000 7fb9d4dfe640 1 --2- 172.21.15.42:0/4151095611 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55dc763941c0 0x55dc76825fc0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bff90126e9941c61 server_cookie=e36d5ed83529addb in_seq=0 out_seq=0 2024-07-26T06:51:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.801+0000 7fb9affff640 10 client.15636 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:59.907 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.905+0000 7fb9affff640 1 -- 172.21.15.42:0/4151095611 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dc767134e0 con 0x55dc763941c0 2024-07-26T06:51:59.907 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.905+0000 7fb9affff640 10 client.15636 handle_command_reply: tid=0 2024-07-26T06:51:59.907 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.905+0000 7fb9ddf7d740 1 client.15636 shutdown 2024-07-26T06:51:59.908 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.905+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/4151095611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 msgr2=0x55dc767b5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:59.908 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.905+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/4151095611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767b5790 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0012040 tx=0x7fb9c00a6c80 comp rx=0 tx=0).stop 2024-07-26T06:51:59.908 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.905+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/4151095611 shutdown_connections 2024-07-26T06:51:59.908 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.905+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/4151095611 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55dc763941c0 0x55dc76825fc0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.908 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.905+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/4151095611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55dc764001b0 0x55dc76822000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.908 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.905+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/4151095611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55dc763df4d0 0x55dc767b5790 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.909 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.905+0000 7fb9affff640 0 client.15636 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:51:59.909 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.905+0000 7fb9ddf7d740 1 --2- 172.21.15.42:0/4151095611 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dc763d72f0 0x55dc767b5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.909 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.905+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/4151095611 >> 172.21.15.42:0/4151095611 conn(0x55dc763f4300 msgr2=0x55dc767ac8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:59.909 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.905+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/4151095611 shutdown_connections 2024-07-26T06:51:59.909 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.905+0000 7fb9ddf7d740 1 -- 172.21.15.42:0/4151095611 wait complete. 2024-07-26T06:51:59.909 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.905+0000 7fb9ddf7d740 20 client.15636 trim_cache size 0 max 16384 2024-07-26T06:51:59.909 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:51:59.909 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:51:59.909 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:51:59.909 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:51:59.910 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.906+0000 7fb9d7062640 1 -- 172.21.15.42:0/3130815202 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb9b80b3c50 msgr2=0x7fb9b80b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:59.910 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.906+0000 7fb9d7062640 1 --2- 172.21.15.42:0/3130815202 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb9b80b3c50 0x7fb9b80b6160 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc000c00 tx=0x7fb9cc005a20 comp rx=0 tx=0).stop 2024-07-26T06:51:59.911 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.906+0000 7fb9d7062640 1 -- 172.21.15.42:0/3130815202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9d010eb70 msgr2=0x7fb9d01a4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:51:59.911 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.906+0000 7fb9d7062640 1 --2- 172.21.15.42:0/3130815202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9d010eb70 0x7fb9d01a4a20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb9c000d950 tx=0x7fb9c000de10 comp rx=0 tx=0).stop 2024-07-26T06:51:59.911 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.906+0000 7fb9d7062640 1 -- 172.21.15.42:0/3130815202 shutdown_connections 2024-07-26T06:51:59.911 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.906+0000 7fb9d7062640 1 --2- 172.21.15.42:0/3130815202 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fb9b80b3c50 0x7fb9b80b6160 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.911 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.906+0000 7fb9d7062640 1 --2- 172.21.15.42:0/3130815202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb9d010f510 0x7fb9d01a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.911 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.906+0000 7fb9d7062640 1 --2- 172.21.15.42:0/3130815202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb9d010eb70 0x7fb9d01a4a20 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.911 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.906+0000 7fb9d7062640 1 --2- 172.21.15.42:0/3130815202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb9d010daf0 0x7fb9d01a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:51:59.912 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.906+0000 7fb9d7062640 1 -- 172.21.15.42:0/3130815202 >> 172.21.15.42:0/3130815202 conn(0x7fb9d006dd50 msgr2=0x7fb9d0111e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:51:59.912 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.906+0000 7fb9d7062640 1 -- 172.21.15.42:0/3130815202 shutdown_connections 2024-07-26T06:51:59.912 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:51:59.906+0000 7fb9d7062640 1 -- 172.21.15.42:0/3130815202 wait complete. 2024-07-26T06:52:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:00 smithi164 ceph-mon[36970]: pgmap v449: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 1.3 MiB/s wr, 46 op/s 2024-07-26T06:52:00.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:00 smithi163 ceph-mon[36873]: pgmap v449: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 1.3 MiB/s wr, 46 op/s 2024-07-26T06:52:00.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:00 smithi042 ceph-mon[32775]: pgmap v449: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 1.3 MiB/s wr, 46 op/s 2024-07-26T06:52:02.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:02 smithi164 ceph-mon[36970]: pgmap v450: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 1.3 MiB/s wr, 58 op/s 2024-07-26T06:52:02.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:02 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:02.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:02 smithi163 ceph-mon[36873]: pgmap v450: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 1.3 MiB/s wr, 58 op/s 2024-07-26T06:52:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:02 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:02.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:02 smithi042 ceph-mon[32775]: pgmap v450: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 1.3 MiB/s wr, 58 op/s 2024-07-26T06:52:02.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:02 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:04.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:04 smithi164 ceph-mon[36970]: pgmap v451: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-07-26T06:52:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:04 smithi163 ceph-mon[36873]: pgmap v451: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-07-26T06:52:04.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:04 smithi042 ceph-mon[32775]: pgmap v451: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 1.3 MiB/s wr, 84 op/s 2024-07-26T06:52:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:06 smithi164 ceph-mon[36970]: pgmap v452: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 343 KiB/s wr, 122 op/s 2024-07-26T06:52:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:06 smithi163 ceph-mon[36873]: pgmap v452: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 343 KiB/s wr, 122 op/s 2024-07-26T06:52:06.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:06 smithi042 ceph-mon[32775]: pgmap v452: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 343 KiB/s wr, 122 op/s 2024-07-26T06:52:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:08 smithi163 ceph-mon[36873]: pgmap v453: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 315 KiB/s wr, 119 op/s 2024-07-26T06:52:08.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:08 smithi042 ceph-mon[32775]: pgmap v453: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 315 KiB/s wr, 119 op/s 2024-07-26T06:52:08.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:08 smithi164 ceph-mon[36970]: pgmap v453: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 315 KiB/s wr, 119 op/s 2024-07-26T06:52:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:52:09.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:52:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:52:10.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:10 smithi163 ceph-mon[36873]: pgmap v454: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 186 KiB/s wr, 120 op/s 2024-07-26T06:52:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:52:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:52:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:52:10.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:10 smithi042 ceph-mon[32775]: pgmap v454: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 186 KiB/s wr, 120 op/s 2024-07-26T06:52:10.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:52:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:52:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:52:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:10 smithi164 ceph-mon[36970]: pgmap v454: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 186 KiB/s wr, 120 op/s 2024-07-26T06:52:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:52:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:52:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:52:11.718 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:52:12.000 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:52:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:12 smithi163 ceph-mon[36873]: pgmap v455: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 KiB/s wr, 86 op/s 2024-07-26T06:52:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:12 smithi042 ceph-mon[32775]: pgmap v455: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 KiB/s wr, 86 op/s 2024-07-26T06:52:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:12 smithi164 ceph-mon[36970]: pgmap v455: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 33 KiB/s wr, 86 op/s 2024-07-26T06:52:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.015+0000 7f6e8226f640 1 -- 172.21.15.42:0/1224271738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c104180 msgr2=0x7f6e7c104590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.015+0000 7f6e8226f640 1 --2- 172.21.15.42:0/1224271738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c104180 0x7f6e7c104590 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009900 tx=0x7f6e6402a410 comp rx=0 tx=0).stop 2024-07-26T06:52:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.015+0000 7f6e8226f640 1 -- 172.21.15.42:0/1224271738 shutdown_connections 2024-07-26T06:52:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.015+0000 7f6e8226f640 1 --2- 172.21.15.42:0/1224271738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e7c104b20 0x7f6e7c10b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.015+0000 7f6e8226f640 1 --2- 172.21.15.42:0/1224271738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c104180 0x7f6e7c104590 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.015+0000 7f6e8226f640 1 --2- 172.21.15.42:0/1224271738 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e7c103100 0x7f6e7c103570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.015+0000 7f6e8226f640 1 -- 172.21.15.42:0/1224271738 >> 172.21.15.42:0/1224271738 conn(0x7f6e7c0fea60 msgr2=0x7f6e7c100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.015+0000 7f6e8226f640 1 -- 172.21.15.42:0/1224271738 shutdown_connections 2024-07-26T06:52:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.015+0000 7f6e8226f640 1 -- 172.21.15.42:0/1224271738 wait complete. 2024-07-26T06:52:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e8226f640 1 Processor -- start 2024-07-26T06:52:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e8226f640 1 -- start start 2024-07-26T06:52:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e8226f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e7c103100 0x7f6e7c076880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e8226f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e7c104180 0x7f6e7c075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e8226f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c104b20 0x7f6e7c075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e8226f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7c10edd0 con 0x7f6e7c104b20 2024-07-26T06:52:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e8226f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7c10ec50 con 0x7f6e7c104180 2024-07-26T06:52:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e8226f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7c10ef50 con 0x7f6e7c103100 2024-07-26T06:52:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e8080c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c104b20 0x7f6e7c075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e8080c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c104b20 0x7f6e7c075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48270/0 (socket says 172.21.15.42:48270) 2024-07-26T06:52:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e8080c640 1 -- 172.21.15.42:0/3772110969 learned_addr learned my addr 172.21.15.42:0/3772110969 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e7bfff640 1 --2- 172.21.15.42:0/3772110969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e7c103100 0x7f6e7c076880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e73fff640 1 --2- 172.21.15.42:0/3772110969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e7c104180 0x7f6e7c075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e8080c640 1 -- 172.21.15.42:0/3772110969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e7c103100 msgr2=0x7f6e7c076880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e8080c640 1 --2- 172.21.15.42:0/3772110969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e7c103100 0x7f6e7c076880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e8080c640 1 -- 172.21.15.42:0/3772110969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e7c104180 msgr2=0x7f6e7c075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e8080c640 1 --2- 172.21.15.42:0/3772110969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e7c104180 0x7f6e7c075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e8080c640 1 -- 172.21.15.42:0/3772110969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e7c075db0 con 0x7f6e7c104b20 2024-07-26T06:52:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.016+0000 7f6e7bfff640 1 --2- 172.21.15.42:0/3772110969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e7c103100 0x7f6e7c076880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:52:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.017+0000 7f6e8080c640 1 --2- 172.21.15.42:0/3772110969 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c104b20 0x7f6e7c075680 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6e74009fd0 tx=0x7f6e7400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.017+0000 7f6e79ffb640 1 -- 172.21.15.42:0/3772110969 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e74017070 con 0x7f6e7c104b20 2024-07-26T06:52:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.017+0000 7f6e79ffb640 1 -- 172.21.15.42:0/3772110969 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e74003e20 con 0x7f6e7c104b20 2024-07-26T06:52:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.017+0000 7f6e79ffb640 1 -- 172.21.15.42:0/3772110969 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e74010460 con 0x7f6e7c104b20 2024-07-26T06:52:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.017+0000 7f6e8226f640 1 -- 172.21.15.42:0/3772110969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e7c1a3a40 con 0x7f6e7c104b20 2024-07-26T06:52:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.017+0000 7f6e8226f640 1 -- 172.21.15.42:0/3772110969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e7c1a3eb0 con 0x7f6e7c104b20 2024-07-26T06:52:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.020+0000 7f6e89189740 1 Processor -- start 2024-07-26T06:52:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.020+0000 7f6e79ffb640 1 -- 172.21.15.42:0/3772110969 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f6e74003970 con 0x7f6e7c104b20 2024-07-26T06:52:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.020+0000 7f6e89189740 1 -- start start 2024-07-26T06:52:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.020+0000 7f6e89189740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55630398e1b0 0x556303d32b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.020+0000 7f6e89189740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5563039652f0 0x556303d331c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.020+0000 7f6e89189740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55630396d4d0 0x556303d33d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.020+0000 7f6e89189740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x556303ca63b0 con 0x55630396d4d0 2024-07-26T06:52:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.020+0000 7f6e89189740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556303c958b0 con 0x55630398e1b0 2024-07-26T06:52:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.020+0000 7f6e89189740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x556303c998f0 con 0x5563039652f0 2024-07-26T06:52:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.020+0000 7f6e73fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55630396d4d0 0x556303d33d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.020+0000 7f6e73fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55630396d4d0 0x556303d33d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48274/0 (socket says 172.21.15.42:48274) 2024-07-26T06:52:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.020+0000 7f6e73fff640 1 -- 172.21.15.42:0/692349107 learned_addr learned my addr 172.21.15.42:0/692349107 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.020+0000 7f6e7bfff640 1 --2- 172.21.15.42:0/692349107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5563039652f0 0x556303d331c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.021+0000 7f6e73fff640 1 --2- 172.21.15.42:0/692349107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55630398e1b0 0x556303d32b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.021+0000 7f6e7bfff640 1 -- 172.21.15.42:0/692349107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55630398e1b0 msgr2=0x556303d32b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.021+0000 7f6e7bfff640 1 --2- 172.21.15.42:0/692349107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55630398e1b0 0x556303d32b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.021+0000 7f6e7bfff640 1 -- 172.21.15.42:0/692349107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55630396d4d0 msgr2=0x556303d33d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.021+0000 7f6e79ffb640 1 --2- 172.21.15.42:0/3772110969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e680b3bd0 0x7f6e680b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.021+0000 7f6e7bfff640 1 --2- 172.21.15.42:0/692349107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55630396d4d0 0x556303d33d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.021+0000 7f6e7bfff640 1 -- 172.21.15.42:0/692349107 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556303c3c9a0 con 0x5563039652f0 2024-07-26T06:52:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.021+0000 7f6e73fff640 1 --2- 172.21.15.42:0/692349107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55630398e1b0 0x556303d32b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:52:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.021+0000 7f6e8080c640 1 --2- 172.21.15.42:0/3772110969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e680b3bd0 0x7f6e680b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.021+0000 7f6e7bfff640 1 --2- 172.21.15.42:0/692349107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5563039652f0 0x556303d331c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00bc80 tx=0x7f6e6c00f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=723eaed319963f6c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.021+0000 7f6e79ffb640 1 -- 172.21.15.42:0/3772110969 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f6e740e3a90 con 0x7f6e7c104b20 2024-07-26T06:52:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.021+0000 7f6e727fc640 1 -- 172.21.15.42:0/692349107 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c015070 con 0x5563039652f0 2024-07-26T06:52:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.021+0000 7f6e727fc640 1 -- 172.21.15.42:0/692349107 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e6c0118f0 con 0x5563039652f0 2024-07-26T06:52:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.021+0000 7f6e727fc640 1 -- 172.21.15.42:0/692349107 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c010870 con 0x5563039652f0 2024-07-26T06:52:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.022+0000 7f6e89189740 1 -- 172.21.15.42:0/692349107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5563039652f0 msgr2=0x556303d331c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.022+0000 7f6e8080c640 1 --2- 172.21.15.42:0/3772110969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e680b3bd0 0x7f6e680b60e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6e740aa6c0 tx=0x7f6e740a4f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.022+0000 7f6e89189740 1 --2- 172.21.15.42:0/692349107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5563039652f0 0x556303d331c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00bc80 tx=0x7f6e6c00f730 comp rx=0 tx=0).stop 2024-07-26T06:52:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.022+0000 7f6e89189740 1 -- 172.21.15.42:0/692349107 shutdown_connections 2024-07-26T06:52:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.022+0000 7f6e89189740 1 --2- 172.21.15.42:0/692349107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55630396d4d0 0x556303d33d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.022+0000 7f6e89189740 1 --2- 172.21.15.42:0/692349107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5563039652f0 0x556303d331c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.022+0000 7f6e89189740 1 --2- 172.21.15.42:0/692349107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55630398e1b0 0x556303d32b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.022+0000 7f6e89189740 1 -- 172.21.15.42:0/692349107 >> 172.21.15.42:0/692349107 conn(0x556303982300 msgr2=0x556303d2e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.022+0000 7f6e89189740 1 -- 172.21.15.42:0/692349107 shutdown_connections 2024-07-26T06:52:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.022+0000 7f6e89189740 1 -- 172.21.15.42:0/692349107 wait complete. 2024-07-26T06:52:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.023+0000 7f6e89189740 1 -- start start 2024-07-26T06:52:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.023+0000 7f6e89189740 1 Processor -- start 2024-07-26T06:52:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.023+0000 7f6e89189740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563039652f0 0x556303d3c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.023+0000 7f6e89189740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55630396d4d0 0x556303d3cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.023+0000 7f6e89189740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55630398e1b0 0x556303d3d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.023+0000 7f6e89189740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x556303bcfab0 con 0x55630398e1b0 2024-07-26T06:52:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.023+0000 7f6e89189740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556303c1dcf0 con 0x5563039652f0 2024-07-26T06:52:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.023+0000 7f6e89189740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x556303d34830 con 0x55630396d4d0 2024-07-26T06:52:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.023+0000 7f6e89189740 10 client.0 _ll_register_callbacks cb 0x556303c8f3e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:52:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.023+0000 7f6e73fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55630398e1b0 0x556303d3d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.024+0000 7f6e73fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55630398e1b0 0x556303d3d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48290/0 (socket says 172.21.15.42:48290) 2024-07-26T06:52:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.024+0000 7f6e73fff640 1 -- 172.21.15.42:0/666950904 learned_addr learned my addr 172.21.15.42:0/666950904 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.024+0000 7f6e73fff640 1 --2- 172.21.15.42:0/666950904 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563039652f0 0x556303d3c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.024+0000 7f6e7bfff640 1 --2- 172.21.15.42:0/666950904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55630396d4d0 0x556303d3cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.024+0000 7f6e73fff640 1 -- 172.21.15.42:0/666950904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55630396d4d0 msgr2=0x556303d3cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.024+0000 7f6e73fff640 1 --2- 172.21.15.42:0/666950904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55630396d4d0 0x556303d3cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.024+0000 7f6e73fff640 1 -- 172.21.15.42:0/666950904 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563039652f0 msgr2=0x556303d3c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.024+0000 7f6e73fff640 1 --2- 172.21.15.42:0/666950904 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563039652f0 0x556303d3c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.024+0000 7f6e73fff640 1 -- 172.21.15.42:0/666950904 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563039f9790 con 0x55630398e1b0 2024-07-26T06:52:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.024+0000 7f6e73fff640 1 --2- 172.21.15.42:0/666950904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55630398e1b0 0x556303d3d740 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6e64006010 tx=0x7f6e6402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f8fd607ba1e312 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.024+0000 7f6e5b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:52:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.024+0000 7f6e5b7fe640 1 -- 172.21.15.42:0/666950904 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e64035040 con 0x55630398e1b0 2024-07-26T06:52:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.024+0000 7f6e5b7fe640 1 -- 172.21.15.42:0/666950904 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f6e64005020 con 0x55630398e1b0 2024-07-26T06:52:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.024+0000 7f6e89189740 1 -- 172.21.15.42:0/666950904 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e6c0118f0 con 0x55630398e1b0 2024-07-26T06:52:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.024+0000 7f6e5b7fe640 1 -- 172.21.15.42:0/666950904 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6e64051070 con 0x55630398e1b0 2024-07-26T06:52:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.024+0000 7f6e89189740 10 client.15648 fetch_fsmap learned FSMap version 30 2024-07-26T06:52:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.024+0000 7f6e89189740 1 -- 172.21.15.42:0/666950904 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556303a72620 con 0x55630398e1b0 2024-07-26T06:52:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.025+0000 7f6e5b7fe640 1 -- 172.21.15.42:0/666950904 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e6c0118f0 con 0x55630398e1b0 2024-07-26T06:52:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.025+0000 7f6e5b7fe640 1 -- 172.21.15.42:0/666950904 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f6e6403a040 con 0x55630398e1b0 2024-07-26T06:52:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.025+0000 7f6e89189740 10 client.15648 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T06:52:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.025+0000 7f6e89189740 10 client.15648 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:52:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.025+0000 7f6e89189740 20 client.15648 populate_metadata read hostname 'smithi042' 2024-07-26T06:52:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.025+0000 7f6e89189740 1 --2- 172.21.15.42:0/666950904 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55630394cd80 0x556303d49c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.025+0000 7f6e89189740 4 client.15648 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:52:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.025+0000 7f6e89189740 1 -- 172.21.15.42:0/666950904 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556303ca8300 con 0x55630394cd80 2024-07-26T06:52:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.025+0000 7f6e7bfff640 1 --2- 172.21.15.42:0/666950904 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55630394cd80 0x556303d49c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.026+0000 7f6e7bfff640 1 --2- 172.21.15.42:0/666950904 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55630394cd80 0x556303d49c90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9e103d760052ea6 server_cookie=8fa11c5e7618edb0 in_seq=0 out_seq=0 2024-07-26T06:52:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.026+0000 7f6e5b7fe640 10 client.15648 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.026+0000 7f6e5b7fe640 1 -- 172.21.15.42:0/666950904 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556303ca8300 con 0x55630394cd80 2024-07-26T06:52:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.026+0000 7f6e5b7fe640 10 client.15648 handle_command_reply: tid=0 2024-07-26T06:52:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.027+0000 7f6e89189740 1 client.15648 shutdown 2024-07-26T06:52:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.027+0000 7f6e89189740 1 -- 172.21.15.42:0/666950904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55630398e1b0 msgr2=0x556303d3d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.027+0000 7f6e89189740 1 --2- 172.21.15.42:0/666950904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55630398e1b0 0x556303d3d740 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6e64006010 tx=0x7f6e6402a410 comp rx=0 tx=0).stop 2024-07-26T06:52:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.027+0000 7f6e89189740 1 -- 172.21.15.42:0/666950904 shutdown_connections 2024-07-26T06:52:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.027+0000 7f6e89189740 1 --2- 172.21.15.42:0/666950904 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55630394cd80 0x556303d49c90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.027+0000 7f6e89189740 1 --2- 172.21.15.42:0/666950904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55630398e1b0 0x556303d3d740 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.027+0000 7f6e89189740 1 --2- 172.21.15.42:0/666950904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55630396d4d0 0x556303d3cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.027+0000 7f6e89189740 1 --2- 172.21.15.42:0/666950904 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563039652f0 0x556303d3c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.027+0000 7f6e5b7fe640 0 client.15648 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.027+0000 7f6e89189740 1 -- 172.21.15.42:0/666950904 >> 172.21.15.42:0/666950904 conn(0x556303982300 msgr2=0x556303d2cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.027+0000 7f6e89189740 1 -- 172.21.15.42:0/666950904 shutdown_connections 2024-07-26T06:52:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.027+0000 7f6e89189740 1 -- 172.21.15.42:0/666950904 wait complete. 2024-07-26T06:52:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.027+0000 7f6e89189740 20 client.15648 trim_cache size 0 max 16384 2024-07-26T06:52:13.045 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.042+0000 7f6e89189740 1 Processor -- start 2024-07-26T06:52:13.045 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.042+0000 7f6e89189740 1 -- start start 2024-07-26T06:52:13.045 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e89189740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5563039652f0 0x556303d32810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.045 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e89189740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55630396d4d0 0x556303d32d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.045 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e89189740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55630398e1b0 0x556303d4e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e89189740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7c1acb70 con 0x5563039652f0 2024-07-26T06:52:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e89189740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7c1ae6e0 con 0x55630398e1b0 2024-07-26T06:52:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e89189740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7c1ad240 con 0x55630396d4d0 2024-07-26T06:52:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e73fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5563039652f0 0x556303d32810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e73fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5563039652f0 0x556303d32810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48306/0 (socket says 172.21.15.42:48306) 2024-07-26T06:52:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e73fff640 1 -- 172.21.15.42:0/1942916035 learned_addr learned my addr 172.21.15.42:0/1942916035 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e7bfff640 1 --2- 172.21.15.42:0/1942916035 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55630396d4d0 0x556303d32d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e73fff640 1 -- 172.21.15.42:0/1942916035 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55630396d4d0 msgr2=0x556303d32d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e73fff640 1 --2- 172.21.15.42:0/1942916035 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55630396d4d0 0x556303d32d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e73fff640 1 -- 172.21.15.42:0/1942916035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55630398e1b0 msgr2=0x556303d4e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e73fff640 1 --2- 172.21.15.42:0/1942916035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55630398e1b0 0x556303d4e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e73fff640 1 -- 172.21.15.42:0/1942916035 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556303d3c2a0 con 0x5563039652f0 2024-07-26T06:52:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e7bfff640 1 --2- 172.21.15.42:0/1942916035 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55630396d4d0 0x556303d32d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:52:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e73fff640 1 --2- 172.21.15.42:0/1942916035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5563039652f0 0x556303d32810 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6e64000c00 tx=0x7f6e64009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eff203f8340324d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e71ffb640 1 -- 172.21.15.42:0/1942916035 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6405b070 con 0x5563039652f0 2024-07-26T06:52:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e71ffb640 1 -- 172.21.15.42:0/1942916035 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e64031c60 con 0x5563039652f0 2024-07-26T06:52:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.043+0000 7f6e71ffb640 1 -- 172.21.15.42:0/1942916035 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e64035070 con 0x5563039652f0 2024-07-26T06:52:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.044+0000 7f6e89189740 1 -- 172.21.15.42:0/1942916035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5563039652f0 msgr2=0x556303d32810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.044+0000 7f6e89189740 1 --2- 172.21.15.42:0/1942916035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5563039652f0 0x556303d32810 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6e64000c00 tx=0x7f6e64009230 comp rx=0 tx=0).stop 2024-07-26T06:52:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.044+0000 7f6e89189740 1 -- 172.21.15.42:0/1942916035 shutdown_connections 2024-07-26T06:52:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.044+0000 7f6e89189740 1 --2- 172.21.15.42:0/1942916035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55630398e1b0 0x556303d4e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.044+0000 7f6e89189740 1 --2- 172.21.15.42:0/1942916035 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55630396d4d0 0x556303d32d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.044+0000 7f6e89189740 1 --2- 172.21.15.42:0/1942916035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5563039652f0 0x556303d32810 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.044+0000 7f6e89189740 1 -- 172.21.15.42:0/1942916035 >> 172.21.15.42:0/1942916035 conn(0x556303982300 msgr2=0x556303d37560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.044+0000 7f6e89189740 1 -- 172.21.15.42:0/1942916035 shutdown_connections 2024-07-26T06:52:13.048 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.044+0000 7f6e89189740 1 -- 172.21.15.42:0/1942916035 wait complete. 2024-07-26T06:52:13.048 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.044+0000 7f6e89189740 1 -- start start 2024-07-26T06:52:13.048 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.045+0000 7f6e89189740 1 Processor -- start 2024-07-26T06:52:13.050 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.045+0000 7f6e89189740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5563039652f0 0x556303da8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.050 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.045+0000 7f6e89189740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55630396d4d0 0x556303da9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.050 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.045+0000 7f6e89189740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55630398e1b0 0x556303da9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.051 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.045+0000 7f6e89189740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7c1adfe0 con 0x5563039652f0 2024-07-26T06:52:13.051 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.045+0000 7f6e89189740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7c1ad910 con 0x55630398e1b0 2024-07-26T06:52:13.051 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.045+0000 7f6e89189740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7c1aab50 con 0x55630396d4d0 2024-07-26T06:52:13.051 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.045+0000 7f6e89189740 10 client.0 _ll_register_callbacks cb 0x7f6e64004270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:52:13.051 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.045+0000 7f6e73fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5563039652f0 0x556303da8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.051 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.045+0000 7f6e73fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5563039652f0 0x556303da8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48308/0 (socket says 172.21.15.42:48308) 2024-07-26T06:52:13.051 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.045+0000 7f6e73fff640 1 -- 172.21.15.42:0/622752275 learned_addr learned my addr 172.21.15.42:0/622752275 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:13.051 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.045+0000 7f6e7bfff640 1 --2- 172.21.15.42:0/622752275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55630396d4d0 0x556303da9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.051 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.045+0000 7f6e73fff640 1 -- 172.21.15.42:0/622752275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55630396d4d0 msgr2=0x556303da9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:13.051 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.046+0000 7f6e73fff640 1 --2- 172.21.15.42:0/622752275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55630396d4d0 0x556303da9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.051 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.046+0000 7f6e73fff640 1 -- 172.21.15.42:0/622752275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55630398e1b0 msgr2=0x556303da9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:13.051 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.046+0000 7f6e73fff640 1 --2- 172.21.15.42:0/622752275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55630398e1b0 0x556303da9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.051 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.046+0000 7f6e73fff640 1 -- 172.21.15.42:0/622752275 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556303daa660 con 0x5563039652f0 2024-07-26T06:52:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.046+0000 7f6e73fff640 1 --2- 172.21.15.42:0/622752275 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5563039652f0 0x556303da8cd0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6e6402aa90 tx=0x7f6e640314a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678f20032cc27c56 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.046+0000 7f6e5b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:52:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.046+0000 7f6e5b7fe640 1 -- 172.21.15.42:0/622752275 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6405b070 con 0x5563039652f0 2024-07-26T06:52:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.046+0000 7f6e5b7fe640 1 -- 172.21.15.42:0/622752275 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f6e640317b0 con 0x5563039652f0 2024-07-26T06:52:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.046+0000 7f6e89189740 1 -- 172.21.15.42:0/622752275 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e64031c60 con 0x5563039652f0 2024-07-26T06:52:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.046+0000 7f6e5b7fe640 1 -- 172.21.15.42:0/622752275 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6e64035040 con 0x5563039652f0 2024-07-26T06:52:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.046+0000 7f6e89189740 10 client.15654 fetch_fsmap learned FSMap version 30 2024-07-26T06:52:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.046+0000 7f6e89189740 1 -- 172.21.15.42:0/622752275 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x556303db1900 con 0x5563039652f0 2024-07-26T06:52:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.046+0000 7f6e5b7fe640 1 -- 172.21.15.42:0/622752275 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e64031c60 con 0x5563039652f0 2024-07-26T06:52:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.046+0000 7f6e5b7fe640 1 -- 172.21.15.42:0/622752275 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f6e64004e60 con 0x5563039652f0 2024-07-26T06:52:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.046+0000 7f6e89189740 10 client.15654 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T06:52:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.046+0000 7f6e89189740 10 client.15654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:52:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.046+0000 7f6e89189740 20 client.15654 populate_metadata read hostname 'smithi042' 2024-07-26T06:52:13.052 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.047+0000 7f6e89189740 1 --2- 172.21.15.42:0/622752275 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5563039221c0 0x556303db4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.047+0000 7f6e89189740 4 client.15654 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:52:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.047+0000 7f6e89189740 1 -- 172.21.15.42:0/622752275 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556303ca3da0 con 0x5563039221c0 2024-07-26T06:52:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.047+0000 7f6e7bfff640 1 --2- 172.21.15.42:0/622752275 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5563039221c0 0x556303db4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.047+0000 7f6e7bfff640 1 --2- 172.21.15.42:0/622752275 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5563039221c0 0x556303db4270 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6ec1b8f725dd2ed server_cookie=b537d6a174fb7642 in_seq=0 out_seq=0 2024-07-26T06:52:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.047+0000 7f6e5b7fe640 10 client.15654 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.067+0000 7f6e5b7fe640 1 -- 172.21.15.42:0/622752275 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556303ca3da0 con 0x5563039221c0 2024-07-26T06:52:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.067+0000 7f6e5b7fe640 10 client.15654 handle_command_reply: tid=0 2024-07-26T06:52:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.067+0000 7f6e89189740 1 client.15654 shutdown 2024-07-26T06:52:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.067+0000 7f6e89189740 1 -- 172.21.15.42:0/622752275 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5563039652f0 msgr2=0x556303da8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.067+0000 7f6e89189740 1 --2- 172.21.15.42:0/622752275 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5563039652f0 0x556303da8cd0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6e6402aa90 tx=0x7f6e640314a0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.067+0000 7f6e89189740 1 -- 172.21.15.42:0/622752275 shutdown_connections 2024-07-26T06:52:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.067+0000 7f6e89189740 1 --2- 172.21.15.42:0/622752275 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5563039221c0 0x556303db4270 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.067+0000 7f6e89189740 1 --2- 172.21.15.42:0/622752275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55630398e1b0 0x556303da9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.067+0000 7f6e89189740 1 --2- 172.21.15.42:0/622752275 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55630396d4d0 0x556303da9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.067+0000 7f6e5b7fe640 0 client.15654 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.067+0000 7f6e89189740 1 --2- 172.21.15.42:0/622752275 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5563039652f0 0x556303da8cd0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.067+0000 7f6e89189740 1 -- 172.21.15.42:0/622752275 >> 172.21.15.42:0/622752275 conn(0x556303982300 msgr2=0x556303d3a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.067+0000 7f6e89189740 1 -- 172.21.15.42:0/622752275 shutdown_connections 2024-07-26T06:52:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.067+0000 7f6e89189740 1 -- 172.21.15.42:0/622752275 wait complete. 2024-07-26T06:52:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.067+0000 7f6e89189740 20 client.15654 trim_cache size 0 max 16384 2024-07-26T06:52:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:52:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:52:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:52:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:52:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.068+0000 7f6e8226f640 1 -- 172.21.15.42:0/3772110969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e680b3bd0 msgr2=0x7f6e680b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.068+0000 7f6e8226f640 1 --2- 172.21.15.42:0/3772110969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e680b3bd0 0x7f6e680b60e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6e740aa6c0 tx=0x7f6e740a4f90 comp rx=0 tx=0).stop 2024-07-26T06:52:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.068+0000 7f6e8226f640 1 -- 172.21.15.42:0/3772110969 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c104b20 msgr2=0x7f6e7c075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.068+0000 7f6e8226f640 1 --2- 172.21.15.42:0/3772110969 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c104b20 0x7f6e7c075680 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6e74009fd0 tx=0x7f6e7400b260 comp rx=0 tx=0).stop 2024-07-26T06:52:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.068+0000 7f6e8226f640 1 -- 172.21.15.42:0/3772110969 shutdown_connections 2024-07-26T06:52:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.068+0000 7f6e8226f640 1 --2- 172.21.15.42:0/3772110969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6e680b3bd0 0x7f6e680b60e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.068+0000 7f6e8226f640 1 --2- 172.21.15.42:0/3772110969 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c104b20 0x7f6e7c075680 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.068+0000 7f6e8226f640 1 --2- 172.21.15.42:0/3772110969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e7c104180 0x7f6e7c075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.068+0000 7f6e8226f640 1 --2- 172.21.15.42:0/3772110969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e7c103100 0x7f6e7c076880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.072 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.068+0000 7f6e8226f640 1 -- 172.21.15.42:0/3772110969 >> 172.21.15.42:0/3772110969 conn(0x7f6e7c0fea60 msgr2=0x7f6e7c1074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:13.072 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.068+0000 7f6e8226f640 1 -- 172.21.15.42:0/3772110969 shutdown_connections 2024-07-26T06:52:13.072 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:13.068+0000 7f6e8226f640 1 -- 172.21.15.42:0/3772110969 wait complete. 2024-07-26T06:52:13.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.947+0000 7f0432b3b640 1 -- 172.21.15.42:0/1834383418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f042c108ed0 msgr2=0x7f042c1092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:13.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.947+0000 7f0432b3b640 1 --2- 172.21.15.42:0/1834383418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f042c108ed0 0x7f042c1092e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0418009900 tx=0x7f041802a410 comp rx=0 tx=0).stop 2024-07-26T06:52:13.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.947+0000 7f0432b3b640 1 -- 172.21.15.42:0/1834383418 shutdown_connections 2024-07-26T06:52:13.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.947+0000 7f0432b3b640 1 --2- 172.21.15.42:0/1834383418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f042c109820 0x7f042c110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.947+0000 7f0432b3b640 1 --2- 172.21.15.42:0/1834383418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f042c108ed0 0x7f042c1092e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.947+0000 7f0432b3b640 1 --2- 172.21.15.42:0/1834383418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f042c103590 0x7f042c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.947+0000 7f0432b3b640 1 -- 172.21.15.42:0/1834383418 >> 172.21.15.42:0/1834383418 conn(0x7f042c07b280 msgr2=0x7f042c07b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:13.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.947+0000 7f0432b3b640 1 -- 172.21.15.42:0/1834383418 shutdown_connections 2024-07-26T06:52:13.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.947+0000 7f0432b3b640 1 -- 172.21.15.42:0/1834383418 wait complete. 2024-07-26T06:52:13.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.948+0000 7f0432b3b640 1 Processor -- start 2024-07-26T06:52:13.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.948+0000 7f0432b3b640 1 -- start start 2024-07-26T06:52:13.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.948+0000 7f0432b3b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f042c103590 0x7f042c199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.948+0000 7f0432b3b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f042c108ed0 0x7f042c19a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.948+0000 7f0432b3b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f042c109820 0x7f042c19f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.948+0000 7f0432b3b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f042c113580 con 0x7f042c103590 2024-07-26T06:52:13.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.948+0000 7f0432b3b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f042c113400 con 0x7f042c108ed0 2024-07-26T06:52:13.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.948+0000 7f0432b3b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f042c113700 con 0x7f042c109820 2024-07-26T06:52:13.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.948+0000 7f04308d7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f042c103590 0x7f042c199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.948+0000 7f04308d7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f042c103590 0x7f042c199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48328/0 (socket says 172.21.15.42:48328) 2024-07-26T06:52:13.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.948+0000 7f04308d7640 1 -- 172.21.15.42:0/639690421 learned_addr learned my addr 172.21.15.42:0/639690421 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:13.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.948+0000 7f04310d8640 1 --2- 172.21.15.42:0/639690421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f042c109820 0x7f042c19f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.948+0000 7f042bfff640 1 --2- 172.21.15.42:0/639690421 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f042c108ed0 0x7f042c19a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.949+0000 7f04308d7640 1 -- 172.21.15.42:0/639690421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f042c109820 msgr2=0x7f042c19f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:13.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.949+0000 7f04308d7640 1 --2- 172.21.15.42:0/639690421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f042c109820 0x7f042c19f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.949+0000 7f04308d7640 1 -- 172.21.15.42:0/639690421 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f042c108ed0 msgr2=0x7f042c19a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:13.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.949+0000 7f04308d7640 1 --2- 172.21.15.42:0/639690421 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f042c108ed0 0x7f042c19a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:13.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.949+0000 7f04308d7640 1 -- 172.21.15.42:0/639690421 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f042c19ff60 con 0x7f042c103590 2024-07-26T06:52:13.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.949+0000 7f04308d7640 1 --2- 172.21.15.42:0/639690421 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f042c103590 0x7f042c199ff0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f042000b4a0 tx=0x7f042000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:13.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.949+0000 7f0429ffb640 1 -- 172.21.15.42:0/639690421 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0420014070 con 0x7f042c103590 2024-07-26T06:52:13.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.949+0000 7f0429ffb640 1 -- 172.21.15.42:0/639690421 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0420010ac0 con 0x7f042c103590 2024-07-26T06:52:13.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.949+0000 7f0432b3b640 1 -- 172.21.15.42:0/639690421 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f042c1a0250 con 0x7f042c103590 2024-07-26T06:52:13.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.949+0000 7f0429ffb640 1 -- 172.21.15.42:0/639690421 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0420017c30 con 0x7f042c103590 2024-07-26T06:52:13.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.949+0000 7f0432b3b640 1 -- 172.21.15.42:0/639690421 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f042c1a69b0 con 0x7f042c103590 2024-07-26T06:52:13.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.950+0000 7f0432b3b640 1 -- 172.21.15.42:0/639690421 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f042c103a20 con 0x7f042c103590 2024-07-26T06:52:13.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.952+0000 7f0429ffb640 1 -- 172.21.15.42:0/639690421 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f0420017070 con 0x7f042c103590 2024-07-26T06:52:13.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.953+0000 7f0429ffb640 1 --2- 172.21.15.42:0/639690421 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f041c0b3c50 0x7f041c0b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:13.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.953+0000 7f0429ffb640 1 -- 172.21.15.42:0/639690421 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f04200e3270 con 0x7f042c103590 2024-07-26T06:52:13.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.953+0000 7f042bfff640 1 --2- 172.21.15.42:0/639690421 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f041c0b3c50 0x7f041c0b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:13.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.953+0000 7f042bfff640 1 --2- 172.21.15.42:0/639690421 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f041c0b3c50 0x7f041c0b6160 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0418009230 tx=0x7f0418031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:13.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:13.956+0000 7f0429ffb640 1 -- 172.21.15.42:0/639690421 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f04200aa8a0 con 0x7f042c103590 2024-07-26T06:52:14.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:14.179+0000 7f0432b3b640 1 -- 172.21.15.42:0/639690421 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f042c19b120 con 0x7f042c103590 2024-07-26T06:52:14.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:14.180+0000 7f0429ffb640 1 -- 172.21.15.42:0/639690421 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11979 (secure 0 0 0) 0x7f042c19b120 con 0x7f042c103590 2024-07-26T06:52:14.181 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:52:14.181 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":30,"btime":"2024-07-26T06:52:01:457118+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:52:00.467337+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/2604534474","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":2604534474},{"type":"v1","addr":"172.21.15.164:6835","nonce":2604534474}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24578,24583,24593,24601,24620]},"id":1}]} 2024-07-26T06:52:14.181 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 30 2024-07-26T06:52:14.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:14.182+0000 7f0432b3b640 1 -- 172.21.15.42:0/639690421 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f041c0b3c50 msgr2=0x7f041c0b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:14.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:14.182+0000 7f0432b3b640 1 --2- 172.21.15.42:0/639690421 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f041c0b3c50 0x7f041c0b6160 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0418009230 tx=0x7f0418031400 comp rx=0 tx=0).stop 2024-07-26T06:52:14.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:14.182+0000 7f0432b3b640 1 -- 172.21.15.42:0/639690421 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f042c103590 msgr2=0x7f042c199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:14.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:14.182+0000 7f0432b3b640 1 --2- 172.21.15.42:0/639690421 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f042c103590 0x7f042c199ff0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f042000b4a0 tx=0x7f042000b960 comp rx=0 tx=0).stop 2024-07-26T06:52:14.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:14.182+0000 7f0432b3b640 1 -- 172.21.15.42:0/639690421 shutdown_connections 2024-07-26T06:52:14.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:14.182+0000 7f0432b3b640 1 --2- 172.21.15.42:0/639690421 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f041c0b3c50 0x7f041c0b6160 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:14.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:14.182+0000 7f0432b3b640 1 --2- 172.21.15.42:0/639690421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f042c109820 0x7f042c19f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:14.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:14.182+0000 7f0432b3b640 1 --2- 172.21.15.42:0/639690421 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f042c108ed0 0x7f042c19a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:14.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:14.182+0000 7f0432b3b640 1 --2- 172.21.15.42:0/639690421 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f042c103590 0x7f042c199ff0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:14.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:14.182+0000 7f0432b3b640 1 -- 172.21.15.42:0/639690421 >> 172.21.15.42:0/639690421 conn(0x7f042c07b280 msgr2=0x7f042c106fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:14.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:14.182+0000 7f0432b3b640 1 -- 172.21.15.42:0/639690421 shutdown_connections 2024-07-26T06:52:14.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:14.182+0000 7f0432b3b640 1 -- 172.21.15.42:0/639690421 wait complete. 2024-07-26T06:52:14.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:14 smithi163 ceph-mon[36873]: pgmap v456: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 32 KiB/s wr, 73 op/s 2024-07-26T06:52:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:14 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/639690421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:52:14.844 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:14 smithi042 ceph-mon[32775]: pgmap v456: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 32 KiB/s wr, 73 op/s 2024-07-26T06:52:14.844 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:14 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/639690421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:52:14.875 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24620 quiesce db --set-id c6bd3542 --release --await' 2024-07-26T06:52:14.875 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.24620 quiesce db --set-id c6bd3542 --release --await 2024-07-26T06:52:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:14 smithi164 ceph-mon[36970]: pgmap v456: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 32 KiB/s wr, 73 op/s 2024-07-26T06:52:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:14 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/639690421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:52:15.140 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:52:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:16 smithi042 ceph-mon[32775]: pgmap v457: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 31 KiB/s wr, 52 op/s 2024-07-26T06:52:16.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:16 smithi163 ceph-mon[36873]: pgmap v457: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 31 KiB/s wr, 52 op/s 2024-07-26T06:52:16.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.935+0000 7f0501088640 1 -- 172.21.15.42:0/4071375619 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04fc108750 msgr2=0x7f04fc10ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:16.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.935+0000 7f0501088640 1 --2- 172.21.15.42:0/4071375619 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04fc108750 0x7f04fc10ef80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f04f4009ca0 tx=0x7f04f402a8c0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.935+0000 7f0501088640 1 -- 172.21.15.42:0/4071375619 shutdown_connections 2024-07-26T06:52:16.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.935+0000 7f0501088640 1 --2- 172.21.15.42:0/4071375619 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04fc108750 0x7f04fc10ef80 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.935+0000 7f0501088640 1 --2- 172.21.15.42:0/4071375619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04fc107e00 0x7f04fc108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.935+0000 7f0501088640 1 --2- 172.21.15.42:0/4071375619 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04fc102510 0x7f04fc102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.935+0000 7f0501088640 1 -- 172.21.15.42:0/4071375619 >> 172.21.15.42:0/4071375619 conn(0x7f04fc079190 msgr2=0x7f04fc0795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:16.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.935+0000 7f0501088640 1 -- 172.21.15.42:0/4071375619 shutdown_connections 2024-07-26T06:52:16.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.935+0000 7f0501088640 1 -- 172.21.15.42:0/4071375619 wait complete. 2024-07-26T06:52:16.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.936+0000 7f0501088640 1 Processor -- start 2024-07-26T06:52:16.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.936+0000 7f0501088640 1 -- start start 2024-07-26T06:52:16.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.936+0000 7f0501088640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04fc102510 0x7f04fc0ff0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.936+0000 7f0501088640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04fc107e00 0x7f04fc0ff5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.936+0000 7f0501088640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04fc108750 0x7f04fc0ffb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.936+0000 7f0501088640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f04fc1123d0 con 0x7f04fc102510 2024-07-26T06:52:16.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.936+0000 7f0501088640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04fc112250 con 0x7f04fc107e00 2024-07-26T06:52:16.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.936+0000 7f0501088640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f04fc112550 con 0x7f04fc108750 2024-07-26T06:52:16.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.936+0000 7f04fad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04fc102510 0x7f04fc0ff0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:16.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.936+0000 7f04fad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04fc102510 0x7f04fc0ff0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57966/0 (socket says 172.21.15.42:57966) 2024-07-26T06:52:16.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.936+0000 7f04fad9d640 1 -- 172.21.15.42:0/80354580 learned_addr learned my addr 172.21.15.42:0/80354580 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:16.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.937+0000 7f04fb59e640 1 --2- 172.21.15.42:0/80354580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04fc108750 0x7f04fc0ffb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:16.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.937+0000 7f04f3fff640 1 --2- 172.21.15.42:0/80354580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04fc107e00 0x7f04fc0ff5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:16.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.937+0000 7f04fad9d640 1 -- 172.21.15.42:0/80354580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04fc108750 msgr2=0x7f04fc0ffb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:16.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.937+0000 7f04fad9d640 1 --2- 172.21.15.42:0/80354580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04fc108750 0x7f04fc0ffb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.937+0000 7f04fad9d640 1 -- 172.21.15.42:0/80354580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04fc107e00 msgr2=0x7f04fc0ff5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:16.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.937+0000 7f04fad9d640 1 --2- 172.21.15.42:0/80354580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04fc107e00 0x7f04fc0ff5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.937+0000 7f04fad9d640 1 -- 172.21.15.42:0/80354580 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04fc1a8130 con 0x7f04fc102510 2024-07-26T06:52:16.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.937+0000 7f04fad9d640 1 --2- 172.21.15.42:0/80354580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04fc102510 0x7f04fc0ff0a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f04ec00e9f0 tx=0x7f04ec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:16.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.937+0000 7f04f8d99640 1 -- 172.21.15.42:0/80354580 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ec00cd80 con 0x7f04fc102510 2024-07-26T06:52:16.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.937+0000 7f0501088640 1 -- 172.21.15.42:0/80354580 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04fc1a8420 con 0x7f04fc102510 2024-07-26T06:52:16.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.937+0000 7f04f8d99640 1 -- 172.21.15.42:0/80354580 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04ec0054d0 con 0x7f04fc102510 2024-07-26T06:52:16.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.937+0000 7f04f8d99640 1 -- 172.21.15.42:0/80354580 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ec010690 con 0x7f04fc102510 2024-07-26T06:52:16.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.937+0000 7f0501088640 1 -- 172.21.15.42:0/80354580 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04fc1a8910 con 0x7f04fc102510 2024-07-26T06:52:16.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.940+0000 7f050760c740 1 Processor -- start 2024-07-26T06:52:16.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.940+0000 7f050760c740 1 -- start start 2024-07-26T06:52:16.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.940+0000 7f050760c740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8ac0e520 0x559f8afb4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.940+0000 7f050760c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8abe5660 0x559f8afb45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.940+0000 7f050760c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8abed840 0x559f8afb4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.940+0000 7f050760c740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x559f8af29e70 con 0x559f8ac0e520 2024-07-26T06:52:16.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.940+0000 7f050760c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559f8af19370 con 0x559f8abed840 2024-07-26T06:52:16.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.940+0000 7f050760c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x559f8af1d3b0 con 0x559f8abe5660 2024-07-26T06:52:16.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.940+0000 7f04f8d99640 1 -- 172.21.15.42:0/80354580 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f04ec004e90 con 0x7f04fc102510 2024-07-26T06:52:16.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.940+0000 7f04fb59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8ac0e520 0x559f8afb4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:16.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.940+0000 7f04fb59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8ac0e520 0x559f8afb4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57970/0 (socket says 172.21.15.42:57970) 2024-07-26T06:52:16.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.940+0000 7f04fb59e640 1 -- 172.21.15.42:0/3087816236 learned_addr learned my addr 172.21.15.42:0/3087816236 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:16.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.941+0000 7f04f3fff640 1 --2- 172.21.15.42:0/3087816236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8abed840 0x559f8afb4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:16.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.941+0000 7f04fad9d640 1 --2- 172.21.15.42:0/3087816236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8abe5660 0x559f8afb45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:16.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.941+0000 7f04fb59e640 1 -- 172.21.15.42:0/3087816236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8abe5660 msgr2=0x559f8afb45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:16.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.941+0000 7f04fb59e640 1 --2- 172.21.15.42:0/3087816236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8abe5660 0x559f8afb45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.941+0000 7f04fb59e640 1 -- 172.21.15.42:0/3087816236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8abed840 msgr2=0x559f8afb4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:16.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.941+0000 7f04fb59e640 1 --2- 172.21.15.42:0/3087816236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8abed840 0x559f8afb4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.941+0000 7f04fb59e640 1 -- 172.21.15.42:0/3087816236 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f8aee6040 con 0x559f8ac0e520 2024-07-26T06:52:16.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.941+0000 7f04fb59e640 1 --2- 172.21.15.42:0/3087816236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8ac0e520 0x559f8afb4070 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f04f4005cc0 tx=0x7f04f402f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7da6fb1ac4fcd5b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:16.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.941+0000 7f04f17fa640 1 -- 172.21.15.42:0/3087816236 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4031cf0 con 0x559f8ac0e520 2024-07-26T06:52:16.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.941+0000 7f04f17fa640 1 -- 172.21.15.42:0/3087816236 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04f4036040 con 0x559f8ac0e520 2024-07-26T06:52:16.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.941+0000 7f04f17fa640 1 -- 172.21.15.42:0/3087816236 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4004040 con 0x559f8ac0e520 2024-07-26T06:52:16.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.942+0000 7f04f8d99640 1 --2- 172.21.15.42:0/80354580 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f04e80b3bd0 0x7f04e80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.942+0000 7f04fad9d640 1 --2- 172.21.15.42:0/80354580 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f04e80b3bd0 0x7f04e80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:16.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.942+0000 7f04f8d99640 1 -- 172.21.15.42:0/80354580 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f04ec0e41e0 con 0x7f04fc102510 2024-07-26T06:52:16.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.942+0000 7f050760c740 1 -- 172.21.15.42:0/3087816236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8ac0e520 msgr2=0x559f8afb4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:16.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.942+0000 7f050760c740 1 --2- 172.21.15.42:0/3087816236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8ac0e520 0x559f8afb4070 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f04f4005cc0 tx=0x7f04f402f420 comp rx=0 tx=0).stop 2024-07-26T06:52:16.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.942+0000 7f050760c740 1 -- 172.21.15.42:0/3087816236 shutdown_connections 2024-07-26T06:52:16.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.942+0000 7f050760c740 1 --2- 172.21.15.42:0/3087816236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8abed840 0x559f8afb4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.942+0000 7f050760c740 1 --2- 172.21.15.42:0/3087816236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8abe5660 0x559f8afb45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.942+0000 7f050760c740 1 --2- 172.21.15.42:0/3087816236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8ac0e520 0x559f8afb4070 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.942+0000 7f04fad9d640 1 --2- 172.21.15.42:0/80354580 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f04e80b3bd0 0x7f04e80b60e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0a7130 tx=0x7f04ec004a90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:16.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.942+0000 7f050760c740 1 -- 172.21.15.42:0/3087816236 >> 172.21.15.42:0/3087816236 conn(0x559f8ac02910 msgr2=0x559f8afb1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:16.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.943+0000 7f050760c740 1 -- 172.21.15.42:0/3087816236 shutdown_connections 2024-07-26T06:52:16.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.943+0000 7f050760c740 1 -- 172.21.15.42:0/3087816236 wait complete. 2024-07-26T06:52:16.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.943+0000 7f050760c740 1 -- start start 2024-07-26T06:52:16.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.943+0000 7f050760c740 1 Processor -- start 2024-07-26T06:52:16.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.943+0000 7f050760c740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abe5660 0x559f8afbfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.943+0000 7f050760c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8abed840 0x559f8afc00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.943+0000 7f050760c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8ac0e520 0x559f8afcbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.943+0000 7f050760c740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x559f8ad7cbb0 con 0x559f8abe5660 2024-07-26T06:52:16.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.943+0000 7f050760c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559f8aebbac0 con 0x559f8ac0e520 2024-07-26T06:52:16.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.943+0000 7f050760c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x559f8afc27f0 con 0x559f8abed840 2024-07-26T06:52:16.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.943+0000 7f050760c740 10 client.0 _ll_register_callbacks cb 0x559f8aec43b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:52:16.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.943+0000 7f04fb59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abe5660 0x559f8afbfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:16.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.944+0000 7f04fb59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abe5660 0x559f8afbfb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57974/0 (socket says 172.21.15.42:57974) 2024-07-26T06:52:16.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.944+0000 7f04fb59e640 1 -- 172.21.15.42:0/1797427033 learned_addr learned my addr 172.21.15.42:0/1797427033 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:16.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.944+0000 7f04f3fff640 1 --2- 172.21.15.42:0/1797427033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8abed840 0x559f8afc00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:16.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.944+0000 7f04fb59e640 1 --2- 172.21.15.42:0/1797427033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8ac0e520 0x559f8afcbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:16.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.944+0000 7f04fb59e640 1 -- 172.21.15.42:0/1797427033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8abed840 msgr2=0x559f8afc00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:16.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.944+0000 7f04fb59e640 1 --2- 172.21.15.42:0/1797427033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8abed840 0x559f8afc00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.944+0000 7f04fb59e640 1 -- 172.21.15.42:0/1797427033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8ac0e520 msgr2=0x559f8afcbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:16.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.944+0000 7f04fb59e640 1 --2- 172.21.15.42:0/1797427033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8ac0e520 0x559f8afcbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.944+0000 7f04fb59e640 1 -- 172.21.15.42:0/1797427033 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f8ac79b00 con 0x559f8abe5660 2024-07-26T06:52:16.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.944+0000 7f04fb59e640 1 --2- 172.21.15.42:0/1797427033 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abe5660 0x559f8afbfb80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f04f402a900 tx=0x7f04f4003b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e4474510764598 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:16.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.944+0000 7f04e27fc640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:52:16.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.944+0000 7f04e27fc640 1 -- 172.21.15.42:0/1797427033 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4058060 con 0x559f8abe5660 2024-07-26T06:52:16.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.944+0000 7f04e27fc640 1 -- 172.21.15.42:0/1797427033 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f04f4031650 con 0x559f8abe5660 2024-07-26T06:52:16.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.944+0000 7f050760c740 1 -- 172.21.15.42:0/1797427033 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04f4036040 con 0x559f8abe5660 2024-07-26T06:52:16.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.944+0000 7f04e27fc640 1 -- 172.21.15.42:0/1797427033 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f04f4031850 con 0x559f8abe5660 2024-07-26T06:52:16.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.945+0000 7f050760c740 10 client.15672 fetch_fsmap learned FSMap version 30 2024-07-26T06:52:16.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.945+0000 7f050760c740 1 -- 172.21.15.42:0/1797427033 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559f8acf0960 con 0x559f8abe5660 2024-07-26T06:52:16.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.945+0000 7f04e27fc640 1 -- 172.21.15.42:0/1797427033 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04f4036040 con 0x559f8abe5660 2024-07-26T06:52:16.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.945+0000 7f04e27fc640 1 -- 172.21.15.42:0/1797427033 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f04f40305c0 con 0x559f8abe5660 2024-07-26T06:52:16.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.945+0000 7f050760c740 10 client.15672 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T06:52:16.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.945+0000 7f050760c740 10 client.15672 resolve_mds: validated gid 24620 aka daemon mds.b 2024-07-26T06:52:16.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.945+0000 7f050760c740 20 client.15672 populate_metadata read hostname 'smithi042' 2024-07-26T06:52:16.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.945+0000 7f050760c740 1 --2- 172.21.15.42:0/1797427033 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x559f8abccea0 0x559f8afd18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.945+0000 7f050760c740 4 client.15672 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:52:16.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.945+0000 7f050760c740 1 -- 172.21.15.42:0/1797427033 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f8af2bdc0 con 0x559f8abccea0 2024-07-26T06:52:16.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.945+0000 7f04f3fff640 1 --2- 172.21.15.42:0/1797427033 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x559f8abccea0 0x559f8afd18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:16.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.946+0000 7f04f3fff640 1 --2- 172.21.15.42:0/1797427033 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x559f8abccea0 0x559f8afd18b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32d62ce84ae4ac27 server_cookie=a9c88fccb591f1f5 in_seq=0 out_seq=0 2024-07-26T06:52:16.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.946+0000 7f04e27fc640 10 client.15672 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:16.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.946+0000 7f04e27fc640 1 -- 172.21.15.42:0/1797427033 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559f8af2bdc0 con 0x559f8abccea0 2024-07-26T06:52:16.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.946+0000 7f04e27fc640 10 client.15672 handle_command_reply: tid=0 2024-07-26T06:52:16.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.946+0000 7f050760c740 1 client.15672 shutdown 2024-07-26T06:52:16.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.947+0000 7f050760c740 1 -- 172.21.15.42:0/1797427033 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abe5660 msgr2=0x559f8afbfb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:16.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.947+0000 7f050760c740 1 --2- 172.21.15.42:0/1797427033 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abe5660 0x559f8afbfb80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f04f402a900 tx=0x7f04f4003b10 comp rx=0 tx=0).stop 2024-07-26T06:52:16.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.947+0000 7f050760c740 1 -- 172.21.15.42:0/1797427033 shutdown_connections 2024-07-26T06:52:16.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.947+0000 7f050760c740 1 --2- 172.21.15.42:0/1797427033 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x559f8abccea0 0x559f8afd18b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.947+0000 7f050760c740 1 --2- 172.21.15.42:0/1797427033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8ac0e520 0x559f8afcbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.947+0000 7f050760c740 1 --2- 172.21.15.42:0/1797427033 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8abed840 0x559f8afc00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.947+0000 7f04e27fc640 0 client.15672 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:16.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.947+0000 7f050760c740 1 --2- 172.21.15.42:0/1797427033 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abe5660 0x559f8afbfb80 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.947+0000 7f050760c740 1 -- 172.21.15.42:0/1797427033 >> 172.21.15.42:0/1797427033 conn(0x559f8ac02910 msgr2=0x559f8afb7250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:16.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.947+0000 7f050760c740 1 -- 172.21.15.42:0/1797427033 shutdown_connections 2024-07-26T06:52:16.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.947+0000 7f050760c740 1 -- 172.21.15.42:0/1797427033 wait complete. 2024-07-26T06:52:16.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.947+0000 7f050760c740 20 client.15672 trim_cache size 0 max 16384 2024-07-26T06:52:16.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.964+0000 7f050760c740 1 Processor -- start 2024-07-26T06:52:16.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.964+0000 7f050760c740 1 -- start start 2024-07-26T06:52:16.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.964+0000 7f050760c740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abe5660 0x559f8afc1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.964+0000 7f050760c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8abed840 0x559f8afcba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.964+0000 7f050760c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8ac0e520 0x559f8afcbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.964+0000 7f050760c740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f04fc1b1100 con 0x559f8abe5660 2024-07-26T06:52:16.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.964+0000 7f050760c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04fc1b2c70 con 0x559f8abed840 2024-07-26T06:52:16.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.964+0000 7f050760c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f04fc1b17d0 con 0x559f8ac0e520 2024-07-26T06:52:16.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.965+0000 7f04fb59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abe5660 0x559f8afc1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:16.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.965+0000 7f04fb59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abe5660 0x559f8afc1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57978/0 (socket says 172.21.15.42:57978) 2024-07-26T06:52:16.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.965+0000 7f04fb59e640 1 -- 172.21.15.42:0/3894575659 learned_addr learned my addr 172.21.15.42:0/3894575659 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:16.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.965+0000 7f04f3fff640 1 --2- 172.21.15.42:0/3894575659 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8abed840 0x559f8afcba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:16.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.965+0000 7f04fb59e640 1 -- 172.21.15.42:0/3894575659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8ac0e520 msgr2=0x559f8afcbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:16.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.965+0000 7f04fb59e640 1 --2- 172.21.15.42:0/3894575659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8ac0e520 0x559f8afcbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.965+0000 7f04fb59e640 1 -- 172.21.15.42:0/3894575659 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8abed840 msgr2=0x559f8afcba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:16.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.965+0000 7f04fb59e640 1 --2- 172.21.15.42:0/3894575659 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8abed840 0x559f8afcba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.965+0000 7f04fb59e640 1 -- 172.21.15.42:0/3894575659 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f8afc0c40 con 0x559f8abe5660 2024-07-26T06:52:16.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.965+0000 7f04f3fff640 1 --2- 172.21.15.42:0/3894575659 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8abed840 0x559f8afcba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:52:16.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.965+0000 7f04fb59e640 1 --2- 172.21.15.42:0/3894575659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abe5660 0x559f8afc1860 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f04f4031fd0 tx=0x7f04f40093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45ac45ea4b9441b4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:16.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.965+0000 7f04f0ff9640 1 -- 172.21.15.42:0/3894575659 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4062070 con 0x559f8abe5660 2024-07-26T06:52:16.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.965+0000 7f04f0ff9640 1 -- 172.21.15.42:0/3894575659 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04f4036070 con 0x559f8abe5660 2024-07-26T06:52:16.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.965+0000 7f04f0ff9640 1 -- 172.21.15.42:0/3894575659 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f40316b0 con 0x559f8abe5660 2024-07-26T06:52:16.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.966+0000 7f050760c740 1 -- 172.21.15.42:0/3894575659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abe5660 msgr2=0x559f8afc1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:16.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.966+0000 7f050760c740 1 --2- 172.21.15.42:0/3894575659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abe5660 0x559f8afc1860 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f04f4031fd0 tx=0x7f04f40093f0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.966+0000 7f050760c740 1 -- 172.21.15.42:0/3894575659 shutdown_connections 2024-07-26T06:52:16.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.966+0000 7f050760c740 1 --2- 172.21.15.42:0/3894575659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8ac0e520 0x559f8afcbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.966+0000 7f050760c740 1 --2- 172.21.15.42:0/3894575659 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8abed840 0x559f8afcba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.966+0000 7f050760c740 1 --2- 172.21.15.42:0/3894575659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abe5660 0x559f8afc1860 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.966+0000 7f050760c740 1 -- 172.21.15.42:0/3894575659 >> 172.21.15.42:0/3894575659 conn(0x559f8ac02910 msgr2=0x559f8afb50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:16.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.966+0000 7f050760c740 1 -- 172.21.15.42:0/3894575659 shutdown_connections 2024-07-26T06:52:16.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.966+0000 7f050760c740 1 -- 172.21.15.42:0/3894575659 wait complete. 2024-07-26T06:52:16.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.967+0000 7f050760c740 1 -- start start 2024-07-26T06:52:16.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.967+0000 7f050760c740 1 Processor -- start 2024-07-26T06:52:16.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.967+0000 7f050760c740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8abe5660 0x559f8afd2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.967+0000 7f050760c740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abed840 0x559f8afd3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.967+0000 7f050760c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8ac0e520 0x559f8b038e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.967+0000 7f050760c740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f04fc1b2570 con 0x559f8abed840 2024-07-26T06:52:16.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.967+0000 7f050760c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04fc1b1ea0 con 0x559f8ac0e520 2024-07-26T06:52:16.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.967+0000 7f050760c740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f04fc1af0e0 con 0x559f8abe5660 2024-07-26T06:52:16.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.967+0000 7f050760c740 10 client.0 _ll_register_callbacks cb 0x7f04e400b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:52:16.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.968+0000 7f04fb59e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8ac0e520 0x559f8b038e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:16.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.968+0000 7f04fb59e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8ac0e520 0x559f8b038e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:34336/0 (socket says 172.21.15.42:34336) 2024-07-26T06:52:16.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.968+0000 7f04fb59e640 1 -- 172.21.15.42:0/1211964599 learned_addr learned my addr 172.21.15.42:0/1211964599 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:16.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.968+0000 7f04fb59e640 1 --2- 172.21.15.42:0/1211964599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8abe5660 0x559f8afd2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:16.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.968+0000 7f04f3fff640 1 --2- 172.21.15.42:0/1211964599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abed840 0x559f8afd3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:16.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.968+0000 7f04fb59e640 1 -- 172.21.15.42:0/1211964599 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8ac0e520 msgr2=0x559f8b038e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:16.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.968+0000 7f04fb59e640 1 --2- 172.21.15.42:0/1211964599 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8ac0e520 0x559f8b038e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.968+0000 7f04fb59e640 1 -- 172.21.15.42:0/1211964599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abed840 msgr2=0x559f8afd3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:16.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.968+0000 7f04fb59e640 1 --2- 172.21.15.42:0/1211964599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abed840 0x559f8afd3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.968+0000 7f04fb59e640 1 -- 172.21.15.42:0/1211964599 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f8afd3830 con 0x559f8abe5660 2024-07-26T06:52:16.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.968+0000 7f04fb59e640 1 --2- 172.21.15.42:0/1211964599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8abe5660 0x559f8afd2ae0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f04f4009a60 tx=0x7f04f4003b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0cde38a76f66815 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:16.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.968+0000 7f04e27fc640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-07-26T06:52:16.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.969+0000 7f04e27fc640 1 -- 172.21.15.42:0/1211964599 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f406a060 con 0x559f8abe5660 2024-07-26T06:52:16.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.969+0000 7f050760c740 1 -- 172.21.15.42:0/1211964599 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04f4036070 con 0x559f8abe5660 2024-07-26T06:52:16.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.969+0000 7f04e27fc640 1 -- 172.21.15.42:0/1211964599 <== mon.1 v2:172.21.15.164:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7f04f402f860 con 0x559f8abe5660 2024-07-26T06:52:16.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.969+0000 7f04e27fc640 1 -- 172.21.15.42:0/1211964599 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f04f4062070 con 0x559f8abe5660 2024-07-26T06:52:16.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.969+0000 7f050760c740 10 client.25330 fetch_fsmap learned FSMap version 30 2024-07-26T06:52:16.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.969+0000 7f050760c740 1 -- 172.21.15.42:0/1211964599 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559f8b03a340 con 0x559f8abe5660 2024-07-26T06:52:16.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.969+0000 7f04e27fc640 1 -- 172.21.15.42:0/1211964599 <== mon.1 v2:172.21.15.164:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04f4036070 con 0x559f8abe5660 2024-07-26T06:52:16.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.969+0000 7f04e27fc640 1 -- 172.21.15.42:0/1211964599 <== mon.1 v2:172.21.15.164:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f04f4054940 con 0x559f8abe5660 2024-07-26T06:52:16.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.969+0000 7f050760c740 10 client.25330 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T06:52:16.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.969+0000 7f050760c740 10 client.25330 resolve_mds: validated gid 24620 aka daemon mds.b 2024-07-26T06:52:16.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.969+0000 7f050760c740 20 client.25330 populate_metadata read hostname 'smithi042' 2024-07-26T06:52:16.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.969+0000 7f050760c740 1 --2- 172.21.15.42:0/1211964599 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x559f8aba2250 0x559f8b03c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:16.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.969+0000 7f050760c740 4 client.25330 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c6bd3542", "release": true, "await": true}] 2024-07-26T06:52:16.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.969+0000 7f050760c740 1 -- 172.21.15.42:0/1211964599 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c6bd3542", "release": true, "await": true}) -- 0x559f8af27860 con 0x559f8aba2250 2024-07-26T06:52:16.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.970+0000 7f04f3fff640 1 --2- 172.21.15.42:0/1211964599 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x559f8aba2250 0x559f8b03c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:16 smithi164 ceph-mon[36970]: pgmap v457: 97 pgs: 97 active+clean; 16 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 31 KiB/s wr, 52 op/s 2024-07-26T06:52:16.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.970+0000 7f04f3fff640 1 --2- 172.21.15.42:0/1211964599 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x559f8aba2250 0x559f8b03c770 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc2c8027aa7451c6 server_cookie=a9f78b97cdff9bd3 in_seq=0 out_seq=0 2024-07-26T06:52:16.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.970+0000 7f04e27fc640 10 client.25330 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:16.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.996+0000 7f04e27fc640 1 -- 172.21.15.42:0/1211964599 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x559f8af27860 con 0x559f8aba2250 2024-07-26T06:52:16.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.996+0000 7f04e27fc640 10 client.25330 handle_command_reply: tid=0 2024-07-26T06:52:16.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.996+0000 7f050760c740 1 client.25330 shutdown 2024-07-26T06:52:16.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.996+0000 7f050760c740 1 -- 172.21.15.42:0/1211964599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8abe5660 msgr2=0x559f8afd2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:16.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.996+0000 7f050760c740 1 --2- 172.21.15.42:0/1211964599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8abe5660 0x559f8afd2ae0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f04f4009a60 tx=0x7f04f4003b10 comp rx=0 tx=0).stop 2024-07-26T06:52:16.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.996+0000 7f050760c740 1 -- 172.21.15.42:0/1211964599 shutdown_connections 2024-07-26T06:52:16.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.996+0000 7f050760c740 1 --2- 172.21.15.42:0/1211964599 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x559f8aba2250 0x559f8b03c770 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.996+0000 7f050760c740 1 --2- 172.21.15.42:0/1211964599 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559f8ac0e520 0x559f8b038e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.996+0000 7f050760c740 1 --2- 172.21.15.42:0/1211964599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x559f8abed840 0x559f8afd3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.996+0000 7f04e27fc640 0 client.25330 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:16.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.996+0000 7f050760c740 1 --2- 172.21.15.42:0/1211964599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x559f8abe5660 0x559f8afd2ae0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:16.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.996+0000 7f050760c740 1 -- 172.21.15.42:0/1211964599 >> 172.21.15.42:0/1211964599 conn(0x559f8ac02910 msgr2=0x559f8afbe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:16.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.996+0000 7f050760c740 1 -- 172.21.15.42:0/1211964599 shutdown_connections 2024-07-26T06:52:16.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.996+0000 7f050760c740 1 -- 172.21.15.42:0/1211964599 wait complete. 2024-07-26T06:52:16.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.996+0000 7f050760c740 20 client.25330 trim_cache size 0 max 16384 2024-07-26T06:52:16.999 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:52:16.999 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 30, 2024-07-26T06:52:16.999 INFO:teuthology.orchestra.run.smithi042.stdout: "leader": 24620, 2024-07-26T06:52:16.999 INFO:teuthology.orchestra.run.smithi042.stdout: "set_version": 8, 2024-07-26T06:52:16.999 INFO:teuthology.orchestra.run.smithi042.stdout: "sets": { 2024-07-26T06:52:16.999 INFO:teuthology.orchestra.run.smithi042.stdout: "c6bd3542": { 2024-07-26T06:52:16.999 INFO:teuthology.orchestra.run.smithi042.stdout: "version": 8, 2024-07-26T06:52:17.000 INFO:teuthology.orchestra.run.smithi042.stdout: "age_ref": 0.0, 2024-07-26T06:52:17.000 INFO:teuthology.orchestra.run.smithi042.stdout: "state": { 2024-07-26T06:52:17.000 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "RELEASED", 2024-07-26T06:52:17.000 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 0.0 2024-07-26T06:52:17.000 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-07-26T06:52:17.000 INFO:teuthology.orchestra.run.smithi042.stdout: "timeout": 90.0, 2024-07-26T06:52:17.000 INFO:teuthology.orchestra.run.smithi042.stdout: "expiration": 142.6, 2024-07-26T06:52:17.000 INFO:teuthology.orchestra.run.smithi042.stdout: "members": { 2024-07-26T06:52:17.000 INFO:teuthology.orchestra.run.smithi042.stdout: "file:/": { 2024-07-26T06:52:17.000 INFO:teuthology.orchestra.run.smithi042.stdout: "excluded": false, 2024-07-26T06:52:17.000 INFO:teuthology.orchestra.run.smithi042.stdout: "state": { 2024-07-26T06:52:17.000 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "RELEASED", 2024-07-26T06:52:17.000 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 0.0 2024-07-26T06:52:17.001 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:52:17.001 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:52:17.001 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:52:17.001 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:52:17.001 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:52:17.001 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:52:17.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.997+0000 7f0501088640 1 -- 172.21.15.42:0/80354580 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f04e80b3bd0 msgr2=0x7f04e80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:17.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.997+0000 7f0501088640 1 --2- 172.21.15.42:0/80354580 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f04e80b3bd0 0x7f04e80b60e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0a7130 tx=0x7f04ec004a90 comp rx=0 tx=0).stop 2024-07-26T06:52:17.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.997+0000 7f0501088640 1 -- 172.21.15.42:0/80354580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04fc102510 msgr2=0x7f04fc0ff0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:17.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.997+0000 7f0501088640 1 --2- 172.21.15.42:0/80354580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04fc102510 0x7f04fc0ff0a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f04ec00e9f0 tx=0x7f04ec00edf0 comp rx=0 tx=0).stop 2024-07-26T06:52:17.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.997+0000 7f0501088640 1 -- 172.21.15.42:0/80354580 shutdown_connections 2024-07-26T06:52:17.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.997+0000 7f0501088640 1 --2- 172.21.15.42:0/80354580 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f04e80b3bd0 0x7f04e80b60e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:17.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.997+0000 7f0501088640 1 --2- 172.21.15.42:0/80354580 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f04fc108750 0x7f04fc0ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:17.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.997+0000 7f0501088640 1 --2- 172.21.15.42:0/80354580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04fc107e00 0x7f04fc0ff5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:17.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.997+0000 7f0501088640 1 --2- 172.21.15.42:0/80354580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04fc102510 0x7f04fc0ff0a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:17.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.997+0000 7f0501088640 1 -- 172.21.15.42:0/80354580 >> 172.21.15.42:0/80354580 conn(0x7f04fc079190 msgr2=0x7f04fc101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:17.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.997+0000 7f0501088640 1 -- 172.21.15.42:0/80354580 shutdown_connections 2024-07-26T06:52:17.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:52:16.997+0000 7f0501088640 1 -- 172.21.15.42:0/80354580 wait complete. 2024-07-26T06:52:18.350 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c6bd3542', total seconds elapsed: 29.2 2024-07-26T06:52:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:18 smithi042 ceph-mon[32775]: pgmap v458: 97 pgs: 97 active+clean; 17 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 89 KiB/s wr, 26 op/s 2024-07-26T06:52:18.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:18 smithi163 ceph-mon[36873]: pgmap v458: 97 pgs: 97 active+clean; 17 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 89 KiB/s wr, 26 op/s 2024-07-26T06:52:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:18 smithi164 ceph-mon[36970]: pgmap v458: 97 pgs: 97 active+clean; 17 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 89 KiB/s wr, 26 op/s 2024-07-26T06:52:20.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:20 smithi163 ceph-mon[36873]: pgmap v459: 97 pgs: 97 active+clean; 19 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 277 KiB/s wr, 48 op/s 2024-07-26T06:52:20.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:20 smithi042 ceph-mon[32775]: pgmap v459: 97 pgs: 97 active+clean; 19 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 277 KiB/s wr, 48 op/s 2024-07-26T06:52:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:20 smithi164 ceph-mon[36970]: pgmap v459: 97 pgs: 97 active+clean; 19 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 277 KiB/s wr, 48 op/s 2024-07-26T06:52:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:22 smithi163 ceph-mon[36873]: pgmap v460: 97 pgs: 97 active+clean; 19 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 277 KiB/s wr, 46 op/s 2024-07-26T06:52:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:22 smithi042 ceph-mon[32775]: pgmap v460: 97 pgs: 97 active+clean; 19 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 277 KiB/s wr, 46 op/s 2024-07-26T06:52:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:22 smithi164 ceph-mon[36970]: pgmap v460: 97 pgs: 97 active+clean; 19 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 277 KiB/s wr, 46 op/s 2024-07-26T06:52:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:24 smithi163 ceph-mon[36873]: pgmap v461: 97 pgs: 97 active+clean; 19 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 291 KiB/s wr, 48 op/s 2024-07-26T06:52:24.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:24 smithi042 ceph-mon[32775]: pgmap v461: 97 pgs: 97 active+clean; 19 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 291 KiB/s wr, 48 op/s 2024-07-26T06:52:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:24 smithi164 ceph-mon[36970]: pgmap v461: 97 pgs: 97 active+clean; 19 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 291 KiB/s wr, 48 op/s 2024-07-26T06:52:26.182 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.181+0000 7ff305a6f640 1 -- 172.21.15.42:0/3637788297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3001062b0 msgr2=0x7ff3001086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:26.183 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.181+0000 7ff305a6f640 1 --2- 172.21.15.42:0/3637788297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3001062b0 0x7ff3001086d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4009900 tx=0x7ff2f402a410 comp rx=0 tx=0).stop 2024-07-26T06:52:26.183 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.181+0000 7ff305a6f640 1 -- 172.21.15.42:0/3637788297 shutdown_connections 2024-07-26T06:52:26.183 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.181+0000 7ff305a6f640 1 --2- 172.21.15.42:0/3637788297 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff300108e10 0x7ff30010b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.183 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.181+0000 7ff305a6f640 1 --2- 172.21.15.42:0/3637788297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff3001062b0 0x7ff3001086d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.183 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.181+0000 7ff305a6f640 1 --2- 172.21.15.42:0/3637788297 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff30006b0e0 0x7ff300105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.183 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.181+0000 7ff305a6f640 1 -- 172.21.15.42:0/3637788297 >> 172.21.15.42:0/3637788297 conn(0x7ff3000fd910 msgr2=0x7ff3000ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:26.183 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.181+0000 7ff305a6f640 1 -- 172.21.15.42:0/3637788297 shutdown_connections 2024-07-26T06:52:26.183 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.182+0000 7ff305a6f640 1 -- 172.21.15.42:0/3637788297 wait complete. 2024-07-26T06:52:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.182+0000 7ff305a6f640 1 Processor -- start 2024-07-26T06:52:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.182+0000 7ff305a6f640 1 -- start start 2024-07-26T06:52:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff305a6f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff30006b0e0 0x7ff3001033b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff305a6f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3001062b0 0x7ff300101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff305a6f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff300108e10 0x7ff3001021b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff305a6f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff30010ed30 con 0x7ff3001062b0 2024-07-26T06:52:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff305a6f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff30010ebb0 con 0x7ff30006b0e0 2024-07-26T06:52:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff305a6f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff30010eeb0 con 0x7ff300108e10 2024-07-26T06:52:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff2feffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3001062b0 0x7ff300101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff2feffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3001062b0 0x7ff300101c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58008/0 (socket says 172.21.15.42:58008) 2024-07-26T06:52:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff2feffd640 1 -- 172.21.15.42:0/3550932323 learned_addr learned my addr 172.21.15.42:0/3550932323 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff2ff7fe640 1 --2- 172.21.15.42:0/3550932323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff30006b0e0 0x7ff3001033b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff2fffff640 1 --2- 172.21.15.42:0/3550932323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff300108e10 0x7ff3001021b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff2ff7fe640 1 -- 172.21.15.42:0/3550932323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff300108e10 msgr2=0x7ff3001021b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff2ff7fe640 1 --2- 172.21.15.42:0/3550932323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff300108e10 0x7ff3001021b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff2ff7fe640 1 -- 172.21.15.42:0/3550932323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3001062b0 msgr2=0x7ff300101c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff2ff7fe640 1 --2- 172.21.15.42:0/3550932323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3001062b0 0x7ff300101c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff2ff7fe640 1 -- 172.21.15.42:0/3550932323 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3001028e0 con 0x7ff30006b0e0 2024-07-26T06:52:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff2feffd640 1 --2- 172.21.15.42:0/3550932323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3001062b0 0x7ff300101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:52:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.183+0000 7ff2fffff640 1 --2- 172.21.15.42:0/3550932323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff300108e10 0x7ff3001021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:52:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.184+0000 7ff2ff7fe640 1 --2- 172.21.15.42:0/3550932323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff30006b0e0 0x7ff3001033b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff2f000e9f0 tx=0x7ff2f000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.184+0000 7ff2fcff9640 1 -- 172.21.15.42:0/3550932323 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f000ccc0 con 0x7ff30006b0e0 2024-07-26T06:52:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.184+0000 7ff2fcff9640 1 -- 172.21.15.42:0/3550932323 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2f0005020 con 0x7ff30006b0e0 2024-07-26T06:52:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.184+0000 7ff305a6f640 1 -- 172.21.15.42:0/3550932323 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3001a3a40 con 0x7ff30006b0e0 2024-07-26T06:52:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.184+0000 7ff2fcff9640 1 -- 172.21.15.42:0/3550932323 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f0004a00 con 0x7ff30006b0e0 2024-07-26T06:52:26.188 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.184+0000 7ff305a6f640 1 -- 172.21.15.42:0/3550932323 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3001a3eb0 con 0x7ff30006b0e0 2024-07-26T06:52:26.194 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.186+0000 7ff30c982740 1 Processor -- start 2024-07-26T06:52:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.186+0000 7ff30c982740 1 -- start start 2024-07-26T06:52:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff30c982740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4521b0 0x560e8c7f7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff30c982740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 0x560e8c7f77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff30c982740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c7f8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff30c982740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x560e8c76a3b0 con 0x560e8c4292f0 2024-07-26T06:52:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff30c982740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560e8c7598b0 con 0x560e8c4521b0 2024-07-26T06:52:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff30c982740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x560e8c75d8f0 con 0x560e8c4314d0 2024-07-26T06:52:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff2fcff9640 1 -- 172.21.15.42:0/3550932323 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7ff2f0004ba0 con 0x7ff30006b0e0 2024-07-26T06:52:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff2fffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4521b0 0x560e8c7f7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff2fffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4521b0 0x560e8c7f7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:34354/0 (socket says 172.21.15.42:34354) 2024-07-26T06:52:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff2fffff640 1 -- 172.21.15.42:0/244504130 learned_addr learned my addr 172.21.15.42:0/244504130 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff2ff7fe640 1 --2- 172.21.15.42:0/244504130 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 0x560e8c7f77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff2feffd640 1 --2- 172.21.15.42:0/244504130 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c7f8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff2fffff640 1 -- 172.21.15.42:0/244504130 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 msgr2=0x560e8c7f8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff2fffff640 1 --2- 172.21.15.42:0/244504130 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c7f8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff2fffff640 1 -- 172.21.15.42:0/244504130 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 msgr2=0x560e8c7f77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff2fffff640 1 --2- 172.21.15.42:0/244504130 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 0x560e8c7f77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff2fffff640 1 -- 172.21.15.42:0/244504130 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e8c7009a0 con 0x560e8c4521b0 2024-07-26T06:52:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff2feffd640 1 --2- 172.21.15.42:0/244504130 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c7f8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:52:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.187+0000 7ff2fffff640 1 --2- 172.21.15.42:0/244504130 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4521b0 0x560e8c7f7180 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8011410 tx=0x7ff2f80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fd3bdfd8d6e47b9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.188+0000 7ff2e5ffb640 1 -- 172.21.15.42:0/244504130 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8012770 con 0x560e8c4521b0 2024-07-26T06:52:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.188+0000 7ff2e5ffb640 1 -- 172.21.15.42:0/244504130 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2f8012dc0 con 0x560e8c4521b0 2024-07-26T06:52:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.188+0000 7ff2fcff9640 1 --2- 172.21.15.42:0/3550932323 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff2e80b3bd0 0x7ff2e80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.188+0000 7ff2e5ffb640 1 -- 172.21.15.42:0/244504130 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f801a3f0 con 0x560e8c4521b0 2024-07-26T06:52:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.188+0000 7ff2ff7fe640 1 --2- 172.21.15.42:0/3550932323 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff2e80b3bd0 0x7ff2e80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.188+0000 7ff2fcff9640 1 -- 172.21.15.42:0/3550932323 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7ff2f00e3cb0 con 0x7ff30006b0e0 2024-07-26T06:52:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.189+0000 7ff2ff7fe640 1 --2- 172.21.15.42:0/3550932323 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff2e80b3bd0 0x7ff2e80b60e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff2f00aab70 tx=0x7ff2f00e7790 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.189+0000 7ff30c982740 1 -- 172.21.15.42:0/244504130 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4521b0 msgr2=0x560e8c7f7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.189+0000 7ff30c982740 1 --2- 172.21.15.42:0/244504130 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4521b0 0x560e8c7f7180 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8011410 tx=0x7ff2f80118d0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.189+0000 7ff30c982740 1 -- 172.21.15.42:0/244504130 shutdown_connections 2024-07-26T06:52:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.189+0000 7ff30c982740 1 --2- 172.21.15.42:0/244504130 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c7f8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.189+0000 7ff30c982740 1 --2- 172.21.15.42:0/244504130 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 0x560e8c7f77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.189+0000 7ff30c982740 1 --2- 172.21.15.42:0/244504130 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4521b0 0x560e8c7f7180 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.189+0000 7ff30c982740 1 -- 172.21.15.42:0/244504130 >> 172.21.15.42:0/244504130 conn(0x560e8c446300 msgr2=0x560e8c7f2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.189+0000 7ff30c982740 1 -- 172.21.15.42:0/244504130 shutdown_connections 2024-07-26T06:52:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.189+0000 7ff30c982740 1 -- 172.21.15.42:0/244504130 wait complete. 2024-07-26T06:52:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.189+0000 7ff30c982740 1 -- start start 2024-07-26T06:52:26.198 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff30c982740 1 Processor -- start 2024-07-26T06:52:26.198 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff30c982740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 0x560e8c800ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.198 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff30c982740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c801260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.198 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff30c982740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4521b0 0x560e8c801dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.198 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff30c982740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x560e8c693ab0 con 0x560e8c4292f0 2024-07-26T06:52:26.198 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff30c982740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560e8c6e1cf0 con 0x560e8c4521b0 2024-07-26T06:52:26.198 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff30c982740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x560e8c7f8e60 con 0x560e8c4314d0 2024-07-26T06:52:26.198 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff30c982740 10 client.0 _ll_register_callbacks cb 0x560e8c7539f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:52:26.198 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff2fffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 0x560e8c800ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.198 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff2fffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 0x560e8c800ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58016/0 (socket says 172.21.15.42:58016) 2024-07-26T06:52:26.198 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff2fffff640 1 -- 172.21.15.42:0/747736156 learned_addr learned my addr 172.21.15.42:0/747736156 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff2fffff640 1 --2- 172.21.15.42:0/747736156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4521b0 0x560e8c801dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff2feffd640 1 --2- 172.21.15.42:0/747736156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c801260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff2fffff640 1 -- 172.21.15.42:0/747736156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 msgr2=0x560e8c801260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff2fffff640 1 --2- 172.21.15.42:0/747736156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c801260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff2fffff640 1 -- 172.21.15.42:0/747736156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4521b0 msgr2=0x560e8c801dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff2fffff640 1 --2- 172.21.15.42:0/747736156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4521b0 0x560e8c801dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff2fffff640 1 -- 172.21.15.42:0/747736156 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e8c4bd790 con 0x560e8c4292f0 2024-07-26T06:52:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff2feffd640 1 --2- 172.21.15.42:0/747736156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c801260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:52:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.190+0000 7ff2fffff640 1 --2- 172.21.15.42:0/747736156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 0x560e8c800ba0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff2f800f950 tx=0x7ff2f8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=617eae7a81e8cb29 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:26.202 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.191+0000 7ff2deffd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:52:26.202 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.191+0000 7ff2deffd640 1 -- 172.21.15.42:0/747736156 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8012770 con 0x560e8c4292f0 2024-07-26T06:52:26.202 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.191+0000 7ff2deffd640 1 -- 172.21.15.42:0/747736156 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7ff2f8012910 con 0x560e8c4292f0 2024-07-26T06:52:26.202 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.191+0000 7ff30c982740 1 -- 172.21.15.42:0/747736156 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2f8012dc0 con 0x560e8c4292f0 2024-07-26T06:52:26.202 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.191+0000 7ff2deffd640 1 -- 172.21.15.42:0/747736156 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff2f8012b10 con 0x560e8c4292f0 2024-07-26T06:52:26.203 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.191+0000 7ff30c982740 10 client.15690 fetch_fsmap learned FSMap version 31 2024-07-26T06:52:26.203 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.191+0000 7ff30c982740 1 -- 172.21.15.42:0/747736156 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560e8c536620 con 0x560e8c4292f0 2024-07-26T06:52:26.203 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.191+0000 7ff2deffd640 1 -- 172.21.15.42:0/747736156 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2f8012dc0 con 0x560e8c4292f0 2024-07-26T06:52:26.203 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.191+0000 7ff2deffd640 1 -- 172.21.15.42:0/747736156 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7ff2f8026490 con 0x560e8c4292f0 2024-07-26T06:52:26.203 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.191+0000 7ff30c982740 10 client.15690 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T06:52:26.203 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.191+0000 7ff30c982740 10 client.15690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:52:26.203 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.191+0000 7ff30c982740 20 client.15690 populate_metadata read hostname 'smithi042' 2024-07-26T06:52:26.203 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.191+0000 7ff30c982740 1 --2- 172.21.15.42:0/747736156 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x560e8c410d80 0x560e8c80e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.203 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.191+0000 7ff30c982740 4 client.15690 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:52:26.203 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.191+0000 7ff30c982740 1 -- 172.21.15.42:0/747736156 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e8c76c300 con 0x560e8c410d80 2024-07-26T06:52:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.192+0000 7ff2feffd640 1 --2- 172.21.15.42:0/747736156 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x560e8c410d80 0x560e8c80e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.192+0000 7ff2deffd640 10 client.15690 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.192+0000 7ff2feffd640 1 --2- 172.21.15.42:0/747736156 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x560e8c410d80 0x560e8c80e020 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1a43f3fa5db2e62 server_cookie=8137deef713d0e2c in_seq=0 out_seq=0 2024-07-26T06:52:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.193+0000 7ff2deffd640 1 -- 172.21.15.42:0/747736156 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560e8c76c300 con 0x560e8c410d80 2024-07-26T06:52:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.193+0000 7ff2deffd640 10 client.15690 handle_command_reply: tid=0 2024-07-26T06:52:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.193+0000 7ff30c982740 1 client.15690 shutdown 2024-07-26T06:52:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.193+0000 7ff30c982740 1 -- 172.21.15.42:0/747736156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 msgr2=0x560e8c800ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.193+0000 7ff30c982740 1 --2- 172.21.15.42:0/747736156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 0x560e8c800ba0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff2f800f950 tx=0x7ff2f8009220 comp rx=0 tx=0).stop 2024-07-26T06:52:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.193+0000 7ff30c982740 1 -- 172.21.15.42:0/747736156 shutdown_connections 2024-07-26T06:52:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.193+0000 7ff30c982740 1 --2- 172.21.15.42:0/747736156 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x560e8c410d80 0x560e8c80e020 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.193+0000 7ff30c982740 1 --2- 172.21.15.42:0/747736156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4521b0 0x560e8c801dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.193+0000 7ff30c982740 1 --2- 172.21.15.42:0/747736156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c801260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.193+0000 7ff30c982740 1 --2- 172.21.15.42:0/747736156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 0x560e8c800ba0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.193+0000 7ff30c982740 1 -- 172.21.15.42:0/747736156 >> 172.21.15.42:0/747736156 conn(0x560e8c446300 msgr2=0x560e8c7f12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.193+0000 7ff2deffd640 0 client.15690 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.193+0000 7ff30c982740 1 -- 172.21.15.42:0/747736156 shutdown_connections 2024-07-26T06:52:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.193+0000 7ff30c982740 1 -- 172.21.15.42:0/747736156 wait complete. 2024-07-26T06:52:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.193+0000 7ff30c982740 20 client.15690 trim_cache size 0 max 16384 2024-07-26T06:52:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff30c982740 1 Processor -- start 2024-07-26T06:52:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff30c982740 1 -- start start 2024-07-26T06:52:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff30c982740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 0x560e8c7ff0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff30c982740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c7ff630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff30c982740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4521b0 0x560e8c7f3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff30c982740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff3001acc60 con 0x560e8c4292f0 2024-07-26T06:52:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff30c982740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff3001ae800 con 0x560e8c4521b0 2024-07-26T06:52:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff30c982740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff3001ad330 con 0x560e8c4314d0 2024-07-26T06:52:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff2fffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 0x560e8c7ff0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff2fffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 0x560e8c7ff0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58030/0 (socket says 172.21.15.42:58030) 2024-07-26T06:52:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff2fffff640 1 -- 172.21.15.42:0/629704520 learned_addr learned my addr 172.21.15.42:0/629704520 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff2fffff640 1 --2- 172.21.15.42:0/629704520 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4521b0 0x560e8c7f3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff2feffd640 1 --2- 172.21.15.42:0/629704520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c7ff630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff2fffff640 1 -- 172.21.15.42:0/629704520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 msgr2=0x560e8c7ff630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff2fffff640 1 --2- 172.21.15.42:0/629704520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c7ff630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff2fffff640 1 -- 172.21.15.42:0/629704520 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4521b0 msgr2=0x560e8c7f3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff2fffff640 1 --2- 172.21.15.42:0/629704520 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4521b0 0x560e8c7f3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.209+0000 7ff2fffff640 1 -- 172.21.15.42:0/629704520 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e8c410fb0 con 0x560e8c4292f0 2024-07-26T06:52:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.210+0000 7ff2fffff640 1 --2- 172.21.15.42:0/629704520 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 0x560e8c7ff0f0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009f50 tx=0x7ff2f80118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d4167eb6d390e2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.210+0000 7ff2e57fa640 1 -- 172.21.15.42:0/629704520 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f800f530 con 0x560e8c4292f0 2024-07-26T06:52:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.210+0000 7ff2e57fa640 1 -- 172.21.15.42:0/629704520 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2f802e070 con 0x560e8c4292f0 2024-07-26T06:52:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.210+0000 7ff2e57fa640 1 -- 172.21.15.42:0/629704520 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8026710 con 0x560e8c4292f0 2024-07-26T06:52:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.210+0000 7ff30c982740 1 -- 172.21.15.42:0/629704520 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 msgr2=0x560e8c7ff0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.210+0000 7ff30c982740 1 --2- 172.21.15.42:0/629704520 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 0x560e8c7ff0f0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009f50 tx=0x7ff2f80118d0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.211+0000 7ff30c982740 1 -- 172.21.15.42:0/629704520 shutdown_connections 2024-07-26T06:52:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.211+0000 7ff30c982740 1 --2- 172.21.15.42:0/629704520 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4521b0 0x560e8c7f3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.211+0000 7ff30c982740 1 --2- 172.21.15.42:0/629704520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c7ff630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.211+0000 7ff30c982740 1 --2- 172.21.15.42:0/629704520 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4292f0 0x560e8c7ff0f0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.211+0000 7ff30c982740 1 -- 172.21.15.42:0/629704520 >> 172.21.15.42:0/629704520 conn(0x560e8c446300 msgr2=0x560e8c800700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.211+0000 7ff30c982740 1 -- 172.21.15.42:0/629704520 shutdown_connections 2024-07-26T06:52:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.211+0000 7ff30c982740 1 -- 172.21.15.42:0/629704520 wait complete. 2024-07-26T06:52:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.211+0000 7ff30c982740 1 -- start start 2024-07-26T06:52:26.217 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.211+0000 7ff30c982740 1 Processor -- start 2024-07-26T06:52:26.217 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.211+0000 7ff30c982740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4292f0 0x560e8c8083c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.217 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.211+0000 7ff30c982740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c808a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.217 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff30c982740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4521b0 0x560e8c806cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.217 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff30c982740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff3001ae0d0 con 0x560e8c4521b0 2024-07-26T06:52:26.217 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff30c982740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff3001ada00 con 0x560e8c4292f0 2024-07-26T06:52:26.218 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff30c982740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff3001aac40 con 0x560e8c4314d0 2024-07-26T06:52:26.218 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff30c982740 10 client.0 _ll_register_callbacks cb 0x7ff2f00e7660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:52:26.218 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff2fffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4292f0 0x560e8c8083c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.218 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff2fffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4521b0 0x560e8c806cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.218 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff2feffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c808a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.218 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff2feffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c808a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:46874/0 (socket says 172.21.15.42:46874) 2024-07-26T06:52:26.218 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff2fffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4521b0 0x560e8c806cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58038/0 (socket says 172.21.15.42:58038) 2024-07-26T06:52:26.219 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff2feffd640 1 -- 172.21.15.42:0/946657157 learned_addr learned my addr 172.21.15.42:0/946657157 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:26.219 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff2feffd640 1 -- 172.21.15.42:0/946657157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4292f0 msgr2=0x560e8c8083c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:26.219 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff2feffd640 1 --2- 172.21.15.42:0/946657157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4292f0 0x560e8c8083c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.219 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff2feffd640 1 -- 172.21.15.42:0/946657157 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4521b0 msgr2=0x560e8c806cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:26.220 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff2feffd640 1 --2- 172.21.15.42:0/946657157 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4521b0 0x560e8c806cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.220 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff2feffd640 1 -- 172.21.15.42:0/946657157 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e8c807480 con 0x560e8c4314d0 2024-07-26T06:52:26.220 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff2fffff640 1 --2- 172.21.15.42:0/946657157 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4521b0 0x560e8c806cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:52:26.220 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff2feffd640 1 --2- 172.21.15.42:0/946657157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c808a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4009230 tx=0x7ff2f4004120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8db3ea5d06f1b2c7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:26.220 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff2deffd640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-07-26T06:52:26.220 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.212+0000 7ff2deffd640 1 -- 172.21.15.42:0/946657157 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4035040 con 0x560e8c4314d0 2024-07-26T06:52:26.220 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.213+0000 7ff30c982740 1 -- 172.21.15.42:0/946657157 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2f802e070 con 0x560e8c4314d0 2024-07-26T06:52:26.220 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.213+0000 7ff2deffd640 1 -- 172.21.15.42:0/946657157 <== mon.1 v2:172.21.15.164:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7ff2f4031900 con 0x560e8c4314d0 2024-07-26T06:52:26.220 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.213+0000 7ff2deffd640 1 -- 172.21.15.42:0/946657157 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff2f4053070 con 0x560e8c4314d0 2024-07-26T06:52:26.220 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.213+0000 7ff30c982740 10 client.25345 fetch_fsmap learned FSMap version 31 2024-07-26T06:52:26.220 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.213+0000 7ff30c982740 1 -- 172.21.15.42:0/946657157 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560e8c8752d0 con 0x560e8c4314d0 2024-07-26T06:52:26.220 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.213+0000 7ff2deffd640 1 -- 172.21.15.42:0/946657157 <== mon.1 v2:172.21.15.164:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2f802e070 con 0x560e8c4314d0 2024-07-26T06:52:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.213+0000 7ff2deffd640 1 -- 172.21.15.42:0/946657157 <== mon.1 v2:172.21.15.164:3300/0 5 ==== fsmap(e 31) ==== 8370+0+0 (secure 0 0 0) 0x7ff2f4032040 con 0x560e8c4314d0 2024-07-26T06:52:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.213+0000 7ff30c982740 10 client.25345 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T06:52:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.213+0000 7ff30c982740 10 client.25345 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:52:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.213+0000 7ff30c982740 20 client.25345 populate_metadata read hostname 'smithi042' 2024-07-26T06:52:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.213+0000 7ff30c982740 1 --2- 172.21.15.42:0/946657157 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x560e8c3e61c0 0x560e8c877d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.213+0000 7ff30c982740 4 client.25345 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:52:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.213+0000 7ff30c982740 1 -- 172.21.15.42:0/946657157 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560e8c767da0 con 0x560e8c3e61c0 2024-07-26T06:52:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.213+0000 7ff2fffff640 1 --2- 172.21.15.42:0/946657157 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x560e8c3e61c0 0x560e8c877d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.214+0000 7ff2fffff640 1 --2- 172.21.15.42:0/946657157 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x560e8c3e61c0 0x560e8c877d00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9af2832de33cdfc7 server_cookie=8323bd1fb21e1baa in_seq=0 out_seq=0 2024-07-26T06:52:26.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.214+0000 7ff2deffd640 10 client.25345 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.831+0000 7ff2deffd640 1 -- 172.21.15.42:0/946657157 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560e8c767da0 con 0x560e8c3e61c0 2024-07-26T06:52:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.831+0000 7ff2deffd640 10 client.25345 handle_command_reply: tid=0 2024-07-26T06:52:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.831+0000 7ff30c982740 1 client.25345 shutdown 2024-07-26T06:52:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.831+0000 7ff30c982740 1 -- 172.21.15.42:0/946657157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 msgr2=0x560e8c808a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:26.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.831+0000 7ff30c982740 1 --2- 172.21.15.42:0/946657157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c808a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4009230 tx=0x7ff2f4004120 comp rx=0 tx=0).stop 2024-07-26T06:52:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.831+0000 7ff30c982740 1 -- 172.21.15.42:0/946657157 shutdown_connections 2024-07-26T06:52:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.831+0000 7ff30c982740 1 --2- 172.21.15.42:0/946657157 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x560e8c3e61c0 0x560e8c877d00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.831+0000 7ff30c982740 1 --2- 172.21.15.42:0/946657157 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x560e8c4521b0 0x560e8c806cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.831+0000 7ff30c982740 1 --2- 172.21.15.42:0/946657157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560e8c4314d0 0x560e8c808a60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.831+0000 7ff2deffd640 0 client.25345 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.831+0000 7ff30c982740 1 --2- 172.21.15.42:0/946657157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560e8c4292f0 0x560e8c8083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.831+0000 7ff30c982740 1 -- 172.21.15.42:0/946657157 >> 172.21.15.42:0/946657157 conn(0x560e8c446300 msgr2=0x560e8c7fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.831+0000 7ff30c982740 1 -- 172.21.15.42:0/946657157 shutdown_connections 2024-07-26T06:52:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.831+0000 7ff30c982740 1 -- 172.21.15.42:0/946657157 wait complete. 2024-07-26T06:52:26.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.831+0000 7ff30c982740 20 client.25345 trim_cache size 0 max 16384 2024-07-26T06:52:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:52:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:52:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:52:26.834 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:52:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.832+0000 7ff305a6f640 1 -- 172.21.15.42:0/3550932323 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff2e80b3bd0 msgr2=0x7ff2e80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.832+0000 7ff305a6f640 1 --2- 172.21.15.42:0/3550932323 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff2e80b3bd0 0x7ff2e80b60e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff2f00aab70 tx=0x7ff2f00e7790 comp rx=0 tx=0).stop 2024-07-26T06:52:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.832+0000 7ff305a6f640 1 -- 172.21.15.42:0/3550932323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff30006b0e0 msgr2=0x7ff3001033b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.832+0000 7ff305a6f640 1 --2- 172.21.15.42:0/3550932323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff30006b0e0 0x7ff3001033b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff2f000e9f0 tx=0x7ff2f000eeb0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.832+0000 7ff305a6f640 1 -- 172.21.15.42:0/3550932323 shutdown_connections 2024-07-26T06:52:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.832+0000 7ff305a6f640 1 --2- 172.21.15.42:0/3550932323 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff2e80b3bd0 0x7ff2e80b60e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.832+0000 7ff305a6f640 1 --2- 172.21.15.42:0/3550932323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff300108e10 0x7ff3001021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.832+0000 7ff305a6f640 1 --2- 172.21.15.42:0/3550932323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3001062b0 0x7ff300101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.832+0000 7ff305a6f640 1 --2- 172.21.15.42:0/3550932323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff30006b0e0 0x7ff3001033b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.832+0000 7ff305a6f640 1 -- 172.21.15.42:0/3550932323 >> 172.21.15.42:0/3550932323 conn(0x7ff3000fd910 msgr2=0x7ff300104860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.832+0000 7ff305a6f640 1 -- 172.21.15.42:0/3550932323 shutdown_connections 2024-07-26T06:52:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:26.832+0000 7ff305a6f640 1 -- 172.21.15.42:0/3550932323 wait complete. 2024-07-26T06:52:26.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:26 smithi042 ceph-mon[32775]: pgmap v462: 97 pgs: 97 active+clean; 20 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 418 KiB/s wr, 52 op/s 2024-07-26T06:52:26.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:26 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:26 smithi164 ceph-mon[36970]: pgmap v462: 97 pgs: 97 active+clean; 20 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 418 KiB/s wr, 52 op/s 2024-07-26T06:52:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:26 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:27.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:26 smithi163 ceph-mon[36873]: pgmap v462: 97 pgs: 97 active+clean; 20 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 418 KiB/s wr, 52 op/s 2024-07-26T06:52:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:26 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:28.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:28 smithi042 ceph-mon[32775]: pgmap v463: 97 pgs: 97 active+clean; 22 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 618 KiB/s wr, 80 op/s 2024-07-26T06:52:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:28 smithi164 ceph-mon[36970]: pgmap v463: 97 pgs: 97 active+clean; 22 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 618 KiB/s wr, 80 op/s 2024-07-26T06:52:29.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:28 smithi163 ceph-mon[36873]: pgmap v463: 97 pgs: 97 active+clean; 22 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 618 KiB/s wr, 80 op/s 2024-07-26T06:52:30.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:30 smithi042 ceph-mon[32775]: pgmap v464: 97 pgs: 97 active+clean; 21 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 951 KiB/s wr, 373 op/s 2024-07-26T06:52:30.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:30 smithi164 ceph-mon[36970]: pgmap v464: 97 pgs: 97 active+clean; 21 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 951 KiB/s wr, 373 op/s 2024-07-26T06:52:31.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:30 smithi163 ceph-mon[36873]: pgmap v464: 97 pgs: 97 active+clean; 21 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 951 KiB/s wr, 373 op/s 2024-07-26T06:52:31.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:31 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:31 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:32.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:31 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:32.612 INFO:tasks.workunit.client.0.smithi042.stderr:Updating files: 7% (322/4329) Updating files: 8% (347/4329) Updating files: 9% (390/4329) Updating files: 10% (433/4329) Updating files: 11% (477/4329) Updating files: 11% (491/4329) Updating files: 12% (520/4329) Updating files: 13% (563/4329) Updating files: 14% (607/4329) Updating files: 15% (650/4329) Updating files: 16% (693/4329) Updating files: 17% (736/4329) Updating files: 18% (780/4329) Updating files: 19% (823/4329) Updating files: 20% (866/4329) Updating files: 21% (910/4329) Updating files: 22% (953/4329) Updating files: 23% (996/4329) Updating files: 24% (1039/4329) Updating files: 25% (1083/4329) Updating files: 26% (1126/4329) Updating files: 27% (1169/4329) Updating files: 28% (1213/4329) Updating files: 28% (1225/4329) Updating files: 29% (1256/4329) Updating files: 29% (1289/4329) Updating files: 30% (1299/4329) Updating files: 31% (1342/4329) Updating files: 32% (1386/4329) Updating files: 33% (1429/4329) Updating files: 34% (1472/4329) Updating files: 35% (1516/4329) Updating files: 36% (1559/4329) Updating files: 37% (1602/4329) Updating files: 38% (1646/4329) Updating files: 39% (1689/4329) Updating files: 40% (1732/4329) Updating files: 41% (1775/4329) Updating files: 42% (1819/4329) Updating files: 43% (1862/4329) Updating files: 44% (1905/4329) Updating files: 44% (1919/4329) Updating files: 45% (1949/4329) Updating files: 46% (1992/4329) Updating files: 47% (2035/4329) Updating files: 48% (2078/4329) Updating files: 49% (2122/4329) Updating files: 50% (2165/4329) Updating files: 51% (2208/4329) Updating files: 52% (2252/4329) Updating files: 53% (2295/4329) Updating files: 54% (2338/4329) Updating files: 55% (2381/4329) Updating files: 56% (2425/4329) Updating files: 56% (2457/4329) Updating files: 57% (2468/4329) Updating files: 58% (2511/4329) Updating files: 59% (2555/4329) Updating files: 60% (2598/4329) Updating files: 61% (2641/4329) Updating files: 62% (2684/4329) Updating files: 63% (2728/4329) Updating files: 64% (2771/4329) Updating files: 65% (2814/4329) Updating files: 65% (2850/4329) Updating files: 66% (2858/4329) Updating files: 67% (2901/4329) Updating files: 68% (2944/4329) Updating files: 69% (2988/4329) Updating files: 69% (3022/4329) Updating files: 70% (3031/4329) Updating files: 71% (3074/4329) Updating files: 72% (3117/4329) Updating files: 73% (3161/4329) Updating files: 74% (3204/4329) Updating files: 75% (3247/4329) Updating files: 76% (3291/4329) Updating files: 77% (3334/4329) Updating files: 78% (3377/4329) Updating files: 79% (3420/4329) Updating files: 80% (3464/4329) Updating files: 81% (3507/4329) Updating files: 82% (3550/4329) Updating files: 83% (3594/4329) Updating files: 84% (3637/4329) Updating files: 85% (3680/4329) Updating files: 86% (3723/4329) Updating files: 86% (3759/4329) Updating files: 87% (3767/4329) Updating files: 88% (3810/4329) Updating files: 89% (3853/4329) Updating files: 90% (3897/4329) Updating files: 91% (3940/4329) Updating files: 92% (3983/4329) Updating files: 93% (4026/4329) Updating files: 94% (4070/4329) Updating files: 95% (4113/4329) Updating files: 96% (4156/4329) Updating files: 97% (4200/4329) Updating files: 97% (4216/4329) Updating files: 98% (4243/4329) Updating files: 99% (4286/4329) Updating files: 100% (4329/4329) Updating files: 100% (4329/4329), done. 2024-07-26T06:52:32.626 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd xfstests-dev 2024-07-26T06:52:32.626 INFO:tasks.workunit.client.0.smithi042.stderr:+ git checkout 0e5c12dfd008efc2848c98108c9237487e91ef35 2024-07-26T06:52:32.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:32 smithi042 ceph-mon[32775]: pgmap v465: 97 pgs: 97 active+clean; 22 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 848 KiB/s wr, 442 op/s 2024-07-26T06:52:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:32 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:32 smithi164 ceph-mon[36970]: pgmap v465: 97 pgs: 97 active+clean; 22 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 848 KiB/s wr, 442 op/s 2024-07-26T06:52:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:32 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:33.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:32 smithi163 ceph-mon[36873]: pgmap v465: 97 pgs: 97 active+clean; 22 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 848 KiB/s wr, 442 op/s 2024-07-26T06:52:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:32 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:34.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:34 smithi042 ceph-mon[32775]: pgmap v466: 97 pgs: 97 active+clean; 24 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 483 op/s 2024-07-26T06:52:34.929 INFO:tasks.workunit.client.0.smithi042.stderr:Note: switching to '0e5c12dfd008efc2848c98108c9237487e91ef35'. 2024-07-26T06:52:34.930 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-07-26T06:52:34.930 INFO:tasks.workunit.client.0.smithi042.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-07-26T06:52:34.930 INFO:tasks.workunit.client.0.smithi042.stderr:changes and commit them, and you can discard any commits you make in this 2024-07-26T06:52:34.930 INFO:tasks.workunit.client.0.smithi042.stderr:state without impacting any branches by switching back to a branch. 2024-07-26T06:52:34.930 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-07-26T06:52:34.931 INFO:tasks.workunit.client.0.smithi042.stderr:If you want to create a new branch to retain commits you create, you may 2024-07-26T06:52:34.931 INFO:tasks.workunit.client.0.smithi042.stderr:do so (now or later) by using -c with the switch command. Example: 2024-07-26T06:52:34.931 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-07-26T06:52:34.931 INFO:tasks.workunit.client.0.smithi042.stderr: git switch -c 2024-07-26T06:52:34.931 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-07-26T06:52:34.931 INFO:tasks.workunit.client.0.smithi042.stderr:Or undo this operation with: 2024-07-26T06:52:34.931 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-07-26T06:52:34.932 INFO:tasks.workunit.client.0.smithi042.stderr: git switch - 2024-07-26T06:52:34.932 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-07-26T06:52:34.932 INFO:tasks.workunit.client.0.smithi042.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-07-26T06:52:34.932 INFO:tasks.workunit.client.0.smithi042.stderr: 2024-07-26T06:52:34.932 INFO:tasks.workunit.client.0.smithi042.stderr:HEAD is now at 0e5c12df xfs/522: use reflink instead of crc as test feature 2024-07-26T06:52:34.937 INFO:tasks.workunit.client.0.smithi042.stderr:+ make -j4 2024-07-26T06:52:34.947 INFO:tasks.workunit.client.0.smithi042.stdout:libtoolize -cfi 2024-07-26T06:52:34.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:34 smithi164 ceph-mon[36970]: pgmap v466: 97 pgs: 97 active+clean; 24 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 483 op/s 2024-07-26T06:52:35.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:34 smithi163 ceph-mon[36873]: pgmap v466: 97 pgs: 97 active+clean; 24 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 483 op/s 2024-07-26T06:52:35.695 INFO:tasks.workunit.client.0.smithi042.stdout:libtoolize: putting auxiliary files in '.'. 2024-07-26T06:52:35.695 INFO:tasks.workunit.client.0.smithi042.stdout:libtoolize: copying file './config.guess' 2024-07-26T06:52:35.721 INFO:tasks.workunit.client.0.smithi042.stdout:libtoolize: copying file './config.sub' 2024-07-26T06:52:35.737 INFO:tasks.workunit.client.0.smithi042.stdout:libtoolize: copying file './install-sh' 2024-07-26T06:52:35.746 INFO:tasks.workunit.client.0.smithi042.stdout:libtoolize: copying file './ltmain.sh' 2024-07-26T06:52:35.907 INFO:tasks.workunit.client.0.smithi042.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2024-07-26T06:52:35.907 INFO:tasks.workunit.client.0.smithi042.stdout:libtoolize: copying file 'm4/libtool.m4' 2024-07-26T06:52:35.922 INFO:tasks.workunit.client.0.smithi042.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2024-07-26T06:52:35.930 INFO:tasks.workunit.client.0.smithi042.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2024-07-26T06:52:35.936 INFO:tasks.workunit.client.0.smithi042.stdout:libtoolize: copying file 'm4/ltversion.m4' 2024-07-26T06:52:35.945 INFO:tasks.workunit.client.0.smithi042.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2024-07-26T06:52:35.952 INFO:tasks.workunit.client.0.smithi042.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2024-07-26T06:52:35.953 INFO:tasks.workunit.client.0.smithi042.stdout:cp include/install-sh . 2024-07-26T06:52:35.979 INFO:tasks.workunit.client.0.smithi042.stdout:aclocal -I m4 2024-07-26T06:52:36.807 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:36 smithi163 ceph-mon[36873]: pgmap v467: 97 pgs: 97 active+clean; 32 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 496 op/s 2024-07-26T06:52:36.808 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:36 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:36.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:36 smithi042 ceph-mon[32775]: pgmap v467: 97 pgs: 97 active+clean; 32 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 496 op/s 2024-07-26T06:52:36.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:36 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:36 smithi164 ceph-mon[36970]: pgmap v467: 97 pgs: 97 active+clean; 32 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 496 op/s 2024-07-26T06:52:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:36 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:37.061 INFO:tasks.workunit.client.0.smithi042.stdout:autoheader 2024-07-26T06:52:37.391 INFO:tasks.workunit.client.0.smithi042.stdout:autoconf 2024-07-26T06:52:37.580 INFO:tasks.workunit.client.0.smithi042.stdout:./configure \ 2024-07-26T06:52:37.580 INFO:tasks.workunit.client.0.smithi042.stdout: --libexecdir=/usr/lib \ 2024-07-26T06:52:37.580 INFO:tasks.workunit.client.0.smithi042.stdout: --exec_prefix=/var/lib 2024-07-26T06:52:37.715 INFO:tasks.workunit.client.0.smithi042.stdout:checking build system type... x86_64-pc-linux-gnu 2024-07-26T06:52:37.715 INFO:tasks.workunit.client.0.smithi042.stdout:checking host system type... x86_64-pc-linux-gnu 2024-07-26T06:52:37.716 INFO:tasks.workunit.client.0.smithi042.stdout:checking how to print strings... printf 2024-07-26T06:52:37.716 INFO:tasks.workunit.client.0.smithi042.stdout:checking for gcc... gcc 2024-07-26T06:52:37.805 INFO:tasks.workunit.client.0.smithi042.stdout:checking whether the C compiler works... yes 2024-07-26T06:52:37.805 INFO:tasks.workunit.client.0.smithi042.stdout:checking for C compiler default output file name... a.out 2024-07-26T06:52:37.844 INFO:tasks.workunit.client.0.smithi042.stdout:checking for suffix of executables... 2024-07-26T06:52:37.883 INFO:tasks.workunit.client.0.smithi042.stdout:checking whether we are cross compiling... no 2024-07-26T06:52:37.921 INFO:tasks.workunit.client.0.smithi042.stdout:checking for suffix of object files... o 2024-07-26T06:52:37.946 INFO:tasks.workunit.client.0.smithi042.stdout:checking whether we are using the GNU C compiler... yes 2024-07-26T06:52:37.970 INFO:tasks.workunit.client.0.smithi042.stdout:checking whether gcc accepts -g... yes 2024-07-26T06:52:38.053 INFO:tasks.workunit.client.0.smithi042.stdout:checking for gcc option to accept ISO C89... none needed 2024-07-26T06:52:38.058 INFO:tasks.workunit.client.0.smithi042.stdout:checking for a sed that does not truncate output... /usr/bin/sed 2024-07-26T06:52:38.061 INFO:tasks.workunit.client.0.smithi042.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-07-26T06:52:38.063 INFO:tasks.workunit.client.0.smithi042.stdout:checking for egrep... /usr/bin/grep -E 2024-07-26T06:52:38.064 INFO:tasks.workunit.client.0.smithi042.stdout:checking for fgrep... /usr/bin/grep -F 2024-07-26T06:52:38.067 INFO:tasks.workunit.client.0.smithi042.stdout:checking for ld used by gcc... /usr/bin/ld 2024-07-26T06:52:38.069 INFO:tasks.workunit.client.0.smithi042.stdout:checking if the linker (/usr/bin/ld) is GNU ld... yes 2024-07-26T06:52:38.082 INFO:tasks.workunit.client.0.smithi042.stdout:checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B 2024-07-26T06:52:38.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:38 smithi042 ceph-mon[32775]: pgmap v468: 97 pgs: 97 active+clean; 33 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 515 op/s 2024-07-26T06:52:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:38 smithi164 ceph-mon[36970]: pgmap v468: 97 pgs: 97 active+clean; 33 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 515 op/s 2024-07-26T06:52:39.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:38 smithi163 ceph-mon[36873]: pgmap v468: 97 pgs: 97 active+clean; 33 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 515 op/s 2024-07-26T06:52:39.064 INFO:tasks.workunit.client.0.smithi042.stdout:checking the name lister (/usr/bin/nm -B) interface... BSD nm 2024-07-26T06:52:39.065 INFO:tasks.workunit.client.0.smithi042.stdout:checking whether ln -s works... yes 2024-07-26T06:52:39.073 INFO:tasks.workunit.client.0.smithi042.stdout:checking the maximum length of command line arguments... 1572864 2024-07-26T06:52:39.077 INFO:tasks.workunit.client.0.smithi042.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2024-07-26T06:52:39.077 INFO:tasks.workunit.client.0.smithi042.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2024-07-26T06:52:39.077 INFO:tasks.workunit.client.0.smithi042.stdout:checking for /usr/bin/ld option to reload object files... -r 2024-07-26T06:52:39.077 INFO:tasks.workunit.client.0.smithi042.stdout:checking for objdump... objdump 2024-07-26T06:52:39.077 INFO:tasks.workunit.client.0.smithi042.stdout:checking how to recognize dependent libraries... pass_all 2024-07-26T06:52:39.078 INFO:tasks.workunit.client.0.smithi042.stdout:checking for dlltool... no 2024-07-26T06:52:39.078 INFO:tasks.workunit.client.0.smithi042.stdout:checking how to associate runtime and link libraries... printf %s\n 2024-07-26T06:52:39.078 INFO:tasks.workunit.client.0.smithi042.stdout:checking for ar... ar 2024-07-26T06:52:39.166 INFO:tasks.workunit.client.0.smithi042.stdout:checking for archiver @FILE support... @ 2024-07-26T06:52:39.166 INFO:tasks.workunit.client.0.smithi042.stdout:checking for strip... strip 2024-07-26T06:52:39.167 INFO:tasks.workunit.client.0.smithi042.stdout:checking for ranlib... ranlib 2024-07-26T06:52:39.167 INFO:tasks.workunit.client.0.smithi042.stdout:checking for gawk... gawk 2024-07-26T06:52:39.276 INFO:tasks.workunit.client.0.smithi042.stdout:checking command to parse /usr/bin/nm -B output from gcc object... ok 2024-07-26T06:52:39.278 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sysroot... no 2024-07-26T06:52:39.290 INFO:tasks.workunit.client.0.smithi042.stdout:checking for a working dd... /usr/bin/dd 2024-07-26T06:52:39.301 INFO:tasks.workunit.client.0.smithi042.stdout:checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 2024-07-26T06:52:39.334 INFO:tasks.workunit.client.0.smithi042.stdout:checking for mt... no 2024-07-26T06:52:39.346 INFO:tasks.workunit.client.0.smithi042.stdout:checking if : is a manifest tool... no 2024-07-26T06:52:39.438 INFO:tasks.workunit.client.0.smithi042.stdout:checking how to run the C preprocessor... gcc -E 2024-07-26T06:52:39.625 INFO:tasks.workunit.client.0.smithi042.stdout:checking for ANSI C header files... yes 2024-07-26T06:52:39.659 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/types.h... yes 2024-07-26T06:52:39.697 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/stat.h... yes 2024-07-26T06:52:39.736 INFO:tasks.workunit.client.0.smithi042.stdout:checking for stdlib.h... yes 2024-07-26T06:52:39.779 INFO:tasks.workunit.client.0.smithi042.stdout:checking for string.h... yes 2024-07-26T06:52:39.819 INFO:tasks.workunit.client.0.smithi042.stdout:checking for memory.h... yes 2024-07-26T06:52:39.859 INFO:tasks.workunit.client.0.smithi042.stdout:checking for strings.h... yes 2024-07-26T06:52:39.901 INFO:tasks.workunit.client.0.smithi042.stdout:checking for inttypes.h... yes 2024-07-26T06:52:39.937 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.936+0000 7fc7ec9e8640 1 -- 172.21.15.42:0/3835111654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7e8102ef0 msgr2=0x7fc7e81033c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:39.938 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.936+0000 7fc7ec9e8640 1 --2- 172.21.15.42:0/3835111654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7e8102ef0 0x7fc7e81033c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0009900 tx=0x7fc7d002a410 comp rx=0 tx=0).stop 2024-07-26T06:52:39.938 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.937+0000 7fc7ec9e8640 1 -- 172.21.15.42:0/3835111654 shutdown_connections 2024-07-26T06:52:39.938 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.937+0000 7fc7ec9e8640 1 --2- 172.21.15.42:0/3835111654 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7e8103b00 0x7fc7e810a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.938 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.937+0000 7fc7ec9e8640 1 --2- 172.21.15.42:0/3835111654 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7e8102ef0 0x7fc7e81033c0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.938 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.937+0000 7fc7ec9e8640 1 --2- 172.21.15.42:0/3835111654 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7e8102510 0x7fc7e8102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.938 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.937+0000 7fc7ec9e8640 1 -- 172.21.15.42:0/3835111654 >> 172.21.15.42:0/3835111654 conn(0x7fc7e80fd0c0 msgr2=0x7fc7e80ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:39.938 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.937+0000 7fc7ec9e8640 1 -- 172.21.15.42:0/3835111654 shutdown_connections 2024-07-26T06:52:39.939 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.937+0000 7fc7ec9e8640 1 -- 172.21.15.42:0/3835111654 wait complete. 2024-07-26T06:52:39.941 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.937+0000 7fc7ec9e8640 1 Processor -- start 2024-07-26T06:52:39.941 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.938+0000 7fc7ec9e8640 1 -- start start 2024-07-26T06:52:39.941 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.938+0000 7fc7ec9e8640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7e8102510 0x7fc7e81988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.941 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.938+0000 7fc7ec9e8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7e8102ef0 0x7fc7e8198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.941 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.938+0000 7fc7ec9e8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7e8103b00 0x7fc7e8199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.941 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.938+0000 7fc7ec9e8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e810dc70 con 0x7fc7e8102ef0 2024-07-26T06:52:39.941 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.938+0000 7fc7ec9e8640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e810daf0 con 0x7fc7e8102510 2024-07-26T06:52:39.941 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.938+0000 7fc7ec9e8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e810ddf0 con 0x7fc7e8103b00 2024-07-26T06:52:39.942 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.938+0000 7fc7e5d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7e8102ef0 0x7fc7e8198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.942 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.938+0000 7fc7e5d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7e8102ef0 0x7fc7e8198f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48568/0 (socket says 172.21.15.42:48568) 2024-07-26T06:52:39.942 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.938+0000 7fc7e5d9b640 1 -- 172.21.15.42:0/1732282903 learned_addr learned my addr 172.21.15.42:0/1732282903 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:39.942 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.938+0000 7fc7e659c640 1 --2- 172.21.15.42:0/1732282903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7e8102510 0x7fc7e81988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.942 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.938+0000 7fc7e6d9d640 1 --2- 172.21.15.42:0/1732282903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7e8103b00 0x7fc7e8199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.942 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.938+0000 7fc7e5d9b640 1 -- 172.21.15.42:0/1732282903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7e8103b00 msgr2=0x7fc7e8199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:39.942 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.938+0000 7fc7e5d9b640 1 --2- 172.21.15.42:0/1732282903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7e8103b00 0x7fc7e8199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.942 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.938+0000 7fc7e5d9b640 1 -- 172.21.15.42:0/1732282903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7e8102510 msgr2=0x7fc7e81988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:39.943 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.938+0000 7fc7e5d9b640 1 --2- 172.21.15.42:0/1732282903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7e8102510 0x7fc7e81988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.943 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.938+0000 7fc7e5d9b640 1 -- 172.21.15.42:0/1732282903 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7e819a320 con 0x7fc7e8102ef0 2024-07-26T06:52:39.943 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.939+0000 7fc7e5d9b640 1 --2- 172.21.15.42:0/1732282903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7e8102ef0 0x7fc7e8198f50 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0006010 tx=0x7fc7d0031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:39.943 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.939+0000 7fc7df7fe640 1 -- 172.21.15.42:0/1732282903 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d003f030 con 0x7fc7e8102ef0 2024-07-26T06:52:39.943 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.939+0000 7fc7df7fe640 1 -- 172.21.15.42:0/1732282903 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7d0003aa0 con 0x7fc7e8102ef0 2024-07-26T06:52:39.943 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.939+0000 7fc7ec9e8640 1 -- 172.21.15.42:0/1732282903 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7e819a5b0 con 0x7fc7e8102ef0 2024-07-26T06:52:39.943 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.939+0000 7fc7df7fe640 1 -- 172.21.15.42:0/1732282903 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0035070 con 0x7fc7e8102ef0 2024-07-26T06:52:39.943 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.939+0000 7fc7ec9e8640 1 -- 172.21.15.42:0/1732282903 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7e819a8c0 con 0x7fc7e8102ef0 2024-07-26T06:52:39.945 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.941+0000 7fc7df7fe640 1 -- 172.21.15.42:0/1732282903 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fc7d0042030 con 0x7fc7e8102ef0 2024-07-26T06:52:39.945 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.942+0000 7fc7f39d3740 1 Processor -- start 2024-07-26T06:52:39.945 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.942+0000 7fc7f39d3740 1 -- start start 2024-07-26T06:52:39.945 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.942+0000 7fc7f39d3740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddeb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.945 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.942+0000 7fc7f39d3740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda1d2f0 0x55fddddeb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.945 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.942+0000 7fc7f39d3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda254d0 0x55fddddec340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.942+0000 7fc7f39d3740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55fdddd5e3b0 con 0x55fddda461b0 2024-07-26T06:52:39.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.942+0000 7fc7f39d3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fdddd4d8b0 con 0x55fddda254d0 2024-07-26T06:52:39.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.942+0000 7fc7f39d3740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55fdddd518f0 con 0x55fddda1d2f0 2024-07-26T06:52:39.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.942+0000 7fc7e6d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddeb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.942+0000 7fc7e6d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddeb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48580/0 (socket says 172.21.15.42:48580) 2024-07-26T06:52:39.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.942+0000 7fc7e6d9d640 1 -- 172.21.15.42:0/2353078947 learned_addr learned my addr 172.21.15.42:0/2353078947 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:39.947 INFO:tasks.workunit.client.0.smithi042.stdout:checking for stdint.h... yes 2024-07-26T06:52:39.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.943+0000 7fc7e5d9b640 1 --2- 172.21.15.42:0/2353078947 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda254d0 0x55fddddec340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.943+0000 7fc7e659c640 1 --2- 172.21.15.42:0/2353078947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda1d2f0 0x55fddddeb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.943+0000 7fc7df7fe640 1 --2- 172.21.15.42:0/1732282903 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc7d40b3bd0 0x7fc7d40b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.943+0000 7fc7e6d9d640 1 -- 172.21.15.42:0/2353078947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda1d2f0 msgr2=0x55fddddeb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:39.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.943+0000 7fc7e6d9d640 1 --2- 172.21.15.42:0/2353078947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda1d2f0 0x55fddddeb7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.943+0000 7fc7e6d9d640 1 -- 172.21.15.42:0/2353078947 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda254d0 msgr2=0x55fddddec340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:39.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.943+0000 7fc7e6d9d640 1 --2- 172.21.15.42:0/2353078947 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda254d0 0x55fddddec340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.943+0000 7fc7e6d9d640 1 -- 172.21.15.42:0/2353078947 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdddcf49a0 con 0x55fddda461b0 2024-07-26T06:52:39.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.943+0000 7fc7e5d9b640 1 --2- 172.21.15.42:0/2353078947 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda254d0 0x55fddddec340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:52:39.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.943+0000 7fc7e6d9d640 1 --2- 172.21.15.42:0/1732282903 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc7d40b3bd0 0x7fc7d40b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.943+0000 7fc7df7fe640 1 -- 172.21.15.42:0/1732282903 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fc7d01041e0 con 0x7fc7e8102ef0 2024-07-26T06:52:39.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.943+0000 7fc7e6d9d640 1 --2- 172.21.15.42:0/2353078947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddeb180 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0003a90 tx=0x7fc7e0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c20c9eca8cf667c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:39.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.943+0000 7fc7dcff9640 1 -- 172.21.15.42:0/2353078947 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e001b370 con 0x55fddda461b0 2024-07-26T06:52:39.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.944+0000 7fc7dcff9640 1 -- 172.21.15.42:0/2353078947 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7e001cd10 con 0x55fddda461b0 2024-07-26T06:52:39.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.944+0000 7fc7e6d9d640 1 --2- 172.21.15.42:0/1732282903 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc7d40b3bd0 0x7fc7d40b60e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0023000 tx=0x7fc7e0020000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:39.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.944+0000 7fc7dcff9640 1 -- 172.21.15.42:0/2353078947 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e001bc60 con 0x55fddda461b0 2024-07-26T06:52:39.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.944+0000 7fc7f39d3740 1 -- 172.21.15.42:0/2353078947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 msgr2=0x55fddddeb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:39.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.944+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/2353078947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddeb180 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0003a90 tx=0x7fc7e0009f90 comp rx=0 tx=0).stop 2024-07-26T06:52:39.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.945+0000 7fc7f39d3740 1 -- 172.21.15.42:0/2353078947 shutdown_connections 2024-07-26T06:52:39.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.945+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/2353078947 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda254d0 0x55fddddec340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.945+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/2353078947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda1d2f0 0x55fddddeb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.945+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/2353078947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddeb180 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.945+0000 7fc7f39d3740 1 -- 172.21.15.42:0/2353078947 >> 172.21.15.42:0/2353078947 conn(0x55fddda3a300 msgr2=0x55fdddde6c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:39.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.945+0000 7fc7f39d3740 1 -- 172.21.15.42:0/2353078947 shutdown_connections 2024-07-26T06:52:39.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.945+0000 7fc7f39d3740 1 -- 172.21.15.42:0/2353078947 wait complete. 2024-07-26T06:52:39.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.945+0000 7fc7f39d3740 1 -- start start 2024-07-26T06:52:39.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.945+0000 7fc7f39d3740 1 Processor -- start 2024-07-26T06:52:39.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.946+0000 7fc7f39d3740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda1d2f0 0x55fddddf4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.946+0000 7fc7f39d3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda254d0 0x55fddddf51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.946+0000 7fc7f39d3740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddf5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.946+0000 7fc7f39d3740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55fdddc87ab0 con 0x55fddda461b0 2024-07-26T06:52:39.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.946+0000 7fc7f39d3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fdddcd5cf0 con 0x55fddda254d0 2024-07-26T06:52:39.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.946+0000 7fc7f39d3740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55fddddece60 con 0x55fddda1d2f0 2024-07-26T06:52:39.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.946+0000 7fc7f39d3740 10 client.0 _ll_register_callbacks cb 0x55fdddd479f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:52:39.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.946+0000 7fc7e6d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddf5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.946+0000 7fc7e6d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddf5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48582/0 (socket says 172.21.15.42:48582) 2024-07-26T06:52:39.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.946+0000 7fc7e6d9d640 1 -- 172.21.15.42:0/1307919031 learned_addr learned my addr 172.21.15.42:0/1307919031 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:39.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.946+0000 7fc7e659c640 1 --2- 172.21.15.42:0/1307919031 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda1d2f0 0x55fddddf4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.946+0000 7fc7e5d9b640 1 --2- 172.21.15.42:0/1307919031 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda254d0 0x55fddddf51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.946+0000 7fc7e6d9d640 1 -- 172.21.15.42:0/1307919031 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda1d2f0 msgr2=0x55fddddf4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:39.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.946+0000 7fc7e6d9d640 1 --2- 172.21.15.42:0/1307919031 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda1d2f0 0x55fddddf4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.946+0000 7fc7e6d9d640 1 -- 172.21.15.42:0/1307919031 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda254d0 msgr2=0x55fddddf51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:39.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.946+0000 7fc7e6d9d640 1 --2- 172.21.15.42:0/1307919031 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda254d0 0x55fddddf51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.946+0000 7fc7e6d9d640 1 -- 172.21.15.42:0/1307919031 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdddab1790 con 0x55fddda461b0 2024-07-26T06:52:39.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.947+0000 7fc7e6d9d640 1 --2- 172.21.15.42:0/1307919031 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddf5d50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00096e0 tx=0x7fc7e001cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ba18385033b2e5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:39.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.947+0000 7fc7cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:52:39.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.947+0000 7fc7cdffb640 1 -- 172.21.15.42:0/1307919031 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e0030030 con 0x55fddda461b0 2024-07-26T06:52:39.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.947+0000 7fc7f39d3740 1 -- 172.21.15.42:0/1307919031 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7e001cd10 con 0x55fddda461b0 2024-07-26T06:52:39.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.947+0000 7fc7cdffb640 1 -- 172.21.15.42:0/1307919031 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fc7e001bc40 con 0x55fddda461b0 2024-07-26T06:52:39.959 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.947+0000 7fc7cdffb640 1 -- 172.21.15.42:0/1307919031 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc7e0010d30 con 0x55fddda461b0 2024-07-26T06:52:39.959 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.947+0000 7fc7f39d3740 10 client.15708 fetch_fsmap learned FSMap version 34 2024-07-26T06:52:39.959 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.947+0000 7fc7f39d3740 1 -- 172.21.15.42:0/1307919031 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55fdddb2a620 con 0x55fddda461b0 2024-07-26T06:52:39.959 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.947+0000 7fc7cdffb640 1 -- 172.21.15.42:0/1307919031 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7e001cd10 con 0x55fddda461b0 2024-07-26T06:52:39.959 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.947+0000 7fc7cdffb640 1 -- 172.21.15.42:0/1307919031 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7fc7e0004af0 con 0x55fddda461b0 2024-07-26T06:52:39.959 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.947+0000 7fc7f39d3740 10 client.15708 fetch_fsmap finished waiting for FSMap version 34 2024-07-26T06:52:39.959 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.947+0000 7fc7f39d3740 10 client.15708 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:52:39.959 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.948+0000 7fc7f39d3740 20 client.15708 populate_metadata read hostname 'smithi042' 2024-07-26T06:52:39.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.948+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/1307919031 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fddda04d80 0x55fddde02270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.948+0000 7fc7f39d3740 4 client.15708 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:52:39.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.948+0000 7fc7f39d3740 1 -- 172.21.15.42:0/1307919031 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdddd60300 con 0x55fddda04d80 2024-07-26T06:52:39.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.948+0000 7fc7e659c640 1 --2- 172.21.15.42:0/1307919031 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fddda04d80 0x55fddde02270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.948+0000 7fc7e659c640 1 --2- 172.21.15.42:0/1307919031 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fddda04d80 0x55fddde02270 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76391a0de4d8738f server_cookie=e37ad2ad9b536a20 in_seq=0 out_seq=0 2024-07-26T06:52:39.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.948+0000 7fc7cdffb640 10 client.15708 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:39.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.949+0000 7fc7cdffb640 1 -- 172.21.15.42:0/1307919031 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fdddd60300 con 0x55fddda04d80 2024-07-26T06:52:39.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.949+0000 7fc7cdffb640 10 client.15708 handle_command_reply: tid=0 2024-07-26T06:52:39.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.950+0000 7fc7f39d3740 1 client.15708 shutdown 2024-07-26T06:52:39.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.950+0000 7fc7f39d3740 1 -- 172.21.15.42:0/1307919031 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 msgr2=0x55fddddf5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:39.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.950+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/1307919031 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddf5d50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00096e0 tx=0x7fc7e001cf40 comp rx=0 tx=0).stop 2024-07-26T06:52:39.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.950+0000 7fc7f39d3740 1 -- 172.21.15.42:0/1307919031 shutdown_connections 2024-07-26T06:52:39.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.950+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/1307919031 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fddda04d80 0x55fddde02270 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.950+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/1307919031 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddf5d50 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.950+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/1307919031 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda254d0 0x55fddddf51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.950+0000 7fc7cdffb640 0 client.15708 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:39.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.950+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/1307919031 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda1d2f0 0x55fddddf4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.950+0000 7fc7f39d3740 1 -- 172.21.15.42:0/1307919031 >> 172.21.15.42:0/1307919031 conn(0x55fddda3a300 msgr2=0x55fdddde52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:39.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.950+0000 7fc7f39d3740 1 -- 172.21.15.42:0/1307919031 shutdown_connections 2024-07-26T06:52:39.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.950+0000 7fc7f39d3740 1 -- 172.21.15.42:0/1307919031 wait complete. 2024-07-26T06:52:39.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.950+0000 7fc7f39d3740 20 client.15708 trim_cache size 0 max 16384 2024-07-26T06:52:39.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.966+0000 7fc7f39d3740 1 Processor -- start 2024-07-26T06:52:39.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.966+0000 7fc7f39d3740 1 -- start start 2024-07-26T06:52:39.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.966+0000 7fc7f39d3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda1d2f0 0x55fddddec090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.966+0000 7fc7f39d3740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda254d0 0x55fddddf3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.966+0000 7fc7f39d3740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddefb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.966+0000 7fc7f39d3740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e81a88e0 con 0x55fddda461b0 2024-07-26T06:52:39.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.966+0000 7fc7f39d3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e81aa4e0 con 0x55fddda1d2f0 2024-07-26T06:52:39.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.966+0000 7fc7f39d3740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e81a8fb0 con 0x55fddda254d0 2024-07-26T06:52:39.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.966+0000 7fc7e6d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddefb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.966+0000 7fc7e6d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddefb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48584/0 (socket says 172.21.15.42:48584) 2024-07-26T06:52:39.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.966+0000 7fc7e6d9d640 1 -- 172.21.15.42:0/146383936 learned_addr learned my addr 172.21.15.42:0/146383936 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:39.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.966+0000 7fc7e659c640 1 --2- 172.21.15.42:0/146383936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda1d2f0 0x55fddddec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.966+0000 7fc7e5d9b640 1 --2- 172.21.15.42:0/146383936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda254d0 0x55fddddf3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.967+0000 7fc7e659c640 1 -- 172.21.15.42:0/146383936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda254d0 msgr2=0x55fddddf3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:39.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.967+0000 7fc7e659c640 1 --2- 172.21.15.42:0/146383936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda254d0 0x55fddddf3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.967+0000 7fc7e659c640 1 -- 172.21.15.42:0/146383936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 msgr2=0x55fddddefb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:39.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.967+0000 7fc7e659c640 1 --2- 172.21.15.42:0/146383936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddefb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.967+0000 7fc7e659c640 1 -- 172.21.15.42:0/146383936 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdddde0fe0 con 0x55fddda1d2f0 2024-07-26T06:52:39.971 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.967+0000 7fc7e6d9d640 1 --2- 172.21.15.42:0/146383936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddefb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:52:39.971 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.967+0000 7fc7e659c640 1 --2- 172.21.15.42:0/146383936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda1d2f0 0x55fddddec090 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x55fddde027d0 tx=0x7fc7d8016c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a96d9c2210b3fb7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:39.971 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.967+0000 7fc7cffff640 1 -- 172.21.15.42:0/146383936 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8026030 con 0x55fddda1d2f0 2024-07-26T06:52:39.971 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.967+0000 7fc7cffff640 1 -- 172.21.15.42:0/146383936 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7d8004310 con 0x55fddda1d2f0 2024-07-26T06:52:39.971 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.967+0000 7fc7cffff640 1 -- 172.21.15.42:0/146383936 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8014aa0 con 0x55fddda1d2f0 2024-07-26T06:52:39.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.968+0000 7fc7f39d3740 1 -- 172.21.15.42:0/146383936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda1d2f0 msgr2=0x55fddddec090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:39.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.968+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/146383936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda1d2f0 0x55fddddec090 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x55fddde027d0 tx=0x7fc7d8016c30 comp rx=0 tx=0).stop 2024-07-26T06:52:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.968+0000 7fc7f39d3740 1 -- 172.21.15.42:0/146383936 shutdown_connections 2024-07-26T06:52:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.968+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/146383936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda461b0 0x55fddddefb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.968+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/146383936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda254d0 0x55fddddf3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.968+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/146383936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda1d2f0 0x55fddddec090 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.968+0000 7fc7f39d3740 1 -- 172.21.15.42:0/146383936 >> 172.21.15.42:0/146383936 conn(0x55fddda3a300 msgr2=0x55fdddde19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.968+0000 7fc7f39d3740 1 -- 172.21.15.42:0/146383936 shutdown_connections 2024-07-26T06:52:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.968+0000 7fc7f39d3740 1 -- 172.21.15.42:0/146383936 wait complete. 2024-07-26T06:52:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.968+0000 7fc7f39d3740 1 -- start start 2024-07-26T06:52:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7f39d3740 1 Processor -- start 2024-07-26T06:52:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7f39d3740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda1d2f0 0x55fddddfaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7f39d3740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda254d0 0x55fddddfb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7f39d3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda461b0 0x55fddddfc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7f39d3740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e81a9d50 con 0x55fddda254d0 2024-07-26T06:52:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7f39d3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e81a9680 con 0x55fddda461b0 2024-07-26T06:52:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7f39d3740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e81a68c0 con 0x55fddda1d2f0 2024-07-26T06:52:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7f39d3740 10 client.0 _ll_register_callbacks cb 0x7fc7d0107b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:52:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7e5d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda254d0 0x55fddddfb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7e5d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda254d0 0x55fddddfb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48598/0 (socket says 172.21.15.42:48598) 2024-07-26T06:52:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7e5d9b640 1 -- 172.21.15.42:0/1270934933 learned_addr learned my addr 172.21.15.42:0/1270934933 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7e659c640 1 --2- 172.21.15.42:0/1270934933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda1d2f0 0x55fddddfaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7e6d9d640 1 --2- 172.21.15.42:0/1270934933 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda461b0 0x55fddddfc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7e5d9b640 1 -- 172.21.15.42:0/1270934933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda1d2f0 msgr2=0x55fddddfaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7e5d9b640 1 --2- 172.21.15.42:0/1270934933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda1d2f0 0x55fddddfaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7e5d9b640 1 -- 172.21.15.42:0/1270934933 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda461b0 msgr2=0x55fddddfc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7e5d9b640 1 --2- 172.21.15.42:0/1270934933 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda461b0 0x55fddddfc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7e5d9b640 1 -- 172.21.15.42:0/1270934933 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fddddfc960 con 0x55fddda254d0 2024-07-26T06:52:39.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7e659c640 1 --2- 172.21.15.42:0/1270934933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda1d2f0 0x55fddddfaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:52:39.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.969+0000 7fc7e6d9d640 1 --2- 172.21.15.42:0/1270934933 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda461b0 0x55fddddfc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:52:39.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.970+0000 7fc7e5d9b640 1 --2- 172.21.15.42:0/1270934933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda254d0 0x55fddddfb690 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc7d00cb150 tx=0x7fc7d0003e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffcd1b5c664e16c3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:39.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.970+0000 7fc7cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:52:39.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.970+0000 7fc7cdffb640 1 -- 172.21.15.42:0/1270934933 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0049040 con 0x55fddda254d0 2024-07-26T06:52:39.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.970+0000 7fc7cdffb640 1 -- 172.21.15.42:0/1270934933 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(80..80 src has 1..80) ==== 8269+0+0 (secure 0 0 0) 0x7fc7d00cb620 con 0x55fddda254d0 2024-07-26T06:52:39.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.970+0000 7fc7f39d3740 1 -- 172.21.15.42:0/1270934933 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7d8004310 con 0x55fddda254d0 2024-07-26T06:52:39.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.970+0000 7fc7cdffb640 1 -- 172.21.15.42:0/1270934933 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc7d00caf90 con 0x55fddda254d0 2024-07-26T06:52:39.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.970+0000 7fc7f39d3740 10 client.15714 fetch_fsmap learned FSMap version 34 2024-07-26T06:52:39.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.970+0000 7fc7f39d3740 1 -- 172.21.15.42:0/1270934933 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55fddde69550 con 0x55fddda254d0 2024-07-26T06:52:39.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.970+0000 7fc7cdffb640 1 -- 172.21.15.42:0/1270934933 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7d8004310 con 0x55fddda254d0 2024-07-26T06:52:39.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.970+0000 7fc7cdffb640 1 -- 172.21.15.42:0/1270934933 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 34) ==== 8382+0+0 (secure 0 0 0) 0x7fc7d00d3d30 con 0x55fddda254d0 2024-07-26T06:52:39.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.970+0000 7fc7f39d3740 10 client.15714 fetch_fsmap finished waiting for FSMap version 34 2024-07-26T06:52:39.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.970+0000 7fc7f39d3740 10 client.15714 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:52:39.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.970+0000 7fc7f39d3740 20 client.15714 populate_metadata read hostname 'smithi042' 2024-07-26T06:52:39.978 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.970+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/1270934933 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fddd9da1c0 0x55fddde6bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:39.978 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.970+0000 7fc7f39d3740 4 client.15714 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:52:39.978 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.970+0000 7fc7f39d3740 1 -- 172.21.15.42:0/1270934933 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fdddd5bda0 con 0x55fddd9da1c0 2024-07-26T06:52:39.978 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.971+0000 7fc7e659c640 1 --2- 172.21.15.42:0/1270934933 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fddd9da1c0 0x55fddde6bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:39.978 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.971+0000 7fc7e659c640 1 --2- 172.21.15.42:0/1270934933 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fddd9da1c0 0x55fddde6bf80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ca30085d06369d2 server_cookie=c2846883fbc51a2c in_seq=0 out_seq=0 2024-07-26T06:52:39.978 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:39.971+0000 7fc7cdffb640 10 client.15714 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:40.218 INFO:tasks.workunit.client.0.smithi042.stdout:checking for unistd.h... yes 2024-07-26T06:52:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.226+0000 7fc7cdffb640 1 -- 172.21.15.42:0/1270934933 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fdddd5bda0 con 0x55fddd9da1c0 2024-07-26T06:52:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.226+0000 7fc7cdffb640 10 client.15714 handle_command_reply: tid=0 2024-07-26T06:52:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.226+0000 7fc7f39d3740 1 client.15714 shutdown 2024-07-26T06:52:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.227+0000 7fc7f39d3740 1 -- 172.21.15.42:0/1270934933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda254d0 msgr2=0x55fddddfb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.227+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/1270934933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda254d0 0x55fddddfb690 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc7d00cb150 tx=0x7fc7d0003e80 comp rx=0 tx=0).stop 2024-07-26T06:52:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.227+0000 7fc7f39d3740 1 -- 172.21.15.42:0/1270934933 shutdown_connections 2024-07-26T06:52:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.227+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/1270934933 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fddd9da1c0 0x55fddde6bf80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.227+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/1270934933 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fddda461b0 0x55fddddfc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.227+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/1270934933 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fddda254d0 0x55fddddfb690 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.227+0000 7fc7cdffb640 0 client.15714 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.227+0000 7fc7f39d3740 1 --2- 172.21.15.42:0/1270934933 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fddda1d2f0 0x55fddddfaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.227+0000 7fc7f39d3740 1 -- 172.21.15.42:0/1270934933 >> 172.21.15.42:0/1270934933 conn(0x55fddda3a300 msgr2=0x55fdddde19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.227+0000 7fc7f39d3740 1 -- 172.21.15.42:0/1270934933 shutdown_connections 2024-07-26T06:52:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.227+0000 7fc7f39d3740 1 -- 172.21.15.42:0/1270934933 wait complete. 2024-07-26T06:52:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.227+0000 7fc7f39d3740 20 client.15714 trim_cache size 0 max 16384 2024-07-26T06:52:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:52:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:52:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:52:40.232 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:52:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.228+0000 7fc7ec9e8640 1 -- 172.21.15.42:0/1732282903 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc7d40b3bd0 msgr2=0x7fc7d40b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.228+0000 7fc7ec9e8640 1 --2- 172.21.15.42:0/1732282903 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc7d40b3bd0 0x7fc7d40b60e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0023000 tx=0x7fc7e0020000 comp rx=0 tx=0).stop 2024-07-26T06:52:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.228+0000 7fc7ec9e8640 1 -- 172.21.15.42:0/1732282903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7e8102ef0 msgr2=0x7fc7e8198f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.228+0000 7fc7ec9e8640 1 --2- 172.21.15.42:0/1732282903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7e8102ef0 0x7fc7e8198f50 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0006010 tx=0x7fc7d0031a50 comp rx=0 tx=0).stop 2024-07-26T06:52:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.228+0000 7fc7ec9e8640 1 -- 172.21.15.42:0/1732282903 shutdown_connections 2024-07-26T06:52:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.228+0000 7fc7ec9e8640 1 --2- 172.21.15.42:0/1732282903 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc7d40b3bd0 0x7fc7d40b60e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.228+0000 7fc7ec9e8640 1 --2- 172.21.15.42:0/1732282903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7e8103b00 0x7fc7e8199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.228+0000 7fc7ec9e8640 1 --2- 172.21.15.42:0/1732282903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7e8102ef0 0x7fc7e8198f50 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.228+0000 7fc7ec9e8640 1 --2- 172.21.15.42:0/1732282903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc7e8102510 0x7fc7e81988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.228+0000 7fc7ec9e8640 1 -- 172.21.15.42:0/1732282903 >> 172.21.15.42:0/1732282903 conn(0x7fc7e80fd0c0 msgr2=0x7fc7e8106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.228+0000 7fc7ec9e8640 1 -- 172.21.15.42:0/1732282903 shutdown_connections 2024-07-26T06:52:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:40.228+0000 7fc7ec9e8640 1 -- 172.21.15.42:0/1732282903 wait complete. 2024-07-26T06:52:40.276 INFO:tasks.workunit.client.0.smithi042.stdout:checking for dlfcn.h... yes 2024-07-26T06:52:40.282 INFO:tasks.workunit.client.0.smithi042.stdout:checking for objdir... .libs 2024-07-26T06:52:40.475 INFO:tasks.workunit.client.0.smithi042.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2024-07-26T06:52:40.475 INFO:tasks.workunit.client.0.smithi042.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2024-07-26T06:52:40.528 INFO:tasks.workunit.client.0.smithi042.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2024-07-26T06:52:40.711 INFO:tasks.workunit.client.0.smithi042.stdout:checking if gcc static flag -static works... no 2024-07-26T06:52:40.774 INFO:tasks.workunit.client.0.smithi042.stdout:checking if gcc supports -c -o file.o... yes 2024-07-26T06:52:40.774 INFO:tasks.workunit.client.0.smithi042.stdout:checking if gcc supports -c -o file.o... (cached) yes 2024-07-26T06:52:40.780 INFO:tasks.workunit.client.0.smithi042.stdout:checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes 2024-07-26T06:52:40.851 INFO:tasks.workunit.client.0.smithi042.stdout:checking whether -lc should be explicitly linked in... no 2024-07-26T06:52:40.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:40 smithi042 ceph-mon[32775]: pgmap v469: 97 pgs: 97 active+clean; 37 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 521 op/s 2024-07-26T06:52:40.912 INFO:tasks.workunit.client.0.smithi042.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2024-07-26T06:52:40.912 INFO:tasks.workunit.client.0.smithi042.stdout:checking how to hardcode library paths into programs... immediate 2024-07-26T06:52:40.914 INFO:tasks.workunit.client.0.smithi042.stdout:checking whether stripping libraries is possible... yes 2024-07-26T06:52:40.914 INFO:tasks.workunit.client.0.smithi042.stdout:checking if libtool supports shared libraries... yes 2024-07-26T06:52:40.914 INFO:tasks.workunit.client.0.smithi042.stdout:checking whether to build shared libraries... yes 2024-07-26T06:52:40.914 INFO:tasks.workunit.client.0.smithi042.stdout:checking whether to build static libraries... yes 2024-07-26T06:52:40.921 INFO:tasks.workunit.client.0.smithi042.stdout:checking for gcc... (cached) gcc 2024-07-26T06:52:40.957 INFO:tasks.workunit.client.0.smithi042.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-07-26T06:52:40.957 INFO:tasks.workunit.client.0.smithi042.stdout:checking whether gcc accepts -g... (cached) yes 2024-07-26T06:52:40.957 INFO:tasks.workunit.client.0.smithi042.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-07-26T06:52:40.958 INFO:tasks.workunit.client.0.smithi042.stdout:checking for gmake... /usr/bin/gmake 2024-07-26T06:52:40.958 INFO:tasks.workunit.client.0.smithi042.stderr:./configure: line 12082: test: $(SHELL): binary operator expected 2024-07-26T06:52:40.959 INFO:tasks.workunit.client.0.smithi042.stdout:checking for tar... /bin/tar 2024-07-26T06:52:40.959 INFO:tasks.workunit.client.0.smithi042.stdout:checking for gzip... /bin/gzip 2024-07-26T06:52:40.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:40 smithi164 ceph-mon[36970]: pgmap v469: 97 pgs: 97 active+clean; 37 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 521 op/s 2024-07-26T06:52:40.972 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:52:40 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-f[66559]: ignoring --setuser ceph since I am not root 2024-07-26T06:52:40.972 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:52:40 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-f[66559]: ignoring --setgroup ceph since I am not root 2024-07-26T06:52:40.972 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:52:40 smithi164 ceph-mds[66563]: -- 172.21.15.164:0/1242014288 <== mon.1 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563dda1a5860 con 0x563dde347600 2024-07-26T06:52:40.972 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:52:40 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-f[66559]: starting mds.f at 2024-07-26T06:52:41.001 INFO:tasks.workunit.client.0.smithi042.stdout:checking whether gcc -MM is supported... yes 2024-07-26T06:52:41.001 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sort... /bin/sort 2024-07-26T06:52:41.001 INFO:tasks.workunit.client.0.smithi042.stdout:checking whether ln -s works... yes 2024-07-26T06:52:41.002 INFO:tasks.workunit.client.0.smithi042.stdout:checking for rpm... /bin/rpm 2024-07-26T06:52:41.004 INFO:tasks.workunit.client.0.smithi042.stdout:checking for rpmbuild... no 2024-07-26T06:52:41.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:40 smithi163 ceph-mon[36873]: pgmap v469: 97 pgs: 97 active+clean; 37 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 521 op/s 2024-07-26T06:52:41.049 INFO:tasks.workunit.client.0.smithi042.stdout:checking assert.h usability... yes 2024-07-26T06:52:41.065 INFO:tasks.workunit.client.0.smithi042.stdout:checking assert.h presence... yes 2024-07-26T06:52:41.065 INFO:tasks.workunit.client.0.smithi042.stdout:checking for assert.h... yes 2024-07-26T06:52:41.120 INFO:tasks.workunit.client.0.smithi042.stdout:checking bstring.h usability... no 2024-07-26T06:52:41.150 INFO:tasks.workunit.client.0.smithi042.stdout:checking bstring.h presence... no 2024-07-26T06:52:41.150 INFO:tasks.workunit.client.0.smithi042.stdout:checking for bstring.h... no 2024-07-26T06:52:41.202 INFO:tasks.workunit.client.0.smithi042.stdout:checking libgen.h usability... yes 2024-07-26T06:52:41.219 INFO:tasks.workunit.client.0.smithi042.stdout:checking libgen.h presence... yes 2024-07-26T06:52:41.219 INFO:tasks.workunit.client.0.smithi042.stdout:checking for libgen.h... yes 2024-07-26T06:52:41.277 INFO:tasks.workunit.client.0.smithi042.stdout:checking dirent.h usability... yes 2024-07-26T06:52:41.294 INFO:tasks.workunit.client.0.smithi042.stdout:checking dirent.h presence... yes 2024-07-26T06:52:41.295 INFO:tasks.workunit.client.0.smithi042.stdout:checking for dirent.h... yes 2024-07-26T06:52:41.341 INFO:tasks.workunit.client.0.smithi042.stdout:checking errno.h usability... yes 2024-07-26T06:52:41.358 INFO:tasks.workunit.client.0.smithi042.stdout:checking errno.h presence... yes 2024-07-26T06:52:41.358 INFO:tasks.workunit.client.0.smithi042.stdout:checking for errno.h... yes 2024-07-26T06:52:41.405 INFO:tasks.workunit.client.0.smithi042.stdout:checking malloc.h usability... yes 2024-07-26T06:52:41.424 INFO:tasks.workunit.client.0.smithi042.stdout:checking malloc.h presence... yes 2024-07-26T06:52:41.424 INFO:tasks.workunit.client.0.smithi042.stdout:checking for malloc.h... yes 2024-07-26T06:52:41.480 INFO:tasks.workunit.client.0.smithi042.stdout:checking uuid.h usability... no 2024-07-26T06:52:41.514 INFO:tasks.workunit.client.0.smithi042.stdout:checking uuid.h presence... no 2024-07-26T06:52:41.514 INFO:tasks.workunit.client.0.smithi042.stdout:checking for uuid.h... no 2024-07-26T06:52:41.565 INFO:tasks.workunit.client.0.smithi042.stdout:checking uuid/uuid.h usability... yes 2024-07-26T06:52:41.584 INFO:tasks.workunit.client.0.smithi042.stdout:checking uuid/uuid.h presence... yes 2024-07-26T06:52:41.584 INFO:tasks.workunit.client.0.smithi042.stdout:checking for uuid/uuid.h... yes 2024-07-26T06:52:41.640 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/uuid.h usability... no 2024-07-26T06:52:41.668 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/uuid.h presence... no 2024-07-26T06:52:41.668 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/uuid.h... no 2024-07-26T06:52:41.713 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/file.h usability... yes 2024-07-26T06:52:41.729 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/file.h presence... yes 2024-07-26T06:52:41.730 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/file.h... yes 2024-07-26T06:52:41.776 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/fcntl.h usability... yes 2024-07-26T06:52:41.793 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/fcntl.h presence... yes 2024-07-26T06:52:41.793 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/fcntl.h... yes 2024-07-26T06:52:41.844 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/syssgi.h usability... no 2024-07-26T06:52:41.873 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/syssgi.h presence... no 2024-07-26T06:52:41.873 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/syssgi.h... no 2024-07-26T06:52:41.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:41 smithi042 ceph-mon[32775]: daemon mds.f restarted 2024-07-26T06:52:41.940 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/param.h usability... yes 2024-07-26T06:52:41.960 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/param.h presence... yes 2024-07-26T06:52:41.960 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/param.h... yes 2024-07-26T06:52:41.971 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:41 smithi164 ceph-mon[36970]: daemon mds.f restarted 2024-07-26T06:52:41.975 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/stat.h... (cached) yes 2024-07-26T06:52:42.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:41 smithi163 ceph-mon[36873]: daemon mds.f restarted 2024-07-26T06:52:42.012 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/statvfs.h usability... yes 2024-07-26T06:52:42.029 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/statvfs.h presence... yes 2024-07-26T06:52:42.029 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/statvfs.h... yes 2024-07-26T06:52:42.076 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/time.h usability... yes 2024-07-26T06:52:42.093 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/time.h presence... yes 2024-07-26T06:52:42.093 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/time.h... yes 2024-07-26T06:52:42.143 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/ioctl.h usability... yes 2024-07-26T06:52:42.156 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/ioctl.h presence... yes 2024-07-26T06:52:42.156 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/ioctl.h... yes 2024-07-26T06:52:42.212 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/wait.h usability... yes 2024-07-26T06:52:42.231 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/wait.h presence... yes 2024-07-26T06:52:42.232 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/wait.h... yes 2024-07-26T06:52:42.247 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/types.h... (cached) yes 2024-07-26T06:52:42.251 INFO:tasks.workunit.client.0.smithi042.stdout:checking for strings.h... (cached) yes 2024-07-26T06:52:42.288 INFO:tasks.workunit.client.0.smithi042.stdout:checking err.h usability... yes 2024-07-26T06:52:42.306 INFO:tasks.workunit.client.0.smithi042.stdout:checking err.h presence... yes 2024-07-26T06:52:42.306 INFO:tasks.workunit.client.0.smithi042.stdout:checking for err.h... yes 2024-07-26T06:52:42.353 INFO:tasks.workunit.client.0.smithi042.stdout:checking linux/falloc.h usability... yes 2024-07-26T06:52:42.369 INFO:tasks.workunit.client.0.smithi042.stdout:checking linux/falloc.h presence... yes 2024-07-26T06:52:42.369 INFO:tasks.workunit.client.0.smithi042.stdout:checking for linux/falloc.h... yes 2024-07-26T06:52:42.420 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/fs/xfs_fsops.h usability... no 2024-07-26T06:52:42.449 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/fs/xfs_fsops.h presence... no 2024-07-26T06:52:42.449 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/fs/xfs_fsops.h... no 2024-07-26T06:52:42.493 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/fs/xfs_itable.h usability... no 2024-07-26T06:52:42.522 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/fs/xfs_itable.h presence... no 2024-07-26T06:52:42.522 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/fs/xfs_itable.h... no 2024-07-26T06:52:42.570 INFO:tasks.workunit.client.0.smithi042.stdout:checking xfs/platform_defs.h usability... no 2024-07-26T06:52:42.604 INFO:tasks.workunit.client.0.smithi042.stdout:checking xfs/platform_defs.h presence... no 2024-07-26T06:52:42.605 INFO:tasks.workunit.client.0.smithi042.stdout:checking for xfs/platform_defs.h... no 2024-07-26T06:52:42.654 INFO:tasks.workunit.client.0.smithi042.stdout:checking btrfs/ioctl.h usability... no 2024-07-26T06:52:42.687 INFO:tasks.workunit.client.0.smithi042.stdout:checking btrfs/ioctl.h presence... no 2024-07-26T06:52:42.687 INFO:tasks.workunit.client.0.smithi042.stdout:checking for btrfs/ioctl.h... no 2024-07-26T06:52:42.736 INFO:tasks.workunit.client.0.smithi042.stdout:checking cifs/ioctl.h usability... no 2024-07-26T06:52:42.771 INFO:tasks.workunit.client.0.smithi042.stdout:checking cifs/ioctl.h presence... no 2024-07-26T06:52:42.772 INFO:tasks.workunit.client.0.smithi042.stdout:checking for cifs/ioctl.h... no 2024-07-26T06:52:42.830 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/mman.h usability... yes 2024-07-26T06:52:42.850 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/mman.h presence... yes 2024-07-26T06:52:42.850 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/mman.h... yes 2024-07-26T06:52:42.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:42 smithi042 ceph-mon[32775]: pgmap v470: 97 pgs: 97 active+clean; 37 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 264 op/s 2024-07-26T06:52:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:42 smithi042 ceph-mon[32775]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T06:52:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:42 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:boot 2024-07-26T06:52:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:42 smithi042 ceph-mon[32775]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T06:52:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T06:52:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:42 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:42.906 INFO:tasks.workunit.client.0.smithi042.stdout:checking linux/ext4.h usability... no 2024-07-26T06:52:42.941 INFO:tasks.workunit.client.0.smithi042.stdout:checking linux/ext4.h presence... no 2024-07-26T06:52:42.941 INFO:tasks.workunit.client.0.smithi042.stdout:checking for linux/ext4.h... no 2024-07-26T06:52:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:42 smithi164 ceph-mon[36970]: pgmap v470: 97 pgs: 97 active+clean; 37 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 264 op/s 2024-07-26T06:52:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:42 smithi164 ceph-mon[36970]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T06:52:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:42 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:boot 2024-07-26T06:52:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:42 smithi164 ceph-mon[36970]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T06:52:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T06:52:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:42 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:42.974 INFO:tasks.workunit.client.0.smithi042.stdout:checking for xfs/xfs_log_format.h... no 2024-07-26T06:52:43.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:42 smithi163 ceph-mon[36873]: pgmap v470: 97 pgs: 97 active+clean; 37 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 264 op/s 2024-07-26T06:52:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:42 smithi163 ceph-mon[36873]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T06:52:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:42 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:boot 2024-07-26T06:52:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:42 smithi163 ceph-mon[36873]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T06:52:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T06:52:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:42 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:52:43.035 INFO:tasks.workunit.client.0.smithi042.stdout:checking for uuid_compare... no 2024-07-26T06:52:43.079 INFO:tasks.workunit.client.0.smithi042.stdout:checking for uuid_compare in -luuid... yes 2024-07-26T06:52:43.149 INFO:tasks.workunit.client.0.smithi042.stdout:checking for xfs/xfs.h... yes 2024-07-26T06:52:43.188 INFO:tasks.workunit.client.0.smithi042.stdout:checking for xfs/libxfs.h... no 2024-07-26T06:52:43.225 INFO:tasks.workunit.client.0.smithi042.stdout:checking whether xlog_assign_lsn is declared... no 2024-07-26T06:52:43.269 INFO:tasks.workunit.client.0.smithi042.stdout:checking for xfs/xqm.h... yes 2024-07-26T06:52:43.325 INFO:tasks.workunit.client.0.smithi042.stdout:checking xfsctl from xfs/xfs.h... ok 2024-07-26T06:52:43.366 INFO:tasks.workunit.client.0.smithi042.stdout:checking xfs/handle.h usability... yes 2024-07-26T06:52:43.382 INFO:tasks.workunit.client.0.smithi042.stdout:checking xfs/handle.h presence... yes 2024-07-26T06:52:43.382 INFO:tasks.workunit.client.0.smithi042.stdout:checking for xfs/handle.h... yes 2024-07-26T06:52:44.282 INFO:tasks.workunit.client.0.smithi042.stdout:checking for attr_list_by_handle in -lhandle... yes 2024-07-26T06:52:44.331 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/acl.h usability... yes 2024-07-26T06:52:44.355 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/acl.h presence... yes 2024-07-26T06:52:44.355 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/acl.h... yes 2024-07-26T06:52:44.402 INFO:tasks.workunit.client.0.smithi042.stdout:checking acl/libacl.h usability... yes 2024-07-26T06:52:44.421 INFO:tasks.workunit.client.0.smithi042.stdout:checking acl/libacl.h presence... yes 2024-07-26T06:52:44.421 INFO:tasks.workunit.client.0.smithi042.stdout:checking for acl/libacl.h... yes 2024-07-26T06:52:44.463 INFO:tasks.workunit.client.0.smithi042.stdout:checking for acl_init in -lacl... yes 2024-07-26T06:52:44.557 INFO:tasks.workunit.client.0.smithi042.stdout:checking gdbm-ndbm.h usability... no 2024-07-26T06:52:44.585 INFO:tasks.workunit.client.0.smithi042.stdout:checking gdbm-ndbm.h presence... no 2024-07-26T06:52:44.585 INFO:tasks.workunit.client.0.smithi042.stdout:checking for gdbm-ndbm.h... no 2024-07-26T06:52:44.624 INFO:tasks.workunit.client.0.smithi042.stdout:checking gdbm/ndbm.h usability... no 2024-07-26T06:52:44.653 INFO:tasks.workunit.client.0.smithi042.stdout:checking gdbm/ndbm.h presence... no 2024-07-26T06:52:44.653 INFO:tasks.workunit.client.0.smithi042.stdout:checking for gdbm/ndbm.h... no 2024-07-26T06:52:44.691 INFO:tasks.workunit.client.0.smithi042.stdout:checking ndbm.h usability... no 2024-07-26T06:52:44.720 INFO:tasks.workunit.client.0.smithi042.stdout:checking ndbm.h presence... no 2024-07-26T06:52:44.720 INFO:tasks.workunit.client.0.smithi042.stdout:checking for ndbm.h... no 2024-07-26T06:52:44.752 INFO:tasks.workunit.client.0.smithi042.stdout:checking libaio.h usability... yes 2024-07-26T06:52:44.772 INFO:tasks.workunit.client.0.smithi042.stdout:checking libaio.h presence... yes 2024-07-26T06:52:44.772 INFO:tasks.workunit.client.0.smithi042.stdout:checking for libaio.h... yes 2024-07-26T06:52:44.817 INFO:tasks.workunit.client.0.smithi042.stdout:checking liburing.h usability... no 2024-07-26T06:52:44.845 INFO:tasks.workunit.client.0.smithi042.stdout:checking liburing.h presence... no 2024-07-26T06:52:44.845 INFO:tasks.workunit.client.0.smithi042.stdout:checking for liburing.h... no 2024-07-26T06:52:44.884 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/capability.h usability... no 2024-07-26T06:52:44.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:44 smithi042 ceph-mon[32775]: pgmap v472: 97 pgs: 97 active+clean; 33 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 279 op/s 2024-07-26T06:52:44.912 INFO:tasks.workunit.client.0.smithi042.stdout:checking sys/capability.h presence... no 2024-07-26T06:52:44.912 INFO:tasks.workunit.client.0.smithi042.stdout:checking for sys/capability.h... no 2024-07-26T06:52:44.944 INFO:tasks.workunit.client.0.smithi042.stdout:checking linux/fiemap.h usability... yes 2024-07-26T06:52:44.960 INFO:tasks.workunit.client.0.smithi042.stdout:checking linux/fiemap.h presence... yes 2024-07-26T06:52:44.960 INFO:tasks.workunit.client.0.smithi042.stdout:checking for linux/fiemap.h... yes 2024-07-26T06:52:44.971 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:44 smithi164 ceph-mon[36970]: pgmap v472: 97 pgs: 97 active+clean; 33 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 279 op/s 2024-07-26T06:52:44.998 INFO:tasks.workunit.client.0.smithi042.stdout:checking for fallocate... yes 2024-07-26T06:52:45.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:44 smithi163 ceph-mon[36873]: pgmap v472: 97 pgs: 97 active+clean; 33 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 279 op/s 2024-07-26T06:52:45.051 INFO:tasks.workunit.client.0.smithi042.stdout:checking for open_by_handle_at... yes 2024-07-26T06:52:45.089 INFO:tasks.workunit.client.0.smithi042.stdout:checking linux/fs.h usability... yes 2024-07-26T06:52:45.105 INFO:tasks.workunit.client.0.smithi042.stdout:checking linux/fs.h presence... yes 2024-07-26T06:52:45.106 INFO:tasks.workunit.client.0.smithi042.stdout:checking for linux/fs.h... yes 2024-07-26T06:52:45.162 INFO:tasks.workunit.client.0.smithi042.stdout:checking btrfsutil.h usability... no 2024-07-26T06:52:45.189 INFO:tasks.workunit.client.0.smithi042.stdout:checking btrfsutil.h presence... no 2024-07-26T06:52:45.190 INFO:tasks.workunit.client.0.smithi042.stdout:checking for btrfsutil.h... no 2024-07-26T06:52:45.237 INFO:tasks.workunit.client.0.smithi042.stdout:checking for copy_file_range... yes 2024-07-26T06:52:45.297 INFO:tasks.workunit.client.0.smithi042.stdout:checking for SEEK_DATA... yes 2024-07-26T06:52:45.356 INFO:tasks.workunit.client.0.smithi042.stdout:checking for BMV_OF_SHARED... yes 2024-07-26T06:52:45.401 INFO:tasks.workunit.client.0.smithi042.stdout:checking for nftw... yes 2024-07-26T06:52:45.454 INFO:tasks.workunit.client.0.smithi042.stdout:checking for RLIMIT_NOFILE... yes 2024-07-26T06:52:45.500 INFO:tasks.workunit.client.0.smithi042.stdout:checking for XFS_IOC_EXCHANGE_RANGE... no 2024-07-26T06:52:45.549 INFO:tasks.workunit.client.0.smithi042.stdout:checking for FICLONE... yes 2024-07-26T06:52:45.597 INFO:tasks.workunit.client.0.smithi042.stdout:checking for renameat2... yes 2024-07-26T06:52:45.640 INFO:tasks.workunit.client.0.smithi042.stdout:checking for reallocarray... yes 2024-07-26T06:52:45.705 INFO:tasks.workunit.client.0.smithi042.stdout:checking for struct mount_attr... yes 2024-07-26T06:52:45.768 INFO:tasks.workunit.client.0.smithi042.stdout:checking for struct btrfs_qgroup_limit... yes 2024-07-26T06:52:45.825 INFO:tasks.workunit.client.0.smithi042.stdout:checking for struct btrfs_qgroup_inherit... yes 2024-07-26T06:52:45.883 INFO:tasks.workunit.client.0.smithi042.stdout:checking for struct btrfs_ioctl_vol_args... yes 2024-07-26T06:52:45.941 INFO:tasks.workunit.client.0.smithi042.stdout:checking for struct btrfs_ioctl_vol_args_v2... yes 2024-07-26T06:52:45.997 INFO:tasks.workunit.client.0.smithi042.stdout:checking for struct btrfs_ioctl_ino_lookup_args... yes 2024-07-26T06:52:46.055 INFO:tasks.workunit.client.0.smithi042.stdout:checking for struct btrfs_ioctl_ino_lookup_user_args... yes 2024-07-26T06:52:46.112 INFO:tasks.workunit.client.0.smithi042.stdout:checking for struct btrfs_ioctl_get_subvol_rootref_args... yes 2024-07-26T06:52:46.152 INFO:tasks.workunit.client.0.smithi042.stdout:checking linux/btrfs.h usability... yes 2024-07-26T06:52:46.169 INFO:tasks.workunit.client.0.smithi042.stdout:checking linux/btrfs.h presence... yes 2024-07-26T06:52:46.170 INFO:tasks.workunit.client.0.smithi042.stdout:checking for linux/btrfs.h... yes 2024-07-26T06:52:46.217 INFO:tasks.workunit.client.0.smithi042.stdout:checking linux/btrfs_tree.h usability... yes 2024-07-26T06:52:46.238 INFO:tasks.workunit.client.0.smithi042.stdout:checking linux/btrfs_tree.h presence... yes 2024-07-26T06:52:46.238 INFO:tasks.workunit.client.0.smithi042.stdout:checking for linux/btrfs_tree.h... yes 2024-07-26T06:52:46.275 INFO:tasks.workunit.client.0.smithi042.stdout:checking for struct btrfs_ioctl_vol_args_v2.subvolid... yes 2024-07-26T06:52:46.306 INFO:tasks.workunit.client.0.smithi042.stdout:checking whether BTRFS_IOC_SNAP_DESTROY_V2 is declared... yes 2024-07-26T06:52:46.326 INFO:tasks.workunit.client.0.smithi042.stdout:configure: creating ./config.status 2024-07-26T06:52:46.676 INFO:tasks.workunit.client.0.smithi042.stdout:config.status: creating include/builddefs 2024-07-26T06:52:46.690 INFO:tasks.workunit.client.0.smithi042.stdout:config.status: creating include/config.h 2024-07-26T06:52:46.703 INFO:tasks.workunit.client.0.smithi042.stdout:config.status: executing libtool commands 2024-07-26T06:52:46.783 INFO:tasks.workunit.client.0.smithi042.stdout:Building include 2024-07-26T06:52:46.783 INFO:tasks.workunit.client.0.smithi042.stdout:Building lib 2024-07-26T06:52:46.784 INFO:tasks.workunit.client.0.smithi042.stdout:Building tests 2024-07-26T06:52:46.788 INFO:tasks.workunit.client.0.smithi042.stdout: [LTDEP] 2024-07-26T06:52:46.804 INFO:tasks.workunit.client.0.smithi042.stdout:Building btrfs 2024-07-26T06:52:46.805 INFO:tasks.workunit.client.0.smithi042.stdout:Building ceph 2024-07-26T06:52:46.805 INFO:tasks.workunit.client.0.smithi042.stdout:Building cifs 2024-07-26T06:52:46.812 INFO:tasks.workunit.client.0.smithi042.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/ceph/group.list 2024-07-26T06:52:46.813 INFO:tasks.workunit.client.0.smithi042.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/cifs/group.list 2024-07-26T06:52:46.830 INFO:tasks.workunit.client.0.smithi042.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/btrfs/group.list 2024-07-26T06:52:46.831 INFO:tasks.workunit.client.0.smithi042.stdout:Building ext4 2024-07-26T06:52:46.833 INFO:tasks.workunit.client.0.smithi042.stdout:Building f2fs 2024-07-26T06:52:46.840 INFO:tasks.workunit.client.0.smithi042.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/f2fs/group.list 2024-07-26T06:52:46.846 INFO:tasks.workunit.client.0.smithi042.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/ext4/group.list 2024-07-26T06:52:46.864 INFO:tasks.workunit.client.0.smithi042.stdout:Building generic 2024-07-26T06:52:46.894 INFO:tasks.workunit.client.0.smithi042.stdout:Building nfs 2024-07-26T06:52:46.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:46 smithi042 ceph-mon[32775]: pgmap v473: 97 pgs: 97 active+clean; 27 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 539 op/s 2024-07-26T06:52:46.900 INFO:tasks.workunit.client.0.smithi042.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/nfs/group.list 2024-07-26T06:52:46.918 INFO:tasks.workunit.client.0.smithi042.stdout:Building ocfs2 2024-07-26T06:52:46.925 INFO:tasks.workunit.client.0.smithi042.stdout:Building overlay 2024-07-26T06:52:46.926 INFO:tasks.workunit.client.0.smithi042.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/ocfs2/group.list 2024-07-26T06:52:46.943 INFO:tasks.workunit.client.0.smithi042.stdout:Building perf 2024-07-26T06:52:46.948 INFO:tasks.workunit.client.0.smithi042.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/overlay/group.list 2024-07-26T06:52:46.950 INFO:tasks.workunit.client.0.smithi042.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/perf/group.list 2024-07-26T06:52:46.955 INFO:tasks.workunit.client.0.smithi042.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/generic/group.list 2024-07-26T06:52:46.966 INFO:tasks.workunit.client.0.smithi042.stdout:Building selftest 2024-07-26T06:52:46.971 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:46 smithi164 ceph-mon[36970]: pgmap v473: 97 pgs: 97 active+clean; 27 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 539 op/s 2024-07-26T06:52:46.973 INFO:tasks.workunit.client.0.smithi042.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/selftest/group.list 2024-07-26T06:52:46.990 INFO:tasks.workunit.client.0.smithi042.stdout:Building shared 2024-07-26T06:52:46.995 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] dataascii.lo 2024-07-26T06:52:46.998 INFO:tasks.workunit.client.0.smithi042.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/shared/group.list 2024-07-26T06:52:47.002 INFO:tasks.workunit.client.0.smithi042.stdout:Building tmpfs 2024-07-26T06:52:47.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:46 smithi163 ceph-mon[36873]: pgmap v473: 97 pgs: 97 active+clean; 27 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 539 op/s 2024-07-26T06:52:47.009 INFO:tasks.workunit.client.0.smithi042.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/tmpfs/group.list 2024-07-26T06:52:47.021 INFO:tasks.workunit.client.0.smithi042.stdout:Building udf 2024-07-26T06:52:47.027 INFO:tasks.workunit.client.0.smithi042.stdout:Building xfs 2024-07-26T06:52:47.028 INFO:tasks.workunit.client.0.smithi042.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/udf/group.list 2024-07-26T06:52:47.044 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] databin.lo 2024-07-26T06:52:47.101 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] datapid.lo 2024-07-26T06:52:47.111 INFO:tasks.workunit.client.0.smithi042.stdout: [GROUP] /home/ubuntu/cephtest/mnt.0/client.0/tmp/xfstests-dev/tests/xfs/group.list 2024-07-26T06:52:47.130 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] file_lock.lo 2024-07-26T06:52:47.212 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] forker.lo 2024-07-26T06:52:47.222 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] pattern.lo 2024-07-26T06:52:47.252 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] open_flags.lo 2024-07-26T06:52:47.310 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] random_range.lo 2024-07-26T06:52:47.357 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] string_to_tokens.lo 2024-07-26T06:52:47.362 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] str_to_bytes.lo 2024-07-26T06:52:47.410 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] tlibio.lo 2024-07-26T06:52:47.480 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] write_log.lo 2024-07-26T06:52:47.517 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] random.lo 2024-07-26T06:52:47.736 INFO:tasks.workunit.client.0.smithi042.stdout: [LD] 2024-07-26T06:52:47.864 INFO:tasks.workunit.client.0.smithi042.stdout:Building ltp 2024-07-26T06:52:47.865 INFO:tasks.workunit.client.0.smithi042.stdout:Building src 2024-07-26T06:52:47.865 INFO:tasks.workunit.client.0.smithi042.stdout:Building m4 2024-07-26T06:52:47.865 INFO:tasks.workunit.client.0.smithi042.stdout:Building common 2024-07-26T06:52:47.869 INFO:tasks.workunit.client.0.smithi042.stdout: [DEP] 2024-07-26T06:52:47.883 INFO:tasks.workunit.client.0.smithi042.stdout: [DEP] 2024-07-26T06:52:48.078 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] doio 2024-07-26T06:52:48.079 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] fsstress 2024-07-26T06:52:48.079 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] fsx 2024-07-26T06:52:48.780 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] growfiles 2024-07-26T06:52:48.836 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] iogen 2024-07-26T06:52:48.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:48 smithi042 ceph-mon[32775]: pgmap v474: 97 pgs: 97 active+clean; 31 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 1.7 MiB/s wr, 610 op/s 2024-07-26T06:52:48.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:48 smithi164 ceph-mon[36970]: pgmap v474: 97 pgs: 97 active+clean; 31 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 1.7 MiB/s wr, 610 op/s 2024-07-26T06:52:49.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:48 smithi163 ceph-mon[36873]: pgmap v474: 97 pgs: 97 active+clean; 31 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 1.7 MiB/s wr, 610 op/s 2024-07-26T06:52:49.230 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aio-stress 2024-07-26T06:52:49.941 INFO:tasks.workunit.client.0.smithi042.stdout: [DEP] 2024-07-26T06:52:49.942 INFO:tasks.workunit.client.0.smithi042.stdout: [DEP] 2024-07-26T06:52:49.942 INFO:tasks.workunit.client.0.smithi042.stdout: [DEP] 2024-07-26T06:52:49.943 INFO:tasks.workunit.client.0.smithi042.stdout: [DEP] 2024-07-26T06:52:50.191 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] dirstress 2024-07-26T06:52:50.192 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] fill 2024-07-26T06:52:50.192 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] fill2 2024-07-26T06:52:50.192 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] getpagesize 2024-07-26T06:52:50.368 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] holes 2024-07-26T06:52:50.378 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] lstat64 2024-07-26T06:52:50.460 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] nametest 2024-07-26T06:52:50.466 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] permname 2024-07-26T06:52:50.707 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] randholes 2024-07-26T06:52:50.707 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] runas 2024-07-26T06:52:50.707 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] truncfile 2024-07-26T06:52:50.780 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] usemem 2024-07-26T06:52:50.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:50 smithi042 ceph-mon[32775]: pgmap v475: 97 pgs: 97 active+clean; 35 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 1.9 MiB/s wr, 676 op/s 2024-07-26T06:52:50.924 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] mmapcat 2024-07-26T06:52:50.924 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] append_reader 2024-07-26T06:52:50.960 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] append_writer 2024-07-26T06:52:50.971 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:50 smithi164 ceph-mon[36970]: pgmap v475: 97 pgs: 97 active+clean; 35 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 1.9 MiB/s wr, 676 op/s 2024-07-26T06:52:50.999 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] dirperf 2024-07-26T06:52:51.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:50 smithi163 ceph-mon[36873]: pgmap v475: 97 pgs: 97 active+clean; 35 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 1.9 MiB/s wr, 676 op/s 2024-07-26T06:52:51.084 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] metaperf 2024-07-26T06:52:51.089 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] devzero 2024-07-26T06:52:51.113 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] feature 2024-07-26T06:52:51.266 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] alloc 2024-07-26T06:52:51.268 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] fault 2024-07-26T06:52:51.356 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] fstest 2024-07-26T06:52:51.367 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_access_root 2024-07-26T06:52:51.447 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] godown 2024-07-26T06:52:51.496 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] resvtest 2024-07-26T06:52:51.521 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] writemod 2024-07-26T06:52:51.640 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] writev_on_pagefault 2024-07-26T06:52:51.647 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] makeextents 2024-07-26T06:52:51.703 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] itrash 2024-07-26T06:52:51.704 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] rename 2024-07-26T06:52:51.806 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] multi_open_unlink 2024-07-26T06:52:51.853 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] unwritten_sync 2024-07-26T06:52:51.854 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] genhashnames 2024-07-26T06:52:51.880 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_holes 2024-07-26T06:52:51.985 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_mmap_writev 2024-07-26T06:52:52.044 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_truncate_cmtime 2024-07-26T06:52:52.045 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] dirhash_collide 2024-07-26T06:52:52.063 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_rename_overwrite 2024-07-26T06:52:52.141 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] holetest 2024-07-26T06:52:52.216 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_truncate_self 2024-07-26T06:52:52.216 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] af_unix 2024-07-26T06:52:52.256 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_mmap_stale_pmd 2024-07-26T06:52:52.367 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_mmap_cow_race 2024-07-26T06:52:52.375 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_mmap_fallocate 2024-07-26T06:52:52.417 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] fsync-err 2024-07-26T06:52:52.423 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_mmap_write_ro 2024-07-26T06:52:52.564 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_ext4_dax_journal_corruption 2024-07-26T06:52:52.566 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_ext4_dax_inline_corruption 2024-07-26T06:52:52.573 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_ofd_locks 2024-07-26T06:52:52.617 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_mmap_collision 2024-07-26T06:52:52.733 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] mmap-write-concurrent 2024-07-26T06:52:52.743 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_get_file_time 2024-07-26T06:52:52.778 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_create_short_dirs 2024-07-26T06:52:52.850 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_create_long_dirs 2024-07-26T06:52:52.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:52 smithi042 ceph-mon[32775]: pgmap v476: 97 pgs: 97 active+clean; 36 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 1.9 MiB/s wr, 626 op/s 2024-07-26T06:52:52.902 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_enospc 2024-07-26T06:52:52.922 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_mmap_writev_overlap 2024-07-26T06:52:52.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:52 smithi164 ceph-mon[36970]: pgmap v476: 97 pgs: 97 active+clean; 36 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 1.9 MiB/s wr, 626 op/s 2024-07-26T06:52:52.979 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] checkpoint_journal 2024-07-26T06:52:53.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:52 smithi163 ceph-mon[36873]: pgmap v476: 97 pgs: 97 active+clean; 36 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 1.9 MiB/s wr, 626 op/s 2024-07-26T06:52:53.036 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] mmap-rw-fault 2024-07-26T06:52:53.108 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] allocstale 2024-07-26T06:52:53.125 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_mmap_cow_memory_failure 2024-07-26T06:52:53.151 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] fake-dump-rootino 2024-07-26T06:52:53.248 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] dio-buf-fault 2024-07-26T06:52:53.296 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] rewinddir-test 2024-07-26T06:52:53.321 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] readdir-while-renames 2024-07-26T06:52:53.328 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] xfsctl 2024-07-26T06:52:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.354+0000 7f0c6494f640 1 -- 172.21.15.42:0/2400581647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c601039f0 msgr2=0x7f0c6010a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.354+0000 7f0c6494f640 1 --2- 172.21.15.42:0/2400581647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c601039f0 0x7f0c6010a390 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0c48009910 tx=0x7f0c4802a3f0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.354+0000 7f0c6494f640 1 -- 172.21.15.42:0/2400581647 shutdown_connections 2024-07-26T06:52:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.354+0000 7f0c6494f640 1 --2- 172.21.15.42:0/2400581647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c601039f0 0x7f0c6010a390 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.354+0000 7f0c6494f640 1 --2- 172.21.15.42:0/2400581647 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c60103050 0x7f0c60103460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.354+0000 7f0c6494f640 1 --2- 172.21.15.42:0/2400581647 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c60101fd0 0x7f0c60102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.354+0000 7f0c6494f640 1 -- 172.21.15.42:0/2400581647 >> 172.21.15.42:0/2400581647 conn(0x7f0c600fd910 msgr2=0x7f0c600ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.354+0000 7f0c6494f640 1 -- 172.21.15.42:0/2400581647 shutdown_connections 2024-07-26T06:52:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.354+0000 7f0c6494f640 1 -- 172.21.15.42:0/2400581647 wait complete. 2024-07-26T06:52:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.355+0000 7f0c6494f640 1 Processor -- start 2024-07-26T06:52:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.355+0000 7f0c6494f640 1 -- start start 2024-07-26T06:52:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c6494f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c60101fd0 0x7f0c60199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c6494f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c60103050 0x7f0c60199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c6494f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c601039f0 0x7f0c6019ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c6494f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6010dc50 con 0x7f0c601039f0 2024-07-26T06:52:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c6494f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6010dad0 con 0x7f0c60103050 2024-07-26T06:52:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c6494f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6010ddd0 con 0x7f0c60101fd0 2024-07-26T06:52:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c5dd9b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c60103050 0x7f0c60199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c5e59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c60101fd0 0x7f0c60199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c5ed9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c601039f0 0x7f0c6019ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c5e59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c60101fd0 0x7f0c60199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:49488/0 (socket says 172.21.15.42:49488) 2024-07-26T06:52:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c5ed9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c601039f0 0x7f0c6019ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35026/0 (socket says 172.21.15.42:35026) 2024-07-26T06:52:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c5e59c640 1 -- 172.21.15.42:0/1079837859 learned_addr learned my addr 172.21.15.42:0/1079837859 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c5dd9b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c60103050 0x7f0c60199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:37294/0 (socket says 172.21.15.42:37294) 2024-07-26T06:52:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c5ed9d640 1 -- 172.21.15.42:0/1079837859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c60101fd0 msgr2=0x7f0c60199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c5ed9d640 1 --2- 172.21.15.42:0/1079837859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c60101fd0 0x7f0c60199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c5ed9d640 1 -- 172.21.15.42:0/1079837859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c60103050 msgr2=0x7f0c60199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c5ed9d640 1 --2- 172.21.15.42:0/1079837859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c60103050 0x7f0c60199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c5ed9d640 1 -- 172.21.15.42:0/1079837859 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c6019b360 con 0x7f0c601039f0 2024-07-26T06:52:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.356+0000 7f0c5dd9b640 1 --2- 172.21.15.42:0/1079837859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c60103050 0x7f0c60199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:52:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.357+0000 7f0c5ed9d640 1 --2- 172.21.15.42:0/1079837859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c601039f0 0x7f0c6019ab40 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f0c4802a910 tx=0x7f0c48031890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.357+0000 7f0c577fe640 1 -- 172.21.15.42:0/1079837859 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c48041020 con 0x7f0c601039f0 2024-07-26T06:52:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.357+0000 7f0c6494f640 1 -- 172.21.15.42:0/1079837859 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c6019b5f0 con 0x7f0c601039f0 2024-07-26T06:52:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.357+0000 7f0c6494f640 1 -- 172.21.15.42:0/1079837859 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c601a1d40 con 0x7f0c601039f0 2024-07-26T06:52:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.357+0000 7f0c577fe640 1 -- 172.21.15.42:0/1079837859 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c48009520 con 0x7f0c601039f0 2024-07-26T06:52:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.357+0000 7f0c577fe640 1 -- 172.21.15.42:0/1079837859 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c48036040 con 0x7f0c601039f0 2024-07-26T06:52:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.360+0000 7f0c6b85f740 1 Processor -- start 2024-07-26T06:52:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.360+0000 7f0c577fe640 1 -- 172.21.15.42:0/1079837859 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f0c48044020 con 0x7f0c601039f0 2024-07-26T06:52:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.361+0000 7f0c6b85f740 1 -- start start 2024-07-26T06:52:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.361+0000 7f0c6b85f740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a4b1b0 0x564eb8df0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.361+0000 7f0c6b85f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a222f0 0x564eb8df07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.361+0000 7f0c6b85f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a2a4d0 0x564eb8df1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.361+0000 7f0c6b85f740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x564eb8d633b0 con 0x564eb8a4b1b0 2024-07-26T06:52:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.361+0000 7f0c6b85f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564eb8d528b0 con 0x564eb8a222f0 2024-07-26T06:52:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.361+0000 7f0c6b85f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x564eb8d568f0 con 0x564eb8a2a4d0 2024-07-26T06:52:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.361+0000 7f0c5e59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a222f0 0x564eb8df07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.362+0000 7f0c5e59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a222f0 0x564eb8df07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:37298/0 (socket says 172.21.15.42:37298) 2024-07-26T06:52:53.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.362+0000 7f0c5e59c640 1 -- 172.21.15.42:0/3375717825 learned_addr learned my addr 172.21.15.42:0/3375717825 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:53.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.362+0000 7f0c5dd9b640 1 --2- 172.21.15.42:0/3375717825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a4b1b0 0x564eb8df0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.362+0000 7f0c577fe640 1 --2- 172.21.15.42:0/1079837859 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0c4c0b3c50 0x7f0c4c0b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.362+0000 7f0c5dd9b640 1 -- 172.21.15.42:0/3375717825 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a2a4d0 msgr2=0x564eb8df1340 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T06:52:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.362+0000 7f0c5dd9b640 1 --2- 172.21.15.42:0/3375717825 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a2a4d0 0x564eb8df1340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.362+0000 7f0c5dd9b640 1 -- 172.21.15.42:0/3375717825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a222f0 msgr2=0x564eb8df07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.362+0000 7f0c5dd9b640 1 --2- 172.21.15.42:0/3375717825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a222f0 0x564eb8df07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.362+0000 7f0c5dd9b640 1 -- 172.21.15.42:0/3375717825 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564eb8cf99a0 con 0x564eb8a4b1b0 2024-07-26T06:52:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.362+0000 7f0c577fe640 1 -- 172.21.15.42:0/1079837859 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f0c48103f90 con 0x7f0c601039f0 2024-07-26T06:52:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.362+0000 7f0c5e59c640 1 --2- 172.21.15.42:0/3375717825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a222f0 0x564eb8df07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:52:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.362+0000 7f0c5ed9d640 1 --2- 172.21.15.42:0/1079837859 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0c4c0b3c50 0x7f0c4c0b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.363+0000 7f0c5dd9b640 1 --2- 172.21.15.42:0/3375717825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a4b1b0 0x564eb8df0180 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009fb0 tx=0x7f0c500118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b127afca046ff14c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.363+0000 7f0c54ff9640 1 -- 172.21.15.42:0/3375717825 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c50004020 con 0x564eb8a4b1b0 2024-07-26T06:52:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.363+0000 7f0c54ff9640 1 -- 172.21.15.42:0/3375717825 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c500054d0 con 0x564eb8a4b1b0 2024-07-26T06:52:53.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.363+0000 7f0c54ff9640 1 -- 172.21.15.42:0/3375717825 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c50004ce0 con 0x564eb8a4b1b0 2024-07-26T06:52:53.365 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.364+0000 7f0c6b85f740 1 -- 172.21.15.42:0/3375717825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a4b1b0 msgr2=0x564eb8df0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:53.365 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.364+0000 7f0c5ed9d640 1 --2- 172.21.15.42:0/1079837859 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0c4c0b3c50 0x7f0c4c0b6160 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0c480c7170 tx=0x7f0c48005ee0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.364+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/3375717825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a4b1b0 0x564eb8df0180 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009fb0 tx=0x7f0c500118d0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.365+0000 7f0c6b85f740 1 -- 172.21.15.42:0/3375717825 shutdown_connections 2024-07-26T06:52:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.365+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/3375717825 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a2a4d0 0x564eb8df1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.365+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/3375717825 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a222f0 0x564eb8df07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.365+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/3375717825 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a4b1b0 0x564eb8df0180 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.365+0000 7f0c6b85f740 1 -- 172.21.15.42:0/3375717825 >> 172.21.15.42:0/3375717825 conn(0x564eb8a3f300 msgr2=0x564eb8debc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.365+0000 7f0c6b85f740 1 -- 172.21.15.42:0/3375717825 shutdown_connections 2024-07-26T06:52:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.365+0000 7f0c6b85f740 1 -- 172.21.15.42:0/3375717825 wait complete. 2024-07-26T06:52:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.365+0000 7f0c6b85f740 1 -- start start 2024-07-26T06:52:53.366 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.365+0000 7f0c6b85f740 1 Processor -- start 2024-07-26T06:52:53.368 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.365+0000 7f0c6b85f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a222f0 0x564eb8df9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.365+0000 7f0c6b85f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a2a4d0 0x564eb8dfa290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.365+0000 7f0c6b85f740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a4b1b0 0x564eb8dfae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.366+0000 7f0c6b85f740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x564eb8c8cab0 con 0x564eb8a4b1b0 2024-07-26T06:52:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.366+0000 7f0c6b85f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564eb8cdacf0 con 0x564eb8a2a4d0 2024-07-26T06:52:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.366+0000 7f0c6b85f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x564eb8df1e00 con 0x564eb8a222f0 2024-07-26T06:52:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.366+0000 7f0c6b85f740 10 client.0 _ll_register_callbacks cb 0x564eb8d4c9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:52:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.366+0000 7f0c5dd9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a222f0 0x564eb8df9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.366+0000 7f0c5dd9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a4b1b0 0x564eb8dfae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.366+0000 7f0c5dd9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a4b1b0 0x564eb8dfae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35044/0 (socket says 172.21.15.42:35044) 2024-07-26T06:52:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.366+0000 7f0c5dd9b640 1 -- 172.21.15.42:0/2620289289 learned_addr learned my addr 172.21.15.42:0/2620289289 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.366+0000 7f0c5e59c640 1 --2- 172.21.15.42:0/2620289289 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a2a4d0 0x564eb8dfa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.366+0000 7f0c5e59c640 1 -- 172.21.15.42:0/2620289289 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a222f0 msgr2=0x564eb8df9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:53.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.366+0000 7f0c5e59c640 1 --2- 172.21.15.42:0/2620289289 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a222f0 0x564eb8df9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.366+0000 7f0c5e59c640 1 -- 172.21.15.42:0/2620289289 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a4b1b0 msgr2=0x564eb8dfae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.366+0000 7f0c5e59c640 1 --2- 172.21.15.42:0/2620289289 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a4b1b0 0x564eb8dfae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.366+0000 7f0c5e59c640 1 -- 172.21.15.42:0/2620289289 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564eb8ab6790 con 0x564eb8a2a4d0 2024-07-26T06:52:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.366+0000 7f0c5dd9b640 1 --2- 172.21.15.42:0/2620289289 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a4b1b0 0x564eb8dfae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:52:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.366+0000 7f0c5e59c640 1 --2- 172.21.15.42:0/2620289289 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a2a4d0 0x564eb8dfa290 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0c58011d10 tx=0x7f0c5800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b755985fcbcd725 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.366+0000 7f0c45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:52:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.367+0000 7f0c45ffb640 1 -- 172.21.15.42:0/2620289289 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5801e060 con 0x564eb8a2a4d0 2024-07-26T06:52:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.367+0000 7f0c45ffb640 1 -- 172.21.15.42:0/2620289289 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f0c58017050 con 0x564eb8a2a4d0 2024-07-26T06:52:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.367+0000 7f0c6b85f740 1 -- 172.21.15.42:0/2620289289 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c500054d0 con 0x564eb8a2a4d0 2024-07-26T06:52:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.367+0000 7f0c45ffb640 1 -- 172.21.15.42:0/2620289289 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0c58004b20 con 0x564eb8a2a4d0 2024-07-26T06:52:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.367+0000 7f0c6b85f740 10 client.25331 fetch_fsmap learned FSMap version 36 2024-07-26T06:52:53.370 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.367+0000 7f0c6b85f740 1 -- 172.21.15.42:0/2620289289 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564eb8b2f620 con 0x564eb8a2a4d0 2024-07-26T06:52:53.371 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.367+0000 7f0c45ffb640 1 -- 172.21.15.42:0/2620289289 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c500054d0 con 0x564eb8a2a4d0 2024-07-26T06:52:53.371 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.367+0000 7f0c45ffb640 1 -- 172.21.15.42:0/2620289289 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f0c58019070 con 0x564eb8a2a4d0 2024-07-26T06:52:53.371 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.367+0000 7f0c6b85f740 10 client.25331 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:52:53.371 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.367+0000 7f0c6b85f740 10 client.25331 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:52:53.371 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.367+0000 7f0c6b85f740 20 client.25331 populate_metadata read hostname 'smithi042' 2024-07-26T06:52:53.371 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.368+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/2620289289 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x564eb8a09d80 0x564eb8e0b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.371 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.368+0000 7f0c6b85f740 4 client.25331 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:52:53.371 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.368+0000 7f0c6b85f740 1 -- 172.21.15.42:0/2620289289 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564eb8d65300 con 0x564eb8a09d80 2024-07-26T06:52:53.371 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.368+0000 7f0c5dd9b640 1 --2- 172.21.15.42:0/2620289289 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x564eb8a09d80 0x564eb8e0b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:53.371 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.368+0000 7f0c5dd9b640 1 --2- 172.21.15.42:0/2620289289 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x564eb8a09d80 0x564eb8e0b940 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b38b11c5104db74d server_cookie=c562167bbfdf0d27 in_seq=0 out_seq=0 2024-07-26T06:52:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.370+0000 7f0c45ffb640 10 client.25331 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.370+0000 7f0c45ffb640 1 -- 172.21.15.42:0/2620289289 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564eb8d65300 con 0x564eb8a09d80 2024-07-26T06:52:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.370+0000 7f0c45ffb640 10 client.25331 handle_command_reply: tid=0 2024-07-26T06:52:53.372 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.371+0000 7f0c6b85f740 1 client.25331 shutdown 2024-07-26T06:52:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.371+0000 7f0c6b85f740 1 -- 172.21.15.42:0/2620289289 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a2a4d0 msgr2=0x564eb8dfa290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.371+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/2620289289 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a2a4d0 0x564eb8dfa290 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0c58011d10 tx=0x7f0c5800f790 comp rx=0 tx=0).stop 2024-07-26T06:52:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.371+0000 7f0c6b85f740 1 -- 172.21.15.42:0/2620289289 shutdown_connections 2024-07-26T06:52:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.371+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/2620289289 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x564eb8a09d80 0x564eb8e0b940 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.371+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/2620289289 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a4b1b0 0x564eb8dfae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.371+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/2620289289 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a2a4d0 0x564eb8dfa290 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.371+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/2620289289 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a222f0 0x564eb8df9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.371+0000 7f0c45ffb640 0 client.25331 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.371+0000 7f0c6b85f740 1 -- 172.21.15.42:0/2620289289 >> 172.21.15.42:0/2620289289 conn(0x564eb8a3f300 msgr2=0x564eb8dea2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.371+0000 7f0c6b85f740 1 -- 172.21.15.42:0/2620289289 shutdown_connections 2024-07-26T06:52:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.371+0000 7f0c6b85f740 1 -- 172.21.15.42:0/2620289289 wait complete. 2024-07-26T06:52:53.373 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.371+0000 7f0c6b85f740 20 client.25331 trim_cache size 0 max 16384 2024-07-26T06:52:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.398+0000 7f0c6b85f740 1 Processor -- start 2024-07-26T06:52:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.398+0000 7f0c6b85f740 1 -- start start 2024-07-26T06:52:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.398+0000 7f0c6b85f740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a222f0 0x564eb8deb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.398+0000 7f0c6b85f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a2a4d0 0x564eb8df9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.399+0000 7f0c6b85f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a4b1b0 0x564eb8dfa110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.399+0000 7f0c6b85f740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4000e520 con 0x564eb8a222f0 2024-07-26T06:52:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.399+0000 7f0c6b85f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0c40010150 con 0x564eb8a4b1b0 2024-07-26T06:52:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.399+0000 7f0c6b85f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4000ec20 con 0x564eb8a2a4d0 2024-07-26T06:52:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.399+0000 7f0c5dd9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a222f0 0x564eb8deb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:53.401 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.399+0000 7f0c5dd9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a222f0 0x564eb8deb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35056/0 (socket says 172.21.15.42:35056) 2024-07-26T06:52:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.399+0000 7f0c5dd9b640 1 -- 172.21.15.42:0/364647626 learned_addr learned my addr 172.21.15.42:0/364647626 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.399+0000 7f0c5e59c640 1 --2- 172.21.15.42:0/364647626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a2a4d0 0x564eb8df9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.399+0000 7f0c5dd9b640 1 --2- 172.21.15.42:0/364647626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a4b1b0 0x564eb8dfa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.399+0000 7f0c5dd9b640 1 -- 172.21.15.42:0/364647626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a2a4d0 msgr2=0x564eb8df9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.399+0000 7f0c5dd9b640 1 --2- 172.21.15.42:0/364647626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a2a4d0 0x564eb8df9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.399+0000 7f0c5dd9b640 1 -- 172.21.15.42:0/364647626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a4b1b0 msgr2=0x564eb8dfa110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.399+0000 7f0c5dd9b640 1 --2- 172.21.15.42:0/364647626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a4b1b0 0x564eb8dfa110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.399+0000 7f0c5dd9b640 1 -- 172.21.15.42:0/364647626 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564eb8df7d50 con 0x564eb8a222f0 2024-07-26T06:52:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.399+0000 7f0c5dd9b640 1 --2- 172.21.15.42:0/364647626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a222f0 0x564eb8deb550 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009250 tx=0x7f0c50005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e76b1ad6b3b54504 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.400+0000 7f0c47fff640 1 -- 172.21.15.42:0/364647626 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5001d040 con 0x564eb8a222f0 2024-07-26T06:52:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.400+0000 7f0c47fff640 1 -- 172.21.15.42:0/364647626 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c50005430 con 0x564eb8a222f0 2024-07-26T06:52:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.400+0000 7f0c47fff640 1 -- 172.21.15.42:0/364647626 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5000fc60 con 0x564eb8a222f0 2024-07-26T06:52:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.401+0000 7f0c6b85f740 1 -- 172.21.15.42:0/364647626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a222f0 msgr2=0x564eb8deb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.401+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/364647626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a222f0 0x564eb8deb550 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009250 tx=0x7f0c50005f90 comp rx=0 tx=0).stop 2024-07-26T06:52:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.401+0000 7f0c6b85f740 1 -- 172.21.15.42:0/364647626 shutdown_connections 2024-07-26T06:52:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.401+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/364647626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb8a4b1b0 0x564eb8dfa110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.401+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/364647626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a2a4d0 0x564eb8df9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.401+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/364647626 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a222f0 0x564eb8deb550 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.401+0000 7f0c6b85f740 1 -- 172.21.15.42:0/364647626 >> 172.21.15.42:0/364647626 conn(0x564eb8a3f300 msgr2=0x564eb8df4bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.401+0000 7f0c6b85f740 1 -- 172.21.15.42:0/364647626 shutdown_connections 2024-07-26T06:52:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.401+0000 7f0c6b85f740 1 -- 172.21.15.42:0/364647626 wait complete. 2024-07-26T06:52:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.402+0000 7f0c6b85f740 1 -- start start 2024-07-26T06:52:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.402+0000 7f0c6b85f740 1 Processor -- start 2024-07-26T06:52:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.402+0000 7f0c6b85f740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a2a4d0 0x564eb8e00280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.402+0000 7f0c6b85f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a4b1b0 0x564eb8e007c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.402+0000 7f0c6b85f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb89df1c0 0x564eb8e00d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.402+0000 7f0c6b85f740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4000fa80 con 0x564eb8a2a4d0 2024-07-26T06:52:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.402+0000 7f0c6b85f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4000f2f0 con 0x564eb89df1c0 2024-07-26T06:52:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.402+0000 7f0c6b85f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4000c500 con 0x564eb8a4b1b0 2024-07-26T06:52:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.402+0000 7f0c6b85f740 10 client.0 _ll_register_callbacks cb 0x7f0c58009d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:52:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.403+0000 7f0c5dd9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a2a4d0 0x564eb8e00280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.403+0000 7f0c5e59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a4b1b0 0x564eb8e007c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.403+0000 7f0c5dd9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a2a4d0 0x564eb8e00280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35062/0 (socket says 172.21.15.42:35062) 2024-07-26T06:52:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.403+0000 7f0c5dd9b640 1 -- 172.21.15.42:0/2428666743 learned_addr learned my addr 172.21.15.42:0/2428666743 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:52:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.403+0000 7f0c5dd9b640 1 --2- 172.21.15.42:0/2428666743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb89df1c0 0x564eb8e00d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.403+0000 7f0c5e59c640 1 -- 172.21.15.42:0/2428666743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb89df1c0 msgr2=0x564eb8e00d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.403+0000 7f0c5e59c640 1 --2- 172.21.15.42:0/2428666743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb89df1c0 0x564eb8e00d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.403+0000 7f0c5e59c640 1 -- 172.21.15.42:0/2428666743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a2a4d0 msgr2=0x564eb8e00280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:52:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.403+0000 7f0c5e59c640 1 --2- 172.21.15.42:0/2428666743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a2a4d0 0x564eb8e00280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.403+0000 7f0c5e59c640 1 -- 172.21.15.42:0/2428666743 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564eb8e72230 con 0x564eb8a4b1b0 2024-07-26T06:52:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.403+0000 7f0c5e59c640 1 --2- 172.21.15.42:0/2428666743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a4b1b0 0x564eb8e007c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0c58004630 tx=0x7f0c5800eda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a55f01476b20abfd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:52:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.403+0000 7f0c45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-07-26T06:52:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.404+0000 7f0c45ffb640 1 -- 172.21.15.42:0/2428666743 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c58028040 con 0x564eb8a4b1b0 2024-07-26T06:52:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.404+0000 7f0c6b85f740 1 -- 172.21.15.42:0/2428666743 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0c50005430 con 0x564eb8a4b1b0 2024-07-26T06:52:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.404+0000 7f0c45ffb640 1 -- 172.21.15.42:0/2428666743 <== mon.1 v2:172.21.15.164:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f0c58011e00 con 0x564eb8a4b1b0 2024-07-26T06:52:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.404+0000 7f0c45ffb640 1 -- 172.21.15.42:0/2428666743 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0c58004820 con 0x564eb8a4b1b0 2024-07-26T06:52:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.405+0000 7f0c6b85f740 10 client.25384 fetch_fsmap learned FSMap version 36 2024-07-26T06:52:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.405+0000 7f0c6b85f740 1 -- 172.21.15.42:0/2428666743 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564eb8e73530 con 0x564eb8a4b1b0 2024-07-26T06:52:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.405+0000 7f0c45ffb640 1 -- 172.21.15.42:0/2428666743 <== mon.1 v2:172.21.15.164:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c50005430 con 0x564eb8a4b1b0 2024-07-26T06:52:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.405+0000 7f0c45ffb640 1 -- 172.21.15.42:0/2428666743 <== mon.1 v2:172.21.15.164:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f0c58019070 con 0x564eb8a4b1b0 2024-07-26T06:52:53.409 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.405+0000 7f0c6b85f740 10 client.25384 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:52:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.405+0000 7f0c6b85f740 10 client.25384 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:52:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.405+0000 7f0c6b85f740 20 client.25384 populate_metadata read hostname 'smithi042' 2024-07-26T06:52:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.405+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/2428666743 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x564eb8e74020 0x564eb8e76440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:52:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.405+0000 7f0c6b85f740 4 client.25384 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:52:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.405+0000 7f0c6b85f740 1 -- 172.21.15.42:0/2428666743 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564eb8d60da0 con 0x564eb8e74020 2024-07-26T06:52:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.405+0000 7f0c5dd9b640 1 --2- 172.21.15.42:0/2428666743 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x564eb8e74020 0x564eb8e76440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:52:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.406+0000 7f0c45ffb640 10 client.25384 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:53.410 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.406+0000 7f0c5dd9b640 1 --2- 172.21.15.42:0/2428666743 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x564eb8e74020 0x564eb8e76440 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a460264cb766ed69 server_cookie=b3a68f382971d3a7 in_seq=0 out_seq=0 2024-07-26T06:52:53.424 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] bstat 2024-07-26T06:52:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.468+0000 7f0c45ffb640 1 -- 172.21.15.42:0/2428666743 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564eb8d60da0 con 0x564eb8e74020 2024-07-26T06:52:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.468+0000 7f0c45ffb640 10 client.25384 handle_command_reply: tid=0 2024-07-26T06:52:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.468+0000 7f0c6b85f740 1 client.25384 shutdown 2024-07-26T06:52:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.468+0000 7f0c6b85f740 1 -- 172.21.15.42:0/2428666743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a4b1b0 msgr2=0x564eb8e007c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.468+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/2428666743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a4b1b0 0x564eb8e007c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0c58004630 tx=0x7f0c5800eda0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.468+0000 7f0c6b85f740 1 -- 172.21.15.42:0/2428666743 shutdown_connections 2024-07-26T06:52:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.468+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/2428666743 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x564eb8e74020 0x564eb8e76440 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.468+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/2428666743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564eb89df1c0 0x564eb8e00d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.468+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/2428666743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x564eb8a4b1b0 0x564eb8e007c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.468+0000 7f0c6b85f740 1 --2- 172.21.15.42:0/2428666743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x564eb8a2a4d0 0x564eb8e00280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.468+0000 7f0c6b85f740 1 -- 172.21.15.42:0/2428666743 >> 172.21.15.42:0/2428666743 conn(0x564eb8a3f300 msgr2=0x564eb8dea2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:53.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.468+0000 7f0c45ffb640 0 client.25384 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:52:53.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.468+0000 7f0c6b85f740 1 -- 172.21.15.42:0/2428666743 shutdown_connections 2024-07-26T06:52:53.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.470+0000 7f0c6b85f740 1 -- 172.21.15.42:0/2428666743 wait complete. 2024-07-26T06:52:53.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.470+0000 7f0c6b85f740 20 client.25384 trim_cache size 0 max 16384 2024-07-26T06:52:53.471 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:52:53.471 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:52:53.471 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:52:53.471 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:52:53.472 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.471+0000 7f0c6494f640 1 -- 172.21.15.42:0/1079837859 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0c4c0b3c50 msgr2=0x7f0c4c0b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:53.472 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.471+0000 7f0c6494f640 1 --2- 172.21.15.42:0/1079837859 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0c4c0b3c50 0x7f0c4c0b6160 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0c480c7170 tx=0x7f0c48005ee0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.471+0000 7f0c6494f640 1 -- 172.21.15.42:0/1079837859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c601039f0 msgr2=0x7f0c6019ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:52:53.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.471+0000 7f0c6494f640 1 --2- 172.21.15.42:0/1079837859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c601039f0 0x7f0c6019ab40 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f0c4802a910 tx=0x7f0c48031890 comp rx=0 tx=0).stop 2024-07-26T06:52:53.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.472+0000 7f0c6494f640 1 -- 172.21.15.42:0/1079837859 shutdown_connections 2024-07-26T06:52:53.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.472+0000 7f0c6494f640 1 --2- 172.21.15.42:0/1079837859 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f0c4c0b3c50 0x7f0c4c0b6160 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.472+0000 7f0c6494f640 1 --2- 172.21.15.42:0/1079837859 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c601039f0 0x7f0c6019ab40 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.472+0000 7f0c6494f640 1 --2- 172.21.15.42:0/1079837859 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c60103050 0x7f0c60199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.472+0000 7f0c6494f640 1 --2- 172.21.15.42:0/1079837859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c60101fd0 0x7f0c60199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:52:53.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.472+0000 7f0c6494f640 1 -- 172.21.15.42:0/1079837859 >> 172.21.15.42:0/1079837859 conn(0x7f0c600fd910 msgr2=0x7f0c600ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:52:53.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.472+0000 7f0c6494f640 1 -- 172.21.15.42:0/1079837859 shutdown_connections 2024-07-26T06:52:53.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:52:53.472+0000 7f0c6494f640 1 -- 172.21.15.42:0/1079837859 wait complete. 2024-07-26T06:52:53.525 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_mtab 2024-07-26T06:52:53.527 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] getdevicesize 2024-07-26T06:52:53.594 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] preallo_rw_pattern_reader 2024-07-26T06:52:53.688 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] preallo_rw_pattern_writer 2024-07-26T06:52:53.698 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] ftrunc 2024-07-26T06:52:53.733 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] trunc 2024-07-26T06:52:53.770 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] fs_perms 2024-07-26T06:52:53.869 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] testx 2024-07-26T06:52:53.869 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] looptest 2024-07-26T06:52:53.899 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] locktest 2024-07-26T06:52:53.966 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] unwritten_mmap 2024-07-26T06:52:54.009 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] bulkstat_unlink_test 2024-07-26T06:52:54.084 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] deduperace 2024-07-26T06:52:54.135 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] bulkstat_unlink_test_modified 2024-07-26T06:52:54.235 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_dir_offset 2024-07-26T06:52:54.240 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_futimens 2024-07-26T06:52:54.299 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_immutable 2024-07-26T06:52:54.356 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] stale_handle 2024-07-26T06:52:54.401 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] pwrite_mmap_blocked 2024-07-26T06:52:54.406 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_dir_offset2 2024-07-26T06:52:54.532 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] seek_sanity_test 2024-07-26T06:52:54.555 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] seek_copy_test 2024-07-26T06:52:54.612 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_readdir_1 2024-07-26T06:52:54.741 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_readdir_2 2024-07-26T06:52:54.791 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] fsync-tester 2024-07-26T06:52:54.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:54 smithi042 ceph-mon[32775]: pgmap v477: 97 pgs: 97 active+clean; 36 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 1.7 MiB/s wr, 539 op/s 2024-07-26T06:52:54.897 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] nsexec 2024-07-26T06:52:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:54 smithi164 ceph-mon[36970]: pgmap v477: 97 pgs: 97 active+clean; 36 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 1.7 MiB/s wr, 539 op/s 2024-07-26T06:52:55.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:54 smithi163 ceph-mon[36873]: pgmap v477: 97 pgs: 97 active+clean; 36 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 1.7 MiB/s wr, 539 op/s 2024-07-26T06:52:55.023 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] cloner 2024-07-26T06:52:55.034 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] renameat2 2024-07-26T06:52:55.051 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_getcwd 2024-07-26T06:52:55.073 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] e4compact 2024-07-26T06:52:55.218 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] test-nextquota 2024-07-26T06:52:55.218 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] punch-alternating 2024-07-26T06:52:55.222 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] attr-list-by-handle-cursor-test 2024-07-26T06:52:55.287 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] listxattr 2024-07-26T06:52:55.402 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] dio-interleaved 2024-07-26T06:52:55.403 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_dir_type 2024-07-26T06:52:55.404 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] dio-invalidate-cache 2024-07-26T06:52:55.459 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] stat_test 2024-07-26T06:52:55.570 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_encrypted_d_revalidate 2024-07-26T06:52:55.572 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] attr_replace_test 2024-07-26T06:52:55.646 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] swapon 2024-07-26T06:52:55.750 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] mkswap 2024-07-26T06:52:55.787 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_attr_corruption 2024-07-26T06:52:55.829 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_open_tmpfiles 2024-07-26T06:52:55.840 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] fscrypt-crypt-util 2024-07-26T06:52:55.920 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] bulkstat_null_ocount 2024-07-26T06:52:55.946 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] splice-test 2024-07-26T06:52:56.024 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] chprojid_fail 2024-07-26T06:52:56.088 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] detached_mounts_propagation 2024-07-26T06:52:56.154 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] ext4_resize 2024-07-26T06:52:56.198 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_readdir_3 2024-07-26T06:52:56.369 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] splice2pipe 2024-07-26T06:52:56.369 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] uuid_ioctl 2024-07-26T06:52:56.418 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_snapshot_deleted_subvolume 2024-07-26T06:52:56.528 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] fiemap-fault 2024-07-26T06:52:56.536 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] fiemap-tester 2024-07-26T06:52:56.602 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_stripealign 2024-07-26T06:52:56.622 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] open_by_handle 2024-07-26T06:52:56.709 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_mmap_dio 2024-07-26T06:52:56.790 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] xfsfind 2024-07-26T06:52:56.825 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] t_reflink_read_race 2024-07-26T06:52:56.891 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] fssum 2024-07-26T06:52:56.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:56 smithi042 ceph-mon[32775]: pgmap v478: 97 pgs: 97 active+clean; 34 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 1.8 MiB/s wr, 470 op/s 2024-07-26T06:52:56.910 INFO:tasks.workunit.client.0.smithi042.stdout:Building log-writes 2024-07-26T06:52:56.917 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] replay-log 2024-07-26T06:52:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:56 smithi164 ceph-mon[36970]: pgmap v478: 97 pgs: 97 active+clean; 34 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 1.8 MiB/s wr, 470 op/s 2024-07-26T06:52:56.995 INFO:tasks.workunit.client.0.smithi042.stdout:Building perf 2024-07-26T06:52:56.999 INFO:tasks.workunit.client.0.smithi042.stdout:Building aio-dio-regress 2024-07-26T06:52:57.006 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aiocp 2024-07-26T06:52:57.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:56 smithi163 ceph-mon[36873]: pgmap v478: 97 pgs: 97 active+clean; 34 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 1.8 MiB/s wr, 470 op/s 2024-07-26T06:52:57.070 INFO:tasks.workunit.client.0.smithi042.stdout:Building vfs 2024-07-26T06:52:57.076 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] vfstest 2024-07-26T06:52:57.241 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aio-dio-append-write-fallocate-race 2024-07-26T06:52:57.275 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aio-dio-append-write-read-race 2024-07-26T06:52:57.424 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] mount-idmapped 2024-07-26T06:52:57.425 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aio-dio-cow-race 2024-07-26T06:52:57.442 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aio-dio-cycle-write 2024-07-26T06:52:57.591 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aio-dio-eof-race 2024-07-26T06:52:57.636 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aio-dio-extend-stat 2024-07-26T06:52:57.783 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aio-dio-fcntl-race 2024-07-26T06:52:57.817 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aio-dio-hole-filling-race 2024-07-26T06:52:57.841 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aio-dio-invalidate-failure 2024-07-26T06:52:57.951 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aio-dio-invalidate-readahead 2024-07-26T06:52:57.970 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aiodio_sparse2 2024-07-26T06:52:58.010 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aio-dio-subblock-eof-read 2024-07-26T06:52:58.137 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aio-dio-write-verify 2024-07-26T06:52:58.150 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aio-free-ring-with-bogus-nr-pages 2024-07-26T06:52:58.194 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aio-io-setup-with-nonwritable-context-pointer 2024-07-26T06:52:58.286 INFO:tasks.workunit.client.0.smithi042.stdout: [CC] aio-last-ref-held-by-io 2024-07-26T06:52:58.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:52:58 smithi042 ceph-mon[32775]: pgmap v479: 97 pgs: 97 active+clean; 34 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 1.2 MiB/s wr, 255 op/s 2024-07-26T06:52:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:52:58 smithi164 ceph-mon[36970]: pgmap v479: 97 pgs: 97 active+clean; 34 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 1.2 MiB/s wr, 255 op/s 2024-07-26T06:52:59.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:52:58 smithi163 ceph-mon[36873]: pgmap v479: 97 pgs: 97 active+clean; 34 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 1.2 MiB/s wr, 255 op/s 2024-07-26T06:52:59.193 INFO:tasks.workunit.client.0.smithi042.stderr:+ cd .. 2024-07-26T06:52:59.193 INFO:tasks.workunit.client.0.smithi042.stderr:+ cp xfstests-dev/ltp/fsx . 2024-07-26T06:52:59.196 INFO:tasks.workunit.client.0.smithi042.stderr:+ OPTIONS=-z 2024-07-26T06:52:59.197 INFO:tasks.workunit.client.0.smithi042.stderr:+ ./fsx -z 1MB -N 50000 -p 10000 -l 1048576 2024-07-26T06:52:59.197 INFO:tasks.workunit.client.0.smithi042.stdout:Seed set to 1 2024-07-26T06:52:59.221 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support fallocate mode 0, disabling! 2024-07-26T06:52:59.222 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support fallocate mode FALLOC_FL_KEEP_SIZE, disabling! 2024-07-26T06:52:59.224 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support fallocate mode FALLOC_FL_COLLAPSE_RANGE, disabling! 2024-07-26T06:52:59.224 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support fallocate mode FALLOC_FL_INSERT_RANGE, disabling! 2024-07-26T06:52:59.224 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support clone range, disabling! 2024-07-26T06:52:59.226 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support dedupe range, disabling! 2024-07-26T06:52:59.229 INFO:tasks.workunit.client.0.smithi042.stdout:skipping zero size read 2024-07-26T06:52:59.229 INFO:tasks.workunit.client.0.smithi042.stdout:skipping zero size read 2024-07-26T06:52:59.230 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support exchange range, disabling! 2024-07-26T06:52:59.230 INFO:tasks.workunit.client.0.smithi042.stdout:truncating to largest ever: 0x81d54 2024-07-26T06:52:59.234 INFO:tasks.workunit.client.0.smithi042.stdout:copying to largest ever: 0xb86d7 2024-07-26T06:52:59.242 INFO:tasks.workunit.client.0.smithi042.stdout:copying to largest ever: 0xe627e 2024-07-26T06:52:59.284 INFO:tasks.workunit.client.0.smithi042.stdout:truncating to largest ever: 0xf3194 2024-07-26T06:52:59.306 INFO:tasks.workunit.client.0.smithi042.stdout:copying to largest ever: 0xf3ceb 2024-07-26T06:52:59.313 INFO:tasks.workunit.client.0.smithi042.stdout:truncating to largest ever: 0xf47e4 2024-07-26T06:52:59.401 INFO:tasks.workunit.client.0.smithi042.stdout:copying to largest ever: 0xfaccc 2024-07-26T06:52:59.764 INFO:tasks.workunit.client.0.smithi042.stdout:copying to largest ever: 0xfe423 2024-07-26T06:53:00.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:00 smithi042 ceph-mon[32775]: pgmap v480: 97 pgs: 97 active+clean; 34 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 868 KiB/s wr, 193 op/s 2024-07-26T06:53:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:00 smithi164 ceph-mon[36970]: pgmap v480: 97 pgs: 97 active+clean; 34 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 868 KiB/s wr, 193 op/s 2024-07-26T06:53:01.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:00 smithi163 ceph-mon[36873]: pgmap v480: 97 pgs: 97 active+clean; 34 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 868 KiB/s wr, 193 op/s 2024-07-26T06:53:02.007 INFO:tasks.workunit.client.0.smithi042.stdout:copying to largest ever: 0xfe492 2024-07-26T06:53:02.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:02 smithi042 ceph-mon[32775]: pgmap v481: 97 pgs: 97 active+clean; 35 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 895 KiB/s wr, 168 op/s 2024-07-26T06:53:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:02 smithi164 ceph-mon[36970]: pgmap v481: 97 pgs: 97 active+clean; 35 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 895 KiB/s wr, 168 op/s 2024-07-26T06:53:03.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:02 smithi163 ceph-mon[36873]: pgmap v481: 97 pgs: 97 active+clean; 35 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 895 KiB/s wr, 168 op/s 2024-07-26T06:53:03.860 INFO:tasks.workunit.client.0.smithi042.stdout:copying to largest ever: 0xff038 2024-07-26T06:53:04.802 INFO:tasks.workunit.client.0.smithi042.stdout:truncating to largest ever: 0xffcac 2024-07-26T06:53:04.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:04 smithi042 ceph-mon[32775]: pgmap v482: 97 pgs: 97 active+clean; 36 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 879 KiB/s wr, 163 op/s 2024-07-26T06:53:04.919 INFO:tasks.workunit.client.0.smithi042.stdout:10000 copy from 0x2d237 to 0x30552, (0x331b bytes) at 0x82aed 2024-07-26T06:53:04.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:04 smithi164 ceph-mon[36970]: pgmap v482: 97 pgs: 97 active+clean; 36 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 879 KiB/s wr, 163 op/s 2024-07-26T06:53:05.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:04 smithi163 ceph-mon[36873]: pgmap v482: 97 pgs: 97 active+clean; 36 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 879 KiB/s wr, 163 op/s 2024-07-26T06:53:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.585+0000 7f364df60640 1 -- 172.21.15.42:0/2120406942 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36481062e0 msgr2=0x7f3648108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.585+0000 7f364df60640 1 --2- 172.21.15.42:0/2120406942 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36481062e0 0x7f3648108700 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f363c009900 tx=0x7f363c02a410 comp rx=0 tx=0).stop 2024-07-26T06:53:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.585+0000 7f364df60640 1 -- 172.21.15.42:0/2120406942 shutdown_connections 2024-07-26T06:53:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.585+0000 7f364df60640 1 --2- 172.21.15.42:0/2120406942 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3648108e40 0x7f364810b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.585+0000 7f364df60640 1 --2- 172.21.15.42:0/2120406942 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36481062e0 0x7f3648108700 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.585+0000 7f364df60640 1 --2- 172.21.15.42:0/2120406942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f364806be20 0x7f3648105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.585+0000 7f364df60640 1 -- 172.21.15.42:0/2120406942 >> 172.21.15.42:0/2120406942 conn(0x7f36480fd960 msgr2=0x7f36480ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.585+0000 7f364df60640 1 -- 172.21.15.42:0/2120406942 shutdown_connections 2024-07-26T06:53:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.585+0000 7f364df60640 1 -- 172.21.15.42:0/2120406942 wait complete. 2024-07-26T06:53:06.587 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.585+0000 7f364df60640 1 Processor -- start 2024-07-26T06:53:06.587 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f364df60640 1 -- start start 2024-07-26T06:53:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f364df60640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f364806be20 0x7f36481999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f364df60640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36481062e0 0x7f364819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f364df60640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3648108e40 0x7f364819ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f364df60640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f364810edb0 con 0x7f36481062e0 2024-07-26T06:53:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f364df60640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f364810ec30 con 0x7f364806be20 2024-07-26T06:53:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f364df60640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f364810ef30 con 0x7f3648108e40 2024-07-26T06:53:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f3646ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36481062e0 0x7f364819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f3646ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36481062e0 0x7f364819a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38698/0 (socket says 172.21.15.42:38698) 2024-07-26T06:53:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f3646ffd640 1 -- 172.21.15.42:0/703685661 learned_addr learned my addr 172.21.15.42:0/703685661 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f36477fe640 1 --2- 172.21.15.42:0/703685661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f364806be20 0x7f36481999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f3647fff640 1 --2- 172.21.15.42:0/703685661 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3648108e40 0x7f364819ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.590 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f36477fe640 1 -- 172.21.15.42:0/703685661 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3648108e40 msgr2=0x7f364819ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:06.590 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f36477fe640 1 --2- 172.21.15.42:0/703685661 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3648108e40 0x7f364819ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.590 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f36477fe640 1 -- 172.21.15.42:0/703685661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36481062e0 msgr2=0x7f364819a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:06.590 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f36477fe640 1 --2- 172.21.15.42:0/703685661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36481062e0 0x7f364819a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.590 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f36477fe640 1 -- 172.21.15.42:0/703685661 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f364819b430 con 0x7f364806be20 2024-07-26T06:53:06.590 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.586+0000 7f3646ffd640 1 --2- 172.21.15.42:0/703685661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36481062e0 0x7f364819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:06.590 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.587+0000 7f36477fe640 1 --2- 172.21.15.42:0/703685661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f364806be20 0x7f36481999f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f363800e9f0 tx=0x7f363800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:06.590 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.587+0000 7f3644ff9640 1 -- 172.21.15.42:0/703685661 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363800ccc0 con 0x7f364806be20 2024-07-26T06:53:06.590 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.587+0000 7f3644ff9640 1 -- 172.21.15.42:0/703685661 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36380108e0 con 0x7f364806be20 2024-07-26T06:53:06.590 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.587+0000 7f364df60640 1 -- 172.21.15.42:0/703685661 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f364819b720 con 0x7f364806be20 2024-07-26T06:53:06.590 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.587+0000 7f3644ff9640 1 -- 172.21.15.42:0/703685661 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3638018880 con 0x7f364806be20 2024-07-26T06:53:06.590 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.587+0000 7f364df60640 1 -- 172.21.15.42:0/703685661 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36481a1e10 con 0x7f364806be20 2024-07-26T06:53:06.593 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.590+0000 7f3644ff9640 1 -- 172.21.15.42:0/703685661 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f3638010430 con 0x7f364806be20 2024-07-26T06:53:06.593 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.590+0000 7f3654e70740 1 Processor -- start 2024-07-26T06:53:06.594 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.590+0000 7f3654e70740 1 -- start start 2024-07-26T06:53:06.594 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f3654e70740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3901b0 0x561eab735180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.594 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f3654e70740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab7357f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.594 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f3654e70740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab736340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.594 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f3654e70740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x561eab6a83b0 con 0x561eab3672f0 2024-07-26T06:53:06.594 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f3654e70740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561eab6978b0 con 0x561eab3901b0 2024-07-26T06:53:06.594 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f3654e70740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x561eab69b8f0 con 0x561eab36f4d0 2024-07-26T06:53:06.594 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f36477fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab7357f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.594 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f3647fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3901b0 0x561eab735180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.594 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f3646ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab736340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.594 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f36477fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab7357f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38710/0 (socket says 172.21.15.42:38710) 2024-07-26T06:53:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f36477fe640 1 -- 172.21.15.42:0/1728437689 learned_addr learned my addr 172.21.15.42:0/1728437689 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f3644ff9640 1 --2- 172.21.15.42:0/703685661 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36300b3bd0 0x7f36300b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f3646ffd640 1 -- 172.21.15.42:0/1728437689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3901b0 msgr2=0x561eab735180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f3646ffd640 1 --2- 172.21.15.42:0/1728437689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3901b0 0x561eab735180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f3646ffd640 1 -- 172.21.15.42:0/1728437689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 msgr2=0x561eab7357f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f3646ffd640 1 --2- 172.21.15.42:0/1728437689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab7357f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f3646ffd640 1 -- 172.21.15.42:0/1728437689 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561eab63e9a0 con 0x561eab36f4d0 2024-07-26T06:53:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f3647fff640 1 --2- 172.21.15.42:0/703685661 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36300b3bd0 0x7f36300b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f36477fe640 1 --2- 172.21.15.42:0/1728437689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab7357f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f3644ff9640 1 -- 172.21.15.42:0/703685661 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f36380e3fe0 con 0x7f364806be20 2024-07-26T06:53:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.591+0000 7f3646ffd640 1 --2- 172.21.15.42:0/1728437689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab736340 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f363c006010 tx=0x7f363c0043f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd87a39471edba26 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.592+0000 7f3647fff640 1 --2- 172.21.15.42:0/703685661 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36300b3bd0 0x7f36300b60e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3640009fd0 tx=0x7f3640004a40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.592+0000 7f362dffb640 1 -- 172.21.15.42:0/1728437689 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c040070 con 0x561eab36f4d0 2024-07-26T06:53:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.592+0000 7f362dffb640 1 -- 172.21.15.42:0/1728437689 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f363c0047a0 con 0x561eab36f4d0 2024-07-26T06:53:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.592+0000 7f362dffb640 1 -- 172.21.15.42:0/1728437689 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c035040 con 0x561eab36f4d0 2024-07-26T06:53:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.593+0000 7f3654e70740 1 -- 172.21.15.42:0/1728437689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 msgr2=0x561eab736340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.593+0000 7f3654e70740 1 --2- 172.21.15.42:0/1728437689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab736340 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f363c006010 tx=0x7f363c0043f0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.593+0000 7f3654e70740 1 -- 172.21.15.42:0/1728437689 shutdown_connections 2024-07-26T06:53:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.593+0000 7f3654e70740 1 --2- 172.21.15.42:0/1728437689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab736340 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.593+0000 7f3654e70740 1 --2- 172.21.15.42:0/1728437689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab7357f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.593+0000 7f3654e70740 1 --2- 172.21.15.42:0/1728437689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3901b0 0x561eab735180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.593+0000 7f3654e70740 1 -- 172.21.15.42:0/1728437689 >> 172.21.15.42:0/1728437689 conn(0x561eab384300 msgr2=0x561eab730c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.593+0000 7f3654e70740 1 -- 172.21.15.42:0/1728437689 shutdown_connections 2024-07-26T06:53:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.593+0000 7f3654e70740 1 -- 172.21.15.42:0/1728437689 wait complete. 2024-07-26T06:53:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.593+0000 7f3654e70740 1 -- start start 2024-07-26T06:53:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.593+0000 7f3654e70740 1 Processor -- start 2024-07-26T06:53:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f3654e70740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab73eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f3654e70740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab73f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f3654e70740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3901b0 0x561eab73fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f3654e70740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x561eab5d1ab0 con 0x561eab3672f0 2024-07-26T06:53:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f3654e70740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561eab61fcf0 con 0x561eab3901b0 2024-07-26T06:53:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f3654e70740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x561eab736e00 con 0x561eab36f4d0 2024-07-26T06:53:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f36477fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab73eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f36477fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab73eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38720/0 (socket says 172.21.15.42:38720) 2024-07-26T06:53:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f36477fe640 1 -- 172.21.15.42:0/719042853 learned_addr learned my addr 172.21.15.42:0/719042853 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f3647fff640 1 --2- 172.21.15.42:0/719042853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3901b0 0x561eab73fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f3646ffd640 1 --2- 172.21.15.42:0/719042853 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab73f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f3654e70740 10 client.0 _ll_register_callbacks cb 0x561eab6919f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:53:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f36477fe640 1 -- 172.21.15.42:0/719042853 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 msgr2=0x561eab73f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f36477fe640 1 --2- 172.21.15.42:0/719042853 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab73f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f36477fe640 1 -- 172.21.15.42:0/719042853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3901b0 msgr2=0x561eab73fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f36477fe640 1 --2- 172.21.15.42:0/719042853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3901b0 0x561eab73fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f36477fe640 1 -- 172.21.15.42:0/719042853 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561eab3fb790 con 0x561eab3672f0 2024-07-26T06:53:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f36477fe640 1 --2- 172.21.15.42:0/719042853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab73eb40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f36380e77c0 tx=0x7f363800cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30d80294a7f5f6a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f3626ffd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:53:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.594+0000 7f3626ffd640 1 -- 172.21.15.42:0/719042853 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36380aa6d0 con 0x561eab3672f0 2024-07-26T06:53:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.595+0000 7f3626ffd640 1 -- 172.21.15.42:0/719042853 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f36380aa3c0 con 0x561eab3672f0 2024-07-26T06:53:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.595+0000 7f3654e70740 1 -- 172.21.15.42:0/719042853 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f363c0047a0 con 0x561eab3672f0 2024-07-26T06:53:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.595+0000 7f3626ffd640 1 -- 172.21.15.42:0/719042853 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f3638010720 con 0x561eab3672f0 2024-07-26T06:53:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.595+0000 7f3654e70740 10 client.15750 fetch_fsmap learned FSMap version 36 2024-07-26T06:53:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.595+0000 7f3654e70740 1 -- 172.21.15.42:0/719042853 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x561eab474620 con 0x561eab3672f0 2024-07-26T06:53:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.595+0000 7f3626ffd640 1 -- 172.21.15.42:0/719042853 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f363c0047a0 con 0x561eab3672f0 2024-07-26T06:53:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.595+0000 7f3626ffd640 1 -- 172.21.15.42:0/719042853 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f36380a68d0 con 0x561eab3672f0 2024-07-26T06:53:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.595+0000 7f3654e70740 10 client.15750 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:53:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.595+0000 7f3654e70740 10 client.15750 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:53:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.595+0000 7f3654e70740 20 client.15750 populate_metadata read hostname 'smithi042' 2024-07-26T06:53:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.595+0000 7f3654e70740 1 --2- 172.21.15.42:0/719042853 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561eab34ed80 0x561eab74c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.595+0000 7f3654e70740 4 client.15750 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:53:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.595+0000 7f3654e70740 1 -- 172.21.15.42:0/719042853 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561eab6aa300 con 0x561eab34ed80 2024-07-26T06:53:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.596+0000 7f3646ffd640 1 --2- 172.21.15.42:0/719042853 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561eab34ed80 0x561eab74c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.596+0000 7f3626ffd640 10 client.15750 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.596+0000 7f3646ffd640 1 --2- 172.21.15.42:0/719042853 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561eab34ed80 0x561eab74c2d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7840927d4da2545f server_cookie=dc0e6ffc7f7e06b3 in_seq=0 out_seq=0 2024-07-26T06:53:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.597+0000 7f3626ffd640 1 -- 172.21.15.42:0/719042853 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561eab6aa300 con 0x561eab34ed80 2024-07-26T06:53:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.597+0000 7f3626ffd640 10 client.15750 handle_command_reply: tid=0 2024-07-26T06:53:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.597+0000 7f3654e70740 1 client.15750 shutdown 2024-07-26T06:53:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.597+0000 7f3654e70740 1 -- 172.21.15.42:0/719042853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 msgr2=0x561eab73eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.597+0000 7f3654e70740 1 --2- 172.21.15.42:0/719042853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab73eb40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f36380e77c0 tx=0x7f363800cef0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.597+0000 7f3654e70740 1 -- 172.21.15.42:0/719042853 shutdown_connections 2024-07-26T06:53:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.597+0000 7f3654e70740 1 --2- 172.21.15.42:0/719042853 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561eab34ed80 0x561eab74c2d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.597+0000 7f3654e70740 1 --2- 172.21.15.42:0/719042853 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3901b0 0x561eab73fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.597+0000 7f3654e70740 1 --2- 172.21.15.42:0/719042853 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab73f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.597+0000 7f3626ffd640 0 client.15750 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.597+0000 7f3654e70740 1 --2- 172.21.15.42:0/719042853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab73eb40 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.597+0000 7f3654e70740 1 -- 172.21.15.42:0/719042853 >> 172.21.15.42:0/719042853 conn(0x561eab384300 msgr2=0x561eab72f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.597+0000 7f3654e70740 1 -- 172.21.15.42:0/719042853 shutdown_connections 2024-07-26T06:53:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.597+0000 7f3654e70740 1 -- 172.21.15.42:0/719042853 wait complete. 2024-07-26T06:53:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.597+0000 7f3654e70740 20 client.15750 trim_cache size 0 max 16384 2024-07-26T06:53:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.613+0000 7f3654e70740 1 Processor -- start 2024-07-26T06:53:06.615 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.613+0000 7f3654e70740 1 -- start start 2024-07-26T06:53:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f3654e70740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3672f0 0x561eab735fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f3654e70740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab735480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f3654e70740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3901b0 0x561eab72f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f3654e70740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f36481aab10 con 0x561eab3901b0 2024-07-26T06:53:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f3654e70740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f36481ac740 con 0x561eab3672f0 2024-07-26T06:53:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f3654e70740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f36481ab210 con 0x561eab36f4d0 2024-07-26T06:53:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f3647fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3901b0 0x561eab72f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f3647fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3901b0 0x561eab72f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38722/0 (socket says 172.21.15.42:38722) 2024-07-26T06:53:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f3647fff640 1 -- 172.21.15.42:0/1745159069 learned_addr learned my addr 172.21.15.42:0/1745159069 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f36477fe640 1 --2- 172.21.15.42:0/1745159069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3672f0 0x561eab735fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f3646ffd640 1 --2- 172.21.15.42:0/1745159069 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab735480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.616 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f3647fff640 1 -- 172.21.15.42:0/1745159069 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 msgr2=0x561eab735480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f3647fff640 1 --2- 172.21.15.42:0/1745159069 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab735480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f3647fff640 1 -- 172.21.15.42:0/1745159069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3672f0 msgr2=0x561eab735fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f3647fff640 1 --2- 172.21.15.42:0/1745159069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3672f0 0x561eab735fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f3647fff640 1 -- 172.21.15.42:0/1745159069 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561eab72afe0 con 0x561eab3901b0 2024-07-26T06:53:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f36477fe640 1 --2- 172.21.15.42:0/1745159069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3672f0 0x561eab735fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.614+0000 7f3646ffd640 1 --2- 172.21.15.42:0/1745159069 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab735480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.615+0000 7f3647fff640 1 --2- 172.21.15.42:0/1745159069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3901b0 0x561eab72f2b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f364000faa0 tx=0x7f3640013030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f058b8535ffbcb4f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.615+0000 7f362d7fa640 1 -- 172.21.15.42:0/1745159069 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3640004b20 con 0x561eab3901b0 2024-07-26T06:53:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.615+0000 7f362d7fa640 1 -- 172.21.15.42:0/1745159069 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3640004cc0 con 0x561eab3901b0 2024-07-26T06:53:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.615+0000 7f362d7fa640 1 -- 172.21.15.42:0/1745159069 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3640005790 con 0x561eab3901b0 2024-07-26T06:53:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.616+0000 7f3654e70740 1 -- 172.21.15.42:0/1745159069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3901b0 msgr2=0x561eab72f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.616+0000 7f3654e70740 1 --2- 172.21.15.42:0/1745159069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3901b0 0x561eab72f2b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f364000faa0 tx=0x7f3640013030 comp rx=0 tx=0).stop 2024-07-26T06:53:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.616+0000 7f3654e70740 1 -- 172.21.15.42:0/1745159069 shutdown_connections 2024-07-26T06:53:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.616+0000 7f3654e70740 1 --2- 172.21.15.42:0/1745159069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3901b0 0x561eab72f2b0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.616+0000 7f3654e70740 1 --2- 172.21.15.42:0/1745159069 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab735480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.616+0000 7f3654e70740 1 --2- 172.21.15.42:0/1745159069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3672f0 0x561eab735fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.616+0000 7f3654e70740 1 -- 172.21.15.42:0/1745159069 >> 172.21.15.42:0/1745159069 conn(0x561eab384300 msgr2=0x561eab734d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.616+0000 7f3654e70740 1 -- 172.21.15.42:0/1745159069 shutdown_connections 2024-07-26T06:53:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.616+0000 7f3654e70740 1 -- 172.21.15.42:0/1745159069 wait complete. 2024-07-26T06:53:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.616+0000 7f3654e70740 1 -- start start 2024-07-26T06:53:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f3654e70740 1 Processor -- start 2024-07-26T06:53:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f3654e70740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab746bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f3654e70740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab745550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f3654e70740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3901b0 0x561eab745c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f3654e70740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f36481ac070 con 0x561eab3672f0 2024-07-26T06:53:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f3654e70740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f36481ab8e0 con 0x561eab3901b0 2024-07-26T06:53:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f3654e70740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f36481a8af0 con 0x561eab36f4d0 2024-07-26T06:53:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f3654e70740 10 client.0 _ll_register_callbacks cb 0x7f363800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:53:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f36477fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab746bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f36477fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab746bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38728/0 (socket says 172.21.15.42:38728) 2024-07-26T06:53:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f36477fe640 1 -- 172.21.15.42:0/2270482926 learned_addr learned my addr 172.21.15.42:0/2270482926 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f3646ffd640 1 --2- 172.21.15.42:0/2270482926 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab745550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f3647fff640 1 --2- 172.21.15.42:0/2270482926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3901b0 0x561eab745c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f36477fe640 1 -- 172.21.15.42:0/2270482926 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 msgr2=0x561eab745550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f36477fe640 1 --2- 172.21.15.42:0/2270482926 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab745550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f36477fe640 1 -- 172.21.15.42:0/2270482926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3901b0 msgr2=0x561eab745c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f36477fe640 1 --2- 172.21.15.42:0/2270482926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3901b0 0x561eab745c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f36477fe640 1 -- 172.21.15.42:0/2270482926 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561eab7b2a40 con 0x561eab3672f0 2024-07-26T06:53:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.617+0000 7f3646ffd640 1 --2- 172.21.15.42:0/2270482926 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab745550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.618+0000 7f36477fe640 1 --2- 172.21.15.42:0/2270482926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab746bb0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f36380a6cc0 tx=0x7f36380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab3aa3cb00c6f682 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.618+0000 7f3626ffd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:53:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.618+0000 7f3626ffd640 1 -- 172.21.15.42:0/2270482926 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36380c9060 con 0x561eab3672f0 2024-07-26T06:53:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.618+0000 7f3626ffd640 1 -- 172.21.15.42:0/2270482926 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f36380a9c30 con 0x561eab3672f0 2024-07-26T06:53:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.618+0000 7f3654e70740 1 -- 172.21.15.42:0/2270482926 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3640004cc0 con 0x561eab3672f0 2024-07-26T06:53:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.618+0000 7f3626ffd640 1 -- 172.21.15.42:0/2270482926 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f36380aaa70 con 0x561eab3672f0 2024-07-26T06:53:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.618+0000 7f3654e70740 10 client.15756 fetch_fsmap learned FSMap version 36 2024-07-26T06:53:06.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.618+0000 7f3654e70740 1 -- 172.21.15.42:0/2270482926 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x561eab7b3d10 con 0x561eab3672f0 2024-07-26T06:53:06.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.618+0000 7f3626ffd640 1 -- 172.21.15.42:0/2270482926 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3640004cc0 con 0x561eab3672f0 2024-07-26T06:53:06.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.618+0000 7f3626ffd640 1 -- 172.21.15.42:0/2270482926 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f36380a9e30 con 0x561eab3672f0 2024-07-26T06:53:06.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.618+0000 7f3654e70740 10 client.15756 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:53:06.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.618+0000 7f3654e70740 10 client.15756 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:53:06.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.618+0000 7f3654e70740 20 client.15756 populate_metadata read hostname 'smithi042' 2024-07-26T06:53:06.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.618+0000 7f3654e70740 1 --2- 172.21.15.42:0/2270482926 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561eab3241c0 0x561eab7b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:06.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.618+0000 7f3654e70740 4 client.15756 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:53:06.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.618+0000 7f3654e70740 1 -- 172.21.15.42:0/2270482926 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561eab6a5da0 con 0x561eab3241c0 2024-07-26T06:53:06.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.619+0000 7f3646ffd640 1 --2- 172.21.15.42:0/2270482926 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561eab3241c0 0x561eab7b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:06.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.619+0000 7f3626ffd640 10 client.15756 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:06.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.619+0000 7f3646ffd640 1 --2- 172.21.15.42:0/2270482926 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561eab3241c0 0x561eab7b65f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a0a07a8c8d21ea0 server_cookie=2d70d145ddb4858 in_seq=0 out_seq=0 2024-07-26T06:53:06.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.640+0000 7f3626ffd640 1 -- 172.21.15.42:0/2270482926 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561eab6a5da0 con 0x561eab3241c0 2024-07-26T06:53:06.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.640+0000 7f3626ffd640 10 client.15756 handle_command_reply: tid=0 2024-07-26T06:53:06.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.640+0000 7f3654e70740 1 client.15756 shutdown 2024-07-26T06:53:06.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.640+0000 7f3654e70740 1 -- 172.21.15.42:0/2270482926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 msgr2=0x561eab746bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:06.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.640+0000 7f3654e70740 1 --2- 172.21.15.42:0/2270482926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab746bb0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f36380a6cc0 tx=0x7f36380057f0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.640+0000 7f3654e70740 1 -- 172.21.15.42:0/2270482926 shutdown_connections 2024-07-26T06:53:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.640+0000 7f3654e70740 1 --2- 172.21.15.42:0/2270482926 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561eab3241c0 0x561eab7b65f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.640+0000 7f3654e70740 1 --2- 172.21.15.42:0/2270482926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561eab3901b0 0x561eab745c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.640+0000 7f3654e70740 1 --2- 172.21.15.42:0/2270482926 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561eab36f4d0 0x561eab745550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.640+0000 7f3654e70740 1 --2- 172.21.15.42:0/2270482926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561eab3672f0 0x561eab746bb0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.640+0000 7f3626ffd640 0 client.15756 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.640+0000 7f3654e70740 1 -- 172.21.15.42:0/2270482926 >> 172.21.15.42:0/2270482926 conn(0x561eab384300 msgr2=0x561eab731690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.640+0000 7f3654e70740 1 -- 172.21.15.42:0/2270482926 shutdown_connections 2024-07-26T06:53:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.640+0000 7f3654e70740 1 -- 172.21.15.42:0/2270482926 wait complete. 2024-07-26T06:53:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.640+0000 7f3654e70740 20 client.15756 trim_cache size 0 max 16384 2024-07-26T06:53:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:53:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:53:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:53:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:53:06.644 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.641+0000 7f364df60640 1 -- 172.21.15.42:0/703685661 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36300b3bd0 msgr2=0x7f36300b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:06.644 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.641+0000 7f364df60640 1 --2- 172.21.15.42:0/703685661 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36300b3bd0 0x7f36300b60e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3640009fd0 tx=0x7f3640004a40 comp rx=0 tx=0).stop 2024-07-26T06:53:06.644 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.641+0000 7f364df60640 1 -- 172.21.15.42:0/703685661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f364806be20 msgr2=0x7f36481999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:06.644 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.641+0000 7f364df60640 1 --2- 172.21.15.42:0/703685661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f364806be20 0x7f36481999f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f363800e9f0 tx=0x7f363800eeb0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.644 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.641+0000 7f364df60640 1 -- 172.21.15.42:0/703685661 shutdown_connections 2024-07-26T06:53:06.644 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.641+0000 7f364df60640 1 --2- 172.21.15.42:0/703685661 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36300b3bd0 0x7f36300b60e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.644 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.641+0000 7f364df60640 1 --2- 172.21.15.42:0/703685661 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3648108e40 0x7f364819ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.644 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.641+0000 7f364df60640 1 --2- 172.21.15.42:0/703685661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36481062e0 0x7f364819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.645 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.641+0000 7f364df60640 1 --2- 172.21.15.42:0/703685661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f364806be20 0x7f36481999f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:06.645 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.641+0000 7f364df60640 1 -- 172.21.15.42:0/703685661 >> 172.21.15.42:0/703685661 conn(0x7f36480fd960 msgr2=0x7f3648104810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:06.645 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.641+0000 7f364df60640 1 -- 172.21.15.42:0/703685661 shutdown_connections 2024-07-26T06:53:06.645 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:06.641+0000 7f364df60640 1 -- 172.21.15.42:0/703685661 wait complete. 2024-07-26T06:53:06.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:06 smithi042 ceph-mon[32775]: pgmap v483: 97 pgs: 97 active+clean; 38 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 1.1 MiB/s wr, 268 op/s 2024-07-26T06:53:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:06 smithi164 ceph-mon[36970]: pgmap v483: 97 pgs: 97 active+clean; 38 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 1.1 MiB/s wr, 268 op/s 2024-07-26T06:53:07.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:06 smithi163 ceph-mon[36873]: pgmap v483: 97 pgs: 97 active+clean; 38 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 1.1 MiB/s wr, 268 op/s 2024-07-26T06:53:08.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:08 smithi042 ceph-mon[32775]: pgmap v484: 97 pgs: 97 active+clean; 38 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 446 op/s 2024-07-26T06:53:08.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:08 smithi164 ceph-mon[36970]: pgmap v484: 97 pgs: 97 active+clean; 38 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 446 op/s 2024-07-26T06:53:09.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:08 smithi163 ceph-mon[36873]: pgmap v484: 97 pgs: 97 active+clean; 38 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 446 op/s 2024-07-26T06:53:10.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:10 smithi042 ceph-mon[32775]: pgmap v485: 97 pgs: 97 active+clean; 37 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 556 op/s 2024-07-26T06:53:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:53:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:53:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:53:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:53:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:10 smithi164 ceph-mon[36970]: pgmap v485: 97 pgs: 97 active+clean; 37 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 556 op/s 2024-07-26T06:53:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:53:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:53:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:53:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:53:11.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:10 smithi163 ceph-mon[36873]: pgmap v485: 97 pgs: 97 active+clean; 37 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 556 op/s 2024-07-26T06:53:11.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:53:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:53:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:53:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:53:12.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:12 smithi042 ceph-mon[32775]: pgmap v486: 97 pgs: 97 active+clean; 37 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 729 op/s 2024-07-26T06:53:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:12 smithi164 ceph-mon[36970]: pgmap v486: 97 pgs: 97 active+clean; 37 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 729 op/s 2024-07-26T06:53:13.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:12 smithi163 ceph-mon[36873]: pgmap v486: 97 pgs: 97 active+clean; 37 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 729 op/s 2024-07-26T06:53:14.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:14 smithi042 ceph-mon[32775]: pgmap v487: 97 pgs: 97 active+clean; 37 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.1 MiB/s wr, 660 op/s 2024-07-26T06:53:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:14 smithi164 ceph-mon[36970]: pgmap v487: 97 pgs: 97 active+clean; 37 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.1 MiB/s wr, 660 op/s 2024-07-26T06:53:15.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:14 smithi163 ceph-mon[36873]: pgmap v487: 97 pgs: 97 active+clean; 37 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.1 MiB/s wr, 660 op/s 2024-07-26T06:53:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:16 smithi042 ceph-mon[32775]: pgmap v488: 97 pgs: 97 active+clean; 40 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 762 op/s 2024-07-26T06:53:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:16 smithi164 ceph-mon[36970]: pgmap v488: 97 pgs: 97 active+clean; 40 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 762 op/s 2024-07-26T06:53:17.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:16 smithi163 ceph-mon[36873]: pgmap v488: 97 pgs: 97 active+clean; 40 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 762 op/s 2024-07-26T06:53:18.684 INFO:tasks.workunit.client.0.smithi042.stdout:30000 trunc from 0x7bb2f to 0xcdf83 2024-07-26T06:53:18.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:18 smithi042 ceph-mon[32775]: pgmap v489: 97 pgs: 97 active+clean; 40 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.5 MiB/s wr, 841 op/s 2024-07-26T06:53:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:18 smithi164 ceph-mon[36970]: pgmap v489: 97 pgs: 97 active+clean; 40 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.5 MiB/s wr, 841 op/s 2024-07-26T06:53:19.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:18 smithi163 ceph-mon[36873]: pgmap v489: 97 pgs: 97 active+clean; 40 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.5 MiB/s wr, 841 op/s 2024-07-26T06:53:19.016 INFO:tasks.workunit.client.0.smithi042.stdout:copying to largest ever: 0xffcdd 2024-07-26T06:53:19.769 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.767+0000 7fe8affff640 1 -- 172.21.15.42:0/994109318 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8a8095ce0 msgr2=0x7fe8a80960f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:19.769 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.767+0000 7fe8affff640 1 --2- 172.21.15.42:0/994109318 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8a8095ce0 0x7fe8a80960f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4009900 tx=0x7fe8a402a410 comp rx=0 tx=0).stop 2024-07-26T06:53:19.769 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.767+0000 7fe8affff640 1 -- 172.21.15.42:0/994109318 shutdown_connections 2024-07-26T06:53:19.769 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.767+0000 7fe8affff640 1 --2- 172.21.15.42:0/994109318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8a8096680 0x7fe8a809d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.770 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.767+0000 7fe8affff640 1 --2- 172.21.15.42:0/994109318 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8a8095ce0 0x7fe8a80960f0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.770 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.767+0000 7fe8affff640 1 --2- 172.21.15.42:0/994109318 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8a8094c60 0x7fe8a80950d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.770 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.767+0000 7fe8affff640 1 -- 172.21.15.42:0/994109318 >> 172.21.15.42:0/994109318 conn(0x7fe8a80905a0 msgr2=0x7fe8a8092a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:19.770 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.767+0000 7fe8affff640 1 -- 172.21.15.42:0/994109318 shutdown_connections 2024-07-26T06:53:19.770 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.768+0000 7fe8affff640 1 -- 172.21.15.42:0/994109318 wait complete. 2024-07-26T06:53:19.770 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.768+0000 7fe8affff640 1 Processor -- start 2024-07-26T06:53:19.770 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.768+0000 7fe8affff640 1 -- start start 2024-07-26T06:53:19.772 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.769+0000 7fe8affff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8a8094c60 0x7fe8a812b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.769+0000 7fe8affff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8a8095ce0 0x7fe8a812bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.769+0000 7fe8affff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8a8096680 0x7fe8a812c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.769+0000 7fe8affff640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a80a08b0 con 0x7fe8a8096680 2024-07-26T06:53:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.769+0000 7fe8affff640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a80a0730 con 0x7fe8a8095ce0 2024-07-26T06:53:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.769+0000 7fe8affff640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a80a0a30 con 0x7fe8a8094c60 2024-07-26T06:53:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.769+0000 7fe8af7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8a8096680 0x7fe8a812c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.769+0000 7fe8af7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8a8096680 0x7fe8a812c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54086/0 (socket says 172.21.15.42:54086) 2024-07-26T06:53:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.769+0000 7fe8af7fe640 1 -- 172.21.15.42:0/1522315207 learned_addr learned my addr 172.21.15.42:0/1522315207 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.769+0000 7fe8ae7fc640 1 --2- 172.21.15.42:0/1522315207 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8a8095ce0 0x7fe8a812bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.769+0000 7fe8aeffd640 1 --2- 172.21.15.42:0/1522315207 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8a8094c60 0x7fe8a812b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.769+0000 7fe8af7fe640 1 -- 172.21.15.42:0/1522315207 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8a8094c60 msgr2=0x7fe8a812b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:19.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.769+0000 7fe8af7fe640 1 --2- 172.21.15.42:0/1522315207 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8a8094c60 0x7fe8a812b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.769+0000 7fe8af7fe640 1 -- 172.21.15.42:0/1522315207 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8a8095ce0 msgr2=0x7fe8a812bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:19.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.769+0000 7fe8af7fe640 1 --2- 172.21.15.42:0/1522315207 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8a8095ce0 0x7fe8a812bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.769+0000 7fe8af7fe640 1 -- 172.21.15.42:0/1522315207 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a812cf00 con 0x7fe8a8096680 2024-07-26T06:53:19.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.769+0000 7fe8af7fe640 1 --2- 172.21.15.42:0/1522315207 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8a8096680 0x7fe8a812c6e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0052340 tx=0x7fe8b007c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:19.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.770+0000 7fe897fff640 1 -- 172.21.15.42:0/1522315207 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b006c170 con 0x7fe8a8096680 2024-07-26T06:53:19.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.770+0000 7fe897fff640 1 -- 172.21.15.42:0/1522315207 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8b0083030 con 0x7fe8a8096680 2024-07-26T06:53:19.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.770+0000 7fe8affff640 1 -- 172.21.15.42:0/1522315207 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a812d190 con 0x7fe8a8096680 2024-07-26T06:53:19.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.770+0000 7fe897fff640 1 -- 172.21.15.42:0/1522315207 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b007f7d0 con 0x7fe8a8096680 2024-07-26T06:53:19.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.770+0000 7fe8affff640 1 -- 172.21.15.42:0/1522315207 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a8004e90 con 0x7fe8a8096680 2024-07-26T06:53:19.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.773+0000 7fe8bc23b740 1 Processor -- start 2024-07-26T06:53:19.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.773+0000 7fe897fff640 1 -- 172.21.15.42:0/1522315207 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fe8b0069ac0 con 0x7fe8a8096680 2024-07-26T06:53:19.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8bc23b740 1 -- start start 2024-07-26T06:53:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8bc23b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4fcb1b0 0x55d7b536dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8bc23b740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fa22f0 0x55d7b536e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8bc23b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4faa4d0 0x55d7b537fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8bc23b740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55d7b52e33b0 con 0x55d7b4fa22f0 2024-07-26T06:53:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8bc23b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d7b52d28b0 con 0x55d7b4fcb1b0 2024-07-26T06:53:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8bc23b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55d7b52d68f0 con 0x55d7b4faa4d0 2024-07-26T06:53:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8aeffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fa22f0 0x55d7b536e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8ae7fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4fcb1b0 0x55d7b536dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8aeffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fa22f0 0x55d7b536e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54090/0 (socket says 172.21.15.42:54090) 2024-07-26T06:53:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8ae7fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4fcb1b0 0x55d7b536dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:45924/0 (socket says 172.21.15.42:45924) 2024-07-26T06:53:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8aeffd640 1 -- 172.21.15.42:0/3365865163 learned_addr learned my addr 172.21.15.42:0/3365865163 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8ae7fc640 1 --2- 172.21.15.42:0/3365865163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4faa4d0 0x55d7b537fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe897fff640 1 --2- 172.21.15.42:0/1522315207 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8980b3c50 0x7fe8980b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8ae7fc640 1 -- 172.21.15.42:0/3365865163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4faa4d0 msgr2=0x55d7b537fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8ae7fc640 1 --2- 172.21.15.42:0/3365865163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4faa4d0 0x55d7b537fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8ae7fc640 1 -- 172.21.15.42:0/3365865163 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fa22f0 msgr2=0x55d7b536e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8ae7fc640 1 --2- 172.21.15.42:0/3365865163 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fa22f0 0x55d7b536e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8ae7fc640 1 -- 172.21.15.42:0/3365865163 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7b52799a0 con 0x55d7b4fcb1b0 2024-07-26T06:53:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8aeffd640 1 --2- 172.21.15.42:0/3365865163 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fa22f0 0x55d7b536e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.774+0000 7fe8af7fe640 1 --2- 172.21.15.42:0/1522315207 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8980b3c50 0x7fe8980b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.775+0000 7fe897fff640 1 -- 172.21.15.42:0/1522315207 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fe8b0150cc0 con 0x7fe8a8096680 2024-07-26T06:53:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.775+0000 7fe8ae7fc640 1 --2- 172.21.15.42:0/3365865163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4fcb1b0 0x55d7b536dbb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4005ab0 tx=0x7fe8a4009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fccfcaa968c926f4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.775+0000 7fe8957fa640 1 -- 172.21.15.42:0/3365865163 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4035040 con 0x55d7b4fcb1b0 2024-07-26T06:53:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.775+0000 7fe8957fa640 1 -- 172.21.15.42:0/3365865163 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8a4032b90 con 0x55d7b4fcb1b0 2024-07-26T06:53:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.775+0000 7fe8957fa640 1 -- 172.21.15.42:0/3365865163 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4003a90 con 0x55d7b4fcb1b0 2024-07-26T06:53:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.775+0000 7fe8af7fe640 1 --2- 172.21.15.42:0/1522315207 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8980b3c50 0x7fe8980b6160 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0051740 tx=0x7fe8b008bd10 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.776+0000 7fe8bc23b740 1 -- 172.21.15.42:0/3365865163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4fcb1b0 msgr2=0x55d7b536dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.776+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/3365865163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4fcb1b0 0x55d7b536dbb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4005ab0 tx=0x7fe8a4009560 comp rx=0 tx=0).stop 2024-07-26T06:53:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.776+0000 7fe8bc23b740 1 -- 172.21.15.42:0/3365865163 shutdown_connections 2024-07-26T06:53:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.776+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/3365865163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4faa4d0 0x55d7b537fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.776+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/3365865163 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fa22f0 0x55d7b536e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.776+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/3365865163 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4fcb1b0 0x55d7b536dbb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.776+0000 7fe8bc23b740 1 -- 172.21.15.42:0/3365865163 >> 172.21.15.42:0/3365865163 conn(0x55d7b4fbf300 msgr2=0x55d7b536b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.776+0000 7fe8bc23b740 1 -- 172.21.15.42:0/3365865163 shutdown_connections 2024-07-26T06:53:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.776+0000 7fe8bc23b740 1 -- 172.21.15.42:0/3365865163 wait complete. 2024-07-26T06:53:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.776+0000 7fe8bc23b740 1 -- start start 2024-07-26T06:53:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.776+0000 7fe8bc23b740 1 Processor -- start 2024-07-26T06:53:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8bc23b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4fa22f0 0x55d7b5381ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8bc23b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4faa4d0 0x55d7b5382580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8bc23b740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 0x55d7b5383080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8bc23b740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55d7b520cab0 con 0x55d7b4fcb1b0 2024-07-26T06:53:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8bc23b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d7b525acf0 con 0x55d7b4fa22f0 2024-07-26T06:53:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8bc23b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55d7b5380240 con 0x55d7b4faa4d0 2024-07-26T06:53:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8bc23b740 10 client.0 _ll_register_callbacks cb 0x55d7b52cc3e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:53:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8ae7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 0x55d7b5383080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8ae7fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4fa22f0 0x55d7b5381ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8ae7fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4fa22f0 0x55d7b5381ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:45938/0 (socket says 172.21.15.42:45938) 2024-07-26T06:53:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8ae7fc640 1 -- 172.21.15.42:0/1474971366 learned_addr learned my addr 172.21.15.42:0/1474971366 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8aeffd640 1 --2- 172.21.15.42:0/1474971366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4faa4d0 0x55d7b5382580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8ae7fc640 1 -- 172.21.15.42:0/1474971366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4faa4d0 msgr2=0x55d7b5382580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8ae7fc640 1 --2- 172.21.15.42:0/1474971366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4faa4d0 0x55d7b5382580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8ae7fc640 1 -- 172.21.15.42:0/1474971366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 msgr2=0x55d7b5383080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8ae7fc640 1 --2- 172.21.15.42:0/1474971366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 0x55d7b5383080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8ae7fc640 1 -- 172.21.15.42:0/1474971366 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7b5036790 con 0x55d7b4fa22f0 2024-07-26T06:53:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe8ae7fc640 1 --2- 172.21.15.42:0/1474971366 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4fa22f0 0x55d7b5381ec0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4009a20 tx=0x7fe8a4031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6b66c934a4df7cf server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.777+0000 7fe892ff5640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:53:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.778+0000 7fe892ff5640 1 -- 172.21.15.42:0/1474971366 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a404e030 con 0x55d7b4fa22f0 2024-07-26T06:53:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.778+0000 7fe892ff5640 1 -- 172.21.15.42:0/1474971366 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fe8a404f070 con 0x55d7b4fa22f0 2024-07-26T06:53:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.778+0000 7fe8bc23b740 1 -- 172.21.15.42:0/1474971366 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8a4032b90 con 0x55d7b4fa22f0 2024-07-26T06:53:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.778+0000 7fe892ff5640 1 -- 172.21.15.42:0/1474971366 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fe8a4035070 con 0x55d7b4fa22f0 2024-07-26T06:53:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.778+0000 7fe8bc23b740 10 client.25367 fetch_fsmap learned FSMap version 36 2024-07-26T06:53:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.778+0000 7fe8bc23b740 1 -- 172.21.15.42:0/1474971366 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55d7b50af620 con 0x55d7b4fa22f0 2024-07-26T06:53:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.778+0000 7fe892ff5640 1 -- 172.21.15.42:0/1474971366 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8a4032b90 con 0x55d7b4fa22f0 2024-07-26T06:53:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.778+0000 7fe892ff5640 1 -- 172.21.15.42:0/1474971366 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7fe8a4059820 con 0x55d7b4fa22f0 2024-07-26T06:53:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.778+0000 7fe8bc23b740 10 client.25367 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:53:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.778+0000 7fe8bc23b740 10 client.25367 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:53:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.778+0000 7fe8bc23b740 20 client.25367 populate_metadata read hostname 'smithi042' 2024-07-26T06:53:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.778+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/1474971366 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55d7b4f89d80 0x55d7b538f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.778+0000 7fe8bc23b740 4 client.25367 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:53:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.778+0000 7fe8bc23b740 1 -- 172.21.15.42:0/1474971366 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7b52e5300 con 0x55d7b4f89d80 2024-07-26T06:53:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.779+0000 7fe8aeffd640 1 --2- 172.21.15.42:0/1474971366 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55d7b4f89d80 0x55d7b538f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.779+0000 7fe8aeffd640 1 --2- 172.21.15.42:0/1474971366 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55d7b4f89d80 0x55d7b538f160 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86ccc5ee407c569c server_cookie=b0e0b9751fdf3d71 in_seq=0 out_seq=0 2024-07-26T06:53:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.779+0000 7fe892ff5640 10 client.25367 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.780+0000 7fe892ff5640 1 -- 172.21.15.42:0/1474971366 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d7b52e5300 con 0x55d7b4f89d80 2024-07-26T06:53:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.780+0000 7fe892ff5640 10 client.25367 handle_command_reply: tid=0 2024-07-26T06:53:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.780+0000 7fe8bc23b740 1 client.25367 shutdown 2024-07-26T06:53:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.780+0000 7fe8bc23b740 1 -- 172.21.15.42:0/1474971366 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4fa22f0 msgr2=0x55d7b5381ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.780+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/1474971366 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4fa22f0 0x55d7b5381ec0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4009a20 tx=0x7fe8a4031400 comp rx=0 tx=0).stop 2024-07-26T06:53:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.780+0000 7fe8bc23b740 1 -- 172.21.15.42:0/1474971366 shutdown_connections 2024-07-26T06:53:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.780+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/1474971366 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55d7b4f89d80 0x55d7b538f160 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.780+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/1474971366 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 0x55d7b5383080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.780+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/1474971366 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4faa4d0 0x55d7b5382580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.780+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/1474971366 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4fa22f0 0x55d7b5381ec0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.780+0000 7fe892ff5640 0 client.25367 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:19.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.780+0000 7fe8bc23b740 1 -- 172.21.15.42:0/1474971366 >> 172.21.15.42:0/1474971366 conn(0x55d7b4fbf300 msgr2=0x55d7b536c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:19.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.781+0000 7fe8bc23b740 1 -- 172.21.15.42:0/1474971366 shutdown_connections 2024-07-26T06:53:19.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.781+0000 7fe8bc23b740 1 -- 172.21.15.42:0/1474971366 wait complete. 2024-07-26T06:53:19.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.781+0000 7fe8bc23b740 20 client.25367 trim_cache size 0 max 16384 2024-07-26T06:53:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.797+0000 7fe8bc23b740 1 Processor -- start 2024-07-26T06:53:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.797+0000 7fe8bc23b740 1 -- start start 2024-07-26T06:53:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.797+0000 7fe8bc23b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4fa22f0 0x55d7b537ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.797+0000 7fe8bc23b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4faa4d0 0x55d7b536e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.797+0000 7fe8bc23b740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 0x55d7b536c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.797+0000 7fe8bc23b740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a813b410 con 0x55d7b4fcb1b0 2024-07-26T06:53:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.797+0000 7fe8bc23b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a813d040 con 0x55d7b4faa4d0 2024-07-26T06:53:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.797+0000 7fe8bc23b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a813bb10 con 0x55d7b4fa22f0 2024-07-26T06:53:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.797+0000 7fe8ae7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 0x55d7b536c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.797+0000 7fe8aeffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4faa4d0 0x55d7b536e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.797+0000 7fe8aeffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4faa4d0 0x55d7b536e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:45948/0 (socket says 172.21.15.42:45948) 2024-07-26T06:53:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.797+0000 7fe8aeffd640 1 -- 172.21.15.42:0/1847564075 learned_addr learned my addr 172.21.15.42:0/1847564075 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.797+0000 7fe8ae7fc640 1 --2- 172.21.15.42:0/1847564075 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4fa22f0 0x55d7b537ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.797+0000 7fe8aeffd640 1 -- 172.21.15.42:0/1847564075 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4fa22f0 msgr2=0x55d7b537ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.797+0000 7fe8aeffd640 1 --2- 172.21.15.42:0/1847564075 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4fa22f0 0x55d7b537ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.798+0000 7fe8aeffd640 1 -- 172.21.15.42:0/1847564075 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 msgr2=0x55d7b536c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.798+0000 7fe8aeffd640 1 --2- 172.21.15.42:0/1847564075 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 0x55d7b536c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.798+0000 7fe8aeffd640 1 -- 172.21.15.42:0/1847564075 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7b537c280 con 0x55d7b4faa4d0 2024-07-26T06:53:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.798+0000 7fe8ae7fc640 1 --2- 172.21.15.42:0/1847564075 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 0x55d7b536c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.798+0000 7fe8aeffd640 1 --2- 172.21.15.42:0/1847564075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4faa4d0 0x55d7b536e490 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x55d7b538f7e0 tx=0x7fe8a000b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbb74ef2d0669d1b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.798+0000 7fe894ff9640 1 -- 172.21.15.42:0/1847564075 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a0004220 con 0x55d7b4faa4d0 2024-07-26T06:53:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.798+0000 7fe894ff9640 1 -- 172.21.15.42:0/1847564075 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8a00043c0 con 0x55d7b4faa4d0 2024-07-26T06:53:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.798+0000 7fe894ff9640 1 -- 172.21.15.42:0/1847564075 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a001da00 con 0x55d7b4faa4d0 2024-07-26T06:53:19.804 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.799+0000 7fe8bc23b740 1 -- 172.21.15.42:0/1847564075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4faa4d0 msgr2=0x55d7b536e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.799+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/1847564075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4faa4d0 0x55d7b536e490 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x55d7b538f7e0 tx=0x7fe8a000b410 comp rx=0 tx=0).stop 2024-07-26T06:53:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.799+0000 7fe8bc23b740 1 -- 172.21.15.42:0/1847564075 shutdown_connections 2024-07-26T06:53:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.799+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/1847564075 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 0x55d7b536c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.799+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/1847564075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4faa4d0 0x55d7b536e490 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.799+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/1847564075 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4fa22f0 0x55d7b537ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.799+0000 7fe8bc23b740 1 -- 172.21.15.42:0/1847564075 >> 172.21.15.42:0/1847564075 conn(0x55d7b4fbf300 msgr2=0x55d7b536bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.799+0000 7fe8bc23b740 1 -- 172.21.15.42:0/1847564075 shutdown_connections 2024-07-26T06:53:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.799+0000 7fe8bc23b740 1 -- 172.21.15.42:0/1847564075 wait complete. 2024-07-26T06:53:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.799+0000 7fe8bc23b740 1 -- start start 2024-07-26T06:53:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.799+0000 7fe8bc23b740 1 Processor -- start 2024-07-26T06:53:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8bc23b740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4fa22f0 0x55d7b5389880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.805 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8bc23b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4faa4d0 0x55d7b5389f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8bc23b740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 0x55d7b53882c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8bc23b740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a813c970 con 0x55d7b4fcb1b0 2024-07-26T06:53:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8bc23b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a813c1e0 con 0x55d7b4faa4d0 2024-07-26T06:53:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8bc23b740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a81393f0 con 0x55d7b4fa22f0 2024-07-26T06:53:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8bc23b740 10 client.0 _ll_register_callbacks cb 0x7fe8a000b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:53:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8ae7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 0x55d7b53882c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8ae7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 0x55d7b53882c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:54110/0 (socket says 172.21.15.42:54110) 2024-07-26T06:53:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8ae7fc640 1 -- 172.21.15.42:0/882053603 learned_addr learned my addr 172.21.15.42:0/882053603 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8aeffd640 1 --2- 172.21.15.42:0/882053603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4faa4d0 0x55d7b5389f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8ae7fc640 1 --2- 172.21.15.42:0/882053603 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4fa22f0 0x55d7b5389880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8ae7fc640 1 -- 172.21.15.42:0/882053603 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4fa22f0 msgr2=0x55d7b5389880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8ae7fc640 1 --2- 172.21.15.42:0/882053603 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4fa22f0 0x55d7b5389880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8ae7fc640 1 -- 172.21.15.42:0/882053603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4faa4d0 msgr2=0x55d7b5389f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8ae7fc640 1 --2- 172.21.15.42:0/882053603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4faa4d0 0x55d7b5389f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8ae7fc640 1 -- 172.21.15.42:0/882053603 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7b5388a50 con 0x55d7b4fcb1b0 2024-07-26T06:53:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe8ae7fc640 1 --2- 172.21.15.42:0/882053603 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 0x55d7b53882c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe8a402a450 tx=0x7fe8a4042040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cdc9cd85c08d166 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe892ff5640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:53:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.800+0000 7fe892ff5640 1 -- 172.21.15.42:0/882053603 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a404e040 con 0x55d7b4fcb1b0 2024-07-26T06:53:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.801+0000 7fe8bc23b740 1 -- 172.21.15.42:0/882053603 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8a00043c0 con 0x55d7b4fcb1b0 2024-07-26T06:53:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.801+0000 7fe892ff5640 1 -- 172.21.15.42:0/882053603 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fe8a4059890 con 0x55d7b4fcb1b0 2024-07-26T06:53:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.801+0000 7fe892ff5640 1 -- 172.21.15.42:0/882053603 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fe8a4059e60 con 0x55d7b4fcb1b0 2024-07-26T06:53:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.801+0000 7fe8bc23b740 10 client.15774 fetch_fsmap learned FSMap version 36 2024-07-26T06:53:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.801+0000 7fe8bc23b740 1 -- 172.21.15.42:0/882053603 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55d7b53f6370 con 0x55d7b4fcb1b0 2024-07-26T06:53:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.801+0000 7fe892ff5640 1 -- 172.21.15.42:0/882053603 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8a00043c0 con 0x55d7b4fcb1b0 2024-07-26T06:53:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.801+0000 7fe892ff5640 1 -- 172.21.15.42:0/882053603 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7fe8a403d5c0 con 0x55d7b4fcb1b0 2024-07-26T06:53:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.801+0000 7fe8bc23b740 10 client.15774 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:53:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.801+0000 7fe8bc23b740 10 client.15774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:53:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.801+0000 7fe8bc23b740 20 client.15774 populate_metadata read hostname 'smithi042' 2024-07-26T06:53:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.801+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/882053603 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55d7b4f5f1c0 0x55d7b53f8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.801+0000 7fe8bc23b740 4 client.15774 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:53:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.801+0000 7fe8bc23b740 1 -- 172.21.15.42:0/882053603 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d7b52e0da0 con 0x55d7b4f5f1c0 2024-07-26T06:53:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.801+0000 7fe8aeffd640 1 --2- 172.21.15.42:0/882053603 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55d7b4f5f1c0 0x55d7b53f8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.802+0000 7fe8aeffd640 1 --2- 172.21.15.42:0/882053603 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55d7b4f5f1c0 0x55d7b53f8ba0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f9c96424427d8cd server_cookie=a63fdecabd7abdc1 in_seq=0 out_seq=0 2024-07-26T06:53:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.802+0000 7fe892ff5640 10 client.15774 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:19.837 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.836+0000 7fe892ff5640 1 -- 172.21.15.42:0/882053603 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d7b52e0da0 con 0x55d7b4f5f1c0 2024-07-26T06:53:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.836+0000 7fe892ff5640 10 client.15774 handle_command_reply: tid=0 2024-07-26T06:53:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.836+0000 7fe8bc23b740 1 client.15774 shutdown 2024-07-26T06:53:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.836+0000 7fe8bc23b740 1 -- 172.21.15.42:0/882053603 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 msgr2=0x55d7b53882c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.836+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/882053603 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 0x55d7b53882c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe8a402a450 tx=0x7fe8a4042040 comp rx=0 tx=0).stop 2024-07-26T06:53:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.836+0000 7fe8bc23b740 1 -- 172.21.15.42:0/882053603 shutdown_connections 2024-07-26T06:53:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.836+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/882053603 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55d7b4f5f1c0 0x55d7b53f8ba0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.836+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/882053603 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55d7b4fcb1b0 0x55d7b53882c0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.836+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/882053603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d7b4faa4d0 0x55d7b5389f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.836+0000 7fe8bc23b740 1 --2- 172.21.15.42:0/882053603 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55d7b4fa22f0 0x55d7b5389880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.836+0000 7fe892ff5640 0 client.15774 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:19.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.836+0000 7fe8bc23b740 1 -- 172.21.15.42:0/882053603 >> 172.21.15.42:0/882053603 conn(0x55d7b4fbf300 msgr2=0x55d7b536e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.836+0000 7fe8bc23b740 1 -- 172.21.15.42:0/882053603 shutdown_connections 2024-07-26T06:53:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.836+0000 7fe8bc23b740 1 -- 172.21.15.42:0/882053603 wait complete. 2024-07-26T06:53:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.837+0000 7fe8bc23b740 20 client.15774 trim_cache size 0 max 16384 2024-07-26T06:53:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:53:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:53:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:53:19.839 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:53:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.837+0000 7fe8affff640 1 -- 172.21.15.42:0/1522315207 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8980b3c50 msgr2=0x7fe8980b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.837+0000 7fe8affff640 1 --2- 172.21.15.42:0/1522315207 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8980b3c50 0x7fe8980b6160 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0051740 tx=0x7fe8b008bd10 comp rx=0 tx=0).stop 2024-07-26T06:53:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.837+0000 7fe8affff640 1 -- 172.21.15.42:0/1522315207 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8a8096680 msgr2=0x7fe8a812c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.837+0000 7fe8affff640 1 --2- 172.21.15.42:0/1522315207 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8a8096680 0x7fe8a812c6e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0052340 tx=0x7fe8b007c320 comp rx=0 tx=0).stop 2024-07-26T06:53:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.837+0000 7fe8affff640 1 -- 172.21.15.42:0/1522315207 shutdown_connections 2024-07-26T06:53:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.837+0000 7fe8affff640 1 --2- 172.21.15.42:0/1522315207 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe8980b3c50 0x7fe8980b6160 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.837+0000 7fe8affff640 1 --2- 172.21.15.42:0/1522315207 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8a8096680 0x7fe8a812c6e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.837+0000 7fe8affff640 1 --2- 172.21.15.42:0/1522315207 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe8a8095ce0 0x7fe8a812bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.837+0000 7fe8affff640 1 --2- 172.21.15.42:0/1522315207 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8a8094c60 0x7fe8a812b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.837+0000 7fe8affff640 1 -- 172.21.15.42:0/1522315207 >> 172.21.15.42:0/1522315207 conn(0x7fe8a80905a0 msgr2=0x7fe8a8098fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.837+0000 7fe8affff640 1 -- 172.21.15.42:0/1522315207 shutdown_connections 2024-07-26T06:53:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:19.837+0000 7fe8affff640 1 -- 172.21.15.42:0/1522315207 wait complete. 2024-07-26T06:53:20.717 INFO:tasks.workunit.client.0.smithi042.stdout:copying to largest ever: 0xffd97 2024-07-26T06:53:20.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:20 smithi042 ceph-mon[32775]: pgmap v490: 97 pgs: 97 active+clean; 41 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 660 op/s 2024-07-26T06:53:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:20 smithi164 ceph-mon[36970]: pgmap v490: 97 pgs: 97 active+clean; 41 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 660 op/s 2024-07-26T06:53:21.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:20 smithi163 ceph-mon[36873]: pgmap v490: 97 pgs: 97 active+clean; 41 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 660 op/s 2024-07-26T06:53:22.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:22 smithi042 ceph-mon[32775]: pgmap v491: 97 pgs: 97 active+clean; 44 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.3 MiB/s wr, 794 op/s 2024-07-26T06:53:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:22 smithi164 ceph-mon[36970]: pgmap v491: 97 pgs: 97 active+clean; 44 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.3 MiB/s wr, 794 op/s 2024-07-26T06:53:23.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:22 smithi163 ceph-mon[36873]: pgmap v491: 97 pgs: 97 active+clean; 44 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.3 MiB/s wr, 794 op/s 2024-07-26T06:53:24.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:24 smithi042 ceph-mon[32775]: pgmap v492: 97 pgs: 97 active+clean; 45 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.9 MiB/s wr, 614 op/s 2024-07-26T06:53:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:24 smithi164 ceph-mon[36970]: pgmap v492: 97 pgs: 97 active+clean; 45 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.9 MiB/s wr, 614 op/s 2024-07-26T06:53:25.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:24 smithi163 ceph-mon[36873]: pgmap v492: 97 pgs: 97 active+clean; 45 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.9 MiB/s wr, 614 op/s 2024-07-26T06:53:26.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:26 smithi042 ceph-mon[32775]: pgmap v493: 97 pgs: 97 active+clean; 39 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.0 MiB/s wr, 629 op/s 2024-07-26T06:53:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:26 smithi164 ceph-mon[36970]: pgmap v493: 97 pgs: 97 active+clean; 39 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.0 MiB/s wr, 629 op/s 2024-07-26T06:53:27.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:26 smithi163 ceph-mon[36873]: pgmap v493: 97 pgs: 97 active+clean; 39 MiB data, 714 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.0 MiB/s wr, 629 op/s 2024-07-26T06:53:28.380 INFO:tasks.workunit.client.0.smithi042.stdout:copying to largest ever: 0xffdb8 2024-07-26T06:53:28.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:28 smithi042 ceph-mon[32775]: pgmap v494: 97 pgs: 97 active+clean; 41 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 5.2 MiB/s wr, 741 op/s 2024-07-26T06:53:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:28 smithi164 ceph-mon[36970]: pgmap v494: 97 pgs: 97 active+clean; 41 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 5.2 MiB/s wr, 741 op/s 2024-07-26T06:53:29.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:28 smithi163 ceph-mon[36873]: pgmap v494: 97 pgs: 97 active+clean; 41 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 5.2 MiB/s wr, 741 op/s 2024-07-26T06:53:30.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:30 smithi042 ceph-mon[32775]: pgmap v495: 97 pgs: 97 active+clean; 44 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.0 MiB/s wr, 625 op/s 2024-07-26T06:53:30.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:30 smithi164 ceph-mon[36970]: pgmap v495: 97 pgs: 97 active+clean; 44 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.0 MiB/s wr, 625 op/s 2024-07-26T06:53:31.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:30 smithi163 ceph-mon[36873]: pgmap v495: 97 pgs: 97 active+clean; 44 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.0 MiB/s wr, 625 op/s 2024-07-26T06:53:32.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:32 smithi042 ceph-mon[32775]: pgmap v496: 97 pgs: 97 active+clean; 45 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.1 MiB/s wr, 736 op/s 2024-07-26T06:53:32.950 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.949+0000 7f454c30e640 1 -- 172.21.15.42:0/413606272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4544101fe0 msgr2=0x7f4544102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:32.950 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.949+0000 7f454c30e640 1 --2- 172.21.15.42:0/413606272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4544101fe0 0x7f4544102450 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4540009900 tx=0x7f454002a410 comp rx=0 tx=0).stop 2024-07-26T06:53:32.950 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.949+0000 7f454c30e640 1 -- 172.21.15.42:0/413606272 shutdown_connections 2024-07-26T06:53:32.950 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.949+0000 7f454c30e640 1 --2- 172.21.15.42:0/413606272 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4544103a00 0x7f454410a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.950 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.949+0000 7f454c30e640 1 --2- 172.21.15.42:0/413606272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4544103060 0x7f4544103470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.950 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.949+0000 7f454c30e640 1 --2- 172.21.15.42:0/413606272 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4544101fe0 0x7f4544102450 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.950 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.949+0000 7f454c30e640 1 -- 172.21.15.42:0/413606272 >> 172.21.15.42:0/413606272 conn(0x7f45440fd960 msgr2=0x7f45440ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:32.950 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.949+0000 7f454c30e640 1 -- 172.21.15.42:0/413606272 shutdown_connections 2024-07-26T06:53:32.951 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.949+0000 7f454c30e640 1 -- 172.21.15.42:0/413606272 wait complete. 2024-07-26T06:53:32.952 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.950+0000 7f454c30e640 1 Processor -- start 2024-07-26T06:53:32.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.950+0000 7f454c30e640 1 -- start start 2024-07-26T06:53:32.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.950+0000 7f454c30e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4544101fe0 0x7f4544105bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.950+0000 7f454c30e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4544103060 0x7f4544104490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.950+0000 7f454c30e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4544103a00 0x7f45441049d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.950+0000 7f454c30e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f454410dc60 con 0x7f4544103060 2024-07-26T06:53:32.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.950+0000 7f454c30e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f454410dae0 con 0x7f4544103a00 2024-07-26T06:53:32.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.950+0000 7f454c30e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f454410dde0 con 0x7f4544101fe0 2024-07-26T06:53:32.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.950+0000 7f45498a9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4544103060 0x7f4544104490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.950+0000 7f454a0aa640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4544101fe0 0x7f4544105bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.950+0000 7f45498a9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4544103060 0x7f4544104490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50694/0 (socket says 172.21.15.42:50694) 2024-07-26T06:53:32.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.950+0000 7f45498a9640 1 -- 172.21.15.42:0/1322847034 learned_addr learned my addr 172.21.15.42:0/1322847034 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.950+0000 7f454a8ab640 1 --2- 172.21.15.42:0/1322847034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4544103a00 0x7f45441049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.951+0000 7f45498a9640 1 -- 172.21.15.42:0/1322847034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4544101fe0 msgr2=0x7f4544105bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.951+0000 7f45498a9640 1 --2- 172.21.15.42:0/1322847034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4544101fe0 0x7f4544105bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.951+0000 7f45498a9640 1 -- 172.21.15.42:0/1322847034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4544103a00 msgr2=0x7f45441049d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.951+0000 7f45498a9640 1 --2- 172.21.15.42:0/1322847034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4544103a00 0x7f45441049d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.951+0000 7f45498a9640 1 -- 172.21.15.42:0/1322847034 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4544105100 con 0x7f4544103060 2024-07-26T06:53:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.951+0000 7f45498a9640 1 --2- 172.21.15.42:0/1322847034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4544103060 0x7f4544104490 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f453400e9f0 tx=0x7f453400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.951+0000 7f453b7fe640 1 -- 172.21.15.42:0/1322847034 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453400cd80 con 0x7f4544103060 2024-07-26T06:53:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.951+0000 7f453b7fe640 1 -- 172.21.15.42:0/1322847034 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45340054d0 con 0x7f4544103060 2024-07-26T06:53:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.951+0000 7f454c30e640 1 -- 172.21.15.42:0/1322847034 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45441a3a20 con 0x7f4544103060 2024-07-26T06:53:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.951+0000 7f453b7fe640 1 -- 172.21.15.42:0/1322847034 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4534010690 con 0x7f4544103060 2024-07-26T06:53:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.951+0000 7f454c30e640 1 -- 172.21.15.42:0/1322847034 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45441a3ee0 con 0x7f4544103060 2024-07-26T06:53:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.954+0000 7f4553225740 1 Processor -- start 2024-07-26T06:53:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.954+0000 7f4553225740 1 -- start start 2024-07-26T06:53:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.954+0000 7f453b7fe640 1 -- 172.21.15.42:0/1322847034 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f4534010830 con 0x7f4544103060 2024-07-26T06:53:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.954+0000 7f4553225740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a18041b0 0x5595a1ba8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.954+0000 7f4553225740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17db2f0 0x5595a1ba91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.954+0000 7f4553225740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a17e34d0 0x5595a1ba9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.954+0000 7f4553225740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5595a1b1c3b0 con 0x5595a17e34d0 2024-07-26T06:53:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.954+0000 7f4553225740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5595a1b0b8b0 con 0x5595a18041b0 2024-07-26T06:53:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.954+0000 7f4553225740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5595a1b0f8f0 con 0x5595a17db2f0 2024-07-26T06:53:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.955+0000 7f454a8ab640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a18041b0 0x5595a1ba8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.955+0000 7f454a8ab640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a18041b0 0x5595a1ba8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:57726/0 (socket says 172.21.15.42:57726) 2024-07-26T06:53:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.955+0000 7f45498a9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a17e34d0 0x5595a1ba9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.955+0000 7f454a8ab640 1 -- 172.21.15.42:0/1015717439 learned_addr learned my addr 172.21.15.42:0/1015717439 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.955+0000 7f454a0aa640 1 --2- 172.21.15.42:0/1015717439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17db2f0 0x5595a1ba91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.955+0000 7f454a8ab640 1 -- 172.21.15.42:0/1015717439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17db2f0 msgr2=0x5595a1ba91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.955+0000 7f454a8ab640 1 --2- 172.21.15.42:0/1015717439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17db2f0 0x5595a1ba91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.955+0000 7f454a8ab640 1 -- 172.21.15.42:0/1015717439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a17e34d0 msgr2=0x5595a1ba9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.955+0000 7f454a8ab640 1 --2- 172.21.15.42:0/1015717439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a17e34d0 0x5595a1ba9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.955+0000 7f454a8ab640 1 -- 172.21.15.42:0/1015717439 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595a1ab29a0 con 0x5595a18041b0 2024-07-26T06:53:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.955+0000 7f454a8ab640 1 --2- 172.21.15.42:0/1015717439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a18041b0 0x5595a1ba8b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f453c009fd0 tx=0x7f453c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=183e079c3fd3a342 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.956+0000 7f4538bfe640 1 -- 172.21.15.42:0/1015717439 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453c004d20 con 0x5595a18041b0 2024-07-26T06:53:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.956+0000 7f4538bfe640 1 -- 172.21.15.42:0/1015717439 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f453c004050 con 0x5595a18041b0 2024-07-26T06:53:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.956+0000 7f4538bfe640 1 -- 172.21.15.42:0/1015717439 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453c005280 con 0x5595a18041b0 2024-07-26T06:53:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.956+0000 7f453b7fe640 1 --2- 172.21.15.42:0/1322847034 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f45300b3c50 0x7f45300b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.956+0000 7f454a0aa640 1 --2- 172.21.15.42:0/1322847034 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f45300b3c50 0x7f45300b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.959 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.956+0000 7f453b7fe640 1 -- 172.21.15.42:0/1322847034 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f45340e3fb0 con 0x7f4544103060 2024-07-26T06:53:32.962 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.957+0000 7f454a0aa640 1 --2- 172.21.15.42:0/1322847034 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f45300b3c50 0x7f45300b6160 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4540000c00 tx=0x7f4540031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.957+0000 7f4553225740 1 -- 172.21.15.42:0/1015717439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a18041b0 msgr2=0x5595a1ba8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.957+0000 7f4553225740 1 --2- 172.21.15.42:0/1015717439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a18041b0 0x5595a1ba8b50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f453c009fd0 tx=0x7f453c0118d0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.957+0000 7f4553225740 1 -- 172.21.15.42:0/1015717439 shutdown_connections 2024-07-26T06:53:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.957+0000 7f4553225740 1 --2- 172.21.15.42:0/1015717439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a17e34d0 0x5595a1ba9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.957+0000 7f4553225740 1 --2- 172.21.15.42:0/1015717439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17db2f0 0x5595a1ba91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.957+0000 7f4553225740 1 --2- 172.21.15.42:0/1015717439 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a18041b0 0x5595a1ba8b50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.957+0000 7f4553225740 1 -- 172.21.15.42:0/1015717439 >> 172.21.15.42:0/1015717439 conn(0x5595a17f8300 msgr2=0x5595a1ba45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.957+0000 7f4553225740 1 -- 172.21.15.42:0/1015717439 shutdown_connections 2024-07-26T06:53:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.957+0000 7f4553225740 1 -- 172.21.15.42:0/1015717439 wait complete. 2024-07-26T06:53:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.957+0000 7f4553225740 1 -- start start 2024-07-26T06:53:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.957+0000 7f4553225740 1 Processor -- start 2024-07-26T06:53:32.963 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.957+0000 7f4553225740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a17db2f0 0x5595a1bb24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.964 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.957+0000 7f4553225740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17e34d0 0x5595a1bb2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.964 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f4553225740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a18041b0 0x5595a1bb3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.964 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f4553225740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5595a1a45ab0 con 0x5595a17db2f0 2024-07-26T06:53:32.964 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f4553225740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5595a1a93cf0 con 0x5595a18041b0 2024-07-26T06:53:32.964 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f4553225740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5595a1baa830 con 0x5595a17e34d0 2024-07-26T06:53:32.964 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f4553225740 10 client.0 _ll_register_callbacks cb 0x5595a1b053e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:53:32.964 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f454a0aa640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a17db2f0 0x5595a1bb24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.964 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f454a0aa640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a17db2f0 0x5595a1bb24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50714/0 (socket says 172.21.15.42:50714) 2024-07-26T06:53:32.964 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f454a0aa640 1 -- 172.21.15.42:0/3239805909 learned_addr learned my addr 172.21.15.42:0/3239805909 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:32.964 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f45498a9640 1 --2- 172.21.15.42:0/3239805909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17e34d0 0x5595a1bb2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.964 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f454a8ab640 1 --2- 172.21.15.42:0/3239805909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a18041b0 0x5595a1bb3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.964 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f454a0aa640 1 -- 172.21.15.42:0/3239805909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17e34d0 msgr2=0x5595a1bb2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:32.964 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f454a0aa640 1 --2- 172.21.15.42:0/3239805909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17e34d0 0x5595a1bb2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f454a0aa640 1 -- 172.21.15.42:0/3239805909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a18041b0 msgr2=0x5595a1bb3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:32.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f454a0aa640 1 --2- 172.21.15.42:0/3239805909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a18041b0 0x5595a1bb3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f454a0aa640 1 -- 172.21.15.42:0/3239805909 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595a186f790 con 0x5595a17db2f0 2024-07-26T06:53:32.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f454a8ab640 1 --2- 172.21.15.42:0/3239805909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a18041b0 0x5595a1bb3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:32.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f45498a9640 1 --2- 172.21.15.42:0/3239805909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17e34d0 0x5595a1bb2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:32.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f454a0aa640 1 --2- 172.21.15.42:0/3239805909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a17db2f0 0x5595a1bb24e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4540009900 tx=0x7f4540009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5498c5e7a17008ca server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:32.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f452dffb640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:53:32.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f452dffb640 1 -- 172.21.15.42:0/3239805909 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4540035040 con 0x5595a17db2f0 2024-07-26T06:53:32.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f4553225740 1 -- 172.21.15.42:0/3239805909 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f453c004050 con 0x5595a17db2f0 2024-07-26T06:53:32.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.958+0000 7f452dffb640 1 -- 172.21.15.42:0/3239805909 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f4540004300 con 0x5595a17db2f0 2024-07-26T06:53:32.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.959+0000 7f452dffb640 1 -- 172.21.15.42:0/3239805909 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f45400035f0 con 0x5595a17db2f0 2024-07-26T06:53:32.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.959+0000 7f4553225740 10 client.15786 fetch_fsmap learned FSMap version 36 2024-07-26T06:53:32.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.959+0000 7f4553225740 1 -- 172.21.15.42:0/3239805909 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5595a18e8620 con 0x5595a17db2f0 2024-07-26T06:53:32.966 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.959+0000 7f452dffb640 1 -- 172.21.15.42:0/3239805909 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f453c004050 con 0x5595a17db2f0 2024-07-26T06:53:32.966 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.959+0000 7f452dffb640 1 -- 172.21.15.42:0/3239805909 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f4540044070 con 0x5595a17db2f0 2024-07-26T06:53:32.966 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.959+0000 7f4553225740 10 client.15786 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:53:32.966 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.959+0000 7f4553225740 10 client.15786 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:53:32.966 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.959+0000 7f4553225740 20 client.15786 populate_metadata read hostname 'smithi042' 2024-07-26T06:53:32.966 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.959+0000 7f4553225740 1 --2- 172.21.15.42:0/3239805909 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5595a17c2d80 0x5595a1bc0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.966 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.959+0000 7f4553225740 4 client.15786 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:53:32.966 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.959+0000 7f4553225740 1 -- 172.21.15.42:0/3239805909 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595a1b1e300 con 0x5595a17c2d80 2024-07-26T06:53:32.966 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.960+0000 7f45498a9640 1 --2- 172.21.15.42:0/3239805909 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5595a17c2d80 0x5595a1bc0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.966 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.960+0000 7f452dffb640 10 client.15786 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:32.966 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.960+0000 7f45498a9640 1 --2- 172.21.15.42:0/3239805909 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5595a17c2d80 0x5595a1bc0230 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae7bab43b8660de server_cookie=accdc381be888660 in_seq=0 out_seq=0 2024-07-26T06:53:32.966 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.961+0000 7f452dffb640 1 -- 172.21.15.42:0/3239805909 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5595a1b1e300 con 0x5595a17c2d80 2024-07-26T06:53:32.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.961+0000 7f452dffb640 10 client.15786 handle_command_reply: tid=0 2024-07-26T06:53:32.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.961+0000 7f4553225740 1 client.15786 shutdown 2024-07-26T06:53:32.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.961+0000 7f4553225740 1 -- 172.21.15.42:0/3239805909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a17db2f0 msgr2=0x5595a1bb24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:32.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.961+0000 7f4553225740 1 --2- 172.21.15.42:0/3239805909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a17db2f0 0x5595a1bb24e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4540009900 tx=0x7f4540009260 comp rx=0 tx=0).stop 2024-07-26T06:53:32.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.961+0000 7f4553225740 1 -- 172.21.15.42:0/3239805909 shutdown_connections 2024-07-26T06:53:32.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.961+0000 7f4553225740 1 --2- 172.21.15.42:0/3239805909 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5595a17c2d80 0x5595a1bc0230 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.961+0000 7f4553225740 1 --2- 172.21.15.42:0/3239805909 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a18041b0 0x5595a1bb3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.961+0000 7f4553225740 1 --2- 172.21.15.42:0/3239805909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17e34d0 0x5595a1bb2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.961+0000 7f4553225740 1 --2- 172.21.15.42:0/3239805909 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a17db2f0 0x5595a1bb24e0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.961+0000 7f4553225740 1 -- 172.21.15.42:0/3239805909 >> 172.21.15.42:0/3239805909 conn(0x5595a17f8300 msgr2=0x5595a1ba2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:32.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.961+0000 7f452dffb640 0 client.15786 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:32.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.961+0000 7f4553225740 1 -- 172.21.15.42:0/3239805909 shutdown_connections 2024-07-26T06:53:32.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.962+0000 7f4553225740 1 -- 172.21.15.42:0/3239805909 wait complete. 2024-07-26T06:53:32.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.962+0000 7f4553225740 20 client.15786 trim_cache size 0 max 16384 2024-07-26T06:53:32.971 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:32 smithi164 ceph-mon[36970]: pgmap v496: 97 pgs: 97 active+clean; 45 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.1 MiB/s wr, 736 op/s 2024-07-26T06:53:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.978+0000 7f4553225740 1 Processor -- start 2024-07-26T06:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.978+0000 7f4553225740 1 -- start start 2024-07-26T06:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.979+0000 7f4553225740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17db2f0 0x5595a1ba99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.979+0000 7f4553225740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a17e34d0 0x5595a1ba8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.979+0000 7f4553225740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a18041b0 0x5595a1ba2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.979+0000 7f4553225740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f45441acc10 con 0x5595a18041b0 2024-07-26T06:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.979+0000 7f4553225740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f45441ae840 con 0x5595a17e34d0 2024-07-26T06:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.979+0000 7f4553225740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f45441ad310 con 0x5595a17db2f0 2024-07-26T06:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.979+0000 7f454a8ab640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a18041b0 0x5595a1ba2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.979+0000 7f454a8ab640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a18041b0 0x5595a1ba2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50724/0 (socket says 172.21.15.42:50724) 2024-07-26T06:53:32.990 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.979+0000 7f454a8ab640 1 -- 172.21.15.42:0/925158265 learned_addr learned my addr 172.21.15.42:0/925158265 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.979+0000 7f454a0aa640 1 --2- 172.21.15.42:0/925158265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17db2f0 0x5595a1ba99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.979+0000 7f45498a9640 1 --2- 172.21.15.42:0/925158265 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a17e34d0 0x5595a1ba8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.979+0000 7f454a0aa640 1 -- 172.21.15.42:0/925158265 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a17e34d0 msgr2=0x5595a1ba8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.979+0000 7f454a0aa640 1 --2- 172.21.15.42:0/925158265 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a17e34d0 0x5595a1ba8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.991 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.979+0000 7f454a0aa640 1 -- 172.21.15.42:0/925158265 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a18041b0 msgr2=0x5595a1ba2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.979+0000 7f454a0aa640 1 --2- 172.21.15.42:0/925158265 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a18041b0 0x5595a1ba2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.979+0000 7f454a0aa640 1 -- 172.21.15.42:0/925158265 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595a17c2fb0 con 0x5595a17db2f0 2024-07-26T06:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.979+0000 7f454a8ab640 1 --2- 172.21.15.42:0/925158265 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a18041b0 0x5595a1ba2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.980+0000 7f454a0aa640 1 --2- 172.21.15.42:0/925158265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17db2f0 0x5595a1ba99e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4540033000 tx=0x7f454002a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21275272b48f11e7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:32.992 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.980+0000 7f452ffff640 1 -- 172.21.15.42:0/925158265 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4540035040 con 0x5595a17db2f0 2024-07-26T06:53:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.980+0000 7f452ffff640 1 -- 172.21.15.42:0/925158265 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f454005d020 con 0x5595a17db2f0 2024-07-26T06:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.980+0000 7f452ffff640 1 -- 172.21.15.42:0/925158265 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4540003a80 con 0x5595a17db2f0 2024-07-26T06:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.981+0000 7f4553225740 1 -- 172.21.15.42:0/925158265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17db2f0 msgr2=0x5595a1ba99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.981+0000 7f4553225740 1 --2- 172.21.15.42:0/925158265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17db2f0 0x5595a1ba99e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4540033000 tx=0x7f454002a410 comp rx=0 tx=0).stop 2024-07-26T06:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.981+0000 7f4553225740 1 -- 172.21.15.42:0/925158265 shutdown_connections 2024-07-26T06:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.981+0000 7f4553225740 1 --2- 172.21.15.42:0/925158265 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a18041b0 0x5595a1ba2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.981+0000 7f4553225740 1 --2- 172.21.15.42:0/925158265 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a17e34d0 0x5595a1ba8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.981+0000 7f4553225740 1 --2- 172.21.15.42:0/925158265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17db2f0 0x5595a1ba99e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.981+0000 7f4553225740 1 -- 172.21.15.42:0/925158265 >> 172.21.15.42:0/925158265 conn(0x5595a17f8300 msgr2=0x5595a1bc0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.981+0000 7f4553225740 1 -- 172.21.15.42:0/925158265 shutdown_connections 2024-07-26T06:53:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.981+0000 7f4553225740 1 -- 172.21.15.42:0/925158265 wait complete. 2024-07-26T06:53:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.981+0000 7f4553225740 1 -- start start 2024-07-26T06:53:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.981+0000 7f4553225740 1 Processor -- start 2024-07-26T06:53:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f4553225740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17db2f0 0x5595a1bb9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f4553225740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a17e34d0 0x5595a1bb98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f4553225740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a18041b0 0x5595a1bb9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f4553225740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f45441ae170 con 0x5595a18041b0 2024-07-26T06:53:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f4553225740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f45441ad9e0 con 0x5595a17e34d0 2024-07-26T06:53:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f4553225740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f45441aabf0 con 0x5595a17db2f0 2024-07-26T06:53:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f4553225740 10 client.0 _ll_register_callbacks cb 0x7f4540009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:53:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f454a8ab640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a18041b0 0x5595a1bb9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f454a8ab640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a18041b0 0x5595a1bb9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50738/0 (socket says 172.21.15.42:50738) 2024-07-26T06:53:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f454a8ab640 1 -- 172.21.15.42:0/3473524047 learned_addr learned my addr 172.21.15.42:0/3473524047 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f454a0aa640 1 --2- 172.21.15.42:0/3473524047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17db2f0 0x5595a1bb9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f45498a9640 1 --2- 172.21.15.42:0/3473524047 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a17e34d0 0x5595a1bb98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f454a8ab640 1 -- 172.21.15.42:0/3473524047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17db2f0 msgr2=0x5595a1bb9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f454a8ab640 1 --2- 172.21.15.42:0/3473524047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17db2f0 0x5595a1bb9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f454a8ab640 1 -- 172.21.15.42:0/3473524047 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a17e34d0 msgr2=0x5595a1bb98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f454a8ab640 1 --2- 172.21.15.42:0/3473524047 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a17e34d0 0x5595a1bb98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f454a8ab640 1 -- 172.21.15.42:0/3473524047 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595a1bb8170 con 0x5595a18041b0 2024-07-26T06:53:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f454a0aa640 1 --2- 172.21.15.42:0/3473524047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17db2f0 0x5595a1bb9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f454a8ab640 1 --2- 172.21.15.42:0/3473524047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a18041b0 0x5595a1bb9e20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f453c009fd0 tx=0x7f453c009d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89672d530406eb7a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.982+0000 7f452dffb640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:53:32.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.983+0000 7f452dffb640 1 -- 172.21.15.42:0/3473524047 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453c0047d0 con 0x5595a18041b0 2024-07-26T06:53:32.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.983+0000 7f452dffb640 1 -- 172.21.15.42:0/3473524047 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f453c004260 con 0x5595a18041b0 2024-07-26T06:53:32.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.983+0000 7f4553225740 1 -- 172.21.15.42:0/3473524047 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f454005d020 con 0x5595a18041b0 2024-07-26T06:53:32.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.983+0000 7f452dffb640 1 -- 172.21.15.42:0/3473524047 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f453c004460 con 0x5595a18041b0 2024-07-26T06:53:32.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.983+0000 7f4553225740 10 client.15792 fetch_fsmap learned FSMap version 36 2024-07-26T06:53:32.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.983+0000 7f4553225740 1 -- 172.21.15.42:0/3473524047 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5595a1c26e90 con 0x5595a18041b0 2024-07-26T06:53:32.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.983+0000 7f452dffb640 1 -- 172.21.15.42:0/3473524047 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f454005d020 con 0x5595a18041b0 2024-07-26T06:53:32.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.983+0000 7f452dffb640 1 -- 172.21.15.42:0/3473524047 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f453c00f570 con 0x5595a18041b0 2024-07-26T06:53:32.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.983+0000 7f4553225740 10 client.15792 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:53:32.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.983+0000 7f4553225740 10 client.15792 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:53:32.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.983+0000 7f4553225740 20 client.15792 populate_metadata read hostname 'smithi042' 2024-07-26T06:53:32.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.983+0000 7f4553225740 1 --2- 172.21.15.42:0/3473524047 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5595a17981c0 0x5595a1c29730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:32.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.983+0000 7f4553225740 4 client.15792 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:53:32.998 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.983+0000 7f4553225740 1 -- 172.21.15.42:0/3473524047 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5595a1b19da0 con 0x5595a17981c0 2024-07-26T06:53:32.998 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.983+0000 7f454a0aa640 1 --2- 172.21.15.42:0/3473524047 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5595a17981c0 0x5595a1c29730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:32.998 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.984+0000 7f454a0aa640 1 --2- 172.21.15.42:0/3473524047 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5595a17981c0 0x5595a1c29730 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abbd1c696a36f059 server_cookie=d0c9384ed1fe1a7d in_seq=0 out_seq=0 2024-07-26T06:53:32.998 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:32.984+0000 7f452dffb640 10 client.15792 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:33.005 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.004+0000 7f452dffb640 1 -- 172.21.15.42:0/3473524047 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5595a1b19da0 con 0x5595a17981c0 2024-07-26T06:53:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.004+0000 7f452dffb640 10 client.15792 handle_command_reply: tid=0 2024-07-26T06:53:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.004+0000 7f4553225740 1 client.15792 shutdown 2024-07-26T06:53:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.005+0000 7f4553225740 1 -- 172.21.15.42:0/3473524047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a18041b0 msgr2=0x5595a1bb9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.005+0000 7f4553225740 1 --2- 172.21.15.42:0/3473524047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a18041b0 0x5595a1bb9e20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f453c009fd0 tx=0x7f453c009d60 comp rx=0 tx=0).stop 2024-07-26T06:53:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.005+0000 7f4553225740 1 -- 172.21.15.42:0/3473524047 shutdown_connections 2024-07-26T06:53:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.005+0000 7f4553225740 1 --2- 172.21.15.42:0/3473524047 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5595a17981c0 0x5595a1c29730 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.005+0000 7f4553225740 1 --2- 172.21.15.42:0/3473524047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5595a18041b0 0x5595a1bb9e20 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.005+0000 7f452dffb640 0 client.15792 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.005+0000 7f4553225740 1 --2- 172.21.15.42:0/3473524047 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5595a17e34d0 0x5595a1bb98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:33.006 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.005+0000 7f4553225740 1 --2- 172.21.15.42:0/3473524047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5595a17db2f0 0x5595a1bb9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.005+0000 7f4553225740 1 -- 172.21.15.42:0/3473524047 >> 172.21.15.42:0/3473524047 conn(0x5595a17f8300 msgr2=0x5595a1bc0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.005+0000 7f4553225740 1 -- 172.21.15.42:0/3473524047 shutdown_connections 2024-07-26T06:53:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.005+0000 7f4553225740 1 -- 172.21.15.42:0/3473524047 wait complete. 2024-07-26T06:53:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.005+0000 7f4553225740 20 client.15792 trim_cache size 0 max 16384 2024-07-26T06:53:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:53:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:53:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:53:33.007 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:53:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.005+0000 7f454c30e640 1 -- 172.21.15.42:0/1322847034 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f45300b3c50 msgr2=0x7f45300b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.005+0000 7f454c30e640 1 --2- 172.21.15.42:0/1322847034 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f45300b3c50 0x7f45300b6160 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4540000c00 tx=0x7f4540031400 comp rx=0 tx=0).stop 2024-07-26T06:53:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.005+0000 7f454c30e640 1 -- 172.21.15.42:0/1322847034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4544103060 msgr2=0x7f4544104490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.005+0000 7f454c30e640 1 --2- 172.21.15.42:0/1322847034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4544103060 0x7f4544104490 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f453400e9f0 tx=0x7f453400edf0 comp rx=0 tx=0).stop 2024-07-26T06:53:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.006+0000 7f454c30e640 1 -- 172.21.15.42:0/1322847034 shutdown_connections 2024-07-26T06:53:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.006+0000 7f454c30e640 1 --2- 172.21.15.42:0/1322847034 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f45300b3c50 0x7f45300b6160 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.006+0000 7f454c30e640 1 --2- 172.21.15.42:0/1322847034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4544103a00 0x7f45441049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.006+0000 7f454c30e640 1 --2- 172.21.15.42:0/1322847034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4544103060 0x7f4544104490 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.006+0000 7f454c30e640 1 --2- 172.21.15.42:0/1322847034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4544101fe0 0x7f4544105bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.006+0000 7f454c30e640 1 -- 172.21.15.42:0/1322847034 >> 172.21.15.42:0/1322847034 conn(0x7f45440fd960 msgr2=0x7f45440ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.006+0000 7f454c30e640 1 -- 172.21.15.42:0/1322847034 shutdown_connections 2024-07-26T06:53:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:33.006+0000 7f454c30e640 1 -- 172.21.15.42:0/1322847034 wait complete. 2024-07-26T06:53:33.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:32 smithi163 ceph-mon[36873]: pgmap v496: 97 pgs: 97 active+clean; 45 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.1 MiB/s wr, 736 op/s 2024-07-26T06:53:34.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:34 smithi042 ceph-mon[32775]: pgmap v497: 97 pgs: 97 active+clean; 46 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.4 MiB/s wr, 480 op/s 2024-07-26T06:53:34.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:34 smithi164 ceph-mon[36970]: pgmap v497: 97 pgs: 97 active+clean; 46 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.4 MiB/s wr, 480 op/s 2024-07-26T06:53:35.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:34 smithi163 ceph-mon[36873]: pgmap v497: 97 pgs: 97 active+clean; 46 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.4 MiB/s wr, 480 op/s 2024-07-26T06:53:36.661 INFO:tasks.workunit.client.0.smithi042.stdout:50000 copy from 0x898b8 to 0x92015, (0x875d bytes) at 0xb142c 2024-07-26T06:53:36.661 INFO:tasks.workunit.client.0.smithi042.stdout:All 50000 operations completed A-OK! 2024-07-26T06:53:36.662 INFO:tasks.workunit.client.0.smithi042.stderr:+ ./fsx -z 10MB -N 50000 -p 10000 -l 10485760 2024-07-26T06:53:36.662 INFO:tasks.workunit.client.0.smithi042.stdout:Seed set to 1 2024-07-26T06:53:36.789 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support fallocate mode 0, disabling! 2024-07-26T06:53:36.789 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support fallocate mode FALLOC_FL_KEEP_SIZE, disabling! 2024-07-26T06:53:36.791 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support fallocate mode FALLOC_FL_COLLAPSE_RANGE, disabling! 2024-07-26T06:53:36.791 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support fallocate mode FALLOC_FL_INSERT_RANGE, disabling! 2024-07-26T06:53:36.791 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support clone range, disabling! 2024-07-26T06:53:36.794 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support dedupe range, disabling! 2024-07-26T06:53:36.796 INFO:tasks.workunit.client.0.smithi042.stdout:skipping zero length punch hole 2024-07-26T06:53:36.797 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support exchange range, disabling! 2024-07-26T06:53:36.797 INFO:tasks.workunit.client.0.smithi042.stdout:skipping zero length punch hole 2024-07-26T06:53:36.797 INFO:tasks.workunit.client.0.smithi042.stdout:skipping zero length punch hole 2024-07-26T06:53:36.797 INFO:tasks.workunit.client.0.smithi042.stdout:skipping zero size read 2024-07-26T06:53:36.797 INFO:tasks.workunit.client.0.smithi042.stdout:skipping zero length punch hole 2024-07-26T06:53:36.797 INFO:tasks.workunit.client.0.smithi042.stdout:skipping zero size read 2024-07-26T06:53:36.797 INFO:tasks.workunit.client.0.smithi042.stdout:skipping zero length punch hole 2024-07-26T06:53:36.797 INFO:tasks.workunit.client.0.smithi042.stdout:truncating to largest ever: 0x4310e9 2024-07-26T06:53:36.803 INFO:tasks.workunit.client.0.smithi042.stdout:copying to largest ever: 0x954a45 2024-07-26T06:53:36.834 INFO:tasks.workunit.client.0.smithi042.stdout:truncating to largest ever: 0x9bc582 2024-07-26T06:53:36.855 INFO:tasks.workunit.client.0.smithi042.stdout:truncating to largest ever: 0x9eecbc 2024-07-26T06:53:36.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:36 smithi042 ceph-mon[32775]: pgmap v498: 97 pgs: 97 active+clean; 49 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.5 MiB/s wr, 547 op/s 2024-07-26T06:53:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:36 smithi164 ceph-mon[36970]: pgmap v498: 97 pgs: 97 active+clean; 49 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.5 MiB/s wr, 547 op/s 2024-07-26T06:53:37.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:36 smithi163 ceph-mon[36873]: pgmap v498: 97 pgs: 97 active+clean; 49 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.5 MiB/s wr, 547 op/s 2024-07-26T06:53:38.722 INFO:tasks.workunit.client.0.smithi042.stdout:copying to largest ever: 0x9fdd84 2024-07-26T06:53:38.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:38 smithi042 ceph-mon[32775]: pgmap v499: 97 pgs: 97 active+clean; 44 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.3 MiB/s wr, 658 op/s 2024-07-26T06:53:38.971 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:38 smithi164 ceph-mon[36970]: pgmap v499: 97 pgs: 97 active+clean; 44 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.3 MiB/s wr, 658 op/s 2024-07-26T06:53:39.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:38 smithi163 ceph-mon[36873]: pgmap v499: 97 pgs: 97 active+clean; 44 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.3 MiB/s wr, 658 op/s 2024-07-26T06:53:40.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:40 smithi042 ceph-mon[32775]: pgmap v500: 97 pgs: 97 active+clean; 50 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.5 MiB/s wr, 603 op/s 2024-07-26T06:53:40.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:40 smithi164 ceph-mon[36970]: pgmap v500: 97 pgs: 97 active+clean; 50 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.5 MiB/s wr, 603 op/s 2024-07-26T06:53:41.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:40 smithi163 ceph-mon[36873]: pgmap v500: 97 pgs: 97 active+clean; 50 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.5 MiB/s wr, 603 op/s 2024-07-26T06:53:42.108 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 31.1 2024-07-26T06:53:42.109 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:53:42.396 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:53:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:42 smithi042 ceph-mon[32775]: pgmap v501: 97 pgs: 97 active+clean; 51 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.6 MiB/s wr, 591 op/s 2024-07-26T06:53:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:42 smithi164 ceph-mon[36970]: pgmap v501: 97 pgs: 97 active+clean; 51 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.6 MiB/s wr, 591 op/s 2024-07-26T06:53:43.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:42 smithi163 ceph-mon[36873]: pgmap v501: 97 pgs: 97 active+clean; 51 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.6 MiB/s wr, 591 op/s 2024-07-26T06:53:44.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.321+0000 7f6f677b0640 1 -- 172.21.15.42:0/1964952475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6f60107560 msgr2=0x7f6f60107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:44.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.321+0000 7f6f677b0640 1 --2- 172.21.15.42:0/1964952475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6f60107560 0x7f6f60107970 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6f5400aa10 tx=0x7f6f5402b720 comp rx=0 tx=0).stop 2024-07-26T06:53:44.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.321+0000 7f6f677b0640 1 -- 172.21.15.42:0/1964952475 shutdown_connections 2024-07-26T06:53:44.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.321+0000 7f6f677b0640 1 --2- 172.21.15.42:0/1964952475 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f60107eb0 0x7f6f6010e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:44.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.321+0000 7f6f677b0640 1 --2- 172.21.15.42:0/1964952475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6f60107560 0x7f6f60107970 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:44.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.321+0000 7f6f677b0640 1 --2- 172.21.15.42:0/1964952475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f60101c20 0x7f6f60102030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:44.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.321+0000 7f6f677b0640 1 -- 172.21.15.42:0/1964952475 >> 172.21.15.42:0/1964952475 conn(0x7f6f60077690 msgr2=0x7f6f60077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:44.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.321+0000 7f6f677b0640 1 -- 172.21.15.42:0/1964952475 shutdown_connections 2024-07-26T06:53:44.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.321+0000 7f6f677b0640 1 -- 172.21.15.42:0/1964952475 wait complete. 2024-07-26T06:53:44.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.322+0000 7f6f677b0640 1 Processor -- start 2024-07-26T06:53:44.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.322+0000 7f6f677b0640 1 -- start start 2024-07-26T06:53:44.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.322+0000 7f6f677b0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6f60101c20 0x7f6f60199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:44.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.322+0000 7f6f677b0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f60107560 0x7f6f6019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:44.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.322+0000 7f6f677b0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f60107eb0 0x7f6f6019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:44.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.322+0000 7f6f677b0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6f601112f0 con 0x7f6f60101c20 2024-07-26T06:53:44.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.322+0000 7f6f677b0640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6f60111170 con 0x7f6f60107560 2024-07-26T06:53:44.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.322+0000 7f6f677b0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6f60111470 con 0x7f6f60107eb0 2024-07-26T06:53:44.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.322+0000 7f6f6554c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6f60101c20 0x7f6f60199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:44.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.322+0000 7f6f6554c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6f60101c20 0x7f6f60199ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49532/0 (socket says 172.21.15.42:49532) 2024-07-26T06:53:44.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.322+0000 7f6f6554c640 1 -- 172.21.15.42:0/48498443 learned_addr learned my addr 172.21.15.42:0/48498443 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:44.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.323+0000 7f6f64d4b640 1 --2- 172.21.15.42:0/48498443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f60107560 0x7f6f6019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:44.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.323+0000 7f6f65d4d640 1 --2- 172.21.15.42:0/48498443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f60107eb0 0x7f6f6019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:44.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.323+0000 7f6f6554c640 1 -- 172.21.15.42:0/48498443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f60107eb0 msgr2=0x7f6f6019f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:44.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.323+0000 7f6f6554c640 1 --2- 172.21.15.42:0/48498443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f60107eb0 0x7f6f6019f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:44.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.323+0000 7f6f6554c640 1 -- 172.21.15.42:0/48498443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f60107560 msgr2=0x7f6f6019a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:44.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.323+0000 7f6f6554c640 1 --2- 172.21.15.42:0/48498443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f60107560 0x7f6f6019a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:44.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.323+0000 7f6f6554c640 1 -- 172.21.15.42:0/48498443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f6019fe50 con 0x7f6f60101c20 2024-07-26T06:53:44.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.323+0000 7f6f65d4d640 1 --2- 172.21.15.42:0/48498443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f60107eb0 0x7f6f6019f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:53:44.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.323+0000 7f6f6554c640 1 --2- 172.21.15.42:0/48498443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6f60101c20 0x7f6f60199ee0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6f5800e9f0 tx=0x7f6f5800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:44.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.323+0000 7f6f5e7fc640 1 -- 172.21.15.42:0/48498443 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5800cd80 con 0x7f6f60101c20 2024-07-26T06:53:44.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.323+0000 7f6f5e7fc640 1 -- 172.21.15.42:0/48498443 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f580054d0 con 0x7f6f60101c20 2024-07-26T06:53:44.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.323+0000 7f6f5e7fc640 1 -- 172.21.15.42:0/48498443 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f58010690 con 0x7f6f60101c20 2024-07-26T06:53:44.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.323+0000 7f6f677b0640 1 -- 172.21.15.42:0/48498443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f601a0140 con 0x7f6f60101c20 2024-07-26T06:53:44.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.323+0000 7f6f677b0640 1 -- 172.21.15.42:0/48498443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f601a68a0 con 0x7f6f60101c20 2024-07-26T06:53:44.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.324+0000 7f6f677b0640 1 -- 172.21.15.42:0/48498443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f60102030 con 0x7f6f60101c20 2024-07-26T06:53:44.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.326+0000 7f6f5e7fc640 1 -- 172.21.15.42:0/48498443 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f6f58004e30 con 0x7f6f60101c20 2024-07-26T06:53:44.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.327+0000 7f6f5e7fc640 1 --2- 172.21.15.42:0/48498443 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6f480b3bd0 0x7f6f480b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:44.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.327+0000 7f6f5e7fc640 1 -- 172.21.15.42:0/48498443 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f6f580e42c0 con 0x7f6f60101c20 2024-07-26T06:53:44.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.327+0000 7f6f64d4b640 1 --2- 172.21.15.42:0/48498443 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6f480b3bd0 0x7f6f480b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:44.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.327+0000 7f6f64d4b640 1 --2- 172.21.15.42:0/48498443 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6f480b3bd0 0x7f6f480b60e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6f5400a340 tx=0x7f6f54030510 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:44.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.330+0000 7f6f5e7fc640 1 -- 172.21.15.42:0/48498443 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6f58055600 con 0x7f6f60101c20 2024-07-26T06:53:44.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.575+0000 7f6f677b0640 1 -- 172.21.15.42:0/48498443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f6019b010 con 0x7f6f60101c20 2024-07-26T06:53:44.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.576+0000 7f6f5e7fc640 1 -- 172.21.15.42:0/48498443 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11986 (secure 0 0 0) 0x7f6f6019b010 con 0x7f6f60101c20 2024-07-26T06:53:44.577 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:53:44.577 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":36,"btime":"2024-07-26T06:52:41:554060+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:52:41.554043+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25369":{"gid":25369,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/1072602530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1072602530},{"type":"v1","addr":"172.21.15.164:6835","nonce":1072602530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24578,24583,24593,24601,24620]},"id":1}]} 2024-07-26T06:53:44.577 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 36 2024-07-26T06:53:44.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.579+0000 7f6f677b0640 1 -- 172.21.15.42:0/48498443 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6f480b3bd0 msgr2=0x7f6f480b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:44.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.579+0000 7f6f677b0640 1 --2- 172.21.15.42:0/48498443 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6f480b3bd0 0x7f6f480b60e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6f5400a340 tx=0x7f6f54030510 comp rx=0 tx=0).stop 2024-07-26T06:53:44.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.579+0000 7f6f677b0640 1 -- 172.21.15.42:0/48498443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6f60101c20 msgr2=0x7f6f60199ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:44.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.579+0000 7f6f677b0640 1 --2- 172.21.15.42:0/48498443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6f60101c20 0x7f6f60199ee0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6f5800e9f0 tx=0x7f6f5800edf0 comp rx=0 tx=0).stop 2024-07-26T06:53:44.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.579+0000 7f6f677b0640 1 -- 172.21.15.42:0/48498443 shutdown_connections 2024-07-26T06:53:44.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.579+0000 7f6f677b0640 1 --2- 172.21.15.42:0/48498443 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f6f480b3bd0 0x7f6f480b60e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:44.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.579+0000 7f6f677b0640 1 --2- 172.21.15.42:0/48498443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f60107eb0 0x7f6f6019f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:44.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.579+0000 7f6f677b0640 1 --2- 172.21.15.42:0/48498443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f60107560 0x7f6f6019a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:44.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.579+0000 7f6f677b0640 1 --2- 172.21.15.42:0/48498443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6f60101c20 0x7f6f60199ee0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:44.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.579+0000 7f6f677b0640 1 -- 172.21.15.42:0/48498443 >> 172.21.15.42:0/48498443 conn(0x7f6f60077690 msgr2=0x7f6f601037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:44.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.579+0000 7f6f677b0640 1 -- 172.21.15.42:0/48498443 shutdown_connections 2024-07-26T06:53:44.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:44.579+0000 7f6f677b0640 1 -- 172.21.15.42:0/48498443 wait complete. 2024-07-26T06:53:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:44 smithi042 ceph-mon[32775]: pgmap v502: 97 pgs: 97 active+clean; 49 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 2.9 MiB/s wr, 533 op/s 2024-07-26T06:53:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:44 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/48498443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:53:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:44 smithi164 ceph-mon[36970]: pgmap v502: 97 pgs: 97 active+clean; 49 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 2.9 MiB/s wr, 533 op/s 2024-07-26T06:53:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:44 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/48498443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:53:45.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:44 smithi163 ceph-mon[36873]: pgmap v502: 97 pgs: 97 active+clean; 49 MiB data, 826 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 2.9 MiB/s wr, 533 op/s 2024-07-26T06:53:45.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:44 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/48498443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:53:45.340 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24620 quiesce db / --timeout 90 --expiration 151.1 --await-for 45.0' 2024-07-26T06:53:45.341 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.24620 quiesce db / --timeout 90 --expiration 151.1 --await-for 45.0 2024-07-26T06:53:45.646 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:53:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.119+0000 7fc18dce7640 1 -- 172.21.15.42:0/1063239878 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc1881031a0 msgr2=0x7fc188109b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.119+0000 7fc18dce7640 1 --2- 172.21.15.42:0/1063239878 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc1881031a0 0x7fc188109b40 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc180009ca0 tx=0x7fc18002a8c0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.119+0000 7fc18dce7640 1 -- 172.21.15.42:0/1063239878 shutdown_connections 2024-07-26T06:53:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.119+0000 7fc18dce7640 1 --2- 172.21.15.42:0/1063239878 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc1881031a0 0x7fc188109b40 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.119+0000 7fc18dce7640 1 --2- 172.21.15.42:0/1063239878 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc188102800 0x7fc188102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.119+0000 7fc18dce7640 1 --2- 172.21.15.42:0/1063239878 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc188101780 0x7fc188101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.119+0000 7fc18dce7640 1 -- 172.21.15.42:0/1063239878 >> 172.21.15.42:0/1063239878 conn(0x7fc1880fd100 msgr2=0x7fc1880ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.119+0000 7fc18dce7640 1 -- 172.21.15.42:0/1063239878 shutdown_connections 2024-07-26T06:53:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.119+0000 7fc18dce7640 1 -- 172.21.15.42:0/1063239878 wait complete. 2024-07-26T06:53:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.120+0000 7fc18dce7640 1 Processor -- start 2024-07-26T06:53:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.120+0000 7fc18dce7640 1 -- start start 2024-07-26T06:53:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.120+0000 7fc18dce7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc188101780 0x7fc188199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.120+0000 7fc18dce7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc188102800 0x7fc188199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.120+0000 7fc18dce7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc1881031a0 0x7fc18819ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.120+0000 7fc18dce7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc18810cb70 con 0x7fc188101780 2024-07-26T06:53:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.120+0000 7fc18dce7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc18810c9f0 con 0x7fc1881031a0 2024-07-26T06:53:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.120+0000 7fc18dce7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc18810ccf0 con 0x7fc188102800 2024-07-26T06:53:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.121+0000 7fc1877fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc188101780 0x7fc188199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.121+0000 7fc1877fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc188101780 0x7fc188199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49538/0 (socket says 172.21.15.42:49538) 2024-07-26T06:53:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.121+0000 7fc1877fe640 1 -- 172.21.15.42:0/2261656532 learned_addr learned my addr 172.21.15.42:0/2261656532 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.121+0000 7fc17ffff640 1 --2- 172.21.15.42:0/2261656532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc188102800 0x7fc188199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.121+0000 7fc187fff640 1 --2- 172.21.15.42:0/2261656532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc1881031a0 0x7fc18819ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.121+0000 7fc1877fe640 1 -- 172.21.15.42:0/2261656532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc188102800 msgr2=0x7fc188199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.121+0000 7fc1877fe640 1 --2- 172.21.15.42:0/2261656532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc188102800 0x7fc188199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.121+0000 7fc1877fe640 1 -- 172.21.15.42:0/2261656532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc1881031a0 msgr2=0x7fc18819ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.121+0000 7fc1877fe640 1 --2- 172.21.15.42:0/2261656532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc1881031a0 0x7fc18819ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.121+0000 7fc1877fe640 1 -- 172.21.15.42:0/2261656532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc18819b390 con 0x7fc188101780 2024-07-26T06:53:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.121+0000 7fc1877fe640 1 --2- 172.21.15.42:0/2261656532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc188101780 0x7fc188199950 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc17800e9f0 tx=0x7fc17800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.121+0000 7fc1857fa640 1 -- 172.21.15.42:0/2261656532 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17800cd80 con 0x7fc188101780 2024-07-26T06:53:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.121+0000 7fc1857fa640 1 -- 172.21.15.42:0/2261656532 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1780108e0 con 0x7fc188101780 2024-07-26T06:53:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.121+0000 7fc18dce7640 1 -- 172.21.15.42:0/2261656532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc18819b680 con 0x7fc188101780 2024-07-26T06:53:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.121+0000 7fc1857fa640 1 -- 172.21.15.42:0/2261656532 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178018980 con 0x7fc188101780 2024-07-26T06:53:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.121+0000 7fc18dce7640 1 -- 172.21.15.42:0/2261656532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1881a1d70 con 0x7fc188101780 2024-07-26T06:53:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc194bfd740 1 Processor -- start 2024-07-26T06:53:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc194bfd740 1 -- start start 2024-07-26T06:53:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc194bfd740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 0x561225d8fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc194bfd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225d901c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc194bfd740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 0x561225d90d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc194bfd740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x561225d033b0 con 0x5612259eb1b0 2024-07-26T06:53:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc194bfd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561225cf28b0 con 0x5612259c22f0 2024-07-26T06:53:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc194bfd740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x561225cf68f0 con 0x5612259ca4d0 2024-07-26T06:53:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc187fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 0x561225d8fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc187fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 0x561225d8fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49546/0 (socket says 172.21.15.42:49546) 2024-07-26T06:53:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc187fff640 1 -- 172.21.15.42:0/1003164704 learned_addr learned my addr 172.21.15.42:0/1003164704 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc17ffff640 1 --2- 172.21.15.42:0/1003164704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 0x561225d90d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc1857fa640 1 -- 172.21.15.42:0/2261656532 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fc178010430 con 0x7fc188101780 2024-07-26T06:53:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc1877fe640 1 --2- 172.21.15.42:0/1003164704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225d901c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc187fff640 1 -- 172.21.15.42:0/1003164704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 msgr2=0x561225d90d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc187fff640 1 --2- 172.21.15.42:0/1003164704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 0x561225d90d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc187fff640 1 -- 172.21.15.42:0/1003164704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 msgr2=0x561225d901c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc187fff640 1 --2- 172.21.15.42:0/1003164704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225d901c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.125+0000 7fc187fff640 1 -- 172.21.15.42:0/1003164704 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561225c999a0 con 0x5612259eb1b0 2024-07-26T06:53:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.126+0000 7fc17ffff640 1 --2- 172.21.15.42:0/1003164704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 0x561225d90d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.126+0000 7fc1877fe640 1 --2- 172.21.15.42:0/1003164704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225d901c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:53:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.126+0000 7fc187fff640 1 --2- 172.21.15.42:0/1003164704 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 0x561225d8fb50 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc180035000 tx=0x7fc18002f410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d44a08553fc7fd4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.126+0000 7fc17dffb640 1 -- 172.21.15.42:0/1003164704 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc180031650 con 0x5612259eb1b0 2024-07-26T06:53:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.126+0000 7fc17dffb640 1 -- 172.21.15.42:0/1003164704 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc180036030 con 0x5612259eb1b0 2024-07-26T06:53:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.126+0000 7fc17dffb640 1 -- 172.21.15.42:0/1003164704 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc180004790 con 0x5612259eb1b0 2024-07-26T06:53:46.136 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.127+0000 7fc1857fa640 1 --2- 172.21.15.42:0/2261656532 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc1740b3bd0 0x7fc1740b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.127+0000 7fc1877fe640 1 --2- 172.21.15.42:0/2261656532 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc1740b3bd0 0x7fc1740b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.127+0000 7fc1857fa640 1 -- 172.21.15.42:0/2261656532 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fc1780e3f20 con 0x7fc188101780 2024-07-26T06:53:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.127+0000 7fc194bfd740 1 -- 172.21.15.42:0/1003164704 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 msgr2=0x561225d8fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.127+0000 7fc194bfd740 1 --2- 172.21.15.42:0/1003164704 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 0x561225d8fb50 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc180035000 tx=0x7fc18002f410 comp rx=0 tx=0).stop 2024-07-26T06:53:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.127+0000 7fc1877fe640 1 --2- 172.21.15.42:0/2261656532 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc1740b3bd0 0x7fc1740b60e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc1780a8170 tx=0x7fc1780a5480 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.127+0000 7fc194bfd740 1 -- 172.21.15.42:0/1003164704 shutdown_connections 2024-07-26T06:53:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.127+0000 7fc194bfd740 1 --2- 172.21.15.42:0/1003164704 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 0x561225d90d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.127+0000 7fc194bfd740 1 --2- 172.21.15.42:0/1003164704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225d901c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.127+0000 7fc194bfd740 1 --2- 172.21.15.42:0/1003164704 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 0x561225d8fb50 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.127+0000 7fc194bfd740 1 -- 172.21.15.42:0/1003164704 >> 172.21.15.42:0/1003164704 conn(0x5612259df300 msgr2=0x561225d8b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.128+0000 7fc194bfd740 1 -- 172.21.15.42:0/1003164704 shutdown_connections 2024-07-26T06:53:46.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.128+0000 7fc194bfd740 1 -- 172.21.15.42:0/1003164704 wait complete. 2024-07-26T06:53:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.128+0000 7fc194bfd740 1 -- start start 2024-07-26T06:53:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.128+0000 7fc194bfd740 1 Processor -- start 2024-07-26T06:53:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc194bfd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225d99330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc194bfd740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259ca4d0 0x561225d999f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc194bfd740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259eb1b0 0x561225d9a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc194bfd740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x561225c2cab0 con 0x5612259ca4d0 2024-07-26T06:53:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc194bfd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561225c7acf0 con 0x5612259c22f0 2024-07-26T06:53:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc194bfd740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x561225d91650 con 0x5612259eb1b0 2024-07-26T06:53:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc194bfd740 10 client.0 _ll_register_callbacks cb 0x561225cec3e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:53:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc187fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225d99330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc17ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259ca4d0 0x561225d999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc187fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225d99330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:45770/0 (socket says 172.21.15.42:45770) 2024-07-26T06:53:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc187fff640 1 -- 172.21.15.42:0/1767106403 learned_addr learned my addr 172.21.15.42:0/1767106403 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc187fff640 1 --2- 172.21.15.42:0/1767106403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259eb1b0 0x561225d9a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc17ffff640 1 -- 172.21.15.42:0/1767106403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259eb1b0 msgr2=0x561225d9a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc17ffff640 1 --2- 172.21.15.42:0/1767106403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259eb1b0 0x561225d9a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc17ffff640 1 -- 172.21.15.42:0/1767106403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 msgr2=0x561225d99330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc17ffff640 1 --2- 172.21.15.42:0/1767106403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225d99330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc17ffff640 1 -- 172.21.15.42:0/1767106403 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561225a56790 con 0x5612259ca4d0 2024-07-26T06:53:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc187fff640 1 --2- 172.21.15.42:0/1767106403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225d99330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc17ffff640 1 --2- 172.21.15.42:0/1767106403 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259ca4d0 0x561225d999f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc17000e700 tx=0x7fc17000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=240d25bba61c82fa server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc166ffd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:53:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.129+0000 7fc166ffd640 1 -- 172.21.15.42:0/1767106403 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17001e060 con 0x5612259ca4d0 2024-07-26T06:53:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.130+0000 7fc166ffd640 1 -- 172.21.15.42:0/1767106403 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fc170004240 con 0x5612259ca4d0 2024-07-26T06:53:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.130+0000 7fc194bfd740 1 -- 172.21.15.42:0/1767106403 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc180036030 con 0x5612259ca4d0 2024-07-26T06:53:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.130+0000 7fc166ffd640 1 -- 172.21.15.42:0/1767106403 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fc170004440 con 0x5612259ca4d0 2024-07-26T06:53:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.130+0000 7fc194bfd740 10 client.15810 fetch_fsmap learned FSMap version 36 2024-07-26T06:53:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.130+0000 7fc194bfd740 1 -- 172.21.15.42:0/1767106403 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x561225acf620 con 0x5612259ca4d0 2024-07-26T06:53:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.130+0000 7fc166ffd640 1 -- 172.21.15.42:0/1767106403 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc180036030 con 0x5612259ca4d0 2024-07-26T06:53:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.130+0000 7fc166ffd640 1 -- 172.21.15.42:0/1767106403 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7fc17000b040 con 0x5612259ca4d0 2024-07-26T06:53:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.130+0000 7fc194bfd740 10 client.15810 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:53:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.130+0000 7fc194bfd740 10 client.15810 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:53:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.130+0000 7fc194bfd740 20 client.15810 populate_metadata read hostname 'smithi042' 2024-07-26T06:53:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.130+0000 7fc194bfd740 1 --2- 172.21.15.42:0/1767106403 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5612259a9d80 0x561225da6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.130+0000 7fc194bfd740 4 client.15810 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:53:46.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.130+0000 7fc194bfd740 1 -- 172.21.15.42:0/1767106403 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561225d05300 con 0x5612259a9d80 2024-07-26T06:53:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.130+0000 7fc187fff640 1 --2- 172.21.15.42:0/1767106403 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5612259a9d80 0x561225da6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.131+0000 7fc166ffd640 10 client.15810 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.131+0000 7fc187fff640 1 --2- 172.21.15.42:0/1767106403 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5612259a9d80 0x561225da6b10 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23ca7354f18a7d95 server_cookie=4e37b3109cbc8b19 in_seq=0 out_seq=0 2024-07-26T06:53:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.132+0000 7fc166ffd640 1 -- 172.21.15.42:0/1767106403 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561225d05300 con 0x5612259a9d80 2024-07-26T06:53:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.132+0000 7fc166ffd640 10 client.15810 handle_command_reply: tid=0 2024-07-26T06:53:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.132+0000 7fc194bfd740 1 client.15810 shutdown 2024-07-26T06:53:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.132+0000 7fc194bfd740 1 -- 172.21.15.42:0/1767106403 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259ca4d0 msgr2=0x561225d999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.132+0000 7fc194bfd740 1 --2- 172.21.15.42:0/1767106403 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259ca4d0 0x561225d999f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc17000e700 tx=0x7fc17000ebc0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.132+0000 7fc194bfd740 1 -- 172.21.15.42:0/1767106403 shutdown_connections 2024-07-26T06:53:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.132+0000 7fc194bfd740 1 --2- 172.21.15.42:0/1767106403 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5612259a9d80 0x561225da6b10 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.141 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.132+0000 7fc194bfd740 1 --2- 172.21.15.42:0/1767106403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259eb1b0 0x561225d9a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.132+0000 7fc194bfd740 1 --2- 172.21.15.42:0/1767106403 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259ca4d0 0x561225d999f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.132+0000 7fc194bfd740 1 --2- 172.21.15.42:0/1767106403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225d99330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.132+0000 7fc194bfd740 1 -- 172.21.15.42:0/1767106403 >> 172.21.15.42:0/1767106403 conn(0x5612259df300 msgr2=0x561225d89c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.132+0000 7fc166ffd640 0 client.15810 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.132+0000 7fc194bfd740 1 -- 172.21.15.42:0/1767106403 shutdown_connections 2024-07-26T06:53:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.132+0000 7fc194bfd740 1 -- 172.21.15.42:0/1767106403 wait complete. 2024-07-26T06:53:46.142 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.132+0000 7fc194bfd740 20 client.15810 trim_cache size 0 max 16384 2024-07-26T06:53:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.150+0000 7fc194bfd740 1 Processor -- start 2024-07-26T06:53:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.150+0000 7fc194bfd740 1 -- start start 2024-07-26T06:53:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc194bfd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225d8c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc194bfd740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 0x561225d8c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc194bfd740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 0x561225d89c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc194bfd740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc1881aaa60 con 0x5612259eb1b0 2024-07-26T06:53:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc194bfd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc1881ac690 con 0x5612259c22f0 2024-07-26T06:53:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc194bfd740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc1881ab160 con 0x5612259ca4d0 2024-07-26T06:53:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc17ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 0x561225d8c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc187fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 0x561225d89c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.155 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc17ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 0x561225d8c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:54582/0 (socket says 172.21.15.42:54582) 2024-07-26T06:53:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc17ffff640 1 -- 172.21.15.42:0/3607907156 learned_addr learned my addr 172.21.15.42:0/3607907156 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc187fff640 1 --2- 172.21.15.42:0/3607907156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225d8c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc17ffff640 1 -- 172.21.15.42:0/3607907156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 msgr2=0x561225d8c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc17ffff640 1 --2- 172.21.15.42:0/3607907156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225d8c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc17ffff640 1 -- 172.21.15.42:0/3607907156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 msgr2=0x561225d89c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc17ffff640 1 --2- 172.21.15.42:0/3607907156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 0x561225d89c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc17ffff640 1 -- 172.21.15.42:0/3607907156 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561225d990c0 con 0x5612259ca4d0 2024-07-26T06:53:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc187fff640 1 --2- 172.21.15.42:0/3607907156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 0x561225d89c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.151+0000 7fc17ffff640 1 --2- 172.21.15.42:0/3607907156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 0x561225d8c6c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc170001eb0 tx=0x7fc170024480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64b7cebeabeeee1f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.152+0000 7fc17d7fa640 1 -- 172.21.15.42:0/3607907156 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc170031070 con 0x5612259ca4d0 2024-07-26T06:53:46.156 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.152+0000 7fc17d7fa640 1 -- 172.21.15.42:0/3607907156 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc170003dc0 con 0x5612259ca4d0 2024-07-26T06:53:46.157 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.152+0000 7fc17d7fa640 1 -- 172.21.15.42:0/3607907156 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc170024590 con 0x5612259ca4d0 2024-07-26T06:53:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.153+0000 7fc194bfd740 1 -- 172.21.15.42:0/3607907156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 msgr2=0x561225d8c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.153+0000 7fc194bfd740 1 --2- 172.21.15.42:0/3607907156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 0x561225d8c6c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc170001eb0 tx=0x7fc170024480 comp rx=0 tx=0).stop 2024-07-26T06:53:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.153+0000 7fc194bfd740 1 -- 172.21.15.42:0/3607907156 shutdown_connections 2024-07-26T06:53:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.153+0000 7fc194bfd740 1 --2- 172.21.15.42:0/3607907156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 0x561225d89c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.153+0000 7fc194bfd740 1 --2- 172.21.15.42:0/3607907156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 0x561225d8c6c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.153+0000 7fc194bfd740 1 --2- 172.21.15.42:0/3607907156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225d8c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.153+0000 7fc194bfd740 1 -- 172.21.15.42:0/3607907156 >> 172.21.15.42:0/3607907156 conn(0x5612259df300 msgr2=0x561225d978c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.153+0000 7fc194bfd740 1 -- 172.21.15.42:0/3607907156 shutdown_connections 2024-07-26T06:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.153+0000 7fc194bfd740 1 -- 172.21.15.42:0/3607907156 wait complete. 2024-07-26T06:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.153+0000 7fc194bfd740 1 -- start start 2024-07-26T06:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.153+0000 7fc194bfd740 1 Processor -- start 2024-07-26T06:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.154+0000 7fc194bfd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225da04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.154+0000 7fc194bfd740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 0x561225da0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.154+0000 7fc194bfd740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 0x561225d9f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.154+0000 7fc194bfd740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc1881abfc0 con 0x5612259eb1b0 2024-07-26T06:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.154+0000 7fc194bfd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc1881ab830 con 0x5612259c22f0 2024-07-26T06:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.154+0000 7fc194bfd740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc1881a8a40 con 0x5612259ca4d0 2024-07-26T06:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.154+0000 7fc194bfd740 10 client.0 _ll_register_callbacks cb 0x7fc17000a9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.154+0000 7fc187fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 0x561225d9f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.154+0000 7fc187fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 0x561225d9f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49582/0 (socket says 172.21.15.42:49582) 2024-07-26T06:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.154+0000 7fc187fff640 1 -- 172.21.15.42:0/1605217119 learned_addr learned my addr 172.21.15.42:0/1605217119 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.154+0000 7fc187fff640 1 --2- 172.21.15.42:0/1605217119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225da04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.154+0000 7fc17ffff640 1 --2- 172.21.15.42:0/1605217119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 0x561225da0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.154+0000 7fc187fff640 1 -- 172.21.15.42:0/1605217119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 msgr2=0x561225da0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.154+0000 7fc187fff640 1 --2- 172.21.15.42:0/1605217119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 0x561225da0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.154+0000 7fc187fff640 1 -- 172.21.15.42:0/1605217119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 msgr2=0x561225da04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.154+0000 7fc187fff640 1 --2- 172.21.15.42:0/1605217119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225da04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.154+0000 7fc187fff640 1 -- 172.21.15.42:0/1605217119 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561225d9f5f0 con 0x5612259eb1b0 2024-07-26T06:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.155+0000 7fc187fff640 1 --2- 172.21.15.42:0/1605217119 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 0x561225d9f0b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fc18002ade0 tx=0x7fc18002f410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b17f93ff979f499d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.155+0000 7fc166ffd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.155+0000 7fc166ffd640 1 -- 172.21.15.42:0/1605217119 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc180003970 con 0x5612259eb1b0 2024-07-26T06:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.155+0000 7fc166ffd640 1 -- 172.21.15.42:0/1605217119 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fc180003b10 con 0x5612259eb1b0 2024-07-26T06:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.155+0000 7fc194bfd740 1 -- 172.21.15.42:0/1605217119 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc170003dc0 con 0x5612259eb1b0 2024-07-26T06:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.155+0000 7fc166ffd640 1 -- 172.21.15.42:0/1605217119 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fc180003d10 con 0x5612259eb1b0 2024-07-26T06:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.155+0000 7fc194bfd740 10 client.15816 fetch_fsmap learned FSMap version 36 2024-07-26T06:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.155+0000 7fc194bfd740 1 -- 172.21.15.42:0/1605217119 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x561225e0d8c0 con 0x5612259eb1b0 2024-07-26T06:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.155+0000 7fc166ffd640 1 -- 172.21.15.42:0/1605217119 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc170003dc0 con 0x5612259eb1b0 2024-07-26T06:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.155+0000 7fc166ffd640 1 -- 172.21.15.42:0/1605217119 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7fc180043070 con 0x5612259eb1b0 2024-07-26T06:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.156+0000 7fc194bfd740 10 client.15816 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.156+0000 7fc194bfd740 10 client.15816 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.156+0000 7fc194bfd740 20 client.15816 populate_metadata read hostname 'smithi042' 2024-07-26T06:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.156+0000 7fc194bfd740 1 --2- 172.21.15.42:0/1605217119 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56122597f1c0 0x561225e10050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.156+0000 7fc194bfd740 4 client.15816 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.156+0000 7fc194bfd740 1 -- 172.21.15.42:0/1605217119 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561225d00da0 con 0x56122597f1c0 2024-07-26T06:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.156+0000 7fc17ffff640 1 --2- 172.21.15.42:0/1605217119 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56122597f1c0 0x561225e10050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.156+0000 7fc166ffd640 10 client.15816 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.156+0000 7fc17ffff640 1 --2- 172.21.15.42:0/1605217119 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56122597f1c0 0x561225e10050 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=403d7ac407991126 server_cookie=a6dabc39298e4bd1 in_seq=0 out_seq=0 2024-07-26T06:53:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.175+0000 7fc166ffd640 1 -- 172.21.15.42:0/1605217119 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561225d00da0 con 0x56122597f1c0 2024-07-26T06:53:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.175+0000 7fc166ffd640 10 client.15816 handle_command_reply: tid=0 2024-07-26T06:53:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.175+0000 7fc194bfd740 1 client.15816 shutdown 2024-07-26T06:53:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.175+0000 7fc194bfd740 1 -- 172.21.15.42:0/1605217119 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 msgr2=0x561225d9f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.175+0000 7fc194bfd740 1 --2- 172.21.15.42:0/1605217119 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 0x561225d9f0b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fc18002ade0 tx=0x7fc18002f410 comp rx=0 tx=0).stop 2024-07-26T06:53:46.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.175+0000 7fc194bfd740 1 -- 172.21.15.42:0/1605217119 shutdown_connections 2024-07-26T06:53:46.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.175+0000 7fc194bfd740 1 --2- 172.21.15.42:0/1605217119 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56122597f1c0 0x561225e10050 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.175+0000 7fc194bfd740 1 --2- 172.21.15.42:0/1605217119 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5612259eb1b0 0x561225d9f0b0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.175+0000 7fc194bfd740 1 --2- 172.21.15.42:0/1605217119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5612259ca4d0 0x561225da0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.175+0000 7fc166ffd640 0 client.15816 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:46.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.175+0000 7fc194bfd740 1 --2- 172.21.15.42:0/1605217119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5612259c22f0 0x561225da04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.176+0000 7fc194bfd740 1 -- 172.21.15.42:0/1605217119 >> 172.21.15.42:0/1605217119 conn(0x5612259df300 msgr2=0x561225d8c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:46.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.176+0000 7fc194bfd740 1 -- 172.21.15.42:0/1605217119 shutdown_connections 2024-07-26T06:53:46.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.176+0000 7fc194bfd740 1 -- 172.21.15.42:0/1605217119 wait complete. 2024-07-26T06:53:46.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.176+0000 7fc194bfd740 20 client.15816 trim_cache size 0 max 16384 2024-07-26T06:53:46.179 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:53:46.179 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:53:46.179 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:53:46.179 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:53:46.180 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.176+0000 7fc18dce7640 1 -- 172.21.15.42:0/2261656532 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc1740b3bd0 msgr2=0x7fc1740b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:46.180 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.176+0000 7fc18dce7640 1 --2- 172.21.15.42:0/2261656532 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc1740b3bd0 0x7fc1740b60e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc1780a8170 tx=0x7fc1780a5480 comp rx=0 tx=0).stop 2024-07-26T06:53:46.180 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.176+0000 7fc18dce7640 1 -- 172.21.15.42:0/2261656532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc188101780 msgr2=0x7fc188199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:46.180 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.176+0000 7fc18dce7640 1 --2- 172.21.15.42:0/2261656532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc188101780 0x7fc188199950 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc17800e9f0 tx=0x7fc17800eeb0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.180 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.176+0000 7fc18dce7640 1 -- 172.21.15.42:0/2261656532 shutdown_connections 2024-07-26T06:53:46.180 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.176+0000 7fc18dce7640 1 --2- 172.21.15.42:0/2261656532 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc1740b3bd0 0x7fc1740b60e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.180 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.176+0000 7fc18dce7640 1 --2- 172.21.15.42:0/2261656532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc1881031a0 0x7fc18819ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.180 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.176+0000 7fc18dce7640 1 --2- 172.21.15.42:0/2261656532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc188102800 0x7fc188199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.180 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.176+0000 7fc18dce7640 1 --2- 172.21.15.42:0/2261656532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc188101780 0x7fc188199950 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:46.180 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.176+0000 7fc18dce7640 1 -- 172.21.15.42:0/2261656532 >> 172.21.15.42:0/2261656532 conn(0x7fc1880fd100 msgr2=0x7fc188107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:46.180 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.177+0000 7fc18dce7640 1 -- 172.21.15.42:0/2261656532 shutdown_connections 2024-07-26T06:53:46.181 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:46.177+0000 7fc18dce7640 1 -- 172.21.15.42:0/2261656532 wait complete. 2024-07-26T06:53:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:46 smithi042 ceph-mon[32775]: pgmap v503: 97 pgs: 97 active+clean; 51 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 3.7 MiB/s wr, 719 op/s 2024-07-26T06:53:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:46 smithi164 ceph-mon[36970]: pgmap v503: 97 pgs: 97 active+clean; 51 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 3.7 MiB/s wr, 719 op/s 2024-07-26T06:53:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:46 smithi163 ceph-mon[36873]: pgmap v503: 97 pgs: 97 active+clean; 51 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 3.7 MiB/s wr, 719 op/s 2024-07-26T06:53:47.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.574+0000 7f8255fc2640 1 -- 172.21.15.42:0/2398837224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82501024b0 msgr2=0x7f82501028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:47.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.574+0000 7f8255fc2640 1 --2- 172.21.15.42:0/2398837224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82501024b0 0x7f82501028c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8240009900 tx=0x7f824002a410 comp rx=0 tx=0).stop 2024-07-26T06:53:47.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.574+0000 7f8255fc2640 1 -- 172.21.15.42:0/2398837224 shutdown_connections 2024-07-26T06:53:47.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.574+0000 7f8255fc2640 1 --2- 172.21.15.42:0/2398837224 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8250108740 0x7f825010ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.574+0000 7f8255fc2640 1 --2- 172.21.15.42:0/2398837224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8250107df0 0x7f8250108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.575+0000 7f8255fc2640 1 --2- 172.21.15.42:0/2398837224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f82501024b0 0x7f82501028c0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.575+0000 7f8255fc2640 1 -- 172.21.15.42:0/2398837224 >> 172.21.15.42:0/2398837224 conn(0x7f8250079190 msgr2=0x7f82500795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:47.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.575+0000 7f8255fc2640 1 -- 172.21.15.42:0/2398837224 shutdown_connections 2024-07-26T06:53:47.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.575+0000 7f8255fc2640 1 -- 172.21.15.42:0/2398837224 wait complete. 2024-07-26T06:53:47.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.575+0000 7f8255fc2640 1 Processor -- start 2024-07-26T06:53:47.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f8255fc2640 1 -- start start 2024-07-26T06:53:47.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f8255fc2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82501024b0 0x7f825019c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f8255fc2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8250107df0 0x7f825019c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f8255fc2640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8250108740 0x7f82501a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f8255fc2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f82501112e0 con 0x7f8250107df0 2024-07-26T06:53:47.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f8255fc2640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8250111160 con 0x7f8250108740 2024-07-26T06:53:47.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f8255fc2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8250111460 con 0x7f82501024b0 2024-07-26T06:53:47.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f824f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82501024b0 0x7f825019c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f824effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8250107df0 0x7f825019c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f824f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82501024b0 0x7f825019c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:48768/0 (socket says 172.21.15.42:48768) 2024-07-26T06:53:47.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f824f7fe640 1 -- 172.21.15.42:0/4247520530 learned_addr learned my addr 172.21.15.42:0/4247520530 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:47.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f824ffff640 1 --2- 172.21.15.42:0/4247520530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8250108740 0x7f82501a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f824effd640 1 -- 172.21.15.42:0/4247520530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82501024b0 msgr2=0x7f825019c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:47.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f824effd640 1 --2- 172.21.15.42:0/4247520530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82501024b0 0x7f825019c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f824effd640 1 -- 172.21.15.42:0/4247520530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8250108740 msgr2=0x7f82501a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:47.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f824effd640 1 --2- 172.21.15.42:0/4247520530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8250108740 0x7f82501a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f824effd640 1 -- 172.21.15.42:0/4247520530 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82501a2070 con 0x7f8250107df0 2024-07-26T06:53:47.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.576+0000 7f824f7fe640 1 --2- 172.21.15.42:0/4247520530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82501024b0 0x7f825019c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:47.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.577+0000 7f824ffff640 1 --2- 172.21.15.42:0/4247520530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8250108740 0x7f82501a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:47.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.577+0000 7f824effd640 1 --2- 172.21.15.42:0/4247520530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8250107df0 0x7f825019c6b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f824400e9c0 tx=0x7f824400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:47.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.577+0000 7f824cff9640 1 -- 172.21.15.42:0/4247520530 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824400ccc0 con 0x7f8250107df0 2024-07-26T06:53:47.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.577+0000 7f8255fc2640 1 -- 172.21.15.42:0/4247520530 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82501a2360 con 0x7f8250107df0 2024-07-26T06:53:47.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.577+0000 7f824cff9640 1 -- 172.21.15.42:0/4247520530 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82440108e0 con 0x7f8250107df0 2024-07-26T06:53:47.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.577+0000 7f824cff9640 1 -- 172.21.15.42:0/4247520530 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82440187a0 con 0x7f8250107df0 2024-07-26T06:53:47.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.577+0000 7f8255fc2640 1 -- 172.21.15.42:0/4247520530 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82501a8ac0 con 0x7f8250107df0 2024-07-26T06:53:47.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.580+0000 7f825c546740 1 Processor -- start 2024-07-26T06:53:47.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.580+0000 7f825c546740 1 -- start start 2024-07-26T06:53:47.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.581+0000 7f824cff9640 1 -- 172.21.15.42:0/4247520530 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f8244010430 con 0x7f8250107df0 2024-07-26T06:53:47.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.581+0000 7f825c546740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb9a7840 0x5592bbd4d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.581+0000 7f825c546740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 0x5592bbd4dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.581+0000 7f825c546740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb986b60 0x5592bbd4e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.581+0000 7f825c546740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5592bbcc2780 con 0x5592bb9a7840 2024-07-26T06:53:47.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.581+0000 7f825c546740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5592bbcb1c80 con 0x5592bb986b60 2024-07-26T06:53:47.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.581+0000 7f825c546740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5592bbcb5cc0 con 0x5592bb97e980 2024-07-26T06:53:47.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.581+0000 7f824ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb9a7840 0x5592bbd4d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.581+0000 7f824f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 0x5592bbd4dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.581+0000 7f824f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 0x5592bbd4dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:48772/0 (socket says 172.21.15.42:48772) 2024-07-26T06:53:47.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.581+0000 7f824f7fe640 1 -- 172.21.15.42:0/112343667 learned_addr learned my addr 172.21.15.42:0/112343667 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:47.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.581+0000 7f824effd640 1 --2- 172.21.15.42:0/112343667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb986b60 0x5592bbd4e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.582+0000 7f824f7fe640 1 -- 172.21.15.42:0/112343667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb986b60 msgr2=0x5592bbd4e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:47.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.582+0000 7f824f7fe640 1 --2- 172.21.15.42:0/112343667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb986b60 0x5592bbd4e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.582+0000 7f824f7fe640 1 -- 172.21.15.42:0/112343667 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb9a7840 msgr2=0x5592bbd4d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:47.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.582+0000 7f824f7fe640 1 --2- 172.21.15.42:0/112343667 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb9a7840 0x5592bbd4d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.582+0000 7f824f7fe640 1 -- 172.21.15.42:0/112343667 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592bbc47080 con 0x5592bb97e980 2024-07-26T06:53:47.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.582+0000 7f824f7fe640 1 --2- 172.21.15.42:0/112343667 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 0x5592bbd4dd80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f82400044b0 tx=0x7f82400044e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc1f8f0287506c61 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:47.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.582+0000 7f8235ffb640 1 -- 172.21.15.42:0/112343667 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8240035040 con 0x5592bb97e980 2024-07-26T06:53:47.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.583+0000 7f8235ffb640 1 -- 172.21.15.42:0/112343667 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82400047a0 con 0x5592bb97e980 2024-07-26T06:53:47.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.583+0000 7f8235ffb640 1 -- 172.21.15.42:0/112343667 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8240003a90 con 0x5592bb97e980 2024-07-26T06:53:47.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.583+0000 7f824cff9640 1 --2- 172.21.15.42:0/4247520530 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82380b3bd0 0x7f82380b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.583+0000 7f824ffff640 1 --2- 172.21.15.42:0/4247520530 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82380b3bd0 0x7f82380b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.583+0000 7f824cff9640 1 -- 172.21.15.42:0/4247520530 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f82440e3f40 con 0x7f8250107df0 2024-07-26T06:53:47.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.584+0000 7f825c546740 1 -- 172.21.15.42:0/112343667 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 msgr2=0x5592bbd4dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:47.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.584+0000 7f824ffff640 1 --2- 172.21.15.42:0/4247520530 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82380b3bd0 0x7f82380b60e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8248009fd0 tx=0x7f8248009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:47.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.584+0000 7f825c546740 1 --2- 172.21.15.42:0/112343667 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 0x5592bbd4dd80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f82400044b0 tx=0x7f82400044e0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.584+0000 7f825c546740 1 -- 172.21.15.42:0/112343667 shutdown_connections 2024-07-26T06:53:47.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.584+0000 7f825c546740 1 --2- 172.21.15.42:0/112343667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb986b60 0x5592bbd4e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.584+0000 7f825c546740 1 --2- 172.21.15.42:0/112343667 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 0x5592bbd4dd80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.584+0000 7f825c546740 1 --2- 172.21.15.42:0/112343667 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb9a7840 0x5592bbd4d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.584+0000 7f825c546740 1 -- 172.21.15.42:0/112343667 >> 172.21.15.42:0/112343667 conn(0x5592bb99b990 msgr2=0x5592bbd4b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:47.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.584+0000 7f825c546740 1 -- 172.21.15.42:0/112343667 shutdown_connections 2024-07-26T06:53:47.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.584+0000 7f825c546740 1 -- 172.21.15.42:0/112343667 wait complete. 2024-07-26T06:53:47.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.584+0000 7f825c546740 1 -- start start 2024-07-26T06:53:47.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.584+0000 7f825c546740 1 Processor -- start 2024-07-26T06:53:47.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.585+0000 7f825c546740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 0x5592bbd69df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.585+0000 7f825c546740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb986b60 0x5592bbd6a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.585+0000 7f825c546740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb9a7840 0x5592bbd6a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.585+0000 7f825c546740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5592bbb157a0 con 0x5592bb9a7840 2024-07-26T06:53:47.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.585+0000 7f825c546740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5592bbc542f0 con 0x5592bb986b60 2024-07-26T06:53:47.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.585+0000 7f825c546740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5592bbd6cef0 con 0x5592bb97e980 2024-07-26T06:53:47.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.585+0000 7f825c546740 10 client.0 _ll_register_callbacks cb 0x5592bbc5e110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:53:47.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.585+0000 7f824f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 0x5592bbd69df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.585+0000 7f824effd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb986b60 0x5592bbd6a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.585+0000 7f824f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 0x5592bbd69df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:48788/0 (socket says 172.21.15.42:48788) 2024-07-26T06:53:47.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.585+0000 7f824ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb9a7840 0x5592bbd6a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.585+0000 7f824f7fe640 1 -- 172.21.15.42:0/261202770 learned_addr learned my addr 172.21.15.42:0/261202770 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:47.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.585+0000 7f824f7fe640 1 -- 172.21.15.42:0/261202770 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb986b60 msgr2=0x5592bbd6a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:47.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.585+0000 7f824f7fe640 1 --2- 172.21.15.42:0/261202770 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb986b60 0x5592bbd6a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.585+0000 7f824f7fe640 1 -- 172.21.15.42:0/261202770 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb9a7840 msgr2=0x5592bbd6a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:47.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.585+0000 7f824f7fe640 1 --2- 172.21.15.42:0/261202770 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb9a7840 0x5592bbd6a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.585+0000 7f824f7fe640 1 -- 172.21.15.42:0/261202770 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592bbc58480 con 0x5592bb97e980 2024-07-26T06:53:47.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.586+0000 7f824f7fe640 1 --2- 172.21.15.42:0/261202770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 0x5592bbd69df0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8240005ee0 tx=0x7f8240004b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51cb4061a6b3db2e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:47.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.586+0000 7f822effd640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-07-26T06:53:47.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.586+0000 7f822effd640 1 -- 172.21.15.42:0/261202770 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8240035070 con 0x5592bb97e980 2024-07-26T06:53:47.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.586+0000 7f822effd640 1 -- 172.21.15.42:0/261202770 <== mon.1 v2:172.21.15.164:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f8240003a20 con 0x5592bb97e980 2024-07-26T06:53:47.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.586+0000 7f825c546740 1 -- 172.21.15.42:0/261202770 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82400047a0 con 0x5592bb97e980 2024-07-26T06:53:47.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.586+0000 7f822effd640 1 -- 172.21.15.42:0/261202770 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8240003c20 con 0x5592bb97e980 2024-07-26T06:53:47.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.586+0000 7f825c546740 10 client.25474 fetch_fsmap learned FSMap version 36 2024-07-26T06:53:47.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.586+0000 7f825c546740 1 -- 172.21.15.42:0/261202770 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5592bba21b90 con 0x5592bb97e980 2024-07-26T06:53:47.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.587+0000 7f822effd640 1 -- 172.21.15.42:0/261202770 <== mon.1 v2:172.21.15.164:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82400047a0 con 0x5592bb97e980 2024-07-26T06:53:47.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.587+0000 7f822effd640 1 -- 172.21.15.42:0/261202770 <== mon.1 v2:172.21.15.164:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f8240031660 con 0x5592bb97e980 2024-07-26T06:53:47.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.587+0000 7f825c546740 10 client.25474 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:53:47.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.587+0000 7f825c546740 10 client.25474 resolve_mds: validated gid 24620 aka daemon mds.b 2024-07-26T06:53:47.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.587+0000 7f825c546740 20 client.25474 populate_metadata read hostname 'smithi042' 2024-07-26T06:53:47.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.587+0000 7f825c546740 1 --2- 172.21.15.42:0/261202770 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5592bb9661c0 0x5592bbd7bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.587+0000 7f825c546740 4 client.25474 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:53:47.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.587+0000 7f825c546740 1 -- 172.21.15.42:0/261202770 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592bbcc46d0 con 0x5592bb9661c0 2024-07-26T06:53:47.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.587+0000 7f824effd640 1 --2- 172.21.15.42:0/261202770 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5592bb9661c0 0x5592bbd7bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.588+0000 7f822effd640 10 client.25474 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:47.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.588+0000 7f824effd640 1 --2- 172.21.15.42:0/261202770 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5592bb9661c0 0x5592bbd7bc70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed973c0491716ca5 server_cookie=939106e8de779d43 in_seq=0 out_seq=0 2024-07-26T06:53:47.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.588+0000 7f822effd640 1 -- 172.21.15.42:0/261202770 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5592bbcc46d0 con 0x5592bb9661c0 2024-07-26T06:53:47.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.588+0000 7f822effd640 10 client.25474 handle_command_reply: tid=0 2024-07-26T06:53:47.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.588+0000 7f825c546740 1 client.25474 shutdown 2024-07-26T06:53:47.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.589+0000 7f825c546740 1 -- 172.21.15.42:0/261202770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 msgr2=0x5592bbd69df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:47.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.589+0000 7f825c546740 1 --2- 172.21.15.42:0/261202770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 0x5592bbd69df0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8240005ee0 tx=0x7f8240004b80 comp rx=0 tx=0).stop 2024-07-26T06:53:47.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.589+0000 7f825c546740 1 -- 172.21.15.42:0/261202770 shutdown_connections 2024-07-26T06:53:47.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.589+0000 7f825c546740 1 --2- 172.21.15.42:0/261202770 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5592bb9661c0 0x5592bbd7bc70 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.589+0000 7f825c546740 1 --2- 172.21.15.42:0/261202770 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb9a7840 0x5592bbd6a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.589+0000 7f825c546740 1 --2- 172.21.15.42:0/261202770 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb986b60 0x5592bbd6a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.589+0000 7f822effd640 0 client.25474 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:47.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.589+0000 7f825c546740 1 --2- 172.21.15.42:0/261202770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 0x5592bbd69df0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.589+0000 7f825c546740 1 -- 172.21.15.42:0/261202770 >> 172.21.15.42:0/261202770 conn(0x5592bb99b990 msgr2=0x5592bbd509d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:47.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.589+0000 7f825c546740 1 -- 172.21.15.42:0/261202770 shutdown_connections 2024-07-26T06:53:47.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.589+0000 7f825c546740 1 -- 172.21.15.42:0/261202770 wait complete. 2024-07-26T06:53:47.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.589+0000 7f825c546740 20 client.25474 trim_cache size 0 max 16384 2024-07-26T06:53:47.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.608+0000 7f825c546740 1 Processor -- start 2024-07-26T06:53:47.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.608+0000 7f825c546740 1 -- start start 2024-07-26T06:53:47.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.609+0000 7f825c546740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 0x5592bbd6bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.609+0000 7f825c546740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb986b60 0x5592bbd52b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.609+0000 7f825c546740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb9a7840 0x5592bbd76010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.609+0000 7f825c546740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8250074c80 con 0x5592bb986b60 2024-07-26T06:53:47.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.609+0000 7f825c546740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f82501aa530 con 0x5592bb9a7840 2024-07-26T06:53:47.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.609+0000 7f825c546740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f82501a9000 con 0x5592bb97e980 2024-07-26T06:53:47.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.609+0000 7f824effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb986b60 0x5592bbd52b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.609+0000 7f824effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb986b60 0x5592bbd52b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36146/0 (socket says 172.21.15.42:36146) 2024-07-26T06:53:47.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.609+0000 7f824effd640 1 -- 172.21.15.42:0/1825372544 learned_addr learned my addr 172.21.15.42:0/1825372544 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:47.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.609+0000 7f824f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 0x5592bbd6bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.609+0000 7f824ffff640 1 --2- 172.21.15.42:0/1825372544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb9a7840 0x5592bbd76010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.609+0000 7f824effd640 1 -- 172.21.15.42:0/1825372544 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 msgr2=0x5592bbd6bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:47.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.609+0000 7f824effd640 1 --2- 172.21.15.42:0/1825372544 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 0x5592bbd6bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.609+0000 7f824effd640 1 -- 172.21.15.42:0/1825372544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb9a7840 msgr2=0x5592bbd76010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:47.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.609+0000 7f824effd640 1 --2- 172.21.15.42:0/1825372544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb9a7840 0x5592bbd76010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.609+0000 7f824effd640 1 -- 172.21.15.42:0/1825372544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592bbd682f0 con 0x5592bb986b60 2024-07-26T06:53:47.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.610+0000 7f824ffff640 1 --2- 172.21.15.42:0/1825372544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb9a7840 0x5592bbd76010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:53:47.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.610+0000 7f824effd640 1 --2- 172.21.15.42:0/1825372544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb986b60 0x5592bbd52b30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f82440a6fa0 tx=0x7f82440e75e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db893a1b2e6a5021 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:47.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.610+0000 7f82357fa640 1 -- 172.21.15.42:0/1825372544 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82440af470 con 0x5592bb986b60 2024-07-26T06:53:47.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.610+0000 7f82357fa640 1 -- 172.21.15.42:0/1825372544 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82440b3f20 con 0x5592bb986b60 2024-07-26T06:53:47.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.610+0000 7f82357fa640 1 -- 172.21.15.42:0/1825372544 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82440e60b0 con 0x5592bb986b60 2024-07-26T06:53:47.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.611+0000 7f825c546740 1 -- 172.21.15.42:0/1825372544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb986b60 msgr2=0x5592bbd52b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:47.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.611+0000 7f825c546740 1 --2- 172.21.15.42:0/1825372544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb986b60 0x5592bbd52b30 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f82440a6fa0 tx=0x7f82440e75e0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.611+0000 7f825c546740 1 -- 172.21.15.42:0/1825372544 shutdown_connections 2024-07-26T06:53:47.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.611+0000 7f825c546740 1 --2- 172.21.15.42:0/1825372544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb9a7840 0x5592bbd76010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.611+0000 7f825c546740 1 --2- 172.21.15.42:0/1825372544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb986b60 0x5592bbd52b30 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.611+0000 7f825c546740 1 --2- 172.21.15.42:0/1825372544 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb97e980 0x5592bbd6bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.611+0000 7f825c546740 1 -- 172.21.15.42:0/1825372544 >> 172.21.15.42:0/1825372544 conn(0x5592bb99b990 msgr2=0x5592bbd4f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:47.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.611+0000 7f825c546740 1 -- 172.21.15.42:0/1825372544 shutdown_connections 2024-07-26T06:53:47.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.611+0000 7f825c546740 1 -- 172.21.15.42:0/1825372544 wait complete. 2024-07-26T06:53:47.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.612+0000 7f825c546740 1 -- start start 2024-07-26T06:53:47.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.612+0000 7f825c546740 1 Processor -- start 2024-07-26T06:53:47.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.612+0000 7f825c546740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb97e980 0x5592bbd72a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.612+0000 7f825c546740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb986b60 0x5592bbd6deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.612+0000 7f825c546740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb9a7840 0x5592bbd6e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.612+0000 7f825c546740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f82501a9e60 con 0x5592bb9a7840 2024-07-26T06:53:47.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.612+0000 7f825c546740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f82501a96d0 con 0x5592bb97e980 2024-07-26T06:53:47.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.612+0000 7f825c546740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f82501028c0 con 0x5592bb986b60 2024-07-26T06:53:47.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f824ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb9a7840 0x5592bbd6e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f825c546740 10 client.0 _ll_register_callbacks cb 0x7f8240009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:53:47.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f824effd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb986b60 0x5592bbd6deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f824f7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb97e980 0x5592bbd72a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f824ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb9a7840 0x5592bbd6e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36152/0 (socket says 172.21.15.42:36152) 2024-07-26T06:53:47.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f824effd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb986b60 0x5592bbd6deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:48812/0 (socket says 172.21.15.42:48812) 2024-07-26T06:53:47.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f824f7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb97e980 0x5592bbd72a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:51884/0 (socket says 172.21.15.42:51884) 2024-07-26T06:53:47.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f824ffff640 1 -- 172.21.15.42:0/3653847948 learned_addr learned my addr 172.21.15.42:0/3653847948 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:47.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f824ffff640 1 -- 172.21.15.42:0/3653847948 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb986b60 msgr2=0x5592bbd6deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:47.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f824ffff640 1 --2- 172.21.15.42:0/3653847948 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb986b60 0x5592bbd6deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f824ffff640 1 -- 172.21.15.42:0/3653847948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb97e980 msgr2=0x5592bbd72a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:47.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f824ffff640 1 --2- 172.21.15.42:0/3653847948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb97e980 0x5592bbd72a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:47.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f824ffff640 1 -- 172.21.15.42:0/3653847948 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592bbde31e0 con 0x5592bb9a7840 2024-07-26T06:53:47.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f824ffff640 1 --2- 172.21.15.42:0/3653847948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb9a7840 0x5592bbd6e5a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f824800fb30 tx=0x7f8248004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a129000890f44a2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:47.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f822effd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:53:47.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f822effd640 1 -- 172.21.15.42:0/3653847948 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8248012020 con 0x5592bb9a7840 2024-07-26T06:53:47.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f822effd640 1 -- 172.21.15.42:0/3653847948 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f82480121c0 con 0x5592bb9a7840 2024-07-26T06:53:47.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.613+0000 7f825c546740 1 -- 172.21.15.42:0/3653847948 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82440b3f20 con 0x5592bb9a7840 2024-07-26T06:53:47.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.614+0000 7f822effd640 1 -- 172.21.15.42:0/3653847948 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8248011440 con 0x5592bb9a7840 2024-07-26T06:53:47.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.614+0000 7f825c546740 10 client.15834 fetch_fsmap learned FSMap version 36 2024-07-26T06:53:47.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.614+0000 7f825c546740 1 -- 172.21.15.42:0/3653847948 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5592bbde4310 con 0x5592bb9a7840 2024-07-26T06:53:47.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.614+0000 7f822effd640 1 -- 172.21.15.42:0/3653847948 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82440b3f20 con 0x5592bb9a7840 2024-07-26T06:53:47.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.614+0000 7f822effd640 1 -- 172.21.15.42:0/3653847948 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f824801f5c0 con 0x5592bb9a7840 2024-07-26T06:53:47.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.614+0000 7f825c546740 10 client.15834 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:53:47.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.614+0000 7f825c546740 10 client.15834 resolve_mds: validated gid 24620 aka daemon mds.b 2024-07-26T06:53:47.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.614+0000 7f825c546740 20 client.15834 populate_metadata read hostname 'smithi042' 2024-07-26T06:53:47.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.614+0000 7f825c546740 1 --2- 172.21.15.42:0/3653847948 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5592bb93b570 0x5592bbde6c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:47.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.614+0000 7f825c546740 4 client.15834 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 151.1, "await_for": 45.0}] 2024-07-26T06:53:47.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.614+0000 7f825c546740 1 -- 172.21.15.42:0/3653847948 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 151.1, "await_for": 45.0}) -- 0x5592bbcc0170 con 0x5592bb93b570 2024-07-26T06:53:47.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.614+0000 7f824f7fe640 1 --2- 172.21.15.42:0/3653847948 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5592bb93b570 0x5592bbde6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:47.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.615+0000 7f822effd640 10 client.15834 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:47.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:47.615+0000 7f824f7fe640 1 --2- 172.21.15.42:0/3653847948 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5592bb93b570 0x5592bbde6c80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c0693dbb7911277 server_cookie=d925ca409b6f2905 in_seq=0 out_seq=0 2024-07-26T06:53:48.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.187+0000 7f822effd640 1 -- 172.21.15.42:0/3653847948 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5592bbcc0170 con 0x5592bb93b570 2024-07-26T06:53:48.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.188+0000 7f822effd640 10 client.15834 handle_command_reply: tid=0 2024-07-26T06:53:48.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.188+0000 7f825c546740 1 client.15834 shutdown 2024-07-26T06:53:48.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.188+0000 7f825c546740 1 -- 172.21.15.42:0/3653847948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb9a7840 msgr2=0x5592bbd6e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:48.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.188+0000 7f825c546740 1 --2- 172.21.15.42:0/3653847948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb9a7840 0x5592bbd6e5a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f824800fb30 tx=0x7f8248004a40 comp rx=0 tx=0).stop 2024-07-26T06:53:48.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.188+0000 7f825c546740 1 -- 172.21.15.42:0/3653847948 shutdown_connections 2024-07-26T06:53:48.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.188+0000 7f825c546740 1 --2- 172.21.15.42:0/3653847948 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5592bb93b570 0x5592bbde6c80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:48.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.188+0000 7f822effd640 0 client.15834 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:48.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.188+0000 7f825c546740 1 --2- 172.21.15.42:0/3653847948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5592bb9a7840 0x5592bbd6e5a0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:48.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.188+0000 7f825c546740 1 --2- 172.21.15.42:0/3653847948 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5592bb986b60 0x5592bbd6deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:48.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.188+0000 7f825c546740 1 --2- 172.21.15.42:0/3653847948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5592bb97e980 0x5592bbd72a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:48.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.188+0000 7f825c546740 1 -- 172.21.15.42:0/3653847948 >> 172.21.15.42:0/3653847948 conn(0x5592bb99b990 msgr2=0x5592bbd4e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:48.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.188+0000 7f825c546740 1 -- 172.21.15.42:0/3653847948 shutdown_connections 2024-07-26T06:53:48.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.188+0000 7f825c546740 1 -- 172.21.15.42:0/3653847948 wait complete. 2024-07-26T06:53:48.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.188+0000 7f825c546740 20 client.15834 trim_cache size 0 max 16384 2024-07-26T06:53:48.192 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:53:48.192 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 36, 2024-07-26T06:53:48.192 INFO:teuthology.orchestra.run.smithi042.stdout: "leader": 24620, 2024-07-26T06:53:48.192 INFO:teuthology.orchestra.run.smithi042.stdout: "set_version": 10, 2024-07-26T06:53:48.192 INFO:teuthology.orchestra.run.smithi042.stdout: "sets": { 2024-07-26T06:53:48.192 INFO:teuthology.orchestra.run.smithi042.stdout: "c8d66e1d": { 2024-07-26T06:53:48.192 INFO:teuthology.orchestra.run.smithi042.stdout: "version": 10, 2024-07-26T06:53:48.192 INFO:teuthology.orchestra.run.smithi042.stdout: "age_ref": 0.0, 2024-07-26T06:53:48.192 INFO:teuthology.orchestra.run.smithi042.stdout: "state": { 2024-07-26T06:53:48.192 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "QUIESCED", 2024-07-26T06:53:48.193 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 0.0 2024-07-26T06:53:48.193 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-07-26T06:53:48.193 INFO:teuthology.orchestra.run.smithi042.stdout: "timeout": 90.0, 2024-07-26T06:53:48.193 INFO:teuthology.orchestra.run.smithi042.stdout: "expiration": 151.1, 2024-07-26T06:53:48.193 INFO:teuthology.orchestra.run.smithi042.stdout: "members": { 2024-07-26T06:53:48.193 INFO:teuthology.orchestra.run.smithi042.stdout: "file:/": { 2024-07-26T06:53:48.193 INFO:teuthology.orchestra.run.smithi042.stdout: "excluded": false, 2024-07-26T06:53:48.193 INFO:teuthology.orchestra.run.smithi042.stdout: "state": { 2024-07-26T06:53:48.193 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "QUIESCED", 2024-07-26T06:53:48.193 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 0.0 2024-07-26T06:53:48.193 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:53:48.193 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:53:48.194 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:53:48.194 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:53:48.194 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:53:48.194 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:53:48.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.189+0000 7f8255fc2640 1 -- 172.21.15.42:0/4247520530 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82380b3bd0 msgr2=0x7f82380b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:48.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.189+0000 7f8255fc2640 1 --2- 172.21.15.42:0/4247520530 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82380b3bd0 0x7f82380b60e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8248009fd0 tx=0x7f8248009500 comp rx=0 tx=0).stop 2024-07-26T06:53:48.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.189+0000 7f8255fc2640 1 -- 172.21.15.42:0/4247520530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8250107df0 msgr2=0x7f825019c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:48.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.189+0000 7f8255fc2640 1 --2- 172.21.15.42:0/4247520530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8250107df0 0x7f825019c6b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f824400e9c0 tx=0x7f824400ee80 comp rx=0 tx=0).stop 2024-07-26T06:53:48.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.189+0000 7f8255fc2640 1 -- 172.21.15.42:0/4247520530 shutdown_connections 2024-07-26T06:53:48.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.189+0000 7f8255fc2640 1 --2- 172.21.15.42:0/4247520530 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f82380b3bd0 0x7f82380b60e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:48.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.189+0000 7f8255fc2640 1 --2- 172.21.15.42:0/4247520530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8250108740 0x7f82501a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:48.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.189+0000 7f8255fc2640 1 --2- 172.21.15.42:0/4247520530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8250107df0 0x7f825019c6b0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:48.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.189+0000 7f8255fc2640 1 --2- 172.21.15.42:0/4247520530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f82501024b0 0x7f825019c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:48.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.189+0000 7f8255fc2640 1 -- 172.21.15.42:0/4247520530 >> 172.21.15.42:0/4247520530 conn(0x7f8250079190 msgr2=0x7f8250100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:48.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.189+0000 7f8255fc2640 1 -- 172.21.15.42:0/4247520530 shutdown_connections 2024-07-26T06:53:48.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:53:48.189+0000 7f8255fc2640 1 -- 172.21.15.42:0/4247520530 wait complete. 2024-07-26T06:53:48.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:48 smithi042 ceph-mon[32775]: pgmap v504: 97 pgs: 97 active+clean; 52 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.9 MiB/s wr, 725 op/s 2024-07-26T06:53:48.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:48 smithi164 ceph-mon[36970]: pgmap v504: 97 pgs: 97 active+clean; 52 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.9 MiB/s wr, 725 op/s 2024-07-26T06:53:48.982 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c8d66e1d', quiesce took 6.9 seconds. Will release after: 24.200000000000003 2024-07-26T06:53:49.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:48 smithi163 ceph-mon[36873]: pgmap v504: 97 pgs: 97 active+clean; 52 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.9 MiB/s wr, 725 op/s 2024-07-26T06:53:50.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:50 smithi042 ceph-mon[32775]: pgmap v505: 97 pgs: 97 active+clean; 45 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 752 op/s 2024-07-26T06:53:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:50 smithi164 ceph-mon[36970]: pgmap v505: 97 pgs: 97 active+clean; 45 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 752 op/s 2024-07-26T06:53:51.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:50 smithi163 ceph-mon[36873]: pgmap v505: 97 pgs: 97 active+clean; 45 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 752 op/s 2024-07-26T06:53:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:52 smithi042 ceph-mon[32775]: pgmap v506: 97 pgs: 97 active+clean; 46 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.5 MiB/s wr, 646 op/s 2024-07-26T06:53:52.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:52 smithi164 ceph-mon[36970]: pgmap v506: 97 pgs: 97 active+clean; 46 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.5 MiB/s wr, 646 op/s 2024-07-26T06:53:53.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:52 smithi163 ceph-mon[36873]: pgmap v506: 97 pgs: 97 active+clean; 46 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.5 MiB/s wr, 646 op/s 2024-07-26T06:53:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:54 smithi042 ceph-mon[32775]: pgmap v507: 97 pgs: 97 active+clean; 45 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 3.1 MiB/s wr, 586 op/s 2024-07-26T06:53:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:54 smithi164 ceph-mon[36970]: pgmap v507: 97 pgs: 97 active+clean; 45 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 3.1 MiB/s wr, 586 op/s 2024-07-26T06:53:55.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:54 smithi163 ceph-mon[36873]: pgmap v507: 97 pgs: 97 active+clean; 45 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 3.1 MiB/s wr, 586 op/s 2024-07-26T06:53:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:56 smithi164 ceph-mon[36970]: pgmap v508: 97 pgs: 97 active+clean; 45 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 2.6 MiB/s wr, 497 op/s 2024-07-26T06:53:57.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:56 smithi163 ceph-mon[36873]: pgmap v508: 97 pgs: 97 active+clean; 45 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 2.6 MiB/s wr, 497 op/s 2024-07-26T06:53:57.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:56 smithi042 ceph-mon[32775]: pgmap v508: 97 pgs: 97 active+clean; 45 MiB data, 825 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 2.6 MiB/s wr, 497 op/s 2024-07-26T06:53:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:53:58 smithi164 ceph-mon[36970]: pgmap v509: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 1.7 MiB/s wr, 310 op/s 2024-07-26T06:53:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:53:58 smithi163 ceph-mon[36873]: pgmap v509: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 1.7 MiB/s wr, 310 op/s 2024-07-26T06:53:59.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:53:58 smithi042 ceph-mon[32775]: pgmap v509: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 1.7 MiB/s wr, 310 op/s 2024-07-26T06:53:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.294+0000 7f9487276640 1 -- 172.21.15.42:0/2201282421 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9480106550 msgr2=0x7f9480108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.294+0000 7f9487276640 1 --2- 172.21.15.42:0/2201282421 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9480106550 0x7f9480108970 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9470009900 tx=0x7f947002a410 comp rx=0 tx=0).stop 2024-07-26T06:53:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.294+0000 7f9487276640 1 -- 172.21.15.42:0/2201282421 shutdown_connections 2024-07-26T06:53:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.294+0000 7f9487276640 1 --2- 172.21.15.42:0/2201282421 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9480108eb0 0x7f948010b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.294+0000 7f9487276640 1 --2- 172.21.15.42:0/2201282421 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9480106550 0x7f9480108970 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.294+0000 7f9487276640 1 --2- 172.21.15.42:0/2201282421 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94801014c0 0x7f94801018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.294+0000 7f9487276640 1 -- 172.21.15.42:0/2201282421 >> 172.21.15.42:0/2201282421 conn(0x7f94800fd100 msgr2=0x7f94800ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.294+0000 7f9487276640 1 -- 172.21.15.42:0/2201282421 shutdown_connections 2024-07-26T06:53:59.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.294+0000 7f9487276640 1 -- 172.21.15.42:0/2201282421 wait complete. 2024-07-26T06:53:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.295+0000 7f9487276640 1 Processor -- start 2024-07-26T06:53:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.295+0000 7f9487276640 1 -- start start 2024-07-26T06:53:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.295+0000 7f9487276640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94801014c0 0x7f9480103b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.295+0000 7f9487276640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9480106550 0x7f9480102450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.295+0000 7f9487276640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9480108eb0 0x7f9480102990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.295+0000 7f9487276640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f948010ed60 con 0x7f9480106550 2024-07-26T06:53:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.295+0000 7f9487276640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f948010ebe0 con 0x7f9480108eb0 2024-07-26T06:53:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.295+0000 7f9487276640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f948010eee0 con 0x7f94801014c0 2024-07-26T06:53:59.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.295+0000 7f9484811640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9480106550 0x7f9480102450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.295+0000 7f9484811640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9480106550 0x7f9480102450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52860/0 (socket says 172.21.15.42:52860) 2024-07-26T06:53:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.295+0000 7f9484811640 1 -- 172.21.15.42:0/3067327873 learned_addr learned my addr 172.21.15.42:0/3067327873 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.295+0000 7f9485813640 1 --2- 172.21.15.42:0/3067327873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9480108eb0 0x7f9480102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.295+0000 7f9485012640 1 --2- 172.21.15.42:0/3067327873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94801014c0 0x7f9480103b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.296+0000 7f9484811640 1 -- 172.21.15.42:0/3067327873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94801014c0 msgr2=0x7f9480103b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.296+0000 7f9484811640 1 --2- 172.21.15.42:0/3067327873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94801014c0 0x7f9480103b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.296+0000 7f9484811640 1 -- 172.21.15.42:0/3067327873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9480108eb0 msgr2=0x7f9480102990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.296+0000 7f9484811640 1 --2- 172.21.15.42:0/3067327873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9480108eb0 0x7f9480102990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.296+0000 7f9484811640 1 -- 172.21.15.42:0/3067327873 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94801030c0 con 0x7f9480106550 2024-07-26T06:53:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.296+0000 7f9485012640 1 --2- 172.21.15.42:0/3067327873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94801014c0 0x7f9480103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:53:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.296+0000 7f9484811640 1 --2- 172.21.15.42:0/3067327873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9480106550 0x7f9480102450 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9470006010 tx=0x7f9470031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.296+0000 7f94767fc640 1 -- 172.21.15.42:0/3067327873 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947003f030 con 0x7f9480106550 2024-07-26T06:53:59.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.296+0000 7f94767fc640 1 -- 172.21.15.42:0/3067327873 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9470003aa0 con 0x7f9480106550 2024-07-26T06:53:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.296+0000 7f9487276640 1 -- 172.21.15.42:0/3067327873 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94801a39f0 con 0x7f9480106550 2024-07-26T06:53:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.296+0000 7f94767fc640 1 -- 172.21.15.42:0/3067327873 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9470035070 con 0x7f9480106550 2024-07-26T06:53:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.296+0000 7f9487276640 1 -- 172.21.15.42:0/3067327873 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94801a3df0 con 0x7f9480106550 2024-07-26T06:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.299+0000 7f948e18d740 1 Processor -- start 2024-07-26T06:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.299+0000 7f948e18d740 1 -- start start 2024-07-26T06:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.299+0000 7f94767fc640 1 -- 172.21.15.42:0/3067327873 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f9470042030 con 0x7f9480106550 2024-07-26T06:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.299+0000 7f948e18d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4701b0 0x55a52b815180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.299+0000 7f948e18d740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b4472f0 0x55a52b8157f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.299+0000 7f948e18d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b44f4d0 0x55a52b816340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.299+0000 7f948e18d740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55a52b7883b0 con 0x55a52b4472f0 2024-07-26T06:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.299+0000 7f948e18d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a52b7778b0 con 0x55a52b4701b0 2024-07-26T06:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.299+0000 7f948e18d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55a52b77b8f0 con 0x55a52b44f4d0 2024-07-26T06:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.299+0000 7f9485012640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b4472f0 0x55a52b8157f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.299+0000 7f9485012640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b4472f0 0x55a52b8157f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52866/0 (socket says 172.21.15.42:52866) 2024-07-26T06:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.299+0000 7f9485012640 1 -- 172.21.15.42:0/1755803806 learned_addr learned my addr 172.21.15.42:0/1755803806 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.299+0000 7f9485813640 1 --2- 172.21.15.42:0/1755803806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4701b0 0x55a52b815180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.300+0000 7f9484811640 1 --2- 172.21.15.42:0/1755803806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b44f4d0 0x55a52b816340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.300+0000 7f9485012640 1 -- 172.21.15.42:0/1755803806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b44f4d0 msgr2=0x55a52b816340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:59.306 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.300+0000 7f9485012640 1 --2- 172.21.15.42:0/1755803806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b44f4d0 0x55a52b816340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.300+0000 7f9485012640 1 -- 172.21.15.42:0/1755803806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4701b0 msgr2=0x55a52b815180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.300+0000 7f9485012640 1 --2- 172.21.15.42:0/1755803806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4701b0 0x55a52b815180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.300+0000 7f9485012640 1 -- 172.21.15.42:0/1755803806 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a52b71e9a0 con 0x55a52b4472f0 2024-07-26T06:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.300+0000 7f9485813640 1 --2- 172.21.15.42:0/1755803806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4701b0 0x55a52b815180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.300+0000 7f94767fc640 1 --2- 172.21.15.42:0/3067327873 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f946c0b3bd0 0x7f946c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.300+0000 7f9485012640 1 --2- 172.21.15.42:0/1755803806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b4472f0 0x55a52b8157f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f947c00bd40 tx=0x7f947c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33678225d385c903 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.300+0000 7f9485813640 1 --2- 172.21.15.42:0/3067327873 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f946c0b3bd0 0x7f946c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.300+0000 7f946b7fe640 1 -- 172.21.15.42:0/1755803806 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c015070 con 0x55a52b4472f0 2024-07-26T06:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.300+0000 7f94767fc640 1 -- 172.21.15.42:0/3067327873 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f9470104220 con 0x7f9480106550 2024-07-26T06:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.300+0000 7f946b7fe640 1 -- 172.21.15.42:0/1755803806 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f947c00be30 con 0x55a52b4472f0 2024-07-26T06:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.300+0000 7f946b7fe640 1 -- 172.21.15.42:0/1755803806 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c011940 con 0x55a52b4472f0 2024-07-26T06:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.301+0000 7f9485813640 1 --2- 172.21.15.42:0/3067327873 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f946c0b3bd0 0x7f946c0b60e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9478009fd0 tx=0x7f9478009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.301+0000 7f948e18d740 1 -- 172.21.15.42:0/1755803806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b4472f0 msgr2=0x55a52b8157f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.301+0000 7f948e18d740 1 --2- 172.21.15.42:0/1755803806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b4472f0 0x55a52b8157f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f947c00bd40 tx=0x7f947c00f730 comp rx=0 tx=0).stop 2024-07-26T06:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.301+0000 7f948e18d740 1 -- 172.21.15.42:0/1755803806 shutdown_connections 2024-07-26T06:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.301+0000 7f948e18d740 1 --2- 172.21.15.42:0/1755803806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b44f4d0 0x55a52b816340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.301+0000 7f948e18d740 1 --2- 172.21.15.42:0/1755803806 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b4472f0 0x55a52b8157f0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.301+0000 7f948e18d740 1 --2- 172.21.15.42:0/1755803806 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4701b0 0x55a52b815180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.301+0000 7f948e18d740 1 -- 172.21.15.42:0/1755803806 >> 172.21.15.42:0/1755803806 conn(0x55a52b464300 msgr2=0x55a52b810c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.301+0000 7f948e18d740 1 -- 172.21.15.42:0/1755803806 shutdown_connections 2024-07-26T06:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.301+0000 7f948e18d740 1 -- 172.21.15.42:0/1755803806 wait complete. 2024-07-26T06:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.302+0000 7f948e18d740 1 -- start start 2024-07-26T06:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.302+0000 7f948e18d740 1 Processor -- start 2024-07-26T06:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.302+0000 7f948e18d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4472f0 0x55a52b81ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.302+0000 7f948e18d740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b44f4d0 0x55a52b81f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.302+0000 7f948e18d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b4701b0 0x55a52b81fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.302+0000 7f948e18d740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55a52b6b1ab0 con 0x55a52b44f4d0 2024-07-26T06:53:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.302+0000 7f948e18d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a52b6ffcf0 con 0x55a52b4472f0 2024-07-26T06:53:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.302+0000 7f948e18d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55a52b816e60 con 0x55a52b4701b0 2024-07-26T06:53:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.303+0000 7f948e18d740 10 client.0 _ll_register_callbacks cb 0x55a52b7719f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:53:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.303+0000 7f9484811640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b44f4d0 0x55a52b81f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.303+0000 7f9484811640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b44f4d0 0x55a52b81f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52880/0 (socket says 172.21.15.42:52880) 2024-07-26T06:53:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.303+0000 7f9484811640 1 -- 172.21.15.42:0/498725313 learned_addr learned my addr 172.21.15.42:0/498725313 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.303+0000 7f9485012640 1 --2- 172.21.15.42:0/498725313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4472f0 0x55a52b81ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.303+0000 7f9485813640 1 --2- 172.21.15.42:0/498725313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b4701b0 0x55a52b81fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.303+0000 7f9484811640 1 -- 172.21.15.42:0/498725313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b4701b0 msgr2=0x55a52b81fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.303+0000 7f9484811640 1 --2- 172.21.15.42:0/498725313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b4701b0 0x55a52b81fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.303+0000 7f9484811640 1 -- 172.21.15.42:0/498725313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4472f0 msgr2=0x55a52b81ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:59.310 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.303+0000 7f9484811640 1 --2- 172.21.15.42:0/498725313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4472f0 0x55a52b81ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.310 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.303+0000 7f9484811640 1 -- 172.21.15.42:0/498725313 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a52b4db790 con 0x55a52b44f4d0 2024-07-26T06:53:59.310 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.303+0000 7f9484811640 1 --2- 172.21.15.42:0/498725313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b44f4d0 0x55a52b81f270 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9470031440 tx=0x7f9470009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d190c64aa954d87c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:59.310 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.303+0000 7f9468ff9640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:53:59.310 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.303+0000 7f9468ff9640 1 -- 172.21.15.42:0/498725313 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9470049040 con 0x55a52b44f4d0 2024-07-26T06:53:59.310 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.303+0000 7f9468ff9640 1 -- 172.21.15.42:0/498725313 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f94700cb3b0 con 0x55a52b44f4d0 2024-07-26T06:53:59.310 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.303+0000 7f948e18d740 1 -- 172.21.15.42:0/498725313 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f947c00be30 con 0x55a52b44f4d0 2024-07-26T06:53:59.310 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.304+0000 7f9468ff9640 1 -- 172.21.15.42:0/498725313 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f94700ca8c0 con 0x55a52b44f4d0 2024-07-26T06:53:59.310 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.304+0000 7f948e18d740 10 client.15846 fetch_fsmap learned FSMap version 36 2024-07-26T06:53:59.310 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.304+0000 7f948e18d740 1 -- 172.21.15.42:0/498725313 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55a52b554620 con 0x55a52b44f4d0 2024-07-26T06:53:59.310 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.304+0000 7f9468ff9640 1 -- 172.21.15.42:0/498725313 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f947c00be30 con 0x55a52b44f4d0 2024-07-26T06:53:59.310 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.304+0000 7f9468ff9640 1 -- 172.21.15.42:0/498725313 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f94700d3d70 con 0x55a52b44f4d0 2024-07-26T06:53:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.304+0000 7f948e18d740 10 client.15846 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:53:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.304+0000 7f948e18d740 10 client.15846 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:53:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.304+0000 7f948e18d740 20 client.15846 populate_metadata read hostname 'smithi042' 2024-07-26T06:53:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.304+0000 7f948e18d740 1 --2- 172.21.15.42:0/498725313 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55a52b42ed80 0x55a52b82c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.304+0000 7f948e18d740 4 client.15846 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:53:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.304+0000 7f948e18d740 1 -- 172.21.15.42:0/498725313 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a52b78a300 con 0x55a52b42ed80 2024-07-26T06:53:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.304+0000 7f9485012640 1 --2- 172.21.15.42:0/498725313 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55a52b42ed80 0x55a52b82c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.305+0000 7f9468ff9640 10 client.15846 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:59.311 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.305+0000 7f9485012640 1 --2- 172.21.15.42:0/498725313 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55a52b42ed80 0x55a52b82c340 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0f0856419cbaee server_cookie=710d6e4a8d53bd3e in_seq=0 out_seq=0 2024-07-26T06:53:59.312 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.306+0000 7f9468ff9640 1 -- 172.21.15.42:0/498725313 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a52b78a300 con 0x55a52b42ed80 2024-07-26T06:53:59.312 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.306+0000 7f9468ff9640 10 client.15846 handle_command_reply: tid=0 2024-07-26T06:53:59.312 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.306+0000 7f948e18d740 1 client.15846 shutdown 2024-07-26T06:53:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.306+0000 7f948e18d740 1 -- 172.21.15.42:0/498725313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b44f4d0 msgr2=0x55a52b81f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.306+0000 7f948e18d740 1 --2- 172.21.15.42:0/498725313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b44f4d0 0x55a52b81f270 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9470031440 tx=0x7f9470009620 comp rx=0 tx=0).stop 2024-07-26T06:53:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.306+0000 7f948e18d740 1 -- 172.21.15.42:0/498725313 shutdown_connections 2024-07-26T06:53:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.306+0000 7f948e18d740 1 --2- 172.21.15.42:0/498725313 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55a52b42ed80 0x55a52b82c340 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.306+0000 7f948e18d740 1 --2- 172.21.15.42:0/498725313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b4701b0 0x55a52b81fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.306+0000 7f948e18d740 1 --2- 172.21.15.42:0/498725313 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b44f4d0 0x55a52b81f270 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.306+0000 7f9468ff9640 0 client.15846 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.306+0000 7f948e18d740 1 --2- 172.21.15.42:0/498725313 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4472f0 0x55a52b81ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.306+0000 7f948e18d740 1 -- 172.21.15.42:0/498725313 >> 172.21.15.42:0/498725313 conn(0x55a52b464300 msgr2=0x55a52b80f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.306+0000 7f948e18d740 1 -- 172.21.15.42:0/498725313 shutdown_connections 2024-07-26T06:53:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.306+0000 7f948e18d740 1 -- 172.21.15.42:0/498725313 wait complete. 2024-07-26T06:53:59.313 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.306+0000 7f948e18d740 20 client.15846 trim_cache size 0 max 16384 2024-07-26T06:53:59.323 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.321+0000 7f948e18d740 1 Processor -- start 2024-07-26T06:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f948e18d740 1 -- start start 2024-07-26T06:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f948e18d740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b42ed80 0x55a52b816010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f948e18d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4472f0 0x55a52b81d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f948e18d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b44f4d0 0x55a52b81d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f948e18d740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f94801acbc0 con 0x55a52b42ed80 2024-07-26T06:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f948e18d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f94801ae7f0 con 0x55a52b4472f0 2024-07-26T06:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f948e18d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f94801ad2c0 con 0x55a52b44f4d0 2024-07-26T06:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f9485012640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b42ed80 0x55a52b816010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f9485012640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b42ed80 0x55a52b816010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52886/0 (socket says 172.21.15.42:52886) 2024-07-26T06:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f9485012640 1 -- 172.21.15.42:0/654924864 learned_addr learned my addr 172.21.15.42:0/654924864 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f9485813640 1 --2- 172.21.15.42:0/654924864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b44f4d0 0x55a52b81d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f9484811640 1 --2- 172.21.15.42:0/654924864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4472f0 0x55a52b81d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f9485012640 1 -- 172.21.15.42:0/654924864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b44f4d0 msgr2=0x55a52b81d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f9485012640 1 --2- 172.21.15.42:0/654924864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b44f4d0 0x55a52b81d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f9485012640 1 -- 172.21.15.42:0/654924864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4472f0 msgr2=0x55a52b81d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f9485012640 1 --2- 172.21.15.42:0/654924864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4472f0 0x55a52b81d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f9485012640 1 -- 172.21.15.42:0/654924864 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a52b81e4e0 con 0x55a52b42ed80 2024-07-26T06:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f9485813640 1 --2- 172.21.15.42:0/654924864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b44f4d0 0x55a52b81d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f9484811640 1 --2- 172.21.15.42:0/654924864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4472f0 0x55a52b81d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.322+0000 7f9485012640 1 --2- 172.21.15.42:0/654924864 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b42ed80 0x55a52b816010 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f947c00bd40 tx=0x7f947c00b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882111e6c5ba774e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.323+0000 7f946affd640 1 -- 172.21.15.42:0/654924864 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c015040 con 0x55a52b42ed80 2024-07-26T06:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.323+0000 7f946affd640 1 -- 172.21.15.42:0/654924864 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f947c004680 con 0x55a52b42ed80 2024-07-26T06:53:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.323+0000 7f946affd640 1 -- 172.21.15.42:0/654924864 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c011850 con 0x55a52b42ed80 2024-07-26T06:53:59.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.323+0000 7f948e18d740 1 -- 172.21.15.42:0/654924864 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b42ed80 msgr2=0x55a52b816010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:59.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.323+0000 7f948e18d740 1 --2- 172.21.15.42:0/654924864 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b42ed80 0x55a52b816010 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f947c00bd40 tx=0x7f947c00b920 comp rx=0 tx=0).stop 2024-07-26T06:53:59.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.324+0000 7f948e18d740 1 -- 172.21.15.42:0/654924864 shutdown_connections 2024-07-26T06:53:59.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.324+0000 7f948e18d740 1 --2- 172.21.15.42:0/654924864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b44f4d0 0x55a52b81d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.324+0000 7f948e18d740 1 --2- 172.21.15.42:0/654924864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4472f0 0x55a52b81d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.324+0000 7f948e18d740 1 --2- 172.21.15.42:0/654924864 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b42ed80 0x55a52b816010 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.324+0000 7f948e18d740 1 -- 172.21.15.42:0/654924864 >> 172.21.15.42:0/654924864 conn(0x55a52b464300 msgr2=0x55a52b81ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:59.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.324+0000 7f948e18d740 1 -- 172.21.15.42:0/654924864 shutdown_connections 2024-07-26T06:53:59.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.324+0000 7f948e18d740 1 -- 172.21.15.42:0/654924864 wait complete. 2024-07-26T06:53:59.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.324+0000 7f948e18d740 1 -- start start 2024-07-26T06:53:59.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.324+0000 7f948e18d740 1 Processor -- start 2024-07-26T06:53:59.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.324+0000 7f948e18d740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b42ed80 0x55a52b826870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f948e18d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4472f0 0x55a52b825100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f948e18d740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b44f4d0 0x55a52b825640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f948e18d740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f94801ae120 con 0x55a52b42ed80 2024-07-26T06:53:59.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f948e18d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f94801ad990 con 0x55a52b4472f0 2024-07-26T06:53:59.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f948e18d740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f94801aaba0 con 0x55a52b44f4d0 2024-07-26T06:53:59.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f948e18d740 10 client.0 _ll_register_callbacks cb 0x7f9470005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:53:59.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f9485012640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b42ed80 0x55a52b826870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f9485012640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b42ed80 0x55a52b826870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52888/0 (socket says 172.21.15.42:52888) 2024-07-26T06:53:59.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f9485012640 1 -- 172.21.15.42:0/2786969511 learned_addr learned my addr 172.21.15.42:0/2786969511 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:53:59.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f9484811640 1 --2- 172.21.15.42:0/2786969511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4472f0 0x55a52b825100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f9485813640 1 --2- 172.21.15.42:0/2786969511 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b44f4d0 0x55a52b825640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f9485012640 1 -- 172.21.15.42:0/2786969511 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b44f4d0 msgr2=0x55a52b825640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:59.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f9485012640 1 --2- 172.21.15.42:0/2786969511 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b44f4d0 0x55a52b825640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f9485012640 1 -- 172.21.15.42:0/2786969511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4472f0 msgr2=0x55a52b825100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:53:59.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f9485012640 1 --2- 172.21.15.42:0/2786969511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4472f0 0x55a52b825100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f9485012640 1 -- 172.21.15.42:0/2786969511 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a52b825d30 con 0x55a52b42ed80 2024-07-26T06:53:59.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f9485813640 1 --2- 172.21.15.42:0/2786969511 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b44f4d0 0x55a52b825640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:53:59.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f9484811640 1 --2- 172.21.15.42:0/2786969511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4472f0 0x55a52b825100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:53:59.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f9485012640 1 --2- 172.21.15.42:0/2786969511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b42ed80 0x55a52b826870 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f947c011480 tx=0x7f947c0048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a461c8f00e199b2e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:53:59.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f9468ff9640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:53:59.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.325+0000 7f9468ff9640 1 -- 172.21.15.42:0/2786969511 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c005200 con 0x55a52b42ed80 2024-07-26T06:53:59.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.326+0000 7f9468ff9640 1 -- 172.21.15.42:0/2786969511 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f947c0053a0 con 0x55a52b42ed80 2024-07-26T06:53:59.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.326+0000 7f948e18d740 1 -- 172.21.15.42:0/2786969511 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f947c004680 con 0x55a52b42ed80 2024-07-26T06:53:59.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.326+0000 7f9468ff9640 1 -- 172.21.15.42:0/2786969511 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f947c0055a0 con 0x55a52b42ed80 2024-07-26T06:53:59.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.326+0000 7f948e18d740 10 client.15852 fetch_fsmap learned FSMap version 36 2024-07-26T06:53:59.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.326+0000 7f948e18d740 1 -- 172.21.15.42:0/2786969511 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55a52b8938e0 con 0x55a52b42ed80 2024-07-26T06:53:59.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.326+0000 7f9468ff9640 1 -- 172.21.15.42:0/2786969511 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f947c004680 con 0x55a52b42ed80 2024-07-26T06:53:59.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.326+0000 7f9468ff9640 1 -- 172.21.15.42:0/2786969511 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f947c01a620 con 0x55a52b42ed80 2024-07-26T06:53:59.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.326+0000 7f948e18d740 10 client.15852 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:53:59.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.326+0000 7f948e18d740 10 client.15852 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:53:59.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.326+0000 7f948e18d740 20 client.15852 populate_metadata read hostname 'smithi042' 2024-07-26T06:53:59.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.326+0000 7f948e18d740 1 --2- 172.21.15.42:0/2786969511 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55a52b4701b0 0x55a52b896200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:53:59.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.326+0000 7f948e18d740 4 client.15852 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:53:59.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.326+0000 7f948e18d740 1 -- 172.21.15.42:0/2786969511 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a52b785da0 con 0x55a52b4701b0 2024-07-26T06:53:59.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.327+0000 7f9484811640 1 --2- 172.21.15.42:0/2786969511 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55a52b4701b0 0x55a52b896200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:53:59.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.327+0000 7f9484811640 1 --2- 172.21.15.42:0/2786969511 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55a52b4701b0 0x55a52b896200 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7ba8572ae3787f5 server_cookie=28f587db7b7f4543 in_seq=0 out_seq=0 2024-07-26T06:53:59.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.327+0000 7f9468ff9640 10 client.15852 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:59.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.345+0000 7f9468ff9640 1 -- 172.21.15.42:0/2786969511 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a52b785da0 con 0x55a52b4701b0 2024-07-26T06:53:59.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.345+0000 7f9468ff9640 10 client.15852 handle_command_reply: tid=0 2024-07-26T06:53:59.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.346+0000 7f948e18d740 1 client.15852 shutdown 2024-07-26T06:53:59.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.346+0000 7f948e18d740 1 -- 172.21.15.42:0/2786969511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b42ed80 msgr2=0x55a52b826870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:59.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.346+0000 7f948e18d740 1 --2- 172.21.15.42:0/2786969511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b42ed80 0x55a52b826870 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f947c011480 tx=0x7f947c0048b0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.346+0000 7f948e18d740 1 -- 172.21.15.42:0/2786969511 shutdown_connections 2024-07-26T06:53:59.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.346+0000 7f948e18d740 1 --2- 172.21.15.42:0/2786969511 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55a52b4701b0 0x55a52b896200 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.346+0000 7f948e18d740 1 --2- 172.21.15.42:0/2786969511 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55a52b44f4d0 0x55a52b825640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.346+0000 7f948e18d740 1 --2- 172.21.15.42:0/2786969511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a52b4472f0 0x55a52b825100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.346+0000 7f9468ff9640 0 client.15852 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:53:59.348 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.346+0000 7f948e18d740 1 --2- 172.21.15.42:0/2786969511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55a52b42ed80 0x55a52b826870 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.348 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.346+0000 7f948e18d740 1 -- 172.21.15.42:0/2786969511 >> 172.21.15.42:0/2786969511 conn(0x55a52b464300 msgr2=0x55a52b81ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:59.348 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.346+0000 7f948e18d740 1 -- 172.21.15.42:0/2786969511 shutdown_connections 2024-07-26T06:53:59.348 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.346+0000 7f948e18d740 1 -- 172.21.15.42:0/2786969511 wait complete. 2024-07-26T06:53:59.348 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.346+0000 7f948e18d740 20 client.15852 trim_cache size 0 max 16384 2024-07-26T06:53:59.348 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:53:59.348 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:53:59.348 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:53:59.348 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:53:59.349 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.347+0000 7f9487276640 1 -- 172.21.15.42:0/3067327873 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f946c0b3bd0 msgr2=0x7f946c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:59.349 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.347+0000 7f9487276640 1 --2- 172.21.15.42:0/3067327873 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f946c0b3bd0 0x7f946c0b60e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9478009fd0 tx=0x7f9478009500 comp rx=0 tx=0).stop 2024-07-26T06:53:59.349 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.347+0000 7f9487276640 1 -- 172.21.15.42:0/3067327873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9480106550 msgr2=0x7f9480102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:53:59.349 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.347+0000 7f9487276640 1 --2- 172.21.15.42:0/3067327873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9480106550 0x7f9480102450 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9470006010 tx=0x7f9470031a50 comp rx=0 tx=0).stop 2024-07-26T06:53:59.349 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.347+0000 7f9487276640 1 -- 172.21.15.42:0/3067327873 shutdown_connections 2024-07-26T06:53:59.349 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.347+0000 7f9487276640 1 --2- 172.21.15.42:0/3067327873 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f946c0b3bd0 0x7f946c0b60e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.349 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.347+0000 7f9487276640 1 --2- 172.21.15.42:0/3067327873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9480108eb0 0x7f9480102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.350 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.347+0000 7f9487276640 1 --2- 172.21.15.42:0/3067327873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9480106550 0x7f9480102450 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.350 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.347+0000 7f9487276640 1 --2- 172.21.15.42:0/3067327873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f94801014c0 0x7f9480103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:53:59.350 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.347+0000 7f9487276640 1 -- 172.21.15.42:0/3067327873 >> 172.21.15.42:0/3067327873 conn(0x7f94800fd100 msgr2=0x7f9480107360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:53:59.350 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.347+0000 7f9487276640 1 -- 172.21.15.42:0/3067327873 shutdown_connections 2024-07-26T06:53:59.350 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:53:59.347+0000 7f9487276640 1 -- 172.21.15.42:0/3067327873 wait complete. 2024-07-26T06:54:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:00 smithi164 ceph-mon[36970]: pgmap v510: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.3 MiB/s wr, 225 op/s 2024-07-26T06:54:01.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:00 smithi163 ceph-mon[36873]: pgmap v510: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.3 MiB/s wr, 225 op/s 2024-07-26T06:54:01.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:00 smithi042 ceph-mon[32775]: pgmap v510: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.3 MiB/s wr, 225 op/s 2024-07-26T06:54:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:02 smithi164 ceph-mon[36970]: pgmap v511: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 258 KiB/s wr, 72 op/s 2024-07-26T06:54:03.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:02 smithi163 ceph-mon[36873]: pgmap v511: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 258 KiB/s wr, 72 op/s 2024-07-26T06:54:03.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:02 smithi042 ceph-mon[32775]: pgmap v511: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 258 KiB/s wr, 72 op/s 2024-07-26T06:54:04.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:04 smithi164 ceph-mon[36970]: pgmap v512: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 27 KiB/s wr, 18 op/s 2024-07-26T06:54:05.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:04 smithi163 ceph-mon[36873]: pgmap v512: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 27 KiB/s wr, 18 op/s 2024-07-26T06:54:05.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:04 smithi042 ceph-mon[32775]: pgmap v512: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 27 KiB/s wr, 18 op/s 2024-07-26T06:54:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:06 smithi164 ceph-mon[36970]: pgmap v513: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.1 KiB/s wr, 16 op/s 2024-07-26T06:54:07.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:06 smithi163 ceph-mon[36873]: pgmap v513: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.1 KiB/s wr, 16 op/s 2024-07-26T06:54:07.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:06 smithi042 ceph-mon[32775]: pgmap v513: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.1 KiB/s wr, 16 op/s 2024-07-26T06:54:08.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:08 smithi164 ceph-mon[36970]: pgmap v514: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.3 KiB/s wr, 15 op/s 2024-07-26T06:54:09.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:08 smithi163 ceph-mon[36873]: pgmap v514: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.3 KiB/s wr, 15 op/s 2024-07-26T06:54:09.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:08 smithi042 ceph-mon[32775]: pgmap v514: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.3 KiB/s wr, 15 op/s 2024-07-26T06:54:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:10 smithi164 ceph-mon[36970]: pgmap v515: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.0 KiB/s wr, 15 op/s 2024-07-26T06:54:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:54:11.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:10 smithi163 ceph-mon[36873]: pgmap v515: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.0 KiB/s wr, 15 op/s 2024-07-26T06:54:11.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:54:11.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:10 smithi042 ceph-mon[32775]: pgmap v515: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.0 KiB/s wr, 15 op/s 2024-07-26T06:54:11.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:54:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:54:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:54:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:54:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:54:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:54:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:54:12.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:54:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:54:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:54:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.461+0000 7fc591cb0640 1 -- 172.21.15.42:0/3936253218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58c101ff0 msgr2=0x7fc58c102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.461+0000 7fc591cb0640 1 --2- 172.21.15.42:0/3936253218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58c101ff0 0x7fc58c102460 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc57c009900 tx=0x7fc57c02a410 comp rx=0 tx=0).stop 2024-07-26T06:54:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.461+0000 7fc591cb0640 1 -- 172.21.15.42:0/3936253218 shutdown_connections 2024-07-26T06:54:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.461+0000 7fc591cb0640 1 --2- 172.21.15.42:0/3936253218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc58c103a10 0x7fc58c10a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.461+0000 7fc591cb0640 1 --2- 172.21.15.42:0/3936253218 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc58c103070 0x7fc58c103480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.462 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.461+0000 7fc591cb0640 1 --2- 172.21.15.42:0/3936253218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58c101ff0 0x7fc58c102460 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.463 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.461+0000 7fc591cb0640 1 -- 172.21.15.42:0/3936253218 >> 172.21.15.42:0/3936253218 conn(0x7fc58c0fd930 msgr2=0x7fc58c0ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:12.463 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.461+0000 7fc591cb0640 1 -- 172.21.15.42:0/3936253218 shutdown_connections 2024-07-26T06:54:12.463 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.461+0000 7fc591cb0640 1 -- 172.21.15.42:0/3936253218 wait complete. 2024-07-26T06:54:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.462+0000 7fc591cb0640 1 Processor -- start 2024-07-26T06:54:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.462+0000 7fc591cb0640 1 -- start start 2024-07-26T06:54:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.462+0000 7fc591cb0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc58c101ff0 0x7fc58c1999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.462+0000 7fc591cb0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58c103070 0x7fc58c19a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.462+0000 7fc591cb0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc58c103a10 0x7fc58c19ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.462+0000 7fc591cb0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c10dc70 con 0x7fc58c103070 2024-07-26T06:54:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.462+0000 7fc591cb0640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c10daf0 con 0x7fc58c103a10 2024-07-26T06:54:12.465 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.462+0000 7fc591cb0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c10ddf0 con 0x7fc58c101ff0 2024-07-26T06:54:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.462+0000 7fc58affd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58c103070 0x7fc58c19a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.462+0000 7fc58affd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58c103070 0x7fc58c19a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53612/0 (socket says 172.21.15.42:53612) 2024-07-26T06:54:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.462+0000 7fc58affd640 1 -- 172.21.15.42:0/1538602937 learned_addr learned my addr 172.21.15.42:0/1538602937 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.462+0000 7fc58b7fe640 1 --2- 172.21.15.42:0/1538602937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc58c101ff0 0x7fc58c1999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.463+0000 7fc58bfff640 1 --2- 172.21.15.42:0/1538602937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc58c103a10 0x7fc58c19ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.463+0000 7fc58affd640 1 -- 172.21.15.42:0/1538602937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc58c101ff0 msgr2=0x7fc58c1999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.463+0000 7fc58affd640 1 --2- 172.21.15.42:0/1538602937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc58c101ff0 0x7fc58c1999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.463+0000 7fc58affd640 1 -- 172.21.15.42:0/1538602937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc58c103a10 msgr2=0x7fc58c19ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.463+0000 7fc58affd640 1 --2- 172.21.15.42:0/1538602937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc58c103a10 0x7fc58c19ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.463+0000 7fc58affd640 1 -- 172.21.15.42:0/1538602937 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58c19b430 con 0x7fc58c103070 2024-07-26T06:54:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.463+0000 7fc58bfff640 1 --2- 172.21.15.42:0/1538602937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc58c103a10 0x7fc58c19ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:54:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.463+0000 7fc58affd640 1 --2- 172.21.15.42:0/1538602937 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58c103070 0x7fc58c19a060 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc58000e9f0 tx=0x7fc58000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.463+0000 7fc588ff9640 1 -- 172.21.15.42:0/1538602937 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58000cd80 con 0x7fc58c103070 2024-07-26T06:54:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.463+0000 7fc588ff9640 1 -- 172.21.15.42:0/1538602937 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5800054d0 con 0x7fc58c103070 2024-07-26T06:54:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.463+0000 7fc591cb0640 1 -- 172.21.15.42:0/1538602937 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc58c19b720 con 0x7fc58c103070 2024-07-26T06:54:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.463+0000 7fc588ff9640 1 -- 172.21.15.42:0/1538602937 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc580010690 con 0x7fc58c103070 2024-07-26T06:54:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.463+0000 7fc591cb0640 1 -- 172.21.15.42:0/1538602937 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc58c1a1e10 con 0x7fc58c103070 2024-07-26T06:54:12.476 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.466+0000 7fc598bc7740 1 Processor -- start 2024-07-26T06:54:12.476 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.466+0000 7fc598bc7740 1 -- start start 2024-07-26T06:54:12.477 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.466+0000 7fc588ff9640 1 -- 172.21.15.42:0/1538602937 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fc580004e90 con 0x7fc58c103070 2024-07-26T06:54:12.477 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.466+0000 7fc598bc7740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a660551b0 0x558a663f9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.477 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.466+0000 7fc598bc7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6602c2f0 0x558a663fa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.477 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.466+0000 7fc598bc7740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660344d0 0x558a663fad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.477 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.466+0000 7fc598bc7740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x558a6636d3b0 con 0x558a660344d0 2024-07-26T06:54:12.477 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.466+0000 7fc598bc7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558a6635c8b0 con 0x558a6602c2f0 2024-07-26T06:54:12.477 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.466+0000 7fc598bc7740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x558a663608f0 con 0x558a660551b0 2024-07-26T06:54:12.477 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.466+0000 7fc58affd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660344d0 0x558a663fad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.477 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.466+0000 7fc58affd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660344d0 0x558a663fad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53624/0 (socket says 172.21.15.42:53624) 2024-07-26T06:54:12.477 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.466+0000 7fc58affd640 1 -- 172.21.15.42:0/2896806596 learned_addr learned my addr 172.21.15.42:0/2896806596 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:12.478 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.466+0000 7fc58bfff640 1 --2- 172.21.15.42:0/2896806596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a660551b0 0x558a663f9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.478 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.466+0000 7fc58b7fe640 1 --2- 172.21.15.42:0/2896806596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6602c2f0 0x558a663fa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.478 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.467+0000 7fc58affd640 1 -- 172.21.15.42:0/2896806596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a660551b0 msgr2=0x558a663f9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:12.478 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.467+0000 7fc58affd640 1 --2- 172.21.15.42:0/2896806596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a660551b0 0x558a663f9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.478 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.467+0000 7fc58affd640 1 -- 172.21.15.42:0/2896806596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6602c2f0 msgr2=0x558a663fa1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:12.478 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.467+0000 7fc58affd640 1 --2- 172.21.15.42:0/2896806596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6602c2f0 0x558a663fa1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.478 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.467+0000 7fc58affd640 1 -- 172.21.15.42:0/2896806596 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a663039a0 con 0x558a660344d0 2024-07-26T06:54:12.478 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.467+0000 7fc58affd640 1 --2- 172.21.15.42:0/2896806596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660344d0 0x558a663fad10 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc5800ec160 tx=0x7fc5800e7f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=927f210ce6ab9692 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:12.478 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.467+0000 7fc588ff9640 1 --2- 172.21.15.42:0/1538602937 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc5740b3bd0 0x7fc5740b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.478 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.467+0000 7fc58bfff640 1 --2- 172.21.15.42:0/1538602937 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc5740b3bd0 0x7fc5740b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.478 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.467+0000 7fc571ffb640 1 -- 172.21.15.42:0/2896806596 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5800f7070 con 0x558a660344d0 2024-07-26T06:54:12.478 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.467+0000 7fc588ff9640 1 -- 172.21.15.42:0/1538602937 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fc5800e4000 con 0x7fc58c103070 2024-07-26T06:54:12.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.467+0000 7fc571ffb640 1 -- 172.21.15.42:0/2896806596 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5800aaae0 con 0x558a660344d0 2024-07-26T06:54:12.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.467+0000 7fc571ffb640 1 -- 172.21.15.42:0/2896806596 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5800aa740 con 0x558a660344d0 2024-07-26T06:54:12.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.468+0000 7fc598bc7740 1 -- 172.21.15.42:0/2896806596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660344d0 msgr2=0x558a663fad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:12.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.468+0000 7fc58bfff640 1 --2- 172.21.15.42:0/1538602937 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc5740b3bd0 0x7fc5740b60e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc584005fd0 tx=0x7fc584009790 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:12.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.468+0000 7fc598bc7740 1 --2- 172.21.15.42:0/2896806596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660344d0 0x558a663fad10 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc5800ec160 tx=0x7fc5800e7f90 comp rx=0 tx=0).stop 2024-07-26T06:54:12.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.468+0000 7fc598bc7740 1 -- 172.21.15.42:0/2896806596 shutdown_connections 2024-07-26T06:54:12.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.468+0000 7fc598bc7740 1 --2- 172.21.15.42:0/2896806596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660344d0 0x558a663fad10 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.468+0000 7fc598bc7740 1 --2- 172.21.15.42:0/2896806596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6602c2f0 0x558a663fa1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.468+0000 7fc598bc7740 1 --2- 172.21.15.42:0/2896806596 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a660551b0 0x558a663f9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.468+0000 7fc598bc7740 1 -- 172.21.15.42:0/2896806596 >> 172.21.15.42:0/2896806596 conn(0x558a66049300 msgr2=0x558a663f55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:12.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.468+0000 7fc598bc7740 1 -- 172.21.15.42:0/2896806596 shutdown_connections 2024-07-26T06:54:12.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.468+0000 7fc598bc7740 1 -- 172.21.15.42:0/2896806596 wait complete. 2024-07-26T06:54:12.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.469+0000 7fc598bc7740 1 -- start start 2024-07-26T06:54:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.469+0000 7fc598bc7740 1 Processor -- start 2024-07-26T06:54:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.469+0000 7fc598bc7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6602c2f0 0x558a66403570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.469+0000 7fc598bc7740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660344d0 0x558a66403c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.469+0000 7fc598bc7740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a660551b0 0x558a664047a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.469+0000 7fc598bc7740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x558a66296ab0 con 0x558a660344d0 2024-07-26T06:54:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.469+0000 7fc598bc7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558a662e4cf0 con 0x558a6602c2f0 2024-07-26T06:54:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.469+0000 7fc598bc7740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x558a663fb830 con 0x558a660551b0 2024-07-26T06:54:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.469+0000 7fc598bc7740 10 client.0 _ll_register_callbacks cb 0x558a663563e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:54:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.469+0000 7fc58affd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660344d0 0x558a66403c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.469+0000 7fc58affd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660344d0 0x558a66403c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53636/0 (socket says 172.21.15.42:53636) 2024-07-26T06:54:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.469+0000 7fc58affd640 1 -- 172.21.15.42:0/2614759559 learned_addr learned my addr 172.21.15.42:0/2614759559 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:12.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc58b7fe640 1 --2- 172.21.15.42:0/2614759559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6602c2f0 0x558a66403570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc58bfff640 1 --2- 172.21.15.42:0/2614759559 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a660551b0 0x558a664047a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc58affd640 1 -- 172.21.15.42:0/2614759559 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a660551b0 msgr2=0x558a664047a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc58affd640 1 --2- 172.21.15.42:0/2614759559 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a660551b0 0x558a664047a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc58affd640 1 -- 172.21.15.42:0/2614759559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6602c2f0 msgr2=0x558a66403570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc58affd640 1 --2- 172.21.15.42:0/2614759559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6602c2f0 0x558a66403570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc58affd640 1 -- 172.21.15.42:0/2614759559 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a660c0790 con 0x558a660344d0 2024-07-26T06:54:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc58b7fe640 1 --2- 172.21.15.42:0/2614759559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6602c2f0 0x558a66403570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:54:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc58bfff640 1 --2- 172.21.15.42:0/2614759559 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a660551b0 0x558a664047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:54:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc58affd640 1 --2- 172.21.15.42:0/2614759559 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660344d0 0x558a66403c30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc5800f5040 tx=0x7fc58000c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4826ff12d001922 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc56affd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:54:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc56affd640 1 -- 172.21.15.42:0/2614759559 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5800e7c60 con 0x558a660344d0 2024-07-26T06:54:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc56affd640 1 -- 172.21.15.42:0/2614759559 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fc5800e7350 con 0x558a660344d0 2024-07-26T06:54:12.481 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc598bc7740 1 -- 172.21.15.42:0/2614759559 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5800aaae0 con 0x558a660344d0 2024-07-26T06:54:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc56affd640 1 -- 172.21.15.42:0/2614759559 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fc580005180 con 0x558a660344d0 2024-07-26T06:54:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc598bc7740 10 client.15864 fetch_fsmap learned FSMap version 36 2024-07-26T06:54:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc598bc7740 1 -- 172.21.15.42:0/2614759559 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x558a66139620 con 0x558a660344d0 2024-07-26T06:54:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.470+0000 7fc56affd640 1 -- 172.21.15.42:0/2614759559 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5800aaae0 con 0x558a660344d0 2024-07-26T06:54:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.471+0000 7fc56affd640 1 -- 172.21.15.42:0/2614759559 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7fc5800ab230 con 0x558a660344d0 2024-07-26T06:54:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.471+0000 7fc598bc7740 10 client.15864 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:54:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.471+0000 7fc598bc7740 10 client.15864 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:54:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.471+0000 7fc598bc7740 20 client.15864 populate_metadata read hostname 'smithi042' 2024-07-26T06:54:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.471+0000 7fc598bc7740 1 --2- 172.21.15.42:0/2614759559 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x558a66013d80 0x558a66410880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.471+0000 7fc598bc7740 4 client.15864 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:54:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.471+0000 7fc598bc7740 1 -- 172.21.15.42:0/2614759559 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a6636f300 con 0x558a66013d80 2024-07-26T06:54:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.471+0000 7fc58b7fe640 1 --2- 172.21.15.42:0/2614759559 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x558a66013d80 0x558a66410880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.482 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.472+0000 7fc56affd640 10 client.15864 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:12.483 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.472+0000 7fc58b7fe640 1 --2- 172.21.15.42:0/2614759559 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x558a66013d80 0x558a66410880 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b2e055028067ac6 server_cookie=5cfabc22aec1b117 in_seq=0 out_seq=0 2024-07-26T06:54:12.483 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.472+0000 7fc56affd640 1 -- 172.21.15.42:0/2614759559 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558a6636f300 con 0x558a66013d80 2024-07-26T06:54:12.483 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.472+0000 7fc56affd640 10 client.15864 handle_command_reply: tid=0 2024-07-26T06:54:12.483 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.472+0000 7fc598bc7740 1 client.15864 shutdown 2024-07-26T06:54:12.483 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.473+0000 7fc598bc7740 1 -- 172.21.15.42:0/2614759559 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660344d0 msgr2=0x558a66403c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:12.483 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.473+0000 7fc598bc7740 1 --2- 172.21.15.42:0/2614759559 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660344d0 0x558a66403c30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc5800f5040 tx=0x7fc58000c7d0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.483 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.473+0000 7fc598bc7740 1 -- 172.21.15.42:0/2614759559 shutdown_connections 2024-07-26T06:54:12.483 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.473+0000 7fc598bc7740 1 --2- 172.21.15.42:0/2614759559 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x558a66013d80 0x558a66410880 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.483 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.473+0000 7fc598bc7740 1 --2- 172.21.15.42:0/2614759559 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a660551b0 0x558a664047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.483 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.473+0000 7fc598bc7740 1 --2- 172.21.15.42:0/2614759559 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660344d0 0x558a66403c30 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.483 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.473+0000 7fc598bc7740 1 --2- 172.21.15.42:0/2614759559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a6602c2f0 0x558a66403570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.483 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.473+0000 7fc56affd640 0 client.15864 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:12.484 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.473+0000 7fc598bc7740 1 -- 172.21.15.42:0/2614759559 >> 172.21.15.42:0/2614759559 conn(0x558a66049300 msgr2=0x558a663f3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:12.484 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.473+0000 7fc598bc7740 1 -- 172.21.15.42:0/2614759559 shutdown_connections 2024-07-26T06:54:12.484 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.473+0000 7fc598bc7740 1 -- 172.21.15.42:0/2614759559 wait complete. 2024-07-26T06:54:12.484 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.473+0000 7fc598bc7740 20 client.15864 trim_cache size 0 max 16384 2024-07-26T06:54:12.490 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.488+0000 7fc598bc7740 1 Processor -- start 2024-07-26T06:54:12.490 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.488+0000 7fc598bc7740 1 -- start start 2024-07-26T06:54:12.490 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc598bc7740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a6602c2f0 0x558a663f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.490 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc598bc7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a660344d0 0x558a66402a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc598bc7740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660551b0 0x558a66402fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc598bc7740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c1aaa70 con 0x558a660551b0 2024-07-26T06:54:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc598bc7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c1ac6a0 con 0x558a660344d0 2024-07-26T06:54:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc598bc7740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c1ab170 con 0x558a6602c2f0 2024-07-26T06:54:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc58bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660551b0 0x558a66402fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc58bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660551b0 0x558a66402fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53638/0 (socket says 172.21.15.42:53638) 2024-07-26T06:54:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc58bfff640 1 -- 172.21.15.42:0/1517536854 learned_addr learned my addr 172.21.15.42:0/1517536854 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc58affd640 1 --2- 172.21.15.42:0/1517536854 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a660344d0 0x558a66402a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc58b7fe640 1 --2- 172.21.15.42:0/1517536854 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a6602c2f0 0x558a663f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc58bfff640 1 -- 172.21.15.42:0/1517536854 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a6602c2f0 msgr2=0x558a663f69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc58bfff640 1 --2- 172.21.15.42:0/1517536854 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a6602c2f0 0x558a663f69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc58bfff640 1 -- 172.21.15.42:0/1517536854 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a660344d0 msgr2=0x558a66402a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc58bfff640 1 --2- 172.21.15.42:0/1517536854 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a660344d0 0x558a66402a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc58bfff640 1 -- 172.21.15.42:0/1517536854 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a66401700 con 0x558a660551b0 2024-07-26T06:54:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc58b7fe640 1 --2- 172.21.15.42:0/1517536854 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a6602c2f0 0x558a663f69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:54:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.489+0000 7fc58bfff640 1 --2- 172.21.15.42:0/1517536854 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660551b0 0x558a66402fd0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc584012830 tx=0x7fc584014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf0011303efc60c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.490+0000 7fc5717fa640 1 -- 172.21.15.42:0/1517536854 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc584004b20 con 0x558a660551b0 2024-07-26T06:54:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.490+0000 7fc5717fa640 1 -- 172.21.15.42:0/1517536854 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc58401ede0 con 0x558a660551b0 2024-07-26T06:54:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.490+0000 7fc5717fa640 1 -- 172.21.15.42:0/1517536854 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc584005810 con 0x558a660551b0 2024-07-26T06:54:12.496 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.490+0000 7fc598bc7740 1 -- 172.21.15.42:0/1517536854 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660551b0 msgr2=0x558a66402fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:12.496 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.490+0000 7fc598bc7740 1 --2- 172.21.15.42:0/1517536854 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660551b0 0x558a66402fd0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc584012830 tx=0x7fc584014000 comp rx=0 tx=0).stop 2024-07-26T06:54:12.496 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.490+0000 7fc598bc7740 1 -- 172.21.15.42:0/1517536854 shutdown_connections 2024-07-26T06:54:12.496 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.490+0000 7fc598bc7740 1 --2- 172.21.15.42:0/1517536854 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a660551b0 0x558a66402fd0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.496 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.490+0000 7fc598bc7740 1 --2- 172.21.15.42:0/1517536854 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a660344d0 0x558a66402a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.490+0000 7fc598bc7740 1 --2- 172.21.15.42:0/1517536854 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a6602c2f0 0x558a663f69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.490+0000 7fc598bc7740 1 -- 172.21.15.42:0/1517536854 >> 172.21.15.42:0/1517536854 conn(0x558a66049300 msgr2=0x558a663f9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.490+0000 7fc598bc7740 1 -- 172.21.15.42:0/1517536854 shutdown_connections 2024-07-26T06:54:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.491+0000 7fc598bc7740 1 -- 172.21.15.42:0/1517536854 wait complete. 2024-07-26T06:54:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.491+0000 7fc598bc7740 1 -- start start 2024-07-26T06:54:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.491+0000 7fc598bc7740 1 Processor -- start 2024-07-26T06:54:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.491+0000 7fc598bc7740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a6602c2f0 0x558a6640b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.491+0000 7fc598bc7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a660344d0 0x558a66409d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.491+0000 7fc598bc7740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a660551b0 0x558a6640a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.491+0000 7fc598bc7740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c1abfd0 con 0x558a6602c2f0 2024-07-26T06:54:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.491+0000 7fc598bc7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c1ab840 con 0x558a660344d0 2024-07-26T06:54:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.491+0000 7fc598bc7740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c1a8a50 con 0x558a660551b0 2024-07-26T06:54:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.491+0000 7fc598bc7740 10 client.0 _ll_register_callbacks cb 0x7fc5800f5070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:54:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.491+0000 7fc58b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a6602c2f0 0x558a6640b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.492+0000 7fc58b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a6602c2f0 0x558a6640b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53652/0 (socket says 172.21.15.42:53652) 2024-07-26T06:54:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.492+0000 7fc58b7fe640 1 -- 172.21.15.42:0/2572360188 learned_addr learned my addr 172.21.15.42:0/2572360188 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.492+0000 7fc58affd640 1 --2- 172.21.15.42:0/2572360188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a660344d0 0x558a66409d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.492+0000 7fc58bfff640 1 --2- 172.21.15.42:0/2572360188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a660551b0 0x558a6640a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.492+0000 7fc58b7fe640 1 -- 172.21.15.42:0/2572360188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a660551b0 msgr2=0x558a6640a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.492+0000 7fc58b7fe640 1 --2- 172.21.15.42:0/2572360188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a660551b0 0x558a6640a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.492+0000 7fc58b7fe640 1 -- 172.21.15.42:0/2572360188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a660344d0 msgr2=0x558a66409d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.492+0000 7fc58b7fe640 1 --2- 172.21.15.42:0/2572360188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a660344d0 0x558a66409d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.492+0000 7fc58b7fe640 1 -- 172.21.15.42:0/2572360188 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a6640aaf0 con 0x558a6602c2f0 2024-07-26T06:54:12.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.492+0000 7fc58b7fe640 1 --2- 172.21.15.42:0/2572360188 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a6602c2f0 0x558a6640b450 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc57c02a930 tx=0x7fc57c0041a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f37d691b8f4422 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.492+0000 7fc56affd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:54:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.492+0000 7fc56affd640 1 -- 172.21.15.42:0/2572360188 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57c04b060 con 0x558a6602c2f0 2024-07-26T06:54:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.492+0000 7fc56affd640 1 -- 172.21.15.42:0/2572360188 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fc57c031a60 con 0x558a6602c2f0 2024-07-26T06:54:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.492+0000 7fc598bc7740 1 -- 172.21.15.42:0/2572360188 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc58401ede0 con 0x558a6602c2f0 2024-07-26T06:54:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.492+0000 7fc56affd640 1 -- 172.21.15.42:0/2572360188 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fc57c035040 con 0x558a6602c2f0 2024-07-26T06:54:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.492+0000 7fc598bc7740 10 client.15870 fetch_fsmap learned FSMap version 36 2024-07-26T06:54:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.492+0000 7fc598bc7740 1 -- 172.21.15.42:0/2572360188 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x558a66478090 con 0x558a6602c2f0 2024-07-26T06:54:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.493+0000 7fc56affd640 1 -- 172.21.15.42:0/2572360188 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc58401ede0 con 0x558a6602c2f0 2024-07-26T06:54:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.493+0000 7fc56affd640 1 -- 172.21.15.42:0/2572360188 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7fc57c0463c0 con 0x558a6602c2f0 2024-07-26T06:54:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.493+0000 7fc598bc7740 10 client.15870 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:54:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.493+0000 7fc598bc7740 10 client.15870 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:54:12.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.493+0000 7fc598bc7740 20 client.15870 populate_metadata read hostname 'smithi042' 2024-07-26T06:54:12.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.493+0000 7fc598bc7740 1 --2- 172.21.15.42:0/2572360188 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x558a65fe91c0 0x558a6647ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:12.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.493+0000 7fc598bc7740 4 client.15870 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:54:12.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.493+0000 7fc598bc7740 1 -- 172.21.15.42:0/2572360188 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558a6636ada0 con 0x558a65fe91c0 2024-07-26T06:54:12.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.493+0000 7fc58affd640 1 --2- 172.21.15.42:0/2572360188 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x558a65fe91c0 0x558a6647ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:12.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.494+0000 7fc56affd640 10 client.15870 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:12.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.494+0000 7fc58affd640 1 --2- 172.21.15.42:0/2572360188 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x558a65fe91c0 0x558a6647ab20 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f8ade010dc10e12 server_cookie=7c704f0370e76966 in_seq=0 out_seq=0 2024-07-26T06:54:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.508+0000 7fc56affd640 1 -- 172.21.15.42:0/2572360188 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558a6636ada0 con 0x558a65fe91c0 2024-07-26T06:54:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.508+0000 7fc56affd640 10 client.15870 handle_command_reply: tid=0 2024-07-26T06:54:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.508+0000 7fc598bc7740 1 client.15870 shutdown 2024-07-26T06:54:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.508+0000 7fc598bc7740 1 -- 172.21.15.42:0/2572360188 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a6602c2f0 msgr2=0x558a6640b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.508+0000 7fc598bc7740 1 --2- 172.21.15.42:0/2572360188 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a6602c2f0 0x558a6640b450 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc57c02a930 tx=0x7fc57c0041a0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.508+0000 7fc598bc7740 1 -- 172.21.15.42:0/2572360188 shutdown_connections 2024-07-26T06:54:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.508+0000 7fc598bc7740 1 --2- 172.21.15.42:0/2572360188 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x558a65fe91c0 0x558a6647ab20 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.508+0000 7fc598bc7740 1 --2- 172.21.15.42:0/2572360188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x558a660551b0 0x558a6640a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.508+0000 7fc598bc7740 1 --2- 172.21.15.42:0/2572360188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558a660344d0 0x558a66409d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.508+0000 7fc56affd640 0 client.15870 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.508+0000 7fc598bc7740 1 --2- 172.21.15.42:0/2572360188 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x558a6602c2f0 0x558a6640b450 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.508+0000 7fc598bc7740 1 -- 172.21.15.42:0/2572360188 >> 172.21.15.42:0/2572360188 conn(0x558a66049300 msgr2=0x558a663f9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.508+0000 7fc598bc7740 1 -- 172.21.15.42:0/2572360188 shutdown_connections 2024-07-26T06:54:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.508+0000 7fc598bc7740 1 -- 172.21.15.42:0/2572360188 wait complete. 2024-07-26T06:54:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.508+0000 7fc598bc7740 20 client.15870 trim_cache size 0 max 16384 2024-07-26T06:54:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:54:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:54:12.510 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:54:12.511 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:54:12.511 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.509+0000 7fc591cb0640 1 -- 172.21.15.42:0/1538602937 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc5740b3bd0 msgr2=0x7fc5740b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:12.511 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.509+0000 7fc591cb0640 1 --2- 172.21.15.42:0/1538602937 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc5740b3bd0 0x7fc5740b60e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc584005fd0 tx=0x7fc584009790 comp rx=0 tx=0).stop 2024-07-26T06:54:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.509+0000 7fc591cb0640 1 -- 172.21.15.42:0/1538602937 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58c103070 msgr2=0x7fc58c19a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.509+0000 7fc591cb0640 1 --2- 172.21.15.42:0/1538602937 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58c103070 0x7fc58c19a060 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc58000e9f0 tx=0x7fc58000edf0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.509+0000 7fc591cb0640 1 -- 172.21.15.42:0/1538602937 shutdown_connections 2024-07-26T06:54:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.509+0000 7fc591cb0640 1 --2- 172.21.15.42:0/1538602937 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fc5740b3bd0 0x7fc5740b60e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.509+0000 7fc591cb0640 1 --2- 172.21.15.42:0/1538602937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc58c103a10 0x7fc58c19ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.509+0000 7fc591cb0640 1 --2- 172.21.15.42:0/1538602937 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc58c103070 0x7fc58c19a060 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.509+0000 7fc591cb0640 1 --2- 172.21.15.42:0/1538602937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc58c101ff0 0x7fc58c1999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.509+0000 7fc591cb0640 1 -- 172.21.15.42:0/1538602937 >> 172.21.15.42:0/1538602937 conn(0x7fc58c0fd930 msgr2=0x7fc58c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.509+0000 7fc591cb0640 1 -- 172.21.15.42:0/1538602937 shutdown_connections 2024-07-26T06:54:12.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:12.509+0000 7fc591cb0640 1 -- 172.21.15.42:0/1538602937 wait complete. 2024-07-26T06:54:13.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:12 smithi042 ceph-mon[32775]: pgmap v516: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.9 KiB/s wr, 13 op/s 2024-07-26T06:54:13.183 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:54:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:12 smithi164 ceph-mon[36970]: pgmap v516: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.9 KiB/s wr, 13 op/s 2024-07-26T06:54:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:12 smithi163 ceph-mon[36873]: pgmap v516: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.9 KiB/s wr, 13 op/s 2024-07-26T06:54:13.458 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:54:14.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:14 smithi042 ceph-mon[32775]: pgmap v517: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-07-26T06:54:15.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.208+0000 7fbbaab5d640 1 -- 172.21.15.42:0/4157708559 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbba4108ee0 msgr2=0x7fbba41092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:15.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.208+0000 7fbbaab5d640 1 --2- 172.21.15.42:0/4157708559 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbba4108ee0 0x7fbba41092f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fbb94009900 tx=0x7fbb9402a3f0 comp rx=0 tx=0).stop 2024-07-26T06:54:15.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.208+0000 7fbbaab5d640 1 -- 172.21.15.42:0/4157708559 shutdown_connections 2024-07-26T06:54:15.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.208+0000 7fbbaab5d640 1 --2- 172.21.15.42:0/4157708559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbba4109830 0x7fbba4110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:15.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.208+0000 7fbbaab5d640 1 --2- 172.21.15.42:0/4157708559 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbba4108ee0 0x7fbba41092f0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:15.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.208+0000 7fbbaab5d640 1 --2- 172.21.15.42:0/4157708559 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbba41035f0 0x7fbba4103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:15.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.208+0000 7fbbaab5d640 1 -- 172.21.15.42:0/4157708559 >> 172.21.15.42:0/4157708559 conn(0x7fbba407b340 msgr2=0x7fbba407b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:15.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.209+0000 7fbbaab5d640 1 -- 172.21.15.42:0/4157708559 shutdown_connections 2024-07-26T06:54:15.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.209+0000 7fbbaab5d640 1 -- 172.21.15.42:0/4157708559 wait complete. 2024-07-26T06:54:15.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.209+0000 7fbbaab5d640 1 Processor -- start 2024-07-26T06:54:15.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.210+0000 7fbbaab5d640 1 -- start start 2024-07-26T06:54:15.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.210+0000 7fbbaab5d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbba41035f0 0x7fbba4075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:15.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.210+0000 7fbbaab5d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbba4108ee0 0x7fbba4075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:15.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.210+0000 7fbbaab5d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbba4109830 0x7fbba4075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:15.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.210+0000 7fbbaab5d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbba41134c0 con 0x7fbba41035f0 2024-07-26T06:54:15.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.210+0000 7fbbaab5d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbba4113340 con 0x7fbba4109830 2024-07-26T06:54:15.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.210+0000 7fbbaab5d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbba4113640 con 0x7fbba4108ee0 2024-07-26T06:54:15.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.210+0000 7fbba88f9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbba41035f0 0x7fbba4075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:15.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.210+0000 7fbba88f9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbba41035f0 0x7fbba4075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53674/0 (socket says 172.21.15.42:53674) 2024-07-26T06:54:15.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.210+0000 7fbba88f9640 1 -- 172.21.15.42:0/2205476584 learned_addr learned my addr 172.21.15.42:0/2205476584 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:15.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.210+0000 7fbb9bfff640 1 --2- 172.21.15.42:0/2205476584 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbba4108ee0 0x7fbba4075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:15.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.210+0000 7fbba90fa640 1 --2- 172.21.15.42:0/2205476584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbba4109830 0x7fbba4075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:15.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.210+0000 7fbba88f9640 1 -- 172.21.15.42:0/2205476584 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbba4108ee0 msgr2=0x7fbba4075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:15.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.210+0000 7fbba88f9640 1 --2- 172.21.15.42:0/2205476584 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbba4108ee0 0x7fbba4075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:15.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.210+0000 7fbba88f9640 1 -- 172.21.15.42:0/2205476584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbba4109830 msgr2=0x7fbba4075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:15.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.210+0000 7fbba88f9640 1 --2- 172.21.15.42:0/2205476584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbba4109830 0x7fbba4075bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:15.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.210+0000 7fbba88f9640 1 -- 172.21.15.42:0/2205476584 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba41a8120 con 0x7fbba41035f0 2024-07-26T06:54:15.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.211+0000 7fbba88f9640 1 --2- 172.21.15.42:0/2205476584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbba41035f0 0x7fbba4075140 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fbba000e9f0 tx=0x7fbba000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:15.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.211+0000 7fbb99ffb640 1 -- 172.21.15.42:0/2205476584 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba000cd80 con 0x7fbba41035f0 2024-07-26T06:54:15.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.211+0000 7fbb99ffb640 1 -- 172.21.15.42:0/2205476584 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbba00054d0 con 0x7fbba41035f0 2024-07-26T06:54:15.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.211+0000 7fbb99ffb640 1 -- 172.21.15.42:0/2205476584 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba0010690 con 0x7fbba41035f0 2024-07-26T06:54:15.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.211+0000 7fbbaab5d640 1 -- 172.21.15.42:0/2205476584 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba41a8410 con 0x7fbba41035f0 2024-07-26T06:54:15.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.211+0000 7fbbaab5d640 1 -- 172.21.15.42:0/2205476584 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba41a88d0 con 0x7fbba41035f0 2024-07-26T06:54:15.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.212+0000 7fbbaab5d640 1 -- 172.21.15.42:0/2205476584 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbba4103a00 con 0x7fbba41035f0 2024-07-26T06:54:15.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.213+0000 7fbb99ffb640 1 -- 172.21.15.42:0/2205476584 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fbba0004e90 con 0x7fbba41035f0 2024-07-26T06:54:15.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.214+0000 7fbb99ffb640 1 --2- 172.21.15.42:0/2205476584 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbb8c0b3bd0 0x7fbb8c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:15.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.214+0000 7fbb99ffb640 1 -- 172.21.15.42:0/2205476584 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fbba00e4000 con 0x7fbba41035f0 2024-07-26T06:54:15.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.214+0000 7fbb9bfff640 1 --2- 172.21.15.42:0/2205476584 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbb8c0b3bd0 0x7fbb8c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:15.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.215+0000 7fbb9bfff640 1 --2- 172.21.15.42:0/2205476584 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbb8c0b3bd0 0x7fbb8c0b60e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbb94009230 tx=0x7fbb94031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:15.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.219+0000 7fbb99ffb640 1 -- 172.21.15.42:0/2205476584 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbba00ab6d0 con 0x7fbba41035f0 2024-07-26T06:54:15.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:14 smithi164 ceph-mon[36970]: pgmap v517: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-07-26T06:54:15.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:14 smithi163 ceph-mon[36873]: pgmap v517: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-07-26T06:54:15.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.441+0000 7fbbaab5d640 1 -- 172.21.15.42:0/2205476584 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbba4076440 con 0x7fbba41035f0 2024-07-26T06:54:15.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.442+0000 7fbb99ffb640 1 -- 172.21.15.42:0/2205476584 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11986 (secure 0 0 0) 0x7fbba00aafd0 con 0x7fbba41035f0 2024-07-26T06:54:15.443 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:54:15.443 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":36,"btime":"2024-07-26T06:52:41:554060+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:52:41.554043+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25369":{"gid":25369,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/1072602530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1072602530},{"type":"v1","addr":"172.21.15.164:6835","nonce":1072602530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24578,24583,24593,24601,24620]},"id":1}]} 2024-07-26T06:54:15.443 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 36 2024-07-26T06:54:15.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.444+0000 7fbbaab5d640 1 -- 172.21.15.42:0/2205476584 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbb8c0b3bd0 msgr2=0x7fbb8c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:15.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.444+0000 7fbbaab5d640 1 --2- 172.21.15.42:0/2205476584 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbb8c0b3bd0 0x7fbb8c0b60e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbb94009230 tx=0x7fbb94031400 comp rx=0 tx=0).stop 2024-07-26T06:54:15.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.444+0000 7fbbaab5d640 1 -- 172.21.15.42:0/2205476584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbba41035f0 msgr2=0x7fbba4075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:15.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.444+0000 7fbbaab5d640 1 --2- 172.21.15.42:0/2205476584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbba41035f0 0x7fbba4075140 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fbba000e9f0 tx=0x7fbba000edf0 comp rx=0 tx=0).stop 2024-07-26T06:54:15.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.444+0000 7fbbaab5d640 1 -- 172.21.15.42:0/2205476584 shutdown_connections 2024-07-26T06:54:15.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.444+0000 7fbbaab5d640 1 --2- 172.21.15.42:0/2205476584 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fbb8c0b3bd0 0x7fbb8c0b60e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:15.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.444+0000 7fbbaab5d640 1 --2- 172.21.15.42:0/2205476584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbba4109830 0x7fbba4075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:15.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.444+0000 7fbbaab5d640 1 --2- 172.21.15.42:0/2205476584 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbba4108ee0 0x7fbba4075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:15.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.444+0000 7fbbaab5d640 1 --2- 172.21.15.42:0/2205476584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbba41035f0 0x7fbba4075140 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:15.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.444+0000 7fbbaab5d640 1 -- 172.21.15.42:0/2205476584 >> 172.21.15.42:0/2205476584 conn(0x7fbba407b340 msgr2=0x7fbba4104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:15.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.444+0000 7fbbaab5d640 1 -- 172.21.15.42:0/2205476584 shutdown_connections 2024-07-26T06:54:15.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:15.445+0000 7fbbaab5d640 1 -- 172.21.15.42:0/2205476584 wait complete. 2024-07-26T06:54:16.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:15 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2205476584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:54:16.189 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24620 quiesce db --set-id c8d66e1d --release --await' 2024-07-26T06:54:16.189 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.24620 quiesce db --set-id c8d66e1d --release --await 2024-07-26T06:54:16.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:15 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2205476584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:54:16.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:15 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2205476584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:54:16.463 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:54:16.868 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:16 smithi042 ceph-mon[32775]: pgmap v518: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-07-26T06:54:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:16 smithi164 ceph-mon[36970]: pgmap v518: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-07-26T06:54:17.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:16 smithi163 ceph-mon[36873]: pgmap v518: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 11 KiB/s wr, 14 op/s 2024-07-26T06:54:18.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.295+0000 7f57f673e640 1 -- 172.21.15.42:0/3681408910 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57f0103590 msgr2=0x7f57f01039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:18.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.295+0000 7f57f673e640 1 --2- 172.21.15.42:0/3681408910 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57f0103590 0x7f57f01039a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f57e0009900 tx=0x7f57e002a410 comp rx=0 tx=0).stop 2024-07-26T06:54:18.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.295+0000 7f57f673e640 1 -- 172.21.15.42:0/3681408910 shutdown_connections 2024-07-26T06:54:18.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.295+0000 7f57f673e640 1 --2- 172.21.15.42:0/3681408910 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57f0109820 0x7f57f0110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.295+0000 7f57f673e640 1 --2- 172.21.15.42:0/3681408910 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57f0108ed0 0x7f57f01092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.295+0000 7f57f673e640 1 --2- 172.21.15.42:0/3681408910 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57f0103590 0x7f57f01039a0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.295+0000 7f57f673e640 1 -- 172.21.15.42:0/3681408910 >> 172.21.15.42:0/3681408910 conn(0x7f57f007b280 msgr2=0x7f57f007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:18.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.295+0000 7f57f673e640 1 -- 172.21.15.42:0/3681408910 shutdown_connections 2024-07-26T06:54:18.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.295+0000 7f57f673e640 1 -- 172.21.15.42:0/3681408910 wait complete. 2024-07-26T06:54:18.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.296+0000 7f57f673e640 1 Processor -- start 2024-07-26T06:54:18.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.296+0000 7f57f673e640 1 -- start start 2024-07-26T06:54:18.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.296+0000 7f57f673e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57f0103590 0x7f57f0075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.296+0000 7f57f673e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57f0108ed0 0x7f57f0075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.296+0000 7f57f673e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57f0109820 0x7f57f0075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.296+0000 7f57f673e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f57f01134f0 con 0x7f57f0109820 2024-07-26T06:54:18.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.296+0000 7f57f673e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f57f0113370 con 0x7f57f0108ed0 2024-07-26T06:54:18.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.296+0000 7f57f673e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f57f0113670 con 0x7f57f0103590 2024-07-26T06:54:18.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.297+0000 7f57f4cdb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57f0109820 0x7f57f0075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:18.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.297+0000 7f57f4cdb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57f0109820 0x7f57f0075bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60142/0 (socket says 172.21.15.42:60142) 2024-07-26T06:54:18.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.297+0000 7f57f4cdb640 1 -- 172.21.15.42:0/2755176056 learned_addr learned my addr 172.21.15.42:0/2755176056 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:18.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.297+0000 7f57effff640 1 --2- 172.21.15.42:0/2755176056 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57f0103590 0x7f57f0075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:18.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.297+0000 7f57ef7fe640 1 --2- 172.21.15.42:0/2755176056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57f0108ed0 0x7f57f0075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:18.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.297+0000 7f57f4cdb640 1 -- 172.21.15.42:0/2755176056 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57f0103590 msgr2=0x7f57f0075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:18.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.297+0000 7f57f4cdb640 1 --2- 172.21.15.42:0/2755176056 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57f0103590 0x7f57f0075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.297+0000 7f57f4cdb640 1 -- 172.21.15.42:0/2755176056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57f0108ed0 msgr2=0x7f57f0075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:18.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.297+0000 7f57f4cdb640 1 --2- 172.21.15.42:0/2755176056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57f0108ed0 0x7f57f0075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.297+0000 7f57f4cdb640 1 -- 172.21.15.42:0/2755176056 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f01a8260 con 0x7f57f0109820 2024-07-26T06:54:18.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.297+0000 7f57ef7fe640 1 --2- 172.21.15.42:0/2755176056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57f0108ed0 0x7f57f0075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:54:18.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.297+0000 7f57f4cdb640 1 --2- 172.21.15.42:0/2755176056 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57f0109820 0x7f57f0075bc0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f57e8009fd0 tx=0x7f57e800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:18.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.297+0000 7f57ed7fa640 1 -- 172.21.15.42:0/2755176056 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8017070 con 0x7f57f0109820 2024-07-26T06:54:18.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.297+0000 7f57ed7fa640 1 -- 172.21.15.42:0/2755176056 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57e8003e20 con 0x7f57f0109820 2024-07-26T06:54:18.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.297+0000 7f57f673e640 1 -- 172.21.15.42:0/2755176056 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f01a8550 con 0x7f57f0109820 2024-07-26T06:54:18.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.297+0000 7f57ed7fa640 1 -- 172.21.15.42:0/2755176056 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8010460 con 0x7f57f0109820 2024-07-26T06:54:18.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.297+0000 7f57f673e640 1 -- 172.21.15.42:0/2755176056 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f01a8a40 con 0x7f57f0109820 2024-07-26T06:54:18.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.300+0000 7f57fccc2740 1 Processor -- start 2024-07-26T06:54:18.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.300+0000 7f57ed7fa640 1 -- 172.21.15.42:0/2755176056 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f57e8003970 con 0x7f57f0109820 2024-07-26T06:54:18.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.300+0000 7f57fccc2740 1 -- start start 2024-07-26T06:54:18.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.300+0000 7f57fccc2740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c27520 0x55eb25fcd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.300+0000 7f57fccc2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25bfe660 0x55eb25fcd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.300+0000 7f57fccc2740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25c06840 0x55eb25fcdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.301+0000 7f57fccc2740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55eb25f42e70 con 0x55eb25c27520 2024-07-26T06:54:18.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.301+0000 7f57fccc2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55eb25f32370 con 0x55eb25bfe660 2024-07-26T06:54:18.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.301+0000 7f57fccc2740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55eb25f363b0 con 0x55eb25c06840 2024-07-26T06:54:18.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.301+0000 7f57ef7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c27520 0x55eb25fcd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:18.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.301+0000 7f57ef7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c27520 0x55eb25fcd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60150/0 (socket says 172.21.15.42:60150) 2024-07-26T06:54:18.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.301+0000 7f57ef7fe640 1 -- 172.21.15.42:0/204846487 learned_addr learned my addr 172.21.15.42:0/204846487 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:18.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.301+0000 7f57effff640 1 --2- 172.21.15.42:0/204846487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25bfe660 0x55eb25fcd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:18.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.301+0000 7f57ef7fe640 1 --2- 172.21.15.42:0/204846487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25c06840 0x55eb25fcdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:18.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.301+0000 7f57ed7fa640 1 --2- 172.21.15.42:0/2755176056 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f57d80b3bd0 0x7f57d80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.301+0000 7f57ef7fe640 1 -- 172.21.15.42:0/204846487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25c06840 msgr2=0x55eb25fcdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:18.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.301+0000 7f57ef7fe640 1 --2- 172.21.15.42:0/204846487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25c06840 0x55eb25fcdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.301+0000 7f57ef7fe640 1 -- 172.21.15.42:0/204846487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25bfe660 msgr2=0x55eb25fcd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:18.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.301+0000 7f57ef7fe640 1 --2- 172.21.15.42:0/204846487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25bfe660 0x55eb25fcd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.301+0000 7f57ef7fe640 1 -- 172.21.15.42:0/204846487 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb25eff040 con 0x55eb25c27520 2024-07-26T06:54:18.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.301+0000 7f57f4cdb640 1 --2- 172.21.15.42:0/2755176056 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f57d80b3bd0 0x7f57d80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:18.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.301+0000 7f57ed7fa640 1 -- 172.21.15.42:0/2755176056 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f57e80e3960 con 0x7f57f0109820 2024-07-26T06:54:18.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.301+0000 7f57ef7fe640 1 --2- 172.21.15.42:0/204846487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c27520 0x55eb25fcd070 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f57e4013560 tx=0x7f57e400b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=224a7700846c4e1e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:18.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.302+0000 7f57d67fc640 1 -- 172.21.15.42:0/204846487 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e4019030 con 0x55eb25c27520 2024-07-26T06:54:18.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.302+0000 7f57d67fc640 1 -- 172.21.15.42:0/204846487 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57e4011970 con 0x55eb25c27520 2024-07-26T06:54:18.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.302+0000 7f57d67fc640 1 -- 172.21.15.42:0/204846487 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e400f8f0 con 0x55eb25c27520 2024-07-26T06:54:18.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.303+0000 7f57fccc2740 1 -- 172.21.15.42:0/204846487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c27520 msgr2=0x55eb25fcd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:18.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.303+0000 7f57f4cdb640 1 --2- 172.21.15.42:0/2755176056 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f57d80b3bd0 0x7f57d80b60e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f57e80aa820 tx=0x7f57e809f160 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:18.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.303+0000 7f57fccc2740 1 --2- 172.21.15.42:0/204846487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c27520 0x55eb25fcd070 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f57e4013560 tx=0x7f57e400b6d0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.303+0000 7f57fccc2740 1 -- 172.21.15.42:0/204846487 shutdown_connections 2024-07-26T06:54:18.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.303+0000 7f57fccc2740 1 --2- 172.21.15.42:0/204846487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25c06840 0x55eb25fcdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.303+0000 7f57fccc2740 1 --2- 172.21.15.42:0/204846487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25bfe660 0x55eb25fcd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.303+0000 7f57fccc2740 1 --2- 172.21.15.42:0/204846487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c27520 0x55eb25fcd070 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.303+0000 7f57fccc2740 1 -- 172.21.15.42:0/204846487 >> 172.21.15.42:0/204846487 conn(0x55eb25c1b910 msgr2=0x55eb25fcaf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:18.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.303+0000 7f57fccc2740 1 -- 172.21.15.42:0/204846487 shutdown_connections 2024-07-26T06:54:18.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.303+0000 7f57fccc2740 1 -- 172.21.15.42:0/204846487 wait complete. 2024-07-26T06:54:18.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.304+0000 7f57fccc2740 1 -- start start 2024-07-26T06:54:18.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.304+0000 7f57fccc2740 1 Processor -- start 2024-07-26T06:54:18.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.304+0000 7f57fccc2740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25bfe660 0x55eb25fd8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.304+0000 7f57fccc2740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c06840 0x55eb25fd9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.304+0000 7f57fccc2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25c27520 0x55eb25fe4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.304+0000 7f57fccc2740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55eb25d95bb0 con 0x55eb25c06840 2024-07-26T06:54:18.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.304+0000 7f57fccc2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55eb25ed4ac0 con 0x55eb25c27520 2024-07-26T06:54:18.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.304+0000 7f57fccc2740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55eb25fdb8a0 con 0x55eb25bfe660 2024-07-26T06:54:18.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.304+0000 7f57fccc2740 10 client.0 _ll_register_callbacks cb 0x55eb25edd3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:54:18.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.304+0000 7f57effff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c06840 0x55eb25fd9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:18.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.304+0000 7f57effff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c06840 0x55eb25fd9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60154/0 (socket says 172.21.15.42:60154) 2024-07-26T06:54:18.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.304+0000 7f57effff640 1 -- 172.21.15.42:0/3988320852 learned_addr learned my addr 172.21.15.42:0/3988320852 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:18.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.305+0000 7f57ef7fe640 1 --2- 172.21.15.42:0/3988320852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25c27520 0x55eb25fe4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:18.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.305+0000 7f57ef7fe640 1 --2- 172.21.15.42:0/3988320852 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25bfe660 0x55eb25fd8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:18.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.305+0000 7f57effff640 1 -- 172.21.15.42:0/3988320852 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25bfe660 msgr2=0x55eb25fd8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:18.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.305+0000 7f57effff640 1 --2- 172.21.15.42:0/3988320852 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25bfe660 0x55eb25fd8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.305+0000 7f57effff640 1 -- 172.21.15.42:0/3988320852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25c27520 msgr2=0x55eb25fe4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:18.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.305+0000 7f57effff640 1 --2- 172.21.15.42:0/3988320852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25c27520 0x55eb25fe4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.305+0000 7f57effff640 1 -- 172.21.15.42:0/3988320852 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb25c92b00 con 0x55eb25c06840 2024-07-26T06:54:18.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.305+0000 7f57effff640 1 --2- 172.21.15.42:0/3988320852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c06840 0x55eb25fd9190 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f57e0009900 tx=0x7f57e00316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ae500ed95e9c67 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:18.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.305+0000 7f57cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:54:18.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.305+0000 7f57cf7fe640 1 -- 172.21.15.42:0/3988320852 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0035040 con 0x55eb25c06840 2024-07-26T06:54:18.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.306+0000 7f57cf7fe640 1 -- 172.21.15.42:0/3988320852 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f57e0004040 con 0x55eb25c06840 2024-07-26T06:54:18.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.306+0000 7f57fccc2740 1 -- 172.21.15.42:0/3988320852 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57e4011970 con 0x55eb25c06840 2024-07-26T06:54:18.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.306+0000 7f57cf7fe640 1 -- 172.21.15.42:0/3988320852 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f57e004c070 con 0x55eb25c06840 2024-07-26T06:54:18.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.306+0000 7f57fccc2740 10 client.15888 fetch_fsmap learned FSMap version 36 2024-07-26T06:54:18.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.306+0000 7f57fccc2740 1 -- 172.21.15.42:0/3988320852 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55eb25d09960 con 0x55eb25c06840 2024-07-26T06:54:18.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.306+0000 7f57cf7fe640 1 -- 172.21.15.42:0/3988320852 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57e4011970 con 0x55eb25c06840 2024-07-26T06:54:18.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.306+0000 7f57cf7fe640 1 -- 172.21.15.42:0/3988320852 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f57e0004750 con 0x55eb25c06840 2024-07-26T06:54:18.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.306+0000 7f57fccc2740 10 client.15888 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:54:18.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.306+0000 7f57fccc2740 10 client.15888 resolve_mds: validated gid 24620 aka daemon mds.b 2024-07-26T06:54:18.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.306+0000 7f57fccc2740 20 client.15888 populate_metadata read hostname 'smithi042' 2024-07-26T06:54:18.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.306+0000 7f57fccc2740 1 --2- 172.21.15.42:0/3988320852 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55eb25be5ea0 0x55eb25feac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.306+0000 7f57fccc2740 4 client.15888 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:54:18.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.306+0000 7f57fccc2740 1 -- 172.21.15.42:0/3988320852 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb25f44dc0 con 0x55eb25be5ea0 2024-07-26T06:54:18.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.307+0000 7f57ef7fe640 1 --2- 172.21.15.42:0/3988320852 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55eb25be5ea0 0x55eb25feac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:18.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.307+0000 7f57cf7fe640 10 client.15888 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:18.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.307+0000 7f57ef7fe640 1 --2- 172.21.15.42:0/3988320852 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55eb25be5ea0 0x55eb25feac90 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0b004d3255c206b server_cookie=4b9bcff396750b0a in_seq=0 out_seq=0 2024-07-26T06:54:18.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.308+0000 7f57cf7fe640 1 -- 172.21.15.42:0/3988320852 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55eb25f44dc0 con 0x55eb25be5ea0 2024-07-26T06:54:18.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.308+0000 7f57cf7fe640 10 client.15888 handle_command_reply: tid=0 2024-07-26T06:54:18.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.308+0000 7f57fccc2740 1 client.15888 shutdown 2024-07-26T06:54:18.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.308+0000 7f57fccc2740 1 -- 172.21.15.42:0/3988320852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c06840 msgr2=0x55eb25fd9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:18.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.308+0000 7f57fccc2740 1 --2- 172.21.15.42:0/3988320852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c06840 0x55eb25fd9190 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f57e0009900 tx=0x7f57e00316d0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.308+0000 7f57fccc2740 1 -- 172.21.15.42:0/3988320852 shutdown_connections 2024-07-26T06:54:18.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.308+0000 7f57fccc2740 1 --2- 172.21.15.42:0/3988320852 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55eb25be5ea0 0x55eb25feac90 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.308+0000 7f57fccc2740 1 --2- 172.21.15.42:0/3988320852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25c27520 0x55eb25fe4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.308+0000 7f57fccc2740 1 --2- 172.21.15.42:0/3988320852 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c06840 0x55eb25fd9190 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.308+0000 7f57fccc2740 1 --2- 172.21.15.42:0/3988320852 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25bfe660 0x55eb25fd8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.308+0000 7f57cf7fe640 0 client.15888 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:18.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.308+0000 7f57fccc2740 1 -- 172.21.15.42:0/3988320852 >> 172.21.15.42:0/3988320852 conn(0x55eb25c1b910 msgr2=0x55eb25fd0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:18.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.308+0000 7f57fccc2740 1 -- 172.21.15.42:0/3988320852 shutdown_connections 2024-07-26T06:54:18.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.308+0000 7f57fccc2740 1 -- 172.21.15.42:0/3988320852 wait complete. 2024-07-26T06:54:18.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.309+0000 7f57fccc2740 20 client.15888 trim_cache size 0 max 16384 2024-07-26T06:54:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.326+0000 7f57fccc2740 1 Processor -- start 2024-07-26T06:54:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.326+0000 7f57fccc2740 1 -- start start 2024-07-26T06:54:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.326+0000 7f57fccc2740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25bfe660 0x55eb25fc9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.326+0000 7f57fccc2740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c06840 0x55eb25fcea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.326+0000 7f57fccc2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25c27520 0x55eb25fda680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.326+0000 7f57fccc2740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f57f01b1010 con 0x55eb25c06840 2024-07-26T06:54:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.326+0000 7f57fccc2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f57f01b2c40 con 0x55eb25c27520 2024-07-26T06:54:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.326+0000 7f57fccc2740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f57f01b1710 con 0x55eb25bfe660 2024-07-26T06:54:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.326+0000 7f57effff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c06840 0x55eb25fcea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.326+0000 7f57effff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c06840 0x55eb25fcea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60168/0 (socket says 172.21.15.42:60168) 2024-07-26T06:54:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.326+0000 7f57effff640 1 -- 172.21.15.42:0/3944017377 learned_addr learned my addr 172.21.15.42:0/3944017377 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.327+0000 7f57ef7fe640 1 --2- 172.21.15.42:0/3944017377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25bfe660 0x55eb25fc9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.327+0000 7f57ef7fe640 1 --2- 172.21.15.42:0/3944017377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25c27520 0x55eb25fda680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.327+0000 7f57effff640 1 -- 172.21.15.42:0/3944017377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25bfe660 msgr2=0x55eb25fc9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.327+0000 7f57effff640 1 --2- 172.21.15.42:0/3944017377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25bfe660 0x55eb25fc9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.327+0000 7f57effff640 1 -- 172.21.15.42:0/3944017377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25c27520 msgr2=0x55eb25fda680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.327+0000 7f57effff640 1 --2- 172.21.15.42:0/3944017377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25c27520 0x55eb25fda680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.327+0000 7f57effff640 1 -- 172.21.15.42:0/3944017377 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb25fcf360 con 0x55eb25c06840 2024-07-26T06:54:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.327+0000 7f57ef7fe640 1 --2- 172.21.15.42:0/3944017377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25c27520 0x55eb25fda680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:54:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.327+0000 7f57effff640 1 --2- 172.21.15.42:0/3944017377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c06840 0x55eb25fcea60 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f57e0006010 tx=0x7f57e0032bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bfe257ba4a7b566 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.327+0000 7f57d5ffb640 1 -- 172.21.15.42:0/3944017377 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0035030 con 0x55eb25c06840 2024-07-26T06:54:18.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.327+0000 7f57d5ffb640 1 -- 172.21.15.42:0/3944017377 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57e0055070 con 0x55eb25c06840 2024-07-26T06:54:18.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.327+0000 7f57d5ffb640 1 -- 172.21.15.42:0/3944017377 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0051480 con 0x55eb25c06840 2024-07-26T06:54:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.328+0000 7f57fccc2740 1 -- 172.21.15.42:0/3944017377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c06840 msgr2=0x55eb25fcea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.328+0000 7f57fccc2740 1 --2- 172.21.15.42:0/3944017377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c06840 0x55eb25fcea60 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f57e0006010 tx=0x7f57e0032bf0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.328+0000 7f57fccc2740 1 -- 172.21.15.42:0/3944017377 shutdown_connections 2024-07-26T06:54:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.328+0000 7f57fccc2740 1 --2- 172.21.15.42:0/3944017377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25c27520 0x55eb25fda680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.328+0000 7f57fccc2740 1 --2- 172.21.15.42:0/3944017377 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25c06840 0x55eb25fcea60 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.328+0000 7f57fccc2740 1 --2- 172.21.15.42:0/3944017377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25bfe660 0x55eb25fc9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.328+0000 7f57fccc2740 1 -- 172.21.15.42:0/3944017377 >> 172.21.15.42:0/3944017377 conn(0x55eb25c1b910 msgr2=0x55eb25fd9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.328+0000 7f57fccc2740 1 -- 172.21.15.42:0/3944017377 shutdown_connections 2024-07-26T06:54:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.328+0000 7f57fccc2740 1 -- 172.21.15.42:0/3944017377 wait complete. 2024-07-26T06:54:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.329+0000 7f57fccc2740 1 -- start start 2024-07-26T06:54:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.329+0000 7f57fccc2740 1 Processor -- start 2024-07-26T06:54:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.329+0000 7f57fccc2740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25bfe660 0x55eb25fe0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.329+0000 7f57fccc2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25c06840 0x55eb25fe1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.329+0000 7f57fccc2740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25c27520 0x55eb25fdbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.329+0000 7f57fccc2740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f57f01b2570 con 0x55eb25bfe660 2024-07-26T06:54:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.329+0000 7f57fccc2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f57f01b1de0 con 0x55eb25c06840 2024-07-26T06:54:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.329+0000 7f57fccc2740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f57f01aeff0 con 0x55eb25c27520 2024-07-26T06:54:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.329+0000 7f57fccc2740 10 client.0 _ll_register_callbacks cb 0x7f57e0005db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:54:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.329+0000 7f57ef7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25bfe660 0x55eb25fe0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.329+0000 7f57ef7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25bfe660 0x55eb25fe0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60178/0 (socket says 172.21.15.42:60178) 2024-07-26T06:54:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.329+0000 7f57ef7fe640 1 -- 172.21.15.42:0/2962849236 learned_addr learned my addr 172.21.15.42:0/2962849236 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.330+0000 7f57effff640 1 --2- 172.21.15.42:0/2962849236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25c06840 0x55eb25fe1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.330+0000 7f57ef7fe640 1 -- 172.21.15.42:0/2962849236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25c27520 msgr2=0x55eb25fdbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.330+0000 7f57ef7fe640 1 --2- 172.21.15.42:0/2962849236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25c27520 0x55eb25fdbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.330+0000 7f57ef7fe640 1 -- 172.21.15.42:0/2962849236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25c06840 msgr2=0x55eb25fe1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.330+0000 7f57ef7fe640 1 --2- 172.21.15.42:0/2962849236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25c06840 0x55eb25fe1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.330+0000 7f57ef7fe640 1 -- 172.21.15.42:0/2962849236 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb25fdbff0 con 0x55eb25bfe660 2024-07-26T06:54:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.330+0000 7f57ef7fe640 1 --2- 172.21.15.42:0/2962849236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25bfe660 0x55eb25fe0b00 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f57e4005fd0 tx=0x7f57e400b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409f1a64188015ca server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.330+0000 7f57cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:54:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.330+0000 7f57cf7fe640 1 -- 172.21.15.42:0/2962849236 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e4019040 con 0x55eb25bfe660 2024-07-26T06:54:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.330+0000 7f57fccc2740 1 -- 172.21.15.42:0/2962849236 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57e0055070 con 0x55eb25bfe660 2024-07-26T06:54:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.330+0000 7f57cf7fe640 1 -- 172.21.15.42:0/2962849236 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f57e402de80 con 0x55eb25bfe660 2024-07-26T06:54:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.330+0000 7f57cf7fe640 1 -- 172.21.15.42:0/2962849236 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f57e403c070 con 0x55eb25bfe660 2024-07-26T06:54:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.330+0000 7f57fccc2740 10 client.15894 fetch_fsmap learned FSMap version 36 2024-07-26T06:54:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.330+0000 7f57fccc2740 1 -- 172.21.15.42:0/2962849236 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55eb26053be0 con 0x55eb25bfe660 2024-07-26T06:54:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.331+0000 7f57cf7fe640 1 -- 172.21.15.42:0/2962849236 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57e0055070 con 0x55eb25bfe660 2024-07-26T06:54:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.331+0000 7f57cf7fe640 1 -- 172.21.15.42:0/2962849236 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7f57e4035030 con 0x55eb25bfe660 2024-07-26T06:54:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.331+0000 7f57fccc2740 10 client.15894 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:54:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.331+0000 7f57fccc2740 10 client.15894 resolve_mds: validated gid 24620 aka daemon mds.b 2024-07-26T06:54:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.331+0000 7f57fccc2740 20 client.15894 populate_metadata read hostname 'smithi042' 2024-07-26T06:54:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.331+0000 7f57fccc2740 1 --2- 172.21.15.42:0/2962849236 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55eb25bbb250 0x55eb260561e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.331+0000 7f57fccc2740 4 client.15894 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c8d66e1d", "release": true, "await": true}] 2024-07-26T06:54:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.331+0000 7f57fccc2740 1 -- 172.21.15.42:0/2962849236 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c8d66e1d", "release": true, "await": true}) -- 0x55eb25f40860 con 0x55eb25bbb250 2024-07-26T06:54:18.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.331+0000 7f57effff640 1 --2- 172.21.15.42:0/2962849236 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55eb25bbb250 0x55eb260561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:18.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.332+0000 7f57cf7fe640 10 client.15894 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:18.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.332+0000 7f57effff640 1 --2- 172.21.15.42:0/2962849236 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55eb25bbb250 0x55eb260561e0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fa267311fc13f72 server_cookie=d2bba59aefff988 in_seq=0 out_seq=0 2024-07-26T06:54:18.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.442+0000 7f57cf7fe640 1 -- 172.21.15.42:0/2962849236 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55eb25f40860 con 0x55eb25bbb250 2024-07-26T06:54:18.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.442+0000 7f57cf7fe640 10 client.15894 handle_command_reply: tid=0 2024-07-26T06:54:18.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.442+0000 7f57fccc2740 1 client.15894 shutdown 2024-07-26T06:54:18.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.443+0000 7f57fccc2740 1 -- 172.21.15.42:0/2962849236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25bfe660 msgr2=0x55eb25fe0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:18.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.443+0000 7f57fccc2740 1 --2- 172.21.15.42:0/2962849236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25bfe660 0x55eb25fe0b00 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f57e4005fd0 tx=0x7f57e400b650 comp rx=0 tx=0).stop 2024-07-26T06:54:18.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.443+0000 7f57fccc2740 1 -- 172.21.15.42:0/2962849236 shutdown_connections 2024-07-26T06:54:18.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.443+0000 7f57fccc2740 1 --2- 172.21.15.42:0/2962849236 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55eb25bbb250 0x55eb260561e0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.443+0000 7f57fccc2740 1 --2- 172.21.15.42:0/2962849236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55eb25c27520 0x55eb25fdbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.443+0000 7f57fccc2740 1 --2- 172.21.15.42:0/2962849236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55eb25c06840 0x55eb25fe1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.443+0000 7f57cf7fe640 0 client.15894 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:18.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.443+0000 7f57fccc2740 1 --2- 172.21.15.42:0/2962849236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55eb25bfe660 0x55eb25fe0b00 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.443+0000 7f57fccc2740 1 -- 172.21.15.42:0/2962849236 >> 172.21.15.42:0/2962849236 conn(0x55eb25c1b910 msgr2=0x55eb25fce830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:18.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.443+0000 7f57fccc2740 1 -- 172.21.15.42:0/2962849236 shutdown_connections 2024-07-26T06:54:18.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.443+0000 7f57fccc2740 1 -- 172.21.15.42:0/2962849236 wait complete. 2024-07-26T06:54:18.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.443+0000 7f57fccc2740 20 client.15894 trim_cache size 0 max 16384 2024-07-26T06:54:18.445 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:54:18.445 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 36, 2024-07-26T06:54:18.445 INFO:teuthology.orchestra.run.smithi042.stdout: "leader": 24620, 2024-07-26T06:54:18.445 INFO:teuthology.orchestra.run.smithi042.stdout: "set_version": 12, 2024-07-26T06:54:18.445 INFO:teuthology.orchestra.run.smithi042.stdout: "sets": { 2024-07-26T06:54:18.445 INFO:teuthology.orchestra.run.smithi042.stdout: "c8d66e1d": { 2024-07-26T06:54:18.446 INFO:teuthology.orchestra.run.smithi042.stdout: "version": 12, 2024-07-26T06:54:18.446 INFO:teuthology.orchestra.run.smithi042.stdout: "age_ref": 0.0, 2024-07-26T06:54:18.446 INFO:teuthology.orchestra.run.smithi042.stdout: "state": { 2024-07-26T06:54:18.446 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "RELEASED", 2024-07-26T06:54:18.446 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 0.0 2024-07-26T06:54:18.446 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-07-26T06:54:18.446 INFO:teuthology.orchestra.run.smithi042.stdout: "timeout": 90.0, 2024-07-26T06:54:18.446 INFO:teuthology.orchestra.run.smithi042.stdout: "expiration": 151.1, 2024-07-26T06:54:18.446 INFO:teuthology.orchestra.run.smithi042.stdout: "members": { 2024-07-26T06:54:18.446 INFO:teuthology.orchestra.run.smithi042.stdout: "file:/": { 2024-07-26T06:54:18.446 INFO:teuthology.orchestra.run.smithi042.stdout: "excluded": false, 2024-07-26T06:54:18.446 INFO:teuthology.orchestra.run.smithi042.stdout: "state": { 2024-07-26T06:54:18.447 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "RELEASED", 2024-07-26T06:54:18.447 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 0.0 2024-07-26T06:54:18.447 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:54:18.447 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:54:18.447 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:54:18.447 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:54:18.447 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:54:18.447 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:54:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.444+0000 7f57f673e640 1 -- 172.21.15.42:0/2755176056 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f57d80b3bd0 msgr2=0x7f57d80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.444+0000 7f57f673e640 1 --2- 172.21.15.42:0/2755176056 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f57d80b3bd0 0x7f57d80b60e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f57e80aa820 tx=0x7f57e809f160 comp rx=0 tx=0).stop 2024-07-26T06:54:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.444+0000 7f57f673e640 1 -- 172.21.15.42:0/2755176056 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57f0109820 msgr2=0x7f57f0075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.444+0000 7f57f673e640 1 --2- 172.21.15.42:0/2755176056 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57f0109820 0x7f57f0075bc0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f57e8009fd0 tx=0x7f57e800b260 comp rx=0 tx=0).stop 2024-07-26T06:54:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.444+0000 7f57f673e640 1 -- 172.21.15.42:0/2755176056 shutdown_connections 2024-07-26T06:54:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.444+0000 7f57f673e640 1 --2- 172.21.15.42:0/2755176056 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f57d80b3bd0 0x7f57d80b60e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.444+0000 7f57f673e640 1 --2- 172.21.15.42:0/2755176056 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f57f0109820 0x7f57f0075bc0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.444+0000 7f57f673e640 1 --2- 172.21.15.42:0/2755176056 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57f0108ed0 0x7f57f0075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.444+0000 7f57f673e640 1 --2- 172.21.15.42:0/2755176056 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f57f0103590 0x7f57f0075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:18.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.444+0000 7f57f673e640 1 -- 172.21.15.42:0/2755176056 >> 172.21.15.42:0/2755176056 conn(0x7f57f007b280 msgr2=0x7f57f0107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:18.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.444+0000 7f57f673e640 1 -- 172.21.15.42:0/2755176056 shutdown_connections 2024-07-26T06:54:18.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:54:18.444+0000 7f57f673e640 1 -- 172.21.15.42:0/2755176056 wait complete. 2024-07-26T06:54:19.101 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:18 smithi042 ceph-mon[32775]: pgmap v519: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-07-26T06:54:19.135 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c8d66e1d', total seconds elapsed: 37.0 2024-07-26T06:54:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:18 smithi164 ceph-mon[36970]: pgmap v519: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-07-26T06:54:19.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:18 smithi163 ceph-mon[36873]: pgmap v519: 97 pgs: 97 active+clean; 45 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 KiB/s wr, 13 op/s 2024-07-26T06:54:20.345 INFO:tasks.workunit.client.0.smithi042.stdout:copying to largest ever: 0x9ff773 2024-07-26T06:54:20.753 INFO:tasks.workunit.client.0.smithi042.stdout:20000 punch from 0x34ddcb to 0x35324f, (0x5484 bytes) 2024-07-26T06:54:21.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:20 smithi042 ceph-mon[32775]: pgmap v520: 97 pgs: 97 active+clean; 48 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 104 KiB/s wr, 27 op/s 2024-07-26T06:54:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:20 smithi164 ceph-mon[36970]: pgmap v520: 97 pgs: 97 active+clean; 48 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 104 KiB/s wr, 27 op/s 2024-07-26T06:54:21.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:20 smithi163 ceph-mon[36873]: pgmap v520: 97 pgs: 97 active+clean; 48 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 104 KiB/s wr, 27 op/s 2024-07-26T06:54:23.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:22 smithi042 ceph-mon[32775]: pgmap v521: 97 pgs: 97 active+clean; 49 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 428 KiB/s wr, 101 op/s 2024-07-26T06:54:23.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:22 smithi164 ceph-mon[36970]: pgmap v521: 97 pgs: 97 active+clean; 49 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 428 KiB/s wr, 101 op/s 2024-07-26T06:54:23.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:22 smithi163 ceph-mon[36873]: pgmap v521: 97 pgs: 97 active+clean; 49 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 428 KiB/s wr, 101 op/s 2024-07-26T06:54:25.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:24 smithi042 ceph-mon[32775]: pgmap v522: 97 pgs: 97 active+clean; 51 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 894 KiB/s wr, 172 op/s 2024-07-26T06:54:25.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:24 smithi164 ceph-mon[36970]: pgmap v522: 97 pgs: 97 active+clean; 51 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 894 KiB/s wr, 172 op/s 2024-07-26T06:54:25.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:24 smithi163 ceph-mon[36873]: pgmap v522: 97 pgs: 97 active+clean; 51 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 894 KiB/s wr, 172 op/s 2024-07-26T06:54:25.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.619+0000 7fca5890e640 1 -- 172.21.15.42:0/1434484733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca54106250 msgr2=0x7fca541086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:25.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.619+0000 7fca5890e640 1 --2- 172.21.15.42:0/1434484733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca54106250 0x7fca541086d0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fca48009900 tx=0x7fca4802a410 comp rx=0 tx=0).stop 2024-07-26T06:54:25.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.619+0000 7fca5890e640 1 -- 172.21.15.42:0/1434484733 shutdown_connections 2024-07-26T06:54:25.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.619+0000 7fca5890e640 1 --2- 172.21.15.42:0/1434484733 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca54108e10 0x7fca5410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.619+0000 7fca5890e640 1 --2- 172.21.15.42:0/1434484733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca54106250 0x7fca541086d0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.619+0000 7fca5890e640 1 --2- 172.21.15.42:0/1434484733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca5406b140 0x7fca54105d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.619+0000 7fca5890e640 1 -- 172.21.15.42:0/1434484733 >> 172.21.15.42:0/1434484733 conn(0x7fca540fd910 msgr2=0x7fca540ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:25.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.619+0000 7fca5890e640 1 -- 172.21.15.42:0/1434484733 shutdown_connections 2024-07-26T06:54:25.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.619+0000 7fca5890e640 1 -- 172.21.15.42:0/1434484733 wait complete. 2024-07-26T06:54:25.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca5890e640 1 Processor -- start 2024-07-26T06:54:25.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca5890e640 1 -- start start 2024-07-26T06:54:25.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca5890e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca5406b140 0x7fca54103350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca5890e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca54106250 0x7fca54101c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca5890e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca54108e10 0x7fca54102150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca5890e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fca5410ed30 con 0x7fca54106250 2024-07-26T06:54:25.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca5890e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fca5410ebb0 con 0x7fca5406b140 2024-07-26T06:54:25.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca5890e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fca5410eeb0 con 0x7fca54108e10 2024-07-26T06:54:25.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca51d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca54106250 0x7fca54101c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca52d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca54108e10 0x7fca54102150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca51d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca54106250 0x7fca54101c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60200/0 (socket says 172.21.15.42:60200) 2024-07-26T06:54:25.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca5259c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca5406b140 0x7fca54103350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca51d9b640 1 -- 172.21.15.42:0/2280355577 learned_addr learned my addr 172.21.15.42:0/2280355577 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:25.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca52d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca54108e10 0x7fca54102150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:46416/0 (socket says 172.21.15.42:46416) 2024-07-26T06:54:25.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca51d9b640 1 -- 172.21.15.42:0/2280355577 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca54108e10 msgr2=0x7fca54102150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:25.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca51d9b640 1 --2- 172.21.15.42:0/2280355577 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca54108e10 0x7fca54102150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca51d9b640 1 -- 172.21.15.42:0/2280355577 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca5406b140 msgr2=0x7fca54103350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:25.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca51d9b640 1 --2- 172.21.15.42:0/2280355577 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca5406b140 0x7fca54103350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca51d9b640 1 -- 172.21.15.42:0/2280355577 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca54102880 con 0x7fca54106250 2024-07-26T06:54:25.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca52d9d640 1 --2- 172.21.15.42:0/2280355577 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca54108e10 0x7fca54102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:54:25.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.620+0000 7fca5259c640 1 --2- 172.21.15.42:0/2280355577 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca5406b140 0x7fca54103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:54:25.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.621+0000 7fca51d9b640 1 --2- 172.21.15.42:0/2280355577 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca54106250 0x7fca54101c10 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fca48006010 tx=0x7fca48004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:25.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.621+0000 7fca437fe640 1 -- 172.21.15.42:0/2280355577 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4803f030 con 0x7fca54106250 2024-07-26T06:54:25.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.621+0000 7fca437fe640 1 -- 172.21.15.42:0/2280355577 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca48031910 con 0x7fca54106250 2024-07-26T06:54:25.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.621+0000 7fca5890e640 1 -- 172.21.15.42:0/2280355577 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca541a3a20 con 0x7fca54106250 2024-07-26T06:54:25.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.621+0000 7fca437fe640 1 -- 172.21.15.42:0/2280355577 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca48035070 con 0x7fca54106250 2024-07-26T06:54:25.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.621+0000 7fca5890e640 1 -- 172.21.15.42:0/2280355577 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca541a3e20 con 0x7fca54106250 2024-07-26T06:54:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.624+0000 7fca5f82f740 1 Processor -- start 2024-07-26T06:54:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.624+0000 7fca5f82f740 1 -- start start 2024-07-26T06:54:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.624+0000 7fca437fe640 1 -- 172.21.15.42:0/2280355577 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fca48042030 con 0x7fca54106250 2024-07-26T06:54:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.624+0000 7fca5f82f740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e499c81b0 0x563e49d6d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.624+0000 7fca5f82f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e4999f2f0 0x563e49d6d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.624+0000 7fca5f82f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 0x563e49d6e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.624+0000 7fca5f82f740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x563e49ce03b0 con 0x563e499c81b0 2024-07-26T06:54:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.624+0000 7fca5f82f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563e49ccf8b0 con 0x563e499a74d0 2024-07-26T06:54:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.624+0000 7fca5f82f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x563e49cd38f0 con 0x563e4999f2f0 2024-07-26T06:54:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.624+0000 7fca52d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e499c81b0 0x563e49d6d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.624+0000 7fca52d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e499c81b0 0x563e49d6d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60216/0 (socket says 172.21.15.42:60216) 2024-07-26T06:54:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.624+0000 7fca5259c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e4999f2f0 0x563e49d6d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.624+0000 7fca52d9d640 1 -- 172.21.15.42:0/580576670 learned_addr learned my addr 172.21.15.42:0/580576670 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.624+0000 7fca51d9b640 1 --2- 172.21.15.42:0/580576670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 0x563e49d6e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.625+0000 7fca52d9d640 1 -- 172.21.15.42:0/580576670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e4999f2f0 msgr2=0x563e49d6d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.625+0000 7fca52d9d640 1 --2- 172.21.15.42:0/580576670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e4999f2f0 0x563e49d6d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.625+0000 7fca52d9d640 1 -- 172.21.15.42:0/580576670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 msgr2=0x563e49d6e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.625+0000 7fca52d9d640 1 --2- 172.21.15.42:0/580576670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 0x563e49d6e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.625+0000 7fca52d9d640 1 -- 172.21.15.42:0/580576670 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e49c769a0 con 0x563e499c81b0 2024-07-26T06:54:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.625+0000 7fca5259c640 1 --2- 172.21.15.42:0/580576670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e4999f2f0 0x563e49d6d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:54:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.625+0000 7fca51d9b640 1 --2- 172.21.15.42:0/580576670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 0x563e49d6e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:54:25.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.625+0000 7fca52d9d640 1 --2- 172.21.15.42:0/580576670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e499c81b0 0x563e49d6d180 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fca4c011820 tx=0x7fca4c011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc6c6991315f6b26 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.625+0000 7fca40ff9640 1 -- 172.21.15.42:0/580576670 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c011330 con 0x563e499c81b0 2024-07-26T06:54:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.625+0000 7fca40ff9640 1 -- 172.21.15.42:0/580576670 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca4c00fe50 con 0x563e499c81b0 2024-07-26T06:54:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.625+0000 7fca40ff9640 1 -- 172.21.15.42:0/580576670 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c010430 con 0x563e499c81b0 2024-07-26T06:54:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.625+0000 7fca437fe640 1 --2- 172.21.15.42:0/2280355577 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fca3c0b3bd0 0x7fca3c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.625+0000 7fca5259c640 1 --2- 172.21.15.42:0/2280355577 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fca3c0b3bd0 0x7fca3c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.626+0000 7fca437fe640 1 -- 172.21.15.42:0/2280355577 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fca48103fb0 con 0x7fca54106250 2024-07-26T06:54:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.626+0000 7fca5f82f740 1 -- 172.21.15.42:0/580576670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e499c81b0 msgr2=0x563e49d6d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.626+0000 7fca5259c640 1 --2- 172.21.15.42:0/2280355577 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fca3c0b3bd0 0x7fca3c0b60e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fca44009fd0 tx=0x7fca44005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.626+0000 7fca5f82f740 1 --2- 172.21.15.42:0/580576670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e499c81b0 0x563e49d6d180 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fca4c011820 tx=0x7fca4c011ce0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.626+0000 7fca5f82f740 1 -- 172.21.15.42:0/580576670 shutdown_connections 2024-07-26T06:54:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.626+0000 7fca5f82f740 1 --2- 172.21.15.42:0/580576670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 0x563e49d6e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.626+0000 7fca5f82f740 1 --2- 172.21.15.42:0/580576670 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e4999f2f0 0x563e49d6d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.626+0000 7fca5f82f740 1 --2- 172.21.15.42:0/580576670 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e499c81b0 0x563e49d6d180 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.626+0000 7fca5f82f740 1 -- 172.21.15.42:0/580576670 >> 172.21.15.42:0/580576670 conn(0x563e499bc300 msgr2=0x563e49d68c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.627+0000 7fca5f82f740 1 -- 172.21.15.42:0/580576670 shutdown_connections 2024-07-26T06:54:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.627+0000 7fca5f82f740 1 -- 172.21.15.42:0/580576670 wait complete. 2024-07-26T06:54:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.627+0000 7fca5f82f740 1 -- start start 2024-07-26T06:54:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.627+0000 7fca5f82f740 1 Processor -- start 2024-07-26T06:54:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.627+0000 7fca5f82f740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e4999f2f0 0x563e49d76ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.627+0000 7fca5f82f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 0x563e49d77260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.627+0000 7fca5f82f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e499c81b0 0x563e49d77dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.627+0000 7fca5f82f740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x563e49c09ab0 con 0x563e4999f2f0 2024-07-26T06:54:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.627+0000 7fca5f82f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563e49c57cf0 con 0x563e499a74d0 2024-07-26T06:54:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.627+0000 7fca5f82f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x563e49d6ee60 con 0x563e499c81b0 2024-07-26T06:54:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.628+0000 7fca5f82f740 10 client.0 _ll_register_callbacks cb 0x563e49cc99f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:54:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.628+0000 7fca5259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e4999f2f0 0x563e49d76ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.628+0000 7fca51d9b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 0x563e49d77260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.628+0000 7fca5259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e4999f2f0 0x563e49d76ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60222/0 (socket says 172.21.15.42:60222) 2024-07-26T06:54:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.628+0000 7fca51d9b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 0x563e49d77260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:46192/0 (socket says 172.21.15.42:46192) 2024-07-26T06:54:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.628+0000 7fca5259c640 1 -- 172.21.15.42:0/463416185 learned_addr learned my addr 172.21.15.42:0/463416185 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.628+0000 7fca52d9d640 1 --2- 172.21.15.42:0/463416185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e499c81b0 0x563e49d77dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.628+0000 7fca51d9b640 1 -- 172.21.15.42:0/463416185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e499c81b0 msgr2=0x563e49d77dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.628+0000 7fca51d9b640 1 --2- 172.21.15.42:0/463416185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e499c81b0 0x563e49d77dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.628+0000 7fca51d9b640 1 -- 172.21.15.42:0/463416185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e4999f2f0 msgr2=0x563e49d76ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.628+0000 7fca51d9b640 1 --2- 172.21.15.42:0/463416185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e4999f2f0 0x563e49d76ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.628+0000 7fca51d9b640 1 -- 172.21.15.42:0/463416185 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e49a33790 con 0x563e499a74d0 2024-07-26T06:54:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.628+0000 7fca52d9d640 1 --2- 172.21.15.42:0/463416185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e499c81b0 0x563e49d77dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:54:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.628+0000 7fca5259c640 1 --2- 172.21.15.42:0/463416185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e4999f2f0 0x563e49d76ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:54:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.628+0000 7fca51d9b640 1 --2- 172.21.15.42:0/463416185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 0x563e49d77260 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fca48101ae0 tx=0x7fca4802a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2496694e0b5e0360 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.628+0000 7fca39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:54:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.628+0000 7fca39ffb640 1 -- 172.21.15.42:0/463416185 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca48049040 con 0x563e499a74d0 2024-07-26T06:54:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.629+0000 7fca5f82f740 1 -- 172.21.15.42:0/463416185 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca4c00fe50 con 0x563e499a74d0 2024-07-26T06:54:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.629+0000 7fca39ffb640 1 -- 172.21.15.42:0/463416185 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fca48106110 con 0x563e499a74d0 2024-07-26T06:54:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.629+0000 7fca39ffb640 1 -- 172.21.15.42:0/463416185 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fca480cad60 con 0x563e499a74d0 2024-07-26T06:54:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.629+0000 7fca5f82f740 10 client.25490 fetch_fsmap learned FSMap version 36 2024-07-26T06:54:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.629+0000 7fca5f82f740 1 -- 172.21.15.42:0/463416185 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563e49aac620 con 0x563e499a74d0 2024-07-26T06:54:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.629+0000 7fca39ffb640 1 -- 172.21.15.42:0/463416185 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca4c00fe50 con 0x563e499a74d0 2024-07-26T06:54:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.629+0000 7fca39ffb640 1 -- 172.21.15.42:0/463416185 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7fca48107c40 con 0x563e499a74d0 2024-07-26T06:54:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.629+0000 7fca5f82f740 10 client.25490 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:54:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.629+0000 7fca5f82f740 10 client.25490 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:54:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.629+0000 7fca5f82f740 20 client.25490 populate_metadata read hostname 'smithi042' 2024-07-26T06:54:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.629+0000 7fca5f82f740 1 --2- 172.21.15.42:0/463416185 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x563e49986d80 0x563e49d84480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.629+0000 7fca5f82f740 4 client.25490 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:54:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.629+0000 7fca5f82f740 1 -- 172.21.15.42:0/463416185 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e49ce2300 con 0x563e49986d80 2024-07-26T06:54:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.629+0000 7fca5259c640 1 --2- 172.21.15.42:0/463416185 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x563e49986d80 0x563e49d84480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.630+0000 7fca5259c640 1 --2- 172.21.15.42:0/463416185 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x563e49986d80 0x563e49d84480 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37bde6f64a15e4f server_cookie=8ca9504ecd0bbe04 in_seq=0 out_seq=0 2024-07-26T06:54:25.637 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.630+0000 7fca39ffb640 10 client.25490 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:25.639 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.631+0000 7fca39ffb640 1 -- 172.21.15.42:0/463416185 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563e49ce2300 con 0x563e49986d80 2024-07-26T06:54:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.631+0000 7fca39ffb640 10 client.25490 handle_command_reply: tid=0 2024-07-26T06:54:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.631+0000 7fca5f82f740 1 client.25490 shutdown 2024-07-26T06:54:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.631+0000 7fca5f82f740 1 -- 172.21.15.42:0/463416185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 msgr2=0x563e49d77260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.631+0000 7fca5f82f740 1 --2- 172.21.15.42:0/463416185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 0x563e49d77260 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fca48101ae0 tx=0x7fca4802a410 comp rx=0 tx=0).stop 2024-07-26T06:54:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.631+0000 7fca5f82f740 1 -- 172.21.15.42:0/463416185 shutdown_connections 2024-07-26T06:54:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.631+0000 7fca5f82f740 1 --2- 172.21.15.42:0/463416185 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x563e49986d80 0x563e49d84480 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.631+0000 7fca5f82f740 1 --2- 172.21.15.42:0/463416185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e499c81b0 0x563e49d77dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.631+0000 7fca39ffb640 0 client.25490 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.631+0000 7fca5f82f740 1 --2- 172.21.15.42:0/463416185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 0x563e49d77260 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.631+0000 7fca5f82f740 1 --2- 172.21.15.42:0/463416185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e4999f2f0 0x563e49d76ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.631+0000 7fca5f82f740 1 -- 172.21.15.42:0/463416185 >> 172.21.15.42:0/463416185 conn(0x563e499bc300 msgr2=0x563e49d672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.631+0000 7fca5f82f740 1 -- 172.21.15.42:0/463416185 shutdown_connections 2024-07-26T06:54:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.631+0000 7fca5f82f740 1 -- 172.21.15.42:0/463416185 wait complete. 2024-07-26T06:54:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.631+0000 7fca5f82f740 20 client.25490 trim_cache size 0 max 16384 2024-07-26T06:54:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.648+0000 7fca5f82f740 1 Processor -- start 2024-07-26T06:54:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.648+0000 7fca5f82f740 1 -- start start 2024-07-26T06:54:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.648+0000 7fca5f82f740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e49986d80 0x563e49d6a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.648+0000 7fca5f82f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e4999f2f0 0x563e49d6e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.648+0000 7fca5f82f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 0x563e49d67c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.649+0000 7fca5f82f740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fca541acc20 con 0x563e49986d80 2024-07-26T06:54:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.649+0000 7fca5f82f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fca541ae850 con 0x563e499a74d0 2024-07-26T06:54:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.649+0000 7fca5f82f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fca541ad320 con 0x563e4999f2f0 2024-07-26T06:54:25.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.649+0000 7fca5259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e49986d80 0x563e49d6a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.649+0000 7fca5259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e49986d80 0x563e49d6a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60236/0 (socket says 172.21.15.42:60236) 2024-07-26T06:54:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.649+0000 7fca5259c640 1 -- 172.21.15.42:0/1959109982 learned_addr learned my addr 172.21.15.42:0/1959109982 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.649+0000 7fca51d9b640 1 --2- 172.21.15.42:0/1959109982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e4999f2f0 0x563e49d6e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.649+0000 7fca52d9d640 1 --2- 172.21.15.42:0/1959109982 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 0x563e49d67c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.649+0000 7fca5259c640 1 -- 172.21.15.42:0/1959109982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e4999f2f0 msgr2=0x563e49d6e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.649+0000 7fca5259c640 1 --2- 172.21.15.42:0/1959109982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e4999f2f0 0x563e49d6e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.649+0000 7fca5259c640 1 -- 172.21.15.42:0/1959109982 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 msgr2=0x563e49d67c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.649+0000 7fca5259c640 1 --2- 172.21.15.42:0/1959109982 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 0x563e49d67c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.649+0000 7fca5259c640 1 -- 172.21.15.42:0/1959109982 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e49d74d30 con 0x563e49986d80 2024-07-26T06:54:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.649+0000 7fca51d9b640 1 --2- 172.21.15.42:0/1959109982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e4999f2f0 0x563e49d6e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:54:25.651 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.649+0000 7fca5259c640 1 --2- 172.21.15.42:0/1959109982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e49986d80 0x563e49d6a060 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fca4400f790 tx=0x7fca4401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9102864035ee5c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:25.652 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.649+0000 7fca3bfff640 1 -- 172.21.15.42:0/1959109982 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca440040f0 con 0x563e49986d80 2024-07-26T06:54:25.652 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.650+0000 7fca3bfff640 1 -- 172.21.15.42:0/1959109982 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca44004290 con 0x563e49986d80 2024-07-26T06:54:25.652 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.650+0000 7fca3bfff640 1 -- 172.21.15.42:0/1959109982 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4402b780 con 0x563e49986d80 2024-07-26T06:54:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.650+0000 7fca5f82f740 1 -- 172.21.15.42:0/1959109982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e49986d80 msgr2=0x563e49d6a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.650+0000 7fca5f82f740 1 --2- 172.21.15.42:0/1959109982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e49986d80 0x563e49d6a060 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fca4400f790 tx=0x7fca4401a000 comp rx=0 tx=0).stop 2024-07-26T06:54:25.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.650+0000 7fca5f82f740 1 -- 172.21.15.42:0/1959109982 shutdown_connections 2024-07-26T06:54:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.650+0000 7fca5f82f740 1 --2- 172.21.15.42:0/1959109982 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e499a74d0 0x563e49d67c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.650+0000 7fca5f82f740 1 --2- 172.21.15.42:0/1959109982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e4999f2f0 0x563e49d6e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.650+0000 7fca5f82f740 1 --2- 172.21.15.42:0/1959109982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e49986d80 0x563e49d6a060 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.650+0000 7fca5f82f740 1 -- 172.21.15.42:0/1959109982 >> 172.21.15.42:0/1959109982 conn(0x563e499bc300 msgr2=0x563e49d639d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.651+0000 7fca5f82f740 1 -- 172.21.15.42:0/1959109982 shutdown_connections 2024-07-26T06:54:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.651+0000 7fca5f82f740 1 -- 172.21.15.42:0/1959109982 wait complete. 2024-07-26T06:54:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.651+0000 7fca5f82f740 1 -- start start 2024-07-26T06:54:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.651+0000 7fca5f82f740 1 Processor -- start 2024-07-26T06:54:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.651+0000 7fca5f82f740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e49986d80 0x563e49d7e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.651+0000 7fca5f82f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4999f2f0 0x563e49d7e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.651+0000 7fca5f82f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e499a74d0 0x563e49d7cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.651+0000 7fca5f82f740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fca541ae180 con 0x563e49986d80 2024-07-26T06:54:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.651+0000 7fca5f82f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fca541ad9f0 con 0x563e4999f2f0 2024-07-26T06:54:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.651+0000 7fca5f82f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fca541aac00 con 0x563e499a74d0 2024-07-26T06:54:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.651+0000 7fca5f82f740 10 client.0 _ll_register_callbacks cb 0x7fca48005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:54:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.651+0000 7fca5259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e49986d80 0x563e49d7e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.651+0000 7fca5259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e49986d80 0x563e49d7e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60242/0 (socket says 172.21.15.42:60242) 2024-07-26T06:54:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.651+0000 7fca5259c640 1 -- 172.21.15.42:0/4075494000 learned_addr learned my addr 172.21.15.42:0/4075494000 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.652+0000 7fca52d9d640 1 --2- 172.21.15.42:0/4075494000 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e499a74d0 0x563e49d7cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.652+0000 7fca51d9b640 1 --2- 172.21.15.42:0/4075494000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4999f2f0 0x563e49d7e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.652+0000 7fca5259c640 1 -- 172.21.15.42:0/4075494000 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e499a74d0 msgr2=0x563e49d7cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.652+0000 7fca5259c640 1 --2- 172.21.15.42:0/4075494000 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e499a74d0 0x563e49d7cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.652+0000 7fca5259c640 1 -- 172.21.15.42:0/4075494000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4999f2f0 msgr2=0x563e49d7e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.652+0000 7fca5259c640 1 --2- 172.21.15.42:0/4075494000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4999f2f0 0x563e49d7e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.652+0000 7fca5259c640 1 -- 172.21.15.42:0/4075494000 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e49d7d2a0 con 0x563e49986d80 2024-07-26T06:54:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.652+0000 7fca5259c640 1 --2- 172.21.15.42:0/4075494000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e49986d80 0x563e49d7e290 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fca44005f60 tx=0x7fca44003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f143d57773f3bfa server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.652+0000 7fca39ffb640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:54:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.652+0000 7fca39ffb640 1 -- 172.21.15.42:0/4075494000 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca440035f0 con 0x563e49986d80 2024-07-26T06:54:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.652+0000 7fca39ffb640 1 -- 172.21.15.42:0/4075494000 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fca44003790 con 0x563e49986d80 2024-07-26T06:54:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.652+0000 7fca5f82f740 1 -- 172.21.15.42:0/4075494000 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca44004290 con 0x563e49986d80 2024-07-26T06:54:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.652+0000 7fca39ffb640 1 -- 172.21.15.42:0/4075494000 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fca44003990 con 0x563e49986d80 2024-07-26T06:54:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.652+0000 7fca5f82f740 10 client.15912 fetch_fsmap learned FSMap version 36 2024-07-26T06:54:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.652+0000 7fca5f82f740 1 -- 172.21.15.42:0/4075494000 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563e49deb560 con 0x563e49986d80 2024-07-26T06:54:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.653+0000 7fca39ffb640 1 -- 172.21.15.42:0/4075494000 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca44004290 con 0x563e49986d80 2024-07-26T06:54:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.653+0000 7fca39ffb640 1 -- 172.21.15.42:0/4075494000 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 36) ==== 8382+0+0 (secure 0 0 0) 0x7fca4403e070 con 0x563e49986d80 2024-07-26T06:54:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.653+0000 7fca5f82f740 10 client.15912 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T06:54:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.653+0000 7fca5f82f740 10 client.15912 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:54:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.653+0000 7fca5f82f740 20 client.15912 populate_metadata read hostname 'smithi042' 2024-07-26T06:54:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.653+0000 7fca5f82f740 1 --2- 172.21.15.42:0/4075494000 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x563e499c81b0 0x563e49dedfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.653+0000 7fca5f82f740 4 client.15912 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:54:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.653+0000 7fca5f82f740 1 -- 172.21.15.42:0/4075494000 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563e49cddda0 con 0x563e499c81b0 2024-07-26T06:54:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.653+0000 7fca51d9b640 1 --2- 172.21.15.42:0/4075494000 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x563e499c81b0 0x563e49dedfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.654+0000 7fca39ffb640 10 client.15912 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.654+0000 7fca51d9b640 1 --2- 172.21.15.42:0/4075494000 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x563e499c81b0 0x563e49dedfd0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c98aa2763338b754 server_cookie=926301be7a855e44 in_seq=0 out_seq=0 2024-07-26T06:54:25.667 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.665+0000 7fca39ffb640 1 -- 172.21.15.42:0/4075494000 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563e49cddda0 con 0x563e499c81b0 2024-07-26T06:54:25.667 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.665+0000 7fca39ffb640 10 client.15912 handle_command_reply: tid=0 2024-07-26T06:54:25.667 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.665+0000 7fca5f82f740 1 client.15912 shutdown 2024-07-26T06:54:25.667 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.665+0000 7fca5f82f740 1 -- 172.21.15.42:0/4075494000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e49986d80 msgr2=0x563e49d7e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:25.667 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.665+0000 7fca5f82f740 1 --2- 172.21.15.42:0/4075494000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e49986d80 0x563e49d7e290 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fca44005f60 tx=0x7fca44003d70 comp rx=0 tx=0).stop 2024-07-26T06:54:25.667 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.666+0000 7fca5f82f740 1 -- 172.21.15.42:0/4075494000 shutdown_connections 2024-07-26T06:54:25.667 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.666+0000 7fca5f82f740 1 --2- 172.21.15.42:0/4075494000 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x563e499c81b0 0x563e49dedfd0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.667 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.666+0000 7fca5f82f740 1 --2- 172.21.15.42:0/4075494000 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x563e499a74d0 0x563e49d7cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.667 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.666+0000 7fca39ffb640 0 client.15912 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:25.667 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.666+0000 7fca5f82f740 1 --2- 172.21.15.42:0/4075494000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563e4999f2f0 0x563e49d7e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.667 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.666+0000 7fca5f82f740 1 --2- 172.21.15.42:0/4075494000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x563e49986d80 0x563e49d7e290 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.666+0000 7fca5f82f740 1 -- 172.21.15.42:0/4075494000 >> 172.21.15.42:0/4075494000 conn(0x563e499bc300 msgr2=0x563e49d639d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:25.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.666+0000 7fca5f82f740 1 -- 172.21.15.42:0/4075494000 shutdown_connections 2024-07-26T06:54:25.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.666+0000 7fca5f82f740 1 -- 172.21.15.42:0/4075494000 wait complete. 2024-07-26T06:54:25.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.666+0000 7fca5f82f740 20 client.15912 trim_cache size 0 max 16384 2024-07-26T06:54:25.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:54:25.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:54:25.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:54:25.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:54:25.669 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.666+0000 7fca5890e640 1 -- 172.21.15.42:0/2280355577 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fca3c0b3bd0 msgr2=0x7fca3c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:25.669 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.666+0000 7fca5890e640 1 --2- 172.21.15.42:0/2280355577 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fca3c0b3bd0 0x7fca3c0b60e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fca44009fd0 tx=0x7fca44005f90 comp rx=0 tx=0).stop 2024-07-26T06:54:25.669 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.666+0000 7fca5890e640 1 -- 172.21.15.42:0/2280355577 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca54106250 msgr2=0x7fca54101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:25.669 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.666+0000 7fca5890e640 1 --2- 172.21.15.42:0/2280355577 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca54106250 0x7fca54101c10 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fca48006010 tx=0x7fca48004270 comp rx=0 tx=0).stop 2024-07-26T06:54:25.669 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.667+0000 7fca5890e640 1 -- 172.21.15.42:0/2280355577 shutdown_connections 2024-07-26T06:54:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.667+0000 7fca5890e640 1 --2- 172.21.15.42:0/2280355577 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fca3c0b3bd0 0x7fca3c0b60e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.667+0000 7fca5890e640 1 --2- 172.21.15.42:0/2280355577 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca54108e10 0x7fca54102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.667+0000 7fca5890e640 1 --2- 172.21.15.42:0/2280355577 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca54106250 0x7fca54101c10 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.667+0000 7fca5890e640 1 --2- 172.21.15.42:0/2280355577 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fca5406b140 0x7fca54103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.667+0000 7fca5890e640 1 -- 172.21.15.42:0/2280355577 >> 172.21.15.42:0/2280355577 conn(0x7fca540fd910 msgr2=0x7fca541047a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.667+0000 7fca5890e640 1 -- 172.21.15.42:0/2280355577 shutdown_connections 2024-07-26T06:54:25.670 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:25.667+0000 7fca5890e640 1 -- 172.21.15.42:0/2280355577 wait complete. 2024-07-26T06:54:27.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:26 smithi042 ceph-mon[32775]: pgmap v523: 97 pgs: 97 active+clean; 50 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.5 MiB/s wr, 254 op/s 2024-07-26T06:54:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:26 smithi164 ceph-mon[36970]: pgmap v523: 97 pgs: 97 active+clean; 50 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.5 MiB/s wr, 254 op/s 2024-07-26T06:54:27.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:26 smithi163 ceph-mon[36873]: pgmap v523: 97 pgs: 97 active+clean; 50 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.5 MiB/s wr, 254 op/s 2024-07-26T06:54:28.213 INFO:tasks.workunit.client.0.smithi042.stdout:30000 trunc from 0x7a5444 to 0x143a5 2024-07-26T06:54:29.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:28 smithi042 ceph-mon[32775]: pgmap v524: 97 pgs: 97 active+clean; 49 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.0 MiB/s wr, 404 op/s 2024-07-26T06:54:29.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:28 smithi164 ceph-mon[36970]: pgmap v524: 97 pgs: 97 active+clean; 49 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.0 MiB/s wr, 404 op/s 2024-07-26T06:54:29.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:28 smithi163 ceph-mon[36873]: pgmap v524: 97 pgs: 97 active+clean; 49 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 2.0 MiB/s wr, 404 op/s 2024-07-26T06:54:31.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:30 smithi042 ceph-mon[32775]: pgmap v525: 97 pgs: 97 active+clean; 44 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.2 MiB/s wr, 571 op/s 2024-07-26T06:54:31.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:30 smithi164 ceph-mon[36970]: pgmap v525: 97 pgs: 97 active+clean; 44 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.2 MiB/s wr, 571 op/s 2024-07-26T06:54:31.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:30 smithi163 ceph-mon[36873]: pgmap v525: 97 pgs: 97 active+clean; 44 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.2 MiB/s wr, 571 op/s 2024-07-26T06:54:32.391 INFO:tasks.workunit.client.0.smithi042.stdout:truncating to largest ever: 0x9ffb01 2024-07-26T06:54:33.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:32 smithi042 ceph-mon[32775]: pgmap v526: 97 pgs: 97 active+clean; 44 MiB data, 931 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.3 MiB/s wr, 596 op/s 2024-07-26T06:54:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:32 smithi164 ceph-mon[36970]: pgmap v526: 97 pgs: 97 active+clean; 44 MiB data, 931 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.3 MiB/s wr, 596 op/s 2024-07-26T06:54:33.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:32 smithi163 ceph-mon[36873]: pgmap v526: 97 pgs: 97 active+clean; 44 MiB data, 931 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.3 MiB/s wr, 596 op/s 2024-07-26T06:54:35.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:34 smithi042 ceph-mon[32775]: pgmap v527: 97 pgs: 97 active+clean; 45 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.8 MiB/s wr, 680 op/s 2024-07-26T06:54:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:34 smithi164 ceph-mon[36970]: pgmap v527: 97 pgs: 97 active+clean; 45 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.8 MiB/s wr, 680 op/s 2024-07-26T06:54:35.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:34 smithi163 ceph-mon[36873]: pgmap v527: 97 pgs: 97 active+clean; 45 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.8 MiB/s wr, 680 op/s 2024-07-26T06:54:35.959 INFO:tasks.workunit.client.0.smithi042.stdout:40000 mapwrite 0x1becad thru 0x1c6a92 (0x7de6 bytes) 2024-07-26T06:54:37.058 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:36 smithi163 ceph-mon[36873]: pgmap v528: 97 pgs: 97 active+clean; 45 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.9 MiB/s wr, 687 op/s 2024-07-26T06:54:37.058 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:36 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:54:37.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:36 smithi042 ceph-mon[32775]: pgmap v528: 97 pgs: 97 active+clean; 45 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.9 MiB/s wr, 687 op/s 2024-07-26T06:54:37.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:36 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:54:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:36 smithi164 ceph-mon[36970]: pgmap v528: 97 pgs: 97 active+clean; 45 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.9 MiB/s wr, 687 op/s 2024-07-26T06:54:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:36 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:54:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.775+0000 7f368690e640 1 -- 172.21.15.42:0/738078107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3680101fd0 msgr2=0x7f3680102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.775+0000 7f368690e640 1 --2- 172.21.15.42:0/738078107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3680101fd0 0x7f3680102440 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3670009900 tx=0x7f367002a410 comp rx=0 tx=0).stop 2024-07-26T06:54:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.775+0000 7f368690e640 1 -- 172.21.15.42:0/738078107 shutdown_connections 2024-07-26T06:54:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.775+0000 7f368690e640 1 --2- 172.21.15.42:0/738078107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36801039f0 0x7f368010a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.775+0000 7f368690e640 1 --2- 172.21.15.42:0/738078107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3680103050 0x7f3680103460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.777 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.775+0000 7f368690e640 1 --2- 172.21.15.42:0/738078107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3680101fd0 0x7f3680102440 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.777 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.775+0000 7f368690e640 1 -- 172.21.15.42:0/738078107 >> 172.21.15.42:0/738078107 conn(0x7f36800fd910 msgr2=0x7f36800ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:38.777 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.775+0000 7f368690e640 1 -- 172.21.15.42:0/738078107 shutdown_connections 2024-07-26T06:54:38.777 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.775+0000 7f368690e640 1 -- 172.21.15.42:0/738078107 wait complete. 2024-07-26T06:54:38.777 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.776+0000 7f368690e640 1 Processor -- start 2024-07-26T06:54:38.777 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.776+0000 7f368690e640 1 -- start start 2024-07-26T06:54:38.779 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.776+0000 7f368690e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3680101fd0 0x7f3680198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.779 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.776+0000 7f368690e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3680103050 0x7f3680198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.779 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.776+0000 7f368690e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36801039f0 0x7f3680199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.779 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.776+0000 7f368690e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f368010dc50 con 0x7f36801039f0 2024-07-26T06:54:38.779 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.776+0000 7f368690e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f368010dad0 con 0x7f3680101fd0 2024-07-26T06:54:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.776+0000 7f368690e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f368010ddd0 con 0x7f3680103050 2024-07-26T06:54:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.776+0000 7f3684eab640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36801039f0 0x7f3680199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.776+0000 7f367ffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3680101fd0 0x7f3680198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.776+0000 7f367ffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3680101fd0 0x7f3680198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:53588/0 (socket says 172.21.15.42:53588) 2024-07-26T06:54:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.776+0000 7f3684eab640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36801039f0 0x7f3680199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35462/0 (socket says 172.21.15.42:35462) 2024-07-26T06:54:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.776+0000 7f367ffff640 1 -- 172.21.15.42:0/572005245 learned_addr learned my addr 172.21.15.42:0/572005245 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.776+0000 7f367f7fe640 1 --2- 172.21.15.42:0/572005245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3680103050 0x7f3680198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.777+0000 7f3684eab640 1 -- 172.21.15.42:0/572005245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3680103050 msgr2=0x7f3680198ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.777+0000 7f3684eab640 1 --2- 172.21.15.42:0/572005245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3680103050 0x7f3680198ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.777+0000 7f3684eab640 1 -- 172.21.15.42:0/572005245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3680101fd0 msgr2=0x7f3680198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.777+0000 7f3684eab640 1 --2- 172.21.15.42:0/572005245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3680101fd0 0x7f3680198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.777+0000 7f3684eab640 1 -- 172.21.15.42:0/572005245 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f368019a2b0 con 0x7f36801039f0 2024-07-26T06:54:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.777+0000 7f3684eab640 1 --2- 172.21.15.42:0/572005245 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36801039f0 0x7f3680199a90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f367800eb20 tx=0x7f367800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.777+0000 7f367d7fa640 1 -- 172.21.15.42:0/572005245 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36780039c0 con 0x7f36801039f0 2024-07-26T06:54:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.777+0000 7f367d7fa640 1 -- 172.21.15.42:0/572005245 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3678003b60 con 0x7f36801039f0 2024-07-26T06:54:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.777+0000 7f368690e640 1 -- 172.21.15.42:0/572005245 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f368019a5a0 con 0x7f36801039f0 2024-07-26T06:54:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.777+0000 7f367d7fa640 1 -- 172.21.15.42:0/572005245 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3678010670 con 0x7f36801039f0 2024-07-26T06:54:38.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.777+0000 7f368690e640 1 -- 172.21.15.42:0/572005245 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36801a0c90 con 0x7f36801039f0 2024-07-26T06:54:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.780+0000 7f368d823740 1 Processor -- start 2024-07-26T06:54:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.780+0000 7f367d7fa640 1 -- 172.21.15.42:0/572005245 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f3678010810 con 0x7f36801039f0 2024-07-26T06:54:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.780+0000 7f368d823740 1 -- start start 2024-07-26T06:54:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.780+0000 7f368d823740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631561b0 0x5606634fd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.780+0000 7f368d823740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56066312d2f0 0x5606634fda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.780+0000 7f368d823740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5606631354d0 0x5606634fe550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.780+0000 7f368d823740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x56066346e3b0 con 0x5606631354d0 2024-07-26T06:54:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.780+0000 7f368d823740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56066345d8b0 con 0x5606631561b0 2024-07-26T06:54:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.780+0000 7f368d823740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5606634618f0 con 0x56066312d2f0 2024-07-26T06:54:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.781+0000 7f367ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56066312d2f0 0x5606634fda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.781+0000 7f367f7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631561b0 0x5606634fd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.781+0000 7f367ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56066312d2f0 0x5606634fda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:57572/0 (socket says 172.21.15.42:57572) 2024-07-26T06:54:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.781+0000 7f367ffff640 1 -- 172.21.15.42:0/1038072674 learned_addr learned my addr 172.21.15.42:0/1038072674 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.781+0000 7f367f7fe640 1 --2- 172.21.15.42:0/1038072674 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5606631354d0 0x5606634fe550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.781+0000 7f367ffff640 1 -- 172.21.15.42:0/1038072674 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631561b0 msgr2=0x5606634fd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.781+0000 7f367ffff640 1 --2- 172.21.15.42:0/1038072674 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631561b0 0x5606634fd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.781+0000 7f367ffff640 1 -- 172.21.15.42:0/1038072674 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5606631354d0 msgr2=0x5606634fe550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.781+0000 7f367ffff640 1 --2- 172.21.15.42:0/1038072674 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5606631354d0 0x5606634fe550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.781+0000 7f367ffff640 1 -- 172.21.15.42:0/1038072674 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606634049a0 con 0x56066312d2f0 2024-07-26T06:54:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.781+0000 7f367d7fa640 1 --2- 172.21.15.42:0/572005245 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36680b3bd0 0x7f36680b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.781+0000 7f367ffff640 1 --2- 172.21.15.42:0/1038072674 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56066312d2f0 0x5606634fda00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3670009a20 tx=0x7f36700315a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=715ec3a5c3843c85 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:38.788 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.781+0000 7f367f7fe640 1 --2- 172.21.15.42:0/572005245 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36680b3bd0 0x7f36680b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.781+0000 7f367d7fa640 1 -- 172.21.15.42:0/572005245 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f36780e4310 con 0x7f36801039f0 2024-07-26T06:54:38.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.781+0000 7f36667fc640 1 -- 172.21.15.42:0/1038072674 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3670035030 con 0x56066312d2f0 2024-07-26T06:54:38.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.782+0000 7f36667fc640 1 -- 172.21.15.42:0/1038072674 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3670040070 con 0x56066312d2f0 2024-07-26T06:54:38.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.782+0000 7f36667fc640 1 -- 172.21.15.42:0/1038072674 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3670003970 con 0x56066312d2f0 2024-07-26T06:54:38.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.782+0000 7f368d823740 1 -- 172.21.15.42:0/1038072674 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56066312d2f0 msgr2=0x5606634fda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:38.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.782+0000 7f368d823740 1 --2- 172.21.15.42:0/1038072674 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56066312d2f0 0x5606634fda00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3670009a20 tx=0x7f36700315a0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.782+0000 7f367f7fe640 1 --2- 172.21.15.42:0/572005245 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36680b3bd0 0x7f36680b60e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f367400f560 tx=0x7f3674004730 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:38.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.783+0000 7f368d823740 1 -- 172.21.15.42:0/1038072674 shutdown_connections 2024-07-26T06:54:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.783+0000 7f368d823740 1 --2- 172.21.15.42:0/1038072674 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5606631354d0 0x5606634fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.783+0000 7f368d823740 1 --2- 172.21.15.42:0/1038072674 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56066312d2f0 0x5606634fda00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.783+0000 7f368d823740 1 --2- 172.21.15.42:0/1038072674 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631561b0 0x5606634fd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.783+0000 7f368d823740 1 -- 172.21.15.42:0/1038072674 >> 172.21.15.42:0/1038072674 conn(0x56066314a300 msgr2=0x5606634f6c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.783+0000 7f368d823740 1 -- 172.21.15.42:0/1038072674 shutdown_connections 2024-07-26T06:54:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.783+0000 7f368d823740 1 -- 172.21.15.42:0/1038072674 wait complete. 2024-07-26T06:54:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.783+0000 7f368d823740 1 -- start start 2024-07-26T06:54:38.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.783+0000 7f368d823740 1 Processor -- start 2024-07-26T06:54:38.791 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f368d823740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x560663506cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.791 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f368d823740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5606631354d0 0x5606635073b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.791 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f368d823740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631561b0 0x560663507f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.791 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f368d823740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x560663397ab0 con 0x56066312d2f0 2024-07-26T06:54:38.791 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f368d823740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5606633e5cf0 con 0x5606631561b0 2024-07-26T06:54:38.791 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f368d823740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5606634ff010 con 0x5606631354d0 2024-07-26T06:54:38.791 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f368d823740 10 client.0 _ll_register_callbacks cb 0x5606634579f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:54:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f367ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x560663506cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f3684eab640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631561b0 0x560663507f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f367ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x560663506cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35470/0 (socket says 172.21.15.42:35470) 2024-07-26T06:54:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f367ffff640 1 -- 172.21.15.42:0/3075993247 learned_addr learned my addr 172.21.15.42:0/3075993247 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f367f7fe640 1 --2- 172.21.15.42:0/3075993247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5606631354d0 0x5606635073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f3684eab640 1 -- 172.21.15.42:0/3075993247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5606631354d0 msgr2=0x5606635073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f3684eab640 1 --2- 172.21.15.42:0/3075993247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5606631354d0 0x5606635073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f3684eab640 1 -- 172.21.15.42:0/3075993247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 msgr2=0x560663506cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f3684eab640 1 --2- 172.21.15.42:0/3075993247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x560663506cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f3684eab640 1 -- 172.21.15.42:0/3075993247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606631c1790 con 0x5606631561b0 2024-07-26T06:54:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f367ffff640 1 --2- 172.21.15.42:0/3075993247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x560663506cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:54:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f3684eab640 1 --2- 172.21.15.42:0/3075993247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631561b0 0x560663507f20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f36780e1e40 tx=0x7f36780aba80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=214f828450c5eb9f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.784+0000 7f365f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:54:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.785+0000 7f365f7fe640 1 -- 172.21.15.42:0/3075993247 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36780c4020 con 0x5606631561b0 2024-07-26T06:54:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.785+0000 7f365f7fe640 1 -- 172.21.15.42:0/3075993247 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f36780b3e60 con 0x5606631561b0 2024-07-26T06:54:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.785+0000 7f368d823740 1 -- 172.21.15.42:0/3075993247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3670040070 con 0x5606631561b0 2024-07-26T06:54:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.785+0000 7f365f7fe640 1 -- 172.21.15.42:0/3075993247 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f36780b4060 con 0x5606631561b0 2024-07-26T06:54:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.785+0000 7f368d823740 10 client.25508 fetch_fsmap learned FSMap version 37 2024-07-26T06:54:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.785+0000 7f368d823740 1 -- 172.21.15.42:0/3075993247 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56066323a620 con 0x5606631561b0 2024-07-26T06:54:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.785+0000 7f365f7fe640 1 -- 172.21.15.42:0/3075993247 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3670040070 con 0x5606631561b0 2024-07-26T06:54:38.794 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.785+0000 7f365f7fe640 1 -- 172.21.15.42:0/3075993247 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7f36780a6bb0 con 0x5606631561b0 2024-07-26T06:54:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.785+0000 7f368d823740 10 client.25508 fetch_fsmap finished waiting for FSMap version 37 2024-07-26T06:54:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.785+0000 7f368d823740 10 client.25508 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:54:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.785+0000 7f368d823740 20 client.25508 populate_metadata read hostname 'smithi042' 2024-07-26T06:54:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.785+0000 7f368d823740 1 --2- 172.21.15.42:0/3075993247 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x560663114d80 0x5606635145c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.785+0000 7f368d823740 4 client.25508 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:54:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.785+0000 7f368d823740 1 -- 172.21.15.42:0/3075993247 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560663470300 con 0x560663114d80 2024-07-26T06:54:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.785+0000 7f367ffff640 1 --2- 172.21.15.42:0/3075993247 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x560663114d80 0x5606635145c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.786+0000 7f365f7fe640 10 client.25508 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:38.796 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.786+0000 7f367ffff640 1 --2- 172.21.15.42:0/3075993247 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x560663114d80 0x5606635145c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=216c25a780883e80 server_cookie=cb6525f78dae012a in_seq=0 out_seq=0 2024-07-26T06:54:38.797 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.787+0000 7f365f7fe640 1 -- 172.21.15.42:0/3075993247 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560663470300 con 0x560663114d80 2024-07-26T06:54:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.787+0000 7f365f7fe640 10 client.25508 handle_command_reply: tid=0 2024-07-26T06:54:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.787+0000 7f368d823740 1 client.25508 shutdown 2024-07-26T06:54:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.787+0000 7f368d823740 1 -- 172.21.15.42:0/3075993247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631561b0 msgr2=0x560663507f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.787+0000 7f368d823740 1 --2- 172.21.15.42:0/3075993247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631561b0 0x560663507f20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f36780e1e40 tx=0x7f36780aba80 comp rx=0 tx=0).stop 2024-07-26T06:54:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.787+0000 7f368d823740 1 -- 172.21.15.42:0/3075993247 shutdown_connections 2024-07-26T06:54:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.787+0000 7f368d823740 1 --2- 172.21.15.42:0/3075993247 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x560663114d80 0x5606635145c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.787+0000 7f365f7fe640 0 client.25508 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:38.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.787+0000 7f368d823740 1 --2- 172.21.15.42:0/3075993247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631561b0 0x560663507f20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.787+0000 7f368d823740 1 --2- 172.21.15.42:0/3075993247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5606631354d0 0x5606635073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.787+0000 7f368d823740 1 --2- 172.21.15.42:0/3075993247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x560663506cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.787+0000 7f368d823740 1 -- 172.21.15.42:0/3075993247 >> 172.21.15.42:0/3075993247 conn(0x56066314a300 msgr2=0x5606634f52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:38.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.787+0000 7f368d823740 1 -- 172.21.15.42:0/3075993247 shutdown_connections 2024-07-26T06:54:38.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.787+0000 7f368d823740 1 -- 172.21.15.42:0/3075993247 wait complete. 2024-07-26T06:54:38.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.787+0000 7f368d823740 20 client.25508 trim_cache size 0 max 16384 2024-07-26T06:54:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.805+0000 7f368d823740 1 Processor -- start 2024-07-26T06:54:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.805+0000 7f368d823740 1 -- start start 2024-07-26T06:54:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.805+0000 7f368d823740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560663114d80 0x5606634f19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.805+0000 7f368d823740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x5606634fd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.805+0000 7f368d823740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631354d0 0x5606634fe220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.805+0000 7f368d823740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f36801a8870 con 0x56066312d2f0 2024-07-26T06:54:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.805+0000 7f368d823740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f36801aa470 con 0x5606631354d0 2024-07-26T06:54:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.805+0000 7f368d823740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f36801a8f40 con 0x560663114d80 2024-07-26T06:54:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.805+0000 7f367f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x5606634fd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.805+0000 7f367f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x5606634fd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35478/0 (socket says 172.21.15.42:35478) 2024-07-26T06:54:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.805+0000 7f367f7fe640 1 -- 172.21.15.42:0/290128470 learned_addr learned my addr 172.21.15.42:0/290128470 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.805+0000 7f3684eab640 1 --2- 172.21.15.42:0/290128470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631354d0 0x5606634fe220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.805+0000 7f367ffff640 1 --2- 172.21.15.42:0/290128470 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560663114d80 0x5606634f19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.806+0000 7f367f7fe640 1 -- 172.21.15.42:0/290128470 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560663114d80 msgr2=0x5606634f19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.806+0000 7f367f7fe640 1 --2- 172.21.15.42:0/290128470 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560663114d80 0x5606634f19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.806+0000 7f367f7fe640 1 -- 172.21.15.42:0/290128470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631354d0 msgr2=0x5606634fe220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.806+0000 7f367f7fe640 1 --2- 172.21.15.42:0/290128470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631354d0 0x5606634fe220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.806+0000 7f367f7fe640 1 -- 172.21.15.42:0/290128470 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560663507720 con 0x56066312d2f0 2024-07-26T06:54:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.806+0000 7f367ffff640 1 --2- 172.21.15.42:0/290128470 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560663114d80 0x5606634f19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:54:38.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.806+0000 7f367f7fe640 1 --2- 172.21.15.42:0/290128470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x5606634fd050 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f3674010740 tx=0x7f3674012070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e3dd8b1a3ce3b4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:38.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.806+0000 7f3665ffb640 1 -- 172.21.15.42:0/290128470 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3674019040 con 0x56066312d2f0 2024-07-26T06:54:38.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.806+0000 7f3665ffb640 1 -- 172.21.15.42:0/290128470 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f367402f070 con 0x56066312d2f0 2024-07-26T06:54:38.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.806+0000 7f3665ffb640 1 -- 172.21.15.42:0/290128470 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367402a9e0 con 0x56066312d2f0 2024-07-26T06:54:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.807+0000 7f368d823740 1 -- 172.21.15.42:0/290128470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 msgr2=0x5606634fd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:38.811 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.807+0000 7f368d823740 1 --2- 172.21.15.42:0/290128470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x5606634fd050 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f3674010740 tx=0x7f3674012070 comp rx=0 tx=0).stop 2024-07-26T06:54:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.807+0000 7f368d823740 1 -- 172.21.15.42:0/290128470 shutdown_connections 2024-07-26T06:54:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.807+0000 7f368d823740 1 --2- 172.21.15.42:0/290128470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631354d0 0x5606634fe220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.807+0000 7f368d823740 1 --2- 172.21.15.42:0/290128470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x5606634fd050 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.807+0000 7f368d823740 1 --2- 172.21.15.42:0/290128470 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560663114d80 0x5606634f19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.807+0000 7f368d823740 1 -- 172.21.15.42:0/290128470 >> 172.21.15.42:0/290128470 conn(0x56066314a300 msgr2=0x5606634f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.807+0000 7f368d823740 1 -- 172.21.15.42:0/290128470 shutdown_connections 2024-07-26T06:54:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.807+0000 7f368d823740 1 -- 172.21.15.42:0/290128470 wait complete. 2024-07-26T06:54:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.807+0000 7f368d823740 1 -- start start 2024-07-26T06:54:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.808+0000 7f368d823740 1 Processor -- start 2024-07-26T06:54:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.808+0000 7f368d823740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560663114d80 0x56066350e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.808+0000 7f368d823740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x56066350d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.808+0000 7f368d823740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631354d0 0x56066350d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.808+0000 7f368d823740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f36801a9da0 con 0x56066312d2f0 2024-07-26T06:54:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.808+0000 7f368d823740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f36801a9610 con 0x5606631354d0 2024-07-26T06:54:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.808+0000 7f368d823740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f36801a6850 con 0x560663114d80 2024-07-26T06:54:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.808+0000 7f368d823740 10 client.0 _ll_register_callbacks cb 0x7f36780a7100 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:54:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.808+0000 7f367f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x56066350d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.808+0000 7f367f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x56066350d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35480/0 (socket says 172.21.15.42:35480) 2024-07-26T06:54:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.808+0000 7f367f7fe640 1 -- 172.21.15.42:0/2640186937 learned_addr learned my addr 172.21.15.42:0/2640186937 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.808+0000 7f367ffff640 1 --2- 172.21.15.42:0/2640186937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560663114d80 0x56066350e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.808+0000 7f3684eab640 1 --2- 172.21.15.42:0/2640186937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631354d0 0x56066350d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.808+0000 7f367f7fe640 1 -- 172.21.15.42:0/2640186937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560663114d80 msgr2=0x56066350e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.809+0000 7f367f7fe640 1 --2- 172.21.15.42:0/2640186937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560663114d80 0x56066350e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.809+0000 7f367f7fe640 1 -- 172.21.15.42:0/2640186937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631354d0 msgr2=0x56066350d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.809+0000 7f367f7fe640 1 --2- 172.21.15.42:0/2640186937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631354d0 0x56066350d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.809+0000 7f367f7fe640 1 -- 172.21.15.42:0/2640186937 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56066350dbc0 con 0x56066312d2f0 2024-07-26T06:54:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.809+0000 7f367ffff640 1 --2- 172.21.15.42:0/2640186937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560663114d80 0x56066350e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:54:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.809+0000 7f3684eab640 1 --2- 172.21.15.42:0/2640186937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631354d0 0x56066350d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:54:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.809+0000 7f367f7fe640 1 --2- 172.21.15.42:0/2640186937 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x56066350d140 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f367402ae60 tx=0x7f367402ae90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1855af78e18ffe97 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.809+0000 7f365f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:54:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.809+0000 7f365f7fe640 1 -- 172.21.15.42:0/2640186937 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367401eb10 con 0x56066312d2f0 2024-07-26T06:54:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.809+0000 7f365f7fe640 1 -- 172.21.15.42:0/2640186937 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f367401ecb0 con 0x56066312d2f0 2024-07-26T06:54:38.814 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.809+0000 7f368d823740 1 -- 172.21.15.42:0/2640186937 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f367402f070 con 0x56066312d2f0 2024-07-26T06:54:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.813+0000 7f365f7fe640 1 -- 172.21.15.42:0/2640186937 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3674035460 con 0x56066312d2f0 2024-07-26T06:54:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.813+0000 7f368d823740 10 client.15930 fetch_fsmap learned FSMap version 37 2024-07-26T06:54:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.813+0000 7f368d823740 1 -- 172.21.15.42:0/2640186937 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56066357ba30 con 0x56066312d2f0 2024-07-26T06:54:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.813+0000 7f365f7fe640 1 -- 172.21.15.42:0/2640186937 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f367403a070 con 0x56066312d2f0 2024-07-26T06:54:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.813+0000 7f365f7fe640 1 -- 172.21.15.42:0/2640186937 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7f3674033410 con 0x56066312d2f0 2024-07-26T06:54:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.813+0000 7f368d823740 10 client.15930 fetch_fsmap finished waiting for FSMap version 37 2024-07-26T06:54:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.813+0000 7f368d823740 10 client.15930 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:54:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.813+0000 7f368d823740 20 client.15930 populate_metadata read hostname 'smithi042' 2024-07-26T06:54:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.813+0000 7f368d823740 1 --2- 172.21.15.42:0/2640186937 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5606631561b0 0x56066357e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.813+0000 7f368d823740 4 client.15930 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:54:38.816 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.813+0000 7f368d823740 1 -- 172.21.15.42:0/2640186937 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f367402f070 con 0x5606631561b0 2024-07-26T06:54:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.813+0000 7f367ffff640 1 --2- 172.21.15.42:0/2640186937 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5606631561b0 0x56066357e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.814+0000 7f365f7fe640 10 client.15930 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:38.817 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.814+0000 7f367ffff640 1 --2- 172.21.15.42:0/2640186937 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5606631561b0 0x56066357e3c0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=585eb4e281f19237 server_cookie=f7212cb5697e2d2 in_seq=0 out_seq=0 2024-07-26T06:54:38.830 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.829+0000 7f365f7fe640 1 -- 172.21.15.42:0/2640186937 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f367402f070 con 0x5606631561b0 2024-07-26T06:54:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.829+0000 7f365f7fe640 10 client.15930 handle_command_reply: tid=0 2024-07-26T06:54:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.829+0000 7f368d823740 1 client.15930 shutdown 2024-07-26T06:54:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.829+0000 7f368d823740 1 -- 172.21.15.42:0/2640186937 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 msgr2=0x56066350d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.829+0000 7f368d823740 1 --2- 172.21.15.42:0/2640186937 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x56066350d140 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f367402ae60 tx=0x7f367402ae90 comp rx=0 tx=0).stop 2024-07-26T06:54:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.829+0000 7f368d823740 1 -- 172.21.15.42:0/2640186937 shutdown_connections 2024-07-26T06:54:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.829+0000 7f368d823740 1 --2- 172.21.15.42:0/2640186937 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5606631561b0 0x56066357e3c0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.829+0000 7f368d823740 1 --2- 172.21.15.42:0/2640186937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5606631354d0 0x56066350d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.829+0000 7f368d823740 1 --2- 172.21.15.42:0/2640186937 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56066312d2f0 0x56066350d140 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.829+0000 7f365f7fe640 0 client.15930 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.829+0000 7f368d823740 1 --2- 172.21.15.42:0/2640186937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x560663114d80 0x56066350e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.829+0000 7f368d823740 1 -- 172.21.15.42:0/2640186937 >> 172.21.15.42:0/2640186937 conn(0x56066314a300 msgr2=0x5606634f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:38.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.830+0000 7f368d823740 1 -- 172.21.15.42:0/2640186937 shutdown_connections 2024-07-26T06:54:38.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.830+0000 7f368d823740 1 -- 172.21.15.42:0/2640186937 wait complete. 2024-07-26T06:54:38.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.830+0000 7f368d823740 20 client.15930 trim_cache size 0 max 16384 2024-07-26T06:54:38.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:54:38.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:54:38.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:54:38.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:54:38.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.830+0000 7f368690e640 1 -- 172.21.15.42:0/572005245 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36680b3bd0 msgr2=0x7f36680b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:38.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.830+0000 7f368690e640 1 --2- 172.21.15.42:0/572005245 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36680b3bd0 0x7f36680b60e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f367400f560 tx=0x7f3674004730 comp rx=0 tx=0).stop 2024-07-26T06:54:38.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.830+0000 7f368690e640 1 -- 172.21.15.42:0/572005245 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36801039f0 msgr2=0x7f3680199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:38.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.830+0000 7f368690e640 1 --2- 172.21.15.42:0/572005245 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36801039f0 0x7f3680199a90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f367800eb20 tx=0x7f367800c2d0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.830+0000 7f368690e640 1 -- 172.21.15.42:0/572005245 shutdown_connections 2024-07-26T06:54:38.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.830+0000 7f368690e640 1 --2- 172.21.15.42:0/572005245 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f36680b3bd0 0x7f36680b60e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.830+0000 7f368690e640 1 --2- 172.21.15.42:0/572005245 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f36801039f0 0x7f3680199a90 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.830+0000 7f368690e640 1 --2- 172.21.15.42:0/572005245 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3680103050 0x7f3680198ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.830+0000 7f368690e640 1 --2- 172.21.15.42:0/572005245 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3680101fd0 0x7f3680198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.830+0000 7f368690e640 1 -- 172.21.15.42:0/572005245 >> 172.21.15.42:0/572005245 conn(0x7f36800fd910 msgr2=0x7f3680106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.830+0000 7f368690e640 1 -- 172.21.15.42:0/572005245 shutdown_connections 2024-07-26T06:54:38.834 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:38.830+0000 7f368690e640 1 -- 172.21.15.42:0/572005245 wait complete. 2024-07-26T06:54:39.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:38 smithi042 ceph-mon[32775]: pgmap v529: 97 pgs: 97 active+clean; 46 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3.6 MiB/s wr, 661 op/s 2024-07-26T06:54:39.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:38 smithi164 ceph-mon[36970]: pgmap v529: 97 pgs: 97 active+clean; 46 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3.6 MiB/s wr, 661 op/s 2024-07-26T06:54:39.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:38 smithi163 ceph-mon[36873]: pgmap v529: 97 pgs: 97 active+clean; 46 MiB data, 932 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3.6 MiB/s wr, 661 op/s 2024-07-26T06:54:41.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:40 smithi042 ceph-mon[32775]: pgmap v530: 97 pgs: 97 active+clean; 54 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 4.3 MiB/s wr, 763 op/s 2024-07-26T06:54:41.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:40 smithi164 ceph-mon[36970]: pgmap v530: 97 pgs: 97 active+clean; 54 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 4.3 MiB/s wr, 763 op/s 2024-07-26T06:54:41.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:40 smithi163 ceph-mon[36873]: pgmap v530: 97 pgs: 97 active+clean; 54 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 4.3 MiB/s wr, 763 op/s 2024-07-26T06:54:43.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:42 smithi042 ceph-mon[32775]: pgmap v531: 97 pgs: 97 active+clean; 54 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 3.5 MiB/s wr, 635 op/s 2024-07-26T06:54:43.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:42 smithi164 ceph-mon[36970]: pgmap v531: 97 pgs: 97 active+clean; 54 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 3.5 MiB/s wr, 635 op/s 2024-07-26T06:54:43.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:42 smithi163 ceph-mon[36873]: pgmap v531: 97 pgs: 97 active+clean; 54 MiB data, 968 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 3.5 MiB/s wr, 635 op/s 2024-07-26T06:54:44.414 INFO:tasks.workunit.client.0.smithi042.stdout:50000 write 0x5304f4 thru 0x53fa11 (0xf51e bytes) 2024-07-26T06:54:44.415 INFO:tasks.workunit.client.0.smithi042.stdout:All 50000 operations completed A-OK! 2024-07-26T06:54:44.416 INFO:tasks.workunit.client.0.smithi042.stderr:+ ./fsx -z 100MB -N 50000 -p 10000 -l 104857600 2024-07-26T06:54:44.418 INFO:tasks.workunit.client.0.smithi042.stdout:Seed set to 1 2024-07-26T06:54:45.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:44 smithi042 ceph-mon[32775]: pgmap v532: 97 pgs: 97 active+clean; 52 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 743 op/s 2024-07-26T06:54:45.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:44 smithi164 ceph-mon[36970]: pgmap v532: 97 pgs: 97 active+clean; 52 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 743 op/s 2024-07-26T06:54:45.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:44 smithi163 ceph-mon[36873]: pgmap v532: 97 pgs: 97 active+clean; 52 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 743 op/s 2024-07-26T06:54:45.606 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support fallocate mode 0, disabling! 2024-07-26T06:54:45.606 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support fallocate mode FALLOC_FL_KEEP_SIZE, disabling! 2024-07-26T06:54:45.608 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support fallocate mode FALLOC_FL_COLLAPSE_RANGE, disabling! 2024-07-26T06:54:45.608 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support fallocate mode FALLOC_FL_INSERT_RANGE, disabling! 2024-07-26T06:54:45.608 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support clone range, disabling! 2024-07-26T06:54:45.611 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support dedupe range, disabling! 2024-07-26T06:54:45.614 INFO:tasks.workunit.client.0.smithi042.stderr:main: filesystem does not support exchange range, disabling! 2024-07-26T06:54:45.614 INFO:tasks.workunit.client.0.smithi042.stdout:skipping zero size read 2024-07-26T06:54:45.625 INFO:tasks.workunit.client.0.smithi042.stdout:truncating to largest ever: 0x1c37b90 2024-07-26T06:54:45.633 INFO:tasks.workunit.client.0.smithi042.stdout:truncating to largest ever: 0x5bd330d 2024-07-26T06:54:45.648 INFO:tasks.workunit.client.0.smithi042.stdout:copying to largest ever: 0x637bf99 2024-07-26T06:54:45.818 INFO:tasks.workunit.client.0.smithi042.stdout:copying to largest ever: 0x63d9b9c 2024-07-26T06:54:47.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:46 smithi042 ceph-mon[32775]: pgmap v533: 97 pgs: 97 active+clean; 51 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.7 MiB/s wr, 656 op/s 2024-07-26T06:54:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:46 smithi164 ceph-mon[36970]: pgmap v533: 97 pgs: 97 active+clean; 51 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.7 MiB/s wr, 656 op/s 2024-07-26T06:54:47.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:46 smithi163 ceph-mon[36873]: pgmap v533: 97 pgs: 97 active+clean; 51 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.7 MiB/s wr, 656 op/s 2024-07-26T06:54:49.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:48 smithi042 ceph-mon[32775]: pgmap v534: 97 pgs: 97 active+clean; 55 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.5 MiB/s wr, 668 op/s 2024-07-26T06:54:49.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:48 smithi164 ceph-mon[36970]: pgmap v534: 97 pgs: 97 active+clean; 55 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.5 MiB/s wr, 668 op/s 2024-07-26T06:54:49.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:48 smithi163 ceph-mon[36873]: pgmap v534: 97 pgs: 97 active+clean; 55 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.5 MiB/s wr, 668 op/s 2024-07-26T06:54:51.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:50 smithi042 ceph-mon[32775]: pgmap v535: 97 pgs: 97 active+clean; 60 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 790 op/s 2024-07-26T06:54:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:50 smithi164 ceph-mon[36970]: pgmap v535: 97 pgs: 97 active+clean; 60 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 790 op/s 2024-07-26T06:54:51.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:50 smithi163 ceph-mon[36873]: pgmap v535: 97 pgs: 97 active+clean; 60 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 790 op/s 2024-07-26T06:54:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.951+0000 7fa9ff0c2640 1 -- 172.21.15.42:0/3593642409 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9f8108eb0 msgr2=0x7fa9f810b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.951+0000 7fa9ff0c2640 1 --2- 172.21.15.42:0/3593642409 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9f8108eb0 0x7fa9f810b350 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009ca0 tx=0x7fa9f002a8c0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.951+0000 7fa9ff0c2640 1 -- 172.21.15.42:0/3593642409 shutdown_connections 2024-07-26T06:54:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.951+0000 7fa9ff0c2640 1 --2- 172.21.15.42:0/3593642409 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9f8108eb0 0x7fa9f810b350 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.951+0000 7fa9ff0c2640 1 --2- 172.21.15.42:0/3593642409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9f8106550 0x7fa9f8108970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.951+0000 7fa9ff0c2640 1 --2- 172.21.15.42:0/3593642409 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9f81014c0 0x7fa9f81018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.952+0000 7fa9ff0c2640 1 -- 172.21.15.42:0/3593642409 >> 172.21.15.42:0/3593642409 conn(0x7fa9f80fd0c0 msgr2=0x7fa9f80ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.952+0000 7fa9ff0c2640 1 -- 172.21.15.42:0/3593642409 shutdown_connections 2024-07-26T06:54:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.952+0000 7fa9ff0c2640 1 -- 172.21.15.42:0/3593642409 wait complete. 2024-07-26T06:54:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.952+0000 7fa9ff0c2640 1 Processor -- start 2024-07-26T06:54:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.952+0000 7fa9ff0c2640 1 -- start start 2024-07-26T06:54:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9ff0c2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9f81014c0 0x7fa9f81998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9ff0c2640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9f8106550 0x7fa9f8199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9ff0c2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9f8108eb0 0x7fa9f819aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9ff0c2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f810ed60 con 0x7fa9f81014c0 2024-07-26T06:54:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9ff0c2640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f810ebe0 con 0x7fa9f8106550 2024-07-26T06:54:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9ff0c2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f810eee0 con 0x7fa9f8108eb0 2024-07-26T06:54:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9fce5e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9f81014c0 0x7fa9f81998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9fd65f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9f8108eb0 0x7fa9f819aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9fce5e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9f81014c0 0x7fa9f81998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53246/0 (socket says 172.21.15.42:53246) 2024-07-26T06:54:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9fce5e640 1 -- 172.21.15.42:0/2087570019 learned_addr learned my addr 172.21.15.42:0/2087570019 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9f7fff640 1 --2- 172.21.15.42:0/2087570019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9f8106550 0x7fa9f8199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9fce5e640 1 -- 172.21.15.42:0/2087570019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9f8108eb0 msgr2=0x7fa9f819aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9fce5e640 1 --2- 172.21.15.42:0/2087570019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9f8108eb0 0x7fa9f819aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9fce5e640 1 -- 172.21.15.42:0/2087570019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9f8106550 msgr2=0x7fa9f8199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9fce5e640 1 --2- 172.21.15.42:0/2087570019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9f8106550 0x7fa9f8199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9fce5e640 1 -- 172.21.15.42:0/2087570019 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f819b300 con 0x7fa9f81014c0 2024-07-26T06:54:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9f7fff640 1 --2- 172.21.15.42:0/2087570019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9f8106550 0x7fa9f8199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:54:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.953+0000 7fa9fce5e640 1 --2- 172.21.15.42:0/2087570019 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9f81014c0 0x7fa9f81998c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec00ea60 tx=0x7fa9ec00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.954+0000 7fa9f5ffb640 1 -- 172.21.15.42:0/2087570019 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec00cde0 con 0x7fa9f81014c0 2024-07-26T06:54:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.954+0000 7fa9ff0c2640 1 -- 172.21.15.42:0/2087570019 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9f819b5f0 con 0x7fa9f81014c0 2024-07-26T06:54:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.954+0000 7fa9f5ffb640 1 -- 172.21.15.42:0/2087570019 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9ec0035f0 con 0x7fa9f81014c0 2024-07-26T06:54:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.954+0000 7fa9f5ffb640 1 -- 172.21.15.42:0/2087570019 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec010690 con 0x7fa9f81014c0 2024-07-26T06:54:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.954+0000 7fa9ff0c2640 1 -- 172.21.15.42:0/2087570019 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9f81a1ce0 con 0x7fa9f81014c0 2024-07-26T06:54:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.957+0000 7faa05fe3740 1 Processor -- start 2024-07-26T06:54:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.957+0000 7faa05fe3740 1 -- start start 2024-07-26T06:54:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.957+0000 7fa9f5ffb640 1 -- 172.21.15.42:0/2087570019 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fa9ec010830 con 0x7fa9f81014c0 2024-07-26T06:54:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.957+0000 7faa05fe3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8d11b0 0x56293ec76180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.957+0000 7faa05fe3740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8a82f0 0x56293ec767f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.957+0000 7faa05fe3740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec77340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.957+0000 7faa05fe3740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x56293ebe93b0 con 0x56293e8b04d0 2024-07-26T06:54:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.957+0000 7faa05fe3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56293ebd88b0 con 0x56293e8d11b0 2024-07-26T06:54:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.957+0000 7faa05fe3740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x56293ebdc8f0 con 0x56293e8a82f0 2024-07-26T06:54:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.957+0000 7fa9f7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec77340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.957+0000 7fa9fce5e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8a82f0 0x56293ec767f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.958+0000 7fa9f7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec77340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53260/0 (socket says 172.21.15.42:53260) 2024-07-26T06:54:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.958+0000 7fa9f7fff640 1 -- 172.21.15.42:0/3203087677 learned_addr learned my addr 172.21.15.42:0/3203087677 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.958+0000 7fa9fd65f640 1 --2- 172.21.15.42:0/3203087677 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8d11b0 0x56293ec76180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.958+0000 7fa9f7fff640 1 -- 172.21.15.42:0/3203087677 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8a82f0 msgr2=0x56293ec767f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:51.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.958+0000 7fa9f7fff640 1 --2- 172.21.15.42:0/3203087677 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8a82f0 0x56293ec767f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.958+0000 7fa9f7fff640 1 -- 172.21.15.42:0/3203087677 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8d11b0 msgr2=0x56293ec76180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:51.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.958+0000 7fa9f7fff640 1 --2- 172.21.15.42:0/3203087677 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8d11b0 0x56293ec76180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.958+0000 7fa9f7fff640 1 -- 172.21.15.42:0/3203087677 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56293eb7f9a0 con 0x56293e8b04d0 2024-07-26T06:54:51.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.958+0000 7fa9fd65f640 1 --2- 172.21.15.42:0/3203087677 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8d11b0 0x56293ec76180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:54:51.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.958+0000 7fa9fce5e640 1 --2- 172.21.15.42:0/3203087677 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8a82f0 0x56293ec767f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:54:51.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.958+0000 7fa9f7fff640 1 --2- 172.21.15.42:0/3203087677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec77340 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4011540 tx=0x7fa9e4011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689cd91c0071c9a5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:51.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.958+0000 7fa9e2ffd640 1 -- 172.21.15.42:0/3203087677 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e400f640 con 0x56293e8b04d0 2024-07-26T06:54:51.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.958+0000 7fa9e2ffd640 1 -- 172.21.15.42:0/3203087677 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9e400fc90 con 0x56293e8b04d0 2024-07-26T06:54:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.958+0000 7fa9e2ffd640 1 -- 172.21.15.42:0/3203087677 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4019c80 con 0x56293e8b04d0 2024-07-26T06:54:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.959+0000 7fa9f5ffb640 1 --2- 172.21.15.42:0/2087570019 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa9e80b3bd0 0x7fa9e80b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.959+0000 7fa9fd65f640 1 --2- 172.21.15.42:0/2087570019 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa9e80b3bd0 0x7fa9e80b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.959+0000 7fa9f5ffb640 1 -- 172.21.15.42:0/2087570019 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fa9ec0e3ef0 con 0x7fa9f81014c0 2024-07-26T06:54:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.959+0000 7faa05fe3740 1 -- 172.21.15.42:0/3203087677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 msgr2=0x56293ec77340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.959+0000 7faa05fe3740 1 --2- 172.21.15.42:0/3203087677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec77340 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4011540 tx=0x7fa9e4011a00 comp rx=0 tx=0).stop 2024-07-26T06:54:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.960+0000 7fa9fd65f640 1 --2- 172.21.15.42:0/2087570019 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa9e80b3bd0 0x7fa9e80b60e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009a60 tx=0x7fa9f002f310 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.960+0000 7faa05fe3740 1 -- 172.21.15.42:0/3203087677 shutdown_connections 2024-07-26T06:54:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.960+0000 7faa05fe3740 1 --2- 172.21.15.42:0/3203087677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec77340 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.960+0000 7faa05fe3740 1 --2- 172.21.15.42:0/3203087677 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8a82f0 0x56293ec767f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.960+0000 7faa05fe3740 1 --2- 172.21.15.42:0/3203087677 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8d11b0 0x56293ec76180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.960+0000 7faa05fe3740 1 -- 172.21.15.42:0/3203087677 >> 172.21.15.42:0/3203087677 conn(0x56293e8c5300 msgr2=0x56293ec71c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.960+0000 7faa05fe3740 1 -- 172.21.15.42:0/3203087677 shutdown_connections 2024-07-26T06:54:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.960+0000 7faa05fe3740 1 -- 172.21.15.42:0/3203087677 wait complete. 2024-07-26T06:54:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.960+0000 7faa05fe3740 1 -- start start 2024-07-26T06:54:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.960+0000 7faa05fe3740 1 Processor -- start 2024-07-26T06:54:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7faa05fe3740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8a82f0 0x56293ec7fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7faa05fe3740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec80200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7faa05fe3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8d11b0 0x56293ec80d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7faa05fe3740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x56293eb12ab0 con 0x56293e8b04d0 2024-07-26T06:54:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7faa05fe3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56293eb60cf0 con 0x56293e8d11b0 2024-07-26T06:54:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7faa05fe3740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x56293ec77e60 con 0x56293e8a82f0 2024-07-26T06:54:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7faa05fe3740 10 client.0 _ll_register_callbacks cb 0x56293ebd29f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:54:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7fa9f7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec80200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7fa9f7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec80200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53268/0 (socket says 172.21.15.42:53268) 2024-07-26T06:54:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7fa9f7fff640 1 -- 172.21.15.42:0/4003866629 learned_addr learned my addr 172.21.15.42:0/4003866629 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7fa9fd65f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8d11b0 0x56293ec80d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7fa9fce5e640 1 --2- 172.21.15.42:0/4003866629 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8a82f0 0x56293ec7fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7fa9f7fff640 1 -- 172.21.15.42:0/4003866629 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8a82f0 msgr2=0x56293ec7fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7fa9f7fff640 1 --2- 172.21.15.42:0/4003866629 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8a82f0 0x56293ec7fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7fa9f7fff640 1 -- 172.21.15.42:0/4003866629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8d11b0 msgr2=0x56293ec80d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7fa9f7fff640 1 --2- 172.21.15.42:0/4003866629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8d11b0 0x56293ec80d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7fa9f7fff640 1 -- 172.21.15.42:0/4003866629 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56293e93c790 con 0x56293e8b04d0 2024-07-26T06:54:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.961+0000 7fa9fd65f640 1 --2- 172.21.15.42:0/4003866629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8d11b0 0x56293ec80d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:54:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.962+0000 7fa9f7fff640 1 --2- 172.21.15.42:0/4003866629 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec80200 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4000c00 tx=0x7fa9e4019c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1163a681050277c7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.962+0000 7fa9d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:54:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.962+0000 7fa9d7fff640 1 -- 172.21.15.42:0/4003866629 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4019e40 con 0x56293e8b04d0 2024-07-26T06:54:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.962+0000 7faa05fe3740 1 -- 172.21.15.42:0/4003866629 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9e400fc90 con 0x56293e8b04d0 2024-07-26T06:54:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.962+0000 7fa9d7fff640 1 -- 172.21.15.42:0/4003866629 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fa9e400f850 con 0x56293e8b04d0 2024-07-26T06:54:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.962+0000 7fa9d7fff640 1 -- 172.21.15.42:0/4003866629 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa9e400fa50 con 0x56293e8b04d0 2024-07-26T06:54:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.962+0000 7faa05fe3740 10 client.15942 fetch_fsmap learned FSMap version 37 2024-07-26T06:54:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.962+0000 7faa05fe3740 1 -- 172.21.15.42:0/4003866629 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56293e9b5620 con 0x56293e8b04d0 2024-07-26T06:54:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.962+0000 7fa9d7fff640 1 -- 172.21.15.42:0/4003866629 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9e400fc90 con 0x56293e8b04d0 2024-07-26T06:54:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.962+0000 7fa9d7fff640 1 -- 172.21.15.42:0/4003866629 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7fa9e4004b30 con 0x56293e8b04d0 2024-07-26T06:54:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.963+0000 7faa05fe3740 10 client.15942 fetch_fsmap finished waiting for FSMap version 37 2024-07-26T06:54:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.963+0000 7faa05fe3740 10 client.15942 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:54:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.963+0000 7faa05fe3740 20 client.15942 populate_metadata read hostname 'smithi042' 2024-07-26T06:54:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.963+0000 7faa05fe3740 1 --2- 172.21.15.42:0/4003866629 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56293e88fd80 0x56293ec8d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.963+0000 7faa05fe3740 4 client.15942 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:54:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.963+0000 7faa05fe3740 1 -- 172.21.15.42:0/4003866629 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56293ebeb300 con 0x56293e88fd80 2024-07-26T06:54:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.963+0000 7fa9fce5e640 1 --2- 172.21.15.42:0/4003866629 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56293e88fd80 0x56293ec8d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.963+0000 7fa9d7fff640 10 client.15942 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.963+0000 7fa9fce5e640 1 --2- 172.21.15.42:0/4003866629 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56293e88fd80 0x56293ec8d140 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=378dc4631a7a29e6 server_cookie=b7d2b6f1f68981b3 in_seq=0 out_seq=0 2024-07-26T06:54:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.964+0000 7fa9d7fff640 1 -- 172.21.15.42:0/4003866629 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56293ebeb300 con 0x56293e88fd80 2024-07-26T06:54:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.964+0000 7fa9d7fff640 10 client.15942 handle_command_reply: tid=0 2024-07-26T06:54:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.964+0000 7faa05fe3740 1 client.15942 shutdown 2024-07-26T06:54:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.964+0000 7faa05fe3740 1 -- 172.21.15.42:0/4003866629 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 msgr2=0x56293ec80200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.964+0000 7faa05fe3740 1 --2- 172.21.15.42:0/4003866629 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec80200 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4000c00 tx=0x7fa9e4019c50 comp rx=0 tx=0).stop 2024-07-26T06:54:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.964+0000 7faa05fe3740 1 -- 172.21.15.42:0/4003866629 shutdown_connections 2024-07-26T06:54:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.964+0000 7faa05fe3740 1 --2- 172.21.15.42:0/4003866629 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56293e88fd80 0x56293ec8d140 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.964+0000 7faa05fe3740 1 --2- 172.21.15.42:0/4003866629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8d11b0 0x56293ec80d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.964+0000 7fa9d7fff640 0 client.15942 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.964+0000 7faa05fe3740 1 --2- 172.21.15.42:0/4003866629 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec80200 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.964+0000 7faa05fe3740 1 --2- 172.21.15.42:0/4003866629 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8a82f0 0x56293ec7fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.964+0000 7faa05fe3740 1 -- 172.21.15.42:0/4003866629 >> 172.21.15.42:0/4003866629 conn(0x56293e8c5300 msgr2=0x56293ec702b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.964+0000 7faa05fe3740 1 -- 172.21.15.42:0/4003866629 shutdown_connections 2024-07-26T06:54:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.964+0000 7faa05fe3740 1 -- 172.21.15.42:0/4003866629 wait complete. 2024-07-26T06:54:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.965+0000 7faa05fe3740 20 client.15942 trim_cache size 0 max 16384 2024-07-26T06:54:51.985 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.983+0000 7faa05fe3740 1 Processor -- start 2024-07-26T06:54:51.985 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.983+0000 7faa05fe3740 1 -- start start 2024-07-26T06:54:51.985 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.984+0000 7faa05fe3740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8a82f0 0x56293ec80250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.985 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.984+0000 7faa05fe3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8b04d0 0x56293ec87c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.985 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.984+0000 7faa05fe3740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8d11b0 0x56293ec702b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.985 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.984+0000 7faa05fe3740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f81a8830 con 0x56293e8d11b0 2024-07-26T06:54:51.985 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.984+0000 7faa05fe3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f81aa430 con 0x56293e8b04d0 2024-07-26T06:54:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.984+0000 7faa05fe3740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f81a8f00 con 0x56293e8a82f0 2024-07-26T06:54:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.984+0000 7fa9fd65f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8d11b0 0x56293ec702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.984+0000 7fa9fd65f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8d11b0 0x56293ec702b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53270/0 (socket says 172.21.15.42:53270) 2024-07-26T06:54:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.984+0000 7fa9f7fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8b04d0 0x56293ec87c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.984+0000 7fa9fd65f640 1 -- 172.21.15.42:0/448964269 learned_addr learned my addr 172.21.15.42:0/448964269 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.984+0000 7fa9fce5e640 1 --2- 172.21.15.42:0/448964269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8a82f0 0x56293ec80250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.984+0000 7fa9f7fff640 1 -- 172.21.15.42:0/448964269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8a82f0 msgr2=0x56293ec80250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.984+0000 7fa9f7fff640 1 --2- 172.21.15.42:0/448964269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8a82f0 0x56293ec80250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.984+0000 7fa9f7fff640 1 -- 172.21.15.42:0/448964269 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8d11b0 msgr2=0x56293ec702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.984+0000 7fa9f7fff640 1 --2- 172.21.15.42:0/448964269 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8d11b0 0x56293ec702b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.984+0000 7fa9f7fff640 1 -- 172.21.15.42:0/448964269 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56293ec7dcd0 con 0x56293e8b04d0 2024-07-26T06:54:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.984+0000 7fa9fce5e640 1 --2- 172.21.15.42:0/448964269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8a82f0 0x56293ec80250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:54:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.985+0000 7fa9f7fff640 1 --2- 172.21.15.42:0/448964269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8b04d0 0x56293ec87c50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4004ed0 tx=0x7fa9e4005d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f74a07fb5d7fd0fe server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.985+0000 7fa9e27fc640 1 -- 172.21.15.42:0/448964269 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e400fbf0 con 0x56293e8b04d0 2024-07-26T06:54:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.985+0000 7fa9e27fc640 1 -- 172.21.15.42:0/448964269 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9e4027070 con 0x56293e8b04d0 2024-07-26T06:54:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.985+0000 7fa9e27fc640 1 -- 172.21.15.42:0/448964269 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4005450 con 0x56293e8b04d0 2024-07-26T06:54:51.992 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.986+0000 7faa05fe3740 1 -- 172.21.15.42:0/448964269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8b04d0 msgr2=0x56293ec87c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:51.992 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.986+0000 7faa05fe3740 1 --2- 172.21.15.42:0/448964269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8b04d0 0x56293ec87c50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4004ed0 tx=0x7fa9e4005d70 comp rx=0 tx=0).stop 2024-07-26T06:54:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.986+0000 7faa05fe3740 1 -- 172.21.15.42:0/448964269 shutdown_connections 2024-07-26T06:54:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.986+0000 7faa05fe3740 1 --2- 172.21.15.42:0/448964269 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8d11b0 0x56293ec702b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.986+0000 7faa05fe3740 1 --2- 172.21.15.42:0/448964269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8b04d0 0x56293ec87c50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.986+0000 7faa05fe3740 1 --2- 172.21.15.42:0/448964269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8a82f0 0x56293ec80250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.986+0000 7faa05fe3740 1 -- 172.21.15.42:0/448964269 >> 172.21.15.42:0/448964269 conn(0x56293e8c5300 msgr2=0x56293ec7fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.986+0000 7faa05fe3740 1 -- 172.21.15.42:0/448964269 shutdown_connections 2024-07-26T06:54:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.986+0000 7faa05fe3740 1 -- 172.21.15.42:0/448964269 wait complete. 2024-07-26T06:54:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.986+0000 7faa05fe3740 1 -- start start 2024-07-26T06:54:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.986+0000 7faa05fe3740 1 Processor -- start 2024-07-26T06:54:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7faa05fe3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8a82f0 0x56293ec869b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7faa05fe3740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec86ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7faa05fe3740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8d11b0 0x56293ec87430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7faa05fe3740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f81a9d60 con 0x56293e8b04d0 2024-07-26T06:54:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7faa05fe3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f81a95d0 con 0x56293e8a82f0 2024-07-26T06:54:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7faa05fe3740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f81a6810 con 0x56293e8d11b0 2024-07-26T06:54:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7faa05fe3740 10 client.0 _ll_register_callbacks cb 0x7fa9e4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:54:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7fa9f7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec86ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7fa9fce5e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8a82f0 0x56293ec869b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7fa9f7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec86ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53278/0 (socket says 172.21.15.42:53278) 2024-07-26T06:54:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7fa9fd65f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8d11b0 0x56293ec87430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7fa9f7fff640 1 -- 172.21.15.42:0/807877445 learned_addr learned my addr 172.21.15.42:0/807877445 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:54:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7fa9fce5e640 1 -- 172.21.15.42:0/807877445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8d11b0 msgr2=0x56293ec87430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7fa9fce5e640 1 --2- 172.21.15.42:0/807877445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8d11b0 0x56293ec87430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7fa9fce5e640 1 -- 172.21.15.42:0/807877445 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 msgr2=0x56293ec86ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:54:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7fa9fce5e640 1 --2- 172.21.15.42:0/807877445 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec86ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7fa9fce5e640 1 -- 172.21.15.42:0/807877445 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56293ec85690 con 0x56293e8a82f0 2024-07-26T06:54:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.987+0000 7fa9f7fff640 1 --2- 172.21.15.42:0/807877445 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec86ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:54:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.988+0000 7fa9fce5e640 1 --2- 172.21.15.42:0/807877445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8a82f0 0x56293ec869b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec0e1a20 tx=0x7fa9ec0aa830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39d18dc3f21f0543 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:54:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.988+0000 7fa9d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:54:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.988+0000 7fa9d7fff640 1 -- 172.21.15.42:0/807877445 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec0aae20 con 0x56293e8a82f0 2024-07-26T06:54:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.988+0000 7fa9d7fff640 1 -- 172.21.15.42:0/807877445 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fa9ec0037a0 con 0x56293e8a82f0 2024-07-26T06:54:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.988+0000 7faa05fe3740 1 -- 172.21.15.42:0/807877445 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9e4027070 con 0x56293e8a82f0 2024-07-26T06:54:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.988+0000 7fa9d7fff640 1 -- 172.21.15.42:0/807877445 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa9ec0af420 con 0x56293e8a82f0 2024-07-26T06:54:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.988+0000 7faa05fe3740 10 client.25532 fetch_fsmap learned FSMap version 37 2024-07-26T06:54:51.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.988+0000 7faa05fe3740 1 -- 172.21.15.42:0/807877445 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56293ecf3c70 con 0x56293e8a82f0 2024-07-26T06:54:51.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.988+0000 7fa9d7fff640 1 -- 172.21.15.42:0/807877445 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9e4027070 con 0x56293e8a82f0 2024-07-26T06:54:51.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.988+0000 7fa9d7fff640 1 -- 172.21.15.42:0/807877445 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7fa9ec0b3a40 con 0x56293e8a82f0 2024-07-26T06:54:51.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.988+0000 7faa05fe3740 10 client.25532 fetch_fsmap finished waiting for FSMap version 37 2024-07-26T06:54:51.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.988+0000 7faa05fe3740 10 client.25532 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:54:51.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.988+0000 7faa05fe3740 20 client.25532 populate_metadata read hostname 'smithi042' 2024-07-26T06:54:51.997 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.988+0000 7faa05fe3740 1 --2- 172.21.15.42:0/807877445 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56293e8651c0 0x56293ecf66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:54:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.988+0000 7faa05fe3740 4 client.25532 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:54:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.988+0000 7faa05fe3740 1 -- 172.21.15.42:0/807877445 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56293ebe6da0 con 0x56293e8651c0 2024-07-26T06:54:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.989+0000 7fa9f7fff640 1 --2- 172.21.15.42:0/807877445 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56293e8651c0 0x56293ecf66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:54:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.989+0000 7fa9d7fff640 10 client.25532 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:51.989+0000 7fa9f7fff640 1 --2- 172.21.15.42:0/807877445 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56293e8651c0 0x56293ecf66d0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc8135195b88d88b server_cookie=dabaaf7bc1c69eb2 in_seq=0 out_seq=0 2024-07-26T06:54:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.007+0000 7fa9d7fff640 1 -- 172.21.15.42:0/807877445 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56293ebe6da0 con 0x56293e8651c0 2024-07-26T06:54:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.008+0000 7fa9d7fff640 10 client.25532 handle_command_reply: tid=0 2024-07-26T06:54:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.008+0000 7faa05fe3740 1 client.25532 shutdown 2024-07-26T06:54:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.008+0000 7faa05fe3740 1 -- 172.21.15.42:0/807877445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8a82f0 msgr2=0x56293ec869b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.008+0000 7faa05fe3740 1 --2- 172.21.15.42:0/807877445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8a82f0 0x56293ec869b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec0e1a20 tx=0x7fa9ec0aa830 comp rx=0 tx=0).stop 2024-07-26T06:54:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.008+0000 7faa05fe3740 1 -- 172.21.15.42:0/807877445 shutdown_connections 2024-07-26T06:54:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.008+0000 7faa05fe3740 1 --2- 172.21.15.42:0/807877445 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56293e8651c0 0x56293ecf66d0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.008+0000 7faa05fe3740 1 --2- 172.21.15.42:0/807877445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56293e8d11b0 0x56293ec87430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.008+0000 7faa05fe3740 1 --2- 172.21.15.42:0/807877445 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56293e8b04d0 0x56293ec86ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.008+0000 7fa9d7fff640 0 client.25532 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:54:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.008+0000 7faa05fe3740 1 --2- 172.21.15.42:0/807877445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56293e8a82f0 0x56293ec869b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.008+0000 7faa05fe3740 1 -- 172.21.15.42:0/807877445 >> 172.21.15.42:0/807877445 conn(0x56293e8c5300 msgr2=0x56293ec75d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.008+0000 7faa05fe3740 1 -- 172.21.15.42:0/807877445 shutdown_connections 2024-07-26T06:54:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.008+0000 7faa05fe3740 1 -- 172.21.15.42:0/807877445 wait complete. 2024-07-26T06:54:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.008+0000 7faa05fe3740 20 client.25532 trim_cache size 0 max 16384 2024-07-26T06:54:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:54:52.012 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:54:52.012 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:54:52.012 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:54:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.009+0000 7fa9ff0c2640 1 -- 172.21.15.42:0/2087570019 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa9e80b3bd0 msgr2=0x7fa9e80b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.009+0000 7fa9ff0c2640 1 --2- 172.21.15.42:0/2087570019 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa9e80b3bd0 0x7fa9e80b60e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009a60 tx=0x7fa9f002f310 comp rx=0 tx=0).stop 2024-07-26T06:54:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.009+0000 7fa9ff0c2640 1 -- 172.21.15.42:0/2087570019 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9f81014c0 msgr2=0x7fa9f81998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:54:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.009+0000 7fa9ff0c2640 1 --2- 172.21.15.42:0/2087570019 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9f81014c0 0x7fa9f81998c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec00ea60 tx=0x7fa9ec00ef20 comp rx=0 tx=0).stop 2024-07-26T06:54:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.009+0000 7fa9ff0c2640 1 -- 172.21.15.42:0/2087570019 shutdown_connections 2024-07-26T06:54:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.009+0000 7fa9ff0c2640 1 --2- 172.21.15.42:0/2087570019 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa9e80b3bd0 0x7fa9e80b60e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.009+0000 7fa9ff0c2640 1 --2- 172.21.15.42:0/2087570019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9f8108eb0 0x7fa9f819aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.009+0000 7fa9ff0c2640 1 --2- 172.21.15.42:0/2087570019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9f8106550 0x7fa9f8199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.009+0000 7fa9ff0c2640 1 --2- 172.21.15.42:0/2087570019 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9f81014c0 0x7fa9f81998c0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:54:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.009+0000 7fa9ff0c2640 1 -- 172.21.15.42:0/2087570019 >> 172.21.15.42:0/2087570019 conn(0x7fa9f80fd0c0 msgr2=0x7fa9f8107320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:54:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.009+0000 7fa9ff0c2640 1 -- 172.21.15.42:0/2087570019 shutdown_connections 2024-07-26T06:54:52.014 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:54:52.009+0000 7fa9ff0c2640 1 -- 172.21.15.42:0/2087570019 wait complete. 2024-07-26T06:54:53.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:52 smithi042 ceph-mon[32775]: pgmap v536: 97 pgs: 97 active+clean; 63 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 2.7 MiB/s wr, 618 op/s 2024-07-26T06:54:53.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:52 smithi164 ceph-mon[36970]: pgmap v536: 97 pgs: 97 active+clean; 63 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 2.7 MiB/s wr, 618 op/s 2024-07-26T06:54:53.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:52 smithi163 ceph-mon[36873]: pgmap v536: 97 pgs: 97 active+clean; 63 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 2.7 MiB/s wr, 618 op/s 2024-07-26T06:54:55.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:54 smithi042 ceph-mon[32775]: pgmap v537: 97 pgs: 97 active+clean; 68 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.8 MiB/s wr, 745 op/s 2024-07-26T06:54:55.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:54 smithi164 ceph-mon[36970]: pgmap v537: 97 pgs: 97 active+clean; 68 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.8 MiB/s wr, 745 op/s 2024-07-26T06:54:55.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:54 smithi163 ceph-mon[36873]: pgmap v537: 97 pgs: 97 active+clean; 68 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.8 MiB/s wr, 745 op/s 2024-07-26T06:54:55.952 INFO:tasks.workunit.client.0.smithi042.stdout:10000 mapread 0x33cc4e3 thru 0x33cec16 (0x2734 bytes) 2024-07-26T06:54:57.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:56 smithi042 ceph-mon[32775]: pgmap v538: 97 pgs: 97 active+clean; 64 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.6 MiB/s wr, 812 op/s 2024-07-26T06:54:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:56 smithi164 ceph-mon[36970]: pgmap v538: 97 pgs: 97 active+clean; 64 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.6 MiB/s wr, 812 op/s 2024-07-26T06:54:57.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:56 smithi163 ceph-mon[36873]: pgmap v538: 97 pgs: 97 active+clean; 64 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.6 MiB/s wr, 812 op/s 2024-07-26T06:54:59.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:54:58 smithi042 ceph-mon[32775]: pgmap v539: 97 pgs: 97 active+clean; 68 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 2.6 MiB/s wr, 937 op/s 2024-07-26T06:54:59.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:54:58 smithi164 ceph-mon[36970]: pgmap v539: 97 pgs: 97 active+clean; 68 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 2.6 MiB/s wr, 937 op/s 2024-07-26T06:54:59.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:54:58 smithi163 ceph-mon[36873]: pgmap v539: 97 pgs: 97 active+clean; 68 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 2.6 MiB/s wr, 937 op/s 2024-07-26T06:55:01.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:00 smithi042 ceph-mon[32775]: pgmap v540: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 3.0 MiB/s wr, 1.12k op/s 2024-07-26T06:55:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:00 smithi164 ceph-mon[36970]: pgmap v540: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 3.0 MiB/s wr, 1.12k op/s 2024-07-26T06:55:01.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:00 smithi163 ceph-mon[36873]: pgmap v540: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 3.0 MiB/s wr, 1.12k op/s 2024-07-26T06:55:02.463 INFO:tasks.workunit.client.0.smithi042.stdout:truncating to largest ever: 0x63e5d2a 2024-07-26T06:55:03.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:02 smithi042 ceph-mon[32775]: pgmap v541: 97 pgs: 97 active+clean; 71 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 2.6 MiB/s wr, 1.03k op/s 2024-07-26T06:55:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:02 smithi164 ceph-mon[36970]: pgmap v541: 97 pgs: 97 active+clean; 71 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 2.6 MiB/s wr, 1.03k op/s 2024-07-26T06:55:03.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:02 smithi163 ceph-mon[36873]: pgmap v541: 97 pgs: 97 active+clean; 71 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 2.6 MiB/s wr, 1.03k op/s 2024-07-26T06:55:05.036 INFO:tasks.workunit.client.0.smithi042.stdout:truncating to largest ever: 0x63e9b52 2024-07-26T06:55:05.133 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.132+0000 7fee79941640 1 -- 172.21.15.42:0/3844373179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee7406b140 msgr2=0x7fee74105d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:05.133 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.132+0000 7fee79941640 1 --2- 172.21.15.42:0/3844373179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee7406b140 0x7fee74105d10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fee5c009900 tx=0x7fee5c02a410 comp rx=0 tx=0).stop 2024-07-26T06:55:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.132+0000 7fee79941640 1 -- 172.21.15.42:0/3844373179 shutdown_connections 2024-07-26T06:55:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.132+0000 7fee79941640 1 --2- 172.21.15.42:0/3844373179 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee74108e10 0x7fee7410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.132+0000 7fee79941640 1 --2- 172.21.15.42:0/3844373179 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee74106250 0x7fee741086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.132+0000 7fee79941640 1 --2- 172.21.15.42:0/3844373179 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee7406b140 0x7fee74105d10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.132+0000 7fee79941640 1 -- 172.21.15.42:0/3844373179 >> 172.21.15.42:0/3844373179 conn(0x7fee740fd910 msgr2=0x7fee740ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.132+0000 7fee79941640 1 -- 172.21.15.42:0/3844373179 shutdown_connections 2024-07-26T06:55:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.132+0000 7fee79941640 1 -- 172.21.15.42:0/3844373179 wait complete. 2024-07-26T06:55:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.133+0000 7fee79941640 1 Processor -- start 2024-07-26T06:55:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.133+0000 7fee79941640 1 -- start start 2024-07-26T06:55:05.136 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.133+0000 7fee79941640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee7406b140 0x7fee741999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.136 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.133+0000 7fee79941640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee74106250 0x7fee7419a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.136 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.133+0000 7fee79941640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee74108e10 0x7fee7419abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.136 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.133+0000 7fee79941640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fee7410ed80 con 0x7fee7406b140 2024-07-26T06:55:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.133+0000 7fee79941640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fee7410ec00 con 0x7fee74106250 2024-07-26T06:55:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.133+0000 7fee79941640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fee7410ef00 con 0x7fee74108e10 2024-07-26T06:55:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.133+0000 7fee72ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee7406b140 0x7fee741999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.134+0000 7fee727fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee74106250 0x7fee7419a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.134+0000 7fee72ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee7406b140 0x7fee741999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57094/0 (socket says 172.21.15.42:57094) 2024-07-26T06:55:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.134+0000 7fee72ffd640 1 -- 172.21.15.42:0/48548624 learned_addr learned my addr 172.21.15.42:0/48548624 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.134+0000 7fee737fe640 1 --2- 172.21.15.42:0/48548624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee74108e10 0x7fee7419abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.134+0000 7fee727fc640 1 -- 172.21.15.42:0/48548624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee74108e10 msgr2=0x7fee7419abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.134+0000 7fee727fc640 1 --2- 172.21.15.42:0/48548624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee74108e10 0x7fee7419abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.134+0000 7fee727fc640 1 -- 172.21.15.42:0/48548624 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee7406b140 msgr2=0x7fee741999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.134+0000 7fee727fc640 1 --2- 172.21.15.42:0/48548624 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee7406b140 0x7fee741999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.134+0000 7fee727fc640 1 -- 172.21.15.42:0/48548624 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee7419b3f0 con 0x7fee74106250 2024-07-26T06:55:05.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.134+0000 7fee72ffd640 1 --2- 172.21.15.42:0/48548624 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee7406b140 0x7fee741999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:55:05.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.134+0000 7fee727fc640 1 --2- 172.21.15.42:0/48548624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee74106250 0x7fee7419a020 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fee6400e9c0 tx=0x7fee6400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:05.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.134+0000 7fee7893f640 1 -- 172.21.15.42:0/48548624 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6400ccc0 con 0x7fee74106250 2024-07-26T06:55:05.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.134+0000 7fee79941640 1 -- 172.21.15.42:0/48548624 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee7419b6e0 con 0x7fee74106250 2024-07-26T06:55:05.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.134+0000 7fee7893f640 1 -- 172.21.15.42:0/48548624 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee640108e0 con 0x7fee74106250 2024-07-26T06:55:05.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.134+0000 7fee7893f640 1 -- 172.21.15.42:0/48548624 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee640187a0 con 0x7fee74106250 2024-07-26T06:55:05.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.134+0000 7fee79941640 1 -- 172.21.15.42:0/48548624 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee741a1dd0 con 0x7fee74106250 2024-07-26T06:55:05.143 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.137+0000 7fee7893f640 1 -- 172.21.15.42:0/48548624 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fee64018940 con 0x7fee74106250 2024-07-26T06:55:05.143 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.137+0000 7fee80856740 1 Processor -- start 2024-07-26T06:55:05.143 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.137+0000 7fee80856740 1 -- start start 2024-07-26T06:55:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee80856740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624648241b0 0x562464bc8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee80856740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624647fb2f0 0x562464bc91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee80856740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624648034d0 0x562464bc9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee80856740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x562464b3c3b0 con 0x5624648241b0 2024-07-26T06:55:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee80856740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562464b2b8b0 con 0x5624648034d0 2024-07-26T06:55:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee80856740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x562464b2f8f0 con 0x5624647fb2f0 2024-07-26T06:55:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee737fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624648241b0 0x562464bc8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee737fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624648241b0 0x562464bc8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57098/0 (socket says 172.21.15.42:57098) 2024-07-26T06:55:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee737fe640 1 -- 172.21.15.42:0/3333437453 learned_addr learned my addr 172.21.15.42:0/3333437453 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee727fc640 1 --2- 172.21.15.42:0/3333437453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624648034d0 0x562464bc9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee72ffd640 1 --2- 172.21.15.42:0/3333437453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624647fb2f0 0x562464bc91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee7893f640 1 --2- 172.21.15.42:0/48548624 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fee600b3bd0 0x7fee600b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee737fe640 1 -- 172.21.15.42:0/3333437453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624647fb2f0 msgr2=0x562464bc91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee737fe640 1 --2- 172.21.15.42:0/3333437453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624647fb2f0 0x562464bc91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee737fe640 1 -- 172.21.15.42:0/3333437453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624648034d0 msgr2=0x562464bc9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee737fe640 1 --2- 172.21.15.42:0/3333437453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624648034d0 0x562464bc9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee737fe640 1 -- 172.21.15.42:0/3333437453 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562464ad29a0 con 0x5624648241b0 2024-07-26T06:55:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee7893f640 1 -- 172.21.15.42:0/48548624 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fee640e3fe0 con 0x7fee74106250 2024-07-26T06:55:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.138+0000 7fee737fe640 1 --2- 172.21.15.42:0/48548624 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fee600b3bd0 0x7fee600b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.139+0000 7fee737fe640 1 --2- 172.21.15.42:0/3333437453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624648241b0 0x562464bc8b50 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fee6c011590 tx=0x7fee6c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7170377336d19cd8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.139+0000 7fee69ffb640 1 -- 172.21.15.42:0/3333437453 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c017050 con 0x5624648241b0 2024-07-26T06:55:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.139+0000 7fee69ffb640 1 -- 172.21.15.42:0/3333437453 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee6c01bc50 con 0x5624648241b0 2024-07-26T06:55:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.139+0000 7fee69ffb640 1 -- 172.21.15.42:0/3333437453 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c00e3b0 con 0x5624648241b0 2024-07-26T06:55:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.140+0000 7fee80856740 1 -- 172.21.15.42:0/3333437453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624648241b0 msgr2=0x562464bc8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.140+0000 7fee80856740 1 --2- 172.21.15.42:0/3333437453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624648241b0 0x562464bc8b50 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fee6c011590 tx=0x7fee6c011a50 comp rx=0 tx=0).stop 2024-07-26T06:55:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.140+0000 7fee737fe640 1 --2- 172.21.15.42:0/48548624 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fee600b3bd0 0x7fee600b60e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fee6c022000 tx=0x7fee6c023000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.140+0000 7fee80856740 1 -- 172.21.15.42:0/3333437453 shutdown_connections 2024-07-26T06:55:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.140+0000 7fee80856740 1 --2- 172.21.15.42:0/3333437453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624648034d0 0x562464bc9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.140+0000 7fee80856740 1 --2- 172.21.15.42:0/3333437453 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624647fb2f0 0x562464bc91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.140+0000 7fee80856740 1 --2- 172.21.15.42:0/3333437453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624648241b0 0x562464bc8b50 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.140+0000 7fee80856740 1 -- 172.21.15.42:0/3333437453 >> 172.21.15.42:0/3333437453 conn(0x562464818300 msgr2=0x562464bc45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.140+0000 7fee80856740 1 -- 172.21.15.42:0/3333437453 shutdown_connections 2024-07-26T06:55:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.140+0000 7fee80856740 1 -- 172.21.15.42:0/3333437453 wait complete. 2024-07-26T06:55:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.140+0000 7fee80856740 1 -- start start 2024-07-26T06:55:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.141+0000 7fee80856740 1 Processor -- start 2024-07-26T06:55:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.141+0000 7fee80856740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624647fb2f0 0x562464bd24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.141+0000 7fee80856740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624648034d0 0x562464bd2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.141+0000 7fee80856740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624648241b0 0x562464bd3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.141+0000 7fee72ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624647fb2f0 0x562464bd24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.141+0000 7fee72ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624647fb2f0 0x562464bd24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57100/0 (socket says 172.21.15.42:57100) 2024-07-26T06:55:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.141+0000 7fee72ffd640 1 -- 172.21.15.42:0/3010382694 learned_addr learned my addr 172.21.15.42:0/3010382694 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.141+0000 7fee80856740 1 -- 172.21.15.42:0/3010382694 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x562464a65ab0 con 0x5624647fb2f0 2024-07-26T06:55:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.141+0000 7fee80856740 1 -- 172.21.15.42:0/3010382694 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562464ab3cf0 con 0x5624648241b0 2024-07-26T06:55:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.141+0000 7fee80856740 1 -- 172.21.15.42:0/3010382694 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x562464bca830 con 0x5624648034d0 2024-07-26T06:55:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.141+0000 7fee737fe640 1 --2- 172.21.15.42:0/3010382694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624648241b0 0x562464bd3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.141+0000 7fee80856740 10 client.0 _ll_register_callbacks cb 0x562464b253e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:55:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.141+0000 7fee727fc640 1 --2- 172.21.15.42:0/3010382694 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624648034d0 0x562464bd2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.142+0000 7fee72ffd640 1 -- 172.21.15.42:0/3010382694 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624648034d0 msgr2=0x562464bd2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.142+0000 7fee72ffd640 1 --2- 172.21.15.42:0/3010382694 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624648034d0 0x562464bd2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.142+0000 7fee72ffd640 1 -- 172.21.15.42:0/3010382694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624648241b0 msgr2=0x562464bd3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.142+0000 7fee72ffd640 1 --2- 172.21.15.42:0/3010382694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624648241b0 0x562464bd3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.142+0000 7fee72ffd640 1 -- 172.21.15.42:0/3010382694 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56246488f790 con 0x5624647fb2f0 2024-07-26T06:55:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.142+0000 7fee72ffd640 1 --2- 172.21.15.42:0/3010382694 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624647fb2f0 0x562464bd24f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fee5c003820 tx=0x7fee5c003850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d56fd7c5ede581 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.142+0000 7fee52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:55:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.142+0000 7fee52ffd640 1 -- 172.21.15.42:0/3010382694 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c035070 con 0x5624647fb2f0 2024-07-26T06:55:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.142+0000 7fee52ffd640 1 -- 172.21.15.42:0/3010382694 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fee5c003c00 con 0x5624647fb2f0 2024-07-26T06:55:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.142+0000 7fee80856740 1 -- 172.21.15.42:0/3010382694 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee6c01bc50 con 0x5624647fb2f0 2024-07-26T06:55:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.142+0000 7fee52ffd640 1 -- 172.21.15.42:0/3010382694 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fee5c04c070 con 0x5624647fb2f0 2024-07-26T06:55:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.142+0000 7fee80856740 10 client.15960 fetch_fsmap learned FSMap version 37 2024-07-26T06:55:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.142+0000 7fee80856740 1 -- 172.21.15.42:0/3010382694 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562464908620 con 0x5624647fb2f0 2024-07-26T06:55:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.143+0000 7fee52ffd640 1 -- 172.21.15.42:0/3010382694 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee6c01bc50 con 0x5624647fb2f0 2024-07-26T06:55:05.149 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.143+0000 7fee52ffd640 1 -- 172.21.15.42:0/3010382694 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7fee5c046460 con 0x5624647fb2f0 2024-07-26T06:55:05.149 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.144+0000 7fee80856740 10 client.15960 fetch_fsmap finished waiting for FSMap version 37 2024-07-26T06:55:05.150 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.144+0000 7fee80856740 10 client.15960 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:55:05.150 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.144+0000 7fee80856740 20 client.15960 populate_metadata read hostname 'smithi042' 2024-07-26T06:55:05.150 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.144+0000 7fee80856740 1 --2- 172.21.15.42:0/3010382694 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5624647e2d80 0x562464be01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.150 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.144+0000 7fee80856740 4 client.15960 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:55:05.150 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.144+0000 7fee80856740 1 -- 172.21.15.42:0/3010382694 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562464b3e300 con 0x5624647e2d80 2024-07-26T06:55:05.150 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:04 smithi042 ceph-mon[32775]: pgmap v542: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 2.6 MiB/s wr, 1.04k op/s 2024-07-26T06:55:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.145+0000 7fee727fc640 1 --2- 172.21.15.42:0/3010382694 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5624647e2d80 0x562464be01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.145+0000 7fee52ffd640 10 client.15960 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.145+0000 7fee727fc640 1 --2- 172.21.15.42:0/3010382694 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5624647e2d80 0x562464be01b0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c108be1145f3f50 server_cookie=50a6523cec2a55f5 in_seq=0 out_seq=0 2024-07-26T06:55:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.147+0000 7fee52ffd640 1 -- 172.21.15.42:0/3010382694 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562464b3e300 con 0x5624647e2d80 2024-07-26T06:55:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.147+0000 7fee52ffd640 10 client.15960 handle_command_reply: tid=0 2024-07-26T06:55:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.147+0000 7fee80856740 1 client.15960 shutdown 2024-07-26T06:55:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.147+0000 7fee80856740 1 -- 172.21.15.42:0/3010382694 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624647fb2f0 msgr2=0x562464bd24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.147+0000 7fee80856740 1 --2- 172.21.15.42:0/3010382694 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624647fb2f0 0x562464bd24f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fee5c003820 tx=0x7fee5c003850 comp rx=0 tx=0).stop 2024-07-26T06:55:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.147+0000 7fee80856740 1 -- 172.21.15.42:0/3010382694 shutdown_connections 2024-07-26T06:55:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.147+0000 7fee80856740 1 --2- 172.21.15.42:0/3010382694 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5624647e2d80 0x562464be01b0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.147+0000 7fee80856740 1 --2- 172.21.15.42:0/3010382694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624648241b0 0x562464bd3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.147+0000 7fee52ffd640 0 client.15960 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.147+0000 7fee80856740 1 --2- 172.21.15.42:0/3010382694 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624648034d0 0x562464bd2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.147+0000 7fee80856740 1 --2- 172.21.15.42:0/3010382694 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624647fb2f0 0x562464bd24f0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.147+0000 7fee80856740 1 -- 172.21.15.42:0/3010382694 >> 172.21.15.42:0/3010382694 conn(0x562464818300 msgr2=0x562464bc2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.148+0000 7fee80856740 1 -- 172.21.15.42:0/3010382694 shutdown_connections 2024-07-26T06:55:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.148+0000 7fee80856740 1 -- 172.21.15.42:0/3010382694 wait complete. 2024-07-26T06:55:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.148+0000 7fee80856740 20 client.15960 trim_cache size 0 max 16384 2024-07-26T06:55:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.168+0000 7fee80856740 1 Processor -- start 2024-07-26T06:55:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.168+0000 7fee80856740 1 -- start start 2024-07-26T06:55:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.168+0000 7fee80856740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624647e2d80 0x562464bbf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.168+0000 7fee80856740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624647fb2f0 0x562464bd1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.168+0000 7fee80856740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624648034d0 0x562464bcd530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.168+0000 7fee80856740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fee741aaa30 con 0x5624647fb2f0 2024-07-26T06:55:05.170 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.168+0000 7fee80856740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fee741ac630 con 0x5624648034d0 2024-07-26T06:55:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.168+0000 7fee80856740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fee741ab100 con 0x5624647e2d80 2024-07-26T06:55:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.168+0000 7fee727fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624647fb2f0 0x562464bd1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.168+0000 7fee727fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624647fb2f0 0x562464bd1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57106/0 (socket says 172.21.15.42:57106) 2024-07-26T06:55:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.168+0000 7fee727fc640 1 -- 172.21.15.42:0/1829114076 learned_addr learned my addr 172.21.15.42:0/1829114076 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.168+0000 7fee72ffd640 1 --2- 172.21.15.42:0/1829114076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624647e2d80 0x562464bbf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.168+0000 7fee737fe640 1 --2- 172.21.15.42:0/1829114076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624648034d0 0x562464bcd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.169+0000 7fee727fc640 1 -- 172.21.15.42:0/1829114076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624647e2d80 msgr2=0x562464bbf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.169+0000 7fee727fc640 1 --2- 172.21.15.42:0/1829114076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624647e2d80 0x562464bbf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.169+0000 7fee727fc640 1 -- 172.21.15.42:0/1829114076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624648034d0 msgr2=0x562464bcd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.169+0000 7fee727fc640 1 --2- 172.21.15.42:0/1829114076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624648034d0 0x562464bcd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.169+0000 7fee727fc640 1 -- 172.21.15.42:0/1829114076 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562464bbe9b0 con 0x5624647fb2f0 2024-07-26T06:55:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.169+0000 7fee72ffd640 1 --2- 172.21.15.42:0/1829114076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624647e2d80 0x562464bbf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:55:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.169+0000 7fee727fc640 1 --2- 172.21.15.42:0/1829114076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624647fb2f0 0x562464bd1710 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fee640a7040 tx=0x7fee640e7ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39984707925e1d0b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.169+0000 7fee697fa640 1 -- 172.21.15.42:0/1829114076 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee640aaad0 con 0x5624647fb2f0 2024-07-26T06:55:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.169+0000 7fee697fa640 1 -- 172.21.15.42:0/1829114076 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee640aafc0 con 0x5624647fb2f0 2024-07-26T06:55:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.169+0000 7fee697fa640 1 -- 172.21.15.42:0/1829114076 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee64010a90 con 0x5624647fb2f0 2024-07-26T06:55:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.170+0000 7fee80856740 1 -- 172.21.15.42:0/1829114076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624647fb2f0 msgr2=0x562464bd1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.170+0000 7fee80856740 1 --2- 172.21.15.42:0/1829114076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624647fb2f0 0x562464bd1710 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fee640a7040 tx=0x7fee640e7ac0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.170+0000 7fee80856740 1 -- 172.21.15.42:0/1829114076 shutdown_connections 2024-07-26T06:55:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.170+0000 7fee80856740 1 --2- 172.21.15.42:0/1829114076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624648034d0 0x562464bcd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.170+0000 7fee80856740 1 --2- 172.21.15.42:0/1829114076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624647fb2f0 0x562464bd1710 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.170+0000 7fee80856740 1 --2- 172.21.15.42:0/1829114076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624647e2d80 0x562464bbf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.170+0000 7fee80856740 1 -- 172.21.15.42:0/1829114076 >> 172.21.15.42:0/1829114076 conn(0x562464818300 msgr2=0x562464bc9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.170+0000 7fee80856740 1 -- 172.21.15.42:0/1829114076 shutdown_connections 2024-07-26T06:55:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.171+0000 7fee80856740 1 -- 172.21.15.42:0/1829114076 wait complete. 2024-07-26T06:55:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.171+0000 7fee80856740 1 -- start start 2024-07-26T06:55:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.171+0000 7fee80856740 1 Processor -- start 2024-07-26T06:55:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee80856740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624647e2d80 0x562464c40a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee80856740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624647fb2f0 0x562464c43100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee80856740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624648034d0 0x562464c45820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee80856740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fee741abf60 con 0x5624648034d0 2024-07-26T06:55:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee80856740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fee741ab7d0 con 0x5624647e2d80 2024-07-26T06:55:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee80856740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fee741a8a10 con 0x5624647fb2f0 2024-07-26T06:55:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee80856740 10 client.0 _ll_register_callbacks cb 0x7fee5c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:55:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee737fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624648034d0 0x562464c45820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee72ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624647e2d80 0x562464c40a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee737fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624648034d0 0x562464c45820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57108/0 (socket says 172.21.15.42:57108) 2024-07-26T06:55:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee737fe640 1 -- 172.21.15.42:0/2467407135 learned_addr learned my addr 172.21.15.42:0/2467407135 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee727fc640 1 --2- 172.21.15.42:0/2467407135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624647fb2f0 0x562464c43100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee72ffd640 1 -- 172.21.15.42:0/2467407135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624647fb2f0 msgr2=0x562464c43100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee72ffd640 1 --2- 172.21.15.42:0/2467407135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624647fb2f0 0x562464c43100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee72ffd640 1 -- 172.21.15.42:0/2467407135 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624648034d0 msgr2=0x562464c45820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee72ffd640 1 --2- 172.21.15.42:0/2467407135 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624648034d0 0x562464c45820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee72ffd640 1 -- 172.21.15.42:0/2467407135 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562464c45fb0 con 0x5624647e2d80 2024-07-26T06:55:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee737fe640 1 --2- 172.21.15.42:0/2467407135 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624648034d0 0x562464c45820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:55:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee72ffd640 1 --2- 172.21.15.42:0/2467407135 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624647e2d80 0x562464c40a30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fee5c004650 tx=0x7fee5c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff6d1d1aab7312f2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.172+0000 7fee52ffd640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:55:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.173+0000 7fee52ffd640 1 -- 172.21.15.42:0/2467407135 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c056060 con 0x5624647e2d80 2024-07-26T06:55:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.173+0000 7fee52ffd640 1 -- 172.21.15.42:0/2467407135 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fee5c046cb0 con 0x5624647e2d80 2024-07-26T06:55:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.173+0000 7fee80856740 1 -- 172.21.15.42:0/2467407135 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee640aafc0 con 0x5624647e2d80 2024-07-26T06:55:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.173+0000 7fee52ffd640 1 -- 172.21.15.42:0/2467407135 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fee5c035070 con 0x5624647e2d80 2024-07-26T06:55:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.173+0000 7fee80856740 10 client.25550 fetch_fsmap learned FSMap version 37 2024-07-26T06:55:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.173+0000 7fee80856740 1 -- 172.21.15.42:0/2467407135 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562464c472b0 con 0x5624647e2d80 2024-07-26T06:55:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.173+0000 7fee52ffd640 1 -- 172.21.15.42:0/2467407135 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee640aafc0 con 0x5624647e2d80 2024-07-26T06:55:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.173+0000 7fee52ffd640 1 -- 172.21.15.42:0/2467407135 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7fee5c031c10 con 0x5624647e2d80 2024-07-26T06:55:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.173+0000 7fee80856740 10 client.25550 fetch_fsmap finished waiting for FSMap version 37 2024-07-26T06:55:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.173+0000 7fee80856740 10 client.25550 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:55:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.173+0000 7fee80856740 20 client.25550 populate_metadata read hostname 'smithi042' 2024-07-26T06:55:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.173+0000 7fee80856740 1 --2- 172.21.15.42:0/2467407135 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5624648241b0 0x562464c49d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.173+0000 7fee80856740 4 client.25550 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:55:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.173+0000 7fee80856740 1 -- 172.21.15.42:0/2467407135 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562464b39da0 con 0x5624648241b0 2024-07-26T06:55:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.174+0000 7fee727fc640 1 --2- 172.21.15.42:0/2467407135 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5624648241b0 0x562464c49d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.174+0000 7fee52ffd640 10 client.25550 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.174+0000 7fee727fc640 1 --2- 172.21.15.42:0/2467407135 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5624648241b0 0x562464c49d30 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=504420c01075106e server_cookie=443783c6207bb5ce in_seq=0 out_seq=0 2024-07-26T06:55:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.199+0000 7fee52ffd640 1 -- 172.21.15.42:0/2467407135 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562464b39da0 con 0x5624648241b0 2024-07-26T06:55:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.199+0000 7fee52ffd640 10 client.25550 handle_command_reply: tid=0 2024-07-26T06:55:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.199+0000 7fee80856740 1 client.25550 shutdown 2024-07-26T06:55:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.200+0000 7fee80856740 1 -- 172.21.15.42:0/2467407135 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624647e2d80 msgr2=0x562464c40a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.200+0000 7fee80856740 1 --2- 172.21.15.42:0/2467407135 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624647e2d80 0x562464c40a30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fee5c004650 tx=0x7fee5c009260 comp rx=0 tx=0).stop 2024-07-26T06:55:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.200+0000 7fee80856740 1 -- 172.21.15.42:0/2467407135 shutdown_connections 2024-07-26T06:55:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.200+0000 7fee80856740 1 --2- 172.21.15.42:0/2467407135 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5624648241b0 0x562464c49d30 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.200+0000 7fee80856740 1 --2- 172.21.15.42:0/2467407135 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5624648034d0 0x562464c45820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.200+0000 7fee80856740 1 --2- 172.21.15.42:0/2467407135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5624647fb2f0 0x562464c43100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.201 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.200+0000 7fee52ffd640 0 client.25550 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.200+0000 7fee80856740 1 --2- 172.21.15.42:0/2467407135 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5624647e2d80 0x562464c40a30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.200+0000 7fee80856740 1 -- 172.21.15.42:0/2467407135 >> 172.21.15.42:0/2467407135 conn(0x562464818300 msgr2=0x562464bc9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.200+0000 7fee80856740 1 -- 172.21.15.42:0/2467407135 shutdown_connections 2024-07-26T06:55:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.200+0000 7fee80856740 1 -- 172.21.15.42:0/2467407135 wait complete. 2024-07-26T06:55:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.200+0000 7fee80856740 20 client.25550 trim_cache size 0 max 16384 2024-07-26T06:55:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:55:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:55:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:55:05.202 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:55:05.203 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.200+0000 7fee79941640 1 -- 172.21.15.42:0/48548624 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fee600b3bd0 msgr2=0x7fee600b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:05.203 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.201+0000 7fee79941640 1 --2- 172.21.15.42:0/48548624 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fee600b3bd0 0x7fee600b60e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fee6c022000 tx=0x7fee6c023000 comp rx=0 tx=0).stop 2024-07-26T06:55:05.203 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.201+0000 7fee79941640 1 -- 172.21.15.42:0/48548624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee74106250 msgr2=0x7fee7419a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:05.203 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.201+0000 7fee79941640 1 --2- 172.21.15.42:0/48548624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee74106250 0x7fee7419a020 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fee6400e9c0 tx=0x7fee6400ee80 comp rx=0 tx=0).stop 2024-07-26T06:55:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.201+0000 7fee79941640 1 -- 172.21.15.42:0/48548624 shutdown_connections 2024-07-26T06:55:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.201+0000 7fee79941640 1 --2- 172.21.15.42:0/48548624 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fee600b3bd0 0x7fee600b60e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.201+0000 7fee79941640 1 --2- 172.21.15.42:0/48548624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee74108e10 0x7fee7419abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.201+0000 7fee79941640 1 --2- 172.21.15.42:0/48548624 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee74106250 0x7fee7419a020 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.201+0000 7fee79941640 1 --2- 172.21.15.42:0/48548624 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fee7406b140 0x7fee741999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.201+0000 7fee79941640 1 -- 172.21.15.42:0/48548624 >> 172.21.15.42:0/48548624 conn(0x7fee740fd910 msgr2=0x7fee741047a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.201+0000 7fee79941640 1 -- 172.21.15.42:0/48548624 shutdown_connections 2024-07-26T06:55:05.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:05.201+0000 7fee79941640 1 -- 172.21.15.42:0/48548624 wait complete. 2024-07-26T06:55:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:04 smithi164 ceph-mon[36970]: pgmap v542: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 2.6 MiB/s wr, 1.04k op/s 2024-07-26T06:55:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:04 smithi163 ceph-mon[36873]: pgmap v542: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 2.6 MiB/s wr, 1.04k op/s 2024-07-26T06:55:07.033 INFO:tasks.workunit.client.0.smithi042.stdout:20000 read 0xa6cbf1 thru 0xa6e358 (0x1768 bytes) 2024-07-26T06:55:07.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:06 smithi042 ceph-mon[32775]: pgmap v543: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 2.9 MiB/s wr, 1.12k op/s 2024-07-26T06:55:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:06 smithi164 ceph-mon[36970]: pgmap v543: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 2.9 MiB/s wr, 1.12k op/s 2024-07-26T06:55:07.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:06 smithi163 ceph-mon[36873]: pgmap v543: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 2.9 MiB/s wr, 1.12k op/s 2024-07-26T06:55:08.933 INFO:tasks.workunit.client.0.smithi042.stdout:copying to largest ever: 0x63ed6b9 2024-07-26T06:55:09.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:08 smithi042 ceph-mon[32775]: pgmap v544: 97 pgs: 97 active+clean; 71 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 2.8 MiB/s wr, 1.05k op/s 2024-07-26T06:55:09.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:08 smithi164 ceph-mon[36970]: pgmap v544: 97 pgs: 97 active+clean; 71 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 2.8 MiB/s wr, 1.05k op/s 2024-07-26T06:55:09.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:08 smithi163 ceph-mon[36873]: pgmap v544: 97 pgs: 97 active+clean; 71 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 2.8 MiB/s wr, 1.05k op/s 2024-07-26T06:55:11.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:10 smithi042 ceph-mon[32775]: pgmap v545: 97 pgs: 97 active+clean; 68 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 2.9 MiB/s wr, 1.12k op/s 2024-07-26T06:55:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:10 smithi164 ceph-mon[36970]: pgmap v545: 97 pgs: 97 active+clean; 68 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 2.9 MiB/s wr, 1.12k op/s 2024-07-26T06:55:11.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:10 smithi163 ceph-mon[36873]: pgmap v545: 97 pgs: 97 active+clean; 68 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 2.9 MiB/s wr, 1.12k op/s 2024-07-26T06:55:12.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:55:12.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:55:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:55:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:55:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:55:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:55:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:55:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:55:12.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:55:12.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:55:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:55:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:55:13.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:12 smithi042 ceph-mon[32775]: pgmap v546: 97 pgs: 97 active+clean; 71 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 2.4 MiB/s wr, 919 op/s 2024-07-26T06:55:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:12 smithi164 ceph-mon[36970]: pgmap v546: 97 pgs: 97 active+clean; 71 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 2.4 MiB/s wr, 919 op/s 2024-07-26T06:55:13.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:12 smithi163 ceph-mon[36873]: pgmap v546: 97 pgs: 97 active+clean; 71 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 2.4 MiB/s wr, 919 op/s 2024-07-26T06:55:15.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:14 smithi042 ceph-mon[32775]: pgmap v547: 97 pgs: 97 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 2.5 MiB/s wr, 945 op/s 2024-07-26T06:55:15.221 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:14 smithi164 ceph-mon[36970]: pgmap v547: 97 pgs: 97 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 2.5 MiB/s wr, 945 op/s 2024-07-26T06:55:15.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:14 smithi163 ceph-mon[36873]: pgmap v547: 97 pgs: 97 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 2.5 MiB/s wr, 945 op/s 2024-07-26T06:55:16.893 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-07-26T06:55:16.894 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:55:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:16 smithi042 ceph-mon[32775]: pgmap v548: 97 pgs: 97 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 2.8 MiB/s wr, 1.04k op/s 2024-07-26T06:55:17.201 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:55:17.221 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:16 smithi164 ceph-mon[36970]: pgmap v548: 97 pgs: 97 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 2.8 MiB/s wr, 1.04k op/s 2024-07-26T06:55:17.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:16 smithi163 ceph-mon[36873]: pgmap v548: 97 pgs: 97 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 2.8 MiB/s wr, 1.04k op/s 2024-07-26T06:55:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.324+0000 7f1bf6f35640 1 -- 172.21.15.42:0/2080604955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf01039f0 msgr2=0x7f1bf010a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.324+0000 7f1bf6f35640 1 --2- 172.21.15.42:0/2080604955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf01039f0 0x7f1bf010a390 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009c90 tx=0x7f1be802a860 comp rx=0 tx=0).stop 2024-07-26T06:55:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.324+0000 7f1bf6f35640 1 -- 172.21.15.42:0/2080604955 shutdown_connections 2024-07-26T06:55:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.324+0000 7f1bf6f35640 1 --2- 172.21.15.42:0/2080604955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf01039f0 0x7f1bf010a390 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.324+0000 7f1bf6f35640 1 --2- 172.21.15.42:0/2080604955 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1bf0103050 0x7f1bf0103460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.324+0000 7f1bf6f35640 1 --2- 172.21.15.42:0/2080604955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bf0101fd0 0x7f1bf0102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.324+0000 7f1bf6f35640 1 -- 172.21.15.42:0/2080604955 >> 172.21.15.42:0/2080604955 conn(0x7f1bf00fd910 msgr2=0x7f1bf00ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.324+0000 7f1bf6f35640 1 -- 172.21.15.42:0/2080604955 shutdown_connections 2024-07-26T06:55:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.324+0000 7f1bf6f35640 1 -- 172.21.15.42:0/2080604955 wait complete. 2024-07-26T06:55:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.324+0000 7f1bf6f35640 1 Processor -- start 2024-07-26T06:55:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.324+0000 7f1bf6f35640 1 -- start start 2024-07-26T06:55:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.325+0000 7f1bf6f35640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf0101fd0 0x7f1bf0198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.325+0000 7f1bf6f35640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1bf0103050 0x7f1bf0198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.325+0000 7f1bf6f35640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bf01039f0 0x7f1bf0199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.325+0000 7f1bf6f35640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf010dca0 con 0x7f1bf0101fd0 2024-07-26T06:55:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.325+0000 7f1bf6f35640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf010db20 con 0x7f1bf01039f0 2024-07-26T06:55:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.325+0000 7f1bf6f35640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf010de20 con 0x7f1bf0103050 2024-07-26T06:55:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.325+0000 7f1bf4cd1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf0101fd0 0x7f1bf0198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.325+0000 7f1bf4cd1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf0101fd0 0x7f1bf0198890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48124/0 (socket says 172.21.15.42:48124) 2024-07-26T06:55:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.325+0000 7f1bf4cd1640 1 -- 172.21.15.42:0/4020404995 learned_addr learned my addr 172.21.15.42:0/4020404995 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.325+0000 7f1beffff640 1 --2- 172.21.15.42:0/4020404995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1bf0103050 0x7f1bf0198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.325+0000 7f1bf54d2640 1 --2- 172.21.15.42:0/4020404995 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bf01039f0 0x7f1bf0199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.325+0000 7f1bf4cd1640 1 -- 172.21.15.42:0/4020404995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1bf0103050 msgr2=0x7f1bf0198f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.325+0000 7f1bf4cd1640 1 --2- 172.21.15.42:0/4020404995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1bf0103050 0x7f1bf0198f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.325+0000 7f1bf4cd1640 1 -- 172.21.15.42:0/4020404995 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bf01039f0 msgr2=0x7f1bf0199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.325+0000 7f1bf4cd1640 1 --2- 172.21.15.42:0/4020404995 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bf01039f0 0x7f1bf0199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.325+0000 7f1bf4cd1640 1 -- 172.21.15.42:0/4020404995 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf019a2d0 con 0x7f1bf0101fd0 2024-07-26T06:55:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.325+0000 7f1bf4cd1640 1 --2- 172.21.15.42:0/4020404995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf0101fd0 0x7f1bf0198890 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1be400eb00 tx=0x7f1be4009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.326+0000 7f1bedffb640 1 -- 172.21.15.42:0/4020404995 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be401c070 con 0x7f1bf0101fd0 2024-07-26T06:55:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.326+0000 7f1bedffb640 1 -- 172.21.15.42:0/4020404995 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1be400bc30 con 0x7f1bf0101fd0 2024-07-26T06:55:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.326+0000 7f1bf6f35640 1 -- 172.21.15.42:0/4020404995 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf019a5c0 con 0x7f1bf0101fd0 2024-07-26T06:55:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.326+0000 7f1bedffb640 1 -- 172.21.15.42:0/4020404995 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be4016610 con 0x7f1bf0101fd0 2024-07-26T06:55:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.326+0000 7f1bf6f35640 1 -- 172.21.15.42:0/4020404995 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf006a7f0 con 0x7f1bf0101fd0 2024-07-26T06:55:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.330+0000 7f1bfde48740 1 Processor -- start 2024-07-26T06:55:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.330+0000 7f1bfde48740 1 -- start start 2024-07-26T06:55:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.330+0000 7f1bfde48740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 0x56195b32bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.330+0000 7f1bfde48740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af5e2f0 0x56195b32c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.330+0000 7f1bfde48740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af664d0 0x56195b32cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.330+0000 7f1bfde48740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x56195b29f3b0 con 0x56195af871b0 2024-07-26T06:55:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.330+0000 7f1bfde48740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56195b28e8b0 con 0x56195af664d0 2024-07-26T06:55:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.330+0000 7f1bfde48740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x56195b2928f0 con 0x56195af5e2f0 2024-07-26T06:55:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.330+0000 7f1bf54d2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 0x56195b32bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.330+0000 7f1beffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af664d0 0x56195b32cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.330+0000 7f1bf54d2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 0x56195b32bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48132/0 (socket says 172.21.15.42:48132) 2024-07-26T06:55:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.330+0000 7f1beffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af664d0 0x56195b32cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:56082/0 (socket says 172.21.15.42:56082) 2024-07-26T06:55:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.330+0000 7f1bf54d2640 1 -- 172.21.15.42:0/677233328 learned_addr learned my addr 172.21.15.42:0/677233328 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.331+0000 7f1beffff640 1 -- 172.21.15.42:0/677233328 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af5e2f0 msgr2=0x56195b32c1c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T06:55:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.331+0000 7f1beffff640 1 --2- 172.21.15.42:0/677233328 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af5e2f0 0x56195b32c1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.331+0000 7f1beffff640 1 -- 172.21.15.42:0/677233328 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 msgr2=0x56195b32bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.331+0000 7f1beffff640 1 --2- 172.21.15.42:0/677233328 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 0x56195b32bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.331+0000 7f1beffff640 1 -- 172.21.15.42:0/677233328 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56195b2359a0 con 0x56195af664d0 2024-07-26T06:55:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.331+0000 7f1bf54d2640 1 --2- 172.21.15.42:0/677233328 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 0x56195b32bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:55:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.331+0000 7f1bedffb640 1 -- 172.21.15.42:0/4020404995 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f1be401f070 con 0x7f1bf0101fd0 2024-07-26T06:55:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.331+0000 7f1beffff640 1 --2- 172.21.15.42:0/677233328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af664d0 0x56195b32cd10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc004630 tx=0x7f1bdc00bb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38158005b6bad090 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.331+0000 7f1bdaffd640 1 -- 172.21.15.42:0/677233328 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc003bd0 con 0x56195af664d0 2024-07-26T06:55:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.331+0000 7f1bdaffd640 1 -- 172.21.15.42:0/677233328 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bdc015070 con 0x56195af664d0 2024-07-26T06:55:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.331+0000 7f1bdaffd640 1 -- 172.21.15.42:0/677233328 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc004c70 con 0x56195af664d0 2024-07-26T06:55:18.338 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.332+0000 7f1bedffb640 1 --2- 172.21.15.42:0/4020404995 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1be00b3bd0 0x7f1be00b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.338 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.332+0000 7f1bfde48740 1 -- 172.21.15.42:0/677233328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af664d0 msgr2=0x56195b32cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.332+0000 7f1bfde48740 1 --2- 172.21.15.42:0/677233328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af664d0 0x56195b32cd10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc004630 tx=0x7f1bdc00bb20 comp rx=0 tx=0).stop 2024-07-26T06:55:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.332+0000 7f1bedffb640 1 -- 172.21.15.42:0/4020404995 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f1be40e5240 con 0x7f1bf0101fd0 2024-07-26T06:55:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.332+0000 7f1bf54d2640 1 --2- 172.21.15.42:0/4020404995 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1be00b3bd0 0x7f1be00b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.332+0000 7f1bfde48740 1 -- 172.21.15.42:0/677233328 shutdown_connections 2024-07-26T06:55:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.332+0000 7f1bfde48740 1 --2- 172.21.15.42:0/677233328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af664d0 0x56195b32cd10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.332+0000 7f1bfde48740 1 --2- 172.21.15.42:0/677233328 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af5e2f0 0x56195b32c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.332+0000 7f1bfde48740 1 --2- 172.21.15.42:0/677233328 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 0x56195b32bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.332+0000 7f1bfde48740 1 -- 172.21.15.42:0/677233328 >> 172.21.15.42:0/677233328 conn(0x56195af7b300 msgr2=0x56195b3275e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.332+0000 7f1bfde48740 1 -- 172.21.15.42:0/677233328 shutdown_connections 2024-07-26T06:55:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.332+0000 7f1bfde48740 1 -- 172.21.15.42:0/677233328 wait complete. 2024-07-26T06:55:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.333+0000 7f1bf54d2640 1 --2- 172.21.15.42:0/4020404995 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1be00b3bd0 0x7f1be00b60e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009a20 tx=0x7f1be8031000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:18.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.333+0000 7f1bfde48740 1 -- start start 2024-07-26T06:55:18.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.333+0000 7f1bfde48740 1 Processor -- start 2024-07-26T06:55:18.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.333+0000 7f1bfde48740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af5e2f0 0x56195b335570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.333+0000 7f1bfde48740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af664d0 0x56195b335c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.333+0000 7f1bfde48740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 0x56195b3367a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.333+0000 7f1bfde48740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x56195b1c8ab0 con 0x56195af871b0 2024-07-26T06:55:18.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.333+0000 7f1bfde48740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56195b216cf0 con 0x56195af664d0 2024-07-26T06:55:18.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.333+0000 7f1bfde48740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x56195b32d830 con 0x56195af5e2f0 2024-07-26T06:55:18.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.334+0000 7f1bfde48740 10 client.0 _ll_register_callbacks cb 0x56195b2883e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:55:18.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.334+0000 7f1bf4cd1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af5e2f0 0x56195b335570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:18.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.334+0000 7f1bf54d2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 0x56195b3367a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:18.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.334+0000 7f1beffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af664d0 0x56195b335c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.334+0000 7f1bf4cd1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af5e2f0 0x56195b335570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:58674/0 (socket says 172.21.15.42:58674) 2024-07-26T06:55:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.334+0000 7f1bf54d2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 0x56195b3367a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48134/0 (socket says 172.21.15.42:48134) 2024-07-26T06:55:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.334+0000 7f1bf4cd1640 1 -- 172.21.15.42:0/1523086105 learned_addr learned my addr 172.21.15.42:0/1523086105 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.334+0000 7f1beffff640 1 -- 172.21.15.42:0/1523086105 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af5e2f0 msgr2=0x56195b335570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.334+0000 7f1beffff640 1 --2- 172.21.15.42:0/1523086105 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af5e2f0 0x56195b335570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.334+0000 7f1beffff640 1 -- 172.21.15.42:0/1523086105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 msgr2=0x56195b3367a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.334+0000 7f1beffff640 1 --2- 172.21.15.42:0/1523086105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 0x56195b3367a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.334+0000 7f1beffff640 1 -- 172.21.15.42:0/1523086105 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56195aff2790 con 0x56195af664d0 2024-07-26T06:55:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.334+0000 7f1beffff640 1 --2- 172.21.15.42:0/1523086105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af664d0 0x56195b335c30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc004630 tx=0x7f1bdc003bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7043b05496cdf153 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.334+0000 7f1bcffff640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:55:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.334+0000 7f1bcffff640 1 -- 172.21.15.42:0/1523086105 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc028060 con 0x56195af664d0 2024-07-26T06:55:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.335+0000 7f1bfde48740 1 -- 172.21.15.42:0/1523086105 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bdc015070 con 0x56195af664d0 2024-07-26T06:55:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.335+0000 7f1bcffff640 1 -- 172.21.15.42:0/1523086105 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f1bdc011640 con 0x56195af664d0 2024-07-26T06:55:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.335+0000 7f1bcffff640 1 -- 172.21.15.42:0/1523086105 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1bdc011840 con 0x56195af664d0 2024-07-26T06:55:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.335+0000 7f1bfde48740 10 client.25562 fetch_fsmap learned FSMap version 37 2024-07-26T06:55:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.335+0000 7f1bfde48740 1 -- 172.21.15.42:0/1523086105 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56195b06b620 con 0x56195af664d0 2024-07-26T06:55:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.335+0000 7f1bcffff640 1 -- 172.21.15.42:0/1523086105 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bdc015070 con 0x56195af664d0 2024-07-26T06:55:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.335+0000 7f1bcffff640 1 -- 172.21.15.42:0/1523086105 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7f1bdc022070 con 0x56195af664d0 2024-07-26T06:55:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.335+0000 7f1bfde48740 10 client.25562 fetch_fsmap finished waiting for FSMap version 37 2024-07-26T06:55:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.335+0000 7f1bfde48740 10 client.25562 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:55:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.335+0000 7f1bfde48740 20 client.25562 populate_metadata read hostname 'smithi042' 2024-07-26T06:55:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.335+0000 7f1bfde48740 1 --2- 172.21.15.42:0/1523086105 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56195af45d80 0x56195b3429b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.335+0000 7f1bfde48740 4 client.25562 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:55:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.335+0000 7f1bfde48740 1 -- 172.21.15.42:0/1523086105 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56195b2a1300 con 0x56195af45d80 2024-07-26T06:55:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.336+0000 7f1bf4cd1640 1 --2- 172.21.15.42:0/1523086105 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56195af45d80 0x56195b3429b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.336+0000 7f1bcffff640 10 client.25562 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.336+0000 7f1bf4cd1640 1 --2- 172.21.15.42:0/1523086105 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56195af45d80 0x56195b3429b0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec613e4296fbcb53 server_cookie=d104e098ea5a9e1e in_seq=0 out_seq=0 2024-07-26T06:55:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.337+0000 7f1bcffff640 1 -- 172.21.15.42:0/1523086105 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56195b2a1300 con 0x56195af45d80 2024-07-26T06:55:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.337+0000 7f1bcffff640 10 client.25562 handle_command_reply: tid=0 2024-07-26T06:55:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.337+0000 7f1bfde48740 1 client.25562 shutdown 2024-07-26T06:55:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.337+0000 7f1bfde48740 1 -- 172.21.15.42:0/1523086105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af664d0 msgr2=0x56195b335c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.337+0000 7f1bfde48740 1 --2- 172.21.15.42:0/1523086105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af664d0 0x56195b335c30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc004630 tx=0x7f1bdc003bd0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.337+0000 7f1bfde48740 1 -- 172.21.15.42:0/1523086105 shutdown_connections 2024-07-26T06:55:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.337+0000 7f1bfde48740 1 --2- 172.21.15.42:0/1523086105 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56195af45d80 0x56195b3429b0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.337+0000 7f1bfde48740 1 --2- 172.21.15.42:0/1523086105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 0x56195b3367a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.337+0000 7f1bfde48740 1 --2- 172.21.15.42:0/1523086105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af664d0 0x56195b335c30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.337+0000 7f1bfde48740 1 --2- 172.21.15.42:0/1523086105 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af5e2f0 0x56195b335570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.337+0000 7f1bfde48740 1 -- 172.21.15.42:0/1523086105 >> 172.21.15.42:0/1523086105 conn(0x56195af7b300 msgr2=0x56195b325c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.337+0000 7f1bcffff640 0 client.25562 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.337+0000 7f1bfde48740 1 -- 172.21.15.42:0/1523086105 shutdown_connections 2024-07-26T06:55:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.337+0000 7f1bfde48740 1 -- 172.21.15.42:0/1523086105 wait complete. 2024-07-26T06:55:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.337+0000 7f1bfde48740 20 client.25562 trim_cache size 0 max 16384 2024-07-26T06:55:18.356 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.355+0000 7f1bfde48740 1 Processor -- start 2024-07-26T06:55:18.356 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.355+0000 7f1bfde48740 1 -- start start 2024-07-26T06:55:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.355+0000 7f1bfde48740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af7b300 0x56195b328d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.355+0000 7f1bfde48740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af5e2f0 0x56195b326390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.355+0000 7f1bfde48740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af664d0 0x56195b3268d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.355+0000 7f1bfde48740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf01a8860 con 0x56195af7b300 2024-07-26T06:55:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.355+0000 7f1bfde48740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf01aa460 con 0x56195af5e2f0 2024-07-26T06:55:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.355+0000 7f1bfde48740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf01a8f30 con 0x56195af664d0 2024-07-26T06:55:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.355+0000 7f1bf4cd1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af7b300 0x56195b328d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.355+0000 7f1bf4cd1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af7b300 0x56195b328d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48148/0 (socket says 172.21.15.42:48148) 2024-07-26T06:55:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.355+0000 7f1bf4cd1640 1 -- 172.21.15.42:0/3524603689 learned_addr learned my addr 172.21.15.42:0/3524603689 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.355+0000 7f1beffff640 1 --2- 172.21.15.42:0/3524603689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af5e2f0 0x56195b326390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.356+0000 7f1bf4cd1640 1 -- 172.21.15.42:0/3524603689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af664d0 msgr2=0x56195b3268d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:18.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.356+0000 7f1bf4cd1640 1 --2- 172.21.15.42:0/3524603689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af664d0 0x56195b3268d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.356+0000 7f1bf4cd1640 1 -- 172.21.15.42:0/3524603689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af5e2f0 msgr2=0x56195b326390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.356+0000 7f1bf4cd1640 1 --2- 172.21.15.42:0/3524603689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af5e2f0 0x56195b326390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.356+0000 7f1bf4cd1640 1 -- 172.21.15.42:0/3524603689 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56195b32bf80 con 0x56195af7b300 2024-07-26T06:55:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.356+0000 7f1bf4cd1640 1 --2- 172.21.15.42:0/3524603689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af7b300 0x56195b328d20 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1be40a82a0 tx=0x7f1be4017c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a8ca1b363675e56 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.356+0000 7f1bda7fc640 1 -- 172.21.15.42:0/3524603689 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be40b5220 con 0x56195af7b300 2024-07-26T06:55:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.356+0000 7f1bda7fc640 1 -- 172.21.15.42:0/3524603689 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1be40cb070 con 0x56195af7b300 2024-07-26T06:55:18.358 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.356+0000 7f1bda7fc640 1 -- 172.21.15.42:0/3524603689 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be40b3d50 con 0x56195af7b300 2024-07-26T06:55:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.357+0000 7f1bfde48740 1 -- 172.21.15.42:0/3524603689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af7b300 msgr2=0x56195b328d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.357+0000 7f1bfde48740 1 --2- 172.21.15.42:0/3524603689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af7b300 0x56195b328d20 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1be40a82a0 tx=0x7f1be4017c90 comp rx=0 tx=0).stop 2024-07-26T06:55:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.357+0000 7f1bfde48740 1 -- 172.21.15.42:0/3524603689 shutdown_connections 2024-07-26T06:55:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.357+0000 7f1bfde48740 1 --2- 172.21.15.42:0/3524603689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af664d0 0x56195b3268d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.357+0000 7f1bfde48740 1 --2- 172.21.15.42:0/3524603689 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af5e2f0 0x56195b326390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.357+0000 7f1bfde48740 1 --2- 172.21.15.42:0/3524603689 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af7b300 0x56195b328d20 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.357+0000 7f1bfde48740 1 -- 172.21.15.42:0/3524603689 >> 172.21.15.42:0/3524603689 conn(0x56195b342d00 msgr2=0x56195b325c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.357+0000 7f1bfde48740 1 -- 172.21.15.42:0/3524603689 shutdown_connections 2024-07-26T06:55:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.357+0000 7f1bfde48740 1 -- 172.21.15.42:0/3524603689 wait complete. 2024-07-26T06:55:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.358+0000 7f1bfde48740 1 -- start start 2024-07-26T06:55:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.358+0000 7f1bfde48740 1 Processor -- start 2024-07-26T06:55:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.358+0000 7f1bfde48740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af5e2f0 0x56195b33c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.358+0000 7f1bfde48740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af664d0 0x56195b33c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.358+0000 7f1bfde48740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 0x56195b33ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.358+0000 7f1bfde48740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf01a9d90 con 0x56195af871b0 2024-07-26T06:55:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.358+0000 7f1bfde48740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf01a9600 con 0x56195af664d0 2024-07-26T06:55:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.358+0000 7f1bfde48740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf01a6840 con 0x56195af5e2f0 2024-07-26T06:55:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.358+0000 7f1bfde48740 10 client.0 _ll_register_callbacks cb 0x7f1be4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:55:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.358+0000 7f1bf54d2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 0x56195b33ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.358+0000 7f1bf54d2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 0x56195b33ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48158/0 (socket says 172.21.15.42:48158) 2024-07-26T06:55:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.358+0000 7f1bf54d2640 1 -- 172.21.15.42:0/2081735924 learned_addr learned my addr 172.21.15.42:0/2081735924 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.359+0000 7f1bf54d2640 1 -- 172.21.15.42:0/2081735924 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af5e2f0 msgr2=0x56195b33c010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T06:55:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.359+0000 7f1bf54d2640 1 --2- 172.21.15.42:0/2081735924 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af5e2f0 0x56195b33c010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.359+0000 7f1bf54d2640 1 -- 172.21.15.42:0/2081735924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af664d0 msgr2=0x56195b33c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.359+0000 7f1bf54d2640 1 --2- 172.21.15.42:0/2081735924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af664d0 0x56195b33c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.359+0000 7f1bf54d2640 1 -- 172.21.15.42:0/2081735924 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56195b3a9180 con 0x56195af871b0 2024-07-26T06:55:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.359+0000 7f1bf54d2640 1 --2- 172.21.15.42:0/2081735924 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 0x56195b33ca90 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009c90 tx=0x7f1be802a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f373fa18ba3372 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.359+0000 7f1bcffff640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:55:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.359+0000 7f1bcffff640 1 -- 172.21.15.42:0/2081735924 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8030560 con 0x56195af871b0 2024-07-26T06:55:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.359+0000 7f1bcffff640 1 -- 172.21.15.42:0/2081735924 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f1be8030700 con 0x56195af871b0 2024-07-26T06:55:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.359+0000 7f1bfde48740 1 -- 172.21.15.42:0/2081735924 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1be40cb070 con 0x56195af871b0 2024-07-26T06:55:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.359+0000 7f1bcffff640 1 -- 172.21.15.42:0/2081735924 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1be8030900 con 0x56195af871b0 2024-07-26T06:55:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.360+0000 7f1bfde48740 10 client.15984 fetch_fsmap learned FSMap version 37 2024-07-26T06:55:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.360+0000 7f1bfde48740 1 -- 172.21.15.42:0/2081735924 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56195b3aa4b0 con 0x56195af871b0 2024-07-26T06:55:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.360+0000 7f1bcffff640 1 -- 172.21.15.42:0/2081735924 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1be40cb070 con 0x56195af871b0 2024-07-26T06:55:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.360+0000 7f1bcffff640 1 -- 172.21.15.42:0/2081735924 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7f1be8044050 con 0x56195af871b0 2024-07-26T06:55:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.360+0000 7f1bfde48740 10 client.15984 fetch_fsmap finished waiting for FSMap version 37 2024-07-26T06:55:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.360+0000 7f1bfde48740 10 client.15984 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:55:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.360+0000 7f1bfde48740 20 client.15984 populate_metadata read hostname 'smithi042' 2024-07-26T06:55:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.360+0000 7f1bfde48740 1 --2- 172.21.15.42:0/2081735924 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56195af1b1c0 0x56195b3acf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.360+0000 7f1bfde48740 4 client.15984 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:55:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.360+0000 7f1bfde48740 1 -- 172.21.15.42:0/2081735924 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56195b29cda0 con 0x56195af1b1c0 2024-07-26T06:55:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.360+0000 7f1bf4cd1640 1 --2- 172.21.15.42:0/2081735924 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56195af1b1c0 0x56195b3acf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.361+0000 7f1bcffff640 10 client.15984 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.361+0000 7f1bf4cd1640 1 --2- 172.21.15.42:0/2081735924 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56195af1b1c0 0x56195b3acf90 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=538eaf18aad982e3 server_cookie=a2ee259a7617a27a in_seq=0 out_seq=0 2024-07-26T06:55:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.380+0000 7f1bcffff640 1 -- 172.21.15.42:0/2081735924 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56195b29cda0 con 0x56195af1b1c0 2024-07-26T06:55:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.380+0000 7f1bcffff640 10 client.15984 handle_command_reply: tid=0 2024-07-26T06:55:18.381 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.381+0000 7f1bfde48740 1 client.15984 shutdown 2024-07-26T06:55:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.381+0000 7f1bfde48740 1 -- 172.21.15.42:0/2081735924 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 msgr2=0x56195b33ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.381+0000 7f1bfde48740 1 --2- 172.21.15.42:0/2081735924 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 0x56195b33ca90 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009c90 tx=0x7f1be802a860 comp rx=0 tx=0).stop 2024-07-26T06:55:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.381+0000 7f1bfde48740 1 -- 172.21.15.42:0/2081735924 shutdown_connections 2024-07-26T06:55:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.381+0000 7f1bfde48740 1 --2- 172.21.15.42:0/2081735924 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x56195af1b1c0 0x56195b3acf90 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.381+0000 7f1bfde48740 1 --2- 172.21.15.42:0/2081735924 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x56195af871b0 0x56195b33ca90 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.381+0000 7f1bfde48740 1 --2- 172.21.15.42:0/2081735924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56195af664d0 0x56195b33c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.381+0000 7f1bfde48740 1 --2- 172.21.15.42:0/2081735924 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x56195af5e2f0 0x56195b33c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.382+0000 7f1bfde48740 1 -- 172.21.15.42:0/2081735924 >> 172.21.15.42:0/2081735924 conn(0x56195b342d00 msgr2=0x56195b326640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.382+0000 7f1bcffff640 0 client.15984 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:18.384 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.383+0000 7f1bfde48740 1 -- 172.21.15.42:0/2081735924 shutdown_connections 2024-07-26T06:55:18.384 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.383+0000 7f1bfde48740 1 -- 172.21.15.42:0/2081735924 wait complete. 2024-07-26T06:55:18.384 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.383+0000 7f1bfde48740 20 client.15984 trim_cache size 0 max 16384 2024-07-26T06:55:18.384 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:55:18.384 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:55:18.384 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:55:18.384 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:55:18.387 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.386+0000 7f1bf6f35640 1 -- 172.21.15.42:0/4020404995 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1be00b3bd0 msgr2=0x7f1be00b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:18.387 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.386+0000 7f1bf6f35640 1 --2- 172.21.15.42:0/4020404995 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1be00b3bd0 0x7f1be00b60e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009a20 tx=0x7f1be8031000 comp rx=0 tx=0).stop 2024-07-26T06:55:18.387 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.386+0000 7f1bf6f35640 1 -- 172.21.15.42:0/4020404995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf0101fd0 msgr2=0x7f1bf0198890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:18.387 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.386+0000 7f1bf6f35640 1 --2- 172.21.15.42:0/4020404995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf0101fd0 0x7f1bf0198890 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1be400eb00 tx=0x7f1be4009f20 comp rx=0 tx=0).stop 2024-07-26T06:55:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.386+0000 7f1bf6f35640 1 -- 172.21.15.42:0/4020404995 shutdown_connections 2024-07-26T06:55:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.386+0000 7f1bf6f35640 1 --2- 172.21.15.42:0/4020404995 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f1be00b3bd0 0x7f1be00b60e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.386+0000 7f1bf6f35640 1 --2- 172.21.15.42:0/4020404995 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bf01039f0 0x7f1bf0199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.386+0000 7f1bf6f35640 1 --2- 172.21.15.42:0/4020404995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1bf0103050 0x7f1bf0198f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.386+0000 7f1bf6f35640 1 --2- 172.21.15.42:0/4020404995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1bf0101fd0 0x7f1bf0198890 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.386+0000 7f1bf6f35640 1 -- 172.21.15.42:0/4020404995 >> 172.21.15.42:0/4020404995 conn(0x7f1bf00fd910 msgr2=0x7f1bf0106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.386+0000 7f1bf6f35640 1 -- 172.21.15.42:0/4020404995 shutdown_connections 2024-07-26T06:55:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:18.386+0000 7f1bf6f35640 1 -- 172.21.15.42:0/4020404995 wait complete. 2024-07-26T06:55:18.967 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:18 smithi042 ceph-mon[32775]: pgmap v549: 97 pgs: 97 active+clean; 67 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 2.6 MiB/s wr, 992 op/s 2024-07-26T06:55:19.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.083+0000 7f59d8dfb640 1 -- 172.21.15.42:0/2991263770 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59d41019a0 msgr2=0x7f59d4101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:19.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.083+0000 7f59d8dfb640 1 --2- 172.21.15.42:0/2991263770 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59d41019a0 0x7f59d4101db0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f59c4009ca0 tx=0x7f59c402a8c0 comp rx=0 tx=0).stop 2024-07-26T06:55:19.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.083+0000 7f59d8dfb640 1 -- 172.21.15.42:0/2991263770 shutdown_connections 2024-07-26T06:55:19.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.083+0000 7f59d8dfb640 1 --2- 172.21.15.42:0/2991263770 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d4109830 0x7f59d4110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:19.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.083+0000 7f59d8dfb640 1 --2- 172.21.15.42:0/2991263770 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59d4108ee0 0x7f59d41092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:19.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.083+0000 7f59d8dfb640 1 --2- 172.21.15.42:0/2991263770 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59d41019a0 0x7f59d4101db0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:19.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.083+0000 7f59d8dfb640 1 -- 172.21.15.42:0/2991263770 >> 172.21.15.42:0/2991263770 conn(0x7f59d4077690 msgr2=0x7f59d4077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:19.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.083+0000 7f59d8dfb640 1 -- 172.21.15.42:0/2991263770 shutdown_connections 2024-07-26T06:55:19.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.083+0000 7f59d8dfb640 1 -- 172.21.15.42:0/2991263770 wait complete. 2024-07-26T06:55:19.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.084+0000 7f59d8dfb640 1 Processor -- start 2024-07-26T06:55:19.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.084+0000 7f59d8dfb640 1 -- start start 2024-07-26T06:55:19.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.084+0000 7f59d8dfb640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59d41019a0 0x7f59d4199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:19.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.084+0000 7f59d8dfb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59d4108ee0 0x7f59d419a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:19.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.084+0000 7f59d8dfb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d4109830 0x7f59d419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:19.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.084+0000 7f59d8dfb640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f59d4113550 con 0x7f59d4108ee0 2024-07-26T06:55:19.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.084+0000 7f59d8dfb640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f59d41133d0 con 0x7f59d41019a0 2024-07-26T06:55:19.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.084+0000 7f59d8dfb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f59d41136d0 con 0x7f59d4109830 2024-07-26T06:55:19.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.085+0000 7f59d1d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59d4108ee0 0x7f59d419a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:19.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.085+0000 7f59d259c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59d41019a0 0x7f59d4199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:19.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.085+0000 7f59d1d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59d4108ee0 0x7f59d419a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48186/0 (socket says 172.21.15.42:48186) 2024-07-26T06:55:19.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.085+0000 7f59d1d9b640 1 -- 172.21.15.42:0/836490465 learned_addr learned my addr 172.21.15.42:0/836490465 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:19.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.085+0000 7f59d2d9d640 1 --2- 172.21.15.42:0/836490465 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d4109830 0x7f59d419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:19.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.085+0000 7f59d259c640 1 -- 172.21.15.42:0/836490465 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d4109830 msgr2=0x7f59d419f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:19.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.085+0000 7f59d259c640 1 --2- 172.21.15.42:0/836490465 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d4109830 0x7f59d419f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:19.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.085+0000 7f59d259c640 1 -- 172.21.15.42:0/836490465 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59d4108ee0 msgr2=0x7f59d419a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:19.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.085+0000 7f59d259c640 1 --2- 172.21.15.42:0/836490465 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59d4108ee0 0x7f59d419a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:19.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.085+0000 7f59d259c640 1 -- 172.21.15.42:0/836490465 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d419fe10 con 0x7f59d41019a0 2024-07-26T06:55:19.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.085+0000 7f59d1d9b640 1 --2- 172.21.15.42:0/836490465 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59d4108ee0 0x7f59d419a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:55:19.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.085+0000 7f59d259c640 1 --2- 172.21.15.42:0/836490465 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59d41019a0 0x7f59d4199ea0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f59c4000f80 tx=0x7f59c4031750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:19.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.086+0000 7f59cb7fe640 1 -- 172.21.15.42:0/836490465 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c4003b70 con 0x7f59d41019a0 2024-07-26T06:55:19.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.086+0000 7f59d8dfb640 1 -- 172.21.15.42:0/836490465 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59d41a00a0 con 0x7f59d41019a0 2024-07-26T06:55:19.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.086+0000 7f59cb7fe640 1 -- 172.21.15.42:0/836490465 <== mon.2 v2:172.21.15.163:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59c4003d10 con 0x7f59d41019a0 2024-07-26T06:55:19.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.086+0000 7f59cb7fe640 1 -- 172.21.15.42:0/836490465 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c402fa30 con 0x7f59d41019a0 2024-07-26T06:55:19.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.086+0000 7f59d8dfb640 1 -- 172.21.15.42:0/836490465 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59d41a6860 con 0x7f59d41019a0 2024-07-26T06:55:19.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.087+0000 7f59d8dfb640 1 -- 172.21.15.42:0/836490465 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59d4101db0 con 0x7f59d41019a0 2024-07-26T06:55:19.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.088+0000 7f59cb7fe640 1 -- 172.21.15.42:0/836490465 <== mon.2 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f59c4004580 con 0x7f59d41019a0 2024-07-26T06:55:19.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.089+0000 7f59cb7fe640 1 --2- 172.21.15.42:0/836490465 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f59c00b3bd0 0x7f59c00b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:19.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.089+0000 7f59d1d9b640 1 --2- 172.21.15.42:0/836490465 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f59c00b3bd0 0x7f59c00b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:19.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.089+0000 7f59cb7fe640 1 -- 172.21.15.42:0/836490465 <== mon.2 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f59c4105060 con 0x7f59d41019a0 2024-07-26T06:55:19.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.090+0000 7f59d1d9b640 1 --2- 172.21.15.42:0/836490465 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f59c00b3bd0 0x7f59c00b60e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f59d4108b70 tx=0x7f59bc005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:19.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.094+0000 7f59cb7fe640 1 -- 172.21.15.42:0/836490465 <== mon.2 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f59c40cc690 con 0x7f59d41019a0 2024-07-26T06:55:19.221 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:18 smithi164 ceph-mon[36970]: pgmap v549: 97 pgs: 97 active+clean; 67 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 2.6 MiB/s wr, 992 op/s 2024-07-26T06:55:19.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:18 smithi163 ceph-mon[36873]: pgmap v549: 97 pgs: 97 active+clean; 67 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 2.6 MiB/s wr, 992 op/s 2024-07-26T06:55:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.352+0000 7f59d8dfb640 1 -- 172.21.15.42:0/836490465 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59d419afd0 con 0x7f59d41019a0 2024-07-26T06:55:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.353+0000 7f59cb7fe640 1 -- 172.21.15.42:0/836490465 <== mon.2 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11985 (secure 0 0 0) 0x7f59c40cbf90 con 0x7f59d41019a0 2024-07-26T06:55:19.354 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:55:19.354 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":37,"btime":"2024-07-26T06:54:35:802227+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:54:35.748994+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25369":{"gid":25369,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/1072602530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1072602530},{"type":"v1","addr":"172.21.15.164:6835","nonce":1072602530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24620,24601,24593,24583,24578]},"id":1}]} 2024-07-26T06:55:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 37 2024-07-26T06:55:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.357+0000 7f59d8dfb640 1 -- 172.21.15.42:0/836490465 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f59c00b3bd0 msgr2=0x7f59c00b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.357+0000 7f59d8dfb640 1 --2- 172.21.15.42:0/836490465 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f59c00b3bd0 0x7f59c00b60e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f59d4108b70 tx=0x7f59bc005f90 comp rx=0 tx=0).stop 2024-07-26T06:55:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.357+0000 7f59d8dfb640 1 -- 172.21.15.42:0/836490465 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59d41019a0 msgr2=0x7f59d4199ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.357+0000 7f59d8dfb640 1 --2- 172.21.15.42:0/836490465 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59d41019a0 0x7f59d4199ea0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f59c4000f80 tx=0x7f59c4031750 comp rx=0 tx=0).stop 2024-07-26T06:55:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.358+0000 7f59d8dfb640 1 -- 172.21.15.42:0/836490465 shutdown_connections 2024-07-26T06:55:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.358+0000 7f59d8dfb640 1 --2- 172.21.15.42:0/836490465 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f59c00b3bd0 0x7f59c00b60e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.358+0000 7f59d8dfb640 1 --2- 172.21.15.42:0/836490465 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d4109830 0x7f59d419f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.358+0000 7f59d8dfb640 1 --2- 172.21.15.42:0/836490465 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59d4108ee0 0x7f59d419a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.358+0000 7f59d8dfb640 1 --2- 172.21.15.42:0/836490465 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f59d41019a0 0x7f59d4199ea0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.358+0000 7f59d8dfb640 1 -- 172.21.15.42:0/836490465 >> 172.21.15.42:0/836490465 conn(0x7f59d4077690 msgr2=0x7f59d4106f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.358+0000 7f59d8dfb640 1 -- 172.21.15.42:0/836490465 shutdown_connections 2024-07-26T06:55:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:19.358+0000 7f59d8dfb640 1 -- 172.21.15.42:0/836490465 wait complete. 2024-07-26T06:55:19.493 INFO:tasks.workunit.client.0.smithi042.stdout:copying to largest ever: 0x63ef9ee 2024-07-26T06:55:20.115 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:19 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/836490465' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:55:20.151 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24620 quiesce db / --timeout 90 --expiration 130 --await-for 45.0' 2024-07-26T06:55:20.151 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.24620 quiesce db / --timeout 90 --expiration 130 --await-for 45.0 2024-07-26T06:55:20.221 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:19 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/836490465' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:55:20.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:19 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/836490465' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:55:20.475 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:55:21.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:20 smithi042 ceph-mon[32775]: pgmap v550: 97 pgs: 97 active+clean; 57 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 2.7 MiB/s wr, 1.06k op/s 2024-07-26T06:55:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:20 smithi164 ceph-mon[36970]: pgmap v550: 97 pgs: 97 active+clean; 57 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 2.7 MiB/s wr, 1.06k op/s 2024-07-26T06:55:21.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:20 smithi163 ceph-mon[36873]: pgmap v550: 97 pgs: 97 active+clean; 57 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 2.7 MiB/s wr, 1.06k op/s 2024-07-26T06:55:22.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:22 smithi042 ceph-mon[32775]: pgmap v551: 97 pgs: 97 active+clean; 63 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.3 MiB/s wr, 862 op/s 2024-07-26T06:55:23.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:22 smithi164 ceph-mon[36970]: pgmap v551: 97 pgs: 97 active+clean; 63 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.3 MiB/s wr, 862 op/s 2024-07-26T06:55:23.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:22 smithi163 ceph-mon[36873]: pgmap v551: 97 pgs: 97 active+clean; 63 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.3 MiB/s wr, 862 op/s 2024-07-26T06:55:23.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.227+0000 7efec94a2640 1 -- 172.21.15.42:0/1059821203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efec41024b0 msgr2=0x7efec41028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:23.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.227+0000 7efec94a2640 1 --2- 172.21.15.42:0/1059821203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efec41024b0 0x7efec41028c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7efeb4009900 tx=0x7efeb402a410 comp rx=0 tx=0).stop 2024-07-26T06:55:23.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.227+0000 7efec94a2640 1 -- 172.21.15.42:0/1059821203 shutdown_connections 2024-07-26T06:55:23.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.227+0000 7efec94a2640 1 --2- 172.21.15.42:0/1059821203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efec4108740 0x7efec410ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.227+0000 7efec94a2640 1 --2- 172.21.15.42:0/1059821203 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efec4107df0 0x7efec4108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.227+0000 7efec94a2640 1 --2- 172.21.15.42:0/1059821203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efec41024b0 0x7efec41028c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.227+0000 7efec94a2640 1 -- 172.21.15.42:0/1059821203 >> 172.21.15.42:0/1059821203 conn(0x7efec4079190 msgr2=0x7efec40795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:23.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.227+0000 7efec94a2640 1 -- 172.21.15.42:0/1059821203 shutdown_connections 2024-07-26T06:55:23.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.227+0000 7efec94a2640 1 -- 172.21.15.42:0/1059821203 wait complete. 2024-07-26T06:55:23.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.228+0000 7efec94a2640 1 Processor -- start 2024-07-26T06:55:23.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.228+0000 7efec94a2640 1 -- start start 2024-07-26T06:55:23.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.228+0000 7efec94a2640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efec41024b0 0x7efec4199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.228+0000 7efec94a2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efec4107df0 0x7efec419a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.228+0000 7efec94a2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efec4108740 0x7efec419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.228+0000 7efec94a2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7efec41123c0 con 0x7efec4108740 2024-07-26T06:55:23.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.228+0000 7efec94a2640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efec4112240 con 0x7efec41024b0 2024-07-26T06:55:23.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.228+0000 7efec94a2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7efec4112540 con 0x7efec4107df0 2024-07-26T06:55:23.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.228+0000 7efec37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efec4108740 0x7efec419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.228+0000 7efec37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efec4108740 0x7efec419f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48200/0 (socket says 172.21.15.42:48200) 2024-07-26T06:55:23.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.228+0000 7efec37fe640 1 -- 172.21.15.42:0/3782781323 learned_addr learned my addr 172.21.15.42:0/3782781323 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:23.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.228+0000 7efec2ffd640 1 --2- 172.21.15.42:0/3782781323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efec41024b0 0x7efec4199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.228+0000 7efebbfff640 1 --2- 172.21.15.42:0/3782781323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efec4107df0 0x7efec419a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.229+0000 7efec37fe640 1 -- 172.21.15.42:0/3782781323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efec4107df0 msgr2=0x7efec419a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:23.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.229+0000 7efec37fe640 1 --2- 172.21.15.42:0/3782781323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efec4107df0 0x7efec419a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.229+0000 7efec37fe640 1 -- 172.21.15.42:0/3782781323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efec41024b0 msgr2=0x7efec4199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:23.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.229+0000 7efec37fe640 1 --2- 172.21.15.42:0/3782781323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efec41024b0 0x7efec4199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.229+0000 7efec37fe640 1 -- 172.21.15.42:0/3782781323 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec419fe70 con 0x7efec4108740 2024-07-26T06:55:23.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.229+0000 7efec37fe640 1 --2- 172.21.15.42:0/3782781323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efec4108740 0x7efec419f740 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7efebc00eb10 tx=0x7efebc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:23.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.229+0000 7efec0ff9640 1 -- 172.21.15.42:0/3782781323 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc004b20 con 0x7efec4108740 2024-07-26T06:55:23.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.229+0000 7efec0ff9640 1 -- 172.21.15.42:0/3782781323 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efebc004cc0 con 0x7efec4108740 2024-07-26T06:55:23.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.229+0000 7efec94a2640 1 -- 172.21.15.42:0/3782781323 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efec41a0160 con 0x7efec4108740 2024-07-26T06:55:23.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.229+0000 7efec0ff9640 1 -- 172.21.15.42:0/3782781323 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc010670 con 0x7efec4108740 2024-07-26T06:55:23.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.229+0000 7efec94a2640 1 -- 172.21.15.42:0/3782781323 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efec41a68c0 con 0x7efec4108740 2024-07-26T06:55:23.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.232+0000 7efecfa26740 1 Processor -- start 2024-07-26T06:55:23.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.232+0000 7efec0ff9640 1 -- 172.21.15.42:0/3782781323 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7efebc005020 con 0x7efec4108740 2024-07-26T06:55:23.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.232+0000 7efecfa26740 1 -- start start 2024-07-26T06:55:23.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efecfa26740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee1be840 0x55fdee564840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efecfa26740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee195980 0x55fdee564d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efecfa26740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee19db60 0x55fdee5652c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efecfa26740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55fdee4d9780 con 0x55fdee19db60 2024-07-26T06:55:23.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efecfa26740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fdee4c8c80 con 0x55fdee1be840 2024-07-26T06:55:23.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efecfa26740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55fdee4cccc0 con 0x55fdee195980 2024-07-26T06:55:23.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efec2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee195980 0x55fdee564d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efebbfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee1be840 0x55fdee564840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efec2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee195980 0x55fdee564d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:58708/0 (socket says 172.21.15.42:58708) 2024-07-26T06:55:23.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efec2ffd640 1 -- 172.21.15.42:0/3793287616 learned_addr learned my addr 172.21.15.42:0/3793287616 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:23.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efebbfff640 1 --2- 172.21.15.42:0/3793287616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee19db60 0x55fdee5652c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efec0ff9640 1 --2- 172.21.15.42:0/3782781323 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7efeb00b3c50 0x7efeb00b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efec2ffd640 1 -- 172.21.15.42:0/3793287616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee1be840 msgr2=0x55fdee564840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:23.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efec2ffd640 1 --2- 172.21.15.42:0/3793287616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee1be840 0x55fdee564840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efec2ffd640 1 -- 172.21.15.42:0/3793287616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee19db60 msgr2=0x55fdee5652c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:23.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efec2ffd640 1 --2- 172.21.15.42:0/3793287616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee19db60 0x55fdee5652c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efec2ffd640 1 -- 172.21.15.42:0/3793287616 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdee45e080 con 0x55fdee195980 2024-07-26T06:55:23.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efec37fe640 1 --2- 172.21.15.42:0/3782781323 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7efeb00b3c50 0x7efeb00b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efec0ff9640 1 -- 172.21.15.42:0/3782781323 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7efebc0e40e0 con 0x7efec4108740 2024-07-26T06:55:23.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.233+0000 7efec2ffd640 1 --2- 172.21.15.42:0/3793287616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee195980 0x55fdee564d80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efeb4009900 tx=0x7efeb4032b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a30eb157a71476a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:23.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.234+0000 7efeb97fa640 1 -- 172.21.15.42:0/3793287616 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb4040070 con 0x55fdee195980 2024-07-26T06:55:23.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.234+0000 7efeb97fa640 1 -- 172.21.15.42:0/3793287616 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efeb4004030 con 0x55fdee195980 2024-07-26T06:55:23.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.234+0000 7efeb97fa640 1 -- 172.21.15.42:0/3793287616 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb4035040 con 0x55fdee195980 2024-07-26T06:55:23.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.234+0000 7efec37fe640 1 --2- 172.21.15.42:0/3782781323 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7efeb00b3c50 0x7efeb00b6160 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efebc0e1f30 tx=0x7efebc00cee0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:23.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.235+0000 7efecfa26740 1 -- 172.21.15.42:0/3793287616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee195980 msgr2=0x55fdee564d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:23.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.235+0000 7efecfa26740 1 --2- 172.21.15.42:0/3793287616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee195980 0x55fdee564d80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efeb4009900 tx=0x7efeb4032b90 comp rx=0 tx=0).stop 2024-07-26T06:55:23.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.236+0000 7efecfa26740 1 -- 172.21.15.42:0/3793287616 shutdown_connections 2024-07-26T06:55:23.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.236+0000 7efecfa26740 1 --2- 172.21.15.42:0/3793287616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee19db60 0x55fdee5652c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.236+0000 7efecfa26740 1 --2- 172.21.15.42:0/3793287616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee195980 0x55fdee564d80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.236+0000 7efecfa26740 1 --2- 172.21.15.42:0/3793287616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee1be840 0x55fdee564840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.236+0000 7efecfa26740 1 -- 172.21.15.42:0/3793287616 >> 172.21.15.42:0/3793287616 conn(0x55fdee1b2990 msgr2=0x55fdee562730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:23.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.236+0000 7efecfa26740 1 -- 172.21.15.42:0/3793287616 shutdown_connections 2024-07-26T06:55:23.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.236+0000 7efecfa26740 1 -- 172.21.15.42:0/3793287616 wait complete. 2024-07-26T06:55:23.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.236+0000 7efecfa26740 1 -- start start 2024-07-26T06:55:23.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.236+0000 7efecfa26740 1 Processor -- start 2024-07-26T06:55:23.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.237+0000 7efecfa26740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee195980 0x55fdee56fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.237+0000 7efecfa26740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 0x55fdee570320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.237+0000 7efecfa26740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee1be840 0x55fdee570860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.237+0000 7efecfa26740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55fdee32c7a0 con 0x55fdee195980 2024-07-26T06:55:23.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.237+0000 7efecfa26740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fdee46b2f0 con 0x55fdee19db60 2024-07-26T06:55:23.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.237+0000 7efecfa26740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55fdee572fc0 con 0x55fdee1be840 2024-07-26T06:55:23.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.237+0000 7efecfa26740 10 client.0 _ll_register_callbacks cb 0x55fdee475110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:55:23.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.237+0000 7efebbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee195980 0x55fdee56fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.237+0000 7efec2ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 0x55fdee570320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.237+0000 7efebbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee195980 0x55fdee56fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48208/0 (socket says 172.21.15.42:48208) 2024-07-26T06:55:23.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.237+0000 7efebbfff640 1 -- 172.21.15.42:0/2526351998 learned_addr learned my addr 172.21.15.42:0/2526351998 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:23.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.237+0000 7efebbfff640 1 --2- 172.21.15.42:0/2526351998 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee1be840 0x55fdee570860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.237+0000 7efebbfff640 1 -- 172.21.15.42:0/2526351998 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee1be840 msgr2=0x55fdee570860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.237+0000 7efebbfff640 1 --2- 172.21.15.42:0/2526351998 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee1be840 0x55fdee570860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.237+0000 7efebbfff640 1 -- 172.21.15.42:0/2526351998 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 msgr2=0x55fdee570320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.237+0000 7efebbfff640 1 --2- 172.21.15.42:0/2526351998 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 0x55fdee570320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.237+0000 7efebbfff640 1 -- 172.21.15.42:0/2526351998 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdee46f480 con 0x55fdee195980 2024-07-26T06:55:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.238+0000 7efebbfff640 1 --2- 172.21.15.42:0/2526351998 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee195980 0x55fdee56fde0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7efeac00f3c0 tx=0x7efeac009aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1716a45542d0bcb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.238+0000 7efeaa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:55:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.238+0000 7efeaa7fc640 1 -- 172.21.15.42:0/2526351998 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac00fe00 con 0x55fdee195980 2024-07-26T06:55:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.238+0000 7efecfa26740 1 -- 172.21.15.42:0/2526351998 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efeb4004030 con 0x55fdee195980 2024-07-26T06:55:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.238+0000 7efeaa7fc640 1 -- 172.21.15.42:0/2526351998 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7efeac004250 con 0x55fdee195980 2024-07-26T06:55:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.238+0000 7efeaa7fc640 1 -- 172.21.15.42:0/2526351998 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7efeac029070 con 0x55fdee195980 2024-07-26T06:55:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.238+0000 7efecfa26740 10 client.16002 fetch_fsmap learned FSMap version 37 2024-07-26T06:55:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.238+0000 7efecfa26740 1 -- 172.21.15.42:0/2526351998 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55fdee238b90 con 0x55fdee195980 2024-07-26T06:55:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.238+0000 7efeaa7fc640 1 -- 172.21.15.42:0/2526351998 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efeb4004030 con 0x55fdee195980 2024-07-26T06:55:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.239+0000 7efeaa7fc640 1 -- 172.21.15.42:0/2526351998 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7efeac01d030 con 0x55fdee195980 2024-07-26T06:55:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.239+0000 7efecfa26740 10 client.16002 fetch_fsmap finished waiting for FSMap version 37 2024-07-26T06:55:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.239+0000 7efecfa26740 10 client.16002 resolve_mds: validated gid 24620 aka daemon mds.b 2024-07-26T06:55:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.239+0000 7efecfa26740 20 client.16002 populate_metadata read hostname 'smithi042' 2024-07-26T06:55:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.239+0000 7efecfa26740 1 --2- 172.21.15.42:0/2526351998 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fdee17d1c0 0x55fdee582000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.239+0000 7efecfa26740 4 client.16002 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:55:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.239+0000 7efecfa26740 1 -- 172.21.15.42:0/2526351998 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdee4db6d0 con 0x55fdee17d1c0 2024-07-26T06:55:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.239+0000 7efec2ffd640 1 --2- 172.21.15.42:0/2526351998 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fdee17d1c0 0x55fdee582000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.239+0000 7efec2ffd640 1 --2- 172.21.15.42:0/2526351998 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fdee17d1c0 0x55fdee582000 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b830340ee8958375 server_cookie=2c8e4a0f3107c656 in_seq=0 out_seq=0 2024-07-26T06:55:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.239+0000 7efeaa7fc640 10 client.16002 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.240+0000 7efeaa7fc640 1 -- 172.21.15.42:0/2526351998 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fdee4db6d0 con 0x55fdee17d1c0 2024-07-26T06:55:23.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.240+0000 7efeaa7fc640 10 client.16002 handle_command_reply: tid=0 2024-07-26T06:55:23.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.240+0000 7efecfa26740 1 client.16002 shutdown 2024-07-26T06:55:23.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.240+0000 7efecfa26740 1 -- 172.21.15.42:0/2526351998 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee195980 msgr2=0x55fdee56fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:23.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.240+0000 7efecfa26740 1 --2- 172.21.15.42:0/2526351998 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee195980 0x55fdee56fde0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7efeac00f3c0 tx=0x7efeac009aa0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.240+0000 7efecfa26740 1 -- 172.21.15.42:0/2526351998 shutdown_connections 2024-07-26T06:55:23.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.240+0000 7efecfa26740 1 --2- 172.21.15.42:0/2526351998 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fdee17d1c0 0x55fdee582000 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.241+0000 7efecfa26740 1 --2- 172.21.15.42:0/2526351998 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee1be840 0x55fdee570860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.241+0000 7efecfa26740 1 --2- 172.21.15.42:0/2526351998 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 0x55fdee570320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.241+0000 7efecfa26740 1 --2- 172.21.15.42:0/2526351998 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee195980 0x55fdee56fde0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.241+0000 7efecfa26740 1 -- 172.21.15.42:0/2526351998 >> 172.21.15.42:0/2526351998 conn(0x55fdee1b2990 msgr2=0x55fdee5679d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:23.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.241+0000 7efeaa7fc640 0 client.16002 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:23.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.241+0000 7efecfa26740 1 -- 172.21.15.42:0/2526351998 shutdown_connections 2024-07-26T06:55:23.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.241+0000 7efecfa26740 1 -- 172.21.15.42:0/2526351998 wait complete. 2024-07-26T06:55:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.241+0000 7efecfa26740 20 client.16002 trim_cache size 0 max 16384 2024-07-26T06:55:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.261+0000 7efecfa26740 1 Processor -- start 2024-07-26T06:55:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.261+0000 7efecfa26740 1 -- start start 2024-07-26T06:55:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.261+0000 7efecfa26740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee195980 0x55fdee56b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.261+0000 7efecfa26740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 0x55fdee57ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.261+0000 7efecfa26740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee1be840 0x55fdee5698d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.261+0000 7efecfa26740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7efec41aeef0 con 0x55fdee1be840 2024-07-26T06:55:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.261+0000 7efecfa26740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efec41b0af0 con 0x55fdee19db60 2024-07-26T06:55:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.261+0000 7efecfa26740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7efec41af5c0 con 0x55fdee195980 2024-07-26T06:55:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.262+0000 7efec2ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 0x55fdee57ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.262+0000 7efebbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee1be840 0x55fdee5698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.262+0000 7efec2ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 0x55fdee57ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:56172/0 (socket says 172.21.15.42:56172) 2024-07-26T06:55:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.262+0000 7efec2ffd640 1 -- 172.21.15.42:0/3016000451 learned_addr learned my addr 172.21.15.42:0/3016000451 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.262+0000 7efebbfff640 1 --2- 172.21.15.42:0/3016000451 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee195980 0x55fdee56b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.262+0000 7efebbfff640 1 -- 172.21.15.42:0/3016000451 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee195980 msgr2=0x55fdee56b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.262+0000 7efebbfff640 1 --2- 172.21.15.42:0/3016000451 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee195980 0x55fdee56b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.262+0000 7efebbfff640 1 -- 172.21.15.42:0/3016000451 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 msgr2=0x55fdee57ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.262+0000 7efebbfff640 1 --2- 172.21.15.42:0/3016000451 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 0x55fdee57ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.262+0000 7efebbfff640 1 -- 172.21.15.42:0/3016000451 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdee567080 con 0x55fdee1be840 2024-07-26T06:55:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.262+0000 7efec2ffd640 1 --2- 172.21.15.42:0/3016000451 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 0x55fdee57ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:55:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.262+0000 7efebbfff640 1 --2- 172.21.15.42:0/3016000451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee1be840 0x55fdee5698d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7efeac0114a0 tx=0x7efeac005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=108021fa566d547d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.262+0000 7efeb8ff9640 1 -- 172.21.15.42:0/3016000451 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac00fe00 con 0x55fdee1be840 2024-07-26T06:55:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.262+0000 7efeb8ff9640 1 -- 172.21.15.42:0/3016000451 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efeac004250 con 0x55fdee1be840 2024-07-26T06:55:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.262+0000 7efeb8ff9640 1 -- 172.21.15.42:0/3016000451 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac005520 con 0x55fdee1be840 2024-07-26T06:55:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.263+0000 7efecfa26740 1 -- 172.21.15.42:0/3016000451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee1be840 msgr2=0x55fdee5698d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.263+0000 7efecfa26740 1 --2- 172.21.15.42:0/3016000451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee1be840 0x55fdee5698d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7efeac0114a0 tx=0x7efeac005f90 comp rx=0 tx=0).stop 2024-07-26T06:55:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.263+0000 7efecfa26740 1 -- 172.21.15.42:0/3016000451 shutdown_connections 2024-07-26T06:55:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.263+0000 7efecfa26740 1 --2- 172.21.15.42:0/3016000451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee1be840 0x55fdee5698d0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.263+0000 7efecfa26740 1 --2- 172.21.15.42:0/3016000451 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 0x55fdee57ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.263+0000 7efecfa26740 1 --2- 172.21.15.42:0/3016000451 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee195980 0x55fdee56b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.263+0000 7efecfa26740 1 -- 172.21.15.42:0/3016000451 >> 172.21.15.42:0/3016000451 conn(0x55fdee1b2990 msgr2=0x55fdee566110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.264+0000 7efecfa26740 1 -- 172.21.15.42:0/3016000451 shutdown_connections 2024-07-26T06:55:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.264+0000 7efecfa26740 1 -- 172.21.15.42:0/3016000451 wait complete. 2024-07-26T06:55:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.264+0000 7efecfa26740 1 -- start start 2024-07-26T06:55:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.264+0000 7efecfa26740 1 Processor -- start 2024-07-26T06:55:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.264+0000 7efecfa26740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee195980 0x55fdee573830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.265+0000 7efecfa26740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 0x55fdee573d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.265+0000 7efecfa26740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee1be840 0x55fdee5e87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.265+0000 7efecfa26740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7efec41b0420 con 0x55fdee1be840 2024-07-26T06:55:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.265+0000 7efecfa26740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efec41afc90 con 0x55fdee19db60 2024-07-26T06:55:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.265+0000 7efecfa26740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7efec41aced0 con 0x55fdee195980 2024-07-26T06:55:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.265+0000 7efecfa26740 10 client.0 _ll_register_callbacks cb 0x7efeac011d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:55:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.265+0000 7efebbfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee195980 0x55fdee573830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.265+0000 7efebbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee1be840 0x55fdee5e87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.266+0000 7efec2ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 0x55fdee573d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.266+0000 7efebbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee1be840 0x55fdee5e87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48218/0 (socket says 172.21.15.42:48218) 2024-07-26T06:55:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.266+0000 7efec2ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 0x55fdee573d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.42:56178/0 (socket says 172.21.15.42:56178) 2024-07-26T06:55:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.266+0000 7efebbfff640 1 -- 172.21.15.42:0/4261871550 learned_addr learned my addr 172.21.15.42:0/4261871550 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.266+0000 7efec2ffd640 1 -- 172.21.15.42:0/4261871550 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee195980 msgr2=0x55fdee573830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.266+0000 7efec2ffd640 1 --2- 172.21.15.42:0/4261871550 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee195980 0x55fdee573830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.266+0000 7efec2ffd640 1 -- 172.21.15.42:0/4261871550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee1be840 msgr2=0x55fdee5e87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.266+0000 7efec2ffd640 1 --2- 172.21.15.42:0/4261871550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee1be840 0x55fdee5e87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.266+0000 7efec2ffd640 1 -- 172.21.15.42:0/4261871550 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdee5e8d00 con 0x55fdee19db60 2024-07-26T06:55:23.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.266+0000 7efec2ffd640 1 --2- 172.21.15.42:0/4261871550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 0x55fdee573d70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efeb4009850 tx=0x7efeb4032e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b5470769bf7d246 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:23.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.266+0000 7efeaa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-07-26T06:55:23.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.267+0000 7efeaa7fc640 1 -- 172.21.15.42:0/4261871550 <== mon.2 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb4035040 con 0x55fdee19db60 2024-07-26T06:55:23.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.267+0000 7efeaa7fc640 1 -- 172.21.15.42:0/4261871550 <== mon.2 v2:172.21.15.163:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7efeb40314a0 con 0x55fdee19db60 2024-07-26T06:55:23.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.267+0000 7efecfa26740 1 -- 172.21.15.42:0/4261871550 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efeac004250 con 0x55fdee19db60 2024-07-26T06:55:23.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.267+0000 7efeaa7fc640 1 -- 172.21.15.42:0/4261871550 <== mon.2 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7efeb4040040 con 0x55fdee19db60 2024-07-26T06:55:23.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.267+0000 7efecfa26740 10 client.25589 fetch_fsmap learned FSMap version 37 2024-07-26T06:55:23.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.267+0000 7efecfa26740 1 -- 172.21.15.42:0/4261871550 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55fdee5e9f20 con 0x55fdee19db60 2024-07-26T06:55:23.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.267+0000 7efeaa7fc640 1 -- 172.21.15.42:0/4261871550 <== mon.2 v2:172.21.15.163:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efeac004250 con 0x55fdee19db60 2024-07-26T06:55:23.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.267+0000 7efeaa7fc640 1 -- 172.21.15.42:0/4261871550 <== mon.2 v2:172.21.15.163:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7efeb4056070 con 0x55fdee19db60 2024-07-26T06:55:23.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.267+0000 7efecfa26740 10 client.25589 fetch_fsmap finished waiting for FSMap version 37 2024-07-26T06:55:23.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.267+0000 7efecfa26740 10 client.25589 resolve_mds: validated gid 24620 aka daemon mds.b 2024-07-26T06:55:23.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.267+0000 7efecfa26740 20 client.25589 populate_metadata read hostname 'smithi042' 2024-07-26T06:55:23.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.267+0000 7efecfa26740 1 --2- 172.21.15.42:0/4261871550 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fdee152570 0x55fdee5ec3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:23.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.267+0000 7efecfa26740 4 client.25589 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}] 2024-07-26T06:55:23.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.267+0000 7efecfa26740 1 -- 172.21.15.42:0/4261871550 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 130.0, "await_for": 45.0}) -- 0x55fdee4d7170 con 0x55fdee152570 2024-07-26T06:55:23.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.268+0000 7efebbfff640 1 --2- 172.21.15.42:0/4261871550 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fdee152570 0x55fdee5ec3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:23.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.268+0000 7efeaa7fc640 10 client.25589 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:23.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.268+0000 7efebbfff640 1 --2- 172.21.15.42:0/4261871550 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fdee152570 0x55fdee5ec3f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b63c69d897a32a33 server_cookie=b17a2eaf53381078 in_seq=0 out_seq=0 2024-07-26T06:55:23.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.936+0000 7efeaa7fc640 1 -- 172.21.15.42:0/4261871550 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55fdee4d7170 con 0x55fdee152570 2024-07-26T06:55:23.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.936+0000 7efeaa7fc640 10 client.25589 handle_command_reply: tid=0 2024-07-26T06:55:23.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.936+0000 7efecfa26740 1 client.25589 shutdown 2024-07-26T06:55:23.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.936+0000 7efecfa26740 1 -- 172.21.15.42:0/4261871550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 msgr2=0x55fdee573d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:23.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.936+0000 7efecfa26740 1 --2- 172.21.15.42:0/4261871550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 0x55fdee573d70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efeb4009850 tx=0x7efeb4032e10 comp rx=0 tx=0).stop 2024-07-26T06:55:23.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.936+0000 7efecfa26740 1 -- 172.21.15.42:0/4261871550 shutdown_connections 2024-07-26T06:55:23.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.936+0000 7efecfa26740 1 --2- 172.21.15.42:0/4261871550 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55fdee152570 0x55fdee5ec3f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.936+0000 7efecfa26740 1 --2- 172.21.15.42:0/4261871550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55fdee1be840 0x55fdee5e87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.936+0000 7efecfa26740 1 --2- 172.21.15.42:0/4261871550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fdee19db60 0x55fdee573d70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.936+0000 7efeaa7fc640 0 client.25589 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:23.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.936+0000 7efecfa26740 1 --2- 172.21.15.42:0/4261871550 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55fdee195980 0x55fdee573830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.936+0000 7efecfa26740 1 -- 172.21.15.42:0/4261871550 >> 172.21.15.42:0/4261871550 conn(0x55fdee1b2990 msgr2=0x55fdee57ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:23.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.936+0000 7efecfa26740 1 -- 172.21.15.42:0/4261871550 shutdown_connections 2024-07-26T06:55:23.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.936+0000 7efecfa26740 1 -- 172.21.15.42:0/4261871550 wait complete. 2024-07-26T06:55:23.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.936+0000 7efecfa26740 20 client.25589 trim_cache size 0 max 16384 2024-07-26T06:55:23.939 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:55:23.939 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 37, 2024-07-26T06:55:23.939 INFO:teuthology.orchestra.run.smithi042.stdout: "leader": 24620, 2024-07-26T06:55:23.939 INFO:teuthology.orchestra.run.smithi042.stdout: "set_version": 14, 2024-07-26T06:55:23.939 INFO:teuthology.orchestra.run.smithi042.stdout: "sets": { 2024-07-26T06:55:23.939 INFO:teuthology.orchestra.run.smithi042.stdout: "84ce04c6": { 2024-07-26T06:55:23.939 INFO:teuthology.orchestra.run.smithi042.stdout: "version": 14, 2024-07-26T06:55:23.939 INFO:teuthology.orchestra.run.smithi042.stdout: "age_ref": 0.0, 2024-07-26T06:55:23.939 INFO:teuthology.orchestra.run.smithi042.stdout: "state": { 2024-07-26T06:55:23.939 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "QUIESCED", 2024-07-26T06:55:23.939 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 0.0 2024-07-26T06:55:23.940 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-07-26T06:55:23.940 INFO:teuthology.orchestra.run.smithi042.stdout: "timeout": 90.0, 2024-07-26T06:55:23.940 INFO:teuthology.orchestra.run.smithi042.stdout: "expiration": 130.0, 2024-07-26T06:55:23.940 INFO:teuthology.orchestra.run.smithi042.stdout: "members": { 2024-07-26T06:55:23.940 INFO:teuthology.orchestra.run.smithi042.stdout: "file:/": { 2024-07-26T06:55:23.940 INFO:teuthology.orchestra.run.smithi042.stdout: "excluded": false, 2024-07-26T06:55:23.940 INFO:teuthology.orchestra.run.smithi042.stdout: "state": { 2024-07-26T06:55:23.940 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "QUIESCED", 2024-07-26T06:55:23.940 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 0.0 2024-07-26T06:55:23.940 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:55:23.940 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:55:23.940 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:55:23.941 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:55:23.941 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:55:23.941 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:55:23.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.937+0000 7efec94a2640 1 -- 172.21.15.42:0/3782781323 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7efeb00b3c50 msgr2=0x7efeb00b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:23.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.937+0000 7efec94a2640 1 --2- 172.21.15.42:0/3782781323 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7efeb00b3c50 0x7efeb00b6160 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efebc0e1f30 tx=0x7efebc00cee0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.937+0000 7efec94a2640 1 -- 172.21.15.42:0/3782781323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efec4108740 msgr2=0x7efec419f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:23.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.937+0000 7efec94a2640 1 --2- 172.21.15.42:0/3782781323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efec4108740 0x7efec419f740 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7efebc00eb10 tx=0x7efebc00c2d0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.937+0000 7efec94a2640 1 -- 172.21.15.42:0/3782781323 shutdown_connections 2024-07-26T06:55:23.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.937+0000 7efec94a2640 1 --2- 172.21.15.42:0/3782781323 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7efeb00b3c50 0x7efeb00b6160 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.937+0000 7efec94a2640 1 --2- 172.21.15.42:0/3782781323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efec4108740 0x7efec419f740 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.937+0000 7efec94a2640 1 --2- 172.21.15.42:0/3782781323 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efec4107df0 0x7efec419a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.937+0000 7efec94a2640 1 --2- 172.21.15.42:0/3782781323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efec41024b0 0x7efec4199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:23.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.937+0000 7efec94a2640 1 -- 172.21.15.42:0/3782781323 >> 172.21.15.42:0/3782781323 conn(0x7efec4079190 msgr2=0x7efec41013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:23.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.937+0000 7efec94a2640 1 -- 172.21.15.42:0/3782781323 shutdown_connections 2024-07-26T06:55:23.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:23.937+0000 7efec94a2640 1 -- 172.21.15.42:0/3782781323 wait complete. 2024-07-26T06:55:24.823 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '84ce04c6', quiesce took 7.9 seconds. Will release after: 2.0999999999999996 2024-07-26T06:55:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:24 smithi042 ceph-mon[32775]: pgmap v552: 97 pgs: 97 active+clean; 67 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 928 op/s 2024-07-26T06:55:25.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:24 smithi164 ceph-mon[36970]: pgmap v552: 97 pgs: 97 active+clean; 67 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 928 op/s 2024-07-26T06:55:25.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:24 smithi163 ceph-mon[36873]: pgmap v552: 97 pgs: 97 active+clean; 67 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 928 op/s 2024-07-26T06:55:26.925 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:55:27.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:26 smithi042 ceph-mon[32775]: pgmap v553: 97 pgs: 97 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 2.5 MiB/s wr, 983 op/s 2024-07-26T06:55:27.207 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:55:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:26 smithi164 ceph-mon[36970]: pgmap v553: 97 pgs: 97 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 2.5 MiB/s wr, 983 op/s 2024-07-26T06:55:27.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:26 smithi163 ceph-mon[36873]: pgmap v553: 97 pgs: 97 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 2.5 MiB/s wr, 983 op/s 2024-07-26T06:55:28.960 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:28 smithi042 ceph-mon[32775]: pgmap v554: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.2 MiB/s wr, 873 op/s 2024-07-26T06:55:29.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.197+0000 7fe06e1e3640 1 -- 172.21.15.42:0/716985491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068109820 msgr2=0x7fe068110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.197+0000 7fe06e1e3640 1 --2- 172.21.15.42:0/716985491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068109820 0x7fe068110050 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fe060009ca0 tx=0x7fe06002a8c0 comp rx=0 tx=0).stop 2024-07-26T06:55:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.197+0000 7fe06e1e3640 1 -- 172.21.15.42:0/716985491 shutdown_connections 2024-07-26T06:55:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.197+0000 7fe06e1e3640 1 --2- 172.21.15.42:0/716985491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068109820 0x7fe068110050 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.197+0000 7fe06e1e3640 1 --2- 172.21.15.42:0/716985491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe068108ed0 0x7fe0681092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.197+0000 7fe06e1e3640 1 --2- 172.21.15.42:0/716985491 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe068103590 0x7fe0681039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.197+0000 7fe06e1e3640 1 -- 172.21.15.42:0/716985491 >> 172.21.15.42:0/716985491 conn(0x7fe06807b280 msgr2=0x7fe06807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.197+0000 7fe06e1e3640 1 -- 172.21.15.42:0/716985491 shutdown_connections 2024-07-26T06:55:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.197+0000 7fe06e1e3640 1 -- 172.21.15.42:0/716985491 wait complete. 2024-07-26T06:55:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.198+0000 7fe06e1e3640 1 Processor -- start 2024-07-26T06:55:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.198+0000 7fe06e1e3640 1 -- start start 2024-07-26T06:55:29.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.198+0000 7fe06e1e3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe068103590 0x7fe068199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:29.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.199+0000 7fe06e1e3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe068108ed0 0x7fe06819a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:29.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.199+0000 7fe06e1e3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068109820 0x7fe06819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:29.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.199+0000 7fe06e1e3640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe0681134f0 con 0x7fe068109820 2024-07-26T06:55:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.199+0000 7fe06e1e3640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe068113370 con 0x7fe068103590 2024-07-26T06:55:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.199+0000 7fe06e1e3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe068113670 con 0x7fe068108ed0 2024-07-26T06:55:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.199+0000 7fe067fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068109820 0x7fe06819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.199+0000 7fe067fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068109820 0x7fe06819f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45730/0 (socket says 172.21.15.42:45730) 2024-07-26T06:55:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.199+0000 7fe067fff640 1 -- 172.21.15.42:0/2487199429 learned_addr learned my addr 172.21.15.42:0/2487199429 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.199+0000 7fe05ffff640 1 --2- 172.21.15.42:0/2487199429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe068108ed0 0x7fe06819a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.199+0000 7fe0677fe640 1 --2- 172.21.15.42:0/2487199429 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe068103590 0x7fe068199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.199+0000 7fe067fff640 1 -- 172.21.15.42:0/2487199429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe068108ed0 msgr2=0x7fe06819a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.199+0000 7fe067fff640 1 --2- 172.21.15.42:0/2487199429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe068108ed0 0x7fe06819a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.199+0000 7fe067fff640 1 -- 172.21.15.42:0/2487199429 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe068103590 msgr2=0x7fe068199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.199+0000 7fe067fff640 1 --2- 172.21.15.42:0/2487199429 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe068103590 0x7fe068199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.199+0000 7fe067fff640 1 -- 172.21.15.42:0/2487199429 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe06819fe10 con 0x7fe068109820 2024-07-26T06:55:29.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.199+0000 7fe067fff640 1 --2- 172.21.15.42:0/2487199429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068109820 0x7fe06819f6e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe060005cc0 tx=0x7fe06002f380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:29.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.200+0000 7fe0657fa640 1 -- 172.21.15.42:0/2487199429 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe060003bf0 con 0x7fe068109820 2024-07-26T06:55:29.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.200+0000 7fe06e1e3640 1 -- 172.21.15.42:0/2487199429 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0681a00a0 con 0x7fe068109820 2024-07-26T06:55:29.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.200+0000 7fe0657fa640 1 -- 172.21.15.42:0/2487199429 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe060003d90 con 0x7fe068109820 2024-07-26T06:55:29.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.200+0000 7fe0657fa640 1 -- 172.21.15.42:0/2487199429 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06002fbc0 con 0x7fe068109820 2024-07-26T06:55:29.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.200+0000 7fe06e1e3640 1 -- 172.21.15.42:0/2487199429 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0681a6860 con 0x7fe068109820 2024-07-26T06:55:29.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.201+0000 7fe06e1e3640 1 -- 172.21.15.42:0/2487199429 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0681039a0 con 0x7fe068109820 2024-07-26T06:55:29.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.202+0000 7fe0657fa640 1 -- 172.21.15.42:0/2487199429 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fe06002fd60 con 0x7fe068109820 2024-07-26T06:55:29.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.203+0000 7fe0657fa640 1 --2- 172.21.15.42:0/2487199429 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe0540b3bd0 0x7fe0540b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:29.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.203+0000 7fe0677fe640 1 --2- 172.21.15.42:0/2487199429 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe0540b3bd0 0x7fe0540b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:29.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.203+0000 7fe0657fa640 1 -- 172.21.15.42:0/2487199429 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fe0601054b0 con 0x7fe068109820 2024-07-26T06:55:29.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.204+0000 7fe0677fe640 1 --2- 172.21.15.42:0/2487199429 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe0540b3bd0 0x7fe0540b60e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe0580058c0 tx=0x7fe05800a250 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:29.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.206+0000 7fe0657fa640 1 -- 172.21.15.42:0/2487199429 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe0600cca80 con 0x7fe068109820 2024-07-26T06:55:29.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:28 smithi164 ceph-mon[36970]: pgmap v554: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.2 MiB/s wr, 873 op/s 2024-07-26T06:55:29.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:28 smithi163 ceph-mon[36873]: pgmap v554: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.2 MiB/s wr, 873 op/s 2024-07-26T06:55:29.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.431+0000 7fe06e1e3640 1 -- 172.21.15.42:0/2487199429 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe06819afd0 con 0x7fe068109820 2024-07-26T06:55:29.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.432+0000 7fe0657fa640 1 -- 172.21.15.42:0/2487199429 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11985 (secure 0 0 0) 0x7fe06819afd0 con 0x7fe068109820 2024-07-26T06:55:29.433 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:55:29.433 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":37,"btime":"2024-07-26T06:54:35:802227+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":37,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:54:35.748994+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25369":{"gid":25369,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/1072602530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1072602530},{"type":"v1","addr":"172.21.15.164:6835","nonce":1072602530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24620,24601,24593,24583,24578]},"id":1}]} 2024-07-26T06:55:29.433 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 37 2024-07-26T06:55:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.434+0000 7fe06e1e3640 1 -- 172.21.15.42:0/2487199429 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe0540b3bd0 msgr2=0x7fe0540b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.434+0000 7fe06e1e3640 1 --2- 172.21.15.42:0/2487199429 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe0540b3bd0 0x7fe0540b60e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe0580058c0 tx=0x7fe05800a250 comp rx=0 tx=0).stop 2024-07-26T06:55:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.434+0000 7fe06e1e3640 1 -- 172.21.15.42:0/2487199429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068109820 msgr2=0x7fe06819f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.434+0000 7fe06e1e3640 1 --2- 172.21.15.42:0/2487199429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068109820 0x7fe06819f6e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe060005cc0 tx=0x7fe06002f380 comp rx=0 tx=0).stop 2024-07-26T06:55:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.434+0000 7fe06e1e3640 1 -- 172.21.15.42:0/2487199429 shutdown_connections 2024-07-26T06:55:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.434+0000 7fe06e1e3640 1 --2- 172.21.15.42:0/2487199429 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fe0540b3bd0 0x7fe0540b60e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.434+0000 7fe06e1e3640 1 --2- 172.21.15.42:0/2487199429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe068109820 0x7fe06819f6e0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:29.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.434+0000 7fe06e1e3640 1 --2- 172.21.15.42:0/2487199429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe068108ed0 0x7fe06819a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:29.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.434+0000 7fe06e1e3640 1 --2- 172.21.15.42:0/2487199429 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe068103590 0x7fe068199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:29.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.434+0000 7fe06e1e3640 1 -- 172.21.15.42:0/2487199429 >> 172.21.15.42:0/2487199429 conn(0x7fe06807b280 msgr2=0x7fe068104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:29.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.434+0000 7fe06e1e3640 1 -- 172.21.15.42:0/2487199429 shutdown_connections 2024-07-26T06:55:29.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:29.434+0000 7fe06e1e3640 1 -- 172.21.15.42:0/2487199429 wait complete. 2024-07-26T06:55:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:29 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2487199429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:55:30.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:29 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2487199429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:55:30.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:29 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2487199429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:55:30.555 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24620 quiesce db --set-id 84ce04c6 --release --await' 2024-07-26T06:55:30.555 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.24620 quiesce db --set-id 84ce04c6 --release --await 2024-07-26T06:55:30.830 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:55:31.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:30 smithi042 ceph-mon[32775]: pgmap v555: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.8 MiB/s wr, 683 op/s 2024-07-26T06:55:31.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:30 smithi164 ceph-mon[36970]: pgmap v555: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.8 MiB/s wr, 683 op/s 2024-07-26T06:55:31.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:30 smithi163 ceph-mon[36873]: pgmap v555: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.8 MiB/s wr, 683 op/s 2024-07-26T06:55:31.496 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.495+0000 7f895a3c4640 1 -- 172.21.15.42:0/3573054613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8954103080 msgr2=0x7f8954103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:31.496 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.495+0000 7f895a3c4640 1 --2- 172.21.15.42:0/3573054613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8954103080 0x7f8954103490 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f8948009900 tx=0x7f894802a410 comp rx=0 tx=0).stop 2024-07-26T06:55:31.496 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.495+0000 7f895a3c4640 1 -- 172.21.15.42:0/3573054613 shutdown_connections 2024-07-26T06:55:31.496 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.495+0000 7f895a3c4640 1 --2- 172.21.15.42:0/3573054613 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8954103a20 0x7f895410a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.495+0000 7f895a3c4640 1 --2- 172.21.15.42:0/3573054613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8954103080 0x7f8954103490 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.495+0000 7f895a3c4640 1 --2- 172.21.15.42:0/3573054613 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8954102000 0x7f8954102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.495+0000 7f895a3c4640 1 -- 172.21.15.42:0/3573054613 >> 172.21.15.42:0/3573054613 conn(0x7f89540fd960 msgr2=0x7f89540ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:31.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.495+0000 7f895a3c4640 1 -- 172.21.15.42:0/3573054613 shutdown_connections 2024-07-26T06:55:31.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.495+0000 7f895a3c4640 1 -- 172.21.15.42:0/3573054613 wait complete. 2024-07-26T06:55:31.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.496+0000 7f895a3c4640 1 Processor -- start 2024-07-26T06:55:31.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.496+0000 7f895a3c4640 1 -- start start 2024-07-26T06:55:31.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.496+0000 7f895a3c4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8954102000 0x7f895406bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.496+0000 7f895a3c4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8954103080 0x7f895419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.496+0000 7f895a3c4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8954103a20 0x7f89541a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.496+0000 7f895a3c4640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f895410dc80 con 0x7f8954102000 2024-07-26T06:55:31.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.496+0000 7f895a3c4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f895410db00 con 0x7f8954103080 2024-07-26T06:55:31.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.496+0000 7f895a3c4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f895410de00 con 0x7f8954103a20 2024-07-26T06:55:31.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.496+0000 7f8953fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8954102000 0x7f895406bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:31.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.496+0000 7f8953fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8954102000 0x7f895406bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45750/0 (socket says 172.21.15.42:45750) 2024-07-26T06:55:31.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.496+0000 7f8953fff640 1 -- 172.21.15.42:0/2166920291 learned_addr learned my addr 172.21.15.42:0/2166920291 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:31.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.496+0000 7f8958961640 1 --2- 172.21.15.42:0/2166920291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8954103a20 0x7f89541a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:31.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.497+0000 7f89537fe640 1 --2- 172.21.15.42:0/2166920291 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8954103080 0x7f895419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:31.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.497+0000 7f8953fff640 1 -- 172.21.15.42:0/2166920291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8954103a20 msgr2=0x7f89541a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:31.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.497+0000 7f8953fff640 1 --2- 172.21.15.42:0/2166920291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8954103a20 0x7f89541a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.497+0000 7f8953fff640 1 -- 172.21.15.42:0/2166920291 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8954103080 msgr2=0x7f895419e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:31.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.497+0000 7f8953fff640 1 --2- 172.21.15.42:0/2166920291 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8954103080 0x7f895419e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.497+0000 7f8953fff640 1 -- 172.21.15.42:0/2166920291 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89541a1400 con 0x7f8954102000 2024-07-26T06:55:31.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.497+0000 7f8958961640 1 --2- 172.21.15.42:0/2166920291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8954103a20 0x7f89541a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:55:31.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.497+0000 7f89537fe640 1 --2- 172.21.15.42:0/2166920291 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8954103080 0x7f895419e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:55:31.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.497+0000 7f8953fff640 1 --2- 172.21.15.42:0/2166920291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8954102000 0x7f895406bc60 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f894400d950 tx=0x7f894400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:31.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.497+0000 7f89517fa640 1 -- 172.21.15.42:0/2166920291 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944017070 con 0x7f8954102000 2024-07-26T06:55:31.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.497+0000 7f895a3c4640 1 -- 172.21.15.42:0/2166920291 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89541a16f0 con 0x7f8954102000 2024-07-26T06:55:31.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.497+0000 7f89517fa640 1 -- 172.21.15.42:0/2166920291 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8944009c30 con 0x7f8954102000 2024-07-26T06:55:31.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.497+0000 7f89517fa640 1 -- 172.21.15.42:0/2166920291 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894400f620 con 0x7f8954102000 2024-07-26T06:55:31.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.497+0000 7f895a3c4640 1 -- 172.21.15.42:0/2166920291 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89541a1c90 con 0x7f8954102000 2024-07-26T06:55:31.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f89613b5740 1 Processor -- start 2024-07-26T06:55:31.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f89613b5740 1 -- start start 2024-07-26T06:55:31.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f89517fa640 1 -- 172.21.15.42:0/2166920291 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f8944010460 con 0x7f8954102000 2024-07-26T06:55:31.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f89613b5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd7e1b0 0x5627ee123180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f89613b5740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd552f0 0x5627ee1237f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f89613b5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd5d4d0 0x5627ee124340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f89613b5740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5627ee0963b0 con 0x5627edd552f0 2024-07-26T06:55:31.515 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f89613b5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5627ee0858b0 con 0x5627edd5d4d0 2024-07-26T06:55:31.515 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f89613b5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5627ee0898f0 con 0x5627edd7e1b0 2024-07-26T06:55:31.515 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f8953fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd552f0 0x5627ee1237f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:31.515 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f8953fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd552f0 0x5627ee1237f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45752/0 (socket says 172.21.15.42:45752) 2024-07-26T06:55:31.515 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f8953fff640 1 -- 172.21.15.42:0/705871627 learned_addr learned my addr 172.21.15.42:0/705871627 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:31.515 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f8958961640 1 --2- 172.21.15.42:0/705871627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd7e1b0 0x5627ee123180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:31.515 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f89537fe640 1 --2- 172.21.15.42:0/705871627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd5d4d0 0x5627ee124340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:31.515 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f8953fff640 1 -- 172.21.15.42:0/705871627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd7e1b0 msgr2=0x5627ee123180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:31.515 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f8953fff640 1 --2- 172.21.15.42:0/705871627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd7e1b0 0x5627ee123180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.515 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f8953fff640 1 -- 172.21.15.42:0/705871627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd5d4d0 msgr2=0x5627ee124340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:31.515 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f8953fff640 1 --2- 172.21.15.42:0/705871627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd5d4d0 0x5627ee124340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.515 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.500+0000 7f8953fff640 1 -- 172.21.15.42:0/705871627 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627ee02c9a0 con 0x5627edd552f0 2024-07-26T06:55:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.501+0000 7f8953fff640 1 --2- 172.21.15.42:0/705871627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd552f0 0x5627ee1237f0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f89440e9c10 tx=0x7f89440eb690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f9f5e63916c1c5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.501+0000 7f89517fa640 1 --2- 172.21.15.42:0/2166920291 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f893c0b3bd0 0x7f893c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.501+0000 7f893a7fc640 1 -- 172.21.15.42:0/705871627 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89440f2070 con 0x5627edd552f0 2024-07-26T06:55:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.501+0000 7f8958961640 1 --2- 172.21.15.42:0/2166920291 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f893c0b3bd0 0x7f893c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.501+0000 7f89517fa640 1 -- 172.21.15.42:0/2166920291 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f89440e3900 con 0x7f8954102000 2024-07-26T06:55:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.501+0000 7f893a7fc640 1 -- 172.21.15.42:0/705871627 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89440aa340 con 0x5627edd552f0 2024-07-26T06:55:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.501+0000 7f893a7fc640 1 -- 172.21.15.42:0/705871627 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89440aa600 con 0x5627edd552f0 2024-07-26T06:55:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.502+0000 7f89613b5740 1 -- 172.21.15.42:0/705871627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd552f0 msgr2=0x5627ee1237f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.502+0000 7f8958961640 1 --2- 172.21.15.42:0/2166920291 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f893c0b3bd0 0x7f893c0b60e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f894c005fd0 tx=0x7f894c009580 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.502+0000 7f89613b5740 1 --2- 172.21.15.42:0/705871627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd552f0 0x5627ee1237f0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f89440e9c10 tx=0x7f89440eb690 comp rx=0 tx=0).stop 2024-07-26T06:55:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.502+0000 7f89613b5740 1 -- 172.21.15.42:0/705871627 shutdown_connections 2024-07-26T06:55:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.502+0000 7f89613b5740 1 --2- 172.21.15.42:0/705871627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd5d4d0 0x5627ee124340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.502+0000 7f89613b5740 1 --2- 172.21.15.42:0/705871627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd552f0 0x5627ee1237f0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.502+0000 7f89613b5740 1 --2- 172.21.15.42:0/705871627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd7e1b0 0x5627ee123180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.502+0000 7f89613b5740 1 -- 172.21.15.42:0/705871627 >> 172.21.15.42:0/705871627 conn(0x5627edd72300 msgr2=0x5627ee11ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.502+0000 7f89613b5740 1 -- 172.21.15.42:0/705871627 shutdown_connections 2024-07-26T06:55:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.502+0000 7f89613b5740 1 -- 172.21.15.42:0/705871627 wait complete. 2024-07-26T06:55:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.502+0000 7f89613b5740 1 -- start start 2024-07-26T06:55:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f89613b5740 1 Processor -- start 2024-07-26T06:55:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f89613b5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd552f0 0x5627ee12cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f89613b5740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd5d4d0 0x5627ee12d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f89613b5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd7e1b0 0x5627ee12dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f89613b5740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5627edfbfab0 con 0x5627edd5d4d0 2024-07-26T06:55:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f89613b5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5627ee00dcf0 con 0x5627edd7e1b0 2024-07-26T06:55:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f89613b5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5627ee124e60 con 0x5627edd552f0 2024-07-26T06:55:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f89613b5740 10 client.0 _ll_register_callbacks cb 0x5627ee07f9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:55:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f89537fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd5d4d0 0x5627ee12d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f89537fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd5d4d0 0x5627ee12d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45754/0 (socket says 172.21.15.42:45754) 2024-07-26T06:55:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f89537fe640 1 -- 172.21.15.42:0/3369815902 learned_addr learned my addr 172.21.15.42:0/3369815902 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f8953fff640 1 --2- 172.21.15.42:0/3369815902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd552f0 0x5627ee12cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f8958961640 1 --2- 172.21.15.42:0/3369815902 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd7e1b0 0x5627ee12dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f89537fe640 1 -- 172.21.15.42:0/3369815902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd552f0 msgr2=0x5627ee12cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f89537fe640 1 --2- 172.21.15.42:0/3369815902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd552f0 0x5627ee12cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f89537fe640 1 -- 172.21.15.42:0/3369815902 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd7e1b0 msgr2=0x5627ee12dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f89537fe640 1 --2- 172.21.15.42:0/3369815902 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd7e1b0 0x5627ee12dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.503+0000 7f89537fe640 1 -- 172.21.15.42:0/3369815902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627edde9790 con 0x5627edd5d4d0 2024-07-26T06:55:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.504+0000 7f89537fe640 1 --2- 172.21.15.42:0/3369815902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd5d4d0 0x5627ee12d290 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f894802ac00 tx=0x7f8948032d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=625a1f060beeb6dd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.504+0000 7f89337fe640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:55:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.504+0000 7f89337fe640 1 -- 172.21.15.42:0/3369815902 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894803f070 con 0x5627edd5d4d0 2024-07-26T06:55:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.504+0000 7f89337fe640 1 -- 172.21.15.42:0/3369815902 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f8948004450 con 0x5627edd5d4d0 2024-07-26T06:55:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.504+0000 7f89613b5740 1 -- 172.21.15.42:0/3369815902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89440aa340 con 0x5627edd5d4d0 2024-07-26T06:55:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.504+0000 7f89337fe640 1 -- 172.21.15.42:0/3369815902 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8948049070 con 0x5627edd5d4d0 2024-07-26T06:55:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.504+0000 7f89613b5740 10 client.16026 fetch_fsmap learned FSMap version 37 2024-07-26T06:55:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.504+0000 7f89613b5740 1 -- 172.21.15.42:0/3369815902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5627ede62620 con 0x5627edd5d4d0 2024-07-26T06:55:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.504+0000 7f89337fe640 1 -- 172.21.15.42:0/3369815902 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89440aa340 con 0x5627edd5d4d0 2024-07-26T06:55:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.504+0000 7f89337fe640 1 -- 172.21.15.42:0/3369815902 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7f8948004bb0 con 0x5627edd5d4d0 2024-07-26T06:55:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.504+0000 7f89613b5740 10 client.16026 fetch_fsmap finished waiting for FSMap version 37 2024-07-26T06:55:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.504+0000 7f89613b5740 10 client.16026 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:55:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.504+0000 7f89613b5740 20 client.16026 populate_metadata read hostname 'smithi042' 2024-07-26T06:55:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.505+0000 7f89613b5740 1 --2- 172.21.15.42:0/3369815902 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5627edd3cd80 0x5627ee13a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.505+0000 7f89613b5740 4 client.16026 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:55:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.505+0000 7f89613b5740 1 -- 172.21.15.42:0/3369815902 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627ee098300 con 0x5627edd3cd80 2024-07-26T06:55:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.505+0000 7f8953fff640 1 --2- 172.21.15.42:0/3369815902 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5627edd3cd80 0x5627ee13a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.505+0000 7f89337fe640 10 client.16026 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.505+0000 7f8953fff640 1 --2- 172.21.15.42:0/3369815902 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5627edd3cd80 0x5627ee13a370 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fb5c3e9adcbc1d2 server_cookie=805cff82b1a3f26f in_seq=0 out_seq=0 2024-07-26T06:55:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.506+0000 7f89337fe640 1 -- 172.21.15.42:0/3369815902 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5627ee098300 con 0x5627edd3cd80 2024-07-26T06:55:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.506+0000 7f89337fe640 10 client.16026 handle_command_reply: tid=0 2024-07-26T06:55:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.506+0000 7f89613b5740 1 client.16026 shutdown 2024-07-26T06:55:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.506+0000 7f89613b5740 1 -- 172.21.15.42:0/3369815902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd5d4d0 msgr2=0x5627ee12d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.506+0000 7f89613b5740 1 --2- 172.21.15.42:0/3369815902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd5d4d0 0x5627ee12d290 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f894802ac00 tx=0x7f8948032d90 comp rx=0 tx=0).stop 2024-07-26T06:55:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.506+0000 7f89613b5740 1 -- 172.21.15.42:0/3369815902 shutdown_connections 2024-07-26T06:55:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.506+0000 7f89613b5740 1 --2- 172.21.15.42:0/3369815902 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5627edd3cd80 0x5627ee13a370 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.506+0000 7f89613b5740 1 --2- 172.21.15.42:0/3369815902 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd7e1b0 0x5627ee12dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.506+0000 7f89613b5740 1 --2- 172.21.15.42:0/3369815902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd5d4d0 0x5627ee12d290 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.506+0000 7f89613b5740 1 --2- 172.21.15.42:0/3369815902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd552f0 0x5627ee12cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.506+0000 7f89337fe640 0 client.16026 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.506+0000 7f89613b5740 1 -- 172.21.15.42:0/3369815902 >> 172.21.15.42:0/3369815902 conn(0x5627edd72300 msgr2=0x5627ee11d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.506+0000 7f89613b5740 1 -- 172.21.15.42:0/3369815902 shutdown_connections 2024-07-26T06:55:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.506+0000 7f89613b5740 1 -- 172.21.15.42:0/3369815902 wait complete. 2024-07-26T06:55:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.507+0000 7f89613b5740 20 client.16026 trim_cache size 0 max 16384 2024-07-26T06:55:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.522+0000 7f89613b5740 1 Processor -- start 2024-07-26T06:55:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.522+0000 7f89613b5740 1 -- start start 2024-07-26T06:55:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.522+0000 7f89613b5740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd552f0 0x5627ee12bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.522+0000 7f89613b5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd5d4d0 0x5627ee124010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.522+0000 7f89613b5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd7e1b0 0x5627ee12b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.522+0000 7f89613b5740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f89541aaaa0 con 0x5627edd552f0 2024-07-26T06:55:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.522+0000 7f89613b5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f89541ac6a0 con 0x5627edd7e1b0 2024-07-26T06:55:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.522+0000 7f89613b5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f89541ab170 con 0x5627edd5d4d0 2024-07-26T06:55:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.522+0000 7f8953fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd552f0 0x5627ee12bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.522+0000 7f8953fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd552f0 0x5627ee12bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45756/0 (socket says 172.21.15.42:45756) 2024-07-26T06:55:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.522+0000 7f8953fff640 1 -- 172.21.15.42:0/2221736515 learned_addr learned my addr 172.21.15.42:0/2221736515 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.522+0000 7f8958961640 1 --2- 172.21.15.42:0/2221736515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd7e1b0 0x5627ee12b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.522+0000 7f8953fff640 1 -- 172.21.15.42:0/2221736515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd5d4d0 msgr2=0x5627ee124010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.522+0000 7f8953fff640 1 --2- 172.21.15.42:0/2221736515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd5d4d0 0x5627ee124010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.522+0000 7f8953fff640 1 -- 172.21.15.42:0/2221736515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd7e1b0 msgr2=0x5627ee12b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.522+0000 7f8953fff640 1 --2- 172.21.15.42:0/2221736515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd7e1b0 0x5627ee12b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.522+0000 7f8953fff640 1 -- 172.21.15.42:0/2221736515 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627ee136ba0 con 0x5627edd552f0 2024-07-26T06:55:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.523+0000 7f8953fff640 1 --2- 172.21.15.42:0/2221736515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd552f0 0x5627ee12bdd0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f89440ebb90 tx=0x7f8944005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f786fe6fa13c6ac4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.523+0000 7f8939ffb640 1 -- 172.21.15.42:0/2221736515 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944020070 con 0x5627edd552f0 2024-07-26T06:55:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.523+0000 7f8939ffb640 1 -- 172.21.15.42:0/2221736515 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89440e9320 con 0x5627edd552f0 2024-07-26T06:55:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.523+0000 7f8939ffb640 1 -- 172.21.15.42:0/2221736515 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89440f2040 con 0x5627edd552f0 2024-07-26T06:55:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.523+0000 7f89613b5740 1 -- 172.21.15.42:0/2221736515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd552f0 msgr2=0x5627ee12bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.523+0000 7f89613b5740 1 --2- 172.21.15.42:0/2221736515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd552f0 0x5627ee12bdd0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f89440ebb90 tx=0x7f8944005880 comp rx=0 tx=0).stop 2024-07-26T06:55:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.524+0000 7f89613b5740 1 -- 172.21.15.42:0/2221736515 shutdown_connections 2024-07-26T06:55:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.524+0000 7f89613b5740 1 --2- 172.21.15.42:0/2221736515 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd7e1b0 0x5627ee12b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.524+0000 7f89613b5740 1 --2- 172.21.15.42:0/2221736515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd5d4d0 0x5627ee124010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.524+0000 7f89613b5740 1 --2- 172.21.15.42:0/2221736515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd552f0 0x5627ee12bdd0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.524+0000 7f89613b5740 1 -- 172.21.15.42:0/2221736515 >> 172.21.15.42:0/2221736515 conn(0x5627edd72300 msgr2=0x5627ee122e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.524+0000 7f89613b5740 1 -- 172.21.15.42:0/2221736515 shutdown_connections 2024-07-26T06:55:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.524+0000 7f89613b5740 1 -- 172.21.15.42:0/2221736515 wait complete. 2024-07-26T06:55:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.524+0000 7f89613b5740 1 -- start start 2024-07-26T06:55:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.524+0000 7f89613b5740 1 Processor -- start 2024-07-26T06:55:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.524+0000 7f89613b5740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd552f0 0x5627ee133f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.524+0000 7f89613b5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd5d4d0 0x5627ee1345f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.524+0000 7f89613b5740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd7e1b0 0x5627ee132ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.524+0000 7f89613b5740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f89541abfd0 con 0x5627edd7e1b0 2024-07-26T06:55:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.524+0000 7f89613b5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f89541ab840 con 0x5627edd5d4d0 2024-07-26T06:55:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.524+0000 7f89613b5740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f89541a8a80 con 0x5627edd552f0 2024-07-26T06:55:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.524+0000 7f89613b5740 10 client.0 _ll_register_callbacks cb 0x7f8948009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:55:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.525+0000 7f8958961640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd7e1b0 0x5627ee132ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:31.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.525+0000 7f8958961640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd7e1b0 0x5627ee132ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45768/0 (socket says 172.21.15.42:45768) 2024-07-26T06:55:31.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.525+0000 7f8958961640 1 -- 172.21.15.42:0/1488189110 learned_addr learned my addr 172.21.15.42:0/1488189110 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:31.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.525+0000 7f8953fff640 1 --2- 172.21.15.42:0/1488189110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd552f0 0x5627ee133f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:31.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.525+0000 7f89537fe640 1 --2- 172.21.15.42:0/1488189110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd5d4d0 0x5627ee1345f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:31.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.525+0000 7f8958961640 1 -- 172.21.15.42:0/1488189110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd552f0 msgr2=0x5627ee133f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:31.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.525+0000 7f8958961640 1 --2- 172.21.15.42:0/1488189110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd552f0 0x5627ee133f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.525+0000 7f8958961640 1 -- 172.21.15.42:0/1488189110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd5d4d0 msgr2=0x5627ee1345f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:31.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.525+0000 7f8958961640 1 --2- 172.21.15.42:0/1488189110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd5d4d0 0x5627ee1345f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.525+0000 7f8958961640 1 -- 172.21.15.42:0/1488189110 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627ee133040 con 0x5627edd7e1b0 2024-07-26T06:55:31.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.525+0000 7f8958961640 1 --2- 172.21.15.42:0/1488189110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd7e1b0 0x5627ee132ab0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f894c01e9e0 tx=0x7f894c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b37141e65378c562 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:31.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.525+0000 7f89337fe640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:55:31.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.525+0000 7f89337fe640 1 -- 172.21.15.42:0/1488189110 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c01e320 con 0x5627edd7e1b0 2024-07-26T06:55:31.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.525+0000 7f89337fe640 1 -- 172.21.15.42:0/1488189110 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f894c004b20 con 0x5627edd7e1b0 2024-07-26T06:55:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.525+0000 7f89613b5740 1 -- 172.21.15.42:0/1488189110 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89440e9320 con 0x5627edd7e1b0 2024-07-26T06:55:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.525+0000 7f89337fe640 1 -- 172.21.15.42:0/1488189110 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f894c004d40 con 0x5627edd7e1b0 2024-07-26T06:55:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.525+0000 7f89613b5740 10 client.16032 fetch_fsmap learned FSMap version 37 2024-07-26T06:55:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.526+0000 7f89613b5740 1 -- 172.21.15.42:0/1488189110 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5627ee1a1370 con 0x5627edd7e1b0 2024-07-26T06:55:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.526+0000 7f89337fe640 1 -- 172.21.15.42:0/1488189110 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89440e9320 con 0x5627edd7e1b0 2024-07-26T06:55:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.526+0000 7f89337fe640 1 -- 172.21.15.42:0/1488189110 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7f894c00f040 con 0x5627edd7e1b0 2024-07-26T06:55:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.526+0000 7f89613b5740 10 client.16032 fetch_fsmap finished waiting for FSMap version 37 2024-07-26T06:55:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.526+0000 7f89613b5740 10 client.16032 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:55:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.526+0000 7f89613b5740 20 client.16032 populate_metadata read hostname 'smithi042' 2024-07-26T06:55:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.526+0000 7f89613b5740 1 --2- 172.21.15.42:0/1488189110 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5627edd121c0 0x5627ee1a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.526+0000 7f89613b5740 4 client.16032 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:55:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.526+0000 7f89613b5740 1 -- 172.21.15.42:0/1488189110 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5627ee093da0 con 0x5627edd121c0 2024-07-26T06:55:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.526+0000 7f8953fff640 1 --2- 172.21.15.42:0/1488189110 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5627edd121c0 0x5627ee1a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.527+0000 7f8953fff640 1 --2- 172.21.15.42:0/1488189110 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5627edd121c0 0x5627ee1a3ea0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ce50b078edbae33 server_cookie=eeebcf65bb4eab0c in_seq=0 out_seq=0 2024-07-26T06:55:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.527+0000 7f89337fe640 10 client.16032 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.550+0000 7f89337fe640 1 -- 172.21.15.42:0/1488189110 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5627ee093da0 con 0x5627edd121c0 2024-07-26T06:55:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.550+0000 7f89337fe640 10 client.16032 handle_command_reply: tid=0 2024-07-26T06:55:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.550+0000 7f89613b5740 1 client.16032 shutdown 2024-07-26T06:55:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.550+0000 7f89613b5740 1 -- 172.21.15.42:0/1488189110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd7e1b0 msgr2=0x5627ee132ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.550+0000 7f89613b5740 1 --2- 172.21.15.42:0/1488189110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd7e1b0 0x5627ee132ab0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f894c01e9e0 tx=0x7f894c004a40 comp rx=0 tx=0).stop 2024-07-26T06:55:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.550+0000 7f89613b5740 1 -- 172.21.15.42:0/1488189110 shutdown_connections 2024-07-26T06:55:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.550+0000 7f89613b5740 1 --2- 172.21.15.42:0/1488189110 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5627edd121c0 0x5627ee1a3ea0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.550+0000 7f89613b5740 1 --2- 172.21.15.42:0/1488189110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5627edd7e1b0 0x5627ee132ab0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.550+0000 7f89613b5740 1 --2- 172.21.15.42:0/1488189110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627edd5d4d0 0x5627ee1345f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.550+0000 7f89337fe640 0 client.16032 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.550+0000 7f89613b5740 1 --2- 172.21.15.42:0/1488189110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5627edd552f0 0x5627ee133f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.550+0000 7f89613b5740 1 -- 172.21.15.42:0/1488189110 >> 172.21.15.42:0/1488189110 conn(0x5627edd72300 msgr2=0x5627ee124010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.550+0000 7f89613b5740 1 -- 172.21.15.42:0/1488189110 shutdown_connections 2024-07-26T06:55:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.550+0000 7f89613b5740 1 -- 172.21.15.42:0/1488189110 wait complete. 2024-07-26T06:55:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.550+0000 7f89613b5740 20 client.16032 trim_cache size 0 max 16384 2024-07-26T06:55:31.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:55:31.553 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:55:31.553 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:55:31.553 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:55:31.553 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.551+0000 7f895a3c4640 1 -- 172.21.15.42:0/2166920291 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f893c0b3bd0 msgr2=0x7f893c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:31.553 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.551+0000 7f895a3c4640 1 --2- 172.21.15.42:0/2166920291 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f893c0b3bd0 0x7f893c0b60e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f894c005fd0 tx=0x7f894c009580 comp rx=0 tx=0).stop 2024-07-26T06:55:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.551+0000 7f895a3c4640 1 -- 172.21.15.42:0/2166920291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8954102000 msgr2=0x7f895406bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.551+0000 7f895a3c4640 1 --2- 172.21.15.42:0/2166920291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8954102000 0x7f895406bc60 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f894400d950 tx=0x7f894400de10 comp rx=0 tx=0).stop 2024-07-26T06:55:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.551+0000 7f895a3c4640 1 -- 172.21.15.42:0/2166920291 shutdown_connections 2024-07-26T06:55:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.551+0000 7f895a3c4640 1 --2- 172.21.15.42:0/2166920291 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f893c0b3bd0 0x7f893c0b60e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.551+0000 7f895a3c4640 1 --2- 172.21.15.42:0/2166920291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8954103a20 0x7f89541a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.551+0000 7f895a3c4640 1 --2- 172.21.15.42:0/2166920291 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8954103080 0x7f895419e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.551+0000 7f895a3c4640 1 --2- 172.21.15.42:0/2166920291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8954102000 0x7f895406bc60 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.551+0000 7f895a3c4640 1 -- 172.21.15.42:0/2166920291 >> 172.21.15.42:0/2166920291 conn(0x7f89540fd960 msgr2=0x7f89541063a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.551+0000 7f895a3c4640 1 -- 172.21.15.42:0/2166920291 shutdown_connections 2024-07-26T06:55:31.554 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:31.551+0000 7f895a3c4640 1 -- 172.21.15.42:0/2166920291 wait complete. 2024-07-26T06:55:32.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.690+0000 7fa9abb8b640 1 -- 172.21.15.42:0/3419608216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9a4109820 msgr2=0x7fa9a4110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:32.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.690+0000 7fa9abb8b640 1 --2- 172.21.15.42:0/3419608216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9a4109820 0x7fa9a4110050 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa99c009900 tx=0x7fa99c02a3f0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.690+0000 7fa9abb8b640 1 -- 172.21.15.42:0/3419608216 shutdown_connections 2024-07-26T06:55:32.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.690+0000 7fa9abb8b640 1 --2- 172.21.15.42:0/3419608216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9a4109820 0x7fa9a4110050 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.690+0000 7fa9abb8b640 1 --2- 172.21.15.42:0/3419608216 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9a4108ed0 0x7fa9a41092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.690+0000 7fa9abb8b640 1 --2- 172.21.15.42:0/3419608216 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9a4103590 0x7fa9a41039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.690+0000 7fa9abb8b640 1 -- 172.21.15.42:0/3419608216 >> 172.21.15.42:0/3419608216 conn(0x7fa9a407b280 msgr2=0x7fa9a407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:32.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.690+0000 7fa9abb8b640 1 -- 172.21.15.42:0/3419608216 shutdown_connections 2024-07-26T06:55:32.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.690+0000 7fa9abb8b640 1 -- 172.21.15.42:0/3419608216 wait complete. 2024-07-26T06:55:32.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.691+0000 7fa9abb8b640 1 Processor -- start 2024-07-26T06:55:32.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.691+0000 7fa9abb8b640 1 -- start start 2024-07-26T06:55:32.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.691+0000 7fa9abb8b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9a4103590 0x7fa9a4199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.691+0000 7fa9abb8b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9a4108ed0 0x7fa9a419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9abb8b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9a4109820 0x7fa9a419f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9abb8b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a41134f0 con 0x7fa9a4109820 2024-07-26T06:55:32.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9abb8b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a4113370 con 0x7fa9a4108ed0 2024-07-26T06:55:32.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9abb8b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a4113670 con 0x7fa9a4103590 2024-07-26T06:55:32.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9aa128640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9a4109820 0x7fa9a419f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9aa128640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9a4109820 0x7fa9a419f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45794/0 (socket says 172.21.15.42:45794) 2024-07-26T06:55:32.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9aa128640 1 -- 172.21.15.42:0/1212569804 learned_addr learned my addr 172.21.15.42:0/1212569804 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:32.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9a9927640 1 --2- 172.21.15.42:0/1212569804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9a4103590 0x7fa9a4199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9a9126640 1 --2- 172.21.15.42:0/1212569804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9a4108ed0 0x7fa9a419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9aa128640 1 -- 172.21.15.42:0/1212569804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9a4103590 msgr2=0x7fa9a4199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:32.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9aa128640 1 --2- 172.21.15.42:0/1212569804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9a4103590 0x7fa9a4199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9aa128640 1 -- 172.21.15.42:0/1212569804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9a4108ed0 msgr2=0x7fa9a419a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:32.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9aa128640 1 --2- 172.21.15.42:0/1212569804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9a4108ed0 0x7fa9a419a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9aa128640 1 -- 172.21.15.42:0/1212569804 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a419fdd0 con 0x7fa9a4109820 2024-07-26T06:55:32.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9aa128640 1 --2- 172.21.15.42:0/1212569804 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9a4109820 0x7fa9a419f760 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa99c004770 tx=0x7fa99c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:32.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9a2ffd640 1 -- 172.21.15.42:0/1212569804 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c03f030 con 0x7fa9a4109820 2024-07-26T06:55:32.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9abb8b640 1 -- 172.21.15.42:0/1212569804 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9a41a0060 con 0x7fa9a4109820 2024-07-26T06:55:32.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9a2ffd640 1 -- 172.21.15.42:0/1212569804 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa99c003aa0 con 0x7fa9a4109820 2024-07-26T06:55:32.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.692+0000 7fa9a2ffd640 1 -- 172.21.15.42:0/1212569804 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c035070 con 0x7fa9a4109820 2024-07-26T06:55:32.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.693+0000 7fa9abb8b640 1 -- 172.21.15.42:0/1212569804 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9a41a0590 con 0x7fa9a4109820 2024-07-26T06:55:32.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.695+0000 7fa9b210f740 1 Processor -- start 2024-07-26T06:55:32.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.695+0000 7fa9a2ffd640 1 -- 172.21.15.42:0/1212569804 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7fa99c042030 con 0x7fa9a4109820 2024-07-26T06:55:32.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.696+0000 7fa9b210f740 1 -- start start 2024-07-26T06:55:32.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.696+0000 7fa9b210f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae63b520 0x55c4ae9e1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.696+0000 7fa9b210f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae612660 0x55c4ae9e15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.696+0000 7fa9b210f740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae61a840 0x55c4ae9e1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.696+0000 7fa9b210f740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55c4ae956e70 con 0x55c4ae61a840 2024-07-26T06:55:32.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.696+0000 7fa9b210f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c4ae946370 con 0x55c4ae63b520 2024-07-26T06:55:32.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.696+0000 7fa9b210f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55c4ae94a3b0 con 0x55c4ae612660 2024-07-26T06:55:32.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.696+0000 7fa9a9126640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae61a840 0x55c4ae9e1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.696+0000 7fa9a9126640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae61a840 0x55c4ae9e1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45806/0 (socket says 172.21.15.42:45806) 2024-07-26T06:55:32.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.696+0000 7fa9a9126640 1 -- 172.21.15.42:0/1623098519 learned_addr learned my addr 172.21.15.42:0/1623098519 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:32.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.696+0000 7fa9a9927640 1 --2- 172.21.15.42:0/1623098519 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae612660 0x55c4ae9e15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.696+0000 7fa9a9126640 1 --2- 172.21.15.42:0/1623098519 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae63b520 0x55c4ae9e1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.697+0000 7fa9a9126640 1 -- 172.21.15.42:0/1623098519 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae612660 msgr2=0x55c4ae9e15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:32.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.697+0000 7fa9a9126640 1 --2- 172.21.15.42:0/1623098519 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae612660 0x55c4ae9e15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.697+0000 7fa9a9126640 1 -- 172.21.15.42:0/1623098519 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae63b520 msgr2=0x55c4ae9e1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:32.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.697+0000 7fa9a9126640 1 --2- 172.21.15.42:0/1623098519 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae63b520 0x55c4ae9e1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.697+0000 7fa9a9126640 1 -- 172.21.15.42:0/1623098519 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4ae913040 con 0x55c4ae61a840 2024-07-26T06:55:32.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.697+0000 7fa9a2ffd640 1 --2- 172.21.15.42:0/1212569804 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa98c0b3c00 0x7fa98c0b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.697+0000 7fa9a9126640 1 --2- 172.21.15.42:0/1623098519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae61a840 0x55c4ae9e1af0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa998009f30 tx=0x7fa9980096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1405c890c20535 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:32.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.697+0000 7fa9a9927640 1 --2- 172.21.15.42:0/1212569804 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa98c0b3c00 0x7fa98c0b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.697+0000 7fa9a2ffd640 1 -- 172.21.15.42:0/1212569804 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fa99c1041b0 con 0x7fa9a4109820 2024-07-26T06:55:32.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.697+0000 7fa98bfff640 1 -- 172.21.15.42:0/1623098519 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa998004050 con 0x55c4ae61a840 2024-07-26T06:55:32.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.697+0000 7fa98bfff640 1 -- 172.21.15.42:0/1623098519 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9980054d0 con 0x55c4ae61a840 2024-07-26T06:55:32.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.697+0000 7fa98bfff640 1 -- 172.21.15.42:0/1623098519 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa998004c40 con 0x55c4ae61a840 2024-07-26T06:55:32.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.698+0000 7fa9b210f740 1 -- 172.21.15.42:0/1623098519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae61a840 msgr2=0x55c4ae9e1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:32.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.698+0000 7fa9a9927640 1 --2- 172.21.15.42:0/1212569804 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa98c0b3c00 0x7fa98c0b6110 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x55c4ae9dfcf0 tx=0x7fa9940075b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:32.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.698+0000 7fa9b210f740 1 --2- 172.21.15.42:0/1623098519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae61a840 0x55c4ae9e1af0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa998009f30 tx=0x7fa9980096a0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.698+0000 7fa9b210f740 1 -- 172.21.15.42:0/1623098519 shutdown_connections 2024-07-26T06:55:32.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.698+0000 7fa9b210f740 1 --2- 172.21.15.42:0/1623098519 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae61a840 0x55c4ae9e1af0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.698+0000 7fa9b210f740 1 --2- 172.21.15.42:0/1623098519 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae612660 0x55c4ae9e15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.698+0000 7fa9b210f740 1 --2- 172.21.15.42:0/1623098519 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae63b520 0x55c4ae9e1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.698+0000 7fa9b210f740 1 -- 172.21.15.42:0/1623098519 >> 172.21.15.42:0/1623098519 conn(0x55c4ae62f910 msgr2=0x55c4ae9def60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:32.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.698+0000 7fa9b210f740 1 -- 172.21.15.42:0/1623098519 shutdown_connections 2024-07-26T06:55:32.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.698+0000 7fa9b210f740 1 -- 172.21.15.42:0/1623098519 wait complete. 2024-07-26T06:55:32.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.699+0000 7fa9b210f740 1 -- start start 2024-07-26T06:55:32.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.699+0000 7fa9b210f740 1 Processor -- start 2024-07-26T06:55:32.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.699+0000 7fa9b210f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae612660 0x55c4ae9ecd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.699+0000 7fa9b210f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae61a840 0x55c4ae9ed2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.699+0000 7fa9b210f740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae63b520 0x55c4ae9f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.699+0000 7fa9b210f740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55c4ae7a9bb0 con 0x55c4ae63b520 2024-07-26T06:55:32.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.699+0000 7fa9b210f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c4ae8e8ac0 con 0x55c4ae61a840 2024-07-26T06:55:32.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.699+0000 7fa9b210f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55c4ae9ef850 con 0x55c4ae612660 2024-07-26T06:55:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.699+0000 7fa9b210f740 10 client.0 _ll_register_callbacks cb 0x55c4ae8f13b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:55:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.699+0000 7fa9aa128640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae63b520 0x55c4ae9f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.699+0000 7fa9aa128640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae63b520 0x55c4ae9f2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45808/0 (socket says 172.21.15.42:45808) 2024-07-26T06:55:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.699+0000 7fa9aa128640 1 -- 172.21.15.42:0/2440529907 learned_addr learned my addr 172.21.15.42:0/2440529907 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.700+0000 7fa9a9126640 1 --2- 172.21.15.42:0/2440529907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae61a840 0x55c4ae9ed2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.700+0000 7fa9aa128640 1 -- 172.21.15.42:0/2440529907 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae612660 msgr2=0x55c4ae9ecd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.700+0000 7fa9a9927640 1 --2- 172.21.15.42:0/2440529907 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae612660 0x55c4ae9ecd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.700+0000 7fa9aa128640 1 --2- 172.21.15.42:0/2440529907 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae612660 0x55c4ae9ecd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.700+0000 7fa9aa128640 1 -- 172.21.15.42:0/2440529907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae61a840 msgr2=0x55c4ae9ed2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.700+0000 7fa9aa128640 1 --2- 172.21.15.42:0/2440529907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae61a840 0x55c4ae9ed2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.700+0000 7fa9aa128640 1 -- 172.21.15.42:0/2440529907 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4ae6a6b00 con 0x55c4ae63b520 2024-07-26T06:55:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.700+0000 7fa9aa128640 1 --2- 172.21.15.42:0/2440529907 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae63b520 0x55c4ae9f2b10 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0c7210 tx=0x7fa99c0caae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80806abd5bbfe0fc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.701+0000 7fa9a9126640 1 --2- 172.21.15.42:0/2440529907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae61a840 0x55c4ae9ed2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:55:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.701+0000 7fa9897fa640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:55:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.701+0000 7fa9a9927640 1 --2- 172.21.15.42:0/2440529907 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae612660 0x55c4ae9ecd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T06:55:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.701+0000 7fa9897fa640 1 -- 172.21.15.42:0/2440529907 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c049040 con 0x55c4ae63b520 2024-07-26T06:55:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.701+0000 7fa9897fa640 1 -- 172.21.15.42:0/2440529907 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fa99c0c7420 con 0x55c4ae63b520 2024-07-26T06:55:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.701+0000 7fa9b210f740 1 -- 172.21.15.42:0/2440529907 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9980054d0 con 0x55c4ae63b520 2024-07-26T06:55:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.701+0000 7fa9897fa640 1 -- 172.21.15.42:0/2440529907 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa99c0cb190 con 0x55c4ae63b520 2024-07-26T06:55:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.701+0000 7fa9b210f740 10 client.16044 fetch_fsmap learned FSMap version 37 2024-07-26T06:55:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.701+0000 7fa9b210f740 1 -- 172.21.15.42:0/2440529907 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c4ae71d960 con 0x55c4ae63b520 2024-07-26T06:55:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.701+0000 7fa9897fa640 1 -- 172.21.15.42:0/2440529907 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9980054d0 con 0x55c4ae63b520 2024-07-26T06:55:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.701+0000 7fa9897fa640 1 -- 172.21.15.42:0/2440529907 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7fa99c0d3d00 con 0x55c4ae63b520 2024-07-26T06:55:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.701+0000 7fa9b210f740 10 client.16044 fetch_fsmap finished waiting for FSMap version 37 2024-07-26T06:55:32.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.701+0000 7fa9b210f740 10 client.16044 resolve_mds: validated gid 24620 aka daemon mds.b 2024-07-26T06:55:32.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.701+0000 7fa9b210f740 20 client.16044 populate_metadata read hostname 'smithi042' 2024-07-26T06:55:32.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.701+0000 7fa9b210f740 1 --2- 172.21.15.42:0/2440529907 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55c4ae5f9ea0 0x55c4ae9feed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.701+0000 7fa9b210f740 4 client.16044 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:55:32.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.701+0000 7fa9b210f740 1 -- 172.21.15.42:0/2440529907 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4ae958dc0 con 0x55c4ae5f9ea0 2024-07-26T06:55:32.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.702+0000 7fa9a9927640 1 --2- 172.21.15.42:0/2440529907 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55c4ae5f9ea0 0x55c4ae9feed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.702+0000 7fa9a9927640 1 --2- 172.21.15.42:0/2440529907 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55c4ae5f9ea0 0x55c4ae9feed0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d39bd8cfeb0285ac server_cookie=f436908683c0744 in_seq=0 out_seq=0 2024-07-26T06:55:32.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.702+0000 7fa9897fa640 10 client.16044 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:32.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.703+0000 7fa9897fa640 1 -- 172.21.15.42:0/2440529907 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c4ae958dc0 con 0x55c4ae5f9ea0 2024-07-26T06:55:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.703+0000 7fa9897fa640 10 client.16044 handle_command_reply: tid=0 2024-07-26T06:55:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.703+0000 7fa9b210f740 1 client.16044 shutdown 2024-07-26T06:55:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.703+0000 7fa9b210f740 1 -- 172.21.15.42:0/2440529907 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae63b520 msgr2=0x55c4ae9f2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.703+0000 7fa9b210f740 1 --2- 172.21.15.42:0/2440529907 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae63b520 0x55c4ae9f2b10 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0c7210 tx=0x7fa99c0caae0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.703+0000 7fa9b210f740 1 -- 172.21.15.42:0/2440529907 shutdown_connections 2024-07-26T06:55:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.703+0000 7fa9b210f740 1 --2- 172.21.15.42:0/2440529907 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55c4ae5f9ea0 0x55c4ae9feed0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.703+0000 7fa9b210f740 1 --2- 172.21.15.42:0/2440529907 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae63b520 0x55c4ae9f2b10 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.703+0000 7fa9b210f740 1 --2- 172.21.15.42:0/2440529907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae61a840 0x55c4ae9ed2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.703+0000 7fa9897fa640 0 client.16044 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.703+0000 7fa9b210f740 1 --2- 172.21.15.42:0/2440529907 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae612660 0x55c4ae9ecd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.703+0000 7fa9b210f740 1 -- 172.21.15.42:0/2440529907 >> 172.21.15.42:0/2440529907 conn(0x55c4ae62f910 msgr2=0x55c4ae9e4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.704+0000 7fa9b210f740 1 -- 172.21.15.42:0/2440529907 shutdown_connections 2024-07-26T06:55:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.704+0000 7fa9b210f740 1 -- 172.21.15.42:0/2440529907 wait complete. 2024-07-26T06:55:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.704+0000 7fa9b210f740 20 client.16044 trim_cache size 0 max 16384 2024-07-26T06:55:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9b210f740 1 Processor -- start 2024-07-26T06:55:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9b210f740 1 -- start start 2024-07-26T06:55:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9b210f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae5f9ea0 0x55c4ae9ee730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9b210f740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae612660 0x55c4ae9f98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9b210f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae61a840 0x55c4ae9e62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9b210f740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a41acc30 con 0x55c4ae612660 2024-07-26T06:55:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9b210f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a41ae830 con 0x55c4ae61a840 2024-07-26T06:55:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9b210f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a41ad300 con 0x55c4ae5f9ea0 2024-07-26T06:55:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9a9126640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae612660 0x55c4ae9f98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9a9126640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae612660 0x55c4ae9f98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45816/0 (socket says 172.21.15.42:45816) 2024-07-26T06:55:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9a9126640 1 -- 172.21.15.42:0/2992928500 learned_addr learned my addr 172.21.15.42:0/2992928500 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9aa128640 1 --2- 172.21.15.42:0/2992928500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae61a840 0x55c4ae9e62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9a9927640 1 --2- 172.21.15.42:0/2992928500 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae5f9ea0 0x55c4ae9ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9a9126640 1 -- 172.21.15.42:0/2992928500 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae5f9ea0 msgr2=0x55c4ae9ee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9a9126640 1 --2- 172.21.15.42:0/2992928500 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae5f9ea0 0x55c4ae9ee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9a9126640 1 -- 172.21.15.42:0/2992928500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae61a840 msgr2=0x55c4ae9e62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9a9126640 1 --2- 172.21.15.42:0/2992928500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae61a840 0x55c4ae9e62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.721+0000 7fa9a9126640 1 -- 172.21.15.42:0/2992928500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4ae9eb000 con 0x55c4ae612660 2024-07-26T06:55:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.722+0000 7fa9a9126640 1 --2- 172.21.15.42:0/2992928500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae612660 0x55c4ae9f98d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa998011d60 tx=0x7fa9980048e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26abbe340ffa41ff server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.722+0000 7fa98b7fe640 1 -- 172.21.15.42:0/2992928500 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa998005330 con 0x55c4ae612660 2024-07-26T06:55:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.722+0000 7fa98b7fe640 1 -- 172.21.15.42:0/2992928500 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa998004e00 con 0x55c4ae612660 2024-07-26T06:55:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.722+0000 7fa98b7fe640 1 -- 172.21.15.42:0/2992928500 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99801eb10 con 0x55c4ae612660 2024-07-26T06:55:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.723+0000 7fa9b210f740 1 -- 172.21.15.42:0/2992928500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae612660 msgr2=0x55c4ae9f98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.723+0000 7fa9b210f740 1 --2- 172.21.15.42:0/2992928500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae612660 0x55c4ae9f98d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa998011d60 tx=0x7fa9980048e0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.723+0000 7fa9b210f740 1 -- 172.21.15.42:0/2992928500 shutdown_connections 2024-07-26T06:55:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.723+0000 7fa9b210f740 1 --2- 172.21.15.42:0/2992928500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae61a840 0x55c4ae9e62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.723+0000 7fa9b210f740 1 --2- 172.21.15.42:0/2992928500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae612660 0x55c4ae9f98d0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.723+0000 7fa9b210f740 1 --2- 172.21.15.42:0/2992928500 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae5f9ea0 0x55c4ae9ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.723+0000 7fa9b210f740 1 -- 172.21.15.42:0/2992928500 >> 172.21.15.42:0/2992928500 conn(0x55c4ae62f910 msgr2=0x55c4ae9ee020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:32.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.723+0000 7fa9b210f740 1 -- 172.21.15.42:0/2992928500 shutdown_connections 2024-07-26T06:55:32.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.723+0000 7fa9b210f740 1 -- 172.21.15.42:0/2992928500 wait complete. 2024-07-26T06:55:32.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.723+0000 7fa9b210f740 1 -- start start 2024-07-26T06:55:32.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.723+0000 7fa9b210f740 1 Processor -- start 2024-07-26T06:55:32.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9b210f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae5f9ea0 0x55c4ae9f92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9b210f740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae612660 0x55c4ae9f98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9b210f740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae61a840 0x55c4ae9f2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9b210f740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a41ae160 con 0x55c4ae612660 2024-07-26T06:55:32.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9b210f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a41ad9d0 con 0x55c4ae5f9ea0 2024-07-26T06:55:32.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9b210f740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a41aac10 con 0x55c4ae61a840 2024-07-26T06:55:32.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9b210f740 10 client.0 _ll_register_callbacks cb 0x7fa99c031950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:55:32.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9a9126640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae612660 0x55c4ae9f98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9a9126640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae612660 0x55c4ae9f98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45822/0 (socket says 172.21.15.42:45822) 2024-07-26T06:55:32.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9a9126640 1 -- 172.21.15.42:0/262866737 learned_addr learned my addr 172.21.15.42:0/262866737 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:32.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9a9927640 1 --2- 172.21.15.42:0/262866737 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae5f9ea0 0x55c4ae9f92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9aa128640 1 --2- 172.21.15.42:0/262866737 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae61a840 0x55c4ae9f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9a9126640 1 -- 172.21.15.42:0/262866737 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae61a840 msgr2=0x55c4ae9f2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9a9126640 1 --2- 172.21.15.42:0/262866737 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae61a840 0x55c4ae9f2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9a9126640 1 -- 172.21.15.42:0/262866737 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae5f9ea0 msgr2=0x55c4ae9f92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9a9126640 1 --2- 172.21.15.42:0/262866737 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae5f9ea0 0x55c4ae9f92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9a9126640 1 -- 172.21.15.42:0/262866737 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4ae9f30b0 con 0x55c4ae612660 2024-07-26T06:55:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.724+0000 7fa9aa128640 1 --2- 172.21.15.42:0/262866737 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae61a840 0x55c4ae9f2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:55:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.725+0000 7fa9a9126640 1 --2- 172.21.15.42:0/262866737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae612660 0x55c4ae9f98a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa9980098f0 tx=0x7fa9980096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c02ba35fe40f62 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.725+0000 7fa9897fa640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:55:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.725+0000 7fa9897fa640 1 -- 172.21.15.42:0/262866737 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa998004b60 con 0x55c4ae612660 2024-07-26T06:55:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.725+0000 7fa9897fa640 1 -- 172.21.15.42:0/262866737 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7fa998025050 con 0x55c4ae612660 2024-07-26T06:55:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.725+0000 7fa9b210f740 1 -- 172.21.15.42:0/262866737 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa998004e00 con 0x55c4ae612660 2024-07-26T06:55:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.725+0000 7fa9897fa640 1 -- 172.21.15.42:0/262866737 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa998004040 con 0x55c4ae612660 2024-07-26T06:55:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.725+0000 7fa9b210f740 10 client.16050 fetch_fsmap learned FSMap version 37 2024-07-26T06:55:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.725+0000 7fa9b210f740 1 -- 172.21.15.42:0/262866737 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c4ae9f8b00 con 0x55c4ae612660 2024-07-26T06:55:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.725+0000 7fa9897fa640 1 -- 172.21.15.42:0/262866737 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa998004e00 con 0x55c4ae612660 2024-07-26T06:55:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.725+0000 7fa9897fa640 1 -- 172.21.15.42:0/262866737 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7fa9980035f0 con 0x55c4ae612660 2024-07-26T06:55:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.725+0000 7fa9b210f740 10 client.16050 fetch_fsmap finished waiting for FSMap version 37 2024-07-26T06:55:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.725+0000 7fa9b210f740 10 client.16050 resolve_mds: validated gid 24620 aka daemon mds.b 2024-07-26T06:55:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.725+0000 7fa9b210f740 20 client.16050 populate_metadata read hostname 'smithi042' 2024-07-26T06:55:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.725+0000 7fa9b210f740 1 --2- 172.21.15.42:0/262866737 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55c4ae63b520 0x55c4aea69850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.725+0000 7fa9b210f740 4 client.16050 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "84ce04c6", "release": true, "await": true}] 2024-07-26T06:55:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.725+0000 7fa9b210f740 1 -- 172.21.15.42:0/262866737 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "quiesce db", "set_id": "84ce04c6", "release": true, "await": true}) -- 0x55c4ae9e86b0 con 0x55c4ae63b520 2024-07-26T06:55:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.726+0000 7fa9a9927640 1 --2- 172.21.15.42:0/262866737 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55c4ae63b520 0x55c4aea69850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:32.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.726+0000 7fa9897fa640 10 client.16050 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:32.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.726+0000 7fa9a9927640 1 --2- 172.21.15.42:0/262866737 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55c4ae63b520 0x55c4aea69850 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97ba34d024dc5cf3 server_cookie=3bad8a630dfb6a21 in_seq=0 out_seq=0 2024-07-26T06:55:32.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.805+0000 7fa9897fa640 1 -- 172.21.15.42:0/262866737 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55c4ae9e86b0 con 0x55c4ae63b520 2024-07-26T06:55:32.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.805+0000 7fa9897fa640 10 client.16050 handle_command_reply: tid=0 2024-07-26T06:55:32.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.805+0000 7fa9b210f740 1 client.16050 shutdown 2024-07-26T06:55:32.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.805+0000 7fa9b210f740 1 -- 172.21.15.42:0/262866737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae612660 msgr2=0x55c4ae9f98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:32.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.805+0000 7fa9b210f740 1 --2- 172.21.15.42:0/262866737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae612660 0x55c4ae9f98a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa9980098f0 tx=0x7fa9980096d0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.805+0000 7fa9b210f740 1 -- 172.21.15.42:0/262866737 shutdown_connections 2024-07-26T06:55:32.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.805+0000 7fa9b210f740 1 --2- 172.21.15.42:0/262866737 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55c4ae63b520 0x55c4aea69850 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.805+0000 7fa9b210f740 1 --2- 172.21.15.42:0/262866737 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55c4ae61a840 0x55c4ae9f2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.805+0000 7fa9b210f740 1 --2- 172.21.15.42:0/262866737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55c4ae612660 0x55c4ae9f98a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.805+0000 7fa9897fa640 0 client.16050 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:32.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.805+0000 7fa9b210f740 1 --2- 172.21.15.42:0/262866737 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c4ae5f9ea0 0x55c4ae9f92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.805+0000 7fa9b210f740 1 -- 172.21.15.42:0/262866737 >> 172.21.15.42:0/262866737 conn(0x55c4ae62f910 msgr2=0x55c4ae9e20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:32.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.805+0000 7fa9b210f740 1 -- 172.21.15.42:0/262866737 shutdown_connections 2024-07-26T06:55:32.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.805+0000 7fa9b210f740 1 -- 172.21.15.42:0/262866737 wait complete. 2024-07-26T06:55:32.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.806+0000 7fa9b210f740 20 client.16050 trim_cache size 0 max 16384 2024-07-26T06:55:32.808 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-07-26T06:55:32.808 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 37, 2024-07-26T06:55:32.808 INFO:teuthology.orchestra.run.smithi042.stdout: "leader": 24620, 2024-07-26T06:55:32.808 INFO:teuthology.orchestra.run.smithi042.stdout: "set_version": 16, 2024-07-26T06:55:32.808 INFO:teuthology.orchestra.run.smithi042.stdout: "sets": { 2024-07-26T06:55:32.808 INFO:teuthology.orchestra.run.smithi042.stdout: "84ce04c6": { 2024-07-26T06:55:32.808 INFO:teuthology.orchestra.run.smithi042.stdout: "version": 16, 2024-07-26T06:55:32.809 INFO:teuthology.orchestra.run.smithi042.stdout: "age_ref": 0.0, 2024-07-26T06:55:32.809 INFO:teuthology.orchestra.run.smithi042.stdout: "state": { 2024-07-26T06:55:32.809 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "RELEASED", 2024-07-26T06:55:32.809 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 0.0 2024-07-26T06:55:32.809 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-07-26T06:55:32.809 INFO:teuthology.orchestra.run.smithi042.stdout: "timeout": 90.0, 2024-07-26T06:55:32.809 INFO:teuthology.orchestra.run.smithi042.stdout: "expiration": 130.0, 2024-07-26T06:55:32.809 INFO:teuthology.orchestra.run.smithi042.stdout: "members": { 2024-07-26T06:55:32.809 INFO:teuthology.orchestra.run.smithi042.stdout: "file:/": { 2024-07-26T06:55:32.809 INFO:teuthology.orchestra.run.smithi042.stdout: "excluded": false, 2024-07-26T06:55:32.809 INFO:teuthology.orchestra.run.smithi042.stdout: "state": { 2024-07-26T06:55:32.809 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "RELEASED", 2024-07-26T06:55:32.809 INFO:teuthology.orchestra.run.smithi042.stdout: "age": 0.0 2024-07-26T06:55:32.810 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:55:32.810 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:55:32.810 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:55:32.810 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:55:32.810 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-07-26T06:55:32.810 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-07-26T06:55:32.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.806+0000 7fa9abb8b640 1 -- 172.21.15.42:0/1212569804 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa98c0b3c00 msgr2=0x7fa98c0b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:32.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.806+0000 7fa9abb8b640 1 --2- 172.21.15.42:0/1212569804 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa98c0b3c00 0x7fa98c0b6110 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x55c4ae9dfcf0 tx=0x7fa9940075b0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.806+0000 7fa9abb8b640 1 -- 172.21.15.42:0/1212569804 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9a4109820 msgr2=0x7fa9a419f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:32.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.806+0000 7fa9abb8b640 1 --2- 172.21.15.42:0/1212569804 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9a4109820 0x7fa9a419f760 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa99c004770 tx=0x7fa99c004270 comp rx=0 tx=0).stop 2024-07-26T06:55:32.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.806+0000 7fa9abb8b640 1 -- 172.21.15.42:0/1212569804 shutdown_connections 2024-07-26T06:55:32.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.806+0000 7fa9abb8b640 1 --2- 172.21.15.42:0/1212569804 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7fa98c0b3c00 0x7fa98c0b6110 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.806+0000 7fa9abb8b640 1 --2- 172.21.15.42:0/1212569804 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa9a4109820 0x7fa9a419f760 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.806+0000 7fa9abb8b640 1 --2- 172.21.15.42:0/1212569804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9a4108ed0 0x7fa9a419a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.806+0000 7fa9abb8b640 1 --2- 172.21.15.42:0/1212569804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9a4103590 0x7fa9a4199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:32.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.806+0000 7fa9abb8b640 1 -- 172.21.15.42:0/1212569804 >> 172.21.15.42:0/1212569804 conn(0x7fa9a407b280 msgr2=0x7fa9a4106f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:32.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.806+0000 7fa9abb8b640 1 -- 172.21.15.42:0/1212569804 shutdown_connections 2024-07-26T06:55:32.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:32.806+0000 7fa9abb8b640 1 -- 172.21.15.42:0/1212569804 wait complete. 2024-07-26T06:55:33.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:32 smithi042 ceph-mon[32775]: pgmap v556: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 1.2 MiB/s wr, 473 op/s 2024-07-26T06:55:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:32 smithi164 ceph-mon[36970]: pgmap v556: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 1.2 MiB/s wr, 473 op/s 2024-07-26T06:55:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:32 smithi163 ceph-mon[36873]: pgmap v556: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 1.2 MiB/s wr, 473 op/s 2024-07-26T06:55:33.749 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '84ce04c6', total seconds elapsed: 16.9 2024-07-26T06:55:35.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:34 smithi042 ceph-mon[32775]: pgmap v557: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 1.0 MiB/s wr, 403 op/s 2024-07-26T06:55:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:34 smithi164 ceph-mon[36970]: pgmap v557: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 1.0 MiB/s wr, 403 op/s 2024-07-26T06:55:35.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:34 smithi163 ceph-mon[36873]: pgmap v557: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 1.0 MiB/s wr, 403 op/s 2024-07-26T06:55:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:36 smithi164 ceph-mon[36970]: pgmap v558: 97 pgs: 97 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 838 KiB/s wr, 312 op/s 2024-07-26T06:55:37.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:36 smithi163 ceph-mon[36873]: pgmap v558: 97 pgs: 97 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 838 KiB/s wr, 312 op/s 2024-07-26T06:55:37.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:36 smithi042 ceph-mon[32775]: pgmap v558: 97 pgs: 97 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 838 KiB/s wr, 312 op/s 2024-07-26T06:55:39.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:38 smithi164 ceph-mon[36970]: pgmap v559: 97 pgs: 97 active+clean; 71 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 758 KiB/s wr, 292 op/s 2024-07-26T06:55:39.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:38 smithi163 ceph-mon[36873]: pgmap v559: 97 pgs: 97 active+clean; 71 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 758 KiB/s wr, 292 op/s 2024-07-26T06:55:39.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:38 smithi042 ceph-mon[32775]: pgmap v559: 97 pgs: 97 active+clean; 71 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 758 KiB/s wr, 292 op/s 2024-07-26T06:55:41.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:40 smithi164 ceph-mon[36970]: pgmap v560: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 1.1 MiB/s wr, 432 op/s 2024-07-26T06:55:41.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:40 smithi163 ceph-mon[36873]: pgmap v560: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 1.1 MiB/s wr, 432 op/s 2024-07-26T06:55:41.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:40 smithi042 ceph-mon[32775]: pgmap v560: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 1.1 MiB/s wr, 432 op/s 2024-07-26T06:55:43.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:42 smithi164 ceph-mon[36970]: pgmap v561: 97 pgs: 97 active+clean; 69 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 504 op/s 2024-07-26T06:55:43.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:42 smithi163 ceph-mon[36873]: pgmap v561: 97 pgs: 97 active+clean; 69 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 504 op/s 2024-07-26T06:55:43.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:42 smithi042 ceph-mon[32775]: pgmap v561: 97 pgs: 97 active+clean; 69 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 504 op/s 2024-07-26T06:55:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.672+0000 7f376a570640 1 -- 172.21.15.42:0/2976318903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3764103060 msgr2=0x7f3764103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.672+0000 7f376a570640 1 --2- 172.21.15.42:0/2976318903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3764103060 0x7f3764103470 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3758009900 tx=0x7f375802a410 comp rx=0 tx=0).stop 2024-07-26T06:55:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.672+0000 7f376a570640 1 -- 172.21.15.42:0/2976318903 shutdown_connections 2024-07-26T06:55:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.672+0000 7f376a570640 1 --2- 172.21.15.42:0/2976318903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3764103a00 0x7f376410a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.672+0000 7f376a570640 1 --2- 172.21.15.42:0/2976318903 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3764103060 0x7f3764103470 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.675 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.672+0000 7f376a570640 1 --2- 172.21.15.42:0/2976318903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3764101fe0 0x7f3764102450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.672+0000 7f376a570640 1 -- 172.21.15.42:0/2976318903 >> 172.21.15.42:0/2976318903 conn(0x7f37640fd960 msgr2=0x7f37640ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.672+0000 7f376a570640 1 -- 172.21.15.42:0/2976318903 shutdown_connections 2024-07-26T06:55:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.672+0000 7f376a570640 1 -- 172.21.15.42:0/2976318903 wait complete. 2024-07-26T06:55:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.673+0000 7f376a570640 1 Processor -- start 2024-07-26T06:55:44.676 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.673+0000 7f376a570640 1 -- start start 2024-07-26T06:55:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f376a570640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3764101fe0 0x7f37641999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f376a570640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3764103060 0x7f376419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f376a570640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3764103a00 0x7f376419ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f376a570640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f376410dc60 con 0x7f3764101fe0 2024-07-26T06:55:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f376a570640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f376410dae0 con 0x7f3764103a00 2024-07-26T06:55:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f376a570640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f376410dde0 con 0x7f3764103060 2024-07-26T06:55:44.678 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f3763fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3764101fe0 0x7f37641999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f3763fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3764101fe0 0x7f37641999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56208/0 (socket says 172.21.15.42:56208) 2024-07-26T06:55:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f3763fff640 1 -- 172.21.15.42:0/2411303981 learned_addr learned my addr 172.21.15.42:0/2411303981 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f37637fe640 1 --2- 172.21.15.42:0/2411303981 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3764103060 0x7f376419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f3768b0d640 1 --2- 172.21.15.42:0/2411303981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3764103a00 0x7f376419ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f3763fff640 1 -- 172.21.15.42:0/2411303981 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3764103060 msgr2=0x7f376419a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f3763fff640 1 --2- 172.21.15.42:0/2411303981 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3764103060 0x7f376419a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f3763fff640 1 -- 172.21.15.42:0/2411303981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3764103a00 msgr2=0x7f376419ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f3763fff640 1 --2- 172.21.15.42:0/2411303981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3764103a00 0x7f376419ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f3763fff640 1 -- 172.21.15.42:0/2411303981 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376419b420 con 0x7f3764101fe0 2024-07-26T06:55:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f37637fe640 1 --2- 172.21.15.42:0/2411303981 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3764103060 0x7f376419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:55:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f3768b0d640 1 --2- 172.21.15.42:0/2411303981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3764103a00 0x7f376419ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:55:44.679 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.674+0000 7f3763fff640 1 --2- 172.21.15.42:0/2411303981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3764101fe0 0x7f37641999e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f375400ea60 tx=0x7f375400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.675+0000 7f37617fa640 1 -- 172.21.15.42:0/2411303981 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375400cde0 con 0x7f3764101fe0 2024-07-26T06:55:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.675+0000 7f376a570640 1 -- 172.21.15.42:0/2411303981 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376419b710 con 0x7f3764101fe0 2024-07-26T06:55:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.675+0000 7f37617fa640 1 -- 172.21.15.42:0/2411303981 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37540035f0 con 0x7f3764101fe0 2024-07-26T06:55:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.675+0000 7f37617fa640 1 -- 172.21.15.42:0/2411303981 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3754010690 con 0x7f3764101fe0 2024-07-26T06:55:44.680 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.675+0000 7f376a570640 1 -- 172.21.15.42:0/2411303981 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3764104530 con 0x7f3764101fe0 2024-07-26T06:55:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.678+0000 7f3771564740 1 Processor -- start 2024-07-26T06:55:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.679+0000 7f3771564740 1 -- start start 2024-07-26T06:55:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.679+0000 7f37617fa640 1 -- 172.21.15.42:0/2411303981 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f37540037a0 con 0x7f3764101fe0 2024-07-26T06:55:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.679+0000 7f3771564740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991ca1b0 0x557099580190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.679+0000 7f3771564740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5570991a12f0 0x557099580800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.679+0000 7f3771564740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991a94d0 0x557099581350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.679+0000 7f3771564740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5570994e23b0 con 0x5570991ca1b0 2024-07-26T06:55:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.679+0000 7f3771564740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5570994d18b0 con 0x5570991a94d0 2024-07-26T06:55:44.681 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.679+0000 7f3771564740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x5570994d58f0 con 0x5570991a12f0 2024-07-26T06:55:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.679+0000 7f3763fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5570991a12f0 0x557099580800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.680+0000 7f3763fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5570991a12f0 0x557099580800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:42260/0 (socket says 172.21.15.42:42260) 2024-07-26T06:55:44.682 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.680+0000 7f3763fff640 1 -- 172.21.15.42:0/1883788871 learned_addr learned my addr 172.21.15.42:0/1883788871 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.680+0000 7f3763fff640 1 -- 172.21.15.42:0/1883788871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991a94d0 msgr2=0x557099581350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.680+0000 7f3763fff640 1 --2- 172.21.15.42:0/1883788871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991a94d0 0x557099581350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.680+0000 7f3763fff640 1 -- 172.21.15.42:0/1883788871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991ca1b0 msgr2=0x557099580190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.680+0000 7f3763fff640 1 --2- 172.21.15.42:0/1883788871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991ca1b0 0x557099580190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.680+0000 7f3763fff640 1 -- 172.21.15.42:0/1883788871 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570994789a0 con 0x5570991a12f0 2024-07-26T06:55:44.683 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.681+0000 7f37617fa640 1 --2- 172.21.15.42:0/2411303981 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f374c0b3c50 0x7f374c0b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.681+0000 7f3768b0d640 1 --2- 172.21.15.42:0/2411303981 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f374c0b3c50 0x7f374c0b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.681+0000 7f37617fa640 1 -- 172.21.15.42:0/2411303981 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f37540e3f40 con 0x7f3764101fe0 2024-07-26T06:55:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.681+0000 7f3763fff640 1 --2- 172.21.15.42:0/1883788871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5570991a12f0 0x557099580800 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f37540e7dd0 tx=0x7f37540ec4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7729b48e4974df2b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.681+0000 7f374a7fc640 1 -- 172.21.15.42:0/1883788871 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37540a6cd0 con 0x5570991a12f0 2024-07-26T06:55:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.681+0000 7f3768b0d640 1 --2- 172.21.15.42:0/2411303981 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f374c0b3c50 0x7f374c0b6160 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f375c005fd0 tx=0x7f375c009540 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.681+0000 7f374a7fc640 1 -- 172.21.15.42:0/1883788871 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37540aa920 con 0x5570991a12f0 2024-07-26T06:55:44.684 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.681+0000 7f374a7fc640 1 -- 172.21.15.42:0/1883788871 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37540aa580 con 0x5570991a12f0 2024-07-26T06:55:44.686 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.682+0000 7f3771564740 1 -- 172.21.15.42:0/1883788871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5570991a12f0 msgr2=0x557099580800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:44.686 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.682+0000 7f3771564740 1 --2- 172.21.15.42:0/1883788871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5570991a12f0 0x557099580800 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f37540e7dd0 tx=0x7f37540ec4b0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.686 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.682+0000 7f3771564740 1 -- 172.21.15.42:0/1883788871 shutdown_connections 2024-07-26T06:55:44.686 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.682+0000 7f3771564740 1 --2- 172.21.15.42:0/1883788871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991a94d0 0x557099581350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.682+0000 7f3771564740 1 --2- 172.21.15.42:0/1883788871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5570991a12f0 0x557099580800 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.682+0000 7f3771564740 1 --2- 172.21.15.42:0/1883788871 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991ca1b0 0x557099580190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.682+0000 7f3771564740 1 -- 172.21.15.42:0/1883788871 >> 172.21.15.42:0/1883788871 conn(0x5570991be300 msgr2=0x55709956ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:44.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.682+0000 7f3771564740 1 -- 172.21.15.42:0/1883788871 shutdown_connections 2024-07-26T06:55:44.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.682+0000 7f3771564740 1 -- 172.21.15.42:0/1883788871 wait complete. 2024-07-26T06:55:44.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.683+0000 7f3771564740 1 -- start start 2024-07-26T06:55:44.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.683+0000 7f3771564740 1 Processor -- start 2024-07-26T06:55:44.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.683+0000 7f3771564740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5570991a12f0 0x557099589bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.683+0000 7f3771564740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a94d0 0x55709958a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.683+0000 7f3771564740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991ca1b0 0x55709958ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.683+0000 7f3771564740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55709940bab0 con 0x5570991a94d0 2024-07-26T06:55:44.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.683+0000 7f3771564740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557099459cf0 con 0x5570991ca1b0 2024-07-26T06:55:44.688 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.683+0000 7f3771564740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x557099581e70 con 0x5570991a12f0 2024-07-26T06:55:44.688 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.683+0000 7f3771564740 10 client.0 _ll_register_callbacks cb 0x5570994cb9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:55:44.688 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.684+0000 7f3763fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5570991a12f0 0x557099589bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:44.688 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.684+0000 7f37637fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a94d0 0x55709958a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:44.688 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.684+0000 7f3763fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5570991a12f0 0x557099589bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:42270/0 (socket says 172.21.15.42:42270) 2024-07-26T06:55:44.688 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.684+0000 7f3763fff640 1 -- 172.21.15.42:0/3189642546 learned_addr learned my addr 172.21.15.42:0/3189642546 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:44.688 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.684+0000 7f37637fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a94d0 0x55709958a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56234/0 (socket says 172.21.15.42:56234) 2024-07-26T06:55:44.688 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.684+0000 7f3768b0d640 1 --2- 172.21.15.42:0/3189642546 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991ca1b0 0x55709958ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:44.688 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.684+0000 7f3763fff640 1 -- 172.21.15.42:0/3189642546 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991ca1b0 msgr2=0x55709958ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:44.688 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.684+0000 7f3763fff640 1 --2- 172.21.15.42:0/3189642546 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991ca1b0 0x55709958ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.688 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.684+0000 7f3763fff640 1 -- 172.21.15.42:0/3189642546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a94d0 msgr2=0x55709958a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:44.688 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.684+0000 7f3763fff640 1 --2- 172.21.15.42:0/3189642546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a94d0 0x55709958a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.688 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.684+0000 7f3763fff640 1 -- 172.21.15.42:0/3189642546 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557099235790 con 0x5570991a12f0 2024-07-26T06:55:44.689 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.684+0000 7f37637fe640 1 --2- 172.21.15.42:0/3189642546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a94d0 0x55709958a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:55:44.689 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.684+0000 7f3763fff640 1 --2- 172.21.15.42:0/3189642546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5570991a12f0 0x557099589bb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f37540e7da0 tx=0x7f3754003cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdd60c630839a13e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:44.689 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.684+0000 7f37437fe640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-07-26T06:55:44.689 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.684+0000 7f37437fe640 1 -- 172.21.15.42:0/3189642546 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37540c5060 con 0x5570991a12f0 2024-07-26T06:55:44.689 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.685+0000 7f37437fe640 1 -- 172.21.15.42:0/3189642546 <== mon.1 v2:172.21.15.164:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f37540ab070 con 0x5570991a12f0 2024-07-26T06:55:44.689 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.685+0000 7f3771564740 1 -- 172.21.15.42:0/3189642546 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37540aa920 con 0x5570991a12f0 2024-07-26T06:55:44.689 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.685+0000 7f37437fe640 1 -- 172.21.15.42:0/3189642546 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f37540aa760 con 0x5570991a12f0 2024-07-26T06:55:44.689 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.685+0000 7f3771564740 10 client.25681 fetch_fsmap learned FSMap version 37 2024-07-26T06:55:44.689 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.685+0000 7f3771564740 1 -- 172.21.15.42:0/3189642546 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5570992ae620 con 0x5570991a12f0 2024-07-26T06:55:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.685+0000 7f37437fe640 1 -- 172.21.15.42:0/3189642546 <== mon.1 v2:172.21.15.164:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37540aa920 con 0x5570991a12f0 2024-07-26T06:55:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.685+0000 7f37437fe640 1 -- 172.21.15.42:0/3189642546 <== mon.1 v2:172.21.15.164:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7f37540e5020 con 0x5570991a12f0 2024-07-26T06:55:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.686+0000 7f3771564740 10 client.25681 fetch_fsmap finished waiting for FSMap version 37 2024-07-26T06:55:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.686+0000 7f3771564740 10 client.25681 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:55:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.686+0000 7f3771564740 20 client.25681 populate_metadata read hostname 'smithi042' 2024-07-26T06:55:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.686+0000 7f3771564740 1 --2- 172.21.15.42:0/3189642546 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x557099188d80 0x55709959f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.686+0000 7f3771564740 4 client.25681 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:55:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.686+0000 7f3771564740 1 -- 172.21.15.42:0/3189642546 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570994e4300 con 0x557099188d80 2024-07-26T06:55:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.686+0000 7f37637fe640 1 --2- 172.21.15.42:0/3189642546 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x557099188d80 0x55709959f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.686+0000 7f37437fe640 10 client.25681 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.686+0000 7f37637fe640 1 --2- 172.21.15.42:0/3189642546 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x557099188d80 0x55709959f6d0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ea656179a24a309 server_cookie=8fd224aa1b853e8 in_seq=0 out_seq=0 2024-07-26T06:55:44.691 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.687+0000 7f37437fe640 1 -- 172.21.15.42:0/3189642546 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5570994e4300 con 0x557099188d80 2024-07-26T06:55:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.687+0000 7f37437fe640 10 client.25681 handle_command_reply: tid=0 2024-07-26T06:55:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.687+0000 7f3771564740 1 client.25681 shutdown 2024-07-26T06:55:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.687+0000 7f3771564740 1 -- 172.21.15.42:0/3189642546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5570991a12f0 msgr2=0x557099589bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.687+0000 7f3771564740 1 --2- 172.21.15.42:0/3189642546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5570991a12f0 0x557099589bb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f37540e7da0 tx=0x7f3754003cf0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.687+0000 7f3771564740 1 -- 172.21.15.42:0/3189642546 shutdown_connections 2024-07-26T06:55:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.687+0000 7f3771564740 1 --2- 172.21.15.42:0/3189642546 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x557099188d80 0x55709959f6d0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.687+0000 7f3771564740 1 --2- 172.21.15.42:0/3189642546 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991ca1b0 0x55709958ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.687+0000 7f37437fe640 0 client.25681 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.687+0000 7f3771564740 1 --2- 172.21.15.42:0/3189642546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a94d0 0x55709958a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.687+0000 7f3771564740 1 --2- 172.21.15.42:0/3189642546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x5570991a12f0 0x557099589bb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.687+0000 7f3771564740 1 -- 172.21.15.42:0/3189642546 >> 172.21.15.42:0/3189642546 conn(0x5570991be300 msgr2=0x5570995692b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.687+0000 7f3771564740 1 -- 172.21.15.42:0/3189642546 shutdown_connections 2024-07-26T06:55:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.687+0000 7f3771564740 1 -- 172.21.15.42:0/3189642546 wait complete. 2024-07-26T06:55:44.693 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.688+0000 7f3771564740 20 client.25681 trim_cache size 0 max 16384 2024-07-26T06:55:44.708 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.705+0000 7f3771564740 1 Processor -- start 2024-07-26T06:55:44.708 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.705+0000 7f3771564740 1 -- start start 2024-07-26T06:55:44.708 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f3771564740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557099188d80 0x55709956c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f3771564740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a12f0 0x557099584c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f3771564740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991a94d0 0x5570995699c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f3771564740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3764073590 con 0x5570991a12f0 2024-07-26T06:55:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f3771564740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f37641a1d90 con 0x5570991a94d0 2024-07-26T06:55:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f3771564740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3764073c60 con 0x557099188d80 2024-07-26T06:55:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f37637fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a12f0 0x557099584c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f37637fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a12f0 0x557099584c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56244/0 (socket says 172.21.15.42:56244) 2024-07-26T06:55:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f37637fe640 1 -- 172.21.15.42:0/265609712 learned_addr learned my addr 172.21.15.42:0/265609712 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f3768b0d640 1 --2- 172.21.15.42:0/265609712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991a94d0 0x5570995699c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f3763fff640 1 --2- 172.21.15.42:0/265609712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557099188d80 0x55709956c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f37637fe640 1 -- 172.21.15.42:0/265609712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557099188d80 msgr2=0x55709956c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f37637fe640 1 --2- 172.21.15.42:0/265609712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557099188d80 0x55709956c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f37637fe640 1 -- 172.21.15.42:0/265609712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991a94d0 msgr2=0x5570995699c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f37637fe640 1 --2- 172.21.15.42:0/265609712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991a94d0 0x5570995699c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f37637fe640 1 -- 172.21.15.42:0/265609712 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557099587d30 con 0x5570991a12f0 2024-07-26T06:55:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f3768b0d640 1 --2- 172.21.15.42:0/265609712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991a94d0 0x5570995699c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:55:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.706+0000 7f37637fe640 1 --2- 172.21.15.42:0/265609712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a12f0 0x557099584c00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f3758000c00 tx=0x7f3758009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7017d7c89e3327 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.707+0000 7f3749ffb640 1 -- 172.21.15.42:0/265609712 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375804a070 con 0x5570991a12f0 2024-07-26T06:55:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.707+0000 7f3749ffb640 1 -- 172.21.15.42:0/265609712 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3758003aa0 con 0x5570991a12f0 2024-07-26T06:55:44.710 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.707+0000 7f3749ffb640 1 -- 172.21.15.42:0/265609712 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3758035040 con 0x5570991a12f0 2024-07-26T06:55:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.708+0000 7f3771564740 1 -- 172.21.15.42:0/265609712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a12f0 msgr2=0x557099584c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.708+0000 7f3771564740 1 --2- 172.21.15.42:0/265609712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a12f0 0x557099584c00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f3758000c00 tx=0x7f3758009260 comp rx=0 tx=0).stop 2024-07-26T06:55:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.708+0000 7f3771564740 1 -- 172.21.15.42:0/265609712 shutdown_connections 2024-07-26T06:55:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.708+0000 7f3771564740 1 --2- 172.21.15.42:0/265609712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991a94d0 0x5570995699c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.708+0000 7f3771564740 1 --2- 172.21.15.42:0/265609712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a12f0 0x557099584c00 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.708+0000 7f3771564740 1 --2- 172.21.15.42:0/265609712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557099188d80 0x55709956c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.708+0000 7f3771564740 1 -- 172.21.15.42:0/265609712 >> 172.21.15.42:0/265609712 conn(0x5570991be300 msgr2=0x5570995692b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.708+0000 7f3771564740 1 -- 172.21.15.42:0/265609712 shutdown_connections 2024-07-26T06:55:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.708+0000 7f3771564740 1 -- 172.21.15.42:0/265609712 wait complete. 2024-07-26T06:55:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.708+0000 7f3771564740 1 -- start start 2024-07-26T06:55:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3771564740 1 Processor -- start 2024-07-26T06:55:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3771564740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557099188d80 0x557099591630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3771564740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991a12f0 0x55709958fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3771564740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a94d0 0x557099590510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3771564740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3764074ac0 con 0x5570991a94d0 2024-07-26T06:55:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3771564740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3764074330 con 0x5570991a12f0 2024-07-26T06:55:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3771564740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3764071570 con 0x557099188d80 2024-07-26T06:55:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3771564740 10 client.0 _ll_register_callbacks cb 0x7f37540e7c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:55:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3768b0d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a94d0 0x557099590510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3763fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557099188d80 0x557099591630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3768b0d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a94d0 0x557099590510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56252/0 (socket says 172.21.15.42:56252) 2024-07-26T06:55:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f37637fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991a12f0 0x55709958fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:44.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3763fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557099188d80 0x557099591630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.42:42296/0 (socket says 172.21.15.42:42296) 2024-07-26T06:55:44.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3768b0d640 1 -- 172.21.15.42:0/506512744 learned_addr learned my addr 172.21.15.42:0/506512744 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:44.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3763fff640 1 -- 172.21.15.42:0/506512744 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991a12f0 msgr2=0x55709958fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:44.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3763fff640 1 --2- 172.21.15.42:0/506512744 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991a12f0 0x55709958fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3763fff640 1 -- 172.21.15.42:0/506512744 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a94d0 msgr2=0x557099590510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:44.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3763fff640 1 --2- 172.21.15.42:0/506512744 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a94d0 0x557099590510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.709+0000 7f3763fff640 1 -- 172.21.15.42:0/506512744 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557099590ca0 con 0x557099188d80 2024-07-26T06:55:44.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.710+0000 7f3763fff640 1 --2- 172.21.15.42:0/506512744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557099188d80 0x557099591630 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f37540ab6c0 tx=0x7f3754003d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=165564ac2733b7ed server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:44.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.710+0000 7f37437fe640 10 client.0 ms_handle_connect on v2:172.21.15.164:3300/0 2024-07-26T06:55:44.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.710+0000 7f37437fe640 1 -- 172.21.15.42:0/506512744 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37540c5040 con 0x557099188d80 2024-07-26T06:55:44.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.710+0000 7f37437fe640 1 -- 172.21.15.42:0/506512744 <== mon.1 v2:172.21.15.164:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f37540b3a30 con 0x557099188d80 2024-07-26T06:55:44.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.710+0000 7f3771564740 1 -- 172.21.15.42:0/506512744 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3758003aa0 con 0x557099188d80 2024-07-26T06:55:44.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.710+0000 7f37437fe640 1 -- 172.21.15.42:0/506512744 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f37540aabe0 con 0x557099188d80 2024-07-26T06:55:44.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.710+0000 7f3771564740 10 client.25687 fetch_fsmap learned FSMap version 37 2024-07-26T06:55:44.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.710+0000 7f3771564740 1 -- 172.21.15.42:0/506512744 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557099606e40 con 0x557099188d80 2024-07-26T06:55:44.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.711+0000 7f37437fe640 1 -- 172.21.15.42:0/506512744 <== mon.1 v2:172.21.15.164:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3758003aa0 con 0x557099188d80 2024-07-26T06:55:44.717 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.711+0000 7f37437fe640 1 -- 172.21.15.42:0/506512744 <== mon.1 v2:172.21.15.164:3300/0 5 ==== fsmap(e 37) ==== 8378+0+0 (secure 0 0 0) 0x7f3754004000 con 0x557099188d80 2024-07-26T06:55:44.717 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.711+0000 7f3771564740 10 client.25687 fetch_fsmap finished waiting for FSMap version 37 2024-07-26T06:55:44.717 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.711+0000 7f3771564740 10 client.25687 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:55:44.717 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.711+0000 7f3771564740 20 client.25687 populate_metadata read hostname 'smithi042' 2024-07-26T06:55:44.717 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.711+0000 7f3771564740 1 --2- 172.21.15.42:0/506512744 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5570991ca1b0 0x5570996096e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:44.717 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.711+0000 7f3771564740 4 client.25687 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T06:55:44.717 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.711+0000 7f3771564740 1 -- 172.21.15.42:0/506512744 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5570994dfda0 con 0x5570991ca1b0 2024-07-26T06:55:44.717 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.711+0000 7f37637fe640 1 --2- 172.21.15.42:0/506512744 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5570991ca1b0 0x5570996096e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:44.717 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.712+0000 7f37437fe640 10 client.25687 ms_handle_connect on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:44.717 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.712+0000 7f37637fe640 1 --2- 172.21.15.42:0/506512744 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5570991ca1b0 0x5570996096e0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b94b8e0bb989feba server_cookie=dc04354ec0824802 in_seq=0 out_seq=0 2024-07-26T06:55:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.737+0000 7f37437fe640 1 -- 172.21.15.42:0/506512744 <== mds.0 v2:172.21.15.163:6838/563658453 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5570994dfda0 con 0x5570991ca1b0 2024-07-26T06:55:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.737+0000 7f37437fe640 10 client.25687 handle_command_reply: tid=0 2024-07-26T06:55:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.737+0000 7f3771564740 1 client.25687 shutdown 2024-07-26T06:55:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.737+0000 7f3771564740 1 -- 172.21.15.42:0/506512744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557099188d80 msgr2=0x557099591630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.737+0000 7f3771564740 1 --2- 172.21.15.42:0/506512744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557099188d80 0x557099591630 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f37540ab6c0 tx=0x7f3754003d90 comp rx=0 tx=0).stop 2024-07-26T06:55:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.737+0000 7f3771564740 1 -- 172.21.15.42:0/506512744 shutdown_connections 2024-07-26T06:55:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.737+0000 7f3771564740 1 --2- 172.21.15.42:0/506512744 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x5570991ca1b0 0x5570996096e0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.737+0000 7f3771564740 1 --2- 172.21.15.42:0/506512744 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5570991a94d0 0x557099590510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.737+0000 7f37437fe640 0 client.25687 ms_handle_reset on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.737+0000 7f3771564740 1 --2- 172.21.15.42:0/506512744 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570991a12f0 0x55709958fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.737+0000 7f3771564740 1 --2- 172.21.15.42:0/506512744 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x557099188d80 0x557099591630 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.737+0000 7f3771564740 1 -- 172.21.15.42:0/506512744 >> 172.21.15.42:0/506512744 conn(0x5570991be300 msgr2=0x5570995a4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.737+0000 7f3771564740 1 -- 172.21.15.42:0/506512744 shutdown_connections 2024-07-26T06:55:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.737+0000 7f3771564740 1 -- 172.21.15.42:0/506512744 wait complete. 2024-07-26T06:55:44.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.737+0000 7f3771564740 20 client.25687 trim_cache size 0 max 16384 2024-07-26T06:55:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:{ 2024-07-26T06:55:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "message": "", 2024-07-26T06:55:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout: "return_code": 0 2024-07-26T06:55:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stdout:} 2024-07-26T06:55:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.738+0000 7f376a570640 1 -- 172.21.15.42:0/2411303981 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f374c0b3c50 msgr2=0x7f374c0b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.738+0000 7f376a570640 1 --2- 172.21.15.42:0/2411303981 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f374c0b3c50 0x7f374c0b6160 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f375c005fd0 tx=0x7f375c009540 comp rx=0 tx=0).stop 2024-07-26T06:55:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.738+0000 7f376a570640 1 -- 172.21.15.42:0/2411303981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3764101fe0 msgr2=0x7f37641999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.738+0000 7f376a570640 1 --2- 172.21.15.42:0/2411303981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3764101fe0 0x7f37641999e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f375400ea60 tx=0x7f375400ef20 comp rx=0 tx=0).stop 2024-07-26T06:55:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.738+0000 7f376a570640 1 -- 172.21.15.42:0/2411303981 shutdown_connections 2024-07-26T06:55:44.743 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.738+0000 7f376a570640 1 --2- 172.21.15.42:0/2411303981 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f374c0b3c50 0x7f374c0b6160 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.743 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.738+0000 7f376a570640 1 --2- 172.21.15.42:0/2411303981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3764103a00 0x7f376419ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.743 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.738+0000 7f376a570640 1 --2- 172.21.15.42:0/2411303981 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3764103060 0x7f376419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.743 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.738+0000 7f376a570640 1 --2- 172.21.15.42:0/2411303981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3764101fe0 0x7f37641999e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:44.743 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.738+0000 7f376a570640 1 -- 172.21.15.42:0/2411303981 >> 172.21.15.42:0/2411303981 conn(0x7f37640fd960 msgr2=0x7f37640ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:44.743 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.738+0000 7f376a570640 1 -- 172.21.15.42:0/2411303981 shutdown_connections 2024-07-26T06:55:44.743 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:44.738+0000 7f376a570640 1 -- 172.21.15.42:0/2411303981 wait complete. 2024-07-26T06:55:45.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:44 smithi164 ceph-mon[36970]: pgmap v562: 97 pgs: 97 active+clean; 76 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.7 MiB/s wr, 654 op/s 2024-07-26T06:55:45.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:44 smithi163 ceph-mon[36873]: pgmap v562: 97 pgs: 97 active+clean; 76 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.7 MiB/s wr, 654 op/s 2024-07-26T06:55:45.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:44 smithi042 ceph-mon[32775]: pgmap v562: 97 pgs: 97 active+clean; 76 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.7 MiB/s wr, 654 op/s 2024-07-26T06:55:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:46 smithi164 ceph-mon[36970]: pgmap v563: 97 pgs: 97 active+clean; 73 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 2.1 MiB/s wr, 831 op/s 2024-07-26T06:55:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:46 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:55:47.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:46 smithi163 ceph-mon[36873]: pgmap v563: 97 pgs: 97 active+clean; 73 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 2.1 MiB/s wr, 831 op/s 2024-07-26T06:55:47.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:46 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:55:47.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:46 smithi042 ceph-mon[32775]: pgmap v563: 97 pgs: 97 active+clean; 73 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 2.1 MiB/s wr, 831 op/s 2024-07-26T06:55:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:46 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:55:49.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:48 smithi164 ceph-mon[36970]: pgmap v564: 97 pgs: 97 active+clean; 62 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 2.5 MiB/s wr, 966 op/s 2024-07-26T06:55:49.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:48 smithi163 ceph-mon[36873]: pgmap v564: 97 pgs: 97 active+clean; 62 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 2.5 MiB/s wr, 966 op/s 2024-07-26T06:55:49.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:48 smithi042 ceph-mon[32775]: pgmap v564: 97 pgs: 97 active+clean; 62 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 2.5 MiB/s wr, 966 op/s 2024-07-26T06:55:51.140 INFO:tasks.workunit.client.0.smithi042.stdout:All 50000 operations completed A-OK! 2024-07-26T06:55:51.145 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-07-26T06:55:51.145 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-07-26T06:55:51.208 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:50 smithi042 ceph-mon[32775]: pgmap v565: 97 pgs: 97 active+clean; 63 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 2.7 MiB/s wr, 1.03k op/s 2024-07-26T06:55:51.221 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:50 smithi164 ceph-mon[36970]: pgmap v565: 97 pgs: 97 active+clean; 63 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 2.7 MiB/s wr, 1.03k op/s 2024-07-26T06:55:51.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:50 smithi163 ceph-mon[36873]: pgmap v565: 97 pgs: 97 active+clean; 63 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 2.7 MiB/s wr, 1.03k op/s 2024-07-26T06:55:52.193 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:51 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:55:52.194 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:51 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:55:52.223 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:51 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:55:52.321 INFO:tasks.quiescer.fs.[cephfs]:Total duration (43.9) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-07-26T06:55:52.321 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.95 2024-07-26T06:55:52.321 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- ceph fs dump --format=json 2024-07-26T06:55:52.473 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:55:52.473 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: *** Caught signal (Aborted) ** 2024-07-26T06:55:52.473 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: in thread 7f07ceddc640 thread_name:ms_dispatch 2024-07-26T06:55:52.473 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:55:52.473 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f07d41b36f0] 2024-07-26T06:55:52.474 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f07d420094c] 2024-07-26T06:55:52.474 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 3: raise() 2024-07-26T06:55:52.474 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 4: abort() 2024-07-26T06:55:52.474 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 5: /lib64/libc.so.6(+0x2871b) [0x7f07d419d71b] 2024-07-26T06:55:52.474 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 6: /lib64/libc.so.6(+0x37386) [0x7f07d41ac386] 2024-07-26T06:55:52.474 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55e4128d56b9] 2024-07-26T06:55:52.474 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55e412a9380c] 2024-07-26T06:55:52.474 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55e4129d32ca] 2024-07-26T06:55:52.475 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55e4129a3753] 2024-07-26T06:55:52.475 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55e4129a4a1b] 2024-07-26T06:55:52.475 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55e412988752] 2024-07-26T06:55:52.475 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55e4128d6a56] 2024-07-26T06:55:52.475 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55e4128da637] 2024-07-26T06:55:52.475 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55e4128daf5e] 2024-07-26T06:55:52.475 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55e4128be03a] 2024-07-26T06:55:52.475 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f07d4a80407] 2024-07-26T06:55:52.475 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 18: (DispatchQueue::entry()+0x60f) [0x7f07d4a80cf9] 2024-07-26T06:55:52.475 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f07d4b5d5a1] 2024-07-26T06:55:52.475 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 20: (Thread::entry_wrapper()+0x43) [0x7f07d490acdd] 2024-07-26T06:55:52.476 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 21: (Thread::_entry_func(void*)+0xd) [0x7f07d490acf9] 2024-07-26T06:55:52.476 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 22: /lib64/libc.so.6(+0x89c02) [0x7f07d41fec02] 2024-07-26T06:55:52.476 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f07d4283c40] 2024-07-26T06:55:52.476 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 2024-07-26T06:55:52.194+0000 7f07ceddc640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:55:52.476 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: in thread 7f07ceddc640 thread_name:ms_dispatch 2024-07-26T06:55:52.476 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 2024-07-26T06:55:52.476 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:55:52.476 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f07d41b36f0] 2024-07-26T06:55:52.476 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f07d420094c] 2024-07-26T06:55:52.476 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 3: raise() 2024-07-26T06:55:52.477 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 4: abort() 2024-07-26T06:55:52.477 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 5: /lib64/libc.so.6(+0x2871b) [0x7f07d419d71b] 2024-07-26T06:55:52.477 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 6: /lib64/libc.so.6(+0x37386) [0x7f07d41ac386] 2024-07-26T06:55:52.477 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55e4128d56b9] 2024-07-26T06:55:52.477 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55e412a9380c] 2024-07-26T06:55:52.477 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55e4129d32ca] 2024-07-26T06:55:52.477 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55e4129a3753] 2024-07-26T06:55:52.477 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55e4129a4a1b] 2024-07-26T06:55:52.477 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55e412988752] 2024-07-26T06:55:52.477 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55e4128d6a56] 2024-07-26T06:55:52.477 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55e4128da637] 2024-07-26T06:55:52.477 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55e4128daf5e] 2024-07-26T06:55:52.477 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55e4128be03a] 2024-07-26T06:55:52.478 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f07d4a80407] 2024-07-26T06:55:52.478 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 18: (DispatchQueue::entry()+0x60f) [0x7f07d4a80cf9] 2024-07-26T06:55:52.478 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f07d4b5d5a1] 2024-07-26T06:55:52.478 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 20: (Thread::entry_wrapper()+0x43) [0x7f07d490acdd] 2024-07-26T06:55:52.479 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 21: (Thread::_entry_func(void*)+0xd) [0x7f07d490acf9] 2024-07-26T06:55:52.479 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 22: /lib64/libc.so.6(+0x89c02) [0x7f07d41fec02] 2024-07-26T06:55:52.479 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f07d4283c40] 2024-07-26T06:55:52.479 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:55:52.479 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 2024-07-26T06:55:52.479 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 0> 2024-07-26T06:55:52.194+0000 7f07ceddc640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:55:52.479 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: in thread 7f07ceddc640 thread_name:ms_dispatch 2024-07-26T06:55:52.479 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 2024-07-26T06:55:52.479 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:55:52.479 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f07d41b36f0] 2024-07-26T06:55:52.479 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f07d420094c] 2024-07-26T06:55:52.480 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 3: raise() 2024-07-26T06:55:52.480 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 4: abort() 2024-07-26T06:55:52.480 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 5: /lib64/libc.so.6(+0x2871b) [0x7f07d419d71b] 2024-07-26T06:55:52.480 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 6: /lib64/libc.so.6(+0x37386) [0x7f07d41ac386] 2024-07-26T06:55:52.480 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55e4128d56b9] 2024-07-26T06:55:52.480 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55e412a9380c] 2024-07-26T06:55:52.480 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55e4129d32ca] 2024-07-26T06:55:52.480 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55e4129a3753] 2024-07-26T06:55:52.480 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55e4129a4a1b] 2024-07-26T06:55:52.481 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55e412988752] 2024-07-26T06:55:52.481 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55e4128d6a56] 2024-07-26T06:55:52.481 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55e4128da637] 2024-07-26T06:55:52.481 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55e4128daf5e] 2024-07-26T06:55:52.481 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55e4128be03a] 2024-07-26T06:55:52.481 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f07d4a80407] 2024-07-26T06:55:52.481 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 18: (DispatchQueue::entry()+0x60f) [0x7f07d4a80cf9] 2024-07-26T06:55:52.481 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f07d4b5d5a1] 2024-07-26T06:55:52.481 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 20: (Thread::entry_wrapper()+0x43) [0x7f07d490acdd] 2024-07-26T06:55:52.481 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 21: (Thread::_entry_func(void*)+0xd) [0x7f07d490acf9] 2024-07-26T06:55:52.481 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 22: /lib64/libc.so.6(+0x89c02) [0x7f07d41fec02] 2024-07-26T06:55:52.481 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f07d4283c40] 2024-07-26T06:55:52.481 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:55:52.482 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 2024-07-26T06:55:52.482 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 0> 2024-07-26T06:55:52.194+0000 7f07ceddc640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:55:52.482 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: in thread 7f07ceddc640 thread_name:ms_dispatch 2024-07-26T06:55:52.482 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 2024-07-26T06:55:52.482 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:55:52.482 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f07d41b36f0] 2024-07-26T06:55:52.482 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f07d420094c] 2024-07-26T06:55:52.482 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 3: raise() 2024-07-26T06:55:52.482 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 4: abort() 2024-07-26T06:55:52.482 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 5: /lib64/libc.so.6(+0x2871b) [0x7f07d419d71b] 2024-07-26T06:55:52.483 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 6: /lib64/libc.so.6(+0x37386) [0x7f07d41ac386] 2024-07-26T06:55:52.483 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55e4128d56b9] 2024-07-26T06:55:52.483 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55e412a9380c] 2024-07-26T06:55:52.483 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55e4129d32ca] 2024-07-26T06:55:52.483 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55e4129a3753] 2024-07-26T06:55:52.483 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55e4129a4a1b] 2024-07-26T06:55:52.483 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55e412988752] 2024-07-26T06:55:52.483 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55e4128d6a56] 2024-07-26T06:55:52.483 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55e4128da637] 2024-07-26T06:55:52.483 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55e4128daf5e] 2024-07-26T06:55:52.483 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55e4128be03a] 2024-07-26T06:55:52.484 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f07d4a80407] 2024-07-26T06:55:52.484 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 18: (DispatchQueue::entry()+0x60f) [0x7f07d4a80cf9] 2024-07-26T06:55:52.484 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f07d4b5d5a1] 2024-07-26T06:55:52.484 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 20: (Thread::entry_wrapper()+0x43) [0x7f07d490acdd] 2024-07-26T06:55:52.484 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 21: (Thread::_entry_func(void*)+0xd) [0x7f07d490acf9] 2024-07-26T06:55:52.484 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 22: /lib64/libc.so.6(+0x89c02) [0x7f07d41fec02] 2024-07-26T06:55:52.484 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f07d4283c40] 2024-07-26T06:55:52.484 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:55:52.484 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[66874]: 2024-07-26T06:55:52.484 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:52 smithi164 podman[70126]: 2024-07-26 06:55:52.250646455 +0000 UTC m=+0.019633961 container died 016ec20fcc4a6cad347b127dc4f1d44df1a72be5b76c2edc452451b641173eda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T06:55:52.507 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:55:52.507 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: *** Caught signal (Aborted) ** 2024-07-26T06:55:52.507 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: in thread 7fe539bd0640 thread_name:ms_dispatch 2024-07-26T06:55:52.508 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:55:52.508 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe53efa76f0] 2024-07-26T06:55:52.508 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe53eff494c] 2024-07-26T06:55:52.508 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 3: raise() 2024-07-26T06:55:52.508 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 4: abort() 2024-07-26T06:55:52.508 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe53ef9171b] 2024-07-26T06:55:52.508 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 6: /lib64/libc.so.6(+0x37386) [0x7fe53efa0386] 2024-07-26T06:55:52.508 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55564439f6b9] 2024-07-26T06:55:52.508 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55564455d80c] 2024-07-26T06:55:52.508 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55564449d2ca] 2024-07-26T06:55:52.508 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55564446d753] 2024-07-26T06:55:52.508 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55564446ea1b] 2024-07-26T06:55:52.509 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x555644452752] 2024-07-26T06:55:52.509 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5556443a0a56] 2024-07-26T06:55:52.509 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5556443a4637] 2024-07-26T06:55:52.509 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5556443a4f5e] 2024-07-26T06:55:52.509 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55564438803a] 2024-07-26T06:55:52.509 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe53f874407] 2024-07-26T06:55:52.509 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 18: (DispatchQueue::entry()+0x60f) [0x7fe53f874cf9] 2024-07-26T06:55:52.509 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe53f9515a1] 2024-07-26T06:55:52.509 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:55:52.509 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: *** Caught signal (Aborted) ** 2024-07-26T06:55:52.510 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: in thread 7fd8708a6640 thread_name:ms_dispatch 2024-07-26T06:55:52.510 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:55:52.510 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd875c7d6f0] 2024-07-26T06:55:52.510 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd875cca94c] 2024-07-26T06:55:52.510 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 3: raise() 2024-07-26T06:55:52.510 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 4: abort() 2024-07-26T06:55:52.510 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd875c6771b] 2024-07-26T06:55:52.510 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 6: /lib64/libc.so.6(+0x37386) [0x7fd875c76386] 2024-07-26T06:55:52.510 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d7e7ce26b9] 2024-07-26T06:55:52.510 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d7e7ea080c] 2024-07-26T06:55:52.510 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55d7e7de02ca] 2024-07-26T06:55:52.510 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55d7e7db0753] 2024-07-26T06:55:52.511 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d7e7db1a1b] 2024-07-26T06:55:52.511 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d7e7d95752] 2024-07-26T06:55:52.511 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d7e7ce3a56] 2024-07-26T06:55:52.511 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d7e7ce7637] 2024-07-26T06:55:52.511 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d7e7ce7f5e] 2024-07-26T06:55:52.511 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d7e7ccb03a] 2024-07-26T06:55:52.511 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd87654a407] 2024-07-26T06:55:52.511 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 18: (DispatchQueue::entry()+0x60f) [0x7fd87654acf9] 2024-07-26T06:55:52.511 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd8766275a1] 2024-07-26T06:55:52.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:55:52.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: *** Caught signal (Aborted) ** 2024-07-26T06:55:52.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: in thread 7fc997f78640 thread_name:ms_dispatch 2024-07-26T06:55:52.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:55:52.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc99d34f6f0] 2024-07-26T06:55:52.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc99d39c94c] 2024-07-26T06:55:52.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 3: raise() 2024-07-26T06:55:52.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 4: abort() 2024-07-26T06:55:52.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 5: /lib64/libc.so.6(+0x2871b) [0x7fc99d33971b] 2024-07-26T06:55:52.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 6: /lib64/libc.so.6(+0x37386) [0x7fc99d348386] 2024-07-26T06:55:52.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ba6de7c6b9] 2024-07-26T06:55:52.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ba6e03a80c] 2024-07-26T06:55:52.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ba6df7a2ca] 2024-07-26T06:55:52.512 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ba6df4a753] 2024-07-26T06:55:52.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ba6df4ba1b] 2024-07-26T06:55:52.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ba6df2f752] 2024-07-26T06:55:52.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ba6de7da56] 2024-07-26T06:55:52.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ba6de81637] 2024-07-26T06:55:52.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ba6de81f5e] 2024-07-26T06:55:52.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ba6de6503a] 2024-07-26T06:55:52.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fc99dc1c407] 2024-07-26T06:55:52.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 18: (DispatchQueue::entry()+0x60f) [0x7fc99dc1ccf9] 2024-07-26T06:55:52.513 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fc99dcf95a1] 2024-07-26T06:55:52.514 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 20: (Thread::entry_wrapper()+0x43) [0x7fe53f6fecdd] 2024-07-26T06:55:52.514 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe53f6fecf9] 2024-07-26T06:55:52.514 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe53eff2c02] 2024-07-26T06:55:52.514 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe53f077c40] 2024-07-26T06:55:52.514 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 2024-07-26T06:55:52.193+0000 7fe539bd0640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:55:52.514 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: in thread 7fe539bd0640 thread_name:ms_dispatch 2024-07-26T06:55:52.514 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 2024-07-26T06:55:52.514 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:55:52.514 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe53efa76f0] 2024-07-26T06:55:52.514 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe53eff494c] 2024-07-26T06:55:52.515 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 3: raise() 2024-07-26T06:55:52.515 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 4: abort() 2024-07-26T06:55:52.515 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe53ef9171b] 2024-07-26T06:55:52.515 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 6: /lib64/libc.so.6(+0x37386) [0x7fe53efa0386] 2024-07-26T06:55:52.515 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55564439f6b9] 2024-07-26T06:55:52.515 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55564455d80c] 2024-07-26T06:55:52.515 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55564449d2ca] 2024-07-26T06:55:52.515 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55564446d753] 2024-07-26T06:55:52.516 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55564446ea1b] 2024-07-26T06:55:52.516 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x555644452752] 2024-07-26T06:55:52.516 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5556443a0a56] 2024-07-26T06:55:52.516 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5556443a4637] 2024-07-26T06:55:52.516 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5556443a4f5e] 2024-07-26T06:55:52.516 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55564438803a] 2024-07-26T06:55:52.516 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe53f874407] 2024-07-26T06:55:52.516 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 18: (DispatchQueue::entry()+0x60f) [0x7fe53f874cf9] 2024-07-26T06:55:52.516 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe53f9515a1] 2024-07-26T06:55:52.516 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 20: (Thread::entry_wrapper()+0x43) [0x7fe53f6fecdd] 2024-07-26T06:55:52.516 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe53f6fecf9] 2024-07-26T06:55:52.517 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe53eff2c02] 2024-07-26T06:55:52.517 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe53f077c40] 2024-07-26T06:55:52.517 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:55:52.517 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 2024-07-26T06:55:52.517 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 0> 2024-07-26T06:55:52.193+0000 7fe539bd0640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:55:52.517 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: in thread 7fe539bd0640 thread_name:ms_dispatch 2024-07-26T06:55:52.517 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 2024-07-26T06:55:52.517 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:55:52.517 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe53efa76f0] 2024-07-26T06:55:52.517 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe53eff494c] 2024-07-26T06:55:52.517 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 3: raise() 2024-07-26T06:55:52.517 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 4: abort() 2024-07-26T06:55:52.517 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe53ef9171b] 2024-07-26T06:55:52.518 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 6: /lib64/libc.so.6(+0x37386) [0x7fe53efa0386] 2024-07-26T06:55:52.518 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55564439f6b9] 2024-07-26T06:55:52.518 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55564455d80c] 2024-07-26T06:55:52.518 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55564449d2ca] 2024-07-26T06:55:52.518 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55564446d753] 2024-07-26T06:55:52.518 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55564446ea1b] 2024-07-26T06:55:52.518 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x555644452752] 2024-07-26T06:55:52.518 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5556443a0a56] 2024-07-26T06:55:52.518 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5556443a4637] 2024-07-26T06:55:52.518 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5556443a4f5e] 2024-07-26T06:55:52.518 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55564438803a] 2024-07-26T06:55:52.519 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe53f874407] 2024-07-26T06:55:52.519 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 20: (Thread::entry_wrapper()+0x43) [0x7fd8763d4cdd] 2024-07-26T06:55:52.519 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd8763d4cf9] 2024-07-26T06:55:52.519 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd875cc8c02] 2024-07-26T06:55:52.519 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd875d4dc40] 2024-07-26T06:55:52.519 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 2024-07-26T06:55:52.193+0000 7fd8708a6640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:55:52.519 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: in thread 7fd8708a6640 thread_name:ms_dispatch 2024-07-26T06:55:52.519 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 2024-07-26T06:55:52.520 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:55:52.520 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd875c7d6f0] 2024-07-26T06:55:52.520 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd875cca94c] 2024-07-26T06:55:52.520 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 3: raise() 2024-07-26T06:55:52.520 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 4: abort() 2024-07-26T06:55:52.520 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd875c6771b] 2024-07-26T06:55:52.520 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 6: /lib64/libc.so.6(+0x37386) [0x7fd875c76386] 2024-07-26T06:55:52.520 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d7e7ce26b9] 2024-07-26T06:55:52.520 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d7e7ea080c] 2024-07-26T06:55:52.520 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55d7e7de02ca] 2024-07-26T06:55:52.520 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55d7e7db0753] 2024-07-26T06:55:52.521 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d7e7db1a1b] 2024-07-26T06:55:52.521 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d7e7d95752] 2024-07-26T06:55:52.521 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d7e7ce3a56] 2024-07-26T06:55:52.521 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d7e7ce7637] 2024-07-26T06:55:52.521 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d7e7ce7f5e] 2024-07-26T06:55:52.521 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d7e7ccb03a] 2024-07-26T06:55:52.521 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd87654a407] 2024-07-26T06:55:52.521 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 18: (DispatchQueue::entry()+0x60f) [0x7fd87654acf9] 2024-07-26T06:55:52.521 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd8766275a1] 2024-07-26T06:55:52.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 20: (Thread::entry_wrapper()+0x43) [0x7fc99daa6cdd] 2024-07-26T06:55:52.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc99daa6cf9] 2024-07-26T06:55:52.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 22: /lib64/libc.so.6(+0x89c02) [0x7fc99d39ac02] 2024-07-26T06:55:52.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fc99d41fc40] 2024-07-26T06:55:52.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 2024-07-26T06:55:52.193+0000 7fc997f78640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:55:52.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: in thread 7fc997f78640 thread_name:ms_dispatch 2024-07-26T06:55:52.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 2024-07-26T06:55:52.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:55:52.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc99d34f6f0] 2024-07-26T06:55:52.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc99d39c94c] 2024-07-26T06:55:52.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 3: raise() 2024-07-26T06:55:52.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 4: abort() 2024-07-26T06:55:52.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 5: /lib64/libc.so.6(+0x2871b) [0x7fc99d33971b] 2024-07-26T06:55:52.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 6: /lib64/libc.so.6(+0x37386) [0x7fc99d348386] 2024-07-26T06:55:52.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ba6de7c6b9] 2024-07-26T06:55:52.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ba6e03a80c] 2024-07-26T06:55:52.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ba6df7a2ca] 2024-07-26T06:55:52.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ba6df4a753] 2024-07-26T06:55:52.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ba6df4ba1b] 2024-07-26T06:55:52.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ba6df2f752] 2024-07-26T06:55:52.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ba6de7da56] 2024-07-26T06:55:52.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ba6de81637] 2024-07-26T06:55:52.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ba6de81f5e] 2024-07-26T06:55:52.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ba6de6503a] 2024-07-26T06:55:52.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fc99dc1c407] 2024-07-26T06:55:52.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 18: (DispatchQueue::entry()+0x60f) [0x7fc99dc1ccf9] 2024-07-26T06:55:52.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fc99dcf95a1] 2024-07-26T06:55:52.524 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 18: (DispatchQueue::entry()+0x60f) [0x7fe53f874cf9] 2024-07-26T06:55:52.524 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe53f9515a1] 2024-07-26T06:55:52.524 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 20: (Thread::entry_wrapper()+0x43) [0x7fe53f6fecdd] 2024-07-26T06:55:52.524 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe53f6fecf9] 2024-07-26T06:55:52.524 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe53eff2c02] 2024-07-26T06:55:52.524 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe53f077c40] 2024-07-26T06:55:52.524 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:55:52.525 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 2024-07-26T06:55:52.525 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 0> 2024-07-26T06:55:52.193+0000 7fe539bd0640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:55:52.525 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: in thread 7fe539bd0640 thread_name:ms_dispatch 2024-07-26T06:55:52.525 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 2024-07-26T06:55:52.525 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:55:52.525 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe53efa76f0] 2024-07-26T06:55:52.525 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe53eff494c] 2024-07-26T06:55:52.525 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 3: raise() 2024-07-26T06:55:52.525 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 4: abort() 2024-07-26T06:55:52.525 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe53ef9171b] 2024-07-26T06:55:52.525 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 6: /lib64/libc.so.6(+0x37386) [0x7fe53efa0386] 2024-07-26T06:55:52.525 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55564439f6b9] 2024-07-26T06:55:52.526 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55564455d80c] 2024-07-26T06:55:52.526 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55564449d2ca] 2024-07-26T06:55:52.526 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55564446d753] 2024-07-26T06:55:52.526 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55564446ea1b] 2024-07-26T06:55:52.526 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x555644452752] 2024-07-26T06:55:52.526 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5556443a0a56] 2024-07-26T06:55:52.526 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5556443a4637] 2024-07-26T06:55:52.526 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5556443a4f5e] 2024-07-26T06:55:52.526 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55564438803a] 2024-07-26T06:55:52.527 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe53f874407] 2024-07-26T06:55:52.527 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 18: (DispatchQueue::entry()+0x60f) [0x7fe53f874cf9] 2024-07-26T06:55:52.527 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe53f9515a1] 2024-07-26T06:55:52.527 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 20: (Thread::entry_wrapper()+0x43) [0x7fe53f6fecdd] 2024-07-26T06:55:52.527 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe53f6fecf9] 2024-07-26T06:55:52.527 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe53eff2c02] 2024-07-26T06:55:52.527 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe53f077c40] 2024-07-26T06:55:52.527 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:55:52.527 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[67028]: 2024-07-26T06:55:52.527 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:52 smithi163 podman[69833]: 2024-07-26 06:55:52.25943207 +0000 UTC m=+0.021048729 container died 464ee1df1f53533c9a64fd32aeb32517814d1cba132245c46a16d7bc4aaf372f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-07-26T06:55:52.528 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 20: (Thread::entry_wrapper()+0x43) [0x7fd8763d4cdd] 2024-07-26T06:55:52.528 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd8763d4cf9] 2024-07-26T06:55:52.528 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd875cc8c02] 2024-07-26T06:55:52.528 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd875d4dc40] 2024-07-26T06:55:52.528 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:55:52.528 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 2024-07-26T06:55:52.528 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 0> 2024-07-26T06:55:52.193+0000 7fd8708a6640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:55:52.528 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: in thread 7fd8708a6640 thread_name:ms_dispatch 2024-07-26T06:55:52.528 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 2024-07-26T06:55:52.528 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:55:52.528 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd875c7d6f0] 2024-07-26T06:55:52.528 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd875cca94c] 2024-07-26T06:55:52.528 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 3: raise() 2024-07-26T06:55:52.528 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 4: abort() 2024-07-26T06:55:52.529 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd875c6771b] 2024-07-26T06:55:52.529 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 6: /lib64/libc.so.6(+0x37386) [0x7fd875c76386] 2024-07-26T06:55:52.529 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d7e7ce26b9] 2024-07-26T06:55:52.529 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d7e7ea080c] 2024-07-26T06:55:52.529 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55d7e7de02ca] 2024-07-26T06:55:52.529 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55d7e7db0753] 2024-07-26T06:55:52.529 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d7e7db1a1b] 2024-07-26T06:55:52.529 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d7e7d95752] 2024-07-26T06:55:52.529 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d7e7ce3a56] 2024-07-26T06:55:52.529 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d7e7ce7637] 2024-07-26T06:55:52.529 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d7e7ce7f5e] 2024-07-26T06:55:52.529 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d7e7ccb03a] 2024-07-26T06:55:52.529 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd87654a407] 2024-07-26T06:55:52.530 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 20: (Thread::entry_wrapper()+0x43) [0x7fc99daa6cdd] 2024-07-26T06:55:52.530 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc99daa6cf9] 2024-07-26T06:55:52.530 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 22: /lib64/libc.so.6(+0x89c02) [0x7fc99d39ac02] 2024-07-26T06:55:52.530 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fc99d41fc40] 2024-07-26T06:55:52.530 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:55:52.530 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 2024-07-26T06:55:52.530 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 0> 2024-07-26T06:55:52.193+0000 7fc997f78640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:55:52.531 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: in thread 7fc997f78640 thread_name:ms_dispatch 2024-07-26T06:55:52.531 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 2024-07-26T06:55:52.531 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:55:52.531 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc99d34f6f0] 2024-07-26T06:55:52.531 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc99d39c94c] 2024-07-26T06:55:52.531 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 3: raise() 2024-07-26T06:55:52.531 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 4: abort() 2024-07-26T06:55:52.531 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 5: /lib64/libc.so.6(+0x2871b) [0x7fc99d33971b] 2024-07-26T06:55:52.531 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 6: /lib64/libc.so.6(+0x37386) [0x7fc99d348386] 2024-07-26T06:55:52.531 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ba6de7c6b9] 2024-07-26T06:55:52.531 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ba6e03a80c] 2024-07-26T06:55:52.531 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ba6df7a2ca] 2024-07-26T06:55:52.531 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ba6df4a753] 2024-07-26T06:55:52.531 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ba6df4ba1b] 2024-07-26T06:55:52.531 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ba6df2f752] 2024-07-26T06:55:52.531 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ba6de7da56] 2024-07-26T06:55:52.532 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ba6de81637] 2024-07-26T06:55:52.532 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ba6de81f5e] 2024-07-26T06:55:52.532 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ba6de6503a] 2024-07-26T06:55:52.532 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fc99dc1c407] 2024-07-26T06:55:52.532 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 18: (DispatchQueue::entry()+0x60f) [0x7fd87654acf9] 2024-07-26T06:55:52.532 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd8766275a1] 2024-07-26T06:55:52.532 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 20: (Thread::entry_wrapper()+0x43) [0x7fd8763d4cdd] 2024-07-26T06:55:52.532 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd8763d4cf9] 2024-07-26T06:55:52.532 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd875cc8c02] 2024-07-26T06:55:52.532 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd875d4dc40] 2024-07-26T06:55:52.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:55:52.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 2024-07-26T06:55:52.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 0> 2024-07-26T06:55:52.193+0000 7fd8708a6640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:55:52.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: in thread 7fd8708a6640 thread_name:ms_dispatch 2024-07-26T06:55:52.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 2024-07-26T06:55:52.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:55:52.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd875c7d6f0] 2024-07-26T06:55:52.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd875cca94c] 2024-07-26T06:55:52.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 3: raise() 2024-07-26T06:55:52.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 4: abort() 2024-07-26T06:55:52.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd875c6771b] 2024-07-26T06:55:52.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 6: /lib64/libc.so.6(+0x37386) [0x7fd875c76386] 2024-07-26T06:55:52.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d7e7ce26b9] 2024-07-26T06:55:52.534 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d7e7ea080c] 2024-07-26T06:55:52.534 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55d7e7de02ca] 2024-07-26T06:55:52.534 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55d7e7db0753] 2024-07-26T06:55:52.534 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d7e7db1a1b] 2024-07-26T06:55:52.534 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d7e7d95752] 2024-07-26T06:55:52.534 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d7e7ce3a56] 2024-07-26T06:55:52.534 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d7e7ce7637] 2024-07-26T06:55:52.534 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d7e7ce7f5e] 2024-07-26T06:55:52.534 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 18: (DispatchQueue::entry()+0x60f) [0x7fc99dc1ccf9] 2024-07-26T06:55:52.534 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fc99dcf95a1] 2024-07-26T06:55:52.535 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 20: (Thread::entry_wrapper()+0x43) [0x7fc99daa6cdd] 2024-07-26T06:55:52.535 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc99daa6cf9] 2024-07-26T06:55:52.535 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 22: /lib64/libc.so.6(+0x89c02) [0x7fc99d39ac02] 2024-07-26T06:55:52.535 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fc99d41fc40] 2024-07-26T06:55:52.535 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:55:52.535 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 2024-07-26T06:55:52.535 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 0> 2024-07-26T06:55:52.193+0000 7fc997f78640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:55:52.535 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: in thread 7fc997f78640 thread_name:ms_dispatch 2024-07-26T06:55:52.535 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 2024-07-26T06:55:52.535 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:55:52.535 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc99d34f6f0] 2024-07-26T06:55:52.535 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc99d39c94c] 2024-07-26T06:55:52.536 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 3: raise() 2024-07-26T06:55:52.536 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 4: abort() 2024-07-26T06:55:52.536 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 5: /lib64/libc.so.6(+0x2871b) [0x7fc99d33971b] 2024-07-26T06:55:52.536 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 6: /lib64/libc.so.6(+0x37386) [0x7fc99d348386] 2024-07-26T06:55:52.536 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ba6de7c6b9] 2024-07-26T06:55:52.536 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ba6e03a80c] 2024-07-26T06:55:52.536 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ba6df7a2ca] 2024-07-26T06:55:52.536 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ba6df4a753] 2024-07-26T06:55:52.536 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ba6df4ba1b] 2024-07-26T06:55:52.536 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ba6df2f752] 2024-07-26T06:55:52.536 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ba6de7da56] 2024-07-26T06:55:52.536 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ba6de81637] 2024-07-26T06:55:52.536 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ba6de81f5e] 2024-07-26T06:55:52.537 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d7e7ccb03a] 2024-07-26T06:55:52.537 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd87654a407] 2024-07-26T06:55:52.537 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 18: (DispatchQueue::entry()+0x60f) [0x7fd87654acf9] 2024-07-26T06:55:52.537 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd8766275a1] 2024-07-26T06:55:52.537 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 20: (Thread::entry_wrapper()+0x43) [0x7fd8763d4cdd] 2024-07-26T06:55:52.537 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd8763d4cf9] 2024-07-26T06:55:52.537 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd875cc8c02] 2024-07-26T06:55:52.538 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd875d4dc40] 2024-07-26T06:55:52.538 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:55:52.538 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[66426]: 2024-07-26T06:55:52.538 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ba6de6503a] 2024-07-26T06:55:52.538 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fc99dc1c407] 2024-07-26T06:55:52.538 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 18: (DispatchQueue::entry()+0x60f) [0x7fc99dc1ccf9] 2024-07-26T06:55:52.538 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fc99dcf95a1] 2024-07-26T06:55:52.538 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 20: (Thread::entry_wrapper()+0x43) [0x7fc99daa6cdd] 2024-07-26T06:55:52.538 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc99daa6cf9] 2024-07-26T06:55:52.539 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 22: /lib64/libc.so.6(+0x89c02) [0x7fc99d39ac02] 2024-07-26T06:55:52.539 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fc99d41fc40] 2024-07-26T06:55:52.539 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:55:52.539 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[66726]: 2024-07-26T06:55:52.591 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:55:53.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:52 smithi042 ceph-mon[32775]: pgmap v566: 97 pgs: 97 active+clean; 63 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 2.3 MiB/s wr, 879 op/s 2024-07-26T06:55:53.209 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:52 smithi163 ceph-mon[36873]: pgmap v566: 97 pgs: 97 active+clean; 63 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 2.3 MiB/s wr, 879 op/s 2024-07-26T06:55:53.209 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:53 smithi163 podman[69833]: 2024-07-26 06:55:53.111106643 +0000 UTC m=+0.872723348 container remove 464ee1df1f53533c9a64fd32aeb32517814d1cba132245c46a16d7bc4aaf372f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-07-26T06:55:53.210 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:53 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:55:53.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:52 smithi164 ceph-mon[36970]: pgmap v566: 97 pgs: 97 active+clean; 63 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 2.3 MiB/s wr, 879 op/s 2024-07-26T06:55:53.506 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:53 smithi163 podman[69844]: 2024-07-26 06:55:53.209884739 +0000 UTC m=+0.954347958 container died c47b4288f0737602408393f380d6895893ae90b6b7388dd5840f1a27ba7c98a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-07-26T06:55:54.184 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:53 smithi164 podman[70126]: 2024-07-26 06:55:53.932019799 +0000 UTC m=+1.701007299 container remove 016ec20fcc4a6cad347b127dc4f1d44df1a72be5b76c2edc452451b641173eda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0) 2024-07-26T06:55:54.185 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:53 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:55:54.253 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:53 smithi163 podman[69846]: 2024-07-26 06:55:53.943937011 +0000 UTC m=+1.681854803 container died eff753a144e58fd48fb44fbae126322c40d5550dddd5afda0cef03fff94cb86e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-07-26T06:55:54.254 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:53 smithi163 podman[69844]: 2024-07-26 06:55:53.924711045 +0000 UTC m=+1.669174266 container remove c47b4288f0737602408393f380d6895893ae90b6b7388dd5840f1a27ba7c98a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-07-26T06:55:54.254 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:53 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:55:54.472 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:54 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-07-26T06:55:54.472 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:55:54 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.c.service: Consumed 5.107s CPU time. 2024-07-26T06:55:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.515+0000 7f90831a1640 1 -- 172.21.15.42:0/3019423804 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f907c1022b0 msgr2=0x7f907c1026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.515+0000 7f90831a1640 1 --2- 172.21.15.42:0/3019423804 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f907c1022b0 0x7f907c1026c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9078009900 tx=0x7f907802a410 comp rx=0 tx=0).stop 2024-07-26T06:55:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.515+0000 7f90831a1640 1 -- 172.21.15.42:0/3019423804 shutdown_connections 2024-07-26T06:55:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.515+0000 7f90831a1640 1 --2- 172.21.15.42:0/3019423804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f907c108540 0x7f907c10ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.515+0000 7f90831a1640 1 --2- 172.21.15.42:0/3019423804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f907c107bf0 0x7f907c108000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.515+0000 7f90831a1640 1 --2- 172.21.15.42:0/3019423804 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f907c1022b0 0x7f907c1026c0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.515+0000 7f90831a1640 1 -- 172.21.15.42:0/3019423804 >> 172.21.15.42:0/3019423804 conn(0x7f907c0fdd00 msgr2=0x7f907c100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.515+0000 7f90831a1640 1 -- 172.21.15.42:0/3019423804 shutdown_connections 2024-07-26T06:55:54.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.515+0000 7f90831a1640 1 -- 172.21.15.42:0/3019423804 wait complete. 2024-07-26T06:55:54.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.516+0000 7f90831a1640 1 Processor -- start 2024-07-26T06:55:54.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.516+0000 7f90831a1640 1 -- start start 2024-07-26T06:55:54.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.516+0000 7f90831a1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f907c1022b0 0x7f907c19ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:54.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.516+0000 7f90831a1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f907c107bf0 0x7f907c19b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.516+0000 7f90831a1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f907c108540 0x7f907c19b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.516+0000 7f90831a1640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f907c1121f0 con 0x7f907c107bf0 2024-07-26T06:55:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.516+0000 7f90831a1640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f907c112070 con 0x7f907c1022b0 2024-07-26T06:55:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.516+0000 7f90831a1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f907c112370 con 0x7f907c108540 2024-07-26T06:55:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.516+0000 7f908199e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f907c107bf0 0x7f907c19b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.516+0000 7f908199e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f907c107bf0 0x7f907c19b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38434/0 (socket says 172.21.15.42:38434) 2024-07-26T06:55:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.516+0000 7f908199e640 1 -- 172.21.15.42:0/2667929691 learned_addr learned my addr 172.21.15.42:0/2667929691 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.516+0000 7f908219f640 1 --2- 172.21.15.42:0/2667929691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f907c1022b0 0x7f907c19ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.516+0000 7f90829a0640 1 --2- 172.21.15.42:0/2667929691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f907c108540 0x7f907c19b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.517+0000 7f908199e640 1 -- 172.21.15.42:0/2667929691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f907c108540 msgr2=0x7f907c19b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.517+0000 7f908199e640 1 --2- 172.21.15.42:0/2667929691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f907c108540 0x7f907c19b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.517+0000 7f908199e640 1 -- 172.21.15.42:0/2667929691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f907c1022b0 msgr2=0x7f907c19ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.517+0000 7f908199e640 1 --2- 172.21.15.42:0/2667929691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f907c1022b0 0x7f907c19ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.517+0000 7f908199e640 1 -- 172.21.15.42:0/2667929691 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f907c1a5db0 con 0x7f907c107bf0 2024-07-26T06:55:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.517+0000 7f908199e640 1 --2- 172.21.15.42:0/2667929691 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f907c107bf0 0x7f907c19b060 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f906c00e9f0 tx=0x7f906c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.517+0000 7f90737fe640 1 -- 172.21.15.42:0/2667929691 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906c00ccc0 con 0x7f907c107bf0 2024-07-26T06:55:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.517+0000 7f90737fe640 1 -- 172.21.15.42:0/2667929691 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f906c0108e0 con 0x7f907c107bf0 2024-07-26T06:55:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.517+0000 7f90831a1640 1 -- 172.21.15.42:0/2667929691 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f907c1a60a0 con 0x7f907c107bf0 2024-07-26T06:55:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.517+0000 7f90737fe640 1 -- 172.21.15.42:0/2667929691 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906c018860 con 0x7f907c107bf0 2024-07-26T06:55:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.517+0000 7f90831a1640 1 -- 172.21.15.42:0/2667929691 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f907c1a6630 con 0x7f907c107bf0 2024-07-26T06:55:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.518+0000 7f90831a1640 1 -- 172.21.15.42:0/2667929691 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f907c103530 con 0x7f907c107bf0 2024-07-26T06:55:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.520+0000 7f90737fe640 1 -- 172.21.15.42:0/2667929691 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7f906c010430 con 0x7f907c107bf0 2024-07-26T06:55:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.520+0000 7f90737fe640 1 --2- 172.21.15.42:0/2667929691 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f90680b3c80 0x7f90680b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.521+0000 7f90737fe640 1 -- 172.21.15.42:0/2667929691 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7f906c0e3e80 con 0x7f907c107bf0 2024-07-26T06:55:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.521+0000 7f908219f640 1 --2- 172.21.15.42:0/2667929691 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f90680b3c80 0x7f90680b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:54.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.521+0000 7f908219f640 1 --2- 172.21.15.42:0/2667929691 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f90680b3c80 0x7f90680b6190 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9078000c00 tx=0x7f9078031400 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:54.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.525+0000 7f90737fe640 1 -- 172.21.15.42:0/2667929691 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f906c0ab4b0 con 0x7f907c107bf0 2024-07-26T06:55:54.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.749+0000 7f90831a1640 1 -- 172.21.15.42:0/2667929691 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f907c1026c0 con 0x7f907c107bf0 2024-07-26T06:55:54.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.750+0000 7f90737fe640 1 -- 172.21.15.42:0/2667929691 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11979 (secure 0 0 0) 0x7f906c0aadb0 con 0x7f907c107bf0 2024-07-26T06:55:54.751 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-07-26T06:55:54.751 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":39,"btime":"2024-07-26T06:55:50:917631+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14658,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6835/3548166563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6834","nonce":3548166563},{"type":"v1","addr":"172.21.15.42:6835","nonce":3548166563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14664,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.42:6837/1420377891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6836","nonce":1420377891},{"type":"v1","addr":"172.21.15.42:6837","nonce":1420377891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T06:47:06.780568+0000","modified":"2024-07-26T06:55:50.473392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24620,"mds_1":24601,"mds_2":24593,"mds_3":24583,"mds_4":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14703":{"gid":14703,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6839/4044632348","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6838","nonce":4044632348},{"type":"v1","addr":"172.21.15.42:6839","nonce":4044632348}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14709":{"gid":14709,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.42:6841/1430155791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6840","nonce":1430155791},{"type":"v1","addr":"172.21.15.42:6841","nonce":1430155791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6833/4144543280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":4144543280},{"type":"v1","addr":"172.21.15.164:6833","nonce":4144543280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24557":{"gid":24557,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6833/845444723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":845444723},{"type":"v1","addr":"172.21.15.163:6833","nonce":845444723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24578":{"gid":24578,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":15,"addr":"172.21.15.163:6835/375978170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":375978170},{"type":"v1","addr":"172.21.15.163:6835","nonce":375978170}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24583":{"gid":24583,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":13,"addr":"172.21.15.164:6837/3642457979","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6836","nonce":3642457979},{"type":"v1","addr":"172.21.15.164:6837","nonce":3642457979}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.163:6837/154041066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":154041066},{"type":"v1","addr":"172.21.15.163:6837","nonce":154041066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24601":{"gid":24601,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.164:6839/3593178314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6838","nonce":3593178314},{"type":"v1","addr":"172.21.15.164:6839","nonce":3593178314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24620":{"gid":24620,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.163:6839/563658453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":563658453},{"type":"v1","addr":"172.21.15.163:6839","nonce":563658453}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25369":{"gid":25369,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.164:6835/1072602530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1072602530},{"type":"v1","addr":"172.21.15.164:6835","nonce":1072602530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24620,"qdb_cluster":[24620,24601,24593,24583,24578]},"id":1}]} 2024-07-26T06:55:54.751 INFO:teuthology.orchestra.run.smithi042.stderr:dumped fsmap epoch 39 2024-07-26T06:55:54.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.752+0000 7f90831a1640 1 -- 172.21.15.42:0/2667929691 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f90680b3c80 msgr2=0x7f90680b6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:54.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.752+0000 7f90831a1640 1 --2- 172.21.15.42:0/2667929691 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f90680b3c80 0x7f90680b6190 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9078000c00 tx=0x7f9078031400 comp rx=0 tx=0).stop 2024-07-26T06:55:54.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.752+0000 7f90831a1640 1 -- 172.21.15.42:0/2667929691 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f907c107bf0 msgr2=0x7f907c19b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:54.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.752+0000 7f90831a1640 1 --2- 172.21.15.42:0/2667929691 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f907c107bf0 0x7f907c19b060 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f906c00e9f0 tx=0x7f906c00eeb0 comp rx=0 tx=0).stop 2024-07-26T06:55:54.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.752+0000 7f90831a1640 1 -- 172.21.15.42:0/2667929691 shutdown_connections 2024-07-26T06:55:54.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.752+0000 7f90831a1640 1 --2- 172.21.15.42:0/2667929691 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7f90680b3c80 0x7f90680b6190 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:54.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.752+0000 7f90831a1640 1 --2- 172.21.15.42:0/2667929691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f907c108540 0x7f907c19b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:54.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.752+0000 7f90831a1640 1 --2- 172.21.15.42:0/2667929691 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f907c107bf0 0x7f907c19b060 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:54.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.752+0000 7f90831a1640 1 --2- 172.21.15.42:0/2667929691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f907c1022b0 0x7f907c19ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:54.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.752+0000 7f90831a1640 1 -- 172.21.15.42:0/2667929691 >> 172.21.15.42:0/2667929691 conn(0x7f907c0fdd00 msgr2=0x7f907c100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:54.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.752+0000 7f90831a1640 1 -- 172.21.15.42:0/2667929691 shutdown_connections 2024-07-26T06:55:54.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:54.752+0000 7f90831a1640 1 -- 172.21.15.42:0/2667929691 wait complete. 2024-07-26T06:55:54.932 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:54 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T06:55:54.932 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:55:54 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Consumed 1min 16.802s CPU time. 2024-07-26T06:55:54.932 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:54 smithi163 podman[69846]: 2024-07-26 06:55:54.66336687 +0000 UTC m=+2.401284661 container remove eff753a144e58fd48fb44fbae126322c40d5550dddd5afda0cef03fff94cb86e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-07-26T06:55:54.932 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:54 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:55:55.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:54 smithi042 ceph-mon[32775]: pgmap v567: 97 pgs: 97 active+clean; 60 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 920 op/s 2024-07-26T06:55:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:54 smithi042 ceph-mon[32775]: from='client.? 172.21.15.42:0/2667929691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:55:55.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:54 smithi164 ceph-mon[36970]: pgmap v567: 97 pgs: 97 active+clean; 60 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 920 op/s 2024-07-26T06:55:55.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:54 smithi164 ceph-mon[36970]: from='client.? 172.21.15.42:0/2667929691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:55:55.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:54 smithi163 ceph-mon[36873]: pgmap v567: 97 pgs: 97 active+clean; 60 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 920 op/s 2024-07-26T06:55:55.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:54 smithi163 ceph-mon[36873]: from='client.? 172.21.15.42:0/2667929691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T06:55:55.257 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:54 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-26T06:55:55.257 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:55:54 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Consumed 4.976s CPU time. 2024-07-26T06:55:55.257 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:54 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T06:55:55.257 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:55:54 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Consumed 5.273s CPU time. 2024-07-26T06:55:55.789 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24620 quiesce db / --timeout 90 --expiration 141.95 --await-for 45.0' 2024-07-26T06:55:55.790 DEBUG:teuthology.orchestra.run.smithi042:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 8b3e2750-4b19-11ef-bcb6-c7b262605968 -- timeout 105 ceph tell mds.24620 quiesce db / --timeout 90 --expiration 141.95 --await-for 45.0 2024-07-26T06:55:56.065 INFO:teuthology.orchestra.run.smithi042.stderr:Inferring config /var/lib/ceph/8b3e2750-4b19-11ef-bcb6-c7b262605968/mon.a/config 2024-07-26T06:55:57.167 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:56 smithi042 ceph-mon[32775]: pgmap v568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.3 MiB/s wr, 852 op/s 2024-07-26T06:55:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:56 smithi164 ceph-mon[36970]: pgmap v568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.3 MiB/s wr, 852 op/s 2024-07-26T06:55:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:56 smithi163 ceph-mon[36873]: pgmap v568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.3 MiB/s wr, 852 op/s 2024-07-26T06:55:57.854 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.853+0000 7ff686072640 1 -- 172.21.15.42:0/3400899982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff680103a20 msgr2=0x7ff68010a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:57.855 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.853+0000 7ff686072640 1 --2- 172.21.15.42:0/3400899982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff680103a20 0x7ff68010a3c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff678009ca0 tx=0x7ff67802a8c0 comp rx=0 tx=0).stop 2024-07-26T06:55:57.855 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.853+0000 7ff686072640 1 -- 172.21.15.42:0/3400899982 shutdown_connections 2024-07-26T06:55:57.855 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.853+0000 7ff686072640 1 --2- 172.21.15.42:0/3400899982 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff680103a20 0x7ff68010a3c0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:57.855 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.853+0000 7ff686072640 1 --2- 172.21.15.42:0/3400899982 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff680103080 0x7ff680103490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:57.855 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.853+0000 7ff686072640 1 --2- 172.21.15.42:0/3400899982 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff680102000 0x7ff680102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:57.855 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.853+0000 7ff686072640 1 -- 172.21.15.42:0/3400899982 >> 172.21.15.42:0/3400899982 conn(0x7ff6800fd960 msgr2=0x7ff6800ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:57.855 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.853+0000 7ff686072640 1 -- 172.21.15.42:0/3400899982 shutdown_connections 2024-07-26T06:55:57.855 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.853+0000 7ff686072640 1 -- 172.21.15.42:0/3400899982 wait complete. 2024-07-26T06:55:57.855 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.854+0000 7ff686072640 1 Processor -- start 2024-07-26T06:55:57.857 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.854+0000 7ff686072640 1 -- start start 2024-07-26T06:55:57.858 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff686072640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff680102000 0x7ff68006b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:57.858 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff686072640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff680103080 0x7ff68019e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:57.858 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff686072640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff680103a20 0x7ff6801a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:57.858 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff686072640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff68010dc80 con 0x7ff680102000 2024-07-26T06:55:57.858 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff686072640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff68010db00 con 0x7ff680103080 2024-07-26T06:55:57.858 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff686072640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff68010de00 con 0x7ff680103a20 2024-07-26T06:55:57.858 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff67f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff680102000 0x7ff68006b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:57.858 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff67f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff680102000 0x7ff68006b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32798/0 (socket says 172.21.15.42:32798) 2024-07-26T06:55:57.858 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff67f7fe640 1 -- 172.21.15.42:0/2402090969 learned_addr learned my addr 172.21.15.42:0/2402090969 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:57.858 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff680103a20 0x7ff6801a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:57.858 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff67effd640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff680103080 0x7ff68019e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:57.859 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff67f7fe640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff680103a20 msgr2=0x7ff6801a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:57.859 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff680103a20 0x7ff6801a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:57.859 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff67f7fe640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff680103080 msgr2=0x7ff68019e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:57.859 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff680103080 0x7ff68019e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:57.859 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff67f7fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6801a1420 con 0x7ff680102000 2024-07-26T06:55:57.859 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff67effd640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff680103080 0x7ff68019e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:55:57.859 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff680102000 0x7ff68006b820 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ff67000d770 tx=0x7ff67000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:57.859 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.855+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff670017070 con 0x7ff680102000 2024-07-26T06:55:57.859 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.856+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6700054d0 con 0x7ff680102000 2024-07-26T06:55:57.859 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.856+0000 7ff686072640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6801a1710 con 0x7ff680102000 2024-07-26T06:55:57.859 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.856+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff670004c30 con 0x7ff680102000 2024-07-26T06:55:57.860 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.856+0000 7ff686072640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6801a1c10 con 0x7ff680102000 2024-07-26T06:55:57.865 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.858+0000 7ff68cf83740 1 Processor -- start 2024-07-26T06:55:57.865 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.858+0000 7ff68cf83740 1 -- start start 2024-07-26T06:55:57.865 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.858+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7ff670004e10 con 0x7ff680102000 2024-07-26T06:55:57.865 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.858+0000 7ff68cf83740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db6e0771b0 0x55db6e41bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:57.865 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.858+0000 7ff68cf83740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55db6e04e2f0 0x55db6e41c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:57.865 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.858+0000 7ff68cf83740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db6e0564d0 0x55db6e41cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:57.865 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.858+0000 7ff68cf83740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55db6e38f3b0 con 0x55db6e04e2f0 2024-07-26T06:55:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.858+0000 7ff68cf83740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55db6e37e8b0 con 0x55db6e0771b0 2024-07-26T06:55:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.858+0000 7ff68cf83740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55db6e3828f0 con 0x55db6e0564d0 2024-07-26T06:55:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.859+0000 7ff67f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55db6e04e2f0 0x55db6e41c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.859+0000 7ff67f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55db6e04e2f0 0x55db6e41c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32806/0 (socket says 172.21.15.42:32806) 2024-07-26T06:55:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.859+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1046411109 learned_addr learned my addr 172.21.15.42:0/1046411109 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.859+0000 7ff67ffff640 1 --2- 172.21.15.42:0/1046411109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db6e0771b0 0x55db6e41bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.859+0000 7ff67effd640 1 --2- 172.21.15.42:0/1046411109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db6e0564d0 0x55db6e41cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.859+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1046411109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db6e0564d0 msgr2=0x55db6e41cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.859+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1046411109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db6e0564d0 0x55db6e41cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.859+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1046411109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db6e0771b0 msgr2=0x55db6e41bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.859+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1046411109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db6e0771b0 0x55db6e41bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.859+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1046411109 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db6e3259a0 con 0x55db6e04e2f0 2024-07-26T06:55:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.859+0000 7ff67effd640 1 --2- 172.21.15.42:0/1046411109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db6e0564d0 0x55db6e41cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:55:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.859+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1046411109 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55db6e04e2f0 0x55db6e41c1c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff6700e9c10 tx=0x7ff6700eb690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe2e8ec34c970c2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.859+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.859+0000 7ff675ffb640 1 -- 172.21.15.42:0/1046411109 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6700f2070 con 0x55db6e04e2f0 2024-07-26T06:55:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.859+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.859+0000 7ff675ffb640 1 -- 172.21.15.42:0/1046411109 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff67006eea0 con 0x55db6e04e2f0 2024-07-26T06:55:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.859+0000 7ff675ffb640 1 -- 172.21.15.42:0/1046411109 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6700a9fb0 con 0x55db6e04e2f0 2024-07-26T06:55:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.860+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7ff6700e3730 con 0x7ff680102000 2024-07-26T06:55:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.860+0000 7ff68cf83740 1 -- 172.21.15.42:0/1046411109 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55db6e04e2f0 msgr2=0x55db6e41c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.860+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b60e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff6780094e0 tx=0x7ff678009450 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.860+0000 7ff68cf83740 1 --2- 172.21.15.42:0/1046411109 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55db6e04e2f0 0x55db6e41c1c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff6700e9c10 tx=0x7ff6700eb690 comp rx=0 tx=0).stop 2024-07-26T06:55:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.860+0000 7ff68cf83740 1 -- 172.21.15.42:0/1046411109 shutdown_connections 2024-07-26T06:55:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.860+0000 7ff68cf83740 1 --2- 172.21.15.42:0/1046411109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db6e0564d0 0x55db6e41cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:57.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.860+0000 7ff68cf83740 1 --2- 172.21.15.42:0/1046411109 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55db6e04e2f0 0x55db6e41c1c0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:57.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.860+0000 7ff68cf83740 1 --2- 172.21.15.42:0/1046411109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db6e0771b0 0x55db6e41bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:57.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.860+0000 7ff68cf83740 1 -- 172.21.15.42:0/1046411109 >> 172.21.15.42:0/1046411109 conn(0x55db6e06b300 msgr2=0x55db6e4175e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:57.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.860+0000 7ff68cf83740 1 -- 172.21.15.42:0/1046411109 shutdown_connections 2024-07-26T06:55:57.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.860+0000 7ff68cf83740 1 -- 172.21.15.42:0/1046411109 wait complete. 2024-07-26T06:55:57.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.861+0000 7ff68cf83740 1 -- start start 2024-07-26T06:55:57.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.861+0000 7ff68cf83740 1 Processor -- start 2024-07-26T06:55:57.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.861+0000 7ff68cf83740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db6e04e2f0 0x55db6e425480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:57.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.861+0000 7ff68cf83740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55db6e0564d0 0x55db6e425b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:57.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.861+0000 7ff68cf83740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db6e0771b0 0x55db6e4266b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:57.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.861+0000 7ff68cf83740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x55db6e2b8ab0 con 0x55db6e0564d0 2024-07-26T06:55:57.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.861+0000 7ff68cf83740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55db6e306cf0 con 0x55db6e04e2f0 2024-07-26T06:55:57.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.861+0000 7ff68cf83740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55db6e41d830 con 0x55db6e0771b0 2024-07-26T06:55:57.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.861+0000 7ff68cf83740 10 client.0 _ll_register_callbacks cb 0x55db6e3783e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:55:57.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.861+0000 7ff67effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55db6e0564d0 0x55db6e425b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.861+0000 7ff67effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55db6e0564d0 0x55db6e425b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32820/0 (socket says 172.21.15.42:32820) 2024-07-26T06:55:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.862+0000 7ff67effd640 1 -- 172.21.15.42:0/1831263443 learned_addr learned my addr 172.21.15.42:0/1831263443 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.862+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db6e04e2f0 0x55db6e425480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.862+0000 7ff67ffff640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db6e0771b0 0x55db6e4266b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.862+0000 7ff67effd640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db6e0771b0 msgr2=0x55db6e4266b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.862+0000 7ff67effd640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55db6e0771b0 0x55db6e4266b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.862+0000 7ff67effd640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db6e04e2f0 msgr2=0x55db6e425480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.862+0000 7ff67effd640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db6e04e2f0 0x55db6e425480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.862+0000 7ff67effd640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db6e0e2790 con 0x55db6e0564d0 2024-07-26T06:55:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.862+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db6e04e2f0 0x55db6e425480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T06:55:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.862+0000 7ff67effd640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x55db6e0564d0 0x55db6e425b40 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff66800bfc0 tx=0x7ff668011870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=389e48ffdd1cd760 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.862+0000 7ff65effd640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:55:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.862+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff668012710 con 0x55db6e0564d0 2024-07-26T06:55:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.862+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7ff6680128b0 con 0x55db6e0564d0 2024-07-26T06:55:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.862+0000 7ff68cf83740 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff67006eea0 con 0x55db6e0564d0 2024-07-26T06:55:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.862+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7ff668012ab0 con 0x55db6e0564d0 2024-07-26T06:55:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.862+0000 7ff68cf83740 10 client.16083 fetch_fsmap learned FSMap version 39 2024-07-26T06:55:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.862+0000 7ff68cf83740 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55db6e15b620 con 0x55db6e0564d0 2024-07-26T06:55:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.863+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff67006eea0 con 0x55db6e0564d0 2024-07-26T06:55:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.863+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 39) ==== 8366+0+0 (secure 0 0 0) 0x7ff668025420 con 0x55db6e0564d0 2024-07-26T06:55:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.863+0000 7ff68cf83740 10 client.16083 fetch_fsmap finished waiting for FSMap version 39 2024-07-26T06:55:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.863+0000 7ff68cf83740 10 client.16083 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T06:55:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.863+0000 7ff68cf83740 20 client.16083 populate_metadata read hostname 'smithi042' 2024-07-26T06:55:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.863+0000 7ff68cf83740 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.863+0000 7ff68cf83740 4 client.16083 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:55:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.863+0000 7ff68cf83740 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db6e391300 con 0x55db6e035d80 2024-07-26T06:55:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.863+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:55:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.863+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:57.863+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T06:55:58.065 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:58.065+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:55:58.066 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:58.065+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T06:55:58.066 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:58.065+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:58.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.229+0000 7faa2ff1d640 1 -- 172.21.15.42:0/3086815017 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa281024b0 msgr2=0x7faa281028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:58.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.229+0000 7faa2ff1d640 1 --2- 172.21.15.42:0/3086815017 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa281024b0 0x7faa281028c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7faa24009900 tx=0x7faa2402a410 comp rx=0 tx=0).stop 2024-07-26T06:55:58.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.229+0000 7faa2ff1d640 1 -- 172.21.15.42:0/3086815017 shutdown_connections 2024-07-26T06:55:58.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.229+0000 7faa2ff1d640 1 --2- 172.21.15.42:0/3086815017 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa28108740 0x7faa2810ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:58.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.229+0000 7faa2ff1d640 1 --2- 172.21.15.42:0/3086815017 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa28107df0 0x7faa28108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:58.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.229+0000 7faa2ff1d640 1 --2- 172.21.15.42:0/3086815017 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa281024b0 0x7faa281028c0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:58.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.229+0000 7faa2ff1d640 1 -- 172.21.15.42:0/3086815017 >> 172.21.15.42:0/3086815017 conn(0x7faa28079190 msgr2=0x7faa280795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:58.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.229+0000 7faa2ff1d640 1 -- 172.21.15.42:0/3086815017 shutdown_connections 2024-07-26T06:55:58.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.229+0000 7faa2ff1d640 1 -- 172.21.15.42:0/3086815017 wait complete. 2024-07-26T06:55:58.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.230+0000 7faa2ff1d640 1 Processor -- start 2024-07-26T06:55:58.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.230+0000 7faa2ff1d640 1 -- start start 2024-07-26T06:55:58.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.230+0000 7faa2ff1d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa281024b0 0x7faa2819ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:58.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.230+0000 7faa2ff1d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa28107df0 0x7faa2819b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:58.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.230+0000 7faa2ff1d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa28108740 0x7faa2819b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:58.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.230+0000 7faa2ff1d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faa281123c0 con 0x7faa28108740 2024-07-26T06:55:58.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.230+0000 7faa2ff1d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7faa28112240 con 0x7faa281024b0 2024-07-26T06:55:58.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.230+0000 7faa2ff1d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faa28112540 con 0x7faa28107df0 2024-07-26T06:55:58.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.230+0000 7faa2e4ba640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa28108740 0x7faa2819b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.231+0000 7faa2e4ba640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa28108740 0x7faa2819b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32838/0 (socket says 172.21.15.42:32838) 2024-07-26T06:55:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.231+0000 7faa2e4ba640 1 -- 172.21.15.42:0/320079342 learned_addr learned my addr 172.21.15.42:0/320079342 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.231+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/320079342 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa281024b0 0x7faa2819ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.231+0000 7faa2d4b8640 1 --2- 172.21.15.42:0/320079342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa28107df0 0x7faa2819b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.231+0000 7faa2e4ba640 1 -- 172.21.15.42:0/320079342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa28107df0 msgr2=0x7faa2819b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.231+0000 7faa2e4ba640 1 --2- 172.21.15.42:0/320079342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa28107df0 0x7faa2819b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.231+0000 7faa2e4ba640 1 -- 172.21.15.42:0/320079342 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa281024b0 msgr2=0x7faa2819ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.231+0000 7faa2e4ba640 1 --2- 172.21.15.42:0/320079342 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa281024b0 0x7faa2819ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.231+0000 7faa2e4ba640 1 -- 172.21.15.42:0/320079342 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa281a5f60 con 0x7faa28108740 2024-07-26T06:55:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.231+0000 7faa2e4ba640 1 --2- 172.21.15.42:0/320079342 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa28108740 0x7faa2819b6d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7faa2000eaa0 tx=0x7faa2000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.231+0000 7faa1effd640 1 -- 172.21.15.42:0/320079342 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2000cdb0 con 0x7faa28108740 2024-07-26T06:55:58.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.231+0000 7faa1effd640 1 -- 172.21.15.42:0/320079342 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa20005020 con 0x7faa28108740 2024-07-26T06:55:58.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.231+0000 7faa1effd640 1 -- 172.21.15.42:0/320079342 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa200116d0 con 0x7faa28108740 2024-07-26T06:55:58.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.231+0000 7faa2ff1d640 1 -- 172.21.15.42:0/320079342 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa281a6250 con 0x7faa28108740 2024-07-26T06:55:58.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.231+0000 7faa2ff1d640 1 -- 172.21.15.42:0/320079342 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa281a6740 con 0x7faa28108740 2024-07-26T06:55:58.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.234+0000 7faa364a1740 1 Processor -- start 2024-07-26T06:55:58.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.235+0000 7faa1effd640 1 -- 172.21.15.42:0/320079342 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 18) ==== 151059+0+0 (secure 0 0 0) 0x7faa200051d0 con 0x7faa28108740 2024-07-26T06:55:58.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.235+0000 7faa364a1740 1 -- start start 2024-07-26T06:55:58.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa364a1740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561ca1d0d850 0x561ca20b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:58.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa364a1740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561ca1ce4990 0x561ca20b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa364a1740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561ca1cecb70 0x561ca20b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa364a1740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x561ca2028690 con 0x561ca1ce4990 2024-07-26T06:55:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa364a1740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561ca2017b90 con 0x561ca1d0d850 2024-07-26T06:55:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa364a1740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x561ca201bbd0 con 0x561ca1cecb70 2024-07-26T06:55:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa2dcb9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561ca1ce4990 0x561ca20b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa2dcb9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561ca1ce4990 0x561ca20b3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32844/0 (socket says 172.21.15.42:32844) 2024-07-26T06:55:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa2d4b8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561ca1cecb70 0x561ca20b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa2dcb9640 1 -- 172.21.15.42:0/1230423487 learned_addr learned my addr 172.21.15.42:0/1230423487 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa2d4b8640 1 --2- 172.21.15.42:0/1230423487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561ca1d0d850 0x561ca20b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa1effd640 1 --2- 172.21.15.42:0/320079342 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faa140b3bd0 0x7faa140b60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa2dcb9640 1 -- 172.21.15.42:0/1230423487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561ca1cecb70 msgr2=0x561ca20b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/1230423487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561ca1cecb70 0x561ca20b3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa2dcb9640 1 -- 172.21.15.42:0/1230423487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561ca1d0d850 msgr2=0x561ca20b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/1230423487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561ca1d0d850 0x561ca20b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa2dcb9640 1 -- 172.21.15.42:0/1230423487 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ca1fa3a10 con 0x561ca1ce4990 2024-07-26T06:55:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa1effd640 1 -- 172.21.15.42:0/320079342 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7faa200e3d20 con 0x7faa28108740 2024-07-26T06:55:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.236+0000 7faa2d4b8640 1 --2- 172.21.15.42:0/1230423487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561ca1d0d850 0x561ca20b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T06:55:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.237+0000 7faa2e4ba640 1 --2- 172.21.15.42:0/320079342 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faa140b3bd0 0x7faa140b60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.237+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/1230423487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561ca1ce4990 0x561ca20b3a50 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7faa24033000 tx=0x7faa24032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b998cf4ad848c32 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.237+0000 7faa13fff640 1 -- 172.21.15.42:0/1230423487 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa24035040 con 0x561ca1ce4990 2024-07-26T06:55:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.237+0000 7faa13fff640 1 -- 172.21.15.42:0/1230423487 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa240316a0 con 0x561ca1ce4990 2024-07-26T06:55:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.237+0000 7faa13fff640 1 -- 172.21.15.42:0/1230423487 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa24005530 con 0x561ca1ce4990 2024-07-26T06:55:58.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.238+0000 7faa364a1740 1 -- 172.21.15.42:0/1230423487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561ca1ce4990 msgr2=0x561ca20b3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T06:55:58.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.238+0000 7faa2e4ba640 1 --2- 172.21.15.42:0/320079342 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7faa140b3bd0 0x7faa140b60e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7faa200aaa00 tx=0x7faa2000cf50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:58.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.238+0000 7faa364a1740 1 --2- 172.21.15.42:0/1230423487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561ca1ce4990 0x561ca20b3a50 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7faa24033000 tx=0x7faa24032c60 comp rx=0 tx=0).stop 2024-07-26T06:55:58.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.238+0000 7faa364a1740 1 -- 172.21.15.42:0/1230423487 shutdown_connections 2024-07-26T06:55:58.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.238+0000 7faa364a1740 1 --2- 172.21.15.42:0/1230423487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561ca1cecb70 0x561ca20b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:58.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.238+0000 7faa364a1740 1 --2- 172.21.15.42:0/1230423487 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561ca1ce4990 0x561ca20b3a50 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:58.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.238+0000 7faa364a1740 1 --2- 172.21.15.42:0/1230423487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561ca1d0d850 0x561ca20b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:58.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.238+0000 7faa364a1740 1 -- 172.21.15.42:0/1230423487 >> 172.21.15.42:0/1230423487 conn(0x561ca1d01c40 msgr2=0x561ca20b1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T06:55:58.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.238+0000 7faa364a1740 1 -- 172.21.15.42:0/1230423487 shutdown_connections 2024-07-26T06:55:58.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.238+0000 7faa364a1740 1 -- 172.21.15.42:0/1230423487 wait complete. 2024-07-26T06:55:58.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.239+0000 7faa364a1740 1 -- start start 2024-07-26T06:55:58.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.239+0000 7faa364a1740 1 Processor -- start 2024-07-26T06:55:58.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.239+0000 7faa364a1740 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561ca1ce4990 0x561ca20bebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:58.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.239+0000 7faa364a1740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561ca1cecb70 0x561ca20bf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:58.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.239+0000 7faa364a1740 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561ca1d0d850 0x561ca20caef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:58.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.239+0000 7faa364a1740 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x561ca1e7b7b0 con 0x561ca1ce4990 2024-07-26T06:55:58.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.239+0000 7faa364a1740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561ca1fba210 con 0x561ca1cecb70 2024-07-26T06:55:58.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.239+0000 7faa364a1740 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x561ca20c1c00 con 0x561ca1d0d850 2024-07-26T06:55:58.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.239+0000 7faa2d4b8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561ca1ce4990 0x561ca20bebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:58.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.240+0000 7faa2d4b8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561ca1ce4990 0x561ca20bebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32856/0 (socket says 172.21.15.42:32856) 2024-07-26T06:55:58.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.240+0000 7faa2d4b8640 1 -- 172.21.15.42:0/2285256131 learned_addr learned my addr 172.21.15.42:0/2285256131 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-07-26T06:55:58.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.240+0000 7faa364a1740 10 client.0 _ll_register_callbacks cb 0x561ca1f9b2c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T06:55:58.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.240+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561ca1cecb70 0x561ca20bf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:58.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.240+0000 7faa2d4b8640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561ca1d0d850 0x561ca20caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:55:58.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.240+0000 7faa2d4b8640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561ca1d0d850 msgr2=0x561ca20caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:58.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.240+0000 7faa2d4b8640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x561ca1d0d850 0x561ca20caef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:58.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.240+0000 7faa2d4b8640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561ca1cecb70 msgr2=0x561ca20bf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T06:55:58.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.240+0000 7faa2d4b8640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561ca1cecb70 0x561ca20bf120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T06:55:58.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.240+0000 7faa2d4b8640 1 -- 172.21.15.42:0/2285256131 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ca1fcb710 con 0x561ca1ce4990 2024-07-26T06:55:58.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.240+0000 7faa2d4b8640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x561ca1ce4990 0x561ca20bebe0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7faa18012d40 tx=0x7faa1800bb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d544cdd15b391e4f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T06:55:58.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.240+0000 7faa117fa640 10 client.0 ms_handle_connect on v2:172.21.15.42:3300/0 2024-07-26T06:55:58.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.240+0000 7faa117fa640 1 -- 172.21.15.42:0/2285256131 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa18005220 con 0x561ca1ce4990 2024-07-26T06:55:58.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.240+0000 7faa117fa640 1 -- 172.21.15.42:0/2285256131 <== mon.0 v2:172.21.15.42:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8269+0+0 (secure 0 0 0) 0x7faa180053c0 con 0x561ca1ce4990 2024-07-26T06:55:58.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.240+0000 7faa364a1740 1 -- 172.21.15.42:0/2285256131 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561ca202a5e0 con 0x561ca1ce4990 2024-07-26T06:55:58.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.241+0000 7faa117fa640 1 -- 172.21.15.42:0/2285256131 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7faa1800fe60 con 0x561ca1ce4990 2024-07-26T06:55:58.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.241+0000 7faa364a1740 10 client.16095 fetch_fsmap learned FSMap version 39 2024-07-26T06:55:58.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.241+0000 7faa364a1740 1 -- 172.21.15.42:0/2285256131 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x561ca1d87ba0 con 0x561ca1ce4990 2024-07-26T06:55:58.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.241+0000 7faa117fa640 1 -- 172.21.15.42:0/2285256131 <== mon.0 v2:172.21.15.42:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x561ca202a5e0 con 0x561ca1ce4990 2024-07-26T06:55:58.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.241+0000 7faa117fa640 1 -- 172.21.15.42:0/2285256131 <== mon.0 v2:172.21.15.42:3300/0 5 ==== fsmap(e 39) ==== 8366+0+0 (secure 0 0 0) 0x7faa18025430 con 0x561ca1ce4990 2024-07-26T06:55:58.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.241+0000 7faa364a1740 10 client.16095 fetch_fsmap finished waiting for FSMap version 39 2024-07-26T06:55:58.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.241+0000 7faa364a1740 10 client.16095 resolve_mds: validated gid 24620 aka daemon mds.b 2024-07-26T06:55:58.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.241+0000 7faa364a1740 20 client.16095 populate_metadata read hostname 'smithi042' 2024-07-26T06:55:58.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.241+0000 7faa364a1740 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T06:55:58.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.241+0000 7faa364a1740 4 client.16095 mds_command: new command op to 24620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T06:55:58.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.241+0000 7faa364a1740 1 -- 172.21.15.42:0/2285256131 --> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ca2026080 con 0x561ca1ccc1d0 2024-07-26T06:55:58.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.241+0000 7faa2dcb9640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 msgr2=0x561ca20d0cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:55:58.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.241+0000 7faa117fa640 1 client.16095 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:58.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.241+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T06:55:58.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.442+0000 7faa2dcb9640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 msgr2=0x561ca20d0cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:55:58.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.442+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T06:55:58.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.442+0000 7faa117fa640 1 client.16095 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:58.467+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:55:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:58.467+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T06:55:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:58.467+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:58.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.844+0000 7faa2dcb9640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 msgr2=0x561ca20d0cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:55:58.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.844+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T06:55:58.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:58.844+0000 7faa117fa640 1 client.16095 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:59.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:55:58 smithi042 ceph-mon[32775]: pgmap v569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 675 op/s 2024-07-26T06:55:59.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:55:58 smithi164 ceph-mon[36970]: pgmap v569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 675 op/s 2024-07-26T06:55:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:55:58 smithi163 ceph-mon[36873]: pgmap v569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.9 MiB/s wr, 675 op/s 2024-07-26T06:55:59.269 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:59.269+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:55:59.269 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:59.269+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T06:55:59.269 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:55:59.269+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:55:59.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:59.645+0000 7faa2dcb9640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 msgr2=0x561ca20d0cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:55:59.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:59.645+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T06:55:59.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:55:59.645+0000 7faa117fa640 1 client.16095 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:56:00.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:00.871+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:56:00.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:00.871+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:56:00.872 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:00.871+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-07-26T06:56:01.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:00 smithi042 ceph-mon[32775]: pgmap v570: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 1.4 MiB/s wr, 493 op/s 2024-07-26T06:56:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:00 smithi164 ceph-mon[36970]: pgmap v570: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 1.4 MiB/s wr, 493 op/s 2024-07-26T06:56:01.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:01.246+0000 7faa2dcb9640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 msgr2=0x561ca20d0cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:56:01.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:01.246+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-07-26T06:56:01.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:01.246+0000 7faa117fa640 1 client.16095 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:56:01.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:00 smithi163 ceph-mon[36873]: pgmap v570: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 1.4 MiB/s wr, 493 op/s 2024-07-26T06:56:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:02 smithi164 ceph-mon[36970]: pgmap v571: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 848 KiB/s wr, 284 op/s 2024-07-26T06:56:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:02 smithi163 ceph-mon[36873]: pgmap v571: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 848 KiB/s wr, 284 op/s 2024-07-26T06:56:03.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:02 smithi042 ceph-mon[32775]: pgmap v571: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 848 KiB/s wr, 284 op/s 2024-07-26T06:56:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:04.072+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:56:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:04.073+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:56:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:04.073+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-07-26T06:56:04.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:04.449+0000 7faa2dcb9640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 msgr2=0x561ca20d0cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:56:04.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:04.449+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-07-26T06:56:04.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:04.449+0000 7faa117fa640 1 client.16095 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:56:04.483 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:56:04 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-07-26T06:56:04.483 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:56:04 smithi164 systemd[1]: Stopped Ceph mds.c for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:04.483 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:56:04 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.c.service: Consumed 5.107s CPU time. 2024-07-26T06:56:04.483 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:56:04 smithi164 systemd[1]: Starting Ceph mds.c for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:56:04.483 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:56:04 smithi164 podman[70222]: 2024-07-26 06:56:04.361550902 +0000 UTC m=+0.017594127 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:56:04.736 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:56:04 smithi164 podman[70222]: 2024-07-26 06:56:04.48338522 +0000 UTC m=+0.139428486 container create 018308366e3f2d07d1c8bf9503f2c723e276513ebb51eeecf9757bc8964cc8fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-07-26T06:56:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:04 smithi164 ceph-mon[36970]: pgmap v572: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 658 KiB/s wr, 213 op/s 2024-07-26T06:56:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:04 smithi164 ceph-mon[36970]: Replacing daemon mds.c as rank 3 with standby daemon mds.a 2024-07-26T06:56:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:04 smithi164 ceph-mon[36970]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-07-26T06:56:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:04 smithi164 ceph-mon[36970]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:56:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:04 smithi164 ceph-mon[36970]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T06:56:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:04 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T06:56:05.223 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:56:04 smithi164 podman[70222]: 2024-07-26 06:56:04.736209225 +0000 UTC m=+0.392252433 container init 018308366e3f2d07d1c8bf9503f2c723e276513ebb51eeecf9757bc8964cc8fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-07-26T06:56:05.223 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:56:04 smithi164 podman[70222]: 2024-07-26 06:56:04.738937502 +0000 UTC m=+0.394980718 container start 018308366e3f2d07d1c8bf9503f2c723e276513ebb51eeecf9757bc8964cc8fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T06:56:05.223 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:56:04 smithi164 ceph-mds[70241]: -- 172.21.15.164:0/3306865929 <== mon.1 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560b364c7860 con 0x560b3a669a80 2024-07-26T06:56:05.223 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:56:04 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-c[70237]: starting mds.c at 2024-07-26T06:56:05.223 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:56:04 smithi164 bash[70222]: 018308366e3f2d07d1c8bf9503f2c723e276513ebb51eeecf9757bc8964cc8fe 2024-07-26T06:56:05.223 INFO:journalctl@ceph.mds.c.smithi164.stdout:Jul 26 06:56:04 smithi164 systemd[1]: Started Ceph mds.c for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:05.256 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 1. 2024-07-26T06:56:05.256 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:05 smithi163 systemd[1]: Stopped Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:05.256 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Consumed 5.273s CPU time. 2024-07-26T06:56:05.256 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:05 smithi163 systemd[1]: Starting Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:56:05.256 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 1. 2024-07-26T06:56:05.256 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:05 smithi163 systemd[1]: Stopped Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:05.257 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Consumed 4.976s CPU time. 2024-07-26T06:56:05.257 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:05 smithi163 systemd[1]: Starting Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:56:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:04 smithi163 ceph-mon[36873]: pgmap v572: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 658 KiB/s wr, 213 op/s 2024-07-26T06:56:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:04 smithi163 ceph-mon[36873]: Replacing daemon mds.c as rank 3 with standby daemon mds.a 2024-07-26T06:56:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:04 smithi163 ceph-mon[36873]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-07-26T06:56:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:04 smithi163 ceph-mon[36873]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:56:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:04 smithi163 ceph-mon[36873]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T06:56:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:04 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T06:56:05.258 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 1. 2024-07-26T06:56:05.258 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:05 smithi163 systemd[1]: Stopped Ceph mds.b for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:05.258 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Consumed 1min 16.802s CPU time. 2024-07-26T06:56:05.258 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:05 smithi163 systemd[1]: Starting Ceph mds.b for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:56:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:04 smithi042 ceph-mon[32775]: pgmap v572: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 658 KiB/s wr, 213 op/s 2024-07-26T06:56:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:04 smithi042 ceph-mon[32775]: Replacing daemon mds.c as rank 3 with standby daemon mds.a 2024-07-26T06:56:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:04 smithi042 ceph-mon[32775]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-07-26T06:56:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:04 smithi042 ceph-mon[32775]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:56:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:04 smithi042 ceph-mon[32775]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T06:56:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:04 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T06:56:05.507 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:05 smithi163 podman[70117]: 2024-07-26 06:56:05.164664347 +0000 UTC m=+0.021934483 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:56:05.507 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:05 smithi163 podman[70118]: 2024-07-26 06:56:05.167076368 +0000 UTC m=+0.022510223 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:56:05.507 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:05 smithi163 podman[70107]: 2024-07-26 06:56:05.159034925 +0000 UTC m=+0.021623175 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:56:05.507 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:05 smithi163 podman[70107]: 2024-07-26 06:56:05.308157403 +0000 UTC m=+0.170745641 container create 85bdfb5f68dd6d00023df89da5d1a8290ae3a708bfa5e599060fcda834992c9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.license=GPLv2) 2024-07-26T06:56:05.781 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:05 smithi163 podman[70117]: 2024-07-26 06:56:05.544190431 +0000 UTC m=+0.401460552 container create 407f493356c218e61d85a124f2469e8567115ab3f538adbe23ea5e60c9801379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-07-26T06:56:06.051 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:06 smithi163 podman[70117]: 2024-07-26 06:56:06.017371366 +0000 UTC m=+0.874641499 container init 407f493356c218e61d85a124f2469e8567115ab3f538adbe23ea5e60c9801379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T06:56:06.051 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:06 smithi163 podman[70117]: 2024-07-26 06:56:06.020289502 +0000 UTC m=+0.877559625 container start 407f493356c218e61d85a124f2469e8567115ab3f538adbe23ea5e60c9801379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T06:56:06.051 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:05 smithi163 podman[70107]: 2024-07-26 06:56:05.781885631 +0000 UTC m=+0.644473881 container init 85bdfb5f68dd6d00023df89da5d1a8290ae3a708bfa5e599060fcda834992c9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-07-26T06:56:06.051 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:05 smithi163 podman[70107]: 2024-07-26 06:56:05.784382491 +0000 UTC m=+0.646970728 container start 85bdfb5f68dd6d00023df89da5d1a8290ae3a708bfa5e599060fcda834992c9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, io.buildah.version=1.36.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T06:56:06.052 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:05 smithi163 ceph-mds[70163]: -- 172.21.15.163:0/3610705310 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5614532fd860 con 0x56145749f180 2024-07-26T06:56:06.052 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:05 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: starting mds.e at 2024-07-26T06:56:06.052 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:05 smithi163 bash[70107]: 85bdfb5f68dd6d00023df89da5d1a8290ae3a708bfa5e599060fcda834992c9a 2024-07-26T06:56:06.052 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:05 smithi163 systemd[1]: Started Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:06.365 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:06 smithi163 podman[70118]: 2024-07-26 06:56:06.244642502 +0000 UTC m=+1.100076355 container create 5c257bedce62e39379a6a9f9148fdf7e6433e7bb16259e2676ad6e77b52538ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T06:56:06.366 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mds[70184]: -- 172.21.15.163:0/2916324796 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5619dbdaf860 con 0x5619dff51180 2024-07-26T06:56:06.366 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: starting mds.b at 2024-07-26T06:56:06.366 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:06 smithi163 bash[70117]: 407f493356c218e61d85a124f2469e8567115ab3f538adbe23ea5e60c9801379 2024-07-26T06:56:06.366 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:06 smithi163 systemd[1]: Started Ceph mds.b for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:06.366 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6836/3683796852,v1:172.21.15.164:6837/3683796852] up:boot 2024-07-26T06:56:06.366 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/1430155791,v1:172.21.15.42:6841/1430155791] up:resolve 2024-07-26T06:56:06.366 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T06:56:06.366 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T06:56:06.366 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: pgmap v574: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 52 KiB/s wr, 16 op/s 2024-07-26T06:56:06.367 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: daemon mds.e restarted 2024-07-26T06:56:06.367 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T06:56:06.367 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/148229526,v1:172.21.15.163:6835/148229526] up:boot 2024-07-26T06:56:06.367 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-07-26T06:56:06.367 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=b=up:active,1=i=up:active,2=h=up:active,3=a=up:resolve} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:56:06.367 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:56:06.367 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:56:06.367 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: daemon mds.b restarted 2024-07-26T06:56:06.367 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T06:56:06.367 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/3426812468,v1:172.21.15.163:6837/3426812468] up:boot 2024-07-26T06:56:06.367 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T06:56:06.367 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {1=i=up:active,2=h=up:active,3=a=up:resolve,4=g=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:56:06.368 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T06:56:06.368 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:56:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6836/3683796852,v1:172.21.15.164:6837/3683796852] up:boot 2024-07-26T06:56:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/1430155791,v1:172.21.15.42:6841/1430155791] up:resolve 2024-07-26T06:56:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T06:56:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T06:56:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: pgmap v574: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 52 KiB/s wr, 16 op/s 2024-07-26T06:56:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: daemon mds.e restarted 2024-07-26T06:56:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T06:56:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/148229526,v1:172.21.15.163:6835/148229526] up:boot 2024-07-26T06:56:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-07-26T06:56:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=b=up:active,1=i=up:active,2=h=up:active,3=a=up:resolve} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:56:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:56:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:56:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: daemon mds.b restarted 2024-07-26T06:56:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T06:56:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/3426812468,v1:172.21.15.163:6837/3426812468] up:boot 2024-07-26T06:56:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T06:56:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {1=i=up:active,2=h=up:active,3=a=up:resolve,4=g=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:56:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T06:56:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:06 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:56:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6836/3683796852,v1:172.21.15.164:6837/3683796852] up:boot 2024-07-26T06:56:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/1430155791,v1:172.21.15.42:6841/1430155791] up:resolve 2024-07-26T06:56:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T06:56:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T06:56:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: pgmap v574: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 52 KiB/s wr, 16 op/s 2024-07-26T06:56:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: daemon mds.e restarted 2024-07-26T06:56:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T06:56:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/148229526,v1:172.21.15.163:6835/148229526] up:boot 2024-07-26T06:56:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-07-26T06:56:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=b=up:active,1=i=up:active,2=h=up:active,3=a=up:resolve} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:56:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:56:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:56:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: daemon mds.b restarted 2024-07-26T06:56:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T06:56:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/3426812468,v1:172.21.15.163:6837/3426812468] up:boot 2024-07-26T06:56:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-07-26T06:56:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {1=i=up:active,2=h=up:active,3=a=up:resolve,4=g=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:56:06.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T06:56:06.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:06 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:56:06.756 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:06 smithi163 podman[70118]: 2024-07-26 06:56:06.481639285 +0000 UTC m=+1.337073139 container init 5c257bedce62e39379a6a9f9148fdf7e6433e7bb16259e2676ad6e77b52538ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux ) 2024-07-26T06:56:06.757 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:06 smithi163 podman[70118]: 2024-07-26 06:56:06.484238019 +0000 UTC m=+1.339671874 container start 5c257bedce62e39379a6a9f9148fdf7e6433e7bb16259e2676ad6e77b52538ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-07-26T06:56:06.757 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-mds[70210]: -- 172.21.15.163:0/3866244818 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cc12c61860 con 0x55cc16e03600 2024-07-26T06:56:06.757 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:06 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: starting mds.h at 2024-07-26T06:56:06.757 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:06 smithi163 bash[70118]: 5c257bedce62e39379a6a9f9148fdf7e6433e7bb16259e2676ad6e77b52538ef 2024-07-26T06:56:06.757 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:06 smithi163 systemd[1]: Started Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:07.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:07 smithi164 ceph-mon[36970]: daemon mds.h restarted 2024-07-26T06:56:07.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:07 smithi164 ceph-mon[36970]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T06:56:07.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:07 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/4044632348,v1:172.21.15.42:6839/4044632348] up:resolve 2024-07-26T06:56:07.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:07 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:resolve 2024-07-26T06:56:07.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:07 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/692571500,v1:172.21.15.163:6839/692571500] up:boot 2024-07-26T06:56:07.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:07 smithi164 ceph-mon[36970]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-07-26T06:56:07.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:07 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:resolve,1=i=up:active,3=a=up:resolve,4=g=up:resolve} 2 up:standby-replay 6 up:standby, 1 failed 2024-07-26T06:56:07.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:07 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:56:07.626 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:07 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 1 up:replay 3 up:resolve 1 up:active 2024-07-26T06:56:07.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:07 smithi042 ceph-mon[32775]: daemon mds.h restarted 2024-07-26T06:56:07.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:07 smithi042 ceph-mon[32775]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T06:56:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:07 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/4044632348,v1:172.21.15.42:6839/4044632348] up:resolve 2024-07-26T06:56:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:07 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:resolve 2024-07-26T06:56:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:07 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/692571500,v1:172.21.15.163:6839/692571500] up:boot 2024-07-26T06:56:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:07 smithi042 ceph-mon[32775]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-07-26T06:56:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:07 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:resolve,1=i=up:active,3=a=up:resolve,4=g=up:resolve} 2 up:standby-replay 6 up:standby, 1 failed 2024-07-26T06:56:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:07 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:56:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:07 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 1 up:replay 3 up:resolve 1 up:active 2024-07-26T06:56:07.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:07 smithi163 ceph-mon[36873]: daemon mds.h restarted 2024-07-26T06:56:07.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:07 smithi163 ceph-mon[36873]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T06:56:07.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:07 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/4044632348,v1:172.21.15.42:6839/4044632348] up:resolve 2024-07-26T06:56:07.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:07 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:resolve 2024-07-26T06:56:07.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:07 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/692571500,v1:172.21.15.163:6839/692571500] up:boot 2024-07-26T06:56:07.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:07 smithi163 ceph-mon[36873]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-07-26T06:56:07.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:07 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:resolve,1=i=up:active,3=a=up:resolve,4=g=up:resolve} 2 up:standby-replay 6 up:standby, 1 failed 2024-07-26T06:56:07.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:07 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:56:07.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:07 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 1 up:replay 3 up:resolve 1 up:active 2024-07-26T06:56:08.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:08 smithi042 ceph-mon[32775]: pgmap v578: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 13 op/s 2024-07-26T06:56:08.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:08 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/4144543280,v1:172.21.15.164:6833/4144543280] up:resolve 2024-07-26T06:56:08.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:08 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:resolve 1 up:active 2024-07-26T06:56:08.898 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:08 smithi164 ceph-mon[36970]: pgmap v578: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 13 op/s 2024-07-26T06:56:08.898 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:08 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/4144543280,v1:172.21.15.164:6833/4144543280] up:resolve 2024-07-26T06:56:08.898 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:08 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:resolve 1 up:active 2024-07-26T06:56:09.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:08 smithi163 ceph-mon[36873]: pgmap v578: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 13 op/s 2024-07-26T06:56:09.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:08 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/4144543280,v1:172.21.15.164:6833/4144543280] up:resolve 2024-07-26T06:56:09.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:08 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:resolve 1 up:active 2024-07-26T06:56:09.832 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:09 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00199998 2024-07-26T06:56:09.832 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:09 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00299997 2024-07-26T06:56:09.832 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:09 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00299995 2024-07-26T06:56:09.832 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:09 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00299995 2024-07-26T06:56:09.832 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:09 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:reconnect 2024-07-26T06:56:09.832 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:09 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/4044632348,v1:172.21.15.42:6839/4044632348] up:reconnect 2024-07-26T06:56:09.833 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:09 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/4144543280,v1:172.21.15.164:6833/4144543280] up:reconnect 2024-07-26T06:56:09.833 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:09 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/1430155791,v1:172.21.15.42:6841/1430155791] up:reconnect 2024-07-26T06:56:09.833 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:09 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-07-26T06:56:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:09 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00199998 2024-07-26T06:56:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:09 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00299997 2024-07-26T06:56:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:09 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00299995 2024-07-26T06:56:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:09 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00299995 2024-07-26T06:56:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:09 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:reconnect 2024-07-26T06:56:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:09 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/4044632348,v1:172.21.15.42:6839/4044632348] up:reconnect 2024-07-26T06:56:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:09 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/4144543280,v1:172.21.15.164:6833/4144543280] up:reconnect 2024-07-26T06:56:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:09 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/1430155791,v1:172.21.15.42:6841/1430155791] up:reconnect 2024-07-26T06:56:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:09 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-07-26T06:56:10.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:09 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00199998 2024-07-26T06:56:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:09 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00299997 2024-07-26T06:56:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:09 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00299995 2024-07-26T06:56:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:09 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00299995 2024-07-26T06:56:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:09 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:reconnect 2024-07-26T06:56:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:09 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/4044632348,v1:172.21.15.42:6839/4044632348] up:reconnect 2024-07-26T06:56:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:09 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/4144543280,v1:172.21.15.164:6833/4144543280] up:reconnect 2024-07-26T06:56:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:09 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/1430155791,v1:172.21.15.42:6841/1430155791] up:reconnect 2024-07-26T06:56:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:09 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-07-26T06:56:10.478 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:10.477+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:56:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:10.478+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T06:56:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:10.478+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T06:56:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:10.478+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T06:56:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:10.478+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-07-26T06:56:10.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:10.855+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:56:10.856 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-mon[32775]: pgmap v579: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 12 op/s 2024-07-26T06:56:10.856 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/4044632348,v1:172.21.15.42:6839/4044632348] up:rejoin 2024-07-26T06:56:10.856 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/1430155791,v1:172.21.15.42:6841/1430155791] up:rejoin 2024-07-26T06:56:10.856 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/4144543280,v1:172.21.15.164:6833/4144543280] up:rejoin 2024-07-26T06:56:10.856 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:rejoin 2024-07-26T06:56:10.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-07-26T06:56:10.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-mon[32775]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:56:10.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-mon[32775]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-07-26T06:56:10.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-mon[32775]: daemon mds.a is now active in filesystem cephfs as rank 3 2024-07-26T06:56:10.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:10.856+0000 7faa2dcb9640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 msgr2=0x561ca20d0cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T06:56:10.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:10.856+0000 7faa2dcb9640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 msgr2=0x561ca20d0cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T06:56:10.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:10.856+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T06:56:10.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:10.856+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-07-26T06:56:10.858 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:56:10.858 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: *** Caught signal (Aborted) ** 2024-07-26T06:56:10.858 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: in thread 7f27ea91c640 thread_name:ms_dispatch 2024-07-26T06:56:10.858 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:10.858 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f27efcf36f0] 2024-07-26T06:56:10.858 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f27efd4094c] 2024-07-26T06:56:10.858 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 3: raise() 2024-07-26T06:56:10.858 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 4: abort() 2024-07-26T06:56:10.859 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 5: /lib64/libc.so.6(+0x2871b) [0x7f27efcdd71b] 2024-07-26T06:56:10.859 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 6: /lib64/libc.so.6(+0x37386) [0x7f27efcec386] 2024-07-26T06:56:10.859 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5602857976b9] 2024-07-26T06:56:10.859 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56028595580c] 2024-07-26T06:56:10.859 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5602858952ca] 2024-07-26T06:56:10.859 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560285865753] 2024-07-26T06:56:10.859 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560285866a1b] 2024-07-26T06:56:10.859 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56028584a752] 2024-07-26T06:56:10.859 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560285798a56] 2024-07-26T06:56:10.859 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56028579c637] 2024-07-26T06:56:10.859 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56028579cf5e] 2024-07-26T06:56:10.859 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56028578003a] 2024-07-26T06:56:10.859 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f27f05c0407] 2024-07-26T06:56:10.859 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 18: (DispatchQueue::entry()+0x60f) [0x7f27f05c0cf9] 2024-07-26T06:56:10.860 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f27f069d5a1] 2024-07-26T06:56:10.860 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:56:10.860 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: *** Caught signal (Aborted) ** 2024-07-26T06:56:10.861 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: in thread 7fb5852b4640 thread_name:ms_dispatch 2024-07-26T06:56:10.861 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:10.861 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb58a68b6f0] 2024-07-26T06:56:10.861 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb58a6d894c] 2024-07-26T06:56:10.861 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 3: raise() 2024-07-26T06:56:10.861 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 4: abort() 2024-07-26T06:56:10.861 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb58a67571b] 2024-07-26T06:56:10.861 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 6: /lib64/libc.so.6(+0x37386) [0x7fb58a684386] 2024-07-26T06:56:10.861 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55925d3556b9] 2024-07-26T06:56:10.861 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55925d51380c] 2024-07-26T06:56:10.861 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55925d4532ca] 2024-07-26T06:56:10.861 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55925d423753] 2024-07-26T06:56:10.862 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55925d424a1b] 2024-07-26T06:56:10.862 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55925d408752] 2024-07-26T06:56:10.862 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55925d356a56] 2024-07-26T06:56:10.862 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55925d35a637] 2024-07-26T06:56:10.862 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55925d35af5e] 2024-07-26T06:56:10.862 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55925d33e03a] 2024-07-26T06:56:10.862 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb58af58407] 2024-07-26T06:56:10.862 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 18: (DispatchQueue::entry()+0x60f) [0x7fb58af58cf9] 2024-07-26T06:56:10.862 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb58b0355a1] 2024-07-26T06:56:10.863 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 20: (Thread::entry_wrapper()+0x43) [0x7f27f044acdd] 2024-07-26T06:56:10.863 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 21: (Thread::_entry_func(void*)+0xd) [0x7f27f044acf9] 2024-07-26T06:56:10.863 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 22: /lib64/libc.so.6(+0x89c02) [0x7f27efd3ec02] 2024-07-26T06:56:10.863 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f27efdc3c40] 2024-07-26T06:56:10.863 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 2024-07-26T06:56:10.564+0000 7f27ea91c640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:10.863 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: in thread 7f27ea91c640 thread_name:ms_dispatch 2024-07-26T06:56:10.863 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 2024-07-26T06:56:10.863 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:10.863 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f27efcf36f0] 2024-07-26T06:56:10.863 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f27efd4094c] 2024-07-26T06:56:10.863 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 3: raise() 2024-07-26T06:56:10.863 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 4: abort() 2024-07-26T06:56:10.864 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 5: /lib64/libc.so.6(+0x2871b) [0x7f27efcdd71b] 2024-07-26T06:56:10.864 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 6: /lib64/libc.so.6(+0x37386) [0x7f27efcec386] 2024-07-26T06:56:10.864 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5602857976b9] 2024-07-26T06:56:10.864 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56028595580c] 2024-07-26T06:56:10.864 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5602858952ca] 2024-07-26T06:56:10.864 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560285865753] 2024-07-26T06:56:10.864 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560285866a1b] 2024-07-26T06:56:10.864 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56028584a752] 2024-07-26T06:56:10.864 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560285798a56] 2024-07-26T06:56:10.864 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56028579c637] 2024-07-26T06:56:10.864 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56028579cf5e] 2024-07-26T06:56:10.864 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56028578003a] 2024-07-26T06:56:10.865 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f27f05c0407] 2024-07-26T06:56:10.865 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 18: (DispatchQueue::entry()+0x60f) [0x7f27f05c0cf9] 2024-07-26T06:56:10.865 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f27f069d5a1] 2024-07-26T06:56:10.865 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 20: (Thread::entry_wrapper()+0x43) [0x7fb58ade2cdd] 2024-07-26T06:56:10.865 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb58ade2cf9] 2024-07-26T06:56:10.865 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb58a6d6c02] 2024-07-26T06:56:10.865 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb58a75bc40] 2024-07-26T06:56:10.865 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 2024-07-26T06:56:10.564+0000 7fb5852b4640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:10.865 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: in thread 7fb5852b4640 thread_name:ms_dispatch 2024-07-26T06:56:10.866 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 2024-07-26T06:56:10.866 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:10.866 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb58a68b6f0] 2024-07-26T06:56:10.866 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb58a6d894c] 2024-07-26T06:56:10.866 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 3: raise() 2024-07-26T06:56:10.866 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 4: abort() 2024-07-26T06:56:10.866 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb58a67571b] 2024-07-26T06:56:10.866 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 6: /lib64/libc.so.6(+0x37386) [0x7fb58a684386] 2024-07-26T06:56:10.866 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55925d3556b9] 2024-07-26T06:56:10.866 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55925d51380c] 2024-07-26T06:56:10.866 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55925d4532ca] 2024-07-26T06:56:10.867 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55925d423753] 2024-07-26T06:56:10.867 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55925d424a1b] 2024-07-26T06:56:10.867 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55925d408752] 2024-07-26T06:56:10.867 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55925d356a56] 2024-07-26T06:56:10.867 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55925d35a637] 2024-07-26T06:56:10.867 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55925d35af5e] 2024-07-26T06:56:10.867 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55925d33e03a] 2024-07-26T06:56:10.867 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb58af58407] 2024-07-26T06:56:10.867 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 18: (DispatchQueue::entry()+0x60f) [0x7fb58af58cf9] 2024-07-26T06:56:10.867 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb58b0355a1] 2024-07-26T06:56:10.868 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 20: (Thread::entry_wrapper()+0x43) [0x7f27f044acdd] 2024-07-26T06:56:10.868 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 21: (Thread::_entry_func(void*)+0xd) [0x7f27f044acf9] 2024-07-26T06:56:10.868 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 22: /lib64/libc.so.6(+0x89c02) [0x7f27efd3ec02] 2024-07-26T06:56:10.868 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f27efdc3c40] 2024-07-26T06:56:10.868 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:10.868 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 2024-07-26T06:56:10.868 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 0> 2024-07-26T06:56:10.564+0000 7f27ea91c640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:10.868 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: in thread 7f27ea91c640 thread_name:ms_dispatch 2024-07-26T06:56:10.868 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 2024-07-26T06:56:10.868 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:10.868 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f27efcf36f0] 2024-07-26T06:56:10.868 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f27efd4094c] 2024-07-26T06:56:10.868 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 3: raise() 2024-07-26T06:56:10.869 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 4: abort() 2024-07-26T06:56:10.869 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 5: /lib64/libc.so.6(+0x2871b) [0x7f27efcdd71b] 2024-07-26T06:56:10.869 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 6: /lib64/libc.so.6(+0x37386) [0x7f27efcec386] 2024-07-26T06:56:10.869 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5602857976b9] 2024-07-26T06:56:10.869 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56028595580c] 2024-07-26T06:56:10.869 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5602858952ca] 2024-07-26T06:56:10.869 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560285865753] 2024-07-26T06:56:10.869 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560285866a1b] 2024-07-26T06:56:10.869 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56028584a752] 2024-07-26T06:56:10.869 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560285798a56] 2024-07-26T06:56:10.869 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56028579c637] 2024-07-26T06:56:10.869 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56028579cf5e] 2024-07-26T06:56:10.870 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56028578003a] 2024-07-26T06:56:10.870 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f27f05c0407] 2024-07-26T06:56:10.870 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 20: (Thread::entry_wrapper()+0x43) [0x7fb58ade2cdd] 2024-07-26T06:56:10.870 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb58ade2cf9] 2024-07-26T06:56:10.870 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb58a6d6c02] 2024-07-26T06:56:10.870 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb58a75bc40] 2024-07-26T06:56:10.870 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:10.870 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 2024-07-26T06:56:10.870 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 0> 2024-07-26T06:56:10.564+0000 7fb5852b4640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:10.871 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: in thread 7fb5852b4640 thread_name:ms_dispatch 2024-07-26T06:56:10.871 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 2024-07-26T06:56:10.871 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:10.871 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb58a68b6f0] 2024-07-26T06:56:10.871 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb58a6d894c] 2024-07-26T06:56:10.871 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 3: raise() 2024-07-26T06:56:10.871 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 4: abort() 2024-07-26T06:56:10.871 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb58a67571b] 2024-07-26T06:56:10.871 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 6: /lib64/libc.so.6(+0x37386) [0x7fb58a684386] 2024-07-26T06:56:10.872 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55925d3556b9] 2024-07-26T06:56:10.872 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55925d51380c] 2024-07-26T06:56:10.872 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55925d4532ca] 2024-07-26T06:56:10.872 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55925d423753] 2024-07-26T06:56:10.872 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55925d424a1b] 2024-07-26T06:56:10.872 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55925d408752] 2024-07-26T06:56:10.872 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55925d356a56] 2024-07-26T06:56:10.872 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55925d35a637] 2024-07-26T06:56:10.872 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55925d35af5e] 2024-07-26T06:56:10.872 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55925d33e03a] 2024-07-26T06:56:10.873 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb58af58407] 2024-07-26T06:56:10.873 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 18: (DispatchQueue::entry()+0x60f) [0x7f27f05c0cf9] 2024-07-26T06:56:10.873 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f27f069d5a1] 2024-07-26T06:56:10.873 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 20: (Thread::entry_wrapper()+0x43) [0x7f27f044acdd] 2024-07-26T06:56:10.873 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 21: (Thread::_entry_func(void*)+0xd) [0x7f27f044acf9] 2024-07-26T06:56:10.873 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 22: /lib64/libc.so.6(+0x89c02) [0x7f27efd3ec02] 2024-07-26T06:56:10.874 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f27efdc3c40] 2024-07-26T06:56:10.874 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:10.874 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 2024-07-26T06:56:10.874 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 0> 2024-07-26T06:56:10.564+0000 7f27ea91c640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:10.874 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: in thread 7f27ea91c640 thread_name:ms_dispatch 2024-07-26T06:56:10.874 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 2024-07-26T06:56:10.874 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:10.874 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f27efcf36f0] 2024-07-26T06:56:10.874 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f27efd4094c] 2024-07-26T06:56:10.874 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 3: raise() 2024-07-26T06:56:10.874 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 4: abort() 2024-07-26T06:56:10.875 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 5: /lib64/libc.so.6(+0x2871b) [0x7f27efcdd71b] 2024-07-26T06:56:10.875 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 6: /lib64/libc.so.6(+0x37386) [0x7f27efcec386] 2024-07-26T06:56:10.875 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5602857976b9] 2024-07-26T06:56:10.875 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56028595580c] 2024-07-26T06:56:10.875 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5602858952ca] 2024-07-26T06:56:10.875 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560285865753] 2024-07-26T06:56:10.875 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560285866a1b] 2024-07-26T06:56:10.875 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56028584a752] 2024-07-26T06:56:10.875 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560285798a56] 2024-07-26T06:56:10.875 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56028579c637] 2024-07-26T06:56:10.875 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56028579cf5e] 2024-07-26T06:56:10.875 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56028578003a] 2024-07-26T06:56:10.875 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f27f05c0407] 2024-07-26T06:56:10.876 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 18: (DispatchQueue::entry()+0x60f) [0x7f27f05c0cf9] 2024-07-26T06:56:10.876 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f27f069d5a1] 2024-07-26T06:56:10.876 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 20: (Thread::entry_wrapper()+0x43) [0x7f27f044acdd] 2024-07-26T06:56:10.876 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 21: (Thread::_entry_func(void*)+0xd) [0x7f27f044acf9] 2024-07-26T06:56:10.876 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 22: /lib64/libc.so.6(+0x89c02) [0x7f27efd3ec02] 2024-07-26T06:56:10.876 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f27efdc3c40] 2024-07-26T06:56:10.876 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:10.876 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[77027]: 2024-07-26T06:56:10.876 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:10 smithi042 podman[118447]: 2024-07-26 06:56:10.644801844 +0000 UTC m=+0.023670855 container died 3e3da20f6e4dca345cc00478eaf1049f852fda21c2ba2f4022d9c69642d83050 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T06:56:10.877 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 18: (DispatchQueue::entry()+0x60f) [0x7fb58af58cf9] 2024-07-26T06:56:10.877 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb58b0355a1] 2024-07-26T06:56:10.877 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 20: (Thread::entry_wrapper()+0x43) [0x7fb58ade2cdd] 2024-07-26T06:56:10.877 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb58ade2cf9] 2024-07-26T06:56:10.877 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb58a6d6c02] 2024-07-26T06:56:10.877 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb58a75bc40] 2024-07-26T06:56:10.877 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:10.877 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 2024-07-26T06:56:10.877 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 0> 2024-07-26T06:56:10.564+0000 7fb5852b4640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:10.877 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: in thread 7fb5852b4640 thread_name:ms_dispatch 2024-07-26T06:56:10.877 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 2024-07-26T06:56:10.877 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:10.878 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb58a68b6f0] 2024-07-26T06:56:10.878 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb58a6d894c] 2024-07-26T06:56:10.878 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 3: raise() 2024-07-26T06:56:10.878 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 4: abort() 2024-07-26T06:56:10.878 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb58a67571b] 2024-07-26T06:56:10.878 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 6: /lib64/libc.so.6(+0x37386) [0x7fb58a684386] 2024-07-26T06:56:10.878 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55925d3556b9] 2024-07-26T06:56:10.878 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55925d51380c] 2024-07-26T06:56:10.878 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55925d4532ca] 2024-07-26T06:56:10.878 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55925d423753] 2024-07-26T06:56:10.878 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55925d424a1b] 2024-07-26T06:56:10.878 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55925d408752] 2024-07-26T06:56:10.879 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55925d356a56] 2024-07-26T06:56:10.879 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55925d35a637] 2024-07-26T06:56:10.879 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55925d35af5e] 2024-07-26T06:56:10.879 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55925d33e03a] 2024-07-26T06:56:10.879 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb58af58407] 2024-07-26T06:56:10.879 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 18: (DispatchQueue::entry()+0x60f) [0x7fb58af58cf9] 2024-07-26T06:56:10.879 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb58b0355a1] 2024-07-26T06:56:10.879 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 20: (Thread::entry_wrapper()+0x43) [0x7fb58ade2cdd] 2024-07-26T06:56:10.879 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb58ade2cf9] 2024-07-26T06:56:10.879 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb58a6d6c02] 2024-07-26T06:56:10.879 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb58a75bc40] 2024-07-26T06:56:10.879 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:10.879 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:10 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[77364]: 2024-07-26T06:56:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-mon[36970]: pgmap v579: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 12 op/s 2024-07-26T06:56:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/4044632348,v1:172.21.15.42:6839/4044632348] up:rejoin 2024-07-26T06:56:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/1430155791,v1:172.21.15.42:6841/1430155791] up:rejoin 2024-07-26T06:56:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/4144543280,v1:172.21.15.164:6833/4144543280] up:rejoin 2024-07-26T06:56:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:rejoin 2024-07-26T06:56:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-07-26T06:56:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-mon[36970]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:56:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-mon[36970]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-07-26T06:56:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-mon[36970]: daemon mds.a is now active in filesystem cephfs as rank 3 2024-07-26T06:56:10.973 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:56:10.973 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: *** Caught signal (Aborted) ** 2024-07-26T06:56:10.973 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: in thread 7f3366070640 thread_name:ms_dispatch 2024-07-26T06:56:10.973 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:10.973 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f336b4476f0] 2024-07-26T06:56:10.973 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f336b49494c] 2024-07-26T06:56:10.974 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 3: raise() 2024-07-26T06:56:10.974 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 4: abort() 2024-07-26T06:56:10.974 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 5: /lib64/libc.so.6(+0x2871b) [0x7f336b43171b] 2024-07-26T06:56:10.974 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 6: /lib64/libc.so.6(+0x37386) [0x7f336b440386] 2024-07-26T06:56:10.974 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55934826a6b9] 2024-07-26T06:56:10.974 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55934842880c] 2024-07-26T06:56:10.974 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5593483682ca] 2024-07-26T06:56:10.974 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559348338753] 2024-07-26T06:56:10.975 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559348339a1b] 2024-07-26T06:56:10.975 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55934831d752] 2024-07-26T06:56:10.975 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55934826ba56] 2024-07-26T06:56:10.975 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55934826f637] 2024-07-26T06:56:10.975 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55934826ff5e] 2024-07-26T06:56:10.975 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55934825303a] 2024-07-26T06:56:10.975 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f336bd14407] 2024-07-26T06:56:10.975 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 18: (DispatchQueue::entry()+0x60f) [0x7f336bd14cf9] 2024-07-26T06:56:10.975 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f336bdf15a1] 2024-07-26T06:56:10.976 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 20: (Thread::entry_wrapper()+0x43) [0x7f336bb9ecdd] 2024-07-26T06:56:10.976 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 21: (Thread::_entry_func(void*)+0xd) [0x7f336bb9ecf9] 2024-07-26T06:56:10.976 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 22: /lib64/libc.so.6(+0x89c02) [0x7f336b492c02] 2024-07-26T06:56:10.976 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f336b517c40] 2024-07-26T06:56:10.976 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 2024-07-26T06:56:10.563+0000 7f3366070640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:10.976 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: in thread 7f3366070640 thread_name:ms_dispatch 2024-07-26T06:56:10.976 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 2024-07-26T06:56:10.977 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:10.977 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f336b4476f0] 2024-07-26T06:56:10.977 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f336b49494c] 2024-07-26T06:56:10.977 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 3: raise() 2024-07-26T06:56:10.977 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 4: abort() 2024-07-26T06:56:10.977 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 5: /lib64/libc.so.6(+0x2871b) [0x7f336b43171b] 2024-07-26T06:56:10.977 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 6: /lib64/libc.so.6(+0x37386) [0x7f336b440386] 2024-07-26T06:56:10.977 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55934826a6b9] 2024-07-26T06:56:10.977 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55934842880c] 2024-07-26T06:56:10.977 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5593483682ca] 2024-07-26T06:56:10.977 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559348338753] 2024-07-26T06:56:10.977 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559348339a1b] 2024-07-26T06:56:10.978 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55934831d752] 2024-07-26T06:56:10.978 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55934826ba56] 2024-07-26T06:56:10.978 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55934826f637] 2024-07-26T06:56:10.978 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55934826ff5e] 2024-07-26T06:56:10.978 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55934825303a] 2024-07-26T06:56:10.978 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f336bd14407] 2024-07-26T06:56:10.978 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 18: (DispatchQueue::entry()+0x60f) [0x7f336bd14cf9] 2024-07-26T06:56:10.978 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f336bdf15a1] 2024-07-26T06:56:10.978 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 20: (Thread::entry_wrapper()+0x43) [0x7f336bb9ecdd] 2024-07-26T06:56:10.978 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 21: (Thread::_entry_func(void*)+0xd) [0x7f336bb9ecf9] 2024-07-26T06:56:10.978 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 22: /lib64/libc.so.6(+0x89c02) [0x7f336b492c02] 2024-07-26T06:56:10.978 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f336b517c40] 2024-07-26T06:56:10.979 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:10.979 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 2024-07-26T06:56:10.979 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 0> 2024-07-26T06:56:10.563+0000 7f3366070640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:10.979 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: in thread 7f3366070640 thread_name:ms_dispatch 2024-07-26T06:56:10.979 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 2024-07-26T06:56:10.979 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:10.979 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f336b4476f0] 2024-07-26T06:56:10.979 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f336b49494c] 2024-07-26T06:56:10.979 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 3: raise() 2024-07-26T06:56:10.979 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 4: abort() 2024-07-26T06:56:10.979 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 5: /lib64/libc.so.6(+0x2871b) [0x7f336b43171b] 2024-07-26T06:56:10.979 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 6: /lib64/libc.so.6(+0x37386) [0x7f336b440386] 2024-07-26T06:56:10.980 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55934826a6b9] 2024-07-26T06:56:10.980 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55934842880c] 2024-07-26T06:56:10.980 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5593483682ca] 2024-07-26T06:56:10.980 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559348338753] 2024-07-26T06:56:10.980 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559348339a1b] 2024-07-26T06:56:10.980 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55934831d752] 2024-07-26T06:56:10.980 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55934826ba56] 2024-07-26T06:56:10.980 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55934826f637] 2024-07-26T06:56:10.980 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55934826ff5e] 2024-07-26T06:56:10.980 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55934825303a] 2024-07-26T06:56:10.980 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f336bd14407] 2024-07-26T06:56:10.980 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 18: (DispatchQueue::entry()+0x60f) [0x7f336bd14cf9] 2024-07-26T06:56:10.980 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f336bdf15a1] 2024-07-26T06:56:10.981 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 20: (Thread::entry_wrapper()+0x43) [0x7f336bb9ecdd] 2024-07-26T06:56:10.981 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 21: (Thread::_entry_func(void*)+0xd) [0x7f336bb9ecf9] 2024-07-26T06:56:10.981 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 22: /lib64/libc.so.6(+0x89c02) [0x7f336b492c02] 2024-07-26T06:56:10.981 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f336b517c40] 2024-07-26T06:56:10.981 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:10.981 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 2024-07-26T06:56:10.981 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 0> 2024-07-26T06:56:10.563+0000 7f3366070640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:10.981 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: in thread 7f3366070640 thread_name:ms_dispatch 2024-07-26T06:56:10.981 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 2024-07-26T06:56:10.981 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:10.981 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f336b4476f0] 2024-07-26T06:56:10.981 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f336b49494c] 2024-07-26T06:56:10.981 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 3: raise() 2024-07-26T06:56:10.981 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 4: abort() 2024-07-26T06:56:10.982 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 5: /lib64/libc.so.6(+0x2871b) [0x7f336b43171b] 2024-07-26T06:56:10.982 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 6: /lib64/libc.so.6(+0x37386) [0x7f336b440386] 2024-07-26T06:56:10.982 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55934826a6b9] 2024-07-26T06:56:10.982 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55934842880c] 2024-07-26T06:56:10.982 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5593483682ca] 2024-07-26T06:56:10.982 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559348338753] 2024-07-26T06:56:10.982 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559348339a1b] 2024-07-26T06:56:10.982 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55934831d752] 2024-07-26T06:56:10.982 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55934826ba56] 2024-07-26T06:56:10.982 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55934826f637] 2024-07-26T06:56:10.982 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55934826ff5e] 2024-07-26T06:56:10.983 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55934825303a] 2024-07-26T06:56:10.983 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f336bd14407] 2024-07-26T06:56:10.983 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 18: (DispatchQueue::entry()+0x60f) [0x7f336bd14cf9] 2024-07-26T06:56:10.983 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f336bdf15a1] 2024-07-26T06:56:10.983 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 20: (Thread::entry_wrapper()+0x43) [0x7f336bb9ecdd] 2024-07-26T06:56:10.983 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 21: (Thread::_entry_func(void*)+0xd) [0x7f336bb9ecf9] 2024-07-26T06:56:10.983 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 22: /lib64/libc.so.6(+0x89c02) [0x7f336b492c02] 2024-07-26T06:56:10.983 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f336b517c40] 2024-07-26T06:56:10.983 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:10.983 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[66254]: 2024-07-26T06:56:10.983 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:10 smithi164 podman[70276]: 2024-07-26 06:56:10.619098767 +0000 UTC m=+0.018423591 container died e9a5e5169b93b9e5b0a686baa294eee7a03693fbe18bc4733061f1edb7041fe9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-07-26T06:56:11.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:10 smithi163 ceph-mon[36873]: pgmap v579: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 12 op/s 2024-07-26T06:56:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:10 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/4044632348,v1:172.21.15.42:6839/4044632348] up:rejoin 2024-07-26T06:56:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:10 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/1430155791,v1:172.21.15.42:6841/1430155791] up:rejoin 2024-07-26T06:56:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:10 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/4144543280,v1:172.21.15.164:6833/4144543280] up:rejoin 2024-07-26T06:56:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:10 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:rejoin 2024-07-26T06:56:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:10 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-07-26T06:56:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:10 smithi163 ceph-mon[36873]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:56:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:10 smithi163 ceph-mon[36873]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-07-26T06:56:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:10 smithi163 ceph-mon[36873]: daemon mds.a is now active in filesystem cephfs as rank 3 2024-07-26T06:56:11.627 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:11 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:clientreplay 2024-07-26T06:56:11.627 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:11 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/4144543280,v1:172.21.15.164:6833/4144543280] up:active 2024-07-26T06:56:11.627 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:11 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/4044632348,v1:172.21.15.42:6839/4044632348] up:active 2024-07-26T06:56:11.627 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:11 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/1430155791,v1:172.21.15.42:6841/1430155791] up:active 2024-07-26T06:56:11.627 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:11 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 1 up:clientreplay 4 up:active 2024-07-26T06:56:11.627 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:11 smithi164 ceph-mon[36970]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T06:56:11.627 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:56:11.628 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:11 smithi164 podman[70276]: 2024-07-26 06:56:11.375555051 +0000 UTC m=+0.774879871 container remove e9a5e5169b93b9e5b0a686baa294eee7a03693fbe18bc4733061f1edb7041fe9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-07-26T06:56:11.628 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:11 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:56:11.628 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:56:11 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-f[66559]: 2024-07-26T06:56:11.570+0000 7fdd38b66640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T06:56:11.628 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:56:11 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-f[66559]: 2024-07-26T06:56:11.570+0000 7fdd38b66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T06:56:11.628 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:56:11 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-f[66559]: 2024-07-26T06:56:11.570+0000 7fdd38b66640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-07-26T06:56:11.628 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:56:11 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-f[66559]: 2024-07-26T06:56:11.570+0000 7fdd38b66640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-07-26T06:56:11.628 INFO:journalctl@ceph.mds.f.smithi164.stdout:Jul 26 06:56:11 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-f[66559]: 2024-07-26T06:56:11.570+0000 7fdd38b66640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-07-26T06:56:11.644 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:11 smithi042 podman[118448]: 2024-07-26 06:56:11.476240409 +0000 UTC m=+0.849895518 container died 84e1094b936b61dbe1abfb18a38ee674f356d345b9102572498214a2967d416a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, ceph=True) 2024-07-26T06:56:11.645 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:11 smithi042 podman[118447]: 2024-07-26 06:56:11.391788019 +0000 UTC m=+0.770657076 container remove 3e3da20f6e4dca345cc00478eaf1049f852fda21c2ba2f4022d9c69642d83050 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-07-26T06:56:11.645 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:11 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:56:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:11 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:clientreplay 2024-07-26T06:56:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:11 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/4144543280,v1:172.21.15.164:6833/4144543280] up:active 2024-07-26T06:56:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:11 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/4044632348,v1:172.21.15.42:6839/4044632348] up:active 2024-07-26T06:56:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:11 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/1430155791,v1:172.21.15.42:6841/1430155791] up:active 2024-07-26T06:56:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:11 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 1 up:clientreplay 4 up:active 2024-07-26T06:56:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:11 smithi042 ceph-mon[32775]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T06:56:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:56:11.972 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:11 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-07-26T06:56:11.972 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:11 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Consumed 1.726s CPU time. 2024-07-26T06:56:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:11 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:clientreplay 2024-07-26T06:56:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:11 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/4144543280,v1:172.21.15.164:6833/4144543280] up:active 2024-07-26T06:56:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:11 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/4044632348,v1:172.21.15.42:6839/4044632348] up:active 2024-07-26T06:56:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:11 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/1430155791,v1:172.21.15.42:6841/1430155791] up:active 2024-07-26T06:56:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:11 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 1 up:clientreplay 4 up:active 2024-07-26T06:56:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:11 smithi163 ceph-mon[36873]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T06:56:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:56:12.345 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:12 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-07-26T06:56:12.345 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:12 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Consumed 1.752s CPU time. 2024-07-26T06:56:12.345 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:12 smithi042 podman[118448]: 2024-07-26 06:56:12.080197542 +0000 UTC m=+1.453852637 container remove 84e1094b936b61dbe1abfb18a38ee674f356d345b9102572498214a2967d416a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-07-26T06:56:12.345 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:12 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:56:12.644 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:12 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T06:56:12.645 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:12 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Consumed 1.804s CPU time. 2024-07-26T06:56:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: pgmap v580: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T06:56:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:56:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: Cluster is now healthy 2024-07-26T06:56:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:active 2024-07-26T06:56:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: fsmap cephfs:5 1 up:standby-replay 6 up:standby 5 up:active 2024-07-26T06:56:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:56:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T06:56:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T06:56:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T06:56:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T06:56:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:56:12.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:56:12.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:56:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: pgmap v580: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T06:56:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:56:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: Cluster is now healthy 2024-07-26T06:56:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:active 2024-07-26T06:56:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: fsmap cephfs:5 1 up:standby-replay 6 up:standby 5 up:active 2024-07-26T06:56:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:56:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T06:56:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T06:56:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T06:56:12.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T06:56:12.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:12.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:56:12.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:56:12.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:56:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: pgmap v580: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T06:56:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:56:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: Cluster is now healthy 2024-07-26T06:56:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:active 2024-07-26T06:56:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: fsmap cephfs:5 1 up:standby-replay 6 up:standby 5 up:active 2024-07-26T06:56:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:56:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T06:56:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T06:56:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T06:56:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T06:56:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:13.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T06:56:13.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:56:13.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:56:13.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:56:13.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:13 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T06:56:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:13 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T06:56:13.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:13 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T06:56:13.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:13 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T06:56:14.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:13 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T06:56:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:13 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T06:56:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:14 smithi042 ceph-mon[32775]: pgmap v581: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 7.1 KiB/s wr, 22 op/s 2024-07-26T06:56:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:14 smithi164 ceph-mon[36970]: pgmap v581: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 7.1 KiB/s wr, 22 op/s 2024-07-26T06:56:15.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:14 smithi163 ceph-mon[36873]: pgmap v581: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 7.1 KiB/s wr, 22 op/s 2024-07-26T06:56:16.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:16 smithi042 ceph-mon[32775]: pgmap v582: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 6.6 KiB/s wr, 28 op/s 2024-07-26T06:56:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:16 smithi164 ceph-mon[36970]: pgmap v582: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 6.6 KiB/s wr, 28 op/s 2024-07-26T06:56:17.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:16 smithi163 ceph-mon[36873]: pgmap v582: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 6.6 KiB/s wr, 28 op/s 2024-07-26T06:56:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:18 smithi042 ceph-mon[32775]: pgmap v583: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 6.0 KiB/s wr, 26 op/s 2024-07-26T06:56:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:18 smithi164 ceph-mon[36970]: pgmap v583: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 6.0 KiB/s wr, 26 op/s 2024-07-26T06:56:19.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:18 smithi163 ceph-mon[36873]: pgmap v583: 97 pgs: 97 active+clean; 69 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 6.0 KiB/s wr, 26 op/s 2024-07-26T06:56:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:20 smithi042 ceph-mon[32775]: pgmap v584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 5.8 KiB/s wr, 25 op/s 2024-07-26T06:56:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:20 smithi164 ceph-mon[36970]: pgmap v584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 5.8 KiB/s wr, 25 op/s 2024-07-26T06:56:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:20 smithi163 ceph-mon[36873]: pgmap v584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 5.8 KiB/s wr, 25 op/s 2024-07-26T06:56:22.001 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:21 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 1. 2024-07-26T06:56:22.001 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:21 smithi164 systemd[1]: Stopped Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:22.001 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:21 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Consumed 1.726s CPU time. 2024-07-26T06:56:22.001 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:21 smithi164 systemd[1]: Starting Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:56:22.001 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:21 smithi164 podman[70518]: 2024-07-26 06:56:21.860915855 +0000 UTC m=+0.017493927 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:56:22.001 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:22 smithi164 podman[70518]: 2024-07-26 06:56:22.001609938 +0000 UTC m=+0.158188002 container create 8b180b3acc87dfbc396e1af8b6e20f4917a902f4c48e20557de55ca78b8a989f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-07-26T06:56:22.584 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:22 smithi164 podman[70518]: 2024-07-26 06:56:22.263226502 +0000 UTC m=+0.419804580 container init 8b180b3acc87dfbc396e1af8b6e20f4917a902f4c48e20557de55ca78b8a989f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T06:56:22.584 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:22 smithi164 podman[70518]: 2024-07-26 06:56:22.26583778 +0000 UTC m=+0.422415850 container start 8b180b3acc87dfbc396e1af8b6e20f4917a902f4c48e20557de55ca78b8a989f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T06:56:22.584 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:22 smithi164 ceph-mds[70537]: -- 172.21.15.164:0/2462368263 <== mon.1 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5586264f3860 con 0x55862a695a80 2024-07-26T06:56:22.584 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:22 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: starting mds.l at 2024-07-26T06:56:22.584 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:22 smithi164 bash[70518]: 8b180b3acc87dfbc396e1af8b6e20f4917a902f4c48e20557de55ca78b8a989f 2024-07-26T06:56:22.584 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:22 smithi164 systemd[1]: Started Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:22.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:22 smithi042 ceph-mon[32775]: pgmap v585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 5.8 KiB/s wr, 22 op/s 2024-07-26T06:56:22.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:22 smithi042 ceph-mon[32775]: daemon mds.l restarted 2024-07-26T06:56:22.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:22 smithi042 ceph-mon[32775]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:56:22.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:22 smithi042 ceph-mon[32775]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T06:56:22.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:22 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/2610757455,v1:172.21.15.164:6833/2610757455] up:boot 2024-07-26T06:56:22.649 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:22 smithi042 ceph-mon[32775]: Standby daemon mds.h assigned to filesystem cephfs as rank 2 2024-07-26T06:56:22.649 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:22 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:56:22.649 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:56:22.649 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:22 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:56:22.649 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:22 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 1. 2024-07-26T06:56:22.649 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:22 smithi042 systemd[1]: Stopped Ceph mds.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:22.649 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:22 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Consumed 1.804s CPU time. 2024-07-26T06:56:22.649 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:22 smithi042 systemd[1]: Starting Ceph mds.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:56:22.649 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:22 smithi042 podman[118753]: 2024-07-26 06:56:22.547964373 +0000 UTC m=+0.023621672 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:56:22.650 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:22 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 1. 2024-07-26T06:56:22.650 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:22 smithi042 systemd[1]: Stopped Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:22.650 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:22 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Consumed 1.752s CPU time. 2024-07-26T06:56:22.650 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:22 smithi042 systemd[1]: Starting Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:56:22.650 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:22 smithi042 podman[118752]: 2024-07-26 06:56:22.545518923 +0000 UTC m=+0.021038201 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:56:22.960 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:22 smithi042 podman[118753]: 2024-07-26 06:56:22.85914262 +0000 UTC m=+0.334799925 container create ae33759893203d9fb8f6c64b90598dbc6dab22d34d3d9971b67e3e7d2e6c574b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-07-26T06:56:22.960 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:22 smithi042 podman[118752]: 2024-07-26 06:56:22.666410844 +0000 UTC m=+0.141930122 container create 75a652fcdbdd7b936f12ae774f5861533e5759a015bcfdcb1c753589a7dfd0b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-07-26T06:56:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:22 smithi164 ceph-mon[36970]: pgmap v585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 5.8 KiB/s wr, 22 op/s 2024-07-26T06:56:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:22 smithi164 ceph-mon[36970]: daemon mds.l restarted 2024-07-26T06:56:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:22 smithi164 ceph-mon[36970]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:56:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:22 smithi164 ceph-mon[36970]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T06:56:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:22 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/2610757455,v1:172.21.15.164:6833/2610757455] up:boot 2024-07-26T06:56:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:22 smithi164 ceph-mon[36970]: Standby daemon mds.h assigned to filesystem cephfs as rank 2 2024-07-26T06:56:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:22 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:56:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:56:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:22 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:56:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:22 smithi163 ceph-mon[36873]: pgmap v585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 5.8 KiB/s wr, 22 op/s 2024-07-26T06:56:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:22 smithi163 ceph-mon[36873]: daemon mds.l restarted 2024-07-26T06:56:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:22 smithi163 ceph-mon[36873]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:56:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:22 smithi163 ceph-mon[36873]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T06:56:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:22 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/2610757455,v1:172.21.15.164:6833/2610757455] up:boot 2024-07-26T06:56:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:22 smithi163 ceph-mon[36873]: Standby daemon mds.h assigned to filesystem cephfs as rank 2 2024-07-26T06:56:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:22 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:56:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:56:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:22 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:56:23.281 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:23.281+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:56:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:23.281+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T06:56:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:23.281+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T06:56:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:23.281+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T06:56:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:23.281+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:56:23.322 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:22 smithi042 podman[118752]: 2024-07-26 06:56:22.960693388 +0000 UTC m=+0.436212679 container init 75a652fcdbdd7b936f12ae774f5861533e5759a015bcfdcb1c753589a7dfd0b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-07-26T06:56:23.322 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:22 smithi042 podman[118752]: 2024-07-26 06:56:22.963828418 +0000 UTC m=+0.439347697 container start 75a652fcdbdd7b936f12ae774f5861533e5759a015bcfdcb1c753589a7dfd0b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-07-26T06:56:23.322 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:22 smithi042 ceph-mds[118795]: -- 172.21.15.42:0/3208367997 <== mon.0 v2:172.21.15.42:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5557c2211c20 con 0x5557c63b3180 2024-07-26T06:56:23.322 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:22 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: starting mds.g at 2024-07-26T06:56:23.322 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:23 smithi042 bash[118752]: 75a652fcdbdd7b936f12ae774f5861533e5759a015bcfdcb1c753589a7dfd0b3 2024-07-26T06:56:23.322 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:23 smithi042 systemd[1]: Started Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:23.585 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:23 smithi042 podman[118753]: 2024-07-26 06:56:23.321864995 +0000 UTC m=+0.797522301 container init ae33759893203d9fb8f6c64b90598dbc6dab22d34d3d9971b67e3e7d2e6c574b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-07-26T06:56:23.585 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:23 smithi042 podman[118753]: 2024-07-26 06:56:23.324871318 +0000 UTC m=+0.800528617 container start ae33759893203d9fb8f6c64b90598dbc6dab22d34d3d9971b67e3e7d2e6c574b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-07-26T06:56:23.586 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:23 smithi042 ceph-mds[118820]: -- 172.21.15.42:0/2948233212 <== mon.0 v2:172.21.15.42:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fee673b860 con 0x55feea8dd600 2024-07-26T06:56:23.586 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:23 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: starting mds.a at 2024-07-26T06:56:23.586 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:23 smithi042 bash[118753]: ae33759893203d9fb8f6c64b90598dbc6dab22d34d3d9971b67e3e7d2e6c574b 2024-07-26T06:56:23.586 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:23 smithi042 systemd[1]: Started Ceph mds.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:23.669+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:56:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:23.670+0000 7faa2dcb9640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 msgr2=0x561ca20d0cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T06:56:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:23.670+0000 7faa2dcb9640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 msgr2=0x561ca20d0cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T06:56:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:23.670+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T06:56:23.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:23.670+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:56:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:23 smithi042 ceph-mon[32775]: daemon mds.g restarted 2024-07-26T06:56:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:23 smithi042 ceph-mon[32775]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T06:56:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:23 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/692571500,v1:172.21.15.163:6839/692571500] up:resolve 2024-07-26T06:56:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:23 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3874449504,v1:172.21.15.42:6839/3874449504] up:boot 2024-07-26T06:56:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:23 smithi042 ceph-mon[32775]: Standby daemon mds.e assigned to filesystem cephfs as rank 4 2024-07-26T06:56:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:23 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,3=a=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:56:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:56:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:23 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:56:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:23 smithi042 ceph-mon[32775]: daemon mds.a restarted 2024-07-26T06:56:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:23 smithi042 ceph-mon[32775]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T06:56:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:23 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/1032688799,v1:172.21.15.42:6841/1032688799] up:boot 2024-07-26T06:56:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:23 smithi042 ceph-mon[32775]: Standby daemon mds.b assigned to filesystem cephfs as rank 3 2024-07-26T06:56:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:23 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,4=e=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:56:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T06:56:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:23 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:56:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:23 smithi164 ceph-mon[36970]: daemon mds.g restarted 2024-07-26T06:56:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:23 smithi164 ceph-mon[36970]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T06:56:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:23 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/692571500,v1:172.21.15.163:6839/692571500] up:resolve 2024-07-26T06:56:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:23 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3874449504,v1:172.21.15.42:6839/3874449504] up:boot 2024-07-26T06:56:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:23 smithi164 ceph-mon[36970]: Standby daemon mds.e assigned to filesystem cephfs as rank 4 2024-07-26T06:56:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:23 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,3=a=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:56:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:56:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:23 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:56:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:23 smithi164 ceph-mon[36970]: daemon mds.a restarted 2024-07-26T06:56:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:23 smithi164 ceph-mon[36970]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T06:56:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:23 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/1032688799,v1:172.21.15.42:6841/1032688799] up:boot 2024-07-26T06:56:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:23 smithi164 ceph-mon[36970]: Standby daemon mds.b assigned to filesystem cephfs as rank 3 2024-07-26T06:56:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:23 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,4=e=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:56:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T06:56:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:23 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:56:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:23 smithi163 ceph-mon[36873]: daemon mds.g restarted 2024-07-26T06:56:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:23 smithi163 ceph-mon[36873]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T06:56:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:23 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/692571500,v1:172.21.15.163:6839/692571500] up:resolve 2024-07-26T06:56:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:23 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3874449504,v1:172.21.15.42:6839/3874449504] up:boot 2024-07-26T06:56:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:23 smithi163 ceph-mon[36873]: Standby daemon mds.e assigned to filesystem cephfs as rank 4 2024-07-26T06:56:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:23 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,3=a=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:56:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:56:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:23 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:56:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:23 smithi163 ceph-mon[36873]: daemon mds.a restarted 2024-07-26T06:56:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:23 smithi163 ceph-mon[36873]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T06:56:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:23 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/1032688799,v1:172.21.15.42:6841/1032688799] up:boot 2024-07-26T06:56:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:23 smithi163 ceph-mon[36873]: Standby daemon mds.b assigned to filesystem cephfs as rank 3 2024-07-26T06:56:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:23 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,4=e=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:56:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T06:56:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:23 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:56:24.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:24 smithi042 ceph-mon[32775]: pgmap v589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 11 op/s 2024-07-26T06:56:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:24 smithi164 ceph-mon[36970]: pgmap v589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 11 op/s 2024-07-26T06:56:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:24 smithi163 ceph-mon[36873]: pgmap v589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 11 op/s 2024-07-26T06:56:25.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:25 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/148229526,v1:172.21.15.163:6835/148229526] up:resolve 2024-07-26T06:56:25.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:25 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/3426812468,v1:172.21.15.163:6837/3426812468] up:resolve 2024-07-26T06:56:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:25 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:56:25.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:25 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/148229526,v1:172.21.15.163:6835/148229526] up:resolve 2024-07-26T06:56:25.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:25 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/3426812468,v1:172.21.15.163:6837/3426812468] up:resolve 2024-07-26T06:56:25.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:25 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:56:26.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:25 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/148229526,v1:172.21.15.163:6835/148229526] up:resolve 2024-07-26T06:56:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:25 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/3426812468,v1:172.21.15.163:6837/3426812468] up:resolve 2024-07-26T06:56:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:25 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:56:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:26 smithi042 ceph-mon[32775]: pgmap v590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T06:56:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:26 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:56:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:26 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:56:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:26 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:56:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:26 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/148229526,v1:172.21.15.163:6835/148229526] up:reconnect 2024-07-26T06:56:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:26 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/692571500,v1:172.21.15.163:6839/692571500] up:reconnect 2024-07-26T06:56:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:26 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/3426812468,v1:172.21.15.163:6837/3426812468] up:reconnect 2024-07-26T06:56:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:26 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:56:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:26 smithi164 ceph-mon[36970]: pgmap v590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T06:56:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:26 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:56:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:26 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:56:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:26 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:56:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:26 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/148229526,v1:172.21.15.163:6835/148229526] up:reconnect 2024-07-26T06:56:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:26 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/692571500,v1:172.21.15.163:6839/692571500] up:reconnect 2024-07-26T06:56:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:26 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/3426812468,v1:172.21.15.163:6837/3426812468] up:reconnect 2024-07-26T06:56:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:26 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:56:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:26 smithi163 ceph-mon[36873]: pgmap v590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T06:56:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:26 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:56:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:26 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:56:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:26 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:56:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:26 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/148229526,v1:172.21.15.163:6835/148229526] up:reconnect 2024-07-26T06:56:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:26 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/692571500,v1:172.21.15.163:6839/692571500] up:reconnect 2024-07-26T06:56:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:26 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/3426812468,v1:172.21.15.163:6837/3426812468] up:reconnect 2024-07-26T06:56:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:26 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:56:27.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:27 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/3426812468,v1:172.21.15.163:6837/3426812468] up:rejoin 2024-07-26T06:56:27.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:27 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/692571500,v1:172.21.15.163:6839/692571500] up:rejoin 2024-07-26T06:56:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:27 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/148229526,v1:172.21.15.163:6835/148229526] up:rejoin 2024-07-26T06:56:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:27 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:56:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:27 smithi042 ceph-mon[32775]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-07-26T06:56:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:27 smithi042 ceph-mon[32775]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-07-26T06:56:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:27 smithi042 ceph-mon[32775]: daemon mds.b is now active in filesystem cephfs as rank 3 2024-07-26T06:56:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:27 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/3426812468,v1:172.21.15.163:6837/3426812468] up:rejoin 2024-07-26T06:56:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:27 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/692571500,v1:172.21.15.163:6839/692571500] up:rejoin 2024-07-26T06:56:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:27 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/148229526,v1:172.21.15.163:6835/148229526] up:rejoin 2024-07-26T06:56:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:27 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:56:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:27 smithi164 ceph-mon[36970]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-07-26T06:56:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:27 smithi164 ceph-mon[36970]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-07-26T06:56:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:27 smithi164 ceph-mon[36970]: daemon mds.b is now active in filesystem cephfs as rank 3 2024-07-26T06:56:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/3426812468,v1:172.21.15.163:6837/3426812468] up:rejoin 2024-07-26T06:56:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/692571500,v1:172.21.15.163:6839/692571500] up:rejoin 2024-07-26T06:56:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/148229526,v1:172.21.15.163:6835/148229526] up:rejoin 2024-07-26T06:56:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:56:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-mon[36873]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-07-26T06:56:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-mon[36873]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-07-26T06:56:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-mon[36873]: daemon mds.b is now active in filesystem cephfs as rank 3 2024-07-26T06:56:28.008 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:56:28.008 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: *** Caught signal (Aborted) ** 2024-07-26T06:56:28.009 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: in thread 7f051545e640 thread_name:ms_dispatch 2024-07-26T06:56:28.009 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:28.009 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f051a8356f0] 2024-07-26T06:56:28.009 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f051a88294c] 2024-07-26T06:56:28.009 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 3: raise() 2024-07-26T06:56:28.010 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 4: abort() 2024-07-26T06:56:28.010 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 5: /lib64/libc.so.6(+0x2871b) [0x7f051a81f71b] 2024-07-26T06:56:28.010 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 6: /lib64/libc.so.6(+0x37386) [0x7f051a82e386] 2024-07-26T06:56:28.010 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5619da3d96b9] 2024-07-26T06:56:28.010 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5619da59780c] 2024-07-26T06:56:28.010 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5619da4d72ca] 2024-07-26T06:56:28.011 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5619da4a7753] 2024-07-26T06:56:28.011 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5619da4a8a1b] 2024-07-26T06:56:28.011 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5619da48c752] 2024-07-26T06:56:28.011 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5619da3daa56] 2024-07-26T06:56:28.011 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5619da3de637] 2024-07-26T06:56:28.011 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5619da3def5e] 2024-07-26T06:56:28.012 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5619da3c203a] 2024-07-26T06:56:28.012 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f051b102407] 2024-07-26T06:56:28.012 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 18: (DispatchQueue::entry()+0x60f) [0x7f051b102cf9] 2024-07-26T06:56:28.012 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f051b1df5a1] 2024-07-26T06:56:28.013 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:56:28.013 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: *** Caught signal (Aborted) ** 2024-07-26T06:56:28.013 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: in thread 7f2fab2d0640 thread_name:ms_dispatch 2024-07-26T06:56:28.013 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:28.013 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2fb06a76f0] 2024-07-26T06:56:28.014 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2fb06f494c] 2024-07-26T06:56:28.014 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 3: raise() 2024-07-26T06:56:28.014 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 4: abort() 2024-07-26T06:56:28.014 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2fb069171b] 2024-07-26T06:56:28.014 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 6: /lib64/libc.so.6(+0x37386) [0x7f2fb06a0386] 2024-07-26T06:56:28.015 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cc114e66b9] 2024-07-26T06:56:28.015 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cc116a480c] 2024-07-26T06:56:28.015 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55cc115e42ca] 2024-07-26T06:56:28.015 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55cc115b4753] 2024-07-26T06:56:28.015 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cc115b5a1b] 2024-07-26T06:56:28.015 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cc11599752] 2024-07-26T06:56:28.016 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cc114e7a56] 2024-07-26T06:56:28.016 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cc114eb637] 2024-07-26T06:56:28.016 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cc114ebf5e] 2024-07-26T06:56:28.016 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cc114cf03a] 2024-07-26T06:56:28.016 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f2fb0f74407] 2024-07-26T06:56:28.016 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 18: (DispatchQueue::entry()+0x60f) [0x7f2fb0f74cf9] 2024-07-26T06:56:28.017 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2fb10515a1] 2024-07-26T06:56:28.017 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:56:28.017 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: *** Caught signal (Aborted) ** 2024-07-26T06:56:28.018 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: in thread 7f284bb69640 thread_name:ms_dispatch 2024-07-26T06:56:28.018 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:28.018 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2850f406f0] 2024-07-26T06:56:28.018 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2850f8d94c] 2024-07-26T06:56:28.018 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 3: raise() 2024-07-26T06:56:28.019 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 4: abort() 2024-07-26T06:56:28.019 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2850f2a71b] 2024-07-26T06:56:28.019 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 6: /lib64/libc.so.6(+0x37386) [0x7f2850f39386] 2024-07-26T06:56:28.019 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5614506316b9] 2024-07-26T06:56:28.019 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5614507ef80c] 2024-07-26T06:56:28.019 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56145072f2ca] 2024-07-26T06:56:28.019 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5614506ff753] 2024-07-26T06:56:28.020 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561450700a1b] 2024-07-26T06:56:28.020 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5614506e4752] 2024-07-26T06:56:28.020 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561450632a56] 2024-07-26T06:56:28.020 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561450636637] 2024-07-26T06:56:28.020 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561450636f5e] 2024-07-26T06:56:28.020 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56145061a03a] 2024-07-26T06:56:28.021 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f285180d407] 2024-07-26T06:56:28.021 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 18: (DispatchQueue::entry()+0x60f) [0x7f285180dcf9] 2024-07-26T06:56:28.021 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f28518ea5a1] 2024-07-26T06:56:28.021 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 20: (Thread::entry_wrapper()+0x43) [0x7f051af8ccdd] 2024-07-26T06:56:28.021 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 21: (Thread::_entry_func(void*)+0xd) [0x7f051af8ccf9] 2024-07-26T06:56:28.022 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 22: /lib64/libc.so.6(+0x89c02) [0x7f051a880c02] 2024-07-26T06:56:28.022 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f051a905c40] 2024-07-26T06:56:28.022 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 2024-07-26T06:56:27.627+0000 7f051545e640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:28.022 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: in thread 7f051545e640 thread_name:ms_dispatch 2024-07-26T06:56:28.022 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 2024-07-26T06:56:28.022 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:28.022 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f051a8356f0] 2024-07-26T06:56:28.022 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f051a88294c] 2024-07-26T06:56:28.022 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 3: raise() 2024-07-26T06:56:28.022 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 4: abort() 2024-07-26T06:56:28.022 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 5: /lib64/libc.so.6(+0x2871b) [0x7f051a81f71b] 2024-07-26T06:56:28.023 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 6: /lib64/libc.so.6(+0x37386) [0x7f051a82e386] 2024-07-26T06:56:28.023 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5619da3d96b9] 2024-07-26T06:56:28.023 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5619da59780c] 2024-07-26T06:56:28.023 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5619da4d72ca] 2024-07-26T06:56:28.023 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5619da4a7753] 2024-07-26T06:56:28.023 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5619da4a8a1b] 2024-07-26T06:56:28.023 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5619da48c752] 2024-07-26T06:56:28.023 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5619da3daa56] 2024-07-26T06:56:28.023 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5619da3de637] 2024-07-26T06:56:28.023 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5619da3def5e] 2024-07-26T06:56:28.023 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5619da3c203a] 2024-07-26T06:56:28.023 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f051b102407] 2024-07-26T06:56:28.024 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 18: (DispatchQueue::entry()+0x60f) [0x7f051b102cf9] 2024-07-26T06:56:28.024 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f051b1df5a1] 2024-07-26T06:56:28.024 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 20: (Thread::entry_wrapper()+0x43) [0x7f2fb0dfecdd] 2024-07-26T06:56:28.024 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2fb0dfecf9] 2024-07-26T06:56:28.024 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2fb06f2c02] 2024-07-26T06:56:28.024 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2fb0777c40] 2024-07-26T06:56:28.024 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 2024-07-26T06:56:27.624+0000 7f2fab2d0640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:28.024 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: in thread 7f2fab2d0640 thread_name:ms_dispatch 2024-07-26T06:56:28.025 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 2024-07-26T06:56:28.025 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:28.025 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2fb06a76f0] 2024-07-26T06:56:28.025 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2fb06f494c] 2024-07-26T06:56:28.025 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 3: raise() 2024-07-26T06:56:28.025 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 4: abort() 2024-07-26T06:56:28.025 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2fb069171b] 2024-07-26T06:56:28.025 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 6: /lib64/libc.so.6(+0x37386) [0x7f2fb06a0386] 2024-07-26T06:56:28.025 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cc114e66b9] 2024-07-26T06:56:28.026 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cc116a480c] 2024-07-26T06:56:28.026 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55cc115e42ca] 2024-07-26T06:56:28.026 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55cc115b4753] 2024-07-26T06:56:28.026 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cc115b5a1b] 2024-07-26T06:56:28.026 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cc11599752] 2024-07-26T06:56:28.026 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cc114e7a56] 2024-07-26T06:56:28.026 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cc114eb637] 2024-07-26T06:56:28.026 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cc114ebf5e] 2024-07-26T06:56:28.026 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cc114cf03a] 2024-07-26T06:56:28.026 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f2fb0f74407] 2024-07-26T06:56:28.026 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 18: (DispatchQueue::entry()+0x60f) [0x7f2fb0f74cf9] 2024-07-26T06:56:28.026 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2fb10515a1] 2024-07-26T06:56:28.027 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 20: (Thread::entry_wrapper()+0x43) [0x7f2851697cdd] 2024-07-26T06:56:28.027 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2851697cf9] 2024-07-26T06:56:28.027 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2850f8bc02] 2024-07-26T06:56:28.027 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2851010c40] 2024-07-26T06:56:28.027 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 2024-07-26T06:56:27.625+0000 7f284bb69640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:28.027 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: in thread 7f284bb69640 thread_name:ms_dispatch 2024-07-26T06:56:28.027 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 2024-07-26T06:56:28.028 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:28.028 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2850f406f0] 2024-07-26T06:56:28.028 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2850f8d94c] 2024-07-26T06:56:28.028 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 3: raise() 2024-07-26T06:56:28.028 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 4: abort() 2024-07-26T06:56:28.028 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2850f2a71b] 2024-07-26T06:56:28.028 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 6: /lib64/libc.so.6(+0x37386) [0x7f2850f39386] 2024-07-26T06:56:28.028 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5614506316b9] 2024-07-26T06:56:28.028 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5614507ef80c] 2024-07-26T06:56:28.028 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56145072f2ca] 2024-07-26T06:56:28.028 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5614506ff753] 2024-07-26T06:56:28.028 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561450700a1b] 2024-07-26T06:56:28.029 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5614506e4752] 2024-07-26T06:56:28.029 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561450632a56] 2024-07-26T06:56:28.029 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561450636637] 2024-07-26T06:56:28.029 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561450636f5e] 2024-07-26T06:56:28.029 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56145061a03a] 2024-07-26T06:56:28.029 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f285180d407] 2024-07-26T06:56:28.029 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 18: (DispatchQueue::entry()+0x60f) [0x7f285180dcf9] 2024-07-26T06:56:28.029 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f28518ea5a1] 2024-07-26T06:56:28.029 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 20: (Thread::entry_wrapper()+0x43) [0x7f051af8ccdd] 2024-07-26T06:56:28.029 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 21: (Thread::_entry_func(void*)+0xd) [0x7f051af8ccf9] 2024-07-26T06:56:28.030 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 22: /lib64/libc.so.6(+0x89c02) [0x7f051a880c02] 2024-07-26T06:56:28.030 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f051a905c40] 2024-07-26T06:56:28.030 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:28.030 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 2024-07-26T06:56:28.030 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 0> 2024-07-26T06:56:27.627+0000 7f051545e640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:28.030 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: in thread 7f051545e640 thread_name:ms_dispatch 2024-07-26T06:56:28.030 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 2024-07-26T06:56:28.030 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:28.030 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f051a8356f0] 2024-07-26T06:56:28.030 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f051a88294c] 2024-07-26T06:56:28.030 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 3: raise() 2024-07-26T06:56:28.030 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 4: abort() 2024-07-26T06:56:28.031 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 5: /lib64/libc.so.6(+0x2871b) [0x7f051a81f71b] 2024-07-26T06:56:28.031 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 6: /lib64/libc.so.6(+0x37386) [0x7f051a82e386] 2024-07-26T06:56:28.031 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5619da3d96b9] 2024-07-26T06:56:28.031 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5619da59780c] 2024-07-26T06:56:28.031 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5619da4d72ca] 2024-07-26T06:56:28.031 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5619da4a7753] 2024-07-26T06:56:28.031 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5619da4a8a1b] 2024-07-26T06:56:28.031 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5619da48c752] 2024-07-26T06:56:28.031 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5619da3daa56] 2024-07-26T06:56:28.031 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5619da3de637] 2024-07-26T06:56:28.031 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5619da3def5e] 2024-07-26T06:56:28.032 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5619da3c203a] 2024-07-26T06:56:28.032 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f051b102407] 2024-07-26T06:56:28.032 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 20: (Thread::entry_wrapper()+0x43) [0x7f2fb0dfecdd] 2024-07-26T06:56:28.032 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2fb0dfecf9] 2024-07-26T06:56:28.033 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2fb06f2c02] 2024-07-26T06:56:28.033 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2fb0777c40] 2024-07-26T06:56:28.033 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:28.033 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 2024-07-26T06:56:28.033 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 0> 2024-07-26T06:56:27.624+0000 7f2fab2d0640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:28.033 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: in thread 7f2fab2d0640 thread_name:ms_dispatch 2024-07-26T06:56:28.033 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 2024-07-26T06:56:28.033 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:28.033 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2fb06a76f0] 2024-07-26T06:56:28.033 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2fb06f494c] 2024-07-26T06:56:28.033 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 3: raise() 2024-07-26T06:56:28.033 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 4: abort() 2024-07-26T06:56:28.034 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2fb069171b] 2024-07-26T06:56:28.034 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 6: /lib64/libc.so.6(+0x37386) [0x7f2fb06a0386] 2024-07-26T06:56:28.034 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cc114e66b9] 2024-07-26T06:56:28.034 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cc116a480c] 2024-07-26T06:56:28.034 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55cc115e42ca] 2024-07-26T06:56:28.034 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55cc115b4753] 2024-07-26T06:56:28.034 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cc115b5a1b] 2024-07-26T06:56:28.034 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cc11599752] 2024-07-26T06:56:28.034 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cc114e7a56] 2024-07-26T06:56:28.034 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cc114eb637] 2024-07-26T06:56:28.035 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cc114ebf5e] 2024-07-26T06:56:28.035 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cc114cf03a] 2024-07-26T06:56:28.035 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f2fb0f74407] 2024-07-26T06:56:28.035 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 20: (Thread::entry_wrapper()+0x43) [0x7f2851697cdd] 2024-07-26T06:56:28.035 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2851697cf9] 2024-07-26T06:56:28.035 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2850f8bc02] 2024-07-26T06:56:28.035 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2851010c40] 2024-07-26T06:56:28.035 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:28.035 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 2024-07-26T06:56:28.035 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 0> 2024-07-26T06:56:27.625+0000 7f284bb69640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:28.036 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: in thread 7f284bb69640 thread_name:ms_dispatch 2024-07-26T06:56:28.036 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 2024-07-26T06:56:28.036 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:28.036 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2850f406f0] 2024-07-26T06:56:28.036 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2850f8d94c] 2024-07-26T06:56:28.036 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 3: raise() 2024-07-26T06:56:28.036 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 4: abort() 2024-07-26T06:56:28.036 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2850f2a71b] 2024-07-26T06:56:28.036 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 6: /lib64/libc.so.6(+0x37386) [0x7f2850f39386] 2024-07-26T06:56:28.036 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5614506316b9] 2024-07-26T06:56:28.036 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5614507ef80c] 2024-07-26T06:56:28.036 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56145072f2ca] 2024-07-26T06:56:28.037 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5614506ff753] 2024-07-26T06:56:28.037 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561450700a1b] 2024-07-26T06:56:28.037 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5614506e4752] 2024-07-26T06:56:28.037 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561450632a56] 2024-07-26T06:56:28.037 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561450636637] 2024-07-26T06:56:28.037 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561450636f5e] 2024-07-26T06:56:28.037 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56145061a03a] 2024-07-26T06:56:28.037 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f285180d407] 2024-07-26T06:56:28.037 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 18: (DispatchQueue::entry()+0x60f) [0x7f051b102cf9] 2024-07-26T06:56:28.037 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f051b1df5a1] 2024-07-26T06:56:28.037 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 20: (Thread::entry_wrapper()+0x43) [0x7f051af8ccdd] 2024-07-26T06:56:28.038 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 21: (Thread::_entry_func(void*)+0xd) [0x7f051af8ccf9] 2024-07-26T06:56:28.038 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 22: /lib64/libc.so.6(+0x89c02) [0x7f051a880c02] 2024-07-26T06:56:28.038 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f051a905c40] 2024-07-26T06:56:28.038 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:28.038 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 2024-07-26T06:56:28.038 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 0> 2024-07-26T06:56:27.627+0000 7f051545e640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:28.038 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: in thread 7f051545e640 thread_name:ms_dispatch 2024-07-26T06:56:28.038 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 2024-07-26T06:56:28.038 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:28.039 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f051a8356f0] 2024-07-26T06:56:28.039 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f051a88294c] 2024-07-26T06:56:28.039 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 3: raise() 2024-07-26T06:56:28.039 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 4: abort() 2024-07-26T06:56:28.040 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 5: /lib64/libc.so.6(+0x2871b) [0x7f051a81f71b] 2024-07-26T06:56:28.040 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 6: /lib64/libc.so.6(+0x37386) [0x7f051a82e386] 2024-07-26T06:56:28.040 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5619da3d96b9] 2024-07-26T06:56:28.040 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5619da59780c] 2024-07-26T06:56:28.040 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5619da4d72ca] 2024-07-26T06:56:28.040 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5619da4a7753] 2024-07-26T06:56:28.041 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5619da4a8a1b] 2024-07-26T06:56:28.041 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5619da48c752] 2024-07-26T06:56:28.041 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5619da3daa56] 2024-07-26T06:56:28.041 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5619da3de637] 2024-07-26T06:56:28.041 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5619da3def5e] 2024-07-26T06:56:28.042 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 18: (DispatchQueue::entry()+0x60f) [0x7f2fb0f74cf9] 2024-07-26T06:56:28.042 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2fb10515a1] 2024-07-26T06:56:28.042 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 20: (Thread::entry_wrapper()+0x43) [0x7f2fb0dfecdd] 2024-07-26T06:56:28.042 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2fb0dfecf9] 2024-07-26T06:56:28.042 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2fb06f2c02] 2024-07-26T06:56:28.043 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2fb0777c40] 2024-07-26T06:56:28.043 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:28.043 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 2024-07-26T06:56:28.043 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 0> 2024-07-26T06:56:27.624+0000 7f2fab2d0640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:28.043 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: in thread 7f2fab2d0640 thread_name:ms_dispatch 2024-07-26T06:56:28.043 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 2024-07-26T06:56:28.044 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:28.044 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2fb06a76f0] 2024-07-26T06:56:28.044 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2fb06f494c] 2024-07-26T06:56:28.044 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 3: raise() 2024-07-26T06:56:28.044 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 4: abort() 2024-07-26T06:56:28.044 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2fb069171b] 2024-07-26T06:56:28.045 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 6: /lib64/libc.so.6(+0x37386) [0x7f2fb06a0386] 2024-07-26T06:56:28.045 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cc114e66b9] 2024-07-26T06:56:28.045 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cc116a480c] 2024-07-26T06:56:28.045 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55cc115e42ca] 2024-07-26T06:56:28.045 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55cc115b4753] 2024-07-26T06:56:28.045 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cc115b5a1b] 2024-07-26T06:56:28.046 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cc11599752] 2024-07-26T06:56:28.046 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cc114e7a56] 2024-07-26T06:56:28.046 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cc114eb637] 2024-07-26T06:56:28.046 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cc114ebf5e] 2024-07-26T06:56:28.047 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 18: (DispatchQueue::entry()+0x60f) [0x7f285180dcf9] 2024-07-26T06:56:28.047 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f28518ea5a1] 2024-07-26T06:56:28.047 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 20: (Thread::entry_wrapper()+0x43) [0x7f2851697cdd] 2024-07-26T06:56:28.047 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2851697cf9] 2024-07-26T06:56:28.047 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2850f8bc02] 2024-07-26T06:56:28.047 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2851010c40] 2024-07-26T06:56:28.048 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:28.048 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 2024-07-26T06:56:28.048 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 0> 2024-07-26T06:56:27.625+0000 7f284bb69640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:28.048 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: in thread 7f284bb69640 thread_name:ms_dispatch 2024-07-26T06:56:28.048 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 2024-07-26T06:56:28.048 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:28.048 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2850f406f0] 2024-07-26T06:56:28.049 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2850f8d94c] 2024-07-26T06:56:28.049 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 3: raise() 2024-07-26T06:56:28.049 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 4: abort() 2024-07-26T06:56:28.049 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2850f2a71b] 2024-07-26T06:56:28.049 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 6: /lib64/libc.so.6(+0x37386) [0x7f2850f39386] 2024-07-26T06:56:28.049 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5614506316b9] 2024-07-26T06:56:28.050 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5614507ef80c] 2024-07-26T06:56:28.050 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56145072f2ca] 2024-07-26T06:56:28.050 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5614506ff753] 2024-07-26T06:56:28.050 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561450700a1b] 2024-07-26T06:56:28.050 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5614506e4752] 2024-07-26T06:56:28.050 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561450632a56] 2024-07-26T06:56:28.050 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561450636637] 2024-07-26T06:56:28.050 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561450636f5e] 2024-07-26T06:56:28.051 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cc114cf03a] 2024-07-26T06:56:28.051 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f2fb0f74407] 2024-07-26T06:56:28.051 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 18: (DispatchQueue::entry()+0x60f) [0x7f2fb0f74cf9] 2024-07-26T06:56:28.051 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2fb10515a1] 2024-07-26T06:56:28.051 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 20: (Thread::entry_wrapper()+0x43) [0x7f2fb0dfecdd] 2024-07-26T06:56:28.051 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2fb0dfecf9] 2024-07-26T06:56:28.051 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2fb06f2c02] 2024-07-26T06:56:28.052 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2fb0777c40] 2024-07-26T06:56:28.052 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:28.052 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70205]: 2024-07-26T06:56:28.052 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5619da3c203a] 2024-07-26T06:56:28.052 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f051b102407] 2024-07-26T06:56:28.052 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 18: (DispatchQueue::entry()+0x60f) [0x7f051b102cf9] 2024-07-26T06:56:28.053 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f051b1df5a1] 2024-07-26T06:56:28.053 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 20: (Thread::entry_wrapper()+0x43) [0x7f051af8ccdd] 2024-07-26T06:56:28.053 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 21: (Thread::_entry_func(void*)+0xd) [0x7f051af8ccf9] 2024-07-26T06:56:28.053 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 22: /lib64/libc.so.6(+0x89c02) [0x7f051a880c02] 2024-07-26T06:56:28.053 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f051a905c40] 2024-07-26T06:56:28.053 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:28.053 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70159]: 2024-07-26T06:56:28.053 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:27 smithi163 podman[70404]: 2024-07-26 06:56:27.689958599 +0000 UTC m=+0.020693610 container died 407f493356c218e61d85a124f2469e8567115ab3f538adbe23ea5e60c9801379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T06:56:28.054 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56145061a03a] 2024-07-26T06:56:28.054 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f285180d407] 2024-07-26T06:56:28.054 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 18: (DispatchQueue::entry()+0x60f) [0x7f285180dcf9] 2024-07-26T06:56:28.054 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f28518ea5a1] 2024-07-26T06:56:28.054 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 20: (Thread::entry_wrapper()+0x43) [0x7f2851697cdd] 2024-07-26T06:56:28.055 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2851697cf9] 2024-07-26T06:56:28.055 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2850f8bc02] 2024-07-26T06:56:28.055 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2851010c40] 2024-07-26T06:56:28.055 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:28.055 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:27 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70153]: 2024-07-26T06:56:28.726 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:28 smithi163 podman[70405]: 2024-07-26 06:56:28.422444757 +0000 UTC m=+0.750806111 container died 5c257bedce62e39379a6a9f9148fdf7e6433e7bb16259e2676ad6e77b52538ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T06:56:28.727 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:28 smithi163 podman[70404]: 2024-07-26 06:56:28.397615156 +0000 UTC m=+0.728350198 container remove 407f493356c218e61d85a124f2469e8567115ab3f538adbe23ea5e60c9801379 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T06:56:28.727 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:28 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:56:28.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:28 smithi163 ceph-mon[36873]: pgmap v591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T06:56:28.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:28 smithi163 ceph-mon[36873]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:56:28.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:28 smithi163 ceph-mon[36873]: Cluster is now healthy 2024-07-26T06:56:28.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:28 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/692571500,v1:172.21.15.163:6839/692571500] up:active 2024-07-26T06:56:28.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:28 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/148229526,v1:172.21.15.163:6835/148229526] up:active 2024-07-26T06:56:28.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:28 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/3426812468,v1:172.21.15.163:6837/3426812468] up:active 2024-07-26T06:56:28.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:28 smithi163 ceph-mon[36873]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:56:28.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:28 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:56:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:28 smithi042 ceph-mon[32775]: pgmap v591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T06:56:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:28 smithi042 ceph-mon[32775]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:56:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:28 smithi042 ceph-mon[32775]: Cluster is now healthy 2024-07-26T06:56:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:28 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/692571500,v1:172.21.15.163:6839/692571500] up:active 2024-07-26T06:56:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:28 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/148229526,v1:172.21.15.163:6835/148229526] up:active 2024-07-26T06:56:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:28 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/3426812468,v1:172.21.15.163:6837/3426812468] up:active 2024-07-26T06:56:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:28 smithi042 ceph-mon[32775]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:56:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:28 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:56:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:28 smithi164 ceph-mon[36970]: pgmap v591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T06:56:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:28 smithi164 ceph-mon[36970]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:56:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:28 smithi164 ceph-mon[36970]: Cluster is now healthy 2024-07-26T06:56:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:28 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/692571500,v1:172.21.15.163:6839/692571500] up:active 2024-07-26T06:56:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:28 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/148229526,v1:172.21.15.163:6835/148229526] up:active 2024-07-26T06:56:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:28 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/3426812468,v1:172.21.15.163:6837/3426812468] up:active 2024-07-26T06:56:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:28 smithi164 ceph-mon[36970]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:56:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:28 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:56:29.382 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:29 smithi163 podman[70411]: 2024-07-26 06:56:29.058228301 +0000 UTC m=+1.381613735 container died 85bdfb5f68dd6d00023df89da5d1a8290ae3a708bfa5e599060fcda834992c9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS) 2024-07-26T06:56:29.382 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:29 smithi163 podman[70405]: 2024-07-26 06:56:29.044226583 +0000 UTC m=+1.372587974 container remove 5c257bedce62e39379a6a9f9148fdf7e6433e7bb16259e2676ad6e77b52538ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_CLEAN=True) 2024-07-26T06:56:29.382 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:29 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:56:30.097 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:29 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T06:56:30.097 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:29 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Consumed 1.099s CPU time. 2024-07-26T06:56:30.097 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:29 smithi163 podman[70411]: 2024-07-26 06:56:29.824603335 +0000 UTC m=+2.147988761 container remove 85bdfb5f68dd6d00023df89da5d1a8290ae3a708bfa5e599060fcda834992c9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0) 2024-07-26T06:56:30.097 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:29 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:56:30.507 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:30 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T06:56:30.507 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:30 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Consumed 1.073s CPU time. 2024-07-26T06:56:30.507 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:30 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-26T06:56:30.507 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:30 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Consumed 1.006s CPU time. 2024-07-26T06:56:30.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:30 smithi042 ceph-mon[32775]: pgmap v592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 1.8 KiB/s wr, 15 op/s 2024-07-26T06:56:30.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:30 smithi164 ceph-mon[36970]: pgmap v592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 1.8 KiB/s wr, 15 op/s 2024-07-26T06:56:31.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:30 smithi163 ceph-mon[36873]: pgmap v592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 1.8 KiB/s wr, 15 op/s 2024-07-26T06:56:32.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:32 smithi042 ceph-mon[32775]: pgmap v593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-07-26T06:56:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:32 smithi164 ceph-mon[36970]: pgmap v593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-07-26T06:56:33.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:32 smithi163 ceph-mon[36873]: pgmap v593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-07-26T06:56:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:34 smithi042 ceph-mon[32775]: pgmap v594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-07-26T06:56:34.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:34 smithi164 ceph-mon[36970]: pgmap v594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-07-26T06:56:35.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:34 smithi163 ceph-mon[36873]: pgmap v594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-07-26T06:56:36.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:36 smithi042 ceph-mon[32775]: pgmap v595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-07-26T06:56:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:36 smithi164 ceph-mon[36970]: pgmap v595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-07-26T06:56:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:36 smithi163 ceph-mon[36873]: pgmap v595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-07-26T06:56:38.289 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:38.288+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:56:38.289 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:38.288+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:56:38.289 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:38.288+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:56:38.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:38.685+0000 7faa2dcb9640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 msgr2=0x561ca20d0cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:56:38.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:38.685+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:56:38.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:38.685+0000 7faa117fa640 1 client.16095 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:56:38.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:38 smithi042 ceph-mon[32775]: pgmap v596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T06:56:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:38 smithi164 ceph-mon[36970]: pgmap v596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T06:56:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:38 smithi163 ceph-mon[36873]: pgmap v596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T06:56:40.258 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:40 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 2. 2024-07-26T06:56:40.258 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:40 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 2. 2024-07-26T06:56:40.258 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:40 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 2. 2024-07-26T06:56:40.258 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:40 smithi163 systemd[1]: Stopped Ceph mds.b for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:40.258 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:40 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Consumed 1.099s CPU time. 2024-07-26T06:56:40.259 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:40 smithi163 systemd[1]: Starting Ceph mds.b for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:56:40.259 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:40 smithi163 podman[70598]: 2024-07-26 06:56:40.141499504 +0000 UTC m=+0.017591670 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:56:40.529 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:40 smithi163 ceph-mon[36873]: pgmap v597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-07-26T06:56:40.529 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:40 smithi163 podman[70598]: 2024-07-26 06:56:40.258829881 +0000 UTC m=+0.134922050 container create ba88b2e97fb9f381bec2797db9c8f98408a62c2c7333486a72e8f0786cedc41b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-07-26T06:56:40.529 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:40 smithi163 systemd[1]: Stopped Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:40.530 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:40 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Consumed 1.006s CPU time. 2024-07-26T06:56:40.530 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:40 smithi163 systemd[1]: Starting Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:56:40.530 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:40 smithi163 systemd[1]: Stopped Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:40.530 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:40 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Consumed 1.073s CPU time. 2024-07-26T06:56:40.530 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:40 smithi163 systemd[1]: Starting Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:56:40.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:40 smithi042 ceph-mon[32775]: pgmap v597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-07-26T06:56:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:40 smithi164 ceph-mon[36970]: pgmap v597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-07-26T06:56:40.801 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:40 smithi163 podman[70598]: 2024-07-26 06:56:40.529304942 +0000 UTC m=+0.405397108 container init ba88b2e97fb9f381bec2797db9c8f98408a62c2c7333486a72e8f0786cedc41b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-07-26T06:56:40.801 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:40 smithi163 podman[70598]: 2024-07-26 06:56:40.53227504 +0000 UTC m=+0.408367210 container start ba88b2e97fb9f381bec2797db9c8f98408a62c2c7333486a72e8f0786cedc41b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-07-26T06:56:40.802 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:40 smithi163 ceph-mds[70679]: -- 172.21.15.163:0/3673041914 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56322ce7d860 con 0x56323101f180 2024-07-26T06:56:40.802 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:40 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: starting mds.b at 2024-07-26T06:56:40.802 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:40 smithi163 bash[70598]: ba88b2e97fb9f381bec2797db9c8f98408a62c2c7333486a72e8f0786cedc41b 2024-07-26T06:56:40.802 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:56:40 smithi163 systemd[1]: Started Ceph mds.b for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:41.109 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:40 smithi163 podman[70739]: 2024-07-26 06:56:40.71533669 +0000 UTC m=+0.019079472 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:56:41.109 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:40 smithi163 podman[70728]: 2024-07-26 06:56:40.704141268 +0000 UTC m=+0.018491444 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:56:41.109 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:40 smithi163 podman[70728]: 2024-07-26 06:56:40.848702853 +0000 UTC m=+0.163053014 container create c511a9857c5805705d77cd227cd8b73652aea95691724568135033e092af9b8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_CLEAN=True) 2024-07-26T06:56:41.370 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:41 smithi163 podman[70739]: 2024-07-26 06:56:41.109166704 +0000 UTC m=+0.412909477 container create e30626c9e226c7d2232311324bfa47293dcaaab4427c823e2765455fd1cb3f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True) 2024-07-26T06:56:41.371 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:41 smithi163 ceph-mon[36873]: daemon mds.b restarted 2024-07-26T06:56:41.371 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:41 smithi163 ceph-mon[36873]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:56:41.371 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:41 smithi163 ceph-mon[36873]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T06:56:41.371 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:41 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/2760283174,v1:172.21.15.163:6835/2760283174] up:boot 2024-07-26T06:56:41.371 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:41 smithi163 ceph-mon[36873]: Standby daemon mds.a assigned to filesystem cephfs as rank 3 2024-07-26T06:56:41.371 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:41 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:56:41.371 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T06:56:41.371 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:41 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:56:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:41 smithi042 ceph-mon[32775]: daemon mds.b restarted 2024-07-26T06:56:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:41 smithi042 ceph-mon[32775]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:56:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:41 smithi042 ceph-mon[32775]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T06:56:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:41 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/2760283174,v1:172.21.15.163:6835/2760283174] up:boot 2024-07-26T06:56:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:41 smithi042 ceph-mon[32775]: Standby daemon mds.a assigned to filesystem cephfs as rank 3 2024-07-26T06:56:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:41 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:56:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T06:56:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:41 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:56:41.672 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:41 smithi163 podman[70728]: 2024-07-26 06:56:41.370762175 +0000 UTC m=+0.685112337 container init c511a9857c5805705d77cd227cd8b73652aea95691724568135033e092af9b8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T06:56:41.672 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:41 smithi163 podman[70728]: 2024-07-26 06:56:41.373439854 +0000 UTC m=+0.687790024 container start c511a9857c5805705d77cd227cd8b73652aea95691724568135033e092af9b8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-07-26T06:56:41.672 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:41 smithi163 ceph-mds[70770]: -- 172.21.15.163:0/1096330083 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557e2ca29860 con 0x557e30bcba80 2024-07-26T06:56:41.672 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:41 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: starting mds.e at 2024-07-26T06:56:41.672 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:41 smithi163 bash[70728]: c511a9857c5805705d77cd227cd8b73652aea95691724568135033e092af9b8a 2024-07-26T06:56:41.672 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:56:41 smithi163 systemd[1]: Started Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:41 smithi164 ceph-mon[36970]: daemon mds.b restarted 2024-07-26T06:56:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:41 smithi164 ceph-mon[36970]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:56:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:41 smithi164 ceph-mon[36970]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T06:56:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:41 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/2760283174,v1:172.21.15.163:6835/2760283174] up:boot 2024-07-26T06:56:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:41 smithi164 ceph-mon[36970]: Standby daemon mds.a assigned to filesystem cephfs as rank 3 2024-07-26T06:56:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:41 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:56:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T06:56:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:41 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:56:42.007 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:41 smithi163 podman[70739]: 2024-07-26 06:56:41.672277894 +0000 UTC m=+0.976020672 container init e30626c9e226c7d2232311324bfa47293dcaaab4427c823e2765455fd1cb3f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0) 2024-07-26T06:56:42.007 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:41 smithi163 podman[70739]: 2024-07-26 06:56:41.675085273 +0000 UTC m=+0.978828050 container start e30626c9e226c7d2232311324bfa47293dcaaab4427c823e2765455fd1cb3f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T06:56:42.007 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:41 smithi163 ceph-mds[70791]: -- 172.21.15.163:0/2180669113 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5643d83f3860 con 0x5643dc595a80 2024-07-26T06:56:42.007 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:41 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: starting mds.h at 2024-07-26T06:56:42.007 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:41 smithi163 bash[70739]: e30626c9e226c7d2232311324bfa47293dcaaab4427c823e2765455fd1cb3f19 2024-07-26T06:56:42.007 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:56:41 smithi163 systemd[1]: Started Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:42 smithi042 ceph-mon[32775]: daemon mds.e restarted 2024-07-26T06:56:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:42 smithi042 ceph-mon[32775]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T06:56:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:42 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/1032688799,v1:172.21.15.42:6841/1032688799] up:resolve 2024-07-26T06:56:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:42 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/2332407499,v1:172.21.15.163:6837/2332407499] up:boot 2024-07-26T06:56:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:42 smithi042 ceph-mon[32775]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-07-26T06:56:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:42 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:active,3=a=up:resolve} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:56:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:56:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:42 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:56:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:42 smithi042 ceph-mon[32775]: pgmap v600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-07-26T06:56:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:42 smithi042 ceph-mon[32775]: daemon mds.h restarted 2024-07-26T06:56:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:42 smithi042 ceph-mon[32775]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T06:56:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:42 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/2567296059,v1:172.21.15.163:6839/2567296059] up:boot 2024-07-26T06:56:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:42 smithi042 ceph-mon[32775]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-07-26T06:56:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:42 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:resolve,4=g=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:56:42.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:56:42.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:42 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:56:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:42 smithi164 ceph-mon[36970]: daemon mds.e restarted 2024-07-26T06:56:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:42 smithi164 ceph-mon[36970]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T06:56:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:42 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/1032688799,v1:172.21.15.42:6841/1032688799] up:resolve 2024-07-26T06:56:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:42 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/2332407499,v1:172.21.15.163:6837/2332407499] up:boot 2024-07-26T06:56:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:42 smithi164 ceph-mon[36970]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-07-26T06:56:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:42 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:active,3=a=up:resolve} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:56:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:56:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:42 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:56:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:42 smithi164 ceph-mon[36970]: pgmap v600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-07-26T06:56:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:42 smithi164 ceph-mon[36970]: daemon mds.h restarted 2024-07-26T06:56:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:42 smithi164 ceph-mon[36970]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T06:56:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:42 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/2567296059,v1:172.21.15.163:6839/2567296059] up:boot 2024-07-26T06:56:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:42 smithi164 ceph-mon[36970]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-07-26T06:56:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:42 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:resolve,4=g=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:56:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:56:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:42 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:56:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:42 smithi163 ceph-mon[36873]: daemon mds.e restarted 2024-07-26T06:56:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:42 smithi163 ceph-mon[36873]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T06:56:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:42 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/1032688799,v1:172.21.15.42:6841/1032688799] up:resolve 2024-07-26T06:56:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:42 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/2332407499,v1:172.21.15.163:6837/2332407499] up:boot 2024-07-26T06:56:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:42 smithi163 ceph-mon[36873]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-07-26T06:56:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:42 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:active,3=a=up:resolve} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:56:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:56:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:42 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:56:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:42 smithi163 ceph-mon[36873]: pgmap v600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-07-26T06:56:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:42 smithi163 ceph-mon[36873]: daemon mds.h restarted 2024-07-26T06:56:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:42 smithi163 ceph-mon[36873]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T06:56:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:42 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/2567296059,v1:172.21.15.163:6839/2567296059] up:boot 2024-07-26T06:56:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:42 smithi163 ceph-mon[36873]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-07-26T06:56:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:42 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:resolve,4=g=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:56:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:56:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:42 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:56:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:43 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/2610757455,v1:172.21.15.164:6833/2610757455] up:resolve 2024-07-26T06:56:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:43 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3874449504,v1:172.21.15.42:6839/3874449504] up:resolve 2024-07-26T06:56:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:43 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:56:44.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:43 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/2610757455,v1:172.21.15.164:6833/2610757455] up:resolve 2024-07-26T06:56:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:43 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3874449504,v1:172.21.15.42:6839/3874449504] up:resolve 2024-07-26T06:56:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:43 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:56:44.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:43 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/2610757455,v1:172.21.15.164:6833/2610757455] up:resolve 2024-07-26T06:56:44.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:43 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3874449504,v1:172.21.15.42:6839/3874449504] up:resolve 2024-07-26T06:56:44.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:43 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:56:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:44 smithi163 ceph-mon[36873]: pgmap v602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 11 op/s 2024-07-26T06:56:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:44 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:56:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:44 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:56:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:44 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:56:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:44 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/1032688799,v1:172.21.15.42:6841/1032688799] up:reconnect 2024-07-26T06:56:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:44 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/2610757455,v1:172.21.15.164:6833/2610757455] up:reconnect 2024-07-26T06:56:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:44 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3874449504,v1:172.21.15.42:6839/3874449504] up:reconnect 2024-07-26T06:56:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:44 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:56:45.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:44 smithi042 ceph-mon[32775]: pgmap v602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 11 op/s 2024-07-26T06:56:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:44 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:56:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:44 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:56:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:44 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:56:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:44 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/1032688799,v1:172.21.15.42:6841/1032688799] up:reconnect 2024-07-26T06:56:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:44 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/2610757455,v1:172.21.15.164:6833/2610757455] up:reconnect 2024-07-26T06:56:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:44 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3874449504,v1:172.21.15.42:6839/3874449504] up:reconnect 2024-07-26T06:56:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:44 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:56:45.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:44 smithi164 ceph-mon[36970]: pgmap v602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 11 op/s 2024-07-26T06:56:45.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:44 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:56:45.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:44 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:56:45.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:44 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:56:45.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:44 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/1032688799,v1:172.21.15.42:6841/1032688799] up:reconnect 2024-07-26T06:56:45.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:44 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/2610757455,v1:172.21.15.164:6833/2610757455] up:reconnect 2024-07-26T06:56:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:44 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3874449504,v1:172.21.15.42:6839/3874449504] up:reconnect 2024-07-26T06:56:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:44 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:56:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:45 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3874449504,v1:172.21.15.42:6839/3874449504] up:rejoin 2024-07-26T06:56:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:45 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/1032688799,v1:172.21.15.42:6841/1032688799] up:rejoin 2024-07-26T06:56:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:45 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/2610757455,v1:172.21.15.164:6833/2610757455] up:rejoin 2024-07-26T06:56:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:45 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:56:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:45 smithi163 ceph-mon[36873]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:56:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:45 smithi163 ceph-mon[36873]: daemon mds.a is now active in filesystem cephfs as rank 3 2024-07-26T06:56:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:45 smithi163 ceph-mon[36873]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-07-26T06:56:46.063 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3874449504,v1:172.21.15.42:6839/3874449504] up:rejoin 2024-07-26T06:56:46.063 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/1032688799,v1:172.21.15.42:6841/1032688799] up:rejoin 2024-07-26T06:56:46.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/2610757455,v1:172.21.15.164:6833/2610757455] up:rejoin 2024-07-26T06:56:46.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:56:46.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-mon[32775]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:56:46.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-mon[32775]: daemon mds.a is now active in filesystem cephfs as rank 3 2024-07-26T06:56:46.064 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-mon[32775]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-07-26T06:56:46.064 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:56:46.065 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: *** Caught signal (Aborted) ** 2024-07-26T06:56:46.065 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: in thread 7f4d12f80640 thread_name:ms_dispatch 2024-07-26T06:56:46.065 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:46.065 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4d183576f0] 2024-07-26T06:56:46.065 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4d183a494c] 2024-07-26T06:56:46.065 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 3: raise() 2024-07-26T06:56:46.065 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 4: abort() 2024-07-26T06:56:46.065 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4d1834171b] 2024-07-26T06:56:46.065 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 6: /lib64/libc.so.6(+0x37386) [0x7f4d18350386] 2024-07-26T06:56:46.065 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55fee3a9e6b9] 2024-07-26T06:56:46.065 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55fee3c5c80c] 2024-07-26T06:56:46.066 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55fee3b9c2ca] 2024-07-26T06:56:46.066 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55fee3b6c753] 2024-07-26T06:56:46.066 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55fee3b6da1b] 2024-07-26T06:56:46.066 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55fee3b51752] 2024-07-26T06:56:46.066 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55fee3a9fa56] 2024-07-26T06:56:46.066 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55fee3aa3637] 2024-07-26T06:56:46.066 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55fee3aa3f5e] 2024-07-26T06:56:46.066 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55fee3a8703a] 2024-07-26T06:56:46.066 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4d18c24407] 2024-07-26T06:56:46.066 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 18: (DispatchQueue::entry()+0x60f) [0x7f4d18c24cf9] 2024-07-26T06:56:46.066 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4d18d015a1] 2024-07-26T06:56:46.067 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:56:46.067 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: *** Caught signal (Aborted) ** 2024-07-26T06:56:46.067 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: in thread 7f6e4feac640 thread_name:ms_dispatch 2024-07-26T06:56:46.067 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:46.067 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6e552836f0] 2024-07-26T06:56:46.067 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6e552d094c] 2024-07-26T06:56:46.067 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 3: raise() 2024-07-26T06:56:46.067 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 4: abort() 2024-07-26T06:56:46.067 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6e5526d71b] 2024-07-26T06:56:46.067 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 6: /lib64/libc.so.6(+0x37386) [0x7f6e5527c386] 2024-07-26T06:56:46.067 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5557c0ec86b9] 2024-07-26T06:56:46.068 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5557c108680c] 2024-07-26T06:56:46.068 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5557c0fc62ca] 2024-07-26T06:56:46.068 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5557c0f96753] 2024-07-26T06:56:46.068 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5557c0f97a1b] 2024-07-26T06:56:46.068 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5557c0f7b752] 2024-07-26T06:56:46.068 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5557c0ec9a56] 2024-07-26T06:56:46.068 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5557c0ecd637] 2024-07-26T06:56:46.068 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5557c0ecdf5e] 2024-07-26T06:56:46.068 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5557c0eb103a] 2024-07-26T06:56:46.068 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6e55b50407] 2024-07-26T06:56:46.068 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 18: (DispatchQueue::entry()+0x60f) [0x7f6e55b50cf9] 2024-07-26T06:56:46.068 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6e55c2d5a1] 2024-07-26T06:56:46.069 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 20: (Thread::entry_wrapper()+0x43) [0x7f4d18aaecdd] 2024-07-26T06:56:46.069 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4d18aaecf9] 2024-07-26T06:56:46.069 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4d183a2c02] 2024-07-26T06:56:46.069 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4d18427c40] 2024-07-26T06:56:46.069 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 2024-07-26T06:56:45.756+0000 7f4d12f80640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:46.069 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: in thread 7f4d12f80640 thread_name:ms_dispatch 2024-07-26T06:56:46.069 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 2024-07-26T06:56:46.069 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:46.069 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4d183576f0] 2024-07-26T06:56:46.069 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4d183a494c] 2024-07-26T06:56:46.069 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 3: raise() 2024-07-26T06:56:46.070 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 4: abort() 2024-07-26T06:56:46.070 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4d1834171b] 2024-07-26T06:56:46.070 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 6: /lib64/libc.so.6(+0x37386) [0x7f4d18350386] 2024-07-26T06:56:46.070 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55fee3a9e6b9] 2024-07-26T06:56:46.070 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55fee3c5c80c] 2024-07-26T06:56:46.070 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55fee3b9c2ca] 2024-07-26T06:56:46.070 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55fee3b6c753] 2024-07-26T06:56:46.070 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55fee3b6da1b] 2024-07-26T06:56:46.070 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55fee3b51752] 2024-07-26T06:56:46.070 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55fee3a9fa56] 2024-07-26T06:56:46.070 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55fee3aa3637] 2024-07-26T06:56:46.070 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55fee3aa3f5e] 2024-07-26T06:56:46.071 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55fee3a8703a] 2024-07-26T06:56:46.071 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4d18c24407] 2024-07-26T06:56:46.071 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 18: (DispatchQueue::entry()+0x60f) [0x7f4d18c24cf9] 2024-07-26T06:56:46.071 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4d18d015a1] 2024-07-26T06:56:46.071 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 20: (Thread::entry_wrapper()+0x43) [0x7f6e559dacdd] 2024-07-26T06:56:46.071 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6e559dacf9] 2024-07-26T06:56:46.071 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6e552cec02] 2024-07-26T06:56:46.071 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6e55353c40] 2024-07-26T06:56:46.071 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 2024-07-26T06:56:45.757+0000 7f6e4feac640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:46.072 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: in thread 7f6e4feac640 thread_name:ms_dispatch 2024-07-26T06:56:46.072 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 2024-07-26T06:56:46.072 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:46.072 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6e552836f0] 2024-07-26T06:56:46.072 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6e552d094c] 2024-07-26T06:56:46.072 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 3: raise() 2024-07-26T06:56:46.072 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 4: abort() 2024-07-26T06:56:46.072 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6e5526d71b] 2024-07-26T06:56:46.072 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 6: /lib64/libc.so.6(+0x37386) [0x7f6e5527c386] 2024-07-26T06:56:46.072 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5557c0ec86b9] 2024-07-26T06:56:46.072 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5557c108680c] 2024-07-26T06:56:46.072 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5557c0fc62ca] 2024-07-26T06:56:46.072 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5557c0f96753] 2024-07-26T06:56:46.073 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5557c0f97a1b] 2024-07-26T06:56:46.073 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5557c0f7b752] 2024-07-26T06:56:46.073 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5557c0ec9a56] 2024-07-26T06:56:46.073 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5557c0ecd637] 2024-07-26T06:56:46.073 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5557c0ecdf5e] 2024-07-26T06:56:46.073 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5557c0eb103a] 2024-07-26T06:56:46.073 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6e55b50407] 2024-07-26T06:56:46.073 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 18: (DispatchQueue::entry()+0x60f) [0x7f6e55b50cf9] 2024-07-26T06:56:46.073 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6e55c2d5a1] 2024-07-26T06:56:46.074 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 20: (Thread::entry_wrapper()+0x43) [0x7f4d18aaecdd] 2024-07-26T06:56:46.074 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4d18aaecf9] 2024-07-26T06:56:46.074 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4d183a2c02] 2024-07-26T06:56:46.074 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4d18427c40] 2024-07-26T06:56:46.074 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:46.074 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 2024-07-26T06:56:46.074 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 0> 2024-07-26T06:56:45.756+0000 7f4d12f80640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:46.074 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: in thread 7f4d12f80640 thread_name:ms_dispatch 2024-07-26T06:56:46.074 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 2024-07-26T06:56:46.074 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:46.074 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4d183576f0] 2024-07-26T06:56:46.075 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4d183a494c] 2024-07-26T06:56:46.075 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 3: raise() 2024-07-26T06:56:46.075 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 4: abort() 2024-07-26T06:56:46.075 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4d1834171b] 2024-07-26T06:56:46.075 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 6: /lib64/libc.so.6(+0x37386) [0x7f4d18350386] 2024-07-26T06:56:46.075 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55fee3a9e6b9] 2024-07-26T06:56:46.075 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55fee3c5c80c] 2024-07-26T06:56:46.075 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55fee3b9c2ca] 2024-07-26T06:56:46.075 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55fee3b6c753] 2024-07-26T06:56:46.075 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55fee3b6da1b] 2024-07-26T06:56:46.075 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55fee3b51752] 2024-07-26T06:56:46.075 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55fee3a9fa56] 2024-07-26T06:56:46.075 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55fee3aa3637] 2024-07-26T06:56:46.076 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55fee3aa3f5e] 2024-07-26T06:56:46.076 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55fee3a8703a] 2024-07-26T06:56:46.076 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 20: (Thread::entry_wrapper()+0x43) [0x7f6e559dacdd] 2024-07-26T06:56:46.076 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6e559dacf9] 2024-07-26T06:56:46.076 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6e552cec02] 2024-07-26T06:56:46.076 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6e55353c40] 2024-07-26T06:56:46.076 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:46.076 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 2024-07-26T06:56:46.076 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 0> 2024-07-26T06:56:45.757+0000 7f6e4feac640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:46.077 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: in thread 7f6e4feac640 thread_name:ms_dispatch 2024-07-26T06:56:46.077 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 2024-07-26T06:56:46.077 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:46.077 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6e552836f0] 2024-07-26T06:56:46.077 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6e552d094c] 2024-07-26T06:56:46.077 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 3: raise() 2024-07-26T06:56:46.077 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 4: abort() 2024-07-26T06:56:46.077 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6e5526d71b] 2024-07-26T06:56:46.077 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 6: /lib64/libc.so.6(+0x37386) [0x7f6e5527c386] 2024-07-26T06:56:46.077 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5557c0ec86b9] 2024-07-26T06:56:46.077 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5557c108680c] 2024-07-26T06:56:46.077 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5557c0fc62ca] 2024-07-26T06:56:46.077 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5557c0f96753] 2024-07-26T06:56:46.078 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5557c0f97a1b] 2024-07-26T06:56:46.078 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5557c0f7b752] 2024-07-26T06:56:46.078 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5557c0ec9a56] 2024-07-26T06:56:46.078 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5557c0ecd637] 2024-07-26T06:56:46.078 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5557c0ecdf5e] 2024-07-26T06:56:46.078 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5557c0eb103a] 2024-07-26T06:56:46.078 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4d18c24407] 2024-07-26T06:56:46.078 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 18: (DispatchQueue::entry()+0x60f) [0x7f4d18c24cf9] 2024-07-26T06:56:46.078 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4d18d015a1] 2024-07-26T06:56:46.078 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 20: (Thread::entry_wrapper()+0x43) [0x7f4d18aaecdd] 2024-07-26T06:56:46.079 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4d18aaecf9] 2024-07-26T06:56:46.079 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4d183a2c02] 2024-07-26T06:56:46.079 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4d18427c40] 2024-07-26T06:56:46.079 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:46.079 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 2024-07-26T06:56:46.079 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 0> 2024-07-26T06:56:45.756+0000 7f4d12f80640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:46.079 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: in thread 7f4d12f80640 thread_name:ms_dispatch 2024-07-26T06:56:46.079 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 2024-07-26T06:56:46.079 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:46.079 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4d183576f0] 2024-07-26T06:56:46.079 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4d183a494c] 2024-07-26T06:56:46.079 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 3: raise() 2024-07-26T06:56:46.079 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 4: abort() 2024-07-26T06:56:46.080 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4d1834171b] 2024-07-26T06:56:46.080 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 6: /lib64/libc.so.6(+0x37386) [0x7f4d18350386] 2024-07-26T06:56:46.080 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55fee3a9e6b9] 2024-07-26T06:56:46.080 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55fee3c5c80c] 2024-07-26T06:56:46.080 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55fee3b9c2ca] 2024-07-26T06:56:46.080 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55fee3b6c753] 2024-07-26T06:56:46.080 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55fee3b6da1b] 2024-07-26T06:56:46.080 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55fee3b51752] 2024-07-26T06:56:46.080 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55fee3a9fa56] 2024-07-26T06:56:46.080 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55fee3aa3637] 2024-07-26T06:56:46.081 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6e55b50407] 2024-07-26T06:56:46.081 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 18: (DispatchQueue::entry()+0x60f) [0x7f6e55b50cf9] 2024-07-26T06:56:46.081 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6e55c2d5a1] 2024-07-26T06:56:46.081 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 20: (Thread::entry_wrapper()+0x43) [0x7f6e559dacdd] 2024-07-26T06:56:46.081 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6e559dacf9] 2024-07-26T06:56:46.081 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6e552cec02] 2024-07-26T06:56:46.081 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6e55353c40] 2024-07-26T06:56:46.081 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:46.082 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 2024-07-26T06:56:46.082 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 0> 2024-07-26T06:56:45.757+0000 7f6e4feac640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:46.082 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: in thread 7f6e4feac640 thread_name:ms_dispatch 2024-07-26T06:56:46.082 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 2024-07-26T06:56:46.082 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:46.082 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6e552836f0] 2024-07-26T06:56:46.082 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6e552d094c] 2024-07-26T06:56:46.082 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 3: raise() 2024-07-26T06:56:46.082 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 4: abort() 2024-07-26T06:56:46.082 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6e5526d71b] 2024-07-26T06:56:46.082 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 6: /lib64/libc.so.6(+0x37386) [0x7f6e5527c386] 2024-07-26T06:56:46.082 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5557c0ec86b9] 2024-07-26T06:56:46.083 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5557c108680c] 2024-07-26T06:56:46.083 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5557c0fc62ca] 2024-07-26T06:56:46.083 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5557c0f96753] 2024-07-26T06:56:46.083 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5557c0f97a1b] 2024-07-26T06:56:46.083 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5557c0f7b752] 2024-07-26T06:56:46.083 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5557c0ec9a56] 2024-07-26T06:56:46.083 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5557c0ecd637] 2024-07-26T06:56:46.083 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5557c0ecdf5e] 2024-07-26T06:56:46.083 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5557c0eb103a] 2024-07-26T06:56:46.083 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6e55b50407] 2024-07-26T06:56:46.083 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 18: (DispatchQueue::entry()+0x60f) [0x7f6e55b50cf9] 2024-07-26T06:56:46.083 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6e55c2d5a1] 2024-07-26T06:56:46.083 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 20: (Thread::entry_wrapper()+0x43) [0x7f6e559dacdd] 2024-07-26T06:56:46.084 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6e559dacf9] 2024-07-26T06:56:46.084 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6e552cec02] 2024-07-26T06:56:46.084 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6e55353c40] 2024-07-26T06:56:46.084 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:46.084 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[118788]: 2024-07-26T06:56:46.084 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55fee3aa3f5e] 2024-07-26T06:56:46.084 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55fee3a8703a] 2024-07-26T06:56:46.084 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4d18c24407] 2024-07-26T06:56:46.084 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 18: (DispatchQueue::entry()+0x60f) [0x7f4d18c24cf9] 2024-07-26T06:56:46.084 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4d18d015a1] 2024-07-26T06:56:46.084 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 20: (Thread::entry_wrapper()+0x43) [0x7f4d18aaecdd] 2024-07-26T06:56:46.085 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4d18aaecf9] 2024-07-26T06:56:46.085 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4d183a2c02] 2024-07-26T06:56:46.085 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4d18427c40] 2024-07-26T06:56:46.085 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:46.085 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[118816]: 2024-07-26T06:56:46.085 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:45 smithi042 podman[118869]: 2024-07-26 06:56:45.823289157 +0000 UTC m=+0.021458952 container died ae33759893203d9fb8f6c64b90598dbc6dab22d34d3d9971b67e3e7d2e6c574b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-07-26T06:56:46.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3874449504,v1:172.21.15.42:6839/3874449504] up:rejoin 2024-07-26T06:56:46.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/1032688799,v1:172.21.15.42:6841/1032688799] up:rejoin 2024-07-26T06:56:46.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/2610757455,v1:172.21.15.164:6833/2610757455] up:rejoin 2024-07-26T06:56:46.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:56:46.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-mon[36970]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:56:46.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-mon[36970]: daemon mds.a is now active in filesystem cephfs as rank 3 2024-07-26T06:56:46.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-mon[36970]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-07-26T06:56:46.125 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:56:46.125 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: *** Caught signal (Aborted) ** 2024-07-26T06:56:46.125 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: in thread 7f9d984ac640 thread_name:ms_dispatch 2024-07-26T06:56:46.125 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:46.125 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9d9d8836f0] 2024-07-26T06:56:46.125 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9d9d8d094c] 2024-07-26T06:56:46.125 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 3: raise() 2024-07-26T06:56:46.125 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 4: abort() 2024-07-26T06:56:46.125 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9d9d86d71b] 2024-07-26T06:56:46.125 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 6: /lib64/libc.so.6(+0x37386) [0x7f9d9d87c386] 2024-07-26T06:56:46.125 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5586244ad6b9] 2024-07-26T06:56:46.126 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55862466b80c] 2024-07-26T06:56:46.126 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5586245ab2ca] 2024-07-26T06:56:46.126 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55862457b753] 2024-07-26T06:56:46.126 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55862457ca1b] 2024-07-26T06:56:46.126 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558624560752] 2024-07-26T06:56:46.126 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5586244aea56] 2024-07-26T06:56:46.126 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5586244b2637] 2024-07-26T06:56:46.126 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5586244b2f5e] 2024-07-26T06:56:46.126 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55862449603a] 2024-07-26T06:56:46.126 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9d9e150407] 2024-07-26T06:56:46.126 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 18: (DispatchQueue::entry()+0x60f) [0x7f9d9e150cf9] 2024-07-26T06:56:46.126 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9d9e22d5a1] 2024-07-26T06:56:46.127 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 20: (Thread::entry_wrapper()+0x43) [0x7f9d9dfdacdd] 2024-07-26T06:56:46.127 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9d9dfdacf9] 2024-07-26T06:56:46.127 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9d9d8cec02] 2024-07-26T06:56:46.128 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9d9d953c40] 2024-07-26T06:56:46.128 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 2024-07-26T06:56:45.756+0000 7f9d984ac640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:46.128 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: in thread 7f9d984ac640 thread_name:ms_dispatch 2024-07-26T06:56:46.128 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 2024-07-26T06:56:46.128 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:46.128 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9d9d8836f0] 2024-07-26T06:56:46.128 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9d9d8d094c] 2024-07-26T06:56:46.128 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 3: raise() 2024-07-26T06:56:46.128 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 4: abort() 2024-07-26T06:56:46.128 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9d9d86d71b] 2024-07-26T06:56:46.128 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 6: /lib64/libc.so.6(+0x37386) [0x7f9d9d87c386] 2024-07-26T06:56:46.128 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5586244ad6b9] 2024-07-26T06:56:46.129 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55862466b80c] 2024-07-26T06:56:46.129 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5586245ab2ca] 2024-07-26T06:56:46.129 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55862457b753] 2024-07-26T06:56:46.129 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55862457ca1b] 2024-07-26T06:56:46.129 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558624560752] 2024-07-26T06:56:46.129 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5586244aea56] 2024-07-26T06:56:46.129 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5586244b2637] 2024-07-26T06:56:46.129 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5586244b2f5e] 2024-07-26T06:56:46.129 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55862449603a] 2024-07-26T06:56:46.129 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9d9e150407] 2024-07-26T06:56:46.130 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 18: (DispatchQueue::entry()+0x60f) [0x7f9d9e150cf9] 2024-07-26T06:56:46.130 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9d9e22d5a1] 2024-07-26T06:56:46.130 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 20: (Thread::entry_wrapper()+0x43) [0x7f9d9dfdacdd] 2024-07-26T06:56:46.130 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9d9dfdacf9] 2024-07-26T06:56:46.130 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9d9d8cec02] 2024-07-26T06:56:46.130 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9d9d953c40] 2024-07-26T06:56:46.130 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:46.130 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 2024-07-26T06:56:46.130 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 0> 2024-07-26T06:56:45.756+0000 7f9d984ac640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:46.130 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: in thread 7f9d984ac640 thread_name:ms_dispatch 2024-07-26T06:56:46.130 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 2024-07-26T06:56:46.130 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:46.131 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9d9d8836f0] 2024-07-26T06:56:46.131 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9d9d8d094c] 2024-07-26T06:56:46.131 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 3: raise() 2024-07-26T06:56:46.131 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 4: abort() 2024-07-26T06:56:46.131 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9d9d86d71b] 2024-07-26T06:56:46.131 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 6: /lib64/libc.so.6(+0x37386) [0x7f9d9d87c386] 2024-07-26T06:56:46.131 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5586244ad6b9] 2024-07-26T06:56:46.131 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55862466b80c] 2024-07-26T06:56:46.131 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5586245ab2ca] 2024-07-26T06:56:46.131 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55862457b753] 2024-07-26T06:56:46.131 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55862457ca1b] 2024-07-26T06:56:46.131 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558624560752] 2024-07-26T06:56:46.132 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5586244aea56] 2024-07-26T06:56:46.132 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5586244b2637] 2024-07-26T06:56:46.132 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5586244b2f5e] 2024-07-26T06:56:46.132 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55862449603a] 2024-07-26T06:56:46.132 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9d9e150407] 2024-07-26T06:56:46.132 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 18: (DispatchQueue::entry()+0x60f) [0x7f9d9e150cf9] 2024-07-26T06:56:46.132 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9d9e22d5a1] 2024-07-26T06:56:46.132 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 20: (Thread::entry_wrapper()+0x43) [0x7f9d9dfdacdd] 2024-07-26T06:56:46.132 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9d9dfdacf9] 2024-07-26T06:56:46.132 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9d9d8cec02] 2024-07-26T06:56:46.132 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9d9d953c40] 2024-07-26T06:56:46.132 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:46.133 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 2024-07-26T06:56:46.133 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 0> 2024-07-26T06:56:45.756+0000 7f9d984ac640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:56:46.133 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: in thread 7f9d984ac640 thread_name:ms_dispatch 2024-07-26T06:56:46.133 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 2024-07-26T06:56:46.133 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:56:46.133 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9d9d8836f0] 2024-07-26T06:56:46.133 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9d9d8d094c] 2024-07-26T06:56:46.133 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 3: raise() 2024-07-26T06:56:46.133 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 4: abort() 2024-07-26T06:56:46.133 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9d9d86d71b] 2024-07-26T06:56:46.133 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 6: /lib64/libc.so.6(+0x37386) [0x7f9d9d87c386] 2024-07-26T06:56:46.133 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5586244ad6b9] 2024-07-26T06:56:46.133 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55862466b80c] 2024-07-26T06:56:46.133 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5586245ab2ca] 2024-07-26T06:56:46.134 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55862457b753] 2024-07-26T06:56:46.134 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55862457ca1b] 2024-07-26T06:56:46.134 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558624560752] 2024-07-26T06:56:46.134 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5586244aea56] 2024-07-26T06:56:46.134 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5586244b2637] 2024-07-26T06:56:46.134 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5586244b2f5e] 2024-07-26T06:56:46.134 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55862449603a] 2024-07-26T06:56:46.134 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9d9e150407] 2024-07-26T06:56:46.134 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 18: (DispatchQueue::entry()+0x60f) [0x7f9d9e150cf9] 2024-07-26T06:56:46.134 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9d9e22d5a1] 2024-07-26T06:56:46.134 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 20: (Thread::entry_wrapper()+0x43) [0x7f9d9dfdacdd] 2024-07-26T06:56:46.134 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9d9dfdacf9] 2024-07-26T06:56:46.135 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9d9d8cec02] 2024-07-26T06:56:46.135 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9d9d953c40] 2024-07-26T06:56:46.135 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:56:46.135 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70533]: 2024-07-26T06:56:46.135 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:45 smithi164 podman[70573]: 2024-07-26 06:56:45.815710003 +0000 UTC m=+0.020798580 container died 8b180b3acc87dfbc396e1af8b6e20f4917a902f4c48e20557de55ca78b8a989f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-07-26T06:56:46.727 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:46 smithi164 podman[70573]: 2024-07-26 06:56:46.476081883 +0000 UTC m=+0.681170460 container remove 8b180b3acc87dfbc396e1af8b6e20f4917a902f4c48e20557de55ca78b8a989f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-07-26T06:56:46.728 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:46 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:56:46.824 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:46 smithi042 podman[118875]: 2024-07-26 06:56:46.564558623 +0000 UTC m=+0.755885432 container died 75a652fcdbdd7b936f12ae774f5861533e5759a015bcfdcb1c753589a7dfd0b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T06:56:46.824 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:46 smithi042 podman[118869]: 2024-07-26 06:56:46.546043086 +0000 UTC m=+0.744212880 container remove ae33759893203d9fb8f6c64b90598dbc6dab22d34d3d9971b67e3e7d2e6c574b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-07-26T06:56:46.824 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:46 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:56:46.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:46 smithi042 ceph-mon[32775]: pgmap v603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T06:56:46.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:46 smithi042 ceph-mon[32775]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:56:46.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:46 smithi042 ceph-mon[32775]: Cluster is now healthy 2024-07-26T06:56:46.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:46 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/2610757455,v1:172.21.15.164:6833/2610757455] up:active 2024-07-26T06:56:46.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:46 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/1032688799,v1:172.21.15.42:6841/1032688799] up:active 2024-07-26T06:56:46.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:46 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3874449504,v1:172.21.15.42:6839/3874449504] up:active 2024-07-26T06:56:46.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:46 smithi042 ceph-mon[32775]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:56:46.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:46 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:56:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:46 smithi163 ceph-mon[36873]: pgmap v603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T06:56:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:46 smithi163 ceph-mon[36873]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:56:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:46 smithi163 ceph-mon[36873]: Cluster is now healthy 2024-07-26T06:56:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:46 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/2610757455,v1:172.21.15.164:6833/2610757455] up:active 2024-07-26T06:56:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:46 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/1032688799,v1:172.21.15.42:6841/1032688799] up:active 2024-07-26T06:56:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:46 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3874449504,v1:172.21.15.42:6839/3874449504] up:active 2024-07-26T06:56:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:46 smithi163 ceph-mon[36873]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:56:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:46 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:56:47.222 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:46 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-07-26T06:56:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:46 smithi164 ceph-mon[36970]: pgmap v603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T06:56:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:46 smithi164 ceph-mon[36970]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:56:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:46 smithi164 ceph-mon[36970]: Cluster is now healthy 2024-07-26T06:56:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:46 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/2610757455,v1:172.21.15.164:6833/2610757455] up:active 2024-07-26T06:56:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:46 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/1032688799,v1:172.21.15.42:6841/1032688799] up:active 2024-07-26T06:56:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:46 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3874449504,v1:172.21.15.42:6839/3874449504] up:active 2024-07-26T06:56:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:46 smithi164 ceph-mon[36970]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:56:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:46 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:56:47.462 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:47 smithi042 podman[118875]: 2024-07-26 06:56:47.192984789 +0000 UTC m=+1.384311597 container remove 75a652fcdbdd7b936f12ae774f5861533e5759a015bcfdcb1c753589a7dfd0b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-07-26T06:56:47.463 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:47 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:56:47.791 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:47 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T06:56:47.792 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:47 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Consumed 1.065s CPU time. 2024-07-26T06:56:47.792 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:47 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-07-26T06:56:47.792 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:47 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Consumed 1.021s CPU time. 2024-07-26T06:56:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:48 smithi163 ceph-mon[36873]: pgmap v604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 296 B/s wr, 12 op/s 2024-07-26T06:56:49.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:48 smithi042 ceph-mon[32775]: pgmap v604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 296 B/s wr, 12 op/s 2024-07-26T06:56:49.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:48 smithi164 ceph-mon[36970]: pgmap v604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 296 B/s wr, 12 op/s 2024-07-26T06:56:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:50 smithi163 ceph-mon[36873]: pgmap v605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-07-26T06:56:51.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:50 smithi042 ceph-mon[32775]: pgmap v605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-07-26T06:56:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:50 smithi164 ceph-mon[36970]: pgmap v605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 1.2 KiB/s wr, 15 op/s 2024-07-26T06:56:53.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:52 smithi042 ceph-mon[32775]: pgmap v606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 1023 B/s wr, 13 op/s 2024-07-26T06:56:53.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:52 smithi164 ceph-mon[36970]: pgmap v606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 1023 B/s wr, 13 op/s 2024-07-26T06:56:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:52 smithi163 ceph-mon[36873]: pgmap v606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 1023 B/s wr, 13 op/s 2024-07-26T06:56:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:53.292+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:56:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:53.292+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T06:56:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:53.292+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T06:56:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:53.292+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T06:56:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:56:53.292+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:56:53.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:53.700+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:56:53.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:53.701+0000 7faa2dcb9640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 msgr2=0x561ca20d0cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T06:56:53.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:53.701+0000 7faa2dcb9640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 msgr2=0x561ca20d0cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T06:56:53.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:53.701+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T06:56:53.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:56:53.701+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:56:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:54 smithi042 ceph-mon[32775]: pgmap v607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 870 B/s wr, 12 op/s 2024-07-26T06:56:55.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:54 smithi164 ceph-mon[36970]: pgmap v607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 870 B/s wr, 12 op/s 2024-07-26T06:56:55.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:54 smithi163 ceph-mon[36873]: pgmap v607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 870 B/s wr, 12 op/s 2024-07-26T06:56:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:56 smithi042 ceph-mon[32775]: pgmap v608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 853 B/s wr, 12 op/s 2024-07-26T06:56:57.207 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:56 smithi164 ceph-mon[36970]: pgmap v608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 853 B/s wr, 12 op/s 2024-07-26T06:56:57.207 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:56 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 2. 2024-07-26T06:56:57.208 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:56 smithi164 systemd[1]: Stopped Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:57.208 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:56 smithi164 systemd[1]: Starting Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:56:57.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:56 smithi163 ceph-mon[36873]: pgmap v608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 853 B/s wr, 12 op/s 2024-07-26T06:56:57.472 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:57 smithi164 podman[70669]: 2024-07-26 06:56:57.110134935 +0000 UTC m=+0.018113608 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:56:57.472 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:57 smithi164 podman[70669]: 2024-07-26 06:56:57.235344036 +0000 UTC m=+0.143322696 container create 05c159eaa36fa59ba074a0fdb1e433cfd52662120704644479d9d1c34b0ac93f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T06:56:57.770 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:57 smithi164 podman[70669]: 2024-07-26 06:56:57.479798795 +0000 UTC m=+0.387777471 container init 05c159eaa36fa59ba074a0fdb1e433cfd52662120704644479d9d1c34b0ac93f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-07-26T06:56:57.770 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:57 smithi164 podman[70669]: 2024-07-26 06:56:57.48234237 +0000 UTC m=+0.390321030 container start 05c159eaa36fa59ba074a0fdb1e433cfd52662120704644479d9d1c34b0ac93f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True) 2024-07-26T06:56:57.770 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:57 smithi164 ceph-mds[70689]: -- 172.21.15.164:0/74897356 <== mon.1 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559bef23f860 con 0x559bf33e1600 2024-07-26T06:56:57.770 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:57 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: starting mds.l at 2024-07-26T06:56:57.770 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:57 smithi164 bash[70669]: 05c159eaa36fa59ba074a0fdb1e433cfd52662120704644479d9d1c34b0ac93f 2024-07-26T06:56:57.770 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:56:57 smithi164 systemd[1]: Started Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:57.903 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:57 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 2. 2024-07-26T06:56:57.903 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:57 smithi042 systemd[1]: Stopped Ceph mds.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:57.903 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:57 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Consumed 1.065s CPU time. 2024-07-26T06:56:57.904 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:57 smithi042 systemd[1]: Starting Ceph mds.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:56:57.904 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:57 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 2. 2024-07-26T06:56:57.904 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:57 smithi042 systemd[1]: Stopped Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:57.904 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:57 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Consumed 1.021s CPU time. 2024-07-26T06:56:57.904 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:57 smithi042 systemd[1]: Starting Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:56:57.904 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:57 smithi042 ceph-mon[32775]: daemon mds.l restarted 2024-07-26T06:56:57.905 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:57 smithi042 ceph-mon[32775]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:56:57.905 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:57 smithi042 ceph-mon[32775]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T06:56:57.905 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:57 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/2696987957,v1:172.21.15.164:6833/2696987957] up:boot 2024-07-26T06:56:57.905 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:57 smithi042 ceph-mon[32775]: Standby daemon mds.h assigned to filesystem cephfs as rank 2 2024-07-26T06:56:57.905 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:57 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:56:57.905 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:56:57.905 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:57 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:56:58.213 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:57 smithi042 podman[119046]: 2024-07-26 06:56:57.807588096 +0000 UTC m=+0.022269453 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:56:58.214 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:57 smithi042 podman[119045]: 2024-07-26 06:56:57.806619503 +0000 UTC m=+0.021292726 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:56:58.214 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:57 smithi042 podman[119045]: 2024-07-26 06:56:57.969199507 +0000 UTC m=+0.183872731 container create 3d9471d11705f0c5c3b772637f63fa1d994778f5f1ffda4d6507a239179d9325 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS) 2024-07-26T06:56:58.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:57 smithi164 ceph-mon[36970]: daemon mds.l restarted 2024-07-26T06:56:58.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:57 smithi164 ceph-mon[36970]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:56:58.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:57 smithi164 ceph-mon[36970]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T06:56:58.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:57 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/2696987957,v1:172.21.15.164:6833/2696987957] up:boot 2024-07-26T06:56:58.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:57 smithi164 ceph-mon[36970]: Standby daemon mds.h assigned to filesystem cephfs as rank 2 2024-07-26T06:56:58.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:57 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:56:58.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:56:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:57 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:56:58.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:57 smithi163 ceph-mon[36873]: daemon mds.l restarted 2024-07-26T06:56:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:57 smithi163 ceph-mon[36873]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:56:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:57 smithi163 ceph-mon[36873]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T06:56:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:57 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/2696987957,v1:172.21.15.164:6833/2696987957] up:boot 2024-07-26T06:56:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:57 smithi163 ceph-mon[36873]: Standby daemon mds.h assigned to filesystem cephfs as rank 2 2024-07-26T06:56:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:57 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:56:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:56:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:57 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:56:58.580 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:58 smithi042 podman[119046]: 2024-07-26 06:56:58.456685314 +0000 UTC m=+0.671366700 container create f83bce301f9db709c7b092d0096e0516b1a3545b1de9b29b5b754d2371029ee7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T06:56:58.581 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:58 smithi042 podman[119045]: 2024-07-26 06:56:58.213849485 +0000 UTC m=+0.428522710 container init 3d9471d11705f0c5c3b772637f63fa1d994778f5f1ffda4d6507a239179d9325 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-07-26T06:56:58.581 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:58 smithi042 podman[119045]: 2024-07-26 06:56:58.217725715 +0000 UTC m=+0.432398938 container start 3d9471d11705f0c5c3b772637f63fa1d994778f5f1ffda4d6507a239179d9325 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, io.buildah.version=1.36.0) 2024-07-26T06:56:58.581 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mds[119077]: -- 172.21.15.42:0/1693588676 <== mon.0 v2:172.21.15.42:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c6a33dbc20 con 0x55c6a757da80 2024-07-26T06:56:58.581 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: starting mds.g at 2024-07-26T06:56:58.581 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:58 smithi042 bash[119045]: 3d9471d11705f0c5c3b772637f63fa1d994778f5f1ffda4d6507a239179d9325 2024-07-26T06:56:58.581 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:56:58 smithi042 systemd[1]: Started Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:58.894 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:58 smithi042 podman[119046]: 2024-07-26 06:56:58.701271531 +0000 UTC m=+0.915952878 container init f83bce301f9db709c7b092d0096e0516b1a3545b1de9b29b5b754d2371029ee7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T06:56:58.895 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:58 smithi042 podman[119046]: 2024-07-26 06:56:58.704229668 +0000 UTC m=+0.918911012 container start f83bce301f9db709c7b092d0096e0516b1a3545b1de9b29b5b754d2371029ee7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T06:56:58.895 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mds[119103]: -- 172.21.15.42:0/2054207054 <== mon.0 v2:172.21.15.42:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560bccf0b860 con 0x560bd10ad600 2024-07-26T06:56:58.895 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: starting mds.a at 2024-07-26T06:56:58.895 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:58 smithi042 bash[119046]: f83bce301f9db709c7b092d0096e0516b1a3545b1de9b29b5b754d2371029ee7 2024-07-26T06:56:58.895 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:56:58 smithi042 systemd[1]: Started Ceph mds.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:56:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mon[32775]: pgmap v609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 853 B/s wr, 11 op/s 2024-07-26T06:56:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mon[32775]: daemon mds.g restarted 2024-07-26T06:56:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mon[32775]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T06:56:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/2567296059,v1:172.21.15.163:6839/2567296059] up:resolve 2024-07-26T06:56:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3995951481,v1:172.21.15.42:6839/3995951481] up:boot 2024-07-26T06:56:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mon[32775]: Standby daemon mds.b assigned to filesystem cephfs as rank 4 2024-07-26T06:56:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,3=a=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:56:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:56:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:56:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mon[32775]: daemon mds.a restarted 2024-07-26T06:56:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mon[32775]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T06:56:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/2505332261,v1:172.21.15.42:6841/2505332261] up:boot 2024-07-26T06:56:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mon[32775]: Standby daemon mds.e assigned to filesystem cephfs as rank 3 2024-07-26T06:56:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,4=b=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:56:58.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T06:56:58.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:56:58 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:56:59.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:58 smithi164 ceph-mon[36970]: pgmap v609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 853 B/s wr, 11 op/s 2024-07-26T06:56:59.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:58 smithi164 ceph-mon[36970]: daemon mds.g restarted 2024-07-26T06:56:59.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:58 smithi164 ceph-mon[36970]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T06:56:59.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:58 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/2567296059,v1:172.21.15.163:6839/2567296059] up:resolve 2024-07-26T06:56:59.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:58 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3995951481,v1:172.21.15.42:6839/3995951481] up:boot 2024-07-26T06:56:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:58 smithi164 ceph-mon[36970]: Standby daemon mds.b assigned to filesystem cephfs as rank 4 2024-07-26T06:56:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:58 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,3=a=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:56:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:56:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:58 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:56:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:58 smithi164 ceph-mon[36970]: daemon mds.a restarted 2024-07-26T06:56:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:58 smithi164 ceph-mon[36970]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T06:56:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:58 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/2505332261,v1:172.21.15.42:6841/2505332261] up:boot 2024-07-26T06:56:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:58 smithi164 ceph-mon[36970]: Standby daemon mds.e assigned to filesystem cephfs as rank 3 2024-07-26T06:56:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:58 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,4=b=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:56:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T06:56:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:56:58 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:56:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:58 smithi163 ceph-mon[36873]: pgmap v609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 853 B/s wr, 11 op/s 2024-07-26T06:56:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:58 smithi163 ceph-mon[36873]: daemon mds.g restarted 2024-07-26T06:56:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:58 smithi163 ceph-mon[36873]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T06:56:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:58 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/2567296059,v1:172.21.15.163:6839/2567296059] up:resolve 2024-07-26T06:56:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:58 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3995951481,v1:172.21.15.42:6839/3995951481] up:boot 2024-07-26T06:56:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:58 smithi163 ceph-mon[36873]: Standby daemon mds.b assigned to filesystem cephfs as rank 4 2024-07-26T06:56:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:58 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,3=a=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:56:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:56:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:58 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:56:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:58 smithi163 ceph-mon[36873]: daemon mds.a restarted 2024-07-26T06:56:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:58 smithi163 ceph-mon[36873]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T06:56:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:58 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/2505332261,v1:172.21.15.42:6841/2505332261] up:boot 2024-07-26T06:56:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:58 smithi163 ceph-mon[36873]: Standby daemon mds.e assigned to filesystem cephfs as rank 3 2024-07-26T06:56:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:58 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,4=b=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:56:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T06:56:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:56:58 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:57:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:00 smithi042 ceph-mon[32775]: pgmap v613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 13 op/s 2024-07-26T06:57:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:00 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/2332407499,v1:172.21.15.163:6837/2332407499] up:resolve 2024-07-26T06:57:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:00 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/2760283174,v1:172.21.15.163:6835/2760283174] up:resolve 2024-07-26T06:57:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:00 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:57:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:00 smithi164 ceph-mon[36970]: pgmap v613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 13 op/s 2024-07-26T06:57:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:00 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/2332407499,v1:172.21.15.163:6837/2332407499] up:resolve 2024-07-26T06:57:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:00 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/2760283174,v1:172.21.15.163:6835/2760283174] up:resolve 2024-07-26T06:57:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:00 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:57:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:00 smithi163 ceph-mon[36873]: pgmap v613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 13 op/s 2024-07-26T06:57:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:00 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/2332407499,v1:172.21.15.163:6837/2332407499] up:resolve 2024-07-26T06:57:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:00 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/2760283174,v1:172.21.15.163:6835/2760283174] up:resolve 2024-07-26T06:57:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:00 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:57:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:01 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:57:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:01 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:57:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:01 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:57:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:01 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/2567296059,v1:172.21.15.163:6839/2567296059] up:reconnect 2024-07-26T06:57:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:01 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/2760283174,v1:172.21.15.163:6835/2760283174] up:reconnect 2024-07-26T06:57:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:01 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/2332407499,v1:172.21.15.163:6837/2332407499] up:reconnect 2024-07-26T06:57:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:01 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:57:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:01 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:57:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:01 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:57:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:01 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:57:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:01 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/2567296059,v1:172.21.15.163:6839/2567296059] up:reconnect 2024-07-26T06:57:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:01 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/2760283174,v1:172.21.15.163:6835/2760283174] up:reconnect 2024-07-26T06:57:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:01 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/2332407499,v1:172.21.15.163:6837/2332407499] up:reconnect 2024-07-26T06:57:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:01 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:57:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:01 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:57:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:01 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:57:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:01 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0 2024-07-26T06:57:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:01 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/2567296059,v1:172.21.15.163:6839/2567296059] up:reconnect 2024-07-26T06:57:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:01 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/2760283174,v1:172.21.15.163:6835/2760283174] up:reconnect 2024-07-26T06:57:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:01 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/2332407499,v1:172.21.15.163:6837/2332407499] up:reconnect 2024-07-26T06:57:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:01 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:57:03.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:02 smithi042 ceph-mon[32775]: pgmap v614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-07-26T06:57:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:02 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/2760283174,v1:172.21.15.163:6835/2760283174] up:rejoin 2024-07-26T06:57:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:02 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/2332407499,v1:172.21.15.163:6837/2332407499] up:rejoin 2024-07-26T06:57:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:02 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/2567296059,v1:172.21.15.163:6839/2567296059] up:rejoin 2024-07-26T06:57:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:02 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:57:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:02 smithi042 ceph-mon[32775]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-07-26T06:57:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:02 smithi042 ceph-mon[32775]: daemon mds.b is now active in filesystem cephfs as rank 4 2024-07-26T06:57:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:02 smithi042 ceph-mon[32775]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-07-26T06:57:03.180 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-mon[36873]: pgmap v614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-07-26T06:57:03.180 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/2760283174,v1:172.21.15.163:6835/2760283174] up:rejoin 2024-07-26T06:57:03.180 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/2332407499,v1:172.21.15.163:6837/2332407499] up:rejoin 2024-07-26T06:57:03.180 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/2567296059,v1:172.21.15.163:6839/2567296059] up:rejoin 2024-07-26T06:57:03.181 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:57:03.181 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-mon[36873]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-07-26T06:57:03.181 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-mon[36873]: daemon mds.b is now active in filesystem cephfs as rank 4 2024-07-26T06:57:03.181 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-mon[36873]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-07-26T06:57:03.181 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:57:03.181 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: *** Caught signal (Aborted) ** 2024-07-26T06:57:03.181 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: in thread 7f389793b640 thread_name:ms_dispatch 2024-07-26T06:57:03.182 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:03.182 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f389cd126f0] 2024-07-26T06:57:03.182 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f389cd5f94c] 2024-07-26T06:57:03.182 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 3: raise() 2024-07-26T06:57:03.182 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 4: abort() 2024-07-26T06:57:03.182 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 5: /lib64/libc.so.6(+0x2871b) [0x7f389ccfc71b] 2024-07-26T06:57:03.182 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 6: /lib64/libc.so.6(+0x37386) [0x7f389cd0b386] 2024-07-26T06:57:03.182 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5643d5a0e6b9] 2024-07-26T06:57:03.182 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5643d5bcc80c] 2024-07-26T06:57:03.182 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5643d5b0c2ca] 2024-07-26T06:57:03.182 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5643d5adc753] 2024-07-26T06:57:03.182 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5643d5adda1b] 2024-07-26T06:57:03.183 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5643d5ac1752] 2024-07-26T06:57:03.183 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5643d5a0fa56] 2024-07-26T06:57:03.183 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5643d5a13637] 2024-07-26T06:57:03.183 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5643d5a13f5e] 2024-07-26T06:57:03.183 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5643d59f703a] 2024-07-26T06:57:03.183 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f389d5df407] 2024-07-26T06:57:03.183 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 18: (DispatchQueue::entry()+0x60f) [0x7f389d5dfcf9] 2024-07-26T06:57:03.183 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f389d6bc5a1] 2024-07-26T06:57:03.184 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:57:03.184 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: *** Caught signal (Aborted) ** 2024-07-26T06:57:03.184 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: in thread 7f858e4c9640 thread_name:ms_dispatch 2024-07-26T06:57:03.184 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:03.184 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f85938a06f0] 2024-07-26T06:57:03.184 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f85938ed94c] 2024-07-26T06:57:03.184 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 3: raise() 2024-07-26T06:57:03.184 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 4: abort() 2024-07-26T06:57:03.184 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 5: /lib64/libc.so.6(+0x2871b) [0x7f859388a71b] 2024-07-26T06:57:03.184 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 6: /lib64/libc.so.6(+0x37386) [0x7f8593899386] 2024-07-26T06:57:03.185 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56322bb336b9] 2024-07-26T06:57:03.185 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56322bcf180c] 2024-07-26T06:57:03.185 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56322bc312ca] 2024-07-26T06:57:03.185 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56322bc01753] 2024-07-26T06:57:03.185 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56322bc02a1b] 2024-07-26T06:57:03.185 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56322bbe6752] 2024-07-26T06:57:03.185 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56322bb34a56] 2024-07-26T06:57:03.185 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56322bb38637] 2024-07-26T06:57:03.185 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56322bb38f5e] 2024-07-26T06:57:03.185 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56322bb1c03a] 2024-07-26T06:57:03.185 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f859416d407] 2024-07-26T06:57:03.185 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 18: (DispatchQueue::entry()+0x60f) [0x7f859416dcf9] 2024-07-26T06:57:03.185 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f859424a5a1] 2024-07-26T06:57:03.186 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:57:03.186 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: *** Caught signal (Aborted) ** 2024-07-26T06:57:03.186 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: in thread 7f4abd463640 thread_name:ms_dispatch 2024-07-26T06:57:03.186 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:03.186 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4ac283a6f0] 2024-07-26T06:57:03.186 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4ac288794c] 2024-07-26T06:57:03.186 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 3: raise() 2024-07-26T06:57:03.186 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 4: abort() 2024-07-26T06:57:03.187 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4ac282471b] 2024-07-26T06:57:03.187 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 6: /lib64/libc.so.6(+0x37386) [0x7f4ac2833386] 2024-07-26T06:57:03.187 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557e2b3176b9] 2024-07-26T06:57:03.187 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557e2b4d580c] 2024-07-26T06:57:03.187 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557e2b4152ca] 2024-07-26T06:57:03.187 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x557e2b3e5753] 2024-07-26T06:57:03.187 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557e2b3e6a1b] 2024-07-26T06:57:03.187 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557e2b3ca752] 2024-07-26T06:57:03.187 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557e2b318a56] 2024-07-26T06:57:03.187 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557e2b31c637] 2024-07-26T06:57:03.187 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557e2b31cf5e] 2024-07-26T06:57:03.187 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557e2b30003a] 2024-07-26T06:57:03.188 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4ac3107407] 2024-07-26T06:57:03.188 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 18: (DispatchQueue::entry()+0x60f) [0x7f4ac3107cf9] 2024-07-26T06:57:03.188 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4ac31e45a1] 2024-07-26T06:57:03.188 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 20: (Thread::entry_wrapper()+0x43) [0x7f389d469cdd] 2024-07-26T06:57:03.188 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 21: (Thread::_entry_func(void*)+0xd) [0x7f389d469cf9] 2024-07-26T06:57:03.189 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 22: /lib64/libc.so.6(+0x89c02) [0x7f389cd5dc02] 2024-07-26T06:57:03.189 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f389cde2c40] 2024-07-26T06:57:03.189 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 2024-07-26T06:57:02.805+0000 7f389793b640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:03.189 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: in thread 7f389793b640 thread_name:ms_dispatch 2024-07-26T06:57:03.189 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 2024-07-26T06:57:03.189 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:03.189 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f389cd126f0] 2024-07-26T06:57:03.189 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f389cd5f94c] 2024-07-26T06:57:03.189 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 3: raise() 2024-07-26T06:57:03.189 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 4: abort() 2024-07-26T06:57:03.189 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 5: /lib64/libc.so.6(+0x2871b) [0x7f389ccfc71b] 2024-07-26T06:57:03.189 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 6: /lib64/libc.so.6(+0x37386) [0x7f389cd0b386] 2024-07-26T06:57:03.190 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5643d5a0e6b9] 2024-07-26T06:57:03.190 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5643d5bcc80c] 2024-07-26T06:57:03.190 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5643d5b0c2ca] 2024-07-26T06:57:03.190 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5643d5adc753] 2024-07-26T06:57:03.190 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5643d5adda1b] 2024-07-26T06:57:03.190 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5643d5ac1752] 2024-07-26T06:57:03.190 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5643d5a0fa56] 2024-07-26T06:57:03.190 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5643d5a13637] 2024-07-26T06:57:03.190 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5643d5a13f5e] 2024-07-26T06:57:03.190 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5643d59f703a] 2024-07-26T06:57:03.191 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f389d5df407] 2024-07-26T06:57:03.191 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 18: (DispatchQueue::entry()+0x60f) [0x7f389d5dfcf9] 2024-07-26T06:57:03.191 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f389d6bc5a1] 2024-07-26T06:57:03.191 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 20: (Thread::entry_wrapper()+0x43) [0x7f389d469cdd] 2024-07-26T06:57:03.191 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 21: (Thread::_entry_func(void*)+0xd) [0x7f389d469cf9] 2024-07-26T06:57:03.191 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 22: /lib64/libc.so.6(+0x89c02) [0x7f389cd5dc02] 2024-07-26T06:57:03.191 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f389cde2c40] 2024-07-26T06:57:03.191 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:03.191 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 2024-07-26T06:57:03.191 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 0> 2024-07-26T06:57:02.805+0000 7f389793b640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:03.191 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: in thread 7f389793b640 thread_name:ms_dispatch 2024-07-26T06:57:03.191 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 2024-07-26T06:57:03.191 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:03.192 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f389cd126f0] 2024-07-26T06:57:03.192 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f389cd5f94c] 2024-07-26T06:57:03.192 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 3: raise() 2024-07-26T06:57:03.192 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 4: abort() 2024-07-26T06:57:03.192 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 5: /lib64/libc.so.6(+0x2871b) [0x7f389ccfc71b] 2024-07-26T06:57:03.192 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 6: /lib64/libc.so.6(+0x37386) [0x7f389cd0b386] 2024-07-26T06:57:03.192 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5643d5a0e6b9] 2024-07-26T06:57:03.192 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5643d5bcc80c] 2024-07-26T06:57:03.192 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5643d5b0c2ca] 2024-07-26T06:57:03.192 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5643d5adc753] 2024-07-26T06:57:03.192 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5643d5adda1b] 2024-07-26T06:57:03.192 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5643d5ac1752] 2024-07-26T06:57:03.192 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5643d5a0fa56] 2024-07-26T06:57:03.192 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5643d5a13637] 2024-07-26T06:57:03.193 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5643d5a13f5e] 2024-07-26T06:57:03.193 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5643d59f703a] 2024-07-26T06:57:03.193 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f389d5df407] 2024-07-26T06:57:03.193 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 20: (Thread::entry_wrapper()+0x43) [0x7f8593ff7cdd] 2024-07-26T06:57:03.193 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8593ff7cf9] 2024-07-26T06:57:03.193 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 22: /lib64/libc.so.6(+0x89c02) [0x7f85938ebc02] 2024-07-26T06:57:03.193 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8593970c40] 2024-07-26T06:57:03.193 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 2024-07-26T06:57:02.806+0000 7f858e4c9640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:03.193 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: in thread 7f858e4c9640 thread_name:ms_dispatch 2024-07-26T06:57:03.193 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 2024-07-26T06:57:03.194 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:03.194 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f85938a06f0] 2024-07-26T06:57:03.194 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f85938ed94c] 2024-07-26T06:57:03.194 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 3: raise() 2024-07-26T06:57:03.194 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 4: abort() 2024-07-26T06:57:03.194 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 5: /lib64/libc.so.6(+0x2871b) [0x7f859388a71b] 2024-07-26T06:57:03.194 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 6: /lib64/libc.so.6(+0x37386) [0x7f8593899386] 2024-07-26T06:57:03.194 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56322bb336b9] 2024-07-26T06:57:03.194 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56322bcf180c] 2024-07-26T06:57:03.194 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56322bc312ca] 2024-07-26T06:57:03.194 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56322bc01753] 2024-07-26T06:57:03.194 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56322bc02a1b] 2024-07-26T06:57:03.195 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56322bbe6752] 2024-07-26T06:57:03.195 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56322bb34a56] 2024-07-26T06:57:03.195 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56322bb38637] 2024-07-26T06:57:03.195 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56322bb38f5e] 2024-07-26T06:57:03.195 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56322bb1c03a] 2024-07-26T06:57:03.195 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f859416d407] 2024-07-26T06:57:03.195 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 18: (DispatchQueue::entry()+0x60f) [0x7f859416dcf9] 2024-07-26T06:57:03.195 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f859424a5a1] 2024-07-26T06:57:03.195 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 20: (Thread::entry_wrapper()+0x43) [0x7f4ac2f91cdd] 2024-07-26T06:57:03.195 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4ac2f91cf9] 2024-07-26T06:57:03.196 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4ac2885c02] 2024-07-26T06:57:03.196 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4ac290ac40] 2024-07-26T06:57:03.196 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 2024-07-26T06:57:02.804+0000 7f4abd463640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:03.196 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: in thread 7f4abd463640 thread_name:ms_dispatch 2024-07-26T06:57:03.196 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 2024-07-26T06:57:03.196 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:03.196 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4ac283a6f0] 2024-07-26T06:57:03.196 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4ac288794c] 2024-07-26T06:57:03.196 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 3: raise() 2024-07-26T06:57:03.196 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 4: abort() 2024-07-26T06:57:03.196 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4ac282471b] 2024-07-26T06:57:03.196 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 6: /lib64/libc.so.6(+0x37386) [0x7f4ac2833386] 2024-07-26T06:57:03.196 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557e2b3176b9] 2024-07-26T06:57:03.196 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557e2b4d580c] 2024-07-26T06:57:03.197 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557e2b4152ca] 2024-07-26T06:57:03.197 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x557e2b3e5753] 2024-07-26T06:57:03.197 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557e2b3e6a1b] 2024-07-26T06:57:03.197 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557e2b3ca752] 2024-07-26T06:57:03.197 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557e2b318a56] 2024-07-26T06:57:03.197 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557e2b31c637] 2024-07-26T06:57:03.197 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557e2b31cf5e] 2024-07-26T06:57:03.197 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557e2b30003a] 2024-07-26T06:57:03.197 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4ac3107407] 2024-07-26T06:57:03.197 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 18: (DispatchQueue::entry()+0x60f) [0x7f4ac3107cf9] 2024-07-26T06:57:03.197 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4ac31e45a1] 2024-07-26T06:57:03.198 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 18: (DispatchQueue::entry()+0x60f) [0x7f389d5dfcf9] 2024-07-26T06:57:03.198 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f389d6bc5a1] 2024-07-26T06:57:03.198 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 20: (Thread::entry_wrapper()+0x43) [0x7f389d469cdd] 2024-07-26T06:57:03.198 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 21: (Thread::_entry_func(void*)+0xd) [0x7f389d469cf9] 2024-07-26T06:57:03.198 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 22: /lib64/libc.so.6(+0x89c02) [0x7f389cd5dc02] 2024-07-26T06:57:03.198 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f389cde2c40] 2024-07-26T06:57:03.198 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:03.198 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 2024-07-26T06:57:03.199 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 0> 2024-07-26T06:57:02.805+0000 7f389793b640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:03.199 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: in thread 7f389793b640 thread_name:ms_dispatch 2024-07-26T06:57:03.199 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 2024-07-26T06:57:03.199 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:03.199 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f389cd126f0] 2024-07-26T06:57:03.199 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f389cd5f94c] 2024-07-26T06:57:03.199 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 3: raise() 2024-07-26T06:57:03.199 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 4: abort() 2024-07-26T06:57:03.199 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 5: /lib64/libc.so.6(+0x2871b) [0x7f389ccfc71b] 2024-07-26T06:57:03.199 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 6: /lib64/libc.so.6(+0x37386) [0x7f389cd0b386] 2024-07-26T06:57:03.199 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5643d5a0e6b9] 2024-07-26T06:57:03.199 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5643d5bcc80c] 2024-07-26T06:57:03.200 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5643d5b0c2ca] 2024-07-26T06:57:03.200 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5643d5adc753] 2024-07-26T06:57:03.200 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5643d5adda1b] 2024-07-26T06:57:03.200 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5643d5ac1752] 2024-07-26T06:57:03.200 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5643d5a0fa56] 2024-07-26T06:57:03.200 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5643d5a13637] 2024-07-26T06:57:03.200 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5643d5a13f5e] 2024-07-26T06:57:03.200 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5643d59f703a] 2024-07-26T06:57:03.200 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f389d5df407] 2024-07-26T06:57:03.200 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 18: (DispatchQueue::entry()+0x60f) [0x7f389d5dfcf9] 2024-07-26T06:57:03.200 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f389d6bc5a1] 2024-07-26T06:57:03.200 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 20: (Thread::entry_wrapper()+0x43) [0x7f389d469cdd] 2024-07-26T06:57:03.201 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 21: (Thread::_entry_func(void*)+0xd) [0x7f389d469cf9] 2024-07-26T06:57:03.201 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 22: /lib64/libc.so.6(+0x89c02) [0x7f389cd5dc02] 2024-07-26T06:57:03.201 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f389cde2c40] 2024-07-26T06:57:03.201 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:03.201 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[70766]: 2024-07-26T06:57:03.201 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:02 smithi163 podman[70853]: 2024-07-26 06:57:02.870559975 +0000 UTC m=+0.020752296 container died e30626c9e226c7d2232311324bfa47293dcaaab4427c823e2765455fd1cb3f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2024-07-26T06:57:03.201 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 20: (Thread::entry_wrapper()+0x43) [0x7f8593ff7cdd] 2024-07-26T06:57:03.201 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8593ff7cf9] 2024-07-26T06:57:03.201 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 22: /lib64/libc.so.6(+0x89c02) [0x7f85938ebc02] 2024-07-26T06:57:03.201 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8593970c40] 2024-07-26T06:57:03.202 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:03.202 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 2024-07-26T06:57:03.202 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 0> 2024-07-26T06:57:02.806+0000 7f858e4c9640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:03.202 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: in thread 7f858e4c9640 thread_name:ms_dispatch 2024-07-26T06:57:03.202 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 2024-07-26T06:57:03.202 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:03.202 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f85938a06f0] 2024-07-26T06:57:03.202 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f85938ed94c] 2024-07-26T06:57:03.202 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 3: raise() 2024-07-26T06:57:03.202 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 4: abort() 2024-07-26T06:57:03.202 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 5: /lib64/libc.so.6(+0x2871b) [0x7f859388a71b] 2024-07-26T06:57:03.202 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 6: /lib64/libc.so.6(+0x37386) [0x7f8593899386] 2024-07-26T06:57:03.203 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56322bb336b9] 2024-07-26T06:57:03.203 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56322bcf180c] 2024-07-26T06:57:03.203 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56322bc312ca] 2024-07-26T06:57:03.203 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56322bc01753] 2024-07-26T06:57:03.203 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56322bc02a1b] 2024-07-26T06:57:03.203 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56322bbe6752] 2024-07-26T06:57:03.203 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56322bb34a56] 2024-07-26T06:57:03.203 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56322bb38637] 2024-07-26T06:57:03.203 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56322bb38f5e] 2024-07-26T06:57:03.203 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56322bb1c03a] 2024-07-26T06:57:03.203 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f859416d407] 2024-07-26T06:57:03.204 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 20: (Thread::entry_wrapper()+0x43) [0x7f4ac2f91cdd] 2024-07-26T06:57:03.204 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4ac2f91cf9] 2024-07-26T06:57:03.204 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4ac2885c02] 2024-07-26T06:57:03.204 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4ac290ac40] 2024-07-26T06:57:03.204 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:03.204 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 2024-07-26T06:57:03.204 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 0> 2024-07-26T06:57:02.804+0000 7f4abd463640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:03.204 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: in thread 7f4abd463640 thread_name:ms_dispatch 2024-07-26T06:57:03.204 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 2024-07-26T06:57:03.204 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:03.204 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4ac283a6f0] 2024-07-26T06:57:03.205 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4ac288794c] 2024-07-26T06:57:03.205 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 3: raise() 2024-07-26T06:57:03.205 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 4: abort() 2024-07-26T06:57:03.205 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4ac282471b] 2024-07-26T06:57:03.205 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 6: /lib64/libc.so.6(+0x37386) [0x7f4ac2833386] 2024-07-26T06:57:03.205 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557e2b3176b9] 2024-07-26T06:57:03.205 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557e2b4d580c] 2024-07-26T06:57:03.205 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557e2b4152ca] 2024-07-26T06:57:03.205 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x557e2b3e5753] 2024-07-26T06:57:03.205 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557e2b3e6a1b] 2024-07-26T06:57:03.206 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557e2b3ca752] 2024-07-26T06:57:03.206 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557e2b318a56] 2024-07-26T06:57:03.206 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557e2b31c637] 2024-07-26T06:57:03.206 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557e2b31cf5e] 2024-07-26T06:57:03.206 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557e2b30003a] 2024-07-26T06:57:03.206 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4ac3107407] 2024-07-26T06:57:03.206 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 18: (DispatchQueue::entry()+0x60f) [0x7f859416dcf9] 2024-07-26T06:57:03.207 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f859424a5a1] 2024-07-26T06:57:03.207 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 20: (Thread::entry_wrapper()+0x43) [0x7f8593ff7cdd] 2024-07-26T06:57:03.207 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8593ff7cf9] 2024-07-26T06:57:03.207 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 22: /lib64/libc.so.6(+0x89c02) [0x7f85938ebc02] 2024-07-26T06:57:03.207 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8593970c40] 2024-07-26T06:57:03.207 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:03.207 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 2024-07-26T06:57:03.207 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: -2> 2024-07-26T06:57:02.806+0000 7f858e4c9640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:03.207 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: in thread 7f858e4c9640 thread_name:ms_dispatch 2024-07-26T06:57:03.207 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 2024-07-26T06:57:03.208 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:03.208 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f85938a06f0] 2024-07-26T06:57:03.208 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f85938ed94c] 2024-07-26T06:57:03.208 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 3: raise() 2024-07-26T06:57:03.208 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 4: abort() 2024-07-26T06:57:03.208 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 5: /lib64/libc.so.6(+0x2871b) [0x7f859388a71b] 2024-07-26T06:57:03.208 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 6: /lib64/libc.so.6(+0x37386) [0x7f8593899386] 2024-07-26T06:57:03.208 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56322bb336b9] 2024-07-26T06:57:03.208 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56322bcf180c] 2024-07-26T06:57:03.208 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56322bc312ca] 2024-07-26T06:57:03.208 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56322bc01753] 2024-07-26T06:57:03.208 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56322bc02a1b] 2024-07-26T06:57:03.208 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56322bbe6752] 2024-07-26T06:57:03.209 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56322bb34a56] 2024-07-26T06:57:03.209 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56322bb38637] 2024-07-26T06:57:03.209 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56322bb38f5e] 2024-07-26T06:57:03.209 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 18: (DispatchQueue::entry()+0x60f) [0x7f4ac3107cf9] 2024-07-26T06:57:03.209 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4ac31e45a1] 2024-07-26T06:57:03.209 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 20: (Thread::entry_wrapper()+0x43) [0x7f4ac2f91cdd] 2024-07-26T06:57:03.209 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4ac2f91cf9] 2024-07-26T06:57:03.209 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4ac2885c02] 2024-07-26T06:57:03.209 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4ac290ac40] 2024-07-26T06:57:03.210 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:03.210 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 2024-07-26T06:57:03.210 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 0> 2024-07-26T06:57:02.804+0000 7f4abd463640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:03.210 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: in thread 7f4abd463640 thread_name:ms_dispatch 2024-07-26T06:57:03.210 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 2024-07-26T06:57:03.210 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:03.210 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4ac283a6f0] 2024-07-26T06:57:03.210 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4ac288794c] 2024-07-26T06:57:03.210 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 3: raise() 2024-07-26T06:57:03.210 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 4: abort() 2024-07-26T06:57:03.210 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4ac282471b] 2024-07-26T06:57:03.211 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 6: /lib64/libc.so.6(+0x37386) [0x7f4ac2833386] 2024-07-26T06:57:03.211 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557e2b3176b9] 2024-07-26T06:57:03.211 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557e2b4d580c] 2024-07-26T06:57:03.211 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557e2b4152ca] 2024-07-26T06:57:03.211 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x557e2b3e5753] 2024-07-26T06:57:03.211 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557e2b3e6a1b] 2024-07-26T06:57:03.211 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557e2b3ca752] 2024-07-26T06:57:03.211 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557e2b318a56] 2024-07-26T06:57:03.211 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557e2b31c637] 2024-07-26T06:57:03.211 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557e2b31cf5e] 2024-07-26T06:57:03.212 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56322bb1c03a] 2024-07-26T06:57:03.212 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f859416d407] 2024-07-26T06:57:03.212 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 18: (DispatchQueue::entry()+0x60f) [0x7f859416dcf9] 2024-07-26T06:57:03.212 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f859424a5a1] 2024-07-26T06:57:03.212 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 20: (Thread::entry_wrapper()+0x43) [0x7f8593ff7cdd] 2024-07-26T06:57:03.212 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 21: (Thread::_entry_func(void*)+0xd) [0x7f8593ff7cf9] 2024-07-26T06:57:03.212 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 22: /lib64/libc.so.6(+0x89c02) [0x7f85938ebc02] 2024-07-26T06:57:03.212 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f8593970c40] 2024-07-26T06:57:03.212 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:03.212 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[70641]: 2024-07-26T06:57:03.213 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557e2b30003a] 2024-07-26T06:57:03.213 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4ac3107407] 2024-07-26T06:57:03.213 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 18: (DispatchQueue::entry()+0x60f) [0x7f4ac3107cf9] 2024-07-26T06:57:03.213 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4ac31e45a1] 2024-07-26T06:57:03.213 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 20: (Thread::entry_wrapper()+0x43) [0x7f4ac2f91cdd] 2024-07-26T06:57:03.213 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4ac2f91cf9] 2024-07-26T06:57:03.213 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4ac2885c02] 2024-07-26T06:57:03.213 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4ac290ac40] 2024-07-26T06:57:03.213 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:03.213 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:02 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[70761]: 2024-07-26T06:57:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:02 smithi164 ceph-mon[36970]: pgmap v614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-07-26T06:57:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:02 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/2760283174,v1:172.21.15.163:6835/2760283174] up:rejoin 2024-07-26T06:57:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:02 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/2332407499,v1:172.21.15.163:6837/2332407499] up:rejoin 2024-07-26T06:57:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:02 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/2567296059,v1:172.21.15.163:6839/2567296059] up:rejoin 2024-07-26T06:57:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:02 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:57:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:02 smithi164 ceph-mon[36970]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-07-26T06:57:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:02 smithi164 ceph-mon[36970]: daemon mds.b is now active in filesystem cephfs as rank 4 2024-07-26T06:57:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:02 smithi164 ceph-mon[36970]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-07-26T06:57:04.007 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:03 smithi163 podman[70854]: 2024-07-26 06:57:03.703099464 +0000 UTC m=+0.853068851 container died c511a9857c5805705d77cd227cd8b73652aea95691724568135033e092af9b8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS) 2024-07-26T06:57:04.007 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:03 smithi163 podman[70853]: 2024-07-26 06:57:03.62151619 +0000 UTC m=+0.771708507 container remove e30626c9e226c7d2232311324bfa47293dcaaab4427c823e2765455fd1cb3f19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T06:57:04.007 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:03 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:57:04.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:03 smithi163 ceph-mon[36873]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:57:04.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:03 smithi163 ceph-mon[36873]: Cluster is now healthy 2024-07-26T06:57:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:03 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/2567296059,v1:172.21.15.163:6839/2567296059] up:active 2024-07-26T06:57:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:03 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/2760283174,v1:172.21.15.163:6835/2760283174] up:active 2024-07-26T06:57:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:03 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/2332407499,v1:172.21.15.163:6837/2332407499] up:active 2024-07-26T06:57:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:03 smithi163 ceph-mon[36873]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:57:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:03 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:57:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:03 smithi042 ceph-mon[32775]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:57:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:03 smithi042 ceph-mon[32775]: Cluster is now healthy 2024-07-26T06:57:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:03 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/2567296059,v1:172.21.15.163:6839/2567296059] up:active 2024-07-26T06:57:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:03 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/2760283174,v1:172.21.15.163:6835/2760283174] up:active 2024-07-26T06:57:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:03 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/2332407499,v1:172.21.15.163:6837/2332407499] up:active 2024-07-26T06:57:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:03 smithi042 ceph-mon[32775]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:57:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:03 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:57:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:03 smithi164 ceph-mon[36970]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:57:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:03 smithi164 ceph-mon[36970]: Cluster is now healthy 2024-07-26T06:57:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:03 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/2567296059,v1:172.21.15.163:6839/2567296059] up:active 2024-07-26T06:57:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:03 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/2760283174,v1:172.21.15.163:6835/2760283174] up:active 2024-07-26T06:57:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:03 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/2332407499,v1:172.21.15.163:6837/2332407499] up:active 2024-07-26T06:57:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:03 smithi164 ceph-mon[36970]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:57:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:03 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:57:04.757 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:04 smithi163 podman[70889]: 2024-07-26 06:57:04.390561145 +0000 UTC m=+0.734622832 container died ba88b2e97fb9f381bec2797db9c8f98408a62c2c7333486a72e8f0786cedc41b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-07-26T06:57:04.757 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:04 smithi163 podman[70854]: 2024-07-26 06:57:04.352069276 +0000 UTC m=+1.502038643 container remove c511a9857c5805705d77cd227cd8b73652aea95691724568135033e092af9b8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T06:57:04.757 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:04 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:57:05.124 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:04 smithi163 ceph-mon[36873]: pgmap v615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T06:57:05.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:04 smithi042 ceph-mon[32775]: pgmap v615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T06:57:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:04 smithi164 ceph-mon[36970]: pgmap v615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T06:57:05.507 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-26T06:57:05.507 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T06:57:05.507 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Consumed 1.064s CPU time. 2024-07-26T06:57:05.507 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:05 smithi163 podman[70855]: 2024-07-26 06:57:05.124330721 +0000 UTC m=+2.269614577 container remove ba88b2e97fb9f381bec2797db9c8f98408a62c2c7333486a72e8f0786cedc41b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True) 2024-07-26T06:57:05.507 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:57:05.508 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T06:57:05.508 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Consumed 1.036s CPU time. 2024-07-26T06:57:07.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:06 smithi042 ceph-mon[32775]: pgmap v616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 514 B/s wr, 13 op/s 2024-07-26T06:57:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:06 smithi164 ceph-mon[36970]: pgmap v616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 514 B/s wr, 13 op/s 2024-07-26T06:57:07.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:06 smithi163 ceph-mon[36873]: pgmap v616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 514 B/s wr, 13 op/s 2024-07-26T06:57:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:57:08.300+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:57:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:57:08.300+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:57:08.301 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:57:08.300+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:57:08.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:57:08.702+0000 7faa2dcb9640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 msgr2=0x561ca20d0cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:57:08.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:57:08.702+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:57:08.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:57:08.702+0000 7faa117fa640 1 client.16095 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:57:09.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:08 smithi042 ceph-mon[32775]: pgmap v617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 665 B/s wr, 14 op/s 2024-07-26T06:57:09.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:08 smithi164 ceph-mon[36970]: pgmap v617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 665 B/s wr, 14 op/s 2024-07-26T06:57:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:08 smithi163 ceph-mon[36873]: pgmap v617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 665 B/s wr, 14 op/s 2024-07-26T06:57:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:10 smithi042 ceph-mon[32775]: pgmap v618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 571 B/s wr, 14 op/s 2024-07-26T06:57:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:10 smithi164 ceph-mon[36970]: pgmap v618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 571 B/s wr, 14 op/s 2024-07-26T06:57:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:10 smithi163 ceph-mon[36873]: pgmap v618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 571 B/s wr, 14 op/s 2024-07-26T06:57:13.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:12 smithi042 ceph-mon[32775]: pgmap v619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T06:57:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:57:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:12 smithi164 ceph-mon[36970]: pgmap v619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T06:57:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:57:13.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:12 smithi163 ceph-mon[36873]: pgmap v619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T06:57:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:57:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:57:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:57:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:57:14.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:57:14.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:57:14.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:57:14.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:57:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:57:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:57:15.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:14 smithi042 ceph-mon[32775]: pgmap v620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 12 op/s 2024-07-26T06:57:15.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:14 smithi164 ceph-mon[36970]: pgmap v620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 12 op/s 2024-07-26T06:57:15.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:14 smithi163 ceph-mon[36873]: pgmap v620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 12 op/s 2024-07-26T06:57:15.507 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:15 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 3. 2024-07-26T06:57:15.507 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:15 smithi163 systemd[1]: Stopped Ceph mds.b for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:15.508 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:15 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 3. 2024-07-26T06:57:15.508 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:15 smithi163 systemd[1]: Stopped Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:15.508 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:15 smithi163 systemd[1]: Starting Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:57:15.508 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:15 smithi163 podman[71205]: 2024-07-26 06:57:15.404362411 +0000 UTC m=+0.018182460 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:57:15.508 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:15 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 3. 2024-07-26T06:57:15.508 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:15 smithi163 systemd[1]: Stopped Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:15.508 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:15 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Consumed 1.064s CPU time. 2024-07-26T06:57:15.509 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:15 smithi163 systemd[1]: Starting Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:57:15.813 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:15 smithi163 podman[71211]: 2024-07-26 06:57:15.414633662 +0000 UTC m=+0.018751257 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:57:15.813 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:15 smithi163 podman[71205]: 2024-07-26 06:57:15.542786851 +0000 UTC m=+0.156606890 container create 439da4e38b4aa7fb023fa80fbb154c22db751d2014f4cfe96d9e41ac9f4f4cff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.build-date=20240724) 2024-07-26T06:57:15.814 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:15 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Consumed 1.036s CPU time. 2024-07-26T06:57:15.814 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:15 smithi163 systemd[1]: Starting Ceph mds.b for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:57:16.129 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:15 smithi163 podman[71211]: 2024-07-26 06:57:15.813544451 +0000 UTC m=+0.417662031 container create 57543711caff8acd1bab1301fdebdf5eb4a4a9ee66881151a7eaf86e1e76d555 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T06:57:16.129 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:15 smithi163 podman[71205]: 2024-07-26 06:57:15.913011579 +0000 UTC m=+0.526831619 container init 439da4e38b4aa7fb023fa80fbb154c22db751d2014f4cfe96d9e41ac9f4f4cff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T06:57:16.129 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:15 smithi163 podman[71205]: 2024-07-26 06:57:15.915736996 +0000 UTC m=+0.529557036 container start 439da4e38b4aa7fb023fa80fbb154c22db751d2014f4cfe96d9e41ac9f4f4cff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, ceph=True) 2024-07-26T06:57:16.129 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:15 smithi163 ceph-mds[71261]: -- 172.21.15.163:0/952108651 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b0c5c87c20 con 0x55b0c9e29a80 2024-07-26T06:57:16.130 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:15 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: starting mds.h at 2024-07-26T06:57:16.130 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:16 smithi163 bash[71205]: 439da4e38b4aa7fb023fa80fbb154c22db751d2014f4cfe96d9e41ac9f4f4cff 2024-07-26T06:57:16.130 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:16 smithi163 systemd[1]: Started Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:16.575 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:16 smithi163 podman[71211]: 2024-07-26 06:57:16.274730175 +0000 UTC m=+0.878847758 container init 57543711caff8acd1bab1301fdebdf5eb4a4a9ee66881151a7eaf86e1e76d555 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-07-26T06:57:16.576 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:16 smithi163 podman[71211]: 2024-07-26 06:57:16.27792634 +0000 UTC m=+0.882043929 container start 57543711caff8acd1bab1301fdebdf5eb4a4a9ee66881151a7eaf86e1e76d555 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD) 2024-07-26T06:57:16.576 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mds[71288]: -- 172.21.15.163:0/680099876 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55563ee3d860 con 0x555642fdfa80 2024-07-26T06:57:16.576 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: starting mds.e at 2024-07-26T06:57:16.576 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:16 smithi163 bash[71211]: 57543711caff8acd1bab1301fdebdf5eb4a4a9ee66881151a7eaf86e1e76d555 2024-07-26T06:57:16.576 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:16 smithi163 systemd[1]: Started Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:16.854 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:16 smithi163 podman[71338]: 2024-07-26 06:57:16.478497009 +0000 UTC m=+0.019696297 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:57:16.854 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:16 smithi163 podman[71338]: 2024-07-26 06:57:16.609726359 +0000 UTC m=+0.150925635 container create 390e1aebaa15489b885e957e73b09c261d81176a6489d10fba2d9fd1a28c26ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T06:57:16.855 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:16 smithi163 podman[71338]: 2024-07-26 06:57:16.8543829 +0000 UTC m=+0.395582185 container init 390e1aebaa15489b885e957e73b09c261d81176a6489d10fba2d9fd1a28c26ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724) 2024-07-26T06:57:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:16 smithi042 ceph-mon[32775]: pgmap v621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 426 B/s wr, 12 op/s 2024-07-26T06:57:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:16 smithi042 ceph-mon[32775]: daemon mds.h restarted 2024-07-26T06:57:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:16 smithi042 ceph-mon[32775]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:57:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:16 smithi042 ceph-mon[32775]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T06:57:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:16 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/4111555088,v1:172.21.15.163:6835/4111555088] up:boot 2024-07-26T06:57:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:16 smithi042 ceph-mon[32775]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-07-26T06:57:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:16 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=e=up:active,4=b=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:57:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:57:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:16 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:57:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:16 smithi042 ceph-mon[32775]: daemon mds.e restarted 2024-07-26T06:57:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:16 smithi042 ceph-mon[32775]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T06:57:17.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:16 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/850726105,v1:172.21.15.163:6837/850726105] up:boot 2024-07-26T06:57:17.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:16 smithi042 ceph-mon[32775]: Standby daemon mds.a assigned to filesystem cephfs as rank 3 2024-07-26T06:57:17.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:16 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=l=up:replay,4=b=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:57:17.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:57:17.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:16 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-07-26T06:57:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:16 smithi164 ceph-mon[36970]: pgmap v621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 426 B/s wr, 12 op/s 2024-07-26T06:57:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:16 smithi164 ceph-mon[36970]: daemon mds.h restarted 2024-07-26T06:57:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:16 smithi164 ceph-mon[36970]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:57:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:16 smithi164 ceph-mon[36970]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T06:57:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:16 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/4111555088,v1:172.21.15.163:6835/4111555088] up:boot 2024-07-26T06:57:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:16 smithi164 ceph-mon[36970]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-07-26T06:57:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:16 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=e=up:active,4=b=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:57:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:57:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:16 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:57:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:16 smithi164 ceph-mon[36970]: daemon mds.e restarted 2024-07-26T06:57:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:16 smithi164 ceph-mon[36970]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T06:57:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:16 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/850726105,v1:172.21.15.163:6837/850726105] up:boot 2024-07-26T06:57:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:16 smithi164 ceph-mon[36970]: Standby daemon mds.a assigned to filesystem cephfs as rank 3 2024-07-26T06:57:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:16 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=l=up:replay,4=b=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:57:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:57:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:16 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-07-26T06:57:17.257 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:16 smithi163 podman[71338]: 2024-07-26 06:57:16.857448223 +0000 UTC m=+0.398647496 container start 390e1aebaa15489b885e957e73b09c261d81176a6489d10fba2d9fd1a28c26ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-07-26T06:57:17.257 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mds[71359]: -- 172.21.15.163:0/3568445058 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564ed9163860 con 0x564edd305a80 2024-07-26T06:57:17.257 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: starting mds.b at 2024-07-26T06:57:17.257 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:16 smithi163 bash[71338]: 390e1aebaa15489b885e957e73b09c261d81176a6489d10fba2d9fd1a28c26ca 2024-07-26T06:57:17.257 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:16 smithi163 systemd[1]: Started Ceph mds.b for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mon[36873]: pgmap v621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 426 B/s wr, 12 op/s 2024-07-26T06:57:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mon[36873]: daemon mds.h restarted 2024-07-26T06:57:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mon[36873]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:57:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mon[36873]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T06:57:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/4111555088,v1:172.21.15.163:6835/4111555088] up:boot 2024-07-26T06:57:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mon[36873]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-07-26T06:57:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=e=up:active,4=b=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:57:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:57:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:57:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mon[36873]: daemon mds.e restarted 2024-07-26T06:57:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mon[36873]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T06:57:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/850726105,v1:172.21.15.163:6837/850726105] up:boot 2024-07-26T06:57:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mon[36873]: Standby daemon mds.a assigned to filesystem cephfs as rank 3 2024-07-26T06:57:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=l=up:replay,4=b=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:57:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:57:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:16 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-07-26T06:57:18.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:17 smithi042 ceph-mon[32775]: daemon mds.b restarted 2024-07-26T06:57:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:17 smithi042 ceph-mon[32775]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T06:57:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:17 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/734940828,v1:172.21.15.163:6839/734940828] up:boot 2024-07-26T06:57:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:17 smithi042 ceph-mon[32775]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-07-26T06:57:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:17 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=l=up:replay,3=a=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:57:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T06:57:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:17 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:replay 2 up:active 2024-07-26T06:57:18.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:17 smithi164 ceph-mon[36970]: daemon mds.b restarted 2024-07-26T06:57:18.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:17 smithi164 ceph-mon[36970]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T06:57:18.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:17 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/734940828,v1:172.21.15.163:6839/734940828] up:boot 2024-07-26T06:57:18.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:17 smithi164 ceph-mon[36970]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-07-26T06:57:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:17 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=l=up:replay,3=a=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:57:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T06:57:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:17 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:replay 2 up:active 2024-07-26T06:57:18.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:17 smithi163 ceph-mon[36873]: daemon mds.b restarted 2024-07-26T06:57:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:17 smithi163 ceph-mon[36873]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T06:57:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:17 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/734940828,v1:172.21.15.163:6839/734940828] up:boot 2024-07-26T06:57:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:17 smithi163 ceph-mon[36873]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-07-26T06:57:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:17 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=l=up:replay,3=a=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:57:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T06:57:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:17 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:replay 2 up:active 2024-07-26T06:57:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:18 smithi164 ceph-mon[36970]: pgmap v625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T06:57:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:18 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3995951481,v1:172.21.15.42:6839/3995951481] up:resolve 2024-07-26T06:57:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:18 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/2505332261,v1:172.21.15.42:6841/2505332261] up:resolve 2024-07-26T06:57:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:18 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/2696987957,v1:172.21.15.164:6833/2696987957] up:resolve 2024-07-26T06:57:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:18 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:57:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:18 smithi163 ceph-mon[36873]: pgmap v625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T06:57:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:18 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3995951481,v1:172.21.15.42:6839/3995951481] up:resolve 2024-07-26T06:57:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:18 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/2505332261,v1:172.21.15.42:6841/2505332261] up:resolve 2024-07-26T06:57:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:18 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/2696987957,v1:172.21.15.164:6833/2696987957] up:resolve 2024-07-26T06:57:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:18 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:57:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:18 smithi042 ceph-mon[32775]: pgmap v625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T06:57:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:18 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3995951481,v1:172.21.15.42:6839/3995951481] up:resolve 2024-07-26T06:57:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:18 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/2505332261,v1:172.21.15.42:6841/2505332261] up:resolve 2024-07-26T06:57:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:18 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/2696987957,v1:172.21.15.164:6833/2696987957] up:resolve 2024-07-26T06:57:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:18 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:57:20.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:19 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00199998 2024-07-26T06:57:20.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:19 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00199998 2024-07-26T06:57:20.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:19 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999983 2024-07-26T06:57:20.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:19 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/2696987957,v1:172.21.15.164:6833/2696987957] up:reconnect 2024-07-26T06:57:20.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:19 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3995951481,v1:172.21.15.42:6839/3995951481] up:reconnect 2024-07-26T06:57:20.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:19 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/2505332261,v1:172.21.15.42:6841/2505332261] up:reconnect 2024-07-26T06:57:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:19 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:57:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:19 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/2505332261,v1:172.21.15.42:6841/2505332261] up:rejoin 2024-07-26T06:57:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:19 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3995951481,v1:172.21.15.42:6839/3995951481] up:rejoin 2024-07-26T06:57:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:19 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/2696987957,v1:172.21.15.164:6833/2696987957] up:rejoin 2024-07-26T06:57:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:19 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:57:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:19 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00199998 2024-07-26T06:57:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:19 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00199998 2024-07-26T06:57:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:19 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999983 2024-07-26T06:57:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:19 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/2696987957,v1:172.21.15.164:6833/2696987957] up:reconnect 2024-07-26T06:57:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:19 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3995951481,v1:172.21.15.42:6839/3995951481] up:reconnect 2024-07-26T06:57:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:19 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/2505332261,v1:172.21.15.42:6841/2505332261] up:reconnect 2024-07-26T06:57:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:19 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:57:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:19 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/2505332261,v1:172.21.15.42:6841/2505332261] up:rejoin 2024-07-26T06:57:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:19 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3995951481,v1:172.21.15.42:6839/3995951481] up:rejoin 2024-07-26T06:57:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:19 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/2696987957,v1:172.21.15.164:6833/2696987957] up:rejoin 2024-07-26T06:57:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:19 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:57:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:19 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00199998 2024-07-26T06:57:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:19 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.00199998 2024-07-26T06:57:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:19 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999983 2024-07-26T06:57:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:19 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/2696987957,v1:172.21.15.164:6833/2696987957] up:reconnect 2024-07-26T06:57:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:19 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3995951481,v1:172.21.15.42:6839/3995951481] up:reconnect 2024-07-26T06:57:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:19 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/2505332261,v1:172.21.15.42:6841/2505332261] up:reconnect 2024-07-26T06:57:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:19 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:57:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:19 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/2505332261,v1:172.21.15.42:6841/2505332261] up:rejoin 2024-07-26T06:57:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:19 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3995951481,v1:172.21.15.42:6839/3995951481] up:rejoin 2024-07-26T06:57:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:19 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/2696987957,v1:172.21.15.164:6833/2696987957] up:rejoin 2024-07-26T06:57:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:19 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:57:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-mon[36970]: pgmap v626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 13 op/s 2024-07-26T06:57:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-mon[36970]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:57:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-mon[36970]: daemon mds.a is now active in filesystem cephfs as rank 3 2024-07-26T06:57:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-mon[36970]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-07-26T06:57:21.223 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:57:21.223 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: *** Caught signal (Aborted) ** 2024-07-26T06:57:21.223 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: in thread 7f4fe154e640 thread_name:ms_dispatch 2024-07-26T06:57:21.223 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:21.223 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4fe69256f0] 2024-07-26T06:57:21.223 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4fe697294c] 2024-07-26T06:57:21.224 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 3: raise() 2024-07-26T06:57:21.224 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 4: abort() 2024-07-26T06:57:21.224 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4fe690f71b] 2024-07-26T06:57:21.224 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 6: /lib64/libc.so.6(+0x37386) [0x7f4fe691e386] 2024-07-26T06:57:21.224 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559bed9426b9] 2024-07-26T06:57:21.224 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559bedb0080c] 2024-07-26T06:57:21.224 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559beda402ca] 2024-07-26T06:57:21.224 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559beda10753] 2024-07-26T06:57:21.224 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559beda11a1b] 2024-07-26T06:57:21.224 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559bed9f5752] 2024-07-26T06:57:21.225 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559bed943a56] 2024-07-26T06:57:21.225 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559bed947637] 2024-07-26T06:57:21.225 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559bed947f5e] 2024-07-26T06:57:21.225 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559bed92b03a] 2024-07-26T06:57:21.225 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4fe71f2407] 2024-07-26T06:57:21.225 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 18: (DispatchQueue::entry()+0x60f) [0x7f4fe71f2cf9] 2024-07-26T06:57:21.225 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4fe72cf5a1] 2024-07-26T06:57:21.225 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 20: (Thread::entry_wrapper()+0x43) [0x7f4fe707ccdd] 2024-07-26T06:57:21.225 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4fe707ccf9] 2024-07-26T06:57:21.225 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4fe6970c02] 2024-07-26T06:57:21.225 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4fe69f5c40] 2024-07-26T06:57:21.225 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 2024-07-26T06:57:20.942+0000 7f4fe154e640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:21.226 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: in thread 7f4fe154e640 thread_name:ms_dispatch 2024-07-26T06:57:21.226 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 2024-07-26T06:57:21.226 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:21.226 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4fe69256f0] 2024-07-26T06:57:21.226 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4fe697294c] 2024-07-26T06:57:21.226 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 3: raise() 2024-07-26T06:57:21.226 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 4: abort() 2024-07-26T06:57:21.226 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4fe690f71b] 2024-07-26T06:57:21.226 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 6: /lib64/libc.so.6(+0x37386) [0x7f4fe691e386] 2024-07-26T06:57:21.226 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559bed9426b9] 2024-07-26T06:57:21.226 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559bedb0080c] 2024-07-26T06:57:21.226 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559beda402ca] 2024-07-26T06:57:21.226 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559beda10753] 2024-07-26T06:57:21.227 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559beda11a1b] 2024-07-26T06:57:21.227 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559bed9f5752] 2024-07-26T06:57:21.227 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559bed943a56] 2024-07-26T06:57:21.227 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559bed947637] 2024-07-26T06:57:21.227 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559bed947f5e] 2024-07-26T06:57:21.227 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559bed92b03a] 2024-07-26T06:57:21.227 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4fe71f2407] 2024-07-26T06:57:21.227 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 18: (DispatchQueue::entry()+0x60f) [0x7f4fe71f2cf9] 2024-07-26T06:57:21.227 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4fe72cf5a1] 2024-07-26T06:57:21.228 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 20: (Thread::entry_wrapper()+0x43) [0x7f4fe707ccdd] 2024-07-26T06:57:21.228 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4fe707ccf9] 2024-07-26T06:57:21.228 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4fe6970c02] 2024-07-26T06:57:21.228 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4fe69f5c40] 2024-07-26T06:57:21.229 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:21.229 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 2024-07-26T06:57:21.229 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 0> 2024-07-26T06:57:20.942+0000 7f4fe154e640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:21.229 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: in thread 7f4fe154e640 thread_name:ms_dispatch 2024-07-26T06:57:21.229 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 2024-07-26T06:57:21.229 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:21.229 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4fe69256f0] 2024-07-26T06:57:21.229 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4fe697294c] 2024-07-26T06:57:21.229 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 3: raise() 2024-07-26T06:57:21.230 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 4: abort() 2024-07-26T06:57:21.230 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4fe690f71b] 2024-07-26T06:57:21.230 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 6: /lib64/libc.so.6(+0x37386) [0x7f4fe691e386] 2024-07-26T06:57:21.230 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559bed9426b9] 2024-07-26T06:57:21.230 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559bedb0080c] 2024-07-26T06:57:21.230 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559beda402ca] 2024-07-26T06:57:21.230 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559beda10753] 2024-07-26T06:57:21.230 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559beda11a1b] 2024-07-26T06:57:21.230 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559bed9f5752] 2024-07-26T06:57:21.230 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559bed943a56] 2024-07-26T06:57:21.230 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559bed947637] 2024-07-26T06:57:21.230 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559bed947f5e] 2024-07-26T06:57:21.231 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559bed92b03a] 2024-07-26T06:57:21.231 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4fe71f2407] 2024-07-26T06:57:21.231 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 18: (DispatchQueue::entry()+0x60f) [0x7f4fe71f2cf9] 2024-07-26T06:57:21.231 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4fe72cf5a1] 2024-07-26T06:57:21.231 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 20: (Thread::entry_wrapper()+0x43) [0x7f4fe707ccdd] 2024-07-26T06:57:21.231 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4fe707ccf9] 2024-07-26T06:57:21.231 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4fe6970c02] 2024-07-26T06:57:21.231 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4fe69f5c40] 2024-07-26T06:57:21.231 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:21.231 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 2024-07-26T06:57:21.231 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 0> 2024-07-26T06:57:20.942+0000 7f4fe154e640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:21.231 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: in thread 7f4fe154e640 thread_name:ms_dispatch 2024-07-26T06:57:21.232 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 2024-07-26T06:57:21.232 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:21.232 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4fe69256f0] 2024-07-26T06:57:21.232 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4fe697294c] 2024-07-26T06:57:21.232 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 3: raise() 2024-07-26T06:57:21.232 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 4: abort() 2024-07-26T06:57:21.232 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4fe690f71b] 2024-07-26T06:57:21.232 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 6: /lib64/libc.so.6(+0x37386) [0x7f4fe691e386] 2024-07-26T06:57:21.232 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559bed9426b9] 2024-07-26T06:57:21.232 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559bedb0080c] 2024-07-26T06:57:21.232 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559beda402ca] 2024-07-26T06:57:21.232 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559beda10753] 2024-07-26T06:57:21.232 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559beda11a1b] 2024-07-26T06:57:21.233 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559bed9f5752] 2024-07-26T06:57:21.233 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559bed943a56] 2024-07-26T06:57:21.233 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559bed947637] 2024-07-26T06:57:21.233 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559bed947f5e] 2024-07-26T06:57:21.233 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559bed92b03a] 2024-07-26T06:57:21.233 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4fe71f2407] 2024-07-26T06:57:21.233 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 18: (DispatchQueue::entry()+0x60f) [0x7f4fe71f2cf9] 2024-07-26T06:57:21.233 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4fe72cf5a1] 2024-07-26T06:57:21.233 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 20: (Thread::entry_wrapper()+0x43) [0x7f4fe707ccdd] 2024-07-26T06:57:21.233 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4fe707ccf9] 2024-07-26T06:57:21.233 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4fe6970c02] 2024-07-26T06:57:21.233 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4fe69f5c40] 2024-07-26T06:57:21.234 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:21.234 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70685]: 2024-07-26T06:57:21.234 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:20 smithi164 podman[70852]: 2024-07-26 06:57:20.999598892 +0000 UTC m=+0.018897169 container died 05c159eaa36fa59ba074a0fdb1e433cfd52662120704644479d9d1c34b0ac93f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-07-26T06:57:21.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:20 smithi163 ceph-mon[36873]: pgmap v626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 13 op/s 2024-07-26T06:57:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:20 smithi163 ceph-mon[36873]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:57:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:20 smithi163 ceph-mon[36873]: daemon mds.a is now active in filesystem cephfs as rank 3 2024-07-26T06:57:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:20 smithi163 ceph-mon[36873]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-07-26T06:57:21.306 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-mon[32775]: pgmap v626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 13 op/s 2024-07-26T06:57:21.306 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-mon[32775]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:57:21.306 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-mon[32775]: daemon mds.a is now active in filesystem cephfs as rank 3 2024-07-26T06:57:21.306 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-mon[32775]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-07-26T06:57:21.306 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:57:21.307 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: *** Caught signal (Aborted) ** 2024-07-26T06:57:21.307 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: in thread 7fba13b86640 thread_name:ms_dispatch 2024-07-26T06:57:21.307 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:21.307 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fba18f5d6f0] 2024-07-26T06:57:21.307 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fba18faa94c] 2024-07-26T06:57:21.307 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 3: raise() 2024-07-26T06:57:21.307 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 4: abort() 2024-07-26T06:57:21.307 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 5: /lib64/libc.so.6(+0x2871b) [0x7fba18f4771b] 2024-07-26T06:57:21.307 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 6: /lib64/libc.so.6(+0x37386) [0x7fba18f56386] 2024-07-26T06:57:21.307 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560bca10e6b9] 2024-07-26T06:57:21.307 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560bca2cc80c] 2024-07-26T06:57:21.308 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560bca20c2ca] 2024-07-26T06:57:21.308 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560bca1dc753] 2024-07-26T06:57:21.308 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560bca1dda1b] 2024-07-26T06:57:21.308 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560bca1c1752] 2024-07-26T06:57:21.308 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560bca10fa56] 2024-07-26T06:57:21.308 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560bca113637] 2024-07-26T06:57:21.308 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560bca113f5e] 2024-07-26T06:57:21.308 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560bca0f703a] 2024-07-26T06:57:21.308 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fba1982a407] 2024-07-26T06:57:21.308 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 18: (DispatchQueue::entry()+0x60f) [0x7fba1982acf9] 2024-07-26T06:57:21.308 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fba199075a1] 2024-07-26T06:57:21.309 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:57:21.309 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: *** Caught signal (Aborted) ** 2024-07-26T06:57:21.309 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: in thread 7f80bb34b640 thread_name:ms_dispatch 2024-07-26T06:57:21.309 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:21.309 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f80c07226f0] 2024-07-26T06:57:21.309 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f80c076f94c] 2024-07-26T06:57:21.309 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 3: raise() 2024-07-26T06:57:21.309 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 4: abort() 2024-07-26T06:57:21.309 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 5: /lib64/libc.so.6(+0x2871b) [0x7f80c070c71b] 2024-07-26T06:57:21.309 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 6: /lib64/libc.so.6(+0x37386) [0x7f80c071b386] 2024-07-26T06:57:21.310 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c6a1a0a6b9] 2024-07-26T06:57:21.310 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c6a1bc880c] 2024-07-26T06:57:21.310 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c6a1b082ca] 2024-07-26T06:57:21.310 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c6a1ad8753] 2024-07-26T06:57:21.310 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c6a1ad9a1b] 2024-07-26T06:57:21.310 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c6a1abd752] 2024-07-26T06:57:21.310 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c6a1a0ba56] 2024-07-26T06:57:21.310 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c6a1a0f637] 2024-07-26T06:57:21.310 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c6a1a0ff5e] 2024-07-26T06:57:21.310 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c6a19f303a] 2024-07-26T06:57:21.310 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f80c0fef407] 2024-07-26T06:57:21.310 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 18: (DispatchQueue::entry()+0x60f) [0x7f80c0fefcf9] 2024-07-26T06:57:21.311 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f80c10cc5a1] 2024-07-26T06:57:21.311 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 20: (Thread::entry_wrapper()+0x43) [0x7fba196b4cdd] 2024-07-26T06:57:21.311 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 21: (Thread::_entry_func(void*)+0xd) [0x7fba196b4cf9] 2024-07-26T06:57:21.311 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 22: /lib64/libc.so.6(+0x89c02) [0x7fba18fa8c02] 2024-07-26T06:57:21.311 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fba1902dc40] 2024-07-26T06:57:21.311 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 2024-07-26T06:57:20.944+0000 7fba13b86640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:21.311 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: in thread 7fba13b86640 thread_name:ms_dispatch 2024-07-26T06:57:21.311 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 2024-07-26T06:57:21.311 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:21.311 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fba18f5d6f0] 2024-07-26T06:57:21.311 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fba18faa94c] 2024-07-26T06:57:21.312 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 3: raise() 2024-07-26T06:57:21.312 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 4: abort() 2024-07-26T06:57:21.312 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 5: /lib64/libc.so.6(+0x2871b) [0x7fba18f4771b] 2024-07-26T06:57:21.312 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 6: /lib64/libc.so.6(+0x37386) [0x7fba18f56386] 2024-07-26T06:57:21.312 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560bca10e6b9] 2024-07-26T06:57:21.312 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560bca2cc80c] 2024-07-26T06:57:21.312 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560bca20c2ca] 2024-07-26T06:57:21.312 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560bca1dc753] 2024-07-26T06:57:21.312 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560bca1dda1b] 2024-07-26T06:57:21.312 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560bca1c1752] 2024-07-26T06:57:21.312 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560bca10fa56] 2024-07-26T06:57:21.312 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560bca113637] 2024-07-26T06:57:21.312 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560bca113f5e] 2024-07-26T06:57:21.313 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560bca0f703a] 2024-07-26T06:57:21.313 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fba1982a407] 2024-07-26T06:57:21.313 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 18: (DispatchQueue::entry()+0x60f) [0x7fba1982acf9] 2024-07-26T06:57:21.313 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fba199075a1] 2024-07-26T06:57:21.313 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 20: (Thread::entry_wrapper()+0x43) [0x7f80c0e79cdd] 2024-07-26T06:57:21.313 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 21: (Thread::_entry_func(void*)+0xd) [0x7f80c0e79cf9] 2024-07-26T06:57:21.313 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 22: /lib64/libc.so.6(+0x89c02) [0x7f80c076dc02] 2024-07-26T06:57:21.313 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f80c07f2c40] 2024-07-26T06:57:21.313 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 2024-07-26T06:57:20.944+0000 7f80bb34b640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:21.314 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: in thread 7f80bb34b640 thread_name:ms_dispatch 2024-07-26T06:57:21.314 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 2024-07-26T06:57:21.314 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:21.314 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f80c07226f0] 2024-07-26T06:57:21.314 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f80c076f94c] 2024-07-26T06:57:21.314 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 3: raise() 2024-07-26T06:57:21.314 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 4: abort() 2024-07-26T06:57:21.314 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 5: /lib64/libc.so.6(+0x2871b) [0x7f80c070c71b] 2024-07-26T06:57:21.314 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 6: /lib64/libc.so.6(+0x37386) [0x7f80c071b386] 2024-07-26T06:57:21.314 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c6a1a0a6b9] 2024-07-26T06:57:21.314 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c6a1bc880c] 2024-07-26T06:57:21.314 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c6a1b082ca] 2024-07-26T06:57:21.315 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c6a1ad8753] 2024-07-26T06:57:21.315 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c6a1ad9a1b] 2024-07-26T06:57:21.315 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c6a1abd752] 2024-07-26T06:57:21.315 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c6a1a0ba56] 2024-07-26T06:57:21.315 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c6a1a0f637] 2024-07-26T06:57:21.315 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c6a1a0ff5e] 2024-07-26T06:57:21.315 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c6a19f303a] 2024-07-26T06:57:21.315 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f80c0fef407] 2024-07-26T06:57:21.315 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 18: (DispatchQueue::entry()+0x60f) [0x7f80c0fefcf9] 2024-07-26T06:57:21.315 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f80c10cc5a1] 2024-07-26T06:57:21.316 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 20: (Thread::entry_wrapper()+0x43) [0x7fba196b4cdd] 2024-07-26T06:57:21.316 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 21: (Thread::_entry_func(void*)+0xd) [0x7fba196b4cf9] 2024-07-26T06:57:21.316 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 22: /lib64/libc.so.6(+0x89c02) [0x7fba18fa8c02] 2024-07-26T06:57:21.316 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fba1902dc40] 2024-07-26T06:57:21.316 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:21.316 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 2024-07-26T06:57:21.316 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 0> 2024-07-26T06:57:20.944+0000 7fba13b86640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:21.316 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: in thread 7fba13b86640 thread_name:ms_dispatch 2024-07-26T06:57:21.316 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 2024-07-26T06:57:21.316 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:21.316 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fba18f5d6f0] 2024-07-26T06:57:21.316 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fba18faa94c] 2024-07-26T06:57:21.317 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 3: raise() 2024-07-26T06:57:21.317 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 4: abort() 2024-07-26T06:57:21.317 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 5: /lib64/libc.so.6(+0x2871b) [0x7fba18f4771b] 2024-07-26T06:57:21.317 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 6: /lib64/libc.so.6(+0x37386) [0x7fba18f56386] 2024-07-26T06:57:21.317 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560bca10e6b9] 2024-07-26T06:57:21.317 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560bca2cc80c] 2024-07-26T06:57:21.317 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560bca20c2ca] 2024-07-26T06:57:21.317 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560bca1dc753] 2024-07-26T06:57:21.317 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560bca1dda1b] 2024-07-26T06:57:21.317 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560bca1c1752] 2024-07-26T06:57:21.317 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560bca10fa56] 2024-07-26T06:57:21.317 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560bca113637] 2024-07-26T06:57:21.317 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560bca113f5e] 2024-07-26T06:57:21.318 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560bca0f703a] 2024-07-26T06:57:21.318 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 20: (Thread::entry_wrapper()+0x43) [0x7f80c0e79cdd] 2024-07-26T06:57:21.318 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 21: (Thread::_entry_func(void*)+0xd) [0x7f80c0e79cf9] 2024-07-26T06:57:21.318 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 22: /lib64/libc.so.6(+0x89c02) [0x7f80c076dc02] 2024-07-26T06:57:21.318 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f80c07f2c40] 2024-07-26T06:57:21.318 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:21.318 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 2024-07-26T06:57:21.318 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 0> 2024-07-26T06:57:20.944+0000 7f80bb34b640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:21.318 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: in thread 7f80bb34b640 thread_name:ms_dispatch 2024-07-26T06:57:21.318 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 2024-07-26T06:57:21.319 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:21.319 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f80c07226f0] 2024-07-26T06:57:21.319 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f80c076f94c] 2024-07-26T06:57:21.319 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 3: raise() 2024-07-26T06:57:21.319 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 4: abort() 2024-07-26T06:57:21.319 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 5: /lib64/libc.so.6(+0x2871b) [0x7f80c070c71b] 2024-07-26T06:57:21.319 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 6: /lib64/libc.so.6(+0x37386) [0x7f80c071b386] 2024-07-26T06:57:21.319 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c6a1a0a6b9] 2024-07-26T06:57:21.319 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c6a1bc880c] 2024-07-26T06:57:21.319 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c6a1b082ca] 2024-07-26T06:57:21.319 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c6a1ad8753] 2024-07-26T06:57:21.319 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c6a1ad9a1b] 2024-07-26T06:57:21.319 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c6a1abd752] 2024-07-26T06:57:21.320 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c6a1a0ba56] 2024-07-26T06:57:21.320 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c6a1a0f637] 2024-07-26T06:57:21.320 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c6a1a0ff5e] 2024-07-26T06:57:21.320 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c6a19f303a] 2024-07-26T06:57:21.320 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fba1982a407] 2024-07-26T06:57:21.320 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 18: (DispatchQueue::entry()+0x60f) [0x7fba1982acf9] 2024-07-26T06:57:21.320 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fba199075a1] 2024-07-26T06:57:21.320 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 20: (Thread::entry_wrapper()+0x43) [0x7fba196b4cdd] 2024-07-26T06:57:21.321 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 21: (Thread::_entry_func(void*)+0xd) [0x7fba196b4cf9] 2024-07-26T06:57:21.321 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 22: /lib64/libc.so.6(+0x89c02) [0x7fba18fa8c02] 2024-07-26T06:57:21.321 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fba1902dc40] 2024-07-26T06:57:21.321 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:21.321 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 2024-07-26T06:57:21.321 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 0> 2024-07-26T06:57:20.944+0000 7fba13b86640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:21.321 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: in thread 7fba13b86640 thread_name:ms_dispatch 2024-07-26T06:57:21.321 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 2024-07-26T06:57:21.321 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:21.321 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fba18f5d6f0] 2024-07-26T06:57:21.321 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fba18faa94c] 2024-07-26T06:57:21.321 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 3: raise() 2024-07-26T06:57:21.322 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 4: abort() 2024-07-26T06:57:21.322 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 5: /lib64/libc.so.6(+0x2871b) [0x7fba18f4771b] 2024-07-26T06:57:21.322 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 6: /lib64/libc.so.6(+0x37386) [0x7fba18f56386] 2024-07-26T06:57:21.322 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560bca10e6b9] 2024-07-26T06:57:21.322 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x560bca2cc80c] 2024-07-26T06:57:21.322 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560bca20c2ca] 2024-07-26T06:57:21.322 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560bca1dc753] 2024-07-26T06:57:21.322 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560bca1dda1b] 2024-07-26T06:57:21.322 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560bca1c1752] 2024-07-26T06:57:21.322 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560bca10fa56] 2024-07-26T06:57:21.322 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560bca113637] 2024-07-26T06:57:21.322 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560bca113f5e] 2024-07-26T06:57:21.323 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560bca0f703a] 2024-07-26T06:57:21.323 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fba1982a407] 2024-07-26T06:57:21.323 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 18: (DispatchQueue::entry()+0x60f) [0x7fba1982acf9] 2024-07-26T06:57:21.323 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fba199075a1] 2024-07-26T06:57:21.323 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 20: (Thread::entry_wrapper()+0x43) [0x7fba196b4cdd] 2024-07-26T06:57:21.323 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 21: (Thread::_entry_func(void*)+0xd) [0x7fba196b4cf9] 2024-07-26T06:57:21.323 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 22: /lib64/libc.so.6(+0x89c02) [0x7fba18fa8c02] 2024-07-26T06:57:21.323 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fba1902dc40] 2024-07-26T06:57:21.323 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:21.323 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119099]: 2024-07-26T06:57:21.324 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f80c0fef407] 2024-07-26T06:57:21.324 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 18: (DispatchQueue::entry()+0x60f) [0x7f80c0fefcf9] 2024-07-26T06:57:21.324 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f80c10cc5a1] 2024-07-26T06:57:21.324 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 20: (Thread::entry_wrapper()+0x43) [0x7f80c0e79cdd] 2024-07-26T06:57:21.324 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 21: (Thread::_entry_func(void*)+0xd) [0x7f80c0e79cf9] 2024-07-26T06:57:21.324 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 22: /lib64/libc.so.6(+0x89c02) [0x7f80c076dc02] 2024-07-26T06:57:21.324 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f80c07f2c40] 2024-07-26T06:57:21.324 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:21.324 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 2024-07-26T06:57:21.324 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 0> 2024-07-26T06:57:20.944+0000 7f80bb34b640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:21.325 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: in thread 7f80bb34b640 thread_name:ms_dispatch 2024-07-26T06:57:21.325 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 2024-07-26T06:57:21.325 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:21.325 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f80c07226f0] 2024-07-26T06:57:21.325 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f80c076f94c] 2024-07-26T06:57:21.325 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 3: raise() 2024-07-26T06:57:21.325 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 4: abort() 2024-07-26T06:57:21.325 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 5: /lib64/libc.so.6(+0x2871b) [0x7f80c070c71b] 2024-07-26T06:57:21.325 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 6: /lib64/libc.so.6(+0x37386) [0x7f80c071b386] 2024-07-26T06:57:21.325 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55c6a1a0a6b9] 2024-07-26T06:57:21.325 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55c6a1bc880c] 2024-07-26T06:57:21.325 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55c6a1b082ca] 2024-07-26T06:57:21.326 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55c6a1ad8753] 2024-07-26T06:57:21.326 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55c6a1ad9a1b] 2024-07-26T06:57:21.326 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55c6a1abd752] 2024-07-26T06:57:21.326 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55c6a1a0ba56] 2024-07-26T06:57:21.326 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55c6a1a0f637] 2024-07-26T06:57:21.326 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55c6a1a0ff5e] 2024-07-26T06:57:21.326 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55c6a19f303a] 2024-07-26T06:57:21.326 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f80c0fef407] 2024-07-26T06:57:21.326 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 18: (DispatchQueue::entry()+0x60f) [0x7f80c0fefcf9] 2024-07-26T06:57:21.326 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f80c10cc5a1] 2024-07-26T06:57:21.326 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 20: (Thread::entry_wrapper()+0x43) [0x7f80c0e79cdd] 2024-07-26T06:57:21.326 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 21: (Thread::_entry_func(void*)+0xd) [0x7f80c0e79cf9] 2024-07-26T06:57:21.327 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 22: /lib64/libc.so.6(+0x89c02) [0x7f80c076dc02] 2024-07-26T06:57:21.327 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f80c07f2c40] 2024-07-26T06:57:21.327 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:21.327 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:20 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119073]: 2024-07-26T06:57:21.327 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:21 smithi042 podman[119280]: 2024-07-26 06:57:21.011289537 +0000 UTC m=+0.022542539 container died 3d9471d11705f0c5c3b772637f63fa1d994778f5f1ffda4d6507a239179d9325 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240724) 2024-07-26T06:57:21.933 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:21 smithi164 podman[70852]: 2024-07-26 06:57:21.609694916 +0000 UTC m=+0.628993226 container remove 05c159eaa36fa59ba074a0fdb1e433cfd52662120704644479d9d1c34b0ac93f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-07-26T06:57:21.933 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:21 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:57:21.933 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:21 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-07-26T06:57:22.137 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:21 smithi042 podman[119286]: 2024-07-26 06:57:21.84609612 +0000 UTC m=+0.851155392 container died f83bce301f9db709c7b092d0096e0516b1a3545b1de9b29b5b754d2371029ee7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, io.buildah.version=1.36.0) 2024-07-26T06:57:22.138 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:21 smithi042 podman[119280]: 2024-07-26 06:57:21.833696406 +0000 UTC m=+0.844949409 container remove 3d9471d11705f0c5c3b772637f63fa1d994778f5f1ffda4d6507a239179d9325 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-07-26T06:57:22.138 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:21 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:57:22.138 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:21 smithi042 ceph-mon[32775]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:57:22.138 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:21 smithi042 ceph-mon[32775]: Cluster is now healthy 2024-07-26T06:57:22.138 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:21 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/2696987957,v1:172.21.15.164:6833/2696987957] up:active 2024-07-26T06:57:22.139 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:21 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/2505332261,v1:172.21.15.42:6841/2505332261] up:active 2024-07-26T06:57:22.139 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:21 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3995951481,v1:172.21.15.42:6839/3995951481] up:active 2024-07-26T06:57:22.139 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:21 smithi042 ceph-mon[32775]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:57:22.139 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:21 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:57:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:21 smithi164 ceph-mon[36970]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:57:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:21 smithi164 ceph-mon[36970]: Cluster is now healthy 2024-07-26T06:57:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:21 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/2696987957,v1:172.21.15.164:6833/2696987957] up:active 2024-07-26T06:57:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:21 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/2505332261,v1:172.21.15.42:6841/2505332261] up:active 2024-07-26T06:57:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:21 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3995951481,v1:172.21.15.42:6839/3995951481] up:active 2024-07-26T06:57:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:21 smithi164 ceph-mon[36970]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:57:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:21 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:57:22.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:21 smithi163 ceph-mon[36873]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:57:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:21 smithi163 ceph-mon[36873]: Cluster is now healthy 2024-07-26T06:57:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:21 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/2696987957,v1:172.21.15.164:6833/2696987957] up:active 2024-07-26T06:57:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:21 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/2505332261,v1:172.21.15.42:6841/2505332261] up:active 2024-07-26T06:57:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:21 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3995951481,v1:172.21.15.42:6839/3995951481] up:active 2024-07-26T06:57:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:21 smithi163 ceph-mon[36873]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:57:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:21 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:57:22.708 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:22 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-07-26T06:57:22.709 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:22 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Consumed 1.005s CPU time. 2024-07-26T06:57:22.709 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:22 smithi042 podman[119286]: 2024-07-26 06:57:22.44725932 +0000 UTC m=+1.452318589 container remove f83bce301f9db709c7b092d0096e0516b1a3545b1de9b29b5b754d2371029ee7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=) 2024-07-26T06:57:22.709 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:22 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:57:23.144 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:22 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T06:57:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:22 smithi042 ceph-mon[32775]: pgmap v627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T06:57:23.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:22 smithi164 ceph-mon[36970]: pgmap v627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T06:57:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:22 smithi163 ceph-mon[36873]: pgmap v627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T06:57:23.303 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:57:23.303+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:57:23.304 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:57:23.303+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T06:57:23.304 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:57:23.303+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T06:57:23.304 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:57:23.303+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T06:57:23.304 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:57:23.303+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:57:23.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:57:23.706+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:57:23.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:57:23.707+0000 7faa2dcb9640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 msgr2=0x561ca20d0cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T06:57:23.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:57:23.707+0000 7faa2dcb9640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 msgr2=0x561ca20d0cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T06:57:23.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:57:23.707+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T06:57:23.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:57:23.707+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:57:25.002 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:24 smithi042 ceph-mon[32775]: pgmap v628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 407 B/s wr, 13 op/s 2024-07-26T06:57:25.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:24 smithi164 ceph-mon[36970]: pgmap v628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 407 B/s wr, 13 op/s 2024-07-26T06:57:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:24 smithi163 ceph-mon[36873]: pgmap v628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 407 B/s wr, 13 op/s 2024-07-26T06:57:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:26 smithi163 ceph-mon[36873]: pgmap v629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 669 B/s wr, 14 op/s 2024-07-26T06:57:27.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:26 smithi042 ceph-mon[32775]: pgmap v629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 669 B/s wr, 14 op/s 2024-07-26T06:57:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:26 smithi164 ceph-mon[36970]: pgmap v629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 669 B/s wr, 14 op/s 2024-07-26T06:57:29.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:28 smithi042 ceph-mon[32775]: pgmap v630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 579 B/s wr, 13 op/s 2024-07-26T06:57:29.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:28 smithi164 ceph-mon[36970]: pgmap v630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 579 B/s wr, 13 op/s 2024-07-26T06:57:29.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:28 smithi163 ceph-mon[36873]: pgmap v630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 579 B/s wr, 13 op/s 2024-07-26T06:57:31.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:30 smithi164 ceph-mon[36970]: pgmap v631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 511 B/s wr, 14 op/s 2024-07-26T06:57:31.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:30 smithi163 ceph-mon[36873]: pgmap v631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 511 B/s wr, 14 op/s 2024-07-26T06:57:31.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:30 smithi042 ceph-mon[32775]: pgmap v631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 511 B/s wr, 14 op/s 2024-07-26T06:57:32.235 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:31 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 3. 2024-07-26T06:57:32.287 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:31 smithi164 systemd[1]: Stopped Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:32.287 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:31 smithi164 systemd[1]: Starting Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:57:32.287 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:32 smithi164 podman[70943]: 2024-07-26 06:57:32.112443032 +0000 UTC m=+0.017799657 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:57:32.287 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:32 smithi164 podman[70943]: 2024-07-26 06:57:32.235904263 +0000 UTC m=+0.141260878 container create 3d91911e4e96facc925304f591eafe9c446657b82424378d484d0d415cd797db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-07-26T06:57:32.498 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:32 smithi164 podman[70943]: 2024-07-26 06:57:32.463668727 +0000 UTC m=+0.369025359 container init 3d91911e4e96facc925304f591eafe9c446657b82424378d484d0d415cd797db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-07-26T06:57:32.498 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:32 smithi164 podman[70943]: 2024-07-26 06:57:32.466319285 +0000 UTC m=+0.371675908 container start 3d91911e4e96facc925304f591eafe9c446657b82424378d484d0d415cd797db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-07-26T06:57:32.831 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:32 smithi164 ceph-mds[70963]: -- 172.21.15.164:0/3938477877 <== mon.1 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558c2a62b860 con 0x558c2e7cda80 2024-07-26T06:57:32.832 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: starting mds.l at 2024-07-26T06:57:32.832 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:32 smithi164 bash[70943]: 3d91911e4e96facc925304f591eafe9c446657b82424378d484d0d415cd797db 2024-07-26T06:57:32.832 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:32 smithi164 systemd[1]: Started Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:32.906 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:32 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 3. 2024-07-26T06:57:32.907 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:32 smithi042 systemd[1]: Stopped Ceph mds.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:32.907 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:32 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 3. 2024-07-26T06:57:32.907 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:32 smithi042 systemd[1]: Stopped Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:32.907 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:32 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Consumed 1.005s CPU time. 2024-07-26T06:57:32.907 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:32 smithi042 systemd[1]: Starting Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:57:32.907 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:32 smithi042 podman[119424]: 2024-07-26 06:57:32.792993698 +0000 UTC m=+0.018179966 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:57:32.907 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:32 smithi042 podman[119424]: 2024-07-26 06:57:32.906783596 +0000 UTC m=+0.131969888 container create f225d4b8aad6754fce903925827868192e05a9ed3a09c37689763cebc83a1e9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-07-26T06:57:33.168 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:32 smithi042 systemd[1]: Starting Ceph mds.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:57:33.168 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:32 smithi042 ceph-mon[32775]: pgmap v632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T06:57:33.168 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:32 smithi042 ceph-mon[32775]: daemon mds.l restarted 2024-07-26T06:57:33.168 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:32 smithi042 ceph-mon[32775]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:57:33.168 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:32 smithi042 ceph-mon[32775]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T06:57:33.168 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:32 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/351656606,v1:172.21.15.164:6833/351656606] up:boot 2024-07-26T06:57:33.168 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:32 smithi042 ceph-mon[32775]: Standby daemon mds.b assigned to filesystem cephfs as rank 2 2024-07-26T06:57:33.169 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:32 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:57:33.169 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:57:33.169 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:32 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:57:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:32 smithi164 ceph-mon[36970]: pgmap v632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T06:57:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:32 smithi164 ceph-mon[36970]: daemon mds.l restarted 2024-07-26T06:57:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:32 smithi164 ceph-mon[36970]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:57:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:32 smithi164 ceph-mon[36970]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T06:57:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:32 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/351656606,v1:172.21.15.164:6833/351656606] up:boot 2024-07-26T06:57:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:32 smithi164 ceph-mon[36970]: Standby daemon mds.b assigned to filesystem cephfs as rank 2 2024-07-26T06:57:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:32 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:57:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:57:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:32 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:57:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:32 smithi163 ceph-mon[36873]: pgmap v632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T06:57:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:32 smithi163 ceph-mon[36873]: daemon mds.l restarted 2024-07-26T06:57:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:32 smithi163 ceph-mon[36873]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:57:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:32 smithi163 ceph-mon[36873]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T06:57:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:32 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/351656606,v1:172.21.15.164:6833/351656606] up:boot 2024-07-26T06:57:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:32 smithi163 ceph-mon[36873]: Standby daemon mds.b assigned to filesystem cephfs as rank 2 2024-07-26T06:57:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:32 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:57:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:57:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:32 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:57:33.496 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:33 smithi042 podman[119424]: 2024-07-26 06:57:33.168190791 +0000 UTC m=+0.393377056 container init f225d4b8aad6754fce903925827868192e05a9ed3a09c37689763cebc83a1e9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240724) 2024-07-26T06:57:33.497 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:33 smithi042 podman[119424]: 2024-07-26 06:57:33.171801102 +0000 UTC m=+0.396987359 container start f225d4b8aad6754fce903925827868192e05a9ed3a09c37689763cebc83a1e9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-07-26T06:57:33.497 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-mds[119482]: -- 172.21.15.42:0/3980705442 <== mon.0 v2:172.21.15.42:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5611a34b5860 con 0x5611a7657180 2024-07-26T06:57:33.497 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: starting mds.g at 2024-07-26T06:57:33.497 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:33 smithi042 bash[119424]: f225d4b8aad6754fce903925827868192e05a9ed3a09c37689763cebc83a1e9a 2024-07-26T06:57:33.497 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:33 smithi042 systemd[1]: Started Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:33.797 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:33 smithi042 podman[119513]: 2024-07-26 06:57:33.399598563 +0000 UTC m=+0.017991612 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:57:33.798 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:33 smithi042 podman[119513]: 2024-07-26 06:57:33.536763837 +0000 UTC m=+0.155156917 container create 28ee803f4704cd586a4e3d7579fba64777b6e7d834c99c68b512f2d81ec09344 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T06:57:34.145 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:33 smithi042 podman[119513]: 2024-07-26 06:57:33.798011555 +0000 UTC m=+0.416404592 container init 28ee803f4704cd586a4e3d7579fba64777b6e7d834c99c68b512f2d81ec09344 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux ) 2024-07-26T06:57:34.145 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:33 smithi042 podman[119513]: 2024-07-26 06:57:33.80103666 +0000 UTC m=+0.419429700 container start 28ee803f4704cd586a4e3d7579fba64777b6e7d834c99c68b512f2d81ec09344 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-07-26T06:57:34.145 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-mds[119533]: -- 172.21.15.42:0/3809015861 <== mon.0 v2:172.21.15.42:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561884ebfc20 con 0x561889061180 2024-07-26T06:57:34.145 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: starting mds.a at 2024-07-26T06:57:34.145 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:33 smithi042 bash[119513]: 28ee803f4704cd586a4e3d7579fba64777b6e7d834c99c68b512f2d81ec09344 2024-07-26T06:57:34.145 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:33 smithi042 systemd[1]: Started Ceph mds.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:34.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-mon[32775]: daemon mds.g restarted 2024-07-26T06:57:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-mon[32775]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T06:57:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/734940828,v1:172.21.15.163:6839/734940828] up:resolve 2024-07-26T06:57:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3863572604,v1:172.21.15.42:6839/3863572604] up:boot 2024-07-26T06:57:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-mon[32775]: Standby daemon mds.h assigned to filesystem cephfs as rank 4 2024-07-26T06:57:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=b=up:resolve,3=a=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:57:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:57:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:57:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-mon[32775]: daemon mds.a restarted 2024-07-26T06:57:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-mon[32775]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T06:57:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/1877922073,v1:172.21.15.42:6841/1877922073] up:boot 2024-07-26T06:57:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-mon[32775]: Standby daemon mds.e assigned to filesystem cephfs as rank 3 2024-07-26T06:57:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=b=up:resolve,4=h=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:57:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T06:57:34.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:33 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:57:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:33 smithi164 ceph-mon[36970]: daemon mds.g restarted 2024-07-26T06:57:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:33 smithi164 ceph-mon[36970]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T06:57:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:33 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/734940828,v1:172.21.15.163:6839/734940828] up:resolve 2024-07-26T06:57:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:33 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3863572604,v1:172.21.15.42:6839/3863572604] up:boot 2024-07-26T06:57:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:33 smithi164 ceph-mon[36970]: Standby daemon mds.h assigned to filesystem cephfs as rank 4 2024-07-26T06:57:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:33 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=b=up:resolve,3=a=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:57:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:57:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:33 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:57:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:33 smithi164 ceph-mon[36970]: daemon mds.a restarted 2024-07-26T06:57:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:33 smithi164 ceph-mon[36970]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T06:57:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:33 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/1877922073,v1:172.21.15.42:6841/1877922073] up:boot 2024-07-26T06:57:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:33 smithi164 ceph-mon[36970]: Standby daemon mds.e assigned to filesystem cephfs as rank 3 2024-07-26T06:57:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:33 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=b=up:resolve,4=h=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:57:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T06:57:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:33 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:57:34.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:33 smithi163 ceph-mon[36873]: daemon mds.g restarted 2024-07-26T06:57:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:33 smithi163 ceph-mon[36873]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T06:57:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:33 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/734940828,v1:172.21.15.163:6839/734940828] up:resolve 2024-07-26T06:57:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:33 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3863572604,v1:172.21.15.42:6839/3863572604] up:boot 2024-07-26T06:57:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:33 smithi163 ceph-mon[36873]: Standby daemon mds.h assigned to filesystem cephfs as rank 4 2024-07-26T06:57:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:33 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=b=up:resolve,3=a=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:57:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:57:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:33 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:57:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:33 smithi163 ceph-mon[36873]: daemon mds.a restarted 2024-07-26T06:57:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:33 smithi163 ceph-mon[36873]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T06:57:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:33 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/1877922073,v1:172.21.15.42:6841/1877922073] up:boot 2024-07-26T06:57:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:33 smithi163 ceph-mon[36873]: Standby daemon mds.e assigned to filesystem cephfs as rank 3 2024-07-26T06:57:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:33 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=b=up:resolve,4=h=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:57:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T06:57:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:33 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:57:35.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:34 smithi163 ceph-mon[36873]: pgmap v635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2024-07-26T06:57:35.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:34 smithi042 ceph-mon[32775]: pgmap v635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2024-07-26T06:57:35.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:34 smithi164 ceph-mon[36970]: pgmap v635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2024-07-26T06:57:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:35 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/4111555088,v1:172.21.15.163:6835/4111555088] up:resolve 2024-07-26T06:57:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:35 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/850726105,v1:172.21.15.163:6837/850726105] up:resolve 2024-07-26T06:57:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:35 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:57:36.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:35 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/4111555088,v1:172.21.15.163:6835/4111555088] up:resolve 2024-07-26T06:57:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:35 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/850726105,v1:172.21.15.163:6837/850726105] up:resolve 2024-07-26T06:57:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:35 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:57:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:35 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/4111555088,v1:172.21.15.163:6835/4111555088] up:resolve 2024-07-26T06:57:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:35 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/850726105,v1:172.21.15.163:6837/850726105] up:resolve 2024-07-26T06:57:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:35 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:57:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:36 smithi163 ceph-mon[36873]: pgmap v637: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T06:57:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:36 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/4111555088,v1:172.21.15.163:6835/4111555088] up:reconnect 2024-07-26T06:57:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:36 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/734940828,v1:172.21.15.163:6839/734940828] up:reconnect 2024-07-26T06:57:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:36 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/850726105,v1:172.21.15.163:6837/850726105] up:reconnect 2024-07-26T06:57:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:36 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:57:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:36 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.0129997 2024-07-26T06:57:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:36 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.0129997 2024-07-26T06:57:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:36 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.0129997 2024-07-26T06:57:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:36 smithi042 ceph-mon[32775]: pgmap v637: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T06:57:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:36 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/4111555088,v1:172.21.15.163:6835/4111555088] up:reconnect 2024-07-26T06:57:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:36 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/734940828,v1:172.21.15.163:6839/734940828] up:reconnect 2024-07-26T06:57:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:36 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/850726105,v1:172.21.15.163:6837/850726105] up:reconnect 2024-07-26T06:57:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:36 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:57:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:36 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.0129997 2024-07-26T06:57:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:36 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.0129997 2024-07-26T06:57:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:36 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.0129997 2024-07-26T06:57:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:36 smithi164 ceph-mon[36970]: pgmap v637: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T06:57:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:36 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/4111555088,v1:172.21.15.163:6835/4111555088] up:reconnect 2024-07-26T06:57:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:36 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/734940828,v1:172.21.15.163:6839/734940828] up:reconnect 2024-07-26T06:57:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:36 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/850726105,v1:172.21.15.163:6837/850726105] up:reconnect 2024-07-26T06:57:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:36 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:57:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:36 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.0129997 2024-07-26T06:57:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:36 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.0129997 2024-07-26T06:57:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:36 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.0129997 2024-07-26T06:57:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:37 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/734940828,v1:172.21.15.163:6839/734940828] up:rejoin 2024-07-26T06:57:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:37 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/4111555088,v1:172.21.15.163:6835/4111555088] up:rejoin 2024-07-26T06:57:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:37 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/850726105,v1:172.21.15.163:6837/850726105] up:rejoin 2024-07-26T06:57:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:37 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:57:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:37 smithi163 ceph-mon[36873]: daemon mds.b is now active in filesystem cephfs as rank 2 2024-07-26T06:57:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:37 smithi163 ceph-mon[36873]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-07-26T06:57:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:37 smithi163 ceph-mon[36873]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-07-26T06:57:38.258 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:57:38.258 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: *** Caught signal (Aborted) ** 2024-07-26T06:57:38.258 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: in thread 7f0e6385b640 thread_name:ms_dispatch 2024-07-26T06:57:38.258 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:38.258 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0e68c326f0] 2024-07-26T06:57:38.258 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0e68c7f94c] 2024-07-26T06:57:38.258 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 3: raise() 2024-07-26T06:57:38.258 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 4: abort() 2024-07-26T06:57:38.258 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0e68c1c71b] 2024-07-26T06:57:38.258 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 6: /lib64/libc.so.6(+0x37386) [0x7f0e68c2b386] 2024-07-26T06:57:38.259 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55563c0306b9] 2024-07-26T06:57:38.259 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55563c1ee80c] 2024-07-26T06:57:38.259 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55563c12e2ca] 2024-07-26T06:57:38.259 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55563c0fe753] 2024-07-26T06:57:38.259 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55563c0ffa1b] 2024-07-26T06:57:38.259 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55563c0e3752] 2024-07-26T06:57:38.259 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55563c031a56] 2024-07-26T06:57:38.259 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55563c035637] 2024-07-26T06:57:38.259 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55563c035f5e] 2024-07-26T06:57:38.259 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55563c01903a] 2024-07-26T06:57:38.259 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0e694ff407] 2024-07-26T06:57:38.259 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 18: (DispatchQueue::entry()+0x60f) [0x7f0e694ffcf9] 2024-07-26T06:57:38.260 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0e695dc5a1] 2024-07-26T06:57:38.260 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:57:38.260 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: *** Caught signal (Aborted) ** 2024-07-26T06:57:38.260 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: in thread 7fc1f5cc1640 thread_name:ms_dispatch 2024-07-26T06:57:38.260 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:38.260 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc1fb0986f0] 2024-07-26T06:57:38.260 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc1fb0e594c] 2024-07-26T06:57:38.260 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 3: raise() 2024-07-26T06:57:38.260 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 4: abort() 2024-07-26T06:57:38.261 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 5: /lib64/libc.so.6(+0x2871b) [0x7fc1fb08271b] 2024-07-26T06:57:38.261 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 6: /lib64/libc.so.6(+0x37386) [0x7fc1fb091386] 2024-07-26T06:57:38.261 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b0c4b7d6b9] 2024-07-26T06:57:38.261 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b0c4d3b80c] 2024-07-26T06:57:38.261 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b0c4c7b2ca] 2024-07-26T06:57:38.261 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b0c4c4b753] 2024-07-26T06:57:38.261 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b0c4c4ca1b] 2024-07-26T06:57:38.261 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b0c4c30752] 2024-07-26T06:57:38.261 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b0c4b7ea56] 2024-07-26T06:57:38.261 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b0c4b82637] 2024-07-26T06:57:38.261 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b0c4b82f5e] 2024-07-26T06:57:38.261 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b0c4b6603a] 2024-07-26T06:57:38.261 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fc1fb965407] 2024-07-26T06:57:38.262 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 18: (DispatchQueue::entry()+0x60f) [0x7fc1fb965cf9] 2024-07-26T06:57:38.262 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fc1fba425a1] 2024-07-26T06:57:38.262 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:57:38.262 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: *** Caught signal (Aborted) ** 2024-07-26T06:57:38.262 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: in thread 7f77fdb44640 thread_name:ms_dispatch 2024-07-26T06:57:38.262 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:38.262 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7802f1b6f0] 2024-07-26T06:57:38.262 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7802f6894c] 2024-07-26T06:57:38.263 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 3: raise() 2024-07-26T06:57:38.263 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 4: abort() 2024-07-26T06:57:38.263 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7802f0571b] 2024-07-26T06:57:38.263 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 6: /lib64/libc.so.6(+0x37386) [0x7f7802f14386] 2024-07-26T06:57:38.263 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x564ed64c16b9] 2024-07-26T06:57:38.263 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x564ed667f80c] 2024-07-26T06:57:38.263 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x564ed65bf2ca] 2024-07-26T06:57:38.263 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564ed658f753] 2024-07-26T06:57:38.263 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564ed6590a1b] 2024-07-26T06:57:38.263 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564ed6574752] 2024-07-26T06:57:38.263 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564ed64c2a56] 2024-07-26T06:57:38.263 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x564ed64c6637] 2024-07-26T06:57:38.263 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x564ed64c6f5e] 2024-07-26T06:57:38.264 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x564ed64aa03a] 2024-07-26T06:57:38.264 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f78037e8407] 2024-07-26T06:57:38.264 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 18: (DispatchQueue::entry()+0x60f) [0x7f78037e8cf9] 2024-07-26T06:57:38.264 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f78038c55a1] 2024-07-26T06:57:38.264 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 20: (Thread::entry_wrapper()+0x43) [0x7f0e69389cdd] 2024-07-26T06:57:38.264 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0e69389cf9] 2024-07-26T06:57:38.264 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0e68c7dc02] 2024-07-26T06:57:38.264 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0e68d02c40] 2024-07-26T06:57:38.265 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 2024-07-26T06:57:38.004+0000 7f0e6385b640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:38.265 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: in thread 7f0e6385b640 thread_name:ms_dispatch 2024-07-26T06:57:38.265 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 2024-07-26T06:57:38.265 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:38.265 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0e68c326f0] 2024-07-26T06:57:38.265 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0e68c7f94c] 2024-07-26T06:57:38.265 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 3: raise() 2024-07-26T06:57:38.265 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 4: abort() 2024-07-26T06:57:38.265 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0e68c1c71b] 2024-07-26T06:57:38.265 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 6: /lib64/libc.so.6(+0x37386) [0x7f0e68c2b386] 2024-07-26T06:57:38.265 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55563c0306b9] 2024-07-26T06:57:38.265 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55563c1ee80c] 2024-07-26T06:57:38.266 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55563c12e2ca] 2024-07-26T06:57:38.266 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55563c0fe753] 2024-07-26T06:57:38.266 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55563c0ffa1b] 2024-07-26T06:57:38.266 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55563c0e3752] 2024-07-26T06:57:38.266 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55563c031a56] 2024-07-26T06:57:38.266 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55563c035637] 2024-07-26T06:57:38.266 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55563c035f5e] 2024-07-26T06:57:38.266 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55563c01903a] 2024-07-26T06:57:38.266 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0e694ff407] 2024-07-26T06:57:38.266 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 18: (DispatchQueue::entry()+0x60f) [0x7f0e694ffcf9] 2024-07-26T06:57:38.267 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0e695dc5a1] 2024-07-26T06:57:38.267 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 20: (Thread::entry_wrapper()+0x43) [0x7f0e69389cdd] 2024-07-26T06:57:38.267 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0e69389cf9] 2024-07-26T06:57:38.267 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0e68c7dc02] 2024-07-26T06:57:38.267 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0e68d02c40] 2024-07-26T06:57:38.267 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:38.267 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 2024-07-26T06:57:38.267 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 0> 2024-07-26T06:57:38.004+0000 7f0e6385b640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:38.267 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: in thread 7f0e6385b640 thread_name:ms_dispatch 2024-07-26T06:57:38.267 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 2024-07-26T06:57:38.267 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:38.267 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0e68c326f0] 2024-07-26T06:57:38.267 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0e68c7f94c] 2024-07-26T06:57:38.268 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 3: raise() 2024-07-26T06:57:38.268 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 4: abort() 2024-07-26T06:57:38.268 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0e68c1c71b] 2024-07-26T06:57:38.268 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 6: /lib64/libc.so.6(+0x37386) [0x7f0e68c2b386] 2024-07-26T06:57:38.268 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55563c0306b9] 2024-07-26T06:57:38.268 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55563c1ee80c] 2024-07-26T06:57:38.268 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55563c12e2ca] 2024-07-26T06:57:38.268 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55563c0fe753] 2024-07-26T06:57:38.268 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55563c0ffa1b] 2024-07-26T06:57:38.268 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55563c0e3752] 2024-07-26T06:57:38.268 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55563c031a56] 2024-07-26T06:57:38.268 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55563c035637] 2024-07-26T06:57:38.269 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55563c035f5e] 2024-07-26T06:57:38.269 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55563c01903a] 2024-07-26T06:57:38.269 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0e694ff407] 2024-07-26T06:57:38.269 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 20: (Thread::entry_wrapper()+0x43) [0x7fc1fb7efcdd] 2024-07-26T06:57:38.269 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc1fb7efcf9] 2024-07-26T06:57:38.269 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 22: /lib64/libc.so.6(+0x89c02) [0x7fc1fb0e3c02] 2024-07-26T06:57:38.269 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fc1fb168c40] 2024-07-26T06:57:38.269 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 2024-07-26T06:57:38.003+0000 7fc1f5cc1640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:38.269 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: in thread 7fc1f5cc1640 thread_name:ms_dispatch 2024-07-26T06:57:38.269 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 2024-07-26T06:57:38.269 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:38.270 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc1fb0986f0] 2024-07-26T06:57:38.270 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc1fb0e594c] 2024-07-26T06:57:38.270 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 3: raise() 2024-07-26T06:57:38.270 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 4: abort() 2024-07-26T06:57:38.270 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 5: /lib64/libc.so.6(+0x2871b) [0x7fc1fb08271b] 2024-07-26T06:57:38.270 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 6: /lib64/libc.so.6(+0x37386) [0x7fc1fb091386] 2024-07-26T06:57:38.270 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b0c4b7d6b9] 2024-07-26T06:57:38.270 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b0c4d3b80c] 2024-07-26T06:57:38.270 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b0c4c7b2ca] 2024-07-26T06:57:38.270 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b0c4c4b753] 2024-07-26T06:57:38.270 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b0c4c4ca1b] 2024-07-26T06:57:38.270 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b0c4c30752] 2024-07-26T06:57:38.271 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b0c4b7ea56] 2024-07-26T06:57:38.271 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b0c4b82637] 2024-07-26T06:57:38.271 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b0c4b82f5e] 2024-07-26T06:57:38.271 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b0c4b6603a] 2024-07-26T06:57:38.271 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fc1fb965407] 2024-07-26T06:57:38.271 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 18: (DispatchQueue::entry()+0x60f) [0x7fc1fb965cf9] 2024-07-26T06:57:38.271 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fc1fba425a1] 2024-07-26T06:57:38.271 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 20: (Thread::entry_wrapper()+0x43) [0x7f7803672cdd] 2024-07-26T06:57:38.271 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7803672cf9] 2024-07-26T06:57:38.272 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7802f66c02] 2024-07-26T06:57:38.272 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7802febc40] 2024-07-26T06:57:38.272 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 2024-07-26T06:57:38.004+0000 7f77fdb44640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:38.272 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: in thread 7f77fdb44640 thread_name:ms_dispatch 2024-07-26T06:57:38.272 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 2024-07-26T06:57:38.272 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:38.272 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7802f1b6f0] 2024-07-26T06:57:38.272 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7802f6894c] 2024-07-26T06:57:38.272 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 3: raise() 2024-07-26T06:57:38.272 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 4: abort() 2024-07-26T06:57:38.272 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7802f0571b] 2024-07-26T06:57:38.272 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 6: /lib64/libc.so.6(+0x37386) [0x7f7802f14386] 2024-07-26T06:57:38.272 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x564ed64c16b9] 2024-07-26T06:57:38.273 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x564ed667f80c] 2024-07-26T06:57:38.273 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x564ed65bf2ca] 2024-07-26T06:57:38.273 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564ed658f753] 2024-07-26T06:57:38.273 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564ed6590a1b] 2024-07-26T06:57:38.273 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564ed6574752] 2024-07-26T06:57:38.273 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564ed64c2a56] 2024-07-26T06:57:38.273 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x564ed64c6637] 2024-07-26T06:57:38.273 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x564ed64c6f5e] 2024-07-26T06:57:38.273 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x564ed64aa03a] 2024-07-26T06:57:38.273 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f78037e8407] 2024-07-26T06:57:38.273 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 18: (DispatchQueue::entry()+0x60f) [0x7f78037e8cf9] 2024-07-26T06:57:38.273 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f78038c55a1] 2024-07-26T06:57:38.274 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 18: (DispatchQueue::entry()+0x60f) [0x7f0e694ffcf9] 2024-07-26T06:57:38.274 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0e695dc5a1] 2024-07-26T06:57:38.274 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 20: (Thread::entry_wrapper()+0x43) [0x7f0e69389cdd] 2024-07-26T06:57:38.274 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0e69389cf9] 2024-07-26T06:57:38.274 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0e68c7dc02] 2024-07-26T06:57:38.274 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0e68d02c40] 2024-07-26T06:57:38.275 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:38.275 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 2024-07-26T06:57:38.275 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 0> 2024-07-26T06:57:38.004+0000 7f0e6385b640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:38.275 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: in thread 7f0e6385b640 thread_name:ms_dispatch 2024-07-26T06:57:38.275 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 2024-07-26T06:57:38.275 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:38.275 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0e68c326f0] 2024-07-26T06:57:38.275 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0e68c7f94c] 2024-07-26T06:57:38.275 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 3: raise() 2024-07-26T06:57:38.275 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 4: abort() 2024-07-26T06:57:38.275 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0e68c1c71b] 2024-07-26T06:57:38.275 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 6: /lib64/libc.so.6(+0x37386) [0x7f0e68c2b386] 2024-07-26T06:57:38.275 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55563c0306b9] 2024-07-26T06:57:38.276 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55563c1ee80c] 2024-07-26T06:57:38.276 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55563c12e2ca] 2024-07-26T06:57:38.276 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55563c0fe753] 2024-07-26T06:57:38.276 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55563c0ffa1b] 2024-07-26T06:57:38.276 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55563c0e3752] 2024-07-26T06:57:38.276 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55563c031a56] 2024-07-26T06:57:38.276 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55563c035637] 2024-07-26T06:57:38.276 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55563c035f5e] 2024-07-26T06:57:38.276 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55563c01903a] 2024-07-26T06:57:38.276 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0e694ff407] 2024-07-26T06:57:38.276 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 18: (DispatchQueue::entry()+0x60f) [0x7f0e694ffcf9] 2024-07-26T06:57:38.277 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0e695dc5a1] 2024-07-26T06:57:38.277 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 20: (Thread::entry_wrapper()+0x43) [0x7f0e69389cdd] 2024-07-26T06:57:38.277 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0e69389cf9] 2024-07-26T06:57:38.277 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0e68c7dc02] 2024-07-26T06:57:38.277 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0e68d02c40] 2024-07-26T06:57:38.277 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:38.277 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71283]: 2024-07-26T06:57:38.277 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 20: (Thread::entry_wrapper()+0x43) [0x7fc1fb7efcdd] 2024-07-26T06:57:38.277 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc1fb7efcf9] 2024-07-26T06:57:38.277 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 22: /lib64/libc.so.6(+0x89c02) [0x7fc1fb0e3c02] 2024-07-26T06:57:38.278 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fc1fb168c40] 2024-07-26T06:57:38.278 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:38.278 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 2024-07-26T06:57:38.278 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 0> 2024-07-26T06:57:38.003+0000 7fc1f5cc1640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:38.278 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: in thread 7fc1f5cc1640 thread_name:ms_dispatch 2024-07-26T06:57:38.278 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 2024-07-26T06:57:38.278 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:38.278 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc1fb0986f0] 2024-07-26T06:57:38.278 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc1fb0e594c] 2024-07-26T06:57:38.279 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 3: raise() 2024-07-26T06:57:38.279 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 4: abort() 2024-07-26T06:57:38.279 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 5: /lib64/libc.so.6(+0x2871b) [0x7fc1fb08271b] 2024-07-26T06:57:38.279 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 6: /lib64/libc.so.6(+0x37386) [0x7fc1fb091386] 2024-07-26T06:57:38.279 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b0c4b7d6b9] 2024-07-26T06:57:38.279 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b0c4d3b80c] 2024-07-26T06:57:38.279 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b0c4c7b2ca] 2024-07-26T06:57:38.279 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b0c4c4b753] 2024-07-26T06:57:38.279 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b0c4c4ca1b] 2024-07-26T06:57:38.279 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b0c4c30752] 2024-07-26T06:57:38.279 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b0c4b7ea56] 2024-07-26T06:57:38.279 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b0c4b82637] 2024-07-26T06:57:38.279 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b0c4b82f5e] 2024-07-26T06:57:38.280 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b0c4b6603a] 2024-07-26T06:57:38.280 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fc1fb965407] 2024-07-26T06:57:38.280 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 20: (Thread::entry_wrapper()+0x43) [0x7f7803672cdd] 2024-07-26T06:57:38.280 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7803672cf9] 2024-07-26T06:57:38.280 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7802f66c02] 2024-07-26T06:57:38.280 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7802febc40] 2024-07-26T06:57:38.280 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:38.280 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 2024-07-26T06:57:38.280 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 0> 2024-07-26T06:57:38.004+0000 7f77fdb44640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:38.281 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: in thread 7f77fdb44640 thread_name:ms_dispatch 2024-07-26T06:57:38.281 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 2024-07-26T06:57:38.281 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:38.281 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7802f1b6f0] 2024-07-26T06:57:38.281 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7802f6894c] 2024-07-26T06:57:38.281 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 3: raise() 2024-07-26T06:57:38.281 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 4: abort() 2024-07-26T06:57:38.281 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7802f0571b] 2024-07-26T06:57:38.281 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 6: /lib64/libc.so.6(+0x37386) [0x7f7802f14386] 2024-07-26T06:57:38.281 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x564ed64c16b9] 2024-07-26T06:57:38.281 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x564ed667f80c] 2024-07-26T06:57:38.281 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x564ed65bf2ca] 2024-07-26T06:57:38.281 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564ed658f753] 2024-07-26T06:57:38.282 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564ed6590a1b] 2024-07-26T06:57:38.282 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564ed6574752] 2024-07-26T06:57:38.282 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564ed64c2a56] 2024-07-26T06:57:38.282 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x564ed64c6637] 2024-07-26T06:57:38.282 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x564ed64c6f5e] 2024-07-26T06:57:38.282 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x564ed64aa03a] 2024-07-26T06:57:38.282 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f78037e8407] 2024-07-26T06:57:38.282 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 18: (DispatchQueue::entry()+0x60f) [0x7fc1fb965cf9] 2024-07-26T06:57:38.283 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fc1fba425a1] 2024-07-26T06:57:38.283 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 20: (Thread::entry_wrapper()+0x43) [0x7fc1fb7efcdd] 2024-07-26T06:57:38.283 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc1fb7efcf9] 2024-07-26T06:57:38.283 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 22: /lib64/libc.so.6(+0x89c02) [0x7fc1fb0e3c02] 2024-07-26T06:57:38.283 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fc1fb168c40] 2024-07-26T06:57:38.283 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:38.283 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 2024-07-26T06:57:38.283 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 0> 2024-07-26T06:57:38.003+0000 7fc1f5cc1640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:38.283 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: in thread 7fc1f5cc1640 thread_name:ms_dispatch 2024-07-26T06:57:38.283 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 2024-07-26T06:57:38.283 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:38.283 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc1fb0986f0] 2024-07-26T06:57:38.283 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc1fb0e594c] 2024-07-26T06:57:38.284 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 3: raise() 2024-07-26T06:57:38.284 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 4: abort() 2024-07-26T06:57:38.284 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 5: /lib64/libc.so.6(+0x2871b) [0x7fc1fb08271b] 2024-07-26T06:57:38.284 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 6: /lib64/libc.so.6(+0x37386) [0x7fc1fb091386] 2024-07-26T06:57:38.284 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b0c4b7d6b9] 2024-07-26T06:57:38.284 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b0c4d3b80c] 2024-07-26T06:57:38.284 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b0c4c7b2ca] 2024-07-26T06:57:38.284 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b0c4c4b753] 2024-07-26T06:57:38.284 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b0c4c4ca1b] 2024-07-26T06:57:38.284 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b0c4c30752] 2024-07-26T06:57:38.284 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b0c4b7ea56] 2024-07-26T06:57:38.284 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b0c4b82637] 2024-07-26T06:57:38.284 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b0c4b82f5e] 2024-07-26T06:57:38.285 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 18: (DispatchQueue::entry()+0x60f) [0x7f78037e8cf9] 2024-07-26T06:57:38.285 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f78038c55a1] 2024-07-26T06:57:38.285 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 20: (Thread::entry_wrapper()+0x43) [0x7f7803672cdd] 2024-07-26T06:57:38.285 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7803672cf9] 2024-07-26T06:57:38.285 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7802f66c02] 2024-07-26T06:57:38.285 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7802febc40] 2024-07-26T06:57:38.285 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:38.286 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 2024-07-26T06:57:38.286 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 0> 2024-07-26T06:57:38.004+0000 7f77fdb44640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:38.286 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: in thread 7f77fdb44640 thread_name:ms_dispatch 2024-07-26T06:57:38.286 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 2024-07-26T06:57:38.286 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:38.286 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7802f1b6f0] 2024-07-26T06:57:38.286 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7802f6894c] 2024-07-26T06:57:38.286 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 3: raise() 2024-07-26T06:57:38.286 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 4: abort() 2024-07-26T06:57:38.287 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7802f0571b] 2024-07-26T06:57:38.287 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 6: /lib64/libc.so.6(+0x37386) [0x7f7802f14386] 2024-07-26T06:57:38.287 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x564ed64c16b9] 2024-07-26T06:57:38.287 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x564ed667f80c] 2024-07-26T06:57:38.287 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x564ed65bf2ca] 2024-07-26T06:57:38.287 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x564ed658f753] 2024-07-26T06:57:38.287 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x564ed6590a1b] 2024-07-26T06:57:38.287 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x564ed6574752] 2024-07-26T06:57:38.287 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x564ed64c2a56] 2024-07-26T06:57:38.287 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x564ed64c6637] 2024-07-26T06:57:38.287 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x564ed64c6f5e] 2024-07-26T06:57:38.287 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x564ed64aa03a] 2024-07-26T06:57:38.287 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f78037e8407] 2024-07-26T06:57:38.288 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 18: (DispatchQueue::entry()+0x60f) [0x7f78037e8cf9] 2024-07-26T06:57:38.288 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f78038c55a1] 2024-07-26T06:57:38.288 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 20: (Thread::entry_wrapper()+0x43) [0x7f7803672cdd] 2024-07-26T06:57:38.288 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7803672cf9] 2024-07-26T06:57:38.288 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7802f66c02] 2024-07-26T06:57:38.288 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7802febc40] 2024-07-26T06:57:38.288 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:38.288 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71355]: 2024-07-26T06:57:38.288 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b0c4b6603a] 2024-07-26T06:57:38.288 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fc1fb965407] 2024-07-26T06:57:38.288 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 18: (DispatchQueue::entry()+0x60f) [0x7fc1fb965cf9] 2024-07-26T06:57:38.288 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fc1fba425a1] 2024-07-26T06:57:38.289 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 20: (Thread::entry_wrapper()+0x43) [0x7fc1fb7efcdd] 2024-07-26T06:57:38.289 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc1fb7efcf9] 2024-07-26T06:57:38.289 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 22: /lib64/libc.so.6(+0x89c02) [0x7fc1fb0e3c02] 2024-07-26T06:57:38.289 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fc1fb168c40] 2024-07-26T06:57:38.289 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:38.289 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71252]: 2024-07-26T06:57:38.289 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 podman[71423]: 2024-07-26 06:57:38.065227706 +0000 UTC m=+0.019788108 container died 439da4e38b4aa7fb023fa80fbb154c22db751d2014f4cfe96d9e41ac9f4f4cff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-07-26T06:57:38.311 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:57:38.310+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:57:38.311 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:57:38.310+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:57:38.311 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:57:38.310+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:57:38.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:37 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/734940828,v1:172.21.15.163:6839/734940828] up:rejoin 2024-07-26T06:57:38.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:37 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/4111555088,v1:172.21.15.163:6835/4111555088] up:rejoin 2024-07-26T06:57:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:37 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/850726105,v1:172.21.15.163:6837/850726105] up:rejoin 2024-07-26T06:57:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:37 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:57:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:37 smithi042 ceph-mon[32775]: daemon mds.b is now active in filesystem cephfs as rank 2 2024-07-26T06:57:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:37 smithi042 ceph-mon[32775]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-07-26T06:57:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:37 smithi042 ceph-mon[32775]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-07-26T06:57:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:37 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/734940828,v1:172.21.15.163:6839/734940828] up:rejoin 2024-07-26T06:57:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:37 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/4111555088,v1:172.21.15.163:6835/4111555088] up:rejoin 2024-07-26T06:57:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:37 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/850726105,v1:172.21.15.163:6837/850726105] up:rejoin 2024-07-26T06:57:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:37 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:57:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:37 smithi164 ceph-mon[36970]: daemon mds.b is now active in filesystem cephfs as rank 2 2024-07-26T06:57:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:37 smithi164 ceph-mon[36970]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-07-26T06:57:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:37 smithi164 ceph-mon[36970]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-07-26T06:57:38.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:57:38.722+0000 7faa2dcb9640 1 -- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 msgr2=0x561ca20d0cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:57:38.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:57:38.722+0000 7faa2dcb9640 1 --2- 172.21.15.42:0/2285256131 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x561ca1ccc1d0 0x561ca20d0cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:57:38.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-07-26T06:57:38.722+0000 7faa117fa640 1 client.16095 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:57:39.106 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 podman[71423]: 2024-07-26 06:57:38.769878443 +0000 UTC m=+0.724438910 container remove 439da4e38b4aa7fb023fa80fbb154c22db751d2014f4cfe96d9e41ac9f4f4cff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-07-26T06:57:39.107 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:38 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:57:39.107 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-mon[36873]: pgmap v638: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-07-26T06:57:39.107 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-mon[36873]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:57:39.107 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-mon[36873]: Cluster is now healthy 2024-07-26T06:57:39.107 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/734940828,v1:172.21.15.163:6839/734940828] up:active 2024-07-26T06:57:39.107 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/4111555088,v1:172.21.15.163:6835/4111555088] up:active 2024-07-26T06:57:39.107 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/850726105,v1:172.21.15.163:6837/850726105] up:active 2024-07-26T06:57:39.107 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-mon[36873]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:57:39.107 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:38 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:57:39.108 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:38 smithi163 podman[71429]: 2024-07-26 06:57:38.807934672 +0000 UTC m=+0.754881019 container died 57543711caff8acd1bab1301fdebdf5eb4a4a9ee66881151a7eaf86e1e76d555 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS) 2024-07-26T06:57:39.373 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:38 smithi164 ceph-mon[36970]: pgmap v638: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-07-26T06:57:39.373 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:38 smithi164 ceph-mon[36970]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:57:39.373 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:38 smithi164 ceph-mon[36970]: Cluster is now healthy 2024-07-26T06:57:39.373 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:38 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/734940828,v1:172.21.15.163:6839/734940828] up:active 2024-07-26T06:57:39.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:38 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/4111555088,v1:172.21.15.163:6835/4111555088] up:active 2024-07-26T06:57:39.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:38 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/850726105,v1:172.21.15.163:6837/850726105] up:active 2024-07-26T06:57:39.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:38 smithi164 ceph-mon[36970]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:57:39.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:38 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:57:39.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:38 smithi042 ceph-mon[32775]: pgmap v638: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-07-26T06:57:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:38 smithi042 ceph-mon[32775]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:57:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:38 smithi042 ceph-mon[32775]: Cluster is now healthy 2024-07-26T06:57:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:38 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/734940828,v1:172.21.15.163:6839/734940828] up:active 2024-07-26T06:57:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:38 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/4111555088,v1:172.21.15.163:6835/4111555088] up:active 2024-07-26T06:57:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:38 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/850726105,v1:172.21.15.163:6837/850726105] up:active 2024-07-26T06:57:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:38 smithi042 ceph-mon[32775]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:57:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:38 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:57:39.754 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:39 smithi163 podman[71430]: 2024-07-26 06:57:39.436542732 +0000 UTC m=+1.383824181 container died 390e1aebaa15489b885e957e73b09c261d81176a6489d10fba2d9fd1a28c26ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T06:57:39.755 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:39 smithi163 podman[71429]: 2024-07-26 06:57:39.40007304 +0000 UTC m=+1.347019389 container remove 57543711caff8acd1bab1301fdebdf5eb4a4a9ee66881151a7eaf86e1e76d555 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2) 2024-07-26T06:57:39.755 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:39 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:57:40.313 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:40 smithi163 podman[71430]: 2024-07-26 06:57:40.055321779 +0000 UTC m=+2.002603273 container remove 390e1aebaa15489b885e957e73b09c261d81176a6489d10fba2d9fd1a28c26ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T06:57:40.313 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:40 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:57:40.756 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:40 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T06:57:40.757 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:40 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T06:57:40.757 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:40 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Consumed 1.166s CPU time. 2024-07-26T06:57:40.757 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:40 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-26T06:57:40.757 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:40 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Consumed 1.056s CPU time. 2024-07-26T06:57:41.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:41 smithi163 ceph-mon[36873]: pgmap v639: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 586 B/s wr, 15 op/s 2024-07-26T06:57:41.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:40 smithi042 ceph-mon[32775]: pgmap v639: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 586 B/s wr, 15 op/s 2024-07-26T06:57:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:41 smithi164 ceph-mon[36970]: pgmap v639: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 586 B/s wr, 15 op/s 2024-07-26T06:57:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:43 smithi163 ceph-mon[36873]: pgmap v640: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 618 B/s wr, 14 op/s 2024-07-26T06:57:43.271 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:43 smithi164 ceph-mon[36970]: pgmap v640: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 618 B/s wr, 14 op/s 2024-07-26T06:57:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:43 smithi042 ceph-mon[32775]: pgmap v640: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 618 B/s wr, 14 op/s 2024-07-26T06:57:43.814 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-07-26T06:57:43.815 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't parse response with error Expecting value: line 1 column 1 (char 0); rc: 124 (EMEDIUMTYPE); stdout: 2024-07-26T06:57:43.816 ERROR:tasks.quiescer.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/quiescer.py", line 294, in do_quiesce response = json.loads(stdout) File "/usr/lib/python3.10/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib/python3.10/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/quiescer.py", line 357, in _run self.do_quiesce(d) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/quiescer.py", line 298, in do_quiesce raise RuntimeError(f"Error parsing quiesce response: {e}") RuntimeError: Error parsing quiesce response: Expecting value: line 1 column 1 (char 0) 2024-07-26T06:57:45.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:45 smithi163 ceph-mon[36873]: pgmap v641: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 614 B/s wr, 13 op/s 2024-07-26T06:57:45.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:45 smithi042 ceph-mon[32775]: pgmap v641: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 614 B/s wr, 13 op/s 2024-07-26T06:57:45.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:45 smithi164 ceph-mon[36970]: pgmap v641: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 614 B/s wr, 13 op/s 2024-07-26T06:57:47.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:47 smithi042 ceph-mon[32775]: pgmap v642: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 527 B/s wr, 13 op/s 2024-07-26T06:57:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:47 smithi164 ceph-mon[36970]: pgmap v642: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 527 B/s wr, 13 op/s 2024-07-26T06:57:47.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:47 smithi163 ceph-mon[36873]: pgmap v642: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 527 B/s wr, 13 op/s 2024-07-26T06:57:49.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:49 smithi042 ceph-mon[32775]: pgmap v643: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 511 B/s wr, 12 op/s 2024-07-26T06:57:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:49 smithi164 ceph-mon[36970]: pgmap v643: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 511 B/s wr, 12 op/s 2024-07-26T06:57:49.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:49 smithi163 ceph-mon[36873]: pgmap v643: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 511 B/s wr, 12 op/s 2024-07-26T06:57:50.760 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:50 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 4. 2024-07-26T06:57:50.760 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:50 smithi163 systemd[1]: Stopped Ceph mds.b for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:50.760 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:50 smithi163 systemd[1]: Starting Ceph mds.b for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:57:50.761 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:50 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 4. 2024-07-26T06:57:50.761 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:50 smithi163 systemd[1]: Stopped Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:50.761 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:50 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Consumed 1.056s CPU time. 2024-07-26T06:57:50.761 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:50 smithi163 systemd[1]: Starting Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:57:50.761 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:50 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 4. 2024-07-26T06:57:50.761 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:50 smithi163 systemd[1]: Stopped Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:50.761 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:50 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Consumed 1.166s CPU time. 2024-07-26T06:57:50.761 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:50 smithi163 systemd[1]: Starting Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:57:50.762 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:50 smithi163 podman[71691]: 2024-07-26 06:57:50.660099604 +0000 UTC m=+0.018867218 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:57:51.016 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:50 smithi163 podman[71690]: 2024-07-26 06:57:50.663214731 +0000 UTC m=+0.021627247 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:57:51.016 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:50 smithi163 podman[71691]: 2024-07-26 06:57:50.815566906 +0000 UTC m=+0.174334507 container create a05826108af7fbec56367fd19716c45523f9c2d98b81190335da0924dca14e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T06:57:51.016 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:50 smithi163 podman[71704]: 2024-07-26 06:57:50.698346448 +0000 UTC m=+0.050713956 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:57:51.370 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:51 smithi163 ceph-mon[36873]: pgmap v644: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 511 B/s wr, 13 op/s 2024-07-26T06:57:51.371 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:51 smithi163 podman[71691]: 2024-07-26 06:57:51.136443141 +0000 UTC m=+0.495210743 container init a05826108af7fbec56367fd19716c45523f9c2d98b81190335da0924dca14e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-07-26T06:57:51.371 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:51 smithi163 podman[71691]: 2024-07-26 06:57:51.139197059 +0000 UTC m=+0.497964662 container start a05826108af7fbec56367fd19716c45523f9c2d98b81190335da0924dca14e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-07-26T06:57:51.371 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:51 smithi163 ceph-mds[71738]: -- 172.21.15.163:0/2932758164 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55640b7b7860 con 0x55640f959600 2024-07-26T06:57:51.371 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:51 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: starting mds.e at 2024-07-26T06:57:51.371 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:51 smithi163 bash[71691]: a05826108af7fbec56367fd19716c45523f9c2d98b81190335da0924dca14e52 2024-07-26T06:57:51.371 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:57:51 smithi163 systemd[1]: Started Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:51.371 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:51 smithi163 podman[71690]: 2024-07-26 06:57:51.035961445 +0000 UTC m=+0.394373957 container create 7d7ea37a484b7929fae7ff42deaf12bfb3a5fc34af3a2f0afe1bde88ec478ab1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-07-26T06:57:51.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:51 smithi042 ceph-mon[32775]: pgmap v644: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 511 B/s wr, 13 op/s 2024-07-26T06:57:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:51 smithi164 ceph-mon[36970]: pgmap v644: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 511 B/s wr, 13 op/s 2024-07-26T06:57:51.656 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:51 smithi163 podman[71704]: 2024-07-26 06:57:51.370742217 +0000 UTC m=+0.723109763 container create 9828358e61fb570143edaedd969eeb83b66f397c23ed662dea795dbe31c6511e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS) 2024-07-26T06:57:52.001 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:51 smithi163 podman[71704]: 2024-07-26 06:57:51.656825389 +0000 UTC m=+1.009192909 container init 9828358e61fb570143edaedd969eeb83b66f397c23ed662dea795dbe31c6511e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724) 2024-07-26T06:57:52.002 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:51 smithi163 podman[71704]: 2024-07-26 06:57:51.65939528 +0000 UTC m=+1.011762788 container start 9828358e61fb570143edaedd969eeb83b66f397c23ed662dea795dbe31c6511e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T06:57:52.002 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:51 smithi163 ceph-mds[71764]: -- 172.21.15.163:0/3040446858 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5602a1ae5860 con 0x5602a5c87180 2024-07-26T06:57:52.002 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:51 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: starting mds.b at 2024-07-26T06:57:52.002 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:51 smithi163 bash[71704]: 9828358e61fb570143edaedd969eeb83b66f397c23ed662dea795dbe31c6511e 2024-07-26T06:57:52.002 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:57:51 smithi163 systemd[1]: Started Ceph mds.b for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:52.257 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:52 smithi163 podman[71690]: 2024-07-26 06:57:52.00184606 +0000 UTC m=+1.360258589 container init 7d7ea37a484b7929fae7ff42deaf12bfb3a5fc34af3a2f0afe1bde88ec478ab1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0) 2024-07-26T06:57:52.257 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:52 smithi163 podman[71690]: 2024-07-26 06:57:52.004738891 +0000 UTC m=+1.363151411 container start 7d7ea37a484b7929fae7ff42deaf12bfb3a5fc34af3a2f0afe1bde88ec478ab1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-07-26T06:57:52.257 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:52 smithi163 ceph-mds[71791]: -- 172.21.15.163:0/1997634518 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5612856ff860 con 0x5612898a1a80 2024-07-26T06:57:52.257 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:52 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: starting mds.h at 2024-07-26T06:57:52.257 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:52 smithi163 bash[71690]: 7d7ea37a484b7929fae7ff42deaf12bfb3a5fc34af3a2f0afe1bde88ec478ab1 2024-07-26T06:57:52.258 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:57:52 smithi163 systemd[1]: Started Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:57:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:52 smithi163 ceph-mon[36873]: daemon mds.e restarted 2024-07-26T06:57:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:52 smithi163 ceph-mon[36873]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:57:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:52 smithi163 ceph-mon[36873]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T06:57:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:52 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/873838612,v1:172.21.15.163:6835/873838612] up:boot 2024-07-26T06:57:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:52 smithi163 ceph-mon[36873]: Standby daemon mds.a assigned to filesystem cephfs as rank 3 2024-07-26T06:57:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:52 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=b=up:active,4=h=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:57:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:57:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:52 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:57:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:52 smithi163 ceph-mon[36873]: daemon mds.b restarted 2024-07-26T06:57:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:52 smithi163 ceph-mon[36873]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T06:57:52.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:52 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/3101627593,v1:172.21.15.163:6837/3101627593] up:boot 2024-07-26T06:57:52.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:52 smithi163 ceph-mon[36873]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-07-26T06:57:52.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:52 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:replay,4=h=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:57:52.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T06:57:52.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:52 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-07-26T06:57:52.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:52 smithi042 ceph-mon[32775]: daemon mds.e restarted 2024-07-26T06:57:52.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:52 smithi042 ceph-mon[32775]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:57:52.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:52 smithi042 ceph-mon[32775]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T06:57:52.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:52 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/873838612,v1:172.21.15.163:6835/873838612] up:boot 2024-07-26T06:57:52.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:52 smithi042 ceph-mon[32775]: Standby daemon mds.a assigned to filesystem cephfs as rank 3 2024-07-26T06:57:52.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:52 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=b=up:active,4=h=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:57:52.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:57:52.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:52 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:57:52.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:52 smithi042 ceph-mon[32775]: daemon mds.b restarted 2024-07-26T06:57:52.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:52 smithi042 ceph-mon[32775]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T06:57:52.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:52 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/3101627593,v1:172.21.15.163:6837/3101627593] up:boot 2024-07-26T06:57:52.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:52 smithi042 ceph-mon[32775]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-07-26T06:57:52.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:52 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:replay,4=h=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:57:52.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T06:57:52.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:52 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-07-26T06:57:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:52 smithi164 ceph-mon[36970]: daemon mds.e restarted 2024-07-26T06:57:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:52 smithi164 ceph-mon[36970]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:57:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:52 smithi164 ceph-mon[36970]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T06:57:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:52 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/873838612,v1:172.21.15.163:6835/873838612] up:boot 2024-07-26T06:57:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:52 smithi164 ceph-mon[36970]: Standby daemon mds.a assigned to filesystem cephfs as rank 3 2024-07-26T06:57:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:52 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=b=up:active,4=h=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:57:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:57:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:52 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:57:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:52 smithi164 ceph-mon[36970]: daemon mds.b restarted 2024-07-26T06:57:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:52 smithi164 ceph-mon[36970]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T06:57:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:52 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/3101627593,v1:172.21.15.163:6837/3101627593] up:boot 2024-07-26T06:57:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:52 smithi164 ceph-mon[36970]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-07-26T06:57:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:52 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:replay,4=h=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:57:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T06:57:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:52 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-07-26T06:57:53.314 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:57:53.313+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:57:53.314 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:57:53.314+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T06:57:53.314 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:57:53.314+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T06:57:53.314 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:57:53.314+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T06:57:53.314 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:57:53.314+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:57:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:53 smithi042 ceph-mon[32775]: pgmap v646: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T06:57:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:53 smithi042 ceph-mon[32775]: daemon mds.h restarted 2024-07-26T06:57:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:53 smithi042 ceph-mon[32775]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T06:57:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:53 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/193486687,v1:172.21.15.163:6839/193486687] up:boot 2024-07-26T06:57:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:53 smithi042 ceph-mon[32775]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-07-26T06:57:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:53 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=l=up:replay,3=a=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:57:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:57:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:53 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:replay 2 up:active 2024-07-26T06:57:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:53 smithi164 ceph-mon[36970]: pgmap v646: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T06:57:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:53 smithi164 ceph-mon[36970]: daemon mds.h restarted 2024-07-26T06:57:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:53 smithi164 ceph-mon[36970]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T06:57:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:53 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/193486687,v1:172.21.15.163:6839/193486687] up:boot 2024-07-26T06:57:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:53 smithi164 ceph-mon[36970]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-07-26T06:57:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:53 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=l=up:replay,3=a=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:57:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:57:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:53 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:replay 2 up:active 2024-07-26T06:57:53.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:53 smithi163 ceph-mon[36873]: pgmap v646: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T06:57:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:53 smithi163 ceph-mon[36873]: daemon mds.h restarted 2024-07-26T06:57:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:53 smithi163 ceph-mon[36873]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T06:57:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:53 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/193486687,v1:172.21.15.163:6839/193486687] up:boot 2024-07-26T06:57:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:53 smithi163 ceph-mon[36873]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-07-26T06:57:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:53 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=l=up:replay,3=a=up:replay} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:57:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:57:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:53 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:replay 2 up:active 2024-07-26T06:57:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:54 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/351656606,v1:172.21.15.164:6833/351656606] up:resolve 2024-07-26T06:57:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:54 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3863572604,v1:172.21.15.42:6839/3863572604] up:resolve 2024-07-26T06:57:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:54 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/1877922073,v1:172.21.15.42:6841/1877922073] up:resolve 2024-07-26T06:57:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:54 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:57:54.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:54 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/351656606,v1:172.21.15.164:6833/351656606] up:resolve 2024-07-26T06:57:54.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:54 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3863572604,v1:172.21.15.42:6839/3863572604] up:resolve 2024-07-26T06:57:54.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:54 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/1877922073,v1:172.21.15.42:6841/1877922073] up:resolve 2024-07-26T06:57:54.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:54 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:57:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:54 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/351656606,v1:172.21.15.164:6833/351656606] up:resolve 2024-07-26T06:57:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:54 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3863572604,v1:172.21.15.42:6839/3863572604] up:resolve 2024-07-26T06:57:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:54 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/1877922073,v1:172.21.15.42:6841/1877922073] up:resolve 2024-07-26T06:57:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:54 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:57:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:55 smithi042 ceph-mon[32775]: pgmap v649: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 10 op/s 2024-07-26T06:57:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:55 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999991 2024-07-26T06:57:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:55 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999991 2024-07-26T06:57:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:55 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999983 2024-07-26T06:57:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:55 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/1877922073,v1:172.21.15.42:6841/1877922073] up:reconnect 2024-07-26T06:57:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:55 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/351656606,v1:172.21.15.164:6833/351656606] up:reconnect 2024-07-26T06:57:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:55 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3863572604,v1:172.21.15.42:6839/3863572604] up:reconnect 2024-07-26T06:57:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:55 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:57:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:55 smithi164 ceph-mon[36970]: pgmap v649: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 10 op/s 2024-07-26T06:57:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:55 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999991 2024-07-26T06:57:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:55 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999991 2024-07-26T06:57:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:55 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999983 2024-07-26T06:57:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:55 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/1877922073,v1:172.21.15.42:6841/1877922073] up:reconnect 2024-07-26T06:57:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:55 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/351656606,v1:172.21.15.164:6833/351656606] up:reconnect 2024-07-26T06:57:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:55 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3863572604,v1:172.21.15.42:6839/3863572604] up:reconnect 2024-07-26T06:57:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:55 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:57:55.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:55 smithi163 ceph-mon[36873]: pgmap v649: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 10 op/s 2024-07-26T06:57:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:55 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999991 2024-07-26T06:57:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:55 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999991 2024-07-26T06:57:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:55 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999983 2024-07-26T06:57:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:55 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/1877922073,v1:172.21.15.42:6841/1877922073] up:reconnect 2024-07-26T06:57:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:55 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/351656606,v1:172.21.15.164:6833/351656606] up:reconnect 2024-07-26T06:57:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:55 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3863572604,v1:172.21.15.42:6839/3863572604] up:reconnect 2024-07-26T06:57:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:55 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:57:56.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3863572604,v1:172.21.15.42:6839/3863572604] up:rejoin 2024-07-26T06:57:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/1877922073,v1:172.21.15.42:6841/1877922073] up:rejoin 2024-07-26T06:57:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/351656606,v1:172.21.15.164:6833/351656606] up:rejoin 2024-07-26T06:57:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:57:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-mon[32775]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:57:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-mon[32775]: daemon mds.a is now active in filesystem cephfs as rank 3 2024-07-26T06:57:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-mon[32775]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-07-26T06:57:56.395 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:57:56.396 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: *** Caught signal (Aborted) ** 2024-07-26T06:57:56.396 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: in thread 7f7098ce2640 thread_name:ms_dispatch 2024-07-26T06:57:56.396 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:56.396 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f709e0b96f0] 2024-07-26T06:57:56.396 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f709e10694c] 2024-07-26T06:57:56.396 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 3: raise() 2024-07-26T06:57:56.396 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 4: abort() 2024-07-26T06:57:56.396 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 5: /lib64/libc.so.6(+0x2871b) [0x7f709e0a371b] 2024-07-26T06:57:56.396 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 6: /lib64/libc.so.6(+0x37386) [0x7f709e0b2386] 2024-07-26T06:57:56.396 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5611a247b6b9] 2024-07-26T06:57:56.396 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5611a263980c] 2024-07-26T06:57:56.396 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5611a25792ca] 2024-07-26T06:57:56.397 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5611a2549753] 2024-07-26T06:57:56.397 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5611a254aa1b] 2024-07-26T06:57:56.397 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5611a252e752] 2024-07-26T06:57:56.397 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5611a247ca56] 2024-07-26T06:57:56.397 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5611a2480637] 2024-07-26T06:57:56.397 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5611a2480f5e] 2024-07-26T06:57:56.397 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5611a246403a] 2024-07-26T06:57:56.397 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f709e986407] 2024-07-26T06:57:56.397 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 18: (DispatchQueue::entry()+0x60f) [0x7f709e986cf9] 2024-07-26T06:57:56.397 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f709ea635a1] 2024-07-26T06:57:56.398 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:57:56.398 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: *** Caught signal (Aborted) ** 2024-07-26T06:57:56.398 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: in thread 7f0c35603640 thread_name:ms_dispatch 2024-07-26T06:57:56.398 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:56.398 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0c3a9da6f0] 2024-07-26T06:57:56.398 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0c3aa2794c] 2024-07-26T06:57:56.398 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 3: raise() 2024-07-26T06:57:56.398 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 4: abort() 2024-07-26T06:57:56.398 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0c3a9c471b] 2024-07-26T06:57:56.398 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 6: /lib64/libc.so.6(+0x37386) [0x7f0c3a9d3386] 2024-07-26T06:57:56.398 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5618837206b9] 2024-07-26T06:57:56.398 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5618838de80c] 2024-07-26T06:57:56.399 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56188381e2ca] 2024-07-26T06:57:56.399 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5618837ee753] 2024-07-26T06:57:56.399 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5618837efa1b] 2024-07-26T06:57:56.399 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5618837d3752] 2024-07-26T06:57:56.399 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561883721a56] 2024-07-26T06:57:56.399 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561883725637] 2024-07-26T06:57:56.399 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561883725f5e] 2024-07-26T06:57:56.399 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56188370903a] 2024-07-26T06:57:56.399 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0c3b2a7407] 2024-07-26T06:57:56.399 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 18: (DispatchQueue::entry()+0x60f) [0x7f0c3b2a7cf9] 2024-07-26T06:57:56.399 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0c3b3845a1] 2024-07-26T06:57:56.400 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 20: (Thread::entry_wrapper()+0x43) [0x7f709e810cdd] 2024-07-26T06:57:56.400 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 21: (Thread::_entry_func(void*)+0xd) [0x7f709e810cf9] 2024-07-26T06:57:56.400 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 22: /lib64/libc.so.6(+0x89c02) [0x7f709e104c02] 2024-07-26T06:57:56.400 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f709e189c40] 2024-07-26T06:57:56.400 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 2024-07-26T06:57:56.080+0000 7f7098ce2640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:56.400 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: in thread 7f7098ce2640 thread_name:ms_dispatch 2024-07-26T06:57:56.400 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 2024-07-26T06:57:56.400 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:56.400 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f709e0b96f0] 2024-07-26T06:57:56.400 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f709e10694c] 2024-07-26T06:57:56.400 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 3: raise() 2024-07-26T06:57:56.401 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 4: abort() 2024-07-26T06:57:56.401 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 5: /lib64/libc.so.6(+0x2871b) [0x7f709e0a371b] 2024-07-26T06:57:56.401 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 6: /lib64/libc.so.6(+0x37386) [0x7f709e0b2386] 2024-07-26T06:57:56.401 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5611a247b6b9] 2024-07-26T06:57:56.401 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5611a263980c] 2024-07-26T06:57:56.401 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5611a25792ca] 2024-07-26T06:57:56.401 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5611a2549753] 2024-07-26T06:57:56.401 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5611a254aa1b] 2024-07-26T06:57:56.401 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5611a252e752] 2024-07-26T06:57:56.401 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5611a247ca56] 2024-07-26T06:57:56.401 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5611a2480637] 2024-07-26T06:57:56.401 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5611a2480f5e] 2024-07-26T06:57:56.401 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5611a246403a] 2024-07-26T06:57:56.402 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f709e986407] 2024-07-26T06:57:56.402 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 18: (DispatchQueue::entry()+0x60f) [0x7f709e986cf9] 2024-07-26T06:57:56.402 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f709ea635a1] 2024-07-26T06:57:56.402 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 20: (Thread::entry_wrapper()+0x43) [0x7f0c3b131cdd] 2024-07-26T06:57:56.402 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0c3b131cf9] 2024-07-26T06:57:56.402 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0c3aa25c02] 2024-07-26T06:57:56.402 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0c3aaaac40] 2024-07-26T06:57:56.402 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 2024-07-26T06:57:56.083+0000 7f0c35603640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:56.402 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: in thread 7f0c35603640 thread_name:ms_dispatch 2024-07-26T06:57:56.402 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 2024-07-26T06:57:56.403 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:56.403 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0c3a9da6f0] 2024-07-26T06:57:56.403 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0c3aa2794c] 2024-07-26T06:57:56.403 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 3: raise() 2024-07-26T06:57:56.403 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 4: abort() 2024-07-26T06:57:56.403 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0c3a9c471b] 2024-07-26T06:57:56.403 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 6: /lib64/libc.so.6(+0x37386) [0x7f0c3a9d3386] 2024-07-26T06:57:56.403 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5618837206b9] 2024-07-26T06:57:56.403 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5618838de80c] 2024-07-26T06:57:56.403 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56188381e2ca] 2024-07-26T06:57:56.403 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5618837ee753] 2024-07-26T06:57:56.403 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5618837efa1b] 2024-07-26T06:57:56.403 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5618837d3752] 2024-07-26T06:57:56.403 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561883721a56] 2024-07-26T06:57:56.403 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561883725637] 2024-07-26T06:57:56.404 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561883725f5e] 2024-07-26T06:57:56.404 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56188370903a] 2024-07-26T06:57:56.404 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0c3b2a7407] 2024-07-26T06:57:56.404 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 18: (DispatchQueue::entry()+0x60f) [0x7f0c3b2a7cf9] 2024-07-26T06:57:56.404 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0c3b3845a1] 2024-07-26T06:57:56.404 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 20: (Thread::entry_wrapper()+0x43) [0x7f709e810cdd] 2024-07-26T06:57:56.404 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 21: (Thread::_entry_func(void*)+0xd) [0x7f709e810cf9] 2024-07-26T06:57:56.404 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 22: /lib64/libc.so.6(+0x89c02) [0x7f709e104c02] 2024-07-26T06:57:56.404 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f709e189c40] 2024-07-26T06:57:56.404 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:56.404 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 2024-07-26T06:57:56.404 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 0> 2024-07-26T06:57:56.080+0000 7f7098ce2640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:56.405 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: in thread 7f7098ce2640 thread_name:ms_dispatch 2024-07-26T06:57:56.405 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 2024-07-26T06:57:56.405 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:56.405 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f709e0b96f0] 2024-07-26T06:57:56.405 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f709e10694c] 2024-07-26T06:57:56.405 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 3: raise() 2024-07-26T06:57:56.405 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 4: abort() 2024-07-26T06:57:56.405 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 5: /lib64/libc.so.6(+0x2871b) [0x7f709e0a371b] 2024-07-26T06:57:56.405 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 6: /lib64/libc.so.6(+0x37386) [0x7f709e0b2386] 2024-07-26T06:57:56.405 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5611a247b6b9] 2024-07-26T06:57:56.405 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5611a263980c] 2024-07-26T06:57:56.405 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5611a25792ca] 2024-07-26T06:57:56.405 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5611a2549753] 2024-07-26T06:57:56.405 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5611a254aa1b] 2024-07-26T06:57:56.406 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5611a252e752] 2024-07-26T06:57:56.406 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5611a247ca56] 2024-07-26T06:57:56.406 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5611a2480637] 2024-07-26T06:57:56.406 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5611a2480f5e] 2024-07-26T06:57:56.406 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5611a246403a] 2024-07-26T06:57:56.406 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 20: (Thread::entry_wrapper()+0x43) [0x7f0c3b131cdd] 2024-07-26T06:57:56.406 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0c3b131cf9] 2024-07-26T06:57:56.406 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0c3aa25c02] 2024-07-26T06:57:56.406 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0c3aaaac40] 2024-07-26T06:57:56.407 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:56.407 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 2024-07-26T06:57:56.407 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 0> 2024-07-26T06:57:56.083+0000 7f0c35603640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:56.407 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: in thread 7f0c35603640 thread_name:ms_dispatch 2024-07-26T06:57:56.407 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 2024-07-26T06:57:56.407 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:56.407 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0c3a9da6f0] 2024-07-26T06:57:56.407 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0c3aa2794c] 2024-07-26T06:57:56.407 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 3: raise() 2024-07-26T06:57:56.407 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 4: abort() 2024-07-26T06:57:56.407 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0c3a9c471b] 2024-07-26T06:57:56.407 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 6: /lib64/libc.so.6(+0x37386) [0x7f0c3a9d3386] 2024-07-26T06:57:56.408 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5618837206b9] 2024-07-26T06:57:56.408 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5618838de80c] 2024-07-26T06:57:56.408 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56188381e2ca] 2024-07-26T06:57:56.408 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5618837ee753] 2024-07-26T06:57:56.408 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5618837efa1b] 2024-07-26T06:57:56.408 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5618837d3752] 2024-07-26T06:57:56.408 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561883721a56] 2024-07-26T06:57:56.408 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561883725637] 2024-07-26T06:57:56.408 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561883725f5e] 2024-07-26T06:57:56.408 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56188370903a] 2024-07-26T06:57:56.409 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f709e986407] 2024-07-26T06:57:56.409 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 18: (DispatchQueue::entry()+0x60f) [0x7f709e986cf9] 2024-07-26T06:57:56.409 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f709ea635a1] 2024-07-26T06:57:56.409 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 20: (Thread::entry_wrapper()+0x43) [0x7f709e810cdd] 2024-07-26T06:57:56.409 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 21: (Thread::_entry_func(void*)+0xd) [0x7f709e810cf9] 2024-07-26T06:57:56.409 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 22: /lib64/libc.so.6(+0x89c02) [0x7f709e104c02] 2024-07-26T06:57:56.409 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f709e189c40] 2024-07-26T06:57:56.409 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:56.409 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 2024-07-26T06:57:56.409 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 0> 2024-07-26T06:57:56.080+0000 7f7098ce2640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:56.409 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: in thread 7f7098ce2640 thread_name:ms_dispatch 2024-07-26T06:57:56.410 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 2024-07-26T06:57:56.410 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:56.410 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f709e0b96f0] 2024-07-26T06:57:56.410 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f709e10694c] 2024-07-26T06:57:56.410 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 3: raise() 2024-07-26T06:57:56.410 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 4: abort() 2024-07-26T06:57:56.410 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 5: /lib64/libc.so.6(+0x2871b) [0x7f709e0a371b] 2024-07-26T06:57:56.410 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 6: /lib64/libc.so.6(+0x37386) [0x7f709e0b2386] 2024-07-26T06:57:56.410 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5611a247b6b9] 2024-07-26T06:57:56.410 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5611a263980c] 2024-07-26T06:57:56.410 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5611a25792ca] 2024-07-26T06:57:56.410 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5611a2549753] 2024-07-26T06:57:56.410 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5611a254aa1b] 2024-07-26T06:57:56.411 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5611a252e752] 2024-07-26T06:57:56.411 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5611a247ca56] 2024-07-26T06:57:56.411 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5611a2480637] 2024-07-26T06:57:56.411 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5611a2480f5e] 2024-07-26T06:57:56.411 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5611a246403a] 2024-07-26T06:57:56.411 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f709e986407] 2024-07-26T06:57:56.411 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 18: (DispatchQueue::entry()+0x60f) [0x7f709e986cf9] 2024-07-26T06:57:56.411 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f709ea635a1] 2024-07-26T06:57:56.411 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 20: (Thread::entry_wrapper()+0x43) [0x7f709e810cdd] 2024-07-26T06:57:56.411 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 21: (Thread::_entry_func(void*)+0xd) [0x7f709e810cf9] 2024-07-26T06:57:56.411 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 22: /lib64/libc.so.6(+0x89c02) [0x7f709e104c02] 2024-07-26T06:57:56.411 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f709e189c40] 2024-07-26T06:57:56.411 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:56.411 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119456]: 2024-07-26T06:57:56.411 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:56 smithi042 podman[119608]: 2024-07-26 06:57:56.156881211 +0000 UTC m=+0.024099895 container died f225d4b8aad6754fce903925827868192e05a9ed3a09c37689763cebc83a1e9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-07-26T06:57:56.412 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0c3b2a7407] 2024-07-26T06:57:56.412 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 18: (DispatchQueue::entry()+0x60f) [0x7f0c3b2a7cf9] 2024-07-26T06:57:56.412 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0c3b3845a1] 2024-07-26T06:57:56.412 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 20: (Thread::entry_wrapper()+0x43) [0x7f0c3b131cdd] 2024-07-26T06:57:56.412 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0c3b131cf9] 2024-07-26T06:57:56.412 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0c3aa25c02] 2024-07-26T06:57:56.412 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0c3aaaac40] 2024-07-26T06:57:56.412 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:56.412 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 2024-07-26T06:57:56.412 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 0> 2024-07-26T06:57:56.083+0000 7f0c35603640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:56.413 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: in thread 7f0c35603640 thread_name:ms_dispatch 2024-07-26T06:57:56.413 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 2024-07-26T06:57:56.413 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:56.413 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0c3a9da6f0] 2024-07-26T06:57:56.413 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0c3aa2794c] 2024-07-26T06:57:56.413 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 3: raise() 2024-07-26T06:57:56.413 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 4: abort() 2024-07-26T06:57:56.413 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0c3a9c471b] 2024-07-26T06:57:56.413 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 6: /lib64/libc.so.6(+0x37386) [0x7f0c3a9d3386] 2024-07-26T06:57:56.413 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5618837206b9] 2024-07-26T06:57:56.413 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5618838de80c] 2024-07-26T06:57:56.413 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56188381e2ca] 2024-07-26T06:57:56.413 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5618837ee753] 2024-07-26T06:57:56.413 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5618837efa1b] 2024-07-26T06:57:56.414 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5618837d3752] 2024-07-26T06:57:56.414 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561883721a56] 2024-07-26T06:57:56.414 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561883725637] 2024-07-26T06:57:56.414 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561883725f5e] 2024-07-26T06:57:56.414 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56188370903a] 2024-07-26T06:57:56.414 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0c3b2a7407] 2024-07-26T06:57:56.414 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 18: (DispatchQueue::entry()+0x60f) [0x7f0c3b2a7cf9] 2024-07-26T06:57:56.414 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0c3b3845a1] 2024-07-26T06:57:56.414 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 20: (Thread::entry_wrapper()+0x43) [0x7f0c3b131cdd] 2024-07-26T06:57:56.414 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0c3b131cf9] 2024-07-26T06:57:56.414 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0c3aa25c02] 2024-07-26T06:57:56.414 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0c3aaaac40] 2024-07-26T06:57:56.414 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:56.415 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119529]: 2024-07-26T06:57:56.415 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 podman[119609]: 2024-07-26 06:57:56.156123856 +0000 UTC m=+0.023518652 container died 28ee803f4704cd586a4e3d7579fba64777b6e7d834c99c68b512f2d81ec09344 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-07-26T06:57:56.459 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3863572604,v1:172.21.15.42:6839/3863572604] up:rejoin 2024-07-26T06:57:56.459 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/1877922073,v1:172.21.15.42:6841/1877922073] up:rejoin 2024-07-26T06:57:56.459 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/351656606,v1:172.21.15.164:6833/351656606] up:rejoin 2024-07-26T06:57:56.459 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:57:56.459 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-mon[36970]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:57:56.459 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-mon[36970]: daemon mds.a is now active in filesystem cephfs as rank 3 2024-07-26T06:57:56.459 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-mon[36970]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-07-26T06:57:56.460 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:57:56.460 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: *** Caught signal (Aborted) ** 2024-07-26T06:57:56.460 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: in thread 7f5ff1691640 thread_name:ms_dispatch 2024-07-26T06:57:56.460 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:56.460 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5ff6a686f0] 2024-07-26T06:57:56.460 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5ff6ab594c] 2024-07-26T06:57:56.460 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 3: raise() 2024-07-26T06:57:56.460 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 4: abort() 2024-07-26T06:57:56.460 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5ff6a5271b] 2024-07-26T06:57:56.460 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 6: /lib64/libc.so.6(+0x37386) [0x7f5ff6a61386] 2024-07-26T06:57:56.460 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558c28a7c6b9] 2024-07-26T06:57:56.460 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558c28c3a80c] 2024-07-26T06:57:56.461 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558c28b7a2ca] 2024-07-26T06:57:56.461 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558c28b4a753] 2024-07-26T06:57:56.461 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558c28b4ba1b] 2024-07-26T06:57:56.461 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558c28b2f752] 2024-07-26T06:57:56.461 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558c28a7da56] 2024-07-26T06:57:56.461 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558c28a81637] 2024-07-26T06:57:56.461 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558c28a81f5e] 2024-07-26T06:57:56.461 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558c28a6503a] 2024-07-26T06:57:56.461 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5ff7335407] 2024-07-26T06:57:56.461 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 18: (DispatchQueue::entry()+0x60f) [0x7f5ff7335cf9] 2024-07-26T06:57:56.461 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5ff74125a1] 2024-07-26T06:57:56.461 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 20: (Thread::entry_wrapper()+0x43) [0x7f5ff71bfcdd] 2024-07-26T06:57:56.461 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5ff71bfcf9] 2024-07-26T06:57:56.462 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5ff6ab3c02] 2024-07-26T06:57:56.462 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5ff6b38c40] 2024-07-26T06:57:56.462 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 2024-07-26T06:57:56.080+0000 7f5ff1691640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:56.462 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: in thread 7f5ff1691640 thread_name:ms_dispatch 2024-07-26T06:57:56.462 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 2024-07-26T06:57:56.462 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:56.462 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5ff6a686f0] 2024-07-26T06:57:56.462 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5ff6ab594c] 2024-07-26T06:57:56.462 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 3: raise() 2024-07-26T06:57:56.462 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 4: abort() 2024-07-26T06:57:56.462 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5ff6a5271b] 2024-07-26T06:57:56.462 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 6: /lib64/libc.so.6(+0x37386) [0x7f5ff6a61386] 2024-07-26T06:57:56.462 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558c28a7c6b9] 2024-07-26T06:57:56.463 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558c28c3a80c] 2024-07-26T06:57:56.463 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558c28b7a2ca] 2024-07-26T06:57:56.463 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558c28b4a753] 2024-07-26T06:57:56.463 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558c28b4ba1b] 2024-07-26T06:57:56.463 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558c28b2f752] 2024-07-26T06:57:56.463 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558c28a7da56] 2024-07-26T06:57:56.463 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558c28a81637] 2024-07-26T06:57:56.463 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558c28a81f5e] 2024-07-26T06:57:56.463 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558c28a6503a] 2024-07-26T06:57:56.463 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5ff7335407] 2024-07-26T06:57:56.463 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 18: (DispatchQueue::entry()+0x60f) [0x7f5ff7335cf9] 2024-07-26T06:57:56.463 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5ff74125a1] 2024-07-26T06:57:56.464 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 20: (Thread::entry_wrapper()+0x43) [0x7f5ff71bfcdd] 2024-07-26T06:57:56.464 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5ff71bfcf9] 2024-07-26T06:57:56.464 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5ff6ab3c02] 2024-07-26T06:57:56.464 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5ff6b38c40] 2024-07-26T06:57:56.464 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:56.464 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 2024-07-26T06:57:56.464 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 0> 2024-07-26T06:57:56.080+0000 7f5ff1691640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:56.464 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: in thread 7f5ff1691640 thread_name:ms_dispatch 2024-07-26T06:57:56.465 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 2024-07-26T06:57:56.465 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:56.465 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5ff6a686f0] 2024-07-26T06:57:56.465 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5ff6ab594c] 2024-07-26T06:57:56.465 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 3: raise() 2024-07-26T06:57:56.465 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 4: abort() 2024-07-26T06:57:56.465 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5ff6a5271b] 2024-07-26T06:57:56.465 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 6: /lib64/libc.so.6(+0x37386) [0x7f5ff6a61386] 2024-07-26T06:57:56.465 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558c28a7c6b9] 2024-07-26T06:57:56.465 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558c28c3a80c] 2024-07-26T06:57:56.465 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558c28b7a2ca] 2024-07-26T06:57:56.465 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558c28b4a753] 2024-07-26T06:57:56.465 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558c28b4ba1b] 2024-07-26T06:57:56.465 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558c28b2f752] 2024-07-26T06:57:56.466 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558c28a7da56] 2024-07-26T06:57:56.466 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558c28a81637] 2024-07-26T06:57:56.466 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558c28a81f5e] 2024-07-26T06:57:56.466 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558c28a6503a] 2024-07-26T06:57:56.466 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5ff7335407] 2024-07-26T06:57:56.466 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 18: (DispatchQueue::entry()+0x60f) [0x7f5ff7335cf9] 2024-07-26T06:57:56.466 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5ff74125a1] 2024-07-26T06:57:56.466 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 20: (Thread::entry_wrapper()+0x43) [0x7f5ff71bfcdd] 2024-07-26T06:57:56.466 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5ff71bfcf9] 2024-07-26T06:57:56.466 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5ff6ab3c02] 2024-07-26T06:57:56.466 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5ff6b38c40] 2024-07-26T06:57:56.466 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:56.467 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 2024-07-26T06:57:56.467 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 0> 2024-07-26T06:57:56.080+0000 7f5ff1691640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:57:56.467 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: in thread 7f5ff1691640 thread_name:ms_dispatch 2024-07-26T06:57:56.467 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 2024-07-26T06:57:56.467 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:57:56.467 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5ff6a686f0] 2024-07-26T06:57:56.467 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5ff6ab594c] 2024-07-26T06:57:56.467 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 3: raise() 2024-07-26T06:57:56.467 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 4: abort() 2024-07-26T06:57:56.467 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5ff6a5271b] 2024-07-26T06:57:56.467 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 6: /lib64/libc.so.6(+0x37386) [0x7f5ff6a61386] 2024-07-26T06:57:56.467 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558c28a7c6b9] 2024-07-26T06:57:56.468 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558c28c3a80c] 2024-07-26T06:57:56.468 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558c28b7a2ca] 2024-07-26T06:57:56.468 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558c28b4a753] 2024-07-26T06:57:56.468 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558c28b4ba1b] 2024-07-26T06:57:56.468 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558c28b2f752] 2024-07-26T06:57:56.468 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558c28a7da56] 2024-07-26T06:57:56.468 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558c28a81637] 2024-07-26T06:57:56.468 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558c28a81f5e] 2024-07-26T06:57:56.468 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558c28a6503a] 2024-07-26T06:57:56.468 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5ff7335407] 2024-07-26T06:57:56.468 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 18: (DispatchQueue::entry()+0x60f) [0x7f5ff7335cf9] 2024-07-26T06:57:56.468 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5ff74125a1] 2024-07-26T06:57:56.468 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 20: (Thread::entry_wrapper()+0x43) [0x7f5ff71bfcdd] 2024-07-26T06:57:56.468 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5ff71bfcf9] 2024-07-26T06:57:56.469 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5ff6ab3c02] 2024-07-26T06:57:56.469 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5ff6b38c40] 2024-07-26T06:57:56.469 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:57:56.469 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[70959]: 2024-07-26T06:57:56.469 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 podman[71009]: 2024-07-26 06:57:56.139780877 +0000 UTC m=+0.019594169 container died 3d91911e4e96facc925304f591eafe9c446657b82424378d484d0d415cd797db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T06:57:56.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:56 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3863572604,v1:172.21.15.42:6839/3863572604] up:rejoin 2024-07-26T06:57:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:56 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/1877922073,v1:172.21.15.42:6841/1877922073] up:rejoin 2024-07-26T06:57:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:56 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/351656606,v1:172.21.15.164:6833/351656606] up:rejoin 2024-07-26T06:57:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:56 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:57:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:56 smithi163 ceph-mon[36873]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:57:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:56 smithi163 ceph-mon[36873]: daemon mds.a is now active in filesystem cephfs as rank 3 2024-07-26T06:57:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:56 smithi163 ceph-mon[36873]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-07-26T06:57:57.045 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 podman[71009]: 2024-07-26 06:57:56.793677251 +0000 UTC m=+0.673490544 container remove 3d91911e4e96facc925304f591eafe9c446657b82424378d484d0d415cd797db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.license=GPLv2) 2024-07-26T06:57:57.045 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:56 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:57:57.145 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 podman[119609]: 2024-07-26 06:57:56.834858071 +0000 UTC m=+0.702252912 container remove 28ee803f4704cd586a4e3d7579fba64777b6e7d834c99c68b512f2d81ec09344 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T06:57:57.145 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:56 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:57:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:57 smithi042 ceph-mon[32775]: pgmap v650: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-07-26T06:57:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:57 smithi042 ceph-mon[32775]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:57:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:57 smithi042 ceph-mon[32775]: Cluster is now healthy 2024-07-26T06:57:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:57 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/351656606,v1:172.21.15.164:6833/351656606] up:active 2024-07-26T06:57:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:57 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/1877922073,v1:172.21.15.42:6841/1877922073] up:active 2024-07-26T06:57:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:57 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3863572604,v1:172.21.15.42:6839/3863572604] up:active 2024-07-26T06:57:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:57 smithi042 ceph-mon[32775]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:57:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:57 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:57:57.472 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:57:57 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-07-26T06:57:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:57 smithi164 ceph-mon[36970]: pgmap v650: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-07-26T06:57:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:57 smithi164 ceph-mon[36970]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:57:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:57 smithi164 ceph-mon[36970]: Cluster is now healthy 2024-07-26T06:57:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:57 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/351656606,v1:172.21.15.164:6833/351656606] up:active 2024-07-26T06:57:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:57 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/1877922073,v1:172.21.15.42:6841/1877922073] up:active 2024-07-26T06:57:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:57 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3863572604,v1:172.21.15.42:6839/3863572604] up:active 2024-07-26T06:57:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:57 smithi164 ceph-mon[36970]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:57:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:57 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:57:57.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:57 smithi163 ceph-mon[36873]: pgmap v650: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-07-26T06:57:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:57 smithi163 ceph-mon[36873]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:57:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:57 smithi163 ceph-mon[36873]: Cluster is now healthy 2024-07-26T06:57:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:57 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/351656606,v1:172.21.15.164:6833/351656606] up:active 2024-07-26T06:57:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:57 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/1877922073,v1:172.21.15.42:6841/1877922073] up:active 2024-07-26T06:57:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:57 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3863572604,v1:172.21.15.42:6839/3863572604] up:active 2024-07-26T06:57:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:57 smithi163 ceph-mon[36873]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:57:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:57 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:57:57.710 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:57:57 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T06:57:57.710 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:57 smithi042 podman[119608]: 2024-07-26 06:57:57.44807554 +0000 UTC m=+1.315294220 container remove f225d4b8aad6754fce903925827868192e05a9ed3a09c37689763cebc83a1e9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T06:57:57.710 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:57 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:57:58.144 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:57:57 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-07-26T06:57:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:57:59 smithi042 ceph-mon[32775]: pgmap v651: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 323 B/s wr, 14 op/s 2024-07-26T06:57:59.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:57:59 smithi164 ceph-mon[36970]: pgmap v651: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 323 B/s wr, 14 op/s 2024-07-26T06:57:59.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:57:59 smithi163 ceph-mon[36873]: pgmap v651: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 323 B/s wr, 14 op/s 2024-07-26T06:58:01.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:01 smithi042 ceph-mon[32775]: pgmap v652: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 767 B/s wr, 16 op/s 2024-07-26T06:58:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:01 smithi164 ceph-mon[36970]: pgmap v652: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 767 B/s wr, 16 op/s 2024-07-26T06:58:01.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:01 smithi163 ceph-mon[36873]: pgmap v652: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 767 B/s wr, 16 op/s 2024-07-26T06:58:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:03 smithi042 ceph-mon[32775]: pgmap v653: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 626 B/s wr, 13 op/s 2024-07-26T06:58:03.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:03 smithi164 ceph-mon[36970]: pgmap v653: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 626 B/s wr, 13 op/s 2024-07-26T06:58:03.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:03 smithi163 ceph-mon[36873]: pgmap v653: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 626 B/s wr, 13 op/s 2024-07-26T06:58:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:05 smithi042 ceph-mon[32775]: pgmap v654: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 536 B/s wr, 13 op/s 2024-07-26T06:58:05.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:05 smithi164 ceph-mon[36970]: pgmap v654: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 536 B/s wr, 13 op/s 2024-07-26T06:58:05.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:05 smithi163 ceph-mon[36873]: pgmap v654: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 536 B/s wr, 13 op/s 2024-07-26T06:58:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:07 smithi042 ceph-mon[32775]: pgmap v655: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 511 B/s wr, 13 op/s 2024-07-26T06:58:07.459 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:07 smithi164 ceph-mon[36970]: pgmap v655: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 511 B/s wr, 13 op/s 2024-07-26T06:58:07.459 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:07 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 4. 2024-07-26T06:58:07.459 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:07 smithi164 systemd[1]: Stopped Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:07.460 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:07 smithi164 systemd[1]: Starting Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:58:07.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:07 smithi163 ceph-mon[36873]: pgmap v655: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 511 B/s wr, 13 op/s 2024-07-26T06:58:07.714 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:07 smithi164 podman[71100]: 2024-07-26 06:58:07.36213731 +0000 UTC m=+0.017521100 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:58:07.714 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:07 smithi164 podman[71100]: 2024-07-26 06:58:07.477802743 +0000 UTC m=+0.133186567 container create 5bb60cf0fda784128b905112dbb68da162bcaedceffa4e7d5d6ed0d050d9ad79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-07-26T06:58:07.895 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:07 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 4. 2024-07-26T06:58:07.896 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:07 smithi042 systemd[1]: Stopped Ceph mds.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:07.896 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:07 smithi042 systemd[1]: Starting Ceph mds.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:58:07.896 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:07 smithi042 podman[119743]: 2024-07-26 06:58:07.890837096 +0000 UTC m=+0.115072456 container create e0cde491b61e7b7e3a1b3d1a808bc47583b6fc12009c8564de164186e996251c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-07-26T06:58:07.896 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:07 smithi042 podman[119743]: 2024-07-26 06:58:07.794278298 +0000 UTC m=+0.018513622 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:58:07.972 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:07 smithi164 podman[71100]: 2024-07-26 06:58:07.714396797 +0000 UTC m=+0.369780575 container init 5bb60cf0fda784128b905112dbb68da162bcaedceffa4e7d5d6ed0d050d9ad79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-07-26T06:58:07.972 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:07 smithi164 podman[71100]: 2024-07-26 06:58:07.717469362 +0000 UTC m=+0.372853149 container start 5bb60cf0fda784128b905112dbb68da162bcaedceffa4e7d5d6ed0d050d9ad79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, ceph=True, org.label-schema.build-date=20240724, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-07-26T06:58:07.972 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:07 smithi164 ceph-mds[71119]: -- 172.21.15.164:0/4014375565 <== mon.1 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5583eafb3860 con 0x5583ef155600 2024-07-26T06:58:07.973 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:07 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: starting mds.l at 2024-07-26T06:58:07.973 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:07 smithi164 bash[71100]: 5bb60cf0fda784128b905112dbb68da162bcaedceffa4e7d5d6ed0d050d9ad79 2024-07-26T06:58:07.973 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:07 smithi164 systemd[1]: Started Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:08.169 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:07 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 4. 2024-07-26T06:58:08.169 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:07 smithi042 systemd[1]: Stopped Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:08.169 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:07 smithi042 systemd[1]: Starting Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:58:08.170 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:08 smithi042 ceph-mon[32775]: daemon mds.l restarted 2024-07-26T06:58:08.170 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:08 smithi042 ceph-mon[32775]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:58:08.170 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:08 smithi042 ceph-mon[32775]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T06:58:08.170 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:08 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/4063114254,v1:172.21.15.164:6833/4063114254] up:boot 2024-07-26T06:58:08.170 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:08 smithi042 ceph-mon[32775]: Standby daemon mds.h assigned to filesystem cephfs as rank 2 2024-07-26T06:58:08.170 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:08 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:58:08.170 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:08 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:58:08.170 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:08 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:58:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:08.322+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T06:58:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:08.322+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-07-26T06:58:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:08.322+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T06:58:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:08.322+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T06:58:08.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:08.323+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:58:08.425 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:08 smithi042 podman[119743]: 2024-07-26 06:58:08.169400654 +0000 UTC m=+0.393635976 container init e0cde491b61e7b7e3a1b3d1a808bc47583b6fc12009c8564de164186e996251c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724) 2024-07-26T06:58:08.425 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:08 smithi042 podman[119743]: 2024-07-26 06:58:08.173280856 +0000 UTC m=+0.397516188 container start e0cde491b61e7b7e3a1b3d1a808bc47583b6fc12009c8564de164186e996251c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-07-26T06:58:08.425 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:08 smithi042 ceph-mds[119799]: -- 172.21.15.42:0/763737247 <== mon.0 v2:172.21.15.42:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5599c63f7860 con 0x5599ca599a80 2024-07-26T06:58:08.425 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:08 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: starting mds.a at 2024-07-26T06:58:08.425 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:08 smithi042 bash[119743]: e0cde491b61e7b7e3a1b3d1a808bc47583b6fc12009c8564de164186e996251c 2024-07-26T06:58:08.425 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:08 smithi042 systemd[1]: Started Ceph mds.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:08 smithi164 ceph-mon[36970]: daemon mds.l restarted 2024-07-26T06:58:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:08 smithi164 ceph-mon[36970]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:58:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:08 smithi164 ceph-mon[36970]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T06:58:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:08 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/4063114254,v1:172.21.15.164:6833/4063114254] up:boot 2024-07-26T06:58:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:08 smithi164 ceph-mon[36970]: Standby daemon mds.h assigned to filesystem cephfs as rank 2 2024-07-26T06:58:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:08 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:58:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:08 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:58:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:08 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:58:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:08 smithi163 ceph-mon[36873]: daemon mds.l restarted 2024-07-26T06:58:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:08 smithi163 ceph-mon[36873]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:58:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:08 smithi163 ceph-mon[36873]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T06:58:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:08 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/4063114254,v1:172.21.15.164:6833/4063114254] up:boot 2024-07-26T06:58:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:08 smithi163 ceph-mon[36873]: Standby daemon mds.h assigned to filesystem cephfs as rank 2 2024-07-26T06:58:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:08 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=a=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:58:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:08 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:58:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:08 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:58:08.715 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:08 smithi042 podman[119831]: 2024-07-26 06:58:08.327952409 +0000 UTC m=+0.017851923 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:58:08.715 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:08 smithi042 podman[119831]: 2024-07-26 06:58:08.445646757 +0000 UTC m=+0.135546262 container create 0a4993e9b9a2d3e242642b01d380d902803d4b656b759335cdf5e7aff2aa43ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T06:58:08.715 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:08 smithi042 podman[119831]: 2024-07-26 06:58:08.715270665 +0000 UTC m=+0.405170170 container init 0a4993e9b9a2d3e242642b01d380d902803d4b656b759335cdf5e7aff2aa43ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-07-26T06:58:09.098 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:08 smithi042 podman[119831]: 2024-07-26 06:58:08.71843641 +0000 UTC m=+0.408335913 container start 0a4993e9b9a2d3e242642b01d380d902803d4b656b759335cdf5e7aff2aa43ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-07-26T06:58:09.098 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:08 smithi042 ceph-mds[119851]: -- 172.21.15.42:0/3574612909 <== mon.0 v2:172.21.15.42:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cbd3f0bc20 con 0x55cbd80ad180 2024-07-26T06:58:09.098 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:08 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: starting mds.g at 2024-07-26T06:58:09.098 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:08 smithi042 bash[119831]: 0a4993e9b9a2d3e242642b01d380d902803d4b656b759335cdf5e7aff2aa43ae 2024-07-26T06:58:09.098 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:08 smithi042 systemd[1]: Started Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:09.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:09 smithi164 ceph-mon[36970]: pgmap v656: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T06:58:09.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:09 smithi164 ceph-mon[36970]: daemon mds.a restarted 2024-07-26T06:58:09.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:09 smithi164 ceph-mon[36970]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T06:58:09.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:09 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/2749045649,v1:172.21.15.42:6839/2749045649] up:boot 2024-07-26T06:58:09.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:09 smithi164 ceph-mon[36970]: Standby daemon mds.b assigned to filesystem cephfs as rank 3 2024-07-26T06:58:09.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:09 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:replay,4=g=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:58:09.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T06:58:09.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:09 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-07-26T06:58:09.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:09 smithi164 ceph-mon[36970]: daemon mds.g restarted 2024-07-26T06:58:09.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:09 smithi164 ceph-mon[36970]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T06:58:09.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:09 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/193486687,v1:172.21.15.163:6839/193486687] up:resolve 2024-07-26T06:58:09.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:09 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/3101627593,v1:172.21.15.163:6837/3101627593] up:resolve 2024-07-26T06:58:09.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:09 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/582863084,v1:172.21.15.42:6841/582863084] up:boot 2024-07-26T06:58:09.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:09 smithi164 ceph-mon[36970]: Standby daemon mds.e assigned to filesystem cephfs as rank 4 2024-07-26T06:58:09.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:09 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,3=b=up:resolve} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:58:09.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:58:09.376 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:09 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 1 up:replay 2 up:resolve 2 up:active 2024-07-26T06:58:09.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:09 smithi042 ceph-mon[32775]: pgmap v656: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T06:58:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:09 smithi042 ceph-mon[32775]: daemon mds.a restarted 2024-07-26T06:58:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:09 smithi042 ceph-mon[32775]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T06:58:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:09 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/2749045649,v1:172.21.15.42:6839/2749045649] up:boot 2024-07-26T06:58:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:09 smithi042 ceph-mon[32775]: Standby daemon mds.b assigned to filesystem cephfs as rank 3 2024-07-26T06:58:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:09 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:replay,4=g=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:58:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T06:58:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:09 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-07-26T06:58:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:09 smithi042 ceph-mon[32775]: daemon mds.g restarted 2024-07-26T06:58:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:09 smithi042 ceph-mon[32775]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T06:58:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:09 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/193486687,v1:172.21.15.163:6839/193486687] up:resolve 2024-07-26T06:58:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:09 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/3101627593,v1:172.21.15.163:6837/3101627593] up:resolve 2024-07-26T06:58:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:09 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/582863084,v1:172.21.15.42:6841/582863084] up:boot 2024-07-26T06:58:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:09 smithi042 ceph-mon[32775]: Standby daemon mds.e assigned to filesystem cephfs as rank 4 2024-07-26T06:58:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:09 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,3=b=up:resolve} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:58:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:58:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:09 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 1 up:replay 2 up:resolve 2 up:active 2024-07-26T06:58:09.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:09 smithi163 ceph-mon[36873]: pgmap v656: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T06:58:09.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:09 smithi163 ceph-mon[36873]: daemon mds.a restarted 2024-07-26T06:58:09.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:09 smithi163 ceph-mon[36873]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T06:58:09.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:09 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/2749045649,v1:172.21.15.42:6839/2749045649] up:boot 2024-07-26T06:58:09.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:09 smithi163 ceph-mon[36873]: Standby daemon mds.b assigned to filesystem cephfs as rank 3 2024-07-26T06:58:09.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:09 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:replay,4=g=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:58:09.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T06:58:09.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:09 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-07-26T06:58:09.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:09 smithi163 ceph-mon[36873]: daemon mds.g restarted 2024-07-26T06:58:09.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:09 smithi163 ceph-mon[36873]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T06:58:09.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:09 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/193486687,v1:172.21.15.163:6839/193486687] up:resolve 2024-07-26T06:58:09.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:09 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/3101627593,v1:172.21.15.163:6837/3101627593] up:resolve 2024-07-26T06:58:09.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:09 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/582863084,v1:172.21.15.42:6841/582863084] up:boot 2024-07-26T06:58:09.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:09 smithi163 ceph-mon[36873]: Standby daemon mds.e assigned to filesystem cephfs as rank 4 2024-07-26T06:58:09.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:09 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,3=b=up:resolve} 3 up:standby-replay 5 up:standby, 1 failed 2024-07-26T06:58:09.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:58:09.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:09 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 1 up:replay 2 up:resolve 2 up:active 2024-07-26T06:58:11.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:11 smithi042 ceph-mon[32775]: pgmap v660: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 13 op/s 2024-07-26T06:58:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:11 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/873838612,v1:172.21.15.163:6835/873838612] up:resolve 2024-07-26T06:58:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:11 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:58:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:11 smithi164 ceph-mon[36970]: pgmap v660: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 13 op/s 2024-07-26T06:58:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:11 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/873838612,v1:172.21.15.163:6835/873838612] up:resolve 2024-07-26T06:58:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:11 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:58:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:11 smithi163 ceph-mon[36873]: pgmap v660: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 13 op/s 2024-07-26T06:58:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:11 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/873838612,v1:172.21.15.163:6835/873838612] up:resolve 2024-07-26T06:58:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:11 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-07-26T06:58:12.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:12 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:58:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:12 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:58:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:12 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:58:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:12 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/193486687,v1:172.21.15.163:6839/193486687] up:reconnect 2024-07-26T06:58:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:12 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/3101627593,v1:172.21.15.163:6837/3101627593] up:reconnect 2024-07-26T06:58:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:12 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/873838612,v1:172.21.15.163:6835/873838612] up:reconnect 2024-07-26T06:58:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:12 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:58:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:12 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:58:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:12 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:58:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:12 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:58:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:12 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/193486687,v1:172.21.15.163:6839/193486687] up:reconnect 2024-07-26T06:58:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:12 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/3101627593,v1:172.21.15.163:6837/3101627593] up:reconnect 2024-07-26T06:58:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:12 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/873838612,v1:172.21.15.163:6835/873838612] up:reconnect 2024-07-26T06:58:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:12 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:58:12.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:12 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:58:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:12 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:58:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:12 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:58:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:12 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/193486687,v1:172.21.15.163:6839/193486687] up:reconnect 2024-07-26T06:58:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:12 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/3101627593,v1:172.21.15.163:6837/3101627593] up:reconnect 2024-07-26T06:58:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:12 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/873838612,v1:172.21.15.163:6835/873838612] up:reconnect 2024-07-26T06:58:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:12 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:58:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:13 smithi042 ceph-mon[32775]: pgmap v661: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-07-26T06:58:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:13 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/3101627593,v1:172.21.15.163:6837/3101627593] up:rejoin 2024-07-26T06:58:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:13 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/193486687,v1:172.21.15.163:6839/193486687] up:rejoin 2024-07-26T06:58:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:13 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/873838612,v1:172.21.15.163:6835/873838612] up:rejoin 2024-07-26T06:58:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:13 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:58:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:13 smithi042 ceph-mon[32775]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-07-26T06:58:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:13 smithi042 ceph-mon[32775]: daemon mds.b is now active in filesystem cephfs as rank 3 2024-07-26T06:58:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:13 smithi042 ceph-mon[32775]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-07-26T06:58:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:58:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:13 smithi164 ceph-mon[36970]: pgmap v661: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-07-26T06:58:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:13 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/3101627593,v1:172.21.15.163:6837/3101627593] up:rejoin 2024-07-26T06:58:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:13 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/193486687,v1:172.21.15.163:6839/193486687] up:rejoin 2024-07-26T06:58:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:13 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/873838612,v1:172.21.15.163:6835/873838612] up:rejoin 2024-07-26T06:58:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:13 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:58:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:13 smithi164 ceph-mon[36970]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-07-26T06:58:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:13 smithi164 ceph-mon[36970]: daemon mds.b is now active in filesystem cephfs as rank 3 2024-07-26T06:58:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:13 smithi164 ceph-mon[36970]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-07-26T06:58:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:58:13.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-mon[36873]: pgmap v661: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-07-26T06:58:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/3101627593,v1:172.21.15.163:6837/3101627593] up:rejoin 2024-07-26T06:58:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/193486687,v1:172.21.15.163:6839/193486687] up:rejoin 2024-07-26T06:58:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/873838612,v1:172.21.15.163:6835/873838612] up:rejoin 2024-07-26T06:58:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:58:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-mon[36873]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-07-26T06:58:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-mon[36873]: daemon mds.b is now active in filesystem cephfs as rank 3 2024-07-26T06:58:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-mon[36873]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-07-26T06:58:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:58:13.508 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:58:13.508 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: *** Caught signal (Aborted) ** 2024-07-26T06:58:13.508 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: in thread 7fba495c1640 thread_name:ms_dispatch 2024-07-26T06:58:13.508 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:13.509 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fba4e9986f0] 2024-07-26T06:58:13.509 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fba4e9e594c] 2024-07-26T06:58:13.509 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 3: raise() 2024-07-26T06:58:13.509 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 4: abort() 2024-07-26T06:58:13.509 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 5: /lib64/libc.so.6(+0x2871b) [0x7fba4e98271b] 2024-07-26T06:58:13.509 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 6: /lib64/libc.so.6(+0x37386) [0x7fba4e991386] 2024-07-26T06:58:13.509 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561282a3b6b9] 2024-07-26T06:58:13.509 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561282bf980c] 2024-07-26T06:58:13.509 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x561282b392ca] 2024-07-26T06:58:13.509 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x561282b09753] 2024-07-26T06:58:13.509 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561282b0aa1b] 2024-07-26T06:58:13.509 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561282aee752] 2024-07-26T06:58:13.510 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561282a3ca56] 2024-07-26T06:58:13.510 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561282a40637] 2024-07-26T06:58:13.510 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561282a40f5e] 2024-07-26T06:58:13.510 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561282a2403a] 2024-07-26T06:58:13.510 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fba4f265407] 2024-07-26T06:58:13.510 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 18: (DispatchQueue::entry()+0x60f) [0x7fba4f265cf9] 2024-07-26T06:58:13.510 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fba4f3425a1] 2024-07-26T06:58:13.510 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:58:13.511 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: *** Caught signal (Aborted) ** 2024-07-26T06:58:13.511 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: in thread 7f5e709e7640 thread_name:ms_dispatch 2024-07-26T06:58:13.511 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:13.511 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5e75dbe6f0] 2024-07-26T06:58:13.511 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5e75e0b94c] 2024-07-26T06:58:13.511 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 3: raise() 2024-07-26T06:58:13.511 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 4: abort() 2024-07-26T06:58:13.511 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5e75da871b] 2024-07-26T06:58:13.511 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 6: /lib64/libc.so.6(+0x37386) [0x7f5e75db7386] 2024-07-26T06:58:13.511 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5602a03e56b9] 2024-07-26T06:58:13.511 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5602a05a380c] 2024-07-26T06:58:13.511 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5602a04e32ca] 2024-07-26T06:58:13.512 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5602a04b3753] 2024-07-26T06:58:13.512 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5602a04b4a1b] 2024-07-26T06:58:13.512 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5602a0498752] 2024-07-26T06:58:13.512 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5602a03e6a56] 2024-07-26T06:58:13.512 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5602a03ea637] 2024-07-26T06:58:13.512 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5602a03eaf5e] 2024-07-26T06:58:13.512 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5602a03ce03a] 2024-07-26T06:58:13.512 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5e7668b407] 2024-07-26T06:58:13.512 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 18: (DispatchQueue::entry()+0x60f) [0x7f5e7668bcf9] 2024-07-26T06:58:13.512 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5e767685a1] 2024-07-26T06:58:13.513 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:58:13.513 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: *** Caught signal (Aborted) ** 2024-07-26T06:58:13.513 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: in thread 7fe4eedb4640 thread_name:ms_dispatch 2024-07-26T06:58:13.513 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:13.513 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe4f418b6f0] 2024-07-26T06:58:13.513 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe4f41d894c] 2024-07-26T06:58:13.513 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 3: raise() 2024-07-26T06:58:13.513 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 4: abort() 2024-07-26T06:58:13.513 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe4f417571b] 2024-07-26T06:58:13.513 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 6: /lib64/libc.so.6(+0x37386) [0x7fe4f4184386] 2024-07-26T06:58:13.513 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5564096856b9] 2024-07-26T06:58:13.513 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55640984380c] 2024-07-26T06:58:13.514 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5564097832ca] 2024-07-26T06:58:13.514 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x556409753753] 2024-07-26T06:58:13.514 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x556409754a1b] 2024-07-26T06:58:13.514 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x556409738752] 2024-07-26T06:58:13.514 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x556409686a56] 2024-07-26T06:58:13.514 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55640968a637] 2024-07-26T06:58:13.514 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55640968af5e] 2024-07-26T06:58:13.515 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55640966e03a] 2024-07-26T06:58:13.515 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe4f4a58407] 2024-07-26T06:58:13.515 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 18: (DispatchQueue::entry()+0x60f) [0x7fe4f4a58cf9] 2024-07-26T06:58:13.515 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe4f4b355a1] 2024-07-26T06:58:13.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 20: (Thread::entry_wrapper()+0x43) [0x7fba4f0efcdd] 2024-07-26T06:58:13.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 21: (Thread::_entry_func(void*)+0xd) [0x7fba4f0efcf9] 2024-07-26T06:58:13.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 22: /lib64/libc.so.6(+0x89c02) [0x7fba4e9e3c02] 2024-07-26T06:58:13.516 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fba4ea68c40] 2024-07-26T06:58:13.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 2024-07-26T06:58:13.136+0000 7fba495c1640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:13.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: in thread 7fba495c1640 thread_name:ms_dispatch 2024-07-26T06:58:13.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 2024-07-26T06:58:13.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:13.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fba4e9986f0] 2024-07-26T06:58:13.517 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fba4e9e594c] 2024-07-26T06:58:13.518 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 3: raise() 2024-07-26T06:58:13.518 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 4: abort() 2024-07-26T06:58:13.518 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 5: /lib64/libc.so.6(+0x2871b) [0x7fba4e98271b] 2024-07-26T06:58:13.518 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 6: /lib64/libc.so.6(+0x37386) [0x7fba4e991386] 2024-07-26T06:58:13.518 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561282a3b6b9] 2024-07-26T06:58:13.519 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561282bf980c] 2024-07-26T06:58:13.519 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x561282b392ca] 2024-07-26T06:58:13.519 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x561282b09753] 2024-07-26T06:58:13.519 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561282b0aa1b] 2024-07-26T06:58:13.519 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561282aee752] 2024-07-26T06:58:13.519 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561282a3ca56] 2024-07-26T06:58:13.520 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561282a40637] 2024-07-26T06:58:13.520 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561282a40f5e] 2024-07-26T06:58:13.520 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561282a2403a] 2024-07-26T06:58:13.520 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fba4f265407] 2024-07-26T06:58:13.520 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 18: (DispatchQueue::entry()+0x60f) [0x7fba4f265cf9] 2024-07-26T06:58:13.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fba4f3425a1] 2024-07-26T06:58:13.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 20: (Thread::entry_wrapper()+0x43) [0x7fba4f0efcdd] 2024-07-26T06:58:13.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 21: (Thread::_entry_func(void*)+0xd) [0x7fba4f0efcf9] 2024-07-26T06:58:13.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 22: /lib64/libc.so.6(+0x89c02) [0x7fba4e9e3c02] 2024-07-26T06:58:13.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fba4ea68c40] 2024-07-26T06:58:13.521 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:13.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 2024-07-26T06:58:13.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 0> 2024-07-26T06:58:13.136+0000 7fba495c1640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:13.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: in thread 7fba495c1640 thread_name:ms_dispatch 2024-07-26T06:58:13.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 2024-07-26T06:58:13.522 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:13.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fba4e9986f0] 2024-07-26T06:58:13.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fba4e9e594c] 2024-07-26T06:58:13.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 3: raise() 2024-07-26T06:58:13.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 4: abort() 2024-07-26T06:58:13.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 5: /lib64/libc.so.6(+0x2871b) [0x7fba4e98271b] 2024-07-26T06:58:13.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 6: /lib64/libc.so.6(+0x37386) [0x7fba4e991386] 2024-07-26T06:58:13.523 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561282a3b6b9] 2024-07-26T06:58:13.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561282bf980c] 2024-07-26T06:58:13.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x561282b392ca] 2024-07-26T06:58:13.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x561282b09753] 2024-07-26T06:58:13.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561282b0aa1b] 2024-07-26T06:58:13.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561282aee752] 2024-07-26T06:58:13.524 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561282a3ca56] 2024-07-26T06:58:13.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561282a40637] 2024-07-26T06:58:13.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561282a40f5e] 2024-07-26T06:58:13.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561282a2403a] 2024-07-26T06:58:13.525 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fba4f265407] 2024-07-26T06:58:13.526 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 20: (Thread::entry_wrapper()+0x43) [0x7f5e76515cdd] 2024-07-26T06:58:13.526 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5e76515cf9] 2024-07-26T06:58:13.526 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5e75e09c02] 2024-07-26T06:58:13.526 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5e75e8ec40] 2024-07-26T06:58:13.526 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 2024-07-26T06:58:13.137+0000 7f5e709e7640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:13.527 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: in thread 7f5e709e7640 thread_name:ms_dispatch 2024-07-26T06:58:13.527 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 2024-07-26T06:58:13.527 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:13.527 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5e75dbe6f0] 2024-07-26T06:58:13.527 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5e75e0b94c] 2024-07-26T06:58:13.527 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 3: raise() 2024-07-26T06:58:13.528 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 4: abort() 2024-07-26T06:58:13.528 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5e75da871b] 2024-07-26T06:58:13.528 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 6: /lib64/libc.so.6(+0x37386) [0x7f5e75db7386] 2024-07-26T06:58:13.528 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5602a03e56b9] 2024-07-26T06:58:13.528 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5602a05a380c] 2024-07-26T06:58:13.529 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5602a04e32ca] 2024-07-26T06:58:13.529 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5602a04b3753] 2024-07-26T06:58:13.529 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5602a04b4a1b] 2024-07-26T06:58:13.529 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5602a0498752] 2024-07-26T06:58:13.529 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5602a03e6a56] 2024-07-26T06:58:13.529 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5602a03ea637] 2024-07-26T06:58:13.530 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5602a03eaf5e] 2024-07-26T06:58:13.530 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5602a03ce03a] 2024-07-26T06:58:13.530 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5e7668b407] 2024-07-26T06:58:13.530 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 18: (DispatchQueue::entry()+0x60f) [0x7f5e7668bcf9] 2024-07-26T06:58:13.530 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5e767685a1] 2024-07-26T06:58:13.531 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 20: (Thread::entry_wrapper()+0x43) [0x7fe4f48e2cdd] 2024-07-26T06:58:13.531 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe4f48e2cf9] 2024-07-26T06:58:13.531 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe4f41d6c02] 2024-07-26T06:58:13.532 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe4f425bc40] 2024-07-26T06:58:13.532 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 2024-07-26T06:58:13.136+0000 7fe4eedb4640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:13.532 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: in thread 7fe4eedb4640 thread_name:ms_dispatch 2024-07-26T06:58:13.532 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 2024-07-26T06:58:13.532 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:13.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe4f418b6f0] 2024-07-26T06:58:13.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe4f41d894c] 2024-07-26T06:58:13.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 3: raise() 2024-07-26T06:58:13.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 4: abort() 2024-07-26T06:58:13.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe4f417571b] 2024-07-26T06:58:13.533 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 6: /lib64/libc.so.6(+0x37386) [0x7fe4f4184386] 2024-07-26T06:58:13.534 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5564096856b9] 2024-07-26T06:58:13.534 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55640984380c] 2024-07-26T06:58:13.534 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5564097832ca] 2024-07-26T06:58:13.534 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x556409753753] 2024-07-26T06:58:13.534 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x556409754a1b] 2024-07-26T06:58:13.535 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x556409738752] 2024-07-26T06:58:13.535 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x556409686a56] 2024-07-26T06:58:13.535 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55640968a637] 2024-07-26T06:58:13.535 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55640968af5e] 2024-07-26T06:58:13.535 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55640966e03a] 2024-07-26T06:58:13.535 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe4f4a58407] 2024-07-26T06:58:13.536 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 18: (DispatchQueue::entry()+0x60f) [0x7fe4f4a58cf9] 2024-07-26T06:58:13.536 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe4f4b355a1] 2024-07-26T06:58:13.537 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 18: (DispatchQueue::entry()+0x60f) [0x7fba4f265cf9] 2024-07-26T06:58:13.537 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fba4f3425a1] 2024-07-26T06:58:13.537 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 20: (Thread::entry_wrapper()+0x43) [0x7fba4f0efcdd] 2024-07-26T06:58:13.537 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 21: (Thread::_entry_func(void*)+0xd) [0x7fba4f0efcf9] 2024-07-26T06:58:13.538 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 22: /lib64/libc.so.6(+0x89c02) [0x7fba4e9e3c02] 2024-07-26T06:58:13.538 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fba4ea68c40] 2024-07-26T06:58:13.538 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:13.538 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 2024-07-26T06:58:13.538 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 0> 2024-07-26T06:58:13.136+0000 7fba495c1640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:13.539 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: in thread 7fba495c1640 thread_name:ms_dispatch 2024-07-26T06:58:13.539 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 2024-07-26T06:58:13.539 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:13.539 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fba4e9986f0] 2024-07-26T06:58:13.539 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fba4e9e594c] 2024-07-26T06:58:13.540 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 3: raise() 2024-07-26T06:58:13.540 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 4: abort() 2024-07-26T06:58:13.540 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 5: /lib64/libc.so.6(+0x2871b) [0x7fba4e98271b] 2024-07-26T06:58:13.540 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 6: /lib64/libc.so.6(+0x37386) [0x7fba4e991386] 2024-07-26T06:58:13.540 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561282a3b6b9] 2024-07-26T06:58:13.541 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561282bf980c] 2024-07-26T06:58:13.541 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x561282b392ca] 2024-07-26T06:58:13.541 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x561282b09753] 2024-07-26T06:58:13.541 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561282b0aa1b] 2024-07-26T06:58:13.541 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561282aee752] 2024-07-26T06:58:13.541 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561282a3ca56] 2024-07-26T06:58:13.542 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561282a40637] 2024-07-26T06:58:13.542 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561282a40f5e] 2024-07-26T06:58:13.542 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561282a2403a] 2024-07-26T06:58:13.542 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fba4f265407] 2024-07-26T06:58:13.542 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 18: (DispatchQueue::entry()+0x60f) [0x7fba4f265cf9] 2024-07-26T06:58:13.542 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fba4f3425a1] 2024-07-26T06:58:13.542 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 20: (Thread::entry_wrapper()+0x43) [0x7fba4f0efcdd] 2024-07-26T06:58:13.543 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 21: (Thread::_entry_func(void*)+0xd) [0x7fba4f0efcf9] 2024-07-26T06:58:13.543 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 22: /lib64/libc.so.6(+0x89c02) [0x7fba4e9e3c02] 2024-07-26T06:58:13.543 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fba4ea68c40] 2024-07-26T06:58:13.543 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:13.543 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[71787]: 2024-07-26T06:58:13.544 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 podman[71919]: 2024-07-26 06:58:13.205989076 +0000 UTC m=+0.028019656 container died 7d7ea37a484b7929fae7ff42deaf12bfb3a5fc34af3a2f0afe1bde88ec478ab1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-07-26T06:58:13.544 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 20: (Thread::entry_wrapper()+0x43) [0x7f5e76515cdd] 2024-07-26T06:58:13.544 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5e76515cf9] 2024-07-26T06:58:13.544 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5e75e09c02] 2024-07-26T06:58:13.545 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5e75e8ec40] 2024-07-26T06:58:13.545 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:13.545 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 2024-07-26T06:58:13.545 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 0> 2024-07-26T06:58:13.137+0000 7f5e709e7640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:13.545 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: in thread 7f5e709e7640 thread_name:ms_dispatch 2024-07-26T06:58:13.546 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 2024-07-26T06:58:13.546 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:13.546 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5e75dbe6f0] 2024-07-26T06:58:13.546 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5e75e0b94c] 2024-07-26T06:58:13.546 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 3: raise() 2024-07-26T06:58:13.547 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 4: abort() 2024-07-26T06:58:13.547 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5e75da871b] 2024-07-26T06:58:13.547 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 6: /lib64/libc.so.6(+0x37386) [0x7f5e75db7386] 2024-07-26T06:58:13.547 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5602a03e56b9] 2024-07-26T06:58:13.547 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5602a05a380c] 2024-07-26T06:58:13.547 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5602a04e32ca] 2024-07-26T06:58:13.548 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5602a04b3753] 2024-07-26T06:58:13.548 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5602a04b4a1b] 2024-07-26T06:58:13.548 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5602a0498752] 2024-07-26T06:58:13.548 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5602a03e6a56] 2024-07-26T06:58:13.548 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5602a03ea637] 2024-07-26T06:58:13.548 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5602a03eaf5e] 2024-07-26T06:58:13.548 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5602a03ce03a] 2024-07-26T06:58:13.548 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5e7668b407] 2024-07-26T06:58:13.549 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 20: (Thread::entry_wrapper()+0x43) [0x7fe4f48e2cdd] 2024-07-26T06:58:13.549 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe4f48e2cf9] 2024-07-26T06:58:13.549 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe4f41d6c02] 2024-07-26T06:58:13.549 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe4f425bc40] 2024-07-26T06:58:13.549 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:13.549 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 2024-07-26T06:58:13.549 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 0> 2024-07-26T06:58:13.136+0000 7fe4eedb4640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:13.549 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: in thread 7fe4eedb4640 thread_name:ms_dispatch 2024-07-26T06:58:13.549 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 2024-07-26T06:58:13.549 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:13.549 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe4f418b6f0] 2024-07-26T06:58:13.549 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe4f41d894c] 2024-07-26T06:58:13.549 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 3: raise() 2024-07-26T06:58:13.550 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 4: abort() 2024-07-26T06:58:13.550 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe4f417571b] 2024-07-26T06:58:13.550 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 6: /lib64/libc.so.6(+0x37386) [0x7fe4f4184386] 2024-07-26T06:58:13.550 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5564096856b9] 2024-07-26T06:58:13.550 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55640984380c] 2024-07-26T06:58:13.550 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5564097832ca] 2024-07-26T06:58:13.550 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x556409753753] 2024-07-26T06:58:13.550 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x556409754a1b] 2024-07-26T06:58:13.550 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x556409738752] 2024-07-26T06:58:13.550 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x556409686a56] 2024-07-26T06:58:13.550 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55640968a637] 2024-07-26T06:58:13.550 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55640968af5e] 2024-07-26T06:58:13.550 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55640966e03a] 2024-07-26T06:58:13.551 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe4f4a58407] 2024-07-26T06:58:13.551 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 18: (DispatchQueue::entry()+0x60f) [0x7f5e7668bcf9] 2024-07-26T06:58:13.551 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5e767685a1] 2024-07-26T06:58:13.551 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 20: (Thread::entry_wrapper()+0x43) [0x7f5e76515cdd] 2024-07-26T06:58:13.551 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5e76515cf9] 2024-07-26T06:58:13.551 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5e75e09c02] 2024-07-26T06:58:13.551 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5e75e8ec40] 2024-07-26T06:58:13.551 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:13.551 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 2024-07-26T06:58:13.552 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 0> 2024-07-26T06:58:13.137+0000 7f5e709e7640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:13.552 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: in thread 7f5e709e7640 thread_name:ms_dispatch 2024-07-26T06:58:13.552 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 2024-07-26T06:58:13.552 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:13.552 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5e75dbe6f0] 2024-07-26T06:58:13.552 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5e75e0b94c] 2024-07-26T06:58:13.552 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 3: raise() 2024-07-26T06:58:13.552 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 4: abort() 2024-07-26T06:58:13.552 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5e75da871b] 2024-07-26T06:58:13.552 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 6: /lib64/libc.so.6(+0x37386) [0x7f5e75db7386] 2024-07-26T06:58:13.552 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5602a03e56b9] 2024-07-26T06:58:13.552 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5602a05a380c] 2024-07-26T06:58:13.552 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5602a04e32ca] 2024-07-26T06:58:13.553 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5602a04b3753] 2024-07-26T06:58:13.553 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5602a04b4a1b] 2024-07-26T06:58:13.553 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5602a0498752] 2024-07-26T06:58:13.553 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5602a03e6a56] 2024-07-26T06:58:13.553 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5602a03ea637] 2024-07-26T06:58:13.553 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5602a03eaf5e] 2024-07-26T06:58:13.553 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 18: (DispatchQueue::entry()+0x60f) [0x7fe4f4a58cf9] 2024-07-26T06:58:13.553 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe4f4b355a1] 2024-07-26T06:58:13.553 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 20: (Thread::entry_wrapper()+0x43) [0x7fe4f48e2cdd] 2024-07-26T06:58:13.553 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe4f48e2cf9] 2024-07-26T06:58:13.554 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe4f41d6c02] 2024-07-26T06:58:13.554 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe4f425bc40] 2024-07-26T06:58:13.554 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:13.554 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 2024-07-26T06:58:13.554 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: -4> 2024-07-26T06:58:13.136+0000 7fe4eedb4640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:13.554 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: in thread 7fe4eedb4640 thread_name:ms_dispatch 2024-07-26T06:58:13.554 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 2024-07-26T06:58:13.554 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:13.554 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe4f418b6f0] 2024-07-26T06:58:13.554 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe4f41d894c] 2024-07-26T06:58:13.554 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 3: raise() 2024-07-26T06:58:13.555 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 4: abort() 2024-07-26T06:58:13.555 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe4f417571b] 2024-07-26T06:58:13.555 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 6: /lib64/libc.so.6(+0x37386) [0x7fe4f4184386] 2024-07-26T06:58:13.555 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5564096856b9] 2024-07-26T06:58:13.555 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55640984380c] 2024-07-26T06:58:13.555 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5564097832ca] 2024-07-26T06:58:13.555 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x556409753753] 2024-07-26T06:58:13.555 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x556409754a1b] 2024-07-26T06:58:13.555 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x556409738752] 2024-07-26T06:58:13.555 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x556409686a56] 2024-07-26T06:58:13.555 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55640968a637] 2024-07-26T06:58:13.555 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55640968af5e] 2024-07-26T06:58:13.556 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5602a03ce03a] 2024-07-26T06:58:13.556 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5e7668b407] 2024-07-26T06:58:13.556 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 18: (DispatchQueue::entry()+0x60f) [0x7f5e7668bcf9] 2024-07-26T06:58:13.556 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5e767685a1] 2024-07-26T06:58:13.556 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 20: (Thread::entry_wrapper()+0x43) [0x7f5e76515cdd] 2024-07-26T06:58:13.556 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5e76515cf9] 2024-07-26T06:58:13.556 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5e75e09c02] 2024-07-26T06:58:13.556 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5e75e8ec40] 2024-07-26T06:58:13.556 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:13.556 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b[71760]: 2024-07-26T06:58:13.557 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55640966e03a] 2024-07-26T06:58:13.557 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe4f4a58407] 2024-07-26T06:58:13.557 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 18: (DispatchQueue::entry()+0x60f) [0x7fe4f4a58cf9] 2024-07-26T06:58:13.557 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe4f4b355a1] 2024-07-26T06:58:13.557 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 20: (Thread::entry_wrapper()+0x43) [0x7fe4f48e2cdd] 2024-07-26T06:58:13.557 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe4f48e2cf9] 2024-07-26T06:58:13.557 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe4f41d6c02] 2024-07-26T06:58:13.557 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe4f425bc40] 2024-07-26T06:58:13.557 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:13.557 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[71734]: 2024-07-26T06:58:14.228 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:14 smithi163 ceph-mon[36873]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:58:14.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:14 smithi163 ceph-mon[36873]: Cluster is now healthy 2024-07-26T06:58:14.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:14 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/873838612,v1:172.21.15.163:6835/873838612] up:active 2024-07-26T06:58:14.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:14 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/3101627593,v1:172.21.15.163:6837/3101627593] up:active 2024-07-26T06:58:14.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:14 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6838/193486687,v1:172.21.15.163:6839/193486687] up:active 2024-07-26T06:58:14.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:14 smithi163 ceph-mon[36873]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:58:14.229 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:14 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:58:14.229 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:13 smithi163 podman[71920]: 2024-07-26 06:58:13.93878352 +0000 UTC m=+0.760949565 container died a05826108af7fbec56367fd19716c45523f9c2d98b81190335da0924dca14e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724) 2024-07-26T06:58:14.230 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 podman[71919]: 2024-07-26 06:58:13.925162167 +0000 UTC m=+0.747192778 container remove 7d7ea37a484b7929fae7ff42deaf12bfb3a5fc34af3a2f0afe1bde88ec478ab1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T06:58:14.230 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:13 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:58:14.383 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:14 smithi042 ceph-mon[32775]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:58:14.383 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:14 smithi042 ceph-mon[32775]: Cluster is now healthy 2024-07-26T06:58:14.383 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:14 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/873838612,v1:172.21.15.163:6835/873838612] up:active 2024-07-26T06:58:14.383 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:14 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/3101627593,v1:172.21.15.163:6837/3101627593] up:active 2024-07-26T06:58:14.383 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:14 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6838/193486687,v1:172.21.15.163:6839/193486687] up:active 2024-07-26T06:58:14.383 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:14 smithi042 ceph-mon[32775]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:58:14.384 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:14 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:58:14.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:14 smithi164 ceph-mon[36970]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:58:14.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:14 smithi164 ceph-mon[36970]: Cluster is now healthy 2024-07-26T06:58:14.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:14 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/873838612,v1:172.21.15.163:6835/873838612] up:active 2024-07-26T06:58:14.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:14 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/3101627593,v1:172.21.15.163:6837/3101627593] up:active 2024-07-26T06:58:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:14 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6838/193486687,v1:172.21.15.163:6839/193486687] up:active 2024-07-26T06:58:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:14 smithi164 ceph-mon[36970]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-07-26T06:58:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:14 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T06:58:14.909 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:14 smithi163 podman[71939]: 2024-07-26 06:58:14.564366302 +0000 UTC m=+1.365659353 container died 9828358e61fb570143edaedd969eeb83b66f397c23ed662dea795dbe31c6511e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, ceph=True, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T06:58:14.909 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:14 smithi163 podman[71920]: 2024-07-26 06:58:14.547027917 +0000 UTC m=+1.369193960 container remove a05826108af7fbec56367fd19716c45523f9c2d98b81190335da0924dca14e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T06:58:14.909 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:14 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:58:15.194 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:15 smithi163 ceph-mon[36873]: pgmap v662: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T06:58:15.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:15 smithi042 ceph-mon[32775]: pgmap v662: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T06:58:15.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:15 smithi164 ceph-mon[36970]: pgmap v662: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T06:58:15.489 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:15 smithi163 podman[71939]: 2024-07-26 06:58:15.194169267 +0000 UTC m=+1.995462354 container remove 9828358e61fb570143edaedd969eeb83b66f397c23ed662dea795dbe31c6511e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-07-26T06:58:15.489 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:15 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:58:15.757 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:15 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-26T06:58:15.757 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:15 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T06:58:15.757 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:15 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Consumed 1.094s CPU time. 2024-07-26T06:58:15.757 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:15 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T06:58:15.757 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:15 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Consumed 1.128s CPU time. 2024-07-26T06:58:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:58:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:58:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:58:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:58:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:58:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:58:16.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:58:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:58:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:58:17.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:17 smithi042 ceph-mon[32775]: pgmap v663: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 528 B/s wr, 13 op/s 2024-07-26T06:58:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:17 smithi164 ceph-mon[36970]: pgmap v663: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 528 B/s wr, 13 op/s 2024-07-26T06:58:17.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:17 smithi163 ceph-mon[36873]: pgmap v663: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 528 B/s wr, 13 op/s 2024-07-26T06:58:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:19 smithi164 ceph-mon[36970]: pgmap v664: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 661 B/s wr, 14 op/s 2024-07-26T06:58:19.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:19 smithi163 ceph-mon[36873]: pgmap v664: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 661 B/s wr, 14 op/s 2024-07-26T06:58:19.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:19 smithi042 ceph-mon[32775]: pgmap v664: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 661 B/s wr, 14 op/s 2024-07-26T06:58:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:21 smithi164 ceph-mon[36970]: pgmap v665: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 571 B/s wr, 14 op/s 2024-07-26T06:58:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:21 smithi163 ceph-mon[36873]: pgmap v665: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 571 B/s wr, 14 op/s 2024-07-26T06:58:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:21 smithi042 ceph-mon[32775]: pgmap v665: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 571 B/s wr, 14 op/s 2024-07-26T06:58:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:23.325+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:58:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:23.325+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:58:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:23.325+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:58:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:23 smithi164 ceph-mon[36970]: pgmap v666: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T06:58:23.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:23 smithi163 ceph-mon[36873]: pgmap v666: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T06:58:23.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:23 smithi042 ceph-mon[32775]: pgmap v666: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T06:58:25.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:25 smithi042 ceph-mon[32775]: pgmap v667: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T06:58:25.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:25 smithi164 ceph-mon[36970]: pgmap v667: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T06:58:25.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:25 smithi163 ceph-mon[36873]: pgmap v667: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T06:58:25.786 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:25 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 5. 2024-07-26T06:58:25.786 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:25 smithi163 systemd[1]: Stopped Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:25.786 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:25 smithi163 systemd[1]: Starting Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:58:25.787 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:25 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 5. 2024-07-26T06:58:25.787 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:25 smithi163 systemd[1]: Stopped Ceph mds.b for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:25.787 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:25 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Consumed 1.094s CPU time. 2024-07-26T06:58:25.787 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:25 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Start request repeated too quickly. 2024-07-26T06:58:25.787 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:25 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T06:58:25.788 INFO:journalctl@ceph.mds.b.smithi163.stdout:Jul 26 06:58:25 smithi163 systemd[1]: Failed to start Ceph mds.b for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:25.788 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:25 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 5. 2024-07-26T06:58:25.788 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:25 smithi163 systemd[1]: Stopped Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:25.788 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:25 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Consumed 1.128s CPU time. 2024-07-26T06:58:25.789 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:25 smithi163 systemd[1]: Starting Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:58:25.789 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:25 smithi163 podman[72196]: 2024-07-26 06:58:25.653428743 +0000 UTC m=+0.021788474 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:58:25.789 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:25 smithi163 podman[72196]: 2024-07-26 06:58:25.786683336 +0000 UTC m=+0.155043050 container create ad0f5dfa19605c39ceab57820dbbe83db9e710a72667b35f8056d2ea468946da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-07-26T06:58:26.078 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:26 smithi163 podman[72196]: 2024-07-26 06:58:26.039537708 +0000 UTC m=+0.407897431 container init ad0f5dfa19605c39ceab57820dbbe83db9e710a72667b35f8056d2ea468946da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-07-26T06:58:26.078 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:26 smithi163 podman[72196]: 2024-07-26 06:58:26.042216092 +0000 UTC m=+0.410575812 container start ad0f5dfa19605c39ceab57820dbbe83db9e710a72667b35f8056d2ea468946da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, ceph=True, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-07-26T06:58:26.078 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:26 smithi163 ceph-mds[72232]: -- 172.21.15.163:0/2708477910 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b13766dc20 con 0x55b13b80f180 2024-07-26T06:58:26.078 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:26 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: starting mds.e at 2024-07-26T06:58:26.295 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:26 smithi163 bash[72196]: ad0f5dfa19605c39ceab57820dbbe83db9e710a72667b35f8056d2ea468946da 2024-07-26T06:58:26.295 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:26 smithi163 systemd[1]: Started Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:26.295 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:26 smithi163 ceph-mon[36873]: daemon mds.e restarted 2024-07-26T06:58:26.295 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:26 smithi163 ceph-mon[36873]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:58:26.295 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:26 smithi163 ceph-mon[36873]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T06:58:26.295 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:26 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/3386290186,v1:172.21.15.163:6835/3386290186] up:boot 2024-07-26T06:58:26.295 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:26 smithi163 ceph-mon[36873]: Standby daemon mds.a assigned to filesystem cephfs as rank 4 2024-07-26T06:58:26.295 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:26 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:active,3=b=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:58:26.296 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:58:26.296 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:26 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:58:26.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:26 smithi042 ceph-mon[32775]: daemon mds.e restarted 2024-07-26T06:58:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:26 smithi042 ceph-mon[32775]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:58:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:26 smithi042 ceph-mon[32775]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T06:58:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:26 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/3386290186,v1:172.21.15.163:6835/3386290186] up:boot 2024-07-26T06:58:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:26 smithi042 ceph-mon[32775]: Standby daemon mds.a assigned to filesystem cephfs as rank 4 2024-07-26T06:58:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:26 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:active,3=b=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:58:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:58:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:26 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:58:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:26 smithi164 ceph-mon[36970]: daemon mds.e restarted 2024-07-26T06:58:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:26 smithi164 ceph-mon[36970]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:58:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:26 smithi164 ceph-mon[36970]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T06:58:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:26 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/3386290186,v1:172.21.15.163:6835/3386290186] up:boot 2024-07-26T06:58:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:26 smithi164 ceph-mon[36970]: Standby daemon mds.a assigned to filesystem cephfs as rank 4 2024-07-26T06:58:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:26 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:active,3=b=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:58:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T06:58:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:26 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T06:58:26.586 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:26 smithi163 podman[72207]: 2024-07-26 06:58:26.197459864 +0000 UTC m=+0.551542235 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:58:26.586 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:26 smithi163 podman[72207]: 2024-07-26 06:58:26.333592549 +0000 UTC m=+0.687674906 container create f5a843b15f3fe3a4235bb17d16b3e75b4e1ba0b79aff9d5f58ca191c5784dbd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T06:58:26.587 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:26 smithi163 podman[72207]: 2024-07-26 06:58:26.586475816 +0000 UTC m=+0.940558173 container init f5a843b15f3fe3a4235bb17d16b3e75b4e1ba0b79aff9d5f58ca191c5784dbd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-07-26T06:58:27.007 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:26 smithi163 podman[72207]: 2024-07-26 06:58:26.589366079 +0000 UTC m=+0.943448436 container start f5a843b15f3fe3a4235bb17d16b3e75b4e1ba0b79aff9d5f58ca191c5784dbd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-07-26T06:58:27.007 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:26 smithi163 ceph-mds[72268]: -- 172.21.15.163:0/271366758 <== mon.2 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558779f7b860 con 0x55877e11d600 2024-07-26T06:58:27.007 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:26 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: starting mds.h at 2024-07-26T06:58:27.007 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:26 smithi163 bash[72207]: f5a843b15f3fe3a4235bb17d16b3e75b4e1ba0b79aff9d5f58ca191c5784dbd8 2024-07-26T06:58:27.007 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:26 smithi163 systemd[1]: Started Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:27 smithi164 ceph-mon[36970]: pgmap v668: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T06:58:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:27 smithi164 ceph-mon[36970]: daemon mds.h restarted 2024-07-26T06:58:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:27 smithi164 ceph-mon[36970]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T06:58:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:27 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/2749045649,v1:172.21.15.42:6839/2749045649] up:resolve 2024-07-26T06:58:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:27 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/2363093091,v1:172.21.15.163:6837/2363093091] up:boot 2024-07-26T06:58:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:27 smithi164 ceph-mon[36970]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-07-26T06:58:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:27 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=b=up:active,4=a=up:resolve} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:58:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:58:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:27 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:58:27.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:27 smithi163 ceph-mon[36873]: pgmap v668: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T06:58:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:27 smithi163 ceph-mon[36873]: daemon mds.h restarted 2024-07-26T06:58:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:27 smithi163 ceph-mon[36873]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T06:58:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:27 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/2749045649,v1:172.21.15.42:6839/2749045649] up:resolve 2024-07-26T06:58:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:27 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/2363093091,v1:172.21.15.163:6837/2363093091] up:boot 2024-07-26T06:58:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:27 smithi163 ceph-mon[36873]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-07-26T06:58:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:27 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=b=up:active,4=a=up:resolve} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:58:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:58:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:27 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:58:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:27 smithi042 ceph-mon[32775]: pgmap v668: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T06:58:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:27 smithi042 ceph-mon[32775]: daemon mds.h restarted 2024-07-26T06:58:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:27 smithi042 ceph-mon[32775]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T06:58:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:27 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/2749045649,v1:172.21.15.42:6839/2749045649] up:resolve 2024-07-26T06:58:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:27 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/2363093091,v1:172.21.15.163:6837/2363093091] up:boot 2024-07-26T06:58:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:27 smithi042 ceph-mon[32775]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-07-26T06:58:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:27 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=b=up:active,4=a=up:resolve} 4 up:standby-replay 4 up:standby, 1 failed 2024-07-26T06:58:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T06:58:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:27 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:58:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:29 smithi164 ceph-mon[36970]: pgmap v671: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T06:58:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:29 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/4063114254,v1:172.21.15.164:6833/4063114254] up:resolve 2024-07-26T06:58:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:29 smithi164 ceph-mon[36970]: Replacing daemon mds.b as rank 3 with standby daemon mds.g 2024-07-26T06:58:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:29 smithi164 ceph-mon[36970]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-07-26T06:58:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:29 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-07-26T06:58:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:29 smithi164 ceph-mon[36970]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T06:58:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:29 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 1 up:replay 2 up:resolve 2 up:active 2024-07-26T06:58:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:29 smithi163 ceph-mon[36873]: pgmap v671: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T06:58:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:29 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/4063114254,v1:172.21.15.164:6833/4063114254] up:resolve 2024-07-26T06:58:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:29 smithi163 ceph-mon[36873]: Replacing daemon mds.b as rank 3 with standby daemon mds.g 2024-07-26T06:58:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:29 smithi163 ceph-mon[36873]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-07-26T06:58:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:29 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-07-26T06:58:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:29 smithi163 ceph-mon[36873]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T06:58:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:29 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 1 up:replay 2 up:resolve 2 up:active 2024-07-26T06:58:29.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:29 smithi042 ceph-mon[32775]: pgmap v671: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T06:58:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:29 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/4063114254,v1:172.21.15.164:6833/4063114254] up:resolve 2024-07-26T06:58:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:29 smithi042 ceph-mon[32775]: Replacing daemon mds.b as rank 3 with standby daemon mds.g 2024-07-26T06:58:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:29 smithi042 ceph-mon[32775]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-07-26T06:58:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:29 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-07-26T06:58:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:29 smithi042 ceph-mon[32775]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T06:58:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:29 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 1 up:replay 2 up:resolve 2 up:active 2024-07-26T06:58:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:30 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/582863084,v1:172.21.15.42:6841/582863084] up:resolve 2024-07-26T06:58:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:30 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 3 up:resolve 2 up:active 2024-07-26T06:58:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:30 smithi164 ceph-mon[36970]: pgmap v673: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 13 op/s 2024-07-26T06:58:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:30 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/582863084,v1:172.21.15.42:6841/582863084] up:resolve 2024-07-26T06:58:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:30 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 3 up:resolve 2 up:active 2024-07-26T06:58:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:30 smithi163 ceph-mon[36873]: pgmap v673: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 13 op/s 2024-07-26T06:58:30.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:30 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/582863084,v1:172.21.15.42:6841/582863084] up:resolve 2024-07-26T06:58:30.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:30 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 3 up:resolve 2 up:active 2024-07-26T06:58:30.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:30 smithi042 ceph-mon[32775]: pgmap v673: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 13 op/s 2024-07-26T06:58:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:31 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:58:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:31 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:58:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:31 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999983 2024-07-26T06:58:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:31 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/4063114254,v1:172.21.15.164:6833/4063114254] up:reconnect 2024-07-26T06:58:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:31 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/2749045649,v1:172.21.15.42:6839/2749045649] up:reconnect 2024-07-26T06:58:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:31 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/582863084,v1:172.21.15.42:6841/582863084] up:reconnect 2024-07-26T06:58:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:31 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:58:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:31 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:58:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:31 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:58:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:31 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999983 2024-07-26T06:58:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:31 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/4063114254,v1:172.21.15.164:6833/4063114254] up:reconnect 2024-07-26T06:58:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:31 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/2749045649,v1:172.21.15.42:6839/2749045649] up:reconnect 2024-07-26T06:58:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:31 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/582863084,v1:172.21.15.42:6841/582863084] up:reconnect 2024-07-26T06:58:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:31 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:58:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:31 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:58:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:31 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:58:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:31 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999983 2024-07-26T06:58:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:31 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/4063114254,v1:172.21.15.164:6833/4063114254] up:reconnect 2024-07-26T06:58:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:31 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/2749045649,v1:172.21.15.42:6839/2749045649] up:reconnect 2024-07-26T06:58:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:31 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/582863084,v1:172.21.15.42:6841/582863084] up:reconnect 2024-07-26T06:58:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:31 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:58:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/2749045649,v1:172.21.15.42:6839/2749045649] up:rejoin 2024-07-26T06:58:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/582863084,v1:172.21.15.42:6841/582863084] up:rejoin 2024-07-26T06:58:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/4063114254,v1:172.21.15.164:6833/4063114254] up:rejoin 2024-07-26T06:58:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:58:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-mon[36970]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:58:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-mon[36970]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-07-26T06:58:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-mon[36970]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-07-26T06:58:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-mon[36970]: pgmap v674: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T06:58:32.473 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:58:32.474 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: *** Caught signal (Aborted) ** 2024-07-26T06:58:32.474 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: in thread 7fb4352b3640 thread_name:ms_dispatch 2024-07-26T06:58:32.474 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:32.474 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb43a68a6f0] 2024-07-26T06:58:32.474 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb43a6d794c] 2024-07-26T06:58:32.474 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 3: raise() 2024-07-26T06:58:32.474 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 4: abort() 2024-07-26T06:58:32.474 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb43a67471b] 2024-07-26T06:58:32.474 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 6: /lib64/libc.so.6(+0x37386) [0x7fb43a683386] 2024-07-26T06:58:32.474 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5583e89996b9] 2024-07-26T06:58:32.474 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5583e8b5780c] 2024-07-26T06:58:32.475 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5583e8a972ca] 2024-07-26T06:58:32.475 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5583e8a67753] 2024-07-26T06:58:32.475 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5583e8a68a1b] 2024-07-26T06:58:32.475 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5583e8a4c752] 2024-07-26T06:58:32.475 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5583e899aa56] 2024-07-26T06:58:32.475 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5583e899e637] 2024-07-26T06:58:32.475 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5583e899ef5e] 2024-07-26T06:58:32.475 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5583e898203a] 2024-07-26T06:58:32.475 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb43af57407] 2024-07-26T06:58:32.475 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 18: (DispatchQueue::entry()+0x60f) [0x7fb43af57cf9] 2024-07-26T06:58:32.475 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb43b0345a1] 2024-07-26T06:58:32.476 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 20: (Thread::entry_wrapper()+0x43) [0x7fb43ade1cdd] 2024-07-26T06:58:32.476 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb43ade1cf9] 2024-07-26T06:58:32.476 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb43a6d5c02] 2024-07-26T06:58:32.476 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb43a75ac40] 2024-07-26T06:58:32.476 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 2024-07-26T06:58:32.207+0000 7fb4352b3640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:32.476 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: in thread 7fb4352b3640 thread_name:ms_dispatch 2024-07-26T06:58:32.476 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 2024-07-26T06:58:32.476 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:32.476 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb43a68a6f0] 2024-07-26T06:58:32.476 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb43a6d794c] 2024-07-26T06:58:32.476 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 3: raise() 2024-07-26T06:58:32.476 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 4: abort() 2024-07-26T06:58:32.476 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb43a67471b] 2024-07-26T06:58:32.477 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 6: /lib64/libc.so.6(+0x37386) [0x7fb43a683386] 2024-07-26T06:58:32.477 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5583e89996b9] 2024-07-26T06:58:32.477 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5583e8b5780c] 2024-07-26T06:58:32.477 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5583e8a972ca] 2024-07-26T06:58:32.477 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5583e8a67753] 2024-07-26T06:58:32.477 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5583e8a68a1b] 2024-07-26T06:58:32.477 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5583e8a4c752] 2024-07-26T06:58:32.477 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5583e899aa56] 2024-07-26T06:58:32.477 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5583e899e637] 2024-07-26T06:58:32.477 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5583e899ef5e] 2024-07-26T06:58:32.477 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5583e898203a] 2024-07-26T06:58:32.477 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb43af57407] 2024-07-26T06:58:32.477 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 18: (DispatchQueue::entry()+0x60f) [0x7fb43af57cf9] 2024-07-26T06:58:32.478 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb43b0345a1] 2024-07-26T06:58:32.478 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 20: (Thread::entry_wrapper()+0x43) [0x7fb43ade1cdd] 2024-07-26T06:58:32.478 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb43ade1cf9] 2024-07-26T06:58:32.478 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb43a6d5c02] 2024-07-26T06:58:32.478 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb43a75ac40] 2024-07-26T06:58:32.478 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:32.478 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 2024-07-26T06:58:32.478 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 0> 2024-07-26T06:58:32.207+0000 7fb4352b3640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:32.478 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: in thread 7fb4352b3640 thread_name:ms_dispatch 2024-07-26T06:58:32.478 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 2024-07-26T06:58:32.478 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:32.478 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb43a68a6f0] 2024-07-26T06:58:32.479 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb43a6d794c] 2024-07-26T06:58:32.479 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 3: raise() 2024-07-26T06:58:32.479 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 4: abort() 2024-07-26T06:58:32.479 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb43a67471b] 2024-07-26T06:58:32.479 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 6: /lib64/libc.so.6(+0x37386) [0x7fb43a683386] 2024-07-26T06:58:32.479 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5583e89996b9] 2024-07-26T06:58:32.479 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5583e8b5780c] 2024-07-26T06:58:32.479 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5583e8a972ca] 2024-07-26T06:58:32.479 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5583e8a67753] 2024-07-26T06:58:32.479 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5583e8a68a1b] 2024-07-26T06:58:32.479 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5583e8a4c752] 2024-07-26T06:58:32.479 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5583e899aa56] 2024-07-26T06:58:32.479 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5583e899e637] 2024-07-26T06:58:32.480 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5583e899ef5e] 2024-07-26T06:58:32.480 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5583e898203a] 2024-07-26T06:58:32.480 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb43af57407] 2024-07-26T06:58:32.480 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 18: (DispatchQueue::entry()+0x60f) [0x7fb43af57cf9] 2024-07-26T06:58:32.480 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb43b0345a1] 2024-07-26T06:58:32.480 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 20: (Thread::entry_wrapper()+0x43) [0x7fb43ade1cdd] 2024-07-26T06:58:32.480 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb43ade1cf9] 2024-07-26T06:58:32.481 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb43a6d5c02] 2024-07-26T06:58:32.481 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb43a75ac40] 2024-07-26T06:58:32.481 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:32.481 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 2024-07-26T06:58:32.481 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 0> 2024-07-26T06:58:32.207+0000 7fb4352b3640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:32.481 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: in thread 7fb4352b3640 thread_name:ms_dispatch 2024-07-26T06:58:32.481 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 2024-07-26T06:58:32.481 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:32.481 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb43a68a6f0] 2024-07-26T06:58:32.481 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb43a6d794c] 2024-07-26T06:58:32.481 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 3: raise() 2024-07-26T06:58:32.481 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 4: abort() 2024-07-26T06:58:32.481 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb43a67471b] 2024-07-26T06:58:32.482 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 6: /lib64/libc.so.6(+0x37386) [0x7fb43a683386] 2024-07-26T06:58:32.482 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5583e89996b9] 2024-07-26T06:58:32.482 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5583e8b5780c] 2024-07-26T06:58:32.482 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5583e8a972ca] 2024-07-26T06:58:32.482 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5583e8a67753] 2024-07-26T06:58:32.482 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5583e8a68a1b] 2024-07-26T06:58:32.482 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5583e8a4c752] 2024-07-26T06:58:32.482 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5583e899aa56] 2024-07-26T06:58:32.482 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5583e899e637] 2024-07-26T06:58:32.482 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5583e899ef5e] 2024-07-26T06:58:32.482 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5583e898203a] 2024-07-26T06:58:32.482 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb43af57407] 2024-07-26T06:58:32.483 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 18: (DispatchQueue::entry()+0x60f) [0x7fb43af57cf9] 2024-07-26T06:58:32.484 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb43b0345a1] 2024-07-26T06:58:32.484 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 20: (Thread::entry_wrapper()+0x43) [0x7fb43ade1cdd] 2024-07-26T06:58:32.484 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb43ade1cf9] 2024-07-26T06:58:32.484 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb43a6d5c02] 2024-07-26T06:58:32.484 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb43a75ac40] 2024-07-26T06:58:32.484 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:32.484 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71115]: 2024-07-26T06:58:32.484 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 podman[71285]: 2024-07-26 06:58:32.271634681 +0000 UTC m=+0.019898044 container died 5bb60cf0fda784128b905112dbb68da162bcaedceffa4e7d5d6ed0d050d9ad79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-07-26T06:58:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:32 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/2749045649,v1:172.21.15.42:6839/2749045649] up:rejoin 2024-07-26T06:58:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:32 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/582863084,v1:172.21.15.42:6841/582863084] up:rejoin 2024-07-26T06:58:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:32 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/4063114254,v1:172.21.15.164:6833/4063114254] up:rejoin 2024-07-26T06:58:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:32 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:58:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:32 smithi163 ceph-mon[36873]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:58:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:32 smithi163 ceph-mon[36873]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-07-26T06:58:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:32 smithi163 ceph-mon[36873]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-07-26T06:58:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:32 smithi163 ceph-mon[36873]: pgmap v674: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T06:58:32.617 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/2749045649,v1:172.21.15.42:6839/2749045649] up:rejoin 2024-07-26T06:58:32.617 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/582863084,v1:172.21.15.42:6841/582863084] up:rejoin 2024-07-26T06:58:32.617 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/4063114254,v1:172.21.15.164:6833/4063114254] up:rejoin 2024-07-26T06:58:32.617 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:58:32.617 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-mon[32775]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:58:32.617 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-mon[32775]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-07-26T06:58:32.617 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-mon[32775]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-07-26T06:58:32.618 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-mon[32775]: pgmap v674: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T06:58:32.618 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:58:32.618 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: *** Caught signal (Aborted) ** 2024-07-26T06:58:32.618 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: in thread 7fab97d66640 thread_name:ms_dispatch 2024-07-26T06:58:32.618 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:32.619 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fab9d13d6f0] 2024-07-26T06:58:32.619 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fab9d18a94c] 2024-07-26T06:58:32.619 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 3: raise() 2024-07-26T06:58:32.619 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 4: abort() 2024-07-26T06:58:32.619 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 5: /lib64/libc.so.6(+0x2871b) [0x7fab9d12771b] 2024-07-26T06:58:32.619 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 6: /lib64/libc.so.6(+0x37386) [0x7fab9d136386] 2024-07-26T06:58:32.619 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5599c55216b9] 2024-07-26T06:58:32.619 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5599c56df80c] 2024-07-26T06:58:32.619 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5599c561f2ca] 2024-07-26T06:58:32.619 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5599c55ef753] 2024-07-26T06:58:32.619 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5599c55f0a1b] 2024-07-26T06:58:32.619 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5599c55d4752] 2024-07-26T06:58:32.620 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5599c5522a56] 2024-07-26T06:58:32.620 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5599c5526637] 2024-07-26T06:58:32.620 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5599c5526f5e] 2024-07-26T06:58:32.620 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5599c550a03a] 2024-07-26T06:58:32.621 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fab9da0a407] 2024-07-26T06:58:32.621 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 18: (DispatchQueue::entry()+0x60f) [0x7fab9da0acf9] 2024-07-26T06:58:32.621 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fab9dae75a1] 2024-07-26T06:58:32.621 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:58:32.621 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: *** Caught signal (Aborted) ** 2024-07-26T06:58:32.621 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: in thread 7f53b8439640 thread_name:ms_dispatch 2024-07-26T06:58:32.621 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:32.622 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f53bd8106f0] 2024-07-26T06:58:32.622 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f53bd85d94c] 2024-07-26T06:58:32.622 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 3: raise() 2024-07-26T06:58:32.622 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 4: abort() 2024-07-26T06:58:32.622 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 5: /lib64/libc.so.6(+0x2871b) [0x7f53bd7fa71b] 2024-07-26T06:58:32.622 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 6: /lib64/libc.so.6(+0x37386) [0x7f53bd809386] 2024-07-26T06:58:32.622 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cbd178c6b9] 2024-07-26T06:58:32.622 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cbd194a80c] 2024-07-26T06:58:32.622 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55cbd188a2ca] 2024-07-26T06:58:32.622 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55cbd185a753] 2024-07-26T06:58:32.622 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cbd185ba1b] 2024-07-26T06:58:32.622 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cbd183f752] 2024-07-26T06:58:32.623 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cbd178da56] 2024-07-26T06:58:32.623 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cbd1791637] 2024-07-26T06:58:32.623 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cbd1791f5e] 2024-07-26T06:58:32.623 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cbd177503a] 2024-07-26T06:58:32.623 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f53be0dd407] 2024-07-26T06:58:32.623 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 18: (DispatchQueue::entry()+0x60f) [0x7f53be0ddcf9] 2024-07-26T06:58:32.623 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f53be1ba5a1] 2024-07-26T06:58:32.623 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 20: (Thread::entry_wrapper()+0x43) [0x7fab9d894cdd] 2024-07-26T06:58:32.623 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 21: (Thread::_entry_func(void*)+0xd) [0x7fab9d894cf9] 2024-07-26T06:58:32.623 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 22: /lib64/libc.so.6(+0x89c02) [0x7fab9d188c02] 2024-07-26T06:58:32.624 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fab9d20dc40] 2024-07-26T06:58:32.624 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 2024-07-26T06:58:32.208+0000 7fab97d66640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:32.624 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: in thread 7fab97d66640 thread_name:ms_dispatch 2024-07-26T06:58:32.624 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 2024-07-26T06:58:32.624 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:32.624 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fab9d13d6f0] 2024-07-26T06:58:32.624 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fab9d18a94c] 2024-07-26T06:58:32.624 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 3: raise() 2024-07-26T06:58:32.624 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 4: abort() 2024-07-26T06:58:32.624 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 5: /lib64/libc.so.6(+0x2871b) [0x7fab9d12771b] 2024-07-26T06:58:32.624 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 6: /lib64/libc.so.6(+0x37386) [0x7fab9d136386] 2024-07-26T06:58:32.624 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5599c55216b9] 2024-07-26T06:58:32.624 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5599c56df80c] 2024-07-26T06:58:32.625 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5599c561f2ca] 2024-07-26T06:58:32.625 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5599c55ef753] 2024-07-26T06:58:32.625 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5599c55f0a1b] 2024-07-26T06:58:32.625 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5599c55d4752] 2024-07-26T06:58:32.625 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5599c5522a56] 2024-07-26T06:58:32.625 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5599c5526637] 2024-07-26T06:58:32.625 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5599c5526f5e] 2024-07-26T06:58:32.625 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5599c550a03a] 2024-07-26T06:58:32.625 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fab9da0a407] 2024-07-26T06:58:32.625 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 18: (DispatchQueue::entry()+0x60f) [0x7fab9da0acf9] 2024-07-26T06:58:32.625 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fab9dae75a1] 2024-07-26T06:58:32.626 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 20: (Thread::entry_wrapper()+0x43) [0x7f53bdf67cdd] 2024-07-26T06:58:32.626 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 21: (Thread::_entry_func(void*)+0xd) [0x7f53bdf67cf9] 2024-07-26T06:58:32.626 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 22: /lib64/libc.so.6(+0x89c02) [0x7f53bd85bc02] 2024-07-26T06:58:32.626 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f53bd8e0c40] 2024-07-26T06:58:32.626 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 2024-07-26T06:58:32.209+0000 7f53b8439640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:32.626 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: in thread 7f53b8439640 thread_name:ms_dispatch 2024-07-26T06:58:32.626 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 2024-07-26T06:58:32.626 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:32.626 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f53bd8106f0] 2024-07-26T06:58:32.626 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f53bd85d94c] 2024-07-26T06:58:32.626 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 3: raise() 2024-07-26T06:58:32.626 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 4: abort() 2024-07-26T06:58:32.627 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 5: /lib64/libc.so.6(+0x2871b) [0x7f53bd7fa71b] 2024-07-26T06:58:32.627 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 6: /lib64/libc.so.6(+0x37386) [0x7f53bd809386] 2024-07-26T06:58:32.627 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cbd178c6b9] 2024-07-26T06:58:32.627 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cbd194a80c] 2024-07-26T06:58:32.627 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55cbd188a2ca] 2024-07-26T06:58:32.627 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55cbd185a753] 2024-07-26T06:58:32.627 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cbd185ba1b] 2024-07-26T06:58:32.627 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cbd183f752] 2024-07-26T06:58:32.627 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cbd178da56] 2024-07-26T06:58:32.627 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cbd1791637] 2024-07-26T06:58:32.627 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cbd1791f5e] 2024-07-26T06:58:32.627 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cbd177503a] 2024-07-26T06:58:32.628 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f53be0dd407] 2024-07-26T06:58:32.628 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 18: (DispatchQueue::entry()+0x60f) [0x7f53be0ddcf9] 2024-07-26T06:58:32.628 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f53be1ba5a1] 2024-07-26T06:58:32.628 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 20: (Thread::entry_wrapper()+0x43) [0x7fab9d894cdd] 2024-07-26T06:58:32.628 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 21: (Thread::_entry_func(void*)+0xd) [0x7fab9d894cf9] 2024-07-26T06:58:32.628 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 22: /lib64/libc.so.6(+0x89c02) [0x7fab9d188c02] 2024-07-26T06:58:32.628 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fab9d20dc40] 2024-07-26T06:58:32.628 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:32.628 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 2024-07-26T06:58:32.628 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 0> 2024-07-26T06:58:32.208+0000 7fab97d66640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:32.629 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: in thread 7fab97d66640 thread_name:ms_dispatch 2024-07-26T06:58:32.629 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 2024-07-26T06:58:32.629 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:32.629 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fab9d13d6f0] 2024-07-26T06:58:32.629 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fab9d18a94c] 2024-07-26T06:58:32.629 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 3: raise() 2024-07-26T06:58:32.629 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 4: abort() 2024-07-26T06:58:32.629 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 5: /lib64/libc.so.6(+0x2871b) [0x7fab9d12771b] 2024-07-26T06:58:32.629 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 6: /lib64/libc.so.6(+0x37386) [0x7fab9d136386] 2024-07-26T06:58:32.629 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5599c55216b9] 2024-07-26T06:58:32.629 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5599c56df80c] 2024-07-26T06:58:32.629 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5599c561f2ca] 2024-07-26T06:58:32.629 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5599c55ef753] 2024-07-26T06:58:32.629 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5599c55f0a1b] 2024-07-26T06:58:32.630 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5599c55d4752] 2024-07-26T06:58:32.630 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5599c5522a56] 2024-07-26T06:58:32.630 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5599c5526637] 2024-07-26T06:58:32.630 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5599c5526f5e] 2024-07-26T06:58:32.630 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5599c550a03a] 2024-07-26T06:58:32.630 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 20: (Thread::entry_wrapper()+0x43) [0x7f53bdf67cdd] 2024-07-26T06:58:32.630 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 21: (Thread::_entry_func(void*)+0xd) [0x7f53bdf67cf9] 2024-07-26T06:58:32.630 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 22: /lib64/libc.so.6(+0x89c02) [0x7f53bd85bc02] 2024-07-26T06:58:32.631 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f53bd8e0c40] 2024-07-26T06:58:32.631 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:32.631 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 2024-07-26T06:58:32.631 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 0> 2024-07-26T06:58:32.209+0000 7f53b8439640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:32.631 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: in thread 7f53b8439640 thread_name:ms_dispatch 2024-07-26T06:58:32.631 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 2024-07-26T06:58:32.631 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:32.631 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f53bd8106f0] 2024-07-26T06:58:32.631 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f53bd85d94c] 2024-07-26T06:58:32.631 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 3: raise() 2024-07-26T06:58:32.631 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 4: abort() 2024-07-26T06:58:32.632 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 5: /lib64/libc.so.6(+0x2871b) [0x7f53bd7fa71b] 2024-07-26T06:58:32.632 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 6: /lib64/libc.so.6(+0x37386) [0x7f53bd809386] 2024-07-26T06:58:32.632 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cbd178c6b9] 2024-07-26T06:58:32.632 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cbd194a80c] 2024-07-26T06:58:32.632 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55cbd188a2ca] 2024-07-26T06:58:32.632 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55cbd185a753] 2024-07-26T06:58:32.632 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cbd185ba1b] 2024-07-26T06:58:32.632 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cbd183f752] 2024-07-26T06:58:32.632 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cbd178da56] 2024-07-26T06:58:32.632 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cbd1791637] 2024-07-26T06:58:32.632 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cbd1791f5e] 2024-07-26T06:58:32.632 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cbd177503a] 2024-07-26T06:58:32.633 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fab9da0a407] 2024-07-26T06:58:32.633 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 18: (DispatchQueue::entry()+0x60f) [0x7fab9da0acf9] 2024-07-26T06:58:32.633 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fab9dae75a1] 2024-07-26T06:58:32.633 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 20: (Thread::entry_wrapper()+0x43) [0x7fab9d894cdd] 2024-07-26T06:58:32.633 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 21: (Thread::_entry_func(void*)+0xd) [0x7fab9d894cf9] 2024-07-26T06:58:32.633 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 22: /lib64/libc.so.6(+0x89c02) [0x7fab9d188c02] 2024-07-26T06:58:32.633 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fab9d20dc40] 2024-07-26T06:58:32.633 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:32.634 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 2024-07-26T06:58:32.634 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: -3> 2024-07-26T06:58:32.208+0000 7fab97d66640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:32.634 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: in thread 7fab97d66640 thread_name:ms_dispatch 2024-07-26T06:58:32.634 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 2024-07-26T06:58:32.634 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:32.634 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fab9d13d6f0] 2024-07-26T06:58:32.634 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fab9d18a94c] 2024-07-26T06:58:32.634 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 3: raise() 2024-07-26T06:58:32.634 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 4: abort() 2024-07-26T06:58:32.634 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 5: /lib64/libc.so.6(+0x2871b) [0x7fab9d12771b] 2024-07-26T06:58:32.634 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 6: /lib64/libc.so.6(+0x37386) [0x7fab9d136386] 2024-07-26T06:58:32.634 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5599c55216b9] 2024-07-26T06:58:32.635 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5599c56df80c] 2024-07-26T06:58:32.635 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5599c561f2ca] 2024-07-26T06:58:32.635 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5599c55ef753] 2024-07-26T06:58:32.635 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5599c55f0a1b] 2024-07-26T06:58:32.635 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5599c55d4752] 2024-07-26T06:58:32.635 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5599c5522a56] 2024-07-26T06:58:32.635 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5599c5526637] 2024-07-26T06:58:32.635 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5599c5526f5e] 2024-07-26T06:58:32.635 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5599c550a03a] 2024-07-26T06:58:32.635 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fab9da0a407] 2024-07-26T06:58:32.635 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 18: (DispatchQueue::entry()+0x60f) [0x7fab9da0acf9] 2024-07-26T06:58:32.636 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fab9dae75a1] 2024-07-26T06:58:32.636 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 20: (Thread::entry_wrapper()+0x43) [0x7fab9d894cdd] 2024-07-26T06:58:32.636 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 21: (Thread::_entry_func(void*)+0xd) [0x7fab9d894cf9] 2024-07-26T06:58:32.636 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 22: /lib64/libc.so.6(+0x89c02) [0x7fab9d188c02] 2024-07-26T06:58:32.636 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fab9d20dc40] 2024-07-26T06:58:32.636 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:32.636 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a[119784]: 2024-07-26T06:58:32.636 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f53be0dd407] 2024-07-26T06:58:32.636 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 18: (DispatchQueue::entry()+0x60f) [0x7f53be0ddcf9] 2024-07-26T06:58:32.637 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f53be1ba5a1] 2024-07-26T06:58:32.637 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 20: (Thread::entry_wrapper()+0x43) [0x7f53bdf67cdd] 2024-07-26T06:58:32.637 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 21: (Thread::_entry_func(void*)+0xd) [0x7f53bdf67cf9] 2024-07-26T06:58:32.637 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 22: /lib64/libc.so.6(+0x89c02) [0x7f53bd85bc02] 2024-07-26T06:58:32.637 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f53bd8e0c40] 2024-07-26T06:58:32.637 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:32.637 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 2024-07-26T06:58:32.637 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 0> 2024-07-26T06:58:32.209+0000 7f53b8439640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:32.637 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: in thread 7f53b8439640 thread_name:ms_dispatch 2024-07-26T06:58:32.637 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 2024-07-26T06:58:32.637 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:32.637 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f53bd8106f0] 2024-07-26T06:58:32.638 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f53bd85d94c] 2024-07-26T06:58:32.638 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 3: raise() 2024-07-26T06:58:32.638 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 4: abort() 2024-07-26T06:58:32.638 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 5: /lib64/libc.so.6(+0x2871b) [0x7f53bd7fa71b] 2024-07-26T06:58:32.638 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 6: /lib64/libc.so.6(+0x37386) [0x7f53bd809386] 2024-07-26T06:58:32.638 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55cbd178c6b9] 2024-07-26T06:58:32.638 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55cbd194a80c] 2024-07-26T06:58:32.638 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55cbd188a2ca] 2024-07-26T06:58:32.638 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55cbd185a753] 2024-07-26T06:58:32.638 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55cbd185ba1b] 2024-07-26T06:58:32.638 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55cbd183f752] 2024-07-26T06:58:32.639 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55cbd178da56] 2024-07-26T06:58:32.639 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55cbd1791637] 2024-07-26T06:58:32.639 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55cbd1791f5e] 2024-07-26T06:58:32.639 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55cbd177503a] 2024-07-26T06:58:32.639 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f53be0dd407] 2024-07-26T06:58:32.639 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 18: (DispatchQueue::entry()+0x60f) [0x7f53be0ddcf9] 2024-07-26T06:58:32.639 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f53be1ba5a1] 2024-07-26T06:58:32.639 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 20: (Thread::entry_wrapper()+0x43) [0x7f53bdf67cdd] 2024-07-26T06:58:32.639 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 21: (Thread::_entry_func(void*)+0xd) [0x7f53bdf67cf9] 2024-07-26T06:58:32.639 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 22: /lib64/libc.so.6(+0x89c02) [0x7f53bd85bc02] 2024-07-26T06:58:32.639 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f53bd8e0c40] 2024-07-26T06:58:32.639 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:32.640 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[119847]: 2024-07-26T06:58:32.640 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 podman[120052]: 2024-07-26 06:58:32.289253607 +0000 UTC m=+0.026607040 container died 0a4993e9b9a2d3e242642b01d380d902803d4b656b759335cdf5e7aff2aa43ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-07-26T06:58:33.181 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 podman[71285]: 2024-07-26 06:58:32.927624126 +0000 UTC m=+0.675887535 container remove 5bb60cf0fda784128b905112dbb68da162bcaedceffa4e7d5d6ed0d050d9ad79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-07-26T06:58:33.181 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:32 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:58:33.201 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:33 smithi042 podman[120053]: 2024-07-26 06:58:33.022849182 +0000 UTC m=+0.760203425 container died e0cde491b61e7b7e3a1b3d1a808bc47583b6fc12009c8564de164186e996251c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-07-26T06:58:33.202 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 podman[120052]: 2024-07-26 06:58:32.947124202 +0000 UTC m=+0.684477681 container remove 0a4993e9b9a2d3e242642b01d380d902803d4b656b759335cdf5e7aff2aa43ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.build-date=20240724, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-07-26T06:58:33.202 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:32 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:58:33.472 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:33 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-07-26T06:58:33.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:33 smithi164 ceph-mon[36970]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:58:33.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:33 smithi164 ceph-mon[36970]: Cluster is now healthy 2024-07-26T06:58:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:33 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/4063114254,v1:172.21.15.164:6833/4063114254] up:active 2024-07-26T06:58:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:33 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/2749045649,v1:172.21.15.42:6839/2749045649] up:active 2024-07-26T06:58:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:33 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6840/582863084,v1:172.21.15.42:6841/582863084] up:active 2024-07-26T06:58:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:33 smithi164 ceph-mon[36970]: fsmap cephfs:5 2 up:standby-replay 4 up:standby 5 up:active 2024-07-26T06:58:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:33 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T06:58:33.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:33 smithi163 ceph-mon[36873]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:58:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:33 smithi163 ceph-mon[36873]: Cluster is now healthy 2024-07-26T06:58:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:33 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/4063114254,v1:172.21.15.164:6833/4063114254] up:active 2024-07-26T06:58:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:33 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/2749045649,v1:172.21.15.42:6839/2749045649] up:active 2024-07-26T06:58:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:33 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6840/582863084,v1:172.21.15.42:6841/582863084] up:active 2024-07-26T06:58:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:33 smithi163 ceph-mon[36873]: fsmap cephfs:5 2 up:standby-replay 4 up:standby 5 up:active 2024-07-26T06:58:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:33 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T06:58:33.602 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:33 smithi042 ceph-mon[32775]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:58:33.602 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:33 smithi042 ceph-mon[32775]: Cluster is now healthy 2024-07-26T06:58:33.602 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:33 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/4063114254,v1:172.21.15.164:6833/4063114254] up:active 2024-07-26T06:58:33.602 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:33 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/2749045649,v1:172.21.15.42:6839/2749045649] up:active 2024-07-26T06:58:33.603 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:33 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6840/582863084,v1:172.21.15.42:6841/582863084] up:active 2024-07-26T06:58:33.603 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:33 smithi042 ceph-mon[32775]: fsmap cephfs:5 2 up:standby-replay 4 up:standby 5 up:active 2024-07-26T06:58:33.603 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:33 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T06:58:33.855 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:33 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-07-26T06:58:33.856 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:33 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Consumed 1.006s CPU time. 2024-07-26T06:58:33.856 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:33 smithi042 podman[120053]: 2024-07-26 06:58:33.602197998 +0000 UTC m=+1.339552231 container remove e0cde491b61e7b7e3a1b3d1a808bc47583b6fc12009c8564de164186e996251c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724) 2024-07-26T06:58:33.856 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:33 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:58:34.144 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:33 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T06:58:34.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:34 smithi164 ceph-mon[36970]: pgmap v675: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 413 B/s wr, 12 op/s 2024-07-26T06:58:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:34 smithi163 ceph-mon[36873]: pgmap v675: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 413 B/s wr, 12 op/s 2024-07-26T06:58:34.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:34 smithi042 ceph-mon[32775]: pgmap v675: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 413 B/s wr, 12 op/s 2024-07-26T06:58:36.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:36 smithi042 ceph-mon[32775]: pgmap v676: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 691 B/s wr, 12 op/s 2024-07-26T06:58:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:36 smithi164 ceph-mon[36970]: pgmap v676: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 691 B/s wr, 12 op/s 2024-07-26T06:58:37.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:36 smithi163 ceph-mon[36873]: pgmap v676: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 691 B/s wr, 12 op/s 2024-07-26T06:58:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:38.331+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:58:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:38.331+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:58:38.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:38.331+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:58:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:38 smithi042 ceph-mon[32775]: pgmap v677: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 614 B/s wr, 12 op/s 2024-07-26T06:58:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:38 smithi164 ceph-mon[36970]: pgmap v677: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 614 B/s wr, 12 op/s 2024-07-26T06:58:39.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:38 smithi163 ceph-mon[36873]: pgmap v677: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 614 B/s wr, 12 op/s 2024-07-26T06:58:40.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:40 smithi042 ceph-mon[32775]: pgmap v678: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 541 B/s wr, 14 op/s 2024-07-26T06:58:40.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:40 smithi164 ceph-mon[36970]: pgmap v678: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 541 B/s wr, 14 op/s 2024-07-26T06:58:41.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:40 smithi163 ceph-mon[36873]: pgmap v678: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 541 B/s wr, 14 op/s 2024-07-26T06:58:42.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:42 smithi042 ceph-mon[32775]: pgmap v679: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T06:58:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:42 smithi164 ceph-mon[36970]: pgmap v679: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T06:58:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:42 smithi163 ceph-mon[36873]: pgmap v679: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T06:58:43.232 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:43 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 5. 2024-07-26T06:58:43.233 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:43 smithi164 systemd[1]: Stopped Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:43.600 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:43 smithi164 systemd[1]: Starting Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:58:43.601 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:43 smithi164 podman[71390]: 2024-07-26 06:58:43.348860894 +0000 UTC m=+0.017652140 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:58:43.601 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:43 smithi164 podman[71390]: 2024-07-26 06:58:43.478464068 +0000 UTC m=+0.147255304 container create d5d7fddc93186985800143b64102c14a94c095a4aef748110da1ebea5242e092 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-07-26T06:58:43.972 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:43 smithi164 podman[71390]: 2024-07-26 06:58:43.71473494 +0000 UTC m=+0.383526177 container init d5d7fddc93186985800143b64102c14a94c095a4aef748110da1ebea5242e092 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724) 2024-07-26T06:58:43.972 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:43 smithi164 podman[71390]: 2024-07-26 06:58:43.717450157 +0000 UTC m=+0.386241402 container start d5d7fddc93186985800143b64102c14a94c095a4aef748110da1ebea5242e092 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux ) 2024-07-26T06:58:43.972 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:43 smithi164 ceph-mds[71409]: -- 172.21.15.164:0/472208770 <== mon.1 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556879b75c20 con 0x55687dd17600 2024-07-26T06:58:43.972 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:43 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: starting mds.l at 2024-07-26T06:58:43.972 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:43 smithi164 bash[71390]: d5d7fddc93186985800143b64102c14a94c095a4aef748110da1ebea5242e092 2024-07-26T06:58:43.972 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:58:43 smithi164 systemd[1]: Started Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:44.141 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:43 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 5. 2024-07-26T06:58:44.141 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:43 smithi042 systemd[1]: Stopped Ceph mds.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:44.141 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:43 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Start request repeated too quickly. 2024-07-26T06:58:44.141 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:43 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T06:58:44.141 INFO:journalctl@ceph.mds.a.smithi042.stdout:Jul 26 06:58:43 smithi042 systemd[1]: Failed to start Ceph mds.a for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:44.141 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:43 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 5. 2024-07-26T06:58:44.142 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:43 smithi042 systemd[1]: Stopped Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:44.142 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:43 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Consumed 1.006s CPU time. 2024-07-26T06:58:44.142 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:43 smithi042 systemd[1]: Starting Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:58:44.394 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:44 smithi042 podman[120186]: 2024-07-26 06:58:44.043870499 +0000 UTC m=+0.018828884 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:58:44.394 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:44 smithi042 podman[120186]: 2024-07-26 06:58:44.178597196 +0000 UTC m=+0.153555559 container create 76596ac32e19bb6708b6bc51fa825d8a3221be79da9cbb466af37f1121a1adeb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-07-26T06:58:44.895 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:44 smithi042 podman[120186]: 2024-07-26 06:58:44.44843877 +0000 UTC m=+0.423397145 container init 76596ac32e19bb6708b6bc51fa825d8a3221be79da9cbb466af37f1121a1adeb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-07-26T06:58:44.895 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:44 smithi042 podman[120186]: 2024-07-26 06:58:44.451505364 +0000 UTC m=+0.426463737 container start 76596ac32e19bb6708b6bc51fa825d8a3221be79da9cbb466af37f1121a1adeb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-07-26T06:58:44.895 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mds[120205]: -- 172.21.15.42:0/3683475877 <== mon.0 v2:172.21.15.42:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56144ffd3c20 con 0x561454175600 2024-07-26T06:58:44.895 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: starting mds.g at 2024-07-26T06:58:44.895 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:44 smithi042 bash[120186]: 76596ac32e19bb6708b6bc51fa825d8a3221be79da9cbb466af37f1121a1adeb 2024-07-26T06:58:44.895 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:58:44 smithi042 systemd[1]: Started Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:58:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mon[32775]: pgmap v680: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T06:58:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mon[32775]: daemon mds.l restarted 2024-07-26T06:58:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mon[32775]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:58:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mon[32775]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T06:58:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/570239393,v1:172.21.15.164:6833/570239393] up:boot 2024-07-26T06:58:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mon[32775]: Standby daemon mds.h assigned to filesystem cephfs as rank 2 2024-07-26T06:58:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=g=up:active,4=a=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-26T06:58:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:58:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T06:58:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mon[32775]: daemon mds.g restarted 2024-07-26T06:58:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mon[32775]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T06:58:44.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/2363093091,v1:172.21.15.163:6837/2363093091] up:resolve 2024-07-26T06:58:44.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3851243510,v1:172.21.15.42:6839/3851243510] up:boot 2024-07-26T06:58:44.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mon[32775]: Standby daemon mds.e assigned to filesystem cephfs as rank 3 2024-07-26T06:58:44.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,4=a=up:active} 4 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:58:44.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:58:44.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:44 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:58:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:44 smithi164 ceph-mon[36970]: pgmap v680: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T06:58:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:44 smithi164 ceph-mon[36970]: daemon mds.l restarted 2024-07-26T06:58:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:44 smithi164 ceph-mon[36970]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:58:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:44 smithi164 ceph-mon[36970]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T06:58:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:44 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/570239393,v1:172.21.15.164:6833/570239393] up:boot 2024-07-26T06:58:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:44 smithi164 ceph-mon[36970]: Standby daemon mds.h assigned to filesystem cephfs as rank 2 2024-07-26T06:58:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:44 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=g=up:active,4=a=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-26T06:58:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:58:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:44 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T06:58:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:44 smithi164 ceph-mon[36970]: daemon mds.g restarted 2024-07-26T06:58:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:44 smithi164 ceph-mon[36970]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T06:58:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:44 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/2363093091,v1:172.21.15.163:6837/2363093091] up:resolve 2024-07-26T06:58:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:44 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3851243510,v1:172.21.15.42:6839/3851243510] up:boot 2024-07-26T06:58:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:44 smithi164 ceph-mon[36970]: Standby daemon mds.e assigned to filesystem cephfs as rank 3 2024-07-26T06:58:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:44 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,4=a=up:active} 4 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:58:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:58:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:44 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:58:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:44 smithi163 ceph-mon[36873]: pgmap v680: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T06:58:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:44 smithi163 ceph-mon[36873]: daemon mds.l restarted 2024-07-26T06:58:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:44 smithi163 ceph-mon[36873]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:58:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:44 smithi163 ceph-mon[36873]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T06:58:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:44 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/570239393,v1:172.21.15.164:6833/570239393] up:boot 2024-07-26T06:58:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:44 smithi163 ceph-mon[36873]: Standby daemon mds.h assigned to filesystem cephfs as rank 2 2024-07-26T06:58:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:44 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,3=g=up:active,4=a=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-26T06:58:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T06:58:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:44 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T06:58:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:44 smithi163 ceph-mon[36873]: daemon mds.g restarted 2024-07-26T06:58:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:44 smithi163 ceph-mon[36873]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T06:58:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:44 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/2363093091,v1:172.21.15.163:6837/2363093091] up:resolve 2024-07-26T06:58:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:44 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3851243510,v1:172.21.15.42:6839/3851243510] up:boot 2024-07-26T06:58:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:44 smithi163 ceph-mon[36873]: Standby daemon mds.e assigned to filesystem cephfs as rank 3 2024-07-26T06:58:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:44 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:resolve,4=a=up:active} 4 up:standby-replay 3 up:standby, 1 failed 2024-07-26T06:58:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T06:58:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:44 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:58:46.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:46 smithi042 ceph-mon[32775]: pgmap v683: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-07-26T06:58:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:46 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/3386290186,v1:172.21.15.163:6835/3386290186] up:resolve 2024-07-26T06:58:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:46 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:resolve 3 up:active 2024-07-26T06:58:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:46 smithi164 ceph-mon[36970]: pgmap v683: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-07-26T06:58:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:46 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/3386290186,v1:172.21.15.163:6835/3386290186] up:resolve 2024-07-26T06:58:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:46 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:resolve 3 up:active 2024-07-26T06:58:47.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:46 smithi163 ceph-mon[36873]: pgmap v683: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-07-26T06:58:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:46 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/3386290186,v1:172.21.15.163:6835/3386290186] up:resolve 2024-07-26T06:58:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:46 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:resolve 3 up:active 2024-07-26T06:58:48.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:48 smithi042 ceph-mon[32775]: pgmap v684: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T06:58:48.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:48 smithi164 ceph-mon[36970]: pgmap v684: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T06:58:49.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:48 smithi163 ceph-mon[36873]: pgmap v684: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T06:58:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:49 smithi042 ceph-mon[32775]: Replacing daemon mds.a as rank 4 with standby daemon mds.d 2024-07-26T06:58:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:49 smithi042 ceph-mon[32775]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T06:58:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:49 smithi042 ceph-mon[32775]: osdmap e112: 12 total, 12 up, 12 in 2024-07-26T06:58:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:49 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 3 up:standby 1 up:replay 2 up:resolve 2 up:active 2024-07-26T06:58:49.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:49 smithi164 ceph-mon[36970]: Replacing daemon mds.a as rank 4 with standby daemon mds.d 2024-07-26T06:58:49.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:49 smithi164 ceph-mon[36970]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T06:58:49.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:49 smithi164 ceph-mon[36970]: osdmap e112: 12 total, 12 up, 12 in 2024-07-26T06:58:49.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:49 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 3 up:standby 1 up:replay 2 up:resolve 2 up:active 2024-07-26T06:58:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:49 smithi163 ceph-mon[36873]: Replacing daemon mds.a as rank 4 with standby daemon mds.d 2024-07-26T06:58:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:49 smithi163 ceph-mon[36873]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T06:58:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:49 smithi163 ceph-mon[36873]: osdmap e112: 12 total, 12 up, 12 in 2024-07-26T06:58:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:49 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 3 up:standby 1 up:replay 2 up:resolve 2 up:active 2024-07-26T06:58:50.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:50 smithi042 ceph-mon[32775]: pgmap v686: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 13 op/s 2024-07-26T06:58:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:50 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6836/1420377891,v1:172.21.15.42:6837/1420377891] up:resolve 2024-07-26T06:58:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:50 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 3 up:standby 3 up:resolve 2 up:active 2024-07-26T06:58:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:50 smithi164 ceph-mon[36970]: pgmap v686: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 13 op/s 2024-07-26T06:58:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:50 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6836/1420377891,v1:172.21.15.42:6837/1420377891] up:resolve 2024-07-26T06:58:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:50 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 3 up:standby 3 up:resolve 2 up:active 2024-07-26T06:58:51.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:50 smithi163 ceph-mon[36873]: pgmap v686: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 13 op/s 2024-07-26T06:58:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:50 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6836/1420377891,v1:172.21.15.42:6837/1420377891] up:resolve 2024-07-26T06:58:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:50 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 3 up:standby 3 up:resolve 2 up:active 2024-07-26T06:58:52.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:52 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999991 2024-07-26T06:58:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:52 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:58:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:52 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:58:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:52 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/2363093091,v1:172.21.15.163:6837/2363093091] up:reconnect 2024-07-26T06:58:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:52 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/3386290186,v1:172.21.15.163:6835/3386290186] up:reconnect 2024-07-26T06:58:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:52 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6836/1420377891,v1:172.21.15.42:6837/1420377891] up:reconnect 2024-07-26T06:58:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:52 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 3 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:58:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:52 smithi042 ceph-mon[32775]: pgmap v687: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 10 op/s 2024-07-26T06:58:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:52 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999991 2024-07-26T06:58:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:52 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:58:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:52 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:58:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:52 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/2363093091,v1:172.21.15.163:6837/2363093091] up:reconnect 2024-07-26T06:58:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:52 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/3386290186,v1:172.21.15.163:6835/3386290186] up:reconnect 2024-07-26T06:58:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:52 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6836/1420377891,v1:172.21.15.42:6837/1420377891] up:reconnect 2024-07-26T06:58:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:52 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 3 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:58:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:52 smithi164 ceph-mon[36970]: pgmap v687: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 10 op/s 2024-07-26T06:58:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:52 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999991 2024-07-26T06:58:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:52 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:58:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:52 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999978 2024-07-26T06:58:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:52 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/2363093091,v1:172.21.15.163:6837/2363093091] up:reconnect 2024-07-26T06:58:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:52 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/3386290186,v1:172.21.15.163:6835/3386290186] up:reconnect 2024-07-26T06:58:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:52 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6836/1420377891,v1:172.21.15.42:6837/1420377891] up:reconnect 2024-07-26T06:58:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:52 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 3 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:58:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:52 smithi163 ceph-mon[36873]: pgmap v687: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 10 op/s 2024-07-26T06:58:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:53.334+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:58:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:53.334+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:58:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:53.334+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:58:53.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6836/1420377891,v1:172.21.15.42:6837/1420377891] up:rejoin 2024-07-26T06:58:53.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/3386290186,v1:172.21.15.163:6835/3386290186] up:rejoin 2024-07-26T06:58:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/2363093091,v1:172.21.15.163:6837/2363093091] up:rejoin 2024-07-26T06:58:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 3 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:58:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-mon[32775]: daemon mds.d is now active in filesystem cephfs as rank 4 2024-07-26T06:58:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-mon[32775]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-07-26T06:58:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-mon[32775]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-07-26T06:58:53.646 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:58:53.646 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: *** Caught signal (Aborted) ** 2024-07-26T06:58:53.646 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: in thread 7f7ef06cb640 thread_name:ms_dispatch 2024-07-26T06:58:53.646 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:53.646 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7ef5aa26f0] 2024-07-26T06:58:53.646 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7ef5aef94c] 2024-07-26T06:58:53.646 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 3: raise() 2024-07-26T06:58:53.647 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 4: abort() 2024-07-26T06:58:53.647 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7ef5a8c71b] 2024-07-26T06:58:53.647 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 6: /lib64/libc.so.6(+0x37386) [0x7f7ef5a9b386] 2024-07-26T06:58:53.647 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5559f5c666b9] 2024-07-26T06:58:53.647 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5559f5e2480c] 2024-07-26T06:58:53.647 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5559f5d642ca] 2024-07-26T06:58:53.647 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5559f5d34753] 2024-07-26T06:58:53.647 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5559f5d35a1b] 2024-07-26T06:58:53.647 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5559f5d19752] 2024-07-26T06:58:53.647 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5559f5c67a56] 2024-07-26T06:58:53.647 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5559f5c6b637] 2024-07-26T06:58:53.647 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5559f5c6bf5e] 2024-07-26T06:58:53.647 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5559f5c4f03a] 2024-07-26T06:58:53.648 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7ef636f407] 2024-07-26T06:58:53.648 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 18: (DispatchQueue::entry()+0x60f) [0x7f7ef636fcf9] 2024-07-26T06:58:53.648 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f7ef644c5a1] 2024-07-26T06:58:53.648 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 20: (Thread::entry_wrapper()+0x43) [0x7f7ef61f9cdd] 2024-07-26T06:58:53.648 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7ef61f9cf9] 2024-07-26T06:58:53.648 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7ef5aedc02] 2024-07-26T06:58:53.648 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7ef5b72c40] 2024-07-26T06:58:53.648 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 2024-07-26T06:58:53.347+0000 7f7ef06cb640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:53.648 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: in thread 7f7ef06cb640 thread_name:ms_dispatch 2024-07-26T06:58:53.648 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 2024-07-26T06:58:53.648 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:53.648 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7ef5aa26f0] 2024-07-26T06:58:53.648 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7ef5aef94c] 2024-07-26T06:58:53.648 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 3: raise() 2024-07-26T06:58:53.649 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 4: abort() 2024-07-26T06:58:53.649 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7ef5a8c71b] 2024-07-26T06:58:53.649 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 6: /lib64/libc.so.6(+0x37386) [0x7f7ef5a9b386] 2024-07-26T06:58:53.649 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5559f5c666b9] 2024-07-26T06:58:53.649 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5559f5e2480c] 2024-07-26T06:58:53.649 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5559f5d642ca] 2024-07-26T06:58:53.649 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5559f5d34753] 2024-07-26T06:58:53.649 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5559f5d35a1b] 2024-07-26T06:58:53.649 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5559f5d19752] 2024-07-26T06:58:53.649 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5559f5c67a56] 2024-07-26T06:58:53.649 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5559f5c6b637] 2024-07-26T06:58:53.649 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5559f5c6bf5e] 2024-07-26T06:58:53.649 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5559f5c4f03a] 2024-07-26T06:58:53.650 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7ef636f407] 2024-07-26T06:58:53.650 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 18: (DispatchQueue::entry()+0x60f) [0x7f7ef636fcf9] 2024-07-26T06:58:53.650 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f7ef644c5a1] 2024-07-26T06:58:53.650 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 20: (Thread::entry_wrapper()+0x43) [0x7f7ef61f9cdd] 2024-07-26T06:58:53.650 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7ef61f9cf9] 2024-07-26T06:58:53.650 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7ef5aedc02] 2024-07-26T06:58:53.650 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7ef5b72c40] 2024-07-26T06:58:53.650 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:53.650 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 2024-07-26T06:58:53.650 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 0> 2024-07-26T06:58:53.347+0000 7f7ef06cb640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:53.650 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: in thread 7f7ef06cb640 thread_name:ms_dispatch 2024-07-26T06:58:53.651 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 2024-07-26T06:58:53.651 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:53.651 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7ef5aa26f0] 2024-07-26T06:58:53.651 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7ef5aef94c] 2024-07-26T06:58:53.651 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 3: raise() 2024-07-26T06:58:53.651 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 4: abort() 2024-07-26T06:58:53.651 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7ef5a8c71b] 2024-07-26T06:58:53.651 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 6: /lib64/libc.so.6(+0x37386) [0x7f7ef5a9b386] 2024-07-26T06:58:53.651 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5559f5c666b9] 2024-07-26T06:58:53.651 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5559f5e2480c] 2024-07-26T06:58:53.651 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5559f5d642ca] 2024-07-26T06:58:53.651 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5559f5d34753] 2024-07-26T06:58:53.651 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5559f5d35a1b] 2024-07-26T06:58:53.651 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5559f5d19752] 2024-07-26T06:58:53.652 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5559f5c67a56] 2024-07-26T06:58:53.652 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5559f5c6b637] 2024-07-26T06:58:53.652 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5559f5c6bf5e] 2024-07-26T06:58:53.652 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5559f5c4f03a] 2024-07-26T06:58:53.652 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7ef636f407] 2024-07-26T06:58:53.652 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 18: (DispatchQueue::entry()+0x60f) [0x7f7ef636fcf9] 2024-07-26T06:58:53.652 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f7ef644c5a1] 2024-07-26T06:58:53.653 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 20: (Thread::entry_wrapper()+0x43) [0x7f7ef61f9cdd] 2024-07-26T06:58:53.653 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7ef61f9cf9] 2024-07-26T06:58:53.653 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7ef5aedc02] 2024-07-26T06:58:53.653 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7ef5b72c40] 2024-07-26T06:58:53.653 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:53.653 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 2024-07-26T06:58:53.653 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 0> 2024-07-26T06:58:53.347+0000 7f7ef06cb640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:53.653 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: in thread 7f7ef06cb640 thread_name:ms_dispatch 2024-07-26T06:58:53.653 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 2024-07-26T06:58:53.653 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:53.653 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7ef5aa26f0] 2024-07-26T06:58:53.654 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7ef5aef94c] 2024-07-26T06:58:53.654 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 3: raise() 2024-07-26T06:58:53.654 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 4: abort() 2024-07-26T06:58:53.654 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7ef5a8c71b] 2024-07-26T06:58:53.654 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 6: /lib64/libc.so.6(+0x37386) [0x7f7ef5a9b386] 2024-07-26T06:58:53.654 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5559f5c666b9] 2024-07-26T06:58:53.654 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5559f5e2480c] 2024-07-26T06:58:53.654 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5559f5d642ca] 2024-07-26T06:58:53.654 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5559f5d34753] 2024-07-26T06:58:53.654 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5559f5d35a1b] 2024-07-26T06:58:53.654 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5559f5d19752] 2024-07-26T06:58:53.654 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5559f5c67a56] 2024-07-26T06:58:53.655 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5559f5c6b637] 2024-07-26T06:58:53.655 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5559f5c6bf5e] 2024-07-26T06:58:53.655 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5559f5c4f03a] 2024-07-26T06:58:53.655 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7ef636f407] 2024-07-26T06:58:53.655 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 18: (DispatchQueue::entry()+0x60f) [0x7f7ef636fcf9] 2024-07-26T06:58:53.655 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f7ef644c5a1] 2024-07-26T06:58:53.655 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 20: (Thread::entry_wrapper()+0x43) [0x7f7ef61f9cdd] 2024-07-26T06:58:53.655 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7ef61f9cf9] 2024-07-26T06:58:53.656 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7ef5aedc02] 2024-07-26T06:58:53.656 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7ef5b72c40] 2024-07-26T06:58:53.656 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:53.656 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[76187]: 2024-07-26T06:58:53.656 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:53 smithi042 podman[120250]: 2024-07-26 06:58:53.425590466 +0000 UTC m=+0.022709460 container died 4e393b9ca4095f4d7dfbcbdf5d466c8c74c17d726e5c27b040584902c7f6d270 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T06:58:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:53 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6836/1420377891,v1:172.21.15.42:6837/1420377891] up:rejoin 2024-07-26T06:58:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:53 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/3386290186,v1:172.21.15.163:6835/3386290186] up:rejoin 2024-07-26T06:58:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:53 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/2363093091,v1:172.21.15.163:6837/2363093091] up:rejoin 2024-07-26T06:58:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:53 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 3 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:58:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:53 smithi164 ceph-mon[36970]: daemon mds.d is now active in filesystem cephfs as rank 4 2024-07-26T06:58:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:53 smithi164 ceph-mon[36970]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-07-26T06:58:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:53 smithi164 ceph-mon[36970]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-07-26T06:58:53.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6836/1420377891,v1:172.21.15.42:6837/1420377891] up:rejoin 2024-07-26T06:58:53.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/3386290186,v1:172.21.15.163:6835/3386290186] up:rejoin 2024-07-26T06:58:53.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/2363093091,v1:172.21.15.163:6837/2363093091] up:rejoin 2024-07-26T06:58:53.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 3 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:58:53.727 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-mon[36873]: daemon mds.d is now active in filesystem cephfs as rank 4 2024-07-26T06:58:53.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-mon[36873]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-07-26T06:58:53.728 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-mon[36873]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-07-26T06:58:53.728 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:58:53.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: *** Caught signal (Aborted) ** 2024-07-26T06:58:53.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: in thread 7efbfff75640 thread_name:ms_dispatch 2024-07-26T06:58:53.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:53.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7efc0534c6f0] 2024-07-26T06:58:53.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 2: /lib64/libc.so.6(+0x8b94c) [0x7efc0539994c] 2024-07-26T06:58:53.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 3: raise() 2024-07-26T06:58:53.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 4: abort() 2024-07-26T06:58:53.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 5: /lib64/libc.so.6(+0x2871b) [0x7efc0533671b] 2024-07-26T06:58:53.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 6: /lib64/libc.so.6(+0x37386) [0x7efc05345386] 2024-07-26T06:58:53.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558778eee6b9] 2024-07-26T06:58:53.729 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5587790ac80c] 2024-07-26T06:58:53.730 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558778fec2ca] 2024-07-26T06:58:53.730 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558778fbc753] 2024-07-26T06:58:53.730 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558778fbda1b] 2024-07-26T06:58:53.730 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558778fa1752] 2024-07-26T06:58:53.730 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558778eefa56] 2024-07-26T06:58:53.730 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558778ef3637] 2024-07-26T06:58:53.730 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558778ef3f5e] 2024-07-26T06:58:53.730 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558778ed703a] 2024-07-26T06:58:53.730 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7efc05c19407] 2024-07-26T06:58:53.730 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 18: (DispatchQueue::entry()+0x60f) [0x7efc05c19cf9] 2024-07-26T06:58:53.730 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7efc05cf65a1] 2024-07-26T06:58:53.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:58:53.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: *** Caught signal (Aborted) ** 2024-07-26T06:58:53.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: in thread 7f13ba33a640 thread_name:ms_dispatch 2024-07-26T06:58:53.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:53.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f13bf7116f0] 2024-07-26T06:58:53.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f13bf75e94c] 2024-07-26T06:58:53.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 3: raise() 2024-07-26T06:58:53.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 4: abort() 2024-07-26T06:58:53.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 5: /lib64/libc.so.6(+0x2871b) [0x7f13bf6fb71b] 2024-07-26T06:58:53.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 6: /lib64/libc.so.6(+0x37386) [0x7f13bf70a386] 2024-07-26T06:58:53.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b135cf76b9] 2024-07-26T06:58:53.731 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b135eb580c] 2024-07-26T06:58:53.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b135df52ca] 2024-07-26T06:58:53.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b135dc5753] 2024-07-26T06:58:53.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b135dc6a1b] 2024-07-26T06:58:53.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b135daa752] 2024-07-26T06:58:53.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b135cf8a56] 2024-07-26T06:58:53.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b135cfc637] 2024-07-26T06:58:53.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b135cfcf5e] 2024-07-26T06:58:53.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b135ce003a] 2024-07-26T06:58:53.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f13bffde407] 2024-07-26T06:58:53.732 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 18: (DispatchQueue::entry()+0x60f) [0x7f13bffdecf9] 2024-07-26T06:58:53.733 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f13c00bb5a1] 2024-07-26T06:58:53.733 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 20: (Thread::entry_wrapper()+0x43) [0x7efc05aa3cdd] 2024-07-26T06:58:53.733 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 21: (Thread::_entry_func(void*)+0xd) [0x7efc05aa3cf9] 2024-07-26T06:58:53.733 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 22: /lib64/libc.so.6(+0x89c02) [0x7efc05397c02] 2024-07-26T06:58:53.733 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 23: /lib64/libc.so.6(+0x10ec40) [0x7efc0541cc40] 2024-07-26T06:58:53.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 2024-07-26T06:58:53.348+0000 7efbfff75640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:53.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: in thread 7efbfff75640 thread_name:ms_dispatch 2024-07-26T06:58:53.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 2024-07-26T06:58:53.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:53.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7efc0534c6f0] 2024-07-26T06:58:53.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 2: /lib64/libc.so.6(+0x8b94c) [0x7efc0539994c] 2024-07-26T06:58:53.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 3: raise() 2024-07-26T06:58:53.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 4: abort() 2024-07-26T06:58:53.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 5: /lib64/libc.so.6(+0x2871b) [0x7efc0533671b] 2024-07-26T06:58:53.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 6: /lib64/libc.so.6(+0x37386) [0x7efc05345386] 2024-07-26T06:58:53.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558778eee6b9] 2024-07-26T06:58:53.734 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5587790ac80c] 2024-07-26T06:58:53.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558778fec2ca] 2024-07-26T06:58:53.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558778fbc753] 2024-07-26T06:58:53.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558778fbda1b] 2024-07-26T06:58:53.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558778fa1752] 2024-07-26T06:58:53.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558778eefa56] 2024-07-26T06:58:53.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558778ef3637] 2024-07-26T06:58:53.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558778ef3f5e] 2024-07-26T06:58:53.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558778ed703a] 2024-07-26T06:58:53.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7efc05c19407] 2024-07-26T06:58:53.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 18: (DispatchQueue::entry()+0x60f) [0x7efc05c19cf9] 2024-07-26T06:58:53.735 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7efc05cf65a1] 2024-07-26T06:58:53.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 20: (Thread::entry_wrapper()+0x43) [0x7efc05aa3cdd] 2024-07-26T06:58:53.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 21: (Thread::_entry_func(void*)+0xd) [0x7efc05aa3cf9] 2024-07-26T06:58:53.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 22: /lib64/libc.so.6(+0x89c02) [0x7efc05397c02] 2024-07-26T06:58:53.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 23: /lib64/libc.so.6(+0x10ec40) [0x7efc0541cc40] 2024-07-26T06:58:53.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:53.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 2024-07-26T06:58:53.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 0> 2024-07-26T06:58:53.348+0000 7efbfff75640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:53.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: in thread 7efbfff75640 thread_name:ms_dispatch 2024-07-26T06:58:53.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 2024-07-26T06:58:53.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:53.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7efc0534c6f0] 2024-07-26T06:58:53.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 2: /lib64/libc.so.6(+0x8b94c) [0x7efc0539994c] 2024-07-26T06:58:53.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 3: raise() 2024-07-26T06:58:53.736 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 4: abort() 2024-07-26T06:58:53.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 5: /lib64/libc.so.6(+0x2871b) [0x7efc0533671b] 2024-07-26T06:58:53.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 6: /lib64/libc.so.6(+0x37386) [0x7efc05345386] 2024-07-26T06:58:53.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558778eee6b9] 2024-07-26T06:58:53.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5587790ac80c] 2024-07-26T06:58:53.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558778fec2ca] 2024-07-26T06:58:53.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558778fbc753] 2024-07-26T06:58:53.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558778fbda1b] 2024-07-26T06:58:53.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558778fa1752] 2024-07-26T06:58:53.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558778eefa56] 2024-07-26T06:58:53.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558778ef3637] 2024-07-26T06:58:53.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558778ef3f5e] 2024-07-26T06:58:53.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558778ed703a] 2024-07-26T06:58:53.737 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7efc05c19407] 2024-07-26T06:58:53.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 20: (Thread::entry_wrapper()+0x43) [0x7f13bfe68cdd] 2024-07-26T06:58:53.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 21: (Thread::_entry_func(void*)+0xd) [0x7f13bfe68cf9] 2024-07-26T06:58:53.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 22: /lib64/libc.so.6(+0x89c02) [0x7f13bf75cc02] 2024-07-26T06:58:53.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f13bf7e1c40] 2024-07-26T06:58:53.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 2024-07-26T06:58:53.349+0000 7f13ba33a640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:53.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: in thread 7f13ba33a640 thread_name:ms_dispatch 2024-07-26T06:58:53.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 2024-07-26T06:58:53.738 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:53.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f13bf7116f0] 2024-07-26T06:58:53.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f13bf75e94c] 2024-07-26T06:58:53.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 3: raise() 2024-07-26T06:58:53.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 4: abort() 2024-07-26T06:58:53.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 5: /lib64/libc.so.6(+0x2871b) [0x7f13bf6fb71b] 2024-07-26T06:58:53.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 6: /lib64/libc.so.6(+0x37386) [0x7f13bf70a386] 2024-07-26T06:58:53.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b135cf76b9] 2024-07-26T06:58:53.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b135eb580c] 2024-07-26T06:58:53.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b135df52ca] 2024-07-26T06:58:53.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b135dc5753] 2024-07-26T06:58:53.739 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b135dc6a1b] 2024-07-26T06:58:53.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b135daa752] 2024-07-26T06:58:53.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b135cf8a56] 2024-07-26T06:58:53.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b135cfc637] 2024-07-26T06:58:53.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b135cfcf5e] 2024-07-26T06:58:53.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b135ce003a] 2024-07-26T06:58:53.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f13bffde407] 2024-07-26T06:58:53.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 18: (DispatchQueue::entry()+0x60f) [0x7f13bffdecf9] 2024-07-26T06:58:53.740 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f13c00bb5a1] 2024-07-26T06:58:53.741 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 18: (DispatchQueue::entry()+0x60f) [0x7efc05c19cf9] 2024-07-26T06:58:53.741 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7efc05cf65a1] 2024-07-26T06:58:53.741 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 20: (Thread::entry_wrapper()+0x43) [0x7efc05aa3cdd] 2024-07-26T06:58:53.741 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 21: (Thread::_entry_func(void*)+0xd) [0x7efc05aa3cf9] 2024-07-26T06:58:53.741 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 22: /lib64/libc.so.6(+0x89c02) [0x7efc05397c02] 2024-07-26T06:58:53.741 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 23: /lib64/libc.so.6(+0x10ec40) [0x7efc0541cc40] 2024-07-26T06:58:53.741 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:53.741 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 2024-07-26T06:58:53.741 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 0> 2024-07-26T06:58:53.348+0000 7efbfff75640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:53.741 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: in thread 7efbfff75640 thread_name:ms_dispatch 2024-07-26T06:58:53.741 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 2024-07-26T06:58:53.741 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:53.742 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7efc0534c6f0] 2024-07-26T06:58:53.742 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 2: /lib64/libc.so.6(+0x8b94c) [0x7efc0539994c] 2024-07-26T06:58:53.742 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 3: raise() 2024-07-26T06:58:53.742 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 4: abort() 2024-07-26T06:58:53.742 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 5: /lib64/libc.so.6(+0x2871b) [0x7efc0533671b] 2024-07-26T06:58:53.742 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 6: /lib64/libc.so.6(+0x37386) [0x7efc05345386] 2024-07-26T06:58:53.742 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558778eee6b9] 2024-07-26T06:58:53.742 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5587790ac80c] 2024-07-26T06:58:53.742 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558778fec2ca] 2024-07-26T06:58:53.742 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558778fbc753] 2024-07-26T06:58:53.742 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558778fbda1b] 2024-07-26T06:58:53.742 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558778fa1752] 2024-07-26T06:58:53.743 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558778eefa56] 2024-07-26T06:58:53.743 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558778ef3637] 2024-07-26T06:58:53.743 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558778ef3f5e] 2024-07-26T06:58:53.743 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558778ed703a] 2024-07-26T06:58:53.743 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7efc05c19407] 2024-07-26T06:58:53.743 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 18: (DispatchQueue::entry()+0x60f) [0x7efc05c19cf9] 2024-07-26T06:58:53.743 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7efc05cf65a1] 2024-07-26T06:58:53.743 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 20: (Thread::entry_wrapper()+0x43) [0x7efc05aa3cdd] 2024-07-26T06:58:53.743 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 21: (Thread::_entry_func(void*)+0xd) [0x7efc05aa3cf9] 2024-07-26T06:58:53.743 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 22: /lib64/libc.so.6(+0x89c02) [0x7efc05397c02] 2024-07-26T06:58:53.744 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 23: /lib64/libc.so.6(+0x10ec40) [0x7efc0541cc40] 2024-07-26T06:58:53.744 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:53.744 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h[72259]: 2024-07-26T06:58:53.744 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:53 smithi163 podman[72320]: 2024-07-26 06:58:53.411281922 +0000 UTC m=+0.020525603 container died f5a843b15f3fe3a4235bb17d16b3e75b4e1ba0b79aff9d5f58ca191c5784dbd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-07-26T06:58:53.744 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 20: (Thread::entry_wrapper()+0x43) [0x7f13bfe68cdd] 2024-07-26T06:58:53.744 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 21: (Thread::_entry_func(void*)+0xd) [0x7f13bfe68cf9] 2024-07-26T06:58:53.744 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 22: /lib64/libc.so.6(+0x89c02) [0x7f13bf75cc02] 2024-07-26T06:58:53.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f13bf7e1c40] 2024-07-26T06:58:53.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:53.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 2024-07-26T06:58:53.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 0> 2024-07-26T06:58:53.349+0000 7f13ba33a640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:53.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: in thread 7f13ba33a640 thread_name:ms_dispatch 2024-07-26T06:58:53.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 2024-07-26T06:58:53.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:53.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f13bf7116f0] 2024-07-26T06:58:53.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f13bf75e94c] 2024-07-26T06:58:53.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 3: raise() 2024-07-26T06:58:53.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 4: abort() 2024-07-26T06:58:53.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 5: /lib64/libc.so.6(+0x2871b) [0x7f13bf6fb71b] 2024-07-26T06:58:53.745 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 6: /lib64/libc.so.6(+0x37386) [0x7f13bf70a386] 2024-07-26T06:58:53.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b135cf76b9] 2024-07-26T06:58:53.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b135eb580c] 2024-07-26T06:58:53.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b135df52ca] 2024-07-26T06:58:53.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b135dc5753] 2024-07-26T06:58:53.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b135dc6a1b] 2024-07-26T06:58:53.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b135daa752] 2024-07-26T06:58:53.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b135cf8a56] 2024-07-26T06:58:53.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b135cfc637] 2024-07-26T06:58:53.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b135cfcf5e] 2024-07-26T06:58:53.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b135ce003a] 2024-07-26T06:58:53.746 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f13bffde407] 2024-07-26T06:58:53.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 18: (DispatchQueue::entry()+0x60f) [0x7f13bffdecf9] 2024-07-26T06:58:53.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f13c00bb5a1] 2024-07-26T06:58:53.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 20: (Thread::entry_wrapper()+0x43) [0x7f13bfe68cdd] 2024-07-26T06:58:53.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 21: (Thread::_entry_func(void*)+0xd) [0x7f13bfe68cf9] 2024-07-26T06:58:53.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 22: /lib64/libc.so.6(+0x89c02) [0x7f13bf75cc02] 2024-07-26T06:58:53.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f13bf7e1c40] 2024-07-26T06:58:53.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:53.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 2024-07-26T06:58:53.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 0> 2024-07-26T06:58:53.349+0000 7f13ba33a640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:58:53.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: in thread 7f13ba33a640 thread_name:ms_dispatch 2024-07-26T06:58:53.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 2024-07-26T06:58:53.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:58:53.747 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f13bf7116f0] 2024-07-26T06:58:53.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f13bf75e94c] 2024-07-26T06:58:53.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 3: raise() 2024-07-26T06:58:53.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 4: abort() 2024-07-26T06:58:53.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 5: /lib64/libc.so.6(+0x2871b) [0x7f13bf6fb71b] 2024-07-26T06:58:53.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 6: /lib64/libc.so.6(+0x37386) [0x7f13bf70a386] 2024-07-26T06:58:53.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b135cf76b9] 2024-07-26T06:58:53.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b135eb580c] 2024-07-26T06:58:53.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b135df52ca] 2024-07-26T06:58:53.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b135dc5753] 2024-07-26T06:58:53.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b135dc6a1b] 2024-07-26T06:58:53.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b135daa752] 2024-07-26T06:58:53.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b135cf8a56] 2024-07-26T06:58:53.748 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b135cfc637] 2024-07-26T06:58:53.749 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b135cfcf5e] 2024-07-26T06:58:53.749 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b135ce003a] 2024-07-26T06:58:53.749 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f13bffde407] 2024-07-26T06:58:53.749 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 18: (DispatchQueue::entry()+0x60f) [0x7f13bffdecf9] 2024-07-26T06:58:53.749 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f13c00bb5a1] 2024-07-26T06:58:53.749 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 20: (Thread::entry_wrapper()+0x43) [0x7f13bfe68cdd] 2024-07-26T06:58:53.749 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 21: (Thread::_entry_func(void*)+0xd) [0x7f13bfe68cf9] 2024-07-26T06:58:53.749 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 22: /lib64/libc.so.6(+0x89c02) [0x7f13bf75cc02] 2024-07-26T06:58:53.750 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f13bf7e1c40] 2024-07-26T06:58:53.750 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:58:53.750 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e[72228]: 2024-07-26T06:58:53.750 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:53 smithi163 podman[72319]: 2024-07-26 06:58:53.411033624 +0000 UTC m=+0.020464768 container died ad0f5dfa19605c39ceab57820dbbe83db9e710a72667b35f8056d2ea468946da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.build-date=20240724) 2024-07-26T06:58:54.447 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:54 smithi163 podman[72319]: 2024-07-26 06:58:54.101094474 +0000 UTC m=+0.710525620 container remove ad0f5dfa19605c39ceab57820dbbe83db9e710a72667b35f8056d2ea468946da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-e, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T06:58:54.447 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:54 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:58:54.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:54 smithi163 ceph-mon[36873]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:58:54.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:54 smithi163 ceph-mon[36873]: Cluster is now healthy 2024-07-26T06:58:54.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:54 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6836/1420377891,v1:172.21.15.42:6837/1420377891] up:active 2024-07-26T06:58:54.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:54 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6834/3386290186,v1:172.21.15.163:6835/3386290186] up:active 2024-07-26T06:58:54.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:54 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.163:6836/2363093091,v1:172.21.15.163:6837/2363093091] up:active 2024-07-26T06:58:54.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:54 smithi163 ceph-mon[36873]: fsmap cephfs:5 2 up:standby-replay 3 up:standby 5 up:active 2024-07-26T06:58:54.447 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:54 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T06:58:54.448 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:54 smithi163 ceph-mon[36873]: pgmap v688: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 113 B/s wr, 7 op/s 2024-07-26T06:58:54.645 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:54 smithi042 podman[120250]: 2024-07-26 06:58:54.330033961 +0000 UTC m=+0.927152946 container remove 4e393b9ca4095f4d7dfbcbdf5d466c8c74c17d726e5c27b040584902c7f6d270 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T06:58:54.645 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:54 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:58:54.645 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:54 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T06:58:54.645 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:58:54 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.d.service: Consumed 1.126s CPU time. 2024-07-26T06:58:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:54 smithi042 ceph-mon[32775]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:58:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:54 smithi042 ceph-mon[32775]: Cluster is now healthy 2024-07-26T06:58:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:54 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6836/1420377891,v1:172.21.15.42:6837/1420377891] up:active 2024-07-26T06:58:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:54 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6834/3386290186,v1:172.21.15.163:6835/3386290186] up:active 2024-07-26T06:58:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:54 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.163:6836/2363093091,v1:172.21.15.163:6837/2363093091] up:active 2024-07-26T06:58:54.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:54 smithi042 ceph-mon[32775]: fsmap cephfs:5 2 up:standby-replay 3 up:standby 5 up:active 2024-07-26T06:58:54.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:54 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T06:58:54.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:54 smithi042 ceph-mon[32775]: pgmap v688: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 113 B/s wr, 7 op/s 2024-07-26T06:58:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:54 smithi164 ceph-mon[36970]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:58:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:54 smithi164 ceph-mon[36970]: Cluster is now healthy 2024-07-26T06:58:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:54 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6836/1420377891,v1:172.21.15.42:6837/1420377891] up:active 2024-07-26T06:58:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:54 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6834/3386290186,v1:172.21.15.163:6835/3386290186] up:active 2024-07-26T06:58:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:54 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.163:6836/2363093091,v1:172.21.15.163:6837/2363093091] up:active 2024-07-26T06:58:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:54 smithi164 ceph-mon[36970]: fsmap cephfs:5 2 up:standby-replay 3 up:standby 5 up:active 2024-07-26T06:58:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:54 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T06:58:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:54 smithi164 ceph-mon[36970]: pgmap v688: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 113 B/s wr, 7 op/s 2024-07-26T06:58:55.001 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:58:54 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T06:58:55.001 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:54 smithi163 podman[72320]: 2024-07-26 06:58:54.748067687 +0000 UTC m=+1.357311370 container remove f5a843b15f3fe3a4235bb17d16b3e75b4e1ba0b79aff9d5f58ca191c5784dbd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-h, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T06:58:55.001 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:54 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:58:55.257 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:58:55 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-26T06:58:56.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:56 smithi042 ceph-mon[32775]: pgmap v689: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 409 B/s wr, 10 op/s 2024-07-26T06:58:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:56 smithi042 ceph-mon[32775]: 1 slow requests, 1 included below; oldest blocked for > 183.849999 secs 2024-07-26T06:58:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:56 smithi042 ceph-mon[32775]: slow request 183.849999 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T06:58:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:56 smithi164 ceph-mon[36970]: pgmap v689: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 409 B/s wr, 10 op/s 2024-07-26T06:58:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:56 smithi164 ceph-mon[36970]: 1 slow requests, 1 included below; oldest blocked for > 183.849999 secs 2024-07-26T06:58:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:56 smithi164 ceph-mon[36970]: slow request 183.849999 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T06:58:57.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:56 smithi163 ceph-mon[36873]: pgmap v689: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 409 B/s wr, 10 op/s 2024-07-26T06:58:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:56 smithi163 ceph-mon[36873]: 1 slow requests, 1 included below; oldest blocked for > 183.849999 secs 2024-07-26T06:58:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:56 smithi163 ceph-mon[36873]: slow request 183.849999 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T06:58:57.882 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:57.882+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580034c0 con 0x7ff680102000 2024-07-26T06:58:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:57.882+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500034c0 con 0x55db6e0564d0 2024-07-26T06:58:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:57.882+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580034c0 con 0x7ff680102000 2024-07-26T06:58:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:58:57.882+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500034c0 con 0x55db6e0564d0 2024-07-26T06:58:58.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:58 smithi042 ceph-mon[32775]: pgmap v690: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 614 B/s wr, 11 op/s 2024-07-26T06:58:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:58 smithi164 ceph-mon[36970]: pgmap v690: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 614 B/s wr, 11 op/s 2024-07-26T06:58:59.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:58 smithi163 ceph-mon[36873]: pgmap v690: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 614 B/s wr, 11 op/s 2024-07-26T06:58:59.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:59 smithi042 ceph-mon[32775]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T06:58:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:59 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] up:active 2024-07-26T06:58:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:58:59 smithi042 ceph-mon[32775]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T06:58:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:59 smithi164 ceph-mon[36970]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T06:58:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:59 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] up:active 2024-07-26T06:58:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:58:59 smithi164 ceph-mon[36970]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T06:59:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:59 smithi163 ceph-mon[36873]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T06:59:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:59 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6838/3593178314,v1:172.21.15.164:6839/3593178314] up:active 2024-07-26T06:59:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:58:59 smithi163 ceph-mon[36873]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T06:59:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:00 smithi042 ceph-mon[32775]: pgmap v691: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 601 B/s wr, 11 op/s 2024-07-26T06:59:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:00 smithi164 ceph-mon[36970]: pgmap v691: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 601 B/s wr, 11 op/s 2024-07-26T06:59:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:00 smithi163 ceph-mon[36873]: pgmap v691: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 601 B/s wr, 11 op/s 2024-07-26T06:59:01.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 188.850121 secs 2024-07-26T06:59:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 188.850121 secs 2024-07-26T06:59:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 188.850121 secs 2024-07-26T06:59:02.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:02 smithi042 ceph-mon[32775]: pgmap v692: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T06:59:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:02 smithi164 ceph-mon[36970]: pgmap v692: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T06:59:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:02 smithi163 ceph-mon[36873]: pgmap v692: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 10 op/s 2024-07-26T06:59:04.906 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:04 smithi042 ceph-mon[32775]: pgmap v693: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T06:59:04.906 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:59:04 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-07-26T06:59:04.906 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:59:04 smithi042 systemd[1]: Stopped Ceph mds.d for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:59:04.906 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:59:04 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.d.service: Consumed 1.126s CPU time. 2024-07-26T06:59:04.906 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:59:04 smithi042 systemd[1]: Starting Ceph mds.d for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:59:04.906 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:59:04 smithi042 podman[120340]: 2024-07-26 06:59:04.787296912 +0000 UTC m=+0.018727146 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:59:04.906 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:59:04 smithi042 podman[120340]: 2024-07-26 06:59:04.906044833 +0000 UTC m=+0.137475104 container create 4fa5f65047a40431cff693ea430ab000a3cb8d2ad05e671cfabc375b280520de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0) 2024-07-26T06:59:04.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:04 smithi164 ceph-mon[36970]: pgmap v693: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T06:59:05.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:04 smithi163 ceph-mon[36873]: pgmap v693: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T06:59:05.185 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:59:05 smithi042 podman[120340]: 2024-07-26 06:59:05.150607431 +0000 UTC m=+0.382037662 container init 4fa5f65047a40431cff693ea430ab000a3cb8d2ad05e671cfabc375b280520de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-07-26T06:59:05.185 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:59:05 smithi042 podman[120340]: 2024-07-26 06:59:05.153649583 +0000 UTC m=+0.385079798 container start 4fa5f65047a40431cff693ea430ab000a3cb8d2ad05e671cfabc375b280520de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T06:59:05.507 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:59:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 6. 2024-07-26T06:59:05.507 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:59:05 smithi163 systemd[1]: Stopped Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:59:05.507 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:59:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Start request repeated too quickly. 2024-07-26T06:59:05.507 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:59:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T06:59:05.507 INFO:journalctl@ceph.mds.e.smithi163.stdout:Jul 26 06:59:05 smithi163 systemd[1]: Failed to start Ceph mds.e for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:59:05.507 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:59:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 6. 2024-07-26T06:59:05.507 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:59:05 smithi163 systemd[1]: Stopped Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:59:05.507 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:59:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Start request repeated too quickly. 2024-07-26T06:59:05.507 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:59:05 smithi163 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-26T06:59:05.508 INFO:journalctl@ceph.mds.h.smithi163.stdout:Jul 26 06:59:05 smithi163 systemd[1]: Failed to start Ceph mds.h for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:59:05.644 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:59:05 smithi042 ceph-mds[120359]: -- 172.21.15.42:0/4227040752 <== mon.0 v2:172.21.15.42:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562805ba3860 con 0x562809d45600 2024-07-26T06:59:05.645 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:59:05 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-d[120355]: starting mds.d at 2024-07-26T06:59:05.645 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:59:05 smithi042 bash[120340]: 4fa5f65047a40431cff693ea430ab000a3cb8d2ad05e671cfabc375b280520de 2024-07-26T06:59:05.645 INFO:journalctl@ceph.mds.d.smithi042.stdout:Jul 26 06:59:05 smithi042 systemd[1]: Started Ceph mds.d for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:59:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:05 smithi164 ceph-mon[36970]: daemon mds.d restarted 2024-07-26T06:59:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:05 smithi164 ceph-mon[36970]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:59:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:05 smithi164 ceph-mon[36970]: osdmap e113: 12 total, 12 up, 12 in 2024-07-26T06:59:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:05 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6836/74811888,v1:172.21.15.42:6837/74811888] up:boot 2024-07-26T06:59:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:05 smithi164 ceph-mon[36970]: Standby daemon mds.j assigned to filesystem cephfs as rank 4 2024-07-26T06:59:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:05 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:active,3=e=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T06:59:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T06:59:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:05 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T06:59:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:05 smithi163 ceph-mon[36873]: daemon mds.d restarted 2024-07-26T06:59:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:05 smithi163 ceph-mon[36873]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:59:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:05 smithi163 ceph-mon[36873]: osdmap e113: 12 total, 12 up, 12 in 2024-07-26T06:59:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:05 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6836/74811888,v1:172.21.15.42:6837/74811888] up:boot 2024-07-26T06:59:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:05 smithi163 ceph-mon[36873]: Standby daemon mds.j assigned to filesystem cephfs as rank 4 2024-07-26T06:59:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:05 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:active,3=e=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T06:59:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T06:59:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:05 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T06:59:06.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:05 smithi042 ceph-mon[32775]: daemon mds.d restarted 2024-07-26T06:59:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:05 smithi042 ceph-mon[32775]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:59:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:05 smithi042 ceph-mon[32775]: osdmap e113: 12 total, 12 up, 12 in 2024-07-26T06:59:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:05 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6836/74811888,v1:172.21.15.42:6837/74811888] up:boot 2024-07-26T06:59:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:05 smithi042 ceph-mon[32775]: Standby daemon mds.j assigned to filesystem cephfs as rank 4 2024-07-26T06:59:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:05 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=h=up:active,3=e=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T06:59:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T06:59:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:05 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T06:59:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:06 smithi164 ceph-mon[36970]: pgmap v695: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T06:59:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 193.850245 secs 2024-07-26T06:59:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:06 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6834/3548166563,v1:172.21.15.42:6835/3548166563] up:resolve 2024-07-26T06:59:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:06 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T06:59:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:06 smithi163 ceph-mon[36873]: pgmap v695: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T06:59:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 193.850245 secs 2024-07-26T06:59:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:06 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6834/3548166563,v1:172.21.15.42:6835/3548166563] up:resolve 2024-07-26T06:59:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:06 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T06:59:07.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:06 smithi042 ceph-mon[32775]: pgmap v695: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T06:59:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 193.850245 secs 2024-07-26T06:59:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:06 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6834/3548166563,v1:172.21.15.42:6835/3548166563] up:resolve 2024-07-26T06:59:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:06 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T06:59:08.342 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:59:08.341+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:59:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:59:08.341+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:59:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:59:08.342+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:59:08.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:08 smithi164 ceph-mon[36970]: pgmap v696: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 10 op/s 2024-07-26T06:59:09.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:08 smithi163 ceph-mon[36873]: pgmap v696: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 10 op/s 2024-07-26T06:59:09.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:08 smithi042 ceph-mon[32775]: pgmap v696: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 10 op/s 2024-07-26T06:59:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:09 smithi164 ceph-mon[36970]: Replacing daemon mds.e as rank 3 with standby daemon mds.g 2024-07-26T06:59:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:09 smithi164 ceph-mon[36970]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-07-26T06:59:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:09 smithi164 ceph-mon[36970]: Replacing daemon mds.h as rank 2 with standby daemon mds.l 2024-07-26T06:59:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:09 smithi164 ceph-mon[36970]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-07-26T06:59:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:09 smithi164 ceph-mon[36970]: osdmap e114: 12 total, 12 up, 12 in 2024-07-26T06:59:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:09 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:59:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:09 smithi163 ceph-mon[36873]: Replacing daemon mds.e as rank 3 with standby daemon mds.g 2024-07-26T06:59:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:09 smithi163 ceph-mon[36873]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-07-26T06:59:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:09 smithi163 ceph-mon[36873]: Replacing daemon mds.h as rank 2 with standby daemon mds.l 2024-07-26T06:59:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:09 smithi163 ceph-mon[36873]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-07-26T06:59:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:09 smithi163 ceph-mon[36873]: osdmap e114: 12 total, 12 up, 12 in 2024-07-26T06:59:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:09 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:59:10.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:09 smithi042 ceph-mon[32775]: Replacing daemon mds.e as rank 3 with standby daemon mds.g 2024-07-26T06:59:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:09 smithi042 ceph-mon[32775]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-07-26T06:59:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:09 smithi042 ceph-mon[32775]: Replacing daemon mds.h as rank 2 with standby daemon mds.l 2024-07-26T06:59:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:09 smithi042 ceph-mon[32775]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-07-26T06:59:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:09 smithi042 ceph-mon[32775]: osdmap e114: 12 total, 12 up, 12 in 2024-07-26T06:59:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:09 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 2 up:replay 1 up:resolve 2 up:active 2024-07-26T06:59:10.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:10 smithi163 ceph-mon[36873]: pgmap v698: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-07-26T06:59:10.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:10 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/570239393,v1:172.21.15.164:6833/570239393] up:resolve 2024-07-26T06:59:10.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:10 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3851243510,v1:172.21.15.42:6839/3851243510] up:resolve 2024-07-26T06:59:10.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:10 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 3 up:resolve 2 up:active 2024-07-26T06:59:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:10 smithi164 ceph-mon[36970]: pgmap v698: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-07-26T06:59:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:10 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/570239393,v1:172.21.15.164:6833/570239393] up:resolve 2024-07-26T06:59:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:10 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3851243510,v1:172.21.15.42:6839/3851243510] up:resolve 2024-07-26T06:59:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:10 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 3 up:resolve 2 up:active 2024-07-26T06:59:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:10 smithi042 ceph-mon[32775]: pgmap v698: 97 pgs: 97 active+clean; 61 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-07-26T06:59:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:10 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/570239393,v1:172.21.15.164:6833/570239393] up:resolve 2024-07-26T06:59:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:10 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3851243510,v1:172.21.15.42:6839/3851243510] up:resolve 2024-07-26T06:59:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:10 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 3 up:resolve 2 up:active 2024-07-26T06:59:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:11 smithi164 ceph-mon[36970]: 1 slow requests, 1 included below; oldest blocked for > 181.108067 secs 2024-07-26T06:59:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:11 smithi164 ceph-mon[36970]: slow request 181.108066 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T06:59:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 198.850345 secs 2024-07-26T06:59:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:11 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:59:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:11 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/570239393,v1:172.21.15.164:6833/570239393] up:reconnect 2024-07-26T06:59:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:11 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6834/3548166563,v1:172.21.15.42:6835/3548166563] up:reconnect 2024-07-26T06:59:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:11 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3851243510,v1:172.21.15.42:6839/3851243510] up:reconnect 2024-07-26T06:59:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:11 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:59:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:11 smithi163 ceph-mon[36873]: 1 slow requests, 1 included below; oldest blocked for > 181.108067 secs 2024-07-26T06:59:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:11 smithi163 ceph-mon[36873]: slow request 181.108066 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T06:59:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 198.850345 secs 2024-07-26T06:59:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:11 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:59:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:11 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/570239393,v1:172.21.15.164:6833/570239393] up:reconnect 2024-07-26T06:59:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:11 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6834/3548166563,v1:172.21.15.42:6835/3548166563] up:reconnect 2024-07-26T06:59:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:11 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3851243510,v1:172.21.15.42:6839/3851243510] up:reconnect 2024-07-26T06:59:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:11 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:59:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:11 smithi042 ceph-mon[32775]: 1 slow requests, 1 included below; oldest blocked for > 181.108067 secs 2024-07-26T06:59:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:11 smithi042 ceph-mon[32775]: slow request 181.108066 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T06:59:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 198.850345 secs 2024-07-26T06:59:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:11 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:59:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:11 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/570239393,v1:172.21.15.164:6833/570239393] up:reconnect 2024-07-26T06:59:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:11 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6834/3548166563,v1:172.21.15.42:6835/3548166563] up:reconnect 2024-07-26T06:59:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:11 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3851243510,v1:172.21.15.42:6839/3851243510] up:reconnect 2024-07-26T06:59:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:11 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 3 up:reconnect 2 up:active 2024-07-26T06:59:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:12 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:59:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:12 smithi164 ceph-mon[36970]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.0199997 2024-07-26T06:59:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:12 smithi164 ceph-mon[36970]: pgmap v699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-07-26T06:59:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:12 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3851243510,v1:172.21.15.42:6839/3851243510] up:rejoin 2024-07-26T06:59:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:12 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6834/3548166563,v1:172.21.15.42:6835/3548166563] up:rejoin 2024-07-26T06:59:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:12 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/570239393,v1:172.21.15.164:6833/570239393] up:rejoin 2024-07-26T06:59:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:12 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:59:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:12 smithi164 ceph-mon[36970]: daemon mds.j is now active in filesystem cephfs as rank 4 2024-07-26T06:59:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:12 smithi164 ceph-mon[36970]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:59:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:12 smithi164 ceph-mon[36970]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-07-26T06:59:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:12 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:59:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:12 smithi163 ceph-mon[36873]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.0199997 2024-07-26T06:59:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:12 smithi163 ceph-mon[36873]: pgmap v699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-07-26T06:59:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:12 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3851243510,v1:172.21.15.42:6839/3851243510] up:rejoin 2024-07-26T06:59:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:12 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6834/3548166563,v1:172.21.15.42:6835/3548166563] up:rejoin 2024-07-26T06:59:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:12 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/570239393,v1:172.21.15.164:6833/570239393] up:rejoin 2024-07-26T06:59:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:12 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:59:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:12 smithi163 ceph-mon[36873]: daemon mds.j is now active in filesystem cephfs as rank 4 2024-07-26T06:59:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:12 smithi163 ceph-mon[36873]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:59:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:12 smithi163 ceph-mon[36873]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-07-26T06:59:13.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:12 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.000999992 2024-07-26T06:59:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:12 smithi042 ceph-mon[32775]: reconnect by client.25018 v1:172.21.15.42:0/110110209 after 0.0199997 2024-07-26T06:59:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:12 smithi042 ceph-mon[32775]: pgmap v699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-07-26T06:59:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:12 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3851243510,v1:172.21.15.42:6839/3851243510] up:rejoin 2024-07-26T06:59:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:12 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6834/3548166563,v1:172.21.15.42:6835/3548166563] up:rejoin 2024-07-26T06:59:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:12 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/570239393,v1:172.21.15.164:6833/570239393] up:rejoin 2024-07-26T06:59:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:12 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 1 up:standby 3 up:rejoin 2 up:active 2024-07-26T06:59:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:12 smithi042 ceph-mon[32775]: daemon mds.j is now active in filesystem cephfs as rank 4 2024-07-26T06:59:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:12 smithi042 ceph-mon[32775]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-07-26T06:59:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:12 smithi042 ceph-mon[32775]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-07-26T06:59:13.645 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:59:13.645 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: *** Caught signal (Aborted) ** 2024-07-26T06:59:13.645 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: in thread 7f21a5540640 thread_name:ms_dispatch 2024-07-26T06:59:13.645 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:59:13.645 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f21aa9176f0] 2024-07-26T06:59:13.646 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f21aa96494c] 2024-07-26T06:59:13.646 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 3: raise() 2024-07-26T06:59:13.646 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 4: abort() 2024-07-26T06:59:13.646 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 5: /lib64/libc.so.6(+0x2871b) [0x7f21aa90171b] 2024-07-26T06:59:13.646 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 6: /lib64/libc.so.6(+0x37386) [0x7f21aa910386] 2024-07-26T06:59:13.646 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56144e4326b9] 2024-07-26T06:59:13.646 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56144e5f080c] 2024-07-26T06:59:13.646 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56144e5302ca] 2024-07-26T06:59:13.646 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56144e500753] 2024-07-26T06:59:13.646 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56144e501a1b] 2024-07-26T06:59:13.646 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56144e4e5752] 2024-07-26T06:59:13.647 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56144e433a56] 2024-07-26T06:59:13.647 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56144e437637] 2024-07-26T06:59:13.647 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56144e437f5e] 2024-07-26T06:59:13.647 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56144e41b03a] 2024-07-26T06:59:13.647 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f21ab1e4407] 2024-07-26T06:59:13.647 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 18: (DispatchQueue::entry()+0x60f) [0x7f21ab1e4cf9] 2024-07-26T06:59:13.647 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f21ab2c15a1] 2024-07-26T06:59:13.647 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:59:13.648 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: *** Caught signal (Aborted) ** 2024-07-26T06:59:13.648 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: in thread 7f032de5d640 thread_name:ms_dispatch 2024-07-26T06:59:13.648 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:59:13.648 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f03332346f0] 2024-07-26T06:59:13.648 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f033328194c] 2024-07-26T06:59:13.648 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 3: raise() 2024-07-26T06:59:13.648 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 4: abort() 2024-07-26T06:59:13.648 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 5: /lib64/libc.so.6(+0x2871b) [0x7f033321e71b] 2024-07-26T06:59:13.648 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 6: /lib64/libc.so.6(+0x37386) [0x7f033322d386] 2024-07-26T06:59:13.648 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5654c25f76b9] 2024-07-26T06:59:13.648 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5654c27b580c] 2024-07-26T06:59:13.648 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5654c26f52ca] 2024-07-26T06:59:13.649 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5654c26c5753] 2024-07-26T06:59:13.649 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5654c26c6a1b] 2024-07-26T06:59:13.649 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5654c26aa752] 2024-07-26T06:59:13.649 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5654c25f8a56] 2024-07-26T06:59:13.649 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5654c25fc637] 2024-07-26T06:59:13.649 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5654c25fcf5e] 2024-07-26T06:59:13.649 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5654c25e003a] 2024-07-26T06:59:13.649 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0333b01407] 2024-07-26T06:59:13.649 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 18: (DispatchQueue::entry()+0x60f) [0x7f0333b01cf9] 2024-07-26T06:59:13.649 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0333bde5a1] 2024-07-26T06:59:13.650 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 20: (Thread::entry_wrapper()+0x43) [0x7f21ab06ecdd] 2024-07-26T06:59:13.650 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 21: (Thread::_entry_func(void*)+0xd) [0x7f21ab06ecf9] 2024-07-26T06:59:13.650 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 22: /lib64/libc.so.6(+0x89c02) [0x7f21aa962c02] 2024-07-26T06:59:13.650 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f21aa9e7c40] 2024-07-26T06:59:13.650 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 2024-07-26T06:59:13.348+0000 7f21a5540640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:59:13.650 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: in thread 7f21a5540640 thread_name:ms_dispatch 2024-07-26T06:59:13.650 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 2024-07-26T06:59:13.651 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:59:13.651 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f21aa9176f0] 2024-07-26T06:59:13.651 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f21aa96494c] 2024-07-26T06:59:13.651 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 3: raise() 2024-07-26T06:59:13.651 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 4: abort() 2024-07-26T06:59:13.651 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 5: /lib64/libc.so.6(+0x2871b) [0x7f21aa90171b] 2024-07-26T06:59:13.651 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 6: /lib64/libc.so.6(+0x37386) [0x7f21aa910386] 2024-07-26T06:59:13.651 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56144e4326b9] 2024-07-26T06:59:13.651 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56144e5f080c] 2024-07-26T06:59:13.651 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56144e5302ca] 2024-07-26T06:59:13.651 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56144e500753] 2024-07-26T06:59:13.652 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56144e501a1b] 2024-07-26T06:59:13.652 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56144e4e5752] 2024-07-26T06:59:13.652 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56144e433a56] 2024-07-26T06:59:13.652 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56144e437637] 2024-07-26T06:59:13.652 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56144e437f5e] 2024-07-26T06:59:13.652 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56144e41b03a] 2024-07-26T06:59:13.652 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f21ab1e4407] 2024-07-26T06:59:13.652 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 18: (DispatchQueue::entry()+0x60f) [0x7f21ab1e4cf9] 2024-07-26T06:59:13.652 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f21ab2c15a1] 2024-07-26T06:59:13.652 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 20: (Thread::entry_wrapper()+0x43) [0x7f21ab06ecdd] 2024-07-26T06:59:13.652 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 21: (Thread::_entry_func(void*)+0xd) [0x7f21ab06ecf9] 2024-07-26T06:59:13.652 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 22: /lib64/libc.so.6(+0x89c02) [0x7f21aa962c02] 2024-07-26T06:59:13.652 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f21aa9e7c40] 2024-07-26T06:59:13.653 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:59:13.653 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 2024-07-26T06:59:13.653 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 0> 2024-07-26T06:59:13.348+0000 7f21a5540640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:59:13.653 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: in thread 7f21a5540640 thread_name:ms_dispatch 2024-07-26T06:59:13.653 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 2024-07-26T06:59:13.653 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:59:13.653 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f21aa9176f0] 2024-07-26T06:59:13.653 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f21aa96494c] 2024-07-26T06:59:13.653 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 3: raise() 2024-07-26T06:59:13.653 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 4: abort() 2024-07-26T06:59:13.653 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 5: /lib64/libc.so.6(+0x2871b) [0x7f21aa90171b] 2024-07-26T06:59:13.653 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 6: /lib64/libc.so.6(+0x37386) [0x7f21aa910386] 2024-07-26T06:59:13.654 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56144e4326b9] 2024-07-26T06:59:13.654 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56144e5f080c] 2024-07-26T06:59:13.654 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56144e5302ca] 2024-07-26T06:59:13.654 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56144e500753] 2024-07-26T06:59:13.654 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56144e501a1b] 2024-07-26T06:59:13.654 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56144e4e5752] 2024-07-26T06:59:13.654 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56144e433a56] 2024-07-26T06:59:13.654 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56144e437637] 2024-07-26T06:59:13.654 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56144e437f5e] 2024-07-26T06:59:13.654 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56144e41b03a] 2024-07-26T06:59:13.655 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 20: (Thread::entry_wrapper()+0x43) [0x7f033398bcdd] 2024-07-26T06:59:13.655 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 21: (Thread::_entry_func(void*)+0xd) [0x7f033398bcf9] 2024-07-26T06:59:13.655 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 22: /lib64/libc.so.6(+0x89c02) [0x7f033327fc02] 2024-07-26T06:59:13.655 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0333304c40] 2024-07-26T06:59:13.655 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 2024-07-26T06:59:13.346+0000 7f032de5d640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:59:13.655 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: in thread 7f032de5d640 thread_name:ms_dispatch 2024-07-26T06:59:13.655 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 2024-07-26T06:59:13.655 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:59:13.655 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f03332346f0] 2024-07-26T06:59:13.655 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f033328194c] 2024-07-26T06:59:13.655 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 3: raise() 2024-07-26T06:59:13.655 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 4: abort() 2024-07-26T06:59:13.656 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 5: /lib64/libc.so.6(+0x2871b) [0x7f033321e71b] 2024-07-26T06:59:13.656 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 6: /lib64/libc.so.6(+0x37386) [0x7f033322d386] 2024-07-26T06:59:13.656 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5654c25f76b9] 2024-07-26T06:59:13.656 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5654c27b580c] 2024-07-26T06:59:13.656 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5654c26f52ca] 2024-07-26T06:59:13.656 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5654c26c5753] 2024-07-26T06:59:13.656 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5654c26c6a1b] 2024-07-26T06:59:13.656 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5654c26aa752] 2024-07-26T06:59:13.656 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5654c25f8a56] 2024-07-26T06:59:13.656 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5654c25fc637] 2024-07-26T06:59:13.656 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5654c25fcf5e] 2024-07-26T06:59:13.656 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5654c25e003a] 2024-07-26T06:59:13.656 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0333b01407] 2024-07-26T06:59:13.656 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 18: (DispatchQueue::entry()+0x60f) [0x7f0333b01cf9] 2024-07-26T06:59:13.657 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0333bde5a1] 2024-07-26T06:59:13.657 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f21ab1e4407] 2024-07-26T06:59:13.657 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 18: (DispatchQueue::entry()+0x60f) [0x7f21ab1e4cf9] 2024-07-26T06:59:13.657 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f21ab2c15a1] 2024-07-26T06:59:13.657 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 20: (Thread::entry_wrapper()+0x43) [0x7f21ab06ecdd] 2024-07-26T06:59:13.657 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 21: (Thread::_entry_func(void*)+0xd) [0x7f21ab06ecf9] 2024-07-26T06:59:13.657 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 22: /lib64/libc.so.6(+0x89c02) [0x7f21aa962c02] 2024-07-26T06:59:13.657 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f21aa9e7c40] 2024-07-26T06:59:13.657 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:59:13.658 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 2024-07-26T06:59:13.658 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 0> 2024-07-26T06:59:13.348+0000 7f21a5540640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:59:13.658 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: in thread 7f21a5540640 thread_name:ms_dispatch 2024-07-26T06:59:13.658 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 2024-07-26T06:59:13.658 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:59:13.658 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f21aa9176f0] 2024-07-26T06:59:13.658 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f21aa96494c] 2024-07-26T06:59:13.658 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 3: raise() 2024-07-26T06:59:13.658 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 4: abort() 2024-07-26T06:59:13.658 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 5: /lib64/libc.so.6(+0x2871b) [0x7f21aa90171b] 2024-07-26T06:59:13.658 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 6: /lib64/libc.so.6(+0x37386) [0x7f21aa910386] 2024-07-26T06:59:13.658 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56144e4326b9] 2024-07-26T06:59:13.659 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56144e5f080c] 2024-07-26T06:59:13.659 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56144e5302ca] 2024-07-26T06:59:13.659 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56144e500753] 2024-07-26T06:59:13.659 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56144e501a1b] 2024-07-26T06:59:13.659 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56144e4e5752] 2024-07-26T06:59:13.659 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56144e433a56] 2024-07-26T06:59:13.659 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56144e437637] 2024-07-26T06:59:13.659 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 20: (Thread::entry_wrapper()+0x43) [0x7f033398bcdd] 2024-07-26T06:59:13.659 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 21: (Thread::_entry_func(void*)+0xd) [0x7f033398bcf9] 2024-07-26T06:59:13.660 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 22: /lib64/libc.so.6(+0x89c02) [0x7f033327fc02] 2024-07-26T06:59:13.660 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0333304c40] 2024-07-26T06:59:13.660 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:59:13.660 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 2024-07-26T06:59:13.660 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 0> 2024-07-26T06:59:13.346+0000 7f032de5d640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:59:13.660 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: in thread 7f032de5d640 thread_name:ms_dispatch 2024-07-26T06:59:13.660 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 2024-07-26T06:59:13.660 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:59:13.660 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f03332346f0] 2024-07-26T06:59:13.660 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f033328194c] 2024-07-26T06:59:13.660 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 3: raise() 2024-07-26T06:59:13.660 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 4: abort() 2024-07-26T06:59:13.661 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 5: /lib64/libc.so.6(+0x2871b) [0x7f033321e71b] 2024-07-26T06:59:13.661 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 6: /lib64/libc.so.6(+0x37386) [0x7f033322d386] 2024-07-26T06:59:13.661 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5654c25f76b9] 2024-07-26T06:59:13.661 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5654c27b580c] 2024-07-26T06:59:13.661 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5654c26f52ca] 2024-07-26T06:59:13.661 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5654c26c5753] 2024-07-26T06:59:13.661 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5654c26c6a1b] 2024-07-26T06:59:13.661 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5654c26aa752] 2024-07-26T06:59:13.661 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5654c25f8a56] 2024-07-26T06:59:13.661 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5654c25fc637] 2024-07-26T06:59:13.661 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5654c25fcf5e] 2024-07-26T06:59:13.661 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5654c25e003a] 2024-07-26T06:59:13.661 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0333b01407] 2024-07-26T06:59:13.662 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56144e437f5e] 2024-07-26T06:59:13.662 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56144e41b03a] 2024-07-26T06:59:13.662 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f21ab1e4407] 2024-07-26T06:59:13.662 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 18: (DispatchQueue::entry()+0x60f) [0x7f21ab1e4cf9] 2024-07-26T06:59:13.662 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f21ab2c15a1] 2024-07-26T06:59:13.662 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 20: (Thread::entry_wrapper()+0x43) [0x7f21ab06ecdd] 2024-07-26T06:59:13.662 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 21: (Thread::_entry_func(void*)+0xd) [0x7f21ab06ecf9] 2024-07-26T06:59:13.662 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 22: /lib64/libc.so.6(+0x89c02) [0x7f21aa962c02] 2024-07-26T06:59:13.662 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f21aa9e7c40] 2024-07-26T06:59:13.663 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:59:13.663 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g[120201]: 2024-07-26T06:59:13.663 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:13 smithi042 podman[120402]: 2024-07-26 06:59:13.423990354 +0000 UTC m=+0.023047143 container died 76596ac32e19bb6708b6bc51fa825d8a3221be79da9cbb466af37f1121a1adeb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-07-26T06:59:13.663 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 18: (DispatchQueue::entry()+0x60f) [0x7f0333b01cf9] 2024-07-26T06:59:13.663 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0333bde5a1] 2024-07-26T06:59:13.663 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 20: (Thread::entry_wrapper()+0x43) [0x7f033398bcdd] 2024-07-26T06:59:13.663 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 21: (Thread::_entry_func(void*)+0xd) [0x7f033398bcf9] 2024-07-26T06:59:13.664 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 22: /lib64/libc.so.6(+0x89c02) [0x7f033327fc02] 2024-07-26T06:59:13.664 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0333304c40] 2024-07-26T06:59:13.664 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:59:13.664 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 2024-07-26T06:59:13.664 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 0> 2024-07-26T06:59:13.346+0000 7f032de5d640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:59:13.664 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: in thread 7f032de5d640 thread_name:ms_dispatch 2024-07-26T06:59:13.664 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 2024-07-26T06:59:13.664 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:59:13.664 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f03332346f0] 2024-07-26T06:59:13.664 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f033328194c] 2024-07-26T06:59:13.664 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 3: raise() 2024-07-26T06:59:13.664 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 4: abort() 2024-07-26T06:59:13.665 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 5: /lib64/libc.so.6(+0x2871b) [0x7f033321e71b] 2024-07-26T06:59:13.665 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 6: /lib64/libc.so.6(+0x37386) [0x7f033322d386] 2024-07-26T06:59:13.665 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5654c25f76b9] 2024-07-26T06:59:13.665 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5654c27b580c] 2024-07-26T06:59:13.665 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5654c26f52ca] 2024-07-26T06:59:13.665 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5654c26c5753] 2024-07-26T06:59:13.665 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5654c26c6a1b] 2024-07-26T06:59:13.665 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5654c26aa752] 2024-07-26T06:59:13.665 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5654c25f8a56] 2024-07-26T06:59:13.665 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5654c25fc637] 2024-07-26T06:59:13.665 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5654c25fcf5e] 2024-07-26T06:59:13.665 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5654c25e003a] 2024-07-26T06:59:13.665 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0333b01407] 2024-07-26T06:59:13.666 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 18: (DispatchQueue::entry()+0x60f) [0x7f0333b01cf9] 2024-07-26T06:59:13.666 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0333bde5a1] 2024-07-26T06:59:13.666 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 20: (Thread::entry_wrapper()+0x43) [0x7f033398bcdd] 2024-07-26T06:59:13.666 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 21: (Thread::_entry_func(void*)+0xd) [0x7f033398bcf9] 2024-07-26T06:59:13.666 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 22: /lib64/libc.so.6(+0x89c02) [0x7f033327fc02] 2024-07-26T06:59:13.666 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0333304c40] 2024-07-26T06:59:13.666 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:59:13.666 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[75789]: 2024-07-26T06:59:13.667 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T06:59:13.667 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: *** Caught signal (Aborted) ** 2024-07-26T06:59:13.667 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: in thread 7f343d7e5640 thread_name:ms_dispatch 2024-07-26T06:59:13.667 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:59:13.667 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3442bbc6f0] 2024-07-26T06:59:13.667 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3442c0994c] 2024-07-26T06:59:13.667 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 3: raise() 2024-07-26T06:59:13.668 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 4: abort() 2024-07-26T06:59:13.668 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3442ba671b] 2024-07-26T06:59:13.668 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 6: /lib64/libc.so.6(+0x37386) [0x7f3442bb5386] 2024-07-26T06:59:13.668 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5568787ee6b9] 2024-07-26T06:59:13.668 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5568789ac80c] 2024-07-26T06:59:13.668 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5568788ec2ca] 2024-07-26T06:59:13.668 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5568788bc753] 2024-07-26T06:59:13.668 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5568788bda1b] 2024-07-26T06:59:13.668 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5568788a1752] 2024-07-26T06:59:13.668 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5568787efa56] 2024-07-26T06:59:13.668 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5568787f3637] 2024-07-26T06:59:13.668 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5568787f3f5e] 2024-07-26T06:59:13.668 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5568787d703a] 2024-07-26T06:59:13.669 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3443489407] 2024-07-26T06:59:13.669 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 18: (DispatchQueue::entry()+0x60f) [0x7f3443489cf9] 2024-07-26T06:59:13.669 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f34435665a1] 2024-07-26T06:59:13.669 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 20: (Thread::entry_wrapper()+0x43) [0x7f3443313cdd] 2024-07-26T06:59:13.669 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3443313cf9] 2024-07-26T06:59:13.669 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3442c07c02] 2024-07-26T06:59:13.669 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3442c8cc40] 2024-07-26T06:59:13.669 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 2024-07-26T06:59:13.348+0000 7f343d7e5640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:59:13.669 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: in thread 7f343d7e5640 thread_name:ms_dispatch 2024-07-26T06:59:13.669 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 2024-07-26T06:59:13.669 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:59:13.669 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3442bbc6f0] 2024-07-26T06:59:13.669 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3442c0994c] 2024-07-26T06:59:13.670 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 3: raise() 2024-07-26T06:59:13.670 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 4: abort() 2024-07-26T06:59:13.670 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3442ba671b] 2024-07-26T06:59:13.670 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 6: /lib64/libc.so.6(+0x37386) [0x7f3442bb5386] 2024-07-26T06:59:13.670 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5568787ee6b9] 2024-07-26T06:59:13.670 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5568789ac80c] 2024-07-26T06:59:13.670 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5568788ec2ca] 2024-07-26T06:59:13.670 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5568788bc753] 2024-07-26T06:59:13.670 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5568788bda1b] 2024-07-26T06:59:13.670 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5568788a1752] 2024-07-26T06:59:13.670 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5568787efa56] 2024-07-26T06:59:13.670 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5568787f3637] 2024-07-26T06:59:13.670 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5568787f3f5e] 2024-07-26T06:59:13.670 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5568787d703a] 2024-07-26T06:59:13.671 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3443489407] 2024-07-26T06:59:13.671 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 18: (DispatchQueue::entry()+0x60f) [0x7f3443489cf9] 2024-07-26T06:59:13.671 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f34435665a1] 2024-07-26T06:59:13.671 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 20: (Thread::entry_wrapper()+0x43) [0x7f3443313cdd] 2024-07-26T06:59:13.671 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3443313cf9] 2024-07-26T06:59:13.671 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3442c07c02] 2024-07-26T06:59:13.671 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3442c8cc40] 2024-07-26T06:59:13.671 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:59:13.671 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 2024-07-26T06:59:13.671 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 0> 2024-07-26T06:59:13.348+0000 7f343d7e5640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:59:13.671 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: in thread 7f343d7e5640 thread_name:ms_dispatch 2024-07-26T06:59:13.671 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 2024-07-26T06:59:13.671 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:59:13.671 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3442bbc6f0] 2024-07-26T06:59:13.672 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3442c0994c] 2024-07-26T06:59:13.672 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 3: raise() 2024-07-26T06:59:13.672 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 4: abort() 2024-07-26T06:59:13.672 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3442ba671b] 2024-07-26T06:59:13.672 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 6: /lib64/libc.so.6(+0x37386) [0x7f3442bb5386] 2024-07-26T06:59:13.672 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5568787ee6b9] 2024-07-26T06:59:13.672 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5568789ac80c] 2024-07-26T06:59:13.672 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5568788ec2ca] 2024-07-26T06:59:13.672 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5568788bc753] 2024-07-26T06:59:13.672 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5568788bda1b] 2024-07-26T06:59:13.672 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5568788a1752] 2024-07-26T06:59:13.672 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5568787efa56] 2024-07-26T06:59:13.673 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5568787f3637] 2024-07-26T06:59:13.673 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5568787f3f5e] 2024-07-26T06:59:13.673 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5568787d703a] 2024-07-26T06:59:13.673 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3443489407] 2024-07-26T06:59:13.673 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 18: (DispatchQueue::entry()+0x60f) [0x7f3443489cf9] 2024-07-26T06:59:13.673 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f34435665a1] 2024-07-26T06:59:13.673 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 20: (Thread::entry_wrapper()+0x43) [0x7f3443313cdd] 2024-07-26T06:59:13.673 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3443313cf9] 2024-07-26T06:59:13.673 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3442c07c02] 2024-07-26T06:59:13.673 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3442c8cc40] 2024-07-26T06:59:13.674 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:59:13.674 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 2024-07-26T06:59:13.674 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 0> 2024-07-26T06:59:13.348+0000 7f343d7e5640 -1 *** Caught signal (Aborted) ** 2024-07-26T06:59:13.674 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: in thread 7f343d7e5640 thread_name:ms_dispatch 2024-07-26T06:59:13.674 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 2024-07-26T06:59:13.674 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T06:59:13.674 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3442bbc6f0] 2024-07-26T06:59:13.674 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3442c0994c] 2024-07-26T06:59:13.674 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 3: raise() 2024-07-26T06:59:13.674 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 4: abort() 2024-07-26T06:59:13.674 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3442ba671b] 2024-07-26T06:59:13.674 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 6: /lib64/libc.so.6(+0x37386) [0x7f3442bb5386] 2024-07-26T06:59:13.674 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5568787ee6b9] 2024-07-26T06:59:13.675 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5568789ac80c] 2024-07-26T06:59:13.675 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5568788ec2ca] 2024-07-26T06:59:13.675 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5568788bc753] 2024-07-26T06:59:13.675 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5568788bda1b] 2024-07-26T06:59:13.675 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5568788a1752] 2024-07-26T06:59:13.675 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5568787efa56] 2024-07-26T06:59:13.675 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5568787f3637] 2024-07-26T06:59:13.675 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5568787f3f5e] 2024-07-26T06:59:13.675 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5568787d703a] 2024-07-26T06:59:13.675 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3443489407] 2024-07-26T06:59:13.675 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 18: (DispatchQueue::entry()+0x60f) [0x7f3443489cf9] 2024-07-26T06:59:13.675 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f34435665a1] 2024-07-26T06:59:13.676 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 20: (Thread::entry_wrapper()+0x43) [0x7f3443313cdd] 2024-07-26T06:59:13.676 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3443313cf9] 2024-07-26T06:59:13.676 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3442c07c02] 2024-07-26T06:59:13.676 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3442c8cc40] 2024-07-26T06:59:13.676 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T06:59:13.676 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l[71405]: 2024-07-26T06:59:13.676 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:13 smithi164 podman[71443]: 2024-07-26 06:59:13.406540557 +0000 UTC m=+0.020014691 container died d5d7fddc93186985800143b64102c14a94c095a4aef748110da1ebea5242e092 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-07-26T06:59:13.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-mon[36970]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:59:13.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6834/3548166563,v1:172.21.15.42:6835/3548166563] up:active 2024-07-26T06:59:13.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6832/570239393,v1:172.21.15.164:6833/570239393] up:active 2024-07-26T06:59:13.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6838/3851243510,v1:172.21.15.42:6839/3851243510] up:active 2024-07-26T06:59:13.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-mon[36970]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-07-26T06:59:13.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:13 smithi164 ceph-mon[36970]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-07-26T06:59:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:13 smithi163 ceph-mon[36873]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:59:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:13 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6834/3548166563,v1:172.21.15.42:6835/3548166563] up:active 2024-07-26T06:59:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:13 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6832/570239393,v1:172.21.15.164:6833/570239393] up:active 2024-07-26T06:59:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:13 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6838/3851243510,v1:172.21.15.42:6839/3851243510] up:active 2024-07-26T06:59:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:13 smithi163 ceph-mon[36873]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-07-26T06:59:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:13 smithi163 ceph-mon[36873]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-07-26T06:59:14.031 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-mon[32775]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T06:59:14.032 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6834/3548166563,v1:172.21.15.42:6835/3548166563] up:active 2024-07-26T06:59:14.032 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6832/570239393,v1:172.21.15.164:6833/570239393] up:active 2024-07-26T06:59:14.032 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6838/3851243510,v1:172.21.15.42:6839/3851243510] up:active 2024-07-26T06:59:14.032 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-mon[32775]: fsmap cephfs:5 2 up:standby-replay 1 up:standby 5 up:active 2024-07-26T06:59:14.032 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:13 smithi042 ceph-mon[32775]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-07-26T06:59:14.474 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:14 smithi164 podman[71443]: 2024-07-26 06:59:14.094005627 +0000 UTC m=+0.707479751 container remove d5d7fddc93186985800143b64102c14a94c095a4aef748110da1ebea5242e092 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-l, io.buildah.version=1.36.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T06:59:14.474 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:14 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:59:14.474 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:14 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-07-26T06:59:14.570 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:14 smithi042 podman[120402]: 2024-07-26 06:59:14.199749878 +0000 UTC m=+0.798806650 container remove 76596ac32e19bb6708b6bc51fa825d8a3221be79da9cbb466af37f1121a1adeb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-07-26T06:59:14.570 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:14 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:59:14.570 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:14 smithi042 podman[120396]: 2024-07-26 06:59:14.256645198 +0000 UTC m=+0.861966932 container died 68440941ad331dda7900f6e75f9ac7502b50e8b1dc84d6ff3e1771b98b526003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2) 2024-07-26T06:59:14.888 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:14 smithi042 ceph-mon[32775]: pgmap v700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 122 B/s wr, 9 op/s 2024-07-26T06:59:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:14 smithi164 ceph-mon[36970]: pgmap v700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 122 B/s wr, 9 op/s 2024-07-26T06:59:15.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:14 smithi163 ceph-mon[36873]: pgmap v700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 122 B/s wr, 9 op/s 2024-07-26T06:59:15.145 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:15 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-07-26T06:59:15.145 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:15 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Consumed 1.023s CPU time. 2024-07-26T06:59:15.145 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:14 smithi042 podman[120396]: 2024-07-26 06:59:14.888126241 +0000 UTC m=+1.493447970 container remove 68440941ad331dda7900f6e75f9ac7502b50e8b1dc84d6ff3e1771b98b526003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True) 2024-07-26T06:59:15.145 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:14 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-07-26T06:59:15.644 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:15 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-07-26T06:59:15.645 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:15 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.j.service: Consumed 1.154s CPU time. 2024-07-26T06:59:15.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:15 smithi164 ceph-mon[36970]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T06:59:15.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:15 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:active 2024-07-26T06:59:15.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:15 smithi164 ceph-mon[36970]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-07-26T06:59:15.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:59:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:15 smithi163 ceph-mon[36873]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T06:59:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:15 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:active 2024-07-26T06:59:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:15 smithi163 ceph-mon[36873]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-07-26T06:59:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:59:16.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:15 smithi042 ceph-mon[32775]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T06:59:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:15 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.164:6834/1072602530,v1:172.21.15.164:6835/1072602530] up:active 2024-07-26T06:59:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:15 smithi042 ceph-mon[32775]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-07-26T06:59:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T06:59:16.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:16 smithi042 ceph-mon[32775]: pgmap v701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 409 B/s wr, 10 op/s 2024-07-26T06:59:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 186.108154 secs 2024-07-26T06:59:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 203.850466 secs 2024-07-26T06:59:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:16 smithi164 ceph-mon[36970]: pgmap v701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 409 B/s wr, 10 op/s 2024-07-26T06:59:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 186.108154 secs 2024-07-26T06:59:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 203.850466 secs 2024-07-26T06:59:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:16 smithi163 ceph-mon[36873]: pgmap v701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 409 B/s wr, 10 op/s 2024-07-26T06:59:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 186.108154 secs 2024-07-26T06:59:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 203.850466 secs 2024-07-26T06:59:18.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:59:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:59:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:59:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:59:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T06:59:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T06:59:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T06:59:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:19 smithi164 ceph-mon[36970]: pgmap v702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 614 B/s wr, 10 op/s 2024-07-26T06:59:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:19 smithi164 ceph-mon[36970]: pgmap v703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 703 B/s wr, 8 op/s 2024-07-26T06:59:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:19 smithi164 ceph-mon[36970]: Health check failed: 7 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T06:59:19.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:19 smithi163 ceph-mon[36873]: pgmap v702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 614 B/s wr, 10 op/s 2024-07-26T06:59:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:19 smithi163 ceph-mon[36873]: pgmap v703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 703 B/s wr, 8 op/s 2024-07-26T06:59:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:19 smithi163 ceph-mon[36873]: Health check failed: 7 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T06:59:19.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:19 smithi042 ceph-mon[32775]: pgmap v702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 614 B/s wr, 10 op/s 2024-07-26T06:59:19.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:19 smithi042 ceph-mon[32775]: pgmap v703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 703 B/s wr, 8 op/s 2024-07-26T06:59:19.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:19 smithi042 ceph-mon[32775]: Health check failed: 7 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T06:59:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:21 smithi164 ceph-mon[36970]: pgmap v704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 583 B/s wr, 8 op/s 2024-07-26T06:59:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 191.108294 secs 2024-07-26T06:59:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 208.850610 secs 2024-07-26T06:59:21.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:21 smithi163 ceph-mon[36873]: pgmap v704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 583 B/s wr, 8 op/s 2024-07-26T06:59:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 191.108294 secs 2024-07-26T06:59:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 208.850610 secs 2024-07-26T06:59:21.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:21 smithi042 ceph-mon[32775]: pgmap v704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 583 B/s wr, 8 op/s 2024-07-26T06:59:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 191.108294 secs 2024-07-26T06:59:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 208.850610 secs 2024-07-26T06:59:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:59:23.345+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:59:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:59:23.345+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:59:23.345 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:59:23.345+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:59:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:23 smithi164 ceph-mon[36970]: pgmap v705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 583 B/s wr, 8 op/s 2024-07-26T06:59:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:23 smithi163 ceph-mon[36873]: pgmap v705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 583 B/s wr, 8 op/s 2024-07-26T06:59:23.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:23 smithi042 ceph-mon[32775]: pgmap v705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 583 B/s wr, 8 op/s 2024-07-26T06:59:24.972 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:24 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 6. 2024-07-26T06:59:24.972 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:24 smithi164 systemd[1]: Stopped Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:59:24.972 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:24 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Start request repeated too quickly. 2024-07-26T06:59:24.972 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:24 smithi164 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-07-26T06:59:24.972 INFO:journalctl@ceph.mds.l.smithi164.stdout:Jul 26 06:59:24 smithi164 systemd[1]: Failed to start Ceph mds.l for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:59:25.173 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:25 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 6. 2024-07-26T06:59:25.173 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:25 smithi042 systemd[1]: Stopped Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:59:25.173 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:25 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Consumed 1.023s CPU time. 2024-07-26T06:59:25.173 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:25 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Start request repeated too quickly. 2024-07-26T06:59:25.174 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:25 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-07-26T06:59:25.174 INFO:journalctl@ceph.mds.g.smithi042.stdout:Jul 26 06:59:25 smithi042 systemd[1]: Failed to start Ceph mds.g for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:59:25.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:25 smithi164 ceph-mon[36970]: pgmap v706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 486 B/s wr, 9 op/s 2024-07-26T06:59:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:25 smithi163 ceph-mon[36873]: pgmap v706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 486 B/s wr, 9 op/s 2024-07-26T06:59:25.633 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:25 smithi042 ceph-mon[32775]: pgmap v706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 486 B/s wr, 9 op/s 2024-07-26T06:59:25.634 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:25 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 1. 2024-07-26T06:59:25.634 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:25 smithi042 systemd[1]: Stopped Ceph mds.j for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:59:25.634 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:25 smithi042 systemd[1]: ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968@mds.j.service: Consumed 1.154s CPU time. 2024-07-26T06:59:25.634 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:25 smithi042 systemd[1]: Starting Ceph mds.j for 8b3e2750-4b19-11ef-bcb6-c7b262605968... 2024-07-26T06:59:25.894 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:25 smithi042 podman[121052]: 2024-07-26 06:59:25.536276002 +0000 UTC m=+0.018449479 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T06:59:25.894 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:25 smithi042 podman[121052]: 2024-07-26 06:59:25.661522446 +0000 UTC m=+0.143695908 container create f390ed009c1c43ac636de1ea0c603670b67852b9a5a427d33ac00de3e70e692d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-07-26T06:59:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 196.108385 secs 2024-07-26T06:59:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:26 smithi042 ceph-mon[32775]: daemon mds.j restarted 2024-07-26T06:59:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:26 smithi042 ceph-mon[32775]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:59:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 213.850750 secs 2024-07-26T06:59:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:26 smithi042 ceph-mon[32775]: osdmap e115: 12 total, 12 up, 12 in 2024-07-26T06:59:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:26 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6834/3615723129,v1:172.21.15.42:6835/3615723129] up:boot 2024-07-26T06:59:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:26 smithi042 ceph-mon[32775]: Standby daemon mds.j assigned to filesystem cephfs as rank 4 2024-07-26T06:59:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:26 smithi042 ceph-mon[32775]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=l=up:active,3=g=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-07-26T06:59:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T06:59:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:26 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2024-07-26T06:59:26.396 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:25 smithi042 podman[121052]: 2024-07-26 06:59:25.948305647 +0000 UTC m=+0.430479123 container init f390ed009c1c43ac636de1ea0c603670b67852b9a5a427d33ac00de3e70e692d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-07-26T06:59:26.396 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:25 smithi042 podman[121052]: 2024-07-26 06:59:25.951545692 +0000 UTC m=+0.433719155 container start f390ed009c1c43ac636de1ea0c603670b67852b9a5a427d33ac00de3e70e692d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T06:59:26.396 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:25 smithi042 ceph-mds[121071]: -- 172.21.15.42:0/3474241145 <== mon.0 v2:172.21.15.42:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5626c93e9860 con 0x5626cd58ba80 2024-07-26T06:59:26.396 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:25 smithi042 ceph-8b3e2750-4b19-11ef-bcb6-c7b262605968-mds-j[121067]: starting mds.j at 2024-07-26T06:59:26.396 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:26 smithi042 bash[121052]: f390ed009c1c43ac636de1ea0c603670b67852b9a5a427d33ac00de3e70e692d 2024-07-26T06:59:26.396 INFO:journalctl@ceph.mds.j.smithi042.stdout:Jul 26 06:59:26 smithi042 systemd[1]: Started Ceph mds.j for 8b3e2750-4b19-11ef-bcb6-c7b262605968. 2024-07-26T06:59:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 196.108385 secs 2024-07-26T06:59:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:26 smithi164 ceph-mon[36970]: daemon mds.j restarted 2024-07-26T06:59:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:26 smithi164 ceph-mon[36970]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:59:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 213.850750 secs 2024-07-26T06:59:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:26 smithi164 ceph-mon[36970]: osdmap e115: 12 total, 12 up, 12 in 2024-07-26T06:59:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:26 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6834/3615723129,v1:172.21.15.42:6835/3615723129] up:boot 2024-07-26T06:59:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:26 smithi164 ceph-mon[36970]: Standby daemon mds.j assigned to filesystem cephfs as rank 4 2024-07-26T06:59:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:26 smithi164 ceph-mon[36970]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=l=up:active,3=g=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-07-26T06:59:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T06:59:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:26 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2024-07-26T06:59:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 196.108385 secs 2024-07-26T06:59:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:26 smithi163 ceph-mon[36873]: daemon mds.j restarted 2024-07-26T06:59:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:26 smithi163 ceph-mon[36873]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T06:59:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 213.850750 secs 2024-07-26T06:59:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:26 smithi163 ceph-mon[36873]: osdmap e115: 12 total, 12 up, 12 in 2024-07-26T06:59:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:26 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6834/3615723129,v1:172.21.15.42:6835/3615723129] up:boot 2024-07-26T06:59:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:26 smithi163 ceph-mon[36873]: Standby daemon mds.j assigned to filesystem cephfs as rank 4 2024-07-26T06:59:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:26 smithi163 ceph-mon[36873]: fsmap cephfs:4/5 {0=f=up:active,1=i=up:active,2=l=up:active,3=g=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-07-26T06:59:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T06:59:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:26 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2024-07-26T06:59:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:27 smithi164 ceph-mon[36970]: pgmap v708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 6 op/s 2024-07-26T06:59:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:27 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6834/3615723129,v1:172.21.15.42:6835/3615723129] up:resolve 2024-07-26T06:59:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:27 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 3 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T06:59:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:27 smithi163 ceph-mon[36873]: pgmap v708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 6 op/s 2024-07-26T06:59:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:27 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6834/3615723129,v1:172.21.15.42:6835/3615723129] up:resolve 2024-07-26T06:59:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:27 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 3 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T06:59:27.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:27 smithi042 ceph-mon[32775]: pgmap v708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 6 op/s 2024-07-26T06:59:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:27 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6834/3615723129,v1:172.21.15.42:6835/3615723129] up:resolve 2024-07-26T06:59:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:27 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 3 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T06:59:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:29 smithi164 ceph-mon[36970]: pgmap v709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 6 op/s 2024-07-26T06:59:29.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:29 smithi163 ceph-mon[36873]: pgmap v709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 6 op/s 2024-07-26T06:59:29.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:29 smithi042 ceph-mon[32775]: pgmap v709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 6 op/s 2024-07-26T06:59:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:30 smithi164 ceph-mon[36970]: Replacing daemon mds.l as rank 2 with standby daemon mds.d 2024-07-26T06:59:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:30 smithi164 ceph-mon[36970]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-07-26T06:59:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:30 smithi164 ceph-mon[36970]: osdmap e116: 12 total, 12 up, 12 in 2024-07-26T06:59:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:30 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:59:30.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:30 smithi163 ceph-mon[36873]: Replacing daemon mds.l as rank 2 with standby daemon mds.d 2024-07-26T06:59:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:30 smithi163 ceph-mon[36873]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-07-26T06:59:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:30 smithi163 ceph-mon[36873]: osdmap e116: 12 total, 12 up, 12 in 2024-07-26T06:59:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:30 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:59:30.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:30 smithi042 ceph-mon[32775]: Replacing daemon mds.l as rank 2 with standby daemon mds.d 2024-07-26T06:59:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:30 smithi042 ceph-mon[32775]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-07-26T06:59:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:30 smithi042 ceph-mon[32775]: osdmap e116: 12 total, 12 up, 12 in 2024-07-26T06:59:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:30 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 1 up:replay 1 up:resolve 3 up:active 2024-07-26T06:59:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:31 smithi164 ceph-mon[36970]: pgmap v711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 7 op/s 2024-07-26T06:59:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:31 smithi164 ceph-mon[36970]: mds.? [v2:172.21.15.42:6836/74811888,v1:172.21.15.42:6837/74811888] up:resolve 2024-07-26T06:59:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:31 smithi164 ceph-mon[36970]: fsmap cephfs:5/5 2 up:standby-replay 2 up:resolve 3 up:active 2024-07-26T06:59:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 201.108467 secs 2024-07-26T06:59:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 218.850872 secs 2024-07-26T06:59:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:31 smithi163 ceph-mon[36873]: pgmap v711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 7 op/s 2024-07-26T06:59:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:31 smithi163 ceph-mon[36873]: mds.? [v2:172.21.15.42:6836/74811888,v1:172.21.15.42:6837/74811888] up:resolve 2024-07-26T06:59:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:31 smithi163 ceph-mon[36873]: fsmap cephfs:5/5 2 up:standby-replay 2 up:resolve 3 up:active 2024-07-26T06:59:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 201.108467 secs 2024-07-26T06:59:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 218.850872 secs 2024-07-26T06:59:31.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:31 smithi042 ceph-mon[32775]: pgmap v711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 7 op/s 2024-07-26T06:59:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:31 smithi042 ceph-mon[32775]: mds.? [v2:172.21.15.42:6836/74811888,v1:172.21.15.42:6837/74811888] up:resolve 2024-07-26T06:59:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:31 smithi042 ceph-mon[32775]: fsmap cephfs:5/5 2 up:standby-replay 2 up:resolve 3 up:active 2024-07-26T06:59:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 201.108467 secs 2024-07-26T06:59:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 218.850872 secs 2024-07-26T06:59:33.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:33 smithi164 ceph-mon[36970]: pgmap v712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 6 op/s 2024-07-26T06:59:33.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:33 smithi163 ceph-mon[36873]: pgmap v712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 6 op/s 2024-07-26T06:59:33.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:33 smithi042 ceph-mon[32775]: pgmap v712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 6 op/s 2024-07-26T06:59:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:34 smithi163 ceph-mon[36873]: pgmap v713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 9 op/s 2024-07-26T06:59:34.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:34 smithi042 ceph-mon[32775]: pgmap v713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 9 op/s 2024-07-26T06:59:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:34 smithi164 ceph-mon[36970]: pgmap v713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 9 op/s 2024-07-26T06:59:36.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:35 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 206.108595 secs 2024-07-26T06:59:36.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:35 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 206.108595 secs 2024-07-26T06:59:36.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:35 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 206.108595 secs 2024-07-26T06:59:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 223.851004 secs 2024-07-26T06:59:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:36 smithi163 ceph-mon[36873]: pgmap v714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 7 op/s 2024-07-26T06:59:37.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 223.851004 secs 2024-07-26T06:59:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:36 smithi042 ceph-mon[32775]: pgmap v714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 7 op/s 2024-07-26T06:59:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 223.851004 secs 2024-07-26T06:59:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:36 smithi164 ceph-mon[36970]: pgmap v714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 7 op/s 2024-07-26T06:59:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:59:38.348+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:59:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:59:38.348+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:59:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:59:38.348+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:59:39.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:39 smithi042 ceph-mon[32775]: pgmap v715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 6 op/s 2024-07-26T06:59:39.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:39 smithi164 ceph-mon[36970]: pgmap v715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 6 op/s 2024-07-26T06:59:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:39 smithi163 ceph-mon[36873]: pgmap v715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 6 op/s 2024-07-26T06:59:41.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:41 smithi042 ceph-mon[32775]: pgmap v716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-07-26T06:59:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 211.108696 secs 2024-07-26T06:59:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 228.851128 secs 2024-07-26T06:59:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:41 smithi164 ceph-mon[36970]: pgmap v716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-07-26T06:59:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 211.108696 secs 2024-07-26T06:59:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 228.851128 secs 2024-07-26T06:59:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:41 smithi163 ceph-mon[36873]: pgmap v716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-07-26T06:59:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 211.108696 secs 2024-07-26T06:59:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 228.851128 secs 2024-07-26T06:59:43.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:43 smithi042 ceph-mon[32775]: pgmap v717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T06:59:43.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:43 smithi164 ceph-mon[36970]: pgmap v717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T06:59:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:43 smithi163 ceph-mon[36873]: pgmap v717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T06:59:45.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:45 smithi042 ceph-mon[32775]: pgmap v718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-07-26T06:59:45.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:45 smithi164 ceph-mon[36970]: pgmap v718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-07-26T06:59:45.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:45 smithi163 ceph-mon[36873]: pgmap v718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-07-26T06:59:46.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 216.108815 secs 2024-07-26T06:59:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 233.851275 secs 2024-07-26T06:59:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 216.108815 secs 2024-07-26T06:59:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 233.851275 secs 2024-07-26T06:59:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 216.108815 secs 2024-07-26T06:59:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 233.851275 secs 2024-07-26T06:59:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:47 smithi042 ceph-mon[32775]: pgmap v719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T06:59:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:47 smithi164 ceph-mon[36970]: pgmap v719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T06:59:47.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:47 smithi163 ceph-mon[36873]: pgmap v719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T06:59:49.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:49 smithi042 ceph-mon[32775]: pgmap v720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T06:59:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:49 smithi164 ceph-mon[36970]: pgmap v720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T06:59:49.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:49 smithi163 ceph-mon[36873]: pgmap v720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T06:59:51.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:51 smithi042 ceph-mon[32775]: pgmap v721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T06:59:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 221.108946 secs 2024-07-26T06:59:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 238.851396 secs 2024-07-26T06:59:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:51 smithi164 ceph-mon[36970]: pgmap v721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T06:59:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 221.108946 secs 2024-07-26T06:59:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 238.851396 secs 2024-07-26T06:59:51.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:51 smithi163 ceph-mon[36873]: pgmap v721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T06:59:51.511 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 221.108946 secs 2024-07-26T06:59:51.511 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 238.851396 secs 2024-07-26T06:59:53.351 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:59:53.350+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T06:59:53.351 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:59:53.350+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T06:59:53.351 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T06:59:53.350+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T06:59:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:53 smithi164 ceph-mon[36970]: pgmap v722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T06:59:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:53 smithi163 ceph-mon[36873]: pgmap v722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T06:59:53.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:53 smithi042 ceph-mon[32775]: pgmap v722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T06:59:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:55 smithi164 ceph-mon[36970]: pgmap v723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T06:59:55.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:55 smithi163 ceph-mon[36873]: pgmap v723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T06:59:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:55 smithi042 ceph-mon[32775]: pgmap v723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T06:59:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 226.109077 secs 2024-07-26T06:59:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 243.851526 secs 2024-07-26T06:59:56.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 226.109077 secs 2024-07-26T06:59:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 243.851526 secs 2024-07-26T06:59:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 226.109077 secs 2024-07-26T06:59:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 243.851526 secs 2024-07-26T06:59:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:57 smithi164 ceph-mon[36970]: pgmap v724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T06:59:57.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:57 smithi163 ceph-mon[36873]: pgmap v724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T06:59:57.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:57 smithi042 ceph-mon[32775]: pgmap v724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T06:59:59.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 06:59:59 smithi164 ceph-mon[36970]: pgmap v725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T06:59:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 06:59:59 smithi163 ceph-mon[36873]: pgmap v725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T06:59:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 06:59:59 smithi042 ceph-mon[32775]: pgmap v725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:00.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:00 smithi164 ceph-mon[36970]: Health detail: HEALTH_WARN 7 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:00:00.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:00 smithi164 ceph-mon[36970]: [WRN] CEPHADM_FAILED_DAEMON: 7 failed cephadm daemon(s) 2024-07-26T07:00:00.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:00 smithi164 ceph-mon[36970]: daemon mds.j on smithi042 is in unknown state 2024-07-26T07:00:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:00 smithi164 ceph-mon[36970]: daemon mds.g on smithi042 is in unknown state 2024-07-26T07:00:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:00 smithi164 ceph-mon[36970]: daemon mds.a on smithi042 is in error state 2024-07-26T07:00:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:00 smithi164 ceph-mon[36970]: daemon mds.e on smithi163 is in error state 2024-07-26T07:00:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:00 smithi164 ceph-mon[36970]: daemon mds.h on smithi163 is in error state 2024-07-26T07:00:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:00 smithi164 ceph-mon[36970]: daemon mds.b on smithi163 is in error state 2024-07-26T07:00:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:00 smithi164 ceph-mon[36970]: daemon mds.l on smithi164 is in unknown state 2024-07-26T07:00:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:00 smithi164 ceph-mon[36970]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-07-26T07:00:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:00 smithi164 ceph-mon[36970]: fs cephfs is degraded 2024-07-26T07:00:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:00 smithi164 ceph-mon[36970]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-07-26T07:00:00.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:00 smithi164 ceph-mon[36970]: mds.i(mds.1): 1 slow requests are blocked > 180 secs 2024-07-26T07:00:00.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:00 smithi164 ceph-mon[36970]: mds.f(mds.0): 1 slow requests are blocked > 180 secs 2024-07-26T07:00:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:00 smithi163 ceph-mon[36873]: Health detail: HEALTH_WARN 7 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:00:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:00 smithi163 ceph-mon[36873]: [WRN] CEPHADM_FAILED_DAEMON: 7 failed cephadm daemon(s) 2024-07-26T07:00:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:00 smithi163 ceph-mon[36873]: daemon mds.j on smithi042 is in unknown state 2024-07-26T07:00:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:00 smithi163 ceph-mon[36873]: daemon mds.g on smithi042 is in unknown state 2024-07-26T07:00:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:00 smithi163 ceph-mon[36873]: daemon mds.a on smithi042 is in error state 2024-07-26T07:00:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:00 smithi163 ceph-mon[36873]: daemon mds.e on smithi163 is in error state 2024-07-26T07:00:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:00 smithi163 ceph-mon[36873]: daemon mds.h on smithi163 is in error state 2024-07-26T07:00:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:00 smithi163 ceph-mon[36873]: daemon mds.b on smithi163 is in error state 2024-07-26T07:00:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:00 smithi163 ceph-mon[36873]: daemon mds.l on smithi164 is in unknown state 2024-07-26T07:00:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:00 smithi163 ceph-mon[36873]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-07-26T07:00:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:00 smithi163 ceph-mon[36873]: fs cephfs is degraded 2024-07-26T07:00:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:00 smithi163 ceph-mon[36873]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-07-26T07:00:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:00 smithi163 ceph-mon[36873]: mds.i(mds.1): 1 slow requests are blocked > 180 secs 2024-07-26T07:00:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:00 smithi163 ceph-mon[36873]: mds.f(mds.0): 1 slow requests are blocked > 180 secs 2024-07-26T07:00:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:00 smithi042 ceph-mon[32775]: Health detail: HEALTH_WARN 7 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:00:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:00 smithi042 ceph-mon[32775]: [WRN] CEPHADM_FAILED_DAEMON: 7 failed cephadm daemon(s) 2024-07-26T07:00:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:00 smithi042 ceph-mon[32775]: daemon mds.j on smithi042 is in unknown state 2024-07-26T07:00:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:00 smithi042 ceph-mon[32775]: daemon mds.g on smithi042 is in unknown state 2024-07-26T07:00:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:00 smithi042 ceph-mon[32775]: daemon mds.a on smithi042 is in error state 2024-07-26T07:00:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:00 smithi042 ceph-mon[32775]: daemon mds.e on smithi163 is in error state 2024-07-26T07:00:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:00 smithi042 ceph-mon[32775]: daemon mds.h on smithi163 is in error state 2024-07-26T07:00:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:00 smithi042 ceph-mon[32775]: daemon mds.b on smithi163 is in error state 2024-07-26T07:00:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:00 smithi042 ceph-mon[32775]: daemon mds.l on smithi164 is in unknown state 2024-07-26T07:00:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:00 smithi042 ceph-mon[32775]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-07-26T07:00:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:00 smithi042 ceph-mon[32775]: fs cephfs is degraded 2024-07-26T07:00:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:00 smithi042 ceph-mon[32775]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-07-26T07:00:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:00 smithi042 ceph-mon[32775]: mds.i(mds.1): 1 slow requests are blocked > 180 secs 2024-07-26T07:00:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:00 smithi042 ceph-mon[32775]: mds.f(mds.0): 1 slow requests are blocked > 180 secs 2024-07-26T07:00:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:01 smithi042 ceph-mon[32775]: pgmap v726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 231.109155 secs 2024-07-26T07:00:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 248.851647 secs 2024-07-26T07:00:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:01 smithi164 ceph-mon[36970]: pgmap v726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 231.109155 secs 2024-07-26T07:00:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 248.851647 secs 2024-07-26T07:00:01.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:01 smithi163 ceph-mon[36873]: pgmap v726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 231.109155 secs 2024-07-26T07:00:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 248.851647 secs 2024-07-26T07:00:03.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:03 smithi164 ceph-mon[36970]: pgmap v727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:03 smithi163 ceph-mon[36873]: pgmap v727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:03 smithi042 ceph-mon[32775]: pgmap v727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:05.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:05 smithi164 ceph-mon[36970]: pgmap v728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:05.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:05 smithi163 ceph-mon[36873]: pgmap v728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:05.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:05 smithi042 ceph-mon[32775]: pgmap v728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 236.109296 secs 2024-07-26T07:00:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 253.851758 secs 2024-07-26T07:00:06.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 236.109296 secs 2024-07-26T07:00:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 253.851758 secs 2024-07-26T07:00:06.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 236.109296 secs 2024-07-26T07:00:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 253.851758 secs 2024-07-26T07:00:07.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:07 smithi164 ceph-mon[36970]: pgmap v729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:07.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:07 smithi163 ceph-mon[36873]: pgmap v729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:07.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:07 smithi042 ceph-mon[32775]: pgmap v729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:00:08.352+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:00:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:00:08.352+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:00:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:00:08.352+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:00:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:09 smithi164 ceph-mon[36970]: pgmap v730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:09 smithi163 ceph-mon[36873]: pgmap v730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:09.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:09 smithi042 ceph-mon[32775]: pgmap v730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:11 smithi164 ceph-mon[36970]: pgmap v731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 241.109439 secs 2024-07-26T07:00:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 258.851879 secs 2024-07-26T07:00:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:11 smithi163 ceph-mon[36873]: pgmap v731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 241.109439 secs 2024-07-26T07:00:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 258.851879 secs 2024-07-26T07:00:11.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:11 smithi042 ceph-mon[32775]: pgmap v731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 241.109439 secs 2024-07-26T07:00:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 258.851879 secs 2024-07-26T07:00:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:13 smithi164 ceph-mon[36970]: pgmap v732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:13.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:13 smithi163 ceph-mon[36873]: pgmap v732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:13.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:13 smithi042 ceph-mon[32775]: pgmap v732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:15.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:15 smithi164 ceph-mon[36970]: pgmap v733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:15 smithi163 ceph-mon[36873]: pgmap v733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:15.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:15 smithi042 ceph-mon[32775]: pgmap v733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 246.109507 secs 2024-07-26T07:00:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 263.851999 secs 2024-07-26T07:00:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:16 smithi164 ceph-mon[36970]: pgmap v734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:16.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 246.109507 secs 2024-07-26T07:00:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 263.851999 secs 2024-07-26T07:00:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:16 smithi163 ceph-mon[36873]: pgmap v734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 246.109507 secs 2024-07-26T07:00:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 263.851999 secs 2024-07-26T07:00:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:16 smithi042 ceph-mon[32775]: pgmap v734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:19 smithi042 ceph-mon[32775]: pgmap v735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:00:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:00:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:00:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:19 smithi164 ceph-mon[36970]: pgmap v735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:00:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:00:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:00:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:19 smithi163 ceph-mon[36873]: pgmap v735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:00:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:00:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:00:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:00:20.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:20 smithi042 ceph-mon[32775]: pgmap v736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:00:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:20 smithi042 ceph-mon[32775]: pgmap v737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T07:00:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:20 smithi164 ceph-mon[36970]: pgmap v736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:00:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:20 smithi164 ceph-mon[36970]: pgmap v737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T07:00:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:20 smithi163 ceph-mon[36873]: pgmap v736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:00:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:20 smithi163 ceph-mon[36873]: pgmap v737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T07:00:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 251.109639 secs 2024-07-26T07:00:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 268.852123 secs 2024-07-26T07:00:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 251.109639 secs 2024-07-26T07:00:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 268.852123 secs 2024-07-26T07:00:21.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 251.109639 secs 2024-07-26T07:00:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 268.852123 secs 2024-07-26T07:00:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:22 smithi042 ceph-mon[32775]: pgmap v738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:00:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:22 smithi164 ceph-mon[36970]: pgmap v738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:00:22.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:22 smithi163 ceph-mon[36873]: pgmap v738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:00:23.354 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:00:23.353+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:00:23.354 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:00:23.353+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:00:23.354 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:00:23.353+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:00:24.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:24 smithi042 ceph-mon[32775]: pgmap v739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:00:24.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:24 smithi164 ceph-mon[36970]: pgmap v739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:00:24.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:24 smithi163 ceph-mon[36873]: pgmap v739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:00:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:26 smithi164 ceph-mon[36970]: pgmap v740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:00:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 256.109770 secs 2024-07-26T07:00:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 273.852245 secs 2024-07-26T07:00:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:26 smithi163 ceph-mon[36873]: pgmap v740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:00:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 256.109770 secs 2024-07-26T07:00:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 273.852245 secs 2024-07-26T07:00:26.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:26 smithi042 ceph-mon[32775]: pgmap v740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:00:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 256.109770 secs 2024-07-26T07:00:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 273.852245 secs 2024-07-26T07:00:28.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:28 smithi164 ceph-mon[36970]: pgmap v741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:00:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:28 smithi163 ceph-mon[36873]: pgmap v741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:00:28.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:28 smithi042 ceph-mon[32775]: pgmap v741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:00:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:30 smithi164 ceph-mon[36970]: pgmap v742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:00:30.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:30 smithi163 ceph-mon[36873]: pgmap v742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:00:30.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:30 smithi042 ceph-mon[32775]: pgmap v742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:00:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 261.109903 secs 2024-07-26T07:00:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 278.852366 secs 2024-07-26T07:00:31.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 261.109903 secs 2024-07-26T07:00:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 278.852366 secs 2024-07-26T07:00:31.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 261.109903 secs 2024-07-26T07:00:31.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 278.852366 secs 2024-07-26T07:00:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:32 smithi164 ceph-mon[36970]: pgmap v743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:32.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:32 smithi163 ceph-mon[36873]: pgmap v743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:32 smithi042 ceph-mon[32775]: pgmap v743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:34.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:34 smithi164 ceph-mon[36970]: pgmap v744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:34 smithi163 ceph-mon[36873]: pgmap v744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:34 smithi042 ceph-mon[32775]: pgmap v744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:36 smithi164 ceph-mon[36970]: pgmap v745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 266.110029 secs 2024-07-26T07:00:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 283.852432 secs 2024-07-26T07:00:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:36 smithi163 ceph-mon[36873]: pgmap v745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 266.110029 secs 2024-07-26T07:00:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 283.852432 secs 2024-07-26T07:00:36.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:36 smithi042 ceph-mon[32775]: pgmap v745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 266.110029 secs 2024-07-26T07:00:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 283.852432 secs 2024-07-26T07:00:38.360 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:00:38.359+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:00:38.360 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:00:38.359+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:00:38.360 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:00:38.359+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:00:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:38 smithi164 ceph-mon[36970]: pgmap v746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:38 smithi163 ceph-mon[36873]: pgmap v746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:38.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:38 smithi042 ceph-mon[32775]: pgmap v746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:40.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:40 smithi164 ceph-mon[36970]: pgmap v747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:00:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:40 smithi163 ceph-mon[36873]: pgmap v747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:00:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:40 smithi042 ceph-mon[32775]: pgmap v747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:00:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 271.110145 secs 2024-07-26T07:00:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 288.852556 secs 2024-07-26T07:00:41.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 271.110145 secs 2024-07-26T07:00:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 288.852556 secs 2024-07-26T07:00:41.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 271.110145 secs 2024-07-26T07:00:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 288.852556 secs 2024-07-26T07:00:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:42 smithi164 ceph-mon[36970]: pgmap v748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:42.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:42 smithi163 ceph-mon[36873]: pgmap v748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:42.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:42 smithi042 ceph-mon[32775]: pgmap v748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:44.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:44 smithi164 ceph-mon[36970]: pgmap v749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:44.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:44 smithi163 ceph-mon[36873]: pgmap v749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:44 smithi042 ceph-mon[32775]: pgmap v749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:46 smithi164 ceph-mon[36970]: pgmap v750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 276.110271 secs 2024-07-26T07:00:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 293.852717 secs 2024-07-26T07:00:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:46 smithi163 ceph-mon[36873]: pgmap v750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 276.110271 secs 2024-07-26T07:00:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 293.852717 secs 2024-07-26T07:00:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:46 smithi042 ceph-mon[32775]: pgmap v750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 276.110271 secs 2024-07-26T07:00:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 293.852717 secs 2024-07-26T07:00:48.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:48 smithi164 ceph-mon[36970]: pgmap v751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:48 smithi163 ceph-mon[36873]: pgmap v751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:48.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:48 smithi042 ceph-mon[32775]: pgmap v751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:50.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:50 smithi164 ceph-mon[36970]: pgmap v752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:00:50.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:50 smithi163 ceph-mon[36873]: pgmap v752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:00:50.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:50 smithi042 ceph-mon[32775]: pgmap v752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:00:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 281.110398 secs 2024-07-26T07:00:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 298.852836 secs 2024-07-26T07:00:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 281.110398 secs 2024-07-26T07:00:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 298.852836 secs 2024-07-26T07:00:51.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 281.110398 secs 2024-07-26T07:00:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 298.852836 secs 2024-07-26T07:00:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:52 smithi164 ceph-mon[36970]: pgmap v753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:52.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:52 smithi163 ceph-mon[36873]: pgmap v753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:52.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:52 smithi042 ceph-mon[32775]: pgmap v753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:00:53.361+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:00:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:00:53.361+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:00:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:00:53.361+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:00:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:53 smithi164 ceph-mon[36970]: pgmap v754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:53.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:53 smithi163 ceph-mon[36873]: pgmap v754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:53.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:53 smithi042 ceph-mon[32775]: pgmap v754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:55 smithi163 ceph-mon[36873]: pgmap v755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:55 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 286.110512 secs 2024-07-26T07:00:56.348 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:55 smithi042 ceph-mon[32775]: pgmap v755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:56.348 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:55 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 286.110512 secs 2024-07-26T07:00:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:55 smithi164 ceph-mon[36970]: pgmap v755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:00:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:55 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 286.110512 secs 2024-07-26T07:00:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 303.852916 secs 2024-07-26T07:00:57.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 303.852916 secs 2024-07-26T07:00:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 303.852916 secs 2024-07-26T07:00:58.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:00:58 smithi163 ceph-mon[36873]: pgmap v756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:58.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:00:58 smithi042 ceph-mon[32775]: pgmap v756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:00:58.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:00:58 smithi164 ceph-mon[36970]: pgmap v756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:00.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:00 smithi042 ceph-mon[32775]: pgmap v757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:00.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:00 smithi164 ceph-mon[36970]: pgmap v757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:00.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:00 smithi163 ceph-mon[36873]: pgmap v757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:01.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 291.110637 secs 2024-07-26T07:01:01.420 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 291.110637 secs 2024-07-26T07:01:01.482 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 291.110637 secs 2024-07-26T07:01:02.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:02 smithi042 ceph-mon[32775]: pgmap v758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 308.853058 secs 2024-07-26T07:01:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:02 smithi164 ceph-mon[36970]: pgmap v758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 308.853058 secs 2024-07-26T07:01:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:02 smithi163 ceph-mon[36873]: pgmap v758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 308.853058 secs 2024-07-26T07:01:04.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:04 smithi042 ceph-mon[32775]: pgmap v759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:04.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:04 smithi164 ceph-mon[36970]: pgmap v759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:04.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:04 smithi163 ceph-mon[36873]: pgmap v759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:06.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:06 smithi042 ceph-mon[32775]: pgmap v760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 296.110756 secs 2024-07-26T07:01:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:06 smithi164 ceph-mon[36970]: pgmap v760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 296.110756 secs 2024-07-26T07:01:06.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:06 smithi163 ceph-mon[36873]: pgmap v760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 296.110756 secs 2024-07-26T07:01:07.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 313.853178 secs 2024-07-26T07:01:07.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 313.853178 secs 2024-07-26T07:01:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 313.853178 secs 2024-07-26T07:01:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:01:08.368+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:01:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:01:08.368+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:01:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:01:08.368+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:01:08.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:08 smithi042 ceph-mon[32775]: pgmap v761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:08 smithi164 ceph-mon[36970]: pgmap v761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:08 smithi163 ceph-mon[36873]: pgmap v761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:10.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:10 smithi042 ceph-mon[32775]: pgmap v762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:10.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:10 smithi164 ceph-mon[36970]: pgmap v762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:10 smithi163 ceph-mon[36873]: pgmap v762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:11.373 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 301.110884 secs 2024-07-26T07:01:11.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 301.110884 secs 2024-07-26T07:01:11.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 301.110884 secs 2024-07-26T07:01:12.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:12 smithi042 ceph-mon[32775]: pgmap v763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 318.853304 secs 2024-07-26T07:01:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:12 smithi164 ceph-mon[36970]: pgmap v763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 318.853304 secs 2024-07-26T07:01:12.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:12 smithi163 ceph-mon[36873]: pgmap v763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 318.853304 secs 2024-07-26T07:01:14.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:14 smithi042 ceph-mon[32775]: pgmap v764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:14.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:14 smithi164 ceph-mon[36970]: pgmap v764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:14 smithi163 ceph-mon[36873]: pgmap v764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:16.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:16 smithi042 ceph-mon[32775]: pgmap v765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 306.111000 secs 2024-07-26T07:01:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:16 smithi164 ceph-mon[36970]: pgmap v765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 306.111000 secs 2024-07-26T07:01:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:16 smithi163 ceph-mon[36873]: pgmap v765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 306.111000 secs 2024-07-26T07:01:17.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 323.853416 secs 2024-07-26T07:01:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 323.853416 secs 2024-07-26T07:01:17.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 323.853416 secs 2024-07-26T07:01:18.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:18 smithi042 ceph-mon[32775]: pgmap v766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:18.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:18 smithi164 ceph-mon[36970]: pgmap v766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:18.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:18 smithi163 ceph-mon[36873]: pgmap v766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:19.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:01:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:01:19.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:01:20.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:20 smithi042 ceph-mon[32775]: pgmap v767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:01:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:01:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:01:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:20 smithi164 ceph-mon[36970]: pgmap v767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:01:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:01:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:01:20.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:20 smithi163 ceph-mon[36873]: pgmap v767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:01:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:01:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:01:21.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:21 smithi042 ceph-mon[32775]: pgmap v768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:01:21.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:21 smithi042 ceph-mon[32775]: pgmap v769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:01:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 311.111098 secs 2024-07-26T07:01:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:21 smithi164 ceph-mon[36970]: pgmap v768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:01:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:21 smithi164 ceph-mon[36970]: pgmap v769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:01:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 311.111098 secs 2024-07-26T07:01:21.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:21 smithi163 ceph-mon[36873]: pgmap v768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:01:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:21 smithi163 ceph-mon[36873]: pgmap v769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:01:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 311.111098 secs 2024-07-26T07:01:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 328.853561 secs 2024-07-26T07:01:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 328.853561 secs 2024-07-26T07:01:22.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 328.853561 secs 2024-07-26T07:01:23.371 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:01:23.370+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:01:23.371 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:01:23.370+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:01:23.371 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:01:23.370+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:01:23.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:23 smithi042 ceph-mon[32775]: pgmap v770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:01:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:23 smithi164 ceph-mon[36970]: pgmap v770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:01:23.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:23 smithi163 ceph-mon[36873]: pgmap v770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:01:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:25 smithi042 ceph-mon[32775]: pgmap v771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:01:25.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:25 smithi164 ceph-mon[36970]: pgmap v771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:01:25.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:25 smithi163 ceph-mon[36873]: pgmap v771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:01:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 316.111256 secs 2024-07-26T07:01:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 316.111256 secs 2024-07-26T07:01:26.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 316.111256 secs 2024-07-26T07:01:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:27 smithi042 ceph-mon[32775]: pgmap v772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:01:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 333.853686 secs 2024-07-26T07:01:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:27 smithi164 ceph-mon[36970]: pgmap v772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:01:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 333.853686 secs 2024-07-26T07:01:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:27 smithi163 ceph-mon[36873]: pgmap v772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:01:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 333.853686 secs 2024-07-26T07:01:29.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:29 smithi042 ceph-mon[32775]: pgmap v773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:01:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:29 smithi164 ceph-mon[36970]: pgmap v773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:01:29.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:29 smithi163 ceph-mon[36873]: pgmap v773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:01:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:31 smithi042 ceph-mon[32775]: pgmap v774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:01:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 321.111399 secs 2024-07-26T07:01:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:31 smithi164 ceph-mon[36970]: pgmap v774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:01:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 321.111399 secs 2024-07-26T07:01:31.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:31 smithi163 ceph-mon[36873]: pgmap v774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:01:31.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 321.111399 secs 2024-07-26T07:01:32.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 338.853829 secs 2024-07-26T07:01:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 338.853829 secs 2024-07-26T07:01:32.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 338.853829 secs 2024-07-26T07:01:33.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:33 smithi042 ceph-mon[32775]: pgmap v775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:33.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:33 smithi164 ceph-mon[36970]: pgmap v775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:33.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:33 smithi163 ceph-mon[36873]: pgmap v775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:35.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:35 smithi042 ceph-mon[32775]: pgmap v776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:35.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:35 smithi164 ceph-mon[36970]: pgmap v776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:35 smithi163 ceph-mon[36873]: pgmap v776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:36.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 326.111489 secs 2024-07-26T07:01:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 343.853969 secs 2024-07-26T07:01:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 326.111489 secs 2024-07-26T07:01:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 343.853969 secs 2024-07-26T07:01:36.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 326.111489 secs 2024-07-26T07:01:36.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 343.853969 secs 2024-07-26T07:01:37.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:37 smithi042 ceph-mon[32775]: pgmap v777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:37 smithi164 ceph-mon[36970]: pgmap v777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:37.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:37 smithi163 ceph-mon[36873]: pgmap v777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:38.374 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:01:38.374+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:01:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:01:38.374+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:01:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:01:38.374+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:01:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:39 smithi042 ceph-mon[32775]: pgmap v778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:39.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:39 smithi164 ceph-mon[36970]: pgmap v778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:39.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:39 smithi163 ceph-mon[36873]: pgmap v778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:41.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:41 smithi042 ceph-mon[32775]: pgmap v779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:41.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 331.111615 secs 2024-07-26T07:01:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 348.854090 secs 2024-07-26T07:01:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:41 smithi164 ceph-mon[36970]: pgmap v779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 331.111615 secs 2024-07-26T07:01:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 348.854090 secs 2024-07-26T07:01:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:41 smithi163 ceph-mon[36873]: pgmap v779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 331.111615 secs 2024-07-26T07:01:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 348.854090 secs 2024-07-26T07:01:43.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:43 smithi042 ceph-mon[32775]: pgmap v780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:43.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:43 smithi164 ceph-mon[36970]: pgmap v780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:43.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:43 smithi163 ceph-mon[36873]: pgmap v780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:45.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:45 smithi042 ceph-mon[32775]: pgmap v781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:45.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:45 smithi164 ceph-mon[36970]: pgmap v781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:45.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:45 smithi163 ceph-mon[36873]: pgmap v781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:46.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 336.111709 secs 2024-07-26T07:01:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 353.854229 secs 2024-07-26T07:01:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 336.111709 secs 2024-07-26T07:01:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 353.854229 secs 2024-07-26T07:01:46.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 336.111709 secs 2024-07-26T07:01:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 353.854229 secs 2024-07-26T07:01:47.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:47 smithi042 ceph-mon[32775]: pgmap v782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:47 smithi164 ceph-mon[36970]: pgmap v782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:47.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:47 smithi163 ceph-mon[36873]: pgmap v782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:49 smithi042 ceph-mon[32775]: pgmap v783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:49 smithi164 ceph-mon[36970]: pgmap v783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:49.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:49 smithi163 ceph-mon[36873]: pgmap v783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:51.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:51 smithi042 ceph-mon[32775]: pgmap v784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 341.111834 secs 2024-07-26T07:01:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 358.854304 secs 2024-07-26T07:01:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:51 smithi164 ceph-mon[36970]: pgmap v784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 341.111834 secs 2024-07-26T07:01:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 358.854304 secs 2024-07-26T07:01:51.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:51 smithi163 ceph-mon[36873]: pgmap v784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:51.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 341.111834 secs 2024-07-26T07:01:51.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 358.854304 secs 2024-07-26T07:01:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:01:53.377+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:01:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:01:53.377+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:01:53.378 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:01:53.377+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:01:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:53 smithi042 ceph-mon[32775]: pgmap v785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:53 smithi164 ceph-mon[36970]: pgmap v785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:53 smithi163 ceph-mon[36873]: pgmap v785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:55 smithi164 ceph-mon[36970]: pgmap v786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:55.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:55 smithi163 ceph-mon[36873]: pgmap v786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:55.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:55 smithi042 ceph-mon[32775]: pgmap v786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:01:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 346.111953 secs 2024-07-26T07:01:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:56 smithi164 ceph-mon[36970]: 1 slow requests, 1 included below; oldest blocked for > 363.854443 secs 2024-07-26T07:01:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:56 smithi164 ceph-mon[36970]: slow request 363.854443 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T07:01:56.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 346.111953 secs 2024-07-26T07:01:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:56 smithi163 ceph-mon[36873]: 1 slow requests, 1 included below; oldest blocked for > 363.854443 secs 2024-07-26T07:01:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:56 smithi163 ceph-mon[36873]: slow request 363.854443 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T07:01:56.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 346.111953 secs 2024-07-26T07:01:56.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:56 smithi042 ceph-mon[32775]: 1 slow requests, 1 included below; oldest blocked for > 363.854443 secs 2024-07-26T07:01:56.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:56 smithi042 ceph-mon[32775]: slow request 363.854443 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T07:01:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:57 smithi164 ceph-mon[36970]: pgmap v787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:57.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:57 smithi163 ceph-mon[36873]: pgmap v787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:57.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:57 smithi042 ceph-mon[32775]: pgmap v787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:01:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:01:57.910+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650005850 con 0x55db6e0564d0 2024-07-26T07:01:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:01:57.910+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658005850 con 0x7ff680102000 2024-07-26T07:01:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:01:57.910+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658005850 con 0x7ff680102000 2024-07-26T07:01:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:01:57.911+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650005850 con 0x55db6e0564d0 2024-07-26T07:01:59.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:01:59 smithi164 ceph-mon[36970]: pgmap v788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:59.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:01:59 smithi163 ceph-mon[36873]: pgmap v788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:01:59.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:01:59 smithi042 ceph-mon[32775]: pgmap v788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:01 smithi164 ceph-mon[36970]: pgmap v789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 351.112081 secs 2024-07-26T07:02:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 368.854565 secs 2024-07-26T07:02:01.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:01 smithi163 ceph-mon[36873]: pgmap v789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 351.112081 secs 2024-07-26T07:02:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 368.854565 secs 2024-07-26T07:02:01.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:01 smithi042 ceph-mon[32775]: pgmap v789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 351.112081 secs 2024-07-26T07:02:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 368.854565 secs 2024-07-26T07:02:03.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:03 smithi164 ceph-mon[36970]: pgmap v790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:03 smithi163 ceph-mon[36873]: pgmap v790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:03.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:03 smithi042 ceph-mon[32775]: pgmap v790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:05.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:05 smithi042 ceph-mon[32775]: pgmap v791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:02:05.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:05 smithi164 ceph-mon[36970]: pgmap v791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:02:05.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:05 smithi163 ceph-mon[36873]: pgmap v791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:02:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 356.112205 secs 2024-07-26T07:02:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 373.854688 secs 2024-07-26T07:02:06.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 356.112205 secs 2024-07-26T07:02:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 373.854688 secs 2024-07-26T07:02:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 356.112205 secs 2024-07-26T07:02:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 373.854688 secs 2024-07-26T07:02:07.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:07 smithi164 ceph-mon[36970]: pgmap v792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:07.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:07 smithi163 ceph-mon[36873]: pgmap v792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:07.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:07 smithi042 ceph-mon[32775]: pgmap v792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:02:08.385+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:02:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:02:08.385+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:02:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:02:08.385+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:02:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:09 smithi164 ceph-mon[36970]: pgmap v793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:09.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:09 smithi163 ceph-mon[36873]: pgmap v793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:09.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:09 smithi042 ceph-mon[32775]: pgmap v793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:11 smithi164 ceph-mon[36970]: pgmap v794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:11 smithi164 ceph-mon[36970]: 1 slow requests, 1 included below; oldest blocked for > 361.112338 secs 2024-07-26T07:02:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:11 smithi164 ceph-mon[36970]: slow request 361.112338 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T07:02:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 378.854820 secs 2024-07-26T07:02:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:11 smithi163 ceph-mon[36873]: pgmap v794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:11 smithi163 ceph-mon[36873]: 1 slow requests, 1 included below; oldest blocked for > 361.112338 secs 2024-07-26T07:02:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:11 smithi163 ceph-mon[36873]: slow request 361.112338 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T07:02:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 378.854820 secs 2024-07-26T07:02:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:11 smithi042 ceph-mon[32775]: pgmap v794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:11 smithi042 ceph-mon[32775]: 1 slow requests, 1 included below; oldest blocked for > 361.112338 secs 2024-07-26T07:02:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:11 smithi042 ceph-mon[32775]: slow request 361.112338 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T07:02:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 378.854820 secs 2024-07-26T07:02:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:13 smithi164 ceph-mon[36970]: pgmap v795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:13.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:13 smithi163 ceph-mon[36873]: pgmap v795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:13.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:13 smithi042 ceph-mon[32775]: pgmap v795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:15.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:15 smithi164 ceph-mon[36970]: pgmap v796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:02:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:15 smithi163 ceph-mon[36873]: pgmap v796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:02:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:15 smithi042 ceph-mon[32775]: pgmap v796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:02:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 366.112438 secs 2024-07-26T07:02:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 383.854981 secs 2024-07-26T07:02:16.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 366.112438 secs 2024-07-26T07:02:16.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 383.854981 secs 2024-07-26T07:02:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 366.112438 secs 2024-07-26T07:02:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 383.854981 secs 2024-07-26T07:02:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:17 smithi164 ceph-mon[36970]: pgmap v797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:17.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:17 smithi163 ceph-mon[36873]: pgmap v797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:17.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:17 smithi042 ceph-mon[32775]: pgmap v797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:19 smithi164 ceph-mon[36970]: pgmap v798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:19.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:19 smithi163 ceph-mon[36873]: pgmap v798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:19.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:19 smithi042 ceph-mon[32775]: pgmap v798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:02:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:02:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:02:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:02:20.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:02:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:02:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:02:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:02:20.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:02:20.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:02:20.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:02:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:02:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:21 smithi164 ceph-mon[36970]: pgmap v799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:21 smithi164 ceph-mon[36970]: pgmap v800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:02:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:21 smithi164 ceph-mon[36970]: pgmap v801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:02:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 371.112552 secs 2024-07-26T07:02:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 388.855145 secs 2024-07-26T07:02:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:21 smithi163 ceph-mon[36873]: pgmap v799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:21 smithi163 ceph-mon[36873]: pgmap v800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:02:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:21 smithi163 ceph-mon[36873]: pgmap v801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:02:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 371.112552 secs 2024-07-26T07:02:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 388.855145 secs 2024-07-26T07:02:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:21 smithi042 ceph-mon[32775]: pgmap v799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:21 smithi042 ceph-mon[32775]: pgmap v800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:02:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:21 smithi042 ceph-mon[32775]: pgmap v801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:02:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 371.112552 secs 2024-07-26T07:02:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 388.855145 secs 2024-07-26T07:02:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:02:23.388+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:02:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:02:23.388+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:02:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:02:23.388+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:02:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:23 smithi164 ceph-mon[36970]: pgmap v802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:02:23.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:23 smithi163 ceph-mon[36873]: pgmap v802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:02:23.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:23 smithi042 ceph-mon[32775]: pgmap v802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:02:25.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:25 smithi164 ceph-mon[36970]: pgmap v803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:02:25.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:25 smithi163 ceph-mon[36873]: pgmap v803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:02:25.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:25 smithi042 ceph-mon[32775]: pgmap v803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:02:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 376.112633 secs 2024-07-26T07:02:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 393.855308 secs 2024-07-26T07:02:26.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 376.112633 secs 2024-07-26T07:02:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 393.855308 secs 2024-07-26T07:02:26.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 376.112633 secs 2024-07-26T07:02:26.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 393.855308 secs 2024-07-26T07:02:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:27 smithi164 ceph-mon[36970]: pgmap v804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:02:27.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:27 smithi163 ceph-mon[36873]: pgmap v804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:02:27.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:27 smithi042 ceph-mon[32775]: pgmap v804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:02:28.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:28 smithi163 ceph-mon[36873]: pgmap v805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:02:28.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:28 smithi042 ceph-mon[32775]: pgmap v805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:02:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:28 smithi164 ceph-mon[36970]: pgmap v805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:02:31.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:31 smithi042 ceph-mon[32775]: pgmap v806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:02:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 381.112750 secs 2024-07-26T07:02:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:31 smithi164 ceph-mon[36970]: pgmap v806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:02:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 381.112750 secs 2024-07-26T07:02:31.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:31 smithi163 ceph-mon[36873]: pgmap v806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:02:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 381.112750 secs 2024-07-26T07:02:32.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 398.855437 secs 2024-07-26T07:02:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 398.855437 secs 2024-07-26T07:02:32.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 398.855437 secs 2024-07-26T07:02:33.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:33 smithi042 ceph-mon[32775]: pgmap v807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:33.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:33 smithi164 ceph-mon[36970]: pgmap v807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:33.527 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:33 smithi163 ceph-mon[36873]: pgmap v807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:35.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:35 smithi042 ceph-mon[32775]: pgmap v808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:35.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:35 smithi164 ceph-mon[36970]: pgmap v808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:35 smithi163 ceph-mon[36873]: pgmap v808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:36.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 386.112894 secs 2024-07-26T07:02:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 403.855539 secs 2024-07-26T07:02:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 386.112894 secs 2024-07-26T07:02:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 403.855539 secs 2024-07-26T07:02:36.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 386.112894 secs 2024-07-26T07:02:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 403.855539 secs 2024-07-26T07:02:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:37 smithi042 ceph-mon[32775]: pgmap v809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:37 smithi164 ceph-mon[36970]: pgmap v809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:37.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:37 smithi163 ceph-mon[36873]: pgmap v809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:02:38.396+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:02:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:02:38.396+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:02:38.396 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:02:38.396+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:02:39.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:39 smithi042 ceph-mon[32775]: pgmap v810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:39.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:39 smithi164 ceph-mon[36970]: pgmap v810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:39.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:39 smithi163 ceph-mon[36873]: pgmap v810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:41 smithi042 ceph-mon[32775]: pgmap v811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 391.113038 secs 2024-07-26T07:02:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 408.855662 secs 2024-07-26T07:02:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:41 smithi164 ceph-mon[36970]: pgmap v811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 391.113038 secs 2024-07-26T07:02:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 408.855662 secs 2024-07-26T07:02:41.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:41 smithi163 ceph-mon[36873]: pgmap v811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:41.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 391.113038 secs 2024-07-26T07:02:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 408.855662 secs 2024-07-26T07:02:43.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:43 smithi042 ceph-mon[32775]: pgmap v812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:43.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:43 smithi164 ceph-mon[36970]: pgmap v812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:43.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:43 smithi163 ceph-mon[36873]: pgmap v812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:45.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:45 smithi042 ceph-mon[32775]: pgmap v813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:45.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:45 smithi164 ceph-mon[36970]: pgmap v813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:45.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:45 smithi163 ceph-mon[36873]: pgmap v813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:46.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 396.113180 secs 2024-07-26T07:02:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 413.855783 secs 2024-07-26T07:02:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 396.113180 secs 2024-07-26T07:02:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 413.855783 secs 2024-07-26T07:02:46.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 396.113180 secs 2024-07-26T07:02:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 413.855783 secs 2024-07-26T07:02:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:47 smithi042 ceph-mon[32775]: pgmap v814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:47 smithi164 ceph-mon[36970]: pgmap v814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:47.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:47 smithi163 ceph-mon[36873]: pgmap v814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:49.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:49 smithi042 ceph-mon[32775]: pgmap v815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:49 smithi164 ceph-mon[36970]: pgmap v815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:49.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:49 smithi163 ceph-mon[36873]: pgmap v815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:51 smithi164 ceph-mon[36970]: pgmap v816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 401.113309 secs 2024-07-26T07:02:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 418.855923 secs 2024-07-26T07:02:51.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:51 smithi163 ceph-mon[36873]: pgmap v816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 401.113309 secs 2024-07-26T07:02:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 418.855923 secs 2024-07-26T07:02:51.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:51 smithi042 ceph-mon[32775]: pgmap v816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 401.113309 secs 2024-07-26T07:02:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 418.855923 secs 2024-07-26T07:02:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:02:53.399+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:02:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:02:53.399+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:02:53.400 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:02:53.399+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:02:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:53 smithi164 ceph-mon[36970]: pgmap v817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:53.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:53 smithi163 ceph-mon[36873]: pgmap v817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:53.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:53 smithi042 ceph-mon[32775]: pgmap v817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:55 smithi164 ceph-mon[36970]: pgmap v818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:55.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:55 smithi163 ceph-mon[36873]: pgmap v818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:55.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:55 smithi042 ceph-mon[32775]: pgmap v818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:02:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 406.113405 secs 2024-07-26T07:02:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 423.856051 secs 2024-07-26T07:02:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 406.113405 secs 2024-07-26T07:02:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 423.856051 secs 2024-07-26T07:02:56.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 406.113405 secs 2024-07-26T07:02:56.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 423.856051 secs 2024-07-26T07:02:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:57 smithi164 ceph-mon[36970]: pgmap v819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:57.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:57 smithi163 ceph-mon[36873]: pgmap v819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:57 smithi042 ceph-mon[32775]: pgmap v819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:59.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:02:59 smithi164 ceph-mon[36970]: pgmap v820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:59.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:02:59 smithi163 ceph-mon[36873]: pgmap v820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:02:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:02:59 smithi042 ceph-mon[32775]: pgmap v820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:01 smithi164 ceph-mon[36970]: pgmap v821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 411.113541 secs 2024-07-26T07:03:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 428.856172 secs 2024-07-26T07:03:01.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:01 smithi163 ceph-mon[36873]: pgmap v821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 411.113541 secs 2024-07-26T07:03:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 428.856172 secs 2024-07-26T07:03:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:01 smithi042 ceph-mon[32775]: pgmap v821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 411.113541 secs 2024-07-26T07:03:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 428.856172 secs 2024-07-26T07:03:03.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:03 smithi164 ceph-mon[36970]: pgmap v822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:03.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:03 smithi163 ceph-mon[36873]: pgmap v822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:03 smithi042 ceph-mon[32775]: pgmap v822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:05.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:05 smithi164 ceph-mon[36970]: pgmap v823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:05.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:05 smithi163 ceph-mon[36873]: pgmap v823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:05.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:05 smithi042 ceph-mon[32775]: pgmap v823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 416.113661 secs 2024-07-26T07:03:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 433.856292 secs 2024-07-26T07:03:06.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 416.113661 secs 2024-07-26T07:03:06.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 433.856292 secs 2024-07-26T07:03:06.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 416.113661 secs 2024-07-26T07:03:06.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 433.856292 secs 2024-07-26T07:03:07.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:07 smithi164 ceph-mon[36970]: pgmap v824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:07.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:07 smithi163 ceph-mon[36873]: pgmap v824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:07.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:07 smithi042 ceph-mon[32775]: pgmap v824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:03:08.406+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:03:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:03:08.406+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:03:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:03:08.406+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:03:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:09 smithi164 ceph-mon[36970]: pgmap v825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:09 smithi163 ceph-mon[36873]: pgmap v825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:09.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:09 smithi042 ceph-mon[32775]: pgmap v825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:11 smithi164 ceph-mon[36970]: pgmap v826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 421.113804 secs 2024-07-26T07:03:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 438.856416 secs 2024-07-26T07:03:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:11 smithi163 ceph-mon[36873]: pgmap v826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 421.113804 secs 2024-07-26T07:03:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 438.856416 secs 2024-07-26T07:03:11.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:11 smithi042 ceph-mon[32775]: pgmap v826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 421.113804 secs 2024-07-26T07:03:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 438.856416 secs 2024-07-26T07:03:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:13 smithi164 ceph-mon[36970]: pgmap v827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:13 smithi163 ceph-mon[36873]: pgmap v827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:13.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:13 smithi042 ceph-mon[32775]: pgmap v827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:15.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:15 smithi164 ceph-mon[36970]: pgmap v828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:15.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:15 smithi163 ceph-mon[36873]: pgmap v828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:15.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:15 smithi042 ceph-mon[32775]: pgmap v828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 426.113946 secs 2024-07-26T07:03:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 443.856562 secs 2024-07-26T07:03:16.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 426.113946 secs 2024-07-26T07:03:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 443.856562 secs 2024-07-26T07:03:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 426.113946 secs 2024-07-26T07:03:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 443.856562 secs 2024-07-26T07:03:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:17 smithi164 ceph-mon[36970]: pgmap v829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:17.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:17 smithi163 ceph-mon[36873]: pgmap v829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:17.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:17 smithi042 ceph-mon[32775]: pgmap v829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:19 smithi164 ceph-mon[36970]: pgmap v830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:19.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:19 smithi163 ceph-mon[36873]: pgmap v830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:03:20.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:03:20.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:19 smithi042 ceph-mon[32775]: pgmap v830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:20.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:03:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:21 smithi164 ceph-mon[36970]: pgmap v831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:03:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:03:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:03:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 431.114092 secs 2024-07-26T07:03:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 448.856707 secs 2024-07-26T07:03:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:21 smithi163 ceph-mon[36873]: pgmap v831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:03:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:03:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:03:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 431.114092 secs 2024-07-26T07:03:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 448.856707 secs 2024-07-26T07:03:21.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:21 smithi042 ceph-mon[32775]: pgmap v831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:03:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:03:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:03:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 431.114092 secs 2024-07-26T07:03:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 448.856707 secs 2024-07-26T07:03:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:22 smithi164 ceph-mon[36970]: pgmap v832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:03:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:22 smithi164 ceph-mon[36970]: pgmap v833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:03:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:22 smithi163 ceph-mon[36873]: pgmap v832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:03:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:22 smithi163 ceph-mon[36873]: pgmap v833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:03:22.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:22 smithi042 ceph-mon[32775]: pgmap v832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:03:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:22 smithi042 ceph-mon[32775]: pgmap v833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:03:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:03:23.409+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:03:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:03:23.409+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:03:23.409 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:03:23.409+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:03:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:24 smithi163 ceph-mon[36873]: pgmap v834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:03:24.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:24 smithi042 ceph-mon[32775]: pgmap v834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:03:24.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:24 smithi164 ceph-mon[36970]: pgmap v834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:03:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:26 smithi163 ceph-mon[36873]: pgmap v835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:03:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 436.114224 secs 2024-07-26T07:03:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 453.856829 secs 2024-07-26T07:03:26.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:26 smithi042 ceph-mon[32775]: pgmap v835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:03:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 436.114224 secs 2024-07-26T07:03:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 453.856829 secs 2024-07-26T07:03:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:26 smithi164 ceph-mon[36970]: pgmap v835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:03:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 436.114224 secs 2024-07-26T07:03:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 453.856829 secs 2024-07-26T07:03:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:27 smithi042 ceph-mon[32775]: pgmap v836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:03:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:27 smithi164 ceph-mon[36970]: pgmap v836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:03:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:27 smithi163 ceph-mon[36873]: pgmap v836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:03:29.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:29 smithi164 ceph-mon[36970]: pgmap v837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:03:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:29 smithi163 ceph-mon[36873]: pgmap v837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:03:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:29 smithi042 ceph-mon[32775]: pgmap v837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:03:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:31 smithi164 ceph-mon[36970]: pgmap v838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:03:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 441.114338 secs 2024-07-26T07:03:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 458.856967 secs 2024-07-26T07:03:32.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:31 smithi163 ceph-mon[36873]: pgmap v838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:03:32.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 441.114338 secs 2024-07-26T07:03:32.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 458.856967 secs 2024-07-26T07:03:32.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:31 smithi042 ceph-mon[32775]: pgmap v838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:03:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 441.114338 secs 2024-07-26T07:03:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 458.856967 secs 2024-07-26T07:03:33.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:33 smithi164 ceph-mon[36970]: pgmap v839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:34.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:33 smithi163 ceph-mon[36873]: pgmap v839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:34.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:33 smithi042 ceph-mon[32775]: pgmap v839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:35.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:35 smithi164 ceph-mon[36970]: pgmap v840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:36.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:35 smithi163 ceph-mon[36873]: pgmap v840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:36.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:35 smithi042 ceph-mon[32775]: pgmap v840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 446.114412 secs 2024-07-26T07:03:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 463.857090 secs 2024-07-26T07:03:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 446.114412 secs 2024-07-26T07:03:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 463.857090 secs 2024-07-26T07:03:37.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 446.114412 secs 2024-07-26T07:03:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 463.857090 secs 2024-07-26T07:03:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:37 smithi164 ceph-mon[36970]: pgmap v841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:37 smithi163 ceph-mon[36873]: pgmap v841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:38.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:37 smithi042 ceph-mon[32775]: pgmap v841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:03:38.416+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:03:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:03:38.416+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:03:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:03:38.416+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:03:39.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:39 smithi164 ceph-mon[36970]: pgmap v842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:03:40.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:39 smithi163 ceph-mon[36873]: pgmap v842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:03:40.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:39 smithi042 ceph-mon[32775]: pgmap v842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:03:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:41 smithi164 ceph-mon[36970]: pgmap v843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 451.114516 secs 2024-07-26T07:03:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 468.857217 secs 2024-07-26T07:03:42.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:41 smithi163 ceph-mon[36873]: pgmap v843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 451.114516 secs 2024-07-26T07:03:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 468.857217 secs 2024-07-26T07:03:42.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:41 smithi042 ceph-mon[32775]: pgmap v843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 451.114516 secs 2024-07-26T07:03:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 468.857217 secs 2024-07-26T07:03:43.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:43 smithi164 ceph-mon[36970]: pgmap v844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:44.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:43 smithi163 ceph-mon[36873]: pgmap v844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:43 smithi042 ceph-mon[32775]: pgmap v844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:45.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:45 smithi164 ceph-mon[36970]: pgmap v845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:46.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:45 smithi163 ceph-mon[36873]: pgmap v845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:46.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:45 smithi042 ceph-mon[32775]: pgmap v845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 456.114639 secs 2024-07-26T07:03:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 473.857345 secs 2024-07-26T07:03:47.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 456.114639 secs 2024-07-26T07:03:47.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 473.857345 secs 2024-07-26T07:03:47.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 456.114639 secs 2024-07-26T07:03:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 473.857345 secs 2024-07-26T07:03:47.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:47 smithi164 ceph-mon[36970]: pgmap v846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:48.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:47 smithi163 ceph-mon[36873]: pgmap v846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:48.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:47 smithi042 ceph-mon[32775]: pgmap v846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:49.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:49 smithi164 ceph-mon[36970]: pgmap v847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:03:50.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:49 smithi163 ceph-mon[36873]: pgmap v847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:03:50.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:49 smithi042 ceph-mon[32775]: pgmap v847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:03:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:51 smithi164 ceph-mon[36970]: pgmap v848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 461.114778 secs 2024-07-26T07:03:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 478.857468 secs 2024-07-26T07:03:52.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:51 smithi163 ceph-mon[36873]: pgmap v848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 461.114778 secs 2024-07-26T07:03:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 478.857468 secs 2024-07-26T07:03:52.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:51 smithi042 ceph-mon[32775]: pgmap v848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 461.114778 secs 2024-07-26T07:03:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 478.857468 secs 2024-07-26T07:03:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:03:53.419+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:03:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:03:53.419+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:03:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:03:53.419+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:03:53.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:53 smithi164 ceph-mon[36970]: pgmap v849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:54.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:53 smithi163 ceph-mon[36873]: pgmap v849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:54.095 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:53 smithi042 ceph-mon[32775]: pgmap v849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:55.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:55 smithi164 ceph-mon[36970]: pgmap v850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:56.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:55 smithi163 ceph-mon[36873]: pgmap v850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:55 smithi042 ceph-mon[32775]: pgmap v850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:03:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 466.114917 secs 2024-07-26T07:03:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 483.857590 secs 2024-07-26T07:03:57.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 466.114917 secs 2024-07-26T07:03:57.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 483.857590 secs 2024-07-26T07:03:57.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 466.114917 secs 2024-07-26T07:03:57.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 483.857590 secs 2024-07-26T07:03:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:57 smithi164 ceph-mon[36970]: pgmap v851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:58.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:57 smithi163 ceph-mon[36873]: pgmap v851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:58.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:57 smithi042 ceph-mon[32775]: pgmap v851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:03:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:03:59 smithi164 ceph-mon[36970]: pgmap v852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:04:00.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:03:59 smithi163 ceph-mon[36873]: pgmap v852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:04:00.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:03:59 smithi042 ceph-mon[32775]: pgmap v852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:04:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:01 smithi164 ceph-mon[36970]: pgmap v853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 471.115042 secs 2024-07-26T07:04:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 488.857712 secs 2024-07-26T07:04:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:01 smithi163 ceph-mon[36873]: pgmap v853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 471.115042 secs 2024-07-26T07:04:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 488.857712 secs 2024-07-26T07:04:02.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:01 smithi042 ceph-mon[32775]: pgmap v853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:02.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 471.115042 secs 2024-07-26T07:04:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 488.857712 secs 2024-07-26T07:04:03.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:03 smithi164 ceph-mon[36970]: pgmap v854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:04.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:03 smithi163 ceph-mon[36873]: pgmap v854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:04.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:03 smithi042 ceph-mon[32775]: pgmap v854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:05 smithi164 ceph-mon[36970]: pgmap v855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:06.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:05 smithi163 ceph-mon[36873]: pgmap v855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:06.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:05 smithi042 ceph-mon[32775]: pgmap v855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:07.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 476.115195 secs 2024-07-26T07:04:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 493.857834 secs 2024-07-26T07:04:07.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 476.115195 secs 2024-07-26T07:04:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 493.857834 secs 2024-07-26T07:04:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 476.115195 secs 2024-07-26T07:04:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 493.857834 secs 2024-07-26T07:04:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:07 smithi163 ceph-mon[36873]: pgmap v856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:07 smithi042 ceph-mon[32775]: pgmap v856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:08.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:07 smithi164 ceph-mon[36970]: pgmap v856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:08.426 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:08.425+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:04:08.426 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:08.425+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:04:08.426 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:08.425+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:04:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:09 smithi163 ceph-mon[36873]: pgmap v857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:04:10.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:09 smithi042 ceph-mon[32775]: pgmap v857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:04:10.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:09 smithi164 ceph-mon[36970]: pgmap v857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:04:11.734 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:11 smithi164 ceph-mon[36970]: pgmap v858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:11.735 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 481.115341 secs 2024-07-26T07:04:11.735 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 498.857981 secs 2024-07-26T07:04:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:11 smithi163 ceph-mon[36873]: pgmap v858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 481.115341 secs 2024-07-26T07:04:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 498.857981 secs 2024-07-26T07:04:12.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:11 smithi042 ceph-mon[32775]: pgmap v858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 481.115341 secs 2024-07-26T07:04:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 498.857981 secs 2024-07-26T07:04:12.933 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:12.932+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658005b10 con 0x7ff680102000 2024-07-26T07:04:12.933 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:12.932+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650005b50 con 0x55db6e0564d0 2024-07-26T07:04:12.933 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:12.932+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658005b10 con 0x7ff680102000 2024-07-26T07:04:12.933 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:12.932+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff650005b50 con 0x55db6e0564d0 2024-07-26T07:04:14.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:13 smithi163 ceph-mon[36873]: pgmap v859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:14.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:13 smithi042 ceph-mon[32775]: pgmap v859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:14.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:13 smithi164 ceph-mon[36970]: pgmap v859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:15 smithi163 ceph-mon[36873]: pgmap v860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:16.111 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:15 smithi042 ceph-mon[32775]: pgmap v860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:16.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:15 smithi164 ceph-mon[36970]: pgmap v860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 486.115428 secs 2024-07-26T07:04:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 503.858103 secs 2024-07-26T07:04:17.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 486.115428 secs 2024-07-26T07:04:17.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 503.858103 secs 2024-07-26T07:04:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 486.115428 secs 2024-07-26T07:04:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 503.858103 secs 2024-07-26T07:04:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:17 smithi163 ceph-mon[36873]: pgmap v861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:18.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:17 smithi042 ceph-mon[32775]: pgmap v861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:18.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:17 smithi164 ceph-mon[36970]: pgmap v861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:20.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:19 smithi163 ceph-mon[36873]: pgmap v862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:04:20.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:19 smithi042 ceph-mon[32775]: pgmap v862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:04:20.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:19 smithi164 ceph-mon[36970]: pgmap v862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:04:21.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:20 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 491.115555 secs 2024-07-26T07:04:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:04:21.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:20 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 491.115555 secs 2024-07-26T07:04:21.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:04:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:20 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 491.115555 secs 2024-07-26T07:04:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:04:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:21 smithi042 ceph-mon[32775]: pgmap v863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 508.858169 secs 2024-07-26T07:04:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:04:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:04:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:21 smithi042 ceph-mon[32775]: pgmap v864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:04:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:21 smithi042 ceph-mon[32775]: pgmap v865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:04:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:04:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:21 smithi164 ceph-mon[36970]: pgmap v863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 508.858169 secs 2024-07-26T07:04:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:04:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:04:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:21 smithi164 ceph-mon[36970]: pgmap v864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:04:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:21 smithi164 ceph-mon[36970]: pgmap v865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:04:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:04:22.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:21 smithi163 ceph-mon[36873]: pgmap v863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 508.858169 secs 2024-07-26T07:04:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:04:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:04:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:21 smithi163 ceph-mon[36873]: pgmap v864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:04:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:21 smithi163 ceph-mon[36873]: pgmap v865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:04:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:04:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:23.428+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:04:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:23.428+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:04:23.429 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:23.428+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:04:24.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:24 smithi164 ceph-mon[36970]: pgmap v866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:04:24.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:24 smithi163 ceph-mon[36873]: pgmap v866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:04:24.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:24 smithi042 ceph-mon[32775]: pgmap v866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:04:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:26 smithi164 ceph-mon[36970]: pgmap v867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:04:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 496.115677 secs 2024-07-26T07:04:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 513.858260 secs 2024-07-26T07:04:26.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:26 smithi163 ceph-mon[36873]: pgmap v867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:04:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 496.115677 secs 2024-07-26T07:04:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 513.858260 secs 2024-07-26T07:04:26.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:26 smithi042 ceph-mon[32775]: pgmap v867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:04:26.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 496.115677 secs 2024-07-26T07:04:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 513.858260 secs 2024-07-26T07:04:28.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:28 smithi164 ceph-mon[36970]: pgmap v868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:04:28.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:28 smithi163 ceph-mon[36873]: pgmap v868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:04:28.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:28 smithi042 ceph-mon[32775]: pgmap v868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:04:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:30 smithi164 ceph-mon[36970]: pgmap v869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:04:30.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:30 smithi163 ceph-mon[36873]: pgmap v869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:04:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:30 smithi042 ceph-mon[32775]: pgmap v869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:04:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 501.115760 secs 2024-07-26T07:04:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 518.858406 secs 2024-07-26T07:04:31.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 501.115760 secs 2024-07-26T07:04:31.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 518.858406 secs 2024-07-26T07:04:31.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 501.115760 secs 2024-07-26T07:04:31.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 518.858406 secs 2024-07-26T07:04:32.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:32 smithi163 ceph-mon[36873]: pgmap v870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:04:32.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:32 smithi042 ceph-mon[32775]: pgmap v870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:04:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:32 smithi164 ceph-mon[36970]: pgmap v870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:04:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:34 smithi163 ceph-mon[36873]: pgmap v871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:34.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:34 smithi042 ceph-mon[32775]: pgmap v871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:34 smithi164 ceph-mon[36970]: pgmap v871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:36 smithi163 ceph-mon[36873]: pgmap v872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 506.115895 secs 2024-07-26T07:04:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 523.858532 secs 2024-07-26T07:04:36.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:36 smithi042 ceph-mon[32775]: pgmap v872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:36.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 506.115895 secs 2024-07-26T07:04:36.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 523.858532 secs 2024-07-26T07:04:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:36 smithi164 ceph-mon[36970]: pgmap v872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 506.115895 secs 2024-07-26T07:04:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 523.858532 secs 2024-07-26T07:04:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:38.436+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:04:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:38.436+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:04:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:38.436+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:04:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:38 smithi163 ceph-mon[36873]: pgmap v873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:38.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:38 smithi042 ceph-mon[32775]: pgmap v873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:38 smithi164 ceph-mon[36970]: pgmap v873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:40.447 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:40 smithi042 ceph-mon[32775]: pgmap v874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:40.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:40 smithi163 ceph-mon[36873]: pgmap v874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:40 smithi164 ceph-mon[36970]: pgmap v874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:41.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 511.115967 secs 2024-07-26T07:04:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 528.858673 secs 2024-07-26T07:04:41.623 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 511.115967 secs 2024-07-26T07:04:41.623 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 528.858673 secs 2024-07-26T07:04:41.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 511.115967 secs 2024-07-26T07:04:41.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 528.858673 secs 2024-07-26T07:04:42.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:42 smithi163 ceph-mon[36873]: pgmap v875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:42.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:42 smithi042 ceph-mon[32775]: pgmap v875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:42 smithi164 ceph-mon[36970]: pgmap v875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:43.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:43 smithi163 ceph-mon[36873]: pgmap v876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:43.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:43 smithi042 ceph-mon[32775]: pgmap v876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:43 smithi164 ceph-mon[36970]: pgmap v876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:46 smithi164 ceph-mon[36970]: pgmap v877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 516.116036 secs 2024-07-26T07:04:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 533.858813 secs 2024-07-26T07:04:46.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:46 smithi163 ceph-mon[36873]: pgmap v877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 516.116036 secs 2024-07-26T07:04:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 533.858813 secs 2024-07-26T07:04:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:46 smithi042 ceph-mon[32775]: pgmap v877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 516.116036 secs 2024-07-26T07:04:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 533.858813 secs 2024-07-26T07:04:48.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:48 smithi164 ceph-mon[36970]: pgmap v878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:48.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:48 smithi163 ceph-mon[36873]: pgmap v878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:48.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:48 smithi042 ceph-mon[32775]: pgmap v878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:50.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:50 smithi164 ceph-mon[36970]: pgmap v879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:50.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:50 smithi163 ceph-mon[36873]: pgmap v879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:50.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:50 smithi042 ceph-mon[32775]: pgmap v879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:51.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 521.116176 secs 2024-07-26T07:04:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 538.858954 secs 2024-07-26T07:04:51.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 521.116176 secs 2024-07-26T07:04:51.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 538.858954 secs 2024-07-26T07:04:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 521.116176 secs 2024-07-26T07:04:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 538.858954 secs 2024-07-26T07:04:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:52 smithi163 ceph-mon[36873]: pgmap v880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:52.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:52 smithi042 ceph-mon[32775]: pgmap v880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:52 smithi164 ceph-mon[36970]: pgmap v880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:53.439+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:04:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:53.439+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:04:53.440 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:53.439+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:04:54.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:54 smithi163 ceph-mon[36873]: pgmap v881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:54.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:54 smithi042 ceph-mon[32775]: pgmap v881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:54 smithi164 ceph-mon[36970]: pgmap v881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:55 smithi164 ceph-mon[36970]: pgmap v882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:55.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:55 smithi163 ceph-mon[36873]: pgmap v882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:55.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:55 smithi042 ceph-mon[32775]: pgmap v882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:04:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 526.116254 secs 2024-07-26T07:04:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 543.859072 secs 2024-07-26T07:04:56.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 526.116254 secs 2024-07-26T07:04:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 543.859072 secs 2024-07-26T07:04:56.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 526.116254 secs 2024-07-26T07:04:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 543.859072 secs 2024-07-26T07:04:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:04:57 smithi164 ceph-mon[36970]: pgmap v883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:57.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:04:57 smithi163 ceph-mon[36873]: pgmap v883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:57.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:04:57 smithi042 ceph-mon[32775]: pgmap v883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:04:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:57.940+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650005e50 con 0x55db6e0564d0 2024-07-26T07:04:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:57.940+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658005e10 con 0x7ff680102000 2024-07-26T07:04:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:57.940+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658005e10 con 0x7ff680102000 2024-07-26T07:04:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:04:57.940+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650005e50 con 0x55db6e0564d0 2024-07-26T07:05:00.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:00 smithi164 ceph-mon[36970]: pgmap v884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:00.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:00 smithi163 ceph-mon[36873]: pgmap v884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:00.598 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:00 smithi042 ceph-mon[32775]: pgmap v884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 531.116383 secs 2024-07-26T07:05:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 548.859213 secs 2024-07-26T07:05:01.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 531.116383 secs 2024-07-26T07:05:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 548.859213 secs 2024-07-26T07:05:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 531.116383 secs 2024-07-26T07:05:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 548.859213 secs 2024-07-26T07:05:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:02 smithi163 ceph-mon[36873]: pgmap v885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:02.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:02 smithi042 ceph-mon[32775]: pgmap v885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:02.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:02 smithi164 ceph-mon[36970]: pgmap v885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:04.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:04 smithi163 ceph-mon[36873]: pgmap v886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:04.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:04 smithi042 ceph-mon[32775]: pgmap v886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:04.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:04 smithi164 ceph-mon[36970]: pgmap v886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:06 smithi163 ceph-mon[36873]: pgmap v887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 536.116503 secs 2024-07-26T07:05:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 553.859369 secs 2024-07-26T07:05:06.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:06 smithi042 ceph-mon[32775]: pgmap v887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:06.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 536.116503 secs 2024-07-26T07:05:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 553.859369 secs 2024-07-26T07:05:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:06 smithi164 ceph-mon[36970]: pgmap v887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 536.116503 secs 2024-07-26T07:05:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 553.859369 secs 2024-07-26T07:05:08.447 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:05:08.447+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:05:08.447 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:05:08.447+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:05:08.447 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:05:08.447+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:05:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:08 smithi163 ceph-mon[36873]: pgmap v888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:08.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:08 smithi042 ceph-mon[32775]: pgmap v888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:08 smithi164 ceph-mon[36970]: pgmap v888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:10 smithi163 ceph-mon[36873]: pgmap v889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:10.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:10 smithi042 ceph-mon[32775]: pgmap v889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:10 smithi164 ceph-mon[36970]: pgmap v889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 541.116628 secs 2024-07-26T07:05:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 558.859514 secs 2024-07-26T07:05:11.623 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 541.116628 secs 2024-07-26T07:05:11.623 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 558.859514 secs 2024-07-26T07:05:11.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 541.116628 secs 2024-07-26T07:05:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 558.859514 secs 2024-07-26T07:05:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:12 smithi163 ceph-mon[36873]: pgmap v890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:12.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:12 smithi042 ceph-mon[32775]: pgmap v890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:12 smithi164 ceph-mon[36970]: pgmap v890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:13.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:13 smithi042 ceph-mon[32775]: pgmap v891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:13.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:13 smithi164 ceph-mon[36970]: pgmap v891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:13.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:13 smithi163 ceph-mon[36873]: pgmap v891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:16 smithi164 ceph-mon[36970]: pgmap v892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 546.116746 secs 2024-07-26T07:05:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 563.859637 secs 2024-07-26T07:05:16.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:16 smithi163 ceph-mon[36873]: pgmap v892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 546.116746 secs 2024-07-26T07:05:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 563.859637 secs 2024-07-26T07:05:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:16 smithi042 ceph-mon[32775]: pgmap v892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 546.116746 secs 2024-07-26T07:05:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 563.859637 secs 2024-07-26T07:05:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:18 smithi163 ceph-mon[36873]: pgmap v893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:18.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:18 smithi042 ceph-mon[32775]: pgmap v893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:18.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:18 smithi164 ceph-mon[36970]: pgmap v893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:20 smithi163 ceph-mon[36873]: pgmap v894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:20.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:20 smithi042 ceph-mon[32775]: pgmap v894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:20.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:20 smithi164 ceph-mon[36970]: pgmap v894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:21.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 551.116870 secs 2024-07-26T07:05:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 568.859762 secs 2024-07-26T07:05:21.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 551.116870 secs 2024-07-26T07:05:21.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 568.859762 secs 2024-07-26T07:05:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 551.116870 secs 2024-07-26T07:05:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 568.859762 secs 2024-07-26T07:05:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:22 smithi163 ceph-mon[36873]: pgmap v895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:05:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:05:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:05:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:05:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:22 smithi042 ceph-mon[32775]: pgmap v895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:05:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:05:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:05:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:05:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:22 smithi164 ceph-mon[36970]: pgmap v895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:05:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:05:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:05:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:05:23.449 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:05:23.448+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:05:23.450 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:05:23.448+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:05:23.450 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:05:23.448+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:05:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:23 smithi163 ceph-mon[36873]: pgmap v896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:05:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:23 smithi163 ceph-mon[36873]: pgmap v897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:05:23.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:23 smithi042 ceph-mon[32775]: pgmap v896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:05:23.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:23 smithi042 ceph-mon[32775]: pgmap v897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:05:23.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:23 smithi164 ceph-mon[36970]: pgmap v896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:05:23.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:23 smithi164 ceph-mon[36970]: pgmap v897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:05:25.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:25 smithi163 ceph-mon[36873]: pgmap v898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:05:25.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:25 smithi042 ceph-mon[32775]: pgmap v898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:05:25.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:25 smithi164 ceph-mon[36970]: pgmap v898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:05:26.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 556.116995 secs 2024-07-26T07:05:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 573.859904 secs 2024-07-26T07:05:26.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 556.116995 secs 2024-07-26T07:05:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 573.859904 secs 2024-07-26T07:05:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 556.116995 secs 2024-07-26T07:05:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 573.859904 secs 2024-07-26T07:05:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:27 smithi163 ceph-mon[36873]: pgmap v899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:05:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:27 smithi042 ceph-mon[32775]: pgmap v899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:05:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:27 smithi164 ceph-mon[36970]: pgmap v899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:05:28.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:28 smithi042 ceph-mon[32775]: pgmap v900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:05:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:28 smithi164 ceph-mon[36970]: pgmap v900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:05:28.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:28 smithi163 ceph-mon[36873]: pgmap v900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:05:31.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:30 smithi042 ceph-mon[32775]: pgmap v901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:05:31.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:30 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 561.117140 secs 2024-07-26T07:05:31.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:30 smithi164 ceph-mon[36970]: pgmap v901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:05:31.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:30 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 561.117140 secs 2024-07-26T07:05:31.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:30 smithi163 ceph-mon[36873]: pgmap v901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:05:31.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:30 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 561.117140 secs 2024-07-26T07:05:32.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 578.860005 secs 2024-07-26T07:05:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 578.860005 secs 2024-07-26T07:05:32.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 578.860005 secs 2024-07-26T07:05:33.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:32 smithi042 ceph-mon[32775]: pgmap v902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:05:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:32 smithi164 ceph-mon[36970]: pgmap v902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:05:33.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:32 smithi163 ceph-mon[36873]: pgmap v902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:05:35.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:34 smithi042 ceph-mon[32775]: pgmap v903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-07-26T07:05:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:34 smithi164 ceph-mon[36970]: pgmap v903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-07-26T07:05:35.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:34 smithi163 ceph-mon[36873]: pgmap v903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-07-26T07:05:36.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:35 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 566.117262 secs 2024-07-26T07:05:36.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:35 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 566.117262 secs 2024-07-26T07:05:36.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:35 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 566.117262 secs 2024-07-26T07:05:37.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:36 smithi042 ceph-mon[32775]: pgmap v904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:05:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 583.860132 secs 2024-07-26T07:05:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:36 smithi164 ceph-mon[36970]: pgmap v904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:05:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 583.860132 secs 2024-07-26T07:05:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:36 smithi163 ceph-mon[36873]: pgmap v904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:05:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 583.860132 secs 2024-07-26T07:05:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:05:38.454+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:05:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:05:38.454+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:05:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:05:38.454+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:05:39.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:38 smithi042 ceph-mon[32775]: pgmap v905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:05:39.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:38 smithi164 ceph-mon[36970]: pgmap v905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:05:39.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:38 smithi163 ceph-mon[36873]: pgmap v905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:05:41.059 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:40 smithi163 ceph-mon[36873]: pgmap v906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:05:41.059 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:40 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 571.117411 secs 2024-07-26T07:05:41.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:40 smithi042 ceph-mon[32775]: pgmap v906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:05:41.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:40 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 571.117411 secs 2024-07-26T07:05:41.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:40 smithi164 ceph-mon[36970]: pgmap v906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:05:41.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:40 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 571.117411 secs 2024-07-26T07:05:42.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 588.860233 secs 2024-07-26T07:05:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 588.860233 secs 2024-07-26T07:05:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 588.860233 secs 2024-07-26T07:05:43.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:42 smithi042 ceph-mon[32775]: pgmap v907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:43.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:42 smithi164 ceph-mon[36970]: pgmap v907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:43.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:42 smithi163 ceph-mon[36873]: pgmap v907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:45.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:44 smithi042 ceph-mon[32775]: pgmap v908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:05:45.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:44 smithi164 ceph-mon[36970]: pgmap v908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:05:45.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:44 smithi163 ceph-mon[36873]: pgmap v908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:05:46.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:45 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 576.117529 secs 2024-07-26T07:05:46.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:45 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 576.117529 secs 2024-07-26T07:05:46.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:45 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 576.117529 secs 2024-07-26T07:05:47.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:46 smithi042 ceph-mon[32775]: pgmap v909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:47.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 593.860377 secs 2024-07-26T07:05:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:46 smithi164 ceph-mon[36970]: pgmap v909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 593.860377 secs 2024-07-26T07:05:47.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:46 smithi163 ceph-mon[36873]: pgmap v909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 593.860377 secs 2024-07-26T07:05:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:48 smithi042 ceph-mon[32775]: pgmap v910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:49.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:48 smithi164 ceph-mon[36970]: pgmap v910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:49.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:48 smithi163 ceph-mon[36873]: pgmap v910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:51.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:50 smithi042 ceph-mon[32775]: pgmap v911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:50 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 581.117654 secs 2024-07-26T07:05:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:50 smithi164 ceph-mon[36970]: pgmap v911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:50 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 581.117654 secs 2024-07-26T07:05:51.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:50 smithi163 ceph-mon[36873]: pgmap v911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:50 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 581.117654 secs 2024-07-26T07:05:52.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 598.860520 secs 2024-07-26T07:05:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 598.860520 secs 2024-07-26T07:05:52.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 598.860520 secs 2024-07-26T07:05:53.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:52 smithi042 ceph-mon[32775]: pgmap v912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:53.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:52 smithi164 ceph-mon[36970]: pgmap v912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:53.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:52 smithi163 ceph-mon[36873]: pgmap v912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:05:53.457+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:05:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:05:53.457+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:05:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:05:53.457+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:05:55.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:54 smithi042 ceph-mon[32775]: pgmap v913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:05:55.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:54 smithi164 ceph-mon[36970]: pgmap v913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:05:55.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:54 smithi163 ceph-mon[36873]: pgmap v913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:05:56.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:55 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 586.117781 secs 2024-07-26T07:05:56.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:55 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 586.117781 secs 2024-07-26T07:05:56.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:55 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 586.117781 secs 2024-07-26T07:05:57.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:56 smithi042 ceph-mon[32775]: pgmap v914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:57.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 603.860662 secs 2024-07-26T07:05:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:56 smithi164 ceph-mon[36970]: pgmap v914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 603.860662 secs 2024-07-26T07:05:57.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:56 smithi163 ceph-mon[36873]: pgmap v914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:05:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 603.860662 secs 2024-07-26T07:05:59.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:05:58 smithi042 ceph-mon[32775]: pgmap v915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:59.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:05:58 smithi164 ceph-mon[36970]: pgmap v915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:05:59.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:05:58 smithi163 ceph-mon[36873]: pgmap v915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:01.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:00 smithi042 ceph-mon[32775]: pgmap v916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:00 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 591.117918 secs 2024-07-26T07:06:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:00 smithi164 ceph-mon[36970]: pgmap v916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:00 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 591.117918 secs 2024-07-26T07:06:01.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:00 smithi163 ceph-mon[36873]: pgmap v916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:00 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 591.117918 secs 2024-07-26T07:06:02.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 608.860816 secs 2024-07-26T07:06:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 608.860816 secs 2024-07-26T07:06:02.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 608.860816 secs 2024-07-26T07:06:03.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:02 smithi042 ceph-mon[32775]: pgmap v917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:02 smithi164 ceph-mon[36970]: pgmap v917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:02 smithi163 ceph-mon[36873]: pgmap v917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:05.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:04 smithi042 ceph-mon[32775]: pgmap v918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:06:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:04 smithi164 ceph-mon[36970]: pgmap v918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:06:05.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:04 smithi163 ceph-mon[36873]: pgmap v918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:06:06.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:05 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 596.118058 secs 2024-07-26T07:06:06.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:05 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 596.118058 secs 2024-07-26T07:06:06.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:05 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 596.118058 secs 2024-07-26T07:06:07.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:06 smithi042 ceph-mon[32775]: pgmap v919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 613.860960 secs 2024-07-26T07:06:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:06 smithi164 ceph-mon[36970]: pgmap v919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 613.860960 secs 2024-07-26T07:06:07.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:06 smithi163 ceph-mon[36873]: pgmap v919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 613.860960 secs 2024-07-26T07:06:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:06:08.465+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:06:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:06:08.465+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:06:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:06:08.465+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:06:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:08 smithi042 ceph-mon[32775]: pgmap v920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:09.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:08 smithi164 ceph-mon[36970]: pgmap v920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:09.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:08 smithi163 ceph-mon[36873]: pgmap v920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:11.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:10 smithi042 ceph-mon[32775]: pgmap v921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:11.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:10 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 601.118201 secs 2024-07-26T07:06:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:10 smithi164 ceph-mon[36970]: pgmap v921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:10 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 601.118201 secs 2024-07-26T07:06:11.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:10 smithi163 ceph-mon[36873]: pgmap v921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:11.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:10 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 601.118201 secs 2024-07-26T07:06:12.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 618.861111 secs 2024-07-26T07:06:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 618.861111 secs 2024-07-26T07:06:12.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 618.861111 secs 2024-07-26T07:06:13.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:12 smithi042 ceph-mon[32775]: pgmap v922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:12 smithi164 ceph-mon[36970]: pgmap v922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:13.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:12 smithi163 ceph-mon[36873]: pgmap v922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:15.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:14 smithi042 ceph-mon[32775]: pgmap v923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:06:15.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:14 smithi164 ceph-mon[36970]: pgmap v923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:06:15.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:14 smithi163 ceph-mon[36873]: pgmap v923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:06:16.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:15 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 606.118250 secs 2024-07-26T07:06:16.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:15 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 606.118250 secs 2024-07-26T07:06:16.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:15 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 606.118250 secs 2024-07-26T07:06:17.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:16 smithi042 ceph-mon[32775]: pgmap v924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:17.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 623.861256 secs 2024-07-26T07:06:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:16 smithi164 ceph-mon[36970]: pgmap v924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 623.861256 secs 2024-07-26T07:06:17.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:16 smithi163 ceph-mon[36873]: pgmap v924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:17.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 623.861256 secs 2024-07-26T07:06:19.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:18 smithi042 ceph-mon[32775]: pgmap v925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:18 smithi164 ceph-mon[36970]: pgmap v925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:19.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:18 smithi163 ceph-mon[36873]: pgmap v925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:20 smithi164 ceph-mon[36970]: pgmap v926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:20 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 611.118369 secs 2024-07-26T07:06:21.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:20 smithi163 ceph-mon[36873]: pgmap v926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:21.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:20 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 611.118369 secs 2024-07-26T07:06:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:20 smithi042 ceph-mon[32775]: pgmap v926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:20 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 611.118369 secs 2024-07-26T07:06:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 628.861342 secs 2024-07-26T07:06:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:06:22.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 628.861342 secs 2024-07-26T07:06:22.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:06:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 628.861342 secs 2024-07-26T07:06:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:06:23.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:22 smithi164 ceph-mon[36970]: pgmap v927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:23.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:06:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:06:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:06:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:22 smithi163 ceph-mon[36873]: pgmap v927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:06:23.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:06:23.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:06:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:22 smithi042 ceph-mon[32775]: pgmap v927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:06:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:06:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:06:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:06:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:06:23.468+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:06:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:06:23.468+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:06:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:06:23.468+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:06:24.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:23 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:06:24.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:23 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:06:24.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:23 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:06:24.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:23 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:06:24.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:23 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:06:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:23 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:06:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:23 smithi164 ceph-mon[36970]: pgmap v928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:06:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:23 smithi164 ceph-mon[36970]: pgmap v929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:06:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:23 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:06:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:23 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:06:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:23 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:06:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:23 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:06:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:23 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:06:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:23 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:06:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:23 smithi163 ceph-mon[36873]: pgmap v928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:06:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:23 smithi163 ceph-mon[36873]: pgmap v929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:06:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:23 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:06:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:23 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:06:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:23 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:06:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:23 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:06:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:23 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:06:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:23 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:06:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:23 smithi042 ceph-mon[32775]: pgmap v928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:06:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:23 smithi042 ceph-mon[32775]: pgmap v929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:06:26.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:25 smithi164 ceph-mon[36970]: pgmap v930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:06:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:25 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 616.118432 secs 2024-07-26T07:06:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:25 smithi163 ceph-mon[36873]: pgmap v930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:06:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:25 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 616.118432 secs 2024-07-26T07:06:26.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:25 smithi042 ceph-mon[32775]: pgmap v930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:06:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:25 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 616.118432 secs 2024-07-26T07:06:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 633.861486 secs 2024-07-26T07:06:27.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 633.861486 secs 2024-07-26T07:06:27.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 633.861486 secs 2024-07-26T07:06:28.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:27 smithi164 ceph-mon[36970]: pgmap v931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:06:28.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:27 smithi163 ceph-mon[36873]: pgmap v931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:06:28.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:27 smithi042 ceph-mon[32775]: pgmap v931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:06:30.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:29 smithi164 ceph-mon[36970]: pgmap v932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:06:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:29 smithi163 ceph-mon[36873]: pgmap v932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:06:30.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:29 smithi042 ceph-mon[32775]: pgmap v932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:06:31.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:30 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 621.118562 secs 2024-07-26T07:06:31.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:30 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 621.118562 secs 2024-07-26T07:06:31.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:30 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 621.118562 secs 2024-07-26T07:06:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:31 smithi164 ceph-mon[36970]: pgmap v933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:06:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 638.861616 secs 2024-07-26T07:06:32.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:31 smithi163 ceph-mon[36873]: pgmap v933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:06:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 638.861616 secs 2024-07-26T07:06:32.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:31 smithi042 ceph-mon[32775]: pgmap v933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:06:32.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 638.861616 secs 2024-07-26T07:06:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:33 smithi164 ceph-mon[36970]: pgmap v934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:06:34.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:33 smithi163 ceph-mon[36873]: pgmap v934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:06:34.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:33 smithi042 ceph-mon[32775]: pgmap v934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:06:36.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:35 smithi164 ceph-mon[36970]: pgmap v935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:36.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:35 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 626.118658 secs 2024-07-26T07:06:36.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:35 smithi163 ceph-mon[36873]: pgmap v935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:36.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:35 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 626.118658 secs 2024-07-26T07:06:36.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:35 smithi042 ceph-mon[32775]: pgmap v935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:35 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 626.118658 secs 2024-07-26T07:06:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 643.861699 secs 2024-07-26T07:06:37.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 643.861699 secs 2024-07-26T07:06:37.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 643.861699 secs 2024-07-26T07:06:38.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:37 smithi164 ceph-mon[36970]: pgmap v936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:38.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:37 smithi163 ceph-mon[36873]: pgmap v936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:38.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:37 smithi042 ceph-mon[32775]: pgmap v936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:38.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:06:38.473+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:06:38.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:06:38.473+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:06:38.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:06:38.473+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:06:40.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:39 smithi164 ceph-mon[36970]: pgmap v937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:40.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:39 smithi163 ceph-mon[36873]: pgmap v937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:39 smithi042 ceph-mon[32775]: pgmap v937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:41.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:40 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 631.118803 secs 2024-07-26T07:06:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:40 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 631.118803 secs 2024-07-26T07:06:41.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:40 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 631.118803 secs 2024-07-26T07:06:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:41 smithi164 ceph-mon[36970]: pgmap v938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 648.861778 secs 2024-07-26T07:06:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:41 smithi163 ceph-mon[36873]: pgmap v938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 648.861778 secs 2024-07-26T07:06:42.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:41 smithi042 ceph-mon[32775]: pgmap v938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:42.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 648.861778 secs 2024-07-26T07:06:44.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:43 smithi164 ceph-mon[36970]: pgmap v939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:44.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:43 smithi163 ceph-mon[36873]: pgmap v939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:44.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:43 smithi042 ceph-mon[32775]: pgmap v939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:46.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:45 smithi164 ceph-mon[36970]: pgmap v940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:46.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:45 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 636.118935 secs 2024-07-26T07:06:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:45 smithi163 ceph-mon[36873]: pgmap v940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:45 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 636.118935 secs 2024-07-26T07:06:46.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:45 smithi042 ceph-mon[32775]: pgmap v940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:45 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 636.118935 secs 2024-07-26T07:06:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 653.861920 secs 2024-07-26T07:06:47.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 653.861920 secs 2024-07-26T07:06:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 653.861920 secs 2024-07-26T07:06:48.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:47 smithi164 ceph-mon[36970]: pgmap v941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:48.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:47 smithi163 ceph-mon[36873]: pgmap v941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:48.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:47 smithi042 ceph-mon[32775]: pgmap v941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:50.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:49 smithi163 ceph-mon[36873]: pgmap v942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:50.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:49 smithi042 ceph-mon[32775]: pgmap v942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:50.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:49 smithi164 ceph-mon[36970]: pgmap v942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:50 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 641.119061 secs 2024-07-26T07:06:51.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:50 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 641.119061 secs 2024-07-26T07:06:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:50 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 641.119061 secs 2024-07-26T07:06:52.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:51 smithi163 ceph-mon[36873]: pgmap v943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 658.862046 secs 2024-07-26T07:06:52.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:51 smithi042 ceph-mon[32775]: pgmap v943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 658.862046 secs 2024-07-26T07:06:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:51 smithi164 ceph-mon[36970]: pgmap v943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 658.862046 secs 2024-07-26T07:06:53.476 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:06:53.476+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:06:53.476 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:06:53.476+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:06:53.476 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:06:53.476+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:06:54.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:53 smithi163 ceph-mon[36873]: pgmap v944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:54.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:53 smithi042 ceph-mon[32775]: pgmap v944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:54.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:53 smithi164 ceph-mon[36970]: pgmap v944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:56.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:55 smithi163 ceph-mon[36873]: pgmap v945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:56.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:55 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 646.119182 secs 2024-07-26T07:06:56.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:55 smithi042 ceph-mon[32775]: pgmap v945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:55 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 646.119182 secs 2024-07-26T07:06:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:55 smithi164 ceph-mon[36970]: pgmap v945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:06:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:55 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 646.119182 secs 2024-07-26T07:06:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 663.862189 secs 2024-07-26T07:06:57.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 663.862189 secs 2024-07-26T07:06:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 663.862189 secs 2024-07-26T07:06:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:06:57 smithi163 ceph-mon[36873]: pgmap v946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:58.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:06:57 smithi042 ceph-mon[32775]: pgmap v946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:06:58.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:06:57 smithi164 ceph-mon[36970]: pgmap v946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:00.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:00 smithi163 ceph-mon[36873]: pgmap v947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:00.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:00 smithi042 ceph-mon[32775]: pgmap v947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:00.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:00 smithi164 ceph-mon[36970]: pgmap v947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 651.119309 secs 2024-07-26T07:07:01.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 651.119309 secs 2024-07-26T07:07:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 651.119309 secs 2024-07-26T07:07:02.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:02 smithi042 ceph-mon[32775]: pgmap v948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:02.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 668.862257 secs 2024-07-26T07:07:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:02 smithi164 ceph-mon[36970]: pgmap v948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 668.862257 secs 2024-07-26T07:07:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:02 smithi163 ceph-mon[36873]: pgmap v948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 668.862257 secs 2024-07-26T07:07:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:04 smithi042 ceph-mon[32775]: pgmap v949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:04.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:04 smithi164 ceph-mon[36970]: pgmap v949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:04.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:04 smithi163 ceph-mon[36873]: pgmap v949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:06 smithi042 ceph-mon[32775]: pgmap v950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 656.119430 secs 2024-07-26T07:07:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:06 smithi164 ceph-mon[36970]: pgmap v950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 656.119430 secs 2024-07-26T07:07:06.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:06 smithi163 ceph-mon[36873]: pgmap v950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 656.119430 secs 2024-07-26T07:07:07.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 673.862398 secs 2024-07-26T07:07:07.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 673.862398 secs 2024-07-26T07:07:07.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 673.862398 secs 2024-07-26T07:07:08.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:08 smithi042 ceph-mon[32775]: pgmap v951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:08 smithi164 ceph-mon[36970]: pgmap v951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:08.483 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:07:08.482+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:07:08.484 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:07:08.482+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:07:08.484 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:07:08.482+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:07:08.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:08 smithi163 ceph-mon[36873]: pgmap v951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:10.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:10 smithi042 ceph-mon[32775]: pgmap v952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:10.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:10 smithi164 ceph-mon[36970]: pgmap v952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:10.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:10 smithi163 ceph-mon[36873]: pgmap v952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:11.373 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 661.119557 secs 2024-07-26T07:07:11.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 661.119557 secs 2024-07-26T07:07:11.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 661.119557 secs 2024-07-26T07:07:12.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:12 smithi042 ceph-mon[32775]: pgmap v953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:12.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 678.862539 secs 2024-07-26T07:07:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:12 smithi164 ceph-mon[36970]: pgmap v953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 678.862539 secs 2024-07-26T07:07:12.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:12 smithi163 ceph-mon[36873]: pgmap v953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 678.862539 secs 2024-07-26T07:07:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:14 smithi042 ceph-mon[32775]: pgmap v954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:14.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:14 smithi164 ceph-mon[36970]: pgmap v954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:14.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:14 smithi163 ceph-mon[36873]: pgmap v954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:16 smithi042 ceph-mon[32775]: pgmap v955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 666.119680 secs 2024-07-26T07:07:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:16 smithi164 ceph-mon[36970]: pgmap v955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 666.119680 secs 2024-07-26T07:07:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:16 smithi163 ceph-mon[36873]: pgmap v955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 666.119680 secs 2024-07-26T07:07:17.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 683.862679 secs 2024-07-26T07:07:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 683.862679 secs 2024-07-26T07:07:17.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 683.862679 secs 2024-07-26T07:07:18.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:18 smithi042 ceph-mon[32775]: pgmap v956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:18.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:18 smithi164 ceph-mon[36970]: pgmap v956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:18 smithi163 ceph-mon[36873]: pgmap v956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:20.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:20 smithi042 ceph-mon[32775]: pgmap v957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:20 smithi164 ceph-mon[36970]: pgmap v957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:20 smithi163 ceph-mon[36873]: pgmap v957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:21.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 671.119812 secs 2024-07-26T07:07:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 671.119812 secs 2024-07-26T07:07:21.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 671.119812 secs 2024-07-26T07:07:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:22 smithi042 ceph-mon[32775]: pgmap v958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 688.862822 secs 2024-07-26T07:07:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:22 smithi164 ceph-mon[36970]: pgmap v958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 688.862822 secs 2024-07-26T07:07:22.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:22 smithi163 ceph-mon[36873]: pgmap v958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:22.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 688.862822 secs 2024-07-26T07:07:23.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:07:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:07:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:07:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:07:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:07:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:07:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:07:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:07:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:07:23.485+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:07:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:07:23.485+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:07:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:07:23.485+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:07:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:07:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:07:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:07:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:07:24.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:24 smithi042 ceph-mon[32775]: pgmap v959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:24 smithi042 ceph-mon[32775]: pgmap v960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:07:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:24 smithi042 ceph-mon[32775]: pgmap v961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:07:24.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:24 smithi164 ceph-mon[36970]: pgmap v959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:24.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:24 smithi164 ceph-mon[36970]: pgmap v960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:07:24.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:24 smithi164 ceph-mon[36970]: pgmap v961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:07:24.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:24 smithi163 ceph-mon[36873]: pgmap v959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:24 smithi163 ceph-mon[36873]: pgmap v960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:07:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:24 smithi163 ceph-mon[36873]: pgmap v961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:07:26.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:26 smithi042 ceph-mon[32775]: pgmap v962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:07:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 676.119937 secs 2024-07-26T07:07:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:26 smithi164 ceph-mon[36970]: pgmap v962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:07:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 676.119937 secs 2024-07-26T07:07:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:26 smithi163 ceph-mon[36873]: pgmap v962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:07:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 676.119937 secs 2024-07-26T07:07:27.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 693.862978 secs 2024-07-26T07:07:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 693.862978 secs 2024-07-26T07:07:27.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 693.862978 secs 2024-07-26T07:07:28.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:28 smithi042 ceph-mon[32775]: pgmap v963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:07:28.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:28 smithi164 ceph-mon[36970]: pgmap v963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:07:28.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:28 smithi163 ceph-mon[36873]: pgmap v963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:07:30.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:30 smithi042 ceph-mon[32775]: pgmap v964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:07:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:30 smithi164 ceph-mon[36970]: pgmap v964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:07:30.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:30 smithi163 ceph-mon[36873]: pgmap v964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:07:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 681.120038 secs 2024-07-26T07:07:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 681.120038 secs 2024-07-26T07:07:31.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 681.120038 secs 2024-07-26T07:07:32.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:32 smithi042 ceph-mon[32775]: pgmap v965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:07:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 698.863122 secs 2024-07-26T07:07:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:32 smithi164 ceph-mon[36970]: pgmap v965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:07:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 698.863122 secs 2024-07-26T07:07:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:32 smithi163 ceph-mon[36873]: pgmap v965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:07:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 698.863122 secs 2024-07-26T07:07:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:34 smithi042 ceph-mon[32775]: pgmap v966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:07:34.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:34 smithi164 ceph-mon[36970]: pgmap v966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:07:34.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:34 smithi163 ceph-mon[36873]: pgmap v966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:07:36.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:36 smithi042 ceph-mon[32775]: pgmap v967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 686.120160 secs 2024-07-26T07:07:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:36 smithi164 ceph-mon[36970]: pgmap v967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 686.120160 secs 2024-07-26T07:07:36.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:36 smithi163 ceph-mon[36873]: pgmap v967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 686.120160 secs 2024-07-26T07:07:37.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 703.863186 secs 2024-07-26T07:07:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 703.863186 secs 2024-07-26T07:07:37.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 703.863186 secs 2024-07-26T07:07:38.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:38 smithi042 ceph-mon[32775]: pgmap v968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:38 smithi164 ceph-mon[36970]: pgmap v968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:07:38.493+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:07:38.494 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:07:38.493+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:07:38.494 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:07:38.493+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:07:38.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:38 smithi163 ceph-mon[36873]: pgmap v968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:40.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:40 smithi042 ceph-mon[32775]: pgmap v969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:07:40.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:40 smithi164 ceph-mon[36970]: pgmap v969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:07:40.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:40 smithi163 ceph-mon[36873]: pgmap v969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:07:41.373 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 691.120285 secs 2024-07-26T07:07:41.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 691.120285 secs 2024-07-26T07:07:41.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 691.120285 secs 2024-07-26T07:07:42.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:42 smithi042 ceph-mon[32775]: pgmap v970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:42.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 708.863260 secs 2024-07-26T07:07:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:42 smithi164 ceph-mon[36970]: pgmap v970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 708.863260 secs 2024-07-26T07:07:42.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:42 smithi163 ceph-mon[36873]: pgmap v970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 708.863260 secs 2024-07-26T07:07:44.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:44 smithi042 ceph-mon[32775]: pgmap v971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:44.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:44 smithi164 ceph-mon[36970]: pgmap v971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:44.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:44 smithi163 ceph-mon[36873]: pgmap v971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:46.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:46 smithi042 ceph-mon[32775]: pgmap v972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 696.120375 secs 2024-07-26T07:07:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:46 smithi164 ceph-mon[36970]: pgmap v972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 696.120375 secs 2024-07-26T07:07:46.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:46 smithi163 ceph-mon[36873]: pgmap v972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 696.120375 secs 2024-07-26T07:07:47.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 713.863329 secs 2024-07-26T07:07:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 713.863329 secs 2024-07-26T07:07:47.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 713.863329 secs 2024-07-26T07:07:48.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:48 smithi042 ceph-mon[32775]: pgmap v973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:48.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:48 smithi164 ceph-mon[36970]: pgmap v973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:48.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:48 smithi163 ceph-mon[36873]: pgmap v973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:50.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:50 smithi042 ceph-mon[32775]: pgmap v974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:07:50.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:50 smithi164 ceph-mon[36970]: pgmap v974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:07:50.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:50 smithi163 ceph-mon[36873]: pgmap v974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:07:51.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 701.120501 secs 2024-07-26T07:07:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 701.120501 secs 2024-07-26T07:07:51.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 701.120501 secs 2024-07-26T07:07:52.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:52 smithi042 ceph-mon[32775]: pgmap v975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:52.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 718.863391 secs 2024-07-26T07:07:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:52 smithi164 ceph-mon[36970]: pgmap v975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 718.863391 secs 2024-07-26T07:07:52.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:52 smithi163 ceph-mon[36873]: pgmap v975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:52.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 718.863391 secs 2024-07-26T07:07:53.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:07:53.496+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:07:53.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:07:53.496+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:07:53.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:07:53.496+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:07:54.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:54 smithi164 ceph-mon[36970]: pgmap v976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:54.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:54 smithi163 ceph-mon[36873]: pgmap v976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:54.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:54 smithi042 ceph-mon[32775]: pgmap v976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:56 smithi164 ceph-mon[36970]: pgmap v977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 706.120593 secs 2024-07-26T07:07:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:56 smithi163 ceph-mon[36873]: pgmap v977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 706.120593 secs 2024-07-26T07:07:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:56 smithi042 ceph-mon[32775]: pgmap v977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:07:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 706.120593 secs 2024-07-26T07:07:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:57 smithi164 ceph-mon[36970]: 1 slow requests, 1 included below; oldest blocked for > 723.863463 secs 2024-07-26T07:07:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:57 smithi164 ceph-mon[36970]: slow request 723.863462 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T07:07:57.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:57 smithi163 ceph-mon[36873]: 1 slow requests, 1 included below; oldest blocked for > 723.863463 secs 2024-07-26T07:07:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:57 smithi163 ceph-mon[36873]: slow request 723.863462 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T07:07:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:57 smithi042 ceph-mon[32775]: 1 slow requests, 1 included below; oldest blocked for > 723.863463 secs 2024-07-26T07:07:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:57 smithi042 ceph-mon[32775]: slow request 723.863462 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T07:07:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:07:57.968+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006070 con 0x7ff680102000 2024-07-26T07:07:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:07:57.968+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650006020 con 0x55db6e0564d0 2024-07-26T07:07:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:07:57.969+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006070 con 0x7ff680102000 2024-07-26T07:07:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:07:57.969+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650006020 con 0x55db6e0564d0 2024-07-26T07:07:58.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:07:58 smithi164 ceph-mon[36970]: pgmap v978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:58.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:07:58 smithi163 ceph-mon[36873]: pgmap v978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:07:58.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:07:58 smithi042 ceph-mon[32775]: pgmap v978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:00.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:00 smithi164 ceph-mon[36970]: pgmap v979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:08:00.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:00 smithi163 ceph-mon[36873]: pgmap v979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:08:00.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:00 smithi042 ceph-mon[32775]: pgmap v979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:08:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 711.120741 secs 2024-07-26T07:08:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 728.863532 secs 2024-07-26T07:08:01.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 711.120741 secs 2024-07-26T07:08:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 728.863532 secs 2024-07-26T07:08:01.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 711.120741 secs 2024-07-26T07:08:01.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 728.863532 secs 2024-07-26T07:08:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:02 smithi164 ceph-mon[36970]: pgmap v980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:02.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:02 smithi163 ceph-mon[36873]: pgmap v980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:02.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:02 smithi042 ceph-mon[32775]: pgmap v980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:04.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:04 smithi164 ceph-mon[36970]: pgmap v981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:04.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:04 smithi163 ceph-mon[36873]: pgmap v981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:04.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:04 smithi042 ceph-mon[32775]: pgmap v981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:06 smithi164 ceph-mon[36970]: pgmap v982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 716.120880 secs 2024-07-26T07:08:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 733.863652 secs 2024-07-26T07:08:06.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:06 smithi163 ceph-mon[36873]: pgmap v982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 716.120880 secs 2024-07-26T07:08:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 733.863652 secs 2024-07-26T07:08:06.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:06 smithi042 ceph-mon[32775]: pgmap v982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 716.120880 secs 2024-07-26T07:08:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 733.863652 secs 2024-07-26T07:08:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:08 smithi164 ceph-mon[36970]: pgmap v983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:08.504 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:08:08.504+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:08:08.505 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:08:08.504+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:08:08.505 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:08:08.504+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:08:08.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:08 smithi163 ceph-mon[36873]: pgmap v983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:08.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:08 smithi042 ceph-mon[32775]: pgmap v983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:10.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:10 smithi164 ceph-mon[36970]: pgmap v984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:08:10.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:10 smithi163 ceph-mon[36873]: pgmap v984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:08:10.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:10 smithi042 ceph-mon[32775]: pgmap v984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:08:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:11 smithi164 ceph-mon[36970]: 1 slow requests, 1 included below; oldest blocked for > 721.121026 secs 2024-07-26T07:08:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:11 smithi164 ceph-mon[36970]: slow request 721.121025 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T07:08:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 738.863767 secs 2024-07-26T07:08:11.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:11 smithi163 ceph-mon[36873]: 1 slow requests, 1 included below; oldest blocked for > 721.121026 secs 2024-07-26T07:08:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:11 smithi163 ceph-mon[36873]: slow request 721.121025 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T07:08:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 738.863767 secs 2024-07-26T07:08:11.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:11 smithi042 ceph-mon[32775]: 1 slow requests, 1 included below; oldest blocked for > 721.121026 secs 2024-07-26T07:08:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:11 smithi042 ceph-mon[32775]: slow request 721.121025 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T07:08:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 738.863767 secs 2024-07-26T07:08:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:12 smithi164 ceph-mon[36970]: pgmap v985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:12.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:12 smithi163 ceph-mon[36873]: pgmap v985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:12.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:12 smithi042 ceph-mon[32775]: pgmap v985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:14.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:14 smithi164 ceph-mon[36970]: pgmap v986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:14.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:14 smithi163 ceph-mon[36873]: pgmap v986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:14.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:14 smithi042 ceph-mon[32775]: pgmap v986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:16 smithi164 ceph-mon[36970]: pgmap v987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 726.121167 secs 2024-07-26T07:08:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 743.863885 secs 2024-07-26T07:08:16.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:16 smithi163 ceph-mon[36873]: pgmap v987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 726.121167 secs 2024-07-26T07:08:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 743.863885 secs 2024-07-26T07:08:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:16 smithi042 ceph-mon[32775]: pgmap v987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 726.121167 secs 2024-07-26T07:08:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 743.863885 secs 2024-07-26T07:08:18.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:18 smithi164 ceph-mon[36970]: pgmap v988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:18.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:18 smithi163 ceph-mon[36873]: pgmap v988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:18.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:18 smithi042 ceph-mon[32775]: pgmap v988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:20 smithi164 ceph-mon[36970]: pgmap v989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:08:20.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:20 smithi163 ceph-mon[36873]: pgmap v989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:08:20.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:20 smithi042 ceph-mon[32775]: pgmap v989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:08:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 731.121352 secs 2024-07-26T07:08:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 748.863998 secs 2024-07-26T07:08:21.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 731.121352 secs 2024-07-26T07:08:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 748.863998 secs 2024-07-26T07:08:21.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 731.121352 secs 2024-07-26T07:08:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 748.863998 secs 2024-07-26T07:08:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:22 smithi164 ceph-mon[36970]: pgmap v990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:08:22.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:22 smithi163 ceph-mon[36873]: pgmap v990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:08:22.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:22 smithi042 ceph-mon[32775]: pgmap v990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:08:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:08:23.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:08:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:08:23.507+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:08:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:08:23.507+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:08:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:08:23.507+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:08:23.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:08:24.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:24 smithi164 ceph-mon[36970]: pgmap v991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:08:24.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:08:24.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:08:24.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:08:24.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:24 smithi163 ceph-mon[36873]: pgmap v991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:08:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:08:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:08:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:08:24.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:24 smithi042 ceph-mon[32775]: pgmap v991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:08:24.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:08:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:08:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:08:25.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:25 smithi164 ceph-mon[36970]: pgmap v992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:08:25.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:25 smithi164 ceph-mon[36970]: pgmap v993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T07:08:25.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:25 smithi163 ceph-mon[36873]: pgmap v992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:08:25.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:25 smithi163 ceph-mon[36873]: pgmap v993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T07:08:25.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:25 smithi042 ceph-mon[32775]: pgmap v992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:08:25.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:25 smithi042 ceph-mon[32775]: pgmap v993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T07:08:26.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 736.121459 secs 2024-07-26T07:08:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 753.864115 secs 2024-07-26T07:08:26.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 736.121459 secs 2024-07-26T07:08:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 753.864115 secs 2024-07-26T07:08:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 736.121459 secs 2024-07-26T07:08:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 753.864115 secs 2024-07-26T07:08:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:27 smithi163 ceph-mon[36873]: pgmap v994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:08:27.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:27 smithi042 ceph-mon[32775]: pgmap v994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:08:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:27 smithi164 ceph-mon[36970]: pgmap v994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:08:29.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:29 smithi163 ceph-mon[36873]: pgmap v995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T07:08:29.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:29 smithi042 ceph-mon[32775]: pgmap v995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T07:08:29.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:29 smithi164 ceph-mon[36970]: pgmap v995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T07:08:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:31 smithi163 ceph-mon[36873]: pgmap v996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:08:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 741.121583 secs 2024-07-26T07:08:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 758.864184 secs 2024-07-26T07:08:31.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:31 smithi042 ceph-mon[32775]: pgmap v996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:08:31.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 741.121583 secs 2024-07-26T07:08:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 758.864184 secs 2024-07-26T07:08:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:31 smithi164 ceph-mon[36970]: pgmap v996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:08:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 741.121583 secs 2024-07-26T07:08:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 758.864184 secs 2024-07-26T07:08:33.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:33 smithi042 ceph-mon[32775]: pgmap v997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:08:33.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:33 smithi163 ceph-mon[36873]: pgmap v997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:08:33.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:33 smithi164 ceph-mon[36970]: pgmap v997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:08:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:35 smithi163 ceph-mon[36873]: pgmap v998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:08:35.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:35 smithi042 ceph-mon[32775]: pgmap v998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:08:35.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:35 smithi164 ceph-mon[36970]: pgmap v998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:08:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 746.121653 secs 2024-07-26T07:08:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 763.864280 secs 2024-07-26T07:08:36.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 746.121653 secs 2024-07-26T07:08:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 763.864280 secs 2024-07-26T07:08:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 746.121653 secs 2024-07-26T07:08:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 763.864280 secs 2024-07-26T07:08:37.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:37 smithi163 ceph-mon[36873]: pgmap v999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:37.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:37 smithi042 ceph-mon[32775]: pgmap v999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:37 smithi164 ceph-mon[36970]: pgmap v999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:08:38.513+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:08:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:08:38.513+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:08:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:08:38.513+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:08:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:39 smithi163 ceph-mon[36873]: pgmap v1000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:39.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:39 smithi042 ceph-mon[32775]: pgmap v1000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:39 smithi164 ceph-mon[36970]: pgmap v1000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:41.524 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:41 smithi164 ceph-mon[36970]: pgmap v1001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:41.524 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 751.121729 secs 2024-07-26T07:08:41.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 768.864376 secs 2024-07-26T07:08:41.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:41 smithi042 ceph-mon[32775]: pgmap v1001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:41.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 751.121729 secs 2024-07-26T07:08:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 768.864376 secs 2024-07-26T07:08:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:41 smithi163 ceph-mon[36873]: pgmap v1001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 751.121729 secs 2024-07-26T07:08:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 768.864376 secs 2024-07-26T07:08:43.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:43 smithi042 ceph-mon[32775]: pgmap v1002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:43 smithi164 ceph-mon[36970]: pgmap v1002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:43 smithi163 ceph-mon[36873]: pgmap v1002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:45 smithi042 ceph-mon[32775]: pgmap v1003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:08:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:45 smithi164 ceph-mon[36970]: pgmap v1003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:08:45.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:45 smithi163 ceph-mon[36873]: pgmap v1003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:08:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 756.121826 secs 2024-07-26T07:08:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 773.864473 secs 2024-07-26T07:08:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 756.121826 secs 2024-07-26T07:08:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 773.864473 secs 2024-07-26T07:08:46.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 756.121826 secs 2024-07-26T07:08:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 773.864473 secs 2024-07-26T07:08:47.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:47 smithi042 ceph-mon[32775]: pgmap v1004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:47 smithi164 ceph-mon[36970]: pgmap v1004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:47.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:47 smithi163 ceph-mon[36873]: pgmap v1004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:49.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:49 smithi042 ceph-mon[32775]: pgmap v1005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:49 smithi164 ceph-mon[36970]: pgmap v1005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:49.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:49 smithi163 ceph-mon[36873]: pgmap v1005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:51 smithi042 ceph-mon[32775]: pgmap v1006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 761.121957 secs 2024-07-26T07:08:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 778.864590 secs 2024-07-26T07:08:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:51 smithi164 ceph-mon[36970]: pgmap v1006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 761.121957 secs 2024-07-26T07:08:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 778.864590 secs 2024-07-26T07:08:51.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:51 smithi163 ceph-mon[36873]: pgmap v1006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 761.121957 secs 2024-07-26T07:08:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 778.864590 secs 2024-07-26T07:08:52.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:52 smithi042 ceph-mon[32775]: pgmap v1007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:52 smithi164 ceph-mon[36970]: pgmap v1007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:52.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:52 smithi163 ceph-mon[36873]: pgmap v1007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:53.515 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:08:53.516+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:08:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:08:53.516+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:08:53.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:08:53.516+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:08:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:54 smithi164 ceph-mon[36970]: pgmap v1008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:08:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:54 smithi163 ceph-mon[36873]: pgmap v1008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:08:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:54 smithi042 ceph-mon[32775]: pgmap v1008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:08:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:56 smithi164 ceph-mon[36970]: pgmap v1009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 766.122076 secs 2024-07-26T07:08:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 783.864711 secs 2024-07-26T07:08:56.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:56 smithi163 ceph-mon[36873]: pgmap v1009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 766.122076 secs 2024-07-26T07:08:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 783.864711 secs 2024-07-26T07:08:56.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:56 smithi042 ceph-mon[32775]: pgmap v1009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:08:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 766.122076 secs 2024-07-26T07:08:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 783.864711 secs 2024-07-26T07:08:58.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:08:58 smithi164 ceph-mon[36970]: pgmap v1010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:58.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:08:58 smithi163 ceph-mon[36873]: pgmap v1010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:08:58.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:08:58 smithi042 ceph-mon[32775]: pgmap v1010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:00 smithi164 ceph-mon[36970]: pgmap v1011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:00 smithi163 ceph-mon[36873]: pgmap v1011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:00.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:00 smithi042 ceph-mon[32775]: pgmap v1011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:01.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 771.122210 secs 2024-07-26T07:09:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 788.864839 secs 2024-07-26T07:09:01.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 771.122210 secs 2024-07-26T07:09:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 788.864839 secs 2024-07-26T07:09:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 771.122210 secs 2024-07-26T07:09:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 788.864839 secs 2024-07-26T07:09:02.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:02 smithi163 ceph-mon[36873]: pgmap v1012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:02.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:02 smithi042 ceph-mon[32775]: pgmap v1012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:02 smithi164 ceph-mon[36970]: pgmap v1012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:04.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:04 smithi163 ceph-mon[36873]: pgmap v1013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:09:04.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:04 smithi042 ceph-mon[32775]: pgmap v1013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:09:04.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:04 smithi164 ceph-mon[36970]: pgmap v1013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:09:06.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:06 smithi163 ceph-mon[36873]: pgmap v1014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 776.122317 secs 2024-07-26T07:09:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 793.864967 secs 2024-07-26T07:09:06.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:06 smithi042 ceph-mon[32775]: pgmap v1014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 776.122317 secs 2024-07-26T07:09:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 793.864967 secs 2024-07-26T07:09:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:06 smithi164 ceph-mon[36970]: pgmap v1014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 776.122317 secs 2024-07-26T07:09:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 793.864967 secs 2024-07-26T07:09:08.524 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:09:08.524+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:09:08.524 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:09:08.524+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:09:08.524 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:09:08.524+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:09:08.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:08 smithi163 ceph-mon[36873]: pgmap v1015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:08.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:08 smithi042 ceph-mon[32775]: pgmap v1015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:08.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:08 smithi164 ceph-mon[36970]: pgmap v1015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:10 smithi163 ceph-mon[36873]: pgmap v1016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:10.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:10 smithi042 ceph-mon[32775]: pgmap v1016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:10 smithi164 ceph-mon[36970]: pgmap v1016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 781.122435 secs 2024-07-26T07:09:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 798.865102 secs 2024-07-26T07:09:11.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 781.122435 secs 2024-07-26T07:09:11.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 798.865102 secs 2024-07-26T07:09:11.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 781.122435 secs 2024-07-26T07:09:11.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 798.865102 secs 2024-07-26T07:09:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:12 smithi163 ceph-mon[36873]: pgmap v1017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:12.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:12 smithi042 ceph-mon[32775]: pgmap v1017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:12 smithi164 ceph-mon[36970]: pgmap v1017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:14 smithi163 ceph-mon[36873]: pgmap v1018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:09:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:14 smithi042 ceph-mon[32775]: pgmap v1018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:09:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:14 smithi164 ceph-mon[36970]: pgmap v1018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:09:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:16 smithi163 ceph-mon[36873]: pgmap v1019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 786.122497 secs 2024-07-26T07:09:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 803.865233 secs 2024-07-26T07:09:16.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:16 smithi042 ceph-mon[32775]: pgmap v1019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 786.122497 secs 2024-07-26T07:09:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 803.865233 secs 2024-07-26T07:09:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:16 smithi164 ceph-mon[36970]: pgmap v1019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 786.122497 secs 2024-07-26T07:09:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 803.865233 secs 2024-07-26T07:09:18.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:18 smithi042 ceph-mon[32775]: pgmap v1020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:18 smithi164 ceph-mon[36970]: pgmap v1020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:19.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:18 smithi163 ceph-mon[36873]: pgmap v1020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:20.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:20 smithi042 ceph-mon[32775]: pgmap v1021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:20 smithi164 ceph-mon[36970]: pgmap v1021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:21.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:20 smithi163 ceph-mon[36873]: pgmap v1021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:21.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 791.122624 secs 2024-07-26T07:09:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 808.865335 secs 2024-07-26T07:09:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 791.122624 secs 2024-07-26T07:09:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 808.865335 secs 2024-07-26T07:09:22.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 791.122624 secs 2024-07-26T07:09:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 808.865335 secs 2024-07-26T07:09:22.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:22 smithi042 ceph-mon[32775]: pgmap v1022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:22 smithi164 ceph-mon[36970]: pgmap v1022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:23.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:22 smithi163 ceph-mon[36873]: pgmap v1022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:09:23.527+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:09:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:09:23.527+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:09:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:09:23.527+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:09:23.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:09:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:09:24.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:09:24.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:24 smithi042 ceph-mon[32775]: pgmap v1023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:09:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:24 smithi164 ceph-mon[36970]: pgmap v1023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:09:25.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:24 smithi163 ceph-mon[36873]: pgmap v1023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:09:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 796.122740 secs 2024-07-26T07:09:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 813.865453 secs 2024-07-26T07:09:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:09:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:09:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 796.122740 secs 2024-07-26T07:09:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 813.865453 secs 2024-07-26T07:09:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:09:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:09:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 796.122740 secs 2024-07-26T07:09:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 813.865453 secs 2024-07-26T07:09:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:09:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:09:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:09:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:27 smithi164 ceph-mon[36970]: pgmap v1024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:27 smithi164 ceph-mon[36970]: pgmap v1025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:09:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:27 smithi164 ceph-mon[36970]: pgmap v1026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:09:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:27 smithi164 ceph-mon[36970]: Health check update: 6 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T07:09:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:27 smithi163 ceph-mon[36873]: pgmap v1024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:27 smithi163 ceph-mon[36873]: pgmap v1025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:09:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:27 smithi163 ceph-mon[36873]: pgmap v1026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:09:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:27 smithi163 ceph-mon[36873]: Health check update: 6 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T07:09:27.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:27 smithi042 ceph-mon[32775]: pgmap v1024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:27 smithi042 ceph-mon[32775]: pgmap v1025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:09:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:27 smithi042 ceph-mon[32775]: pgmap v1026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:09:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:27 smithi042 ceph-mon[32775]: Health check update: 6 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T07:09:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:29 smithi164 ceph-mon[36970]: pgmap v1027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:09:29.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:29 smithi163 ceph-mon[36873]: pgmap v1027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:09:29.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:29 smithi042 ceph-mon[32775]: pgmap v1027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:09:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:31 smithi164 ceph-mon[36970]: pgmap v1028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:09:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 801.122866 secs 2024-07-26T07:09:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 818.865575 secs 2024-07-26T07:09:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:31 smithi163 ceph-mon[36873]: pgmap v1028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:09:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 801.122866 secs 2024-07-26T07:09:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 818.865575 secs 2024-07-26T07:09:31.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:31 smithi042 ceph-mon[32775]: pgmap v1028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:09:31.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 801.122866 secs 2024-07-26T07:09:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 818.865575 secs 2024-07-26T07:09:33.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:33 smithi164 ceph-mon[36970]: pgmap v1029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:09:33.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:33 smithi163 ceph-mon[36873]: pgmap v1029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:09:33.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:33 smithi042 ceph-mon[32775]: pgmap v1029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:09:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:35 smithi042 ceph-mon[32775]: pgmap v1030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:09:35.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:35 smithi164 ceph-mon[36970]: pgmap v1030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:09:35.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:35 smithi163 ceph-mon[36873]: pgmap v1030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:09:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 806.123007 secs 2024-07-26T07:09:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 823.865699 secs 2024-07-26T07:09:36.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 806.123007 secs 2024-07-26T07:09:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 823.865699 secs 2024-07-26T07:09:36.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 806.123007 secs 2024-07-26T07:09:36.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 823.865699 secs 2024-07-26T07:09:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:37 smithi164 ceph-mon[36970]: pgmap v1031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:09:37.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:37 smithi163 ceph-mon[36873]: pgmap v1031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:09:37.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:37 smithi042 ceph-mon[32775]: pgmap v1031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:09:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:09:38.536+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:09:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:09:38.536+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:09:38.537 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:09:38.536+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:09:39.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:39 smithi164 ceph-mon[36970]: pgmap v1032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:39 smithi163 ceph-mon[36873]: pgmap v1032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:39 smithi042 ceph-mon[32775]: pgmap v1032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:41 smithi164 ceph-mon[36970]: pgmap v1033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 811.123129 secs 2024-07-26T07:09:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 828.865821 secs 2024-07-26T07:09:41.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:41 smithi163 ceph-mon[36873]: pgmap v1033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:41.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 811.123129 secs 2024-07-26T07:09:41.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 828.865821 secs 2024-07-26T07:09:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:41 smithi042 ceph-mon[32775]: pgmap v1033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 811.123129 secs 2024-07-26T07:09:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 828.865821 secs 2024-07-26T07:09:43.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:43 smithi164 ceph-mon[36970]: pgmap v1034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:43.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:43 smithi163 ceph-mon[36873]: pgmap v1034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:43.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:43 smithi042 ceph-mon[32775]: pgmap v1034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:45.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:45 smithi164 ceph-mon[36970]: pgmap v1035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:45.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:45 smithi163 ceph-mon[36873]: pgmap v1035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:45.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:45 smithi042 ceph-mon[32775]: pgmap v1035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 816.123249 secs 2024-07-26T07:09:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 833.865919 secs 2024-07-26T07:09:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 816.123249 secs 2024-07-26T07:09:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 833.865919 secs 2024-07-26T07:09:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 816.123249 secs 2024-07-26T07:09:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 833.865919 secs 2024-07-26T07:09:47.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:47 smithi042 ceph-mon[32775]: pgmap v1036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:47 smithi164 ceph-mon[36970]: pgmap v1036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:47.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:47 smithi163 ceph-mon[36873]: pgmap v1036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:49 smithi164 ceph-mon[36970]: pgmap v1037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:49.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:49 smithi163 ceph-mon[36873]: pgmap v1037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:49.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:49 smithi042 ceph-mon[32775]: pgmap v1037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:51 smithi163 ceph-mon[36873]: pgmap v1038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 821.123392 secs 2024-07-26T07:09:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 838.866020 secs 2024-07-26T07:09:51.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:51 smithi042 ceph-mon[32775]: pgmap v1038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:51.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 821.123392 secs 2024-07-26T07:09:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 838.866020 secs 2024-07-26T07:09:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:51 smithi164 ceph-mon[36970]: pgmap v1038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 821.123392 secs 2024-07-26T07:09:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 838.866020 secs 2024-07-26T07:09:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:53 smithi163 ceph-mon[36873]: pgmap v1039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:09:53.539+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:09:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:09:53.539+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:09:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:09:53.539+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:09:53.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:53 smithi042 ceph-mon[32775]: pgmap v1039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:53 smithi164 ceph-mon[36970]: pgmap v1039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:55 smithi163 ceph-mon[36873]: pgmap v1040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:55.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:55 smithi042 ceph-mon[32775]: pgmap v1040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:55 smithi164 ceph-mon[36970]: pgmap v1040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:09:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 826.123509 secs 2024-07-26T07:09:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 843.866145 secs 2024-07-26T07:09:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 826.123509 secs 2024-07-26T07:09:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 843.866145 secs 2024-07-26T07:09:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 826.123509 secs 2024-07-26T07:09:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 843.866145 secs 2024-07-26T07:09:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:57 smithi163 ceph-mon[36873]: pgmap v1041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:57.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:57 smithi042 ceph-mon[32775]: pgmap v1041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:57 smithi164 ceph-mon[36970]: pgmap v1041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:09:59 smithi163 ceph-mon[36873]: pgmap v1042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:59.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:09:59 smithi042 ceph-mon[32775]: pgmap v1042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:09:59.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:09:59 smithi164 ceph-mon[36970]: pgmap v1042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:00 smithi163 ceph-mon[36873]: Health detail: HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:10:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:00 smithi163 ceph-mon[36873]: [WRN] CEPHADM_FAILED_DAEMON: 6 failed cephadm daemon(s) 2024-07-26T07:10:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:00 smithi163 ceph-mon[36873]: daemon mds.g on smithi042 is in error state 2024-07-26T07:10:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:00 smithi163 ceph-mon[36873]: daemon mds.a on smithi042 is in error state 2024-07-26T07:10:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:00 smithi163 ceph-mon[36873]: daemon mds.e on smithi163 is in error state 2024-07-26T07:10:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:00 smithi163 ceph-mon[36873]: daemon mds.h on smithi163 is in error state 2024-07-26T07:10:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:00 smithi163 ceph-mon[36873]: daemon mds.b on smithi163 is in error state 2024-07-26T07:10:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:00 smithi163 ceph-mon[36873]: daemon mds.l on smithi164 is in error state 2024-07-26T07:10:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:00 smithi163 ceph-mon[36873]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-07-26T07:10:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:00 smithi163 ceph-mon[36873]: fs cephfs is degraded 2024-07-26T07:10:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:00 smithi163 ceph-mon[36873]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-07-26T07:10:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:00 smithi163 ceph-mon[36873]: mds.i(mds.1): 1 slow requests are blocked > 180 secs 2024-07-26T07:10:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:00 smithi163 ceph-mon[36873]: mds.f(mds.0): 1 slow requests are blocked > 180 secs 2024-07-26T07:10:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:00 smithi042 ceph-mon[32775]: Health detail: HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:10:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:00 smithi042 ceph-mon[32775]: [WRN] CEPHADM_FAILED_DAEMON: 6 failed cephadm daemon(s) 2024-07-26T07:10:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:00 smithi042 ceph-mon[32775]: daemon mds.g on smithi042 is in error state 2024-07-26T07:10:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:00 smithi042 ceph-mon[32775]: daemon mds.a on smithi042 is in error state 2024-07-26T07:10:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:00 smithi042 ceph-mon[32775]: daemon mds.e on smithi163 is in error state 2024-07-26T07:10:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:00 smithi042 ceph-mon[32775]: daemon mds.h on smithi163 is in error state 2024-07-26T07:10:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:00 smithi042 ceph-mon[32775]: daemon mds.b on smithi163 is in error state 2024-07-26T07:10:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:00 smithi042 ceph-mon[32775]: daemon mds.l on smithi164 is in error state 2024-07-26T07:10:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:00 smithi042 ceph-mon[32775]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-07-26T07:10:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:00 smithi042 ceph-mon[32775]: fs cephfs is degraded 2024-07-26T07:10:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:00 smithi042 ceph-mon[32775]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-07-26T07:10:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:00 smithi042 ceph-mon[32775]: mds.i(mds.1): 1 slow requests are blocked > 180 secs 2024-07-26T07:10:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:00 smithi042 ceph-mon[32775]: mds.f(mds.0): 1 slow requests are blocked > 180 secs 2024-07-26T07:10:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:00 smithi164 ceph-mon[36970]: Health detail: HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:10:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:00 smithi164 ceph-mon[36970]: [WRN] CEPHADM_FAILED_DAEMON: 6 failed cephadm daemon(s) 2024-07-26T07:10:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:00 smithi164 ceph-mon[36970]: daemon mds.g on smithi042 is in error state 2024-07-26T07:10:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:00 smithi164 ceph-mon[36970]: daemon mds.a on smithi042 is in error state 2024-07-26T07:10:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:00 smithi164 ceph-mon[36970]: daemon mds.e on smithi163 is in error state 2024-07-26T07:10:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:00 smithi164 ceph-mon[36970]: daemon mds.h on smithi163 is in error state 2024-07-26T07:10:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:00 smithi164 ceph-mon[36970]: daemon mds.b on smithi163 is in error state 2024-07-26T07:10:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:00 smithi164 ceph-mon[36970]: daemon mds.l on smithi164 is in error state 2024-07-26T07:10:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:00 smithi164 ceph-mon[36970]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-07-26T07:10:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:00 smithi164 ceph-mon[36970]: fs cephfs is degraded 2024-07-26T07:10:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:00 smithi164 ceph-mon[36970]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-07-26T07:10:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:00 smithi164 ceph-mon[36970]: mds.i(mds.1): 1 slow requests are blocked > 180 secs 2024-07-26T07:10:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:00 smithi164 ceph-mon[36970]: mds.f(mds.0): 1 slow requests are blocked > 180 secs 2024-07-26T07:10:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:01 smithi163 ceph-mon[36873]: pgmap v1043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 831.123636 secs 2024-07-26T07:10:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 848.866267 secs 2024-07-26T07:10:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:01 smithi042 ceph-mon[32775]: pgmap v1043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 831.123636 secs 2024-07-26T07:10:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 848.866267 secs 2024-07-26T07:10:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:01 smithi164 ceph-mon[36970]: pgmap v1043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 831.123636 secs 2024-07-26T07:10:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 848.866267 secs 2024-07-26T07:10:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:03 smithi163 ceph-mon[36873]: pgmap v1044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:03.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:03 smithi042 ceph-mon[32775]: pgmap v1044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:03.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:03 smithi164 ceph-mon[36970]: pgmap v1044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:05.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:05 smithi042 ceph-mon[32775]: pgmap v1045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:05.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:05 smithi164 ceph-mon[36970]: pgmap v1045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:05.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:05 smithi163 ceph-mon[36873]: pgmap v1045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:06.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 836.123766 secs 2024-07-26T07:10:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 853.866388 secs 2024-07-26T07:10:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 836.123766 secs 2024-07-26T07:10:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 853.866388 secs 2024-07-26T07:10:06.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 836.123766 secs 2024-07-26T07:10:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 853.866388 secs 2024-07-26T07:10:07.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:07 smithi042 ceph-mon[32775]: pgmap v1046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:07 smithi164 ceph-mon[36970]: pgmap v1046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:07.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:07 smithi163 ceph-mon[36873]: pgmap v1046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:08.542 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:08.540+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:10:08.542 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:08.540+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:10:08.542 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:08.540+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:10:09.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:09 smithi042 ceph-mon[32775]: pgmap v1047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:09 smithi164 ceph-mon[36970]: pgmap v1047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:09.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:09 smithi163 ceph-mon[36873]: pgmap v1047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:11.524 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:11 smithi164 ceph-mon[36970]: pgmap v1048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:11.524 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 841.123909 secs 2024-07-26T07:10:11.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 858.866525 secs 2024-07-26T07:10:11.559 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:11 smithi163 ceph-mon[36873]: pgmap v1048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:11.559 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 841.123909 secs 2024-07-26T07:10:11.559 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 858.866525 secs 2024-07-26T07:10:11.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:11 smithi042 ceph-mon[32775]: pgmap v1048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:11.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 841.123909 secs 2024-07-26T07:10:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 858.866525 secs 2024-07-26T07:10:13.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:13 smithi042 ceph-mon[32775]: pgmap v1049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:13.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:13 smithi164 ceph-mon[36970]: pgmap v1049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:13.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:13 smithi163 ceph-mon[36873]: pgmap v1049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:15.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:15 smithi042 ceph-mon[32775]: pgmap v1050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:15.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:15 smithi164 ceph-mon[36970]: pgmap v1050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:15.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:15 smithi163 ceph-mon[36873]: pgmap v1050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 846.124007 secs 2024-07-26T07:10:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 863.866650 secs 2024-07-26T07:10:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 846.124007 secs 2024-07-26T07:10:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 863.866650 secs 2024-07-26T07:10:16.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 846.124007 secs 2024-07-26T07:10:16.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 863.866650 secs 2024-07-26T07:10:17.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:17 smithi042 ceph-mon[32775]: pgmap v1051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:17 smithi164 ceph-mon[36970]: pgmap v1051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:17 smithi163 ceph-mon[36873]: pgmap v1051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:19.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:19 smithi042 ceph-mon[32775]: pgmap v1052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:19.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:19 smithi164 ceph-mon[36970]: pgmap v1052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:19.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:19 smithi163 ceph-mon[36873]: pgmap v1052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:21.447 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:21 smithi042 ceph-mon[32775]: pgmap v1053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:21.447 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 851.124141 secs 2024-07-26T07:10:21.447 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 868.866773 secs 2024-07-26T07:10:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:21 smithi164 ceph-mon[36970]: pgmap v1053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 851.124141 secs 2024-07-26T07:10:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 868.866773 secs 2024-07-26T07:10:21.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:21 smithi163 ceph-mon[36873]: pgmap v1053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 851.124141 secs 2024-07-26T07:10:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 868.866773 secs 2024-07-26T07:10:22.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:22 smithi042 ceph-mon[32775]: pgmap v1054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:22 smithi164 ceph-mon[36970]: pgmap v1054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:22.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:22 smithi163 ceph-mon[36873]: pgmap v1054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:23.543 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:23.543+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:10:23.543 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:23.543+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:10:23.543 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:23.543+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:10:25.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:25 smithi164 ceph-mon[36970]: pgmap v1055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:25.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:25 smithi163 ceph-mon[36873]: pgmap v1055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:25.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:25 smithi042 ceph-mon[32775]: pgmap v1055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:26.470 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 856.124295 secs 2024-07-26T07:10:26.471 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 856.124295 secs 2024-07-26T07:10:26.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 856.124295 secs 2024-07-26T07:10:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 873.866844 secs 2024-07-26T07:10:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:27 smithi164 ceph-mon[36970]: pgmap v1056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:10:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:27.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 873.866844 secs 2024-07-26T07:10:27.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:27 smithi163 ceph-mon[36873]: pgmap v1056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:10:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:27.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 873.866844 secs 2024-07-26T07:10:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:27 smithi042 ceph-mon[32775]: pgmap v1056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:10:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:10:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:10:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:28 smithi163 ceph-mon[36873]: pgmap v1057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:10:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:28 smithi163 ceph-mon[36873]: pgmap v1058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:10:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:10:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:10:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:28 smithi042 ceph-mon[32775]: pgmap v1057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:10:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:28 smithi042 ceph-mon[32775]: pgmap v1058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:10:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:10:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:10:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:28 smithi164 ceph-mon[36970]: pgmap v1057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:10:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:28 smithi164 ceph-mon[36970]: pgmap v1058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:10:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:10:29.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:29 smithi042 ceph-mon[32775]: pgmap v1059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:10:29.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:29 smithi164 ceph-mon[36970]: pgmap v1059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:10:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:29 smithi163 ceph-mon[36873]: pgmap v1059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:10:31.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:30 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 861.124380 secs 2024-07-26T07:10:31.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:30 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 861.124380 secs 2024-07-26T07:10:31.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:30 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 861.124380 secs 2024-07-26T07:10:32.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 878.866974 secs 2024-07-26T07:10:32.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:31 smithi042 ceph-mon[32775]: pgmap v1060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T07:10:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 878.866974 secs 2024-07-26T07:10:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:31 smithi164 ceph-mon[36970]: pgmap v1060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T07:10:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 878.866974 secs 2024-07-26T07:10:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:31 smithi163 ceph-mon[36873]: pgmap v1060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T07:10:33.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:33 smithi042 ceph-mon[32775]: pgmap v1061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:10:33.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:33 smithi164 ceph-mon[36970]: pgmap v1061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:10:33.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:33 smithi163 ceph-mon[36873]: pgmap v1061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:10:35.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:35 smithi042 ceph-mon[32775]: pgmap v1062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:10:35.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:35 smithi164 ceph-mon[36970]: pgmap v1062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:10:36.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:35 smithi163 ceph-mon[36873]: pgmap v1062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:10:36.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 866.124504 secs 2024-07-26T07:10:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 883.867101 secs 2024-07-26T07:10:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 866.124504 secs 2024-07-26T07:10:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 883.867101 secs 2024-07-26T07:10:37.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 866.124504 secs 2024-07-26T07:10:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 883.867101 secs 2024-07-26T07:10:37.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:37 smithi042 ceph-mon[32775]: pgmap v1063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:10:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:37 smithi164 ceph-mon[36970]: pgmap v1063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:10:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:37 smithi163 ceph-mon[36873]: pgmap v1063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:10:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:38.552+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:10:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:38.552+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:10:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:38.552+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:10:39.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:39 smithi042 ceph-mon[32775]: pgmap v1064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:39 smithi164 ceph-mon[36970]: pgmap v1064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:39.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:39 smithi163 ceph-mon[36873]: pgmap v1064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:41.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:40 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 871.124604 secs 2024-07-26T07:10:41.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:40 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 871.124604 secs 2024-07-26T07:10:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:40 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 871.124604 secs 2024-07-26T07:10:42.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 888.867223 secs 2024-07-26T07:10:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:41 smithi042 ceph-mon[32775]: pgmap v1065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 888.867223 secs 2024-07-26T07:10:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:41 smithi164 ceph-mon[36970]: pgmap v1065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:42.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 888.867223 secs 2024-07-26T07:10:42.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:41 smithi163 ceph-mon[36873]: pgmap v1065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:43.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:43 smithi042 ceph-mon[32775]: pgmap v1066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:43 smithi164 ceph-mon[36970]: pgmap v1066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:43 smithi163 ceph-mon[36873]: pgmap v1066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:45.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:45 smithi042 ceph-mon[32775]: pgmap v1067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:45 smithi164 ceph-mon[36970]: pgmap v1067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:45.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:45 smithi163 ceph-mon[36873]: pgmap v1067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 876.124723 secs 2024-07-26T07:10:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 893.867346 secs 2024-07-26T07:10:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 876.124723 secs 2024-07-26T07:10:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 893.867346 secs 2024-07-26T07:10:46.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 876.124723 secs 2024-07-26T07:10:46.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 893.867346 secs 2024-07-26T07:10:47.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:47 smithi042 ceph-mon[32775]: pgmap v1068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:47 smithi164 ceph-mon[36970]: pgmap v1068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:47 smithi163 ceph-mon[36873]: pgmap v1068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:49.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:49 smithi042 ceph-mon[32775]: pgmap v1069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:49 smithi164 ceph-mon[36970]: pgmap v1069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:49.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:49 smithi163 ceph-mon[36873]: pgmap v1069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:51.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:50 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 881.124839 secs 2024-07-26T07:10:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:50 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 881.124839 secs 2024-07-26T07:10:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:50 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 881.124839 secs 2024-07-26T07:10:52.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 898.867422 secs 2024-07-26T07:10:52.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:51 smithi042 ceph-mon[32775]: pgmap v1070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 898.867422 secs 2024-07-26T07:10:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:51 smithi164 ceph-mon[36970]: pgmap v1070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:52.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 898.867422 secs 2024-07-26T07:10:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:51 smithi163 ceph-mon[36873]: pgmap v1070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:53.554+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:10:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:53.554+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:10:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:53.554+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:10:53.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:53 smithi042 ceph-mon[32775]: pgmap v1071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:53 smithi164 ceph-mon[36970]: pgmap v1071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:53.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:53 smithi163 ceph-mon[36873]: pgmap v1071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:55.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:55 smithi042 ceph-mon[32775]: pgmap v1072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:55 smithi164 ceph-mon[36970]: pgmap v1072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:55.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:55 smithi163 ceph-mon[36873]: pgmap v1072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 886.124966 secs 2024-07-26T07:10:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 903.867560 secs 2024-07-26T07:10:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 886.124966 secs 2024-07-26T07:10:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 903.867560 secs 2024-07-26T07:10:56.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 886.124966 secs 2024-07-26T07:10:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 903.867560 secs 2024-07-26T07:10:57.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:57 smithi042 ceph-mon[32775]: pgmap v1073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:57 smithi164 ceph-mon[36970]: pgmap v1073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:57.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:57 smithi163 ceph-mon[36873]: pgmap v1073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:10:57.861 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:57.860+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T07:10:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:57.860+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T07:10:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:57.860+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b60e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff6780094e0 tx=0x7ff678009450 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:10:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:57.860+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b60e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff6780094e0 tx=0x7ff678009450 comp rx=0 tx=0).stop 2024-07-26T07:10:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:57.860+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0b60e0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T07:10:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:57.860+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0b60e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T07:10:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:57.860+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b60e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:10:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:57.860+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:10:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:57.861+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:10:57.863 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:57.861+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff678000f80 tx=0x7ff678045000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:10:57.998 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:57.997+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500062e0 con 0x55db6e0564d0 2024-07-26T07:10:57.998 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:57.997+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006290 con 0x7ff680102000 2024-07-26T07:10:57.998 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:57.998+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006290 con 0x7ff680102000 2024-07-26T07:10:57.999 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:10:57.998+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500062e0 con 0x55db6e0564d0 2024-07-26T07:10:59.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:10:59 smithi042 ceph-mon[32775]: pgmap v1074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:59.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:10:59 smithi164 ceph-mon[36970]: pgmap v1074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:10:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:10:59 smithi163 ceph-mon[36873]: pgmap v1074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:00 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 891.125087 secs 2024-07-26T07:11:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:00 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 891.125087 secs 2024-07-26T07:11:01.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:00 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 891.125087 secs 2024-07-26T07:11:02.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 908.867664 secs 2024-07-26T07:11:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:01 smithi042 ceph-mon[32775]: pgmap v1075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 908.867664 secs 2024-07-26T07:11:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:01 smithi164 ceph-mon[36970]: pgmap v1075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:02.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 908.867664 secs 2024-07-26T07:11:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:01 smithi163 ceph-mon[36873]: pgmap v1075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:03.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:03 smithi042 ceph-mon[32775]: pgmap v1076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:03.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:03 smithi164 ceph-mon[36970]: pgmap v1076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:03.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:03 smithi163 ceph-mon[36873]: pgmap v1076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:05.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:05 smithi042 ceph-mon[32775]: pgmap v1077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:05.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:05 smithi164 ceph-mon[36970]: pgmap v1077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:05.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:05 smithi163 ceph-mon[36873]: pgmap v1077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 896.125241 secs 2024-07-26T07:11:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 913.867805 secs 2024-07-26T07:11:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 896.125241 secs 2024-07-26T07:11:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 913.867805 secs 2024-07-26T07:11:06.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 896.125241 secs 2024-07-26T07:11:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 913.867805 secs 2024-07-26T07:11:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:07 smithi042 ceph-mon[32775]: pgmap v1078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:07 smithi164 ceph-mon[36970]: pgmap v1078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:07 smithi163 ceph-mon[36873]: pgmap v1078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:11:08.556+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:11:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:11:08.556+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:11:08.557 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:11:08.556+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:11:09.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:09 smithi042 ceph-mon[32775]: pgmap v1079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:09 smithi164 ceph-mon[36970]: pgmap v1079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:09.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:09 smithi163 ceph-mon[36873]: pgmap v1079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:11.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:10 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 901.125328 secs 2024-07-26T07:11:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:10 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 901.125328 secs 2024-07-26T07:11:11.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:10 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 901.125328 secs 2024-07-26T07:11:12.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 918.867903 secs 2024-07-26T07:11:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:11 smithi042 ceph-mon[32775]: pgmap v1080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 918.867903 secs 2024-07-26T07:11:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:11 smithi164 ceph-mon[36970]: pgmap v1080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 918.867903 secs 2024-07-26T07:11:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:11 smithi163 ceph-mon[36873]: pgmap v1080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:13.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:13 smithi042 ceph-mon[32775]: pgmap v1081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:13.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:13 smithi164 ceph-mon[36970]: pgmap v1081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:13 smithi163 ceph-mon[36873]: pgmap v1081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:15.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:15 smithi042 ceph-mon[32775]: pgmap v1082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:15.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:15 smithi164 ceph-mon[36970]: pgmap v1082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:15.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:15 smithi163 ceph-mon[36873]: pgmap v1082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 906.125449 secs 2024-07-26T07:11:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 923.868029 secs 2024-07-26T07:11:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 906.125449 secs 2024-07-26T07:11:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 923.868029 secs 2024-07-26T07:11:16.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 906.125449 secs 2024-07-26T07:11:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 923.868029 secs 2024-07-26T07:11:17.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:17 smithi042 ceph-mon[32775]: pgmap v1083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:17 smithi164 ceph-mon[36970]: pgmap v1083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:17.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:17 smithi163 ceph-mon[36873]: pgmap v1083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:19.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:19 smithi042 ceph-mon[32775]: pgmap v1084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:19.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:19 smithi164 ceph-mon[36970]: pgmap v1084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:19.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:19 smithi163 ceph-mon[36873]: pgmap v1084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:21.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:20 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 911.125592 secs 2024-07-26T07:11:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:20 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 911.125592 secs 2024-07-26T07:11:21.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:20 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 911.125592 secs 2024-07-26T07:11:22.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 928.868133 secs 2024-07-26T07:11:22.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:21 smithi042 ceph-mon[32775]: pgmap v1085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 928.868133 secs 2024-07-26T07:11:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:21 smithi164 ceph-mon[36970]: pgmap v1085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 928.868133 secs 2024-07-26T07:11:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:21 smithi163 ceph-mon[36873]: pgmap v1085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:23.559 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:11:23.559+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:11:23.560 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:11:23.559+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:11:23.560 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:11:23.559+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:11:23.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:23 smithi042 ceph-mon[32775]: pgmap v1086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:23.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:23 smithi164 ceph-mon[36970]: pgmap v1086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:23.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:23 smithi163 ceph-mon[36873]: pgmap v1086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:25.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:25 smithi042 ceph-mon[32775]: pgmap v1087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:25.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:25 smithi164 ceph-mon[36970]: pgmap v1087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:25.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:25 smithi163 ceph-mon[36873]: pgmap v1087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:26.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 916.125732 secs 2024-07-26T07:11:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 933.868233 secs 2024-07-26T07:11:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 916.125732 secs 2024-07-26T07:11:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 933.868233 secs 2024-07-26T07:11:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 916.125732 secs 2024-07-26T07:11:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 933.868233 secs 2024-07-26T07:11:27.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:27 smithi042 ceph-mon[32775]: pgmap v1088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:27 smithi164 ceph-mon[36970]: pgmap v1088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:27.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:27 smithi163 ceph-mon[36873]: pgmap v1088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:11:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:11:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:11:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:28 smithi042 ceph-mon[32775]: pgmap v1089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:11:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:28 smithi042 ceph-mon[32775]: pgmap v1090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:11:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:11:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:11:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:11:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:11:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:28 smithi164 ceph-mon[36970]: pgmap v1089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:11:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:28 smithi164 ceph-mon[36970]: pgmap v1090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:11:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:11:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:11:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:11:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:11:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:28 smithi163 ceph-mon[36873]: pgmap v1089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:11:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:28 smithi163 ceph-mon[36873]: pgmap v1090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:11:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:11:30.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:30 smithi042 ceph-mon[32775]: pgmap v1091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:11:30.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:30 smithi164 ceph-mon[36970]: pgmap v1091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:11:31.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:30 smithi163 ceph-mon[36873]: pgmap v1091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:11:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 921.125888 secs 2024-07-26T07:11:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 938.868337 secs 2024-07-26T07:11:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 921.125888 secs 2024-07-26T07:11:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 938.868337 secs 2024-07-26T07:11:32.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 921.125888 secs 2024-07-26T07:11:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 938.868337 secs 2024-07-26T07:11:32.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:32 smithi042 ceph-mon[32775]: pgmap v1092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:11:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:32 smithi164 ceph-mon[36970]: pgmap v1092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:11:33.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:32 smithi163 ceph-mon[36873]: pgmap v1092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:11:35.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:34 smithi042 ceph-mon[32775]: pgmap v1093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:11:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:34 smithi164 ceph-mon[36970]: pgmap v1093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:11:35.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:34 smithi163 ceph-mon[36873]: pgmap v1093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:11:36.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:35 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 926.126010 secs 2024-07-26T07:11:36.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:35 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 926.126010 secs 2024-07-26T07:11:36.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:35 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 926.126010 secs 2024-07-26T07:11:37.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:36 smithi042 ceph-mon[32775]: pgmap v1094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:11:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 943.868473 secs 2024-07-26T07:11:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:36 smithi164 ceph-mon[36970]: pgmap v1094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:11:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 943.868473 secs 2024-07-26T07:11:37.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:36 smithi163 ceph-mon[36873]: pgmap v1094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:11:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 943.868473 secs 2024-07-26T07:11:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:11:38.560+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:11:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:11:38.560+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:11:38.561 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:11:38.560+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:11:39.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:38 smithi042 ceph-mon[32775]: pgmap v1095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:11:39.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:38 smithi164 ceph-mon[36970]: pgmap v1095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:11:39.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:38 smithi163 ceph-mon[36873]: pgmap v1095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:11:41.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:40 smithi042 ceph-mon[32775]: pgmap v1096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:40 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 931.126137 secs 2024-07-26T07:11:41.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:40 smithi164 ceph-mon[36970]: pgmap v1096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:41.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:40 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 931.126137 secs 2024-07-26T07:11:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:40 smithi163 ceph-mon[36873]: pgmap v1096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:40 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 931.126137 secs 2024-07-26T07:11:42.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 948.868575 secs 2024-07-26T07:11:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 948.868575 secs 2024-07-26T07:11:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 948.868575 secs 2024-07-26T07:11:43.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:42 smithi042 ceph-mon[32775]: pgmap v1097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:43.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:42 smithi164 ceph-mon[36970]: pgmap v1097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:43.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:42 smithi163 ceph-mon[36873]: pgmap v1097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:45.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:44 smithi042 ceph-mon[32775]: pgmap v1098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:11:45.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:44 smithi164 ceph-mon[36970]: pgmap v1098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:11:45.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:44 smithi163 ceph-mon[36873]: pgmap v1098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:11:46.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:45 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 936.126290 secs 2024-07-26T07:11:46.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:45 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 936.126290 secs 2024-07-26T07:11:46.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:45 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 936.126290 secs 2024-07-26T07:11:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:46 smithi164 ceph-mon[36970]: pgmap v1099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 953.868683 secs 2024-07-26T07:11:47.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:46 smithi163 ceph-mon[36873]: pgmap v1099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 953.868683 secs 2024-07-26T07:11:47.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:46 smithi042 ceph-mon[32775]: pgmap v1099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:47.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 953.868683 secs 2024-07-26T07:11:49.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:48 smithi164 ceph-mon[36970]: pgmap v1100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:48 smithi163 ceph-mon[36873]: pgmap v1100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:49.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:48 smithi042 ceph-mon[32775]: pgmap v1100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:50 smithi164 ceph-mon[36970]: pgmap v1101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:50 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 941.126365 secs 2024-07-26T07:11:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:50 smithi163 ceph-mon[36873]: pgmap v1101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:50 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 941.126365 secs 2024-07-26T07:11:51.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:50 smithi042 ceph-mon[32775]: pgmap v1101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:50 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 941.126365 secs 2024-07-26T07:11:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 958.868832 secs 2024-07-26T07:11:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 958.868832 secs 2024-07-26T07:11:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 958.868832 secs 2024-07-26T07:11:53.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:52 smithi164 ceph-mon[36970]: pgmap v1102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:53.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:52 smithi163 ceph-mon[36873]: pgmap v1102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:53.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:52 smithi042 ceph-mon[32775]: pgmap v1102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:11:53.563+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:11:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:11:53.563+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:11:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:11:53.563+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:11:55.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:54 smithi042 ceph-mon[32775]: pgmap v1103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:11:55.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:54 smithi164 ceph-mon[36970]: pgmap v1103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:11:55.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:54 smithi163 ceph-mon[36873]: pgmap v1103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:11:56.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:55 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 946.126484 secs 2024-07-26T07:11:56.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:55 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 946.126484 secs 2024-07-26T07:11:56.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:55 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 946.126484 secs 2024-07-26T07:11:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:56 smithi164 ceph-mon[36970]: pgmap v1104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 963.868990 secs 2024-07-26T07:11:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:56 smithi163 ceph-mon[36873]: pgmap v1104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 963.868990 secs 2024-07-26T07:11:57.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:56 smithi042 ceph-mon[32775]: pgmap v1104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:11:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 963.868990 secs 2024-07-26T07:11:59.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:11:58 smithi164 ceph-mon[36970]: pgmap v1105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:11:58 smithi163 ceph-mon[36873]: pgmap v1105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:11:59.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:11:58 smithi042 ceph-mon[32775]: pgmap v1105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:00 smithi164 ceph-mon[36970]: pgmap v1106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:00 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 951.126563 secs 2024-07-26T07:12:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:00 smithi163 ceph-mon[36873]: pgmap v1106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:00 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 951.126563 secs 2024-07-26T07:12:01.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:00 smithi042 ceph-mon[32775]: pgmap v1106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:01.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:00 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 951.126563 secs 2024-07-26T07:12:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 968.869111 secs 2024-07-26T07:12:02.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 968.869111 secs 2024-07-26T07:12:02.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 968.869111 secs 2024-07-26T07:12:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:02 smithi164 ceph-mon[36970]: pgmap v1107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:02 smithi163 ceph-mon[36873]: pgmap v1107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:02 smithi042 ceph-mon[32775]: pgmap v1107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:05.196 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:04 smithi042 ceph-mon[32775]: pgmap v1108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:12:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:04 smithi164 ceph-mon[36970]: pgmap v1108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:12:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:04 smithi163 ceph-mon[36873]: pgmap v1108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:12:06.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:05 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 956.126689 secs 2024-07-26T07:12:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:05 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 956.126689 secs 2024-07-26T07:12:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:05 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 956.126689 secs 2024-07-26T07:12:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:06 smithi164 ceph-mon[36970]: pgmap v1109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 973.869238 secs 2024-07-26T07:12:07.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:06 smithi163 ceph-mon[36873]: pgmap v1109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 973.869238 secs 2024-07-26T07:12:07.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:06 smithi042 ceph-mon[32775]: pgmap v1109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 973.869238 secs 2024-07-26T07:12:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:12:08.571+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:12:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:12:08.571+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:12:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:12:08.571+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:12:09.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:08 smithi164 ceph-mon[36970]: pgmap v1110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:09.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:08 smithi163 ceph-mon[36873]: pgmap v1110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:09.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:08 smithi042 ceph-mon[32775]: pgmap v1110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:10 smithi164 ceph-mon[36970]: pgmap v1111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:10 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 961.126809 secs 2024-07-26T07:12:11.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:10 smithi163 ceph-mon[36873]: pgmap v1111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:10 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 961.126809 secs 2024-07-26T07:12:11.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:10 smithi042 ceph-mon[32775]: pgmap v1111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:10 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 961.126809 secs 2024-07-26T07:12:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 978.869326 secs 2024-07-26T07:12:12.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 978.869326 secs 2024-07-26T07:12:12.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 978.869326 secs 2024-07-26T07:12:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:12 smithi164 ceph-mon[36970]: pgmap v1112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:13.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:12 smithi163 ceph-mon[36873]: pgmap v1112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:12 smithi042 ceph-mon[32775]: pgmap v1112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:15.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:14 smithi164 ceph-mon[36970]: pgmap v1113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:12:15.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:14 smithi163 ceph-mon[36873]: pgmap v1113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:12:15.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:14 smithi042 ceph-mon[32775]: pgmap v1113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:12:16.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:15 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 966.126929 secs 2024-07-26T07:12:16.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:15 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 966.126929 secs 2024-07-26T07:12:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:15 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 966.126929 secs 2024-07-26T07:12:17.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:16 smithi163 ceph-mon[36873]: pgmap v1114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 983.869453 secs 2024-07-26T07:12:17.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:16 smithi042 ceph-mon[32775]: pgmap v1114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 983.869453 secs 2024-07-26T07:12:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:16 smithi164 ceph-mon[36970]: pgmap v1114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 983.869453 secs 2024-07-26T07:12:19.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:18 smithi163 ceph-mon[36873]: pgmap v1115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:19.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:18 smithi042 ceph-mon[32775]: pgmap v1115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:18 smithi164 ceph-mon[36970]: pgmap v1115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:20 smithi163 ceph-mon[36873]: pgmap v1116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:20 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 971.127050 secs 2024-07-26T07:12:21.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:20 smithi042 ceph-mon[32775]: pgmap v1116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:21.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:20 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 971.127050 secs 2024-07-26T07:12:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:20 smithi164 ceph-mon[36970]: pgmap v1116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:20 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 971.127050 secs 2024-07-26T07:12:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 988.869609 secs 2024-07-26T07:12:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 988.869609 secs 2024-07-26T07:12:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 988.869609 secs 2024-07-26T07:12:23.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:22 smithi163 ceph-mon[36873]: pgmap v1117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:23.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:22 smithi042 ceph-mon[32775]: pgmap v1117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:22 smithi164 ceph-mon[36970]: pgmap v1117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:12:23.574+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:12:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:12:23.574+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:12:23.575 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:12:23.574+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:12:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:24 smithi163 ceph-mon[36873]: pgmap v1118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:12:25.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:24 smithi042 ceph-mon[32775]: pgmap v1118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:12:25.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:24 smithi164 ceph-mon[36970]: pgmap v1118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:12:26.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 976.127173 secs 2024-07-26T07:12:26.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 976.127173 secs 2024-07-26T07:12:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 976.127173 secs 2024-07-26T07:12:27.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:27 smithi163 ceph-mon[36873]: pgmap v1119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 993.869756 secs 2024-07-26T07:12:27.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:27 smithi042 ceph-mon[32775]: pgmap v1119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 993.869756 secs 2024-07-26T07:12:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:27 smithi164 ceph-mon[36970]: pgmap v1119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 993.869756 secs 2024-07-26T07:12:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:12:28.011+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:12:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:12:28.011+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:12:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:12:28.012+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:12:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:12:28.012+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:12:28.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:12:28.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:12:28.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:12:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:29 smithi042 ceph-mon[32775]: pgmap v1120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:12:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:12:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:12:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:29 smithi164 ceph-mon[36970]: pgmap v1120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:12:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:12:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:12:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:29 smithi163 ceph-mon[36873]: pgmap v1120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:12:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:12:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:12:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:30 smithi042 ceph-mon[32775]: pgmap v1121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:12:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:30 smithi042 ceph-mon[32775]: pgmap v1122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:12:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:30 smithi164 ceph-mon[36970]: pgmap v1121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:12:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:30 smithi164 ceph-mon[36970]: pgmap v1122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:12:30.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:30 smithi163 ceph-mon[36873]: pgmap v1121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:12:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:30 smithi163 ceph-mon[36873]: pgmap v1122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:12:31.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 981.127291 secs 2024-07-26T07:12:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 981.127291 secs 2024-07-26T07:12:31.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 981.127291 secs 2024-07-26T07:12:32.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:32 smithi042 ceph-mon[32775]: pgmap v1123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:12:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 998.869852 secs 2024-07-26T07:12:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:32 smithi164 ceph-mon[36970]: pgmap v1123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:12:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 998.869852 secs 2024-07-26T07:12:32.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:32 smithi163 ceph-mon[36873]: pgmap v1123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:12:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 998.869852 secs 2024-07-26T07:12:34.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:34 smithi042 ceph-mon[32775]: pgmap v1124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:12:34.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:34 smithi164 ceph-mon[36970]: pgmap v1124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:12:34.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:34 smithi163 ceph-mon[36873]: pgmap v1124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:12:36.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:36 smithi042 ceph-mon[32775]: pgmap v1125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:12:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 986.127364 secs 2024-07-26T07:12:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:36 smithi164 ceph-mon[36970]: pgmap v1125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:12:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 986.127364 secs 2024-07-26T07:12:36.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:36 smithi163 ceph-mon[36873]: pgmap v1125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:12:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 986.127364 secs 2024-07-26T07:12:37.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1003.870005 secs 2024-07-26T07:12:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1003.870005 secs 2024-07-26T07:12:37.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1003.870005 secs 2024-07-26T07:12:38.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:38 smithi042 ceph-mon[32775]: pgmap v1126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:12:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:38 smithi164 ceph-mon[36970]: pgmap v1126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:12:38.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:38 smithi163 ceph-mon[36873]: pgmap v1126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:12:38.581 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:12:38.581+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:12:38.581 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:12:38.581+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:12:38.582 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:12:38.581+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:12:40.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:40 smithi042 ceph-mon[32775]: pgmap v1127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:12:40.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:40 smithi164 ceph-mon[36970]: pgmap v1127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:12:40.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:40 smithi163 ceph-mon[36873]: pgmap v1127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:12:41.372 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 991.127434 secs 2024-07-26T07:12:41.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 991.127434 secs 2024-07-26T07:12:41.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 991.127434 secs 2024-07-26T07:12:42.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:42 smithi042 ceph-mon[32775]: pgmap v1128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1008.870136 secs 2024-07-26T07:12:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:42 smithi164 ceph-mon[36970]: pgmap v1128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1008.870136 secs 2024-07-26T07:12:42.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:42 smithi163 ceph-mon[36873]: pgmap v1128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1008.870136 secs 2024-07-26T07:12:44.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:44 smithi042 ceph-mon[32775]: pgmap v1129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:44.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:44 smithi164 ceph-mon[36970]: pgmap v1129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:44.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:44 smithi163 ceph-mon[36873]: pgmap v1129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:46.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:46 smithi042 ceph-mon[32775]: pgmap v1130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 996.127500 secs 2024-07-26T07:12:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:46 smithi164 ceph-mon[36970]: pgmap v1130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 996.127500 secs 2024-07-26T07:12:46.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:46 smithi163 ceph-mon[36873]: pgmap v1130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 996.127500 secs 2024-07-26T07:12:47.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1013.870263 secs 2024-07-26T07:12:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1013.870263 secs 2024-07-26T07:12:47.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1013.870263 secs 2024-07-26T07:12:48.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:48 smithi042 ceph-mon[32775]: pgmap v1131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:48.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:48 smithi164 ceph-mon[36970]: pgmap v1131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:48.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:48 smithi163 ceph-mon[36873]: pgmap v1131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:50.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:50 smithi042 ceph-mon[32775]: pgmap v1132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:12:50.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:50 smithi164 ceph-mon[36970]: pgmap v1132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:12:50.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:50 smithi163 ceph-mon[36873]: pgmap v1132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:12:51.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1001.127576 secs 2024-07-26T07:12:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1001.127576 secs 2024-07-26T07:12:51.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1001.127576 secs 2024-07-26T07:12:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:52 smithi042 ceph-mon[32775]: pgmap v1133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1018.870395 secs 2024-07-26T07:12:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:52 smithi164 ceph-mon[36970]: pgmap v1133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1018.870395 secs 2024-07-26T07:12:52.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:52 smithi163 ceph-mon[36873]: pgmap v1133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:52.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1018.870395 secs 2024-07-26T07:12:53.585 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:12:53.584+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:12:53.585 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:12:53.584+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:12:53.585 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:12:53.584+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:12:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:54 smithi042 ceph-mon[32775]: pgmap v1134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:54.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:54 smithi164 ceph-mon[36970]: pgmap v1134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:54.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:54 smithi163 ceph-mon[36873]: pgmap v1134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:56.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:56 smithi042 ceph-mon[32775]: pgmap v1135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1006.127695 secs 2024-07-26T07:12:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:56 smithi164 ceph-mon[36970]: pgmap v1135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1006.127695 secs 2024-07-26T07:12:56.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:56 smithi163 ceph-mon[36873]: pgmap v1135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:12:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1006.127695 secs 2024-07-26T07:12:57.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1023.870516 secs 2024-07-26T07:12:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1023.870516 secs 2024-07-26T07:12:57.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1023.870516 secs 2024-07-26T07:12:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:12:58 smithi042 ceph-mon[32775]: pgmap v1136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:58.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:12:58 smithi164 ceph-mon[36970]: pgmap v1136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:12:58.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:12:58 smithi163 ceph-mon[36873]: pgmap v1136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:00.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:00 smithi042 ceph-mon[32775]: pgmap v1137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:13:00.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:00 smithi164 ceph-mon[36970]: pgmap v1137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:13:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:00 smithi163 ceph-mon[36873]: pgmap v1137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:13:01.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1011.127821 secs 2024-07-26T07:13:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1011.127821 secs 2024-07-26T07:13:01.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1011.127821 secs 2024-07-26T07:13:02.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:02 smithi042 ceph-mon[32775]: pgmap v1138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:02.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1028.870641 secs 2024-07-26T07:13:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:02 smithi164 ceph-mon[36970]: pgmap v1138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1028.870641 secs 2024-07-26T07:13:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:02 smithi163 ceph-mon[36873]: pgmap v1138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1028.870641 secs 2024-07-26T07:13:04.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:04 smithi042 ceph-mon[32775]: pgmap v1139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:04.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:04 smithi164 ceph-mon[36970]: pgmap v1139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:04.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:04 smithi163 ceph-mon[36873]: pgmap v1139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:06.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:06 smithi042 ceph-mon[32775]: pgmap v1140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:06.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1016.127960 secs 2024-07-26T07:13:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:06 smithi164 ceph-mon[36970]: pgmap v1140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1016.127960 secs 2024-07-26T07:13:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:06 smithi163 ceph-mon[36873]: pgmap v1140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1016.127960 secs 2024-07-26T07:13:07.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1033.870787 secs 2024-07-26T07:13:07.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1033.870787 secs 2024-07-26T07:13:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1033.870787 secs 2024-07-26T07:13:08.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:08 smithi042 ceph-mon[32775]: pgmap v1141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:08 smithi164 ceph-mon[36970]: pgmap v1141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:08.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:08 smithi163 ceph-mon[36873]: pgmap v1141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:08.591 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:13:08.591+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:13:08.592 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:13:08.591+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:13:08.592 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:13:08.591+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:13:10.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:10 smithi042 ceph-mon[32775]: pgmap v1142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:13:10.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:10 smithi164 ceph-mon[36970]: pgmap v1142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:13:10.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:10 smithi163 ceph-mon[36873]: pgmap v1142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:13:11.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1021.128048 secs 2024-07-26T07:13:11.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1021.128048 secs 2024-07-26T07:13:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1021.128048 secs 2024-07-26T07:13:12.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:12 smithi042 ceph-mon[32775]: pgmap v1143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1038.870908 secs 2024-07-26T07:13:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:12 smithi164 ceph-mon[36970]: pgmap v1143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1038.870908 secs 2024-07-26T07:13:12.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:12 smithi163 ceph-mon[36873]: pgmap v1143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:12.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1038.870908 secs 2024-07-26T07:13:14.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:14 smithi042 ceph-mon[32775]: pgmap v1144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:14.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:14 smithi164 ceph-mon[36970]: pgmap v1144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:14.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:14 smithi163 ceph-mon[36873]: pgmap v1144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:16.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:16 smithi042 ceph-mon[32775]: pgmap v1145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1026.128188 secs 2024-07-26T07:13:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:16 smithi164 ceph-mon[36970]: pgmap v1145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1026.128188 secs 2024-07-26T07:13:16.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:16 smithi163 ceph-mon[36873]: pgmap v1145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1026.128188 secs 2024-07-26T07:13:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1043.871031 secs 2024-07-26T07:13:17.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1043.871031 secs 2024-07-26T07:13:17.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1043.871031 secs 2024-07-26T07:13:18.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:18 smithi164 ceph-mon[36970]: pgmap v1146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:18 smithi163 ceph-mon[36873]: pgmap v1146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:18.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:18 smithi042 ceph-mon[32775]: pgmap v1146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:20 smithi164 ceph-mon[36970]: pgmap v1147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:13:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:20 smithi163 ceph-mon[36873]: pgmap v1147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:13:20.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:20 smithi042 ceph-mon[32775]: pgmap v1147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:13:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1031.128309 secs 2024-07-26T07:13:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1031.128309 secs 2024-07-26T07:13:21.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1031.128309 secs 2024-07-26T07:13:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:22 smithi164 ceph-mon[36970]: pgmap v1148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1048.871127 secs 2024-07-26T07:13:22.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:22 smithi163 ceph-mon[36873]: pgmap v1148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1048.871127 secs 2024-07-26T07:13:22.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:22 smithi042 ceph-mon[32775]: pgmap v1148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1048.871127 secs 2024-07-26T07:13:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:13:23.594+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:13:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:13:23.594+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:13:23.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:13:23.594+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:13:24.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:24 smithi164 ceph-mon[36970]: pgmap v1149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:24.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:24 smithi163 ceph-mon[36873]: pgmap v1149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:24.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:24 smithi042 ceph-mon[32775]: pgmap v1149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:26 smithi164 ceph-mon[36970]: pgmap v1150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1036.128414 secs 2024-07-26T07:13:26.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:26 smithi163 ceph-mon[36873]: pgmap v1150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1036.128414 secs 2024-07-26T07:13:26.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:26 smithi042 ceph-mon[32775]: pgmap v1150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1036.128414 secs 2024-07-26T07:13:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1053.871233 secs 2024-07-26T07:13:27.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1053.871233 secs 2024-07-26T07:13:27.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1053.871233 secs 2024-07-26T07:13:28.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:28 smithi042 ceph-mon[32775]: pgmap v1151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:28.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:28 smithi164 ceph-mon[36970]: pgmap v1151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:28.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:28 smithi163 ceph-mon[36873]: pgmap v1151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:13:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:13:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:13:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:13:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:13:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:13:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:13:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:13:29.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:13:29.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:13:29.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:13:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:13:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:30 smithi164 ceph-mon[36970]: pgmap v1152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:13:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:30 smithi164 ceph-mon[36970]: pgmap v1153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:13:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:30 smithi164 ceph-mon[36970]: pgmap v1154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:13:30.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:30 smithi163 ceph-mon[36873]: pgmap v1152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:13:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:30 smithi163 ceph-mon[36873]: pgmap v1153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:13:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:30 smithi163 ceph-mon[36873]: pgmap v1154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:13:30.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:30 smithi042 ceph-mon[32775]: pgmap v1152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:13:30.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:30 smithi042 ceph-mon[32775]: pgmap v1153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:13:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:30 smithi042 ceph-mon[32775]: pgmap v1154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:13:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1041.128537 secs 2024-07-26T07:13:31.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1041.128537 secs 2024-07-26T07:13:31.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1041.128537 secs 2024-07-26T07:13:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:32 smithi164 ceph-mon[36970]: pgmap v1155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:13:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1058.871356 secs 2024-07-26T07:13:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:32 smithi163 ceph-mon[36873]: pgmap v1155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:13:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1058.871356 secs 2024-07-26T07:13:32.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:32 smithi042 ceph-mon[32775]: pgmap v1155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:13:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1058.871356 secs 2024-07-26T07:13:34.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:34 smithi164 ceph-mon[36970]: pgmap v1156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:13:34.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:34 smithi163 ceph-mon[36873]: pgmap v1156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:13:34.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:34 smithi042 ceph-mon[32775]: pgmap v1156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:13:36.447 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:36 smithi042 ceph-mon[32775]: pgmap v1157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:13:36.447 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1046.128658 secs 2024-07-26T07:13:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:36 smithi164 ceph-mon[36970]: pgmap v1157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:13:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1046.128658 secs 2024-07-26T07:13:36.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:36 smithi163 ceph-mon[36873]: pgmap v1157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:13:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1046.128658 secs 2024-07-26T07:13:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1063.871496 secs 2024-07-26T07:13:37.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1063.871496 secs 2024-07-26T07:13:37.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1063.871496 secs 2024-07-26T07:13:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:38 smithi164 ceph-mon[36970]: pgmap v1158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:13:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:38 smithi163 ceph-mon[36873]: pgmap v1158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:13:38.601 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:13:38.601+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:13:38.602 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:13:38.601+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:13:38.602 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:13:38.601+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:13:38.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:38 smithi042 ceph-mon[32775]: pgmap v1158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:13:40.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:40 smithi164 ceph-mon[36970]: pgmap v1159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:13:40.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:40 smithi163 ceph-mon[36873]: pgmap v1159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:13:40.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:40 smithi042 ceph-mon[32775]: pgmap v1159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:13:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1051.128800 secs 2024-07-26T07:13:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1068.871626 secs 2024-07-26T07:13:41.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1051.128800 secs 2024-07-26T07:13:41.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1068.871626 secs 2024-07-26T07:13:41.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1051.128800 secs 2024-07-26T07:13:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1068.871626 secs 2024-07-26T07:13:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:42 smithi164 ceph-mon[36970]: pgmap v1160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:42 smithi163 ceph-mon[36873]: pgmap v1160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:42.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:42 smithi042 ceph-mon[32775]: pgmap v1160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:44.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:44 smithi164 ceph-mon[36970]: pgmap v1161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:44.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:44 smithi163 ceph-mon[36873]: pgmap v1161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:44.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:44 smithi042 ceph-mon[32775]: pgmap v1161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:46 smithi164 ceph-mon[36970]: pgmap v1162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1056.128919 secs 2024-07-26T07:13:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1073.871754 secs 2024-07-26T07:13:46.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:46 smithi163 ceph-mon[36873]: pgmap v1162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1056.128919 secs 2024-07-26T07:13:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1073.871754 secs 2024-07-26T07:13:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:46 smithi042 ceph-mon[32775]: pgmap v1162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1056.128919 secs 2024-07-26T07:13:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1073.871754 secs 2024-07-26T07:13:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:48 smithi163 ceph-mon[36873]: pgmap v1163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:48.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:48 smithi042 ceph-mon[32775]: pgmap v1163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:48.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:48 smithi164 ceph-mon[36970]: pgmap v1163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:50 smithi163 ceph-mon[36873]: pgmap v1164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:13:50.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:50 smithi042 ceph-mon[32775]: pgmap v1164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:13:50.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:50 smithi164 ceph-mon[36970]: pgmap v1164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:13:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1061.129043 secs 2024-07-26T07:13:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1078.871874 secs 2024-07-26T07:13:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1061.129043 secs 2024-07-26T07:13:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1078.871874 secs 2024-07-26T07:13:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1061.129043 secs 2024-07-26T07:13:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1078.871874 secs 2024-07-26T07:13:52.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:52 smithi163 ceph-mon[36873]: pgmap v1165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:52.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:52 smithi042 ceph-mon[32775]: pgmap v1165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:52 smithi164 ceph-mon[36970]: pgmap v1165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:53.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:13:53.604+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:13:53.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:13:53.604+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:13:53.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:13:53.604+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:13:54.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:54 smithi163 ceph-mon[36873]: pgmap v1166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:54.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:54 smithi042 ceph-mon[32775]: pgmap v1166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:54 smithi164 ceph-mon[36970]: pgmap v1166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:56.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:56 smithi163 ceph-mon[36873]: pgmap v1167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1066.129163 secs 2024-07-26T07:13:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1083.872021 secs 2024-07-26T07:13:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:56 smithi042 ceph-mon[32775]: pgmap v1167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1066.129163 secs 2024-07-26T07:13:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1083.872021 secs 2024-07-26T07:13:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:56 smithi164 ceph-mon[36970]: pgmap v1167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:13:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1066.129163 secs 2024-07-26T07:13:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1083.872021 secs 2024-07-26T07:13:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:13:58.026+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T07:13:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:13:58.026+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T07:13:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:13:58.026+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T07:13:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:13:58.026+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T07:13:58.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:13:58 smithi163 ceph-mon[36873]: pgmap v1168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:58.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:13:58 smithi042 ceph-mon[32775]: pgmap v1168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:13:58.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:13:58 smithi164 ceph-mon[36970]: pgmap v1168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:00.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:00 smithi163 ceph-mon[36873]: pgmap v1169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:00.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:00 smithi042 ceph-mon[32775]: pgmap v1169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:00 smithi164 ceph-mon[36970]: pgmap v1169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:01.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1071.129286 secs 2024-07-26T07:14:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1088.872137 secs 2024-07-26T07:14:01.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1071.129286 secs 2024-07-26T07:14:01.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1088.872137 secs 2024-07-26T07:14:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1071.129286 secs 2024-07-26T07:14:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1088.872137 secs 2024-07-26T07:14:02.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:02 smithi042 ceph-mon[32775]: pgmap v1170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:02.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:02 smithi164 ceph-mon[36970]: pgmap v1170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:02.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:02 smithi163 ceph-mon[36873]: pgmap v1170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:04.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:04 smithi042 ceph-mon[32775]: pgmap v1171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:04.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:04 smithi164 ceph-mon[36970]: pgmap v1171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:04.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:04 smithi163 ceph-mon[36873]: pgmap v1171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:06.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:06 smithi042 ceph-mon[32775]: pgmap v1172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1076.129367 secs 2024-07-26T07:14:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1093.872274 secs 2024-07-26T07:14:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:06 smithi164 ceph-mon[36970]: pgmap v1172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1076.129367 secs 2024-07-26T07:14:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1093.872274 secs 2024-07-26T07:14:06.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:06 smithi163 ceph-mon[36873]: pgmap v1172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1076.129367 secs 2024-07-26T07:14:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1093.872274 secs 2024-07-26T07:14:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:14:08.612+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:14:08.612 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:14:08.612+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:14:08.613 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:14:08.612+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:14:08.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:08 smithi042 ceph-mon[32775]: pgmap v1173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:08 smithi164 ceph-mon[36970]: pgmap v1173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:08.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:08 smithi163 ceph-mon[36873]: pgmap v1173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:10.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:10 smithi042 ceph-mon[32775]: pgmap v1174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:10 smithi164 ceph-mon[36970]: pgmap v1174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:10.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:10 smithi163 ceph-mon[36873]: pgmap v1174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:11.623 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1081.129487 secs 2024-07-26T07:14:11.623 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1098.872414 secs 2024-07-26T07:14:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1081.129487 secs 2024-07-26T07:14:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1098.872414 secs 2024-07-26T07:14:11.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1081.129487 secs 2024-07-26T07:14:11.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1098.872414 secs 2024-07-26T07:14:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:12 smithi042 ceph-mon[32775]: pgmap v1175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:12 smithi164 ceph-mon[36970]: pgmap v1175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:12.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:12 smithi163 ceph-mon[36873]: pgmap v1175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:14 smithi042 ceph-mon[32775]: pgmap v1176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:14.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:14 smithi164 ceph-mon[36970]: pgmap v1176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:14 smithi163 ceph-mon[36873]: pgmap v1176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:16 smithi042 ceph-mon[32775]: pgmap v1177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1086.129605 secs 2024-07-26T07:14:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1103.872548 secs 2024-07-26T07:14:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:16 smithi164 ceph-mon[36970]: pgmap v1177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1086.129605 secs 2024-07-26T07:14:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1103.872548 secs 2024-07-26T07:14:16.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:16 smithi163 ceph-mon[36873]: pgmap v1177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:16.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1086.129605 secs 2024-07-26T07:14:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1103.872548 secs 2024-07-26T07:14:18.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:18 smithi042 ceph-mon[32775]: pgmap v1178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:18.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:18 smithi164 ceph-mon[36970]: pgmap v1178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:18.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:18 smithi163 ceph-mon[36873]: pgmap v1178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:20.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:20 smithi042 ceph-mon[32775]: pgmap v1179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:20.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:20 smithi164 ceph-mon[36970]: pgmap v1179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:20.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:20 smithi163 ceph-mon[36873]: pgmap v1179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:21.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1091.129750 secs 2024-07-26T07:14:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1108.872700 secs 2024-07-26T07:14:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1091.129750 secs 2024-07-26T07:14:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1108.872700 secs 2024-07-26T07:14:21.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1091.129750 secs 2024-07-26T07:14:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1108.872700 secs 2024-07-26T07:14:22.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:22 smithi042 ceph-mon[32775]: pgmap v1180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:22 smithi164 ceph-mon[36970]: pgmap v1180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:22.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:22 smithi163 ceph-mon[36873]: pgmap v1180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:14:23.615+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:14:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:14:23.615+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:14:23.616 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:14:23.615+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:14:24.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:24 smithi042 ceph-mon[32775]: pgmap v1181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:24.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:24 smithi164 ceph-mon[36970]: pgmap v1181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:24.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:24 smithi163 ceph-mon[36873]: pgmap v1181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:26 smithi042 ceph-mon[32775]: pgmap v1182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1096.129871 secs 2024-07-26T07:14:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1113.872829 secs 2024-07-26T07:14:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:26 smithi164 ceph-mon[36970]: pgmap v1182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1096.129871 secs 2024-07-26T07:14:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1113.872829 secs 2024-07-26T07:14:26.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:26 smithi163 ceph-mon[36873]: pgmap v1182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:26.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1096.129871 secs 2024-07-26T07:14:26.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1113.872829 secs 2024-07-26T07:14:28.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:28 smithi042 ceph-mon[32775]: pgmap v1183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:28 smithi164 ceph-mon[36970]: pgmap v1183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:28.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:28 smithi163 ceph-mon[36873]: pgmap v1183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:29.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:14:29.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:14:29.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:14:30.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:30 smithi042 ceph-mon[32775]: pgmap v1184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:14:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:14:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:14:30.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:30 smithi164 ceph-mon[36970]: pgmap v1184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:30.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:14:30.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:14:30.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:14:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:30 smithi163 ceph-mon[36873]: pgmap v1184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:14:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:14:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:14:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:31 smithi042 ceph-mon[32775]: pgmap v1185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:14:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:31 smithi042 ceph-mon[32775]: pgmap v1186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:14:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1101.130016 secs 2024-07-26T07:14:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1118.872979 secs 2024-07-26T07:14:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:31 smithi164 ceph-mon[36970]: pgmap v1185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:14:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:31 smithi164 ceph-mon[36970]: pgmap v1186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:14:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1101.130016 secs 2024-07-26T07:14:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1118.872979 secs 2024-07-26T07:14:31.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:31 smithi163 ceph-mon[36873]: pgmap v1185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:14:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:31 smithi163 ceph-mon[36873]: pgmap v1186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:14:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1101.130016 secs 2024-07-26T07:14:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1118.872979 secs 2024-07-26T07:14:33.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:33 smithi042 ceph-mon[32775]: pgmap v1187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:14:33.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:33 smithi164 ceph-mon[36970]: pgmap v1187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:14:33.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:33 smithi163 ceph-mon[36873]: pgmap v1187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:14:35.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:35 smithi042 ceph-mon[32775]: pgmap v1188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:14:35.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:35 smithi164 ceph-mon[36970]: pgmap v1188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:14:35.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:35 smithi163 ceph-mon[36873]: pgmap v1188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:14:36.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1106.130135 secs 2024-07-26T07:14:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1123.873109 secs 2024-07-26T07:14:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1106.130135 secs 2024-07-26T07:14:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1123.873109 secs 2024-07-26T07:14:36.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1106.130135 secs 2024-07-26T07:14:36.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1123.873109 secs 2024-07-26T07:14:37.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:37 smithi042 ceph-mon[32775]: pgmap v1189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:14:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:37 smithi164 ceph-mon[36970]: pgmap v1189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:14:37.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:37 smithi163 ceph-mon[36873]: pgmap v1189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:14:38.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:14:38.622+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:14:38.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:14:38.622+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:14:38.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:14:38.622+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:14:38.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:38 smithi042 ceph-mon[32775]: pgmap v1190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:14:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:38 smithi164 ceph-mon[36970]: pgmap v1190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:14:38.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:38 smithi163 ceph-mon[36873]: pgmap v1190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:14:40.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:40 smithi042 ceph-mon[32775]: pgmap v1191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:14:40.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:40 smithi164 ceph-mon[36970]: pgmap v1191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:14:41.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:40 smithi163 ceph-mon[36873]: pgmap v1191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:14:41.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1111.130241 secs 2024-07-26T07:14:41.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1128.873184 secs 2024-07-26T07:14:41.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1111.130241 secs 2024-07-26T07:14:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1128.873184 secs 2024-07-26T07:14:41.907 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1111.130241 secs 2024-07-26T07:14:41.907 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1128.873184 secs 2024-07-26T07:14:42.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:42 smithi042 ceph-mon[32775]: pgmap v1192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:42 smithi164 ceph-mon[36970]: pgmap v1192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:43.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:42 smithi163 ceph-mon[36873]: pgmap v1192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:44.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:44 smithi042 ceph-mon[32775]: pgmap v1193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:44 smithi164 ceph-mon[36970]: pgmap v1193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:45.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:44 smithi163 ceph-mon[36873]: pgmap v1193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:46.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:46 smithi042 ceph-mon[32775]: pgmap v1194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1116.130362 secs 2024-07-26T07:14:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1133.873324 secs 2024-07-26T07:14:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:46 smithi164 ceph-mon[36970]: pgmap v1194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1116.130362 secs 2024-07-26T07:14:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1133.873324 secs 2024-07-26T07:14:47.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:46 smithi163 ceph-mon[36873]: pgmap v1194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:47.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1116.130362 secs 2024-07-26T07:14:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1133.873324 secs 2024-07-26T07:14:48.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:48 smithi042 ceph-mon[32775]: pgmap v1195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:48.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:48 smithi164 ceph-mon[36970]: pgmap v1195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:49.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:48 smithi163 ceph-mon[36873]: pgmap v1195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:50 smithi042 ceph-mon[32775]: pgmap v1196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:50 smithi164 ceph-mon[36970]: pgmap v1196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:51.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:50 smithi163 ceph-mon[36873]: pgmap v1196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:51.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1121.130461 secs 2024-07-26T07:14:51.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1138.873444 secs 2024-07-26T07:14:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1121.130461 secs 2024-07-26T07:14:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1138.873444 secs 2024-07-26T07:14:52.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1121.130461 secs 2024-07-26T07:14:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1138.873444 secs 2024-07-26T07:14:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:52 smithi042 ceph-mon[32775]: pgmap v1197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:52.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:52 smithi164 ceph-mon[36970]: pgmap v1197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:52 smithi163 ceph-mon[36873]: pgmap v1197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:53.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:14:53.625+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:14:53.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:14:53.625+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:14:53.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:14:53.625+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:14:54.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:54 smithi042 ceph-mon[32775]: pgmap v1198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:54 smithi164 ceph-mon[36970]: pgmap v1198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:55.061 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:54 smithi163 ceph-mon[36873]: pgmap v1198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:14:56.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:56 smithi042 ceph-mon[32775]: pgmap v1199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1126.130559 secs 2024-07-26T07:14:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1143.873548 secs 2024-07-26T07:14:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:56 smithi164 ceph-mon[36970]: pgmap v1199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1126.130559 secs 2024-07-26T07:14:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1143.873548 secs 2024-07-26T07:14:57.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:56 smithi163 ceph-mon[36873]: pgmap v1199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:14:57.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1126.130559 secs 2024-07-26T07:14:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1143.873548 secs 2024-07-26T07:14:58.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:14:58 smithi042 ceph-mon[32775]: pgmap v1200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:14:58 smithi164 ceph-mon[36970]: pgmap v1200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:14:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:14:58 smithi163 ceph-mon[36873]: pgmap v1200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:00.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:00 smithi042 ceph-mon[32775]: pgmap v1201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:00 smithi164 ceph-mon[36970]: pgmap v1201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:01.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:00 smithi163 ceph-mon[36873]: pgmap v1201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:01.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1131.130685 secs 2024-07-26T07:15:01.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1148.873671 secs 2024-07-26T07:15:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1131.130685 secs 2024-07-26T07:15:01.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1148.873671 secs 2024-07-26T07:15:02.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1131.130685 secs 2024-07-26T07:15:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1148.873671 secs 2024-07-26T07:15:02.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:02 smithi042 ceph-mon[32775]: pgmap v1202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:02 smithi164 ceph-mon[36970]: pgmap v1202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:03.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:02 smithi163 ceph-mon[36873]: pgmap v1202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:04.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:04 smithi042 ceph-mon[32775]: pgmap v1203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:15:04.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:04 smithi164 ceph-mon[36970]: pgmap v1203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:15:05.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:04 smithi163 ceph-mon[36873]: pgmap v1203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:15:06.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:06 smithi042 ceph-mon[32775]: pgmap v1204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1136.130800 secs 2024-07-26T07:15:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1153.873794 secs 2024-07-26T07:15:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:06 smithi164 ceph-mon[36970]: pgmap v1204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1136.130800 secs 2024-07-26T07:15:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1153.873794 secs 2024-07-26T07:15:07.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:06 smithi163 ceph-mon[36873]: pgmap v1204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1136.130800 secs 2024-07-26T07:15:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1153.873794 secs 2024-07-26T07:15:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:15:08.633+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:15:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:15:08.633+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:15:08.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:15:08.633+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:15:08.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:08 smithi042 ceph-mon[32775]: pgmap v1205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:08.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:08 smithi164 ceph-mon[36970]: pgmap v1205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:09.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:08 smithi163 ceph-mon[36873]: pgmap v1205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:10.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:10 smithi042 ceph-mon[32775]: pgmap v1206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:10 smithi164 ceph-mon[36970]: pgmap v1206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:11.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:10 smithi163 ceph-mon[36873]: pgmap v1206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:11.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1141.130905 secs 2024-07-26T07:15:11.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1158.873954 secs 2024-07-26T07:15:11.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1141.130905 secs 2024-07-26T07:15:11.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1158.873954 secs 2024-07-26T07:15:11.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1141.130905 secs 2024-07-26T07:15:11.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1158.873954 secs 2024-07-26T07:15:12.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:12 smithi042 ceph-mon[32775]: pgmap v1207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:12 smithi164 ceph-mon[36970]: pgmap v1207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:13.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:12 smithi163 ceph-mon[36873]: pgmap v1207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:14.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:14 smithi042 ceph-mon[32775]: pgmap v1208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:15:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:14 smithi164 ceph-mon[36970]: pgmap v1208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:15:15.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:14 smithi163 ceph-mon[36873]: pgmap v1208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:15:16.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:16 smithi042 ceph-mon[32775]: pgmap v1209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1146.131055 secs 2024-07-26T07:15:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1163.874074 secs 2024-07-26T07:15:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:16 smithi164 ceph-mon[36970]: pgmap v1209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1146.131055 secs 2024-07-26T07:15:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1163.874074 secs 2024-07-26T07:15:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:16 smithi163 ceph-mon[36873]: pgmap v1209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1146.131055 secs 2024-07-26T07:15:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1163.874074 secs 2024-07-26T07:15:18.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:18 smithi042 ceph-mon[32775]: pgmap v1210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:18 smithi164 ceph-mon[36970]: pgmap v1210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:19.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:18 smithi163 ceph-mon[36873]: pgmap v1210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:20.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:20 smithi042 ceph-mon[32775]: pgmap v1211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:20 smithi164 ceph-mon[36970]: pgmap v1211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:20 smithi163 ceph-mon[36873]: pgmap v1211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:21.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1151.131201 secs 2024-07-26T07:15:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1168.874136 secs 2024-07-26T07:15:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1151.131201 secs 2024-07-26T07:15:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1168.874136 secs 2024-07-26T07:15:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1151.131201 secs 2024-07-26T07:15:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1168.874136 secs 2024-07-26T07:15:22.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:22 smithi042 ceph-mon[32775]: pgmap v1212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:22 smithi164 ceph-mon[36970]: pgmap v1212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:23.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:22 smithi163 ceph-mon[36873]: pgmap v1212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:23.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:15:23.635+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:15:23.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:15:23.635+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:15:23.636 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:15:23.635+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:15:24.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:24 smithi042 ceph-mon[32775]: pgmap v1213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:15:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:24 smithi164 ceph-mon[36970]: pgmap v1213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:15:25.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:24 smithi163 ceph-mon[36873]: pgmap v1213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:15:26.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:26 smithi042 ceph-mon[32775]: pgmap v1214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1156.131323 secs 2024-07-26T07:15:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1173.874251 secs 2024-07-26T07:15:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:26 smithi164 ceph-mon[36970]: pgmap v1214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1156.131323 secs 2024-07-26T07:15:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1173.874251 secs 2024-07-26T07:15:27.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:26 smithi163 ceph-mon[36873]: pgmap v1214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1156.131323 secs 2024-07-26T07:15:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1173.874251 secs 2024-07-26T07:15:28.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:28 smithi042 ceph-mon[32775]: pgmap v1215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:28.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:28 smithi164 ceph-mon[36970]: pgmap v1215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:29.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:28 smithi163 ceph-mon[36873]: pgmap v1215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:29.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:15:29.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:15:30.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:15:30.794 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:30 smithi042 ceph-mon[32775]: pgmap v1216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:30.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:30 smithi164 ceph-mon[36970]: pgmap v1216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:31.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:30 smithi163 ceph-mon[36873]: pgmap v1216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:31.779 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1161.131405 secs 2024-07-26T07:15:31.779 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1178.874384 secs 2024-07-26T07:15:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1161.131405 secs 2024-07-26T07:15:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1178.874384 secs 2024-07-26T07:15:32.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1161.131405 secs 2024-07-26T07:15:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1178.874384 secs 2024-07-26T07:15:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:32 smithi164 ceph-mon[36970]: pgmap v1217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:32.975 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:32 smithi042 ceph-mon[32775]: pgmap v1217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:32 smithi163 ceph-mon[36873]: pgmap v1217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:15:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:15:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:15:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:15:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:34 smithi164 ceph-mon[36970]: pgmap v1218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:15:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:34 smithi164 ceph-mon[36970]: pgmap v1219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:15:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:15:34.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:15:34.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:15:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:15:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:15:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:34 smithi163 ceph-mon[36873]: pgmap v1218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:15:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:34 smithi163 ceph-mon[36873]: pgmap v1219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:15:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:15:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:15:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:15:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:15:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:15:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:34 smithi042 ceph-mon[32775]: pgmap v1218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:15:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:34 smithi042 ceph-mon[32775]: pgmap v1219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:15:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:15:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:36 smithi163 ceph-mon[36873]: pgmap v1220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:15:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1166.131520 secs 2024-07-26T07:15:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1183.874534 secs 2024-07-26T07:15:36.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:36 smithi042 ceph-mon[32775]: pgmap v1220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:15:36.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1166.131520 secs 2024-07-26T07:15:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1183.874534 secs 2024-07-26T07:15:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:36 smithi164 ceph-mon[36970]: pgmap v1220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:15:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1166.131520 secs 2024-07-26T07:15:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1183.874534 secs 2024-07-26T07:15:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:15:38.642+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:15:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:15:38.642+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:15:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:15:38.642+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:15:38.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:38 smithi163 ceph-mon[36873]: pgmap v1221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:15:38.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:38 smithi042 ceph-mon[32775]: pgmap v1221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:15:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:38 smithi164 ceph-mon[36970]: pgmap v1221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:15:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:40 smithi163 ceph-mon[36873]: pgmap v1222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:15:40.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:40 smithi042 ceph-mon[32775]: pgmap v1222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:15:40.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:40 smithi164 ceph-mon[36970]: pgmap v1222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:15:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1171.131603 secs 2024-07-26T07:15:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1188.874650 secs 2024-07-26T07:15:41.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1171.131603 secs 2024-07-26T07:15:41.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1188.874650 secs 2024-07-26T07:15:41.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1171.131603 secs 2024-07-26T07:15:41.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1188.874650 secs 2024-07-26T07:15:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:42 smithi163 ceph-mon[36873]: pgmap v1223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:15:42.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:42 smithi042 ceph-mon[32775]: pgmap v1223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:15:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:42 smithi164 ceph-mon[36970]: pgmap v1223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:15:44.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:44 smithi042 ceph-mon[32775]: pgmap v1224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:15:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:44 smithi164 ceph-mon[36970]: pgmap v1224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:15:45.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:44 smithi163 ceph-mon[36873]: pgmap v1224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:15:46.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:46 smithi042 ceph-mon[32775]: pgmap v1225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:46.793 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1176.131748 secs 2024-07-26T07:15:46.794 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1193.874777 secs 2024-07-26T07:15:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:46 smithi164 ceph-mon[36970]: pgmap v1225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1176.131748 secs 2024-07-26T07:15:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1193.874777 secs 2024-07-26T07:15:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:46 smithi163 ceph-mon[36873]: pgmap v1225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1176.131748 secs 2024-07-26T07:15:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1193.874777 secs 2024-07-26T07:15:48.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:48 smithi042 ceph-mon[32775]: pgmap v1226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:48.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:48 smithi164 ceph-mon[36970]: pgmap v1226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:49.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:48 smithi163 ceph-mon[36873]: pgmap v1226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:50 smithi042 ceph-mon[32775]: pgmap v1227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:50 smithi164 ceph-mon[36970]: pgmap v1227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:51.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:50 smithi163 ceph-mon[36873]: pgmap v1227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:51.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1181.131896 secs 2024-07-26T07:15:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1198.874899 secs 2024-07-26T07:15:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1181.131896 secs 2024-07-26T07:15:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1198.874899 secs 2024-07-26T07:15:52.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1181.131896 secs 2024-07-26T07:15:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1198.874899 secs 2024-07-26T07:15:52.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:52 smithi042 ceph-mon[32775]: pgmap v1228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:52.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:52 smithi164 ceph-mon[36970]: pgmap v1228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:53.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:52 smithi163 ceph-mon[36873]: pgmap v1228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:53.645 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:15:53.645+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:15:53.645 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:15:53.645+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:15:53.645 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:15:53.645+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:15:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:54 smithi042 ceph-mon[32775]: pgmap v1229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:15:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:54 smithi164 ceph-mon[36970]: pgmap v1229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:15:55.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:54 smithi163 ceph-mon[36873]: pgmap v1229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:15:56.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:56 smithi042 ceph-mon[32775]: pgmap v1230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1186.132015 secs 2024-07-26T07:15:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1203.875021 secs 2024-07-26T07:15:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:56 smithi164 ceph-mon[36970]: pgmap v1230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1186.132015 secs 2024-07-26T07:15:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1203.875021 secs 2024-07-26T07:15:57.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:56 smithi163 ceph-mon[36873]: pgmap v1230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:15:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1186.132015 secs 2024-07-26T07:15:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1203.875021 secs 2024-07-26T07:15:58.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:15:58 smithi042 ceph-mon[32775]: pgmap v1231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:15:58 smithi164 ceph-mon[36970]: pgmap v1231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:15:59.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:15:58 smithi163 ceph-mon[36873]: pgmap v1231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:00.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:00 smithi042 ceph-mon[32775]: pgmap v1232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:16:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:00 smithi164 ceph-mon[36970]: pgmap v1232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:16:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:00 smithi163 ceph-mon[36873]: pgmap v1232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:16:01.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1191.132142 secs 2024-07-26T07:16:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1208.875149 secs 2024-07-26T07:16:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1191.132142 secs 2024-07-26T07:16:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1208.875149 secs 2024-07-26T07:16:02.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1191.132142 secs 2024-07-26T07:16:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1208.875149 secs 2024-07-26T07:16:02.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:02 smithi042 ceph-mon[32775]: pgmap v1233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:16:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:02 smithi164 ceph-mon[36970]: pgmap v1233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:16:03.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:02 smithi163 ceph-mon[36873]: pgmap v1233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:16:04.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:04 smithi042 ceph-mon[32775]: pgmap v1234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-07-26T07:16:04.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:04 smithi164 ceph-mon[36970]: pgmap v1234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-07-26T07:16:05.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:04 smithi163 ceph-mon[36873]: pgmap v1234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-07-26T07:16:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:06 smithi042 ceph-mon[32775]: pgmap v1235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:16:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1196.132269 secs 2024-07-26T07:16:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1213.875269 secs 2024-07-26T07:16:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:06 smithi164 ceph-mon[36970]: pgmap v1235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:16:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1196.132269 secs 2024-07-26T07:16:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1213.875269 secs 2024-07-26T07:16:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:06 smithi163 ceph-mon[36873]: pgmap v1235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:16:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1196.132269 secs 2024-07-26T07:16:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1213.875269 secs 2024-07-26T07:16:08.652 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:16:08.653+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:16:08.653 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:16:08.653+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:16:08.653 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:16:08.653+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:16:08.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:08 smithi042 ceph-mon[32775]: pgmap v1236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:16:08.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:08 smithi164 ceph-mon[36970]: pgmap v1236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:16:09.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:08 smithi163 ceph-mon[36873]: pgmap v1236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:16:10.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:10 smithi042 ceph-mon[32775]: pgmap v1237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:16:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:10 smithi164 ceph-mon[36970]: pgmap v1237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:16:11.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:10 smithi163 ceph-mon[36873]: pgmap v1237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:16:11.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1201.132414 secs 2024-07-26T07:16:11.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1218.875413 secs 2024-07-26T07:16:11.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1201.132414 secs 2024-07-26T07:16:11.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1218.875413 secs 2024-07-26T07:16:11.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1201.132414 secs 2024-07-26T07:16:11.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1218.875413 secs 2024-07-26T07:16:12.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:12 smithi042 ceph-mon[32775]: pgmap v1238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:12 smithi164 ceph-mon[36970]: pgmap v1238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:13.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:12 smithi163 ceph-mon[36873]: pgmap v1238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:14.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:14 smithi042 ceph-mon[32775]: pgmap v1239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:16:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:14 smithi164 ceph-mon[36970]: pgmap v1239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:16:15.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:14 smithi163 ceph-mon[36873]: pgmap v1239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:16:16.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:16 smithi042 ceph-mon[32775]: pgmap v1240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1206.132552 secs 2024-07-26T07:16:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1223.875542 secs 2024-07-26T07:16:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:16 smithi164 ceph-mon[36970]: pgmap v1240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1206.132552 secs 2024-07-26T07:16:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1223.875542 secs 2024-07-26T07:16:17.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:16 smithi163 ceph-mon[36873]: pgmap v1240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:17.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1206.132552 secs 2024-07-26T07:16:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1223.875542 secs 2024-07-26T07:16:18.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:18 smithi042 ceph-mon[32775]: pgmap v1241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:18 smithi164 ceph-mon[36970]: pgmap v1241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:19.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:18 smithi163 ceph-mon[36873]: pgmap v1241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:20.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:20 smithi042 ceph-mon[32775]: pgmap v1242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:20 smithi164 ceph-mon[36970]: pgmap v1242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:21.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:20 smithi163 ceph-mon[36873]: pgmap v1242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:21.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1211.132698 secs 2024-07-26T07:16:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1228.875663 secs 2024-07-26T07:16:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1211.132698 secs 2024-07-26T07:16:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1228.875663 secs 2024-07-26T07:16:22.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1211.132698 secs 2024-07-26T07:16:22.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1228.875663 secs 2024-07-26T07:16:22.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:22 smithi042 ceph-mon[32775]: pgmap v1243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:22 smithi164 ceph-mon[36970]: pgmap v1243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:23.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:22 smithi163 ceph-mon[36873]: pgmap v1243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:16:23.656+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:16:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:16:23.656+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:16:23.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:16:23.656+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:16:24.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:24 smithi042 ceph-mon[32775]: pgmap v1244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:16:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:24 smithi164 ceph-mon[36970]: pgmap v1244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:16:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:24 smithi163 ceph-mon[36873]: pgmap v1244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:16:26.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:26 smithi042 ceph-mon[32775]: pgmap v1245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1216.132818 secs 2024-07-26T07:16:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1233.875747 secs 2024-07-26T07:16:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:26 smithi164 ceph-mon[36970]: pgmap v1245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1216.132818 secs 2024-07-26T07:16:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1233.875747 secs 2024-07-26T07:16:27.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:26 smithi163 ceph-mon[36873]: pgmap v1245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1216.132818 secs 2024-07-26T07:16:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1233.875747 secs 2024-07-26T07:16:28.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:28 smithi042 ceph-mon[32775]: pgmap v1246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:28 smithi164 ceph-mon[36970]: pgmap v1246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:29.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:28 smithi163 ceph-mon[36873]: pgmap v1246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:30.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:30 smithi042 ceph-mon[32775]: pgmap v1247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:30.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:30 smithi164 ceph-mon[36970]: pgmap v1247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:31.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:30 smithi163 ceph-mon[36873]: pgmap v1247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:31.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1221.132963 secs 2024-07-26T07:16:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1238.875868 secs 2024-07-26T07:16:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1221.132963 secs 2024-07-26T07:16:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1238.875868 secs 2024-07-26T07:16:32.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1221.132963 secs 2024-07-26T07:16:32.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1238.875868 secs 2024-07-26T07:16:32.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:32 smithi042 ceph-mon[32775]: pgmap v1248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:32 smithi164 ceph-mon[36970]: pgmap v1248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:33.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:32 smithi163 ceph-mon[36873]: pgmap v1248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:33.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:16:33.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:16:34.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:16:34.688 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:34 smithi164 ceph-mon[36970]: pgmap v1249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:16:34.688 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:34.688 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:34.689 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:34.689 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:34.689 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:34 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:16:34.689 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:34 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:16:34.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:34 smithi163 ceph-mon[36873]: pgmap v1249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:16:34.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:34.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:34.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:34.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:34.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:34 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:16:34.691 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:34 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:16:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:34 smithi042 ceph-mon[32775]: pgmap v1249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:16:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:34 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:16:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:34 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:16:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:36 smithi164 ceph-mon[36970]: pgmap v1250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1226.133060 secs 2024-07-26T07:16:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1243.875971 secs 2024-07-26T07:16:36.873 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:36 smithi163 ceph-mon[36873]: pgmap v1250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:36.873 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1226.133060 secs 2024-07-26T07:16:36.874 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1243.875971 secs 2024-07-26T07:16:36.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:36 smithi042 ceph-mon[32775]: pgmap v1250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:36.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1226.133060 secs 2024-07-26T07:16:36.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1243.875971 secs 2024-07-26T07:16:38.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:16:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:16:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:38 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:16:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:38 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:16:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:16:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:16:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:16:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:16:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:16:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:16:38.664+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:16:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:16:38.664+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:16:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:16:38.664+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:16:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:16:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:16:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:38 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:16:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:38 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:16:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:16:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:16:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:16:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:16:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:16:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:16:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:16:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:38 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:16:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:38 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:16:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:16:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:16:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:16:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:16:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:16:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:16:39.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:39 smithi042 ceph-mon[32775]: pgmap v1251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:39 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:16:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:39 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:16:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:39 smithi042 ceph-mon[32775]: pgmap v1252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:16:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:39 smithi042 ceph-mon[32775]: pgmap v1253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:16:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:39 smithi164 ceph-mon[36970]: pgmap v1251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:39 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:16:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:39 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:16:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:39 smithi164 ceph-mon[36970]: pgmap v1252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:16:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:39 smithi164 ceph-mon[36970]: pgmap v1253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:16:39.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:39 smithi163 ceph-mon[36873]: pgmap v1251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:39 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:16:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:39 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:16:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:39 smithi163 ceph-mon[36873]: pgmap v1252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:16:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:39 smithi163 ceph-mon[36873]: pgmap v1253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:16:41.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:41 smithi042 ceph-mon[32775]: pgmap v1254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1231.133202 secs 2024-07-26T07:16:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1248.876092 secs 2024-07-26T07:16:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:41 smithi164 ceph-mon[36970]: pgmap v1254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1231.133202 secs 2024-07-26T07:16:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1248.876092 secs 2024-07-26T07:16:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:41 smithi163 ceph-mon[36873]: pgmap v1254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1231.133202 secs 2024-07-26T07:16:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1248.876092 secs 2024-07-26T07:16:43.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:43 smithi042 ceph-mon[32775]: pgmap v1255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3 op/s 2024-07-26T07:16:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:43 smithi164 ceph-mon[36970]: pgmap v1255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3 op/s 2024-07-26T07:16:43.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:43 smithi163 ceph-mon[36873]: pgmap v1255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3 op/s 2024-07-26T07:16:45.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:45 smithi042 ceph-mon[32775]: pgmap v1256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:16:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:45 smithi164 ceph-mon[36970]: pgmap v1256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:16:45.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:45 smithi163 ceph-mon[36873]: pgmap v1256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:16:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1236.133317 secs 2024-07-26T07:16:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1253.876234 secs 2024-07-26T07:16:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1236.133317 secs 2024-07-26T07:16:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1253.876234 secs 2024-07-26T07:16:46.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1236.133317 secs 2024-07-26T07:16:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1253.876234 secs 2024-07-26T07:16:47.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:47 smithi042 ceph-mon[32775]: pgmap v1257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:47 smithi164 ceph-mon[36970]: pgmap v1257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:47.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:47 smithi163 ceph-mon[36873]: pgmap v1257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:49.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:49 smithi042 ceph-mon[32775]: pgmap v1258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:16:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:49 smithi164 ceph-mon[36970]: pgmap v1258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:16:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:49 smithi163 ceph-mon[36873]: pgmap v1258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:16:51.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:51 smithi042 ceph-mon[32775]: pgmap v1259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1241.133437 secs 2024-07-26T07:16:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1258.876359 secs 2024-07-26T07:16:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:51 smithi164 ceph-mon[36970]: pgmap v1259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1241.133437 secs 2024-07-26T07:16:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1258.876359 secs 2024-07-26T07:16:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:51 smithi163 ceph-mon[36873]: pgmap v1259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1241.133437 secs 2024-07-26T07:16:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1258.876359 secs 2024-07-26T07:16:53.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:53 smithi042 ceph-mon[32775]: pgmap v1260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:53.667 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:16:53.667+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:16:53.667 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:16:53.667+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:16:53.667 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:16:53.667+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:16:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:53 smithi164 ceph-mon[36970]: pgmap v1260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:53.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:53 smithi163 ceph-mon[36873]: pgmap v1260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:55.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:55 smithi042 ceph-mon[32775]: pgmap v1261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:16:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:55 smithi164 ceph-mon[36970]: pgmap v1261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:16:55.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:55 smithi163 ceph-mon[36873]: pgmap v1261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:16:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1246.133575 secs 2024-07-26T07:16:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1263.876480 secs 2024-07-26T07:16:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1246.133575 secs 2024-07-26T07:16:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1263.876480 secs 2024-07-26T07:16:56.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1246.133575 secs 2024-07-26T07:16:56.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1263.876480 secs 2024-07-26T07:16:57.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:57 smithi042 ceph-mon[32775]: pgmap v1262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:57 smithi164 ceph-mon[36970]: pgmap v1262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:57.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:57 smithi163 ceph-mon[36873]: pgmap v1262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:16:58.056 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:16:58.055+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T07:16:58.056 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:16:58.055+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T07:16:58.057 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:16:58.056+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T07:16:58.057 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:16:58.056+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T07:16:59.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:16:59 smithi042 ceph-mon[32775]: pgmap v1263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:59.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:16:59 smithi164 ceph-mon[36970]: pgmap v1263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:16:59.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:16:59 smithi163 ceph-mon[36873]: pgmap v1263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:01.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:01 smithi042 ceph-mon[32775]: pgmap v1264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1251.133651 secs 2024-07-26T07:17:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1268.876611 secs 2024-07-26T07:17:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:01 smithi164 ceph-mon[36970]: pgmap v1264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1251.133651 secs 2024-07-26T07:17:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1268.876611 secs 2024-07-26T07:17:01.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:01 smithi163 ceph-mon[36873]: pgmap v1264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:01.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1251.133651 secs 2024-07-26T07:17:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1268.876611 secs 2024-07-26T07:17:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:03 smithi042 ceph-mon[32775]: pgmap v1265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:03.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:03 smithi164 ceph-mon[36970]: pgmap v1265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:03.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:03 smithi163 ceph-mon[36873]: pgmap v1265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:05 smithi042 ceph-mon[32775]: pgmap v1266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:17:05.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:05 smithi164 ceph-mon[36970]: pgmap v1266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:17:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:05 smithi163 ceph-mon[36873]: pgmap v1266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:17:06.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:06 smithi042 ceph-mon[32775]: pgmap v1267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1256.133793 secs 2024-07-26T07:17:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1273.876712 secs 2024-07-26T07:17:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:06 smithi164 ceph-mon[36970]: pgmap v1267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1256.133793 secs 2024-07-26T07:17:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1273.876712 secs 2024-07-26T07:17:06.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:06 smithi163 ceph-mon[36873]: pgmap v1267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1256.133793 secs 2024-07-26T07:17:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1273.876712 secs 2024-07-26T07:17:08.676 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:17:08.675+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:17:08.676 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:17:08.675+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:17:08.676 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:17:08.675+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:17:08.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:08 smithi042 ceph-mon[32775]: pgmap v1268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:08.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:08 smithi164 ceph-mon[36970]: pgmap v1268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:09.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:08 smithi163 ceph-mon[36873]: pgmap v1268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:10.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:10 smithi042 ceph-mon[32775]: pgmap v1269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:10 smithi164 ceph-mon[36970]: pgmap v1269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:11.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:10 smithi163 ceph-mon[36873]: pgmap v1269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:11.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1261.133881 secs 2024-07-26T07:17:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1278.876817 secs 2024-07-26T07:17:11.899 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1261.133881 secs 2024-07-26T07:17:11.899 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1278.876817 secs 2024-07-26T07:17:11.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1261.133881 secs 2024-07-26T07:17:11.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1278.876817 secs 2024-07-26T07:17:12.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:12 smithi042 ceph-mon[32775]: pgmap v1270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:12 smithi164 ceph-mon[36970]: pgmap v1270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:13.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:12 smithi163 ceph-mon[36873]: pgmap v1270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:14.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:14 smithi042 ceph-mon[32775]: pgmap v1271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:17:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:14 smithi164 ceph-mon[36970]: pgmap v1271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:17:15.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:14 smithi163 ceph-mon[36873]: pgmap v1271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:17:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:16 smithi164 ceph-mon[36970]: pgmap v1272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1266.134026 secs 2024-07-26T07:17:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1283.876960 secs 2024-07-26T07:17:17.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:16 smithi163 ceph-mon[36873]: pgmap v1272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1266.134026 secs 2024-07-26T07:17:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1283.876960 secs 2024-07-26T07:17:17.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:16 smithi042 ceph-mon[32775]: pgmap v1272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:17.171 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1266.134026 secs 2024-07-26T07:17:17.171 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1283.876960 secs 2024-07-26T07:17:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:18 smithi164 ceph-mon[36970]: pgmap v1273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:19.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:18 smithi163 ceph-mon[36873]: pgmap v1273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:19.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:18 smithi042 ceph-mon[32775]: pgmap v1273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:20 smithi164 ceph-mon[36970]: pgmap v1274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:21.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:20 smithi163 ceph-mon[36873]: pgmap v1274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:21.007 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:20 smithi042 ceph-mon[32775]: pgmap v1274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1271.134172 secs 2024-07-26T07:17:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1288.877037 secs 2024-07-26T07:17:22.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1271.134172 secs 2024-07-26T07:17:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1288.877037 secs 2024-07-26T07:17:22.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1271.134172 secs 2024-07-26T07:17:22.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1288.877037 secs 2024-07-26T07:17:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:22 smithi164 ceph-mon[36970]: pgmap v1275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:22 smithi163 ceph-mon[36873]: pgmap v1275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:23.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:22 smithi042 ceph-mon[32775]: pgmap v1275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:17:23.678+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:17:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:17:23.678+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:17:23.678 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:17:23.678+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:17:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:24 smithi164 ceph-mon[36970]: pgmap v1276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:17:25.036 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:24 smithi163 ceph-mon[36873]: pgmap v1276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:17:25.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:24 smithi042 ceph-mon[32775]: pgmap v1276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:17:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:26 smithi164 ceph-mon[36970]: pgmap v1277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1276.134312 secs 2024-07-26T07:17:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1293.877158 secs 2024-07-26T07:17:27.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:26 smithi163 ceph-mon[36873]: pgmap v1277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1276.134312 secs 2024-07-26T07:17:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1293.877158 secs 2024-07-26T07:17:27.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:26 smithi042 ceph-mon[32775]: pgmap v1277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1276.134312 secs 2024-07-26T07:17:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1293.877158 secs 2024-07-26T07:17:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:28 smithi164 ceph-mon[36970]: pgmap v1278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:29.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:28 smithi163 ceph-mon[36873]: pgmap v1278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:29.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:28 smithi042 ceph-mon[32775]: pgmap v1278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:30.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:30 smithi164 ceph-mon[36970]: pgmap v1279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:31.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:30 smithi163 ceph-mon[36873]: pgmap v1279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:31.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:30 smithi042 ceph-mon[32775]: pgmap v1279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1281.134383 secs 2024-07-26T07:17:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1298.877223 secs 2024-07-26T07:17:32.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1281.134383 secs 2024-07-26T07:17:32.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1298.877223 secs 2024-07-26T07:17:32.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1281.134383 secs 2024-07-26T07:17:32.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1298.877223 secs 2024-07-26T07:17:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:32 smithi164 ceph-mon[36970]: pgmap v1280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:33.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:32 smithi163 ceph-mon[36873]: pgmap v1280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:33.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:32 smithi042 ceph-mon[32775]: pgmap v1280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:34.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:34 smithi164 ceph-mon[36970]: pgmap v1281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:17:35.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:34 smithi163 ceph-mon[36873]: pgmap v1281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:17:35.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:34 smithi042 ceph-mon[32775]: pgmap v1281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:17:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:36 smithi164 ceph-mon[36970]: pgmap v1282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1286.134533 secs 2024-07-26T07:17:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1303.877317 secs 2024-07-26T07:17:37.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:36 smithi163 ceph-mon[36873]: pgmap v1282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:37.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1286.134533 secs 2024-07-26T07:17:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1303.877317 secs 2024-07-26T07:17:37.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:36 smithi042 ceph-mon[32775]: pgmap v1282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:37.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1286.134533 secs 2024-07-26T07:17:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1303.877317 secs 2024-07-26T07:17:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:17:38.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:17:38.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:17:38.685 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:17:38.685+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:17:38.685 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:17:38.685+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:17:38.685 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:17:38.685+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:17:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:38 smithi164 ceph-mon[36970]: pgmap v1283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:17:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:17:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:38 smithi164 ceph-mon[36970]: pgmap v1284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:17:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:38 smithi164 ceph-mon[36970]: pgmap v1285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:17:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:17:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:38 smithi163 ceph-mon[36873]: pgmap v1283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:17:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:17:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:38 smithi163 ceph-mon[36873]: pgmap v1284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:17:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:38 smithi163 ceph-mon[36873]: pgmap v1285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:17:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:17:39.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:38 smithi042 ceph-mon[32775]: pgmap v1283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:17:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:17:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:38 smithi042 ceph-mon[32775]: pgmap v1284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:17:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:38 smithi042 ceph-mon[32775]: pgmap v1285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:17:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:17:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:41 smithi164 ceph-mon[36970]: pgmap v1286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:17:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1291.134680 secs 2024-07-26T07:17:41.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:41 smithi163 ceph-mon[36873]: pgmap v1286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:17:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1291.134680 secs 2024-07-26T07:17:41.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:41 smithi042 ceph-mon[32775]: pgmap v1286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:17:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1291.134680 secs 2024-07-26T07:17:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1308.877466 secs 2024-07-26T07:17:42.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1308.877466 secs 2024-07-26T07:17:42.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1308.877466 secs 2024-07-26T07:17:43.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:43 smithi164 ceph-mon[36970]: pgmap v1287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:17:43.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:43 smithi163 ceph-mon[36873]: pgmap v1287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:17:43.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:43 smithi042 ceph-mon[32775]: pgmap v1287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:17:45.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:45 smithi164 ceph-mon[36970]: pgmap v1288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:17:45.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:45 smithi163 ceph-mon[36873]: pgmap v1288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:17:45.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:45 smithi042 ceph-mon[32775]: pgmap v1288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:17:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1296.134819 secs 2024-07-26T07:17:46.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1296.134819 secs 2024-07-26T07:17:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1296.134819 secs 2024-07-26T07:17:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1313.877563 secs 2024-07-26T07:17:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:47 smithi164 ceph-mon[36970]: pgmap v1289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:17:47.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1313.877563 secs 2024-07-26T07:17:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:47 smithi163 ceph-mon[36873]: pgmap v1289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:17:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1313.877563 secs 2024-07-26T07:17:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:47 smithi042 ceph-mon[32775]: pgmap v1289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:17:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:49 smithi164 ceph-mon[36970]: pgmap v1290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:17:49.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:49 smithi163 ceph-mon[36873]: pgmap v1290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:17:49.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:49 smithi042 ceph-mon[32775]: pgmap v1290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:17:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:51 smithi163 ceph-mon[36873]: pgmap v1291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1301.134940 secs 2024-07-26T07:17:51.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:51 smithi042 ceph-mon[32775]: pgmap v1291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:51.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1301.134940 secs 2024-07-26T07:17:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:51 smithi164 ceph-mon[36970]: pgmap v1291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1301.134940 secs 2024-07-26T07:17:52.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1318.877686 secs 2024-07-26T07:17:52.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1318.877686 secs 2024-07-26T07:17:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1318.877686 secs 2024-07-26T07:17:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:53 smithi163 ceph-mon[36873]: pgmap v1292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:53.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:53 smithi042 ceph-mon[32775]: pgmap v1292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:17:53.689+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:17:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:17:53.689+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:17:53.689 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:17:53.689+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:17:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:53 smithi164 ceph-mon[36970]: pgmap v1292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:55 smithi163 ceph-mon[36873]: pgmap v1293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:55.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:55 smithi042 ceph-mon[32775]: pgmap v1293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:55 smithi164 ceph-mon[36970]: pgmap v1293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:17:56.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1306.135038 secs 2024-07-26T07:17:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1306.135038 secs 2024-07-26T07:17:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1306.135038 secs 2024-07-26T07:17:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1323.877823 secs 2024-07-26T07:17:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:57 smithi163 ceph-mon[36873]: pgmap v1294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:57.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1323.877823 secs 2024-07-26T07:17:57.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:57 smithi042 ceph-mon[32775]: pgmap v1294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1323.877823 secs 2024-07-26T07:17:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:57 smithi164 ceph-mon[36970]: pgmap v1294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:59.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:17:59 smithi163 ceph-mon[36873]: pgmap v1295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:59.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:17:59 smithi042 ceph-mon[32775]: pgmap v1295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:17:59.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:17:59 smithi164 ceph-mon[36970]: pgmap v1295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:01 smithi163 ceph-mon[36873]: pgmap v1296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1311.135169 secs 2024-07-26T07:18:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1328.877940 secs 2024-07-26T07:18:01.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:01 smithi042 ceph-mon[32775]: pgmap v1296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1311.135169 secs 2024-07-26T07:18:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1328.877940 secs 2024-07-26T07:18:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:01 smithi164 ceph-mon[36970]: pgmap v1296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1311.135169 secs 2024-07-26T07:18:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1328.877940 secs 2024-07-26T07:18:03.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:03 smithi163 ceph-mon[36873]: pgmap v1297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:03.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:03 smithi042 ceph-mon[32775]: pgmap v1297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:03.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:03 smithi164 ceph-mon[36970]: pgmap v1297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:05.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:05 smithi163 ceph-mon[36873]: pgmap v1298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:05.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:05 smithi042 ceph-mon[32775]: pgmap v1298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:05.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:05 smithi164 ceph-mon[36970]: pgmap v1298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:06.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1316.135230 secs 2024-07-26T07:18:06.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1333.878062 secs 2024-07-26T07:18:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1316.135230 secs 2024-07-26T07:18:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1333.878062 secs 2024-07-26T07:18:06.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1316.135230 secs 2024-07-26T07:18:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1333.878062 secs 2024-07-26T07:18:07.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:07 smithi042 ceph-mon[32775]: pgmap v1299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:07 smithi164 ceph-mon[36970]: pgmap v1299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:07.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:07 smithi163 ceph-mon[36873]: pgmap v1299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:18:08.696+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:18:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:18:08.696+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:18:08.697 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:18:08.696+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:18:09.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:09 smithi042 ceph-mon[32775]: pgmap v1300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:09 smithi164 ceph-mon[36970]: pgmap v1300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:09.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:09 smithi163 ceph-mon[36873]: pgmap v1300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:11.524 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:11 smithi164 ceph-mon[36970]: pgmap v1301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:11.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1321.135380 secs 2024-07-26T07:18:11.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1338.878182 secs 2024-07-26T07:18:11.558 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:11 smithi163 ceph-mon[36873]: pgmap v1301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:11.559 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1321.135380 secs 2024-07-26T07:18:11.559 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1338.878182 secs 2024-07-26T07:18:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:11 smithi042 ceph-mon[32775]: pgmap v1301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1321.135380 secs 2024-07-26T07:18:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1338.878182 secs 2024-07-26T07:18:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:13 smithi042 ceph-mon[32775]: pgmap v1302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:13.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:13 smithi164 ceph-mon[36970]: pgmap v1302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:13.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:13 smithi163 ceph-mon[36873]: pgmap v1302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:15.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:15 smithi042 ceph-mon[32775]: pgmap v1303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:15.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:15 smithi164 ceph-mon[36970]: pgmap v1303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:15.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:15 smithi163 ceph-mon[36873]: pgmap v1303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1326.135531 secs 2024-07-26T07:18:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1343.878302 secs 2024-07-26T07:18:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1326.135531 secs 2024-07-26T07:18:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1343.878302 secs 2024-07-26T07:18:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1326.135531 secs 2024-07-26T07:18:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1343.878302 secs 2024-07-26T07:18:17.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:17 smithi042 ceph-mon[32775]: pgmap v1304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:17 smithi164 ceph-mon[36970]: pgmap v1304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:17.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:17 smithi163 ceph-mon[36873]: pgmap v1304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:19.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:19 smithi042 ceph-mon[32775]: pgmap v1305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:19.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:19 smithi164 ceph-mon[36970]: pgmap v1305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:19.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:19 smithi163 ceph-mon[36873]: pgmap v1305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:21.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:21 smithi042 ceph-mon[32775]: pgmap v1306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:18:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1331.135613 secs 2024-07-26T07:18:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1348.878428 secs 2024-07-26T07:18:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:21 smithi164 ceph-mon[36970]: pgmap v1306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:18:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1331.135613 secs 2024-07-26T07:18:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1348.878428 secs 2024-07-26T07:18:21.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:21 smithi163 ceph-mon[36873]: pgmap v1306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:18:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1331.135613 secs 2024-07-26T07:18:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1348.878428 secs 2024-07-26T07:18:23.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:23 smithi042 ceph-mon[32775]: pgmap v1307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:18:23.700 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:18:23.699+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:18:23.700 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:18:23.699+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:18:23.700 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:18:23.699+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:18:23.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:23 smithi164 ceph-mon[36970]: pgmap v1307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:18:23.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:23 smithi163 ceph-mon[36873]: pgmap v1307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:18:25.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:25 smithi042 ceph-mon[32775]: pgmap v1308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:18:25.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:25 smithi164 ceph-mon[36970]: pgmap v1308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:18:25.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:25 smithi163 ceph-mon[36873]: pgmap v1308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:18:26.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1336.135728 secs 2024-07-26T07:18:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1353.878559 secs 2024-07-26T07:18:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1336.135728 secs 2024-07-26T07:18:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1353.878559 secs 2024-07-26T07:18:26.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1336.135728 secs 2024-07-26T07:18:26.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1353.878559 secs 2024-07-26T07:18:27.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:27 smithi042 ceph-mon[32775]: pgmap v1309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:18:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:27 smithi164 ceph-mon[36970]: pgmap v1309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:18:27.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:27 smithi163 ceph-mon[36873]: pgmap v1309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:18:29.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:29 smithi042 ceph-mon[32775]: pgmap v1310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:18:29.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:29 smithi164 ceph-mon[36970]: pgmap v1310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:18:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:29 smithi163 ceph-mon[36873]: pgmap v1310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:18:31.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:31 smithi042 ceph-mon[32775]: pgmap v1311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:18:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1341.135852 secs 2024-07-26T07:18:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1358.878701 secs 2024-07-26T07:18:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:31 smithi164 ceph-mon[36970]: pgmap v1311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:18:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1341.135852 secs 2024-07-26T07:18:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1358.878701 secs 2024-07-26T07:18:31.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:31 smithi163 ceph-mon[36873]: pgmap v1311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:18:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1341.135852 secs 2024-07-26T07:18:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1358.878701 secs 2024-07-26T07:18:33.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:33 smithi042 ceph-mon[32775]: pgmap v1312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:33.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:33 smithi164 ceph-mon[36970]: pgmap v1312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:33.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:33 smithi163 ceph-mon[36873]: pgmap v1312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:35.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:35 smithi042 ceph-mon[32775]: pgmap v1313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:35.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:35 smithi164 ceph-mon[36970]: pgmap v1313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:35.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:35 smithi163 ceph-mon[36873]: pgmap v1313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:36.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1346.135919 secs 2024-07-26T07:18:36.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1363.878782 secs 2024-07-26T07:18:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1346.135919 secs 2024-07-26T07:18:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1363.878782 secs 2024-07-26T07:18:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1346.135919 secs 2024-07-26T07:18:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1363.878782 secs 2024-07-26T07:18:37.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:37 smithi042 ceph-mon[32775]: pgmap v1314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:37 smithi164 ceph-mon[36970]: pgmap v1314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:37.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:37 smithi163 ceph-mon[36873]: pgmap v1314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:38.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:18:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:18:38.706+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:18:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:18:38.706+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:18:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:18:38.706+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:18:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:18:38.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:18:39.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:39 smithi042 ceph-mon[32775]: pgmap v1315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:39.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:18:39.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:18:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:18:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:39 smithi164 ceph-mon[36970]: pgmap v1315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:18:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:18:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:18:39.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:39 smithi163 ceph-mon[36873]: pgmap v1315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:18:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:18:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:18:40.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:40 smithi042 ceph-mon[32775]: pgmap v1316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:18:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:40 smithi042 ceph-mon[32775]: pgmap v1317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:18:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:40 smithi164 ceph-mon[36970]: pgmap v1316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:18:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:40 smithi164 ceph-mon[36970]: pgmap v1317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:18:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:40 smithi163 ceph-mon[36873]: pgmap v1316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:18:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:40 smithi163 ceph-mon[36873]: pgmap v1317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:18:41.623 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1351.136041 secs 2024-07-26T07:18:41.623 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1368.878901 secs 2024-07-26T07:18:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1351.136041 secs 2024-07-26T07:18:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1368.878901 secs 2024-07-26T07:18:41.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1351.136041 secs 2024-07-26T07:18:41.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1368.878901 secs 2024-07-26T07:18:42.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:42 smithi042 ceph-mon[32775]: pgmap v1318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:18:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:42 smithi164 ceph-mon[36970]: pgmap v1318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:18:42.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:42 smithi163 ceph-mon[36873]: pgmap v1318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:18:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:44 smithi042 ceph-mon[32775]: pgmap v1319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:18:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:44 smithi164 ceph-mon[36970]: pgmap v1319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:18:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:44 smithi163 ceph-mon[36873]: pgmap v1319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:18:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:46 smithi042 ceph-mon[32775]: pgmap v1320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:18:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1356.136156 secs 2024-07-26T07:18:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1373.879025 secs 2024-07-26T07:18:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:46 smithi164 ceph-mon[36970]: pgmap v1320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:18:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1356.136156 secs 2024-07-26T07:18:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1373.879025 secs 2024-07-26T07:18:46.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:46 smithi163 ceph-mon[36873]: pgmap v1320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:18:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1356.136156 secs 2024-07-26T07:18:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1373.879025 secs 2024-07-26T07:18:48.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:48 smithi042 ceph-mon[32775]: pgmap v1321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:18:48.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:48 smithi164 ceph-mon[36970]: pgmap v1321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:18:48.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:48 smithi163 ceph-mon[36873]: pgmap v1321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:18:50.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:50 smithi042 ceph-mon[32775]: pgmap v1322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:18:50.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:50 smithi164 ceph-mon[36970]: pgmap v1322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:18:50.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:50 smithi163 ceph-mon[36873]: pgmap v1322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:18:51.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1361.136277 secs 2024-07-26T07:18:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1378.879092 secs 2024-07-26T07:18:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1361.136277 secs 2024-07-26T07:18:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1378.879092 secs 2024-07-26T07:18:51.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1361.136277 secs 2024-07-26T07:18:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1378.879092 secs 2024-07-26T07:18:52.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:52 smithi042 ceph-mon[32775]: pgmap v1323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:52 smithi164 ceph-mon[36970]: pgmap v1323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:52.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:52 smithi163 ceph-mon[36873]: pgmap v1323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:18:53.709+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:18:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:18:53.709+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:18:53.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:18:53.709+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:18:54.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:54 smithi042 ceph-mon[32775]: pgmap v1324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:54 smithi164 ceph-mon[36970]: pgmap v1324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:54 smithi163 ceph-mon[36873]: pgmap v1324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:56 smithi042 ceph-mon[32775]: pgmap v1325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1366.136344 secs 2024-07-26T07:18:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1383.879236 secs 2024-07-26T07:18:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:56 smithi164 ceph-mon[36970]: pgmap v1325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1366.136344 secs 2024-07-26T07:18:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1383.879236 secs 2024-07-26T07:18:56.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:56 smithi163 ceph-mon[36873]: pgmap v1325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:18:56.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1366.136344 secs 2024-07-26T07:18:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1383.879236 secs 2024-07-26T07:18:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:58 smithi042 ceph-mon[32775]: pgmap v1326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:58.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:58 smithi164 ceph-mon[36970]: pgmap v1326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:58 smithi163 ceph-mon[36873]: pgmap v1326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:18:59.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:18:59 smithi042 ceph-mon[32775]: pgmap v1327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:18:59.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:18:59 smithi164 ceph-mon[36970]: pgmap v1327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:18:59.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:18:59 smithi163 ceph-mon[36873]: pgmap v1327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:19:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1371.136442 secs 2024-07-26T07:19:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:01 smithi042 ceph-mon[32775]: pgmap v1328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1388.879359 secs 2024-07-26T07:19:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1371.136442 secs 2024-07-26T07:19:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:01 smithi164 ceph-mon[36970]: pgmap v1328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1388.879359 secs 2024-07-26T07:19:02.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1371.136442 secs 2024-07-26T07:19:02.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:01 smithi163 ceph-mon[36873]: pgmap v1328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1388.879359 secs 2024-07-26T07:19:04.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:03 smithi042 ceph-mon[32775]: pgmap v1329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:03 smithi164 ceph-mon[36970]: pgmap v1329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:04.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:03 smithi163 ceph-mon[36873]: pgmap v1329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:06.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:05 smithi042 ceph-mon[32775]: pgmap v1330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:06.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:05 smithi164 ceph-mon[36970]: pgmap v1330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:06.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:05 smithi163 ceph-mon[36873]: pgmap v1330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:07.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1376.136574 secs 2024-07-26T07:19:07.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1393.879481 secs 2024-07-26T07:19:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1376.136574 secs 2024-07-26T07:19:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1393.879481 secs 2024-07-26T07:19:07.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1376.136574 secs 2024-07-26T07:19:07.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1393.879481 secs 2024-07-26T07:19:08.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:07 smithi042 ceph-mon[32775]: pgmap v1331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:08.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:07 smithi164 ceph-mon[36970]: pgmap v1331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:08.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:07 smithi163 ceph-mon[36873]: pgmap v1331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:08.717 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:19:08.717+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:19:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:19:08.717+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:19:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:19:08.717+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:19:10.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:09 smithi042 ceph-mon[32775]: pgmap v1332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:19:10.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:09 smithi164 ceph-mon[36970]: pgmap v1332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:19:10.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:09 smithi163 ceph-mon[36873]: pgmap v1332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:19:12.059 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1381.136694 secs 2024-07-26T07:19:12.059 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:11 smithi163 ceph-mon[36873]: pgmap v1333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:12.059 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1398.879602 secs 2024-07-26T07:19:12.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1381.136694 secs 2024-07-26T07:19:12.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:11 smithi042 ceph-mon[32775]: pgmap v1333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:12.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1398.879602 secs 2024-07-26T07:19:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1381.136694 secs 2024-07-26T07:19:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:11 smithi164 ceph-mon[36970]: pgmap v1333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1398.879602 secs 2024-07-26T07:19:14.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:13 smithi042 ceph-mon[32775]: pgmap v1334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:14.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:13 smithi164 ceph-mon[36970]: pgmap v1334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:14.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:13 smithi163 ceph-mon[36873]: pgmap v1334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:16.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:15 smithi042 ceph-mon[32775]: pgmap v1335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:16.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:15 smithi164 ceph-mon[36970]: pgmap v1335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:16.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:15 smithi163 ceph-mon[36873]: pgmap v1335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1386.136808 secs 2024-07-26T07:19:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1403.879746 secs 2024-07-26T07:19:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1386.136808 secs 2024-07-26T07:19:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1403.879746 secs 2024-07-26T07:19:17.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1386.136808 secs 2024-07-26T07:19:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1403.879746 secs 2024-07-26T07:19:18.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:17 smithi042 ceph-mon[32775]: pgmap v1336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:18.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:17 smithi164 ceph-mon[36970]: pgmap v1336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:18.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:17 smithi163 ceph-mon[36873]: pgmap v1336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:20.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:19 smithi042 ceph-mon[32775]: pgmap v1337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:19:20.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:19 smithi164 ceph-mon[36970]: pgmap v1337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:19:20.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:19 smithi163 ceph-mon[36873]: pgmap v1337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:19:22.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1391.136906 secs 2024-07-26T07:19:22.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:21 smithi042 ceph-mon[32775]: pgmap v1338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1408.879876 secs 2024-07-26T07:19:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1391.136906 secs 2024-07-26T07:19:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:21 smithi164 ceph-mon[36970]: pgmap v1338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1408.879876 secs 2024-07-26T07:19:22.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1391.136906 secs 2024-07-26T07:19:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:21 smithi163 ceph-mon[36873]: pgmap v1338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1408.879876 secs 2024-07-26T07:19:23.721 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:19:23.721+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:19:23.721 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:19:23.721+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:19:23.721 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:19:23.721+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:19:24.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:23 smithi042 ceph-mon[32775]: pgmap v1339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:24.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:23 smithi164 ceph-mon[36970]: pgmap v1339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:24.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:23 smithi163 ceph-mon[36873]: pgmap v1339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:26.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:25 smithi042 ceph-mon[32775]: pgmap v1340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:26.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:25 smithi164 ceph-mon[36970]: pgmap v1340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:26.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:25 smithi163 ceph-mon[36873]: pgmap v1340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:27.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1396.136988 secs 2024-07-26T07:19:27.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1413.879999 secs 2024-07-26T07:19:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1396.136988 secs 2024-07-26T07:19:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1413.879999 secs 2024-07-26T07:19:27.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1396.136988 secs 2024-07-26T07:19:27.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1413.879999 secs 2024-07-26T07:19:28.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:27 smithi042 ceph-mon[32775]: pgmap v1341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:28.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:27 smithi164 ceph-mon[36970]: pgmap v1341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:28.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:27 smithi163 ceph-mon[36873]: pgmap v1341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:30.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:29 smithi042 ceph-mon[32775]: pgmap v1342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:19:30.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:29 smithi164 ceph-mon[36970]: pgmap v1342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:19:30.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:29 smithi163 ceph-mon[36873]: pgmap v1342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:19:32.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1401.137085 secs 2024-07-26T07:19:32.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:31 smithi042 ceph-mon[32775]: pgmap v1343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1418.880092 secs 2024-07-26T07:19:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1401.137085 secs 2024-07-26T07:19:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:31 smithi164 ceph-mon[36970]: pgmap v1343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1418.880092 secs 2024-07-26T07:19:32.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1401.137085 secs 2024-07-26T07:19:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:31 smithi163 ceph-mon[36873]: pgmap v1343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1418.880092 secs 2024-07-26T07:19:34.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:33 smithi042 ceph-mon[32775]: pgmap v1344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:33 smithi164 ceph-mon[36970]: pgmap v1344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:34.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:33 smithi163 ceph-mon[36873]: pgmap v1344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:36.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:35 smithi042 ceph-mon[32775]: pgmap v1345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:36.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:35 smithi164 ceph-mon[36970]: pgmap v1345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:36.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:35 smithi163 ceph-mon[36873]: pgmap v1345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:37.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1406.137202 secs 2024-07-26T07:19:37.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1423.880222 secs 2024-07-26T07:19:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1406.137202 secs 2024-07-26T07:19:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1423.880222 secs 2024-07-26T07:19:37.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1406.137202 secs 2024-07-26T07:19:37.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1423.880222 secs 2024-07-26T07:19:38.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:37 smithi042 ceph-mon[32775]: pgmap v1346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:38.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:37 smithi164 ceph-mon[36970]: pgmap v1346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:38.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:37 smithi163 ceph-mon[36873]: pgmap v1346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:19:38.729+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:19:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:19:38.729+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:19:38.729 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:19:38.729+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:19:39.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:19:39.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:19:39.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:19:39.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:39 smithi164 ceph-mon[36970]: pgmap v1347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:19:40.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:39 smithi042 ceph-mon[32775]: pgmap v1347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:19:40.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:39 smithi163 ceph-mon[36873]: pgmap v1347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:19:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1411.137295 secs 2024-07-26T07:19:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:41 smithi164 ceph-mon[36970]: pgmap v1348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1428.880331 secs 2024-07-26T07:19:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:19:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:19:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:41 smithi164 ceph-mon[36970]: pgmap v1349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:19:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:41 smithi164 ceph-mon[36970]: pgmap v1350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:19:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1411.137295 secs 2024-07-26T07:19:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:41 smithi163 ceph-mon[36873]: pgmap v1348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1428.880331 secs 2024-07-26T07:19:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:19:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:19:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:41 smithi163 ceph-mon[36873]: pgmap v1349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:19:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:41 smithi163 ceph-mon[36873]: pgmap v1350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:19:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1411.137295 secs 2024-07-26T07:19:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:41 smithi042 ceph-mon[32775]: pgmap v1348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1428.880331 secs 2024-07-26T07:19:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:19:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:19:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:41 smithi042 ceph-mon[32775]: pgmap v1349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:19:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:41 smithi042 ceph-mon[32775]: pgmap v1350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:19:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:19:44.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:44 smithi042 ceph-mon[32775]: pgmap v1351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:19:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:44 smithi164 ceph-mon[36970]: pgmap v1351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:19:44.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:44 smithi163 ceph-mon[36873]: pgmap v1351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:19:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:46 smithi042 ceph-mon[32775]: pgmap v1352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:19:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1416.137411 secs 2024-07-26T07:19:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1433.880455 secs 2024-07-26T07:19:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:46 smithi164 ceph-mon[36970]: pgmap v1352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:19:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1416.137411 secs 2024-07-26T07:19:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1433.880455 secs 2024-07-26T07:19:46.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:46 smithi163 ceph-mon[36873]: pgmap v1352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:19:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1416.137411 secs 2024-07-26T07:19:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1433.880455 secs 2024-07-26T07:19:48.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:48 smithi042 ceph-mon[32775]: pgmap v1353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:19:48.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:48 smithi164 ceph-mon[36970]: pgmap v1353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:19:48.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:48 smithi163 ceph-mon[36873]: pgmap v1353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:19:50.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:50 smithi042 ceph-mon[32775]: pgmap v1354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:19:50.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:50 smithi164 ceph-mon[36970]: pgmap v1354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:19:50.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:50 smithi163 ceph-mon[36873]: pgmap v1354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:19:51.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1421.137542 secs 2024-07-26T07:19:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1438.880578 secs 2024-07-26T07:19:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1421.137542 secs 2024-07-26T07:19:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1438.880578 secs 2024-07-26T07:19:51.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1421.137542 secs 2024-07-26T07:19:51.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1438.880578 secs 2024-07-26T07:19:52.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:52 smithi042 ceph-mon[32775]: pgmap v1355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:19:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:52 smithi164 ceph-mon[36970]: pgmap v1355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:19:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:52 smithi163 ceph-mon[36873]: pgmap v1355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:19:53.732 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:19:53.732+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:19:53.732 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:19:53.732+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:19:53.732 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:19:53.732+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:19:54.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:54 smithi042 ceph-mon[32775]: pgmap v1356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:19:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:54 smithi164 ceph-mon[36970]: pgmap v1356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:19:54.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:54 smithi163 ceph-mon[36873]: pgmap v1356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:19:56.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:56 smithi042 ceph-mon[32775]: pgmap v1357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1426.137669 secs 2024-07-26T07:19:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:56 smithi042 ceph-mon[32775]: 1 slow requests, 1 included below; oldest blocked for > 1443.880676 secs 2024-07-26T07:19:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:56 smithi042 ceph-mon[32775]: slow request 1443.880676 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T07:19:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:56 smithi164 ceph-mon[36970]: pgmap v1357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1426.137669 secs 2024-07-26T07:19:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:56 smithi164 ceph-mon[36970]: 1 slow requests, 1 included below; oldest blocked for > 1443.880676 secs 2024-07-26T07:19:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:56 smithi164 ceph-mon[36970]: slow request 1443.880676 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T07:19:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:56 smithi163 ceph-mon[36873]: pgmap v1357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:19:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1426.137669 secs 2024-07-26T07:19:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:56 smithi163 ceph-mon[36873]: 1 slow requests, 1 included below; oldest blocked for > 1443.880676 secs 2024-07-26T07:19:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:56 smithi163 ceph-mon[36873]: slow request 1443.880676 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T07:19:58.084 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:19:58.083+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T07:19:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:19:58.083+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T07:19:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:19:58.083+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T07:19:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:19:58.083+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T07:19:58.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:19:58 smithi042 ceph-mon[32775]: pgmap v1358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:58.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:19:58 smithi164 ceph-mon[36970]: pgmap v1358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:19:58.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:19:58 smithi163 ceph-mon[36873]: pgmap v1358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:00.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:00 smithi042 ceph-mon[32775]: pgmap v1359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:20:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:00 smithi164 ceph-mon[36970]: pgmap v1359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:20:00.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:00 smithi163 ceph-mon[36873]: pgmap v1359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:20:01.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1431.137803 secs 2024-07-26T07:20:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1448.880818 secs 2024-07-26T07:20:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1431.137803 secs 2024-07-26T07:20:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1448.880818 secs 2024-07-26T07:20:01.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1431.137803 secs 2024-07-26T07:20:01.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1448.880818 secs 2024-07-26T07:20:02.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:02 smithi042 ceph-mon[32775]: pgmap v1360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:02.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:02 smithi164 ceph-mon[36970]: pgmap v1360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:02.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:02 smithi163 ceph-mon[36873]: pgmap v1360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:04.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:04 smithi042 ceph-mon[32775]: pgmap v1361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:20:04.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:04 smithi164 ceph-mon[36970]: pgmap v1361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:20:04.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:04 smithi163 ceph-mon[36873]: pgmap v1361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:20:06.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:06 smithi042 ceph-mon[32775]: pgmap v1362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1436.137901 secs 2024-07-26T07:20:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1453.880948 secs 2024-07-26T07:20:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:06 smithi164 ceph-mon[36970]: pgmap v1362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1436.137901 secs 2024-07-26T07:20:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1453.880948 secs 2024-07-26T07:20:06.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:06 smithi163 ceph-mon[36873]: pgmap v1362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1436.137901 secs 2024-07-26T07:20:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1453.880948 secs 2024-07-26T07:20:08.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:08 smithi042 ceph-mon[32775]: pgmap v1363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:08 smithi164 ceph-mon[36970]: pgmap v1363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:20:08.739+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:20:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:20:08.739+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:20:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:20:08.739+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:20:08.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:08 smithi163 ceph-mon[36873]: pgmap v1363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:10.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:10 smithi042 ceph-mon[32775]: pgmap v1364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:10 smithi164 ceph-mon[36970]: pgmap v1364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:10.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:10 smithi163 ceph-mon[36873]: pgmap v1364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:11.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:11 smithi042 ceph-mon[32775]: 1 slow requests, 1 included below; oldest blocked for > 1441.138019 secs 2024-07-26T07:20:11.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:11 smithi042 ceph-mon[32775]: slow request 1441.138019 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T07:20:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1458.881069 secs 2024-07-26T07:20:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:11 smithi164 ceph-mon[36970]: 1 slow requests, 1 included below; oldest blocked for > 1441.138019 secs 2024-07-26T07:20:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:11 smithi164 ceph-mon[36970]: slow request 1441.138019 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T07:20:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1458.881069 secs 2024-07-26T07:20:11.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:11 smithi163 ceph-mon[36873]: 1 slow requests, 1 included below; oldest blocked for > 1441.138019 secs 2024-07-26T07:20:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:11 smithi163 ceph-mon[36873]: slow request 1441.138019 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T07:20:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1458.881069 secs 2024-07-26T07:20:12.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:12 smithi042 ceph-mon[32775]: pgmap v1365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:12 smithi164 ceph-mon[36970]: pgmap v1365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:12.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:12 smithi163 ceph-mon[36873]: pgmap v1365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:14.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:14 smithi042 ceph-mon[32775]: pgmap v1366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:20:14.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:14 smithi164 ceph-mon[36970]: pgmap v1366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:20:14.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:14 smithi163 ceph-mon[36873]: pgmap v1366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:20:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:16 smithi042 ceph-mon[32775]: pgmap v1367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1446.138173 secs 2024-07-26T07:20:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1463.881186 secs 2024-07-26T07:20:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:16 smithi164 ceph-mon[36970]: pgmap v1367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1446.138173 secs 2024-07-26T07:20:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1463.881186 secs 2024-07-26T07:20:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:16 smithi163 ceph-mon[36873]: pgmap v1367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1446.138173 secs 2024-07-26T07:20:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1463.881186 secs 2024-07-26T07:20:18.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:18 smithi042 ceph-mon[32775]: pgmap v1368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:18.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:18 smithi164 ceph-mon[36970]: pgmap v1368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:18.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:18 smithi163 ceph-mon[36873]: pgmap v1368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:20.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:20 smithi042 ceph-mon[32775]: pgmap v1369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:20.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:20 smithi164 ceph-mon[36970]: pgmap v1369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:20 smithi163 ceph-mon[36873]: pgmap v1369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:21.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1451.138316 secs 2024-07-26T07:20:21.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1468.881308 secs 2024-07-26T07:20:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1451.138316 secs 2024-07-26T07:20:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1468.881308 secs 2024-07-26T07:20:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1451.138316 secs 2024-07-26T07:20:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1468.881308 secs 2024-07-26T07:20:22.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:22 smithi042 ceph-mon[32775]: pgmap v1370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:22 smithi164 ceph-mon[36970]: pgmap v1370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:22.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:22 smithi163 ceph-mon[36873]: pgmap v1370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:23.741 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:20:23.741+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:20:23.741 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:20:23.741+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:20:23.742 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:20:23.741+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:20:24.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:24 smithi042 ceph-mon[32775]: pgmap v1371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:20:24.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:24 smithi164 ceph-mon[36970]: pgmap v1371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:20:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:24 smithi163 ceph-mon[36873]: pgmap v1371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:20:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:26 smithi042 ceph-mon[32775]: pgmap v1372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1456.138446 secs 2024-07-26T07:20:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1473.881438 secs 2024-07-26T07:20:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:26 smithi164 ceph-mon[36970]: pgmap v1372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1456.138446 secs 2024-07-26T07:20:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1473.881438 secs 2024-07-26T07:20:26.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:26 smithi163 ceph-mon[36873]: pgmap v1372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:26.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1456.138446 secs 2024-07-26T07:20:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1473.881438 secs 2024-07-26T07:20:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:27 smithi164 ceph-mon[36970]: pgmap v1373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:27.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:27 smithi163 ceph-mon[36873]: pgmap v1373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:27.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:27 smithi042 ceph-mon[32775]: pgmap v1373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:30 smithi042 ceph-mon[32775]: pgmap v1374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:30.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:30 smithi164 ceph-mon[36970]: pgmap v1374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:30.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:30 smithi163 ceph-mon[36873]: pgmap v1374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:31.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1461.138571 secs 2024-07-26T07:20:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1478.881576 secs 2024-07-26T07:20:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1461.138571 secs 2024-07-26T07:20:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1478.881576 secs 2024-07-26T07:20:31.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1461.138571 secs 2024-07-26T07:20:31.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1478.881576 secs 2024-07-26T07:20:32.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:32 smithi042 ceph-mon[32775]: pgmap v1375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:32 smithi164 ceph-mon[36970]: pgmap v1375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:32.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:32 smithi163 ceph-mon[36873]: pgmap v1375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:34.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:34 smithi042 ceph-mon[32775]: pgmap v1376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:20:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:34 smithi164 ceph-mon[36970]: pgmap v1376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:20:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:34 smithi163 ceph-mon[36873]: pgmap v1376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:20:36.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:36 smithi042 ceph-mon[32775]: pgmap v1377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1466.138650 secs 2024-07-26T07:20:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1483.881682 secs 2024-07-26T07:20:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:36 smithi164 ceph-mon[36970]: pgmap v1377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1466.138650 secs 2024-07-26T07:20:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1483.881682 secs 2024-07-26T07:20:36.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:36 smithi163 ceph-mon[36873]: pgmap v1377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1466.138650 secs 2024-07-26T07:20:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1483.881682 secs 2024-07-26T07:20:38.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:38 smithi042 ceph-mon[32775]: pgmap v1378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:38 smithi164 ceph-mon[36970]: pgmap v1378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:20:38.749+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:20:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:20:38.749+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:20:38.750 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:20:38.749+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:20:38.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:38 smithi163 ceph-mon[36873]: pgmap v1378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:40.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:40 smithi042 ceph-mon[32775]: pgmap v1379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:40 smithi164 ceph-mon[36970]: pgmap v1379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:40.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:40 smithi163 ceph-mon[36873]: pgmap v1379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:41.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1471.138716 secs 2024-07-26T07:20:41.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1488.881803 secs 2024-07-26T07:20:41.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1471.138716 secs 2024-07-26T07:20:41.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1488.881803 secs 2024-07-26T07:20:41.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1471.138716 secs 2024-07-26T07:20:41.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1488.881803 secs 2024-07-26T07:20:42.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:42 smithi042 ceph-mon[32775]: pgmap v1380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:42.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:20:42.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:20:42.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:20:42.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:42 smithi164 ceph-mon[36970]: pgmap v1380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:20:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:20:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:20:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:42 smithi163 ceph-mon[36873]: pgmap v1380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:20:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:20:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:20:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:20:43.091 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:20:43.091+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T07:20:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:20:43.091+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T07:20:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:20:43.091+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T07:20:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:20:43.091+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T07:20:43.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:43 smithi042 ceph-mon[32775]: pgmap v1381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:20:43.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:43 smithi042 ceph-mon[32775]: pgmap v1382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:20:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:43 smithi164 ceph-mon[36970]: pgmap v1381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:20:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:43 smithi164 ceph-mon[36970]: pgmap v1382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:20:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:43 smithi163 ceph-mon[36873]: pgmap v1381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:20:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:43 smithi163 ceph-mon[36873]: pgmap v1382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:20:45.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:45 smithi042 ceph-mon[32775]: pgmap v1383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T07:20:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:45 smithi164 ceph-mon[36970]: pgmap v1383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T07:20:45.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:45 smithi163 ceph-mon[36873]: pgmap v1383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T07:20:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1476.138813 secs 2024-07-26T07:20:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1493.881958 secs 2024-07-26T07:20:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1476.138813 secs 2024-07-26T07:20:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1493.881958 secs 2024-07-26T07:20:46.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1476.138813 secs 2024-07-26T07:20:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1493.881958 secs 2024-07-26T07:20:47.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:47 smithi042 ceph-mon[32775]: pgmap v1384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:20:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:47 smithi164 ceph-mon[36970]: pgmap v1384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:20:47.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:47 smithi163 ceph-mon[36873]: pgmap v1384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:20:49.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:49 smithi042 ceph-mon[32775]: pgmap v1385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:20:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:49 smithi164 ceph-mon[36970]: pgmap v1385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:20:49.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:49 smithi163 ceph-mon[36873]: pgmap v1385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:20:51.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:51 smithi042 ceph-mon[32775]: pgmap v1386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:20:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1481.138879 secs 2024-07-26T07:20:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1498.882095 secs 2024-07-26T07:20:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:51 smithi164 ceph-mon[36970]: pgmap v1386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:20:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1481.138879 secs 2024-07-26T07:20:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1498.882095 secs 2024-07-26T07:20:51.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:51 smithi163 ceph-mon[36873]: pgmap v1386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:20:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1481.138879 secs 2024-07-26T07:20:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1498.882095 secs 2024-07-26T07:20:53.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:53 smithi042 ceph-mon[32775]: pgmap v1387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:20:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:53 smithi164 ceph-mon[36970]: pgmap v1387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:20:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:20:53.752+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:20:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:20:53.752+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:20:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:20:53.752+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:20:53.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:53 smithi163 ceph-mon[36873]: pgmap v1387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:20:55.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:55 smithi042 ceph-mon[32775]: pgmap v1388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:55 smithi164 ceph-mon[36970]: pgmap v1388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:55 smithi163 ceph-mon[36873]: pgmap v1388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1486.138985 secs 2024-07-26T07:20:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1503.882226 secs 2024-07-26T07:20:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1486.138985 secs 2024-07-26T07:20:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1503.882226 secs 2024-07-26T07:20:56.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1486.138985 secs 2024-07-26T07:20:56.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1503.882226 secs 2024-07-26T07:20:57.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:57 smithi042 ceph-mon[32775]: pgmap v1389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:57 smithi164 ceph-mon[36970]: pgmap v1389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:57.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:57 smithi163 ceph-mon[36873]: pgmap v1389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:20:59.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:20:59 smithi042 ceph-mon[32775]: pgmap v1390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:59.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:20:59 smithi164 ceph-mon[36970]: pgmap v1390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:20:59.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:20:59 smithi163 ceph-mon[36873]: pgmap v1390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:01.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:01 smithi042 ceph-mon[32775]: pgmap v1391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1491.139089 secs 2024-07-26T07:21:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1508.882332 secs 2024-07-26T07:21:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:01 smithi164 ceph-mon[36970]: pgmap v1391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1491.139089 secs 2024-07-26T07:21:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1508.882332 secs 2024-07-26T07:21:01.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:01 smithi163 ceph-mon[36873]: pgmap v1391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:01.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1491.139089 secs 2024-07-26T07:21:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1508.882332 secs 2024-07-26T07:21:03.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:03 smithi042 ceph-mon[32775]: pgmap v1392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:03.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:03 smithi164 ceph-mon[36970]: pgmap v1392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:03.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:03 smithi163 ceph-mon[36873]: pgmap v1392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:05.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:05 smithi042 ceph-mon[32775]: pgmap v1393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:05.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:05 smithi164 ceph-mon[36970]: pgmap v1393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:05.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:05 smithi163 ceph-mon[36873]: pgmap v1393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1496.139237 secs 2024-07-26T07:21:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1513.882474 secs 2024-07-26T07:21:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1496.139237 secs 2024-07-26T07:21:06.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1513.882474 secs 2024-07-26T07:21:06.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1496.139237 secs 2024-07-26T07:21:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1513.882474 secs 2024-07-26T07:21:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:07 smithi164 ceph-mon[36970]: pgmap v1394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:07.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:07 smithi163 ceph-mon[36873]: pgmap v1394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:07.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:07 smithi042 ceph-mon[32775]: pgmap v1394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:08 smithi164 ceph-mon[36970]: pgmap v1395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:08.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:08 smithi163 ceph-mon[36873]: pgmap v1395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:21:08.760+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:21:08.760 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:21:08.760+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:21:08.761 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:21:08.760+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:21:08.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:08 smithi042 ceph-mon[32775]: pgmap v1395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:11 smithi164 ceph-mon[36970]: pgmap v1396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1501.139327 secs 2024-07-26T07:21:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:11 smithi163 ceph-mon[36873]: pgmap v1396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1501.139327 secs 2024-07-26T07:21:11.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:11 smithi042 ceph-mon[32775]: pgmap v1396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1501.139327 secs 2024-07-26T07:21:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1518.882611 secs 2024-07-26T07:21:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1518.882611 secs 2024-07-26T07:21:12.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1518.882611 secs 2024-07-26T07:21:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:13 smithi164 ceph-mon[36970]: pgmap v1397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:13.513 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:13 smithi163 ceph-mon[36873]: pgmap v1397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:13.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:13 smithi042 ceph-mon[32775]: pgmap v1397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:15.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:15 smithi164 ceph-mon[36970]: pgmap v1398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:15 smithi163 ceph-mon[36873]: pgmap v1398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:15.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:15 smithi042 ceph-mon[32775]: pgmap v1398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1506.139449 secs 2024-07-26T07:21:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1506.139449 secs 2024-07-26T07:21:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1506.139449 secs 2024-07-26T07:21:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1523.882737 secs 2024-07-26T07:21:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:17 smithi164 ceph-mon[36970]: pgmap v1399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:17.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1523.882737 secs 2024-07-26T07:21:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:17 smithi163 ceph-mon[36873]: pgmap v1399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:17.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1523.882737 secs 2024-07-26T07:21:17.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:17 smithi042 ceph-mon[32775]: pgmap v1399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:19 smithi164 ceph-mon[36970]: pgmap v1400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:19 smithi163 ceph-mon[36873]: pgmap v1400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:19.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:19 smithi042 ceph-mon[32775]: pgmap v1400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:21 smithi164 ceph-mon[36970]: pgmap v1401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1511.139569 secs 2024-07-26T07:21:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:21 smithi163 ceph-mon[36873]: pgmap v1401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1511.139569 secs 2024-07-26T07:21:21.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:21 smithi042 ceph-mon[32775]: pgmap v1401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1511.139569 secs 2024-07-26T07:21:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1528.882860 secs 2024-07-26T07:21:22.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1528.882860 secs 2024-07-26T07:21:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1528.882860 secs 2024-07-26T07:21:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:23 smithi164 ceph-mon[36970]: pgmap v1402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:23.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:23 smithi163 ceph-mon[36873]: pgmap v1402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:23.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:23 smithi042 ceph-mon[32775]: pgmap v1402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:23.763 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:21:23.763+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:21:23.763 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:21:23.763+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:21:23.763 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:21:23.763+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:21:25.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:25 smithi164 ceph-mon[36970]: pgmap v1403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:25.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:25 smithi163 ceph-mon[36873]: pgmap v1403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:25.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:25 smithi042 ceph-mon[32775]: pgmap v1403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1516.139652 secs 2024-07-26T07:21:26.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1516.139652 secs 2024-07-26T07:21:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1516.139652 secs 2024-07-26T07:21:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1533.883003 secs 2024-07-26T07:21:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:27 smithi164 ceph-mon[36970]: pgmap v1404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:27.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1533.883003 secs 2024-07-26T07:21:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:27 smithi163 ceph-mon[36873]: pgmap v1404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:27.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1533.883003 secs 2024-07-26T07:21:27.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:27 smithi042 ceph-mon[32775]: pgmap v1404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:29 smithi164 ceph-mon[36970]: pgmap v1405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:29.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:29 smithi163 ceph-mon[36873]: pgmap v1405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:29.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:29 smithi042 ceph-mon[32775]: pgmap v1405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:31 smithi164 ceph-mon[36970]: pgmap v1406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1521.139777 secs 2024-07-26T07:21:31.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:31 smithi163 ceph-mon[36873]: pgmap v1406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:31.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1521.139777 secs 2024-07-26T07:21:31.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:31 smithi042 ceph-mon[32775]: pgmap v1406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1521.139777 secs 2024-07-26T07:21:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1538.883131 secs 2024-07-26T07:21:32.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1538.883131 secs 2024-07-26T07:21:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1538.883131 secs 2024-07-26T07:21:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:33 smithi163 ceph-mon[36873]: pgmap v1407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:33.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:33 smithi042 ceph-mon[32775]: pgmap v1407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:33.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:33 smithi164 ceph-mon[36970]: pgmap v1407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:35 smithi163 ceph-mon[36873]: pgmap v1408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:35.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:35 smithi042 ceph-mon[32775]: pgmap v1408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:35.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:35 smithi164 ceph-mon[36970]: pgmap v1408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1526.139890 secs 2024-07-26T07:21:36.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1526.139890 secs 2024-07-26T07:21:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1526.139890 secs 2024-07-26T07:21:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1543.883226 secs 2024-07-26T07:21:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:37 smithi163 ceph-mon[36873]: pgmap v1409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:37.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1543.883226 secs 2024-07-26T07:21:37.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:37 smithi042 ceph-mon[32775]: pgmap v1409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1543.883226 secs 2024-07-26T07:21:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:37 smithi164 ceph-mon[36970]: pgmap v1409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:38.771 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:21:38.771+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:21:38.772 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:21:38.771+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:21:38.772 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:21:38.771+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:21:39.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:39 smithi163 ceph-mon[36873]: pgmap v1410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:39.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:39 smithi042 ceph-mon[32775]: pgmap v1410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:39 smithi164 ceph-mon[36970]: pgmap v1410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:41 smithi163 ceph-mon[36873]: pgmap v1411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1531.140017 secs 2024-07-26T07:21:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:41 smithi042 ceph-mon[32775]: pgmap v1411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1531.140017 secs 2024-07-26T07:21:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:41 smithi164 ceph-mon[36970]: pgmap v1411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1531.140017 secs 2024-07-26T07:21:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1548.883349 secs 2024-07-26T07:21:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:21:42.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1548.883349 secs 2024-07-26T07:21:42.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:21:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1548.883349 secs 2024-07-26T07:21:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:21:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:43 smithi163 ceph-mon[36873]: pgmap v1412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:21:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:21:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:21:43.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:43 smithi042 ceph-mon[32775]: pgmap v1412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:43.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:21:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:21:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:21:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:43 smithi164 ceph-mon[36970]: pgmap v1412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:21:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:21:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:21:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:44 smithi163 ceph-mon[36873]: pgmap v1413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:21:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:44 smithi163 ceph-mon[36873]: pgmap v1414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:21:44.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:44 smithi042 ceph-mon[32775]: pgmap v1413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:21:44.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:44 smithi042 ceph-mon[32775]: pgmap v1414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:21:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:44 smithi164 ceph-mon[36970]: pgmap v1413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:21:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:44 smithi164 ceph-mon[36970]: pgmap v1414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:21:46.524 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:46 smithi164 ceph-mon[36970]: pgmap v1415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:21:46.524 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1536.140084 secs 2024-07-26T07:21:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:46 smithi042 ceph-mon[32775]: pgmap v1415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:21:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1536.140084 secs 2024-07-26T07:21:46.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:46 smithi163 ceph-mon[36873]: pgmap v1415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:21:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1536.140084 secs 2024-07-26T07:21:47.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1553.883475 secs 2024-07-26T07:21:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1553.883475 secs 2024-07-26T07:21:47.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1553.883475 secs 2024-07-26T07:21:48.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:48 smithi042 ceph-mon[32775]: pgmap v1416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:21:48.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:48 smithi164 ceph-mon[36970]: pgmap v1416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:21:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:48 smithi163 ceph-mon[36873]: pgmap v1416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:21:50.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:50 smithi042 ceph-mon[32775]: pgmap v1417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:21:50.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:50 smithi164 ceph-mon[36970]: pgmap v1417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:21:50.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:50 smithi163 ceph-mon[36873]: pgmap v1417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:21:51.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1541.140244 secs 2024-07-26T07:21:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1541.140244 secs 2024-07-26T07:21:51.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1541.140244 secs 2024-07-26T07:21:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:52 smithi042 ceph-mon[32775]: pgmap v1418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:21:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1558.883608 secs 2024-07-26T07:21:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:52 smithi164 ceph-mon[36970]: pgmap v1418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:21:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1558.883608 secs 2024-07-26T07:21:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:52 smithi163 ceph-mon[36873]: pgmap v1418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:21:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1558.883608 secs 2024-07-26T07:21:53.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:21:53.773+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:21:53.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:21:53.773+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:21:53.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:21:53.773+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:21:54.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:54 smithi042 ceph-mon[32775]: pgmap v1419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:21:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:54 smithi164 ceph-mon[36970]: pgmap v1419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:21:54.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:54 smithi163 ceph-mon[36873]: pgmap v1419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:21:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:56 smithi042 ceph-mon[32775]: pgmap v1420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1546.140297 secs 2024-07-26T07:21:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1563.883771 secs 2024-07-26T07:21:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:56 smithi164 ceph-mon[36970]: pgmap v1420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1546.140297 secs 2024-07-26T07:21:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1563.883771 secs 2024-07-26T07:21:56.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:56 smithi163 ceph-mon[36873]: pgmap v1420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:21:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1546.140297 secs 2024-07-26T07:21:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1563.883771 secs 2024-07-26T07:21:58.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:21:58 smithi042 ceph-mon[32775]: pgmap v1421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:58.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:21:58 smithi164 ceph-mon[36970]: pgmap v1421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:21:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:21:58 smithi163 ceph-mon[36873]: pgmap v1421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:00.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:00 smithi042 ceph-mon[32775]: pgmap v1422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:22:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:00 smithi164 ceph-mon[36970]: pgmap v1422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:22:00.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:00 smithi163 ceph-mon[36873]: pgmap v1422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:22:01.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1551.140416 secs 2024-07-26T07:22:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1568.883886 secs 2024-07-26T07:22:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1551.140416 secs 2024-07-26T07:22:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1568.883886 secs 2024-07-26T07:22:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1551.140416 secs 2024-07-26T07:22:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1568.883886 secs 2024-07-26T07:22:02.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:02 smithi042 ceph-mon[32775]: pgmap v1423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:02.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:02 smithi164 ceph-mon[36970]: pgmap v1423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:02.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:02 smithi163 ceph-mon[36873]: pgmap v1423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:04.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:04 smithi042 ceph-mon[32775]: pgmap v1424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:04.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:04 smithi164 ceph-mon[36970]: pgmap v1424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:04 smithi163 ceph-mon[36873]: pgmap v1424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:06.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:06 smithi042 ceph-mon[32775]: pgmap v1425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1556.140487 secs 2024-07-26T07:22:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1573.883974 secs 2024-07-26T07:22:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:06 smithi164 ceph-mon[36970]: pgmap v1425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1556.140487 secs 2024-07-26T07:22:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1573.883974 secs 2024-07-26T07:22:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:06 smithi163 ceph-mon[36873]: pgmap v1425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1556.140487 secs 2024-07-26T07:22:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1573.883974 secs 2024-07-26T07:22:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:08 smithi042 ceph-mon[32775]: pgmap v1426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:08 smithi164 ceph-mon[36970]: pgmap v1426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:08 smithi163 ceph-mon[36873]: pgmap v1426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:22:08.781+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:22:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:22:08.781+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:22:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:22:08.781+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:22:10.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:10 smithi042 ceph-mon[32775]: pgmap v1427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:22:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:10 smithi164 ceph-mon[36970]: pgmap v1427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:22:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:10 smithi163 ceph-mon[36873]: pgmap v1427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:22:11.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1561.140630 secs 2024-07-26T07:22:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1578.884074 secs 2024-07-26T07:22:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1561.140630 secs 2024-07-26T07:22:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1578.884074 secs 2024-07-26T07:22:11.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1561.140630 secs 2024-07-26T07:22:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1578.884074 secs 2024-07-26T07:22:12.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:12 smithi042 ceph-mon[32775]: pgmap v1428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:12 smithi164 ceph-mon[36970]: pgmap v1428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:12.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:12 smithi163 ceph-mon[36873]: pgmap v1428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:14.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:14 smithi042 ceph-mon[32775]: pgmap v1429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:14.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:14 smithi164 ceph-mon[36970]: pgmap v1429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:14.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:14 smithi163 ceph-mon[36873]: pgmap v1429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:16 smithi042 ceph-mon[32775]: pgmap v1430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1566.140756 secs 2024-07-26T07:22:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1583.884198 secs 2024-07-26T07:22:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:16 smithi164 ceph-mon[36970]: pgmap v1430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1566.140756 secs 2024-07-26T07:22:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1583.884198 secs 2024-07-26T07:22:16.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:16 smithi163 ceph-mon[36873]: pgmap v1430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1566.140756 secs 2024-07-26T07:22:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1583.884198 secs 2024-07-26T07:22:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:18 smithi042 ceph-mon[32775]: pgmap v1431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:18.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:18 smithi164 ceph-mon[36970]: pgmap v1431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:18.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:18 smithi163 ceph-mon[36873]: pgmap v1431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:20.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:20 smithi042 ceph-mon[32775]: pgmap v1432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:22:20.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:20 smithi164 ceph-mon[36970]: pgmap v1432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:22:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:20 smithi163 ceph-mon[36873]: pgmap v1432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:22:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1571.140885 secs 2024-07-26T07:22:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1588.884314 secs 2024-07-26T07:22:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1571.140885 secs 2024-07-26T07:22:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1588.884314 secs 2024-07-26T07:22:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1571.140885 secs 2024-07-26T07:22:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1588.884314 secs 2024-07-26T07:22:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:22 smithi042 ceph-mon[32775]: pgmap v1433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:22 smithi164 ceph-mon[36970]: pgmap v1433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:22 smithi163 ceph-mon[36873]: pgmap v1433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:22:23.784+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:22:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:22:23.784+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:22:23.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:22:23.784+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:22:24.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:24 smithi042 ceph-mon[32775]: pgmap v1434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:24.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:24 smithi164 ceph-mon[36970]: pgmap v1434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:24 smithi163 ceph-mon[36873]: pgmap v1434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:26.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:26 smithi042 ceph-mon[32775]: pgmap v1435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1576.141025 secs 2024-07-26T07:22:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1593.884413 secs 2024-07-26T07:22:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:26 smithi164 ceph-mon[36970]: pgmap v1435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1576.141025 secs 2024-07-26T07:22:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1593.884413 secs 2024-07-26T07:22:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:26 smithi163 ceph-mon[36873]: pgmap v1435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1576.141025 secs 2024-07-26T07:22:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1593.884413 secs 2024-07-26T07:22:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:28 smithi042 ceph-mon[32775]: pgmap v1436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:28 smithi164 ceph-mon[36970]: pgmap v1436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:28 smithi163 ceph-mon[36873]: pgmap v1436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:30.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:30 smithi042 ceph-mon[32775]: pgmap v1437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:22:30.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:30 smithi164 ceph-mon[36970]: pgmap v1437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:22:30.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:30 smithi163 ceph-mon[36873]: pgmap v1437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:22:31.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1581.141151 secs 2024-07-26T07:22:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1598.884537 secs 2024-07-26T07:22:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1581.141151 secs 2024-07-26T07:22:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1598.884537 secs 2024-07-26T07:22:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1581.141151 secs 2024-07-26T07:22:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1598.884537 secs 2024-07-26T07:22:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:32 smithi042 ceph-mon[32775]: pgmap v1438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:32 smithi164 ceph-mon[36970]: pgmap v1438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:32.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:32 smithi163 ceph-mon[36873]: pgmap v1438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:34 smithi042 ceph-mon[32775]: pgmap v1439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:34 smithi164 ceph-mon[36970]: pgmap v1439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:34.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:34 smithi163 ceph-mon[36873]: pgmap v1439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:36.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:36 smithi042 ceph-mon[32775]: pgmap v1440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1586.141318 secs 2024-07-26T07:22:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1603.884682 secs 2024-07-26T07:22:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:36 smithi164 ceph-mon[36970]: pgmap v1440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1586.141318 secs 2024-07-26T07:22:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1603.884682 secs 2024-07-26T07:22:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:36 smithi163 ceph-mon[36873]: pgmap v1440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1586.141318 secs 2024-07-26T07:22:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1603.884682 secs 2024-07-26T07:22:38.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:38 smithi042 ceph-mon[32775]: pgmap v1441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:38 smithi164 ceph-mon[36970]: pgmap v1441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:38 smithi163 ceph-mon[36873]: pgmap v1441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:22:38.792+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:22:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:22:38.792+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:22:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:22:38.792+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:22:40.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:40 smithi042 ceph-mon[32775]: pgmap v1442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:22:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:40 smithi164 ceph-mon[36970]: pgmap v1442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:22:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:40 smithi163 ceph-mon[36873]: pgmap v1442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:22:41.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1591.141448 secs 2024-07-26T07:22:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1608.884809 secs 2024-07-26T07:22:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1591.141448 secs 2024-07-26T07:22:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1608.884809 secs 2024-07-26T07:22:41.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1591.141448 secs 2024-07-26T07:22:41.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1608.884809 secs 2024-07-26T07:22:42.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:42 smithi042 ceph-mon[32775]: pgmap v1443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:42 smithi164 ceph-mon[36970]: pgmap v1443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:42.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:42 smithi163 ceph-mon[36873]: pgmap v1443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:43.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:22:43.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:22:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:22:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:22:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:22:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:22:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:22:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:22:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:22:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:22:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:22:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:22:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:44 smithi042 ceph-mon[32775]: pgmap v1444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:44 smithi042 ceph-mon[32775]: pgmap v1445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:22:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:44 smithi042 ceph-mon[32775]: pgmap v1446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:22:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:44 smithi164 ceph-mon[36970]: pgmap v1444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:44 smithi164 ceph-mon[36970]: pgmap v1445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:22:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:44 smithi164 ceph-mon[36970]: pgmap v1446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:22:44.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:44 smithi163 ceph-mon[36873]: pgmap v1444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:44 smithi163 ceph-mon[36873]: pgmap v1445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:22:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:44 smithi163 ceph-mon[36873]: pgmap v1446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:22:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:46 smithi042 ceph-mon[32775]: pgmap v1447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:22:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1596.141534 secs 2024-07-26T07:22:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1613.884955 secs 2024-07-26T07:22:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:46 smithi164 ceph-mon[36970]: pgmap v1447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:22:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1596.141534 secs 2024-07-26T07:22:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1613.884955 secs 2024-07-26T07:22:46.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:46 smithi163 ceph-mon[36873]: pgmap v1447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:22:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1596.141534 secs 2024-07-26T07:22:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1613.884955 secs 2024-07-26T07:22:48.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:48 smithi164 ceph-mon[36970]: pgmap v1448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:22:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:48 smithi163 ceph-mon[36873]: pgmap v1448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:22:48.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:48 smithi042 ceph-mon[32775]: pgmap v1448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:22:50.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:50 smithi164 ceph-mon[36970]: pgmap v1449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:22:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:50 smithi163 ceph-mon[36873]: pgmap v1449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:22:50.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:50 smithi042 ceph-mon[32775]: pgmap v1449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:22:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1601.141661 secs 2024-07-26T07:22:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1618.885083 secs 2024-07-26T07:22:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1601.141661 secs 2024-07-26T07:22:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1618.885083 secs 2024-07-26T07:22:51.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1601.141661 secs 2024-07-26T07:22:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1618.885083 secs 2024-07-26T07:22:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:52 smithi164 ceph-mon[36970]: pgmap v1450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:22:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:52 smithi163 ceph-mon[36873]: pgmap v1450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:22:52.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:52 smithi042 ceph-mon[32775]: pgmap v1450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:22:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:22:53.795+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:22:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:22:53.795+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:22:53.796 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:22:53.795+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:22:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:54 smithi164 ceph-mon[36970]: pgmap v1451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:22:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:54 smithi163 ceph-mon[36873]: pgmap v1451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:22:54.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:54 smithi042 ceph-mon[32775]: pgmap v1451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:22:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:56 smithi164 ceph-mon[36970]: pgmap v1452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1606.141737 secs 2024-07-26T07:22:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1623.885185 secs 2024-07-26T07:22:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:56 smithi163 ceph-mon[36873]: pgmap v1452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1606.141737 secs 2024-07-26T07:22:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1623.885185 secs 2024-07-26T07:22:56.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:56 smithi042 ceph-mon[32775]: pgmap v1452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1606.141737 secs 2024-07-26T07:22:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1623.885185 secs 2024-07-26T07:22:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:22:58.112+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T07:22:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:22:58.112+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T07:22:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:22:58.112+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T07:22:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:22:58.112+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T07:22:58.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:58 smithi164 ceph-mon[36970]: pgmap v1453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:58 smithi163 ceph-mon[36873]: pgmap v1453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:58.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:58 smithi042 ceph-mon[32775]: pgmap v1453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:22:59.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:22:59 smithi164 ceph-mon[36970]: pgmap v1454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:59.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:22:59 smithi163 ceph-mon[36873]: pgmap v1454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:22:59.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:22:59 smithi042 ceph-mon[32775]: pgmap v1454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:00 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1611.141837 secs 2024-07-26T07:23:01.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:00 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1611.141837 secs 2024-07-26T07:23:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:00 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1611.141837 secs 2024-07-26T07:23:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1628.885309 secs 2024-07-26T07:23:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:01 smithi164 ceph-mon[36970]: pgmap v1455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:02.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1628.885309 secs 2024-07-26T07:23:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:01 smithi163 ceph-mon[36873]: pgmap v1455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1628.885309 secs 2024-07-26T07:23:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:01 smithi042 ceph-mon[32775]: pgmap v1455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:04.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:04 smithi042 ceph-mon[32775]: pgmap v1456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:23:04.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:04 smithi164 ceph-mon[36970]: pgmap v1456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:23:04.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:04 smithi163 ceph-mon[36873]: pgmap v1456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:23:06.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:06 smithi042 ceph-mon[32775]: pgmap v1457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1616.141960 secs 2024-07-26T07:23:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:06 smithi164 ceph-mon[36970]: pgmap v1457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1616.141960 secs 2024-07-26T07:23:06.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:06 smithi163 ceph-mon[36873]: pgmap v1457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1616.141960 secs 2024-07-26T07:23:07.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1633.885430 secs 2024-07-26T07:23:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1633.885430 secs 2024-07-26T07:23:07.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1633.885430 secs 2024-07-26T07:23:08.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:08 smithi042 ceph-mon[32775]: pgmap v1458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:08 smithi164 ceph-mon[36970]: pgmap v1458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:08.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:08 smithi163 ceph-mon[36873]: pgmap v1458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:08.804 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:23:08.804+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:23:08.804 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:23:08.804+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:23:08.804 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:23:08.804+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:23:10.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:10 smithi042 ceph-mon[32775]: pgmap v1459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:10 smithi164 ceph-mon[36970]: pgmap v1459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:10 smithi163 ceph-mon[36873]: pgmap v1459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:11.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1621.142082 secs 2024-07-26T07:23:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1621.142082 secs 2024-07-26T07:23:11.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1621.142082 secs 2024-07-26T07:23:12.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1638.885556 secs 2024-07-26T07:23:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:12 smithi042 ceph-mon[32775]: pgmap v1460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1638.885556 secs 2024-07-26T07:23:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:12 smithi164 ceph-mon[36970]: pgmap v1460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:12.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1638.885556 secs 2024-07-26T07:23:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:12 smithi163 ceph-mon[36873]: pgmap v1460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:14 smithi042 ceph-mon[32775]: pgmap v1461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:23:14.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:14 smithi164 ceph-mon[36970]: pgmap v1461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:23:14.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:14 smithi163 ceph-mon[36873]: pgmap v1461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:23:16.559 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:16 smithi163 ceph-mon[36873]: pgmap v1462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:16.559 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1626.142248 secs 2024-07-26T07:23:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:16 smithi042 ceph-mon[32775]: pgmap v1462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1626.142248 secs 2024-07-26T07:23:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:16 smithi164 ceph-mon[36970]: pgmap v1462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1626.142248 secs 2024-07-26T07:23:17.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1643.885685 secs 2024-07-26T07:23:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1643.885685 secs 2024-07-26T07:23:17.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1643.885685 secs 2024-07-26T07:23:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:18 smithi042 ceph-mon[32775]: pgmap v1463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:18.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:18 smithi164 ceph-mon[36970]: pgmap v1463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:18.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:18 smithi163 ceph-mon[36873]: pgmap v1463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:20.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:20 smithi042 ceph-mon[32775]: pgmap v1464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:20.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:20 smithi164 ceph-mon[36970]: pgmap v1464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:20.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:20 smithi163 ceph-mon[36873]: pgmap v1464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:21.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1631.142389 secs 2024-07-26T07:23:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1648.885809 secs 2024-07-26T07:23:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1631.142389 secs 2024-07-26T07:23:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1648.885809 secs 2024-07-26T07:23:21.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1631.142389 secs 2024-07-26T07:23:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1648.885809 secs 2024-07-26T07:23:22.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:22 smithi042 ceph-mon[32775]: pgmap v1465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:22 smithi164 ceph-mon[36970]: pgmap v1465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:22.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:22 smithi163 ceph-mon[36873]: pgmap v1465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:23:23.807+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:23:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:23:23.807+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:23:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:23:23.807+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:23:24.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:24 smithi042 ceph-mon[32775]: pgmap v1466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:23:24.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:24 smithi164 ceph-mon[36970]: pgmap v1466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:23:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:24 smithi163 ceph-mon[36873]: pgmap v1466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:23:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:25 smithi163 ceph-mon[36873]: pgmap v1467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:26.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:25 smithi042 ceph-mon[32775]: pgmap v1467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:26.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:25 smithi164 ceph-mon[36970]: pgmap v1467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1636.142530 secs 2024-07-26T07:23:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1653.885958 secs 2024-07-26T07:23:27.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1636.142530 secs 2024-07-26T07:23:27.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1653.885958 secs 2024-07-26T07:23:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1636.142530 secs 2024-07-26T07:23:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1653.885958 secs 2024-07-26T07:23:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:27 smithi163 ceph-mon[36873]: pgmap v1468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:27 smithi042 ceph-mon[32775]: pgmap v1468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:28.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:27 smithi164 ceph-mon[36970]: pgmap v1468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:30 smithi042 ceph-mon[32775]: pgmap v1469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:30.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:30 smithi164 ceph-mon[36970]: pgmap v1469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:30.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:30 smithi163 ceph-mon[36873]: pgmap v1469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:31.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1641.142643 secs 2024-07-26T07:23:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1641.142643 secs 2024-07-26T07:23:31.772 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1641.142643 secs 2024-07-26T07:23:32.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1658.886087 secs 2024-07-26T07:23:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:32 smithi042 ceph-mon[32775]: pgmap v1470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1658.886087 secs 2024-07-26T07:23:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:32 smithi164 ceph-mon[36970]: pgmap v1470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1658.886087 secs 2024-07-26T07:23:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:32 smithi163 ceph-mon[36873]: pgmap v1470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:34.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:34 smithi042 ceph-mon[32775]: pgmap v1471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:23:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:34 smithi164 ceph-mon[36970]: pgmap v1471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:23:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:34 smithi163 ceph-mon[36873]: pgmap v1471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:23:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:36 smithi042 ceph-mon[32775]: pgmap v1472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1646.142768 secs 2024-07-26T07:23:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:36 smithi164 ceph-mon[36970]: pgmap v1472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1646.142768 secs 2024-07-26T07:23:36.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:36 smithi163 ceph-mon[36873]: pgmap v1472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1646.142768 secs 2024-07-26T07:23:37.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1663.886209 secs 2024-07-26T07:23:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1663.886209 secs 2024-07-26T07:23:37.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1663.886209 secs 2024-07-26T07:23:38.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:38 smithi042 ceph-mon[32775]: pgmap v1473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:38 smithi164 ceph-mon[36970]: pgmap v1473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:38 smithi163 ceph-mon[36873]: pgmap v1473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:23:38.814+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:23:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:23:38.814+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:23:38.815 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:23:38.814+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:23:40.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:40 smithi042 ceph-mon[32775]: pgmap v1474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:40 smithi164 ceph-mon[36970]: pgmap v1474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:40 smithi163 ceph-mon[36873]: pgmap v1474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:41.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1651.142907 secs 2024-07-26T07:23:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1651.142907 secs 2024-07-26T07:23:41.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1651.142907 secs 2024-07-26T07:23:42.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1668.886293 secs 2024-07-26T07:23:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:42 smithi042 ceph-mon[32775]: pgmap v1475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1668.886293 secs 2024-07-26T07:23:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:42 smithi164 ceph-mon[36970]: pgmap v1475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:42.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1668.886293 secs 2024-07-26T07:23:42.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:42 smithi163 ceph-mon[36873]: pgmap v1475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:44 smithi042 ceph-mon[32775]: pgmap v1476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:23:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:23:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:23:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:23:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:23:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:44 smithi164 ceph-mon[36970]: pgmap v1476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:23:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:23:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:23:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:23:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:23:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:44 smithi163 ceph-mon[36873]: pgmap v1476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:23:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:23:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:23:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:23:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:23:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:45 smithi042 ceph-mon[32775]: pgmap v1477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:23:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:45 smithi042 ceph-mon[32775]: pgmap v1478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:23:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:45 smithi164 ceph-mon[36970]: pgmap v1477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:23:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:45 smithi164 ceph-mon[36970]: pgmap v1478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:23:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:45 smithi163 ceph-mon[36873]: pgmap v1477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:23:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:45 smithi163 ceph-mon[36873]: pgmap v1478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:23:46.623 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1656.143032 secs 2024-07-26T07:23:46.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1673.886418 secs 2024-07-26T07:23:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1656.143032 secs 2024-07-26T07:23:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1673.886418 secs 2024-07-26T07:23:46.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1656.143032 secs 2024-07-26T07:23:46.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1673.886418 secs 2024-07-26T07:23:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:47 smithi042 ceph-mon[32775]: pgmap v1479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:23:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:47 smithi164 ceph-mon[36970]: pgmap v1479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:23:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:47 smithi163 ceph-mon[36873]: pgmap v1479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:23:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:49 smithi042 ceph-mon[32775]: pgmap v1480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:23:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:49 smithi164 ceph-mon[36970]: pgmap v1480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:23:49.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:49 smithi163 ceph-mon[36873]: pgmap v1480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:23:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:51 smithi042 ceph-mon[32775]: pgmap v1481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:23:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1661.143171 secs 2024-07-26T07:23:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1678.886539 secs 2024-07-26T07:23:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:51 smithi164 ceph-mon[36970]: pgmap v1481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:23:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1661.143171 secs 2024-07-26T07:23:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1678.886539 secs 2024-07-26T07:23:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:51 smithi163 ceph-mon[36873]: pgmap v1481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:23:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1661.143171 secs 2024-07-26T07:23:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1678.886539 secs 2024-07-26T07:23:53.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:53 smithi042 ceph-mon[32775]: pgmap v1482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:23:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:53 smithi164 ceph-mon[36970]: pgmap v1482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:23:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:53 smithi163 ceph-mon[36873]: pgmap v1482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:23:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:23:53.817+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:23:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:23:53.817+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:23:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:23:53.817+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:23:55.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:55 smithi042 ceph-mon[32775]: pgmap v1483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:23:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:55 smithi164 ceph-mon[36970]: pgmap v1483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:23:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:55 smithi163 ceph-mon[36873]: pgmap v1483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:23:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1666.143306 secs 2024-07-26T07:23:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1683.886662 secs 2024-07-26T07:23:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1666.143306 secs 2024-07-26T07:23:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1683.886662 secs 2024-07-26T07:23:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1666.143306 secs 2024-07-26T07:23:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1683.886662 secs 2024-07-26T07:23:57.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:57 smithi042 ceph-mon[32775]: pgmap v1484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:57 smithi164 ceph-mon[36970]: pgmap v1484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:57.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:57 smithi163 ceph-mon[36873]: pgmap v1484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:23:59.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:23:59 smithi042 ceph-mon[32775]: pgmap v1485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:59.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:23:59 smithi164 ceph-mon[36970]: pgmap v1485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:23:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:23:59 smithi163 ceph-mon[36873]: pgmap v1485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:01.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:01 smithi042 ceph-mon[32775]: pgmap v1486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1671.143430 secs 2024-07-26T07:24:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1688.886789 secs 2024-07-26T07:24:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:01 smithi164 ceph-mon[36970]: pgmap v1486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1671.143430 secs 2024-07-26T07:24:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1688.886789 secs 2024-07-26T07:24:01.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:01 smithi163 ceph-mon[36873]: pgmap v1486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1671.143430 secs 2024-07-26T07:24:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1688.886789 secs 2024-07-26T07:24:03.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:03 smithi042 ceph-mon[32775]: pgmap v1487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:03.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:03 smithi164 ceph-mon[36970]: pgmap v1487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:03 smithi163 ceph-mon[36873]: pgmap v1487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:05.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:05 smithi042 ceph-mon[32775]: pgmap v1488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:24:05.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:05 smithi164 ceph-mon[36970]: pgmap v1488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:24:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:05 smithi163 ceph-mon[36873]: pgmap v1488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:24:06.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1676.143558 secs 2024-07-26T07:24:06.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1693.886892 secs 2024-07-26T07:24:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1676.143558 secs 2024-07-26T07:24:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1693.886892 secs 2024-07-26T07:24:06.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1676.143558 secs 2024-07-26T07:24:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1693.886892 secs 2024-07-26T07:24:07.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:07 smithi042 ceph-mon[32775]: pgmap v1489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:07 smithi164 ceph-mon[36970]: pgmap v1489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:07.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:07 smithi163 ceph-mon[36873]: pgmap v1489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:08.826 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:24:08.825+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:24:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:24:08.825+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:24:08.827 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:24:08.826+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:24:09.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:09 smithi042 ceph-mon[32775]: pgmap v1490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:09 smithi164 ceph-mon[36970]: pgmap v1490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:09.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:09 smithi163 ceph-mon[36873]: pgmap v1490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:11.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:11 smithi042 ceph-mon[32775]: pgmap v1491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1681.143688 secs 2024-07-26T07:24:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1698.887011 secs 2024-07-26T07:24:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:11 smithi164 ceph-mon[36970]: pgmap v1491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1681.143688 secs 2024-07-26T07:24:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1698.887011 secs 2024-07-26T07:24:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:11 smithi163 ceph-mon[36873]: pgmap v1491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1681.143688 secs 2024-07-26T07:24:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1698.887011 secs 2024-07-26T07:24:13.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:13 smithi042 ceph-mon[32775]: pgmap v1492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:13.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:13 smithi164 ceph-mon[36970]: pgmap v1492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:13 smithi163 ceph-mon[36873]: pgmap v1492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:15.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:15 smithi042 ceph-mon[32775]: pgmap v1493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:24:15.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:15 smithi164 ceph-mon[36970]: pgmap v1493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:24:15.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:15 smithi163 ceph-mon[36873]: pgmap v1493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:24:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1686.143851 secs 2024-07-26T07:24:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1703.887135 secs 2024-07-26T07:24:16.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1686.143851 secs 2024-07-26T07:24:16.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1703.887135 secs 2024-07-26T07:24:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1686.143851 secs 2024-07-26T07:24:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1703.887135 secs 2024-07-26T07:24:17.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:17 smithi042 ceph-mon[32775]: pgmap v1494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:17 smithi164 ceph-mon[36970]: pgmap v1494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:17 smithi163 ceph-mon[36873]: pgmap v1494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:19.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:19 smithi042 ceph-mon[32775]: pgmap v1495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:19.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:19 smithi164 ceph-mon[36970]: pgmap v1495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:19.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:19 smithi163 ceph-mon[36873]: pgmap v1495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:21 smithi042 ceph-mon[32775]: pgmap v1496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1691.143975 secs 2024-07-26T07:24:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1708.887259 secs 2024-07-26T07:24:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:21 smithi164 ceph-mon[36970]: pgmap v1496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1691.143975 secs 2024-07-26T07:24:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1708.887259 secs 2024-07-26T07:24:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:21 smithi163 ceph-mon[36873]: pgmap v1496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1691.143975 secs 2024-07-26T07:24:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1708.887259 secs 2024-07-26T07:24:23.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:23 smithi042 ceph-mon[32775]: pgmap v1497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:23.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:23 smithi164 ceph-mon[36970]: pgmap v1497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:23 smithi163 ceph-mon[36873]: pgmap v1497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:24:23.829+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:24:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:24:23.829+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:24:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:24:23.829+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:24:25.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:25 smithi042 ceph-mon[32775]: pgmap v1498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:24:25.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:25 smithi164 ceph-mon[36970]: pgmap v1498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:24:25.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:25 smithi163 ceph-mon[36873]: pgmap v1498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:24:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1696.144103 secs 2024-07-26T07:24:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1713.887395 secs 2024-07-26T07:24:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1696.144103 secs 2024-07-26T07:24:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1713.887395 secs 2024-07-26T07:24:26.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1696.144103 secs 2024-07-26T07:24:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1713.887395 secs 2024-07-26T07:24:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:27 smithi164 ceph-mon[36970]: pgmap v1499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:27.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:27 smithi163 ceph-mon[36873]: pgmap v1499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:27.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:27 smithi042 ceph-mon[32775]: pgmap v1499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:29.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:29 smithi164 ceph-mon[36970]: pgmap v1500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:29.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:29 smithi163 ceph-mon[36873]: pgmap v1500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:29.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:29 smithi042 ceph-mon[32775]: pgmap v1500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:31 smithi164 ceph-mon[36970]: pgmap v1501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1701.144178 secs 2024-07-26T07:24:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1718.887525 secs 2024-07-26T07:24:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:31 smithi163 ceph-mon[36873]: pgmap v1501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1701.144178 secs 2024-07-26T07:24:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1718.887525 secs 2024-07-26T07:24:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:31 smithi042 ceph-mon[32775]: pgmap v1501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1701.144178 secs 2024-07-26T07:24:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1718.887525 secs 2024-07-26T07:24:33.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:33 smithi164 ceph-mon[36970]: pgmap v1502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:33 smithi163 ceph-mon[36873]: pgmap v1502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:33.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:33 smithi042 ceph-mon[32775]: pgmap v1502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:35.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:35 smithi042 ceph-mon[32775]: pgmap v1503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:24:35.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:35 smithi164 ceph-mon[36970]: pgmap v1503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:24:35.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:35 smithi163 ceph-mon[36873]: pgmap v1503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:24:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1706.144317 secs 2024-07-26T07:24:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1723.887669 secs 2024-07-26T07:24:36.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1706.144317 secs 2024-07-26T07:24:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1723.887669 secs 2024-07-26T07:24:36.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1706.144317 secs 2024-07-26T07:24:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1723.887669 secs 2024-07-26T07:24:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:37 smithi164 ceph-mon[36970]: pgmap v1504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:37 smithi163 ceph-mon[36873]: pgmap v1504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:37.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:37 smithi042 ceph-mon[32775]: pgmap v1504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:38.837 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:24:38.837+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:24:38.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:24:38.837+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:24:38.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:24:38.837+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:24:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:39 smithi164 ceph-mon[36970]: pgmap v1505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:39 smithi163 ceph-mon[36873]: pgmap v1505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:39.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:39 smithi042 ceph-mon[32775]: pgmap v1505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:41.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:41 smithi042 ceph-mon[32775]: pgmap v1506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:41.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1711.144427 secs 2024-07-26T07:24:41.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1728.887772 secs 2024-07-26T07:24:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:41 smithi164 ceph-mon[36970]: pgmap v1506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1711.144427 secs 2024-07-26T07:24:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1728.887772 secs 2024-07-26T07:24:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:41 smithi163 ceph-mon[36873]: pgmap v1506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:24:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1711.144427 secs 2024-07-26T07:24:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1728.887772 secs 2024-07-26T07:24:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:43 smithi164 ceph-mon[36970]: pgmap v1507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:43 smithi163 ceph-mon[36873]: pgmap v1507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:43.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:43 smithi042 ceph-mon[32775]: pgmap v1507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:24:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:24:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:24:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:24:44.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:24:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:24:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:24:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:24:44.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:24:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:24:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:24:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:24:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:45 smithi164 ceph-mon[36970]: pgmap v1508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:24:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:45 smithi164 ceph-mon[36970]: pgmap v1509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:24:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:45 smithi164 ceph-mon[36970]: pgmap v1510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:24:45.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:45 smithi163 ceph-mon[36873]: pgmap v1508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:24:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:45 smithi163 ceph-mon[36873]: pgmap v1509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:24:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:45 smithi163 ceph-mon[36873]: pgmap v1510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:24:45.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:45 smithi042 ceph-mon[32775]: pgmap v1508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:24:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:45 smithi042 ceph-mon[32775]: pgmap v1509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:24:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:45 smithi042 ceph-mon[32775]: pgmap v1510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:24:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1716.144558 secs 2024-07-26T07:24:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1733.887893 secs 2024-07-26T07:24:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:46 smithi164 ceph-mon[36970]: pgmap v1511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:24:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1716.144558 secs 2024-07-26T07:24:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1733.887893 secs 2024-07-26T07:24:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:46 smithi163 ceph-mon[36873]: pgmap v1511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:24:46.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1716.144558 secs 2024-07-26T07:24:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1733.887893 secs 2024-07-26T07:24:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:46 smithi042 ceph-mon[32775]: pgmap v1511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:24:49.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:49 smithi042 ceph-mon[32775]: pgmap v1512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:24:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:49 smithi164 ceph-mon[36970]: pgmap v1512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:24:49.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:49 smithi163 ceph-mon[36873]: pgmap v1512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:24:51.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:51 smithi042 ceph-mon[32775]: pgmap v1513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:24:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1721.144679 secs 2024-07-26T07:24:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1738.888015 secs 2024-07-26T07:24:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:51 smithi164 ceph-mon[36970]: pgmap v1513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:24:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1721.144679 secs 2024-07-26T07:24:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1738.888015 secs 2024-07-26T07:24:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:51 smithi163 ceph-mon[36873]: pgmap v1513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:24:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1721.144679 secs 2024-07-26T07:24:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1738.888015 secs 2024-07-26T07:24:53.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:53 smithi042 ceph-mon[32775]: pgmap v1514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:24:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:53 smithi164 ceph-mon[36970]: pgmap v1514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:24:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:53 smithi163 ceph-mon[36873]: pgmap v1514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:24:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:24:53.838+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:24:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:24:53.838+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:24:53.839 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:24:53.838+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:24:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:55 smithi042 ceph-mon[32775]: pgmap v1515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:24:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:55 smithi164 ceph-mon[36970]: pgmap v1515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:24:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:55 smithi163 ceph-mon[36873]: pgmap v1515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:24:56.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1726.144826 secs 2024-07-26T07:24:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1743.888139 secs 2024-07-26T07:24:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1726.144826 secs 2024-07-26T07:24:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1743.888139 secs 2024-07-26T07:24:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1726.144826 secs 2024-07-26T07:24:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1743.888139 secs 2024-07-26T07:24:57.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:57 smithi042 ceph-mon[32775]: pgmap v1516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:57 smithi164 ceph-mon[36970]: pgmap v1516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:57 smithi163 ceph-mon[36873]: pgmap v1516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:24:59.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:24:59 smithi042 ceph-mon[32775]: pgmap v1517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:24:59.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:24:59 smithi164 ceph-mon[36970]: pgmap v1517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:24:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:24:59 smithi163 ceph-mon[36873]: pgmap v1517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:25:01.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:01 smithi042 ceph-mon[32775]: pgmap v1518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1731.144956 secs 2024-07-26T07:25:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1748.888252 secs 2024-07-26T07:25:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:01 smithi164 ceph-mon[36970]: pgmap v1518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1731.144956 secs 2024-07-26T07:25:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1748.888252 secs 2024-07-26T07:25:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:01 smithi163 ceph-mon[36873]: pgmap v1518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1731.144956 secs 2024-07-26T07:25:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1748.888252 secs 2024-07-26T07:25:03.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:03 smithi164 ceph-mon[36970]: pgmap v1519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:03 smithi163 ceph-mon[36873]: pgmap v1519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:03.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:03 smithi042 ceph-mon[32775]: pgmap v1519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:05.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:05 smithi164 ceph-mon[36970]: pgmap v1520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:05.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:05 smithi163 ceph-mon[36873]: pgmap v1520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:05.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:05 smithi042 ceph-mon[32775]: pgmap v1520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1736.145061 secs 2024-07-26T07:25:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1753.888396 secs 2024-07-26T07:25:06.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1736.145061 secs 2024-07-26T07:25:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1753.888396 secs 2024-07-26T07:25:06.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1736.145061 secs 2024-07-26T07:25:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1753.888396 secs 2024-07-26T07:25:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:07 smithi164 ceph-mon[36970]: pgmap v1521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:07 smithi163 ceph-mon[36873]: pgmap v1521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:07 smithi042 ceph-mon[32775]: pgmap v1521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:08.846+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:25:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:08.846+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:25:08.846 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:08.846+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:25:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:09 smithi164 ceph-mon[36970]: pgmap v1522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:25:09.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:09 smithi163 ceph-mon[36873]: pgmap v1522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:25:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:09 smithi042 ceph-mon[32775]: pgmap v1522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:25:11.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:11 smithi042 ceph-mon[32775]: pgmap v1523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:11.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1741.145188 secs 2024-07-26T07:25:11.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1758.888542 secs 2024-07-26T07:25:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:11 smithi164 ceph-mon[36970]: pgmap v1523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1741.145188 secs 2024-07-26T07:25:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1758.888542 secs 2024-07-26T07:25:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:11 smithi163 ceph-mon[36873]: pgmap v1523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1741.145188 secs 2024-07-26T07:25:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1758.888542 secs 2024-07-26T07:25:13.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:13 smithi164 ceph-mon[36970]: pgmap v1524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:13 smithi163 ceph-mon[36873]: pgmap v1524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:13.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:13 smithi042 ceph-mon[32775]: pgmap v1524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:15.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:15 smithi164 ceph-mon[36970]: pgmap v1525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:15 smithi163 ceph-mon[36873]: pgmap v1525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:15.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:15 smithi042 ceph-mon[32775]: pgmap v1525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1746.145312 secs 2024-07-26T07:25:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1763.888664 secs 2024-07-26T07:25:16.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1746.145312 secs 2024-07-26T07:25:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1763.888664 secs 2024-07-26T07:25:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1746.145312 secs 2024-07-26T07:25:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1763.888664 secs 2024-07-26T07:25:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:17 smithi164 ceph-mon[36970]: pgmap v1526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:17 smithi163 ceph-mon[36873]: pgmap v1526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:17.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:17 smithi042 ceph-mon[32775]: pgmap v1526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:19.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:19 smithi164 ceph-mon[36970]: pgmap v1527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:25:19.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:19 smithi163 ceph-mon[36873]: pgmap v1527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:25:19.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:19 smithi042 ceph-mon[32775]: pgmap v1527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:25:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:21 smithi164 ceph-mon[36970]: pgmap v1528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1751.145382 secs 2024-07-26T07:25:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1768.888763 secs 2024-07-26T07:25:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:21 smithi163 ceph-mon[36873]: pgmap v1528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1751.145382 secs 2024-07-26T07:25:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1768.888763 secs 2024-07-26T07:25:21.775 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:21 smithi042 ceph-mon[32775]: pgmap v1528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:21.775 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1751.145382 secs 2024-07-26T07:25:21.775 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1768.888763 secs 2024-07-26T07:25:23.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:23 smithi164 ceph-mon[36970]: pgmap v1529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:23 smithi163 ceph-mon[36873]: pgmap v1529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:23.848+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:25:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:23.848+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:25:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:23.848+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:25:23.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:23 smithi042 ceph-mon[32775]: pgmap v1529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:24.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:24 smithi164 ceph-mon[36970]: pgmap v1530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:24 smithi163 ceph-mon[36873]: pgmap v1530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:24.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:24 smithi042 ceph-mon[32775]: pgmap v1530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:25 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1756.145504 secs 2024-07-26T07:25:26.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:25 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1756.145504 secs 2024-07-26T07:25:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:25 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1756.145504 secs 2024-07-26T07:25:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1773.888923 secs 2024-07-26T07:25:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:26 smithi163 ceph-mon[36873]: pgmap v1531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:27.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1773.888923 secs 2024-07-26T07:25:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:26 smithi042 ceph-mon[32775]: pgmap v1531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1773.888923 secs 2024-07-26T07:25:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:26 smithi164 ceph-mon[36970]: pgmap v1531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:29.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:29 smithi042 ceph-mon[32775]: pgmap v1532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:25:29.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:29 smithi164 ceph-mon[36970]: pgmap v1532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:25:29.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:29 smithi163 ceph-mon[36873]: pgmap v1532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:25:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:31 smithi042 ceph-mon[32775]: pgmap v1533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1761.145630 secs 2024-07-26T07:25:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1778.889060 secs 2024-07-26T07:25:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:31 smithi164 ceph-mon[36970]: pgmap v1533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1761.145630 secs 2024-07-26T07:25:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1778.889060 secs 2024-07-26T07:25:31.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:31 smithi163 ceph-mon[36873]: pgmap v1533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1761.145630 secs 2024-07-26T07:25:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1778.889060 secs 2024-07-26T07:25:33.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:33 smithi042 ceph-mon[32775]: pgmap v1534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:33.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:33 smithi164 ceph-mon[36970]: pgmap v1534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:33 smithi163 ceph-mon[36873]: pgmap v1534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:35.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:35 smithi042 ceph-mon[32775]: pgmap v1535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:35.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:35 smithi164 ceph-mon[36970]: pgmap v1535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:35.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:35 smithi163 ceph-mon[36873]: pgmap v1535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1766.145751 secs 2024-07-26T07:25:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1783.889184 secs 2024-07-26T07:25:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1766.145751 secs 2024-07-26T07:25:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1783.889184 secs 2024-07-26T07:25:36.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1766.145751 secs 2024-07-26T07:25:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1783.889184 secs 2024-07-26T07:25:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:37 smithi164 ceph-mon[36970]: pgmap v1536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:37 smithi163 ceph-mon[36873]: pgmap v1536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:37.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:37 smithi042 ceph-mon[32775]: pgmap v1536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:38.856+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:25:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:38.856+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:25:38.857 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:38.856+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:25:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:39 smithi164 ceph-mon[36970]: pgmap v1537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:25:39.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:39 smithi163 ceph-mon[36873]: pgmap v1537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:25:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:39 smithi042 ceph-mon[32775]: pgmap v1537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:25:41.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:41 smithi042 ceph-mon[32775]: pgmap v1538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:41.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1771.145896 secs 2024-07-26T07:25:41.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1788.889312 secs 2024-07-26T07:25:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:41 smithi164 ceph-mon[36970]: pgmap v1538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1771.145896 secs 2024-07-26T07:25:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1788.889312 secs 2024-07-26T07:25:41.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:41 smithi163 ceph-mon[36873]: pgmap v1538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1771.145896 secs 2024-07-26T07:25:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1788.889312 secs 2024-07-26T07:25:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:43 smithi164 ceph-mon[36970]: pgmap v1539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:43.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:43 smithi163 ceph-mon[36873]: pgmap v1539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:43.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:43 smithi042 ceph-mon[32775]: pgmap v1539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:45 smithi164 ceph-mon[36970]: pgmap v1540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:25:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:25:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:25:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:25:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:45 smithi163 ceph-mon[36873]: pgmap v1540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:25:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:25:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:25:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:25:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:45 smithi042 ceph-mon[32775]: pgmap v1540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:25:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:25:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:25:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:25:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:25:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:46 smithi164 ceph-mon[36970]: pgmap v1541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:25:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:46 smithi164 ceph-mon[36970]: pgmap v1542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:25:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1776.146024 secs 2024-07-26T07:25:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1793.889392 secs 2024-07-26T07:25:46.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:46 smithi163 ceph-mon[36873]: pgmap v1541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:25:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:46 smithi163 ceph-mon[36873]: pgmap v1542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:25:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1776.146024 secs 2024-07-26T07:25:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1793.889392 secs 2024-07-26T07:25:46.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:46 smithi042 ceph-mon[32775]: pgmap v1541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:25:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:46 smithi042 ceph-mon[32775]: pgmap v1542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:25:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1776.146024 secs 2024-07-26T07:25:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1793.889392 secs 2024-07-26T07:25:48.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:48 smithi164 ceph-mon[36970]: pgmap v1543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:25:48.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:48 smithi163 ceph-mon[36873]: pgmap v1543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:25:48.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:48 smithi042 ceph-mon[32775]: pgmap v1543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:25:50.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:50 smithi164 ceph-mon[36970]: pgmap v1544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:25:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:50 smithi163 ceph-mon[36873]: pgmap v1544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:25:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:50 smithi042 ceph-mon[32775]: pgmap v1544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:25:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1781.146150 secs 2024-07-26T07:25:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1798.889468 secs 2024-07-26T07:25:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1781.146150 secs 2024-07-26T07:25:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1798.889468 secs 2024-07-26T07:25:51.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1781.146150 secs 2024-07-26T07:25:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1798.889468 secs 2024-07-26T07:25:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:52 smithi164 ceph-mon[36970]: pgmap v1545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:25:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:52 smithi163 ceph-mon[36873]: pgmap v1545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:25:52.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:52 smithi042 ceph-mon[32775]: pgmap v1545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:25:53.860 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:53.859+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:25:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:53.859+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:25:53.861 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:53.859+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:25:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:54 smithi164 ceph-mon[36970]: pgmap v1546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:25:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:54 smithi163 ceph-mon[36873]: pgmap v1546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:25:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:54 smithi042 ceph-mon[32775]: pgmap v1546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:25:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:56 smithi164 ceph-mon[36970]: pgmap v1547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:25:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1786.146276 secs 2024-07-26T07:25:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1803.889612 secs 2024-07-26T07:25:56.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:56 smithi163 ceph-mon[36873]: pgmap v1547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:25:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1786.146276 secs 2024-07-26T07:25:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1803.889612 secs 2024-07-26T07:25:56.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:56 smithi042 ceph-mon[32775]: pgmap v1547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:25:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1786.146276 secs 2024-07-26T07:25:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1803.889612 secs 2024-07-26T07:25:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:57 smithi163 ceph-mon[36873]: pgmap v1548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:57.863 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:57.862+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T07:25:57.863 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:57.862+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T07:25:57.863 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:57.862+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff678000f80 tx=0x7ff678045000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:25:57.863 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:57.862+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff678000f80 tx=0x7ff678045000 comp rx=0 tx=0).stop 2024-07-26T07:25:57.863 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:57.862+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T07:25:57.863 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:57.862+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T07:25:57.864 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:57.862+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:25:57.864 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:57.862+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0c03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:25:57.864 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:57.862+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0c03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:25:57.864 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:57.863+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0c03f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff678031550 tx=0x7ff67000d3d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:25:57.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:57 smithi042 ceph-mon[32775]: pgmap v1548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:57 smithi164 ceph-mon[36970]: pgmap v1548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:25:58.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:58.139+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T07:25:58.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:58.140+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T07:25:58.143 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:58.143+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T07:25:58.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:25:58.144+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T07:26:00.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:25:59 smithi164 ceph-mon[36970]: pgmap v1549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:26:00.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:25:59 smithi163 ceph-mon[36873]: pgmap v1549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:26:00.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:25:59 smithi042 ceph-mon[32775]: pgmap v1549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:26:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1791.146375 secs 2024-07-26T07:26:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:01 smithi164 ceph-mon[36970]: pgmap v1550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1808.889740 secs 2024-07-26T07:26:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1791.146375 secs 2024-07-26T07:26:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:01 smithi163 ceph-mon[36873]: pgmap v1550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1808.889740 secs 2024-07-26T07:26:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1791.146375 secs 2024-07-26T07:26:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:01 smithi042 ceph-mon[32775]: pgmap v1550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1808.889740 secs 2024-07-26T07:26:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:03 smithi164 ceph-mon[36970]: pgmap v1551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:03 smithi163 ceph-mon[36873]: pgmap v1551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:03 smithi042 ceph-mon[32775]: pgmap v1551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:06.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:05 smithi164 ceph-mon[36970]: pgmap v1552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:05 smithi163 ceph-mon[36873]: pgmap v1552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:06.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:05 smithi042 ceph-mon[32775]: pgmap v1552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1796.146472 secs 2024-07-26T07:26:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1813.889887 secs 2024-07-26T07:26:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1796.146472 secs 2024-07-26T07:26:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1813.889887 secs 2024-07-26T07:26:07.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1796.146472 secs 2024-07-26T07:26:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1813.889887 secs 2024-07-26T07:26:08.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:07 smithi042 ceph-mon[32775]: pgmap v1553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:08.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:07 smithi164 ceph-mon[36970]: pgmap v1553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:07 smithi163 ceph-mon[36873]: pgmap v1553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:08.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:26:08.867+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:26:08.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:26:08.867+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:26:08.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:26:08.867+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:26:10.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:09 smithi164 ceph-mon[36970]: pgmap v1554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:26:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:09 smithi163 ceph-mon[36873]: pgmap v1554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:26:10.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:09 smithi042 ceph-mon[32775]: pgmap v1554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:26:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:10 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1801.146597 secs 2024-07-26T07:26:11.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:10 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1801.146597 secs 2024-07-26T07:26:11.295 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:10 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1801.146597 secs 2024-07-26T07:26:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:11 smithi164 ceph-mon[36970]: pgmap v1555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1818.890011 secs 2024-07-26T07:26:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:11 smithi163 ceph-mon[36873]: pgmap v1555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1818.890011 secs 2024-07-26T07:26:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:11 smithi042 ceph-mon[32775]: pgmap v1555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1818.890011 secs 2024-07-26T07:26:14.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:13 smithi164 ceph-mon[36970]: pgmap v1556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:13 smithi163 ceph-mon[36873]: pgmap v1556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:14.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:13 smithi042 ceph-mon[32775]: pgmap v1556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:15 smithi164 ceph-mon[36970]: pgmap v1557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:15 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1806.146724 secs 2024-07-26T07:26:16.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:15 smithi163 ceph-mon[36873]: pgmap v1557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:15 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1806.146724 secs 2024-07-26T07:26:16.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:15 smithi042 ceph-mon[32775]: pgmap v1557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:15 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1806.146724 secs 2024-07-26T07:26:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1823.890099 secs 2024-07-26T07:26:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1823.890099 secs 2024-07-26T07:26:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1823.890099 secs 2024-07-26T07:26:18.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:17 smithi164 ceph-mon[36970]: pgmap v1558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:17 smithi163 ceph-mon[36873]: pgmap v1558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:18.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:17 smithi042 ceph-mon[32775]: pgmap v1558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:19 smithi163 ceph-mon[36873]: pgmap v1559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:26:20.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:19 smithi042 ceph-mon[32775]: pgmap v1559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:26:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:19 smithi164 ceph-mon[36970]: pgmap v1559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:26:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:20 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1811.146830 secs 2024-07-26T07:26:21.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:20 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1811.146830 secs 2024-07-26T07:26:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:20 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1811.146830 secs 2024-07-26T07:26:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:21 smithi163 ceph-mon[36873]: pgmap v1560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1828.890180 secs 2024-07-26T07:26:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:21 smithi042 ceph-mon[32775]: pgmap v1560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1828.890180 secs 2024-07-26T07:26:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:21 smithi164 ceph-mon[36970]: pgmap v1560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1828.890180 secs 2024-07-26T07:26:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:26:23.870+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:26:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:26:23.870+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:26:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:26:23.870+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:26:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:23 smithi163 ceph-mon[36873]: pgmap v1561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:23 smithi042 ceph-mon[32775]: pgmap v1561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:24.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:23 smithi164 ceph-mon[36970]: pgmap v1561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:26.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:25 smithi163 ceph-mon[36873]: pgmap v1562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:26.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:25 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1816.146963 secs 2024-07-26T07:26:26.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:25 smithi042 ceph-mon[32775]: pgmap v1562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:25 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1816.146963 secs 2024-07-26T07:26:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:25 smithi164 ceph-mon[36970]: pgmap v1562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:25 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1816.146963 secs 2024-07-26T07:26:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1833.890275 secs 2024-07-26T07:26:27.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1833.890275 secs 2024-07-26T07:26:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1833.890275 secs 2024-07-26T07:26:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:27 smithi163 ceph-mon[36873]: pgmap v1563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:28.299 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:27 smithi164 ceph-mon[36970]: pgmap v1563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:28.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:27 smithi042 ceph-mon[32775]: pgmap v1563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:29 smithi163 ceph-mon[36873]: pgmap v1564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:26:30.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:29 smithi042 ceph-mon[32775]: pgmap v1564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:26:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:29 smithi164 ceph-mon[36970]: pgmap v1564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:26:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1821.147107 secs 2024-07-26T07:26:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1821.147107 secs 2024-07-26T07:26:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1821.147107 secs 2024-07-26T07:26:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:32 smithi163 ceph-mon[36873]: pgmap v1565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1838.890404 secs 2024-07-26T07:26:32.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:32 smithi042 ceph-mon[32775]: pgmap v1565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1838.890404 secs 2024-07-26T07:26:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:32 smithi164 ceph-mon[36970]: pgmap v1565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1838.890404 secs 2024-07-26T07:26:34.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:34 smithi042 ceph-mon[32775]: pgmap v1566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:34.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:34 smithi164 ceph-mon[36970]: pgmap v1566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:34.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:34 smithi163 ceph-mon[36873]: pgmap v1566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:35 smithi042 ceph-mon[32775]: pgmap v1567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:26:36.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:35 smithi164 ceph-mon[36970]: pgmap v1567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:26:36.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:35 smithi163 ceph-mon[36873]: pgmap v1567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:26:37.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1826.147251 secs 2024-07-26T07:26:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1843.890504 secs 2024-07-26T07:26:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1826.147251 secs 2024-07-26T07:26:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1843.890504 secs 2024-07-26T07:26:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1826.147251 secs 2024-07-26T07:26:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1843.890504 secs 2024-07-26T07:26:38.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:37 smithi042 ceph-mon[32775]: pgmap v1568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:26:38.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:37 smithi164 ceph-mon[36970]: pgmap v1568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:26:38.256 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:37 smithi163 ceph-mon[36873]: pgmap v1568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:26:38.872 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:26:38.871+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:26:38.872 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:26:38.871+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:26:38.872 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:26:38.871+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:26:40.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:39 smithi164 ceph-mon[36970]: pgmap v1569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-07-26T07:26:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:39 smithi163 ceph-mon[36873]: pgmap v1569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-07-26T07:26:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:39 smithi042 ceph-mon[32775]: pgmap v1569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-07-26T07:26:42.197 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1831.147373 secs 2024-07-26T07:26:42.197 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:41 smithi042 ceph-mon[32775]: pgmap v1570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:26:42.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1848.890649 secs 2024-07-26T07:26:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1831.147373 secs 2024-07-26T07:26:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:41 smithi164 ceph-mon[36970]: pgmap v1570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:26:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1848.890649 secs 2024-07-26T07:26:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1831.147373 secs 2024-07-26T07:26:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:41 smithi163 ceph-mon[36873]: pgmap v1570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:26:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1848.890649 secs 2024-07-26T07:26:44.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:43 smithi164 ceph-mon[36970]: pgmap v1571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:26:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:43 smithi163 ceph-mon[36873]: pgmap v1571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:26:44.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:43 smithi042 ceph-mon[32775]: pgmap v1571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:26:46.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: pgmap v1572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:26:46.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:26:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:26:46.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:26:46.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:26:46.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:26:46.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:26:46.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:26:46.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:26:46.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:26:46.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: pgmap v1573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:26:46.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: pgmap v1574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:26:46.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:26:46.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:45 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1836.147439 secs 2024-07-26T07:26:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: pgmap v1572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:26:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:26:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:26:46.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:26:46.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:26:46.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:26:46.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:26:46.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:26:46.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:26:46.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:26:46.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: pgmap v1573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:26:46.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: pgmap v1574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:26:46.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:26:46.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:45 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1836.147439 secs 2024-07-26T07:26:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: pgmap v1572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:26:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:26:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:26:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:26:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:26:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:26:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:26:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:26:46.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:26:46.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:26:46.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:26:46.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: pgmap v1573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:26:46.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: pgmap v1574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:26:46.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:26:46.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:45 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1836.147439 secs 2024-07-26T07:26:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1853.890799 secs 2024-07-26T07:26:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1853.890799 secs 2024-07-26T07:26:47.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1853.890799 secs 2024-07-26T07:26:48.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:47 smithi164 ceph-mon[36970]: pgmap v1575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:26:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:47 smithi163 ceph-mon[36873]: pgmap v1575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:26:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:47 smithi042 ceph-mon[32775]: pgmap v1575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:26:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:49 smithi163 ceph-mon[36873]: pgmap v1576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:26:49.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:49 smithi042 ceph-mon[32775]: pgmap v1576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:26:49.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:49 smithi164 ceph-mon[36970]: pgmap v1576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:26:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:50 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1841.147572 secs 2024-07-26T07:26:51.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:50 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1841.147572 secs 2024-07-26T07:26:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:50 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1841.147572 secs 2024-07-26T07:26:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1858.890923 secs 2024-07-26T07:26:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:52 smithi163 ceph-mon[36873]: pgmap v1577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:26:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1858.890923 secs 2024-07-26T07:26:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:52 smithi042 ceph-mon[32775]: pgmap v1577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:26:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1858.890923 secs 2024-07-26T07:26:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:52 smithi164 ceph-mon[36970]: pgmap v1577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:26:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:53 smithi163 ceph-mon[36873]: pgmap v1578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:26:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:26:53.874+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:26:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:26:53.874+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:26:53.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:26:53.874+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:26:53.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:53 smithi042 ceph-mon[32775]: pgmap v1578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:26:53.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:53 smithi164 ceph-mon[36970]: pgmap v1578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:26:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:55 smithi163 ceph-mon[36873]: pgmap v1579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:26:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:55 smithi042 ceph-mon[32775]: pgmap v1579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:26:55.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:55 smithi164 ceph-mon[36970]: pgmap v1579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:26:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1846.147697 secs 2024-07-26T07:26:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1863.891045 secs 2024-07-26T07:26:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1846.147697 secs 2024-07-26T07:26:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1863.891045 secs 2024-07-26T07:26:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1846.147697 secs 2024-07-26T07:26:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1863.891045 secs 2024-07-26T07:26:57.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:57 smithi042 ceph-mon[32775]: pgmap v1580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:57 smithi164 ceph-mon[36970]: pgmap v1580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:57 smithi163 ceph-mon[36873]: pgmap v1580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:26:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:26:59 smithi163 ceph-mon[36873]: pgmap v1581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:26:59 smithi042 ceph-mon[32775]: pgmap v1581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:26:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:26:59 smithi164 ceph-mon[36970]: pgmap v1581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:00 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1851.147817 secs 2024-07-26T07:27:01.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:00 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1851.147817 secs 2024-07-26T07:27:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:00 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1851.147817 secs 2024-07-26T07:27:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1868.891169 secs 2024-07-26T07:27:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:02 smithi163 ceph-mon[36873]: pgmap v1582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1868.891169 secs 2024-07-26T07:27:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:02 smithi042 ceph-mon[32775]: pgmap v1582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1868.891169 secs 2024-07-26T07:27:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:02 smithi164 ceph-mon[36970]: pgmap v1582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:03 smithi163 ceph-mon[36873]: pgmap v1583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:27:03.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:03 smithi042 ceph-mon[32775]: pgmap v1583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:27:03.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:03 smithi164 ceph-mon[36970]: pgmap v1583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:27:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:05 smithi163 ceph-mon[36873]: pgmap v1584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:05.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:05 smithi042 ceph-mon[32775]: pgmap v1584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:05 smithi164 ceph-mon[36970]: pgmap v1584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:06.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1856.147943 secs 2024-07-26T07:27:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1873.891235 secs 2024-07-26T07:27:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1856.147943 secs 2024-07-26T07:27:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1873.891235 secs 2024-07-26T07:27:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1856.147943 secs 2024-07-26T07:27:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1873.891235 secs 2024-07-26T07:27:07.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:07 smithi042 ceph-mon[32775]: pgmap v1585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:07.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:07 smithi164 ceph-mon[36970]: pgmap v1585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:08.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:07 smithi163 ceph-mon[36873]: pgmap v1585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:27:08.881+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:27:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:27:08.881+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:27:08.882 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:27:08.881+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:27:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:09 smithi163 ceph-mon[36873]: pgmap v1586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:09.807 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:09 smithi042 ceph-mon[32775]: pgmap v1586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:09 smithi164 ceph-mon[36970]: pgmap v1586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:10 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1861.148071 secs 2024-07-26T07:27:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:10 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1861.148071 secs 2024-07-26T07:27:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:10 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1861.148071 secs 2024-07-26T07:27:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1878.891354 secs 2024-07-26T07:27:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:12 smithi163 ceph-mon[36873]: pgmap v1587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:12.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1878.891354 secs 2024-07-26T07:27:12.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:12 smithi042 ceph-mon[32775]: pgmap v1587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1878.891354 secs 2024-07-26T07:27:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:12 smithi164 ceph-mon[36970]: pgmap v1587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:13 smithi163 ceph-mon[36873]: pgmap v1588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:27:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:13 smithi042 ceph-mon[32775]: pgmap v1588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:27:13.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:13 smithi164 ceph-mon[36970]: pgmap v1588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:27:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:15 smithi163 ceph-mon[36873]: pgmap v1589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:15.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:15 smithi042 ceph-mon[32775]: pgmap v1589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:15.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:15 smithi164 ceph-mon[36970]: pgmap v1589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:16.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1866.148203 secs 2024-07-26T07:27:16.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1883.891484 secs 2024-07-26T07:27:16.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1866.148203 secs 2024-07-26T07:27:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1883.891484 secs 2024-07-26T07:27:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1866.148203 secs 2024-07-26T07:27:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1883.891484 secs 2024-07-26T07:27:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:17 smithi042 ceph-mon[32775]: pgmap v1590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:17.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:17 smithi164 ceph-mon[36970]: pgmap v1590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:17 smithi163 ceph-mon[36873]: pgmap v1590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:19.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:19 smithi042 ceph-mon[32775]: pgmap v1591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:19.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:19 smithi164 ceph-mon[36970]: pgmap v1591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:20.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:19 smithi163 ceph-mon[36873]: pgmap v1591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1871.148331 secs 2024-07-26T07:27:21.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1871.148331 secs 2024-07-26T07:27:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1871.148331 secs 2024-07-26T07:27:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1888.891619 secs 2024-07-26T07:27:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:22 smithi042 ceph-mon[32775]: pgmap v1592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1888.891619 secs 2024-07-26T07:27:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:22 smithi164 ceph-mon[36970]: pgmap v1592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:22.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1888.891619 secs 2024-07-26T07:27:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:22 smithi163 ceph-mon[36873]: pgmap v1592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:27:23.884+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:27:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:27:23.884+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:27:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:27:23.884+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:27:23.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:23 smithi042 ceph-mon[32775]: pgmap v1593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:27:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:23 smithi164 ceph-mon[36970]: pgmap v1593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:27:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:23 smithi163 ceph-mon[36873]: pgmap v1593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:27:25.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:25 smithi042 ceph-mon[32775]: pgmap v1594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:25.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:25 smithi164 ceph-mon[36970]: pgmap v1594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:25 smithi163 ceph-mon[36873]: pgmap v1594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:26.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1876.148419 secs 2024-07-26T07:27:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1893.891730 secs 2024-07-26T07:27:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1876.148419 secs 2024-07-26T07:27:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1893.891730 secs 2024-07-26T07:27:27.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1876.148419 secs 2024-07-26T07:27:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1893.891730 secs 2024-07-26T07:27:27.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:27 smithi042 ceph-mon[32775]: pgmap v1595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:27 smithi164 ceph-mon[36970]: pgmap v1595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:27 smithi163 ceph-mon[36873]: pgmap v1595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:29.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:29 smithi042 ceph-mon[32775]: pgmap v1596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:29.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:29 smithi164 ceph-mon[36970]: pgmap v1596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:29 smithi163 ceph-mon[36873]: pgmap v1596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1881.148545 secs 2024-07-26T07:27:31.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1881.148545 secs 2024-07-26T07:27:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1881.148545 secs 2024-07-26T07:27:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1898.891853 secs 2024-07-26T07:27:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:32 smithi042 ceph-mon[32775]: pgmap v1597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1898.891853 secs 2024-07-26T07:27:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:32 smithi164 ceph-mon[36970]: pgmap v1597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1898.891853 secs 2024-07-26T07:27:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:32 smithi163 ceph-mon[36873]: pgmap v1597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:33 smithi042 ceph-mon[32775]: pgmap v1598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:27:33.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:33 smithi164 ceph-mon[36970]: pgmap v1598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:27:34.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:33 smithi163 ceph-mon[36873]: pgmap v1598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:27:35.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:35 smithi042 ceph-mon[32775]: pgmap v1599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:35.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:35 smithi164 ceph-mon[36970]: pgmap v1599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:36.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:35 smithi163 ceph-mon[36873]: pgmap v1599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:36.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1886.148677 secs 2024-07-26T07:27:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1903.891981 secs 2024-07-26T07:27:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1886.148677 secs 2024-07-26T07:27:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1903.891981 secs 2024-07-26T07:27:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1886.148677 secs 2024-07-26T07:27:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1903.891981 secs 2024-07-26T07:27:37.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:37 smithi042 ceph-mon[32775]: pgmap v1600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:37 smithi164 ceph-mon[36970]: pgmap v1600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:37 smithi163 ceph-mon[36873]: pgmap v1600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:38.893 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:27:38.892+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:27:38.893 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:27:38.892+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:27:38.893 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:27:38.892+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:27:39.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:39 smithi042 ceph-mon[32775]: pgmap v1601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:39.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:39 smithi164 ceph-mon[36970]: pgmap v1601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:40.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:39 smithi163 ceph-mon[36873]: pgmap v1601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1891.148802 secs 2024-07-26T07:27:41.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1891.148802 secs 2024-07-26T07:27:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1891.148802 secs 2024-07-26T07:27:42.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1908.892049 secs 2024-07-26T07:27:42.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:42 smithi042 ceph-mon[32775]: pgmap v1602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1908.892049 secs 2024-07-26T07:27:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:42 smithi164 ceph-mon[36970]: pgmap v1602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:42.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1908.892049 secs 2024-07-26T07:27:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:42 smithi163 ceph-mon[36873]: pgmap v1602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:43.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:43 smithi042 ceph-mon[32775]: pgmap v1603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:27:43.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:43 smithi164 ceph-mon[36970]: pgmap v1603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:27:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:43 smithi163 ceph-mon[36873]: pgmap v1603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:27:45.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:45 smithi042 ceph-mon[32775]: pgmap v1604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:45.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:45 smithi164 ceph-mon[36970]: pgmap v1604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:45 smithi163 ceph-mon[36873]: pgmap v1604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:27:46.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:27:46.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1896.148898 secs 2024-07-26T07:27:46.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:27:46.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:27:46.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:46 smithi164 ceph-mon[36970]: pgmap v1605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:27:46.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:46 smithi164 ceph-mon[36970]: pgmap v1606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:27:46.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:27:46.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1913.892171 secs 2024-07-26T07:27:46.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:27:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1896.148898 secs 2024-07-26T07:27:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:27:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:27:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:46 smithi042 ceph-mon[32775]: pgmap v1605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:27:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:46 smithi042 ceph-mon[32775]: pgmap v1606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:27:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:27:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1913.892171 secs 2024-07-26T07:27:46.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:27:46.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1896.148898 secs 2024-07-26T07:27:46.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:27:46.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:27:46.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:46 smithi163 ceph-mon[36873]: pgmap v1605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:27:46.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:46 smithi163 ceph-mon[36873]: pgmap v1606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:27:46.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:27:46.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1913.892171 secs 2024-07-26T07:27:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:49 smithi042 ceph-mon[32775]: pgmap v1607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:27:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:49 smithi164 ceph-mon[36970]: pgmap v1607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:27:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:49 smithi163 ceph-mon[36873]: pgmap v1607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:27:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:51 smithi042 ceph-mon[32775]: pgmap v1608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:27:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1901.149029 secs 2024-07-26T07:27:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:51 smithi164 ceph-mon[36970]: pgmap v1608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:27:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1901.149029 secs 2024-07-26T07:27:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:51 smithi163 ceph-mon[36873]: pgmap v1608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:27:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1901.149029 secs 2024-07-26T07:27:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1918.892295 secs 2024-07-26T07:27:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1918.892295 secs 2024-07-26T07:27:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1918.892295 secs 2024-07-26T07:27:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:53 smithi042 ceph-mon[32775]: pgmap v1609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:27:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:53 smithi164 ceph-mon[36970]: pgmap v1609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:27:53.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:53 smithi163 ceph-mon[36873]: pgmap v1609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:27:53.896 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:27:53.895+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:27:53.896 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:27:53.895+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:27:53.896 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:27:53.895+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:27:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:55 smithi042 ceph-mon[32775]: pgmap v1610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:27:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:55 smithi164 ceph-mon[36970]: pgmap v1610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:27:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:55 smithi163 ceph-mon[36873]: pgmap v1610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:27:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1906.149133 secs 2024-07-26T07:27:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1906.149133 secs 2024-07-26T07:27:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1906.149133 secs 2024-07-26T07:27:57.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:57 smithi042 ceph-mon[32775]: pgmap v1611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:27:57.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1923.892413 secs 2024-07-26T07:27:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:57 smithi164 ceph-mon[36970]: pgmap v1611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:27:57.503 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1923.892413 secs 2024-07-26T07:27:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:57 smithi163 ceph-mon[36873]: pgmap v1611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:27:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1923.892413 secs 2024-07-26T07:27:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:27:59 smithi042 ceph-mon[32775]: pgmap v1612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:59.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:27:59 smithi164 ceph-mon[36970]: pgmap v1612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:27:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:27:59 smithi163 ceph-mon[36873]: pgmap v1612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:01 smithi042 ceph-mon[32775]: pgmap v1613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1911.149256 secs 2024-07-26T07:28:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:01 smithi164 ceph-mon[36970]: pgmap v1613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1911.149256 secs 2024-07-26T07:28:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:01 smithi163 ceph-mon[36873]: pgmap v1613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1911.149256 secs 2024-07-26T07:28:02.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1928.892537 secs 2024-07-26T07:28:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1928.892537 secs 2024-07-26T07:28:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1928.892537 secs 2024-07-26T07:28:03.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:03 smithi042 ceph-mon[32775]: pgmap v1614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:03.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:03 smithi164 ceph-mon[36970]: pgmap v1614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:03 smithi163 ceph-mon[36873]: pgmap v1614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:05 smithi042 ceph-mon[32775]: pgmap v1615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:28:05.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:05 smithi164 ceph-mon[36970]: pgmap v1615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:28:05.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:05 smithi163 ceph-mon[36873]: pgmap v1615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:28:06.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1916.149385 secs 2024-07-26T07:28:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1916.149385 secs 2024-07-26T07:28:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1916.149385 secs 2024-07-26T07:28:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:07 smithi042 ceph-mon[32775]: pgmap v1616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1933.892681 secs 2024-07-26T07:28:07.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:07 smithi164 ceph-mon[36970]: pgmap v1616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:07.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1933.892681 secs 2024-07-26T07:28:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:07 smithi163 ceph-mon[36873]: pgmap v1616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1933.892681 secs 2024-07-26T07:28:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:08.899+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:28:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:08.899+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:28:08.900 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:08.899+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:28:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:09 smithi042 ceph-mon[32775]: pgmap v1617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:09 smithi164 ceph-mon[36970]: pgmap v1617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:09 smithi163 ceph-mon[36873]: pgmap v1617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:11 smithi042 ceph-mon[32775]: pgmap v1618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1921.149507 secs 2024-07-26T07:28:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:11 smithi164 ceph-mon[36970]: pgmap v1618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1921.149507 secs 2024-07-26T07:28:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:11 smithi163 ceph-mon[36873]: pgmap v1618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1921.149507 secs 2024-07-26T07:28:12.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1938.892803 secs 2024-07-26T07:28:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1938.892803 secs 2024-07-26T07:28:12.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1938.892803 secs 2024-07-26T07:28:13.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:13 smithi042 ceph-mon[32775]: pgmap v1619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:13 smithi164 ceph-mon[36970]: pgmap v1619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:13 smithi163 ceph-mon[36873]: pgmap v1619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:15.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:15 smithi042 ceph-mon[32775]: pgmap v1620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:28:15.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:15 smithi164 ceph-mon[36970]: pgmap v1620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:28:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:15 smithi163 ceph-mon[36873]: pgmap v1620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:28:16.373 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1926.149611 secs 2024-07-26T07:28:16.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1926.149611 secs 2024-07-26T07:28:16.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1926.149611 secs 2024-07-26T07:28:17.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:17 smithi042 ceph-mon[32775]: pgmap v1621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1943.892923 secs 2024-07-26T07:28:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:17 smithi164 ceph-mon[36970]: pgmap v1621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1943.892923 secs 2024-07-26T07:28:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:17 smithi163 ceph-mon[36873]: pgmap v1621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1943.892923 secs 2024-07-26T07:28:19.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:19 smithi042 ceph-mon[32775]: pgmap v1622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:19 smithi164 ceph-mon[36970]: pgmap v1622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:19.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:19 smithi163 ceph-mon[36873]: pgmap v1622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:21.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:21 smithi042 ceph-mon[32775]: pgmap v1623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:28:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1931.149690 secs 2024-07-26T07:28:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:21 smithi164 ceph-mon[36970]: pgmap v1623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:28:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1931.149690 secs 2024-07-26T07:28:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:21 smithi163 ceph-mon[36873]: pgmap v1623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:28:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1931.149690 secs 2024-07-26T07:28:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1948.893055 secs 2024-07-26T07:28:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1948.893055 secs 2024-07-26T07:28:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1948.893055 secs 2024-07-26T07:28:23.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:23 smithi042 ceph-mon[32775]: pgmap v1624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:28:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:23 smithi164 ceph-mon[36970]: pgmap v1624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:28:23.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:23 smithi163 ceph-mon[36873]: pgmap v1624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:28:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:23.903+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:28:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:23.903+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:28:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:23.903+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:28:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:25 smithi042 ceph-mon[32775]: pgmap v1625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-07-26T07:28:25.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:25 smithi164 ceph-mon[36970]: pgmap v1625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-07-26T07:28:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:25 smithi163 ceph-mon[36873]: pgmap v1625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-07-26T07:28:26.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1936.149846 secs 2024-07-26T07:28:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1936.149846 secs 2024-07-26T07:28:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1936.149846 secs 2024-07-26T07:28:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:27 smithi042 ceph-mon[32775]: pgmap v1626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:28:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1953.893202 secs 2024-07-26T07:28:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:27 smithi164 ceph-mon[36970]: pgmap v1626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:28:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1953.893202 secs 2024-07-26T07:28:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:27 smithi163 ceph-mon[36873]: pgmap v1626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:28:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1953.893202 secs 2024-07-26T07:28:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:29 smithi042 ceph-mon[32775]: pgmap v1627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:28:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:29 smithi164 ceph-mon[36970]: pgmap v1627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:28:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:29 smithi163 ceph-mon[36873]: pgmap v1627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:28:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:31 smithi164 ceph-mon[36970]: pgmap v1628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:28:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1941.149927 secs 2024-07-26T07:28:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:31 smithi163 ceph-mon[36873]: pgmap v1628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:28:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1941.149927 secs 2024-07-26T07:28:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:31 smithi042 ceph-mon[32775]: pgmap v1628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:28:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1941.149927 secs 2024-07-26T07:28:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1958.893347 secs 2024-07-26T07:28:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1958.893347 secs 2024-07-26T07:28:32.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1958.893347 secs 2024-07-26T07:28:33.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:33 smithi164 ceph-mon[36970]: pgmap v1629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:33 smithi163 ceph-mon[36873]: pgmap v1629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:33.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:33 smithi042 ceph-mon[32775]: pgmap v1629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:35.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:35 smithi164 ceph-mon[36970]: pgmap v1630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:28:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:35 smithi163 ceph-mon[36873]: pgmap v1630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:28:35.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:35 smithi042 ceph-mon[32775]: pgmap v1630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:28:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1946.150033 secs 2024-07-26T07:28:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1946.150033 secs 2024-07-26T07:28:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1946.150033 secs 2024-07-26T07:28:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:37 smithi164 ceph-mon[36970]: pgmap v1631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1963.893466 secs 2024-07-26T07:28:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:37 smithi163 ceph-mon[36873]: pgmap v1631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1963.893466 secs 2024-07-26T07:28:37.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:37 smithi042 ceph-mon[32775]: pgmap v1631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1963.893466 secs 2024-07-26T07:28:38.911 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:38.911+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:28:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:38.911+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:28:38.912 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:38.911+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:28:39.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:39 smithi164 ceph-mon[36970]: pgmap v1632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:39 smithi163 ceph-mon[36873]: pgmap v1632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:39.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:39 smithi042 ceph-mon[32775]: pgmap v1632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:41 smithi164 ceph-mon[36970]: pgmap v1633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1951.150170 secs 2024-07-26T07:28:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:41 smithi163 ceph-mon[36873]: pgmap v1633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1951.150170 secs 2024-07-26T07:28:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:41 smithi042 ceph-mon[32775]: pgmap v1633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1951.150170 secs 2024-07-26T07:28:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1968.893589 secs 2024-07-26T07:28:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1968.893589 secs 2024-07-26T07:28:42.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1968.893589 secs 2024-07-26T07:28:43.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:43 smithi164 ceph-mon[36970]: pgmap v1634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:43 smithi163 ceph-mon[36873]: pgmap v1634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:43 smithi042 ceph-mon[32775]: pgmap v1634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:28:45.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:45 smithi164 ceph-mon[36970]: pgmap v1635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:28:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:45 smithi163 ceph-mon[36873]: pgmap v1635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:28:45.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:45 smithi042 ceph-mon[32775]: pgmap v1635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:28:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1956.150296 secs 2024-07-26T07:28:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:28:46.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1956.150296 secs 2024-07-26T07:28:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:28:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1956.150296 secs 2024-07-26T07:28:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:28:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:47 smithi164 ceph-mon[36970]: pgmap v1636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1973.893719 secs 2024-07-26T07:28:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:28:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:28:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:28:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:47 smithi163 ceph-mon[36873]: pgmap v1636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1973.893719 secs 2024-07-26T07:28:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:28:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:28:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:28:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:47 smithi042 ceph-mon[32775]: pgmap v1636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:28:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1973.893719 secs 2024-07-26T07:28:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:28:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:28:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:28:48.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:48 smithi164 ceph-mon[36970]: pgmap v1637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:28:48.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:48 smithi164 ceph-mon[36970]: pgmap v1638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:28:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:48 smithi163 ceph-mon[36873]: pgmap v1637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:28:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:48 smithi163 ceph-mon[36873]: pgmap v1638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:28:48.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:48 smithi042 ceph-mon[32775]: pgmap v1637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:28:48.652 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:48 smithi042 ceph-mon[32775]: pgmap v1638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:28:50.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:50 smithi164 ceph-mon[36970]: pgmap v1639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:28:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:50 smithi163 ceph-mon[36873]: pgmap v1639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:28:50.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:50 smithi042 ceph-mon[32775]: pgmap v1639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:28:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1961.150453 secs 2024-07-26T07:28:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1961.150453 secs 2024-07-26T07:28:51.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1961.150453 secs 2024-07-26T07:28:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:52 smithi164 ceph-mon[36970]: pgmap v1640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:28:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1978.893849 secs 2024-07-26T07:28:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:52 smithi163 ceph-mon[36873]: pgmap v1640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:28:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1978.893849 secs 2024-07-26T07:28:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:52 smithi042 ceph-mon[32775]: pgmap v1640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:28:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1978.893849 secs 2024-07-26T07:28:53.914 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:53.914+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:28:53.914 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:53.914+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:28:53.914 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:53.914+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:28:54.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:54 smithi164 ceph-mon[36970]: pgmap v1641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:28:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:54 smithi163 ceph-mon[36873]: pgmap v1641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:28:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:54 smithi042 ceph-mon[32775]: pgmap v1641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:28:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:56 smithi164 ceph-mon[36970]: pgmap v1642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:28:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1966.150577 secs 2024-07-26T07:28:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:56 smithi163 ceph-mon[36873]: pgmap v1642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:28:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1966.150577 secs 2024-07-26T07:28:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:56 smithi042 ceph-mon[32775]: pgmap v1642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:28:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1966.150577 secs 2024-07-26T07:28:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1983.893970 secs 2024-07-26T07:28:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1983.893970 secs 2024-07-26T07:28:57.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1983.893970 secs 2024-07-26T07:28:58.166 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:58.165+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T07:28:58.167 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:58.165+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T07:28:58.167 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:58.165+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:28:58.167 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:58.165+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:28:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:58.167+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T07:28:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:58.167+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T07:28:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:58.167+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:28:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:28:58.168+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:28:58.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:28:58 smithi164 ceph-mon[36970]: pgmap v1643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:28:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:28:58 smithi163 ceph-mon[36873]: pgmap v1643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:28:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:28:58 smithi042 ceph-mon[32775]: pgmap v1643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:29:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:00 smithi163 ceph-mon[36873]: pgmap v1644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:00.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:00 smithi042 ceph-mon[32775]: pgmap v1644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:00 smithi164 ceph-mon[36970]: pgmap v1644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1971.150699 secs 2024-07-26T07:29:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1971.150699 secs 2024-07-26T07:29:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1971.150699 secs 2024-07-26T07:29:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:02 smithi163 ceph-mon[36873]: pgmap v1645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1988.894089 secs 2024-07-26T07:29:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:02 smithi042 ceph-mon[32775]: pgmap v1645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1988.894089 secs 2024-07-26T07:29:02.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:02 smithi164 ceph-mon[36970]: pgmap v1645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:02.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1988.894089 secs 2024-07-26T07:29:04.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:04 smithi163 ceph-mon[36873]: pgmap v1646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:04.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:04 smithi042 ceph-mon[32775]: pgmap v1646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:04.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:04 smithi164 ceph-mon[36970]: pgmap v1646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:06 smithi163 ceph-mon[36873]: pgmap v1647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1976.150846 secs 2024-07-26T07:29:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:06 smithi042 ceph-mon[32775]: pgmap v1647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1976.150846 secs 2024-07-26T07:29:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:06 smithi164 ceph-mon[36970]: pgmap v1647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1976.150846 secs 2024-07-26T07:29:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1993.894213 secs 2024-07-26T07:29:07.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1993.894213 secs 2024-07-26T07:29:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1993.894213 secs 2024-07-26T07:29:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:08 smithi163 ceph-mon[36873]: pgmap v1648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:08.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:08 smithi042 ceph-mon[32775]: pgmap v1648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:08 smithi164 ceph-mon[36970]: pgmap v1648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:08.922 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:29:08.922+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:29:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:29:08.922+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:29:08.923 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:29:08.922+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:29:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:10 smithi163 ceph-mon[36873]: pgmap v1649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:10.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:10 smithi042 ceph-mon[32775]: pgmap v1649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:10 smithi164 ceph-mon[36970]: pgmap v1649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1981.150967 secs 2024-07-26T07:29:11.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1981.150967 secs 2024-07-26T07:29:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1981.150967 secs 2024-07-26T07:29:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:12 smithi163 ceph-mon[36873]: pgmap v1650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1998.894314 secs 2024-07-26T07:29:12.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:12 smithi042 ceph-mon[32775]: pgmap v1650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:12.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1998.894314 secs 2024-07-26T07:29:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:12 smithi164 ceph-mon[36970]: pgmap v1650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1998.894314 secs 2024-07-26T07:29:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:14 smithi042 ceph-mon[32775]: pgmap v1651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:14.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:14 smithi164 ceph-mon[36970]: pgmap v1651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:14 smithi163 ceph-mon[36873]: pgmap v1651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:16.524 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:16 smithi164 ceph-mon[36970]: pgmap v1652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:16.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1986.151091 secs 2024-07-26T07:29:16.559 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:16 smithi163 ceph-mon[36873]: pgmap v1652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:16.559 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1986.151091 secs 2024-07-26T07:29:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:16 smithi042 ceph-mon[32775]: pgmap v1652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1986.151091 secs 2024-07-26T07:29:17.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2003.894466 secs 2024-07-26T07:29:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2003.894466 secs 2024-07-26T07:29:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2003.894466 secs 2024-07-26T07:29:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:18 smithi042 ceph-mon[32775]: pgmap v1653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:18.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:18 smithi164 ceph-mon[36970]: pgmap v1653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:18 smithi163 ceph-mon[36873]: pgmap v1653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:20.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:20 smithi042 ceph-mon[32775]: pgmap v1654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:20.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:20 smithi164 ceph-mon[36970]: pgmap v1654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:20 smithi163 ceph-mon[36873]: pgmap v1654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1991.151211 secs 2024-07-26T07:29:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1991.151211 secs 2024-07-26T07:29:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1991.151211 secs 2024-07-26T07:29:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:22 smithi042 ceph-mon[32775]: pgmap v1655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2008.894589 secs 2024-07-26T07:29:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:22 smithi164 ceph-mon[36970]: pgmap v1655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2008.894589 secs 2024-07-26T07:29:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:22 smithi163 ceph-mon[36873]: pgmap v1655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2008.894589 secs 2024-07-26T07:29:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:29:23.925+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:29:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:29:23.925+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:29:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:29:23.925+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:29:24.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:24 smithi042 ceph-mon[32775]: pgmap v1656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:24.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:24 smithi164 ceph-mon[36970]: pgmap v1656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:24 smithi163 ceph-mon[36873]: pgmap v1656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:26 smithi042 ceph-mon[32775]: pgmap v1657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 1996.151339 secs 2024-07-26T07:29:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:26 smithi164 ceph-mon[36970]: pgmap v1657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 1996.151339 secs 2024-07-26T07:29:26.756 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:26 smithi163 ceph-mon[36873]: pgmap v1657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 1996.151339 secs 2024-07-26T07:29:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2013.894736 secs 2024-07-26T07:29:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2013.894736 secs 2024-07-26T07:29:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2013.894736 secs 2024-07-26T07:29:28.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:28 smithi042 ceph-mon[32775]: pgmap v1658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:28 smithi164 ceph-mon[36970]: pgmap v1658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:28 smithi163 ceph-mon[36873]: pgmap v1658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:30.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:30 smithi042 ceph-mon[32775]: pgmap v1659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:30.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:30 smithi164 ceph-mon[36970]: pgmap v1659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:30 smithi163 ceph-mon[36873]: pgmap v1659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2001.151461 secs 2024-07-26T07:29:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2001.151461 secs 2024-07-26T07:29:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2001.151461 secs 2024-07-26T07:29:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:32 smithi042 ceph-mon[32775]: pgmap v1660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2018.894879 secs 2024-07-26T07:29:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:32 smithi164 ceph-mon[36970]: pgmap v1660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2018.894879 secs 2024-07-26T07:29:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:32 smithi163 ceph-mon[36873]: pgmap v1660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2018.894879 secs 2024-07-26T07:29:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:34 smithi042 ceph-mon[32775]: pgmap v1661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:34 smithi164 ceph-mon[36970]: pgmap v1661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:34 smithi163 ceph-mon[36873]: pgmap v1661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:36 smithi042 ceph-mon[32775]: pgmap v1662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2006.151548 secs 2024-07-26T07:29:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:36 smithi164 ceph-mon[36970]: pgmap v1662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2006.151548 secs 2024-07-26T07:29:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:36 smithi163 ceph-mon[36873]: pgmap v1662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2006.151548 secs 2024-07-26T07:29:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2023.895001 secs 2024-07-26T07:29:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2023.895001 secs 2024-07-26T07:29:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2023.895001 secs 2024-07-26T07:29:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:38 smithi042 ceph-mon[32775]: pgmap v1663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:38 smithi164 ceph-mon[36970]: pgmap v1663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:38 smithi163 ceph-mon[36873]: pgmap v1663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:29:38.933+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:29:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:29:38.933+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:29:38.934 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:29:38.933+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:29:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:40 smithi042 ceph-mon[32775]: pgmap v1664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:40 smithi164 ceph-mon[36970]: pgmap v1664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:40 smithi163 ceph-mon[36873]: pgmap v1664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2011.151675 secs 2024-07-26T07:29:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2011.151675 secs 2024-07-26T07:29:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2011.151675 secs 2024-07-26T07:29:42.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:42 smithi042 ceph-mon[32775]: pgmap v1665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2028.895101 secs 2024-07-26T07:29:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:42 smithi164 ceph-mon[36970]: pgmap v1665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2028.895101 secs 2024-07-26T07:29:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:42 smithi163 ceph-mon[36873]: pgmap v1665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2028.895101 secs 2024-07-26T07:29:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:44 smithi042 ceph-mon[32775]: pgmap v1666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:44 smithi164 ceph-mon[36970]: pgmap v1666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:44 smithi163 ceph-mon[36873]: pgmap v1666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:46.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:46 smithi042 ceph-mon[32775]: pgmap v1667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2016.151775 secs 2024-07-26T07:29:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:46 smithi164 ceph-mon[36970]: pgmap v1667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2016.151775 secs 2024-07-26T07:29:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:46 smithi163 ceph-mon[36873]: pgmap v1667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:29:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2016.151775 secs 2024-07-26T07:29:47.536 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2033.895202 secs 2024-07-26T07:29:47.536 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:29:47.610 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2033.895202 secs 2024-07-26T07:29:47.610 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:29:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2033.895202 secs 2024-07-26T07:29:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:29:48.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:48 smithi042 ceph-mon[32775]: pgmap v1668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:48.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:48 smithi164 ceph-mon[36970]: pgmap v1668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:48 smithi163 ceph-mon[36873]: pgmap v1668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:29:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:29:49.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:29:49.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:29:49.760 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:29:49.760 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:29:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:29:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:29:50.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:50 smithi164 ceph-mon[36970]: pgmap v1669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:50.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:50 smithi164 ceph-mon[36970]: pgmap v1670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:29:50.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:50 smithi164 ceph-mon[36970]: pgmap v1671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:29:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:50 smithi163 ceph-mon[36873]: pgmap v1669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:50 smithi163 ceph-mon[36873]: pgmap v1670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:29:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:50 smithi163 ceph-mon[36873]: pgmap v1671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:29:50.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:50 smithi042 ceph-mon[32775]: pgmap v1669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:29:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:50 smithi042 ceph-mon[32775]: pgmap v1670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:29:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:50 smithi042 ceph-mon[32775]: pgmap v1671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:29:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2021.151898 secs 2024-07-26T07:29:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2038.895316 secs 2024-07-26T07:29:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2021.151898 secs 2024-07-26T07:29:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2038.895316 secs 2024-07-26T07:29:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2021.151898 secs 2024-07-26T07:29:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2038.895316 secs 2024-07-26T07:29:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:52 smithi164 ceph-mon[36970]: pgmap v1672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:29:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:52 smithi163 ceph-mon[36873]: pgmap v1672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:29:52.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:52 smithi042 ceph-mon[32775]: pgmap v1672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:29:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:29:53.936+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:29:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:29:53.936+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:29:53.937 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:29:53.936+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:29:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:54 smithi164 ceph-mon[36970]: pgmap v1673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:29:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:54 smithi163 ceph-mon[36873]: pgmap v1673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:29:54.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:54 smithi042 ceph-mon[32775]: pgmap v1673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:29:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:56 smithi164 ceph-mon[36970]: pgmap v1674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:29:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2026.151965 secs 2024-07-26T07:29:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2043.895437 secs 2024-07-26T07:29:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:56 smithi163 ceph-mon[36873]: pgmap v1674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:29:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2026.151965 secs 2024-07-26T07:29:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2043.895437 secs 2024-07-26T07:29:56.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:56 smithi042 ceph-mon[32775]: pgmap v1674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:29:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2026.151965 secs 2024-07-26T07:29:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2043.895437 secs 2024-07-26T07:29:58.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:29:58 smithi164 ceph-mon[36970]: pgmap v1675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:29:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:29:58 smithi163 ceph-mon[36873]: pgmap v1675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:29:58.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:29:58 smithi042 ceph-mon[32775]: pgmap v1675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:30:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:00 smithi164 ceph-mon[36970]: pgmap v1676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:30:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:30:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:00 smithi163 ceph-mon[36873]: pgmap v1676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:30:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:30:00.816 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:00 smithi042 ceph-mon[32775]: pgmap v1676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:30:00.816 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:30:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2031.152128 secs 2024-07-26T07:30:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2048.895557 secs 2024-07-26T07:30:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2031.152128 secs 2024-07-26T07:30:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2048.895557 secs 2024-07-26T07:30:01.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2031.152128 secs 2024-07-26T07:30:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2048.895557 secs 2024-07-26T07:30:02.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:02 smithi164 ceph-mon[36970]: pgmap v1677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:02 smithi163 ceph-mon[36873]: pgmap v1677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:02.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:02 smithi042 ceph-mon[32775]: pgmap v1677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:04.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:04 smithi164 ceph-mon[36970]: pgmap v1678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:04 smithi163 ceph-mon[36873]: pgmap v1678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:04.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:04 smithi042 ceph-mon[32775]: pgmap v1678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:06 smithi164 ceph-mon[36970]: pgmap v1679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2036.152257 secs 2024-07-26T07:30:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2053.895678 secs 2024-07-26T07:30:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:06 smithi163 ceph-mon[36873]: pgmap v1679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2036.152257 secs 2024-07-26T07:30:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2053.895678 secs 2024-07-26T07:30:06.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:06 smithi042 ceph-mon[32775]: pgmap v1679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2036.152257 secs 2024-07-26T07:30:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2053.895678 secs 2024-07-26T07:30:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:08 smithi164 ceph-mon[36970]: pgmap v1680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:08 smithi163 ceph-mon[36873]: pgmap v1680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:08.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:08 smithi042 ceph-mon[32775]: pgmap v1680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:30:08.944+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:30:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:30:08.944+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:30:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:30:08.944+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:30:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:10 smithi164 ceph-mon[36970]: pgmap v1681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:10 smithi163 ceph-mon[36873]: pgmap v1681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:10 smithi042 ceph-mon[32775]: pgmap v1681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2041.152379 secs 2024-07-26T07:30:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2058.895803 secs 2024-07-26T07:30:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2041.152379 secs 2024-07-26T07:30:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2058.895803 secs 2024-07-26T07:30:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2041.152379 secs 2024-07-26T07:30:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2058.895803 secs 2024-07-26T07:30:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:12 smithi164 ceph-mon[36970]: pgmap v1682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:12 smithi163 ceph-mon[36873]: pgmap v1682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:12.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:12 smithi042 ceph-mon[32775]: pgmap v1682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:14.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:14 smithi164 ceph-mon[36970]: pgmap v1683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:14 smithi163 ceph-mon[36873]: pgmap v1683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:14 smithi042 ceph-mon[32775]: pgmap v1683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:16 smithi163 ceph-mon[36873]: pgmap v1684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2046.152450 secs 2024-07-26T07:30:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2063.895925 secs 2024-07-26T07:30:16.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:16 smithi042 ceph-mon[32775]: pgmap v1684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2046.152450 secs 2024-07-26T07:30:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2063.895925 secs 2024-07-26T07:30:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:16 smithi164 ceph-mon[36970]: pgmap v1684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2046.152450 secs 2024-07-26T07:30:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2063.895925 secs 2024-07-26T07:30:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:18 smithi163 ceph-mon[36873]: pgmap v1685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:18.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:18 smithi042 ceph-mon[32775]: pgmap v1685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:18 smithi164 ceph-mon[36970]: pgmap v1685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:20 smithi163 ceph-mon[36873]: pgmap v1686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:20.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:20 smithi042 ceph-mon[32775]: pgmap v1686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:20 smithi164 ceph-mon[36970]: pgmap v1686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2051.152550 secs 2024-07-26T07:30:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2068.896046 secs 2024-07-26T07:30:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2051.152550 secs 2024-07-26T07:30:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2068.896046 secs 2024-07-26T07:30:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2051.152550 secs 2024-07-26T07:30:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2068.896046 secs 2024-07-26T07:30:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:22 smithi163 ceph-mon[36873]: pgmap v1687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:22.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:22 smithi042 ceph-mon[32775]: pgmap v1687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:22 smithi164 ceph-mon[36970]: pgmap v1687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:23 smithi163 ceph-mon[36873]: pgmap v1688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:23.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:23 smithi042 ceph-mon[32775]: pgmap v1688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:30:23.947+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:30:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:30:23.947+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:30:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:30:23.947+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:30:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:23 smithi164 ceph-mon[36970]: pgmap v1688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:26.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:26 smithi042 ceph-mon[32775]: pgmap v1689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2056.152635 secs 2024-07-26T07:30:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:26 smithi164 ceph-mon[36970]: pgmap v1689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2056.152635 secs 2024-07-26T07:30:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:26 smithi163 ceph-mon[36873]: pgmap v1689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2056.152635 secs 2024-07-26T07:30:27.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2073.896171 secs 2024-07-26T07:30:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2073.896171 secs 2024-07-26T07:30:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2073.896171 secs 2024-07-26T07:30:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:28 smithi042 ceph-mon[32775]: pgmap v1690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:28.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:28 smithi164 ceph-mon[36970]: pgmap v1690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:28 smithi163 ceph-mon[36873]: pgmap v1690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:30 smithi042 ceph-mon[32775]: pgmap v1691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:30 smithi164 ceph-mon[36970]: pgmap v1691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:30 smithi163 ceph-mon[36873]: pgmap v1691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:31.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2061.152756 secs 2024-07-26T07:30:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2061.152756 secs 2024-07-26T07:30:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2061.152756 secs 2024-07-26T07:30:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:32 smithi042 ceph-mon[32775]: pgmap v1692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2078.896292 secs 2024-07-26T07:30:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:32 smithi164 ceph-mon[36970]: pgmap v1692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2078.896292 secs 2024-07-26T07:30:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:32 smithi163 ceph-mon[36873]: pgmap v1692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2078.896292 secs 2024-07-26T07:30:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:34 smithi042 ceph-mon[32775]: pgmap v1693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:34.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:34 smithi164 ceph-mon[36970]: pgmap v1693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:34 smithi163 ceph-mon[36873]: pgmap v1693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:35 smithi042 ceph-mon[32775]: pgmap v1694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:36.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:35 smithi164 ceph-mon[36970]: pgmap v1694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:35 smithi163 ceph-mon[36873]: pgmap v1694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:37.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2066.152885 secs 2024-07-26T07:30:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2083.896402 secs 2024-07-26T07:30:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2066.152885 secs 2024-07-26T07:30:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2083.896402 secs 2024-07-26T07:30:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2066.152885 secs 2024-07-26T07:30:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2083.896402 secs 2024-07-26T07:30:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:37 smithi042 ceph-mon[32775]: pgmap v1695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:38.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:37 smithi164 ceph-mon[36970]: pgmap v1695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:37 smithi163 ceph-mon[36873]: pgmap v1695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:30:38.955+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:30:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:30:38.955+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:30:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:30:38.955+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:30:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:40 smithi042 ceph-mon[32775]: pgmap v1696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:40.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:40 smithi164 ceph-mon[36970]: pgmap v1696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:40 smithi163 ceph-mon[36873]: pgmap v1696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2071.153008 secs 2024-07-26T07:30:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2071.153008 secs 2024-07-26T07:30:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2071.153008 secs 2024-07-26T07:30:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2088.896522 secs 2024-07-26T07:30:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:42 smithi042 ceph-mon[32775]: pgmap v1697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2088.896522 secs 2024-07-26T07:30:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:42 smithi164 ceph-mon[36970]: pgmap v1697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2088.896522 secs 2024-07-26T07:30:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:42 smithi163 ceph-mon[36873]: pgmap v1697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:44 smithi042 ceph-mon[32775]: pgmap v1698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:44.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:44 smithi164 ceph-mon[36970]: pgmap v1698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:44 smithi163 ceph-mon[36873]: pgmap v1698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:46.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:46 smithi042 ceph-mon[32775]: pgmap v1699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2076.153149 secs 2024-07-26T07:30:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:46 smithi164 ceph-mon[36970]: pgmap v1699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2076.153149 secs 2024-07-26T07:30:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:46 smithi163 ceph-mon[36873]: pgmap v1699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2076.153149 secs 2024-07-26T07:30:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2093.896650 secs 2024-07-26T07:30:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2093.896650 secs 2024-07-26T07:30:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2093.896650 secs 2024-07-26T07:30:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:48 smithi042 ceph-mon[32775]: pgmap v1700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:48.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:48 smithi164 ceph-mon[36970]: pgmap v1700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:48 smithi163 ceph-mon[36873]: pgmap v1700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:30:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:50 smithi042 ceph-mon[32775]: pgmap v1701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:30:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:30:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:30:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:50 smithi164 ceph-mon[36970]: pgmap v1701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:50.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:30:50.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:30:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:30:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:50 smithi163 ceph-mon[36873]: pgmap v1701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:30:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:30:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:30:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:30:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:30:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:51 smithi164 ceph-mon[36970]: pgmap v1702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T07:30:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:51 smithi164 ceph-mon[36970]: pgmap v1703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:30:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2081.153274 secs 2024-07-26T07:30:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2098.896789 secs 2024-07-26T07:30:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:51 smithi163 ceph-mon[36873]: pgmap v1702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T07:30:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:51 smithi163 ceph-mon[36873]: pgmap v1703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:30:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2081.153274 secs 2024-07-26T07:30:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2098.896789 secs 2024-07-26T07:30:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:51 smithi042 ceph-mon[32775]: pgmap v1702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T07:30:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:51 smithi042 ceph-mon[32775]: pgmap v1703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:30:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2081.153274 secs 2024-07-26T07:30:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2098.896789 secs 2024-07-26T07:30:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:53 smithi163 ceph-mon[36873]: pgmap v1704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:30:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:53 smithi042 ceph-mon[32775]: pgmap v1704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:30:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:30:53.957+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:30:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:30:53.957+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:30:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:30:53.957+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:30:53.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:53 smithi164 ceph-mon[36970]: pgmap v1704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:30:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:55 smithi163 ceph-mon[36873]: pgmap v1705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T07:30:55.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:55 smithi042 ceph-mon[32775]: pgmap v1705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T07:30:55.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:55 smithi164 ceph-mon[36970]: pgmap v1705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T07:30:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2086.153427 secs 2024-07-26T07:30:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2103.896950 secs 2024-07-26T07:30:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2086.153427 secs 2024-07-26T07:30:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2103.896950 secs 2024-07-26T07:30:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2086.153427 secs 2024-07-26T07:30:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2103.896950 secs 2024-07-26T07:30:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:57 smithi163 ceph-mon[36873]: pgmap v1706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:30:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:57 smithi042 ceph-mon[32775]: pgmap v1706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:30:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:57 smithi164 ceph-mon[36970]: pgmap v1706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:30:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:30:59 smithi163 ceph-mon[36873]: pgmap v1707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:30:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:30:59 smithi042 ceph-mon[32775]: pgmap v1707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:30:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:30:59 smithi164 ceph-mon[36970]: pgmap v1707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:31:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:01 smithi163 ceph-mon[36873]: pgmap v1708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:31:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2091.153557 secs 2024-07-26T07:31:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2108.897078 secs 2024-07-26T07:31:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:01 smithi042 ceph-mon[32775]: pgmap v1708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:31:01.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2091.153557 secs 2024-07-26T07:31:01.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2108.897078 secs 2024-07-26T07:31:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:01 smithi164 ceph-mon[36970]: pgmap v1708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:31:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2091.153557 secs 2024-07-26T07:31:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2108.897078 secs 2024-07-26T07:31:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:02 smithi042 ceph-mon[32775]: pgmap v1709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:02 smithi164 ceph-mon[36970]: pgmap v1709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:02 smithi163 ceph-mon[36873]: pgmap v1709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:04 smithi164 ceph-mon[36970]: pgmap v1710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:31:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:04 smithi163 ceph-mon[36873]: pgmap v1710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:31:05.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:04 smithi042 ceph-mon[32775]: pgmap v1710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:31:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2096.153683 secs 2024-07-26T07:31:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:06 smithi164 ceph-mon[36970]: pgmap v1711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2113.897204 secs 2024-07-26T07:31:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2096.153683 secs 2024-07-26T07:31:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:06 smithi163 ceph-mon[36873]: pgmap v1711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2113.897204 secs 2024-07-26T07:31:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2096.153683 secs 2024-07-26T07:31:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:06 smithi042 ceph-mon[32775]: pgmap v1711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2113.897204 secs 2024-07-26T07:31:08.966 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:31:08.966+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:31:08.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:31:08.966+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:31:08.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:31:08.966+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:31:09.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:08 smithi164 ceph-mon[36970]: pgmap v1712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:08 smithi163 ceph-mon[36873]: pgmap v1712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:08 smithi042 ceph-mon[32775]: pgmap v1712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:10 smithi164 ceph-mon[36970]: pgmap v1713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:10 smithi163 ceph-mon[36873]: pgmap v1713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:11.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:10 smithi042 ceph-mon[32775]: pgmap v1713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2101.153781 secs 2024-07-26T07:31:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2118.897327 secs 2024-07-26T07:31:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2101.153781 secs 2024-07-26T07:31:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2118.897327 secs 2024-07-26T07:31:12.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2101.153781 secs 2024-07-26T07:31:12.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2118.897327 secs 2024-07-26T07:31:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:12 smithi164 ceph-mon[36970]: pgmap v1714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:12 smithi163 ceph-mon[36873]: pgmap v1714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:13.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:12 smithi042 ceph-mon[32775]: pgmap v1714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:15.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:14 smithi164 ceph-mon[36970]: pgmap v1715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:31:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:14 smithi163 ceph-mon[36873]: pgmap v1715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:31:15.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:14 smithi042 ceph-mon[32775]: pgmap v1715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:31:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2106.153907 secs 2024-07-26T07:31:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:16 smithi164 ceph-mon[36970]: pgmap v1716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2123.897409 secs 2024-07-26T07:31:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2106.153907 secs 2024-07-26T07:31:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:16 smithi163 ceph-mon[36873]: pgmap v1716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2123.897409 secs 2024-07-26T07:31:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2106.153907 secs 2024-07-26T07:31:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:16 smithi042 ceph-mon[32775]: pgmap v1716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2123.897409 secs 2024-07-26T07:31:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:18 smithi164 ceph-mon[36970]: pgmap v1717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:18 smithi163 ceph-mon[36873]: pgmap v1717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:18 smithi042 ceph-mon[32775]: pgmap v1717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:20 smithi164 ceph-mon[36970]: pgmap v1718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:20 smithi163 ceph-mon[36873]: pgmap v1718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:21.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:20 smithi042 ceph-mon[32775]: pgmap v1718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2111.154027 secs 2024-07-26T07:31:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2128.897562 secs 2024-07-26T07:31:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2111.154027 secs 2024-07-26T07:31:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2128.897562 secs 2024-07-26T07:31:22.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2111.154027 secs 2024-07-26T07:31:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2128.897562 secs 2024-07-26T07:31:23.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:22 smithi164 ceph-mon[36970]: pgmap v1719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:22 smithi163 ceph-mon[36873]: pgmap v1719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:22 smithi042 ceph-mon[32775]: pgmap v1719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:31:23.968+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:31:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:31:23.968+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:31:23.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:31:23.968+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:31:25.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:24 smithi164 ceph-mon[36970]: pgmap v1720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:31:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:24 smithi163 ceph-mon[36873]: pgmap v1720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:31:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:24 smithi042 ceph-mon[32775]: pgmap v1720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:31:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2116.154155 secs 2024-07-26T07:31:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:26 smithi164 ceph-mon[36970]: pgmap v1721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2133.897682 secs 2024-07-26T07:31:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2116.154155 secs 2024-07-26T07:31:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:26 smithi163 ceph-mon[36873]: pgmap v1721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2133.897682 secs 2024-07-26T07:31:27.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2116.154155 secs 2024-07-26T07:31:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:26 smithi042 ceph-mon[32775]: pgmap v1721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2133.897682 secs 2024-07-26T07:31:29.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:28 smithi164 ceph-mon[36970]: pgmap v1722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:28 smithi163 ceph-mon[36873]: pgmap v1722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:28 smithi042 ceph-mon[32775]: pgmap v1722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:30 smithi163 ceph-mon[36873]: pgmap v1723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:30 smithi042 ceph-mon[32775]: pgmap v1723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:30 smithi164 ceph-mon[36970]: pgmap v1723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2121.154279 secs 2024-07-26T07:31:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2138.897805 secs 2024-07-26T07:31:32.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2121.154279 secs 2024-07-26T07:31:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2138.897805 secs 2024-07-26T07:31:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2121.154279 secs 2024-07-26T07:31:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2138.897805 secs 2024-07-26T07:31:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:32 smithi163 ceph-mon[36873]: pgmap v1724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:32 smithi042 ceph-mon[32775]: pgmap v1724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:33.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:32 smithi164 ceph-mon[36970]: pgmap v1724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:34 smithi163 ceph-mon[36873]: pgmap v1725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:31:35.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:34 smithi042 ceph-mon[32775]: pgmap v1725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:31:35.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:34 smithi164 ceph-mon[36970]: pgmap v1725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:31:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2126.154423 secs 2024-07-26T07:31:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:36 smithi163 ceph-mon[36873]: pgmap v1726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2143.897912 secs 2024-07-26T07:31:37.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2126.154423 secs 2024-07-26T07:31:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:36 smithi042 ceph-mon[32775]: pgmap v1726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2143.897912 secs 2024-07-26T07:31:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2126.154423 secs 2024-07-26T07:31:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:36 smithi164 ceph-mon[36970]: pgmap v1726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2143.897912 secs 2024-07-26T07:31:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:31:38.976+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:31:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:31:38.976+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:31:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:31:38.976+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:31:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:38 smithi163 ceph-mon[36873]: pgmap v1727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:39.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:38 smithi042 ceph-mon[32775]: pgmap v1727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:39.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:38 smithi164 ceph-mon[36970]: pgmap v1727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:40 smithi163 ceph-mon[36873]: pgmap v1728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:40 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2131.154580 secs 2024-07-26T07:31:41.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:40 smithi042 ceph-mon[32775]: pgmap v1728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:40 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2131.154580 secs 2024-07-26T07:31:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:40 smithi164 ceph-mon[36970]: pgmap v1728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:40 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2131.154580 secs 2024-07-26T07:31:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2148.898056 secs 2024-07-26T07:31:42.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2148.898056 secs 2024-07-26T07:31:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2148.898056 secs 2024-07-26T07:31:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:43 smithi163 ceph-mon[36873]: pgmap v1729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:43.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:43 smithi042 ceph-mon[32775]: pgmap v1729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:43.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:43 smithi164 ceph-mon[36970]: pgmap v1729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:45 smithi163 ceph-mon[36873]: pgmap v1730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:31:45.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:45 smithi042 ceph-mon[32775]: pgmap v1730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:31:45.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:45 smithi164 ceph-mon[36970]: pgmap v1730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:31:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2136.154731 secs 2024-07-26T07:31:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2136.154731 secs 2024-07-26T07:31:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2136.154731 secs 2024-07-26T07:31:47.373 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:47 smithi164 ceph-mon[36970]: pgmap v1731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:47.373 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2153.898203 secs 2024-07-26T07:31:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:47 smithi042 ceph-mon[32775]: pgmap v1731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2153.898203 secs 2024-07-26T07:31:47.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:47 smithi163 ceph-mon[36873]: pgmap v1731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:47.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2153.898203 secs 2024-07-26T07:31:49.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:49 smithi042 ceph-mon[32775]: pgmap v1732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:49.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:49 smithi164 ceph-mon[36970]: pgmap v1732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:49 smithi163 ceph-mon[36873]: pgmap v1732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:31:50.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:31:50.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:31:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:31:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:51 smithi042 ceph-mon[32775]: pgmap v1733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:31:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:31:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:51 smithi042 ceph-mon[32775]: pgmap v1734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:31:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:51 smithi042 ceph-mon[32775]: pgmap v1735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:31:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:31:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2141.154832 secs 2024-07-26T07:31:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:51 smithi164 ceph-mon[36970]: pgmap v1733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:31:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:31:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:51 smithi164 ceph-mon[36970]: pgmap v1734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:31:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:51 smithi164 ceph-mon[36970]: pgmap v1735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:31:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:31:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2141.154832 secs 2024-07-26T07:31:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:51 smithi163 ceph-mon[36873]: pgmap v1733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:31:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:31:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:31:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:51 smithi163 ceph-mon[36873]: pgmap v1734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:31:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:51 smithi163 ceph-mon[36873]: pgmap v1735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:31:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:31:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2141.154832 secs 2024-07-26T07:31:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2158.898284 secs 2024-07-26T07:31:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2158.898284 secs 2024-07-26T07:31:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2158.898284 secs 2024-07-26T07:31:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:53 smithi042 ceph-mon[32775]: pgmap v1736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:31:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:53 smithi164 ceph-mon[36970]: pgmap v1736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:31:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:53 smithi163 ceph-mon[36873]: pgmap v1736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:31:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:31:53.979+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:31:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:31:53.979+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:31:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:31:53.979+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:31:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:55 smithi163 ceph-mon[36873]: pgmap v1737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:31:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:55 smithi042 ceph-mon[32775]: pgmap v1737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:31:55.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:55 smithi164 ceph-mon[36970]: pgmap v1737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:31:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2146.154965 secs 2024-07-26T07:31:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2163.898382 secs 2024-07-26T07:31:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2146.154965 secs 2024-07-26T07:31:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2163.898382 secs 2024-07-26T07:31:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2146.154965 secs 2024-07-26T07:31:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2163.898382 secs 2024-07-26T07:31:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:57 smithi163 ceph-mon[36873]: pgmap v1738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:31:57.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:57 smithi042 ceph-mon[32775]: pgmap v1738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:31:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:57 smithi164 ceph-mon[36970]: pgmap v1738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:31:58.194 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:31:58.194+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:31:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:31:58.194+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:31:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:31:58.194+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:31:58.195 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:31:58.194+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:31:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:31:59 smithi163 ceph-mon[36873]: pgmap v1739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:31:59.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:31:59 smithi042 ceph-mon[32775]: pgmap v1739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:31:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:31:59 smithi164 ceph-mon[36970]: pgmap v1739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:32:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:01 smithi163 ceph-mon[36873]: pgmap v1740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:32:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2151.155089 secs 2024-07-26T07:32:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2168.898504 secs 2024-07-26T07:32:01.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:01 smithi042 ceph-mon[32775]: pgmap v1740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:32:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2151.155089 secs 2024-07-26T07:32:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2168.898504 secs 2024-07-26T07:32:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:01 smithi164 ceph-mon[36970]: pgmap v1740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:32:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2151.155089 secs 2024-07-26T07:32:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2168.898504 secs 2024-07-26T07:32:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:03 smithi163 ceph-mon[36873]: pgmap v1741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:03.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:03 smithi042 ceph-mon[32775]: pgmap v1741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:03.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:03 smithi164 ceph-mon[36970]: pgmap v1741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:05 smithi163 ceph-mon[36873]: pgmap v1742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:05 smithi042 ceph-mon[32775]: pgmap v1742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:05 smithi164 ceph-mon[36970]: pgmap v1742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2156.155246 secs 2024-07-26T07:32:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2173.898630 secs 2024-07-26T07:32:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2156.155246 secs 2024-07-26T07:32:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2173.898630 secs 2024-07-26T07:32:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2156.155246 secs 2024-07-26T07:32:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2173.898630 secs 2024-07-26T07:32:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:07 smithi163 ceph-mon[36873]: pgmap v1743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:07 smithi042 ceph-mon[32775]: pgmap v1743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:07.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:07 smithi164 ceph-mon[36970]: pgmap v1743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:08.988 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:32:08.988+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:32:08.989 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:32:08.988+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:32:08.989 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:32:08.988+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:32:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:09 smithi163 ceph-mon[36873]: pgmap v1744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:32:09.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:09 smithi042 ceph-mon[32775]: pgmap v1744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:32:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:09 smithi164 ceph-mon[36970]: pgmap v1744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:32:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:10 smithi042 ceph-mon[32775]: pgmap v1745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:10 smithi164 ceph-mon[36970]: pgmap v1745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:10 smithi163 ceph-mon[36873]: pgmap v1745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:11.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2161.155310 secs 2024-07-26T07:32:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2178.898750 secs 2024-07-26T07:32:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2161.155310 secs 2024-07-26T07:32:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2178.898750 secs 2024-07-26T07:32:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2161.155310 secs 2024-07-26T07:32:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2178.898750 secs 2024-07-26T07:32:12.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:12 smithi042 ceph-mon[32775]: pgmap v1746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:12 smithi164 ceph-mon[36970]: pgmap v1746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:12 smithi163 ceph-mon[36873]: pgmap v1746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:15 smithi163 ceph-mon[36873]: pgmap v1747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:15.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:15 smithi042 ceph-mon[32775]: pgmap v1747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:15.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:15 smithi164 ceph-mon[36970]: pgmap v1747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2166.155434 secs 2024-07-26T07:32:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2183.898872 secs 2024-07-26T07:32:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2166.155434 secs 2024-07-26T07:32:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2183.898872 secs 2024-07-26T07:32:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2166.155434 secs 2024-07-26T07:32:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2183.898872 secs 2024-07-26T07:32:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:17 smithi163 ceph-mon[36873]: pgmap v1748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:17.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:17 smithi164 ceph-mon[36970]: pgmap v1748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:17 smithi042 ceph-mon[32775]: pgmap v1748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:19 smithi163 ceph-mon[36873]: pgmap v1749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:32:19.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:19 smithi042 ceph-mon[32775]: pgmap v1749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:32:19.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:19 smithi164 ceph-mon[36970]: pgmap v1749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:32:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:21 smithi163 ceph-mon[36873]: pgmap v1750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2171.155542 secs 2024-07-26T07:32:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2188.898998 secs 2024-07-26T07:32:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:21 smithi042 ceph-mon[32775]: pgmap v1750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2171.155542 secs 2024-07-26T07:32:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2188.898998 secs 2024-07-26T07:32:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:21 smithi164 ceph-mon[36970]: pgmap v1750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2171.155542 secs 2024-07-26T07:32:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2188.898998 secs 2024-07-26T07:32:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:23 smithi163 ceph-mon[36873]: pgmap v1751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:23.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:23 smithi042 ceph-mon[32775]: pgmap v1751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:23 smithi164 ceph-mon[36970]: pgmap v1751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:32:23.991+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:32:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:32:23.991+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:32:23.991 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:32:23.991+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:32:25.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:25 smithi163 ceph-mon[36873]: pgmap v1752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:25.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:25 smithi042 ceph-mon[32775]: pgmap v1752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:25.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:25 smithi164 ceph-mon[36970]: pgmap v1752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2176.155666 secs 2024-07-26T07:32:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2193.899064 secs 2024-07-26T07:32:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2176.155666 secs 2024-07-26T07:32:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2193.899064 secs 2024-07-26T07:32:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2176.155666 secs 2024-07-26T07:32:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2193.899064 secs 2024-07-26T07:32:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:27 smithi042 ceph-mon[32775]: pgmap v1753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:27 smithi164 ceph-mon[36970]: pgmap v1753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:27 smithi163 ceph-mon[36873]: pgmap v1753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:28 smithi042 ceph-mon[32775]: pgmap v1754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:32:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:28 smithi164 ceph-mon[36970]: pgmap v1754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:32:29.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:28 smithi163 ceph-mon[36873]: pgmap v1754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:32:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:31 smithi163 ceph-mon[36873]: pgmap v1755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2181.155791 secs 2024-07-26T07:32:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2198.899161 secs 2024-07-26T07:32:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:31 smithi042 ceph-mon[32775]: pgmap v1755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2181.155791 secs 2024-07-26T07:32:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2198.899161 secs 2024-07-26T07:32:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:31 smithi164 ceph-mon[36970]: pgmap v1755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2181.155791 secs 2024-07-26T07:32:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2198.899161 secs 2024-07-26T07:32:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:33 smithi163 ceph-mon[36873]: pgmap v1756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:33 smithi042 ceph-mon[32775]: pgmap v1756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:33.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:33 smithi164 ceph-mon[36970]: pgmap v1756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:35.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:35 smithi163 ceph-mon[36873]: pgmap v1757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:35.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:35 smithi042 ceph-mon[32775]: pgmap v1757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:35.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:35 smithi164 ceph-mon[36970]: pgmap v1757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2186.155931 secs 2024-07-26T07:32:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2203.899278 secs 2024-07-26T07:32:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2186.155931 secs 2024-07-26T07:32:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2203.899278 secs 2024-07-26T07:32:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2186.155931 secs 2024-07-26T07:32:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2203.899278 secs 2024-07-26T07:32:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:37 smithi163 ceph-mon[36873]: pgmap v1758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:37 smithi042 ceph-mon[32775]: pgmap v1758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:37 smithi164 ceph-mon[36970]: pgmap v1758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:39.000 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:32:38.999+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:32:39.000 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:32:38.999+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:32:39.000 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:32:38.999+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:32:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:39 smithi163 ceph-mon[36873]: pgmap v1759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:32:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:39 smithi042 ceph-mon[32775]: pgmap v1759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:32:39.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:39 smithi164 ceph-mon[36970]: pgmap v1759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:32:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:41 smithi042 ceph-mon[32775]: pgmap v1760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2191.156071 secs 2024-07-26T07:32:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2208.899407 secs 2024-07-26T07:32:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:41 smithi164 ceph-mon[36970]: pgmap v1760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2191.156071 secs 2024-07-26T07:32:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2208.899407 secs 2024-07-26T07:32:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:41 smithi163 ceph-mon[36873]: pgmap v1760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2191.156071 secs 2024-07-26T07:32:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2208.899407 secs 2024-07-26T07:32:43.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:43 smithi042 ceph-mon[32775]: pgmap v1761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:43.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:43 smithi164 ceph-mon[36970]: pgmap v1761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:43 smithi163 ceph-mon[36873]: pgmap v1761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:44 smithi042 ceph-mon[32775]: pgmap v1762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:44 smithi164 ceph-mon[36970]: pgmap v1762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:45.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:44 smithi163 ceph-mon[36873]: pgmap v1762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2196.156195 secs 2024-07-26T07:32:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2196.156195 secs 2024-07-26T07:32:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2196.156195 secs 2024-07-26T07:32:47.373 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2213.899536 secs 2024-07-26T07:32:47.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:47 smithi164 ceph-mon[36970]: pgmap v1763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2213.899536 secs 2024-07-26T07:32:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:47 smithi042 ceph-mon[32775]: pgmap v1763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:47.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2213.899536 secs 2024-07-26T07:32:47.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:47 smithi163 ceph-mon[36873]: pgmap v1763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:32:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:49 smithi163 ceph-mon[36873]: pgmap v1764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:32:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:49 smithi042 ceph-mon[32775]: pgmap v1764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:32:49.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:49 smithi164 ceph-mon[36970]: pgmap v1764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:32:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:51 smithi163 ceph-mon[36873]: pgmap v1765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:32:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2201.156289 secs 2024-07-26T07:32:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:32:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:32:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:32:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2218.899657 secs 2024-07-26T07:32:51.823 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:51 smithi042 ceph-mon[32775]: pgmap v1765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:51.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:32:51.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2201.156289 secs 2024-07-26T07:32:51.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:32:51.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:32:51.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:32:51.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2218.899657 secs 2024-07-26T07:32:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:51 smithi164 ceph-mon[36970]: pgmap v1765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:32:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:32:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2201.156289 secs 2024-07-26T07:32:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:32:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:32:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:32:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2218.899657 secs 2024-07-26T07:32:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:52 smithi163 ceph-mon[36873]: pgmap v1766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:32:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:52 smithi163 ceph-mon[36873]: pgmap v1767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:32:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:52 smithi042 ceph-mon[32775]: pgmap v1766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:32:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:52 smithi042 ceph-mon[32775]: pgmap v1767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:32:52.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:52 smithi164 ceph-mon[36970]: pgmap v1766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:32:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:52 smithi164 ceph-mon[36970]: pgmap v1767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:32:54.003 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:32:54.002+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:32:54.003 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:32:54.002+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:32:54.003 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:32:54.002+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:32:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:54 smithi163 ceph-mon[36873]: pgmap v1768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:32:54.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:54 smithi042 ceph-mon[32775]: pgmap v1768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:32:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:54 smithi164 ceph-mon[36970]: pgmap v1768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:32:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:56 smithi042 ceph-mon[32775]: pgmap v1769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:32:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2206.156415 secs 2024-07-26T07:32:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2223.899778 secs 2024-07-26T07:32:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:56 smithi164 ceph-mon[36970]: pgmap v1769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:32:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2206.156415 secs 2024-07-26T07:32:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2223.899778 secs 2024-07-26T07:32:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:56 smithi163 ceph-mon[36873]: pgmap v1769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:32:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2206.156415 secs 2024-07-26T07:32:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2223.899778 secs 2024-07-26T07:32:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:58 smithi042 ceph-mon[32775]: pgmap v1770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:32:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:58 smithi164 ceph-mon[36970]: pgmap v1770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:32:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:58 smithi163 ceph-mon[36873]: pgmap v1770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:32:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:32:59 smithi042 ceph-mon[32775]: pgmap v1771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:32:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:32:59 smithi164 ceph-mon[36970]: pgmap v1771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:33:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:32:59 smithi163 ceph-mon[36873]: pgmap v1771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:33:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2211.156541 secs 2024-07-26T07:33:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2211.156541 secs 2024-07-26T07:33:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2211.156541 secs 2024-07-26T07:33:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:02 smithi042 ceph-mon[32775]: pgmap v1772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:33:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2228.899912 secs 2024-07-26T07:33:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:02 smithi164 ceph-mon[36970]: pgmap v1772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:33:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2228.899912 secs 2024-07-26T07:33:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:02 smithi163 ceph-mon[36873]: pgmap v1772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:33:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2228.899912 secs 2024-07-26T07:33:04.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:04 smithi042 ceph-mon[32775]: pgmap v1773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:04.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:04 smithi164 ceph-mon[36970]: pgmap v1773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:04.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:04 smithi163 ceph-mon[36873]: pgmap v1773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:06 smithi042 ceph-mon[32775]: pgmap v1774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2216.156647 secs 2024-07-26T07:33:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:06 smithi164 ceph-mon[36970]: pgmap v1774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2216.156647 secs 2024-07-26T07:33:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:06 smithi163 ceph-mon[36873]: pgmap v1774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2216.156647 secs 2024-07-26T07:33:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2233.900050 secs 2024-07-26T07:33:07.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2233.900050 secs 2024-07-26T07:33:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2233.900050 secs 2024-07-26T07:33:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:08 smithi042 ceph-mon[32775]: pgmap v1775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:08 smithi164 ceph-mon[36970]: pgmap v1775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:08 smithi163 ceph-mon[36873]: pgmap v1775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:33:09.003+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:33:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:33:09.003+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:33:09.004 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:33:09.003+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:33:10.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:10 smithi042 ceph-mon[32775]: pgmap v1776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:33:10.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:10 smithi164 ceph-mon[36970]: pgmap v1776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:33:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:10 smithi163 ceph-mon[36873]: pgmap v1776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:33:11.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2221.156775 secs 2024-07-26T07:33:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2221.156775 secs 2024-07-26T07:33:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2221.156775 secs 2024-07-26T07:33:12.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:12 smithi042 ceph-mon[32775]: pgmap v1777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2238.900149 secs 2024-07-26T07:33:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:12 smithi164 ceph-mon[36970]: pgmap v1777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2238.900149 secs 2024-07-26T07:33:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:12 smithi163 ceph-mon[36873]: pgmap v1777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2238.900149 secs 2024-07-26T07:33:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:14 smithi042 ceph-mon[32775]: pgmap v1778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:14 smithi164 ceph-mon[36970]: pgmap v1778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:14 smithi163 ceph-mon[36873]: pgmap v1778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:16.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:16 smithi042 ceph-mon[32775]: pgmap v1779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2226.156875 secs 2024-07-26T07:33:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:16 smithi164 ceph-mon[36970]: pgmap v1779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2226.156875 secs 2024-07-26T07:33:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:16 smithi163 ceph-mon[36873]: pgmap v1779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2226.156875 secs 2024-07-26T07:33:17.373 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2243.900245 secs 2024-07-26T07:33:17.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2243.900245 secs 2024-07-26T07:33:17.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2243.900245 secs 2024-07-26T07:33:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:18 smithi042 ceph-mon[32775]: pgmap v1780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:18.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:18 smithi164 ceph-mon[36970]: pgmap v1780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:18 smithi163 ceph-mon[36873]: pgmap v1780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:20 smithi042 ceph-mon[32775]: pgmap v1781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:33:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:20 smithi164 ceph-mon[36970]: pgmap v1781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:33:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:20 smithi163 ceph-mon[36873]: pgmap v1781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:33:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2231.157019 secs 2024-07-26T07:33:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2231.157019 secs 2024-07-26T07:33:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2231.157019 secs 2024-07-26T07:33:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:22 smithi042 ceph-mon[32775]: pgmap v1782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2248.900369 secs 2024-07-26T07:33:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:22 smithi164 ceph-mon[36970]: pgmap v1782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2248.900369 secs 2024-07-26T07:33:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:22 smithi163 ceph-mon[36873]: pgmap v1782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2248.900369 secs 2024-07-26T07:33:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:33:24.006+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:33:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:33:24.006+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:33:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:33:24.006+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:33:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:24 smithi042 ceph-mon[32775]: pgmap v1783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:24.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:24 smithi164 ceph-mon[36970]: pgmap v1783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:24 smithi163 ceph-mon[36873]: pgmap v1783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:26 smithi042 ceph-mon[32775]: pgmap v1784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2236.157159 secs 2024-07-26T07:33:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:26 smithi164 ceph-mon[36970]: pgmap v1784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2236.157159 secs 2024-07-26T07:33:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:26 smithi163 ceph-mon[36873]: pgmap v1784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2236.157159 secs 2024-07-26T07:33:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2253.900524 secs 2024-07-26T07:33:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2253.900524 secs 2024-07-26T07:33:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2253.900524 secs 2024-07-26T07:33:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:28 smithi042 ceph-mon[32775]: pgmap v1785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:28.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:28 smithi164 ceph-mon[36970]: pgmap v1785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:28 smithi163 ceph-mon[36873]: pgmap v1785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:30 smithi042 ceph-mon[32775]: pgmap v1786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:33:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:30 smithi164 ceph-mon[36970]: pgmap v1786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:33:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:30 smithi163 ceph-mon[36873]: pgmap v1786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:33:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2241.157285 secs 2024-07-26T07:33:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2241.157285 secs 2024-07-26T07:33:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2241.157285 secs 2024-07-26T07:33:32.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:32 smithi042 ceph-mon[32775]: pgmap v1787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2258.900651 secs 2024-07-26T07:33:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:32 smithi164 ceph-mon[36970]: pgmap v1787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2258.900651 secs 2024-07-26T07:33:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:32 smithi163 ceph-mon[36873]: pgmap v1787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2258.900651 secs 2024-07-26T07:33:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:34 smithi042 ceph-mon[32775]: pgmap v1788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:34.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:34 smithi164 ceph-mon[36970]: pgmap v1788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:34 smithi163 ceph-mon[36873]: pgmap v1788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:36.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:36 smithi042 ceph-mon[32775]: pgmap v1789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2246.157405 secs 2024-07-26T07:33:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:36 smithi164 ceph-mon[36970]: pgmap v1789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2246.157405 secs 2024-07-26T07:33:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:36 smithi163 ceph-mon[36873]: pgmap v1789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2246.157405 secs 2024-07-26T07:33:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2263.900774 secs 2024-07-26T07:33:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2263.900774 secs 2024-07-26T07:33:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2263.900774 secs 2024-07-26T07:33:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:38 smithi042 ceph-mon[32775]: pgmap v1790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:38 smithi164 ceph-mon[36970]: pgmap v1790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:38 smithi163 ceph-mon[36873]: pgmap v1790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:39.012 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:33:39.012+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:33:39.012 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:33:39.012+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:33:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:33:39.012+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:33:40.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:40 smithi164 ceph-mon[36970]: pgmap v1791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:40 smithi163 ceph-mon[36873]: pgmap v1791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:40 smithi042 ceph-mon[32775]: pgmap v1791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2251.157570 secs 2024-07-26T07:33:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2251.157570 secs 2024-07-26T07:33:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2251.157570 secs 2024-07-26T07:33:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:42 smithi164 ceph-mon[36970]: pgmap v1792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2268.900903 secs 2024-07-26T07:33:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:42 smithi163 ceph-mon[36873]: pgmap v1792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2268.900903 secs 2024-07-26T07:33:42.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:42 smithi042 ceph-mon[32775]: pgmap v1792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2268.900903 secs 2024-07-26T07:33:44.447 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:44 smithi042 ceph-mon[32775]: pgmap v1793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:44.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:44 smithi164 ceph-mon[36970]: pgmap v1793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:44 smithi163 ceph-mon[36873]: pgmap v1793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:46 smithi164 ceph-mon[36970]: pgmap v1794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2256.157637 secs 2024-07-26T07:33:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:46 smithi163 ceph-mon[36873]: pgmap v1794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2256.157637 secs 2024-07-26T07:33:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:46 smithi042 ceph-mon[32775]: pgmap v1794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2256.157637 secs 2024-07-26T07:33:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2273.901023 secs 2024-07-26T07:33:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2273.901023 secs 2024-07-26T07:33:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2273.901023 secs 2024-07-26T07:33:48.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:48 smithi164 ceph-mon[36970]: pgmap v1795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:48 smithi163 ceph-mon[36873]: pgmap v1795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:48.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:48 smithi042 ceph-mon[32775]: pgmap v1795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:50.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:50 smithi164 ceph-mon[36970]: pgmap v1796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:50 smithi163 ceph-mon[36873]: pgmap v1796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:50 smithi042 ceph-mon[32775]: pgmap v1796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:33:51.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2261.157767 secs 2024-07-26T07:33:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:33:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2261.157767 secs 2024-07-26T07:33:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:33:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2261.157767 secs 2024-07-26T07:33:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:33:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:52 smithi164 ceph-mon[36970]: pgmap v1797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2278.901146 secs 2024-07-26T07:33:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:33:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:33:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:33:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:52 smithi163 ceph-mon[36873]: pgmap v1797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2278.901146 secs 2024-07-26T07:33:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:33:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:33:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:33:52.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:52 smithi042 ceph-mon[32775]: pgmap v1797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:33:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2278.901146 secs 2024-07-26T07:33:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:33:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:33:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:33:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:53 smithi164 ceph-mon[36970]: pgmap v1798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:33:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:53 smithi164 ceph-mon[36970]: pgmap v1799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:33:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:53 smithi163 ceph-mon[36873]: pgmap v1798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:33:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:53 smithi163 ceph-mon[36873]: pgmap v1799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:33:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:53 smithi042 ceph-mon[32775]: pgmap v1798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:33:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:53 smithi042 ceph-mon[32775]: pgmap v1799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:33:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:33:54.015+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:33:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:33:54.015+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:33:54.016 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:33:54.015+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:33:55.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:55 smithi164 ceph-mon[36970]: pgmap v1800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:33:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:55 smithi163 ceph-mon[36873]: pgmap v1800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:33:55.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:55 smithi042 ceph-mon[32775]: pgmap v1800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:33:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2266.157895 secs 2024-07-26T07:33:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2266.157895 secs 2024-07-26T07:33:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2266.157895 secs 2024-07-26T07:33:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:57 smithi164 ceph-mon[36970]: pgmap v1801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:33:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2283.901298 secs 2024-07-26T07:33:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:57 smithi163 ceph-mon[36873]: pgmap v1801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:33:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2283.901298 secs 2024-07-26T07:33:57.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:57 smithi042 ceph-mon[32775]: pgmap v1801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:33:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2283.901298 secs 2024-07-26T07:33:59.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:33:59 smithi164 ceph-mon[36970]: pgmap v1802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:33:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:33:59 smithi163 ceph-mon[36873]: pgmap v1802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:33:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:33:59 smithi042 ceph-mon[32775]: pgmap v1802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:34:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:01 smithi164 ceph-mon[36970]: pgmap v1803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:34:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2271.158017 secs 2024-07-26T07:34:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:01 smithi163 ceph-mon[36873]: pgmap v1803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:34:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2271.158017 secs 2024-07-26T07:34:01.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:01 smithi042 ceph-mon[32775]: pgmap v1803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:34:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2271.158017 secs 2024-07-26T07:34:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2288.901418 secs 2024-07-26T07:34:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2288.901418 secs 2024-07-26T07:34:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2288.901418 secs 2024-07-26T07:34:03.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:03 smithi164 ceph-mon[36970]: pgmap v1804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:34:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:03 smithi163 ceph-mon[36873]: pgmap v1804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:34:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:03 smithi042 ceph-mon[32775]: pgmap v1804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:34:05.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:05 smithi164 ceph-mon[36970]: pgmap v1805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:34:05.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:05 smithi163 ceph-mon[36873]: pgmap v1805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:34:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:05 smithi042 ceph-mon[32775]: pgmap v1805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:34:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2276.158137 secs 2024-07-26T07:34:06.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2276.158137 secs 2024-07-26T07:34:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2276.158137 secs 2024-07-26T07:34:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:07 smithi163 ceph-mon[36873]: pgmap v1806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2293.901544 secs 2024-07-26T07:34:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:07 smithi042 ceph-mon[32775]: pgmap v1806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2293.901544 secs 2024-07-26T07:34:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:07 smithi164 ceph-mon[36970]: pgmap v1806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2293.901544 secs 2024-07-26T07:34:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:34:09.023+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:34:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:34:09.023+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:34:09.024 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:34:09.023+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:34:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:09 smithi042 ceph-mon[32775]: pgmap v1807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:09 smithi163 ceph-mon[36873]: pgmap v1807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:09 smithi164 ceph-mon[36970]: pgmap v1807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:11 smithi163 ceph-mon[36873]: pgmap v1808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2281.158261 secs 2024-07-26T07:34:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:11 smithi042 ceph-mon[32775]: pgmap v1808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2281.158261 secs 2024-07-26T07:34:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:11 smithi164 ceph-mon[36970]: pgmap v1808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2281.158261 secs 2024-07-26T07:34:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2298.901637 secs 2024-07-26T07:34:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2298.901637 secs 2024-07-26T07:34:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2298.901637 secs 2024-07-26T07:34:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:13 smithi163 ceph-mon[36873]: pgmap v1809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:13.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:13 smithi042 ceph-mon[32775]: pgmap v1809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:13.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:13 smithi164 ceph-mon[36970]: pgmap v1809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:15 smithi163 ceph-mon[36873]: pgmap v1810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:34:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:15 smithi042 ceph-mon[32775]: pgmap v1810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:34:15.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:15 smithi164 ceph-mon[36970]: pgmap v1810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:34:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2286.158384 secs 2024-07-26T07:34:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2286.158384 secs 2024-07-26T07:34:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2286.158384 secs 2024-07-26T07:34:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:17 smithi163 ceph-mon[36873]: pgmap v1811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2303.901757 secs 2024-07-26T07:34:17.623 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:17 smithi164 ceph-mon[36970]: pgmap v1811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:17.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2303.901757 secs 2024-07-26T07:34:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:17 smithi042 ceph-mon[32775]: pgmap v1811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2303.901757 secs 2024-07-26T07:34:19.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:19 smithi042 ceph-mon[32775]: pgmap v1812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:19.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:19 smithi164 ceph-mon[36970]: pgmap v1812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:19 smithi163 ceph-mon[36873]: pgmap v1812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:21 smithi042 ceph-mon[32775]: pgmap v1813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2291.158507 secs 2024-07-26T07:34:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:21 smithi164 ceph-mon[36970]: pgmap v1813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2291.158507 secs 2024-07-26T07:34:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:21 smithi163 ceph-mon[36873]: pgmap v1813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2291.158507 secs 2024-07-26T07:34:22.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2308.901872 secs 2024-07-26T07:34:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2308.901872 secs 2024-07-26T07:34:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2308.901872 secs 2024-07-26T07:34:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:23 smithi042 ceph-mon[32775]: pgmap v1814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:23.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:23 smithi164 ceph-mon[36970]: pgmap v1814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:23 smithi163 ceph-mon[36873]: pgmap v1814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:34:24.026+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:34:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:34:24.026+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:34:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:34:24.026+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:34:25.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:25 smithi042 ceph-mon[32775]: pgmap v1815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:34:25.818 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:25 smithi163 ceph-mon[36873]: pgmap v1815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:34:25.818 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:25 smithi164 ceph-mon[36970]: pgmap v1815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:34:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2296.158589 secs 2024-07-26T07:34:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2296.158589 secs 2024-07-26T07:34:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2296.158589 secs 2024-07-26T07:34:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:27 smithi042 ceph-mon[32775]: pgmap v1816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2313.901961 secs 2024-07-26T07:34:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:27 smithi164 ceph-mon[36970]: pgmap v1816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2313.901961 secs 2024-07-26T07:34:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:27 smithi163 ceph-mon[36873]: pgmap v1816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2313.901961 secs 2024-07-26T07:34:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:29 smithi042 ceph-mon[32775]: pgmap v1817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:29.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:29 smithi164 ceph-mon[36970]: pgmap v1817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:29 smithi163 ceph-mon[36873]: pgmap v1817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:31 smithi042 ceph-mon[32775]: pgmap v1818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2301.158664 secs 2024-07-26T07:34:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:31 smithi164 ceph-mon[36970]: pgmap v1818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2301.158664 secs 2024-07-26T07:34:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:31 smithi163 ceph-mon[36873]: pgmap v1818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2301.158664 secs 2024-07-26T07:34:32.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2318.902061 secs 2024-07-26T07:34:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2318.902061 secs 2024-07-26T07:34:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2318.902061 secs 2024-07-26T07:34:33.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:33 smithi042 ceph-mon[32775]: pgmap v1819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:33.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:33 smithi164 ceph-mon[36970]: pgmap v1819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:33 smithi163 ceph-mon[36873]: pgmap v1819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:35.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:35 smithi042 ceph-mon[32775]: pgmap v1820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:34:35.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:35 smithi164 ceph-mon[36970]: pgmap v1820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:34:35.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:35 smithi163 ceph-mon[36873]: pgmap v1820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:34:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2306.158805 secs 2024-07-26T07:34:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2306.158805 secs 2024-07-26T07:34:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2306.158805 secs 2024-07-26T07:34:37.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:37 smithi042 ceph-mon[32775]: pgmap v1821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2323.902161 secs 2024-07-26T07:34:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:37 smithi164 ceph-mon[36970]: pgmap v1821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2323.902161 secs 2024-07-26T07:34:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:37 smithi163 ceph-mon[36873]: pgmap v1821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2323.902161 secs 2024-07-26T07:34:39.035 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:34:39.034+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:34:39.036 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:34:39.035+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:34:39.036 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:34:39.035+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:34:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:39 smithi042 ceph-mon[32775]: pgmap v1822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:39 smithi164 ceph-mon[36970]: pgmap v1822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:39 smithi163 ceph-mon[36873]: pgmap v1822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:41 smithi042 ceph-mon[32775]: pgmap v1823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2311.158929 secs 2024-07-26T07:34:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:41 smithi164 ceph-mon[36970]: pgmap v1823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2311.158929 secs 2024-07-26T07:34:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:41 smithi163 ceph-mon[36873]: pgmap v1823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2311.158929 secs 2024-07-26T07:34:42.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2328.902283 secs 2024-07-26T07:34:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2328.902283 secs 2024-07-26T07:34:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2328.902283 secs 2024-07-26T07:34:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:43 smithi042 ceph-mon[32775]: pgmap v1824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:43 smithi164 ceph-mon[36970]: pgmap v1824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:43 smithi163 ceph-mon[36873]: pgmap v1824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:45.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:45 smithi042 ceph-mon[32775]: pgmap v1825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:34:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:45 smithi164 ceph-mon[36970]: pgmap v1825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:34:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:45 smithi163 ceph-mon[36873]: pgmap v1825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:34:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2316.159055 secs 2024-07-26T07:34:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2316.159055 secs 2024-07-26T07:34:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2316.159055 secs 2024-07-26T07:34:47.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:47 smithi164 ceph-mon[36970]: pgmap v1826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:47.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2333.902407 secs 2024-07-26T07:34:47.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:47 smithi042 ceph-mon[32775]: pgmap v1826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2333.902407 secs 2024-07-26T07:34:47.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:47 smithi163 ceph-mon[36873]: pgmap v1826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:47.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2333.902407 secs 2024-07-26T07:34:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:49 smithi042 ceph-mon[32775]: pgmap v1827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:49 smithi164 ceph-mon[36970]: pgmap v1827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:49 smithi163 ceph-mon[36873]: pgmap v1827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:51 smithi042 ceph-mon[32775]: pgmap v1828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2321.159181 secs 2024-07-26T07:34:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:51 smithi164 ceph-mon[36970]: pgmap v1828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2321.159181 secs 2024-07-26T07:34:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:51 smithi163 ceph-mon[36873]: pgmap v1828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:34:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2321.159181 secs 2024-07-26T07:34:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2338.902531 secs 2024-07-26T07:34:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:34:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:34:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:34:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:34:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2338.902531 secs 2024-07-26T07:34:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:34:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:34:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:34:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:34:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2338.902531 secs 2024-07-26T07:34:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:34:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:34:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:34:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:34:53.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:53 smithi042 ceph-mon[32775]: pgmap v1829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:53 smithi042 ceph-mon[32775]: pgmap v1830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:34:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:53 smithi042 ceph-mon[32775]: pgmap v1831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:34:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:53 smithi164 ceph-mon[36970]: pgmap v1829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:53 smithi164 ceph-mon[36970]: pgmap v1830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:34:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:53 smithi164 ceph-mon[36970]: pgmap v1831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:34:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:53 smithi163 ceph-mon[36873]: pgmap v1829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:34:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:53 smithi163 ceph-mon[36873]: pgmap v1830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:34:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:53 smithi163 ceph-mon[36873]: pgmap v1831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:34:54.038 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:34:54.036+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:34:54.038 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:34:54.037+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:34:54.038 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:34:54.037+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:34:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:55 smithi042 ceph-mon[32775]: pgmap v1832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:34:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:55 smithi164 ceph-mon[36970]: pgmap v1832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:34:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:55 smithi163 ceph-mon[36873]: pgmap v1832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:34:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2326.159328 secs 2024-07-26T07:34:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2326.159328 secs 2024-07-26T07:34:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2326.159328 secs 2024-07-26T07:34:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2343.902658 secs 2024-07-26T07:34:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:57 smithi042 ceph-mon[32775]: pgmap v1833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:34:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2343.902658 secs 2024-07-26T07:34:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:57 smithi164 ceph-mon[36970]: pgmap v1833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:34:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2343.902658 secs 2024-07-26T07:34:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:57 smithi163 ceph-mon[36873]: pgmap v1833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:34:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:34:58.222+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:34:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:34:58.222+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:34:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:34:58.223+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:34:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:34:58.223+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:34:59.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:34:59 smithi042 ceph-mon[32775]: pgmap v1834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:34:59.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:34:59 smithi164 ceph-mon[36970]: pgmap v1834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:34:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:34:59 smithi163 ceph-mon[36873]: pgmap v1834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:35:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:01 smithi042 ceph-mon[32775]: pgmap v1835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:35:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2331.159477 secs 2024-07-26T07:35:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:01 smithi164 ceph-mon[36970]: pgmap v1835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:35:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2331.159477 secs 2024-07-26T07:35:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:01 smithi163 ceph-mon[36873]: pgmap v1835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:35:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2331.159477 secs 2024-07-26T07:35:02.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2348.902733 secs 2024-07-26T07:35:02.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2348.902733 secs 2024-07-26T07:35:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2348.902733 secs 2024-07-26T07:35:03.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:03 smithi042 ceph-mon[32775]: pgmap v1836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:35:03.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:03 smithi164 ceph-mon[36970]: pgmap v1836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:35:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:03 smithi163 ceph-mon[36873]: pgmap v1836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:35:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:05 smithi042 ceph-mon[32775]: pgmap v1837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:35:05.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:05 smithi164 ceph-mon[36970]: pgmap v1837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:35:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:05 smithi163 ceph-mon[36873]: pgmap v1837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:35:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2336.159607 secs 2024-07-26T07:35:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2336.159607 secs 2024-07-26T07:35:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2336.159607 secs 2024-07-26T07:35:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2353.902879 secs 2024-07-26T07:35:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:07 smithi042 ceph-mon[32775]: pgmap v1838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:35:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2353.902879 secs 2024-07-26T07:35:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:07 smithi164 ceph-mon[36970]: pgmap v1838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:35:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2353.902879 secs 2024-07-26T07:35:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:07 smithi163 ceph-mon[36873]: pgmap v1838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:35:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:35:09.040+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:35:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:35:09.040+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:35:09.041 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:35:09.040+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:35:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:09 smithi042 ceph-mon[32775]: pgmap v1839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:35:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:09 smithi164 ceph-mon[36970]: pgmap v1839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:35:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:09 smithi163 ceph-mon[36873]: pgmap v1839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:35:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:11 smithi042 ceph-mon[32775]: pgmap v1840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:35:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2341.159752 secs 2024-07-26T07:35:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:11 smithi164 ceph-mon[36970]: pgmap v1840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:35:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2341.159752 secs 2024-07-26T07:35:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:11 smithi163 ceph-mon[36873]: pgmap v1840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:35:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2341.159752 secs 2024-07-26T07:35:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2358.903022 secs 2024-07-26T07:35:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2358.903022 secs 2024-07-26T07:35:12.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2358.903022 secs 2024-07-26T07:35:13.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:13 smithi164 ceph-mon[36970]: pgmap v1841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:35:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:13 smithi163 ceph-mon[36873]: pgmap v1841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:35:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:13 smithi042 ceph-mon[32775]: pgmap v1841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:35:15.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:15 smithi042 ceph-mon[32775]: pgmap v1842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:35:15.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:15 smithi164 ceph-mon[36970]: pgmap v1842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:35:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:15 smithi163 ceph-mon[36873]: pgmap v1842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:35:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2346.159856 secs 2024-07-26T07:35:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2363.903145 secs 2024-07-26T07:35:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2346.159856 secs 2024-07-26T07:35:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2363.903145 secs 2024-07-26T07:35:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2346.159856 secs 2024-07-26T07:35:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2363.903145 secs 2024-07-26T07:35:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:17 smithi163 ceph-mon[36873]: pgmap v1843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:17 smithi164 ceph-mon[36970]: pgmap v1843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:17.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:17 smithi042 ceph-mon[32775]: pgmap v1843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:19.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:19 smithi164 ceph-mon[36970]: pgmap v1844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:19 smithi163 ceph-mon[36873]: pgmap v1844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:19.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:19 smithi042 ceph-mon[32775]: pgmap v1844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:21 smithi164 ceph-mon[36970]: pgmap v1845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2351.160003 secs 2024-07-26T07:35:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2368.903221 secs 2024-07-26T07:35:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:21 smithi163 ceph-mon[36873]: pgmap v1845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2351.160003 secs 2024-07-26T07:35:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2368.903221 secs 2024-07-26T07:35:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:21 smithi042 ceph-mon[32775]: pgmap v1845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2351.160003 secs 2024-07-26T07:35:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2368.903221 secs 2024-07-26T07:35:23.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:23 smithi164 ceph-mon[36970]: pgmap v1846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:23 smithi163 ceph-mon[36873]: pgmap v1846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:23 smithi042 ceph-mon[32775]: pgmap v1846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:24.044 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:35:24.043+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:35:24.044 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:35:24.043+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:35:24.044 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:35:24.043+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:35:25.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:25 smithi164 ceph-mon[36970]: pgmap v1847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:25.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:25 smithi163 ceph-mon[36873]: pgmap v1847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:25 smithi042 ceph-mon[32775]: pgmap v1847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2356.160145 secs 2024-07-26T07:35:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2373.903344 secs 2024-07-26T07:35:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2356.160145 secs 2024-07-26T07:35:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2373.903344 secs 2024-07-26T07:35:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2356.160145 secs 2024-07-26T07:35:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2373.903344 secs 2024-07-26T07:35:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:27 smithi164 ceph-mon[36970]: pgmap v1848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:27 smithi163 ceph-mon[36873]: pgmap v1848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:27 smithi042 ceph-mon[32775]: pgmap v1848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:29.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:29 smithi164 ceph-mon[36970]: pgmap v1849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:29 smithi163 ceph-mon[36873]: pgmap v1849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:29.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:29 smithi042 ceph-mon[32775]: pgmap v1849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:31 smithi164 ceph-mon[36970]: pgmap v1850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2361.160270 secs 2024-07-26T07:35:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2378.903466 secs 2024-07-26T07:35:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:31 smithi163 ceph-mon[36873]: pgmap v1850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2361.160270 secs 2024-07-26T07:35:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2378.903466 secs 2024-07-26T07:35:31.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:31 smithi042 ceph-mon[32775]: pgmap v1850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2361.160270 secs 2024-07-26T07:35:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2378.903466 secs 2024-07-26T07:35:33.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:33 smithi164 ceph-mon[36970]: pgmap v1851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:33 smithi163 ceph-mon[36873]: pgmap v1851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:33 smithi042 ceph-mon[32775]: pgmap v1851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:35.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:35 smithi164 ceph-mon[36970]: pgmap v1852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:35.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:35 smithi163 ceph-mon[36873]: pgmap v1852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:35.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:35 smithi042 ceph-mon[32775]: pgmap v1852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2366.160386 secs 2024-07-26T07:35:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2383.903587 secs 2024-07-26T07:35:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2366.160386 secs 2024-07-26T07:35:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2383.903587 secs 2024-07-26T07:35:36.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2366.160386 secs 2024-07-26T07:35:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2383.903587 secs 2024-07-26T07:35:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:37 smithi164 ceph-mon[36970]: pgmap v1853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:37 smithi163 ceph-mon[36873]: pgmap v1853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:37 smithi042 ceph-mon[32775]: pgmap v1853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:35:39.051+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:35:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:35:39.051+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:35:39.052 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:35:39.051+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:35:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:39 smithi164 ceph-mon[36970]: pgmap v1854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:39 smithi163 ceph-mon[36873]: pgmap v1854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:39.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:39 smithi042 ceph-mon[32775]: pgmap v1854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:41 smithi164 ceph-mon[36970]: pgmap v1855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2371.160452 secs 2024-07-26T07:35:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2388.903713 secs 2024-07-26T07:35:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:41 smithi163 ceph-mon[36873]: pgmap v1855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2371.160452 secs 2024-07-26T07:35:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2388.903713 secs 2024-07-26T07:35:41.808 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:41 smithi042 ceph-mon[32775]: pgmap v1855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:41.808 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2371.160452 secs 2024-07-26T07:35:41.808 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2388.903713 secs 2024-07-26T07:35:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:43 smithi164 ceph-mon[36970]: pgmap v1856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:43 smithi163 ceph-mon[36873]: pgmap v1856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:43.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:43 smithi042 ceph-mon[32775]: pgmap v1856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:45 smithi164 ceph-mon[36970]: pgmap v1857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:45 smithi163 ceph-mon[36873]: pgmap v1857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:45.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:45 smithi042 ceph-mon[32775]: pgmap v1857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2376.160589 secs 2024-07-26T07:35:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2393.903837 secs 2024-07-26T07:35:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2376.160589 secs 2024-07-26T07:35:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2393.903837 secs 2024-07-26T07:35:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2376.160589 secs 2024-07-26T07:35:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2393.903837 secs 2024-07-26T07:35:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:47 smithi163 ceph-mon[36873]: pgmap v1858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:47.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:47 smithi042 ceph-mon[32775]: pgmap v1858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:47.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:47 smithi164 ceph-mon[36970]: pgmap v1858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:49 smithi163 ceph-mon[36873]: pgmap v1859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:49 smithi042 ceph-mon[32775]: pgmap v1859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:49.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:49 smithi164 ceph-mon[36970]: pgmap v1859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:51 smithi163 ceph-mon[36873]: pgmap v1860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2381.160714 secs 2024-07-26T07:35:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2398.903963 secs 2024-07-26T07:35:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:51 smithi042 ceph-mon[32775]: pgmap v1860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2381.160714 secs 2024-07-26T07:35:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2398.903963 secs 2024-07-26T07:35:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:51 smithi164 ceph-mon[36970]: pgmap v1860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:35:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2381.160714 secs 2024-07-26T07:35:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2398.903963 secs 2024-07-26T07:35:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:35:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:35:52.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:35:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:53 smithi163 ceph-mon[36873]: pgmap v1861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:35:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:35:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:35:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:53 smithi042 ceph-mon[32775]: pgmap v1861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:35:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:35:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:35:53.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:53 smithi164 ceph-mon[36970]: pgmap v1861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:35:53.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:35:53.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:35:53.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:35:54.056 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:35:54.055+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:35:54.056 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:35:54.055+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:35:54.056 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:35:54.055+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:35:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:54 smithi163 ceph-mon[36873]: pgmap v1862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:35:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:54 smithi163 ceph-mon[36873]: pgmap v1863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:35:54.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:54 smithi042 ceph-mon[32775]: pgmap v1862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:35:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:54 smithi042 ceph-mon[32775]: pgmap v1863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:35:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:54 smithi164 ceph-mon[36970]: pgmap v1862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:35:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:54 smithi164 ceph-mon[36970]: pgmap v1863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:35:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:56 smithi163 ceph-mon[36873]: pgmap v1864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:35:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2386.160829 secs 2024-07-26T07:35:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2403.904087 secs 2024-07-26T07:35:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:56 smithi042 ceph-mon[32775]: pgmap v1864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:35:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2386.160829 secs 2024-07-26T07:35:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2403.904087 secs 2024-07-26T07:35:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:56 smithi164 ceph-mon[36970]: pgmap v1864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:35:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2386.160829 secs 2024-07-26T07:35:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2403.904087 secs 2024-07-26T07:35:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:35:58 smithi163 ceph-mon[36873]: pgmap v1865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:35:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:35:58 smithi042 ceph-mon[32775]: pgmap v1865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:35:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:35:58 smithi164 ceph-mon[36970]: pgmap v1865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:36:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:00 smithi163 ceph-mon[36873]: pgmap v1866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:36:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:00 smithi042 ceph-mon[32775]: pgmap v1866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:36:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:00 smithi164 ceph-mon[36970]: pgmap v1866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:36:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2391.160949 secs 2024-07-26T07:36:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2408.904170 secs 2024-07-26T07:36:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2391.160949 secs 2024-07-26T07:36:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2408.904170 secs 2024-07-26T07:36:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2391.160949 secs 2024-07-26T07:36:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2408.904170 secs 2024-07-26T07:36:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:02 smithi042 ceph-mon[32775]: pgmap v1867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:36:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:02 smithi164 ceph-mon[36970]: pgmap v1867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:36:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:02 smithi163 ceph-mon[36873]: pgmap v1867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:36:04.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:04 smithi042 ceph-mon[32775]: pgmap v1868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:36:04.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:04 smithi164 ceph-mon[36970]: pgmap v1868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:36:05.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:04 smithi163 ceph-mon[36873]: pgmap v1868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:36:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:06 smithi042 ceph-mon[32775]: pgmap v1869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2396.161068 secs 2024-07-26T07:36:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2413.904269 secs 2024-07-26T07:36:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:06 smithi164 ceph-mon[36970]: pgmap v1869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2396.161068 secs 2024-07-26T07:36:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2413.904269 secs 2024-07-26T07:36:07.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:06 smithi163 ceph-mon[36873]: pgmap v1869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:07.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2396.161068 secs 2024-07-26T07:36:07.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2413.904269 secs 2024-07-26T07:36:08.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:08 smithi042 ceph-mon[32775]: pgmap v1870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:08.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:08 smithi164 ceph-mon[36970]: pgmap v1870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:09.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:08 smithi163 ceph-mon[36873]: pgmap v1870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:09.061 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:36:09.060+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:36:09.062 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:36:09.060+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:36:09.062 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:36:09.060+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:36:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:10 smithi042 ceph-mon[32775]: pgmap v1871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:36:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:10 smithi164 ceph-mon[36970]: pgmap v1871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:36:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:10 smithi163 ceph-mon[36873]: pgmap v1871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:36:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2401.161175 secs 2024-07-26T07:36:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2418.904412 secs 2024-07-26T07:36:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2401.161175 secs 2024-07-26T07:36:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2418.904412 secs 2024-07-26T07:36:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2401.161175 secs 2024-07-26T07:36:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2418.904412 secs 2024-07-26T07:36:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:12 smithi042 ceph-mon[32775]: pgmap v1872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:36:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:12 smithi164 ceph-mon[36970]: pgmap v1872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:36:13.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:12 smithi163 ceph-mon[36873]: pgmap v1872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:36:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:14 smithi042 ceph-mon[32775]: pgmap v1873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:36:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:14 smithi164 ceph-mon[36970]: pgmap v1873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:36:15.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:14 smithi163 ceph-mon[36873]: pgmap v1873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:36:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:16 smithi042 ceph-mon[32775]: pgmap v1874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:36:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2406.161301 secs 2024-07-26T07:36:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2423.904512 secs 2024-07-26T07:36:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:16 smithi164 ceph-mon[36970]: pgmap v1874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:36:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2406.161301 secs 2024-07-26T07:36:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2423.904512 secs 2024-07-26T07:36:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:16 smithi163 ceph-mon[36873]: pgmap v1874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:36:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2406.161301 secs 2024-07-26T07:36:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2423.904512 secs 2024-07-26T07:36:18.809 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:18 smithi163 ceph-mon[36873]: pgmap v1875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:36:18.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:18 smithi042 ceph-mon[32775]: pgmap v1875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:36:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:18 smithi164 ceph-mon[36970]: pgmap v1875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s 2024-07-26T07:36:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:20 smithi042 ceph-mon[32775]: pgmap v1876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-07-26T07:36:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:20 smithi164 ceph-mon[36970]: pgmap v1876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-07-26T07:36:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:20 smithi163 ceph-mon[36873]: pgmap v1876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s 2024-07-26T07:36:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2411.161426 secs 2024-07-26T07:36:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2428.904670 secs 2024-07-26T07:36:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2411.161426 secs 2024-07-26T07:36:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2428.904670 secs 2024-07-26T07:36:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2411.161426 secs 2024-07-26T07:36:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2428.904670 secs 2024-07-26T07:36:22.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:22 smithi042 ceph-mon[32775]: pgmap v1877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:36:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:22 smithi164 ceph-mon[36970]: pgmap v1877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:36:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:22 smithi163 ceph-mon[36873]: pgmap v1877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:36:23.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:23 smithi042 ceph-mon[32775]: pgmap v1878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:23 smithi164 ceph-mon[36970]: pgmap v1878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:24.006 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:23 smithi163 ceph-mon[36873]: pgmap v1878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:24.062 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:36:24.062+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:36:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:36:24.062+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:36:24.063 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:36:24.062+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:36:25.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:25 smithi164 ceph-mon[36970]: pgmap v1879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:25 smithi163 ceph-mon[36873]: pgmap v1879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:25 smithi042 ceph-mon[32775]: pgmap v1879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2416.161549 secs 2024-07-26T07:36:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2433.904790 secs 2024-07-26T07:36:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2416.161549 secs 2024-07-26T07:36:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2433.904790 secs 2024-07-26T07:36:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2416.161549 secs 2024-07-26T07:36:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2433.904790 secs 2024-07-26T07:36:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:27 smithi164 ceph-mon[36970]: pgmap v1880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:27 smithi163 ceph-mon[36873]: pgmap v1880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:27 smithi042 ceph-mon[32775]: pgmap v1880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:29.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:29 smithi164 ceph-mon[36970]: pgmap v1881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:36:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:29 smithi163 ceph-mon[36873]: pgmap v1881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:36:30.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:29 smithi042 ceph-mon[32775]: pgmap v1881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:36:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:31 smithi164 ceph-mon[36970]: pgmap v1882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2421.161672 secs 2024-07-26T07:36:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2438.904906 secs 2024-07-26T07:36:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:31 smithi163 ceph-mon[36873]: pgmap v1882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2421.161672 secs 2024-07-26T07:36:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2438.904906 secs 2024-07-26T07:36:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:31 smithi042 ceph-mon[32775]: pgmap v1882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2421.161672 secs 2024-07-26T07:36:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2438.904906 secs 2024-07-26T07:36:34.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:33 smithi163 ceph-mon[36873]: pgmap v1883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:34.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:33 smithi042 ceph-mon[32775]: pgmap v1883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:33 smithi164 ceph-mon[36970]: pgmap v1883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:36.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:35 smithi163 ceph-mon[36873]: pgmap v1884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:35 smithi042 ceph-mon[32775]: pgmap v1884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:36.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:35 smithi164 ceph-mon[36970]: pgmap v1884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2426.161795 secs 2024-07-26T07:36:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2443.905007 secs 2024-07-26T07:36:37.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2426.161795 secs 2024-07-26T07:36:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2443.905007 secs 2024-07-26T07:36:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2426.161795 secs 2024-07-26T07:36:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2443.905007 secs 2024-07-26T07:36:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:37 smithi163 ceph-mon[36873]: pgmap v1885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:37 smithi042 ceph-mon[32775]: pgmap v1885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:38.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:37 smithi164 ceph-mon[36970]: pgmap v1885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:36:39.069+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:36:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:36:39.069+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:36:39.070 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:36:39.069+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:36:40.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:39 smithi163 ceph-mon[36873]: pgmap v1886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:36:40.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:39 smithi042 ceph-mon[32775]: pgmap v1886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:36:40.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:39 smithi164 ceph-mon[36970]: pgmap v1886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:36:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:41 smithi163 ceph-mon[36873]: pgmap v1887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2431.161936 secs 2024-07-26T07:36:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2448.905129 secs 2024-07-26T07:36:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:41 smithi042 ceph-mon[32775]: pgmap v1887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2431.161936 secs 2024-07-26T07:36:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2448.905129 secs 2024-07-26T07:36:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:41 smithi164 ceph-mon[36970]: pgmap v1887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2431.161936 secs 2024-07-26T07:36:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2448.905129 secs 2024-07-26T07:36:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:43 smithi163 ceph-mon[36873]: pgmap v1888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:43 smithi042 ceph-mon[32775]: pgmap v1888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:44.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:43 smithi164 ceph-mon[36970]: pgmap v1888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:45 smithi163 ceph-mon[36873]: pgmap v1889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:46.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:45 smithi042 ceph-mon[32775]: pgmap v1889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:46.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:45 smithi164 ceph-mon[36970]: pgmap v1889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2436.162052 secs 2024-07-26T07:36:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2453.905245 secs 2024-07-26T07:36:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2436.162052 secs 2024-07-26T07:36:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2453.905245 secs 2024-07-26T07:36:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2436.162052 secs 2024-07-26T07:36:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2453.905245 secs 2024-07-26T07:36:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:47 smithi163 ceph-mon[36873]: pgmap v1890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:48.123 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:47 smithi164 ceph-mon[36970]: pgmap v1890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:48.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:47 smithi042 ceph-mon[32775]: pgmap v1890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:50.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:49 smithi042 ceph-mon[32775]: pgmap v1891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:36:50.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:49 smithi164 ceph-mon[36970]: pgmap v1891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:36:50.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:49 smithi163 ceph-mon[36873]: pgmap v1891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:36:52.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:51 smithi042 ceph-mon[32775]: pgmap v1892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2441.162175 secs 2024-07-26T07:36:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2458.905342 secs 2024-07-26T07:36:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:51 smithi164 ceph-mon[36970]: pgmap v1892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2441.162175 secs 2024-07-26T07:36:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2458.905342 secs 2024-07-26T07:36:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:51 smithi163 ceph-mon[36873]: pgmap v1892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:36:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2441.162175 secs 2024-07-26T07:36:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2458.905342 secs 2024-07-26T07:36:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:36:53.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:36:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:36:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:36:54.072+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:36:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:36:54.072+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:36:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:36:54.072+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:36:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: pgmap v1893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:36:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:36:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:36:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:36:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:36:54.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:36:54.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:36:54.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:36:54.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: pgmap v1894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:36:54.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: pgmap v1895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:36:54.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:36:54.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: pgmap v1893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:54.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:36:54.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:36:54.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:36:54.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:36:54.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:36:54.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:36:54.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:36:54.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:36:54.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: pgmap v1894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:36:54.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: pgmap v1895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:36:54.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:36:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: pgmap v1893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:36:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:36:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:36:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:36:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:36:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:36:54.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:36:54.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:36:54.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:36:54.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:36:54.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: pgmap v1894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:36:54.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: pgmap v1895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:36:54.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:36:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:56 smithi042 ceph-mon[32775]: pgmap v1896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:36:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2446.162328 secs 2024-07-26T07:36:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:56 smithi164 ceph-mon[36970]: pgmap v1896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:36:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2446.162328 secs 2024-07-26T07:36:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:56 smithi163 ceph-mon[36873]: pgmap v1896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:36:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2446.162328 secs 2024-07-26T07:36:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2463.905464 secs 2024-07-26T07:36:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2463.905464 secs 2024-07-26T07:36:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2463.905464 secs 2024-07-26T07:36:58.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:36:58 smithi042 ceph-mon[32775]: pgmap v1897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:36:58.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:36:58 smithi164 ceph-mon[36970]: pgmap v1897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:36:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:36:58 smithi163 ceph-mon[36873]: pgmap v1897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:37:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:00 smithi042 ceph-mon[32775]: pgmap v1898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:37:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:00 smithi164 ceph-mon[36970]: pgmap v1898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:37:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:00 smithi163 ceph-mon[36873]: pgmap v1898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:37:01.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2451.162468 secs 2024-07-26T07:37:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2451.162468 secs 2024-07-26T07:37:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2451.162468 secs 2024-07-26T07:37:02.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2468.905568 secs 2024-07-26T07:37:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:02 smithi042 ceph-mon[32775]: pgmap v1899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:37:02.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2468.905568 secs 2024-07-26T07:37:02.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:02 smithi164 ceph-mon[36970]: pgmap v1899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:37:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2468.905568 secs 2024-07-26T07:37:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:02 smithi163 ceph-mon[36873]: pgmap v1899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:37:04.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:04 smithi042 ceph-mon[32775]: pgmap v1900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:37:04.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:04 smithi164 ceph-mon[36970]: pgmap v1900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:37:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:04 smithi163 ceph-mon[36873]: pgmap v1900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:37:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:06 smithi042 ceph-mon[32775]: pgmap v1901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2456.162575 secs 2024-07-26T07:37:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:06 smithi164 ceph-mon[36970]: pgmap v1901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2456.162575 secs 2024-07-26T07:37:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:06 smithi163 ceph-mon[36873]: pgmap v1901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2456.162575 secs 2024-07-26T07:37:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2473.905671 secs 2024-07-26T07:37:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2473.905671 secs 2024-07-26T07:37:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2473.905671 secs 2024-07-26T07:37:08.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:08 smithi042 ceph-mon[32775]: pgmap v1902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:08 smithi164 ceph-mon[36970]: pgmap v1902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:08 smithi163 ceph-mon[36873]: pgmap v1902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:09.081 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:09.080+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:37:09.081 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:09.080+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:37:09.081 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:09.080+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:37:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:10 smithi042 ceph-mon[32775]: pgmap v1903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:10 smithi164 ceph-mon[36970]: pgmap v1903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:10 smithi163 ceph-mon[36873]: pgmap v1903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2461.162698 secs 2024-07-26T07:37:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2461.162698 secs 2024-07-26T07:37:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2461.162698 secs 2024-07-26T07:37:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2478.905817 secs 2024-07-26T07:37:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:12 smithi042 ceph-mon[32775]: pgmap v1904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2478.905817 secs 2024-07-26T07:37:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:12 smithi164 ceph-mon[36970]: pgmap v1904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2478.905817 secs 2024-07-26T07:37:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:12 smithi163 ceph-mon[36873]: pgmap v1904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:13.245 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:13.244+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:37:13.245 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:13.244+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:37:13.245 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:13.244+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:37:13.245 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:13.244+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:37:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:14 smithi042 ceph-mon[32775]: pgmap v1905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:37:14.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:14 smithi164 ceph-mon[36970]: pgmap v1905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:37:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:14 smithi163 ceph-mon[36873]: pgmap v1905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:37:16.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:16 smithi042 ceph-mon[32775]: pgmap v1906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2466.162822 secs 2024-07-26T07:37:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:16 smithi164 ceph-mon[36970]: pgmap v1906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2466.162822 secs 2024-07-26T07:37:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:16 smithi163 ceph-mon[36873]: pgmap v1906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2466.162822 secs 2024-07-26T07:37:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2483.905952 secs 2024-07-26T07:37:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2483.905952 secs 2024-07-26T07:37:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2483.905952 secs 2024-07-26T07:37:18.623 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:18 smithi164 ceph-mon[36970]: pgmap v1907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:18.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:18 smithi042 ceph-mon[32775]: pgmap v1907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:18 smithi163 ceph-mon[36873]: pgmap v1907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:20 smithi042 ceph-mon[32775]: pgmap v1908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:20.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:20 smithi164 ceph-mon[36970]: pgmap v1908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:20 smithi163 ceph-mon[36873]: pgmap v1908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2471.162926 secs 2024-07-26T07:37:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2471.162926 secs 2024-07-26T07:37:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2471.162926 secs 2024-07-26T07:37:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2488.906068 secs 2024-07-26T07:37:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:22 smithi042 ceph-mon[32775]: pgmap v1909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2488.906068 secs 2024-07-26T07:37:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:22 smithi164 ceph-mon[36970]: pgmap v1909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2488.906068 secs 2024-07-26T07:37:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:22 smithi163 ceph-mon[36873]: pgmap v1909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:24.084+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:37:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:24.084+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:37:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:24.084+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:37:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:24 smithi042 ceph-mon[32775]: pgmap v1910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:37:24.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:24 smithi164 ceph-mon[36970]: pgmap v1910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:37:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:24 smithi163 ceph-mon[36873]: pgmap v1910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:37:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:26 smithi042 ceph-mon[32775]: pgmap v1911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2476.163055 secs 2024-07-26T07:37:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:26 smithi164 ceph-mon[36970]: pgmap v1911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2476.163055 secs 2024-07-26T07:37:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:26 smithi163 ceph-mon[36873]: pgmap v1911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2476.163055 secs 2024-07-26T07:37:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2493.906169 secs 2024-07-26T07:37:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2493.906169 secs 2024-07-26T07:37:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2493.906169 secs 2024-07-26T07:37:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:28 smithi042 ceph-mon[32775]: pgmap v1912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:28 smithi164 ceph-mon[36970]: pgmap v1912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:28 smithi163 ceph-mon[36873]: pgmap v1912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:30 smithi042 ceph-mon[32775]: pgmap v1913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:30.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:30 smithi164 ceph-mon[36970]: pgmap v1913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:30 smithi163 ceph-mon[36873]: pgmap v1913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2481.163156 secs 2024-07-26T07:37:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2481.163156 secs 2024-07-26T07:37:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2481.163156 secs 2024-07-26T07:37:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2498.906238 secs 2024-07-26T07:37:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:32 smithi042 ceph-mon[32775]: pgmap v1914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2498.906238 secs 2024-07-26T07:37:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:32 smithi164 ceph-mon[36970]: pgmap v1914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2498.906238 secs 2024-07-26T07:37:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:32 smithi163 ceph-mon[36873]: pgmap v1914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:34 smithi042 ceph-mon[32775]: pgmap v1915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:37:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:34 smithi164 ceph-mon[36970]: pgmap v1915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:37:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:34 smithi163 ceph-mon[36873]: pgmap v1915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:37:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:36 smithi042 ceph-mon[32775]: pgmap v1916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2486.163279 secs 2024-07-26T07:37:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:36 smithi164 ceph-mon[36970]: pgmap v1916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2486.163279 secs 2024-07-26T07:37:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:36 smithi163 ceph-mon[36873]: pgmap v1916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2486.163279 secs 2024-07-26T07:37:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2503.906366 secs 2024-07-26T07:37:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2503.906366 secs 2024-07-26T07:37:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2503.906366 secs 2024-07-26T07:37:38.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:38 smithi042 ceph-mon[32775]: pgmap v1917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:38 smithi164 ceph-mon[36970]: pgmap v1917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:38 smithi163 ceph-mon[36873]: pgmap v1917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:39.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:39.092+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:37:39.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:39.092+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:37:39.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:39.092+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:37:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:40 smithi042 ceph-mon[32775]: pgmap v1918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:40 smithi164 ceph-mon[36970]: pgmap v1918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:40 smithi163 ceph-mon[36873]: pgmap v1918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2491.163375 secs 2024-07-26T07:37:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2491.163375 secs 2024-07-26T07:37:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2491.163375 secs 2024-07-26T07:37:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2508.906489 secs 2024-07-26T07:37:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:42 smithi042 ceph-mon[32775]: pgmap v1919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2508.906489 secs 2024-07-26T07:37:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:42 smithi164 ceph-mon[36970]: pgmap v1919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2508.906489 secs 2024-07-26T07:37:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:42 smithi163 ceph-mon[36873]: pgmap v1919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:44 smithi042 ceph-mon[32775]: pgmap v1920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:37:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:44 smithi164 ceph-mon[36970]: pgmap v1920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:37:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:44 smithi163 ceph-mon[36873]: pgmap v1920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:37:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:46 smithi042 ceph-mon[32775]: pgmap v1921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2496.163444 secs 2024-07-26T07:37:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:46 smithi164 ceph-mon[36970]: pgmap v1921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2496.163444 secs 2024-07-26T07:37:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:46 smithi163 ceph-mon[36873]: pgmap v1921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2496.163444 secs 2024-07-26T07:37:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2513.906610 secs 2024-07-26T07:37:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2513.906610 secs 2024-07-26T07:37:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2513.906610 secs 2024-07-26T07:37:48.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:48 smithi042 ceph-mon[32775]: pgmap v1922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:48.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:48 smithi164 ceph-mon[36970]: pgmap v1922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:48 smithi163 ceph-mon[36873]: pgmap v1922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:50.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:50 smithi164 ceph-mon[36970]: pgmap v1923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:50 smithi163 ceph-mon[36873]: pgmap v1923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:50 smithi042 ceph-mon[32775]: pgmap v1923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:37:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2501.163572 secs 2024-07-26T07:37:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2501.163572 secs 2024-07-26T07:37:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2501.163572 secs 2024-07-26T07:37:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2518.906677 secs 2024-07-26T07:37:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:52 smithi164 ceph-mon[36970]: pgmap v1924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2518.906677 secs 2024-07-26T07:37:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:52 smithi163 ceph-mon[36873]: pgmap v1924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2518.906677 secs 2024-07-26T07:37:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:52 smithi042 ceph-mon[32775]: pgmap v1924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:37:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:37:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:37:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:37:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:54.096+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:37:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:54.096+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:37:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:54.096+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:37:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:54 smithi164 ceph-mon[36970]: pgmap v1925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:37:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:37:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:37:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:37:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:54 smithi163 ceph-mon[36873]: pgmap v1925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:37:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:37:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:37:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:37:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:54 smithi042 ceph-mon[32775]: pgmap v1925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:37:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:37:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:37:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:37:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:55 smithi164 ceph-mon[36970]: pgmap v1926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:37:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:55 smithi164 ceph-mon[36970]: pgmap v1927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:37:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:55 smithi163 ceph-mon[36873]: pgmap v1926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:37:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:55 smithi163 ceph-mon[36873]: pgmap v1927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:37:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:55 smithi042 ceph-mon[32775]: pgmap v1926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:37:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:55 smithi042 ceph-mon[32775]: pgmap v1927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:37:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2506.163698 secs 2024-07-26T07:37:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2506.163698 secs 2024-07-26T07:37:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2506.163698 secs 2024-07-26T07:37:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:57 smithi164 ceph-mon[36970]: pgmap v1928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:37:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2523.906822 secs 2024-07-26T07:37:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:57 smithi163 ceph-mon[36873]: pgmap v1928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:37:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2523.906822 secs 2024-07-26T07:37:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:57 smithi042 ceph-mon[32775]: pgmap v1928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:37:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2523.906822 secs 2024-07-26T07:37:58.252 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:58.251+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:37:58.252 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:58.251+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:37:58.252 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:58.251+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:37:58.252 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:37:58.251+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:37:59.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:37:59 smithi164 ceph-mon[36970]: pgmap v1929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:37:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:37:59 smithi163 ceph-mon[36873]: pgmap v1929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:37:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:37:59 smithi042 ceph-mon[32775]: pgmap v1929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:38:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:01 smithi164 ceph-mon[36970]: pgmap v1930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:38:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2511.163824 secs 2024-07-26T07:38:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2528.906896 secs 2024-07-26T07:38:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:01 smithi163 ceph-mon[36873]: pgmap v1930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:38:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2511.163824 secs 2024-07-26T07:38:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2528.906896 secs 2024-07-26T07:38:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:01 smithi042 ceph-mon[32775]: pgmap v1930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:38:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2511.163824 secs 2024-07-26T07:38:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2528.906896 secs 2024-07-26T07:38:03.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:03 smithi164 ceph-mon[36970]: pgmap v1931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:38:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:03 smithi163 ceph-mon[36873]: pgmap v1931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:38:03.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:03 smithi042 ceph-mon[32775]: pgmap v1931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:38:05.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:05 smithi164 ceph-mon[36970]: pgmap v1932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:38:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:05 smithi163 ceph-mon[36873]: pgmap v1932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:38:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:05 smithi042 ceph-mon[32775]: pgmap v1932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:38:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2516.163949 secs 2024-07-26T07:38:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2533.907026 secs 2024-07-26T07:38:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2516.163949 secs 2024-07-26T07:38:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2533.907026 secs 2024-07-26T07:38:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2516.163949 secs 2024-07-26T07:38:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2533.907026 secs 2024-07-26T07:38:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:07 smithi164 ceph-mon[36970]: pgmap v1933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:07 smithi163 ceph-mon[36873]: pgmap v1933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:07 smithi042 ceph-mon[32775]: pgmap v1933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:38:09.104+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:38:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:38:09.104+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:38:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:38:09.104+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:38:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:09 smithi164 ceph-mon[36970]: pgmap v1934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:09 smithi163 ceph-mon[36873]: pgmap v1934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:09 smithi042 ceph-mon[32775]: pgmap v1934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:11 smithi164 ceph-mon[36970]: pgmap v1935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2521.164074 secs 2024-07-26T07:38:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2538.907093 secs 2024-07-26T07:38:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:11 smithi163 ceph-mon[36873]: pgmap v1935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2521.164074 secs 2024-07-26T07:38:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2538.907093 secs 2024-07-26T07:38:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:11 smithi042 ceph-mon[32775]: pgmap v1935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2521.164074 secs 2024-07-26T07:38:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2538.907093 secs 2024-07-26T07:38:13.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:13 smithi164 ceph-mon[36970]: pgmap v1936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:13 smithi163 ceph-mon[36873]: pgmap v1936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:13 smithi042 ceph-mon[32775]: pgmap v1936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:15.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:15 smithi164 ceph-mon[36970]: pgmap v1937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:38:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:15 smithi163 ceph-mon[36873]: pgmap v1937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:38:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:15 smithi042 ceph-mon[32775]: pgmap v1937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:38:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2526.164235 secs 2024-07-26T07:38:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2543.907173 secs 2024-07-26T07:38:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2526.164235 secs 2024-07-26T07:38:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2543.907173 secs 2024-07-26T07:38:16.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2526.164235 secs 2024-07-26T07:38:16.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2543.907173 secs 2024-07-26T07:38:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:17 smithi164 ceph-mon[36970]: pgmap v1938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:17 smithi163 ceph-mon[36873]: pgmap v1938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:17 smithi042 ceph-mon[32775]: pgmap v1938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:19.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:19 smithi164 ceph-mon[36970]: pgmap v1939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:19 smithi163 ceph-mon[36873]: pgmap v1939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:19.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:19 smithi042 ceph-mon[32775]: pgmap v1939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:21 smithi163 ceph-mon[36873]: pgmap v1940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2531.164368 secs 2024-07-26T07:38:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2548.907271 secs 2024-07-26T07:38:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:21 smithi042 ceph-mon[32775]: pgmap v1940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2531.164368 secs 2024-07-26T07:38:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2548.907271 secs 2024-07-26T07:38:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:21 smithi164 ceph-mon[36970]: pgmap v1940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2531.164368 secs 2024-07-26T07:38:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2548.907271 secs 2024-07-26T07:38:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:23 smithi163 ceph-mon[36873]: pgmap v1941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:23 smithi042 ceph-mon[32775]: pgmap v1941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:23 smithi164 ceph-mon[36970]: pgmap v1941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:24.107 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:38:24.107+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:38:24.108 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:38:24.107+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:38:24.108 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:38:24.107+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:38:25.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:25 smithi163 ceph-mon[36873]: pgmap v1942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:38:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:25 smithi042 ceph-mon[32775]: pgmap v1942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:38:25.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:25 smithi164 ceph-mon[36970]: pgmap v1942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:38:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2536.164489 secs 2024-07-26T07:38:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2553.907396 secs 2024-07-26T07:38:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2536.164489 secs 2024-07-26T07:38:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2553.907396 secs 2024-07-26T07:38:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2536.164489 secs 2024-07-26T07:38:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2553.907396 secs 2024-07-26T07:38:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:27 smithi163 ceph-mon[36873]: pgmap v1943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:27 smithi042 ceph-mon[32775]: pgmap v1943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:27 smithi164 ceph-mon[36970]: pgmap v1943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:29 smithi163 ceph-mon[36873]: pgmap v1944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:29 smithi042 ceph-mon[32775]: pgmap v1944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:29.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:29 smithi164 ceph-mon[36970]: pgmap v1944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:31 smithi163 ceph-mon[36873]: pgmap v1945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2541.164597 secs 2024-07-26T07:38:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2558.907551 secs 2024-07-26T07:38:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:31 smithi042 ceph-mon[32775]: pgmap v1945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2541.164597 secs 2024-07-26T07:38:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2558.907551 secs 2024-07-26T07:38:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:31 smithi164 ceph-mon[36970]: pgmap v1945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2541.164597 secs 2024-07-26T07:38:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2558.907551 secs 2024-07-26T07:38:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:33 smithi163 ceph-mon[36873]: pgmap v1946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:33 smithi042 ceph-mon[32775]: pgmap v1946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:33.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:33 smithi164 ceph-mon[36970]: pgmap v1946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:35.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:35 smithi163 ceph-mon[36873]: pgmap v1947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:38:35.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:35 smithi042 ceph-mon[32775]: pgmap v1947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:38:35.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:35 smithi164 ceph-mon[36970]: pgmap v1947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:38:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2546.164745 secs 2024-07-26T07:38:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2563.907674 secs 2024-07-26T07:38:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2546.164745 secs 2024-07-26T07:38:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2563.907674 secs 2024-07-26T07:38:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2546.164745 secs 2024-07-26T07:38:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2563.907674 secs 2024-07-26T07:38:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:37 smithi042 ceph-mon[32775]: pgmap v1948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:37 smithi164 ceph-mon[36970]: pgmap v1948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:37 smithi163 ceph-mon[36873]: pgmap v1948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:38:39.115+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:38:39.117 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:38:39.115+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:38:39.117 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:38:39.115+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:38:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:39 smithi042 ceph-mon[32775]: pgmap v1949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:39.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:39 smithi164 ceph-mon[36970]: pgmap v1949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:40.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:39 smithi163 ceph-mon[36873]: pgmap v1949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:41 smithi042 ceph-mon[32775]: pgmap v1950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2551.164869 secs 2024-07-26T07:38:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2568.907823 secs 2024-07-26T07:38:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:41 smithi164 ceph-mon[36970]: pgmap v1950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2551.164869 secs 2024-07-26T07:38:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2568.907823 secs 2024-07-26T07:38:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:41 smithi163 ceph-mon[36873]: pgmap v1950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2551.164869 secs 2024-07-26T07:38:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2568.907823 secs 2024-07-26T07:38:43.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:43 smithi042 ceph-mon[32775]: pgmap v1951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:43.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:43 smithi164 ceph-mon[36970]: pgmap v1951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:43 smithi163 ceph-mon[36873]: pgmap v1951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:45 smithi042 ceph-mon[32775]: pgmap v1952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:38:45.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:45 smithi164 ceph-mon[36970]: pgmap v1952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:38:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:45 smithi163 ceph-mon[36873]: pgmap v1952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:38:46.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2556.165000 secs 2024-07-26T07:38:46.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2573.907974 secs 2024-07-26T07:38:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2556.165000 secs 2024-07-26T07:38:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2573.907974 secs 2024-07-26T07:38:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2556.165000 secs 2024-07-26T07:38:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2573.907974 secs 2024-07-26T07:38:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:47 smithi042 ceph-mon[32775]: pgmap v1953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:47.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:47 smithi164 ceph-mon[36970]: pgmap v1953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:47 smithi163 ceph-mon[36873]: pgmap v1953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:49 smithi042 ceph-mon[32775]: pgmap v1954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:49.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:49 smithi164 ceph-mon[36970]: pgmap v1954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:49 smithi163 ceph-mon[36873]: pgmap v1954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:51.809 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:51 smithi163 ceph-mon[36873]: pgmap v1955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:51.810 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2561.165127 secs 2024-07-26T07:38:51.810 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2578.908103 secs 2024-07-26T07:38:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:51 smithi042 ceph-mon[32775]: pgmap v1955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2561.165127 secs 2024-07-26T07:38:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2578.908103 secs 2024-07-26T07:38:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:51 smithi164 ceph-mon[36970]: pgmap v1955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:38:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2561.165127 secs 2024-07-26T07:38:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2578.908103 secs 2024-07-26T07:38:53.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:53 smithi042 ceph-mon[32775]: pgmap v1956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:53.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:53 smithi164 ceph-mon[36970]: pgmap v1956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:53 smithi163 ceph-mon[36873]: pgmap v1956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:38:54.119 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:38:54.118+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:38:54.119 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:38:54.118+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:38:54.119 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:38:54.118+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:38:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:38:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:38:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:38:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:38:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:38:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:38:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:38:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:38:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:38:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:38:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:55 smithi042 ceph-mon[32775]: pgmap v1957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:38:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:55 smithi042 ceph-mon[32775]: pgmap v1958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:38:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:55 smithi042 ceph-mon[32775]: pgmap v1959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:38:55.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:55 smithi164 ceph-mon[36970]: pgmap v1957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:38:55.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:55 smithi164 ceph-mon[36970]: pgmap v1958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:38:55.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:55 smithi164 ceph-mon[36970]: pgmap v1959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:38:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:55 smithi163 ceph-mon[36873]: pgmap v1957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:38:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:55 smithi163 ceph-mon[36873]: pgmap v1958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:38:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:55 smithi163 ceph-mon[36873]: pgmap v1959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:38:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2566.165276 secs 2024-07-26T07:38:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2583.908217 secs 2024-07-26T07:38:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2566.165276 secs 2024-07-26T07:38:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2583.908217 secs 2024-07-26T07:38:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2566.165276 secs 2024-07-26T07:38:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2583.908217 secs 2024-07-26T07:38:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:57 smithi042 ceph-mon[32775]: pgmap v1960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:38:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:57 smithi164 ceph-mon[36970]: pgmap v1960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:38:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:57 smithi163 ceph-mon[36873]: pgmap v1960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:38:59.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:38:59 smithi042 ceph-mon[32775]: pgmap v1961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:38:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:38:59 smithi164 ceph-mon[36970]: pgmap v1961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:39:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:38:59 smithi163 ceph-mon[36873]: pgmap v1961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:39:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:01 smithi042 ceph-mon[32775]: pgmap v1962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:39:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2571.165399 secs 2024-07-26T07:39:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2588.908295 secs 2024-07-26T07:39:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:01 smithi164 ceph-mon[36970]: pgmap v1962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:39:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2571.165399 secs 2024-07-26T07:39:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2588.908295 secs 2024-07-26T07:39:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:01 smithi163 ceph-mon[36873]: pgmap v1962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:39:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2571.165399 secs 2024-07-26T07:39:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2588.908295 secs 2024-07-26T07:39:03.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:03 smithi042 ceph-mon[32775]: pgmap v1963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:39:03.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:03 smithi164 ceph-mon[36970]: pgmap v1963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:39:04.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:03 smithi163 ceph-mon[36873]: pgmap v1963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:39:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:05 smithi042 ceph-mon[32775]: pgmap v1964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:39:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:05 smithi164 ceph-mon[36970]: pgmap v1964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:39:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:05 smithi163 ceph-mon[36873]: pgmap v1964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:39:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2576.165525 secs 2024-07-26T07:39:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2593.908427 secs 2024-07-26T07:39:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2576.165525 secs 2024-07-26T07:39:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2593.908427 secs 2024-07-26T07:39:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2576.165525 secs 2024-07-26T07:39:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2593.908427 secs 2024-07-26T07:39:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:07 smithi042 ceph-mon[32775]: pgmap v1965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:07.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:07 smithi164 ceph-mon[36970]: pgmap v1965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:07 smithi163 ceph-mon[36873]: pgmap v1965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:09.127 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:39:09.126+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:39:09.127 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:39:09.126+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:39:09.127 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:39:09.126+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:39:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:09 smithi042 ceph-mon[32775]: pgmap v1966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:39:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:09 smithi164 ceph-mon[36970]: pgmap v1966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:39:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:09 smithi163 ceph-mon[36873]: pgmap v1966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:39:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:10 smithi042 ceph-mon[32775]: pgmap v1967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:10 smithi164 ceph-mon[36970]: pgmap v1967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:11.018 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:10 smithi163 ceph-mon[36873]: pgmap v1967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2581.165656 secs 2024-07-26T07:39:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2598.908551 secs 2024-07-26T07:39:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2581.165656 secs 2024-07-26T07:39:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2598.908551 secs 2024-07-26T07:39:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2581.165656 secs 2024-07-26T07:39:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2598.908551 secs 2024-07-26T07:39:12.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:12 smithi042 ceph-mon[32775]: pgmap v1968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:12 smithi164 ceph-mon[36970]: pgmap v1968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:12 smithi163 ceph-mon[36873]: pgmap v1968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:15 smithi042 ceph-mon[32775]: pgmap v1969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:15.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:15 smithi164 ceph-mon[36970]: pgmap v1969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:15 smithi163 ceph-mon[36873]: pgmap v1969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2586.165781 secs 2024-07-26T07:39:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2586.165781 secs 2024-07-26T07:39:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2586.165781 secs 2024-07-26T07:39:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2603.908677 secs 2024-07-26T07:39:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:17 smithi042 ceph-mon[32775]: pgmap v1970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2603.908677 secs 2024-07-26T07:39:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:17 smithi164 ceph-mon[36970]: pgmap v1970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2603.908677 secs 2024-07-26T07:39:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:17 smithi163 ceph-mon[36873]: pgmap v1970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:19.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:19 smithi042 ceph-mon[32775]: pgmap v1971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:39:19.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:19 smithi164 ceph-mon[36970]: pgmap v1971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:39:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:19 smithi163 ceph-mon[36873]: pgmap v1971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:39:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:21 smithi042 ceph-mon[32775]: pgmap v1972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2591.165905 secs 2024-07-26T07:39:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:21 smithi164 ceph-mon[36970]: pgmap v1972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2591.165905 secs 2024-07-26T07:39:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:21 smithi163 ceph-mon[36873]: pgmap v1972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2591.165905 secs 2024-07-26T07:39:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2608.908741 secs 2024-07-26T07:39:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2608.908741 secs 2024-07-26T07:39:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2608.908741 secs 2024-07-26T07:39:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:23 smithi042 ceph-mon[32775]: pgmap v1973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:23.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:23 smithi164 ceph-mon[36970]: pgmap v1973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:23 smithi163 ceph-mon[36873]: pgmap v1973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:24.130 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:39:24.130+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:39:24.130 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:39:24.130+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:39:24.130 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:39:24.130+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:39:25.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:25 smithi164 ceph-mon[36970]: pgmap v1974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:25.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:25 smithi163 ceph-mon[36873]: pgmap v1974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:25 smithi042 ceph-mon[32775]: pgmap v1974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2596.166029 secs 2024-07-26T07:39:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2596.166029 secs 2024-07-26T07:39:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2596.166029 secs 2024-07-26T07:39:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2613.908864 secs 2024-07-26T07:39:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:27 smithi164 ceph-mon[36970]: pgmap v1975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2613.908864 secs 2024-07-26T07:39:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:27 smithi163 ceph-mon[36873]: pgmap v1975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2613.908864 secs 2024-07-26T07:39:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:27 smithi042 ceph-mon[32775]: pgmap v1975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:29.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:29 smithi164 ceph-mon[36970]: pgmap v1976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:39:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:29 smithi163 ceph-mon[36873]: pgmap v1976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:39:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:29 smithi042 ceph-mon[32775]: pgmap v1976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:39:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:31 smithi164 ceph-mon[36970]: pgmap v1977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2601.166152 secs 2024-07-26T07:39:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:31 smithi163 ceph-mon[36873]: pgmap v1977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2601.166152 secs 2024-07-26T07:39:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:31 smithi042 ceph-mon[32775]: pgmap v1977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2601.166152 secs 2024-07-26T07:39:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2618.909010 secs 2024-07-26T07:39:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2618.909010 secs 2024-07-26T07:39:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2618.909010 secs 2024-07-26T07:39:33.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:33 smithi164 ceph-mon[36970]: pgmap v1978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:33 smithi163 ceph-mon[36873]: pgmap v1978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:33.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:33 smithi042 ceph-mon[32775]: pgmap v1978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:35.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:35 smithi042 ceph-mon[32775]: pgmap v1979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:35.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:35 smithi164 ceph-mon[36970]: pgmap v1979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:35.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:35 smithi163 ceph-mon[36873]: pgmap v1979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2606.166302 secs 2024-07-26T07:39:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2606.166302 secs 2024-07-26T07:39:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2606.166302 secs 2024-07-26T07:39:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2623.909094 secs 2024-07-26T07:39:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:37 smithi164 ceph-mon[36970]: pgmap v1980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2623.909094 secs 2024-07-26T07:39:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:37 smithi163 ceph-mon[36873]: pgmap v1980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2623.909094 secs 2024-07-26T07:39:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:37 smithi042 ceph-mon[32775]: pgmap v1980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:39:39.138+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:39:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:39:39.138+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:39:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:39:39.138+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:39:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:39 smithi164 ceph-mon[36970]: pgmap v1981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:39:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:39 smithi163 ceph-mon[36873]: pgmap v1981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:39:39.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:39 smithi042 ceph-mon[32775]: pgmap v1981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:39:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:41 smithi164 ceph-mon[36970]: pgmap v1982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2611.166368 secs 2024-07-26T07:39:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:41 smithi163 ceph-mon[36873]: pgmap v1982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2611.166368 secs 2024-07-26T07:39:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:41 smithi042 ceph-mon[32775]: pgmap v1982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2611.166368 secs 2024-07-26T07:39:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2628.909215 secs 2024-07-26T07:39:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2628.909215 secs 2024-07-26T07:39:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2628.909215 secs 2024-07-26T07:39:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:43 smithi164 ceph-mon[36970]: pgmap v1983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:43 smithi163 ceph-mon[36873]: pgmap v1983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:43.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:43 smithi042 ceph-mon[32775]: pgmap v1983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:45 smithi164 ceph-mon[36970]: pgmap v1984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:45 smithi163 ceph-mon[36873]: pgmap v1984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:45.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:45 smithi042 ceph-mon[32775]: pgmap v1984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2616.166440 secs 2024-07-26T07:39:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2633.909341 secs 2024-07-26T07:39:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2616.166440 secs 2024-07-26T07:39:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2633.909341 secs 2024-07-26T07:39:46.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2616.166440 secs 2024-07-26T07:39:46.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2633.909341 secs 2024-07-26T07:39:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:47 smithi164 ceph-mon[36970]: pgmap v1985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:47 smithi163 ceph-mon[36873]: pgmap v1985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:47 smithi042 ceph-mon[32775]: pgmap v1985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:49 smithi164 ceph-mon[36970]: pgmap v1986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:39:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:49 smithi163 ceph-mon[36873]: pgmap v1986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:39:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:49 smithi042 ceph-mon[32775]: pgmap v1986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:39:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:51 smithi164 ceph-mon[36970]: pgmap v1987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2621.166566 secs 2024-07-26T07:39:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2638.909466 secs 2024-07-26T07:39:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:51 smithi163 ceph-mon[36873]: pgmap v1987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2621.166566 secs 2024-07-26T07:39:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2638.909466 secs 2024-07-26T07:39:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:51 smithi042 ceph-mon[32775]: pgmap v1987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2621.166566 secs 2024-07-26T07:39:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2638.909466 secs 2024-07-26T07:39:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:53 smithi164 ceph-mon[36970]: pgmap v1988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:53 smithi163 ceph-mon[36873]: pgmap v1988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:53 smithi042 ceph-mon[32775]: pgmap v1988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:54.141 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:39:54.140+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:39:54.141 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:39:54.140+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:39:54.141 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:39:54.140+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:39:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:39:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:39:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:39:55.472 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:55 smithi163 ceph-mon[36873]: pgmap v1989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:55 smithi164 ceph-mon[36970]: pgmap v1989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:55.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:55 smithi042 ceph-mon[32775]: pgmap v1989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:39:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2626.166631 secs 2024-07-26T07:39:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2643.909538 secs 2024-07-26T07:39:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:57 smithi042 ceph-mon[32775]: pgmap v1990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2626.166631 secs 2024-07-26T07:39:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2643.909538 secs 2024-07-26T07:39:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:57 smithi164 ceph-mon[36970]: pgmap v1990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2626.166631 secs 2024-07-26T07:39:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2643.909538 secs 2024-07-26T07:39:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:57 smithi163 ceph-mon[36873]: pgmap v1990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:39:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:39:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:39:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:58 smithi042 ceph-mon[32775]: pgmap v1991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:39:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:58 smithi042 ceph-mon[32775]: pgmap v1992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:39:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:58.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:39:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:39:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:58 smithi164 ceph-mon[36970]: pgmap v1991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:39:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:58 smithi164 ceph-mon[36970]: pgmap v1992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:39:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:39:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:39:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:39:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:58 smithi163 ceph-mon[36873]: pgmap v1991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:39:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:58 smithi163 ceph-mon[36873]: pgmap v1992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:39:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:00 smithi042 ceph-mon[32775]: pgmap v1993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:40:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:40:00.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:00 smithi164 ceph-mon[36970]: pgmap v1993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:40:00.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:40:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:00 smithi163 ceph-mon[36873]: pgmap v1993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:40:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:40:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2631.166775 secs 2024-07-26T07:40:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2631.166775 secs 2024-07-26T07:40:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2631.166775 secs 2024-07-26T07:40:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:02 smithi042 ceph-mon[32775]: pgmap v1994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:40:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2648.909658 secs 2024-07-26T07:40:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:02 smithi164 ceph-mon[36970]: pgmap v1994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:40:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2648.909658 secs 2024-07-26T07:40:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:02 smithi163 ceph-mon[36873]: pgmap v1994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:40:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2648.909658 secs 2024-07-26T07:40:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:04 smithi042 ceph-mon[32775]: pgmap v1995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:40:04.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:04 smithi164 ceph-mon[36970]: pgmap v1995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:40:04.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:04 smithi163 ceph-mon[36873]: pgmap v1995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:40:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:06 smithi042 ceph-mon[32775]: pgmap v1996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:40:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2636.166928 secs 2024-07-26T07:40:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:06 smithi164 ceph-mon[36970]: pgmap v1996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:40:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2636.166928 secs 2024-07-26T07:40:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:06 smithi163 ceph-mon[36873]: pgmap v1996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:40:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2636.166928 secs 2024-07-26T07:40:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2653.909786 secs 2024-07-26T07:40:07.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2653.909786 secs 2024-07-26T07:40:07.506 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2653.909786 secs 2024-07-26T07:40:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:08 smithi042 ceph-mon[32775]: pgmap v1997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:40:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:08 smithi164 ceph-mon[36970]: pgmap v1997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:40:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:08 smithi163 ceph-mon[36873]: pgmap v1997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:40:09.149 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:09.148+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:40:09.150 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:09.148+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:40:09.150 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:09.148+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:40:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:10 smithi042 ceph-mon[32775]: pgmap v1998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:10.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:10 smithi164 ceph-mon[36970]: pgmap v1998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:10 smithi163 ceph-mon[36873]: pgmap v1998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2641.167075 secs 2024-07-26T07:40:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2641.167075 secs 2024-07-26T07:40:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2641.167075 secs 2024-07-26T07:40:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2658.909888 secs 2024-07-26T07:40:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:12 smithi042 ceph-mon[32775]: pgmap v1999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2658.909888 secs 2024-07-26T07:40:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:12 smithi164 ceph-mon[36970]: pgmap v1999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2658.909888 secs 2024-07-26T07:40:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:12 smithi163 ceph-mon[36873]: pgmap v1999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:14 smithi042 ceph-mon[32775]: pgmap v2000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:14.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:14 smithi164 ceph-mon[36970]: pgmap v2000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:14.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:14 smithi163 ceph-mon[36873]: pgmap v2000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:16 smithi042 ceph-mon[32775]: pgmap v2001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2646.167192 secs 2024-07-26T07:40:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:16 smithi164 ceph-mon[36970]: pgmap v2001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2646.167192 secs 2024-07-26T07:40:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:16 smithi163 ceph-mon[36873]: pgmap v2001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2646.167192 secs 2024-07-26T07:40:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2663.910013 secs 2024-07-26T07:40:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2663.910013 secs 2024-07-26T07:40:17.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2663.910013 secs 2024-07-26T07:40:18.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:18 smithi164 ceph-mon[36970]: pgmap v2002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:18 smithi163 ceph-mon[36873]: pgmap v2002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:18 smithi042 ceph-mon[32775]: pgmap v2002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:20 smithi164 ceph-mon[36970]: pgmap v2003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:20 smithi163 ceph-mon[36873]: pgmap v2003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:20 smithi042 ceph-mon[32775]: pgmap v2003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2651.167322 secs 2024-07-26T07:40:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2651.167322 secs 2024-07-26T07:40:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2651.167322 secs 2024-07-26T07:40:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2668.910107 secs 2024-07-26T07:40:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:22 smithi164 ceph-mon[36970]: pgmap v2004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2668.910107 secs 2024-07-26T07:40:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:22 smithi163 ceph-mon[36873]: pgmap v2004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2668.910107 secs 2024-07-26T07:40:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:22 smithi042 ceph-mon[32775]: pgmap v2004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:24.150+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:40:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:24.150+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:40:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:24.150+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:40:24.420 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:24 smithi042 ceph-mon[32775]: pgmap v2005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:24.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:24 smithi164 ceph-mon[36970]: pgmap v2005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:24 smithi163 ceph-mon[36873]: pgmap v2005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:26 smithi164 ceph-mon[36970]: pgmap v2006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2656.167440 secs 2024-07-26T07:40:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:26 smithi163 ceph-mon[36873]: pgmap v2006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2656.167440 secs 2024-07-26T07:40:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:26 smithi042 ceph-mon[32775]: pgmap v2006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2656.167440 secs 2024-07-26T07:40:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2673.910163 secs 2024-07-26T07:40:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2673.910163 secs 2024-07-26T07:40:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2673.910163 secs 2024-07-26T07:40:28.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:28 smithi164 ceph-mon[36970]: pgmap v2007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:28 smithi163 ceph-mon[36873]: pgmap v2007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:28 smithi042 ceph-mon[32775]: pgmap v2007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:30 smithi164 ceph-mon[36970]: pgmap v2008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:30 smithi163 ceph-mon[36873]: pgmap v2008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:30 smithi042 ceph-mon[32775]: pgmap v2008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2661.167569 secs 2024-07-26T07:40:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2661.167569 secs 2024-07-26T07:40:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2661.167569 secs 2024-07-26T07:40:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2678.910277 secs 2024-07-26T07:40:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:32 smithi164 ceph-mon[36970]: pgmap v2009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2678.910277 secs 2024-07-26T07:40:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:32 smithi163 ceph-mon[36873]: pgmap v2009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2678.910277 secs 2024-07-26T07:40:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:32 smithi042 ceph-mon[32775]: pgmap v2009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:34.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:34 smithi164 ceph-mon[36970]: pgmap v2010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:34.482 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:34 smithi042 ceph-mon[32775]: pgmap v2010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:34 smithi163 ceph-mon[36873]: pgmap v2010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:36 smithi164 ceph-mon[36970]: pgmap v2011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2666.167726 secs 2024-07-26T07:40:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:36 smithi163 ceph-mon[36873]: pgmap v2011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2666.167726 secs 2024-07-26T07:40:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:36 smithi042 ceph-mon[32775]: pgmap v2011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2666.167726 secs 2024-07-26T07:40:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2683.910397 secs 2024-07-26T07:40:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2683.910397 secs 2024-07-26T07:40:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2683.910397 secs 2024-07-26T07:40:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:38 smithi164 ceph-mon[36970]: pgmap v2012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:38 smithi163 ceph-mon[36873]: pgmap v2012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:38 smithi042 ceph-mon[32775]: pgmap v2012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:39.159 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:39.158+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:40:39.159 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:39.158+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:40:39.159 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:39.158+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:40:40.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:40 smithi164 ceph-mon[36970]: pgmap v2013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:40 smithi163 ceph-mon[36873]: pgmap v2013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:40 smithi042 ceph-mon[32775]: pgmap v2013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2671.167849 secs 2024-07-26T07:40:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2671.167849 secs 2024-07-26T07:40:41.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2671.167849 secs 2024-07-26T07:40:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2688.910517 secs 2024-07-26T07:40:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:42 smithi164 ceph-mon[36970]: pgmap v2014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2688.910517 secs 2024-07-26T07:40:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:42 smithi163 ceph-mon[36873]: pgmap v2014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2688.910517 secs 2024-07-26T07:40:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:42 smithi042 ceph-mon[32775]: pgmap v2014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:44 smithi163 ceph-mon[36873]: pgmap v2015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:44 smithi042 ceph-mon[32775]: pgmap v2015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:44 smithi164 ceph-mon[36970]: pgmap v2015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:46 smithi163 ceph-mon[36873]: pgmap v2016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2676.167989 secs 2024-07-26T07:40:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:46 smithi042 ceph-mon[32775]: pgmap v2016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2676.167989 secs 2024-07-26T07:40:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:46 smithi164 ceph-mon[36970]: pgmap v2016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2676.167989 secs 2024-07-26T07:40:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2693.910638 secs 2024-07-26T07:40:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2693.910638 secs 2024-07-26T07:40:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2693.910638 secs 2024-07-26T07:40:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:48 smithi163 ceph-mon[36873]: pgmap v2017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:48.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:48 smithi042 ceph-mon[32775]: pgmap v2017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:48.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:48 smithi164 ceph-mon[36970]: pgmap v2017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:50 smithi163 ceph-mon[36873]: pgmap v2018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:50.524 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:50 smithi164 ceph-mon[36970]: pgmap v2018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:50 smithi042 ceph-mon[32775]: pgmap v2018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2681.168111 secs 2024-07-26T07:40:51.540 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2681.168111 secs 2024-07-26T07:40:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2681.168111 secs 2024-07-26T07:40:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2698.910781 secs 2024-07-26T07:40:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:52 smithi163 ceph-mon[36873]: pgmap v2019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2698.910781 secs 2024-07-26T07:40:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:52 smithi042 ceph-mon[32775]: pgmap v2019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2698.910781 secs 2024-07-26T07:40:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:52 smithi164 ceph-mon[36970]: pgmap v2019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:54.162+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:40:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:54.162+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:40:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:54.162+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:40:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:54 smithi163 ceph-mon[36873]: pgmap v2020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:54 smithi042 ceph-mon[32775]: pgmap v2020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:54 smithi164 ceph-mon[36970]: pgmap v2020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:56 smithi163 ceph-mon[36873]: pgmap v2021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2686.168254 secs 2024-07-26T07:40:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:56 smithi042 ceph-mon[32775]: pgmap v2021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2686.168254 secs 2024-07-26T07:40:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:56 smithi164 ceph-mon[36970]: pgmap v2021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:40:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2686.168254 secs 2024-07-26T07:40:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2703.910920 secs 2024-07-26T07:40:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:40:57.644 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2703.910920 secs 2024-07-26T07:40:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:40:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2703.910920 secs 2024-07-26T07:40:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:40:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:57.864+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0c03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T07:40:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:57.865+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0c03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T07:40:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:57.865+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0c03f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff678031550 tx=0x7ff67000d3d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:40:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:57.865+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0c03f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff678031550 tx=0x7ff67000d3d0 comp rx=0 tx=0).stop 2024-07-26T07:40:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:57.865+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0c03f0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T07:40:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:57.865+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0c03f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T07:40:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:57.865+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0c03f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:40:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:57.865+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:40:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:57.865+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:40:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:57.865+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff6780315a0 tx=0x7ff67804b070 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:40:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:58.280+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:40:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:58.280+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:40:58.281 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:58.280+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:40:58.281 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:40:58.280+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:40:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:58 smithi163 ceph-mon[36873]: pgmap v2022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:40:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:58 smithi042 ceph-mon[32775]: pgmap v2022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:40:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:58 smithi164 ceph-mon[36970]: pgmap v2022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:40:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:40:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:59 smithi163 ceph-mon[36873]: pgmap v2023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:40:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:40:59 smithi163 ceph-mon[36873]: pgmap v2024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:40:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:59 smithi042 ceph-mon[32775]: pgmap v2023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:40:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:40:59 smithi042 ceph-mon[32775]: pgmap v2024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:40:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:59 smithi164 ceph-mon[36970]: pgmap v2023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:40:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:40:59 smithi164 ceph-mon[36970]: pgmap v2024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T07:41:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:01 smithi163 ceph-mon[36873]: pgmap v2025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:41:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2691.168370 secs 2024-07-26T07:41:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2708.911039 secs 2024-07-26T07:41:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:01 smithi042 ceph-mon[32775]: pgmap v2025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:41:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2691.168370 secs 2024-07-26T07:41:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2708.911039 secs 2024-07-26T07:41:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:01 smithi164 ceph-mon[36970]: pgmap v2025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:41:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2691.168370 secs 2024-07-26T07:41:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2708.911039 secs 2024-07-26T07:41:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:03 smithi163 ceph-mon[36873]: pgmap v2026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:41:03.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:03 smithi042 ceph-mon[32775]: pgmap v2026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:41:03.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:03 smithi164 ceph-mon[36970]: pgmap v2026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:41:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:05 smithi163 ceph-mon[36873]: pgmap v2027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:41:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:05 smithi042 ceph-mon[32775]: pgmap v2027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:41:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:05 smithi164 ceph-mon[36970]: pgmap v2027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:41:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2696.168502 secs 2024-07-26T07:41:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2713.911156 secs 2024-07-26T07:41:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2696.168502 secs 2024-07-26T07:41:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2713.911156 secs 2024-07-26T07:41:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2696.168502 secs 2024-07-26T07:41:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2713.911156 secs 2024-07-26T07:41:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:07 smithi163 ceph-mon[36873]: pgmap v2028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:41:07.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:07 smithi042 ceph-mon[32775]: pgmap v2028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:41:07.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:07 smithi164 ceph-mon[36970]: pgmap v2028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T07:41:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:41:09.163+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:41:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:41:09.163+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:41:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:41:09.163+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:41:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:09 smithi163 ceph-mon[36873]: pgmap v2029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:41:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:09 smithi042 ceph-mon[32775]: pgmap v2029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:41:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:09 smithi164 ceph-mon[36970]: pgmap v2029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:41:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:11 smithi163 ceph-mon[36873]: pgmap v2030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2701.168567 secs 2024-07-26T07:41:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2718.911284 secs 2024-07-26T07:41:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:11 smithi042 ceph-mon[32775]: pgmap v2030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2701.168567 secs 2024-07-26T07:41:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2718.911284 secs 2024-07-26T07:41:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:11 smithi164 ceph-mon[36970]: pgmap v2030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2701.168567 secs 2024-07-26T07:41:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2718.911284 secs 2024-07-26T07:41:13.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:13 smithi042 ceph-mon[32775]: pgmap v2031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:13.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:13 smithi164 ceph-mon[36970]: pgmap v2031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:13 smithi163 ceph-mon[36873]: pgmap v2031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:15 smithi042 ceph-mon[32775]: pgmap v2032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:15.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:15 smithi164 ceph-mon[36970]: pgmap v2032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:15 smithi163 ceph-mon[36873]: pgmap v2032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:16.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2706.168699 secs 2024-07-26T07:41:16.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2723.911405 secs 2024-07-26T07:41:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2706.168699 secs 2024-07-26T07:41:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2723.911405 secs 2024-07-26T07:41:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2706.168699 secs 2024-07-26T07:41:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2723.911405 secs 2024-07-26T07:41:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:17 smithi042 ceph-mon[32775]: pgmap v2033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:17.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:17 smithi164 ceph-mon[36970]: pgmap v2033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:17 smithi163 ceph-mon[36873]: pgmap v2033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:19.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:19 smithi042 ceph-mon[32775]: pgmap v2034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:19.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:19 smithi164 ceph-mon[36970]: pgmap v2034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:20.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:19 smithi163 ceph-mon[36873]: pgmap v2034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:21.809 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:21 smithi163 ceph-mon[36873]: pgmap v2035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:21.809 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2711.168766 secs 2024-07-26T07:41:21.810 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2728.911526 secs 2024-07-26T07:41:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:21 smithi042 ceph-mon[32775]: pgmap v2035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2711.168766 secs 2024-07-26T07:41:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2728.911526 secs 2024-07-26T07:41:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:21 smithi164 ceph-mon[36970]: pgmap v2035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2711.168766 secs 2024-07-26T07:41:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2728.911526 secs 2024-07-26T07:41:23.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:23 smithi042 ceph-mon[32775]: pgmap v2036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:23 smithi164 ceph-mon[36970]: pgmap v2036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:23 smithi163 ceph-mon[36873]: pgmap v2036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:41:24.165+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:41:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:41:24.165+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:41:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:41:24.165+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:41:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:25 smithi042 ceph-mon[32775]: pgmap v2037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:25.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:25 smithi164 ceph-mon[36970]: pgmap v2037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:25 smithi163 ceph-mon[36873]: pgmap v2037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2716.168897 secs 2024-07-26T07:41:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2733.911647 secs 2024-07-26T07:41:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2716.168897 secs 2024-07-26T07:41:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2733.911647 secs 2024-07-26T07:41:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2716.168897 secs 2024-07-26T07:41:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2733.911647 secs 2024-07-26T07:41:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:27 smithi042 ceph-mon[32775]: pgmap v2038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:27 smithi164 ceph-mon[36970]: pgmap v2038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:27 smithi163 ceph-mon[36873]: pgmap v2038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:29 smithi042 ceph-mon[32775]: pgmap v2039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:29.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:29 smithi164 ceph-mon[36970]: pgmap v2039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:29 smithi163 ceph-mon[36873]: pgmap v2039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:31 smithi042 ceph-mon[32775]: pgmap v2040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2721.169040 secs 2024-07-26T07:41:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2738.911792 secs 2024-07-26T07:41:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:31 smithi164 ceph-mon[36970]: pgmap v2040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2721.169040 secs 2024-07-26T07:41:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2738.911792 secs 2024-07-26T07:41:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:31 smithi163 ceph-mon[36873]: pgmap v2040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2721.169040 secs 2024-07-26T07:41:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2738.911792 secs 2024-07-26T07:41:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:33 smithi042 ceph-mon[32775]: pgmap v2041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:33.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:33 smithi164 ceph-mon[36970]: pgmap v2041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:34.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:33 smithi163 ceph-mon[36873]: pgmap v2041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:35.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:35 smithi042 ceph-mon[32775]: pgmap v2042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:35.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:35 smithi164 ceph-mon[36970]: pgmap v2042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:36.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:35 smithi163 ceph-mon[36873]: pgmap v2042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2726.169158 secs 2024-07-26T07:41:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2743.911920 secs 2024-07-26T07:41:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2726.169158 secs 2024-07-26T07:41:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2743.911920 secs 2024-07-26T07:41:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2726.169158 secs 2024-07-26T07:41:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2743.911920 secs 2024-07-26T07:41:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:37 smithi042 ceph-mon[32775]: pgmap v2043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:37 smithi164 ceph-mon[36970]: pgmap v2043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:37 smithi163 ceph-mon[36873]: pgmap v2043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:39.167 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:41:39.166+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:41:39.167 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:41:39.166+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:41:39.167 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:41:39.166+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:41:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:39 smithi042 ceph-mon[32775]: pgmap v2044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:39.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:39 smithi164 ceph-mon[36970]: pgmap v2044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:40.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:39 smithi163 ceph-mon[36873]: pgmap v2044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:41 smithi042 ceph-mon[32775]: pgmap v2045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2731.169303 secs 2024-07-26T07:41:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2748.912010 secs 2024-07-26T07:41:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:41 smithi164 ceph-mon[36970]: pgmap v2045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2731.169303 secs 2024-07-26T07:41:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2748.912010 secs 2024-07-26T07:41:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:41 smithi163 ceph-mon[36873]: pgmap v2045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2731.169303 secs 2024-07-26T07:41:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2748.912010 secs 2024-07-26T07:41:43.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:43 smithi042 ceph-mon[32775]: pgmap v2046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:43.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:43 smithi164 ceph-mon[36970]: pgmap v2046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:43 smithi163 ceph-mon[36873]: pgmap v2046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:45 smithi042 ceph-mon[32775]: pgmap v2047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:45.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:45 smithi164 ceph-mon[36970]: pgmap v2047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:45 smithi163 ceph-mon[36873]: pgmap v2047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2736.169408 secs 2024-07-26T07:41:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2753.912075 secs 2024-07-26T07:41:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2736.169408 secs 2024-07-26T07:41:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2753.912075 secs 2024-07-26T07:41:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2736.169408 secs 2024-07-26T07:41:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2753.912075 secs 2024-07-26T07:41:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:47 smithi042 ceph-mon[32775]: pgmap v2048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:47.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:47 smithi164 ceph-mon[36970]: pgmap v2048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:47 smithi163 ceph-mon[36873]: pgmap v2048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:49 smithi042 ceph-mon[32775]: pgmap v2049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:49.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:49 smithi164 ceph-mon[36970]: pgmap v2049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:49 smithi163 ceph-mon[36873]: pgmap v2049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:51 smithi042 ceph-mon[32775]: pgmap v2050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2741.169567 secs 2024-07-26T07:41:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2758.912190 secs 2024-07-26T07:41:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:51 smithi164 ceph-mon[36970]: pgmap v2050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2741.169567 secs 2024-07-26T07:41:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2758.912190 secs 2024-07-26T07:41:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:51 smithi163 ceph-mon[36873]: pgmap v2050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2741.169567 secs 2024-07-26T07:41:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2758.912190 secs 2024-07-26T07:41:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:53 smithi042 ceph-mon[32775]: pgmap v2051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:53.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:53 smithi164 ceph-mon[36970]: pgmap v2051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:53 smithi163 ceph-mon[36873]: pgmap v2051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:54.170 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:41:54.169+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:41:54.170 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:41:54.169+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:41:54.170 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:41:54.169+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:41:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:55 smithi042 ceph-mon[32775]: pgmap v2052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:55.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:55 smithi164 ceph-mon[36970]: pgmap v2052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:55 smithi163 ceph-mon[36873]: pgmap v2052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:41:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2746.169691 secs 2024-07-26T07:41:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2763.912336 secs 2024-07-26T07:41:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2746.169691 secs 2024-07-26T07:41:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2763.912336 secs 2024-07-26T07:41:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2746.169691 secs 2024-07-26T07:41:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2763.912336 secs 2024-07-26T07:41:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:57 smithi042 ceph-mon[32775]: pgmap v2053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:57 smithi164 ceph-mon[36970]: pgmap v2053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:57 smithi163 ceph-mon[36873]: pgmap v2053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:58.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:41:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:41:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:41:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:59 smithi042 ceph-mon[32775]: pgmap v2054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:41:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:59 smithi042 ceph-mon[32775]: pgmap v2055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:41:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:59 smithi042 ceph-mon[32775]: pgmap v2056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:41:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:41:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:41:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:59 smithi164 ceph-mon[36970]: pgmap v2054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:41:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:41:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:41:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:59 smithi164 ceph-mon[36970]: pgmap v2055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:41:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:59 smithi164 ceph-mon[36970]: pgmap v2056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:41:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:41:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:42:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:59 smithi163 ceph-mon[36873]: pgmap v2054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:42:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:42:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:59 smithi163 ceph-mon[36873]: pgmap v2055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:42:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:59 smithi163 ceph-mon[36873]: pgmap v2056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:42:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:41:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:42:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2751.169814 secs 2024-07-26T07:42:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:01 smithi042 ceph-mon[32775]: pgmap v2057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:42:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2768.912484 secs 2024-07-26T07:42:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2751.169814 secs 2024-07-26T07:42:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:01 smithi164 ceph-mon[36970]: pgmap v2057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:42:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2768.912484 secs 2024-07-26T07:42:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2751.169814 secs 2024-07-26T07:42:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:01 smithi163 ceph-mon[36873]: pgmap v2057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:42:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2768.912484 secs 2024-07-26T07:42:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:03 smithi042 ceph-mon[32775]: pgmap v2058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:42:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:03 smithi164 ceph-mon[36970]: pgmap v2058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:42:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:03 smithi163 ceph-mon[36873]: pgmap v2058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:42:06.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:05 smithi042 ceph-mon[32775]: pgmap v2059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:42:06.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:05 smithi164 ceph-mon[36970]: pgmap v2059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:42:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:05 smithi163 ceph-mon[36873]: pgmap v2059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:42:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2756.169937 secs 2024-07-26T07:42:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2773.912611 secs 2024-07-26T07:42:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2756.169937 secs 2024-07-26T07:42:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2773.912611 secs 2024-07-26T07:42:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2756.169937 secs 2024-07-26T07:42:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2773.912611 secs 2024-07-26T07:42:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:07 smithi042 ceph-mon[32775]: pgmap v2060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:42:08.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:07 smithi164 ceph-mon[36970]: pgmap v2060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:42:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:07 smithi163 ceph-mon[36873]: pgmap v2060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:42:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:42:09.177+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:42:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:42:09.177+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:42:09.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:42:09.177+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:42:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:09 smithi042 ceph-mon[32775]: pgmap v2061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:42:10.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:09 smithi164 ceph-mon[36970]: pgmap v2061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:42:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:09 smithi163 ceph-mon[36873]: pgmap v2061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:42:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2761.170079 secs 2024-07-26T07:42:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:11 smithi042 ceph-mon[32775]: pgmap v2062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2778.912733 secs 2024-07-26T07:42:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2761.170079 secs 2024-07-26T07:42:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:11 smithi164 ceph-mon[36970]: pgmap v2062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2778.912733 secs 2024-07-26T07:42:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2761.170079 secs 2024-07-26T07:42:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:11 smithi163 ceph-mon[36873]: pgmap v2062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2778.912733 secs 2024-07-26T07:42:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:13 smithi042 ceph-mon[32775]: pgmap v2063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:14.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:13 smithi164 ceph-mon[36970]: pgmap v2063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:13 smithi163 ceph-mon[36873]: pgmap v2063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:15 smithi042 ceph-mon[32775]: pgmap v2064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:16.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:15 smithi164 ceph-mon[36970]: pgmap v2064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:15 smithi163 ceph-mon[36873]: pgmap v2064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2766.170169 secs 2024-07-26T07:42:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2783.912855 secs 2024-07-26T07:42:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2766.170169 secs 2024-07-26T07:42:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2783.912855 secs 2024-07-26T07:42:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2766.170169 secs 2024-07-26T07:42:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2783.912855 secs 2024-07-26T07:42:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:17 smithi042 ceph-mon[32775]: pgmap v2065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:18.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:17 smithi164 ceph-mon[36970]: pgmap v2065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:17 smithi163 ceph-mon[36873]: pgmap v2065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:19 smithi042 ceph-mon[32775]: pgmap v2066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:42:20.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:19 smithi164 ceph-mon[36970]: pgmap v2066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:42:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:19 smithi163 ceph-mon[36873]: pgmap v2066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:42:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2771.170290 secs 2024-07-26T07:42:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:21 smithi042 ceph-mon[32775]: pgmap v2067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2788.912987 secs 2024-07-26T07:42:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2771.170290 secs 2024-07-26T07:42:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:21 smithi164 ceph-mon[36970]: pgmap v2067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2788.912987 secs 2024-07-26T07:42:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2771.170290 secs 2024-07-26T07:42:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:21 smithi163 ceph-mon[36873]: pgmap v2067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2788.912987 secs 2024-07-26T07:42:24.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:23 smithi042 ceph-mon[32775]: pgmap v2068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:42:24.179+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:42:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:42:24.179+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:42:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:42:24.179+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:42:24.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:23 smithi164 ceph-mon[36970]: pgmap v2068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:23 smithi163 ceph-mon[36873]: pgmap v2068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:25 smithi042 ceph-mon[32775]: pgmap v2069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:26.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:25 smithi164 ceph-mon[36970]: pgmap v2069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:25 smithi163 ceph-mon[36873]: pgmap v2069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2776.170419 secs 2024-07-26T07:42:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2793.913071 secs 2024-07-26T07:42:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2776.170419 secs 2024-07-26T07:42:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2793.913071 secs 2024-07-26T07:42:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2776.170419 secs 2024-07-26T07:42:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2793.913071 secs 2024-07-26T07:42:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:27 smithi042 ceph-mon[32775]: pgmap v2070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:28.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:27 smithi164 ceph-mon[36970]: pgmap v2070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:27 smithi163 ceph-mon[36873]: pgmap v2070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:29 smithi042 ceph-mon[32775]: pgmap v2071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:42:30.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:29 smithi164 ceph-mon[36970]: pgmap v2071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:42:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:29 smithi163 ceph-mon[36873]: pgmap v2071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:42:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2781.170542 secs 2024-07-26T07:42:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:32 smithi042 ceph-mon[32775]: pgmap v2072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2798.913218 secs 2024-07-26T07:42:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2781.170542 secs 2024-07-26T07:42:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:32 smithi164 ceph-mon[36970]: pgmap v2072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2798.913218 secs 2024-07-26T07:42:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2781.170542 secs 2024-07-26T07:42:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:32 smithi163 ceph-mon[36873]: pgmap v2072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2798.913218 secs 2024-07-26T07:42:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:34 smithi042 ceph-mon[32775]: pgmap v2073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:34.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:34 smithi164 ceph-mon[36970]: pgmap v2073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:34 smithi163 ceph-mon[36873]: pgmap v2073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:36 smithi042 ceph-mon[32775]: pgmap v2074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:36 smithi164 ceph-mon[36970]: pgmap v2074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:36 smithi163 ceph-mon[36873]: pgmap v2074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2786.170661 secs 2024-07-26T07:42:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2803.913286 secs 2024-07-26T07:42:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2786.170661 secs 2024-07-26T07:42:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2803.913286 secs 2024-07-26T07:42:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2786.170661 secs 2024-07-26T07:42:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2803.913286 secs 2024-07-26T07:42:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:38 smithi042 ceph-mon[32775]: pgmap v2075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:38 smithi164 ceph-mon[36970]: pgmap v2075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:38 smithi163 ceph-mon[36873]: pgmap v2075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:39.187 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:42:39.187+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:42:39.188 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:42:39.187+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:42:39.188 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:42:39.187+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:42:40.394 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:40 smithi042 ceph-mon[32775]: pgmap v2076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:42:40.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:40 smithi164 ceph-mon[36970]: pgmap v2076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:42:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:40 smithi163 ceph-mon[36873]: pgmap v2076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:42:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2791.170752 secs 2024-07-26T07:42:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:42 smithi042 ceph-mon[32775]: pgmap v2077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2808.913408 secs 2024-07-26T07:42:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2791.170752 secs 2024-07-26T07:42:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:42 smithi164 ceph-mon[36970]: pgmap v2077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2808.913408 secs 2024-07-26T07:42:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2791.170752 secs 2024-07-26T07:42:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:42 smithi163 ceph-mon[36873]: pgmap v2077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2808.913408 secs 2024-07-26T07:42:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:44 smithi042 ceph-mon[32775]: pgmap v2078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:44.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:44 smithi164 ceph-mon[36970]: pgmap v2078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:44 smithi163 ceph-mon[36873]: pgmap v2078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:46 smithi042 ceph-mon[32775]: pgmap v2079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:46 smithi164 ceph-mon[36970]: pgmap v2079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:46 smithi163 ceph-mon[36873]: pgmap v2079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2796.170876 secs 2024-07-26T07:42:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2813.913489 secs 2024-07-26T07:42:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2796.170876 secs 2024-07-26T07:42:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2813.913489 secs 2024-07-26T07:42:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2796.170876 secs 2024-07-26T07:42:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2813.913489 secs 2024-07-26T07:42:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:48 smithi042 ceph-mon[32775]: pgmap v2080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:48.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:48 smithi164 ceph-mon[36970]: pgmap v2080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:48 smithi163 ceph-mon[36873]: pgmap v2080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:50 smithi042 ceph-mon[32775]: pgmap v2081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:42:50.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:50 smithi164 ceph-mon[36970]: pgmap v2081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:42:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:50 smithi163 ceph-mon[36873]: pgmap v2081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:42:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2801.171015 secs 2024-07-26T07:42:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:52 smithi042 ceph-mon[32775]: pgmap v2082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2818.913610 secs 2024-07-26T07:42:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2801.171015 secs 2024-07-26T07:42:52.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:52 smithi164 ceph-mon[36970]: pgmap v2082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2818.913610 secs 2024-07-26T07:42:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2801.171015 secs 2024-07-26T07:42:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:52 smithi163 ceph-mon[36873]: pgmap v2082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2818.913610 secs 2024-07-26T07:42:54.190 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:42:54.189+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:42:54.190 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:42:54.189+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:42:54.191 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:42:54.189+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:42:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:54 smithi042 ceph-mon[32775]: pgmap v2083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:54.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:54 smithi164 ceph-mon[36970]: pgmap v2083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:54 smithi163 ceph-mon[36873]: pgmap v2083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:56 smithi042 ceph-mon[32775]: pgmap v2084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:56.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:56 smithi164 ceph-mon[36970]: pgmap v2084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:56 smithi163 ceph-mon[36873]: pgmap v2084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:42:57.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2806.171138 secs 2024-07-26T07:42:57.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2823.913715 secs 2024-07-26T07:42:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2806.171138 secs 2024-07-26T07:42:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2823.913715 secs 2024-07-26T07:42:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2806.171138 secs 2024-07-26T07:42:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2823.913715 secs 2024-07-26T07:42:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:58 smithi042 ceph-mon[32775]: pgmap v2085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:58.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:58 smithi164 ceph-mon[36970]: pgmap v2085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:58 smithi163 ceph-mon[36873]: pgmap v2085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:42:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:42:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:42:59.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:42:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:42:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:42:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:43:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:00 smithi042 ceph-mon[32775]: pgmap v2086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:43:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:00 smithi042 ceph-mon[32775]: pgmap v2087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:43:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:00 smithi042 ceph-mon[32775]: pgmap v2088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:43:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:43:00.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:00 smithi164 ceph-mon[36970]: pgmap v2086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:00.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:00.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:43:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:00 smithi164 ceph-mon[36970]: pgmap v2087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:43:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:00 smithi164 ceph-mon[36970]: pgmap v2088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:43:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:43:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:00 smithi163 ceph-mon[36873]: pgmap v2086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:43:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:43:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:00 smithi163 ceph-mon[36873]: pgmap v2087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:43:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:00 smithi163 ceph-mon[36873]: pgmap v2088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:43:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:43:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2811.171260 secs 2024-07-26T07:43:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2811.171260 secs 2024-07-26T07:43:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2811.171260 secs 2024-07-26T07:43:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2828.913839 secs 2024-07-26T07:43:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:02 smithi042 ceph-mon[32775]: pgmap v2089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:43:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2828.913839 secs 2024-07-26T07:43:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:02 smithi164 ceph-mon[36970]: pgmap v2089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:43:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2828.913839 secs 2024-07-26T07:43:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:02 smithi163 ceph-mon[36873]: pgmap v2089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:43:04.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:04 smithi042 ceph-mon[32775]: pgmap v2090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:43:04.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:04 smithi164 ceph-mon[36970]: pgmap v2090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:43:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:04 smithi163 ceph-mon[36873]: pgmap v2090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:43:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:06 smithi042 ceph-mon[32775]: pgmap v2091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:43:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2816.171410 secs 2024-07-26T07:43:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:06 smithi164 ceph-mon[36970]: pgmap v2091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:43:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2816.171410 secs 2024-07-26T07:43:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:06 smithi163 ceph-mon[36873]: pgmap v2091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:43:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2816.171410 secs 2024-07-26T07:43:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2833.913964 secs 2024-07-26T07:43:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2833.913964 secs 2024-07-26T07:43:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2833.913964 secs 2024-07-26T07:43:08.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:08 smithi042 ceph-mon[32775]: pgmap v2092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:43:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:08 smithi164 ceph-mon[36970]: pgmap v2092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:43:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:08 smithi163 ceph-mon[36873]: pgmap v2092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:43:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:43:09.196+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:43:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:43:09.196+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:43:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:43:09.196+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:43:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:10 smithi042 ceph-mon[32775]: pgmap v2093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:43:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:10 smithi164 ceph-mon[36970]: pgmap v2093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:43:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:10 smithi163 ceph-mon[36873]: pgmap v2093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:43:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2821.171526 secs 2024-07-26T07:43:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2821.171526 secs 2024-07-26T07:43:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2821.171526 secs 2024-07-26T07:43:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2838.914084 secs 2024-07-26T07:43:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:12 smithi042 ceph-mon[32775]: pgmap v2094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2838.914084 secs 2024-07-26T07:43:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:12 smithi164 ceph-mon[36970]: pgmap v2094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2838.914084 secs 2024-07-26T07:43:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:12 smithi163 ceph-mon[36873]: pgmap v2094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:14 smithi042 ceph-mon[32775]: pgmap v2095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:14.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:14 smithi164 ceph-mon[36970]: pgmap v2095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:14 smithi163 ceph-mon[36873]: pgmap v2095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:16 smithi164 ceph-mon[36970]: pgmap v2096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2826.171651 secs 2024-07-26T07:43:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:16 smithi163 ceph-mon[36873]: pgmap v2096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2826.171651 secs 2024-07-26T07:43:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:16 smithi042 ceph-mon[32775]: pgmap v2096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2826.171651 secs 2024-07-26T07:43:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2843.914205 secs 2024-07-26T07:43:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2843.914205 secs 2024-07-26T07:43:17.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2843.914205 secs 2024-07-26T07:43:18.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:18 smithi164 ceph-mon[36970]: pgmap v2097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:18 smithi163 ceph-mon[36873]: pgmap v2097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:18 smithi042 ceph-mon[32775]: pgmap v2097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:20.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:20 smithi164 ceph-mon[36970]: pgmap v2098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:20 smithi163 ceph-mon[36873]: pgmap v2098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:20 smithi042 ceph-mon[32775]: pgmap v2098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2831.171772 secs 2024-07-26T07:43:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2831.171772 secs 2024-07-26T07:43:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2831.171772 secs 2024-07-26T07:43:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2848.914313 secs 2024-07-26T07:43:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:22 smithi164 ceph-mon[36970]: pgmap v2099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2848.914313 secs 2024-07-26T07:43:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:22 smithi163 ceph-mon[36873]: pgmap v2099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2848.914313 secs 2024-07-26T07:43:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:22 smithi042 ceph-mon[32775]: pgmap v2099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:43:24.199+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:43:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:43:24.199+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:43:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:43:24.199+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:43:24.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:24 smithi164 ceph-mon[36970]: pgmap v2100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:24 smithi163 ceph-mon[36873]: pgmap v2100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:24 smithi042 ceph-mon[32775]: pgmap v2100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:26 smithi164 ceph-mon[36970]: pgmap v2101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2836.171928 secs 2024-07-26T07:43:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:26 smithi163 ceph-mon[36873]: pgmap v2101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2836.171928 secs 2024-07-26T07:43:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:26 smithi042 ceph-mon[32775]: pgmap v2101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2836.171928 secs 2024-07-26T07:43:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2853.914414 secs 2024-07-26T07:43:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2853.914414 secs 2024-07-26T07:43:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2853.914414 secs 2024-07-26T07:43:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:28 smithi164 ceph-mon[36970]: pgmap v2102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:28 smithi163 ceph-mon[36873]: pgmap v2102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:28 smithi042 ceph-mon[32775]: pgmap v2102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:30.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:30 smithi164 ceph-mon[36970]: pgmap v2103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:30 smithi163 ceph-mon[36873]: pgmap v2103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:30 smithi042 ceph-mon[32775]: pgmap v2103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2841.172069 secs 2024-07-26T07:43:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2841.172069 secs 2024-07-26T07:43:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2841.172069 secs 2024-07-26T07:43:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2858.914481 secs 2024-07-26T07:43:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:32 smithi164 ceph-mon[36970]: pgmap v2104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2858.914481 secs 2024-07-26T07:43:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:32 smithi163 ceph-mon[36873]: pgmap v2104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2858.914481 secs 2024-07-26T07:43:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:32 smithi042 ceph-mon[32775]: pgmap v2104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:34 smithi164 ceph-mon[36970]: pgmap v2105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:34 smithi163 ceph-mon[36873]: pgmap v2105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:34 smithi042 ceph-mon[32775]: pgmap v2105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:36 smithi164 ceph-mon[36970]: pgmap v2106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2846.172192 secs 2024-07-26T07:43:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:36 smithi163 ceph-mon[36873]: pgmap v2106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2846.172192 secs 2024-07-26T07:43:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:36 smithi042 ceph-mon[32775]: pgmap v2106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2846.172192 secs 2024-07-26T07:43:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2863.914607 secs 2024-07-26T07:43:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2863.914607 secs 2024-07-26T07:43:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2863.914607 secs 2024-07-26T07:43:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:38 smithi164 ceph-mon[36970]: pgmap v2107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:38 smithi163 ceph-mon[36873]: pgmap v2107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:38 smithi042 ceph-mon[32775]: pgmap v2107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:39.207 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:43:39.206+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:43:39.207 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:43:39.206+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:43:39.208 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:43:39.206+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:43:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:40 smithi164 ceph-mon[36970]: pgmap v2108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:40 smithi163 ceph-mon[36873]: pgmap v2108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:40 smithi042 ceph-mon[32775]: pgmap v2108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2851.172324 secs 2024-07-26T07:43:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2851.172324 secs 2024-07-26T07:43:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2851.172324 secs 2024-07-26T07:43:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2868.914712 secs 2024-07-26T07:43:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:42 smithi164 ceph-mon[36970]: pgmap v2109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2868.914712 secs 2024-07-26T07:43:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:42 smithi163 ceph-mon[36873]: pgmap v2109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2868.914712 secs 2024-07-26T07:43:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:42 smithi042 ceph-mon[32775]: pgmap v2109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:44 smithi164 ceph-mon[36970]: pgmap v2110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:44 smithi163 ceph-mon[36873]: pgmap v2110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:44 smithi042 ceph-mon[32775]: pgmap v2110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:46 smithi163 ceph-mon[36873]: pgmap v2111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2856.172427 secs 2024-07-26T07:43:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:46 smithi042 ceph-mon[32775]: pgmap v2111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2856.172427 secs 2024-07-26T07:43:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:46 smithi164 ceph-mon[36970]: pgmap v2111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2856.172427 secs 2024-07-26T07:43:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2873.914842 secs 2024-07-26T07:43:47.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2873.914842 secs 2024-07-26T07:43:47.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2873.914842 secs 2024-07-26T07:43:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:48 smithi163 ceph-mon[36873]: pgmap v2112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:48.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:48 smithi042 ceph-mon[32775]: pgmap v2112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:48.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:48 smithi164 ceph-mon[36970]: pgmap v2112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:50 smithi163 ceph-mon[36873]: pgmap v2113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:50 smithi042 ceph-mon[32775]: pgmap v2113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:50 smithi164 ceph-mon[36970]: pgmap v2113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2861.172532 secs 2024-07-26T07:43:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2878.914967 secs 2024-07-26T07:43:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2861.172532 secs 2024-07-26T07:43:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2878.914967 secs 2024-07-26T07:43:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2861.172532 secs 2024-07-26T07:43:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2878.914967 secs 2024-07-26T07:43:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:52 smithi163 ceph-mon[36873]: pgmap v2114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:52.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:52 smithi164 ceph-mon[36970]: pgmap v2114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:52 smithi042 ceph-mon[32775]: pgmap v2114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:54.210 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:43:54.209+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:43:54.211 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:43:54.209+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:43:54.211 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:43:54.209+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:43:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:54 smithi163 ceph-mon[36873]: pgmap v2115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:54 smithi042 ceph-mon[32775]: pgmap v2115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:54 smithi164 ceph-mon[36970]: pgmap v2115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:43:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:56 smithi163 ceph-mon[36873]: pgmap v2116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2866.172661 secs 2024-07-26T07:43:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:56 smithi163 ceph-mon[36873]: 1 slow requests, 1 included below; oldest blocked for > 2883.915087 secs 2024-07-26T07:43:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:56 smithi163 ceph-mon[36873]: slow request 2883.915087 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T07:43:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:56 smithi042 ceph-mon[32775]: pgmap v2116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2866.172661 secs 2024-07-26T07:43:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:56 smithi042 ceph-mon[32775]: 1 slow requests, 1 included below; oldest blocked for > 2883.915087 secs 2024-07-26T07:43:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:56 smithi042 ceph-mon[32775]: slow request 2883.915087 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T07:43:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:56 smithi164 ceph-mon[36970]: pgmap v2116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:43:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2866.172661 secs 2024-07-26T07:43:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:56 smithi164 ceph-mon[36970]: 1 slow requests, 1 included below; oldest blocked for > 2883.915087 secs 2024-07-26T07:43:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:56 smithi164 ceph-mon[36970]: slow request 2883.915087 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T07:43:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:43:58.309+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:43:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:43:58.309+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:43:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:43:58.309+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:43:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:43:58.309+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:43:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:58 smithi163 ceph-mon[36873]: pgmap v2117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:58 smithi042 ceph-mon[32775]: pgmap v2117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:58 smithi164 ceph-mon[36970]: pgmap v2117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:43:59.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:43:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:43:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:43:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:44:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:43:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:44:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:00 smithi042 ceph-mon[32775]: pgmap v2118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:44:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:44:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:00 smithi164 ceph-mon[36970]: pgmap v2118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:44:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:44:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:00 smithi163 ceph-mon[36873]: pgmap v2118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:44:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:44:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:44:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:01 smithi042 ceph-mon[32775]: pgmap v2119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:44:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:01 smithi042 ceph-mon[32775]: pgmap v2120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:44:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2871.172767 secs 2024-07-26T07:44:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2888.915232 secs 2024-07-26T07:44:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:01 smithi164 ceph-mon[36970]: pgmap v2119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:44:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:01 smithi164 ceph-mon[36970]: pgmap v2120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:44:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2871.172767 secs 2024-07-26T07:44:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2888.915232 secs 2024-07-26T07:44:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:01 smithi163 ceph-mon[36873]: pgmap v2119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:44:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:01 smithi163 ceph-mon[36873]: pgmap v2120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:44:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2871.172767 secs 2024-07-26T07:44:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2888.915232 secs 2024-07-26T07:44:03.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:03 smithi042 ceph-mon[32775]: pgmap v2121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:44:03.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:03 smithi164 ceph-mon[36970]: pgmap v2121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:44:04.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:03 smithi163 ceph-mon[36873]: pgmap v2121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:44:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:05 smithi042 ceph-mon[32775]: pgmap v2122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:44:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:05 smithi164 ceph-mon[36970]: pgmap v2122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:44:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:05 smithi163 ceph-mon[36873]: pgmap v2122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:44:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2876.172895 secs 2024-07-26T07:44:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2893.915362 secs 2024-07-26T07:44:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2876.172895 secs 2024-07-26T07:44:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2893.915362 secs 2024-07-26T07:44:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2876.172895 secs 2024-07-26T07:44:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2893.915362 secs 2024-07-26T07:44:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:07 smithi042 ceph-mon[32775]: pgmap v2123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:44:07.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:07 smithi164 ceph-mon[36970]: pgmap v2123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:44:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:07 smithi163 ceph-mon[36873]: pgmap v2123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:44:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:44:09.214+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:44:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:44:09.214+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:44:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:44:09.214+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:44:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:09 smithi042 ceph-mon[32775]: pgmap v2124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:44:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:09 smithi164 ceph-mon[36970]: pgmap v2124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:44:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:09 smithi163 ceph-mon[36873]: pgmap v2124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:44:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:11 smithi042 ceph-mon[32775]: pgmap v2125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:44:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:11 smithi042 ceph-mon[32775]: 1 slow requests, 1 included below; oldest blocked for > 2881.173017 secs 2024-07-26T07:44:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:11 smithi042 ceph-mon[32775]: slow request 2881.173016 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T07:44:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2898.915487 secs 2024-07-26T07:44:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:11 smithi164 ceph-mon[36970]: pgmap v2125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:44:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:11 smithi164 ceph-mon[36970]: 1 slow requests, 1 included below; oldest blocked for > 2881.173017 secs 2024-07-26T07:44:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:11 smithi164 ceph-mon[36970]: slow request 2881.173016 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T07:44:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2898.915487 secs 2024-07-26T07:44:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:11 smithi163 ceph-mon[36873]: pgmap v2125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:44:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:11 smithi163 ceph-mon[36873]: 1 slow requests, 1 included below; oldest blocked for > 2881.173017 secs 2024-07-26T07:44:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:11 smithi163 ceph-mon[36873]: slow request 2881.173016 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T07:44:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2898.915487 secs 2024-07-26T07:44:13.808 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:13 smithi042 ceph-mon[32775]: pgmap v2126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:13.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:13 smithi164 ceph-mon[36970]: pgmap v2126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:13 smithi163 ceph-mon[36873]: pgmap v2126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:15 smithi042 ceph-mon[32775]: pgmap v2127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:44:15.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:15 smithi164 ceph-mon[36970]: pgmap v2127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:44:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:15 smithi163 ceph-mon[36873]: pgmap v2127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:44:16.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2886.173143 secs 2024-07-26T07:44:16.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2903.915633 secs 2024-07-26T07:44:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2886.173143 secs 2024-07-26T07:44:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2903.915633 secs 2024-07-26T07:44:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2886.173143 secs 2024-07-26T07:44:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2903.915633 secs 2024-07-26T07:44:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:17 smithi042 ceph-mon[32775]: pgmap v2128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:18.113 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:17 smithi163 ceph-mon[36873]: pgmap v2128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:18.113 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:17 smithi164 ceph-mon[36970]: pgmap v2128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:19.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:19 smithi042 ceph-mon[32775]: pgmap v2129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:19.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:19 smithi164 ceph-mon[36970]: pgmap v2129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:20.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:19 smithi163 ceph-mon[36873]: pgmap v2129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:21 smithi042 ceph-mon[32775]: pgmap v2130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2891.173268 secs 2024-07-26T07:44:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2908.915753 secs 2024-07-26T07:44:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:21 smithi164 ceph-mon[36970]: pgmap v2130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2891.173268 secs 2024-07-26T07:44:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2908.915753 secs 2024-07-26T07:44:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:21 smithi163 ceph-mon[36873]: pgmap v2130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2891.173268 secs 2024-07-26T07:44:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2908.915753 secs 2024-07-26T07:44:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:23 smithi042 ceph-mon[32775]: pgmap v2131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:23 smithi164 ceph-mon[36970]: pgmap v2131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:23 smithi163 ceph-mon[36873]: pgmap v2131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:44:24.218+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:44:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:44:24.218+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:44:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:44:24.218+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:44:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:25 smithi042 ceph-mon[32775]: pgmap v2132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:44:25.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:25 smithi164 ceph-mon[36970]: pgmap v2132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:44:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:25 smithi163 ceph-mon[36873]: pgmap v2132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:44:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2896.173401 secs 2024-07-26T07:44:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2913.915855 secs 2024-07-26T07:44:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2896.173401 secs 2024-07-26T07:44:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2913.915855 secs 2024-07-26T07:44:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2896.173401 secs 2024-07-26T07:44:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2913.915855 secs 2024-07-26T07:44:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:27 smithi042 ceph-mon[32775]: pgmap v2133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:27 smithi164 ceph-mon[36970]: pgmap v2133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:27 smithi163 ceph-mon[36873]: pgmap v2133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:29 smithi042 ceph-mon[32775]: pgmap v2134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:29.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:29 smithi164 ceph-mon[36970]: pgmap v2134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:29 smithi163 ceph-mon[36873]: pgmap v2134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:31 smithi042 ceph-mon[32775]: pgmap v2135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2901.173546 secs 2024-07-26T07:44:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2918.916003 secs 2024-07-26T07:44:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:31 smithi164 ceph-mon[36970]: pgmap v2135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2901.173546 secs 2024-07-26T07:44:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2918.916003 secs 2024-07-26T07:44:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:31 smithi163 ceph-mon[36873]: pgmap v2135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2901.173546 secs 2024-07-26T07:44:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2918.916003 secs 2024-07-26T07:44:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:33 smithi042 ceph-mon[32775]: pgmap v2136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:33.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:33 smithi164 ceph-mon[36970]: pgmap v2136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:34.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:33 smithi163 ceph-mon[36873]: pgmap v2136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:35.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:35 smithi042 ceph-mon[32775]: pgmap v2137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:44:35.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:35 smithi164 ceph-mon[36970]: pgmap v2137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:44:36.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:35 smithi163 ceph-mon[36873]: pgmap v2137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:44:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2906.173645 secs 2024-07-26T07:44:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2923.916123 secs 2024-07-26T07:44:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2906.173645 secs 2024-07-26T07:44:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2923.916123 secs 2024-07-26T07:44:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2906.173645 secs 2024-07-26T07:44:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2923.916123 secs 2024-07-26T07:44:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:37 smithi042 ceph-mon[32775]: pgmap v2138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:37 smithi164 ceph-mon[36970]: pgmap v2138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:37 smithi163 ceph-mon[36873]: pgmap v2138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:44:39.226+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:44:39.227 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:44:39.226+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:44:39.228 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:44:39.226+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:44:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:39 smithi042 ceph-mon[32775]: pgmap v2139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:39.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:39 smithi164 ceph-mon[36970]: pgmap v2139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:40.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:39 smithi163 ceph-mon[36873]: pgmap v2139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:41 smithi042 ceph-mon[32775]: pgmap v2140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2911.173784 secs 2024-07-26T07:44:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2928.916209 secs 2024-07-26T07:44:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:41 smithi164 ceph-mon[36970]: pgmap v2140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2911.173784 secs 2024-07-26T07:44:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2928.916209 secs 2024-07-26T07:44:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:41 smithi163 ceph-mon[36873]: pgmap v2140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2911.173784 secs 2024-07-26T07:44:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2928.916209 secs 2024-07-26T07:44:43.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:43 smithi042 ceph-mon[32775]: pgmap v2141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:43.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:43 smithi164 ceph-mon[36970]: pgmap v2141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:43 smithi163 ceph-mon[36873]: pgmap v2141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:45 smithi042 ceph-mon[32775]: pgmap v2142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:44:45.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:45 smithi164 ceph-mon[36970]: pgmap v2142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:44:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:45 smithi163 ceph-mon[36873]: pgmap v2142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:44:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2916.173934 secs 2024-07-26T07:44:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2933.916295 secs 2024-07-26T07:44:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2916.173934 secs 2024-07-26T07:44:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2933.916295 secs 2024-07-26T07:44:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2916.173934 secs 2024-07-26T07:44:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2933.916295 secs 2024-07-26T07:44:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:47 smithi042 ceph-mon[32775]: pgmap v2143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:47.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:47 smithi164 ceph-mon[36970]: pgmap v2143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:48.116 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:47 smithi163 ceph-mon[36873]: pgmap v2143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:49.907 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:49 smithi042 ceph-mon[32775]: pgmap v2144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:49.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:49 smithi164 ceph-mon[36970]: pgmap v2144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:49 smithi163 ceph-mon[36873]: pgmap v2144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:51 smithi042 ceph-mon[32775]: pgmap v2145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2921.174051 secs 2024-07-26T07:44:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2938.916435 secs 2024-07-26T07:44:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:51 smithi164 ceph-mon[36970]: pgmap v2145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2921.174051 secs 2024-07-26T07:44:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2938.916435 secs 2024-07-26T07:44:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:51 smithi163 ceph-mon[36873]: pgmap v2145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2921.174051 secs 2024-07-26T07:44:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2938.916435 secs 2024-07-26T07:44:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:53 smithi042 ceph-mon[32775]: pgmap v2146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:53.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:53 smithi164 ceph-mon[36970]: pgmap v2146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:53 smithi163 ceph-mon[36873]: pgmap v2146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:54.230 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:44:54.229+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:44:54.291 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:44:54.229+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:44:54.291 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:44:54.229+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:44:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:55 smithi042 ceph-mon[32775]: pgmap v2147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:44:55.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:55 smithi164 ceph-mon[36970]: pgmap v2147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:44:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:55 smithi163 ceph-mon[36873]: pgmap v2147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:44:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2926.174173 secs 2024-07-26T07:44:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2943.916565 secs 2024-07-26T07:44:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2926.174173 secs 2024-07-26T07:44:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2943.916565 secs 2024-07-26T07:44:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2926.174173 secs 2024-07-26T07:44:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2943.916565 secs 2024-07-26T07:44:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:57 smithi042 ceph-mon[32775]: pgmap v2148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:57 smithi164 ceph-mon[36970]: pgmap v2148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:57 smithi163 ceph-mon[36873]: pgmap v2148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:44:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:44:58 smithi164 ceph-mon[36970]: pgmap v2149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:59.145 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:44:58 smithi163 ceph-mon[36873]: pgmap v2149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:44:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:44:58 smithi042 ceph-mon[32775]: pgmap v2149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:00 smithi164 ceph-mon[36970]: pgmap v2150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:45:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:45:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:00 smithi164 ceph-mon[36970]: pgmap v2151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:45:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:00 smithi164 ceph-mon[36970]: pgmap v2152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:45:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:45:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:00 smithi163 ceph-mon[36873]: pgmap v2150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:45:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:45:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:00 smithi163 ceph-mon[36873]: pgmap v2151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:45:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:00 smithi163 ceph-mon[36873]: pgmap v2152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:45:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:45:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:00 smithi042 ceph-mon[32775]: pgmap v2150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:45:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:45:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:45:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:00 smithi042 ceph-mon[32775]: pgmap v2151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:45:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:00 smithi042 ceph-mon[32775]: pgmap v2152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:45:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:45:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2931.174287 secs 2024-07-26T07:45:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2948.916659 secs 2024-07-26T07:45:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2931.174287 secs 2024-07-26T07:45:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2948.916659 secs 2024-07-26T07:45:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2931.174287 secs 2024-07-26T07:45:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2948.916659 secs 2024-07-26T07:45:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:02 smithi164 ceph-mon[36970]: pgmap v2153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:45:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:02 smithi163 ceph-mon[36873]: pgmap v2153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:45:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:02 smithi042 ceph-mon[32775]: pgmap v2153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:45:05.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:05 smithi164 ceph-mon[36970]: pgmap v2154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:45:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:05 smithi163 ceph-mon[36873]: pgmap v2154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:45:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:05 smithi042 ceph-mon[32775]: pgmap v2154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:45:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2936.174416 secs 2024-07-26T07:45:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2936.174416 secs 2024-07-26T07:45:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2936.174416 secs 2024-07-26T07:45:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2953.916790 secs 2024-07-26T07:45:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:07 smithi164 ceph-mon[36970]: pgmap v2155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:45:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2953.916790 secs 2024-07-26T07:45:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:07 smithi163 ceph-mon[36873]: pgmap v2155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:45:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2953.916790 secs 2024-07-26T07:45:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:07 smithi042 ceph-mon[32775]: pgmap v2155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:45:09.238 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:45:09.237+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:45:09.238 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:45:09.237+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:45:09.238 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:45:09.237+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:45:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:09 smithi163 ceph-mon[36873]: pgmap v2156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:45:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:09 smithi042 ceph-mon[32775]: pgmap v2156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:45:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:09 smithi164 ceph-mon[36970]: pgmap v2156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:45:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:11 smithi163 ceph-mon[36873]: pgmap v2157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:45:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2941.174517 secs 2024-07-26T07:45:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:11 smithi042 ceph-mon[32775]: pgmap v2157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:45:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2941.174517 secs 2024-07-26T07:45:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:11 smithi164 ceph-mon[36970]: pgmap v2157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:45:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2941.174517 secs 2024-07-26T07:45:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2958.916895 secs 2024-07-26T07:45:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2958.916895 secs 2024-07-26T07:45:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2958.916895 secs 2024-07-26T07:45:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:13 smithi163 ceph-mon[36873]: pgmap v2158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:13 smithi042 ceph-mon[32775]: pgmap v2158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:13.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:13 smithi164 ceph-mon[36970]: pgmap v2158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:15 smithi163 ceph-mon[36873]: pgmap v2159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:15 smithi042 ceph-mon[32775]: pgmap v2159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:15.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:15 smithi164 ceph-mon[36970]: pgmap v2159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2946.174653 secs 2024-07-26T07:45:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2946.174653 secs 2024-07-26T07:45:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2946.174653 secs 2024-07-26T07:45:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2963.917015 secs 2024-07-26T07:45:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:17 smithi163 ceph-mon[36873]: pgmap v2160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2963.917015 secs 2024-07-26T07:45:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:17 smithi042 ceph-mon[32775]: pgmap v2160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:17.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2963.917015 secs 2024-07-26T07:45:17.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:17 smithi164 ceph-mon[36970]: pgmap v2160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:19 smithi163 ceph-mon[36873]: pgmap v2161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:45:19.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:19 smithi042 ceph-mon[32775]: pgmap v2161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:45:19.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:19 smithi164 ceph-mon[36970]: pgmap v2161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:45:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:21 smithi163 ceph-mon[36873]: pgmap v2162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2951.174797 secs 2024-07-26T07:45:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2968.917094 secs 2024-07-26T07:45:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:21 smithi042 ceph-mon[32775]: pgmap v2162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2951.174797 secs 2024-07-26T07:45:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2968.917094 secs 2024-07-26T07:45:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:21 smithi164 ceph-mon[36970]: pgmap v2162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2951.174797 secs 2024-07-26T07:45:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2968.917094 secs 2024-07-26T07:45:23.775 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:23 smithi164 ceph-mon[36970]: pgmap v2163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:23 smithi042 ceph-mon[32775]: pgmap v2163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:23 smithi163 ceph-mon[36873]: pgmap v2163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:24.241 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:45:24.240+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:45:24.241 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:45:24.240+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:45:24.241 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:45:24.240+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:45:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:25 smithi042 ceph-mon[32775]: pgmap v2164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:25.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:25 smithi164 ceph-mon[36970]: pgmap v2164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:25 smithi163 ceph-mon[36873]: pgmap v2164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2956.174944 secs 2024-07-26T07:45:27.404 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2973.917218 secs 2024-07-26T07:45:27.405 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2956.174944 secs 2024-07-26T07:45:27.405 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2973.917218 secs 2024-07-26T07:45:27.405 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2956.174944 secs 2024-07-26T07:45:27.405 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2973.917218 secs 2024-07-26T07:45:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:27 smithi042 ceph-mon[32775]: pgmap v2165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:27 smithi164 ceph-mon[36970]: pgmap v2165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:27 smithi163 ceph-mon[36873]: pgmap v2165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:28.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:45:28.324+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:45:28.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:45:28.324+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:45:28.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:45:28.324+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:45:28.326 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:45:28.324+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:45:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:29 smithi042 ceph-mon[32775]: pgmap v2166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:45:29.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:29 smithi164 ceph-mon[36970]: pgmap v2166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:45:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:29 smithi163 ceph-mon[36873]: pgmap v2166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:45:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:31 smithi042 ceph-mon[32775]: pgmap v2167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2961.175094 secs 2024-07-26T07:45:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2978.917362 secs 2024-07-26T07:45:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:31 smithi164 ceph-mon[36970]: pgmap v2167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2961.175094 secs 2024-07-26T07:45:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2978.917362 secs 2024-07-26T07:45:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:31 smithi163 ceph-mon[36873]: pgmap v2167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2961.175094 secs 2024-07-26T07:45:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2978.917362 secs 2024-07-26T07:45:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:33 smithi042 ceph-mon[32775]: pgmap v2168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:33.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:33 smithi164 ceph-mon[36970]: pgmap v2168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:35.797 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:33 smithi163 ceph-mon[36873]: pgmap v2168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:35.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:35 smithi042 ceph-mon[32775]: pgmap v2169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:35.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:35 smithi164 ceph-mon[36970]: pgmap v2169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:36.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:35 smithi163 ceph-mon[36873]: pgmap v2169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2966.175265 secs 2024-07-26T07:45:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2983.917432 secs 2024-07-26T07:45:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2966.175265 secs 2024-07-26T07:45:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2983.917432 secs 2024-07-26T07:45:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2966.175265 secs 2024-07-26T07:45:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2983.917432 secs 2024-07-26T07:45:37.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:37 smithi042 ceph-mon[32775]: pgmap v2170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:37 smithi164 ceph-mon[36970]: pgmap v2170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:37 smithi163 ceph-mon[36873]: pgmap v2170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:45:39.249+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:45:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:45:39.249+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:45:39.250 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:45:39.249+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:45:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:39 smithi042 ceph-mon[32775]: pgmap v2171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:45:39.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:39 smithi164 ceph-mon[36970]: pgmap v2171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:45:40.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:39 smithi163 ceph-mon[36873]: pgmap v2171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:45:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:41 smithi042 ceph-mon[32775]: pgmap v2172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:45:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2971.175405 secs 2024-07-26T07:45:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2988.917573 secs 2024-07-26T07:45:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:41 smithi164 ceph-mon[36970]: pgmap v2172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:45:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2971.175405 secs 2024-07-26T07:45:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2988.917573 secs 2024-07-26T07:45:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:41 smithi163 ceph-mon[36873]: pgmap v2172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:45:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2971.175405 secs 2024-07-26T07:45:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2988.917573 secs 2024-07-26T07:45:43.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:43 smithi042 ceph-mon[32775]: pgmap v2173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T07:45:43.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:43 smithi164 ceph-mon[36970]: pgmap v2173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T07:45:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:43 smithi163 ceph-mon[36873]: pgmap v2173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T07:45:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:45 smithi042 ceph-mon[32775]: pgmap v2174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:45:45.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:45 smithi164 ceph-mon[36970]: pgmap v2174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:45:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:45 smithi163 ceph-mon[36873]: pgmap v2174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:45:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2976.175543 secs 2024-07-26T07:45:49.788 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2993.917721 secs 2024-07-26T07:45:49.790 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:47 smithi042 ceph-mon[32775]: pgmap v2175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T07:45:49.791 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2976.175543 secs 2024-07-26T07:45:49.791 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2993.917721 secs 2024-07-26T07:45:49.791 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:47 smithi163 ceph-mon[36873]: pgmap v2175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T07:45:49.791 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2976.175543 secs 2024-07-26T07:45:49.791 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2993.917721 secs 2024-07-26T07:45:49.792 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:47 smithi164 ceph-mon[36970]: pgmap v2175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T07:45:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:49 smithi042 ceph-mon[32775]: pgmap v2176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T07:45:49.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:49 smithi164 ceph-mon[36970]: pgmap v2176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T07:45:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:49 smithi163 ceph-mon[36873]: pgmap v2176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T07:45:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:51 smithi042 ceph-mon[32775]: pgmap v2177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:45:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2981.175663 secs 2024-07-26T07:45:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2998.917833 secs 2024-07-26T07:45:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:51 smithi164 ceph-mon[36970]: pgmap v2177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:45:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2981.175663 secs 2024-07-26T07:45:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2998.917833 secs 2024-07-26T07:45:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:51 smithi163 ceph-mon[36873]: pgmap v2177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:45:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2981.175663 secs 2024-07-26T07:45:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2998.917833 secs 2024-07-26T07:45:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:53 smithi042 ceph-mon[32775]: pgmap v2178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:53.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:53 smithi164 ceph-mon[36970]: pgmap v2178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:53 smithi163 ceph-mon[36873]: pgmap v2178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:45:54.252+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:45:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:45:54.252+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:45:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:45:54.252+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:45:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:55 smithi042 ceph-mon[32775]: pgmap v2179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:55.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:55 smithi164 ceph-mon[36970]: pgmap v2179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:55 smithi163 ceph-mon[36873]: pgmap v2179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:45:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2986.175792 secs 2024-07-26T07:45:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3003.917945 secs 2024-07-26T07:45:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2986.175792 secs 2024-07-26T07:45:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3003.917945 secs 2024-07-26T07:45:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2986.175792 secs 2024-07-26T07:45:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3003.917945 secs 2024-07-26T07:45:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:57 smithi042 ceph-mon[32775]: pgmap v2180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:57 smithi164 ceph-mon[36970]: pgmap v2180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:57 smithi163 ceph-mon[36873]: pgmap v2180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:45:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:45:59 smithi042 ceph-mon[32775]: pgmap v2181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:45:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:45:59 smithi164 ceph-mon[36970]: pgmap v2181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:46:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:45:59 smithi163 ceph-mon[36873]: pgmap v2181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:46:00.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:46:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:46:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:46:01.639 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:01 smithi042 ceph-mon[32775]: pgmap v2182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:01.640 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2991.175902 secs 2024-07-26T07:46:01.640 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3008.918066 secs 2024-07-26T07:46:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:01 smithi164 ceph-mon[36970]: pgmap v2182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2991.175902 secs 2024-07-26T07:46:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3008.918066 secs 2024-07-26T07:46:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:01 smithi163 ceph-mon[36873]: pgmap v2182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2991.175902 secs 2024-07-26T07:46:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3008.918066 secs 2024-07-26T07:46:03.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:03 smithi042 ceph-mon[32775]: pgmap v2183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:03.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:03 smithi164 ceph-mon[36970]: pgmap v2183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:04.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:03 smithi163 ceph-mon[36873]: pgmap v2183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:46:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:46:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:46:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:05 smithi042 ceph-mon[32775]: pgmap v2184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:05 smithi042 ceph-mon[32775]: pgmap v2185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:46:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:46:05.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:46:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:46:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:46:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:05 smithi164 ceph-mon[36970]: pgmap v2184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:05 smithi164 ceph-mon[36970]: pgmap v2185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:46:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:46:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:46:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:46:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:46:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:46:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:05 smithi163 ceph-mon[36873]: pgmap v2184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:05 smithi163 ceph-mon[36873]: pgmap v2185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:46:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:46:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 2996.176035 secs 2024-07-26T07:46:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 2996.176035 secs 2024-07-26T07:46:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 2996.176035 secs 2024-07-26T07:46:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3013.918164 secs 2024-07-26T07:46:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:07 smithi042 ceph-mon[32775]: pgmap v2186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:46:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3013.918164 secs 2024-07-26T07:46:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:07 smithi164 ceph-mon[36970]: pgmap v2186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:46:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3013.918164 secs 2024-07-26T07:46:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:07 smithi163 ceph-mon[36873]: pgmap v2186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:46:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:46:09.254+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:46:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:46:09.254+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:46:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:46:09.254+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:46:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:09 smithi042 ceph-mon[32775]: pgmap v2187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:46:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:09 smithi164 ceph-mon[36970]: pgmap v2187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:46:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:09 smithi163 ceph-mon[36873]: pgmap v2187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:46:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:11 smithi042 ceph-mon[32775]: pgmap v2188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:46:11.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:11 smithi164 ceph-mon[36970]: pgmap v2188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:46:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:11 smithi163 ceph-mon[36873]: pgmap v2188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:46:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3001.176137 secs 2024-07-26T07:46:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3018.918271 secs 2024-07-26T07:46:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3001.176137 secs 2024-07-26T07:46:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3018.918271 secs 2024-07-26T07:46:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3001.176137 secs 2024-07-26T07:46:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3018.918271 secs 2024-07-26T07:46:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:13 smithi042 ceph-mon[32775]: pgmap v2189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:46:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:13 smithi164 ceph-mon[36970]: pgmap v2189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:46:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:13 smithi163 ceph-mon[36873]: pgmap v2189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:46:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:15 smithi042 ceph-mon[32775]: pgmap v2190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:46:15.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:15 smithi164 ceph-mon[36970]: pgmap v2190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:46:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:15 smithi163 ceph-mon[36873]: pgmap v2190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:46:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3006.176302 secs 2024-07-26T07:46:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3006.176302 secs 2024-07-26T07:46:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3006.176302 secs 2024-07-26T07:46:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3023.918372 secs 2024-07-26T07:46:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:17 smithi042 ceph-mon[32775]: pgmap v2191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3023.918372 secs 2024-07-26T07:46:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:17 smithi164 ceph-mon[36970]: pgmap v2191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3023.918372 secs 2024-07-26T07:46:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:17 smithi163 ceph-mon[36873]: pgmap v2191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:19.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:19 smithi042 ceph-mon[32775]: pgmap v2192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:46:19.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:19 smithi164 ceph-mon[36970]: pgmap v2192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:46:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:19 smithi163 ceph-mon[36873]: pgmap v2192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:46:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:21 smithi042 ceph-mon[32775]: pgmap v2193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3011.176431 secs 2024-07-26T07:46:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:21 smithi164 ceph-mon[36970]: pgmap v2193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3011.176431 secs 2024-07-26T07:46:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:21 smithi163 ceph-mon[36873]: pgmap v2193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3011.176431 secs 2024-07-26T07:46:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3028.918515 secs 2024-07-26T07:46:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3028.918515 secs 2024-07-26T07:46:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3028.918515 secs 2024-07-26T07:46:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:23 smithi042 ceph-mon[32775]: pgmap v2194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:23.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:23 smithi163 ceph-mon[36873]: pgmap v2194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:23.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:23 smithi164 ceph-mon[36970]: pgmap v2194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:46:24.257+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:46:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:46:24.257+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:46:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:46:24.257+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:46:25.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:25 smithi042 ceph-mon[32775]: pgmap v2195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:25.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:25 smithi164 ceph-mon[36970]: pgmap v2195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:25.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:25 smithi163 ceph-mon[36873]: pgmap v2195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3016.176552 secs 2024-07-26T07:46:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3016.176552 secs 2024-07-26T07:46:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3016.176552 secs 2024-07-26T07:46:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3033.918656 secs 2024-07-26T07:46:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:27 smithi164 ceph-mon[36970]: pgmap v2196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3033.918656 secs 2024-07-26T07:46:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:27 smithi163 ceph-mon[36873]: pgmap v2196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:27.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3033.918656 secs 2024-07-26T07:46:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:27 smithi042 ceph-mon[32775]: pgmap v2196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:29.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:29 smithi164 ceph-mon[36970]: pgmap v2197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:46:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:29 smithi163 ceph-mon[36873]: pgmap v2197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:46:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:29 smithi042 ceph-mon[32775]: pgmap v2197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:46:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:31 smithi164 ceph-mon[36970]: pgmap v2198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3021.176673 secs 2024-07-26T07:46:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:31 smithi163 ceph-mon[36873]: pgmap v2198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3021.176673 secs 2024-07-26T07:46:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:31 smithi042 ceph-mon[32775]: pgmap v2198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:31.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3021.176673 secs 2024-07-26T07:46:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3038.918762 secs 2024-07-26T07:46:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3038.918762 secs 2024-07-26T07:46:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3038.918762 secs 2024-07-26T07:46:33.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:33 smithi164 ceph-mon[36970]: pgmap v2199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:33 smithi163 ceph-mon[36873]: pgmap v2199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:33 smithi042 ceph-mon[32775]: pgmap v2199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:35.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:35 smithi164 ceph-mon[36970]: pgmap v2200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:35.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:35 smithi163 ceph-mon[36873]: pgmap v2200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:35.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:35 smithi042 ceph-mon[32775]: pgmap v2200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3026.176839 secs 2024-07-26T07:46:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3026.176839 secs 2024-07-26T07:46:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3026.176839 secs 2024-07-26T07:46:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3043.918906 secs 2024-07-26T07:46:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:37 smithi164 ceph-mon[36970]: pgmap v2201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3043.918906 secs 2024-07-26T07:46:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:37 smithi163 ceph-mon[36873]: pgmap v2201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3043.918906 secs 2024-07-26T07:46:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:37 smithi042 ceph-mon[32775]: pgmap v2201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:39.263 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:46:39.262+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:46:39.263 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:46:39.262+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:46:39.263 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:46:39.262+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:46:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:39 smithi164 ceph-mon[36970]: pgmap v2202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:46:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:39 smithi163 ceph-mon[36873]: pgmap v2202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:46:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:39 smithi042 ceph-mon[32775]: pgmap v2202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:46:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:41 smithi164 ceph-mon[36970]: pgmap v2203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3031.176971 secs 2024-07-26T07:46:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:41 smithi163 ceph-mon[36873]: pgmap v2203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3031.176971 secs 2024-07-26T07:46:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:41 smithi042 ceph-mon[32775]: pgmap v2203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3031.176971 secs 2024-07-26T07:46:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3048.919026 secs 2024-07-26T07:46:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3048.919026 secs 2024-07-26T07:46:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3048.919026 secs 2024-07-26T07:46:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:43 smithi164 ceph-mon[36970]: pgmap v2204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:43 smithi163 ceph-mon[36873]: pgmap v2204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:43.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:43 smithi042 ceph-mon[32775]: pgmap v2204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:45 smithi164 ceph-mon[36970]: pgmap v2205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:45.906 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:45 smithi042 ceph-mon[32775]: pgmap v2205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:45.906 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:45 smithi163 ceph-mon[36873]: pgmap v2205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3036.177105 secs 2024-07-26T07:46:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3036.177105 secs 2024-07-26T07:46:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3036.177105 secs 2024-07-26T07:46:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3053.919149 secs 2024-07-26T07:46:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:47 smithi164 ceph-mon[36970]: pgmap v2206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3053.919149 secs 2024-07-26T07:46:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:47 smithi163 ceph-mon[36873]: pgmap v2206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:47.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3053.919149 secs 2024-07-26T07:46:47.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:47 smithi042 ceph-mon[32775]: pgmap v2206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:49 smithi164 ceph-mon[36970]: pgmap v2207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:46:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:49 smithi163 ceph-mon[36873]: pgmap v2207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:46:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:49 smithi042 ceph-mon[32775]: pgmap v2207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:46:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:51 smithi164 ceph-mon[36970]: pgmap v2208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3041.177255 secs 2024-07-26T07:46:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:51 smithi163 ceph-mon[36873]: pgmap v2208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3041.177255 secs 2024-07-26T07:46:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:51 smithi042 ceph-mon[32775]: pgmap v2208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3041.177255 secs 2024-07-26T07:46:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3058.919256 secs 2024-07-26T07:46:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3058.919256 secs 2024-07-26T07:46:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3058.919256 secs 2024-07-26T07:46:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:53 smithi164 ceph-mon[36970]: pgmap v2209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:53 smithi163 ceph-mon[36873]: pgmap v2209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:53 smithi042 ceph-mon[32775]: pgmap v2209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:46:54.265+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:46:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:46:54.265+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:46:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:46:54.265+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:46:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:55 smithi164 ceph-mon[36970]: pgmap v2210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:55 smithi163 ceph-mon[36873]: pgmap v2210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:55 smithi042 ceph-mon[32775]: pgmap v2210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:46:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3046.177392 secs 2024-07-26T07:46:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3046.177392 secs 2024-07-26T07:46:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3046.177392 secs 2024-07-26T07:46:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3063.919382 secs 2024-07-26T07:46:57.817 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:57 smithi164 ceph-mon[36970]: pgmap v2211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:57.818 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3063.919382 secs 2024-07-26T07:46:57.818 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:57 smithi163 ceph-mon[36873]: pgmap v2211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3063.919382 secs 2024-07-26T07:46:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:57 smithi042 ceph-mon[32775]: pgmap v2211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:46:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:46:58.339+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:46:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:46:58.339+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:46:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:46:58.339+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:46:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:46:58.339+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:46:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:46:59 smithi163 ceph-mon[36873]: pgmap v2212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:46:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:46:59 smithi042 ceph-mon[32775]: pgmap v2212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:46:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:46:59 smithi164 ceph-mon[36970]: pgmap v2212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:47:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:01 smithi163 ceph-mon[36873]: pgmap v2213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3051.177497 secs 2024-07-26T07:47:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:01 smithi042 ceph-mon[32775]: pgmap v2213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3051.177497 secs 2024-07-26T07:47:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:01 smithi164 ceph-mon[36970]: pgmap v2213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3051.177497 secs 2024-07-26T07:47:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3068.919485 secs 2024-07-26T07:47:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3068.919485 secs 2024-07-26T07:47:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3068.919485 secs 2024-07-26T07:47:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:03 smithi163 ceph-mon[36873]: pgmap v2214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:03.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:03 smithi042 ceph-mon[32775]: pgmap v2214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:03.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:03 smithi164 ceph-mon[36970]: pgmap v2214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:47:04.816 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:47:04.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:47:05.541 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:05 smithi163 ceph-mon[36873]: pgmap v2215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:05.541 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:05.541 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:05.541 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:05.541 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:05.610 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:05 smithi164 ceph-mon[36970]: pgmap v2215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:05.610 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:05.610 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:05.610 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:05.610 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:05 smithi042 ceph-mon[32775]: pgmap v2215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:06.595 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:06 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:47:06.595 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:06 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:47:06.595 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3056.177611 secs 2024-07-26T07:47:06.602 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:06 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:47:06.602 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:06 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:47:06.602 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3056.177611 secs 2024-07-26T07:47:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:06 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:47:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:06 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:47:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3056.177611 secs 2024-07-26T07:47:07.699 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3073.919607 secs 2024-07-26T07:47:07.699 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:07 smithi163 ceph-mon[36873]: pgmap v2216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:07.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3073.919607 secs 2024-07-26T07:47:07.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:07 smithi164 ceph-mon[36970]: pgmap v2216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3073.919607 secs 2024-07-26T07:47:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:07 smithi042 ceph-mon[32775]: pgmap v2216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:47:09.271+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:47:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:47:09.271+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:47:09.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:47:09.271+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:47:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:47:09.750 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:47:09.752 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.753 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.754 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.755 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.757 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:47:09.757 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:47:09.757 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:47:09.757 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:47:09.757 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.757 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.757 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.757 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.757 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:47:09.757 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:47:09.757 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:09.758 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:47:09.758 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: pgmap v2217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:09.758 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: pgmap v2218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:47:09.758 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:47:09.759 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:47:09.759 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:47:09.759 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.759 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.759 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.759 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.759 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:47:09.759 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:47:09.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:47:09.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:47:09.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:47:09.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:47:09.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:09.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:47:09.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: pgmap v2217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:09.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: pgmap v2218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:47:09.760 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:47:09.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:47:09.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:47:09.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:47:09.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:47:09.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:47:09.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:47:09.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.762 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:47:09.762 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:47:09.762 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:47:09.762 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:47:09.762 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:47:09.762 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: pgmap v2217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:09.762 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: pgmap v2218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:47:09.762 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:47:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:11 smithi042 ceph-mon[32775]: pgmap v2219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:47:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3061.177738 secs 2024-07-26T07:47:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:11 smithi164 ceph-mon[36970]: pgmap v2219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:47:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3061.177738 secs 2024-07-26T07:47:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:11 smithi163 ceph-mon[36873]: pgmap v2219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:47:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3061.177738 secs 2024-07-26T07:47:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3078.919751 secs 2024-07-26T07:47:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3078.919751 secs 2024-07-26T07:47:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3078.919751 secs 2024-07-26T07:47:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:13 smithi042 ceph-mon[32775]: pgmap v2220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:47:13.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:13 smithi164 ceph-mon[36970]: pgmap v2220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:47:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:13 smithi163 ceph-mon[36873]: pgmap v2220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:47:15.615 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:15 smithi042 ceph-mon[32775]: pgmap v2221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:47:15.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:15 smithi164 ceph-mon[36970]: pgmap v2221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:47:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:15 smithi163 ceph-mon[36873]: pgmap v2221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:47:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3066.177875 secs 2024-07-26T07:47:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3066.177875 secs 2024-07-26T07:47:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3066.177875 secs 2024-07-26T07:47:17.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3083.919852 secs 2024-07-26T07:47:17.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:17 smithi042 ceph-mon[32775]: pgmap v2222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:47:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3083.919852 secs 2024-07-26T07:47:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:17 smithi164 ceph-mon[36970]: pgmap v2222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:47:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3083.919852 secs 2024-07-26T07:47:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:17 smithi163 ceph-mon[36873]: pgmap v2222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T07:47:19.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:19 smithi042 ceph-mon[32775]: pgmap v2223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:47:19.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:19 smithi164 ceph-mon[36970]: pgmap v2223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:47:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:19 smithi163 ceph-mon[36873]: pgmap v2223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:47:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:21 smithi042 ceph-mon[32775]: pgmap v2224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3071.178011 secs 2024-07-26T07:47:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:21 smithi164 ceph-mon[36970]: pgmap v2224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3071.178011 secs 2024-07-26T07:47:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:21 smithi163 ceph-mon[36873]: pgmap v2224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3071.178011 secs 2024-07-26T07:47:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3088.919998 secs 2024-07-26T07:47:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3088.919998 secs 2024-07-26T07:47:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3088.919998 secs 2024-07-26T07:47:23.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:23 smithi164 ceph-mon[36970]: pgmap v2225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:23 smithi042 ceph-mon[32775]: pgmap v2225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:23.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:23 smithi163 ceph-mon[36873]: pgmap v2225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:47:24.275+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:47:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:47:24.275+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:47:24.276 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:47:24.275+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:47:25.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:25 smithi042 ceph-mon[32775]: pgmap v2226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:25.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:25 smithi164 ceph-mon[36970]: pgmap v2226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:25.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:25 smithi163 ceph-mon[36873]: pgmap v2226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3076.178158 secs 2024-07-26T07:47:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3076.178158 secs 2024-07-26T07:47:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3076.178158 secs 2024-07-26T07:47:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3093.920126 secs 2024-07-26T07:47:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:27 smithi042 ceph-mon[32775]: pgmap v2227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3093.920126 secs 2024-07-26T07:47:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:27 smithi164 ceph-mon[36970]: pgmap v2227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3093.920126 secs 2024-07-26T07:47:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:27 smithi163 ceph-mon[36873]: pgmap v2227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:29 smithi042 ceph-mon[32775]: pgmap v2228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:29.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:29 smithi164 ceph-mon[36970]: pgmap v2228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:29 smithi163 ceph-mon[36873]: pgmap v2228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:31 smithi042 ceph-mon[32775]: pgmap v2229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3081.178294 secs 2024-07-26T07:47:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:31 smithi164 ceph-mon[36970]: pgmap v2229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3081.178294 secs 2024-07-26T07:47:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:31 smithi163 ceph-mon[36873]: pgmap v2229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3081.178294 secs 2024-07-26T07:47:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3098.920224 secs 2024-07-26T07:47:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3098.920224 secs 2024-07-26T07:47:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3098.920224 secs 2024-07-26T07:47:33.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:33 smithi042 ceph-mon[32775]: pgmap v2230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:33.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:33 smithi164 ceph-mon[36970]: pgmap v2230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:33 smithi163 ceph-mon[36873]: pgmap v2230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:35.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:35 smithi042 ceph-mon[32775]: pgmap v2231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:35.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:35 smithi164 ceph-mon[36970]: pgmap v2231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:35.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:35 smithi163 ceph-mon[36873]: pgmap v2231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3086.178416 secs 2024-07-26T07:47:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3086.178416 secs 2024-07-26T07:47:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3086.178416 secs 2024-07-26T07:47:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3103.920310 secs 2024-07-26T07:47:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:37 smithi042 ceph-mon[32775]: pgmap v2232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3103.920310 secs 2024-07-26T07:47:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:37 smithi164 ceph-mon[36970]: pgmap v2232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3103.920310 secs 2024-07-26T07:47:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:37 smithi163 ceph-mon[36873]: pgmap v2232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:47:39.283+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:47:39.284 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:47:39.283+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:47:39.285 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:47:39.283+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:47:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:39 smithi042 ceph-mon[32775]: pgmap v2233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:39 smithi164 ceph-mon[36970]: pgmap v2233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:39 smithi163 ceph-mon[36873]: pgmap v2233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:41 smithi042 ceph-mon[32775]: pgmap v2234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3091.178532 secs 2024-07-26T07:47:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:41 smithi164 ceph-mon[36970]: pgmap v2234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3091.178532 secs 2024-07-26T07:47:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:41 smithi163 ceph-mon[36873]: pgmap v2234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3091.178532 secs 2024-07-26T07:47:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3108.920458 secs 2024-07-26T07:47:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3108.920458 secs 2024-07-26T07:47:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3108.920458 secs 2024-07-26T07:47:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:43 smithi042 ceph-mon[32775]: pgmap v2235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:43 smithi164 ceph-mon[36970]: pgmap v2235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:43 smithi163 ceph-mon[36873]: pgmap v2235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:45 smithi042 ceph-mon[32775]: pgmap v2236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:45.748 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:45 smithi164 ceph-mon[36970]: pgmap v2236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:45 smithi163 ceph-mon[36873]: pgmap v2236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3096.178653 secs 2024-07-26T07:47:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3096.178653 secs 2024-07-26T07:47:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3096.178653 secs 2024-07-26T07:47:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3113.920590 secs 2024-07-26T07:47:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:47 smithi042 ceph-mon[32775]: pgmap v2237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3113.920590 secs 2024-07-26T07:47:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:47 smithi164 ceph-mon[36970]: pgmap v2237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3113.920590 secs 2024-07-26T07:47:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:47 smithi163 ceph-mon[36873]: pgmap v2237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:49 smithi042 ceph-mon[32775]: pgmap v2238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:49 smithi164 ceph-mon[36970]: pgmap v2238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:49 smithi163 ceph-mon[36873]: pgmap v2238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:51 smithi042 ceph-mon[32775]: pgmap v2239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3101.178776 secs 2024-07-26T07:47:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:51 smithi164 ceph-mon[36970]: pgmap v2239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3101.178776 secs 2024-07-26T07:47:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:51 smithi163 ceph-mon[36873]: pgmap v2239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3101.178776 secs 2024-07-26T07:47:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3118.920712 secs 2024-07-26T07:47:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3118.920712 secs 2024-07-26T07:47:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3118.920712 secs 2024-07-26T07:47:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:53 smithi042 ceph-mon[32775]: pgmap v2240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:53.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:53 smithi163 ceph-mon[36873]: pgmap v2240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:53 smithi164 ceph-mon[36970]: pgmap v2240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:47:54.286+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:47:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:47:54.286+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:47:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:47:54.286+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:47:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:55 smithi042 ceph-mon[32775]: pgmap v2241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:55 smithi164 ceph-mon[36970]: pgmap v2241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:55 smithi163 ceph-mon[36873]: pgmap v2241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:47:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3106.178921 secs 2024-07-26T07:47:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3106.178921 secs 2024-07-26T07:47:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3106.178921 secs 2024-07-26T07:47:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3123.920854 secs 2024-07-26T07:47:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:57 smithi164 ceph-mon[36970]: pgmap v2242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3123.920854 secs 2024-07-26T07:47:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:57 smithi163 ceph-mon[36873]: pgmap v2242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3123.920854 secs 2024-07-26T07:47:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:57 smithi042 ceph-mon[32775]: pgmap v2242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:59.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:47:59 smithi164 ceph-mon[36970]: pgmap v2243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:47:59 smithi163 ceph-mon[36873]: pgmap v2243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:47:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:47:59 smithi042 ceph-mon[32775]: pgmap v2243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:01 smithi164 ceph-mon[36970]: pgmap v2244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3111.179092 secs 2024-07-26T07:48:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:01 smithi163 ceph-mon[36873]: pgmap v2244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3111.179092 secs 2024-07-26T07:48:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:01 smithi042 ceph-mon[32775]: pgmap v2244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3111.179092 secs 2024-07-26T07:48:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3128.921013 secs 2024-07-26T07:48:02.765 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3128.921013 secs 2024-07-26T07:48:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3128.921013 secs 2024-07-26T07:48:03.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:03 smithi164 ceph-mon[36970]: pgmap v2245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:03 smithi163 ceph-mon[36873]: pgmap v2245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:03.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:03 smithi042 ceph-mon[32775]: pgmap v2245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:05.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:05 smithi164 ceph-mon[36970]: pgmap v2246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:05 smithi163 ceph-mon[36873]: pgmap v2246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:05 smithi042 ceph-mon[32775]: pgmap v2246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3116.179261 secs 2024-07-26T07:48:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3116.179261 secs 2024-07-26T07:48:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3116.179261 secs 2024-07-26T07:48:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3133.921138 secs 2024-07-26T07:48:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:07 smithi042 ceph-mon[32775]: pgmap v2247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3133.921138 secs 2024-07-26T07:48:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:07 smithi164 ceph-mon[36970]: pgmap v2247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3133.921138 secs 2024-07-26T07:48:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:07 smithi163 ceph-mon[36873]: pgmap v2247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:08 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:48:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:08 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:48:08.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:08 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:48:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:48:09.294+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:48:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:48:09.294+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:48:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:48:09.294+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:48:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:09 smithi164 ceph-mon[36970]: pgmap v2248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:48:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:48:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:48:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:09 smithi163 ceph-mon[36873]: pgmap v2248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:48:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:48:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:48:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:09 smithi042 ceph-mon[32775]: pgmap v2248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:48:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:48:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:48:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:10 smithi164 ceph-mon[36970]: pgmap v2249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:48:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:10 smithi164 ceph-mon[36970]: pgmap v2250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:48:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:10 smithi163 ceph-mon[36873]: pgmap v2249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:48:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:10 smithi163 ceph-mon[36873]: pgmap v2250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:48:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:10 smithi042 ceph-mon[32775]: pgmap v2249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:48:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:10 smithi042 ceph-mon[32775]: pgmap v2250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:48:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3121.179393 secs 2024-07-26T07:48:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3121.179393 secs 2024-07-26T07:48:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3121.179393 secs 2024-07-26T07:48:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:12 smithi164 ceph-mon[36970]: pgmap v2251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:48:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3138.921260 secs 2024-07-26T07:48:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:12 smithi163 ceph-mon[36873]: pgmap v2251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:48:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3138.921260 secs 2024-07-26T07:48:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:12 smithi042 ceph-mon[32775]: pgmap v2251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:48:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3138.921260 secs 2024-07-26T07:48:14.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:14 smithi164 ceph-mon[36970]: pgmap v2252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:48:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:14 smithi163 ceph-mon[36873]: pgmap v2252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:48:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:14 smithi042 ceph-mon[32775]: pgmap v2252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:48:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:16 smithi164 ceph-mon[36970]: pgmap v2253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:48:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3126.179481 secs 2024-07-26T07:48:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:16 smithi163 ceph-mon[36873]: pgmap v2253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:48:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3126.179481 secs 2024-07-26T07:48:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:16 smithi042 ceph-mon[32775]: pgmap v2253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:48:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3126.179481 secs 2024-07-26T07:48:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3143.921359 secs 2024-07-26T07:48:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3143.921359 secs 2024-07-26T07:48:17.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3143.921359 secs 2024-07-26T07:48:18.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:18 smithi164 ceph-mon[36970]: pgmap v2254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:48:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:18 smithi163 ceph-mon[36873]: pgmap v2254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:48:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:18 smithi042 ceph-mon[32775]: pgmap v2254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:48:20.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:20 smithi164 ceph-mon[36970]: pgmap v2255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:48:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:20 smithi163 ceph-mon[36873]: pgmap v2255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:48:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:20 smithi042 ceph-mon[32775]: pgmap v2255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:48:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3131.179613 secs 2024-07-26T07:48:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3131.179613 secs 2024-07-26T07:48:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3131.179613 secs 2024-07-26T07:48:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:22 smithi164 ceph-mon[36970]: pgmap v2256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3148.921486 secs 2024-07-26T07:48:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:22 smithi163 ceph-mon[36873]: pgmap v2256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3148.921486 secs 2024-07-26T07:48:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:22 smithi042 ceph-mon[32775]: pgmap v2256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3148.921486 secs 2024-07-26T07:48:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:48:24.297+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:48:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:48:24.297+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:48:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:48:24.297+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:48:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:24 smithi163 ceph-mon[36873]: pgmap v2257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:24.789 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:24 smithi164 ceph-mon[36970]: pgmap v2257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:24 smithi042 ceph-mon[32775]: pgmap v2257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:26 smithi163 ceph-mon[36873]: pgmap v2258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3136.179679 secs 2024-07-26T07:48:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:26 smithi042 ceph-mon[32775]: pgmap v2258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3136.179679 secs 2024-07-26T07:48:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:26 smithi164 ceph-mon[36970]: pgmap v2258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3136.179679 secs 2024-07-26T07:48:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3153.921579 secs 2024-07-26T07:48:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3153.921579 secs 2024-07-26T07:48:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3153.921579 secs 2024-07-26T07:48:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:28 smithi163 ceph-mon[36873]: pgmap v2259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:28 smithi042 ceph-mon[32775]: pgmap v2259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:28 smithi164 ceph-mon[36970]: pgmap v2259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:30 smithi163 ceph-mon[36873]: pgmap v2260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:48:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:30 smithi042 ceph-mon[32775]: pgmap v2260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:48:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:30 smithi164 ceph-mon[36970]: pgmap v2260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:48:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3141.179777 secs 2024-07-26T07:48:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3141.179777 secs 2024-07-26T07:48:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3141.179777 secs 2024-07-26T07:48:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:32 smithi163 ceph-mon[36873]: pgmap v2261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3158.921705 secs 2024-07-26T07:48:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:32 smithi042 ceph-mon[32775]: pgmap v2261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3158.921705 secs 2024-07-26T07:48:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:32 smithi164 ceph-mon[36970]: pgmap v2261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3158.921705 secs 2024-07-26T07:48:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:34 smithi163 ceph-mon[36873]: pgmap v2262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:34 smithi042 ceph-mon[32775]: pgmap v2262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:35.192 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:34 smithi164 ceph-mon[36970]: pgmap v2262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:36 smithi163 ceph-mon[36873]: pgmap v2263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:36.916 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3146.179884 secs 2024-07-26T07:48:36.917 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:36 smithi042 ceph-mon[32775]: pgmap v2263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:36.917 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3146.179884 secs 2024-07-26T07:48:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:36 smithi164 ceph-mon[36970]: pgmap v2263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3146.179884 secs 2024-07-26T07:48:37.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3163.921830 secs 2024-07-26T07:48:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3163.921830 secs 2024-07-26T07:48:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3163.921830 secs 2024-07-26T07:48:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:38 smithi042 ceph-mon[32775]: pgmap v2264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:38 smithi164 ceph-mon[36970]: pgmap v2264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:38 smithi163 ceph-mon[36873]: pgmap v2264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:39.305 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:48:39.304+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:48:39.306 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:48:39.304+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:48:39.306 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:48:39.304+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:48:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:40 smithi042 ceph-mon[32775]: pgmap v2265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:48:40.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:40 smithi164 ceph-mon[36970]: pgmap v2265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:48:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:40 smithi163 ceph-mon[36873]: pgmap v2265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:48:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3151.179986 secs 2024-07-26T07:48:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3151.179986 secs 2024-07-26T07:48:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3151.179986 secs 2024-07-26T07:48:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:42 smithi042 ceph-mon[32775]: pgmap v2266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3168.921975 secs 2024-07-26T07:48:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:42 smithi164 ceph-mon[36970]: pgmap v2266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3168.921975 secs 2024-07-26T07:48:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:42 smithi163 ceph-mon[36873]: pgmap v2266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3168.921975 secs 2024-07-26T07:48:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:44 smithi042 ceph-mon[32775]: pgmap v2267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:44 smithi164 ceph-mon[36970]: pgmap v2267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:44 smithi163 ceph-mon[36873]: pgmap v2267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:46 smithi042 ceph-mon[32775]: pgmap v2268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3156.180114 secs 2024-07-26T07:48:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3173.922126 secs 2024-07-26T07:48:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:46 smithi164 ceph-mon[36970]: pgmap v2268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3156.180114 secs 2024-07-26T07:48:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3173.922126 secs 2024-07-26T07:48:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:46 smithi163 ceph-mon[36873]: pgmap v2268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3156.180114 secs 2024-07-26T07:48:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3173.922126 secs 2024-07-26T07:48:48.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:48 smithi042 ceph-mon[32775]: pgmap v2269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:48.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:48 smithi164 ceph-mon[36970]: pgmap v2269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:48 smithi163 ceph-mon[36873]: pgmap v2269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:50 smithi042 ceph-mon[32775]: pgmap v2270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:48:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:50 smithi164 ceph-mon[36970]: pgmap v2270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:48:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:50 smithi163 ceph-mon[36873]: pgmap v2270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:48:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3161.180241 secs 2024-07-26T07:48:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3178.922250 secs 2024-07-26T07:48:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3161.180241 secs 2024-07-26T07:48:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3178.922250 secs 2024-07-26T07:48:52.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3161.180241 secs 2024-07-26T07:48:52.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3178.922250 secs 2024-07-26T07:48:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:52 smithi042 ceph-mon[32775]: pgmap v2271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:52.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:52 smithi164 ceph-mon[36970]: pgmap v2271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:52 smithi163 ceph-mon[36873]: pgmap v2271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:48:54.307+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:48:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:48:54.307+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:48:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:48:54.307+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:48:54.809 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:54 smithi163 ceph-mon[36873]: pgmap v2272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:54 smithi042 ceph-mon[32775]: pgmap v2272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:54 smithi164 ceph-mon[36970]: pgmap v2272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:56 smithi042 ceph-mon[32775]: pgmap v2273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3166.180350 secs 2024-07-26T07:48:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3183.922374 secs 2024-07-26T07:48:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:56 smithi164 ceph-mon[36970]: pgmap v2273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3166.180350 secs 2024-07-26T07:48:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3183.922374 secs 2024-07-26T07:48:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:56 smithi163 ceph-mon[36873]: pgmap v2273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:48:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3166.180350 secs 2024-07-26T07:48:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3183.922374 secs 2024-07-26T07:48:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:48:58 smithi042 ceph-mon[32775]: pgmap v2274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:48:58 smithi164 ceph-mon[36970]: pgmap v2274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:48:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:48:58 smithi163 ceph-mon[36873]: pgmap v2274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:00 smithi042 ceph-mon[32775]: pgmap v2275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:00 smithi164 ceph-mon[36970]: pgmap v2275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:00 smithi163 ceph-mon[36873]: pgmap v2275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3171.180481 secs 2024-07-26T07:49:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3188.922498 secs 2024-07-26T07:49:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3171.180481 secs 2024-07-26T07:49:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3188.922498 secs 2024-07-26T07:49:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3171.180481 secs 2024-07-26T07:49:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3188.922498 secs 2024-07-26T07:49:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:02 smithi042 ceph-mon[32775]: pgmap v2276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:02 smithi164 ceph-mon[36970]: pgmap v2276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:02 smithi163 ceph-mon[36873]: pgmap v2276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:04.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:04 smithi042 ceph-mon[32775]: pgmap v2277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:04.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:04 smithi164 ceph-mon[36970]: pgmap v2277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:05.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:04 smithi163 ceph-mon[36873]: pgmap v2277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:06 smithi042 ceph-mon[32775]: pgmap v2278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3176.180612 secs 2024-07-26T07:49:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3193.922635 secs 2024-07-26T07:49:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:06 smithi164 ceph-mon[36970]: pgmap v2278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3176.180612 secs 2024-07-26T07:49:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3193.922635 secs 2024-07-26T07:49:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:06 smithi163 ceph-mon[36873]: pgmap v2278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3176.180612 secs 2024-07-26T07:49:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3193.922635 secs 2024-07-26T07:49:08.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:08 smithi042 ceph-mon[32775]: pgmap v2279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:08.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:08 smithi164 ceph-mon[36970]: pgmap v2279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:09.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:08 smithi163 ceph-mon[36873]: pgmap v2279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:49:09.315+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:49:09.428 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:49:09.315+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:49:09.428 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:49:09.315+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:49:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:49:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:49:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:49:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:49:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:49:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:49:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:49:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:49:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:49:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:49:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:49:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:49:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:10 smithi042 ceph-mon[32775]: pgmap v2280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:10 smithi042 ceph-mon[32775]: pgmap v2281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:49:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:10 smithi042 ceph-mon[32775]: pgmap v2282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:49:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:10 smithi164 ceph-mon[36970]: pgmap v2280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:10 smithi164 ceph-mon[36970]: pgmap v2281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:49:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:10 smithi164 ceph-mon[36970]: pgmap v2282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:49:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:10 smithi163 ceph-mon[36873]: pgmap v2280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:10 smithi163 ceph-mon[36873]: pgmap v2281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:49:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:10 smithi163 ceph-mon[36873]: pgmap v2282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:49:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3181.180740 secs 2024-07-26T07:49:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3198.922774 secs 2024-07-26T07:49:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3181.180740 secs 2024-07-26T07:49:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3198.922774 secs 2024-07-26T07:49:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3181.180740 secs 2024-07-26T07:49:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3198.922774 secs 2024-07-26T07:49:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:12 smithi042 ceph-mon[32775]: pgmap v2283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:49:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:12 smithi164 ceph-mon[36970]: pgmap v2283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:49:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:12 smithi163 ceph-mon[36873]: pgmap v2283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:49:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:14 smithi042 ceph-mon[32775]: pgmap v2284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:49:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:14 smithi164 ceph-mon[36970]: pgmap v2284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:49:15.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:14 smithi163 ceph-mon[36873]: pgmap v2284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:49:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:16 smithi042 ceph-mon[32775]: pgmap v2285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:49:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3186.180886 secs 2024-07-26T07:49:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3203.922901 secs 2024-07-26T07:49:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:16 smithi164 ceph-mon[36970]: pgmap v2285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:49:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3186.180886 secs 2024-07-26T07:49:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3203.922901 secs 2024-07-26T07:49:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:16 smithi163 ceph-mon[36873]: pgmap v2285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:49:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3186.180886 secs 2024-07-26T07:49:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3203.922901 secs 2024-07-26T07:49:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:18 smithi042 ceph-mon[32775]: pgmap v2286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:49:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:18 smithi164 ceph-mon[36970]: pgmap v2286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:49:19.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:18 smithi163 ceph-mon[36873]: pgmap v2286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:49:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:20 smithi042 ceph-mon[32775]: pgmap v2287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:49:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:20 smithi164 ceph-mon[36970]: pgmap v2287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:49:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:20 smithi163 ceph-mon[36873]: pgmap v2287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:49:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3191.181049 secs 2024-07-26T07:49:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3208.923029 secs 2024-07-26T07:49:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3191.181049 secs 2024-07-26T07:49:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3208.923029 secs 2024-07-26T07:49:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3191.181049 secs 2024-07-26T07:49:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3208.923029 secs 2024-07-26T07:49:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:22 smithi042 ceph-mon[32775]: pgmap v2288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:22 smithi164 ceph-mon[36970]: pgmap v2288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:22 smithi163 ceph-mon[36873]: pgmap v2288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:49:24.318+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:49:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:49:24.318+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:49:24.320 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:49:24.319+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:49:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:24 smithi042 ceph-mon[32775]: pgmap v2289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:24 smithi164 ceph-mon[36970]: pgmap v2289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:24 smithi163 ceph-mon[36873]: pgmap v2289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:26 smithi042 ceph-mon[32775]: pgmap v2290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3196.181191 secs 2024-07-26T07:49:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3213.923117 secs 2024-07-26T07:49:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:26 smithi164 ceph-mon[36970]: pgmap v2290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3196.181191 secs 2024-07-26T07:49:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3213.923117 secs 2024-07-26T07:49:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:26 smithi163 ceph-mon[36873]: pgmap v2290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3196.181191 secs 2024-07-26T07:49:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3213.923117 secs 2024-07-26T07:49:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:28 smithi042 ceph-mon[32775]: pgmap v2291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:28 smithi164 ceph-mon[36970]: pgmap v2291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:29.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:28 smithi163 ceph-mon[36873]: pgmap v2291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:30 smithi042 ceph-mon[32775]: pgmap v2292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:30.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:30 smithi164 ceph-mon[36970]: pgmap v2292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:31.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:30 smithi163 ceph-mon[36873]: pgmap v2292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3201.181321 secs 2024-07-26T07:49:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3218.923246 secs 2024-07-26T07:49:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3201.181321 secs 2024-07-26T07:49:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3218.923246 secs 2024-07-26T07:49:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3201.181321 secs 2024-07-26T07:49:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3218.923246 secs 2024-07-26T07:49:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:32 smithi042 ceph-mon[32775]: pgmap v2293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:32 smithi164 ceph-mon[36970]: pgmap v2293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:32 smithi163 ceph-mon[36873]: pgmap v2293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:34 smithi042 ceph-mon[32775]: pgmap v2294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:34.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:34 smithi164 ceph-mon[36970]: pgmap v2294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:35.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:34 smithi163 ceph-mon[36873]: pgmap v2294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:36 smithi042 ceph-mon[32775]: pgmap v2295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3206.181427 secs 2024-07-26T07:49:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3223.923369 secs 2024-07-26T07:49:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:36 smithi164 ceph-mon[36970]: pgmap v2295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3206.181427 secs 2024-07-26T07:49:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3223.923369 secs 2024-07-26T07:49:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:36 smithi163 ceph-mon[36873]: pgmap v2295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3206.181427 secs 2024-07-26T07:49:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3223.923369 secs 2024-07-26T07:49:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:38 smithi042 ceph-mon[32775]: pgmap v2296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:38 smithi164 ceph-mon[36970]: pgmap v2296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:38 smithi163 ceph-mon[36873]: pgmap v2296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:49:39.326+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:49:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:49:39.326+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:49:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:49:39.326+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:49:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:40 smithi042 ceph-mon[32775]: pgmap v2297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:40.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:40 smithi164 ceph-mon[36970]: pgmap v2297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:40 smithi163 ceph-mon[36873]: pgmap v2297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3211.181540 secs 2024-07-26T07:49:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3228.923513 secs 2024-07-26T07:49:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3211.181540 secs 2024-07-26T07:49:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3228.923513 secs 2024-07-26T07:49:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3211.181540 secs 2024-07-26T07:49:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3228.923513 secs 2024-07-26T07:49:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:42 smithi164 ceph-mon[36970]: pgmap v2298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:42 smithi163 ceph-mon[36873]: pgmap v2298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:42 smithi042 ceph-mon[32775]: pgmap v2298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:44 smithi164 ceph-mon[36970]: pgmap v2299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:44 smithi163 ceph-mon[36873]: pgmap v2299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:44 smithi042 ceph-mon[32775]: pgmap v2299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:46 smithi164 ceph-mon[36970]: pgmap v2300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3216.181634 secs 2024-07-26T07:49:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3233.923629 secs 2024-07-26T07:49:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:46 smithi163 ceph-mon[36873]: pgmap v2300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3216.181634 secs 2024-07-26T07:49:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3233.923629 secs 2024-07-26T07:49:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:46 smithi042 ceph-mon[32775]: pgmap v2300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3216.181634 secs 2024-07-26T07:49:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3233.923629 secs 2024-07-26T07:49:48.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:48 smithi042 ceph-mon[32775]: pgmap v2301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:48.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:48 smithi164 ceph-mon[36970]: pgmap v2301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:48 smithi163 ceph-mon[36873]: pgmap v2301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:50 smithi164 ceph-mon[36970]: pgmap v2302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:50 smithi163 ceph-mon[36873]: pgmap v2302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:50 smithi042 ceph-mon[32775]: pgmap v2302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3221.181753 secs 2024-07-26T07:49:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3238.923754 secs 2024-07-26T07:49:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3221.181753 secs 2024-07-26T07:49:52.169 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3238.923754 secs 2024-07-26T07:49:52.170 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3221.181753 secs 2024-07-26T07:49:52.170 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3238.923754 secs 2024-07-26T07:49:52.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:52 smithi164 ceph-mon[36970]: pgmap v2303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:52 smithi163 ceph-mon[36873]: pgmap v2303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:52 smithi042 ceph-mon[32775]: pgmap v2303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:49:54.330+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:49:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:49:54.330+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:49:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:49:54.330+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:49:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:54 smithi164 ceph-mon[36970]: pgmap v2304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:54 smithi163 ceph-mon[36873]: pgmap v2304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:54 smithi042 ceph-mon[32775]: pgmap v2304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:49:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:56 smithi164 ceph-mon[36970]: pgmap v2305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3226.181880 secs 2024-07-26T07:49:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3243.923852 secs 2024-07-26T07:49:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:56 smithi163 ceph-mon[36873]: pgmap v2305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3226.181880 secs 2024-07-26T07:49:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3243.923852 secs 2024-07-26T07:49:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:56 smithi042 ceph-mon[32775]: pgmap v2305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:49:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3226.181880 secs 2024-07-26T07:49:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3243.923852 secs 2024-07-26T07:49:58.368 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:49:58.367+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:49:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:49:58.367+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:49:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:49:58.368+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:49:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:49:58.368+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:49:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:49:58 smithi164 ceph-mon[36970]: pgmap v2306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:49:58 smithi163 ceph-mon[36873]: pgmap v2306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:49:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:49:58 smithi042 ceph-mon[32775]: pgmap v2306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:00 smithi164 ceph-mon[36970]: pgmap v2307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:50:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:00 smithi163 ceph-mon[36873]: pgmap v2307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:50:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:00 smithi042 ceph-mon[32775]: pgmap v2307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T07:50:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3231.182034 secs 2024-07-26T07:50:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3248.923976 secs 2024-07-26T07:50:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3231.182034 secs 2024-07-26T07:50:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3248.923976 secs 2024-07-26T07:50:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3231.182034 secs 2024-07-26T07:50:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3248.923976 secs 2024-07-26T07:50:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:02 smithi164 ceph-mon[36970]: pgmap v2308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:02 smithi163 ceph-mon[36873]: pgmap v2308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:02 smithi042 ceph-mon[32775]: pgmap v2308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:04.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:04 smithi164 ceph-mon[36970]: pgmap v2309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:50:05.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:04 smithi163 ceph-mon[36873]: pgmap v2309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:50:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:04 smithi042 ceph-mon[32775]: pgmap v2309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:50:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:05 smithi163 ceph-mon[36873]: pgmap v2310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:05 smithi042 ceph-mon[32775]: pgmap v2310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:06.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:05 smithi164 ceph-mon[36970]: pgmap v2310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3236.182153 secs 2024-07-26T07:50:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3253.924098 secs 2024-07-26T07:50:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3236.182153 secs 2024-07-26T07:50:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3253.924098 secs 2024-07-26T07:50:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3236.182153 secs 2024-07-26T07:50:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3253.924098 secs 2024-07-26T07:50:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:07 smithi163 ceph-mon[36873]: pgmap v2311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:07 smithi042 ceph-mon[32775]: pgmap v2311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:08.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:07 smithi164 ceph-mon[36970]: pgmap v2311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:09.339 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:50:09.339+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:50:09.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:50:09.339+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:50:09.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:50:09.339+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:50:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:10 smithi164 ceph-mon[36970]: pgmap v2312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:50:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:10 smithi163 ceph-mon[36873]: pgmap v2312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:50:10.894 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:10 smithi042 ceph-mon[32775]: pgmap v2312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:50:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3241.182227 secs 2024-07-26T07:50:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3258.924172 secs 2024-07-26T07:50:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:12 smithi042 ceph-mon[32775]: pgmap v2313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:50:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:50:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3241.182227 secs 2024-07-26T07:50:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3258.924172 secs 2024-07-26T07:50:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:12 smithi164 ceph-mon[36970]: pgmap v2313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:50:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:50:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3241.182227 secs 2024-07-26T07:50:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3258.924172 secs 2024-07-26T07:50:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:12 smithi163 ceph-mon[36873]: pgmap v2313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:50:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:50:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:50:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:13 smithi042 ceph-mon[32775]: pgmap v2314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:50:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:13 smithi042 ceph-mon[32775]: pgmap v2315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T07:50:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:13 smithi164 ceph-mon[36970]: pgmap v2314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:50:13.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:13 smithi164 ceph-mon[36970]: pgmap v2315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T07:50:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:13 smithi163 ceph-mon[36873]: pgmap v2314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:50:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:13 smithi163 ceph-mon[36873]: pgmap v2315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T07:50:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:15 smithi042 ceph-mon[32775]: pgmap v2316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:50:15.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:15 smithi164 ceph-mon[36970]: pgmap v2316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:50:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:15 smithi163 ceph-mon[36873]: pgmap v2316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:50:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3246.182347 secs 2024-07-26T07:50:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:17 smithi042 ceph-mon[32775]: pgmap v2317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:50:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3263.924295 secs 2024-07-26T07:50:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3246.182347 secs 2024-07-26T07:50:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:17 smithi164 ceph-mon[36970]: pgmap v2317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:50:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3263.924295 secs 2024-07-26T07:50:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3246.182347 secs 2024-07-26T07:50:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:17 smithi163 ceph-mon[36873]: pgmap v2317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:50:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3263.924295 secs 2024-07-26T07:50:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:19 smithi042 ceph-mon[32775]: pgmap v2318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:50:19.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:19 smithi164 ceph-mon[36970]: pgmap v2318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:50:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:19 smithi163 ceph-mon[36873]: pgmap v2318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:50:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:21 smithi042 ceph-mon[32775]: pgmap v2319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:50:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:21 smithi164 ceph-mon[36970]: pgmap v2319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:50:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:21 smithi163 ceph-mon[36873]: pgmap v2319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T07:50:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3251.182443 secs 2024-07-26T07:50:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3268.924421 secs 2024-07-26T07:50:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3251.182443 secs 2024-07-26T07:50:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3268.924421 secs 2024-07-26T07:50:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3251.182443 secs 2024-07-26T07:50:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3268.924421 secs 2024-07-26T07:50:23.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:23 smithi164 ceph-mon[36970]: pgmap v2320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:50:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:23 smithi163 ceph-mon[36873]: pgmap v2320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:50:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:23 smithi042 ceph-mon[32775]: pgmap v2320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:50:24.342 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:50:24.342+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:50:24.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:50:24.342+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:50:24.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:50:24.342+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:50:25.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:25 smithi164 ceph-mon[36970]: pgmap v2321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:50:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:25 smithi163 ceph-mon[36873]: pgmap v2321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:50:25.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:25 smithi042 ceph-mon[32775]: pgmap v2321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:50:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3256.182541 secs 2024-07-26T07:50:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:27 smithi042 ceph-mon[32775]: pgmap v2322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3273.924534 secs 2024-07-26T07:50:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3256.182541 secs 2024-07-26T07:50:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:27 smithi164 ceph-mon[36970]: pgmap v2322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3273.924534 secs 2024-07-26T07:50:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3256.182541 secs 2024-07-26T07:50:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:27 smithi163 ceph-mon[36873]: pgmap v2322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3273.924534 secs 2024-07-26T07:50:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:29 smithi164 ceph-mon[36970]: pgmap v2323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:29 smithi163 ceph-mon[36873]: pgmap v2323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:29 smithi042 ceph-mon[32775]: pgmap v2323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:31 smithi164 ceph-mon[36970]: pgmap v2324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3261.182663 secs 2024-07-26T07:50:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:31 smithi163 ceph-mon[36873]: pgmap v2324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3261.182663 secs 2024-07-26T07:50:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:31 smithi042 ceph-mon[32775]: pgmap v2324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3261.182663 secs 2024-07-26T07:50:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3278.924658 secs 2024-07-26T07:50:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3278.924658 secs 2024-07-26T07:50:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3278.924658 secs 2024-07-26T07:50:33.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:33 smithi164 ceph-mon[36970]: pgmap v2325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:33 smithi163 ceph-mon[36873]: pgmap v2325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:33.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:33 smithi042 ceph-mon[32775]: pgmap v2325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:35.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:35 smithi164 ceph-mon[36970]: pgmap v2326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:50:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:35 smithi163 ceph-mon[36873]: pgmap v2326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:50:35.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:35 smithi042 ceph-mon[32775]: pgmap v2326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:50:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3266.182789 secs 2024-07-26T07:50:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3266.182789 secs 2024-07-26T07:50:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3266.182789 secs 2024-07-26T07:50:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:37 smithi164 ceph-mon[36970]: pgmap v2327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3283.924802 secs 2024-07-26T07:50:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:37 smithi163 ceph-mon[36873]: pgmap v2327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3283.924802 secs 2024-07-26T07:50:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:37 smithi042 ceph-mon[32775]: pgmap v2327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3283.924802 secs 2024-07-26T07:50:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:50:39.350+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:50:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:50:39.350+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:50:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:50:39.350+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:50:39.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:39 smithi164 ceph-mon[36970]: pgmap v2328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:39 smithi163 ceph-mon[36873]: pgmap v2328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:39 smithi042 ceph-mon[32775]: pgmap v2328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:41 smithi164 ceph-mon[36970]: pgmap v2329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:41.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3271.182913 secs 2024-07-26T07:50:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:41 smithi163 ceph-mon[36873]: pgmap v2329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3271.182913 secs 2024-07-26T07:50:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:41 smithi042 ceph-mon[32775]: pgmap v2329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3271.182913 secs 2024-07-26T07:50:42.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3288.924951 secs 2024-07-26T07:50:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3288.924951 secs 2024-07-26T07:50:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3288.924951 secs 2024-07-26T07:50:43.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:43 smithi164 ceph-mon[36970]: pgmap v2330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:43.527 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:43 smithi163 ceph-mon[36873]: pgmap v2330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:43 smithi042 ceph-mon[32775]: pgmap v2330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:45.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:45 smithi164 ceph-mon[36970]: pgmap v2331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:50:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:45 smithi163 ceph-mon[36873]: pgmap v2331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:50:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:45 smithi042 ceph-mon[32775]: pgmap v2331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:50:46.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3276.183039 secs 2024-07-26T07:50:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3276.183039 secs 2024-07-26T07:50:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3276.183039 secs 2024-07-26T07:50:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:47 smithi164 ceph-mon[36970]: pgmap v2332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3293.925054 secs 2024-07-26T07:50:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:47 smithi163 ceph-mon[36873]: pgmap v2332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3293.925054 secs 2024-07-26T07:50:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:47 smithi042 ceph-mon[32775]: pgmap v2332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3293.925054 secs 2024-07-26T07:50:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:49 smithi163 ceph-mon[36873]: pgmap v2333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:49 smithi042 ceph-mon[32775]: pgmap v2333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:49 smithi164 ceph-mon[36970]: pgmap v2333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:51 smithi163 ceph-mon[36873]: pgmap v2334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3281.183118 secs 2024-07-26T07:50:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:51 smithi042 ceph-mon[32775]: pgmap v2334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3281.183118 secs 2024-07-26T07:50:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:51 smithi164 ceph-mon[36970]: pgmap v2334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3281.183118 secs 2024-07-26T07:50:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3298.925176 secs 2024-07-26T07:50:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3298.925176 secs 2024-07-26T07:50:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3298.925176 secs 2024-07-26T07:50:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:53 smithi163 ceph-mon[36873]: pgmap v2335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:53 smithi042 ceph-mon[32775]: pgmap v2335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:53 smithi164 ceph-mon[36970]: pgmap v2335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:54.354 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:50:54.353+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:50:54.354 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:50:54.353+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:50:54.354 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:50:54.353+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:50:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:55 smithi163 ceph-mon[36873]: pgmap v2336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:50:55.524 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:55 smithi164 ceph-mon[36970]: pgmap v2336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:50:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:55 smithi042 ceph-mon[32775]: pgmap v2336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:50:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3286.183258 secs 2024-07-26T07:50:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3286.183258 secs 2024-07-26T07:50:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3286.183258 secs 2024-07-26T07:50:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:57 smithi163 ceph-mon[36873]: pgmap v2337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3303.925278 secs 2024-07-26T07:50:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:57 smithi042 ceph-mon[32775]: pgmap v2337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3303.925278 secs 2024-07-26T07:50:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:57 smithi164 ceph-mon[36970]: pgmap v2337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:50:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3303.925278 secs 2024-07-26T07:50:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:50:59 smithi163 ceph-mon[36873]: pgmap v2338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:50:59 smithi042 ceph-mon[32775]: pgmap v2338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:50:59.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:50:59 smithi164 ceph-mon[36970]: pgmap v2338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:01 smithi163 ceph-mon[36873]: pgmap v2339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3291.183384 secs 2024-07-26T07:51:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:01 smithi042 ceph-mon[32775]: pgmap v2339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3291.183384 secs 2024-07-26T07:51:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:01 smithi164 ceph-mon[36970]: pgmap v2339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3291.183384 secs 2024-07-26T07:51:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3308.925406 secs 2024-07-26T07:51:02.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3308.925406 secs 2024-07-26T07:51:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3308.925406 secs 2024-07-26T07:51:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:03 smithi042 ceph-mon[32775]: pgmap v2340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:03.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:03 smithi164 ceph-mon[36970]: pgmap v2340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:03 smithi163 ceph-mon[36873]: pgmap v2340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:05 smithi042 ceph-mon[32775]: pgmap v2341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:51:05.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:05 smithi164 ceph-mon[36970]: pgmap v2341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:51:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:05 smithi163 ceph-mon[36873]: pgmap v2341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:51:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3296.183457 secs 2024-07-26T07:51:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3296.183457 secs 2024-07-26T07:51:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3296.183457 secs 2024-07-26T07:51:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:07 smithi042 ceph-mon[32775]: pgmap v2342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3313.925531 secs 2024-07-26T07:51:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:07 smithi164 ceph-mon[36970]: pgmap v2342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3313.925531 secs 2024-07-26T07:51:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:07 smithi163 ceph-mon[36873]: pgmap v2342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3313.925531 secs 2024-07-26T07:51:09.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:51:09.361+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:51:09.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:51:09.361+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:51:09.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:51:09.361+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:51:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:09 smithi042 ceph-mon[32775]: pgmap v2343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:09 smithi164 ceph-mon[36970]: pgmap v2343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:09 smithi163 ceph-mon[36873]: pgmap v2343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:11 smithi042 ceph-mon[32775]: pgmap v2344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3301.183579 secs 2024-07-26T07:51:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:11 smithi164 ceph-mon[36970]: pgmap v2344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3301.183579 secs 2024-07-26T07:51:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:11 smithi163 ceph-mon[36873]: pgmap v2344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3301.183579 secs 2024-07-26T07:51:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3318.925654 secs 2024-07-26T07:51:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:51:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3318.925654 secs 2024-07-26T07:51:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:51:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3318.925654 secs 2024-07-26T07:51:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:51:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:13 smithi042 ceph-mon[32775]: pgmap v2345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:51:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:51:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:13 smithi164 ceph-mon[36970]: pgmap v2345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:51:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:51:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:13 smithi163 ceph-mon[36873]: pgmap v2345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:51:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:51:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:51:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:14 smithi042 ceph-mon[32775]: pgmap v2346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:51:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:14 smithi042 ceph-mon[32775]: pgmap v2347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:51:14.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:14 smithi164 ceph-mon[36970]: pgmap v2346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:51:14.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:14 smithi164 ceph-mon[36970]: pgmap v2347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:51:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:14 smithi163 ceph-mon[36873]: pgmap v2346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T07:51:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:14 smithi163 ceph-mon[36873]: pgmap v2347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:51:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:16 smithi042 ceph-mon[32775]: pgmap v2348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:51:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3306.183723 secs 2024-07-26T07:51:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:16 smithi164 ceph-mon[36970]: pgmap v2348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:51:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3306.183723 secs 2024-07-26T07:51:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:16 smithi163 ceph-mon[36873]: pgmap v2348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:51:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3306.183723 secs 2024-07-26T07:51:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3323.925782 secs 2024-07-26T07:51:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3323.925782 secs 2024-07-26T07:51:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3323.925782 secs 2024-07-26T07:51:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:18 smithi042 ceph-mon[32775]: pgmap v2349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:51:18.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:18 smithi164 ceph-mon[36970]: pgmap v2349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:51:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:18 smithi163 ceph-mon[36873]: pgmap v2349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:51:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:20 smithi042 ceph-mon[32775]: pgmap v2350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T07:51:20.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:20 smithi164 ceph-mon[36970]: pgmap v2350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T07:51:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:20 smithi163 ceph-mon[36873]: pgmap v2350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T07:51:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3311.183798 secs 2024-07-26T07:51:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3311.183798 secs 2024-07-26T07:51:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3311.183798 secs 2024-07-26T07:51:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:22 smithi042 ceph-mon[32775]: pgmap v2351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:51:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3328.925951 secs 2024-07-26T07:51:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:22 smithi164 ceph-mon[36970]: pgmap v2351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:51:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3328.925951 secs 2024-07-26T07:51:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:22 smithi163 ceph-mon[36873]: pgmap v2351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:51:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3328.925951 secs 2024-07-26T07:51:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:51:24.364+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:51:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:51:24.364+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:51:24.365 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:51:24.364+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:51:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:24 smithi042 ceph-mon[32775]: pgmap v2352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:51:24.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:24 smithi164 ceph-mon[36970]: pgmap v2352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:51:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:24 smithi163 ceph-mon[36873]: pgmap v2352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:51:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:26 smithi042 ceph-mon[32775]: pgmap v2353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3316.183947 secs 2024-07-26T07:51:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:26 smithi164 ceph-mon[36970]: pgmap v2353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3316.183947 secs 2024-07-26T07:51:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:26 smithi163 ceph-mon[36873]: pgmap v2353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3316.183947 secs 2024-07-26T07:51:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3333.926052 secs 2024-07-26T07:51:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3333.926052 secs 2024-07-26T07:51:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3333.926052 secs 2024-07-26T07:51:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:28 smithi042 ceph-mon[32775]: pgmap v2354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:28 smithi164 ceph-mon[36970]: pgmap v2354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:28 smithi163 ceph-mon[36873]: pgmap v2354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:30 smithi042 ceph-mon[32775]: pgmap v2355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:51:30.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:30 smithi164 ceph-mon[36970]: pgmap v2355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:51:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:30 smithi163 ceph-mon[36873]: pgmap v2355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:51:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3321.184084 secs 2024-07-26T07:51:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3321.184084 secs 2024-07-26T07:51:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3321.184084 secs 2024-07-26T07:51:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:32 smithi042 ceph-mon[32775]: pgmap v2356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3338.926178 secs 2024-07-26T07:51:32.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:32 smithi164 ceph-mon[36970]: pgmap v2356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3338.926178 secs 2024-07-26T07:51:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:32 smithi163 ceph-mon[36873]: pgmap v2356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3338.926178 secs 2024-07-26T07:51:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:34 smithi042 ceph-mon[32775]: pgmap v2357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:34 smithi164 ceph-mon[36970]: pgmap v2357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:34 smithi163 ceph-mon[36873]: pgmap v2357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:36 smithi042 ceph-mon[32775]: pgmap v2358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3326.184212 secs 2024-07-26T07:51:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:36 smithi164 ceph-mon[36970]: pgmap v2358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3326.184212 secs 2024-07-26T07:51:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:36 smithi163 ceph-mon[36873]: pgmap v2358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3326.184212 secs 2024-07-26T07:51:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3343.926303 secs 2024-07-26T07:51:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3343.926303 secs 2024-07-26T07:51:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3343.926303 secs 2024-07-26T07:51:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:38 smithi042 ceph-mon[32775]: pgmap v2359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:38 smithi164 ceph-mon[36970]: pgmap v2359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:38 smithi163 ceph-mon[36873]: pgmap v2359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:51:39.373+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:51:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:51:39.373+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:51:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:51:39.373+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:51:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:40 smithi042 ceph-mon[32775]: pgmap v2360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:51:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:40 smithi164 ceph-mon[36970]: pgmap v2360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:51:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:40 smithi163 ceph-mon[36873]: pgmap v2360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:51:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3331.184296 secs 2024-07-26T07:51:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3331.184296 secs 2024-07-26T07:51:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3331.184296 secs 2024-07-26T07:51:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:42 smithi042 ceph-mon[32775]: pgmap v2361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3348.926426 secs 2024-07-26T07:51:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:42 smithi164 ceph-mon[36970]: pgmap v2361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3348.926426 secs 2024-07-26T07:51:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:42 smithi163 ceph-mon[36873]: pgmap v2361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3348.926426 secs 2024-07-26T07:51:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:44 smithi042 ceph-mon[32775]: pgmap v2362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:44 smithi164 ceph-mon[36970]: pgmap v2362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:44 smithi163 ceph-mon[36873]: pgmap v2362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:46 smithi042 ceph-mon[32775]: pgmap v2363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3336.184383 secs 2024-07-26T07:51:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:46 smithi164 ceph-mon[36970]: pgmap v2363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3336.184383 secs 2024-07-26T07:51:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:46 smithi163 ceph-mon[36873]: pgmap v2363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3336.184383 secs 2024-07-26T07:51:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3353.926556 secs 2024-07-26T07:51:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3353.926556 secs 2024-07-26T07:51:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3353.926556 secs 2024-07-26T07:51:48.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:48 smithi164 ceph-mon[36970]: pgmap v2364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:48 smithi163 ceph-mon[36873]: pgmap v2364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:48.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:48 smithi042 ceph-mon[32775]: pgmap v2364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:50.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:50 smithi042 ceph-mon[32775]: pgmap v2365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:51:50.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:50 smithi164 ceph-mon[36970]: pgmap v2365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:51:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:50 smithi163 ceph-mon[36873]: pgmap v2365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:51:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3341.184469 secs 2024-07-26T07:51:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3341.184469 secs 2024-07-26T07:51:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3341.184469 secs 2024-07-26T07:51:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:52 smithi164 ceph-mon[36970]: pgmap v2366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3358.926680 secs 2024-07-26T07:51:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:52 smithi163 ceph-mon[36873]: pgmap v2366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3358.926680 secs 2024-07-26T07:51:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:52 smithi042 ceph-mon[32775]: pgmap v2366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3358.926680 secs 2024-07-26T07:51:54.376 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:51:54.375+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:51:54.376 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:51:54.375+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:51:54.376 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:51:54.375+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:51:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:54 smithi164 ceph-mon[36970]: pgmap v2367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:54 smithi163 ceph-mon[36873]: pgmap v2367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:54 smithi042 ceph-mon[32775]: pgmap v2367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:56 smithi164 ceph-mon[36970]: pgmap v2368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3346.184551 secs 2024-07-26T07:51:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:56 smithi163 ceph-mon[36873]: pgmap v2368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3346.184551 secs 2024-07-26T07:51:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:56 smithi042 ceph-mon[32775]: pgmap v2368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:51:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3346.184551 secs 2024-07-26T07:51:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3363.926825 secs 2024-07-26T07:51:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3363.926825 secs 2024-07-26T07:51:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3363.926825 secs 2024-07-26T07:51:58.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:51:58 smithi164 ceph-mon[36970]: pgmap v2369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:51:58 smithi163 ceph-mon[36873]: pgmap v2369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:51:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:51:58 smithi042 ceph-mon[32775]: pgmap v2369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:00 smithi164 ceph-mon[36970]: pgmap v2370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:52:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:00 smithi163 ceph-mon[36873]: pgmap v2370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:52:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:00 smithi042 ceph-mon[32775]: pgmap v2370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:52:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3351.184638 secs 2024-07-26T07:52:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3351.184638 secs 2024-07-26T07:52:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3351.184638 secs 2024-07-26T07:52:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:02 smithi164 ceph-mon[36970]: pgmap v2371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3368.926943 secs 2024-07-26T07:52:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:02 smithi163 ceph-mon[36873]: pgmap v2371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3368.926943 secs 2024-07-26T07:52:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:02 smithi042 ceph-mon[32775]: pgmap v2371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3368.926943 secs 2024-07-26T07:52:04.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:04 smithi164 ceph-mon[36970]: pgmap v2372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:04 smithi163 ceph-mon[36873]: pgmap v2372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:04.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:04 smithi042 ceph-mon[32775]: pgmap v2372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:06 smithi164 ceph-mon[36970]: pgmap v2373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3356.184765 secs 2024-07-26T07:52:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:06 smithi163 ceph-mon[36873]: pgmap v2373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3356.184765 secs 2024-07-26T07:52:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:06 smithi042 ceph-mon[32775]: pgmap v2373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3356.184765 secs 2024-07-26T07:52:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3373.927095 secs 2024-07-26T07:52:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3373.927095 secs 2024-07-26T07:52:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3373.927095 secs 2024-07-26T07:52:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:08 smithi164 ceph-mon[36970]: pgmap v2374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:08 smithi163 ceph-mon[36873]: pgmap v2374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:08.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:08 smithi042 ceph-mon[32775]: pgmap v2374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:09.384 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:52:09.384+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:52:09.384 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:52:09.384+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:52:09.384 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:52:09.384+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:52:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:10 smithi164 ceph-mon[36970]: pgmap v2375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:52:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:10 smithi163 ceph-mon[36873]: pgmap v2375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:52:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:10 smithi042 ceph-mon[32775]: pgmap v2375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:52:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3361.184827 secs 2024-07-26T07:52:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3361.184827 secs 2024-07-26T07:52:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3361.184827 secs 2024-07-26T07:52:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:12 smithi164 ceph-mon[36970]: pgmap v2376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3378.927218 secs 2024-07-26T07:52:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:12 smithi163 ceph-mon[36873]: pgmap v2376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3378.927218 secs 2024-07-26T07:52:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:12 smithi042 ceph-mon[32775]: pgmap v2376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3378.927218 secs 2024-07-26T07:52:13.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:52:13.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:52:13.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:52:13.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:52:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:52:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:52:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:52:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:52:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:52:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:52:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:52:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:52:14.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:14 smithi164 ceph-mon[36970]: pgmap v2377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:14 smithi164 ceph-mon[36970]: pgmap v2378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:52:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:14 smithi164 ceph-mon[36970]: pgmap v2379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:52:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:14 smithi163 ceph-mon[36873]: pgmap v2377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:14 smithi163 ceph-mon[36873]: pgmap v2378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:52:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:14 smithi163 ceph-mon[36873]: pgmap v2379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:52:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:14 smithi042 ceph-mon[32775]: pgmap v2377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:14 smithi042 ceph-mon[32775]: pgmap v2378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:52:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:14 smithi042 ceph-mon[32775]: pgmap v2379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:52:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:16 smithi164 ceph-mon[36970]: pgmap v2380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:52:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3366.184947 secs 2024-07-26T07:52:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:16 smithi163 ceph-mon[36873]: pgmap v2380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:52:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3366.184947 secs 2024-07-26T07:52:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:16 smithi042 ceph-mon[32775]: pgmap v2380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:52:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3366.184947 secs 2024-07-26T07:52:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3383.927342 secs 2024-07-26T07:52:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3383.927342 secs 2024-07-26T07:52:17.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3383.927342 secs 2024-07-26T07:52:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:18 smithi163 ceph-mon[36873]: pgmap v2381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:52:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:18 smithi042 ceph-mon[32775]: pgmap v2381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:52:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:18 smithi164 ceph-mon[36970]: pgmap v2381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:52:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:20 smithi163 ceph-mon[36873]: pgmap v2382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:52:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:20 smithi042 ceph-mon[32775]: pgmap v2382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:52:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:20 smithi164 ceph-mon[36970]: pgmap v2382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:52:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3371.185071 secs 2024-07-26T07:52:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3371.185071 secs 2024-07-26T07:52:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3371.185071 secs 2024-07-26T07:52:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3388.927466 secs 2024-07-26T07:52:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:22 smithi163 ceph-mon[36873]: pgmap v2383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:52:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3388.927466 secs 2024-07-26T07:52:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:22 smithi042 ceph-mon[32775]: pgmap v2383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:52:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3388.927466 secs 2024-07-26T07:52:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:22 smithi164 ceph-mon[36970]: pgmap v2383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:52:24.388 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:52:24.387+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:52:24.388 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:52:24.387+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:52:24.388 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:52:24.387+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:52:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:24 smithi163 ceph-mon[36873]: pgmap v2384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:52:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:24 smithi042 ceph-mon[32775]: pgmap v2384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:52:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:24 smithi164 ceph-mon[36970]: pgmap v2384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:52:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:26 smithi163 ceph-mon[36873]: pgmap v2385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3376.185193 secs 2024-07-26T07:52:26.775 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:26 smithi164 ceph-mon[36970]: pgmap v2385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:26.775 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3376.185193 secs 2024-07-26T07:52:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:26 smithi042 ceph-mon[32775]: pgmap v2385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3376.185193 secs 2024-07-26T07:52:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3393.927542 secs 2024-07-26T07:52:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3393.927542 secs 2024-07-26T07:52:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3393.927542 secs 2024-07-26T07:52:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:28 smithi163 ceph-mon[36873]: pgmap v2386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:28 smithi042 ceph-mon[32775]: pgmap v2386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:28 smithi164 ceph-mon[36970]: pgmap v2386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:30 smithi042 ceph-mon[32775]: pgmap v2387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:30.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:30 smithi164 ceph-mon[36970]: pgmap v2387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:31.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:30 smithi163 ceph-mon[36873]: pgmap v2387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3381.185303 secs 2024-07-26T07:52:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3381.185303 secs 2024-07-26T07:52:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3381.185303 secs 2024-07-26T07:52:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3398.927666 secs 2024-07-26T07:52:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:32 smithi042 ceph-mon[32775]: pgmap v2388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3398.927666 secs 2024-07-26T07:52:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:32 smithi164 ceph-mon[36970]: pgmap v2388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3398.927666 secs 2024-07-26T07:52:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:32 smithi163 ceph-mon[36873]: pgmap v2388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:34 smithi042 ceph-mon[32775]: pgmap v2389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:34.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:34 smithi164 ceph-mon[36970]: pgmap v2389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:35.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:34 smithi163 ceph-mon[36873]: pgmap v2389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:36 smithi042 ceph-mon[32775]: pgmap v2390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3386.185428 secs 2024-07-26T07:52:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:36 smithi164 ceph-mon[36970]: pgmap v2390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3386.185428 secs 2024-07-26T07:52:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:36 smithi163 ceph-mon[36873]: pgmap v2390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3386.185428 secs 2024-07-26T07:52:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3403.927793 secs 2024-07-26T07:52:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3403.927793 secs 2024-07-26T07:52:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3403.927793 secs 2024-07-26T07:52:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:38 smithi042 ceph-mon[32775]: pgmap v2391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:38 smithi164 ceph-mon[36970]: pgmap v2391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:38 smithi163 ceph-mon[36873]: pgmap v2391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:52:39.395+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:52:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:52:39.395+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:52:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:52:39.395+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:52:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:40 smithi042 ceph-mon[32775]: pgmap v2392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:40.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:40 smithi164 ceph-mon[36970]: pgmap v2392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:41.323 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:40 smithi163 ceph-mon[36873]: pgmap v2392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3391.185558 secs 2024-07-26T07:52:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3391.185558 secs 2024-07-26T07:52:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3391.185558 secs 2024-07-26T07:52:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3408.927960 secs 2024-07-26T07:52:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:42 smithi042 ceph-mon[32775]: pgmap v2393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3408.927960 secs 2024-07-26T07:52:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:42 smithi164 ceph-mon[36970]: pgmap v2393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3408.927960 secs 2024-07-26T07:52:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:42 smithi163 ceph-mon[36873]: pgmap v2393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:44 smithi042 ceph-mon[32775]: pgmap v2394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:44 smithi164 ceph-mon[36970]: pgmap v2394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:44 smithi163 ceph-mon[36873]: pgmap v2394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:46 smithi042 ceph-mon[32775]: pgmap v2395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3396.185630 secs 2024-07-26T07:52:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:46 smithi164 ceph-mon[36970]: pgmap v2395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3396.185630 secs 2024-07-26T07:52:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:46 smithi163 ceph-mon[36873]: pgmap v2395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3396.185630 secs 2024-07-26T07:52:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3413.928076 secs 2024-07-26T07:52:47.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3413.928076 secs 2024-07-26T07:52:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3413.928076 secs 2024-07-26T07:52:48.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:48 smithi042 ceph-mon[32775]: pgmap v2396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:48.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:48 smithi164 ceph-mon[36970]: pgmap v2396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:48 smithi163 ceph-mon[36873]: pgmap v2396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:50 smithi042 ceph-mon[32775]: pgmap v2397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:50 smithi164 ceph-mon[36970]: pgmap v2397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:50 smithi163 ceph-mon[36873]: pgmap v2397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3401.185759 secs 2024-07-26T07:52:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3401.185759 secs 2024-07-26T07:52:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3401.185759 secs 2024-07-26T07:52:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3418.928150 secs 2024-07-26T07:52:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:52 smithi042 ceph-mon[32775]: pgmap v2398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:52.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3418.928150 secs 2024-07-26T07:52:52.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:52 smithi164 ceph-mon[36970]: pgmap v2398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3418.928150 secs 2024-07-26T07:52:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:52 smithi163 ceph-mon[36873]: pgmap v2398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:54.400 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:52:54.399+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:52:54.400 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:52:54.399+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:52:54.400 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:52:54.399+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:52:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:54 smithi042 ceph-mon[32775]: pgmap v2399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:54 smithi164 ceph-mon[36970]: pgmap v2399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:54 smithi163 ceph-mon[36873]: pgmap v2399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:56 smithi042 ceph-mon[32775]: pgmap v2400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3406.185888 secs 2024-07-26T07:52:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3423.928280 secs 2024-07-26T07:52:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:56 smithi164 ceph-mon[36970]: pgmap v2400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3406.185888 secs 2024-07-26T07:52:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3423.928280 secs 2024-07-26T07:52:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:56 smithi163 ceph-mon[36873]: pgmap v2400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:52:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3406.185888 secs 2024-07-26T07:52:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3423.928280 secs 2024-07-26T07:52:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:52:58.395+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:52:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:52:58.395+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:52:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:52:58.396+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:52:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:52:58.396+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:52:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:52:58 smithi042 ceph-mon[32775]: pgmap v2401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:52:58 smithi164 ceph-mon[36970]: pgmap v2401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:52:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:52:58 smithi163 ceph-mon[36873]: pgmap v2401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:00 smithi042 ceph-mon[32775]: pgmap v2402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:00 smithi164 ceph-mon[36970]: pgmap v2402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:00 smithi163 ceph-mon[36873]: pgmap v2402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:01.865 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3411.185957 secs 2024-07-26T07:53:01.865 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3428.928391 secs 2024-07-26T07:53:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3411.185957 secs 2024-07-26T07:53:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3428.928391 secs 2024-07-26T07:53:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3411.185957 secs 2024-07-26T07:53:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3428.928391 secs 2024-07-26T07:53:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:02 smithi042 ceph-mon[32775]: pgmap v2403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:02 smithi164 ceph-mon[36970]: pgmap v2403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:02 smithi163 ceph-mon[36873]: pgmap v2403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:04.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:04 smithi042 ceph-mon[32775]: pgmap v2404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:04.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:04 smithi164 ceph-mon[36970]: pgmap v2404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:05.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:04 smithi163 ceph-mon[36873]: pgmap v2404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:06 smithi042 ceph-mon[32775]: pgmap v2405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3416.186087 secs 2024-07-26T07:53:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3433.928513 secs 2024-07-26T07:53:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:06 smithi164 ceph-mon[36970]: pgmap v2405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3416.186087 secs 2024-07-26T07:53:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3433.928513 secs 2024-07-26T07:53:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:06 smithi163 ceph-mon[36873]: pgmap v2405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3416.186087 secs 2024-07-26T07:53:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3433.928513 secs 2024-07-26T07:53:08.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:08 smithi042 ceph-mon[32775]: pgmap v2406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:08.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:08 smithi164 ceph-mon[36970]: pgmap v2406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:09.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:08 smithi163 ceph-mon[36873]: pgmap v2406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:09.408 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:53:09.407+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:53:09.408 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:53:09.407+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:53:09.408 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:53:09.407+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:53:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:10 smithi042 ceph-mon[32775]: pgmap v2407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:10 smithi164 ceph-mon[36970]: pgmap v2407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:10 smithi163 ceph-mon[36873]: pgmap v2407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3421.186249 secs 2024-07-26T07:53:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3438.928639 secs 2024-07-26T07:53:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3421.186249 secs 2024-07-26T07:53:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3438.928639 secs 2024-07-26T07:53:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3421.186249 secs 2024-07-26T07:53:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3438.928639 secs 2024-07-26T07:53:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:12 smithi042 ceph-mon[32775]: pgmap v2408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:12 smithi164 ceph-mon[36970]: pgmap v2408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:12 smithi163 ceph-mon[36873]: pgmap v2408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:53:13.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:53:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:53:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:14 smithi042 ceph-mon[32775]: pgmap v2409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:53:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:53:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:53:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:14 smithi164 ceph-mon[36970]: pgmap v2409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:53:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:53:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:53:15.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:14 smithi163 ceph-mon[36873]: pgmap v2409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:15.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:53:15.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:53:15.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:53:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:15 smithi042 ceph-mon[32775]: pgmap v2410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:53:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:15 smithi042 ceph-mon[32775]: pgmap v2411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:53:15.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:15 smithi164 ceph-mon[36970]: pgmap v2410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:53:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:15 smithi164 ceph-mon[36970]: pgmap v2411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:53:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:15 smithi163 ceph-mon[36873]: pgmap v2410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:53:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:15 smithi163 ceph-mon[36873]: pgmap v2411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:53:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3426.186378 secs 2024-07-26T07:53:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3443.928719 secs 2024-07-26T07:53:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3426.186378 secs 2024-07-26T07:53:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3443.928719 secs 2024-07-26T07:53:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3426.186378 secs 2024-07-26T07:53:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3443.928719 secs 2024-07-26T07:53:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:17 smithi042 ceph-mon[32775]: pgmap v2412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:53:17.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:17 smithi164 ceph-mon[36970]: pgmap v2412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:53:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:17 smithi163 ceph-mon[36873]: pgmap v2412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:53:19.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:19 smithi042 ceph-mon[32775]: pgmap v2413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:53:19.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:19 smithi164 ceph-mon[36970]: pgmap v2413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:53:20.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:19 smithi163 ceph-mon[36873]: pgmap v2413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:53:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:21 smithi042 ceph-mon[32775]: pgmap v2414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:53:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3431.186505 secs 2024-07-26T07:53:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3448.928838 secs 2024-07-26T07:53:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:21 smithi164 ceph-mon[36970]: pgmap v2414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:53:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3431.186505 secs 2024-07-26T07:53:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3448.928838 secs 2024-07-26T07:53:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:21 smithi163 ceph-mon[36873]: pgmap v2414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:53:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3431.186505 secs 2024-07-26T07:53:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3448.928838 secs 2024-07-26T07:53:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:23 smithi042 ceph-mon[32775]: pgmap v2415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:53:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:23 smithi164 ceph-mon[36970]: pgmap v2415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:53:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:23 smithi163 ceph-mon[36873]: pgmap v2415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:53:24.411 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:53:24.411+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:53:24.412 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:53:24.411+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:53:24.412 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:53:24.411+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:53:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:25 smithi042 ceph-mon[32775]: pgmap v2416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:53:25.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:25 smithi164 ceph-mon[36970]: pgmap v2416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:53:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:25 smithi163 ceph-mon[36873]: pgmap v2416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:53:26.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3436.186654 secs 2024-07-26T07:53:26.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3453.928962 secs 2024-07-26T07:53:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3436.186654 secs 2024-07-26T07:53:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3453.928962 secs 2024-07-26T07:53:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3436.186654 secs 2024-07-26T07:53:27.157 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3453.928962 secs 2024-07-26T07:53:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:27 smithi164 ceph-mon[36970]: pgmap v2417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:27 smithi163 ceph-mon[36873]: pgmap v2417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:27 smithi042 ceph-mon[32775]: pgmap v2417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:29.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:29 smithi164 ceph-mon[36970]: pgmap v2418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:29 smithi163 ceph-mon[36873]: pgmap v2418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:29 smithi042 ceph-mon[32775]: pgmap v2418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:31 smithi164 ceph-mon[36970]: pgmap v2419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3441.186786 secs 2024-07-26T07:53:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3458.929087 secs 2024-07-26T07:53:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:31 smithi163 ceph-mon[36873]: pgmap v2419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3441.186786 secs 2024-07-26T07:53:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3458.929087 secs 2024-07-26T07:53:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:31 smithi042 ceph-mon[32775]: pgmap v2419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3441.186786 secs 2024-07-26T07:53:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3458.929087 secs 2024-07-26T07:53:33.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:33 smithi164 ceph-mon[36970]: pgmap v2420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:34.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:33 smithi163 ceph-mon[36873]: pgmap v2420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:34.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:33 smithi042 ceph-mon[32775]: pgmap v2420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:35.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:35 smithi164 ceph-mon[36970]: pgmap v2421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:53:36.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:35 smithi163 ceph-mon[36873]: pgmap v2421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:53:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:35 smithi042 ceph-mon[32775]: pgmap v2421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:53:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3446.186940 secs 2024-07-26T07:53:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3463.929185 secs 2024-07-26T07:53:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3446.186940 secs 2024-07-26T07:53:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3463.929185 secs 2024-07-26T07:53:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3446.186940 secs 2024-07-26T07:53:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3463.929185 secs 2024-07-26T07:53:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:37 smithi164 ceph-mon[36970]: pgmap v2422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:37 smithi163 ceph-mon[36873]: pgmap v2422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:37 smithi042 ceph-mon[32775]: pgmap v2422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:39.420 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:53:39.419+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:53:39.420 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:53:39.419+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:53:39.420 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:53:39.419+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:53:39.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:39 smithi164 ceph-mon[36970]: pgmap v2423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:40.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:39 smithi163 ceph-mon[36873]: pgmap v2423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:39 smithi042 ceph-mon[32775]: pgmap v2423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:41 smithi164 ceph-mon[36970]: pgmap v2424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3451.187073 secs 2024-07-26T07:53:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3468.929279 secs 2024-07-26T07:53:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:41 smithi163 ceph-mon[36873]: pgmap v2424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3451.187073 secs 2024-07-26T07:53:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3468.929279 secs 2024-07-26T07:53:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:41 smithi042 ceph-mon[32775]: pgmap v2424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3451.187073 secs 2024-07-26T07:53:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3468.929279 secs 2024-07-26T07:53:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:53:43.402+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:53:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:53:43.402+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:53:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:53:43.403+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:53:43.404 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:53:43.403+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:53:43.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:43 smithi164 ceph-mon[36970]: pgmap v2425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:43 smithi163 ceph-mon[36873]: pgmap v2425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:43 smithi042 ceph-mon[32775]: pgmap v2425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:45.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:45 smithi164 ceph-mon[36970]: pgmap v2426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:53:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:45 smithi163 ceph-mon[36873]: pgmap v2426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:53:46.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:45 smithi042 ceph-mon[32775]: pgmap v2426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:53:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3456.187230 secs 2024-07-26T07:53:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3473.929357 secs 2024-07-26T07:53:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3456.187230 secs 2024-07-26T07:53:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3473.929357 secs 2024-07-26T07:53:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3456.187230 secs 2024-07-26T07:53:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3473.929357 secs 2024-07-26T07:53:47.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:47 smithi164 ceph-mon[36970]: pgmap v2427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:47 smithi163 ceph-mon[36873]: pgmap v2427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:47 smithi042 ceph-mon[32775]: pgmap v2427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:49.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:49 smithi164 ceph-mon[36970]: pgmap v2428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:49 smithi163 ceph-mon[36873]: pgmap v2428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:50.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:49 smithi042 ceph-mon[32775]: pgmap v2428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:51 smithi164 ceph-mon[36970]: pgmap v2429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3461.187359 secs 2024-07-26T07:53:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3478.929462 secs 2024-07-26T07:53:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:51 smithi163 ceph-mon[36873]: pgmap v2429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3461.187359 secs 2024-07-26T07:53:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3478.929462 secs 2024-07-26T07:53:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:51 smithi042 ceph-mon[32775]: pgmap v2429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3461.187359 secs 2024-07-26T07:53:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3478.929462 secs 2024-07-26T07:53:53.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:53 smithi164 ceph-mon[36970]: pgmap v2430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:53 smithi163 ceph-mon[36873]: pgmap v2430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:53 smithi042 ceph-mon[32775]: pgmap v2430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:53:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:53:54.422+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:53:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:53:54.422+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:53:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:53:54.422+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:53:55.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:55 smithi164 ceph-mon[36970]: pgmap v2431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:53:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:55 smithi163 ceph-mon[36873]: pgmap v2431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:53:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:55 smithi042 ceph-mon[32775]: pgmap v2431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:53:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3466.187479 secs 2024-07-26T07:53:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3483.929586 secs 2024-07-26T07:53:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3466.187479 secs 2024-07-26T07:53:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3483.929586 secs 2024-07-26T07:53:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3466.187479 secs 2024-07-26T07:53:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3483.929586 secs 2024-07-26T07:53:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:57 smithi164 ceph-mon[36970]: pgmap v2432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:57 smithi163 ceph-mon[36873]: pgmap v2432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:57 smithi042 ceph-mon[32775]: pgmap v2432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:53:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:53:59 smithi164 ceph-mon[36970]: pgmap v2433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:53:59 smithi163 ceph-mon[36873]: pgmap v2433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:53:59 smithi042 ceph-mon[32775]: pgmap v2433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:01 smithi164 ceph-mon[36970]: pgmap v2434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3471.187604 secs 2024-07-26T07:54:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3488.929708 secs 2024-07-26T07:54:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:01 smithi163 ceph-mon[36873]: pgmap v2434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3471.187604 secs 2024-07-26T07:54:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3488.929708 secs 2024-07-26T07:54:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:01 smithi042 ceph-mon[32775]: pgmap v2434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3471.187604 secs 2024-07-26T07:54:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3488.929708 secs 2024-07-26T07:54:04.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:03 smithi163 ceph-mon[36873]: pgmap v2435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:04.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:03 smithi042 ceph-mon[32775]: pgmap v2435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:03 smithi164 ceph-mon[36970]: pgmap v2435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:05 smithi163 ceph-mon[36873]: pgmap v2436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:54:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:05 smithi042 ceph-mon[32775]: pgmap v2436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:54:06.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:05 smithi164 ceph-mon[36970]: pgmap v2436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:54:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3476.187727 secs 2024-07-26T07:54:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3493.929827 secs 2024-07-26T07:54:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3476.187727 secs 2024-07-26T07:54:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3493.929827 secs 2024-07-26T07:54:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3476.187727 secs 2024-07-26T07:54:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3493.929827 secs 2024-07-26T07:54:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:07 smithi163 ceph-mon[36873]: pgmap v2437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:07 smithi042 ceph-mon[32775]: pgmap v2437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:08.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:07 smithi164 ceph-mon[36970]: pgmap v2437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:08 smithi042 ceph-mon[32775]: pgmap v2438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:09.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:08 smithi164 ceph-mon[36970]: pgmap v2438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:08 smithi163 ceph-mon[36873]: pgmap v2438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:09.430 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:54:09.430+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:54:09.431 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:54:09.430+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:54:09.431 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:54:09.430+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:54:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:10 smithi164 ceph-mon[36970]: pgmap v2439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:10 smithi163 ceph-mon[36873]: pgmap v2439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:10 smithi042 ceph-mon[32775]: pgmap v2439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3481.187851 secs 2024-07-26T07:54:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3498.929958 secs 2024-07-26T07:54:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3481.187851 secs 2024-07-26T07:54:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3498.929958 secs 2024-07-26T07:54:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3481.187851 secs 2024-07-26T07:54:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3498.929958 secs 2024-07-26T07:54:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:12 smithi164 ceph-mon[36970]: pgmap v2440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:12 smithi163 ceph-mon[36873]: pgmap v2440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:12 smithi042 ceph-mon[32775]: pgmap v2440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:15.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:14 smithi164 ceph-mon[36970]: pgmap v2441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:54:15.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:54:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:54:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:54:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:14 smithi164 ceph-mon[36970]: pgmap v2442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:54:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:14 smithi164 ceph-mon[36970]: pgmap v2443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:54:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:54:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:14 smithi163 ceph-mon[36873]: pgmap v2441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:54:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:54:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:54:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:54:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:14 smithi163 ceph-mon[36873]: pgmap v2442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:54:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:14 smithi163 ceph-mon[36873]: pgmap v2443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:54:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:54:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:14 smithi042 ceph-mon[32775]: pgmap v2441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:54:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:54:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:54:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:54:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:14 smithi042 ceph-mon[32775]: pgmap v2442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T07:54:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:14 smithi042 ceph-mon[32775]: pgmap v2443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:54:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:54:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3486.187973 secs 2024-07-26T07:54:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3486.187973 secs 2024-07-26T07:54:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3486.187973 secs 2024-07-26T07:54:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3503.930078 secs 2024-07-26T07:54:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:17 smithi163 ceph-mon[36873]: pgmap v2444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:54:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3503.930078 secs 2024-07-26T07:54:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:17 smithi042 ceph-mon[32775]: pgmap v2444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:54:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3503.930078 secs 2024-07-26T07:54:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:17 smithi164 ceph-mon[36970]: pgmap v2444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:54:19.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:19 smithi164 ceph-mon[36970]: pgmap v2445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:54:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:19 smithi163 ceph-mon[36873]: pgmap v2445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:54:19.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:19 smithi042 ceph-mon[32775]: pgmap v2445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:54:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:21 smithi164 ceph-mon[36970]: pgmap v2446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:54:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3491.188099 secs 2024-07-26T07:54:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:21 smithi163 ceph-mon[36873]: pgmap v2446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:54:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3491.188099 secs 2024-07-26T07:54:21.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:21 smithi042 ceph-mon[32775]: pgmap v2446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:54:21.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3491.188099 secs 2024-07-26T07:54:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3508.930197 secs 2024-07-26T07:54:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3508.930197 secs 2024-07-26T07:54:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3508.930197 secs 2024-07-26T07:54:23.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:23 smithi164 ceph-mon[36970]: pgmap v2447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:54:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:23 smithi163 ceph-mon[36873]: pgmap v2447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:54:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:23 smithi042 ceph-mon[32775]: pgmap v2447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:54:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:54:24.433+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:54:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:54:24.433+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:54:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:54:24.433+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:54:25.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:25 smithi164 ceph-mon[36970]: pgmap v2448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:54:25.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:25 smithi163 ceph-mon[36873]: pgmap v2448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:54:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:25 smithi042 ceph-mon[32775]: pgmap v2448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:54:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3496.188259 secs 2024-07-26T07:54:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3496.188259 secs 2024-07-26T07:54:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3496.188259 secs 2024-07-26T07:54:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3513.930327 secs 2024-07-26T07:54:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:27 smithi164 ceph-mon[36970]: pgmap v2449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3513.930327 secs 2024-07-26T07:54:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:27 smithi163 ceph-mon[36873]: pgmap v2449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3513.930327 secs 2024-07-26T07:54:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:27 smithi042 ceph-mon[32775]: pgmap v2449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:29 smithi163 ceph-mon[36873]: pgmap v2450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:54:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:29 smithi042 ceph-mon[32775]: pgmap v2450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:54:29.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:29 smithi164 ceph-mon[36970]: pgmap v2450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:54:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:31 smithi163 ceph-mon[36873]: pgmap v2451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3501.188387 secs 2024-07-26T07:54:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:31 smithi042 ceph-mon[32775]: pgmap v2451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3501.188387 secs 2024-07-26T07:54:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:31 smithi164 ceph-mon[36970]: pgmap v2451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3501.188387 secs 2024-07-26T07:54:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3518.930451 secs 2024-07-26T07:54:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3518.930451 secs 2024-07-26T07:54:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3518.930451 secs 2024-07-26T07:54:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:33 smithi163 ceph-mon[36873]: pgmap v2452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:33 smithi042 ceph-mon[32775]: pgmap v2452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:33.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:33 smithi164 ceph-mon[36970]: pgmap v2452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:35.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:35 smithi163 ceph-mon[36873]: pgmap v2453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:35.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:35 smithi042 ceph-mon[32775]: pgmap v2453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:35.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:35 smithi164 ceph-mon[36970]: pgmap v2453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3506.188522 secs 2024-07-26T07:54:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3506.188522 secs 2024-07-26T07:54:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3506.188522 secs 2024-07-26T07:54:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3523.930592 secs 2024-07-26T07:54:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:37 smithi163 ceph-mon[36873]: pgmap v2454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3523.930592 secs 2024-07-26T07:54:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:37 smithi042 ceph-mon[32775]: pgmap v2454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3523.930592 secs 2024-07-26T07:54:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:37 smithi164 ceph-mon[36970]: pgmap v2454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:54:39.441+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:54:39.443 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:54:39.441+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:54:39.443 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:54:39.441+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:54:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:39 smithi163 ceph-mon[36873]: pgmap v2455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:54:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:39 smithi042 ceph-mon[32775]: pgmap v2455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:54:39.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:39 smithi164 ceph-mon[36970]: pgmap v2455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:54:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:41 smithi163 ceph-mon[36873]: pgmap v2456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3511.188678 secs 2024-07-26T07:54:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:41 smithi042 ceph-mon[32775]: pgmap v2456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3511.188678 secs 2024-07-26T07:54:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:41 smithi164 ceph-mon[36970]: pgmap v2456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3511.188678 secs 2024-07-26T07:54:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3528.930712 secs 2024-07-26T07:54:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3528.930712 secs 2024-07-26T07:54:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3528.930712 secs 2024-07-26T07:54:43.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:43 smithi042 ceph-mon[32775]: pgmap v2457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:43.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:43 smithi164 ceph-mon[36970]: pgmap v2457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:43 smithi163 ceph-mon[36873]: pgmap v2457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:45 smithi042 ceph-mon[32775]: pgmap v2458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:45.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:45 smithi164 ceph-mon[36970]: pgmap v2458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:45 smithi163 ceph-mon[36873]: pgmap v2458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3516.188748 secs 2024-07-26T07:54:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3516.188748 secs 2024-07-26T07:54:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3516.188748 secs 2024-07-26T07:54:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3533.930852 secs 2024-07-26T07:54:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:47 smithi042 ceph-mon[32775]: pgmap v2459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:47.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3533.930852 secs 2024-07-26T07:54:47.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:47 smithi164 ceph-mon[36970]: pgmap v2459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3533.930852 secs 2024-07-26T07:54:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:47 smithi163 ceph-mon[36873]: pgmap v2459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:49 smithi042 ceph-mon[32775]: pgmap v2460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:54:49.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:49 smithi164 ceph-mon[36970]: pgmap v2460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:54:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:49 smithi163 ceph-mon[36873]: pgmap v2460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:54:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:51 smithi042 ceph-mon[32775]: pgmap v2461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3521.188905 secs 2024-07-26T07:54:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:51 smithi164 ceph-mon[36970]: pgmap v2461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3521.188905 secs 2024-07-26T07:54:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:51 smithi163 ceph-mon[36873]: pgmap v2461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3521.188905 secs 2024-07-26T07:54:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3538.930992 secs 2024-07-26T07:54:52.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3538.930992 secs 2024-07-26T07:54:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3538.930992 secs 2024-07-26T07:54:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:53 smithi042 ceph-mon[32775]: pgmap v2462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:53.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:53 smithi164 ceph-mon[36970]: pgmap v2462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:53 smithi163 ceph-mon[36873]: pgmap v2462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:54:54.445+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:54:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:54:54.445+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:54:54.446 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:54:54.445+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:54:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:55 smithi042 ceph-mon[32775]: pgmap v2463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:55.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:55 smithi164 ceph-mon[36970]: pgmap v2463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:55 smithi163 ceph-mon[36873]: pgmap v2463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:54:56.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3526.188981 secs 2024-07-26T07:54:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3526.188981 secs 2024-07-26T07:54:56.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3526.188981 secs 2024-07-26T07:54:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3543.931122 secs 2024-07-26T07:54:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:57 smithi042 ceph-mon[32775]: pgmap v2464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3543.931122 secs 2024-07-26T07:54:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:57 smithi164 ceph-mon[36970]: pgmap v2464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3543.931122 secs 2024-07-26T07:54:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:57 smithi163 ceph-mon[36873]: pgmap v2464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:54:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:54:59 smithi042 ceph-mon[32775]: pgmap v2465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:54:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:54:59 smithi164 ceph-mon[36970]: pgmap v2465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:55:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:54:59 smithi163 ceph-mon[36873]: pgmap v2465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:55:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:01 smithi042 ceph-mon[32775]: pgmap v2466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3531.189124 secs 2024-07-26T07:55:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:01 smithi164 ceph-mon[36970]: pgmap v2466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3531.189124 secs 2024-07-26T07:55:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:01 smithi163 ceph-mon[36873]: pgmap v2466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3531.189124 secs 2024-07-26T07:55:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3548.931212 secs 2024-07-26T07:55:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3548.931212 secs 2024-07-26T07:55:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3548.931212 secs 2024-07-26T07:55:03.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:03 smithi042 ceph-mon[32775]: pgmap v2467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:03.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:03 smithi164 ceph-mon[36970]: pgmap v2467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:04.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:03 smithi163 ceph-mon[36873]: pgmap v2467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:05 smithi042 ceph-mon[32775]: pgmap v2468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:05.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:05 smithi164 ceph-mon[36970]: pgmap v2468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:05 smithi163 ceph-mon[36873]: pgmap v2468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3536.189251 secs 2024-07-26T07:55:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3536.189251 secs 2024-07-26T07:55:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3536.189251 secs 2024-07-26T07:55:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3553.931313 secs 2024-07-26T07:55:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:07 smithi042 ceph-mon[32775]: pgmap v2469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:07.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3553.931313 secs 2024-07-26T07:55:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:07 smithi164 ceph-mon[36970]: pgmap v2469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3553.931313 secs 2024-07-26T07:55:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:07 smithi163 ceph-mon[36873]: pgmap v2469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:09.454+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:55:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:09.454+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:55:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:09.454+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:55:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:09 smithi042 ceph-mon[32775]: pgmap v2470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:55:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:09 smithi164 ceph-mon[36970]: pgmap v2470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:55:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:09 smithi163 ceph-mon[36873]: pgmap v2470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:55:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:11 smithi042 ceph-mon[32775]: pgmap v2471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3541.189344 secs 2024-07-26T07:55:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:11 smithi164 ceph-mon[36970]: pgmap v2471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3541.189344 secs 2024-07-26T07:55:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:11 smithi163 ceph-mon[36873]: pgmap v2471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3541.189344 secs 2024-07-26T07:55:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3558.931412 secs 2024-07-26T07:55:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3558.931412 secs 2024-07-26T07:55:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3558.931412 secs 2024-07-26T07:55:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:13 smithi042 ceph-mon[32775]: pgmap v2472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:13.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:13 smithi164 ceph-mon[36970]: pgmap v2472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:13 smithi163 ceph-mon[36873]: pgmap v2472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:55:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:55:15.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:55:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:15 smithi042 ceph-mon[32775]: pgmap v2473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:55:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:55:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:55:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:55:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:15 smithi164 ceph-mon[36970]: pgmap v2473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:55:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:55:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:55:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:55:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:15 smithi163 ceph-mon[36873]: pgmap v2473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:55:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:55:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:55:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:55:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:16 smithi042 ceph-mon[32775]: pgmap v2474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:16 smithi042 ceph-mon[32775]: pgmap v2475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T07:55:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3546.189518 secs 2024-07-26T07:55:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3563.931546 secs 2024-07-26T07:55:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:16 smithi164 ceph-mon[36970]: pgmap v2474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:16 smithi164 ceph-mon[36970]: pgmap v2475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T07:55:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3546.189518 secs 2024-07-26T07:55:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3563.931546 secs 2024-07-26T07:55:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:16 smithi163 ceph-mon[36873]: pgmap v2474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:16 smithi163 ceph-mon[36873]: pgmap v2475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T07:55:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3546.189518 secs 2024-07-26T07:55:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3563.931546 secs 2024-07-26T07:55:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:18 smithi042 ceph-mon[32775]: pgmap v2476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T07:55:18.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:18 smithi164 ceph-mon[36970]: pgmap v2476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T07:55:19.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:18 smithi163 ceph-mon[36873]: pgmap v2476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T07:55:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:20 smithi042 ceph-mon[32775]: pgmap v2477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 959 B/s rd, 3 op/s 2024-07-26T07:55:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:20 smithi164 ceph-mon[36970]: pgmap v2477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 959 B/s rd, 3 op/s 2024-07-26T07:55:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:20 smithi163 ceph-mon[36873]: pgmap v2477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 959 B/s rd, 3 op/s 2024-07-26T07:55:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3551.189651 secs 2024-07-26T07:55:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3568.931689 secs 2024-07-26T07:55:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3551.189651 secs 2024-07-26T07:55:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3568.931689 secs 2024-07-26T07:55:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3551.189651 secs 2024-07-26T07:55:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3568.931689 secs 2024-07-26T07:55:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:22 smithi042 ceph-mon[32775]: pgmap v2478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T07:55:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:22 smithi164 ceph-mon[36970]: pgmap v2478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T07:55:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:22 smithi163 ceph-mon[36873]: pgmap v2478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T07:55:24.458 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:24.457+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:55:24.458 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:24.457+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:55:24.458 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:24.457+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:55:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:24 smithi042 ceph-mon[32775]: pgmap v2479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:55:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:24 smithi164 ceph-mon[36970]: pgmap v2479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:55:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:24 smithi163 ceph-mon[36873]: pgmap v2479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:55:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:26 smithi042 ceph-mon[32775]: pgmap v2480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:55:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3556.189773 secs 2024-07-26T07:55:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3573.931833 secs 2024-07-26T07:55:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:26 smithi164 ceph-mon[36970]: pgmap v2480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:55:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3556.189773 secs 2024-07-26T07:55:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3573.931833 secs 2024-07-26T07:55:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:26 smithi163 ceph-mon[36873]: pgmap v2480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:55:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3556.189773 secs 2024-07-26T07:55:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3573.931833 secs 2024-07-26T07:55:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:28 smithi042 ceph-mon[32775]: pgmap v2481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:28 smithi164 ceph-mon[36970]: pgmap v2481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:29.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:28 smithi163 ceph-mon[36873]: pgmap v2481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:30 smithi042 ceph-mon[32775]: pgmap v2482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:55:30.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:30 smithi164 ceph-mon[36970]: pgmap v2482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:55:31.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:30 smithi163 ceph-mon[36873]: pgmap v2482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:55:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3561.189899 secs 2024-07-26T07:55:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3578.931961 secs 2024-07-26T07:55:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3561.189899 secs 2024-07-26T07:55:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3578.931961 secs 2024-07-26T07:55:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3561.189899 secs 2024-07-26T07:55:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3578.931961 secs 2024-07-26T07:55:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:32 smithi042 ceph-mon[32775]: pgmap v2483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:32 smithi164 ceph-mon[36970]: pgmap v2483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:32 smithi163 ceph-mon[36873]: pgmap v2483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:34.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:34 smithi042 ceph-mon[32775]: pgmap v2484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:34.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:34 smithi164 ceph-mon[36970]: pgmap v2484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:35.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:34 smithi163 ceph-mon[36873]: pgmap v2484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:36 smithi042 ceph-mon[32775]: pgmap v2485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3566.190020 secs 2024-07-26T07:55:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3583.932084 secs 2024-07-26T07:55:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:36 smithi164 ceph-mon[36970]: pgmap v2485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3566.190020 secs 2024-07-26T07:55:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3583.932084 secs 2024-07-26T07:55:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:36 smithi163 ceph-mon[36873]: pgmap v2485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3566.190020 secs 2024-07-26T07:55:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3583.932084 secs 2024-07-26T07:55:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:38 smithi042 ceph-mon[32775]: pgmap v2486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:38 smithi164 ceph-mon[36970]: pgmap v2486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:38 smithi163 ceph-mon[36873]: pgmap v2486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:39.465+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:55:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:39.465+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:55:39.466 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:39.465+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:55:40.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:40 smithi164 ceph-mon[36970]: pgmap v2487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:55:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:40 smithi163 ceph-mon[36873]: pgmap v2487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:55:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:40 smithi042 ceph-mon[32775]: pgmap v2487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:55:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3571.190144 secs 2024-07-26T07:55:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3588.932214 secs 2024-07-26T07:55:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3571.190144 secs 2024-07-26T07:55:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3588.932214 secs 2024-07-26T07:55:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3571.190144 secs 2024-07-26T07:55:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3588.932214 secs 2024-07-26T07:55:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:42 smithi164 ceph-mon[36970]: pgmap v2488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:42 smithi163 ceph-mon[36873]: pgmap v2488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:42 smithi042 ceph-mon[32775]: pgmap v2488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:44 smithi164 ceph-mon[36970]: pgmap v2489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:44 smithi163 ceph-mon[36873]: pgmap v2489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:44 smithi042 ceph-mon[32775]: pgmap v2489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:46 smithi164 ceph-mon[36970]: pgmap v2490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3576.190295 secs 2024-07-26T07:55:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3593.932337 secs 2024-07-26T07:55:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:46 smithi163 ceph-mon[36873]: pgmap v2490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3576.190295 secs 2024-07-26T07:55:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3593.932337 secs 2024-07-26T07:55:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:46 smithi042 ceph-mon[32775]: pgmap v2490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3576.190295 secs 2024-07-26T07:55:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3593.932337 secs 2024-07-26T07:55:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:48 smithi164 ceph-mon[36970]: pgmap v2491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:48 smithi163 ceph-mon[36873]: pgmap v2491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:48 smithi042 ceph-mon[32775]: pgmap v2491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:50 smithi164 ceph-mon[36970]: pgmap v2492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:55:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:50 smithi163 ceph-mon[36873]: pgmap v2492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:55:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:50 smithi042 ceph-mon[32775]: pgmap v2492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:55:51.189 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 745, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() TimeoutError 2024-07-26T07:55:51.256 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() TimeoutError 2024-07-26T07:55:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3581.190425 secs 2024-07-26T07:55:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3598.932478 secs 2024-07-26T07:55:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3581.190425 secs 2024-07-26T07:55:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3598.932478 secs 2024-07-26T07:55:52.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3581.190425 secs 2024-07-26T07:55:52.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3598.932478 secs 2024-07-26T07:55:52.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:52 smithi164 ceph-mon[36970]: pgmap v2493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:52 smithi163 ceph-mon[36873]: pgmap v2493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:52 smithi042 ceph-mon[32775]: pgmap v2493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:54.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:54.469+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:55:54.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:54.469+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:55:54.469 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:54.469+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:55:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:54 smithi164 ceph-mon[36970]: pgmap v2494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:54 smithi163 ceph-mon[36873]: pgmap v2494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:54 smithi042 ceph-mon[32775]: pgmap v2494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:56 smithi164 ceph-mon[36970]: pgmap v2495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3586.190548 secs 2024-07-26T07:55:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3603.932619 secs 2024-07-26T07:55:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:56 smithi163 ceph-mon[36873]: pgmap v2495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3586.190548 secs 2024-07-26T07:55:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3603.932619 secs 2024-07-26T07:55:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:56 smithi042 ceph-mon[32775]: pgmap v2495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:55:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3586.190548 secs 2024-07-26T07:55:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3603.932619 secs 2024-07-26T07:55:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:57.865+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T07:55:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:57.865+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T07:55:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:57.865+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff6780315a0 tx=0x7ff67804b070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T07:55:57.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:57.865+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff6780315a0 tx=0x7ff67804b070 comp rx=0 tx=0).stop 2024-07-26T07:55:57.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:57.865+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T07:55:57.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:57.865+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T07:55:57.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:57.865+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T07:55:57.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:57.865+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T07:55:57.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:57.865+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T07:55:57.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:57.866+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b45c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff678031980 tx=0x7ff67803e040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T07:55:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:58.424+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:55:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:58.424+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:55:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:58.424+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:55:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:55:58.424+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:55:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:55:58 smithi164 ceph-mon[36970]: pgmap v2496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:55:58 smithi163 ceph-mon[36873]: pgmap v2496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:55:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:55:58 smithi042 ceph-mon[32775]: pgmap v2496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:00 smithi164 ceph-mon[36970]: pgmap v2497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:56:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:00 smithi163 ceph-mon[36873]: pgmap v2497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:56:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:00 smithi042 ceph-mon[32775]: pgmap v2497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:56:01.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3591.190682 secs 2024-07-26T07:56:01.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3608.932739 secs 2024-07-26T07:56:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3591.190682 secs 2024-07-26T07:56:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3608.932739 secs 2024-07-26T07:56:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3591.190682 secs 2024-07-26T07:56:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3608.932739 secs 2024-07-26T07:56:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:02 smithi164 ceph-mon[36970]: pgmap v2498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:02 smithi163 ceph-mon[36873]: pgmap v2498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:02 smithi042 ceph-mon[32775]: pgmap v2498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:04.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:04 smithi164 ceph-mon[36970]: pgmap v2499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:05.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:04 smithi163 ceph-mon[36873]: pgmap v2499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:04 smithi042 ceph-mon[32775]: pgmap v2499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:06 smithi164 ceph-mon[36970]: pgmap v2500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3596.190811 secs 2024-07-26T07:56:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3613.932868 secs 2024-07-26T07:56:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:06 smithi163 ceph-mon[36873]: pgmap v2500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3596.190811 secs 2024-07-26T07:56:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3613.932868 secs 2024-07-26T07:56:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:06 smithi042 ceph-mon[32775]: pgmap v2500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3596.190811 secs 2024-07-26T07:56:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3613.932868 secs 2024-07-26T07:56:08.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:08 smithi164 ceph-mon[36970]: pgmap v2501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:09.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:08 smithi163 ceph-mon[36873]: pgmap v2501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:08 smithi042 ceph-mon[32775]: pgmap v2501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:56:09.476+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:56:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:56:09.476+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:56:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:56:09.476+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:56:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:10 smithi164 ceph-mon[36970]: pgmap v2502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:56:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:10 smithi163 ceph-mon[36873]: pgmap v2502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:56:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:10 smithi042 ceph-mon[32775]: pgmap v2502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:56:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3601.190933 secs 2024-07-26T07:56:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3618.933015 secs 2024-07-26T07:56:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3601.190933 secs 2024-07-26T07:56:12.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3618.933015 secs 2024-07-26T07:56:12.261 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3601.190933 secs 2024-07-26T07:56:12.261 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3618.933015 secs 2024-07-26T07:56:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:12 smithi164 ceph-mon[36970]: pgmap v2503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:12 smithi163 ceph-mon[36873]: pgmap v2503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:12 smithi042 ceph-mon[32775]: pgmap v2503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:14 smithi164 ceph-mon[36970]: pgmap v2504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:15.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:14 smithi163 ceph-mon[36873]: pgmap v2504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:14 smithi042 ceph-mon[32775]: pgmap v2504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:56:16.018 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:56:16.018 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:56:16.018 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:56:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:56:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:56:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:56:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:56:16.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:56:16.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:56:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:56:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:56:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:16 smithi163 ceph-mon[36873]: pgmap v2505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:16 smithi163 ceph-mon[36873]: pgmap v2506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:56:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:16 smithi163 ceph-mon[36873]: pgmap v2507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:56:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3606.191093 secs 2024-07-26T07:56:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3623.933105 secs 2024-07-26T07:56:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:16 smithi042 ceph-mon[32775]: pgmap v2505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:16 smithi042 ceph-mon[32775]: pgmap v2506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:56:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:16 smithi042 ceph-mon[32775]: pgmap v2507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:56:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3606.191093 secs 2024-07-26T07:56:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3623.933105 secs 2024-07-26T07:56:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:16 smithi164 ceph-mon[36970]: pgmap v2505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:16 smithi164 ceph-mon[36970]: pgmap v2506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:56:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:16 smithi164 ceph-mon[36970]: pgmap v2507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:56:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3606.191093 secs 2024-07-26T07:56:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3623.933105 secs 2024-07-26T07:56:19.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:18 smithi163 ceph-mon[36873]: pgmap v2508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:56:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:18 smithi042 ceph-mon[32775]: pgmap v2508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:56:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:18 smithi164 ceph-mon[36970]: pgmap v2508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:56:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:20 smithi163 ceph-mon[36873]: pgmap v2509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:56:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:20 smithi042 ceph-mon[32775]: pgmap v2509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:56:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:20 smithi164 ceph-mon[36970]: pgmap v2509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:56:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3611.191164 secs 2024-07-26T07:56:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3628.933225 secs 2024-07-26T07:56:22.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3611.191164 secs 2024-07-26T07:56:22.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3628.933225 secs 2024-07-26T07:56:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3611.191164 secs 2024-07-26T07:56:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3628.933225 secs 2024-07-26T07:56:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:22 smithi163 ceph-mon[36873]: pgmap v2510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:56:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:22 smithi042 ceph-mon[32775]: pgmap v2510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:56:23.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:22 smithi164 ceph-mon[36970]: pgmap v2510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:56:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:56:24.479+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:56:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:56:24.479+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:56:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:56:24.479+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:56:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:24 smithi163 ceph-mon[36873]: pgmap v2511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:56:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:24 smithi042 ceph-mon[32775]: pgmap v2511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:56:25.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:24 smithi164 ceph-mon[36970]: pgmap v2511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:56:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:26 smithi163 ceph-mon[36873]: pgmap v2512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:56:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3616.191296 secs 2024-07-26T07:56:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3633.933375 secs 2024-07-26T07:56:27.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:26 smithi164 ceph-mon[36970]: pgmap v2512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:56:27.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3616.191296 secs 2024-07-26T07:56:27.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3633.933375 secs 2024-07-26T07:56:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:26 smithi042 ceph-mon[32775]: pgmap v2512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:56:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3616.191296 secs 2024-07-26T07:56:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3633.933375 secs 2024-07-26T07:56:29.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:28 smithi163 ceph-mon[36873]: pgmap v2513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:28 smithi042 ceph-mon[32775]: pgmap v2513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:29.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:28 smithi164 ceph-mon[36970]: pgmap v2513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:31.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:30 smithi163 ceph-mon[36873]: pgmap v2514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:30 smithi042 ceph-mon[32775]: pgmap v2514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:31.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:30 smithi164 ceph-mon[36970]: pgmap v2514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3621.191434 secs 2024-07-26T07:56:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3638.933448 secs 2024-07-26T07:56:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3621.191434 secs 2024-07-26T07:56:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3638.933448 secs 2024-07-26T07:56:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3621.191434 secs 2024-07-26T07:56:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3638.933448 secs 2024-07-26T07:56:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:32 smithi042 ceph-mon[32775]: pgmap v2515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:32 smithi164 ceph-mon[36970]: pgmap v2515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:32 smithi163 ceph-mon[36873]: pgmap v2515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:35.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:34 smithi042 ceph-mon[32775]: pgmap v2516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:56:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:34 smithi164 ceph-mon[36970]: pgmap v2516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:56:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:34 smithi163 ceph-mon[36873]: pgmap v2516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:56:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:35 smithi042 ceph-mon[32775]: pgmap v2517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:36.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:35 smithi164 ceph-mon[36970]: pgmap v2517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:35 smithi163 ceph-mon[36873]: pgmap v2517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3626.191564 secs 2024-07-26T07:56:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3643.933520 secs 2024-07-26T07:56:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3626.191564 secs 2024-07-26T07:56:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3643.933520 secs 2024-07-26T07:56:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3626.191564 secs 2024-07-26T07:56:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3643.933520 secs 2024-07-26T07:56:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:37 smithi042 ceph-mon[32775]: pgmap v2518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:38.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:37 smithi164 ceph-mon[36970]: pgmap v2518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:37 smithi163 ceph-mon[36873]: pgmap v2518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:56:39.488+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:56:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:56:39.488+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:56:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:56:39.488+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:56:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:40 smithi042 ceph-mon[32775]: pgmap v2519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:40.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:40 smithi164 ceph-mon[36970]: pgmap v2519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:41.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:40 smithi163 ceph-mon[36873]: pgmap v2519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3631.191705 secs 2024-07-26T07:56:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3631.191705 secs 2024-07-26T07:56:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3631.191705 secs 2024-07-26T07:56:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3648.933642 secs 2024-07-26T07:56:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:42 smithi042 ceph-mon[32775]: pgmap v2520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3648.933642 secs 2024-07-26T07:56:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:42 smithi164 ceph-mon[36970]: pgmap v2520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3648.933642 secs 2024-07-26T07:56:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:42 smithi163 ceph-mon[36873]: pgmap v2520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:44 smithi042 ceph-mon[32775]: pgmap v2521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:56:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:44 smithi164 ceph-mon[36970]: pgmap v2521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:56:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:44 smithi163 ceph-mon[36873]: pgmap v2521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:56:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:46 smithi042 ceph-mon[32775]: pgmap v2522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3636.191862 secs 2024-07-26T07:56:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:46 smithi164 ceph-mon[36970]: pgmap v2522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3636.191862 secs 2024-07-26T07:56:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:46 smithi163 ceph-mon[36873]: pgmap v2522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3636.191862 secs 2024-07-26T07:56:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3653.933764 secs 2024-07-26T07:56:47.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3653.933764 secs 2024-07-26T07:56:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3653.933764 secs 2024-07-26T07:56:48.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:48 smithi042 ceph-mon[32775]: pgmap v2523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:48.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:48 smithi164 ceph-mon[36970]: pgmap v2523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:48 smithi163 ceph-mon[36873]: pgmap v2523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:50 smithi042 ceph-mon[32775]: pgmap v2524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:50 smithi164 ceph-mon[36970]: pgmap v2524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:50 smithi163 ceph-mon[36873]: pgmap v2524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3641.191977 secs 2024-07-26T07:56:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3641.191977 secs 2024-07-26T07:56:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3641.191977 secs 2024-07-26T07:56:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3658.933883 secs 2024-07-26T07:56:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:52 smithi042 ceph-mon[32775]: pgmap v2525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:52.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3658.933883 secs 2024-07-26T07:56:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:52 smithi164 ceph-mon[36970]: pgmap v2525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3658.933883 secs 2024-07-26T07:56:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:52 smithi163 ceph-mon[36873]: pgmap v2525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:56:54.491+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:56:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:56:54.491+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:56:54.492 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:56:54.491+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:56:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:54 smithi042 ceph-mon[32775]: pgmap v2526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:56:54.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:54 smithi164 ceph-mon[36970]: pgmap v2526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:56:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:54 smithi163 ceph-mon[36873]: pgmap v2526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:56:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:56 smithi042 ceph-mon[32775]: pgmap v2527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3646.192129 secs 2024-07-26T07:56:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3663.934004 secs 2024-07-26T07:56:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:56 smithi164 ceph-mon[36970]: pgmap v2527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3646.192129 secs 2024-07-26T07:56:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3663.934004 secs 2024-07-26T07:56:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:56 smithi163 ceph-mon[36873]: pgmap v2527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:56:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3646.192129 secs 2024-07-26T07:56:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3663.934004 secs 2024-07-26T07:56:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:56:58 smithi042 ceph-mon[32775]: pgmap v2528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:58.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:56:58 smithi164 ceph-mon[36970]: pgmap v2528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:56:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:56:58 smithi163 ceph-mon[36873]: pgmap v2528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:00 smithi042 ceph-mon[32775]: pgmap v2529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:00.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:00 smithi164 ceph-mon[36970]: pgmap v2529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:00 smithi163 ceph-mon[36873]: pgmap v2529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3651.192291 secs 2024-07-26T07:57:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3668.934126 secs 2024-07-26T07:57:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3651.192291 secs 2024-07-26T07:57:02.257 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3668.934126 secs 2024-07-26T07:57:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3651.192291 secs 2024-07-26T07:57:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3668.934126 secs 2024-07-26T07:57:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:02 smithi042 ceph-mon[32775]: pgmap v2530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:02 smithi164 ceph-mon[36970]: pgmap v2530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:02 smithi163 ceph-mon[36873]: pgmap v2530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:04.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:04 smithi042 ceph-mon[32775]: pgmap v2531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:57:04.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:04 smithi164 ceph-mon[36970]: pgmap v2531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:57:05.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:04 smithi163 ceph-mon[36873]: pgmap v2531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:57:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:06 smithi042 ceph-mon[32775]: pgmap v2532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3656.192389 secs 2024-07-26T07:57:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3673.934245 secs 2024-07-26T07:57:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:06 smithi164 ceph-mon[36970]: pgmap v2532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3656.192389 secs 2024-07-26T07:57:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3673.934245 secs 2024-07-26T07:57:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:06 smithi163 ceph-mon[36873]: pgmap v2532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3656.192389 secs 2024-07-26T07:57:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3673.934245 secs 2024-07-26T07:57:08.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:08 smithi042 ceph-mon[32775]: pgmap v2533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:08.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:08 smithi164 ceph-mon[36970]: pgmap v2533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:09.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:08 smithi163 ceph-mon[36873]: pgmap v2533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:09.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:57:09.500+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:57:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:57:09.500+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:57:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:57:09.500+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:57:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:10 smithi042 ceph-mon[32775]: pgmap v2534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:10.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:10 smithi164 ceph-mon[36970]: pgmap v2534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:10 smithi163 ceph-mon[36873]: pgmap v2534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3661.192511 secs 2024-07-26T07:57:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3678.934346 secs 2024-07-26T07:57:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3661.192511 secs 2024-07-26T07:57:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3678.934346 secs 2024-07-26T07:57:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3661.192511 secs 2024-07-26T07:57:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3678.934346 secs 2024-07-26T07:57:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:12 smithi042 ceph-mon[32775]: pgmap v2535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:12 smithi164 ceph-mon[36970]: pgmap v2535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:12 smithi163 ceph-mon[36873]: pgmap v2535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:14 smithi042 ceph-mon[32775]: pgmap v2536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:57:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:14 smithi164 ceph-mon[36970]: pgmap v2536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:57:15.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:14 smithi163 ceph-mon[36873]: pgmap v2536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:57:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:57:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:57:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:57:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: pgmap v2537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3666.192627 secs 2024-07-26T07:57:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:57:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:57:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:57:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3683.934504 secs 2024-07-26T07:57:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: pgmap v2537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3666.192627 secs 2024-07-26T07:57:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:57:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:57:17.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:57:17.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3683.934504 secs 2024-07-26T07:57:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: pgmap v2537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3666.192627 secs 2024-07-26T07:57:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T07:57:17.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:57:17.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:57:17.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:57:17.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3683.934504 secs 2024-07-26T07:57:17.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:17 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:57:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:17 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:57:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:17 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:57:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:17 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:57:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:17 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:57:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:17 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:57:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:17 smithi164 ceph-mon[36970]: pgmap v2538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:57:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:17 smithi164 ceph-mon[36970]: pgmap v2539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:57:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:17 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:57:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:17 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:57:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:17 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:57:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:17 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:57:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:17 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:57:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:17 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:57:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:17 smithi163 ceph-mon[36873]: pgmap v2538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:57:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:17 smithi163 ceph-mon[36873]: pgmap v2539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:57:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:17 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T07:57:18.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:17 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T07:57:18.351 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:17 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T07:57:18.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:17 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T07:57:18.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:17 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T07:57:18.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:17 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T07:57:18.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:17 smithi042 ceph-mon[32775]: pgmap v2538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:57:18.352 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:17 smithi042 ceph-mon[32775]: pgmap v2539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:57:19.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:19 smithi164 ceph-mon[36970]: pgmap v2540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:57:20.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:19 smithi163 ceph-mon[36873]: pgmap v2540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:57:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:19 smithi042 ceph-mon[32775]: pgmap v2540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:57:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:21 smithi164 ceph-mon[36970]: pgmap v2541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:57:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3671.192753 secs 2024-07-26T07:57:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3688.934628 secs 2024-07-26T07:57:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:21 smithi163 ceph-mon[36873]: pgmap v2541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:57:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3671.192753 secs 2024-07-26T07:57:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3688.934628 secs 2024-07-26T07:57:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:21 smithi042 ceph-mon[32775]: pgmap v2541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:57:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3671.192753 secs 2024-07-26T07:57:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3688.934628 secs 2024-07-26T07:57:23.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:23 smithi042 ceph-mon[32775]: pgmap v2542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:57:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:23 smithi164 ceph-mon[36970]: pgmap v2542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:57:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:23 smithi163 ceph-mon[36873]: pgmap v2542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T07:57:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:57:24.504+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:57:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:57:24.504+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:57:24.505 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:57:24.504+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:57:25.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:25 smithi164 ceph-mon[36970]: pgmap v2543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:57:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:25 smithi163 ceph-mon[36873]: pgmap v2543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:57:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:25 smithi042 ceph-mon[32775]: pgmap v2543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:57:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3676.192895 secs 2024-07-26T07:57:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3693.934770 secs 2024-07-26T07:57:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3676.192895 secs 2024-07-26T07:57:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3693.934770 secs 2024-07-26T07:57:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3676.192895 secs 2024-07-26T07:57:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3693.934770 secs 2024-07-26T07:57:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:27 smithi164 ceph-mon[36970]: pgmap v2544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:57:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:27 smithi163 ceph-mon[36873]: pgmap v2544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:57:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:27 smithi042 ceph-mon[32775]: pgmap v2544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:57:29.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:29 smithi164 ceph-mon[36970]: pgmap v2545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:29 smithi163 ceph-mon[36873]: pgmap v2545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:29 smithi042 ceph-mon[32775]: pgmap v2545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:31 smithi164 ceph-mon[36970]: pgmap v2546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3681.192996 secs 2024-07-26T07:57:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3698.934873 secs 2024-07-26T07:57:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:31 smithi163 ceph-mon[36873]: pgmap v2546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3681.192996 secs 2024-07-26T07:57:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3698.934873 secs 2024-07-26T07:57:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:31 smithi042 ceph-mon[32775]: pgmap v2546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3681.192996 secs 2024-07-26T07:57:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3698.934873 secs 2024-07-26T07:57:33.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:33 smithi164 ceph-mon[36970]: pgmap v2547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:34.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:33 smithi163 ceph-mon[36873]: pgmap v2547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:34.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:33 smithi042 ceph-mon[32775]: pgmap v2547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:35.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:35 smithi164 ceph-mon[36970]: pgmap v2548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:36.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:35 smithi163 ceph-mon[36873]: pgmap v2548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:35 smithi042 ceph-mon[32775]: pgmap v2548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3686.193101 secs 2024-07-26T07:57:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3703.935000 secs 2024-07-26T07:57:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3686.193101 secs 2024-07-26T07:57:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3703.935000 secs 2024-07-26T07:57:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3686.193101 secs 2024-07-26T07:57:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3703.935000 secs 2024-07-26T07:57:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:37 smithi164 ceph-mon[36970]: pgmap v2549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:37 smithi163 ceph-mon[36873]: pgmap v2549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:37 smithi042 ceph-mon[32775]: pgmap v2549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:39.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:57:39.512+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:57:39.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:57:39.512+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:57:39.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:57:39.512+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:57:39.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:39 smithi164 ceph-mon[36970]: pgmap v2550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:40.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:39 smithi163 ceph-mon[36873]: pgmap v2550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:39 smithi042 ceph-mon[32775]: pgmap v2550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:41.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:41 smithi164 ceph-mon[36970]: pgmap v2551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3691.193244 secs 2024-07-26T07:57:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3708.935122 secs 2024-07-26T07:57:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:41 smithi163 ceph-mon[36873]: pgmap v2551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3691.193244 secs 2024-07-26T07:57:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3708.935122 secs 2024-07-26T07:57:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:41 smithi042 ceph-mon[32775]: pgmap v2551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3691.193244 secs 2024-07-26T07:57:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3708.935122 secs 2024-07-26T07:57:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:43 smithi164 ceph-mon[36970]: pgmap v2552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:43 smithi163 ceph-mon[36873]: pgmap v2552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:43 smithi042 ceph-mon[32775]: pgmap v2552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:45.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:45 smithi164 ceph-mon[36970]: pgmap v2553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:45 smithi163 ceph-mon[36873]: pgmap v2553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:46.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:45 smithi042 ceph-mon[32775]: pgmap v2553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3696.193342 secs 2024-07-26T07:57:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3713.935245 secs 2024-07-26T07:57:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3696.193342 secs 2024-07-26T07:57:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3713.935245 secs 2024-07-26T07:57:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3696.193342 secs 2024-07-26T07:57:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3713.935245 secs 2024-07-26T07:57:47.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:47 smithi164 ceph-mon[36970]: pgmap v2554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:47 smithi163 ceph-mon[36873]: pgmap v2554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:47 smithi042 ceph-mon[32775]: pgmap v2554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:49 smithi163 ceph-mon[36873]: pgmap v2555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:50.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:49 smithi042 ceph-mon[32775]: pgmap v2555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:50.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:49 smithi164 ceph-mon[36970]: pgmap v2555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:51 smithi163 ceph-mon[36873]: pgmap v2556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3701.193490 secs 2024-07-26T07:57:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3718.935367 secs 2024-07-26T07:57:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:51 smithi042 ceph-mon[32775]: pgmap v2556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3701.193490 secs 2024-07-26T07:57:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3718.935367 secs 2024-07-26T07:57:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:51 smithi164 ceph-mon[36970]: pgmap v2556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3701.193490 secs 2024-07-26T07:57:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3718.935367 secs 2024-07-26T07:57:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:53 smithi163 ceph-mon[36873]: pgmap v2557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:53 smithi042 ceph-mon[32775]: pgmap v2557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:54.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:53 smithi164 ceph-mon[36970]: pgmap v2557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:57:54.515+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:57:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:57:54.515+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:57:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:57:54.515+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:57:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:55 smithi163 ceph-mon[36873]: pgmap v2558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:55 smithi042 ceph-mon[32775]: pgmap v2558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:56.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:55 smithi164 ceph-mon[36970]: pgmap v2558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:57:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3706.193553 secs 2024-07-26T07:57:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3723.935472 secs 2024-07-26T07:57:57.104 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3706.193553 secs 2024-07-26T07:57:57.104 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3723.935472 secs 2024-07-26T07:57:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3706.193553 secs 2024-07-26T07:57:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3723.935472 secs 2024-07-26T07:57:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:57 smithi163 ceph-mon[36873]: pgmap v2559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:57 smithi042 ceph-mon[32775]: pgmap v2559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:57:58.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:57 smithi164 ceph-mon[36970]: pgmap v2559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:57:59 smithi163 ceph-mon[36873]: pgmap v2560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:00.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:57:59 smithi164 ceph-mon[36970]: pgmap v2560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:57:59 smithi042 ceph-mon[32775]: pgmap v2560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:01 smithi163 ceph-mon[36873]: pgmap v2561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3711.193709 secs 2024-07-26T07:58:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3728.935593 secs 2024-07-26T07:58:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:01 smithi042 ceph-mon[32775]: pgmap v2561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3711.193709 secs 2024-07-26T07:58:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3728.935593 secs 2024-07-26T07:58:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:01 smithi164 ceph-mon[36970]: pgmap v2561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3711.193709 secs 2024-07-26T07:58:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3728.935593 secs 2024-07-26T07:58:04.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:03 smithi163 ceph-mon[36873]: pgmap v2562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:03 smithi042 ceph-mon[32775]: pgmap v2562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:03 smithi164 ceph-mon[36970]: pgmap v2562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:05 smithi163 ceph-mon[36873]: pgmap v2563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:05 smithi042 ceph-mon[32775]: pgmap v2563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:06.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:05 smithi164 ceph-mon[36970]: pgmap v2563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3716.193840 secs 2024-07-26T07:58:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3733.935712 secs 2024-07-26T07:58:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3716.193840 secs 2024-07-26T07:58:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3733.935712 secs 2024-07-26T07:58:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3716.193840 secs 2024-07-26T07:58:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3733.935712 secs 2024-07-26T07:58:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:07 smithi042 ceph-mon[32775]: pgmap v2564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:08.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:07 smithi164 ceph-mon[36970]: pgmap v2564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:07 smithi163 ceph-mon[36873]: pgmap v2564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:09.525 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:58:09.523+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:58:09.525 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:58:09.523+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:58:09.525 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:58:09.524+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:58:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:09 smithi042 ceph-mon[32775]: pgmap v2565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:10.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:09 smithi164 ceph-mon[36970]: pgmap v2565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:09 smithi163 ceph-mon[36873]: pgmap v2565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:11 smithi042 ceph-mon[32775]: pgmap v2566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3721.193917 secs 2024-07-26T07:58:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3738.935845 secs 2024-07-26T07:58:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:11 smithi164 ceph-mon[36970]: pgmap v2566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:12.292 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3721.193917 secs 2024-07-26T07:58:12.292 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3738.935845 secs 2024-07-26T07:58:12.293 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:11 smithi163 ceph-mon[36873]: pgmap v2566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:12.293 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3721.193917 secs 2024-07-26T07:58:12.293 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3738.935845 secs 2024-07-26T07:58:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:13 smithi042 ceph-mon[32775]: pgmap v2567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:14.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:13 smithi164 ceph-mon[36970]: pgmap v2567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:13 smithi163 ceph-mon[36873]: pgmap v2567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:15 smithi042 ceph-mon[32775]: pgmap v2568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:16.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:15 smithi164 ceph-mon[36970]: pgmap v2568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:15 smithi163 ceph-mon[36873]: pgmap v2568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3726.194041 secs 2024-07-26T07:58:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3743.935965 secs 2024-07-26T07:58:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:58:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:58:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:58:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:58:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3726.194041 secs 2024-07-26T07:58:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3743.935965 secs 2024-07-26T07:58:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:58:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:58:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:58:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:58:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3726.194041 secs 2024-07-26T07:58:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3743.935965 secs 2024-07-26T07:58:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:58:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:58:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:58:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:58:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:17 smithi042 ceph-mon[32775]: pgmap v2569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:17 smithi042 ceph-mon[32775]: pgmap v2570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:58:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:17 smithi042 ceph-mon[32775]: pgmap v2571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:58:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:17 smithi164 ceph-mon[36970]: pgmap v2569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:17 smithi164 ceph-mon[36970]: pgmap v2570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:58:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:17 smithi164 ceph-mon[36970]: pgmap v2571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:58:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:17 smithi163 ceph-mon[36873]: pgmap v2569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:17 smithi163 ceph-mon[36873]: pgmap v2570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T07:58:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:17 smithi163 ceph-mon[36873]: pgmap v2571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:58:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:18 smithi042 ceph-mon[32775]: pgmap v2572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:58:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:18 smithi164 ceph-mon[36970]: pgmap v2572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:58:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:18 smithi163 ceph-mon[36873]: pgmap v2572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T07:58:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:21 smithi164 ceph-mon[36970]: pgmap v2573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:58:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3731.194163 secs 2024-07-26T07:58:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3748.936086 secs 2024-07-26T07:58:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:21 smithi163 ceph-mon[36873]: pgmap v2573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:58:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3731.194163 secs 2024-07-26T07:58:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3748.936086 secs 2024-07-26T07:58:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:21 smithi042 ceph-mon[32775]: pgmap v2573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:58:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3731.194163 secs 2024-07-26T07:58:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3748.936086 secs 2024-07-26T07:58:23.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:23 smithi042 ceph-mon[32775]: pgmap v2574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:58:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:23 smithi164 ceph-mon[36970]: pgmap v2574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:58:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:23 smithi163 ceph-mon[36873]: pgmap v2574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:58:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:58:24.528+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:58:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:58:24.528+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:58:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:58:24.528+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:58:25.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:25 smithi164 ceph-mon[36970]: pgmap v2575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:58:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:25 smithi163 ceph-mon[36873]: pgmap v2575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:58:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:25 smithi042 ceph-mon[32775]: pgmap v2575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:58:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3736.194273 secs 2024-07-26T07:58:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3753.936208 secs 2024-07-26T07:58:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3736.194273 secs 2024-07-26T07:58:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3753.936208 secs 2024-07-26T07:58:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3736.194273 secs 2024-07-26T07:58:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3753.936208 secs 2024-07-26T07:58:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:27 smithi164 ceph-mon[36970]: pgmap v2576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:58:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:27 smithi163 ceph-mon[36873]: pgmap v2576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:58:28.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:27 smithi042 ceph-mon[32775]: pgmap v2576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:58:29.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:29 smithi164 ceph-mon[36970]: pgmap v2577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:58:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:29 smithi163 ceph-mon[36873]: pgmap v2577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:58:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:29 smithi042 ceph-mon[32775]: pgmap v2577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:58:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:31 smithi164 ceph-mon[36970]: pgmap v2578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3741.194397 secs 2024-07-26T07:58:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3758.936326 secs 2024-07-26T07:58:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:31 smithi163 ceph-mon[36873]: pgmap v2578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3741.194397 secs 2024-07-26T07:58:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3758.936326 secs 2024-07-26T07:58:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:31 smithi042 ceph-mon[32775]: pgmap v2578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:32.150 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3741.194397 secs 2024-07-26T07:58:32.150 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3758.936326 secs 2024-07-26T07:58:33.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:33 smithi164 ceph-mon[36970]: pgmap v2579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:34.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:33 smithi163 ceph-mon[36873]: pgmap v2579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:34.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:33 smithi042 ceph-mon[32775]: pgmap v2579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:35.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:35 smithi164 ceph-mon[36970]: pgmap v2580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:36.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:35 smithi163 ceph-mon[36873]: pgmap v2580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:35 smithi042 ceph-mon[32775]: pgmap v2580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3746.194523 secs 2024-07-26T07:58:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3763.936474 secs 2024-07-26T07:58:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3746.194523 secs 2024-07-26T07:58:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3763.936474 secs 2024-07-26T07:58:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3746.194523 secs 2024-07-26T07:58:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3763.936474 secs 2024-07-26T07:58:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:37 smithi164 ceph-mon[36970]: pgmap v2581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:37 smithi163 ceph-mon[36873]: pgmap v2581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:37 smithi042 ceph-mon[32775]: pgmap v2581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:58:39.537+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:58:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:58:39.537+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:58:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:58:39.537+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:58:39.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:39 smithi164 ceph-mon[36970]: pgmap v2582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:58:40.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:39 smithi163 ceph-mon[36873]: pgmap v2582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:58:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:39 smithi042 ceph-mon[32775]: pgmap v2582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:58:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:41 smithi163 ceph-mon[36873]: pgmap v2583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3751.194622 secs 2024-07-26T07:58:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3768.936550 secs 2024-07-26T07:58:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:41 smithi042 ceph-mon[32775]: pgmap v2583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3751.194622 secs 2024-07-26T07:58:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3768.936550 secs 2024-07-26T07:58:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:41 smithi164 ceph-mon[36970]: pgmap v2583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3751.194622 secs 2024-07-26T07:58:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3768.936550 secs 2024-07-26T07:58:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:43 smithi163 ceph-mon[36873]: pgmap v2584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:43 smithi042 ceph-mon[32775]: pgmap v2584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:44.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:43 smithi164 ceph-mon[36970]: pgmap v2584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:45 smithi163 ceph-mon[36873]: pgmap v2585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:46.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:45 smithi042 ceph-mon[32775]: pgmap v2585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:46.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:45 smithi164 ceph-mon[36970]: pgmap v2585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3756.194701 secs 2024-07-26T07:58:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3773.936691 secs 2024-07-26T07:58:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3756.194701 secs 2024-07-26T07:58:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3773.936691 secs 2024-07-26T07:58:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3756.194701 secs 2024-07-26T07:58:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3773.936691 secs 2024-07-26T07:58:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:47 smithi163 ceph-mon[36873]: pgmap v2586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:47 smithi042 ceph-mon[32775]: pgmap v2586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:48.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:47 smithi164 ceph-mon[36970]: pgmap v2586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:49 smithi163 ceph-mon[36873]: pgmap v2587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:58:50.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:49 smithi042 ceph-mon[32775]: pgmap v2587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:58:50.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:49 smithi164 ceph-mon[36970]: pgmap v2587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:58:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:51 smithi163 ceph-mon[36873]: pgmap v2588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3761.194821 secs 2024-07-26T07:58:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3778.936793 secs 2024-07-26T07:58:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:51 smithi042 ceph-mon[32775]: pgmap v2588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3761.194821 secs 2024-07-26T07:58:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3778.936793 secs 2024-07-26T07:58:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:51 smithi164 ceph-mon[36970]: pgmap v2588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3761.194821 secs 2024-07-26T07:58:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3778.936793 secs 2024-07-26T07:58:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:53 smithi163 ceph-mon[36873]: pgmap v2589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:53 smithi042 ceph-mon[32775]: pgmap v2589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:54.284 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:53 smithi164 ceph-mon[36970]: pgmap v2589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:58:54.540+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:58:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:58:54.540+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:58:54.541 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:58:54.540+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:58:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:55 smithi163 ceph-mon[36873]: pgmap v2590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:55 smithi042 ceph-mon[32775]: pgmap v2590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:56.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:55 smithi164 ceph-mon[36970]: pgmap v2590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:58:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3766.194908 secs 2024-07-26T07:58:57.162 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3783.936918 secs 2024-07-26T07:58:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3766.194908 secs 2024-07-26T07:58:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3783.936918 secs 2024-07-26T07:58:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3766.194908 secs 2024-07-26T07:58:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3783.936918 secs 2024-07-26T07:58:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:57 smithi042 ceph-mon[32775]: pgmap v2591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:58.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:57 smithi164 ceph-mon[36970]: pgmap v2591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:57 smithi163 ceph-mon[36873]: pgmap v2591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:58:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:58:58.455+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:58:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:58:58.455+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:58:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:58:58.455+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006460 con 0x7ff680102000 2024-07-26T07:58:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:58:58.455+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T07:59:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:58:59 smithi042 ceph-mon[32775]: pgmap v2592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:59:00.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:58:59 smithi164 ceph-mon[36970]: pgmap v2592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:59:00.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:58:59 smithi163 ceph-mon[36873]: pgmap v2592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:59:02.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:01 smithi164 ceph-mon[36970]: pgmap v2593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:02.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3771.195016 secs 2024-07-26T07:59:02.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3788.937016 secs 2024-07-26T07:59:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:01 smithi042 ceph-mon[32775]: pgmap v2593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3771.195016 secs 2024-07-26T07:59:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3788.937016 secs 2024-07-26T07:59:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:01 smithi163 ceph-mon[36873]: pgmap v2593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3771.195016 secs 2024-07-26T07:59:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3788.937016 secs 2024-07-26T07:59:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:03 smithi042 ceph-mon[32775]: pgmap v2594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:03 smithi164 ceph-mon[36970]: pgmap v2594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:03 smithi163 ceph-mon[36873]: pgmap v2594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:05 smithi042 ceph-mon[32775]: pgmap v2595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:06.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:05 smithi164 ceph-mon[36970]: pgmap v2595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:05 smithi163 ceph-mon[36873]: pgmap v2595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3776.195139 secs 2024-07-26T07:59:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3793.937137 secs 2024-07-26T07:59:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3776.195139 secs 2024-07-26T07:59:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3793.937137 secs 2024-07-26T07:59:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3776.195139 secs 2024-07-26T07:59:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3793.937137 secs 2024-07-26T07:59:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:07 smithi042 ceph-mon[32775]: pgmap v2596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:08.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:07 smithi164 ceph-mon[36970]: pgmap v2596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:07 smithi163 ceph-mon[36873]: pgmap v2596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:59:09.545+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:59:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:59:09.545+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:59:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:59:09.545+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:59:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:09 smithi042 ceph-mon[32775]: pgmap v2597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:59:10.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:09 smithi164 ceph-mon[36970]: pgmap v2597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:59:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:09 smithi163 ceph-mon[36873]: pgmap v2597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T07:59:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:11 smithi042 ceph-mon[32775]: pgmap v2598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3781.195289 secs 2024-07-26T07:59:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3798.937261 secs 2024-07-26T07:59:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:11 smithi164 ceph-mon[36970]: pgmap v2598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3781.195289 secs 2024-07-26T07:59:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3798.937261 secs 2024-07-26T07:59:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:11 smithi163 ceph-mon[36873]: pgmap v2598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3781.195289 secs 2024-07-26T07:59:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3798.937261 secs 2024-07-26T07:59:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:13 smithi042 ceph-mon[32775]: pgmap v2599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:14.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:13 smithi164 ceph-mon[36970]: pgmap v2599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:13 smithi163 ceph-mon[36873]: pgmap v2599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:15 smithi042 ceph-mon[32775]: pgmap v2600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:16.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:15 smithi164 ceph-mon[36970]: pgmap v2600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:15 smithi163 ceph-mon[36873]: pgmap v2600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3786.195344 secs 2024-07-26T07:59:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3803.937343 secs 2024-07-26T07:59:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:59:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3786.195344 secs 2024-07-26T07:59:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3803.937343 secs 2024-07-26T07:59:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:59:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3786.195344 secs 2024-07-26T07:59:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3803.937343 secs 2024-07-26T07:59:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T07:59:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:17 smithi042 ceph-mon[32775]: pgmap v2601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:59:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:59:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:17 smithi042 ceph-mon[32775]: pgmap v2602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:59:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:17 smithi042 ceph-mon[32775]: pgmap v2603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:59:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:59:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:17 smithi164 ceph-mon[36970]: pgmap v2601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:59:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:59:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:17 smithi164 ceph-mon[36970]: pgmap v2602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:59:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:17 smithi164 ceph-mon[36970]: pgmap v2603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:59:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:59:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:17 smithi163 ceph-mon[36873]: pgmap v2601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T07:59:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T07:59:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:17 smithi163 ceph-mon[36873]: pgmap v2602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T07:59:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:17 smithi163 ceph-mon[36873]: pgmap v2603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:59:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T07:59:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:20 smithi163 ceph-mon[36873]: pgmap v2604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:59:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:20 smithi042 ceph-mon[32775]: pgmap v2604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:59:20.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:20 smithi164 ceph-mon[36970]: pgmap v2604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:59:22.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3791.195430 secs 2024-07-26T07:59:22.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3808.937480 secs 2024-07-26T07:59:22.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:22 smithi163 ceph-mon[36873]: pgmap v2605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:59:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3791.195430 secs 2024-07-26T07:59:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3808.937480 secs 2024-07-26T07:59:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:22 smithi042 ceph-mon[32775]: pgmap v2605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:59:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3791.195430 secs 2024-07-26T07:59:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3808.937480 secs 2024-07-26T07:59:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:22 smithi164 ceph-mon[36970]: pgmap v2605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:59:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:24 smithi163 ceph-mon[36873]: pgmap v2606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:59:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:59:24.548+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:59:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:59:24.548+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:59:24.549 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:59:24.548+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:59:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:24 smithi042 ceph-mon[32775]: pgmap v2606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:59:24.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:24 smithi164 ceph-mon[36970]: pgmap v2606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T07:59:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:26 smithi163 ceph-mon[36873]: pgmap v2607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:59:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:26 smithi042 ceph-mon[32775]: pgmap v2607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:59:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:26 smithi164 ceph-mon[36970]: pgmap v2607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T07:59:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3796.195552 secs 2024-07-26T07:59:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3813.937599 secs 2024-07-26T07:59:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3796.195552 secs 2024-07-26T07:59:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3813.937599 secs 2024-07-26T07:59:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3796.195552 secs 2024-07-26T07:59:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3813.937599 secs 2024-07-26T07:59:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:28 smithi163 ceph-mon[36873]: pgmap v2608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:59:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:28 smithi042 ceph-mon[32775]: pgmap v2608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:59:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:28 smithi164 ceph-mon[36970]: pgmap v2608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T07:59:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:30 smithi163 ceph-mon[36873]: pgmap v2609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:30 smithi042 ceph-mon[32775]: pgmap v2609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:30 smithi164 ceph-mon[36970]: pgmap v2609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3801.195635 secs 2024-07-26T07:59:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3801.195635 secs 2024-07-26T07:59:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3801.195635 secs 2024-07-26T07:59:32.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3818.937727 secs 2024-07-26T07:59:32.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:32 smithi164 ceph-mon[36970]: pgmap v2610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3818.937727 secs 2024-07-26T07:59:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:32 smithi042 ceph-mon[32775]: pgmap v2610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3818.937727 secs 2024-07-26T07:59:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:32 smithi163 ceph-mon[36873]: pgmap v2610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:34 smithi042 ceph-mon[32775]: pgmap v2611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:34 smithi164 ceph-mon[36970]: pgmap v2611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:34 smithi163 ceph-mon[36873]: pgmap v2611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:36 smithi042 ceph-mon[32775]: pgmap v2612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3806.195715 secs 2024-07-26T07:59:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:36 smithi164 ceph-mon[36970]: pgmap v2612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3806.195715 secs 2024-07-26T07:59:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:36 smithi163 ceph-mon[36873]: pgmap v2612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3806.195715 secs 2024-07-26T07:59:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3823.937845 secs 2024-07-26T07:59:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3823.937845 secs 2024-07-26T07:59:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3823.937845 secs 2024-07-26T07:59:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:38 smithi042 ceph-mon[32775]: pgmap v2613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:38 smithi164 ceph-mon[36970]: pgmap v2613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:38 smithi163 ceph-mon[36873]: pgmap v2613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:59:39.554+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:59:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:59:39.554+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:59:39.556 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:59:39.554+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:59:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:40 smithi042 ceph-mon[32775]: pgmap v2614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:40 smithi164 ceph-mon[36970]: pgmap v2614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:40 smithi163 ceph-mon[36873]: pgmap v2614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3811.195843 secs 2024-07-26T07:59:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3811.195843 secs 2024-07-26T07:59:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3811.195843 secs 2024-07-26T07:59:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3828.937965 secs 2024-07-26T07:59:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:42 smithi042 ceph-mon[32775]: pgmap v2615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3828.937965 secs 2024-07-26T07:59:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:42 smithi164 ceph-mon[36970]: pgmap v2615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3828.937965 secs 2024-07-26T07:59:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:42 smithi163 ceph-mon[36873]: pgmap v2615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:44 smithi042 ceph-mon[32775]: pgmap v2616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:44 smithi164 ceph-mon[36970]: pgmap v2616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:44 smithi163 ceph-mon[36873]: pgmap v2616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:46 smithi042 ceph-mon[32775]: pgmap v2617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3816.195987 secs 2024-07-26T07:59:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:46 smithi164 ceph-mon[36970]: pgmap v2617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3816.195987 secs 2024-07-26T07:59:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:46 smithi163 ceph-mon[36873]: pgmap v2617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3816.195987 secs 2024-07-26T07:59:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3833.938092 secs 2024-07-26T07:59:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3833.938092 secs 2024-07-26T07:59:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3833.938092 secs 2024-07-26T07:59:48.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:48 smithi042 ceph-mon[32775]: pgmap v2618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:48.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:48 smithi164 ceph-mon[36970]: pgmap v2618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:48 smithi163 ceph-mon[36873]: pgmap v2618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:50 smithi042 ceph-mon[32775]: pgmap v2619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:50 smithi164 ceph-mon[36970]: pgmap v2619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:50 smithi163 ceph-mon[36873]: pgmap v2619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3821.196125 secs 2024-07-26T07:59:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3821.196125 secs 2024-07-26T07:59:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3821.196125 secs 2024-07-26T07:59:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3838.938214 secs 2024-07-26T07:59:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:52 smithi042 ceph-mon[32775]: pgmap v2620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3838.938214 secs 2024-07-26T07:59:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:52 smithi164 ceph-mon[36970]: pgmap v2620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3838.938214 secs 2024-07-26T07:59:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:52 smithi163 ceph-mon[36873]: pgmap v2620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:54.558 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:59:54.557+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T07:59:54.558 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:59:54.557+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T07:59:54.558 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T07:59:54.557+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T07:59:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:54 smithi042 ceph-mon[32775]: pgmap v2621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:54 smithi164 ceph-mon[36970]: pgmap v2621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:54 smithi163 ceph-mon[36873]: pgmap v2621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:56 smithi042 ceph-mon[32775]: pgmap v2622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3826.196271 secs 2024-07-26T07:59:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:56 smithi164 ceph-mon[36970]: pgmap v2622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3826.196271 secs 2024-07-26T07:59:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:56 smithi163 ceph-mon[36873]: pgmap v2622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T07:59:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3826.196271 secs 2024-07-26T07:59:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3843.938349 secs 2024-07-26T07:59:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3843.938349 secs 2024-07-26T07:59:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3843.938349 secs 2024-07-26T07:59:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 07:59:58 smithi042 ceph-mon[32775]: pgmap v2623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:58.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 07:59:58 smithi164 ceph-mon[36970]: pgmap v2623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T07:59:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 07:59:58 smithi163 ceph-mon[36873]: pgmap v2623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:00 smithi042 ceph-mon[32775]: pgmap v2624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:00:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:00 smithi164 ceph-mon[36970]: pgmap v2624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:00:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:00 smithi163 ceph-mon[36873]: pgmap v2624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:00:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3831.196386 secs 2024-07-26T08:00:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3831.196386 secs 2024-07-26T08:00:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3831.196386 secs 2024-07-26T08:00:02.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3848.938449 secs 2024-07-26T08:00:02.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:02 smithi164 ceph-mon[36970]: pgmap v2625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3848.938449 secs 2024-07-26T08:00:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:02 smithi042 ceph-mon[32775]: pgmap v2625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3848.938449 secs 2024-07-26T08:00:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:02 smithi163 ceph-mon[36873]: pgmap v2625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:04.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:04 smithi042 ceph-mon[32775]: pgmap v2626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:04 smithi164 ceph-mon[36970]: pgmap v2626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:04 smithi163 ceph-mon[36873]: pgmap v2626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:06 smithi042 ceph-mon[32775]: pgmap v2627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3836.196456 secs 2024-07-26T08:00:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:06 smithi164 ceph-mon[36970]: pgmap v2627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3836.196456 secs 2024-07-26T08:00:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:06 smithi163 ceph-mon[36873]: pgmap v2627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3836.196456 secs 2024-07-26T08:00:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3853.938568 secs 2024-07-26T08:00:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3853.938568 secs 2024-07-26T08:00:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3853.938568 secs 2024-07-26T08:00:08.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:08 smithi042 ceph-mon[32775]: pgmap v2628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:08 smithi164 ceph-mon[36970]: pgmap v2628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:08 smithi163 ceph-mon[36873]: pgmap v2628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:09.565 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:00:09.564+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:00:09.565 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:00:09.564+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:00:09.565 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:00:09.564+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:00:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:10 smithi042 ceph-mon[32775]: pgmap v2629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:10 smithi164 ceph-mon[36970]: pgmap v2629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:10 smithi163 ceph-mon[36873]: pgmap v2629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3841.196540 secs 2024-07-26T08:00:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3841.196540 secs 2024-07-26T08:00:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3841.196540 secs 2024-07-26T08:00:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3858.938689 secs 2024-07-26T08:00:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:12 smithi042 ceph-mon[32775]: pgmap v2630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3858.938689 secs 2024-07-26T08:00:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:12 smithi164 ceph-mon[36970]: pgmap v2630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3858.938689 secs 2024-07-26T08:00:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:12 smithi163 ceph-mon[36873]: pgmap v2630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:14 smithi042 ceph-mon[32775]: pgmap v2631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:14.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:14 smithi164 ceph-mon[36970]: pgmap v2631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:14 smithi163 ceph-mon[36873]: pgmap v2631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:16 smithi042 ceph-mon[32775]: pgmap v2632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3846.196634 secs 2024-07-26T08:00:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:16 smithi164 ceph-mon[36970]: pgmap v2632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3846.196634 secs 2024-07-26T08:00:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:16 smithi163 ceph-mon[36873]: pgmap v2632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3846.196634 secs 2024-07-26T08:00:17.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3863.938813 secs 2024-07-26T08:00:17.638 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:00:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3863.938813 secs 2024-07-26T08:00:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:00:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3863.938813 secs 2024-07-26T08:00:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:00:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:18 smithi042 ceph-mon[32775]: pgmap v2633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:18.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:18 smithi164 ceph-mon[36970]: pgmap v2633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:18 smithi163 ceph-mon[36873]: pgmap v2633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:19 smithi164 ceph-mon[36970]: pgmap v2634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:00:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:00:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:19 smithi164 ceph-mon[36970]: pgmap v2635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:00:20.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:19 smithi164 ceph-mon[36970]: pgmap v2636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:00:20.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:19 smithi163 ceph-mon[36873]: pgmap v2634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:00:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:00:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:19 smithi163 ceph-mon[36873]: pgmap v2635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:00:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:19 smithi163 ceph-mon[36873]: pgmap v2636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:00:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:19 smithi042 ceph-mon[32775]: pgmap v2634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:00:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:00:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:19 smithi042 ceph-mon[32775]: pgmap v2635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:00:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:19 smithi042 ceph-mon[32775]: pgmap v2636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:00:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:00:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3851.196777 secs 2024-07-26T08:00:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3851.196777 secs 2024-07-26T08:00:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3851.196777 secs 2024-07-26T08:00:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3868.938953 secs 2024-07-26T08:00:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:22 smithi042 ceph-mon[32775]: pgmap v2637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:00:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3868.938953 secs 2024-07-26T08:00:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:22 smithi164 ceph-mon[36970]: pgmap v2637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:00:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3868.938953 secs 2024-07-26T08:00:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:22 smithi163 ceph-mon[36873]: pgmap v2637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:00:24.569 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:00:24.568+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:00:24.569 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:00:24.568+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:00:24.569 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:00:24.568+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:00:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:24 smithi163 ceph-mon[36873]: pgmap v2638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:00:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:24 smithi042 ceph-mon[32775]: pgmap v2638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:00:25.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:24 smithi164 ceph-mon[36970]: pgmap v2638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:00:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:26 smithi042 ceph-mon[32775]: pgmap v2639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:00:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3856.196877 secs 2024-07-26T08:00:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3873.939073 secs 2024-07-26T08:00:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:26 smithi164 ceph-mon[36970]: pgmap v2639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:00:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3856.196877 secs 2024-07-26T08:00:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3873.939073 secs 2024-07-26T08:00:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:26 smithi163 ceph-mon[36873]: pgmap v2639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:00:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3856.196877 secs 2024-07-26T08:00:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3873.939073 secs 2024-07-26T08:00:29.019 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:28 smithi163 ceph-mon[36873]: pgmap v2640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:00:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:28 smithi042 ceph-mon[32775]: pgmap v2640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:00:29.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:28 smithi164 ceph-mon[36970]: pgmap v2640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:00:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:30 smithi042 ceph-mon[32775]: pgmap v2641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:00:31.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:30 smithi164 ceph-mon[36970]: pgmap v2641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:00:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:30 smithi163 ceph-mon[36873]: pgmap v2641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:00:32.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3861.197003 secs 2024-07-26T08:00:32.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3878.939194 secs 2024-07-26T08:00:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3861.197003 secs 2024-07-26T08:00:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3878.939194 secs 2024-07-26T08:00:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3861.197003 secs 2024-07-26T08:00:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3878.939194 secs 2024-07-26T08:00:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:32 smithi042 ceph-mon[32775]: pgmap v2642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:32 smithi164 ceph-mon[36970]: pgmap v2642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:32 smithi163 ceph-mon[36873]: pgmap v2642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:35.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:34 smithi042 ceph-mon[32775]: pgmap v2643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:00:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:34 smithi164 ceph-mon[36970]: pgmap v2643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:00:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:34 smithi163 ceph-mon[36873]: pgmap v2643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:00:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:36 smithi042 ceph-mon[32775]: pgmap v2644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3866.197103 secs 2024-07-26T08:00:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3883.939314 secs 2024-07-26T08:00:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:36 smithi164 ceph-mon[36970]: pgmap v2644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3866.197103 secs 2024-07-26T08:00:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3883.939314 secs 2024-07-26T08:00:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:36 smithi163 ceph-mon[36873]: pgmap v2644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3866.197103 secs 2024-07-26T08:00:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3883.939314 secs 2024-07-26T08:00:39.086 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:38 smithi163 ceph-mon[36873]: pgmap v2645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:38 smithi042 ceph-mon[32775]: pgmap v2645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:39.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:38 smithi164 ceph-mon[36970]: pgmap v2645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:39.571 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:00:39.570+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:00:39.571 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:00:39.570+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:00:39.571 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:00:39.570+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:00:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:40 smithi042 ceph-mon[32775]: pgmap v2646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:41.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:40 smithi164 ceph-mon[36970]: pgmap v2646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:40 smithi163 ceph-mon[36873]: pgmap v2646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3871.197243 secs 2024-07-26T08:00:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3888.939438 secs 2024-07-26T08:00:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3871.197243 secs 2024-07-26T08:00:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3888.939438 secs 2024-07-26T08:00:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3871.197243 secs 2024-07-26T08:00:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3888.939438 secs 2024-07-26T08:00:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:42 smithi042 ceph-mon[32775]: pgmap v2647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:43.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:42 smithi164 ceph-mon[36970]: pgmap v2647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:42 smithi163 ceph-mon[36873]: pgmap v2647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:44 smithi042 ceph-mon[32775]: pgmap v2648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:00:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:44 smithi164 ceph-mon[36970]: pgmap v2648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:00:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:44 smithi163 ceph-mon[36873]: pgmap v2648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:00:47.088 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:46 smithi042 ceph-mon[32775]: pgmap v2649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:47.088 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3876.197333 secs 2024-07-26T08:00:47.088 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3893.939557 secs 2024-07-26T08:00:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:46 smithi164 ceph-mon[36970]: pgmap v2649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3876.197333 secs 2024-07-26T08:00:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3893.939557 secs 2024-07-26T08:00:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:46 smithi163 ceph-mon[36873]: pgmap v2649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3876.197333 secs 2024-07-26T08:00:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3893.939557 secs 2024-07-26T08:00:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:48 smithi042 ceph-mon[32775]: pgmap v2650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:49.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:48 smithi164 ceph-mon[36970]: pgmap v2650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:48 smithi163 ceph-mon[36873]: pgmap v2650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:50 smithi042 ceph-mon[32775]: pgmap v2651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:50 smithi164 ceph-mon[36970]: pgmap v2651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:50 smithi163 ceph-mon[36873]: pgmap v2651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3881.197453 secs 2024-07-26T08:00:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3898.939678 secs 2024-07-26T08:00:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3881.197453 secs 2024-07-26T08:00:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3898.939678 secs 2024-07-26T08:00:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3881.197453 secs 2024-07-26T08:00:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3898.939678 secs 2024-07-26T08:00:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:52 smithi042 ceph-mon[32775]: pgmap v2652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:53.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:52 smithi164 ceph-mon[36970]: pgmap v2652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:52 smithi163 ceph-mon[36873]: pgmap v2652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:53 smithi042 ceph-mon[32775]: pgmap v2653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:00:54.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:53 smithi164 ceph-mon[36970]: pgmap v2653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:00:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:53 smithi163 ceph-mon[36873]: pgmap v2653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:00:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:00:54.574+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:00:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:00:54.574+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:00:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:00:54.574+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:00:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:56 smithi042 ceph-mon[32775]: pgmap v2654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3886.197595 secs 2024-07-26T08:00:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3903.939799 secs 2024-07-26T08:00:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:56 smithi164 ceph-mon[36970]: pgmap v2654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3886.197595 secs 2024-07-26T08:00:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3903.939799 secs 2024-07-26T08:00:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:56 smithi163 ceph-mon[36873]: pgmap v2654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:00:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3886.197595 secs 2024-07-26T08:00:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3903.939799 secs 2024-07-26T08:00:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:00:58 smithi042 ceph-mon[32775]: pgmap v2655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:00:58 smithi164 ceph-mon[36970]: pgmap v2655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:00:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:00:58 smithi163 ceph-mon[36873]: pgmap v2655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:00 smithi042 ceph-mon[32775]: pgmap v2656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:00 smithi164 ceph-mon[36970]: pgmap v2656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:00 smithi163 ceph-mon[36873]: pgmap v2656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3891.197718 secs 2024-07-26T08:01:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3908.939918 secs 2024-07-26T08:01:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3891.197718 secs 2024-07-26T08:01:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3908.939918 secs 2024-07-26T08:01:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3891.197718 secs 2024-07-26T08:01:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3908.939918 secs 2024-07-26T08:01:03.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:02 smithi042 ceph-mon[32775]: pgmap v2657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:02 smithi164 ceph-mon[36970]: pgmap v2657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:02 smithi163 ceph-mon[36873]: pgmap v2657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:04 smithi042 ceph-mon[32775]: pgmap v2658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:01:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:04 smithi164 ceph-mon[36970]: pgmap v2658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:01:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:04 smithi163 ceph-mon[36873]: pgmap v2658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:01:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:06 smithi042 ceph-mon[32775]: pgmap v2659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3896.197841 secs 2024-07-26T08:01:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3913.940034 secs 2024-07-26T08:01:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:06 smithi164 ceph-mon[36970]: pgmap v2659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3896.197841 secs 2024-07-26T08:01:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3913.940034 secs 2024-07-26T08:01:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:06 smithi163 ceph-mon[36873]: pgmap v2659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3896.197841 secs 2024-07-26T08:01:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3913.940034 secs 2024-07-26T08:01:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:08 smithi042 ceph-mon[32775]: pgmap v2660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:09.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:08 smithi164 ceph-mon[36970]: pgmap v2660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:08 smithi163 ceph-mon[36873]: pgmap v2660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:09.583 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:09.582+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:01:09.583 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:09.582+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:01:09.583 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:09.582+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:01:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:10 smithi042 ceph-mon[32775]: pgmap v2661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:10 smithi164 ceph-mon[36970]: pgmap v2661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:10 smithi163 ceph-mon[36873]: pgmap v2661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3901.197963 secs 2024-07-26T08:01:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3918.940159 secs 2024-07-26T08:01:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3901.197963 secs 2024-07-26T08:01:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3918.940159 secs 2024-07-26T08:01:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3901.197963 secs 2024-07-26T08:01:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3918.940159 secs 2024-07-26T08:01:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:12 smithi042 ceph-mon[32775]: pgmap v2662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:12 smithi164 ceph-mon[36970]: pgmap v2662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:12 smithi163 ceph-mon[36873]: pgmap v2662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:14 smithi042 ceph-mon[32775]: pgmap v2663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:01:15.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:14 smithi164 ceph-mon[36970]: pgmap v2663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:01:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:14 smithi163 ceph-mon[36873]: pgmap v2663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:01:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:16 smithi042 ceph-mon[32775]: pgmap v2664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3906.198106 secs 2024-07-26T08:01:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3923.940286 secs 2024-07-26T08:01:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:16 smithi164 ceph-mon[36970]: pgmap v2664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3906.198106 secs 2024-07-26T08:01:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3923.940286 secs 2024-07-26T08:01:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:16 smithi163 ceph-mon[36873]: pgmap v2664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3906.198106 secs 2024-07-26T08:01:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3923.940286 secs 2024-07-26T08:01:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:17 smithi042 ceph-mon[32775]: pgmap v2665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:18.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:17 smithi164 ceph-mon[36970]: pgmap v2665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:17 smithi163 ceph-mon[36873]: pgmap v2665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:20 smithi042 ceph-mon[32775]: pgmap v2666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:01:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:01:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:01:21.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:20 smithi164 ceph-mon[36970]: pgmap v2666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:01:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:01:21.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:01:21.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:20 smithi163 ceph-mon[36873]: pgmap v2666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:01:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:01:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:01:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:01:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:21 smithi042 ceph-mon[32775]: pgmap v2667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T08:01:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:21 smithi042 ceph-mon[32775]: pgmap v2668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:01:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3911.198263 secs 2024-07-26T08:01:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3928.940375 secs 2024-07-26T08:01:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:21 smithi164 ceph-mon[36970]: pgmap v2667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T08:01:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:21 smithi164 ceph-mon[36970]: pgmap v2668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:01:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3911.198263 secs 2024-07-26T08:01:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3928.940375 secs 2024-07-26T08:01:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:21 smithi163 ceph-mon[36873]: pgmap v2667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T08:01:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:21 smithi163 ceph-mon[36873]: pgmap v2668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:01:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3911.198263 secs 2024-07-26T08:01:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3928.940375 secs 2024-07-26T08:01:24.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:23 smithi042 ceph-mon[32775]: pgmap v2669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:01:24.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:23 smithi164 ceph-mon[36970]: pgmap v2669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:01:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:23 smithi163 ceph-mon[36873]: pgmap v2669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:01:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:24.584+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:01:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:24.584+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:01:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:24.584+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:01:26.059 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:25 smithi163 ceph-mon[36873]: pgmap v2670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:01:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:25 smithi042 ceph-mon[32775]: pgmap v2670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:01:26.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:25 smithi164 ceph-mon[36970]: pgmap v2670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:01:27.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3916.198348 secs 2024-07-26T08:01:27.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3933.940499 secs 2024-07-26T08:01:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3916.198348 secs 2024-07-26T08:01:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3933.940499 secs 2024-07-26T08:01:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3916.198348 secs 2024-07-26T08:01:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3933.940499 secs 2024-07-26T08:01:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:27 smithi042 ceph-mon[32775]: pgmap v2671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:01:28.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:27 smithi164 ceph-mon[36970]: pgmap v2671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:01:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:27 smithi163 ceph-mon[36873]: pgmap v2671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:01:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:29 smithi042 ceph-mon[32775]: pgmap v2672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T08:01:30.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:29 smithi164 ceph-mon[36970]: pgmap v2672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T08:01:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:29 smithi163 ceph-mon[36873]: pgmap v2672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T08:01:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:31 smithi042 ceph-mon[32775]: pgmap v2673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:01:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3921.198466 secs 2024-07-26T08:01:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3938.940638 secs 2024-07-26T08:01:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:31 smithi164 ceph-mon[36970]: pgmap v2673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:01:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3921.198466 secs 2024-07-26T08:01:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3938.940638 secs 2024-07-26T08:01:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:31 smithi163 ceph-mon[36873]: pgmap v2673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:01:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3921.198466 secs 2024-07-26T08:01:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3938.940638 secs 2024-07-26T08:01:34.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:33 smithi042 ceph-mon[32775]: pgmap v2674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:33 smithi164 ceph-mon[36970]: pgmap v2674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:33 smithi163 ceph-mon[36873]: pgmap v2674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:35 smithi042 ceph-mon[32775]: pgmap v2675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:36.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:35 smithi164 ceph-mon[36970]: pgmap v2675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:35 smithi163 ceph-mon[36873]: pgmap v2675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3926.198556 secs 2024-07-26T08:01:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3943.940731 secs 2024-07-26T08:01:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3926.198556 secs 2024-07-26T08:01:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3943.940731 secs 2024-07-26T08:01:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3926.198556 secs 2024-07-26T08:01:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3943.940731 secs 2024-07-26T08:01:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:37 smithi042 ceph-mon[32775]: pgmap v2676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:38.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:37 smithi164 ceph-mon[36970]: pgmap v2676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:37 smithi163 ceph-mon[36873]: pgmap v2676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:39.594 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:39.593+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:01:39.594 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:39.593+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:01:39.594 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:39.593+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:01:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:39 smithi042 ceph-mon[32775]: pgmap v2677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:01:40.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:39 smithi164 ceph-mon[36970]: pgmap v2677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:01:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:39 smithi163 ceph-mon[36873]: pgmap v2677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:01:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:41 smithi042 ceph-mon[32775]: pgmap v2678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3931.198679 secs 2024-07-26T08:01:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3948.940854 secs 2024-07-26T08:01:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:41 smithi164 ceph-mon[36970]: pgmap v2678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3931.198679 secs 2024-07-26T08:01:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3948.940854 secs 2024-07-26T08:01:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:41 smithi163 ceph-mon[36873]: pgmap v2678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3931.198679 secs 2024-07-26T08:01:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3948.940854 secs 2024-07-26T08:01:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:43 smithi042 ceph-mon[32775]: pgmap v2679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:44.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:43 smithi164 ceph-mon[36970]: pgmap v2679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:43 smithi163 ceph-mon[36873]: pgmap v2679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:44 smithi042 ceph-mon[32775]: pgmap v2680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:44 smithi164 ceph-mon[36970]: pgmap v2680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:44 smithi163 ceph-mon[36873]: pgmap v2680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3936.198812 secs 2024-07-26T08:01:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3936.198812 secs 2024-07-26T08:01:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3936.198812 secs 2024-07-26T08:01:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3953.940974 secs 2024-07-26T08:01:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:47 smithi042 ceph-mon[32775]: pgmap v2681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3953.940974 secs 2024-07-26T08:01:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:47 smithi164 ceph-mon[36970]: pgmap v2681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3953.940974 secs 2024-07-26T08:01:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:47 smithi163 ceph-mon[36873]: pgmap v2681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:49 smithi042 ceph-mon[32775]: pgmap v2682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:01:49.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:49 smithi164 ceph-mon[36970]: pgmap v2682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:01:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:49 smithi163 ceph-mon[36873]: pgmap v2682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:01:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:51 smithi042 ceph-mon[32775]: pgmap v2683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3941.198936 secs 2024-07-26T08:01:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:51 smithi164 ceph-mon[36970]: pgmap v2683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3941.198936 secs 2024-07-26T08:01:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:51 smithi163 ceph-mon[36873]: pgmap v2683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3941.198936 secs 2024-07-26T08:01:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3958.941075 secs 2024-07-26T08:01:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3958.941075 secs 2024-07-26T08:01:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3958.941075 secs 2024-07-26T08:01:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:53 smithi042 ceph-mon[32775]: pgmap v2684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:53.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:53 smithi164 ceph-mon[36970]: pgmap v2684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:53 smithi163 ceph-mon[36873]: pgmap v2684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:54.597+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:01:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:54.597+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:01:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:54.597+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:01:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:55 smithi042 ceph-mon[32775]: pgmap v2685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:55.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:55 smithi164 ceph-mon[36970]: pgmap v2685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:55 smithi163 ceph-mon[36873]: pgmap v2685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:01:56.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3946.199082 secs 2024-07-26T08:01:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3946.199082 secs 2024-07-26T08:01:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3946.199082 secs 2024-07-26T08:01:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3963.941194 secs 2024-07-26T08:01:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:57 smithi164 ceph-mon[36970]: pgmap v2686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3963.941194 secs 2024-07-26T08:01:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:57 smithi163 ceph-mon[36873]: pgmap v2686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:58.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3963.941194 secs 2024-07-26T08:01:58.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:57 smithi042 ceph-mon[32775]: pgmap v2686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:01:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:58.483+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658006460 con 0x7ff680102000 2024-07-26T08:01:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:58.483+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:01:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:58.483+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff658006460 con 0x7ff680102000 2024-07-26T08:01:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:58.484+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:01:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:58.484+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:01:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:58.484+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:01:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:58.484+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:01:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:01:58.484+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:01:59.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:01:59 smithi164 ceph-mon[36970]: pgmap v2687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:02:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:01:59 smithi163 ceph-mon[36873]: pgmap v2687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:02:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:01:59 smithi042 ceph-mon[32775]: pgmap v2687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:02:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:01 smithi164 ceph-mon[36970]: pgmap v2688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3951.199247 secs 2024-07-26T08:02:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3968.941314 secs 2024-07-26T08:02:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:01 smithi163 ceph-mon[36873]: pgmap v2688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3951.199247 secs 2024-07-26T08:02:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3968.941314 secs 2024-07-26T08:02:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:01 smithi042 ceph-mon[32775]: pgmap v2688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3951.199247 secs 2024-07-26T08:02:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3968.941314 secs 2024-07-26T08:02:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:03 smithi164 ceph-mon[36970]: pgmap v2689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:04.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:03 smithi163 ceph-mon[36873]: pgmap v2689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:03 smithi042 ceph-mon[32775]: pgmap v2689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:05 smithi164 ceph-mon[36970]: pgmap v2690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:05 smithi163 ceph-mon[36873]: pgmap v2690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:05 smithi042 ceph-mon[32775]: pgmap v2690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3956.199299 secs 2024-07-26T08:02:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3973.941471 secs 2024-07-26T08:02:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3956.199299 secs 2024-07-26T08:02:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3973.941471 secs 2024-07-26T08:02:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3956.199299 secs 2024-07-26T08:02:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3973.941471 secs 2024-07-26T08:02:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:07 smithi164 ceph-mon[36970]: pgmap v2691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:07 smithi163 ceph-mon[36873]: pgmap v2691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:07 smithi042 ceph-mon[32775]: pgmap v2691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:02:09.605+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:02:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:02:09.605+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:02:09.612 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:02:09.605+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:02:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:09 smithi164 ceph-mon[36970]: pgmap v2692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:02:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:09 smithi163 ceph-mon[36873]: pgmap v2692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:02:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:09 smithi042 ceph-mon[32775]: pgmap v2692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:02:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:11 smithi164 ceph-mon[36970]: pgmap v2693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3961.199433 secs 2024-07-26T08:02:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3978.941537 secs 2024-07-26T08:02:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:11 smithi163 ceph-mon[36873]: pgmap v2693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3961.199433 secs 2024-07-26T08:02:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3978.941537 secs 2024-07-26T08:02:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:11 smithi042 ceph-mon[32775]: pgmap v2693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3961.199433 secs 2024-07-26T08:02:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3978.941537 secs 2024-07-26T08:02:13.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:13 smithi164 ceph-mon[36970]: pgmap v2694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:13 smithi163 ceph-mon[36873]: pgmap v2694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:13 smithi042 ceph-mon[32775]: pgmap v2694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:15.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:15 smithi164 ceph-mon[36970]: pgmap v2695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:15 smithi163 ceph-mon[36873]: pgmap v2695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:15 smithi042 ceph-mon[32775]: pgmap v2695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3966.199526 secs 2024-07-26T08:02:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3983.941660 secs 2024-07-26T08:02:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3966.199526 secs 2024-07-26T08:02:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3983.941660 secs 2024-07-26T08:02:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3966.199526 secs 2024-07-26T08:02:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3983.941660 secs 2024-07-26T08:02:17.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:17 smithi164 ceph-mon[36970]: pgmap v2696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:17 smithi163 ceph-mon[36873]: pgmap v2696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:17 smithi042 ceph-mon[32775]: pgmap v2696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:19 smithi164 ceph-mon[36970]: pgmap v2697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:02:20.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:19 smithi163 ceph-mon[36873]: pgmap v2697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:02:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:19 smithi042 ceph-mon[32775]: pgmap v2697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:02:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:02:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:02:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:02:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:21 smithi164 ceph-mon[36970]: pgmap v2698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3971.199625 secs 2024-07-26T08:02:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3988.941813 secs 2024-07-26T08:02:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:02:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:02:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:02:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:21 smithi163 ceph-mon[36873]: pgmap v2698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3971.199625 secs 2024-07-26T08:02:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3988.941813 secs 2024-07-26T08:02:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:02:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:02:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:02:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:21 smithi042 ceph-mon[32775]: pgmap v2698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3971.199625 secs 2024-07-26T08:02:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3988.941813 secs 2024-07-26T08:02:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:02:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:02:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:02:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:22 smithi164 ceph-mon[36970]: pgmap v2699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:02:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:22 smithi164 ceph-mon[36970]: pgmap v2700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:02:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:22 smithi163 ceph-mon[36873]: pgmap v2699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:02:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:22 smithi163 ceph-mon[36873]: pgmap v2700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:02:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:22 smithi042 ceph-mon[32775]: pgmap v2699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:02:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:22 smithi042 ceph-mon[32775]: pgmap v2700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:02:24.609 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:02:24.609+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:02:24.610 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:02:24.609+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:02:24.610 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:02:24.609+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:02:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:24 smithi164 ceph-mon[36970]: pgmap v2701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:02:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:24 smithi163 ceph-mon[36873]: pgmap v2701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:02:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:24 smithi042 ceph-mon[32775]: pgmap v2701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:02:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:26 smithi164 ceph-mon[36970]: pgmap v2702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:02:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3976.199754 secs 2024-07-26T08:02:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3993.941963 secs 2024-07-26T08:02:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:26 smithi163 ceph-mon[36873]: pgmap v2702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:02:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3976.199754 secs 2024-07-26T08:02:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3993.941963 secs 2024-07-26T08:02:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:26 smithi042 ceph-mon[32775]: pgmap v2702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:02:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3976.199754 secs 2024-07-26T08:02:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3993.941963 secs 2024-07-26T08:02:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:28 smithi164 ceph-mon[36970]: pgmap v2703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:02:29.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:28 smithi163 ceph-mon[36873]: pgmap v2703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:02:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:28 smithi042 ceph-mon[32775]: pgmap v2703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:02:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:30 smithi164 ceph-mon[36970]: pgmap v2704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:02:31.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:30 smithi163 ceph-mon[36873]: pgmap v2704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:02:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:30 smithi042 ceph-mon[32775]: pgmap v2704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:02:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3981.199876 secs 2024-07-26T08:02:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3998.942087 secs 2024-07-26T08:02:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3981.199876 secs 2024-07-26T08:02:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3998.942087 secs 2024-07-26T08:02:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3981.199876 secs 2024-07-26T08:02:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3998.942087 secs 2024-07-26T08:02:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:32 smithi163 ceph-mon[36873]: pgmap v2705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:02:33.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:32 smithi164 ceph-mon[36970]: pgmap v2705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:02:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:32 smithi042 ceph-mon[32775]: pgmap v2705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:02:35.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:34 smithi163 ceph-mon[36873]: pgmap v2706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:35.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:34 smithi042 ceph-mon[32775]: pgmap v2706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:34 smithi164 ceph-mon[36970]: pgmap v2706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:36 smithi163 ceph-mon[36873]: pgmap v2707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3986.200000 secs 2024-07-26T08:02:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4003.942215 secs 2024-07-26T08:02:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:36 smithi042 ceph-mon[32775]: pgmap v2707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3986.200000 secs 2024-07-26T08:02:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4003.942215 secs 2024-07-26T08:02:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:36 smithi164 ceph-mon[36970]: pgmap v2707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3986.200000 secs 2024-07-26T08:02:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4003.942215 secs 2024-07-26T08:02:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:38 smithi163 ceph-mon[36873]: pgmap v2708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:38 smithi042 ceph-mon[32775]: pgmap v2708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:39.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:38 smithi164 ceph-mon[36970]: pgmap v2708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:39.619 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:02:39.618+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:02:39.619 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:02:39.618+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:02:39.619 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:02:39.618+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:02:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:40 smithi163 ceph-mon[36873]: pgmap v2709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:40 smithi042 ceph-mon[32775]: pgmap v2709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:41.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:40 smithi164 ceph-mon[36970]: pgmap v2709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3991.200068 secs 2024-07-26T08:02:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4008.942337 secs 2024-07-26T08:02:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3991.200068 secs 2024-07-26T08:02:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4008.942337 secs 2024-07-26T08:02:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3991.200068 secs 2024-07-26T08:02:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4008.942337 secs 2024-07-26T08:02:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:42 smithi163 ceph-mon[36873]: pgmap v2710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:42 smithi042 ceph-mon[32775]: pgmap v2710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:43.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:42 smithi164 ceph-mon[36970]: pgmap v2710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:44 smithi163 ceph-mon[36873]: pgmap v2711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:44 smithi042 ceph-mon[32775]: pgmap v2711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:45.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:44 smithi164 ceph-mon[36970]: pgmap v2711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:46 smithi163 ceph-mon[36873]: pgmap v2712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 3996.200190 secs 2024-07-26T08:02:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4013.942476 secs 2024-07-26T08:02:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:46 smithi042 ceph-mon[32775]: pgmap v2712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 3996.200190 secs 2024-07-26T08:02:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4013.942476 secs 2024-07-26T08:02:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:46 smithi164 ceph-mon[36970]: pgmap v2712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 3996.200190 secs 2024-07-26T08:02:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4013.942476 secs 2024-07-26T08:02:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:48 smithi042 ceph-mon[32775]: pgmap v2713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:49.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:48 smithi164 ceph-mon[36970]: pgmap v2713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:48 smithi163 ceph-mon[36873]: pgmap v2713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:50 smithi042 ceph-mon[32775]: pgmap v2714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:50 smithi164 ceph-mon[36970]: pgmap v2714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:50 smithi163 ceph-mon[36873]: pgmap v2714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4001.200312 secs 2024-07-26T08:02:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4018.942596 secs 2024-07-26T08:02:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4001.200312 secs 2024-07-26T08:02:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4018.942596 secs 2024-07-26T08:02:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4001.200312 secs 2024-07-26T08:02:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4018.942596 secs 2024-07-26T08:02:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:52 smithi042 ceph-mon[32775]: pgmap v2715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:53.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:52 smithi164 ceph-mon[36970]: pgmap v2715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:52 smithi163 ceph-mon[36873]: pgmap v2715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:02:54.622+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:02:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:02:54.622+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:02:54.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:02:54.622+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:02:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:54 smithi042 ceph-mon[32775]: pgmap v2716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:55.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:54 smithi164 ceph-mon[36970]: pgmap v2716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:55.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:54 smithi163 ceph-mon[36873]: pgmap v2716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:57.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:56 smithi163 ceph-mon[36873]: pgmap v2717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:57.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4006.200427 secs 2024-07-26T08:02:57.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4023.942725 secs 2024-07-26T08:02:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:56 smithi042 ceph-mon[32775]: pgmap v2717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4006.200427 secs 2024-07-26T08:02:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4023.942725 secs 2024-07-26T08:02:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:56 smithi164 ceph-mon[36970]: pgmap v2717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:02:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4006.200427 secs 2024-07-26T08:02:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4023.942725 secs 2024-07-26T08:02:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:02:58 smithi042 ceph-mon[32775]: pgmap v2718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:02:58 smithi164 ceph-mon[36970]: pgmap v2718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:02:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:02:58 smithi163 ceph-mon[36873]: pgmap v2718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:00 smithi042 ceph-mon[32775]: pgmap v2719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:00 smithi164 ceph-mon[36970]: pgmap v2719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:00 smithi163 ceph-mon[36873]: pgmap v2719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4011.200515 secs 2024-07-26T08:03:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4028.942853 secs 2024-07-26T08:03:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4011.200515 secs 2024-07-26T08:03:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4028.942853 secs 2024-07-26T08:03:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4011.200515 secs 2024-07-26T08:03:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4028.942853 secs 2024-07-26T08:03:03.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:02 smithi164 ceph-mon[36970]: pgmap v2720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:02 smithi042 ceph-mon[32775]: pgmap v2720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:02 smithi163 ceph-mon[36873]: pgmap v2720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:04 smithi042 ceph-mon[32775]: pgmap v2721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:04 smithi164 ceph-mon[36970]: pgmap v2721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:04 smithi163 ceph-mon[36873]: pgmap v2721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:06 smithi042 ceph-mon[32775]: pgmap v2722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4016.200613 secs 2024-07-26T08:03:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4033.942951 secs 2024-07-26T08:03:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:06 smithi164 ceph-mon[36970]: pgmap v2722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4016.200613 secs 2024-07-26T08:03:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4033.942951 secs 2024-07-26T08:03:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:06 smithi163 ceph-mon[36873]: pgmap v2722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4016.200613 secs 2024-07-26T08:03:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4033.942951 secs 2024-07-26T08:03:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:08 smithi042 ceph-mon[32775]: pgmap v2723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:09.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:08 smithi164 ceph-mon[36970]: pgmap v2723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:08 smithi163 ceph-mon[36873]: pgmap v2723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:09.631 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:03:09.630+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:03:09.631 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:03:09.630+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:03:09.631 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:03:09.630+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:03:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:10 smithi042 ceph-mon[32775]: pgmap v2724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:10 smithi164 ceph-mon[36970]: pgmap v2724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:10 smithi163 ceph-mon[36873]: pgmap v2724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4021.200697 secs 2024-07-26T08:03:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4038.943082 secs 2024-07-26T08:03:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4021.200697 secs 2024-07-26T08:03:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4038.943082 secs 2024-07-26T08:03:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4021.200697 secs 2024-07-26T08:03:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4038.943082 secs 2024-07-26T08:03:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:12 smithi042 ceph-mon[32775]: pgmap v2725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:12 smithi164 ceph-mon[36970]: pgmap v2725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:12 smithi163 ceph-mon[36873]: pgmap v2725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:14 smithi042 ceph-mon[32775]: pgmap v2726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:15.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:14 smithi164 ceph-mon[36970]: pgmap v2726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:14 smithi163 ceph-mon[36873]: pgmap v2726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:16 smithi042 ceph-mon[32775]: pgmap v2727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4026.200839 secs 2024-07-26T08:03:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4043.943197 secs 2024-07-26T08:03:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:16 smithi164 ceph-mon[36970]: pgmap v2727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4026.200839 secs 2024-07-26T08:03:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4043.943197 secs 2024-07-26T08:03:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:16 smithi163 ceph-mon[36873]: pgmap v2727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4026.200839 secs 2024-07-26T08:03:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4043.943197 secs 2024-07-26T08:03:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:18 smithi042 ceph-mon[32775]: pgmap v2728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:18 smithi164 ceph-mon[36970]: pgmap v2728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:18 smithi163 ceph-mon[36873]: pgmap v2728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:20 smithi042 ceph-mon[32775]: pgmap v2729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:20 smithi164 ceph-mon[36970]: pgmap v2729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:20 smithi163 ceph-mon[36873]: pgmap v2729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4031.200982 secs 2024-07-26T08:03:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4048.943285 secs 2024-07-26T08:03:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:03:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:03:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:03:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:03:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4031.200982 secs 2024-07-26T08:03:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4048.943285 secs 2024-07-26T08:03:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:03:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:03:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:03:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:03:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4031.200982 secs 2024-07-26T08:03:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4048.943285 secs 2024-07-26T08:03:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:03:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:03:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:03:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:03:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:22 smithi042 ceph-mon[32775]: pgmap v2730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:22 smithi042 ceph-mon[32775]: pgmap v2731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:03:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:22 smithi042 ceph-mon[32775]: pgmap v2732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:03:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:22 smithi164 ceph-mon[36970]: pgmap v2730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:22 smithi164 ceph-mon[36970]: pgmap v2731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:03:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:22 smithi164 ceph-mon[36970]: pgmap v2732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:03:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:22 smithi163 ceph-mon[36873]: pgmap v2730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:22 smithi163 ceph-mon[36873]: pgmap v2731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:03:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:22 smithi163 ceph-mon[36873]: pgmap v2732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:03:24.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:03:24.633+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:03:24.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:03:24.633+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:03:24.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:03:24.633+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:03:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:24 smithi042 ceph-mon[32775]: pgmap v2733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:03:25.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:24 smithi164 ceph-mon[36970]: pgmap v2733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:03:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:24 smithi163 ceph-mon[36873]: pgmap v2733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:03:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:26 smithi042 ceph-mon[32775]: pgmap v2734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:03:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4036.201120 secs 2024-07-26T08:03:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4053.943407 secs 2024-07-26T08:03:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:26 smithi164 ceph-mon[36970]: pgmap v2734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:03:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4036.201120 secs 2024-07-26T08:03:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4053.943407 secs 2024-07-26T08:03:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:26 smithi163 ceph-mon[36873]: pgmap v2734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:03:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4036.201120 secs 2024-07-26T08:03:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4053.943407 secs 2024-07-26T08:03:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:27 smithi042 ceph-mon[32775]: pgmap v2735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:03:28.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:27 smithi164 ceph-mon[36970]: pgmap v2735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:03:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:27 smithi163 ceph-mon[36873]: pgmap v2735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:03:31.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:30 smithi163 ceph-mon[36873]: pgmap v2736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:03:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:30 smithi042 ceph-mon[32775]: pgmap v2736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:03:31.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:30 smithi164 ceph-mon[36970]: pgmap v2736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:03:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4041.201272 secs 2024-07-26T08:03:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4058.943548 secs 2024-07-26T08:03:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4041.201272 secs 2024-07-26T08:03:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4058.943548 secs 2024-07-26T08:03:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4041.201272 secs 2024-07-26T08:03:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4058.943548 secs 2024-07-26T08:03:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:32 smithi163 ceph-mon[36873]: pgmap v2737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:03:33.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:32 smithi164 ceph-mon[36970]: pgmap v2737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:03:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:32 smithi042 ceph-mon[32775]: pgmap v2737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:03:35.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:34 smithi163 ceph-mon[36873]: pgmap v2738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:03:35.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:34 smithi042 ceph-mon[32775]: pgmap v2738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:03:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:34 smithi164 ceph-mon[36970]: pgmap v2738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:03:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:36 smithi163 ceph-mon[36873]: pgmap v2739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4046.201357 secs 2024-07-26T08:03:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4063.943670 secs 2024-07-26T08:03:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:36 smithi042 ceph-mon[32775]: pgmap v2739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4046.201357 secs 2024-07-26T08:03:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4063.943670 secs 2024-07-26T08:03:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:36 smithi164 ceph-mon[36970]: pgmap v2739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4046.201357 secs 2024-07-26T08:03:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4063.943670 secs 2024-07-26T08:03:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:38 smithi163 ceph-mon[36873]: pgmap v2740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:38 smithi042 ceph-mon[32775]: pgmap v2740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:39.228 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:38 smithi164 ceph-mon[36970]: pgmap v2740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:39.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:03:39.640+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:03:39.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:03:39.640+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:03:39.642 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:03:39.640+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:03:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:40 smithi163 ceph-mon[36873]: pgmap v2741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:40 smithi042 ceph-mon[32775]: pgmap v2741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:40 smithi164 ceph-mon[36970]: pgmap v2741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4051.201517 secs 2024-07-26T08:03:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4068.943791 secs 2024-07-26T08:03:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4051.201517 secs 2024-07-26T08:03:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4068.943791 secs 2024-07-26T08:03:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4051.201517 secs 2024-07-26T08:03:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4068.943791 secs 2024-07-26T08:03:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:42 smithi042 ceph-mon[32775]: pgmap v2742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:43.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:42 smithi164 ceph-mon[36970]: pgmap v2742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:42 smithi163 ceph-mon[36873]: pgmap v2742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:44 smithi042 ceph-mon[32775]: pgmap v2743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:03:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:44 smithi164 ceph-mon[36970]: pgmap v2743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:03:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:44 smithi163 ceph-mon[36873]: pgmap v2743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:03:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:46 smithi042 ceph-mon[32775]: pgmap v2744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4056.201646 secs 2024-07-26T08:03:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4073.943912 secs 2024-07-26T08:03:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:46 smithi164 ceph-mon[36970]: pgmap v2744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4056.201646 secs 2024-07-26T08:03:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4073.943912 secs 2024-07-26T08:03:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:46 smithi163 ceph-mon[36873]: pgmap v2744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4056.201646 secs 2024-07-26T08:03:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4073.943912 secs 2024-07-26T08:03:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:48 smithi042 ceph-mon[32775]: pgmap v2745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:49.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:48 smithi164 ceph-mon[36970]: pgmap v2745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:48 smithi163 ceph-mon[36873]: pgmap v2745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:50 smithi042 ceph-mon[32775]: pgmap v2746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:50 smithi164 ceph-mon[36970]: pgmap v2746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:50 smithi163 ceph-mon[36873]: pgmap v2746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4061.201721 secs 2024-07-26T08:03:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4078.944036 secs 2024-07-26T08:03:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4061.201721 secs 2024-07-26T08:03:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4078.944036 secs 2024-07-26T08:03:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4061.201721 secs 2024-07-26T08:03:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4078.944036 secs 2024-07-26T08:03:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:52 smithi042 ceph-mon[32775]: pgmap v2747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:52 smithi164 ceph-mon[36970]: pgmap v2747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:52 smithi163 ceph-mon[36873]: pgmap v2747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:54.644 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:03:54.643+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:03:54.645 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:03:54.643+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:03:54.645 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:03:54.643+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:03:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:54 smithi042 ceph-mon[32775]: pgmap v2748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:03:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:54 smithi164 ceph-mon[36970]: pgmap v2748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:03:55.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:54 smithi163 ceph-mon[36873]: pgmap v2748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:03:57.059 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:56 smithi163 ceph-mon[36873]: pgmap v2749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:57.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4066.201844 secs 2024-07-26T08:03:57.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4083.944151 secs 2024-07-26T08:03:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:56 smithi042 ceph-mon[32775]: pgmap v2749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4066.201844 secs 2024-07-26T08:03:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4083.944151 secs 2024-07-26T08:03:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:56 smithi164 ceph-mon[36970]: pgmap v2749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:03:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4066.201844 secs 2024-07-26T08:03:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4083.944151 secs 2024-07-26T08:03:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:03:58 smithi042 ceph-mon[32775]: pgmap v2750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:59.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:03:58 smithi164 ceph-mon[36970]: pgmap v2750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:03:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:03:58 smithi163 ceph-mon[36873]: pgmap v2750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:00 smithi042 ceph-mon[32775]: pgmap v2751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:00 smithi164 ceph-mon[36970]: pgmap v2751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:00 smithi163 ceph-mon[36873]: pgmap v2751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4071.201976 secs 2024-07-26T08:04:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4088.944237 secs 2024-07-26T08:04:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4071.201976 secs 2024-07-26T08:04:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4088.944237 secs 2024-07-26T08:04:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4071.201976 secs 2024-07-26T08:04:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4088.944237 secs 2024-07-26T08:04:03.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:02 smithi164 ceph-mon[36970]: pgmap v2752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:02 smithi042 ceph-mon[32775]: pgmap v2752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:02 smithi163 ceph-mon[36873]: pgmap v2752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:04 smithi042 ceph-mon[32775]: pgmap v2753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:04:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:04 smithi164 ceph-mon[36970]: pgmap v2753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:04:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:04 smithi163 ceph-mon[36873]: pgmap v2753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:04:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:06 smithi042 ceph-mon[32775]: pgmap v2754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4076.202098 secs 2024-07-26T08:04:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4093.944362 secs 2024-07-26T08:04:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:06 smithi164 ceph-mon[36970]: pgmap v2754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4076.202098 secs 2024-07-26T08:04:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4093.944362 secs 2024-07-26T08:04:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:06 smithi163 ceph-mon[36873]: pgmap v2754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4076.202098 secs 2024-07-26T08:04:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4093.944362 secs 2024-07-26T08:04:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:08 smithi042 ceph-mon[32775]: pgmap v2755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:09.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:08 smithi164 ceph-mon[36970]: pgmap v2755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:08 smithi163 ceph-mon[36873]: pgmap v2755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:04:09.651+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:04:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:04:09.651+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:04:09.652 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:04:09.651+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:04:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:10 smithi042 ceph-mon[32775]: pgmap v2756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:10 smithi164 ceph-mon[36970]: pgmap v2756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:10 smithi163 ceph-mon[36873]: pgmap v2756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4081.202246 secs 2024-07-26T08:04:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4098.944505 secs 2024-07-26T08:04:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4081.202246 secs 2024-07-26T08:04:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4098.944505 secs 2024-07-26T08:04:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4081.202246 secs 2024-07-26T08:04:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4098.944505 secs 2024-07-26T08:04:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:12 smithi042 ceph-mon[32775]: pgmap v2757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:12 smithi164 ceph-mon[36970]: pgmap v2757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:12 smithi163 ceph-mon[36873]: pgmap v2757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:14 smithi042 ceph-mon[32775]: pgmap v2758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:04:15.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:14 smithi164 ceph-mon[36970]: pgmap v2758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:04:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:14 smithi163 ceph-mon[36873]: pgmap v2758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:04:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:16 smithi042 ceph-mon[32775]: pgmap v2759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4086.202396 secs 2024-07-26T08:04:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4103.944627 secs 2024-07-26T08:04:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:16 smithi164 ceph-mon[36970]: pgmap v2759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4086.202396 secs 2024-07-26T08:04:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4103.944627 secs 2024-07-26T08:04:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:16 smithi163 ceph-mon[36873]: pgmap v2759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4086.202396 secs 2024-07-26T08:04:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4103.944627 secs 2024-07-26T08:04:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:18 smithi042 ceph-mon[32775]: pgmap v2760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:18 smithi164 ceph-mon[36970]: pgmap v2760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:18 smithi163 ceph-mon[36873]: pgmap v2760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:20 smithi042 ceph-mon[32775]: pgmap v2761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:20 smithi164 ceph-mon[36970]: pgmap v2761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:20 smithi163 ceph-mon[36873]: pgmap v2761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4091.202523 secs 2024-07-26T08:04:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4108.944767 secs 2024-07-26T08:04:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:04:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4091.202523 secs 2024-07-26T08:04:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4108.944767 secs 2024-07-26T08:04:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:04:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4091.202523 secs 2024-07-26T08:04:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4108.944767 secs 2024-07-26T08:04:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:04:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:22 smithi042 ceph-mon[32775]: pgmap v2762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:04:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:04:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:04:23.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:22 smithi164 ceph-mon[36970]: pgmap v2762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:04:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:04:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:04:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:22 smithi163 ceph-mon[36873]: pgmap v2762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:04:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:04:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:04:24.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:23 smithi042 ceph-mon[32775]: pgmap v2763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:04:24.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:23 smithi042 ceph-mon[32775]: pgmap v2764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:04:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:23 smithi164 ceph-mon[36970]: pgmap v2763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:04:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:23 smithi164 ceph-mon[36970]: pgmap v2764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:04:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:23 smithi163 ceph-mon[36873]: pgmap v2763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:04:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:23 smithi163 ceph-mon[36873]: pgmap v2764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:04:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:04:24.654+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:04:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:04:24.654+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:04:24.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:04:24.654+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:04:25.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:24 smithi164 ceph-mon[36970]: pgmap v2765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:04:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:24 smithi163 ceph-mon[36873]: pgmap v2765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:04:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:24 smithi042 ceph-mon[32775]: pgmap v2765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:04:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4096.202645 secs 2024-07-26T08:04:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4096.202645 secs 2024-07-26T08:04:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4096.202645 secs 2024-07-26T08:04:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4113.944913 secs 2024-07-26T08:04:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:27 smithi042 ceph-mon[32775]: pgmap v2766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:04:27.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4113.944913 secs 2024-07-26T08:04:27.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:27 smithi163 ceph-mon[36873]: pgmap v2766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:04:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4113.944913 secs 2024-07-26T08:04:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:27 smithi164 ceph-mon[36970]: pgmap v2766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:04:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:29 smithi042 ceph-mon[32775]: pgmap v2767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:04:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:29 smithi164 ceph-mon[36970]: pgmap v2767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:04:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:29 smithi163 ceph-mon[36873]: pgmap v2767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:04:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:31 smithi042 ceph-mon[32775]: pgmap v2768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:04:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4101.202770 secs 2024-07-26T08:04:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:31 smithi164 ceph-mon[36970]: pgmap v2768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:04:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4101.202770 secs 2024-07-26T08:04:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:31 smithi163 ceph-mon[36873]: pgmap v2768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:04:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4101.202770 secs 2024-07-26T08:04:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4118.945028 secs 2024-07-26T08:04:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4118.945028 secs 2024-07-26T08:04:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4118.945028 secs 2024-07-26T08:04:33.623 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:33 smithi164 ceph-mon[36970]: pgmap v2769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:04:33.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:33 smithi042 ceph-mon[32775]: pgmap v2769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:04:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:33 smithi163 ceph-mon[36873]: pgmap v2769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:04:35.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:35 smithi042 ceph-mon[32775]: pgmap v2770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:35.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:35 smithi164 ceph-mon[36970]: pgmap v2770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:35.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:35 smithi163 ceph-mon[36873]: pgmap v2770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4106.202901 secs 2024-07-26T08:04:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4106.202901 secs 2024-07-26T08:04:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4106.202901 secs 2024-07-26T08:04:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4123.945148 secs 2024-07-26T08:04:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:37 smithi042 ceph-mon[32775]: pgmap v2771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4123.945148 secs 2024-07-26T08:04:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:37 smithi164 ceph-mon[36970]: pgmap v2771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4123.945148 secs 2024-07-26T08:04:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:37 smithi163 ceph-mon[36873]: pgmap v2771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:39 smithi042 ceph-mon[32775]: pgmap v2772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:04:39.664 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:04:39.662+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:04:39.664 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:04:39.662+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:04:39.664 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:04:39.663+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:04:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:39 smithi164 ceph-mon[36970]: pgmap v2772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:04:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:39 smithi163 ceph-mon[36873]: pgmap v2772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:04:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:41 smithi042 ceph-mon[32775]: pgmap v2773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4111.202996 secs 2024-07-26T08:04:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:41 smithi164 ceph-mon[36970]: pgmap v2773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4111.202996 secs 2024-07-26T08:04:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:41 smithi163 ceph-mon[36873]: pgmap v2773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4111.202996 secs 2024-07-26T08:04:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4128.945276 secs 2024-07-26T08:04:42.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4128.945276 secs 2024-07-26T08:04:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4128.945276 secs 2024-07-26T08:04:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:43 smithi042 ceph-mon[32775]: pgmap v2774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:43 smithi164 ceph-mon[36970]: pgmap v2774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:43 smithi163 ceph-mon[36873]: pgmap v2774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:45 smithi042 ceph-mon[32775]: pgmap v2775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:45 smithi164 ceph-mon[36970]: pgmap v2775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:45 smithi163 ceph-mon[36873]: pgmap v2775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4116.203147 secs 2024-07-26T08:04:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4116.203147 secs 2024-07-26T08:04:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4116.203147 secs 2024-07-26T08:04:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4133.945398 secs 2024-07-26T08:04:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:47 smithi164 ceph-mon[36970]: pgmap v2776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4133.945398 secs 2024-07-26T08:04:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:47 smithi163 ceph-mon[36873]: pgmap v2776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4133.945398 secs 2024-07-26T08:04:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:47 smithi042 ceph-mon[32775]: pgmap v2776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:49 smithi164 ceph-mon[36970]: pgmap v2777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:04:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:49 smithi163 ceph-mon[36873]: pgmap v2777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:04:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:49 smithi042 ceph-mon[32775]: pgmap v2777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:04:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:51 smithi164 ceph-mon[36970]: pgmap v2778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4121.203251 secs 2024-07-26T08:04:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:51 smithi163 ceph-mon[36873]: pgmap v2778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4121.203251 secs 2024-07-26T08:04:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:51 smithi042 ceph-mon[32775]: pgmap v2778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4121.203251 secs 2024-07-26T08:04:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4138.945519 secs 2024-07-26T08:04:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4138.945519 secs 2024-07-26T08:04:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4138.945519 secs 2024-07-26T08:04:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:53 smithi164 ceph-mon[36970]: pgmap v2779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:53 smithi163 ceph-mon[36873]: pgmap v2779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:53 smithi042 ceph-mon[32775]: pgmap v2779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:54.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:04:54.667+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:04:54.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:04:54.667+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:04:54.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:04:54.667+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:04:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:55 smithi164 ceph-mon[36970]: pgmap v2780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:55 smithi163 ceph-mon[36873]: pgmap v2780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:55 smithi042 ceph-mon[32775]: pgmap v2780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:04:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4126.203363 secs 2024-07-26T08:04:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4126.203363 secs 2024-07-26T08:04:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4126.203363 secs 2024-07-26T08:04:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4143.945659 secs 2024-07-26T08:04:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:57 smithi164 ceph-mon[36970]: pgmap v2781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4143.945659 secs 2024-07-26T08:04:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:57 smithi163 ceph-mon[36873]: pgmap v2781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4143.945659 secs 2024-07-26T08:04:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:57 smithi042 ceph-mon[32775]: pgmap v2781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:04:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:04:58.512+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:04:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:04:58.512+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:04:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:04:58.512+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:04:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:04:58.512+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:04:59.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:04:59 smithi042 ceph-mon[32775]: pgmap v2782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:04:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:04:59 smithi164 ceph-mon[36970]: pgmap v2782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:04:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:04:59 smithi163 ceph-mon[36873]: pgmap v2782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:05:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:01 smithi164 ceph-mon[36970]: pgmap v2783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4131.203480 secs 2024-07-26T08:05:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:01 smithi163 ceph-mon[36873]: pgmap v2783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4131.203480 secs 2024-07-26T08:05:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:01 smithi042 ceph-mon[32775]: pgmap v2783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4131.203480 secs 2024-07-26T08:05:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4148.945735 secs 2024-07-26T08:05:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4148.945735 secs 2024-07-26T08:05:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4148.945735 secs 2024-07-26T08:05:03.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:03 smithi164 ceph-mon[36970]: pgmap v2784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:03 smithi163 ceph-mon[36873]: pgmap v2784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:03.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:03 smithi042 ceph-mon[32775]: pgmap v2784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:05.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:05 smithi164 ceph-mon[36970]: pgmap v2785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:05 smithi163 ceph-mon[36873]: pgmap v2785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:05 smithi042 ceph-mon[32775]: pgmap v2785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4136.203617 secs 2024-07-26T08:05:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4136.203617 secs 2024-07-26T08:05:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4136.203617 secs 2024-07-26T08:05:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4153.945860 secs 2024-07-26T08:05:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:07 smithi164 ceph-mon[36970]: pgmap v2786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4153.945860 secs 2024-07-26T08:05:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:07 smithi163 ceph-mon[36873]: pgmap v2786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4153.945860 secs 2024-07-26T08:05:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:07 smithi042 ceph-mon[32775]: pgmap v2786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:05:09.669+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:05:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:05:09.669+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:05:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:05:09.669+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:05:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:09 smithi164 ceph-mon[36970]: pgmap v2787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:05:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:09 smithi163 ceph-mon[36873]: pgmap v2787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:05:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:09 smithi042 ceph-mon[32775]: pgmap v2787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:05:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:11 smithi042 ceph-mon[32775]: pgmap v2788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4141.203741 secs 2024-07-26T08:05:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:11 smithi164 ceph-mon[36970]: pgmap v2788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4141.203741 secs 2024-07-26T08:05:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:11 smithi163 ceph-mon[36873]: pgmap v2788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4141.203741 secs 2024-07-26T08:05:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4158.945960 secs 2024-07-26T08:05:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4158.945960 secs 2024-07-26T08:05:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4158.945960 secs 2024-07-26T08:05:13.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:13 smithi164 ceph-mon[36970]: pgmap v2789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:13 smithi163 ceph-mon[36873]: pgmap v2789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:13 smithi042 ceph-mon[32775]: pgmap v2789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:15 smithi164 ceph-mon[36970]: pgmap v2790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:15 smithi163 ceph-mon[36873]: pgmap v2790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:15 smithi042 ceph-mon[32775]: pgmap v2790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4146.203862 secs 2024-07-26T08:05:16.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4146.203862 secs 2024-07-26T08:05:16.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4146.203862 secs 2024-07-26T08:05:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4163.946089 secs 2024-07-26T08:05:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:17 smithi163 ceph-mon[36873]: pgmap v2791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4163.946089 secs 2024-07-26T08:05:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:17 smithi042 ceph-mon[32775]: pgmap v2791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4163.946089 secs 2024-07-26T08:05:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:17 smithi164 ceph-mon[36970]: pgmap v2791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:19 smithi163 ceph-mon[36873]: pgmap v2792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:05:19.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:19 smithi042 ceph-mon[32775]: pgmap v2792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:05:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:19 smithi164 ceph-mon[36970]: pgmap v2792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:05:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:21 smithi163 ceph-mon[36873]: pgmap v2793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4151.203991 secs 2024-07-26T08:05:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:21 smithi042 ceph-mon[32775]: pgmap v2793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4151.203991 secs 2024-07-26T08:05:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:21 smithi164 ceph-mon[36970]: pgmap v2793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4151.203991 secs 2024-07-26T08:05:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4168.946181 secs 2024-07-26T08:05:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:05:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4168.946181 secs 2024-07-26T08:05:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:05:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4168.946181 secs 2024-07-26T08:05:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:05:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:23 smithi163 ceph-mon[36873]: pgmap v2794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:05:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:05:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:23 smithi163 ceph-mon[36873]: pgmap v2795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:05:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:23 smithi163 ceph-mon[36873]: pgmap v2796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:05:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:05:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:23 smithi042 ceph-mon[32775]: pgmap v2794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:05:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:05:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:23 smithi042 ceph-mon[32775]: pgmap v2795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:05:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:23 smithi042 ceph-mon[32775]: pgmap v2796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:05:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:05:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:23 smithi164 ceph-mon[36970]: pgmap v2794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:05:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:05:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:23 smithi164 ceph-mon[36970]: pgmap v2795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:05:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:23 smithi164 ceph-mon[36970]: pgmap v2796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:05:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:05:24.673 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:05:24.671+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:05:24.673 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:05:24.671+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:05:24.674 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:05:24.671+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:05:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:24 smithi164 ceph-mon[36970]: pgmap v2797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:05:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:24 smithi163 ceph-mon[36873]: pgmap v2797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:05:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:24 smithi042 ceph-mon[32775]: pgmap v2797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:05:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4156.204077 secs 2024-07-26T08:05:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4156.204077 secs 2024-07-26T08:05:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4156.204077 secs 2024-07-26T08:05:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4173.946336 secs 2024-07-26T08:05:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:27 smithi042 ceph-mon[32775]: pgmap v2798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:05:27.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4173.946336 secs 2024-07-26T08:05:27.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:27 smithi163 ceph-mon[36873]: pgmap v2798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:05:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4173.946336 secs 2024-07-26T08:05:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:27 smithi164 ceph-mon[36970]: pgmap v2798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:05:29.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:28 smithi164 ceph-mon[36970]: pgmap v2799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:05:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:28 smithi163 ceph-mon[36873]: pgmap v2799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:05:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:28 smithi042 ceph-mon[32775]: pgmap v2799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:05:31.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:30 smithi164 ceph-mon[36970]: pgmap v2800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:05:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:30 smithi163 ceph-mon[36873]: pgmap v2800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:05:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:30 smithi042 ceph-mon[32775]: pgmap v2800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:05:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4161.204196 secs 2024-07-26T08:05:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4178.946461 secs 2024-07-26T08:05:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4161.204196 secs 2024-07-26T08:05:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4178.946461 secs 2024-07-26T08:05:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4161.204196 secs 2024-07-26T08:05:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4178.946461 secs 2024-07-26T08:05:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:32 smithi164 ceph-mon[36970]: pgmap v2801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:05:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:32 smithi163 ceph-mon[36873]: pgmap v2801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:05:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:32 smithi042 ceph-mon[32775]: pgmap v2801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:05:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:34 smithi164 ceph-mon[36970]: pgmap v2802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:34 smithi163 ceph-mon[36873]: pgmap v2802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:34 smithi042 ceph-mon[32775]: pgmap v2802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4166.204317 secs 2024-07-26T08:05:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4166.204317 secs 2024-07-26T08:05:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4166.204317 secs 2024-07-26T08:05:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4183.946599 secs 2024-07-26T08:05:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:37 smithi042 ceph-mon[32775]: pgmap v2803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4183.946599 secs 2024-07-26T08:05:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:37 smithi164 ceph-mon[36970]: pgmap v2803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4183.946599 secs 2024-07-26T08:05:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:37 smithi163 ceph-mon[36873]: pgmap v2803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:39.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:38 smithi164 ceph-mon[36970]: pgmap v2804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:05:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:38 smithi163 ceph-mon[36873]: pgmap v2804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:05:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:38 smithi042 ceph-mon[32775]: pgmap v2804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:05:39.674 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:05:39.673+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:05:39.674 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:05:39.673+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:05:39.674 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:05:39.673+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:05:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:40 smithi164 ceph-mon[36970]: pgmap v2805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:05:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:40 smithi163 ceph-mon[36873]: pgmap v2805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:05:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:40 smithi042 ceph-mon[32775]: pgmap v2805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:05:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4171.204437 secs 2024-07-26T08:05:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4188.946749 secs 2024-07-26T08:05:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4171.204437 secs 2024-07-26T08:05:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4188.946749 secs 2024-07-26T08:05:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4171.204437 secs 2024-07-26T08:05:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4188.946749 secs 2024-07-26T08:05:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:42 smithi164 ceph-mon[36970]: pgmap v2806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:05:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:42 smithi163 ceph-mon[36873]: pgmap v2806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:05:43.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:42 smithi042 ceph-mon[32775]: pgmap v2806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:05:45.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:44 smithi164 ceph-mon[36970]: pgmap v2807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:05:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:44 smithi163 ceph-mon[36873]: pgmap v2807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:05:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:44 smithi042 ceph-mon[32775]: pgmap v2807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:05:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4176.204562 secs 2024-07-26T08:05:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4176.204562 secs 2024-07-26T08:05:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4176.204562 secs 2024-07-26T08:05:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4193.946870 secs 2024-07-26T08:05:47.676 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:47 smithi042 ceph-mon[32775]: pgmap v2808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:05:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4193.946870 secs 2024-07-26T08:05:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:47 smithi164 ceph-mon[36970]: pgmap v2808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:05:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4193.946870 secs 2024-07-26T08:05:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:47 smithi163 ceph-mon[36873]: pgmap v2808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:05:49.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:48 smithi164 ceph-mon[36970]: pgmap v2809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:05:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:48 smithi163 ceph-mon[36873]: pgmap v2809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:05:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:48 smithi042 ceph-mon[32775]: pgmap v2809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:05:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:50 smithi164 ceph-mon[36970]: pgmap v2810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:05:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:50 smithi163 ceph-mon[36873]: pgmap v2810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:05:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:50 smithi042 ceph-mon[32775]: pgmap v2810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:05:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4181.204688 secs 2024-07-26T08:05:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4198.946961 secs 2024-07-26T08:05:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4181.204688 secs 2024-07-26T08:05:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4198.946961 secs 2024-07-26T08:05:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4181.204688 secs 2024-07-26T08:05:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4198.946961 secs 2024-07-26T08:05:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:52 smithi164 ceph-mon[36970]: pgmap v2811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:52 smithi163 ceph-mon[36873]: pgmap v2811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:52 smithi042 ceph-mon[32775]: pgmap v2811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:05:54.677+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:05:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:05:54.677+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:05:54.678 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:05:54.677+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:05:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:54 smithi164 ceph-mon[36970]: pgmap v2812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:55.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:54 smithi163 ceph-mon[36873]: pgmap v2812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:54 smithi042 ceph-mon[32775]: pgmap v2812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:05:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4186.204814 secs 2024-07-26T08:05:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4186.204814 secs 2024-07-26T08:05:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4186.204814 secs 2024-07-26T08:05:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4203.947091 secs 2024-07-26T08:05:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:57 smithi042 ceph-mon[32775]: pgmap v2813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:57.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4203.947091 secs 2024-07-26T08:05:57.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:57 smithi163 ceph-mon[36873]: pgmap v2813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4203.947091 secs 2024-07-26T08:05:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:57 smithi164 ceph-mon[36970]: pgmap v2813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:05:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:05:58 smithi164 ceph-mon[36970]: pgmap v2814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:05:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:05:58 smithi163 ceph-mon[36873]: pgmap v2814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:05:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:05:58 smithi042 ceph-mon[32775]: pgmap v2814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:06:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:00 smithi164 ceph-mon[36970]: pgmap v2815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:00 smithi163 ceph-mon[36873]: pgmap v2815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:00 smithi042 ceph-mon[32775]: pgmap v2815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4191.204935 secs 2024-07-26T08:06:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4208.947220 secs 2024-07-26T08:06:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4191.204935 secs 2024-07-26T08:06:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4208.947220 secs 2024-07-26T08:06:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4191.204935 secs 2024-07-26T08:06:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4208.947220 secs 2024-07-26T08:06:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:02 smithi164 ceph-mon[36970]: pgmap v2816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:02 smithi163 ceph-mon[36873]: pgmap v2816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:02 smithi042 ceph-mon[32775]: pgmap v2816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:04 smithi164 ceph-mon[36970]: pgmap v2817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:04 smithi163 ceph-mon[36873]: pgmap v2817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:04 smithi042 ceph-mon[32775]: pgmap v2817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4196.205058 secs 2024-07-26T08:06:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4196.205058 secs 2024-07-26T08:06:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4196.205058 secs 2024-07-26T08:06:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4213.947306 secs 2024-07-26T08:06:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:07 smithi042 ceph-mon[32775]: pgmap v2818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4213.947306 secs 2024-07-26T08:06:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:07 smithi164 ceph-mon[36970]: pgmap v2818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4213.947306 secs 2024-07-26T08:06:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:07 smithi163 ceph-mon[36873]: pgmap v2818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:08 smithi164 ceph-mon[36970]: pgmap v2819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:06:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:08 smithi163 ceph-mon[36873]: pgmap v2819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:06:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:08 smithi042 ceph-mon[32775]: pgmap v2819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:06:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:06:09.686+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:06:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:06:09.686+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:06:09.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:06:09.686+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:06:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:10 smithi164 ceph-mon[36970]: pgmap v2820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:10 smithi163 ceph-mon[36873]: pgmap v2820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:10 smithi042 ceph-mon[32775]: pgmap v2820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4201.205180 secs 2024-07-26T08:06:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4218.947453 secs 2024-07-26T08:06:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4201.205180 secs 2024-07-26T08:06:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4218.947453 secs 2024-07-26T08:06:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4201.205180 secs 2024-07-26T08:06:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4218.947453 secs 2024-07-26T08:06:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:12 smithi164 ceph-mon[36970]: pgmap v2821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:13.232 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:12 smithi042 ceph-mon[32775]: pgmap v2821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:12 smithi163 ceph-mon[36873]: pgmap v2821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:14 smithi164 ceph-mon[36970]: pgmap v2822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:14 smithi163 ceph-mon[36873]: pgmap v2822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:14 smithi042 ceph-mon[32775]: pgmap v2822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4206.205306 secs 2024-07-26T08:06:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4206.205306 secs 2024-07-26T08:06:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4206.205306 secs 2024-07-26T08:06:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4223.947591 secs 2024-07-26T08:06:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:17 smithi042 ceph-mon[32775]: pgmap v2823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4223.947591 secs 2024-07-26T08:06:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:17 smithi164 ceph-mon[36970]: pgmap v2823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4223.947591 secs 2024-07-26T08:06:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:17 smithi163 ceph-mon[36873]: pgmap v2823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:18 smithi164 ceph-mon[36970]: pgmap v2824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:06:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:18 smithi163 ceph-mon[36873]: pgmap v2824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:06:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:18 smithi042 ceph-mon[32775]: pgmap v2824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:06:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:20 smithi164 ceph-mon[36970]: pgmap v2825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:20 smithi163 ceph-mon[36873]: pgmap v2825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:20 smithi042 ceph-mon[32775]: pgmap v2825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4211.205432 secs 2024-07-26T08:06:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4228.947736 secs 2024-07-26T08:06:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4211.205432 secs 2024-07-26T08:06:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4228.947736 secs 2024-07-26T08:06:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4211.205432 secs 2024-07-26T08:06:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4228.947736 secs 2024-07-26T08:06:23.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:22 smithi164 ceph-mon[36970]: pgmap v2826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:06:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:22 smithi163 ceph-mon[36873]: pgmap v2826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:06:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:22 smithi042 ceph-mon[32775]: pgmap v2826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:06:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:06:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:06:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:23 smithi164 ceph-mon[36970]: pgmap v2827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:06:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:23 smithi164 ceph-mon[36970]: pgmap v2828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:06:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:06:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:06:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:06:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:23 smithi163 ceph-mon[36873]: pgmap v2827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:06:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:23 smithi163 ceph-mon[36873]: pgmap v2828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:06:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:06:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:06:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:06:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:23 smithi042 ceph-mon[32775]: pgmap v2827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:06:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:23 smithi042 ceph-mon[32775]: pgmap v2828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:06:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:06:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:06:24.690+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:06:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:06:24.690+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:06:24.691 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:06:24.690+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:06:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:26 smithi164 ceph-mon[36970]: pgmap v2829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:06:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4216.205554 secs 2024-07-26T08:06:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:26 smithi163 ceph-mon[36873]: pgmap v2829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:06:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4216.205554 secs 2024-07-26T08:06:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:26 smithi042 ceph-mon[32775]: pgmap v2829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:06:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4216.205554 secs 2024-07-26T08:06:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4233.947862 secs 2024-07-26T08:06:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4233.947862 secs 2024-07-26T08:06:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4233.947862 secs 2024-07-26T08:06:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:28 smithi164 ceph-mon[36970]: pgmap v2830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:06:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:28 smithi163 ceph-mon[36873]: pgmap v2830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:06:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:28 smithi042 ceph-mon[32775]: pgmap v2830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:06:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:30 smithi164 ceph-mon[36970]: pgmap v2831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:06:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:30 smithi163 ceph-mon[36873]: pgmap v2831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:06:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:30 smithi042 ceph-mon[32775]: pgmap v2831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:06:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4221.205677 secs 2024-07-26T08:06:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4221.205677 secs 2024-07-26T08:06:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4221.205677 secs 2024-07-26T08:06:32.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4238.948014 secs 2024-07-26T08:06:32.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:32 smithi164 ceph-mon[36970]: pgmap v2832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:06:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4238.948014 secs 2024-07-26T08:06:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:32 smithi163 ceph-mon[36873]: pgmap v2832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:06:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4238.948014 secs 2024-07-26T08:06:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:32 smithi042 ceph-mon[32775]: pgmap v2832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:06:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:34 smithi164 ceph-mon[36970]: pgmap v2833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:06:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:34 smithi163 ceph-mon[36873]: pgmap v2833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:06:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:34 smithi042 ceph-mon[32775]: pgmap v2833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:06:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:36 smithi164 ceph-mon[36970]: pgmap v2834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4226.205756 secs 2024-07-26T08:06:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:36 smithi163 ceph-mon[36873]: pgmap v2834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4226.205756 secs 2024-07-26T08:06:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:36 smithi042 ceph-mon[32775]: pgmap v2834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4226.205756 secs 2024-07-26T08:06:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4243.948134 secs 2024-07-26T08:06:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4243.948134 secs 2024-07-26T08:06:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4243.948134 secs 2024-07-26T08:06:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:38 smithi164 ceph-mon[36970]: pgmap v2835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:38 smithi163 ceph-mon[36873]: pgmap v2835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:38 smithi042 ceph-mon[32775]: pgmap v2835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:39.699 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:06:39.699+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:06:39.700 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:06:39.699+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:06:39.700 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:06:39.699+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:06:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:40 smithi164 ceph-mon[36970]: pgmap v2836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:40 smithi163 ceph-mon[36873]: pgmap v2836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:40 smithi042 ceph-mon[32775]: pgmap v2836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4231.205878 secs 2024-07-26T08:06:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4231.205878 secs 2024-07-26T08:06:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4231.205878 secs 2024-07-26T08:06:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4248.948263 secs 2024-07-26T08:06:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:42 smithi163 ceph-mon[36873]: pgmap v2837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4248.948263 secs 2024-07-26T08:06:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:42 smithi042 ceph-mon[32775]: pgmap v2837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4248.948263 secs 2024-07-26T08:06:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:42 smithi164 ceph-mon[36970]: pgmap v2837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:44 smithi163 ceph-mon[36873]: pgmap v2838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:06:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:44 smithi042 ceph-mon[32775]: pgmap v2838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:06:44.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:44 smithi164 ceph-mon[36970]: pgmap v2838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:06:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:46 smithi163 ceph-mon[36873]: pgmap v2839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4236.205986 secs 2024-07-26T08:06:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:46 smithi042 ceph-mon[32775]: pgmap v2839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4236.205986 secs 2024-07-26T08:06:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:46 smithi164 ceph-mon[36970]: pgmap v2839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4236.205986 secs 2024-07-26T08:06:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4253.948382 secs 2024-07-26T08:06:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4253.948382 secs 2024-07-26T08:06:47.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4253.948382 secs 2024-07-26T08:06:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:48 smithi163 ceph-mon[36873]: pgmap v2840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:48.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:48 smithi042 ceph-mon[32775]: pgmap v2840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:48 smithi164 ceph-mon[36970]: pgmap v2840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:50 smithi163 ceph-mon[36873]: pgmap v2841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:50 smithi042 ceph-mon[32775]: pgmap v2841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:50 smithi164 ceph-mon[36970]: pgmap v2841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4241.206111 secs 2024-07-26T08:06:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4241.206111 secs 2024-07-26T08:06:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4241.206111 secs 2024-07-26T08:06:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4258.948502 secs 2024-07-26T08:06:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:52 smithi163 ceph-mon[36873]: pgmap v2842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4258.948502 secs 2024-07-26T08:06:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:52 smithi042 ceph-mon[32775]: pgmap v2842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4258.948502 secs 2024-07-26T08:06:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:52 smithi164 ceph-mon[36970]: pgmap v2842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:06:54.700+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:06:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:06:54.700+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:06:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:06:54.700+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:06:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:54 smithi163 ceph-mon[36873]: pgmap v2843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:06:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:54 smithi042 ceph-mon[32775]: pgmap v2843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:06:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:54 smithi164 ceph-mon[36970]: pgmap v2843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:06:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:56 smithi042 ceph-mon[32775]: pgmap v2844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4246.206254 secs 2024-07-26T08:06:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:56 smithi164 ceph-mon[36970]: pgmap v2844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4246.206254 secs 2024-07-26T08:06:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:56 smithi163 ceph-mon[36873]: pgmap v2844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:06:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4246.206254 secs 2024-07-26T08:06:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4263.948605 secs 2024-07-26T08:06:57.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4263.948605 secs 2024-07-26T08:06:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4263.948605 secs 2024-07-26T08:06:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:06:58 smithi042 ceph-mon[32775]: pgmap v2845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:06:58 smithi164 ceph-mon[36970]: pgmap v2845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:06:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:06:58 smithi163 ceph-mon[36873]: pgmap v2845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:00 smithi042 ceph-mon[32775]: pgmap v2846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:00 smithi164 ceph-mon[36970]: pgmap v2846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:00 smithi163 ceph-mon[36873]: pgmap v2846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4251.206357 secs 2024-07-26T08:07:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4251.206357 secs 2024-07-26T08:07:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4251.206357 secs 2024-07-26T08:07:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4268.948733 secs 2024-07-26T08:07:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:02 smithi042 ceph-mon[32775]: pgmap v2847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4268.948733 secs 2024-07-26T08:07:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:02 smithi164 ceph-mon[36970]: pgmap v2847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4268.948733 secs 2024-07-26T08:07:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:02 smithi163 ceph-mon[36873]: pgmap v2847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:04.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:04 smithi042 ceph-mon[32775]: pgmap v2848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:07:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:04 smithi164 ceph-mon[36970]: pgmap v2848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:07:05.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:04 smithi163 ceph-mon[36873]: pgmap v2848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:07:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:06 smithi042 ceph-mon[32775]: pgmap v2849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4256.206485 secs 2024-07-26T08:07:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:06 smithi164 ceph-mon[36970]: pgmap v2849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4256.206485 secs 2024-07-26T08:07:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:06 smithi163 ceph-mon[36873]: pgmap v2849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4256.206485 secs 2024-07-26T08:07:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4273.948877 secs 2024-07-26T08:07:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4273.948877 secs 2024-07-26T08:07:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4273.948877 secs 2024-07-26T08:07:08.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:08 smithi042 ceph-mon[32775]: pgmap v2850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:08 smithi164 ceph-mon[36970]: pgmap v2850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:09.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:08 smithi163 ceph-mon[36873]: pgmap v2850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:07:09.709+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:07:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:07:09.709+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:07:09.710 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:07:09.709+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:07:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:10 smithi042 ceph-mon[32775]: pgmap v2851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:10.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:10 smithi164 ceph-mon[36970]: pgmap v2851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:10 smithi163 ceph-mon[36873]: pgmap v2851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4261.206616 secs 2024-07-26T08:07:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4261.206616 secs 2024-07-26T08:07:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4261.206616 secs 2024-07-26T08:07:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4278.948997 secs 2024-07-26T08:07:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:12 smithi042 ceph-mon[32775]: pgmap v2852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4278.948997 secs 2024-07-26T08:07:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:12 smithi164 ceph-mon[36970]: pgmap v2852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4278.948997 secs 2024-07-26T08:07:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:12 smithi163 ceph-mon[36873]: pgmap v2852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:14 smithi042 ceph-mon[32775]: pgmap v2853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:07:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:14 smithi164 ceph-mon[36970]: pgmap v2853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:07:15.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:14 smithi163 ceph-mon[36873]: pgmap v2853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:07:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:16 smithi042 ceph-mon[32775]: pgmap v2854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4266.206767 secs 2024-07-26T08:07:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:16 smithi164 ceph-mon[36970]: pgmap v2854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4266.206767 secs 2024-07-26T08:07:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:16 smithi163 ceph-mon[36873]: pgmap v2854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4266.206767 secs 2024-07-26T08:07:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4283.949088 secs 2024-07-26T08:07:17.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4283.949088 secs 2024-07-26T08:07:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4283.949088 secs 2024-07-26T08:07:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:18 smithi042 ceph-mon[32775]: pgmap v2855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:18 smithi164 ceph-mon[36970]: pgmap v2855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:19.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:18 smithi163 ceph-mon[36873]: pgmap v2855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:20 smithi042 ceph-mon[32775]: pgmap v2856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:20.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:20 smithi164 ceph-mon[36970]: pgmap v2856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:20 smithi163 ceph-mon[36873]: pgmap v2856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4271.206897 secs 2024-07-26T08:07:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4271.206897 secs 2024-07-26T08:07:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4271.206897 secs 2024-07-26T08:07:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4288.949225 secs 2024-07-26T08:07:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:22 smithi042 ceph-mon[32775]: pgmap v2857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4288.949225 secs 2024-07-26T08:07:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:22 smithi164 ceph-mon[36970]: pgmap v2857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4288.949225 secs 2024-07-26T08:07:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:22 smithi163 ceph-mon[36873]: pgmap v2857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:07:23.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:07:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:07:24.713 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:07:24.712+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:07:24.713 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:07:24.712+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:07:24.713 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:07:24.712+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:07:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:24 smithi042 ceph-mon[32775]: pgmap v2858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:07:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:07:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:07:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:07:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:24 smithi164 ceph-mon[36970]: pgmap v2858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:07:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:24.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:07:24.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:07:24.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:07:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:24 smithi163 ceph-mon[36873]: pgmap v2858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:07:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:07:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:07:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:07:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:07:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:25 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:07:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:25 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:07:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:25 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:07:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:25 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:07:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:25 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:07:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:25 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:07:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:25 smithi042 ceph-mon[32775]: pgmap v2859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:07:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:25 smithi042 ceph-mon[32775]: pgmap v2860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:07:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:25 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:07:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:25 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:07:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:25 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:07:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:25 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:07:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:25 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:07:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:25 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:07:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:25 smithi164 ceph-mon[36970]: pgmap v2859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:07:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:25 smithi164 ceph-mon[36970]: pgmap v2860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:07:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:25 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:07:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:25 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:07:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:25 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:07:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:25 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:07:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:25 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:07:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:25 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:07:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:25 smithi163 ceph-mon[36873]: pgmap v2859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:07:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:25 smithi163 ceph-mon[36873]: pgmap v2860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:07:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4276.207041 secs 2024-07-26T08:07:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4276.207041 secs 2024-07-26T08:07:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4276.207041 secs 2024-07-26T08:07:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:27 smithi042 ceph-mon[32775]: pgmap v2861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:07:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4293.949343 secs 2024-07-26T08:07:27.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:27 smithi163 ceph-mon[36873]: pgmap v2861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:07:27.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4293.949343 secs 2024-07-26T08:07:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:27 smithi164 ceph-mon[36970]: pgmap v2861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:07:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4293.949343 secs 2024-07-26T08:07:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:29 smithi042 ceph-mon[32775]: pgmap v2862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:07:29.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:29 smithi164 ceph-mon[36970]: pgmap v2862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:07:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:29 smithi163 ceph-mon[36873]: pgmap v2862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:07:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:31 smithi042 ceph-mon[32775]: pgmap v2863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:07:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4281.207166 secs 2024-07-26T08:07:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:31 smithi164 ceph-mon[36970]: pgmap v2863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:07:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4281.207166 secs 2024-07-26T08:07:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:31 smithi163 ceph-mon[36873]: pgmap v2863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:07:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4281.207166 secs 2024-07-26T08:07:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4298.949464 secs 2024-07-26T08:07:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4298.949464 secs 2024-07-26T08:07:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4298.949464 secs 2024-07-26T08:07:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:33 smithi042 ceph-mon[32775]: pgmap v2864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:07:33.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:33 smithi164 ceph-mon[36970]: pgmap v2864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:07:34.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:33 smithi163 ceph-mon[36873]: pgmap v2864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:07:35.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:35 smithi042 ceph-mon[32775]: pgmap v2865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:07:35.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:35 smithi164 ceph-mon[36970]: pgmap v2865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:07:36.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:35 smithi163 ceph-mon[36873]: pgmap v2865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:07:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4286.207292 secs 2024-07-26T08:07:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4286.207292 secs 2024-07-26T08:07:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4286.207292 secs 2024-07-26T08:07:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:37 smithi042 ceph-mon[32775]: pgmap v2866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4303.949571 secs 2024-07-26T08:07:37.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:37 smithi164 ceph-mon[36970]: pgmap v2866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4303.949571 secs 2024-07-26T08:07:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:37 smithi163 ceph-mon[36873]: pgmap v2866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4303.949571 secs 2024-07-26T08:07:39.719 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:07:39.718+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:07:39.719 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:07:39.718+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:07:39.719 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:07:39.718+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:07:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:39 smithi042 ceph-mon[32775]: pgmap v2867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:39 smithi164 ceph-mon[36970]: pgmap v2867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:40.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:39 smithi163 ceph-mon[36873]: pgmap v2867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:41 smithi042 ceph-mon[32775]: pgmap v2868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4291.207415 secs 2024-07-26T08:07:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:41 smithi164 ceph-mon[36970]: pgmap v2868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4291.207415 secs 2024-07-26T08:07:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:41 smithi163 ceph-mon[36873]: pgmap v2868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4291.207415 secs 2024-07-26T08:07:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4308.949670 secs 2024-07-26T08:07:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4308.949670 secs 2024-07-26T08:07:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4308.949670 secs 2024-07-26T08:07:43.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:43 smithi042 ceph-mon[32775]: pgmap v2869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:43.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:43 smithi164 ceph-mon[36970]: pgmap v2869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:43 smithi163 ceph-mon[36873]: pgmap v2869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:45 smithi042 ceph-mon[32775]: pgmap v2870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:07:45.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:45 smithi164 ceph-mon[36970]: pgmap v2870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:07:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:45 smithi163 ceph-mon[36873]: pgmap v2870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:07:46.928 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4296.207546 secs 2024-07-26T08:07:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4296.207546 secs 2024-07-26T08:07:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4296.207546 secs 2024-07-26T08:07:47.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:47 smithi164 ceph-mon[36970]: pgmap v2871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4313.949792 secs 2024-07-26T08:07:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:47 smithi163 ceph-mon[36873]: pgmap v2871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4313.949792 secs 2024-07-26T08:07:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:47 smithi042 ceph-mon[32775]: pgmap v2871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4313.949792 secs 2024-07-26T08:07:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:49 smithi164 ceph-mon[36970]: pgmap v2872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:49 smithi163 ceph-mon[36873]: pgmap v2872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:50.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:49 smithi042 ceph-mon[32775]: pgmap v2872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:51 smithi164 ceph-mon[36970]: pgmap v2873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4301.207675 secs 2024-07-26T08:07:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:51 smithi163 ceph-mon[36873]: pgmap v2873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4301.207675 secs 2024-07-26T08:07:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:51 smithi042 ceph-mon[32775]: pgmap v2873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4301.207675 secs 2024-07-26T08:07:52.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4318.949889 secs 2024-07-26T08:07:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4318.949889 secs 2024-07-26T08:07:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4318.949889 secs 2024-07-26T08:07:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:53 smithi164 ceph-mon[36970]: pgmap v2874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:53 smithi163 ceph-mon[36873]: pgmap v2874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:53 smithi042 ceph-mon[32775]: pgmap v2874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:07:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:07:54.721+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:07:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:07:54.721+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:07:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:07:54.721+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:07:55.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:55 smithi164 ceph-mon[36970]: pgmap v2875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:07:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:55 smithi163 ceph-mon[36873]: pgmap v2875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:07:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:55 smithi042 ceph-mon[32775]: pgmap v2875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:07:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4306.207833 secs 2024-07-26T08:07:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4306.207833 secs 2024-07-26T08:07:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4306.207833 secs 2024-07-26T08:07:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:57 smithi164 ceph-mon[36970]: pgmap v2876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4323.950012 secs 2024-07-26T08:07:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:57 smithi163 ceph-mon[36873]: pgmap v2876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4323.950012 secs 2024-07-26T08:07:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:57 smithi042 ceph-mon[32775]: pgmap v2876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:07:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4323.950012 secs 2024-07-26T08:07:58.542 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:07:58.541+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:07:58.542 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:07:58.541+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:07:58.542 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:07:58.541+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:07:58.542 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:07:58.541+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:07:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:07:59 smithi164 ceph-mon[36970]: pgmap v2877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:07:59 smithi163 ceph-mon[36873]: pgmap v2877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:07:59 smithi042 ceph-mon[32775]: pgmap v2877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:01.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:01 smithi042 ceph-mon[32775]: pgmap v2878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:01.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4311.207979 secs 2024-07-26T08:08:01.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:01 smithi164 ceph-mon[36970]: pgmap v2878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4311.207979 secs 2024-07-26T08:08:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:01 smithi163 ceph-mon[36873]: pgmap v2878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4311.207979 secs 2024-07-26T08:08:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4328.950138 secs 2024-07-26T08:08:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4328.950138 secs 2024-07-26T08:08:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4328.950138 secs 2024-07-26T08:08:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:03 smithi164 ceph-mon[36970]: pgmap v2879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:04.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:03 smithi163 ceph-mon[36873]: pgmap v2879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:03 smithi042 ceph-mon[32775]: pgmap v2879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:05 smithi164 ceph-mon[36970]: pgmap v2880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:08:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:05 smithi163 ceph-mon[36873]: pgmap v2880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:08:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:05 smithi042 ceph-mon[32775]: pgmap v2880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:08:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4316.208123 secs 2024-07-26T08:08:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4316.208123 secs 2024-07-26T08:08:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4316.208123 secs 2024-07-26T08:08:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:07 smithi164 ceph-mon[36970]: pgmap v2881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4333.950236 secs 2024-07-26T08:08:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:07 smithi163 ceph-mon[36873]: pgmap v2881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4333.950236 secs 2024-07-26T08:08:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:07 smithi042 ceph-mon[32775]: pgmap v2881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4333.950236 secs 2024-07-26T08:08:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:08:09.729+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:08:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:08:09.729+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:08:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:08:09.729+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:08:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:09 smithi164 ceph-mon[36970]: pgmap v2882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:09 smithi163 ceph-mon[36873]: pgmap v2882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:09 smithi042 ceph-mon[32775]: pgmap v2882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:11 smithi164 ceph-mon[36970]: pgmap v2883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4321.208247 secs 2024-07-26T08:08:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4338.950364 secs 2024-07-26T08:08:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:11 smithi163 ceph-mon[36873]: pgmap v2883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4321.208247 secs 2024-07-26T08:08:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4338.950364 secs 2024-07-26T08:08:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:11 smithi042 ceph-mon[32775]: pgmap v2883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4321.208247 secs 2024-07-26T08:08:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4338.950364 secs 2024-07-26T08:08:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:13 smithi164 ceph-mon[36970]: pgmap v2884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:13 smithi163 ceph-mon[36873]: pgmap v2884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:13 smithi042 ceph-mon[32775]: pgmap v2884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:15.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:15 smithi164 ceph-mon[36970]: pgmap v2885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:08:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:15 smithi163 ceph-mon[36873]: pgmap v2885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:08:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:15 smithi042 ceph-mon[32775]: pgmap v2885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:08:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4326.208376 secs 2024-07-26T08:08:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4343.950462 secs 2024-07-26T08:08:17.137 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4326.208376 secs 2024-07-26T08:08:17.137 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4343.950462 secs 2024-07-26T08:08:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4326.208376 secs 2024-07-26T08:08:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4343.950462 secs 2024-07-26T08:08:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:17 smithi163 ceph-mon[36873]: pgmap v2886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:17 smithi042 ceph-mon[32775]: pgmap v2886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:18.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:17 smithi164 ceph-mon[36970]: pgmap v2886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:20.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:19 smithi163 ceph-mon[36873]: pgmap v2887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:19 smithi042 ceph-mon[32775]: pgmap v2887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:20.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:19 smithi164 ceph-mon[36970]: pgmap v2887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:21 smithi163 ceph-mon[36873]: pgmap v2888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4331.208507 secs 2024-07-26T08:08:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4348.950525 secs 2024-07-26T08:08:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:21 smithi042 ceph-mon[32775]: pgmap v2888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4331.208507 secs 2024-07-26T08:08:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4348.950525 secs 2024-07-26T08:08:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:21 smithi164 ceph-mon[36970]: pgmap v2888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4331.208507 secs 2024-07-26T08:08:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4348.950525 secs 2024-07-26T08:08:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:23 smithi163 ceph-mon[36873]: pgmap v2889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:24.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:23 smithi042 ceph-mon[32775]: pgmap v2889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:24.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:23 smithi164 ceph-mon[36970]: pgmap v2889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:08:24.732+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:08:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:08:24.732+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:08:24.733 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:08:24.732+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:08:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:08:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:08:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:08:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:08:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:08:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:08:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:08:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:08:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:08:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:08:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:08:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:08:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:25 smithi163 ceph-mon[36873]: pgmap v2890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:08:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:25 smithi163 ceph-mon[36873]: pgmap v2891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T08:08:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:25 smithi163 ceph-mon[36873]: pgmap v2892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:08:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:25 smithi042 ceph-mon[32775]: pgmap v2890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:08:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:25 smithi042 ceph-mon[32775]: pgmap v2891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T08:08:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:25 smithi042 ceph-mon[32775]: pgmap v2892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:08:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:25 smithi164 ceph-mon[36970]: pgmap v2890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:08:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:25 smithi164 ceph-mon[36970]: pgmap v2891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T08:08:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:25 smithi164 ceph-mon[36970]: pgmap v2892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:08:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4336.208575 secs 2024-07-26T08:08:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4353.950652 secs 2024-07-26T08:08:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4336.208575 secs 2024-07-26T08:08:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4353.950652 secs 2024-07-26T08:08:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4336.208575 secs 2024-07-26T08:08:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4353.950652 secs 2024-07-26T08:08:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:27 smithi163 ceph-mon[36873]: pgmap v2893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:08:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:27 smithi042 ceph-mon[32775]: pgmap v2893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:08:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:27 smithi164 ceph-mon[36970]: pgmap v2893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:08:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:29 smithi163 ceph-mon[36873]: pgmap v2894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:08:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:29 smithi042 ceph-mon[32775]: pgmap v2894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:08:30.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:29 smithi164 ceph-mon[36970]: pgmap v2894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:08:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:31 smithi042 ceph-mon[32775]: pgmap v2895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:08:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4341.208653 secs 2024-07-26T08:08:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4358.950773 secs 2024-07-26T08:08:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:31 smithi164 ceph-mon[36970]: pgmap v2895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:08:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4341.208653 secs 2024-07-26T08:08:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4358.950773 secs 2024-07-26T08:08:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:31 smithi163 ceph-mon[36873]: pgmap v2895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:08:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4341.208653 secs 2024-07-26T08:08:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4358.950773 secs 2024-07-26T08:08:34.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:33 smithi042 ceph-mon[32775]: pgmap v2896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:08:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:33 smithi164 ceph-mon[36970]: pgmap v2896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:08:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:33 smithi163 ceph-mon[36873]: pgmap v2896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:08:36.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:35 smithi164 ceph-mon[36970]: pgmap v2897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:08:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:35 smithi042 ceph-mon[32775]: pgmap v2897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:08:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:35 smithi163 ceph-mon[36873]: pgmap v2897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:08:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4346.208771 secs 2024-07-26T08:08:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4363.950920 secs 2024-07-26T08:08:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4346.208771 secs 2024-07-26T08:08:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4363.950920 secs 2024-07-26T08:08:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4346.208771 secs 2024-07-26T08:08:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4363.950920 secs 2024-07-26T08:08:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:37 smithi042 ceph-mon[32775]: pgmap v2898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:37 smithi164 ceph-mon[36970]: pgmap v2898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:37 smithi163 ceph-mon[36873]: pgmap v2898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:08:39.740+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:08:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:08:39.740+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:08:39.741 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:08:39.740+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:08:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:39 smithi042 ceph-mon[32775]: pgmap v2899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:08:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:39 smithi164 ceph-mon[36970]: pgmap v2899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:08:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:39 smithi163 ceph-mon[36873]: pgmap v2899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:08:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:41 smithi042 ceph-mon[32775]: pgmap v2900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4351.208923 secs 2024-07-26T08:08:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4368.951064 secs 2024-07-26T08:08:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:41 smithi164 ceph-mon[36970]: pgmap v2900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4351.208923 secs 2024-07-26T08:08:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4368.951064 secs 2024-07-26T08:08:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:41 smithi163 ceph-mon[36873]: pgmap v2900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4351.208923 secs 2024-07-26T08:08:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4368.951064 secs 2024-07-26T08:08:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:43 smithi042 ceph-mon[32775]: pgmap v2901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:43 smithi164 ceph-mon[36970]: pgmap v2901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:43 smithi163 ceph-mon[36873]: pgmap v2901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:46.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:45 smithi042 ceph-mon[32775]: pgmap v2902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:45 smithi164 ceph-mon[36970]: pgmap v2902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:45 smithi163 ceph-mon[36873]: pgmap v2902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4356.209067 secs 2024-07-26T08:08:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4373.951130 secs 2024-07-26T08:08:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4356.209067 secs 2024-07-26T08:08:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4373.951130 secs 2024-07-26T08:08:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4356.209067 secs 2024-07-26T08:08:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4373.951130 secs 2024-07-26T08:08:48.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:47 smithi042 ceph-mon[32775]: pgmap v2903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:47 smithi164 ceph-mon[36970]: pgmap v2903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:47 smithi163 ceph-mon[36873]: pgmap v2903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:50.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:49 smithi042 ceph-mon[32775]: pgmap v2904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:08:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:49 smithi164 ceph-mon[36970]: pgmap v2904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:08:50.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:49 smithi163 ceph-mon[36873]: pgmap v2904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:08:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:51 smithi042 ceph-mon[32775]: pgmap v2905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4361.209198 secs 2024-07-26T08:08:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4378.951258 secs 2024-07-26T08:08:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:51 smithi164 ceph-mon[36970]: pgmap v2905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4361.209198 secs 2024-07-26T08:08:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4378.951258 secs 2024-07-26T08:08:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:51 smithi163 ceph-mon[36873]: pgmap v2905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4361.209198 secs 2024-07-26T08:08:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4378.951258 secs 2024-07-26T08:08:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:53 smithi042 ceph-mon[32775]: pgmap v2906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:54.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:53 smithi164 ceph-mon[36970]: pgmap v2906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:53 smithi163 ceph-mon[36873]: pgmap v2906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:54.744 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:08:54.743+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:08:54.745 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:08:54.743+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:08:54.745 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:08:54.743+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:08:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:55 smithi042 ceph-mon[32775]: pgmap v2907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:55 smithi164 ceph-mon[36970]: pgmap v2907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:56.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:55 smithi163 ceph-mon[36873]: pgmap v2907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:08:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4366.209313 secs 2024-07-26T08:08:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4383.951378 secs 2024-07-26T08:08:57.158 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4366.209313 secs 2024-07-26T08:08:57.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4383.951378 secs 2024-07-26T08:08:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4366.209313 secs 2024-07-26T08:08:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4383.951378 secs 2024-07-26T08:08:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:57 smithi042 ceph-mon[32775]: pgmap v2908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:57 smithi164 ceph-mon[36970]: pgmap v2908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:08:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:57 smithi163 ceph-mon[36873]: pgmap v2908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:08:59 smithi042 ceph-mon[32775]: pgmap v2909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:09:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:08:59 smithi164 ceph-mon[36970]: pgmap v2909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:09:00.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:08:59 smithi163 ceph-mon[36873]: pgmap v2909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:09:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:01 smithi042 ceph-mon[32775]: pgmap v2910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4371.209433 secs 2024-07-26T08:09:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4388.951447 secs 2024-07-26T08:09:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:01 smithi164 ceph-mon[36970]: pgmap v2910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4371.209433 secs 2024-07-26T08:09:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4388.951447 secs 2024-07-26T08:09:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:01 smithi163 ceph-mon[36873]: pgmap v2910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4371.209433 secs 2024-07-26T08:09:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4388.951447 secs 2024-07-26T08:09:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:03 smithi042 ceph-mon[32775]: pgmap v2911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:03 smithi164 ceph-mon[36970]: pgmap v2911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:03 smithi163 ceph-mon[36873]: pgmap v2911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:05 smithi042 ceph-mon[32775]: pgmap v2912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:06.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:05 smithi164 ceph-mon[36970]: pgmap v2912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:05 smithi163 ceph-mon[36873]: pgmap v2912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4376.209561 secs 2024-07-26T08:09:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4393.951571 secs 2024-07-26T08:09:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4376.209561 secs 2024-07-26T08:09:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4393.951571 secs 2024-07-26T08:09:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4376.209561 secs 2024-07-26T08:09:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4393.951571 secs 2024-07-26T08:09:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:07 smithi042 ceph-mon[32775]: pgmap v2913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:07 smithi164 ceph-mon[36970]: pgmap v2913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:07 smithi163 ceph-mon[36873]: pgmap v2913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:09.753 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:09:09.752+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:09:09.753 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:09:09.752+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:09:09.753 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:09:09.752+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:09:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:09 smithi042 ceph-mon[32775]: pgmap v2914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:09:10.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:09 smithi164 ceph-mon[36970]: pgmap v2914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:09:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:09 smithi163 ceph-mon[36873]: pgmap v2914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:09:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:11 smithi042 ceph-mon[32775]: pgmap v2915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4381.209688 secs 2024-07-26T08:09:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4398.951694 secs 2024-07-26T08:09:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:11 smithi164 ceph-mon[36970]: pgmap v2915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4381.209688 secs 2024-07-26T08:09:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4398.951694 secs 2024-07-26T08:09:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:11 smithi163 ceph-mon[36873]: pgmap v2915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4381.209688 secs 2024-07-26T08:09:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4398.951694 secs 2024-07-26T08:09:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:13 smithi042 ceph-mon[32775]: pgmap v2916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:14.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:13 smithi164 ceph-mon[36970]: pgmap v2916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:13 smithi163 ceph-mon[36873]: pgmap v2916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:15 smithi042 ceph-mon[32775]: pgmap v2917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:15 smithi164 ceph-mon[36970]: pgmap v2917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:15 smithi163 ceph-mon[36873]: pgmap v2917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4386.209808 secs 2024-07-26T08:09:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4403.951830 secs 2024-07-26T08:09:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4386.209808 secs 2024-07-26T08:09:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4403.951830 secs 2024-07-26T08:09:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4386.209808 secs 2024-07-26T08:09:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4403.951830 secs 2024-07-26T08:09:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:17 smithi042 ceph-mon[32775]: pgmap v2918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:18.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:17 smithi164 ceph-mon[36970]: pgmap v2918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:17 smithi163 ceph-mon[36873]: pgmap v2918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:19 smithi042 ceph-mon[32775]: pgmap v2919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:09:20.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:19 smithi164 ceph-mon[36970]: pgmap v2919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:09:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:19 smithi163 ceph-mon[36873]: pgmap v2919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:09:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:21 smithi042 ceph-mon[32775]: pgmap v2920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4391.209931 secs 2024-07-26T08:09:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4408.951958 secs 2024-07-26T08:09:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:21 smithi164 ceph-mon[36970]: pgmap v2920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4391.209931 secs 2024-07-26T08:09:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4408.951958 secs 2024-07-26T08:09:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:21 smithi163 ceph-mon[36873]: pgmap v2920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4391.209931 secs 2024-07-26T08:09:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4408.951958 secs 2024-07-26T08:09:24.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:23 smithi042 ceph-mon[32775]: pgmap v2921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:23 smithi164 ceph-mon[36970]: pgmap v2921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:23 smithi163 ceph-mon[36873]: pgmap v2921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:09:24.756+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:09:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:09:24.756+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:09:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:09:24.756+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:09:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:09:25.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:09:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:09:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:25 smithi042 ceph-mon[32775]: pgmap v2922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:09:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:09:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:09:26.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:25 smithi164 ceph-mon[36970]: pgmap v2922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:09:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:09:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:09:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:25 smithi163 ceph-mon[36873]: pgmap v2922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:09:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:09:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:09:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:26 smithi042 ceph-mon[32775]: pgmap v2923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:09:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:26 smithi042 ceph-mon[32775]: pgmap v2924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:09:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4396.210058 secs 2024-07-26T08:09:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4413.952011 secs 2024-07-26T08:09:27.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:26 smithi163 ceph-mon[36873]: pgmap v2923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:09:27.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:26 smithi163 ceph-mon[36873]: pgmap v2924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:09:27.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4396.210058 secs 2024-07-26T08:09:27.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4413.952011 secs 2024-07-26T08:09:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:26 smithi164 ceph-mon[36970]: pgmap v2923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:09:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:26 smithi164 ceph-mon[36970]: pgmap v2924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:09:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4396.210058 secs 2024-07-26T08:09:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4413.952011 secs 2024-07-26T08:09:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:28 smithi042 ceph-mon[32775]: pgmap v2925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:09:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:28 smithi164 ceph-mon[36970]: pgmap v2925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:09:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:28 smithi163 ceph-mon[36873]: pgmap v2925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:09:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:30 smithi042 ceph-mon[32775]: pgmap v2926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:09:31.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:30 smithi164 ceph-mon[36970]: pgmap v2926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:09:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:30 smithi163 ceph-mon[36873]: pgmap v2926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:09:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4401.210197 secs 2024-07-26T08:09:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4418.952126 secs 2024-07-26T08:09:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4401.210197 secs 2024-07-26T08:09:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4418.952126 secs 2024-07-26T08:09:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4401.210197 secs 2024-07-26T08:09:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4418.952126 secs 2024-07-26T08:09:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:32 smithi042 ceph-mon[32775]: pgmap v2927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:09:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:32 smithi164 ceph-mon[36970]: pgmap v2927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:09:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:32 smithi163 ceph-mon[36873]: pgmap v2927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:09:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:34 smithi164 ceph-mon[36970]: pgmap v2928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:09:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:34 smithi163 ceph-mon[36873]: pgmap v2928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:09:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:34 smithi042 ceph-mon[32775]: pgmap v2928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:09:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:36 smithi164 ceph-mon[36970]: pgmap v2929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:09:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4406.210312 secs 2024-07-26T08:09:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4423.952245 secs 2024-07-26T08:09:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:36 smithi163 ceph-mon[36873]: pgmap v2929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:09:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4406.210312 secs 2024-07-26T08:09:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4423.952245 secs 2024-07-26T08:09:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:36 smithi042 ceph-mon[32775]: pgmap v2929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:09:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4406.210312 secs 2024-07-26T08:09:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4423.952245 secs 2024-07-26T08:09:39.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:38 smithi164 ceph-mon[36970]: pgmap v2930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:38 smithi163 ceph-mon[36873]: pgmap v2930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:38 smithi042 ceph-mon[32775]: pgmap v2930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:09:39.765+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:09:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:09:39.765+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:09:39.766 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:09:39.765+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:09:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:40 smithi164 ceph-mon[36970]: pgmap v2931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:09:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:40 smithi163 ceph-mon[36873]: pgmap v2931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:09:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:40 smithi042 ceph-mon[32775]: pgmap v2931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:09:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4411.210462 secs 2024-07-26T08:09:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4428.952346 secs 2024-07-26T08:09:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4411.210462 secs 2024-07-26T08:09:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4428.952346 secs 2024-07-26T08:09:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4411.210462 secs 2024-07-26T08:09:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4428.952346 secs 2024-07-26T08:09:43.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:42 smithi164 ceph-mon[36970]: pgmap v2932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:42 smithi163 ceph-mon[36873]: pgmap v2932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:42 smithi042 ceph-mon[32775]: pgmap v2932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:43 smithi164 ceph-mon[36970]: pgmap v2933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:43 smithi163 ceph-mon[36873]: pgmap v2933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:43 smithi042 ceph-mon[32775]: pgmap v2933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:46 smithi042 ceph-mon[32775]: pgmap v2934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:46 smithi164 ceph-mon[36970]: pgmap v2934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:46 smithi163 ceph-mon[36873]: pgmap v2934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4416.210587 secs 2024-07-26T08:09:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4433.952469 secs 2024-07-26T08:09:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4416.210587 secs 2024-07-26T08:09:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4433.952469 secs 2024-07-26T08:09:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4416.210587 secs 2024-07-26T08:09:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4433.952469 secs 2024-07-26T08:09:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:48 smithi042 ceph-mon[32775]: pgmap v2935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:48.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:48 smithi164 ceph-mon[36970]: pgmap v2935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:48 smithi163 ceph-mon[36873]: pgmap v2935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:50 smithi042 ceph-mon[32775]: pgmap v2936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:09:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:50 smithi164 ceph-mon[36970]: pgmap v2936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:09:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:50 smithi163 ceph-mon[36873]: pgmap v2936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:09:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4421.210730 secs 2024-07-26T08:09:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:52 smithi164 ceph-mon[36970]: pgmap v2937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4438.952598 secs 2024-07-26T08:09:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4421.210730 secs 2024-07-26T08:09:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:52 smithi163 ceph-mon[36873]: pgmap v2937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4438.952598 secs 2024-07-26T08:09:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4421.210730 secs 2024-07-26T08:09:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:52 smithi042 ceph-mon[32775]: pgmap v2937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4438.952598 secs 2024-07-26T08:09:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:54 smithi164 ceph-mon[36970]: pgmap v2938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:54 smithi163 ceph-mon[36873]: pgmap v2938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:54 smithi042 ceph-mon[32775]: pgmap v2938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:54.769 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:09:54.768+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:09:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:09:54.768+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:09:54.770 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:09:54.768+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:09:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:56 smithi164 ceph-mon[36970]: pgmap v2939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:56 smithi163 ceph-mon[36873]: pgmap v2939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:56 smithi042 ceph-mon[32775]: pgmap v2939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:09:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4426.210851 secs 2024-07-26T08:09:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4443.952751 secs 2024-07-26T08:09:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4426.210851 secs 2024-07-26T08:09:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4443.952751 secs 2024-07-26T08:09:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4426.210851 secs 2024-07-26T08:09:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4443.952751 secs 2024-07-26T08:09:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:09:58 smithi164 ceph-mon[36970]: pgmap v2940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:09:58 smithi163 ceph-mon[36873]: pgmap v2940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:09:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:09:58 smithi042 ceph-mon[32775]: pgmap v2940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:00 smithi164 ceph-mon[36970]: pgmap v2941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:10:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:10:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:00 smithi163 ceph-mon[36873]: pgmap v2941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:10:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:10:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:00 smithi042 ceph-mon[32775]: pgmap v2941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:10:00.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:10:02.447 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4431.210950 secs 2024-07-26T08:10:02.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:02 smithi042 ceph-mon[32775]: pgmap v2942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:02.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4448.952897 secs 2024-07-26T08:10:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4431.210950 secs 2024-07-26T08:10:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:02 smithi164 ceph-mon[36970]: pgmap v2942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4448.952897 secs 2024-07-26T08:10:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4431.210950 secs 2024-07-26T08:10:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:02 smithi163 ceph-mon[36873]: pgmap v2942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4448.952897 secs 2024-07-26T08:10:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:04 smithi164 ceph-mon[36970]: pgmap v2943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:04.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:04 smithi163 ceph-mon[36873]: pgmap v2943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:04.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:04 smithi042 ceph-mon[32775]: pgmap v2943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:06 smithi164 ceph-mon[36970]: pgmap v2944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:06 smithi163 ceph-mon[36873]: pgmap v2944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:06 smithi042 ceph-mon[32775]: pgmap v2944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:07.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4436.211026 secs 2024-07-26T08:10:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4453.953017 secs 2024-07-26T08:10:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4436.211026 secs 2024-07-26T08:10:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4453.953017 secs 2024-07-26T08:10:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4436.211026 secs 2024-07-26T08:10:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4453.953017 secs 2024-07-26T08:10:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:08 smithi164 ceph-mon[36970]: pgmap v2945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:08 smithi163 ceph-mon[36873]: pgmap v2945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:08.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:08 smithi042 ceph-mon[32775]: pgmap v2945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:09.777+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:10:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:09.777+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:10:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:09.777+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:10:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:10 smithi164 ceph-mon[36970]: pgmap v2946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:10:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:10 smithi163 ceph-mon[36873]: pgmap v2946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:10:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:10 smithi042 ceph-mon[32775]: pgmap v2946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:10:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4441.211146 secs 2024-07-26T08:10:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:12 smithi164 ceph-mon[36970]: pgmap v2947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4458.953144 secs 2024-07-26T08:10:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4441.211146 secs 2024-07-26T08:10:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:12 smithi163 ceph-mon[36873]: pgmap v2947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4458.953144 secs 2024-07-26T08:10:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4441.211146 secs 2024-07-26T08:10:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:12 smithi042 ceph-mon[32775]: pgmap v2947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4458.953144 secs 2024-07-26T08:10:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:13.564+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:10:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:13.564+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:10:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:13.564+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:10:13.565 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:13.564+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:10:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:14 smithi164 ceph-mon[36970]: pgmap v2948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:14.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:14 smithi163 ceph-mon[36873]: pgmap v2948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:14 smithi042 ceph-mon[32775]: pgmap v2948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:16 smithi164 ceph-mon[36970]: pgmap v2949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:16 smithi163 ceph-mon[36873]: pgmap v2949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:16 smithi042 ceph-mon[32775]: pgmap v2949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4446.211274 secs 2024-07-26T08:10:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4463.953267 secs 2024-07-26T08:10:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4446.211274 secs 2024-07-26T08:10:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4463.953267 secs 2024-07-26T08:10:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4446.211274 secs 2024-07-26T08:10:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4463.953267 secs 2024-07-26T08:10:18.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:18 smithi164 ceph-mon[36970]: pgmap v2950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:18.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:18 smithi163 ceph-mon[36873]: pgmap v2950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:18 smithi042 ceph-mon[32775]: pgmap v2950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:20 smithi164 ceph-mon[36970]: pgmap v2951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:20 smithi163 ceph-mon[36873]: pgmap v2951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:20 smithi042 ceph-mon[32775]: pgmap v2951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4451.211398 secs 2024-07-26T08:10:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:22 smithi164 ceph-mon[36970]: pgmap v2952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4468.953334 secs 2024-07-26T08:10:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4451.211398 secs 2024-07-26T08:10:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:22 smithi163 ceph-mon[36873]: pgmap v2952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4468.953334 secs 2024-07-26T08:10:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4451.211398 secs 2024-07-26T08:10:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:22 smithi042 ceph-mon[32775]: pgmap v2952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4468.953334 secs 2024-07-26T08:10:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:24 smithi164 ceph-mon[36970]: pgmap v2953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:24 smithi163 ceph-mon[36873]: pgmap v2953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:24 smithi042 ceph-mon[32775]: pgmap v2953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:24.781+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:10:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:24.781+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:10:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:24.781+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:10:25.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:10:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:10:25.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:10:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:26 smithi164 ceph-mon[36970]: pgmap v2954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:26 smithi163 ceph-mon[36873]: pgmap v2954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:26 smithi042 ceph-mon[32775]: pgmap v2954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4456.211487 secs 2024-07-26T08:10:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4473.953417 secs 2024-07-26T08:10:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4456.211487 secs 2024-07-26T08:10:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4473.953417 secs 2024-07-26T08:10:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4456.211487 secs 2024-07-26T08:10:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4473.953417 secs 2024-07-26T08:10:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:28 smithi163 ceph-mon[36873]: pgmap v2955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:10:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:10:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:28 smithi163 ceph-mon[36873]: pgmap v2956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:10:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:28 smithi163 ceph-mon[36873]: pgmap v2957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:10:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:28 smithi042 ceph-mon[32775]: pgmap v2955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:10:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:10:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:28 smithi042 ceph-mon[32775]: pgmap v2956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:10:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:28 smithi042 ceph-mon[32775]: pgmap v2957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:10:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:28 smithi164 ceph-mon[36970]: pgmap v2955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:10:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:10:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:28 smithi164 ceph-mon[36970]: pgmap v2956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:10:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:28 smithi164 ceph-mon[36970]: pgmap v2957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:10:28.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:10:30.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:30 smithi164 ceph-mon[36970]: pgmap v2958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:10:31.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:30 smithi163 ceph-mon[36873]: pgmap v2958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:10:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:30 smithi042 ceph-mon[32775]: pgmap v2958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:10:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4461.211616 secs 2024-07-26T08:10:32.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4461.211616 secs 2024-07-26T08:10:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4461.211616 secs 2024-07-26T08:10:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4478.953528 secs 2024-07-26T08:10:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:32 smithi163 ceph-mon[36873]: pgmap v2959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:10:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4478.953528 secs 2024-07-26T08:10:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:32 smithi042 ceph-mon[32775]: pgmap v2959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:10:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4478.953528 secs 2024-07-26T08:10:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:32 smithi164 ceph-mon[36970]: pgmap v2959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:10:35.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:34 smithi163 ceph-mon[36873]: pgmap v2960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:10:35.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:34 smithi042 ceph-mon[32775]: pgmap v2960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:10:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:34 smithi164 ceph-mon[36970]: pgmap v2960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:10:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:36 smithi042 ceph-mon[32775]: pgmap v2961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:10:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4466.211757 secs 2024-07-26T08:10:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:36 smithi164 ceph-mon[36970]: pgmap v2961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:10:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4466.211757 secs 2024-07-26T08:10:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:36 smithi163 ceph-mon[36873]: pgmap v2961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:10:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4466.211757 secs 2024-07-26T08:10:37.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4483.953646 secs 2024-07-26T08:10:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4483.953646 secs 2024-07-26T08:10:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4483.953646 secs 2024-07-26T08:10:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:38 smithi042 ceph-mon[32775]: pgmap v2962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:10:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:38 smithi164 ceph-mon[36970]: pgmap v2962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:10:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:38 smithi163 ceph-mon[36873]: pgmap v2962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:10:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:39.788+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:10:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:39.788+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:10:39.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:39.788+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:10:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:40 smithi164 ceph-mon[36970]: pgmap v2963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:40 smithi163 ceph-mon[36873]: pgmap v2963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:40 smithi042 ceph-mon[32775]: pgmap v2963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4471.211829 secs 2024-07-26T08:10:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4471.211829 secs 2024-07-26T08:10:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4471.211829 secs 2024-07-26T08:10:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4488.953785 secs 2024-07-26T08:10:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:42 smithi163 ceph-mon[36873]: pgmap v2964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4488.953785 secs 2024-07-26T08:10:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:42 smithi042 ceph-mon[32775]: pgmap v2964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:43.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4488.953785 secs 2024-07-26T08:10:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:42 smithi164 ceph-mon[36970]: pgmap v2964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:44 smithi163 ceph-mon[36873]: pgmap v2965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:10:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:44 smithi042 ceph-mon[32775]: pgmap v2965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:10:45.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:44 smithi164 ceph-mon[36970]: pgmap v2965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:10:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:46 smithi163 ceph-mon[36873]: pgmap v2966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4476.211971 secs 2024-07-26T08:10:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4493.953913 secs 2024-07-26T08:10:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:46 smithi042 ceph-mon[32775]: pgmap v2966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4476.211971 secs 2024-07-26T08:10:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4493.953913 secs 2024-07-26T08:10:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:46 smithi164 ceph-mon[36970]: pgmap v2966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4476.211971 secs 2024-07-26T08:10:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4493.953913 secs 2024-07-26T08:10:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:48 smithi164 ceph-mon[36970]: pgmap v2967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:48 smithi163 ceph-mon[36873]: pgmap v2967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:48 smithi042 ceph-mon[32775]: pgmap v2967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:50 smithi163 ceph-mon[36873]: pgmap v2968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:50 smithi042 ceph-mon[32775]: pgmap v2968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:50 smithi164 ceph-mon[36970]: pgmap v2968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4481.212097 secs 2024-07-26T08:10:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4498.954012 secs 2024-07-26T08:10:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4481.212097 secs 2024-07-26T08:10:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4498.954012 secs 2024-07-26T08:10:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4481.212097 secs 2024-07-26T08:10:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4498.954012 secs 2024-07-26T08:10:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:52 smithi163 ceph-mon[36873]: pgmap v2969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:52 smithi042 ceph-mon[32775]: pgmap v2969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:53.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:52 smithi164 ceph-mon[36970]: pgmap v2969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:54.792+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:10:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:54.792+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:10:54.793 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:54.792+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:10:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:54 smithi163 ceph-mon[36873]: pgmap v2970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:10:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:54 smithi042 ceph-mon[32775]: pgmap v2970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:10:55.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:54 smithi164 ceph-mon[36970]: pgmap v2970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:10:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:56 smithi042 ceph-mon[32775]: pgmap v2971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4486.212231 secs 2024-07-26T08:10:56.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:56 smithi164 ceph-mon[36970]: pgmap v2971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4486.212231 secs 2024-07-26T08:10:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:56 smithi163 ceph-mon[36873]: pgmap v2971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:10:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4486.212231 secs 2024-07-26T08:10:57.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:57.867+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0b45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T08:10:57.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:57.867+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0b45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T08:10:57.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:57.867+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b45c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff678031980 tx=0x7ff67803e040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:10:57.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:57.867+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b45c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff678031980 tx=0x7ff67803e040 comp rx=0 tx=0).stop 2024-07-26T08:10:57.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:57.867+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0b45c0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T08:10:57.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:57.867+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0b45c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T08:10:57.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:57.867+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b45c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:10:57.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:57.867+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:10:57.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:57.867+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:10:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:57.868+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff67802b0b0 tx=0x7ff67804b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:10:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4503.954112 secs 2024-07-26T08:10:57.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4503.954112 secs 2024-07-26T08:10:57.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4503.954112 secs 2024-07-26T08:10:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:58.570+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:10:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:58.570+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:10:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:58.571+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:10:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:10:58.571+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:10:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:10:58 smithi042 ceph-mon[32775]: pgmap v2972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:10:58 smithi164 ceph-mon[36970]: pgmap v2972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:10:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:10:58 smithi163 ceph-mon[36873]: pgmap v2972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:00 smithi163 ceph-mon[36873]: pgmap v2973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:00 smithi042 ceph-mon[32775]: pgmap v2973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:00 smithi164 ceph-mon[36970]: pgmap v2973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4491.212338 secs 2024-07-26T08:11:02.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4491.212338 secs 2024-07-26T08:11:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4491.212338 secs 2024-07-26T08:11:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4508.954232 secs 2024-07-26T08:11:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:02 smithi163 ceph-mon[36873]: pgmap v2974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4508.954232 secs 2024-07-26T08:11:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:02 smithi042 ceph-mon[32775]: pgmap v2974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4508.954232 secs 2024-07-26T08:11:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:02 smithi164 ceph-mon[36970]: pgmap v2974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:05.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:04 smithi163 ceph-mon[36873]: pgmap v2975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:11:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:04 smithi042 ceph-mon[32775]: pgmap v2975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:11:05.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:04 smithi164 ceph-mon[36970]: pgmap v2975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:11:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:06 smithi163 ceph-mon[36873]: pgmap v2976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4496.212469 secs 2024-07-26T08:11:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4513.954295 secs 2024-07-26T08:11:07.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:06 smithi164 ceph-mon[36970]: pgmap v2976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:07.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4496.212469 secs 2024-07-26T08:11:07.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4513.954295 secs 2024-07-26T08:11:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:06 smithi042 ceph-mon[32775]: pgmap v2976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4496.212469 secs 2024-07-26T08:11:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4513.954295 secs 2024-07-26T08:11:09.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:08 smithi163 ceph-mon[36873]: pgmap v2977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:08 smithi042 ceph-mon[32775]: pgmap v2977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:08 smithi164 ceph-mon[36970]: pgmap v2977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:09.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:11:09.798+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:11:09.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:11:09.798+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:11:09.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:11:09.798+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:11:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:10 smithi163 ceph-mon[36873]: pgmap v2978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:10 smithi042 ceph-mon[32775]: pgmap v2978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:10 smithi164 ceph-mon[36970]: pgmap v2978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4501.212555 secs 2024-07-26T08:11:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4518.954422 secs 2024-07-26T08:11:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4501.212555 secs 2024-07-26T08:11:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4518.954422 secs 2024-07-26T08:11:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4501.212555 secs 2024-07-26T08:11:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4518.954422 secs 2024-07-26T08:11:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:12 smithi163 ceph-mon[36873]: pgmap v2979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:12 smithi042 ceph-mon[32775]: pgmap v2979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:12 smithi164 ceph-mon[36970]: pgmap v2979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:15.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:14 smithi163 ceph-mon[36873]: pgmap v2980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:11:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:14 smithi042 ceph-mon[32775]: pgmap v2980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:11:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:14 smithi164 ceph-mon[36970]: pgmap v2980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:11:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:16 smithi042 ceph-mon[32775]: pgmap v2981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4506.212695 secs 2024-07-26T08:11:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4523.954569 secs 2024-07-26T08:11:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:16 smithi164 ceph-mon[36970]: pgmap v2981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4506.212695 secs 2024-07-26T08:11:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4523.954569 secs 2024-07-26T08:11:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:16 smithi163 ceph-mon[36873]: pgmap v2981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4506.212695 secs 2024-07-26T08:11:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4523.954569 secs 2024-07-26T08:11:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:18 smithi042 ceph-mon[32775]: pgmap v2982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:19.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:18 smithi164 ceph-mon[36970]: pgmap v2982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:18 smithi163 ceph-mon[36873]: pgmap v2982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:20 smithi042 ceph-mon[32775]: pgmap v2983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:20 smithi164 ceph-mon[36970]: pgmap v2983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:20 smithi163 ceph-mon[36873]: pgmap v2983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4511.212838 secs 2024-07-26T08:11:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4528.954667 secs 2024-07-26T08:11:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4511.212838 secs 2024-07-26T08:11:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4528.954667 secs 2024-07-26T08:11:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4511.212838 secs 2024-07-26T08:11:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4528.954667 secs 2024-07-26T08:11:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:22 smithi042 ceph-mon[32775]: pgmap v2984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:23.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:22 smithi164 ceph-mon[36970]: pgmap v2984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:22 smithi163 ceph-mon[36873]: pgmap v2984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:24.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:11:24.802+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:11:24.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:11:24.802+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:11:24.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:11:24.802+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:11:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:24 smithi042 ceph-mon[32775]: pgmap v2985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:11:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:24 smithi164 ceph-mon[36970]: pgmap v2985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:11:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:24 smithi163 ceph-mon[36873]: pgmap v2985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:11:27.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:26 smithi163 ceph-mon[36873]: pgmap v2986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:27.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4516.212981 secs 2024-07-26T08:11:27.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4533.954805 secs 2024-07-26T08:11:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:26 smithi042 ceph-mon[32775]: pgmap v2986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4516.212981 secs 2024-07-26T08:11:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4533.954805 secs 2024-07-26T08:11:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:26 smithi164 ceph-mon[36970]: pgmap v2986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4516.212981 secs 2024-07-26T08:11:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4533.954805 secs 2024-07-26T08:11:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:11:28.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:11:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:11:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:28 smithi042 ceph-mon[32775]: pgmap v2987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:11:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:11:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:28 smithi164 ceph-mon[36970]: pgmap v2987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:11:29.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:11:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:28 smithi163 ceph-mon[36873]: pgmap v2987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:11:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:11:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:29 smithi042 ceph-mon[32775]: pgmap v2988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T08:11:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:29 smithi042 ceph-mon[32775]: pgmap v2989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T08:11:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:29 smithi164 ceph-mon[36970]: pgmap v2988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T08:11:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:29 smithi164 ceph-mon[36970]: pgmap v2989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T08:11:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:29 smithi163 ceph-mon[36873]: pgmap v2988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T08:11:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:29 smithi163 ceph-mon[36873]: pgmap v2989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T08:11:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:11:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4521.213107 secs 2024-07-26T08:11:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:31 smithi042 ceph-mon[32775]: pgmap v2990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T08:11:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4521.213107 secs 2024-07-26T08:11:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:31 smithi164 ceph-mon[36970]: pgmap v2990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T08:11:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4521.213107 secs 2024-07-26T08:11:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:31 smithi163 ceph-mon[36873]: pgmap v2990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T08:11:32.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4538.954947 secs 2024-07-26T08:11:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4538.954947 secs 2024-07-26T08:11:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4538.954947 secs 2024-07-26T08:11:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:33 smithi042 ceph-mon[32775]: pgmap v2991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T08:11:33.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:33 smithi164 ceph-mon[36970]: pgmap v2991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T08:11:34.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:33 smithi163 ceph-mon[36873]: pgmap v2991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T08:11:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:35 smithi042 ceph-mon[32775]: pgmap v2992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T08:11:36.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:35 smithi164 ceph-mon[36970]: pgmap v2992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T08:11:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:35 smithi163 ceph-mon[36873]: pgmap v2992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T08:11:37.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4526.213235 secs 2024-07-26T08:11:37.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4543.955066 secs 2024-07-26T08:11:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4526.213235 secs 2024-07-26T08:11:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4543.955066 secs 2024-07-26T08:11:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4526.213235 secs 2024-07-26T08:11:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4543.955066 secs 2024-07-26T08:11:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:37 smithi042 ceph-mon[32775]: pgmap v2993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T08:11:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:37 smithi164 ceph-mon[36970]: pgmap v2993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T08:11:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:37 smithi163 ceph-mon[36873]: pgmap v2993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T08:11:39.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:11:39.811+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:11:39.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:11:39.811+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:11:39.812 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:11:39.811+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:11:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:39 smithi042 ceph-mon[32775]: pgmap v2994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:11:40.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:39 smithi164 ceph-mon[36970]: pgmap v2994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:11:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:39 smithi163 ceph-mon[36873]: pgmap v2994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:11:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4531.213359 secs 2024-07-26T08:11:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:41 smithi042 ceph-mon[32775]: pgmap v2995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4548.955193 secs 2024-07-26T08:11:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4531.213359 secs 2024-07-26T08:11:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:41 smithi164 ceph-mon[36970]: pgmap v2995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4548.955193 secs 2024-07-26T08:11:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4531.213359 secs 2024-07-26T08:11:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:41 smithi163 ceph-mon[36873]: pgmap v2995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4548.955193 secs 2024-07-26T08:11:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:43 smithi042 ceph-mon[32775]: pgmap v2996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:44.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:43 smithi164 ceph-mon[36970]: pgmap v2996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:43 smithi163 ceph-mon[36873]: pgmap v2996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:46.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:45 smithi042 ceph-mon[32775]: pgmap v2997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:45 smithi164 ceph-mon[36970]: pgmap v2997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:45 smithi163 ceph-mon[36873]: pgmap v2997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4536.213475 secs 2024-07-26T08:11:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4553.955312 secs 2024-07-26T08:11:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4536.213475 secs 2024-07-26T08:11:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4553.955312 secs 2024-07-26T08:11:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4536.213475 secs 2024-07-26T08:11:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4553.955312 secs 2024-07-26T08:11:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:47 smithi042 ceph-mon[32775]: pgmap v2998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:47 smithi164 ceph-mon[36970]: pgmap v2998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:47 smithi163 ceph-mon[36873]: pgmap v2998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:50.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:49 smithi042 ceph-mon[32775]: pgmap v2999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:11:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:49 smithi164 ceph-mon[36970]: pgmap v2999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:11:50.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:49 smithi163 ceph-mon[36873]: pgmap v2999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:11:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4541.213539 secs 2024-07-26T08:11:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:51 smithi042 ceph-mon[32775]: pgmap v3000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4558.955434 secs 2024-07-26T08:11:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4541.213539 secs 2024-07-26T08:11:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:51 smithi164 ceph-mon[36970]: pgmap v3000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4558.955434 secs 2024-07-26T08:11:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4541.213539 secs 2024-07-26T08:11:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:51 smithi163 ceph-mon[36873]: pgmap v3000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4558.955434 secs 2024-07-26T08:11:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:53 smithi042 ceph-mon[32775]: pgmap v3001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:54.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:53 smithi164 ceph-mon[36970]: pgmap v3001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:53 smithi163 ceph-mon[36873]: pgmap v3001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:11:54.814+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:11:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:11:54.815+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:11:54.816 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:11:54.815+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:11:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:55 smithi042 ceph-mon[32775]: pgmap v3002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:56.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:55 smithi164 ceph-mon[36970]: pgmap v3002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:56.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:55 smithi163 ceph-mon[36873]: pgmap v3002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:11:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4546.213657 secs 2024-07-26T08:11:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4563.955560 secs 2024-07-26T08:11:57.158 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4546.213657 secs 2024-07-26T08:11:57.158 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4563.955560 secs 2024-07-26T08:11:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4546.213657 secs 2024-07-26T08:11:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4563.955560 secs 2024-07-26T08:11:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:57 smithi164 ceph-mon[36970]: pgmap v3003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:57 smithi163 ceph-mon[36873]: pgmap v3003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:11:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:57 smithi042 ceph-mon[32775]: pgmap v3003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:11:59 smithi164 ceph-mon[36970]: pgmap v3004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:00.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:11:59 smithi163 ceph-mon[36873]: pgmap v3004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:11:59 smithi042 ceph-mon[32775]: pgmap v3004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:02.197 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4551.213780 secs 2024-07-26T08:12:02.197 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:01 smithi042 ceph-mon[32775]: pgmap v3005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:02.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4568.955705 secs 2024-07-26T08:12:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4551.213780 secs 2024-07-26T08:12:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:01 smithi164 ceph-mon[36970]: pgmap v3005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4568.955705 secs 2024-07-26T08:12:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4551.213780 secs 2024-07-26T08:12:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:01 smithi163 ceph-mon[36873]: pgmap v3005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4568.955705 secs 2024-07-26T08:12:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:03 smithi164 ceph-mon[36970]: pgmap v3006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:03 smithi163 ceph-mon[36873]: pgmap v3006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:04.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:03 smithi042 ceph-mon[32775]: pgmap v3006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:05 smithi164 ceph-mon[36970]: pgmap v3007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:05 smithi163 ceph-mon[36873]: pgmap v3007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:05 smithi042 ceph-mon[32775]: pgmap v3007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4556.213913 secs 2024-07-26T08:12:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4573.955850 secs 2024-07-26T08:12:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4556.213913 secs 2024-07-26T08:12:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4573.955850 secs 2024-07-26T08:12:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4556.213913 secs 2024-07-26T08:12:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4573.955850 secs 2024-07-26T08:12:08.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:07 smithi164 ceph-mon[36970]: pgmap v3008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:07 smithi163 ceph-mon[36873]: pgmap v3008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:07 smithi042 ceph-mon[32775]: pgmap v3008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:09.821 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:12:09.820+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:12:09.821 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:12:09.820+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:12:09.821 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:12:09.820+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:12:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:09 smithi164 ceph-mon[36970]: pgmap v3009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:09 smithi163 ceph-mon[36873]: pgmap v3009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:09 smithi042 ceph-mon[32775]: pgmap v3009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4561.214018 secs 2024-07-26T08:12:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:11 smithi164 ceph-mon[36970]: pgmap v3010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4578.956009 secs 2024-07-26T08:12:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4561.214018 secs 2024-07-26T08:12:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:11 smithi163 ceph-mon[36873]: pgmap v3010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4578.956009 secs 2024-07-26T08:12:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4561.214018 secs 2024-07-26T08:12:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:11 smithi042 ceph-mon[32775]: pgmap v3010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4578.956009 secs 2024-07-26T08:12:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:13 smithi164 ceph-mon[36970]: pgmap v3011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:13 smithi163 ceph-mon[36873]: pgmap v3011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:13 smithi042 ceph-mon[32775]: pgmap v3011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:16.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:15 smithi164 ceph-mon[36970]: pgmap v3012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:15 smithi163 ceph-mon[36873]: pgmap v3012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:15 smithi042 ceph-mon[32775]: pgmap v3012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4566.214142 secs 2024-07-26T08:12:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4583.956077 secs 2024-07-26T08:12:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4566.214142 secs 2024-07-26T08:12:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4583.956077 secs 2024-07-26T08:12:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4566.214142 secs 2024-07-26T08:12:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4583.956077 secs 2024-07-26T08:12:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:17 smithi164 ceph-mon[36970]: pgmap v3013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:17 smithi163 ceph-mon[36873]: pgmap v3013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:17 smithi042 ceph-mon[32775]: pgmap v3013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:18 smithi164 ceph-mon[36970]: pgmap v3014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:18 smithi163 ceph-mon[36873]: pgmap v3014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:18 smithi042 ceph-mon[32775]: pgmap v3014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4571.214264 secs 2024-07-26T08:12:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:21 smithi042 ceph-mon[32775]: pgmap v3015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4588.956235 secs 2024-07-26T08:12:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4571.214264 secs 2024-07-26T08:12:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:21 smithi164 ceph-mon[36970]: pgmap v3015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4588.956235 secs 2024-07-26T08:12:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4571.214264 secs 2024-07-26T08:12:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:21 smithi163 ceph-mon[36873]: pgmap v3015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4588.956235 secs 2024-07-26T08:12:24.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:23 smithi042 ceph-mon[32775]: pgmap v3016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:23 smithi164 ceph-mon[36970]: pgmap v3016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:23 smithi163 ceph-mon[36873]: pgmap v3016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:12:24.823+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:12:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:12:24.823+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:12:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:12:24.823+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:12:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:25 smithi042 ceph-mon[32775]: pgmap v3017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:26.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:25 smithi164 ceph-mon[36970]: pgmap v3017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:25 smithi163 ceph-mon[36873]: pgmap v3017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4576.214389 secs 2024-07-26T08:12:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4593.956359 secs 2024-07-26T08:12:27.158 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4576.214389 secs 2024-07-26T08:12:27.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4593.956359 secs 2024-07-26T08:12:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4576.214389 secs 2024-07-26T08:12:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4593.956359 secs 2024-07-26T08:12:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:27 smithi042 ceph-mon[32775]: pgmap v3018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:27 smithi164 ceph-mon[36970]: pgmap v3018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:27 smithi163 ceph-mon[36873]: pgmap v3018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:29 smithi042 ceph-mon[32775]: pgmap v3019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:12:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:12:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:12:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:12:30.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:29 smithi164 ceph-mon[36970]: pgmap v3019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:12:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:12:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:12:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:12:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:29 smithi163 ceph-mon[36873]: pgmap v3019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:12:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:12:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:12:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:12:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:30 smithi042 ceph-mon[32775]: pgmap v3020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T08:12:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:30 smithi042 ceph-mon[32775]: pgmap v3021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:12:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:30 smithi164 ceph-mon[36970]: pgmap v3020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T08:12:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:30 smithi164 ceph-mon[36970]: pgmap v3021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:12:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:30 smithi163 ceph-mon[36873]: pgmap v3020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T08:12:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:30 smithi163 ceph-mon[36873]: pgmap v3021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:12:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4581.214510 secs 2024-07-26T08:12:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4598.956460 secs 2024-07-26T08:12:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4581.214510 secs 2024-07-26T08:12:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4598.956460 secs 2024-07-26T08:12:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4581.214510 secs 2024-07-26T08:12:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4598.956460 secs 2024-07-26T08:12:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:32 smithi042 ceph-mon[32775]: pgmap v3022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:12:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:32 smithi164 ceph-mon[36970]: pgmap v3022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:12:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:32 smithi163 ceph-mon[36873]: pgmap v3022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:12:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:34 smithi164 ceph-mon[36970]: pgmap v3023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:12:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:34 smithi163 ceph-mon[36873]: pgmap v3023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:12:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:34 smithi042 ceph-mon[32775]: pgmap v3023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:12:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:36 smithi164 ceph-mon[36970]: pgmap v3024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:12:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4586.214642 secs 2024-07-26T08:12:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4603.956589 secs 2024-07-26T08:12:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:36 smithi163 ceph-mon[36873]: pgmap v3024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:12:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4586.214642 secs 2024-07-26T08:12:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4603.956589 secs 2024-07-26T08:12:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:36 smithi042 ceph-mon[32775]: pgmap v3024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:12:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4586.214642 secs 2024-07-26T08:12:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4603.956589 secs 2024-07-26T08:12:39.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:38 smithi164 ceph-mon[36970]: pgmap v3025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:12:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:38 smithi163 ceph-mon[36873]: pgmap v3025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:12:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:38 smithi042 ceph-mon[32775]: pgmap v3025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:12:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:12:39.832+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:12:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:12:39.832+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:12:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:12:39.832+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:12:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:40 smithi042 ceph-mon[32775]: pgmap v3026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:12:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:40 smithi164 ceph-mon[36970]: pgmap v3026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:12:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:40 smithi163 ceph-mon[36873]: pgmap v3026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:12:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4591.214766 secs 2024-07-26T08:12:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4608.956689 secs 2024-07-26T08:12:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4591.214766 secs 2024-07-26T08:12:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4608.956689 secs 2024-07-26T08:12:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4591.214766 secs 2024-07-26T08:12:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4608.956689 secs 2024-07-26T08:12:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:42 smithi164 ceph-mon[36970]: pgmap v3027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:42 smithi163 ceph-mon[36873]: pgmap v3027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:42 smithi042 ceph-mon[32775]: pgmap v3027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:44 smithi164 ceph-mon[36970]: pgmap v3028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:44 smithi163 ceph-mon[36873]: pgmap v3028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:44 smithi042 ceph-mon[32775]: pgmap v3028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:46 smithi164 ceph-mon[36970]: pgmap v3029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4596.214909 secs 2024-07-26T08:12:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4613.956790 secs 2024-07-26T08:12:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:46 smithi163 ceph-mon[36873]: pgmap v3029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4596.214909 secs 2024-07-26T08:12:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4613.956790 secs 2024-07-26T08:12:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:46 smithi042 ceph-mon[32775]: pgmap v3029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4596.214909 secs 2024-07-26T08:12:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4613.956790 secs 2024-07-26T08:12:49.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:48 smithi164 ceph-mon[36970]: pgmap v3030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:48 smithi163 ceph-mon[36873]: pgmap v3030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:48 smithi042 ceph-mon[32775]: pgmap v3030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:50 smithi164 ceph-mon[36970]: pgmap v3031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:50 smithi163 ceph-mon[36873]: pgmap v3031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:50 smithi042 ceph-mon[32775]: pgmap v3031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:52.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4601.215031 secs 2024-07-26T08:12:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4618.956914 secs 2024-07-26T08:12:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4601.215031 secs 2024-07-26T08:12:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4618.956914 secs 2024-07-26T08:12:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4601.215031 secs 2024-07-26T08:12:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4618.956914 secs 2024-07-26T08:12:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:52 smithi164 ceph-mon[36970]: pgmap v3032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:52 smithi163 ceph-mon[36873]: pgmap v3032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:52 smithi042 ceph-mon[32775]: pgmap v3032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:54.837 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:12:54.836+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:12:54.837 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:12:54.836+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:12:54.837 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:12:54.836+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:12:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:54 smithi164 ceph-mon[36970]: pgmap v3033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:55.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:54 smithi163 ceph-mon[36873]: pgmap v3033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:55.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:54 smithi042 ceph-mon[32775]: pgmap v3033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:12:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:56 smithi164 ceph-mon[36970]: pgmap v3034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4606.215151 secs 2024-07-26T08:12:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4623.957041 secs 2024-07-26T08:12:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:56 smithi163 ceph-mon[36873]: pgmap v3034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4606.215151 secs 2024-07-26T08:12:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4623.957041 secs 2024-07-26T08:12:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:56 smithi042 ceph-mon[32775]: pgmap v3034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:12:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4606.215151 secs 2024-07-26T08:12:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4623.957041 secs 2024-07-26T08:12:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:58 smithi164 ceph-mon[36970]: pgmap v3035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:58 smithi163 ceph-mon[36873]: pgmap v3035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:12:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:58 smithi042 ceph-mon[32775]: pgmap v3035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:00.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:12:59 smithi164 ceph-mon[36970]: pgmap v3036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:00.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:12:59 smithi163 ceph-mon[36873]: pgmap v3036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:12:59 smithi042 ceph-mon[32775]: pgmap v3036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4611.215290 secs 2024-07-26T08:13:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4611.215290 secs 2024-07-26T08:13:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4611.215290 secs 2024-07-26T08:13:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4628.957169 secs 2024-07-26T08:13:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:02 smithi164 ceph-mon[36970]: pgmap v3037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4628.957169 secs 2024-07-26T08:13:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:02 smithi163 ceph-mon[36873]: pgmap v3037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:02.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4628.957169 secs 2024-07-26T08:13:02.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:02 smithi042 ceph-mon[32775]: pgmap v3037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:04 smithi164 ceph-mon[36970]: pgmap v3038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:13:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:04 smithi163 ceph-mon[36873]: pgmap v3038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:13:04.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:04 smithi042 ceph-mon[32775]: pgmap v3038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:13:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:06 smithi164 ceph-mon[36970]: pgmap v3039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4616.215418 secs 2024-07-26T08:13:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:06 smithi163 ceph-mon[36873]: pgmap v3039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4616.215418 secs 2024-07-26T08:13:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:06 smithi042 ceph-mon[32775]: pgmap v3039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4616.215418 secs 2024-07-26T08:13:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4633.957267 secs 2024-07-26T08:13:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4633.957267 secs 2024-07-26T08:13:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4633.957267 secs 2024-07-26T08:13:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:08 smithi164 ceph-mon[36970]: pgmap v3040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:08 smithi163 ceph-mon[36873]: pgmap v3040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:08.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:08 smithi042 ceph-mon[32775]: pgmap v3040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:13:09.845+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:13:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:13:09.845+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:13:09.846 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:13:09.845+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:13:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:10 smithi164 ceph-mon[36970]: pgmap v3041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:10 smithi163 ceph-mon[36873]: pgmap v3041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:10 smithi042 ceph-mon[32775]: pgmap v3041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4621.215503 secs 2024-07-26T08:13:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4621.215503 secs 2024-07-26T08:13:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4621.215503 secs 2024-07-26T08:13:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4638.957387 secs 2024-07-26T08:13:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:12 smithi164 ceph-mon[36970]: pgmap v3042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4638.957387 secs 2024-07-26T08:13:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:12 smithi163 ceph-mon[36873]: pgmap v3042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4638.957387 secs 2024-07-26T08:13:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:12 smithi042 ceph-mon[32775]: pgmap v3042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:14 smithi164 ceph-mon[36970]: pgmap v3043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:13:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:14 smithi163 ceph-mon[36873]: pgmap v3043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:13:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:14 smithi042 ceph-mon[32775]: pgmap v3043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:13:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:16 smithi164 ceph-mon[36970]: pgmap v3044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4626.215627 secs 2024-07-26T08:13:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:16 smithi163 ceph-mon[36873]: pgmap v3044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4626.215627 secs 2024-07-26T08:13:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:16 smithi042 ceph-mon[32775]: pgmap v3044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4626.215627 secs 2024-07-26T08:13:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4643.957507 secs 2024-07-26T08:13:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4643.957507 secs 2024-07-26T08:13:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4643.957507 secs 2024-07-26T08:13:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:18 smithi164 ceph-mon[36970]: pgmap v3045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:18 smithi163 ceph-mon[36873]: pgmap v3045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:18 smithi042 ceph-mon[32775]: pgmap v3045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:20 smithi164 ceph-mon[36970]: pgmap v3046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:20 smithi163 ceph-mon[36873]: pgmap v3046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:20 smithi042 ceph-mon[32775]: pgmap v3046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4631.215759 secs 2024-07-26T08:13:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4631.215759 secs 2024-07-26T08:13:21.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4631.215759 secs 2024-07-26T08:13:22.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4648.957606 secs 2024-07-26T08:13:22.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:22 smithi163 ceph-mon[36873]: pgmap v3047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4648.957606 secs 2024-07-26T08:13:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:22 smithi042 ceph-mon[32775]: pgmap v3047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:22.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4648.957606 secs 2024-07-26T08:13:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:22 smithi164 ceph-mon[36970]: pgmap v3047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:24 smithi163 ceph-mon[36873]: pgmap v3048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:13:24.849 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:13:24.848+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:13:24.849 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:13:24.848+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:13:24.849 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:13:24.848+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:13:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:24 smithi042 ceph-mon[32775]: pgmap v3048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:13:24.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:24 smithi164 ceph-mon[36970]: pgmap v3048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:13:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:26 smithi163 ceph-mon[36873]: pgmap v3049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4636.215881 secs 2024-07-26T08:13:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:26 smithi042 ceph-mon[32775]: pgmap v3049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4636.215881 secs 2024-07-26T08:13:26.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:26 smithi164 ceph-mon[36970]: pgmap v3049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4636.215881 secs 2024-07-26T08:13:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4653.957729 secs 2024-07-26T08:13:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4653.957729 secs 2024-07-26T08:13:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4653.957729 secs 2024-07-26T08:13:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:28 smithi163 ceph-mon[36873]: pgmap v3050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:28 smithi042 ceph-mon[32775]: pgmap v3050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:28 smithi164 ceph-mon[36970]: pgmap v3050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:13:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:13:29.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:13:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:30 smithi163 ceph-mon[36873]: pgmap v3051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:13:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:13:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:30 smithi163 ceph-mon[36873]: pgmap v3052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:13:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:30 smithi163 ceph-mon[36873]: pgmap v3053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:13:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:13:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:30 smithi042 ceph-mon[32775]: pgmap v3051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:13:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:13:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:30 smithi042 ceph-mon[32775]: pgmap v3052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:13:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:30 smithi042 ceph-mon[32775]: pgmap v3053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:13:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:13:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:30 smithi164 ceph-mon[36970]: pgmap v3051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:13:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:13:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:30 smithi164 ceph-mon[36970]: pgmap v3052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:13:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:30 smithi164 ceph-mon[36970]: pgmap v3053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:13:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:13:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4641.215999 secs 2024-07-26T08:13:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4641.215999 secs 2024-07-26T08:13:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4641.215999 secs 2024-07-26T08:13:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4658.957821 secs 2024-07-26T08:13:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:32 smithi163 ceph-mon[36873]: pgmap v3054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:13:32.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4658.957821 secs 2024-07-26T08:13:32.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:32 smithi042 ceph-mon[32775]: pgmap v3054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:13:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4658.957821 secs 2024-07-26T08:13:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:32 smithi164 ceph-mon[36970]: pgmap v3054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:13:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:34 smithi163 ceph-mon[36873]: pgmap v3055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:13:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:34 smithi042 ceph-mon[32775]: pgmap v3055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:13:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:34 smithi164 ceph-mon[36970]: pgmap v3055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:13:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:36 smithi163 ceph-mon[36873]: pgmap v3056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:13:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:36 smithi042 ceph-mon[32775]: pgmap v3056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:13:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:36 smithi164 ceph-mon[36970]: pgmap v3056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:13:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4646.216121 secs 2024-07-26T08:13:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4663.957952 secs 2024-07-26T08:13:37.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4646.216121 secs 2024-07-26T08:13:37.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4663.957952 secs 2024-07-26T08:13:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4646.216121 secs 2024-07-26T08:13:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4663.957952 secs 2024-07-26T08:13:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:38 smithi042 ceph-mon[32775]: pgmap v3057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:13:38.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:38 smithi164 ceph-mon[36970]: pgmap v3057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:13:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:38 smithi163 ceph-mon[36873]: pgmap v3057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:13:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:13:39.857+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:13:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:13:39.857+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:13:39.858 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:13:39.857+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:13:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:40 smithi163 ceph-mon[36873]: pgmap v3058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:13:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:40 smithi042 ceph-mon[32775]: pgmap v3058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:13:40.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:40 smithi164 ceph-mon[36970]: pgmap v3058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:13:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4651.216247 secs 2024-07-26T08:13:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4651.216247 secs 2024-07-26T08:13:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4651.216247 secs 2024-07-26T08:13:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4668.958073 secs 2024-07-26T08:13:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:42 smithi164 ceph-mon[36970]: pgmap v3059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4668.958073 secs 2024-07-26T08:13:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:42 smithi163 ceph-mon[36873]: pgmap v3059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4668.958073 secs 2024-07-26T08:13:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:42 smithi042 ceph-mon[32775]: pgmap v3059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:44 smithi163 ceph-mon[36873]: pgmap v3060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:13:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:44 smithi042 ceph-mon[32775]: pgmap v3060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:13:44.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:44 smithi164 ceph-mon[36970]: pgmap v3060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:13:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:46 smithi163 ceph-mon[36873]: pgmap v3061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:46 smithi042 ceph-mon[32775]: pgmap v3061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:46 smithi164 ceph-mon[36970]: pgmap v3061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4656.216332 secs 2024-07-26T08:13:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4673.958151 secs 2024-07-26T08:13:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4656.216332 secs 2024-07-26T08:13:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4673.958151 secs 2024-07-26T08:13:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4656.216332 secs 2024-07-26T08:13:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4673.958151 secs 2024-07-26T08:13:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:48 smithi042 ceph-mon[32775]: pgmap v3062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:48.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:48 smithi164 ceph-mon[36970]: pgmap v3062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:48 smithi163 ceph-mon[36873]: pgmap v3062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:50.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:50 smithi163 ceph-mon[36873]: pgmap v3063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:50 smithi042 ceph-mon[32775]: pgmap v3063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:50 smithi164 ceph-mon[36970]: pgmap v3063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4661.216462 secs 2024-07-26T08:13:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4661.216462 secs 2024-07-26T08:13:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4661.216462 secs 2024-07-26T08:13:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4678.958269 secs 2024-07-26T08:13:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:52 smithi164 ceph-mon[36970]: pgmap v3064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4678.958269 secs 2024-07-26T08:13:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:52 smithi163 ceph-mon[36873]: pgmap v3064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4678.958269 secs 2024-07-26T08:13:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:52 smithi042 ceph-mon[32775]: pgmap v3064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:54 smithi163 ceph-mon[36873]: pgmap v3065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:13:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:54 smithi042 ceph-mon[32775]: pgmap v3065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:13:54.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:54 smithi164 ceph-mon[36970]: pgmap v3065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:13:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:13:54.861+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:13:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:13:54.861+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:13:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:13:54.861+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:13:56.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:56 smithi163 ceph-mon[36873]: pgmap v3066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:56.399 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:56 smithi042 ceph-mon[32775]: pgmap v3066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:56 smithi164 ceph-mon[36970]: pgmap v3066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:13:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4666.216546 secs 2024-07-26T08:13:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4683.958396 secs 2024-07-26T08:13:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4666.216546 secs 2024-07-26T08:13:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4683.958396 secs 2024-07-26T08:13:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4666.216546 secs 2024-07-26T08:13:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4683.958396 secs 2024-07-26T08:13:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:13:58 smithi042 ceph-mon[32775]: pgmap v3067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:58.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:13:58 smithi164 ceph-mon[36970]: pgmap v3067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:13:58 smithi163 ceph-mon[36873]: pgmap v3067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:13:58.600 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:13:58.599+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:13:58.600 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:13:58.599+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:13:58.600 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:13:58.599+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:13:58.601 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:13:58.599+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:14:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:00 smithi042 ceph-mon[32775]: pgmap v3068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:00.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:00 smithi164 ceph-mon[36970]: pgmap v3068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:00 smithi163 ceph-mon[36873]: pgmap v3068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:01.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4671.216691 secs 2024-07-26T08:14:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4671.216691 secs 2024-07-26T08:14:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4671.216691 secs 2024-07-26T08:14:02.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4688.958524 secs 2024-07-26T08:14:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:02 smithi164 ceph-mon[36970]: pgmap v3069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4688.958524 secs 2024-07-26T08:14:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:02 smithi163 ceph-mon[36873]: pgmap v3069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:02.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4688.958524 secs 2024-07-26T08:14:02.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:02 smithi042 ceph-mon[32775]: pgmap v3069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:04 smithi042 ceph-mon[32775]: pgmap v3070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:14:04.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:04 smithi164 ceph-mon[36970]: pgmap v3070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:14:04.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:04 smithi163 ceph-mon[36873]: pgmap v3070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:14:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:06 smithi042 ceph-mon[32775]: pgmap v3071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:06 smithi164 ceph-mon[36970]: pgmap v3071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:06 smithi163 ceph-mon[36873]: pgmap v3071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:07.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4676.216832 secs 2024-07-26T08:14:07.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4693.958649 secs 2024-07-26T08:14:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4676.216832 secs 2024-07-26T08:14:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4693.958649 secs 2024-07-26T08:14:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4676.216832 secs 2024-07-26T08:14:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4693.958649 secs 2024-07-26T08:14:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:08 smithi042 ceph-mon[32775]: pgmap v3072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:08.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:08 smithi164 ceph-mon[36970]: pgmap v3072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:08 smithi163 ceph-mon[36873]: pgmap v3072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:09.870 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:14:09.869+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:14:09.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:14:09.869+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:14:09.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:14:09.869+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:14:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:10 smithi042 ceph-mon[32775]: pgmap v3073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:10 smithi164 ceph-mon[36970]: pgmap v3073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:10 smithi163 ceph-mon[36873]: pgmap v3073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4681.216978 secs 2024-07-26T08:14:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4681.216978 secs 2024-07-26T08:14:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4681.216978 secs 2024-07-26T08:14:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4698.958772 secs 2024-07-26T08:14:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:12 smithi164 ceph-mon[36970]: pgmap v3074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4698.958772 secs 2024-07-26T08:14:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:12 smithi163 ceph-mon[36873]: pgmap v3074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4698.958772 secs 2024-07-26T08:14:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:12 smithi042 ceph-mon[32775]: pgmap v3074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:14 smithi042 ceph-mon[32775]: pgmap v3075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:14:14.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:14 smithi164 ceph-mon[36970]: pgmap v3075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:14:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:14 smithi163 ceph-mon[36873]: pgmap v3075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:14:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:16 smithi042 ceph-mon[32775]: pgmap v3076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:16 smithi164 ceph-mon[36970]: pgmap v3076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:16 smithi163 ceph-mon[36873]: pgmap v3076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4686.217106 secs 2024-07-26T08:14:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4703.958898 secs 2024-07-26T08:14:17.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4686.217106 secs 2024-07-26T08:14:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4703.958898 secs 2024-07-26T08:14:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4686.217106 secs 2024-07-26T08:14:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4703.958898 secs 2024-07-26T08:14:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:18 smithi042 ceph-mon[32775]: pgmap v3077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:18.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:18 smithi164 ceph-mon[36970]: pgmap v3077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:18 smithi163 ceph-mon[36873]: pgmap v3077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:20 smithi042 ceph-mon[32775]: pgmap v3078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:20 smithi164 ceph-mon[36970]: pgmap v3078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:20 smithi163 ceph-mon[36873]: pgmap v3078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4691.217209 secs 2024-07-26T08:14:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4691.217209 secs 2024-07-26T08:14:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4691.217209 secs 2024-07-26T08:14:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4708.959022 secs 2024-07-26T08:14:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:22 smithi164 ceph-mon[36970]: pgmap v3079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4708.959022 secs 2024-07-26T08:14:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:22 smithi163 ceph-mon[36873]: pgmap v3079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4708.959022 secs 2024-07-26T08:14:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:22 smithi042 ceph-mon[32775]: pgmap v3079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:24 smithi042 ceph-mon[32775]: pgmap v3080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:14:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:24 smithi164 ceph-mon[36970]: pgmap v3080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:14:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:24 smithi163 ceph-mon[36873]: pgmap v3080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:14:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:14:24.872+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:14:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:14:24.872+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:14:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:14:24.872+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:14:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4696.217296 secs 2024-07-26T08:14:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:26 smithi042 ceph-mon[32775]: pgmap v3081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4696.217296 secs 2024-07-26T08:14:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:26 smithi164 ceph-mon[36970]: pgmap v3081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4696.217296 secs 2024-07-26T08:14:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:26 smithi163 ceph-mon[36873]: pgmap v3081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4713.959145 secs 2024-07-26T08:14:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4713.959145 secs 2024-07-26T08:14:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4713.959145 secs 2024-07-26T08:14:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:28 smithi042 ceph-mon[32775]: pgmap v3082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:28.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:28 smithi164 ceph-mon[36970]: pgmap v3082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:29.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:28 smithi163 ceph-mon[36873]: pgmap v3082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:30 smithi042 ceph-mon[32775]: pgmap v3083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:14:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:30 smithi164 ceph-mon[36970]: pgmap v3083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:14:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:30 smithi163 ceph-mon[36873]: pgmap v3083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:14:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:14:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:14:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:31 smithi042 ceph-mon[32775]: pgmap v3084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:14:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:31 smithi042 ceph-mon[32775]: pgmap v3085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:14:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:14:31.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:14:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:14:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:31 smithi164 ceph-mon[36970]: pgmap v3084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:14:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:31 smithi164 ceph-mon[36970]: pgmap v3085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:14:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:14:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:14:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:14:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:31 smithi163 ceph-mon[36873]: pgmap v3084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:14:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:31 smithi163 ceph-mon[36873]: pgmap v3085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:14:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:14:32.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4701.217362 secs 2024-07-26T08:14:32.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4718.959241 secs 2024-07-26T08:14:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4701.217362 secs 2024-07-26T08:14:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4718.959241 secs 2024-07-26T08:14:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4701.217362 secs 2024-07-26T08:14:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4718.959241 secs 2024-07-26T08:14:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:33 smithi042 ceph-mon[32775]: pgmap v3086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:14:33.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:33 smithi164 ceph-mon[36970]: pgmap v3086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:14:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:33 smithi163 ceph-mon[36873]: pgmap v3086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:14:35.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:35 smithi042 ceph-mon[32775]: pgmap v3087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:14:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:35 smithi164 ceph-mon[36970]: pgmap v3087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:14:36.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:35 smithi163 ceph-mon[36873]: pgmap v3087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:14:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4706.217459 secs 2024-07-26T08:14:36.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4706.217459 secs 2024-07-26T08:14:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4706.217459 secs 2024-07-26T08:14:37.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4723.959365 secs 2024-07-26T08:14:37.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:37 smithi164 ceph-mon[36970]: pgmap v3088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:14:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4723.959365 secs 2024-07-26T08:14:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:37 smithi042 ceph-mon[32775]: pgmap v3088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:14:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4723.959365 secs 2024-07-26T08:14:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:37 smithi163 ceph-mon[36873]: pgmap v3088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:14:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:14:39.878+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:14:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:14:39.878+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:14:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:14:39.878+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:14:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:39 smithi042 ceph-mon[32775]: pgmap v3089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:14:39.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:39 smithi164 ceph-mon[36970]: pgmap v3089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:14:40.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:39 smithi163 ceph-mon[36873]: pgmap v3089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:14:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:41 smithi042 ceph-mon[32775]: pgmap v3090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:14:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4711.217547 secs 2024-07-26T08:14:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:41 smithi164 ceph-mon[36970]: pgmap v3090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:14:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4711.217547 secs 2024-07-26T08:14:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:41 smithi163 ceph-mon[36873]: pgmap v3090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:14:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4711.217547 secs 2024-07-26T08:14:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4728.959488 secs 2024-07-26T08:14:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4728.959488 secs 2024-07-26T08:14:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4728.959488 secs 2024-07-26T08:14:43.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:43 smithi042 ceph-mon[32775]: pgmap v3091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:43 smithi164 ceph-mon[36970]: pgmap v3091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:43 smithi163 ceph-mon[36873]: pgmap v3091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:45 smithi042 ceph-mon[32775]: pgmap v3092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:45.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:45 smithi164 ceph-mon[36970]: pgmap v3092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:45 smithi163 ceph-mon[36873]: pgmap v3092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4716.217633 secs 2024-07-26T08:14:46.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4716.217633 secs 2024-07-26T08:14:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4716.217633 secs 2024-07-26T08:14:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4733.959593 secs 2024-07-26T08:14:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:47 smithi042 ceph-mon[32775]: pgmap v3093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:47.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4733.959593 secs 2024-07-26T08:14:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:47 smithi164 ceph-mon[36970]: pgmap v3093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4733.959593 secs 2024-07-26T08:14:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:47 smithi163 ceph-mon[36873]: pgmap v3093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:49 smithi042 ceph-mon[32775]: pgmap v3094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:14:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:49 smithi164 ceph-mon[36970]: pgmap v3094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:14:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:49 smithi163 ceph-mon[36873]: pgmap v3094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:14:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:51 smithi042 ceph-mon[32775]: pgmap v3095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4721.217755 secs 2024-07-26T08:14:51.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:51 smithi164 ceph-mon[36970]: pgmap v3095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4721.217755 secs 2024-07-26T08:14:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:51 smithi163 ceph-mon[36873]: pgmap v3095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4721.217755 secs 2024-07-26T08:14:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4738.959687 secs 2024-07-26T08:14:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4738.959687 secs 2024-07-26T08:14:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4738.959687 secs 2024-07-26T08:14:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:53 smithi042 ceph-mon[32775]: pgmap v3096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:53 smithi164 ceph-mon[36970]: pgmap v3096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:53 smithi163 ceph-mon[36873]: pgmap v3096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:14:54.882+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:14:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:14:54.882+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:14:54.883 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:14:54.882+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:14:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:55 smithi042 ceph-mon[32775]: pgmap v3097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:55 smithi164 ceph-mon[36970]: pgmap v3097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:55 smithi163 ceph-mon[36873]: pgmap v3097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:14:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4726.217878 secs 2024-07-26T08:14:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4726.217878 secs 2024-07-26T08:14:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4726.217878 secs 2024-07-26T08:14:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4743.959817 secs 2024-07-26T08:14:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:57 smithi042 ceph-mon[32775]: pgmap v3098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:57.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4743.959817 secs 2024-07-26T08:14:57.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:57 smithi163 ceph-mon[36873]: pgmap v3098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4743.959817 secs 2024-07-26T08:14:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:57 smithi164 ceph-mon[36970]: pgmap v3098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:14:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:14:59 smithi042 ceph-mon[32775]: pgmap v3099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:14:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:14:59 smithi164 ceph-mon[36970]: pgmap v3099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:15:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:14:59 smithi163 ceph-mon[36873]: pgmap v3099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:15:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:01 smithi164 ceph-mon[36970]: pgmap v3100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4731.218004 secs 2024-07-26T08:15:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:01 smithi163 ceph-mon[36873]: pgmap v3100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4731.218004 secs 2024-07-26T08:15:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:01 smithi042 ceph-mon[32775]: pgmap v3100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4731.218004 secs 2024-07-26T08:15:02.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4748.959950 secs 2024-07-26T08:15:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4748.959950 secs 2024-07-26T08:15:03.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4748.959950 secs 2024-07-26T08:15:03.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:03 smithi164 ceph-mon[36970]: pgmap v3101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:04.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:03 smithi163 ceph-mon[36873]: pgmap v3101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:03 smithi042 ceph-mon[32775]: pgmap v3101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:05 smithi164 ceph-mon[36970]: pgmap v3102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:05 smithi163 ceph-mon[36873]: pgmap v3102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:05 smithi042 ceph-mon[32775]: pgmap v3102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:06.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4736.218136 secs 2024-07-26T08:15:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4736.218136 secs 2024-07-26T08:15:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4736.218136 secs 2024-07-26T08:15:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4753.960056 secs 2024-07-26T08:15:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:07 smithi164 ceph-mon[36970]: pgmap v3103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4753.960056 secs 2024-07-26T08:15:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:07 smithi163 ceph-mon[36873]: pgmap v3103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4753.960056 secs 2024-07-26T08:15:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:07 smithi042 ceph-mon[32775]: pgmap v3103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:15:09.891+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:15:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:15:09.891+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:15:09.892 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:15:09.891+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:15:09.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:09 smithi164 ceph-mon[36970]: pgmap v3104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:15:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:09 smithi163 ceph-mon[36873]: pgmap v3104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:15:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:09 smithi042 ceph-mon[32775]: pgmap v3104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:15:11.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:11 smithi164 ceph-mon[36970]: pgmap v3105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4741.218266 secs 2024-07-26T08:15:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:11 smithi163 ceph-mon[36873]: pgmap v3105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4741.218266 secs 2024-07-26T08:15:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:11 smithi042 ceph-mon[32775]: pgmap v3105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4741.218266 secs 2024-07-26T08:15:12.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4758.960134 secs 2024-07-26T08:15:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4758.960134 secs 2024-07-26T08:15:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4758.960134 secs 2024-07-26T08:15:13.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:13 smithi164 ceph-mon[36970]: pgmap v3106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:13 smithi163 ceph-mon[36873]: pgmap v3106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:13 smithi042 ceph-mon[32775]: pgmap v3106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:15.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:15 smithi164 ceph-mon[36970]: pgmap v3107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:15 smithi163 ceph-mon[36873]: pgmap v3107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:15 smithi042 ceph-mon[32775]: pgmap v3107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4746.218334 secs 2024-07-26T08:15:16.976 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4746.218334 secs 2024-07-26T08:15:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4746.218334 secs 2024-07-26T08:15:17.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4763.960208 secs 2024-07-26T08:15:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:17 smithi164 ceph-mon[36970]: pgmap v3108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4763.960208 secs 2024-07-26T08:15:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:17 smithi163 ceph-mon[36873]: pgmap v3108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4763.960208 secs 2024-07-26T08:15:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:17 smithi042 ceph-mon[32775]: pgmap v3108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:19 smithi164 ceph-mon[36970]: pgmap v3109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:15:20.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:19 smithi163 ceph-mon[36873]: pgmap v3109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:15:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:19 smithi042 ceph-mon[32775]: pgmap v3109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:15:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:21 smithi164 ceph-mon[36970]: pgmap v3110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4751.218458 secs 2024-07-26T08:15:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:21 smithi163 ceph-mon[36873]: pgmap v3110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4751.218458 secs 2024-07-26T08:15:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:21 smithi042 ceph-mon[32775]: pgmap v3110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4751.218458 secs 2024-07-26T08:15:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4768.960324 secs 2024-07-26T08:15:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4768.960324 secs 2024-07-26T08:15:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4768.960324 secs 2024-07-26T08:15:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:23 smithi164 ceph-mon[36970]: pgmap v3111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:23 smithi163 ceph-mon[36873]: pgmap v3111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:24.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:23 smithi042 ceph-mon[32775]: pgmap v3111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:24.896 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:15:24.895+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:15:24.896 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:15:24.895+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:15:24.896 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:15:24.895+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:15:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:25 smithi164 ceph-mon[36970]: pgmap v3112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:25 smithi163 ceph-mon[36873]: pgmap v3112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:25 smithi042 ceph-mon[32775]: pgmap v3112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4756.218618 secs 2024-07-26T08:15:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4756.218618 secs 2024-07-26T08:15:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4756.218618 secs 2024-07-26T08:15:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4773.960478 secs 2024-07-26T08:15:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:27 smithi164 ceph-mon[36970]: pgmap v3113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4773.960478 secs 2024-07-26T08:15:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:27 smithi163 ceph-mon[36873]: pgmap v3113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4773.960478 secs 2024-07-26T08:15:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:27 smithi042 ceph-mon[32775]: pgmap v3113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:29 smithi164 ceph-mon[36970]: pgmap v3114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:15:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:29 smithi163 ceph-mon[36873]: pgmap v3114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:15:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:29 smithi042 ceph-mon[32775]: pgmap v3114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:15:31.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:15:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:15:31.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:15:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:31 smithi163 ceph-mon[36873]: pgmap v3115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4761.218688 secs 2024-07-26T08:15:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:15:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:15:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:15:32.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:31 smithi042 ceph-mon[32775]: pgmap v3115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:32.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4761.218688 secs 2024-07-26T08:15:32.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:15:32.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:15:32.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:15:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:31 smithi164 ceph-mon[36970]: pgmap v3115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4761.218688 secs 2024-07-26T08:15:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:15:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:15:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:15:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:32 smithi163 ceph-mon[36873]: pgmap v3116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:15:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:32 smithi163 ceph-mon[36873]: pgmap v3117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:15:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4778.960573 secs 2024-07-26T08:15:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:32 smithi042 ceph-mon[32775]: pgmap v3116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:15:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:32 smithi042 ceph-mon[32775]: pgmap v3117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:15:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4778.960573 secs 2024-07-26T08:15:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:32 smithi164 ceph-mon[36970]: pgmap v3116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:15:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:32 smithi164 ceph-mon[36970]: pgmap v3117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:15:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4778.960573 secs 2024-07-26T08:15:35.012 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:34 smithi163 ceph-mon[36873]: pgmap v3118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:15:35.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:34 smithi042 ceph-mon[32775]: pgmap v3118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:15:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:34 smithi164 ceph-mon[36970]: pgmap v3118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:15:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:36 smithi163 ceph-mon[36873]: pgmap v3119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:15:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4766.218813 secs 2024-07-26T08:15:37.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:36 smithi164 ceph-mon[36970]: pgmap v3119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:15:37.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4766.218813 secs 2024-07-26T08:15:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:36 smithi042 ceph-mon[32775]: pgmap v3119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:15:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4766.218813 secs 2024-07-26T08:15:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4783.960716 secs 2024-07-26T08:15:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4783.960716 secs 2024-07-26T08:15:38.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4783.960716 secs 2024-07-26T08:15:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:38 smithi163 ceph-mon[36873]: pgmap v3120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:15:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:38 smithi042 ceph-mon[32775]: pgmap v3120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:15:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:38 smithi164 ceph-mon[36970]: pgmap v3120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:15:39.904 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:15:39.904+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:15:39.905 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:15:39.904+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:15:39.905 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:15:39.904+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:15:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:40 smithi163 ceph-mon[36873]: pgmap v3121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T08:15:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:40 smithi042 ceph-mon[32775]: pgmap v3121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T08:15:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:40 smithi164 ceph-mon[36970]: pgmap v3121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T08:15:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4771.218899 secs 2024-07-26T08:15:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4771.218899 secs 2024-07-26T08:15:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4771.218899 secs 2024-07-26T08:15:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:42 smithi042 ceph-mon[32775]: pgmap v3122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T08:15:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4788.960838 secs 2024-07-26T08:15:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:42 smithi164 ceph-mon[36970]: pgmap v3122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T08:15:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4788.960838 secs 2024-07-26T08:15:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:42 smithi163 ceph-mon[36873]: pgmap v3122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T08:15:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4788.960838 secs 2024-07-26T08:15:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:44 smithi042 ceph-mon[32775]: pgmap v3123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:15:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:44 smithi164 ceph-mon[36970]: pgmap v3123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:15:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:44 smithi163 ceph-mon[36873]: pgmap v3123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:15:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:46 smithi042 ceph-mon[32775]: pgmap v3124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:15:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4776.219021 secs 2024-07-26T08:15:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:46 smithi164 ceph-mon[36970]: pgmap v3124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:15:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4776.219021 secs 2024-07-26T08:15:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:46 smithi163 ceph-mon[36873]: pgmap v3124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:15:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4776.219021 secs 2024-07-26T08:15:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4793.960966 secs 2024-07-26T08:15:48.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4793.960966 secs 2024-07-26T08:15:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4793.960966 secs 2024-07-26T08:15:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:48 smithi042 ceph-mon[32775]: pgmap v3125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:15:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:48 smithi164 ceph-mon[36970]: pgmap v3125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:15:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:48 smithi163 ceph-mon[36873]: pgmap v3125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:15:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:50 smithi042 ceph-mon[32775]: pgmap v3126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:15:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:50 smithi164 ceph-mon[36970]: pgmap v3126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:15:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:50 smithi163 ceph-mon[36873]: pgmap v3126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:15:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4781.219087 secs 2024-07-26T08:15:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4798.961098 secs 2024-07-26T08:15:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4781.219087 secs 2024-07-26T08:15:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4798.961098 secs 2024-07-26T08:15:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4781.219087 secs 2024-07-26T08:15:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4798.961098 secs 2024-07-26T08:15:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:52 smithi042 ceph-mon[32775]: pgmap v3127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:53.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:52 smithi164 ceph-mon[36970]: pgmap v3127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:52 smithi163 ceph-mon[36873]: pgmap v3127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:54.906 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:15:54.904+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:15:54.907 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:15:54.904+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:15:54.907 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:15:54.904+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:15:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:54 smithi042 ceph-mon[32775]: pgmap v3128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:54 smithi164 ceph-mon[36970]: pgmap v3128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:55.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:54 smithi163 ceph-mon[36873]: pgmap v3128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:57.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:56 smithi163 ceph-mon[36873]: pgmap v3129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:57.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4786.219209 secs 2024-07-26T08:15:57.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4803.961222 secs 2024-07-26T08:15:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:56 smithi042 ceph-mon[32775]: pgmap v3129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4786.219209 secs 2024-07-26T08:15:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4803.961222 secs 2024-07-26T08:15:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:56 smithi164 ceph-mon[36970]: pgmap v3129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:15:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4786.219209 secs 2024-07-26T08:15:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4803.961222 secs 2024-07-26T08:15:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:15:58 smithi042 ceph-mon[32775]: pgmap v3130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:59.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:15:58 smithi164 ceph-mon[36970]: pgmap v3130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:15:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:15:58 smithi163 ceph-mon[36873]: pgmap v3130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:00 smithi042 ceph-mon[32775]: pgmap v3131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:00 smithi164 ceph-mon[36970]: pgmap v3131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:00 smithi163 ceph-mon[36873]: pgmap v3131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4791.219295 secs 2024-07-26T08:16:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4808.961345 secs 2024-07-26T08:16:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4791.219295 secs 2024-07-26T08:16:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4808.961345 secs 2024-07-26T08:16:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4791.219295 secs 2024-07-26T08:16:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4808.961345 secs 2024-07-26T08:16:03.056 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:02 smithi042 ceph-mon[32775]: pgmap v3132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:02 smithi164 ceph-mon[36970]: pgmap v3132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:02 smithi163 ceph-mon[36873]: pgmap v3132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:04 smithi042 ceph-mon[32775]: pgmap v3133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:04 smithi164 ceph-mon[36970]: pgmap v3133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:04 smithi163 ceph-mon[36873]: pgmap v3133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:06 smithi042 ceph-mon[32775]: pgmap v3134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4796.219419 secs 2024-07-26T08:16:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4813.961438 secs 2024-07-26T08:16:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:06 smithi164 ceph-mon[36970]: pgmap v3134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4796.219419 secs 2024-07-26T08:16:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4813.961438 secs 2024-07-26T08:16:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:06 smithi163 ceph-mon[36873]: pgmap v3134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4796.219419 secs 2024-07-26T08:16:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4813.961438 secs 2024-07-26T08:16:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:08 smithi042 ceph-mon[32775]: pgmap v3135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:08 smithi164 ceph-mon[36970]: pgmap v3135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:08 smithi163 ceph-mon[36873]: pgmap v3135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:09.914 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:16:09.913+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:16:09.915 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:16:09.913+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:16:09.915 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:16:09.913+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:16:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:10 smithi042 ceph-mon[32775]: pgmap v3136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:10 smithi164 ceph-mon[36970]: pgmap v3136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:10 smithi163 ceph-mon[36873]: pgmap v3136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4801.219542 secs 2024-07-26T08:16:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4818.961573 secs 2024-07-26T08:16:12.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4801.219542 secs 2024-07-26T08:16:12.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4818.961573 secs 2024-07-26T08:16:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4801.219542 secs 2024-07-26T08:16:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4818.961573 secs 2024-07-26T08:16:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:12 smithi042 ceph-mon[32775]: pgmap v3137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:13.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:12 smithi164 ceph-mon[36970]: pgmap v3137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:12 smithi163 ceph-mon[36873]: pgmap v3137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:14 smithi042 ceph-mon[32775]: pgmap v3138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:14 smithi164 ceph-mon[36970]: pgmap v3138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:14 smithi163 ceph-mon[36873]: pgmap v3138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:16 smithi042 ceph-mon[32775]: pgmap v3139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4806.219665 secs 2024-07-26T08:16:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4823.961696 secs 2024-07-26T08:16:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:16 smithi164 ceph-mon[36970]: pgmap v3139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4806.219665 secs 2024-07-26T08:16:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4823.961696 secs 2024-07-26T08:16:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:16 smithi163 ceph-mon[36873]: pgmap v3139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4806.219665 secs 2024-07-26T08:16:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4823.961696 secs 2024-07-26T08:16:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:18 smithi042 ceph-mon[32775]: pgmap v3140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:18 smithi164 ceph-mon[36970]: pgmap v3140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:18 smithi163 ceph-mon[36873]: pgmap v3140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:20 smithi042 ceph-mon[32775]: pgmap v3141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:20 smithi164 ceph-mon[36970]: pgmap v3141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:20 smithi163 ceph-mon[36873]: pgmap v3141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4811.219743 secs 2024-07-26T08:16:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4828.961818 secs 2024-07-26T08:16:22.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4811.219743 secs 2024-07-26T08:16:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4828.961818 secs 2024-07-26T08:16:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4811.219743 secs 2024-07-26T08:16:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4828.961818 secs 2024-07-26T08:16:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:22 smithi042 ceph-mon[32775]: pgmap v3142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:22 smithi164 ceph-mon[36970]: pgmap v3142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:22 smithi163 ceph-mon[36873]: pgmap v3142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:16:24.916+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:16:24.957 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:16:24.916+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:16:24.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:16:24.916+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:16:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:24 smithi042 ceph-mon[32775]: pgmap v3143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:24 smithi164 ceph-mon[36970]: pgmap v3143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:24 smithi163 ceph-mon[36873]: pgmap v3143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:26 smithi042 ceph-mon[32775]: pgmap v3144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4816.219874 secs 2024-07-26T08:16:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4833.961951 secs 2024-07-26T08:16:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:26 smithi164 ceph-mon[36970]: pgmap v3144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4816.219874 secs 2024-07-26T08:16:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4833.961951 secs 2024-07-26T08:16:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:26 smithi163 ceph-mon[36873]: pgmap v3144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4816.219874 secs 2024-07-26T08:16:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4833.961951 secs 2024-07-26T08:16:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:28 smithi042 ceph-mon[32775]: pgmap v3145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:28 smithi164 ceph-mon[36970]: pgmap v3145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:28 smithi163 ceph-mon[36873]: pgmap v3145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:30 smithi042 ceph-mon[32775]: pgmap v3146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:31.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:30 smithi164 ceph-mon[36970]: pgmap v3146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:30 smithi163 ceph-mon[36873]: pgmap v3146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:32.129 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4821.219997 secs 2024-07-26T08:16:32.130 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4838.962074 secs 2024-07-26T08:16:32.130 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:16:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4821.219997 secs 2024-07-26T08:16:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4838.962074 secs 2024-07-26T08:16:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:16:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4821.219997 secs 2024-07-26T08:16:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4838.962074 secs 2024-07-26T08:16:32.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:16:33.119 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:32 smithi042 ceph-mon[32775]: pgmap v3147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:32 smithi164 ceph-mon[36970]: pgmap v3147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:32 smithi163 ceph-mon[36873]: pgmap v3147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:35.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:34 smithi042 ceph-mon[32775]: pgmap v3148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:34 smithi164 ceph-mon[36970]: pgmap v3148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:34 smithi163 ceph-mon[36873]: pgmap v3148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:16:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:16:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:16:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:16:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:35 smithi042 ceph-mon[32775]: pgmap v3149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:35 smithi042 ceph-mon[32775]: pgmap v3150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:16:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:16:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:16:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:16:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:16:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:16:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:35 smithi164 ceph-mon[36970]: pgmap v3149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:35 smithi164 ceph-mon[36970]: pgmap v3150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:16:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:16:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:16:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:16:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:16:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:16:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:35 smithi163 ceph-mon[36873]: pgmap v3149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:35 smithi163 ceph-mon[36873]: pgmap v3150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:16:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:16:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4826.220121 secs 2024-07-26T08:16:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4843.962190 secs 2024-07-26T08:16:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4826.220121 secs 2024-07-26T08:16:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4843.962190 secs 2024-07-26T08:16:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4826.220121 secs 2024-07-26T08:16:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4843.962190 secs 2024-07-26T08:16:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:37 smithi164 ceph-mon[36970]: pgmap v3151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:16:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:37 smithi163 ceph-mon[36873]: pgmap v3151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:16:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:37 smithi042 ceph-mon[32775]: pgmap v3151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:16:39.927 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:16:39.925+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:16:39.927 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:16:39.925+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:16:39.927 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:16:39.925+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:16:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:39 smithi164 ceph-mon[36970]: pgmap v3152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 3 op/s 2024-07-26T08:16:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:39 smithi163 ceph-mon[36873]: pgmap v3152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 3 op/s 2024-07-26T08:16:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:39 smithi042 ceph-mon[32775]: pgmap v3152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 3 op/s 2024-07-26T08:16:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4831.220279 secs 2024-07-26T08:16:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:41 smithi164 ceph-mon[36970]: pgmap v3153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:16:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4848.962306 secs 2024-07-26T08:16:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4831.220279 secs 2024-07-26T08:16:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:41 smithi163 ceph-mon[36873]: pgmap v3153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:16:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4848.962306 secs 2024-07-26T08:16:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4831.220279 secs 2024-07-26T08:16:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:41 smithi042 ceph-mon[32775]: pgmap v3153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:16:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4848.962306 secs 2024-07-26T08:16:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:43 smithi164 ceph-mon[36970]: pgmap v3154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:16:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:43 smithi163 ceph-mon[36873]: pgmap v3154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:16:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:43 smithi042 ceph-mon[32775]: pgmap v3154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:16:46.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:45 smithi164 ceph-mon[36970]: pgmap v3155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:16:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:45 smithi163 ceph-mon[36873]: pgmap v3155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:16:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:45 smithi042 ceph-mon[32775]: pgmap v3155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:16:47.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4836.220370 secs 2024-07-26T08:16:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4853.962428 secs 2024-07-26T08:16:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4836.220370 secs 2024-07-26T08:16:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4853.962428 secs 2024-07-26T08:16:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4836.220370 secs 2024-07-26T08:16:47.403 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4853.962428 secs 2024-07-26T08:16:48.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:47 smithi164 ceph-mon[36970]: pgmap v3156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:47 smithi163 ceph-mon[36873]: pgmap v3156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:47 smithi042 ceph-mon[32775]: pgmap v3156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:49 smithi164 ceph-mon[36970]: pgmap v3157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:16:50.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:49 smithi163 ceph-mon[36873]: pgmap v3157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:16:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:49 smithi042 ceph-mon[32775]: pgmap v3157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:16:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4841.220498 secs 2024-07-26T08:16:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:51 smithi164 ceph-mon[36970]: pgmap v3158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4858.962528 secs 2024-07-26T08:16:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4841.220498 secs 2024-07-26T08:16:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:51 smithi163 ceph-mon[36873]: pgmap v3158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4858.962528 secs 2024-07-26T08:16:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4841.220498 secs 2024-07-26T08:16:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:51 smithi042 ceph-mon[32775]: pgmap v3158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4858.962528 secs 2024-07-26T08:16:54.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:53 smithi164 ceph-mon[36970]: pgmap v3159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:53 smithi163 ceph-mon[36873]: pgmap v3159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:53 smithi042 ceph-mon[32775]: pgmap v3159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:54.927 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:16:54.926+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:16:54.927 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:16:54.926+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:16:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:16:54.926+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:16:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:55 smithi164 ceph-mon[36970]: pgmap v3160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:56.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:55 smithi163 ceph-mon[36873]: pgmap v3160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:55 smithi042 ceph-mon[32775]: pgmap v3160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:16:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4846.220622 secs 2024-07-26T08:16:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4863.962676 secs 2024-07-26T08:16:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4846.220622 secs 2024-07-26T08:16:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4863.962676 secs 2024-07-26T08:16:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4846.220622 secs 2024-07-26T08:16:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4863.962676 secs 2024-07-26T08:16:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:57 smithi164 ceph-mon[36970]: pgmap v3161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:57 smithi163 ceph-mon[36873]: pgmap v3161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:57 smithi042 ceph-mon[32775]: pgmap v3161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:16:58.629 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:16:58.628+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:16:58.629 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:16:58.628+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:16:58.629 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:16:58.628+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:16:58.629 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:16:58.628+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:17:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:16:59 smithi164 ceph-mon[36970]: pgmap v3162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:17:00.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:16:59 smithi163 ceph-mon[36873]: pgmap v3162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:17:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:16:59 smithi042 ceph-mon[32775]: pgmap v3162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:17:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4851.220747 secs 2024-07-26T08:17:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:01 smithi164 ceph-mon[36970]: pgmap v3163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4868.962796 secs 2024-07-26T08:17:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4851.220747 secs 2024-07-26T08:17:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:01 smithi163 ceph-mon[36873]: pgmap v3163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4868.962796 secs 2024-07-26T08:17:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4851.220747 secs 2024-07-26T08:17:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:01 smithi042 ceph-mon[32775]: pgmap v3163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4868.962796 secs 2024-07-26T08:17:04.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:03 smithi042 ceph-mon[32775]: pgmap v3164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:03 smithi164 ceph-mon[36970]: pgmap v3164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:03 smithi163 ceph-mon[36873]: pgmap v3164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:05 smithi164 ceph-mon[36970]: pgmap v3165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:05 smithi163 ceph-mon[36873]: pgmap v3165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:05 smithi042 ceph-mon[32775]: pgmap v3165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4856.220866 secs 2024-07-26T08:17:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4873.962955 secs 2024-07-26T08:17:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4856.220866 secs 2024-07-26T08:17:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4873.962955 secs 2024-07-26T08:17:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4856.220866 secs 2024-07-26T08:17:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4873.962955 secs 2024-07-26T08:17:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:07 smithi164 ceph-mon[36970]: pgmap v3166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:07 smithi163 ceph-mon[36873]: pgmap v3166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:07 smithi042 ceph-mon[32775]: pgmap v3166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:09.936 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:17:09.935+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:17:09.936 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:17:09.935+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:17:09.936 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:17:09.935+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:17:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:09 smithi164 ceph-mon[36970]: pgmap v3167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:17:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:09 smithi163 ceph-mon[36873]: pgmap v3167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:17:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:09 smithi042 ceph-mon[32775]: pgmap v3167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:17:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4861.220935 secs 2024-07-26T08:17:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:11 smithi164 ceph-mon[36970]: pgmap v3168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4878.963076 secs 2024-07-26T08:17:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4861.220935 secs 2024-07-26T08:17:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:11 smithi163 ceph-mon[36873]: pgmap v3168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4878.963076 secs 2024-07-26T08:17:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4861.220935 secs 2024-07-26T08:17:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:11 smithi042 ceph-mon[32775]: pgmap v3168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4878.963076 secs 2024-07-26T08:17:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:13 smithi164 ceph-mon[36970]: pgmap v3169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:13 smithi163 ceph-mon[36873]: pgmap v3169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:13 smithi042 ceph-mon[32775]: pgmap v3169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:15 smithi163 ceph-mon[36873]: pgmap v3170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:15 smithi042 ceph-mon[32775]: pgmap v3170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:16.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:15 smithi164 ceph-mon[36970]: pgmap v3170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4866.221062 secs 2024-07-26T08:17:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4883.963194 secs 2024-07-26T08:17:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4866.221062 secs 2024-07-26T08:17:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4883.963194 secs 2024-07-26T08:17:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4866.221062 secs 2024-07-26T08:17:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4883.963194 secs 2024-07-26T08:17:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:17 smithi163 ceph-mon[36873]: pgmap v3171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:17 smithi042 ceph-mon[32775]: pgmap v3171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:18.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:17 smithi164 ceph-mon[36970]: pgmap v3171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:19 smithi042 ceph-mon[32775]: pgmap v3172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:17:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:19 smithi163 ceph-mon[36873]: pgmap v3172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:17:20.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:19 smithi164 ceph-mon[36970]: pgmap v3172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:17:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4871.221192 secs 2024-07-26T08:17:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:21 smithi163 ceph-mon[36873]: pgmap v3173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4888.963305 secs 2024-07-26T08:17:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4871.221192 secs 2024-07-26T08:17:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:21 smithi042 ceph-mon[32775]: pgmap v3173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4888.963305 secs 2024-07-26T08:17:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4871.221192 secs 2024-07-26T08:17:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:21 smithi164 ceph-mon[36970]: pgmap v3173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4888.963305 secs 2024-07-26T08:17:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:23 smithi163 ceph-mon[36873]: pgmap v3174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:23 smithi042 ceph-mon[32775]: pgmap v3174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:23 smithi164 ceph-mon[36970]: pgmap v3174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:24.940 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:17:24.939+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:17:24.940 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:17:24.939+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:17:24.940 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:17:24.939+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:17:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:25 smithi163 ceph-mon[36873]: pgmap v3175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:25 smithi042 ceph-mon[32775]: pgmap v3175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:25 smithi164 ceph-mon[36970]: pgmap v3175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4876.221319 secs 2024-07-26T08:17:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4893.963429 secs 2024-07-26T08:17:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:27 smithi163 ceph-mon[36873]: pgmap v3176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4876.221319 secs 2024-07-26T08:17:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4893.963429 secs 2024-07-26T08:17:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:26 smithi042 ceph-mon[32775]: pgmap v3176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4876.221319 secs 2024-07-26T08:17:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4893.963429 secs 2024-07-26T08:17:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:26 smithi164 ceph-mon[36970]: pgmap v3176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:29 smithi164 ceph-mon[36970]: pgmap v3177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:17:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:29 smithi163 ceph-mon[36873]: pgmap v3177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:17:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:29 smithi042 ceph-mon[32775]: pgmap v3177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:17:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4881.221416 secs 2024-07-26T08:17:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:31 smithi042 ceph-mon[32775]: pgmap v3178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4881.221416 secs 2024-07-26T08:17:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:31 smithi164 ceph-mon[36970]: pgmap v3178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4881.221416 secs 2024-07-26T08:17:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:31 smithi163 ceph-mon[36873]: pgmap v3178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4898.963555 secs 2024-07-26T08:17:32.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4898.963555 secs 2024-07-26T08:17:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4898.963555 secs 2024-07-26T08:17:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:33 smithi042 ceph-mon[32775]: pgmap v3179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:33 smithi164 ceph-mon[36970]: pgmap v3179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:34.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:33 smithi163 ceph-mon[36873]: pgmap v3179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:36.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:35 smithi164 ceph-mon[36970]: pgmap v3180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:36.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:17:36.141 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:36.142 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:36.142 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:36.142 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:36.166 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:35 smithi163 ceph-mon[36873]: pgmap v3180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:36.166 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:17:36.166 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:36.167 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:36.167 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:36.167 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:35 smithi042 ceph-mon[32775]: pgmap v3180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:17:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:37.206 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:36 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:17:37.206 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:36 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:17:37.206 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4886.221491 secs 2024-07-26T08:17:37.206 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4903.963679 secs 2024-07-26T08:17:37.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:36 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:17:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:36 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:17:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4886.221491 secs 2024-07-26T08:17:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4903.963679 secs 2024-07-26T08:17:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:36 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:17:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:36 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:17:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4886.221491 secs 2024-07-26T08:17:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4903.963679 secs 2024-07-26T08:17:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:37 smithi163 ceph-mon[36873]: pgmap v3181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:38.330 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:37 smithi164 ceph-mon[36970]: pgmap v3181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:37 smithi042 ceph-mon[32775]: pgmap v3181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:39.948 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:17:39.948+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:17:39.948 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:17:39.948+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:17:39.949 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:17:39.948+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:17:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:17:39.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:17:39.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:39.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:39.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:39.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:39.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:17:39.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:17:39.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:17:39.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:17:39.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:39.976 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:39.976 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:39.976 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:39.976 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:17:39.976 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:17:39.976 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:17:39.977 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:17:39.977 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: pgmap v3182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:17:39.977 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: pgmap v3183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:17:39.977 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:17:40.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:17:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:17:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:40.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:17:40.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:17:40.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:17:40.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:17:40.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:40.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:40.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:40.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:40.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:17:40.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:17:40.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:17:40.011 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:17:40.011 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: pgmap v3182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:17:40.011 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: pgmap v3183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:17:40.011 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:17:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:17:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:17:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:40.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:40.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:17:40.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:17:40.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:17:40.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:17:40.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:40.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:40.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:40.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:17:40.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:17:40.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:17:40.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:17:40.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:17:40.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: pgmap v3182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:17:40.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: pgmap v3183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:17:40.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:17:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:41 smithi042 ceph-mon[32775]: pgmap v3184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:17:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4891.221607 secs 2024-07-26T08:17:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4908.963800 secs 2024-07-26T08:17:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:41 smithi164 ceph-mon[36970]: pgmap v3184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:17:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4891.221607 secs 2024-07-26T08:17:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4908.963800 secs 2024-07-26T08:17:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:41 smithi163 ceph-mon[36873]: pgmap v3184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:17:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4891.221607 secs 2024-07-26T08:17:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4908.963800 secs 2024-07-26T08:17:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:43 smithi042 ceph-mon[32775]: pgmap v3185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:17:44.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:43 smithi164 ceph-mon[36970]: pgmap v3185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:17:44.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:43 smithi163 ceph-mon[36873]: pgmap v3185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:17:46.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:45 smithi042 ceph-mon[32775]: pgmap v3186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:17:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:45 smithi164 ceph-mon[36970]: pgmap v3186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:17:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:45 smithi163 ceph-mon[36873]: pgmap v3186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:17:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4896.221730 secs 2024-07-26T08:17:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4913.963922 secs 2024-07-26T08:17:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4896.221730 secs 2024-07-26T08:17:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4913.963922 secs 2024-07-26T08:17:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4896.221730 secs 2024-07-26T08:17:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4913.963922 secs 2024-07-26T08:17:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:47 smithi042 ceph-mon[32775]: pgmap v3187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:17:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:47 smithi164 ceph-mon[36970]: pgmap v3187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:17:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:47 smithi163 ceph-mon[36873]: pgmap v3187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:17:50.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:49 smithi042 ceph-mon[32775]: pgmap v3188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:17:50.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:49 smithi164 ceph-mon[36970]: pgmap v3188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:17:50.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:49 smithi163 ceph-mon[36873]: pgmap v3188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:17:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:51 smithi042 ceph-mon[32775]: pgmap v3189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4901.221859 secs 2024-07-26T08:17:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4918.964037 secs 2024-07-26T08:17:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:51 smithi164 ceph-mon[36970]: pgmap v3189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4901.221859 secs 2024-07-26T08:17:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4918.964037 secs 2024-07-26T08:17:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:51 smithi163 ceph-mon[36873]: pgmap v3189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4901.221859 secs 2024-07-26T08:17:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4918.964037 secs 2024-07-26T08:17:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:53 smithi042 ceph-mon[32775]: pgmap v3190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:53 smithi164 ceph-mon[36970]: pgmap v3190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:53 smithi163 ceph-mon[36873]: pgmap v3190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:17:54.951+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:17:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:17:54.951+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:17:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:17:54.951+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:17:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:55 smithi042 ceph-mon[32775]: pgmap v3191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:56.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:55 smithi164 ceph-mon[36970]: pgmap v3191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:56.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:55 smithi163 ceph-mon[36873]: pgmap v3191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:17:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4906.221963 secs 2024-07-26T08:17:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4923.964178 secs 2024-07-26T08:17:57.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4906.221963 secs 2024-07-26T08:17:57.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4923.964178 secs 2024-07-26T08:17:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4906.221963 secs 2024-07-26T08:17:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4923.964178 secs 2024-07-26T08:17:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:57 smithi042 ceph-mon[32775]: pgmap v3192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:57 smithi164 ceph-mon[36970]: pgmap v3192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:17:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:57 smithi163 ceph-mon[36873]: pgmap v3192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:17:59 smithi042 ceph-mon[32775]: pgmap v3193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:00.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:17:59 smithi164 ceph-mon[36970]: pgmap v3193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:00.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:17:59 smithi163 ceph-mon[36873]: pgmap v3193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:01 smithi042 ceph-mon[32775]: pgmap v3194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4911.222086 secs 2024-07-26T08:18:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4928.964305 secs 2024-07-26T08:18:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:01 smithi164 ceph-mon[36970]: pgmap v3194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4911.222086 secs 2024-07-26T08:18:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4928.964305 secs 2024-07-26T08:18:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:01 smithi163 ceph-mon[36873]: pgmap v3194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4911.222086 secs 2024-07-26T08:18:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4928.964305 secs 2024-07-26T08:18:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:03 smithi042 ceph-mon[32775]: pgmap v3195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:04.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:03 smithi164 ceph-mon[36970]: pgmap v3195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:03 smithi163 ceph-mon[36873]: pgmap v3195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:05 smithi042 ceph-mon[32775]: pgmap v3196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:05 smithi164 ceph-mon[36970]: pgmap v3196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:05 smithi163 ceph-mon[36873]: pgmap v3196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:07.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4916.222252 secs 2024-07-26T08:18:07.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4933.964426 secs 2024-07-26T08:18:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4916.222252 secs 2024-07-26T08:18:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4933.964426 secs 2024-07-26T08:18:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4916.222252 secs 2024-07-26T08:18:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4933.964426 secs 2024-07-26T08:18:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:07 smithi042 ceph-mon[32775]: pgmap v3197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:07 smithi164 ceph-mon[36970]: pgmap v3197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:07 smithi163 ceph-mon[36873]: pgmap v3197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:09.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:18:09.960+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:18:09.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:18:09.960+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:18:09.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:18:09.960+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:18:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:09 smithi042 ceph-mon[32775]: pgmap v3198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:10.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:09 smithi164 ceph-mon[36970]: pgmap v3198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:12.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:11 smithi042 ceph-mon[32775]: pgmap v3199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:12.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4921.222291 secs 2024-07-26T08:18:12.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4938.964506 secs 2024-07-26T08:18:12.748 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:11 smithi164 ceph-mon[36970]: pgmap v3199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:12.748 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4921.222291 secs 2024-07-26T08:18:12.748 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4938.964506 secs 2024-07-26T08:18:12.748 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:09 smithi163 ceph-mon[36873]: pgmap v3198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:12.749 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:11 smithi163 ceph-mon[36873]: pgmap v3199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:12.749 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4921.222291 secs 2024-07-26T08:18:12.749 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4938.964506 secs 2024-07-26T08:18:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:13 smithi042 ceph-mon[32775]: pgmap v3200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:14.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:13 smithi164 ceph-mon[36970]: pgmap v3200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:13 smithi163 ceph-mon[36873]: pgmap v3200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:15 smithi042 ceph-mon[32775]: pgmap v3201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:16.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:15 smithi164 ceph-mon[36970]: pgmap v3201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:15 smithi163 ceph-mon[36873]: pgmap v3201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4926.222371 secs 2024-07-26T08:18:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4943.964629 secs 2024-07-26T08:18:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4926.222371 secs 2024-07-26T08:18:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4943.964629 secs 2024-07-26T08:18:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4926.222371 secs 2024-07-26T08:18:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4943.964629 secs 2024-07-26T08:18:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:17 smithi042 ceph-mon[32775]: pgmap v3202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:18.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:17 smithi164 ceph-mon[36970]: pgmap v3202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:17 smithi163 ceph-mon[36873]: pgmap v3202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:19 smithi042 ceph-mon[32775]: pgmap v3203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:19 smithi164 ceph-mon[36970]: pgmap v3203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:19 smithi163 ceph-mon[36873]: pgmap v3203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:21 smithi042 ceph-mon[32775]: pgmap v3204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4931.222459 secs 2024-07-26T08:18:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4948.964757 secs 2024-07-26T08:18:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:21 smithi164 ceph-mon[36970]: pgmap v3204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4931.222459 secs 2024-07-26T08:18:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4948.964757 secs 2024-07-26T08:18:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:21 smithi163 ceph-mon[36873]: pgmap v3204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4931.222459 secs 2024-07-26T08:18:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4948.964757 secs 2024-07-26T08:18:24.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:23 smithi164 ceph-mon[36970]: pgmap v3205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:23 smithi163 ceph-mon[36873]: pgmap v3205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:23 smithi042 ceph-mon[32775]: pgmap v3205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:18:24.964+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:18:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:18:24.964+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:18:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:18:24.964+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:18:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:25 smithi164 ceph-mon[36970]: pgmap v3206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:25 smithi163 ceph-mon[36873]: pgmap v3206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:25 smithi042 ceph-mon[32775]: pgmap v3206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:27.158 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4936.222576 secs 2024-07-26T08:18:27.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4953.964904 secs 2024-07-26T08:18:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4936.222576 secs 2024-07-26T08:18:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4953.964904 secs 2024-07-26T08:18:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4936.222576 secs 2024-07-26T08:18:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4953.964904 secs 2024-07-26T08:18:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:27 smithi164 ceph-mon[36970]: pgmap v3207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:27 smithi163 ceph-mon[36873]: pgmap v3207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:27 smithi042 ceph-mon[32775]: pgmap v3207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:28.643 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:18:28.641+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:18:28.643 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:18:28.641+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:18:28.643 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:18:28.642+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:18:28.643 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:18:28.642+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:18:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:29 smithi164 ceph-mon[36970]: pgmap v3208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:29 smithi163 ceph-mon[36873]: pgmap v3208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:29 smithi042 ceph-mon[32775]: pgmap v3208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4941.222695 secs 2024-07-26T08:18:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:31 smithi164 ceph-mon[36970]: pgmap v3209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4958.965027 secs 2024-07-26T08:18:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4941.222695 secs 2024-07-26T08:18:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:31 smithi163 ceph-mon[36873]: pgmap v3209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4958.965027 secs 2024-07-26T08:18:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4941.222695 secs 2024-07-26T08:18:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:31 smithi042 ceph-mon[32775]: pgmap v3209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4958.965027 secs 2024-07-26T08:18:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:33 smithi164 ceph-mon[36970]: pgmap v3210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:33 smithi163 ceph-mon[36873]: pgmap v3210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:33 smithi042 ceph-mon[32775]: pgmap v3210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:36.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:35 smithi042 ceph-mon[32775]: pgmap v3211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:36.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:35 smithi164 ceph-mon[36970]: pgmap v3211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:35 smithi163 ceph-mon[36873]: pgmap v3211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4946.222843 secs 2024-07-26T08:18:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4963.965151 secs 2024-07-26T08:18:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4946.222843 secs 2024-07-26T08:18:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4963.965151 secs 2024-07-26T08:18:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4946.222843 secs 2024-07-26T08:18:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4963.965151 secs 2024-07-26T08:18:38.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:37 smithi164 ceph-mon[36970]: pgmap v3212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:37 smithi163 ceph-mon[36873]: pgmap v3212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:37 smithi042 ceph-mon[32775]: pgmap v3212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:18:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:18:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:18:39.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:18:39.972+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:18:39.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:18:39.972+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:18:39.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:18:39.972+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:18:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:39 smithi164 ceph-mon[36970]: pgmap v3213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:18:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:18:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:18:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:39 smithi163 ceph-mon[36873]: pgmap v3213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:18:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:18:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:18:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:39 smithi042 ceph-mon[32775]: pgmap v3213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:18:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:18:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:18:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:40 smithi164 ceph-mon[36970]: pgmap v3214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T08:18:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:40 smithi164 ceph-mon[36970]: pgmap v3215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:18:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:40 smithi163 ceph-mon[36873]: pgmap v3214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T08:18:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:40 smithi163 ceph-mon[36873]: pgmap v3215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:18:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:40 smithi042 ceph-mon[32775]: pgmap v3214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T08:18:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:40 smithi042 ceph-mon[32775]: pgmap v3215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:18:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4951.222960 secs 2024-07-26T08:18:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4968.965286 secs 2024-07-26T08:18:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4951.222960 secs 2024-07-26T08:18:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4968.965286 secs 2024-07-26T08:18:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4951.222960 secs 2024-07-26T08:18:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4968.965286 secs 2024-07-26T08:18:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:42 smithi164 ceph-mon[36970]: pgmap v3216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:18:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:42 smithi163 ceph-mon[36873]: pgmap v3216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:18:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:42 smithi042 ceph-mon[32775]: pgmap v3216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:18:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:44 smithi164 ceph-mon[36970]: pgmap v3217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:18:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:44 smithi163 ceph-mon[36873]: pgmap v3217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:18:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:44 smithi042 ceph-mon[32775]: pgmap v3217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:18:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:46 smithi164 ceph-mon[36970]: pgmap v3218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:18:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4956.223114 secs 2024-07-26T08:18:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4973.965406 secs 2024-07-26T08:18:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:46 smithi163 ceph-mon[36873]: pgmap v3218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:18:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4956.223114 secs 2024-07-26T08:18:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4973.965406 secs 2024-07-26T08:18:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:46 smithi042 ceph-mon[32775]: pgmap v3218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:18:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4956.223114 secs 2024-07-26T08:18:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4973.965406 secs 2024-07-26T08:18:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:48 smithi164 ceph-mon[36970]: pgmap v3219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:18:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:48 smithi163 ceph-mon[36873]: pgmap v3219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:18:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:48 smithi042 ceph-mon[32775]: pgmap v3219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:18:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:50 smithi164 ceph-mon[36970]: pgmap v3220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:18:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:50 smithi163 ceph-mon[36873]: pgmap v3220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:18:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:50 smithi042 ceph-mon[32775]: pgmap v3220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:18:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4961.223238 secs 2024-07-26T08:18:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4978.965543 secs 2024-07-26T08:18:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4961.223238 secs 2024-07-26T08:18:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4978.965543 secs 2024-07-26T08:18:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4961.223238 secs 2024-07-26T08:18:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4978.965543 secs 2024-07-26T08:18:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:52 smithi164 ceph-mon[36970]: pgmap v3221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:52 smithi163 ceph-mon[36873]: pgmap v3221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:52 smithi042 ceph-mon[32775]: pgmap v3221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:54.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:18:54.975+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:18:54.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:18:54.975+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:18:54.977 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:18:54.975+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:18:55.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:54 smithi163 ceph-mon[36873]: pgmap v3222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:54 smithi042 ceph-mon[32775]: pgmap v3222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:54 smithi164 ceph-mon[36970]: pgmap v3222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:18:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:56 smithi163 ceph-mon[36873]: pgmap v3223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4966.223354 secs 2024-07-26T08:18:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4983.965677 secs 2024-07-26T08:18:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:56 smithi042 ceph-mon[32775]: pgmap v3223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4966.223354 secs 2024-07-26T08:18:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4983.965677 secs 2024-07-26T08:18:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:56 smithi164 ceph-mon[36970]: pgmap v3223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:18:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4966.223354 secs 2024-07-26T08:18:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4983.965677 secs 2024-07-26T08:18:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:18:58 smithi163 ceph-mon[36873]: pgmap v3224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:18:58 smithi042 ceph-mon[32775]: pgmap v3224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:18:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:18:58 smithi164 ceph-mon[36970]: pgmap v3224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:00 smithi163 ceph-mon[36873]: pgmap v3225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:00 smithi042 ceph-mon[32775]: pgmap v3225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:00 smithi164 ceph-mon[36970]: pgmap v3225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4971.223436 secs 2024-07-26T08:19:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4988.965806 secs 2024-07-26T08:19:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4971.223436 secs 2024-07-26T08:19:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4988.965806 secs 2024-07-26T08:19:02.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4971.223436 secs 2024-07-26T08:19:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4988.965806 secs 2024-07-26T08:19:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:02 smithi163 ceph-mon[36873]: pgmap v3226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:02 smithi042 ceph-mon[32775]: pgmap v3226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:03.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:02 smithi164 ceph-mon[36970]: pgmap v3226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:04 smithi163 ceph-mon[36873]: pgmap v3227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:19:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:04 smithi042 ceph-mon[32775]: pgmap v3227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:19:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:04 smithi164 ceph-mon[36970]: pgmap v3227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:19:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:06 smithi163 ceph-mon[36873]: pgmap v3228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4976.223503 secs 2024-07-26T08:19:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4993.965945 secs 2024-07-26T08:19:07.274 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:06 smithi164 ceph-mon[36970]: pgmap v3228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:07.274 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4976.223503 secs 2024-07-26T08:19:07.274 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4993.965945 secs 2024-07-26T08:19:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:06 smithi042 ceph-mon[32775]: pgmap v3228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4976.223503 secs 2024-07-26T08:19:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4993.965945 secs 2024-07-26T08:19:09.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:09 smithi042 ceph-mon[32775]: pgmap v3229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:09 smithi163 ceph-mon[36873]: pgmap v3229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:09.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:09 smithi164 ceph-mon[36970]: pgmap v3229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:19:09.983+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:19:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:19:09.983+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:19:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:19:09.983+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:19:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:11 smithi163 ceph-mon[36873]: pgmap v3230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:11 smithi042 ceph-mon[32775]: pgmap v3230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:11 smithi164 ceph-mon[36970]: pgmap v3230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4981.223574 secs 2024-07-26T08:19:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4998.966045 secs 2024-07-26T08:19:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4981.223574 secs 2024-07-26T08:19:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4998.966045 secs 2024-07-26T08:19:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4981.223574 secs 2024-07-26T08:19:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4998.966045 secs 2024-07-26T08:19:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:13 smithi042 ceph-mon[32775]: pgmap v3231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:13 smithi164 ceph-mon[36970]: pgmap v3231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:13 smithi163 ceph-mon[36873]: pgmap v3231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:14 smithi042 ceph-mon[32775]: pgmap v3232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:19:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:14 smithi164 ceph-mon[36970]: pgmap v3232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:19:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:14 smithi163 ceph-mon[36873]: pgmap v3232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:19:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:16 smithi042 ceph-mon[32775]: pgmap v3233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:16 smithi164 ceph-mon[36970]: pgmap v3233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:16 smithi163 ceph-mon[36873]: pgmap v3233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4986.223673 secs 2024-07-26T08:19:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5003.966135 secs 2024-07-26T08:19:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4986.223673 secs 2024-07-26T08:19:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5003.966135 secs 2024-07-26T08:19:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4986.223673 secs 2024-07-26T08:19:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5003.966135 secs 2024-07-26T08:19:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:18 smithi164 ceph-mon[36970]: pgmap v3234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:18.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:18 smithi163 ceph-mon[36873]: pgmap v3234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:18 smithi042 ceph-mon[32775]: pgmap v3234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:20 smithi164 ceph-mon[36970]: pgmap v3235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:20 smithi163 ceph-mon[36873]: pgmap v3235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:20 smithi042 ceph-mon[32775]: pgmap v3235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4991.223755 secs 2024-07-26T08:19:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5008.966266 secs 2024-07-26T08:19:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:22 smithi164 ceph-mon[36970]: pgmap v3236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4991.223755 secs 2024-07-26T08:19:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5008.966266 secs 2024-07-26T08:19:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:22 smithi163 ceph-mon[36873]: pgmap v3236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4991.223755 secs 2024-07-26T08:19:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5008.966266 secs 2024-07-26T08:19:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:22 smithi042 ceph-mon[32775]: pgmap v3236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:24 smithi042 ceph-mon[32775]: pgmap v3237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:19:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:24 smithi164 ceph-mon[36970]: pgmap v3237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:19:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:24 smithi163 ceph-mon[36873]: pgmap v3237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:19:24.988 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:19:24.987+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:19:24.988 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:19:24.987+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:19:24.988 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:19:24.987+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:19:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:26 smithi164 ceph-mon[36970]: pgmap v3238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:26 smithi163 ceph-mon[36873]: pgmap v3238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:26 smithi042 ceph-mon[32775]: pgmap v3238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 4996.223874 secs 2024-07-26T08:19:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5013.966387 secs 2024-07-26T08:19:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 4996.223874 secs 2024-07-26T08:19:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5013.966387 secs 2024-07-26T08:19:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 4996.223874 secs 2024-07-26T08:19:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5013.966387 secs 2024-07-26T08:19:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:28 smithi164 ceph-mon[36970]: pgmap v3239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:28 smithi163 ceph-mon[36873]: pgmap v3239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:28 smithi042 ceph-mon[32775]: pgmap v3239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:30 smithi164 ceph-mon[36970]: pgmap v3240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:30 smithi163 ceph-mon[36873]: pgmap v3240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:30 smithi042 ceph-mon[32775]: pgmap v3240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5001.224007 secs 2024-07-26T08:19:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5018.966458 secs 2024-07-26T08:19:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:32 smithi164 ceph-mon[36970]: pgmap v3241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5001.224007 secs 2024-07-26T08:19:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5018.966458 secs 2024-07-26T08:19:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:32 smithi163 ceph-mon[36873]: pgmap v3241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5001.224007 secs 2024-07-26T08:19:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5018.966458 secs 2024-07-26T08:19:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:32 smithi042 ceph-mon[32775]: pgmap v3241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:34 smithi164 ceph-mon[36970]: pgmap v3242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:19:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:34 smithi163 ceph-mon[36873]: pgmap v3242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:19:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:34 smithi042 ceph-mon[32775]: pgmap v3242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:19:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:36 smithi164 ceph-mon[36970]: pgmap v3243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5006.224132 secs 2024-07-26T08:19:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:36 smithi163 ceph-mon[36873]: pgmap v3243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5006.224132 secs 2024-07-26T08:19:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:36 smithi042 ceph-mon[32775]: pgmap v3243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5006.224132 secs 2024-07-26T08:19:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5023.966601 secs 2024-07-26T08:19:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5023.966601 secs 2024-07-26T08:19:37.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5023.966601 secs 2024-07-26T08:19:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:38 smithi164 ceph-mon[36970]: pgmap v3244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:38 smithi163 ceph-mon[36873]: pgmap v3244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:38 smithi042 ceph-mon[32775]: pgmap v3244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:19:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:19:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:19:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:19:39.988+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:19:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:19:39.988+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:19:39.990 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:19:39.988+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:19:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:40 smithi164 ceph-mon[36970]: pgmap v3245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:19:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:19:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:40 smithi164 ceph-mon[36970]: pgmap v3246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:19:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:40 smithi164 ceph-mon[36970]: pgmap v3247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:19:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:19:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:40 smithi163 ceph-mon[36873]: pgmap v3245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:19:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:19:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:40 smithi163 ceph-mon[36873]: pgmap v3246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:19:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:40 smithi163 ceph-mon[36873]: pgmap v3247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:19:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:19:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:40 smithi042 ceph-mon[32775]: pgmap v3245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:19:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:19:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:40 smithi042 ceph-mon[32775]: pgmap v3246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:19:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:40 smithi042 ceph-mon[32775]: pgmap v3247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:19:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:19:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5011.224258 secs 2024-07-26T08:19:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5011.224258 secs 2024-07-26T08:19:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5011.224258 secs 2024-07-26T08:19:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5028.966746 secs 2024-07-26T08:19:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:42 smithi164 ceph-mon[36970]: pgmap v3248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:19:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5028.966746 secs 2024-07-26T08:19:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:42 smithi163 ceph-mon[36873]: pgmap v3248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:19:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5028.966746 secs 2024-07-26T08:19:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:42 smithi042 ceph-mon[32775]: pgmap v3248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:19:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:44 smithi164 ceph-mon[36970]: pgmap v3249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:19:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:44 smithi163 ceph-mon[36873]: pgmap v3249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:19:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:44 smithi042 ceph-mon[32775]: pgmap v3249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:19:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5016.224324 secs 2024-07-26T08:19:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:46 smithi164 ceph-mon[36970]: pgmap v3250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:19:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5033.966871 secs 2024-07-26T08:19:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5016.224324 secs 2024-07-26T08:19:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:46 smithi163 ceph-mon[36873]: pgmap v3250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:19:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5033.966871 secs 2024-07-26T08:19:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5016.224324 secs 2024-07-26T08:19:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:46 smithi042 ceph-mon[32775]: pgmap v3250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:19:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5033.966871 secs 2024-07-26T08:19:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:48 smithi164 ceph-mon[36970]: pgmap v3251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:19:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:48 smithi163 ceph-mon[36873]: pgmap v3251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:19:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:48 smithi042 ceph-mon[32775]: pgmap v3251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:19:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:50 smithi164 ceph-mon[36970]: pgmap v3252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:19:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:50 smithi163 ceph-mon[36873]: pgmap v3252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:19:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:50 smithi042 ceph-mon[32775]: pgmap v3252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:19:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5021.224488 secs 2024-07-26T08:19:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5038.966999 secs 2024-07-26T08:19:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5021.224488 secs 2024-07-26T08:19:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5038.966999 secs 2024-07-26T08:19:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5021.224488 secs 2024-07-26T08:19:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5038.966999 secs 2024-07-26T08:19:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:52 smithi164 ceph-mon[36970]: pgmap v3253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:52 smithi163 ceph-mon[36873]: pgmap v3253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:52 smithi042 ceph-mon[32775]: pgmap v3253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:54.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:19:54.992+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:19:54.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:19:54.992+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:19:54.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:19:54.992+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:19:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:54 smithi164 ceph-mon[36970]: pgmap v3254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:19:55.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:54 smithi163 ceph-mon[36873]: pgmap v3254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:19:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:54 smithi042 ceph-mon[32775]: pgmap v3254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:19:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5026.224571 secs 2024-07-26T08:19:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:56 smithi164 ceph-mon[36970]: pgmap v3255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5043.967119 secs 2024-07-26T08:19:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5026.224571 secs 2024-07-26T08:19:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:56 smithi163 ceph-mon[36873]: pgmap v3255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5043.967119 secs 2024-07-26T08:19:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5026.224571 secs 2024-07-26T08:19:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:56 smithi042 ceph-mon[32775]: pgmap v3255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:19:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5043.967119 secs 2024-07-26T08:19:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:19:58.655+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:19:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:19:58.655+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:19:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:19:58.655+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:19:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:19:58.655+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:19:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:19:58 smithi164 ceph-mon[36970]: pgmap v3256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:19:58 smithi163 ceph-mon[36873]: pgmap v3256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:19:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:19:58 smithi042 ceph-mon[32775]: pgmap v3256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:00 smithi164 ceph-mon[36970]: pgmap v3257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:20:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:00 smithi163 ceph-mon[36873]: pgmap v3257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:20:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:00 smithi042 ceph-mon[32775]: pgmap v3257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:20:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5031.224671 secs 2024-07-26T08:20:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5048.967253 secs 2024-07-26T08:20:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5031.224671 secs 2024-07-26T08:20:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5048.967253 secs 2024-07-26T08:20:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5031.224671 secs 2024-07-26T08:20:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5048.967253 secs 2024-07-26T08:20:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:02 smithi164 ceph-mon[36970]: pgmap v3258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:02 smithi163 ceph-mon[36873]: pgmap v3258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:02 smithi042 ceph-mon[32775]: pgmap v3258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:04 smithi163 ceph-mon[36873]: pgmap v3259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:20:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:04 smithi042 ceph-mon[32775]: pgmap v3259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:20:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:04 smithi164 ceph-mon[36970]: pgmap v3259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:20:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5036.224811 secs 2024-07-26T08:20:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:06 smithi163 ceph-mon[36873]: pgmap v3260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5053.967376 secs 2024-07-26T08:20:07.275 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5036.224811 secs 2024-07-26T08:20:07.275 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:06 smithi164 ceph-mon[36970]: pgmap v3260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:07.275 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5053.967376 secs 2024-07-26T08:20:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5036.224811 secs 2024-07-26T08:20:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:06 smithi042 ceph-mon[32775]: pgmap v3260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5053.967376 secs 2024-07-26T08:20:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:08 smithi163 ceph-mon[36873]: pgmap v3261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:08 smithi042 ceph-mon[32775]: pgmap v3261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:08 smithi164 ceph-mon[36970]: pgmap v3261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:20:09.999+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:20:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:20:09.999+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:20:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:20:09.999+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:20:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:10 smithi042 ceph-mon[32775]: pgmap v3262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:10 smithi163 ceph-mon[36873]: pgmap v3262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:10 smithi164 ceph-mon[36970]: pgmap v3262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5041.224929 secs 2024-07-26T08:20:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5058.967454 secs 2024-07-26T08:20:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5041.224929 secs 2024-07-26T08:20:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5058.967454 secs 2024-07-26T08:20:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5041.224929 secs 2024-07-26T08:20:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5058.967454 secs 2024-07-26T08:20:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:12 smithi163 ceph-mon[36873]: pgmap v3263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:12 smithi042 ceph-mon[32775]: pgmap v3263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:12 smithi164 ceph-mon[36970]: pgmap v3263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:14 smithi163 ceph-mon[36873]: pgmap v3264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:20:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:14 smithi042 ceph-mon[32775]: pgmap v3264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:20:15.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:14 smithi164 ceph-mon[36970]: pgmap v3264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:20:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5046.225079 secs 2024-07-26T08:20:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:16 smithi163 ceph-mon[36873]: pgmap v3265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5063.967574 secs 2024-07-26T08:20:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5046.225079 secs 2024-07-26T08:20:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:16 smithi042 ceph-mon[32775]: pgmap v3265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5063.967574 secs 2024-07-26T08:20:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5046.225079 secs 2024-07-26T08:20:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:16 smithi164 ceph-mon[36970]: pgmap v3265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5063.967574 secs 2024-07-26T08:20:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:19 smithi163 ceph-mon[36873]: pgmap v3266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:19 smithi042 ceph-mon[32775]: pgmap v3266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:19 smithi164 ceph-mon[36970]: pgmap v3266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:21 smithi163 ceph-mon[36873]: pgmap v3267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:21 smithi042 ceph-mon[32775]: pgmap v3267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:21 smithi164 ceph-mon[36970]: pgmap v3267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5051.225204 secs 2024-07-26T08:20:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5068.967716 secs 2024-07-26T08:20:22.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5051.225204 secs 2024-07-26T08:20:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5068.967716 secs 2024-07-26T08:20:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5051.225204 secs 2024-07-26T08:20:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5068.967716 secs 2024-07-26T08:20:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:23 smithi042 ceph-mon[32775]: pgmap v3268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:23 smithi164 ceph-mon[36970]: pgmap v3268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:23 smithi163 ceph-mon[36873]: pgmap v3268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:20:25.003+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:20:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:20:25.003+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:20:25.004 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:20:25.003+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:20:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:25 smithi042 ceph-mon[32775]: pgmap v3269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:20:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:25 smithi164 ceph-mon[36970]: pgmap v3269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:20:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:25 smithi163 ceph-mon[36873]: pgmap v3269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:20:27.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5056.225332 secs 2024-07-26T08:20:27.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:27 smithi163 ceph-mon[36873]: pgmap v3270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:27.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5073.967862 secs 2024-07-26T08:20:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5056.225332 secs 2024-07-26T08:20:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:27 smithi042 ceph-mon[32775]: pgmap v3270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5073.967862 secs 2024-07-26T08:20:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5056.225332 secs 2024-07-26T08:20:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:27 smithi164 ceph-mon[36970]: pgmap v3270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5073.967862 secs 2024-07-26T08:20:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:29 smithi042 ceph-mon[32775]: pgmap v3271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:29.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:29 smithi164 ceph-mon[36970]: pgmap v3271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:29 smithi163 ceph-mon[36873]: pgmap v3271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:31 smithi042 ceph-mon[32775]: pgmap v3272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:31 smithi164 ceph-mon[36970]: pgmap v3272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:31 smithi163 ceph-mon[36873]: pgmap v3272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5061.225429 secs 2024-07-26T08:20:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5078.967991 secs 2024-07-26T08:20:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:32 smithi042 ceph-mon[32775]: pgmap v3273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5061.225429 secs 2024-07-26T08:20:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5078.967991 secs 2024-07-26T08:20:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:32 smithi164 ceph-mon[36970]: pgmap v3273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5061.225429 secs 2024-07-26T08:20:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5078.967991 secs 2024-07-26T08:20:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:32 smithi163 ceph-mon[36873]: pgmap v3273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:34 smithi164 ceph-mon[36970]: pgmap v3274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:20:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:34 smithi163 ceph-mon[36873]: pgmap v3274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:20:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:34 smithi042 ceph-mon[32775]: pgmap v3274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:20:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5066.225549 secs 2024-07-26T08:20:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:36 smithi164 ceph-mon[36970]: pgmap v3275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5083.968059 secs 2024-07-26T08:20:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5066.225549 secs 2024-07-26T08:20:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:36 smithi163 ceph-mon[36873]: pgmap v3275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5083.968059 secs 2024-07-26T08:20:37.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5066.225549 secs 2024-07-26T08:20:37.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:36 smithi042 ceph-mon[32775]: pgmap v3275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:37.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5083.968059 secs 2024-07-26T08:20:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:38 smithi164 ceph-mon[36970]: pgmap v3276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:38 smithi163 ceph-mon[36873]: pgmap v3276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:38 smithi042 ceph-mon[32775]: pgmap v3276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:20:40.012+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:20:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:20:40.012+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:20:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:20:40.012+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:20:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:40 smithi164 ceph-mon[36970]: pgmap v3277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:20:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:40 smithi163 ceph-mon[36873]: pgmap v3277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:41.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:20:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:40 smithi042 ceph-mon[32775]: pgmap v3277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:20:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5071.225664 secs 2024-07-26T08:20:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5088.968132 secs 2024-07-26T08:20:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5071.225664 secs 2024-07-26T08:20:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5088.968132 secs 2024-07-26T08:20:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:42.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5071.225664 secs 2024-07-26T08:20:42.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5088.968132 secs 2024-07-26T08:20:42.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:42.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:42.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:42.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:42 smithi163 ceph-mon[36873]: pgmap v3278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:20:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:20:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:42 smithi042 ceph-mon[32775]: pgmap v3278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:43.408 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:43.408 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:43.408 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:20:43.408 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:20:43.409 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:42 smithi164 ceph-mon[36970]: pgmap v3278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:20:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:20:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:20:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:43 smithi163 ceph-mon[36873]: pgmap v3279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:20:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:43 smithi163 ceph-mon[36873]: pgmap v3280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:20:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:43 smithi042 ceph-mon[32775]: pgmap v3279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:20:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:43 smithi042 ceph-mon[32775]: pgmap v3280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:20:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:43 smithi164 ceph-mon[36970]: pgmap v3279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:20:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:43 smithi164 ceph-mon[36970]: pgmap v3280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:20:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:45 smithi163 ceph-mon[36873]: pgmap v3281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:20:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:45 smithi042 ceph-mon[32775]: pgmap v3281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:20:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:45 smithi164 ceph-mon[36970]: pgmap v3281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:20:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5076.225797 secs 2024-07-26T08:20:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5093.968257 secs 2024-07-26T08:20:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5076.225797 secs 2024-07-26T08:20:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5093.968257 secs 2024-07-26T08:20:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5076.225797 secs 2024-07-26T08:20:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5093.968257 secs 2024-07-26T08:20:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:47 smithi163 ceph-mon[36873]: pgmap v3282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:20:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:47 smithi042 ceph-mon[32775]: pgmap v3282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:20:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:47 smithi164 ceph-mon[36970]: pgmap v3282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:20:50.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:50 smithi163 ceph-mon[36873]: pgmap v3283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:20:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:50 smithi042 ceph-mon[32775]: pgmap v3283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:20:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:50 smithi164 ceph-mon[36970]: pgmap v3283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:20:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:52 smithi163 ceph-mon[36873]: pgmap v3284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:20:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5081.225897 secs 2024-07-26T08:20:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5098.968383 secs 2024-07-26T08:20:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:52 smithi042 ceph-mon[32775]: pgmap v3284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:20:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5081.225897 secs 2024-07-26T08:20:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5098.968383 secs 2024-07-26T08:20:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:52 smithi164 ceph-mon[36970]: pgmap v3284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:20:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5081.225897 secs 2024-07-26T08:20:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5098.968383 secs 2024-07-26T08:20:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:54 smithi163 ceph-mon[36873]: pgmap v3285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:20:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:54 smithi042 ceph-mon[32775]: pgmap v3285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:20:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:54 smithi164 ceph-mon[36970]: pgmap v3285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:20:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:20:55.016+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:20:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:20:55.016+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:20:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:20:55.016+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:20:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:56 smithi042 ceph-mon[32775]: pgmap v3286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:56 smithi164 ceph-mon[36970]: pgmap v3286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:56 smithi163 ceph-mon[36873]: pgmap v3286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:20:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5086.226024 secs 2024-07-26T08:20:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5103.968508 secs 2024-07-26T08:20:57.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5086.226024 secs 2024-07-26T08:20:57.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5103.968508 secs 2024-07-26T08:20:57.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5086.226024 secs 2024-07-26T08:20:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5103.968508 secs 2024-07-26T08:20:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:20:58 smithi042 ceph-mon[32775]: pgmap v3287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:20:58 smithi164 ceph-mon[36970]: pgmap v3287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:20:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:20:58 smithi163 ceph-mon[36873]: pgmap v3287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:00 smithi042 ceph-mon[32775]: pgmap v3288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:21:00.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:00 smithi164 ceph-mon[36970]: pgmap v3288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:21:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:00 smithi163 ceph-mon[36873]: pgmap v3288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:21:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:02 smithi042 ceph-mon[32775]: pgmap v3289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5091.226174 secs 2024-07-26T08:21:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5108.968630 secs 2024-07-26T08:21:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:02 smithi164 ceph-mon[36970]: pgmap v3289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5091.226174 secs 2024-07-26T08:21:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5108.968630 secs 2024-07-26T08:21:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:02 smithi163 ceph-mon[36873]: pgmap v3289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5091.226174 secs 2024-07-26T08:21:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5108.968630 secs 2024-07-26T08:21:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:04 smithi042 ceph-mon[32775]: pgmap v3290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:04 smithi164 ceph-mon[36970]: pgmap v3290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:04.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:04 smithi163 ceph-mon[36873]: pgmap v3290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:06 smithi042 ceph-mon[32775]: pgmap v3291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:06 smithi164 ceph-mon[36970]: pgmap v3291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:06 smithi163 ceph-mon[36873]: pgmap v3291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:07.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5096.226307 secs 2024-07-26T08:21:07.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5113.968770 secs 2024-07-26T08:21:07.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:07 smithi042 ceph-mon[32775]: pgmap v3292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:07.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5096.226307 secs 2024-07-26T08:21:07.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5113.968770 secs 2024-07-26T08:21:07.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:07 smithi164 ceph-mon[36970]: pgmap v3292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5096.226307 secs 2024-07-26T08:21:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5113.968770 secs 2024-07-26T08:21:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:07 smithi163 ceph-mon[36873]: pgmap v3292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:09 smithi164 ceph-mon[36970]: pgmap v3293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:21:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:09 smithi163 ceph-mon[36873]: pgmap v3293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:21:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:09 smithi042 ceph-mon[32775]: pgmap v3293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:21:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:21:10.026+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:21:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:21:10.026+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:21:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:21:10.026+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:21:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:11 smithi164 ceph-mon[36970]: pgmap v3294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:11 smithi163 ceph-mon[36873]: pgmap v3294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:11 smithi042 ceph-mon[32775]: pgmap v3294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5101.226434 secs 2024-07-26T08:21:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5118.968871 secs 2024-07-26T08:21:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5101.226434 secs 2024-07-26T08:21:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5118.968871 secs 2024-07-26T08:21:12.816 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5101.226434 secs 2024-07-26T08:21:12.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5118.968871 secs 2024-07-26T08:21:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:13 smithi164 ceph-mon[36970]: pgmap v3295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:13 smithi163 ceph-mon[36873]: pgmap v3295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:13 smithi042 ceph-mon[32775]: pgmap v3295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:15 smithi164 ceph-mon[36970]: pgmap v3296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:15 smithi163 ceph-mon[36873]: pgmap v3296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:15 smithi042 ceph-mon[32775]: pgmap v3296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5106.226500 secs 2024-07-26T08:21:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5123.969013 secs 2024-07-26T08:21:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:17 smithi164 ceph-mon[36970]: pgmap v3297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5106.226500 secs 2024-07-26T08:21:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5123.969013 secs 2024-07-26T08:21:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:17 smithi163 ceph-mon[36873]: pgmap v3297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5106.226500 secs 2024-07-26T08:21:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5123.969013 secs 2024-07-26T08:21:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:17 smithi042 ceph-mon[32775]: pgmap v3297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:19 smithi164 ceph-mon[36970]: pgmap v3298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:21:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:19 smithi163 ceph-mon[36873]: pgmap v3298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:21:19.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:19 smithi042 ceph-mon[32775]: pgmap v3298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:21:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:21 smithi164 ceph-mon[36970]: pgmap v3299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:21.838 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:21 smithi163 ceph-mon[36873]: pgmap v3299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:21 smithi042 ceph-mon[32775]: pgmap v3299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5111.226625 secs 2024-07-26T08:21:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5128.969136 secs 2024-07-26T08:21:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5111.226625 secs 2024-07-26T08:21:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5128.969136 secs 2024-07-26T08:21:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5111.226625 secs 2024-07-26T08:21:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5128.969136 secs 2024-07-26T08:21:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:23 smithi164 ceph-mon[36970]: pgmap v3300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:23 smithi163 ceph-mon[36873]: pgmap v3300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:23 smithi042 ceph-mon[32775]: pgmap v3300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:25.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:21:25.029+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:21:25.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:21:25.029+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:21:25.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:21:25.029+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:21:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:25 smithi164 ceph-mon[36970]: pgmap v3301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:25.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:25 smithi163 ceph-mon[36873]: pgmap v3301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:25 smithi042 ceph-mon[32775]: pgmap v3301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5116.226727 secs 2024-07-26T08:21:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5133.969261 secs 2024-07-26T08:21:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:27 smithi164 ceph-mon[36970]: pgmap v3302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5116.226727 secs 2024-07-26T08:21:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5133.969261 secs 2024-07-26T08:21:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:27 smithi163 ceph-mon[36873]: pgmap v3302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5116.226727 secs 2024-07-26T08:21:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5133.969261 secs 2024-07-26T08:21:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:27 smithi042 ceph-mon[32775]: pgmap v3302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:29 smithi164 ceph-mon[36970]: pgmap v3303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:21:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:29 smithi163 ceph-mon[36873]: pgmap v3303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:21:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:29 smithi042 ceph-mon[32775]: pgmap v3303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:21:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:31 smithi164 ceph-mon[36970]: pgmap v3304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:31 smithi163 ceph-mon[36873]: pgmap v3304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:31 smithi042 ceph-mon[32775]: pgmap v3304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5121.226874 secs 2024-07-26T08:21:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5138.969400 secs 2024-07-26T08:21:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5121.226874 secs 2024-07-26T08:21:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5138.969400 secs 2024-07-26T08:21:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5121.226874 secs 2024-07-26T08:21:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5138.969400 secs 2024-07-26T08:21:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:33 smithi164 ceph-mon[36970]: pgmap v3305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:33 smithi163 ceph-mon[36873]: pgmap v3305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:33 smithi042 ceph-mon[32775]: pgmap v3305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:35.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:35 smithi164 ceph-mon[36970]: pgmap v3306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:35.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:35 smithi163 ceph-mon[36873]: pgmap v3306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:35.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:35 smithi042 ceph-mon[32775]: pgmap v3306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5126.227028 secs 2024-07-26T08:21:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5126.227028 secs 2024-07-26T08:21:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5126.227028 secs 2024-07-26T08:21:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5143.969520 secs 2024-07-26T08:21:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:37 smithi164 ceph-mon[36970]: pgmap v3307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5143.969520 secs 2024-07-26T08:21:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:37 smithi163 ceph-mon[36873]: pgmap v3307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:37.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5143.969520 secs 2024-07-26T08:21:37.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:37 smithi042 ceph-mon[32775]: pgmap v3307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:39 smithi164 ceph-mon[36970]: pgmap v3308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:21:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:39 smithi163 ceph-mon[36873]: pgmap v3308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:21:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:39 smithi042 ceph-mon[32775]: pgmap v3308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:21:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:21:40.030+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:21:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:21:40.030+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:21:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:21:40.030+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:21:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:41 smithi164 ceph-mon[36970]: pgmap v3309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5131.227094 secs 2024-07-26T08:21:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:41 smithi163 ceph-mon[36873]: pgmap v3309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5131.227094 secs 2024-07-26T08:21:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:41 smithi042 ceph-mon[32775]: pgmap v3309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5131.227094 secs 2024-07-26T08:21:42.760 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5148.969639 secs 2024-07-26T08:21:42.760 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:21:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5148.969639 secs 2024-07-26T08:21:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:21:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5148.969639 secs 2024-07-26T08:21:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:21:44.048 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:43 smithi042 ceph-mon[32775]: pgmap v3310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:44.048 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.049 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.049 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.049 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.049 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.049 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.049 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:21:44.049 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:21:44.049 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:43 smithi164 ceph-mon[36970]: pgmap v3310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:21:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:21:44.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:43 smithi163 ceph-mon[36873]: pgmap v3310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:21:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:21:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:21:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:44 smithi042 ceph-mon[32775]: pgmap v3311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T08:21:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:44 smithi042 ceph-mon[32775]: pgmap v3312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T08:21:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:44 smithi164 ceph-mon[36970]: pgmap v3311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T08:21:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:44 smithi164 ceph-mon[36970]: pgmap v3312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T08:21:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:44 smithi163 ceph-mon[36873]: pgmap v3311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T08:21:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:44 smithi163 ceph-mon[36873]: pgmap v3312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T08:21:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:46 smithi042 ceph-mon[32775]: pgmap v3313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:21:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5136.227270 secs 2024-07-26T08:21:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:46 smithi164 ceph-mon[36970]: pgmap v3313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:21:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5136.227270 secs 2024-07-26T08:21:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:46 smithi163 ceph-mon[36873]: pgmap v3313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:21:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5136.227270 secs 2024-07-26T08:21:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5153.969703 secs 2024-07-26T08:21:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5153.969703 secs 2024-07-26T08:21:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5153.969703 secs 2024-07-26T08:21:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:48 smithi042 ceph-mon[32775]: pgmap v3314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:21:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:48 smithi164 ceph-mon[36970]: pgmap v3314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:21:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:48 smithi163 ceph-mon[36873]: pgmap v3314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:21:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:50 smithi042 ceph-mon[32775]: pgmap v3315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:21:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:50 smithi164 ceph-mon[36970]: pgmap v3315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:21:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:50 smithi163 ceph-mon[36873]: pgmap v3315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:21:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5141.227336 secs 2024-07-26T08:21:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5141.227336 secs 2024-07-26T08:21:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5141.227336 secs 2024-07-26T08:21:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5158.969825 secs 2024-07-26T08:21:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:52 smithi042 ceph-mon[32775]: pgmap v3316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:21:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5158.969825 secs 2024-07-26T08:21:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:52 smithi164 ceph-mon[36970]: pgmap v3316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:21:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5158.969825 secs 2024-07-26T08:21:53.303 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:52 smithi163 ceph-mon[36873]: pgmap v3316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:21:55.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:21:55.033+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:21:55.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:21:55.033+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:21:55.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:21:55.033+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:21:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:54 smithi042 ceph-mon[32775]: pgmap v3317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:21:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:54 smithi164 ceph-mon[36970]: pgmap v3317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:21:55.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:54 smithi163 ceph-mon[36873]: pgmap v3317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:21:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:56 smithi042 ceph-mon[32775]: pgmap v3318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5146.227473 secs 2024-07-26T08:21:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:56 smithi164 ceph-mon[36970]: pgmap v3318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5146.227473 secs 2024-07-26T08:21:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:56 smithi163 ceph-mon[36873]: pgmap v3318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:21:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5146.227473 secs 2024-07-26T08:21:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5163.969962 secs 2024-07-26T08:21:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5163.969962 secs 2024-07-26T08:21:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5163.969962 secs 2024-07-26T08:21:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:21:58 smithi042 ceph-mon[32775]: pgmap v3319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:21:58 smithi164 ceph-mon[36970]: pgmap v3319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:21:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:21:58 smithi163 ceph-mon[36873]: pgmap v3319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:00 smithi042 ceph-mon[32775]: pgmap v3320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:00 smithi164 ceph-mon[36970]: pgmap v3320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:00 smithi163 ceph-mon[36873]: pgmap v3320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5151.227606 secs 2024-07-26T08:22:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5168.970101 secs 2024-07-26T08:22:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5151.227606 secs 2024-07-26T08:22:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5168.970101 secs 2024-07-26T08:22:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5151.227606 secs 2024-07-26T08:22:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5168.970101 secs 2024-07-26T08:22:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:02 smithi042 ceph-mon[32775]: pgmap v3321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:02 smithi164 ceph-mon[36970]: pgmap v3321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:02 smithi163 ceph-mon[36873]: pgmap v3321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:04 smithi042 ceph-mon[32775]: pgmap v3322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:04 smithi164 ceph-mon[36970]: pgmap v3322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:04 smithi163 ceph-mon[36873]: pgmap v3322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:06 smithi042 ceph-mon[32775]: pgmap v3323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5156.227737 secs 2024-07-26T08:22:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5173.970195 secs 2024-07-26T08:22:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:06 smithi164 ceph-mon[36970]: pgmap v3323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5156.227737 secs 2024-07-26T08:22:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5173.970195 secs 2024-07-26T08:22:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:06 smithi163 ceph-mon[36873]: pgmap v3323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5156.227737 secs 2024-07-26T08:22:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5173.970195 secs 2024-07-26T08:22:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:08 smithi042 ceph-mon[32775]: pgmap v3324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:08 smithi164 ceph-mon[36970]: pgmap v3324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:08 smithi163 ceph-mon[36873]: pgmap v3324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:22:10.041+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:22:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:22:10.041+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:22:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:22:10.041+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:22:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:10 smithi042 ceph-mon[32775]: pgmap v3325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:10 smithi164 ceph-mon[36970]: pgmap v3325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:10 smithi163 ceph-mon[36873]: pgmap v3325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5161.227895 secs 2024-07-26T08:22:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5178.970287 secs 2024-07-26T08:22:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5161.227895 secs 2024-07-26T08:22:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5178.970287 secs 2024-07-26T08:22:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5161.227895 secs 2024-07-26T08:22:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5178.970287 secs 2024-07-26T08:22:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:12 smithi042 ceph-mon[32775]: pgmap v3326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:12 smithi164 ceph-mon[36970]: pgmap v3326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:12 smithi163 ceph-mon[36873]: pgmap v3326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:14 smithi042 ceph-mon[32775]: pgmap v3327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:14 smithi164 ceph-mon[36970]: pgmap v3327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:14 smithi163 ceph-mon[36873]: pgmap v3327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:16 smithi042 ceph-mon[32775]: pgmap v3328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5166.228058 secs 2024-07-26T08:22:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5183.970443 secs 2024-07-26T08:22:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:16 smithi164 ceph-mon[36970]: pgmap v3328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5166.228058 secs 2024-07-26T08:22:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5183.970443 secs 2024-07-26T08:22:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:16 smithi163 ceph-mon[36873]: pgmap v3328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5166.228058 secs 2024-07-26T08:22:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5183.970443 secs 2024-07-26T08:22:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:18 smithi042 ceph-mon[32775]: pgmap v3329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:18 smithi164 ceph-mon[36970]: pgmap v3329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:18 smithi163 ceph-mon[36873]: pgmap v3329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:20 smithi042 ceph-mon[32775]: pgmap v3330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:20 smithi164 ceph-mon[36970]: pgmap v3330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:20 smithi163 ceph-mon[36873]: pgmap v3330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5171.228181 secs 2024-07-26T08:22:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5188.970577 secs 2024-07-26T08:22:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5171.228181 secs 2024-07-26T08:22:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5188.970577 secs 2024-07-26T08:22:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5171.228181 secs 2024-07-26T08:22:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5188.970577 secs 2024-07-26T08:22:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:22 smithi042 ceph-mon[32775]: pgmap v3331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:22 smithi164 ceph-mon[36970]: pgmap v3331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:22 smithi163 ceph-mon[36873]: pgmap v3331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:22:25.044+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:22:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:22:25.044+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:22:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:22:25.044+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:22:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:24 smithi042 ceph-mon[32775]: pgmap v3332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:24 smithi164 ceph-mon[36970]: pgmap v3332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:24 smithi163 ceph-mon[36873]: pgmap v3332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:26 smithi042 ceph-mon[32775]: pgmap v3333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5176.228305 secs 2024-07-26T08:22:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5193.970719 secs 2024-07-26T08:22:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:26 smithi164 ceph-mon[36970]: pgmap v3333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5176.228305 secs 2024-07-26T08:22:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5193.970719 secs 2024-07-26T08:22:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:26 smithi163 ceph-mon[36873]: pgmap v3333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5176.228305 secs 2024-07-26T08:22:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5193.970719 secs 2024-07-26T08:22:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:28 smithi042 ceph-mon[32775]: pgmap v3334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:28 smithi164 ceph-mon[36970]: pgmap v3334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:28 smithi163 ceph-mon[36873]: pgmap v3334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:30 smithi042 ceph-mon[32775]: pgmap v3335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:30 smithi164 ceph-mon[36970]: pgmap v3335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:30 smithi163 ceph-mon[36873]: pgmap v3335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5181.228427 secs 2024-07-26T08:22:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5198.970840 secs 2024-07-26T08:22:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5181.228427 secs 2024-07-26T08:22:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5198.970840 secs 2024-07-26T08:22:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5181.228427 secs 2024-07-26T08:22:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5198.970840 secs 2024-07-26T08:22:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:32 smithi164 ceph-mon[36970]: pgmap v3336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:32 smithi163 ceph-mon[36873]: pgmap v3336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:32 smithi042 ceph-mon[32775]: pgmap v3336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:34 smithi164 ceph-mon[36970]: pgmap v3337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:34 smithi163 ceph-mon[36873]: pgmap v3337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:34 smithi042 ceph-mon[32775]: pgmap v3337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:36 smithi164 ceph-mon[36970]: pgmap v3338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5186.228496 secs 2024-07-26T08:22:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5203.970964 secs 2024-07-26T08:22:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:36 smithi163 ceph-mon[36873]: pgmap v3338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5186.228496 secs 2024-07-26T08:22:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5203.970964 secs 2024-07-26T08:22:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:36 smithi042 ceph-mon[32775]: pgmap v3338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5186.228496 secs 2024-07-26T08:22:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5203.970964 secs 2024-07-26T08:22:39.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:38 smithi042 ceph-mon[32775]: pgmap v3339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:39.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:38 smithi164 ceph-mon[36970]: pgmap v3339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:38 smithi163 ceph-mon[36873]: pgmap v3339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:40.054 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:22:40.053+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:22:40.054 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:22:40.053+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:22:40.055 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:22:40.053+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:22:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:40 smithi164 ceph-mon[36970]: pgmap v3340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:40 smithi163 ceph-mon[36873]: pgmap v3340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:40 smithi042 ceph-mon[32775]: pgmap v3340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5191.228620 secs 2024-07-26T08:22:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5208.971109 secs 2024-07-26T08:22:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5191.228620 secs 2024-07-26T08:22:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5208.971109 secs 2024-07-26T08:22:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5191.228620 secs 2024-07-26T08:22:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5208.971109 secs 2024-07-26T08:22:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:42 smithi164 ceph-mon[36970]: pgmap v3341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:42 smithi163 ceph-mon[36873]: pgmap v3341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:42 smithi042 ceph-mon[32775]: pgmap v3341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:22:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:22:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:22:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:22:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:22:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:22:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:22:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:22:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:22:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:22:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:22:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:22:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:44 smithi164 ceph-mon[36970]: pgmap v3342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:44 smithi164 ceph-mon[36970]: pgmap v3343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:22:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:44 smithi164 ceph-mon[36970]: pgmap v3344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:22:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:44 smithi163 ceph-mon[36873]: pgmap v3342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:44 smithi163 ceph-mon[36873]: pgmap v3343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:22:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:44 smithi163 ceph-mon[36873]: pgmap v3344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:22:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:44 smithi042 ceph-mon[32775]: pgmap v3342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:44 smithi042 ceph-mon[32775]: pgmap v3343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:22:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:44 smithi042 ceph-mon[32775]: pgmap v3344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:22:47.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:46 smithi042 ceph-mon[32775]: pgmap v3345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:22:47.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5196.228703 secs 2024-07-26T08:22:47.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5213.971230 secs 2024-07-26T08:22:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:46 smithi164 ceph-mon[36970]: pgmap v3345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:22:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5196.228703 secs 2024-07-26T08:22:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5213.971230 secs 2024-07-26T08:22:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:46 smithi163 ceph-mon[36873]: pgmap v3345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:22:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5196.228703 secs 2024-07-26T08:22:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5213.971230 secs 2024-07-26T08:22:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:48 smithi164 ceph-mon[36970]: pgmap v3346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:22:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:48 smithi163 ceph-mon[36873]: pgmap v3346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:22:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:48 smithi042 ceph-mon[32775]: pgmap v3346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:22:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:50 smithi164 ceph-mon[36970]: pgmap v3347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:22:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:50 smithi163 ceph-mon[36873]: pgmap v3347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:22:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:50 smithi042 ceph-mon[32775]: pgmap v3347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:22:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5201.228839 secs 2024-07-26T08:22:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5218.971386 secs 2024-07-26T08:22:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5201.228839 secs 2024-07-26T08:22:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5218.971386 secs 2024-07-26T08:22:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5201.228839 secs 2024-07-26T08:22:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5218.971386 secs 2024-07-26T08:22:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:52 smithi164 ceph-mon[36970]: pgmap v3348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:22:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:52 smithi163 ceph-mon[36873]: pgmap v3348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:22:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:52 smithi042 ceph-mon[32775]: pgmap v3348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:22:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:22:55.054+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:22:55.055 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:22:55.054+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:22:55.056 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:22:55.054+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:22:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:54 smithi164 ceph-mon[36970]: pgmap v3349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:22:55.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:54 smithi163 ceph-mon[36873]: pgmap v3349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:22:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:54 smithi042 ceph-mon[32775]: pgmap v3349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:22:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:56 smithi164 ceph-mon[36970]: pgmap v3350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5206.228963 secs 2024-07-26T08:22:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5223.971509 secs 2024-07-26T08:22:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:56 smithi163 ceph-mon[36873]: pgmap v3350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5206.228963 secs 2024-07-26T08:22:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5223.971509 secs 2024-07-26T08:22:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:56 smithi042 ceph-mon[32775]: pgmap v3350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:22:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5206.228963 secs 2024-07-26T08:22:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5223.971509 secs 2024-07-26T08:22:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:22:58.684+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:22:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:22:58.684+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:22:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:22:58.684+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:22:58.685 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:22:58.684+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:22:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:22:58 smithi164 ceph-mon[36970]: pgmap v3351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:22:58 smithi163 ceph-mon[36873]: pgmap v3351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:22:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:22:58 smithi042 ceph-mon[32775]: pgmap v3351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:00 smithi164 ceph-mon[36970]: pgmap v3352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:00 smithi163 ceph-mon[36873]: pgmap v3352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:00 smithi042 ceph-mon[32775]: pgmap v3352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5211.229086 secs 2024-07-26T08:23:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5228.971575 secs 2024-07-26T08:23:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5211.229086 secs 2024-07-26T08:23:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5228.971575 secs 2024-07-26T08:23:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5211.229086 secs 2024-07-26T08:23:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5228.971575 secs 2024-07-26T08:23:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:02 smithi164 ceph-mon[36970]: pgmap v3353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:02 smithi163 ceph-mon[36873]: pgmap v3353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:02 smithi042 ceph-mon[32775]: pgmap v3353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:04 smithi164 ceph-mon[36970]: pgmap v3354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:23:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:04 smithi163 ceph-mon[36873]: pgmap v3354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:23:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:04 smithi042 ceph-mon[32775]: pgmap v3354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:23:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:06 smithi163 ceph-mon[36873]: pgmap v3355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5216.229207 secs 2024-07-26T08:23:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5233.971703 secs 2024-07-26T08:23:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:06 smithi042 ceph-mon[32775]: pgmap v3355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5216.229207 secs 2024-07-26T08:23:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5233.971703 secs 2024-07-26T08:23:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:06 smithi164 ceph-mon[36970]: pgmap v3355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5216.229207 secs 2024-07-26T08:23:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5233.971703 secs 2024-07-26T08:23:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:08 smithi163 ceph-mon[36873]: pgmap v3356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:08 smithi042 ceph-mon[32775]: pgmap v3356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:08 smithi164 ceph-mon[36970]: pgmap v3356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:23:10.061+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:23:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:23:10.061+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:23:10.064 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:23:10.061+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:23:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:10 smithi163 ceph-mon[36873]: pgmap v3357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:10 smithi042 ceph-mon[32775]: pgmap v3357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:10 smithi164 ceph-mon[36970]: pgmap v3357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5221.229326 secs 2024-07-26T08:23:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5238.971850 secs 2024-07-26T08:23:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5221.229326 secs 2024-07-26T08:23:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5238.971850 secs 2024-07-26T08:23:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5221.229326 secs 2024-07-26T08:23:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5238.971850 secs 2024-07-26T08:23:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:12 smithi163 ceph-mon[36873]: pgmap v3358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:12 smithi042 ceph-mon[32775]: pgmap v3358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:12 smithi164 ceph-mon[36970]: pgmap v3358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:14 smithi163 ceph-mon[36873]: pgmap v3359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:23:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:14 smithi042 ceph-mon[32775]: pgmap v3359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:23:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:14 smithi164 ceph-mon[36970]: pgmap v3359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:23:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:16 smithi163 ceph-mon[36873]: pgmap v3360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5226.229446 secs 2024-07-26T08:23:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5243.971993 secs 2024-07-26T08:23:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:16 smithi042 ceph-mon[32775]: pgmap v3360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5226.229446 secs 2024-07-26T08:23:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5243.971993 secs 2024-07-26T08:23:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:16 smithi164 ceph-mon[36970]: pgmap v3360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5226.229446 secs 2024-07-26T08:23:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5243.971993 secs 2024-07-26T08:23:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:18 smithi163 ceph-mon[36873]: pgmap v3361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:18 smithi042 ceph-mon[32775]: pgmap v3361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:18 smithi164 ceph-mon[36970]: pgmap v3361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:21 smithi163 ceph-mon[36873]: pgmap v3362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:21 smithi042 ceph-mon[32775]: pgmap v3362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:21 smithi164 ceph-mon[36970]: pgmap v3362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5231.229559 secs 2024-07-26T08:23:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5248.972115 secs 2024-07-26T08:23:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5231.229559 secs 2024-07-26T08:23:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5248.972115 secs 2024-07-26T08:23:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5231.229559 secs 2024-07-26T08:23:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5248.972115 secs 2024-07-26T08:23:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:23 smithi042 ceph-mon[32775]: pgmap v3363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:23 smithi164 ceph-mon[36970]: pgmap v3363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:23 smithi163 ceph-mon[36873]: pgmap v3363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:25.067 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:23:25.066+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:23:25.068 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:23:25.066+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:23:25.068 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:23:25.066+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:23:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:25 smithi042 ceph-mon[32775]: pgmap v3364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:23:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:25 smithi164 ceph-mon[36970]: pgmap v3364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:23:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:25 smithi163 ceph-mon[36873]: pgmap v3364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:23:27.309 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:27 smithi163 ceph-mon[36873]: pgmap v3365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:27.309 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5236.229661 secs 2024-07-26T08:23:27.309 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5253.972229 secs 2024-07-26T08:23:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:27 smithi042 ceph-mon[32775]: pgmap v3365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5236.229661 secs 2024-07-26T08:23:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5253.972229 secs 2024-07-26T08:23:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:27 smithi164 ceph-mon[36970]: pgmap v3365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5236.229661 secs 2024-07-26T08:23:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5253.972229 secs 2024-07-26T08:23:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:29 smithi042 ceph-mon[32775]: pgmap v3366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:29 smithi164 ceph-mon[36970]: pgmap v3366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:29 smithi163 ceph-mon[36873]: pgmap v3366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:31 smithi042 ceph-mon[32775]: pgmap v3367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:31 smithi164 ceph-mon[36970]: pgmap v3367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:31 smithi163 ceph-mon[36873]: pgmap v3367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5241.229785 secs 2024-07-26T08:23:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5258.972354 secs 2024-07-26T08:23:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5241.229785 secs 2024-07-26T08:23:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5258.972354 secs 2024-07-26T08:23:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5241.229785 secs 2024-07-26T08:23:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5258.972354 secs 2024-07-26T08:23:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:33 smithi042 ceph-mon[32775]: pgmap v3368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:33 smithi164 ceph-mon[36970]: pgmap v3368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:33 smithi163 ceph-mon[36873]: pgmap v3368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:35 smithi042 ceph-mon[32775]: pgmap v3369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:23:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:35 smithi164 ceph-mon[36970]: pgmap v3369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:23:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:35 smithi163 ceph-mon[36873]: pgmap v3369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:23:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:37 smithi042 ceph-mon[32775]: pgmap v3370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5246.229907 secs 2024-07-26T08:23:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5263.972475 secs 2024-07-26T08:23:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:37 smithi164 ceph-mon[36970]: pgmap v3370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5246.229907 secs 2024-07-26T08:23:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5263.972475 secs 2024-07-26T08:23:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:37 smithi163 ceph-mon[36873]: pgmap v3370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5246.229907 secs 2024-07-26T08:23:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5263.972475 secs 2024-07-26T08:23:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:39 smithi042 ceph-mon[32775]: pgmap v3371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:39 smithi164 ceph-mon[36970]: pgmap v3371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:39 smithi163 ceph-mon[36873]: pgmap v3371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:23:40.074+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:23:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:23:40.074+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:23:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:23:40.074+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:23:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:41 smithi042 ceph-mon[32775]: pgmap v3372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:41 smithi164 ceph-mon[36970]: pgmap v3372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:41 smithi163 ceph-mon[36873]: pgmap v3372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:23:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5251.230034 secs 2024-07-26T08:23:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5268.972605 secs 2024-07-26T08:23:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5251.230034 secs 2024-07-26T08:23:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5268.972605 secs 2024-07-26T08:23:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5251.230034 secs 2024-07-26T08:23:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5268.972605 secs 2024-07-26T08:23:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:43 smithi042 ceph-mon[32775]: pgmap v3373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:43 smithi164 ceph-mon[36970]: pgmap v3373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:43 smithi163 ceph-mon[36873]: pgmap v3373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:23:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:23:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:23:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:45 smithi042 ceph-mon[32775]: pgmap v3374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:23:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:23:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:23:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:23:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:45 smithi164 ceph-mon[36970]: pgmap v3374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:23:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:23:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:23:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:23:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:45 smithi163 ceph-mon[36873]: pgmap v3374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:23:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:23:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:23:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:23:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:46 smithi042 ceph-mon[32775]: pgmap v3375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T08:23:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:46 smithi042 ceph-mon[32775]: pgmap v3376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:23:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:46 smithi164 ceph-mon[36970]: pgmap v3375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T08:23:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:46 smithi164 ceph-mon[36970]: pgmap v3376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:23:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:46 smithi163 ceph-mon[36873]: pgmap v3375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T08:23:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:46 smithi163 ceph-mon[36873]: pgmap v3376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:23:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5256.230159 secs 2024-07-26T08:23:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5273.972728 secs 2024-07-26T08:23:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:47 smithi042 ceph-mon[32775]: pgmap v3377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:23:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5256.230159 secs 2024-07-26T08:23:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5273.972728 secs 2024-07-26T08:23:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:47 smithi164 ceph-mon[36970]: pgmap v3377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:23:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5256.230159 secs 2024-07-26T08:23:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5273.972728 secs 2024-07-26T08:23:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:47 smithi163 ceph-mon[36873]: pgmap v3377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:23:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:49 smithi042 ceph-mon[32775]: pgmap v3378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:23:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:49 smithi164 ceph-mon[36970]: pgmap v3378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:23:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:49 smithi163 ceph-mon[36873]: pgmap v3378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:23:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:51 smithi042 ceph-mon[32775]: pgmap v3379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:23:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:51 smithi164 ceph-mon[36970]: pgmap v3379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:23:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:51 smithi163 ceph-mon[36873]: pgmap v3379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:23:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5261.230289 secs 2024-07-26T08:23:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5278.972849 secs 2024-07-26T08:23:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5261.230289 secs 2024-07-26T08:23:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5278.972849 secs 2024-07-26T08:23:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5261.230289 secs 2024-07-26T08:23:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5278.972849 secs 2024-07-26T08:23:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:53 smithi042 ceph-mon[32775]: pgmap v3380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:23:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:53 smithi164 ceph-mon[36970]: pgmap v3380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:23:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:53 smithi163 ceph-mon[36873]: pgmap v3380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:23:55.080 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:23:55.079+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:23:55.080 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:23:55.079+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:23:55.080 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:23:55.079+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:23:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:55 smithi042 ceph-mon[32775]: pgmap v3381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:23:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:55 smithi164 ceph-mon[36970]: pgmap v3381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:23:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:55 smithi163 ceph-mon[36873]: pgmap v3381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:23:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5266.230391 secs 2024-07-26T08:23:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5283.972971 secs 2024-07-26T08:23:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:57 smithi042 ceph-mon[32775]: pgmap v3382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5266.230391 secs 2024-07-26T08:23:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5283.972971 secs 2024-07-26T08:23:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:57 smithi164 ceph-mon[36970]: pgmap v3382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5266.230391 secs 2024-07-26T08:23:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5283.972971 secs 2024-07-26T08:23:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:57 smithi163 ceph-mon[36873]: pgmap v3382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:23:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:23:59 smithi042 ceph-mon[32775]: pgmap v3383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:23:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:23:59 smithi164 ceph-mon[36970]: pgmap v3383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:23:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:23:59 smithi163 ceph-mon[36873]: pgmap v3383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:24:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:01 smithi042 ceph-mon[32775]: pgmap v3384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:01 smithi164 ceph-mon[36970]: pgmap v3384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:01 smithi163 ceph-mon[36873]: pgmap v3384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5271.230523 secs 2024-07-26T08:24:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5288.973092 secs 2024-07-26T08:24:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5271.230523 secs 2024-07-26T08:24:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5288.973092 secs 2024-07-26T08:24:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5271.230523 secs 2024-07-26T08:24:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5288.973092 secs 2024-07-26T08:24:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:03 smithi042 ceph-mon[32775]: pgmap v3385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:03 smithi164 ceph-mon[36970]: pgmap v3385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:03 smithi163 ceph-mon[36873]: pgmap v3385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:05 smithi042 ceph-mon[32775]: pgmap v3386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:05 smithi164 ceph-mon[36970]: pgmap v3386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:05 smithi163 ceph-mon[36873]: pgmap v3386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5276.230651 secs 2024-07-26T08:24:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5293.973217 secs 2024-07-26T08:24:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:07 smithi042 ceph-mon[32775]: pgmap v3387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5276.230651 secs 2024-07-26T08:24:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5293.973217 secs 2024-07-26T08:24:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:07 smithi164 ceph-mon[36970]: pgmap v3387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5276.230651 secs 2024-07-26T08:24:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5293.973217 secs 2024-07-26T08:24:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:07 smithi163 ceph-mon[36873]: pgmap v3387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:09 smithi042 ceph-mon[32775]: pgmap v3388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:24:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:09 smithi164 ceph-mon[36970]: pgmap v3388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:24:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:09 smithi163 ceph-mon[36873]: pgmap v3388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:24:10.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:24:10.088+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:24:10.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:24:10.088+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:24:10.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:24:10.088+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:24:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:11 smithi042 ceph-mon[32775]: pgmap v3389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:11 smithi164 ceph-mon[36970]: pgmap v3389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:11 smithi163 ceph-mon[36873]: pgmap v3389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5281.230734 secs 2024-07-26T08:24:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5298.973340 secs 2024-07-26T08:24:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5281.230734 secs 2024-07-26T08:24:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5298.973340 secs 2024-07-26T08:24:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5281.230734 secs 2024-07-26T08:24:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5298.973340 secs 2024-07-26T08:24:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:13 smithi164 ceph-mon[36970]: pgmap v3390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:13 smithi163 ceph-mon[36873]: pgmap v3390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:13 smithi042 ceph-mon[32775]: pgmap v3390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:15 smithi164 ceph-mon[36970]: pgmap v3391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:15 smithi163 ceph-mon[36873]: pgmap v3391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:15 smithi042 ceph-mon[32775]: pgmap v3391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5286.230875 secs 2024-07-26T08:24:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5303.973462 secs 2024-07-26T08:24:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:17 smithi164 ceph-mon[36970]: pgmap v3392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5286.230875 secs 2024-07-26T08:24:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5303.973462 secs 2024-07-26T08:24:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:17 smithi163 ceph-mon[36873]: pgmap v3392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5286.230875 secs 2024-07-26T08:24:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5303.973462 secs 2024-07-26T08:24:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:17 smithi042 ceph-mon[32775]: pgmap v3392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:19.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:19 smithi042 ceph-mon[32775]: pgmap v3393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:24:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:19 smithi164 ceph-mon[36970]: pgmap v3393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:24:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:19 smithi163 ceph-mon[36873]: pgmap v3393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:24:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:21 smithi164 ceph-mon[36970]: pgmap v3394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:24:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:21 smithi163 ceph-mon[36873]: pgmap v3394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:24:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:21 smithi042 ceph-mon[32775]: pgmap v3394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:24:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5291.231024 secs 2024-07-26T08:24:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5308.973606 secs 2024-07-26T08:24:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5291.231024 secs 2024-07-26T08:24:22.780 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5308.973606 secs 2024-07-26T08:24:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5291.231024 secs 2024-07-26T08:24:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5308.973606 secs 2024-07-26T08:24:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:23 smithi164 ceph-mon[36970]: pgmap v3395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:24:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:23 smithi163 ceph-mon[36873]: pgmap v3395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:24:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:23 smithi042 ceph-mon[32775]: pgmap v3395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:24:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:24:25.092+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:24:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:24:25.092+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:24:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:24:25.092+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:24:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:25 smithi164 ceph-mon[36970]: pgmap v3396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:24:25.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:25 smithi163 ceph-mon[36873]: pgmap v3396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:24:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:25 smithi042 ceph-mon[32775]: pgmap v3396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:24:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5296.231147 secs 2024-07-26T08:24:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5313.973717 secs 2024-07-26T08:24:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:27 smithi164 ceph-mon[36970]: pgmap v3397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:24:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5296.231147 secs 2024-07-26T08:24:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5313.973717 secs 2024-07-26T08:24:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:27 smithi163 ceph-mon[36873]: pgmap v3397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:24:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5296.231147 secs 2024-07-26T08:24:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5313.973717 secs 2024-07-26T08:24:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:27 smithi042 ceph-mon[32775]: pgmap v3397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:24:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:29 smithi164 ceph-mon[36970]: pgmap v3398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:24:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:29 smithi163 ceph-mon[36873]: pgmap v3398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:24:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:29 smithi042 ceph-mon[32775]: pgmap v3398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:24:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:31 smithi164 ceph-mon[36970]: pgmap v3399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:24:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:31 smithi163 ceph-mon[36873]: pgmap v3399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:24:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:31 smithi042 ceph-mon[32775]: pgmap v3399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:24:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5301.231271 secs 2024-07-26T08:24:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5318.973842 secs 2024-07-26T08:24:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5301.231271 secs 2024-07-26T08:24:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5318.973842 secs 2024-07-26T08:24:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5301.231271 secs 2024-07-26T08:24:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5318.973842 secs 2024-07-26T08:24:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:33 smithi164 ceph-mon[36970]: pgmap v3400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:33 smithi163 ceph-mon[36873]: pgmap v3400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:33 smithi042 ceph-mon[32775]: pgmap v3400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:35 smithi164 ceph-mon[36970]: pgmap v3401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:35.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:35 smithi163 ceph-mon[36873]: pgmap v3401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:35.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:35 smithi042 ceph-mon[32775]: pgmap v3401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5306.231392 secs 2024-07-26T08:24:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5323.973941 secs 2024-07-26T08:24:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:37 smithi164 ceph-mon[36970]: pgmap v3402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5306.231392 secs 2024-07-26T08:24:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5323.973941 secs 2024-07-26T08:24:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:37 smithi163 ceph-mon[36873]: pgmap v3402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5306.231392 secs 2024-07-26T08:24:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5323.973941 secs 2024-07-26T08:24:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:37 smithi042 ceph-mon[32775]: pgmap v3402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:39 smithi164 ceph-mon[36970]: pgmap v3403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:24:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:39 smithi163 ceph-mon[36873]: pgmap v3403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:24:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:39 smithi042 ceph-mon[32775]: pgmap v3403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:24:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:24:40.101+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:24:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:24:40.101+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:24:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:24:40.101+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:24:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:41 smithi164 ceph-mon[36970]: pgmap v3404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:41 smithi163 ceph-mon[36873]: pgmap v3404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:41 smithi042 ceph-mon[32775]: pgmap v3404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5311.231514 secs 2024-07-26T08:24:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5328.974069 secs 2024-07-26T08:24:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5311.231514 secs 2024-07-26T08:24:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5328.974069 secs 2024-07-26T08:24:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5311.231514 secs 2024-07-26T08:24:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5328.974069 secs 2024-07-26T08:24:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:43 smithi164 ceph-mon[36970]: pgmap v3405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:43 smithi163 ceph-mon[36873]: pgmap v3405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:43.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:43 smithi042 ceph-mon[32775]: pgmap v3405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:24:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:45 smithi163 ceph-mon[36873]: pgmap v3406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:24:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:45 smithi163 ceph-mon[36873]: pgmap v3407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:24:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:45 smithi163 ceph-mon[36873]: pgmap v3408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:24:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:24:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:45 smithi042 ceph-mon[32775]: pgmap v3406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:24:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:45 smithi042 ceph-mon[32775]: pgmap v3407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:24:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:45 smithi042 ceph-mon[32775]: pgmap v3408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:24:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:24:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:45 smithi164 ceph-mon[36970]: pgmap v3406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:24:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:24:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:24:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:45 smithi164 ceph-mon[36970]: pgmap v3407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:24:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:45 smithi164 ceph-mon[36970]: pgmap v3408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:24:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:24:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5316.231636 secs 2024-07-26T08:24:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5316.231636 secs 2024-07-26T08:24:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5316.231636 secs 2024-07-26T08:24:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5333.974148 secs 2024-07-26T08:24:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:47 smithi042 ceph-mon[32775]: pgmap v3409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:24:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5333.974148 secs 2024-07-26T08:24:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:47 smithi164 ceph-mon[36970]: pgmap v3409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:24:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5333.974148 secs 2024-07-26T08:24:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:47 smithi163 ceph-mon[36873]: pgmap v3409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:24:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:49 smithi164 ceph-mon[36970]: pgmap v3410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:24:50.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:49 smithi163 ceph-mon[36873]: pgmap v3410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:24:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:49 smithi042 ceph-mon[32775]: pgmap v3410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:24:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5321.231760 secs 2024-07-26T08:24:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:51 smithi164 ceph-mon[36970]: pgmap v3411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:24:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5338.974247 secs 2024-07-26T08:24:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5321.231760 secs 2024-07-26T08:24:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:51 smithi163 ceph-mon[36873]: pgmap v3411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:24:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5338.974247 secs 2024-07-26T08:24:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5321.231760 secs 2024-07-26T08:24:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:51 smithi042 ceph-mon[32775]: pgmap v3411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:24:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5338.974247 secs 2024-07-26T08:24:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:53 smithi164 ceph-mon[36970]: pgmap v3412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:24:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:53 smithi163 ceph-mon[36873]: pgmap v3412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:24:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:53 smithi042 ceph-mon[32775]: pgmap v3412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:24:55.106 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:24:55.104+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:24:55.106 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:24:55.104+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:24:55.106 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:24:55.104+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:24:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:55 smithi164 ceph-mon[36970]: pgmap v3413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:24:56.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:55 smithi163 ceph-mon[36873]: pgmap v3413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:24:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:55 smithi042 ceph-mon[32775]: pgmap v3413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:24:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5326.231904 secs 2024-07-26T08:24:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5343.974371 secs 2024-07-26T08:24:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5326.231904 secs 2024-07-26T08:24:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5343.974371 secs 2024-07-26T08:24:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5326.231904 secs 2024-07-26T08:24:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5343.974371 secs 2024-07-26T08:24:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:57 smithi164 ceph-mon[36970]: pgmap v3414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:57 smithi163 ceph-mon[36873]: pgmap v3414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:24:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:57 smithi042 ceph-mon[32775]: pgmap v3414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:24:59 smithi164 ceph-mon[36970]: pgmap v3415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:25:00.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:24:59 smithi163 ceph-mon[36873]: pgmap v3415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:25:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:24:59 smithi042 ceph-mon[32775]: pgmap v3415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:25:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5331.231969 secs 2024-07-26T08:25:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:01 smithi164 ceph-mon[36970]: pgmap v3416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5348.974469 secs 2024-07-26T08:25:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5331.231969 secs 2024-07-26T08:25:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:01 smithi163 ceph-mon[36873]: pgmap v3416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5348.974469 secs 2024-07-26T08:25:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5331.231969 secs 2024-07-26T08:25:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:01 smithi042 ceph-mon[32775]: pgmap v3416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5348.974469 secs 2024-07-26T08:25:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:03 smithi164 ceph-mon[36970]: pgmap v3417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:03 smithi163 ceph-mon[36873]: pgmap v3417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:03 smithi042 ceph-mon[32775]: pgmap v3417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:05 smithi042 ceph-mon[32775]: pgmap v3418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:05 smithi164 ceph-mon[36970]: pgmap v3418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:05 smithi163 ceph-mon[36873]: pgmap v3418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5336.232071 secs 2024-07-26T08:25:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5353.974567 secs 2024-07-26T08:25:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5336.232071 secs 2024-07-26T08:25:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5353.974567 secs 2024-07-26T08:25:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5336.232071 secs 2024-07-26T08:25:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5353.974567 secs 2024-07-26T08:25:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:07 smithi164 ceph-mon[36970]: pgmap v3419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:07 smithi163 ceph-mon[36873]: pgmap v3419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:07 smithi042 ceph-mon[32775]: pgmap v3419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:10.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:10.112+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:25:10.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:10.112+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:25:10.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:10.112+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:25:10.197 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:09 smithi042 ceph-mon[32775]: pgmap v3420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:25:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:09 smithi164 ceph-mon[36970]: pgmap v3420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:25:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:09 smithi163 ceph-mon[36873]: pgmap v3420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:25:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5341.232155 secs 2024-07-26T08:25:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:11 smithi164 ceph-mon[36970]: pgmap v3421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5358.974691 secs 2024-07-26T08:25:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5341.232155 secs 2024-07-26T08:25:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:11 smithi163 ceph-mon[36873]: pgmap v3421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5358.974691 secs 2024-07-26T08:25:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5341.232155 secs 2024-07-26T08:25:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:11 smithi042 ceph-mon[32775]: pgmap v3421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5358.974691 secs 2024-07-26T08:25:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:13 smithi164 ceph-mon[36970]: pgmap v3422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:13 smithi163 ceph-mon[36873]: pgmap v3422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:13 smithi042 ceph-mon[32775]: pgmap v3422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:15 smithi164 ceph-mon[36970]: pgmap v3423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:15 smithi163 ceph-mon[36873]: pgmap v3423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:15 smithi042 ceph-mon[32775]: pgmap v3423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5346.232304 secs 2024-07-26T08:25:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5363.974810 secs 2024-07-26T08:25:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5346.232304 secs 2024-07-26T08:25:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5363.974810 secs 2024-07-26T08:25:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5346.232304 secs 2024-07-26T08:25:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5363.974810 secs 2024-07-26T08:25:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:17 smithi164 ceph-mon[36970]: pgmap v3424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:17 smithi163 ceph-mon[36873]: pgmap v3424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:17 smithi042 ceph-mon[32775]: pgmap v3424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:19 smithi164 ceph-mon[36970]: pgmap v3425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:25:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:19 smithi163 ceph-mon[36873]: pgmap v3425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:25:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:19 smithi042 ceph-mon[32775]: pgmap v3425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:25:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5351.232386 secs 2024-07-26T08:25:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:21 smithi163 ceph-mon[36873]: pgmap v3426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5368.974948 secs 2024-07-26T08:25:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5351.232386 secs 2024-07-26T08:25:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:21 smithi042 ceph-mon[32775]: pgmap v3426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5368.974948 secs 2024-07-26T08:25:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5351.232386 secs 2024-07-26T08:25:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:21 smithi164 ceph-mon[36970]: pgmap v3426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5368.974948 secs 2024-07-26T08:25:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:23 smithi163 ceph-mon[36873]: pgmap v3427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:23 smithi042 ceph-mon[32775]: pgmap v3427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:23 smithi164 ceph-mon[36970]: pgmap v3427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:25.116+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:25:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:25.116+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:25:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:25.116+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:25:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:25 smithi163 ceph-mon[36873]: pgmap v3428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:25 smithi042 ceph-mon[32775]: pgmap v3428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:25 smithi164 ceph-mon[36970]: pgmap v3428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5356.232507 secs 2024-07-26T08:25:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5373.975071 secs 2024-07-26T08:25:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5356.232507 secs 2024-07-26T08:25:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5373.975071 secs 2024-07-26T08:25:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5356.232507 secs 2024-07-26T08:25:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5373.975071 secs 2024-07-26T08:25:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:27 smithi163 ceph-mon[36873]: pgmap v3429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:27 smithi042 ceph-mon[32775]: pgmap v3429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:27 smithi164 ceph-mon[36970]: pgmap v3429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:29 smithi163 ceph-mon[36873]: pgmap v3430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:25:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:29 smithi042 ceph-mon[32775]: pgmap v3430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:25:30.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:29 smithi164 ceph-mon[36970]: pgmap v3430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:25:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5361.232629 secs 2024-07-26T08:25:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:31 smithi163 ceph-mon[36873]: pgmap v3431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5378.975213 secs 2024-07-26T08:25:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5361.232629 secs 2024-07-26T08:25:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:31 smithi042 ceph-mon[32775]: pgmap v3431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5378.975213 secs 2024-07-26T08:25:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5361.232629 secs 2024-07-26T08:25:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:31 smithi164 ceph-mon[36970]: pgmap v3431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5378.975213 secs 2024-07-26T08:25:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:33 smithi163 ceph-mon[36873]: pgmap v3432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:33 smithi042 ceph-mon[32775]: pgmap v3432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:33 smithi164 ceph-mon[36970]: pgmap v3432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:36 smithi163 ceph-mon[36873]: pgmap v3433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:35 smithi042 ceph-mon[32775]: pgmap v3433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:36 smithi164 ceph-mon[36970]: pgmap v3433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5366.232753 secs 2024-07-26T08:25:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5383.975338 secs 2024-07-26T08:25:37.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5366.232753 secs 2024-07-26T08:25:37.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5383.975338 secs 2024-07-26T08:25:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5366.232753 secs 2024-07-26T08:25:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5383.975338 secs 2024-07-26T08:25:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:38 smithi042 ceph-mon[32775]: pgmap v3434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:38 smithi164 ceph-mon[36970]: pgmap v3434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:38 smithi163 ceph-mon[36873]: pgmap v3434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:40.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:40.125+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:25:40.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:40.125+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:25:40.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:40.125+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:25:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:40 smithi042 ceph-mon[32775]: pgmap v3435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:25:40.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:40 smithi164 ceph-mon[36970]: pgmap v3435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:25:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:40 smithi163 ceph-mon[36873]: pgmap v3435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:25:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5371.232896 secs 2024-07-26T08:25:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:42 smithi042 ceph-mon[32775]: pgmap v3436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5388.975419 secs 2024-07-26T08:25:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5371.232896 secs 2024-07-26T08:25:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:42 smithi164 ceph-mon[36970]: pgmap v3436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5388.975419 secs 2024-07-26T08:25:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5371.232896 secs 2024-07-26T08:25:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:42 smithi163 ceph-mon[36873]: pgmap v3436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5388.975419 secs 2024-07-26T08:25:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:44 smithi042 ceph-mon[32775]: pgmap v3437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:44 smithi164 ceph-mon[36970]: pgmap v3437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:44 smithi163 ceph-mon[36873]: pgmap v3437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:45.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:25:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:46 smithi042 ceph-mon[32775]: pgmap v3438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:25:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:46 smithi164 ceph-mon[36970]: pgmap v3438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:25:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:46 smithi163 ceph-mon[36873]: pgmap v3438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:25:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:25:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:25:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:25:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:47 smithi042 ceph-mon[32775]: pgmap v3439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:25:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:47 smithi042 ceph-mon[32775]: pgmap v3440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:25:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5376.232992 secs 2024-07-26T08:25:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5393.975560 secs 2024-07-26T08:25:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:47 smithi164 ceph-mon[36970]: pgmap v3439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:25:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:47 smithi164 ceph-mon[36970]: pgmap v3440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:25:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5376.232992 secs 2024-07-26T08:25:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5393.975560 secs 2024-07-26T08:25:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:47 smithi163 ceph-mon[36873]: pgmap v3439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:25:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:47 smithi163 ceph-mon[36873]: pgmap v3440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:25:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5376.232992 secs 2024-07-26T08:25:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5393.975560 secs 2024-07-26T08:25:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:49 smithi042 ceph-mon[32775]: pgmap v3441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:25:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:49 smithi164 ceph-mon[36970]: pgmap v3441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:25:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:49 smithi163 ceph-mon[36873]: pgmap v3441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:25:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:51 smithi042 ceph-mon[32775]: pgmap v3442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:25:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:51 smithi164 ceph-mon[36970]: pgmap v3442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:25:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:51 smithi163 ceph-mon[36873]: pgmap v3442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:25:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5381.233150 secs 2024-07-26T08:25:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5398.975627 secs 2024-07-26T08:25:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5381.233150 secs 2024-07-26T08:25:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5398.975627 secs 2024-07-26T08:25:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5381.233150 secs 2024-07-26T08:25:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5398.975627 secs 2024-07-26T08:25:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:53 smithi042 ceph-mon[32775]: pgmap v3443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:25:53.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:53 smithi164 ceph-mon[36970]: pgmap v3443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:25:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:53 smithi163 ceph-mon[36873]: pgmap v3443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:25:55.130 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:55.129+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:25:55.130 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:55.129+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:25:55.131 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:55.129+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:25:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:55 smithi042 ceph-mon[32775]: pgmap v3444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:25:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:55 smithi164 ceph-mon[36970]: pgmap v3444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:25:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:55 smithi163 ceph-mon[36873]: pgmap v3444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:25:57.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:57 smithi163 ceph-mon[36873]: pgmap v3445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:25:57.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5386.233272 secs 2024-07-26T08:25:57.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5403.975695 secs 2024-07-26T08:25:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:57 smithi042 ceph-mon[32775]: pgmap v3445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:25:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5386.233272 secs 2024-07-26T08:25:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5403.975695 secs 2024-07-26T08:25:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:57 smithi164 ceph-mon[36970]: pgmap v3445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:25:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5386.233272 secs 2024-07-26T08:25:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5403.975695 secs 2024-07-26T08:25:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:57.869+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T08:25:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:57.869+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T08:25:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:57.869+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff67802b0b0 tx=0x7ff67804b040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:25:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:57.869+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff67802b0b0 tx=0x7ff67804b040 comp rx=0 tx=0).stop 2024-07-26T08:25:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:57.869+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T08:25:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:57.869+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T08:25:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:57.869+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:25:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:57.869+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:25:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:57.869+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:25:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:57.870+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b4d80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff6780318b0 tx=0x7ff678050000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:25:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:58.712+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:25:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:58.712+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:25:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:58.712+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:25:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:25:58.713+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:25:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:25:59 smithi042 ceph-mon[32775]: pgmap v3446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:25:59 smithi164 ceph-mon[36970]: pgmap v3446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:25:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:25:59 smithi163 ceph-mon[36873]: pgmap v3446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:01 smithi042 ceph-mon[32775]: pgmap v3447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:01 smithi164 ceph-mon[36970]: pgmap v3447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:01 smithi163 ceph-mon[36873]: pgmap v3447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5391.233397 secs 2024-07-26T08:26:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5408.975818 secs 2024-07-26T08:26:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5391.233397 secs 2024-07-26T08:26:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5408.975818 secs 2024-07-26T08:26:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5391.233397 secs 2024-07-26T08:26:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5408.975818 secs 2024-07-26T08:26:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:03 smithi042 ceph-mon[32775]: pgmap v3448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:03 smithi164 ceph-mon[36970]: pgmap v3448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:03 smithi163 ceph-mon[36873]: pgmap v3448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:05 smithi042 ceph-mon[32775]: pgmap v3449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:26:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:05 smithi164 ceph-mon[36970]: pgmap v3449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:26:05.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:05 smithi163 ceph-mon[36873]: pgmap v3449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:26:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:07 smithi042 ceph-mon[32775]: pgmap v3450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5396.233514 secs 2024-07-26T08:26:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5413.975957 secs 2024-07-26T08:26:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:07 smithi164 ceph-mon[36970]: pgmap v3450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5396.233514 secs 2024-07-26T08:26:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5413.975957 secs 2024-07-26T08:26:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:07 smithi163 ceph-mon[36873]: pgmap v3450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5396.233514 secs 2024-07-26T08:26:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5413.975957 secs 2024-07-26T08:26:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:09 smithi042 ceph-mon[32775]: pgmap v3451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:09 smithi164 ceph-mon[36970]: pgmap v3451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:09 smithi163 ceph-mon[36873]: pgmap v3451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:26:10.138+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:26:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:26:10.138+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:26:10.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:26:10.138+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:26:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:11 smithi042 ceph-mon[32775]: pgmap v3452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:11 smithi164 ceph-mon[36970]: pgmap v3452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:11 smithi163 ceph-mon[36873]: pgmap v3452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5401.233673 secs 2024-07-26T08:26:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5418.976023 secs 2024-07-26T08:26:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5401.233673 secs 2024-07-26T08:26:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5418.976023 secs 2024-07-26T08:26:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5401.233673 secs 2024-07-26T08:26:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5418.976023 secs 2024-07-26T08:26:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:13 smithi042 ceph-mon[32775]: pgmap v3453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:13 smithi164 ceph-mon[36970]: pgmap v3453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:13 smithi163 ceph-mon[36873]: pgmap v3453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:14 smithi042 ceph-mon[32775]: pgmap v3454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:26:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:14 smithi164 ceph-mon[36970]: pgmap v3454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:26:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:14 smithi163 ceph-mon[36873]: pgmap v3454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:26:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:16 smithi164 ceph-mon[36970]: pgmap v3455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:16 smithi163 ceph-mon[36873]: pgmap v3455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:16 smithi042 ceph-mon[32775]: pgmap v3455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5406.233794 secs 2024-07-26T08:26:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5423.976138 secs 2024-07-26T08:26:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5406.233794 secs 2024-07-26T08:26:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5423.976138 secs 2024-07-26T08:26:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5406.233794 secs 2024-07-26T08:26:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5423.976138 secs 2024-07-26T08:26:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:18 smithi163 ceph-mon[36873]: pgmap v3456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:18 smithi042 ceph-mon[32775]: pgmap v3456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:18 smithi164 ceph-mon[36970]: pgmap v3456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:20 smithi163 ceph-mon[36873]: pgmap v3457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:20 smithi042 ceph-mon[32775]: pgmap v3457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:20 smithi164 ceph-mon[36970]: pgmap v3457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5411.233930 secs 2024-07-26T08:26:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5428.976220 secs 2024-07-26T08:26:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:22 smithi163 ceph-mon[36873]: pgmap v3458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5411.233930 secs 2024-07-26T08:26:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5428.976220 secs 2024-07-26T08:26:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:22 smithi042 ceph-mon[32775]: pgmap v3458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5411.233930 secs 2024-07-26T08:26:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5428.976220 secs 2024-07-26T08:26:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:22 smithi164 ceph-mon[36970]: pgmap v3458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:24 smithi163 ceph-mon[36873]: pgmap v3459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:26:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:24 smithi042 ceph-mon[32775]: pgmap v3459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:26:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:24 smithi164 ceph-mon[36970]: pgmap v3459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:26:25.143 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:26:25.142+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:26:25.143 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:26:25.142+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:26:25.143 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:26:25.142+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:26:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:26 smithi163 ceph-mon[36873]: pgmap v3460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:26 smithi042 ceph-mon[32775]: pgmap v3460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:26 smithi164 ceph-mon[36970]: pgmap v3460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5416.234006 secs 2024-07-26T08:26:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5433.976338 secs 2024-07-26T08:26:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5416.234006 secs 2024-07-26T08:26:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5433.976338 secs 2024-07-26T08:26:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5416.234006 secs 2024-07-26T08:26:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5433.976338 secs 2024-07-26T08:26:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:28 smithi163 ceph-mon[36873]: pgmap v3461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:28 smithi042 ceph-mon[32775]: pgmap v3461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:28 smithi164 ceph-mon[36970]: pgmap v3461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:30 smithi163 ceph-mon[36873]: pgmap v3462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:30 smithi042 ceph-mon[32775]: pgmap v3462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:30 smithi164 ceph-mon[36970]: pgmap v3462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5421.234129 secs 2024-07-26T08:26:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5421.234129 secs 2024-07-26T08:26:31.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5421.234129 secs 2024-07-26T08:26:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5438.976422 secs 2024-07-26T08:26:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:32 smithi163 ceph-mon[36873]: pgmap v3463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5438.976422 secs 2024-07-26T08:26:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:32 smithi042 ceph-mon[32775]: pgmap v3463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5438.976422 secs 2024-07-26T08:26:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:32 smithi164 ceph-mon[36970]: pgmap v3463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:34 smithi042 ceph-mon[32775]: pgmap v3464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:26:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:34 smithi164 ceph-mon[36970]: pgmap v3464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:26:35.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:34 smithi163 ceph-mon[36873]: pgmap v3464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:26:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:36 smithi042 ceph-mon[32775]: pgmap v3465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5426.234285 secs 2024-07-26T08:26:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:36 smithi164 ceph-mon[36970]: pgmap v3465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5426.234285 secs 2024-07-26T08:26:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:36 smithi163 ceph-mon[36873]: pgmap v3465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5426.234285 secs 2024-07-26T08:26:38.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5443.976559 secs 2024-07-26T08:26:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5443.976559 secs 2024-07-26T08:26:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5443.976559 secs 2024-07-26T08:26:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:38 smithi042 ceph-mon[32775]: pgmap v3466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:38 smithi164 ceph-mon[36970]: pgmap v3466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:38 smithi163 ceph-mon[36873]: pgmap v3466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:40.150 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:26:40.149+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:26:40.150 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:26:40.149+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:26:40.150 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:26:40.149+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:26:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:40 smithi042 ceph-mon[32775]: pgmap v3467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:40 smithi164 ceph-mon[36970]: pgmap v3467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:40 smithi163 ceph-mon[36873]: pgmap v3467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:42.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5431.234384 secs 2024-07-26T08:26:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5431.234384 secs 2024-07-26T08:26:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5431.234384 secs 2024-07-26T08:26:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5448.976679 secs 2024-07-26T08:26:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:42 smithi042 ceph-mon[32775]: pgmap v3468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5448.976679 secs 2024-07-26T08:26:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:42 smithi164 ceph-mon[36970]: pgmap v3468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5448.976679 secs 2024-07-26T08:26:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:42 smithi163 ceph-mon[36873]: pgmap v3468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:26:43.719+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:26:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:26:43.719+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:26:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:26:43.719+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:26:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:26:43.719+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:26:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:44 smithi042 ceph-mon[32775]: pgmap v3469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:26:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:44 smithi164 ceph-mon[36970]: pgmap v3469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:26:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:44 smithi163 ceph-mon[36873]: pgmap v3469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:26:46.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:46.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:26:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:46 smithi042 ceph-mon[32775]: pgmap v3470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5436.234454 secs 2024-07-26T08:26:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:46 smithi042 ceph-mon[32775]: pgmap v3471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:26:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:46 smithi042 ceph-mon[32775]: pgmap v3472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:26:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:26:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:46 smithi164 ceph-mon[36970]: pgmap v3470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5436.234454 secs 2024-07-26T08:26:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:46 smithi164 ceph-mon[36970]: pgmap v3471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:26:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:46 smithi164 ceph-mon[36970]: pgmap v3472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:26:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:26:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:46 smithi163 ceph-mon[36873]: pgmap v3470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:26:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5436.234454 secs 2024-07-26T08:26:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:26:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:26:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:46 smithi163 ceph-mon[36873]: pgmap v3471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:26:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:46 smithi163 ceph-mon[36873]: pgmap v3472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:26:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:26:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5453.976803 secs 2024-07-26T08:26:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5453.976803 secs 2024-07-26T08:26:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5453.976803 secs 2024-07-26T08:26:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:48 smithi042 ceph-mon[32775]: pgmap v3473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:26:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:48 smithi164 ceph-mon[36970]: pgmap v3473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:26:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:48 smithi163 ceph-mon[36873]: pgmap v3473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:26:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:51 smithi042 ceph-mon[32775]: pgmap v3474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:26:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:51 smithi164 ceph-mon[36970]: pgmap v3474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:26:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:51 smithi163 ceph-mon[36873]: pgmap v3474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:26:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5441.234581 secs 2024-07-26T08:26:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5458.976948 secs 2024-07-26T08:26:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5441.234581 secs 2024-07-26T08:26:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5458.976948 secs 2024-07-26T08:26:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5441.234581 secs 2024-07-26T08:26:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5458.976948 secs 2024-07-26T08:26:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:53 smithi042 ceph-mon[32775]: pgmap v3475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:26:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:53 smithi164 ceph-mon[36970]: pgmap v3475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:26:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:53 smithi163 ceph-mon[36873]: pgmap v3475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:26:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:26:55.152+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:26:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:26:55.152+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:26:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:26:55.152+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:26:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:55 smithi042 ceph-mon[32775]: pgmap v3476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:26:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:55 smithi164 ceph-mon[36970]: pgmap v3476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:26:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:55 smithi163 ceph-mon[36873]: pgmap v3476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:26:57.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5446.234709 secs 2024-07-26T08:26:57.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:57 smithi163 ceph-mon[36873]: pgmap v3477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:26:57.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5463.977020 secs 2024-07-26T08:26:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5446.234709 secs 2024-07-26T08:26:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:57 smithi042 ceph-mon[32775]: pgmap v3477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:26:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5463.977020 secs 2024-07-26T08:26:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5446.234709 secs 2024-07-26T08:26:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:57 smithi164 ceph-mon[36970]: pgmap v3477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:26:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5463.977020 secs 2024-07-26T08:26:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:26:59 smithi042 ceph-mon[32775]: pgmap v3478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:26:59 smithi164 ceph-mon[36970]: pgmap v3478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:26:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:26:59 smithi163 ceph-mon[36873]: pgmap v3478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:01 smithi042 ceph-mon[32775]: pgmap v3479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:01 smithi164 ceph-mon[36970]: pgmap v3479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:01 smithi163 ceph-mon[36873]: pgmap v3479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5451.234831 secs 2024-07-26T08:27:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5468.977142 secs 2024-07-26T08:27:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5451.234831 secs 2024-07-26T08:27:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5468.977142 secs 2024-07-26T08:27:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5451.234831 secs 2024-07-26T08:27:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5468.977142 secs 2024-07-26T08:27:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:03 smithi042 ceph-mon[32775]: pgmap v3480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:03 smithi164 ceph-mon[36970]: pgmap v3480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:03 smithi163 ceph-mon[36873]: pgmap v3480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:05 smithi042 ceph-mon[32775]: pgmap v3481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:27:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:05 smithi164 ceph-mon[36970]: pgmap v3481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:27:05.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:05 smithi163 ceph-mon[36873]: pgmap v3481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:27:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5456.234985 secs 2024-07-26T08:27:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:07 smithi042 ceph-mon[32775]: pgmap v3482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5473.977263 secs 2024-07-26T08:27:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5456.234985 secs 2024-07-26T08:27:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:07 smithi164 ceph-mon[36970]: pgmap v3482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5473.977263 secs 2024-07-26T08:27:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5456.234985 secs 2024-07-26T08:27:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:07 smithi163 ceph-mon[36873]: pgmap v3482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5473.977263 secs 2024-07-26T08:27:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:09 smithi042 ceph-mon[32775]: pgmap v3483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:09 smithi164 ceph-mon[36970]: pgmap v3483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:09 smithi163 ceph-mon[36873]: pgmap v3483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:27:10.161+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:27:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:27:10.161+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:27:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:27:10.161+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:27:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:11 smithi042 ceph-mon[32775]: pgmap v3484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:11 smithi164 ceph-mon[36970]: pgmap v3484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:11 smithi163 ceph-mon[36873]: pgmap v3484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5461.235050 secs 2024-07-26T08:27:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5478.977386 secs 2024-07-26T08:27:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5461.235050 secs 2024-07-26T08:27:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5478.977386 secs 2024-07-26T08:27:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5461.235050 secs 2024-07-26T08:27:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5478.977386 secs 2024-07-26T08:27:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:13 smithi042 ceph-mon[32775]: pgmap v3485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:13 smithi164 ceph-mon[36970]: pgmap v3485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:13 smithi163 ceph-mon[36873]: pgmap v3485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:15.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:15 smithi042 ceph-mon[32775]: pgmap v3486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:27:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:15 smithi164 ceph-mon[36970]: pgmap v3486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:27:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:15 smithi163 ceph-mon[36873]: pgmap v3486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:27:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5466.235195 secs 2024-07-26T08:27:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:17 smithi042 ceph-mon[32775]: pgmap v3487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5483.977539 secs 2024-07-26T08:27:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5466.235195 secs 2024-07-26T08:27:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:17 smithi164 ceph-mon[36970]: pgmap v3487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5483.977539 secs 2024-07-26T08:27:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5466.235195 secs 2024-07-26T08:27:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:17 smithi163 ceph-mon[36873]: pgmap v3487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5483.977539 secs 2024-07-26T08:27:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:19 smithi042 ceph-mon[32775]: pgmap v3488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:19 smithi164 ceph-mon[36970]: pgmap v3488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:19 smithi163 ceph-mon[36873]: pgmap v3488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:21 smithi042 ceph-mon[32775]: pgmap v3489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:21 smithi164 ceph-mon[36970]: pgmap v3489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:21 smithi163 ceph-mon[36873]: pgmap v3489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5471.235329 secs 2024-07-26T08:27:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5488.977661 secs 2024-07-26T08:27:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5471.235329 secs 2024-07-26T08:27:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5488.977661 secs 2024-07-26T08:27:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5471.235329 secs 2024-07-26T08:27:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5488.977661 secs 2024-07-26T08:27:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:23 smithi042 ceph-mon[32775]: pgmap v3490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:23 smithi164 ceph-mon[36970]: pgmap v3490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:23 smithi163 ceph-mon[36873]: pgmap v3490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:25.167 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:27:25.165+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:27:25.167 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:27:25.165+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:27:25.167 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:27:25.165+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:27:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:25 smithi042 ceph-mon[32775]: pgmap v3491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:27:25.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:25 smithi164 ceph-mon[36970]: pgmap v3491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:27:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:25 smithi163 ceph-mon[36873]: pgmap v3491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:27:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5476.235452 secs 2024-07-26T08:27:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:26 smithi042 ceph-mon[32775]: pgmap v3492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5476.235452 secs 2024-07-26T08:27:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:26 smithi164 ceph-mon[36970]: pgmap v3492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5476.235452 secs 2024-07-26T08:27:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:26 smithi163 ceph-mon[36873]: pgmap v3492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5493.977762 secs 2024-07-26T08:27:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5493.977762 secs 2024-07-26T08:27:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5493.977762 secs 2024-07-26T08:27:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:28 smithi042 ceph-mon[32775]: pgmap v3493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:28 smithi164 ceph-mon[36970]: pgmap v3493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:28 smithi163 ceph-mon[36873]: pgmap v3493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:31 smithi042 ceph-mon[32775]: pgmap v3494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:31 smithi164 ceph-mon[36970]: pgmap v3494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:31 smithi163 ceph-mon[36873]: pgmap v3494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5481.235513 secs 2024-07-26T08:27:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5498.977912 secs 2024-07-26T08:27:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5481.235513 secs 2024-07-26T08:27:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5498.977912 secs 2024-07-26T08:27:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5481.235513 secs 2024-07-26T08:27:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5498.977912 secs 2024-07-26T08:27:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:33 smithi042 ceph-mon[32775]: pgmap v3495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:33 smithi164 ceph-mon[36970]: pgmap v3495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:33 smithi163 ceph-mon[36873]: pgmap v3495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:35 smithi042 ceph-mon[32775]: pgmap v3496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:27:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:35 smithi164 ceph-mon[36970]: pgmap v3496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:27:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:35 smithi163 ceph-mon[36873]: pgmap v3496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:27:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5486.235636 secs 2024-07-26T08:27:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:36 smithi042 ceph-mon[32775]: pgmap v3497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5486.235636 secs 2024-07-26T08:27:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:36 smithi164 ceph-mon[36970]: pgmap v3497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5486.235636 secs 2024-07-26T08:27:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:36 smithi163 ceph-mon[36873]: pgmap v3497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5503.978028 secs 2024-07-26T08:27:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5503.978028 secs 2024-07-26T08:27:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5503.978028 secs 2024-07-26T08:27:39.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:38 smithi164 ceph-mon[36970]: pgmap v3498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:38 smithi042 ceph-mon[32775]: pgmap v3498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:38 smithi163 ceph-mon[36873]: pgmap v3498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:27:40.167+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:27:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:27:40.167+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:27:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:27:40.167+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:27:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:41 smithi042 ceph-mon[32775]: pgmap v3499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:41 smithi164 ceph-mon[36970]: pgmap v3499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:41 smithi163 ceph-mon[36873]: pgmap v3499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:42.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5491.235756 secs 2024-07-26T08:27:42.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5508.978121 secs 2024-07-26T08:27:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5491.235756 secs 2024-07-26T08:27:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5508.978121 secs 2024-07-26T08:27:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5491.235756 secs 2024-07-26T08:27:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5508.978121 secs 2024-07-26T08:27:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:43 smithi042 ceph-mon[32775]: pgmap v3500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:43.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:43 smithi164 ceph-mon[36970]: pgmap v3500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:43 smithi163 ceph-mon[36873]: pgmap v3500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:27:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:45 smithi042 ceph-mon[32775]: pgmap v3501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:27:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:45 smithi164 ceph-mon[36970]: pgmap v3501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:27:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:45 smithi163 ceph-mon[36873]: pgmap v3501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:27:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5496.235878 secs 2024-07-26T08:27:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: pgmap v3502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5513.978199 secs 2024-07-26T08:27:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:47.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:47.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:27:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5496.235878 secs 2024-07-26T08:27:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: pgmap v3502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5513.978199 secs 2024-07-26T08:27:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:47.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:47.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:27:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5496.235878 secs 2024-07-26T08:27:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: pgmap v3502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:27:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:27:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5513.978199 secs 2024-07-26T08:27:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:27:47.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:27:47.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:27:47.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:27:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:48 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:27:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:48 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:27:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:48 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:27:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:48 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:27:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:48 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:27:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:48 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:27:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:48 smithi042 ceph-mon[32775]: pgmap v3503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:27:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:48 smithi042 ceph-mon[32775]: pgmap v3504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:27:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:48 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:27:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:48 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:27:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:48 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:27:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:48 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:27:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:48 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:27:48.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:48 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:27:48.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:48 smithi164 ceph-mon[36970]: pgmap v3503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:27:48.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:48 smithi164 ceph-mon[36970]: pgmap v3504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:27:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:48 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:27:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:48 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:27:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:48 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:27:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:48 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:27:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:48 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:27:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:48 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:27:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:48 smithi163 ceph-mon[36873]: pgmap v3503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:27:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:48 smithi163 ceph-mon[36873]: pgmap v3504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:27:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:50 smithi042 ceph-mon[32775]: pgmap v3505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:27:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:50 smithi164 ceph-mon[36970]: pgmap v3505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:27:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:50 smithi163 ceph-mon[36873]: pgmap v3505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:27:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:52 smithi042 ceph-mon[32775]: pgmap v3506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:27:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5501.236013 secs 2024-07-26T08:27:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5518.978321 secs 2024-07-26T08:27:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:52 smithi164 ceph-mon[36970]: pgmap v3506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:27:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5501.236013 secs 2024-07-26T08:27:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5518.978321 secs 2024-07-26T08:27:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:52 smithi163 ceph-mon[36873]: pgmap v3506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:27:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5501.236013 secs 2024-07-26T08:27:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5518.978321 secs 2024-07-26T08:27:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:54 smithi042 ceph-mon[32775]: pgmap v3507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:27:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:54 smithi164 ceph-mon[36970]: pgmap v3507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:27:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:54 smithi163 ceph-mon[36873]: pgmap v3507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:27:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:27:55.168+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:27:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:27:55.168+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:27:55.169 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:27:55.168+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:27:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:56 smithi042 ceph-mon[32775]: pgmap v3508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:27:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:56 smithi164 ceph-mon[36970]: pgmap v3508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:27:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:56 smithi163 ceph-mon[36873]: pgmap v3508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:27:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5506.236137 secs 2024-07-26T08:27:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5523.978422 secs 2024-07-26T08:27:57.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5506.236137 secs 2024-07-26T08:27:57.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5523.978422 secs 2024-07-26T08:27:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5506.236137 secs 2024-07-26T08:27:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5523.978422 secs 2024-07-26T08:27:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:27:58 smithi042 ceph-mon[32775]: pgmap v3509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:27:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:27:58 smithi164 ceph-mon[36970]: pgmap v3509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:27:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:27:58 smithi163 ceph-mon[36873]: pgmap v3509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:28:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:00 smithi042 ceph-mon[32775]: pgmap v3510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:28:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:00 smithi164 ceph-mon[36970]: pgmap v3510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:28:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:00 smithi163 ceph-mon[36873]: pgmap v3510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:28:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:02 smithi042 ceph-mon[32775]: pgmap v3511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5511.236236 secs 2024-07-26T08:28:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5528.978562 secs 2024-07-26T08:28:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:02 smithi164 ceph-mon[36970]: pgmap v3511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5511.236236 secs 2024-07-26T08:28:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5528.978562 secs 2024-07-26T08:28:02.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:02 smithi163 ceph-mon[36873]: pgmap v3511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:02.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5511.236236 secs 2024-07-26T08:28:02.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5528.978562 secs 2024-07-26T08:28:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:04 smithi042 ceph-mon[32775]: pgmap v3512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:04 smithi164 ceph-mon[36970]: pgmap v3512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:04.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:04 smithi163 ceph-mon[36873]: pgmap v3512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:06 smithi042 ceph-mon[32775]: pgmap v3513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:06 smithi164 ceph-mon[36970]: pgmap v3513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:06 smithi163 ceph-mon[36873]: pgmap v3513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5516.236362 secs 2024-07-26T08:28:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5533.978691 secs 2024-07-26T08:28:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5516.236362 secs 2024-07-26T08:28:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5533.978691 secs 2024-07-26T08:28:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5516.236362 secs 2024-07-26T08:28:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5533.978691 secs 2024-07-26T08:28:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:08 smithi042 ceph-mon[32775]: pgmap v3514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:08 smithi164 ceph-mon[36970]: pgmap v3514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:08 smithi163 ceph-mon[36873]: pgmap v3514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:28:10.177+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:28:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:28:10.177+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:28:10.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:28:10.177+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:28:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:10 smithi042 ceph-mon[32775]: pgmap v3515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:28:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:10 smithi164 ceph-mon[36970]: pgmap v3515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:28:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:10 smithi163 ceph-mon[36873]: pgmap v3515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:28:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:12 smithi042 ceph-mon[32775]: pgmap v3516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5521.236463 secs 2024-07-26T08:28:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5538.978818 secs 2024-07-26T08:28:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:12 smithi164 ceph-mon[36970]: pgmap v3516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5521.236463 secs 2024-07-26T08:28:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5538.978818 secs 2024-07-26T08:28:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:12 smithi163 ceph-mon[36873]: pgmap v3516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5521.236463 secs 2024-07-26T08:28:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5538.978818 secs 2024-07-26T08:28:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:14 smithi042 ceph-mon[32775]: pgmap v3517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:14 smithi164 ceph-mon[36970]: pgmap v3517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:14 smithi163 ceph-mon[36873]: pgmap v3517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:16 smithi042 ceph-mon[32775]: pgmap v3518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:16 smithi164 ceph-mon[36970]: pgmap v3518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:16 smithi163 ceph-mon[36873]: pgmap v3518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5526.236613 secs 2024-07-26T08:28:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5543.978948 secs 2024-07-26T08:28:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5526.236613 secs 2024-07-26T08:28:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5543.978948 secs 2024-07-26T08:28:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5526.236613 secs 2024-07-26T08:28:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5543.978948 secs 2024-07-26T08:28:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:18 smithi042 ceph-mon[32775]: pgmap v3519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:18 smithi164 ceph-mon[36970]: pgmap v3519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:18 smithi163 ceph-mon[36873]: pgmap v3519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:19 smithi164 ceph-mon[36970]: pgmap v3520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:28:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:19 smithi163 ceph-mon[36873]: pgmap v3520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:28:19.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:19 smithi042 ceph-mon[32775]: pgmap v3520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:28:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:21 smithi042 ceph-mon[32775]: pgmap v3521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5531.236683 secs 2024-07-26T08:28:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:21 smithi164 ceph-mon[36970]: pgmap v3521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5531.236683 secs 2024-07-26T08:28:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:21 smithi163 ceph-mon[36873]: pgmap v3521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5531.236683 secs 2024-07-26T08:28:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5548.979071 secs 2024-07-26T08:28:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5548.979071 secs 2024-07-26T08:28:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5548.979071 secs 2024-07-26T08:28:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:23 smithi042 ceph-mon[32775]: pgmap v3522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:23 smithi164 ceph-mon[36970]: pgmap v3522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:23 smithi163 ceph-mon[36873]: pgmap v3522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:25.182 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:28:25.180+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:28:25.182 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:28:25.181+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:28:25.182 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:28:25.181+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:28:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:25 smithi042 ceph-mon[32775]: pgmap v3523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:25 smithi164 ceph-mon[36970]: pgmap v3523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:25 smithi163 ceph-mon[36873]: pgmap v3523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5536.236830 secs 2024-07-26T08:28:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5536.236830 secs 2024-07-26T08:28:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5536.236830 secs 2024-07-26T08:28:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5553.979184 secs 2024-07-26T08:28:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:27 smithi042 ceph-mon[32775]: pgmap v3524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:27.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5553.979184 secs 2024-07-26T08:28:27.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:27 smithi163 ceph-mon[36873]: pgmap v3524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:27.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5553.979184 secs 2024-07-26T08:28:27.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:27 smithi164 ceph-mon[36970]: pgmap v3524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:29 smithi042 ceph-mon[32775]: pgmap v3525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:28:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:29 smithi164 ceph-mon[36970]: pgmap v3525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:28:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:29 smithi163 ceph-mon[36873]: pgmap v3525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:28:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:31 smithi042 ceph-mon[32775]: pgmap v3526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5541.236951 secs 2024-07-26T08:28:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:31 smithi164 ceph-mon[36970]: pgmap v3526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5541.236951 secs 2024-07-26T08:28:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:31 smithi163 ceph-mon[36873]: pgmap v3526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5541.236951 secs 2024-07-26T08:28:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5558.979253 secs 2024-07-26T08:28:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5558.979253 secs 2024-07-26T08:28:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5558.979253 secs 2024-07-26T08:28:34.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:33 smithi042 ceph-mon[32775]: pgmap v3527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:33 smithi164 ceph-mon[36970]: pgmap v3527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:33 smithi163 ceph-mon[36873]: pgmap v3527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:35 smithi042 ceph-mon[32775]: pgmap v3528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:35 smithi164 ceph-mon[36970]: pgmap v3528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:35 smithi163 ceph-mon[36873]: pgmap v3528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5546.237068 secs 2024-07-26T08:28:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5546.237068 secs 2024-07-26T08:28:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5546.237068 secs 2024-07-26T08:28:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5563.979381 secs 2024-07-26T08:28:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:37 smithi042 ceph-mon[32775]: pgmap v3529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5563.979381 secs 2024-07-26T08:28:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:37 smithi164 ceph-mon[36970]: pgmap v3529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5563.979381 secs 2024-07-26T08:28:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:37 smithi163 ceph-mon[36873]: pgmap v3529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:39 smithi042 ceph-mon[32775]: pgmap v3530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:28:40.191 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:28:40.190+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:28:40.192 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:28:40.190+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:28:40.192 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:28:40.190+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:28:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:39 smithi164 ceph-mon[36970]: pgmap v3530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:28:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:39 smithi163 ceph-mon[36873]: pgmap v3530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:28:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:41 smithi042 ceph-mon[32775]: pgmap v3531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5551.237136 secs 2024-07-26T08:28:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:41 smithi164 ceph-mon[36970]: pgmap v3531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5551.237136 secs 2024-07-26T08:28:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:41 smithi163 ceph-mon[36873]: pgmap v3531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5551.237136 secs 2024-07-26T08:28:43.048 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5568.979509 secs 2024-07-26T08:28:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5568.979509 secs 2024-07-26T08:28:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5568.979509 secs 2024-07-26T08:28:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:43 smithi042 ceph-mon[32775]: pgmap v3532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:43 smithi164 ceph-mon[36970]: pgmap v3532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:43 smithi163 ceph-mon[36873]: pgmap v3532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:46.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:45 smithi042 ceph-mon[32775]: pgmap v3533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:45 smithi164 ceph-mon[36970]: pgmap v3533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:45 smithi163 ceph-mon[36873]: pgmap v3533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5556.237275 secs 2024-07-26T08:28:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5556.237275 secs 2024-07-26T08:28:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5556.237275 secs 2024-07-26T08:28:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:28:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5573.979609 secs 2024-07-26T08:28:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:47 smithi042 ceph-mon[32775]: pgmap v3534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:47 smithi042 ceph-mon[32775]: pgmap v3535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:28:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:47 smithi042 ceph-mon[32775]: pgmap v3536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:28:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:28:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5573.979609 secs 2024-07-26T08:28:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:47 smithi164 ceph-mon[36970]: pgmap v3534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:47 smithi164 ceph-mon[36970]: pgmap v3535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:28:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:47 smithi164 ceph-mon[36970]: pgmap v3536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:28:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:28:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5573.979609 secs 2024-07-26T08:28:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:47 smithi163 ceph-mon[36873]: pgmap v3534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:28:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:28:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:28:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:47 smithi163 ceph-mon[36873]: pgmap v3535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:28:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:47 smithi163 ceph-mon[36873]: pgmap v3536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:28:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:28:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:49 smithi164 ceph-mon[36970]: pgmap v3537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:28:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:49 smithi163 ceph-mon[36873]: pgmap v3537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:28:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:49 smithi042 ceph-mon[32775]: pgmap v3537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:28:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:51 smithi164 ceph-mon[36970]: pgmap v3538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:28:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:51 smithi163 ceph-mon[36873]: pgmap v3538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:28:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:51 smithi042 ceph-mon[32775]: pgmap v3538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:28:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5561.237379 secs 2024-07-26T08:28:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5578.979736 secs 2024-07-26T08:28:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5561.237379 secs 2024-07-26T08:28:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5578.979736 secs 2024-07-26T08:28:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5561.237379 secs 2024-07-26T08:28:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5578.979736 secs 2024-07-26T08:28:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:53 smithi164 ceph-mon[36970]: pgmap v3539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:28:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:53 smithi163 ceph-mon[36873]: pgmap v3539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:28:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:53 smithi042 ceph-mon[32775]: pgmap v3539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:28:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:28:55.194+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:28:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:28:55.194+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:28:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:28:55.194+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:28:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:55 smithi164 ceph-mon[36970]: pgmap v3540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:28:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:55 smithi163 ceph-mon[36873]: pgmap v3540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:28:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:55 smithi042 ceph-mon[32775]: pgmap v3540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:28:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5566.237511 secs 2024-07-26T08:28:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5566.237511 secs 2024-07-26T08:28:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5566.237511 secs 2024-07-26T08:28:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5583.979853 secs 2024-07-26T08:28:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:57 smithi042 ceph-mon[32775]: pgmap v3541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:28:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5583.979853 secs 2024-07-26T08:28:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:57 smithi164 ceph-mon[36970]: pgmap v3541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:28:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5583.979853 secs 2024-07-26T08:28:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:57 smithi163 ceph-mon[36873]: pgmap v3541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:28:58.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:28:58.738+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:28:58.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:28:58.738+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:28:58.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:28:58.738+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:28:58.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:28:58.739+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:28:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:28:59 smithi164 ceph-mon[36970]: pgmap v3542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:28:59 smithi163 ceph-mon[36873]: pgmap v3542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:28:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:28:59 smithi042 ceph-mon[32775]: pgmap v3542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:01 smithi164 ceph-mon[36970]: pgmap v3543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:01 smithi163 ceph-mon[36873]: pgmap v3543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:01 smithi042 ceph-mon[32775]: pgmap v3543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5571.237631 secs 2024-07-26T08:29:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5588.980009 secs 2024-07-26T08:29:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5571.237631 secs 2024-07-26T08:29:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5588.980009 secs 2024-07-26T08:29:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5571.237631 secs 2024-07-26T08:29:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5588.980009 secs 2024-07-26T08:29:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:03 smithi164 ceph-mon[36970]: pgmap v3544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:03 smithi163 ceph-mon[36873]: pgmap v3544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:03 smithi042 ceph-mon[32775]: pgmap v3544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:05 smithi164 ceph-mon[36970]: pgmap v3545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:05.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:05 smithi163 ceph-mon[36873]: pgmap v3545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:05 smithi042 ceph-mon[32775]: pgmap v3545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5576.237776 secs 2024-07-26T08:29:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5576.237776 secs 2024-07-26T08:29:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5576.237776 secs 2024-07-26T08:29:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5593.980130 secs 2024-07-26T08:29:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:07 smithi042 ceph-mon[32775]: pgmap v3546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5593.980130 secs 2024-07-26T08:29:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:07 smithi164 ceph-mon[36970]: pgmap v3546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5593.980130 secs 2024-07-26T08:29:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:07 smithi163 ceph-mon[36873]: pgmap v3546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:09 smithi164 ceph-mon[36970]: pgmap v3547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:09 smithi163 ceph-mon[36873]: pgmap v3547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:09 smithi042 ceph-mon[32775]: pgmap v3547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:29:10.195+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:29:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:29:10.195+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:29:10.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:29:10.195+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:29:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:11 smithi164 ceph-mon[36970]: pgmap v3548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:11 smithi163 ceph-mon[36873]: pgmap v3548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:11 smithi042 ceph-mon[32775]: pgmap v3548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5581.237890 secs 2024-07-26T08:29:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5598.980251 secs 2024-07-26T08:29:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5581.237890 secs 2024-07-26T08:29:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5598.980251 secs 2024-07-26T08:29:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5581.237890 secs 2024-07-26T08:29:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5598.980251 secs 2024-07-26T08:29:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:13 smithi164 ceph-mon[36970]: pgmap v3549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:13 smithi163 ceph-mon[36873]: pgmap v3549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:13.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:13 smithi042 ceph-mon[32775]: pgmap v3549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:15 smithi164 ceph-mon[36970]: pgmap v3550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:15 smithi163 ceph-mon[36873]: pgmap v3550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:15 smithi042 ceph-mon[32775]: pgmap v3550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5586.238032 secs 2024-07-26T08:29:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5586.238032 secs 2024-07-26T08:29:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5586.238032 secs 2024-07-26T08:29:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5603.980382 secs 2024-07-26T08:29:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:17 smithi042 ceph-mon[32775]: pgmap v3551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5603.980382 secs 2024-07-26T08:29:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:17 smithi164 ceph-mon[36970]: pgmap v3551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5603.980382 secs 2024-07-26T08:29:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:17 smithi163 ceph-mon[36873]: pgmap v3551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:19 smithi164 ceph-mon[36970]: pgmap v3552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:19 smithi163 ceph-mon[36873]: pgmap v3552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:19.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:19 smithi042 ceph-mon[32775]: pgmap v3552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:21 smithi164 ceph-mon[36970]: pgmap v3553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:21 smithi163 ceph-mon[36873]: pgmap v3553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:21 smithi042 ceph-mon[32775]: pgmap v3553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5591.238154 secs 2024-07-26T08:29:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5608.980505 secs 2024-07-26T08:29:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5591.238154 secs 2024-07-26T08:29:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5608.980505 secs 2024-07-26T08:29:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5591.238154 secs 2024-07-26T08:29:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5608.980505 secs 2024-07-26T08:29:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:23 smithi164 ceph-mon[36970]: pgmap v3554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:23 smithi163 ceph-mon[36873]: pgmap v3554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:23 smithi042 ceph-mon[32775]: pgmap v3554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:29:25.199+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:29:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:29:25.199+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:29:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:29:25.199+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:29:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:25 smithi164 ceph-mon[36970]: pgmap v3555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:25 smithi163 ceph-mon[36873]: pgmap v3555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:25.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:25 smithi042 ceph-mon[32775]: pgmap v3555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5596.238275 secs 2024-07-26T08:29:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5596.238275 secs 2024-07-26T08:29:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5596.238275 secs 2024-07-26T08:29:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5613.980621 secs 2024-07-26T08:29:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:27 smithi042 ceph-mon[32775]: pgmap v3556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5613.980621 secs 2024-07-26T08:29:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:27 smithi164 ceph-mon[36970]: pgmap v3556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5613.980621 secs 2024-07-26T08:29:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:27 smithi163 ceph-mon[36873]: pgmap v3556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:29 smithi164 ceph-mon[36970]: pgmap v3557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:29 smithi163 ceph-mon[36873]: pgmap v3557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:29 smithi042 ceph-mon[32775]: pgmap v3557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:31 smithi164 ceph-mon[36970]: pgmap v3558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:31 smithi163 ceph-mon[36873]: pgmap v3558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:31 smithi042 ceph-mon[32775]: pgmap v3558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5601.238399 secs 2024-07-26T08:29:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5618.980718 secs 2024-07-26T08:29:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5601.238399 secs 2024-07-26T08:29:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5618.980718 secs 2024-07-26T08:29:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5601.238399 secs 2024-07-26T08:29:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5618.980718 secs 2024-07-26T08:29:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:33 smithi164 ceph-mon[36970]: pgmap v3559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:33 smithi163 ceph-mon[36873]: pgmap v3559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:33.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:33 smithi042 ceph-mon[32775]: pgmap v3559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:35 smithi164 ceph-mon[36970]: pgmap v3560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:35 smithi163 ceph-mon[36873]: pgmap v3560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:35.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:35 smithi042 ceph-mon[32775]: pgmap v3560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:36.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5606.238526 secs 2024-07-26T08:29:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5606.238526 secs 2024-07-26T08:29:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5606.238526 secs 2024-07-26T08:29:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5623.980840 secs 2024-07-26T08:29:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:37 smithi042 ceph-mon[32775]: pgmap v3561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5623.980840 secs 2024-07-26T08:29:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:37 smithi164 ceph-mon[36970]: pgmap v3561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5623.980840 secs 2024-07-26T08:29:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:37 smithi163 ceph-mon[36873]: pgmap v3561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:39 smithi164 ceph-mon[36970]: pgmap v3562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:39 smithi163 ceph-mon[36873]: pgmap v3562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:39 smithi042 ceph-mon[32775]: pgmap v3562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:40.210 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:29:40.208+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:29:40.210 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:29:40.208+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:29:40.210 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:29:40.208+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:29:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:41 smithi164 ceph-mon[36970]: pgmap v3563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:41 smithi163 ceph-mon[36873]: pgmap v3563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:41 smithi042 ceph-mon[32775]: pgmap v3563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5611.238650 secs 2024-07-26T08:29:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5628.980956 secs 2024-07-26T08:29:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5611.238650 secs 2024-07-26T08:29:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5628.980956 secs 2024-07-26T08:29:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5611.238650 secs 2024-07-26T08:29:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5628.980956 secs 2024-07-26T08:29:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:43 smithi164 ceph-mon[36970]: pgmap v3564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:43 smithi163 ceph-mon[36873]: pgmap v3564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:43.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:43 smithi042 ceph-mon[32775]: pgmap v3564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:45 smithi164 ceph-mon[36970]: pgmap v3565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:45 smithi163 ceph-mon[36873]: pgmap v3565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:45 smithi042 ceph-mon[32775]: pgmap v3565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:29:46.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5616.238750 secs 2024-07-26T08:29:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5616.238750 secs 2024-07-26T08:29:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5616.238750 secs 2024-07-26T08:29:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5633.981076 secs 2024-07-26T08:29:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:47 smithi042 ceph-mon[32775]: pgmap v3566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5633.981076 secs 2024-07-26T08:29:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:47 smithi164 ceph-mon[36970]: pgmap v3566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5633.981076 secs 2024-07-26T08:29:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:47 smithi163 ceph-mon[36873]: pgmap v3566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:29:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:29:48.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:48 smithi042 ceph-mon[32775]: pgmap v3567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:29:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:48 smithi042 ceph-mon[32775]: pgmap v3568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:29:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:29:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:48 smithi164 ceph-mon[36970]: pgmap v3567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:29:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:48 smithi164 ceph-mon[36970]: pgmap v3568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:29:48.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:29:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:29:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:29:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:48 smithi163 ceph-mon[36873]: pgmap v3567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:29:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:48 smithi163 ceph-mon[36873]: pgmap v3568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:29:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:29:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:50 smithi164 ceph-mon[36970]: pgmap v3569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:29:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:50 smithi163 ceph-mon[36873]: pgmap v3569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:29:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:50 smithi042 ceph-mon[32775]: pgmap v3569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:29:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5621.238885 secs 2024-07-26T08:29:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5621.238885 secs 2024-07-26T08:29:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5621.238885 secs 2024-07-26T08:29:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5638.981157 secs 2024-07-26T08:29:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:52 smithi164 ceph-mon[36970]: pgmap v3570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:29:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5638.981157 secs 2024-07-26T08:29:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:52 smithi163 ceph-mon[36873]: pgmap v3570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:29:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5638.981157 secs 2024-07-26T08:29:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:52 smithi042 ceph-mon[32775]: pgmap v3570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:29:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:54 smithi164 ceph-mon[36970]: pgmap v3571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:29:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:54 smithi163 ceph-mon[36873]: pgmap v3571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:29:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:54 smithi042 ceph-mon[32775]: pgmap v3571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:29:55.212 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:29:55.211+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:29:55.213 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:29:55.211+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:29:55.213 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:29:55.211+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:29:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:56 smithi164 ceph-mon[36970]: pgmap v3572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:29:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5626.239013 secs 2024-07-26T08:29:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:56 smithi163 ceph-mon[36873]: pgmap v3572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:29:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5626.239013 secs 2024-07-26T08:29:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:56 smithi042 ceph-mon[32775]: pgmap v3572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:29:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5626.239013 secs 2024-07-26T08:29:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5643.981290 secs 2024-07-26T08:29:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5643.981290 secs 2024-07-26T08:29:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5643.981290 secs 2024-07-26T08:29:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:29:58 smithi163 ceph-mon[36873]: pgmap v3573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:29:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:29:58 smithi042 ceph-mon[32775]: pgmap v3573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:29:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:29:58 smithi164 ceph-mon[36970]: pgmap v3573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:30:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:00 smithi163 ceph-mon[36873]: pgmap v3574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:30:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:00 smithi042 ceph-mon[32775]: pgmap v3574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:30:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:00 smithi164 ceph-mon[36970]: pgmap v3574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:30:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5631.239143 secs 2024-07-26T08:30:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5631.239143 secs 2024-07-26T08:30:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5631.239143 secs 2024-07-26T08:30:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5648.981420 secs 2024-07-26T08:30:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:02 smithi163 ceph-mon[36873]: pgmap v3575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5648.981420 secs 2024-07-26T08:30:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:02 smithi042 ceph-mon[32775]: pgmap v3575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5648.981420 secs 2024-07-26T08:30:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:02 smithi164 ceph-mon[36970]: pgmap v3575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:05.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:04 smithi163 ceph-mon[36873]: pgmap v3576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:30:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:04 smithi042 ceph-mon[32775]: pgmap v3576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:30:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:04 smithi164 ceph-mon[36970]: pgmap v3576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:30:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:06 smithi163 ceph-mon[36873]: pgmap v3577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5636.239287 secs 2024-07-26T08:30:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:06 smithi042 ceph-mon[32775]: pgmap v3577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5636.239287 secs 2024-07-26T08:30:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:06 smithi164 ceph-mon[36970]: pgmap v3577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5636.239287 secs 2024-07-26T08:30:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5653.981541 secs 2024-07-26T08:30:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5653.981541 secs 2024-07-26T08:30:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5653.981541 secs 2024-07-26T08:30:09.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:08 smithi163 ceph-mon[36873]: pgmap v3578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:08 smithi042 ceph-mon[32775]: pgmap v3578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:08 smithi164 ceph-mon[36970]: pgmap v3578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:10.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:30:10.220+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:30:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:30:10.220+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:30:10.222 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:30:10.220+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:30:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:10 smithi163 ceph-mon[36873]: pgmap v3579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:11.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:10 smithi164 ceph-mon[36970]: pgmap v3579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:10 smithi042 ceph-mon[32775]: pgmap v3579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5641.239407 secs 2024-07-26T08:30:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5641.239407 secs 2024-07-26T08:30:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5641.239407 secs 2024-07-26T08:30:13.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5658.981680 secs 2024-07-26T08:30:13.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:12 smithi042 ceph-mon[32775]: pgmap v3580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5658.981680 secs 2024-07-26T08:30:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:12 smithi164 ceph-mon[36970]: pgmap v3580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5658.981680 secs 2024-07-26T08:30:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:12 smithi163 ceph-mon[36873]: pgmap v3580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:14 smithi042 ceph-mon[32775]: pgmap v3581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:30:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:14 smithi164 ceph-mon[36970]: pgmap v3581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:30:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:14 smithi163 ceph-mon[36873]: pgmap v3581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:30:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:16 smithi042 ceph-mon[32775]: pgmap v3582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5646.239539 secs 2024-07-26T08:30:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:16 smithi164 ceph-mon[36970]: pgmap v3582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5646.239539 secs 2024-07-26T08:30:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:16 smithi163 ceph-mon[36873]: pgmap v3582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5646.239539 secs 2024-07-26T08:30:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5663.981778 secs 2024-07-26T08:30:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5663.981778 secs 2024-07-26T08:30:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5663.981778 secs 2024-07-26T08:30:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:18 smithi042 ceph-mon[32775]: pgmap v3583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:18 smithi164 ceph-mon[36970]: pgmap v3583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:18 smithi163 ceph-mon[36873]: pgmap v3583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:20 smithi042 ceph-mon[32775]: pgmap v3584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:20 smithi164 ceph-mon[36970]: pgmap v3584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:20 smithi163 ceph-mon[36873]: pgmap v3584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5651.239630 secs 2024-07-26T08:30:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5651.239630 secs 2024-07-26T08:30:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5651.239630 secs 2024-07-26T08:30:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5668.981880 secs 2024-07-26T08:30:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:22 smithi042 ceph-mon[32775]: pgmap v3585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5668.981880 secs 2024-07-26T08:30:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:22 smithi164 ceph-mon[36970]: pgmap v3585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5668.981880 secs 2024-07-26T08:30:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:22 smithi163 ceph-mon[36873]: pgmap v3585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:24 smithi042 ceph-mon[32775]: pgmap v3586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:30:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:24 smithi164 ceph-mon[36970]: pgmap v3586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:30:25.225 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:30:25.223+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:30:25.225 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:30:25.223+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:30:25.225 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:30:25.223+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:30:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:24 smithi163 ceph-mon[36873]: pgmap v3586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:30:27.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:26 smithi163 ceph-mon[36873]: pgmap v3587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:27.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5656.239754 secs 2024-07-26T08:30:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:26 smithi042 ceph-mon[32775]: pgmap v3587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5656.239754 secs 2024-07-26T08:30:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:26 smithi164 ceph-mon[36970]: pgmap v3587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5656.239754 secs 2024-07-26T08:30:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5673.982008 secs 2024-07-26T08:30:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5673.982008 secs 2024-07-26T08:30:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5673.982008 secs 2024-07-26T08:30:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:28 smithi042 ceph-mon[32775]: pgmap v3588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:28 smithi164 ceph-mon[36970]: pgmap v3588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:28 smithi163 ceph-mon[36873]: pgmap v3588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:30 smithi042 ceph-mon[32775]: pgmap v3589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:30 smithi164 ceph-mon[36970]: pgmap v3589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:30 smithi163 ceph-mon[36873]: pgmap v3589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5661.239870 secs 2024-07-26T08:30:32.203 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5661.239870 secs 2024-07-26T08:30:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5661.239870 secs 2024-07-26T08:30:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5678.982079 secs 2024-07-26T08:30:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:32 smithi042 ceph-mon[32775]: pgmap v3590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5678.982079 secs 2024-07-26T08:30:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:32 smithi164 ceph-mon[36970]: pgmap v3590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5678.982079 secs 2024-07-26T08:30:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:32 smithi163 ceph-mon[36873]: pgmap v3590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:35.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:34 smithi042 ceph-mon[32775]: pgmap v3591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:30:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:34 smithi164 ceph-mon[36970]: pgmap v3591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:30:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:34 smithi163 ceph-mon[36873]: pgmap v3591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:30:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:36 smithi042 ceph-mon[32775]: pgmap v3592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5666.239990 secs 2024-07-26T08:30:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:36 smithi164 ceph-mon[36970]: pgmap v3592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5666.239990 secs 2024-07-26T08:30:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:36 smithi163 ceph-mon[36873]: pgmap v3592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5666.239990 secs 2024-07-26T08:30:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5683.982194 secs 2024-07-26T08:30:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5683.982194 secs 2024-07-26T08:30:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5683.982194 secs 2024-07-26T08:30:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:38 smithi042 ceph-mon[32775]: pgmap v3593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:38 smithi164 ceph-mon[36970]: pgmap v3593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:38 smithi163 ceph-mon[36873]: pgmap v3593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:40.233 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:30:40.232+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:30:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:30:40.232+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:30:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:30:40.232+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:30:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:40 smithi042 ceph-mon[32775]: pgmap v3594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:40 smithi164 ceph-mon[36970]: pgmap v3594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:40 smithi163 ceph-mon[36873]: pgmap v3594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5671.240114 secs 2024-07-26T08:30:42.220 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5671.240114 secs 2024-07-26T08:30:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5671.240114 secs 2024-07-26T08:30:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5688.982288 secs 2024-07-26T08:30:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:42 smithi042 ceph-mon[32775]: pgmap v3595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5688.982288 secs 2024-07-26T08:30:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:42 smithi164 ceph-mon[36970]: pgmap v3595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5688.982288 secs 2024-07-26T08:30:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:42 smithi163 ceph-mon[36873]: pgmap v3595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:44 smithi042 ceph-mon[32775]: pgmap v3596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:30:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:44 smithi164 ceph-mon[36970]: pgmap v3596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:30:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:44 smithi163 ceph-mon[36873]: pgmap v3596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:30:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:46 smithi042 ceph-mon[32775]: pgmap v3597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5676.240273 secs 2024-07-26T08:30:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:46 smithi164 ceph-mon[36970]: pgmap v3597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5676.240273 secs 2024-07-26T08:30:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:46 smithi163 ceph-mon[36873]: pgmap v3597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5676.240273 secs 2024-07-26T08:30:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5693.982429 secs 2024-07-26T08:30:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5693.982429 secs 2024-07-26T08:30:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5693.982429 secs 2024-07-26T08:30:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:30:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:48 smithi042 ceph-mon[32775]: pgmap v3598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:48 smithi164 ceph-mon[36970]: pgmap v3598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:48 smithi163 ceph-mon[36873]: pgmap v3598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:30:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:50 smithi164 ceph-mon[36970]: pgmap v3599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:50 smithi163 ceph-mon[36873]: pgmap v3599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:50 smithi042 ceph-mon[32775]: pgmap v3599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:30:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:30:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:30:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:30:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:51 smithi042 ceph-mon[32775]: pgmap v3600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:30:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:51 smithi042 ceph-mon[32775]: pgmap v3601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:30:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5681.240405 secs 2024-07-26T08:30:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:51 smithi164 ceph-mon[36970]: pgmap v3600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:30:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:51 smithi164 ceph-mon[36970]: pgmap v3601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:30:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5681.240405 secs 2024-07-26T08:30:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:51 smithi163 ceph-mon[36873]: pgmap v3600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:30:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:51 smithi163 ceph-mon[36873]: pgmap v3601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:30:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5681.240405 secs 2024-07-26T08:30:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5698.982528 secs 2024-07-26T08:30:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5698.982528 secs 2024-07-26T08:30:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5698.982528 secs 2024-07-26T08:30:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:53 smithi164 ceph-mon[36970]: pgmap v3602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:30:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:53 smithi163 ceph-mon[36873]: pgmap v3602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:30:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:53 smithi042 ceph-mon[32775]: pgmap v3602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:30:55.238 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:30:55.236+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:30:55.238 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:30:55.236+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:30:55.238 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:30:55.236+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:30:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:55 smithi164 ceph-mon[36970]: pgmap v3603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:30:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:55 smithi163 ceph-mon[36873]: pgmap v3603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:30:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:55 smithi042 ceph-mon[32775]: pgmap v3603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:30:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5686.240533 secs 2024-07-26T08:30:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5686.240533 secs 2024-07-26T08:30:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5686.240533 secs 2024-07-26T08:30:57.924 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5703.982657 secs 2024-07-26T08:30:57.924 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:57 smithi163 ceph-mon[36873]: pgmap v3604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:30:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5703.982657 secs 2024-07-26T08:30:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:57 smithi164 ceph-mon[36970]: pgmap v3604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:30:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5703.982657 secs 2024-07-26T08:30:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:57 smithi042 ceph-mon[32775]: pgmap v3604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T08:30:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:30:59 smithi164 ceph-mon[36970]: pgmap v3605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:31:00.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:30:59 smithi163 ceph-mon[36873]: pgmap v3605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:31:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:30:59 smithi042 ceph-mon[32775]: pgmap v3605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:31:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:01 smithi164 ceph-mon[36970]: pgmap v3606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:31:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5691.240655 secs 2024-07-26T08:31:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:01 smithi163 ceph-mon[36873]: pgmap v3606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:31:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5691.240655 secs 2024-07-26T08:31:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:01 smithi042 ceph-mon[32775]: pgmap v3606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:31:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5691.240655 secs 2024-07-26T08:31:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5708.982778 secs 2024-07-26T08:31:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5708.982778 secs 2024-07-26T08:31:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5708.982778 secs 2024-07-26T08:31:03.944 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:03 smithi042 ceph-mon[32775]: pgmap v3607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:03 smithi164 ceph-mon[36970]: pgmap v3607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:04.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:03 smithi163 ceph-mon[36873]: pgmap v3607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:05 smithi164 ceph-mon[36970]: pgmap v3608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:05 smithi163 ceph-mon[36873]: pgmap v3608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:05 smithi042 ceph-mon[32775]: pgmap v3608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5696.240783 secs 2024-07-26T08:31:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5696.240783 secs 2024-07-26T08:31:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5696.240783 secs 2024-07-26T08:31:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5713.982879 secs 2024-07-26T08:31:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:07 smithi164 ceph-mon[36970]: pgmap v3609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5713.982879 secs 2024-07-26T08:31:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:07 smithi163 ceph-mon[36873]: pgmap v3609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5713.982879 secs 2024-07-26T08:31:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:07 smithi042 ceph-mon[32775]: pgmap v3609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:09.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:09 smithi164 ceph-mon[36970]: pgmap v3610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:09 smithi163 ceph-mon[36873]: pgmap v3610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:09 smithi042 ceph-mon[32775]: pgmap v3610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:31:10.245+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:31:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:31:10.245+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:31:10.247 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:31:10.245+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:31:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:11 smithi164 ceph-mon[36970]: pgmap v3611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5701.240926 secs 2024-07-26T08:31:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:11 smithi163 ceph-mon[36873]: pgmap v3611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5701.240926 secs 2024-07-26T08:31:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:11 smithi042 ceph-mon[32775]: pgmap v3611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5701.240926 secs 2024-07-26T08:31:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5718.983001 secs 2024-07-26T08:31:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5718.983001 secs 2024-07-26T08:31:13.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5718.983001 secs 2024-07-26T08:31:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:13 smithi164 ceph-mon[36970]: pgmap v3612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:13 smithi163 ceph-mon[36873]: pgmap v3612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:13 smithi042 ceph-mon[32775]: pgmap v3612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:15 smithi164 ceph-mon[36970]: pgmap v3613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:15 smithi163 ceph-mon[36873]: pgmap v3613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:15 smithi042 ceph-mon[32775]: pgmap v3613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5706.241071 secs 2024-07-26T08:31:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5706.241071 secs 2024-07-26T08:31:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5706.241071 secs 2024-07-26T08:31:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5723.983124 secs 2024-07-26T08:31:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:17 smithi164 ceph-mon[36970]: pgmap v3614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5723.983124 secs 2024-07-26T08:31:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:17 smithi163 ceph-mon[36873]: pgmap v3614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5723.983124 secs 2024-07-26T08:31:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:17 smithi042 ceph-mon[32775]: pgmap v3614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:19 smithi164 ceph-mon[36970]: pgmap v3615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:20.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:19 smithi163 ceph-mon[36873]: pgmap v3615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:19 smithi042 ceph-mon[32775]: pgmap v3615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:21 smithi163 ceph-mon[36873]: pgmap v3616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5711.241195 secs 2024-07-26T08:31:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:21 smithi042 ceph-mon[32775]: pgmap v3616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5711.241195 secs 2024-07-26T08:31:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:21 smithi164 ceph-mon[36970]: pgmap v3616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5711.241195 secs 2024-07-26T08:31:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5728.983243 secs 2024-07-26T08:31:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5728.983243 secs 2024-07-26T08:31:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5728.983243 secs 2024-07-26T08:31:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:23 smithi163 ceph-mon[36873]: pgmap v3617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:24.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:23 smithi042 ceph-mon[32775]: pgmap v3617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:23 smithi164 ceph-mon[36970]: pgmap v3617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:25.250 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:31:25.248+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:31:25.250 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:31:25.248+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:31:25.250 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:31:25.248+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:31:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:25 smithi163 ceph-mon[36873]: pgmap v3618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:25 smithi042 ceph-mon[32775]: pgmap v3618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:25 smithi164 ceph-mon[36970]: pgmap v3618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5716.241279 secs 2024-07-26T08:31:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5716.241279 secs 2024-07-26T08:31:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5716.241279 secs 2024-07-26T08:31:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5733.983362 secs 2024-07-26T08:31:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:27 smithi163 ceph-mon[36873]: pgmap v3619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5733.983362 secs 2024-07-26T08:31:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:27 smithi042 ceph-mon[32775]: pgmap v3619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:28.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5733.983362 secs 2024-07-26T08:31:28.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:27 smithi164 ceph-mon[36970]: pgmap v3619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:30.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:29 smithi163 ceph-mon[36873]: pgmap v3620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:29 smithi042 ceph-mon[32775]: pgmap v3620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:29 smithi164 ceph-mon[36970]: pgmap v3620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:31 smithi163 ceph-mon[36873]: pgmap v3621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5721.241346 secs 2024-07-26T08:31:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:31 smithi042 ceph-mon[32775]: pgmap v3621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5721.241346 secs 2024-07-26T08:31:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:31 smithi164 ceph-mon[36970]: pgmap v3621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5721.241346 secs 2024-07-26T08:31:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5738.983503 secs 2024-07-26T08:31:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5738.983503 secs 2024-07-26T08:31:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5738.983503 secs 2024-07-26T08:31:34.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:33 smithi042 ceph-mon[32775]: pgmap v3622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:33 smithi164 ceph-mon[36970]: pgmap v3622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:33 smithi163 ceph-mon[36873]: pgmap v3622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:35 smithi042 ceph-mon[32775]: pgmap v3623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:35 smithi164 ceph-mon[36970]: pgmap v3623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:35 smithi163 ceph-mon[36873]: pgmap v3623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5726.241439 secs 2024-07-26T08:31:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5726.241439 secs 2024-07-26T08:31:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5726.241439 secs 2024-07-26T08:31:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5743.983626 secs 2024-07-26T08:31:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:37 smithi042 ceph-mon[32775]: pgmap v3624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5743.983626 secs 2024-07-26T08:31:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:37 smithi164 ceph-mon[36970]: pgmap v3624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5743.983626 secs 2024-07-26T08:31:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:37 smithi163 ceph-mon[36873]: pgmap v3624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:39 smithi042 ceph-mon[32775]: pgmap v3625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:39 smithi164 ceph-mon[36970]: pgmap v3625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:39 smithi163 ceph-mon[36873]: pgmap v3625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:31:40.257+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:31:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:31:40.257+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:31:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:31:40.257+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:31:42.026 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:41 smithi164 ceph-mon[36970]: pgmap v3626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:42.026 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5731.241542 secs 2024-07-26T08:31:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:41 smithi042 ceph-mon[32775]: pgmap v3626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5731.241542 secs 2024-07-26T08:31:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:41 smithi163 ceph-mon[36873]: pgmap v3626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5731.241542 secs 2024-07-26T08:31:43.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5748.983751 secs 2024-07-26T08:31:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5748.983751 secs 2024-07-26T08:31:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5748.983751 secs 2024-07-26T08:31:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:43 smithi042 ceph-mon[32775]: pgmap v3627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:43 smithi164 ceph-mon[36970]: pgmap v3627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:43 smithi163 ceph-mon[36873]: pgmap v3627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:46.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:45 smithi042 ceph-mon[32775]: pgmap v3628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:45 smithi164 ceph-mon[36970]: pgmap v3628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:45 smithi163 ceph-mon[36873]: pgmap v3628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5736.241668 secs 2024-07-26T08:31:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5736.241668 secs 2024-07-26T08:31:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5736.241668 secs 2024-07-26T08:31:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5753.983880 secs 2024-07-26T08:31:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:47 smithi042 ceph-mon[32775]: pgmap v3629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5753.983880 secs 2024-07-26T08:31:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:47 smithi164 ceph-mon[36970]: pgmap v3629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5753.983880 secs 2024-07-26T08:31:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:47 smithi163 ceph-mon[36873]: pgmap v3629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:50.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:49 smithi042 ceph-mon[32775]: pgmap v3630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:49 smithi164 ceph-mon[36970]: pgmap v3630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:50.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:49 smithi163 ceph-mon[36873]: pgmap v3630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:31:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:31:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:51 smithi042 ceph-mon[32775]: pgmap v3631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5741.241760 secs 2024-07-26T08:31:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:51 smithi042 ceph-mon[32775]: pgmap v3632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T08:31:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:51 smithi042 ceph-mon[32775]: pgmap v3633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:31:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:51 smithi164 ceph-mon[36970]: pgmap v3631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5741.241760 secs 2024-07-26T08:31:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:51 smithi164 ceph-mon[36970]: pgmap v3632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T08:31:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:51 smithi164 ceph-mon[36970]: pgmap v3633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:31:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:51 smithi163 ceph-mon[36873]: pgmap v3631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:31:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5741.241760 secs 2024-07-26T08:31:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:31:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:31:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:51 smithi163 ceph-mon[36873]: pgmap v3632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T08:31:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:51 smithi163 ceph-mon[36873]: pgmap v3633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:31:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:31:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5758.984028 secs 2024-07-26T08:31:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5758.984028 secs 2024-07-26T08:31:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5758.984028 secs 2024-07-26T08:31:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:53 smithi042 ceph-mon[32775]: pgmap v3634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T08:31:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:53 smithi164 ceph-mon[36970]: pgmap v3634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T08:31:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:53 smithi163 ceph-mon[36873]: pgmap v3634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T08:31:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:31:55.261+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:31:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:31:55.261+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:31:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:31:55.261+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:31:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:56 smithi042 ceph-mon[32775]: pgmap v3635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:31:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:56 smithi164 ceph-mon[36970]: pgmap v3635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:31:56.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:56 smithi163 ceph-mon[36873]: pgmap v3635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:31:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5746.241828 secs 2024-07-26T08:31:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:57 smithi042 ceph-mon[32775]: 1 slow requests, 1 included below; oldest blocked for > 5763.984091 secs 2024-07-26T08:31:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:57 smithi042 ceph-mon[32775]: slow request 5763.984091 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:31:57.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5746.241828 secs 2024-07-26T08:31:57.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:57 smithi163 ceph-mon[36873]: 1 slow requests, 1 included below; oldest blocked for > 5763.984091 secs 2024-07-26T08:31:57.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:57 smithi163 ceph-mon[36873]: slow request 5763.984091 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:31:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5746.241828 secs 2024-07-26T08:31:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:57 smithi164 ceph-mon[36970]: 1 slow requests, 1 included below; oldest blocked for > 5763.984091 secs 2024-07-26T08:31:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:57 smithi164 ceph-mon[36970]: slow request 5763.984091 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T08:31:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:31:58 smithi042 ceph-mon[32775]: pgmap v3636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:31:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:31:58 smithi164 ceph-mon[36970]: pgmap v3636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:31:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:31:58 smithi163 ceph-mon[36873]: pgmap v3636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:31:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:31:58.765+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:31:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:31:58.765+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:31:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:31:58.765+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:31:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:31:58.765+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:32:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:00 smithi042 ceph-mon[32775]: pgmap v3637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:32:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:00 smithi164 ceph-mon[36970]: pgmap v3637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:32:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:00 smithi163 ceph-mon[36873]: pgmap v3637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:32:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5751.241957 secs 2024-07-26T08:32:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:02 smithi042 ceph-mon[32775]: pgmap v3638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:32:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5768.984230 secs 2024-07-26T08:32:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5751.241957 secs 2024-07-26T08:32:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:02 smithi164 ceph-mon[36970]: pgmap v3638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:32:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5768.984230 secs 2024-07-26T08:32:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5751.241957 secs 2024-07-26T08:32:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:02 smithi163 ceph-mon[36873]: pgmap v3638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:32:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5768.984230 secs 2024-07-26T08:32:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:04 smithi042 ceph-mon[32775]: pgmap v3639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:04 smithi164 ceph-mon[36970]: pgmap v3639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:04.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:04 smithi163 ceph-mon[36873]: pgmap v3639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:06 smithi042 ceph-mon[32775]: pgmap v3640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:06 smithi164 ceph-mon[36970]: pgmap v3640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:06 smithi163 ceph-mon[36873]: pgmap v3640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5756.242078 secs 2024-07-26T08:32:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5773.984307 secs 2024-07-26T08:32:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5756.242078 secs 2024-07-26T08:32:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5773.984307 secs 2024-07-26T08:32:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5756.242078 secs 2024-07-26T08:32:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5773.984307 secs 2024-07-26T08:32:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:08 smithi042 ceph-mon[32775]: pgmap v3641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:08 smithi164 ceph-mon[36970]: pgmap v3641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:08 smithi163 ceph-mon[36873]: pgmap v3641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:10.267 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:32:10.266+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:32:10.268 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:32:10.266+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:32:10.268 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:32:10.266+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:32:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:10 smithi042 ceph-mon[32775]: pgmap v3642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:10 smithi164 ceph-mon[36970]: pgmap v3642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:10 smithi163 ceph-mon[36873]: pgmap v3642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:12 smithi042 ceph-mon[32775]: 1 slow requests, 1 included below; oldest blocked for > 5761.242239 secs 2024-07-26T08:32:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:12 smithi042 ceph-mon[32775]: slow request 5761.242238 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T08:32:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:12 smithi042 ceph-mon[32775]: pgmap v3643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5778.984439 secs 2024-07-26T08:32:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:12 smithi164 ceph-mon[36970]: 1 slow requests, 1 included below; oldest blocked for > 5761.242239 secs 2024-07-26T08:32:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:12 smithi164 ceph-mon[36970]: slow request 5761.242238 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T08:32:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:12 smithi164 ceph-mon[36970]: pgmap v3643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5778.984439 secs 2024-07-26T08:32:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:12 smithi163 ceph-mon[36873]: 1 slow requests, 1 included below; oldest blocked for > 5761.242239 secs 2024-07-26T08:32:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:12 smithi163 ceph-mon[36873]: slow request 5761.242238 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T08:32:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:12 smithi163 ceph-mon[36873]: pgmap v3643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5778.984439 secs 2024-07-26T08:32:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:14 smithi042 ceph-mon[32775]: pgmap v3644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:14 smithi164 ceph-mon[36970]: pgmap v3644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:14 smithi163 ceph-mon[36873]: pgmap v3644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:16 smithi042 ceph-mon[32775]: pgmap v3645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:16 smithi164 ceph-mon[36970]: pgmap v3645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:16 smithi163 ceph-mon[36873]: pgmap v3645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5766.242348 secs 2024-07-26T08:32:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5783.984590 secs 2024-07-26T08:32:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5766.242348 secs 2024-07-26T08:32:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5783.984590 secs 2024-07-26T08:32:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5766.242348 secs 2024-07-26T08:32:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5783.984590 secs 2024-07-26T08:32:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:18 smithi042 ceph-mon[32775]: pgmap v3646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:18 smithi164 ceph-mon[36970]: pgmap v3646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:18 smithi163 ceph-mon[36873]: pgmap v3646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:20 smithi042 ceph-mon[32775]: pgmap v3647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:20 smithi164 ceph-mon[36970]: pgmap v3647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:20.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:20 smithi163 ceph-mon[36873]: pgmap v3647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5771.242416 secs 2024-07-26T08:32:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:22 smithi042 ceph-mon[32775]: pgmap v3648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5788.984735 secs 2024-07-26T08:32:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5771.242416 secs 2024-07-26T08:32:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:22 smithi164 ceph-mon[36970]: pgmap v3648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5788.984735 secs 2024-07-26T08:32:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5771.242416 secs 2024-07-26T08:32:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:22 smithi163 ceph-mon[36873]: pgmap v3648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5788.984735 secs 2024-07-26T08:32:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:24 smithi042 ceph-mon[32775]: pgmap v3649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:24 smithi164 ceph-mon[36970]: pgmap v3649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:24 smithi163 ceph-mon[36873]: pgmap v3649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:25.270 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:32:25.268+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:32:25.270 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:32:25.268+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:32:25.270 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:32:25.268+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:32:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:26 smithi042 ceph-mon[32775]: pgmap v3650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:26 smithi164 ceph-mon[36970]: pgmap v3650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:26 smithi163 ceph-mon[36873]: pgmap v3650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5776.242515 secs 2024-07-26T08:32:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5793.984829 secs 2024-07-26T08:32:27.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5776.242515 secs 2024-07-26T08:32:27.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5793.984829 secs 2024-07-26T08:32:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5776.242515 secs 2024-07-26T08:32:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5793.984829 secs 2024-07-26T08:32:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:28 smithi042 ceph-mon[32775]: pgmap v3651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:28 smithi164 ceph-mon[36970]: pgmap v3651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:28 smithi163 ceph-mon[36873]: pgmap v3651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:30 smithi042 ceph-mon[32775]: pgmap v3652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:30 smithi164 ceph-mon[36970]: pgmap v3652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:30 smithi163 ceph-mon[36873]: pgmap v3652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5781.242593 secs 2024-07-26T08:32:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:32 smithi042 ceph-mon[32775]: pgmap v3653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5798.984969 secs 2024-07-26T08:32:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5781.242593 secs 2024-07-26T08:32:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:32 smithi164 ceph-mon[36970]: pgmap v3653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5798.984969 secs 2024-07-26T08:32:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5781.242593 secs 2024-07-26T08:32:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:32 smithi163 ceph-mon[36873]: pgmap v3653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5798.984969 secs 2024-07-26T08:32:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:34 smithi042 ceph-mon[32775]: pgmap v3654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:34 smithi164 ceph-mon[36970]: pgmap v3654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:34 smithi163 ceph-mon[36873]: pgmap v3654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:36 smithi042 ceph-mon[32775]: pgmap v3655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:36 smithi164 ceph-mon[36970]: pgmap v3655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:36 smithi163 ceph-mon[36873]: pgmap v3655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5786.242716 secs 2024-07-26T08:32:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5803.985066 secs 2024-07-26T08:32:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5786.242716 secs 2024-07-26T08:32:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5803.985066 secs 2024-07-26T08:32:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5786.242716 secs 2024-07-26T08:32:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5803.985066 secs 2024-07-26T08:32:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:38 smithi164 ceph-mon[36970]: pgmap v3656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:38.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:38 smithi163 ceph-mon[36873]: pgmap v3656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:38 smithi042 ceph-mon[32775]: pgmap v3656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:32:40.277+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:32:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:32:40.277+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:32:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:32:40.277+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:32:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:40 smithi164 ceph-mon[36970]: pgmap v3657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:40 smithi163 ceph-mon[36873]: pgmap v3657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:40 smithi042 ceph-mon[32775]: pgmap v3657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:41 smithi164 ceph-mon[36970]: pgmap v3658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:41 smithi163 ceph-mon[36873]: pgmap v3658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:41 smithi042 ceph-mon[32775]: pgmap v3658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5791.242820 secs 2024-07-26T08:32:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5808.985191 secs 2024-07-26T08:32:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5791.242820 secs 2024-07-26T08:32:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5808.985191 secs 2024-07-26T08:32:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5791.242820 secs 2024-07-26T08:32:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5808.985191 secs 2024-07-26T08:32:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:43 smithi164 ceph-mon[36970]: pgmap v3659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:43 smithi163 ceph-mon[36873]: pgmap v3659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:43 smithi042 ceph-mon[32775]: pgmap v3659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:46.348 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:46 smithi042 ceph-mon[32775]: pgmap v3660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:46 smithi164 ceph-mon[36970]: pgmap v3660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:46 smithi163 ceph-mon[36873]: pgmap v3660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5796.242947 secs 2024-07-26T08:32:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5813.985292 secs 2024-07-26T08:32:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5796.242947 secs 2024-07-26T08:32:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5813.985292 secs 2024-07-26T08:32:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5796.242947 secs 2024-07-26T08:32:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5813.985292 secs 2024-07-26T08:32:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:48 smithi042 ceph-mon[32775]: pgmap v3661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:48 smithi164 ceph-mon[36970]: pgmap v3661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:48 smithi163 ceph-mon[36873]: pgmap v3661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:50 smithi042 ceph-mon[32775]: pgmap v3662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:50 smithi164 ceph-mon[36970]: pgmap v3662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:50 smithi163 ceph-mon[36873]: pgmap v3662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:32:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5801.243068 secs 2024-07-26T08:32:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:52 smithi042 ceph-mon[32775]: pgmap v3663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:32:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5818.985367 secs 2024-07-26T08:32:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:32:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:32:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5801.243068 secs 2024-07-26T08:32:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:52 smithi164 ceph-mon[36970]: pgmap v3663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:32:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5818.985367 secs 2024-07-26T08:32:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:32:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:32:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5801.243068 secs 2024-07-26T08:32:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:52 smithi163 ceph-mon[36873]: pgmap v3663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:32:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:32:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5818.985367 secs 2024-07-26T08:32:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:32:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:32:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:32:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:53 smithi042 ceph-mon[32775]: pgmap v3664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:32:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:53 smithi042 ceph-mon[32775]: pgmap v3665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:32:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:53 smithi164 ceph-mon[36970]: pgmap v3664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:32:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:53 smithi164 ceph-mon[36970]: pgmap v3665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:32:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:53 smithi163 ceph-mon[36873]: pgmap v3664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:32:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:53 smithi163 ceph-mon[36873]: pgmap v3665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:32:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:32:55.281+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:32:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:32:55.281+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:32:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:32:55.281+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:32:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:55 smithi042 ceph-mon[32775]: pgmap v3666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:32:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:55 smithi164 ceph-mon[36970]: pgmap v3666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:32:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:55 smithi163 ceph-mon[36873]: pgmap v3666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:32:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:57 smithi042 ceph-mon[32775]: pgmap v3667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:32:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5806.243190 secs 2024-07-26T08:32:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5823.985491 secs 2024-07-26T08:32:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:57 smithi164 ceph-mon[36970]: pgmap v3667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:32:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5806.243190 secs 2024-07-26T08:32:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5823.985491 secs 2024-07-26T08:32:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:57 smithi163 ceph-mon[36873]: pgmap v3667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:32:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5806.243190 secs 2024-07-26T08:32:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5823.985491 secs 2024-07-26T08:32:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:32:59 smithi042 ceph-mon[32775]: pgmap v3668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:32:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:32:59 smithi164 ceph-mon[36970]: pgmap v3668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:32:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:32:59 smithi163 ceph-mon[36873]: pgmap v3668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:33:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:01 smithi042 ceph-mon[32775]: pgmap v3669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:33:01.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:01 smithi164 ceph-mon[36970]: pgmap v3669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:33:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:01 smithi163 ceph-mon[36873]: pgmap v3669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:33:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5811.243312 secs 2024-07-26T08:33:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5828.985614 secs 2024-07-26T08:33:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5811.243312 secs 2024-07-26T08:33:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5828.985614 secs 2024-07-26T08:33:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5811.243312 secs 2024-07-26T08:33:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5828.985614 secs 2024-07-26T08:33:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:03 smithi042 ceph-mon[32775]: pgmap v3670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:33:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:03 smithi164 ceph-mon[36970]: pgmap v3670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:33:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:03 smithi163 ceph-mon[36873]: pgmap v3670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:33:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:05 smithi042 ceph-mon[32775]: pgmap v3671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:33:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:05 smithi164 ceph-mon[36970]: pgmap v3671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:33:05.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:05 smithi163 ceph-mon[36873]: pgmap v3671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:33:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:07 smithi042 ceph-mon[32775]: pgmap v3672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5816.243398 secs 2024-07-26T08:33:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5833.985711 secs 2024-07-26T08:33:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:07 smithi164 ceph-mon[36970]: pgmap v3672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5816.243398 secs 2024-07-26T08:33:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5833.985711 secs 2024-07-26T08:33:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:07 smithi163 ceph-mon[36873]: pgmap v3672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5816.243398 secs 2024-07-26T08:33:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5833.985711 secs 2024-07-26T08:33:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:09 smithi042 ceph-mon[32775]: pgmap v3673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:09 smithi164 ceph-mon[36970]: pgmap v3673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:09 smithi163 ceph-mon[36873]: pgmap v3673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:10.290 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:33:10.289+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:33:10.290 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:33:10.289+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:33:10.290 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:33:10.289+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:33:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:11 smithi042 ceph-mon[32775]: pgmap v3674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:11 smithi164 ceph-mon[36970]: pgmap v3674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:11 smithi163 ceph-mon[36873]: pgmap v3674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5821.243480 secs 2024-07-26T08:33:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5838.985792 secs 2024-07-26T08:33:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5821.243480 secs 2024-07-26T08:33:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5838.985792 secs 2024-07-26T08:33:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5821.243480 secs 2024-07-26T08:33:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5838.985792 secs 2024-07-26T08:33:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:13 smithi042 ceph-mon[32775]: pgmap v3675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:13.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:13 smithi164 ceph-mon[36970]: pgmap v3675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:13 smithi163 ceph-mon[36873]: pgmap v3675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:15 smithi042 ceph-mon[32775]: pgmap v3676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:33:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:15 smithi164 ceph-mon[36970]: pgmap v3676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:33:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:15 smithi163 ceph-mon[36873]: pgmap v3676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:33:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:17 smithi164 ceph-mon[36970]: pgmap v3677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5826.243604 secs 2024-07-26T08:33:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5843.985913 secs 2024-07-26T08:33:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:17 smithi163 ceph-mon[36873]: pgmap v3677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5826.243604 secs 2024-07-26T08:33:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5843.985913 secs 2024-07-26T08:33:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:17 smithi042 ceph-mon[32775]: pgmap v3677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5826.243604 secs 2024-07-26T08:33:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5843.985913 secs 2024-07-26T08:33:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:19 smithi164 ceph-mon[36970]: pgmap v3678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:19 smithi163 ceph-mon[36873]: pgmap v3678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:19 smithi042 ceph-mon[32775]: pgmap v3678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:21 smithi164 ceph-mon[36970]: pgmap v3679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:21 smithi163 ceph-mon[36873]: pgmap v3679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:21 smithi042 ceph-mon[32775]: pgmap v3679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5831.243702 secs 2024-07-26T08:33:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5848.986035 secs 2024-07-26T08:33:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5831.243702 secs 2024-07-26T08:33:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5848.986035 secs 2024-07-26T08:33:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5831.243702 secs 2024-07-26T08:33:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5848.986035 secs 2024-07-26T08:33:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:23 smithi042 ceph-mon[32775]: pgmap v3680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:23 smithi164 ceph-mon[36970]: pgmap v3680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:23 smithi163 ceph-mon[36873]: pgmap v3680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:24 smithi164 ceph-mon[36970]: pgmap v3681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:33:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:24 smithi163 ceph-mon[36873]: pgmap v3681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:33:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:24 smithi042 ceph-mon[32775]: pgmap v3681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:33:25.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:33:25.292+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:33:25.294 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:33:25.292+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:33:25.294 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:33:25.292+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:33:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:26 smithi164 ceph-mon[36970]: pgmap v3682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5836.243786 secs 2024-07-26T08:33:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:26 smithi163 ceph-mon[36873]: pgmap v3682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5836.243786 secs 2024-07-26T08:33:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:26 smithi042 ceph-mon[32775]: pgmap v3682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5836.243786 secs 2024-07-26T08:33:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5853.986137 secs 2024-07-26T08:33:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5853.986137 secs 2024-07-26T08:33:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5853.986137 secs 2024-07-26T08:33:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:28 smithi164 ceph-mon[36970]: pgmap v3683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:29.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:28 smithi163 ceph-mon[36873]: pgmap v3683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:28 smithi042 ceph-mon[32775]: pgmap v3683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:30 smithi164 ceph-mon[36970]: pgmap v3684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:31.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:30 smithi163 ceph-mon[36873]: pgmap v3684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:30 smithi042 ceph-mon[32775]: pgmap v3684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5841.243926 secs 2024-07-26T08:33:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5841.243926 secs 2024-07-26T08:33:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5841.243926 secs 2024-07-26T08:33:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5858.986261 secs 2024-07-26T08:33:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:32 smithi164 ceph-mon[36970]: pgmap v3685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5858.986261 secs 2024-07-26T08:33:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:32 smithi163 ceph-mon[36873]: pgmap v3685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5858.986261 secs 2024-07-26T08:33:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:32 smithi042 ceph-mon[32775]: pgmap v3685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:34.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:34 smithi164 ceph-mon[36970]: pgmap v3686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:33:35.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:34 smithi163 ceph-mon[36873]: pgmap v3686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:33:35.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:34 smithi042 ceph-mon[32775]: pgmap v3686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:33:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:36 smithi164 ceph-mon[36970]: pgmap v3687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5846.244052 secs 2024-07-26T08:33:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:36 smithi163 ceph-mon[36873]: pgmap v3687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5846.244052 secs 2024-07-26T08:33:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:36 smithi042 ceph-mon[32775]: pgmap v3687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:33:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5846.244052 secs 2024-07-26T08:33:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5863.986382 secs 2024-07-26T08:33:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5863.986382 secs 2024-07-26T08:33:38.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5863.986382 secs 2024-07-26T08:33:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:38 smithi164 ceph-mon[36970]: pgmap v3688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:38 smithi163 ceph-mon[36873]: pgmap v3688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:38 smithi042 ceph-mon[32775]: pgmap v3688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:33:40.302+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:33:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:33:40.302+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:33:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:33:40.302+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:33:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:40 smithi164 ceph-mon[36970]: pgmap v3689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:33:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:40 smithi163 ceph-mon[36873]: pgmap v3689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:33:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:40 smithi042 ceph-mon[32775]: pgmap v3689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:33:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5851.244179 secs 2024-07-26T08:33:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5851.244179 secs 2024-07-26T08:33:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5851.244179 secs 2024-07-26T08:33:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5868.986508 secs 2024-07-26T08:33:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:42 smithi164 ceph-mon[36970]: pgmap v3690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:33:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5868.986508 secs 2024-07-26T08:33:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:42 smithi163 ceph-mon[36873]: pgmap v3690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:33:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5868.986508 secs 2024-07-26T08:33:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:42 smithi042 ceph-mon[32775]: pgmap v3690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:33:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:44 smithi164 ceph-mon[36970]: pgmap v3691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:33:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:44 smithi163 ceph-mon[36873]: pgmap v3691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:33:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:44 smithi042 ceph-mon[32775]: pgmap v3691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:33:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:46 smithi164 ceph-mon[36970]: pgmap v3692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:33:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5856.244292 secs 2024-07-26T08:33:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:46 smithi163 ceph-mon[36873]: pgmap v3692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:33:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5856.244292 secs 2024-07-26T08:33:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:46 smithi042 ceph-mon[32775]: pgmap v3692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:33:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5856.244292 secs 2024-07-26T08:33:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5873.986633 secs 2024-07-26T08:33:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5873.986633 secs 2024-07-26T08:33:48.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5873.986633 secs 2024-07-26T08:33:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:48 smithi164 ceph-mon[36970]: pgmap v3693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:33:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:48 smithi163 ceph-mon[36873]: pgmap v3693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:33:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:48 smithi042 ceph-mon[32775]: pgmap v3693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T08:33:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:50 smithi164 ceph-mon[36970]: pgmap v3694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:33:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:50 smithi163 ceph-mon[36873]: pgmap v3694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:33:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:50 smithi042 ceph-mon[32775]: pgmap v3694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:33:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5861.244370 secs 2024-07-26T08:33:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:33:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5861.244370 secs 2024-07-26T08:33:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:33:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5861.244370 secs 2024-07-26T08:33:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:33:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5878.986753 secs 2024-07-26T08:33:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:52 smithi164 ceph-mon[36970]: pgmap v3695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:53.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:33:53.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:33:53.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:52 smithi164 ceph-mon[36970]: pgmap v3696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:33:53.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:52 smithi164 ceph-mon[36970]: pgmap v3697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:33:53.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:33:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5878.986753 secs 2024-07-26T08:33:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:52 smithi163 ceph-mon[36873]: pgmap v3695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:33:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:33:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:52 smithi163 ceph-mon[36873]: pgmap v3696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:33:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:52 smithi163 ceph-mon[36873]: pgmap v3697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:33:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:33:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5878.986753 secs 2024-07-26T08:33:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:52 smithi042 ceph-mon[32775]: pgmap v3695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:33:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:33:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:33:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:52 smithi042 ceph-mon[32775]: pgmap v3696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:33:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:52 smithi042 ceph-mon[32775]: pgmap v3697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:33:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:33:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:54 smithi164 ceph-mon[36970]: pgmap v3698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:33:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:54 smithi163 ceph-mon[36873]: pgmap v3698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:33:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:54 smithi042 ceph-mon[32775]: pgmap v3698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:33:55.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:33:55.305+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:33:55.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:33:55.305+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:33:55.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:33:55.305+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:33:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:56 smithi163 ceph-mon[36873]: pgmap v3699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:33:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:56 smithi042 ceph-mon[32775]: pgmap v3699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:33:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:56 smithi164 ceph-mon[36970]: pgmap v3699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:33:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5866.244468 secs 2024-07-26T08:33:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5883.986871 secs 2024-07-26T08:33:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5866.244468 secs 2024-07-26T08:33:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5883.986871 secs 2024-07-26T08:33:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5866.244468 secs 2024-07-26T08:33:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5883.986871 secs 2024-07-26T08:33:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:33:58 smithi163 ceph-mon[36873]: pgmap v3700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:33:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:33:58 smithi042 ceph-mon[32775]: pgmap v3700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:33:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:33:58 smithi164 ceph-mon[36970]: pgmap v3700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:34:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:00 smithi164 ceph-mon[36970]: pgmap v3701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:34:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:00 smithi163 ceph-mon[36873]: pgmap v3701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:34:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:00 smithi042 ceph-mon[32775]: pgmap v3701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:34:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5871.244599 secs 2024-07-26T08:34:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5871.244599 secs 2024-07-26T08:34:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5871.244599 secs 2024-07-26T08:34:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5888.986992 secs 2024-07-26T08:34:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:02 smithi042 ceph-mon[32775]: pgmap v3702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:34:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5888.986992 secs 2024-07-26T08:34:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:02 smithi164 ceph-mon[36970]: pgmap v3702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:34:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5888.986992 secs 2024-07-26T08:34:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:02 smithi163 ceph-mon[36873]: pgmap v3702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:34:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:04 smithi164 ceph-mon[36970]: pgmap v3703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:04.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:04 smithi163 ceph-mon[36873]: pgmap v3703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:04 smithi042 ceph-mon[32775]: pgmap v3703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:06 smithi163 ceph-mon[36873]: pgmap v3704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:06 smithi042 ceph-mon[32775]: pgmap v3704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:06 smithi164 ceph-mon[36970]: pgmap v3704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5876.244719 secs 2024-07-26T08:34:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5893.987106 secs 2024-07-26T08:34:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5876.244719 secs 2024-07-26T08:34:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5893.987106 secs 2024-07-26T08:34:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5876.244719 secs 2024-07-26T08:34:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5893.987106 secs 2024-07-26T08:34:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:08 smithi163 ceph-mon[36873]: pgmap v3705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:08.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:08 smithi042 ceph-mon[32775]: pgmap v3705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:08 smithi164 ceph-mon[36970]: pgmap v3705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:10.311 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:10.310+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:34:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:10.310+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:34:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:10.310+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:34:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:10 smithi163 ceph-mon[36873]: pgmap v3706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:10 smithi042 ceph-mon[32775]: pgmap v3706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:10 smithi164 ceph-mon[36970]: pgmap v3706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5881.244864 secs 2024-07-26T08:34:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5881.244864 secs 2024-07-26T08:34:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5881.244864 secs 2024-07-26T08:34:12.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5898.987199 secs 2024-07-26T08:34:12.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:12 smithi164 ceph-mon[36970]: pgmap v3707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5898.987199 secs 2024-07-26T08:34:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:12 smithi042 ceph-mon[32775]: pgmap v3707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5898.987199 secs 2024-07-26T08:34:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:12 smithi163 ceph-mon[36873]: pgmap v3707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:14 smithi163 ceph-mon[36873]: pgmap v3708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:14 smithi042 ceph-mon[32775]: pgmap v3708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:14 smithi164 ceph-mon[36970]: pgmap v3708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:16 smithi163 ceph-mon[36873]: pgmap v3709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:16 smithi042 ceph-mon[32775]: pgmap v3709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:16 smithi164 ceph-mon[36970]: pgmap v3709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5886.244963 secs 2024-07-26T08:34:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5903.987292 secs 2024-07-26T08:34:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5886.244963 secs 2024-07-26T08:34:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5903.987292 secs 2024-07-26T08:34:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5886.244963 secs 2024-07-26T08:34:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5903.987292 secs 2024-07-26T08:34:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:18 smithi163 ceph-mon[36873]: pgmap v3710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:18.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:18 smithi042 ceph-mon[32775]: pgmap v3710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:18 smithi164 ceph-mon[36970]: pgmap v3710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:20 smithi163 ceph-mon[36873]: pgmap v3711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:20 smithi042 ceph-mon[32775]: pgmap v3711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:20 smithi164 ceph-mon[36970]: pgmap v3711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5891.245088 secs 2024-07-26T08:34:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5891.245088 secs 2024-07-26T08:34:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5891.245088 secs 2024-07-26T08:34:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5908.987416 secs 2024-07-26T08:34:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:22 smithi042 ceph-mon[32775]: pgmap v3712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5908.987416 secs 2024-07-26T08:34:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:22 smithi164 ceph-mon[36970]: pgmap v3712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5908.987416 secs 2024-07-26T08:34:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:22 smithi163 ceph-mon[36873]: pgmap v3712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:24 smithi163 ceph-mon[36873]: pgmap v3713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:24 smithi042 ceph-mon[32775]: pgmap v3713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:24 smithi164 ceph-mon[36970]: pgmap v3713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:25.315 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:25.314+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:34:25.315 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:25.314+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:34:25.315 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:25.314+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:34:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:26 smithi163 ceph-mon[36873]: pgmap v3714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:26 smithi042 ceph-mon[32775]: pgmap v3714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:26 smithi164 ceph-mon[36970]: pgmap v3714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5896.245209 secs 2024-07-26T08:34:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5913.987534 secs 2024-07-26T08:34:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5896.245209 secs 2024-07-26T08:34:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5913.987534 secs 2024-07-26T08:34:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5896.245209 secs 2024-07-26T08:34:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5913.987534 secs 2024-07-26T08:34:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:28 smithi163 ceph-mon[36873]: pgmap v3715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:28 smithi042 ceph-mon[32775]: pgmap v3715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:28 smithi164 ceph-mon[36970]: pgmap v3715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:30 smithi163 ceph-mon[36873]: pgmap v3716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:30 smithi042 ceph-mon[32775]: pgmap v3716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:30 smithi164 ceph-mon[36970]: pgmap v3716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5901.245335 secs 2024-07-26T08:34:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5901.245335 secs 2024-07-26T08:34:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5901.245335 secs 2024-07-26T08:34:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5918.987659 secs 2024-07-26T08:34:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:32 smithi042 ceph-mon[32775]: pgmap v3717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5918.987659 secs 2024-07-26T08:34:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:32 smithi164 ceph-mon[36970]: pgmap v3717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5918.987659 secs 2024-07-26T08:34:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:32 smithi163 ceph-mon[36873]: pgmap v3717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:34 smithi163 ceph-mon[36873]: pgmap v3718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:34 smithi042 ceph-mon[32775]: pgmap v3718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:34 smithi164 ceph-mon[36970]: pgmap v3718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5906.245461 secs 2024-07-26T08:34:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:36 smithi164 ceph-mon[36970]: pgmap v3719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5906.245461 secs 2024-07-26T08:34:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:36 smithi163 ceph-mon[36873]: pgmap v3719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5906.245461 secs 2024-07-26T08:34:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:36 smithi042 ceph-mon[32775]: pgmap v3719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5923.987791 secs 2024-07-26T08:34:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5923.987791 secs 2024-07-26T08:34:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5923.987791 secs 2024-07-26T08:34:39.216 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:38 smithi042 ceph-mon[32775]: pgmap v3720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:38 smithi164 ceph-mon[36970]: pgmap v3720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:38 smithi163 ceph-mon[36873]: pgmap v3720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:40.323 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:40.321+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:34:40.323 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:40.321+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:34:40.323 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:40.321+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:34:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:40 smithi163 ceph-mon[36873]: pgmap v3721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:40 smithi042 ceph-mon[32775]: pgmap v3721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:40 smithi164 ceph-mon[36970]: pgmap v3721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5911.245562 secs 2024-07-26T08:34:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5911.245562 secs 2024-07-26T08:34:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5911.245562 secs 2024-07-26T08:34:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5928.987916 secs 2024-07-26T08:34:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:42 smithi042 ceph-mon[32775]: pgmap v3722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5928.987916 secs 2024-07-26T08:34:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:42 smithi164 ceph-mon[36970]: pgmap v3722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5928.987916 secs 2024-07-26T08:34:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:42 smithi163 ceph-mon[36873]: pgmap v3722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:44 smithi163 ceph-mon[36873]: pgmap v3723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:44 smithi042 ceph-mon[32775]: pgmap v3723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:44 smithi164 ceph-mon[36970]: pgmap v3723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:46 smithi163 ceph-mon[36873]: pgmap v3724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:46 smithi042 ceph-mon[32775]: pgmap v3724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:46 smithi164 ceph-mon[36970]: pgmap v3724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5916.245688 secs 2024-07-26T08:34:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5933.988032 secs 2024-07-26T08:34:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5916.245688 secs 2024-07-26T08:34:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5933.988032 secs 2024-07-26T08:34:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5916.245688 secs 2024-07-26T08:34:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5933.988032 secs 2024-07-26T08:34:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:48 smithi163 ceph-mon[36873]: pgmap v3725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:48.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:48 smithi042 ceph-mon[32775]: pgmap v3725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:48 smithi164 ceph-mon[36970]: pgmap v3725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:50 smithi163 ceph-mon[36873]: pgmap v3726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:50 smithi042 ceph-mon[32775]: pgmap v3726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:50 smithi164 ceph-mon[36970]: pgmap v3726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:34:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5921.245776 secs 2024-07-26T08:34:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5921.245776 secs 2024-07-26T08:34:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5921.245776 secs 2024-07-26T08:34:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5938.988156 secs 2024-07-26T08:34:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:52 smithi042 ceph-mon[32775]: pgmap v3727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:34:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5938.988156 secs 2024-07-26T08:34:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:52 smithi164 ceph-mon[36970]: pgmap v3727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:34:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5938.988156 secs 2024-07-26T08:34:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:52 smithi163 ceph-mon[36873]: pgmap v3727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:34:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:34:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:34:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:34:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:53 smithi042 ceph-mon[32775]: pgmap v3728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:34:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:53 smithi042 ceph-mon[32775]: pgmap v3729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:34:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:34:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:34:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:34:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:53 smithi164 ceph-mon[36970]: pgmap v3728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:34:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:53 smithi164 ceph-mon[36970]: pgmap v3729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:34:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:34:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:34:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:34:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:53 smithi163 ceph-mon[36873]: pgmap v3728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:34:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:53 smithi163 ceph-mon[36873]: pgmap v3729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:34:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:34:55.326 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:55.324+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:34:55.326 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:55.324+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:34:55.326 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:55.324+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:34:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:55 smithi042 ceph-mon[32775]: pgmap v3730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:34:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:55 smithi164 ceph-mon[36970]: pgmap v3730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:34:56.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:55 smithi163 ceph-mon[36873]: pgmap v3730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:34:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5926.245901 secs 2024-07-26T08:34:57.158 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5926.245901 secs 2024-07-26T08:34:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5926.245901 secs 2024-07-26T08:34:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5943.988285 secs 2024-07-26T08:34:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:57 smithi042 ceph-mon[32775]: pgmap v3731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:34:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5943.988285 secs 2024-07-26T08:34:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:57 smithi164 ceph-mon[36970]: pgmap v3731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:34:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5943.988285 secs 2024-07-26T08:34:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:57 smithi163 ceph-mon[36873]: pgmap v3731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:34:58.794 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:58.789+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:34:58.794 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:58.789+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:34:58.794 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:58.789+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T08:34:58.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:58.789+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:34:58.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:58.789+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T08:34:58.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:58.789+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006460 con 0x7ff680102000 2024-07-26T08:34:58.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:58.789+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:34:58.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:34:58.790+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006460 con 0x7ff680102000 2024-07-26T08:35:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:34:59 smithi042 ceph-mon[32775]: pgmap v3732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:35:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:34:59 smithi164 ceph-mon[36970]: pgmap v3732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:35:00.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:34:59 smithi163 ceph-mon[36873]: pgmap v3732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:35:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:01 smithi042 ceph-mon[32775]: pgmap v3733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:35:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5931.246023 secs 2024-07-26T08:35:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:01 smithi164 ceph-mon[36970]: pgmap v3733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:35:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5931.246023 secs 2024-07-26T08:35:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:01 smithi163 ceph-mon[36873]: pgmap v3733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:35:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5931.246023 secs 2024-07-26T08:35:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5948.988414 secs 2024-07-26T08:35:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5948.988414 secs 2024-07-26T08:35:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5948.988414 secs 2024-07-26T08:35:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:03 smithi042 ceph-mon[32775]: pgmap v3734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:35:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:03 smithi164 ceph-mon[36970]: pgmap v3734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:35:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:03 smithi163 ceph-mon[36873]: pgmap v3734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:35:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:05 smithi042 ceph-mon[32775]: pgmap v3735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:05 smithi164 ceph-mon[36970]: pgmap v3735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:05 smithi163 ceph-mon[36873]: pgmap v3735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5936.246119 secs 2024-07-26T08:35:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5936.246119 secs 2024-07-26T08:35:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5936.246119 secs 2024-07-26T08:35:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5953.988554 secs 2024-07-26T08:35:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:07 smithi042 ceph-mon[32775]: pgmap v3736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5953.988554 secs 2024-07-26T08:35:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:07 smithi164 ceph-mon[36970]: pgmap v3736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5953.988554 secs 2024-07-26T08:35:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:07 smithi163 ceph-mon[36873]: pgmap v3736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:09 smithi042 ceph-mon[32775]: pgmap v3737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:35:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:09 smithi164 ceph-mon[36970]: pgmap v3737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:35:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:09 smithi163 ceph-mon[36873]: pgmap v3737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:35:10.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:35:10.326+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:35:10.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:35:10.326+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:35:10.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:35:10.326+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:35:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:11 smithi042 ceph-mon[32775]: pgmap v3738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5941.246261 secs 2024-07-26T08:35:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:11 smithi164 ceph-mon[36970]: pgmap v3738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5941.246261 secs 2024-07-26T08:35:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:11 smithi163 ceph-mon[36873]: pgmap v3738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5941.246261 secs 2024-07-26T08:35:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5958.988676 secs 2024-07-26T08:35:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5958.988676 secs 2024-07-26T08:35:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5958.988676 secs 2024-07-26T08:35:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:13 smithi042 ceph-mon[32775]: pgmap v3739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:13 smithi164 ceph-mon[36970]: pgmap v3739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:13 smithi163 ceph-mon[36873]: pgmap v3739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:15 smithi042 ceph-mon[32775]: pgmap v3740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:15 smithi164 ceph-mon[36970]: pgmap v3740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:15 smithi163 ceph-mon[36873]: pgmap v3740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5946.246354 secs 2024-07-26T08:35:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5946.246354 secs 2024-07-26T08:35:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5946.246354 secs 2024-07-26T08:35:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5963.988821 secs 2024-07-26T08:35:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:17 smithi042 ceph-mon[32775]: pgmap v3741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:18.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5963.988821 secs 2024-07-26T08:35:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:17 smithi164 ceph-mon[36970]: pgmap v3741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5963.988821 secs 2024-07-26T08:35:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:17 smithi163 ceph-mon[36873]: pgmap v3741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:19 smithi042 ceph-mon[32775]: pgmap v3742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:35:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:19 smithi164 ceph-mon[36970]: pgmap v3742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:35:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:19 smithi163 ceph-mon[36873]: pgmap v3742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:35:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:21 smithi042 ceph-mon[32775]: pgmap v3743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5951.246421 secs 2024-07-26T08:35:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:21 smithi164 ceph-mon[36970]: pgmap v3743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5951.246421 secs 2024-07-26T08:35:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:21 smithi163 ceph-mon[36873]: pgmap v3743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5951.246421 secs 2024-07-26T08:35:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5968.988958 secs 2024-07-26T08:35:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5968.988958 secs 2024-07-26T08:35:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5968.988958 secs 2024-07-26T08:35:24.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:23 smithi042 ceph-mon[32775]: pgmap v3744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:23 smithi164 ceph-mon[36970]: pgmap v3744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:23 smithi163 ceph-mon[36873]: pgmap v3744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:35:25.329+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:35:25.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:35:25.329+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:35:25.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:35:25.329+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:35:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:25 smithi042 ceph-mon[32775]: pgmap v3745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:25 smithi164 ceph-mon[36970]: pgmap v3745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:25 smithi163 ceph-mon[36873]: pgmap v3745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5956.246546 secs 2024-07-26T08:35:27.158 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5956.246546 secs 2024-07-26T08:35:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5956.246546 secs 2024-07-26T08:35:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5973.989079 secs 2024-07-26T08:35:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:27 smithi042 ceph-mon[32775]: pgmap v3746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5973.989079 secs 2024-07-26T08:35:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:27 smithi164 ceph-mon[36970]: pgmap v3746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5973.989079 secs 2024-07-26T08:35:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:27 smithi163 ceph-mon[36873]: pgmap v3746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:29 smithi042 ceph-mon[32775]: pgmap v3747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:35:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:29 smithi164 ceph-mon[36970]: pgmap v3747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:35:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:29 smithi163 ceph-mon[36873]: pgmap v3747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:35:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:31 smithi042 ceph-mon[32775]: pgmap v3748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5961.246666 secs 2024-07-26T08:35:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:31 smithi164 ceph-mon[36970]: pgmap v3748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5961.246666 secs 2024-07-26T08:35:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:31 smithi163 ceph-mon[36873]: pgmap v3748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5961.246666 secs 2024-07-26T08:35:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5978.989199 secs 2024-07-26T08:35:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5978.989199 secs 2024-07-26T08:35:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5978.989199 secs 2024-07-26T08:35:34.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:33 smithi042 ceph-mon[32775]: pgmap v3749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:33 smithi164 ceph-mon[36970]: pgmap v3749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:33 smithi163 ceph-mon[36873]: pgmap v3749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:35 smithi042 ceph-mon[32775]: pgmap v3750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:35 smithi164 ceph-mon[36970]: pgmap v3750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:35 smithi163 ceph-mon[36873]: pgmap v3750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5966.246749 secs 2024-07-26T08:35:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5966.246749 secs 2024-07-26T08:35:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5966.246749 secs 2024-07-26T08:35:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5983.989342 secs 2024-07-26T08:35:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:37 smithi164 ceph-mon[36970]: pgmap v3751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5983.989342 secs 2024-07-26T08:35:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:37 smithi163 ceph-mon[36873]: pgmap v3751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5983.989342 secs 2024-07-26T08:35:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:37 smithi042 ceph-mon[32775]: pgmap v3751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:39 smithi164 ceph-mon[36970]: pgmap v3752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:35:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:39 smithi163 ceph-mon[36873]: pgmap v3752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:35:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:35:40.338+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:35:40.339 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:35:40.338+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:35:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:35:40.338+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:35:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:39 smithi042 ceph-mon[32775]: pgmap v3752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:35:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:41 smithi164 ceph-mon[36970]: pgmap v3753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5971.246868 secs 2024-07-26T08:35:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:41 smithi163 ceph-mon[36873]: pgmap v3753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5971.246868 secs 2024-07-26T08:35:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:41 smithi042 ceph-mon[32775]: pgmap v3753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5971.246868 secs 2024-07-26T08:35:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5988.989485 secs 2024-07-26T08:35:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5988.989485 secs 2024-07-26T08:35:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5988.989485 secs 2024-07-26T08:35:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:43 smithi164 ceph-mon[36970]: pgmap v3754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:43 smithi163 ceph-mon[36873]: pgmap v3754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:43 smithi042 ceph-mon[32775]: pgmap v3754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:45 smithi164 ceph-mon[36970]: pgmap v3755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:45 smithi163 ceph-mon[36873]: pgmap v3755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:45 smithi042 ceph-mon[32775]: pgmap v3755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5976.246994 secs 2024-07-26T08:35:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5976.246994 secs 2024-07-26T08:35:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5976.246994 secs 2024-07-26T08:35:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5993.989599 secs 2024-07-26T08:35:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:47 smithi164 ceph-mon[36970]: pgmap v3756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5993.989599 secs 2024-07-26T08:35:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:47 smithi163 ceph-mon[36873]: pgmap v3756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:48.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5993.989599 secs 2024-07-26T08:35:48.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:47 smithi042 ceph-mon[32775]: pgmap v3756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:49 smithi164 ceph-mon[36970]: pgmap v3757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:35:50.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:49 smithi163 ceph-mon[36873]: pgmap v3757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:35:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:49 smithi042 ceph-mon[32775]: pgmap v3757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:35:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:51 smithi164 ceph-mon[36970]: pgmap v3758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5981.247116 secs 2024-07-26T08:35:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5998.989723 secs 2024-07-26T08:35:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:51 smithi163 ceph-mon[36873]: pgmap v3758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5981.247116 secs 2024-07-26T08:35:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5998.989723 secs 2024-07-26T08:35:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:51 smithi042 ceph-mon[32775]: pgmap v3758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:35:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5981.247116 secs 2024-07-26T08:35:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5998.989723 secs 2024-07-26T08:35:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:35:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:35:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:35:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:53 smithi163 ceph-mon[36873]: pgmap v3759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:35:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:35:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:35:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:53 smithi042 ceph-mon[32775]: pgmap v3759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:35:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:35:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:35:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:53 smithi164 ceph-mon[36970]: pgmap v3759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:35:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:35:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:35:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:35:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:54 smithi163 ceph-mon[36873]: pgmap v3760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:35:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:54 smithi163 ceph-mon[36873]: pgmap v3761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:35:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:35:55.340+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:35:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:35:55.340+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:35:55.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:35:55.340+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:35:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:54 smithi042 ceph-mon[32775]: pgmap v3760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:35:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:54 smithi042 ceph-mon[32775]: pgmap v3761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:35:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:54 smithi164 ceph-mon[36970]: pgmap v3760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:35:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:54 smithi164 ceph-mon[36970]: pgmap v3761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:35:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:56 smithi163 ceph-mon[36873]: pgmap v3762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:35:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5986.247261 secs 2024-07-26T08:35:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6003.989866 secs 2024-07-26T08:35:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:56 smithi042 ceph-mon[32775]: pgmap v3762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:35:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5986.247261 secs 2024-07-26T08:35:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6003.989866 secs 2024-07-26T08:35:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:56 smithi164 ceph-mon[36970]: pgmap v3762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:35:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5986.247261 secs 2024-07-26T08:35:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6003.989866 secs 2024-07-26T08:35:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:35:58 smithi163 ceph-mon[36873]: pgmap v3763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:35:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:35:58 smithi042 ceph-mon[32775]: pgmap v3763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:35:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:35:58 smithi164 ceph-mon[36970]: pgmap v3763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:36:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:00 smithi163 ceph-mon[36873]: pgmap v3764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:36:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:00 smithi042 ceph-mon[32775]: pgmap v3764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:36:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:00 smithi164 ceph-mon[36970]: pgmap v3764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:36:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5991.247381 secs 2024-07-26T08:36:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6008.990018 secs 2024-07-26T08:36:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5991.247381 secs 2024-07-26T08:36:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6008.990018 secs 2024-07-26T08:36:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5991.247381 secs 2024-07-26T08:36:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6008.990018 secs 2024-07-26T08:36:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:02 smithi163 ceph-mon[36873]: pgmap v3765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:36:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:02 smithi042 ceph-mon[32775]: pgmap v3765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:36:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:02 smithi164 ceph-mon[36970]: pgmap v3765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:36:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:05 smithi163 ceph-mon[36873]: pgmap v3766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:36:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:05 smithi042 ceph-mon[32775]: pgmap v3766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:36:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:05 smithi164 ceph-mon[36970]: pgmap v3766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:36:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:07 smithi163 ceph-mon[36873]: pgmap v3767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 5996.247508 secs 2024-07-26T08:36:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6013.990139 secs 2024-07-26T08:36:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:07 smithi042 ceph-mon[32775]: pgmap v3767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 5996.247508 secs 2024-07-26T08:36:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6013.990139 secs 2024-07-26T08:36:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:07 smithi164 ceph-mon[36970]: pgmap v3767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 5996.247508 secs 2024-07-26T08:36:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6013.990139 secs 2024-07-26T08:36:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:09 smithi042 ceph-mon[32775]: pgmap v3768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:09 smithi164 ceph-mon[36970]: pgmap v3768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:09 smithi163 ceph-mon[36873]: pgmap v3768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:36:10.349+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:36:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:36:10.349+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:36:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:36:10.349+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:36:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:11 smithi042 ceph-mon[32775]: pgmap v3769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:11 smithi164 ceph-mon[36970]: pgmap v3769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:11 smithi163 ceph-mon[36873]: pgmap v3769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:12.373 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6001.247578 secs 2024-07-26T08:36:12.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6018.990254 secs 2024-07-26T08:36:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6001.247578 secs 2024-07-26T08:36:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6018.990254 secs 2024-07-26T08:36:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6001.247578 secs 2024-07-26T08:36:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6018.990254 secs 2024-07-26T08:36:13.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:13 smithi042 ceph-mon[32775]: pgmap v3770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:13 smithi164 ceph-mon[36970]: pgmap v3770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:13 smithi163 ceph-mon[36873]: pgmap v3770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:15 smithi042 ceph-mon[32775]: pgmap v3771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:36:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:15 smithi164 ceph-mon[36970]: pgmap v3771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:36:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:15 smithi163 ceph-mon[36873]: pgmap v3771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:36:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:17 smithi042 ceph-mon[32775]: pgmap v3772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6006.247700 secs 2024-07-26T08:36:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6023.990319 secs 2024-07-26T08:36:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:17 smithi164 ceph-mon[36970]: pgmap v3772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6006.247700 secs 2024-07-26T08:36:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6023.990319 secs 2024-07-26T08:36:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:17 smithi163 ceph-mon[36873]: pgmap v3772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6006.247700 secs 2024-07-26T08:36:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6023.990319 secs 2024-07-26T08:36:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:19 smithi042 ceph-mon[32775]: pgmap v3773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:19 smithi164 ceph-mon[36970]: pgmap v3773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:19 smithi163 ceph-mon[36873]: pgmap v3773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:21 smithi042 ceph-mon[32775]: pgmap v3774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:21 smithi164 ceph-mon[36970]: pgmap v3774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:21 smithi163 ceph-mon[36873]: pgmap v3774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:22.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6011.247824 secs 2024-07-26T08:36:22.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6028.990400 secs 2024-07-26T08:36:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6011.247824 secs 2024-07-26T08:36:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6028.990400 secs 2024-07-26T08:36:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6011.247824 secs 2024-07-26T08:36:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6028.990400 secs 2024-07-26T08:36:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:23 smithi042 ceph-mon[32775]: pgmap v3775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:23 smithi164 ceph-mon[36970]: pgmap v3775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:23 smithi163 ceph-mon[36873]: pgmap v3775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:36:25.353+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:36:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:36:25.353+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:36:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:36:25.353+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:36:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:25 smithi042 ceph-mon[32775]: pgmap v3776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:36:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:25 smithi164 ceph-mon[36970]: pgmap v3776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:36:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:25 smithi163 ceph-mon[36873]: pgmap v3776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:36:27.309 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:27 smithi163 ceph-mon[36873]: pgmap v3777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:27.309 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6016.247966 secs 2024-07-26T08:36:27.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6033.990518 secs 2024-07-26T08:36:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:27 smithi042 ceph-mon[32775]: pgmap v3777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6016.247966 secs 2024-07-26T08:36:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6033.990518 secs 2024-07-26T08:36:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:27 smithi164 ceph-mon[36970]: pgmap v3777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6016.247966 secs 2024-07-26T08:36:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6033.990518 secs 2024-07-26T08:36:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:29 smithi042 ceph-mon[32775]: pgmap v3778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:29 smithi164 ceph-mon[36970]: pgmap v3778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:29 smithi163 ceph-mon[36873]: pgmap v3778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:31 smithi042 ceph-mon[32775]: pgmap v3779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:31 smithi164 ceph-mon[36970]: pgmap v3779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:31 smithi163 ceph-mon[36873]: pgmap v3779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6021.248090 secs 2024-07-26T08:36:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6038.990645 secs 2024-07-26T08:36:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6021.248090 secs 2024-07-26T08:36:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6038.990645 secs 2024-07-26T08:36:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6021.248090 secs 2024-07-26T08:36:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6038.990645 secs 2024-07-26T08:36:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:33 smithi042 ceph-mon[32775]: pgmap v3780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:33 smithi164 ceph-mon[36970]: pgmap v3780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:33 smithi163 ceph-mon[36873]: pgmap v3780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:35 smithi042 ceph-mon[32775]: pgmap v3781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:36:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:35 smithi164 ceph-mon[36970]: pgmap v3781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:36:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:35 smithi163 ceph-mon[36873]: pgmap v3781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:36:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:37 smithi042 ceph-mon[32775]: pgmap v3782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6026.248190 secs 2024-07-26T08:36:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6043.990768 secs 2024-07-26T08:36:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:37 smithi164 ceph-mon[36970]: pgmap v3782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6026.248190 secs 2024-07-26T08:36:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6043.990768 secs 2024-07-26T08:36:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:37 smithi163 ceph-mon[36873]: pgmap v3782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6026.248190 secs 2024-07-26T08:36:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6043.990768 secs 2024-07-26T08:36:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:39 smithi042 ceph-mon[32775]: pgmap v3783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:39 smithi164 ceph-mon[36970]: pgmap v3783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:39 smithi163 ceph-mon[36873]: pgmap v3783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:36:40.362+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:36:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:36:40.362+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:36:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:36:40.362+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:36:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:41 smithi042 ceph-mon[32775]: pgmap v3784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:41 smithi164 ceph-mon[36970]: pgmap v3784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:41 smithi163 ceph-mon[36873]: pgmap v3784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:42.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6031.248278 secs 2024-07-26T08:36:42.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6048.990892 secs 2024-07-26T08:36:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6031.248278 secs 2024-07-26T08:36:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6048.990892 secs 2024-07-26T08:36:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6031.248278 secs 2024-07-26T08:36:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6048.990892 secs 2024-07-26T08:36:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:43 smithi042 ceph-mon[32775]: pgmap v3785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:43 smithi164 ceph-mon[36970]: pgmap v3785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:43 smithi163 ceph-mon[36873]: pgmap v3785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:45 smithi042 ceph-mon[32775]: pgmap v3786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:36:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:45 smithi164 ceph-mon[36970]: pgmap v3786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:36:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:45 smithi163 ceph-mon[36873]: pgmap v3786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:36:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:47 smithi042 ceph-mon[32775]: pgmap v3787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6036.248404 secs 2024-07-26T08:36:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6053.990984 secs 2024-07-26T08:36:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:47 smithi164 ceph-mon[36970]: pgmap v3787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6036.248404 secs 2024-07-26T08:36:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6053.990984 secs 2024-07-26T08:36:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:47 smithi163 ceph-mon[36873]: pgmap v3787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6036.248404 secs 2024-07-26T08:36:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6053.990984 secs 2024-07-26T08:36:49.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:49 smithi042 ceph-mon[32775]: pgmap v3788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:49 smithi164 ceph-mon[36970]: pgmap v3788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:49 smithi163 ceph-mon[36873]: pgmap v3788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:51 smithi042 ceph-mon[32775]: pgmap v3789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:51 smithi164 ceph-mon[36970]: pgmap v3789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:51 smithi163 ceph-mon[36873]: pgmap v3789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:36:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6041.248529 secs 2024-07-26T08:36:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6058.991109 secs 2024-07-26T08:36:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6041.248529 secs 2024-07-26T08:36:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6058.991109 secs 2024-07-26T08:36:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6041.248529 secs 2024-07-26T08:36:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6058.991109 secs 2024-07-26T08:36:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:53 smithi042 ceph-mon[32775]: pgmap v3790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:53 smithi164 ceph-mon[36970]: pgmap v3790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:53 smithi163 ceph-mon[36873]: pgmap v3790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:36:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:36:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:36:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:36:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:36:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:36:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:36:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:36:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:36:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:36:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:36:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:36:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:36:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:36:55.366+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:36:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:36:55.366+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:36:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:36:55.366+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:36:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:55 smithi042 ceph-mon[32775]: pgmap v3791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:36:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:55 smithi042 ceph-mon[32775]: pgmap v3792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:36:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:55 smithi042 ceph-mon[32775]: pgmap v3793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:36:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:55 smithi164 ceph-mon[36970]: pgmap v3791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:36:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:55 smithi164 ceph-mon[36970]: pgmap v3792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:36:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:55 smithi164 ceph-mon[36970]: pgmap v3793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:36:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:55 smithi163 ceph-mon[36873]: pgmap v3791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:36:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:55 smithi163 ceph-mon[36873]: pgmap v3792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:36:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:55 smithi163 ceph-mon[36873]: pgmap v3793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:36:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6046.248630 secs 2024-07-26T08:36:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:57 smithi042 ceph-mon[32775]: pgmap v3794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:36:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6063.991231 secs 2024-07-26T08:36:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6046.248630 secs 2024-07-26T08:36:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:57 smithi164 ceph-mon[36970]: pgmap v3794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:36:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6063.991231 secs 2024-07-26T08:36:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6046.248630 secs 2024-07-26T08:36:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:57 smithi163 ceph-mon[36873]: pgmap v3794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:36:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6063.991231 secs 2024-07-26T08:36:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:36:59 smithi042 ceph-mon[32775]: pgmap v3795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:36:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:36:59 smithi164 ceph-mon[36970]: pgmap v3795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:36:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:36:59 smithi163 ceph-mon[36873]: pgmap v3795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:37:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:01 smithi042 ceph-mon[32775]: pgmap v3796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:37:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:01 smithi164 ceph-mon[36970]: pgmap v3796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:37:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:01 smithi163 ceph-mon[36873]: pgmap v3796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:37:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6051.248753 secs 2024-07-26T08:37:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6068.991356 secs 2024-07-26T08:37:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6051.248753 secs 2024-07-26T08:37:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6068.991356 secs 2024-07-26T08:37:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6051.248753 secs 2024-07-26T08:37:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6068.991356 secs 2024-07-26T08:37:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:03 smithi042 ceph-mon[32775]: pgmap v3797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:37:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:03 smithi164 ceph-mon[36970]: pgmap v3797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:37:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:03 smithi163 ceph-mon[36873]: pgmap v3797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:37:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:05 smithi042 ceph-mon[32775]: pgmap v3798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:37:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:05 smithi164 ceph-mon[36970]: pgmap v3798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:37:05.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:05 smithi163 ceph-mon[36873]: pgmap v3798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:37:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6056.248853 secs 2024-07-26T08:37:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:07 smithi042 ceph-mon[32775]: pgmap v3799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6073.991469 secs 2024-07-26T08:37:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6056.248853 secs 2024-07-26T08:37:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:07 smithi164 ceph-mon[36970]: pgmap v3799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6073.991469 secs 2024-07-26T08:37:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6056.248853 secs 2024-07-26T08:37:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:07 smithi163 ceph-mon[36873]: pgmap v3799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6073.991469 secs 2024-07-26T08:37:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:09 smithi042 ceph-mon[32775]: pgmap v3800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:09 smithi164 ceph-mon[36970]: pgmap v3800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:09 smithi163 ceph-mon[36873]: pgmap v3800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:37:10.374+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:37:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:37:10.374+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:37:10.375 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:37:10.374+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:37:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:11 smithi042 ceph-mon[32775]: pgmap v3801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:11 smithi164 ceph-mon[36970]: pgmap v3801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:11 smithi163 ceph-mon[36873]: pgmap v3801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6061.248972 secs 2024-07-26T08:37:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6078.991600 secs 2024-07-26T08:37:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6061.248972 secs 2024-07-26T08:37:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6078.991600 secs 2024-07-26T08:37:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6061.248972 secs 2024-07-26T08:37:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6078.991600 secs 2024-07-26T08:37:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:13 smithi164 ceph-mon[36970]: pgmap v3802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:13 smithi163 ceph-mon[36873]: pgmap v3802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:13 smithi042 ceph-mon[32775]: pgmap v3802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:15 smithi164 ceph-mon[36970]: pgmap v3803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:37:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:15 smithi163 ceph-mon[36873]: pgmap v3803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:37:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:15 smithi042 ceph-mon[32775]: pgmap v3803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:37:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6066.249122 secs 2024-07-26T08:37:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:17 smithi164 ceph-mon[36970]: pgmap v3804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6083.991694 secs 2024-07-26T08:37:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6066.249122 secs 2024-07-26T08:37:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:17 smithi163 ceph-mon[36873]: pgmap v3804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6083.991694 secs 2024-07-26T08:37:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6066.249122 secs 2024-07-26T08:37:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:17 smithi042 ceph-mon[32775]: pgmap v3804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6083.991694 secs 2024-07-26T08:37:19.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:19 smithi042 ceph-mon[32775]: pgmap v3805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:19 smithi164 ceph-mon[36970]: pgmap v3805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:19 smithi163 ceph-mon[36873]: pgmap v3805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:21 smithi164 ceph-mon[36970]: pgmap v3806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:21 smithi163 ceph-mon[36873]: pgmap v3806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:21 smithi042 ceph-mon[32775]: pgmap v3806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6071.249243 secs 2024-07-26T08:37:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6088.991824 secs 2024-07-26T08:37:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6071.249243 secs 2024-07-26T08:37:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6088.991824 secs 2024-07-26T08:37:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6071.249243 secs 2024-07-26T08:37:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6088.991824 secs 2024-07-26T08:37:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:23 smithi164 ceph-mon[36970]: pgmap v3807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:23 smithi163 ceph-mon[36873]: pgmap v3807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:23 smithi042 ceph-mon[32775]: pgmap v3807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:25.379 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:37:25.377+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:37:25.379 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:37:25.377+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:37:25.379 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:37:25.377+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:37:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:25 smithi164 ceph-mon[36970]: pgmap v3808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:37:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:25 smithi163 ceph-mon[36873]: pgmap v3808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:37:25.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:25 smithi042 ceph-mon[32775]: pgmap v3808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:37:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6076.249330 secs 2024-07-26T08:37:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:27 smithi164 ceph-mon[36970]: pgmap v3809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6093.991920 secs 2024-07-26T08:37:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6076.249330 secs 2024-07-26T08:37:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:27 smithi163 ceph-mon[36873]: pgmap v3809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6093.991920 secs 2024-07-26T08:37:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6076.249330 secs 2024-07-26T08:37:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:27 smithi042 ceph-mon[32775]: pgmap v3809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6093.991920 secs 2024-07-26T08:37:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:29 smithi164 ceph-mon[36970]: pgmap v3810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:29 smithi163 ceph-mon[36873]: pgmap v3810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:29 smithi042 ceph-mon[32775]: pgmap v3810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:31 smithi164 ceph-mon[36970]: pgmap v3811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:31 smithi163 ceph-mon[36873]: pgmap v3811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:31 smithi042 ceph-mon[32775]: pgmap v3811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6081.249436 secs 2024-07-26T08:37:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6098.992019 secs 2024-07-26T08:37:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6081.249436 secs 2024-07-26T08:37:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6098.992019 secs 2024-07-26T08:37:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6081.249436 secs 2024-07-26T08:37:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6098.992019 secs 2024-07-26T08:37:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:33 smithi164 ceph-mon[36970]: pgmap v3812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:33 smithi163 ceph-mon[36873]: pgmap v3812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:33.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:33 smithi042 ceph-mon[32775]: pgmap v3812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:35 smithi164 ceph-mon[36970]: pgmap v3813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:37:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:35 smithi163 ceph-mon[36873]: pgmap v3813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:37:35.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:35 smithi042 ceph-mon[32775]: pgmap v3813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:37:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6086.249505 secs 2024-07-26T08:37:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:37 smithi164 ceph-mon[36970]: pgmap v3814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6103.992085 secs 2024-07-26T08:37:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6086.249505 secs 2024-07-26T08:37:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:37 smithi163 ceph-mon[36873]: pgmap v3814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6103.992085 secs 2024-07-26T08:37:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6086.249505 secs 2024-07-26T08:37:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:37 smithi042 ceph-mon[32775]: pgmap v3814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6103.992085 secs 2024-07-26T08:37:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:39 smithi164 ceph-mon[36970]: pgmap v3815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:39 smithi163 ceph-mon[36873]: pgmap v3815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:39 smithi042 ceph-mon[32775]: pgmap v3815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:37:40.386+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:37:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:37:40.386+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:37:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:37:40.386+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:37:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:41 smithi164 ceph-mon[36970]: pgmap v3816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:41 smithi163 ceph-mon[36873]: pgmap v3816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:41 smithi042 ceph-mon[32775]: pgmap v3816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6091.249588 secs 2024-07-26T08:37:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6108.992203 secs 2024-07-26T08:37:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6091.249588 secs 2024-07-26T08:37:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6108.992203 secs 2024-07-26T08:37:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6091.249588 secs 2024-07-26T08:37:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6108.992203 secs 2024-07-26T08:37:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:43 smithi164 ceph-mon[36970]: pgmap v3817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:43 smithi163 ceph-mon[36873]: pgmap v3817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:43 smithi042 ceph-mon[32775]: pgmap v3817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:45 smithi164 ceph-mon[36970]: pgmap v3818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:37:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:45 smithi163 ceph-mon[36873]: pgmap v3818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:37:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:45 smithi042 ceph-mon[32775]: pgmap v3818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:37:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:46 smithi042 ceph-mon[32775]: pgmap v3819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:46 smithi164 ceph-mon[36970]: pgmap v3819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:46 smithi163 ceph-mon[36873]: pgmap v3819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6096.249669 secs 2024-07-26T08:37:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6113.992284 secs 2024-07-26T08:37:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6096.249669 secs 2024-07-26T08:37:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6113.992284 secs 2024-07-26T08:37:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6096.249669 secs 2024-07-26T08:37:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6113.992284 secs 2024-07-26T08:37:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:48 smithi163 ceph-mon[36873]: pgmap v3820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:48.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:48 smithi042 ceph-mon[32775]: pgmap v3820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:48 smithi164 ceph-mon[36970]: pgmap v3820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:50 smithi042 ceph-mon[32775]: pgmap v3821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:50 smithi164 ceph-mon[36970]: pgmap v3821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:50 smithi163 ceph-mon[36873]: pgmap v3821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:37:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6101.249796 secs 2024-07-26T08:37:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6101.249796 secs 2024-07-26T08:37:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6101.249796 secs 2024-07-26T08:37:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6118.992405 secs 2024-07-26T08:37:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:52 smithi164 ceph-mon[36970]: pgmap v3822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6118.992405 secs 2024-07-26T08:37:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:52 smithi163 ceph-mon[36873]: pgmap v3822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6118.992405 secs 2024-07-26T08:37:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:52 smithi042 ceph-mon[32775]: pgmap v3822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:37:55.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:54 smithi164 ceph-mon[36970]: pgmap v3823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:37:55.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:37:55.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:37:55.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:37:55.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:37:55.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:54 smithi163 ceph-mon[36873]: pgmap v3823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:37:55.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:37:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:37:55.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:37:55.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:37:55.391 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:37:55.390+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:37:55.391 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:37:55.390+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:37:55.391 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:37:55.390+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:37:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:54 smithi042 ceph-mon[32775]: pgmap v3823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:37:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:37:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:37:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:37:55.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:37:55.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:37:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:55 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:37:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:55 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:37:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:55 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:37:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:55 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:37:56.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:55 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:37:56.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:55 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:37:56.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:55 smithi164 ceph-mon[36970]: pgmap v3824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T08:37:56.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:55 smithi164 ceph-mon[36970]: pgmap v3825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:37:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:55 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:37:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:55 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:37:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:55 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:37:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:55 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:37:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:55 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:37:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:55 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:37:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:55 smithi163 ceph-mon[36873]: pgmap v3824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T08:37:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:55 smithi163 ceph-mon[36873]: pgmap v3825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:37:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:55 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:37:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:55 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:37:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:55 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:37:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:55 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:37:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:55 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:37:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:55 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:37:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:55 smithi042 ceph-mon[32775]: pgmap v3824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T08:37:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:55 smithi042 ceph-mon[32775]: pgmap v3825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:37:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6106.249922 secs 2024-07-26T08:37:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6106.249922 secs 2024-07-26T08:37:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6106.249922 secs 2024-07-26T08:37:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6123.992550 secs 2024-07-26T08:37:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:57 smithi164 ceph-mon[36970]: pgmap v3826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:37:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6123.992550 secs 2024-07-26T08:37:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:57 smithi163 ceph-mon[36873]: pgmap v3826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:37:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6123.992550 secs 2024-07-26T08:37:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:57 smithi042 ceph-mon[32775]: pgmap v3826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:37:58.815 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:37:58.814+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:37:58.816 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:37:58.814+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:37:58.816 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:37:58.814+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:37:58.816 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:37:58.814+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:38:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:37:59 smithi164 ceph-mon[36970]: pgmap v3827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:38:00.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:37:59 smithi163 ceph-mon[36873]: pgmap v3827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:38:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:37:59 smithi042 ceph-mon[32775]: pgmap v3827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:38:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:01 smithi164 ceph-mon[36970]: pgmap v3828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:38:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6111.250042 secs 2024-07-26T08:38:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:01 smithi163 ceph-mon[36873]: pgmap v3828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:38:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6111.250042 secs 2024-07-26T08:38:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:01 smithi042 ceph-mon[32775]: pgmap v3828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:38:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6111.250042 secs 2024-07-26T08:38:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6128.992651 secs 2024-07-26T08:38:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6128.992651 secs 2024-07-26T08:38:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6128.992651 secs 2024-07-26T08:38:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:03 smithi164 ceph-mon[36970]: pgmap v3829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:38:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:03 smithi163 ceph-mon[36873]: pgmap v3829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:38:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:03 smithi042 ceph-mon[32775]: pgmap v3829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:38:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:05 smithi164 ceph-mon[36970]: pgmap v3830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:38:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:05 smithi163 ceph-mon[36873]: pgmap v3830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:38:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:05 smithi042 ceph-mon[32775]: pgmap v3830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:38:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6116.250148 secs 2024-07-26T08:38:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6116.250148 secs 2024-07-26T08:38:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6116.250148 secs 2024-07-26T08:38:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6133.992793 secs 2024-07-26T08:38:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:07 smithi164 ceph-mon[36970]: pgmap v3831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6133.992793 secs 2024-07-26T08:38:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:07 smithi163 ceph-mon[36873]: pgmap v3831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6133.992793 secs 2024-07-26T08:38:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:07 smithi042 ceph-mon[32775]: pgmap v3831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:09 smithi164 ceph-mon[36970]: pgmap v3832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:38:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:09 smithi163 ceph-mon[36873]: pgmap v3832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:38:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:09 smithi042 ceph-mon[32775]: pgmap v3832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:38:10.400 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:38:10.399+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:38:10.400 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:38:10.399+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:38:10.400 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:38:10.399+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:38:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:11 smithi164 ceph-mon[36970]: pgmap v3833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6121.250295 secs 2024-07-26T08:38:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:11 smithi163 ceph-mon[36873]: pgmap v3833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6121.250295 secs 2024-07-26T08:38:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:11 smithi042 ceph-mon[32775]: pgmap v3833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6121.250295 secs 2024-07-26T08:38:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6138.992958 secs 2024-07-26T08:38:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6138.992958 secs 2024-07-26T08:38:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6138.992958 secs 2024-07-26T08:38:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:13 smithi164 ceph-mon[36970]: pgmap v3834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:13 smithi163 ceph-mon[36873]: pgmap v3834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:13 smithi042 ceph-mon[32775]: pgmap v3834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:15 smithi164 ceph-mon[36970]: pgmap v3835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:15 smithi163 ceph-mon[36873]: pgmap v3835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:15 smithi042 ceph-mon[32775]: pgmap v3835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6126.250382 secs 2024-07-26T08:38:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6126.250382 secs 2024-07-26T08:38:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6126.250382 secs 2024-07-26T08:38:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6143.993079 secs 2024-07-26T08:38:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:17 smithi164 ceph-mon[36970]: pgmap v3836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6143.993079 secs 2024-07-26T08:38:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:17 smithi163 ceph-mon[36873]: pgmap v3836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6143.993079 secs 2024-07-26T08:38:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:17 smithi042 ceph-mon[32775]: pgmap v3836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:19 smithi164 ceph-mon[36970]: pgmap v3837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:38:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:19 smithi163 ceph-mon[36873]: pgmap v3837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:38:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:19 smithi042 ceph-mon[32775]: pgmap v3837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:38:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:21 smithi163 ceph-mon[36873]: pgmap v3838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6131.250519 secs 2024-07-26T08:38:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:21 smithi042 ceph-mon[32775]: pgmap v3838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6131.250519 secs 2024-07-26T08:38:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:21 smithi164 ceph-mon[36970]: pgmap v3838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6131.250519 secs 2024-07-26T08:38:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6148.993165 secs 2024-07-26T08:38:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6148.993165 secs 2024-07-26T08:38:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6148.993165 secs 2024-07-26T08:38:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:23 smithi163 ceph-mon[36873]: pgmap v3839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:23 smithi042 ceph-mon[32775]: pgmap v3839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:23 smithi164 ceph-mon[36970]: pgmap v3839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:25.404 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:38:25.403+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:38:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:38:25.403+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:38:25.405 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:38:25.403+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:38:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:25 smithi163 ceph-mon[36873]: pgmap v3840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:25 smithi042 ceph-mon[32775]: pgmap v3840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:25 smithi164 ceph-mon[36970]: pgmap v3840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6136.250642 secs 2024-07-26T08:38:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6153.993232 secs 2024-07-26T08:38:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6136.250642 secs 2024-07-26T08:38:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6153.993232 secs 2024-07-26T08:38:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6136.250642 secs 2024-07-26T08:38:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6153.993232 secs 2024-07-26T08:38:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:27 smithi163 ceph-mon[36873]: pgmap v3841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:27 smithi042 ceph-mon[32775]: pgmap v3841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:27 smithi164 ceph-mon[36970]: pgmap v3841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:29 smithi163 ceph-mon[36873]: pgmap v3842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:38:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:29 smithi042 ceph-mon[32775]: pgmap v3842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:38:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:29 smithi164 ceph-mon[36970]: pgmap v3842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:38:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:31 smithi163 ceph-mon[36873]: pgmap v3843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6141.250787 secs 2024-07-26T08:38:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6158.993361 secs 2024-07-26T08:38:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:31 smithi042 ceph-mon[32775]: pgmap v3843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6141.250787 secs 2024-07-26T08:38:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6158.993361 secs 2024-07-26T08:38:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:31 smithi164 ceph-mon[36970]: pgmap v3843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6141.250787 secs 2024-07-26T08:38:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6158.993361 secs 2024-07-26T08:38:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:34 smithi163 ceph-mon[36873]: pgmap v3844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:34 smithi042 ceph-mon[32775]: pgmap v3844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:34 smithi164 ceph-mon[36970]: pgmap v3844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:36 smithi163 ceph-mon[36873]: pgmap v3845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:36 smithi042 ceph-mon[32775]: pgmap v3845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:36 smithi164 ceph-mon[36970]: pgmap v3845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6146.250914 secs 2024-07-26T08:38:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6163.993488 secs 2024-07-26T08:38:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6146.250914 secs 2024-07-26T08:38:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6163.993488 secs 2024-07-26T08:38:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6146.250914 secs 2024-07-26T08:38:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6163.993488 secs 2024-07-26T08:38:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:38 smithi042 ceph-mon[32775]: pgmap v3846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:38 smithi164 ceph-mon[36970]: pgmap v3846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:38 smithi163 ceph-mon[36873]: pgmap v3846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:40 smithi042 ceph-mon[32775]: pgmap v3847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:38:40.413 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:38:40.412+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:38:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:38:40.412+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:38:40.414 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:38:40.412+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:38:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:40 smithi164 ceph-mon[36970]: pgmap v3847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:38:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:40 smithi163 ceph-mon[36873]: pgmap v3847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:38:42.275 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:42 smithi164 ceph-mon[36970]: pgmap v3848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:42.275 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6151.251064 secs 2024-07-26T08:38:42.275 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6168.993619 secs 2024-07-26T08:38:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:42 smithi042 ceph-mon[32775]: pgmap v3848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6151.251064 secs 2024-07-26T08:38:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6168.993619 secs 2024-07-26T08:38:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:42 smithi163 ceph-mon[36873]: pgmap v3848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6151.251064 secs 2024-07-26T08:38:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6168.993619 secs 2024-07-26T08:38:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:44 smithi042 ceph-mon[32775]: pgmap v3849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:44 smithi164 ceph-mon[36970]: pgmap v3849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:44 smithi163 ceph-mon[36873]: pgmap v3849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:46.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:46 smithi042 ceph-mon[32775]: pgmap v3850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:46 smithi164 ceph-mon[36970]: pgmap v3850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:46 smithi163 ceph-mon[36873]: pgmap v3850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6156.251190 secs 2024-07-26T08:38:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6173.993741 secs 2024-07-26T08:38:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6156.251190 secs 2024-07-26T08:38:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6173.993741 secs 2024-07-26T08:38:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6156.251190 secs 2024-07-26T08:38:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6173.993741 secs 2024-07-26T08:38:48.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:48 smithi042 ceph-mon[32775]: pgmap v3851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:48 smithi164 ceph-mon[36970]: pgmap v3851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:48 smithi163 ceph-mon[36873]: pgmap v3851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:50 smithi042 ceph-mon[32775]: pgmap v3852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:38:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:50 smithi164 ceph-mon[36970]: pgmap v3852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:38:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:50 smithi163 ceph-mon[36873]: pgmap v3852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:38:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:52 smithi042 ceph-mon[32775]: pgmap v3853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6161.251315 secs 2024-07-26T08:38:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6178.993876 secs 2024-07-26T08:38:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:52 smithi164 ceph-mon[36970]: pgmap v3853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6161.251315 secs 2024-07-26T08:38:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6178.993876 secs 2024-07-26T08:38:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:52 smithi163 ceph-mon[36873]: pgmap v3853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6161.251315 secs 2024-07-26T08:38:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6178.993876 secs 2024-07-26T08:38:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:54 smithi042 ceph-mon[32775]: pgmap v3854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:54 smithi164 ceph-mon[36970]: pgmap v3854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:54 smithi163 ceph-mon[36873]: pgmap v3854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:38:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:38:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:38:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:38:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:38:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:38:55.414+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:38:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:38:55.414+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:38:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:38:55.414+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:38:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:38:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:38:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:38:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:38:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:38:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:38:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:38:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:38:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:56 smithi042 ceph-mon[32775]: pgmap v3855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:56 smithi042 ceph-mon[32775]: pgmap v3856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:38:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:56 smithi042 ceph-mon[32775]: pgmap v3857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:38:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:56 smithi164 ceph-mon[36970]: pgmap v3855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:56 smithi164 ceph-mon[36970]: pgmap v3856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:38:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:56 smithi164 ceph-mon[36970]: pgmap v3857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:38:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:56 smithi163 ceph-mon[36873]: pgmap v3855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:38:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:56 smithi163 ceph-mon[36873]: pgmap v3856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:38:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:56 smithi163 ceph-mon[36873]: pgmap v3857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:38:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6166.251402 secs 2024-07-26T08:38:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6183.994016 secs 2024-07-26T08:38:57.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6166.251402 secs 2024-07-26T08:38:57.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6183.994016 secs 2024-07-26T08:38:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6166.251402 secs 2024-07-26T08:38:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6183.994016 secs 2024-07-26T08:38:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:38:58 smithi042 ceph-mon[32775]: pgmap v3858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:38:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:38:58 smithi164 ceph-mon[36970]: pgmap v3858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:38:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:38:58 smithi163 ceph-mon[36873]: pgmap v3858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:39:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:00 smithi042 ceph-mon[32775]: pgmap v3859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:39:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:00 smithi164 ceph-mon[36970]: pgmap v3859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:39:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:00 smithi163 ceph-mon[36873]: pgmap v3859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:39:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6171.251532 secs 2024-07-26T08:39:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:02 smithi042 ceph-mon[32775]: pgmap v3860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:39:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6188.994082 secs 2024-07-26T08:39:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6171.251532 secs 2024-07-26T08:39:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:02 smithi164 ceph-mon[36970]: pgmap v3860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:39:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6188.994082 secs 2024-07-26T08:39:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6171.251532 secs 2024-07-26T08:39:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:02 smithi163 ceph-mon[36873]: pgmap v3860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:39:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6188.994082 secs 2024-07-26T08:39:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:04 smithi042 ceph-mon[32775]: pgmap v3861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:39:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:04 smithi164 ceph-mon[36970]: pgmap v3861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:39:04.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:04 smithi163 ceph-mon[36873]: pgmap v3861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:39:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:06 smithi042 ceph-mon[32775]: pgmap v3862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:39:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:06 smithi164 ceph-mon[36970]: pgmap v3862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:39:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:06 smithi163 ceph-mon[36873]: pgmap v3862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:39:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6176.251662 secs 2024-07-26T08:39:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6193.994148 secs 2024-07-26T08:39:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6176.251662 secs 2024-07-26T08:39:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6193.994148 secs 2024-07-26T08:39:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6176.251662 secs 2024-07-26T08:39:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6193.994148 secs 2024-07-26T08:39:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:08 smithi042 ceph-mon[32775]: pgmap v3863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:08 smithi164 ceph-mon[36970]: pgmap v3863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:08 smithi163 ceph-mon[36873]: pgmap v3863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:10 smithi042 ceph-mon[32775]: pgmap v3864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:39:10.424 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:39:10.422+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:39:10.424 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:39:10.422+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:39:10.424 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:39:10.422+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:39:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:10 smithi164 ceph-mon[36970]: pgmap v3864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:39:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:10 smithi163 ceph-mon[36873]: pgmap v3864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:39:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6181.251765 secs 2024-07-26T08:39:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:12 smithi042 ceph-mon[32775]: pgmap v3865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6198.994270 secs 2024-07-26T08:39:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6181.251765 secs 2024-07-26T08:39:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:12 smithi164 ceph-mon[36970]: pgmap v3865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6198.994270 secs 2024-07-26T08:39:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6181.251765 secs 2024-07-26T08:39:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:12 smithi163 ceph-mon[36873]: pgmap v3865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6198.994270 secs 2024-07-26T08:39:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:14 smithi042 ceph-mon[32775]: pgmap v3866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:14 smithi164 ceph-mon[36970]: pgmap v3866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:14 smithi163 ceph-mon[36873]: pgmap v3866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:16 smithi042 ceph-mon[32775]: pgmap v3867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:16 smithi164 ceph-mon[36970]: pgmap v3867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:16 smithi163 ceph-mon[36873]: pgmap v3867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6186.251895 secs 2024-07-26T08:39:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6203.994393 secs 2024-07-26T08:39:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6186.251895 secs 2024-07-26T08:39:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6203.994393 secs 2024-07-26T08:39:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6186.251895 secs 2024-07-26T08:39:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6203.994393 secs 2024-07-26T08:39:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:18 smithi042 ceph-mon[32775]: pgmap v3868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:18 smithi164 ceph-mon[36970]: pgmap v3868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:18 smithi163 ceph-mon[36873]: pgmap v3868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:20 smithi042 ceph-mon[32775]: pgmap v3869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:39:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:20 smithi164 ceph-mon[36970]: pgmap v3869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:39:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:20 smithi163 ceph-mon[36873]: pgmap v3869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:39:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6191.252022 secs 2024-07-26T08:39:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:22 smithi042 ceph-mon[32775]: pgmap v3870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6208.994518 secs 2024-07-26T08:39:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6191.252022 secs 2024-07-26T08:39:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:22 smithi164 ceph-mon[36970]: pgmap v3870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6208.994518 secs 2024-07-26T08:39:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6191.252022 secs 2024-07-26T08:39:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:22 smithi163 ceph-mon[36873]: pgmap v3870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6208.994518 secs 2024-07-26T08:39:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:24 smithi042 ceph-mon[32775]: pgmap v3871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:24 smithi164 ceph-mon[36970]: pgmap v3871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:24 smithi163 ceph-mon[36873]: pgmap v3871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:25.423 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:39:25.422+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:39:25.424 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:39:25.422+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:39:25.424 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:39:25.422+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:39:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:26 smithi042 ceph-mon[32775]: pgmap v3872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:26 smithi164 ceph-mon[36970]: pgmap v3872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:26 smithi163 ceph-mon[36873]: pgmap v3872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6196.252157 secs 2024-07-26T08:39:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6213.994639 secs 2024-07-26T08:39:27.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6196.252157 secs 2024-07-26T08:39:27.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6213.994639 secs 2024-07-26T08:39:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6196.252157 secs 2024-07-26T08:39:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6213.994639 secs 2024-07-26T08:39:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:28 smithi042 ceph-mon[32775]: pgmap v3873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:28 smithi164 ceph-mon[36970]: pgmap v3873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:28 smithi163 ceph-mon[36873]: pgmap v3873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:30 smithi042 ceph-mon[32775]: pgmap v3874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:39:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:30 smithi164 ceph-mon[36970]: pgmap v3874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:39:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:30 smithi163 ceph-mon[36873]: pgmap v3874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:39:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6201.252282 secs 2024-07-26T08:39:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:32 smithi042 ceph-mon[32775]: pgmap v3875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6218.994736 secs 2024-07-26T08:39:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6201.252282 secs 2024-07-26T08:39:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:32 smithi164 ceph-mon[36970]: pgmap v3875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6218.994736 secs 2024-07-26T08:39:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6201.252282 secs 2024-07-26T08:39:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:32 smithi163 ceph-mon[36873]: pgmap v3875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6218.994736 secs 2024-07-26T08:39:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:34 smithi042 ceph-mon[32775]: pgmap v3876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:34 smithi164 ceph-mon[36970]: pgmap v3876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:34 smithi163 ceph-mon[36873]: pgmap v3876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:36 smithi042 ceph-mon[32775]: pgmap v3877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:36 smithi164 ceph-mon[36970]: pgmap v3877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:36 smithi163 ceph-mon[36873]: pgmap v3877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6206.252420 secs 2024-07-26T08:39:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6223.994805 secs 2024-07-26T08:39:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6206.252420 secs 2024-07-26T08:39:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6223.994805 secs 2024-07-26T08:39:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6206.252420 secs 2024-07-26T08:39:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6223.994805 secs 2024-07-26T08:39:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:38 smithi164 ceph-mon[36970]: pgmap v3878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:38 smithi163 ceph-mon[36873]: pgmap v3878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:38 smithi042 ceph-mon[32775]: pgmap v3878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:40.432 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:39:40.431+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:39:40.433 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:39:40.431+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:39:40.433 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:39:40.431+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:39:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:40 smithi164 ceph-mon[36970]: pgmap v3879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:39:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:40 smithi163 ceph-mon[36873]: pgmap v3879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:39:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:40 smithi042 ceph-mon[32775]: pgmap v3879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:39:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6211.252547 secs 2024-07-26T08:39:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:42 smithi164 ceph-mon[36970]: pgmap v3880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6228.994997 secs 2024-07-26T08:39:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6211.252547 secs 2024-07-26T08:39:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:42 smithi163 ceph-mon[36873]: pgmap v3880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6228.994997 secs 2024-07-26T08:39:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6211.252547 secs 2024-07-26T08:39:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:42 smithi042 ceph-mon[32775]: pgmap v3880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6228.994997 secs 2024-07-26T08:39:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:44 smithi164 ceph-mon[36970]: pgmap v3881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:44 smithi163 ceph-mon[36873]: pgmap v3881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:44 smithi042 ceph-mon[32775]: pgmap v3881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:46 smithi164 ceph-mon[36970]: pgmap v3882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:46 smithi163 ceph-mon[36873]: pgmap v3882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:46 smithi042 ceph-mon[32775]: pgmap v3882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6216.252676 secs 2024-07-26T08:39:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6233.995138 secs 2024-07-26T08:39:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6216.252676 secs 2024-07-26T08:39:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6233.995138 secs 2024-07-26T08:39:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6216.252676 secs 2024-07-26T08:39:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6233.995138 secs 2024-07-26T08:39:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:48 smithi164 ceph-mon[36970]: pgmap v3883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:48 smithi163 ceph-mon[36873]: pgmap v3883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:48.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:48 smithi042 ceph-mon[32775]: pgmap v3883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:50 smithi164 ceph-mon[36970]: pgmap v3884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:39:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:50 smithi163 ceph-mon[36873]: pgmap v3884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:39:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:50 smithi042 ceph-mon[32775]: pgmap v3884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:39:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6221.252825 secs 2024-07-26T08:39:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:52 smithi164 ceph-mon[36970]: pgmap v3885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6238.995260 secs 2024-07-26T08:39:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6221.252825 secs 2024-07-26T08:39:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:52 smithi163 ceph-mon[36873]: pgmap v3885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6238.995260 secs 2024-07-26T08:39:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6221.252825 secs 2024-07-26T08:39:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:52 smithi042 ceph-mon[32775]: pgmap v3885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6238.995260 secs 2024-07-26T08:39:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:54 smithi164 ceph-mon[36970]: pgmap v3886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:54 smithi163 ceph-mon[36873]: pgmap v3886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:54 smithi042 ceph-mon[32775]: pgmap v3886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:39:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:39:55.434+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:39:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:39:55.434+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:39:55.435 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:39:55.434+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:39:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:39:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:39:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:39:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:56 smithi164 ceph-mon[36970]: pgmap v3887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:39:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:39:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:39:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:56 smithi163 ceph-mon[36873]: pgmap v3887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:39:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:39:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:39:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:56 smithi042 ceph-mon[32775]: pgmap v3887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:39:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:39:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:39:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:39:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:57 smithi164 ceph-mon[36970]: pgmap v3888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:39:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:57 smithi164 ceph-mon[36970]: pgmap v3889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:39:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6226.252948 secs 2024-07-26T08:39:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6243.995384 secs 2024-07-26T08:39:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:57 smithi163 ceph-mon[36873]: pgmap v3888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:39:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:57 smithi163 ceph-mon[36873]: pgmap v3889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:39:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6226.252948 secs 2024-07-26T08:39:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6243.995384 secs 2024-07-26T08:39:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:57 smithi042 ceph-mon[32775]: pgmap v3888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:39:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:57 smithi042 ceph-mon[32775]: pgmap v3889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:39:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6226.252948 secs 2024-07-26T08:39:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6243.995384 secs 2024-07-26T08:39:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:39:59 smithi164 ceph-mon[36970]: pgmap v3890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:39:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:39:59 smithi163 ceph-mon[36873]: pgmap v3890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:39:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:39:59 smithi042 ceph-mon[32775]: pgmap v3890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:40:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:40:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:40:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:40:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:01 smithi164 ceph-mon[36970]: pgmap v3891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:40:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:01 smithi163 ceph-mon[36873]: pgmap v3891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:40:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:01 smithi042 ceph-mon[32775]: pgmap v3891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:40:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6231.253075 secs 2024-07-26T08:40:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6248.995503 secs 2024-07-26T08:40:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6231.253075 secs 2024-07-26T08:40:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6248.995503 secs 2024-07-26T08:40:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6231.253075 secs 2024-07-26T08:40:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6248.995503 secs 2024-07-26T08:40:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:03 smithi164 ceph-mon[36970]: pgmap v3892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:40:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:03 smithi163 ceph-mon[36873]: pgmap v3892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:40:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:03 smithi042 ceph-mon[32775]: pgmap v3892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:40:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:05 smithi164 ceph-mon[36970]: pgmap v3893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:40:05.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:05 smithi163 ceph-mon[36873]: pgmap v3893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:40:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:05 smithi042 ceph-mon[32775]: pgmap v3893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:40:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:07 smithi164 ceph-mon[36970]: pgmap v3894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:40:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6236.253206 secs 2024-07-26T08:40:07.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6253.995650 secs 2024-07-26T08:40:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:07 smithi163 ceph-mon[36873]: pgmap v3894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:40:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6236.253206 secs 2024-07-26T08:40:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6253.995650 secs 2024-07-26T08:40:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:07 smithi042 ceph-mon[32775]: pgmap v3894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:40:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6236.253206 secs 2024-07-26T08:40:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6253.995650 secs 2024-07-26T08:40:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:09 smithi164 ceph-mon[36970]: pgmap v3895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:09 smithi163 ceph-mon[36873]: pgmap v3895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:09 smithi042 ceph-mon[32775]: pgmap v3895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:10.443+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:40:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:10.443+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:40:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:10.443+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:40:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:11 smithi164 ceph-mon[36970]: pgmap v3896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:11 smithi163 ceph-mon[36873]: pgmap v3896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:11 smithi042 ceph-mon[32775]: pgmap v3896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6241.253338 secs 2024-07-26T08:40:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6258.995771 secs 2024-07-26T08:40:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6241.253338 secs 2024-07-26T08:40:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6258.995771 secs 2024-07-26T08:40:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6241.253338 secs 2024-07-26T08:40:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6258.995771 secs 2024-07-26T08:40:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:13 smithi163 ceph-mon[36873]: pgmap v3897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:13 smithi042 ceph-mon[32775]: pgmap v3897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:13 smithi164 ceph-mon[36970]: pgmap v3897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:15 smithi163 ceph-mon[36873]: pgmap v3898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:40:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:15 smithi042 ceph-mon[32775]: pgmap v3898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:40:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:15 smithi164 ceph-mon[36970]: pgmap v3898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:40:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:17 smithi163 ceph-mon[36873]: pgmap v3899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6246.253468 secs 2024-07-26T08:40:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6263.995895 secs 2024-07-26T08:40:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:17 smithi042 ceph-mon[32775]: pgmap v3899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6246.253468 secs 2024-07-26T08:40:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6263.995895 secs 2024-07-26T08:40:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:17 smithi164 ceph-mon[36970]: pgmap v3899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6246.253468 secs 2024-07-26T08:40:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6263.995895 secs 2024-07-26T08:40:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:19 smithi163 ceph-mon[36873]: pgmap v3900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:19.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:19 smithi042 ceph-mon[32775]: pgmap v3900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:19 smithi164 ceph-mon[36970]: pgmap v3900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:21 smithi163 ceph-mon[36873]: pgmap v3901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:21 smithi042 ceph-mon[32775]: pgmap v3901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:21 smithi164 ceph-mon[36970]: pgmap v3901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:22.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6251.253601 secs 2024-07-26T08:40:22.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6268.996000 secs 2024-07-26T08:40:22.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:22 smithi163 ceph-mon[36873]: pgmap v3902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6251.253601 secs 2024-07-26T08:40:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6268.996000 secs 2024-07-26T08:40:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:22 smithi042 ceph-mon[32775]: pgmap v3902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6251.253601 secs 2024-07-26T08:40:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6268.996000 secs 2024-07-26T08:40:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:22 smithi164 ceph-mon[36970]: pgmap v3902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:24 smithi042 ceph-mon[32775]: pgmap v3903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:40:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:24 smithi164 ceph-mon[36970]: pgmap v3903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:40:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:24 smithi163 ceph-mon[36873]: pgmap v3903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:40:25.449 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:25.447+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:40:25.449 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:25.447+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:40:25.449 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:25.447+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:40:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:26 smithi042 ceph-mon[32775]: pgmap v3904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:26 smithi164 ceph-mon[36970]: pgmap v3904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:26 smithi163 ceph-mon[36873]: pgmap v3904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6256.253732 secs 2024-07-26T08:40:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6273.996121 secs 2024-07-26T08:40:27.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6256.253732 secs 2024-07-26T08:40:27.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6273.996121 secs 2024-07-26T08:40:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6256.253732 secs 2024-07-26T08:40:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6273.996121 secs 2024-07-26T08:40:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:28 smithi042 ceph-mon[32775]: pgmap v3905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:28 smithi164 ceph-mon[36970]: pgmap v3905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:29.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:28 smithi163 ceph-mon[36873]: pgmap v3905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:30 smithi042 ceph-mon[32775]: pgmap v3906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:30 smithi164 ceph-mon[36970]: pgmap v3906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:31.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:30 smithi163 ceph-mon[36873]: pgmap v3906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6261.253858 secs 2024-07-26T08:40:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6278.996245 secs 2024-07-26T08:40:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:32 smithi042 ceph-mon[32775]: pgmap v3907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6261.253858 secs 2024-07-26T08:40:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6278.996245 secs 2024-07-26T08:40:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:32 smithi164 ceph-mon[36970]: pgmap v3907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6261.253858 secs 2024-07-26T08:40:33.191 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6278.996245 secs 2024-07-26T08:40:33.191 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:32 smithi163 ceph-mon[36873]: pgmap v3907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:34 smithi042 ceph-mon[32775]: pgmap v3908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:40:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:34 smithi164 ceph-mon[36970]: pgmap v3908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:40:35.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:34 smithi163 ceph-mon[36873]: pgmap v3908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:40:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:36 smithi042 ceph-mon[32775]: pgmap v3909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:36 smithi164 ceph-mon[36970]: pgmap v3909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:36 smithi163 ceph-mon[36873]: pgmap v3909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6266.253991 secs 2024-07-26T08:40:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6283.996365 secs 2024-07-26T08:40:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6266.253991 secs 2024-07-26T08:40:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6283.996365 secs 2024-07-26T08:40:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6266.253991 secs 2024-07-26T08:40:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6283.996365 secs 2024-07-26T08:40:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:38 smithi042 ceph-mon[32775]: pgmap v3910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:38 smithi164 ceph-mon[36970]: pgmap v3910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:38 smithi163 ceph-mon[36873]: pgmap v3910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:40.456+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:40:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:40.456+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:40:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:40.456+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:40:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:40 smithi042 ceph-mon[32775]: pgmap v3911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:40 smithi164 ceph-mon[36970]: pgmap v3911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:40 smithi163 ceph-mon[36873]: pgmap v3911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6271.254090 secs 2024-07-26T08:40:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6288.996493 secs 2024-07-26T08:40:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:42 smithi042 ceph-mon[32775]: pgmap v3912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6271.254090 secs 2024-07-26T08:40:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6288.996493 secs 2024-07-26T08:40:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:42 smithi164 ceph-mon[36970]: pgmap v3912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6271.254090 secs 2024-07-26T08:40:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6288.996493 secs 2024-07-26T08:40:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:42 smithi163 ceph-mon[36873]: pgmap v3912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:44 smithi042 ceph-mon[32775]: pgmap v3913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:40:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:44 smithi164 ceph-mon[36970]: pgmap v3913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:40:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:44 smithi163 ceph-mon[36873]: pgmap v3913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:40:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:46 smithi042 ceph-mon[32775]: pgmap v3914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6276.254248 secs 2024-07-26T08:40:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:46 smithi164 ceph-mon[36970]: pgmap v3914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6276.254248 secs 2024-07-26T08:40:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:46 smithi163 ceph-mon[36873]: pgmap v3914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6276.254248 secs 2024-07-26T08:40:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6293.996614 secs 2024-07-26T08:40:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6293.996614 secs 2024-07-26T08:40:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6293.996614 secs 2024-07-26T08:40:48.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:48 smithi042 ceph-mon[32775]: pgmap v3915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:48 smithi164 ceph-mon[36970]: pgmap v3915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:48 smithi163 ceph-mon[36873]: pgmap v3915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:50 smithi042 ceph-mon[32775]: pgmap v3916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:50 smithi164 ceph-mon[36970]: pgmap v3916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:50 smithi163 ceph-mon[36873]: pgmap v3916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6281.254352 secs 2024-07-26T08:40:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6281.254352 secs 2024-07-26T08:40:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6281.254352 secs 2024-07-26T08:40:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6298.996754 secs 2024-07-26T08:40:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:52 smithi164 ceph-mon[36970]: pgmap v3917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:52.976 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6298.996754 secs 2024-07-26T08:40:52.976 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:52 smithi042 ceph-mon[32775]: pgmap v3917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6298.996754 secs 2024-07-26T08:40:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:52 smithi163 ceph-mon[36873]: pgmap v3917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:54 smithi164 ceph-mon[36970]: pgmap v3918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:40:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:54 smithi163 ceph-mon[36873]: pgmap v3918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:40:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:54 smithi042 ceph-mon[32775]: pgmap v3918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:40:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:55.460+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:40:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:55.460+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:40:55.461 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:55.460+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:40:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:40:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:40:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:40:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:56 smithi163 ceph-mon[36873]: pgmap v3919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6286.254423 secs 2024-07-26T08:40:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:56 smithi042 ceph-mon[32775]: pgmap v3919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6286.254423 secs 2024-07-26T08:40:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:56 smithi164 ceph-mon[36970]: pgmap v3919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:40:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6286.254423 secs 2024-07-26T08:40:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:57.871+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T08:40:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:57.871+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T08:40:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:57.871+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b4d80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff6780318b0 tx=0x7ff678050000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:40:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:57.871+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b4d80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff6780318b0 tx=0x7ff678050000 comp rx=0 tx=0).stop 2024-07-26T08:40:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:57.871+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0b4d80 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T08:40:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:57.871+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0b4d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T08:40:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:57.871+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b4d80 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:40:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:57.871+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:40:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:57.871+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:40:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:57.872+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff678031980 tx=0x7ff67803e040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:40:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6303.996860 secs 2024-07-26T08:40:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6303.996860 secs 2024-07-26T08:40:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6303.996860 secs 2024-07-26T08:40:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:58.840 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:58.839+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:40:58.841 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:58.839+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:40:58.845 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:58.844+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:40:58.845 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:40:58.844+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:58 smithi164 ceph-mon[36970]: pgmap v3920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:58 smithi163 ceph-mon[36873]: pgmap v3920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:40:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:40:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:58 smithi042 ceph-mon[32775]: pgmap v3920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:40:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:40:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:40:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:40:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:59 smithi164 ceph-mon[36970]: pgmap v3921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:40:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:40:59 smithi164 ceph-mon[36970]: pgmap v3922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T08:41:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:59 smithi163 ceph-mon[36873]: pgmap v3921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:41:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:40:59 smithi163 ceph-mon[36873]: pgmap v3922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T08:41:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:59 smithi042 ceph-mon[32775]: pgmap v3921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:41:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:40:59 smithi042 ceph-mon[32775]: pgmap v3922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T08:41:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:01 smithi164 ceph-mon[36970]: pgmap v3923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:41:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6291.254548 secs 2024-07-26T08:41:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:01 smithi163 ceph-mon[36873]: pgmap v3923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:41:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6291.254548 secs 2024-07-26T08:41:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:01 smithi042 ceph-mon[32775]: pgmap v3923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:41:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6291.254548 secs 2024-07-26T08:41:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6308.996989 secs 2024-07-26T08:41:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6308.996989 secs 2024-07-26T08:41:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6308.996989 secs 2024-07-26T08:41:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:03 smithi164 ceph-mon[36970]: pgmap v3924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:41:04.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:03 smithi163 ceph-mon[36873]: pgmap v3924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:41:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:03 smithi042 ceph-mon[32775]: pgmap v3924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:41:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:05 smithi164 ceph-mon[36970]: pgmap v3925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:41:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:05 smithi163 ceph-mon[36873]: pgmap v3925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:41:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:05 smithi042 ceph-mon[32775]: pgmap v3925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:41:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6296.254631 secs 2024-07-26T08:41:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6296.254631 secs 2024-07-26T08:41:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6296.254631 secs 2024-07-26T08:41:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6313.997089 secs 2024-07-26T08:41:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:07 smithi164 ceph-mon[36970]: pgmap v3926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:41:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6313.997089 secs 2024-07-26T08:41:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:07 smithi163 ceph-mon[36873]: pgmap v3926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:41:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6313.997089 secs 2024-07-26T08:41:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:07 smithi042 ceph-mon[32775]: pgmap v3926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:41:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:09 smithi164 ceph-mon[36970]: pgmap v3927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:41:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:09 smithi163 ceph-mon[36873]: pgmap v3927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:41:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:09 smithi042 ceph-mon[32775]: pgmap v3927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:41:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:41:10.469+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:41:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:41:10.469+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:41:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:41:10.469+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:41:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:11 smithi164 ceph-mon[36970]: pgmap v3928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6301.254787 secs 2024-07-26T08:41:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:11 smithi163 ceph-mon[36873]: pgmap v3928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6301.254787 secs 2024-07-26T08:41:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:11 smithi042 ceph-mon[32775]: pgmap v3928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6301.254787 secs 2024-07-26T08:41:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6318.997212 secs 2024-07-26T08:41:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6318.997212 secs 2024-07-26T08:41:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6318.997212 secs 2024-07-26T08:41:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:13 smithi164 ceph-mon[36970]: pgmap v3929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:13 smithi163 ceph-mon[36873]: pgmap v3929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:13 smithi042 ceph-mon[32775]: pgmap v3929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:15 smithi164 ceph-mon[36970]: pgmap v3930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:15 smithi163 ceph-mon[36873]: pgmap v3930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:15 smithi042 ceph-mon[32775]: pgmap v3930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6306.254919 secs 2024-07-26T08:41:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6306.254919 secs 2024-07-26T08:41:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6306.254919 secs 2024-07-26T08:41:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6323.997332 secs 2024-07-26T08:41:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:17 smithi164 ceph-mon[36970]: pgmap v3931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6323.997332 secs 2024-07-26T08:41:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:17 smithi163 ceph-mon[36873]: pgmap v3931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6323.997332 secs 2024-07-26T08:41:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:17 smithi042 ceph-mon[32775]: pgmap v3931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:19 smithi164 ceph-mon[36970]: pgmap v3932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:41:20.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:19 smithi163 ceph-mon[36873]: pgmap v3932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:41:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:19 smithi042 ceph-mon[32775]: pgmap v3932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:41:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:21 smithi164 ceph-mon[36970]: pgmap v3933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6311.255021 secs 2024-07-26T08:41:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:21 smithi163 ceph-mon[36873]: pgmap v3933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6311.255021 secs 2024-07-26T08:41:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:21 smithi042 ceph-mon[32775]: pgmap v3933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6311.255021 secs 2024-07-26T08:41:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6328.997455 secs 2024-07-26T08:41:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6328.997455 secs 2024-07-26T08:41:23.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6328.997455 secs 2024-07-26T08:41:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:23 smithi164 ceph-mon[36970]: pgmap v3934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:23 smithi163 ceph-mon[36873]: pgmap v3934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:24.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:23 smithi042 ceph-mon[32775]: pgmap v3934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:41:25.473+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:41:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:41:25.473+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:41:25.475 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:41:25.473+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:41:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:25 smithi163 ceph-mon[36873]: pgmap v3935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:25 smithi042 ceph-mon[32775]: pgmap v3935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:25 smithi164 ceph-mon[36970]: pgmap v3935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6316.255175 secs 2024-07-26T08:41:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6316.255175 secs 2024-07-26T08:41:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6316.255175 secs 2024-07-26T08:41:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6333.997575 secs 2024-07-26T08:41:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:27 smithi163 ceph-mon[36873]: pgmap v3936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6333.997575 secs 2024-07-26T08:41:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:27 smithi042 ceph-mon[32775]: pgmap v3936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6333.997575 secs 2024-07-26T08:41:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:27 smithi164 ceph-mon[36970]: pgmap v3936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:29 smithi163 ceph-mon[36873]: pgmap v3937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:41:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:29 smithi042 ceph-mon[32775]: pgmap v3937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:41:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:29 smithi164 ceph-mon[36970]: pgmap v3937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:41:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:31 smithi163 ceph-mon[36873]: pgmap v3938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6321.255313 secs 2024-07-26T08:41:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:31 smithi042 ceph-mon[32775]: pgmap v3938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6321.255313 secs 2024-07-26T08:41:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:31 smithi164 ceph-mon[36970]: pgmap v3938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6321.255313 secs 2024-07-26T08:41:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6338.997704 secs 2024-07-26T08:41:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6338.997704 secs 2024-07-26T08:41:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6338.997704 secs 2024-07-26T08:41:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:33 smithi163 ceph-mon[36873]: pgmap v3939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:34.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:33 smithi042 ceph-mon[32775]: pgmap v3939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:33 smithi164 ceph-mon[36970]: pgmap v3939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:36.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:35 smithi163 ceph-mon[36873]: pgmap v3940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:35 smithi042 ceph-mon[32775]: pgmap v3940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:35 smithi164 ceph-mon[36970]: pgmap v3940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6326.255451 secs 2024-07-26T08:41:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6326.255451 secs 2024-07-26T08:41:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6326.255451 secs 2024-07-26T08:41:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6343.997849 secs 2024-07-26T08:41:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:37 smithi042 ceph-mon[32775]: pgmap v3941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6343.997849 secs 2024-07-26T08:41:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:37 smithi164 ceph-mon[36970]: pgmap v3941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6343.997849 secs 2024-07-26T08:41:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:37 smithi163 ceph-mon[36873]: pgmap v3941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:39 smithi042 ceph-mon[32775]: pgmap v3942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:41:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:39 smithi164 ceph-mon[36970]: pgmap v3942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:41:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:39 smithi163 ceph-mon[36873]: pgmap v3942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:41:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:41:40.482+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:41:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:41:40.482+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:41:40.484 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:41:40.482+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:41:42.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:41 smithi164 ceph-mon[36970]: pgmap v3943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:42.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6331.255587 secs 2024-07-26T08:41:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:41 smithi042 ceph-mon[32775]: pgmap v3943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6331.255587 secs 2024-07-26T08:41:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:41 smithi163 ceph-mon[36873]: pgmap v3943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6331.255587 secs 2024-07-26T08:41:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6348.997971 secs 2024-07-26T08:41:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6348.997971 secs 2024-07-26T08:41:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6348.997971 secs 2024-07-26T08:41:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:43 smithi042 ceph-mon[32775]: pgmap v3944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:43 smithi164 ceph-mon[36970]: pgmap v3944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:43 smithi163 ceph-mon[36873]: pgmap v3944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:46.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:45 smithi042 ceph-mon[32775]: pgmap v3945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:45 smithi164 ceph-mon[36970]: pgmap v3945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:45 smithi163 ceph-mon[36873]: pgmap v3945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6336.255733 secs 2024-07-26T08:41:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6336.255733 secs 2024-07-26T08:41:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6336.255733 secs 2024-07-26T08:41:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6353.998096 secs 2024-07-26T08:41:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:47 smithi042 ceph-mon[32775]: pgmap v3946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6353.998096 secs 2024-07-26T08:41:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:47 smithi164 ceph-mon[36970]: pgmap v3946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6353.998096 secs 2024-07-26T08:41:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:47 smithi163 ceph-mon[36873]: pgmap v3946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:50.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:49 smithi042 ceph-mon[32775]: pgmap v3947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:41:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:49 smithi164 ceph-mon[36970]: pgmap v3947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:41:50.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:49 smithi163 ceph-mon[36873]: pgmap v3947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:41:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:51 smithi042 ceph-mon[32775]: pgmap v3948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6341.255800 secs 2024-07-26T08:41:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:51 smithi164 ceph-mon[36970]: pgmap v3948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6341.255800 secs 2024-07-26T08:41:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:51 smithi163 ceph-mon[36873]: pgmap v3948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6341.255800 secs 2024-07-26T08:41:53.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6358.998220 secs 2024-07-26T08:41:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6358.998220 secs 2024-07-26T08:41:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6358.998220 secs 2024-07-26T08:41:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:53 smithi042 ceph-mon[32775]: pgmap v3949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:53 smithi164 ceph-mon[36970]: pgmap v3949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:53 smithi163 ceph-mon[36873]: pgmap v3949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:41:55.486+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:41:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:41:55.486+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:41:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:41:55.486+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:41:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:55 smithi042 ceph-mon[32775]: pgmap v3950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:55 smithi164 ceph-mon[36970]: pgmap v3950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:56.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:55 smithi163 ceph-mon[36873]: pgmap v3950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:41:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6346.255943 secs 2024-07-26T08:41:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6346.255943 secs 2024-07-26T08:41:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6346.255943 secs 2024-07-26T08:41:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6363.998346 secs 2024-07-26T08:41:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:57 smithi042 ceph-mon[32775]: pgmap v3951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6363.998346 secs 2024-07-26T08:41:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:57 smithi164 ceph-mon[36970]: pgmap v3951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6363.998346 secs 2024-07-26T08:41:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:57 smithi163 ceph-mon[36873]: pgmap v3951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:41:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:41:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:41:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:41:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:41:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:41:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:41:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:41:59.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:41:59.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:41:59.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:41:59.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:41:59.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:41:59.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:41:59.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:41:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:41:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:41:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:41:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:41:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:41:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:41:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:42:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:59 smithi042 ceph-mon[32775]: pgmap v3952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:42:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:42:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:42:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:42:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:42:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:42:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:41:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:42:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:59 smithi164 ceph-mon[36970]: pgmap v3952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:42:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:42:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:42:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:42:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:42:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:42:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:41:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:42:00.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:59 smithi163 ceph-mon[36873]: pgmap v3952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:42:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:42:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:42:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:42:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:42:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:42:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:41:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:42:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:00 smithi042 ceph-mon[32775]: pgmap v3953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-07-26T08:42:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:00 smithi042 ceph-mon[32775]: pgmap v3954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T08:42:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:00 smithi164 ceph-mon[36970]: pgmap v3953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-07-26T08:42:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:00 smithi164 ceph-mon[36970]: pgmap v3954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T08:42:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:00 smithi163 ceph-mon[36873]: pgmap v3953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-07-26T08:42:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:00 smithi163 ceph-mon[36873]: pgmap v3954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T08:42:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6351.256055 secs 2024-07-26T08:42:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6351.256055 secs 2024-07-26T08:42:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6351.256055 secs 2024-07-26T08:42:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6368.998468 secs 2024-07-26T08:42:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:02 smithi042 ceph-mon[32775]: pgmap v3955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T08:42:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6368.998468 secs 2024-07-26T08:42:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:02 smithi164 ceph-mon[36970]: pgmap v3955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T08:42:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6368.998468 secs 2024-07-26T08:42:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:02 smithi163 ceph-mon[36873]: pgmap v3955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T08:42:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:04 smithi042 ceph-mon[32775]: pgmap v3956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T08:42:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:04 smithi164 ceph-mon[36970]: pgmap v3956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T08:42:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:04 smithi163 ceph-mon[36873]: pgmap v3956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T08:42:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:06 smithi042 ceph-mon[32775]: pgmap v3957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T08:42:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6356.256189 secs 2024-07-26T08:42:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:06 smithi164 ceph-mon[36970]: pgmap v3957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T08:42:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6356.256189 secs 2024-07-26T08:42:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:06 smithi163 ceph-mon[36873]: pgmap v3957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T08:42:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6356.256189 secs 2024-07-26T08:42:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6373.998588 secs 2024-07-26T08:42:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6373.998588 secs 2024-07-26T08:42:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6373.998588 secs 2024-07-26T08:42:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:08 smithi042 ceph-mon[32775]: pgmap v3958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T08:42:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:08 smithi164 ceph-mon[36970]: pgmap v3958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T08:42:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:08 smithi163 ceph-mon[36873]: pgmap v3958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T08:42:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:42:10.494+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:42:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:42:10.494+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:42:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:42:10.494+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:42:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:10 smithi042 ceph-mon[32775]: pgmap v3959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:42:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:10 smithi164 ceph-mon[36970]: pgmap v3959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:42:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:10 smithi163 ceph-mon[36873]: pgmap v3959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:42:12.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6361.256312 secs 2024-07-26T08:42:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6361.256312 secs 2024-07-26T08:42:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6361.256312 secs 2024-07-26T08:42:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6378.998727 secs 2024-07-26T08:42:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:12 smithi042 ceph-mon[32775]: pgmap v3960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6378.998727 secs 2024-07-26T08:42:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:12 smithi164 ceph-mon[36970]: pgmap v3960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6378.998727 secs 2024-07-26T08:42:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:12 smithi163 ceph-mon[36873]: pgmap v3960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:14 smithi042 ceph-mon[32775]: pgmap v3961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:14 smithi164 ceph-mon[36970]: pgmap v3961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:14 smithi163 ceph-mon[36873]: pgmap v3961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:16 smithi042 ceph-mon[32775]: pgmap v3962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6366.256430 secs 2024-07-26T08:42:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:16 smithi164 ceph-mon[36970]: pgmap v3962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6366.256430 secs 2024-07-26T08:42:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:16 smithi163 ceph-mon[36873]: pgmap v3962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6366.256430 secs 2024-07-26T08:42:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6383.998859 secs 2024-07-26T08:42:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6383.998859 secs 2024-07-26T08:42:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6383.998859 secs 2024-07-26T08:42:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:18 smithi042 ceph-mon[32775]: pgmap v3963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:18 smithi164 ceph-mon[36970]: pgmap v3963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:18 smithi163 ceph-mon[36873]: pgmap v3963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:20 smithi042 ceph-mon[32775]: pgmap v3964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:20 smithi164 ceph-mon[36970]: pgmap v3964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:20 smithi163 ceph-mon[36873]: pgmap v3964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6371.256567 secs 2024-07-26T08:42:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6371.256567 secs 2024-07-26T08:42:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6371.256567 secs 2024-07-26T08:42:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6388.999006 secs 2024-07-26T08:42:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:22 smithi042 ceph-mon[32775]: pgmap v3965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6388.999006 secs 2024-07-26T08:42:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:22 smithi164 ceph-mon[36970]: pgmap v3965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6388.999006 secs 2024-07-26T08:42:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:22 smithi163 ceph-mon[36873]: pgmap v3965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:25.136 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:24 smithi042 ceph-mon[32775]: pgmap v3966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:24 smithi164 ceph-mon[36970]: pgmap v3966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:24 smithi163 ceph-mon[36873]: pgmap v3966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:25.499 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:42:25.498+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:42:25.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:42:25.498+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:42:25.500 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:42:25.498+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:42:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:26 smithi042 ceph-mon[32775]: pgmap v3967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6376.256697 secs 2024-07-26T08:42:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:26 smithi164 ceph-mon[36970]: pgmap v3967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6376.256697 secs 2024-07-26T08:42:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:26 smithi163 ceph-mon[36873]: pgmap v3967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6376.256697 secs 2024-07-26T08:42:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6393.999152 secs 2024-07-26T08:42:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6393.999152 secs 2024-07-26T08:42:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6393.999152 secs 2024-07-26T08:42:29.158 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:28 smithi163 ceph-mon[36873]: pgmap v3968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:28 smithi164 ceph-mon[36970]: pgmap v3968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:28 smithi042 ceph-mon[32775]: pgmap v3968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:30 smithi164 ceph-mon[36970]: pgmap v3969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:30 smithi163 ceph-mon[36873]: pgmap v3969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:30 smithi042 ceph-mon[32775]: pgmap v3969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6381.256847 secs 2024-07-26T08:42:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6381.256847 secs 2024-07-26T08:42:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6381.256847 secs 2024-07-26T08:42:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6398.999283 secs 2024-07-26T08:42:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:32 smithi164 ceph-mon[36970]: pgmap v3970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6398.999283 secs 2024-07-26T08:42:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:32 smithi163 ceph-mon[36873]: pgmap v3970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6398.999283 secs 2024-07-26T08:42:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:32 smithi042 ceph-mon[32775]: pgmap v3970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:34 smithi164 ceph-mon[36970]: pgmap v3971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:34 smithi163 ceph-mon[36873]: pgmap v3971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:34 smithi042 ceph-mon[32775]: pgmap v3971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:37 smithi042 ceph-mon[32775]: pgmap v3972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6386.256985 secs 2024-07-26T08:42:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6403.999445 secs 2024-07-26T08:42:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:37 smithi164 ceph-mon[36970]: pgmap v3972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6386.256985 secs 2024-07-26T08:42:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6403.999445 secs 2024-07-26T08:42:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:37 smithi163 ceph-mon[36873]: pgmap v3972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6386.256985 secs 2024-07-26T08:42:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6403.999445 secs 2024-07-26T08:42:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:39 smithi042 ceph-mon[32775]: pgmap v3973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:39 smithi164 ceph-mon[36970]: pgmap v3973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:39 smithi163 ceph-mon[36873]: pgmap v3973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:42:40.507+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:42:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:42:40.507+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:42:40.509 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:42:40.507+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:42:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:41 smithi042 ceph-mon[32775]: pgmap v3974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:41 smithi164 ceph-mon[36970]: pgmap v3974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:41 smithi163 ceph-mon[36873]: pgmap v3974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:42.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6391.257122 secs 2024-07-26T08:42:42.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6408.999590 secs 2024-07-26T08:42:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6391.257122 secs 2024-07-26T08:42:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6408.999590 secs 2024-07-26T08:42:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6391.257122 secs 2024-07-26T08:42:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6408.999590 secs 2024-07-26T08:42:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:43 smithi042 ceph-mon[32775]: pgmap v3975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:43 smithi164 ceph-mon[36970]: pgmap v3975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:43 smithi163 ceph-mon[36873]: pgmap v3975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:45 smithi042 ceph-mon[32775]: pgmap v3976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:45 smithi164 ceph-mon[36970]: pgmap v3976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:45 smithi163 ceph-mon[36873]: pgmap v3976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:47 smithi042 ceph-mon[32775]: pgmap v3977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6396.257258 secs 2024-07-26T08:42:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6413.999732 secs 2024-07-26T08:42:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:47 smithi164 ceph-mon[36970]: pgmap v3977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6396.257258 secs 2024-07-26T08:42:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6413.999732 secs 2024-07-26T08:42:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:47 smithi163 ceph-mon[36873]: pgmap v3977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6396.257258 secs 2024-07-26T08:42:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6413.999732 secs 2024-07-26T08:42:49.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:49 smithi042 ceph-mon[32775]: pgmap v3978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:49 smithi164 ceph-mon[36970]: pgmap v3978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:49 smithi163 ceph-mon[36873]: pgmap v3978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:51 smithi042 ceph-mon[32775]: pgmap v3979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:51 smithi164 ceph-mon[36970]: pgmap v3979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:51 smithi163 ceph-mon[36873]: pgmap v3979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6401.257377 secs 2024-07-26T08:42:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6418.999856 secs 2024-07-26T08:42:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6401.257377 secs 2024-07-26T08:42:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6418.999856 secs 2024-07-26T08:42:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6401.257377 secs 2024-07-26T08:42:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6418.999856 secs 2024-07-26T08:42:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:53 smithi042 ceph-mon[32775]: pgmap v3980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:53 smithi164 ceph-mon[36970]: pgmap v3980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:53 smithi163 ceph-mon[36873]: pgmap v3980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:55 smithi042 ceph-mon[32775]: pgmap v3981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:55 smithi164 ceph-mon[36970]: pgmap v3981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:55 smithi163 ceph-mon[36873]: pgmap v3981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:42:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:42:55.511+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:42:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:42:55.511+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:42:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:42:55.511+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:42:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:57 smithi042 ceph-mon[32775]: pgmap v3982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6406.257441 secs 2024-07-26T08:42:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6423.999966 secs 2024-07-26T08:42:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:57 smithi164 ceph-mon[36970]: pgmap v3982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6406.257441 secs 2024-07-26T08:42:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6423.999966 secs 2024-07-26T08:42:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:57 smithi163 ceph-mon[36873]: pgmap v3982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:42:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6406.257441 secs 2024-07-26T08:42:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6423.999966 secs 2024-07-26T08:42:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:42:59 smithi042 ceph-mon[32775]: pgmap v3983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:42:59 smithi164 ceph-mon[36970]: pgmap v3983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:42:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:42:59 smithi163 ceph-mon[36873]: pgmap v3983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:43:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:43:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:43:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:43:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:43:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:43:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:43:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:43:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:43:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:43:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:43:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:43:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:01 smithi042 ceph-mon[32775]: pgmap v3984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:01 smithi042 ceph-mon[32775]: pgmap v3985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:43:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:01 smithi042 ceph-mon[32775]: pgmap v3986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:43:01.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:01 smithi164 ceph-mon[36970]: pgmap v3984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:01.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:01 smithi164 ceph-mon[36970]: pgmap v3985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:43:01.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:01 smithi164 ceph-mon[36970]: pgmap v3986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:43:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:01 smithi163 ceph-mon[36873]: pgmap v3984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:01 smithi163 ceph-mon[36873]: pgmap v3985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:43:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:01 smithi163 ceph-mon[36873]: pgmap v3986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:43:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6411.257509 secs 2024-07-26T08:43:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6429.000087 secs 2024-07-26T08:43:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6411.257509 secs 2024-07-26T08:43:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6429.000087 secs 2024-07-26T08:43:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6411.257509 secs 2024-07-26T08:43:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6429.000087 secs 2024-07-26T08:43:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:03 smithi042 ceph-mon[32775]: pgmap v3987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:43:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:03 smithi164 ceph-mon[36970]: pgmap v3987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:43:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:03 smithi163 ceph-mon[36873]: pgmap v3987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:43:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:05 smithi042 ceph-mon[32775]: pgmap v3988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:43:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:05 smithi164 ceph-mon[36970]: pgmap v3988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:43:05.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:05 smithi163 ceph-mon[36873]: pgmap v3988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:43:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6416.257648 secs 2024-07-26T08:43:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:07 smithi042 ceph-mon[32775]: pgmap v3989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:43:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6434.000175 secs 2024-07-26T08:43:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6416.257648 secs 2024-07-26T08:43:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:07 smithi164 ceph-mon[36970]: pgmap v3989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:43:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6434.000175 secs 2024-07-26T08:43:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6416.257648 secs 2024-07-26T08:43:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:07 smithi163 ceph-mon[36873]: pgmap v3989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:43:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6434.000175 secs 2024-07-26T08:43:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:09 smithi042 ceph-mon[32775]: pgmap v3990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:43:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:09 smithi164 ceph-mon[36970]: pgmap v3990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:43:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:09 smithi163 ceph-mon[36873]: pgmap v3990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:43:10.519 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:10.518+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:43:10.520 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:10.518+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:43:10.520 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:10.518+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:43:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:11 smithi042 ceph-mon[32775]: pgmap v3991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:43:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:11 smithi164 ceph-mon[36970]: pgmap v3991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:43:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:11 smithi163 ceph-mon[36873]: pgmap v3991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:43:12.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6421.257783 secs 2024-07-26T08:43:12.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6439.000302 secs 2024-07-26T08:43:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6421.257783 secs 2024-07-26T08:43:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6439.000302 secs 2024-07-26T08:43:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6421.257783 secs 2024-07-26T08:43:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6439.000302 secs 2024-07-26T08:43:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:13 smithi042 ceph-mon[32775]: pgmap v3992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:13 smithi164 ceph-mon[36970]: pgmap v3992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:13 smithi163 ceph-mon[36873]: pgmap v3992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:13.859 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:13.857+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:43:13.859 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:13.858+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:43:13.864 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:13.862+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:43:13.864 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:13.863+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:43:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:15 smithi042 ceph-mon[32775]: pgmap v3993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:43:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:15 smithi164 ceph-mon[36970]: pgmap v3993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:43:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:15 smithi163 ceph-mon[36873]: pgmap v3993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:43:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6426.257916 secs 2024-07-26T08:43:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:17 smithi042 ceph-mon[32775]: pgmap v3994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6444.000429 secs 2024-07-26T08:43:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6426.257916 secs 2024-07-26T08:43:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:17 smithi164 ceph-mon[36970]: pgmap v3994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6444.000429 secs 2024-07-26T08:43:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6426.257916 secs 2024-07-26T08:43:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:17 smithi163 ceph-mon[36873]: pgmap v3994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6444.000429 secs 2024-07-26T08:43:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:19 smithi042 ceph-mon[32775]: pgmap v3995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:19 smithi164 ceph-mon[36970]: pgmap v3995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:19 smithi163 ceph-mon[36873]: pgmap v3995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:21 smithi042 ceph-mon[32775]: pgmap v3996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:21 smithi164 ceph-mon[36970]: pgmap v3996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:21 smithi163 ceph-mon[36873]: pgmap v3996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6431.258060 secs 2024-07-26T08:43:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6449.000499 secs 2024-07-26T08:43:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6431.258060 secs 2024-07-26T08:43:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6449.000499 secs 2024-07-26T08:43:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6431.258060 secs 2024-07-26T08:43:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6449.000499 secs 2024-07-26T08:43:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:23 smithi042 ceph-mon[32775]: pgmap v3997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:23 smithi164 ceph-mon[36970]: pgmap v3997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:23 smithi163 ceph-mon[36873]: pgmap v3997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:25 smithi164 ceph-mon[36970]: pgmap v3998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:43:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:25 smithi163 ceph-mon[36873]: pgmap v3998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:43:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:25.520+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:43:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:25.520+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:43:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:25.520+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:43:25.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:25 smithi042 ceph-mon[32775]: pgmap v3998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:43:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6436.258219 secs 2024-07-26T08:43:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:27 smithi042 ceph-mon[32775]: pgmap v3999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6454.000657 secs 2024-07-26T08:43:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6436.258219 secs 2024-07-26T08:43:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:27 smithi164 ceph-mon[36970]: pgmap v3999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6454.000657 secs 2024-07-26T08:43:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6436.258219 secs 2024-07-26T08:43:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:27 smithi163 ceph-mon[36873]: pgmap v3999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6454.000657 secs 2024-07-26T08:43:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:29 smithi164 ceph-mon[36970]: pgmap v4000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:29 smithi163 ceph-mon[36873]: pgmap v4000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:29 smithi042 ceph-mon[32775]: pgmap v4000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:31 smithi164 ceph-mon[36970]: pgmap v4001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:31 smithi163 ceph-mon[36873]: pgmap v4001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:31 smithi042 ceph-mon[32775]: pgmap v4001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6441.258319 secs 2024-07-26T08:43:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6459.000779 secs 2024-07-26T08:43:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6441.258319 secs 2024-07-26T08:43:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6459.000779 secs 2024-07-26T08:43:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6441.258319 secs 2024-07-26T08:43:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6459.000779 secs 2024-07-26T08:43:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:33 smithi164 ceph-mon[36970]: pgmap v4002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:33 smithi163 ceph-mon[36873]: pgmap v4002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:33.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:33 smithi042 ceph-mon[32775]: pgmap v4002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:35 smithi164 ceph-mon[36970]: pgmap v4003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:43:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:35 smithi163 ceph-mon[36873]: pgmap v4003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:43:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:35 smithi042 ceph-mon[32775]: pgmap v4003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:43:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6446.258442 secs 2024-07-26T08:43:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:37 smithi164 ceph-mon[36970]: pgmap v4004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6464.000922 secs 2024-07-26T08:43:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6446.258442 secs 2024-07-26T08:43:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:37 smithi163 ceph-mon[36873]: pgmap v4004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6464.000922 secs 2024-07-26T08:43:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6446.258442 secs 2024-07-26T08:43:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:37 smithi042 ceph-mon[32775]: pgmap v4004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6464.000922 secs 2024-07-26T08:43:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:39 smithi164 ceph-mon[36970]: pgmap v4005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:39 smithi163 ceph-mon[36873]: pgmap v4005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:39 smithi042 ceph-mon[32775]: pgmap v4005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:40.530 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:40.529+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:43:40.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:40.529+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:43:40.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:40.529+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:43:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:41 smithi164 ceph-mon[36970]: pgmap v4006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:41 smithi163 ceph-mon[36873]: pgmap v4006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:41 smithi042 ceph-mon[32775]: pgmap v4006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6451.258568 secs 2024-07-26T08:43:42.503 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6469.001045 secs 2024-07-26T08:43:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6451.258568 secs 2024-07-26T08:43:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6469.001045 secs 2024-07-26T08:43:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6451.258568 secs 2024-07-26T08:43:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6469.001045 secs 2024-07-26T08:43:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:43 smithi042 ceph-mon[32775]: pgmap v4007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:43 smithi164 ceph-mon[36970]: pgmap v4007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:43 smithi163 ceph-mon[36873]: pgmap v4007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:45 smithi164 ceph-mon[36970]: pgmap v4008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:43:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:45 smithi163 ceph-mon[36873]: pgmap v4008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:43:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:45 smithi042 ceph-mon[32775]: pgmap v4008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:43:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6456.258692 secs 2024-07-26T08:43:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:47 smithi164 ceph-mon[36970]: pgmap v4009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6474.001185 secs 2024-07-26T08:43:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6456.258692 secs 2024-07-26T08:43:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:47 smithi163 ceph-mon[36873]: pgmap v4009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6474.001185 secs 2024-07-26T08:43:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6456.258692 secs 2024-07-26T08:43:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:47 smithi042 ceph-mon[32775]: pgmap v4009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6474.001185 secs 2024-07-26T08:43:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:49 smithi164 ceph-mon[36970]: pgmap v4010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:49 smithi163 ceph-mon[36873]: pgmap v4010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:49 smithi042 ceph-mon[32775]: pgmap v4010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:51 smithi164 ceph-mon[36970]: pgmap v4011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:51 smithi163 ceph-mon[36873]: pgmap v4011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:51 smithi042 ceph-mon[32775]: pgmap v4011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6461.258841 secs 2024-07-26T08:43:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6479.001327 secs 2024-07-26T08:43:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6461.258841 secs 2024-07-26T08:43:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6479.001327 secs 2024-07-26T08:43:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6461.258841 secs 2024-07-26T08:43:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6479.001327 secs 2024-07-26T08:43:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:53 smithi164 ceph-mon[36970]: pgmap v4012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:53 smithi163 ceph-mon[36873]: pgmap v4012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:53.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:53 smithi042 ceph-mon[32775]: pgmap v4012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:55 smithi164 ceph-mon[36970]: pgmap v4013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:55 smithi163 ceph-mon[36873]: pgmap v4013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:55.534 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:55.533+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:43:55.534 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:55.533+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:43:55.534 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:55.533+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:43:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:55 smithi042 ceph-mon[32775]: pgmap v4013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6466.258966 secs 2024-07-26T08:43:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:57 smithi164 ceph-mon[36970]: pgmap v4014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6484.001442 secs 2024-07-26T08:43:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6466.258966 secs 2024-07-26T08:43:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:57 smithi163 ceph-mon[36873]: pgmap v4014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6484.001442 secs 2024-07-26T08:43:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6466.258966 secs 2024-07-26T08:43:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:57 smithi042 ceph-mon[32775]: pgmap v4014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:43:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6484.001442 secs 2024-07-26T08:43:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:58.864+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:43:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:58.864+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:43:58.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:58.869+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:43:58.870 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:43:58.869+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:43:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:43:59 smithi164 ceph-mon[36970]: pgmap v4015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:43:59 smithi163 ceph-mon[36873]: pgmap v4015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:43:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:43:59 smithi042 ceph-mon[32775]: pgmap v4015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:44:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:44:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:44:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:01 smithi163 ceph-mon[36873]: pgmap v4016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:44:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:44:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:44:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:44:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:01 smithi042 ceph-mon[32775]: pgmap v4016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:44:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:44:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:44:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:44:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:01 smithi164 ceph-mon[36970]: pgmap v4016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:44:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:44:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:44:01.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:44:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:02 smithi163 ceph-mon[36873]: pgmap v4017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:02 smithi163 ceph-mon[36873]: pgmap v4018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T08:44:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6471.259065 secs 2024-07-26T08:44:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6489.001588 secs 2024-07-26T08:44:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:02 smithi042 ceph-mon[32775]: pgmap v4017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:02 smithi042 ceph-mon[32775]: pgmap v4018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T08:44:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6471.259065 secs 2024-07-26T08:44:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6489.001588 secs 2024-07-26T08:44:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:02 smithi164 ceph-mon[36970]: pgmap v4017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:02 smithi164 ceph-mon[36970]: pgmap v4018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T08:44:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6471.259065 secs 2024-07-26T08:44:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6489.001588 secs 2024-07-26T08:44:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:04 smithi163 ceph-mon[36873]: pgmap v4019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T08:44:04.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:04 smithi042 ceph-mon[32775]: pgmap v4019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T08:44:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:04 smithi164 ceph-mon[36970]: pgmap v4019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T08:44:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:06 smithi163 ceph-mon[36873]: pgmap v4020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T08:44:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:06 smithi042 ceph-mon[32775]: pgmap v4020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T08:44:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:06 smithi164 ceph-mon[36970]: pgmap v4020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T08:44:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6476.259184 secs 2024-07-26T08:44:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6494.001712 secs 2024-07-26T08:44:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6476.259184 secs 2024-07-26T08:44:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6494.001712 secs 2024-07-26T08:44:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6476.259184 secs 2024-07-26T08:44:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6494.001712 secs 2024-07-26T08:44:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:08 smithi163 ceph-mon[36873]: pgmap v4021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T08:44:08.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:08 smithi042 ceph-mon[32775]: pgmap v4021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T08:44:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:08 smithi164 ceph-mon[36970]: pgmap v4021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T08:44:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:10 smithi163 ceph-mon[36873]: pgmap v4022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:44:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:44:10.542+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:44:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:44:10.542+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:44:10.543 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:44:10.542+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:44:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:10 smithi042 ceph-mon[32775]: pgmap v4022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:44:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:10 smithi164 ceph-mon[36970]: pgmap v4022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:44:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:12 smithi163 ceph-mon[36873]: pgmap v4023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:44:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6481.259327 secs 2024-07-26T08:44:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6499.001840 secs 2024-07-26T08:44:12.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:12 smithi164 ceph-mon[36970]: pgmap v4023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:44:12.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6481.259327 secs 2024-07-26T08:44:12.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6499.001840 secs 2024-07-26T08:44:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:12 smithi042 ceph-mon[32775]: pgmap v4023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:44:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6481.259327 secs 2024-07-26T08:44:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6499.001840 secs 2024-07-26T08:44:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:14 smithi163 ceph-mon[36873]: pgmap v4024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:14 smithi042 ceph-mon[32775]: pgmap v4024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:14 smithi164 ceph-mon[36970]: pgmap v4024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:16 smithi163 ceph-mon[36873]: pgmap v4025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:16 smithi042 ceph-mon[32775]: pgmap v4025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:16 smithi164 ceph-mon[36970]: pgmap v4025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6486.259463 secs 2024-07-26T08:44:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6504.001994 secs 2024-07-26T08:44:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6486.259463 secs 2024-07-26T08:44:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6504.001994 secs 2024-07-26T08:44:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6486.259463 secs 2024-07-26T08:44:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6504.001994 secs 2024-07-26T08:44:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:18 smithi042 ceph-mon[32775]: pgmap v4026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:18 smithi164 ceph-mon[36970]: pgmap v4026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:18 smithi163 ceph-mon[36873]: pgmap v4026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:20 smithi042 ceph-mon[32775]: pgmap v4027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:44:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:20 smithi164 ceph-mon[36970]: pgmap v4027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:44:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:20 smithi163 ceph-mon[36873]: pgmap v4027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:44:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:22 smithi042 ceph-mon[32775]: pgmap v4028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6491.259563 secs 2024-07-26T08:44:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6509.002128 secs 2024-07-26T08:44:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:22 smithi164 ceph-mon[36970]: pgmap v4028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6491.259563 secs 2024-07-26T08:44:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6509.002128 secs 2024-07-26T08:44:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:22 smithi163 ceph-mon[36873]: pgmap v4028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6491.259563 secs 2024-07-26T08:44:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6509.002128 secs 2024-07-26T08:44:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:24 smithi042 ceph-mon[32775]: pgmap v4029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:24 smithi164 ceph-mon[36970]: pgmap v4029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:24 smithi163 ceph-mon[36873]: pgmap v4029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:25.547 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:44:25.546+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:44:25.547 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:44:25.546+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:44:25.547 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:44:25.546+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:44:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:26 smithi042 ceph-mon[32775]: pgmap v4030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:26 smithi164 ceph-mon[36970]: pgmap v4030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:26 smithi163 ceph-mon[36873]: pgmap v4030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6496.259670 secs 2024-07-26T08:44:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6514.002249 secs 2024-07-26T08:44:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6496.259670 secs 2024-07-26T08:44:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6514.002249 secs 2024-07-26T08:44:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6496.259670 secs 2024-07-26T08:44:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6514.002249 secs 2024-07-26T08:44:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:28 smithi042 ceph-mon[32775]: pgmap v4031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:28 smithi164 ceph-mon[36970]: pgmap v4031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:28 smithi163 ceph-mon[36873]: pgmap v4031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:30.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:30 smithi163 ceph-mon[36873]: pgmap v4032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:44:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:30 smithi042 ceph-mon[32775]: pgmap v4032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:44:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:30 smithi164 ceph-mon[36970]: pgmap v4032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:44:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:31 smithi042 ceph-mon[32775]: pgmap v4033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:31 smithi164 ceph-mon[36970]: pgmap v4033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:31 smithi163 ceph-mon[36873]: pgmap v4033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6501.259799 secs 2024-07-26T08:44:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6519.002376 secs 2024-07-26T08:44:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6501.259799 secs 2024-07-26T08:44:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6519.002376 secs 2024-07-26T08:44:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6501.259799 secs 2024-07-26T08:44:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6519.002376 secs 2024-07-26T08:44:33.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:33 smithi042 ceph-mon[32775]: pgmap v4034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:33 smithi164 ceph-mon[36970]: pgmap v4034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:33 smithi163 ceph-mon[36873]: pgmap v4034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:35 smithi164 ceph-mon[36970]: pgmap v4035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:36.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:35 smithi163 ceph-mon[36873]: pgmap v4035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:35 smithi042 ceph-mon[32775]: pgmap v4035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6506.259941 secs 2024-07-26T08:44:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6524.002497 secs 2024-07-26T08:44:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:37 smithi164 ceph-mon[36970]: pgmap v4036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6506.259941 secs 2024-07-26T08:44:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6524.002497 secs 2024-07-26T08:44:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:37 smithi163 ceph-mon[36873]: pgmap v4036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6506.259941 secs 2024-07-26T08:44:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6524.002497 secs 2024-07-26T08:44:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:37 smithi042 ceph-mon[32775]: pgmap v4036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:39 smithi164 ceph-mon[36970]: pgmap v4037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:44:40.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:39 smithi163 ceph-mon[36873]: pgmap v4037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:44:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:39 smithi042 ceph-mon[32775]: pgmap v4037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:44:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:44:40.549+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:44:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:44:40.549+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:44:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:44:40.549+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:44:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:41 smithi164 ceph-mon[36970]: pgmap v4038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:41 smithi163 ceph-mon[36873]: pgmap v4038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:41 smithi042 ceph-mon[32775]: pgmap v4038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6511.260036 secs 2024-07-26T08:44:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6529.002572 secs 2024-07-26T08:44:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6511.260036 secs 2024-07-26T08:44:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6529.002572 secs 2024-07-26T08:44:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6511.260036 secs 2024-07-26T08:44:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6529.002572 secs 2024-07-26T08:44:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:43 smithi164 ceph-mon[36970]: pgmap v4039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:43 smithi163 ceph-mon[36873]: pgmap v4039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:43 smithi042 ceph-mon[32775]: pgmap v4039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:45 smithi164 ceph-mon[36970]: pgmap v4040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:45 smithi163 ceph-mon[36873]: pgmap v4040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:46.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:45 smithi042 ceph-mon[32775]: pgmap v4040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6516.260162 secs 2024-07-26T08:44:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6516.260162 secs 2024-07-26T08:44:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6516.260162 secs 2024-07-26T08:44:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6534.002707 secs 2024-07-26T08:44:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:47 smithi164 ceph-mon[36970]: pgmap v4041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6534.002707 secs 2024-07-26T08:44:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:47 smithi163 ceph-mon[36873]: pgmap v4041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6534.002707 secs 2024-07-26T08:44:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:47 smithi042 ceph-mon[32775]: pgmap v4041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:49 smithi164 ceph-mon[36970]: pgmap v4042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:44:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:49 smithi163 ceph-mon[36873]: pgmap v4042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:44:50.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:49 smithi042 ceph-mon[32775]: pgmap v4042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:44:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:51 smithi164 ceph-mon[36970]: pgmap v4043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6521.260265 secs 2024-07-26T08:44:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:51 smithi163 ceph-mon[36873]: pgmap v4043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6521.260265 secs 2024-07-26T08:44:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:51 smithi042 ceph-mon[32775]: pgmap v4043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6521.260265 secs 2024-07-26T08:44:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6539.002849 secs 2024-07-26T08:44:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6539.002849 secs 2024-07-26T08:44:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6539.002849 secs 2024-07-26T08:44:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:53 smithi164 ceph-mon[36970]: pgmap v4044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:53 smithi163 ceph-mon[36873]: pgmap v4044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:54.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:53 smithi042 ceph-mon[32775]: pgmap v4044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:44:55.553+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:44:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:44:55.553+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:44:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:44:55.553+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:44:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:55 smithi164 ceph-mon[36970]: pgmap v4045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:55 smithi163 ceph-mon[36873]: pgmap v4045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:55 smithi042 ceph-mon[32775]: pgmap v4045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:44:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6526.260390 secs 2024-07-26T08:44:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6526.260390 secs 2024-07-26T08:44:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6526.260390 secs 2024-07-26T08:44:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6544.002973 secs 2024-07-26T08:44:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:57 smithi164 ceph-mon[36970]: pgmap v4046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6544.002973 secs 2024-07-26T08:44:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:57 smithi163 ceph-mon[36873]: pgmap v4046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6544.002973 secs 2024-07-26T08:44:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:57 smithi042 ceph-mon[32775]: pgmap v4046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:44:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:44:59 smithi164 ceph-mon[36970]: pgmap v4047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:45:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:44:59 smithi163 ceph-mon[36873]: pgmap v4047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:45:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:44:59 smithi042 ceph-mon[32775]: pgmap v4047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:45:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:45:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:45:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:45:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:01 smithi164 ceph-mon[36970]: pgmap v4048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6531.260460 secs 2024-07-26T08:45:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:45:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:45:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:01 smithi164 ceph-mon[36970]: pgmap v4049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:45:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:01 smithi164 ceph-mon[36970]: pgmap v4050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:45:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:45:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:01 smithi163 ceph-mon[36873]: pgmap v4048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6531.260460 secs 2024-07-26T08:45:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:45:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:45:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:01 smithi163 ceph-mon[36873]: pgmap v4049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:45:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:01 smithi163 ceph-mon[36873]: pgmap v4050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:45:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:45:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:01 smithi042 ceph-mon[32775]: pgmap v4048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6531.260460 secs 2024-07-26T08:45:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:45:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:45:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:01 smithi042 ceph-mon[32775]: pgmap v4049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:45:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:01 smithi042 ceph-mon[32775]: pgmap v4050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:45:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:45:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6549.003040 secs 2024-07-26T08:45:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6549.003040 secs 2024-07-26T08:45:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6549.003040 secs 2024-07-26T08:45:04.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:03 smithi163 ceph-mon[36873]: pgmap v4051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:45:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:03 smithi042 ceph-mon[32775]: pgmap v4051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:45:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:03 smithi164 ceph-mon[36970]: pgmap v4051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:45:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:06 smithi164 ceph-mon[36970]: pgmap v4052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:45:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:06 smithi163 ceph-mon[36873]: pgmap v4052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:45:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:06 smithi042 ceph-mon[32775]: pgmap v4052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:45:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6536.260534 secs 2024-07-26T08:45:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6554.003164 secs 2024-07-26T08:45:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6536.260534 secs 2024-07-26T08:45:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6554.003164 secs 2024-07-26T08:45:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6536.260534 secs 2024-07-26T08:45:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6554.003164 secs 2024-07-26T08:45:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:08 smithi164 ceph-mon[36970]: pgmap v4053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:45:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:08 smithi163 ceph-mon[36873]: pgmap v4053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:45:08.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:08 smithi042 ceph-mon[32775]: pgmap v4053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:45:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:10 smithi164 ceph-mon[36970]: pgmap v4054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:45:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:10 smithi163 ceph-mon[36873]: pgmap v4054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:45:10.564 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:45:10.562+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:45:10.564 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:45:10.562+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:45:10.564 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:45:10.562+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:45:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:10 smithi042 ceph-mon[32775]: pgmap v4054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:45:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6541.260610 secs 2024-07-26T08:45:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6559.003252 secs 2024-07-26T08:45:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:12 smithi164 ceph-mon[36970]: pgmap v4055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:45:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6541.260610 secs 2024-07-26T08:45:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6559.003252 secs 2024-07-26T08:45:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:12 smithi163 ceph-mon[36873]: pgmap v4055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:45:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6541.260610 secs 2024-07-26T08:45:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6559.003252 secs 2024-07-26T08:45:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:12 smithi042 ceph-mon[32775]: pgmap v4055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:45:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:14 smithi164 ceph-mon[36970]: pgmap v4056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:14 smithi163 ceph-mon[36873]: pgmap v4056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:14 smithi042 ceph-mon[32775]: pgmap v4056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:16 smithi042 ceph-mon[32775]: pgmap v4057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:16 smithi163 ceph-mon[36873]: pgmap v4057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:16 smithi164 ceph-mon[36970]: pgmap v4057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6546.260670 secs 2024-07-26T08:45:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6564.003391 secs 2024-07-26T08:45:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6546.260670 secs 2024-07-26T08:45:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6564.003391 secs 2024-07-26T08:45:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6546.260670 secs 2024-07-26T08:45:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6564.003391 secs 2024-07-26T08:45:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:18 smithi163 ceph-mon[36873]: pgmap v4058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:18 smithi042 ceph-mon[32775]: pgmap v4058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:18 smithi164 ceph-mon[36970]: pgmap v4058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:20 smithi163 ceph-mon[36873]: pgmap v4059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:20 smithi042 ceph-mon[32775]: pgmap v4059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:20 smithi164 ceph-mon[36970]: pgmap v4059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6551.260736 secs 2024-07-26T08:45:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6569.003461 secs 2024-07-26T08:45:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:22 smithi163 ceph-mon[36873]: pgmap v4060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6551.260736 secs 2024-07-26T08:45:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6569.003461 secs 2024-07-26T08:45:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:22 smithi042 ceph-mon[32775]: pgmap v4060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6551.260736 secs 2024-07-26T08:45:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6569.003461 secs 2024-07-26T08:45:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:22 smithi164 ceph-mon[36970]: pgmap v4060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:24 smithi163 ceph-mon[36873]: pgmap v4061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:24 smithi042 ceph-mon[32775]: pgmap v4061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:24 smithi164 ceph-mon[36970]: pgmap v4061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:45:25.566+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:45:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:45:25.566+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:45:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:45:25.566+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:45:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:26 smithi163 ceph-mon[36873]: pgmap v4062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:26 smithi042 ceph-mon[32775]: pgmap v4062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:26 smithi164 ceph-mon[36970]: pgmap v4062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6556.260817 secs 2024-07-26T08:45:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6574.003584 secs 2024-07-26T08:45:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6556.260817 secs 2024-07-26T08:45:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6574.003584 secs 2024-07-26T08:45:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6556.260817 secs 2024-07-26T08:45:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6574.003584 secs 2024-07-26T08:45:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:28 smithi163 ceph-mon[36873]: pgmap v4063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:28 smithi042 ceph-mon[32775]: pgmap v4063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:28 smithi164 ceph-mon[36970]: pgmap v4063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:30.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:30 smithi163 ceph-mon[36873]: pgmap v4064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:30 smithi042 ceph-mon[32775]: pgmap v4064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:30 smithi164 ceph-mon[36970]: pgmap v4064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6561.260941 secs 2024-07-26T08:45:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6579.003670 secs 2024-07-26T08:45:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:32 smithi042 ceph-mon[32775]: pgmap v4065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6561.260941 secs 2024-07-26T08:45:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6579.003670 secs 2024-07-26T08:45:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:32 smithi164 ceph-mon[36970]: pgmap v4065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6561.260941 secs 2024-07-26T08:45:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6579.003670 secs 2024-07-26T08:45:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:32 smithi163 ceph-mon[36873]: pgmap v4065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:34 smithi042 ceph-mon[32775]: pgmap v4066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:34 smithi164 ceph-mon[36970]: pgmap v4066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:34 smithi163 ceph-mon[36873]: pgmap v4066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:36 smithi042 ceph-mon[32775]: pgmap v4067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:36.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:36 smithi164 ceph-mon[36970]: pgmap v4067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:36 smithi163 ceph-mon[36873]: pgmap v4067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6566.261069 secs 2024-07-26T08:45:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6584.003823 secs 2024-07-26T08:45:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6566.261069 secs 2024-07-26T08:45:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6584.003823 secs 2024-07-26T08:45:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6566.261069 secs 2024-07-26T08:45:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6584.003823 secs 2024-07-26T08:45:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:38 smithi042 ceph-mon[32775]: pgmap v4068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:38 smithi164 ceph-mon[36970]: pgmap v4068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:38 smithi163 ceph-mon[36873]: pgmap v4068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:40.576 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:45:40.575+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:45:40.576 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:45:40.575+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:45:40.577 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:45:40.575+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:45:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:40 smithi042 ceph-mon[32775]: pgmap v4069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:40 smithi164 ceph-mon[36970]: pgmap v4069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:40 smithi163 ceph-mon[36873]: pgmap v4069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6571.261192 secs 2024-07-26T08:45:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6589.003959 secs 2024-07-26T08:45:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:42 smithi042 ceph-mon[32775]: pgmap v4070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6571.261192 secs 2024-07-26T08:45:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6589.003959 secs 2024-07-26T08:45:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:42 smithi164 ceph-mon[36970]: pgmap v4070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6571.261192 secs 2024-07-26T08:45:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6589.003959 secs 2024-07-26T08:45:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:42 smithi163 ceph-mon[36873]: pgmap v4070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:44 smithi042 ceph-mon[32775]: pgmap v4071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:44 smithi164 ceph-mon[36970]: pgmap v4071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:44 smithi163 ceph-mon[36873]: pgmap v4071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:46 smithi042 ceph-mon[32775]: pgmap v4072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:46 smithi164 ceph-mon[36970]: pgmap v4072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:46 smithi163 ceph-mon[36873]: pgmap v4072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6576.261314 secs 2024-07-26T08:45:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6594.004074 secs 2024-07-26T08:45:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6576.261314 secs 2024-07-26T08:45:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6594.004074 secs 2024-07-26T08:45:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6576.261314 secs 2024-07-26T08:45:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6594.004074 secs 2024-07-26T08:45:48.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:48 smithi042 ceph-mon[32775]: pgmap v4073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:48 smithi164 ceph-mon[36970]: pgmap v4073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:48 smithi163 ceph-mon[36873]: pgmap v4073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:49 smithi042 ceph-mon[32775]: pgmap v4074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:49 smithi164 ceph-mon[36970]: pgmap v4074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:49 smithi163 ceph-mon[36873]: pgmap v4074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6581.261434 secs 2024-07-26T08:45:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6599.004195 secs 2024-07-26T08:45:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:52 smithi164 ceph-mon[36970]: pgmap v4075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6581.261434 secs 2024-07-26T08:45:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6599.004195 secs 2024-07-26T08:45:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:52 smithi163 ceph-mon[36873]: pgmap v4075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6581.261434 secs 2024-07-26T08:45:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6599.004195 secs 2024-07-26T08:45:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:52 smithi042 ceph-mon[32775]: pgmap v4075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:54 smithi164 ceph-mon[36970]: pgmap v4076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:54 smithi163 ceph-mon[36873]: pgmap v4076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:54 smithi042 ceph-mon[32775]: pgmap v4076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:45:55.579+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:45:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:45:55.579+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:45:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:45:55.579+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:45:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:56 smithi163 ceph-mon[36873]: pgmap v4077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:56 smithi042 ceph-mon[32775]: pgmap v4077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:56 smithi164 ceph-mon[36970]: pgmap v4077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:45:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6586.261559 secs 2024-07-26T08:45:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6604.004317 secs 2024-07-26T08:45:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6586.261559 secs 2024-07-26T08:45:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6604.004317 secs 2024-07-26T08:45:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6586.261559 secs 2024-07-26T08:45:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6604.004317 secs 2024-07-26T08:45:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:45:58 smithi163 ceph-mon[36873]: pgmap v4078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:45:58 smithi042 ceph-mon[32775]: pgmap v4078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:45:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:45:58 smithi164 ceph-mon[36970]: pgmap v4078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:00 smithi163 ceph-mon[36873]: pgmap v4079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:00 smithi042 ceph-mon[32775]: pgmap v4079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:00 smithi164 ceph-mon[36970]: pgmap v4079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6591.261684 secs 2024-07-26T08:46:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6609.004439 secs 2024-07-26T08:46:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:02 smithi163 ceph-mon[36873]: pgmap v4080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:46:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:46:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:46:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:46:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6591.261684 secs 2024-07-26T08:46:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6609.004439 secs 2024-07-26T08:46:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:02 smithi042 ceph-mon[32775]: pgmap v4080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:46:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:46:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:46:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:46:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6591.261684 secs 2024-07-26T08:46:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6609.004439 secs 2024-07-26T08:46:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:02 smithi164 ceph-mon[36970]: pgmap v4080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:46:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:46:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:46:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:46:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:03 smithi163 ceph-mon[36873]: pgmap v4081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:46:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:03 smithi163 ceph-mon[36873]: pgmap v4082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:46:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:03 smithi042 ceph-mon[32775]: pgmap v4081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:46:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:03 smithi042 ceph-mon[32775]: pgmap v4082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:46:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:03 smithi164 ceph-mon[36970]: pgmap v4081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:46:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:03 smithi164 ceph-mon[36970]: pgmap v4082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:46:05.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:05 smithi163 ceph-mon[36873]: pgmap v4083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:46:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:05 smithi042 ceph-mon[32775]: pgmap v4083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:46:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:05 smithi164 ceph-mon[36970]: pgmap v4083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:46:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:07 smithi163 ceph-mon[36873]: pgmap v4084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:46:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6596.261805 secs 2024-07-26T08:46:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6614.004556 secs 2024-07-26T08:46:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:07 smithi042 ceph-mon[32775]: pgmap v4084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:46:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6596.261805 secs 2024-07-26T08:46:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6614.004556 secs 2024-07-26T08:46:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:07 smithi164 ceph-mon[36970]: pgmap v4084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:46:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6596.261805 secs 2024-07-26T08:46:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6614.004556 secs 2024-07-26T08:46:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:09 smithi163 ceph-mon[36873]: pgmap v4085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:46:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:09 smithi042 ceph-mon[32775]: pgmap v4085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:46:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:09 smithi164 ceph-mon[36970]: pgmap v4085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:46:10.584 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:46:10.582+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:46:10.584 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:46:10.582+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:46:10.584 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:46:10.582+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:46:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:11 smithi042 ceph-mon[32775]: pgmap v4086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:46:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:11 smithi164 ceph-mon[36970]: pgmap v4086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:46:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:11 smithi163 ceph-mon[36873]: pgmap v4086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:46:12.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6601.261928 secs 2024-07-26T08:46:12.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6619.004676 secs 2024-07-26T08:46:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6601.261928 secs 2024-07-26T08:46:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6619.004676 secs 2024-07-26T08:46:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6601.261928 secs 2024-07-26T08:46:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6619.004676 secs 2024-07-26T08:46:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:13 smithi042 ceph-mon[32775]: pgmap v4087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:46:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:13 smithi164 ceph-mon[36970]: pgmap v4087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:46:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:13 smithi163 ceph-mon[36873]: pgmap v4087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:46:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:15 smithi042 ceph-mon[32775]: pgmap v4088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:46:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:15 smithi164 ceph-mon[36970]: pgmap v4088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:46:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:15 smithi163 ceph-mon[36873]: pgmap v4088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:46:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:17 smithi042 ceph-mon[32775]: pgmap v4089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6606.262055 secs 2024-07-26T08:46:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6624.004774 secs 2024-07-26T08:46:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:17 smithi164 ceph-mon[36970]: pgmap v4089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6606.262055 secs 2024-07-26T08:46:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6624.004774 secs 2024-07-26T08:46:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:17 smithi163 ceph-mon[36873]: pgmap v4089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6606.262055 secs 2024-07-26T08:46:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6624.004774 secs 2024-07-26T08:46:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:19 smithi042 ceph-mon[32775]: pgmap v4090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:19 smithi164 ceph-mon[36970]: pgmap v4090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:19 smithi163 ceph-mon[36873]: pgmap v4090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:21 smithi042 ceph-mon[32775]: pgmap v4091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:21 smithi164 ceph-mon[36970]: pgmap v4091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:21 smithi163 ceph-mon[36873]: pgmap v4091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6611.262179 secs 2024-07-26T08:46:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6629.004898 secs 2024-07-26T08:46:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6611.262179 secs 2024-07-26T08:46:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6629.004898 secs 2024-07-26T08:46:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6611.262179 secs 2024-07-26T08:46:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6629.004898 secs 2024-07-26T08:46:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:23 smithi042 ceph-mon[32775]: pgmap v4092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:23 smithi164 ceph-mon[36970]: pgmap v4092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:23 smithi163 ceph-mon[36873]: pgmap v4092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:46:25.585+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:46:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:46:25.585+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:46:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:46:25.585+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:46:25.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:25 smithi042 ceph-mon[32775]: pgmap v4093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:46:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:25 smithi164 ceph-mon[36970]: pgmap v4093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:46:25.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:25 smithi163 ceph-mon[36873]: pgmap v4093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:46:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:27 smithi042 ceph-mon[32775]: pgmap v4094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6616.262308 secs 2024-07-26T08:46:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6634.005024 secs 2024-07-26T08:46:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:27 smithi164 ceph-mon[36970]: pgmap v4094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6616.262308 secs 2024-07-26T08:46:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6634.005024 secs 2024-07-26T08:46:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:27 smithi163 ceph-mon[36873]: pgmap v4094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6616.262308 secs 2024-07-26T08:46:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6634.005024 secs 2024-07-26T08:46:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:29 smithi042 ceph-mon[32775]: pgmap v4095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:29 smithi164 ceph-mon[36970]: pgmap v4095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:29 smithi163 ceph-mon[36873]: pgmap v4095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:31.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:31 smithi163 ceph-mon[36873]: pgmap v4096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:31 smithi042 ceph-mon[32775]: pgmap v4096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:31 smithi164 ceph-mon[36970]: pgmap v4096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6621.262376 secs 2024-07-26T08:46:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6639.005117 secs 2024-07-26T08:46:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6621.262376 secs 2024-07-26T08:46:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6639.005117 secs 2024-07-26T08:46:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6621.262376 secs 2024-07-26T08:46:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6639.005117 secs 2024-07-26T08:46:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:33 smithi042 ceph-mon[32775]: pgmap v4097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:33 smithi164 ceph-mon[36970]: pgmap v4097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:33 smithi163 ceph-mon[36873]: pgmap v4097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:35.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:35 smithi042 ceph-mon[32775]: pgmap v4098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:46:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:35 smithi164 ceph-mon[36970]: pgmap v4098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:46:35.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:35 smithi163 ceph-mon[36873]: pgmap v4098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:46:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:37 smithi042 ceph-mon[32775]: pgmap v4099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6626.262498 secs 2024-07-26T08:46:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6644.005208 secs 2024-07-26T08:46:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:37 smithi164 ceph-mon[36970]: pgmap v4099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6626.262498 secs 2024-07-26T08:46:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6644.005208 secs 2024-07-26T08:46:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:37 smithi163 ceph-mon[36873]: pgmap v4099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6626.262498 secs 2024-07-26T08:46:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6644.005208 secs 2024-07-26T08:46:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:39 smithi042 ceph-mon[32775]: pgmap v4100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:39 smithi164 ceph-mon[36970]: pgmap v4100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:39 smithi163 ceph-mon[36873]: pgmap v4100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:46:40.593+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:46:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:46:40.593+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:46:40.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:46:40.593+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:46:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:41 smithi042 ceph-mon[32775]: pgmap v4101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:41 smithi164 ceph-mon[36970]: pgmap v4101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:41 smithi163 ceph-mon[36873]: pgmap v4101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6631.262635 secs 2024-07-26T08:46:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6649.005327 secs 2024-07-26T08:46:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6631.262635 secs 2024-07-26T08:46:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6649.005327 secs 2024-07-26T08:46:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6631.262635 secs 2024-07-26T08:46:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6649.005327 secs 2024-07-26T08:46:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:43 smithi042 ceph-mon[32775]: pgmap v4102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:43 smithi164 ceph-mon[36970]: pgmap v4102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:43 smithi163 ceph-mon[36873]: pgmap v4102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:45 smithi042 ceph-mon[32775]: pgmap v4103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:46:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:45 smithi164 ceph-mon[36970]: pgmap v4103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:46:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:45 smithi163 ceph-mon[36873]: pgmap v4103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:46:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:47 smithi042 ceph-mon[32775]: pgmap v4104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6636.262757 secs 2024-07-26T08:46:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6654.005448 secs 2024-07-26T08:46:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:47 smithi164 ceph-mon[36970]: pgmap v4104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6636.262757 secs 2024-07-26T08:46:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6654.005448 secs 2024-07-26T08:46:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:47 smithi163 ceph-mon[36873]: pgmap v4104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6636.262757 secs 2024-07-26T08:46:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6654.005448 secs 2024-07-26T08:46:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:49 smithi042 ceph-mon[32775]: pgmap v4105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:49 smithi164 ceph-mon[36970]: pgmap v4105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:49 smithi163 ceph-mon[36873]: pgmap v4105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:51 smithi042 ceph-mon[32775]: pgmap v4106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:51 smithi164 ceph-mon[36970]: pgmap v4106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:51 smithi163 ceph-mon[36873]: pgmap v4106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6641.262855 secs 2024-07-26T08:46:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6659.005514 secs 2024-07-26T08:46:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6641.262855 secs 2024-07-26T08:46:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6659.005514 secs 2024-07-26T08:46:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6641.262855 secs 2024-07-26T08:46:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6659.005514 secs 2024-07-26T08:46:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:53 smithi164 ceph-mon[36970]: pgmap v4107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:53 smithi163 ceph-mon[36873]: pgmap v4107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:53 smithi042 ceph-mon[32775]: pgmap v4107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:55.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:55 smithi042 ceph-mon[32775]: pgmap v4108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:46:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:55 smithi164 ceph-mon[36970]: pgmap v4108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:46:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:55 smithi163 ceph-mon[36873]: pgmap v4108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:46:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:46:55.596+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:46:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:46:55.596+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:46:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:46:55.596+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:46:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:57 smithi164 ceph-mon[36970]: pgmap v4109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6646.262984 secs 2024-07-26T08:46:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6664.005652 secs 2024-07-26T08:46:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:57 smithi163 ceph-mon[36873]: pgmap v4109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6646.262984 secs 2024-07-26T08:46:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6664.005652 secs 2024-07-26T08:46:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:57 smithi042 ceph-mon[32775]: pgmap v4109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:46:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6646.262984 secs 2024-07-26T08:46:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6664.005652 secs 2024-07-26T08:46:58.890 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:46:58.888+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:46:58.890 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:46:58.888+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:46:58.896 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:46:58.894+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:46:58.896 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:46:58.895+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:46:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:46:59 smithi164 ceph-mon[36970]: pgmap v4110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:46:59 smithi163 ceph-mon[36873]: pgmap v4110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:46:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:46:59 smithi042 ceph-mon[32775]: pgmap v4110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:01 smithi164 ceph-mon[36970]: pgmap v4111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:01 smithi163 ceph-mon[36873]: pgmap v4111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:01 smithi042 ceph-mon[32775]: pgmap v4111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6651.263107 secs 2024-07-26T08:47:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6669.005766 secs 2024-07-26T08:47:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:47:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6651.263107 secs 2024-07-26T08:47:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6669.005766 secs 2024-07-26T08:47:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:47:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6651.263107 secs 2024-07-26T08:47:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6669.005766 secs 2024-07-26T08:47:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:47:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:03 smithi042 ceph-mon[32775]: pgmap v4112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:03 smithi164 ceph-mon[36970]: pgmap v4112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:03 smithi163 ceph-mon[36873]: pgmap v4112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:05 smithi042 ceph-mon[32775]: pgmap v4113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:47:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:05 smithi164 ceph-mon[36970]: pgmap v4113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:47:05.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:05 smithi163 ceph-mon[36873]: pgmap v4113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:47:06.522 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:47:06.522 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:47:06.522 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:47:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:47:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:47:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:47:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:47:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:06 smithi042 ceph-mon[32775]: pgmap v4114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:06 smithi042 ceph-mon[32775]: pgmap v4115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:47:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:47:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:47:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:47:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:47:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:47:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:06 smithi164 ceph-mon[36970]: pgmap v4114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:06 smithi164 ceph-mon[36970]: pgmap v4115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:47:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:47:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:47:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:06 smithi163 ceph-mon[36873]: pgmap v4114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:06 smithi163 ceph-mon[36873]: pgmap v4115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:47:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:47:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6656.263249 secs 2024-07-26T08:47:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6674.005891 secs 2024-07-26T08:47:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6656.263249 secs 2024-07-26T08:47:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6674.005891 secs 2024-07-26T08:47:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6656.263249 secs 2024-07-26T08:47:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6674.005891 secs 2024-07-26T08:47:08.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:08 smithi042 ceph-mon[32775]: pgmap v4116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:47:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:08 smithi164 ceph-mon[36970]: pgmap v4116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:47:09.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:08 smithi163 ceph-mon[36873]: pgmap v4116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:47:10.607 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:47:10.605+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:47:10.607 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:47:10.605+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:47:10.607 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:47:10.605+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:47:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:10 smithi042 ceph-mon[32775]: pgmap v4117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:47:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:10 smithi164 ceph-mon[36970]: pgmap v4117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:47:11.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:10 smithi163 ceph-mon[36873]: pgmap v4117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:47:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6661.263372 secs 2024-07-26T08:47:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6679.006015 secs 2024-07-26T08:47:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:12 smithi042 ceph-mon[32775]: pgmap v4118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:47:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6661.263372 secs 2024-07-26T08:47:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6679.006015 secs 2024-07-26T08:47:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:12 smithi164 ceph-mon[36970]: pgmap v4118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:47:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6661.263372 secs 2024-07-26T08:47:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6679.006015 secs 2024-07-26T08:47:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:12 smithi163 ceph-mon[36873]: pgmap v4118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:47:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:14 smithi042 ceph-mon[32775]: pgmap v4119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:47:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:14 smithi164 ceph-mon[36970]: pgmap v4119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:47:15.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:14 smithi163 ceph-mon[36873]: pgmap v4119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:47:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:16 smithi042 ceph-mon[32775]: pgmap v4120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:47:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:16 smithi164 ceph-mon[36970]: pgmap v4120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:47:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:16 smithi163 ceph-mon[36873]: pgmap v4120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:47:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6666.263496 secs 2024-07-26T08:47:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6684.006167 secs 2024-07-26T08:47:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6666.263496 secs 2024-07-26T08:47:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6684.006167 secs 2024-07-26T08:47:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6666.263496 secs 2024-07-26T08:47:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6684.006167 secs 2024-07-26T08:47:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:18 smithi042 ceph-mon[32775]: pgmap v4121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:18 smithi164 ceph-mon[36970]: pgmap v4121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:19.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:18 smithi163 ceph-mon[36873]: pgmap v4121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:20 smithi042 ceph-mon[32775]: pgmap v4122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:20 smithi164 ceph-mon[36970]: pgmap v4122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:20 smithi163 ceph-mon[36873]: pgmap v4122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6671.263619 secs 2024-07-26T08:47:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6689.006295 secs 2024-07-26T08:47:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:22 smithi042 ceph-mon[32775]: pgmap v4123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6671.263619 secs 2024-07-26T08:47:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6689.006295 secs 2024-07-26T08:47:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:22 smithi164 ceph-mon[36970]: pgmap v4123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6671.263619 secs 2024-07-26T08:47:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6689.006295 secs 2024-07-26T08:47:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:22 smithi163 ceph-mon[36873]: pgmap v4123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:24 smithi042 ceph-mon[32775]: pgmap v4124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:47:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:24 smithi164 ceph-mon[36970]: pgmap v4124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:47:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:24 smithi163 ceph-mon[36873]: pgmap v4124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:47:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:47:25.609+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:47:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:47:25.609+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:47:25.611 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:47:25.609+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:47:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:26 smithi042 ceph-mon[32775]: pgmap v4125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:26 smithi164 ceph-mon[36970]: pgmap v4125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:27.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:26 smithi163 ceph-mon[36873]: pgmap v4125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6676.263720 secs 2024-07-26T08:47:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6694.006419 secs 2024-07-26T08:47:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6676.263720 secs 2024-07-26T08:47:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6694.006419 secs 2024-07-26T08:47:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6676.263720 secs 2024-07-26T08:47:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6694.006419 secs 2024-07-26T08:47:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:28 smithi042 ceph-mon[32775]: pgmap v4126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:28 smithi164 ceph-mon[36970]: pgmap v4126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:29.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:28 smithi163 ceph-mon[36873]: pgmap v4126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:30 smithi042 ceph-mon[32775]: pgmap v4127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:30 smithi164 ceph-mon[36970]: pgmap v4127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:31.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:30 smithi163 ceph-mon[36873]: pgmap v4127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6681.263850 secs 2024-07-26T08:47:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6699.006564 secs 2024-07-26T08:47:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:32 smithi042 ceph-mon[32775]: pgmap v4128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6681.263850 secs 2024-07-26T08:47:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6699.006564 secs 2024-07-26T08:47:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:32 smithi164 ceph-mon[36970]: pgmap v4128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6681.263850 secs 2024-07-26T08:47:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6699.006564 secs 2024-07-26T08:47:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:32 smithi163 ceph-mon[36873]: pgmap v4128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:34 smithi042 ceph-mon[32775]: pgmap v4129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:47:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:34 smithi164 ceph-mon[36970]: pgmap v4129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:47:35.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:34 smithi163 ceph-mon[36873]: pgmap v4129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:47:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:36 smithi042 ceph-mon[32775]: pgmap v4130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:36 smithi164 ceph-mon[36970]: pgmap v4130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:36 smithi163 ceph-mon[36873]: pgmap v4130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6686.263971 secs 2024-07-26T08:47:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6704.006700 secs 2024-07-26T08:47:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6686.263971 secs 2024-07-26T08:47:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6704.006700 secs 2024-07-26T08:47:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6686.263971 secs 2024-07-26T08:47:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6704.006700 secs 2024-07-26T08:47:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:38 smithi042 ceph-mon[32775]: pgmap v4131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:38 smithi164 ceph-mon[36970]: pgmap v4131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:38 smithi163 ceph-mon[36873]: pgmap v4131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:40.614 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:47:40.613+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:47:40.615 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:47:40.613+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:47:40.615 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:47:40.613+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:47:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:40 smithi042 ceph-mon[32775]: pgmap v4132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:40 smithi164 ceph-mon[36970]: pgmap v4132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:40 smithi163 ceph-mon[36873]: pgmap v4132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6691.264097 secs 2024-07-26T08:47:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6709.006826 secs 2024-07-26T08:47:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:42 smithi042 ceph-mon[32775]: pgmap v4133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6691.264097 secs 2024-07-26T08:47:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6709.006826 secs 2024-07-26T08:47:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:42 smithi164 ceph-mon[36970]: pgmap v4133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6691.264097 secs 2024-07-26T08:47:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6709.006826 secs 2024-07-26T08:47:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:42 smithi163 ceph-mon[36873]: pgmap v4133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:44 smithi042 ceph-mon[32775]: pgmap v4134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:47:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:44 smithi164 ceph-mon[36970]: pgmap v4134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:47:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:44 smithi163 ceph-mon[36873]: pgmap v4134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:47:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:46 smithi042 ceph-mon[32775]: pgmap v4135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:46 smithi164 ceph-mon[36970]: pgmap v4135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:46 smithi163 ceph-mon[36873]: pgmap v4135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6696.264227 secs 2024-07-26T08:47:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6714.006966 secs 2024-07-26T08:47:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6696.264227 secs 2024-07-26T08:47:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6714.006966 secs 2024-07-26T08:47:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6696.264227 secs 2024-07-26T08:47:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6714.006966 secs 2024-07-26T08:47:48.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:48 smithi042 ceph-mon[32775]: pgmap v4136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:48 smithi164 ceph-mon[36970]: pgmap v4136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:48 smithi163 ceph-mon[36873]: pgmap v4136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:50 smithi042 ceph-mon[32775]: pgmap v4137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:50 smithi164 ceph-mon[36970]: pgmap v4137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:50 smithi163 ceph-mon[36873]: pgmap v4137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6701.264344 secs 2024-07-26T08:47:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6719.007094 secs 2024-07-26T08:47:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:52 smithi042 ceph-mon[32775]: pgmap v4138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6701.264344 secs 2024-07-26T08:47:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6719.007094 secs 2024-07-26T08:47:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:52 smithi164 ceph-mon[36970]: pgmap v4138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6701.264344 secs 2024-07-26T08:47:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6719.007094 secs 2024-07-26T08:47:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:52 smithi163 ceph-mon[36873]: pgmap v4138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:54 smithi042 ceph-mon[32775]: pgmap v4139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:47:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:54 smithi164 ceph-mon[36970]: pgmap v4139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:47:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:54 smithi163 ceph-mon[36873]: pgmap v4139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:47:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:47:55.618+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:47:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:47:55.618+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:47:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:47:55.618+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:47:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:56 smithi042 ceph-mon[32775]: pgmap v4140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:56 smithi164 ceph-mon[36970]: pgmap v4140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:56 smithi163 ceph-mon[36873]: pgmap v4140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:47:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6706.264471 secs 2024-07-26T08:47:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6724.007220 secs 2024-07-26T08:47:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6706.264471 secs 2024-07-26T08:47:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6724.007220 secs 2024-07-26T08:47:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6706.264471 secs 2024-07-26T08:47:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6724.007220 secs 2024-07-26T08:47:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:47:58 smithi042 ceph-mon[32775]: pgmap v4141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:47:58 smithi164 ceph-mon[36970]: pgmap v4141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:47:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:47:58 smithi163 ceph-mon[36873]: pgmap v4141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:00 smithi042 ceph-mon[32775]: pgmap v4142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:00 smithi164 ceph-mon[36970]: pgmap v4142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:00 smithi163 ceph-mon[36873]: pgmap v4142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6711.264593 secs 2024-07-26T08:48:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6729.007350 secs 2024-07-26T08:48:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:02 smithi042 ceph-mon[32775]: pgmap v4143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6711.264593 secs 2024-07-26T08:48:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6729.007350 secs 2024-07-26T08:48:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:02 smithi164 ceph-mon[36970]: pgmap v4143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6711.264593 secs 2024-07-26T08:48:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6729.007350 secs 2024-07-26T08:48:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:02 smithi163 ceph-mon[36873]: pgmap v4143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:04.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:04 smithi042 ceph-mon[32775]: pgmap v4144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:48:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:04 smithi164 ceph-mon[36970]: pgmap v4144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:48:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:04 smithi163 ceph-mon[36873]: pgmap v4144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:48:06.798 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:06 smithi163 ceph-mon[36873]: pgmap v4145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:06.798 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:48:06.799 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:06.799 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:06.799 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:06.799 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:06.799 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:06 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:48:06.799 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:06 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:48:06.808 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:06 smithi164 ceph-mon[36970]: pgmap v4145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:06.809 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:48:06.809 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:06.809 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:06.809 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:06.809 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:06.809 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:06 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:48:06.810 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:06 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:48:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:06 smithi042 ceph-mon[32775]: pgmap v4145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:48:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:06 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:48:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:06 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:48:07.843 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6716.264688 secs 2024-07-26T08:48:07.843 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6734.007418 secs 2024-07-26T08:48:07.921 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6716.264688 secs 2024-07-26T08:48:07.921 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6734.007418 secs 2024-07-26T08:48:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6716.264688 secs 2024-07-26T08:48:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6734.007418 secs 2024-07-26T08:48:08.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:08 smithi164 ceph-mon[36970]: pgmap v4146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:08 smithi163 ceph-mon[36873]: pgmap v4146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:08 smithi042 ceph-mon[32775]: pgmap v4146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:48:10.628+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:48:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:48:10.628+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:48:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:48:10.628+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:48:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:48:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:48:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:48:10.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:48:10.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:48:10.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:48:10.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:48:10.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:48:10.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:48:10.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:48:10.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: pgmap v4147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:10.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: pgmap v4148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:48:10.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:48:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:48:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:48:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:48:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:48:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:48:10.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:48:10.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:48:10.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:48:10.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:48:10.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:48:10.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: pgmap v4147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:10.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: pgmap v4148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:48:10.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:48:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:48:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:48:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:48:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:48:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:48:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:48:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:48:10.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:48:10.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:48:10.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:48:10.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:48:10.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: pgmap v4147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:10.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: pgmap v4148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:48:10.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:48:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6721.264811 secs 2024-07-26T08:48:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6739.007540 secs 2024-07-26T08:48:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:12 smithi042 ceph-mon[32775]: pgmap v4149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:48:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6721.264811 secs 2024-07-26T08:48:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6739.007540 secs 2024-07-26T08:48:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:12 smithi164 ceph-mon[36970]: pgmap v4149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:48:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6721.264811 secs 2024-07-26T08:48:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6739.007540 secs 2024-07-26T08:48:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:12 smithi163 ceph-mon[36873]: pgmap v4149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:48:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:14 smithi042 ceph-mon[32775]: pgmap v4150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:48:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:14 smithi164 ceph-mon[36970]: pgmap v4150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:48:15.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:14 smithi163 ceph-mon[36873]: pgmap v4150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:48:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:16 smithi042 ceph-mon[32775]: pgmap v4151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:48:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:16 smithi164 ceph-mon[36970]: pgmap v4151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:48:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:16 smithi163 ceph-mon[36873]: pgmap v4151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:48:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6726.264933 secs 2024-07-26T08:48:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6744.007685 secs 2024-07-26T08:48:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6726.264933 secs 2024-07-26T08:48:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6744.007685 secs 2024-07-26T08:48:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6726.264933 secs 2024-07-26T08:48:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6744.007685 secs 2024-07-26T08:48:18.846 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:18 smithi163 ceph-mon[36873]: pgmap v4152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:48:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:18 smithi042 ceph-mon[32775]: pgmap v4152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:48:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:18 smithi164 ceph-mon[36970]: pgmap v4152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T08:48:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:20 smithi042 ceph-mon[32775]: pgmap v4153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:48:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:20 smithi164 ceph-mon[36970]: pgmap v4153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:48:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:20 smithi163 ceph-mon[36873]: pgmap v4153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:48:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6731.265070 secs 2024-07-26T08:48:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6749.007750 secs 2024-07-26T08:48:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:22 smithi042 ceph-mon[32775]: pgmap v4154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6731.265070 secs 2024-07-26T08:48:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6749.007750 secs 2024-07-26T08:48:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:22 smithi164 ceph-mon[36970]: pgmap v4154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6731.265070 secs 2024-07-26T08:48:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6749.007750 secs 2024-07-26T08:48:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:22 smithi163 ceph-mon[36873]: pgmap v4154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:24 smithi042 ceph-mon[32775]: pgmap v4155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:48:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:24 smithi164 ceph-mon[36970]: pgmap v4155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:48:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:24 smithi163 ceph-mon[36873]: pgmap v4155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:48:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:48:25.631+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:48:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:48:25.631+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:48:25.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:48:25.631+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:48:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:26 smithi042 ceph-mon[32775]: pgmap v4156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:26 smithi164 ceph-mon[36970]: pgmap v4156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:26 smithi163 ceph-mon[36873]: pgmap v4156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6736.265203 secs 2024-07-26T08:48:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6754.007875 secs 2024-07-26T08:48:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6736.265203 secs 2024-07-26T08:48:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6754.007875 secs 2024-07-26T08:48:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6736.265203 secs 2024-07-26T08:48:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6754.007875 secs 2024-07-26T08:48:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:28 smithi042 ceph-mon[32775]: pgmap v4157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:28 smithi164 ceph-mon[36970]: pgmap v4157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:29.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:28 smithi163 ceph-mon[36873]: pgmap v4157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:30 smithi042 ceph-mon[32775]: pgmap v4158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:30 smithi164 ceph-mon[36970]: pgmap v4158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:31.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:30 smithi163 ceph-mon[36873]: pgmap v4158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6741.265335 secs 2024-07-26T08:48:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6759.007989 secs 2024-07-26T08:48:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:32 smithi042 ceph-mon[32775]: pgmap v4159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6741.265335 secs 2024-07-26T08:48:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6759.007989 secs 2024-07-26T08:48:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:32 smithi164 ceph-mon[36970]: pgmap v4159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6741.265335 secs 2024-07-26T08:48:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6759.007989 secs 2024-07-26T08:48:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:32 smithi163 ceph-mon[36873]: pgmap v4159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:34 smithi042 ceph-mon[32775]: pgmap v4160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:48:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:34 smithi164 ceph-mon[36970]: pgmap v4160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:48:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:34 smithi163 ceph-mon[36873]: pgmap v4160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:48:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:36 smithi042 ceph-mon[32775]: pgmap v4161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:36 smithi164 ceph-mon[36970]: pgmap v4161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:36 smithi163 ceph-mon[36873]: pgmap v4161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6746.265490 secs 2024-07-26T08:48:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6764.008082 secs 2024-07-26T08:48:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6746.265490 secs 2024-07-26T08:48:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6764.008082 secs 2024-07-26T08:48:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6746.265490 secs 2024-07-26T08:48:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6764.008082 secs 2024-07-26T08:48:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:38 smithi042 ceph-mon[32775]: pgmap v4162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:38 smithi164 ceph-mon[36970]: pgmap v4162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:38 smithi163 ceph-mon[36873]: pgmap v4162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:48:40.640+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:48:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:48:40.640+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:48:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:48:40.640+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:48:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:40 smithi042 ceph-mon[32775]: pgmap v4163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:40 smithi164 ceph-mon[36970]: pgmap v4163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:40 smithi163 ceph-mon[36873]: pgmap v4163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6751.265640 secs 2024-07-26T08:48:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6769.008152 secs 2024-07-26T08:48:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:42 smithi042 ceph-mon[32775]: pgmap v4164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6751.265640 secs 2024-07-26T08:48:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6769.008152 secs 2024-07-26T08:48:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:42 smithi164 ceph-mon[36970]: pgmap v4164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6751.265640 secs 2024-07-26T08:48:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6769.008152 secs 2024-07-26T08:48:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:42 smithi163 ceph-mon[36873]: pgmap v4164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:44 smithi042 ceph-mon[32775]: pgmap v4165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:48:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:44 smithi164 ceph-mon[36970]: pgmap v4165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:48:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:44 smithi163 ceph-mon[36873]: pgmap v4165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:48:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:46 smithi042 ceph-mon[32775]: pgmap v4166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:46 smithi164 ceph-mon[36970]: pgmap v4166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:46 smithi163 ceph-mon[36873]: pgmap v4166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6756.265709 secs 2024-07-26T08:48:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6774.008277 secs 2024-07-26T08:48:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6756.265709 secs 2024-07-26T08:48:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6774.008277 secs 2024-07-26T08:48:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6756.265709 secs 2024-07-26T08:48:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6774.008277 secs 2024-07-26T08:48:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:48 smithi164 ceph-mon[36970]: pgmap v4167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:48 smithi163 ceph-mon[36873]: pgmap v4167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:48 smithi042 ceph-mon[32775]: pgmap v4167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:50 smithi164 ceph-mon[36970]: pgmap v4168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:50 smithi163 ceph-mon[36873]: pgmap v4168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:50 smithi042 ceph-mon[32775]: pgmap v4168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6761.265852 secs 2024-07-26T08:48:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6779.008397 secs 2024-07-26T08:48:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:52 smithi164 ceph-mon[36970]: pgmap v4169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6761.265852 secs 2024-07-26T08:48:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6779.008397 secs 2024-07-26T08:48:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:52 smithi163 ceph-mon[36873]: pgmap v4169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6761.265852 secs 2024-07-26T08:48:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6779.008397 secs 2024-07-26T08:48:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:52 smithi042 ceph-mon[32775]: pgmap v4169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:54 smithi164 ceph-mon[36970]: pgmap v4170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:48:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:54 smithi163 ceph-mon[36873]: pgmap v4170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:48:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:54 smithi042 ceph-mon[32775]: pgmap v4170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:48:55.646 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:48:55.644+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:48:55.646 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:48:55.644+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:48:55.646 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:48:55.644+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:48:56.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:56 smithi042 ceph-mon[32775]: pgmap v4171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:56 smithi164 ceph-mon[36970]: pgmap v4171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:56 smithi163 ceph-mon[36873]: pgmap v4171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:48:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6766.265962 secs 2024-07-26T08:48:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6784.008515 secs 2024-07-26T08:48:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6766.265962 secs 2024-07-26T08:48:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6784.008515 secs 2024-07-26T08:48:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6766.265962 secs 2024-07-26T08:48:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6784.008515 secs 2024-07-26T08:48:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:48:58 smithi164 ceph-mon[36970]: pgmap v4172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:48:58 smithi163 ceph-mon[36873]: pgmap v4172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:48:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:48:58 smithi042 ceph-mon[32775]: pgmap v4172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:00 smithi164 ceph-mon[36970]: pgmap v4173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:00 smithi163 ceph-mon[36873]: pgmap v4173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:00 smithi042 ceph-mon[32775]: pgmap v4173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6771.266090 secs 2024-07-26T08:49:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6789.008652 secs 2024-07-26T08:49:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:02 smithi164 ceph-mon[36970]: pgmap v4174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6771.266090 secs 2024-07-26T08:49:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6789.008652 secs 2024-07-26T08:49:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:02 smithi163 ceph-mon[36873]: pgmap v4174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6771.266090 secs 2024-07-26T08:49:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6789.008652 secs 2024-07-26T08:49:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:02 smithi042 ceph-mon[32775]: pgmap v4174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:04 smithi164 ceph-mon[36970]: pgmap v4175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:49:05.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:04 smithi163 ceph-mon[36873]: pgmap v4175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:49:05.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:04 smithi042 ceph-mon[32775]: pgmap v4175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:49:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:06 smithi164 ceph-mon[36970]: pgmap v4176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:06 smithi163 ceph-mon[36873]: pgmap v4176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:06 smithi042 ceph-mon[32775]: pgmap v4176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6776.266236 secs 2024-07-26T08:49:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6794.008807 secs 2024-07-26T08:49:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6776.266236 secs 2024-07-26T08:49:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6794.008807 secs 2024-07-26T08:49:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6776.266236 secs 2024-07-26T08:49:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6794.008807 secs 2024-07-26T08:49:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:08 smithi164 ceph-mon[36970]: pgmap v4177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:08 smithi163 ceph-mon[36873]: pgmap v4177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:09.104 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:08 smithi042 ceph-mon[32775]: pgmap v4177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:49:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:49:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:49:10.654 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:49:10.653+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:49:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:49:10.653+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:49:10.655 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:49:10.653+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:49:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:10 smithi164 ceph-mon[36970]: pgmap v4178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:49:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:49:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:10 smithi164 ceph-mon[36970]: pgmap v4179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:49:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:10 smithi164 ceph-mon[36970]: pgmap v4180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:49:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:49:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:10 smithi163 ceph-mon[36873]: pgmap v4178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:49:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:49:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:10 smithi163 ceph-mon[36873]: pgmap v4179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:49:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:10 smithi163 ceph-mon[36873]: pgmap v4180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:49:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:49:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:10 smithi042 ceph-mon[32775]: pgmap v4178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:49:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:49:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:10 smithi042 ceph-mon[32775]: pgmap v4179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:49:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:10 smithi042 ceph-mon[32775]: pgmap v4180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:49:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:49:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6781.266332 secs 2024-07-26T08:49:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6781.266332 secs 2024-07-26T08:49:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6781.266332 secs 2024-07-26T08:49:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6799.008953 secs 2024-07-26T08:49:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:12 smithi164 ceph-mon[36970]: pgmap v4181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:49:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6799.008953 secs 2024-07-26T08:49:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:12 smithi163 ceph-mon[36873]: pgmap v4181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:49:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6799.008953 secs 2024-07-26T08:49:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:12 smithi042 ceph-mon[32775]: pgmap v4181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:49:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:15 smithi164 ceph-mon[36970]: pgmap v4182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:49:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:15 smithi163 ceph-mon[36873]: pgmap v4182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:49:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:15 smithi042 ceph-mon[32775]: pgmap v4182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:49:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6786.266479 secs 2024-07-26T08:49:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:17 smithi164 ceph-mon[36970]: pgmap v4183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:49:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6804.009018 secs 2024-07-26T08:49:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6786.266479 secs 2024-07-26T08:49:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:17 smithi163 ceph-mon[36873]: pgmap v4183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:49:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6804.009018 secs 2024-07-26T08:49:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6786.266479 secs 2024-07-26T08:49:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:17 smithi042 ceph-mon[32775]: pgmap v4183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:49:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6804.009018 secs 2024-07-26T08:49:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:19 smithi164 ceph-mon[36970]: pgmap v4184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:49:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:19 smithi163 ceph-mon[36873]: pgmap v4184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:49:19.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:19 smithi042 ceph-mon[32775]: pgmap v4184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:49:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:21 smithi164 ceph-mon[36970]: pgmap v4185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:49:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:21 smithi163 ceph-mon[36873]: pgmap v4185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:49:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:21 smithi042 ceph-mon[32775]: pgmap v4185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:49:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6791.266568 secs 2024-07-26T08:49:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6809.009141 secs 2024-07-26T08:49:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6791.266568 secs 2024-07-26T08:49:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6809.009141 secs 2024-07-26T08:49:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6791.266568 secs 2024-07-26T08:49:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6809.009141 secs 2024-07-26T08:49:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:23 smithi164 ceph-mon[36970]: pgmap v4186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:23 smithi163 ceph-mon[36873]: pgmap v4186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:23 smithi042 ceph-mon[32775]: pgmap v4186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:25 smithi042 ceph-mon[32775]: pgmap v4187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:25 smithi164 ceph-mon[36970]: pgmap v4187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:25 smithi163 ceph-mon[36873]: pgmap v4187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:49:25.657+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:49:25.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:49:25.657+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:49:25.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:49:25.657+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:49:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6796.266697 secs 2024-07-26T08:49:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:27 smithi164 ceph-mon[36970]: pgmap v4188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6814.009230 secs 2024-07-26T08:49:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6796.266697 secs 2024-07-26T08:49:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:27 smithi163 ceph-mon[36873]: pgmap v4188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6814.009230 secs 2024-07-26T08:49:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6796.266697 secs 2024-07-26T08:49:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:27 smithi042 ceph-mon[32775]: pgmap v4188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6814.009230 secs 2024-07-26T08:49:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:29 smithi164 ceph-mon[36970]: pgmap v4189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:29 smithi163 ceph-mon[36873]: pgmap v4189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:29 smithi042 ceph-mon[32775]: pgmap v4189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:31.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:31 smithi164 ceph-mon[36970]: pgmap v4190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:31 smithi163 ceph-mon[36873]: pgmap v4190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:31 smithi042 ceph-mon[32775]: pgmap v4190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6801.266822 secs 2024-07-26T08:49:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6819.009358 secs 2024-07-26T08:49:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6801.266822 secs 2024-07-26T08:49:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6819.009358 secs 2024-07-26T08:49:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6801.266822 secs 2024-07-26T08:49:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6819.009358 secs 2024-07-26T08:49:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:33 smithi164 ceph-mon[36970]: pgmap v4191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:33 smithi163 ceph-mon[36873]: pgmap v4191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:33.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:33 smithi042 ceph-mon[32775]: pgmap v4191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:35 smithi164 ceph-mon[36970]: pgmap v4192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:35 smithi163 ceph-mon[36873]: pgmap v4192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:35.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:35 smithi042 ceph-mon[32775]: pgmap v4192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6806.266969 secs 2024-07-26T08:49:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:37 smithi164 ceph-mon[36970]: pgmap v4193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6824.009500 secs 2024-07-26T08:49:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6806.266969 secs 2024-07-26T08:49:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:37 smithi163 ceph-mon[36873]: pgmap v4193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6824.009500 secs 2024-07-26T08:49:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6806.266969 secs 2024-07-26T08:49:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:37 smithi042 ceph-mon[32775]: pgmap v4193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6824.009500 secs 2024-07-26T08:49:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:39 smithi164 ceph-mon[36970]: pgmap v4194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:39 smithi163 ceph-mon[36873]: pgmap v4194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:39 smithi042 ceph-mon[32775]: pgmap v4194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:49:40.666+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:49:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:49:40.666+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:49:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:49:40.666+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:49:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:41 smithi164 ceph-mon[36970]: pgmap v4195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:41 smithi163 ceph-mon[36873]: pgmap v4195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:41 smithi042 ceph-mon[32775]: pgmap v4195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6811.267097 secs 2024-07-26T08:49:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6829.009659 secs 2024-07-26T08:49:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6811.267097 secs 2024-07-26T08:49:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6829.009659 secs 2024-07-26T08:49:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6811.267097 secs 2024-07-26T08:49:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6829.009659 secs 2024-07-26T08:49:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:43 smithi164 ceph-mon[36970]: pgmap v4196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:43 smithi163 ceph-mon[36873]: pgmap v4196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:43 smithi042 ceph-mon[32775]: pgmap v4196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:45 smithi164 ceph-mon[36970]: pgmap v4197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:45 smithi163 ceph-mon[36873]: pgmap v4197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:45 smithi042 ceph-mon[32775]: pgmap v4197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6816.267247 secs 2024-07-26T08:49:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:47 smithi164 ceph-mon[36970]: pgmap v4198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6834.009799 secs 2024-07-26T08:49:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6816.267247 secs 2024-07-26T08:49:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:47 smithi163 ceph-mon[36873]: pgmap v4198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6834.009799 secs 2024-07-26T08:49:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6816.267247 secs 2024-07-26T08:49:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:47 smithi042 ceph-mon[32775]: pgmap v4198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6834.009799 secs 2024-07-26T08:49:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:49 smithi164 ceph-mon[36970]: pgmap v4199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:49 smithi163 ceph-mon[36873]: pgmap v4199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:49 smithi042 ceph-mon[32775]: pgmap v4199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:51 smithi163 ceph-mon[36873]: pgmap v4200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:51 smithi042 ceph-mon[32775]: pgmap v4200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:51 smithi164 ceph-mon[36970]: pgmap v4200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6821.267311 secs 2024-07-26T08:49:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6839.009963 secs 2024-07-26T08:49:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6821.267311 secs 2024-07-26T08:49:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6839.009963 secs 2024-07-26T08:49:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6821.267311 secs 2024-07-26T08:49:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6839.009963 secs 2024-07-26T08:49:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:53 smithi163 ceph-mon[36873]: pgmap v4201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:53 smithi042 ceph-mon[32775]: pgmap v4201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:53 smithi164 ceph-mon[36970]: pgmap v4201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:55 smithi163 ceph-mon[36873]: pgmap v4202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:55 smithi042 ceph-mon[32775]: pgmap v4202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:55.671 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:49:55.670+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:49:55.672 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:49:55.670+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:49:55.672 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:49:55.670+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:49:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:55 smithi164 ceph-mon[36970]: pgmap v4202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:49:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6826.267396 secs 2024-07-26T08:49:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:57 smithi163 ceph-mon[36873]: pgmap v4203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6844.010087 secs 2024-07-26T08:49:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6826.267396 secs 2024-07-26T08:49:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:57 smithi042 ceph-mon[32775]: pgmap v4203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6844.010087 secs 2024-07-26T08:49:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6826.267396 secs 2024-07-26T08:49:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:57 smithi164 ceph-mon[36970]: pgmap v4203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6844.010087 secs 2024-07-26T08:49:58.916 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:49:58.915+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:49:58.916 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:49:58.915+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:49:58.920 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:49:58.920+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:49:58.920 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:49:58.920+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:49:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:49:59 smithi163 ceph-mon[36873]: pgmap v4204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:49:59 smithi042 ceph-mon[32775]: pgmap v4204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:49:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:49:59 smithi164 ceph-mon[36970]: pgmap v4204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:50:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:50:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T08:50:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:01 smithi163 ceph-mon[36873]: pgmap v4205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:01 smithi042 ceph-mon[32775]: pgmap v4205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:01 smithi164 ceph-mon[36970]: pgmap v4205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6831.267525 secs 2024-07-26T08:50:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6849.010210 secs 2024-07-26T08:50:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6831.267525 secs 2024-07-26T08:50:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6849.010210 secs 2024-07-26T08:50:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6831.267525 secs 2024-07-26T08:50:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6849.010210 secs 2024-07-26T08:50:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:03 smithi163 ceph-mon[36873]: pgmap v4206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:03 smithi042 ceph-mon[32775]: pgmap v4206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:03 smithi164 ceph-mon[36970]: pgmap v4206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:05 smithi042 ceph-mon[32775]: pgmap v4207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:05 smithi164 ceph-mon[36970]: pgmap v4207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:05 smithi163 ceph-mon[36873]: pgmap v4207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6836.267690 secs 2024-07-26T08:50:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:07 smithi042 ceph-mon[32775]: pgmap v4208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6854.010330 secs 2024-07-26T08:50:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6836.267690 secs 2024-07-26T08:50:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:07 smithi164 ceph-mon[36970]: pgmap v4208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6854.010330 secs 2024-07-26T08:50:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6836.267690 secs 2024-07-26T08:50:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:07 smithi163 ceph-mon[36873]: pgmap v4208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6854.010330 secs 2024-07-26T08:50:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:09 smithi042 ceph-mon[32775]: pgmap v4209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:09 smithi164 ceph-mon[36970]: pgmap v4209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:09 smithi163 ceph-mon[36873]: pgmap v4209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:50:10.679 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:50:10.678+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:50:10.679 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:50:10.678+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:50:10.679 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:50:10.678+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:50:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:50:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:50:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:11 smithi042 ceph-mon[32775]: pgmap v4210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:50:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:50:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:50:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:11 smithi164 ceph-mon[36970]: pgmap v4210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:50:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:50:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:50:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:11 smithi163 ceph-mon[36873]: pgmap v4210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:50:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:50:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:50:12.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:12 smithi164 ceph-mon[36970]: pgmap v4211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:50:12.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:12 smithi164 ceph-mon[36970]: pgmap v4212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:50:12.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6841.267779 secs 2024-07-26T08:50:12.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6859.010460 secs 2024-07-26T08:50:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:12 smithi042 ceph-mon[32775]: pgmap v4211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:50:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:12 smithi042 ceph-mon[32775]: pgmap v4212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:50:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6841.267779 secs 2024-07-26T08:50:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6859.010460 secs 2024-07-26T08:50:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:12 smithi163 ceph-mon[36873]: pgmap v4211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:50:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:12 smithi163 ceph-mon[36873]: pgmap v4212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:50:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6841.267779 secs 2024-07-26T08:50:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6859.010460 secs 2024-07-26T08:50:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:14 smithi042 ceph-mon[32775]: pgmap v4213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:50:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:14 smithi164 ceph-mon[36970]: pgmap v4213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:50:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:14 smithi163 ceph-mon[36873]: pgmap v4213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:50:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:16 smithi042 ceph-mon[32775]: pgmap v4214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:50:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:16 smithi164 ceph-mon[36970]: pgmap v4214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:50:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:16 smithi163 ceph-mon[36873]: pgmap v4214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:50:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6846.267908 secs 2024-07-26T08:50:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6864.010584 secs 2024-07-26T08:50:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6846.267908 secs 2024-07-26T08:50:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6864.010584 secs 2024-07-26T08:50:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6846.267908 secs 2024-07-26T08:50:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6864.010584 secs 2024-07-26T08:50:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:18 smithi042 ceph-mon[32775]: pgmap v4215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:50:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:18 smithi164 ceph-mon[36970]: pgmap v4215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:50:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:18 smithi163 ceph-mon[36873]: pgmap v4215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:50:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:20 smithi042 ceph-mon[32775]: pgmap v4216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:50:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:20 smithi164 ceph-mon[36970]: pgmap v4216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:50:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:20 smithi163 ceph-mon[36873]: pgmap v4216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:50:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:22 smithi042 ceph-mon[32775]: pgmap v4217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:50:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6851.268037 secs 2024-07-26T08:50:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6869.010705 secs 2024-07-26T08:50:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:22 smithi164 ceph-mon[36970]: pgmap v4217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:50:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6851.268037 secs 2024-07-26T08:50:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6869.010705 secs 2024-07-26T08:50:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:22 smithi163 ceph-mon[36873]: pgmap v4217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:50:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6851.268037 secs 2024-07-26T08:50:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6869.010705 secs 2024-07-26T08:50:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:24 smithi042 ceph-mon[32775]: pgmap v4218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:24 smithi164 ceph-mon[36970]: pgmap v4218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:24 smithi163 ceph-mon[36873]: pgmap v4218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:50:25.682+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:50:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:50:25.682+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:50:25.684 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:50:25.683+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:50:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:26 smithi042 ceph-mon[32775]: pgmap v4219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:26 smithi164 ceph-mon[36970]: pgmap v4219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:26 smithi163 ceph-mon[36873]: pgmap v4219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6856.268172 secs 2024-07-26T08:50:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6874.010847 secs 2024-07-26T08:50:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6856.268172 secs 2024-07-26T08:50:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6874.010847 secs 2024-07-26T08:50:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6856.268172 secs 2024-07-26T08:50:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6874.010847 secs 2024-07-26T08:50:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:28 smithi042 ceph-mon[32775]: pgmap v4220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:28 smithi164 ceph-mon[36970]: pgmap v4220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:28 smithi163 ceph-mon[36873]: pgmap v4220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:30 smithi042 ceph-mon[32775]: pgmap v4221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:50:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:30 smithi164 ceph-mon[36970]: pgmap v4221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:50:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:30 smithi163 ceph-mon[36873]: pgmap v4221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:50:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:32 smithi042 ceph-mon[32775]: pgmap v4222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6861.268271 secs 2024-07-26T08:50:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6879.010976 secs 2024-07-26T08:50:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:32 smithi164 ceph-mon[36970]: pgmap v4222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6861.268271 secs 2024-07-26T08:50:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6879.010976 secs 2024-07-26T08:50:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:32 smithi163 ceph-mon[36873]: pgmap v4222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6861.268271 secs 2024-07-26T08:50:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6879.010976 secs 2024-07-26T08:50:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:34 smithi042 ceph-mon[32775]: pgmap v4223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:34 smithi164 ceph-mon[36970]: pgmap v4223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:34 smithi163 ceph-mon[36873]: pgmap v4223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:36 smithi042 ceph-mon[32775]: pgmap v4224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:36 smithi164 ceph-mon[36970]: pgmap v4224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:36 smithi163 ceph-mon[36873]: pgmap v4224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6866.268397 secs 2024-07-26T08:50:37.693 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6884.011100 secs 2024-07-26T08:50:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6866.268397 secs 2024-07-26T08:50:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6884.011100 secs 2024-07-26T08:50:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6866.268397 secs 2024-07-26T08:50:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6884.011100 secs 2024-07-26T08:50:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:38 smithi042 ceph-mon[32775]: pgmap v4225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:38 smithi164 ceph-mon[36970]: pgmap v4225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:38 smithi163 ceph-mon[36873]: pgmap v4225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:40 smithi042 ceph-mon[32775]: pgmap v4226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:50:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:50:40.692+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:50:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:50:40.692+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:50:40.694 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:50:40.692+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:50:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:40 smithi164 ceph-mon[36970]: pgmap v4226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:50:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:40 smithi163 ceph-mon[36873]: pgmap v4226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:50:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:42 smithi042 ceph-mon[32775]: pgmap v4227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6871.268518 secs 2024-07-26T08:50:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6889.011222 secs 2024-07-26T08:50:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:42 smithi164 ceph-mon[36970]: pgmap v4227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6871.268518 secs 2024-07-26T08:50:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6889.011222 secs 2024-07-26T08:50:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:42 smithi163 ceph-mon[36873]: pgmap v4227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6871.268518 secs 2024-07-26T08:50:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6889.011222 secs 2024-07-26T08:50:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:44 smithi042 ceph-mon[32775]: pgmap v4228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:44 smithi164 ceph-mon[36970]: pgmap v4228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:44 smithi163 ceph-mon[36873]: pgmap v4228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:46 smithi042 ceph-mon[32775]: pgmap v4229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:46 smithi164 ceph-mon[36970]: pgmap v4229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:46 smithi163 ceph-mon[36873]: pgmap v4229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6876.268639 secs 2024-07-26T08:50:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6894.011285 secs 2024-07-26T08:50:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6876.268639 secs 2024-07-26T08:50:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6894.011285 secs 2024-07-26T08:50:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6876.268639 secs 2024-07-26T08:50:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6894.011285 secs 2024-07-26T08:50:48.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:48 smithi042 ceph-mon[32775]: pgmap v4230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:48 smithi164 ceph-mon[36970]: pgmap v4230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:48 smithi163 ceph-mon[36873]: pgmap v4230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:49 smithi164 ceph-mon[36970]: pgmap v4231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:50:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:49 smithi163 ceph-mon[36873]: pgmap v4231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:50:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:49 smithi042 ceph-mon[32775]: pgmap v4231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:50:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:51 smithi164 ceph-mon[36970]: pgmap v4232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:51 smithi163 ceph-mon[36873]: pgmap v4232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:51 smithi042 ceph-mon[32775]: pgmap v4232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6881.268762 secs 2024-07-26T08:50:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6899.011413 secs 2024-07-26T08:50:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6881.268762 secs 2024-07-26T08:50:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6899.011413 secs 2024-07-26T08:50:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6881.268762 secs 2024-07-26T08:50:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6899.011413 secs 2024-07-26T08:50:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:53 smithi163 ceph-mon[36873]: pgmap v4233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:53 smithi042 ceph-mon[32775]: pgmap v4233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:53 smithi164 ceph-mon[36970]: pgmap v4233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:50:55.696+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:50:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:50:55.696+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:50:55.698 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:50:55.696+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:50:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:55 smithi163 ceph-mon[36873]: pgmap v4234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:55 smithi042 ceph-mon[32775]: pgmap v4234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:55 smithi164 ceph-mon[36970]: pgmap v4234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:50:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6886.268908 secs 2024-07-26T08:50:57.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6886.268908 secs 2024-07-26T08:50:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6886.268908 secs 2024-07-26T08:50:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6904.011535 secs 2024-07-26T08:50:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:57 smithi163 ceph-mon[36873]: pgmap v4235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6904.011535 secs 2024-07-26T08:50:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:57 smithi042 ceph-mon[32775]: pgmap v4235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:50:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6904.011535 secs 2024-07-26T08:50:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:57 smithi164 ceph-mon[36970]: pgmap v4235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:50:59 smithi163 ceph-mon[36873]: pgmap v4236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:51:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:50:59 smithi042 ceph-mon[32775]: pgmap v4236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:51:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:50:59 smithi164 ceph-mon[36970]: pgmap v4236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:51:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:01 smithi163 ceph-mon[36873]: pgmap v4237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6891.269042 secs 2024-07-26T08:51:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:01 smithi042 ceph-mon[32775]: pgmap v4237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6891.269042 secs 2024-07-26T08:51:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:01 smithi164 ceph-mon[36970]: pgmap v4237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6891.269042 secs 2024-07-26T08:51:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6909.011677 secs 2024-07-26T08:51:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6909.011677 secs 2024-07-26T08:51:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6909.011677 secs 2024-07-26T08:51:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:03 smithi163 ceph-mon[36873]: pgmap v4238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:03 smithi042 ceph-mon[32775]: pgmap v4238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:03 smithi164 ceph-mon[36970]: pgmap v4238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:05 smithi163 ceph-mon[36873]: pgmap v4239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:05 smithi042 ceph-mon[32775]: pgmap v4239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:05 smithi164 ceph-mon[36970]: pgmap v4239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6896.269112 secs 2024-07-26T08:51:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6896.269112 secs 2024-07-26T08:51:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6896.269112 secs 2024-07-26T08:51:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6914.011826 secs 2024-07-26T08:51:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:07 smithi042 ceph-mon[32775]: pgmap v4240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6914.011826 secs 2024-07-26T08:51:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:07 smithi164 ceph-mon[36970]: pgmap v4240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6914.011826 secs 2024-07-26T08:51:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:07 smithi163 ceph-mon[36873]: pgmap v4240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:09 smithi042 ceph-mon[32775]: pgmap v4241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:51:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:09 smithi164 ceph-mon[36970]: pgmap v4241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:51:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:09 smithi163 ceph-mon[36873]: pgmap v4241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:51:10.707 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:51:10.705+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:51:10.707 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:51:10.705+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:51:10.707 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:51:10.705+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:51:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:10 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:51:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:10 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:51:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:10 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:51:12.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:11 smithi042 ceph-mon[32775]: pgmap v4242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:12.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6901.269226 secs 2024-07-26T08:51:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:11 smithi164 ceph-mon[36970]: pgmap v4242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6901.269226 secs 2024-07-26T08:51:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:11 smithi163 ceph-mon[36873]: pgmap v4242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6901.269226 secs 2024-07-26T08:51:13.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6919.011922 secs 2024-07-26T08:51:13.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:13.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:13.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:13.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6919.011922 secs 2024-07-26T08:51:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:13.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:13.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6919.011922 secs 2024-07-26T08:51:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:13 smithi042 ceph-mon[32775]: pgmap v4243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:51:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:51:14.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:13 smithi042 ceph-mon[32775]: pgmap v4244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:51:14.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:13 smithi042 ceph-mon[32775]: pgmap v4245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:51:14.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:13 smithi164 ceph-mon[36970]: pgmap v4243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:51:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:51:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:13 smithi164 ceph-mon[36970]: pgmap v4244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:51:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:13 smithi164 ceph-mon[36970]: pgmap v4245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:51:14.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:13 smithi163 ceph-mon[36873]: pgmap v4243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:51:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:51:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:13 smithi163 ceph-mon[36873]: pgmap v4244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:51:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:13 smithi163 ceph-mon[36873]: pgmap v4245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:51:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:51:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:16 smithi163 ceph-mon[36873]: pgmap v4246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:51:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:16 smithi042 ceph-mon[32775]: pgmap v4246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:51:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:16 smithi164 ceph-mon[36970]: pgmap v4246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:51:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6906.269339 secs 2024-07-26T08:51:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6924.012049 secs 2024-07-26T08:51:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6906.269339 secs 2024-07-26T08:51:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6924.012049 secs 2024-07-26T08:51:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6906.269339 secs 2024-07-26T08:51:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6924.012049 secs 2024-07-26T08:51:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:18 smithi163 ceph-mon[36873]: pgmap v4247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:51:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:18 smithi042 ceph-mon[32775]: pgmap v4247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:51:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:18 smithi164 ceph-mon[36970]: pgmap v4247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:51:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:20 smithi163 ceph-mon[36873]: pgmap v4248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:51:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:20 smithi042 ceph-mon[32775]: pgmap v4248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:51:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:20 smithi164 ceph-mon[36970]: pgmap v4248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:51:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6911.269440 secs 2024-07-26T08:51:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6929.012199 secs 2024-07-26T08:51:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:22 smithi163 ceph-mon[36873]: pgmap v4249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:51:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6911.269440 secs 2024-07-26T08:51:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6929.012199 secs 2024-07-26T08:51:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:22 smithi042 ceph-mon[32775]: pgmap v4249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:51:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6911.269440 secs 2024-07-26T08:51:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6929.012199 secs 2024-07-26T08:51:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:22 smithi164 ceph-mon[36970]: pgmap v4249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:51:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:24 smithi163 ceph-mon[36873]: pgmap v4250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:51:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:24 smithi042 ceph-mon[32775]: pgmap v4250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:51:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:24 smithi164 ceph-mon[36970]: pgmap v4250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:51:25.710 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:51:25.709+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:51:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:51:25.709+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:51:25.711 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:51:25.709+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:51:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:26 smithi163 ceph-mon[36873]: pgmap v4251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:26 smithi042 ceph-mon[32775]: pgmap v4251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:26 smithi164 ceph-mon[36970]: pgmap v4251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:27.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6916.269564 secs 2024-07-26T08:51:27.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6934.012345 secs 2024-07-26T08:51:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6916.269564 secs 2024-07-26T08:51:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6934.012345 secs 2024-07-26T08:51:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6916.269564 secs 2024-07-26T08:51:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6934.012345 secs 2024-07-26T08:51:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:28 smithi042 ceph-mon[32775]: pgmap v4252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:28 smithi164 ceph-mon[36970]: pgmap v4252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:28 smithi163 ceph-mon[36873]: pgmap v4252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:28.930 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:51:28.929+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:51:28.930 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:51:28.929+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:51:28.933 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:51:28.932+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:51:28.933 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:51:28.932+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:51:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:30 smithi042 ceph-mon[32775]: pgmap v4253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:30 smithi164 ceph-mon[36970]: pgmap v4253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:30 smithi163 ceph-mon[36873]: pgmap v4253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:32.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6921.269688 secs 2024-07-26T08:51:32.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6939.012468 secs 2024-07-26T08:51:32.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:32 smithi163 ceph-mon[36873]: pgmap v4254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6921.269688 secs 2024-07-26T08:51:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6939.012468 secs 2024-07-26T08:51:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:32 smithi042 ceph-mon[32775]: pgmap v4254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6921.269688 secs 2024-07-26T08:51:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6939.012468 secs 2024-07-26T08:51:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:32 smithi164 ceph-mon[36970]: pgmap v4254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:34 smithi042 ceph-mon[32775]: pgmap v4255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:34 smithi164 ceph-mon[36970]: pgmap v4255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:34 smithi163 ceph-mon[36873]: pgmap v4255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:36 smithi042 ceph-mon[32775]: pgmap v4256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:36 smithi164 ceph-mon[36970]: pgmap v4256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:36 smithi163 ceph-mon[36873]: pgmap v4256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6926.269828 secs 2024-07-26T08:51:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6944.012608 secs 2024-07-26T08:51:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6926.269828 secs 2024-07-26T08:51:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6944.012608 secs 2024-07-26T08:51:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6926.269828 secs 2024-07-26T08:51:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6944.012608 secs 2024-07-26T08:51:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:38 smithi042 ceph-mon[32775]: pgmap v4257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:38 smithi164 ceph-mon[36970]: pgmap v4257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:38 smithi163 ceph-mon[36873]: pgmap v4257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:40 smithi042 ceph-mon[32775]: pgmap v4258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:40.720 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:51:40.718+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:51:40.720 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:51:40.718+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:51:40.720 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:51:40.718+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:51:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:40 smithi164 ceph-mon[36970]: pgmap v4258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:40 smithi163 ceph-mon[36873]: pgmap v4258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6931.269956 secs 2024-07-26T08:51:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6949.012729 secs 2024-07-26T08:51:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:42 smithi042 ceph-mon[32775]: pgmap v4259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6931.269956 secs 2024-07-26T08:51:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6949.012729 secs 2024-07-26T08:51:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:42 smithi164 ceph-mon[36970]: pgmap v4259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6931.269956 secs 2024-07-26T08:51:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6949.012729 secs 2024-07-26T08:51:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:42 smithi163 ceph-mon[36873]: pgmap v4259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:44 smithi042 ceph-mon[32775]: pgmap v4260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:44 smithi164 ceph-mon[36970]: pgmap v4260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:44 smithi163 ceph-mon[36873]: pgmap v4260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:46 smithi042 ceph-mon[32775]: pgmap v4261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:46 smithi164 ceph-mon[36970]: pgmap v4261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:46 smithi163 ceph-mon[36873]: pgmap v4261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6936.270080 secs 2024-07-26T08:51:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6954.012848 secs 2024-07-26T08:51:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6936.270080 secs 2024-07-26T08:51:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6954.012848 secs 2024-07-26T08:51:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6936.270080 secs 2024-07-26T08:51:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6954.012848 secs 2024-07-26T08:51:48.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:48 smithi042 ceph-mon[32775]: pgmap v4262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:48 smithi164 ceph-mon[36970]: pgmap v4262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:48 smithi163 ceph-mon[36873]: pgmap v4262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:50 smithi042 ceph-mon[32775]: pgmap v4263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:50 smithi164 ceph-mon[36970]: pgmap v4263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:50 smithi163 ceph-mon[36873]: pgmap v4263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6941.270252 secs 2024-07-26T08:51:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6959.012977 secs 2024-07-26T08:51:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:52 smithi042 ceph-mon[32775]: pgmap v4264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6941.270252 secs 2024-07-26T08:51:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6959.012977 secs 2024-07-26T08:51:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:52 smithi164 ceph-mon[36970]: pgmap v4264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6941.270252 secs 2024-07-26T08:51:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6959.012977 secs 2024-07-26T08:51:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:52 smithi163 ceph-mon[36873]: pgmap v4264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:54 smithi042 ceph-mon[32775]: pgmap v4265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:54 smithi164 ceph-mon[36970]: pgmap v4265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:54 smithi163 ceph-mon[36873]: pgmap v4265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:55.723 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:51:55.721+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:51:55.723 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:51:55.721+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:51:55.723 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:51:55.721+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:51:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:56 smithi042 ceph-mon[32775]: pgmap v4266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:56 smithi164 ceph-mon[36970]: pgmap v4266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:56 smithi163 ceph-mon[36873]: pgmap v4266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:51:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6946.270394 secs 2024-07-26T08:51:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6964.013100 secs 2024-07-26T08:51:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6946.270394 secs 2024-07-26T08:51:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6964.013100 secs 2024-07-26T08:51:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6946.270394 secs 2024-07-26T08:51:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6964.013100 secs 2024-07-26T08:51:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:51:58 smithi042 ceph-mon[32775]: pgmap v4267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:51:58 smithi164 ceph-mon[36970]: pgmap v4267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:51:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:51:58 smithi163 ceph-mon[36873]: pgmap v4267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:00 smithi042 ceph-mon[32775]: pgmap v4268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:00 smithi164 ceph-mon[36970]: pgmap v4268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:00 smithi163 ceph-mon[36873]: pgmap v4268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6951.270490 secs 2024-07-26T08:52:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6969.013164 secs 2024-07-26T08:52:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:02 smithi042 ceph-mon[32775]: pgmap v4269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6951.270490 secs 2024-07-26T08:52:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6969.013164 secs 2024-07-26T08:52:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:02 smithi164 ceph-mon[36970]: pgmap v4269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6951.270490 secs 2024-07-26T08:52:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6969.013164 secs 2024-07-26T08:52:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:02 smithi163 ceph-mon[36873]: pgmap v4269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:04.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:04 smithi042 ceph-mon[32775]: pgmap v4270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:04 smithi164 ceph-mon[36970]: pgmap v4270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:04 smithi163 ceph-mon[36873]: pgmap v4270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:06 smithi042 ceph-mon[32775]: pgmap v4271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:06 smithi164 ceph-mon[36970]: pgmap v4271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:06 smithi163 ceph-mon[36873]: pgmap v4271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6956.270621 secs 2024-07-26T08:52:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6974.013263 secs 2024-07-26T08:52:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6956.270621 secs 2024-07-26T08:52:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6974.013263 secs 2024-07-26T08:52:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6956.270621 secs 2024-07-26T08:52:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6974.013263 secs 2024-07-26T08:52:08.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:08 smithi042 ceph-mon[32775]: pgmap v4272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:08 smithi164 ceph-mon[36970]: pgmap v4272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:08 smithi163 ceph-mon[36873]: pgmap v4272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:10 smithi042 ceph-mon[32775]: pgmap v4273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:10 smithi164 ceph-mon[36970]: pgmap v4273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:52:10.730+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:52:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:52:10.730+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:52:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:52:10.731+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:52:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:10 smithi163 ceph-mon[36873]: pgmap v4273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6961.270746 secs 2024-07-26T08:52:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6979.013364 secs 2024-07-26T08:52:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:12 smithi042 ceph-mon[32775]: pgmap v4274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6961.270746 secs 2024-07-26T08:52:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6979.013364 secs 2024-07-26T08:52:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:12 smithi164 ceph-mon[36970]: pgmap v4274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6961.270746 secs 2024-07-26T08:52:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6979.013364 secs 2024-07-26T08:52:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:12 smithi163 ceph-mon[36873]: pgmap v4274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:52:13.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:52:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:52:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:14 smithi042 ceph-mon[32775]: pgmap v4275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:52:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:52:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:14 smithi164 ceph-mon[36970]: pgmap v4275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:52:14.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:52:14.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:14 smithi163 ceph-mon[36873]: pgmap v4275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:52:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:52:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:52:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:15 smithi042 ceph-mon[32775]: pgmap v4276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-07-26T08:52:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:15 smithi042 ceph-mon[32775]: pgmap v4277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:52:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:15 smithi164 ceph-mon[36970]: pgmap v4276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-07-26T08:52:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:15 smithi164 ceph-mon[36970]: pgmap v4277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:52:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:15 smithi163 ceph-mon[36873]: pgmap v4276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-07-26T08:52:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:15 smithi163 ceph-mon[36873]: pgmap v4277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:52:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6966.270869 secs 2024-07-26T08:52:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:17 smithi042 ceph-mon[32775]: pgmap v4278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:52:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6984.013501 secs 2024-07-26T08:52:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6966.270869 secs 2024-07-26T08:52:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:17 smithi164 ceph-mon[36970]: pgmap v4278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:52:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6984.013501 secs 2024-07-26T08:52:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6966.270869 secs 2024-07-26T08:52:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:17 smithi163 ceph-mon[36873]: pgmap v4278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:52:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6984.013501 secs 2024-07-26T08:52:19.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:19 smithi042 ceph-mon[32775]: pgmap v4279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T08:52:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:19 smithi164 ceph-mon[36970]: pgmap v4279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T08:52:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:19 smithi163 ceph-mon[36873]: pgmap v4279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T08:52:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:20 smithi042 ceph-mon[32775]: pgmap v4280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:52:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:20 smithi164 ceph-mon[36970]: pgmap v4280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:52:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:20 smithi163 ceph-mon[36873]: pgmap v4280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:52:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6971.271032 secs 2024-07-26T08:52:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6971.271032 secs 2024-07-26T08:52:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6971.271032 secs 2024-07-26T08:52:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6989.013646 secs 2024-07-26T08:52:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:22 smithi042 ceph-mon[32775]: pgmap v4281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:52:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6989.013646 secs 2024-07-26T08:52:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:22 smithi164 ceph-mon[36970]: pgmap v4281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:52:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6989.013646 secs 2024-07-26T08:52:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:22 smithi163 ceph-mon[36873]: pgmap v4281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:52:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:25 smithi042 ceph-mon[32775]: pgmap v4282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:52:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:25 smithi164 ceph-mon[36970]: pgmap v4282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:52:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:25 smithi163 ceph-mon[36873]: pgmap v4282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:52:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:52:25.735+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:52:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:52:25.735+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:52:25.737 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:52:25.735+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:52:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6976.271157 secs 2024-07-26T08:52:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:27 smithi042 ceph-mon[32775]: pgmap v4283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6976.271157 secs 2024-07-26T08:52:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:27 smithi164 ceph-mon[36970]: pgmap v4283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6976.271157 secs 2024-07-26T08:52:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:27 smithi163 ceph-mon[36873]: pgmap v4283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:28 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6994.013770 secs 2024-07-26T08:52:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:28 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6994.013770 secs 2024-07-26T08:52:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:28 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6994.013770 secs 2024-07-26T08:52:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:29 smithi042 ceph-mon[32775]: pgmap v4284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:29 smithi164 ceph-mon[36970]: pgmap v4284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:29 smithi163 ceph-mon[36873]: pgmap v4284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:31 smithi042 ceph-mon[32775]: pgmap v4285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:31 smithi164 ceph-mon[36970]: pgmap v4285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:31 smithi163 ceph-mon[36873]: pgmap v4285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6981.271280 secs 2024-07-26T08:52:32.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6981.271280 secs 2024-07-26T08:52:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6981.271280 secs 2024-07-26T08:52:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:33 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6999.013890 secs 2024-07-26T08:52:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:33 smithi042 ceph-mon[32775]: pgmap v4286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:33 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6999.013890 secs 2024-07-26T08:52:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:33 smithi164 ceph-mon[36970]: pgmap v4286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:33 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6999.013890 secs 2024-07-26T08:52:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:33 smithi163 ceph-mon[36873]: pgmap v4286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:35 smithi042 ceph-mon[32775]: pgmap v4287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:35 smithi164 ceph-mon[36970]: pgmap v4287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:35 smithi163 ceph-mon[36873]: pgmap v4287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6986.271404 secs 2024-07-26T08:52:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:37 smithi042 ceph-mon[32775]: pgmap v4288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7004.014013 secs 2024-07-26T08:52:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6986.271404 secs 2024-07-26T08:52:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:37 smithi164 ceph-mon[36970]: pgmap v4288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7004.014013 secs 2024-07-26T08:52:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6986.271404 secs 2024-07-26T08:52:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:37 smithi163 ceph-mon[36873]: pgmap v4288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7004.014013 secs 2024-07-26T08:52:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:39 smithi042 ceph-mon[32775]: pgmap v4289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:39 smithi164 ceph-mon[36970]: pgmap v4289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:39 smithi163 ceph-mon[36873]: pgmap v4289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:40.745 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:52:40.744+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:52:40.746 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:52:40.744+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:52:40.746 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:52:40.744+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:52:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:41 smithi042 ceph-mon[32775]: pgmap v4290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:41 smithi164 ceph-mon[36970]: pgmap v4290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:41.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:41 smithi163 ceph-mon[36873]: pgmap v4290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6991.271548 secs 2024-07-26T08:52:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7009.014129 secs 2024-07-26T08:52:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6991.271548 secs 2024-07-26T08:52:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7009.014129 secs 2024-07-26T08:52:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6991.271548 secs 2024-07-26T08:52:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7009.014129 secs 2024-07-26T08:52:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:43 smithi042 ceph-mon[32775]: pgmap v4291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:43 smithi164 ceph-mon[36970]: pgmap v4291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:43 smithi163 ceph-mon[36873]: pgmap v4291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:45 smithi042 ceph-mon[32775]: pgmap v4292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:45 smithi164 ceph-mon[36970]: pgmap v4292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:45 smithi163 ceph-mon[36873]: pgmap v4292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 6996.271672 secs 2024-07-26T08:52:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:47 smithi042 ceph-mon[32775]: pgmap v4293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7014.014229 secs 2024-07-26T08:52:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 6996.271672 secs 2024-07-26T08:52:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:47 smithi164 ceph-mon[36970]: pgmap v4293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7014.014229 secs 2024-07-26T08:52:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 6996.271672 secs 2024-07-26T08:52:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:47 smithi163 ceph-mon[36873]: pgmap v4293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7014.014229 secs 2024-07-26T08:52:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:49 smithi042 ceph-mon[32775]: pgmap v4294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:49 smithi164 ceph-mon[36970]: pgmap v4294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:49 smithi163 ceph-mon[36873]: pgmap v4294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:51 smithi164 ceph-mon[36970]: pgmap v4295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:51 smithi163 ceph-mon[36873]: pgmap v4295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:51 smithi042 ceph-mon[32775]: pgmap v4295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7001.271795 secs 2024-07-26T08:52:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7019.014353 secs 2024-07-26T08:52:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7001.271795 secs 2024-07-26T08:52:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7019.014353 secs 2024-07-26T08:52:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7001.271795 secs 2024-07-26T08:52:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7019.014353 secs 2024-07-26T08:52:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:53 smithi164 ceph-mon[36970]: pgmap v4296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:53 smithi163 ceph-mon[36873]: pgmap v4296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:53 smithi042 ceph-mon[32775]: pgmap v4296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:55 smithi164 ceph-mon[36970]: pgmap v4297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:55 smithi163 ceph-mon[36873]: pgmap v4297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:55 smithi042 ceph-mon[32775]: pgmap v4297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:52:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:52:55.748+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:52:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:52:55.748+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:52:55.749 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:52:55.748+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:52:57.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7006.271894 secs 2024-07-26T08:52:57.449 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:57 smithi042 ceph-mon[32775]: pgmap v4298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:57.449 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7024.014492 secs 2024-07-26T08:52:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7006.271894 secs 2024-07-26T08:52:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:57 smithi164 ceph-mon[36970]: pgmap v4298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7024.014492 secs 2024-07-26T08:52:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7006.271894 secs 2024-07-26T08:52:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:57 smithi163 ceph-mon[36873]: pgmap v4298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7024.014492 secs 2024-07-26T08:52:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:52:58.942+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:52:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:52:58.942+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:52:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:52:58.945+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:52:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:52:58.945+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:52:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:52:59 smithi164 ceph-mon[36970]: pgmap v4299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:52:59 smithi163 ceph-mon[36873]: pgmap v4299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:52:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:52:59 smithi042 ceph-mon[32775]: pgmap v4299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:01 smithi042 ceph-mon[32775]: pgmap v4300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:01 smithi164 ceph-mon[36970]: pgmap v4300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:01 smithi163 ceph-mon[36873]: pgmap v4300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7011.271974 secs 2024-07-26T08:53:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7029.014614 secs 2024-07-26T08:53:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7011.271974 secs 2024-07-26T08:53:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7029.014614 secs 2024-07-26T08:53:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7011.271974 secs 2024-07-26T08:53:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7029.014614 secs 2024-07-26T08:53:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:03 smithi164 ceph-mon[36970]: pgmap v4301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:03 smithi163 ceph-mon[36873]: pgmap v4301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:03 smithi042 ceph-mon[32775]: pgmap v4301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:05 smithi164 ceph-mon[36970]: pgmap v4302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:05.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:05 smithi163 ceph-mon[36873]: pgmap v4302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:05 smithi042 ceph-mon[32775]: pgmap v4302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7016.272099 secs 2024-07-26T08:53:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:07 smithi164 ceph-mon[36970]: pgmap v4303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7034.014733 secs 2024-07-26T08:53:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7016.272099 secs 2024-07-26T08:53:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:07 smithi163 ceph-mon[36873]: pgmap v4303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7034.014733 secs 2024-07-26T08:53:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7016.272099 secs 2024-07-26T08:53:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:07 smithi042 ceph-mon[32775]: pgmap v4303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7034.014733 secs 2024-07-26T08:53:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:09 smithi164 ceph-mon[36970]: pgmap v4304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:09 smithi163 ceph-mon[36873]: pgmap v4304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:09 smithi042 ceph-mon[32775]: pgmap v4304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:53:10.758+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:53:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:53:10.758+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:53:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:53:10.758+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:53:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:11 smithi164 ceph-mon[36970]: pgmap v4305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:11 smithi163 ceph-mon[36873]: pgmap v4305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:11 smithi042 ceph-mon[32775]: pgmap v4305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7021.272244 secs 2024-07-26T08:53:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7039.014831 secs 2024-07-26T08:53:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7021.272244 secs 2024-07-26T08:53:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7039.014831 secs 2024-07-26T08:53:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7021.272244 secs 2024-07-26T08:53:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7039.014831 secs 2024-07-26T08:53:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:13 smithi164 ceph-mon[36970]: pgmap v4306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:13 smithi163 ceph-mon[36873]: pgmap v4306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:13 smithi042 ceph-mon[32775]: pgmap v4306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:53:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:53:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:53:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:15 smithi164 ceph-mon[36970]: pgmap v4307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:53:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:53:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:53:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:53:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:15 smithi163 ceph-mon[36873]: pgmap v4307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:53:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:53:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:53:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:53:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:15 smithi042 ceph-mon[32775]: pgmap v4307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:53:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:53:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:53:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:53:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:16 smithi164 ceph-mon[36970]: pgmap v4308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:16 smithi164 ceph-mon[36970]: pgmap v4309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T08:53:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:16 smithi163 ceph-mon[36873]: pgmap v4308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:16 smithi163 ceph-mon[36873]: pgmap v4309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T08:53:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:16 smithi042 ceph-mon[32775]: pgmap v4308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:16 smithi042 ceph-mon[32775]: pgmap v4309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T08:53:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7026.272320 secs 2024-07-26T08:53:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7044.014949 secs 2024-07-26T08:53:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7026.272320 secs 2024-07-26T08:53:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7044.014949 secs 2024-07-26T08:53:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7026.272320 secs 2024-07-26T08:53:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7044.014949 secs 2024-07-26T08:53:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:18 smithi163 ceph-mon[36873]: pgmap v4310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T08:53:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:18 smithi042 ceph-mon[32775]: pgmap v4310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T08:53:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:18 smithi164 ceph-mon[36970]: pgmap v4310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T08:53:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:20 smithi163 ceph-mon[36873]: pgmap v4311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 958 B/s rd, 3 op/s 2024-07-26T08:53:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:20 smithi042 ceph-mon[32775]: pgmap v4311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 958 B/s rd, 3 op/s 2024-07-26T08:53:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:20 smithi164 ceph-mon[36970]: pgmap v4311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 958 B/s rd, 3 op/s 2024-07-26T08:53:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:22 smithi163 ceph-mon[36873]: pgmap v4312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T08:53:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7031.272450 secs 2024-07-26T08:53:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7049.015074 secs 2024-07-26T08:53:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:22 smithi042 ceph-mon[32775]: pgmap v4312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T08:53:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7031.272450 secs 2024-07-26T08:53:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7049.015074 secs 2024-07-26T08:53:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:22 smithi164 ceph-mon[36970]: pgmap v4312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T08:53:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7031.272450 secs 2024-07-26T08:53:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7049.015074 secs 2024-07-26T08:53:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:24 smithi163 ceph-mon[36873]: pgmap v4313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:53:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:24 smithi042 ceph-mon[32775]: pgmap v4313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:53:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:24 smithi164 ceph-mon[36970]: pgmap v4313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:53:25.763 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:53:25.762+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:53:25.764 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:53:25.762+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:53:25.764 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:53:25.762+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:53:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:26 smithi163 ceph-mon[36873]: pgmap v4314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:53:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:26 smithi042 ceph-mon[32775]: pgmap v4314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:53:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:26 smithi164 ceph-mon[36970]: pgmap v4314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:53:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7036.272574 secs 2024-07-26T08:53:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7054.015194 secs 2024-07-26T08:53:27.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7036.272574 secs 2024-07-26T08:53:27.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7054.015194 secs 2024-07-26T08:53:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7036.272574 secs 2024-07-26T08:53:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7054.015194 secs 2024-07-26T08:53:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:28 smithi163 ceph-mon[36873]: pgmap v4315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:28 smithi042 ceph-mon[32775]: pgmap v4315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:28 smithi164 ceph-mon[36970]: pgmap v4315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:30 smithi163 ceph-mon[36873]: pgmap v4316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:53:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:30 smithi042 ceph-mon[32775]: pgmap v4316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:53:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:30 smithi164 ceph-mon[36970]: pgmap v4316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:53:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:32 smithi042 ceph-mon[32775]: pgmap v4317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7041.272700 secs 2024-07-26T08:53:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7059.015275 secs 2024-07-26T08:53:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:32 smithi164 ceph-mon[36970]: pgmap v4317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7041.272700 secs 2024-07-26T08:53:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7059.015275 secs 2024-07-26T08:53:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:32 smithi163 ceph-mon[36873]: pgmap v4317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7041.272700 secs 2024-07-26T08:53:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7059.015275 secs 2024-07-26T08:53:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:34 smithi042 ceph-mon[32775]: pgmap v4318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:34 smithi164 ceph-mon[36970]: pgmap v4318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:34 smithi163 ceph-mon[36873]: pgmap v4318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:36 smithi042 ceph-mon[32775]: pgmap v4319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:36 smithi164 ceph-mon[36970]: pgmap v4319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:36 smithi163 ceph-mon[36873]: pgmap v4319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7046.272779 secs 2024-07-26T08:53:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7064.015355 secs 2024-07-26T08:53:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7046.272779 secs 2024-07-26T08:53:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7064.015355 secs 2024-07-26T08:53:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7046.272779 secs 2024-07-26T08:53:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7064.015355 secs 2024-07-26T08:53:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:38 smithi042 ceph-mon[32775]: pgmap v4320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:38 smithi164 ceph-mon[36970]: pgmap v4320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:38 smithi163 ceph-mon[36873]: pgmap v4320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:40 smithi042 ceph-mon[32775]: pgmap v4321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:53:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:40 smithi164 ceph-mon[36970]: pgmap v4321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:53:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:40 smithi163 ceph-mon[36873]: pgmap v4321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:53:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:53:40.770+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:53:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:53:40.770+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:53:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:53:40.770+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:53:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:42 smithi042 ceph-mon[32775]: pgmap v4322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7051.272921 secs 2024-07-26T08:53:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7069.015453 secs 2024-07-26T08:53:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:42 smithi164 ceph-mon[36970]: pgmap v4322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7051.272921 secs 2024-07-26T08:53:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7069.015453 secs 2024-07-26T08:53:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:42 smithi163 ceph-mon[36873]: pgmap v4322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7051.272921 secs 2024-07-26T08:53:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7069.015453 secs 2024-07-26T08:53:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:44 smithi042 ceph-mon[32775]: pgmap v4323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:44 smithi164 ceph-mon[36970]: pgmap v4323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:44 smithi163 ceph-mon[36873]: pgmap v4323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:46 smithi042 ceph-mon[32775]: pgmap v4324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:46 smithi164 ceph-mon[36970]: pgmap v4324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:46 smithi163 ceph-mon[36873]: pgmap v4324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7056.273058 secs 2024-07-26T08:53:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7074.015551 secs 2024-07-26T08:53:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7056.273058 secs 2024-07-26T08:53:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7074.015551 secs 2024-07-26T08:53:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7056.273058 secs 2024-07-26T08:53:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7074.015551 secs 2024-07-26T08:53:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:48 smithi042 ceph-mon[32775]: pgmap v4325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:48 smithi164 ceph-mon[36970]: pgmap v4325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:48 smithi163 ceph-mon[36873]: pgmap v4325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:50 smithi042 ceph-mon[32775]: pgmap v4326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:53:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:50 smithi164 ceph-mon[36970]: pgmap v4326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:53:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:50 smithi163 ceph-mon[36873]: pgmap v4326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:53:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:52 smithi042 ceph-mon[32775]: pgmap v4327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7061.273183 secs 2024-07-26T08:53:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7079.015629 secs 2024-07-26T08:53:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:52 smithi164 ceph-mon[36970]: pgmap v4327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7061.273183 secs 2024-07-26T08:53:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7079.015629 secs 2024-07-26T08:53:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:52 smithi163 ceph-mon[36873]: pgmap v4327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7061.273183 secs 2024-07-26T08:53:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7079.015629 secs 2024-07-26T08:53:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:54 smithi042 ceph-mon[32775]: pgmap v4328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:54 smithi164 ceph-mon[36970]: pgmap v4328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:54 smithi163 ceph-mon[36873]: pgmap v4328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:53:55.774+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:53:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:53:55.774+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:53:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:53:55.774+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:53:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:56 smithi042 ceph-mon[32775]: pgmap v4329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:56 smithi164 ceph-mon[36970]: pgmap v4329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:56 smithi163 ceph-mon[36873]: pgmap v4329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:53:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7066.273330 secs 2024-07-26T08:53:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7084.015780 secs 2024-07-26T08:53:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7066.273330 secs 2024-07-26T08:53:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7084.015780 secs 2024-07-26T08:53:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7066.273330 secs 2024-07-26T08:53:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7084.015780 secs 2024-07-26T08:53:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:53:58 smithi042 ceph-mon[32775]: pgmap v4330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:53:58 smithi164 ceph-mon[36970]: pgmap v4330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:53:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:53:58 smithi163 ceph-mon[36873]: pgmap v4330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:00 smithi042 ceph-mon[32775]: pgmap v4331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:54:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:00 smithi164 ceph-mon[36970]: pgmap v4331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:54:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:00 smithi163 ceph-mon[36873]: pgmap v4331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:54:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:02 smithi042 ceph-mon[32775]: pgmap v4332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7071.273410 secs 2024-07-26T08:54:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7089.015847 secs 2024-07-26T08:54:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:02 smithi164 ceph-mon[36970]: pgmap v4332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7071.273410 secs 2024-07-26T08:54:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7089.015847 secs 2024-07-26T08:54:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:02 smithi163 ceph-mon[36873]: pgmap v4332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7071.273410 secs 2024-07-26T08:54:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7089.015847 secs 2024-07-26T08:54:04.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:04 smithi042 ceph-mon[32775]: pgmap v4333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:04 smithi164 ceph-mon[36970]: pgmap v4333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:04 smithi163 ceph-mon[36873]: pgmap v4333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:06 smithi042 ceph-mon[32775]: pgmap v4334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:06 smithi164 ceph-mon[36970]: pgmap v4334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:06 smithi163 ceph-mon[36873]: pgmap v4334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7076.273525 secs 2024-07-26T08:54:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7094.015966 secs 2024-07-26T08:54:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7076.273525 secs 2024-07-26T08:54:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7094.015966 secs 2024-07-26T08:54:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7076.273525 secs 2024-07-26T08:54:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7094.015966 secs 2024-07-26T08:54:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:08 smithi042 ceph-mon[32775]: pgmap v4335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:08 smithi164 ceph-mon[36970]: pgmap v4335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:08 smithi163 ceph-mon[36873]: pgmap v4335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:10 smithi042 ceph-mon[32775]: pgmap v4336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:54:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:10 smithi164 ceph-mon[36970]: pgmap v4336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:54:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:10 smithi163 ceph-mon[36873]: pgmap v4336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:54:10.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:54:10.782+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:54:10.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:54:10.782+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:54:10.783 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:54:10.782+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:54:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:12 smithi042 ceph-mon[32775]: pgmap v4337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7081.273655 secs 2024-07-26T08:54:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7099.016086 secs 2024-07-26T08:54:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:12 smithi164 ceph-mon[36970]: pgmap v4337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7081.273655 secs 2024-07-26T08:54:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7099.016086 secs 2024-07-26T08:54:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:12 smithi163 ceph-mon[36873]: pgmap v4337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7081.273655 secs 2024-07-26T08:54:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7099.016086 secs 2024-07-26T08:54:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:14 smithi042 ceph-mon[32775]: pgmap v4338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:14 smithi164 ceph-mon[36970]: pgmap v4338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:14 smithi163 ceph-mon[36873]: pgmap v4338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:54:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:54:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:54:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:54:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:54:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:54:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:54:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:54:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:54:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:54:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:54:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:54:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:16 smithi042 ceph-mon[32775]: pgmap v4339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:16 smithi042 ceph-mon[32775]: pgmap v4340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:54:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:16 smithi042 ceph-mon[32775]: pgmap v4341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:54:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:16 smithi164 ceph-mon[36970]: pgmap v4339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:16 smithi164 ceph-mon[36970]: pgmap v4340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:54:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:16 smithi164 ceph-mon[36970]: pgmap v4341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:54:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:16 smithi163 ceph-mon[36873]: pgmap v4339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:16 smithi163 ceph-mon[36873]: pgmap v4340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:54:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:16 smithi163 ceph-mon[36873]: pgmap v4341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:54:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7086.273802 secs 2024-07-26T08:54:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7104.016222 secs 2024-07-26T08:54:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7086.273802 secs 2024-07-26T08:54:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7104.016222 secs 2024-07-26T08:54:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7086.273802 secs 2024-07-26T08:54:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7104.016222 secs 2024-07-26T08:54:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:18 smithi042 ceph-mon[32775]: pgmap v4342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:54:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:18 smithi164 ceph-mon[36970]: pgmap v4342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:54:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:18 smithi163 ceph-mon[36873]: pgmap v4342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:54:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:20 smithi042 ceph-mon[32775]: pgmap v4343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:54:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:20 smithi164 ceph-mon[36970]: pgmap v4343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:54:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:20 smithi163 ceph-mon[36873]: pgmap v4343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:54:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7091.273865 secs 2024-07-26T08:54:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:22 smithi042 ceph-mon[32775]: pgmap v4344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:54:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7109.016336 secs 2024-07-26T08:54:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7091.273865 secs 2024-07-26T08:54:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:22 smithi164 ceph-mon[36970]: pgmap v4344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:54:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7109.016336 secs 2024-07-26T08:54:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7091.273865 secs 2024-07-26T08:54:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:22 smithi163 ceph-mon[36873]: pgmap v4344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:54:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7109.016336 secs 2024-07-26T08:54:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:24 smithi042 ceph-mon[32775]: pgmap v4345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:54:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:24 smithi164 ceph-mon[36970]: pgmap v4345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:54:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:24 smithi163 ceph-mon[36873]: pgmap v4345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:54:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:54:25.786+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:54:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:54:25.786+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:54:25.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:54:25.786+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:54:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:26 smithi042 ceph-mon[32775]: pgmap v4346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:54:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:26 smithi164 ceph-mon[36970]: pgmap v4346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:54:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:26 smithi163 ceph-mon[36873]: pgmap v4346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:54:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7096.273989 secs 2024-07-26T08:54:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7114.016464 secs 2024-07-26T08:54:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7096.273989 secs 2024-07-26T08:54:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7114.016464 secs 2024-07-26T08:54:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7096.273989 secs 2024-07-26T08:54:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7114.016464 secs 2024-07-26T08:54:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:28 smithi042 ceph-mon[32775]: pgmap v4347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:28 smithi164 ceph-mon[36970]: pgmap v4347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:28 smithi163 ceph-mon[36873]: pgmap v4347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:30 smithi042 ceph-mon[32775]: pgmap v4348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:30 smithi164 ceph-mon[36970]: pgmap v4348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:30 smithi163 ceph-mon[36873]: pgmap v4348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7101.274112 secs 2024-07-26T08:54:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:32 smithi042 ceph-mon[32775]: pgmap v4349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7119.016585 secs 2024-07-26T08:54:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7101.274112 secs 2024-07-26T08:54:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:32 smithi164 ceph-mon[36970]: pgmap v4349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7119.016585 secs 2024-07-26T08:54:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7101.274112 secs 2024-07-26T08:54:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:32 smithi163 ceph-mon[36873]: pgmap v4349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7119.016585 secs 2024-07-26T08:54:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:34 smithi042 ceph-mon[32775]: pgmap v4350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:34 smithi164 ceph-mon[36970]: pgmap v4350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:34 smithi163 ceph-mon[36873]: pgmap v4350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:36 smithi042 ceph-mon[32775]: pgmap v4351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:36 smithi164 ceph-mon[36970]: pgmap v4351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:36 smithi163 ceph-mon[36873]: pgmap v4351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7106.274245 secs 2024-07-26T08:54:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7124.016709 secs 2024-07-26T08:54:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7106.274245 secs 2024-07-26T08:54:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7124.016709 secs 2024-07-26T08:54:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7106.274245 secs 2024-07-26T08:54:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7124.016709 secs 2024-07-26T08:54:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:38 smithi164 ceph-mon[36970]: pgmap v4352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:38 smithi163 ceph-mon[36873]: pgmap v4352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:38.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:38 smithi042 ceph-mon[32775]: pgmap v4352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:39 smithi164 ceph-mon[36970]: pgmap v4353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:39 smithi163 ceph-mon[36873]: pgmap v4353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:39 smithi042 ceph-mon[32775]: pgmap v4353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:54:40.795+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:54:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:54:40.795+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:54:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:54:40.796+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:54:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7111.274402 secs 2024-07-26T08:54:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:42 smithi164 ceph-mon[36970]: pgmap v4354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7129.016811 secs 2024-07-26T08:54:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7111.274402 secs 2024-07-26T08:54:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:42 smithi163 ceph-mon[36873]: pgmap v4354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7129.016811 secs 2024-07-26T08:54:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7111.274402 secs 2024-07-26T08:54:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:42 smithi042 ceph-mon[32775]: pgmap v4354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7129.016811 secs 2024-07-26T08:54:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:44 smithi164 ceph-mon[36970]: pgmap v4355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:44 smithi163 ceph-mon[36873]: pgmap v4355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:44 smithi042 ceph-mon[32775]: pgmap v4355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:46 smithi164 ceph-mon[36970]: pgmap v4356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:46 smithi163 ceph-mon[36873]: pgmap v4356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:46 smithi042 ceph-mon[32775]: pgmap v4356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7116.274512 secs 2024-07-26T08:54:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7134.016952 secs 2024-07-26T08:54:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7116.274512 secs 2024-07-26T08:54:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7134.016952 secs 2024-07-26T08:54:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7116.274512 secs 2024-07-26T08:54:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7134.016952 secs 2024-07-26T08:54:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:48 smithi164 ceph-mon[36970]: pgmap v4357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:48 smithi163 ceph-mon[36873]: pgmap v4357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:48.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:48 smithi042 ceph-mon[32775]: pgmap v4357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:50 smithi042 ceph-mon[32775]: pgmap v4358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:50 smithi163 ceph-mon[36873]: pgmap v4358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:50 smithi164 ceph-mon[36970]: pgmap v4358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7121.274640 secs 2024-07-26T08:54:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:52 smithi163 ceph-mon[36873]: pgmap v4359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7139.017089 secs 2024-07-26T08:54:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7121.274640 secs 2024-07-26T08:54:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:52 smithi042 ceph-mon[32775]: pgmap v4359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7139.017089 secs 2024-07-26T08:54:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7121.274640 secs 2024-07-26T08:54:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:52 smithi164 ceph-mon[36970]: pgmap v4359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7139.017089 secs 2024-07-26T08:54:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:54 smithi163 ceph-mon[36873]: pgmap v4360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:54 smithi042 ceph-mon[32775]: pgmap v4360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:54 smithi164 ceph-mon[36970]: pgmap v4360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:54:55.799+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:54:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:54:55.799+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:54:55.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:54:55.799+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:54:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:56 smithi163 ceph-mon[36873]: pgmap v4361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:56 smithi042 ceph-mon[32775]: pgmap v4361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:56 smithi164 ceph-mon[36970]: pgmap v4361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:54:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7126.274760 secs 2024-07-26T08:54:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7144.017206 secs 2024-07-26T08:54:57.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7126.274760 secs 2024-07-26T08:54:57.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7144.017206 secs 2024-07-26T08:54:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7126.274760 secs 2024-07-26T08:54:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7144.017206 secs 2024-07-26T08:54:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:54:58 smithi163 ceph-mon[36873]: pgmap v4362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:54:58 smithi042 ceph-mon[32775]: pgmap v4362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:54:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:54:58 smithi164 ceph-mon[36970]: pgmap v4362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:00 smithi163 ceph-mon[36873]: pgmap v4363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:00 smithi042 ceph-mon[32775]: pgmap v4363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:00 smithi164 ceph-mon[36970]: pgmap v4363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7131.274863 secs 2024-07-26T08:55:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:02 smithi163 ceph-mon[36873]: pgmap v4364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7149.017336 secs 2024-07-26T08:55:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7131.274863 secs 2024-07-26T08:55:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:02 smithi042 ceph-mon[32775]: pgmap v4364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7149.017336 secs 2024-07-26T08:55:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7131.274863 secs 2024-07-26T08:55:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:02 smithi164 ceph-mon[36970]: pgmap v4364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7149.017336 secs 2024-07-26T08:55:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:04 smithi163 ceph-mon[36873]: pgmap v4365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:04.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:04 smithi042 ceph-mon[32775]: pgmap v4365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:04 smithi164 ceph-mon[36970]: pgmap v4365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:06 smithi163 ceph-mon[36873]: pgmap v4366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:06 smithi042 ceph-mon[32775]: pgmap v4366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:06 smithi164 ceph-mon[36970]: pgmap v4366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7136.275030 secs 2024-07-26T08:55:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7154.017400 secs 2024-07-26T08:55:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7136.275030 secs 2024-07-26T08:55:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7154.017400 secs 2024-07-26T08:55:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7136.275030 secs 2024-07-26T08:55:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7154.017400 secs 2024-07-26T08:55:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:08 smithi042 ceph-mon[32775]: pgmap v4367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:08 smithi164 ceph-mon[36970]: pgmap v4367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:08 smithi163 ceph-mon[36873]: pgmap v4367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:10 smithi042 ceph-mon[32775]: pgmap v4368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:10 smithi164 ceph-mon[36970]: pgmap v4368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:10 smithi163 ceph-mon[36873]: pgmap v4368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:10.810 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:10.808+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:55:10.810 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:10.808+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:55:10.811 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:10.808+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:55:12.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7141.275150 secs 2024-07-26T08:55:12.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7159.017523 secs 2024-07-26T08:55:12.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:12 smithi164 ceph-mon[36970]: pgmap v4369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7141.275150 secs 2024-07-26T08:55:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7159.017523 secs 2024-07-26T08:55:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:12 smithi042 ceph-mon[32775]: pgmap v4369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7141.275150 secs 2024-07-26T08:55:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7159.017523 secs 2024-07-26T08:55:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:12 smithi163 ceph-mon[36873]: pgmap v4369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:14 smithi042 ceph-mon[32775]: pgmap v4370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:14 smithi164 ceph-mon[36970]: pgmap v4370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:14 smithi163 ceph-mon[36873]: pgmap v4370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:55:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:55:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:55:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:16 smithi042 ceph-mon[32775]: pgmap v4371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:55:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:55:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:55:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:16 smithi164 ceph-mon[36970]: pgmap v4371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:55:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:55:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:55:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:16 smithi163 ceph-mon[36873]: pgmap v4371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:55:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:55:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:55:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:17 smithi042 ceph-mon[32775]: pgmap v4372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:55:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:17 smithi042 ceph-mon[32775]: pgmap v4373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:55:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7146.275299 secs 2024-07-26T08:55:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7164.017617 secs 2024-07-26T08:55:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:17 smithi164 ceph-mon[36970]: pgmap v4372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:55:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:17 smithi164 ceph-mon[36970]: pgmap v4373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:55:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7146.275299 secs 2024-07-26T08:55:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7164.017617 secs 2024-07-26T08:55:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:17 smithi163 ceph-mon[36873]: pgmap v4372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:55:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:17 smithi163 ceph-mon[36873]: pgmap v4373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:55:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7146.275299 secs 2024-07-26T08:55:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7164.017617 secs 2024-07-26T08:55:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:19 smithi042 ceph-mon[32775]: pgmap v4374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:55:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:19 smithi164 ceph-mon[36970]: pgmap v4374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:55:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:19 smithi163 ceph-mon[36873]: pgmap v4374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:55:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:21 smithi042 ceph-mon[32775]: pgmap v4375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:55:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:21 smithi164 ceph-mon[36970]: pgmap v4375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:55:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:21 smithi163 ceph-mon[36873]: pgmap v4375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:55:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7151.275382 secs 2024-07-26T08:55:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7169.017745 secs 2024-07-26T08:55:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7151.275382 secs 2024-07-26T08:55:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7169.017745 secs 2024-07-26T08:55:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7151.275382 secs 2024-07-26T08:55:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7169.017745 secs 2024-07-26T08:55:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:23 smithi042 ceph-mon[32775]: pgmap v4376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:55:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:23 smithi164 ceph-mon[36970]: pgmap v4376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:55:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:23 smithi163 ceph-mon[36873]: pgmap v4376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:55:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:25 smithi042 ceph-mon[32775]: pgmap v4377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:55:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:25 smithi164 ceph-mon[36970]: pgmap v4377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:55:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:25 smithi163 ceph-mon[36873]: pgmap v4377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:55:25.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:25.812+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:55:25.814 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:25.812+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:55:25.814 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:25.812+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:55:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:27 smithi042 ceph-mon[32775]: pgmap v4378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:55:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7156.275530 secs 2024-07-26T08:55:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7174.017895 secs 2024-07-26T08:55:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:27 smithi164 ceph-mon[36970]: pgmap v4378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:55:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7156.275530 secs 2024-07-26T08:55:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7174.017895 secs 2024-07-26T08:55:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:27 smithi163 ceph-mon[36873]: pgmap v4378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:55:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7156.275530 secs 2024-07-26T08:55:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7174.017895 secs 2024-07-26T08:55:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:29 smithi042 ceph-mon[32775]: pgmap v4379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:29 smithi164 ceph-mon[36970]: pgmap v4379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:29 smithi163 ceph-mon[36873]: pgmap v4379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:31 smithi042 ceph-mon[32775]: pgmap v4380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:31 smithi164 ceph-mon[36970]: pgmap v4380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:31 smithi163 ceph-mon[36873]: pgmap v4380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7161.275655 secs 2024-07-26T08:55:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7179.018017 secs 2024-07-26T08:55:32.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7161.275655 secs 2024-07-26T08:55:32.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7179.018017 secs 2024-07-26T08:55:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7161.275655 secs 2024-07-26T08:55:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7179.018017 secs 2024-07-26T08:55:33.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:33 smithi042 ceph-mon[32775]: pgmap v4381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:33 smithi164 ceph-mon[36970]: pgmap v4381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:33 smithi163 ceph-mon[36873]: pgmap v4381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:35 smithi042 ceph-mon[32775]: pgmap v4382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:55:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:35 smithi164 ceph-mon[36970]: pgmap v4382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:55:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:35 smithi163 ceph-mon[36873]: pgmap v4382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:55:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:37 smithi042 ceph-mon[32775]: pgmap v4383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7166.275808 secs 2024-07-26T08:55:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7184.018140 secs 2024-07-26T08:55:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:37 smithi164 ceph-mon[36970]: pgmap v4383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7166.275808 secs 2024-07-26T08:55:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7184.018140 secs 2024-07-26T08:55:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:37 smithi163 ceph-mon[36873]: pgmap v4383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7166.275808 secs 2024-07-26T08:55:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7184.018140 secs 2024-07-26T08:55:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:39 smithi042 ceph-mon[32775]: pgmap v4384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:39 smithi164 ceph-mon[36970]: pgmap v4384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:39 smithi163 ceph-mon[36873]: pgmap v4384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:40.816+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:55:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:40.816+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:55:40.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:40.816+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:55:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:41 smithi042 ceph-mon[32775]: pgmap v4385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:41 smithi164 ceph-mon[36970]: pgmap v4385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:41 smithi163 ceph-mon[36873]: pgmap v4385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:42.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7171.275946 secs 2024-07-26T08:55:42.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7189.018213 secs 2024-07-26T08:55:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7171.275946 secs 2024-07-26T08:55:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7189.018213 secs 2024-07-26T08:55:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7171.275946 secs 2024-07-26T08:55:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7189.018213 secs 2024-07-26T08:55:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:43 smithi042 ceph-mon[32775]: pgmap v4386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:43 smithi164 ceph-mon[36970]: pgmap v4386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:43 smithi163 ceph-mon[36873]: pgmap v4386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:45 smithi042 ceph-mon[32775]: pgmap v4387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:55:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:45 smithi164 ceph-mon[36970]: pgmap v4387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:55:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:45 smithi163 ceph-mon[36873]: pgmap v4387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:55:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:47 smithi042 ceph-mon[32775]: pgmap v4388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7176.276049 secs 2024-07-26T08:55:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7194.018329 secs 2024-07-26T08:55:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:47 smithi164 ceph-mon[36970]: pgmap v4388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7176.276049 secs 2024-07-26T08:55:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7194.018329 secs 2024-07-26T08:55:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:47 smithi163 ceph-mon[36873]: pgmap v4388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7176.276049 secs 2024-07-26T08:55:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7194.018329 secs 2024-07-26T08:55:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:49 smithi042 ceph-mon[32775]: pgmap v4389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:49 smithi164 ceph-mon[36970]: pgmap v4389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:49 smithi163 ceph-mon[36873]: pgmap v4389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:51 smithi042 ceph-mon[32775]: pgmap v4390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:51 smithi164 ceph-mon[36970]: pgmap v4390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:51 smithi163 ceph-mon[36873]: pgmap v4390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7181.276178 secs 2024-07-26T08:55:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7199.018476 secs 2024-07-26T08:55:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7181.276178 secs 2024-07-26T08:55:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7199.018476 secs 2024-07-26T08:55:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7181.276178 secs 2024-07-26T08:55:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7199.018476 secs 2024-07-26T08:55:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:53 smithi042 ceph-mon[32775]: pgmap v4391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:53 smithi164 ceph-mon[36970]: pgmap v4391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:53 smithi163 ceph-mon[36873]: pgmap v4391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:55 smithi042 ceph-mon[32775]: pgmap v4392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:55:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:55 smithi164 ceph-mon[36970]: pgmap v4392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:55:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:55 smithi163 ceph-mon[36873]: pgmap v4392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:55:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:55.818+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:55:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:55.818+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:55:55.820 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:55.818+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:55:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:57 smithi042 ceph-mon[32775]: pgmap v4393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7186.276302 secs 2024-07-26T08:55:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7204.018614 secs 2024-07-26T08:55:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:57 smithi164 ceph-mon[36970]: pgmap v4393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7186.276302 secs 2024-07-26T08:55:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7204.018614 secs 2024-07-26T08:55:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:57 smithi163 ceph-mon[36873]: pgmap v4393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:55:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7186.276302 secs 2024-07-26T08:55:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7204.018614 secs 2024-07-26T08:55:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:57.872+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T08:55:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:57.872+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T08:55:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:57.872+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff678031980 tx=0x7ff67803e040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T08:55:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:57.872+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff678031980 tx=0x7ff67803e040 comp rx=0 tx=0).stop 2024-07-26T08:55:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:57.872+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T08:55:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:57.872+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T08:55:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:57.872+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T08:55:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:57.873+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T08:55:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:57.873+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T08:55:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:57.873+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b4840 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff678031570 tx=0x7ff67804b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T08:55:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:58.971+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:55:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:58.971+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:55:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:58.971+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:55:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:55:58.971+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:55:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:55:59 smithi042 ceph-mon[32775]: pgmap v4394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:55:59 smithi164 ceph-mon[36970]: pgmap v4394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:55:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:55:59 smithi163 ceph-mon[36873]: pgmap v4394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:01 smithi042 ceph-mon[32775]: pgmap v4395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:01 smithi164 ceph-mon[36970]: pgmap v4395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:01 smithi163 ceph-mon[36873]: pgmap v4395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7191.276427 secs 2024-07-26T08:56:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7209.018741 secs 2024-07-26T08:56:02.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7191.276427 secs 2024-07-26T08:56:02.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7209.018741 secs 2024-07-26T08:56:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7191.276427 secs 2024-07-26T08:56:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7209.018741 secs 2024-07-26T08:56:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:03 smithi042 ceph-mon[32775]: pgmap v4396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:03 smithi164 ceph-mon[36970]: pgmap v4396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:03 smithi163 ceph-mon[36873]: pgmap v4396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:05 smithi042 ceph-mon[32775]: pgmap v4397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:56:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:05 smithi164 ceph-mon[36970]: pgmap v4397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:56:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:05 smithi163 ceph-mon[36873]: pgmap v4397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:56:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:07 smithi042 ceph-mon[32775]: pgmap v4398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7196.276557 secs 2024-07-26T08:56:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7214.018865 secs 2024-07-26T08:56:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:07 smithi164 ceph-mon[36970]: pgmap v4398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7196.276557 secs 2024-07-26T08:56:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7214.018865 secs 2024-07-26T08:56:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:07 smithi163 ceph-mon[36873]: pgmap v4398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7196.276557 secs 2024-07-26T08:56:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7214.018865 secs 2024-07-26T08:56:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:09 smithi042 ceph-mon[32775]: pgmap v4399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:09 smithi164 ceph-mon[36970]: pgmap v4399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:09 smithi163 ceph-mon[36873]: pgmap v4399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:10.829 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:56:10.828+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:56:10.829 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:56:10.828+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:56:10.829 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:56:10.828+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:56:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:11 smithi042 ceph-mon[32775]: pgmap v4400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:11 smithi164 ceph-mon[36970]: pgmap v4400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:11 smithi163 ceph-mon[36873]: pgmap v4400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7201.276682 secs 2024-07-26T08:56:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7219.018988 secs 2024-07-26T08:56:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7201.276682 secs 2024-07-26T08:56:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7219.018988 secs 2024-07-26T08:56:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7201.276682 secs 2024-07-26T08:56:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7219.018988 secs 2024-07-26T08:56:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:13 smithi164 ceph-mon[36970]: pgmap v4401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:13 smithi163 ceph-mon[36873]: pgmap v4401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:13 smithi042 ceph-mon[32775]: pgmap v4401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:15 smithi164 ceph-mon[36970]: pgmap v4402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:56:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:15 smithi163 ceph-mon[36873]: pgmap v4402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:56:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:15 smithi042 ceph-mon[32775]: pgmap v4402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:56:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:56:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:56:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:56:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:56:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:56:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:56:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:56:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:56:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:56:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:56:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:56:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:56:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:17 smithi164 ceph-mon[36970]: pgmap v4403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7206.276751 secs 2024-07-26T08:56:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7224.019066 secs 2024-07-26T08:56:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:17 smithi164 ceph-mon[36970]: pgmap v4404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:56:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:17 smithi164 ceph-mon[36970]: pgmap v4405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:56:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:17 smithi163 ceph-mon[36873]: pgmap v4403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7206.276751 secs 2024-07-26T08:56:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7224.019066 secs 2024-07-26T08:56:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:17 smithi163 ceph-mon[36873]: pgmap v4404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:56:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:17 smithi163 ceph-mon[36873]: pgmap v4405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:56:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:17 smithi042 ceph-mon[32775]: pgmap v4403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7206.276751 secs 2024-07-26T08:56:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7224.019066 secs 2024-07-26T08:56:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:17 smithi042 ceph-mon[32775]: pgmap v4404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:56:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:17 smithi042 ceph-mon[32775]: pgmap v4405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:56:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:18 smithi164 ceph-mon[36970]: pgmap v4406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:56:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:18 smithi163 ceph-mon[36873]: pgmap v4406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:56:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:18 smithi042 ceph-mon[32775]: pgmap v4406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:56:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:21 smithi042 ceph-mon[32775]: pgmap v4407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:56:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:21 smithi164 ceph-mon[36970]: pgmap v4407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:56:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:21 smithi163 ceph-mon[36873]: pgmap v4407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:56:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7211.276879 secs 2024-07-26T08:56:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7229.019201 secs 2024-07-26T08:56:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7211.276879 secs 2024-07-26T08:56:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7229.019201 secs 2024-07-26T08:56:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7211.276879 secs 2024-07-26T08:56:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7229.019201 secs 2024-07-26T08:56:23.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:23 smithi042 ceph-mon[32775]: pgmap v4408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:56:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:23 smithi164 ceph-mon[36970]: pgmap v4408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:56:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:23 smithi163 ceph-mon[36873]: pgmap v4408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:56:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:25 smithi042 ceph-mon[32775]: pgmap v4409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:56:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:25 smithi164 ceph-mon[36970]: pgmap v4409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:56:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:25 smithi163 ceph-mon[36873]: pgmap v4409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:56:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:56:25.832+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:56:25.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:56:25.832+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:56:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:56:25.832+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:56:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7216.277003 secs 2024-07-26T08:56:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7234.019320 secs 2024-07-26T08:56:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:27 smithi042 ceph-mon[32775]: pgmap v4410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:56:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7216.277003 secs 2024-07-26T08:56:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7234.019320 secs 2024-07-26T08:56:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:27 smithi164 ceph-mon[36970]: pgmap v4410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:56:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7216.277003 secs 2024-07-26T08:56:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7234.019320 secs 2024-07-26T08:56:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:27 smithi163 ceph-mon[36873]: pgmap v4410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:56:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:29 smithi042 ceph-mon[32775]: pgmap v4411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:29.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:29 smithi164 ceph-mon[36970]: pgmap v4411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:29 smithi163 ceph-mon[36873]: pgmap v4411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:31 smithi042 ceph-mon[32775]: pgmap v4412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:31 smithi164 ceph-mon[36970]: pgmap v4412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:31 smithi163 ceph-mon[36873]: pgmap v4412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7221.277164 secs 2024-07-26T08:56:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7239.019437 secs 2024-07-26T08:56:32.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7221.277164 secs 2024-07-26T08:56:32.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7239.019437 secs 2024-07-26T08:56:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7221.277164 secs 2024-07-26T08:56:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7239.019437 secs 2024-07-26T08:56:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:33 smithi042 ceph-mon[32775]: pgmap v4413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:33 smithi164 ceph-mon[36970]: pgmap v4413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:33 smithi163 ceph-mon[36873]: pgmap v4413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:35.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:35 smithi042 ceph-mon[32775]: pgmap v4414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:35 smithi164 ceph-mon[36970]: pgmap v4414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:35 smithi163 ceph-mon[36873]: pgmap v4414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7226.277284 secs 2024-07-26T08:56:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7244.019562 secs 2024-07-26T08:56:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:37 smithi042 ceph-mon[32775]: pgmap v4415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7226.277284 secs 2024-07-26T08:56:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7244.019562 secs 2024-07-26T08:56:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:37 smithi164 ceph-mon[36970]: pgmap v4415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7226.277284 secs 2024-07-26T08:56:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7244.019562 secs 2024-07-26T08:56:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:37 smithi163 ceph-mon[36873]: pgmap v4415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:39 smithi042 ceph-mon[32775]: pgmap v4416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:39 smithi164 ceph-mon[36970]: pgmap v4416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:39 smithi163 ceph-mon[36873]: pgmap v4416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:40.843 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:56:40.842+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:56:40.843 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:56:40.842+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:56:40.843 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:56:40.842+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:56:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:41 smithi042 ceph-mon[32775]: pgmap v4417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:41 smithi164 ceph-mon[36970]: pgmap v4417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:41 smithi163 ceph-mon[36873]: pgmap v4417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:42.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7231.277410 secs 2024-07-26T08:56:42.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7249.019703 secs 2024-07-26T08:56:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7231.277410 secs 2024-07-26T08:56:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7249.019703 secs 2024-07-26T08:56:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7231.277410 secs 2024-07-26T08:56:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7249.019703 secs 2024-07-26T08:56:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:43 smithi042 ceph-mon[32775]: pgmap v4418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:43 smithi164 ceph-mon[36970]: pgmap v4418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:43 smithi163 ceph-mon[36873]: pgmap v4418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:45 smithi042 ceph-mon[32775]: pgmap v4419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:45 smithi164 ceph-mon[36970]: pgmap v4419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:45 smithi163 ceph-mon[36873]: pgmap v4419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7236.277536 secs 2024-07-26T08:56:47.650 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7254.019826 secs 2024-07-26T08:56:47.650 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:47 smithi042 ceph-mon[32775]: pgmap v4420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7236.277536 secs 2024-07-26T08:56:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7254.019826 secs 2024-07-26T08:56:47.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:47 smithi164 ceph-mon[36970]: pgmap v4420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7236.277536 secs 2024-07-26T08:56:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7254.019826 secs 2024-07-26T08:56:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:47 smithi163 ceph-mon[36873]: pgmap v4420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:49 smithi042 ceph-mon[32775]: pgmap v4421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:49 smithi164 ceph-mon[36970]: pgmap v4421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:49 smithi163 ceph-mon[36873]: pgmap v4421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:51 smithi042 ceph-mon[32775]: pgmap v4422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:51 smithi164 ceph-mon[36970]: pgmap v4422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:51 smithi163 ceph-mon[36873]: pgmap v4422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7241.277660 secs 2024-07-26T08:56:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7259.019981 secs 2024-07-26T08:56:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7241.277660 secs 2024-07-26T08:56:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7259.019981 secs 2024-07-26T08:56:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7241.277660 secs 2024-07-26T08:56:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7259.019981 secs 2024-07-26T08:56:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:53 smithi042 ceph-mon[32775]: pgmap v4423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:53 smithi164 ceph-mon[36970]: pgmap v4423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:53 smithi163 ceph-mon[36873]: pgmap v4423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:55 smithi042 ceph-mon[32775]: pgmap v4424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:55 smithi164 ceph-mon[36970]: pgmap v4424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:55 smithi163 ceph-mon[36873]: pgmap v4424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:56:55.847 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:56:55.846+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:56:55.848 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:56:55.846+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:56:55.848 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:56:55.846+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:56:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7246.277757 secs 2024-07-26T08:56:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7264.020079 secs 2024-07-26T08:56:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:57 smithi042 ceph-mon[32775]: pgmap v4425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7246.277757 secs 2024-07-26T08:56:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7264.020079 secs 2024-07-26T08:56:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:57 smithi164 ceph-mon[36970]: pgmap v4425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7246.277757 secs 2024-07-26T08:56:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7264.020079 secs 2024-07-26T08:56:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:57 smithi163 ceph-mon[36873]: pgmap v4425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:56:59 smithi042 ceph-mon[32775]: pgmap v4426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:59.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:56:59 smithi164 ceph-mon[36970]: pgmap v4426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:56:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:56:59 smithi163 ceph-mon[36873]: pgmap v4426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:01 smithi042 ceph-mon[32775]: pgmap v4427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:01 smithi164 ceph-mon[36970]: pgmap v4427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:01 smithi163 ceph-mon[36873]: pgmap v4427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:02.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7251.277899 secs 2024-07-26T08:57:02.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7269.020202 secs 2024-07-26T08:57:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7251.277899 secs 2024-07-26T08:57:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7269.020202 secs 2024-07-26T08:57:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7251.277899 secs 2024-07-26T08:57:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7269.020202 secs 2024-07-26T08:57:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:03 smithi164 ceph-mon[36970]: pgmap v4428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:03 smithi163 ceph-mon[36873]: pgmap v4428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:03.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:03 smithi042 ceph-mon[32775]: pgmap v4428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:05 smithi164 ceph-mon[36970]: pgmap v4429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:05 smithi163 ceph-mon[36873]: pgmap v4429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:05 smithi042 ceph-mon[32775]: pgmap v4429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7256.278026 secs 2024-07-26T08:57:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7274.020320 secs 2024-07-26T08:57:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:07 smithi164 ceph-mon[36970]: pgmap v4430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7256.278026 secs 2024-07-26T08:57:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7274.020320 secs 2024-07-26T08:57:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:07 smithi163 ceph-mon[36873]: pgmap v4430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7256.278026 secs 2024-07-26T08:57:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7274.020320 secs 2024-07-26T08:57:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:07 smithi042 ceph-mon[32775]: pgmap v4430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:09 smithi164 ceph-mon[36970]: pgmap v4431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:09 smithi163 ceph-mon[36873]: pgmap v4431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:09.872 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:09 smithi042 ceph-mon[32775]: pgmap v4431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:10.857 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:57:10.855+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:57:10.857 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:57:10.855+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:57:10.857 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:57:10.855+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:57:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:11 smithi164 ceph-mon[36970]: pgmap v4432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:11 smithi163 ceph-mon[36873]: pgmap v4432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:11 smithi042 ceph-mon[32775]: pgmap v4432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7261.278109 secs 2024-07-26T08:57:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7279.020446 secs 2024-07-26T08:57:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7261.278109 secs 2024-07-26T08:57:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7279.020446 secs 2024-07-26T08:57:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7261.278109 secs 2024-07-26T08:57:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7279.020446 secs 2024-07-26T08:57:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:13 smithi164 ceph-mon[36970]: pgmap v4433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:13 smithi163 ceph-mon[36873]: pgmap v4433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:13 smithi042 ceph-mon[32775]: pgmap v4433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:14 smithi163 ceph-mon[36873]: pgmap v4434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:14 smithi042 ceph-mon[32775]: pgmap v4434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:14 smithi164 ceph-mon[36970]: pgmap v4434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7266.278253 secs 2024-07-26T08:57:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7284.020551 secs 2024-07-26T08:57:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:17 smithi042 ceph-mon[32775]: pgmap v4435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:57:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:57:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:57:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:57:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7266.278253 secs 2024-07-26T08:57:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7284.020551 secs 2024-07-26T08:57:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:17 smithi164 ceph-mon[36970]: pgmap v4435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:57:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:57:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:57:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:57:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7266.278253 secs 2024-07-26T08:57:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7284.020551 secs 2024-07-26T08:57:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:17 smithi163 ceph-mon[36873]: pgmap v4435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:57:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:57:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:57:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:57:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:18 smithi042 ceph-mon[32775]: pgmap v4436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:57:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:18 smithi042 ceph-mon[32775]: pgmap v4437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:57:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:18 smithi164 ceph-mon[36970]: pgmap v4436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:57:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:18 smithi164 ceph-mon[36970]: pgmap v4437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:57:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:18 smithi163 ceph-mon[36873]: pgmap v4436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:57:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:18 smithi163 ceph-mon[36873]: pgmap v4437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:57:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:20 smithi042 ceph-mon[32775]: pgmap v4438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:57:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:20 smithi164 ceph-mon[36970]: pgmap v4438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:57:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:20 smithi163 ceph-mon[36873]: pgmap v4438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:57:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:22 smithi042 ceph-mon[32775]: pgmap v4439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:57:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7271.278372 secs 2024-07-26T08:57:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7289.020678 secs 2024-07-26T08:57:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:22 smithi164 ceph-mon[36970]: pgmap v4439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:57:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7271.278372 secs 2024-07-26T08:57:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7289.020678 secs 2024-07-26T08:57:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:22 smithi163 ceph-mon[36873]: pgmap v4439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:57:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7271.278372 secs 2024-07-26T08:57:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7289.020678 secs 2024-07-26T08:57:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:24 smithi042 ceph-mon[32775]: pgmap v4440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:57:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:24 smithi164 ceph-mon[36970]: pgmap v4440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:57:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:24 smithi163 ceph-mon[36873]: pgmap v4440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:57:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:57:25.859+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:57:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:57:25.859+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:57:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:57:25.859+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:57:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:26 smithi042 ceph-mon[32775]: pgmap v4441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:57:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:26 smithi164 ceph-mon[36970]: pgmap v4441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:57:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:26 smithi163 ceph-mon[36873]: pgmap v4441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:57:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7276.278492 secs 2024-07-26T08:57:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7294.020796 secs 2024-07-26T08:57:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7276.278492 secs 2024-07-26T08:57:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7294.020796 secs 2024-07-26T08:57:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7276.278492 secs 2024-07-26T08:57:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7294.020796 secs 2024-07-26T08:57:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:28 smithi042 ceph-mon[32775]: pgmap v4442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:57:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:28 smithi164 ceph-mon[36970]: pgmap v4442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:57:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:28 smithi163 ceph-mon[36873]: pgmap v4442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:57:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:30 smithi042 ceph-mon[32775]: pgmap v4443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:57:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:30 smithi164 ceph-mon[36970]: pgmap v4443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:57:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:30 smithi163 ceph-mon[36873]: pgmap v4443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:57:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:32 smithi042 ceph-mon[32775]: pgmap v4444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7281.278576 secs 2024-07-26T08:57:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7299.020894 secs 2024-07-26T08:57:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:32 smithi164 ceph-mon[36970]: pgmap v4444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7281.278576 secs 2024-07-26T08:57:32.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7299.020894 secs 2024-07-26T08:57:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:32 smithi163 ceph-mon[36873]: pgmap v4444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7281.278576 secs 2024-07-26T08:57:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7299.020894 secs 2024-07-26T08:57:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:34 smithi042 ceph-mon[32775]: pgmap v4445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:34 smithi164 ceph-mon[36970]: pgmap v4445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:34 smithi163 ceph-mon[36873]: pgmap v4445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:36 smithi042 ceph-mon[32775]: pgmap v4446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:36 smithi164 ceph-mon[36970]: pgmap v4446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:36 smithi163 ceph-mon[36873]: pgmap v4446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7286.278675 secs 2024-07-26T08:57:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7304.021016 secs 2024-07-26T08:57:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7286.278675 secs 2024-07-26T08:57:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7304.021016 secs 2024-07-26T08:57:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7286.278675 secs 2024-07-26T08:57:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7304.021016 secs 2024-07-26T08:57:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:38 smithi042 ceph-mon[32775]: pgmap v4447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:38 smithi164 ceph-mon[36970]: pgmap v4447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:38 smithi163 ceph-mon[36873]: pgmap v4447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:40 smithi042 ceph-mon[32775]: pgmap v4448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:57:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:40 smithi164 ceph-mon[36970]: pgmap v4448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:57:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:40 smithi163 ceph-mon[36873]: pgmap v4448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:57:40.870 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:57:40.869+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:57:40.870 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:57:40.869+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:57:40.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:57:40.869+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:57:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:42 smithi042 ceph-mon[32775]: pgmap v4449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7291.278819 secs 2024-07-26T08:57:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7309.021137 secs 2024-07-26T08:57:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:42 smithi164 ceph-mon[36970]: pgmap v4449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7291.278819 secs 2024-07-26T08:57:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7309.021137 secs 2024-07-26T08:57:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:42 smithi163 ceph-mon[36873]: pgmap v4449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7291.278819 secs 2024-07-26T08:57:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7309.021137 secs 2024-07-26T08:57:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:44 smithi042 ceph-mon[32775]: pgmap v4450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:44 smithi164 ceph-mon[36970]: pgmap v4450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:44 smithi163 ceph-mon[36873]: pgmap v4450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:46 smithi042 ceph-mon[32775]: pgmap v4451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:46 smithi164 ceph-mon[36970]: pgmap v4451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:46 smithi163 ceph-mon[36873]: pgmap v4451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7296.278939 secs 2024-07-26T08:57:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7314.021263 secs 2024-07-26T08:57:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7296.278939 secs 2024-07-26T08:57:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7314.021263 secs 2024-07-26T08:57:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7296.278939 secs 2024-07-26T08:57:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7314.021263 secs 2024-07-26T08:57:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:48 smithi164 ceph-mon[36970]: pgmap v4452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:48 smithi163 ceph-mon[36873]: pgmap v4452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:48 smithi042 ceph-mon[32775]: pgmap v4452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:50 smithi164 ceph-mon[36970]: pgmap v4453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:57:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:50 smithi163 ceph-mon[36873]: pgmap v4453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:57:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:50 smithi042 ceph-mon[32775]: pgmap v4453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:57:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:52 smithi164 ceph-mon[36970]: pgmap v4454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7301.279055 secs 2024-07-26T08:57:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7319.021345 secs 2024-07-26T08:57:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:52 smithi163 ceph-mon[36873]: pgmap v4454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7301.279055 secs 2024-07-26T08:57:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7319.021345 secs 2024-07-26T08:57:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:52 smithi042 ceph-mon[32775]: pgmap v4454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7301.279055 secs 2024-07-26T08:57:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7319.021345 secs 2024-07-26T08:57:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:54 smithi164 ceph-mon[36970]: pgmap v4455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:54 smithi163 ceph-mon[36873]: pgmap v4455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:54 smithi042 ceph-mon[32775]: pgmap v4455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:57:55.873+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:57:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:57:55.873+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:57:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:57:55.873+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:57:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:56 smithi164 ceph-mon[36970]: pgmap v4456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:56 smithi163 ceph-mon[36873]: pgmap v4456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:56 smithi042 ceph-mon[32775]: pgmap v4456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:57:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7306.279178 secs 2024-07-26T08:57:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7324.021484 secs 2024-07-26T08:57:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7306.279178 secs 2024-07-26T08:57:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7324.021484 secs 2024-07-26T08:57:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7306.279178 secs 2024-07-26T08:57:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7324.021484 secs 2024-07-26T08:57:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:57:58 smithi164 ceph-mon[36970]: pgmap v4457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:57:58 smithi163 ceph-mon[36873]: pgmap v4457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:57:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:57:58 smithi042 ceph-mon[32775]: pgmap v4457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:00 smithi164 ceph-mon[36970]: pgmap v4458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:58:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:00 smithi163 ceph-mon[36873]: pgmap v4458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:58:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:00 smithi042 ceph-mon[32775]: pgmap v4458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:58:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7311.279308 secs 2024-07-26T08:58:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:02 smithi164 ceph-mon[36970]: pgmap v4459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7329.021548 secs 2024-07-26T08:58:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7311.279308 secs 2024-07-26T08:58:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:02 smithi163 ceph-mon[36873]: pgmap v4459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7329.021548 secs 2024-07-26T08:58:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7311.279308 secs 2024-07-26T08:58:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:02 smithi042 ceph-mon[32775]: pgmap v4459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7329.021548 secs 2024-07-26T08:58:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:04 smithi164 ceph-mon[36970]: pgmap v4460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:04 smithi163 ceph-mon[36873]: pgmap v4460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:04.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:04 smithi042 ceph-mon[32775]: pgmap v4460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:06 smithi164 ceph-mon[36970]: pgmap v4461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:06 smithi163 ceph-mon[36873]: pgmap v4461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:06 smithi042 ceph-mon[32775]: pgmap v4461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7316.279433 secs 2024-07-26T08:58:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7334.021691 secs 2024-07-26T08:58:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7316.279433 secs 2024-07-26T08:58:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7334.021691 secs 2024-07-26T08:58:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7316.279433 secs 2024-07-26T08:58:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7334.021691 secs 2024-07-26T08:58:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:08 smithi164 ceph-mon[36970]: pgmap v4462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:08 smithi163 ceph-mon[36873]: pgmap v4462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:08.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:08 smithi042 ceph-mon[32775]: pgmap v4462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:09 smithi164 ceph-mon[36970]: pgmap v4463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:58:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:09 smithi163 ceph-mon[36873]: pgmap v4463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:58:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:09 smithi042 ceph-mon[32775]: pgmap v4463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:58:10.883 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:58:10.882+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:58:10.884 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:58:10.882+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:58:10.884 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:58:10.882+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:58:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7321.279553 secs 2024-07-26T08:58:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:11 smithi042 ceph-mon[32775]: pgmap v4464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7321.279553 secs 2024-07-26T08:58:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:11 smithi164 ceph-mon[36970]: pgmap v4464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7321.279553 secs 2024-07-26T08:58:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:11 smithi163 ceph-mon[36873]: pgmap v4464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7339.021792 secs 2024-07-26T08:58:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7339.021792 secs 2024-07-26T08:58:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7339.021792 secs 2024-07-26T08:58:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:13 smithi042 ceph-mon[32775]: pgmap v4465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:13 smithi164 ceph-mon[36970]: pgmap v4465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:13 smithi163 ceph-mon[36873]: pgmap v4465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:15 smithi042 ceph-mon[32775]: pgmap v4466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:15 smithi164 ceph-mon[36970]: pgmap v4466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:15 smithi163 ceph-mon[36873]: pgmap v4466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7326.279677 secs 2024-07-26T08:58:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7326.279677 secs 2024-07-26T08:58:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7326.279677 secs 2024-07-26T08:58:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7344.021912 secs 2024-07-26T08:58:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: pgmap v4467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:58:18.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:58:18.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:58:18.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:58:18.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:58:18.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:58:18.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:58:18.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:58:18.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:58:18.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: pgmap v4468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:58:18.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: pgmap v4469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:58:18.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:58:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7344.021912 secs 2024-07-26T08:58:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: pgmap v4467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:58:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:58:18.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:58:18.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:58:18.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:58:18.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:58:18.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:58:18.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:58:18.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:58:18.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: pgmap v4468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:58:18.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: pgmap v4469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:58:18.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:58:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7344.021912 secs 2024-07-26T08:58:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: pgmap v4467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:58:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T08:58:18.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T08:58:18.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T08:58:18.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T08:58:18.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T08:58:18.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T08:58:18.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T08:58:18.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:58:18.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:58:18.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: pgmap v4468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T08:58:18.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: pgmap v4469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:58:18.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:58:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:20 smithi164 ceph-mon[36970]: pgmap v4470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:58:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:20 smithi163 ceph-mon[36873]: pgmap v4470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:58:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:20 smithi042 ceph-mon[32775]: pgmap v4470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:58:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7331.279804 secs 2024-07-26T08:58:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7349.022040 secs 2024-07-26T08:58:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:22 smithi164 ceph-mon[36970]: pgmap v4471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:58:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7331.279804 secs 2024-07-26T08:58:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7349.022040 secs 2024-07-26T08:58:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:22 smithi163 ceph-mon[36873]: pgmap v4471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:58:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7331.279804 secs 2024-07-26T08:58:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7349.022040 secs 2024-07-26T08:58:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:22 smithi042 ceph-mon[32775]: pgmap v4471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:58:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:24 smithi164 ceph-mon[36970]: pgmap v4472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:58:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:24 smithi163 ceph-mon[36873]: pgmap v4472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:58:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:24 smithi042 ceph-mon[32775]: pgmap v4472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:58:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:58:25.886+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:58:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:58:25.886+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:58:25.888 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:58:25.886+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:58:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:26 smithi164 ceph-mon[36970]: pgmap v4473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:58:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:26 smithi163 ceph-mon[36873]: pgmap v4473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:58:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:26 smithi042 ceph-mon[32775]: pgmap v4473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:58:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7336.279948 secs 2024-07-26T08:58:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7354.022164 secs 2024-07-26T08:58:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7336.279948 secs 2024-07-26T08:58:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7354.022164 secs 2024-07-26T08:58:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7336.279948 secs 2024-07-26T08:58:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7354.022164 secs 2024-07-26T08:58:28.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:28 smithi164 ceph-mon[36970]: pgmap v4474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:58:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:28 smithi163 ceph-mon[36873]: pgmap v4474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:58:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:28 smithi042 ceph-mon[32775]: pgmap v4474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:58:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:29 smithi164 ceph-mon[36970]: pgmap v4475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:29 smithi163 ceph-mon[36873]: pgmap v4475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:29 smithi042 ceph-mon[32775]: pgmap v4475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7341.280072 secs 2024-07-26T08:58:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7359.022310 secs 2024-07-26T08:58:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:32 smithi164 ceph-mon[36970]: pgmap v4476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7341.280072 secs 2024-07-26T08:58:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7359.022310 secs 2024-07-26T08:58:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:32 smithi163 ceph-mon[36873]: pgmap v4476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7341.280072 secs 2024-07-26T08:58:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7359.022310 secs 2024-07-26T08:58:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:32 smithi042 ceph-mon[32775]: pgmap v4476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:34 smithi164 ceph-mon[36970]: pgmap v4477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:58:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:34 smithi163 ceph-mon[36873]: pgmap v4477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:58:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:34 smithi042 ceph-mon[32775]: pgmap v4477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:58:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:36 smithi164 ceph-mon[36970]: pgmap v4478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:36 smithi163 ceph-mon[36873]: pgmap v4478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:36 smithi042 ceph-mon[32775]: pgmap v4478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7346.280198 secs 2024-07-26T08:58:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7364.022435 secs 2024-07-26T08:58:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7346.280198 secs 2024-07-26T08:58:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7364.022435 secs 2024-07-26T08:58:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7346.280198 secs 2024-07-26T08:58:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7364.022435 secs 2024-07-26T08:58:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:38 smithi164 ceph-mon[36970]: pgmap v4479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:38 smithi163 ceph-mon[36873]: pgmap v4479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:38 smithi042 ceph-mon[32775]: pgmap v4479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:39 smithi164 ceph-mon[36970]: pgmap v4480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:39 smithi163 ceph-mon[36873]: pgmap v4480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:39 smithi042 ceph-mon[32775]: pgmap v4480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:40.896 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:58:40.895+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:58:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:58:40.895+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:58:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:58:40.895+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:58:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7351.280293 secs 2024-07-26T08:58:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7369.022557 secs 2024-07-26T08:58:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:42 smithi164 ceph-mon[36970]: pgmap v4481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7351.280293 secs 2024-07-26T08:58:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7369.022557 secs 2024-07-26T08:58:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:42 smithi163 ceph-mon[36873]: pgmap v4481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7351.280293 secs 2024-07-26T08:58:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7369.022557 secs 2024-07-26T08:58:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:42 smithi042 ceph-mon[32775]: pgmap v4481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:44 smithi164 ceph-mon[36970]: pgmap v4482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:58:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:44 smithi163 ceph-mon[36873]: pgmap v4482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:58:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:44 smithi042 ceph-mon[32775]: pgmap v4482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:58:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:46 smithi164 ceph-mon[36970]: pgmap v4483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:46 smithi163 ceph-mon[36873]: pgmap v4483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:46 smithi042 ceph-mon[32775]: pgmap v4483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7356.280359 secs 2024-07-26T08:58:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7374.022697 secs 2024-07-26T08:58:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7356.280359 secs 2024-07-26T08:58:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7374.022697 secs 2024-07-26T08:58:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7356.280359 secs 2024-07-26T08:58:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7374.022697 secs 2024-07-26T08:58:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:48 smithi164 ceph-mon[36970]: pgmap v4484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:48 smithi163 ceph-mon[36873]: pgmap v4484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:48 smithi042 ceph-mon[32775]: pgmap v4484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:49 smithi164 ceph-mon[36970]: pgmap v4485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:49 smithi163 ceph-mon[36873]: pgmap v4485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:49 smithi042 ceph-mon[32775]: pgmap v4485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7379.022819 secs 2024-07-26T08:58:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7379.022819 secs 2024-07-26T08:58:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7379.022819 secs 2024-07-26T08:58:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7361.280482 secs 2024-07-26T08:58:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:52 smithi163 ceph-mon[36873]: pgmap v4486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7361.280482 secs 2024-07-26T08:58:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:52 smithi042 ceph-mon[32775]: pgmap v4486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7361.280482 secs 2024-07-26T08:58:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:52 smithi164 ceph-mon[36970]: pgmap v4486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:54 smithi164 ceph-mon[36970]: pgmap v4487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:58:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:54 smithi163 ceph-mon[36873]: pgmap v4487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:58:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:54 smithi042 ceph-mon[32775]: pgmap v4487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:58:55.901 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:58:55.898+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:58:55.901 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:58:55.898+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:58:55.901 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:58:55.898+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:58:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:56 smithi164 ceph-mon[36970]: pgmap v4488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7384.022885 secs 2024-07-26T08:58:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:56 smithi163 ceph-mon[36873]: pgmap v4488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7384.022885 secs 2024-07-26T08:58:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:56 smithi042 ceph-mon[32775]: pgmap v4488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7384.022885 secs 2024-07-26T08:58:57.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7366.280550 secs 2024-07-26T08:58:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7366.280550 secs 2024-07-26T08:58:57.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7366.280550 secs 2024-07-26T08:58:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:58 smithi164 ceph-mon[36970]: pgmap v4489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:58 smithi163 ceph-mon[36873]: pgmap v4489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:58 smithi042 ceph-mon[32775]: pgmap v4489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:58:59.001 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:58:58.999+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:58:59.001 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:58:58.999+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:58:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:58:58.999+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:58:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:58:58.999+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:58:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:58:59 smithi164 ceph-mon[36970]: pgmap v4490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:58:59 smithi163 ceph-mon[36873]: pgmap v4490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:58:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:58:59 smithi042 ceph-mon[32775]: pgmap v4490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7389.023027 secs 2024-07-26T08:59:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7389.023027 secs 2024-07-26T08:59:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7389.023027 secs 2024-07-26T08:59:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7371.280632 secs 2024-07-26T08:59:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:02 smithi163 ceph-mon[36873]: pgmap v4491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7371.280632 secs 2024-07-26T08:59:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:02 smithi042 ceph-mon[32775]: pgmap v4491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7371.280632 secs 2024-07-26T08:59:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:02 smithi164 ceph-mon[36970]: pgmap v4491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:04 smithi164 ceph-mon[36970]: pgmap v4492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:59:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:04 smithi163 ceph-mon[36873]: pgmap v4492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:59:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:04 smithi042 ceph-mon[32775]: pgmap v4492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:59:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:06 smithi164 ceph-mon[36970]: pgmap v4493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7394.023171 secs 2024-07-26T08:59:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:06 smithi163 ceph-mon[36873]: pgmap v4493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7394.023171 secs 2024-07-26T08:59:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:06 smithi042 ceph-mon[32775]: pgmap v4493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7394.023171 secs 2024-07-26T08:59:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7376.280759 secs 2024-07-26T08:59:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:07 smithi164 ceph-mon[36970]: pgmap v4494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7376.280759 secs 2024-07-26T08:59:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:07 smithi163 ceph-mon[36873]: pgmap v4494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7376.280759 secs 2024-07-26T08:59:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:07 smithi042 ceph-mon[32775]: pgmap v4494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:10 smithi164 ceph-mon[36970]: pgmap v4495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:10 smithi163 ceph-mon[36873]: pgmap v4495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:10 smithi042 ceph-mon[32775]: pgmap v4495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:10.909 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:59:10.907+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:59:10.909 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:59:10.908+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:59:10.909 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:59:10.908+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:59:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7399.023296 secs 2024-07-26T08:59:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7399.023296 secs 2024-07-26T08:59:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7399.023296 secs 2024-07-26T08:59:12.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7381.280906 secs 2024-07-26T08:59:12.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:12 smithi164 ceph-mon[36970]: pgmap v4496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7381.280906 secs 2024-07-26T08:59:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:12 smithi163 ceph-mon[36873]: pgmap v4496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7381.280906 secs 2024-07-26T08:59:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:12 smithi042 ceph-mon[32775]: pgmap v4496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:13 smithi164 ceph-mon[36970]: pgmap v4497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:59:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:13 smithi163 ceph-mon[36873]: pgmap v4497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:59:13.776 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:13 smithi042 ceph-mon[32775]: pgmap v4497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:59:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:16 smithi164 ceph-mon[36970]: pgmap v4498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7404.023415 secs 2024-07-26T08:59:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:16 smithi163 ceph-mon[36873]: pgmap v4498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7404.023415 secs 2024-07-26T08:59:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:16 smithi042 ceph-mon[32775]: pgmap v4498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7404.023415 secs 2024-07-26T08:59:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7386.281046 secs 2024-07-26T08:59:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:17 smithi164 ceph-mon[36970]: pgmap v4499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7386.281046 secs 2024-07-26T08:59:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:17 smithi163 ceph-mon[36873]: pgmap v4499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7386.281046 secs 2024-07-26T08:59:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:17 smithi042 ceph-mon[32775]: pgmap v4499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:59:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:59:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:59:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:18 smithi164 ceph-mon[36970]: pgmap v4500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:59:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:18 smithi164 ceph-mon[36970]: pgmap v4501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:59:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:59:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:59:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:59:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:59:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:18 smithi163 ceph-mon[36873]: pgmap v4500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:59:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:18 smithi163 ceph-mon[36873]: pgmap v4501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:59:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:59:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T08:59:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T08:59:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T08:59:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:18 smithi042 ceph-mon[32775]: pgmap v4500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T08:59:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:18 smithi042 ceph-mon[32775]: pgmap v4501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T08:59:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T08:59:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:20 smithi163 ceph-mon[36873]: pgmap v4502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:59:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:20 smithi042 ceph-mon[32775]: pgmap v4502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:59:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:20 smithi164 ceph-mon[36970]: pgmap v4502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:59:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7391.281188 secs 2024-07-26T08:59:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7409.023542 secs 2024-07-26T08:59:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7391.281188 secs 2024-07-26T08:59:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7409.023542 secs 2024-07-26T08:59:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7391.281188 secs 2024-07-26T08:59:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7409.023542 secs 2024-07-26T08:59:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:22 smithi163 ceph-mon[36873]: pgmap v4503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:59:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:22 smithi042 ceph-mon[32775]: pgmap v4503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:59:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:22 smithi164 ceph-mon[36970]: pgmap v4503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T08:59:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:24 smithi163 ceph-mon[36873]: pgmap v4504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:59:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:24 smithi042 ceph-mon[32775]: pgmap v4504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:59:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:24 smithi164 ceph-mon[36970]: pgmap v4504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T08:59:25.912 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:59:25.911+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:59:25.912 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:59:25.911+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:59:25.913 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:59:25.911+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:59:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7396.281307 secs 2024-07-26T08:59:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:26 smithi163 ceph-mon[36873]: pgmap v4505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:59:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7414.023683 secs 2024-07-26T08:59:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7396.281307 secs 2024-07-26T08:59:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:26 smithi042 ceph-mon[32775]: pgmap v4505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:59:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7414.023683 secs 2024-07-26T08:59:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7396.281307 secs 2024-07-26T08:59:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:26 smithi164 ceph-mon[36970]: pgmap v4505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T08:59:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7414.023683 secs 2024-07-26T08:59:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:28 smithi163 ceph-mon[36873]: pgmap v4506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:59:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:28 smithi042 ceph-mon[32775]: pgmap v4506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:59:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:28 smithi164 ceph-mon[36970]: pgmap v4506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T08:59:31.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:30 smithi163 ceph-mon[36873]: pgmap v4507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:30 smithi042 ceph-mon[32775]: pgmap v4507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:30 smithi164 ceph-mon[36970]: pgmap v4507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7401.281431 secs 2024-07-26T08:59:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7419.023815 secs 2024-07-26T08:59:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7401.281431 secs 2024-07-26T08:59:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7419.023815 secs 2024-07-26T08:59:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7401.281431 secs 2024-07-26T08:59:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7419.023815 secs 2024-07-26T08:59:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:33 smithi163 ceph-mon[36873]: pgmap v4508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:33.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:33 smithi042 ceph-mon[32775]: pgmap v4508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:33 smithi164 ceph-mon[36970]: pgmap v4508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:35 smithi042 ceph-mon[32775]: pgmap v4509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:59:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:35 smithi164 ceph-mon[36970]: pgmap v4509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:59:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:35 smithi163 ceph-mon[36873]: pgmap v4509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:59:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7406.281586 secs 2024-07-26T08:59:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:37 smithi042 ceph-mon[32775]: pgmap v4510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7424.023959 secs 2024-07-26T08:59:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7406.281586 secs 2024-07-26T08:59:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:37 smithi164 ceph-mon[36970]: pgmap v4510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7424.023959 secs 2024-07-26T08:59:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7406.281586 secs 2024-07-26T08:59:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:37 smithi163 ceph-mon[36873]: pgmap v4510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7424.023959 secs 2024-07-26T08:59:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:39 smithi042 ceph-mon[32775]: pgmap v4511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:39 smithi164 ceph-mon[36970]: pgmap v4511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:39 smithi163 ceph-mon[36873]: pgmap v4511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:40.922 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:59:40.920+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:59:40.922 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:59:40.920+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:59:40.922 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:59:40.920+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:59:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:41 smithi042 ceph-mon[32775]: pgmap v4512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:41 smithi164 ceph-mon[36970]: pgmap v4512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:41 smithi163 ceph-mon[36873]: pgmap v4512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:42.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7411.281717 secs 2024-07-26T08:59:42.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7429.024056 secs 2024-07-26T08:59:42.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7411.281717 secs 2024-07-26T08:59:42.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7429.024056 secs 2024-07-26T08:59:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7411.281717 secs 2024-07-26T08:59:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7429.024056 secs 2024-07-26T08:59:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:43 smithi042 ceph-mon[32775]: pgmap v4513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:43 smithi164 ceph-mon[36970]: pgmap v4513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:43 smithi163 ceph-mon[36873]: pgmap v4513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:59:44.007+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:59:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:59:44.007+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:59:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:59:44.007+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T08:59:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:59:44.007+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T08:59:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:45 smithi042 ceph-mon[32775]: pgmap v4514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:59:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:45 smithi164 ceph-mon[36970]: pgmap v4514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:59:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:45 smithi163 ceph-mon[36873]: pgmap v4514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:59:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7416.281840 secs 2024-07-26T08:59:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:47 smithi042 ceph-mon[32775]: pgmap v4515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7434.024144 secs 2024-07-26T08:59:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7416.281840 secs 2024-07-26T08:59:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:47 smithi164 ceph-mon[36970]: pgmap v4515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7434.024144 secs 2024-07-26T08:59:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7416.281840 secs 2024-07-26T08:59:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:47 smithi163 ceph-mon[36873]: pgmap v4515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7434.024144 secs 2024-07-26T08:59:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:49 smithi042 ceph-mon[32775]: pgmap v4516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:49 smithi164 ceph-mon[36970]: pgmap v4516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:49 smithi163 ceph-mon[36873]: pgmap v4516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:51 smithi042 ceph-mon[32775]: pgmap v4517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:51 smithi164 ceph-mon[36970]: pgmap v4517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:51 smithi163 ceph-mon[36873]: pgmap v4517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7421.281992 secs 2024-07-26T08:59:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7439.024275 secs 2024-07-26T08:59:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7421.281992 secs 2024-07-26T08:59:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7439.024275 secs 2024-07-26T08:59:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7421.281992 secs 2024-07-26T08:59:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7439.024275 secs 2024-07-26T08:59:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:53 smithi042 ceph-mon[32775]: pgmap v4518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:53 smithi164 ceph-mon[36970]: pgmap v4518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:53 smithi163 ceph-mon[36873]: pgmap v4518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:55 smithi042 ceph-mon[32775]: pgmap v4519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:59:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:55 smithi164 ceph-mon[36970]: pgmap v4519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:59:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:55 smithi163 ceph-mon[36873]: pgmap v4519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T08:59:55.925 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:59:55.924+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T08:59:55.926 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:59:55.924+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T08:59:55.926 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T08:59:55.924+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T08:59:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7426.282086 secs 2024-07-26T08:59:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:57 smithi042 ceph-mon[32775]: pgmap v4520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7444.024376 secs 2024-07-26T08:59:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7426.282086 secs 2024-07-26T08:59:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:57 smithi164 ceph-mon[36970]: pgmap v4520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7444.024376 secs 2024-07-26T08:59:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7426.282086 secs 2024-07-26T08:59:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:57 smithi163 ceph-mon[36873]: pgmap v4520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T08:59:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7444.024376 secs 2024-07-26T08:59:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 08:59:59 smithi042 ceph-mon[32775]: pgmap v4521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 08:59:59 smithi164 ceph-mon[36970]: pgmap v4521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T08:59:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 08:59:59 smithi163 ceph-mon[36873]: pgmap v4521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:00:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:00:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:00:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:01 smithi042 ceph-mon[32775]: pgmap v4522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:01 smithi164 ceph-mon[36970]: pgmap v4522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:01 smithi163 ceph-mon[36873]: pgmap v4522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7431.282227 secs 2024-07-26T09:00:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7449.024492 secs 2024-07-26T09:00:02.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7431.282227 secs 2024-07-26T09:00:02.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7449.024492 secs 2024-07-26T09:00:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7431.282227 secs 2024-07-26T09:00:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7449.024492 secs 2024-07-26T09:00:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:03 smithi042 ceph-mon[32775]: pgmap v4523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:03 smithi164 ceph-mon[36970]: pgmap v4523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:03 smithi163 ceph-mon[36873]: pgmap v4523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:05 smithi042 ceph-mon[32775]: pgmap v4524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:00:05.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:05 smithi164 ceph-mon[36970]: pgmap v4524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:00:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:05 smithi163 ceph-mon[36873]: pgmap v4524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:00:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7454.024630 secs 2024-07-26T09:00:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7454.024630 secs 2024-07-26T09:00:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7454.024630 secs 2024-07-26T09:00:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7436.282353 secs 2024-07-26T09:00:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:07 smithi042 ceph-mon[32775]: pgmap v4525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7436.282353 secs 2024-07-26T09:00:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:07 smithi164 ceph-mon[36970]: pgmap v4525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7436.282353 secs 2024-07-26T09:00:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:07 smithi163 ceph-mon[36873]: pgmap v4525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:09 smithi042 ceph-mon[32775]: pgmap v4526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:09 smithi164 ceph-mon[36970]: pgmap v4526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:09 smithi163 ceph-mon[36873]: pgmap v4526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:10.935 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:00:10.933+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:00:10.935 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:00:10.933+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:00:10.935 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:00:10.933+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:00:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:11 smithi042 ceph-mon[32775]: pgmap v4527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7459.024753 secs 2024-07-26T09:00:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:11 smithi164 ceph-mon[36970]: pgmap v4527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7459.024753 secs 2024-07-26T09:00:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:11 smithi163 ceph-mon[36873]: pgmap v4527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7459.024753 secs 2024-07-26T09:00:12.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7441.282493 secs 2024-07-26T09:00:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7441.282493 secs 2024-07-26T09:00:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7441.282493 secs 2024-07-26T09:00:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:13 smithi042 ceph-mon[32775]: pgmap v4528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:13 smithi164 ceph-mon[36970]: pgmap v4528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:13 smithi163 ceph-mon[36873]: pgmap v4528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:15 smithi042 ceph-mon[32775]: pgmap v4529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:00:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:15 smithi164 ceph-mon[36970]: pgmap v4529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:00:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:15 smithi163 ceph-mon[36873]: pgmap v4529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:00:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7464.024890 secs 2024-07-26T09:00:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7464.024890 secs 2024-07-26T09:00:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7464.024890 secs 2024-07-26T09:00:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7446.282623 secs 2024-07-26T09:00:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:17 smithi042 ceph-mon[32775]: pgmap v4530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7446.282623 secs 2024-07-26T09:00:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:17 smithi164 ceph-mon[36970]: pgmap v4530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7446.282623 secs 2024-07-26T09:00:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:17 smithi163 ceph-mon[36873]: pgmap v4530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:00:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:00:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:00:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:19 smithi042 ceph-mon[32775]: pgmap v4531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:00:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:00:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:00:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:19 smithi164 ceph-mon[36970]: pgmap v4531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:00:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:00:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:00:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:19 smithi163 ceph-mon[36873]: pgmap v4531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:00:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:00:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:00:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:20 smithi042 ceph-mon[32775]: pgmap v4532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:00:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:20 smithi042 ceph-mon[32775]: pgmap v4533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:00:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:20 smithi164 ceph-mon[36970]: pgmap v4532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:00:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:20 smithi164 ceph-mon[36970]: pgmap v4533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:00:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:20 smithi163 ceph-mon[36873]: pgmap v4532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:00:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:20 smithi163 ceph-mon[36873]: pgmap v4533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:00:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7469.025017 secs 2024-07-26T09:00:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7469.025017 secs 2024-07-26T09:00:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7469.025017 secs 2024-07-26T09:00:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:22 smithi042 ceph-mon[32775]: pgmap v4534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:00:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7451.282771 secs 2024-07-26T09:00:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:22 smithi164 ceph-mon[36970]: pgmap v4534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:00:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7451.282771 secs 2024-07-26T09:00:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:22 smithi163 ceph-mon[36873]: pgmap v4534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:00:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7451.282771 secs 2024-07-26T09:00:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:24 smithi164 ceph-mon[36970]: pgmap v4535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:00:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:24 smithi163 ceph-mon[36873]: pgmap v4535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:00:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:24 smithi042 ceph-mon[32775]: pgmap v4535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:00:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:00:25.938+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:00:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:00:25.938+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:00:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:00:25.938+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:00:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:26 smithi164 ceph-mon[36970]: pgmap v4536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:00:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7474.025155 secs 2024-07-26T09:00:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:26 smithi163 ceph-mon[36873]: pgmap v4536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:00:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7474.025155 secs 2024-07-26T09:00:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:26 smithi042 ceph-mon[32775]: pgmap v4536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:00:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7474.025155 secs 2024-07-26T09:00:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7456.282909 secs 2024-07-26T09:00:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7456.282909 secs 2024-07-26T09:00:27.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7456.282909 secs 2024-07-26T09:00:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:28 smithi164 ceph-mon[36970]: pgmap v4537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:00:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:28 smithi163 ceph-mon[36873]: pgmap v4537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:00:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:28 smithi042 ceph-mon[32775]: pgmap v4537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:00:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:30 smithi164 ceph-mon[36970]: pgmap v4538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:00:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:30 smithi163 ceph-mon[36873]: pgmap v4538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:00:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:30 smithi042 ceph-mon[32775]: pgmap v4538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:00:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7479.025266 secs 2024-07-26T09:00:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7479.025266 secs 2024-07-26T09:00:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7479.025266 secs 2024-07-26T09:00:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:32 smithi163 ceph-mon[36873]: pgmap v4539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7461.283040 secs 2024-07-26T09:00:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:32 smithi164 ceph-mon[36970]: pgmap v4539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7461.283040 secs 2024-07-26T09:00:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:32 smithi042 ceph-mon[32775]: pgmap v4539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7461.283040 secs 2024-07-26T09:00:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:34 smithi164 ceph-mon[36970]: pgmap v4540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:34 smithi163 ceph-mon[36873]: pgmap v4540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:34 smithi042 ceph-mon[32775]: pgmap v4540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:36 smithi164 ceph-mon[36970]: pgmap v4541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7484.025387 secs 2024-07-26T09:00:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:36 smithi163 ceph-mon[36873]: pgmap v4541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7484.025387 secs 2024-07-26T09:00:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:36 smithi042 ceph-mon[32775]: pgmap v4541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7484.025387 secs 2024-07-26T09:00:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7466.283117 secs 2024-07-26T09:00:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7466.283117 secs 2024-07-26T09:00:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7466.283117 secs 2024-07-26T09:00:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:38 smithi164 ceph-mon[36970]: pgmap v4542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:38 smithi163 ceph-mon[36873]: pgmap v4542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:38 smithi042 ceph-mon[32775]: pgmap v4542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:40 smithi164 ceph-mon[36970]: pgmap v4543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:00:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:40 smithi163 ceph-mon[36873]: pgmap v4543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:00:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:40 smithi042 ceph-mon[32775]: pgmap v4543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:00:40.947 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:00:40.946+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:00:40.948 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:00:40.946+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:00:40.948 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:00:40.946+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:00:41.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7489.025517 secs 2024-07-26T09:00:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7489.025517 secs 2024-07-26T09:00:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7489.025517 secs 2024-07-26T09:00:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:42 smithi164 ceph-mon[36970]: pgmap v4544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7471.283247 secs 2024-07-26T09:00:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:42 smithi163 ceph-mon[36873]: pgmap v4544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7471.283247 secs 2024-07-26T09:00:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:42 smithi042 ceph-mon[32775]: pgmap v4544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7471.283247 secs 2024-07-26T09:00:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:44 smithi164 ceph-mon[36970]: pgmap v4545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:44 smithi163 ceph-mon[36873]: pgmap v4545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:44 smithi042 ceph-mon[32775]: pgmap v4545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:46 smithi164 ceph-mon[36970]: pgmap v4546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7494.025639 secs 2024-07-26T09:00:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:46 smithi163 ceph-mon[36873]: pgmap v4546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7494.025639 secs 2024-07-26T09:00:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:46 smithi042 ceph-mon[32775]: pgmap v4546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7494.025639 secs 2024-07-26T09:00:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7476.283378 secs 2024-07-26T09:00:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7476.283378 secs 2024-07-26T09:00:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7476.283378 secs 2024-07-26T09:00:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:48 smithi164 ceph-mon[36970]: pgmap v4547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:48 smithi163 ceph-mon[36873]: pgmap v4547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:48.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:48 smithi042 ceph-mon[32775]: pgmap v4547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:50 smithi164 ceph-mon[36970]: pgmap v4548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:00:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:50 smithi163 ceph-mon[36873]: pgmap v4548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:00:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:50 smithi042 ceph-mon[32775]: pgmap v4548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:00:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7499.025760 secs 2024-07-26T09:00:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7499.025760 secs 2024-07-26T09:00:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7499.025760 secs 2024-07-26T09:00:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:52 smithi164 ceph-mon[36970]: pgmap v4549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7481.283443 secs 2024-07-26T09:00:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:52 smithi163 ceph-mon[36873]: pgmap v4549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7481.283443 secs 2024-07-26T09:00:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:52 smithi042 ceph-mon[32775]: pgmap v4549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7481.283443 secs 2024-07-26T09:00:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:53 smithi163 ceph-mon[36873]: pgmap v4550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:53 smithi042 ceph-mon[32775]: pgmap v4550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:53 smithi164 ceph-mon[36970]: pgmap v4550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:55 smithi042 ceph-mon[32775]: pgmap v4551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:00:55.947+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:00:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:00:55.947+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:00:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:00:55.947+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:00:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:55 smithi164 ceph-mon[36970]: pgmap v4551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:55 smithi163 ceph-mon[36873]: pgmap v4551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:00:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7504.025872 secs 2024-07-26T09:00:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7504.025872 secs 2024-07-26T09:00:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7504.025872 secs 2024-07-26T09:00:57.815 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7486.283529 secs 2024-07-26T09:00:57.815 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:57 smithi042 ceph-mon[32775]: pgmap v4552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7486.283529 secs 2024-07-26T09:00:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:57 smithi164 ceph-mon[36970]: pgmap v4552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7486.283529 secs 2024-07-26T09:00:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:57 smithi163 ceph-mon[36873]: pgmap v4552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:00:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:00:59 smithi042 ceph-mon[32775]: pgmap v4553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:00:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:00:59 smithi164 ceph-mon[36970]: pgmap v4553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:01:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:00:59 smithi163 ceph-mon[36873]: pgmap v4553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:01:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:01 smithi042 ceph-mon[32775]: pgmap v4554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7509.025995 secs 2024-07-26T09:01:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:01 smithi164 ceph-mon[36970]: pgmap v4554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7509.025995 secs 2024-07-26T09:01:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:01 smithi163 ceph-mon[36873]: pgmap v4554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7509.025995 secs 2024-07-26T09:01:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7491.283615 secs 2024-07-26T09:01:02.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7491.283615 secs 2024-07-26T09:01:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7491.283615 secs 2024-07-26T09:01:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:03 smithi042 ceph-mon[32775]: pgmap v4555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:03 smithi164 ceph-mon[36970]: pgmap v4555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:03 smithi163 ceph-mon[36873]: pgmap v4555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:05 smithi042 ceph-mon[32775]: pgmap v4556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:05 smithi164 ceph-mon[36970]: pgmap v4556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:05 smithi163 ceph-mon[36873]: pgmap v4556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:06.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7514.026108 secs 2024-07-26T09:01:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7514.026108 secs 2024-07-26T09:01:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7514.026108 secs 2024-07-26T09:01:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7496.283739 secs 2024-07-26T09:01:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:07 smithi042 ceph-mon[32775]: pgmap v4557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7496.283739 secs 2024-07-26T09:01:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:07 smithi164 ceph-mon[36970]: pgmap v4557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7496.283739 secs 2024-07-26T09:01:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:07 smithi163 ceph-mon[36873]: pgmap v4557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:09 smithi042 ceph-mon[32775]: pgmap v4558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:01:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:09 smithi164 ceph-mon[36970]: pgmap v4558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:01:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:09 smithi163 ceph-mon[36873]: pgmap v4558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:01:10.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:01:10.957+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:01:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:01:10.957+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:01:10.959 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:01:10.957+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:01:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:11 smithi042 ceph-mon[32775]: pgmap v4559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7519.026229 secs 2024-07-26T09:01:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:11 smithi164 ceph-mon[36970]: pgmap v4559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7519.026229 secs 2024-07-26T09:01:12.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:11 smithi163 ceph-mon[36873]: pgmap v4559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:12.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7519.026229 secs 2024-07-26T09:01:12.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7501.283843 secs 2024-07-26T09:01:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7501.283843 secs 2024-07-26T09:01:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7501.283843 secs 2024-07-26T09:01:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:13 smithi042 ceph-mon[32775]: pgmap v4560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:13 smithi164 ceph-mon[36970]: pgmap v4560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:13 smithi163 ceph-mon[36873]: pgmap v4560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:15 smithi042 ceph-mon[32775]: pgmap v4561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:15 smithi164 ceph-mon[36970]: pgmap v4561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:15 smithi163 ceph-mon[36873]: pgmap v4561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7524.026328 secs 2024-07-26T09:01:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7524.026328 secs 2024-07-26T09:01:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7524.026328 secs 2024-07-26T09:01:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7506.283972 secs 2024-07-26T09:01:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:17 smithi042 ceph-mon[32775]: pgmap v4562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7506.283972 secs 2024-07-26T09:01:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:17 smithi164 ceph-mon[36970]: pgmap v4562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7506.283972 secs 2024-07-26T09:01:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:17 smithi163 ceph-mon[36873]: pgmap v4562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:01:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:01:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:01:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:19 smithi164 ceph-mon[36970]: pgmap v4563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:01:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:19 smithi163 ceph-mon[36873]: pgmap v4563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:01:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:19 smithi042 ceph-mon[32775]: pgmap v4563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:01:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:01:21.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:01:21.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7529.026416 secs 2024-07-26T09:01:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:01:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:01:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7529.026416 secs 2024-07-26T09:01:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:01:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:01:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:01:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7529.026416 secs 2024-07-26T09:01:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:22 smithi042 ceph-mon[32775]: pgmap v4564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7511.284112 secs 2024-07-26T09:01:22.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:22 smithi042 ceph-mon[32775]: pgmap v4565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:01:22.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:22 smithi042 ceph-mon[32775]: pgmap v4566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T09:01:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:22 smithi164 ceph-mon[36970]: pgmap v4564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7511.284112 secs 2024-07-26T09:01:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:22 smithi164 ceph-mon[36970]: pgmap v4565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:01:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:22 smithi164 ceph-mon[36970]: pgmap v4566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T09:01:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:22 smithi163 ceph-mon[36873]: pgmap v4564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7511.284112 secs 2024-07-26T09:01:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:22 smithi163 ceph-mon[36873]: pgmap v4565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:01:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:22 smithi163 ceph-mon[36873]: pgmap v4566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T09:01:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:24 smithi042 ceph-mon[32775]: pgmap v4567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:01:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:24 smithi164 ceph-mon[36970]: pgmap v4567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:01:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:24 smithi163 ceph-mon[36873]: pgmap v4567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:01:25.963 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:01:25.961+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:01:25.963 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:01:25.961+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:01:25.963 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:01:25.961+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:01:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:26 smithi042 ceph-mon[32775]: pgmap v4568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T09:01:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7534.026533 secs 2024-07-26T09:01:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:26 smithi164 ceph-mon[36970]: pgmap v4568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T09:01:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7534.026533 secs 2024-07-26T09:01:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:26 smithi163 ceph-mon[36873]: pgmap v4568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T09:01:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7534.026533 secs 2024-07-26T09:01:27.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7516.284252 secs 2024-07-26T09:01:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7516.284252 secs 2024-07-26T09:01:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7516.284252 secs 2024-07-26T09:01:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:28 smithi042 ceph-mon[32775]: pgmap v4569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T09:01:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:28 smithi164 ceph-mon[36970]: pgmap v4569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T09:01:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:28 smithi163 ceph-mon[36873]: pgmap v4569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T09:01:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:30 smithi042 ceph-mon[32775]: pgmap v4570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:01:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:30 smithi164 ceph-mon[36970]: pgmap v4570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:01:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:30 smithi163 ceph-mon[36873]: pgmap v4570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:01:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7539.026632 secs 2024-07-26T09:01:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7539.026632 secs 2024-07-26T09:01:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7539.026632 secs 2024-07-26T09:01:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7521.284304 secs 2024-07-26T09:01:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:32 smithi042 ceph-mon[32775]: pgmap v4571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:01:32.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7521.284304 secs 2024-07-26T09:01:32.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:32 smithi163 ceph-mon[36873]: pgmap v4571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:01:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7521.284304 secs 2024-07-26T09:01:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:32 smithi164 ceph-mon[36970]: pgmap v4571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:01:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:34 smithi042 ceph-mon[32775]: pgmap v4572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:34 smithi164 ceph-mon[36970]: pgmap v4572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:34 smithi163 ceph-mon[36873]: pgmap v4572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:36 smithi042 ceph-mon[32775]: pgmap v4573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7544.026754 secs 2024-07-26T09:01:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:36 smithi164 ceph-mon[36970]: pgmap v4573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7544.026754 secs 2024-07-26T09:01:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:36 smithi163 ceph-mon[36873]: pgmap v4573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7544.026754 secs 2024-07-26T09:01:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7526.284394 secs 2024-07-26T09:01:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7526.284394 secs 2024-07-26T09:01:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7526.284394 secs 2024-07-26T09:01:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:38 smithi042 ceph-mon[32775]: pgmap v4574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:38 smithi164 ceph-mon[36970]: pgmap v4574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:38 smithi163 ceph-mon[36873]: pgmap v4574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:40 smithi042 ceph-mon[32775]: pgmap v4575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:01:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:40 smithi164 ceph-mon[36970]: pgmap v4575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:01:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:40 smithi163 ceph-mon[36873]: pgmap v4575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:01:40.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:01:40.965+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:01:40.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:01:40.965+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:01:40.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:01:40.965+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:01:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7549.026892 secs 2024-07-26T09:01:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7549.026892 secs 2024-07-26T09:01:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7549.026892 secs 2024-07-26T09:01:42.639 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7531.284484 secs 2024-07-26T09:01:42.639 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:42 smithi164 ceph-mon[36970]: pgmap v4576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7531.284484 secs 2024-07-26T09:01:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:42 smithi042 ceph-mon[32775]: pgmap v4576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7531.284484 secs 2024-07-26T09:01:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:42 smithi163 ceph-mon[36873]: pgmap v4576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:44 smithi042 ceph-mon[32775]: pgmap v4577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:44.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:44 smithi164 ceph-mon[36970]: pgmap v4577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:44 smithi163 ceph-mon[36873]: pgmap v4577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:46 smithi042 ceph-mon[32775]: pgmap v4578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7554.027016 secs 2024-07-26T09:01:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:46 smithi164 ceph-mon[36970]: pgmap v4578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7554.027016 secs 2024-07-26T09:01:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:46 smithi163 ceph-mon[36873]: pgmap v4578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7554.027016 secs 2024-07-26T09:01:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7536.284637 secs 2024-07-26T09:01:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7536.284637 secs 2024-07-26T09:01:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7536.284637 secs 2024-07-26T09:01:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:48 smithi164 ceph-mon[36970]: pgmap v4579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:48 smithi163 ceph-mon[36873]: pgmap v4579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:48 smithi042 ceph-mon[32775]: pgmap v4579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:50 smithi164 ceph-mon[36970]: pgmap v4580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:01:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:50 smithi163 ceph-mon[36873]: pgmap v4580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:01:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:50 smithi042 ceph-mon[32775]: pgmap v4580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:01:51.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7559.027136 secs 2024-07-26T09:01:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7559.027136 secs 2024-07-26T09:01:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7559.027136 secs 2024-07-26T09:01:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7541.284763 secs 2024-07-26T09:01:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:52 smithi164 ceph-mon[36970]: pgmap v4581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7541.284763 secs 2024-07-26T09:01:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:52 smithi163 ceph-mon[36873]: pgmap v4581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7541.284763 secs 2024-07-26T09:01:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:52 smithi042 ceph-mon[32775]: pgmap v4581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:54 smithi164 ceph-mon[36970]: pgmap v4582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:54 smithi163 ceph-mon[36873]: pgmap v4582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:54 smithi042 ceph-mon[32775]: pgmap v4582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:01:55.969+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:01:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:01:55.969+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:01:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:01:55.969+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:01:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:56 smithi164 ceph-mon[36970]: pgmap v4583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7564.027262 secs 2024-07-26T09:01:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:56 smithi163 ceph-mon[36873]: pgmap v4583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7564.027262 secs 2024-07-26T09:01:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:56 smithi042 ceph-mon[32775]: pgmap v4583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:01:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7564.027262 secs 2024-07-26T09:01:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7546.284931 secs 2024-07-26T09:01:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7546.284931 secs 2024-07-26T09:01:57.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7546.284931 secs 2024-07-26T09:01:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:01:58 smithi164 ceph-mon[36970]: pgmap v4584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:01:58 smithi163 ceph-mon[36873]: pgmap v4584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:01:58 smithi042 ceph-mon[32775]: pgmap v4584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:01:59.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:01:59.029+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:01:59.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:01:59.029+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:01:59.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:01:59.029+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:01:59.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:01:59.029+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:02:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:00 smithi164 ceph-mon[36970]: pgmap v4585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:00 smithi163 ceph-mon[36873]: pgmap v4585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:00 smithi042 ceph-mon[32775]: pgmap v4585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7569.027364 secs 2024-07-26T09:02:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7569.027364 secs 2024-07-26T09:02:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7569.027364 secs 2024-07-26T09:02:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7551.285066 secs 2024-07-26T09:02:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:02 smithi164 ceph-mon[36970]: pgmap v4586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7551.285066 secs 2024-07-26T09:02:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:02 smithi163 ceph-mon[36873]: pgmap v4586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7551.285066 secs 2024-07-26T09:02:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:02 smithi042 ceph-mon[32775]: pgmap v4586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:04 smithi042 ceph-mon[32775]: pgmap v4587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:04 smithi164 ceph-mon[36970]: pgmap v4587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:04 smithi163 ceph-mon[36873]: pgmap v4587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:06 smithi164 ceph-mon[36970]: pgmap v4588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7574.027484 secs 2024-07-26T09:02:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:06 smithi163 ceph-mon[36873]: pgmap v4588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7574.027484 secs 2024-07-26T09:02:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:06 smithi042 ceph-mon[32775]: pgmap v4588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:06.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7574.027484 secs 2024-07-26T09:02:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7556.285246 secs 2024-07-26T09:02:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7556.285246 secs 2024-07-26T09:02:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7556.285246 secs 2024-07-26T09:02:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:08 smithi164 ceph-mon[36970]: pgmap v4589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:08 smithi163 ceph-mon[36873]: pgmap v4589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:08.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:08 smithi042 ceph-mon[32775]: pgmap v4589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:10 smithi164 ceph-mon[36970]: pgmap v4590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:10 smithi163 ceph-mon[36873]: pgmap v4590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:10 smithi042 ceph-mon[32775]: pgmap v4590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:02:10.979+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:02:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:02:10.979+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:02:10.982 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:02:10.979+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:02:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7579.027617 secs 2024-07-26T09:02:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7579.027617 secs 2024-07-26T09:02:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7579.027617 secs 2024-07-26T09:02:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7561.285369 secs 2024-07-26T09:02:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:12 smithi164 ceph-mon[36970]: pgmap v4591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7561.285369 secs 2024-07-26T09:02:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:12 smithi163 ceph-mon[36873]: pgmap v4591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7561.285369 secs 2024-07-26T09:02:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:12 smithi042 ceph-mon[32775]: pgmap v4591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:14 smithi164 ceph-mon[36970]: pgmap v4592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:14 smithi163 ceph-mon[36873]: pgmap v4592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:14 smithi042 ceph-mon[32775]: pgmap v4592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:16 smithi163 ceph-mon[36873]: pgmap v4593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7584.027741 secs 2024-07-26T09:02:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:16 smithi042 ceph-mon[32775]: pgmap v4593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7584.027741 secs 2024-07-26T09:02:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:16 smithi164 ceph-mon[36970]: pgmap v4593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7584.027741 secs 2024-07-26T09:02:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7566.285454 secs 2024-07-26T09:02:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7566.285454 secs 2024-07-26T09:02:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7566.285454 secs 2024-07-26T09:02:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:18 smithi163 ceph-mon[36873]: pgmap v4594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:18 smithi042 ceph-mon[32775]: pgmap v4594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:18 smithi164 ceph-mon[36970]: pgmap v4594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:19 smithi163 ceph-mon[36873]: pgmap v4595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:19.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:19 smithi042 ceph-mon[32775]: pgmap v4595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:19 smithi164 ceph-mon[36970]: pgmap v4595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7571.285576 secs 2024-07-26T09:02:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:22 smithi042 ceph-mon[32775]: pgmap v4596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:02:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7589.027844 secs 2024-07-26T09:02:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:02:22.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:02:22.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7571.285576 secs 2024-07-26T09:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:22 smithi164 ceph-mon[36970]: pgmap v4596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7589.027844 secs 2024-07-26T09:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:02:22.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7571.285576 secs 2024-07-26T09:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:22 smithi163 ceph-mon[36873]: pgmap v4596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7589.027844 secs 2024-07-26T09:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:22.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:02:22.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:02:22.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:02:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:23 smithi164 ceph-mon[36970]: pgmap v4597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T09:02:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:23 smithi164 ceph-mon[36970]: pgmap v4598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:02:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:23 smithi163 ceph-mon[36873]: pgmap v4597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T09:02:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:23 smithi163 ceph-mon[36873]: pgmap v4598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:02:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:23 smithi042 ceph-mon[32775]: pgmap v4597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T09:02:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:23 smithi042 ceph-mon[32775]: pgmap v4598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:02:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:25 smithi164 ceph-mon[36970]: pgmap v4599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T09:02:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:25 smithi163 ceph-mon[36873]: pgmap v4599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T09:02:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:25 smithi042 ceph-mon[32775]: pgmap v4599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T09:02:25.984 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:02:25.983+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:02:25.984 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:02:25.983+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:02:25.984 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:02:25.983+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:02:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7594.027961 secs 2024-07-26T09:02:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7594.027961 secs 2024-07-26T09:02:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7594.027961 secs 2024-07-26T09:02:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7576.285701 secs 2024-07-26T09:02:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:27 smithi164 ceph-mon[36970]: pgmap v4600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:02:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7576.285701 secs 2024-07-26T09:02:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:27 smithi163 ceph-mon[36873]: pgmap v4600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:02:27.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7576.285701 secs 2024-07-26T09:02:27.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:27 smithi042 ceph-mon[32775]: pgmap v4600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:02:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:29 smithi164 ceph-mon[36970]: pgmap v4601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:02:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:29 smithi163 ceph-mon[36873]: pgmap v4601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:02:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:29 smithi042 ceph-mon[32775]: pgmap v4601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:02:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:31 smithi164 ceph-mon[36970]: pgmap v4602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:02:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7599.028101 secs 2024-07-26T09:02:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:31 smithi163 ceph-mon[36873]: pgmap v4602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:02:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7599.028101 secs 2024-07-26T09:02:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:31 smithi042 ceph-mon[32775]: pgmap v4602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:02:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7599.028101 secs 2024-07-26T09:02:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7581.285801 secs 2024-07-26T09:02:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7581.285801 secs 2024-07-26T09:02:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7581.285801 secs 2024-07-26T09:02:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:33 smithi164 ceph-mon[36970]: pgmap v4603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:02:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:33 smithi163 ceph-mon[36873]: pgmap v4603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:02:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:33 smithi042 ceph-mon[32775]: pgmap v4603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:02:35.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:35 smithi164 ceph-mon[36970]: pgmap v4604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:35 smithi163 ceph-mon[36873]: pgmap v4604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:35.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:35 smithi042 ceph-mon[32775]: pgmap v4604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7604.028218 secs 2024-07-26T09:02:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7604.028218 secs 2024-07-26T09:02:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7604.028218 secs 2024-07-26T09:02:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7586.285928 secs 2024-07-26T09:02:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:37 smithi164 ceph-mon[36970]: pgmap v4605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7586.285928 secs 2024-07-26T09:02:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:37 smithi163 ceph-mon[36873]: pgmap v4605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7586.285928 secs 2024-07-26T09:02:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:37 smithi042 ceph-mon[32775]: pgmap v4605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:39 smithi164 ceph-mon[36970]: pgmap v4606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:39 smithi163 ceph-mon[36873]: pgmap v4606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:39 smithi042 ceph-mon[32775]: pgmap v4606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:02:40.984+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:02:40.986 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:02:40.985+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:02:40.987 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:02:40.985+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:02:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:41 smithi163 ceph-mon[36873]: pgmap v4607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7609.028339 secs 2024-07-26T09:02:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:41 smithi042 ceph-mon[32775]: pgmap v4607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7609.028339 secs 2024-07-26T09:02:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:41 smithi164 ceph-mon[36970]: pgmap v4607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7609.028339 secs 2024-07-26T09:02:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7591.286054 secs 2024-07-26T09:02:42.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7591.286054 secs 2024-07-26T09:02:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7591.286054 secs 2024-07-26T09:02:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:43 smithi163 ceph-mon[36873]: pgmap v4608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:43.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:43 smithi042 ceph-mon[32775]: pgmap v4608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:43 smithi164 ceph-mon[36970]: pgmap v4608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:44 smithi163 ceph-mon[36873]: pgmap v4609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:44 smithi042 ceph-mon[32775]: pgmap v4609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:44 smithi164 ceph-mon[36970]: pgmap v4609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7596.286156 secs 2024-07-26T09:02:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:46 smithi164 ceph-mon[36970]: pgmap v4610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7614.028461 secs 2024-07-26T09:02:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7596.286156 secs 2024-07-26T09:02:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:46 smithi163 ceph-mon[36873]: pgmap v4610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7614.028461 secs 2024-07-26T09:02:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7596.286156 secs 2024-07-26T09:02:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:46 smithi042 ceph-mon[32775]: pgmap v4610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7614.028461 secs 2024-07-26T09:02:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:48 smithi164 ceph-mon[36970]: pgmap v4611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:48 smithi163 ceph-mon[36873]: pgmap v4611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:48 smithi042 ceph-mon[32775]: pgmap v4611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:50 smithi042 ceph-mon[32775]: pgmap v4612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:50 smithi164 ceph-mon[36970]: pgmap v4612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:50 smithi163 ceph-mon[36873]: pgmap v4612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7601.286255 secs 2024-07-26T09:02:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7619.028592 secs 2024-07-26T09:02:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7601.286255 secs 2024-07-26T09:02:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7619.028592 secs 2024-07-26T09:02:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7601.286255 secs 2024-07-26T09:02:52.530 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7619.028592 secs 2024-07-26T09:02:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:52 smithi164 ceph-mon[36970]: pgmap v4613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:52 smithi163 ceph-mon[36873]: pgmap v4613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:52 smithi042 ceph-mon[32775]: pgmap v4613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:54 smithi164 ceph-mon[36970]: pgmap v4614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:54 smithi163 ceph-mon[36873]: pgmap v4614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:54 smithi042 ceph-mon[32775]: pgmap v4614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:02:55.991 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:02:55.990+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:02:55.992 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:02:55.990+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:02:55.992 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:02:55.990+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:02:57.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7606.286338 secs 2024-07-26T09:02:57.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:56 smithi042 ceph-mon[32775]: pgmap v4615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:57.199 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7624.028715 secs 2024-07-26T09:02:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7606.286338 secs 2024-07-26T09:02:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:56 smithi164 ceph-mon[36970]: pgmap v4615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7624.028715 secs 2024-07-26T09:02:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7606.286338 secs 2024-07-26T09:02:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:56 smithi163 ceph-mon[36873]: pgmap v4615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:02:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7624.028715 secs 2024-07-26T09:02:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:02:58 smithi164 ceph-mon[36970]: pgmap v4616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:02:58 smithi163 ceph-mon[36873]: pgmap v4616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:02:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:02:58 smithi042 ceph-mon[32775]: pgmap v4616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:01.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:00 smithi164 ceph-mon[36970]: pgmap v4617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:00 smithi163 ceph-mon[36873]: pgmap v4617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:00 smithi042 ceph-mon[32775]: pgmap v4617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7611.286399 secs 2024-07-26T09:03:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7629.028856 secs 2024-07-26T09:03:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7611.286399 secs 2024-07-26T09:03:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7629.028856 secs 2024-07-26T09:03:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7611.286399 secs 2024-07-26T09:03:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7629.028856 secs 2024-07-26T09:03:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:02 smithi164 ceph-mon[36970]: pgmap v4618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:02 smithi163 ceph-mon[36873]: pgmap v4618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:02 smithi042 ceph-mon[32775]: pgmap v4618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:04 smithi164 ceph-mon[36970]: pgmap v4619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:03:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:04 smithi163 ceph-mon[36873]: pgmap v4619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:03:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:04 smithi042 ceph-mon[32775]: pgmap v4619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:03:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7616.286512 secs 2024-07-26T09:03:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:06 smithi164 ceph-mon[36970]: pgmap v4620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7634.028978 secs 2024-07-26T09:03:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7616.286512 secs 2024-07-26T09:03:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:06 smithi163 ceph-mon[36873]: pgmap v4620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:07.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7634.028978 secs 2024-07-26T09:03:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7616.286512 secs 2024-07-26T09:03:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:06 smithi042 ceph-mon[32775]: pgmap v4620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7634.028978 secs 2024-07-26T09:03:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:08 smithi164 ceph-mon[36970]: pgmap v4621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:09.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:08 smithi163 ceph-mon[36873]: pgmap v4621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:08 smithi042 ceph-mon[32775]: pgmap v4621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:03:10.998+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:03:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:03:10.998+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:03:11.000 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:03:10.998+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:03:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:10 smithi164 ceph-mon[36970]: pgmap v4622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:10 smithi163 ceph-mon[36873]: pgmap v4622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:10 smithi042 ceph-mon[32775]: pgmap v4622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7621.286660 secs 2024-07-26T09:03:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7639.029105 secs 2024-07-26T09:03:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7621.286660 secs 2024-07-26T09:03:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7639.029105 secs 2024-07-26T09:03:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7621.286660 secs 2024-07-26T09:03:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7639.029105 secs 2024-07-26T09:03:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:12 smithi164 ceph-mon[36970]: pgmap v4623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:12 smithi163 ceph-mon[36873]: pgmap v4623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:12 smithi042 ceph-mon[32775]: pgmap v4623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:14 smithi164 ceph-mon[36970]: pgmap v4624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:03:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:14 smithi163 ceph-mon[36873]: pgmap v4624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:03:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:14 smithi042 ceph-mon[32775]: pgmap v4624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:03:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7626.286741 secs 2024-07-26T09:03:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:16 smithi164 ceph-mon[36970]: pgmap v4625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7644.029220 secs 2024-07-26T09:03:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7626.286741 secs 2024-07-26T09:03:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:16 smithi163 ceph-mon[36873]: pgmap v4625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7644.029220 secs 2024-07-26T09:03:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7626.286741 secs 2024-07-26T09:03:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:16 smithi042 ceph-mon[32775]: pgmap v4625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7644.029220 secs 2024-07-26T09:03:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:18 smithi164 ceph-mon[36970]: pgmap v4626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:19.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:18 smithi163 ceph-mon[36873]: pgmap v4626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:18 smithi042 ceph-mon[32775]: pgmap v4626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:20 smithi163 ceph-mon[36873]: pgmap v4627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:20 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7631.286884 secs 2024-07-26T09:03:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:20 smithi042 ceph-mon[32775]: pgmap v4627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:20 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7631.286884 secs 2024-07-26T09:03:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:20 smithi164 ceph-mon[36970]: pgmap v4627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:20 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7631.286884 secs 2024-07-26T09:03:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7649.029343 secs 2024-07-26T09:03:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:03:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7649.029343 secs 2024-07-26T09:03:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:03:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7649.029343 secs 2024-07-26T09:03:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:03:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:22 smithi163 ceph-mon[36873]: pgmap v4628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:03:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:03:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:22 smithi163 ceph-mon[36873]: pgmap v4629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:03:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:22 smithi163 ceph-mon[36873]: pgmap v4630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:03:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:03:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:22 smithi042 ceph-mon[32775]: pgmap v4628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:03:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:03:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:22 smithi042 ceph-mon[32775]: pgmap v4629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:03:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:22 smithi042 ceph-mon[32775]: pgmap v4630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:03:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:03:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:22 smithi164 ceph-mon[36970]: pgmap v4628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:03:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:03:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:22 smithi164 ceph-mon[36970]: pgmap v4629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:03:23.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:22 smithi164 ceph-mon[36970]: pgmap v4630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:03:23.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:03:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:25 smithi164 ceph-mon[36970]: pgmap v4631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:03:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:25 smithi163 ceph-mon[36873]: pgmap v4631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:03:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:25 smithi042 ceph-mon[32775]: pgmap v4631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:03:26.004 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:03:26.002+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:03:26.004 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:03:26.002+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:03:26.004 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:03:26.002+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:03:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7636.287033 secs 2024-07-26T09:03:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7654.029480 secs 2024-07-26T09:03:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7636.287033 secs 2024-07-26T09:03:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7654.029480 secs 2024-07-26T09:03:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7636.287033 secs 2024-07-26T09:03:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7654.029480 secs 2024-07-26T09:03:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:27 smithi164 ceph-mon[36970]: pgmap v4632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:03:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:27 smithi163 ceph-mon[36873]: pgmap v4632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:03:27.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:27 smithi042 ceph-mon[32775]: pgmap v4632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:03:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:29 smithi164 ceph-mon[36970]: pgmap v4633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:03:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:29 smithi163 ceph-mon[36873]: pgmap v4633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:03:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:29 smithi042 ceph-mon[32775]: pgmap v4633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:03:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:31 smithi164 ceph-mon[36970]: pgmap v4634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:03:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7641.287164 secs 2024-07-26T09:03:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7659.029600 secs 2024-07-26T09:03:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:31 smithi163 ceph-mon[36873]: pgmap v4634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:03:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7641.287164 secs 2024-07-26T09:03:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7659.029600 secs 2024-07-26T09:03:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:31 smithi042 ceph-mon[32775]: pgmap v4634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:03:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7641.287164 secs 2024-07-26T09:03:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7659.029600 secs 2024-07-26T09:03:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:33 smithi164 ceph-mon[36970]: pgmap v4635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:03:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:33 smithi163 ceph-mon[36873]: pgmap v4635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:03:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:33 smithi042 ceph-mon[32775]: pgmap v4635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:03:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:35 smithi164 ceph-mon[36970]: pgmap v4636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:35 smithi163 ceph-mon[36873]: pgmap v4636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:35.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:35 smithi042 ceph-mon[32775]: pgmap v4636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7646.287295 secs 2024-07-26T09:03:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7664.029723 secs 2024-07-26T09:03:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7646.287295 secs 2024-07-26T09:03:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7664.029723 secs 2024-07-26T09:03:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7646.287295 secs 2024-07-26T09:03:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7664.029723 secs 2024-07-26T09:03:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:37 smithi163 ceph-mon[36873]: pgmap v4637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:37 smithi042 ceph-mon[32775]: pgmap v4637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:37 smithi164 ceph-mon[36970]: pgmap v4637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:39 smithi163 ceph-mon[36873]: pgmap v4638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:03:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:39 smithi042 ceph-mon[32775]: pgmap v4638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:03:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:39 smithi164 ceph-mon[36970]: pgmap v4638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:03:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:03:41.011+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:03:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:03:41.011+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:03:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:03:41.011+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:03:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:41 smithi163 ceph-mon[36873]: pgmap v4639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:03:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7651.287423 secs 2024-07-26T09:03:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7669.029840 secs 2024-07-26T09:03:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:41 smithi042 ceph-mon[32775]: pgmap v4639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:03:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7651.287423 secs 2024-07-26T09:03:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7669.029840 secs 2024-07-26T09:03:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:41 smithi164 ceph-mon[36970]: pgmap v4639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:03:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7651.287423 secs 2024-07-26T09:03:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7669.029840 secs 2024-07-26T09:03:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:43 smithi163 ceph-mon[36873]: pgmap v4640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:03:43.775 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:43 smithi164 ceph-mon[36970]: pgmap v4640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:03:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:43 smithi042 ceph-mon[32775]: pgmap v4640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:03:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:45 smithi163 ceph-mon[36873]: pgmap v4641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:03:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:45 smithi042 ceph-mon[32775]: pgmap v4641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:03:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:45 smithi164 ceph-mon[36970]: pgmap v4641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:03:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7656.287531 secs 2024-07-26T09:03:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7674.029948 secs 2024-07-26T09:03:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7656.287531 secs 2024-07-26T09:03:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7674.029948 secs 2024-07-26T09:03:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7656.287531 secs 2024-07-26T09:03:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7674.029948 secs 2024-07-26T09:03:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:47 smithi163 ceph-mon[36873]: pgmap v4642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:03:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:47 smithi042 ceph-mon[32775]: pgmap v4642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:03:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:47 smithi164 ceph-mon[36970]: pgmap v4642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:03:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:48 smithi163 ceph-mon[36873]: pgmap v4643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:03:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:48 smithi042 ceph-mon[32775]: pgmap v4643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:03:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:48 smithi164 ceph-mon[36970]: pgmap v4643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:03:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:51 smithi164 ceph-mon[36970]: pgmap v4644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:03:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7661.287662 secs 2024-07-26T09:03:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7679.030065 secs 2024-07-26T09:03:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:51 smithi163 ceph-mon[36873]: pgmap v4644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:03:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7661.287662 secs 2024-07-26T09:03:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7679.030065 secs 2024-07-26T09:03:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:51 smithi042 ceph-mon[32775]: pgmap v4644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:03:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7661.287662 secs 2024-07-26T09:03:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7679.030065 secs 2024-07-26T09:03:53.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:53 smithi164 ceph-mon[36970]: pgmap v4645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:53 smithi163 ceph-mon[36873]: pgmap v4645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:53 smithi042 ceph-mon[32775]: pgmap v4645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:55 smithi164 ceph-mon[36970]: pgmap v4646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:55 smithi163 ceph-mon[36873]: pgmap v4646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:55 smithi042 ceph-mon[32775]: pgmap v4646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:03:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:03:56.015+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:03:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:03:56.015+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:03:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:03:56.015+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:03:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7666.287808 secs 2024-07-26T09:03:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7684.030185 secs 2024-07-26T09:03:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7666.287808 secs 2024-07-26T09:03:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7684.030185 secs 2024-07-26T09:03:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7666.287808 secs 2024-07-26T09:03:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7684.030185 secs 2024-07-26T09:03:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:57 smithi164 ceph-mon[36970]: pgmap v4647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:57 smithi163 ceph-mon[36873]: pgmap v4647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:57.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:57 smithi042 ceph-mon[32775]: pgmap v4647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:03:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:03:59 smithi164 ceph-mon[36970]: pgmap v4648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:03:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:03:59 smithi163 ceph-mon[36873]: pgmap v4648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:03:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:03:59 smithi042 ceph-mon[32775]: pgmap v4648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:04:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:01 smithi163 ceph-mon[36873]: pgmap v4649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7671.287956 secs 2024-07-26T09:04:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7689.030304 secs 2024-07-26T09:04:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:01 smithi042 ceph-mon[32775]: pgmap v4649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7671.287956 secs 2024-07-26T09:04:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7689.030304 secs 2024-07-26T09:04:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:01 smithi164 ceph-mon[36970]: pgmap v4649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7671.287956 secs 2024-07-26T09:04:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7689.030304 secs 2024-07-26T09:04:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:03 smithi163 ceph-mon[36873]: pgmap v4650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:03 smithi042 ceph-mon[32775]: pgmap v4650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:03 smithi164 ceph-mon[36970]: pgmap v4650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:05 smithi163 ceph-mon[36873]: pgmap v4651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:05 smithi042 ceph-mon[32775]: pgmap v4651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:05 smithi164 ceph-mon[36970]: pgmap v4651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7676.288091 secs 2024-07-26T09:04:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7694.030426 secs 2024-07-26T09:04:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7676.288091 secs 2024-07-26T09:04:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7694.030426 secs 2024-07-26T09:04:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7676.288091 secs 2024-07-26T09:04:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7694.030426 secs 2024-07-26T09:04:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:07 smithi163 ceph-mon[36873]: pgmap v4652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:07 smithi042 ceph-mon[32775]: pgmap v4652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:07 smithi164 ceph-mon[36970]: pgmap v4652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:09 smithi163 ceph-mon[36873]: pgmap v4653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:04:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:09 smithi042 ceph-mon[32775]: pgmap v4653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:04:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:09 smithi164 ceph-mon[36970]: pgmap v4653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:04:11.026 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:04:11.025+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:04:11.026 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:04:11.025+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:04:11.026 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:04:11.025+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:04:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:11 smithi163 ceph-mon[36873]: pgmap v4654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7681.288248 secs 2024-07-26T09:04:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7699.030561 secs 2024-07-26T09:04:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:11 smithi042 ceph-mon[32775]: pgmap v4654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7681.288248 secs 2024-07-26T09:04:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7699.030561 secs 2024-07-26T09:04:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:11 smithi164 ceph-mon[36970]: pgmap v4654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7681.288248 secs 2024-07-26T09:04:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7699.030561 secs 2024-07-26T09:04:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:13 smithi163 ceph-mon[36873]: pgmap v4655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:13.775 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:13 smithi164 ceph-mon[36970]: pgmap v4655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:13 smithi042 ceph-mon[32775]: pgmap v4655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:14 smithi042 ceph-mon[32775]: pgmap v4656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:14 smithi164 ceph-mon[36970]: pgmap v4656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:15.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:14 smithi163 ceph-mon[36873]: pgmap v4656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:15 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7686.288370 secs 2024-07-26T09:04:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:15 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7686.288370 secs 2024-07-26T09:04:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:15 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7686.288370 secs 2024-07-26T09:04:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7704.030664 secs 2024-07-26T09:04:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:16 smithi164 ceph-mon[36970]: pgmap v4657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7704.030664 secs 2024-07-26T09:04:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:16 smithi163 ceph-mon[36873]: pgmap v4657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7704.030664 secs 2024-07-26T09:04:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:16 smithi042 ceph-mon[32775]: pgmap v4657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:19 smithi164 ceph-mon[36970]: pgmap v4658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:04:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:19 smithi163 ceph-mon[36873]: pgmap v4658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:04:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:19 smithi042 ceph-mon[32775]: pgmap v4658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:04:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:21 smithi164 ceph-mon[36970]: pgmap v4659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7691.288456 secs 2024-07-26T09:04:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7709.030788 secs 2024-07-26T09:04:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:21 smithi163 ceph-mon[36873]: pgmap v4659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7691.288456 secs 2024-07-26T09:04:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7709.030788 secs 2024-07-26T09:04:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:21 smithi042 ceph-mon[32775]: pgmap v4659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7691.288456 secs 2024-07-26T09:04:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7709.030788 secs 2024-07-26T09:04:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:23 smithi163 ceph-mon[36873]: pgmap v4660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:04:23.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:04:23.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:04:23.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:04:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:23 smithi042 ceph-mon[32775]: pgmap v4660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:04:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:04:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:04:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:04:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:23 smithi164 ceph-mon[36970]: pgmap v4660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:04:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:04:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:04:23.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:04:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:24 smithi163 ceph-mon[36873]: pgmap v4661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:04:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:24 smithi163 ceph-mon[36873]: pgmap v4662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:04:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:24 smithi042 ceph-mon[32775]: pgmap v4661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:04:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:24 smithi042 ceph-mon[32775]: pgmap v4662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:04:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:24 smithi164 ceph-mon[36970]: pgmap v4661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:04:24.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:24 smithi164 ceph-mon[36970]: pgmap v4662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:04:26.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:04:26.029+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:04:26.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:04:26.029+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:04:26.031 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:04:26.029+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:04:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:26 smithi163 ceph-mon[36873]: pgmap v4663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:04:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7696.288583 secs 2024-07-26T09:04:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7714.030914 secs 2024-07-26T09:04:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:26 smithi042 ceph-mon[32775]: pgmap v4663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:04:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7696.288583 secs 2024-07-26T09:04:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7714.030914 secs 2024-07-26T09:04:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:26 smithi164 ceph-mon[36970]: pgmap v4663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:04:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7696.288583 secs 2024-07-26T09:04:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7714.030914 secs 2024-07-26T09:04:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:28 smithi163 ceph-mon[36873]: pgmap v4664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:04:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:28 smithi042 ceph-mon[32775]: pgmap v4664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:04:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:28 smithi164 ceph-mon[36970]: pgmap v4664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:04:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:30 smithi163 ceph-mon[36873]: pgmap v4665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:04:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:30 smithi042 ceph-mon[32775]: pgmap v4665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:04:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:30 smithi164 ceph-mon[36970]: pgmap v4665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:04:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7701.288712 secs 2024-07-26T09:04:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7719.031018 secs 2024-07-26T09:04:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7701.288712 secs 2024-07-26T09:04:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7719.031018 secs 2024-07-26T09:04:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7701.288712 secs 2024-07-26T09:04:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7719.031018 secs 2024-07-26T09:04:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:32 smithi163 ceph-mon[36873]: pgmap v4666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:04:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:32 smithi042 ceph-mon[32775]: pgmap v4666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:04:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:32 smithi164 ceph-mon[36970]: pgmap v4666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:04:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:34 smithi163 ceph-mon[36873]: pgmap v4667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:04:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:34 smithi042 ceph-mon[32775]: pgmap v4667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:04:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:34 smithi164 ceph-mon[36970]: pgmap v4667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:04:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:36 smithi163 ceph-mon[36873]: pgmap v4668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7706.288820 secs 2024-07-26T09:04:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7724.031122 secs 2024-07-26T09:04:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:36 smithi042 ceph-mon[32775]: pgmap v4668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7706.288820 secs 2024-07-26T09:04:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7724.031122 secs 2024-07-26T09:04:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:36 smithi164 ceph-mon[36970]: pgmap v4668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7706.288820 secs 2024-07-26T09:04:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7724.031122 secs 2024-07-26T09:04:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:38 smithi163 ceph-mon[36873]: pgmap v4669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:38 smithi042 ceph-mon[32775]: pgmap v4669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:38 smithi164 ceph-mon[36970]: pgmap v4669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:39 smithi042 ceph-mon[32775]: pgmap v4670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:04:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:39 smithi164 ceph-mon[36970]: pgmap v4670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:04:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:39 smithi163 ceph-mon[36873]: pgmap v4670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:04:41.032 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:04:41.030+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:04:41.032 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:04:41.030+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:04:41.032 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:04:41.030+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:04:41.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:40 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7711.288969 secs 2024-07-26T09:04:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:40 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7711.288969 secs 2024-07-26T09:04:41.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:40 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7711.288969 secs 2024-07-26T09:04:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:41 smithi164 ceph-mon[36970]: pgmap v4671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7729.031279 secs 2024-07-26T09:04:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:41 smithi163 ceph-mon[36873]: pgmap v4671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7729.031279 secs 2024-07-26T09:04:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:41 smithi042 ceph-mon[32775]: pgmap v4671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7729.031279 secs 2024-07-26T09:04:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:44 smithi042 ceph-mon[32775]: pgmap v4672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:44 smithi164 ceph-mon[36970]: pgmap v4672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:44 smithi163 ceph-mon[36873]: pgmap v4672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:46 smithi042 ceph-mon[32775]: pgmap v4673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7716.289095 secs 2024-07-26T09:04:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:46 smithi164 ceph-mon[36970]: pgmap v4673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7716.289095 secs 2024-07-26T09:04:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:46 smithi163 ceph-mon[36873]: pgmap v4673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7716.289095 secs 2024-07-26T09:04:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7734.031402 secs 2024-07-26T09:04:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7734.031402 secs 2024-07-26T09:04:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7734.031402 secs 2024-07-26T09:04:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:48 smithi042 ceph-mon[32775]: pgmap v4674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:48 smithi164 ceph-mon[36970]: pgmap v4674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:48 smithi163 ceph-mon[36873]: pgmap v4674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:50 smithi042 ceph-mon[32775]: pgmap v4675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:04:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:50 smithi164 ceph-mon[36970]: pgmap v4675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:04:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:50 smithi163 ceph-mon[36873]: pgmap v4675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:04:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7721.289243 secs 2024-07-26T09:04:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7721.289243 secs 2024-07-26T09:04:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7721.289243 secs 2024-07-26T09:04:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:52 smithi042 ceph-mon[32775]: pgmap v4676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7739.031529 secs 2024-07-26T09:04:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:52 smithi164 ceph-mon[36970]: pgmap v4676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7739.031529 secs 2024-07-26T09:04:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:52 smithi163 ceph-mon[36873]: pgmap v4676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7739.031529 secs 2024-07-26T09:04:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:54 smithi042 ceph-mon[32775]: pgmap v4677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:54 smithi164 ceph-mon[36970]: pgmap v4677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:54 smithi163 ceph-mon[36873]: pgmap v4677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:56.036 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:04:56.034+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:04:56.036 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:04:56.034+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:04:56.036 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:04:56.034+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:04:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:56 smithi042 ceph-mon[32775]: pgmap v4678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7726.289306 secs 2024-07-26T09:04:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:56 smithi164 ceph-mon[36970]: pgmap v4678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7726.289306 secs 2024-07-26T09:04:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:56 smithi163 ceph-mon[36873]: pgmap v4678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:04:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7726.289306 secs 2024-07-26T09:04:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7744.031629 secs 2024-07-26T09:04:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7744.031629 secs 2024-07-26T09:04:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7744.031629 secs 2024-07-26T09:04:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:04:58 smithi042 ceph-mon[32775]: pgmap v4679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:04:58 smithi164 ceph-mon[36970]: pgmap v4679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:04:58 smithi163 ceph-mon[36873]: pgmap v4679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:04:59.058 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:04:59.056+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:04:59.058 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:04:59.056+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:04:59.058 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:04:59.056+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:04:59.058 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:04:59.056+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:05:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:00 smithi042 ceph-mon[32775]: pgmap v4680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:05:00.491 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:00 smithi164 ceph-mon[36970]: pgmap v4680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:05:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:00 smithi163 ceph-mon[36873]: pgmap v4680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:05:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7731.289390 secs 2024-07-26T09:05:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7731.289390 secs 2024-07-26T09:05:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7731.289390 secs 2024-07-26T09:05:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:02 smithi042 ceph-mon[32775]: pgmap v4681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7749.031760 secs 2024-07-26T09:05:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:02 smithi164 ceph-mon[36970]: pgmap v4681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7749.031760 secs 2024-07-26T09:05:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:02 smithi163 ceph-mon[36873]: pgmap v4681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7749.031760 secs 2024-07-26T09:05:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:04 smithi042 ceph-mon[32775]: pgmap v4682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:04 smithi164 ceph-mon[36970]: pgmap v4682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:04 smithi163 ceph-mon[36873]: pgmap v4682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:06 smithi042 ceph-mon[32775]: pgmap v4683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7736.289507 secs 2024-07-26T09:05:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:06 smithi164 ceph-mon[36970]: pgmap v4683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:06.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7736.289507 secs 2024-07-26T09:05:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:06 smithi163 ceph-mon[36873]: pgmap v4683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7736.289507 secs 2024-07-26T09:05:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7754.031904 secs 2024-07-26T09:05:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7754.031904 secs 2024-07-26T09:05:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7754.031904 secs 2024-07-26T09:05:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:08 smithi042 ceph-mon[32775]: pgmap v4684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:08 smithi164 ceph-mon[36970]: pgmap v4684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:08 smithi163 ceph-mon[36873]: pgmap v4684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:10.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:10 smithi042 ceph-mon[32775]: pgmap v4685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:05:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:10 smithi164 ceph-mon[36970]: pgmap v4685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:05:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:10 smithi163 ceph-mon[36873]: pgmap v4685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:05:11.044 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:05:11.042+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:05:11.044 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:05:11.042+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:05:11.044 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:05:11.042+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:05:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7741.289575 secs 2024-07-26T09:05:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7741.289575 secs 2024-07-26T09:05:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7741.289575 secs 2024-07-26T09:05:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:12 smithi042 ceph-mon[32775]: pgmap v4686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7759.032019 secs 2024-07-26T09:05:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:12 smithi164 ceph-mon[36970]: pgmap v4686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7759.032019 secs 2024-07-26T09:05:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:12 smithi163 ceph-mon[36873]: pgmap v4686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7759.032019 secs 2024-07-26T09:05:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:14 smithi042 ceph-mon[32775]: pgmap v4687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:14 smithi164 ceph-mon[36970]: pgmap v4687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:14.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:14 smithi163 ceph-mon[36873]: pgmap v4687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:16 smithi042 ceph-mon[32775]: pgmap v4688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7746.289699 secs 2024-07-26T09:05:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:16 smithi164 ceph-mon[36970]: pgmap v4688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7746.289699 secs 2024-07-26T09:05:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:16 smithi163 ceph-mon[36873]: pgmap v4688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7746.289699 secs 2024-07-26T09:05:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7764.032149 secs 2024-07-26T09:05:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7764.032149 secs 2024-07-26T09:05:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7764.032149 secs 2024-07-26T09:05:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:18 smithi042 ceph-mon[32775]: pgmap v4689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:18 smithi164 ceph-mon[36970]: pgmap v4689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:18 smithi163 ceph-mon[36873]: pgmap v4689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:20 smithi042 ceph-mon[32775]: pgmap v4690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:05:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:20 smithi164 ceph-mon[36970]: pgmap v4690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:05:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:20 smithi163 ceph-mon[36873]: pgmap v4690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:05:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7751.289820 secs 2024-07-26T09:05:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7751.289820 secs 2024-07-26T09:05:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7751.289820 secs 2024-07-26T09:05:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:22 smithi042 ceph-mon[32775]: pgmap v4691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7769.032226 secs 2024-07-26T09:05:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:22 smithi164 ceph-mon[36970]: pgmap v4691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7769.032226 secs 2024-07-26T09:05:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:22 smithi163 ceph-mon[36873]: pgmap v4691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7769.032226 secs 2024-07-26T09:05:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:05:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:05:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:05:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:24 smithi042 ceph-mon[32775]: pgmap v4692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:05:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:05:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:05:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:24 smithi164 ceph-mon[36970]: pgmap v4692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:05:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:05:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:05:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:24 smithi163 ceph-mon[36873]: pgmap v4692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:05:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:05:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:05:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:25 smithi042 ceph-mon[32775]: pgmap v4693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:05:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:25 smithi042 ceph-mon[32775]: pgmap v4694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:05:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:25 smithi164 ceph-mon[36970]: pgmap v4693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:05:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:25 smithi164 ceph-mon[36970]: pgmap v4694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:05:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:25 smithi163 ceph-mon[36873]: pgmap v4693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:05:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:25 smithi163 ceph-mon[36873]: pgmap v4694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:05:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:05:26.047+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:05:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:05:26.047+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:05:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:05:26.047+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:05:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7756.289945 secs 2024-07-26T09:05:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7756.289945 secs 2024-07-26T09:05:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7756.289945 secs 2024-07-26T09:05:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:27 smithi042 ceph-mon[32775]: pgmap v4695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:05:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7774.032351 secs 2024-07-26T09:05:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:27 smithi164 ceph-mon[36970]: pgmap v4695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:05:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7774.032351 secs 2024-07-26T09:05:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:27 smithi163 ceph-mon[36873]: pgmap v4695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:05:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7774.032351 secs 2024-07-26T09:05:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:29 smithi164 ceph-mon[36970]: pgmap v4696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:05:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:29 smithi163 ceph-mon[36873]: pgmap v4696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:05:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:29 smithi042 ceph-mon[32775]: pgmap v4696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:05:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:31 smithi164 ceph-mon[36970]: pgmap v4697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:05:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7761.290068 secs 2024-07-26T09:05:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:31 smithi163 ceph-mon[36873]: pgmap v4697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:05:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7761.290068 secs 2024-07-26T09:05:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:31 smithi042 ceph-mon[32775]: pgmap v4697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:05:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7761.290068 secs 2024-07-26T09:05:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7779.032474 secs 2024-07-26T09:05:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7779.032474 secs 2024-07-26T09:05:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7779.032474 secs 2024-07-26T09:05:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:33 smithi164 ceph-mon[36970]: pgmap v4698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:05:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:33 smithi163 ceph-mon[36873]: pgmap v4698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:05:33.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:33 smithi042 ceph-mon[32775]: pgmap v4698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:05:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:35 smithi164 ceph-mon[36970]: pgmap v4699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:05:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:35 smithi163 ceph-mon[36873]: pgmap v4699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:05:35.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:35 smithi042 ceph-mon[32775]: pgmap v4699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:05:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7766.290194 secs 2024-07-26T09:05:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7766.290194 secs 2024-07-26T09:05:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7766.290194 secs 2024-07-26T09:05:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:37 smithi164 ceph-mon[36970]: pgmap v4700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7784.032569 secs 2024-07-26T09:05:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:37 smithi163 ceph-mon[36873]: pgmap v4700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7784.032569 secs 2024-07-26T09:05:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:37 smithi042 ceph-mon[32775]: pgmap v4700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7784.032569 secs 2024-07-26T09:05:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:39 smithi164 ceph-mon[36970]: pgmap v4701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:39 smithi163 ceph-mon[36873]: pgmap v4701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:39 smithi042 ceph-mon[32775]: pgmap v4701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:05:41.054+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:05:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:05:41.054+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:05:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:05:41.054+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:05:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:41 smithi164 ceph-mon[36970]: pgmap v4702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7771.290323 secs 2024-07-26T09:05:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:41 smithi163 ceph-mon[36873]: pgmap v4702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7771.290323 secs 2024-07-26T09:05:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:41 smithi042 ceph-mon[32775]: pgmap v4702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7771.290323 secs 2024-07-26T09:05:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7789.032711 secs 2024-07-26T09:05:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7789.032711 secs 2024-07-26T09:05:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7789.032711 secs 2024-07-26T09:05:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:43 smithi164 ceph-mon[36970]: pgmap v4703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:43 smithi163 ceph-mon[36873]: pgmap v4703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:43 smithi042 ceph-mon[32775]: pgmap v4703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:45 smithi164 ceph-mon[36970]: pgmap v4704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:05:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:45 smithi163 ceph-mon[36873]: pgmap v4704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:05:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:45 smithi042 ceph-mon[32775]: pgmap v4704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:05:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7776.290447 secs 2024-07-26T09:05:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7776.290447 secs 2024-07-26T09:05:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7776.290447 secs 2024-07-26T09:05:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:47 smithi164 ceph-mon[36970]: pgmap v4705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7794.032865 secs 2024-07-26T09:05:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:47 smithi163 ceph-mon[36873]: pgmap v4705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7794.032865 secs 2024-07-26T09:05:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:47 smithi042 ceph-mon[32775]: pgmap v4705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7794.032865 secs 2024-07-26T09:05:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:49 smithi164 ceph-mon[36970]: pgmap v4706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:49 smithi163 ceph-mon[36873]: pgmap v4706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:49 smithi042 ceph-mon[32775]: pgmap v4706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:51 smithi164 ceph-mon[36970]: pgmap v4707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7781.290576 secs 2024-07-26T09:05:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:51 smithi163 ceph-mon[36873]: pgmap v4707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7781.290576 secs 2024-07-26T09:05:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:51 smithi042 ceph-mon[32775]: pgmap v4707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7781.290576 secs 2024-07-26T09:05:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7799.032985 secs 2024-07-26T09:05:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7799.032985 secs 2024-07-26T09:05:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7799.032985 secs 2024-07-26T09:05:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:53 smithi164 ceph-mon[36970]: pgmap v4708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:53 smithi163 ceph-mon[36873]: pgmap v4708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:53 smithi042 ceph-mon[32775]: pgmap v4708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:55 smithi164 ceph-mon[36970]: pgmap v4709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:05:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:55 smithi163 ceph-mon[36873]: pgmap v4709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:05:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:55 smithi042 ceph-mon[32775]: pgmap v4709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:05:56.060 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:05:56.058+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:05:56.060 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:05:56.058+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:05:56.060 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:05:56.058+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:05:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7786.290683 secs 2024-07-26T09:05:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7786.290683 secs 2024-07-26T09:05:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7786.290683 secs 2024-07-26T09:05:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:57 smithi164 ceph-mon[36970]: pgmap v4710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7804.033103 secs 2024-07-26T09:05:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:57 smithi163 ceph-mon[36873]: pgmap v4710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7804.033103 secs 2024-07-26T09:05:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:57 smithi042 ceph-mon[32775]: pgmap v4710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:05:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7804.033103 secs 2024-07-26T09:05:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:05:59 smithi163 ceph-mon[36873]: pgmap v4711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:05:59 smithi042 ceph-mon[32775]: pgmap v4711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:05:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:05:59 smithi164 ceph-mon[36970]: pgmap v4711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:01 smithi163 ceph-mon[36873]: pgmap v4712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7791.290787 secs 2024-07-26T09:06:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:01 smithi042 ceph-mon[32775]: pgmap v4712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7791.290787 secs 2024-07-26T09:06:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:01 smithi164 ceph-mon[36970]: pgmap v4712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7791.290787 secs 2024-07-26T09:06:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7809.033231 secs 2024-07-26T09:06:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7809.033231 secs 2024-07-26T09:06:02.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7809.033231 secs 2024-07-26T09:06:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:03 smithi163 ceph-mon[36873]: pgmap v4713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:03 smithi042 ceph-mon[32775]: pgmap v4713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:03 smithi164 ceph-mon[36970]: pgmap v4713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:05 smithi163 ceph-mon[36873]: pgmap v4714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:06:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:05 smithi042 ceph-mon[32775]: pgmap v4714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:06:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:05 smithi164 ceph-mon[36970]: pgmap v4714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:06:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7796.290884 secs 2024-07-26T09:06:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7814.033351 secs 2024-07-26T09:06:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7796.290884 secs 2024-07-26T09:06:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7814.033351 secs 2024-07-26T09:06:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7796.290884 secs 2024-07-26T09:06:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7814.033351 secs 2024-07-26T09:06:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:07 smithi163 ceph-mon[36873]: pgmap v4715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:07 smithi042 ceph-mon[32775]: pgmap v4715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:07 smithi164 ceph-mon[36970]: pgmap v4715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:09 smithi163 ceph-mon[36873]: pgmap v4716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:09 smithi042 ceph-mon[32775]: pgmap v4716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:09 smithi164 ceph-mon[36970]: pgmap v4716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:11.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:06:11.067+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:06:11.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:06:11.067+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:06:11.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:06:11.067+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:06:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:11 smithi163 ceph-mon[36873]: pgmap v4717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7801.291035 secs 2024-07-26T09:06:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7819.033471 secs 2024-07-26T09:06:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:11 smithi042 ceph-mon[32775]: pgmap v4717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7801.291035 secs 2024-07-26T09:06:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7819.033471 secs 2024-07-26T09:06:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:11 smithi164 ceph-mon[36970]: pgmap v4717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7801.291035 secs 2024-07-26T09:06:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7819.033471 secs 2024-07-26T09:06:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:13 smithi164 ceph-mon[36970]: pgmap v4718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:13 smithi163 ceph-mon[36873]: pgmap v4718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:13 smithi042 ceph-mon[32775]: pgmap v4718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:15 smithi164 ceph-mon[36970]: pgmap v4719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:06:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:15 smithi163 ceph-mon[36873]: pgmap v4719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:06:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:15 smithi042 ceph-mon[32775]: pgmap v4719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:06:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7806.291183 secs 2024-07-26T09:06:16.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7824.033574 secs 2024-07-26T09:06:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7806.291183 secs 2024-07-26T09:06:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7824.033574 secs 2024-07-26T09:06:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7806.291183 secs 2024-07-26T09:06:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7824.033574 secs 2024-07-26T09:06:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:17 smithi164 ceph-mon[36970]: pgmap v4720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:17 smithi163 ceph-mon[36873]: pgmap v4720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:17 smithi042 ceph-mon[32775]: pgmap v4720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:19 smithi164 ceph-mon[36970]: pgmap v4721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:19 smithi163 ceph-mon[36873]: pgmap v4721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:19 smithi042 ceph-mon[32775]: pgmap v4721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:21 smithi164 ceph-mon[36970]: pgmap v4722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7811.291313 secs 2024-07-26T09:06:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7829.033721 secs 2024-07-26T09:06:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:21 smithi163 ceph-mon[36873]: pgmap v4722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7811.291313 secs 2024-07-26T09:06:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7829.033721 secs 2024-07-26T09:06:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:21 smithi042 ceph-mon[32775]: pgmap v4722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7811.291313 secs 2024-07-26T09:06:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7829.033721 secs 2024-07-26T09:06:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:23 smithi164 ceph-mon[36970]: pgmap v4723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:23 smithi163 ceph-mon[36873]: pgmap v4723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:23 smithi042 ceph-mon[32775]: pgmap v4723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:06:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:06:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:06:24.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:06:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:06:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:06:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:06:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:06:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:06:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:06:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:06:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:06:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:25 smithi164 ceph-mon[36970]: pgmap v4724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:06:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:25 smithi164 ceph-mon[36970]: pgmap v4725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:06:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:25 smithi164 ceph-mon[36970]: pgmap v4726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:06:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:25 smithi163 ceph-mon[36873]: pgmap v4724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:06:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:25 smithi163 ceph-mon[36873]: pgmap v4725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:06:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:25 smithi163 ceph-mon[36873]: pgmap v4726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:06:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:25 smithi042 ceph-mon[32775]: pgmap v4724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:06:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:25 smithi042 ceph-mon[32775]: pgmap v4725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:06:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:25 smithi042 ceph-mon[32775]: pgmap v4726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:06:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:06:26.071+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:06:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:06:26.071+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:06:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:06:26.071+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:06:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7816.291437 secs 2024-07-26T09:06:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7834.033837 secs 2024-07-26T09:06:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7816.291437 secs 2024-07-26T09:06:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7834.033837 secs 2024-07-26T09:06:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7816.291437 secs 2024-07-26T09:06:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7834.033837 secs 2024-07-26T09:06:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:27 smithi163 ceph-mon[36873]: pgmap v4727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:06:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:27 smithi042 ceph-mon[32775]: pgmap v4727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:06:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:27 smithi164 ceph-mon[36970]: pgmap v4727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:06:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:29 smithi163 ceph-mon[36873]: pgmap v4728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:06:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:29 smithi042 ceph-mon[32775]: pgmap v4728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:06:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:29 smithi164 ceph-mon[36970]: pgmap v4728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:06:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:31 smithi163 ceph-mon[36873]: pgmap v4729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:06:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7821.291525 secs 2024-07-26T09:06:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7839.033946 secs 2024-07-26T09:06:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:31 smithi042 ceph-mon[32775]: pgmap v4729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:06:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7821.291525 secs 2024-07-26T09:06:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7839.033946 secs 2024-07-26T09:06:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:31 smithi164 ceph-mon[36970]: pgmap v4729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:06:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7821.291525 secs 2024-07-26T09:06:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7839.033946 secs 2024-07-26T09:06:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:33 smithi163 ceph-mon[36873]: pgmap v4730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:06:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:33 smithi042 ceph-mon[32775]: pgmap v4730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:06:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:33 smithi164 ceph-mon[36970]: pgmap v4730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:06:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:35 smithi163 ceph-mon[36873]: pgmap v4731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:06:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:35 smithi042 ceph-mon[32775]: pgmap v4731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:06:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:35 smithi164 ceph-mon[36970]: pgmap v4731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:06:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7826.291603 secs 2024-07-26T09:06:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7844.034075 secs 2024-07-26T09:06:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7826.291603 secs 2024-07-26T09:06:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7844.034075 secs 2024-07-26T09:06:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7826.291603 secs 2024-07-26T09:06:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7844.034075 secs 2024-07-26T09:06:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:37 smithi163 ceph-mon[36873]: pgmap v4732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:37 smithi042 ceph-mon[32775]: pgmap v4732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:37 smithi164 ceph-mon[36970]: pgmap v4732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:39 smithi163 ceph-mon[36873]: pgmap v4733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:39 smithi042 ceph-mon[32775]: pgmap v4733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:39 smithi164 ceph-mon[36970]: pgmap v4733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:06:41.081+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:06:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:06:41.081+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:06:41.083 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:06:41.081+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:06:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:41 smithi163 ceph-mon[36873]: pgmap v4734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7831.291698 secs 2024-07-26T09:06:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7849.034180 secs 2024-07-26T09:06:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:41 smithi042 ceph-mon[32775]: pgmap v4734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7831.291698 secs 2024-07-26T09:06:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7849.034180 secs 2024-07-26T09:06:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:41 smithi164 ceph-mon[36970]: pgmap v4734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7831.291698 secs 2024-07-26T09:06:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7849.034180 secs 2024-07-26T09:06:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:43 smithi163 ceph-mon[36873]: pgmap v4735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:43.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:43 smithi042 ceph-mon[32775]: pgmap v4735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:43 smithi164 ceph-mon[36970]: pgmap v4735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:45.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:45 smithi164 ceph-mon[36970]: pgmap v4736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:45 smithi042 ceph-mon[32775]: pgmap v4736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:45 smithi163 ceph-mon[36873]: pgmap v4736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7836.291802 secs 2024-07-26T09:06:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7854.034299 secs 2024-07-26T09:06:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7836.291802 secs 2024-07-26T09:06:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7854.034299 secs 2024-07-26T09:06:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7836.291802 secs 2024-07-26T09:06:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7854.034299 secs 2024-07-26T09:06:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:47 smithi042 ceph-mon[32775]: pgmap v4737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:47 smithi164 ceph-mon[36970]: pgmap v4737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:47 smithi163 ceph-mon[36873]: pgmap v4737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:49 smithi042 ceph-mon[32775]: pgmap v4738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:49 smithi164 ceph-mon[36970]: pgmap v4738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:49 smithi163 ceph-mon[36873]: pgmap v4738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:51 smithi042 ceph-mon[32775]: pgmap v4739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7841.291883 secs 2024-07-26T09:06:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7859.034419 secs 2024-07-26T09:06:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:51 smithi164 ceph-mon[36970]: pgmap v4739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7841.291883 secs 2024-07-26T09:06:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7859.034419 secs 2024-07-26T09:06:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:51 smithi163 ceph-mon[36873]: pgmap v4739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7841.291883 secs 2024-07-26T09:06:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7859.034419 secs 2024-07-26T09:06:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:53 smithi042 ceph-mon[32775]: pgmap v4740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:53 smithi164 ceph-mon[36970]: pgmap v4740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:53 smithi163 ceph-mon[36873]: pgmap v4740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:54 smithi042 ceph-mon[32775]: pgmap v4741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:54 smithi164 ceph-mon[36970]: pgmap v4741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:54 smithi163 ceph-mon[36873]: pgmap v4741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:06:56.085+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:06:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:06:56.085+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:06:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:06:56.085+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:06:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:55 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7846.292033 secs 2024-07-26T09:06:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:55 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7846.292033 secs 2024-07-26T09:06:56.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:55 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7846.292033 secs 2024-07-26T09:06:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:56 smithi164 ceph-mon[36970]: pgmap v4742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7864.034542 secs 2024-07-26T09:06:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:56 smithi163 ceph-mon[36873]: pgmap v4742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7864.034542 secs 2024-07-26T09:06:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:56 smithi042 ceph-mon[32775]: pgmap v4742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:06:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7864.034542 secs 2024-07-26T09:06:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:06:59 smithi042 ceph-mon[32775]: pgmap v4743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:06:59 smithi164 ceph-mon[36970]: pgmap v4743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:06:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:06:59 smithi163 ceph-mon[36873]: pgmap v4743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:01 smithi164 ceph-mon[36970]: pgmap v4744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7851.292165 secs 2024-07-26T09:07:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:01 smithi163 ceph-mon[36873]: pgmap v4744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7851.292165 secs 2024-07-26T09:07:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:01 smithi042 ceph-mon[32775]: pgmap v4744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7851.292165 secs 2024-07-26T09:07:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7869.034674 secs 2024-07-26T09:07:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7869.034674 secs 2024-07-26T09:07:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7869.034674 secs 2024-07-26T09:07:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:03 smithi164 ceph-mon[36970]: pgmap v4745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:03 smithi163 ceph-mon[36873]: pgmap v4745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:03 smithi042 ceph-mon[32775]: pgmap v4745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:05 smithi164 ceph-mon[36970]: pgmap v4746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:05 smithi163 ceph-mon[36873]: pgmap v4746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:05 smithi042 ceph-mon[32775]: pgmap v4746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7856.292278 secs 2024-07-26T09:07:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7856.292278 secs 2024-07-26T09:07:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7856.292278 secs 2024-07-26T09:07:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:07 smithi164 ceph-mon[36970]: pgmap v4747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7874.034818 secs 2024-07-26T09:07:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:07 smithi163 ceph-mon[36873]: pgmap v4747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7874.034818 secs 2024-07-26T09:07:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:07 smithi042 ceph-mon[32775]: pgmap v4747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7874.034818 secs 2024-07-26T09:07:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:09 smithi164 ceph-mon[36970]: pgmap v4748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:09 smithi163 ceph-mon[36873]: pgmap v4748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:09 smithi042 ceph-mon[32775]: pgmap v4748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:11.096 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:11.095+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:07:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:11.095+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:07:11.097 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:11.095+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:07:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:11 smithi164 ceph-mon[36970]: pgmap v4749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7861.292344 secs 2024-07-26T09:07:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:11 smithi163 ceph-mon[36873]: pgmap v4749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7861.292344 secs 2024-07-26T09:07:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:11 smithi042 ceph-mon[32775]: pgmap v4749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7861.292344 secs 2024-07-26T09:07:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7879.034920 secs 2024-07-26T09:07:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7879.034920 secs 2024-07-26T09:07:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7879.034920 secs 2024-07-26T09:07:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:13 smithi164 ceph-mon[36970]: pgmap v4750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:13 smithi163 ceph-mon[36873]: pgmap v4750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:13 smithi042 ceph-mon[32775]: pgmap v4750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:15 smithi042 ceph-mon[32775]: pgmap v4751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:15 smithi164 ceph-mon[36970]: pgmap v4751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:15 smithi163 ceph-mon[36873]: pgmap v4751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7866.292481 secs 2024-07-26T09:07:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7866.292481 secs 2024-07-26T09:07:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7866.292481 secs 2024-07-26T09:07:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:17 smithi164 ceph-mon[36970]: pgmap v4752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7884.035041 secs 2024-07-26T09:07:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:17 smithi163 ceph-mon[36873]: pgmap v4752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7884.035041 secs 2024-07-26T09:07:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:17 smithi042 ceph-mon[32775]: pgmap v4752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7884.035041 secs 2024-07-26T09:07:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:19 smithi164 ceph-mon[36970]: pgmap v4753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:19 smithi163 ceph-mon[36873]: pgmap v4753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:19 smithi042 ceph-mon[32775]: pgmap v4753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:21 smithi164 ceph-mon[36970]: pgmap v4754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:21.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7871.292603 secs 2024-07-26T09:07:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:21 smithi163 ceph-mon[36873]: pgmap v4754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7871.292603 secs 2024-07-26T09:07:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:21 smithi042 ceph-mon[32775]: pgmap v4754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7871.292603 secs 2024-07-26T09:07:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7889.035155 secs 2024-07-26T09:07:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7889.035155 secs 2024-07-26T09:07:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7889.035155 secs 2024-07-26T09:07:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:23 smithi164 ceph-mon[36970]: pgmap v4755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:23 smithi163 ceph-mon[36873]: pgmap v4755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:23 smithi042 ceph-mon[32775]: pgmap v4755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:07:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:07:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:07:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:25 smithi164 ceph-mon[36970]: pgmap v4756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:07:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:07:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:07:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:25 smithi163 ceph-mon[36873]: pgmap v4756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:07:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:07:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:07:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:25 smithi042 ceph-mon[32775]: pgmap v4756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:07:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:07:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:07:26.101 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:26.100+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:07:26.101 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:26.100+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:07:26.102 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:26.100+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:07:26.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:26 smithi164 ceph-mon[36970]: pgmap v4757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T09:07:26.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:26 smithi164 ceph-mon[36970]: pgmap v4758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:07:26.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7876.292696 secs 2024-07-26T09:07:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:26 smithi163 ceph-mon[36873]: pgmap v4757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T09:07:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:26 smithi163 ceph-mon[36873]: pgmap v4758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:07:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7876.292696 secs 2024-07-26T09:07:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:26 smithi042 ceph-mon[32775]: pgmap v4757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T09:07:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:26 smithi042 ceph-mon[32775]: pgmap v4758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:07:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7876.292696 secs 2024-07-26T09:07:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7894.035284 secs 2024-07-26T09:07:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7894.035284 secs 2024-07-26T09:07:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7894.035284 secs 2024-07-26T09:07:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:28 smithi163 ceph-mon[36873]: pgmap v4759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:07:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:28 smithi042 ceph-mon[32775]: pgmap v4759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:07:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:28 smithi164 ceph-mon[36970]: pgmap v4759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:07:30.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:30 smithi163 ceph-mon[36873]: pgmap v4760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:07:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:30 smithi042 ceph-mon[32775]: pgmap v4760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:07:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:30 smithi164 ceph-mon[36970]: pgmap v4760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:07:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7881.292841 secs 2024-07-26T09:07:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7881.292841 secs 2024-07-26T09:07:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7881.292841 secs 2024-07-26T09:07:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:32 smithi163 ceph-mon[36873]: pgmap v4761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:07:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7899.035410 secs 2024-07-26T09:07:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:32 smithi042 ceph-mon[32775]: pgmap v4761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:07:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7899.035410 secs 2024-07-26T09:07:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:32 smithi164 ceph-mon[36970]: pgmap v4761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:07:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7899.035410 secs 2024-07-26T09:07:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:34 smithi163 ceph-mon[36873]: pgmap v4762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:07:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:34 smithi042 ceph-mon[32775]: pgmap v4762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:07:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:34 smithi164 ceph-mon[36970]: pgmap v4762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:07:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:36 smithi163 ceph-mon[36873]: pgmap v4763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:07:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7886.292965 secs 2024-07-26T09:07:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:36 smithi042 ceph-mon[32775]: pgmap v4763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:07:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7886.292965 secs 2024-07-26T09:07:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:36 smithi164 ceph-mon[36970]: pgmap v4763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:07:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7886.292965 secs 2024-07-26T09:07:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7904.035531 secs 2024-07-26T09:07:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7904.035531 secs 2024-07-26T09:07:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7904.035531 secs 2024-07-26T09:07:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:38 smithi163 ceph-mon[36873]: pgmap v4764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:38 smithi042 ceph-mon[32775]: pgmap v4764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:38 smithi164 ceph-mon[36970]: pgmap v4764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:40 smithi163 ceph-mon[36873]: pgmap v4765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:07:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:40 smithi042 ceph-mon[32775]: pgmap v4765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:07:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:40 smithi164 ceph-mon[36970]: pgmap v4765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:07:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:41.109+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:07:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:41.109+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:07:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:41.109+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:07:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7891.293098 secs 2024-07-26T09:07:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7909.035661 secs 2024-07-26T09:07:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7891.293098 secs 2024-07-26T09:07:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7909.035661 secs 2024-07-26T09:07:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7891.293098 secs 2024-07-26T09:07:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7909.035661 secs 2024-07-26T09:07:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:42 smithi042 ceph-mon[32775]: pgmap v4766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:42 smithi164 ceph-mon[36970]: pgmap v4766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:42 smithi163 ceph-mon[36873]: pgmap v4766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:44 smithi042 ceph-mon[32775]: pgmap v4767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:44 smithi164 ceph-mon[36970]: pgmap v4767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:44 smithi163 ceph-mon[36873]: pgmap v4767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:46.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:46 smithi164 ceph-mon[36970]: pgmap v4768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:46.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7896.293259 secs 2024-07-26T09:07:46.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7914.035782 secs 2024-07-26T09:07:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:46 smithi042 ceph-mon[32775]: pgmap v4768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7896.293259 secs 2024-07-26T09:07:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7914.035782 secs 2024-07-26T09:07:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:46 smithi163 ceph-mon[36873]: pgmap v4768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7896.293259 secs 2024-07-26T09:07:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7914.035782 secs 2024-07-26T09:07:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:48 smithi042 ceph-mon[32775]: pgmap v4769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:48 smithi164 ceph-mon[36970]: pgmap v4769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:48 smithi163 ceph-mon[36873]: pgmap v4769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:50 smithi042 ceph-mon[32775]: pgmap v4770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:07:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:50 smithi164 ceph-mon[36970]: pgmap v4770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:07:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:50 smithi163 ceph-mon[36873]: pgmap v4770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:07:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7901.293385 secs 2024-07-26T09:07:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7919.035919 secs 2024-07-26T09:07:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7901.293385 secs 2024-07-26T09:07:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7919.035919 secs 2024-07-26T09:07:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7901.293385 secs 2024-07-26T09:07:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7919.035919 secs 2024-07-26T09:07:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:52 smithi042 ceph-mon[32775]: pgmap v4771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:52 smithi164 ceph-mon[36970]: pgmap v4771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:52 smithi163 ceph-mon[36873]: pgmap v4771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:54 smithi042 ceph-mon[32775]: pgmap v4772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:54 smithi164 ceph-mon[36970]: pgmap v4772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:54 smithi163 ceph-mon[36873]: pgmap v4772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:56.115 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:56.113+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:07:56.115 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:56.113+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:07:56.115 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:56.113+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:07:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:56 smithi042 ceph-mon[32775]: pgmap v4773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7906.293518 secs 2024-07-26T09:07:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7924.036036 secs 2024-07-26T09:07:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:56 smithi164 ceph-mon[36970]: pgmap v4773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7906.293518 secs 2024-07-26T09:07:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7924.036036 secs 2024-07-26T09:07:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:56 smithi163 ceph-mon[36873]: pgmap v4773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:07:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7906.293518 secs 2024-07-26T09:07:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7924.036036 secs 2024-07-26T09:07:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:07:58 smithi042 ceph-mon[32775]: pgmap v4774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:07:58 smithi164 ceph-mon[36970]: pgmap v4774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:07:58 smithi163 ceph-mon[36873]: pgmap v4774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:07:59.086 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:59.084+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:07:59.086 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:59.084+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:07:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:59.084+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:07:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:59.084+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:07:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:59.084+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:07:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:59.084+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006630 con 0x7ff680102000 2024-07-26T09:07:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:59.084+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:07:59.087 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:07:59.084+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006630 con 0x7ff680102000 2024-07-26T09:08:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:00 smithi042 ceph-mon[32775]: pgmap v4775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:08:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:00 smithi164 ceph-mon[36970]: pgmap v4775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:08:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:00 smithi163 ceph-mon[36873]: pgmap v4775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:08:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7911.293656 secs 2024-07-26T09:08:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7929.036162 secs 2024-07-26T09:08:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7911.293656 secs 2024-07-26T09:08:01.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7929.036162 secs 2024-07-26T09:08:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7911.293656 secs 2024-07-26T09:08:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7929.036162 secs 2024-07-26T09:08:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:02 smithi042 ceph-mon[32775]: pgmap v4776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:02 smithi164 ceph-mon[36970]: pgmap v4776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:02 smithi163 ceph-mon[36873]: pgmap v4776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:04 smithi042 ceph-mon[32775]: pgmap v4777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:04 smithi164 ceph-mon[36970]: pgmap v4777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:04 smithi163 ceph-mon[36873]: pgmap v4777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:06 smithi042 ceph-mon[32775]: pgmap v4778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7916.293786 secs 2024-07-26T09:08:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7934.036288 secs 2024-07-26T09:08:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:06 smithi164 ceph-mon[36970]: pgmap v4778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7916.293786 secs 2024-07-26T09:08:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7934.036288 secs 2024-07-26T09:08:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:06 smithi163 ceph-mon[36873]: pgmap v4778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7916.293786 secs 2024-07-26T09:08:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7934.036288 secs 2024-07-26T09:08:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:08 smithi042 ceph-mon[32775]: pgmap v4779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:08 smithi164 ceph-mon[36970]: pgmap v4779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:08 smithi163 ceph-mon[36873]: pgmap v4779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:10 smithi042 ceph-mon[32775]: pgmap v4780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:08:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:10 smithi164 ceph-mon[36970]: pgmap v4780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:08:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:10 smithi163 ceph-mon[36873]: pgmap v4780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:08:11.125 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:08:11.123+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:08:11.125 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:08:11.123+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:08:11.125 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:08:11.123+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:08:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7921.293946 secs 2024-07-26T09:08:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7939.036410 secs 2024-07-26T09:08:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7921.293946 secs 2024-07-26T09:08:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7939.036410 secs 2024-07-26T09:08:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7921.293946 secs 2024-07-26T09:08:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7939.036410 secs 2024-07-26T09:08:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:12 smithi042 ceph-mon[32775]: pgmap v4781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:12 smithi164 ceph-mon[36970]: pgmap v4781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:12 smithi163 ceph-mon[36873]: pgmap v4781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:14 smithi042 ceph-mon[32775]: pgmap v4782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:14 smithi164 ceph-mon[36970]: pgmap v4782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:14 smithi163 ceph-mon[36873]: pgmap v4782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:16 smithi042 ceph-mon[32775]: pgmap v4783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7926.294074 secs 2024-07-26T09:08:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7944.036532 secs 2024-07-26T09:08:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:16 smithi164 ceph-mon[36970]: pgmap v4783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7926.294074 secs 2024-07-26T09:08:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7944.036532 secs 2024-07-26T09:08:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:16 smithi163 ceph-mon[36873]: pgmap v4783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7926.294074 secs 2024-07-26T09:08:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7944.036532 secs 2024-07-26T09:08:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:18 smithi042 ceph-mon[32775]: pgmap v4784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:18 smithi164 ceph-mon[36970]: pgmap v4784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:18 smithi163 ceph-mon[36873]: pgmap v4784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:20 smithi042 ceph-mon[32775]: pgmap v4785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:08:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:20 smithi164 ceph-mon[36970]: pgmap v4785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:08:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:20 smithi163 ceph-mon[36873]: pgmap v4785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:08:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7931.294199 secs 2024-07-26T09:08:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7949.036655 secs 2024-07-26T09:08:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7931.294199 secs 2024-07-26T09:08:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7949.036655 secs 2024-07-26T09:08:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7931.294199 secs 2024-07-26T09:08:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7949.036655 secs 2024-07-26T09:08:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:22 smithi042 ceph-mon[32775]: pgmap v4786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:22 smithi164 ceph-mon[36970]: pgmap v4786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:22 smithi163 ceph-mon[36873]: pgmap v4786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:24 smithi042 ceph-mon[32775]: pgmap v4787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:24 smithi164 ceph-mon[36970]: pgmap v4787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:24 smithi163 ceph-mon[36873]: pgmap v4787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:08:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:08:25.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:08:25.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:08:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:08:25.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:08:25.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:08:25.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:08:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:08:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:08:25.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:08:25.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:08:25.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:08:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:08:26.128+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:08:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:08:26.128+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:08:26.130 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:08:26.128+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:08:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:26 smithi042 ceph-mon[32775]: pgmap v4788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:26 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:08:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:26 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:08:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:26 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:08:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:26 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:08:26.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:26 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:08:26.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:26 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:08:26.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:26 smithi042 ceph-mon[32775]: pgmap v4789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:08:26.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:26 smithi042 ceph-mon[32775]: pgmap v4790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:08:26.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7936.294349 secs 2024-07-26T09:08:26.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7954.036778 secs 2024-07-26T09:08:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:26 smithi164 ceph-mon[36970]: pgmap v4788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:26 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:08:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:26 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:08:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:26 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:08:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:26 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:08:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:26 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:08:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:26 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:08:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:26 smithi164 ceph-mon[36970]: pgmap v4789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:08:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:26 smithi164 ceph-mon[36970]: pgmap v4790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:08:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7936.294349 secs 2024-07-26T09:08:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7954.036778 secs 2024-07-26T09:08:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:26 smithi163 ceph-mon[36873]: pgmap v4788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:26 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:08:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:26 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:08:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:26 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:08:26.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:26 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:08:26.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:26 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:08:26.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:26 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:08:26.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:26 smithi163 ceph-mon[36873]: pgmap v4789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:08:26.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:26 smithi163 ceph-mon[36873]: pgmap v4790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:08:26.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7936.294349 secs 2024-07-26T09:08:26.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7954.036778 secs 2024-07-26T09:08:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:28 smithi042 ceph-mon[32775]: pgmap v4791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:08:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:28 smithi164 ceph-mon[36970]: pgmap v4791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:08:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:28 smithi163 ceph-mon[36873]: pgmap v4791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:08:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:30 smithi042 ceph-mon[32775]: pgmap v4792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:08:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:30 smithi164 ceph-mon[36970]: pgmap v4792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:08:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:30 smithi163 ceph-mon[36873]: pgmap v4792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:08:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7941.294464 secs 2024-07-26T09:08:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7959.036899 secs 2024-07-26T09:08:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7941.294464 secs 2024-07-26T09:08:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7959.036899 secs 2024-07-26T09:08:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7941.294464 secs 2024-07-26T09:08:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7959.036899 secs 2024-07-26T09:08:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:32 smithi042 ceph-mon[32775]: pgmap v4793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:08:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:32 smithi164 ceph-mon[36970]: pgmap v4793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:08:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:32 smithi163 ceph-mon[36873]: pgmap v4793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:08:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:34 smithi042 ceph-mon[32775]: pgmap v4794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:08:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:34 smithi164 ceph-mon[36970]: pgmap v4794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:08:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:34 smithi163 ceph-mon[36873]: pgmap v4794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:08:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:36 smithi042 ceph-mon[32775]: pgmap v4795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:08:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7946.294617 secs 2024-07-26T09:08:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7964.036994 secs 2024-07-26T09:08:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:36 smithi164 ceph-mon[36970]: pgmap v4795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:08:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7946.294617 secs 2024-07-26T09:08:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7964.036994 secs 2024-07-26T09:08:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:36 smithi163 ceph-mon[36873]: pgmap v4795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:08:36.780 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7946.294617 secs 2024-07-26T09:08:36.781 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7964.036994 secs 2024-07-26T09:08:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:38 smithi163 ceph-mon[36873]: pgmap v4796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:38 smithi042 ceph-mon[32775]: pgmap v4796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:38 smithi164 ceph-mon[36970]: pgmap v4796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:40 smithi163 ceph-mon[36873]: pgmap v4797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:40 smithi042 ceph-mon[32775]: pgmap v4797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:40.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:40 smithi164 ceph-mon[36970]: pgmap v4797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:41.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:08:41.138+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:08:41.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:08:41.138+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:08:41.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:08:41.138+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:08:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7951.294761 secs 2024-07-26T09:08:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7969.037112 secs 2024-07-26T09:08:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7951.294761 secs 2024-07-26T09:08:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7969.037112 secs 2024-07-26T09:08:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7951.294761 secs 2024-07-26T09:08:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7969.037112 secs 2024-07-26T09:08:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:42 smithi163 ceph-mon[36873]: pgmap v4798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:42 smithi042 ceph-mon[32775]: pgmap v4798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:42 smithi164 ceph-mon[36970]: pgmap v4798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:44 smithi163 ceph-mon[36873]: pgmap v4799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:44 smithi042 ceph-mon[32775]: pgmap v4799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:44 smithi164 ceph-mon[36970]: pgmap v4799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:46 smithi163 ceph-mon[36873]: pgmap v4800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7956.294869 secs 2024-07-26T09:08:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7974.037212 secs 2024-07-26T09:08:46.775 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:46 smithi164 ceph-mon[36970]: pgmap v4800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:46.775 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7956.294869 secs 2024-07-26T09:08:46.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7974.037212 secs 2024-07-26T09:08:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:46 smithi042 ceph-mon[32775]: pgmap v4800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7956.294869 secs 2024-07-26T09:08:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7974.037212 secs 2024-07-26T09:08:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:48 smithi163 ceph-mon[36873]: pgmap v4801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:48 smithi042 ceph-mon[32775]: pgmap v4801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:48 smithi164 ceph-mon[36970]: pgmap v4801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:50 smithi163 ceph-mon[36873]: pgmap v4802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:50 smithi042 ceph-mon[32775]: pgmap v4802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:50 smithi164 ceph-mon[36970]: pgmap v4802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7961.295001 secs 2024-07-26T09:08:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7979.037292 secs 2024-07-26T09:08:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7961.295001 secs 2024-07-26T09:08:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7979.037292 secs 2024-07-26T09:08:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7961.295001 secs 2024-07-26T09:08:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7979.037292 secs 2024-07-26T09:08:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:52 smithi042 ceph-mon[32775]: pgmap v4803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:52 smithi164 ceph-mon[36970]: pgmap v4803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:52 smithi163 ceph-mon[36873]: pgmap v4803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:54 smithi042 ceph-mon[32775]: pgmap v4804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:54 smithi164 ceph-mon[36970]: pgmap v4804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:54 smithi163 ceph-mon[36873]: pgmap v4804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:56.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:08:56.142+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:08:56.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:08:56.142+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:08:56.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:08:56.142+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:08:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:56 smithi042 ceph-mon[32775]: pgmap v4805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7966.295103 secs 2024-07-26T09:08:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7984.037413 secs 2024-07-26T09:08:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:56 smithi164 ceph-mon[36970]: pgmap v4805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7966.295103 secs 2024-07-26T09:08:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7984.037413 secs 2024-07-26T09:08:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:56 smithi163 ceph-mon[36873]: pgmap v4805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:08:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7966.295103 secs 2024-07-26T09:08:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7984.037413 secs 2024-07-26T09:08:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:08:58 smithi042 ceph-mon[32775]: pgmap v4806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:08:58 smithi164 ceph-mon[36970]: pgmap v4806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:08:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:08:58 smithi163 ceph-mon[36873]: pgmap v4806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:00 smithi042 ceph-mon[32775]: pgmap v4807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:00 smithi164 ceph-mon[36970]: pgmap v4807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:00 smithi163 ceph-mon[36873]: pgmap v4807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7971.295246 secs 2024-07-26T09:09:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7989.037536 secs 2024-07-26T09:09:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7971.295246 secs 2024-07-26T09:09:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7989.037536 secs 2024-07-26T09:09:02.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7971.295246 secs 2024-07-26T09:09:02.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7989.037536 secs 2024-07-26T09:09:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:02 smithi042 ceph-mon[32775]: pgmap v4808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:02 smithi164 ceph-mon[36970]: pgmap v4808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:02 smithi163 ceph-mon[36873]: pgmap v4808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:04 smithi042 ceph-mon[32775]: pgmap v4809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:04 smithi164 ceph-mon[36970]: pgmap v4809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:04 smithi163 ceph-mon[36873]: pgmap v4809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:06.810 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:06 smithi163 ceph-mon[36873]: pgmap v4810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:06.810 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7976.295347 secs 2024-07-26T09:09:06.810 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7994.037676 secs 2024-07-26T09:09:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:06 smithi042 ceph-mon[32775]: pgmap v4810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7976.295347 secs 2024-07-26T09:09:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7994.037676 secs 2024-07-26T09:09:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:06 smithi164 ceph-mon[36970]: pgmap v4810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7976.295347 secs 2024-07-26T09:09:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7994.037676 secs 2024-07-26T09:09:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:08 smithi042 ceph-mon[32775]: pgmap v4811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:08 smithi164 ceph-mon[36970]: pgmap v4811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:08 smithi163 ceph-mon[36873]: pgmap v4811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:10 smithi042 ceph-mon[32775]: pgmap v4812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:10 smithi164 ceph-mon[36970]: pgmap v4812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:10 smithi163 ceph-mon[36873]: pgmap v4812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:11.153 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:09:11.152+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:09:11.174 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:09:11.152+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:09:11.174 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:09:11.152+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:09:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7981.295477 secs 2024-07-26T09:09:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7999.037754 secs 2024-07-26T09:09:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7981.295477 secs 2024-07-26T09:09:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7999.037754 secs 2024-07-26T09:09:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7981.295477 secs 2024-07-26T09:09:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7999.037754 secs 2024-07-26T09:09:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:12 smithi042 ceph-mon[32775]: pgmap v4813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:12 smithi164 ceph-mon[36970]: pgmap v4813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:12 smithi163 ceph-mon[36873]: pgmap v4813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:13 smithi042 ceph-mon[32775]: pgmap v4814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:09:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:13 smithi164 ceph-mon[36970]: pgmap v4814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:09:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:13 smithi163 ceph-mon[36873]: pgmap v4814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:09:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:16 smithi042 ceph-mon[32775]: pgmap v4815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7986.295597 secs 2024-07-26T09:09:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8004.037842 secs 2024-07-26T09:09:16.676 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:16 smithi164 ceph-mon[36970]: pgmap v4815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:16.676 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7986.295597 secs 2024-07-26T09:09:16.676 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8004.037842 secs 2024-07-26T09:09:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:16 smithi163 ceph-mon[36873]: pgmap v4815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7986.295597 secs 2024-07-26T09:09:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8004.037842 secs 2024-07-26T09:09:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:18 smithi042 ceph-mon[32775]: pgmap v4816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:18 smithi164 ceph-mon[36970]: pgmap v4816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:18 smithi163 ceph-mon[36873]: pgmap v4816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:20 smithi042 ceph-mon[32775]: pgmap v4817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:20 smithi164 ceph-mon[36970]: pgmap v4817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:20 smithi163 ceph-mon[36873]: pgmap v4817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7991.295731 secs 2024-07-26T09:09:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8009.037968 secs 2024-07-26T09:09:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7991.295731 secs 2024-07-26T09:09:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8009.037968 secs 2024-07-26T09:09:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7991.295731 secs 2024-07-26T09:09:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8009.037968 secs 2024-07-26T09:09:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:22 smithi042 ceph-mon[32775]: pgmap v4818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:22 smithi164 ceph-mon[36970]: pgmap v4818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:22 smithi163 ceph-mon[36873]: pgmap v4818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:24 smithi042 ceph-mon[32775]: pgmap v4819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:09:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:24 smithi164 ceph-mon[36970]: pgmap v4819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:09:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:24 smithi163 ceph-mon[36873]: pgmap v4819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:09:26.158 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:09:26.157+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:09:26.159 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:09:26.157+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:09:26.159 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:09:26.157+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:09:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:26 smithi042 ceph-mon[32775]: pgmap v4820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:09:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:09:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:09:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:09:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 7996.295830 secs 2024-07-26T09:09:26.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8014.038079 secs 2024-07-26T09:09:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:26 smithi164 ceph-mon[36970]: pgmap v4820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:09:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:09:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:09:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:09:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 7996.295830 secs 2024-07-26T09:09:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8014.038079 secs 2024-07-26T09:09:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:26 smithi163 ceph-mon[36873]: pgmap v4820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:09:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:09:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:09:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:09:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 7996.295830 secs 2024-07-26T09:09:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8014.038079 secs 2024-07-26T09:09:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:27 smithi042 ceph-mon[32775]: pgmap v4821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:09:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:27 smithi042 ceph-mon[32775]: pgmap v4822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:09:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:27 smithi164 ceph-mon[36970]: pgmap v4821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:09:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:27 smithi164 ceph-mon[36970]: pgmap v4822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:09:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:27 smithi163 ceph-mon[36873]: pgmap v4821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:09:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:27 smithi163 ceph-mon[36873]: pgmap v4822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:09:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:29 smithi042 ceph-mon[32775]: pgmap v4823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:09:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:29 smithi164 ceph-mon[36970]: pgmap v4823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:09:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:29 smithi163 ceph-mon[36873]: pgmap v4823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:09:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:31 smithi042 ceph-mon[32775]: pgmap v4824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:09:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8001.295915 secs 2024-07-26T09:09:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8019.038201 secs 2024-07-26T09:09:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:31 smithi164 ceph-mon[36970]: pgmap v4824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:09:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8001.295915 secs 2024-07-26T09:09:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8019.038201 secs 2024-07-26T09:09:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:31 smithi163 ceph-mon[36873]: pgmap v4824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:09:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8001.295915 secs 2024-07-26T09:09:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8019.038201 secs 2024-07-26T09:09:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:33 smithi042 ceph-mon[32775]: pgmap v4825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:09:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:33 smithi164 ceph-mon[36970]: pgmap v4825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:09:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:33 smithi163 ceph-mon[36873]: pgmap v4825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:09:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:35 smithi042 ceph-mon[32775]: pgmap v4826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:09:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:35 smithi164 ceph-mon[36970]: pgmap v4826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:09:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:35 smithi163 ceph-mon[36873]: pgmap v4826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:09:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8006.296036 secs 2024-07-26T09:09:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8024.038299 secs 2024-07-26T09:09:36.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8006.296036 secs 2024-07-26T09:09:36.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8024.038299 secs 2024-07-26T09:09:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8006.296036 secs 2024-07-26T09:09:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8024.038299 secs 2024-07-26T09:09:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:37 smithi042 ceph-mon[32775]: pgmap v4827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:09:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:37 smithi164 ceph-mon[36970]: pgmap v4827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:09:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:37 smithi163 ceph-mon[36873]: pgmap v4827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:09:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:39 smithi042 ceph-mon[32775]: pgmap v4828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:39 smithi164 ceph-mon[36970]: pgmap v4828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:39 smithi163 ceph-mon[36873]: pgmap v4828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:41.168 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:09:41.166+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:09:41.168 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:09:41.166+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:09:41.168 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:09:41.166+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:09:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:41 smithi042 ceph-mon[32775]: pgmap v4829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8011.296161 secs 2024-07-26T09:09:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8029.038427 secs 2024-07-26T09:09:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:41 smithi164 ceph-mon[36970]: pgmap v4829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:41.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8011.296161 secs 2024-07-26T09:09:41.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8029.038427 secs 2024-07-26T09:09:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:41 smithi163 ceph-mon[36873]: pgmap v4829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8011.296161 secs 2024-07-26T09:09:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8029.038427 secs 2024-07-26T09:09:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:43 smithi042 ceph-mon[32775]: pgmap v4830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:43 smithi164 ceph-mon[36970]: pgmap v4830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:43 smithi163 ceph-mon[36873]: pgmap v4830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:45 smithi042 ceph-mon[32775]: pgmap v4831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:09:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:45 smithi164 ceph-mon[36970]: pgmap v4831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:09:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:45 smithi163 ceph-mon[36873]: pgmap v4831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:09:46.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8016.296308 secs 2024-07-26T09:09:46.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8034.038567 secs 2024-07-26T09:09:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8016.296308 secs 2024-07-26T09:09:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8034.038567 secs 2024-07-26T09:09:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8016.296308 secs 2024-07-26T09:09:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8034.038567 secs 2024-07-26T09:09:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:47 smithi042 ceph-mon[32775]: pgmap v4832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:47 smithi164 ceph-mon[36970]: pgmap v4832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:47 smithi163 ceph-mon[36873]: pgmap v4832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:49.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:49 smithi042 ceph-mon[32775]: pgmap v4833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:49 smithi164 ceph-mon[36970]: pgmap v4833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:49 smithi163 ceph-mon[36873]: pgmap v4833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:51 smithi042 ceph-mon[32775]: pgmap v4834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8021.296374 secs 2024-07-26T09:09:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8039.038710 secs 2024-07-26T09:09:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:51 smithi164 ceph-mon[36970]: pgmap v4834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:51.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8021.296374 secs 2024-07-26T09:09:51.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8039.038710 secs 2024-07-26T09:09:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:51 smithi163 ceph-mon[36873]: pgmap v4834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8021.296374 secs 2024-07-26T09:09:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8039.038710 secs 2024-07-26T09:09:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:53 smithi042 ceph-mon[32775]: pgmap v4835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:53 smithi164 ceph-mon[36970]: pgmap v4835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:53 smithi163 ceph-mon[36873]: pgmap v4835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:55 smithi042 ceph-mon[32775]: pgmap v4836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:09:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:55 smithi164 ceph-mon[36970]: pgmap v4836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:09:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:55 smithi163 ceph-mon[36873]: pgmap v4836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:09:56.172 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:09:56.170+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:09:56.172 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:09:56.170+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:09:56.172 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:09:56.170+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:09:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8026.296499 secs 2024-07-26T09:09:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8044.038833 secs 2024-07-26T09:09:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8026.296499 secs 2024-07-26T09:09:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8044.038833 secs 2024-07-26T09:09:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8026.296499 secs 2024-07-26T09:09:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8044.038833 secs 2024-07-26T09:09:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:57 smithi042 ceph-mon[32775]: pgmap v4837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:57 smithi164 ceph-mon[36970]: pgmap v4837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:57 smithi163 ceph-mon[36873]: pgmap v4837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:09:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:09:59 smithi042 ceph-mon[32775]: pgmap v4838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:09:59 smithi164 ceph-mon[36970]: pgmap v4838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:09:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:09:59 smithi163 ceph-mon[36873]: pgmap v4838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:10:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:10:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:10:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:01 smithi042 ceph-mon[32775]: pgmap v4839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8031.296621 secs 2024-07-26T09:10:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8049.038972 secs 2024-07-26T09:10:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:01 smithi164 ceph-mon[36970]: pgmap v4839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8031.296621 secs 2024-07-26T09:10:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8049.038972 secs 2024-07-26T09:10:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:01 smithi163 ceph-mon[36873]: pgmap v4839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8031.296621 secs 2024-07-26T09:10:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8049.038972 secs 2024-07-26T09:10:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:03 smithi042 ceph-mon[32775]: pgmap v4840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:03 smithi164 ceph-mon[36970]: pgmap v4840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:03 smithi163 ceph-mon[36873]: pgmap v4840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:05 smithi042 ceph-mon[32775]: pgmap v4841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:10:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:05 smithi164 ceph-mon[36970]: pgmap v4841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:10:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:05 smithi163 ceph-mon[36873]: pgmap v4841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:10:06.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8036.296748 secs 2024-07-26T09:10:06.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8054.039093 secs 2024-07-26T09:10:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8036.296748 secs 2024-07-26T09:10:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8054.039093 secs 2024-07-26T09:10:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8036.296748 secs 2024-07-26T09:10:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8054.039093 secs 2024-07-26T09:10:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:07 smithi164 ceph-mon[36970]: pgmap v4842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:07 smithi163 ceph-mon[36873]: pgmap v4842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:07 smithi042 ceph-mon[32775]: pgmap v4842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:09 smithi164 ceph-mon[36970]: pgmap v4843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:09 smithi163 ceph-mon[36873]: pgmap v4843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:09 smithi042 ceph-mon[32775]: pgmap v4843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:11.182 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:11.180+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:10:11.182 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:11.180+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:10:11.182 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:11.180+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:10:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:11 smithi164 ceph-mon[36970]: pgmap v4844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:11.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8041.296848 secs 2024-07-26T09:10:11.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8059.039216 secs 2024-07-26T09:10:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:11 smithi163 ceph-mon[36873]: pgmap v4844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8041.296848 secs 2024-07-26T09:10:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8059.039216 secs 2024-07-26T09:10:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:11 smithi042 ceph-mon[32775]: pgmap v4844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8041.296848 secs 2024-07-26T09:10:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8059.039216 secs 2024-07-26T09:10:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:13 smithi164 ceph-mon[36970]: pgmap v4845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:13 smithi163 ceph-mon[36873]: pgmap v4845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:13 smithi042 ceph-mon[32775]: pgmap v4845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:15 smithi164 ceph-mon[36970]: pgmap v4846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:10:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:15 smithi163 ceph-mon[36873]: pgmap v4846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:10:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:15 smithi042 ceph-mon[32775]: pgmap v4846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:10:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8046.296979 secs 2024-07-26T09:10:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8064.039343 secs 2024-07-26T09:10:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8046.296979 secs 2024-07-26T09:10:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8064.039343 secs 2024-07-26T09:10:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8046.296979 secs 2024-07-26T09:10:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8064.039343 secs 2024-07-26T09:10:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:17 smithi164 ceph-mon[36970]: pgmap v4847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:17 smithi163 ceph-mon[36873]: pgmap v4847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:17 smithi042 ceph-mon[32775]: pgmap v4847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:19 smithi164 ceph-mon[36970]: pgmap v4848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:19 smithi163 ceph-mon[36873]: pgmap v4848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:19 smithi042 ceph-mon[32775]: pgmap v4848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:21 smithi164 ceph-mon[36970]: pgmap v4849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8051.297103 secs 2024-07-26T09:10:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8069.039457 secs 2024-07-26T09:10:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:21 smithi163 ceph-mon[36873]: pgmap v4849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8051.297103 secs 2024-07-26T09:10:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8069.039457 secs 2024-07-26T09:10:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:21 smithi042 ceph-mon[32775]: pgmap v4849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8051.297103 secs 2024-07-26T09:10:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8069.039457 secs 2024-07-26T09:10:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:23 smithi164 ceph-mon[36970]: pgmap v4850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:23 smithi163 ceph-mon[36873]: pgmap v4850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:23 smithi042 ceph-mon[32775]: pgmap v4850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:25 smithi164 ceph-mon[36970]: pgmap v4851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:10:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:25 smithi163 ceph-mon[36873]: pgmap v4851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:10:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:25 smithi042 ceph-mon[32775]: pgmap v4851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:10:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:26.184+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:10:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:26.184+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:10:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:26.184+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:10:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8056.297252 secs 2024-07-26T09:10:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:10:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8074.039556 secs 2024-07-26T09:10:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:10:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:10:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:10:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8056.297252 secs 2024-07-26T09:10:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:10:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8074.039556 secs 2024-07-26T09:10:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:10:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:10:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:10:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8056.297252 secs 2024-07-26T09:10:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:10:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8074.039556 secs 2024-07-26T09:10:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:10:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:10:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:10:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:27 smithi164 ceph-mon[36970]: pgmap v4852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:27 smithi164 ceph-mon[36970]: pgmap v4853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:10:27.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:27 smithi164 ceph-mon[36970]: pgmap v4854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:10:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:27 smithi163 ceph-mon[36873]: pgmap v4852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:27 smithi163 ceph-mon[36873]: pgmap v4853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:10:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:27 smithi163 ceph-mon[36873]: pgmap v4854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:10:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:27 smithi042 ceph-mon[32775]: pgmap v4852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:27 smithi042 ceph-mon[32775]: pgmap v4853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:10:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:27 smithi042 ceph-mon[32775]: pgmap v4854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:10:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:29 smithi164 ceph-mon[36970]: pgmap v4855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:10:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:29 smithi163 ceph-mon[36873]: pgmap v4855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:10:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:29 smithi042 ceph-mon[32775]: pgmap v4855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:10:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:31 smithi164 ceph-mon[36970]: pgmap v4856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:10:31.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8061.297375 secs 2024-07-26T09:10:31.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8079.039679 secs 2024-07-26T09:10:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:31 smithi163 ceph-mon[36873]: pgmap v4856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:10:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8061.297375 secs 2024-07-26T09:10:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8079.039679 secs 2024-07-26T09:10:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:31 smithi042 ceph-mon[32775]: pgmap v4856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:10:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8061.297375 secs 2024-07-26T09:10:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8079.039679 secs 2024-07-26T09:10:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:33 smithi164 ceph-mon[36970]: pgmap v4857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:10:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:33 smithi163 ceph-mon[36873]: pgmap v4857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:10:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:33 smithi042 ceph-mon[32775]: pgmap v4857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:10:35.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:35 smithi164 ceph-mon[36970]: pgmap v4858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:10:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:35 smithi163 ceph-mon[36873]: pgmap v4858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:10:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:35 smithi042 ceph-mon[32775]: pgmap v4858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:10:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8066.297498 secs 2024-07-26T09:10:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8084.039780 secs 2024-07-26T09:10:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8066.297498 secs 2024-07-26T09:10:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8084.039780 secs 2024-07-26T09:10:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8066.297498 secs 2024-07-26T09:10:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8084.039780 secs 2024-07-26T09:10:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:37 smithi042 ceph-mon[32775]: pgmap v4859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:10:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:37 smithi164 ceph-mon[36970]: pgmap v4859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:10:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:37 smithi163 ceph-mon[36873]: pgmap v4859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:10:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:39 smithi042 ceph-mon[32775]: pgmap v4860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:10:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:39 smithi164 ceph-mon[36970]: pgmap v4860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:10:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:39 smithi163 ceph-mon[36873]: pgmap v4860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:10:41.192 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:41.190+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:10:41.192 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:41.190+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:10:41.192 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:41.190+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:10:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:41 smithi042 ceph-mon[32775]: pgmap v4861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8071.297615 secs 2024-07-26T09:10:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8089.039885 secs 2024-07-26T09:10:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:41 smithi164 ceph-mon[36970]: pgmap v4861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8071.297615 secs 2024-07-26T09:10:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8089.039885 secs 2024-07-26T09:10:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:41 smithi163 ceph-mon[36873]: pgmap v4861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8071.297615 secs 2024-07-26T09:10:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8089.039885 secs 2024-07-26T09:10:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:43 smithi042 ceph-mon[32775]: pgmap v4862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:43 smithi164 ceph-mon[36970]: pgmap v4862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:44.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:43 smithi163 ceph-mon[36873]: pgmap v4862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:45 smithi042 ceph-mon[32775]: pgmap v4863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:46.015 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:45 smithi163 ceph-mon[36873]: pgmap v4863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:46.015 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:45 smithi164 ceph-mon[36970]: pgmap v4863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:46.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8076.297739 secs 2024-07-26T09:10:46.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8094.040012 secs 2024-07-26T09:10:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8076.297739 secs 2024-07-26T09:10:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8094.040012 secs 2024-07-26T09:10:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8076.297739 secs 2024-07-26T09:10:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8094.040012 secs 2024-07-26T09:10:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:47 smithi042 ceph-mon[32775]: pgmap v4864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:47 smithi164 ceph-mon[36970]: pgmap v4864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:47 smithi163 ceph-mon[36873]: pgmap v4864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:49 smithi042 ceph-mon[32775]: pgmap v4865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:10:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:49 smithi164 ceph-mon[36970]: pgmap v4865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:10:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:49 smithi163 ceph-mon[36873]: pgmap v4865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:10:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:51 smithi042 ceph-mon[32775]: pgmap v4866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8081.297807 secs 2024-07-26T09:10:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8099.040134 secs 2024-07-26T09:10:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:51 smithi164 ceph-mon[36970]: pgmap v4866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8081.297807 secs 2024-07-26T09:10:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8099.040134 secs 2024-07-26T09:10:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:51 smithi163 ceph-mon[36873]: pgmap v4866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8081.297807 secs 2024-07-26T09:10:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8099.040134 secs 2024-07-26T09:10:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:53 smithi042 ceph-mon[32775]: pgmap v4867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:53 smithi164 ceph-mon[36970]: pgmap v4867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:53 smithi163 ceph-mon[36873]: pgmap v4867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:55 smithi042 ceph-mon[32775]: pgmap v4868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:55 smithi164 ceph-mon[36970]: pgmap v4868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:55 smithi163 ceph-mon[36873]: pgmap v4868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:10:56.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:56.194+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:10:56.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:56.194+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:10:56.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:56.194+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:10:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8086.297954 secs 2024-07-26T09:10:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8104.040220 secs 2024-07-26T09:10:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8086.297954 secs 2024-07-26T09:10:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8104.040220 secs 2024-07-26T09:10:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8086.297954 secs 2024-07-26T09:10:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8104.040220 secs 2024-07-26T09:10:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:57.874+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0b4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T09:10:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:57.874+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0b4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T09:10:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:57.874+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b4840 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff678031570 tx=0x7ff67804b040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:10:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:57.874+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b4840 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff678031570 tx=0x7ff67804b040 comp rx=0 tx=0).stop 2024-07-26T09:10:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:57.874+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0b4840 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T09:10:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:57.874+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 msgr2=0x7ff66c0b4840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T09:10:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:57.874+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b3bd0 0x7ff66c0b4840 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T09:10:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:57.874+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T09:10:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:57.874+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:10:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:57.874+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff6780094b0 tx=0x7ff67806c040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T09:10:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:57 smithi042 ceph-mon[32775]: pgmap v4869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:57 smithi164 ceph-mon[36970]: pgmap v4869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:57 smithi163 ceph-mon[36873]: pgmap v4869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:10:59.114 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:59.113+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064f0 con 0x7ff680102000 2024-07-26T09:10:59.115 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:59.113+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:10:59.115 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:59.113+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064f0 con 0x7ff680102000 2024-07-26T09:10:59.115 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:10:59.113+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:10:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:10:59 smithi042 ceph-mon[32775]: pgmap v4870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:10:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:10:59 smithi164 ceph-mon[36970]: pgmap v4870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:11:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:10:59 smithi163 ceph-mon[36873]: pgmap v4870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:11:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:01 smithi042 ceph-mon[32775]: pgmap v4871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8091.298101 secs 2024-07-26T09:11:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8109.040345 secs 2024-07-26T09:11:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:01 smithi164 ceph-mon[36970]: pgmap v4871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8091.298101 secs 2024-07-26T09:11:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8109.040345 secs 2024-07-26T09:11:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:01 smithi163 ceph-mon[36873]: pgmap v4871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8091.298101 secs 2024-07-26T09:11:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8109.040345 secs 2024-07-26T09:11:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:03 smithi042 ceph-mon[32775]: pgmap v4872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:03 smithi164 ceph-mon[36970]: pgmap v4872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:03 smithi163 ceph-mon[36873]: pgmap v4872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:05 smithi042 ceph-mon[32775]: pgmap v4873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:05 smithi164 ceph-mon[36970]: pgmap v4873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:05 smithi163 ceph-mon[36873]: pgmap v4873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8096.298242 secs 2024-07-26T09:11:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8114.040465 secs 2024-07-26T09:11:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:06 smithi042 ceph-mon[32775]: pgmap v4874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:06.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8096.298242 secs 2024-07-26T09:11:06.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8114.040465 secs 2024-07-26T09:11:06.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:06 smithi163 ceph-mon[36873]: pgmap v4874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8096.298242 secs 2024-07-26T09:11:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8114.040465 secs 2024-07-26T09:11:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:06 smithi164 ceph-mon[36970]: pgmap v4874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:09 smithi164 ceph-mon[36970]: pgmap v4875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:11:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:09 smithi163 ceph-mon[36873]: pgmap v4875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:11:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:09 smithi042 ceph-mon[32775]: pgmap v4875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:11:11.206 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:11:11.204+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:11:11.206 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:11:11.204+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:11:11.206 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:11:11.204+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:11:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:11 smithi164 ceph-mon[36970]: pgmap v4876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8101.298362 secs 2024-07-26T09:11:11.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8119.040605 secs 2024-07-26T09:11:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:11 smithi163 ceph-mon[36873]: pgmap v4876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8101.298362 secs 2024-07-26T09:11:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8119.040605 secs 2024-07-26T09:11:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:11 smithi042 ceph-mon[32775]: pgmap v4876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8101.298362 secs 2024-07-26T09:11:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8119.040605 secs 2024-07-26T09:11:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:13 smithi164 ceph-mon[36970]: pgmap v4877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:13 smithi163 ceph-mon[36873]: pgmap v4877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:13 smithi042 ceph-mon[32775]: pgmap v4877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:15 smithi164 ceph-mon[36970]: pgmap v4878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:15 smithi163 ceph-mon[36873]: pgmap v4878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:15 smithi042 ceph-mon[32775]: pgmap v4878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8106.298445 secs 2024-07-26T09:11:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8124.040746 secs 2024-07-26T09:11:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8106.298445 secs 2024-07-26T09:11:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8124.040746 secs 2024-07-26T09:11:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8106.298445 secs 2024-07-26T09:11:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8124.040746 secs 2024-07-26T09:11:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:17 smithi164 ceph-mon[36970]: pgmap v4879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:17 smithi163 ceph-mon[36873]: pgmap v4879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:17 smithi042 ceph-mon[32775]: pgmap v4879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:19 smithi164 ceph-mon[36970]: pgmap v4880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:11:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:19 smithi163 ceph-mon[36873]: pgmap v4880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:11:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:19 smithi042 ceph-mon[32775]: pgmap v4880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:11:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:21 smithi164 ceph-mon[36970]: pgmap v4881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8111.298545 secs 2024-07-26T09:11:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8129.040889 secs 2024-07-26T09:11:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:21 smithi163 ceph-mon[36873]: pgmap v4881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8111.298545 secs 2024-07-26T09:11:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8129.040889 secs 2024-07-26T09:11:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:21 smithi042 ceph-mon[32775]: pgmap v4881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8111.298545 secs 2024-07-26T09:11:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8129.040889 secs 2024-07-26T09:11:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:23 smithi164 ceph-mon[36970]: pgmap v4882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:23 smithi163 ceph-mon[36873]: pgmap v4882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:23 smithi042 ceph-mon[32775]: pgmap v4882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:25 smithi164 ceph-mon[36970]: pgmap v4883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:25 smithi163 ceph-mon[36873]: pgmap v4883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:25 smithi042 ceph-mon[32775]: pgmap v4883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:11:26.209+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:11:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:11:26.209+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:11:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:11:26.209+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:11:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8116.298672 secs 2024-07-26T09:11:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8134.041015 secs 2024-07-26T09:11:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8116.298672 secs 2024-07-26T09:11:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8134.041015 secs 2024-07-26T09:11:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8116.298672 secs 2024-07-26T09:11:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8134.041015 secs 2024-07-26T09:11:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:27 smithi164 ceph-mon[36970]: pgmap v4884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:11:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:27 smithi163 ceph-mon[36873]: pgmap v4884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:11:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:27 smithi042 ceph-mon[32775]: pgmap v4884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:11:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:29 smithi042 ceph-mon[32775]: pgmap v4885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:11:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:11:29.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:11:29.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:29 smithi164 ceph-mon[36970]: pgmap v4885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:11:29.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:11:29.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:11:29.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:29 smithi163 ceph-mon[36873]: pgmap v4885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:11:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:29.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:11:29.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:11:29.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:11:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:30 smithi042 ceph-mon[32775]: pgmap v4886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:11:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:30 smithi042 ceph-mon[32775]: pgmap v4887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:11:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:30 smithi164 ceph-mon[36970]: pgmap v4886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:11:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:30 smithi164 ceph-mon[36970]: pgmap v4887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:11:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:30 smithi163 ceph-mon[36873]: pgmap v4886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:11:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:30 smithi163 ceph-mon[36873]: pgmap v4887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:11:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8121.298816 secs 2024-07-26T09:11:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8139.041122 secs 2024-07-26T09:11:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8121.298816 secs 2024-07-26T09:11:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8139.041122 secs 2024-07-26T09:11:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8121.298816 secs 2024-07-26T09:11:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8139.041122 secs 2024-07-26T09:11:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:32 smithi042 ceph-mon[32775]: pgmap v4888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:11:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:32 smithi164 ceph-mon[36970]: pgmap v4888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:11:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:32 smithi163 ceph-mon[36873]: pgmap v4888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:11:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:34 smithi042 ceph-mon[32775]: pgmap v4889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:11:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:34 smithi164 ceph-mon[36970]: pgmap v4889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:11:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:34 smithi163 ceph-mon[36873]: pgmap v4889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:11:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:36 smithi042 ceph-mon[32775]: pgmap v4890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:11:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8126.298938 secs 2024-07-26T09:11:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8144.041236 secs 2024-07-26T09:11:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:36 smithi164 ceph-mon[36970]: pgmap v4890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:11:36.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8126.298938 secs 2024-07-26T09:11:36.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8144.041236 secs 2024-07-26T09:11:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:36 smithi163 ceph-mon[36873]: pgmap v4890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:11:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8126.298938 secs 2024-07-26T09:11:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8144.041236 secs 2024-07-26T09:11:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:38 smithi042 ceph-mon[32775]: pgmap v4891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:11:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:38 smithi164 ceph-mon[36970]: pgmap v4891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:11:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:38 smithi163 ceph-mon[36873]: pgmap v4891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:11:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:40 smithi042 ceph-mon[32775]: pgmap v4892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:11:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:40 smithi164 ceph-mon[36970]: pgmap v4892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:11:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:40 smithi163 ceph-mon[36873]: pgmap v4892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:11:41.220 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:11:41.218+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:11:41.220 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:11:41.218+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:11:41.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:11:41.218+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:11:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8131.299032 secs 2024-07-26T09:11:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8149.041335 secs 2024-07-26T09:11:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8131.299032 secs 2024-07-26T09:11:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8149.041335 secs 2024-07-26T09:11:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8131.299032 secs 2024-07-26T09:11:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8149.041335 secs 2024-07-26T09:11:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:42 smithi042 ceph-mon[32775]: pgmap v4893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:42 smithi164 ceph-mon[36970]: pgmap v4893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:42 smithi163 ceph-mon[36873]: pgmap v4893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:44 smithi042 ceph-mon[32775]: pgmap v4894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:44 smithi164 ceph-mon[36970]: pgmap v4894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:44 smithi163 ceph-mon[36873]: pgmap v4894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:46 smithi042 ceph-mon[32775]: pgmap v4895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8136.299153 secs 2024-07-26T09:11:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8154.041428 secs 2024-07-26T09:11:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:46 smithi164 ceph-mon[36970]: pgmap v4895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8136.299153 secs 2024-07-26T09:11:46.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8154.041428 secs 2024-07-26T09:11:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:46 smithi163 ceph-mon[36873]: pgmap v4895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8136.299153 secs 2024-07-26T09:11:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8154.041428 secs 2024-07-26T09:11:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:48 smithi042 ceph-mon[32775]: pgmap v4896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:48 smithi164 ceph-mon[36970]: pgmap v4896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:48 smithi163 ceph-mon[36873]: pgmap v4896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:50 smithi042 ceph-mon[32775]: pgmap v4897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:11:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:50 smithi164 ceph-mon[36970]: pgmap v4897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:11:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:50 smithi163 ceph-mon[36873]: pgmap v4897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:11:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8141.299280 secs 2024-07-26T09:11:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8159.041559 secs 2024-07-26T09:11:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8141.299280 secs 2024-07-26T09:11:51.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8159.041559 secs 2024-07-26T09:11:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8141.299280 secs 2024-07-26T09:11:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8159.041559 secs 2024-07-26T09:11:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:52 smithi042 ceph-mon[32775]: pgmap v4898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:52 smithi164 ceph-mon[36970]: pgmap v4898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:52 smithi163 ceph-mon[36873]: pgmap v4898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:54 smithi164 ceph-mon[36970]: pgmap v4899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:54 smithi163 ceph-mon[36873]: pgmap v4899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:54 smithi042 ceph-mon[32775]: pgmap v4899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:11:56.222+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:11:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:11:56.222+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:11:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:11:56.222+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:11:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:56 smithi164 ceph-mon[36970]: pgmap v4900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:56.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8146.299409 secs 2024-07-26T09:11:56.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8164.041679 secs 2024-07-26T09:11:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:56 smithi163 ceph-mon[36873]: pgmap v4900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8146.299409 secs 2024-07-26T09:11:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8164.041679 secs 2024-07-26T09:11:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:56 smithi042 ceph-mon[32775]: pgmap v4900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:11:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8146.299409 secs 2024-07-26T09:11:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8164.041679 secs 2024-07-26T09:11:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:11:58 smithi164 ceph-mon[36970]: pgmap v4901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:11:58 smithi163 ceph-mon[36873]: pgmap v4901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:11:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:11:58 smithi042 ceph-mon[32775]: pgmap v4901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:00 smithi164 ceph-mon[36970]: pgmap v4902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:12:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:00 smithi163 ceph-mon[36873]: pgmap v4902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:12:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:00 smithi042 ceph-mon[32775]: pgmap v4902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:12:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8151.299534 secs 2024-07-26T09:12:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8169.041800 secs 2024-07-26T09:12:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8151.299534 secs 2024-07-26T09:12:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8169.041800 secs 2024-07-26T09:12:01.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8151.299534 secs 2024-07-26T09:12:01.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8169.041800 secs 2024-07-26T09:12:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:02 smithi164 ceph-mon[36970]: pgmap v4903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:02 smithi163 ceph-mon[36873]: pgmap v4903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:02 smithi042 ceph-mon[32775]: pgmap v4903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:04 smithi164 ceph-mon[36970]: pgmap v4904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:04 smithi163 ceph-mon[36873]: pgmap v4904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:04 smithi042 ceph-mon[32775]: pgmap v4904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:06 smithi164 ceph-mon[36970]: pgmap v4905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8156.299658 secs 2024-07-26T09:12:06.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8174.041918 secs 2024-07-26T09:12:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:06 smithi163 ceph-mon[36873]: pgmap v4905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8156.299658 secs 2024-07-26T09:12:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8174.041918 secs 2024-07-26T09:12:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:06 smithi042 ceph-mon[32775]: pgmap v4905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8156.299658 secs 2024-07-26T09:12:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8174.041918 secs 2024-07-26T09:12:08.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:08 smithi164 ceph-mon[36970]: pgmap v4906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:08 smithi163 ceph-mon[36873]: pgmap v4906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:08 smithi042 ceph-mon[32775]: pgmap v4906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:10 smithi164 ceph-mon[36970]: pgmap v4907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:12:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:10 smithi163 ceph-mon[36873]: pgmap v4907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:12:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:10 smithi042 ceph-mon[32775]: pgmap v4907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:12:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:12:11.232+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:12:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:12:11.232+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:12:11.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:12:11.232+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:12:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8161.299786 secs 2024-07-26T09:12:11.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8179.042032 secs 2024-07-26T09:12:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8161.299786 secs 2024-07-26T09:12:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8179.042032 secs 2024-07-26T09:12:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8161.299786 secs 2024-07-26T09:12:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8179.042032 secs 2024-07-26T09:12:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:12 smithi164 ceph-mon[36970]: pgmap v4908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:12 smithi163 ceph-mon[36873]: pgmap v4908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:12 smithi042 ceph-mon[32775]: pgmap v4908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:14 smithi164 ceph-mon[36970]: pgmap v4909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:14 smithi163 ceph-mon[36873]: pgmap v4909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:14 smithi042 ceph-mon[32775]: pgmap v4909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:16 smithi164 ceph-mon[36970]: pgmap v4910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:16.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8166.299909 secs 2024-07-26T09:12:16.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8184.042152 secs 2024-07-26T09:12:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:16 smithi163 ceph-mon[36873]: pgmap v4910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8166.299909 secs 2024-07-26T09:12:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8184.042152 secs 2024-07-26T09:12:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:16 smithi042 ceph-mon[32775]: pgmap v4910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8166.299909 secs 2024-07-26T09:12:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8184.042152 secs 2024-07-26T09:12:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:18 smithi164 ceph-mon[36970]: pgmap v4911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:18 smithi163 ceph-mon[36873]: pgmap v4911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:18 smithi042 ceph-mon[32775]: pgmap v4911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:20 smithi164 ceph-mon[36970]: pgmap v4912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:12:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:20 smithi163 ceph-mon[36873]: pgmap v4912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:12:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:20 smithi042 ceph-mon[32775]: pgmap v4912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:12:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8171.300034 secs 2024-07-26T09:12:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8189.042292 secs 2024-07-26T09:12:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8171.300034 secs 2024-07-26T09:12:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8189.042292 secs 2024-07-26T09:12:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8171.300034 secs 2024-07-26T09:12:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8189.042292 secs 2024-07-26T09:12:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:22 smithi164 ceph-mon[36970]: pgmap v4913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:22 smithi163 ceph-mon[36873]: pgmap v4913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:22 smithi042 ceph-mon[32775]: pgmap v4913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:24 smithi164 ceph-mon[36970]: pgmap v4914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:24 smithi163 ceph-mon[36873]: pgmap v4914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:24.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:24 smithi042 ceph-mon[32775]: pgmap v4914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:12:26.236+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:12:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:12:26.236+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:12:26.239 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:12:26.236+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:12:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:26 smithi164 ceph-mon[36970]: pgmap v4915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8176.300142 secs 2024-07-26T09:12:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8194.042413 secs 2024-07-26T09:12:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:26 smithi163 ceph-mon[36873]: pgmap v4915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8176.300142 secs 2024-07-26T09:12:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8194.042413 secs 2024-07-26T09:12:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:26 smithi042 ceph-mon[32775]: pgmap v4915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8176.300142 secs 2024-07-26T09:12:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8194.042413 secs 2024-07-26T09:12:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:28 smithi164 ceph-mon[36970]: pgmap v4916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:28 smithi163 ceph-mon[36873]: pgmap v4916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:28 smithi042 ceph-mon[32775]: pgmap v4916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:12:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:29.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:12:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:12:29.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:29.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:30 smithi163 ceph-mon[36873]: pgmap v4917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:12:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:12:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:12:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:30 smithi042 ceph-mon[32775]: pgmap v4917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:12:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:12:30.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:12:30.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:30 smithi164 ceph-mon[36970]: pgmap v4917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:30.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:12:30.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:12:30.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:12:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:31 smithi163 ceph-mon[36873]: pgmap v4918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-07-26T09:12:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:31 smithi163 ceph-mon[36873]: pgmap v4919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:12:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8181.300303 secs 2024-07-26T09:12:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8199.042559 secs 2024-07-26T09:12:31.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:31 smithi042 ceph-mon[32775]: pgmap v4918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-07-26T09:12:31.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:31 smithi042 ceph-mon[32775]: pgmap v4919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:12:31.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8181.300303 secs 2024-07-26T09:12:31.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8199.042559 secs 2024-07-26T09:12:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:31 smithi164 ceph-mon[36970]: pgmap v4918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-07-26T09:12:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:31 smithi164 ceph-mon[36970]: pgmap v4919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:12:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8181.300303 secs 2024-07-26T09:12:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8199.042559 secs 2024-07-26T09:12:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:33 smithi042 ceph-mon[32775]: pgmap v4920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:12:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:33 smithi164 ceph-mon[36970]: pgmap v4920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:12:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:33 smithi163 ceph-mon[36873]: pgmap v4920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:12:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:35 smithi042 ceph-mon[32775]: pgmap v4921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:12:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:35 smithi164 ceph-mon[36970]: pgmap v4921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:12:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:35 smithi163 ceph-mon[36873]: pgmap v4921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:12:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8186.300427 secs 2024-07-26T09:12:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8204.042682 secs 2024-07-26T09:12:36.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8186.300427 secs 2024-07-26T09:12:36.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8204.042682 secs 2024-07-26T09:12:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8186.300427 secs 2024-07-26T09:12:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8204.042682 secs 2024-07-26T09:12:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:37 smithi042 ceph-mon[32775]: pgmap v4922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:12:37.926 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:37 smithi163 ceph-mon[36873]: pgmap v4922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:12:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:37 smithi164 ceph-mon[36970]: pgmap v4922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:12:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:39 smithi042 ceph-mon[32775]: pgmap v4923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T09:12:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:39 smithi164 ceph-mon[36970]: pgmap v4923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T09:12:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:39 smithi163 ceph-mon[36873]: pgmap v4923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T09:12:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:12:41.246+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:12:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:12:41.246+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:12:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:12:41.246+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:12:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:41 smithi042 ceph-mon[32775]: pgmap v4924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:12:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8191.300550 secs 2024-07-26T09:12:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8209.042805 secs 2024-07-26T09:12:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:41 smithi164 ceph-mon[36970]: pgmap v4924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:12:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8191.300550 secs 2024-07-26T09:12:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8209.042805 secs 2024-07-26T09:12:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:41 smithi163 ceph-mon[36873]: pgmap v4924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:12:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8191.300550 secs 2024-07-26T09:12:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8209.042805 secs 2024-07-26T09:12:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:43 smithi042 ceph-mon[32775]: pgmap v4925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:43 smithi164 ceph-mon[36970]: pgmap v4925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:43 smithi163 ceph-mon[36873]: pgmap v4925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:45 smithi042 ceph-mon[32775]: pgmap v4926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:45 smithi164 ceph-mon[36970]: pgmap v4926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:45 smithi163 ceph-mon[36873]: pgmap v4926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:46.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8196.300676 secs 2024-07-26T09:12:46.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8214.042926 secs 2024-07-26T09:12:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8196.300676 secs 2024-07-26T09:12:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8214.042926 secs 2024-07-26T09:12:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8196.300676 secs 2024-07-26T09:12:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8214.042926 secs 2024-07-26T09:12:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:47 smithi042 ceph-mon[32775]: pgmap v4927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:47 smithi164 ceph-mon[36970]: pgmap v4927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:47 smithi163 ceph-mon[36873]: pgmap v4927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:49.583 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:49 smithi042 ceph-mon[32775]: pgmap v4928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:49 smithi164 ceph-mon[36970]: pgmap v4928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:49 smithi163 ceph-mon[36873]: pgmap v4928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:51 smithi042 ceph-mon[32775]: pgmap v4929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8201.300819 secs 2024-07-26T09:12:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8219.043046 secs 2024-07-26T09:12:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:51 smithi164 ceph-mon[36970]: pgmap v4929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8201.300819 secs 2024-07-26T09:12:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8219.043046 secs 2024-07-26T09:12:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:51 smithi163 ceph-mon[36873]: pgmap v4929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8201.300819 secs 2024-07-26T09:12:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8219.043046 secs 2024-07-26T09:12:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:52 smithi042 ceph-mon[32775]: pgmap v4930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:52 smithi164 ceph-mon[36970]: pgmap v4930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:52 smithi163 ceph-mon[36873]: pgmap v4930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:55 smithi163 ceph-mon[36873]: pgmap v4931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:55 smithi042 ceph-mon[32775]: pgmap v4931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:55 smithi164 ceph-mon[36970]: pgmap v4931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:12:56.252 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:12:56.251+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:12:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:12:56.251+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:12:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:12:56.251+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:12:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8206.300941 secs 2024-07-26T09:12:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8206.300941 secs 2024-07-26T09:12:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8206.300941 secs 2024-07-26T09:12:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:57 smithi163 ceph-mon[36873]: pgmap v4932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8224.043151 secs 2024-07-26T09:12:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:57 smithi042 ceph-mon[32775]: pgmap v4932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8224.043151 secs 2024-07-26T09:12:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:57 smithi164 ceph-mon[36970]: pgmap v4932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8224.043151 secs 2024-07-26T09:12:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:12:59 smithi163 ceph-mon[36873]: pgmap v4933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:12:59 smithi042 ceph-mon[32775]: pgmap v4933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:12:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:12:59 smithi164 ceph-mon[36970]: pgmap v4933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:01 smithi163 ceph-mon[36873]: pgmap v4934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8211.301073 secs 2024-07-26T09:13:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:01 smithi042 ceph-mon[32775]: pgmap v4934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8211.301073 secs 2024-07-26T09:13:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:01 smithi164 ceph-mon[36970]: pgmap v4934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8211.301073 secs 2024-07-26T09:13:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8229.043252 secs 2024-07-26T09:13:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8229.043252 secs 2024-07-26T09:13:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8229.043252 secs 2024-07-26T09:13:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:03 smithi163 ceph-mon[36873]: pgmap v4935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:03 smithi042 ceph-mon[32775]: pgmap v4935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:03 smithi164 ceph-mon[36970]: pgmap v4935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:05 smithi042 ceph-mon[32775]: pgmap v4936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:05 smithi164 ceph-mon[36970]: pgmap v4936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:05 smithi163 ceph-mon[36873]: pgmap v4936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8216.301225 secs 2024-07-26T09:13:06.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8216.301225 secs 2024-07-26T09:13:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8216.301225 secs 2024-07-26T09:13:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:07 smithi042 ceph-mon[32775]: pgmap v4937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8234.043390 secs 2024-07-26T09:13:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:07 smithi164 ceph-mon[36970]: pgmap v4937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:07.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8234.043390 secs 2024-07-26T09:13:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:07 smithi163 ceph-mon[36873]: pgmap v4937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8234.043390 secs 2024-07-26T09:13:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:09 smithi042 ceph-mon[32775]: pgmap v4938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:09 smithi164 ceph-mon[36970]: pgmap v4938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:09 smithi163 ceph-mon[36873]: pgmap v4938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:11.262 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:13:11.261+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:13:11.265 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:13:11.261+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:13:11.265 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:13:11.261+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:13:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:11 smithi042 ceph-mon[32775]: pgmap v4939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8221.301328 secs 2024-07-26T09:13:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:11 smithi164 ceph-mon[36970]: pgmap v4939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8221.301328 secs 2024-07-26T09:13:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:11 smithi163 ceph-mon[36873]: pgmap v4939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8221.301328 secs 2024-07-26T09:13:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8239.043497 secs 2024-07-26T09:13:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8239.043497 secs 2024-07-26T09:13:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8239.043497 secs 2024-07-26T09:13:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:13 smithi042 ceph-mon[32775]: pgmap v4940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:13 smithi164 ceph-mon[36970]: pgmap v4940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:13 smithi163 ceph-mon[36873]: pgmap v4940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:15 smithi042 ceph-mon[32775]: pgmap v4941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:15 smithi164 ceph-mon[36970]: pgmap v4941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:15 smithi163 ceph-mon[36873]: pgmap v4941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:16.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8226.301429 secs 2024-07-26T09:13:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8226.301429 secs 2024-07-26T09:13:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8226.301429 secs 2024-07-26T09:13:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:17 smithi042 ceph-mon[32775]: pgmap v4942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8244.043619 secs 2024-07-26T09:13:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:17 smithi164 ceph-mon[36970]: pgmap v4942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8244.043619 secs 2024-07-26T09:13:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:17 smithi163 ceph-mon[36873]: pgmap v4942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8244.043619 secs 2024-07-26T09:13:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:19 smithi042 ceph-mon[32775]: pgmap v4943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:19 smithi164 ceph-mon[36970]: pgmap v4943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:19 smithi163 ceph-mon[36873]: pgmap v4943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:21 smithi042 ceph-mon[32775]: pgmap v4944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8231.301560 secs 2024-07-26T09:13:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:21 smithi164 ceph-mon[36970]: pgmap v4944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8231.301560 secs 2024-07-26T09:13:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:21 smithi163 ceph-mon[36873]: pgmap v4944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8231.301560 secs 2024-07-26T09:13:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8249.043684 secs 2024-07-26T09:13:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8249.043684 secs 2024-07-26T09:13:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8249.043684 secs 2024-07-26T09:13:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:23 smithi042 ceph-mon[32775]: pgmap v4945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:23 smithi164 ceph-mon[36970]: pgmap v4945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:23 smithi163 ceph-mon[36873]: pgmap v4945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:25 smithi042 ceph-mon[32775]: pgmap v4946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:25 smithi164 ceph-mon[36970]: pgmap v4946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:25 smithi163 ceph-mon[36873]: pgmap v4946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:26.267 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:13:26.265+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:13:26.267 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:13:26.265+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:13:26.268 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:13:26.265+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:13:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8236.301684 secs 2024-07-26T09:13:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8254.043806 secs 2024-07-26T09:13:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8236.301684 secs 2024-07-26T09:13:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8254.043806 secs 2024-07-26T09:13:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8236.301684 secs 2024-07-26T09:13:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8254.043806 secs 2024-07-26T09:13:27.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:27 smithi042 ceph-mon[32775]: pgmap v4947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:27 smithi164 ceph-mon[36970]: pgmap v4947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:27 smithi163 ceph-mon[36873]: pgmap v4947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:29 smithi042 ceph-mon[32775]: pgmap v4948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:29 smithi164 ceph-mon[36970]: pgmap v4948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:29 smithi163 ceph-mon[36873]: pgmap v4948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:13:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:13:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:13:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:31 smithi042 ceph-mon[32775]: pgmap v4949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:13:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:13:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:13:31.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8241.301807 secs 2024-07-26T09:13:31.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8259.043922 secs 2024-07-26T09:13:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:31 smithi164 ceph-mon[36970]: pgmap v4949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:31.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:13:31.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:13:31.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:13:31.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8241.301807 secs 2024-07-26T09:13:31.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8259.043922 secs 2024-07-26T09:13:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:31 smithi163 ceph-mon[36873]: pgmap v4949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:13:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:13:31.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:13:31.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8241.301807 secs 2024-07-26T09:13:31.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8259.043922 secs 2024-07-26T09:13:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:32 smithi042 ceph-mon[32775]: pgmap v4950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:13:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:32 smithi042 ceph-mon[32775]: pgmap v4951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:13:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:32 smithi164 ceph-mon[36970]: pgmap v4950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:13:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:32 smithi164 ceph-mon[36970]: pgmap v4951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:13:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:32 smithi163 ceph-mon[36873]: pgmap v4950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:13:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:32 smithi163 ceph-mon[36873]: pgmap v4951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:13:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:34 smithi042 ceph-mon[32775]: pgmap v4952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:13:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:34 smithi164 ceph-mon[36970]: pgmap v4952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:13:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:34 smithi163 ceph-mon[36873]: pgmap v4952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:13:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:36 smithi042 ceph-mon[32775]: pgmap v4953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:13:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8246.301923 secs 2024-07-26T09:13:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8264.044038 secs 2024-07-26T09:13:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:36 smithi164 ceph-mon[36970]: pgmap v4953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:13:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8246.301923 secs 2024-07-26T09:13:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8264.044038 secs 2024-07-26T09:13:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:36 smithi163 ceph-mon[36873]: pgmap v4953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:13:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8246.301923 secs 2024-07-26T09:13:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8264.044038 secs 2024-07-26T09:13:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:38 smithi042 ceph-mon[32775]: pgmap v4954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:13:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:38 smithi164 ceph-mon[36970]: pgmap v4954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:13:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:38 smithi163 ceph-mon[36873]: pgmap v4954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:13:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:40 smithi042 ceph-mon[32775]: pgmap v4955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:13:40.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:40 smithi164 ceph-mon[36970]: pgmap v4955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:13:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:40 smithi163 ceph-mon[36873]: pgmap v4955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:13:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:13:41.275+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:13:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:13:41.275+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:13:41.277 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:13:41.275+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:13:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8251.302066 secs 2024-07-26T09:13:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8269.044161 secs 2024-07-26T09:13:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8251.302066 secs 2024-07-26T09:13:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8269.044161 secs 2024-07-26T09:13:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8251.302066 secs 2024-07-26T09:13:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8269.044161 secs 2024-07-26T09:13:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:42 smithi042 ceph-mon[32775]: pgmap v4956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:13:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:42 smithi164 ceph-mon[36970]: pgmap v4956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:13:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:42 smithi163 ceph-mon[36873]: pgmap v4956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:13:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:44 smithi042 ceph-mon[32775]: pgmap v4957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:44 smithi164 ceph-mon[36970]: pgmap v4957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:44 smithi163 ceph-mon[36873]: pgmap v4957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:46 smithi042 ceph-mon[32775]: pgmap v4958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8256.302186 secs 2024-07-26T09:13:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8274.044284 secs 2024-07-26T09:13:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:46 smithi164 ceph-mon[36970]: pgmap v4958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8256.302186 secs 2024-07-26T09:13:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8274.044284 secs 2024-07-26T09:13:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:46 smithi163 ceph-mon[36873]: pgmap v4958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8256.302186 secs 2024-07-26T09:13:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8274.044284 secs 2024-07-26T09:13:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:48 smithi042 ceph-mon[32775]: pgmap v4959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:48 smithi164 ceph-mon[36970]: pgmap v4959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:48 smithi163 ceph-mon[36873]: pgmap v4959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:50 smithi042 ceph-mon[32775]: pgmap v4960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:13:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:50 smithi164 ceph-mon[36970]: pgmap v4960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:13:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:50 smithi163 ceph-mon[36873]: pgmap v4960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:13:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8261.302314 secs 2024-07-26T09:13:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8279.044406 secs 2024-07-26T09:13:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8261.302314 secs 2024-07-26T09:13:51.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8279.044406 secs 2024-07-26T09:13:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8261.302314 secs 2024-07-26T09:13:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8279.044406 secs 2024-07-26T09:13:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:52 smithi042 ceph-mon[32775]: pgmap v4961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:52 smithi164 ceph-mon[36970]: pgmap v4961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:52 smithi163 ceph-mon[36873]: pgmap v4961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:54 smithi042 ceph-mon[32775]: pgmap v4962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:54 smithi164 ceph-mon[36970]: pgmap v4962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:54 smithi163 ceph-mon[36873]: pgmap v4962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:13:56.280+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:13:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:13:56.280+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:13:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:13:56.280+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:13:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:56 smithi042 ceph-mon[32775]: pgmap v4963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8266.302406 secs 2024-07-26T09:13:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8284.044537 secs 2024-07-26T09:13:56.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:56 smithi164 ceph-mon[36970]: pgmap v4963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:56.761 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8266.302406 secs 2024-07-26T09:13:56.762 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8284.044537 secs 2024-07-26T09:13:56.763 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:56 smithi163 ceph-mon[36873]: pgmap v4963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:13:56.764 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8266.302406 secs 2024-07-26T09:13:56.764 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8284.044537 secs 2024-07-26T09:13:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:13:58 smithi042 ceph-mon[32775]: pgmap v4964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:13:58 smithi164 ceph-mon[36970]: pgmap v4964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:13:58 smithi163 ceph-mon[36873]: pgmap v4964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:13:59.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:13:59.141+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:13:59.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:13:59.141+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064f0 con 0x7ff680102000 2024-07-26T09:13:59.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:13:59.142+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064f0 con 0x7ff680102000 2024-07-26T09:13:59.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:13:59.142+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:14:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:00 smithi042 ceph-mon[32775]: pgmap v4965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:14:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:00 smithi164 ceph-mon[36970]: pgmap v4965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:14:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:00 smithi163 ceph-mon[36873]: pgmap v4965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:14:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8271.302535 secs 2024-07-26T09:14:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8289.044662 secs 2024-07-26T09:14:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8271.302535 secs 2024-07-26T09:14:01.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8289.044662 secs 2024-07-26T09:14:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8271.302535 secs 2024-07-26T09:14:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8289.044662 secs 2024-07-26T09:14:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:02 smithi042 ceph-mon[32775]: pgmap v4966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:02 smithi164 ceph-mon[36970]: pgmap v4966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:02 smithi163 ceph-mon[36873]: pgmap v4966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:04 smithi164 ceph-mon[36970]: pgmap v4967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:04 smithi163 ceph-mon[36873]: pgmap v4967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:04 smithi042 ceph-mon[32775]: pgmap v4967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:06 smithi164 ceph-mon[36970]: pgmap v4968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:06.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8276.302633 secs 2024-07-26T09:14:06.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8294.044756 secs 2024-07-26T09:14:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:06 smithi163 ceph-mon[36873]: pgmap v4968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8276.302633 secs 2024-07-26T09:14:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8294.044756 secs 2024-07-26T09:14:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:06 smithi042 ceph-mon[32775]: pgmap v4968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8276.302633 secs 2024-07-26T09:14:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8294.044756 secs 2024-07-26T09:14:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:08 smithi164 ceph-mon[36970]: pgmap v4969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:14:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:08 smithi163 ceph-mon[36873]: pgmap v4969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:14:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:08 smithi042 ceph-mon[32775]: pgmap v4969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:14:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:10 smithi164 ceph-mon[36970]: pgmap v4970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:14:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:10 smithi163 ceph-mon[36873]: pgmap v4970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:14:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:10 smithi042 ceph-mon[32775]: pgmap v4970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:14:11.292 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:14:11.290+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:14:11.292 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:14:11.290+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:14:11.292 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:14:11.290+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:14:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8281.302759 secs 2024-07-26T09:14:11.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8299.044874 secs 2024-07-26T09:14:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8281.302759 secs 2024-07-26T09:14:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8299.044874 secs 2024-07-26T09:14:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8281.302759 secs 2024-07-26T09:14:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8299.044874 secs 2024-07-26T09:14:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:12 smithi164 ceph-mon[36970]: pgmap v4971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:14:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:12 smithi163 ceph-mon[36873]: pgmap v4971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:14:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:12 smithi042 ceph-mon[32775]: pgmap v4971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:14:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:14 smithi164 ceph-mon[36970]: pgmap v4972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:14:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:14 smithi163 ceph-mon[36873]: pgmap v4972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:14:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:14 smithi042 ceph-mon[32775]: pgmap v4972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:14:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:16 smithi164 ceph-mon[36970]: pgmap v4973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:16.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8286.302897 secs 2024-07-26T09:14:16.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8304.044995 secs 2024-07-26T09:14:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:16 smithi163 ceph-mon[36873]: pgmap v4973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8286.302897 secs 2024-07-26T09:14:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8304.044995 secs 2024-07-26T09:14:16.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:16 smithi042 ceph-mon[32775]: pgmap v4973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:16.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8286.302897 secs 2024-07-26T09:14:16.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8304.044995 secs 2024-07-26T09:14:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:18 smithi164 ceph-mon[36970]: pgmap v4974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:18 smithi163 ceph-mon[36873]: pgmap v4974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:18 smithi042 ceph-mon[32775]: pgmap v4974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:20 smithi164 ceph-mon[36970]: pgmap v4975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:14:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:20 smithi163 ceph-mon[36873]: pgmap v4975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:14:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:20 smithi042 ceph-mon[32775]: pgmap v4975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:14:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8291.303069 secs 2024-07-26T09:14:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8309.045114 secs 2024-07-26T09:14:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8291.303069 secs 2024-07-26T09:14:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8309.045114 secs 2024-07-26T09:14:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8291.303069 secs 2024-07-26T09:14:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8309.045114 secs 2024-07-26T09:14:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:22 smithi164 ceph-mon[36970]: pgmap v4976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:22 smithi163 ceph-mon[36873]: pgmap v4976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:22 smithi042 ceph-mon[32775]: pgmap v4976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:24 smithi164 ceph-mon[36970]: pgmap v4977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:24 smithi163 ceph-mon[36873]: pgmap v4977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:24 smithi042 ceph-mon[32775]: pgmap v4977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:14:26.294+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:14:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:14:26.294+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:14:26.296 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:14:26.294+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:14:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:26 smithi164 ceph-mon[36970]: pgmap v4978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8296.303211 secs 2024-07-26T09:14:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8314.045178 secs 2024-07-26T09:14:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:26 smithi163 ceph-mon[36873]: pgmap v4978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8296.303211 secs 2024-07-26T09:14:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8314.045178 secs 2024-07-26T09:14:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:26 smithi042 ceph-mon[32775]: pgmap v4978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8296.303211 secs 2024-07-26T09:14:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8314.045178 secs 2024-07-26T09:14:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:28 smithi164 ceph-mon[36970]: pgmap v4979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:28 smithi163 ceph-mon[36873]: pgmap v4979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:28 smithi042 ceph-mon[32775]: pgmap v4979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:30 smithi164 ceph-mon[36970]: pgmap v4980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:14:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:30 smithi163 ceph-mon[36873]: pgmap v4980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:14:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:30 smithi042 ceph-mon[32775]: pgmap v4980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:14:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:14:31.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8301.303292 secs 2024-07-26T09:14:31.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8319.045258 secs 2024-07-26T09:14:31.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:14:31.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:14:31.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:14:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:14:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8301.303292 secs 2024-07-26T09:14:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8319.045258 secs 2024-07-26T09:14:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:14:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:14:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:14:31.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:14:31.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8301.303292 secs 2024-07-26T09:14:31.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8319.045258 secs 2024-07-26T09:14:31.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:14:31.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:14:31.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:14:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:32 smithi164 ceph-mon[36970]: pgmap v4981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:32.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:32 smithi164 ceph-mon[36970]: pgmap v4982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:14:32.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:32 smithi164 ceph-mon[36970]: pgmap v4983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:32 smithi163 ceph-mon[36873]: pgmap v4981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:32.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:32 smithi163 ceph-mon[36873]: pgmap v4982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:14:32.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:32 smithi163 ceph-mon[36873]: pgmap v4983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:32 smithi042 ceph-mon[32775]: pgmap v4981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:32 smithi042 ceph-mon[32775]: pgmap v4982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:14:32.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:32 smithi042 ceph-mon[32775]: pgmap v4983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:34 smithi164 ceph-mon[36970]: pgmap v4984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:14:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:34 smithi163 ceph-mon[36873]: pgmap v4984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:14:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:34 smithi042 ceph-mon[32775]: pgmap v4984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:14:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:36 smithi164 ceph-mon[36970]: pgmap v4985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:36.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8306.303418 secs 2024-07-26T09:14:36.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8324.045380 secs 2024-07-26T09:14:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:36 smithi163 ceph-mon[36873]: pgmap v4985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8306.303418 secs 2024-07-26T09:14:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8324.045380 secs 2024-07-26T09:14:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:36 smithi042 ceph-mon[32775]: pgmap v4985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8306.303418 secs 2024-07-26T09:14:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8324.045380 secs 2024-07-26T09:14:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:38 smithi042 ceph-mon[32775]: pgmap v4986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:38 smithi164 ceph-mon[36970]: pgmap v4986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:38 smithi163 ceph-mon[36873]: pgmap v4986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:40 smithi042 ceph-mon[32775]: pgmap v4987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:40 smithi164 ceph-mon[36970]: pgmap v4987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:40 smithi163 ceph-mon[36873]: pgmap v4987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:14:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:14:41.304+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:14:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:14:41.304+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:14:41.306 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:14:41.304+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:14:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8311.303542 secs 2024-07-26T09:14:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8329.045519 secs 2024-07-26T09:14:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8311.303542 secs 2024-07-26T09:14:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8329.045519 secs 2024-07-26T09:14:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8311.303542 secs 2024-07-26T09:14:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8329.045519 secs 2024-07-26T09:14:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:42 smithi042 ceph-mon[32775]: pgmap v4988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:14:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:42 smithi164 ceph-mon[36970]: pgmap v4988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:14:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:42 smithi163 ceph-mon[36873]: pgmap v4988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:14:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:44 smithi042 ceph-mon[32775]: pgmap v4989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:14:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:44 smithi164 ceph-mon[36970]: pgmap v4989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:14:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:44 smithi163 ceph-mon[36873]: pgmap v4989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:14:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:46 smithi042 ceph-mon[32775]: pgmap v4990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8316.303669 secs 2024-07-26T09:14:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8334.045636 secs 2024-07-26T09:14:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:46 smithi164 ceph-mon[36970]: pgmap v4990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8316.303669 secs 2024-07-26T09:14:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8334.045636 secs 2024-07-26T09:14:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:46 smithi163 ceph-mon[36873]: pgmap v4990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8316.303669 secs 2024-07-26T09:14:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8334.045636 secs 2024-07-26T09:14:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:48 smithi042 ceph-mon[32775]: pgmap v4991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:48 smithi164 ceph-mon[36970]: pgmap v4991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:48 smithi163 ceph-mon[36873]: pgmap v4991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:49 smithi042 ceph-mon[32775]: pgmap v4992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:49 smithi164 ceph-mon[36970]: pgmap v4992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:49 smithi163 ceph-mon[36873]: pgmap v4992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:50 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8321.303807 secs 2024-07-26T09:14:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:50 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8321.303807 secs 2024-07-26T09:14:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:50 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8321.303807 secs 2024-07-26T09:14:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8339.045792 secs 2024-07-26T09:14:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:51 smithi163 ceph-mon[36873]: pgmap v4993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8339.045792 secs 2024-07-26T09:14:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:51 smithi042 ceph-mon[32775]: pgmap v4993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8339.045792 secs 2024-07-26T09:14:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:51 smithi164 ceph-mon[36970]: pgmap v4993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:54 smithi042 ceph-mon[32775]: pgmap v4994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:14:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:54 smithi164 ceph-mon[36970]: pgmap v4994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:14:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:54 smithi163 ceph-mon[36873]: pgmap v4994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:14:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:14:56.308+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:14:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:14:56.308+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:14:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:14:56.308+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:14:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:56 smithi042 ceph-mon[32775]: pgmap v4995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8326.303930 secs 2024-07-26T09:14:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8344.045857 secs 2024-07-26T09:14:56.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:56 smithi164 ceph-mon[36970]: pgmap v4995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:56.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8326.303930 secs 2024-07-26T09:14:56.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8344.045857 secs 2024-07-26T09:14:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:56 smithi163 ceph-mon[36873]: pgmap v4995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:14:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8326.303930 secs 2024-07-26T09:14:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8344.045857 secs 2024-07-26T09:14:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:14:58 smithi042 ceph-mon[32775]: pgmap v4996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:14:58 smithi164 ceph-mon[36970]: pgmap v4996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:14:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:14:58 smithi163 ceph-mon[36873]: pgmap v4996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:00 smithi042 ceph-mon[32775]: pgmap v4997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:00 smithi164 ceph-mon[36970]: pgmap v4997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:00 smithi163 ceph-mon[36873]: pgmap v4997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8331.304078 secs 2024-07-26T09:15:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8349.045986 secs 2024-07-26T09:15:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8331.304078 secs 2024-07-26T09:15:01.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8349.045986 secs 2024-07-26T09:15:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8331.304078 secs 2024-07-26T09:15:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8349.045986 secs 2024-07-26T09:15:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:02 smithi042 ceph-mon[32775]: pgmap v4998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:02 smithi164 ceph-mon[36970]: pgmap v4998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:02 smithi163 ceph-mon[36873]: pgmap v4998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:04 smithi042 ceph-mon[32775]: pgmap v4999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:15:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:04 smithi164 ceph-mon[36970]: pgmap v4999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:15:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:04 smithi163 ceph-mon[36873]: pgmap v4999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:15:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:06 smithi042 ceph-mon[32775]: pgmap v5000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8336.304239 secs 2024-07-26T09:15:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8354.046113 secs 2024-07-26T09:15:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:06 smithi164 ceph-mon[36970]: pgmap v5000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8336.304239 secs 2024-07-26T09:15:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8354.046113 secs 2024-07-26T09:15:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:06 smithi163 ceph-mon[36873]: pgmap v5000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8336.304239 secs 2024-07-26T09:15:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8354.046113 secs 2024-07-26T09:15:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:08 smithi042 ceph-mon[32775]: pgmap v5001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:08 smithi164 ceph-mon[36970]: pgmap v5001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:08 smithi163 ceph-mon[36873]: pgmap v5001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:10 smithi042 ceph-mon[32775]: pgmap v5002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:10 smithi164 ceph-mon[36970]: pgmap v5002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:10 smithi163 ceph-mon[36873]: pgmap v5002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:11.313 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:15:11.308+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:15:11.314 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:15:11.308+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:15:11.314 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:15:11.308+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:15:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8341.304342 secs 2024-07-26T09:15:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8359.046180 secs 2024-07-26T09:15:11.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8341.304342 secs 2024-07-26T09:15:11.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8359.046180 secs 2024-07-26T09:15:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8341.304342 secs 2024-07-26T09:15:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8359.046180 secs 2024-07-26T09:15:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:12 smithi042 ceph-mon[32775]: pgmap v5003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:12 smithi164 ceph-mon[36970]: pgmap v5003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:12 smithi163 ceph-mon[36873]: pgmap v5003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:14 smithi042 ceph-mon[32775]: pgmap v5004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:15:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:14 smithi164 ceph-mon[36970]: pgmap v5004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:15:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:14 smithi163 ceph-mon[36873]: pgmap v5004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:15:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:16 smithi042 ceph-mon[32775]: pgmap v5005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8346.304427 secs 2024-07-26T09:15:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8364.046298 secs 2024-07-26T09:15:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:16 smithi164 ceph-mon[36970]: pgmap v5005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:16.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8346.304427 secs 2024-07-26T09:15:16.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8364.046298 secs 2024-07-26T09:15:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:16 smithi163 ceph-mon[36873]: pgmap v5005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8346.304427 secs 2024-07-26T09:15:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8364.046298 secs 2024-07-26T09:15:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:18 smithi042 ceph-mon[32775]: pgmap v5006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:18 smithi164 ceph-mon[36970]: pgmap v5006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:18 smithi163 ceph-mon[36873]: pgmap v5006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:20 smithi164 ceph-mon[36970]: pgmap v5007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:20 smithi163 ceph-mon[36873]: pgmap v5007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:20 smithi042 ceph-mon[32775]: pgmap v5007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8351.304561 secs 2024-07-26T09:15:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8369.046423 secs 2024-07-26T09:15:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8351.304561 secs 2024-07-26T09:15:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8369.046423 secs 2024-07-26T09:15:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8351.304561 secs 2024-07-26T09:15:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8369.046423 secs 2024-07-26T09:15:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:22 smithi164 ceph-mon[36970]: pgmap v5008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:22 smithi163 ceph-mon[36873]: pgmap v5008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:22 smithi042 ceph-mon[32775]: pgmap v5008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:24 smithi164 ceph-mon[36970]: pgmap v5009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:15:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:24 smithi163 ceph-mon[36873]: pgmap v5009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:15:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:24 smithi042 ceph-mon[32775]: pgmap v5009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:15:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:15:26.312+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:15:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:15:26.312+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:15:26.315 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:15:26.312+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:15:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:26 smithi164 ceph-mon[36970]: pgmap v5010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8356.304687 secs 2024-07-26T09:15:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8374.046541 secs 2024-07-26T09:15:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:26 smithi163 ceph-mon[36873]: pgmap v5010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8356.304687 secs 2024-07-26T09:15:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8374.046541 secs 2024-07-26T09:15:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:26 smithi042 ceph-mon[32775]: pgmap v5010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8356.304687 secs 2024-07-26T09:15:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8374.046541 secs 2024-07-26T09:15:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:28 smithi164 ceph-mon[36970]: pgmap v5011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:28 smithi163 ceph-mon[36873]: pgmap v5011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:28.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:28 smithi042 ceph-mon[32775]: pgmap v5011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:30 smithi164 ceph-mon[36970]: pgmap v5012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:30 smithi163 ceph-mon[36873]: pgmap v5012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:30 smithi042 ceph-mon[32775]: pgmap v5012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8361.304821 secs 2024-07-26T09:15:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8379.046665 secs 2024-07-26T09:15:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:15:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8361.304821 secs 2024-07-26T09:15:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8379.046665 secs 2024-07-26T09:15:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:15:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8361.304821 secs 2024-07-26T09:15:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8379.046665 secs 2024-07-26T09:15:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:15:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:32 smithi164 ceph-mon[36970]: pgmap v5013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:32.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:15:32.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:15:32.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:15:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:32 smithi163 ceph-mon[36873]: pgmap v5013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:15:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:15:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:15:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:32 smithi042 ceph-mon[32775]: pgmap v5013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:15:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:15:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:15:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:33 smithi164 ceph-mon[36970]: pgmap v5014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:15:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:33 smithi164 ceph-mon[36970]: pgmap v5015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:15:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:33 smithi163 ceph-mon[36873]: pgmap v5014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:15:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:33 smithi163 ceph-mon[36873]: pgmap v5015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:15:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:33 smithi042 ceph-mon[32775]: pgmap v5014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:15:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:33 smithi042 ceph-mon[32775]: pgmap v5015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:15:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:35 smithi164 ceph-mon[36970]: pgmap v5016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:15:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:35 smithi163 ceph-mon[36873]: pgmap v5016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:15:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:35 smithi042 ceph-mon[32775]: pgmap v5016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:15:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8366.304966 secs 2024-07-26T09:15:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8384.046785 secs 2024-07-26T09:15:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8366.304966 secs 2024-07-26T09:15:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8384.046785 secs 2024-07-26T09:15:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8366.304966 secs 2024-07-26T09:15:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8384.046785 secs 2024-07-26T09:15:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:37 smithi164 ceph-mon[36970]: pgmap v5017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:15:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:37 smithi163 ceph-mon[36873]: pgmap v5017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:15:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:37 smithi042 ceph-mon[32775]: pgmap v5017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:15:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:39 smithi164 ceph-mon[36970]: pgmap v5018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:15:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:39 smithi163 ceph-mon[36873]: pgmap v5018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:15:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:39 smithi042 ceph-mon[32775]: pgmap v5018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:15:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:15:41.317+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:15:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:15:41.318+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:15:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:15:41.318+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:15:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:41 smithi164 ceph-mon[36970]: pgmap v5019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:15:41.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8371.305097 secs 2024-07-26T09:15:41.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8389.046906 secs 2024-07-26T09:15:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:41 smithi163 ceph-mon[36873]: pgmap v5019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:15:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8371.305097 secs 2024-07-26T09:15:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8389.046906 secs 2024-07-26T09:15:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:41 smithi042 ceph-mon[32775]: pgmap v5019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:15:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8371.305097 secs 2024-07-26T09:15:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8389.046906 secs 2024-07-26T09:15:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:43 smithi164 ceph-mon[36970]: pgmap v5020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:15:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:43 smithi163 ceph-mon[36873]: pgmap v5020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:15:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:43 smithi042 ceph-mon[32775]: pgmap v5020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:15:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:45 smithi164 ceph-mon[36970]: pgmap v5021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:15:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:45 smithi163 ceph-mon[36873]: pgmap v5021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:15:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:45 smithi042 ceph-mon[32775]: pgmap v5021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:15:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8376.305252 secs 2024-07-26T09:15:46.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8394.047049 secs 2024-07-26T09:15:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8376.305252 secs 2024-07-26T09:15:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8394.047049 secs 2024-07-26T09:15:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8376.305252 secs 2024-07-26T09:15:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8394.047049 secs 2024-07-26T09:15:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:47 smithi164 ceph-mon[36970]: pgmap v5022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:47 smithi163 ceph-mon[36873]: pgmap v5022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:47 smithi042 ceph-mon[32775]: pgmap v5022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:49 smithi164 ceph-mon[36970]: pgmap v5023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:49 smithi163 ceph-mon[36873]: pgmap v5023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:49 smithi042 ceph-mon[32775]: pgmap v5023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:51 smithi164 ceph-mon[36970]: pgmap v5024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8381.305352 secs 2024-07-26T09:15:51.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8399.047168 secs 2024-07-26T09:15:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:51 smithi163 ceph-mon[36873]: pgmap v5024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8381.305352 secs 2024-07-26T09:15:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8399.047168 secs 2024-07-26T09:15:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:51 smithi042 ceph-mon[32775]: pgmap v5024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8381.305352 secs 2024-07-26T09:15:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8399.047168 secs 2024-07-26T09:15:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:53 smithi163 ceph-mon[36873]: pgmap v5025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:53 smithi042 ceph-mon[32775]: pgmap v5025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:53 smithi164 ceph-mon[36970]: pgmap v5025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:55 smithi163 ceph-mon[36873]: pgmap v5026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:15:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:55 smithi042 ceph-mon[32775]: pgmap v5026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:15:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:55 smithi164 ceph-mon[36970]: pgmap v5026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:15:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:15:56.323+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:15:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:15:56.323+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:15:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:15:56.323+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:15:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8386.305481 secs 2024-07-26T09:15:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8404.047311 secs 2024-07-26T09:15:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8386.305481 secs 2024-07-26T09:15:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8404.047311 secs 2024-07-26T09:15:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8386.305481 secs 2024-07-26T09:15:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8404.047311 secs 2024-07-26T09:15:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:57 smithi163 ceph-mon[36873]: pgmap v5027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:57 smithi042 ceph-mon[32775]: pgmap v5027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:57 smithi164 ceph-mon[36970]: pgmap v5027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:15:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:15:59 smithi163 ceph-mon[36873]: pgmap v5028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:15:59 smithi042 ceph-mon[32775]: pgmap v5028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:15:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:15:59 smithi164 ceph-mon[36970]: pgmap v5028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:01 smithi163 ceph-mon[36873]: pgmap v5029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8391.305548 secs 2024-07-26T09:16:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8409.047433 secs 2024-07-26T09:16:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:01 smithi042 ceph-mon[32775]: pgmap v5029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8391.305548 secs 2024-07-26T09:16:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8409.047433 secs 2024-07-26T09:16:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:01 smithi164 ceph-mon[36970]: pgmap v5029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8391.305548 secs 2024-07-26T09:16:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8409.047433 secs 2024-07-26T09:16:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:03 smithi163 ceph-mon[36873]: pgmap v5030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:03 smithi042 ceph-mon[32775]: pgmap v5030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:03 smithi164 ceph-mon[36970]: pgmap v5030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:05 smithi163 ceph-mon[36873]: pgmap v5031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:16:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:05 smithi042 ceph-mon[32775]: pgmap v5031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:16:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:05 smithi164 ceph-mon[36970]: pgmap v5031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:16:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8396.305659 secs 2024-07-26T09:16:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8414.047561 secs 2024-07-26T09:16:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8396.305659 secs 2024-07-26T09:16:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8414.047561 secs 2024-07-26T09:16:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8396.305659 secs 2024-07-26T09:16:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8414.047561 secs 2024-07-26T09:16:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:07 smithi163 ceph-mon[36873]: pgmap v5032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:07 smithi042 ceph-mon[32775]: pgmap v5032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:07 smithi164 ceph-mon[36970]: pgmap v5032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:09 smithi042 ceph-mon[32775]: pgmap v5033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:09 smithi164 ceph-mon[36970]: pgmap v5033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:09 smithi163 ceph-mon[36873]: pgmap v5033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:11.333+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:16:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:11.333+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:16:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:11.333+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:16:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:11 smithi042 ceph-mon[32775]: pgmap v5034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8401.305784 secs 2024-07-26T09:16:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8419.047689 secs 2024-07-26T09:16:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:11 smithi164 ceph-mon[36970]: pgmap v5034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8401.305784 secs 2024-07-26T09:16:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8419.047689 secs 2024-07-26T09:16:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:11 smithi163 ceph-mon[36873]: pgmap v5034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8401.305784 secs 2024-07-26T09:16:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8419.047689 secs 2024-07-26T09:16:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:13 smithi042 ceph-mon[32775]: pgmap v5035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:13 smithi164 ceph-mon[36970]: pgmap v5035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:13 smithi163 ceph-mon[36873]: pgmap v5035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:14.165 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:14.163+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:16:14.165 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:14.163+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6580064f0 con 0x7ff680102000 2024-07-26T09:16:14.166 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:14.164+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6580064f0 con 0x7ff680102000 2024-07-26T09:16:14.166 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:14.164+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:16:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:15 smithi042 ceph-mon[32775]: pgmap v5036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:16:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:15 smithi164 ceph-mon[36970]: pgmap v5036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:16:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:15 smithi163 ceph-mon[36873]: pgmap v5036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:16:16.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8406.305925 secs 2024-07-26T09:16:16.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8424.047820 secs 2024-07-26T09:16:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8406.305925 secs 2024-07-26T09:16:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8424.047820 secs 2024-07-26T09:16:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8406.305925 secs 2024-07-26T09:16:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8424.047820 secs 2024-07-26T09:16:17.777 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:17 smithi042 ceph-mon[32775]: pgmap v5037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:17 smithi164 ceph-mon[36970]: pgmap v5037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:17 smithi163 ceph-mon[36873]: pgmap v5037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:19 smithi042 ceph-mon[32775]: pgmap v5038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:19 smithi164 ceph-mon[36970]: pgmap v5038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:19 smithi163 ceph-mon[36873]: pgmap v5038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:21 smithi042 ceph-mon[32775]: pgmap v5039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8411.306049 secs 2024-07-26T09:16:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8429.047899 secs 2024-07-26T09:16:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:21 smithi164 ceph-mon[36970]: pgmap v5039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8411.306049 secs 2024-07-26T09:16:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8429.047899 secs 2024-07-26T09:16:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:21 smithi163 ceph-mon[36873]: pgmap v5039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8411.306049 secs 2024-07-26T09:16:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8429.047899 secs 2024-07-26T09:16:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:23 smithi042 ceph-mon[32775]: pgmap v5040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:23 smithi164 ceph-mon[36970]: pgmap v5040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:23 smithi163 ceph-mon[36873]: pgmap v5040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:25 smithi042 ceph-mon[32775]: pgmap v5041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:16:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:25 smithi164 ceph-mon[36970]: pgmap v5041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:16:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:25 smithi163 ceph-mon[36873]: pgmap v5041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:16:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:26.337+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:16:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:26.337+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:16:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:26.337+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:16:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8416.306177 secs 2024-07-26T09:16:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8434.048005 secs 2024-07-26T09:16:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8416.306177 secs 2024-07-26T09:16:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8434.048005 secs 2024-07-26T09:16:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8416.306177 secs 2024-07-26T09:16:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8434.048005 secs 2024-07-26T09:16:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:27 smithi042 ceph-mon[32775]: pgmap v5042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:27 smithi164 ceph-mon[36970]: pgmap v5042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:27 smithi163 ceph-mon[36873]: pgmap v5042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:29 smithi042 ceph-mon[32775]: pgmap v5043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:29 smithi164 ceph-mon[36970]: pgmap v5043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:29 smithi163 ceph-mon[36873]: pgmap v5043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:31 smithi042 ceph-mon[32775]: pgmap v5044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8421.306300 secs 2024-07-26T09:16:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8439.048131 secs 2024-07-26T09:16:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:31 smithi164 ceph-mon[36970]: pgmap v5044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8421.306300 secs 2024-07-26T09:16:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8439.048131 secs 2024-07-26T09:16:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:31 smithi163 ceph-mon[36873]: pgmap v5044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8421.306300 secs 2024-07-26T09:16:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8439.048131 secs 2024-07-26T09:16:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:16:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:16:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:16:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:16:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:16:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:16:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:16:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:16:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:16:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:16:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:16:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:16:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:33 smithi042 ceph-mon[32775]: pgmap v5045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:33 smithi042 ceph-mon[32775]: pgmap v5046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:16:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:33 smithi042 ceph-mon[32775]: pgmap v5047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:16:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:33 smithi164 ceph-mon[36970]: pgmap v5045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:33 smithi164 ceph-mon[36970]: pgmap v5046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:16:33.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:33 smithi164 ceph-mon[36970]: pgmap v5047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:16:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:33 smithi163 ceph-mon[36873]: pgmap v5045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:33 smithi163 ceph-mon[36873]: pgmap v5046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:16:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:33 smithi163 ceph-mon[36873]: pgmap v5047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:16:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:35 smithi042 ceph-mon[32775]: pgmap v5048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:16:35.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:35 smithi164 ceph-mon[36970]: pgmap v5048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:16:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:35 smithi163 ceph-mon[36873]: pgmap v5048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:16:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8426.306388 secs 2024-07-26T09:16:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8444.048262 secs 2024-07-26T09:16:36.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8426.306388 secs 2024-07-26T09:16:36.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8444.048262 secs 2024-07-26T09:16:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8426.306388 secs 2024-07-26T09:16:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8444.048262 secs 2024-07-26T09:16:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:37 smithi042 ceph-mon[32775]: pgmap v5049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:16:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:37 smithi164 ceph-mon[36970]: pgmap v5049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:16:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:37 smithi163 ceph-mon[36873]: pgmap v5049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:16:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:39 smithi042 ceph-mon[32775]: pgmap v5050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:16:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:39 smithi164 ceph-mon[36970]: pgmap v5050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:16:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:39 smithi163 ceph-mon[36873]: pgmap v5050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:16:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:41.345+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:16:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:41.345+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:16:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:41.345+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:16:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:41 smithi042 ceph-mon[32775]: pgmap v5051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:16:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8431.306514 secs 2024-07-26T09:16:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8449.048331 secs 2024-07-26T09:16:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:41 smithi164 ceph-mon[36970]: pgmap v5051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:16:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8431.306514 secs 2024-07-26T09:16:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8449.048331 secs 2024-07-26T09:16:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:41 smithi163 ceph-mon[36873]: pgmap v5051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:16:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8431.306514 secs 2024-07-26T09:16:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8449.048331 secs 2024-07-26T09:16:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:43 smithi042 ceph-mon[32775]: pgmap v5052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:16:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:43 smithi164 ceph-mon[36970]: pgmap v5052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:16:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:43 smithi163 ceph-mon[36873]: pgmap v5052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:16:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:45 smithi042 ceph-mon[32775]: pgmap v5053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:45 smithi164 ceph-mon[36970]: pgmap v5053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:45 smithi163 ceph-mon[36873]: pgmap v5053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:46.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8436.306609 secs 2024-07-26T09:16:46.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8454.048474 secs 2024-07-26T09:16:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8436.306609 secs 2024-07-26T09:16:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8454.048474 secs 2024-07-26T09:16:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8436.306609 secs 2024-07-26T09:16:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8454.048474 secs 2024-07-26T09:16:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:47 smithi042 ceph-mon[32775]: pgmap v5054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:47 smithi164 ceph-mon[36970]: pgmap v5054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:47 smithi163 ceph-mon[36873]: pgmap v5054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:49 smithi042 ceph-mon[32775]: pgmap v5055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:16:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:49 smithi164 ceph-mon[36970]: pgmap v5055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:16:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:49 smithi163 ceph-mon[36873]: pgmap v5055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:16:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:51 smithi042 ceph-mon[32775]: pgmap v5056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8441.306736 secs 2024-07-26T09:16:51.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8459.048615 secs 2024-07-26T09:16:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:51 smithi164 ceph-mon[36970]: pgmap v5056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8441.306736 secs 2024-07-26T09:16:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8459.048615 secs 2024-07-26T09:16:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:51 smithi163 ceph-mon[36873]: pgmap v5056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8441.306736 secs 2024-07-26T09:16:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8459.048615 secs 2024-07-26T09:16:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:53 smithi042 ceph-mon[32775]: pgmap v5057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:53 smithi164 ceph-mon[36970]: pgmap v5057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:53 smithi163 ceph-mon[36873]: pgmap v5057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:55 smithi042 ceph-mon[32775]: pgmap v5058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:55 smithi164 ceph-mon[36970]: pgmap v5058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:56.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:55 smithi163 ceph-mon[36873]: pgmap v5058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:16:56.351 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:56.348+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:16:56.351 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:56.348+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:16:56.351 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:56.348+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:16:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8446.306854 secs 2024-07-26T09:16:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8464.048740 secs 2024-07-26T09:16:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8446.306854 secs 2024-07-26T09:16:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8464.048740 secs 2024-07-26T09:16:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8446.306854 secs 2024-07-26T09:16:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8464.048740 secs 2024-07-26T09:16:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:57 smithi042 ceph-mon[32775]: pgmap v5059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:57 smithi164 ceph-mon[36970]: pgmap v5059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:57 smithi163 ceph-mon[36873]: pgmap v5059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:16:59.172 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:59.170+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:16:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:59.170+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064f0 con 0x7ff680102000 2024-07-26T09:16:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:59.170+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064f0 con 0x7ff680102000 2024-07-26T09:16:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:16:59.170+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:16:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:16:59 smithi042 ceph-mon[32775]: pgmap v5060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:16:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:16:59 smithi164 ceph-mon[36970]: pgmap v5060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:17:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:16:59 smithi163 ceph-mon[36873]: pgmap v5060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:17:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:00 smithi042 ceph-mon[32775]: pgmap v5061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:00 smithi164 ceph-mon[36970]: pgmap v5061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:00 smithi163 ceph-mon[36873]: pgmap v5061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8451.306954 secs 2024-07-26T09:17:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8469.048864 secs 2024-07-26T09:17:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8451.306954 secs 2024-07-26T09:17:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8469.048864 secs 2024-07-26T09:17:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8451.306954 secs 2024-07-26T09:17:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8469.048864 secs 2024-07-26T09:17:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:02 smithi042 ceph-mon[32775]: pgmap v5062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:02 smithi164 ceph-mon[36970]: pgmap v5062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:02 smithi163 ceph-mon[36873]: pgmap v5062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:05 smithi164 ceph-mon[36970]: pgmap v5063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:05 smithi163 ceph-mon[36873]: pgmap v5063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:05 smithi042 ceph-mon[32775]: pgmap v5063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8456.307077 secs 2024-07-26T09:17:06.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8474.048986 secs 2024-07-26T09:17:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8456.307077 secs 2024-07-26T09:17:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8474.048986 secs 2024-07-26T09:17:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8456.307077 secs 2024-07-26T09:17:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8474.048986 secs 2024-07-26T09:17:07.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:07 smithi164 ceph-mon[36970]: pgmap v5064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:07 smithi163 ceph-mon[36873]: pgmap v5064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:07 smithi042 ceph-mon[32775]: pgmap v5064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:09 smithi164 ceph-mon[36970]: pgmap v5065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:17:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:09 smithi163 ceph-mon[36873]: pgmap v5065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:17:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:09 smithi042 ceph-mon[32775]: pgmap v5065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:17:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:17:11.350+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:17:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:17:11.350+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:17:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:17:11.350+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:17:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:11 smithi164 ceph-mon[36970]: pgmap v5066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:11.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8461.307186 secs 2024-07-26T09:17:11.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8479.049128 secs 2024-07-26T09:17:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:11 smithi163 ceph-mon[36873]: pgmap v5066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8461.307186 secs 2024-07-26T09:17:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8479.049128 secs 2024-07-26T09:17:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:11 smithi042 ceph-mon[32775]: pgmap v5066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8461.307186 secs 2024-07-26T09:17:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8479.049128 secs 2024-07-26T09:17:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:13 smithi163 ceph-mon[36873]: pgmap v5067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:13 smithi042 ceph-mon[32775]: pgmap v5067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:13 smithi164 ceph-mon[36970]: pgmap v5067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:15 smithi163 ceph-mon[36873]: pgmap v5068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:15 smithi042 ceph-mon[32775]: pgmap v5068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:15 smithi164 ceph-mon[36970]: pgmap v5068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8466.307309 secs 2024-07-26T09:17:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8484.049252 secs 2024-07-26T09:17:16.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8466.307309 secs 2024-07-26T09:17:16.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8484.049252 secs 2024-07-26T09:17:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8466.307309 secs 2024-07-26T09:17:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8484.049252 secs 2024-07-26T09:17:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:17 smithi163 ceph-mon[36873]: pgmap v5069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:17 smithi042 ceph-mon[32775]: pgmap v5069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:17 smithi164 ceph-mon[36970]: pgmap v5069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:19 smithi163 ceph-mon[36873]: pgmap v5070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:17:19.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:19 smithi042 ceph-mon[32775]: pgmap v5070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:17:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:19 smithi164 ceph-mon[36970]: pgmap v5070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:17:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:21 smithi163 ceph-mon[36873]: pgmap v5071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8471.307435 secs 2024-07-26T09:17:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8489.049377 secs 2024-07-26T09:17:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:21 smithi042 ceph-mon[32775]: pgmap v5071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8471.307435 secs 2024-07-26T09:17:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8489.049377 secs 2024-07-26T09:17:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:21 smithi164 ceph-mon[36970]: pgmap v5071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8471.307435 secs 2024-07-26T09:17:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8489.049377 secs 2024-07-26T09:17:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:23 smithi163 ceph-mon[36873]: pgmap v5072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:23 smithi042 ceph-mon[32775]: pgmap v5072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:23 smithi164 ceph-mon[36970]: pgmap v5072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:25 smithi163 ceph-mon[36873]: pgmap v5073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:25 smithi042 ceph-mon[32775]: pgmap v5073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:25 smithi164 ceph-mon[36970]: pgmap v5073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:17:26.354+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:17:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:17:26.355+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:17:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:17:26.355+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:17:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8476.307565 secs 2024-07-26T09:17:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8494.049468 secs 2024-07-26T09:17:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8476.307565 secs 2024-07-26T09:17:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8494.049468 secs 2024-07-26T09:17:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8476.307565 secs 2024-07-26T09:17:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8494.049468 secs 2024-07-26T09:17:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:27 smithi042 ceph-mon[32775]: pgmap v5074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:27 smithi164 ceph-mon[36970]: pgmap v5074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:27 smithi163 ceph-mon[36873]: pgmap v5074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:29 smithi042 ceph-mon[32775]: pgmap v5075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:17:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:29 smithi164 ceph-mon[36970]: pgmap v5075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:17:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:29 smithi163 ceph-mon[36873]: pgmap v5075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:17:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:31 smithi042 ceph-mon[32775]: pgmap v5076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8481.307687 secs 2024-07-26T09:17:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8499.049587 secs 2024-07-26T09:17:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:31 smithi164 ceph-mon[36970]: pgmap v5076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8481.307687 secs 2024-07-26T09:17:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8499.049587 secs 2024-07-26T09:17:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:31 smithi163 ceph-mon[36873]: pgmap v5076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8481.307687 secs 2024-07-26T09:17:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8499.049587 secs 2024-07-26T09:17:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:17:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:17:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:17:33.627 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:33 smithi042 ceph-mon[32775]: pgmap v5077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:33 smithi164 ceph-mon[36970]: pgmap v5077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:33 smithi163 ceph-mon[36873]: pgmap v5077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:35.683 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:35 smithi042 ceph-mon[32775]: pgmap v5078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:35 smithi164 ceph-mon[36970]: pgmap v5078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:35 smithi163 ceph-mon[36873]: pgmap v5078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8486.307831 secs 2024-07-26T09:17:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8504.049708 secs 2024-07-26T09:17:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:17:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:17:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:17:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:17:36.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:17:36.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8486.307831 secs 2024-07-26T09:17:36.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8504.049708 secs 2024-07-26T09:17:36.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:17:36.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:17:36.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:17:36.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:17:36.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:17:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8486.307831 secs 2024-07-26T09:17:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8504.049708 secs 2024-07-26T09:17:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:17:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:17:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:17:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:17:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:17:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:37 smithi042 ceph-mon[32775]: pgmap v5079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:37 smithi042 ceph-mon[32775]: pgmap v5080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:37.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:37 smithi164 ceph-mon[36970]: pgmap v5079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:37.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:37 smithi164 ceph-mon[36970]: pgmap v5080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:37 smithi163 ceph-mon[36873]: pgmap v5079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:37 smithi163 ceph-mon[36873]: pgmap v5080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:39 smithi042 ceph-mon[32775]: pgmap v5081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:39 smithi164 ceph-mon[36970]: pgmap v5081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:39 smithi163 ceph-mon[36873]: pgmap v5081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:41.366 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:17:41.364+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:17:41.366 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:17:41.364+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:17:41.367 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:17:41.364+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:17:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:41 smithi042 ceph-mon[32775]: pgmap v5082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8491.307937 secs 2024-07-26T09:17:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8509.049831 secs 2024-07-26T09:17:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:41 smithi164 ceph-mon[36970]: pgmap v5082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8491.307937 secs 2024-07-26T09:17:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8509.049831 secs 2024-07-26T09:17:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:41 smithi163 ceph-mon[36873]: pgmap v5082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:42.015 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8491.307937 secs 2024-07-26T09:17:42.016 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8509.049831 secs 2024-07-26T09:17:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:43 smithi042 ceph-mon[32775]: pgmap v5083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:43 smithi164 ceph-mon[36970]: pgmap v5083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:43 smithi163 ceph-mon[36873]: pgmap v5083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:45 smithi042 ceph-mon[32775]: pgmap v5084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:45 smithi164 ceph-mon[36970]: pgmap v5084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:45 smithi163 ceph-mon[36873]: pgmap v5084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:46.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8496.308022 secs 2024-07-26T09:17:46.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8514.049972 secs 2024-07-26T09:17:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8496.308022 secs 2024-07-26T09:17:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8514.049972 secs 2024-07-26T09:17:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8496.308022 secs 2024-07-26T09:17:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8514.049972 secs 2024-07-26T09:17:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:47 smithi042 ceph-mon[32775]: pgmap v5085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:47 smithi164 ceph-mon[36970]: pgmap v5085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:47 smithi163 ceph-mon[36873]: pgmap v5085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:17:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:49 smithi042 ceph-mon[32775]: pgmap v5086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:17:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:49 smithi164 ceph-mon[36970]: pgmap v5086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:17:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:49 smithi163 ceph-mon[36873]: pgmap v5086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:17:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:51 smithi042 ceph-mon[32775]: pgmap v5087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8501.308144 secs 2024-07-26T09:17:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8519.050091 secs 2024-07-26T09:17:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:51 smithi164 ceph-mon[36970]: pgmap v5087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8501.308144 secs 2024-07-26T09:17:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8519.050091 secs 2024-07-26T09:17:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:51 smithi163 ceph-mon[36873]: pgmap v5087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8501.308144 secs 2024-07-26T09:17:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8519.050091 secs 2024-07-26T09:17:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:53 smithi042 ceph-mon[32775]: pgmap v5088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:53 smithi164 ceph-mon[36970]: pgmap v5088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:53 smithi163 ceph-mon[36873]: pgmap v5088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:55 smithi042 ceph-mon[32775]: pgmap v5089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:55 smithi164 ceph-mon[36970]: pgmap v5089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:55 smithi163 ceph-mon[36873]: pgmap v5089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:17:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:17:56.368+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:17:56.371 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:17:56.368+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:17:56.371 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:17:56.368+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:17:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8506.308270 secs 2024-07-26T09:17:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8524.050220 secs 2024-07-26T09:17:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8506.308270 secs 2024-07-26T09:17:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8524.050220 secs 2024-07-26T09:17:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8506.308270 secs 2024-07-26T09:17:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8524.050220 secs 2024-07-26T09:17:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:57 smithi042 ceph-mon[32775]: pgmap v5090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:57 smithi164 ceph-mon[36970]: pgmap v5090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:57 smithi163 ceph-mon[36873]: pgmap v5090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:17:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:17:59 smithi042 ceph-mon[32775]: pgmap v5091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:17:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:17:59 smithi164 ceph-mon[36970]: pgmap v5091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:18:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:17:59 smithi163 ceph-mon[36873]: pgmap v5091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:18:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:01 smithi042 ceph-mon[32775]: pgmap v5092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8511.308391 secs 2024-07-26T09:18:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8529.050343 secs 2024-07-26T09:18:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:01 smithi164 ceph-mon[36970]: pgmap v5092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8511.308391 secs 2024-07-26T09:18:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8529.050343 secs 2024-07-26T09:18:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:01 smithi163 ceph-mon[36873]: pgmap v5092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8511.308391 secs 2024-07-26T09:18:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8529.050343 secs 2024-07-26T09:18:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:03 smithi042 ceph-mon[32775]: pgmap v5093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:03 smithi164 ceph-mon[36970]: pgmap v5093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:03 smithi163 ceph-mon[36873]: pgmap v5093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:05 smithi042 ceph-mon[32775]: pgmap v5094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:05 smithi164 ceph-mon[36970]: pgmap v5094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:05 smithi163 ceph-mon[36873]: pgmap v5094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8516.308529 secs 2024-07-26T09:18:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8534.050460 secs 2024-07-26T09:18:06.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8516.308529 secs 2024-07-26T09:18:06.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8534.050460 secs 2024-07-26T09:18:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8516.308529 secs 2024-07-26T09:18:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8534.050460 secs 2024-07-26T09:18:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:07 smithi042 ceph-mon[32775]: pgmap v5095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:07 smithi164 ceph-mon[36970]: pgmap v5095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:07 smithi163 ceph-mon[36873]: pgmap v5095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:09 smithi042 ceph-mon[32775]: pgmap v5096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:18:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:09 smithi164 ceph-mon[36970]: pgmap v5096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:18:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:09 smithi163 ceph-mon[36873]: pgmap v5096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:18:11.379 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:18:11.377+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:18:11.379 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:18:11.377+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:18:11.379 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:18:11.377+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:18:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:11 smithi042 ceph-mon[32775]: pgmap v5097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8521.308653 secs 2024-07-26T09:18:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8539.050599 secs 2024-07-26T09:18:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:11 smithi164 ceph-mon[36970]: pgmap v5097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8521.308653 secs 2024-07-26T09:18:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8539.050599 secs 2024-07-26T09:18:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:11 smithi163 ceph-mon[36873]: pgmap v5097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8521.308653 secs 2024-07-26T09:18:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8539.050599 secs 2024-07-26T09:18:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:13 smithi042 ceph-mon[32775]: pgmap v5098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:13 smithi164 ceph-mon[36970]: pgmap v5098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:13 smithi163 ceph-mon[36873]: pgmap v5098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:15 smithi042 ceph-mon[32775]: pgmap v5099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:15 smithi164 ceph-mon[36970]: pgmap v5099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:15 smithi163 ceph-mon[36873]: pgmap v5099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:16.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8526.308777 secs 2024-07-26T09:18:16.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8544.050741 secs 2024-07-26T09:18:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8526.308777 secs 2024-07-26T09:18:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8544.050741 secs 2024-07-26T09:18:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8526.308777 secs 2024-07-26T09:18:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8544.050741 secs 2024-07-26T09:18:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:17 smithi042 ceph-mon[32775]: pgmap v5100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:17 smithi164 ceph-mon[36970]: pgmap v5100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:17 smithi163 ceph-mon[36873]: pgmap v5100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:19 smithi042 ceph-mon[32775]: pgmap v5101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:18:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:19 smithi164 ceph-mon[36970]: pgmap v5101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:18:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:19 smithi163 ceph-mon[36873]: pgmap v5101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:18:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:21 smithi042 ceph-mon[32775]: pgmap v5102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8531.308931 secs 2024-07-26T09:18:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8549.050890 secs 2024-07-26T09:18:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:21 smithi164 ceph-mon[36970]: pgmap v5102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8531.308931 secs 2024-07-26T09:18:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8549.050890 secs 2024-07-26T09:18:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:21 smithi163 ceph-mon[36873]: pgmap v5102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8531.308931 secs 2024-07-26T09:18:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8549.050890 secs 2024-07-26T09:18:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:23 smithi042 ceph-mon[32775]: pgmap v5103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:23 smithi164 ceph-mon[36970]: pgmap v5103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:23 smithi163 ceph-mon[36873]: pgmap v5103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:25 smithi042 ceph-mon[32775]: pgmap v5104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:25 smithi164 ceph-mon[36970]: pgmap v5104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:25 smithi163 ceph-mon[36873]: pgmap v5104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:18:26.381+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:18:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:18:26.381+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:18:26.383 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:18:26.381+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:18:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8536.309084 secs 2024-07-26T09:18:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8554.051012 secs 2024-07-26T09:18:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8536.309084 secs 2024-07-26T09:18:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8554.051012 secs 2024-07-26T09:18:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8536.309084 secs 2024-07-26T09:18:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8554.051012 secs 2024-07-26T09:18:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:27 smithi042 ceph-mon[32775]: pgmap v5105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:27 smithi164 ceph-mon[36970]: pgmap v5105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:27 smithi163 ceph-mon[36873]: pgmap v5105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:28 smithi042 ceph-mon[32775]: pgmap v5106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:18:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:28 smithi164 ceph-mon[36970]: pgmap v5106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:18:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:28 smithi163 ceph-mon[36873]: pgmap v5106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:18:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:31 smithi163 ceph-mon[36873]: pgmap v5107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8541.309251 secs 2024-07-26T09:18:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8559.051125 secs 2024-07-26T09:18:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:31 smithi042 ceph-mon[32775]: pgmap v5107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8541.309251 secs 2024-07-26T09:18:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8559.051125 secs 2024-07-26T09:18:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:31 smithi164 ceph-mon[36970]: pgmap v5107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8541.309251 secs 2024-07-26T09:18:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8559.051125 secs 2024-07-26T09:18:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:33 smithi163 ceph-mon[36873]: pgmap v5108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:33 smithi042 ceph-mon[32775]: pgmap v5108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:33 smithi164 ceph-mon[36970]: pgmap v5108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:35 smithi163 ceph-mon[36873]: pgmap v5109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:35 smithi042 ceph-mon[32775]: pgmap v5109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:35 smithi164 ceph-mon[36970]: pgmap v5109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8546.309389 secs 2024-07-26T09:18:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8564.051245 secs 2024-07-26T09:18:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8546.309389 secs 2024-07-26T09:18:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8564.051245 secs 2024-07-26T09:18:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8546.309389 secs 2024-07-26T09:18:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8564.051245 secs 2024-07-26T09:18:37.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:37 smithi042 ceph-mon[32775]: pgmap v5110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:37.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:18:37.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:37.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:37.674 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:37.675 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:37.707 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:37 smithi164 ceph-mon[36970]: pgmap v5110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:37.708 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:18:37.708 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:37.708 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:37.708 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:37.708 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:37.713 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:37 smithi163 ceph-mon[36873]: pgmap v5110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:37.713 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:18:37.713 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:37.713 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:37.713 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:37.713 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:38.677 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:38 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:18:38.678 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:38 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:18:38.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:38 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:18:38.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:38 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:18:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:38 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:18:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:38 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:18:39.537 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:39 smithi163 ceph-mon[36873]: pgmap v5111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:18:39.812 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:39 smithi164 ceph-mon[36970]: pgmap v5111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:18:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:39 smithi042 ceph-mon[32775]: pgmap v5111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:18:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:18:41.391+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:18:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:18:41.391+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:18:41.393 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:18:41.391+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:18:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:18:41.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:18:41.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:18:41.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:18:41.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:18:41.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:18:41.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:18:41.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:18:41.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:18:41.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:18:41.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: pgmap v5112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:41.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: pgmap v5113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:18:41.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:18:41.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8551.309491 secs 2024-07-26T09:18:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:18:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:18:41.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:18:41.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:18:41.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:18:41.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:18:41.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:18:41.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:18:41.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:18:41.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:18:41.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: pgmap v5112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:41.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: pgmap v5113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:18:41.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:18:41.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8551.309491 secs 2024-07-26T09:18:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:18:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:18:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:18:41.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:18:41.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:18:41.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:18:41.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:18:41.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:18:41.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:18:41.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:18:41.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:18:41.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: pgmap v5112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:41.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: pgmap v5113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:18:41.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:18:41.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8551.309491 secs 2024-07-26T09:18:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8569.051329 secs 2024-07-26T09:18:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8569.051329 secs 2024-07-26T09:18:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8569.051329 secs 2024-07-26T09:18:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:43 smithi042 ceph-mon[32775]: pgmap v5114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:18:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:43 smithi164 ceph-mon[36970]: pgmap v5114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:18:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:43 smithi163 ceph-mon[36873]: pgmap v5114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:18:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:45 smithi042 ceph-mon[32775]: pgmap v5115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:18:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:45 smithi164 ceph-mon[36970]: pgmap v5115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:18:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:45 smithi163 ceph-mon[36873]: pgmap v5115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:18:46.647 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8556.309629 secs 2024-07-26T09:18:46.647 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8574.051452 secs 2024-07-26T09:18:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8556.309629 secs 2024-07-26T09:18:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8574.051452 secs 2024-07-26T09:18:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8556.309629 secs 2024-07-26T09:18:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8574.051452 secs 2024-07-26T09:18:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:47 smithi164 ceph-mon[36970]: pgmap v5116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:18:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:47 smithi163 ceph-mon[36873]: pgmap v5116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:18:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:47 smithi042 ceph-mon[32775]: pgmap v5116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:18:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:49 smithi164 ceph-mon[36970]: pgmap v5117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:18:49.729 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:49 smithi163 ceph-mon[36873]: pgmap v5117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:18:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:49 smithi042 ceph-mon[32775]: pgmap v5117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:18:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:51 smithi164 ceph-mon[36970]: pgmap v5118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:18:51.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8561.309707 secs 2024-07-26T09:18:51.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8579.051575 secs 2024-07-26T09:18:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:51 smithi163 ceph-mon[36873]: pgmap v5118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:18:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8561.309707 secs 2024-07-26T09:18:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8579.051575 secs 2024-07-26T09:18:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:51 smithi042 ceph-mon[32775]: pgmap v5118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:18:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8561.309707 secs 2024-07-26T09:18:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8579.051575 secs 2024-07-26T09:18:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:53 smithi164 ceph-mon[36970]: pgmap v5119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:53 smithi163 ceph-mon[36873]: pgmap v5119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:53 smithi042 ceph-mon[32775]: pgmap v5119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:55 smithi164 ceph-mon[36970]: pgmap v5120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:55 smithi163 ceph-mon[36873]: pgmap v5120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:55 smithi042 ceph-mon[32775]: pgmap v5120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:18:56.398 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:18:56.395+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:18:56.398 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:18:56.395+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:18:56.398 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:18:56.395+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:18:56.676 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8566.309828 secs 2024-07-26T09:18:56.676 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8584.051695 secs 2024-07-26T09:18:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8566.309828 secs 2024-07-26T09:18:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8584.051695 secs 2024-07-26T09:18:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8566.309828 secs 2024-07-26T09:18:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8584.051695 secs 2024-07-26T09:18:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:57 smithi164 ceph-mon[36970]: pgmap v5121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:57 smithi163 ceph-mon[36873]: pgmap v5121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:57 smithi042 ceph-mon[32775]: pgmap v5121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:18:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:18:59 smithi164 ceph-mon[36970]: pgmap v5122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:18:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:18:59 smithi163 ceph-mon[36873]: pgmap v5122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:18:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:18:59 smithi042 ceph-mon[32775]: pgmap v5122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:19:01.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:01 smithi042 ceph-mon[32775]: pgmap v5123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:01.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8571.309951 secs 2024-07-26T09:19:01.699 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8589.051816 secs 2024-07-26T09:19:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:01 smithi164 ceph-mon[36970]: pgmap v5123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8571.309951 secs 2024-07-26T09:19:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8589.051816 secs 2024-07-26T09:19:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:01 smithi163 ceph-mon[36873]: pgmap v5123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8571.309951 secs 2024-07-26T09:19:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8589.051816 secs 2024-07-26T09:19:03.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:03 smithi164 ceph-mon[36970]: pgmap v5124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:03 smithi163 ceph-mon[36873]: pgmap v5124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:03 smithi042 ceph-mon[32775]: pgmap v5124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:05 smithi164 ceph-mon[36970]: pgmap v5125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:05 smithi163 ceph-mon[36873]: pgmap v5125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:05 smithi042 ceph-mon[32775]: pgmap v5125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8576.310076 secs 2024-07-26T09:19:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8594.051882 secs 2024-07-26T09:19:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8576.310076 secs 2024-07-26T09:19:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8594.051882 secs 2024-07-26T09:19:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8576.310076 secs 2024-07-26T09:19:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8594.051882 secs 2024-07-26T09:19:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:07 smithi164 ceph-mon[36970]: pgmap v5126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:07 smithi163 ceph-mon[36873]: pgmap v5126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:07 smithi042 ceph-mon[32775]: pgmap v5126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:09 smithi164 ceph-mon[36970]: pgmap v5127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:19:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:09 smithi163 ceph-mon[36873]: pgmap v5127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:19:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:09 smithi042 ceph-mon[32775]: pgmap v5127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:19:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:19:11.405+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:19:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:19:11.405+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:19:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:19:11.405+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:19:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:11 smithi164 ceph-mon[36970]: pgmap v5128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:11.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8581.310208 secs 2024-07-26T09:19:11.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8599.052024 secs 2024-07-26T09:19:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:11 smithi163 ceph-mon[36873]: pgmap v5128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8581.310208 secs 2024-07-26T09:19:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8599.052024 secs 2024-07-26T09:19:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:11 smithi042 ceph-mon[32775]: pgmap v5128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8581.310208 secs 2024-07-26T09:19:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8599.052024 secs 2024-07-26T09:19:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:13 smithi164 ceph-mon[36970]: pgmap v5129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:13 smithi163 ceph-mon[36873]: pgmap v5129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:13 smithi042 ceph-mon[32775]: pgmap v5129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:15 smithi164 ceph-mon[36970]: pgmap v5130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:15 smithi163 ceph-mon[36873]: pgmap v5130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:15 smithi042 ceph-mon[32775]: pgmap v5130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8586.310330 secs 2024-07-26T09:19:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8604.052141 secs 2024-07-26T09:19:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8586.310330 secs 2024-07-26T09:19:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8604.052141 secs 2024-07-26T09:19:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8586.310330 secs 2024-07-26T09:19:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8604.052141 secs 2024-07-26T09:19:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:17 smithi164 ceph-mon[36970]: pgmap v5131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:17 smithi163 ceph-mon[36873]: pgmap v5131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:17 smithi042 ceph-mon[32775]: pgmap v5131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:19 smithi164 ceph-mon[36970]: pgmap v5132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:19:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:19 smithi163 ceph-mon[36873]: pgmap v5132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:19:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:19 smithi042 ceph-mon[32775]: pgmap v5132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:19:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:21 smithi164 ceph-mon[36970]: pgmap v5133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8591.310447 secs 2024-07-26T09:19:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8609.052265 secs 2024-07-26T09:19:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:21 smithi163 ceph-mon[36873]: pgmap v5133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8591.310447 secs 2024-07-26T09:19:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8609.052265 secs 2024-07-26T09:19:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:21 smithi042 ceph-mon[32775]: pgmap v5133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8591.310447 secs 2024-07-26T09:19:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8609.052265 secs 2024-07-26T09:19:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:23 smithi164 ceph-mon[36970]: pgmap v5134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:23 smithi163 ceph-mon[36873]: pgmap v5134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:23 smithi042 ceph-mon[32775]: pgmap v5134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:25 smithi163 ceph-mon[36873]: pgmap v5135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:25 smithi042 ceph-mon[32775]: pgmap v5135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:25 smithi164 ceph-mon[36970]: pgmap v5135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:19:26.406+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:19:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:19:26.406+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:19:26.408 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:19:26.406+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:19:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8596.310576 secs 2024-07-26T09:19:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8614.052391 secs 2024-07-26T09:19:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8596.310576 secs 2024-07-26T09:19:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8614.052391 secs 2024-07-26T09:19:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8596.310576 secs 2024-07-26T09:19:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8614.052391 secs 2024-07-26T09:19:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:27 smithi163 ceph-mon[36873]: pgmap v5136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:27 smithi042 ceph-mon[32775]: pgmap v5136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:27 smithi164 ceph-mon[36970]: pgmap v5136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:29 smithi163 ceph-mon[36873]: pgmap v5137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:19:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:29 smithi042 ceph-mon[32775]: pgmap v5137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:19:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:29 smithi164 ceph-mon[36970]: pgmap v5137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:19:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:31 smithi163 ceph-mon[36873]: pgmap v5138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8601.310700 secs 2024-07-26T09:19:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8619.052511 secs 2024-07-26T09:19:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:31 smithi042 ceph-mon[32775]: pgmap v5138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8601.310700 secs 2024-07-26T09:19:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8619.052511 secs 2024-07-26T09:19:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:31 smithi164 ceph-mon[36970]: pgmap v5138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8601.310700 secs 2024-07-26T09:19:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8619.052511 secs 2024-07-26T09:19:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:33 smithi163 ceph-mon[36873]: pgmap v5139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:33 smithi042 ceph-mon[32775]: pgmap v5139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:33 smithi164 ceph-mon[36970]: pgmap v5139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:35 smithi163 ceph-mon[36873]: pgmap v5140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:35 smithi042 ceph-mon[32775]: pgmap v5140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:35 smithi164 ceph-mon[36970]: pgmap v5140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8606.310825 secs 2024-07-26T09:19:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8624.052584 secs 2024-07-26T09:19:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8606.310825 secs 2024-07-26T09:19:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8624.052584 secs 2024-07-26T09:19:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8606.310825 secs 2024-07-26T09:19:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8624.052584 secs 2024-07-26T09:19:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:37 smithi163 ceph-mon[36873]: pgmap v5141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:37 smithi042 ceph-mon[32775]: pgmap v5141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:37 smithi164 ceph-mon[36970]: pgmap v5141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:39 smithi163 ceph-mon[36873]: pgmap v5142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:19:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:39 smithi042 ceph-mon[32775]: pgmap v5142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:19:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:39 smithi164 ceph-mon[36970]: pgmap v5142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:19:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:19:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:19:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:19:41.417 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:19:41.416+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:19:41.418 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:19:41.416+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:19:41.418 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:19:41.416+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:19:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:41 smithi042 ceph-mon[32775]: pgmap v5143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8611.310949 secs 2024-07-26T09:19:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:19:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:19:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:19:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8629.052706 secs 2024-07-26T09:19:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:41 smithi164 ceph-mon[36970]: pgmap v5143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8611.310949 secs 2024-07-26T09:19:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:19:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:19:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:19:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8629.052706 secs 2024-07-26T09:19:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:41 smithi163 ceph-mon[36873]: pgmap v5143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8611.310949 secs 2024-07-26T09:19:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:19:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:19:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:19:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8629.052706 secs 2024-07-26T09:19:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:42 smithi042 ceph-mon[32775]: pgmap v5144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:19:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:42 smithi042 ceph-mon[32775]: pgmap v5145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:19:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:42 smithi164 ceph-mon[36970]: pgmap v5144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:19:42.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:42 smithi164 ceph-mon[36970]: pgmap v5145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:19:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:42 smithi163 ceph-mon[36873]: pgmap v5144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:19:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:42 smithi163 ceph-mon[36873]: pgmap v5145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:19:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:44 smithi042 ceph-mon[32775]: pgmap v5146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:19:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:44 smithi164 ceph-mon[36970]: pgmap v5146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:19:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:44 smithi163 ceph-mon[36873]: pgmap v5146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:19:46.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:46 smithi164 ceph-mon[36970]: pgmap v5147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:19:46.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8616.311083 secs 2024-07-26T09:19:46.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8634.052831 secs 2024-07-26T09:19:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:46 smithi042 ceph-mon[32775]: pgmap v5147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:19:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8616.311083 secs 2024-07-26T09:19:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8634.052831 secs 2024-07-26T09:19:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:46 smithi163 ceph-mon[36873]: pgmap v5147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:19:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8616.311083 secs 2024-07-26T09:19:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8634.052831 secs 2024-07-26T09:19:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:48 smithi042 ceph-mon[32775]: pgmap v5148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:19:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:48 smithi164 ceph-mon[36970]: pgmap v5148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:19:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:48 smithi163 ceph-mon[36873]: pgmap v5148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:19:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:50 smithi042 ceph-mon[32775]: pgmap v5149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:19:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:50 smithi164 ceph-mon[36970]: pgmap v5149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:19:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:50 smithi163 ceph-mon[36873]: pgmap v5149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:19:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8621.311206 secs 2024-07-26T09:19:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8639.052970 secs 2024-07-26T09:19:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8621.311206 secs 2024-07-26T09:19:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8639.052970 secs 2024-07-26T09:19:52.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8621.311206 secs 2024-07-26T09:19:52.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8639.052970 secs 2024-07-26T09:19:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:52 smithi042 ceph-mon[32775]: pgmap v5150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:19:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:52 smithi164 ceph-mon[36970]: pgmap v5150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:19:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:52 smithi163 ceph-mon[36873]: pgmap v5150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:19:54.865 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:54 smithi042 ceph-mon[32775]: pgmap v5151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:54 smithi164 ceph-mon[36970]: pgmap v5151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:54 smithi163 ceph-mon[36873]: pgmap v5151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:56.423 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:19:56.421+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:19:56.423 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:19:56.421+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:19:56.423 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:19:56.421+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:19:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:56 smithi042 ceph-mon[32775]: pgmap v5152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8626.311235 secs 2024-07-26T09:19:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8644.053090 secs 2024-07-26T09:19:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:56 smithi164 ceph-mon[36970]: pgmap v5152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8626.311235 secs 2024-07-26T09:19:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8644.053090 secs 2024-07-26T09:19:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:56 smithi163 ceph-mon[36873]: pgmap v5152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:19:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8626.311235 secs 2024-07-26T09:19:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8644.053090 secs 2024-07-26T09:19:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:19:58 smithi042 ceph-mon[32775]: pgmap v5153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:19:58 smithi164 ceph-mon[36970]: pgmap v5153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:19:58 smithi163 ceph-mon[36873]: pgmap v5153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:19:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:19:59.199+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064c0 con 0x7ff680102000 2024-07-26T09:19:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:19:59.199+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:19:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:19:59.199+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:19:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:19:59.199+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064c0 con 0x7ff680102000 2024-07-26T09:20:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:00 smithi042 ceph-mon[32775]: pgmap v5154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:20:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:00 smithi164 ceph-mon[36970]: pgmap v5154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:00.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:20:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:00 smithi163 ceph-mon[36873]: pgmap v5154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:20:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8631.311363 secs 2024-07-26T09:20:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8649.053213 secs 2024-07-26T09:20:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8631.311363 secs 2024-07-26T09:20:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8649.053213 secs 2024-07-26T09:20:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8631.311363 secs 2024-07-26T09:20:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8649.053213 secs 2024-07-26T09:20:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:02 smithi042 ceph-mon[32775]: pgmap v5155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:02 smithi164 ceph-mon[36970]: pgmap v5155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:02 smithi163 ceph-mon[36873]: pgmap v5155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:04 smithi042 ceph-mon[32775]: pgmap v5156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:04 smithi164 ceph-mon[36970]: pgmap v5156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:04 smithi163 ceph-mon[36873]: pgmap v5156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:06.825 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:06 smithi163 ceph-mon[36873]: pgmap v5157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:06.826 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8636.311489 secs 2024-07-26T09:20:06.826 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8654.053337 secs 2024-07-26T09:20:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:06 smithi042 ceph-mon[32775]: pgmap v5157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8636.311489 secs 2024-07-26T09:20:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8654.053337 secs 2024-07-26T09:20:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:06 smithi164 ceph-mon[36970]: pgmap v5157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8636.311489 secs 2024-07-26T09:20:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8654.053337 secs 2024-07-26T09:20:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:08 smithi042 ceph-mon[32775]: pgmap v5158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:08 smithi164 ceph-mon[36970]: pgmap v5158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:08 smithi163 ceph-mon[36873]: pgmap v5158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:10 smithi042 ceph-mon[32775]: pgmap v5159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:10 smithi164 ceph-mon[36970]: pgmap v5159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:10 smithi163 ceph-mon[36873]: pgmap v5159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:20:11.430+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:20:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:20:11.430+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:20:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:20:11.430+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:20:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8641.311590 secs 2024-07-26T09:20:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8659.053458 secs 2024-07-26T09:20:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8641.311590 secs 2024-07-26T09:20:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8659.053458 secs 2024-07-26T09:20:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8641.311590 secs 2024-07-26T09:20:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8659.053458 secs 2024-07-26T09:20:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:12 smithi042 ceph-mon[32775]: pgmap v5160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:12 smithi164 ceph-mon[36970]: pgmap v5160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:12 smithi163 ceph-mon[36873]: pgmap v5160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:14 smithi042 ceph-mon[32775]: pgmap v5161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:14 smithi164 ceph-mon[36970]: pgmap v5161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:14 smithi163 ceph-mon[36873]: pgmap v5161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:16 smithi042 ceph-mon[32775]: pgmap v5162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8646.311676 secs 2024-07-26T09:20:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8664.053578 secs 2024-07-26T09:20:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:16 smithi164 ceph-mon[36970]: pgmap v5162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8646.311676 secs 2024-07-26T09:20:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8664.053578 secs 2024-07-26T09:20:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:16 smithi163 ceph-mon[36873]: pgmap v5162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8646.311676 secs 2024-07-26T09:20:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8664.053578 secs 2024-07-26T09:20:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:18 smithi042 ceph-mon[32775]: pgmap v5163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:18 smithi164 ceph-mon[36970]: pgmap v5163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:18 smithi163 ceph-mon[36873]: pgmap v5163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:20 smithi042 ceph-mon[32775]: pgmap v5164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:20 smithi164 ceph-mon[36970]: pgmap v5164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:20 smithi163 ceph-mon[36873]: pgmap v5164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8651.311781 secs 2024-07-26T09:20:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8669.053697 secs 2024-07-26T09:20:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8651.311781 secs 2024-07-26T09:20:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8669.053697 secs 2024-07-26T09:20:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8651.311781 secs 2024-07-26T09:20:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8669.053697 secs 2024-07-26T09:20:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:22 smithi042 ceph-mon[32775]: pgmap v5165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:22 smithi164 ceph-mon[36970]: pgmap v5165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:22 smithi163 ceph-mon[36873]: pgmap v5165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:24 smithi042 ceph-mon[32775]: pgmap v5166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:24 smithi164 ceph-mon[36970]: pgmap v5166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:24 smithi163 ceph-mon[36873]: pgmap v5166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:26.434 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:20:26.433+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:20:26.434 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:20:26.433+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:20:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:20:26.433+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:20:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:26 smithi042 ceph-mon[32775]: pgmap v5167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8656.311928 secs 2024-07-26T09:20:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8674.053817 secs 2024-07-26T09:20:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:26 smithi164 ceph-mon[36970]: pgmap v5167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8656.311928 secs 2024-07-26T09:20:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8674.053817 secs 2024-07-26T09:20:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:26 smithi163 ceph-mon[36873]: pgmap v5167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8656.311928 secs 2024-07-26T09:20:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8674.053817 secs 2024-07-26T09:20:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:28 smithi042 ceph-mon[32775]: pgmap v5168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:28 smithi164 ceph-mon[36970]: pgmap v5168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:28 smithi163 ceph-mon[36873]: pgmap v5168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:30 smithi042 ceph-mon[32775]: pgmap v5169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:30 smithi164 ceph-mon[36970]: pgmap v5169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:30 smithi163 ceph-mon[36873]: pgmap v5169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8661.312049 secs 2024-07-26T09:20:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8679.053961 secs 2024-07-26T09:20:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8661.312049 secs 2024-07-26T09:20:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8679.053961 secs 2024-07-26T09:20:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8661.312049 secs 2024-07-26T09:20:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8679.053961 secs 2024-07-26T09:20:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:32 smithi042 ceph-mon[32775]: pgmap v5170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:32 smithi164 ceph-mon[36970]: pgmap v5170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:32 smithi163 ceph-mon[36873]: pgmap v5170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:34 smithi042 ceph-mon[32775]: pgmap v5171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:34 smithi164 ceph-mon[36970]: pgmap v5171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:34 smithi163 ceph-mon[36873]: pgmap v5171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:36 smithi042 ceph-mon[32775]: pgmap v5172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8666.312120 secs 2024-07-26T09:20:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8684.054023 secs 2024-07-26T09:20:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:36 smithi164 ceph-mon[36970]: pgmap v5172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8666.312120 secs 2024-07-26T09:20:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8684.054023 secs 2024-07-26T09:20:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:36 smithi163 ceph-mon[36873]: pgmap v5172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8666.312120 secs 2024-07-26T09:20:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8684.054023 secs 2024-07-26T09:20:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:38 smithi042 ceph-mon[32775]: pgmap v5173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:38 smithi164 ceph-mon[36970]: pgmap v5173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:38 smithi163 ceph-mon[36873]: pgmap v5173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:40 smithi042 ceph-mon[32775]: pgmap v5174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:40 smithi164 ceph-mon[36970]: pgmap v5174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:40 smithi163 ceph-mon[36873]: pgmap v5174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:20:41.436+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:20:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:20:41.436+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:20:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:20:41.436+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:20:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8671.312254 secs 2024-07-26T09:20:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:20:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8689.054119 secs 2024-07-26T09:20:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:20:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:20:41.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:20:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8671.312254 secs 2024-07-26T09:20:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:20:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8689.054119 secs 2024-07-26T09:20:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:20:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:20:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:20:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8671.312254 secs 2024-07-26T09:20:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:20:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8689.054119 secs 2024-07-26T09:20:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:20:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:20:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:20:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:42 smithi042 ceph-mon[32775]: pgmap v5175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:42 smithi042 ceph-mon[32775]: pgmap v5176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:20:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:42 smithi042 ceph-mon[32775]: pgmap v5177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:20:42.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:42 smithi164 ceph-mon[36970]: pgmap v5175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:42.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:42 smithi164 ceph-mon[36970]: pgmap v5176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:20:42.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:42 smithi164 ceph-mon[36970]: pgmap v5177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:20:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:42 smithi163 ceph-mon[36873]: pgmap v5175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:42 smithi163 ceph-mon[36873]: pgmap v5176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:20:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:42 smithi163 ceph-mon[36873]: pgmap v5177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:20:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:44 smithi164 ceph-mon[36970]: pgmap v5178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:20:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:44 smithi163 ceph-mon[36873]: pgmap v5178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:20:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:44 smithi042 ceph-mon[32775]: pgmap v5178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:20:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:46 smithi164 ceph-mon[36970]: pgmap v5179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:20:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8676.312322 secs 2024-07-26T09:20:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8694.054239 secs 2024-07-26T09:20:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:46 smithi163 ceph-mon[36873]: pgmap v5179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:20:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8676.312322 secs 2024-07-26T09:20:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8694.054239 secs 2024-07-26T09:20:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:46 smithi042 ceph-mon[32775]: pgmap v5179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:20:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8676.312322 secs 2024-07-26T09:20:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8694.054239 secs 2024-07-26T09:20:47.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:47 smithi164 ceph-mon[36970]: pgmap v5180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:20:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:47 smithi163 ceph-mon[36873]: pgmap v5180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:20:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:47 smithi042 ceph-mon[32775]: pgmap v5180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:20:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:50 smithi042 ceph-mon[32775]: pgmap v5181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:20:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:50 smithi164 ceph-mon[36970]: pgmap v5181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:20:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:50 smithi163 ceph-mon[36873]: pgmap v5181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:20:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8681.312453 secs 2024-07-26T09:20:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8699.054362 secs 2024-07-26T09:20:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8681.312453 secs 2024-07-26T09:20:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8699.054362 secs 2024-07-26T09:20:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8681.312453 secs 2024-07-26T09:20:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8699.054362 secs 2024-07-26T09:20:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:52 smithi042 ceph-mon[32775]: pgmap v5182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:20:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:52 smithi164 ceph-mon[36970]: pgmap v5182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:20:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:52 smithi163 ceph-mon[36873]: pgmap v5182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:20:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:54 smithi042 ceph-mon[32775]: pgmap v5183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:54 smithi164 ceph-mon[36970]: pgmap v5183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:54 smithi163 ceph-mon[36873]: pgmap v5183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:20:56.443 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:20:56.440+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:20:56.443 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:20:56.441+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:20:56.443 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:20:56.441+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:20:56.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:56 smithi042 ceph-mon[32775]: pgmap v5184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:56.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8686.312586 secs 2024-07-26T09:20:56.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8704.054474 secs 2024-07-26T09:20:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:56 smithi164 ceph-mon[36970]: pgmap v5184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8686.312586 secs 2024-07-26T09:20:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8704.054474 secs 2024-07-26T09:20:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:56 smithi163 ceph-mon[36873]: pgmap v5184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:20:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8686.312586 secs 2024-07-26T09:20:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8704.054474 secs 2024-07-26T09:20:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:20:58 smithi042 ceph-mon[32775]: pgmap v5185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:20:58 smithi164 ceph-mon[36970]: pgmap v5185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:20:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:20:58 smithi163 ceph-mon[36873]: pgmap v5185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:00 smithi042 ceph-mon[32775]: pgmap v5186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:00 smithi164 ceph-mon[36970]: pgmap v5186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:00 smithi163 ceph-mon[36873]: pgmap v5186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8691.312683 secs 2024-07-26T09:21:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8709.054607 secs 2024-07-26T09:21:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8691.312683 secs 2024-07-26T09:21:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8709.054607 secs 2024-07-26T09:21:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8691.312683 secs 2024-07-26T09:21:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8709.054607 secs 2024-07-26T09:21:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:02 smithi042 ceph-mon[32775]: pgmap v5187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:02 smithi164 ceph-mon[36970]: pgmap v5187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:03.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:02 smithi163 ceph-mon[36873]: pgmap v5187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:04 smithi042 ceph-mon[32775]: pgmap v5188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:21:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:04 smithi164 ceph-mon[36970]: pgmap v5188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:21:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:04 smithi163 ceph-mon[36873]: pgmap v5188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:21:06.811 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:06 smithi163 ceph-mon[36873]: pgmap v5189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:06.811 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8696.312808 secs 2024-07-26T09:21:06.811 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8714.054745 secs 2024-07-26T09:21:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:06 smithi042 ceph-mon[32775]: pgmap v5189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8696.312808 secs 2024-07-26T09:21:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8714.054745 secs 2024-07-26T09:21:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:06 smithi164 ceph-mon[36970]: pgmap v5189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8696.312808 secs 2024-07-26T09:21:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8714.054745 secs 2024-07-26T09:21:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:08 smithi042 ceph-mon[32775]: pgmap v5190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:08 smithi164 ceph-mon[36970]: pgmap v5190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:08 smithi163 ceph-mon[36873]: pgmap v5190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:10 smithi042 ceph-mon[32775]: pgmap v5191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:10 smithi164 ceph-mon[36970]: pgmap v5191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:10 smithi163 ceph-mon[36873]: pgmap v5191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:11.452 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:21:11.450+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:21:11.452 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:21:11.450+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:21:11.452 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:21:11.450+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:21:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8701.312933 secs 2024-07-26T09:21:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8719.054816 secs 2024-07-26T09:21:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8701.312933 secs 2024-07-26T09:21:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8719.054816 secs 2024-07-26T09:21:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8701.312933 secs 2024-07-26T09:21:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8719.054816 secs 2024-07-26T09:21:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:12 smithi042 ceph-mon[32775]: pgmap v5192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:12 smithi164 ceph-mon[36970]: pgmap v5192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:12 smithi163 ceph-mon[36873]: pgmap v5192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:14 smithi042 ceph-mon[32775]: pgmap v5193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:21:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:14 smithi164 ceph-mon[36970]: pgmap v5193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:21:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:14 smithi163 ceph-mon[36873]: pgmap v5193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:21:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:16 smithi042 ceph-mon[32775]: pgmap v5194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8706.313071 secs 2024-07-26T09:21:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8724.054887 secs 2024-07-26T09:21:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:16 smithi164 ceph-mon[36970]: pgmap v5194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8706.313071 secs 2024-07-26T09:21:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8724.054887 secs 2024-07-26T09:21:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:16 smithi163 ceph-mon[36873]: pgmap v5194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8706.313071 secs 2024-07-26T09:21:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8724.054887 secs 2024-07-26T09:21:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:18 smithi042 ceph-mon[32775]: pgmap v5195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:18 smithi164 ceph-mon[36970]: pgmap v5195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:18 smithi163 ceph-mon[36873]: pgmap v5195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:20 smithi042 ceph-mon[32775]: pgmap v5196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:20 smithi164 ceph-mon[36970]: pgmap v5196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:20 smithi163 ceph-mon[36873]: pgmap v5196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8711.313193 secs 2024-07-26T09:21:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8729.055014 secs 2024-07-26T09:21:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8711.313193 secs 2024-07-26T09:21:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8729.055014 secs 2024-07-26T09:21:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8711.313193 secs 2024-07-26T09:21:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8729.055014 secs 2024-07-26T09:21:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:22 smithi042 ceph-mon[32775]: pgmap v5197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:22 smithi164 ceph-mon[36970]: pgmap v5197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:22 smithi163 ceph-mon[36873]: pgmap v5197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:24 smithi042 ceph-mon[32775]: pgmap v5198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:21:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:24 smithi164 ceph-mon[36970]: pgmap v5198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:21:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:24 smithi163 ceph-mon[36873]: pgmap v5198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:21:26.457 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:21:26.455+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:21:26.458 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:21:26.455+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:21:26.458 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:21:26.455+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:21:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:26 smithi042 ceph-mon[32775]: pgmap v5199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8716.313294 secs 2024-07-26T09:21:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8734.055138 secs 2024-07-26T09:21:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:26 smithi164 ceph-mon[36970]: pgmap v5199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8716.313294 secs 2024-07-26T09:21:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8734.055138 secs 2024-07-26T09:21:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:26 smithi163 ceph-mon[36873]: pgmap v5199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8716.313294 secs 2024-07-26T09:21:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8734.055138 secs 2024-07-26T09:21:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:28 smithi042 ceph-mon[32775]: pgmap v5200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:28 smithi164 ceph-mon[36970]: pgmap v5200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:28 smithi163 ceph-mon[36873]: pgmap v5200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:30 smithi042 ceph-mon[32775]: pgmap v5201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:30 smithi164 ceph-mon[36970]: pgmap v5201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:30 smithi163 ceph-mon[36873]: pgmap v5201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8721.313395 secs 2024-07-26T09:21:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8739.055267 secs 2024-07-26T09:21:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8721.313395 secs 2024-07-26T09:21:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8739.055267 secs 2024-07-26T09:21:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8721.313395 secs 2024-07-26T09:21:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8739.055267 secs 2024-07-26T09:21:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:32 smithi042 ceph-mon[32775]: pgmap v5202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:32 smithi164 ceph-mon[36970]: pgmap v5202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:32 smithi163 ceph-mon[36873]: pgmap v5202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:34 smithi042 ceph-mon[32775]: pgmap v5203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:21:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:34 smithi164 ceph-mon[36970]: pgmap v5203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:21:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:34 smithi163 ceph-mon[36873]: pgmap v5203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:21:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:36 smithi042 ceph-mon[32775]: pgmap v5204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8726.313519 secs 2024-07-26T09:21:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8744.055387 secs 2024-07-26T09:21:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:36 smithi164 ceph-mon[36970]: pgmap v5204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8726.313519 secs 2024-07-26T09:21:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8744.055387 secs 2024-07-26T09:21:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:36 smithi163 ceph-mon[36873]: pgmap v5204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8726.313519 secs 2024-07-26T09:21:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8744.055387 secs 2024-07-26T09:21:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:38 smithi042 ceph-mon[32775]: pgmap v5205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:38 smithi164 ceph-mon[36970]: pgmap v5205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:38 smithi163 ceph-mon[36873]: pgmap v5205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:40 smithi042 ceph-mon[32775]: pgmap v5206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:40 smithi164 ceph-mon[36970]: pgmap v5206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:40 smithi163 ceph-mon[36873]: pgmap v5206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:21:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:21:41.465+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:21:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:21:41.465+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:21:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:21:41.465+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:21:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8731.313674 secs 2024-07-26T09:21:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8749.055506 secs 2024-07-26T09:21:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:41 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:21:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8731.313674 secs 2024-07-26T09:21:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8749.055506 secs 2024-07-26T09:21:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:41 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:21:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8731.313674 secs 2024-07-26T09:21:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8749.055506 secs 2024-07-26T09:21:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:41 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:21:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:42 smithi164 ceph-mon[36970]: pgmap v5207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:42 smithi042 ceph-mon[32775]: pgmap v5207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:42 smithi163 ceph-mon[36873]: pgmap v5207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:44 smithi163 ceph-mon[36873]: pgmap v5208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:21:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:21:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:21:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:44 smithi042 ceph-mon[32775]: pgmap v5208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:21:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:21:44.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:21:44.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:44 smithi164 ceph-mon[36970]: pgmap v5208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:21:44.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:44.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:21:44.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:21:44.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:21:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:45 smithi163 ceph-mon[36873]: pgmap v5209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T09:21:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:45 smithi163 ceph-mon[36873]: pgmap v5210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:21:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:45 smithi042 ceph-mon[32775]: pgmap v5209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T09:21:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:45 smithi042 ceph-mon[32775]: pgmap v5210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:21:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:45 smithi164 ceph-mon[36970]: pgmap v5209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T09:21:45.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:45 smithi164 ceph-mon[36970]: pgmap v5210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:21:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8736.313821 secs 2024-07-26T09:21:46.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8736.313821 secs 2024-07-26T09:21:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8736.313821 secs 2024-07-26T09:21:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:47 smithi163 ceph-mon[36873]: pgmap v5211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:21:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8754.055629 secs 2024-07-26T09:21:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:47 smithi042 ceph-mon[32775]: pgmap v5211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:21:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8754.055629 secs 2024-07-26T09:21:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:47 smithi164 ceph-mon[36970]: pgmap v5211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:21:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8754.055629 secs 2024-07-26T09:21:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:49 smithi163 ceph-mon[36873]: pgmap v5212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:21:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:49 smithi042 ceph-mon[32775]: pgmap v5212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:21:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:49 smithi164 ceph-mon[36970]: pgmap v5212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:21:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:51 smithi163 ceph-mon[36873]: pgmap v5213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:21:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8741.313943 secs 2024-07-26T09:21:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:51 smithi042 ceph-mon[32775]: pgmap v5213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:21:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8741.313943 secs 2024-07-26T09:21:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:51 smithi164 ceph-mon[36970]: pgmap v5213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:21:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8741.313943 secs 2024-07-26T09:21:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8759.055731 secs 2024-07-26T09:21:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8759.055731 secs 2024-07-26T09:21:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8759.055731 secs 2024-07-26T09:21:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:53 smithi042 ceph-mon[32775]: pgmap v5214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:21:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:53 smithi164 ceph-mon[36970]: pgmap v5214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:21:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:53 smithi163 ceph-mon[36873]: pgmap v5214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:21:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:55 smithi042 ceph-mon[32775]: pgmap v5215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:21:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:55 smithi164 ceph-mon[36970]: pgmap v5215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:21:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:55 smithi163 ceph-mon[36873]: pgmap v5215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:21:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:21:56.470+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:21:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:21:56.470+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:21:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:21:56.470+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:21:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8746.314046 secs 2024-07-26T09:21:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8746.314046 secs 2024-07-26T09:21:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8746.314046 secs 2024-07-26T09:21:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:57 smithi042 ceph-mon[32775]: pgmap v5216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8764.055855 secs 2024-07-26T09:21:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:57 smithi164 ceph-mon[36970]: pgmap v5216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:57.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8764.055855 secs 2024-07-26T09:21:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:57 smithi163 ceph-mon[36873]: pgmap v5216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8764.055855 secs 2024-07-26T09:21:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:21:59 smithi042 ceph-mon[32775]: pgmap v5217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:21:59 smithi164 ceph-mon[36970]: pgmap v5217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:21:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:21:59 smithi163 ceph-mon[36873]: pgmap v5217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:01 smithi042 ceph-mon[32775]: pgmap v5218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8751.314188 secs 2024-07-26T09:22:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:01 smithi164 ceph-mon[36970]: pgmap v5218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8751.314188 secs 2024-07-26T09:22:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:01 smithi163 ceph-mon[36873]: pgmap v5218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8751.314188 secs 2024-07-26T09:22:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8769.055989 secs 2024-07-26T09:22:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8769.055989 secs 2024-07-26T09:22:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8769.055989 secs 2024-07-26T09:22:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:03 smithi042 ceph-mon[32775]: pgmap v5219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:03.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:03 smithi164 ceph-mon[36970]: pgmap v5219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:03 smithi163 ceph-mon[36873]: pgmap v5219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:05 smithi042 ceph-mon[32775]: pgmap v5220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:05 smithi164 ceph-mon[36970]: pgmap v5220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:05 smithi163 ceph-mon[36873]: pgmap v5220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8756.314316 secs 2024-07-26T09:22:06.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8756.314316 secs 2024-07-26T09:22:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8756.314316 secs 2024-07-26T09:22:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:07 smithi042 ceph-mon[32775]: pgmap v5221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8774.056052 secs 2024-07-26T09:22:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:07 smithi164 ceph-mon[36970]: pgmap v5221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8774.056052 secs 2024-07-26T09:22:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:07 smithi163 ceph-mon[36873]: pgmap v5221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8774.056052 secs 2024-07-26T09:22:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:09 smithi042 ceph-mon[32775]: pgmap v5222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:09 smithi164 ceph-mon[36970]: pgmap v5222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:09 smithi163 ceph-mon[36873]: pgmap v5222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:22:11.473+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:22:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:22:11.473+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:22:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:22:11.473+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:22:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:11 smithi042 ceph-mon[32775]: pgmap v5223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8761.314435 secs 2024-07-26T09:22:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:11 smithi164 ceph-mon[36970]: pgmap v5223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:11.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8761.314435 secs 2024-07-26T09:22:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:11 smithi163 ceph-mon[36873]: pgmap v5223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8761.314435 secs 2024-07-26T09:22:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8779.056173 secs 2024-07-26T09:22:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8779.056173 secs 2024-07-26T09:22:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8779.056173 secs 2024-07-26T09:22:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:13 smithi042 ceph-mon[32775]: pgmap v5224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:13 smithi164 ceph-mon[36970]: pgmap v5224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:13 smithi163 ceph-mon[36873]: pgmap v5224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:15.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:15 smithi042 ceph-mon[32775]: pgmap v5225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:22:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:15 smithi164 ceph-mon[36970]: pgmap v5225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:22:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:15 smithi163 ceph-mon[36873]: pgmap v5225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:22:16.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8766.314542 secs 2024-07-26T09:22:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8766.314542 secs 2024-07-26T09:22:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8766.314542 secs 2024-07-26T09:22:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:17 smithi042 ceph-mon[32775]: pgmap v5226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:22:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8784.056293 secs 2024-07-26T09:22:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:17 smithi164 ceph-mon[36970]: pgmap v5226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:22:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8784.056293 secs 2024-07-26T09:22:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:17 smithi163 ceph-mon[36873]: pgmap v5226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:22:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8784.056293 secs 2024-07-26T09:22:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:19 smithi042 ceph-mon[32775]: pgmap v5227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:22:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:19 smithi164 ceph-mon[36970]: pgmap v5227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:22:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:19 smithi163 ceph-mon[36873]: pgmap v5227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:22:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:21 smithi042 ceph-mon[32775]: pgmap v5228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:22:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8771.314661 secs 2024-07-26T09:22:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:21 smithi164 ceph-mon[36970]: pgmap v5228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:22:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8771.314661 secs 2024-07-26T09:22:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:21 smithi163 ceph-mon[36873]: pgmap v5228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:22:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8771.314661 secs 2024-07-26T09:22:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8789.056418 secs 2024-07-26T09:22:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8789.056418 secs 2024-07-26T09:22:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8789.056418 secs 2024-07-26T09:22:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:23 smithi042 ceph-mon[32775]: pgmap v5229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:22:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:23 smithi164 ceph-mon[36970]: pgmap v5229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:22:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:23 smithi163 ceph-mon[36873]: pgmap v5229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:22:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:25 smithi042 ceph-mon[32775]: pgmap v5230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:22:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:25 smithi164 ceph-mon[36970]: pgmap v5230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:22:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:25 smithi163 ceph-mon[36873]: pgmap v5230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:22:26.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:22:26.477+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:22:26.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:22:26.477+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:22:26.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:22:26.477+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:22:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8776.314724 secs 2024-07-26T09:22:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8776.314724 secs 2024-07-26T09:22:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8776.314724 secs 2024-07-26T09:22:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:27 smithi042 ceph-mon[32775]: pgmap v5231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8794.056537 secs 2024-07-26T09:22:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:27 smithi164 ceph-mon[36970]: pgmap v5231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8794.056537 secs 2024-07-26T09:22:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:27 smithi163 ceph-mon[36873]: pgmap v5231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8794.056537 secs 2024-07-26T09:22:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:29 smithi042 ceph-mon[32775]: pgmap v5232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:29 smithi164 ceph-mon[36970]: pgmap v5232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:29 smithi163 ceph-mon[36873]: pgmap v5232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:31 smithi042 ceph-mon[32775]: pgmap v5233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8781.314800 secs 2024-07-26T09:22:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:31 smithi164 ceph-mon[36970]: pgmap v5233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8781.314800 secs 2024-07-26T09:22:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:31 smithi163 ceph-mon[36873]: pgmap v5233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8781.314800 secs 2024-07-26T09:22:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8799.056678 secs 2024-07-26T09:22:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8799.056678 secs 2024-07-26T09:22:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8799.056678 secs 2024-07-26T09:22:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:33 smithi042 ceph-mon[32775]: pgmap v5234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:33 smithi164 ceph-mon[36970]: pgmap v5234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:33 smithi163 ceph-mon[36873]: pgmap v5234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:35 smithi042 ceph-mon[32775]: pgmap v5235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:35 smithi164 ceph-mon[36970]: pgmap v5235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:35 smithi163 ceph-mon[36873]: pgmap v5235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8786.314915 secs 2024-07-26T09:22:36.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8786.314915 secs 2024-07-26T09:22:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8786.314915 secs 2024-07-26T09:22:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:37 smithi042 ceph-mon[32775]: pgmap v5236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8804.056814 secs 2024-07-26T09:22:37.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:37 smithi164 ceph-mon[36970]: pgmap v5236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:37.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8804.056814 secs 2024-07-26T09:22:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:37 smithi163 ceph-mon[36873]: pgmap v5236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8804.056814 secs 2024-07-26T09:22:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:39 smithi042 ceph-mon[32775]: pgmap v5237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:39 smithi164 ceph-mon[36970]: pgmap v5237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:39 smithi163 ceph-mon[36873]: pgmap v5237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:22:41.486+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:22:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:22:41.486+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:22:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:22:41.486+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:22:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:41 smithi042 ceph-mon[32775]: pgmap v5238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8791.315036 secs 2024-07-26T09:22:41.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:41 smithi164 ceph-mon[36970]: pgmap v5238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:41.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8791.315036 secs 2024-07-26T09:22:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:41 smithi163 ceph-mon[36873]: pgmap v5238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8791.315036 secs 2024-07-26T09:22:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8809.056955 secs 2024-07-26T09:22:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8809.056955 secs 2024-07-26T09:22:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8809.056955 secs 2024-07-26T09:22:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:43 smithi042 ceph-mon[32775]: pgmap v5239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:43 smithi164 ceph-mon[36970]: pgmap v5239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:43 smithi163 ceph-mon[36873]: pgmap v5239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:22:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:22:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:22:45.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:45 smithi164 ceph-mon[36970]: pgmap v5240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:45.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:22:45.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:22:45.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:45 smithi163 ceph-mon[36873]: pgmap v5240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:22:45.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:22:45.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:45 smithi042 ceph-mon[32775]: pgmap v5240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:22:45.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.874 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.874 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.874 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.874 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:45.874 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:22:45.874 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:22:45.874 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:22:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:46 smithi163 ceph-mon[36873]: pgmap v5241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T09:22:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:46 smithi163 ceph-mon[36873]: pgmap v5242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:22:46.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8796.315195 secs 2024-07-26T09:22:46.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8814.057100 secs 2024-07-26T09:22:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:46 smithi042 ceph-mon[32775]: pgmap v5241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T09:22:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:46 smithi042 ceph-mon[32775]: pgmap v5242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:22:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8796.315195 secs 2024-07-26T09:22:46.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8814.057100 secs 2024-07-26T09:22:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:46 smithi164 ceph-mon[36970]: pgmap v5241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T09:22:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:46 smithi164 ceph-mon[36970]: pgmap v5242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:22:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8796.315195 secs 2024-07-26T09:22:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8814.057100 secs 2024-07-26T09:22:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:48 smithi163 ceph-mon[36873]: pgmap v5243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:22:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:48 smithi042 ceph-mon[32775]: pgmap v5243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:22:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:48 smithi164 ceph-mon[36970]: pgmap v5243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:22:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:50 smithi163 ceph-mon[36873]: pgmap v5244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T09:22:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:50 smithi042 ceph-mon[32775]: pgmap v5244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T09:22:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:50 smithi164 ceph-mon[36970]: pgmap v5244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T09:22:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8801.315342 secs 2024-07-26T09:22:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8819.057220 secs 2024-07-26T09:22:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8801.315342 secs 2024-07-26T09:22:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8819.057220 secs 2024-07-26T09:22:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8801.315342 secs 2024-07-26T09:22:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8819.057220 secs 2024-07-26T09:22:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:52 smithi163 ceph-mon[36873]: pgmap v5245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:22:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:52 smithi042 ceph-mon[32775]: pgmap v5245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:22:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:52 smithi164 ceph-mon[36970]: pgmap v5245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:22:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:54 smithi163 ceph-mon[36873]: pgmap v5246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T09:22:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:54 smithi042 ceph-mon[32775]: pgmap v5246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T09:22:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:54 smithi164 ceph-mon[36970]: pgmap v5246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T09:22:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:22:56.490+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:22:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:22:56.490+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:22:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:22:56.490+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:22:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:56 smithi163 ceph-mon[36873]: pgmap v5247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:22:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8806.315473 secs 2024-07-26T09:22:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8824.057369 secs 2024-07-26T09:22:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:56 smithi042 ceph-mon[32775]: pgmap v5247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:22:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8806.315473 secs 2024-07-26T09:22:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8824.057369 secs 2024-07-26T09:22:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:56 smithi164 ceph-mon[36970]: pgmap v5247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:22:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8806.315473 secs 2024-07-26T09:22:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8824.057369 secs 2024-07-26T09:22:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:22:58 smithi163 ceph-mon[36873]: pgmap v5248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:58.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:22:58 smithi042 ceph-mon[32775]: pgmap v5248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:22:58 smithi164 ceph-mon[36970]: pgmap v5248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:22:59.228 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:22:59.226+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064c0 con 0x7ff680102000 2024-07-26T09:22:59.229 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:22:59.226+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:22:59.229 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:22:59.227+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:22:59.229 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:22:59.227+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064c0 con 0x7ff680102000 2024-07-26T09:23:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:00 smithi163 ceph-mon[36873]: pgmap v5249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:23:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:00 smithi042 ceph-mon[32775]: pgmap v5249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:23:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:00 smithi164 ceph-mon[36970]: pgmap v5249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:23:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8811.315591 secs 2024-07-26T09:23:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8829.057469 secs 2024-07-26T09:23:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8811.315591 secs 2024-07-26T09:23:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8829.057469 secs 2024-07-26T09:23:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8811.315591 secs 2024-07-26T09:23:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8829.057469 secs 2024-07-26T09:23:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:02 smithi042 ceph-mon[32775]: pgmap v5250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:02 smithi164 ceph-mon[36970]: pgmap v5250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:02 smithi163 ceph-mon[36873]: pgmap v5250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:04 smithi042 ceph-mon[32775]: pgmap v5251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:04 smithi164 ceph-mon[36970]: pgmap v5251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:04 smithi163 ceph-mon[36873]: pgmap v5251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:06.811 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:06 smithi163 ceph-mon[36873]: pgmap v5252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:06.811 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8816.315720 secs 2024-07-26T09:23:06.811 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8834.057590 secs 2024-07-26T09:23:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:06 smithi042 ceph-mon[32775]: pgmap v5252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8816.315720 secs 2024-07-26T09:23:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8834.057590 secs 2024-07-26T09:23:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:06 smithi164 ceph-mon[36970]: pgmap v5252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8816.315720 secs 2024-07-26T09:23:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8834.057590 secs 2024-07-26T09:23:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:08 smithi042 ceph-mon[32775]: pgmap v5253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:08 smithi164 ceph-mon[36970]: pgmap v5253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:09.005 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:08 smithi163 ceph-mon[36873]: pgmap v5253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:10 smithi042 ceph-mon[32775]: pgmap v5254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:23:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:10 smithi164 ceph-mon[36970]: pgmap v5254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:23:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:10 smithi163 ceph-mon[36873]: pgmap v5254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:23:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:23:11.500+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:23:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:23:11.500+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:23:11.502 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:23:11.500+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:23:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8821.315842 secs 2024-07-26T09:23:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8839.057720 secs 2024-07-26T09:23:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8821.315842 secs 2024-07-26T09:23:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8839.057720 secs 2024-07-26T09:23:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8821.315842 secs 2024-07-26T09:23:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8839.057720 secs 2024-07-26T09:23:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:12 smithi042 ceph-mon[32775]: pgmap v5255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:12 smithi164 ceph-mon[36970]: pgmap v5255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:12 smithi163 ceph-mon[36873]: pgmap v5255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:14 smithi042 ceph-mon[32775]: pgmap v5256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:14 smithi164 ceph-mon[36970]: pgmap v5256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:14 smithi163 ceph-mon[36873]: pgmap v5256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:16 smithi042 ceph-mon[32775]: pgmap v5257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8826.315963 secs 2024-07-26T09:23:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8844.057848 secs 2024-07-26T09:23:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:16 smithi164 ceph-mon[36970]: pgmap v5257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8826.315963 secs 2024-07-26T09:23:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8844.057848 secs 2024-07-26T09:23:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:16 smithi163 ceph-mon[36873]: pgmap v5257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8826.315963 secs 2024-07-26T09:23:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8844.057848 secs 2024-07-26T09:23:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:18 smithi042 ceph-mon[32775]: pgmap v5258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:18 smithi164 ceph-mon[36970]: pgmap v5258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:18 smithi163 ceph-mon[36873]: pgmap v5258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:20.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:20 smithi042 ceph-mon[32775]: pgmap v5259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:23:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:20 smithi164 ceph-mon[36970]: pgmap v5259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:23:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:20 smithi163 ceph-mon[36873]: pgmap v5259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:23:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8831.316050 secs 2024-07-26T09:23:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8849.057969 secs 2024-07-26T09:23:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8831.316050 secs 2024-07-26T09:23:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8849.057969 secs 2024-07-26T09:23:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8831.316050 secs 2024-07-26T09:23:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8849.057969 secs 2024-07-26T09:23:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:22 smithi042 ceph-mon[32775]: pgmap v5260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:22 smithi164 ceph-mon[36970]: pgmap v5260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:22 smithi163 ceph-mon[36873]: pgmap v5260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:24 smithi042 ceph-mon[32775]: pgmap v5261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:24 smithi164 ceph-mon[36970]: pgmap v5261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:25.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:24 smithi163 ceph-mon[36873]: pgmap v5261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:26.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:23:26.505+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:23:26.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:23:26.505+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:23:26.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:23:26.505+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:23:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:26 smithi042 ceph-mon[32775]: pgmap v5262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8836.316186 secs 2024-07-26T09:23:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8854.058093 secs 2024-07-26T09:23:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:26 smithi164 ceph-mon[36970]: pgmap v5262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8836.316186 secs 2024-07-26T09:23:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8854.058093 secs 2024-07-26T09:23:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:26 smithi163 ceph-mon[36873]: pgmap v5262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8836.316186 secs 2024-07-26T09:23:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8854.058093 secs 2024-07-26T09:23:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:28 smithi042 ceph-mon[32775]: pgmap v5263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:28 smithi164 ceph-mon[36970]: pgmap v5263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:28 smithi163 ceph-mon[36873]: pgmap v5263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:30 smithi042 ceph-mon[32775]: pgmap v5264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:23:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:30 smithi164 ceph-mon[36970]: pgmap v5264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:23:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:30 smithi163 ceph-mon[36873]: pgmap v5264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:23:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8841.316320 secs 2024-07-26T09:23:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8859.058209 secs 2024-07-26T09:23:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8841.316320 secs 2024-07-26T09:23:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8859.058209 secs 2024-07-26T09:23:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8841.316320 secs 2024-07-26T09:23:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8859.058209 secs 2024-07-26T09:23:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:32 smithi042 ceph-mon[32775]: pgmap v5265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:32 smithi164 ceph-mon[36970]: pgmap v5265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:32 smithi163 ceph-mon[36873]: pgmap v5265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:34 smithi042 ceph-mon[32775]: pgmap v5266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:34.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:34 smithi164 ceph-mon[36970]: pgmap v5266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:34 smithi163 ceph-mon[36873]: pgmap v5266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:36 smithi042 ceph-mon[32775]: pgmap v5267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8846.316453 secs 2024-07-26T09:23:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8864.058281 secs 2024-07-26T09:23:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:36 smithi164 ceph-mon[36970]: pgmap v5267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8846.316453 secs 2024-07-26T09:23:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8864.058281 secs 2024-07-26T09:23:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:36 smithi163 ceph-mon[36873]: pgmap v5267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8846.316453 secs 2024-07-26T09:23:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8864.058281 secs 2024-07-26T09:23:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:38 smithi042 ceph-mon[32775]: pgmap v5268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:38 smithi164 ceph-mon[36970]: pgmap v5268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:38 smithi163 ceph-mon[36873]: pgmap v5268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:40 smithi042 ceph-mon[32775]: pgmap v5269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:23:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:40 smithi164 ceph-mon[36970]: pgmap v5269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:23:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:40 smithi163 ceph-mon[36873]: pgmap v5269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:23:41.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:23:41.515+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:23:41.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:23:41.515+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:23:41.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:23:41.515+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:23:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8851.316553 secs 2024-07-26T09:23:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8869.058355 secs 2024-07-26T09:23:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8851.316553 secs 2024-07-26T09:23:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8869.058355 secs 2024-07-26T09:23:42.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8851.316553 secs 2024-07-26T09:23:42.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8869.058355 secs 2024-07-26T09:23:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:42 smithi042 ceph-mon[32775]: pgmap v5270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:42 smithi164 ceph-mon[36970]: pgmap v5270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:42 smithi163 ceph-mon[36873]: pgmap v5270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:44 smithi042 ceph-mon[32775]: pgmap v5271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:44.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:44 smithi164 ceph-mon[36970]: pgmap v5271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:44 smithi163 ceph-mon[36873]: pgmap v5271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:23:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:23:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:23:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:23:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:23:45.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:23:45.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:23:45.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:23:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:23:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:23:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:23:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:23:46.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:46 smithi164 ceph-mon[36970]: pgmap v5272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:46.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:46 smithi164 ceph-mon[36970]: pgmap v5273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:23:46.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:46 smithi164 ceph-mon[36970]: pgmap v5274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:23:46.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8856.316686 secs 2024-07-26T09:23:46.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8874.058487 secs 2024-07-26T09:23:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:46 smithi042 ceph-mon[32775]: pgmap v5272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:46 smithi042 ceph-mon[32775]: pgmap v5273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:23:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:46 smithi042 ceph-mon[32775]: pgmap v5274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:23:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8856.316686 secs 2024-07-26T09:23:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8874.058487 secs 2024-07-26T09:23:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:46 smithi163 ceph-mon[36873]: pgmap v5272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:23:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:46 smithi163 ceph-mon[36873]: pgmap v5273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:23:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:46 smithi163 ceph-mon[36873]: pgmap v5274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:23:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8856.316686 secs 2024-07-26T09:23:47.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8874.058487 secs 2024-07-26T09:23:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:48 smithi042 ceph-mon[32775]: pgmap v5275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:23:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:48 smithi164 ceph-mon[36970]: pgmap v5275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:23:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:48 smithi163 ceph-mon[36873]: pgmap v5275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:23:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:50 smithi042 ceph-mon[32775]: pgmap v5276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:23:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:50 smithi164 ceph-mon[36970]: pgmap v5276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:23:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:50 smithi163 ceph-mon[36873]: pgmap v5276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:23:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8861.316825 secs 2024-07-26T09:23:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8879.058581 secs 2024-07-26T09:23:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8861.316825 secs 2024-07-26T09:23:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8879.058581 secs 2024-07-26T09:23:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8861.316825 secs 2024-07-26T09:23:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8879.058581 secs 2024-07-26T09:23:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:52 smithi042 ceph-mon[32775]: pgmap v5277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:23:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:52 smithi164 ceph-mon[36970]: pgmap v5277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:23:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:52 smithi163 ceph-mon[36873]: pgmap v5277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:23:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:54 smithi042 ceph-mon[32775]: pgmap v5278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:23:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:54 smithi164 ceph-mon[36970]: pgmap v5278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:23:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:54 smithi163 ceph-mon[36873]: pgmap v5278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:23:56.521 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:23:56.519+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:23:56.521 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:23:56.519+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:23:56.521 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:23:56.519+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:23:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:56 smithi042 ceph-mon[32775]: pgmap v5279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:23:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8866.316960 secs 2024-07-26T09:23:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8884.058678 secs 2024-07-26T09:23:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:56 smithi164 ceph-mon[36970]: pgmap v5279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:23:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8866.316960 secs 2024-07-26T09:23:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8884.058678 secs 2024-07-26T09:23:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:56 smithi163 ceph-mon[36873]: pgmap v5279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:23:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8866.316960 secs 2024-07-26T09:23:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8884.058678 secs 2024-07-26T09:23:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:23:58 smithi042 ceph-mon[32775]: pgmap v5280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:23:58 smithi164 ceph-mon[36970]: pgmap v5280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:23:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:23:58 smithi163 ceph-mon[36873]: pgmap v5280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:00 smithi042 ceph-mon[32775]: pgmap v5281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:00 smithi164 ceph-mon[36970]: pgmap v5281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:00 smithi163 ceph-mon[36873]: pgmap v5281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8871.317107 secs 2024-07-26T09:24:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8889.058819 secs 2024-07-26T09:24:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8871.317107 secs 2024-07-26T09:24:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8889.058819 secs 2024-07-26T09:24:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8871.317107 secs 2024-07-26T09:24:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8889.058819 secs 2024-07-26T09:24:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:02 smithi042 ceph-mon[32775]: pgmap v5282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:02 smithi164 ceph-mon[36970]: pgmap v5282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:02 smithi163 ceph-mon[36873]: pgmap v5282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:04.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:04 smithi164 ceph-mon[36970]: pgmap v5283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:24:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:04 smithi163 ceph-mon[36873]: pgmap v5283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:24:05.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:04 smithi042 ceph-mon[32775]: pgmap v5283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:24:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:06 smithi164 ceph-mon[36970]: pgmap v5284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8876.317256 secs 2024-07-26T09:24:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8894.058958 secs 2024-07-26T09:24:07.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:06 smithi163 ceph-mon[36873]: pgmap v5284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:07.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8876.317256 secs 2024-07-26T09:24:07.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8894.058958 secs 2024-07-26T09:24:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:06 smithi042 ceph-mon[32775]: pgmap v5284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8876.317256 secs 2024-07-26T09:24:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8894.058958 secs 2024-07-26T09:24:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:08 smithi164 ceph-mon[36970]: pgmap v5285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:08 smithi163 ceph-mon[36873]: pgmap v5285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:08 smithi042 ceph-mon[32775]: pgmap v5285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:10 smithi164 ceph-mon[36970]: pgmap v5286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:10 smithi163 ceph-mon[36873]: pgmap v5286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:10 smithi042 ceph-mon[32775]: pgmap v5286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:24:11.528+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:24:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:24:11.528+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:24:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:24:11.528+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:24:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8881.317389 secs 2024-07-26T09:24:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8899.059076 secs 2024-07-26T09:24:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8881.317389 secs 2024-07-26T09:24:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8899.059076 secs 2024-07-26T09:24:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8881.317389 secs 2024-07-26T09:24:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8899.059076 secs 2024-07-26T09:24:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:12 smithi164 ceph-mon[36970]: pgmap v5287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:12 smithi163 ceph-mon[36873]: pgmap v5287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:12 smithi042 ceph-mon[32775]: pgmap v5287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:14 smithi164 ceph-mon[36970]: pgmap v5288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:24:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:14 smithi163 ceph-mon[36873]: pgmap v5288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:24:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:14 smithi042 ceph-mon[32775]: pgmap v5288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:24:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:16 smithi164 ceph-mon[36970]: pgmap v5289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8886.317491 secs 2024-07-26T09:24:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8904.059201 secs 2024-07-26T09:24:17.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:16 smithi163 ceph-mon[36873]: pgmap v5289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:17.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8886.317491 secs 2024-07-26T09:24:17.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8904.059201 secs 2024-07-26T09:24:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:16 smithi042 ceph-mon[32775]: pgmap v5289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8886.317491 secs 2024-07-26T09:24:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8904.059201 secs 2024-07-26T09:24:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:18 smithi164 ceph-mon[36970]: pgmap v5290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:18 smithi163 ceph-mon[36873]: pgmap v5290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:19.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:18 smithi042 ceph-mon[32775]: pgmap v5290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:20 smithi164 ceph-mon[36970]: pgmap v5291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:20 smithi163 ceph-mon[36873]: pgmap v5291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:20 smithi042 ceph-mon[32775]: pgmap v5291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8891.317601 secs 2024-07-26T09:24:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8909.059299 secs 2024-07-26T09:24:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8891.317601 secs 2024-07-26T09:24:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8909.059299 secs 2024-07-26T09:24:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8891.317601 secs 2024-07-26T09:24:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8909.059299 secs 2024-07-26T09:24:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:22 smithi164 ceph-mon[36970]: pgmap v5292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:22 smithi163 ceph-mon[36873]: pgmap v5292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:22 smithi042 ceph-mon[32775]: pgmap v5292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:23 smithi164 ceph-mon[36970]: pgmap v5293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:24:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:23 smithi163 ceph-mon[36873]: pgmap v5293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:24:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:23 smithi042 ceph-mon[32775]: pgmap v5293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:24:26.535 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:24:26.532+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:24:26.535 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:24:26.533+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:24:26.535 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:24:26.533+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:24:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:26 smithi042 ceph-mon[32775]: pgmap v5294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8896.317756 secs 2024-07-26T09:24:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8914.059402 secs 2024-07-26T09:24:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:26 smithi164 ceph-mon[36970]: pgmap v5294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8896.317756 secs 2024-07-26T09:24:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8914.059402 secs 2024-07-26T09:24:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:26 smithi163 ceph-mon[36873]: pgmap v5294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8896.317756 secs 2024-07-26T09:24:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8914.059402 secs 2024-07-26T09:24:28.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:28 smithi042 ceph-mon[32775]: pgmap v5295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:28 smithi164 ceph-mon[36970]: pgmap v5295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:28 smithi163 ceph-mon[36873]: pgmap v5295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:24:29.241+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T09:24:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:24:29.241+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:24:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:24:29.241+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:24:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:24:29.241+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T09:24:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:30 smithi042 ceph-mon[32775]: pgmap v5296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:30 smithi164 ceph-mon[36970]: pgmap v5296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:30 smithi163 ceph-mon[36873]: pgmap v5296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:31.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8901.317892 secs 2024-07-26T09:24:31.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8919.059489 secs 2024-07-26T09:24:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8901.317892 secs 2024-07-26T09:24:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8919.059489 secs 2024-07-26T09:24:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8901.317892 secs 2024-07-26T09:24:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8919.059489 secs 2024-07-26T09:24:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:32 smithi042 ceph-mon[32775]: pgmap v5297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:32 smithi164 ceph-mon[36970]: pgmap v5297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:32 smithi163 ceph-mon[36873]: pgmap v5297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:34 smithi042 ceph-mon[32775]: pgmap v5298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:24:34.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:34 smithi164 ceph-mon[36970]: pgmap v5298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:24:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:34 smithi163 ceph-mon[36873]: pgmap v5298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:24:36.811 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:36 smithi163 ceph-mon[36873]: pgmap v5299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:36.811 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8906.318028 secs 2024-07-26T09:24:36.811 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8924.059606 secs 2024-07-26T09:24:36.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:36 smithi042 ceph-mon[32775]: pgmap v5299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:36.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8906.318028 secs 2024-07-26T09:24:36.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8924.059606 secs 2024-07-26T09:24:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:36 smithi164 ceph-mon[36970]: pgmap v5299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8906.318028 secs 2024-07-26T09:24:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8924.059606 secs 2024-07-26T09:24:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:38 smithi042 ceph-mon[32775]: pgmap v5300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:38 smithi164 ceph-mon[36970]: pgmap v5300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:38 smithi163 ceph-mon[36873]: pgmap v5300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:40 smithi042 ceph-mon[32775]: pgmap v5301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:40 smithi164 ceph-mon[36970]: pgmap v5301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:40 smithi163 ceph-mon[36873]: pgmap v5301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:24:41.542+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:24:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:24:41.542+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:24:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:24:41.542+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:24:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8911.318165 secs 2024-07-26T09:24:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8929.059730 secs 2024-07-26T09:24:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8911.318165 secs 2024-07-26T09:24:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8929.059730 secs 2024-07-26T09:24:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8911.318165 secs 2024-07-26T09:24:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8929.059730 secs 2024-07-26T09:24:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:42 smithi042 ceph-mon[32775]: pgmap v5302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:42 smithi164 ceph-mon[36970]: pgmap v5302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:42 smithi163 ceph-mon[36873]: pgmap v5302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:44 smithi042 ceph-mon[32775]: pgmap v5303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:24:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:44 smithi164 ceph-mon[36970]: pgmap v5303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:24:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:44 smithi163 ceph-mon[36873]: pgmap v5303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:24:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:46 smithi042 ceph-mon[32775]: pgmap v5304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:24:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8916.318273 secs 2024-07-26T09:24:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:24:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:24:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:24:46.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8934.059862 secs 2024-07-26T09:24:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:46 smithi164 ceph-mon[36970]: pgmap v5304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:24:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8916.318273 secs 2024-07-26T09:24:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:24:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:24:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:24:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8934.059862 secs 2024-07-26T09:24:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:46 smithi163 ceph-mon[36873]: pgmap v5304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:24:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:24:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8916.318273 secs 2024-07-26T09:24:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:24:47.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:24:47.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:24:47.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8934.059862 secs 2024-07-26T09:24:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:47 smithi042 ceph-mon[32775]: pgmap v5305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:24:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:47 smithi042 ceph-mon[32775]: pgmap v5306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:24:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:47 smithi164 ceph-mon[36970]: pgmap v5305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:24:47.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:47 smithi164 ceph-mon[36970]: pgmap v5306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:24:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:47 smithi163 ceph-mon[36873]: pgmap v5305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:24:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:47 smithi163 ceph-mon[36873]: pgmap v5306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:24:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:49 smithi042 ceph-mon[32775]: pgmap v5307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:24:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:49 smithi164 ceph-mon[36970]: pgmap v5307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:24:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:49 smithi163 ceph-mon[36873]: pgmap v5307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:24:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:51 smithi042 ceph-mon[32775]: pgmap v5308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:24:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8921.318413 secs 2024-07-26T09:24:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8939.059949 secs 2024-07-26T09:24:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:51 smithi164 ceph-mon[36970]: pgmap v5308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:24:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8921.318413 secs 2024-07-26T09:24:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8939.059949 secs 2024-07-26T09:24:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:51 smithi163 ceph-mon[36873]: pgmap v5308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:24:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8921.318413 secs 2024-07-26T09:24:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8939.059949 secs 2024-07-26T09:24:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:53 smithi042 ceph-mon[32775]: pgmap v5309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:24:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:53 smithi164 ceph-mon[36970]: pgmap v5309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:24:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:53 smithi163 ceph-mon[36873]: pgmap v5309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:24:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:55 smithi042 ceph-mon[32775]: pgmap v5310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:24:55.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:55 smithi164 ceph-mon[36970]: pgmap v5310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:24:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:55 smithi163 ceph-mon[36873]: pgmap v5310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:24:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:24:56.547+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:24:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:24:56.547+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:24:56.549 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:24:56.547+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:24:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8926.318548 secs 2024-07-26T09:24:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8944.060064 secs 2024-07-26T09:24:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8926.318548 secs 2024-07-26T09:24:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8944.060064 secs 2024-07-26T09:24:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8926.318548 secs 2024-07-26T09:24:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8944.060064 secs 2024-07-26T09:24:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:57 smithi042 ceph-mon[32775]: pgmap v5311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:24:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:57 smithi164 ceph-mon[36970]: pgmap v5311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:24:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:57 smithi163 ceph-mon[36873]: pgmap v5311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:24:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:24:59 smithi042 ceph-mon[32775]: pgmap v5312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:24:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:24:59 smithi164 ceph-mon[36970]: pgmap v5312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:24:59 smithi163 ceph-mon[36873]: pgmap v5312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:01 smithi042 ceph-mon[32775]: pgmap v5313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8931.318679 secs 2024-07-26T09:25:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8949.060187 secs 2024-07-26T09:25:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:01 smithi164 ceph-mon[36970]: pgmap v5313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8931.318679 secs 2024-07-26T09:25:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8949.060187 secs 2024-07-26T09:25:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:01 smithi163 ceph-mon[36873]: pgmap v5313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8931.318679 secs 2024-07-26T09:25:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8949.060187 secs 2024-07-26T09:25:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:03 smithi042 ceph-mon[32775]: pgmap v5314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:03 smithi164 ceph-mon[36970]: pgmap v5314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:03 smithi163 ceph-mon[36873]: pgmap v5314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:05 smithi042 ceph-mon[32775]: pgmap v5315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:05 smithi164 ceph-mon[36970]: pgmap v5315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:05 smithi163 ceph-mon[36873]: pgmap v5315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8936.318813 secs 2024-07-26T09:25:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8954.060280 secs 2024-07-26T09:25:06.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8936.318813 secs 2024-07-26T09:25:06.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8954.060280 secs 2024-07-26T09:25:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8936.318813 secs 2024-07-26T09:25:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8954.060280 secs 2024-07-26T09:25:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:07 smithi042 ceph-mon[32775]: pgmap v5316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:07 smithi164 ceph-mon[36970]: pgmap v5316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:07 smithi163 ceph-mon[36873]: pgmap v5316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:09 smithi042 ceph-mon[32775]: pgmap v5317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:09 smithi164 ceph-mon[36970]: pgmap v5317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:09 smithi163 ceph-mon[36873]: pgmap v5317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:11.558 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:11.557+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:25:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:11.557+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:25:11.559 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:11.557+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:25:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:11 smithi042 ceph-mon[32775]: pgmap v5318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8941.318949 secs 2024-07-26T09:25:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8959.060402 secs 2024-07-26T09:25:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:11 smithi164 ceph-mon[36970]: pgmap v5318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8941.318949 secs 2024-07-26T09:25:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8959.060402 secs 2024-07-26T09:25:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:11 smithi163 ceph-mon[36873]: pgmap v5318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8941.318949 secs 2024-07-26T09:25:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8959.060402 secs 2024-07-26T09:25:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:13 smithi042 ceph-mon[32775]: pgmap v5319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:13 smithi164 ceph-mon[36970]: pgmap v5319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:13 smithi163 ceph-mon[36873]: pgmap v5319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:15 smithi042 ceph-mon[32775]: pgmap v5320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:15 smithi164 ceph-mon[36970]: pgmap v5320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:15 smithi163 ceph-mon[36873]: pgmap v5320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:16.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8946.319114 secs 2024-07-26T09:25:16.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8964.060521 secs 2024-07-26T09:25:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8946.319114 secs 2024-07-26T09:25:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8964.060521 secs 2024-07-26T09:25:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8946.319114 secs 2024-07-26T09:25:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8964.060521 secs 2024-07-26T09:25:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:17 smithi042 ceph-mon[32775]: pgmap v5321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:17 smithi164 ceph-mon[36970]: pgmap v5321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:17 smithi163 ceph-mon[36873]: pgmap v5321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:19 smithi042 ceph-mon[32775]: pgmap v5322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:19 smithi164 ceph-mon[36970]: pgmap v5322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:19 smithi163 ceph-mon[36873]: pgmap v5322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:21 smithi042 ceph-mon[32775]: pgmap v5323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8951.319254 secs 2024-07-26T09:25:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8969.060662 secs 2024-07-26T09:25:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:21 smithi164 ceph-mon[36970]: pgmap v5323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8951.319254 secs 2024-07-26T09:25:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8969.060662 secs 2024-07-26T09:25:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:21 smithi163 ceph-mon[36873]: pgmap v5323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8951.319254 secs 2024-07-26T09:25:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8969.060662 secs 2024-07-26T09:25:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:23 smithi042 ceph-mon[32775]: pgmap v5324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:23.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:23 smithi164 ceph-mon[36970]: pgmap v5324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:23 smithi163 ceph-mon[36873]: pgmap v5324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:25 smithi042 ceph-mon[32775]: pgmap v5325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:25 smithi164 ceph-mon[36970]: pgmap v5325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:25 smithi163 ceph-mon[36873]: pgmap v5325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:26.562+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:25:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:26.562+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:25:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:26.562+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:25:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8956.319376 secs 2024-07-26T09:25:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8974.060803 secs 2024-07-26T09:25:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8956.319376 secs 2024-07-26T09:25:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8974.060803 secs 2024-07-26T09:25:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8956.319376 secs 2024-07-26T09:25:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8974.060803 secs 2024-07-26T09:25:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:27 smithi042 ceph-mon[32775]: pgmap v5326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:27 smithi164 ceph-mon[36970]: pgmap v5326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:27 smithi163 ceph-mon[36873]: pgmap v5326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:29 smithi042 ceph-mon[32775]: pgmap v5327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:29 smithi164 ceph-mon[36970]: pgmap v5327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:29 smithi163 ceph-mon[36873]: pgmap v5327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:31 smithi164 ceph-mon[36970]: pgmap v5328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8961.319503 secs 2024-07-26T09:25:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8979.060946 secs 2024-07-26T09:25:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:31 smithi163 ceph-mon[36873]: pgmap v5328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8961.319503 secs 2024-07-26T09:25:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8979.060946 secs 2024-07-26T09:25:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:31 smithi042 ceph-mon[32775]: pgmap v5328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8961.319503 secs 2024-07-26T09:25:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8979.060946 secs 2024-07-26T09:25:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:33 smithi164 ceph-mon[36970]: pgmap v5329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:33 smithi163 ceph-mon[36873]: pgmap v5329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:33 smithi042 ceph-mon[32775]: pgmap v5329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:35 smithi164 ceph-mon[36970]: pgmap v5330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:35 smithi163 ceph-mon[36873]: pgmap v5330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:35 smithi042 ceph-mon[32775]: pgmap v5330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8966.319648 secs 2024-07-26T09:25:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8984.061011 secs 2024-07-26T09:25:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8966.319648 secs 2024-07-26T09:25:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8984.061011 secs 2024-07-26T09:25:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8966.319648 secs 2024-07-26T09:25:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8984.061011 secs 2024-07-26T09:25:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:37 smithi164 ceph-mon[36970]: pgmap v5331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:37 smithi163 ceph-mon[36873]: pgmap v5331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:37 smithi042 ceph-mon[32775]: pgmap v5331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:39 smithi164 ceph-mon[36970]: pgmap v5332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:39 smithi163 ceph-mon[36873]: pgmap v5332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:39 smithi042 ceph-mon[32775]: pgmap v5332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:41.566+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:25:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:41.566+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:25:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:41.566+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:25:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:41 smithi164 ceph-mon[36970]: pgmap v5333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8971.319786 secs 2024-07-26T09:25:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8989.061136 secs 2024-07-26T09:25:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:41 smithi163 ceph-mon[36873]: pgmap v5333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8971.319786 secs 2024-07-26T09:25:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8989.061136 secs 2024-07-26T09:25:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:41 smithi042 ceph-mon[32775]: pgmap v5333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8971.319786 secs 2024-07-26T09:25:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8989.061136 secs 2024-07-26T09:25:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:43 smithi164 ceph-mon[36970]: pgmap v5334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:43 smithi163 ceph-mon[36873]: pgmap v5334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:43 smithi042 ceph-mon[32775]: pgmap v5334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:25:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:45 smithi164 ceph-mon[36970]: pgmap v5335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:45 smithi163 ceph-mon[36873]: pgmap v5335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:45 smithi042 ceph-mon[32775]: pgmap v5335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8976.319923 secs 2024-07-26T09:25:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:25:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8994.061207 secs 2024-07-26T09:25:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:25:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:25:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:25:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8976.319923 secs 2024-07-26T09:25:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:25:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8994.061207 secs 2024-07-26T09:25:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:25:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:25:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:25:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8976.319923 secs 2024-07-26T09:25:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:25:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8994.061207 secs 2024-07-26T09:25:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:25:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:25:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:25:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:47 smithi164 ceph-mon[36970]: pgmap v5336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:47.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:47 smithi164 ceph-mon[36970]: pgmap v5337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:25:47.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:47 smithi164 ceph-mon[36970]: pgmap v5338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:25:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:47 smithi163 ceph-mon[36873]: pgmap v5336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:47 smithi163 ceph-mon[36873]: pgmap v5337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:25:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:47 smithi163 ceph-mon[36873]: pgmap v5338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:25:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:47 smithi042 ceph-mon[32775]: pgmap v5336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:25:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:47 smithi042 ceph-mon[32775]: pgmap v5337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:25:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:47 smithi042 ceph-mon[32775]: pgmap v5338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:25:49.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:49 smithi164 ceph-mon[36970]: pgmap v5339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:25:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:49 smithi163 ceph-mon[36873]: pgmap v5339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:25:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:49 smithi042 ceph-mon[32775]: pgmap v5339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:25:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:51 smithi164 ceph-mon[36970]: pgmap v5340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:25:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8981.320058 secs 2024-07-26T09:25:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8999.061335 secs 2024-07-26T09:25:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:51 smithi163 ceph-mon[36873]: pgmap v5340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:25:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8981.320058 secs 2024-07-26T09:25:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8999.061335 secs 2024-07-26T09:25:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:51 smithi042 ceph-mon[32775]: pgmap v5340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:25:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8981.320058 secs 2024-07-26T09:25:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8999.061335 secs 2024-07-26T09:25:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:53 smithi164 ceph-mon[36970]: pgmap v5341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:25:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:53 smithi163 ceph-mon[36873]: pgmap v5341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:25:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:53 smithi042 ceph-mon[32775]: pgmap v5341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:25:55.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:55 smithi164 ceph-mon[36970]: pgmap v5342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:25:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:55 smithi163 ceph-mon[36873]: pgmap v5342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:25:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:55 smithi042 ceph-mon[32775]: pgmap v5342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:25:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:56.571+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:25:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:56.571+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:25:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:56.571+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:25:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8986.320191 secs 2024-07-26T09:25:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9004.061412 secs 2024-07-26T09:25:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8986.320191 secs 2024-07-26T09:25:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9004.061412 secs 2024-07-26T09:25:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8986.320191 secs 2024-07-26T09:25:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9004.061412 secs 2024-07-26T09:25:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:57.875+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000744) for more than 900000000 us, fault. 2024-07-26T09:25:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:57.875+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff6780094b0 tx=0x7ff67806c040 comp rx=0 tx=0).stop 2024-07-26T09:25:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:57.875+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T09:25:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:57.875+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T09:25:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:57.875+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T09:25:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:57.875+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:25:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:57.875+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5000 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff67802b060 tx=0x7ff678055040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T09:25:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:57 smithi164 ceph-mon[36970]: pgmap v5343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:25:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:57 smithi163 ceph-mon[36873]: pgmap v5343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:25:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:57 smithi042 ceph-mon[32775]: pgmap v5343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:25:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:25:58 smithi163 ceph-mon[36873]: pgmap v5344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:25:58 smithi042 ceph-mon[32775]: pgmap v5344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:25:58 smithi164 ceph-mon[36970]: pgmap v5344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:25:59.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:59.256+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T09:25:59.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:59.256+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:25:59.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:59.256+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T09:25:59.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:25:59.256+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:26:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:01 smithi042 ceph-mon[32775]: pgmap v5345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8991.320315 secs 2024-07-26T09:26:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9009.061527 secs 2024-07-26T09:26:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:01 smithi164 ceph-mon[36970]: pgmap v5345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8991.320315 secs 2024-07-26T09:26:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9009.061527 secs 2024-07-26T09:26:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:01 smithi163 ceph-mon[36873]: pgmap v5345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8991.320315 secs 2024-07-26T09:26:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9009.061527 secs 2024-07-26T09:26:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:03 smithi042 ceph-mon[32775]: pgmap v5346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:03 smithi164 ceph-mon[36970]: pgmap v5346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:03 smithi163 ceph-mon[36873]: pgmap v5346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:05 smithi042 ceph-mon[32775]: pgmap v5347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:05 smithi164 ceph-mon[36970]: pgmap v5347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:05 smithi163 ceph-mon[36873]: pgmap v5347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 8996.320385 secs 2024-07-26T09:26:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9014.061670 secs 2024-07-26T09:26:06.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 8996.320385 secs 2024-07-26T09:26:06.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9014.061670 secs 2024-07-26T09:26:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 8996.320385 secs 2024-07-26T09:26:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9014.061670 secs 2024-07-26T09:26:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:07 smithi042 ceph-mon[32775]: pgmap v5348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:07 smithi164 ceph-mon[36970]: pgmap v5348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:07 smithi163 ceph-mon[36873]: pgmap v5348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:09 smithi042 ceph-mon[32775]: pgmap v5349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:26:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:09 smithi164 ceph-mon[36970]: pgmap v5349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:26:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:09 smithi163 ceph-mon[36873]: pgmap v5349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:26:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:26:11.581+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:26:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:26:11.581+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:26:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:26:11.581+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:26:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:11 smithi042 ceph-mon[32775]: pgmap v5350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9001.320459 secs 2024-07-26T09:26:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9019.061790 secs 2024-07-26T09:26:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:11 smithi164 ceph-mon[36970]: pgmap v5350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9001.320459 secs 2024-07-26T09:26:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9019.061790 secs 2024-07-26T09:26:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:11 smithi163 ceph-mon[36873]: pgmap v5350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9001.320459 secs 2024-07-26T09:26:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9019.061790 secs 2024-07-26T09:26:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:13 smithi042 ceph-mon[32775]: pgmap v5351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:13 smithi164 ceph-mon[36970]: pgmap v5351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:13 smithi163 ceph-mon[36873]: pgmap v5351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:15 smithi042 ceph-mon[32775]: pgmap v5352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:15 smithi164 ceph-mon[36970]: pgmap v5352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:15 smithi163 ceph-mon[36873]: pgmap v5352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9006.320574 secs 2024-07-26T09:26:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9024.061949 secs 2024-07-26T09:26:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9006.320574 secs 2024-07-26T09:26:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9024.061949 secs 2024-07-26T09:26:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9006.320574 secs 2024-07-26T09:26:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9024.061949 secs 2024-07-26T09:26:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:17 smithi164 ceph-mon[36970]: pgmap v5353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:17 smithi163 ceph-mon[36873]: pgmap v5353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:17 smithi042 ceph-mon[32775]: pgmap v5353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:19 smithi164 ceph-mon[36970]: pgmap v5354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:26:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:19 smithi163 ceph-mon[36873]: pgmap v5354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:26:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:19 smithi042 ceph-mon[32775]: pgmap v5354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:26:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:21 smithi164 ceph-mon[36970]: pgmap v5355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9011.320711 secs 2024-07-26T09:26:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9029.062048 secs 2024-07-26T09:26:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:21 smithi163 ceph-mon[36873]: pgmap v5355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9011.320711 secs 2024-07-26T09:26:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9029.062048 secs 2024-07-26T09:26:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:21 smithi042 ceph-mon[32775]: pgmap v5355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9011.320711 secs 2024-07-26T09:26:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9029.062048 secs 2024-07-26T09:26:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:23 smithi164 ceph-mon[36970]: pgmap v5356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:23 smithi163 ceph-mon[36873]: pgmap v5356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:23 smithi042 ceph-mon[32775]: pgmap v5356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:25 smithi164 ceph-mon[36970]: pgmap v5357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:25 smithi163 ceph-mon[36873]: pgmap v5357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:25 smithi042 ceph-mon[32775]: pgmap v5357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:26:26.583+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:26:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:26:26.583+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:26:26.585 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:26:26.583+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:26:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9016.320837 secs 2024-07-26T09:26:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9034.062113 secs 2024-07-26T09:26:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9016.320837 secs 2024-07-26T09:26:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9034.062113 secs 2024-07-26T09:26:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9016.320837 secs 2024-07-26T09:26:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9034.062113 secs 2024-07-26T09:26:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:27 smithi164 ceph-mon[36970]: pgmap v5358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:27 smithi163 ceph-mon[36873]: pgmap v5358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:27 smithi042 ceph-mon[32775]: pgmap v5358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:29 smithi164 ceph-mon[36970]: pgmap v5359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:26:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:29 smithi163 ceph-mon[36873]: pgmap v5359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:26:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:29 smithi042 ceph-mon[32775]: pgmap v5359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:26:31.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:31 smithi042 ceph-mon[32775]: pgmap v5360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:31.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9021.320966 secs 2024-07-26T09:26:31.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9039.062208 secs 2024-07-26T09:26:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:31 smithi164 ceph-mon[36970]: pgmap v5360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9021.320966 secs 2024-07-26T09:26:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9039.062208 secs 2024-07-26T09:26:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:31 smithi163 ceph-mon[36873]: pgmap v5360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9021.320966 secs 2024-07-26T09:26:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9039.062208 secs 2024-07-26T09:26:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:33 smithi164 ceph-mon[36970]: pgmap v5361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:33 smithi163 ceph-mon[36873]: pgmap v5361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:33 smithi042 ceph-mon[32775]: pgmap v5361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:35 smithi164 ceph-mon[36970]: pgmap v5362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:35 smithi163 ceph-mon[36873]: pgmap v5362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:35 smithi042 ceph-mon[32775]: pgmap v5362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9026.321112 secs 2024-07-26T09:26:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9044.062367 secs 2024-07-26T09:26:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9026.321112 secs 2024-07-26T09:26:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9044.062367 secs 2024-07-26T09:26:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9026.321112 secs 2024-07-26T09:26:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9044.062367 secs 2024-07-26T09:26:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:37 smithi164 ceph-mon[36970]: pgmap v5363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:37 smithi163 ceph-mon[36873]: pgmap v5363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:37 smithi042 ceph-mon[32775]: pgmap v5363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:39 smithi164 ceph-mon[36970]: pgmap v5364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:26:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:39 smithi163 ceph-mon[36873]: pgmap v5364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:26:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:39 smithi042 ceph-mon[32775]: pgmap v5364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:26:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:26:41.593+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:26:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:26:41.593+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:26:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:26:41.593+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:26:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:41 smithi164 ceph-mon[36970]: pgmap v5365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9031.321264 secs 2024-07-26T09:26:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9049.062467 secs 2024-07-26T09:26:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:41 smithi163 ceph-mon[36873]: pgmap v5365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9031.321264 secs 2024-07-26T09:26:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9049.062467 secs 2024-07-26T09:26:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:41 smithi042 ceph-mon[32775]: pgmap v5365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9031.321264 secs 2024-07-26T09:26:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9049.062467 secs 2024-07-26T09:26:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:43 smithi164 ceph-mon[36970]: pgmap v5366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:43 smithi163 ceph-mon[36873]: pgmap v5366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:43 smithi042 ceph-mon[32775]: pgmap v5366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:45 smithi164 ceph-mon[36970]: pgmap v5367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:45 smithi163 ceph-mon[36873]: pgmap v5367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:45 smithi042 ceph-mon[32775]: pgmap v5367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:26:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9036.321395 secs 2024-07-26T09:26:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9054.062552 secs 2024-07-26T09:26:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:26:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9036.321395 secs 2024-07-26T09:26:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9054.062552 secs 2024-07-26T09:26:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:26:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9036.321395 secs 2024-07-26T09:26:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9054.062552 secs 2024-07-26T09:26:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:26:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:47 smithi164 ceph-mon[36970]: pgmap v5368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:47.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:26:47.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:26:47.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:26:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:47 smithi163 ceph-mon[36873]: pgmap v5368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:26:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:26:48.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:26:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:47 smithi042 ceph-mon[32775]: pgmap v5368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:26:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:26:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:26:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:26:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:48 smithi164 ceph-mon[36970]: pgmap v5369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:26:48.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:48 smithi164 ceph-mon[36970]: pgmap v5370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:26:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:48 smithi163 ceph-mon[36873]: pgmap v5369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:26:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:48 smithi163 ceph-mon[36873]: pgmap v5370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:26:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:48 smithi042 ceph-mon[32775]: pgmap v5369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:26:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:48 smithi042 ceph-mon[32775]: pgmap v5370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:26:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:49 smithi163 ceph-mon[36873]: pgmap v5371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:26:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:49 smithi042 ceph-mon[32775]: pgmap v5371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:26:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:49 smithi164 ceph-mon[36970]: pgmap v5371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:26:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9041.321520 secs 2024-07-26T09:26:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9041.321520 secs 2024-07-26T09:26:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9041.321520 secs 2024-07-26T09:26:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:52 smithi042 ceph-mon[32775]: pgmap v5372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:26:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9059.062678 secs 2024-07-26T09:26:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:52 smithi164 ceph-mon[36970]: pgmap v5372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:26:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9059.062678 secs 2024-07-26T09:26:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:52 smithi163 ceph-mon[36873]: pgmap v5372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:26:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9059.062678 secs 2024-07-26T09:26:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:54 smithi164 ceph-mon[36970]: pgmap v5373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:26:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:54 smithi163 ceph-mon[36873]: pgmap v5373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:26:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:54 smithi042 ceph-mon[32775]: pgmap v5373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:26:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:56 smithi164 ceph-mon[36970]: pgmap v5374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:26:56.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9046.321643 secs 2024-07-26T09:26:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:56 smithi163 ceph-mon[36873]: pgmap v5374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:26:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9046.321643 secs 2024-07-26T09:26:56.599 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:26:56.597+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:26:56.599 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:26:56.597+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:26:56.599 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:26:56.597+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:26:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:56 smithi042 ceph-mon[32775]: pgmap v5374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:26:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9046.321643 secs 2024-07-26T09:26:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9064.062799 secs 2024-07-26T09:26:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9064.062799 secs 2024-07-26T09:26:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9064.062799 secs 2024-07-26T09:26:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:26:58 smithi164 ceph-mon[36970]: pgmap v5375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:26:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:26:58 smithi163 ceph-mon[36873]: pgmap v5375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:26:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:26:58 smithi042 ceph-mon[32775]: pgmap v5375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:27:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:00 smithi164 ceph-mon[36970]: pgmap v5376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:00 smithi163 ceph-mon[36873]: pgmap v5376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:00 smithi042 ceph-mon[32775]: pgmap v5376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9051.321765 secs 2024-07-26T09:27:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9051.321765 secs 2024-07-26T09:27:01.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9051.321765 secs 2024-07-26T09:27:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:02 smithi164 ceph-mon[36970]: pgmap v5377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9069.062924 secs 2024-07-26T09:27:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:02 smithi163 ceph-mon[36873]: pgmap v5377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9069.062924 secs 2024-07-26T09:27:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:02 smithi042 ceph-mon[32775]: pgmap v5377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9069.062924 secs 2024-07-26T09:27:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:04 smithi164 ceph-mon[36970]: pgmap v5378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:04 smithi163 ceph-mon[36873]: pgmap v5378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:04 smithi042 ceph-mon[32775]: pgmap v5378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:06 smithi164 ceph-mon[36970]: pgmap v5379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:06.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9056.321887 secs 2024-07-26T09:27:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:06 smithi163 ceph-mon[36873]: pgmap v5379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9056.321887 secs 2024-07-26T09:27:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:06 smithi042 ceph-mon[32775]: pgmap v5379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9056.321887 secs 2024-07-26T09:27:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9074.062954 secs 2024-07-26T09:27:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9074.062954 secs 2024-07-26T09:27:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9074.062954 secs 2024-07-26T09:27:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:08 smithi164 ceph-mon[36970]: pgmap v5380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:08 smithi163 ceph-mon[36873]: pgmap v5380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:08 smithi042 ceph-mon[32775]: pgmap v5380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:10 smithi164 ceph-mon[36970]: pgmap v5381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:10 smithi163 ceph-mon[36873]: pgmap v5381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:10 smithi042 ceph-mon[32775]: pgmap v5381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9061.322036 secs 2024-07-26T09:27:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:27:11.607+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:27:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:27:11.607+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:27:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:27:11.607+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:27:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9061.322036 secs 2024-07-26T09:27:11.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9061.322036 secs 2024-07-26T09:27:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:12 smithi163 ceph-mon[36873]: pgmap v5382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9079.063079 secs 2024-07-26T09:27:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:12 smithi042 ceph-mon[32775]: pgmap v5382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9079.063079 secs 2024-07-26T09:27:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:12 smithi164 ceph-mon[36970]: pgmap v5382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9079.063079 secs 2024-07-26T09:27:14.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:14 smithi163 ceph-mon[36873]: pgmap v5383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:14 smithi042 ceph-mon[32775]: pgmap v5383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:14.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:14 smithi164 ceph-mon[36970]: pgmap v5383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:16 smithi163 ceph-mon[36873]: pgmap v5384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9066.322157 secs 2024-07-26T09:27:16.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:16 smithi164 ceph-mon[36970]: pgmap v5384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:16.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9066.322157 secs 2024-07-26T09:27:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:16 smithi042 ceph-mon[32775]: pgmap v5384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9066.322157 secs 2024-07-26T09:27:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9084.063204 secs 2024-07-26T09:27:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9084.063204 secs 2024-07-26T09:27:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9084.063204 secs 2024-07-26T09:27:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:18 smithi163 ceph-mon[36873]: pgmap v5385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:18 smithi042 ceph-mon[32775]: pgmap v5385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:18 smithi164 ceph-mon[36970]: pgmap v5385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:20.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:20 smithi163 ceph-mon[36873]: pgmap v5386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:20 smithi042 ceph-mon[32775]: pgmap v5386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:20 smithi164 ceph-mon[36970]: pgmap v5386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9071.322280 secs 2024-07-26T09:27:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9071.322280 secs 2024-07-26T09:27:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9071.322280 secs 2024-07-26T09:27:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:22 smithi163 ceph-mon[36873]: pgmap v5387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9089.063328 secs 2024-07-26T09:27:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:22 smithi042 ceph-mon[32775]: pgmap v5387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9089.063328 secs 2024-07-26T09:27:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:22 smithi164 ceph-mon[36970]: pgmap v5387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9089.063328 secs 2024-07-26T09:27:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:24 smithi163 ceph-mon[36873]: pgmap v5388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:24 smithi042 ceph-mon[32775]: pgmap v5388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:24 smithi164 ceph-mon[36970]: pgmap v5388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:27:26.611+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:27:26.613 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:27:26.611+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:27:26.614 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:27:26.611+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:27:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:26 smithi042 ceph-mon[32775]: pgmap v5389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9076.322400 secs 2024-07-26T09:27:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:26 smithi164 ceph-mon[36970]: pgmap v5389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9076.322400 secs 2024-07-26T09:27:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:26 smithi163 ceph-mon[36873]: pgmap v5389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9076.322400 secs 2024-07-26T09:27:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9094.063449 secs 2024-07-26T09:27:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9094.063449 secs 2024-07-26T09:27:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9094.063449 secs 2024-07-26T09:27:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:28 smithi042 ceph-mon[32775]: pgmap v5390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:28 smithi164 ceph-mon[36970]: pgmap v5390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:28 smithi163 ceph-mon[36873]: pgmap v5390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:30 smithi042 ceph-mon[32775]: pgmap v5391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:30 smithi164 ceph-mon[36970]: pgmap v5391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:30 smithi163 ceph-mon[36873]: pgmap v5391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:31.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9081.322522 secs 2024-07-26T09:27:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9081.322522 secs 2024-07-26T09:27:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9081.322522 secs 2024-07-26T09:27:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:32 smithi042 ceph-mon[32775]: pgmap v5392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9099.063572 secs 2024-07-26T09:27:32.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:32 smithi164 ceph-mon[36970]: pgmap v5392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:32.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9099.063572 secs 2024-07-26T09:27:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:32 smithi163 ceph-mon[36873]: pgmap v5392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9099.063572 secs 2024-07-26T09:27:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:34 smithi042 ceph-mon[32775]: pgmap v5393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:34 smithi164 ceph-mon[36970]: pgmap v5393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:34 smithi163 ceph-mon[36873]: pgmap v5393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:36.561 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:36 smithi163 ceph-mon[36873]: pgmap v5394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:36.561 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9086.322642 secs 2024-07-26T09:27:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:36 smithi042 ceph-mon[32775]: pgmap v5394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9086.322642 secs 2024-07-26T09:27:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:36 smithi164 ceph-mon[36970]: pgmap v5394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9086.322642 secs 2024-07-26T09:27:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9104.063709 secs 2024-07-26T09:27:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9104.063709 secs 2024-07-26T09:27:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9104.063709 secs 2024-07-26T09:27:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:38 smithi042 ceph-mon[32775]: pgmap v5395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:38 smithi164 ceph-mon[36970]: pgmap v5395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:38 smithi163 ceph-mon[36873]: pgmap v5395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:40 smithi042 ceph-mon[32775]: pgmap v5396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:40 smithi164 ceph-mon[36970]: pgmap v5396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:40 smithi163 ceph-mon[36873]: pgmap v5396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:41.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:27:41.621+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:27:41.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:27:41.621+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:27:41.623 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:27:41.621+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:27:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9091.322746 secs 2024-07-26T09:27:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9091.322746 secs 2024-07-26T09:27:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9091.322746 secs 2024-07-26T09:27:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:42 smithi042 ceph-mon[32775]: pgmap v5397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9109.063833 secs 2024-07-26T09:27:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:42 smithi164 ceph-mon[36970]: pgmap v5397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9109.063833 secs 2024-07-26T09:27:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:42 smithi163 ceph-mon[36873]: pgmap v5397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9109.063833 secs 2024-07-26T09:27:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:44 smithi042 ceph-mon[32775]: pgmap v5398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:44 smithi164 ceph-mon[36970]: pgmap v5398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:44 smithi163 ceph-mon[36873]: pgmap v5398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:46 smithi042 ceph-mon[32775]: pgmap v5399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9096.322872 secs 2024-07-26T09:27:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:46 smithi164 ceph-mon[36970]: pgmap v5399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9096.322872 secs 2024-07-26T09:27:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:46 smithi163 ceph-mon[36873]: pgmap v5399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:27:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9096.322872 secs 2024-07-26T09:27:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9114.063955 secs 2024-07-26T09:27:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:27:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9114.063955 secs 2024-07-26T09:27:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:27:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9114.063955 secs 2024-07-26T09:27:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:27:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:48 smithi042 ceph-mon[32775]: pgmap v5400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:27:48.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:27:48.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:48 smithi042 ceph-mon[32775]: pgmap v5401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:27:48.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:48 smithi042 ceph-mon[32775]: pgmap v5402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:27:48.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:27:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:48 smithi164 ceph-mon[36970]: pgmap v5400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:48.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:27:48.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:27:48.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:48 smithi164 ceph-mon[36970]: pgmap v5401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:27:48.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:48 smithi164 ceph-mon[36970]: pgmap v5402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:27:48.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:27:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:48 smithi163 ceph-mon[36873]: pgmap v5400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:27:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:27:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:27:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:48 smithi163 ceph-mon[36873]: pgmap v5401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:27:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:48 smithi163 ceph-mon[36873]: pgmap v5402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:27:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:27:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:49 smithi042 ceph-mon[32775]: pgmap v5403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:27:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:49 smithi164 ceph-mon[36970]: pgmap v5403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:27:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:49 smithi163 ceph-mon[36873]: pgmap v5403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:27:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9101.322991 secs 2024-07-26T09:27:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9101.322991 secs 2024-07-26T09:27:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9101.322991 secs 2024-07-26T09:27:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9119.064082 secs 2024-07-26T09:27:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:52 smithi042 ceph-mon[32775]: pgmap v5404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:27:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9119.064082 secs 2024-07-26T09:27:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:52 smithi164 ceph-mon[36970]: pgmap v5404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:27:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9119.064082 secs 2024-07-26T09:27:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:52 smithi163 ceph-mon[36873]: pgmap v5404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:27:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:53 smithi042 ceph-mon[32775]: pgmap v5405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:27:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:53 smithi164 ceph-mon[36970]: pgmap v5405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:27:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:53 smithi163 ceph-mon[36873]: pgmap v5405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:27:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:55 smithi042 ceph-mon[32775]: pgmap v5406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:27:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:55 smithi164 ceph-mon[36970]: pgmap v5406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:27:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:55 smithi163 ceph-mon[36873]: pgmap v5406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:27:56.627 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:27:56.625+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:27:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:27:56.625+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:27:56.628 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:27:56.625+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:27:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9106.323140 secs 2024-07-26T09:27:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9124.064183 secs 2024-07-26T09:27:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9106.323140 secs 2024-07-26T09:27:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9124.064183 secs 2024-07-26T09:27:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9106.323140 secs 2024-07-26T09:27:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9124.064183 secs 2024-07-26T09:27:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:57 smithi042 ceph-mon[32775]: pgmap v5407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:27:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:57 smithi164 ceph-mon[36970]: pgmap v5407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:27:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:57 smithi163 ceph-mon[36873]: pgmap v5407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:28:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:27:59 smithi042 ceph-mon[32775]: pgmap v5408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:27:59 smithi164 ceph-mon[36970]: pgmap v5408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:27:59 smithi163 ceph-mon[36873]: pgmap v5408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9111.323268 secs 2024-07-26T09:28:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9111.323268 secs 2024-07-26T09:28:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9111.323268 secs 2024-07-26T09:28:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9129.064272 secs 2024-07-26T09:28:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:02 smithi042 ceph-mon[32775]: pgmap v5409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9129.064272 secs 2024-07-26T09:28:02.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:02 smithi164 ceph-mon[36970]: pgmap v5409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9129.064272 secs 2024-07-26T09:28:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:02 smithi163 ceph-mon[36873]: pgmap v5409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:03 smithi042 ceph-mon[32775]: pgmap v5410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:28:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:03 smithi164 ceph-mon[36970]: pgmap v5410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:28:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:03 smithi163 ceph-mon[36873]: pgmap v5410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:28:06.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:05 smithi163 ceph-mon[36873]: pgmap v5411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:05 smithi042 ceph-mon[32775]: pgmap v5411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:05 smithi164 ceph-mon[36970]: pgmap v5411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9116.323387 secs 2024-07-26T09:28:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9134.064400 secs 2024-07-26T09:28:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9116.323387 secs 2024-07-26T09:28:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9134.064400 secs 2024-07-26T09:28:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9116.323387 secs 2024-07-26T09:28:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9134.064400 secs 2024-07-26T09:28:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:07 smithi042 ceph-mon[32775]: pgmap v5412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:07 smithi164 ceph-mon[36970]: pgmap v5412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:07 smithi163 ceph-mon[36873]: pgmap v5412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:09 smithi042 ceph-mon[32775]: pgmap v5413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:09 smithi164 ceph-mon[36970]: pgmap v5413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:09 smithi163 ceph-mon[36873]: pgmap v5413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9121.323533 secs 2024-07-26T09:28:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9121.323533 secs 2024-07-26T09:28:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9121.323533 secs 2024-07-26T09:28:11.630 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:28:11.628+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:28:11.630 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:28:11.628+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:28:11.630 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:28:11.628+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:28:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9139.064542 secs 2024-07-26T09:28:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:12 smithi042 ceph-mon[32775]: pgmap v5414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9139.064542 secs 2024-07-26T09:28:12.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:12 smithi164 ceph-mon[36970]: pgmap v5414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9139.064542 secs 2024-07-26T09:28:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:12 smithi163 ceph-mon[36873]: pgmap v5414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:13 smithi042 ceph-mon[32775]: pgmap v5415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:28:14.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:13 smithi164 ceph-mon[36970]: pgmap v5415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:28:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:13 smithi163 ceph-mon[36873]: pgmap v5415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:28:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:15 smithi042 ceph-mon[32775]: pgmap v5416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:15 smithi164 ceph-mon[36970]: pgmap v5416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:15 smithi163 ceph-mon[36873]: pgmap v5416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9126.323677 secs 2024-07-26T09:28:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9144.064664 secs 2024-07-26T09:28:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9126.323677 secs 2024-07-26T09:28:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9144.064664 secs 2024-07-26T09:28:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9126.323677 secs 2024-07-26T09:28:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9144.064664 secs 2024-07-26T09:28:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:17 smithi042 ceph-mon[32775]: pgmap v5417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:17 smithi164 ceph-mon[36970]: pgmap v5417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:17 smithi163 ceph-mon[36873]: pgmap v5417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:19 smithi042 ceph-mon[32775]: pgmap v5418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:19 smithi164 ceph-mon[36970]: pgmap v5418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:19 smithi163 ceph-mon[36873]: pgmap v5418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9131.323755 secs 2024-07-26T09:28:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9131.323755 secs 2024-07-26T09:28:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9131.323755 secs 2024-07-26T09:28:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9149.064789 secs 2024-07-26T09:28:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:22 smithi042 ceph-mon[32775]: pgmap v5419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9149.064789 secs 2024-07-26T09:28:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:22 smithi164 ceph-mon[36970]: pgmap v5419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9149.064789 secs 2024-07-26T09:28:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:22 smithi163 ceph-mon[36873]: pgmap v5419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:23 smithi042 ceph-mon[32775]: pgmap v5420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:28:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:23 smithi164 ceph-mon[36970]: pgmap v5420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:28:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:23 smithi163 ceph-mon[36873]: pgmap v5420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:28:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:25 smithi042 ceph-mon[32775]: pgmap v5421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:25 smithi164 ceph-mon[36970]: pgmap v5421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:25 smithi163 ceph-mon[36873]: pgmap v5421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:28:26.632+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:28:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:28:26.632+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:28:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:28:26.632+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:28:27.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9136.323880 secs 2024-07-26T09:28:27.048 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9154.064912 secs 2024-07-26T09:28:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9136.323880 secs 2024-07-26T09:28:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9154.064912 secs 2024-07-26T09:28:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9136.323880 secs 2024-07-26T09:28:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9154.064912 secs 2024-07-26T09:28:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:27 smithi042 ceph-mon[32775]: pgmap v5422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:27 smithi164 ceph-mon[36970]: pgmap v5422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:27 smithi163 ceph-mon[36873]: pgmap v5422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:29 smithi042 ceph-mon[32775]: pgmap v5423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:29 smithi164 ceph-mon[36970]: pgmap v5423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:29 smithi163 ceph-mon[36873]: pgmap v5423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9141.324011 secs 2024-07-26T09:28:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9141.324011 secs 2024-07-26T09:28:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9141.324011 secs 2024-07-26T09:28:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9159.065039 secs 2024-07-26T09:28:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:32 smithi042 ceph-mon[32775]: pgmap v5424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9159.065039 secs 2024-07-26T09:28:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:32 smithi164 ceph-mon[36970]: pgmap v5424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9159.065039 secs 2024-07-26T09:28:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:32 smithi163 ceph-mon[36873]: pgmap v5424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:33 smithi042 ceph-mon[32775]: pgmap v5425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:28:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:33 smithi164 ceph-mon[36970]: pgmap v5425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:28:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:33 smithi163 ceph-mon[36873]: pgmap v5425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:28:36.061 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:35 smithi163 ceph-mon[36873]: pgmap v5426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:35 smithi042 ceph-mon[32775]: pgmap v5426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:35 smithi164 ceph-mon[36970]: pgmap v5426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9146.324145 secs 2024-07-26T09:28:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9164.065160 secs 2024-07-26T09:28:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9146.324145 secs 2024-07-26T09:28:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9164.065160 secs 2024-07-26T09:28:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9146.324145 secs 2024-07-26T09:28:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9164.065160 secs 2024-07-26T09:28:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:37 smithi042 ceph-mon[32775]: pgmap v5427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:37 smithi164 ceph-mon[36970]: pgmap v5427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:37 smithi163 ceph-mon[36873]: pgmap v5427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:39 smithi042 ceph-mon[32775]: pgmap v5428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:39 smithi164 ceph-mon[36970]: pgmap v5428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:39 smithi163 ceph-mon[36873]: pgmap v5428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9151.324267 secs 2024-07-26T09:28:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9151.324267 secs 2024-07-26T09:28:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9151.324267 secs 2024-07-26T09:28:41.644 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:28:41.642+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:28:41.644 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:28:41.642+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:28:41.644 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:28:41.642+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:28:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9169.065283 secs 2024-07-26T09:28:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:42 smithi042 ceph-mon[32775]: pgmap v5429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9169.065283 secs 2024-07-26T09:28:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:42 smithi164 ceph-mon[36970]: pgmap v5429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9169.065283 secs 2024-07-26T09:28:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:42 smithi163 ceph-mon[36873]: pgmap v5429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:43 smithi042 ceph-mon[32775]: pgmap v5430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:28:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:43 smithi164 ceph-mon[36970]: pgmap v5430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:28:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:43 smithi163 ceph-mon[36873]: pgmap v5430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:28:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:45 smithi042 ceph-mon[32775]: pgmap v5431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:45 smithi164 ceph-mon[36970]: pgmap v5431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:45 smithi163 ceph-mon[36873]: pgmap v5431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:28:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9156.324291 secs 2024-07-26T09:28:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9174.065405 secs 2024-07-26T09:28:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9156.324291 secs 2024-07-26T09:28:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9174.065405 secs 2024-07-26T09:28:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9156.324291 secs 2024-07-26T09:28:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9174.065405 secs 2024-07-26T09:28:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:47 smithi042 ceph-mon[32775]: pgmap v5432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:28:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:47 smithi164 ceph-mon[36970]: pgmap v5432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:28:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:47 smithi163 ceph-mon[36873]: pgmap v5432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:28:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:28:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:28:49.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:28:49.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:28:49.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:28:49.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:28:49.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:28:49.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:28:49.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:28:49.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: pgmap v5433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:28:49.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: pgmap v5434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:28:49.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:28:49.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:28:49.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:28:49.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:28:49.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:28:49.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:28:49.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:28:49.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:28:49.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:28:49.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: pgmap v5433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:28:49.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: pgmap v5434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:28:49.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:28:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:28:49.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:28:49.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:28:49.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:28:49.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:28:49.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:28:49.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:28:49.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:28:49.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:28:49.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: pgmap v5433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:28:49.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: pgmap v5434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:28:49.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:28:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:51 smithi042 ceph-mon[32775]: pgmap v5435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:28:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9161.324417 secs 2024-07-26T09:28:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:51 smithi164 ceph-mon[36970]: pgmap v5435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:28:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9161.324417 secs 2024-07-26T09:28:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:51 smithi163 ceph-mon[36873]: pgmap v5435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:28:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9161.324417 secs 2024-07-26T09:28:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9179.065501 secs 2024-07-26T09:28:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9179.065501 secs 2024-07-26T09:28:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9179.065501 secs 2024-07-26T09:28:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:53 smithi042 ceph-mon[32775]: pgmap v5436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:28:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:53 smithi164 ceph-mon[36970]: pgmap v5436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:28:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:53 smithi163 ceph-mon[36873]: pgmap v5436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:28:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:55 smithi042 ceph-mon[32775]: pgmap v5437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:28:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:55 smithi164 ceph-mon[36970]: pgmap v5437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:28:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:55 smithi163 ceph-mon[36873]: pgmap v5437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:28:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9166.324551 secs 2024-07-26T09:28:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:28:56.646+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:28:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:28:56.646+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:28:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:28:56.646+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:28:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9166.324551 secs 2024-07-26T09:28:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9166.324551 secs 2024-07-26T09:28:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9184.065566 secs 2024-07-26T09:28:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:57 smithi042 ceph-mon[32775]: pgmap v5438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:28:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9184.065566 secs 2024-07-26T09:28:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:57 smithi164 ceph-mon[36970]: pgmap v5438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:28:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9184.065566 secs 2024-07-26T09:28:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:57 smithi163 ceph-mon[36873]: pgmap v5438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:28:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:28:59.285+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:28:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:28:59.285+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T09:28:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:28:59.285+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:28:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:28:59.285+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T09:28:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:28:59 smithi042 ceph-mon[32775]: pgmap v5439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:28:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:28:59 smithi164 ceph-mon[36970]: pgmap v5439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:28:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:28:59 smithi163 ceph-mon[36873]: pgmap v5439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:29:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:01 smithi042 ceph-mon[32775]: pgmap v5440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9171.324618 secs 2024-07-26T09:29:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:01 smithi164 ceph-mon[36970]: pgmap v5440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:01.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9171.324618 secs 2024-07-26T09:29:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:01 smithi163 ceph-mon[36873]: pgmap v5440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9171.324618 secs 2024-07-26T09:29:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9189.065692 secs 2024-07-26T09:29:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9189.065692 secs 2024-07-26T09:29:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9189.065692 secs 2024-07-26T09:29:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:03 smithi042 ceph-mon[32775]: pgmap v5441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:03 smithi164 ceph-mon[36970]: pgmap v5441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:03 smithi163 ceph-mon[36873]: pgmap v5441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:05 smithi042 ceph-mon[32775]: pgmap v5442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:05 smithi164 ceph-mon[36970]: pgmap v5442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:05 smithi163 ceph-mon[36873]: pgmap v5442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9176.324758 secs 2024-07-26T09:29:06.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9176.324758 secs 2024-07-26T09:29:06.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9176.324758 secs 2024-07-26T09:29:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9194.065812 secs 2024-07-26T09:29:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:07 smithi042 ceph-mon[32775]: pgmap v5443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9194.065812 secs 2024-07-26T09:29:07.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:07 smithi164 ceph-mon[36970]: pgmap v5443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9194.065812 secs 2024-07-26T09:29:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:07 smithi163 ceph-mon[36873]: pgmap v5443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:09 smithi042 ceph-mon[32775]: pgmap v5444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:09 smithi164 ceph-mon[36970]: pgmap v5444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:09 smithi163 ceph-mon[36873]: pgmap v5444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:11 smithi042 ceph-mon[32775]: pgmap v5445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9181.324876 secs 2024-07-26T09:29:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:29:11.656+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:29:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:29:11.656+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:29:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:29:11.656+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:29:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:11 smithi164 ceph-mon[36970]: pgmap v5445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9181.324876 secs 2024-07-26T09:29:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:11 smithi163 ceph-mon[36873]: pgmap v5445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9181.324876 secs 2024-07-26T09:29:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9199.065952 secs 2024-07-26T09:29:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9199.065952 secs 2024-07-26T09:29:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9199.065952 secs 2024-07-26T09:29:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:13 smithi042 ceph-mon[32775]: pgmap v5446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:13.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:13 smithi164 ceph-mon[36970]: pgmap v5446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:13 smithi163 ceph-mon[36873]: pgmap v5446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:15 smithi042 ceph-mon[32775]: pgmap v5447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:15 smithi164 ceph-mon[36970]: pgmap v5447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:15 smithi163 ceph-mon[36873]: pgmap v5447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:16.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9186.324998 secs 2024-07-26T09:29:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9186.324998 secs 2024-07-26T09:29:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9186.324998 secs 2024-07-26T09:29:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9204.066079 secs 2024-07-26T09:29:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:17 smithi042 ceph-mon[32775]: pgmap v5448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9204.066079 secs 2024-07-26T09:29:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:17 smithi164 ceph-mon[36970]: pgmap v5448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9204.066079 secs 2024-07-26T09:29:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:17 smithi163 ceph-mon[36873]: pgmap v5448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:19 smithi042 ceph-mon[32775]: pgmap v5449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:19 smithi164 ceph-mon[36970]: pgmap v5449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:19 smithi163 ceph-mon[36873]: pgmap v5449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:21 smithi042 ceph-mon[32775]: pgmap v5450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9191.325124 secs 2024-07-26T09:29:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:21 smithi164 ceph-mon[36970]: pgmap v5450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9191.325124 secs 2024-07-26T09:29:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:21 smithi163 ceph-mon[36873]: pgmap v5450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9191.325124 secs 2024-07-26T09:29:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9209.066223 secs 2024-07-26T09:29:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9209.066223 secs 2024-07-26T09:29:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9209.066223 secs 2024-07-26T09:29:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:23 smithi042 ceph-mon[32775]: pgmap v5451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:23 smithi164 ceph-mon[36970]: pgmap v5451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:23 smithi163 ceph-mon[36873]: pgmap v5451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:25 smithi042 ceph-mon[32775]: pgmap v5452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:25 smithi164 ceph-mon[36970]: pgmap v5452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:25 smithi163 ceph-mon[36873]: pgmap v5452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9196.325259 secs 2024-07-26T09:29:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:29:26.661+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:29:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:29:26.661+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:29:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:29:26.661+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:29:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9196.325259 secs 2024-07-26T09:29:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9196.325259 secs 2024-07-26T09:29:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9214.066364 secs 2024-07-26T09:29:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:27 smithi042 ceph-mon[32775]: pgmap v5453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9214.066364 secs 2024-07-26T09:29:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:27 smithi164 ceph-mon[36970]: pgmap v5453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9214.066364 secs 2024-07-26T09:29:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:27 smithi163 ceph-mon[36873]: pgmap v5453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:29 smithi042 ceph-mon[32775]: pgmap v5454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:29.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:29 smithi164 ceph-mon[36970]: pgmap v5454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:29 smithi163 ceph-mon[36873]: pgmap v5454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:31 smithi042 ceph-mon[32775]: pgmap v5455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9201.325358 secs 2024-07-26T09:29:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:31 smithi164 ceph-mon[36970]: pgmap v5455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:31.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9201.325358 secs 2024-07-26T09:29:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:31 smithi163 ceph-mon[36873]: pgmap v5455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9201.325358 secs 2024-07-26T09:29:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9219.066483 secs 2024-07-26T09:29:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9219.066483 secs 2024-07-26T09:29:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9219.066483 secs 2024-07-26T09:29:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:33 smithi164 ceph-mon[36970]: pgmap v5456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:33 smithi163 ceph-mon[36873]: pgmap v5456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:33 smithi042 ceph-mon[32775]: pgmap v5456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:35 smithi164 ceph-mon[36970]: pgmap v5457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:35 smithi163 ceph-mon[36873]: pgmap v5457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:35 smithi042 ceph-mon[32775]: pgmap v5457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9206.325485 secs 2024-07-26T09:29:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9206.325485 secs 2024-07-26T09:29:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9206.325485 secs 2024-07-26T09:29:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9224.066603 secs 2024-07-26T09:29:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:37 smithi164 ceph-mon[36970]: pgmap v5458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9224.066603 secs 2024-07-26T09:29:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:37 smithi163 ceph-mon[36873]: pgmap v5458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9224.066603 secs 2024-07-26T09:29:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:37 smithi042 ceph-mon[32775]: pgmap v5458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:39 smithi164 ceph-mon[36970]: pgmap v5459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:39 smithi163 ceph-mon[36873]: pgmap v5459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:39 smithi042 ceph-mon[32775]: pgmap v5459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:41.672 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:29:41.670+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:29:41.672 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:29:41.670+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:29:41.672 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:29:41.670+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:29:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:41 smithi164 ceph-mon[36970]: pgmap v5460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9211.325607 secs 2024-07-26T09:29:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:41 smithi163 ceph-mon[36873]: pgmap v5460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9211.325607 secs 2024-07-26T09:29:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:41 smithi042 ceph-mon[32775]: pgmap v5460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9211.325607 secs 2024-07-26T09:29:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9229.066726 secs 2024-07-26T09:29:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9229.066726 secs 2024-07-26T09:29:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9229.066726 secs 2024-07-26T09:29:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:43 smithi164 ceph-mon[36970]: pgmap v5461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:43 smithi163 ceph-mon[36873]: pgmap v5461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:43 smithi042 ceph-mon[32775]: pgmap v5461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:45 smithi164 ceph-mon[36970]: pgmap v5462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:45 smithi163 ceph-mon[36873]: pgmap v5462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:45 smithi042 ceph-mon[32775]: pgmap v5462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:29:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9216.325734 secs 2024-07-26T09:29:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9216.325734 secs 2024-07-26T09:29:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9216.325734 secs 2024-07-26T09:29:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9234.066846 secs 2024-07-26T09:29:47.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:47 smithi164 ceph-mon[36970]: pgmap v5463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9234.066846 secs 2024-07-26T09:29:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:47 smithi163 ceph-mon[36873]: pgmap v5463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9234.066846 secs 2024-07-26T09:29:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:47 smithi042 ceph-mon[32775]: pgmap v5463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:29:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:29:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:29:49.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:49 smithi164 ceph-mon[36970]: pgmap v5464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:49.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:29:49.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:29:49.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:29:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:49 smithi163 ceph-mon[36873]: pgmap v5464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:29:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:29:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:29:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:49 smithi042 ceph-mon[32775]: pgmap v5464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:29:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:29:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:29:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:29:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:50 smithi164 ceph-mon[36970]: pgmap v5465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:29:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:50 smithi164 ceph-mon[36970]: pgmap v5466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:29:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:50 smithi163 ceph-mon[36873]: pgmap v5465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:29:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:50 smithi163 ceph-mon[36873]: pgmap v5466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:29:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:50 smithi042 ceph-mon[32775]: pgmap v5465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:29:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:50 smithi042 ceph-mon[32775]: pgmap v5466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:29:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9221.325856 secs 2024-07-26T09:29:51.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9239.066986 secs 2024-07-26T09:29:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9221.325856 secs 2024-07-26T09:29:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9239.066986 secs 2024-07-26T09:29:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9221.325856 secs 2024-07-26T09:29:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9239.066986 secs 2024-07-26T09:29:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:52 smithi164 ceph-mon[36970]: pgmap v5467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:29:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:52 smithi163 ceph-mon[36873]: pgmap v5467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:29:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:52 smithi042 ceph-mon[32775]: pgmap v5467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:29:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:54 smithi164 ceph-mon[36970]: pgmap v5468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:29:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:54 smithi163 ceph-mon[36873]: pgmap v5468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:29:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:54 smithi042 ceph-mon[32775]: pgmap v5468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:29:56.677 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:29:56.675+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:29:56.677 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:29:56.675+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:29:56.677 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:29:56.675+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:29:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:56 smithi164 ceph-mon[36970]: pgmap v5469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:29:56.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9226.325982 secs 2024-07-26T09:29:56.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9244.067104 secs 2024-07-26T09:29:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:56 smithi163 ceph-mon[36873]: pgmap v5469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:29:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9226.325982 secs 2024-07-26T09:29:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9244.067104 secs 2024-07-26T09:29:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:56 smithi042 ceph-mon[32775]: pgmap v5469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:29:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9226.325982 secs 2024-07-26T09:29:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9244.067104 secs 2024-07-26T09:29:58.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:29:58 smithi164 ceph-mon[36970]: pgmap v5470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:29:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:29:58 smithi163 ceph-mon[36873]: pgmap v5470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:29:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:29:58 smithi042 ceph-mon[32775]: pgmap v5470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:30:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:00 smithi163 ceph-mon[36873]: pgmap v5471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:30:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:30:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:00 smithi042 ceph-mon[32775]: pgmap v5471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:30:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:30:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:00 smithi164 ceph-mon[36970]: pgmap v5471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:30:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:30:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9231.326103 secs 2024-07-26T09:30:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9249.067225 secs 2024-07-26T09:30:01.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9231.326103 secs 2024-07-26T09:30:01.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9249.067225 secs 2024-07-26T09:30:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9231.326103 secs 2024-07-26T09:30:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9249.067225 secs 2024-07-26T09:30:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:02 smithi163 ceph-mon[36873]: pgmap v5472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:02 smithi042 ceph-mon[32775]: pgmap v5472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:02 smithi164 ceph-mon[36970]: pgmap v5472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:04 smithi163 ceph-mon[36873]: pgmap v5473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:04 smithi042 ceph-mon[32775]: pgmap v5473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:04 smithi164 ceph-mon[36970]: pgmap v5473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:06 smithi163 ceph-mon[36873]: pgmap v5474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:06.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9236.326250 secs 2024-07-26T09:30:06.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9254.067348 secs 2024-07-26T09:30:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:06 smithi042 ceph-mon[32775]: pgmap v5474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9236.326250 secs 2024-07-26T09:30:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9254.067348 secs 2024-07-26T09:30:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:06 smithi164 ceph-mon[36970]: pgmap v5474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9236.326250 secs 2024-07-26T09:30:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9254.067348 secs 2024-07-26T09:30:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:08 smithi163 ceph-mon[36873]: pgmap v5475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:08 smithi042 ceph-mon[32775]: pgmap v5475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:08 smithi164 ceph-mon[36970]: pgmap v5475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:10 smithi163 ceph-mon[36873]: pgmap v5476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:30:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:10 smithi042 ceph-mon[32775]: pgmap v5476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:30:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:10 smithi164 ceph-mon[36970]: pgmap v5476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:30:11.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:30:11.685+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:30:11.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:30:11.685+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:30:11.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:30:11.685+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:30:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9241.326376 secs 2024-07-26T09:30:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9259.067470 secs 2024-07-26T09:30:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9241.326376 secs 2024-07-26T09:30:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9259.067470 secs 2024-07-26T09:30:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9241.326376 secs 2024-07-26T09:30:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9259.067470 secs 2024-07-26T09:30:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:12 smithi163 ceph-mon[36873]: pgmap v5477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:12 smithi042 ceph-mon[32775]: pgmap v5477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:12 smithi164 ceph-mon[36970]: pgmap v5477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:14 smithi163 ceph-mon[36873]: pgmap v5478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:14 smithi042 ceph-mon[32775]: pgmap v5478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:14 smithi164 ceph-mon[36970]: pgmap v5478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:16.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:16 smithi164 ceph-mon[36970]: pgmap v5479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:16.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9246.326497 secs 2024-07-26T09:30:16.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9264.067592 secs 2024-07-26T09:30:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:16 smithi042 ceph-mon[32775]: pgmap v5479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9246.326497 secs 2024-07-26T09:30:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9264.067592 secs 2024-07-26T09:30:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:16 smithi163 ceph-mon[36873]: pgmap v5479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9246.326497 secs 2024-07-26T09:30:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9264.067592 secs 2024-07-26T09:30:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:18 smithi042 ceph-mon[32775]: pgmap v5480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:18 smithi164 ceph-mon[36970]: pgmap v5480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:18 smithi163 ceph-mon[36873]: pgmap v5480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:20 smithi042 ceph-mon[32775]: pgmap v5481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:30:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:20 smithi164 ceph-mon[36970]: pgmap v5481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:30:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:20 smithi163 ceph-mon[36873]: pgmap v5481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:30:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9251.326560 secs 2024-07-26T09:30:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9269.067729 secs 2024-07-26T09:30:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9251.326560 secs 2024-07-26T09:30:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9269.067729 secs 2024-07-26T09:30:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9251.326560 secs 2024-07-26T09:30:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9269.067729 secs 2024-07-26T09:30:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:22 smithi042 ceph-mon[32775]: pgmap v5482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:22 smithi164 ceph-mon[36970]: pgmap v5482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:22 smithi163 ceph-mon[36873]: pgmap v5482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:24 smithi042 ceph-mon[32775]: pgmap v5483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:24 smithi164 ceph-mon[36970]: pgmap v5483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:24 smithi163 ceph-mon[36873]: pgmap v5483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:30:26.690+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:30:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:30:26.690+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:30:26.692 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:30:26.690+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:30:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:26 smithi042 ceph-mon[32775]: pgmap v5484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9256.326718 secs 2024-07-26T09:30:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9274.067873 secs 2024-07-26T09:30:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:26 smithi164 ceph-mon[36970]: pgmap v5484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9256.326718 secs 2024-07-26T09:30:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9274.067873 secs 2024-07-26T09:30:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:26 smithi163 ceph-mon[36873]: pgmap v5484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9256.326718 secs 2024-07-26T09:30:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9274.067873 secs 2024-07-26T09:30:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:28 smithi042 ceph-mon[32775]: pgmap v5485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:28 smithi164 ceph-mon[36970]: pgmap v5485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:28 smithi163 ceph-mon[36873]: pgmap v5485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:30 smithi042 ceph-mon[32775]: pgmap v5486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:30:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:30 smithi164 ceph-mon[36970]: pgmap v5486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:30:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:30 smithi163 ceph-mon[36873]: pgmap v5486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:30:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9261.326814 secs 2024-07-26T09:30:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9279.067992 secs 2024-07-26T09:30:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9261.326814 secs 2024-07-26T09:30:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9279.067992 secs 2024-07-26T09:30:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9261.326814 secs 2024-07-26T09:30:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9279.067992 secs 2024-07-26T09:30:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:32 smithi042 ceph-mon[32775]: pgmap v5487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:32 smithi164 ceph-mon[36970]: pgmap v5487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:32 smithi163 ceph-mon[36873]: pgmap v5487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:34 smithi042 ceph-mon[32775]: pgmap v5488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:34 smithi164 ceph-mon[36970]: pgmap v5488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:34 smithi163 ceph-mon[36873]: pgmap v5488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:36.810 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:36 smithi163 ceph-mon[36873]: pgmap v5489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:36.811 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9266.326938 secs 2024-07-26T09:30:36.811 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9284.068083 secs 2024-07-26T09:30:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:36 smithi042 ceph-mon[32775]: pgmap v5489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9266.326938 secs 2024-07-26T09:30:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9284.068083 secs 2024-07-26T09:30:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:36 smithi164 ceph-mon[36970]: pgmap v5489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9266.326938 secs 2024-07-26T09:30:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9284.068083 secs 2024-07-26T09:30:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:38 smithi042 ceph-mon[32775]: pgmap v5490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:38 smithi164 ceph-mon[36970]: pgmap v5490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:38 smithi163 ceph-mon[36873]: pgmap v5490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:40 smithi042 ceph-mon[32775]: pgmap v5491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:30:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:40 smithi164 ceph-mon[36970]: pgmap v5491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:30:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:40 smithi163 ceph-mon[36873]: pgmap v5491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:30:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:30:41.694+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:30:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:30:41.694+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:30:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:30:41.694+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:30:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9271.327061 secs 2024-07-26T09:30:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9289.068204 secs 2024-07-26T09:30:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9271.327061 secs 2024-07-26T09:30:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9289.068204 secs 2024-07-26T09:30:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9271.327061 secs 2024-07-26T09:30:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9289.068204 secs 2024-07-26T09:30:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:42 smithi042 ceph-mon[32775]: pgmap v5492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:42 smithi164 ceph-mon[36970]: pgmap v5492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:42 smithi163 ceph-mon[36873]: pgmap v5492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:44 smithi042 ceph-mon[32775]: pgmap v5493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:44 smithi164 ceph-mon[36970]: pgmap v5493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:44 smithi163 ceph-mon[36873]: pgmap v5493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:46 smithi042 ceph-mon[32775]: pgmap v5494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9276.327163 secs 2024-07-26T09:30:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9294.068323 secs 2024-07-26T09:30:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:46 smithi164 ceph-mon[36970]: pgmap v5494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9276.327163 secs 2024-07-26T09:30:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9294.068323 secs 2024-07-26T09:30:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:46 smithi163 ceph-mon[36873]: pgmap v5494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:30:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9276.327163 secs 2024-07-26T09:30:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9294.068323 secs 2024-07-26T09:30:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:48 smithi042 ceph-mon[32775]: pgmap v5495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:48 smithi164 ceph-mon[36970]: pgmap v5495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:48 smithi163 ceph-mon[36873]: pgmap v5495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:30:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:30:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:30:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:30:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:30:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:30:49.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:30:49.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:30:49.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:30:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:30:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:30:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:30:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:30:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:50 smithi042 ceph-mon[32775]: pgmap v5496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:30:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:50 smithi042 ceph-mon[32775]: pgmap v5497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T09:30:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:50 smithi042 ceph-mon[32775]: pgmap v5498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:30:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:50 smithi164 ceph-mon[36970]: pgmap v5496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:30:50.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:50 smithi164 ceph-mon[36970]: pgmap v5497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T09:30:50.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:50 smithi164 ceph-mon[36970]: pgmap v5498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:30:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:50 smithi163 ceph-mon[36873]: pgmap v5496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:30:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:50 smithi163 ceph-mon[36873]: pgmap v5497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T09:30:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:50 smithi163 ceph-mon[36873]: pgmap v5498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:30:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9281.327292 secs 2024-07-26T09:30:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9299.068450 secs 2024-07-26T09:30:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9281.327292 secs 2024-07-26T09:30:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9299.068450 secs 2024-07-26T09:30:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9281.327292 secs 2024-07-26T09:30:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9299.068450 secs 2024-07-26T09:30:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:52 smithi042 ceph-mon[32775]: pgmap v5499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:30:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:52 smithi164 ceph-mon[36970]: pgmap v5499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:30:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:52 smithi163 ceph-mon[36873]: pgmap v5499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:30:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:54 smithi042 ceph-mon[32775]: pgmap v5500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:30:54.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:54 smithi164 ceph-mon[36970]: pgmap v5500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:30:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:54 smithi163 ceph-mon[36873]: pgmap v5500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:30:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:30:56.698+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:30:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:30:56.698+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:30:56.700 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:30:56.698+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:30:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:56 smithi042 ceph-mon[32775]: pgmap v5501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:30:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9286.327434 secs 2024-07-26T09:30:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9304.068608 secs 2024-07-26T09:30:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:56 smithi164 ceph-mon[36970]: pgmap v5501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:30:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9286.327434 secs 2024-07-26T09:30:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9304.068608 secs 2024-07-26T09:30:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:56 smithi163 ceph-mon[36873]: pgmap v5501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:30:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9286.327434 secs 2024-07-26T09:30:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9304.068608 secs 2024-07-26T09:30:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:30:58 smithi042 ceph-mon[32775]: pgmap v5502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:30:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:30:58 smithi164 ceph-mon[36970]: pgmap v5502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:30:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:30:58 smithi163 ceph-mon[36873]: pgmap v5502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:31:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:00 smithi042 ceph-mon[32775]: pgmap v5503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:31:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:00 smithi164 ceph-mon[36970]: pgmap v5503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:31:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:00 smithi163 ceph-mon[36873]: pgmap v5503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:31:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9291.327540 secs 2024-07-26T09:31:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9309.068727 secs 2024-07-26T09:31:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9291.327540 secs 2024-07-26T09:31:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9309.068727 secs 2024-07-26T09:31:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9291.327540 secs 2024-07-26T09:31:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9309.068727 secs 2024-07-26T09:31:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:02 smithi042 ceph-mon[32775]: pgmap v5504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:02 smithi164 ceph-mon[36970]: pgmap v5504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:02 smithi163 ceph-mon[36873]: pgmap v5504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:04 smithi042 ceph-mon[32775]: pgmap v5505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:31:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:04 smithi164 ceph-mon[36970]: pgmap v5505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:31:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:04 smithi163 ceph-mon[36873]: pgmap v5505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:31:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:06 smithi042 ceph-mon[32775]: pgmap v5506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9296.327626 secs 2024-07-26T09:31:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9314.068854 secs 2024-07-26T09:31:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:06 smithi164 ceph-mon[36970]: pgmap v5506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9296.327626 secs 2024-07-26T09:31:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9314.068854 secs 2024-07-26T09:31:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:06 smithi163 ceph-mon[36873]: pgmap v5506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9296.327626 secs 2024-07-26T09:31:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9314.068854 secs 2024-07-26T09:31:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:08 smithi042 ceph-mon[32775]: pgmap v5507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:08 smithi164 ceph-mon[36970]: pgmap v5507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:08 smithi163 ceph-mon[36873]: pgmap v5507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:10 smithi042 ceph-mon[32775]: pgmap v5508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:10 smithi164 ceph-mon[36970]: pgmap v5508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:10 smithi163 ceph-mon[36873]: pgmap v5508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:31:11.708+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:31:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:31:11.708+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:31:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:31:11.708+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:31:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9301.327766 secs 2024-07-26T09:31:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9319.068975 secs 2024-07-26T09:31:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9301.327766 secs 2024-07-26T09:31:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9319.068975 secs 2024-07-26T09:31:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9301.327766 secs 2024-07-26T09:31:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9319.068975 secs 2024-07-26T09:31:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:12 smithi042 ceph-mon[32775]: pgmap v5509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:12 smithi164 ceph-mon[36970]: pgmap v5509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:12 smithi163 ceph-mon[36873]: pgmap v5509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:14 smithi042 ceph-mon[32775]: pgmap v5510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:31:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:14 smithi164 ceph-mon[36970]: pgmap v5510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:31:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:14 smithi163 ceph-mon[36873]: pgmap v5510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:31:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:16 smithi164 ceph-mon[36970]: pgmap v5511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9306.327909 secs 2024-07-26T09:31:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9324.069023 secs 2024-07-26T09:31:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:16 smithi163 ceph-mon[36873]: pgmap v5511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9306.327909 secs 2024-07-26T09:31:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9324.069023 secs 2024-07-26T09:31:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:16 smithi042 ceph-mon[32775]: pgmap v5511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9306.327909 secs 2024-07-26T09:31:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9324.069023 secs 2024-07-26T09:31:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:18 smithi164 ceph-mon[36970]: pgmap v5512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:18 smithi163 ceph-mon[36873]: pgmap v5512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:18 smithi042 ceph-mon[32775]: pgmap v5512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:20 smithi164 ceph-mon[36970]: pgmap v5513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:20 smithi163 ceph-mon[36873]: pgmap v5513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:20 smithi042 ceph-mon[32775]: pgmap v5513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9311.328036 secs 2024-07-26T09:31:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9329.069147 secs 2024-07-26T09:31:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9311.328036 secs 2024-07-26T09:31:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9329.069147 secs 2024-07-26T09:31:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9311.328036 secs 2024-07-26T09:31:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9329.069147 secs 2024-07-26T09:31:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:22 smithi164 ceph-mon[36970]: pgmap v5514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:22 smithi163 ceph-mon[36873]: pgmap v5514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:22 smithi042 ceph-mon[32775]: pgmap v5514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:24 smithi164 ceph-mon[36970]: pgmap v5515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:31:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:24 smithi163 ceph-mon[36873]: pgmap v5515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:31:25.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:24 smithi042 ceph-mon[32775]: pgmap v5515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:31:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:31:26.712+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:31:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:31:26.712+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:31:26.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:31:26.712+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:31:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:26 smithi164 ceph-mon[36970]: pgmap v5516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9316.328160 secs 2024-07-26T09:31:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9334.069267 secs 2024-07-26T09:31:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:26 smithi163 ceph-mon[36873]: pgmap v5516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9316.328160 secs 2024-07-26T09:31:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9334.069267 secs 2024-07-26T09:31:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:26 smithi042 ceph-mon[32775]: pgmap v5516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9316.328160 secs 2024-07-26T09:31:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9334.069267 secs 2024-07-26T09:31:28.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:28 smithi164 ceph-mon[36970]: pgmap v5517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:28 smithi163 ceph-mon[36873]: pgmap v5517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:28 smithi042 ceph-mon[32775]: pgmap v5517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:30 smithi164 ceph-mon[36970]: pgmap v5518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:30 smithi163 ceph-mon[36873]: pgmap v5518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:30 smithi042 ceph-mon[32775]: pgmap v5518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9321.328315 secs 2024-07-26T09:31:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9339.069408 secs 2024-07-26T09:31:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9321.328315 secs 2024-07-26T09:31:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9339.069408 secs 2024-07-26T09:31:32.048 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9321.328315 secs 2024-07-26T09:31:32.048 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9339.069408 secs 2024-07-26T09:31:32.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:32 smithi164 ceph-mon[36970]: pgmap v5519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:32 smithi163 ceph-mon[36873]: pgmap v5519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:32 smithi042 ceph-mon[32775]: pgmap v5519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:34 smithi164 ceph-mon[36970]: pgmap v5520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:31:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:34 smithi163 ceph-mon[36873]: pgmap v5520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:31:35.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:34 smithi042 ceph-mon[32775]: pgmap v5520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:31:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:36 smithi164 ceph-mon[36970]: pgmap v5521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:31:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9326.328400 secs 2024-07-26T09:31:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9344.069506 secs 2024-07-26T09:31:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:36 smithi163 ceph-mon[36873]: pgmap v5521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:31:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9326.328400 secs 2024-07-26T09:31:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9344.069506 secs 2024-07-26T09:31:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:36 smithi042 ceph-mon[32775]: pgmap v5521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:31:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9326.328400 secs 2024-07-26T09:31:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9344.069506 secs 2024-07-26T09:31:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:38 smithi164 ceph-mon[36970]: pgmap v5522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:31:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:38 smithi163 ceph-mon[36873]: pgmap v5522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:31:39.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:38 smithi042 ceph-mon[32775]: pgmap v5522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:31:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:40 smithi164 ceph-mon[36970]: pgmap v5523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:31:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:40 smithi163 ceph-mon[36873]: pgmap v5523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:31:41.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:40 smithi042 ceph-mon[32775]: pgmap v5523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:31:41.725 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:31:41.723+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:31:41.725 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:31:41.723+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:31:41.725 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:31:41.723+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:31:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9331.328495 secs 2024-07-26T09:31:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9349.069625 secs 2024-07-26T09:31:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9331.328495 secs 2024-07-26T09:31:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9349.069625 secs 2024-07-26T09:31:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9331.328495 secs 2024-07-26T09:31:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9349.069625 secs 2024-07-26T09:31:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:42 smithi164 ceph-mon[36970]: pgmap v5524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:31:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:42 smithi163 ceph-mon[36873]: pgmap v5524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:31:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:42 smithi042 ceph-mon[32775]: pgmap v5524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:31:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:44 smithi164 ceph-mon[36970]: pgmap v5525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:31:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:44 smithi163 ceph-mon[36873]: pgmap v5525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:31:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:44 smithi042 ceph-mon[32775]: pgmap v5525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:31:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:46 smithi164 ceph-mon[36970]: pgmap v5526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:31:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9336.328629 secs 2024-07-26T09:31:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9354.069731 secs 2024-07-26T09:31:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:46 smithi163 ceph-mon[36873]: pgmap v5526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:31:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9336.328629 secs 2024-07-26T09:31:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9354.069731 secs 2024-07-26T09:31:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:46 smithi042 ceph-mon[32775]: pgmap v5526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:31:47.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9336.328629 secs 2024-07-26T09:31:47.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9354.069731 secs 2024-07-26T09:31:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:48 smithi164 ceph-mon[36970]: pgmap v5527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:48 smithi163 ceph-mon[36873]: pgmap v5527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:49.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:48 smithi042 ceph-mon[32775]: pgmap v5527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:49.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:31:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:31:50.127 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:31:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:50 smithi164 ceph-mon[36970]: pgmap v5528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:50 smithi163 ceph-mon[36873]: pgmap v5528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:50 smithi042 ceph-mon[32775]: pgmap v5528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:31:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9341.328723 secs 2024-07-26T09:31:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9359.069825 secs 2024-07-26T09:31:52.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:52 smithi042 ceph-mon[32775]: pgmap v5529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:52.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:31:52.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:31:52.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9341.328723 secs 2024-07-26T09:31:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9359.069825 secs 2024-07-26T09:31:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:52 smithi164 ceph-mon[36970]: pgmap v5529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:31:52.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:31:52.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9341.328723 secs 2024-07-26T09:31:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9359.069825 secs 2024-07-26T09:31:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:52 smithi163 ceph-mon[36873]: pgmap v5529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:31:52.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:52.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:31:52.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:31:52.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:31:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:53 smithi042 ceph-mon[32775]: pgmap v5530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:31:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:53 smithi042 ceph-mon[32775]: pgmap v5531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:31:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:53 smithi164 ceph-mon[36970]: pgmap v5530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:31:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:53 smithi164 ceph-mon[36970]: pgmap v5531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:31:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:53 smithi163 ceph-mon[36873]: pgmap v5530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:31:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:53 smithi163 ceph-mon[36873]: pgmap v5531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:31:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:55 smithi042 ceph-mon[32775]: pgmap v5532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:31:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:55 smithi164 ceph-mon[36970]: pgmap v5532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:31:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:55 smithi163 ceph-mon[36873]: pgmap v5532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:31:56.730 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:31:56.728+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:31:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:31:56.728+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:31:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:31:56.728+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:31:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9346.328858 secs 2024-07-26T09:31:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:57 smithi042 ceph-mon[32775]: pgmap v5533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:31:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9364.069965 secs 2024-07-26T09:31:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9346.328858 secs 2024-07-26T09:31:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:57 smithi164 ceph-mon[36970]: pgmap v5533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:31:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9364.069965 secs 2024-07-26T09:31:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9346.328858 secs 2024-07-26T09:31:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:57 smithi163 ceph-mon[36873]: pgmap v5533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:31:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9364.069965 secs 2024-07-26T09:31:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:31:59.313+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:31:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:31:59.313+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T09:31:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:31:59.314+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:31:59.316 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:31:59.314+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T09:31:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:31:59 smithi042 ceph-mon[32775]: pgmap v5534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:31:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:31:59 smithi164 ceph-mon[36970]: pgmap v5534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:31:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:31:59 smithi163 ceph-mon[36873]: pgmap v5534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:32:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:01 smithi042 ceph-mon[32775]: pgmap v5535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:32:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:01 smithi164 ceph-mon[36970]: pgmap v5535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:32:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:01 smithi163 ceph-mon[36873]: pgmap v5535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:32:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9351.329009 secs 2024-07-26T09:32:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9369.070082 secs 2024-07-26T09:32:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9351.329009 secs 2024-07-26T09:32:02.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9369.070082 secs 2024-07-26T09:32:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9351.329009 secs 2024-07-26T09:32:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9369.070082 secs 2024-07-26T09:32:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:03 smithi042 ceph-mon[32775]: pgmap v5536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:32:03.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:03 smithi164 ceph-mon[36970]: pgmap v5536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:32:03.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:03 smithi163 ceph-mon[36873]: pgmap v5536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:32:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:05 smithi042 ceph-mon[32775]: pgmap v5537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:32:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:05 smithi164 ceph-mon[36970]: pgmap v5537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:32:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:05 smithi163 ceph-mon[36873]: pgmap v5537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:32:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9356.329130 secs 2024-07-26T09:32:06.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9356.329130 secs 2024-07-26T09:32:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9356.329130 secs 2024-07-26T09:32:07.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:07 smithi042 ceph-mon[32775]: pgmap v5538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:07.399 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9374.070167 secs 2024-07-26T09:32:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:07 smithi164 ceph-mon[36970]: pgmap v5538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9374.070167 secs 2024-07-26T09:32:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:07 smithi163 ceph-mon[36873]: pgmap v5538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9374.070167 secs 2024-07-26T09:32:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:09 smithi042 ceph-mon[32775]: pgmap v5539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:09 smithi164 ceph-mon[36970]: pgmap v5539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:09 smithi163 ceph-mon[36873]: pgmap v5539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:11 smithi042 ceph-mon[32775]: pgmap v5540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9361.329253 secs 2024-07-26T09:32:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:11 smithi164 ceph-mon[36970]: pgmap v5540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:11.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9361.329253 secs 2024-07-26T09:32:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:11 smithi163 ceph-mon[36873]: pgmap v5540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9361.329253 secs 2024-07-26T09:32:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:32:11.734+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:32:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:32:11.734+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:32:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:32:11.734+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:32:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9379.070238 secs 2024-07-26T09:32:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9379.070238 secs 2024-07-26T09:32:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9379.070238 secs 2024-07-26T09:32:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:13 smithi042 ceph-mon[32775]: pgmap v5541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:13 smithi164 ceph-mon[36970]: pgmap v5541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:13 smithi163 ceph-mon[36873]: pgmap v5541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:15 smithi042 ceph-mon[32775]: pgmap v5542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:32:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:15 smithi164 ceph-mon[36970]: pgmap v5542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:32:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:15 smithi163 ceph-mon[36873]: pgmap v5542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:32:16.389 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9366.329373 secs 2024-07-26T09:32:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9366.329373 secs 2024-07-26T09:32:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9366.329373 secs 2024-07-26T09:32:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:17 smithi042 ceph-mon[32775]: pgmap v5543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9384.070384 secs 2024-07-26T09:32:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:17 smithi164 ceph-mon[36970]: pgmap v5543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9384.070384 secs 2024-07-26T09:32:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:17 smithi163 ceph-mon[36873]: pgmap v5543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9384.070384 secs 2024-07-26T09:32:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:19 smithi042 ceph-mon[32775]: pgmap v5544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:19 smithi164 ceph-mon[36970]: pgmap v5544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:19 smithi163 ceph-mon[36873]: pgmap v5544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:21 smithi042 ceph-mon[32775]: pgmap v5545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9371.329509 secs 2024-07-26T09:32:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:21 smithi164 ceph-mon[36970]: pgmap v5545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:21.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9371.329509 secs 2024-07-26T09:32:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:21 smithi163 ceph-mon[36873]: pgmap v5545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9371.329509 secs 2024-07-26T09:32:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9389.070484 secs 2024-07-26T09:32:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9389.070484 secs 2024-07-26T09:32:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9389.070484 secs 2024-07-26T09:32:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:23 smithi164 ceph-mon[36970]: pgmap v5546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:23 smithi163 ceph-mon[36873]: pgmap v5546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:23 smithi042 ceph-mon[32775]: pgmap v5546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:25 smithi164 ceph-mon[36970]: pgmap v5547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:32:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:25 smithi163 ceph-mon[36873]: pgmap v5547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:32:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:25 smithi042 ceph-mon[32775]: pgmap v5547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:32:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9376.329587 secs 2024-07-26T09:32:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9376.329587 secs 2024-07-26T09:32:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9376.329587 secs 2024-07-26T09:32:26.741 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:32:26.739+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:32:26.741 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:32:26.739+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:32:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:32:26.739+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:32:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:27 smithi164 ceph-mon[36970]: pgmap v5548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9394.070617 secs 2024-07-26T09:32:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:27 smithi163 ceph-mon[36873]: pgmap v5548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9394.070617 secs 2024-07-26T09:32:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:27 smithi042 ceph-mon[32775]: pgmap v5548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9394.070617 secs 2024-07-26T09:32:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:29 smithi164 ceph-mon[36970]: pgmap v5549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:29 smithi163 ceph-mon[36873]: pgmap v5549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:29 smithi042 ceph-mon[32775]: pgmap v5549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:31 smithi164 ceph-mon[36970]: pgmap v5550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:31.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9381.329716 secs 2024-07-26T09:32:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:31 smithi163 ceph-mon[36873]: pgmap v5550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9381.329716 secs 2024-07-26T09:32:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:31 smithi042 ceph-mon[32775]: pgmap v5550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9381.329716 secs 2024-07-26T09:32:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9399.070710 secs 2024-07-26T09:32:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9399.070710 secs 2024-07-26T09:32:32.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9399.070710 secs 2024-07-26T09:32:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:33 smithi164 ceph-mon[36970]: pgmap v5551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:33 smithi163 ceph-mon[36873]: pgmap v5551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:33 smithi042 ceph-mon[32775]: pgmap v5551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:35.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:35 smithi042 ceph-mon[32775]: pgmap v5552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:32:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:35 smithi164 ceph-mon[36970]: pgmap v5552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:32:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:35 smithi163 ceph-mon[36873]: pgmap v5552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:32:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9386.329853 secs 2024-07-26T09:32:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9386.329853 secs 2024-07-26T09:32:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9386.329853 secs 2024-07-26T09:32:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:37 smithi164 ceph-mon[36970]: pgmap v5553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9404.070842 secs 2024-07-26T09:32:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:37 smithi163 ceph-mon[36873]: pgmap v5553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9404.070842 secs 2024-07-26T09:32:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:37 smithi042 ceph-mon[32775]: pgmap v5553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9404.070842 secs 2024-07-26T09:32:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:39 smithi164 ceph-mon[36970]: pgmap v5554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:39 smithi163 ceph-mon[36873]: pgmap v5554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:39 smithi042 ceph-mon[32775]: pgmap v5554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:41 smithi164 ceph-mon[36970]: pgmap v5555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:41.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9391.329967 secs 2024-07-26T09:32:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:41 smithi163 ceph-mon[36873]: pgmap v5555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9391.329967 secs 2024-07-26T09:32:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:41 smithi042 ceph-mon[32775]: pgmap v5555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9391.329967 secs 2024-07-26T09:32:41.751 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:32:41.749+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:32:41.751 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:32:41.749+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:32:41.751 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:32:41.749+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:32:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9409.070985 secs 2024-07-26T09:32:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9409.070985 secs 2024-07-26T09:32:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9409.070985 secs 2024-07-26T09:32:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:43 smithi164 ceph-mon[36970]: pgmap v5556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:43 smithi163 ceph-mon[36873]: pgmap v5556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:43 smithi042 ceph-mon[32775]: pgmap v5556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:32:44.321+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:32:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:32:44.321+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T09:32:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:32:44.321+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T09:32:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:32:44.321+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:32:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:45 smithi164 ceph-mon[36970]: pgmap v5557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:32:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:45 smithi163 ceph-mon[36873]: pgmap v5557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:32:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:45 smithi042 ceph-mon[32775]: pgmap v5557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:32:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9396.330089 secs 2024-07-26T09:32:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9396.330089 secs 2024-07-26T09:32:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9396.330089 secs 2024-07-26T09:32:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:47 smithi164 ceph-mon[36970]: pgmap v5558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9414.071050 secs 2024-07-26T09:32:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:47 smithi163 ceph-mon[36873]: pgmap v5558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9414.071050 secs 2024-07-26T09:32:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:47 smithi042 ceph-mon[32775]: pgmap v5558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9414.071050 secs 2024-07-26T09:32:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:49 smithi164 ceph-mon[36970]: pgmap v5559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:49 smithi163 ceph-mon[36873]: pgmap v5559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:49 smithi042 ceph-mon[32775]: pgmap v5559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:51 smithi042 ceph-mon[32775]: pgmap v5560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9401.330227 secs 2024-07-26T09:32:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:51 smithi164 ceph-mon[36970]: pgmap v5560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:51.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9401.330227 secs 2024-07-26T09:32:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:51 smithi163 ceph-mon[36873]: pgmap v5560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:32:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9401.330227 secs 2024-07-26T09:32:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9419.071170 secs 2024-07-26T09:32:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:32:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9419.071170 secs 2024-07-26T09:32:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:32:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9419.071170 secs 2024-07-26T09:32:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:32:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:53 smithi042 ceph-mon[32775]: pgmap v5561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:32:53.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:32:53.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:53 smithi164 ceph-mon[36970]: pgmap v5561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:53.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:32:53.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:32:53.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:53 smithi163 ceph-mon[36873]: pgmap v5561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:32:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:53.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:32:53.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:32:53.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:32:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:54 smithi164 ceph-mon[36970]: pgmap v5562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T09:32:54.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:54 smithi164 ceph-mon[36970]: pgmap v5563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:32:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:54 smithi163 ceph-mon[36873]: pgmap v5562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T09:32:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:54 smithi163 ceph-mon[36873]: pgmap v5563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:32:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:54 smithi042 ceph-mon[32775]: pgmap v5562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T09:32:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:54 smithi042 ceph-mon[32775]: pgmap v5563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:32:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:56 smithi164 ceph-mon[36970]: pgmap v5564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:32:56.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9406.330328 secs 2024-07-26T09:32:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:32:56.750+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:32:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:32:56.750+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:32:56.752 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:32:56.750+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:32:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:56 smithi163 ceph-mon[36873]: pgmap v5564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:32:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9406.330328 secs 2024-07-26T09:32:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:56 smithi042 ceph-mon[32775]: pgmap v5564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:32:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9406.330328 secs 2024-07-26T09:32:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9424.071246 secs 2024-07-26T09:32:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9424.071246 secs 2024-07-26T09:32:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9424.071246 secs 2024-07-26T09:32:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:32:58 smithi164 ceph-mon[36970]: pgmap v5565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:32:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:32:58 smithi163 ceph-mon[36873]: pgmap v5565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:32:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:32:58 smithi042 ceph-mon[32775]: pgmap v5565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:33:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:00 smithi164 ceph-mon[36970]: pgmap v5566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T09:33:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:00 smithi163 ceph-mon[36873]: pgmap v5566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T09:33:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:00 smithi042 ceph-mon[32775]: pgmap v5566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T09:33:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9411.330429 secs 2024-07-26T09:33:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9411.330429 secs 2024-07-26T09:33:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9411.330429 secs 2024-07-26T09:33:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:02 smithi164 ceph-mon[36970]: pgmap v5567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:33:02.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9429.071364 secs 2024-07-26T09:33:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:02 smithi163 ceph-mon[36873]: pgmap v5567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:33:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9429.071364 secs 2024-07-26T09:33:02.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:02 smithi042 ceph-mon[32775]: pgmap v5567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:33:02.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9429.071364 secs 2024-07-26T09:33:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:04 smithi164 ceph-mon[36970]: pgmap v5568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:33:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:04 smithi163 ceph-mon[36873]: pgmap v5568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:33:04.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:04 smithi042 ceph-mon[32775]: pgmap v5568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:33:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:06 smithi164 ceph-mon[36970]: pgmap v5569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:06.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9416.330514 secs 2024-07-26T09:33:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:06 smithi163 ceph-mon[36873]: pgmap v5569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9416.330514 secs 2024-07-26T09:33:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:06 smithi042 ceph-mon[32775]: pgmap v5569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9416.330514 secs 2024-07-26T09:33:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9434.071490 secs 2024-07-26T09:33:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9434.071490 secs 2024-07-26T09:33:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9434.071490 secs 2024-07-26T09:33:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:08 smithi164 ceph-mon[36970]: pgmap v5570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:08 smithi163 ceph-mon[36873]: pgmap v5570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:08 smithi042 ceph-mon[32775]: pgmap v5570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:10 smithi164 ceph-mon[36970]: pgmap v5571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:33:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:10 smithi163 ceph-mon[36873]: pgmap v5571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:33:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:10 smithi042 ceph-mon[32775]: pgmap v5571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:33:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9421.330643 secs 2024-07-26T09:33:11.752 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:33:11.751+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:33:11.753 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:33:11.751+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:33:11.753 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:33:11.751+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:33:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9421.330643 secs 2024-07-26T09:33:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9421.330643 secs 2024-07-26T09:33:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:12 smithi164 ceph-mon[36970]: pgmap v5572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:12.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9439.071616 secs 2024-07-26T09:33:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:12 smithi163 ceph-mon[36873]: pgmap v5572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9439.071616 secs 2024-07-26T09:33:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:12 smithi042 ceph-mon[32775]: pgmap v5572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9439.071616 secs 2024-07-26T09:33:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:14 smithi164 ceph-mon[36970]: pgmap v5573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:14 smithi163 ceph-mon[36873]: pgmap v5573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:14 smithi042 ceph-mon[32775]: pgmap v5573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:16 smithi164 ceph-mon[36970]: pgmap v5574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9426.330706 secs 2024-07-26T09:33:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:16 smithi163 ceph-mon[36873]: pgmap v5574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9426.330706 secs 2024-07-26T09:33:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:16 smithi042 ceph-mon[32775]: pgmap v5574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9426.330706 secs 2024-07-26T09:33:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9444.071737 secs 2024-07-26T09:33:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9444.071737 secs 2024-07-26T09:33:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9444.071737 secs 2024-07-26T09:33:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:18 smithi164 ceph-mon[36970]: pgmap v5575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:18 smithi163 ceph-mon[36873]: pgmap v5575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:18 smithi042 ceph-mon[32775]: pgmap v5575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:20 smithi164 ceph-mon[36970]: pgmap v5576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:33:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:20 smithi163 ceph-mon[36873]: pgmap v5576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:33:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:20 smithi042 ceph-mon[32775]: pgmap v5576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:33:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9431.330831 secs 2024-07-26T09:33:21.777 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9431.330831 secs 2024-07-26T09:33:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9431.330831 secs 2024-07-26T09:33:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:22 smithi163 ceph-mon[36873]: pgmap v5577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9449.071858 secs 2024-07-26T09:33:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:22 smithi042 ceph-mon[32775]: pgmap v5577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9449.071858 secs 2024-07-26T09:33:22.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:22 smithi164 ceph-mon[36970]: pgmap v5577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:22.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9449.071858 secs 2024-07-26T09:33:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:24 smithi163 ceph-mon[36873]: pgmap v5578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:24.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:24 smithi042 ceph-mon[32775]: pgmap v5578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:24 smithi164 ceph-mon[36970]: pgmap v5578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:26.757 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:33:26.755+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:33:26.757 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:33:26.755+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:33:26.757 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:33:26.755+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:33:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:26 smithi163 ceph-mon[36873]: pgmap v5579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9436.330955 secs 2024-07-26T09:33:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9454.071955 secs 2024-07-26T09:33:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:26 smithi042 ceph-mon[32775]: pgmap v5579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9436.330955 secs 2024-07-26T09:33:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9454.071955 secs 2024-07-26T09:33:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:26 smithi164 ceph-mon[36970]: pgmap v5579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9436.330955 secs 2024-07-26T09:33:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9454.071955 secs 2024-07-26T09:33:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:28 smithi163 ceph-mon[36873]: pgmap v5580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:28 smithi042 ceph-mon[32775]: pgmap v5580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:28 smithi164 ceph-mon[36970]: pgmap v5580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:30 smithi163 ceph-mon[36873]: pgmap v5581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:33:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:30 smithi042 ceph-mon[32775]: pgmap v5581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:33:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:30 smithi164 ceph-mon[36970]: pgmap v5581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:33:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9441.331080 secs 2024-07-26T09:33:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9459.072078 secs 2024-07-26T09:33:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9441.331080 secs 2024-07-26T09:33:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9459.072078 secs 2024-07-26T09:33:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9441.331080 secs 2024-07-26T09:33:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9459.072078 secs 2024-07-26T09:33:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:32 smithi163 ceph-mon[36873]: pgmap v5582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:32.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:32 smithi042 ceph-mon[32775]: pgmap v5582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:32 smithi164 ceph-mon[36970]: pgmap v5582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:34 smithi163 ceph-mon[36873]: pgmap v5583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:34 smithi042 ceph-mon[32775]: pgmap v5583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:34 smithi164 ceph-mon[36970]: pgmap v5583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:36 smithi163 ceph-mon[36873]: pgmap v5584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9446.331204 secs 2024-07-26T09:33:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9464.072193 secs 2024-07-26T09:33:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:36 smithi042 ceph-mon[32775]: pgmap v5584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9446.331204 secs 2024-07-26T09:33:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9464.072193 secs 2024-07-26T09:33:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:36 smithi164 ceph-mon[36970]: pgmap v5584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9446.331204 secs 2024-07-26T09:33:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9464.072193 secs 2024-07-26T09:33:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:38 smithi042 ceph-mon[32775]: pgmap v5585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:38 smithi164 ceph-mon[36970]: pgmap v5585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:38 smithi163 ceph-mon[36873]: pgmap v5585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:40 smithi042 ceph-mon[32775]: pgmap v5586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:33:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:40 smithi164 ceph-mon[36970]: pgmap v5586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:33:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:40 smithi163 ceph-mon[36873]: pgmap v5586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:33:41.759 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:33:41.757+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:33:41.759 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:33:41.757+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:33:41.759 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:33:41.757+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:33:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9451.331347 secs 2024-07-26T09:33:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9469.072284 secs 2024-07-26T09:33:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9451.331347 secs 2024-07-26T09:33:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9469.072284 secs 2024-07-26T09:33:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9451.331347 secs 2024-07-26T09:33:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9469.072284 secs 2024-07-26T09:33:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:42 smithi042 ceph-mon[32775]: pgmap v5587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:42 smithi164 ceph-mon[36970]: pgmap v5587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:42 smithi163 ceph-mon[36873]: pgmap v5587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:44 smithi042 ceph-mon[32775]: pgmap v5588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:44 smithi164 ceph-mon[36970]: pgmap v5588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:44 smithi163 ceph-mon[36873]: pgmap v5588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:46 smithi042 ceph-mon[32775]: pgmap v5589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9456.331470 secs 2024-07-26T09:33:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9474.072405 secs 2024-07-26T09:33:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:46 smithi164 ceph-mon[36970]: pgmap v5589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9456.331470 secs 2024-07-26T09:33:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9474.072405 secs 2024-07-26T09:33:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:46 smithi163 ceph-mon[36873]: pgmap v5589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9456.331470 secs 2024-07-26T09:33:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9474.072405 secs 2024-07-26T09:33:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:48 smithi042 ceph-mon[32775]: pgmap v5590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:48 smithi164 ceph-mon[36970]: pgmap v5590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:48 smithi163 ceph-mon[36873]: pgmap v5590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:50 smithi042 ceph-mon[32775]: pgmap v5591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:33:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:50 smithi164 ceph-mon[36970]: pgmap v5591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:33:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:50 smithi163 ceph-mon[36873]: pgmap v5591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:33:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9461.331599 secs 2024-07-26T09:33:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9479.072542 secs 2024-07-26T09:33:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9461.331599 secs 2024-07-26T09:33:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9479.072542 secs 2024-07-26T09:33:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9461.331599 secs 2024-07-26T09:33:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9479.072542 secs 2024-07-26T09:33:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:52 smithi042 ceph-mon[32775]: pgmap v5592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:52 smithi164 ceph-mon[36970]: pgmap v5592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:52 smithi163 ceph-mon[36873]: pgmap v5592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:33:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:33:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:33:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:33:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:33:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:33:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:33:53.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:33:53.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:33:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:33:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:33:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:33:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:33:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:54 smithi042 ceph-mon[32775]: pgmap v5593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:54 smithi042 ceph-mon[32775]: pgmap v5594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:33:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:54 smithi042 ceph-mon[32775]: pgmap v5595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:33:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:54 smithi164 ceph-mon[36970]: pgmap v5593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:54 smithi164 ceph-mon[36970]: pgmap v5594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:33:54.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:54 smithi164 ceph-mon[36970]: pgmap v5595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:33:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:54 smithi163 ceph-mon[36873]: pgmap v5593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:33:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:54 smithi163 ceph-mon[36873]: pgmap v5594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:33:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:54 smithi163 ceph-mon[36873]: pgmap v5595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:33:56.764 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:33:56.762+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:33:56.764 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:33:56.762+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:33:56.765 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:33:56.762+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:33:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:56 smithi042 ceph-mon[32775]: pgmap v5596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:33:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9466.331674 secs 2024-07-26T09:33:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9484.072681 secs 2024-07-26T09:33:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:56 smithi164 ceph-mon[36970]: pgmap v5596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:33:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9466.331674 secs 2024-07-26T09:33:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9484.072681 secs 2024-07-26T09:33:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:56 smithi163 ceph-mon[36873]: pgmap v5596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:33:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9466.331674 secs 2024-07-26T09:33:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9484.072681 secs 2024-07-26T09:33:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:33:58 smithi042 ceph-mon[32775]: pgmap v5597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:33:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:33:58 smithi164 ceph-mon[36970]: pgmap v5597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:33:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:33:58 smithi163 ceph-mon[36873]: pgmap v5597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:34:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:00 smithi042 ceph-mon[32775]: pgmap v5598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:34:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:00 smithi164 ceph-mon[36970]: pgmap v5598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:34:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:00 smithi163 ceph-mon[36873]: pgmap v5598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:34:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9471.331803 secs 2024-07-26T09:34:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9489.072829 secs 2024-07-26T09:34:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9471.331803 secs 2024-07-26T09:34:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9489.072829 secs 2024-07-26T09:34:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9471.331803 secs 2024-07-26T09:34:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9489.072829 secs 2024-07-26T09:34:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:02 smithi042 ceph-mon[32775]: pgmap v5599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:34:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:02 smithi164 ceph-mon[36970]: pgmap v5599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:34:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:02 smithi163 ceph-mon[36873]: pgmap v5599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:34:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:04 smithi042 ceph-mon[32775]: pgmap v5600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:34:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:04 smithi164 ceph-mon[36970]: pgmap v5600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:34:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:04 smithi163 ceph-mon[36873]: pgmap v5600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:34:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:06 smithi042 ceph-mon[32775]: pgmap v5601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9476.331919 secs 2024-07-26T09:34:06.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9494.072994 secs 2024-07-26T09:34:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:06 smithi164 ceph-mon[36970]: pgmap v5601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9476.331919 secs 2024-07-26T09:34:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9494.072994 secs 2024-07-26T09:34:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:06 smithi163 ceph-mon[36873]: pgmap v5601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9476.331919 secs 2024-07-26T09:34:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9494.072994 secs 2024-07-26T09:34:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:08 smithi042 ceph-mon[32775]: pgmap v5602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:08.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:08 smithi164 ceph-mon[36970]: pgmap v5602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:08 smithi163 ceph-mon[36873]: pgmap v5602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:10 smithi042 ceph-mon[32775]: pgmap v5603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:10 smithi164 ceph-mon[36970]: pgmap v5603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:10 smithi163 ceph-mon[36873]: pgmap v5603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:34:11.772+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:34:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:34:11.772+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:34:11.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:34:11.772+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:34:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9481.332046 secs 2024-07-26T09:34:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9499.073109 secs 2024-07-26T09:34:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9481.332046 secs 2024-07-26T09:34:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9499.073109 secs 2024-07-26T09:34:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9481.332046 secs 2024-07-26T09:34:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9499.073109 secs 2024-07-26T09:34:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:12 smithi042 ceph-mon[32775]: pgmap v5604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:12 smithi164 ceph-mon[36970]: pgmap v5604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:12 smithi163 ceph-mon[36873]: pgmap v5604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:14 smithi042 ceph-mon[32775]: pgmap v5605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:34:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:14 smithi164 ceph-mon[36970]: pgmap v5605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:34:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:14 smithi163 ceph-mon[36873]: pgmap v5605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:34:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:16 smithi042 ceph-mon[32775]: pgmap v5606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9486.332143 secs 2024-07-26T09:34:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9504.073228 secs 2024-07-26T09:34:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:16 smithi164 ceph-mon[36970]: pgmap v5606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9486.332143 secs 2024-07-26T09:34:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9504.073228 secs 2024-07-26T09:34:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:16 smithi163 ceph-mon[36873]: pgmap v5606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9486.332143 secs 2024-07-26T09:34:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9504.073228 secs 2024-07-26T09:34:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:18 smithi042 ceph-mon[32775]: pgmap v5607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:18 smithi164 ceph-mon[36970]: pgmap v5607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:18 smithi163 ceph-mon[36873]: pgmap v5607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:20 smithi042 ceph-mon[32775]: pgmap v5608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:20 smithi164 ceph-mon[36970]: pgmap v5608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:20 smithi163 ceph-mon[36873]: pgmap v5608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9491.332275 secs 2024-07-26T09:34:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9509.073349 secs 2024-07-26T09:34:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9491.332275 secs 2024-07-26T09:34:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9509.073349 secs 2024-07-26T09:34:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9491.332275 secs 2024-07-26T09:34:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9509.073349 secs 2024-07-26T09:34:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:22 smithi042 ceph-mon[32775]: pgmap v5609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:22 smithi164 ceph-mon[36970]: pgmap v5609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:22 smithi163 ceph-mon[36873]: pgmap v5609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:24 smithi042 ceph-mon[32775]: pgmap v5610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:34:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:24 smithi164 ceph-mon[36970]: pgmap v5610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:34:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:24 smithi163 ceph-mon[36873]: pgmap v5610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:34:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:34:26.777+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:34:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:34:26.777+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:34:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:34:26.777+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:34:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:26 smithi042 ceph-mon[32775]: pgmap v5611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9496.332390 secs 2024-07-26T09:34:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9514.073488 secs 2024-07-26T09:34:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:26 smithi164 ceph-mon[36970]: pgmap v5611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9496.332390 secs 2024-07-26T09:34:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9514.073488 secs 2024-07-26T09:34:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:26 smithi163 ceph-mon[36873]: pgmap v5611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9496.332390 secs 2024-07-26T09:34:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9514.073488 secs 2024-07-26T09:34:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:28 smithi042 ceph-mon[32775]: pgmap v5612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:28 smithi164 ceph-mon[36970]: pgmap v5612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:28 smithi163 ceph-mon[36873]: pgmap v5612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:30 smithi042 ceph-mon[32775]: pgmap v5613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:30 smithi164 ceph-mon[36970]: pgmap v5613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:30 smithi163 ceph-mon[36873]: pgmap v5613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9501.332517 secs 2024-07-26T09:34:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9519.073632 secs 2024-07-26T09:34:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9501.332517 secs 2024-07-26T09:34:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9519.073632 secs 2024-07-26T09:34:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9501.332517 secs 2024-07-26T09:34:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9519.073632 secs 2024-07-26T09:34:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:32 smithi042 ceph-mon[32775]: pgmap v5614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:32 smithi164 ceph-mon[36970]: pgmap v5614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:33.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:32 smithi163 ceph-mon[36873]: pgmap v5614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:34 smithi042 ceph-mon[32775]: pgmap v5615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:34:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:34 smithi164 ceph-mon[36970]: pgmap v5615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:34:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:34 smithi163 ceph-mon[36873]: pgmap v5615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:34:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:36 smithi042 ceph-mon[32775]: pgmap v5616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9506.332592 secs 2024-07-26T09:34:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9524.073704 secs 2024-07-26T09:34:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:36 smithi164 ceph-mon[36970]: pgmap v5616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9506.332592 secs 2024-07-26T09:34:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9524.073704 secs 2024-07-26T09:34:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:36 smithi163 ceph-mon[36873]: pgmap v5616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9506.332592 secs 2024-07-26T09:34:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9524.073704 secs 2024-07-26T09:34:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:37 smithi164 ceph-mon[36970]: pgmap v5617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:37 smithi163 ceph-mon[36873]: pgmap v5617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:37 smithi042 ceph-mon[32775]: pgmap v5617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:40.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:40 smithi164 ceph-mon[36970]: pgmap v5618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:40 smithi163 ceph-mon[36873]: pgmap v5618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:40 smithi042 ceph-mon[32775]: pgmap v5618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9511.332715 secs 2024-07-26T09:34:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9511.332715 secs 2024-07-26T09:34:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:34:41.787+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:34:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:34:41.787+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:34:41.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:34:41.787+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:34:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9511.332715 secs 2024-07-26T09:34:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9529.073825 secs 2024-07-26T09:34:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:42 smithi164 ceph-mon[36970]: pgmap v5619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9529.073825 secs 2024-07-26T09:34:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:42 smithi163 ceph-mon[36873]: pgmap v5619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9529.073825 secs 2024-07-26T09:34:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:42 smithi042 ceph-mon[32775]: pgmap v5619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:44 smithi164 ceph-mon[36970]: pgmap v5620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:34:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:44 smithi163 ceph-mon[36873]: pgmap v5620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:34:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:44 smithi042 ceph-mon[32775]: pgmap v5620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:34:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:46 smithi164 ceph-mon[36970]: pgmap v5621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:46.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9516.332844 secs 2024-07-26T09:34:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:46 smithi163 ceph-mon[36873]: pgmap v5621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9516.332844 secs 2024-07-26T09:34:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:46 smithi042 ceph-mon[32775]: pgmap v5621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9516.332844 secs 2024-07-26T09:34:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9534.073946 secs 2024-07-26T09:34:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9534.073946 secs 2024-07-26T09:34:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9534.073946 secs 2024-07-26T09:34:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:48 smithi164 ceph-mon[36970]: pgmap v5622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:48 smithi163 ceph-mon[36873]: pgmap v5622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:48 smithi042 ceph-mon[32775]: pgmap v5622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:50 smithi164 ceph-mon[36970]: pgmap v5623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:50 smithi163 ceph-mon[36873]: pgmap v5623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:50 smithi042 ceph-mon[32775]: pgmap v5623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:34:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9521.332974 secs 2024-07-26T09:34:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9521.332974 secs 2024-07-26T09:34:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9521.332974 secs 2024-07-26T09:34:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9539.074044 secs 2024-07-26T09:34:52.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:52 smithi164 ceph-mon[36970]: pgmap v5624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9539.074044 secs 2024-07-26T09:34:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:52 smithi163 ceph-mon[36873]: pgmap v5624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9539.074044 secs 2024-07-26T09:34:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:52 smithi042 ceph-mon[32775]: pgmap v5624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:34:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:54 smithi163 ceph-mon[36873]: pgmap v5625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:34:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:34:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:34:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:34:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:34:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:54 smithi042 ceph-mon[32775]: pgmap v5625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:34:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:34:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:34:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:34:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:34:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:54 smithi164 ceph-mon[36970]: pgmap v5625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:34:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:34:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:34:54.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:34:54.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:34:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:55 smithi163 ceph-mon[36873]: pgmap v5626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:34:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:55 smithi163 ceph-mon[36873]: pgmap v5627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:34:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:55 smithi042 ceph-mon[32775]: pgmap v5626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:34:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:55 smithi042 ceph-mon[32775]: pgmap v5627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:34:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:55 smithi164 ceph-mon[36970]: pgmap v5626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:34:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:55 smithi164 ceph-mon[36970]: pgmap v5627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:34:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9526.333106 secs 2024-07-26T09:34:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:34:56.790+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:34:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:34:56.790+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:34:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:34:56.790+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:34:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9526.333106 secs 2024-07-26T09:34:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9526.333106 secs 2024-07-26T09:34:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:57 smithi163 ceph-mon[36873]: pgmap v5628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:34:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9544.074164 secs 2024-07-26T09:34:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:57 smithi042 ceph-mon[32775]: pgmap v5628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:34:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9544.074164 secs 2024-07-26T09:34:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:57 smithi164 ceph-mon[36970]: pgmap v5628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:34:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9544.074164 secs 2024-07-26T09:34:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:34:59.342+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T09:34:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:34:59.342+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:34:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:34:59.342+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:34:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:34:59.342+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T09:34:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:34:59 smithi163 ceph-mon[36873]: pgmap v5629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:34:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:34:59 smithi042 ceph-mon[32775]: pgmap v5629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:34:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:34:59 smithi164 ceph-mon[36970]: pgmap v5629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:35:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:01 smithi163 ceph-mon[36873]: pgmap v5630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:35:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9531.333260 secs 2024-07-26T09:35:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:01 smithi042 ceph-mon[32775]: pgmap v5630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:35:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9531.333260 secs 2024-07-26T09:35:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:01 smithi164 ceph-mon[36970]: pgmap v5630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:35:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9531.333260 secs 2024-07-26T09:35:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9549.074284 secs 2024-07-26T09:35:02.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9549.074284 secs 2024-07-26T09:35:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9549.074284 secs 2024-07-26T09:35:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:03 smithi163 ceph-mon[36873]: pgmap v5631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:35:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:03 smithi042 ceph-mon[32775]: pgmap v5631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:35:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:03 smithi164 ceph-mon[36970]: pgmap v5631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:35:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:05 smithi163 ceph-mon[36873]: pgmap v5632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:35:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:05 smithi042 ceph-mon[32775]: pgmap v5632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:35:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:05 smithi164 ceph-mon[36970]: pgmap v5632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:35:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9536.333331 secs 2024-07-26T09:35:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9554.074418 secs 2024-07-26T09:35:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9536.333331 secs 2024-07-26T09:35:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9554.074418 secs 2024-07-26T09:35:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9536.333331 secs 2024-07-26T09:35:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9554.074418 secs 2024-07-26T09:35:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:07 smithi042 ceph-mon[32775]: pgmap v5633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:07 smithi164 ceph-mon[36970]: pgmap v5633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:07 smithi163 ceph-mon[36873]: pgmap v5633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:09.809 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:09 smithi042 ceph-mon[32775]: pgmap v5634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:09 smithi164 ceph-mon[36970]: pgmap v5634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:09 smithi163 ceph-mon[36873]: pgmap v5634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:35:11.800+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:35:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:35:11.800+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:35:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:35:11.800+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:35:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:11 smithi042 ceph-mon[32775]: pgmap v5635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9541.333464 secs 2024-07-26T09:35:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9559.074540 secs 2024-07-26T09:35:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:11 smithi164 ceph-mon[36970]: pgmap v5635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9541.333464 secs 2024-07-26T09:35:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9559.074540 secs 2024-07-26T09:35:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:11 smithi163 ceph-mon[36873]: pgmap v5635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9541.333464 secs 2024-07-26T09:35:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9559.074540 secs 2024-07-26T09:35:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:13 smithi042 ceph-mon[32775]: pgmap v5636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:13 smithi164 ceph-mon[36970]: pgmap v5636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:13 smithi163 ceph-mon[36873]: pgmap v5636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:15 smithi042 ceph-mon[32775]: pgmap v5637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:35:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:15 smithi164 ceph-mon[36970]: pgmap v5637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:35:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:15 smithi163 ceph-mon[36873]: pgmap v5637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:35:16.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9546.333552 secs 2024-07-26T09:35:16.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9564.074661 secs 2024-07-26T09:35:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9546.333552 secs 2024-07-26T09:35:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9564.074661 secs 2024-07-26T09:35:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9546.333552 secs 2024-07-26T09:35:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9564.074661 secs 2024-07-26T09:35:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:17 smithi042 ceph-mon[32775]: pgmap v5638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:17 smithi164 ceph-mon[36970]: pgmap v5638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:17 smithi163 ceph-mon[36873]: pgmap v5638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:19 smithi042 ceph-mon[32775]: pgmap v5639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:19 smithi164 ceph-mon[36970]: pgmap v5639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:19 smithi163 ceph-mon[36873]: pgmap v5639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:21 smithi042 ceph-mon[32775]: pgmap v5640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9551.333683 secs 2024-07-26T09:35:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9569.074736 secs 2024-07-26T09:35:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:21 smithi164 ceph-mon[36970]: pgmap v5640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9551.333683 secs 2024-07-26T09:35:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9569.074736 secs 2024-07-26T09:35:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:21 smithi163 ceph-mon[36873]: pgmap v5640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9551.333683 secs 2024-07-26T09:35:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9569.074736 secs 2024-07-26T09:35:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:23 smithi042 ceph-mon[32775]: pgmap v5641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:23 smithi164 ceph-mon[36970]: pgmap v5641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:23 smithi163 ceph-mon[36873]: pgmap v5641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:25 smithi042 ceph-mon[32775]: pgmap v5642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:35:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:25 smithi164 ceph-mon[36970]: pgmap v5642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:35:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:25 smithi163 ceph-mon[36873]: pgmap v5642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:35:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:35:26.805+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:35:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:35:26.805+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:35:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:35:26.805+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:35:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9556.333815 secs 2024-07-26T09:35:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9574.074836 secs 2024-07-26T09:35:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9556.333815 secs 2024-07-26T09:35:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9574.074836 secs 2024-07-26T09:35:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9556.333815 secs 2024-07-26T09:35:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9574.074836 secs 2024-07-26T09:35:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:27 smithi042 ceph-mon[32775]: pgmap v5643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:27 smithi164 ceph-mon[36970]: pgmap v5643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:27 smithi163 ceph-mon[36873]: pgmap v5643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:29 smithi042 ceph-mon[32775]: pgmap v5644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:29 smithi164 ceph-mon[36970]: pgmap v5644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:29 smithi163 ceph-mon[36873]: pgmap v5644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:31 smithi042 ceph-mon[32775]: pgmap v5645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9561.333922 secs 2024-07-26T09:35:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9579.074957 secs 2024-07-26T09:35:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:31 smithi164 ceph-mon[36970]: pgmap v5645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9561.333922 secs 2024-07-26T09:35:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9579.074957 secs 2024-07-26T09:35:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:31 smithi163 ceph-mon[36873]: pgmap v5645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9561.333922 secs 2024-07-26T09:35:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9579.074957 secs 2024-07-26T09:35:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:33 smithi042 ceph-mon[32775]: pgmap v5646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:33 smithi164 ceph-mon[36970]: pgmap v5646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:33 smithi163 ceph-mon[36873]: pgmap v5646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:35 smithi042 ceph-mon[32775]: pgmap v5647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:35:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:35 smithi164 ceph-mon[36970]: pgmap v5647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:35:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:35 smithi163 ceph-mon[36873]: pgmap v5647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:35:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9566.334068 secs 2024-07-26T09:35:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9584.075059 secs 2024-07-26T09:35:36.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9566.334068 secs 2024-07-26T09:35:36.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9584.075059 secs 2024-07-26T09:35:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9566.334068 secs 2024-07-26T09:35:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9584.075059 secs 2024-07-26T09:35:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:37 smithi042 ceph-mon[32775]: pgmap v5648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:37 smithi164 ceph-mon[36970]: pgmap v5648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:37 smithi163 ceph-mon[36873]: pgmap v5648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:39 smithi042 ceph-mon[32775]: pgmap v5649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:39 smithi164 ceph-mon[36970]: pgmap v5649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:39 smithi163 ceph-mon[36873]: pgmap v5649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:41.817 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:35:41.815+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:35:41.817 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:35:41.815+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:35:41.817 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:35:41.815+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:35:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:41 smithi042 ceph-mon[32775]: pgmap v5650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9571.334147 secs 2024-07-26T09:35:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9589.075188 secs 2024-07-26T09:35:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:41 smithi164 ceph-mon[36970]: pgmap v5650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9571.334147 secs 2024-07-26T09:35:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9589.075188 secs 2024-07-26T09:35:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:41 smithi163 ceph-mon[36873]: pgmap v5650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9571.334147 secs 2024-07-26T09:35:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9589.075188 secs 2024-07-26T09:35:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:43 smithi042 ceph-mon[32775]: pgmap v5651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:43 smithi164 ceph-mon[36970]: pgmap v5651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:43 smithi163 ceph-mon[36873]: pgmap v5651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:45 smithi042 ceph-mon[32775]: pgmap v5652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:35:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:45 smithi164 ceph-mon[36970]: pgmap v5652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:35:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:45 smithi163 ceph-mon[36873]: pgmap v5652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:35:46.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9576.334276 secs 2024-07-26T09:35:46.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9594.075326 secs 2024-07-26T09:35:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9576.334276 secs 2024-07-26T09:35:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9594.075326 secs 2024-07-26T09:35:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9576.334276 secs 2024-07-26T09:35:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9594.075326 secs 2024-07-26T09:35:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:47 smithi042 ceph-mon[32775]: pgmap v5653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:47 smithi164 ceph-mon[36970]: pgmap v5653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:47 smithi163 ceph-mon[36873]: pgmap v5653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:49 smithi042 ceph-mon[32775]: pgmap v5654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:49 smithi164 ceph-mon[36970]: pgmap v5654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:49 smithi163 ceph-mon[36873]: pgmap v5654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:51 smithi042 ceph-mon[32775]: pgmap v5655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9581.334399 secs 2024-07-26T09:35:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9599.075430 secs 2024-07-26T09:35:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:51 smithi164 ceph-mon[36970]: pgmap v5655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9581.334399 secs 2024-07-26T09:35:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9599.075430 secs 2024-07-26T09:35:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:51 smithi163 ceph-mon[36873]: pgmap v5655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:35:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9581.334399 secs 2024-07-26T09:35:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9599.075430 secs 2024-07-26T09:35:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:53 smithi042 ceph-mon[32775]: pgmap v5656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:53 smithi164 ceph-mon[36970]: pgmap v5656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:53 smithi163 ceph-mon[36873]: pgmap v5656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:35:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:35:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:35:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:35:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:35:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:35:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:35:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:35:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:35:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:35:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:35:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:35:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:35:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:55 smithi042 ceph-mon[32775]: pgmap v5657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:35:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:55 smithi042 ceph-mon[32775]: pgmap v5658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T09:35:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:55 smithi042 ceph-mon[32775]: pgmap v5659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:35:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:55 smithi164 ceph-mon[36970]: pgmap v5657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:35:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:55 smithi164 ceph-mon[36970]: pgmap v5658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T09:35:55.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:55 smithi164 ceph-mon[36970]: pgmap v5659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:35:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:55 smithi163 ceph-mon[36873]: pgmap v5657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:35:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:55 smithi163 ceph-mon[36873]: pgmap v5658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T09:35:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:55 smithi163 ceph-mon[36873]: pgmap v5659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:35:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:35:56.820+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:35:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:35:56.820+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:35:56.822 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:35:56.820+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:35:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9586.334523 secs 2024-07-26T09:35:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9604.075556 secs 2024-07-26T09:35:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9586.334523 secs 2024-07-26T09:35:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9604.075556 secs 2024-07-26T09:35:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9586.334523 secs 2024-07-26T09:35:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9604.075556 secs 2024-07-26T09:35:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:57 smithi042 ceph-mon[32775]: pgmap v5660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:35:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:57 smithi164 ceph-mon[36970]: pgmap v5660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:35:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:57 smithi163 ceph-mon[36873]: pgmap v5660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:35:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:35:59 smithi042 ceph-mon[32775]: pgmap v5661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:35:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:35:59 smithi164 ceph-mon[36970]: pgmap v5661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:36:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:35:59 smithi163 ceph-mon[36873]: pgmap v5661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:36:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:01 smithi042 ceph-mon[32775]: pgmap v5662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:36:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9591.334610 secs 2024-07-26T09:36:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9609.075680 secs 2024-07-26T09:36:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:01 smithi164 ceph-mon[36970]: pgmap v5662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:36:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9591.334610 secs 2024-07-26T09:36:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9609.075680 secs 2024-07-26T09:36:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:01 smithi163 ceph-mon[36873]: pgmap v5662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:36:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9591.334610 secs 2024-07-26T09:36:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9609.075680 secs 2024-07-26T09:36:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:03 smithi042 ceph-mon[32775]: pgmap v5663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:36:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:03 smithi164 ceph-mon[36970]: pgmap v5663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:36:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:03 smithi163 ceph-mon[36873]: pgmap v5663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:36:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:05 smithi042 ceph-mon[32775]: pgmap v5664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:36:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:05 smithi164 ceph-mon[36970]: pgmap v5664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:36:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:05 smithi163 ceph-mon[36873]: pgmap v5664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:36:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9596.334736 secs 2024-07-26T09:36:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9614.075823 secs 2024-07-26T09:36:06.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9596.334736 secs 2024-07-26T09:36:06.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9614.075823 secs 2024-07-26T09:36:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9596.334736 secs 2024-07-26T09:36:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9614.075823 secs 2024-07-26T09:36:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:07 smithi042 ceph-mon[32775]: pgmap v5665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:07 smithi164 ceph-mon[36970]: pgmap v5665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:07 smithi163 ceph-mon[36873]: pgmap v5665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:09 smithi042 ceph-mon[32775]: pgmap v5666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:36:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:09 smithi164 ceph-mon[36970]: pgmap v5666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:36:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:09 smithi163 ceph-mon[36873]: pgmap v5666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:36:11.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:36:11.830+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:36:11.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:36:11.830+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:36:11.832 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:36:11.830+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:36:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:11 smithi164 ceph-mon[36970]: pgmap v5667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9601.334866 secs 2024-07-26T09:36:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9619.075970 secs 2024-07-26T09:36:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:11 smithi163 ceph-mon[36873]: pgmap v5667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9601.334866 secs 2024-07-26T09:36:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9619.075970 secs 2024-07-26T09:36:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:11 smithi042 ceph-mon[32775]: pgmap v5667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9601.334866 secs 2024-07-26T09:36:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9619.075970 secs 2024-07-26T09:36:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:13 smithi164 ceph-mon[36970]: pgmap v5668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:13 smithi163 ceph-mon[36873]: pgmap v5668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:13 smithi042 ceph-mon[32775]: pgmap v5668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:15 smithi164 ceph-mon[36970]: pgmap v5669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:15 smithi163 ceph-mon[36873]: pgmap v5669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:15 smithi042 ceph-mon[32775]: pgmap v5669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9606.334998 secs 2024-07-26T09:36:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9624.076098 secs 2024-07-26T09:36:17.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9606.334998 secs 2024-07-26T09:36:17.011 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9624.076098 secs 2024-07-26T09:36:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9606.334998 secs 2024-07-26T09:36:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9624.076098 secs 2024-07-26T09:36:17.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:17 smithi164 ceph-mon[36970]: pgmap v5670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:17 smithi163 ceph-mon[36873]: pgmap v5670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:17 smithi042 ceph-mon[32775]: pgmap v5670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:19 smithi164 ceph-mon[36970]: pgmap v5671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:36:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:19 smithi163 ceph-mon[36873]: pgmap v5671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:36:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:19 smithi042 ceph-mon[32775]: pgmap v5671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:36:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:21 smithi164 ceph-mon[36970]: pgmap v5672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9611.335127 secs 2024-07-26T09:36:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9629.076220 secs 2024-07-26T09:36:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:21 smithi163 ceph-mon[36873]: pgmap v5672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9611.335127 secs 2024-07-26T09:36:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9629.076220 secs 2024-07-26T09:36:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:21 smithi042 ceph-mon[32775]: pgmap v5672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9611.335127 secs 2024-07-26T09:36:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9629.076220 secs 2024-07-26T09:36:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:23 smithi164 ceph-mon[36970]: pgmap v5673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:23 smithi163 ceph-mon[36873]: pgmap v5673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:23 smithi042 ceph-mon[32775]: pgmap v5673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:25 smithi164 ceph-mon[36970]: pgmap v5674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:25 smithi163 ceph-mon[36873]: pgmap v5674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:25 smithi042 ceph-mon[32775]: pgmap v5674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:36:26.835+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:36:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:36:26.835+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:36:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:36:26.835+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:36:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9616.335240 secs 2024-07-26T09:36:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9634.076344 secs 2024-07-26T09:36:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9616.335240 secs 2024-07-26T09:36:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9634.076344 secs 2024-07-26T09:36:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9616.335240 secs 2024-07-26T09:36:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9634.076344 secs 2024-07-26T09:36:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:27 smithi164 ceph-mon[36970]: pgmap v5675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:27 smithi163 ceph-mon[36873]: pgmap v5675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:27 smithi042 ceph-mon[32775]: pgmap v5675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:29 smithi164 ceph-mon[36970]: pgmap v5676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:36:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:29 smithi163 ceph-mon[36873]: pgmap v5676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:36:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:29 smithi042 ceph-mon[32775]: pgmap v5676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:36:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:31 smithi164 ceph-mon[36970]: pgmap v5677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9621.335351 secs 2024-07-26T09:36:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9639.076472 secs 2024-07-26T09:36:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:31 smithi163 ceph-mon[36873]: pgmap v5677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9621.335351 secs 2024-07-26T09:36:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9639.076472 secs 2024-07-26T09:36:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:31 smithi042 ceph-mon[32775]: pgmap v5677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9621.335351 secs 2024-07-26T09:36:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9639.076472 secs 2024-07-26T09:36:33.949 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:33 smithi042 ceph-mon[32775]: pgmap v5678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:33 smithi164 ceph-mon[36970]: pgmap v5678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:33 smithi163 ceph-mon[36873]: pgmap v5678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:35 smithi164 ceph-mon[36970]: pgmap v5679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:35 smithi163 ceph-mon[36873]: pgmap v5679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:35 smithi042 ceph-mon[32775]: pgmap v5679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9626.335485 secs 2024-07-26T09:36:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9644.076570 secs 2024-07-26T09:36:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9626.335485 secs 2024-07-26T09:36:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9644.076570 secs 2024-07-26T09:36:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9626.335485 secs 2024-07-26T09:36:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9644.076570 secs 2024-07-26T09:36:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:37 smithi164 ceph-mon[36970]: pgmap v5680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:37 smithi163 ceph-mon[36873]: pgmap v5680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:37 smithi042 ceph-mon[32775]: pgmap v5680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:39 smithi164 ceph-mon[36970]: pgmap v5681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:36:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:39 smithi163 ceph-mon[36873]: pgmap v5681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:36:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:39 smithi042 ceph-mon[32775]: pgmap v5681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:36:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:36:41.845+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:36:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:36:41.845+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:36:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:36:41.845+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:36:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:41 smithi164 ceph-mon[36970]: pgmap v5682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9631.335625 secs 2024-07-26T09:36:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9649.076695 secs 2024-07-26T09:36:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:41 smithi163 ceph-mon[36873]: pgmap v5682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9631.335625 secs 2024-07-26T09:36:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9649.076695 secs 2024-07-26T09:36:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:41 smithi042 ceph-mon[32775]: pgmap v5682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9631.335625 secs 2024-07-26T09:36:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9649.076695 secs 2024-07-26T09:36:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:43 smithi164 ceph-mon[36970]: pgmap v5683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:43 smithi163 ceph-mon[36873]: pgmap v5683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:44.017 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:43 smithi042 ceph-mon[32775]: pgmap v5683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:45 smithi164 ceph-mon[36970]: pgmap v5684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:46.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:45 smithi163 ceph-mon[36873]: pgmap v5684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:45 smithi042 ceph-mon[32775]: pgmap v5684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9636.335721 secs 2024-07-26T09:36:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9654.076816 secs 2024-07-26T09:36:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9636.335721 secs 2024-07-26T09:36:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9654.076816 secs 2024-07-26T09:36:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9636.335721 secs 2024-07-26T09:36:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9654.076816 secs 2024-07-26T09:36:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:47 smithi163 ceph-mon[36873]: pgmap v5685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:47 smithi042 ceph-mon[32775]: pgmap v5685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:47 smithi164 ceph-mon[36970]: pgmap v5685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:49 smithi163 ceph-mon[36873]: pgmap v5686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:36:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:49 smithi042 ceph-mon[32775]: pgmap v5686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:36:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:49 smithi164 ceph-mon[36970]: pgmap v5686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:36:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:51 smithi163 ceph-mon[36873]: pgmap v5687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9641.335853 secs 2024-07-26T09:36:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9659.076915 secs 2024-07-26T09:36:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:51 smithi042 ceph-mon[32775]: pgmap v5687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9641.335853 secs 2024-07-26T09:36:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9659.076915 secs 2024-07-26T09:36:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:51 smithi164 ceph-mon[36970]: pgmap v5687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9641.335853 secs 2024-07-26T09:36:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9659.076915 secs 2024-07-26T09:36:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:53 smithi163 ceph-mon[36873]: pgmap v5688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:53 smithi042 ceph-mon[32775]: pgmap v5688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:53 smithi164 ceph-mon[36970]: pgmap v5688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:36:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:36:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:36:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:36:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:55 smithi163 ceph-mon[36873]: pgmap v5689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:36:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:36:56.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:36:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:55 smithi042 ceph-mon[32775]: pgmap v5689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:36:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:36:56.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:36:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:55 smithi164 ceph-mon[36970]: pgmap v5689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:36:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:36:56.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:36:56.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:36:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:36:56.849+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:36:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:36:56.849+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:36:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:36:56.849+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:36:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:56 smithi163 ceph-mon[36873]: pgmap v5690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:36:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:56 smithi163 ceph-mon[36873]: pgmap v5691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:36:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9646.335978 secs 2024-07-26T09:36:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9664.077040 secs 2024-07-26T09:36:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:56 smithi042 ceph-mon[32775]: pgmap v5690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:36:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:56 smithi042 ceph-mon[32775]: pgmap v5691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:36:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9646.335978 secs 2024-07-26T09:36:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9664.077040 secs 2024-07-26T09:36:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:56 smithi164 ceph-mon[36970]: pgmap v5690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:36:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:56 smithi164 ceph-mon[36970]: pgmap v5691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:36:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9646.335978 secs 2024-07-26T09:36:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9664.077040 secs 2024-07-26T09:36:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:36:58 smithi163 ceph-mon[36873]: pgmap v5692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:36:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:36:58 smithi042 ceph-mon[32775]: pgmap v5692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:36:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:36:58 smithi164 ceph-mon[36970]: pgmap v5692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:37:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:00 smithi163 ceph-mon[36873]: pgmap v5693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:37:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:00 smithi042 ceph-mon[32775]: pgmap v5693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:37:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:00 smithi164 ceph-mon[36970]: pgmap v5693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:37:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9651.336108 secs 2024-07-26T09:37:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9669.077183 secs 2024-07-26T09:37:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9651.336108 secs 2024-07-26T09:37:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9669.077183 secs 2024-07-26T09:37:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9651.336108 secs 2024-07-26T09:37:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9669.077183 secs 2024-07-26T09:37:03.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:02 smithi042 ceph-mon[32775]: pgmap v5694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:37:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:02 smithi164 ceph-mon[36970]: pgmap v5694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:37:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:02 smithi163 ceph-mon[36873]: pgmap v5694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:37:05.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:04 smithi042 ceph-mon[32775]: pgmap v5695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:37:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:04 smithi164 ceph-mon[36970]: pgmap v5695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:37:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:04 smithi163 ceph-mon[36873]: pgmap v5695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:37:07.061 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:06 smithi163 ceph-mon[36873]: pgmap v5696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:37:07.061 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9656.336260 secs 2024-07-26T09:37:07.061 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9674.077304 secs 2024-07-26T09:37:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:06 smithi042 ceph-mon[32775]: pgmap v5696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:37:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9656.336260 secs 2024-07-26T09:37:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9674.077304 secs 2024-07-26T09:37:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:06 smithi164 ceph-mon[36970]: pgmap v5696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:37:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9656.336260 secs 2024-07-26T09:37:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9674.077304 secs 2024-07-26T09:37:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:08 smithi042 ceph-mon[32775]: pgmap v5697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:08 smithi164 ceph-mon[36970]: pgmap v5697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:09.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:08 smithi163 ceph-mon[36873]: pgmap v5697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:10 smithi042 ceph-mon[32775]: pgmap v5698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:10 smithi164 ceph-mon[36970]: pgmap v5698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:10 smithi163 ceph-mon[36873]: pgmap v5698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:11.861 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:37:11.859+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:37:11.861 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:37:11.859+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:37:11.861 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:37:11.859+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:37:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9661.336392 secs 2024-07-26T09:37:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9679.077425 secs 2024-07-26T09:37:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9661.336392 secs 2024-07-26T09:37:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9679.077425 secs 2024-07-26T09:37:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9661.336392 secs 2024-07-26T09:37:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9679.077425 secs 2024-07-26T09:37:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:12 smithi042 ceph-mon[32775]: pgmap v5699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:12 smithi164 ceph-mon[36970]: pgmap v5699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:12 smithi163 ceph-mon[36873]: pgmap v5699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:14 smithi042 ceph-mon[32775]: pgmap v5700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:14 smithi164 ceph-mon[36970]: pgmap v5700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:14 smithi163 ceph-mon[36873]: pgmap v5700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:16 smithi042 ceph-mon[32775]: pgmap v5701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9666.336521 secs 2024-07-26T09:37:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9684.077587 secs 2024-07-26T09:37:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:16 smithi164 ceph-mon[36970]: pgmap v5701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9666.336521 secs 2024-07-26T09:37:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9684.077587 secs 2024-07-26T09:37:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:16 smithi163 ceph-mon[36873]: pgmap v5701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9666.336521 secs 2024-07-26T09:37:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9684.077587 secs 2024-07-26T09:37:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:18 smithi042 ceph-mon[32775]: pgmap v5702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:18 smithi164 ceph-mon[36970]: pgmap v5702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:19.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:18 smithi163 ceph-mon[36873]: pgmap v5702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:20 smithi042 ceph-mon[32775]: pgmap v5703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:20 smithi164 ceph-mon[36970]: pgmap v5703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:20 smithi163 ceph-mon[36873]: pgmap v5703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9671.336603 secs 2024-07-26T09:37:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9689.077694 secs 2024-07-26T09:37:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9671.336603 secs 2024-07-26T09:37:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9689.077694 secs 2024-07-26T09:37:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9671.336603 secs 2024-07-26T09:37:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9689.077694 secs 2024-07-26T09:37:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:22 smithi042 ceph-mon[32775]: pgmap v5704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:22 smithi164 ceph-mon[36970]: pgmap v5704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:22 smithi163 ceph-mon[36873]: pgmap v5704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:25.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:24 smithi042 ceph-mon[32775]: pgmap v5705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:24 smithi164 ceph-mon[36970]: pgmap v5705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:25.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:24 smithi163 ceph-mon[36873]: pgmap v5705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:37:26.861+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:37:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:37:26.861+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:37:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:37:26.861+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:37:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:26 smithi042 ceph-mon[32775]: pgmap v5706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9676.336690 secs 2024-07-26T09:37:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9694.077836 secs 2024-07-26T09:37:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:26 smithi164 ceph-mon[36970]: pgmap v5706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9676.336690 secs 2024-07-26T09:37:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9694.077836 secs 2024-07-26T09:37:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:26 smithi163 ceph-mon[36873]: pgmap v5706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9676.336690 secs 2024-07-26T09:37:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9694.077836 secs 2024-07-26T09:37:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:28 smithi042 ceph-mon[32775]: pgmap v5707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:28 smithi164 ceph-mon[36970]: pgmap v5707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:28 smithi163 ceph-mon[36873]: pgmap v5707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:30.097 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:29 smithi042 ceph-mon[32775]: pgmap v5708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:30.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:29 smithi164 ceph-mon[36970]: pgmap v5708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:29 smithi163 ceph-mon[36873]: pgmap v5708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9681.336795 secs 2024-07-26T09:37:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:32 smithi042 ceph-mon[32775]: pgmap v5709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9699.077959 secs 2024-07-26T09:37:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9681.336795 secs 2024-07-26T09:37:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:32 smithi164 ceph-mon[36970]: pgmap v5709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9699.077959 secs 2024-07-26T09:37:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9681.336795 secs 2024-07-26T09:37:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:32 smithi163 ceph-mon[36873]: pgmap v5709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9699.077959 secs 2024-07-26T09:37:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:34 smithi042 ceph-mon[32775]: pgmap v5710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:34 smithi164 ceph-mon[36970]: pgmap v5710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:34 smithi163 ceph-mon[36873]: pgmap v5710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:36 smithi042 ceph-mon[32775]: pgmap v5711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:36 smithi164 ceph-mon[36970]: pgmap v5711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:36 smithi163 ceph-mon[36873]: pgmap v5711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9686.336944 secs 2024-07-26T09:37:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9704.078082 secs 2024-07-26T09:37:37.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9686.336944 secs 2024-07-26T09:37:37.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9704.078082 secs 2024-07-26T09:37:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9686.336944 secs 2024-07-26T09:37:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9704.078082 secs 2024-07-26T09:37:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:38 smithi042 ceph-mon[32775]: pgmap v5712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:38 smithi164 ceph-mon[36970]: pgmap v5712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:38 smithi163 ceph-mon[36873]: pgmap v5712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:40.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:40 smithi042 ceph-mon[32775]: pgmap v5713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:40 smithi164 ceph-mon[36970]: pgmap v5713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:40 smithi163 ceph-mon[36873]: pgmap v5713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:41.873 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:37:41.871+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:37:41.873 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:37:41.871+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:37:41.873 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:37:41.871+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:37:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9691.337104 secs 2024-07-26T09:37:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:42 smithi042 ceph-mon[32775]: pgmap v5714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9709.078186 secs 2024-07-26T09:37:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9691.337104 secs 2024-07-26T09:37:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:42 smithi164 ceph-mon[36970]: pgmap v5714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9709.078186 secs 2024-07-26T09:37:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9691.337104 secs 2024-07-26T09:37:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:42 smithi163 ceph-mon[36873]: pgmap v5714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9709.078186 secs 2024-07-26T09:37:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:44 smithi042 ceph-mon[32775]: pgmap v5715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:44 smithi164 ceph-mon[36970]: pgmap v5715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:44 smithi163 ceph-mon[36873]: pgmap v5715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:46 smithi042 ceph-mon[32775]: pgmap v5716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:46 smithi164 ceph-mon[36970]: pgmap v5716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:46 smithi163 ceph-mon[36873]: pgmap v5716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9696.337260 secs 2024-07-26T09:37:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9714.078315 secs 2024-07-26T09:37:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9696.337260 secs 2024-07-26T09:37:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9714.078315 secs 2024-07-26T09:37:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9696.337260 secs 2024-07-26T09:37:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9714.078315 secs 2024-07-26T09:37:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:48 smithi042 ceph-mon[32775]: pgmap v5717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:48 smithi164 ceph-mon[36970]: pgmap v5717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:48 smithi163 ceph-mon[36873]: pgmap v5717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:50 smithi042 ceph-mon[32775]: pgmap v5718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:50 smithi164 ceph-mon[36970]: pgmap v5718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:50 smithi163 ceph-mon[36873]: pgmap v5718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9701.337400 secs 2024-07-26T09:37:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:52 smithi164 ceph-mon[36970]: pgmap v5719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9719.078435 secs 2024-07-26T09:37:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9701.337400 secs 2024-07-26T09:37:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:52 smithi163 ceph-mon[36873]: pgmap v5719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9719.078435 secs 2024-07-26T09:37:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9701.337400 secs 2024-07-26T09:37:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:52 smithi042 ceph-mon[32775]: pgmap v5719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9719.078435 secs 2024-07-26T09:37:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:54 smithi164 ceph-mon[36970]: pgmap v5720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:54 smithi163 ceph-mon[36873]: pgmap v5720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:54 smithi042 ceph-mon[32775]: pgmap v5720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:37:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:37:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:37:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:37:56.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:56 smithi164 ceph-mon[36970]: pgmap v5721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:56.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:37:56.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:37:56.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:56 smithi164 ceph-mon[36970]: pgmap v5722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:37:56.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:56 smithi164 ceph-mon[36970]: pgmap v5723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:37:56.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:37:56.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9706.337528 secs 2024-07-26T09:37:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:56 smithi163 ceph-mon[36873]: pgmap v5721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:37:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:37:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:56 smithi163 ceph-mon[36873]: pgmap v5722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:37:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:56 smithi163 ceph-mon[36873]: pgmap v5723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:37:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:37:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9706.337528 secs 2024-07-26T09:37:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:56 smithi042 ceph-mon[32775]: pgmap v5721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:37:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:37:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:37:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:56 smithi042 ceph-mon[32775]: pgmap v5722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:37:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:56 smithi042 ceph-mon[32775]: pgmap v5723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:37:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:37:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9706.337528 secs 2024-07-26T09:37:56.878 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:37:56.876+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:37:56.878 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:37:56.876+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:37:56.878 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:37:56.876+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:37:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9724.078549 secs 2024-07-26T09:37:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9724.078549 secs 2024-07-26T09:37:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9724.078549 secs 2024-07-26T09:37:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:37:58 smithi164 ceph-mon[36970]: pgmap v5724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:37:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:37:58 smithi163 ceph-mon[36873]: pgmap v5724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:37:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:37:58 smithi042 ceph-mon[32775]: pgmap v5724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:37:59.374 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:37:59.372+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T09:37:59.374 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:37:59.372+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:37:59.374 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:37:59.372+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T09:37:59.374 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:37:59.372+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:38:00.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:00 smithi164 ceph-mon[36970]: pgmap v5725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:38:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:00 smithi163 ceph-mon[36873]: pgmap v5725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:38:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:00 smithi042 ceph-mon[32775]: pgmap v5725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:38:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9711.337662 secs 2024-07-26T09:38:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9729.078678 secs 2024-07-26T09:38:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9711.337662 secs 2024-07-26T09:38:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9729.078678 secs 2024-07-26T09:38:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9711.337662 secs 2024-07-26T09:38:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9729.078678 secs 2024-07-26T09:38:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:02 smithi164 ceph-mon[36970]: pgmap v5726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:38:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:02 smithi163 ceph-mon[36873]: pgmap v5726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:38:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:02 smithi042 ceph-mon[32775]: pgmap v5726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:38:04.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:04 smithi042 ceph-mon[32775]: pgmap v5727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:38:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:04 smithi164 ceph-mon[36970]: pgmap v5727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:38:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:04 smithi163 ceph-mon[36873]: pgmap v5727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:38:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:06 smithi164 ceph-mon[36970]: pgmap v5728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:38:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9716.337797 secs 2024-07-26T09:38:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9734.078799 secs 2024-07-26T09:38:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:06 smithi163 ceph-mon[36873]: pgmap v5728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:38:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9716.337797 secs 2024-07-26T09:38:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9734.078799 secs 2024-07-26T09:38:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:06 smithi042 ceph-mon[32775]: pgmap v5728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:38:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9716.337797 secs 2024-07-26T09:38:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9734.078799 secs 2024-07-26T09:38:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:08 smithi164 ceph-mon[36970]: pgmap v5729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:08 smithi163 ceph-mon[36873]: pgmap v5729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:08 smithi042 ceph-mon[32775]: pgmap v5729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:10 smithi164 ceph-mon[36970]: pgmap v5730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:10 smithi163 ceph-mon[36873]: pgmap v5730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:10 smithi042 ceph-mon[32775]: pgmap v5730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:11.887 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:38:11.885+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:38:11.887 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:38:11.885+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:38:11.887 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:38:11.885+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:38:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9721.337925 secs 2024-07-26T09:38:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9739.078922 secs 2024-07-26T09:38:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9721.337925 secs 2024-07-26T09:38:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9739.078922 secs 2024-07-26T09:38:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9721.337925 secs 2024-07-26T09:38:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9739.078922 secs 2024-07-26T09:38:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:12 smithi164 ceph-mon[36970]: pgmap v5731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:12 smithi163 ceph-mon[36873]: pgmap v5731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:12 smithi042 ceph-mon[32775]: pgmap v5731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:14 smithi164 ceph-mon[36970]: pgmap v5732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:38:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:14 smithi163 ceph-mon[36873]: pgmap v5732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:38:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:14 smithi042 ceph-mon[32775]: pgmap v5732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:38:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:16 smithi164 ceph-mon[36970]: pgmap v5733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9726.338071 secs 2024-07-26T09:38:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9744.079041 secs 2024-07-26T09:38:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:16 smithi163 ceph-mon[36873]: pgmap v5733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9726.338071 secs 2024-07-26T09:38:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9744.079041 secs 2024-07-26T09:38:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:16 smithi042 ceph-mon[32775]: pgmap v5733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9726.338071 secs 2024-07-26T09:38:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9744.079041 secs 2024-07-26T09:38:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:18 smithi164 ceph-mon[36970]: pgmap v5734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:18 smithi163 ceph-mon[36873]: pgmap v5734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:18 smithi042 ceph-mon[32775]: pgmap v5734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:20 smithi164 ceph-mon[36970]: pgmap v5735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:20 smithi163 ceph-mon[36873]: pgmap v5735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:20 smithi042 ceph-mon[32775]: pgmap v5735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9731.338246 secs 2024-07-26T09:38:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9749.079132 secs 2024-07-26T09:38:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9731.338246 secs 2024-07-26T09:38:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9749.079132 secs 2024-07-26T09:38:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9731.338246 secs 2024-07-26T09:38:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9749.079132 secs 2024-07-26T09:38:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:22 smithi164 ceph-mon[36970]: pgmap v5736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:22 smithi163 ceph-mon[36873]: pgmap v5736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:22 smithi042 ceph-mon[32775]: pgmap v5736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:24 smithi164 ceph-mon[36970]: pgmap v5737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:38:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:24 smithi163 ceph-mon[36873]: pgmap v5737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:38:25.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:24 smithi042 ceph-mon[32775]: pgmap v5737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:38:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:38:26.890+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:38:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:38:26.890+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:38:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:38:26.890+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:38:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:26 smithi164 ceph-mon[36970]: pgmap v5738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9736.338371 secs 2024-07-26T09:38:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9754.079242 secs 2024-07-26T09:38:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:26 smithi163 ceph-mon[36873]: pgmap v5738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9736.338371 secs 2024-07-26T09:38:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9754.079242 secs 2024-07-26T09:38:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:26 smithi042 ceph-mon[32775]: pgmap v5738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9736.338371 secs 2024-07-26T09:38:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9754.079242 secs 2024-07-26T09:38:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:28 smithi164 ceph-mon[36970]: pgmap v5739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:28 smithi163 ceph-mon[36873]: pgmap v5739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:28 smithi042 ceph-mon[32775]: pgmap v5739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:30 smithi164 ceph-mon[36970]: pgmap v5740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:30 smithi163 ceph-mon[36873]: pgmap v5740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:30 smithi042 ceph-mon[32775]: pgmap v5740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9741.338505 secs 2024-07-26T09:38:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9759.079343 secs 2024-07-26T09:38:32.049 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9741.338505 secs 2024-07-26T09:38:32.049 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9759.079343 secs 2024-07-26T09:38:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9741.338505 secs 2024-07-26T09:38:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9759.079343 secs 2024-07-26T09:38:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:32 smithi163 ceph-mon[36873]: pgmap v5741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:32 smithi042 ceph-mon[32775]: pgmap v5741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:32 smithi164 ceph-mon[36970]: pgmap v5741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:34 smithi163 ceph-mon[36873]: pgmap v5742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:38:35.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:34 smithi042 ceph-mon[32775]: pgmap v5742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:38:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:34 smithi164 ceph-mon[36970]: pgmap v5742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:38:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:36 smithi163 ceph-mon[36873]: pgmap v5743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9746.338639 secs 2024-07-26T09:38:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9764.079409 secs 2024-07-26T09:38:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:36 smithi042 ceph-mon[32775]: pgmap v5743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9746.338639 secs 2024-07-26T09:38:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9764.079409 secs 2024-07-26T09:38:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:36 smithi164 ceph-mon[36970]: pgmap v5743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9746.338639 secs 2024-07-26T09:38:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9764.079409 secs 2024-07-26T09:38:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:37 smithi163 ceph-mon[36873]: pgmap v5744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:37 smithi042 ceph-mon[32775]: pgmap v5744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:37 smithi164 ceph-mon[36970]: pgmap v5744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:40 smithi164 ceph-mon[36970]: pgmap v5745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:40 smithi163 ceph-mon[36873]: pgmap v5745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:41.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:40 smithi042 ceph-mon[32775]: pgmap v5745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:38:41.899+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:38:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:38:41.899+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:38:41.901 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:38:41.899+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:38:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9751.338776 secs 2024-07-26T09:38:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9769.079531 secs 2024-07-26T09:38:42.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9751.338776 secs 2024-07-26T09:38:42.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9769.079531 secs 2024-07-26T09:38:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9751.338776 secs 2024-07-26T09:38:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9769.079531 secs 2024-07-26T09:38:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:42 smithi164 ceph-mon[36970]: pgmap v5746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:42 smithi163 ceph-mon[36873]: pgmap v5746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:42 smithi042 ceph-mon[32775]: pgmap v5746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:44 smithi164 ceph-mon[36970]: pgmap v5747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:38:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:44 smithi163 ceph-mon[36873]: pgmap v5747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:38:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:44 smithi042 ceph-mon[32775]: pgmap v5747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:38:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:46 smithi164 ceph-mon[36970]: pgmap v5748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9756.338916 secs 2024-07-26T09:38:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9774.079656 secs 2024-07-26T09:38:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:46 smithi163 ceph-mon[36873]: pgmap v5748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9756.338916 secs 2024-07-26T09:38:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9774.079656 secs 2024-07-26T09:38:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:46 smithi042 ceph-mon[32775]: pgmap v5748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9756.338916 secs 2024-07-26T09:38:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9774.079656 secs 2024-07-26T09:38:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:48 smithi164 ceph-mon[36970]: pgmap v5749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:48 smithi163 ceph-mon[36873]: pgmap v5749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:48 smithi042 ceph-mon[32775]: pgmap v5749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:50 smithi164 ceph-mon[36970]: pgmap v5750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:51.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:50 smithi163 ceph-mon[36873]: pgmap v5750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:50 smithi042 ceph-mon[32775]: pgmap v5750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9761.339051 secs 2024-07-26T09:38:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9779.079779 secs 2024-07-26T09:38:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9761.339051 secs 2024-07-26T09:38:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9779.079779 secs 2024-07-26T09:38:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9761.339051 secs 2024-07-26T09:38:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9779.079779 secs 2024-07-26T09:38:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:52 smithi164 ceph-mon[36970]: pgmap v5751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:52 smithi163 ceph-mon[36873]: pgmap v5751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:52 smithi042 ceph-mon[32775]: pgmap v5751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:38:54.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:54 smithi164 ceph-mon[36970]: pgmap v5752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:38:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:54 smithi163 ceph-mon[36873]: pgmap v5752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:38:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:54 smithi042 ceph-mon[32775]: pgmap v5752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:38:56.904 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:38:56.902+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:38:56.904 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:38:56.902+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:38:56.904 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:38:56.902+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:38:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: pgmap v5753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:38:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9766.339147 secs 2024-07-26T09:38:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:56.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:56.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:56.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:56.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:56.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:38:56.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:38:56.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:38:56.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9784.079869 secs 2024-07-26T09:38:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: pgmap v5753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:38:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9766.339147 secs 2024-07-26T09:38:57.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:38:57.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:38:57.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:38:57.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9784.079869 secs 2024-07-26T09:38:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: pgmap v5753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:38:57.159 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:38:57.159 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9766.339147 secs 2024-07-26T09:38:57.159 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.159 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.159 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.159 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.160 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.160 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.160 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.160 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.160 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.160 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.160 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.160 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:38:57.160 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:38:57.160 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:38:57.160 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:38:57.160 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9784.079869 secs 2024-07-26T09:38:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:57 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:38:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:57 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:38:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:57 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:38:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:57 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:38:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:57 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:38:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:57 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:38:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:57 smithi164 ceph-mon[36970]: pgmap v5754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:38:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:57 smithi164 ceph-mon[36970]: pgmap v5755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:38:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:57 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:38:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:57 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:38:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:57 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:38:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:57 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:38:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:57 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:38:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:57 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:38:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:57 smithi163 ceph-mon[36873]: pgmap v5754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:38:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:57 smithi163 ceph-mon[36873]: pgmap v5755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:38:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:57 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:38:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:57 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:38:58.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:57 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:38:58.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:57 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:38:58.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:57 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:38:58.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:57 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:38:58.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:57 smithi042 ceph-mon[32775]: pgmap v5754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:38:58.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:57 smithi042 ceph-mon[32775]: pgmap v5755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:38:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:38:59 smithi164 ceph-mon[36970]: pgmap v5756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:39:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:38:59 smithi163 ceph-mon[36873]: pgmap v5756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:39:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:38:59 smithi042 ceph-mon[32775]: pgmap v5756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:39:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:01 smithi164 ceph-mon[36970]: pgmap v5757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:39:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9771.339280 secs 2024-07-26T09:39:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9789.080010 secs 2024-07-26T09:39:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:01 smithi163 ceph-mon[36873]: pgmap v5757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:39:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9771.339280 secs 2024-07-26T09:39:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9789.080010 secs 2024-07-26T09:39:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:01 smithi042 ceph-mon[32775]: pgmap v5757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:39:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9771.339280 secs 2024-07-26T09:39:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9789.080010 secs 2024-07-26T09:39:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:03 smithi164 ceph-mon[36970]: pgmap v5758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:39:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:03 smithi163 ceph-mon[36873]: pgmap v5758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:39:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:03 smithi042 ceph-mon[32775]: pgmap v5758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:39:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:05 smithi164 ceph-mon[36970]: pgmap v5759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:39:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:05 smithi163 ceph-mon[36873]: pgmap v5759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:39:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:05 smithi042 ceph-mon[32775]: pgmap v5759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:39:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9776.339417 secs 2024-07-26T09:39:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9794.080134 secs 2024-07-26T09:39:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9776.339417 secs 2024-07-26T09:39:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9794.080134 secs 2024-07-26T09:39:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9776.339417 secs 2024-07-26T09:39:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9794.080134 secs 2024-07-26T09:39:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:07 smithi163 ceph-mon[36873]: pgmap v5760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:39:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:07 smithi042 ceph-mon[32775]: pgmap v5760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:39:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:07 smithi164 ceph-mon[36970]: pgmap v5760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:39:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:09 smithi163 ceph-mon[36873]: pgmap v5761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:09 smithi042 ceph-mon[32775]: pgmap v5761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:09 smithi164 ceph-mon[36970]: pgmap v5761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:39:11.912+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:39:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:39:11.912+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:39:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:39:11.912+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:39:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:11 smithi163 ceph-mon[36873]: pgmap v5762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9781.339557 secs 2024-07-26T09:39:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9799.080220 secs 2024-07-26T09:39:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:11 smithi042 ceph-mon[32775]: pgmap v5762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9781.339557 secs 2024-07-26T09:39:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9799.080220 secs 2024-07-26T09:39:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:11 smithi164 ceph-mon[36970]: pgmap v5762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9781.339557 secs 2024-07-26T09:39:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9799.080220 secs 2024-07-26T09:39:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:13 smithi163 ceph-mon[36873]: pgmap v5763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:13 smithi042 ceph-mon[32775]: pgmap v5763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:13 smithi164 ceph-mon[36970]: pgmap v5763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:15 smithi163 ceph-mon[36873]: pgmap v5764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:15 smithi042 ceph-mon[32775]: pgmap v5764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:15 smithi164 ceph-mon[36970]: pgmap v5764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9786.339691 secs 2024-07-26T09:39:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9804.080283 secs 2024-07-26T09:39:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9786.339691 secs 2024-07-26T09:39:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9804.080283 secs 2024-07-26T09:39:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9786.339691 secs 2024-07-26T09:39:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9804.080283 secs 2024-07-26T09:39:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:17 smithi163 ceph-mon[36873]: pgmap v5765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:18.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:17 smithi164 ceph-mon[36970]: pgmap v5765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:17 smithi042 ceph-mon[32775]: pgmap v5765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:19 smithi163 ceph-mon[36873]: pgmap v5766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:19 smithi042 ceph-mon[32775]: pgmap v5766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:19 smithi164 ceph-mon[36970]: pgmap v5766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:21 smithi163 ceph-mon[36873]: pgmap v5767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:22.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9791.339823 secs 2024-07-26T09:39:22.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9809.080408 secs 2024-07-26T09:39:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:21 smithi042 ceph-mon[32775]: pgmap v5767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9791.339823 secs 2024-07-26T09:39:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9809.080408 secs 2024-07-26T09:39:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:21 smithi164 ceph-mon[36970]: pgmap v5767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9791.339823 secs 2024-07-26T09:39:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9809.080408 secs 2024-07-26T09:39:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:23 smithi042 ceph-mon[32775]: pgmap v5768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:23 smithi164 ceph-mon[36970]: pgmap v5768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:23 smithi163 ceph-mon[36873]: pgmap v5768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:25 smithi042 ceph-mon[32775]: pgmap v5769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:25 smithi164 ceph-mon[36970]: pgmap v5769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:25 smithi163 ceph-mon[36873]: pgmap v5769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:26.919 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:39:26.917+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:39:26.919 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:39:26.917+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:39:26.919 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:39:26.917+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:39:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9796.339890 secs 2024-07-26T09:39:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9814.080546 secs 2024-07-26T09:39:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9796.339890 secs 2024-07-26T09:39:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9814.080546 secs 2024-07-26T09:39:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9796.339890 secs 2024-07-26T09:39:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9814.080546 secs 2024-07-26T09:39:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:27 smithi042 ceph-mon[32775]: pgmap v5770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:27 smithi164 ceph-mon[36970]: pgmap v5770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:27 smithi163 ceph-mon[36873]: pgmap v5770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:29 smithi042 ceph-mon[32775]: pgmap v5771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:30.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:29 smithi164 ceph-mon[36970]: pgmap v5771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:29 smithi163 ceph-mon[36873]: pgmap v5771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:31 smithi042 ceph-mon[32775]: pgmap v5772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9801.340030 secs 2024-07-26T09:39:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9819.080691 secs 2024-07-26T09:39:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:31 smithi164 ceph-mon[36970]: pgmap v5772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9801.340030 secs 2024-07-26T09:39:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9819.080691 secs 2024-07-26T09:39:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:31 smithi163 ceph-mon[36873]: pgmap v5772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9801.340030 secs 2024-07-26T09:39:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9819.080691 secs 2024-07-26T09:39:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:33 smithi042 ceph-mon[32775]: pgmap v5773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:33 smithi164 ceph-mon[36970]: pgmap v5773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:33 smithi163 ceph-mon[36873]: pgmap v5773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:35 smithi042 ceph-mon[32775]: pgmap v5774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:35 smithi164 ceph-mon[36970]: pgmap v5774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:35 smithi163 ceph-mon[36873]: pgmap v5774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9806.340161 secs 2024-07-26T09:39:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9824.080762 secs 2024-07-26T09:39:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9806.340161 secs 2024-07-26T09:39:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9824.080762 secs 2024-07-26T09:39:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9806.340161 secs 2024-07-26T09:39:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9824.080762 secs 2024-07-26T09:39:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:37 smithi042 ceph-mon[32775]: pgmap v5775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:37 smithi164 ceph-mon[36970]: pgmap v5775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:37 smithi163 ceph-mon[36873]: pgmap v5775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:40.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:39 smithi163 ceph-mon[36873]: pgmap v5776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:39 smithi042 ceph-mon[32775]: pgmap v5776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:39 smithi164 ceph-mon[36970]: pgmap v5776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:41.929 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:39:41.927+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:39:41.929 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:39:41.927+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:39:41.929 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:39:41.927+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:39:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:41 smithi042 ceph-mon[32775]: pgmap v5777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9811.340290 secs 2024-07-26T09:39:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9829.080882 secs 2024-07-26T09:39:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:41 smithi164 ceph-mon[36970]: pgmap v5777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9811.340290 secs 2024-07-26T09:39:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9829.080882 secs 2024-07-26T09:39:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:41 smithi163 ceph-mon[36873]: pgmap v5777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9811.340290 secs 2024-07-26T09:39:42.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9829.080882 secs 2024-07-26T09:39:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:43 smithi042 ceph-mon[32775]: pgmap v5778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:43 smithi164 ceph-mon[36970]: pgmap v5778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:43 smithi163 ceph-mon[36873]: pgmap v5778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:45 smithi042 ceph-mon[32775]: pgmap v5779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:45 smithi164 ceph-mon[36970]: pgmap v5779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:45 smithi163 ceph-mon[36873]: pgmap v5779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9816.340416 secs 2024-07-26T09:39:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9834.080954 secs 2024-07-26T09:39:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9816.340416 secs 2024-07-26T09:39:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9834.080954 secs 2024-07-26T09:39:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9816.340416 secs 2024-07-26T09:39:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9834.080954 secs 2024-07-26T09:39:48.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:47 smithi164 ceph-mon[36970]: pgmap v5780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:47 smithi042 ceph-mon[32775]: pgmap v5780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:47 smithi163 ceph-mon[36873]: pgmap v5780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:49 smithi042 ceph-mon[32775]: pgmap v5781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:49 smithi164 ceph-mon[36970]: pgmap v5781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:49 smithi163 ceph-mon[36873]: pgmap v5781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:51 smithi042 ceph-mon[32775]: pgmap v5782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9821.340537 secs 2024-07-26T09:39:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9839.081019 secs 2024-07-26T09:39:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:51 smithi164 ceph-mon[36970]: pgmap v5782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9821.340537 secs 2024-07-26T09:39:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9839.081019 secs 2024-07-26T09:39:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:51 smithi163 ceph-mon[36873]: pgmap v5782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9821.340537 secs 2024-07-26T09:39:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9839.081019 secs 2024-07-26T09:39:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:53 smithi042 ceph-mon[32775]: pgmap v5783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:53 smithi164 ceph-mon[36970]: pgmap v5783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:53 smithi163 ceph-mon[36873]: pgmap v5783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:55 smithi042 ceph-mon[32775]: pgmap v5784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:55 smithi164 ceph-mon[36970]: pgmap v5784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:55 smithi163 ceph-mon[36873]: pgmap v5784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:39:56.934 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:39:56.932+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:39:56.934 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:39:56.932+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:39:56.934 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:39:56.932+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:39:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9826.340621 secs 2024-07-26T09:39:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9844.081097 secs 2024-07-26T09:39:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:39:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:39:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:39:57.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:39:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9826.340621 secs 2024-07-26T09:39:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9844.081097 secs 2024-07-26T09:39:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:39:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:39:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:39:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:39:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9826.340621 secs 2024-07-26T09:39:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9844.081097 secs 2024-07-26T09:39:57.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:39:57.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:39:57.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:39:57.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:39:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:57 smithi042 ceph-mon[32775]: pgmap v5785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:57 smithi042 ceph-mon[32775]: pgmap v5786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:39:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:57 smithi042 ceph-mon[32775]: pgmap v5787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:39:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:57 smithi164 ceph-mon[36970]: pgmap v5785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:57 smithi164 ceph-mon[36970]: pgmap v5786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:39:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:57 smithi164 ceph-mon[36970]: pgmap v5787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:39:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:57 smithi163 ceph-mon[36873]: pgmap v5785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:39:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:57 smithi163 ceph-mon[36873]: pgmap v5786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:39:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:57 smithi163 ceph-mon[36873]: pgmap v5787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:40:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:39:59 smithi042 ceph-mon[32775]: pgmap v5788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:40:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:39:59 smithi164 ceph-mon[36970]: pgmap v5788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:40:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:39:59 smithi163 ceph-mon[36873]: pgmap v5788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:40:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:40:01.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:40:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:40:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:01 smithi042 ceph-mon[32775]: pgmap v5789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:40:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9831.340706 secs 2024-07-26T09:40:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9849.081162 secs 2024-07-26T09:40:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:01 smithi164 ceph-mon[36970]: pgmap v5789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:40:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9831.340706 secs 2024-07-26T09:40:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9849.081162 secs 2024-07-26T09:40:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:01 smithi163 ceph-mon[36873]: pgmap v5789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:40:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9831.340706 secs 2024-07-26T09:40:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9849.081162 secs 2024-07-26T09:40:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:02 smithi042 ceph-mon[32775]: pgmap v5790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:40:03.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:02 smithi164 ceph-mon[36970]: pgmap v5790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:40:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:02 smithi163 ceph-mon[36873]: pgmap v5790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:40:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:05 smithi042 ceph-mon[32775]: pgmap v5791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:40:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:05 smithi164 ceph-mon[36970]: pgmap v5791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:40:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:05 smithi163 ceph-mon[36873]: pgmap v5791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:40:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9836.340792 secs 2024-07-26T09:40:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9854.081229 secs 2024-07-26T09:40:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9836.340792 secs 2024-07-26T09:40:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9854.081229 secs 2024-07-26T09:40:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9836.340792 secs 2024-07-26T09:40:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9854.081229 secs 2024-07-26T09:40:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:07 smithi042 ceph-mon[32775]: pgmap v5792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:40:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:07 smithi164 ceph-mon[36970]: pgmap v5792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:40:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:07 smithi163 ceph-mon[36873]: pgmap v5792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:40:10.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:09 smithi163 ceph-mon[36873]: pgmap v5793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:40:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:09 smithi042 ceph-mon[32775]: pgmap v5793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:40:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:09 smithi164 ceph-mon[36970]: pgmap v5793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:40:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:11.941+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:40:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:11.941+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:40:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:11.941+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:40:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:11 smithi042 ceph-mon[32775]: pgmap v5794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9841.340913 secs 2024-07-26T09:40:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9859.081349 secs 2024-07-26T09:40:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:11 smithi164 ceph-mon[36970]: pgmap v5794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9841.340913 secs 2024-07-26T09:40:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9859.081349 secs 2024-07-26T09:40:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:11 smithi163 ceph-mon[36873]: pgmap v5794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9841.340913 secs 2024-07-26T09:40:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9859.081349 secs 2024-07-26T09:40:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:13 smithi042 ceph-mon[32775]: pgmap v5795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:13 smithi164 ceph-mon[36970]: pgmap v5795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:13 smithi163 ceph-mon[36873]: pgmap v5795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:15 smithi042 ceph-mon[32775]: pgmap v5796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:15 smithi164 ceph-mon[36970]: pgmap v5796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:15 smithi163 ceph-mon[36873]: pgmap v5796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9846.341051 secs 2024-07-26T09:40:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9864.081434 secs 2024-07-26T09:40:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9846.341051 secs 2024-07-26T09:40:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9864.081434 secs 2024-07-26T09:40:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9846.341051 secs 2024-07-26T09:40:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9864.081434 secs 2024-07-26T09:40:18.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:17 smithi164 ceph-mon[36970]: pgmap v5797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:17 smithi042 ceph-mon[32775]: pgmap v5797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:17 smithi163 ceph-mon[36873]: pgmap v5797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:19 smithi042 ceph-mon[32775]: pgmap v5798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:40:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:19 smithi164 ceph-mon[36970]: pgmap v5798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:40:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:19 smithi163 ceph-mon[36873]: pgmap v5798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:40:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:21 smithi042 ceph-mon[32775]: pgmap v5799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9851.341195 secs 2024-07-26T09:40:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9869.081520 secs 2024-07-26T09:40:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:21 smithi164 ceph-mon[36970]: pgmap v5799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9851.341195 secs 2024-07-26T09:40:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9869.081520 secs 2024-07-26T09:40:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:21 smithi163 ceph-mon[36873]: pgmap v5799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9851.341195 secs 2024-07-26T09:40:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9869.081520 secs 2024-07-26T09:40:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:23 smithi042 ceph-mon[32775]: pgmap v5800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:24.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:23 smithi164 ceph-mon[36970]: pgmap v5800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:24.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:23 smithi163 ceph-mon[36873]: pgmap v5800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:25 smithi042 ceph-mon[32775]: pgmap v5801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:25 smithi164 ceph-mon[36970]: pgmap v5801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:25 smithi163 ceph-mon[36873]: pgmap v5801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:26.944 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:26.941+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:40:26.944 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:26.941+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:40:26.944 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:26.941+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:40:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9856.341329 secs 2024-07-26T09:40:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9874.081580 secs 2024-07-26T09:40:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9856.341329 secs 2024-07-26T09:40:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9874.081580 secs 2024-07-26T09:40:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9856.341329 secs 2024-07-26T09:40:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9874.081580 secs 2024-07-26T09:40:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:27 smithi042 ceph-mon[32775]: pgmap v5802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:27 smithi164 ceph-mon[36970]: pgmap v5802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:27 smithi163 ceph-mon[36873]: pgmap v5802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:29 smithi042 ceph-mon[32775]: pgmap v5803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:40:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:29 smithi164 ceph-mon[36970]: pgmap v5803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:40:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:29 smithi163 ceph-mon[36873]: pgmap v5803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:40:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:31 smithi042 ceph-mon[32775]: pgmap v5804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9861.341452 secs 2024-07-26T09:40:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9879.081725 secs 2024-07-26T09:40:32.206 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:31 smithi163 ceph-mon[36873]: pgmap v5804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:32.206 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9861.341452 secs 2024-07-26T09:40:32.206 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9879.081725 secs 2024-07-26T09:40:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:31 smithi164 ceph-mon[36970]: pgmap v5804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9861.341452 secs 2024-07-26T09:40:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9879.081725 secs 2024-07-26T09:40:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:33 smithi042 ceph-mon[32775]: pgmap v5805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:33 smithi164 ceph-mon[36970]: pgmap v5805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:33 smithi163 ceph-mon[36873]: pgmap v5805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:35 smithi042 ceph-mon[32775]: pgmap v5806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:35 smithi164 ceph-mon[36970]: pgmap v5806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:35 smithi163 ceph-mon[36873]: pgmap v5806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9866.341577 secs 2024-07-26T09:40:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9884.081850 secs 2024-07-26T09:40:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9866.341577 secs 2024-07-26T09:40:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9884.081850 secs 2024-07-26T09:40:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9866.341577 secs 2024-07-26T09:40:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9884.081850 secs 2024-07-26T09:40:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:37 smithi042 ceph-mon[32775]: pgmap v5807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:38.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:37 smithi164 ceph-mon[36970]: pgmap v5807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:37 smithi163 ceph-mon[36873]: pgmap v5807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:39 smithi042 ceph-mon[32775]: pgmap v5808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:40:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:39 smithi164 ceph-mon[36970]: pgmap v5808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:40:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:39 smithi163 ceph-mon[36873]: pgmap v5808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:40:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:41.951+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:40:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:41.951+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:40:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:41.951+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:40:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:41 smithi042 ceph-mon[32775]: pgmap v5809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9871.341707 secs 2024-07-26T09:40:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9889.081971 secs 2024-07-26T09:40:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:41 smithi164 ceph-mon[36970]: pgmap v5809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9871.341707 secs 2024-07-26T09:40:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9889.081971 secs 2024-07-26T09:40:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:41 smithi163 ceph-mon[36873]: pgmap v5809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9871.341707 secs 2024-07-26T09:40:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9889.081971 secs 2024-07-26T09:40:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:43 smithi042 ceph-mon[32775]: pgmap v5810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:43 smithi164 ceph-mon[36970]: pgmap v5810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:43 smithi163 ceph-mon[36873]: pgmap v5810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:45 smithi042 ceph-mon[32775]: pgmap v5811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:45 smithi164 ceph-mon[36970]: pgmap v5811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:45 smithi163 ceph-mon[36873]: pgmap v5811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9876.341833 secs 2024-07-26T09:40:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9894.082048 secs 2024-07-26T09:40:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9876.341833 secs 2024-07-26T09:40:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9894.082048 secs 2024-07-26T09:40:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9876.341833 secs 2024-07-26T09:40:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9894.082048 secs 2024-07-26T09:40:48.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:47 smithi164 ceph-mon[36970]: pgmap v5812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:47 smithi042 ceph-mon[32775]: pgmap v5812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:47 smithi163 ceph-mon[36873]: pgmap v5812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:48 smithi042 ceph-mon[32775]: pgmap v5813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:40:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:48 smithi164 ceph-mon[36970]: pgmap v5813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:40:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:48 smithi163 ceph-mon[36873]: pgmap v5813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:40:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:51 smithi042 ceph-mon[32775]: pgmap v5814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9881.341967 secs 2024-07-26T09:40:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9899.082168 secs 2024-07-26T09:40:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:51 smithi164 ceph-mon[36970]: pgmap v5814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9881.341967 secs 2024-07-26T09:40:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9899.082168 secs 2024-07-26T09:40:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:51 smithi163 ceph-mon[36873]: pgmap v5814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9881.341967 secs 2024-07-26T09:40:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9899.082168 secs 2024-07-26T09:40:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:53 smithi042 ceph-mon[32775]: pgmap v5815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:53 smithi164 ceph-mon[36970]: pgmap v5815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:53 smithi163 ceph-mon[36873]: pgmap v5815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:55 smithi042 ceph-mon[32775]: pgmap v5816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:55 smithi164 ceph-mon[36970]: pgmap v5816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:55 smithi163 ceph-mon[36873]: pgmap v5816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:40:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:56.956+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:40:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:56.956+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:40:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:56.956+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:40:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9886.342082 secs 2024-07-26T09:40:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9904.082250 secs 2024-07-26T09:40:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9886.342082 secs 2024-07-26T09:40:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9904.082250 secs 2024-07-26T09:40:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9886.342082 secs 2024-07-26T09:40:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9904.082250 secs 2024-07-26T09:40:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:57.875+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000694) for more than 900000000 us, fault. 2024-07-26T09:40:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:57.875+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5000 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff67802b060 tx=0x7ff678055040 comp rx=0 tx=0).stop 2024-07-26T09:40:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:57.875+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T09:40:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:57.875+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5000 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T09:40:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:57.875+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T09:40:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:57.875+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:40:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:57.876+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff678003990 tx=0x7ff67804b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T09:40:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:57 smithi042 ceph-mon[32775]: pgmap v5817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:40:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:40:58.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:40:58.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:40:58.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:57 smithi164 ceph-mon[36970]: pgmap v5817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:58.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:40:58.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:40:58.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:40:58.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:40:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:57 smithi163 ceph-mon[36873]: pgmap v5817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:40:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:40:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:40:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:40:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:40:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:58 smithi042 ceph-mon[32775]: pgmap v5818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:40:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:40:58 smithi042 ceph-mon[32775]: pgmap v5819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:40:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:58 smithi164 ceph-mon[36970]: pgmap v5818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:40:59.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:40:58 smithi164 ceph-mon[36970]: pgmap v5819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:40:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:58 smithi163 ceph-mon[36873]: pgmap v5818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:40:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:40:58 smithi163 ceph-mon[36873]: pgmap v5819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:40:59.404 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:59.401+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:40:59.404 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:59.401+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T09:40:59.404 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:59.402+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T09:40:59.404 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:59.402+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:40:59.405 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:59.402+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T09:40:59.405 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:59.402+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:40:59.405 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:59.402+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:40:59.405 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:40:59.402+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:41:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:00 smithi042 ceph-mon[32775]: pgmap v5820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:41:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:00 smithi164 ceph-mon[36970]: pgmap v5820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:41:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:00 smithi163 ceph-mon[36873]: pgmap v5820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:41:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9891.342252 secs 2024-07-26T09:41:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9909.082333 secs 2024-07-26T09:41:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9891.342252 secs 2024-07-26T09:41:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9909.082333 secs 2024-07-26T09:41:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9891.342252 secs 2024-07-26T09:41:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9909.082333 secs 2024-07-26T09:41:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:02 smithi042 ceph-mon[32775]: pgmap v5821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:41:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:02 smithi164 ceph-mon[36970]: pgmap v5821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:41:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:02 smithi163 ceph-mon[36873]: pgmap v5821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:41:05.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:04 smithi042 ceph-mon[32775]: pgmap v5822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:41:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:04 smithi164 ceph-mon[36970]: pgmap v5822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:41:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:04 smithi163 ceph-mon[36873]: pgmap v5822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:41:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:06 smithi042 ceph-mon[32775]: pgmap v5823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:41:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9896.342318 secs 2024-07-26T09:41:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9914.082457 secs 2024-07-26T09:41:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:06 smithi164 ceph-mon[36970]: pgmap v5823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:41:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9896.342318 secs 2024-07-26T09:41:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9914.082457 secs 2024-07-26T09:41:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:06 smithi163 ceph-mon[36873]: pgmap v5823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:41:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9896.342318 secs 2024-07-26T09:41:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9914.082457 secs 2024-07-26T09:41:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:08 smithi042 ceph-mon[32775]: pgmap v5824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:41:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:08 smithi164 ceph-mon[36970]: pgmap v5824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:41:09.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:08 smithi163 ceph-mon[36873]: pgmap v5824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:41:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:10 smithi042 ceph-mon[32775]: pgmap v5825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:10 smithi164 ceph-mon[36970]: pgmap v5825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:10 smithi163 ceph-mon[36873]: pgmap v5825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:41:11.966+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:41:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:41:11.966+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:41:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:41:11.966+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:41:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9901.342441 secs 2024-07-26T09:41:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9919.082596 secs 2024-07-26T09:41:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9901.342441 secs 2024-07-26T09:41:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9919.082596 secs 2024-07-26T09:41:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9901.342441 secs 2024-07-26T09:41:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9919.082596 secs 2024-07-26T09:41:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:12 smithi042 ceph-mon[32775]: pgmap v5826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:12 smithi164 ceph-mon[36970]: pgmap v5826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:12 smithi163 ceph-mon[36873]: pgmap v5826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:14 smithi042 ceph-mon[32775]: pgmap v5827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:41:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:14 smithi164 ceph-mon[36970]: pgmap v5827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:41:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:14 smithi163 ceph-mon[36873]: pgmap v5827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:41:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:16 smithi042 ceph-mon[32775]: pgmap v5828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9906.342538 secs 2024-07-26T09:41:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9924.082745 secs 2024-07-26T09:41:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:16 smithi164 ceph-mon[36970]: pgmap v5828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9906.342538 secs 2024-07-26T09:41:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9924.082745 secs 2024-07-26T09:41:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:16 smithi163 ceph-mon[36873]: pgmap v5828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9906.342538 secs 2024-07-26T09:41:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9924.082745 secs 2024-07-26T09:41:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:18 smithi042 ceph-mon[32775]: pgmap v5829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:18 smithi164 ceph-mon[36970]: pgmap v5829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:19.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:18 smithi163 ceph-mon[36873]: pgmap v5829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:20 smithi042 ceph-mon[32775]: pgmap v5830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:20 smithi164 ceph-mon[36970]: pgmap v5830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:20 smithi163 ceph-mon[36873]: pgmap v5830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9911.342625 secs 2024-07-26T09:41:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9929.082858 secs 2024-07-26T09:41:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9911.342625 secs 2024-07-26T09:41:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9929.082858 secs 2024-07-26T09:41:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9911.342625 secs 2024-07-26T09:41:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9929.082858 secs 2024-07-26T09:41:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:22 smithi042 ceph-mon[32775]: pgmap v5831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:22 smithi164 ceph-mon[36970]: pgmap v5831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:22 smithi163 ceph-mon[36873]: pgmap v5831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:25.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:24 smithi042 ceph-mon[32775]: pgmap v5832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:41:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:24 smithi164 ceph-mon[36970]: pgmap v5832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:41:25.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:24 smithi163 ceph-mon[36873]: pgmap v5832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:41:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:25 smithi164 ceph-mon[36970]: pgmap v5833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:25 smithi163 ceph-mon[36873]: pgmap v5833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:25 smithi042 ceph-mon[32775]: pgmap v5833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:26.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:41:26.968+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:41:26.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:41:26.968+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:41:26.970 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:41:26.968+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:41:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9916.342711 secs 2024-07-26T09:41:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9934.083011 secs 2024-07-26T09:41:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9916.342711 secs 2024-07-26T09:41:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9934.083011 secs 2024-07-26T09:41:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9916.342711 secs 2024-07-26T09:41:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9934.083011 secs 2024-07-26T09:41:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:27 smithi164 ceph-mon[36970]: pgmap v5834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:27 smithi163 ceph-mon[36873]: pgmap v5834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:27 smithi042 ceph-mon[32775]: pgmap v5834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:30 smithi042 ceph-mon[32775]: pgmap v5835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:30 smithi164 ceph-mon[36970]: pgmap v5835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:30 smithi163 ceph-mon[36873]: pgmap v5835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9921.342831 secs 2024-07-26T09:41:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9921.342831 secs 2024-07-26T09:41:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9921.342831 secs 2024-07-26T09:41:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9939.083135 secs 2024-07-26T09:41:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:32 smithi042 ceph-mon[32775]: pgmap v5836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:32.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9939.083135 secs 2024-07-26T09:41:32.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:32 smithi164 ceph-mon[36970]: pgmap v5836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9939.083135 secs 2024-07-26T09:41:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:32 smithi163 ceph-mon[36873]: pgmap v5836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:34 smithi042 ceph-mon[32775]: pgmap v5837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:41:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:34 smithi164 ceph-mon[36970]: pgmap v5837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:41:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:34 smithi163 ceph-mon[36873]: pgmap v5837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:41:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:36 smithi042 ceph-mon[32775]: pgmap v5838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9926.342971 secs 2024-07-26T09:41:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:36 smithi164 ceph-mon[36970]: pgmap v5838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:36.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9926.342971 secs 2024-07-26T09:41:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:36 smithi163 ceph-mon[36873]: pgmap v5838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9926.342971 secs 2024-07-26T09:41:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9944.083251 secs 2024-07-26T09:41:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9944.083251 secs 2024-07-26T09:41:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9944.083251 secs 2024-07-26T09:41:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:38 smithi042 ceph-mon[32775]: pgmap v5839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:38 smithi164 ceph-mon[36970]: pgmap v5839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:38 smithi163 ceph-mon[36873]: pgmap v5839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:40 smithi042 ceph-mon[32775]: pgmap v5840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:40 smithi164 ceph-mon[36970]: pgmap v5840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:40 smithi163 ceph-mon[36873]: pgmap v5840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9931.343112 secs 2024-07-26T09:41:41.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9931.343112 secs 2024-07-26T09:41:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9931.343112 secs 2024-07-26T09:41:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:41:41.971+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:41:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:41:41.971+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:41:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:41:41.971+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:41:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9949.083372 secs 2024-07-26T09:41:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:42 smithi042 ceph-mon[32775]: pgmap v5841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9949.083372 secs 2024-07-26T09:41:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:42 smithi164 ceph-mon[36970]: pgmap v5841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9949.083372 secs 2024-07-26T09:41:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:42 smithi163 ceph-mon[36873]: pgmap v5841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:44 smithi042 ceph-mon[32775]: pgmap v5842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:41:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:44 smithi164 ceph-mon[36970]: pgmap v5842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:41:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:44 smithi163 ceph-mon[36873]: pgmap v5842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:41:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:46 smithi042 ceph-mon[32775]: pgmap v5843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9936.343252 secs 2024-07-26T09:41:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:46 smithi164 ceph-mon[36970]: pgmap v5843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:46.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9936.343252 secs 2024-07-26T09:41:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:46 smithi163 ceph-mon[36873]: pgmap v5843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9936.343252 secs 2024-07-26T09:41:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9954.083471 secs 2024-07-26T09:41:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9954.083471 secs 2024-07-26T09:41:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9954.083471 secs 2024-07-26T09:41:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:48 smithi042 ceph-mon[32775]: pgmap v5844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:48 smithi164 ceph-mon[36970]: pgmap v5844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:48 smithi163 ceph-mon[36873]: pgmap v5844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:50 smithi042 ceph-mon[32775]: pgmap v5845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:50 smithi164 ceph-mon[36970]: pgmap v5845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:50 smithi163 ceph-mon[36873]: pgmap v5845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:41:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9941.343385 secs 2024-07-26T09:41:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9941.343385 secs 2024-07-26T09:41:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9941.343385 secs 2024-07-26T09:41:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9959.083610 secs 2024-07-26T09:41:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:52 smithi042 ceph-mon[32775]: pgmap v5846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9959.083610 secs 2024-07-26T09:41:52.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:52 smithi164 ceph-mon[36970]: pgmap v5846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9959.083610 secs 2024-07-26T09:41:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:52 smithi163 ceph-mon[36873]: pgmap v5846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:41:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:54 smithi042 ceph-mon[32775]: pgmap v5847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:41:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:54 smithi164 ceph-mon[36970]: pgmap v5847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:41:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:54 smithi163 ceph-mon[36873]: pgmap v5847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:41:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:56 smithi042 ceph-mon[32775]: pgmap v5848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:41:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9946.343486 secs 2024-07-26T09:41:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:56 smithi164 ceph-mon[36970]: pgmap v5848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:41:56.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9946.343486 secs 2024-07-26T09:41:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:56 smithi163 ceph-mon[36873]: pgmap v5848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:41:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9946.343486 secs 2024-07-26T09:41:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:41:56.973+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:41:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:41:56.973+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:41:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:41:56.973+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:41:57.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9964.083728 secs 2024-07-26T09:41:57.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:41:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9964.083728 secs 2024-07-26T09:41:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:41:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9964.083728 secs 2024-07-26T09:41:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:41:58.395 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:41:58 smithi164 ceph-mon[36970]: pgmap v5849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:41:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:41:58 smithi042 ceph-mon[32775]: pgmap v5849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:41:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:41:58 smithi163 ceph-mon[36873]: pgmap v5849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:42:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:00 smithi042 ceph-mon[32775]: pgmap v5850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:42:00.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:42:00.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:42:00.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:00 smithi164 ceph-mon[36970]: pgmap v5850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:42:00.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:42:00.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:42:00.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:00 smithi163 ceph-mon[36873]: pgmap v5850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:42:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:00.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:42:00.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:42:00.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:42:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:01 smithi042 ceph-mon[32775]: pgmap v5851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:01 smithi042 ceph-mon[32775]: pgmap v5852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T09:42:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:01 smithi164 ceph-mon[36970]: pgmap v5851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:01.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:01 smithi164 ceph-mon[36970]: pgmap v5852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T09:42:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:01 smithi163 ceph-mon[36873]: pgmap v5851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:01 smithi163 ceph-mon[36873]: pgmap v5852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T09:42:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9951.343606 secs 2024-07-26T09:42:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9969.083851 secs 2024-07-26T09:42:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9951.343606 secs 2024-07-26T09:42:02.492 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9969.083851 secs 2024-07-26T09:42:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9951.343606 secs 2024-07-26T09:42:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9969.083851 secs 2024-07-26T09:42:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:03 smithi042 ceph-mon[32775]: pgmap v5853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T09:42:03.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:03 smithi164 ceph-mon[36970]: pgmap v5853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T09:42:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:03 smithi163 ceph-mon[36873]: pgmap v5853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T09:42:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:05 smithi042 ceph-mon[32775]: pgmap v5854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T09:42:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:05 smithi164 ceph-mon[36970]: pgmap v5854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T09:42:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:05 smithi163 ceph-mon[36873]: pgmap v5854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T09:42:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9956.343731 secs 2024-07-26T09:42:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:07 smithi042 ceph-mon[32775]: pgmap v5855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:42:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9974.083970 secs 2024-07-26T09:42:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9956.343731 secs 2024-07-26T09:42:07.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:07 smithi164 ceph-mon[36970]: pgmap v5855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:42:07.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9974.083970 secs 2024-07-26T09:42:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9956.343731 secs 2024-07-26T09:42:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:07 smithi163 ceph-mon[36873]: pgmap v5855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:42:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9974.083970 secs 2024-07-26T09:42:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:09 smithi042 ceph-mon[32775]: pgmap v5856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T09:42:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:09 smithi164 ceph-mon[36970]: pgmap v5856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T09:42:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:09 smithi163 ceph-mon[36873]: pgmap v5856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T09:42:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:11 smithi042 ceph-mon[32775]: pgmap v5857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:42:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:11 smithi164 ceph-mon[36970]: pgmap v5857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:42:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:11 smithi163 ceph-mon[36873]: pgmap v5857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:42:11.985 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:42:11.983+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:42:11.985 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:42:11.983+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:42:11.985 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:42:11.983+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:42:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9961.343820 secs 2024-07-26T09:42:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9979.084030 secs 2024-07-26T09:42:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9961.343820 secs 2024-07-26T09:42:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9979.084030 secs 2024-07-26T09:42:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9961.343820 secs 2024-07-26T09:42:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9979.084030 secs 2024-07-26T09:42:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:13 smithi042 ceph-mon[32775]: pgmap v5858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:13 smithi164 ceph-mon[36970]: pgmap v5858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:13 smithi163 ceph-mon[36873]: pgmap v5858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:15 smithi042 ceph-mon[32775]: pgmap v5859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:42:15.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:15 smithi164 ceph-mon[36970]: pgmap v5859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:42:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:15 smithi163 ceph-mon[36873]: pgmap v5859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:42:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9966.343927 secs 2024-07-26T09:42:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:17 smithi042 ceph-mon[32775]: pgmap v5860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9984.084117 secs 2024-07-26T09:42:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9966.343927 secs 2024-07-26T09:42:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:17 smithi164 ceph-mon[36970]: pgmap v5860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9984.084117 secs 2024-07-26T09:42:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9966.343927 secs 2024-07-26T09:42:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:17 smithi163 ceph-mon[36873]: pgmap v5860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9984.084117 secs 2024-07-26T09:42:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:19 smithi042 ceph-mon[32775]: pgmap v5861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:19 smithi164 ceph-mon[36970]: pgmap v5861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:19 smithi163 ceph-mon[36873]: pgmap v5861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:21 smithi042 ceph-mon[32775]: pgmap v5862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:21 smithi164 ceph-mon[36970]: pgmap v5862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:21 smithi163 ceph-mon[36873]: pgmap v5862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9971.344019 secs 2024-07-26T09:42:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9989.084213 secs 2024-07-26T09:42:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9971.344019 secs 2024-07-26T09:42:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9989.084213 secs 2024-07-26T09:42:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9971.344019 secs 2024-07-26T09:42:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9989.084213 secs 2024-07-26T09:42:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:23 smithi042 ceph-mon[32775]: pgmap v5863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:23 smithi164 ceph-mon[36970]: pgmap v5863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:23 smithi163 ceph-mon[36873]: pgmap v5863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:25 smithi042 ceph-mon[32775]: pgmap v5864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:42:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:25 smithi164 ceph-mon[36970]: pgmap v5864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:42:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:25 smithi163 ceph-mon[36873]: pgmap v5864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:42:26.989 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:42:26.987+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:42:26.989 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:42:26.987+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:42:26.989 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:42:26.987+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:42:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9976.344145 secs 2024-07-26T09:42:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:27 smithi042 ceph-mon[32775]: pgmap v5865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9994.084334 secs 2024-07-26T09:42:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9976.344145 secs 2024-07-26T09:42:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:27 smithi164 ceph-mon[36970]: pgmap v5865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9994.084334 secs 2024-07-26T09:42:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9976.344145 secs 2024-07-26T09:42:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:27 smithi163 ceph-mon[36873]: pgmap v5865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9994.084334 secs 2024-07-26T09:42:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:29 smithi042 ceph-mon[32775]: pgmap v5866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:29 smithi164 ceph-mon[36970]: pgmap v5866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:29 smithi163 ceph-mon[36873]: pgmap v5866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:31 smithi042 ceph-mon[32775]: pgmap v5867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:31 smithi164 ceph-mon[36970]: pgmap v5867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:31 smithi163 ceph-mon[36873]: pgmap v5867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9981.344269 secs 2024-07-26T09:42:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9999.084480 secs 2024-07-26T09:42:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9981.344269 secs 2024-07-26T09:42:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9999.084480 secs 2024-07-26T09:42:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9981.344269 secs 2024-07-26T09:42:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9999.084480 secs 2024-07-26T09:42:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:33 smithi042 ceph-mon[32775]: pgmap v5868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:33.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:33 smithi164 ceph-mon[36970]: pgmap v5868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:33 smithi163 ceph-mon[36873]: pgmap v5868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:35 smithi042 ceph-mon[32775]: pgmap v5869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:42:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:35 smithi164 ceph-mon[36970]: pgmap v5869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:42:35.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:35 smithi163 ceph-mon[36873]: pgmap v5869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:42:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9986.344372 secs 2024-07-26T09:42:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:37 smithi042 ceph-mon[32775]: pgmap v5870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10004.084620 secs 2024-07-26T09:42:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9986.344372 secs 2024-07-26T09:42:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:37 smithi164 ceph-mon[36970]: pgmap v5870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10004.084620 secs 2024-07-26T09:42:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9986.344372 secs 2024-07-26T09:42:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:37 smithi163 ceph-mon[36873]: pgmap v5870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10004.084620 secs 2024-07-26T09:42:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:39 smithi042 ceph-mon[32775]: pgmap v5871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:39 smithi164 ceph-mon[36970]: pgmap v5871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:39 smithi163 ceph-mon[36873]: pgmap v5871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:41 smithi042 ceph-mon[32775]: pgmap v5872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:41 smithi164 ceph-mon[36970]: pgmap v5872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:41 smithi163 ceph-mon[36873]: pgmap v5872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:42:41.993+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:42:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:42:41.993+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:42:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:42:41.993+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:42:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9991.344489 secs 2024-07-26T09:42:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10009.084689 secs 2024-07-26T09:42:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9991.344489 secs 2024-07-26T09:42:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10009.084689 secs 2024-07-26T09:42:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9991.344489 secs 2024-07-26T09:42:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10009.084689 secs 2024-07-26T09:42:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:43 smithi042 ceph-mon[32775]: pgmap v5873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:43 smithi164 ceph-mon[36970]: pgmap v5873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:43 smithi163 ceph-mon[36873]: pgmap v5873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:45.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:45 smithi042 ceph-mon[32775]: pgmap v5874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:42:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:45 smithi164 ceph-mon[36970]: pgmap v5874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:42:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:45 smithi163 ceph-mon[36873]: pgmap v5874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:42:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 9996.344609 secs 2024-07-26T09:42:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:47 smithi042 ceph-mon[32775]: pgmap v5875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:47.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10014.084819 secs 2024-07-26T09:42:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 9996.344609 secs 2024-07-26T09:42:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:47 smithi164 ceph-mon[36970]: pgmap v5875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10014.084819 secs 2024-07-26T09:42:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 9996.344609 secs 2024-07-26T09:42:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:47 smithi163 ceph-mon[36873]: pgmap v5875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10014.084819 secs 2024-07-26T09:42:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:49 smithi042 ceph-mon[32775]: pgmap v5876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:49 smithi164 ceph-mon[36970]: pgmap v5876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:49 smithi163 ceph-mon[36873]: pgmap v5876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:51 smithi042 ceph-mon[32775]: pgmap v5877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:51 smithi164 ceph-mon[36970]: pgmap v5877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:51 smithi163 ceph-mon[36873]: pgmap v5877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10001.344714 secs 2024-07-26T09:42:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10019.084925 secs 2024-07-26T09:42:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10001.344714 secs 2024-07-26T09:42:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10019.084925 secs 2024-07-26T09:42:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10001.344714 secs 2024-07-26T09:42:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10019.084925 secs 2024-07-26T09:42:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:53 smithi164 ceph-mon[36970]: pgmap v5878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:53 smithi163 ceph-mon[36873]: pgmap v5878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:53 smithi042 ceph-mon[32775]: pgmap v5878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:55.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:55 smithi042 ceph-mon[32775]: pgmap v5879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:42:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:55 smithi164 ceph-mon[36970]: pgmap v5879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:42:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:55 smithi163 ceph-mon[36873]: pgmap v5879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:42:57.000 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:42:56.998+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:42:57.001 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:42:56.998+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:42:57.001 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:42:56.998+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:42:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10006.344867 secs 2024-07-26T09:42:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:57 smithi164 ceph-mon[36970]: pgmap v5880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10024.085015 secs 2024-07-26T09:42:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10006.344867 secs 2024-07-26T09:42:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:57 smithi163 ceph-mon[36873]: pgmap v5880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10024.085015 secs 2024-07-26T09:42:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10006.344867 secs 2024-07-26T09:42:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:57 smithi042 ceph-mon[32775]: pgmap v5880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:42:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10024.085015 secs 2024-07-26T09:42:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:42:59 smithi164 ceph-mon[36970]: pgmap v5881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:42:59 smithi163 ceph-mon[36873]: pgmap v5881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:42:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:42:59 smithi042 ceph-mon[32775]: pgmap v5881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:43:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:43:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:43:01.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:01 smithi164 ceph-mon[36970]: pgmap v5882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:01.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10011.344937 secs 2024-07-26T09:43:01.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:43:01.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:43:01.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:01 smithi163 ceph-mon[36873]: pgmap v5882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10011.344937 secs 2024-07-26T09:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:01 smithi042 ceph-mon[32775]: pgmap v5882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10011.344937 secs 2024-07-26T09:43:01.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:01.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:43:01.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:43:01.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:43:02.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:02 smithi164 ceph-mon[36970]: pgmap v5883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T09:43:02.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:02 smithi164 ceph-mon[36970]: pgmap v5884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:43:02.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10029.085081 secs 2024-07-26T09:43:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:02 smithi163 ceph-mon[36873]: pgmap v5883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T09:43:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:02 smithi163 ceph-mon[36873]: pgmap v5884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:43:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10029.085081 secs 2024-07-26T09:43:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:02 smithi042 ceph-mon[32775]: pgmap v5883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T09:43:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:02 smithi042 ceph-mon[32775]: pgmap v5884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:43:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10029.085081 secs 2024-07-26T09:43:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:04 smithi164 ceph-mon[36970]: pgmap v5885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T09:43:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:04 smithi163 ceph-mon[36873]: pgmap v5885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T09:43:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:04 smithi042 ceph-mon[32775]: pgmap v5885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T09:43:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:06 smithi164 ceph-mon[36970]: pgmap v5886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:43:06.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10016.345081 secs 2024-07-26T09:43:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:06 smithi163 ceph-mon[36873]: pgmap v5886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:43:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10016.345081 secs 2024-07-26T09:43:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:06 smithi042 ceph-mon[32775]: pgmap v5886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:43:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10016.345081 secs 2024-07-26T09:43:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10034.085225 secs 2024-07-26T09:43:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10034.085225 secs 2024-07-26T09:43:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10034.085225 secs 2024-07-26T09:43:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:08 smithi164 ceph-mon[36970]: pgmap v5887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T09:43:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:08 smithi163 ceph-mon[36873]: pgmap v5887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T09:43:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:08 smithi042 ceph-mon[32775]: pgmap v5887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T09:43:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:10 smithi164 ceph-mon[36970]: pgmap v5888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:43:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:10 smithi163 ceph-mon[36873]: pgmap v5888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:43:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:10 smithi042 ceph-mon[32775]: pgmap v5888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T09:43:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10021.345172 secs 2024-07-26T09:43:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10021.345172 secs 2024-07-26T09:43:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10021.345172 secs 2024-07-26T09:43:12.010 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:43:12.008+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:43:12.010 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:43:12.008+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:43:12.010 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:43:12.008+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:43:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:12 smithi164 ceph-mon[36970]: pgmap v5889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:43:12.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10039.085315 secs 2024-07-26T09:43:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:12 smithi163 ceph-mon[36873]: pgmap v5889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:43:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10039.085315 secs 2024-07-26T09:43:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:12 smithi042 ceph-mon[32775]: pgmap v5889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:43:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10039.085315 secs 2024-07-26T09:43:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:14 smithi164 ceph-mon[36970]: pgmap v5890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:14 smithi163 ceph-mon[36873]: pgmap v5890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:14 smithi042 ceph-mon[32775]: pgmap v5890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:16 smithi164 ceph-mon[36970]: pgmap v5891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:16.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10026.345295 secs 2024-07-26T09:43:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:16 smithi163 ceph-mon[36873]: pgmap v5891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10026.345295 secs 2024-07-26T09:43:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:16 smithi042 ceph-mon[32775]: pgmap v5891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10026.345295 secs 2024-07-26T09:43:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10044.085462 secs 2024-07-26T09:43:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10044.085462 secs 2024-07-26T09:43:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10044.085462 secs 2024-07-26T09:43:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:18 smithi164 ceph-mon[36970]: pgmap v5892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:18 smithi163 ceph-mon[36873]: pgmap v5892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:18 smithi042 ceph-mon[32775]: pgmap v5892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:20.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:20 smithi164 ceph-mon[36970]: pgmap v5893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:20 smithi163 ceph-mon[36873]: pgmap v5893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:20 smithi042 ceph-mon[32775]: pgmap v5893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10031.345400 secs 2024-07-26T09:43:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10031.345400 secs 2024-07-26T09:43:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10031.345400 secs 2024-07-26T09:43:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:22 smithi164 ceph-mon[36970]: pgmap v5894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10049.085584 secs 2024-07-26T09:43:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:22 smithi163 ceph-mon[36873]: pgmap v5894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10049.085584 secs 2024-07-26T09:43:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:22 smithi042 ceph-mon[32775]: pgmap v5894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10049.085584 secs 2024-07-26T09:43:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:24 smithi164 ceph-mon[36970]: pgmap v5895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:24 smithi163 ceph-mon[36873]: pgmap v5895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:24 smithi042 ceph-mon[32775]: pgmap v5895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:26 smithi163 ceph-mon[36873]: pgmap v5896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10036.345524 secs 2024-07-26T09:43:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:26 smithi042 ceph-mon[32775]: pgmap v5896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10036.345524 secs 2024-07-26T09:43:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:26 smithi164 ceph-mon[36970]: pgmap v5896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10036.345524 secs 2024-07-26T09:43:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:43:27.013+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:43:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:43:27.013+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:43:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:43:27.013+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:43:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10054.085679 secs 2024-07-26T09:43:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10054.085679 secs 2024-07-26T09:43:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10054.085679 secs 2024-07-26T09:43:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:28 smithi163 ceph-mon[36873]: pgmap v5897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:28 smithi042 ceph-mon[32775]: pgmap v5897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:28 smithi164 ceph-mon[36970]: pgmap v5897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:30 smithi163 ceph-mon[36873]: pgmap v5898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:30 smithi042 ceph-mon[32775]: pgmap v5898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:30 smithi164 ceph-mon[36970]: pgmap v5898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10041.345656 secs 2024-07-26T09:43:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10041.345656 secs 2024-07-26T09:43:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10041.345656 secs 2024-07-26T09:43:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:32 smithi163 ceph-mon[36873]: pgmap v5899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10059.085799 secs 2024-07-26T09:43:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:32 smithi042 ceph-mon[32775]: pgmap v5899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10059.085799 secs 2024-07-26T09:43:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:32 smithi164 ceph-mon[36970]: pgmap v5899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10059.085799 secs 2024-07-26T09:43:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:34 smithi163 ceph-mon[36873]: pgmap v5900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:34 smithi042 ceph-mon[32775]: pgmap v5900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:34 smithi164 ceph-mon[36970]: pgmap v5900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:36 smithi163 ceph-mon[36873]: pgmap v5901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10046.345797 secs 2024-07-26T09:43:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:36 smithi042 ceph-mon[32775]: pgmap v5901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10046.345797 secs 2024-07-26T09:43:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:36 smithi164 ceph-mon[36970]: pgmap v5901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10046.345797 secs 2024-07-26T09:43:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10064.085925 secs 2024-07-26T09:43:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10064.085925 secs 2024-07-26T09:43:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10064.085925 secs 2024-07-26T09:43:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:38 smithi163 ceph-mon[36873]: pgmap v5902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:38 smithi042 ceph-mon[32775]: pgmap v5902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:38 smithi164 ceph-mon[36970]: pgmap v5902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:40 smithi042 ceph-mon[32775]: pgmap v5903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:40 smithi164 ceph-mon[36970]: pgmap v5903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:40 smithi163 ceph-mon[36873]: pgmap v5903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:41.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10051.345924 secs 2024-07-26T09:43:41.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10051.345924 secs 2024-07-26T09:43:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10051.345924 secs 2024-07-26T09:43:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:43:42.014+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:43:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:43:42.014+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:43:42.016 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:43:42.014+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:43:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:42 smithi042 ceph-mon[32775]: pgmap v5904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10069.086026 secs 2024-07-26T09:43:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:42 smithi164 ceph-mon[36970]: pgmap v5904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:42.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10069.086026 secs 2024-07-26T09:43:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:42 smithi163 ceph-mon[36873]: pgmap v5904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:43.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10069.086026 secs 2024-07-26T09:43:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:44 smithi042 ceph-mon[32775]: pgmap v5905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:44 smithi164 ceph-mon[36970]: pgmap v5905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:44 smithi163 ceph-mon[36873]: pgmap v5905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:46 smithi042 ceph-mon[32775]: pgmap v5906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10056.346048 secs 2024-07-26T09:43:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:46 smithi164 ceph-mon[36970]: pgmap v5906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10056.346048 secs 2024-07-26T09:43:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:46 smithi163 ceph-mon[36873]: pgmap v5906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10056.346048 secs 2024-07-26T09:43:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10074.086107 secs 2024-07-26T09:43:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10074.086107 secs 2024-07-26T09:43:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10074.086107 secs 2024-07-26T09:43:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:48 smithi042 ceph-mon[32775]: pgmap v5907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:48.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:48 smithi164 ceph-mon[36970]: pgmap v5907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:48 smithi163 ceph-mon[36873]: pgmap v5907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:50 smithi042 ceph-mon[32775]: pgmap v5908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:50 smithi164 ceph-mon[36970]: pgmap v5908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:50 smithi163 ceph-mon[36873]: pgmap v5908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:51.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10061.346174 secs 2024-07-26T09:43:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10061.346174 secs 2024-07-26T09:43:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10061.346174 secs 2024-07-26T09:43:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:52 smithi042 ceph-mon[32775]: pgmap v5909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10079.086226 secs 2024-07-26T09:43:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:52 smithi164 ceph-mon[36970]: pgmap v5909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:52.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10079.086226 secs 2024-07-26T09:43:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:52 smithi163 ceph-mon[36873]: pgmap v5909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10079.086226 secs 2024-07-26T09:43:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:54 smithi042 ceph-mon[32775]: pgmap v5910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:54 smithi164 ceph-mon[36970]: pgmap v5910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:54 smithi163 ceph-mon[36873]: pgmap v5910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:56 smithi042 ceph-mon[32775]: pgmap v5911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10066.346294 secs 2024-07-26T09:43:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:56 smithi164 ceph-mon[36970]: pgmap v5911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10066.346294 secs 2024-07-26T09:43:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:56 smithi163 ceph-mon[36873]: pgmap v5911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:43:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10066.346294 secs 2024-07-26T09:43:57.021 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:43:57.019+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:43:57.021 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:43:57.019+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:43:57.021 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:43:57.019+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:43:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10084.086308 secs 2024-07-26T09:43:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10084.086308 secs 2024-07-26T09:43:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10084.086308 secs 2024-07-26T09:43:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:43:58 smithi042 ceph-mon[32775]: pgmap v5912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:43:58 smithi164 ceph-mon[36970]: pgmap v5912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:43:58 smithi163 ceph-mon[36873]: pgmap v5912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:43:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:43:59.431+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:43:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:43:59.431+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:43:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:43:59.431+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:43:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:43:59.431+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:44:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:00 smithi042 ceph-mon[32775]: pgmap v5913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:00 smithi164 ceph-mon[36970]: pgmap v5913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:00 smithi163 ceph-mon[36873]: pgmap v5913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10071.346451 secs 2024-07-26T09:44:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:44:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10071.346451 secs 2024-07-26T09:44:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:44:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10071.346451 secs 2024-07-26T09:44:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:44:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:02 smithi042 ceph-mon[32775]: pgmap v5914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10089.086405 secs 2024-07-26T09:44:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:44:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:44:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:02 smithi042 ceph-mon[32775]: pgmap v5915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:44:02.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:02 smithi042 ceph-mon[32775]: pgmap v5916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:44:02.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:44:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:02 smithi164 ceph-mon[36970]: pgmap v5914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10089.086405 secs 2024-07-26T09:44:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:44:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:44:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:02 smithi164 ceph-mon[36970]: pgmap v5915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:44:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:02 smithi164 ceph-mon[36970]: pgmap v5916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:44:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:44:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:02 smithi163 ceph-mon[36873]: pgmap v5914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10089.086405 secs 2024-07-26T09:44:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:44:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:44:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:02 smithi163 ceph-mon[36873]: pgmap v5915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:44:03.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:02 smithi163 ceph-mon[36873]: pgmap v5916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:44:03.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:44:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:04 smithi163 ceph-mon[36873]: pgmap v5917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:44:05.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:04 smithi042 ceph-mon[32775]: pgmap v5917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:44:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:04 smithi164 ceph-mon[36970]: pgmap v5917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:44:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:06 smithi163 ceph-mon[36873]: pgmap v5918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:44:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10076.346573 secs 2024-07-26T09:44:07.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10094.086545 secs 2024-07-26T09:44:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:06 smithi042 ceph-mon[32775]: pgmap v5918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:44:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10076.346573 secs 2024-07-26T09:44:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10094.086545 secs 2024-07-26T09:44:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:06 smithi164 ceph-mon[36970]: pgmap v5918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:44:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10076.346573 secs 2024-07-26T09:44:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10094.086545 secs 2024-07-26T09:44:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:08 smithi163 ceph-mon[36873]: pgmap v5919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:44:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:08 smithi042 ceph-mon[32775]: pgmap v5919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:44:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:08 smithi164 ceph-mon[36970]: pgmap v5919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:44:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:10 smithi163 ceph-mon[36873]: pgmap v5920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:44:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:10 smithi042 ceph-mon[32775]: pgmap v5920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:44:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:10 smithi164 ceph-mon[36970]: pgmap v5920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:44:12.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:44:12.029+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:44:12.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:44:12.029+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:44:12.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:44:12.029+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:44:12.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10081.346697 secs 2024-07-26T09:44:12.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10099.086681 secs 2024-07-26T09:44:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10081.346697 secs 2024-07-26T09:44:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10099.086681 secs 2024-07-26T09:44:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10081.346697 secs 2024-07-26T09:44:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10099.086681 secs 2024-07-26T09:44:13.041 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:12 smithi042 ceph-mon[32775]: pgmap v5921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:44:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:12 smithi164 ceph-mon[36970]: pgmap v5921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:44:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:12 smithi163 ceph-mon[36873]: pgmap v5921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:44:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:14 smithi042 ceph-mon[32775]: pgmap v5922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:44:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:14 smithi164 ceph-mon[36970]: pgmap v5922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:44:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:14 smithi163 ceph-mon[36873]: pgmap v5922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:44:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:16 smithi042 ceph-mon[32775]: pgmap v5923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10086.346839 secs 2024-07-26T09:44:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10104.086792 secs 2024-07-26T09:44:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:16 smithi164 ceph-mon[36970]: pgmap v5923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10086.346839 secs 2024-07-26T09:44:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10104.086792 secs 2024-07-26T09:44:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:16 smithi163 ceph-mon[36873]: pgmap v5923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10086.346839 secs 2024-07-26T09:44:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10104.086792 secs 2024-07-26T09:44:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:18 smithi042 ceph-mon[32775]: pgmap v5924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:19.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:18 smithi164 ceph-mon[36970]: pgmap v5924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:19.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:18 smithi163 ceph-mon[36873]: pgmap v5924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:21.026 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:20 smithi164 ceph-mon[36970]: pgmap v5925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:20 smithi042 ceph-mon[32775]: pgmap v5925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:20 smithi163 ceph-mon[36873]: pgmap v5925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10091.346964 secs 2024-07-26T09:44:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10109.086923 secs 2024-07-26T09:44:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10091.346964 secs 2024-07-26T09:44:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10109.086923 secs 2024-07-26T09:44:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10091.346964 secs 2024-07-26T09:44:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10109.086923 secs 2024-07-26T09:44:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:22 smithi042 ceph-mon[32775]: pgmap v5926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:23.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:22 smithi164 ceph-mon[36970]: pgmap v5926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:22 smithi163 ceph-mon[36873]: pgmap v5926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:25.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:24 smithi042 ceph-mon[32775]: pgmap v5927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:44:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:24 smithi164 ceph-mon[36970]: pgmap v5927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:44:25.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:24 smithi163 ceph-mon[36873]: pgmap v5927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:44:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:44:27.031+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:44:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:44:27.031+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:44:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:44:27.031+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:44:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:26 smithi042 ceph-mon[32775]: pgmap v5928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10096.347092 secs 2024-07-26T09:44:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10114.087053 secs 2024-07-26T09:44:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:26 smithi164 ceph-mon[36970]: pgmap v5928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10096.347092 secs 2024-07-26T09:44:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10114.087053 secs 2024-07-26T09:44:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:26 smithi163 ceph-mon[36873]: pgmap v5928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10096.347092 secs 2024-07-26T09:44:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10114.087053 secs 2024-07-26T09:44:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:28 smithi042 ceph-mon[32775]: pgmap v5929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:28 smithi164 ceph-mon[36970]: pgmap v5929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:28 smithi163 ceph-mon[36873]: pgmap v5929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:30 smithi042 ceph-mon[32775]: pgmap v5930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:30 smithi164 ceph-mon[36970]: pgmap v5930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:31.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:30 smithi163 ceph-mon[36873]: pgmap v5930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10101.347228 secs 2024-07-26T09:44:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10119.087121 secs 2024-07-26T09:44:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10101.347228 secs 2024-07-26T09:44:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10119.087121 secs 2024-07-26T09:44:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10101.347228 secs 2024-07-26T09:44:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10119.087121 secs 2024-07-26T09:44:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:32 smithi042 ceph-mon[32775]: pgmap v5931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:32 smithi164 ceph-mon[36970]: pgmap v5931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:32 smithi163 ceph-mon[36873]: pgmap v5931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:35.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:34 smithi042 ceph-mon[32775]: pgmap v5932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:44:35.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:34 smithi164 ceph-mon[36970]: pgmap v5932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:44:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:34 smithi163 ceph-mon[36873]: pgmap v5932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:44:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:36 smithi042 ceph-mon[32775]: pgmap v5933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10106.347346 secs 2024-07-26T09:44:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10124.087185 secs 2024-07-26T09:44:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:36 smithi164 ceph-mon[36970]: pgmap v5933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10106.347346 secs 2024-07-26T09:44:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10124.087185 secs 2024-07-26T09:44:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:36 smithi163 ceph-mon[36873]: pgmap v5933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10106.347346 secs 2024-07-26T09:44:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10124.087185 secs 2024-07-26T09:44:39.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:38 smithi042 ceph-mon[32775]: pgmap v5934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:38 smithi164 ceph-mon[36970]: pgmap v5934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:38 smithi163 ceph-mon[36873]: pgmap v5934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:41.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:40 smithi042 ceph-mon[32775]: pgmap v5935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:40 smithi164 ceph-mon[36970]: pgmap v5935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:41.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:40 smithi163 ceph-mon[36873]: pgmap v5935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:42.043 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:44:42.041+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:44:42.043 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:44:42.041+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:44:42.043 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:44:42.041+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:44:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10111.347469 secs 2024-07-26T09:44:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10129.087270 secs 2024-07-26T09:44:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10111.347469 secs 2024-07-26T09:44:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10129.087270 secs 2024-07-26T09:44:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10111.347469 secs 2024-07-26T09:44:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10129.087270 secs 2024-07-26T09:44:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:42 smithi042 ceph-mon[32775]: pgmap v5936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:42 smithi164 ceph-mon[36970]: pgmap v5936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:42 smithi163 ceph-mon[36873]: pgmap v5936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:44 smithi042 ceph-mon[32775]: pgmap v5937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:44:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:44 smithi164 ceph-mon[36970]: pgmap v5937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:44:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:44 smithi163 ceph-mon[36873]: pgmap v5937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:44:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:46 smithi042 ceph-mon[32775]: pgmap v5938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10116.347595 secs 2024-07-26T09:44:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10134.087395 secs 2024-07-26T09:44:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:46 smithi164 ceph-mon[36970]: pgmap v5938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10116.347595 secs 2024-07-26T09:44:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10134.087395 secs 2024-07-26T09:44:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:46 smithi163 ceph-mon[36873]: pgmap v5938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10116.347595 secs 2024-07-26T09:44:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10134.087395 secs 2024-07-26T09:44:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:48 smithi042 ceph-mon[32775]: pgmap v5939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:48 smithi164 ceph-mon[36970]: pgmap v5939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:48 smithi163 ceph-mon[36873]: pgmap v5939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:50 smithi042 ceph-mon[32775]: pgmap v5940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:50 smithi164 ceph-mon[36970]: pgmap v5940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:50 smithi163 ceph-mon[36873]: pgmap v5940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10121.347722 secs 2024-07-26T09:44:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10139.087463 secs 2024-07-26T09:44:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10121.347722 secs 2024-07-26T09:44:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10139.087463 secs 2024-07-26T09:44:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10121.347722 secs 2024-07-26T09:44:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10139.087463 secs 2024-07-26T09:44:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:52 smithi042 ceph-mon[32775]: pgmap v5941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:52 smithi164 ceph-mon[36970]: pgmap v5941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:52 smithi163 ceph-mon[36873]: pgmap v5941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:54 smithi042 ceph-mon[32775]: pgmap v5942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:44:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:54 smithi164 ceph-mon[36970]: pgmap v5942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:44:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:54 smithi163 ceph-mon[36873]: pgmap v5942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:44:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:44:57.045+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:44:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:44:57.045+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:44:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:44:57.045+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:44:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:56 smithi042 ceph-mon[32775]: pgmap v5943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10126.347880 secs 2024-07-26T09:44:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10144.087557 secs 2024-07-26T09:44:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:56 smithi164 ceph-mon[36970]: pgmap v5943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10126.347880 secs 2024-07-26T09:44:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10144.087557 secs 2024-07-26T09:44:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:56 smithi163 ceph-mon[36873]: pgmap v5943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:44:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10126.347880 secs 2024-07-26T09:44:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10144.087557 secs 2024-07-26T09:44:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:44:58 smithi042 ceph-mon[32775]: pgmap v5944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:44:58 smithi164 ceph-mon[36970]: pgmap v5944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:44:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:44:58 smithi163 ceph-mon[36873]: pgmap v5944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:00 smithi042 ceph-mon[32775]: pgmap v5945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:00 smithi164 ceph-mon[36970]: pgmap v5945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:00 smithi163 ceph-mon[36873]: pgmap v5945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10131.348035 secs 2024-07-26T09:45:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10149.087698 secs 2024-07-26T09:45:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:45:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10131.348035 secs 2024-07-26T09:45:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10149.087698 secs 2024-07-26T09:45:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:45:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10131.348035 secs 2024-07-26T09:45:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10149.087698 secs 2024-07-26T09:45:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:45:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:02 smithi042 ceph-mon[32775]: pgmap v5946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:45:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:45:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:45:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:02 smithi164 ceph-mon[36970]: pgmap v5946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:03.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:45:03.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:45:03.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:45:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:02 smithi163 ceph-mon[36873]: pgmap v5946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:45:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:45:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:45:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:03 smithi042 ceph-mon[32775]: pgmap v5947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:45:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:03 smithi042 ceph-mon[32775]: pgmap v5948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:45:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:03 smithi164 ceph-mon[36970]: pgmap v5947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:45:04.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:03 smithi164 ceph-mon[36970]: pgmap v5948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:45:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:03 smithi163 ceph-mon[36873]: pgmap v5947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:45:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:03 smithi163 ceph-mon[36873]: pgmap v5948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:45:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:05 smithi042 ceph-mon[32775]: pgmap v5949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:45:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:05 smithi164 ceph-mon[36970]: pgmap v5949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:45:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:05 smithi163 ceph-mon[36873]: pgmap v5949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:45:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10136.348103 secs 2024-07-26T09:45:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10154.087844 secs 2024-07-26T09:45:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10136.348103 secs 2024-07-26T09:45:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10154.087844 secs 2024-07-26T09:45:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10136.348103 secs 2024-07-26T09:45:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10154.087844 secs 2024-07-26T09:45:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:07 smithi042 ceph-mon[32775]: pgmap v5950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:45:08.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:07 smithi164 ceph-mon[36970]: pgmap v5950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:45:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:07 smithi163 ceph-mon[36873]: pgmap v5950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:45:09.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:08 smithi164 ceph-mon[36970]: pgmap v5951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:45:09.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:08 smithi163 ceph-mon[36873]: pgmap v5951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:45:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:08 smithi042 ceph-mon[32775]: pgmap v5951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:45:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:11 smithi042 ceph-mon[32775]: pgmap v5952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:45:11.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10141.348256 secs 2024-07-26T09:45:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:11 smithi164 ceph-mon[36970]: pgmap v5952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:45:11.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10141.348256 secs 2024-07-26T09:45:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:11 smithi163 ceph-mon[36873]: pgmap v5952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:45:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10141.348256 secs 2024-07-26T09:45:12.057 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:45:12.056+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:45:12.058 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:45:12.056+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:45:12.058 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:45:12.056+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:45:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10159.087949 secs 2024-07-26T09:45:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10159.087949 secs 2024-07-26T09:45:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10159.087949 secs 2024-07-26T09:45:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:13 smithi042 ceph-mon[32775]: pgmap v5953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:45:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:13 smithi164 ceph-mon[36970]: pgmap v5953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:45:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:13 smithi163 ceph-mon[36873]: pgmap v5953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:45:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:15 smithi042 ceph-mon[32775]: pgmap v5954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:15 smithi164 ceph-mon[36970]: pgmap v5954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:15 smithi163 ceph-mon[36873]: pgmap v5954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10146.348381 secs 2024-07-26T09:45:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10146.348381 secs 2024-07-26T09:45:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10146.348381 secs 2024-07-26T09:45:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10164.088069 secs 2024-07-26T09:45:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:17 smithi042 ceph-mon[32775]: pgmap v5955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10164.088069 secs 2024-07-26T09:45:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:17 smithi164 ceph-mon[36970]: pgmap v5955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10164.088069 secs 2024-07-26T09:45:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:17 smithi163 ceph-mon[36873]: pgmap v5955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:19 smithi042 ceph-mon[32775]: pgmap v5956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:19 smithi164 ceph-mon[36970]: pgmap v5956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:19 smithi163 ceph-mon[36873]: pgmap v5956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:21 smithi042 ceph-mon[32775]: pgmap v5957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10151.348503 secs 2024-07-26T09:45:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:21 smithi164 ceph-mon[36970]: pgmap v5957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10151.348503 secs 2024-07-26T09:45:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:21 smithi163 ceph-mon[36873]: pgmap v5957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10151.348503 secs 2024-07-26T09:45:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10169.088185 secs 2024-07-26T09:45:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10169.088185 secs 2024-07-26T09:45:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10169.088185 secs 2024-07-26T09:45:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:23 smithi042 ceph-mon[32775]: pgmap v5958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:23 smithi164 ceph-mon[36970]: pgmap v5958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:23 smithi163 ceph-mon[36873]: pgmap v5958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:25 smithi042 ceph-mon[32775]: pgmap v5959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:25 smithi164 ceph-mon[36970]: pgmap v5959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:25 smithi163 ceph-mon[36873]: pgmap v5959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10156.348631 secs 2024-07-26T09:45:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10156.348631 secs 2024-07-26T09:45:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10156.348631 secs 2024-07-26T09:45:27.059 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:45:27.058+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:45:27.060 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:45:27.058+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:45:27.060 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:45:27.058+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:45:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10174.088293 secs 2024-07-26T09:45:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:27 smithi042 ceph-mon[32775]: pgmap v5960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10174.088293 secs 2024-07-26T09:45:27.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:27 smithi164 ceph-mon[36970]: pgmap v5960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10174.088293 secs 2024-07-26T09:45:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:27 smithi163 ceph-mon[36873]: pgmap v5960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:29 smithi042 ceph-mon[32775]: pgmap v5961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:29 smithi164 ceph-mon[36970]: pgmap v5961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:29 smithi163 ceph-mon[36873]: pgmap v5961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:31 smithi042 ceph-mon[32775]: pgmap v5962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10161.348755 secs 2024-07-26T09:45:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:31 smithi164 ceph-mon[36970]: pgmap v5962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:31.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10161.348755 secs 2024-07-26T09:45:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:31 smithi163 ceph-mon[36873]: pgmap v5962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10161.348755 secs 2024-07-26T09:45:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10179.088416 secs 2024-07-26T09:45:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10179.088416 secs 2024-07-26T09:45:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10179.088416 secs 2024-07-26T09:45:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:33 smithi042 ceph-mon[32775]: pgmap v5963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:33 smithi164 ceph-mon[36970]: pgmap v5963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:33 smithi163 ceph-mon[36873]: pgmap v5963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:35 smithi042 ceph-mon[32775]: pgmap v5964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:35 smithi164 ceph-mon[36970]: pgmap v5964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:35 smithi163 ceph-mon[36873]: pgmap v5964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10166.348872 secs 2024-07-26T09:45:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10166.348872 secs 2024-07-26T09:45:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10166.348872 secs 2024-07-26T09:45:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10184.088490 secs 2024-07-26T09:45:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:37 smithi042 ceph-mon[32775]: pgmap v5965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10184.088490 secs 2024-07-26T09:45:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:37 smithi164 ceph-mon[36970]: pgmap v5965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10184.088490 secs 2024-07-26T09:45:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:37 smithi163 ceph-mon[36873]: pgmap v5965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:39 smithi042 ceph-mon[32775]: pgmap v5966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:39 smithi164 ceph-mon[36970]: pgmap v5966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:39 smithi163 ceph-mon[36873]: pgmap v5966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:41 smithi042 ceph-mon[32775]: pgmap v5967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10171.349005 secs 2024-07-26T09:45:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:41 smithi164 ceph-mon[36970]: pgmap v5967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10171.349005 secs 2024-07-26T09:45:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:41 smithi163 ceph-mon[36873]: pgmap v5967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10171.349005 secs 2024-07-26T09:45:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:45:42.068+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:45:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:45:42.068+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:45:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:45:42.068+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:45:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10189.088629 secs 2024-07-26T09:45:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10189.088629 secs 2024-07-26T09:45:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10189.088629 secs 2024-07-26T09:45:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:43 smithi042 ceph-mon[32775]: pgmap v5968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:43 smithi164 ceph-mon[36970]: pgmap v5968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:43 smithi163 ceph-mon[36873]: pgmap v5968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:45 smithi042 ceph-mon[32775]: pgmap v5969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:45 smithi164 ceph-mon[36970]: pgmap v5969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:45 smithi163 ceph-mon[36873]: pgmap v5969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10176.349165 secs 2024-07-26T09:45:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10176.349165 secs 2024-07-26T09:45:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10176.349165 secs 2024-07-26T09:45:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10194.088747 secs 2024-07-26T09:45:47.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:47 smithi164 ceph-mon[36970]: pgmap v5970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10194.088747 secs 2024-07-26T09:45:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:47 smithi163 ceph-mon[36873]: pgmap v5970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10194.088747 secs 2024-07-26T09:45:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:47 smithi042 ceph-mon[32775]: pgmap v5970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:49 smithi164 ceph-mon[36970]: pgmap v5971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:49 smithi163 ceph-mon[36873]: pgmap v5971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:49 smithi042 ceph-mon[32775]: pgmap v5971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:51 smithi164 ceph-mon[36970]: pgmap v5972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:51.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10181.349288 secs 2024-07-26T09:45:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:51 smithi163 ceph-mon[36873]: pgmap v5972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10181.349288 secs 2024-07-26T09:45:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:51 smithi042 ceph-mon[32775]: pgmap v5972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10181.349288 secs 2024-07-26T09:45:52.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10199.088870 secs 2024-07-26T09:45:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10199.088870 secs 2024-07-26T09:45:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10199.088870 secs 2024-07-26T09:45:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:53 smithi164 ceph-mon[36970]: pgmap v5973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:53 smithi163 ceph-mon[36873]: pgmap v5973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:53 smithi042 ceph-mon[32775]: pgmap v5973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:55 smithi164 ceph-mon[36970]: pgmap v5974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:55 smithi163 ceph-mon[36873]: pgmap v5974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:55 smithi042 ceph-mon[32775]: pgmap v5974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:45:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10186.349368 secs 2024-07-26T09:45:56.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10186.349368 secs 2024-07-26T09:45:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10186.349368 secs 2024-07-26T09:45:57.074 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:45:57.072+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:45:57.075 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:45:57.072+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:45:57.075 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:45:57.072+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:45:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10204.088991 secs 2024-07-26T09:45:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:57 smithi164 ceph-mon[36970]: pgmap v5975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10204.088991 secs 2024-07-26T09:45:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:57 smithi163 ceph-mon[36873]: pgmap v5975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10204.088991 secs 2024-07-26T09:45:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:57 smithi042 ceph-mon[32775]: pgmap v5975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:45:59 smithi164 ceph-mon[36970]: pgmap v5976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:45:59 smithi163 ceph-mon[36873]: pgmap v5976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:45:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:45:59 smithi042 ceph-mon[32775]: pgmap v5976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:01 smithi164 ceph-mon[36970]: pgmap v5977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:01.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10191.349494 secs 2024-07-26T09:46:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:01 smithi163 ceph-mon[36873]: pgmap v5977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10191.349494 secs 2024-07-26T09:46:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:01 smithi042 ceph-mon[32775]: pgmap v5977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10191.349494 secs 2024-07-26T09:46:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10209.089123 secs 2024-07-26T09:46:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:46:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10209.089123 secs 2024-07-26T09:46:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:46:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10209.089123 secs 2024-07-26T09:46:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:46:03.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:03 smithi164 ceph-mon[36970]: pgmap v5978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:03.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:46:03.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:46:03.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:03 smithi164 ceph-mon[36970]: pgmap v5979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:46:03.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:03 smithi164 ceph-mon[36970]: pgmap v5980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:46:03.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:46:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:03 smithi163 ceph-mon[36873]: pgmap v5978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:46:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:46:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:03 smithi163 ceph-mon[36873]: pgmap v5979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:46:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:03 smithi163 ceph-mon[36873]: pgmap v5980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:46:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:46:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:03 smithi042 ceph-mon[32775]: pgmap v5978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:46:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:46:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:03 smithi042 ceph-mon[32775]: pgmap v5979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:46:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:03 smithi042 ceph-mon[32775]: pgmap v5980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:46:03.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:46:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:05 smithi042 ceph-mon[32775]: pgmap v5981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:46:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:05 smithi164 ceph-mon[36970]: pgmap v5981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:46:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:05 smithi163 ceph-mon[36873]: pgmap v5981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:46:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10196.349573 secs 2024-07-26T09:46:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10214.089273 secs 2024-07-26T09:46:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10196.349573 secs 2024-07-26T09:46:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10214.089273 secs 2024-07-26T09:46:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10196.349573 secs 2024-07-26T09:46:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10214.089273 secs 2024-07-26T09:46:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:07 smithi042 ceph-mon[32775]: pgmap v5982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:46:08.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:07 smithi164 ceph-mon[36970]: pgmap v5982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:46:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:07 smithi163 ceph-mon[36873]: pgmap v5982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:46:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:09 smithi042 ceph-mon[32775]: pgmap v5983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:46:10.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:09 smithi164 ceph-mon[36970]: pgmap v5983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:46:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:09 smithi163 ceph-mon[36873]: pgmap v5983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:46:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:46:12.082+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:46:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:46:12.082+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:46:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:46:12.082+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:46:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:11 smithi042 ceph-mon[32775]: pgmap v5984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:46:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10201.349702 secs 2024-07-26T09:46:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10219.089391 secs 2024-07-26T09:46:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:11 smithi164 ceph-mon[36970]: pgmap v5984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:46:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10201.349702 secs 2024-07-26T09:46:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10219.089391 secs 2024-07-26T09:46:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:11 smithi163 ceph-mon[36873]: pgmap v5984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:46:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10201.349702 secs 2024-07-26T09:46:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10219.089391 secs 2024-07-26T09:46:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:13 smithi042 ceph-mon[32775]: pgmap v5985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:46:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:13 smithi164 ceph-mon[36970]: pgmap v5985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:46:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:13 smithi163 ceph-mon[36873]: pgmap v5985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:46:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:15 smithi042 ceph-mon[32775]: pgmap v5986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:15 smithi164 ceph-mon[36970]: pgmap v5986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:15 smithi163 ceph-mon[36873]: pgmap v5986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10206.349825 secs 2024-07-26T09:46:17.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10224.089531 secs 2024-07-26T09:46:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10206.349825 secs 2024-07-26T09:46:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10224.089531 secs 2024-07-26T09:46:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10206.349825 secs 2024-07-26T09:46:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10224.089531 secs 2024-07-26T09:46:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:17 smithi042 ceph-mon[32775]: pgmap v5987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:17 smithi164 ceph-mon[36970]: pgmap v5987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:17 smithi163 ceph-mon[36873]: pgmap v5987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:19 smithi042 ceph-mon[32775]: pgmap v5988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:46:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:19 smithi164 ceph-mon[36970]: pgmap v5988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:46:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:19 smithi163 ceph-mon[36873]: pgmap v5988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:46:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:21 smithi042 ceph-mon[32775]: pgmap v5989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10211.349895 secs 2024-07-26T09:46:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10229.089634 secs 2024-07-26T09:46:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:21 smithi164 ceph-mon[36970]: pgmap v5989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10211.349895 secs 2024-07-26T09:46:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10229.089634 secs 2024-07-26T09:46:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:21 smithi163 ceph-mon[36873]: pgmap v5989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10211.349895 secs 2024-07-26T09:46:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10229.089634 secs 2024-07-26T09:46:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:23 smithi042 ceph-mon[32775]: pgmap v5990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:23 smithi164 ceph-mon[36970]: pgmap v5990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:23 smithi163 ceph-mon[36873]: pgmap v5990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:25 smithi164 ceph-mon[36970]: pgmap v5991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:25 smithi163 ceph-mon[36873]: pgmap v5991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:25 smithi042 ceph-mon[32775]: pgmap v5991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:27.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:46:27.087+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:46:27.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:46:27.087+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:46:27.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:46:27.087+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:46:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10216.350021 secs 2024-07-26T09:46:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10234.089754 secs 2024-07-26T09:46:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10216.350021 secs 2024-07-26T09:46:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10234.089754 secs 2024-07-26T09:46:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10216.350021 secs 2024-07-26T09:46:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10234.089754 secs 2024-07-26T09:46:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:27 smithi164 ceph-mon[36970]: pgmap v5992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:27 smithi163 ceph-mon[36873]: pgmap v5992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:27 smithi042 ceph-mon[32775]: pgmap v5992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:28 smithi164 ceph-mon[36970]: pgmap v5993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:46:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:28 smithi163 ceph-mon[36873]: pgmap v5993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:46:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:28 smithi042 ceph-mon[32775]: pgmap v5993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:46:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:31 smithi042 ceph-mon[32775]: pgmap v5994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10221.350148 secs 2024-07-26T09:46:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10239.089894 secs 2024-07-26T09:46:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:31 smithi164 ceph-mon[36970]: pgmap v5994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10221.350148 secs 2024-07-26T09:46:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10239.089894 secs 2024-07-26T09:46:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:31 smithi163 ceph-mon[36873]: pgmap v5994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10221.350148 secs 2024-07-26T09:46:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10239.089894 secs 2024-07-26T09:46:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:33 smithi042 ceph-mon[32775]: pgmap v5995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:33 smithi164 ceph-mon[36970]: pgmap v5995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:33 smithi163 ceph-mon[36873]: pgmap v5995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:35 smithi042 ceph-mon[32775]: pgmap v5996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:35 smithi164 ceph-mon[36970]: pgmap v5996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:35 smithi163 ceph-mon[36873]: pgmap v5996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10226.350295 secs 2024-07-26T09:46:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10244.090033 secs 2024-07-26T09:46:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10226.350295 secs 2024-07-26T09:46:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10244.090033 secs 2024-07-26T09:46:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10226.350295 secs 2024-07-26T09:46:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10244.090033 secs 2024-07-26T09:46:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:37 smithi042 ceph-mon[32775]: pgmap v5997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:37 smithi164 ceph-mon[36970]: pgmap v5997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:37 smithi163 ceph-mon[36873]: pgmap v5997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:39 smithi042 ceph-mon[32775]: pgmap v5998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:46:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:39 smithi164 ceph-mon[36970]: pgmap v5998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:46:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:39 smithi163 ceph-mon[36873]: pgmap v5998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:46:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:46:42.097+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:46:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:46:42.097+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:46:42.099 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:46:42.097+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:46:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:41 smithi042 ceph-mon[32775]: pgmap v5999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10231.350396 secs 2024-07-26T09:46:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10249.090097 secs 2024-07-26T09:46:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:41 smithi164 ceph-mon[36970]: pgmap v5999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10231.350396 secs 2024-07-26T09:46:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10249.090097 secs 2024-07-26T09:46:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:41 smithi163 ceph-mon[36873]: pgmap v5999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10231.350396 secs 2024-07-26T09:46:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10249.090097 secs 2024-07-26T09:46:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:43 smithi042 ceph-mon[32775]: pgmap v6000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:43 smithi164 ceph-mon[36970]: pgmap v6000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:43 smithi163 ceph-mon[36873]: pgmap v6000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:45 smithi042 ceph-mon[32775]: pgmap v6001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:45 smithi164 ceph-mon[36970]: pgmap v6001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:45 smithi163 ceph-mon[36873]: pgmap v6001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10236.350464 secs 2024-07-26T09:46:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10254.090181 secs 2024-07-26T09:46:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10236.350464 secs 2024-07-26T09:46:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10254.090181 secs 2024-07-26T09:46:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10236.350464 secs 2024-07-26T09:46:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10254.090181 secs 2024-07-26T09:46:48.177 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:47 smithi042 ceph-mon[32775]: pgmap v6002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:47 smithi164 ceph-mon[36970]: pgmap v6002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:47 smithi163 ceph-mon[36873]: pgmap v6002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:48 smithi164 ceph-mon[36970]: pgmap v6003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:46:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:48 smithi163 ceph-mon[36873]: pgmap v6003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:46:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:48 smithi042 ceph-mon[32775]: pgmap v6003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:46:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:51 smithi042 ceph-mon[32775]: pgmap v6004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10241.350590 secs 2024-07-26T09:46:52.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10259.090302 secs 2024-07-26T09:46:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:51 smithi164 ceph-mon[36970]: pgmap v6004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10241.350590 secs 2024-07-26T09:46:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10259.090302 secs 2024-07-26T09:46:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:51 smithi163 ceph-mon[36873]: pgmap v6004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10241.350590 secs 2024-07-26T09:46:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10259.090302 secs 2024-07-26T09:46:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:53 smithi042 ceph-mon[32775]: pgmap v6005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:53 smithi164 ceph-mon[36970]: pgmap v6005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:53 smithi163 ceph-mon[36873]: pgmap v6005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:55 smithi042 ceph-mon[32775]: pgmap v6006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:55 smithi164 ceph-mon[36970]: pgmap v6006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:55 smithi163 ceph-mon[36873]: pgmap v6006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:46:57.103 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:46:57.102+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:46:57.104 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:46:57.102+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:46:57.104 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:46:57.102+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:46:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10246.350708 secs 2024-07-26T09:46:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10264.090399 secs 2024-07-26T09:46:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10246.350708 secs 2024-07-26T09:46:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10264.090399 secs 2024-07-26T09:46:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10246.350708 secs 2024-07-26T09:46:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10264.090399 secs 2024-07-26T09:46:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:57 smithi042 ceph-mon[32775]: pgmap v6007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:57 smithi164 ceph-mon[36970]: pgmap v6007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:57 smithi163 ceph-mon[36873]: pgmap v6007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:46:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:46:59.459+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:46:59.462 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:46:59.459+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:46:59.462 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:46:59.459+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:46:59.462 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:46:59.459+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:47:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:46:59 smithi042 ceph-mon[32775]: pgmap v6008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:47:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:46:59 smithi164 ceph-mon[36970]: pgmap v6008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:47:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:46:59 smithi163 ceph-mon[36873]: pgmap v6008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:47:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:01 smithi042 ceph-mon[32775]: pgmap v6009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10251.350771 secs 2024-07-26T09:47:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10269.090516 secs 2024-07-26T09:47:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:01 smithi164 ceph-mon[36970]: pgmap v6009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10251.350771 secs 2024-07-26T09:47:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10269.090516 secs 2024-07-26T09:47:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:01 smithi163 ceph-mon[36873]: pgmap v6009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10251.350771 secs 2024-07-26T09:47:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10269.090516 secs 2024-07-26T09:47:04.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:03 smithi164 ceph-mon[36970]: pgmap v6010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:04.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:47:04.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:47:04.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:47:04.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:47:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:03 smithi163 ceph-mon[36873]: pgmap v6010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:47:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:47:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:47:04.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:47:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:03 smithi042 ceph-mon[32775]: pgmap v6010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:47:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:47:04.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:47:04.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:47:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:04 smithi164 ceph-mon[36970]: pgmap v6011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:47:05.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:04 smithi164 ceph-mon[36970]: pgmap v6012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:47:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:04 smithi163 ceph-mon[36873]: pgmap v6011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:47:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:04 smithi163 ceph-mon[36873]: pgmap v6012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:47:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:04 smithi042 ceph-mon[32775]: pgmap v6011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:47:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:04 smithi042 ceph-mon[32775]: pgmap v6012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:47:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:06 smithi164 ceph-mon[36970]: pgmap v6013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:47:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10256.350850 secs 2024-07-26T09:47:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10274.090644 secs 2024-07-26T09:47:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:06 smithi163 ceph-mon[36873]: pgmap v6013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:47:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10256.350850 secs 2024-07-26T09:47:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10274.090644 secs 2024-07-26T09:47:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:06 smithi042 ceph-mon[32775]: pgmap v6013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:47:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10256.350850 secs 2024-07-26T09:47:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10274.090644 secs 2024-07-26T09:47:08.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:07 smithi164 ceph-mon[36970]: pgmap v6014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:47:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:07 smithi163 ceph-mon[36873]: pgmap v6014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:47:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:07 smithi042 ceph-mon[32775]: pgmap v6014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:47:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:10 smithi164 ceph-mon[36970]: pgmap v6015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:47:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:10 smithi163 ceph-mon[36873]: pgmap v6015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:47:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:10 smithi042 ceph-mon[32775]: pgmap v6015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:47:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10261.350986 secs 2024-07-26T09:47:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10261.350986 secs 2024-07-26T09:47:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10261.350986 secs 2024-07-26T09:47:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:47:12.112+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:47:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:47:12.112+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:47:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:47:12.112+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:47:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10279.090763 secs 2024-07-26T09:47:12.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:12 smithi164 ceph-mon[36970]: pgmap v6016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:47:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10279.090763 secs 2024-07-26T09:47:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:12 smithi163 ceph-mon[36873]: pgmap v6016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:47:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10279.090763 secs 2024-07-26T09:47:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:12 smithi042 ceph-mon[32775]: pgmap v6016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:47:14.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:14 smithi042 ceph-mon[32775]: pgmap v6017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:47:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:14 smithi164 ceph-mon[36970]: pgmap v6017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:47:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:14 smithi163 ceph-mon[36873]: pgmap v6017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:47:16.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:16 smithi164 ceph-mon[36970]: pgmap v6018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:16.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10266.351111 secs 2024-07-26T09:47:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:16 smithi163 ceph-mon[36873]: pgmap v6018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:16.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10266.351111 secs 2024-07-26T09:47:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:16 smithi042 ceph-mon[32775]: pgmap v6018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10266.351111 secs 2024-07-26T09:47:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10284.090882 secs 2024-07-26T09:47:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10284.090882 secs 2024-07-26T09:47:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10284.090882 secs 2024-07-26T09:47:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:18 smithi164 ceph-mon[36970]: pgmap v6019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:18 smithi163 ceph-mon[36873]: pgmap v6019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:18 smithi042 ceph-mon[32775]: pgmap v6019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:20.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:20 smithi164 ceph-mon[36970]: pgmap v6020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:20 smithi163 ceph-mon[36873]: pgmap v6020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:20 smithi042 ceph-mon[32775]: pgmap v6020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10271.351255 secs 2024-07-26T09:47:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10271.351255 secs 2024-07-26T09:47:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10271.351255 secs 2024-07-26T09:47:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10289.090972 secs 2024-07-26T09:47:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:22 smithi164 ceph-mon[36970]: pgmap v6021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10289.090972 secs 2024-07-26T09:47:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:22 smithi163 ceph-mon[36873]: pgmap v6021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10289.090972 secs 2024-07-26T09:47:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:22 smithi042 ceph-mon[32775]: pgmap v6021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:24 smithi164 ceph-mon[36970]: pgmap v6022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:47:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:24 smithi163 ceph-mon[36873]: pgmap v6022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:47:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:24 smithi042 ceph-mon[32775]: pgmap v6022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:47:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:26 smithi164 ceph-mon[36970]: pgmap v6023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:26.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10276.351375 secs 2024-07-26T09:47:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:26 smithi163 ceph-mon[36873]: pgmap v6023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10276.351375 secs 2024-07-26T09:47:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:26 smithi042 ceph-mon[32775]: pgmap v6023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10276.351375 secs 2024-07-26T09:47:27.119 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:47:27.117+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:47:27.119 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:47:27.117+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:47:27.119 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:47:27.117+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:47:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10294.091070 secs 2024-07-26T09:47:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10294.091070 secs 2024-07-26T09:47:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10294.091070 secs 2024-07-26T09:47:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:28 smithi164 ceph-mon[36970]: pgmap v6024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:28 smithi163 ceph-mon[36873]: pgmap v6024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:28 smithi042 ceph-mon[32775]: pgmap v6024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:30 smithi164 ceph-mon[36970]: pgmap v6025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:30 smithi163 ceph-mon[36873]: pgmap v6025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:30 smithi042 ceph-mon[32775]: pgmap v6025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10281.351500 secs 2024-07-26T09:47:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10281.351500 secs 2024-07-26T09:47:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10281.351500 secs 2024-07-26T09:47:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10299.091192 secs 2024-07-26T09:47:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:32 smithi163 ceph-mon[36873]: pgmap v6026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10299.091192 secs 2024-07-26T09:47:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:32 smithi042 ceph-mon[32775]: pgmap v6026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10299.091192 secs 2024-07-26T09:47:32.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:32 smithi164 ceph-mon[36970]: pgmap v6026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:34 smithi163 ceph-mon[36873]: pgmap v6027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:47:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:34 smithi042 ceph-mon[32775]: pgmap v6027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:47:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:34 smithi164 ceph-mon[36970]: pgmap v6027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:47:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:36 smithi163 ceph-mon[36873]: pgmap v6028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10286.351626 secs 2024-07-26T09:47:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:36 smithi042 ceph-mon[32775]: pgmap v6028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10286.351626 secs 2024-07-26T09:47:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:36 smithi164 ceph-mon[36970]: pgmap v6028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10286.351626 secs 2024-07-26T09:47:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10304.091301 secs 2024-07-26T09:47:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10304.091301 secs 2024-07-26T09:47:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10304.091301 secs 2024-07-26T09:47:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:38 smithi163 ceph-mon[36873]: pgmap v6029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:38 smithi042 ceph-mon[32775]: pgmap v6029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:38 smithi164 ceph-mon[36970]: pgmap v6029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:40 smithi163 ceph-mon[36873]: pgmap v6030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:40 smithi042 ceph-mon[32775]: pgmap v6030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:40.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:40 smithi164 ceph-mon[36970]: pgmap v6030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10291.351754 secs 2024-07-26T09:47:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10291.351754 secs 2024-07-26T09:47:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10291.351754 secs 2024-07-26T09:47:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:47:42.117+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:47:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:47:42.117+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:47:42.120 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:47:42.117+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:47:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10309.091421 secs 2024-07-26T09:47:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:42 smithi163 ceph-mon[36873]: pgmap v6031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10309.091421 secs 2024-07-26T09:47:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:42 smithi042 ceph-mon[32775]: pgmap v6031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10309.091421 secs 2024-07-26T09:47:42.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:42 smithi164 ceph-mon[36970]: pgmap v6031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:44 smithi163 ceph-mon[36873]: pgmap v6032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:47:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:44 smithi042 ceph-mon[32775]: pgmap v6032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:47:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:44 smithi164 ceph-mon[36970]: pgmap v6032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:47:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:46 smithi163 ceph-mon[36873]: pgmap v6033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10296.351878 secs 2024-07-26T09:47:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:46 smithi042 ceph-mon[32775]: pgmap v6033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10296.351878 secs 2024-07-26T09:47:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:46 smithi164 ceph-mon[36970]: pgmap v6033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10296.351878 secs 2024-07-26T09:47:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10314.091541 secs 2024-07-26T09:47:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10314.091541 secs 2024-07-26T09:47:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10314.091541 secs 2024-07-26T09:47:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:48 smithi042 ceph-mon[32775]: pgmap v6034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:48 smithi164 ceph-mon[36970]: pgmap v6034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:48 smithi163 ceph-mon[36873]: pgmap v6034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:50 smithi042 ceph-mon[32775]: pgmap v6035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:50.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:50 smithi164 ceph-mon[36970]: pgmap v6035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:51.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:50 smithi163 ceph-mon[36873]: pgmap v6035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10301.352004 secs 2024-07-26T09:47:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10301.352004 secs 2024-07-26T09:47:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10301.352004 secs 2024-07-26T09:47:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10319.091679 secs 2024-07-26T09:47:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:52 smithi042 ceph-mon[32775]: pgmap v6036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:52.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10319.091679 secs 2024-07-26T09:47:52.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:52 smithi164 ceph-mon[36970]: pgmap v6036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10319.091679 secs 2024-07-26T09:47:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:52 smithi163 ceph-mon[36873]: pgmap v6036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:54 smithi042 ceph-mon[32775]: pgmap v6037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:47:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:54 smithi164 ceph-mon[36970]: pgmap v6037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:47:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:54 smithi163 ceph-mon[36873]: pgmap v6037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:47:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:56 smithi042 ceph-mon[32775]: pgmap v6038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10306.352127 secs 2024-07-26T09:47:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:56 smithi164 ceph-mon[36970]: pgmap v6038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10306.352127 secs 2024-07-26T09:47:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:56 smithi163 ceph-mon[36873]: pgmap v6038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:47:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10306.352127 secs 2024-07-26T09:47:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:47:57.122+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:47:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:47:57.122+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:47:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:47:57.122+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:47:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10324.091830 secs 2024-07-26T09:47:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10324.091830 secs 2024-07-26T09:47:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10324.091830 secs 2024-07-26T09:47:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:47:58 smithi042 ceph-mon[32775]: pgmap v6039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:47:58 smithi164 ceph-mon[36970]: pgmap v6039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:47:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:47:58 smithi163 ceph-mon[36873]: pgmap v6039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:00 smithi042 ceph-mon[32775]: pgmap v6040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:00 smithi164 ceph-mon[36970]: pgmap v6040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:00 smithi163 ceph-mon[36873]: pgmap v6040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10311.352254 secs 2024-07-26T09:48:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10311.352254 secs 2024-07-26T09:48:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10311.352254 secs 2024-07-26T09:48:02.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10329.091955 secs 2024-07-26T09:48:02.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:02 smithi042 ceph-mon[32775]: pgmap v6041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10329.091955 secs 2024-07-26T09:48:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:02 smithi164 ceph-mon[36970]: pgmap v6041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10329.091955 secs 2024-07-26T09:48:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:02 smithi163 ceph-mon[36873]: pgmap v6041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:48:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:48:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:48:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:04 smithi042 ceph-mon[32775]: pgmap v6042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:48:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:04 smithi164 ceph-mon[36970]: pgmap v6042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:48:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:04 smithi163 ceph-mon[36873]: pgmap v6042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:48:06.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:06 smithi042 ceph-mon[32775]: pgmap v6043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:06.761 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10316.352380 secs 2024-07-26T09:48:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:06 smithi164 ceph-mon[36970]: pgmap v6043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10316.352380 secs 2024-07-26T09:48:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:06 smithi163 ceph-mon[36873]: pgmap v6043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10316.352380 secs 2024-07-26T09:48:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10334.092052 secs 2024-07-26T09:48:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:07 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:48:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:07 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:48:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:07 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:48:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:07 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:48:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:07 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:48:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10334.092052 secs 2024-07-26T09:48:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:07 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:48:07.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:07 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:48:07.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:07 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:48:07.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:07 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:48:07.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:07 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:48:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10334.092052 secs 2024-07-26T09:48:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:07 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:48:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:07 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:48:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:07 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:48:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:07 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:48:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:07 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:48:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:08 smithi042 ceph-mon[32775]: pgmap v6044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:08 smithi042 ceph-mon[32775]: pgmap v6045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:48:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:08 smithi164 ceph-mon[36970]: pgmap v6044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:08.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:08 smithi164 ceph-mon[36970]: pgmap v6045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:48:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:08 smithi163 ceph-mon[36873]: pgmap v6044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:08 smithi163 ceph-mon[36873]: pgmap v6045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:48:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:10 smithi042 ceph-mon[32775]: pgmap v6046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:48:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:10 smithi164 ceph-mon[36970]: pgmap v6046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:48:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:10 smithi163 ceph-mon[36873]: pgmap v6046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:48:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10321.352500 secs 2024-07-26T09:48:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10321.352500 secs 2024-07-26T09:48:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10321.352500 secs 2024-07-26T09:48:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:48:12.126+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:48:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:48:12.126+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:48:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:48:12.126+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:48:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:12 smithi042 ceph-mon[32775]: pgmap v6047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:48:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10339.092179 secs 2024-07-26T09:48:12.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:12 smithi163 ceph-mon[36873]: pgmap v6047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:48:12.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10339.092179 secs 2024-07-26T09:48:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:12 smithi164 ceph-mon[36970]: pgmap v6047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:48:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10339.092179 secs 2024-07-26T09:48:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:14 smithi042 ceph-mon[32775]: pgmap v6048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:48:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:14 smithi164 ceph-mon[36970]: pgmap v6048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:48:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:14 smithi163 ceph-mon[36873]: pgmap v6048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:48:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:16 smithi042 ceph-mon[32775]: pgmap v6049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:48:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10326.352652 secs 2024-07-26T09:48:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:16 smithi164 ceph-mon[36970]: pgmap v6049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:48:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10326.352652 secs 2024-07-26T09:48:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:16 smithi163 ceph-mon[36873]: pgmap v6049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:48:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10326.352652 secs 2024-07-26T09:48:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10344.092297 secs 2024-07-26T09:48:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10344.092297 secs 2024-07-26T09:48:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10344.092297 secs 2024-07-26T09:48:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:18 smithi042 ceph-mon[32775]: pgmap v6050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:48:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:18 smithi164 ceph-mon[36970]: pgmap v6050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:48:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:18 smithi163 ceph-mon[36873]: pgmap v6050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:48:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:20 smithi042 ceph-mon[32775]: pgmap v6051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:20 smithi164 ceph-mon[36970]: pgmap v6051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:20 smithi163 ceph-mon[36873]: pgmap v6051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:21.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10331.352795 secs 2024-07-26T09:48:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10331.352795 secs 2024-07-26T09:48:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10331.352795 secs 2024-07-26T09:48:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:22 smithi042 ceph-mon[32775]: pgmap v6052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10349.092455 secs 2024-07-26T09:48:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:22 smithi164 ceph-mon[36970]: pgmap v6052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:22.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10349.092455 secs 2024-07-26T09:48:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:22 smithi163 ceph-mon[36873]: pgmap v6052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10349.092455 secs 2024-07-26T09:48:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:24 smithi042 ceph-mon[32775]: pgmap v6053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:24 smithi164 ceph-mon[36970]: pgmap v6053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:24 smithi163 ceph-mon[36873]: pgmap v6053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:26 smithi042 ceph-mon[32775]: pgmap v6054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10336.352920 secs 2024-07-26T09:48:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:26 smithi164 ceph-mon[36970]: pgmap v6054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10336.352920 secs 2024-07-26T09:48:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:26 smithi163 ceph-mon[36873]: pgmap v6054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10336.352920 secs 2024-07-26T09:48:27.133 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:48:27.131+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:48:27.133 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:48:27.131+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:48:27.133 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:48:27.131+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:48:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10354.092572 secs 2024-07-26T09:48:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10354.092572 secs 2024-07-26T09:48:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10354.092572 secs 2024-07-26T09:48:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:28 smithi042 ceph-mon[32775]: pgmap v6055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:28 smithi164 ceph-mon[36970]: pgmap v6055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:28 smithi163 ceph-mon[36873]: pgmap v6055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:30 smithi042 ceph-mon[32775]: pgmap v6056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:30 smithi164 ceph-mon[36970]: pgmap v6056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:30 smithi163 ceph-mon[36873]: pgmap v6056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10341.353047 secs 2024-07-26T09:48:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10359.092694 secs 2024-07-26T09:48:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10341.353047 secs 2024-07-26T09:48:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10359.092694 secs 2024-07-26T09:48:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10341.353047 secs 2024-07-26T09:48:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10359.092694 secs 2024-07-26T09:48:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:32 smithi042 ceph-mon[32775]: pgmap v6057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:32.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:32 smithi164 ceph-mon[36970]: pgmap v6057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:32 smithi163 ceph-mon[36873]: pgmap v6057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:34 smithi042 ceph-mon[32775]: pgmap v6058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:34 smithi164 ceph-mon[36970]: pgmap v6058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:34 smithi163 ceph-mon[36873]: pgmap v6058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:36 smithi042 ceph-mon[32775]: pgmap v6059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10346.353163 secs 2024-07-26T09:48:36.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10364.092832 secs 2024-07-26T09:48:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:36 smithi164 ceph-mon[36970]: pgmap v6059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10346.353163 secs 2024-07-26T09:48:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10364.092832 secs 2024-07-26T09:48:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:36 smithi163 ceph-mon[36873]: pgmap v6059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10346.353163 secs 2024-07-26T09:48:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10364.092832 secs 2024-07-26T09:48:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:38 smithi042 ceph-mon[32775]: pgmap v6060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:38 smithi164 ceph-mon[36970]: pgmap v6060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:38 smithi163 ceph-mon[36873]: pgmap v6060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:40 smithi042 ceph-mon[32775]: pgmap v6061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:40 smithi164 ceph-mon[36970]: pgmap v6061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:40 smithi163 ceph-mon[36873]: pgmap v6061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10351.353293 secs 2024-07-26T09:48:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10369.092962 secs 2024-07-26T09:48:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10351.353293 secs 2024-07-26T09:48:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10369.092962 secs 2024-07-26T09:48:42.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:48:42.137+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:48:42.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:48:42.137+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:48:42.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:48:42.137+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:48:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10351.353293 secs 2024-07-26T09:48:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10369.092962 secs 2024-07-26T09:48:42.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:42 smithi163 ceph-mon[36873]: pgmap v6062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:42 smithi164 ceph-mon[36970]: pgmap v6062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:42 smithi042 ceph-mon[32775]: pgmap v6062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:44.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:44 smithi042 ceph-mon[32775]: pgmap v6063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:48:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:44 smithi164 ceph-mon[36970]: pgmap v6063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:48:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:44 smithi163 ceph-mon[36873]: pgmap v6063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:48:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:46 smithi164 ceph-mon[36970]: pgmap v6064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10356.353416 secs 2024-07-26T09:48:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10374.093111 secs 2024-07-26T09:48:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:46 smithi163 ceph-mon[36873]: pgmap v6064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10356.353416 secs 2024-07-26T09:48:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10374.093111 secs 2024-07-26T09:48:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:46 smithi042 ceph-mon[32775]: pgmap v6064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10356.353416 secs 2024-07-26T09:48:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10374.093111 secs 2024-07-26T09:48:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:48 smithi164 ceph-mon[36970]: pgmap v6065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:48 smithi163 ceph-mon[36873]: pgmap v6065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:48 smithi042 ceph-mon[32775]: pgmap v6065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:50 smithi164 ceph-mon[36970]: pgmap v6066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:50 smithi163 ceph-mon[36873]: pgmap v6066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:50 smithi042 ceph-mon[32775]: pgmap v6066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10361.353534 secs 2024-07-26T09:48:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10379.093233 secs 2024-07-26T09:48:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10361.353534 secs 2024-07-26T09:48:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10379.093233 secs 2024-07-26T09:48:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10361.353534 secs 2024-07-26T09:48:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10379.093233 secs 2024-07-26T09:48:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:52 smithi164 ceph-mon[36970]: pgmap v6067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:52 smithi163 ceph-mon[36873]: pgmap v6067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:53.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:52 smithi042 ceph-mon[32775]: pgmap v6067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:54 smithi164 ceph-mon[36970]: pgmap v6068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:48:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:54 smithi163 ceph-mon[36873]: pgmap v6068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:48:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:54 smithi042 ceph-mon[32775]: pgmap v6068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:48:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:56 smithi164 ceph-mon[36970]: pgmap v6069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10366.353661 secs 2024-07-26T09:48:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10384.093355 secs 2024-07-26T09:48:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:56 smithi163 ceph-mon[36873]: pgmap v6069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10366.353661 secs 2024-07-26T09:48:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10384.093355 secs 2024-07-26T09:48:57.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:48:57.142+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:48:57.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:48:57.142+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:48:57.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:48:57.142+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:48:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:56 smithi042 ceph-mon[32775]: pgmap v6069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:48:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10366.353661 secs 2024-07-26T09:48:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10384.093355 secs 2024-07-26T09:48:58.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:48:58 smithi164 ceph-mon[36970]: pgmap v6070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:48:58 smithi163 ceph-mon[36873]: pgmap v6070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:48:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:48:58 smithi042 ceph-mon[32775]: pgmap v6070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:00 smithi164 ceph-mon[36970]: pgmap v6071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:00 smithi163 ceph-mon[36873]: pgmap v6071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:00 smithi042 ceph-mon[32775]: pgmap v6071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10371.353780 secs 2024-07-26T09:49:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10389.093476 secs 2024-07-26T09:49:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10371.353780 secs 2024-07-26T09:49:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10389.093476 secs 2024-07-26T09:49:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10371.353780 secs 2024-07-26T09:49:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10389.093476 secs 2024-07-26T09:49:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:02 smithi164 ceph-mon[36970]: pgmap v6072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:02 smithi163 ceph-mon[36873]: pgmap v6072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:02 smithi042 ceph-mon[32775]: pgmap v6072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:04 smithi164 ceph-mon[36970]: pgmap v6073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:49:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:04 smithi163 ceph-mon[36873]: pgmap v6073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:49:05.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:04 smithi042 ceph-mon[32775]: pgmap v6073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:49:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:06 smithi164 ceph-mon[36970]: pgmap v6074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10376.353929 secs 2024-07-26T09:49:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10394.093620 secs 2024-07-26T09:49:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:06 smithi163 ceph-mon[36873]: pgmap v6074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10376.353929 secs 2024-07-26T09:49:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10394.093620 secs 2024-07-26T09:49:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:06 smithi042 ceph-mon[32775]: pgmap v6074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10376.353929 secs 2024-07-26T09:49:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10394.093620 secs 2024-07-26T09:49:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:07 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:49:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:07 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:49:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:07 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:49:08.917 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:08 smithi164 ceph-mon[36970]: pgmap v6075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:08.917 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:08 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:08.917 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:08 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:08.917 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:08 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:08.917 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:08 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:08.918 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:08 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:49:08.918 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:08 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:49:08.977 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:08 smithi042 ceph-mon[32775]: pgmap v6075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:08.977 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:08 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:08.977 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:08 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:08.977 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:08 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:08.978 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:08 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:08.978 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:08 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:49:08.978 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:08 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:49:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:08 smithi163 ceph-mon[36873]: pgmap v6075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:08 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:08 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:08 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:08 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:09.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:08 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:49:09.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:08 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:49:10.894 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:10 smithi164 ceph-mon[36970]: pgmap v6076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:10 smithi163 ceph-mon[36873]: pgmap v6076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:10 smithi042 ceph-mon[32775]: pgmap v6076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:12.151+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:49:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:12.151+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:49:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:12.151+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:49:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10381.354004 secs 2024-07-26T09:49:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:49:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:49:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:49:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:49:12.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:49:12.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:49:12.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:49:12.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:49:12.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:49:12.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:49:12.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: pgmap v6077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:12.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: pgmap v6078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:49:12.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:49:12.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10399.093692 secs 2024-07-26T09:49:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10381.354004 secs 2024-07-26T09:49:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:49:12.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:49:12.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:49:12.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:49:12.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:49:12.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:49:12.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:49:12.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:49:12.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:49:12.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:49:12.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: pgmap v6077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:12.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: pgmap v6078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:49:12.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:49:12.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10399.093692 secs 2024-07-26T09:49:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10381.354004 secs 2024-07-26T09:49:12.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:49:12.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:49:12.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:49:12.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:49:12.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:49:12.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:49:12.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:49:12.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:49:12.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:49:12.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:49:12.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:49:12.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: pgmap v6077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:12.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: pgmap v6078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:49:12.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:49:12.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10399.093692 secs 2024-07-26T09:49:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:14 smithi042 ceph-mon[32775]: pgmap v6079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:49:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:14 smithi164 ceph-mon[36970]: pgmap v6079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:49:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:14.480+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:49:14.482 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:14.480+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:49:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:14.481+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:49:14.483 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:14.481+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:49:14.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:14 smithi163 ceph-mon[36873]: pgmap v6079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:49:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:16 smithi042 ceph-mon[32775]: pgmap v6080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:49:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:16 smithi164 ceph-mon[36970]: pgmap v6080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:49:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:16 smithi163 ceph-mon[36873]: pgmap v6080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:49:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10386.354130 secs 2024-07-26T09:49:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10404.093808 secs 2024-07-26T09:49:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10386.354130 secs 2024-07-26T09:49:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10404.093808 secs 2024-07-26T09:49:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10386.354130 secs 2024-07-26T09:49:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10404.093808 secs 2024-07-26T09:49:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:18 smithi042 ceph-mon[32775]: pgmap v6081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:49:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:18 smithi164 ceph-mon[36970]: pgmap v6081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:49:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:18 smithi163 ceph-mon[36873]: pgmap v6081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:49:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:20 smithi042 ceph-mon[32775]: pgmap v6082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:49:20.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:20 smithi164 ceph-mon[36970]: pgmap v6082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:49:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:20 smithi163 ceph-mon[36873]: pgmap v6082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T09:49:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10391.354260 secs 2024-07-26T09:49:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:22 smithi042 ceph-mon[32775]: pgmap v6083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:49:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10409.093951 secs 2024-07-26T09:49:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10391.354260 secs 2024-07-26T09:49:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:22 smithi164 ceph-mon[36970]: pgmap v6083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:49:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10409.093951 secs 2024-07-26T09:49:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10391.354260 secs 2024-07-26T09:49:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:22 smithi163 ceph-mon[36873]: pgmap v6083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:49:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10409.093951 secs 2024-07-26T09:49:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:24 smithi042 ceph-mon[32775]: pgmap v6084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:24 smithi164 ceph-mon[36970]: pgmap v6084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:24 smithi163 ceph-mon[36873]: pgmap v6084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:26 smithi042 ceph-mon[32775]: pgmap v6085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:26 smithi164 ceph-mon[36970]: pgmap v6085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:26 smithi163 ceph-mon[36873]: pgmap v6085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:27.157 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:27.155+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:49:27.157 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:27.155+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:49:27.157 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:27.155+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:49:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10396.354395 secs 2024-07-26T09:49:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10414.094050 secs 2024-07-26T09:49:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10396.354395 secs 2024-07-26T09:49:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10414.094050 secs 2024-07-26T09:49:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10396.354395 secs 2024-07-26T09:49:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10414.094050 secs 2024-07-26T09:49:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:28 smithi164 ceph-mon[36970]: pgmap v6086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:28 smithi163 ceph-mon[36873]: pgmap v6086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:28 smithi042 ceph-mon[32775]: pgmap v6086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:30 smithi164 ceph-mon[36970]: pgmap v6087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:30.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:30 smithi163 ceph-mon[36873]: pgmap v6087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:30 smithi042 ceph-mon[32775]: pgmap v6087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10401.354533 secs 2024-07-26T09:49:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:32 smithi164 ceph-mon[36970]: pgmap v6088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10419.094175 secs 2024-07-26T09:49:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10401.354533 secs 2024-07-26T09:49:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:32 smithi163 ceph-mon[36873]: pgmap v6088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10419.094175 secs 2024-07-26T09:49:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10401.354533 secs 2024-07-26T09:49:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:32 smithi042 ceph-mon[32775]: pgmap v6088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10419.094175 secs 2024-07-26T09:49:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:34 smithi164 ceph-mon[36970]: pgmap v6089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:34 smithi163 ceph-mon[36873]: pgmap v6089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:34 smithi042 ceph-mon[32775]: pgmap v6089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:36 smithi164 ceph-mon[36970]: pgmap v6090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:36 smithi163 ceph-mon[36873]: pgmap v6090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:36 smithi042 ceph-mon[32775]: pgmap v6090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10406.354668 secs 2024-07-26T09:49:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10424.094296 secs 2024-07-26T09:49:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10406.354668 secs 2024-07-26T09:49:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10424.094296 secs 2024-07-26T09:49:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10406.354668 secs 2024-07-26T09:49:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10424.094296 secs 2024-07-26T09:49:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:38 smithi164 ceph-mon[36970]: pgmap v6091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:38 smithi163 ceph-mon[36873]: pgmap v6091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:38 smithi042 ceph-mon[32775]: pgmap v6091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:40 smithi164 ceph-mon[36970]: pgmap v6092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:40 smithi163 ceph-mon[36873]: pgmap v6092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:40 smithi042 ceph-mon[32775]: pgmap v6092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:42.165+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:49:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:42.165+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:49:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:42.165+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:49:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10411.354807 secs 2024-07-26T09:49:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:42 smithi164 ceph-mon[36970]: pgmap v6093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10429.094423 secs 2024-07-26T09:49:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10411.354807 secs 2024-07-26T09:49:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:42 smithi163 ceph-mon[36873]: pgmap v6093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:42.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10429.094423 secs 2024-07-26T09:49:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10411.354807 secs 2024-07-26T09:49:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:42 smithi042 ceph-mon[32775]: pgmap v6093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10429.094423 secs 2024-07-26T09:49:44.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:44 smithi042 ceph-mon[32775]: pgmap v6094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:44 smithi164 ceph-mon[36970]: pgmap v6094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:44 smithi163 ceph-mon[36873]: pgmap v6094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:46.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:46 smithi164 ceph-mon[36970]: pgmap v6095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:46 smithi163 ceph-mon[36873]: pgmap v6095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:46 smithi042 ceph-mon[32775]: pgmap v6095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10416.354944 secs 2024-07-26T09:49:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10434.094543 secs 2024-07-26T09:49:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10416.354944 secs 2024-07-26T09:49:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10434.094543 secs 2024-07-26T09:49:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10416.354944 secs 2024-07-26T09:49:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10434.094543 secs 2024-07-26T09:49:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:48 smithi164 ceph-mon[36970]: pgmap v6096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:48 smithi163 ceph-mon[36873]: pgmap v6096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:48 smithi042 ceph-mon[32775]: pgmap v6096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:50 smithi164 ceph-mon[36970]: pgmap v6097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:50 smithi163 ceph-mon[36873]: pgmap v6097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:50 smithi042 ceph-mon[32775]: pgmap v6097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10421.355040 secs 2024-07-26T09:49:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:52 smithi164 ceph-mon[36970]: pgmap v6098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10439.094672 secs 2024-07-26T09:49:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10421.355040 secs 2024-07-26T09:49:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:52 smithi163 ceph-mon[36873]: pgmap v6098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10439.094672 secs 2024-07-26T09:49:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10421.355040 secs 2024-07-26T09:49:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:52 smithi042 ceph-mon[32775]: pgmap v6098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10439.094672 secs 2024-07-26T09:49:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:54 smithi164 ceph-mon[36970]: pgmap v6099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:54 smithi163 ceph-mon[36873]: pgmap v6099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:54 smithi042 ceph-mon[32775]: pgmap v6099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:56.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:56 smithi164 ceph-mon[36970]: pgmap v6100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:56 smithi163 ceph-mon[36873]: pgmap v6100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:56 smithi042 ceph-mon[32775]: pgmap v6100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:49:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:57.170+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:49:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:57.170+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:49:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:57.170+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:49:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10426.355164 secs 2024-07-26T09:49:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10444.094820 secs 2024-07-26T09:49:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10426.355164 secs 2024-07-26T09:49:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10444.094820 secs 2024-07-26T09:49:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10426.355164 secs 2024-07-26T09:49:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10444.094820 secs 2024-07-26T09:49:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:49:58 smithi164 ceph-mon[36970]: pgmap v6101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:49:58 smithi163 ceph-mon[36873]: pgmap v6101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:49:58 smithi042 ceph-mon[32775]: pgmap v6101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:49:59.490 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:59.488+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:49:59.490 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:59.488+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:49:59.491 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:59.488+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:49:59.491 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:49:59.488+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:50:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:00 smithi164 ceph-mon[36970]: pgmap v6102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:00.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:50:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:00 smithi163 ceph-mon[36873]: pgmap v6102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:50:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:00 smithi042 ceph-mon[32775]: pgmap v6102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T09:50:02.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10431.355292 secs 2024-07-26T09:50:02.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:02 smithi164 ceph-mon[36970]: pgmap v6103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:02.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10449.094975 secs 2024-07-26T09:50:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10431.355292 secs 2024-07-26T09:50:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:02 smithi163 ceph-mon[36873]: pgmap v6103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10449.094975 secs 2024-07-26T09:50:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10431.355292 secs 2024-07-26T09:50:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:02 smithi042 ceph-mon[32775]: pgmap v6103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:02.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10449.094975 secs 2024-07-26T09:50:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:04 smithi163 ceph-mon[36873]: pgmap v6104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:04 smithi042 ceph-mon[32775]: pgmap v6104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:04 smithi164 ceph-mon[36970]: pgmap v6104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:06 smithi163 ceph-mon[36873]: pgmap v6105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:06 smithi042 ceph-mon[32775]: pgmap v6105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:06 smithi164 ceph-mon[36970]: pgmap v6105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10436.355417 secs 2024-07-26T09:50:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10454.095043 secs 2024-07-26T09:50:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10436.355417 secs 2024-07-26T09:50:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10454.095043 secs 2024-07-26T09:50:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10436.355417 secs 2024-07-26T09:50:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10454.095043 secs 2024-07-26T09:50:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:08 smithi163 ceph-mon[36873]: pgmap v6106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:08 smithi042 ceph-mon[32775]: pgmap v6106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:08 smithi164 ceph-mon[36970]: pgmap v6106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:10 smithi163 ceph-mon[36873]: pgmap v6107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:10 smithi042 ceph-mon[32775]: pgmap v6107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:10 smithi164 ceph-mon[36970]: pgmap v6107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:50:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:50:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:50:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:50:12.174+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:50:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:50:12.174+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:50:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:50:12.174+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:50:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10441.355541 secs 2024-07-26T09:50:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:12 smithi163 ceph-mon[36873]: pgmap v6108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10459.095120 secs 2024-07-26T09:50:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:50:12.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:50:12.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:12 smithi163 ceph-mon[36873]: pgmap v6109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:50:12.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:12 smithi163 ceph-mon[36873]: pgmap v6110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:50:12.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:50:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10441.355541 secs 2024-07-26T09:50:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:12 smithi042 ceph-mon[32775]: pgmap v6108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10459.095120 secs 2024-07-26T09:50:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:50:12.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:50:12.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:12 smithi042 ceph-mon[32775]: pgmap v6109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:50:12.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:12 smithi042 ceph-mon[32775]: pgmap v6110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:50:12.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:50:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10441.355541 secs 2024-07-26T09:50:12.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:12 smithi164 ceph-mon[36970]: pgmap v6108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:12.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10459.095120 secs 2024-07-26T09:50:12.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:50:12.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:50:12.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:12 smithi164 ceph-mon[36970]: pgmap v6109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:50:12.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:12 smithi164 ceph-mon[36970]: pgmap v6110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:50:12.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:50:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:14 smithi164 ceph-mon[36970]: pgmap v6111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:50:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:14 smithi163 ceph-mon[36873]: pgmap v6111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:50:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:14 smithi042 ceph-mon[32775]: pgmap v6111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:50:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:16 smithi164 ceph-mon[36970]: pgmap v6112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:50:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10446.355684 secs 2024-07-26T09:50:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10464.095185 secs 2024-07-26T09:50:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:16 smithi163 ceph-mon[36873]: pgmap v6112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:50:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10446.355684 secs 2024-07-26T09:50:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10464.095185 secs 2024-07-26T09:50:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:16 smithi042 ceph-mon[32775]: pgmap v6112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:50:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10446.355684 secs 2024-07-26T09:50:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10464.095185 secs 2024-07-26T09:50:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:18 smithi164 ceph-mon[36970]: pgmap v6113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:50:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:18 smithi163 ceph-mon[36873]: pgmap v6113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:50:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:18 smithi042 ceph-mon[32775]: pgmap v6113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:50:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:20 smithi164 ceph-mon[36970]: pgmap v6114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:50:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:20 smithi163 ceph-mon[36873]: pgmap v6114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:50:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:20 smithi042 ceph-mon[32775]: pgmap v6114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:50:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10451.355788 secs 2024-07-26T09:50:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10469.095310 secs 2024-07-26T09:50:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10451.355788 secs 2024-07-26T09:50:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10469.095310 secs 2024-07-26T09:50:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10451.355788 secs 2024-07-26T09:50:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10469.095310 secs 2024-07-26T09:50:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:22 smithi164 ceph-mon[36970]: pgmap v6115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:50:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:22 smithi163 ceph-mon[36873]: pgmap v6115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:50:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:22 smithi042 ceph-mon[32775]: pgmap v6115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:50:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:24 smithi164 ceph-mon[36970]: pgmap v6116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:50:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:24 smithi163 ceph-mon[36873]: pgmap v6116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:50:25.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:24 smithi042 ceph-mon[32775]: pgmap v6116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:50:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:26 smithi164 ceph-mon[36970]: pgmap v6117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10456.355905 secs 2024-07-26T09:50:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10474.095461 secs 2024-07-26T09:50:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:26 smithi163 ceph-mon[36873]: pgmap v6117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10456.355905 secs 2024-07-26T09:50:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10474.095461 secs 2024-07-26T09:50:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:26 smithi042 ceph-mon[32775]: pgmap v6117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10456.355905 secs 2024-07-26T09:50:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10474.095461 secs 2024-07-26T09:50:27.181 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:50:27.179+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:50:27.181 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:50:27.179+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:50:27.181 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:50:27.179+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:50:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:28 smithi164 ceph-mon[36970]: pgmap v6118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:28 smithi163 ceph-mon[36873]: pgmap v6118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:28 smithi042 ceph-mon[32775]: pgmap v6118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:30 smithi164 ceph-mon[36970]: pgmap v6119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:30 smithi163 ceph-mon[36873]: pgmap v6119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:30 smithi042 ceph-mon[32775]: pgmap v6119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10461.356027 secs 2024-07-26T09:50:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10479.095589 secs 2024-07-26T09:50:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10461.356027 secs 2024-07-26T09:50:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10479.095589 secs 2024-07-26T09:50:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10461.356027 secs 2024-07-26T09:50:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10479.095589 secs 2024-07-26T09:50:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:32 smithi164 ceph-mon[36970]: pgmap v6120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:32 smithi163 ceph-mon[36873]: pgmap v6120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:32 smithi042 ceph-mon[32775]: pgmap v6120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:34 smithi163 ceph-mon[36873]: pgmap v6121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:50:35.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:34 smithi042 ceph-mon[32775]: pgmap v6121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:50:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:34 smithi164 ceph-mon[36970]: pgmap v6121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:50:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:36 smithi163 ceph-mon[36873]: pgmap v6122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:37.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10466.356156 secs 2024-07-26T09:50:37.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10484.095701 secs 2024-07-26T09:50:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:36 smithi042 ceph-mon[32775]: pgmap v6122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10466.356156 secs 2024-07-26T09:50:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10484.095701 secs 2024-07-26T09:50:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:36 smithi164 ceph-mon[36970]: pgmap v6122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10466.356156 secs 2024-07-26T09:50:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10484.095701 secs 2024-07-26T09:50:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:38 smithi164 ceph-mon[36970]: pgmap v6123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:38 smithi163 ceph-mon[36873]: pgmap v6123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:38 smithi042 ceph-mon[32775]: pgmap v6123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:40 smithi164 ceph-mon[36970]: pgmap v6124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:40 smithi163 ceph-mon[36873]: pgmap v6124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:41.137 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:40 smithi042 ceph-mon[32775]: pgmap v6124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10471.356287 secs 2024-07-26T09:50:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10489.095823 secs 2024-07-26T09:50:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10471.356287 secs 2024-07-26T09:50:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10489.095823 secs 2024-07-26T09:50:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10471.356287 secs 2024-07-26T09:50:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10489.095823 secs 2024-07-26T09:50:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:50:42.190+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:50:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:50:42.190+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:50:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:50:42.190+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:50:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:42 smithi164 ceph-mon[36970]: pgmap v6125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:42 smithi163 ceph-mon[36873]: pgmap v6125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:42 smithi042 ceph-mon[32775]: pgmap v6125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:44 smithi164 ceph-mon[36970]: pgmap v6126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:50:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:44 smithi163 ceph-mon[36873]: pgmap v6126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:50:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:44 smithi042 ceph-mon[32775]: pgmap v6126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:50:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:46 smithi164 ceph-mon[36970]: pgmap v6127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10476.356373 secs 2024-07-26T09:50:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10494.095981 secs 2024-07-26T09:50:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:46 smithi163 ceph-mon[36873]: pgmap v6127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10476.356373 secs 2024-07-26T09:50:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10494.095981 secs 2024-07-26T09:50:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:46 smithi042 ceph-mon[32775]: pgmap v6127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10476.356373 secs 2024-07-26T09:50:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10494.095981 secs 2024-07-26T09:50:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:48 smithi164 ceph-mon[36970]: pgmap v6128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:48 smithi163 ceph-mon[36873]: pgmap v6128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:48 smithi042 ceph-mon[32775]: pgmap v6128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:50 smithi164 ceph-mon[36970]: pgmap v6129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:50 smithi163 ceph-mon[36873]: pgmap v6129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:50 smithi042 ceph-mon[32775]: pgmap v6129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10481.356494 secs 2024-07-26T09:50:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10499.096075 secs 2024-07-26T09:50:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10481.356494 secs 2024-07-26T09:50:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10499.096075 secs 2024-07-26T09:50:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10481.356494 secs 2024-07-26T09:50:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10499.096075 secs 2024-07-26T09:50:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:52 smithi164 ceph-mon[36970]: pgmap v6130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:52 smithi163 ceph-mon[36873]: pgmap v6130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:52 smithi042 ceph-mon[32775]: pgmap v6130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:54 smithi163 ceph-mon[36873]: pgmap v6131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:50:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:54 smithi042 ceph-mon[32775]: pgmap v6131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:50:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:54 smithi164 ceph-mon[36970]: pgmap v6131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:50:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:56 smithi163 ceph-mon[36873]: pgmap v6132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10486.356574 secs 2024-07-26T09:50:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10504.096188 secs 2024-07-26T09:50:57.009 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:56 smithi042 ceph-mon[32775]: pgmap v6132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:57.009 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10486.356574 secs 2024-07-26T09:50:57.009 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10504.096188 secs 2024-07-26T09:50:57.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:50:57.195+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:50:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:50:57.195+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:50:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:50:57.195+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:50:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:56 smithi164 ceph-mon[36970]: pgmap v6132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:50:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10486.356574 secs 2024-07-26T09:50:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10504.096188 secs 2024-07-26T09:50:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:50:58 smithi163 ceph-mon[36873]: pgmap v6133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:50:58 smithi042 ceph-mon[32775]: pgmap v6133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:50:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:50:58 smithi164 ceph-mon[36970]: pgmap v6133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:00 smithi163 ceph-mon[36873]: pgmap v6134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:00 smithi042 ceph-mon[32775]: pgmap v6134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:00 smithi164 ceph-mon[36970]: pgmap v6134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10491.356720 secs 2024-07-26T09:51:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10509.096280 secs 2024-07-26T09:51:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10491.356720 secs 2024-07-26T09:51:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10509.096280 secs 2024-07-26T09:51:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10491.356720 secs 2024-07-26T09:51:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10509.096280 secs 2024-07-26T09:51:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:02 smithi163 ceph-mon[36873]: pgmap v6135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:02 smithi042 ceph-mon[32775]: pgmap v6135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:02 smithi164 ceph-mon[36970]: pgmap v6135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:04 smithi163 ceph-mon[36873]: pgmap v6136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:51:05.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:04 smithi042 ceph-mon[32775]: pgmap v6136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:51:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:04 smithi164 ceph-mon[36970]: pgmap v6136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:51:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:06 smithi163 ceph-mon[36873]: pgmap v6137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:51:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10496.356835 secs 2024-07-26T09:51:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10514.096361 secs 2024-07-26T09:51:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:06 smithi042 ceph-mon[32775]: pgmap v6137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:51:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10496.356835 secs 2024-07-26T09:51:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10514.096361 secs 2024-07-26T09:51:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:06 smithi164 ceph-mon[36970]: pgmap v6137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:51:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10496.356835 secs 2024-07-26T09:51:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10514.096361 secs 2024-07-26T09:51:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:08 smithi163 ceph-mon[36873]: pgmap v6138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:51:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:08 smithi042 ceph-mon[32775]: pgmap v6138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:51:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:08 smithi164 ceph-mon[36970]: pgmap v6138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:51:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:10 smithi163 ceph-mon[36873]: pgmap v6139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:51:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:10 smithi042 ceph-mon[32775]: pgmap v6139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:51:11.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:10 smithi164 ceph-mon[36970]: pgmap v6139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:51:12.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10501.356960 secs 2024-07-26T09:51:12.048 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10519.096503 secs 2024-07-26T09:51:12.048 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:51:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:51:12.205+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:51:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:51:12.205+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:51:12.207 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:51:12.205+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:51:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10501.356960 secs 2024-07-26T09:51:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10519.096503 secs 2024-07-26T09:51:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:51:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10501.356960 secs 2024-07-26T09:51:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10519.096503 secs 2024-07-26T09:51:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:51:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:12 smithi042 ceph-mon[32775]: pgmap v6140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:51:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:51:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:51:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:51:13.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:12 smithi163 ceph-mon[36873]: pgmap v6140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:51:13.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:51:13.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:51:13.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:51:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:12 smithi164 ceph-mon[36970]: pgmap v6140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T09:51:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:51:13.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:51:13.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:51:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:13 smithi042 ceph-mon[32775]: pgmap v6141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:51:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:13 smithi042 ceph-mon[32775]: pgmap v6142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T09:51:14.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:13 smithi164 ceph-mon[36970]: pgmap v6141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:51:14.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:13 smithi164 ceph-mon[36970]: pgmap v6142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T09:51:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:13 smithi163 ceph-mon[36873]: pgmap v6141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:51:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:13 smithi163 ceph-mon[36873]: pgmap v6142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T09:51:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:15 smithi042 ceph-mon[32775]: pgmap v6143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:51:16.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:15 smithi164 ceph-mon[36970]: pgmap v6143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:51:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:15 smithi163 ceph-mon[36873]: pgmap v6143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:51:17.048 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10506.357083 secs 2024-07-26T09:51:17.048 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10524.096626 secs 2024-07-26T09:51:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10506.357083 secs 2024-07-26T09:51:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10524.096626 secs 2024-07-26T09:51:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10506.357083 secs 2024-07-26T09:51:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10524.096626 secs 2024-07-26T09:51:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:17 smithi042 ceph-mon[32775]: pgmap v6144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:51:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:17 smithi164 ceph-mon[36970]: pgmap v6144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:51:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:17 smithi163 ceph-mon[36873]: pgmap v6144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:51:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:19 smithi042 ceph-mon[32775]: pgmap v6145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:51:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:19 smithi164 ceph-mon[36970]: pgmap v6145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:51:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:19 smithi163 ceph-mon[36873]: pgmap v6145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:51:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:21 smithi042 ceph-mon[32775]: pgmap v6146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:51:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10511.357206 secs 2024-07-26T09:51:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10529.096762 secs 2024-07-26T09:51:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:21 smithi164 ceph-mon[36970]: pgmap v6146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:51:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10511.357206 secs 2024-07-26T09:51:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10529.096762 secs 2024-07-26T09:51:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:21 smithi163 ceph-mon[36873]: pgmap v6146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:51:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10511.357206 secs 2024-07-26T09:51:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10529.096762 secs 2024-07-26T09:51:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:23 smithi042 ceph-mon[32775]: pgmap v6147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:51:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:23 smithi164 ceph-mon[36970]: pgmap v6147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:51:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:23 smithi163 ceph-mon[36873]: pgmap v6147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:51:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:25 smithi042 ceph-mon[32775]: pgmap v6148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:26.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:25 smithi164 ceph-mon[36970]: pgmap v6148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:25 smithi163 ceph-mon[36873]: pgmap v6148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10516.357334 secs 2024-07-26T09:51:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10534.096882 secs 2024-07-26T09:51:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:51:27.207+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:51:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:51:27.207+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:51:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:51:27.207+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:51:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10516.357334 secs 2024-07-26T09:51:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10534.096882 secs 2024-07-26T09:51:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10516.357334 secs 2024-07-26T09:51:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10534.096882 secs 2024-07-26T09:51:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:27 smithi042 ceph-mon[32775]: pgmap v6149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:27 smithi164 ceph-mon[36970]: pgmap v6149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:27 smithi163 ceph-mon[36873]: pgmap v6149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:29 smithi042 ceph-mon[32775]: pgmap v6150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:29 smithi164 ceph-mon[36970]: pgmap v6150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:29 smithi163 ceph-mon[36873]: pgmap v6150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:31 smithi042 ceph-mon[32775]: pgmap v6151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10521.357481 secs 2024-07-26T09:51:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10539.096990 secs 2024-07-26T09:51:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:31 smithi164 ceph-mon[36970]: pgmap v6151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10521.357481 secs 2024-07-26T09:51:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10539.096990 secs 2024-07-26T09:51:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:31 smithi163 ceph-mon[36873]: pgmap v6151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10521.357481 secs 2024-07-26T09:51:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10539.096990 secs 2024-07-26T09:51:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:33 smithi042 ceph-mon[32775]: pgmap v6152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:33 smithi164 ceph-mon[36970]: pgmap v6152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:33 smithi163 ceph-mon[36873]: pgmap v6152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:35 smithi042 ceph-mon[32775]: pgmap v6153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:35 smithi164 ceph-mon[36970]: pgmap v6153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:35 smithi163 ceph-mon[36873]: pgmap v6153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10526.357562 secs 2024-07-26T09:51:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10544.097094 secs 2024-07-26T09:51:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10526.357562 secs 2024-07-26T09:51:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10544.097094 secs 2024-07-26T09:51:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10526.357562 secs 2024-07-26T09:51:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10544.097094 secs 2024-07-26T09:51:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:37 smithi042 ceph-mon[32775]: pgmap v6154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:37 smithi164 ceph-mon[36970]: pgmap v6154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:37 smithi163 ceph-mon[36873]: pgmap v6154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:39 smithi042 ceph-mon[32775]: pgmap v6155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:39 smithi164 ceph-mon[36970]: pgmap v6155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:39 smithi163 ceph-mon[36873]: pgmap v6155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:41 smithi042 ceph-mon[32775]: pgmap v6156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10531.357632 secs 2024-07-26T09:51:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10549.097196 secs 2024-07-26T09:51:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:51:42.217+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:51:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:51:42.217+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:51:42.219 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:51:42.217+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:51:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:41 smithi164 ceph-mon[36970]: pgmap v6156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10531.357632 secs 2024-07-26T09:51:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10549.097196 secs 2024-07-26T09:51:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:41 smithi163 ceph-mon[36873]: pgmap v6156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10531.357632 secs 2024-07-26T09:51:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10549.097196 secs 2024-07-26T09:51:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:43 smithi042 ceph-mon[32775]: pgmap v6157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:43 smithi164 ceph-mon[36970]: pgmap v6157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:43 smithi163 ceph-mon[36873]: pgmap v6157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:45 smithi042 ceph-mon[32775]: pgmap v6158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:45 smithi164 ceph-mon[36970]: pgmap v6158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:45 smithi163 ceph-mon[36873]: pgmap v6158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10536.357759 secs 2024-07-26T09:51:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10554.097312 secs 2024-07-26T09:51:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10536.357759 secs 2024-07-26T09:51:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10554.097312 secs 2024-07-26T09:51:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10536.357759 secs 2024-07-26T09:51:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10554.097312 secs 2024-07-26T09:51:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:47 smithi042 ceph-mon[32775]: pgmap v6159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:47 smithi164 ceph-mon[36970]: pgmap v6159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:47 smithi163 ceph-mon[36873]: pgmap v6159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:49 smithi042 ceph-mon[32775]: pgmap v6160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:49 smithi164 ceph-mon[36970]: pgmap v6160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:49 smithi163 ceph-mon[36873]: pgmap v6160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:51 smithi042 ceph-mon[32775]: pgmap v6161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:52.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10541.357888 secs 2024-07-26T09:51:52.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10559.097396 secs 2024-07-26T09:51:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:51 smithi164 ceph-mon[36970]: pgmap v6161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:52.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10541.357888 secs 2024-07-26T09:51:52.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10559.097396 secs 2024-07-26T09:51:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:51 smithi163 ceph-mon[36873]: pgmap v6161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10541.357888 secs 2024-07-26T09:51:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10559.097396 secs 2024-07-26T09:51:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:53 smithi042 ceph-mon[32775]: pgmap v6162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:53 smithi164 ceph-mon[36970]: pgmap v6162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:53 smithi163 ceph-mon[36873]: pgmap v6162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:55 smithi042 ceph-mon[32775]: pgmap v6163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:55 smithi164 ceph-mon[36970]: pgmap v6163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:55 smithi163 ceph-mon[36873]: pgmap v6163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:51:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10546.358018 secs 2024-07-26T09:51:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10564.097516 secs 2024-07-26T09:51:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10546.358018 secs 2024-07-26T09:51:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10564.097516 secs 2024-07-26T09:51:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:51:57.222+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:51:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:51:57.222+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:51:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:51:57.222+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:51:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10546.358018 secs 2024-07-26T09:51:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10564.097516 secs 2024-07-26T09:51:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:57 smithi042 ceph-mon[32775]: pgmap v6164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:57 smithi164 ceph-mon[36970]: pgmap v6164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:51:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:57 smithi163 ceph-mon[36873]: pgmap v6164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:51:59 smithi042 ceph-mon[32775]: pgmap v6165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:51:59 smithi164 ceph-mon[36970]: pgmap v6165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:51:59 smithi163 ceph-mon[36873]: pgmap v6165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:01 smithi042 ceph-mon[32775]: pgmap v6166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10551.358118 secs 2024-07-26T09:52:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10569.097616 secs 2024-07-26T09:52:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:01 smithi164 ceph-mon[36970]: pgmap v6166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10551.358118 secs 2024-07-26T09:52:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10569.097616 secs 2024-07-26T09:52:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:01 smithi163 ceph-mon[36873]: pgmap v6166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10551.358118 secs 2024-07-26T09:52:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10569.097616 secs 2024-07-26T09:52:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:03 smithi042 ceph-mon[32775]: pgmap v6167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:03 smithi164 ceph-mon[36970]: pgmap v6167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:03 smithi163 ceph-mon[36873]: pgmap v6167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:05 smithi042 ceph-mon[32775]: pgmap v6168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:05 smithi164 ceph-mon[36970]: pgmap v6168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:05 smithi163 ceph-mon[36873]: pgmap v6168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10556.358267 secs 2024-07-26T09:52:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10574.097760 secs 2024-07-26T09:52:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10556.358267 secs 2024-07-26T09:52:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10574.097760 secs 2024-07-26T09:52:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10556.358267 secs 2024-07-26T09:52:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10574.097760 secs 2024-07-26T09:52:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:07 smithi042 ceph-mon[32775]: pgmap v6169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:07 smithi164 ceph-mon[36970]: pgmap v6169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:07 smithi163 ceph-mon[36873]: pgmap v6169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:09 smithi042 ceph-mon[32775]: pgmap v6170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:09 smithi164 ceph-mon[36970]: pgmap v6170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:09 smithi163 ceph-mon[36873]: pgmap v6170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:11 smithi164 ceph-mon[36970]: pgmap v6171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10561.358334 secs 2024-07-26T09:52:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10579.097886 secs 2024-07-26T09:52:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:52:12.232+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:52:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:52:12.232+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:52:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:52:12.232+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:52:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:11 smithi163 ceph-mon[36873]: pgmap v6171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10561.358334 secs 2024-07-26T09:52:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10579.097886 secs 2024-07-26T09:52:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:11 smithi042 ceph-mon[32775]: pgmap v6171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10561.358334 secs 2024-07-26T09:52:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10579.097886 secs 2024-07-26T09:52:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:13 smithi164 ceph-mon[36970]: pgmap v6172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:13.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:52:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:13 smithi163 ceph-mon[36873]: pgmap v6172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:52:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:13 smithi042 ceph-mon[32775]: pgmap v6172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:52:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:14 smithi164 ceph-mon[36970]: pgmap v6173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:52:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:52:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:14 smithi164 ceph-mon[36970]: pgmap v6174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:52:15.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:14 smithi164 ceph-mon[36970]: pgmap v6175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T09:52:15.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:14 smithi163 ceph-mon[36873]: pgmap v6173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:52:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:52:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:14 smithi163 ceph-mon[36873]: pgmap v6174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:52:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:14 smithi163 ceph-mon[36873]: pgmap v6175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T09:52:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:14 smithi042 ceph-mon[32775]: pgmap v6173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:15.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:15.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:52:15.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:52:15.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:14 smithi042 ceph-mon[32775]: pgmap v6174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:52:15.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:14 smithi042 ceph-mon[32775]: pgmap v6175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T09:52:15.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:52:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10566.358460 secs 2024-07-26T09:52:16.727 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10566.358460 secs 2024-07-26T09:52:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10566.358460 secs 2024-07-26T09:52:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10584.098018 secs 2024-07-26T09:52:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:17 smithi042 ceph-mon[32775]: pgmap v6176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T09:52:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10584.098018 secs 2024-07-26T09:52:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:17 smithi164 ceph-mon[36970]: pgmap v6176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T09:52:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10584.098018 secs 2024-07-26T09:52:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:17 smithi163 ceph-mon[36873]: pgmap v6176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T09:52:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:19 smithi163 ceph-mon[36873]: pgmap v6177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:52:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:19 smithi042 ceph-mon[32775]: pgmap v6177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:52:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:19 smithi164 ceph-mon[36970]: pgmap v6177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:52:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:21 smithi163 ceph-mon[36873]: pgmap v6178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T09:52:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10571.358551 secs 2024-07-26T09:52:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10589.098119 secs 2024-07-26T09:52:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:21 smithi042 ceph-mon[32775]: pgmap v6178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T09:52:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10571.358551 secs 2024-07-26T09:52:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10589.098119 secs 2024-07-26T09:52:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:21 smithi164 ceph-mon[36970]: pgmap v6178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T09:52:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10571.358551 secs 2024-07-26T09:52:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10589.098119 secs 2024-07-26T09:52:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:23 smithi163 ceph-mon[36873]: pgmap v6179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:52:24.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:23 smithi164 ceph-mon[36970]: pgmap v6179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:52:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:23 smithi042 ceph-mon[32775]: pgmap v6179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:52:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:25 smithi163 ceph-mon[36873]: pgmap v6180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:52:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:25 smithi042 ceph-mon[32775]: pgmap v6180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:52:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:25 smithi164 ceph-mon[36970]: pgmap v6180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:52:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10576.358616 secs 2024-07-26T09:52:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10594.098240 secs 2024-07-26T09:52:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10576.358616 secs 2024-07-26T09:52:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10594.098240 secs 2024-07-26T09:52:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10576.358616 secs 2024-07-26T09:52:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10594.098240 secs 2024-07-26T09:52:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:52:27.237+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:52:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:52:27.237+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:52:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:52:27.237+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:52:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:27 smithi163 ceph-mon[36873]: pgmap v6181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:27 smithi042 ceph-mon[32775]: pgmap v6181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:27 smithi164 ceph-mon[36970]: pgmap v6181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:29 smithi042 ceph-mon[32775]: pgmap v6182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:52:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:29 smithi164 ceph-mon[36970]: pgmap v6182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:52:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:29 smithi163 ceph-mon[36873]: pgmap v6182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:52:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:31 smithi042 ceph-mon[32775]: pgmap v6183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10581.358750 secs 2024-07-26T09:52:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10599.098359 secs 2024-07-26T09:52:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:31 smithi164 ceph-mon[36970]: pgmap v6183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10581.358750 secs 2024-07-26T09:52:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10599.098359 secs 2024-07-26T09:52:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:31 smithi163 ceph-mon[36873]: pgmap v6183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10581.358750 secs 2024-07-26T09:52:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10599.098359 secs 2024-07-26T09:52:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:33 smithi042 ceph-mon[32775]: pgmap v6184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:33 smithi164 ceph-mon[36970]: pgmap v6184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:33 smithi163 ceph-mon[36873]: pgmap v6184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:35 smithi042 ceph-mon[32775]: pgmap v6185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:35 smithi164 ceph-mon[36970]: pgmap v6185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:35 smithi163 ceph-mon[36873]: pgmap v6185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10586.358874 secs 2024-07-26T09:52:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10604.098498 secs 2024-07-26T09:52:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10586.358874 secs 2024-07-26T09:52:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10604.098498 secs 2024-07-26T09:52:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10586.358874 secs 2024-07-26T09:52:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10604.098498 secs 2024-07-26T09:52:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:37 smithi042 ceph-mon[32775]: pgmap v6186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:37 smithi164 ceph-mon[36970]: pgmap v6186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:37 smithi163 ceph-mon[36873]: pgmap v6186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:39 smithi042 ceph-mon[32775]: pgmap v6187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:52:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:39 smithi164 ceph-mon[36970]: pgmap v6187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:52:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:39 smithi163 ceph-mon[36873]: pgmap v6187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:52:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:41 smithi042 ceph-mon[32775]: pgmap v6188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10591.359003 secs 2024-07-26T09:52:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10609.098641 secs 2024-07-26T09:52:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:41 smithi164 ceph-mon[36970]: pgmap v6188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10591.359003 secs 2024-07-26T09:52:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10609.098641 secs 2024-07-26T09:52:42.249 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:52:42.248+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:52:42.249 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:52:42.248+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:52:42.249 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:52:42.248+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:52:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:41 smithi163 ceph-mon[36873]: pgmap v6188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10591.359003 secs 2024-07-26T09:52:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10609.098641 secs 2024-07-26T09:52:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:43 smithi042 ceph-mon[32775]: pgmap v6189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:43 smithi164 ceph-mon[36970]: pgmap v6189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:43 smithi163 ceph-mon[36873]: pgmap v6189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:46.061 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:45 smithi163 ceph-mon[36873]: pgmap v6190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:45 smithi042 ceph-mon[32775]: pgmap v6190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:45 smithi164 ceph-mon[36970]: pgmap v6190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10596.359138 secs 2024-07-26T09:52:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10614.098778 secs 2024-07-26T09:52:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10596.359138 secs 2024-07-26T09:52:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10614.098778 secs 2024-07-26T09:52:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10596.359138 secs 2024-07-26T09:52:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10614.098778 secs 2024-07-26T09:52:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:47 smithi042 ceph-mon[32775]: pgmap v6191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:47 smithi164 ceph-mon[36970]: pgmap v6191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:47 smithi163 ceph-mon[36873]: pgmap v6191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:49 smithi042 ceph-mon[32775]: pgmap v6192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:52:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:49 smithi164 ceph-mon[36970]: pgmap v6192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:52:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:49 smithi163 ceph-mon[36873]: pgmap v6192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:52:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:51 smithi042 ceph-mon[32775]: pgmap v6193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10601.359273 secs 2024-07-26T09:52:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10619.098911 secs 2024-07-26T09:52:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:51 smithi164 ceph-mon[36970]: pgmap v6193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10601.359273 secs 2024-07-26T09:52:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10619.098911 secs 2024-07-26T09:52:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:51 smithi163 ceph-mon[36873]: pgmap v6193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10601.359273 secs 2024-07-26T09:52:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10619.098911 secs 2024-07-26T09:52:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:53 smithi042 ceph-mon[32775]: pgmap v6194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:53 smithi164 ceph-mon[36970]: pgmap v6194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:53 smithi163 ceph-mon[36873]: pgmap v6194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:55 smithi042 ceph-mon[32775]: pgmap v6195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:55 smithi164 ceph-mon[36970]: pgmap v6195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:55 smithi163 ceph-mon[36873]: pgmap v6195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:52:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10606.359357 secs 2024-07-26T09:52:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10624.099052 secs 2024-07-26T09:52:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10606.359357 secs 2024-07-26T09:52:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10624.099052 secs 2024-07-26T09:52:57.254 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:52:57.252+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:52:57.254 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:52:57.252+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:52:57.254 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:52:57.252+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:52:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10606.359357 secs 2024-07-26T09:52:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10624.099052 secs 2024-07-26T09:52:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:57 smithi042 ceph-mon[32775]: pgmap v6196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:57 smithi164 ceph-mon[36970]: pgmap v6196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:57 smithi163 ceph-mon[36873]: pgmap v6196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:52:59.518 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:52:59.516+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:52:59.518 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:52:59.516+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:52:59.518 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:52:59.516+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:52:59.518 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:52:59.516+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:53:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:52:59 smithi042 ceph-mon[32775]: pgmap v6197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:53:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:52:59 smithi164 ceph-mon[36970]: pgmap v6197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:53:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:52:59 smithi163 ceph-mon[36873]: pgmap v6197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:53:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:01 smithi042 ceph-mon[32775]: pgmap v6198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10611.359444 secs 2024-07-26T09:53:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10629.099168 secs 2024-07-26T09:53:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:01 smithi164 ceph-mon[36970]: pgmap v6198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10611.359444 secs 2024-07-26T09:53:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10629.099168 secs 2024-07-26T09:53:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:01 smithi163 ceph-mon[36873]: pgmap v6198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10611.359444 secs 2024-07-26T09:53:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10629.099168 secs 2024-07-26T09:53:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:03 smithi042 ceph-mon[32775]: pgmap v6199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:03 smithi164 ceph-mon[36970]: pgmap v6199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:03 smithi163 ceph-mon[36873]: pgmap v6199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:05 smithi042 ceph-mon[32775]: pgmap v6200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:05 smithi164 ceph-mon[36970]: pgmap v6200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:05 smithi163 ceph-mon[36873]: pgmap v6200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10616.359571 secs 2024-07-26T09:53:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10634.099291 secs 2024-07-26T09:53:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10616.359571 secs 2024-07-26T09:53:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10634.099291 secs 2024-07-26T09:53:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10616.359571 secs 2024-07-26T09:53:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10634.099291 secs 2024-07-26T09:53:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:07 smithi042 ceph-mon[32775]: pgmap v6201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:07 smithi164 ceph-mon[36970]: pgmap v6201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:07 smithi163 ceph-mon[36873]: pgmap v6201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:09 smithi042 ceph-mon[32775]: pgmap v6202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:53:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:09 smithi164 ceph-mon[36970]: pgmap v6202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:53:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:09 smithi163 ceph-mon[36873]: pgmap v6202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:53:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:11 smithi042 ceph-mon[32775]: pgmap v6203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10621.359681 secs 2024-07-26T09:53:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10639.099355 secs 2024-07-26T09:53:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:11 smithi164 ceph-mon[36970]: pgmap v6203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10621.359681 secs 2024-07-26T09:53:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10639.099355 secs 2024-07-26T09:53:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:11 smithi163 ceph-mon[36873]: pgmap v6203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10621.359681 secs 2024-07-26T09:53:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10639.099355 secs 2024-07-26T09:53:12.263 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:53:12.261+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:53:12.263 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:53:12.261+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:53:12.263 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:53:12.261+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:53:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:13 smithi042 ceph-mon[32775]: pgmap v6204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:13 smithi164 ceph-mon[36970]: pgmap v6204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:13 smithi163 ceph-mon[36873]: pgmap v6204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:53:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:53:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:53:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:15 smithi042 ceph-mon[32775]: pgmap v6205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:53:16.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:53:16.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:15 smithi163 ceph-mon[36873]: pgmap v6205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:16.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:53:16.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:53:16.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:15 smithi164 ceph-mon[36970]: pgmap v6205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:16.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:16.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:53:16.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:53:16.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:53:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:16 smithi042 ceph-mon[32775]: pgmap v6206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T09:53:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:16 smithi042 ceph-mon[32775]: pgmap v6207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:53:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10626.359811 secs 2024-07-26T09:53:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10644.099479 secs 2024-07-26T09:53:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:16 smithi164 ceph-mon[36970]: pgmap v6206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T09:53:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:16 smithi164 ceph-mon[36970]: pgmap v6207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:53:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10626.359811 secs 2024-07-26T09:53:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10644.099479 secs 2024-07-26T09:53:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:16 smithi163 ceph-mon[36873]: pgmap v6206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T09:53:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:16 smithi163 ceph-mon[36873]: pgmap v6207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:53:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10626.359811 secs 2024-07-26T09:53:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10644.099479 secs 2024-07-26T09:53:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:18 smithi042 ceph-mon[32775]: pgmap v6208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:53:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:18 smithi164 ceph-mon[36970]: pgmap v6208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:53:19.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:18 smithi163 ceph-mon[36873]: pgmap v6208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:53:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:20 smithi042 ceph-mon[32775]: pgmap v6209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:53:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:20 smithi164 ceph-mon[36970]: pgmap v6209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:53:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:20 smithi163 ceph-mon[36873]: pgmap v6209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:53:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10631.359925 secs 2024-07-26T09:53:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10649.099624 secs 2024-07-26T09:53:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10631.359925 secs 2024-07-26T09:53:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10649.099624 secs 2024-07-26T09:53:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10631.359925 secs 2024-07-26T09:53:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10649.099624 secs 2024-07-26T09:53:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:22 smithi042 ceph-mon[32775]: pgmap v6210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:53:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:22 smithi164 ceph-mon[36970]: pgmap v6210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:53:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:22 smithi163 ceph-mon[36873]: pgmap v6210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:53:25.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:24 smithi042 ceph-mon[32775]: pgmap v6211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T09:53:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:24 smithi164 ceph-mon[36970]: pgmap v6211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T09:53:25.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:24 smithi163 ceph-mon[36873]: pgmap v6211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T09:53:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:26 smithi042 ceph-mon[32775]: pgmap v6212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:53:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10636.360050 secs 2024-07-26T09:53:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10654.099763 secs 2024-07-26T09:53:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:26 smithi164 ceph-mon[36970]: pgmap v6212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:53:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10636.360050 secs 2024-07-26T09:53:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10654.099763 secs 2024-07-26T09:53:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:26 smithi163 ceph-mon[36873]: pgmap v6212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:53:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10636.360050 secs 2024-07-26T09:53:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10654.099763 secs 2024-07-26T09:53:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:53:27.266+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:53:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:53:27.266+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:53:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:53:27.266+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:53:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:28 smithi042 ceph-mon[32775]: pgmap v6213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:28 smithi164 ceph-mon[36970]: pgmap v6213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:28 smithi163 ceph-mon[36873]: pgmap v6213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:30 smithi042 ceph-mon[32775]: pgmap v6214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:30 smithi164 ceph-mon[36970]: pgmap v6214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:31.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:30 smithi163 ceph-mon[36873]: pgmap v6214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10641.360189 secs 2024-07-26T09:53:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10659.099880 secs 2024-07-26T09:53:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10641.360189 secs 2024-07-26T09:53:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10659.099880 secs 2024-07-26T09:53:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10641.360189 secs 2024-07-26T09:53:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10659.099880 secs 2024-07-26T09:53:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:32 smithi164 ceph-mon[36970]: pgmap v6215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:32 smithi163 ceph-mon[36873]: pgmap v6215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:32 smithi042 ceph-mon[32775]: pgmap v6215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:34 smithi164 ceph-mon[36970]: pgmap v6216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:53:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:34 smithi163 ceph-mon[36873]: pgmap v6216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:53:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:34 smithi042 ceph-mon[32775]: pgmap v6216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:53:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:36 smithi164 ceph-mon[36970]: pgmap v6217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10646.360266 secs 2024-07-26T09:53:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10664.100004 secs 2024-07-26T09:53:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:36 smithi163 ceph-mon[36873]: pgmap v6217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10646.360266 secs 2024-07-26T09:53:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10664.100004 secs 2024-07-26T09:53:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:36 smithi042 ceph-mon[32775]: pgmap v6217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10646.360266 secs 2024-07-26T09:53:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10664.100004 secs 2024-07-26T09:53:39.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:38 smithi164 ceph-mon[36970]: pgmap v6218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:38 smithi163 ceph-mon[36873]: pgmap v6218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:38 smithi042 ceph-mon[32775]: pgmap v6218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:40 smithi164 ceph-mon[36970]: pgmap v6219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:41.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:40 smithi163 ceph-mon[36873]: pgmap v6219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:40 smithi042 ceph-mon[32775]: pgmap v6219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10651.360393 secs 2024-07-26T09:53:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10669.100103 secs 2024-07-26T09:53:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10651.360393 secs 2024-07-26T09:53:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10669.100103 secs 2024-07-26T09:53:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:53:42.276+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:53:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:53:42.276+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:53:42.278 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:53:42.276+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:53:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10651.360393 secs 2024-07-26T09:53:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10669.100103 secs 2024-07-26T09:53:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:42 smithi164 ceph-mon[36970]: pgmap v6220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:42 smithi163 ceph-mon[36873]: pgmap v6220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:42 smithi042 ceph-mon[32775]: pgmap v6220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:44 smithi164 ceph-mon[36970]: pgmap v6221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:53:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:44 smithi163 ceph-mon[36873]: pgmap v6221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:53:45.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:44 smithi042 ceph-mon[32775]: pgmap v6221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:53:47.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:46 smithi042 ceph-mon[32775]: pgmap v6222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:47.199 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10656.360529 secs 2024-07-26T09:53:47.199 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10674.100204 secs 2024-07-26T09:53:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:46 smithi164 ceph-mon[36970]: pgmap v6222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10656.360529 secs 2024-07-26T09:53:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10674.100204 secs 2024-07-26T09:53:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:46 smithi163 ceph-mon[36873]: pgmap v6222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10656.360529 secs 2024-07-26T09:53:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10674.100204 secs 2024-07-26T09:53:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:48 smithi164 ceph-mon[36970]: pgmap v6223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:48 smithi163 ceph-mon[36873]: pgmap v6223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:48 smithi042 ceph-mon[32775]: pgmap v6223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:50 smithi164 ceph-mon[36970]: pgmap v6224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:50 smithi163 ceph-mon[36873]: pgmap v6224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:50 smithi042 ceph-mon[32775]: pgmap v6224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10661.360596 secs 2024-07-26T09:53:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10679.100280 secs 2024-07-26T09:53:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10661.360596 secs 2024-07-26T09:53:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10679.100280 secs 2024-07-26T09:53:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10661.360596 secs 2024-07-26T09:53:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10679.100280 secs 2024-07-26T09:53:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:52 smithi164 ceph-mon[36970]: pgmap v6225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:52 smithi163 ceph-mon[36873]: pgmap v6225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:52 smithi042 ceph-mon[32775]: pgmap v6225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:54 smithi164 ceph-mon[36970]: pgmap v6226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:53:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:54 smithi163 ceph-mon[36873]: pgmap v6226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:53:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:54 smithi042 ceph-mon[32775]: pgmap v6226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:53:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:56 smithi164 ceph-mon[36970]: pgmap v6227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10666.360719 secs 2024-07-26T09:53:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10684.100399 secs 2024-07-26T09:53:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:56 smithi163 ceph-mon[36873]: pgmap v6227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10666.360719 secs 2024-07-26T09:53:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10684.100399 secs 2024-07-26T09:53:57.283 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:53:57.281+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:53:57.283 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:53:57.281+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:53:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:53:57.281+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:53:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:56 smithi042 ceph-mon[32775]: pgmap v6227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:53:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10666.360719 secs 2024-07-26T09:53:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10684.100399 secs 2024-07-26T09:53:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:53:58 smithi164 ceph-mon[36970]: pgmap v6228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:53:58 smithi163 ceph-mon[36873]: pgmap v6228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:53:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:53:58 smithi042 ceph-mon[32775]: pgmap v6228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:00 smithi164 ceph-mon[36970]: pgmap v6229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:00 smithi163 ceph-mon[36873]: pgmap v6229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:00 smithi042 ceph-mon[32775]: pgmap v6229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10671.360840 secs 2024-07-26T09:54:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10689.100548 secs 2024-07-26T09:54:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10671.360840 secs 2024-07-26T09:54:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10689.100548 secs 2024-07-26T09:54:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10671.360840 secs 2024-07-26T09:54:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10689.100548 secs 2024-07-26T09:54:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:02 smithi042 ceph-mon[32775]: pgmap v6230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:02 smithi164 ceph-mon[36970]: pgmap v6230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:02 smithi163 ceph-mon[36873]: pgmap v6230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:04 smithi164 ceph-mon[36970]: pgmap v6231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:54:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:04 smithi163 ceph-mon[36873]: pgmap v6231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:54:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:04 smithi042 ceph-mon[32775]: pgmap v6231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:54:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:06 smithi164 ceph-mon[36970]: pgmap v6232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10676.360939 secs 2024-07-26T09:54:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10694.100672 secs 2024-07-26T09:54:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:06 smithi163 ceph-mon[36873]: pgmap v6232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10676.360939 secs 2024-07-26T09:54:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10694.100672 secs 2024-07-26T09:54:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:06 smithi042 ceph-mon[32775]: pgmap v6232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10676.360939 secs 2024-07-26T09:54:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10694.100672 secs 2024-07-26T09:54:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:08 smithi164 ceph-mon[36970]: pgmap v6233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:09.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:08 smithi163 ceph-mon[36873]: pgmap v6233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:08 smithi042 ceph-mon[32775]: pgmap v6233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:10 smithi164 ceph-mon[36970]: pgmap v6234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:10 smithi163 ceph-mon[36873]: pgmap v6234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:10 smithi042 ceph-mon[32775]: pgmap v6234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10681.361006 secs 2024-07-26T09:54:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10699.100801 secs 2024-07-26T09:54:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:54:12.291+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:54:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:54:12.291+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:54:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:54:12.291+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:54:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10681.361006 secs 2024-07-26T09:54:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10699.100801 secs 2024-07-26T09:54:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10681.361006 secs 2024-07-26T09:54:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10699.100801 secs 2024-07-26T09:54:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:12 smithi163 ceph-mon[36873]: pgmap v6235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:12 smithi042 ceph-mon[32775]: pgmap v6235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:12 smithi164 ceph-mon[36970]: pgmap v6235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:13 smithi163 ceph-mon[36873]: pgmap v6236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:54:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:13 smithi042 ceph-mon[32775]: pgmap v6236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:54:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:13 smithi164 ceph-mon[36970]: pgmap v6236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:54:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:16 smithi042 ceph-mon[32775]: pgmap v6237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:54:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10686.361076 secs 2024-07-26T09:54:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:54:16.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:54:16.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:54:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:16 smithi164 ceph-mon[36970]: pgmap v6237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:54:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10686.361076 secs 2024-07-26T09:54:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:54:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:54:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:54:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:16 smithi163 ceph-mon[36873]: pgmap v6237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:54:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10686.361076 secs 2024-07-26T09:54:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:54:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:54:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:54:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:17 smithi042 ceph-mon[32775]: pgmap v6238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:54:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:17 smithi042 ceph-mon[32775]: pgmap v6239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:54:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10704.100891 secs 2024-07-26T09:54:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:17 smithi163 ceph-mon[36873]: pgmap v6238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:54:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:17 smithi163 ceph-mon[36873]: pgmap v6239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:54:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10704.100891 secs 2024-07-26T09:54:17.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:17 smithi164 ceph-mon[36970]: pgmap v6238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:54:17.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:17 smithi164 ceph-mon[36970]: pgmap v6239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:54:17.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10704.100891 secs 2024-07-26T09:54:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:19 smithi042 ceph-mon[32775]: pgmap v6240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:54:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:19 smithi164 ceph-mon[36970]: pgmap v6240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:54:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:19 smithi163 ceph-mon[36873]: pgmap v6240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:54:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:21 smithi042 ceph-mon[32775]: pgmap v6241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:54:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10691.361252 secs 2024-07-26T09:54:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:21 smithi164 ceph-mon[36970]: pgmap v6241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:54:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10691.361252 secs 2024-07-26T09:54:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:21 smithi163 ceph-mon[36873]: pgmap v6241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:54:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10691.361252 secs 2024-07-26T09:54:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10709.101016 secs 2024-07-26T09:54:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10709.101016 secs 2024-07-26T09:54:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10709.101016 secs 2024-07-26T09:54:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:23 smithi042 ceph-mon[32775]: pgmap v6242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:54:23.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:23 smithi164 ceph-mon[36970]: pgmap v6242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:54:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:23 smithi163 ceph-mon[36873]: pgmap v6242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:54:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:25 smithi042 ceph-mon[32775]: pgmap v6243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:54:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:25 smithi164 ceph-mon[36970]: pgmap v6243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:54:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:25 smithi163 ceph-mon[36873]: pgmap v6243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:54:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10696.361347 secs 2024-07-26T09:54:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10696.361347 secs 2024-07-26T09:54:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10696.361347 secs 2024-07-26T09:54:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:54:27.296+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:54:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:54:27.296+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:54:27.298 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:54:27.296+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:54:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:27 smithi164 ceph-mon[36970]: pgmap v6244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:54:27.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10714.101105 secs 2024-07-26T09:54:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:27 smithi163 ceph-mon[36873]: pgmap v6244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:54:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10714.101105 secs 2024-07-26T09:54:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:27 smithi042 ceph-mon[32775]: pgmap v6244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:54:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10714.101105 secs 2024-07-26T09:54:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:29 smithi164 ceph-mon[36970]: pgmap v6245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:29 smithi163 ceph-mon[36873]: pgmap v6245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:29 smithi042 ceph-mon[32775]: pgmap v6245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:31 smithi164 ceph-mon[36970]: pgmap v6246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10701.361477 secs 2024-07-26T09:54:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:31 smithi163 ceph-mon[36873]: pgmap v6246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10701.361477 secs 2024-07-26T09:54:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:31 smithi042 ceph-mon[32775]: pgmap v6246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10701.361477 secs 2024-07-26T09:54:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10719.101224 secs 2024-07-26T09:54:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10719.101224 secs 2024-07-26T09:54:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10719.101224 secs 2024-07-26T09:54:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:33 smithi164 ceph-mon[36970]: pgmap v6247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:33 smithi163 ceph-mon[36873]: pgmap v6247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:34.097 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:33 smithi042 ceph-mon[32775]: pgmap v6247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:35 smithi164 ceph-mon[36970]: pgmap v6248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:35 smithi163 ceph-mon[36873]: pgmap v6248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:35 smithi042 ceph-mon[32775]: pgmap v6248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10706.361604 secs 2024-07-26T09:54:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10706.361604 secs 2024-07-26T09:54:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10706.361604 secs 2024-07-26T09:54:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:37 smithi164 ceph-mon[36970]: pgmap v6249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10724.101309 secs 2024-07-26T09:54:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:37 smithi163 ceph-mon[36873]: pgmap v6249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10724.101309 secs 2024-07-26T09:54:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:37 smithi042 ceph-mon[32775]: pgmap v6249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10724.101309 secs 2024-07-26T09:54:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:39 smithi164 ceph-mon[36970]: pgmap v6250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:39 smithi163 ceph-mon[36873]: pgmap v6250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:39 smithi042 ceph-mon[32775]: pgmap v6250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:41 smithi164 ceph-mon[36970]: pgmap v6251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10711.361730 secs 2024-07-26T09:54:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:41 smithi163 ceph-mon[36873]: pgmap v6251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10711.361730 secs 2024-07-26T09:54:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:41 smithi042 ceph-mon[32775]: pgmap v6251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10711.361730 secs 2024-07-26T09:54:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:54:42.306+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:54:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:54:42.306+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:54:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:54:42.306+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:54:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10729.101438 secs 2024-07-26T09:54:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10729.101438 secs 2024-07-26T09:54:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10729.101438 secs 2024-07-26T09:54:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:43 smithi164 ceph-mon[36970]: pgmap v6252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:43 smithi163 ceph-mon[36873]: pgmap v6252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:43 smithi042 ceph-mon[32775]: pgmap v6252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:45.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:45 smithi164 ceph-mon[36970]: pgmap v6253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:45 smithi163 ceph-mon[36873]: pgmap v6253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:45 smithi042 ceph-mon[32775]: pgmap v6253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10716.361860 secs 2024-07-26T09:54:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10734.101578 secs 2024-07-26T09:54:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10716.361860 secs 2024-07-26T09:54:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10734.101578 secs 2024-07-26T09:54:47.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10716.361860 secs 2024-07-26T09:54:47.048 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10734.101578 secs 2024-07-26T09:54:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:47 smithi164 ceph-mon[36970]: pgmap v6254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:47 smithi163 ceph-mon[36873]: pgmap v6254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:47 smithi042 ceph-mon[32775]: pgmap v6254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:49.969 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:49 smithi042 ceph-mon[32775]: pgmap v6255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:49 smithi164 ceph-mon[36970]: pgmap v6255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:49 smithi163 ceph-mon[36873]: pgmap v6255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:51 smithi164 ceph-mon[36970]: pgmap v6256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10721.361944 secs 2024-07-26T09:54:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10739.101715 secs 2024-07-26T09:54:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:51 smithi163 ceph-mon[36873]: pgmap v6256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10721.361944 secs 2024-07-26T09:54:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10739.101715 secs 2024-07-26T09:54:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:51 smithi042 ceph-mon[32775]: pgmap v6256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10721.361944 secs 2024-07-26T09:54:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10739.101715 secs 2024-07-26T09:54:53.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:53 smithi164 ceph-mon[36970]: pgmap v6257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:53 smithi163 ceph-mon[36873]: pgmap v6257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:53 smithi042 ceph-mon[32775]: pgmap v6257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:55 smithi164 ceph-mon[36970]: pgmap v6258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:55 smithi163 ceph-mon[36873]: pgmap v6258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:55 smithi042 ceph-mon[32775]: pgmap v6258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:54:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10726.362069 secs 2024-07-26T09:54:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10744.101829 secs 2024-07-26T09:54:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10726.362069 secs 2024-07-26T09:54:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10744.101829 secs 2024-07-26T09:54:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10726.362069 secs 2024-07-26T09:54:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10744.101829 secs 2024-07-26T09:54:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:54:57.311+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:54:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:54:57.311+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:54:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:54:57.311+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:54:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:57 smithi164 ceph-mon[36970]: pgmap v6259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:57 smithi163 ceph-mon[36873]: pgmap v6259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:57 smithi042 ceph-mon[32775]: pgmap v6259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:54:59.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:54:59 smithi164 ceph-mon[36970]: pgmap v6260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:54:59 smithi163 ceph-mon[36873]: pgmap v6260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:54:59 smithi042 ceph-mon[32775]: pgmap v6260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:01 smithi163 ceph-mon[36873]: pgmap v6261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:02.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10731.362222 secs 2024-07-26T09:55:02.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10749.101966 secs 2024-07-26T09:55:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:01 smithi042 ceph-mon[32775]: pgmap v6261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10731.362222 secs 2024-07-26T09:55:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10749.101966 secs 2024-07-26T09:55:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:01 smithi164 ceph-mon[36970]: pgmap v6261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10731.362222 secs 2024-07-26T09:55:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10749.101966 secs 2024-07-26T09:55:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:03 smithi163 ceph-mon[36873]: pgmap v6262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:03 smithi042 ceph-mon[32775]: pgmap v6262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:03 smithi164 ceph-mon[36970]: pgmap v6262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:05 smithi163 ceph-mon[36873]: pgmap v6263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:05 smithi042 ceph-mon[32775]: pgmap v6263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:05 smithi164 ceph-mon[36970]: pgmap v6263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10736.362335 secs 2024-07-26T09:55:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10754.102050 secs 2024-07-26T09:55:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10736.362335 secs 2024-07-26T09:55:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10754.102050 secs 2024-07-26T09:55:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10736.362335 secs 2024-07-26T09:55:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10754.102050 secs 2024-07-26T09:55:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:07 smithi163 ceph-mon[36873]: pgmap v6264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:07 smithi042 ceph-mon[32775]: pgmap v6264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:07 smithi164 ceph-mon[36970]: pgmap v6264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:09 smithi163 ceph-mon[36873]: pgmap v6265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:09 smithi042 ceph-mon[32775]: pgmap v6265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:09 smithi164 ceph-mon[36970]: pgmap v6265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:11 smithi163 ceph-mon[36873]: pgmap v6266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10741.362455 secs 2024-07-26T09:55:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10759.102118 secs 2024-07-26T09:55:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:11 smithi042 ceph-mon[32775]: pgmap v6266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10741.362455 secs 2024-07-26T09:55:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10759.102118 secs 2024-07-26T09:55:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:11 smithi164 ceph-mon[36970]: pgmap v6266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10741.362455 secs 2024-07-26T09:55:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10759.102118 secs 2024-07-26T09:55:12.316 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:12.314+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:55:12.317 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:12.314+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:55:12.317 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:12.314+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:55:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:13 smithi163 ceph-mon[36873]: pgmap v6267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:13 smithi042 ceph-mon[32775]: pgmap v6267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:13 smithi164 ceph-mon[36970]: pgmap v6267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:15 smithi163 ceph-mon[36873]: pgmap v6268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:15 smithi042 ceph-mon[32775]: pgmap v6268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:15 smithi164 ceph-mon[36970]: pgmap v6268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10746.362536 secs 2024-07-26T09:55:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:55:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10764.102199 secs 2024-07-26T09:55:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:55:17.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:55:17.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:55:17.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10746.362536 secs 2024-07-26T09:55:17.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:55:17.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10764.102199 secs 2024-07-26T09:55:17.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:55:17.048 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:55:17.048 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:55:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10746.362536 secs 2024-07-26T09:55:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:55:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10764.102199 secs 2024-07-26T09:55:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:55:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:55:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:55:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:17 smithi042 ceph-mon[32775]: pgmap v6269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:17 smithi042 ceph-mon[32775]: pgmap v6270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:55:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:17 smithi042 ceph-mon[32775]: pgmap v6271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:55:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:17 smithi164 ceph-mon[36970]: pgmap v6269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:17 smithi164 ceph-mon[36970]: pgmap v6270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:55:18.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:17 smithi164 ceph-mon[36970]: pgmap v6271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:55:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:17 smithi163 ceph-mon[36873]: pgmap v6269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:17 smithi163 ceph-mon[36873]: pgmap v6270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:55:18.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:17 smithi163 ceph-mon[36873]: pgmap v6271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:55:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:19 smithi042 ceph-mon[32775]: pgmap v6272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:55:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:19 smithi164 ceph-mon[36970]: pgmap v6272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:55:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:19 smithi163 ceph-mon[36873]: pgmap v6272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:55:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:21 smithi042 ceph-mon[32775]: pgmap v6273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:55:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10751.362632 secs 2024-07-26T09:55:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10769.102319 secs 2024-07-26T09:55:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:21 smithi164 ceph-mon[36970]: pgmap v6273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:55:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10751.362632 secs 2024-07-26T09:55:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10769.102319 secs 2024-07-26T09:55:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:21 smithi163 ceph-mon[36873]: pgmap v6273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:55:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10751.362632 secs 2024-07-26T09:55:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10769.102319 secs 2024-07-26T09:55:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:23 smithi042 ceph-mon[32775]: pgmap v6274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:55:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:23 smithi164 ceph-mon[36970]: pgmap v6274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:55:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:23 smithi163 ceph-mon[36873]: pgmap v6274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:55:26.026 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:25 smithi164 ceph-mon[36970]: pgmap v6275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:55:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:25 smithi042 ceph-mon[32775]: pgmap v6275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:55:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:25 smithi163 ceph-mon[36873]: pgmap v6275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:55:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10756.362767 secs 2024-07-26T09:55:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10774.102457 secs 2024-07-26T09:55:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10756.362767 secs 2024-07-26T09:55:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10774.102457 secs 2024-07-26T09:55:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10756.362767 secs 2024-07-26T09:55:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10774.102457 secs 2024-07-26T09:55:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:27.319+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:55:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:27.319+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:55:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:27.319+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:55:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:27 smithi042 ceph-mon[32775]: pgmap v6276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:55:28.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:27 smithi164 ceph-mon[36970]: pgmap v6276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:55:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:27 smithi163 ceph-mon[36873]: pgmap v6276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:55:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:29 smithi042 ceph-mon[32775]: pgmap v6277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:55:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:29 smithi164 ceph-mon[36970]: pgmap v6277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:55:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:29 smithi163 ceph-mon[36873]: pgmap v6277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:55:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:31 smithi042 ceph-mon[32775]: pgmap v6278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10761.362929 secs 2024-07-26T09:55:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10779.102576 secs 2024-07-26T09:55:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:31 smithi164 ceph-mon[36970]: pgmap v6278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10761.362929 secs 2024-07-26T09:55:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10779.102576 secs 2024-07-26T09:55:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:31 smithi163 ceph-mon[36873]: pgmap v6278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10761.362929 secs 2024-07-26T09:55:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10779.102576 secs 2024-07-26T09:55:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:33 smithi042 ceph-mon[32775]: pgmap v6279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:33 smithi164 ceph-mon[36970]: pgmap v6279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:33 smithi163 ceph-mon[36873]: pgmap v6279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:36.049 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:35 smithi042 ceph-mon[32775]: pgmap v6280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:35 smithi164 ceph-mon[36970]: pgmap v6280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:35 smithi163 ceph-mon[36873]: pgmap v6280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10766.363065 secs 2024-07-26T09:55:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10784.102676 secs 2024-07-26T09:55:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10766.363065 secs 2024-07-26T09:55:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10784.102676 secs 2024-07-26T09:55:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10766.363065 secs 2024-07-26T09:55:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10784.102676 secs 2024-07-26T09:55:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:37 smithi042 ceph-mon[32775]: pgmap v6281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:37 smithi164 ceph-mon[36970]: pgmap v6281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:37 smithi163 ceph-mon[36873]: pgmap v6281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:39 smithi042 ceph-mon[32775]: pgmap v6282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:55:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:39 smithi164 ceph-mon[36970]: pgmap v6282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:55:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:39 smithi163 ceph-mon[36873]: pgmap v6282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:55:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:41 smithi042 ceph-mon[32775]: pgmap v6283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10771.363212 secs 2024-07-26T09:55:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10789.102803 secs 2024-07-26T09:55:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:41 smithi164 ceph-mon[36970]: pgmap v6283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10771.363212 secs 2024-07-26T09:55:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10789.102803 secs 2024-07-26T09:55:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:41 smithi163 ceph-mon[36873]: pgmap v6283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10771.363212 secs 2024-07-26T09:55:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10789.102803 secs 2024-07-26T09:55:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:42.329+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:55:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:42.329+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:55:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:42.329+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:55:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:43 smithi042 ceph-mon[32775]: pgmap v6284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:43 smithi164 ceph-mon[36970]: pgmap v6284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:43 smithi163 ceph-mon[36873]: pgmap v6284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:45 smithi042 ceph-mon[32775]: pgmap v6285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:45 smithi164 ceph-mon[36970]: pgmap v6285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:45 smithi163 ceph-mon[36873]: pgmap v6285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10776.363341 secs 2024-07-26T09:55:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10794.102924 secs 2024-07-26T09:55:47.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10776.363341 secs 2024-07-26T09:55:47.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10794.102924 secs 2024-07-26T09:55:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10776.363341 secs 2024-07-26T09:55:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10794.102924 secs 2024-07-26T09:55:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:47 smithi042 ceph-mon[32775]: pgmap v6286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:47 smithi164 ceph-mon[36970]: pgmap v6286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:47 smithi163 ceph-mon[36873]: pgmap v6286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:49 smithi042 ceph-mon[32775]: pgmap v6287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:55:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:49 smithi164 ceph-mon[36970]: pgmap v6287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:55:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:49 smithi163 ceph-mon[36873]: pgmap v6287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:55:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:51 smithi042 ceph-mon[32775]: pgmap v6288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10781.363482 secs 2024-07-26T09:55:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10799.103050 secs 2024-07-26T09:55:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:51 smithi164 ceph-mon[36970]: pgmap v6288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10781.363482 secs 2024-07-26T09:55:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10799.103050 secs 2024-07-26T09:55:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:51 smithi163 ceph-mon[36873]: pgmap v6288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10781.363482 secs 2024-07-26T09:55:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10799.103050 secs 2024-07-26T09:55:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:53 smithi042 ceph-mon[32775]: pgmap v6289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:53 smithi164 ceph-mon[36970]: pgmap v6289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:53 smithi163 ceph-mon[36873]: pgmap v6289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:55 smithi042 ceph-mon[32775]: pgmap v6290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:55 smithi164 ceph-mon[36970]: pgmap v6290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:55 smithi163 ceph-mon[36873]: pgmap v6290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:55:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10786.363602 secs 2024-07-26T09:55:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10804.103167 secs 2024-07-26T09:55:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10786.363602 secs 2024-07-26T09:55:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10804.103167 secs 2024-07-26T09:55:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10786.363602 secs 2024-07-26T09:55:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10804.103167 secs 2024-07-26T09:55:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:57.335+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:55:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:57.335+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:55:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:57.335+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:55:57.879 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:57.877+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T09:55:57.879 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:57.877+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T09:55:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:57.877+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff678003990 tx=0x7ff67804b040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T09:55:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:57.877+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff678003990 tx=0x7ff67804b040 comp rx=0 tx=0).stop 2024-07-26T09:55:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:57.877+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T09:55:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:57.877+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T09:55:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:57.877+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T09:55:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:57.877+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T09:55:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:57.877+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T09:55:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:57.877+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff678009a60 tx=0x7ff67803e070 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T09:55:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:57 smithi042 ceph-mon[32775]: pgmap v6291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:57 smithi164 ceph-mon[36970]: pgmap v6291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:57 smithi163 ceph-mon[36873]: pgmap v6291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:55:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:59.544+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:55:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:59.544+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:55:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:59.544+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:55:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:55:59.544+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:56:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:55:59 smithi042 ceph-mon[32775]: pgmap v6292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:56:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:55:59 smithi164 ceph-mon[36970]: pgmap v6292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:56:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:55:59 smithi163 ceph-mon[36873]: pgmap v6292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:56:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:01 smithi042 ceph-mon[32775]: pgmap v6293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10791.363730 secs 2024-07-26T09:56:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10809.103256 secs 2024-07-26T09:56:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:01 smithi164 ceph-mon[36970]: pgmap v6293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10791.363730 secs 2024-07-26T09:56:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10809.103256 secs 2024-07-26T09:56:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:01 smithi163 ceph-mon[36873]: pgmap v6293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10791.363730 secs 2024-07-26T09:56:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10809.103256 secs 2024-07-26T09:56:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:03 smithi042 ceph-mon[32775]: pgmap v6294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:03 smithi164 ceph-mon[36970]: pgmap v6294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:03 smithi163 ceph-mon[36873]: pgmap v6294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:05 smithi042 ceph-mon[32775]: pgmap v6295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:05 smithi164 ceph-mon[36970]: pgmap v6295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:05 smithi163 ceph-mon[36873]: pgmap v6295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10796.363866 secs 2024-07-26T09:56:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10814.103391 secs 2024-07-26T09:56:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10796.363866 secs 2024-07-26T09:56:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10814.103391 secs 2024-07-26T09:56:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10796.363866 secs 2024-07-26T09:56:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10814.103391 secs 2024-07-26T09:56:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:07 smithi042 ceph-mon[32775]: pgmap v6296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:07 smithi164 ceph-mon[36970]: pgmap v6296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:07 smithi163 ceph-mon[36873]: pgmap v6296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:09 smithi042 ceph-mon[32775]: pgmap v6297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:56:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:09 smithi164 ceph-mon[36970]: pgmap v6297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:56:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:09 smithi163 ceph-mon[36873]: pgmap v6297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:56:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:11 smithi042 ceph-mon[32775]: pgmap v6298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10801.363999 secs 2024-07-26T09:56:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10819.103499 secs 2024-07-26T09:56:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:11 smithi164 ceph-mon[36970]: pgmap v6298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10801.363999 secs 2024-07-26T09:56:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10819.103499 secs 2024-07-26T09:56:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:11 smithi163 ceph-mon[36873]: pgmap v6298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10801.363999 secs 2024-07-26T09:56:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10819.103499 secs 2024-07-26T09:56:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:56:12.345+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:56:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:56:12.345+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:56:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:56:12.345+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:56:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:13 smithi042 ceph-mon[32775]: pgmap v6299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:13 smithi164 ceph-mon[36970]: pgmap v6299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:13 smithi163 ceph-mon[36873]: pgmap v6299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:15 smithi042 ceph-mon[32775]: pgmap v6300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:15 smithi164 ceph-mon[36970]: pgmap v6300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:15 smithi163 ceph-mon[36873]: pgmap v6300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10806.364147 secs 2024-07-26T09:56:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10824.103619 secs 2024-07-26T09:56:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:56:17.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10806.364147 secs 2024-07-26T09:56:17.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10824.103619 secs 2024-07-26T09:56:17.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:56:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10806.364147 secs 2024-07-26T09:56:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10824.103619 secs 2024-07-26T09:56:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:56:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:17 smithi042 ceph-mon[32775]: pgmap v6301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:56:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:56:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:56:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:17 smithi164 ceph-mon[36970]: pgmap v6301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:18.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:56:18.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:56:18.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:56:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:17 smithi163 ceph-mon[36873]: pgmap v6301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:56:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:56:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:56:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:18 smithi042 ceph-mon[32775]: pgmap v6302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:56:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:18 smithi042 ceph-mon[32775]: pgmap v6303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:56:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:18 smithi164 ceph-mon[36970]: pgmap v6302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:56:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:18 smithi164 ceph-mon[36970]: pgmap v6303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:56:19.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:18 smithi163 ceph-mon[36873]: pgmap v6302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:56:19.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:18 smithi163 ceph-mon[36873]: pgmap v6303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:56:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:20 smithi042 ceph-mon[32775]: pgmap v6304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:56:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:20 smithi164 ceph-mon[36970]: pgmap v6304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:56:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:20 smithi163 ceph-mon[36873]: pgmap v6304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:56:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10811.364255 secs 2024-07-26T09:56:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10829.103772 secs 2024-07-26T09:56:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10811.364255 secs 2024-07-26T09:56:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10829.103772 secs 2024-07-26T09:56:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10811.364255 secs 2024-07-26T09:56:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10829.103772 secs 2024-07-26T09:56:23.153 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:22 smithi042 ceph-mon[32775]: pgmap v6305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:56:23.266 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:22 smithi163 ceph-mon[36873]: pgmap v6305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:56:23.266 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:22 smithi164 ceph-mon[36970]: pgmap v6305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:56:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:24 smithi164 ceph-mon[36970]: pgmap v6306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:56:25.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:24 smithi163 ceph-mon[36873]: pgmap v6306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:56:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:24 smithi042 ceph-mon[32775]: pgmap v6306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:56:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:26 smithi164 ceph-mon[36970]: pgmap v6307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:56:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10816.364325 secs 2024-07-26T09:56:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10834.103913 secs 2024-07-26T09:56:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:26 smithi163 ceph-mon[36873]: pgmap v6307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:56:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10816.364325 secs 2024-07-26T09:56:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10834.103913 secs 2024-07-26T09:56:27.352 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:56:27.350+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:56:27.352 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:56:27.350+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:56:27.352 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:56:27.350+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:56:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:26 smithi042 ceph-mon[32775]: pgmap v6307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:56:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10816.364325 secs 2024-07-26T09:56:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10834.103913 secs 2024-07-26T09:56:29.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:28 smithi164 ceph-mon[36970]: pgmap v6308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:56:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:28 smithi163 ceph-mon[36873]: pgmap v6308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:56:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:28 smithi042 ceph-mon[32775]: pgmap v6308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:56:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:30 smithi164 ceph-mon[36970]: pgmap v6309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:31.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:30 smithi163 ceph-mon[36873]: pgmap v6309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:30 smithi042 ceph-mon[32775]: pgmap v6309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10821.364447 secs 2024-07-26T09:56:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10839.104035 secs 2024-07-26T09:56:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10821.364447 secs 2024-07-26T09:56:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10839.104035 secs 2024-07-26T09:56:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10821.364447 secs 2024-07-26T09:56:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10839.104035 secs 2024-07-26T09:56:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:32 smithi164 ceph-mon[36970]: pgmap v6310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:32 smithi163 ceph-mon[36873]: pgmap v6310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:32 smithi042 ceph-mon[32775]: pgmap v6310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:34 smithi164 ceph-mon[36970]: pgmap v6311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:34 smithi163 ceph-mon[36873]: pgmap v6311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:34 smithi042 ceph-mon[32775]: pgmap v6311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:36 smithi164 ceph-mon[36970]: pgmap v6312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10826.364562 secs 2024-07-26T09:56:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10844.104137 secs 2024-07-26T09:56:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:36 smithi163 ceph-mon[36873]: pgmap v6312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10826.364562 secs 2024-07-26T09:56:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10844.104137 secs 2024-07-26T09:56:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:36 smithi042 ceph-mon[32775]: pgmap v6312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10826.364562 secs 2024-07-26T09:56:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10844.104137 secs 2024-07-26T09:56:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:38 smithi164 ceph-mon[36970]: pgmap v6313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:38 smithi163 ceph-mon[36873]: pgmap v6313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:38 smithi042 ceph-mon[32775]: pgmap v6313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:41.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:40 smithi164 ceph-mon[36970]: pgmap v6314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:41.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:40 smithi163 ceph-mon[36873]: pgmap v6314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:40 smithi042 ceph-mon[32775]: pgmap v6314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10831.364677 secs 2024-07-26T09:56:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10849.104296 secs 2024-07-26T09:56:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10831.364677 secs 2024-07-26T09:56:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10849.104296 secs 2024-07-26T09:56:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:56:42.360+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:56:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:56:42.360+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:56:42.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:56:42.360+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:56:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10831.364677 secs 2024-07-26T09:56:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10849.104296 secs 2024-07-26T09:56:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:42 smithi164 ceph-mon[36970]: pgmap v6315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:42 smithi163 ceph-mon[36873]: pgmap v6315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:42 smithi042 ceph-mon[32775]: pgmap v6315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:44 smithi164 ceph-mon[36970]: pgmap v6316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:44 smithi163 ceph-mon[36873]: pgmap v6316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:45.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:44 smithi042 ceph-mon[32775]: pgmap v6316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:47.199 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:46 smithi042 ceph-mon[32775]: pgmap v6317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:47.199 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10836.364821 secs 2024-07-26T09:56:47.199 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10854.104444 secs 2024-07-26T09:56:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:46 smithi164 ceph-mon[36970]: pgmap v6317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10836.364821 secs 2024-07-26T09:56:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10854.104444 secs 2024-07-26T09:56:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:46 smithi163 ceph-mon[36873]: pgmap v6317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10836.364821 secs 2024-07-26T09:56:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10854.104444 secs 2024-07-26T09:56:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:48 smithi164 ceph-mon[36970]: pgmap v6318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:48 smithi163 ceph-mon[36873]: pgmap v6318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:48 smithi042 ceph-mon[32775]: pgmap v6318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:50 smithi164 ceph-mon[36970]: pgmap v6319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:50 smithi163 ceph-mon[36873]: pgmap v6319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:50 smithi042 ceph-mon[32775]: pgmap v6319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10841.364907 secs 2024-07-26T09:56:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10859.104590 secs 2024-07-26T09:56:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10841.364907 secs 2024-07-26T09:56:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10859.104590 secs 2024-07-26T09:56:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10841.364907 secs 2024-07-26T09:56:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10859.104590 secs 2024-07-26T09:56:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:52 smithi164 ceph-mon[36970]: pgmap v6320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:52 smithi163 ceph-mon[36873]: pgmap v6320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:52 smithi042 ceph-mon[32775]: pgmap v6320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:54 smithi164 ceph-mon[36970]: pgmap v6321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:54 smithi163 ceph-mon[36873]: pgmap v6321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:54 smithi042 ceph-mon[32775]: pgmap v6321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:56 smithi164 ceph-mon[36970]: pgmap v6322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10846.365023 secs 2024-07-26T09:56:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10864.104683 secs 2024-07-26T09:56:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:56 smithi163 ceph-mon[36873]: pgmap v6322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10846.365023 secs 2024-07-26T09:56:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10864.104683 secs 2024-07-26T09:56:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:56:57.364+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:56:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:56:57.364+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:56:57.367 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:56:57.364+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:56:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:56 smithi042 ceph-mon[32775]: pgmap v6322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:56:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10846.365023 secs 2024-07-26T09:56:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10864.104683 secs 2024-07-26T09:56:59.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:56:58 smithi164 ceph-mon[36970]: pgmap v6323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:56:58 smithi163 ceph-mon[36873]: pgmap v6323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:56:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:56:58 smithi042 ceph-mon[32775]: pgmap v6323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:00 smithi163 ceph-mon[36873]: pgmap v6324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:00 smithi042 ceph-mon[32775]: pgmap v6324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:00 smithi164 ceph-mon[36970]: pgmap v6324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10851.365165 secs 2024-07-26T09:57:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10869.104814 secs 2024-07-26T09:57:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10851.365165 secs 2024-07-26T09:57:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10869.104814 secs 2024-07-26T09:57:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10851.365165 secs 2024-07-26T09:57:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10869.104814 secs 2024-07-26T09:57:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:02 smithi163 ceph-mon[36873]: pgmap v6325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:02 smithi042 ceph-mon[32775]: pgmap v6325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:02 smithi164 ceph-mon[36970]: pgmap v6325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:04 smithi163 ceph-mon[36873]: pgmap v6326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:04 smithi042 ceph-mon[32775]: pgmap v6326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:04 smithi164 ceph-mon[36970]: pgmap v6326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:06 smithi163 ceph-mon[36873]: pgmap v6327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:07.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10856.365301 secs 2024-07-26T09:57:07.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10874.104956 secs 2024-07-26T09:57:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:06 smithi042 ceph-mon[32775]: pgmap v6327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10856.365301 secs 2024-07-26T09:57:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10874.104956 secs 2024-07-26T09:57:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:06 smithi164 ceph-mon[36970]: pgmap v6327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10856.365301 secs 2024-07-26T09:57:07.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10874.104956 secs 2024-07-26T09:57:09.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:08 smithi163 ceph-mon[36873]: pgmap v6328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:08 smithi042 ceph-mon[32775]: pgmap v6328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:08 smithi164 ceph-mon[36970]: pgmap v6328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:10 smithi163 ceph-mon[36873]: pgmap v6329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:10 smithi042 ceph-mon[32775]: pgmap v6329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:10 smithi164 ceph-mon[36970]: pgmap v6329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10861.365430 secs 2024-07-26T09:57:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10879.105075 secs 2024-07-26T09:57:12.376 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:57:12.374+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:57:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:57:12.374+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:57:12.377 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:57:12.374+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:57:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10861.365430 secs 2024-07-26T09:57:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10879.105075 secs 2024-07-26T09:57:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10861.365430 secs 2024-07-26T09:57:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10879.105075 secs 2024-07-26T09:57:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:13 smithi163 ceph-mon[36873]: pgmap v6330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:13 smithi042 ceph-mon[32775]: pgmap v6330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:13 smithi164 ceph-mon[36970]: pgmap v6330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:14 smithi042 ceph-mon[32775]: pgmap v6331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:14 smithi164 ceph-mon[36970]: pgmap v6331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:14.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:14 smithi163 ceph-mon[36873]: pgmap v6331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:16 smithi042 ceph-mon[32775]: pgmap v6332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:16 smithi164 ceph-mon[36970]: pgmap v6332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:16 smithi163 ceph-mon[36873]: pgmap v6332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:17.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10866.365516 secs 2024-07-26T09:57:17.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10884.105196 secs 2024-07-26T09:57:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10866.365516 secs 2024-07-26T09:57:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10884.105196 secs 2024-07-26T09:57:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10866.365516 secs 2024-07-26T09:57:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10884.105196 secs 2024-07-26T09:57:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:18 smithi042 ceph-mon[32775]: pgmap v6333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:57:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:57:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:57:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:18 smithi042 ceph-mon[32775]: pgmap v6334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:57:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:18 smithi042 ceph-mon[32775]: pgmap v6335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:57:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:57:18.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:18 smithi164 ceph-mon[36970]: pgmap v6333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:18.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:57:18.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:57:18.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:57:18.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:18 smithi164 ceph-mon[36970]: pgmap v6334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:57:18.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:18 smithi164 ceph-mon[36970]: pgmap v6335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:57:18.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:57:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:18 smithi163 ceph-mon[36873]: pgmap v6333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:57:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:57:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:57:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:18 smithi163 ceph-mon[36873]: pgmap v6334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:57:18.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:18 smithi163 ceph-mon[36873]: pgmap v6335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:57:18.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:57:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:20 smithi042 ceph-mon[32775]: pgmap v6336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:57:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:20 smithi164 ceph-mon[36970]: pgmap v6336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:57:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:20 smithi163 ceph-mon[36873]: pgmap v6336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:57:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10871.365643 secs 2024-07-26T09:57:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10889.105277 secs 2024-07-26T09:57:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10871.365643 secs 2024-07-26T09:57:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10889.105277 secs 2024-07-26T09:57:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10871.365643 secs 2024-07-26T09:57:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10889.105277 secs 2024-07-26T09:57:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:22 smithi042 ceph-mon[32775]: pgmap v6337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:57:23.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:22 smithi164 ceph-mon[36970]: pgmap v6337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:57:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:22 smithi163 ceph-mon[36873]: pgmap v6337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:57:25.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:24 smithi042 ceph-mon[32775]: pgmap v6338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:57:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:24 smithi164 ceph-mon[36970]: pgmap v6338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:57:25.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:24 smithi163 ceph-mon[36873]: pgmap v6338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:57:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:26 smithi042 ceph-mon[32775]: pgmap v6339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:57:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10876.365743 secs 2024-07-26T09:57:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10894.105414 secs 2024-07-26T09:57:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:26 smithi164 ceph-mon[36970]: pgmap v6339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:57:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10876.365743 secs 2024-07-26T09:57:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10894.105414 secs 2024-07-26T09:57:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:26 smithi163 ceph-mon[36873]: pgmap v6339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:57:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10876.365743 secs 2024-07-26T09:57:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10894.105414 secs 2024-07-26T09:57:27.381 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:57:27.379+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:57:27.382 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:57:27.379+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:57:27.382 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:57:27.379+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:57:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:28 smithi042 ceph-mon[32775]: pgmap v6340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:57:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:28 smithi164 ceph-mon[36970]: pgmap v6340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:57:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:28 smithi163 ceph-mon[36873]: pgmap v6340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:57:29.561 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:57:29.558+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:57:29.561 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:57:29.558+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:57:29.561 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:57:29.559+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:57:29.561 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:57:29.559+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:57:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:30 smithi042 ceph-mon[32775]: pgmap v6341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:30 smithi164 ceph-mon[36970]: pgmap v6341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:31.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:30 smithi163 ceph-mon[36873]: pgmap v6341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10881.365872 secs 2024-07-26T09:57:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10899.105555 secs 2024-07-26T09:57:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10881.365872 secs 2024-07-26T09:57:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10899.105555 secs 2024-07-26T09:57:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10881.365872 secs 2024-07-26T09:57:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10899.105555 secs 2024-07-26T09:57:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:32 smithi042 ceph-mon[32775]: pgmap v6342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:32 smithi164 ceph-mon[36970]: pgmap v6342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:32 smithi163 ceph-mon[36873]: pgmap v6342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:35.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:34 smithi042 ceph-mon[32775]: pgmap v6343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:57:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:34 smithi164 ceph-mon[36970]: pgmap v6343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:57:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:34 smithi163 ceph-mon[36873]: pgmap v6343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:57:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:36 smithi042 ceph-mon[32775]: pgmap v6344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10886.365994 secs 2024-07-26T09:57:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10904.105707 secs 2024-07-26T09:57:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:36 smithi164 ceph-mon[36970]: pgmap v6344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10886.365994 secs 2024-07-26T09:57:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10904.105707 secs 2024-07-26T09:57:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:36 smithi163 ceph-mon[36873]: pgmap v6344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10886.365994 secs 2024-07-26T09:57:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10904.105707 secs 2024-07-26T09:57:39.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:38 smithi042 ceph-mon[32775]: pgmap v6345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:38 smithi164 ceph-mon[36970]: pgmap v6345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:38 smithi163 ceph-mon[36873]: pgmap v6345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:41.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:40 smithi042 ceph-mon[32775]: pgmap v6346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:40 smithi164 ceph-mon[36970]: pgmap v6346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:41.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:40 smithi163 ceph-mon[36873]: pgmap v6346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10891.366117 secs 2024-07-26T09:57:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10909.105823 secs 2024-07-26T09:57:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10891.366117 secs 2024-07-26T09:57:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10909.105823 secs 2024-07-26T09:57:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10891.366117 secs 2024-07-26T09:57:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10909.105823 secs 2024-07-26T09:57:42.391 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:57:42.389+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:57:42.391 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:57:42.389+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:57:42.391 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:57:42.389+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:57:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:42 smithi042 ceph-mon[32775]: pgmap v6347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:42 smithi164 ceph-mon[36970]: pgmap v6347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:42 smithi163 ceph-mon[36873]: pgmap v6347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:44 smithi042 ceph-mon[32775]: pgmap v6348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:57:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:44 smithi164 ceph-mon[36970]: pgmap v6348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:57:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:44 smithi163 ceph-mon[36873]: pgmap v6348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:57:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:46 smithi042 ceph-mon[32775]: pgmap v6349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10896.366261 secs 2024-07-26T09:57:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10914.105959 secs 2024-07-26T09:57:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:46 smithi164 ceph-mon[36970]: pgmap v6349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10896.366261 secs 2024-07-26T09:57:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10914.105959 secs 2024-07-26T09:57:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:46 smithi163 ceph-mon[36873]: pgmap v6349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10896.366261 secs 2024-07-26T09:57:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10914.105959 secs 2024-07-26T09:57:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:48 smithi042 ceph-mon[32775]: pgmap v6350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:48 smithi164 ceph-mon[36970]: pgmap v6350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:48 smithi163 ceph-mon[36873]: pgmap v6350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:50 smithi042 ceph-mon[32775]: pgmap v6351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:51.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:50 smithi164 ceph-mon[36970]: pgmap v6351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:50 smithi163 ceph-mon[36873]: pgmap v6351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10901.366294 secs 2024-07-26T09:57:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10919.106044 secs 2024-07-26T09:57:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10901.366294 secs 2024-07-26T09:57:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10919.106044 secs 2024-07-26T09:57:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10901.366294 secs 2024-07-26T09:57:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10919.106044 secs 2024-07-26T09:57:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:52 smithi042 ceph-mon[32775]: pgmap v6352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:52 smithi164 ceph-mon[36970]: pgmap v6352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:52 smithi163 ceph-mon[36873]: pgmap v6352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:54 smithi042 ceph-mon[32775]: pgmap v6353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:57:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:54 smithi164 ceph-mon[36970]: pgmap v6353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:57:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:54 smithi163 ceph-mon[36873]: pgmap v6353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:57:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:56 smithi042 ceph-mon[32775]: pgmap v6354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10906.366401 secs 2024-07-26T09:57:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10924.106108 secs 2024-07-26T09:57:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:56 smithi164 ceph-mon[36970]: pgmap v6354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10906.366401 secs 2024-07-26T09:57:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10924.106108 secs 2024-07-26T09:57:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:56 smithi163 ceph-mon[36873]: pgmap v6354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:57:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10906.366401 secs 2024-07-26T09:57:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10924.106108 secs 2024-07-26T09:57:57.396 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:57:57.394+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:57:57.397 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:57:57.394+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:57:57.397 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:57:57.394+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:57:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:57:58 smithi164 ceph-mon[36970]: pgmap v6355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:57:58 smithi163 ceph-mon[36873]: pgmap v6355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:57:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:57:58 smithi042 ceph-mon[32775]: pgmap v6355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:00 smithi164 ceph-mon[36970]: pgmap v6356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:00 smithi163 ceph-mon[36873]: pgmap v6356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:00 smithi042 ceph-mon[32775]: pgmap v6356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10911.366525 secs 2024-07-26T09:58:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10929.106231 secs 2024-07-26T09:58:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10911.366525 secs 2024-07-26T09:58:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10929.106231 secs 2024-07-26T09:58:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10911.366525 secs 2024-07-26T09:58:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10929.106231 secs 2024-07-26T09:58:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:02 smithi164 ceph-mon[36970]: pgmap v6357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:02 smithi163 ceph-mon[36873]: pgmap v6357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:02 smithi042 ceph-mon[32775]: pgmap v6357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:04 smithi164 ceph-mon[36970]: pgmap v6358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:58:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:04 smithi163 ceph-mon[36873]: pgmap v6358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:58:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:04 smithi042 ceph-mon[32775]: pgmap v6358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:58:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:06 smithi164 ceph-mon[36970]: pgmap v6359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10916.366605 secs 2024-07-26T09:58:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10934.106355 secs 2024-07-26T09:58:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:06 smithi163 ceph-mon[36873]: pgmap v6359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10916.366605 secs 2024-07-26T09:58:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10934.106355 secs 2024-07-26T09:58:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:06 smithi042 ceph-mon[32775]: pgmap v6359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10916.366605 secs 2024-07-26T09:58:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10934.106355 secs 2024-07-26T09:58:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:08 smithi164 ceph-mon[36970]: pgmap v6360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:09.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:08 smithi163 ceph-mon[36873]: pgmap v6360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:08 smithi042 ceph-mon[32775]: pgmap v6360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:11.185 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:10 smithi042 ceph-mon[32775]: pgmap v6361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:10 smithi164 ceph-mon[36970]: pgmap v6361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:10 smithi163 ceph-mon[36873]: pgmap v6361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10921.366725 secs 2024-07-26T09:58:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10939.106501 secs 2024-07-26T09:58:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10921.366725 secs 2024-07-26T09:58:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10939.106501 secs 2024-07-26T09:58:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10921.366725 secs 2024-07-26T09:58:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10939.106501 secs 2024-07-26T09:58:12.406 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:58:12.405+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:58:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:58:12.405+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:58:12.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:58:12.405+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:58:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:12 smithi164 ceph-mon[36970]: pgmap v6362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:12 smithi163 ceph-mon[36873]: pgmap v6362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:12 smithi042 ceph-mon[32775]: pgmap v6362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:14 smithi164 ceph-mon[36970]: pgmap v6363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:58:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:14 smithi163 ceph-mon[36873]: pgmap v6363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:58:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:14 smithi042 ceph-mon[32775]: pgmap v6363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:58:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:16 smithi164 ceph-mon[36970]: pgmap v6364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10926.366841 secs 2024-07-26T09:58:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10944.106614 secs 2024-07-26T09:58:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:16 smithi163 ceph-mon[36873]: pgmap v6364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:17.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10926.366841 secs 2024-07-26T09:58:17.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10944.106614 secs 2024-07-26T09:58:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:16 smithi042 ceph-mon[32775]: pgmap v6364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10926.366841 secs 2024-07-26T09:58:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10944.106614 secs 2024-07-26T09:58:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:18 smithi042 ceph-mon[32775]: pgmap v6365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:58:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:18 smithi164 ceph-mon[36970]: pgmap v6365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:18.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:58:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:18 smithi163 ceph-mon[36873]: pgmap v6365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:58:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:58:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:58:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:19 smithi042 ceph-mon[32775]: pgmap v6366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:58:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:19 smithi042 ceph-mon[32775]: pgmap v6367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:58:19.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:58:19.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:58:19.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:58:19.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:19 smithi164 ceph-mon[36970]: pgmap v6366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:58:19.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:19 smithi164 ceph-mon[36970]: pgmap v6367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:58:19.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:58:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:58:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:58:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:19 smithi163 ceph-mon[36873]: pgmap v6366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T09:58:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:19 smithi163 ceph-mon[36873]: pgmap v6367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:58:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:58:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:21 smithi042 ceph-mon[32775]: pgmap v6368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:58:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10931.366924 secs 2024-07-26T09:58:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:21 smithi164 ceph-mon[36970]: pgmap v6368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:58:21.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10931.366924 secs 2024-07-26T09:58:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:21 smithi163 ceph-mon[36873]: pgmap v6368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:58:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10931.366924 secs 2024-07-26T09:58:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10949.106735 secs 2024-07-26T09:58:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10949.106735 secs 2024-07-26T09:58:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10949.106735 secs 2024-07-26T09:58:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:23 smithi042 ceph-mon[32775]: pgmap v6369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:58:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:23 smithi164 ceph-mon[36970]: pgmap v6369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:58:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:23 smithi163 ceph-mon[36873]: pgmap v6369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:58:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:25 smithi042 ceph-mon[32775]: pgmap v6370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:58:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:25 smithi164 ceph-mon[36970]: pgmap v6370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:58:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:25 smithi163 ceph-mon[36873]: pgmap v6370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:58:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10936.367075 secs 2024-07-26T09:58:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10936.367075 secs 2024-07-26T09:58:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10936.367075 secs 2024-07-26T09:58:27.412 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:58:27.410+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:58:27.412 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:58:27.410+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:58:27.412 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:58:27.410+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:58:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10954.106858 secs 2024-07-26T09:58:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:27 smithi042 ceph-mon[32775]: pgmap v6371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:58:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10954.106858 secs 2024-07-26T09:58:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:27 smithi164 ceph-mon[36970]: pgmap v6371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:58:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10954.106858 secs 2024-07-26T09:58:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:27 smithi163 ceph-mon[36873]: pgmap v6371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:58:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:29 smithi042 ceph-mon[32775]: pgmap v6372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:58:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:29 smithi164 ceph-mon[36970]: pgmap v6372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:58:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:29 smithi163 ceph-mon[36873]: pgmap v6372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:58:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:31 smithi042 ceph-mon[32775]: pgmap v6373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10941.367197 secs 2024-07-26T09:58:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:31 smithi164 ceph-mon[36970]: pgmap v6373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:31.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10941.367197 secs 2024-07-26T09:58:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:31 smithi163 ceph-mon[36873]: pgmap v6373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10941.367197 secs 2024-07-26T09:58:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10959.106985 secs 2024-07-26T09:58:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10959.106985 secs 2024-07-26T09:58:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10959.106985 secs 2024-07-26T09:58:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:33 smithi164 ceph-mon[36970]: pgmap v6374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:33 smithi163 ceph-mon[36873]: pgmap v6374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:33 smithi042 ceph-mon[32775]: pgmap v6374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:35 smithi164 ceph-mon[36970]: pgmap v6375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:35 smithi163 ceph-mon[36873]: pgmap v6375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:35 smithi042 ceph-mon[32775]: pgmap v6375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:36.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10946.367323 secs 2024-07-26T09:58:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10946.367323 secs 2024-07-26T09:58:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10946.367323 secs 2024-07-26T09:58:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10964.107108 secs 2024-07-26T09:58:37.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:37 smithi164 ceph-mon[36970]: pgmap v6376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10964.107108 secs 2024-07-26T09:58:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:37 smithi163 ceph-mon[36873]: pgmap v6376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10964.107108 secs 2024-07-26T09:58:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:37 smithi042 ceph-mon[32775]: pgmap v6376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:39 smithi164 ceph-mon[36970]: pgmap v6377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:58:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:39 smithi163 ceph-mon[36873]: pgmap v6377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:58:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:39 smithi042 ceph-mon[32775]: pgmap v6377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:58:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:41 smithi164 ceph-mon[36970]: pgmap v6378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:41.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10951.367453 secs 2024-07-26T09:58:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:41 smithi163 ceph-mon[36873]: pgmap v6378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10951.367453 secs 2024-07-26T09:58:41.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:41 smithi042 ceph-mon[32775]: pgmap v6378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:41.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10951.367453 secs 2024-07-26T09:58:42.421 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:58:42.419+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:58:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:58:42.419+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:58:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:58:42.419+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:58:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10969.107235 secs 2024-07-26T09:58:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10969.107235 secs 2024-07-26T09:58:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10969.107235 secs 2024-07-26T09:58:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:43 smithi164 ceph-mon[36970]: pgmap v6379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:43 smithi163 ceph-mon[36873]: pgmap v6379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:43 smithi042 ceph-mon[32775]: pgmap v6379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:45 smithi164 ceph-mon[36970]: pgmap v6380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:45 smithi163 ceph-mon[36873]: pgmap v6380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:45 smithi042 ceph-mon[32775]: pgmap v6380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:46.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10956.367578 secs 2024-07-26T09:58:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10956.367578 secs 2024-07-26T09:58:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10956.367578 secs 2024-07-26T09:58:47.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10974.107314 secs 2024-07-26T09:58:47.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:47 smithi164 ceph-mon[36970]: pgmap v6381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10974.107314 secs 2024-07-26T09:58:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:47 smithi163 ceph-mon[36873]: pgmap v6381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10974.107314 secs 2024-07-26T09:58:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:47 smithi042 ceph-mon[32775]: pgmap v6381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:49.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:49 smithi042 ceph-mon[32775]: pgmap v6382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:58:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:49 smithi164 ceph-mon[36970]: pgmap v6382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:58:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:49 smithi163 ceph-mon[36873]: pgmap v6382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:58:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:51 smithi164 ceph-mon[36970]: pgmap v6383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:51.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10961.367698 secs 2024-07-26T09:58:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:51 smithi163 ceph-mon[36873]: pgmap v6383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10961.367698 secs 2024-07-26T09:58:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:51 smithi042 ceph-mon[32775]: pgmap v6383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10961.367698 secs 2024-07-26T09:58:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10979.107434 secs 2024-07-26T09:58:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10979.107434 secs 2024-07-26T09:58:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10979.107434 secs 2024-07-26T09:58:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:53 smithi164 ceph-mon[36970]: pgmap v6384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:53 smithi163 ceph-mon[36873]: pgmap v6384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:53 smithi042 ceph-mon[32775]: pgmap v6384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:55 smithi164 ceph-mon[36970]: pgmap v6385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:55 smithi163 ceph-mon[36873]: pgmap v6385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:55 smithi042 ceph-mon[32775]: pgmap v6385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:58:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10966.367819 secs 2024-07-26T09:58:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10966.367819 secs 2024-07-26T09:58:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10966.367819 secs 2024-07-26T09:58:57.427 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:58:57.425+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:58:57.427 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:58:57.425+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:58:57.427 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:58:57.425+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:58:57.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10984.107556 secs 2024-07-26T09:58:57.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:57 smithi164 ceph-mon[36970]: pgmap v6386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10984.107556 secs 2024-07-26T09:58:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:57 smithi163 ceph-mon[36873]: pgmap v6386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:57.777 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10984.107556 secs 2024-07-26T09:58:57.777 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:57 smithi042 ceph-mon[32775]: pgmap v6386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:58:59.575 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:58:59.572+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:58:59.575 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:58:59.572+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:58:59.575 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:58:59.573+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T09:58:59.575 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:58:59.573+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T09:58:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:58:59 smithi164 ceph-mon[36970]: pgmap v6387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:58:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:58:59 smithi163 ceph-mon[36873]: pgmap v6387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:58:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:58:59 smithi042 ceph-mon[32775]: pgmap v6387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:59:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:01 smithi163 ceph-mon[36873]: pgmap v6388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10971.367961 secs 2024-07-26T09:59:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:01 smithi042 ceph-mon[32775]: pgmap v6388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10971.367961 secs 2024-07-26T09:59:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:01 smithi164 ceph-mon[36970]: pgmap v6388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10971.367961 secs 2024-07-26T09:59:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10989.107677 secs 2024-07-26T09:59:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10989.107677 secs 2024-07-26T09:59:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10989.107677 secs 2024-07-26T09:59:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:03 smithi163 ceph-mon[36873]: pgmap v6389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:03 smithi042 ceph-mon[32775]: pgmap v6389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:03 smithi164 ceph-mon[36970]: pgmap v6389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:05 smithi163 ceph-mon[36873]: pgmap v6390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:05 smithi042 ceph-mon[32775]: pgmap v6390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:05 smithi164 ceph-mon[36970]: pgmap v6390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10976.368091 secs 2024-07-26T09:59:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10976.368091 secs 2024-07-26T09:59:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10976.368091 secs 2024-07-26T09:59:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10994.107796 secs 2024-07-26T09:59:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:07 smithi163 ceph-mon[36873]: pgmap v6391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10994.107796 secs 2024-07-26T09:59:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:07 smithi042 ceph-mon[32775]: pgmap v6391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10994.107796 secs 2024-07-26T09:59:07.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:07 smithi164 ceph-mon[36970]: pgmap v6391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:09 smithi163 ceph-mon[36873]: pgmap v6392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:59:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:09 smithi042 ceph-mon[32775]: pgmap v6392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:59:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:09 smithi164 ceph-mon[36970]: pgmap v6392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:59:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:11 smithi163 ceph-mon[36873]: pgmap v6393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10981.368258 secs 2024-07-26T09:59:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:11 smithi042 ceph-mon[32775]: pgmap v6393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10981.368258 secs 2024-07-26T09:59:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:11 smithi164 ceph-mon[36970]: pgmap v6393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10981.368258 secs 2024-07-26T09:59:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:59:12.435+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:59:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:59:12.435+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:59:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:59:12.435+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:59:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10999.107893 secs 2024-07-26T09:59:12.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10999.107893 secs 2024-07-26T09:59:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10999.107893 secs 2024-07-26T09:59:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:13 smithi042 ceph-mon[32775]: pgmap v6394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:13.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:13 smithi164 ceph-mon[36970]: pgmap v6394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:13 smithi163 ceph-mon[36873]: pgmap v6394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:15 smithi042 ceph-mon[32775]: pgmap v6395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:15 smithi164 ceph-mon[36970]: pgmap v6395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:15 smithi163 ceph-mon[36873]: pgmap v6395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10986.368382 secs 2024-07-26T09:59:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10986.368382 secs 2024-07-26T09:59:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10986.368382 secs 2024-07-26T09:59:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11004.108037 secs 2024-07-26T09:59:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:17 smithi042 ceph-mon[32775]: pgmap v6396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11004.108037 secs 2024-07-26T09:59:17.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:17 smithi163 ceph-mon[36873]: pgmap v6396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11004.108037 secs 2024-07-26T09:59:17.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:17 smithi164 ceph-mon[36970]: pgmap v6396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:59:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:59:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T09:59:19.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: pgmap v6397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:59:19.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.798 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.799 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:59:19.799 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:59:19.799 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:59:19.799 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:59:19.799 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:59:19.799 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:59:19.799 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:59:19.799 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:59:19.799 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: pgmap v6398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:59:19.799 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: pgmap v6399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:59:19.799 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:59:19.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: pgmap v6397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:59:19.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:19.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:59:19.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:59:19.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:59:19.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:59:19.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:59:19.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:59:19.976 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:59:19.976 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:59:19.976 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: pgmap v6398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:59:19.976 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: pgmap v6399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:59:19.976 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:59:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: pgmap v6397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:59:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:20.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:20.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:20.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:20.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:20.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:20.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:20.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:20.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:20.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:20.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T09:59:20.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T09:59:20.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T09:59:20.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T09:59:20.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T09:59:20.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T09:59:20.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T09:59:20.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T09:59:20.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T09:59:20.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: pgmap v6398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T09:59:20.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: pgmap v6399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T09:59:20.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T09:59:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:21 smithi164 ceph-mon[36970]: pgmap v6400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:59:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10991.368468 secs 2024-07-26T09:59:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11009.108107 secs 2024-07-26T09:59:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:21 smithi163 ceph-mon[36873]: pgmap v6400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:59:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10991.368468 secs 2024-07-26T09:59:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11009.108107 secs 2024-07-26T09:59:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:21 smithi042 ceph-mon[32775]: pgmap v6400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:59:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10991.368468 secs 2024-07-26T09:59:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11009.108107 secs 2024-07-26T09:59:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:23 smithi164 ceph-mon[36970]: pgmap v6401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:59:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:23 smithi163 ceph-mon[36873]: pgmap v6401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:59:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:23 smithi042 ceph-mon[32775]: pgmap v6401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T09:59:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:25 smithi164 ceph-mon[36970]: pgmap v6402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:59:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:25 smithi163 ceph-mon[36873]: pgmap v6402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:59:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:25 smithi042 ceph-mon[32775]: pgmap v6402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T09:59:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 10996.368590 secs 2024-07-26T09:59:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11014.108192 secs 2024-07-26T09:59:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 10996.368590 secs 2024-07-26T09:59:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11014.108192 secs 2024-07-26T09:59:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 10996.368590 secs 2024-07-26T09:59:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11014.108192 secs 2024-07-26T09:59:27.442 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:59:27.440+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:59:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:59:27.440+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:59:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:59:27.440+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:59:28.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:27 smithi164 ceph-mon[36970]: pgmap v6403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:59:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:27 smithi163 ceph-mon[36873]: pgmap v6403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:59:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:27 smithi042 ceph-mon[32775]: pgmap v6403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T09:59:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:29 smithi164 ceph-mon[36970]: pgmap v6404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:59:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:29 smithi163 ceph-mon[36873]: pgmap v6404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:59:30.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:29 smithi042 ceph-mon[32775]: pgmap v6404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T09:59:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:31 smithi164 ceph-mon[36970]: pgmap v6405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11001.368734 secs 2024-07-26T09:59:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11019.108285 secs 2024-07-26T09:59:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:31 smithi163 ceph-mon[36873]: pgmap v6405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11001.368734 secs 2024-07-26T09:59:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11019.108285 secs 2024-07-26T09:59:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:31 smithi042 ceph-mon[32775]: pgmap v6405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11001.368734 secs 2024-07-26T09:59:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11019.108285 secs 2024-07-26T09:59:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:33 smithi164 ceph-mon[36970]: pgmap v6406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:33 smithi163 ceph-mon[36873]: pgmap v6406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:33 smithi042 ceph-mon[32775]: pgmap v6406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:35 smithi164 ceph-mon[36970]: pgmap v6407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:35 smithi163 ceph-mon[36873]: pgmap v6407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:35 smithi042 ceph-mon[32775]: pgmap v6407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11006.368854 secs 2024-07-26T09:59:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11024.108407 secs 2024-07-26T09:59:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11006.368854 secs 2024-07-26T09:59:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11024.108407 secs 2024-07-26T09:59:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11006.368854 secs 2024-07-26T09:59:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11024.108407 secs 2024-07-26T09:59:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:37 smithi164 ceph-mon[36970]: pgmap v6408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:37 smithi163 ceph-mon[36873]: pgmap v6408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:37 smithi042 ceph-mon[32775]: pgmap v6408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:39 smithi164 ceph-mon[36970]: pgmap v6409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:59:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:39 smithi163 ceph-mon[36873]: pgmap v6409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:59:40.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:39 smithi042 ceph-mon[32775]: pgmap v6409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:59:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:41 smithi164 ceph-mon[36970]: pgmap v6410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11011.368929 secs 2024-07-26T09:59:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11029.108547 secs 2024-07-26T09:59:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:41 smithi163 ceph-mon[36873]: pgmap v6410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11011.368929 secs 2024-07-26T09:59:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11029.108547 secs 2024-07-26T09:59:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:41 smithi042 ceph-mon[32775]: pgmap v6410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11011.368929 secs 2024-07-26T09:59:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11029.108547 secs 2024-07-26T09:59:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:59:42.450+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:59:42.452 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:59:42.450+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:59:42.453 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:59:42.450+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:59:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:43 smithi042 ceph-mon[32775]: pgmap v6411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:44.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:43 smithi164 ceph-mon[36970]: pgmap v6411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:43 smithi163 ceph-mon[36873]: pgmap v6411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:45 smithi163 ceph-mon[36873]: pgmap v6412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:45 smithi042 ceph-mon[32775]: pgmap v6412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:45 smithi164 ceph-mon[36970]: pgmap v6412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11016.369069 secs 2024-07-26T09:59:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11034.108684 secs 2024-07-26T09:59:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11016.369069 secs 2024-07-26T09:59:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11034.108684 secs 2024-07-26T09:59:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11016.369069 secs 2024-07-26T09:59:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11034.108684 secs 2024-07-26T09:59:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:47 smithi163 ceph-mon[36873]: pgmap v6413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:47 smithi042 ceph-mon[32775]: pgmap v6413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:47 smithi164 ceph-mon[36970]: pgmap v6413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:49 smithi163 ceph-mon[36873]: pgmap v6414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:59:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:49 smithi042 ceph-mon[32775]: pgmap v6414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:59:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:49 smithi164 ceph-mon[36970]: pgmap v6414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T09:59:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:51 smithi163 ceph-mon[36873]: pgmap v6415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11021.369211 secs 2024-07-26T09:59:52.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11039.108804 secs 2024-07-26T09:59:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:51 smithi042 ceph-mon[32775]: pgmap v6415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11021.369211 secs 2024-07-26T09:59:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11039.108804 secs 2024-07-26T09:59:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:51 smithi164 ceph-mon[36970]: pgmap v6415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11021.369211 secs 2024-07-26T09:59:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11039.108804 secs 2024-07-26T09:59:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:53 smithi163 ceph-mon[36873]: pgmap v6416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:53 smithi042 ceph-mon[32775]: pgmap v6416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:53 smithi164 ceph-mon[36970]: pgmap v6416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:55 smithi163 ceph-mon[36873]: pgmap v6417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:55 smithi042 ceph-mon[32775]: pgmap v6417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:55 smithi164 ceph-mon[36970]: pgmap v6417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T09:59:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11026.369256 secs 2024-07-26T09:59:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11044.108953 secs 2024-07-26T09:59:57.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11026.369256 secs 2024-07-26T09:59:57.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11044.108953 secs 2024-07-26T09:59:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11026.369256 secs 2024-07-26T09:59:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11044.108953 secs 2024-07-26T09:59:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:59:57.455+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T09:59:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:59:57.455+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T09:59:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T09:59:57.455+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T09:59:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 09:59:58 smithi163 ceph-mon[36873]: pgmap v6418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 09:59:58 smithi042 ceph-mon[32775]: pgmap v6418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T09:59:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 09:59:58 smithi164 ceph-mon[36970]: pgmap v6418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:00 smithi163 ceph-mon[36873]: pgmap v6419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:00:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:00 smithi042 ceph-mon[32775]: pgmap v6419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:00:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:00 smithi164 ceph-mon[36970]: pgmap v6419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:00:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:02 smithi042 ceph-mon[32775]: pgmap v6420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:02.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11031.369353 secs 2024-07-26T10:00:02.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11049.109031 secs 2024-07-26T10:00:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:02 smithi164 ceph-mon[36970]: pgmap v6420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11031.369353 secs 2024-07-26T10:00:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11049.109031 secs 2024-07-26T10:00:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:02 smithi163 ceph-mon[36873]: pgmap v6420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11031.369353 secs 2024-07-26T10:00:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11049.109031 secs 2024-07-26T10:00:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:04 smithi042 ceph-mon[32775]: pgmap v6421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:04 smithi164 ceph-mon[36970]: pgmap v6421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:04 smithi163 ceph-mon[36873]: pgmap v6421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:06 smithi042 ceph-mon[32775]: pgmap v6422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:06 smithi164 ceph-mon[36970]: pgmap v6422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:06 smithi163 ceph-mon[36873]: pgmap v6422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11036.369479 secs 2024-07-26T10:00:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11054.109174 secs 2024-07-26T10:00:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11036.369479 secs 2024-07-26T10:00:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11054.109174 secs 2024-07-26T10:00:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11036.369479 secs 2024-07-26T10:00:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11054.109174 secs 2024-07-26T10:00:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:08 smithi042 ceph-mon[32775]: pgmap v6423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:08 smithi164 ceph-mon[36970]: pgmap v6423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:08 smithi163 ceph-mon[36873]: pgmap v6423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:10.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:10 smithi042 ceph-mon[32775]: pgmap v6424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:10 smithi164 ceph-mon[36970]: pgmap v6424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:10 smithi163 ceph-mon[36873]: pgmap v6424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:12 smithi042 ceph-mon[32775]: pgmap v6425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11041.369602 secs 2024-07-26T10:00:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11059.109261 secs 2024-07-26T10:00:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:00:12.466+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:00:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:00:12.466+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:00:12.467 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:00:12.466+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:00:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:12 smithi164 ceph-mon[36970]: pgmap v6425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11041.369602 secs 2024-07-26T10:00:12.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11059.109261 secs 2024-07-26T10:00:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:12 smithi163 ceph-mon[36873]: pgmap v6425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11041.369602 secs 2024-07-26T10:00:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11059.109261 secs 2024-07-26T10:00:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:13 smithi042 ceph-mon[32775]: pgmap v6426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:13 smithi164 ceph-mon[36970]: pgmap v6426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:13 smithi163 ceph-mon[36873]: pgmap v6426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:16.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:15 smithi164 ceph-mon[36970]: pgmap v6427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:15 smithi163 ceph-mon[36873]: pgmap v6427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:15 smithi042 ceph-mon[32775]: pgmap v6427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11046.369724 secs 2024-07-26T10:00:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11064.109405 secs 2024-07-26T10:00:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11046.369724 secs 2024-07-26T10:00:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11064.109405 secs 2024-07-26T10:00:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11046.369724 secs 2024-07-26T10:00:17.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11064.109405 secs 2024-07-26T10:00:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:17 smithi164 ceph-mon[36970]: pgmap v6428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:17 smithi163 ceph-mon[36873]: pgmap v6428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:17 smithi042 ceph-mon[32775]: pgmap v6428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:19 smithi164 ceph-mon[36970]: pgmap v6429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:20.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:00:20.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:00:20.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:00:20.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:00:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:19 smithi163 ceph-mon[36873]: pgmap v6429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:00:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:00:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:00:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:00:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:19 smithi042 ceph-mon[32775]: pgmap v6429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:00:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:00:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:00:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:00:21.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:20 smithi164 ceph-mon[36970]: pgmap v6430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:00:21.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:20 smithi164 ceph-mon[36970]: pgmap v6431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:00:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:20 smithi163 ceph-mon[36873]: pgmap v6430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:00:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:20 smithi163 ceph-mon[36873]: pgmap v6431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:00:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:20 smithi042 ceph-mon[32775]: pgmap v6430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:00:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:20 smithi042 ceph-mon[32775]: pgmap v6431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:00:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11051.369829 secs 2024-07-26T10:00:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11069.109561 secs 2024-07-26T10:00:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11051.369829 secs 2024-07-26T10:00:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11069.109561 secs 2024-07-26T10:00:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11051.369829 secs 2024-07-26T10:00:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11069.109561 secs 2024-07-26T10:00:23.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:22 smithi164 ceph-mon[36970]: pgmap v6432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:00:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:22 smithi163 ceph-mon[36873]: pgmap v6432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:00:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:22 smithi042 ceph-mon[32775]: pgmap v6432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:00:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:24 smithi164 ceph-mon[36970]: pgmap v6433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:00:25.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:24 smithi163 ceph-mon[36873]: pgmap v6433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:00:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:24 smithi042 ceph-mon[32775]: pgmap v6433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:00:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:26 smithi163 ceph-mon[36873]: pgmap v6434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:00:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11056.369956 secs 2024-07-26T10:00:27.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11074.109685 secs 2024-07-26T10:00:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:26 smithi042 ceph-mon[32775]: pgmap v6434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:00:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11056.369956 secs 2024-07-26T10:00:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11074.109685 secs 2024-07-26T10:00:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:00:27.471+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:00:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:00:27.471+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:00:27.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:00:27.471+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:00:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:26 smithi164 ceph-mon[36970]: pgmap v6434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:00:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11056.369956 secs 2024-07-26T10:00:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11074.109685 secs 2024-07-26T10:00:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:28 smithi163 ceph-mon[36873]: pgmap v6435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:00:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:28 smithi042 ceph-mon[32775]: pgmap v6435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:00:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:28 smithi164 ceph-mon[36970]: pgmap v6435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:00:31.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:30 smithi163 ceph-mon[36873]: pgmap v6436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:00:31.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:30 smithi042 ceph-mon[32775]: pgmap v6436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:00:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:30 smithi164 ceph-mon[36970]: pgmap v6436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:00:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11061.370103 secs 2024-07-26T10:00:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11079.109791 secs 2024-07-26T10:00:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11061.370103 secs 2024-07-26T10:00:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11079.109791 secs 2024-07-26T10:00:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11061.370103 secs 2024-07-26T10:00:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11079.109791 secs 2024-07-26T10:00:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:32 smithi163 ceph-mon[36873]: pgmap v6437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:32 smithi042 ceph-mon[32775]: pgmap v6437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:32 smithi164 ceph-mon[36970]: pgmap v6437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:34 smithi163 ceph-mon[36873]: pgmap v6438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:34 smithi042 ceph-mon[32775]: pgmap v6438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:34 smithi164 ceph-mon[36970]: pgmap v6438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:36 smithi163 ceph-mon[36873]: pgmap v6439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:37.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11066.370230 secs 2024-07-26T10:00:37.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11084.109962 secs 2024-07-26T10:00:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:36 smithi042 ceph-mon[32775]: pgmap v6439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11066.370230 secs 2024-07-26T10:00:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11084.109962 secs 2024-07-26T10:00:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:36 smithi164 ceph-mon[36970]: pgmap v6439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11066.370230 secs 2024-07-26T10:00:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11084.109962 secs 2024-07-26T10:00:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:38 smithi163 ceph-mon[36873]: pgmap v6440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:38 smithi042 ceph-mon[32775]: pgmap v6440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:38 smithi164 ceph-mon[36970]: pgmap v6440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:41.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:41 smithi163 ceph-mon[36873]: pgmap v6441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:41 smithi042 ceph-mon[32775]: pgmap v6441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:41 smithi164 ceph-mon[36970]: pgmap v6441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11071.370353 secs 2024-07-26T10:00:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11089.110085 secs 2024-07-26T10:00:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11071.370353 secs 2024-07-26T10:00:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11089.110085 secs 2024-07-26T10:00:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:00:42.482+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:00:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:00:42.482+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:00:42.484 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:00:42.482+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:00:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11071.370353 secs 2024-07-26T10:00:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11089.110085 secs 2024-07-26T10:00:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:43 smithi042 ceph-mon[32775]: pgmap v6442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:43.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:43 smithi164 ceph-mon[36970]: pgmap v6442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:43 smithi163 ceph-mon[36873]: pgmap v6442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:45.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:45 smithi042 ceph-mon[32775]: pgmap v6443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:45 smithi164 ceph-mon[36970]: pgmap v6443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:45 smithi163 ceph-mon[36873]: pgmap v6443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:47.311 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:47 smithi163 ceph-mon[36873]: pgmap v6444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:47.311 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11076.370456 secs 2024-07-26T10:00:47.311 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11094.110170 secs 2024-07-26T10:00:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:47 smithi042 ceph-mon[32775]: pgmap v6444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11076.370456 secs 2024-07-26T10:00:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11094.110170 secs 2024-07-26T10:00:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:47 smithi164 ceph-mon[36970]: pgmap v6444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11076.370456 secs 2024-07-26T10:00:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11094.110170 secs 2024-07-26T10:00:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:49 smithi042 ceph-mon[32775]: pgmap v6445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:49 smithi164 ceph-mon[36970]: pgmap v6445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:49.503 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:49 smithi163 ceph-mon[36873]: pgmap v6445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:51 smithi042 ceph-mon[32775]: pgmap v6446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:51 smithi164 ceph-mon[36970]: pgmap v6446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:51 smithi163 ceph-mon[36873]: pgmap v6446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11081.370579 secs 2024-07-26T10:00:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11099.110294 secs 2024-07-26T10:00:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11081.370579 secs 2024-07-26T10:00:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11099.110294 secs 2024-07-26T10:00:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11081.370579 secs 2024-07-26T10:00:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11099.110294 secs 2024-07-26T10:00:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:53 smithi042 ceph-mon[32775]: pgmap v6447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:53 smithi164 ceph-mon[36970]: pgmap v6447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:53 smithi163 ceph-mon[36873]: pgmap v6447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:55 smithi042 ceph-mon[32775]: pgmap v6448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:55 smithi164 ceph-mon[36970]: pgmap v6448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:55 smithi163 ceph-mon[36873]: pgmap v6448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:00:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:57 smithi042 ceph-mon[32775]: pgmap v6449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11086.370696 secs 2024-07-26T10:00:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11104.110428 secs 2024-07-26T10:00:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:57 smithi164 ceph-mon[36970]: pgmap v6449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11086.370696 secs 2024-07-26T10:00:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11104.110428 secs 2024-07-26T10:00:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:00:57.487+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:00:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:00:57.487+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:00:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:00:57.487+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:00:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:57 smithi163 ceph-mon[36873]: pgmap v6449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:00:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11086.370696 secs 2024-07-26T10:00:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11104.110428 secs 2024-07-26T10:00:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:00:58 smithi042 ceph-mon[32775]: pgmap v6450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:00:58 smithi164 ceph-mon[36970]: pgmap v6450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:00:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:00:58 smithi163 ceph-mon[36873]: pgmap v6450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:00 smithi163 ceph-mon[36873]: pgmap v6451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:00 smithi042 ceph-mon[32775]: pgmap v6451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:00 smithi164 ceph-mon[36970]: pgmap v6451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11091.370786 secs 2024-07-26T10:01:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11091.370786 secs 2024-07-26T10:01:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11091.370786 secs 2024-07-26T10:01:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11109.110526 secs 2024-07-26T10:01:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:02 smithi042 ceph-mon[32775]: pgmap v6452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11109.110526 secs 2024-07-26T10:01:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:02 smithi164 ceph-mon[36970]: pgmap v6452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11109.110526 secs 2024-07-26T10:01:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:02 smithi163 ceph-mon[36873]: pgmap v6452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:04 smithi042 ceph-mon[32775]: pgmap v6453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:01:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:04 smithi164 ceph-mon[36970]: pgmap v6453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:01:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:04 smithi163 ceph-mon[36873]: pgmap v6453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:01:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:06 smithi042 ceph-mon[32775]: pgmap v6454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11096.370914 secs 2024-07-26T10:01:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:06 smithi164 ceph-mon[36970]: pgmap v6454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11096.370914 secs 2024-07-26T10:01:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:06 smithi163 ceph-mon[36873]: pgmap v6454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11096.370914 secs 2024-07-26T10:01:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11114.110671 secs 2024-07-26T10:01:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11114.110671 secs 2024-07-26T10:01:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11114.110671 secs 2024-07-26T10:01:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:08 smithi042 ceph-mon[32775]: pgmap v6455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:08.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:08 smithi164 ceph-mon[36970]: pgmap v6455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:08 smithi163 ceph-mon[36873]: pgmap v6455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:10 smithi042 ceph-mon[32775]: pgmap v6456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:10 smithi164 ceph-mon[36970]: pgmap v6456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:10 smithi163 ceph-mon[36873]: pgmap v6456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11101.371054 secs 2024-07-26T10:01:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11101.371054 secs 2024-07-26T10:01:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11101.371054 secs 2024-07-26T10:01:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:01:12.494+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:01:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:01:12.494+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:01:12.497 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:01:12.494+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:01:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11119.110794 secs 2024-07-26T10:01:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:12 smithi042 ceph-mon[32775]: pgmap v6457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:12.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11119.110794 secs 2024-07-26T10:01:12.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:12 smithi164 ceph-mon[36970]: pgmap v6457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11119.110794 secs 2024-07-26T10:01:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:12 smithi163 ceph-mon[36873]: pgmap v6457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:14 smithi042 ceph-mon[32775]: pgmap v6458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:01:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:14 smithi164 ceph-mon[36970]: pgmap v6458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:01:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:14 smithi163 ceph-mon[36873]: pgmap v6458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:01:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:16 smithi042 ceph-mon[32775]: pgmap v6459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11106.371202 secs 2024-07-26T10:01:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:16 smithi164 ceph-mon[36970]: pgmap v6459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11106.371202 secs 2024-07-26T10:01:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:16 smithi163 ceph-mon[36873]: pgmap v6459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11106.371202 secs 2024-07-26T10:01:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11124.110914 secs 2024-07-26T10:01:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11124.110914 secs 2024-07-26T10:01:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11124.110914 secs 2024-07-26T10:01:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:18 smithi042 ceph-mon[32775]: pgmap v6460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:18 smithi164 ceph-mon[36970]: pgmap v6460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:18 smithi163 ceph-mon[36873]: pgmap v6460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:01:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:01:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:01:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:20 smithi042 ceph-mon[32775]: pgmap v6461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:01:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:01:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:20 smithi042 ceph-mon[32775]: pgmap v6462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:01:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:20 smithi042 ceph-mon[32775]: pgmap v6463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:01:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:01:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:20 smithi164 ceph-mon[36970]: pgmap v6461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:01:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:01:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:20 smithi164 ceph-mon[36970]: pgmap v6462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:01:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:20 smithi164 ceph-mon[36970]: pgmap v6463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:01:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:01:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:20 smithi163 ceph-mon[36873]: pgmap v6461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:01:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:01:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:20 smithi163 ceph-mon[36873]: pgmap v6462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:01:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:20 smithi163 ceph-mon[36873]: pgmap v6463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:01:21.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:01:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11111.371369 secs 2024-07-26T10:01:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11111.371369 secs 2024-07-26T10:01:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11111.371369 secs 2024-07-26T10:01:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11129.111011 secs 2024-07-26T10:01:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:22 smithi042 ceph-mon[32775]: pgmap v6464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:01:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11129.111011 secs 2024-07-26T10:01:22.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:22 smithi164 ceph-mon[36970]: pgmap v6464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:01:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11129.111011 secs 2024-07-26T10:01:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:22 smithi163 ceph-mon[36873]: pgmap v6464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:01:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:24 smithi042 ceph-mon[32775]: pgmap v6465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:01:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:24 smithi164 ceph-mon[36970]: pgmap v6465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:01:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:24 smithi163 ceph-mon[36873]: pgmap v6465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:01:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:26 smithi042 ceph-mon[32775]: pgmap v6466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:01:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:26 smithi164 ceph-mon[36970]: pgmap v6466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:01:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:26 smithi163 ceph-mon[36873]: pgmap v6466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:01:27.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11116.371495 secs 2024-07-26T10:01:27.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11134.111079 secs 2024-07-26T10:01:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11116.371495 secs 2024-07-26T10:01:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11134.111079 secs 2024-07-26T10:01:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:01:27.499+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:01:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:01:27.499+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:01:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:01:27.499+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:01:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11116.371495 secs 2024-07-26T10:01:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11134.111079 secs 2024-07-26T10:01:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:28 smithi042 ceph-mon[32775]: pgmap v6467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:01:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:28 smithi164 ceph-mon[36970]: pgmap v6467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:01:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:28 smithi163 ceph-mon[36873]: pgmap v6467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:01:30.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:30 smithi042 ceph-mon[32775]: pgmap v6468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:01:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:30 smithi164 ceph-mon[36970]: pgmap v6468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:01:30.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:30 smithi163 ceph-mon[36873]: pgmap v6468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:01:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11121.371621 secs 2024-07-26T10:01:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11121.371621 secs 2024-07-26T10:01:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11121.371621 secs 2024-07-26T10:01:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11139.111206 secs 2024-07-26T10:01:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:32 smithi164 ceph-mon[36970]: pgmap v6469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11139.111206 secs 2024-07-26T10:01:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:32 smithi163 ceph-mon[36873]: pgmap v6469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11139.111206 secs 2024-07-26T10:01:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:32 smithi042 ceph-mon[32775]: pgmap v6469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:34 smithi042 ceph-mon[32775]: pgmap v6470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:01:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:34 smithi164 ceph-mon[36970]: pgmap v6470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:01:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:34 smithi163 ceph-mon[36873]: pgmap v6470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:01:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:36 smithi042 ceph-mon[32775]: pgmap v6471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:01:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:36 smithi164 ceph-mon[36970]: pgmap v6471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:01:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:36 smithi163 ceph-mon[36873]: pgmap v6471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:01:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11126.371748 secs 2024-07-26T10:01:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11144.111310 secs 2024-07-26T10:01:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11126.371748 secs 2024-07-26T10:01:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11144.111310 secs 2024-07-26T10:01:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11126.371748 secs 2024-07-26T10:01:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11144.111310 secs 2024-07-26T10:01:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:38 smithi042 ceph-mon[32775]: pgmap v6472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:01:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:38 smithi164 ceph-mon[36970]: pgmap v6472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:01:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:38 smithi163 ceph-mon[36873]: pgmap v6472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:01:40.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:40 smithi042 ceph-mon[32775]: pgmap v6473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:01:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:40 smithi164 ceph-mon[36970]: pgmap v6473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:01:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:40 smithi163 ceph-mon[36873]: pgmap v6473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:01:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11131.371864 secs 2024-07-26T10:01:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11131.371864 secs 2024-07-26T10:01:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11131.371864 secs 2024-07-26T10:01:42.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:01:42.510+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:01:42.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:01:42.510+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:01:42.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:01:42.510+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:01:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11149.111376 secs 2024-07-26T10:01:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:42 smithi164 ceph-mon[36970]: pgmap v6474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:01:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11149.111376 secs 2024-07-26T10:01:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:42 smithi163 ceph-mon[36873]: pgmap v6474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:01:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11149.111376 secs 2024-07-26T10:01:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:42 smithi042 ceph-mon[32775]: pgmap v6474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:01:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:44 smithi042 ceph-mon[32775]: pgmap v6475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:01:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:44 smithi164 ceph-mon[36970]: pgmap v6475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:01:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:44 smithi163 ceph-mon[36873]: pgmap v6475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:01:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:46 smithi042 ceph-mon[32775]: pgmap v6476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:01:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:46 smithi164 ceph-mon[36970]: pgmap v6476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:01:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:46 smithi163 ceph-mon[36873]: pgmap v6476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:01:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11136.372006 secs 2024-07-26T10:01:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11154.111453 secs 2024-07-26T10:01:47.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11136.372006 secs 2024-07-26T10:01:47.410 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11154.111453 secs 2024-07-26T10:01:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11136.372006 secs 2024-07-26T10:01:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11154.111453 secs 2024-07-26T10:01:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:48 smithi042 ceph-mon[32775]: pgmap v6477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:48 smithi164 ceph-mon[36970]: pgmap v6477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:48 smithi163 ceph-mon[36873]: pgmap v6477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:50 smithi042 ceph-mon[32775]: pgmap v6478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:50 smithi164 ceph-mon[36970]: pgmap v6478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:50 smithi163 ceph-mon[36873]: pgmap v6478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11141.372151 secs 2024-07-26T10:01:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11141.372151 secs 2024-07-26T10:01:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11141.372151 secs 2024-07-26T10:01:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11159.111555 secs 2024-07-26T10:01:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:52 smithi164 ceph-mon[36970]: pgmap v6479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11159.111555 secs 2024-07-26T10:01:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:52 smithi163 ceph-mon[36873]: pgmap v6479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11159.111555 secs 2024-07-26T10:01:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:52 smithi042 ceph-mon[32775]: pgmap v6479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:54 smithi042 ceph-mon[32775]: pgmap v6480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:01:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:54 smithi164 ceph-mon[36970]: pgmap v6480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:01:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:54 smithi163 ceph-mon[36873]: pgmap v6480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:01:56.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:56 smithi042 ceph-mon[32775]: pgmap v6481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:56.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:56 smithi164 ceph-mon[36970]: pgmap v6481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:56 smithi163 ceph-mon[36873]: pgmap v6481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:01:57.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11146.372246 secs 2024-07-26T10:01:57.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11164.111678 secs 2024-07-26T10:01:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11146.372246 secs 2024-07-26T10:01:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11164.111678 secs 2024-07-26T10:01:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11146.372246 secs 2024-07-26T10:01:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11164.111678 secs 2024-07-26T10:01:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:01:57.515+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:01:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:01:57.515+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:01:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:01:57.515+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:01:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:01:58 smithi042 ceph-mon[32775]: pgmap v6482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:01:58 smithi164 ceph-mon[36970]: pgmap v6482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:01:58 smithi163 ceph-mon[36873]: pgmap v6482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:01:59.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:01:59.601+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:01:59.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:01:59.601+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:01:59.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:01:59.602+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:01:59.604 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:01:59.602+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:02:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:00 smithi042 ceph-mon[32775]: pgmap v6483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:00 smithi164 ceph-mon[36970]: pgmap v6483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:00 smithi163 ceph-mon[36873]: pgmap v6483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:01.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11151.372374 secs 2024-07-26T10:02:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11151.372374 secs 2024-07-26T10:02:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11151.372374 secs 2024-07-26T10:02:02.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11169.111818 secs 2024-07-26T10:02:02.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:02 smithi164 ceph-mon[36970]: pgmap v6484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11169.111818 secs 2024-07-26T10:02:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:02 smithi163 ceph-mon[36873]: pgmap v6484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11169.111818 secs 2024-07-26T10:02:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:02 smithi042 ceph-mon[32775]: pgmap v6484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:04 smithi042 ceph-mon[32775]: pgmap v6485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:02:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:04 smithi164 ceph-mon[36970]: pgmap v6485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:02:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:04 smithi163 ceph-mon[36873]: pgmap v6485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:02:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:06 smithi042 ceph-mon[32775]: pgmap v6486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:06 smithi164 ceph-mon[36970]: pgmap v6486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:06 smithi163 ceph-mon[36873]: pgmap v6486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11156.372495 secs 2024-07-26T10:02:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11174.111965 secs 2024-07-26T10:02:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11156.372495 secs 2024-07-26T10:02:07.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11174.111965 secs 2024-07-26T10:02:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11156.372495 secs 2024-07-26T10:02:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11174.111965 secs 2024-07-26T10:02:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:08 smithi042 ceph-mon[32775]: pgmap v6487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:08 smithi164 ceph-mon[36970]: pgmap v6487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:08 smithi163 ceph-mon[36873]: pgmap v6487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:10.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:10 smithi042 ceph-mon[32775]: pgmap v6488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:10 smithi164 ceph-mon[36970]: pgmap v6488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:10 smithi163 ceph-mon[36873]: pgmap v6488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11161.372620 secs 2024-07-26T10:02:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11161.372620 secs 2024-07-26T10:02:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11161.372620 secs 2024-07-26T10:02:12.526 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:02:12.525+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:02:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:02:12.525+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:02:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:02:12.525+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:02:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11179.112045 secs 2024-07-26T10:02:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:12 smithi164 ceph-mon[36970]: pgmap v6489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11179.112045 secs 2024-07-26T10:02:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:12 smithi163 ceph-mon[36873]: pgmap v6489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11179.112045 secs 2024-07-26T10:02:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:12 smithi042 ceph-mon[32775]: pgmap v6489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:14 smithi042 ceph-mon[32775]: pgmap v6490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:02:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:14 smithi164 ceph-mon[36970]: pgmap v6490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:02:14.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:14 smithi163 ceph-mon[36873]: pgmap v6490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:02:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:16 smithi042 ceph-mon[32775]: pgmap v6491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:16 smithi164 ceph-mon[36970]: pgmap v6491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:16 smithi163 ceph-mon[36873]: pgmap v6491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11166.372719 secs 2024-07-26T10:02:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11184.112162 secs 2024-07-26T10:02:17.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11166.372719 secs 2024-07-26T10:02:17.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11184.112162 secs 2024-07-26T10:02:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11166.372719 secs 2024-07-26T10:02:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11184.112162 secs 2024-07-26T10:02:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:18 smithi042 ceph-mon[32775]: pgmap v6492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:18 smithi164 ceph-mon[36970]: pgmap v6492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:18 smithi163 ceph-mon[36873]: pgmap v6492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:20 smithi042 ceph-mon[32775]: pgmap v6493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:20 smithi164 ceph-mon[36970]: pgmap v6493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:20 smithi163 ceph-mon[36873]: pgmap v6493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:02:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:02:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:02:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11171.372874 secs 2024-07-26T10:02:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11189.112234 secs 2024-07-26T10:02:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:22 smithi042 ceph-mon[32775]: pgmap v6494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11171.372874 secs 2024-07-26T10:02:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11189.112234 secs 2024-07-26T10:02:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:22 smithi164 ceph-mon[36970]: pgmap v6494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11171.372874 secs 2024-07-26T10:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11189.112234 secs 2024-07-26T10:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:22 smithi163 ceph-mon[36873]: pgmap v6494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:23.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:23.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:23.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:23.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:23.386 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:02:23.387 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:02:23.387 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:23 smithi042 ceph-mon[32775]: pgmap v6495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:02:23.387 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:23 smithi042 ceph-mon[32775]: pgmap v6496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:02:23.387 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:23.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:23.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:23.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:23.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:02:23.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:02:23.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:23 smithi164 ceph-mon[36970]: pgmap v6495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:02:23.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:23 smithi164 ceph-mon[36970]: pgmap v6496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:02:23.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:23.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:23.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:23.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:02:23.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:02:23.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:23 smithi163 ceph-mon[36873]: pgmap v6495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:02:23.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:23 smithi163 ceph-mon[36873]: pgmap v6496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:02:23.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:02:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:25 smithi042 ceph-mon[32775]: pgmap v6497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:02:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:25 smithi164 ceph-mon[36970]: pgmap v6497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:02:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:25 smithi163 ceph-mon[36873]: pgmap v6497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:02:26.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11176.373003 secs 2024-07-26T10:02:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11176.373003 secs 2024-07-26T10:02:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11176.373003 secs 2024-07-26T10:02:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:02:27.530+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:02:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:02:27.530+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:02:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:02:27.530+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:02:27.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11194.112332 secs 2024-07-26T10:02:27.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:27 smithi164 ceph-mon[36970]: pgmap v6498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:02:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11194.112332 secs 2024-07-26T10:02:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:27 smithi042 ceph-mon[32775]: pgmap v6498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:02:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11194.112332 secs 2024-07-26T10:02:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:27 smithi163 ceph-mon[36873]: pgmap v6498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:02:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:29 smithi042 ceph-mon[32775]: pgmap v6499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:02:29.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:29 smithi164 ceph-mon[36970]: pgmap v6499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:02:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:29 smithi163 ceph-mon[36873]: pgmap v6499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:02:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:31 smithi042 ceph-mon[32775]: pgmap v6500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:02:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11181.373121 secs 2024-07-26T10:02:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:31 smithi164 ceph-mon[36970]: pgmap v6500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:02:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11181.373121 secs 2024-07-26T10:02:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:31 smithi163 ceph-mon[36873]: pgmap v6500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:02:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11181.373121 secs 2024-07-26T10:02:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11199.112395 secs 2024-07-26T10:02:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11199.112395 secs 2024-07-26T10:02:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11199.112395 secs 2024-07-26T10:02:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:33 smithi042 ceph-mon[32775]: pgmap v6501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:02:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:33 smithi164 ceph-mon[36970]: pgmap v6501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:02:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:33 smithi163 ceph-mon[36873]: pgmap v6501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:02:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:35 smithi042 ceph-mon[32775]: pgmap v6502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:35 smithi164 ceph-mon[36970]: pgmap v6502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:35 smithi163 ceph-mon[36873]: pgmap v6502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11186.373263 secs 2024-07-26T10:02:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11186.373263 secs 2024-07-26T10:02:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11186.373263 secs 2024-07-26T10:02:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11204.112526 secs 2024-07-26T10:02:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:37 smithi042 ceph-mon[32775]: pgmap v6503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11204.112526 secs 2024-07-26T10:02:37.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:37 smithi164 ceph-mon[36970]: pgmap v6503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11204.112526 secs 2024-07-26T10:02:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:37 smithi163 ceph-mon[36873]: pgmap v6503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:39 smithi042 ceph-mon[32775]: pgmap v6504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:02:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:39 smithi164 ceph-mon[36970]: pgmap v6504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:02:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:39 smithi163 ceph-mon[36873]: pgmap v6504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:02:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:41 smithi164 ceph-mon[36970]: pgmap v6505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11191.373390 secs 2024-07-26T10:02:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:41 smithi163 ceph-mon[36873]: pgmap v6505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11191.373390 secs 2024-07-26T10:02:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:41 smithi042 ceph-mon[32775]: pgmap v6505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11191.373390 secs 2024-07-26T10:02:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:02:42.532+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:02:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:02:42.532+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:02:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:02:42.532+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:02:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11209.112663 secs 2024-07-26T10:02:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11209.112663 secs 2024-07-26T10:02:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11209.112663 secs 2024-07-26T10:02:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:43 smithi164 ceph-mon[36970]: pgmap v6506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:43 smithi163 ceph-mon[36873]: pgmap v6506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:43 smithi042 ceph-mon[32775]: pgmap v6506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:45.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:45 smithi164 ceph-mon[36970]: pgmap v6507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:45 smithi163 ceph-mon[36873]: pgmap v6507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:45 smithi042 ceph-mon[32775]: pgmap v6507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11196.373474 secs 2024-07-26T10:02:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11196.373474 secs 2024-07-26T10:02:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11196.373474 secs 2024-07-26T10:02:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11214.112823 secs 2024-07-26T10:02:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:47 smithi164 ceph-mon[36970]: pgmap v6508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11214.112823 secs 2024-07-26T10:02:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:47 smithi163 ceph-mon[36873]: pgmap v6508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11214.112823 secs 2024-07-26T10:02:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:47 smithi042 ceph-mon[32775]: pgmap v6508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:49 smithi164 ceph-mon[36970]: pgmap v6509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:02:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:49 smithi163 ceph-mon[36873]: pgmap v6509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:02:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:49 smithi042 ceph-mon[32775]: pgmap v6509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:02:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:51 smithi164 ceph-mon[36970]: pgmap v6510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11201.373605 secs 2024-07-26T10:02:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:51 smithi163 ceph-mon[36873]: pgmap v6510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11201.373605 secs 2024-07-26T10:02:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:51 smithi042 ceph-mon[32775]: pgmap v6510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11201.373605 secs 2024-07-26T10:02:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11219.112966 secs 2024-07-26T10:02:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11219.112966 secs 2024-07-26T10:02:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11219.112966 secs 2024-07-26T10:02:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:53 smithi164 ceph-mon[36970]: pgmap v6511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:53 smithi163 ceph-mon[36873]: pgmap v6511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:53 smithi042 ceph-mon[32775]: pgmap v6511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:55 smithi164 ceph-mon[36970]: pgmap v6512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:55 smithi163 ceph-mon[36873]: pgmap v6512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:55 smithi042 ceph-mon[32775]: pgmap v6512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:02:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11206.373737 secs 2024-07-26T10:02:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11206.373737 secs 2024-07-26T10:02:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11206.373737 secs 2024-07-26T10:02:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:02:57.534+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:02:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:02:57.534+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:02:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:02:57.534+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:02:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11224.113052 secs 2024-07-26T10:02:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:57 smithi164 ceph-mon[36970]: pgmap v6513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11224.113052 secs 2024-07-26T10:02:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:57 smithi163 ceph-mon[36873]: pgmap v6513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11224.113052 secs 2024-07-26T10:02:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:57 smithi042 ceph-mon[32775]: pgmap v6513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:02:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:02:59 smithi164 ceph-mon[36970]: pgmap v6514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:03:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:02:59 smithi163 ceph-mon[36873]: pgmap v6514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:03:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:02:59 smithi042 ceph-mon[32775]: pgmap v6514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:03:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:01 smithi164 ceph-mon[36970]: pgmap v6515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11211.373869 secs 2024-07-26T10:03:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:01 smithi163 ceph-mon[36873]: pgmap v6515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11211.373869 secs 2024-07-26T10:03:02.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:01 smithi042 ceph-mon[32775]: pgmap v6515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:02.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11211.373869 secs 2024-07-26T10:03:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11229.113115 secs 2024-07-26T10:03:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11229.113115 secs 2024-07-26T10:03:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11229.113115 secs 2024-07-26T10:03:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:03 smithi164 ceph-mon[36970]: pgmap v6516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:03 smithi163 ceph-mon[36873]: pgmap v6516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:03 smithi042 ceph-mon[32775]: pgmap v6516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:05 smithi164 ceph-mon[36970]: pgmap v6517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:05 smithi163 ceph-mon[36873]: pgmap v6517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:06.097 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:05 smithi042 ceph-mon[32775]: pgmap v6517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11216.373968 secs 2024-07-26T10:03:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11216.373968 secs 2024-07-26T10:03:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11216.373968 secs 2024-07-26T10:03:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11234.113236 secs 2024-07-26T10:03:07.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:07 smithi164 ceph-mon[36970]: pgmap v6518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11234.113236 secs 2024-07-26T10:03:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:07 smithi163 ceph-mon[36873]: pgmap v6518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11234.113236 secs 2024-07-26T10:03:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:07 smithi042 ceph-mon[32775]: pgmap v6518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:09 smithi164 ceph-mon[36970]: pgmap v6519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:03:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:09 smithi163 ceph-mon[36873]: pgmap v6519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:03:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:09 smithi042 ceph-mon[32775]: pgmap v6519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:03:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:11 smithi164 ceph-mon[36970]: pgmap v6520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11221.374091 secs 2024-07-26T10:03:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:11 smithi163 ceph-mon[36873]: pgmap v6520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11221.374091 secs 2024-07-26T10:03:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:11 smithi042 ceph-mon[32775]: pgmap v6520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11221.374091 secs 2024-07-26T10:03:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:03:12.545+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:03:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:03:12.545+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:03:12.547 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:03:12.545+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:03:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11239.113359 secs 2024-07-26T10:03:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11239.113359 secs 2024-07-26T10:03:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11239.113359 secs 2024-07-26T10:03:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:13 smithi163 ceph-mon[36873]: pgmap v6521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:13 smithi042 ceph-mon[32775]: pgmap v6521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:13 smithi164 ceph-mon[36970]: pgmap v6521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:15 smithi163 ceph-mon[36873]: pgmap v6522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:15 smithi042 ceph-mon[32775]: pgmap v6522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:15 smithi164 ceph-mon[36970]: pgmap v6522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11226.374238 secs 2024-07-26T10:03:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11226.374238 secs 2024-07-26T10:03:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11226.374238 secs 2024-07-26T10:03:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11244.113478 secs 2024-07-26T10:03:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:17 smithi163 ceph-mon[36873]: pgmap v6523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11244.113478 secs 2024-07-26T10:03:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:17 smithi042 ceph-mon[32775]: pgmap v6523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11244.113478 secs 2024-07-26T10:03:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:17 smithi164 ceph-mon[36970]: pgmap v6523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:19 smithi163 ceph-mon[36873]: pgmap v6524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:03:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:19 smithi042 ceph-mon[32775]: pgmap v6524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:03:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:19 smithi164 ceph-mon[36970]: pgmap v6524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:03:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:21 smithi163 ceph-mon[36873]: pgmap v6525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11231.374359 secs 2024-07-26T10:03:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:21 smithi042 ceph-mon[32775]: pgmap v6525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11231.374359 secs 2024-07-26T10:03:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:21 smithi164 ceph-mon[36970]: pgmap v6525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11231.374359 secs 2024-07-26T10:03:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11249.113605 secs 2024-07-26T10:03:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:03:23.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11249.113605 secs 2024-07-26T10:03:23.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:03:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11249.113605 secs 2024-07-26T10:03:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:03:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:23 smithi163 ceph-mon[36873]: pgmap v6526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:03:24.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:03:24.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:23 smithi042 ceph-mon[32775]: pgmap v6526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:03:24.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:03:24.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:23 smithi164 ceph-mon[36970]: pgmap v6526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:24.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:03:24.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:03:24.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:03:25.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:25 smithi163 ceph-mon[36873]: pgmap v6527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T10:03:25.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:25 smithi163 ceph-mon[36873]: pgmap v6528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T10:03:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:25 smithi042 ceph-mon[32775]: pgmap v6527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T10:03:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:25 smithi042 ceph-mon[32775]: pgmap v6528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T10:03:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:25 smithi164 ceph-mon[36970]: pgmap v6527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T10:03:25.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:25 smithi164 ceph-mon[36970]: pgmap v6528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T10:03:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:27 smithi163 ceph-mon[36873]: pgmap v6529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:03:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11236.374473 secs 2024-07-26T10:03:27.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11254.113729 secs 2024-07-26T10:03:27.276 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:27 smithi164 ceph-mon[36970]: pgmap v6529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:03:27.276 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11236.374473 secs 2024-07-26T10:03:27.276 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11254.113729 secs 2024-07-26T10:03:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:27 smithi042 ceph-mon[32775]: pgmap v6529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:03:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11236.374473 secs 2024-07-26T10:03:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11254.113729 secs 2024-07-26T10:03:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:03:27.550+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:03:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:03:27.550+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:03:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:03:27.550+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:03:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:29 smithi042 ceph-mon[32775]: pgmap v6530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:03:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:29 smithi164 ceph-mon[36970]: pgmap v6530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:03:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:29 smithi163 ceph-mon[36873]: pgmap v6530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:03:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:31 smithi042 ceph-mon[32775]: pgmap v6531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:03:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:31 smithi164 ceph-mon[36970]: pgmap v6531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:03:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:31 smithi163 ceph-mon[36873]: pgmap v6531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:03:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11241.374575 secs 2024-07-26T10:03:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11259.113882 secs 2024-07-26T10:03:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11241.374575 secs 2024-07-26T10:03:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11259.113882 secs 2024-07-26T10:03:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11241.374575 secs 2024-07-26T10:03:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11259.113882 secs 2024-07-26T10:03:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:33 smithi042 ceph-mon[32775]: pgmap v6532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:03:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:33 smithi164 ceph-mon[36970]: pgmap v6532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:03:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:33 smithi163 ceph-mon[36873]: pgmap v6532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:03:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:35 smithi042 ceph-mon[32775]: pgmap v6533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:03:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:35 smithi164 ceph-mon[36970]: pgmap v6533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:03:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:35 smithi163 ceph-mon[36873]: pgmap v6533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:03:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:37 smithi042 ceph-mon[32775]: pgmap v6534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11246.374641 secs 2024-07-26T10:03:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11264.114028 secs 2024-07-26T10:03:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:37 smithi164 ceph-mon[36970]: pgmap v6534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11246.374641 secs 2024-07-26T10:03:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11264.114028 secs 2024-07-26T10:03:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:37 smithi163 ceph-mon[36873]: pgmap v6534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11246.374641 secs 2024-07-26T10:03:37.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11264.114028 secs 2024-07-26T10:03:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:39 smithi042 ceph-mon[32775]: pgmap v6535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:39 smithi164 ceph-mon[36970]: pgmap v6535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:39 smithi163 ceph-mon[36873]: pgmap v6535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:41 smithi042 ceph-mon[32775]: pgmap v6536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:41 smithi164 ceph-mon[36970]: pgmap v6536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:41 smithi163 ceph-mon[36873]: pgmap v6536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11251.374769 secs 2024-07-26T10:03:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11269.114188 secs 2024-07-26T10:03:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11251.374769 secs 2024-07-26T10:03:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11269.114188 secs 2024-07-26T10:03:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11251.374769 secs 2024-07-26T10:03:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11269.114188 secs 2024-07-26T10:03:42.562 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:03:42.560+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:03:42.562 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:03:42.560+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:03:42.562 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:03:42.560+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:03:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:43 smithi042 ceph-mon[32775]: pgmap v6537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:43 smithi164 ceph-mon[36970]: pgmap v6537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:43 smithi163 ceph-mon[36873]: pgmap v6537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:45.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:45 smithi042 ceph-mon[32775]: pgmap v6538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:03:45.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:45 smithi164 ceph-mon[36970]: pgmap v6538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:03:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:45 smithi163 ceph-mon[36873]: pgmap v6538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:03:47.311 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:47 smithi163 ceph-mon[36873]: pgmap v6539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:47.311 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11256.374912 secs 2024-07-26T10:03:47.311 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11274.114310 secs 2024-07-26T10:03:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:47 smithi042 ceph-mon[32775]: pgmap v6539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11256.374912 secs 2024-07-26T10:03:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11274.114310 secs 2024-07-26T10:03:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:47 smithi164 ceph-mon[36970]: pgmap v6539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11256.374912 secs 2024-07-26T10:03:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11274.114310 secs 2024-07-26T10:03:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:49 smithi042 ceph-mon[32775]: pgmap v6540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:49 smithi164 ceph-mon[36970]: pgmap v6540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:49 smithi163 ceph-mon[36873]: pgmap v6540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:51 smithi042 ceph-mon[32775]: pgmap v6541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:51 smithi164 ceph-mon[36970]: pgmap v6541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:51 smithi163 ceph-mon[36873]: pgmap v6541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11261.375017 secs 2024-07-26T10:03:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11279.114437 secs 2024-07-26T10:03:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11261.375017 secs 2024-07-26T10:03:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11279.114437 secs 2024-07-26T10:03:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11261.375017 secs 2024-07-26T10:03:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11279.114437 secs 2024-07-26T10:03:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:53 smithi042 ceph-mon[32775]: pgmap v6542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:53 smithi164 ceph-mon[36970]: pgmap v6542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:53 smithi163 ceph-mon[36873]: pgmap v6542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:55 smithi042 ceph-mon[32775]: pgmap v6543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:03:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:55 smithi164 ceph-mon[36970]: pgmap v6543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:03:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:55 smithi163 ceph-mon[36873]: pgmap v6543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:03:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:56 smithi042 ceph-mon[32775]: pgmap v6544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:56 smithi164 ceph-mon[36970]: pgmap v6544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:56 smithi163 ceph-mon[36873]: pgmap v6544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:03:57.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11266.375138 secs 2024-07-26T10:03:57.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11284.114570 secs 2024-07-26T10:03:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11266.375138 secs 2024-07-26T10:03:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11284.114570 secs 2024-07-26T10:03:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11266.375138 secs 2024-07-26T10:03:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11284.114570 secs 2024-07-26T10:03:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:03:57.565+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:03:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:03:57.565+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:03:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:03:57.565+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:03:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:03:58 smithi042 ceph-mon[32775]: pgmap v6545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:58.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:03:58 smithi164 ceph-mon[36970]: pgmap v6545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:03:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:03:58 smithi163 ceph-mon[36873]: pgmap v6545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:00 smithi164 ceph-mon[36970]: pgmap v6546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:00 smithi163 ceph-mon[36873]: pgmap v6546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:00 smithi042 ceph-mon[32775]: pgmap v6546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11271.375265 secs 2024-07-26T10:04:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11271.375265 secs 2024-07-26T10:04:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11271.375265 secs 2024-07-26T10:04:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11289.114690 secs 2024-07-26T10:04:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:02 smithi163 ceph-mon[36873]: pgmap v6547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11289.114690 secs 2024-07-26T10:04:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:02 smithi042 ceph-mon[32775]: pgmap v6547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11289.114690 secs 2024-07-26T10:04:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:02 smithi164 ceph-mon[36970]: pgmap v6547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:04 smithi163 ceph-mon[36873]: pgmap v6548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:04:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:04 smithi042 ceph-mon[32775]: pgmap v6548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:04:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:04 smithi164 ceph-mon[36970]: pgmap v6548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:04:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:06 smithi163 ceph-mon[36873]: pgmap v6549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11276.375323 secs 2024-07-26T10:04:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:06 smithi042 ceph-mon[32775]: pgmap v6549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11276.375323 secs 2024-07-26T10:04:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:06 smithi164 ceph-mon[36970]: pgmap v6549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11276.375323 secs 2024-07-26T10:04:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11294.114812 secs 2024-07-26T10:04:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11294.114812 secs 2024-07-26T10:04:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11294.114812 secs 2024-07-26T10:04:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:08 smithi163 ceph-mon[36873]: pgmap v6550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:08 smithi042 ceph-mon[32775]: pgmap v6550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:08 smithi164 ceph-mon[36970]: pgmap v6550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:10 smithi163 ceph-mon[36873]: pgmap v6551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:10 smithi042 ceph-mon[32775]: pgmap v6551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:10 smithi164 ceph-mon[36970]: pgmap v6551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11281.375424 secs 2024-07-26T10:04:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11281.375424 secs 2024-07-26T10:04:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11281.375424 secs 2024-07-26T10:04:12.578 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:04:12.573+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:04:12.578 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:04:12.573+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:04:12.578 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:04:12.573+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:04:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11299.114968 secs 2024-07-26T10:04:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:12 smithi163 ceph-mon[36873]: pgmap v6552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11299.114968 secs 2024-07-26T10:04:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:12 smithi042 ceph-mon[32775]: pgmap v6552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11299.114968 secs 2024-07-26T10:04:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:12 smithi164 ceph-mon[36970]: pgmap v6552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:14 smithi163 ceph-mon[36873]: pgmap v6553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:04:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:14 smithi042 ceph-mon[32775]: pgmap v6553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:04:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:14 smithi164 ceph-mon[36970]: pgmap v6553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:04:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:16 smithi042 ceph-mon[32775]: pgmap v6554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11286.375551 secs 2024-07-26T10:04:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:16 smithi164 ceph-mon[36970]: pgmap v6554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11286.375551 secs 2024-07-26T10:04:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:16 smithi163 ceph-mon[36873]: pgmap v6554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11286.375551 secs 2024-07-26T10:04:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11304.115089 secs 2024-07-26T10:04:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11304.115089 secs 2024-07-26T10:04:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11304.115089 secs 2024-07-26T10:04:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:18 smithi042 ceph-mon[32775]: pgmap v6555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:18 smithi164 ceph-mon[36970]: pgmap v6555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:18 smithi163 ceph-mon[36873]: pgmap v6555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:20 smithi042 ceph-mon[32775]: pgmap v6556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:20 smithi164 ceph-mon[36970]: pgmap v6556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:20 smithi163 ceph-mon[36873]: pgmap v6556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11291.375676 secs 2024-07-26T10:04:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11291.375676 secs 2024-07-26T10:04:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11291.375676 secs 2024-07-26T10:04:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11309.115172 secs 2024-07-26T10:04:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:22 smithi042 ceph-mon[32775]: pgmap v6557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11309.115172 secs 2024-07-26T10:04:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:22 smithi164 ceph-mon[36970]: pgmap v6557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11309.115172 secs 2024-07-26T10:04:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:22 smithi163 ceph-mon[36873]: pgmap v6557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:23.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:04:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:04:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:04:24.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:24 smithi042 ceph-mon[32775]: pgmap v6558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:04:24.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:04:24.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:04:24.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:24 smithi042 ceph-mon[32775]: pgmap v6559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:04:24.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:24 smithi042 ceph-mon[32775]: pgmap v6560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:04:24.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:04:24.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:24 smithi164 ceph-mon[36970]: pgmap v6558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:04:24.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:04:24.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:04:24.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:24 smithi164 ceph-mon[36970]: pgmap v6559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:04:24.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:24 smithi164 ceph-mon[36970]: pgmap v6560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:04:24.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:04:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:24 smithi163 ceph-mon[36873]: pgmap v6558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:04:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:04:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:04:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:24 smithi163 ceph-mon[36873]: pgmap v6559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:04:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:24 smithi163 ceph-mon[36873]: pgmap v6560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:04:25.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:04:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11296.375764 secs 2024-07-26T10:04:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:27 smithi042 ceph-mon[32775]: pgmap v6561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:04:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11314.115291 secs 2024-07-26T10:04:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11296.375764 secs 2024-07-26T10:04:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:27 smithi164 ceph-mon[36970]: pgmap v6561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:04:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11314.115291 secs 2024-07-26T10:04:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11296.375764 secs 2024-07-26T10:04:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:27 smithi163 ceph-mon[36873]: pgmap v6561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:04:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11314.115291 secs 2024-07-26T10:04:27.581 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:04:27.579+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:04:27.581 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:04:27.579+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:04:27.581 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:04:27.579+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:04:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:29 smithi042 ceph-mon[32775]: pgmap v6562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:04:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:29 smithi164 ceph-mon[36970]: pgmap v6562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:04:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:29 smithi163 ceph-mon[36873]: pgmap v6562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:04:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:31 smithi042 ceph-mon[32775]: pgmap v6563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:04:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:31 smithi164 ceph-mon[36970]: pgmap v6563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:04:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:31 smithi163 ceph-mon[36873]: pgmap v6563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:04:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11301.375905 secs 2024-07-26T10:04:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11319.115393 secs 2024-07-26T10:04:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11301.375905 secs 2024-07-26T10:04:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11319.115393 secs 2024-07-26T10:04:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11301.375905 secs 2024-07-26T10:04:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11319.115393 secs 2024-07-26T10:04:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:33 smithi042 ceph-mon[32775]: pgmap v6564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:04:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:33 smithi164 ceph-mon[36970]: pgmap v6564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:04:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:33 smithi163 ceph-mon[36873]: pgmap v6564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:04:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:35 smithi042 ceph-mon[32775]: pgmap v6565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:04:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:35 smithi164 ceph-mon[36970]: pgmap v6565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:04:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:35 smithi163 ceph-mon[36873]: pgmap v6565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:04:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11306.376047 secs 2024-07-26T10:04:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:37 smithi042 ceph-mon[32775]: pgmap v6566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11324.115515 secs 2024-07-26T10:04:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11306.376047 secs 2024-07-26T10:04:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:37 smithi164 ceph-mon[36970]: pgmap v6566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11324.115515 secs 2024-07-26T10:04:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11306.376047 secs 2024-07-26T10:04:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:37 smithi163 ceph-mon[36873]: pgmap v6566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11324.115515 secs 2024-07-26T10:04:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:39 smithi042 ceph-mon[32775]: pgmap v6567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:39 smithi164 ceph-mon[36970]: pgmap v6567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:39 smithi163 ceph-mon[36873]: pgmap v6567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:41 smithi042 ceph-mon[32775]: pgmap v6568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:41 smithi164 ceph-mon[36970]: pgmap v6568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:41 smithi163 ceph-mon[36873]: pgmap v6568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11311.376190 secs 2024-07-26T10:04:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11329.115644 secs 2024-07-26T10:04:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11311.376190 secs 2024-07-26T10:04:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11329.115644 secs 2024-07-26T10:04:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11311.376190 secs 2024-07-26T10:04:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11329.115644 secs 2024-07-26T10:04:42.592 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:04:42.589+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:04:42.592 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:04:42.589+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:04:42.592 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:04:42.589+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:04:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:43 smithi042 ceph-mon[32775]: pgmap v6569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:43 smithi164 ceph-mon[36970]: pgmap v6569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:43 smithi163 ceph-mon[36873]: pgmap v6569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:45.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:45 smithi042 ceph-mon[32775]: pgmap v6570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:04:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:45 smithi164 ceph-mon[36970]: pgmap v6570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:04:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:45 smithi163 ceph-mon[36873]: pgmap v6570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:04:47.311 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11316.376312 secs 2024-07-26T10:04:47.311 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:47 smithi163 ceph-mon[36873]: pgmap v6571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:47.311 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11334.115770 secs 2024-07-26T10:04:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11316.376312 secs 2024-07-26T10:04:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:47 smithi042 ceph-mon[32775]: pgmap v6571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11334.115770 secs 2024-07-26T10:04:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11316.376312 secs 2024-07-26T10:04:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:47 smithi164 ceph-mon[36970]: pgmap v6571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11334.115770 secs 2024-07-26T10:04:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:49 smithi042 ceph-mon[32775]: pgmap v6572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:49 smithi164 ceph-mon[36970]: pgmap v6572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:49 smithi163 ceph-mon[36873]: pgmap v6572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:51 smithi042 ceph-mon[32775]: pgmap v6573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:51 smithi164 ceph-mon[36970]: pgmap v6573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:51 smithi163 ceph-mon[36873]: pgmap v6573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11321.376412 secs 2024-07-26T10:04:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11339.115898 secs 2024-07-26T10:04:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11321.376412 secs 2024-07-26T10:04:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11339.115898 secs 2024-07-26T10:04:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11321.376412 secs 2024-07-26T10:04:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11339.115898 secs 2024-07-26T10:04:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:53 smithi042 ceph-mon[32775]: pgmap v6574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:53 smithi164 ceph-mon[36970]: pgmap v6574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:53 smithi163 ceph-mon[36873]: pgmap v6574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:55 smithi042 ceph-mon[32775]: pgmap v6575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:04:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:55 smithi164 ceph-mon[36970]: pgmap v6575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:04:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:55 smithi163 ceph-mon[36873]: pgmap v6575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:04:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:56 smithi042 ceph-mon[32775]: pgmap v6576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:56 smithi164 ceph-mon[36970]: pgmap v6576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:56 smithi163 ceph-mon[36873]: pgmap v6576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:04:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11326.376530 secs 2024-07-26T10:04:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11344.116031 secs 2024-07-26T10:04:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11326.376530 secs 2024-07-26T10:04:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11344.116031 secs 2024-07-26T10:04:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11326.376530 secs 2024-07-26T10:04:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11344.116031 secs 2024-07-26T10:04:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:04:57.594+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:04:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:04:57.594+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:04:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:04:57.594+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:04:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:04:58 smithi042 ceph-mon[32775]: pgmap v6577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:04:58 smithi164 ceph-mon[36970]: pgmap v6577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:04:58 smithi163 ceph-mon[36873]: pgmap v6577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:04:59.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:04:59.630+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:04:59.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:04:59.630+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:04:59.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:04:59.631+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:04:59.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:04:59.631+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:05:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:01 smithi042 ceph-mon[32775]: pgmap v6578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:01 smithi164 ceph-mon[36970]: pgmap v6578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:01 smithi163 ceph-mon[36873]: pgmap v6578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11331.376630 secs 2024-07-26T10:05:02.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11349.116158 secs 2024-07-26T10:05:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11331.376630 secs 2024-07-26T10:05:02.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11349.116158 secs 2024-07-26T10:05:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11331.376630 secs 2024-07-26T10:05:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11349.116158 secs 2024-07-26T10:05:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:03 smithi042 ceph-mon[32775]: pgmap v6579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:03 smithi164 ceph-mon[36970]: pgmap v6579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:03 smithi163 ceph-mon[36873]: pgmap v6579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:05 smithi042 ceph-mon[32775]: pgmap v6580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:05:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:05 smithi164 ceph-mon[36970]: pgmap v6580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:05:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:05 smithi163 ceph-mon[36873]: pgmap v6580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:05:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11336.376701 secs 2024-07-26T10:05:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:07 smithi042 ceph-mon[32775]: pgmap v6581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11354.116285 secs 2024-07-26T10:05:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11336.376701 secs 2024-07-26T10:05:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:07 smithi164 ceph-mon[36970]: pgmap v6581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:07.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11354.116285 secs 2024-07-26T10:05:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11336.376701 secs 2024-07-26T10:05:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:07 smithi163 ceph-mon[36873]: pgmap v6581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11354.116285 secs 2024-07-26T10:05:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:09 smithi042 ceph-mon[32775]: pgmap v6582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:09 smithi164 ceph-mon[36970]: pgmap v6582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:09 smithi163 ceph-mon[36873]: pgmap v6582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:11 smithi042 ceph-mon[32775]: pgmap v6583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:11 smithi164 ceph-mon[36970]: pgmap v6583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:11 smithi163 ceph-mon[36873]: pgmap v6583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11341.376828 secs 2024-07-26T10:05:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11359.116410 secs 2024-07-26T10:05:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11341.376828 secs 2024-07-26T10:05:12.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11359.116410 secs 2024-07-26T10:05:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11341.376828 secs 2024-07-26T10:05:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11359.116410 secs 2024-07-26T10:05:12.607 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:05:12.605+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:05:12.607 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:05:12.605+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:05:12.607 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:05:12.605+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:05:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:13 smithi042 ceph-mon[32775]: pgmap v6584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:13 smithi164 ceph-mon[36970]: pgmap v6584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:13 smithi163 ceph-mon[36873]: pgmap v6584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:15 smithi042 ceph-mon[32775]: pgmap v6585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:05:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:15 smithi164 ceph-mon[36970]: pgmap v6585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:05:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:15 smithi163 ceph-mon[36873]: pgmap v6585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:05:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11346.376980 secs 2024-07-26T10:05:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:17 smithi042 ceph-mon[32775]: pgmap v6586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11364.116532 secs 2024-07-26T10:05:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11346.376980 secs 2024-07-26T10:05:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:17 smithi164 ceph-mon[36970]: pgmap v6586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11364.116532 secs 2024-07-26T10:05:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11346.376980 secs 2024-07-26T10:05:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:17 smithi163 ceph-mon[36873]: pgmap v6586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11364.116532 secs 2024-07-26T10:05:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:19 smithi042 ceph-mon[32775]: pgmap v6587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:19 smithi164 ceph-mon[36970]: pgmap v6587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:19 smithi163 ceph-mon[36873]: pgmap v6587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:21 smithi042 ceph-mon[32775]: pgmap v6588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:21 smithi164 ceph-mon[36970]: pgmap v6588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:21 smithi163 ceph-mon[36873]: pgmap v6588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11351.377122 secs 2024-07-26T10:05:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11369.116651 secs 2024-07-26T10:05:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11351.377122 secs 2024-07-26T10:05:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11369.116651 secs 2024-07-26T10:05:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11351.377122 secs 2024-07-26T10:05:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11369.116651 secs 2024-07-26T10:05:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:23 smithi042 ceph-mon[32775]: pgmap v6589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:23 smithi164 ceph-mon[36970]: pgmap v6589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:23 smithi163 ceph-mon[36873]: pgmap v6589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:05:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:05:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:05:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:25 smithi042 ceph-mon[32775]: pgmap v6590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:05:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:05:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:05:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:25 smithi042 ceph-mon[32775]: pgmap v6591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:05:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:25 smithi042 ceph-mon[32775]: pgmap v6592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:05:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:05:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:25 smithi164 ceph-mon[36970]: pgmap v6590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:05:25.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:05:25.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:05:25.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:25 smithi164 ceph-mon[36970]: pgmap v6591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:05:25.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:25 smithi164 ceph-mon[36970]: pgmap v6592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:05:25.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:05:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:25 smithi163 ceph-mon[36873]: pgmap v6590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:05:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:05:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:05:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:25 smithi163 ceph-mon[36873]: pgmap v6591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:05:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:25 smithi163 ceph-mon[36873]: pgmap v6592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:05:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:05:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11356.377252 secs 2024-07-26T10:05:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11356.377252 secs 2024-07-26T10:05:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11356.377252 secs 2024-07-26T10:05:27.612 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:05:27.610+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:05:27.612 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:05:27.610+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:05:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:05:27.610+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:05:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11374.116771 secs 2024-07-26T10:05:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:27 smithi164 ceph-mon[36970]: pgmap v6593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:05:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11374.116771 secs 2024-07-26T10:05:27.761 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:27 smithi163 ceph-mon[36873]: pgmap v6593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:05:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11374.116771 secs 2024-07-26T10:05:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:27 smithi042 ceph-mon[32775]: pgmap v6593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:05:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:29 smithi042 ceph-mon[32775]: pgmap v6594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:05:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:29 smithi164 ceph-mon[36970]: pgmap v6594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:05:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:29 smithi163 ceph-mon[36873]: pgmap v6594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:05:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:31 smithi042 ceph-mon[32775]: pgmap v6595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:05:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11361.377384 secs 2024-07-26T10:05:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:31 smithi164 ceph-mon[36970]: pgmap v6595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:05:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11361.377384 secs 2024-07-26T10:05:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:31 smithi163 ceph-mon[36873]: pgmap v6595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:05:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11361.377384 secs 2024-07-26T10:05:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11379.116900 secs 2024-07-26T10:05:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11379.116900 secs 2024-07-26T10:05:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11379.116900 secs 2024-07-26T10:05:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:33 smithi042 ceph-mon[32775]: pgmap v6596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:05:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:33 smithi164 ceph-mon[36970]: pgmap v6596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:05:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:33 smithi163 ceph-mon[36873]: pgmap v6596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:05:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:35 smithi042 ceph-mon[32775]: pgmap v6597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:05:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:35 smithi164 ceph-mon[36970]: pgmap v6597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:05:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:35 smithi163 ceph-mon[36873]: pgmap v6597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:05:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11366.377485 secs 2024-07-26T10:05:36.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11366.377485 secs 2024-07-26T10:05:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11366.377485 secs 2024-07-26T10:05:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11384.117001 secs 2024-07-26T10:05:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:37 smithi042 ceph-mon[32775]: pgmap v6598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11384.117001 secs 2024-07-26T10:05:37.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:37 smithi164 ceph-mon[36970]: pgmap v6598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11384.117001 secs 2024-07-26T10:05:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:37 smithi163 ceph-mon[36873]: pgmap v6598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:39 smithi042 ceph-mon[32775]: pgmap v6599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:05:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:39 smithi164 ceph-mon[36970]: pgmap v6599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:05:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:39 smithi163 ceph-mon[36873]: pgmap v6599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:05:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:41 smithi042 ceph-mon[32775]: pgmap v6600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11371.377617 secs 2024-07-26T10:05:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:41 smithi164 ceph-mon[36970]: pgmap v6600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11371.377617 secs 2024-07-26T10:05:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:41 smithi163 ceph-mon[36873]: pgmap v6600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11371.377617 secs 2024-07-26T10:05:42.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:05:42.620+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:05:42.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:05:42.620+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:05:42.622 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:05:42.620+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:05:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11389.117130 secs 2024-07-26T10:05:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11389.117130 secs 2024-07-26T10:05:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11389.117130 secs 2024-07-26T10:05:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:43 smithi042 ceph-mon[32775]: pgmap v6601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:43 smithi164 ceph-mon[36970]: pgmap v6601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:43 smithi163 ceph-mon[36873]: pgmap v6601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:44.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:05:44.638+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:05:44.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:05:44.638+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:05:44.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:05:44.639+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:05:44.641 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:05:44.639+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:05:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:45 smithi042 ceph-mon[32775]: pgmap v6602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:45 smithi164 ceph-mon[36970]: pgmap v6602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:45 smithi163 ceph-mon[36873]: pgmap v6602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11376.377743 secs 2024-07-26T10:05:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11376.377743 secs 2024-07-26T10:05:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11376.377743 secs 2024-07-26T10:05:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11394.117214 secs 2024-07-26T10:05:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:47 smithi042 ceph-mon[32775]: pgmap v6603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:47.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11394.117214 secs 2024-07-26T10:05:47.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:47 smithi163 ceph-mon[36873]: pgmap v6603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:47.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11394.117214 secs 2024-07-26T10:05:47.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:47 smithi164 ceph-mon[36970]: pgmap v6603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:49 smithi042 ceph-mon[32775]: pgmap v6604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:05:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:49 smithi164 ceph-mon[36970]: pgmap v6604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:05:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:49 smithi163 ceph-mon[36873]: pgmap v6604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:05:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:51 smithi042 ceph-mon[32775]: pgmap v6605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11381.377872 secs 2024-07-26T10:05:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:51 smithi164 ceph-mon[36970]: pgmap v6605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11381.377872 secs 2024-07-26T10:05:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:51 smithi163 ceph-mon[36873]: pgmap v6605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11381.377872 secs 2024-07-26T10:05:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11399.117313 secs 2024-07-26T10:05:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11399.117313 secs 2024-07-26T10:05:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11399.117313 secs 2024-07-26T10:05:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:53 smithi042 ceph-mon[32775]: pgmap v6606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:53 smithi164 ceph-mon[36970]: pgmap v6606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:53 smithi163 ceph-mon[36873]: pgmap v6606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:55 smithi042 ceph-mon[32775]: pgmap v6607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:55 smithi164 ceph-mon[36970]: pgmap v6607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:55 smithi163 ceph-mon[36873]: pgmap v6607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:05:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11386.377998 secs 2024-07-26T10:05:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11386.377998 secs 2024-07-26T10:05:57.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11386.377998 secs 2024-07-26T10:05:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:05:57.625+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:05:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:05:57.625+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:05:57.627 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:05:57.625+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:05:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11404.117392 secs 2024-07-26T10:05:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:57 smithi164 ceph-mon[36970]: pgmap v6608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11404.117392 secs 2024-07-26T10:05:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:57 smithi163 ceph-mon[36873]: pgmap v6608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11404.117392 secs 2024-07-26T10:05:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:57 smithi042 ceph-mon[32775]: pgmap v6608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:05:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:05:59 smithi164 ceph-mon[36970]: pgmap v6609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:06:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:05:59 smithi163 ceph-mon[36873]: pgmap v6609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:06:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:05:59 smithi042 ceph-mon[32775]: pgmap v6609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:06:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:01 smithi164 ceph-mon[36970]: pgmap v6610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11391.378162 secs 2024-07-26T10:06:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:01 smithi163 ceph-mon[36873]: pgmap v6610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11391.378162 secs 2024-07-26T10:06:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:01 smithi042 ceph-mon[32775]: pgmap v6610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11391.378162 secs 2024-07-26T10:06:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11409.117514 secs 2024-07-26T10:06:03.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11409.117514 secs 2024-07-26T10:06:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11409.117514 secs 2024-07-26T10:06:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:03 smithi164 ceph-mon[36970]: pgmap v6611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:04.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:03 smithi163 ceph-mon[36873]: pgmap v6611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:03 smithi042 ceph-mon[32775]: pgmap v6611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:05 smithi164 ceph-mon[36970]: pgmap v6612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:05 smithi163 ceph-mon[36873]: pgmap v6612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:05 smithi042 ceph-mon[32775]: pgmap v6612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11396.378269 secs 2024-07-26T10:06:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11396.378269 secs 2024-07-26T10:06:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11396.378269 secs 2024-07-26T10:06:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11414.117637 secs 2024-07-26T10:06:07.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:07 smithi164 ceph-mon[36970]: pgmap v6613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11414.117637 secs 2024-07-26T10:06:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:07 smithi163 ceph-mon[36873]: pgmap v6613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11414.117637 secs 2024-07-26T10:06:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:07 smithi042 ceph-mon[32775]: pgmap v6613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:09 smithi164 ceph-mon[36970]: pgmap v6614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:06:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:09 smithi163 ceph-mon[36873]: pgmap v6614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:06:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:09 smithi042 ceph-mon[32775]: pgmap v6614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:06:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:11 smithi164 ceph-mon[36970]: pgmap v6615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11401.378386 secs 2024-07-26T10:06:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:11 smithi163 ceph-mon[36873]: pgmap v6615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11401.378386 secs 2024-07-26T10:06:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:11 smithi042 ceph-mon[32775]: pgmap v6615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11401.378386 secs 2024-07-26T10:06:12.638 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:06:12.635+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:06:12.638 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:06:12.635+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:06:12.638 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:06:12.635+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:06:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11419.117737 secs 2024-07-26T10:06:12.977 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11419.117737 secs 2024-07-26T10:06:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11419.117737 secs 2024-07-26T10:06:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:13 smithi164 ceph-mon[36970]: pgmap v6616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:13 smithi163 ceph-mon[36873]: pgmap v6616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:13 smithi042 ceph-mon[32775]: pgmap v6616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:15 smithi164 ceph-mon[36970]: pgmap v6617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:15 smithi163 ceph-mon[36873]: pgmap v6617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:15 smithi042 ceph-mon[32775]: pgmap v6617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11406.378489 secs 2024-07-26T10:06:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11406.378489 secs 2024-07-26T10:06:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11406.378489 secs 2024-07-26T10:06:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11424.117836 secs 2024-07-26T10:06:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:17 smithi164 ceph-mon[36970]: pgmap v6618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11424.117836 secs 2024-07-26T10:06:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:17 smithi163 ceph-mon[36873]: pgmap v6618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11424.117836 secs 2024-07-26T10:06:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:17 smithi042 ceph-mon[32775]: pgmap v6618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:19 smithi164 ceph-mon[36970]: pgmap v6619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:06:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:19 smithi163 ceph-mon[36873]: pgmap v6619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:06:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:19 smithi042 ceph-mon[32775]: pgmap v6619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:06:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:21 smithi164 ceph-mon[36970]: pgmap v6620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11411.378639 secs 2024-07-26T10:06:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:21 smithi163 ceph-mon[36873]: pgmap v6620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11411.378639 secs 2024-07-26T10:06:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:21 smithi042 ceph-mon[32775]: pgmap v6620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11411.378639 secs 2024-07-26T10:06:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11429.117990 secs 2024-07-26T10:06:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11429.117990 secs 2024-07-26T10:06:23.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11429.117990 secs 2024-07-26T10:06:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:23 smithi164 ceph-mon[36970]: pgmap v6621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:23 smithi163 ceph-mon[36873]: pgmap v6621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:23 smithi042 ceph-mon[32775]: pgmap v6621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:06:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:06:25.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:06:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:25 smithi163 ceph-mon[36873]: pgmap v6622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:06:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:06:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:25 smithi163 ceph-mon[36873]: pgmap v6623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:06:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:25 smithi163 ceph-mon[36873]: pgmap v6624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:06:26.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:06:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:25 smithi042 ceph-mon[32775]: pgmap v6622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:06:26.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:06:26.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:25 smithi042 ceph-mon[32775]: pgmap v6623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:06:26.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:25 smithi042 ceph-mon[32775]: pgmap v6624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:06:26.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:06:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:25 smithi164 ceph-mon[36970]: pgmap v6622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:26.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:06:26.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:06:26.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:25 smithi164 ceph-mon[36970]: pgmap v6623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:06:26.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:25 smithi164 ceph-mon[36970]: pgmap v6624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:06:26.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:06:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11416.378766 secs 2024-07-26T10:06:27.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11416.378766 secs 2024-07-26T10:06:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11416.378766 secs 2024-07-26T10:06:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:06:27.641+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:06:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:06:27.641+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:06:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:06:27.641+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:06:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11434.118111 secs 2024-07-26T10:06:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:27 smithi163 ceph-mon[36873]: pgmap v6625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:06:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11434.118111 secs 2024-07-26T10:06:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:27 smithi042 ceph-mon[32775]: pgmap v6625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:06:28.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11434.118111 secs 2024-07-26T10:06:28.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:27 smithi164 ceph-mon[36970]: pgmap v6625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:06:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:29 smithi042 ceph-mon[32775]: pgmap v6626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:06:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:29 smithi164 ceph-mon[36970]: pgmap v6626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:06:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:29 smithi163 ceph-mon[36873]: pgmap v6626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:06:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:31 smithi042 ceph-mon[32775]: pgmap v6627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:06:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:31 smithi164 ceph-mon[36970]: pgmap v6627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:06:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:31 smithi163 ceph-mon[36873]: pgmap v6627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:06:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11421.378827 secs 2024-07-26T10:06:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11439.118237 secs 2024-07-26T10:06:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11421.378827 secs 2024-07-26T10:06:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11439.118237 secs 2024-07-26T10:06:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11421.378827 secs 2024-07-26T10:06:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11439.118237 secs 2024-07-26T10:06:33.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:33 smithi164 ceph-mon[36970]: pgmap v6628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:06:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:33 smithi163 ceph-mon[36873]: pgmap v6628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:06:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:33 smithi042 ceph-mon[32775]: pgmap v6628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:06:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:35 smithi042 ceph-mon[32775]: pgmap v6629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:06:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:35 smithi164 ceph-mon[36970]: pgmap v6629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:06:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:35 smithi163 ceph-mon[36873]: pgmap v6629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:06:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11426.378955 secs 2024-07-26T10:06:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11426.378955 secs 2024-07-26T10:06:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11426.378955 secs 2024-07-26T10:06:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11444.118356 secs 2024-07-26T10:06:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:37 smithi164 ceph-mon[36970]: pgmap v6630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11444.118356 secs 2024-07-26T10:06:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:37 smithi163 ceph-mon[36873]: pgmap v6630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11444.118356 secs 2024-07-26T10:06:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:37 smithi042 ceph-mon[32775]: pgmap v6630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:39 smithi042 ceph-mon[32775]: pgmap v6631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:39 smithi164 ceph-mon[36970]: pgmap v6631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:39 smithi163 ceph-mon[36873]: pgmap v6631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:41 smithi042 ceph-mon[32775]: pgmap v6632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:41 smithi164 ceph-mon[36970]: pgmap v6632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:41 smithi163 ceph-mon[36873]: pgmap v6632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11431.379051 secs 2024-07-26T10:06:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11449.118477 secs 2024-07-26T10:06:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11431.379051 secs 2024-07-26T10:06:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11449.118477 secs 2024-07-26T10:06:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11431.379051 secs 2024-07-26T10:06:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11449.118477 secs 2024-07-26T10:06:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:06:42.648+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:06:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:06:42.648+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:06:42.650 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:06:42.648+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:06:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:43 smithi164 ceph-mon[36970]: pgmap v6633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:43 smithi163 ceph-mon[36873]: pgmap v6633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:43 smithi042 ceph-mon[32775]: pgmap v6633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:45.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:45 smithi042 ceph-mon[32775]: pgmap v6634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:45 smithi164 ceph-mon[36970]: pgmap v6634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:45 smithi163 ceph-mon[36873]: pgmap v6634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11436.379175 secs 2024-07-26T10:06:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11436.379175 secs 2024-07-26T10:06:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11436.379175 secs 2024-07-26T10:06:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11454.118598 secs 2024-07-26T10:06:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:47 smithi164 ceph-mon[36970]: pgmap v6635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11454.118598 secs 2024-07-26T10:06:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:47 smithi163 ceph-mon[36873]: pgmap v6635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11454.118598 secs 2024-07-26T10:06:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:47 smithi042 ceph-mon[32775]: pgmap v6635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:49 smithi042 ceph-mon[32775]: pgmap v6636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:49 smithi164 ceph-mon[36970]: pgmap v6636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:49 smithi163 ceph-mon[36873]: pgmap v6636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:51 smithi042 ceph-mon[32775]: pgmap v6637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:51 smithi164 ceph-mon[36970]: pgmap v6637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:51 smithi163 ceph-mon[36873]: pgmap v6637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11441.379303 secs 2024-07-26T10:06:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11459.118756 secs 2024-07-26T10:06:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11441.379303 secs 2024-07-26T10:06:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11459.118756 secs 2024-07-26T10:06:52.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11441.379303 secs 2024-07-26T10:06:52.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11459.118756 secs 2024-07-26T10:06:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:53 smithi164 ceph-mon[36970]: pgmap v6638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:53 smithi163 ceph-mon[36873]: pgmap v6638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:53.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:53 smithi042 ceph-mon[32775]: pgmap v6638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:55 smithi042 ceph-mon[32775]: pgmap v6639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:55 smithi164 ceph-mon[36970]: pgmap v6639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:55 smithi163 ceph-mon[36873]: pgmap v6639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:06:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11446.379397 secs 2024-07-26T10:06:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11446.379397 secs 2024-07-26T10:06:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11446.379397 secs 2024-07-26T10:06:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:06:57.654+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:06:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:06:57.654+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:06:57.656 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:06:57.654+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:06:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11464.118812 secs 2024-07-26T10:06:57.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:57 smithi164 ceph-mon[36970]: pgmap v6640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11464.118812 secs 2024-07-26T10:06:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:57 smithi163 ceph-mon[36873]: pgmap v6640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11464.118812 secs 2024-07-26T10:06:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:57 smithi042 ceph-mon[32775]: pgmap v6640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:06:59 smithi042 ceph-mon[32775]: pgmap v6641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:06:59 smithi164 ceph-mon[36970]: pgmap v6641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:06:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:06:59 smithi163 ceph-mon[36873]: pgmap v6641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:01 smithi164 ceph-mon[36970]: pgmap v6642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:01 smithi163 ceph-mon[36873]: pgmap v6642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:01 smithi042 ceph-mon[32775]: pgmap v6642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11451.379474 secs 2024-07-26T10:07:02.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11469.118964 secs 2024-07-26T10:07:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11451.379474 secs 2024-07-26T10:07:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11469.118964 secs 2024-07-26T10:07:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11451.379474 secs 2024-07-26T10:07:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11469.118964 secs 2024-07-26T10:07:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:03 smithi164 ceph-mon[36970]: pgmap v6643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:03 smithi163 ceph-mon[36873]: pgmap v6643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:03 smithi042 ceph-mon[32775]: pgmap v6643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:05 smithi164 ceph-mon[36970]: pgmap v6644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:05 smithi163 ceph-mon[36873]: pgmap v6644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:05 smithi042 ceph-mon[32775]: pgmap v6644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11456.379600 secs 2024-07-26T10:07:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11456.379600 secs 2024-07-26T10:07:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11456.379600 secs 2024-07-26T10:07:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11474.119022 secs 2024-07-26T10:07:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:07 smithi163 ceph-mon[36873]: pgmap v6645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11474.119022 secs 2024-07-26T10:07:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:07 smithi042 ceph-mon[32775]: pgmap v6645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11474.119022 secs 2024-07-26T10:07:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:07 smithi164 ceph-mon[36970]: pgmap v6645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:09 smithi164 ceph-mon[36970]: pgmap v6646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:09 smithi163 ceph-mon[36873]: pgmap v6646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:09 smithi042 ceph-mon[32775]: pgmap v6646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:11 smithi164 ceph-mon[36970]: pgmap v6647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:11 smithi163 ceph-mon[36873]: pgmap v6647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:11 smithi042 ceph-mon[32775]: pgmap v6647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11461.379700 secs 2024-07-26T10:07:12.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11479.119088 secs 2024-07-26T10:07:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11461.379700 secs 2024-07-26T10:07:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11479.119088 secs 2024-07-26T10:07:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11461.379700 secs 2024-07-26T10:07:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11479.119088 secs 2024-07-26T10:07:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:07:12.655+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:07:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:07:12.655+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:07:12.657 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:07:12.655+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:07:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:13 smithi164 ceph-mon[36970]: pgmap v6648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:13 smithi163 ceph-mon[36873]: pgmap v6648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:13 smithi042 ceph-mon[32775]: pgmap v6648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:15 smithi042 ceph-mon[32775]: pgmap v6649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:15 smithi164 ceph-mon[36970]: pgmap v6649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:15 smithi163 ceph-mon[36873]: pgmap v6649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11466.379823 secs 2024-07-26T10:07:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11466.379823 secs 2024-07-26T10:07:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11466.379823 secs 2024-07-26T10:07:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11484.119209 secs 2024-07-26T10:07:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:17 smithi164 ceph-mon[36970]: pgmap v6650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11484.119209 secs 2024-07-26T10:07:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:17 smithi163 ceph-mon[36873]: pgmap v6650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11484.119209 secs 2024-07-26T10:07:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:17 smithi042 ceph-mon[32775]: pgmap v6650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:19 smithi164 ceph-mon[36970]: pgmap v6651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:19 smithi163 ceph-mon[36873]: pgmap v6651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:19 smithi042 ceph-mon[32775]: pgmap v6651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:21 smithi164 ceph-mon[36970]: pgmap v6652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:21 smithi163 ceph-mon[36873]: pgmap v6652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:21 smithi042 ceph-mon[32775]: pgmap v6652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11471.379944 secs 2024-07-26T10:07:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11489.119340 secs 2024-07-26T10:07:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11471.379944 secs 2024-07-26T10:07:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11489.119340 secs 2024-07-26T10:07:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11471.379944 secs 2024-07-26T10:07:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11489.119340 secs 2024-07-26T10:07:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:23 smithi164 ceph-mon[36970]: pgmap v6653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:23 smithi163 ceph-mon[36873]: pgmap v6653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:23.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:23 smithi042 ceph-mon[32775]: pgmap v6653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:25 smithi164 ceph-mon[36970]: pgmap v6654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:25 smithi163 ceph-mon[36873]: pgmap v6654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:25 smithi042 ceph-mon[32775]: pgmap v6654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:07:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:07:26.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:07:26.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:26 smithi164 ceph-mon[36970]: pgmap v6655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:07:26.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:26 smithi164 ceph-mon[36970]: pgmap v6656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:07:26.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:07:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:07:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:07:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:07:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:26 smithi163 ceph-mon[36873]: pgmap v6655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:07:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:26 smithi163 ceph-mon[36873]: pgmap v6656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:07:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:07:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:07:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:07:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:07:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:26 smithi042 ceph-mon[32775]: pgmap v6655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:07:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:26 smithi042 ceph-mon[32775]: pgmap v6656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:07:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:07:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11476.380034 secs 2024-07-26T10:07:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11494.119463 secs 2024-07-26T10:07:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11476.380034 secs 2024-07-26T10:07:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11494.119463 secs 2024-07-26T10:07:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11476.380034 secs 2024-07-26T10:07:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11494.119463 secs 2024-07-26T10:07:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:07:27.660+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:07:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:07:27.660+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:07:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:07:27.661+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:07:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:28 smithi164 ceph-mon[36970]: pgmap v6657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:07:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:28 smithi163 ceph-mon[36873]: pgmap v6657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:07:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:28 smithi042 ceph-mon[32775]: pgmap v6657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:07:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:30 smithi164 ceph-mon[36970]: pgmap v6658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:07:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:30 smithi163 ceph-mon[36873]: pgmap v6658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:07:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:30 smithi042 ceph-mon[32775]: pgmap v6658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:07:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11481.380159 secs 2024-07-26T10:07:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11481.380159 secs 2024-07-26T10:07:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11481.380159 secs 2024-07-26T10:07:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11499.119584 secs 2024-07-26T10:07:32.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:32 smithi164 ceph-mon[36970]: pgmap v6659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:07:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11499.119584 secs 2024-07-26T10:07:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:32 smithi163 ceph-mon[36873]: pgmap v6659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:07:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11499.119584 secs 2024-07-26T10:07:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:32 smithi042 ceph-mon[32775]: pgmap v6659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:07:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:34 smithi164 ceph-mon[36970]: pgmap v6660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:07:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:34 smithi163 ceph-mon[36873]: pgmap v6660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:07:35.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:34 smithi042 ceph-mon[32775]: pgmap v6660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:07:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:36 smithi164 ceph-mon[36970]: pgmap v6661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:07:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11486.380281 secs 2024-07-26T10:07:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:36 smithi163 ceph-mon[36873]: pgmap v6661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:07:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11486.380281 secs 2024-07-26T10:07:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:36 smithi042 ceph-mon[32775]: pgmap v6661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:07:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11486.380281 secs 2024-07-26T10:07:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11504.119683 secs 2024-07-26T10:07:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11504.119683 secs 2024-07-26T10:07:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11504.119683 secs 2024-07-26T10:07:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:38 smithi164 ceph-mon[36970]: pgmap v6662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:38 smithi163 ceph-mon[36873]: pgmap v6662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:39.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:38 smithi042 ceph-mon[32775]: pgmap v6662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:40 smithi163 ceph-mon[36873]: pgmap v6663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:41.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:40 smithi042 ceph-mon[32775]: pgmap v6663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:40 smithi164 ceph-mon[36970]: pgmap v6663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11491.380407 secs 2024-07-26T10:07:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11491.380407 secs 2024-07-26T10:07:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11491.380407 secs 2024-07-26T10:07:42.663 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:07:42.661+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:07:42.663 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:07:42.661+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:07:42.663 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:07:42.661+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:07:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11509.119805 secs 2024-07-26T10:07:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:42 smithi163 ceph-mon[36873]: pgmap v6664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11509.119805 secs 2024-07-26T10:07:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:42 smithi042 ceph-mon[32775]: pgmap v6664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11509.119805 secs 2024-07-26T10:07:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:42 smithi164 ceph-mon[36970]: pgmap v6664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:44 smithi163 ceph-mon[36873]: pgmap v6665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:07:45.137 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:44 smithi042 ceph-mon[32775]: pgmap v6665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:07:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:44 smithi164 ceph-mon[36970]: pgmap v6665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:07:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:46 smithi163 ceph-mon[36873]: pgmap v6666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11496.380532 secs 2024-07-26T10:07:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:46 smithi042 ceph-mon[32775]: pgmap v6666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11496.380532 secs 2024-07-26T10:07:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:46 smithi164 ceph-mon[36970]: pgmap v6666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11496.380532 secs 2024-07-26T10:07:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11514.119911 secs 2024-07-26T10:07:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11514.119911 secs 2024-07-26T10:07:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11514.119911 secs 2024-07-26T10:07:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:48 smithi163 ceph-mon[36873]: pgmap v6667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:48 smithi042 ceph-mon[32775]: pgmap v6667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:48 smithi164 ceph-mon[36970]: pgmap v6667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:50 smithi163 ceph-mon[36873]: pgmap v6668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:50 smithi042 ceph-mon[32775]: pgmap v6668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:50 smithi164 ceph-mon[36970]: pgmap v6668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11501.380657 secs 2024-07-26T10:07:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11501.380657 secs 2024-07-26T10:07:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11501.380657 secs 2024-07-26T10:07:53.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11519.120007 secs 2024-07-26T10:07:53.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:52 smithi042 ceph-mon[32775]: pgmap v6669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11519.120007 secs 2024-07-26T10:07:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:52 smithi164 ceph-mon[36970]: pgmap v6669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11519.120007 secs 2024-07-26T10:07:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:52 smithi163 ceph-mon[36873]: pgmap v6669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:54 smithi042 ceph-mon[32775]: pgmap v6670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:07:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:54 smithi164 ceph-mon[36970]: pgmap v6670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:07:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:54 smithi163 ceph-mon[36873]: pgmap v6670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:07:57.026 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:56 smithi164 ceph-mon[36970]: pgmap v6671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:57.026 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11506.380779 secs 2024-07-26T10:07:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:56 smithi042 ceph-mon[32775]: pgmap v6671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11506.380779 secs 2024-07-26T10:07:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:56 smithi163 ceph-mon[36873]: pgmap v6671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:07:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11506.380779 secs 2024-07-26T10:07:57.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:07:57.666+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:07:57.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:07:57.666+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:07:57.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:07:57.666+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:07:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:57 smithi042 ceph-mon[32775]: 1 slow requests, 1 included below; oldest blocked for > 11524.120124 secs 2024-07-26T10:07:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:57 smithi042 ceph-mon[32775]: slow request 11524.120123 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T10:07:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:57 smithi164 ceph-mon[36970]: 1 slow requests, 1 included below; oldest blocked for > 11524.120124 secs 2024-07-26T10:07:58.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:57 smithi164 ceph-mon[36970]: slow request 11524.120123 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T10:07:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:57 smithi163 ceph-mon[36873]: 1 slow requests, 1 included below; oldest blocked for > 11524.120124 secs 2024-07-26T10:07:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:57 smithi163 ceph-mon[36873]: slow request 11524.120123 seconds old, received at 2024-07-26T06:55:52.142048+0000: client_request(client.25018:18310 rmdir #0x10000000007/src 2024-07-26T06:55:52.140899+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T10:07:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:07:58 smithi042 ceph-mon[32775]: pgmap v6672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:07:58 smithi164 ceph-mon[36970]: pgmap v6672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:07:58 smithi163 ceph-mon[36873]: pgmap v6672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:07:59.663 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:07:59.660+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:07:59.663 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:07:59.660+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:07:59.663 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:07:59.661+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:07:59.663 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:07:59.661+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:08:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:00 smithi042 ceph-mon[32775]: pgmap v6673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:00 smithi164 ceph-mon[36970]: pgmap v6673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:00 smithi163 ceph-mon[36873]: pgmap v6673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11511.380903 secs 2024-07-26T10:08:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11511.380903 secs 2024-07-26T10:08:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11511.380903 secs 2024-07-26T10:08:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11529.120192 secs 2024-07-26T10:08:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:02 smithi042 ceph-mon[32775]: pgmap v6674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11529.120192 secs 2024-07-26T10:08:03.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:02 smithi164 ceph-mon[36970]: pgmap v6674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11529.120192 secs 2024-07-26T10:08:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:02 smithi163 ceph-mon[36873]: pgmap v6674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:05.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:04 smithi042 ceph-mon[32775]: pgmap v6675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:08:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:04 smithi164 ceph-mon[36970]: pgmap v6675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:08:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:04 smithi163 ceph-mon[36873]: pgmap v6675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:08:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:06 smithi042 ceph-mon[32775]: pgmap v6676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11516.380969 secs 2024-07-26T10:08:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:06 smithi164 ceph-mon[36970]: pgmap v6676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11516.380969 secs 2024-07-26T10:08:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:06 smithi163 ceph-mon[36873]: pgmap v6676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11516.380969 secs 2024-07-26T10:08:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11534.120284 secs 2024-07-26T10:08:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11534.120284 secs 2024-07-26T10:08:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11534.120284 secs 2024-07-26T10:08:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:08 smithi042 ceph-mon[32775]: pgmap v6677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:08 smithi164 ceph-mon[36970]: pgmap v6677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:09.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:08 smithi163 ceph-mon[36873]: pgmap v6677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:10 smithi042 ceph-mon[32775]: pgmap v6678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:10 smithi164 ceph-mon[36970]: pgmap v6678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:10 smithi163 ceph-mon[36873]: pgmap v6678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:11 smithi042 ceph-mon[32775]: 1 slow requests, 1 included below; oldest blocked for > 11521.381096 secs 2024-07-26T10:08:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:11 smithi042 ceph-mon[32775]: slow request 11521.381096 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T10:08:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:11 smithi164 ceph-mon[36970]: 1 slow requests, 1 included below; oldest blocked for > 11521.381096 secs 2024-07-26T10:08:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:11 smithi164 ceph-mon[36970]: slow request 11521.381096 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T10:08:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:11 smithi163 ceph-mon[36873]: 1 slow requests, 1 included below; oldest blocked for > 11521.381096 secs 2024-07-26T10:08:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:11 smithi163 ceph-mon[36873]: slow request 11521.381096 seconds old, received at 2024-07-26T06:56:09.541762+0000: peer_request:client.25018:18310 currently dispatched 2024-07-26T10:08:12.669 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:08:12.667+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:08:12.669 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:08:12.667+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:08:12.669 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:08:12.667+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:08:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11539.120360 secs 2024-07-26T10:08:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:12 smithi042 ceph-mon[32775]: pgmap v6679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11539.120360 secs 2024-07-26T10:08:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:12 smithi164 ceph-mon[36970]: pgmap v6679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11539.120360 secs 2024-07-26T10:08:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:12 smithi163 ceph-mon[36873]: pgmap v6679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:14 smithi042 ceph-mon[32775]: pgmap v6680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:08:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:14 smithi164 ceph-mon[36970]: pgmap v6680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:08:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:14 smithi163 ceph-mon[36873]: pgmap v6680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:08:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:16 smithi042 ceph-mon[32775]: pgmap v6681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11526.381242 secs 2024-07-26T10:08:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11544.120459 secs 2024-07-26T10:08:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:16 smithi164 ceph-mon[36970]: pgmap v6681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11526.381242 secs 2024-07-26T10:08:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11544.120459 secs 2024-07-26T10:08:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:16 smithi163 ceph-mon[36873]: pgmap v6681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11526.381242 secs 2024-07-26T10:08:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11544.120459 secs 2024-07-26T10:08:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:18 smithi042 ceph-mon[32775]: pgmap v6682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:18 smithi164 ceph-mon[36970]: pgmap v6682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:19.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:18 smithi163 ceph-mon[36873]: pgmap v6682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:20 smithi042 ceph-mon[32775]: pgmap v6683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:20 smithi164 ceph-mon[36970]: pgmap v6683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:20 smithi163 ceph-mon[36873]: pgmap v6683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11531.381351 secs 2024-07-26T10:08:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11549.120597 secs 2024-07-26T10:08:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11531.381351 secs 2024-07-26T10:08:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11549.120597 secs 2024-07-26T10:08:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11531.381351 secs 2024-07-26T10:08:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11549.120597 secs 2024-07-26T10:08:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:22 smithi042 ceph-mon[32775]: pgmap v6684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:22 smithi164 ceph-mon[36970]: pgmap v6684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:22 smithi163 ceph-mon[36873]: pgmap v6684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:25.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:24 smithi042 ceph-mon[32775]: pgmap v6685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:08:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:24 smithi164 ceph-mon[36970]: pgmap v6685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:08:25.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:24 smithi163 ceph-mon[36873]: pgmap v6685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:08:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:08:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:08:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:08:27.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:26 smithi164 ceph-mon[36970]: pgmap v6686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:27.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11536.381448 secs 2024-07-26T10:08:27.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:08:27.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:08:27.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:08:27.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11554.120728 secs 2024-07-26T10:08:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:26 smithi042 ceph-mon[32775]: pgmap v6686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11536.381448 secs 2024-07-26T10:08:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:08:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:08:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:08:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11554.120728 secs 2024-07-26T10:08:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:26 smithi163 ceph-mon[36873]: pgmap v6686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11536.381448 secs 2024-07-26T10:08:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:08:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:08:27.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:08:27.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11554.120728 secs 2024-07-26T10:08:27.674 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:08:27.672+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:08:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:08:27.672+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:08:27.675 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:08:27.672+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:08:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:27 smithi042 ceph-mon[32775]: pgmap v6687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:08:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:27 smithi042 ceph-mon[32775]: pgmap v6688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:08:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:27 smithi164 ceph-mon[36970]: pgmap v6687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:08:28.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:27 smithi164 ceph-mon[36970]: pgmap v6688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:08:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:27 smithi163 ceph-mon[36873]: pgmap v6687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:08:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:27 smithi163 ceph-mon[36873]: pgmap v6688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:08:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:29 smithi042 ceph-mon[32775]: pgmap v6689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:08:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:29 smithi164 ceph-mon[36970]: pgmap v6689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:08:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:29 smithi163 ceph-mon[36873]: pgmap v6689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:08:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:31 smithi042 ceph-mon[32775]: pgmap v6690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:08:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11541.381580 secs 2024-07-26T10:08:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11559.120852 secs 2024-07-26T10:08:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:31 smithi164 ceph-mon[36970]: pgmap v6690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:08:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11541.381580 secs 2024-07-26T10:08:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11559.120852 secs 2024-07-26T10:08:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:31 smithi163 ceph-mon[36873]: pgmap v6690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:08:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11541.381580 secs 2024-07-26T10:08:32.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11559.120852 secs 2024-07-26T10:08:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:33 smithi042 ceph-mon[32775]: pgmap v6691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:08:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:33 smithi164 ceph-mon[36970]: pgmap v6691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:08:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:33 smithi163 ceph-mon[36873]: pgmap v6691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:08:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:35 smithi042 ceph-mon[32775]: pgmap v6692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:08:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:35 smithi164 ceph-mon[36970]: pgmap v6692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:08:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:35 smithi163 ceph-mon[36873]: pgmap v6692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:08:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11546.381704 secs 2024-07-26T10:08:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11564.121000 secs 2024-07-26T10:08:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11546.381704 secs 2024-07-26T10:08:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11564.121000 secs 2024-07-26T10:08:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11546.381704 secs 2024-07-26T10:08:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11564.121000 secs 2024-07-26T10:08:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:37 smithi042 ceph-mon[32775]: pgmap v6693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:08:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:37 smithi164 ceph-mon[36970]: pgmap v6693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:08:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:37 smithi163 ceph-mon[36873]: pgmap v6693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:08:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:39 smithi042 ceph-mon[32775]: pgmap v6694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:39 smithi164 ceph-mon[36970]: pgmap v6694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:39 smithi163 ceph-mon[36873]: pgmap v6694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:41 smithi042 ceph-mon[32775]: pgmap v6695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11551.381865 secs 2024-07-26T10:08:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11569.121130 secs 2024-07-26T10:08:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:41 smithi164 ceph-mon[36970]: pgmap v6695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11551.381865 secs 2024-07-26T10:08:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11569.121130 secs 2024-07-26T10:08:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:41 smithi163 ceph-mon[36873]: pgmap v6695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11551.381865 secs 2024-07-26T10:08:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11569.121130 secs 2024-07-26T10:08:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:08:42.673+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:08:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:08:42.673+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:08:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:08:42.673+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:08:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:43 smithi042 ceph-mon[32775]: pgmap v6696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:43 smithi164 ceph-mon[36970]: pgmap v6696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:43 smithi163 ceph-mon[36873]: pgmap v6696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:45 smithi042 ceph-mon[32775]: pgmap v6697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:45 smithi164 ceph-mon[36970]: pgmap v6697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:45 smithi163 ceph-mon[36873]: pgmap v6697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11556.381990 secs 2024-07-26T10:08:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11574.121256 secs 2024-07-26T10:08:47.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11556.381990 secs 2024-07-26T10:08:47.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11574.121256 secs 2024-07-26T10:08:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11556.381990 secs 2024-07-26T10:08:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11574.121256 secs 2024-07-26T10:08:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:47 smithi042 ceph-mon[32775]: pgmap v6698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:47 smithi164 ceph-mon[36970]: pgmap v6698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:47 smithi163 ceph-mon[36873]: pgmap v6698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:49 smithi042 ceph-mon[32775]: pgmap v6699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:49 smithi164 ceph-mon[36970]: pgmap v6699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:49 smithi163 ceph-mon[36873]: pgmap v6699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:51 smithi164 ceph-mon[36970]: pgmap v6700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11561.382126 secs 2024-07-26T10:08:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11579.121381 secs 2024-07-26T10:08:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:51 smithi163 ceph-mon[36873]: pgmap v6700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11561.382126 secs 2024-07-26T10:08:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11579.121381 secs 2024-07-26T10:08:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:51 smithi042 ceph-mon[32775]: pgmap v6700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11561.382126 secs 2024-07-26T10:08:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11579.121381 secs 2024-07-26T10:08:54.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:53 smithi042 ceph-mon[32775]: pgmap v6701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:54.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:53 smithi164 ceph-mon[36970]: pgmap v6701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:53 smithi163 ceph-mon[36873]: pgmap v6701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:55 smithi164 ceph-mon[36970]: pgmap v6702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:55 smithi163 ceph-mon[36873]: pgmap v6702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:55 smithi042 ceph-mon[32775]: pgmap v6702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:08:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11566.382257 secs 2024-07-26T10:08:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11584.121499 secs 2024-07-26T10:08:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11566.382257 secs 2024-07-26T10:08:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11584.121499 secs 2024-07-26T10:08:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11566.382257 secs 2024-07-26T10:08:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11584.121499 secs 2024-07-26T10:08:57.680 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:08:57.678+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:08:57.680 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:08:57.678+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:08:57.681 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:08:57.678+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:08:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:57 smithi164 ceph-mon[36970]: pgmap v6703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:57 smithi163 ceph-mon[36873]: pgmap v6703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:08:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:57 smithi042 ceph-mon[32775]: pgmap v6703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:08:59 smithi164 ceph-mon[36970]: pgmap v6704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:08:59 smithi163 ceph-mon[36873]: pgmap v6704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:08:59 smithi042 ceph-mon[32775]: pgmap v6704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:01 smithi164 ceph-mon[36970]: pgmap v6705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11571.382356 secs 2024-07-26T10:09:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11589.121618 secs 2024-07-26T10:09:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:01 smithi163 ceph-mon[36873]: pgmap v6705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11571.382356 secs 2024-07-26T10:09:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11589.121618 secs 2024-07-26T10:09:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:01 smithi042 ceph-mon[32775]: pgmap v6705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11571.382356 secs 2024-07-26T10:09:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11589.121618 secs 2024-07-26T10:09:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:03 smithi164 ceph-mon[36970]: pgmap v6706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:03 smithi163 ceph-mon[36873]: pgmap v6706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:03 smithi042 ceph-mon[32775]: pgmap v6706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:05 smithi164 ceph-mon[36970]: pgmap v6707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:05 smithi163 ceph-mon[36873]: pgmap v6707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:05 smithi042 ceph-mon[32775]: pgmap v6707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11576.382422 secs 2024-07-26T10:09:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11594.121742 secs 2024-07-26T10:09:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11576.382422 secs 2024-07-26T10:09:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11594.121742 secs 2024-07-26T10:09:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11576.382422 secs 2024-07-26T10:09:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11594.121742 secs 2024-07-26T10:09:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:07 smithi164 ceph-mon[36970]: pgmap v6708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:07 smithi163 ceph-mon[36873]: pgmap v6708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:07 smithi042 ceph-mon[32775]: pgmap v6708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:10.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:09 smithi164 ceph-mon[36970]: pgmap v6709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:09 smithi163 ceph-mon[36873]: pgmap v6709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:10.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:09 smithi042 ceph-mon[32775]: pgmap v6709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:11 smithi164 ceph-mon[36970]: pgmap v6710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11581.382508 secs 2024-07-26T10:09:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11599.121881 secs 2024-07-26T10:09:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:11 smithi163 ceph-mon[36873]: pgmap v6710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11581.382508 secs 2024-07-26T10:09:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11599.121881 secs 2024-07-26T10:09:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:11 smithi042 ceph-mon[32775]: pgmap v6710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11581.382508 secs 2024-07-26T10:09:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11599.121881 secs 2024-07-26T10:09:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:09:12.679+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:09:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:09:12.679+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:09:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:09:12.679+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:09:14.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:13 smithi164 ceph-mon[36970]: pgmap v6711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:13 smithi163 ceph-mon[36873]: pgmap v6711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:13 smithi042 ceph-mon[32775]: pgmap v6711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:15 smithi164 ceph-mon[36970]: pgmap v6712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:15 smithi163 ceph-mon[36873]: pgmap v6712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:15 smithi042 ceph-mon[32775]: pgmap v6712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11586.382630 secs 2024-07-26T10:09:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11604.122002 secs 2024-07-26T10:09:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11586.382630 secs 2024-07-26T10:09:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11604.122002 secs 2024-07-26T10:09:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11586.382630 secs 2024-07-26T10:09:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11604.122002 secs 2024-07-26T10:09:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:17 smithi042 ceph-mon[32775]: pgmap v6713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:17 smithi164 ceph-mon[36970]: pgmap v6713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:17 smithi163 ceph-mon[36873]: pgmap v6713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:19 smithi164 ceph-mon[36970]: pgmap v6714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:19 smithi163 ceph-mon[36873]: pgmap v6714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:19 smithi042 ceph-mon[32775]: pgmap v6714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:21 smithi164 ceph-mon[36970]: pgmap v6715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11591.382761 secs 2024-07-26T10:09:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11609.122101 secs 2024-07-26T10:09:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:21 smithi163 ceph-mon[36873]: pgmap v6715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11591.382761 secs 2024-07-26T10:09:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11609.122101 secs 2024-07-26T10:09:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:21 smithi042 ceph-mon[32775]: pgmap v6715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11591.382761 secs 2024-07-26T10:09:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11609.122101 secs 2024-07-26T10:09:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:23 smithi164 ceph-mon[36970]: pgmap v6716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:23 smithi163 ceph-mon[36873]: pgmap v6716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:23 smithi042 ceph-mon[32775]: pgmap v6716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:25 smithi163 ceph-mon[36873]: pgmap v6717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:25 smithi042 ceph-mon[32775]: pgmap v6717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:25 smithi164 ceph-mon[36970]: pgmap v6717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11596.382883 secs 2024-07-26T10:09:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11614.122169 secs 2024-07-26T10:09:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:09:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:09:27.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:09:27.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:09:27.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11596.382883 secs 2024-07-26T10:09:27.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11614.122169 secs 2024-07-26T10:09:27.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:09:27.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.376 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.376 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.376 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.376 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.376 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.376 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.376 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.376 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.376 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.376 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.376 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:09:27.376 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:09:27.376 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:09:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11596.382883 secs 2024-07-26T10:09:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11614.122169 secs 2024-07-26T10:09:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:09:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:09:27.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:09:27.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:09:27.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:09:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:09:27.680+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:09:27.681 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:09:27.680+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:09:27.682 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:09:27.680+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:09:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:27 smithi163 ceph-mon[36873]: pgmap v6718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:28.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:27 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T10:09:28.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:27 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T10:09:28.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:27 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T10:09:28.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:27 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T10:09:28.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:27 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:09:28.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:27 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T10:09:28.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:27 smithi163 ceph-mon[36873]: pgmap v6719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:09:28.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:27 smithi163 ceph-mon[36873]: pgmap v6720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:09:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:27 smithi042 ceph-mon[32775]: pgmap v6718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:27 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T10:09:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:27 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T10:09:28.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:27 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T10:09:28.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:27 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T10:09:28.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:27 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:09:28.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:27 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T10:09:28.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:27 smithi042 ceph-mon[32775]: pgmap v6719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:09:28.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:27 smithi042 ceph-mon[32775]: pgmap v6720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:09:28.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:27 smithi164 ceph-mon[36970]: pgmap v6718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:28.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:27 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T10:09:28.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:27 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T10:09:28.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:27 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T10:09:28.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:27 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T10:09:28.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:27 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:09:28.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:27 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T10:09:28.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:27 smithi164 ceph-mon[36970]: pgmap v6719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:09:28.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:27 smithi164 ceph-mon[36970]: pgmap v6720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:09:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:29 smithi163 ceph-mon[36873]: pgmap v6721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:09:30.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:29 smithi042 ceph-mon[32775]: pgmap v6721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:09:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:29 smithi164 ceph-mon[36970]: pgmap v6721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:09:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:31 smithi163 ceph-mon[36873]: pgmap v6722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:09:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11601.383025 secs 2024-07-26T10:09:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11619.122269 secs 2024-07-26T10:09:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:31 smithi042 ceph-mon[32775]: pgmap v6722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:09:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11601.383025 secs 2024-07-26T10:09:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11619.122269 secs 2024-07-26T10:09:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:31 smithi164 ceph-mon[36970]: pgmap v6722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:09:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11601.383025 secs 2024-07-26T10:09:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11619.122269 secs 2024-07-26T10:09:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:33 smithi163 ceph-mon[36873]: pgmap v6723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:09:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:33 smithi042 ceph-mon[32775]: pgmap v6723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:09:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:33 smithi164 ceph-mon[36970]: pgmap v6723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:09:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:35 smithi163 ceph-mon[36873]: pgmap v6724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:09:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:35 smithi042 ceph-mon[32775]: pgmap v6724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:09:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:35 smithi164 ceph-mon[36970]: pgmap v6724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:09:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11606.383171 secs 2024-07-26T10:09:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11624.122401 secs 2024-07-26T10:09:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11606.383171 secs 2024-07-26T10:09:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11624.122401 secs 2024-07-26T10:09:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11606.383171 secs 2024-07-26T10:09:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11624.122401 secs 2024-07-26T10:09:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:38 smithi163 ceph-mon[36873]: pgmap v6725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:09:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:37 smithi042 ceph-mon[32775]: pgmap v6725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:09:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:38 smithi164 ceph-mon[36970]: pgmap v6725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:09:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:40 smithi163 ceph-mon[36873]: pgmap v6726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:09:40.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:40 smithi042 ceph-mon[32775]: pgmap v6726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:09:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:40 smithi164 ceph-mon[36970]: pgmap v6726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:09:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:42 smithi163 ceph-mon[36873]: pgmap v6727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11611.383316 secs 2024-07-26T10:09:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11629.122522 secs 2024-07-26T10:09:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:42 smithi042 ceph-mon[32775]: pgmap v6727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11611.383316 secs 2024-07-26T10:09:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11629.122522 secs 2024-07-26T10:09:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:42 smithi164 ceph-mon[36970]: pgmap v6727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11611.383316 secs 2024-07-26T10:09:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11629.122522 secs 2024-07-26T10:09:42.684 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:09:42.682+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:09:42.684 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:09:42.682+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:09:42.684 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:09:42.682+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:09:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:44 smithi042 ceph-mon[32775]: pgmap v6728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:44 smithi164 ceph-mon[36970]: pgmap v6728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:44 smithi163 ceph-mon[36873]: pgmap v6728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:46.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:46 smithi042 ceph-mon[32775]: pgmap v6729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:46 smithi164 ceph-mon[36970]: pgmap v6729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:46 smithi163 ceph-mon[36873]: pgmap v6729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11616.383437 secs 2024-07-26T10:09:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11634.122644 secs 2024-07-26T10:09:47.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11616.383437 secs 2024-07-26T10:09:47.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11634.122644 secs 2024-07-26T10:09:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11616.383437 secs 2024-07-26T10:09:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11634.122644 secs 2024-07-26T10:09:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:48 smithi042 ceph-mon[32775]: pgmap v6730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:48 smithi164 ceph-mon[36970]: pgmap v6730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:48 smithi163 ceph-mon[36873]: pgmap v6730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:50 smithi042 ceph-mon[32775]: pgmap v6731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:09:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:50 smithi164 ceph-mon[36970]: pgmap v6731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:09:50.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:50 smithi163 ceph-mon[36873]: pgmap v6731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:09:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:52 smithi042 ceph-mon[32775]: pgmap v6732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11621.383560 secs 2024-07-26T10:09:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11639.122762 secs 2024-07-26T10:09:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:52 smithi164 ceph-mon[36970]: pgmap v6732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11621.383560 secs 2024-07-26T10:09:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11639.122762 secs 2024-07-26T10:09:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:52 smithi163 ceph-mon[36873]: pgmap v6732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11621.383560 secs 2024-07-26T10:09:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11639.122762 secs 2024-07-26T10:09:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:54 smithi042 ceph-mon[32775]: pgmap v6733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:54 smithi164 ceph-mon[36970]: pgmap v6733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:54 smithi163 ceph-mon[36873]: pgmap v6733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:56 smithi042 ceph-mon[32775]: pgmap v6734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:56 smithi164 ceph-mon[36970]: pgmap v6734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:56 smithi163 ceph-mon[36873]: pgmap v6734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:09:57.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11626.383684 secs 2024-07-26T10:09:57.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11644.122905 secs 2024-07-26T10:09:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11626.383684 secs 2024-07-26T10:09:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11644.122905 secs 2024-07-26T10:09:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11626.383684 secs 2024-07-26T10:09:57.516 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11644.122905 secs 2024-07-26T10:09:57.689 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:09:57.687+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:09:57.689 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:09:57.687+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:09:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:09:57.687+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:09:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:09:58 smithi042 ceph-mon[32775]: pgmap v6735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:09:58 smithi164 ceph-mon[36970]: pgmap v6735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:09:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:09:58 smithi163 ceph-mon[36873]: pgmap v6735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:00 smithi042 ceph-mon[32775]: pgmap v6736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:10:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:10:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:00 smithi164 ceph-mon[36970]: pgmap v6736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:10:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:10:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:00 smithi163 ceph-mon[36873]: pgmap v6736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:10:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:10:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:02 smithi042 ceph-mon[32775]: pgmap v6737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11631.383749 secs 2024-07-26T10:10:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11649.123028 secs 2024-07-26T10:10:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:02 smithi164 ceph-mon[36970]: pgmap v6737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:02.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11631.383749 secs 2024-07-26T10:10:02.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11649.123028 secs 2024-07-26T10:10:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:02 smithi163 ceph-mon[36873]: pgmap v6737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11631.383749 secs 2024-07-26T10:10:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11649.123028 secs 2024-07-26T10:10:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:04 smithi042 ceph-mon[32775]: pgmap v6738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:04 smithi164 ceph-mon[36970]: pgmap v6738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:04 smithi163 ceph-mon[36873]: pgmap v6738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:06 smithi042 ceph-mon[32775]: pgmap v6739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:06 smithi164 ceph-mon[36970]: pgmap v6739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:06 smithi163 ceph-mon[36873]: pgmap v6739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11636.383873 secs 2024-07-26T10:10:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11654.123152 secs 2024-07-26T10:10:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11636.383873 secs 2024-07-26T10:10:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11654.123152 secs 2024-07-26T10:10:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11636.383873 secs 2024-07-26T10:10:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11654.123152 secs 2024-07-26T10:10:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:08 smithi042 ceph-mon[32775]: pgmap v6740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:08 smithi164 ceph-mon[36970]: pgmap v6740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:08.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:08 smithi163 ceph-mon[36873]: pgmap v6740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:10.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:10 smithi042 ceph-mon[32775]: pgmap v6741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:10:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:10 smithi164 ceph-mon[36970]: pgmap v6741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:10:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:10 smithi163 ceph-mon[36873]: pgmap v6741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:10:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:12 smithi042 ceph-mon[32775]: pgmap v6742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11641.383996 secs 2024-07-26T10:10:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11659.123276 secs 2024-07-26T10:10:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:12 smithi164 ceph-mon[36970]: pgmap v6742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11641.383996 secs 2024-07-26T10:10:12.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11659.123276 secs 2024-07-26T10:10:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:12 smithi163 ceph-mon[36873]: pgmap v6742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11641.383996 secs 2024-07-26T10:10:12.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11659.123276 secs 2024-07-26T10:10:12.690 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:12.688+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:10:12.690 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:12.688+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:10:12.690 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:12.688+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:10:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:14 smithi042 ceph-mon[32775]: pgmap v6743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:14 smithi164 ceph-mon[36970]: pgmap v6743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:14.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:14 smithi163 ceph-mon[36873]: pgmap v6743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:16 smithi042 ceph-mon[32775]: pgmap v6744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:16 smithi164 ceph-mon[36970]: pgmap v6744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:16 smithi163 ceph-mon[36873]: pgmap v6744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11646.384118 secs 2024-07-26T10:10:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11664.123367 secs 2024-07-26T10:10:17.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11646.384118 secs 2024-07-26T10:10:17.410 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11664.123367 secs 2024-07-26T10:10:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11646.384118 secs 2024-07-26T10:10:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11664.123367 secs 2024-07-26T10:10:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:18 smithi042 ceph-mon[32775]: pgmap v6745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:18 smithi164 ceph-mon[36970]: pgmap v6745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:18 smithi163 ceph-mon[36873]: pgmap v6745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:20 smithi042 ceph-mon[32775]: pgmap v6746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:10:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:20 smithi164 ceph-mon[36970]: pgmap v6746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:10:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:20 smithi163 ceph-mon[36873]: pgmap v6746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:10:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:22 smithi042 ceph-mon[32775]: pgmap v6747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11651.384224 secs 2024-07-26T10:10:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11669.123509 secs 2024-07-26T10:10:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:22 smithi164 ceph-mon[36970]: pgmap v6747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11651.384224 secs 2024-07-26T10:10:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11669.123509 secs 2024-07-26T10:10:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:22 smithi163 ceph-mon[36873]: pgmap v6747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11651.384224 secs 2024-07-26T10:10:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11669.123509 secs 2024-07-26T10:10:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:24 smithi042 ceph-mon[32775]: pgmap v6748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:24 smithi164 ceph-mon[36970]: pgmap v6748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:24 smithi163 ceph-mon[36873]: pgmap v6748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:26 smithi042 ceph-mon[32775]: pgmap v6749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:26 smithi164 ceph-mon[36970]: pgmap v6749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:26 smithi163 ceph-mon[36873]: pgmap v6749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:27.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11656.384264 secs 2024-07-26T10:10:27.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11674.123665 secs 2024-07-26T10:10:27.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:10:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11656.384264 secs 2024-07-26T10:10:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11674.123665 secs 2024-07-26T10:10:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:10:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11656.384264 secs 2024-07-26T10:10:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11674.123665 secs 2024-07-26T10:10:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:10:27.696 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:27.694+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:10:27.696 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:27.694+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:10:27.696 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:27.694+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:10:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:28 smithi042 ceph-mon[32775]: pgmap v6750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:10:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:10:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:10:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:28 smithi164 ceph-mon[36970]: pgmap v6750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:10:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:10:28.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:10:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:28 smithi163 ceph-mon[36873]: pgmap v6750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:10:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:10:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:10:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:29 smithi042 ceph-mon[32775]: pgmap v6751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:10:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:29 smithi042 ceph-mon[32775]: pgmap v6752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:10:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:29 smithi164 ceph-mon[36970]: pgmap v6751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:10:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:29 smithi164 ceph-mon[36970]: pgmap v6752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:10:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:29 smithi163 ceph-mon[36873]: pgmap v6751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:10:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:29 smithi163 ceph-mon[36873]: pgmap v6752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:10:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:31 smithi042 ceph-mon[32775]: pgmap v6753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:10:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:31 smithi164 ceph-mon[36970]: pgmap v6753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:10:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:31 smithi163 ceph-mon[36873]: pgmap v6753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:10:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11661.384384 secs 2024-07-26T10:10:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11679.123812 secs 2024-07-26T10:10:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11661.384384 secs 2024-07-26T10:10:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11679.123812 secs 2024-07-26T10:10:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11661.384384 secs 2024-07-26T10:10:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11679.123812 secs 2024-07-26T10:10:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:33 smithi042 ceph-mon[32775]: pgmap v6754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:10:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:33 smithi164 ceph-mon[36970]: pgmap v6754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:10:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:33 smithi163 ceph-mon[36873]: pgmap v6754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:10:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:35 smithi042 ceph-mon[32775]: pgmap v6755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:10:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:35 smithi164 ceph-mon[36970]: pgmap v6755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:10:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:35 smithi163 ceph-mon[36873]: pgmap v6755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:10:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:37 smithi042 ceph-mon[32775]: pgmap v6756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:10:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11666.384470 secs 2024-07-26T10:10:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11684.123913 secs 2024-07-26T10:10:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:37 smithi164 ceph-mon[36970]: pgmap v6756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:10:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11666.384470 secs 2024-07-26T10:10:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11684.123913 secs 2024-07-26T10:10:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:37 smithi163 ceph-mon[36873]: pgmap v6756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:10:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11666.384470 secs 2024-07-26T10:10:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11684.123913 secs 2024-07-26T10:10:39.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:39 smithi042 ceph-mon[32775]: pgmap v6757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:10:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:39 smithi164 ceph-mon[36970]: pgmap v6757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:10:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:39 smithi163 ceph-mon[36873]: pgmap v6757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:10:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:40 smithi164 ceph-mon[36970]: pgmap v6758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:40 smithi163 ceph-mon[36873]: pgmap v6758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:40 smithi042 ceph-mon[32775]: pgmap v6758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11671.384553 secs 2024-07-26T10:10:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11689.124004 secs 2024-07-26T10:10:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:42 smithi042 ceph-mon[32775]: pgmap v6759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:42.695+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:10:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:42.695+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:10:42.697 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:42.695+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:10:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11671.384553 secs 2024-07-26T10:10:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11689.124004 secs 2024-07-26T10:10:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:42 smithi164 ceph-mon[36970]: pgmap v6759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11671.384553 secs 2024-07-26T10:10:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11689.124004 secs 2024-07-26T10:10:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:42 smithi163 ceph-mon[36873]: pgmap v6759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:44 smithi164 ceph-mon[36970]: pgmap v6760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:10:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:44 smithi163 ceph-mon[36873]: pgmap v6760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:10:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:44 smithi042 ceph-mon[32775]: pgmap v6760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:10:46.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:46 smithi164 ceph-mon[36970]: pgmap v6761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:46 smithi163 ceph-mon[36873]: pgmap v6761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:46 smithi042 ceph-mon[32775]: pgmap v6761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:47.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11676.384629 secs 2024-07-26T10:10:47.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11694.124128 secs 2024-07-26T10:10:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11676.384629 secs 2024-07-26T10:10:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11694.124128 secs 2024-07-26T10:10:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11676.384629 secs 2024-07-26T10:10:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11694.124128 secs 2024-07-26T10:10:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:48 smithi164 ceph-mon[36970]: pgmap v6762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:48 smithi163 ceph-mon[36873]: pgmap v6762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:48 smithi042 ceph-mon[32775]: pgmap v6762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:50 smithi164 ceph-mon[36970]: pgmap v6763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:50 smithi163 ceph-mon[36873]: pgmap v6763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:50 smithi042 ceph-mon[32775]: pgmap v6763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11681.384749 secs 2024-07-26T10:10:52.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11699.124205 secs 2024-07-26T10:10:52.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:52 smithi164 ceph-mon[36970]: pgmap v6764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11681.384749 secs 2024-07-26T10:10:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11699.124205 secs 2024-07-26T10:10:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:52 smithi163 ceph-mon[36873]: pgmap v6764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11681.384749 secs 2024-07-26T10:10:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11699.124205 secs 2024-07-26T10:10:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:52 smithi042 ceph-mon[32775]: pgmap v6764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:54.700 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:54 smithi042 ceph-mon[32775]: pgmap v6765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:10:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:54 smithi164 ceph-mon[36970]: pgmap v6765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:10:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:54 smithi163 ceph-mon[36873]: pgmap v6765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:10:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:56 smithi164 ceph-mon[36970]: pgmap v6766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:56 smithi163 ceph-mon[36873]: pgmap v6766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:56 smithi042 ceph-mon[32775]: pgmap v6766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:10:57.703 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:57.701+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:10:57.703 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:57.701+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:10:57.703 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:57.701+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:10:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11686.384831 secs 2024-07-26T10:10:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11704.124326 secs 2024-07-26T10:10:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11686.384831 secs 2024-07-26T10:10:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11704.124326 secs 2024-07-26T10:10:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:57.878+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T10:10:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:57.878+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T10:10:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:57.878+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff678009a60 tx=0x7ff67803e070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:10:57.880 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:57.878+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff678009a60 tx=0x7ff67803e070 comp rx=0 tx=0).stop 2024-07-26T10:10:57.881 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:57.878+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5210 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T10:10:57.881 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:57.878+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T10:10:57.881 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:57.878+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:10:57.881 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:57.878+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:10:57.881 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:57.878+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:10:57.881 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:57.879+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff678009a60 tx=0x7ff67803e040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:10:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11686.384831 secs 2024-07-26T10:10:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11704.124326 secs 2024-07-26T10:10:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:10:58 smithi164 ceph-mon[36970]: pgmap v6767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:10:58 smithi163 ceph-mon[36873]: pgmap v6767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:10:58 smithi042 ceph-mon[32775]: pgmap v6767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:10:59.690 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:59.688+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:10:59.690 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:59.688+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:10:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:59.688+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:10:59.691 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:10:59.689+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:11:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:00 smithi164 ceph-mon[36970]: pgmap v6768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:00 smithi163 ceph-mon[36873]: pgmap v6768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:00 smithi042 ceph-mon[32775]: pgmap v6768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11691.384972 secs 2024-07-26T10:11:02.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11709.124409 secs 2024-07-26T10:11:02.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:02 smithi164 ceph-mon[36970]: pgmap v6769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11691.384972 secs 2024-07-26T10:11:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11709.124409 secs 2024-07-26T10:11:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:02 smithi163 ceph-mon[36873]: pgmap v6769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11691.384972 secs 2024-07-26T10:11:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11709.124409 secs 2024-07-26T10:11:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:02 smithi042 ceph-mon[32775]: pgmap v6769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:04 smithi164 ceph-mon[36970]: pgmap v6770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:11:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:04 smithi163 ceph-mon[36873]: pgmap v6770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:11:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:04 smithi042 ceph-mon[32775]: pgmap v6770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:11:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:06 smithi164 ceph-mon[36970]: pgmap v6771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:06 smithi163 ceph-mon[36873]: pgmap v6771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:06.865 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:06 smithi042 ceph-mon[32775]: pgmap v6771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11696.385107 secs 2024-07-26T10:11:07.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11714.124475 secs 2024-07-26T10:11:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11696.385107 secs 2024-07-26T10:11:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11714.124475 secs 2024-07-26T10:11:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11696.385107 secs 2024-07-26T10:11:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11714.124475 secs 2024-07-26T10:11:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:08 smithi164 ceph-mon[36970]: pgmap v6772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:08 smithi163 ceph-mon[36873]: pgmap v6772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:08 smithi042 ceph-mon[32775]: pgmap v6772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:10.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:10 smithi164 ceph-mon[36970]: pgmap v6773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:10 smithi163 ceph-mon[36873]: pgmap v6773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:10 smithi042 ceph-mon[32775]: pgmap v6773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11701.385251 secs 2024-07-26T10:11:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11701.385251 secs 2024-07-26T10:11:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11701.385251 secs 2024-07-26T10:11:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:11:12.702+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:11:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:11:12.702+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:11:12.704 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:11:12.702+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:11:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11719.124581 secs 2024-07-26T10:11:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:12 smithi164 ceph-mon[36970]: pgmap v6774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11719.124581 secs 2024-07-26T10:11:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:12 smithi163 ceph-mon[36873]: pgmap v6774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:12.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11719.124581 secs 2024-07-26T10:11:12.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:12 smithi042 ceph-mon[32775]: pgmap v6774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:14 smithi164 ceph-mon[36970]: pgmap v6775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:11:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:14 smithi163 ceph-mon[36873]: pgmap v6775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:11:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:14 smithi042 ceph-mon[32775]: pgmap v6775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:11:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:16 smithi164 ceph-mon[36970]: pgmap v6776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11706.385373 secs 2024-07-26T10:11:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:16 smithi163 ceph-mon[36873]: pgmap v6776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11706.385373 secs 2024-07-26T10:11:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:16 smithi042 ceph-mon[32775]: pgmap v6776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11706.385373 secs 2024-07-26T10:11:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11724.124727 secs 2024-07-26T10:11:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11724.124727 secs 2024-07-26T10:11:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11724.124727 secs 2024-07-26T10:11:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:18 smithi163 ceph-mon[36873]: pgmap v6777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:18 smithi042 ceph-mon[32775]: pgmap v6777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:18 smithi164 ceph-mon[36970]: pgmap v6777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:20 smithi163 ceph-mon[36873]: pgmap v6778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:20 smithi042 ceph-mon[32775]: pgmap v6778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:20 smithi164 ceph-mon[36970]: pgmap v6778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11711.385493 secs 2024-07-26T10:11:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11711.385493 secs 2024-07-26T10:11:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11711.385493 secs 2024-07-26T10:11:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11729.124858 secs 2024-07-26T10:11:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:22 smithi163 ceph-mon[36873]: pgmap v6779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11729.124858 secs 2024-07-26T10:11:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:22 smithi042 ceph-mon[32775]: pgmap v6779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11729.124858 secs 2024-07-26T10:11:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:22 smithi164 ceph-mon[36970]: pgmap v6779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:24 smithi163 ceph-mon[36873]: pgmap v6780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:11:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:24 smithi042 ceph-mon[32775]: pgmap v6780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:11:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:24 smithi164 ceph-mon[36970]: pgmap v6780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:11:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:26 smithi163 ceph-mon[36873]: pgmap v6781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11716.385623 secs 2024-07-26T10:11:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:26 smithi042 ceph-mon[32775]: pgmap v6781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11716.385623 secs 2024-07-26T10:11:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:26 smithi164 ceph-mon[36970]: pgmap v6781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11716.385623 secs 2024-07-26T10:11:27.708 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:11:27.705+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:11:27.708 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:11:27.705+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:11:27.708 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:11:27.705+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:11:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11734.125010 secs 2024-07-26T10:11:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:11:27.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11734.125010 secs 2024-07-26T10:11:27.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:11:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11734.125010 secs 2024-07-26T10:11:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:11:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:28 smithi042 ceph-mon[32775]: pgmap v6782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:11:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:11:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:28 smithi042 ceph-mon[32775]: pgmap v6783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:11:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:28 smithi042 ceph-mon[32775]: pgmap v6784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:11:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:11:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:28 smithi164 ceph-mon[36970]: pgmap v6782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:28.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:11:28.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:11:28.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:28 smithi164 ceph-mon[36970]: pgmap v6783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:11:28.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:28 smithi164 ceph-mon[36970]: pgmap v6784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:11:28.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:11:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:28 smithi163 ceph-mon[36873]: pgmap v6782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:11:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:11:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:28 smithi163 ceph-mon[36873]: pgmap v6783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:11:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:28 smithi163 ceph-mon[36873]: pgmap v6784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:11:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:11:31.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:30 smithi164 ceph-mon[36970]: pgmap v6785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:11:31.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:30 smithi163 ceph-mon[36873]: pgmap v6785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:11:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:30 smithi042 ceph-mon[32775]: pgmap v6785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:11:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11721.385685 secs 2024-07-26T10:11:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11739.125161 secs 2024-07-26T10:11:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11721.385685 secs 2024-07-26T10:11:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11739.125161 secs 2024-07-26T10:11:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11721.385685 secs 2024-07-26T10:11:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11739.125161 secs 2024-07-26T10:11:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:32 smithi164 ceph-mon[36970]: pgmap v6786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:11:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:32 smithi163 ceph-mon[36873]: pgmap v6786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:11:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:32 smithi042 ceph-mon[32775]: pgmap v6786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:11:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:34 smithi164 ceph-mon[36970]: pgmap v6787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:11:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:34 smithi163 ceph-mon[36873]: pgmap v6787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:11:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:34 smithi042 ceph-mon[32775]: pgmap v6787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:11:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:36 smithi164 ceph-mon[36970]: pgmap v6788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:11:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11726.385830 secs 2024-07-26T10:11:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11744.125304 secs 2024-07-26T10:11:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:36 smithi163 ceph-mon[36873]: pgmap v6788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:11:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11726.385830 secs 2024-07-26T10:11:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11744.125304 secs 2024-07-26T10:11:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:36 smithi042 ceph-mon[32775]: pgmap v6788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:11:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11726.385830 secs 2024-07-26T10:11:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11744.125304 secs 2024-07-26T10:11:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:38 smithi164 ceph-mon[36970]: pgmap v6789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:11:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:38 smithi163 ceph-mon[36873]: pgmap v6789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:11:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:38 smithi042 ceph-mon[32775]: pgmap v6789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:11:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:40 smithi164 ceph-mon[36970]: pgmap v6790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:41.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:40 smithi163 ceph-mon[36873]: pgmap v6790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:40 smithi042 ceph-mon[32775]: pgmap v6790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11731.385953 secs 2024-07-26T10:11:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11749.125366 secs 2024-07-26T10:11:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11731.385953 secs 2024-07-26T10:11:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11749.125366 secs 2024-07-26T10:11:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11731.385953 secs 2024-07-26T10:11:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11749.125366 secs 2024-07-26T10:11:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:11:42.707+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:11:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:11:42.707+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:11:42.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:11:42.707+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:11:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:42 smithi164 ceph-mon[36970]: pgmap v6791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:42 smithi163 ceph-mon[36873]: pgmap v6791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:42 smithi042 ceph-mon[32775]: pgmap v6791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:44 smithi163 ceph-mon[36873]: pgmap v6792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:11:45.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:44 smithi042 ceph-mon[32775]: pgmap v6792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:11:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:44 smithi164 ceph-mon[36970]: pgmap v6792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:11:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:46 smithi163 ceph-mon[36873]: pgmap v6793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11736.386050 secs 2024-07-26T10:11:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11754.125511 secs 2024-07-26T10:11:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:46 smithi042 ceph-mon[32775]: pgmap v6793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11736.386050 secs 2024-07-26T10:11:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11754.125511 secs 2024-07-26T10:11:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:46 smithi164 ceph-mon[36970]: pgmap v6793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11736.386050 secs 2024-07-26T10:11:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11754.125511 secs 2024-07-26T10:11:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:48 smithi163 ceph-mon[36873]: pgmap v6794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:48 smithi042 ceph-mon[32775]: pgmap v6794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:48 smithi164 ceph-mon[36970]: pgmap v6794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:50 smithi163 ceph-mon[36873]: pgmap v6795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:50 smithi042 ceph-mon[32775]: pgmap v6795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:50 smithi164 ceph-mon[36970]: pgmap v6795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11741.386174 secs 2024-07-26T10:11:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11759.125634 secs 2024-07-26T10:11:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11741.386174 secs 2024-07-26T10:11:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11759.125634 secs 2024-07-26T10:11:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11741.386174 secs 2024-07-26T10:11:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11759.125634 secs 2024-07-26T10:11:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:52 smithi163 ceph-mon[36873]: pgmap v6796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:52 smithi042 ceph-mon[32775]: pgmap v6796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:52 smithi164 ceph-mon[36970]: pgmap v6796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:54 smithi163 ceph-mon[36873]: pgmap v6797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:11:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:54 smithi042 ceph-mon[32775]: pgmap v6797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:11:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:54 smithi164 ceph-mon[36970]: pgmap v6797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:11:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:57 smithi163 ceph-mon[36873]: pgmap v6798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11746.386292 secs 2024-07-26T10:11:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11764.125758 secs 2024-07-26T10:11:57.276 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:57 smithi164 ceph-mon[36970]: pgmap v6798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:57.276 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11746.386292 secs 2024-07-26T10:11:57.276 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11764.125758 secs 2024-07-26T10:11:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:57 smithi042 ceph-mon[32775]: pgmap v6798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:11:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11746.386292 secs 2024-07-26T10:11:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11764.125758 secs 2024-07-26T10:11:57.714 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:11:57.712+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:11:57.714 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:11:57.712+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:11:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:11:57.712+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:11:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:11:59 smithi163 ceph-mon[36873]: pgmap v6799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:11:59 smithi042 ceph-mon[32775]: pgmap v6799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:11:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:11:59 smithi164 ceph-mon[36970]: pgmap v6799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:01 smithi163 ceph-mon[36873]: pgmap v6800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:01 smithi042 ceph-mon[32775]: pgmap v6800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:01 smithi164 ceph-mon[36970]: pgmap v6800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11751.386434 secs 2024-07-26T10:12:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11769.125839 secs 2024-07-26T10:12:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11751.386434 secs 2024-07-26T10:12:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11769.125839 secs 2024-07-26T10:12:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11751.386434 secs 2024-07-26T10:12:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11769.125839 secs 2024-07-26T10:12:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:03 smithi042 ceph-mon[32775]: pgmap v6801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:12:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:03 smithi164 ceph-mon[36970]: pgmap v6801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:12:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:03 smithi163 ceph-mon[36873]: pgmap v6801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:12:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:05 smithi042 ceph-mon[32775]: pgmap v6802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:12:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:05 smithi164 ceph-mon[36970]: pgmap v6802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:12:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:05 smithi163 ceph-mon[36873]: pgmap v6802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:12:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:07 smithi042 ceph-mon[32775]: pgmap v6803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:12:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11756.386549 secs 2024-07-26T10:12:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11774.125999 secs 2024-07-26T10:12:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:07 smithi164 ceph-mon[36970]: pgmap v6803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:12:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11756.386549 secs 2024-07-26T10:12:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11774.125999 secs 2024-07-26T10:12:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:07 smithi163 ceph-mon[36873]: pgmap v6803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:12:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11756.386549 secs 2024-07-26T10:12:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11774.125999 secs 2024-07-26T10:12:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:09 smithi042 ceph-mon[32775]: pgmap v6804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:12:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:09 smithi164 ceph-mon[36970]: pgmap v6804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:12:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:09 smithi163 ceph-mon[36873]: pgmap v6804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:12:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:11 smithi042 ceph-mon[32775]: pgmap v6805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:11 smithi164 ceph-mon[36970]: pgmap v6805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:11 smithi163 ceph-mon[36873]: pgmap v6805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11761.386673 secs 2024-07-26T10:12:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11779.126104 secs 2024-07-26T10:12:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11761.386673 secs 2024-07-26T10:12:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11779.126104 secs 2024-07-26T10:12:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11761.386673 secs 2024-07-26T10:12:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11779.126104 secs 2024-07-26T10:12:12.714 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:12:12.713+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:12:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:12:12.713+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:12:12.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:12:12.713+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:12:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:13 smithi042 ceph-mon[32775]: pgmap v6806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:13 smithi164 ceph-mon[36970]: pgmap v6806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:13 smithi163 ceph-mon[36873]: pgmap v6806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:15 smithi042 ceph-mon[32775]: pgmap v6807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:12:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:15 smithi164 ceph-mon[36970]: pgmap v6807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:12:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:15 smithi163 ceph-mon[36873]: pgmap v6807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:12:17.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:17 smithi163 ceph-mon[36873]: pgmap v6808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:17.311 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11766.386788 secs 2024-07-26T10:12:17.311 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11784.126204 secs 2024-07-26T10:12:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:17 smithi042 ceph-mon[32775]: pgmap v6808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11766.386788 secs 2024-07-26T10:12:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11784.126204 secs 2024-07-26T10:12:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:17 smithi164 ceph-mon[36970]: pgmap v6808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11766.386788 secs 2024-07-26T10:12:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11784.126204 secs 2024-07-26T10:12:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:19 smithi042 ceph-mon[32775]: pgmap v6809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:19 smithi164 ceph-mon[36970]: pgmap v6809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:19 smithi163 ceph-mon[36873]: pgmap v6809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:21 smithi042 ceph-mon[32775]: pgmap v6810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:21 smithi164 ceph-mon[36970]: pgmap v6810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:21 smithi163 ceph-mon[36873]: pgmap v6810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11771.386905 secs 2024-07-26T10:12:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11789.126328 secs 2024-07-26T10:12:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11771.386905 secs 2024-07-26T10:12:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11789.126328 secs 2024-07-26T10:12:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11771.386905 secs 2024-07-26T10:12:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11789.126328 secs 2024-07-26T10:12:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:23 smithi042 ceph-mon[32775]: pgmap v6811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:23 smithi164 ceph-mon[36970]: pgmap v6811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:23 smithi163 ceph-mon[36873]: pgmap v6811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:25 smithi042 ceph-mon[32775]: pgmap v6812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:12:25.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:25 smithi164 ceph-mon[36970]: pgmap v6812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:12:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:25 smithi163 ceph-mon[36873]: pgmap v6812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:12:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:27 smithi042 ceph-mon[32775]: pgmap v6813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11776.387027 secs 2024-07-26T10:12:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11794.126421 secs 2024-07-26T10:12:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:27 smithi164 ceph-mon[36970]: pgmap v6813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11776.387027 secs 2024-07-26T10:12:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11794.126421 secs 2024-07-26T10:12:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:27 smithi163 ceph-mon[36873]: pgmap v6813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11776.387027 secs 2024-07-26T10:12:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11794.126421 secs 2024-07-26T10:12:27.720 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:12:27.718+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:12:27.720 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:12:27.718+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:12:27.720 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:12:27.718+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:12:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:12:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:12:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:12:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:29 smithi042 ceph-mon[32775]: pgmap v6814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:29 smithi164 ceph-mon[36970]: pgmap v6814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:29 smithi163 ceph-mon[36873]: pgmap v6814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:30 smithi042 ceph-mon[32775]: pgmap v6815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:30.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:30.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:30.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:30.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:12:30.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:12:30.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:30 smithi164 ceph-mon[36970]: pgmap v6815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:12:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:30 smithi163 ceph-mon[36873]: pgmap v6815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:31.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:31.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:31.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:31.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:12:31.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:12:31.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:12:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:31 smithi164 ceph-mon[36970]: pgmap v6816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:12:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:31 smithi164 ceph-mon[36970]: pgmap v6817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11781.387178 secs 2024-07-26T10:12:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11799.126548 secs 2024-07-26T10:12:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:31 smithi163 ceph-mon[36873]: pgmap v6816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:12:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:31 smithi163 ceph-mon[36873]: pgmap v6817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11781.387178 secs 2024-07-26T10:12:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11799.126548 secs 2024-07-26T10:12:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:31 smithi042 ceph-mon[32775]: pgmap v6816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:12:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:31 smithi042 ceph-mon[32775]: pgmap v6817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11781.387178 secs 2024-07-26T10:12:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11799.126548 secs 2024-07-26T10:12:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:33 smithi164 ceph-mon[36970]: pgmap v6818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:33 smithi163 ceph-mon[36873]: pgmap v6818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:33 smithi042 ceph-mon[32775]: pgmap v6818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:35 smithi164 ceph-mon[36970]: pgmap v6819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:35 smithi163 ceph-mon[36873]: pgmap v6819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:35 smithi042 ceph-mon[32775]: pgmap v6819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11786.387291 secs 2024-07-26T10:12:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11804.126666 secs 2024-07-26T10:12:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11786.387291 secs 2024-07-26T10:12:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11804.126666 secs 2024-07-26T10:12:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11786.387291 secs 2024-07-26T10:12:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11804.126666 secs 2024-07-26T10:12:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:37 smithi164 ceph-mon[36970]: pgmap v6820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:37 smithi163 ceph-mon[36873]: pgmap v6820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:37 smithi042 ceph-mon[32775]: pgmap v6820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:12:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:39 smithi164 ceph-mon[36970]: pgmap v6821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:12:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:39 smithi163 ceph-mon[36873]: pgmap v6821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:12:40.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:39 smithi042 ceph-mon[32775]: pgmap v6821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:12:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:41 smithi164 ceph-mon[36970]: pgmap v6822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:12:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11791.387390 secs 2024-07-26T10:12:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11809.126789 secs 2024-07-26T10:12:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:41 smithi163 ceph-mon[36873]: pgmap v6822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:12:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11791.387390 secs 2024-07-26T10:12:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11809.126789 secs 2024-07-26T10:12:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:41 smithi042 ceph-mon[32775]: pgmap v6822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:12:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11791.387390 secs 2024-07-26T10:12:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11809.126789 secs 2024-07-26T10:12:42.722 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:12:42.720+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:12:42.722 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:12:42.720+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:12:42.722 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:12:42.720+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:12:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:43 smithi164 ceph-mon[36970]: pgmap v6823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:43 smithi163 ceph-mon[36873]: pgmap v6823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:43 smithi042 ceph-mon[32775]: pgmap v6823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:45 smithi164 ceph-mon[36970]: pgmap v6824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:45 smithi163 ceph-mon[36873]: pgmap v6824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:45 smithi042 ceph-mon[32775]: pgmap v6824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11796.387519 secs 2024-07-26T10:12:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11814.126919 secs 2024-07-26T10:12:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11796.387519 secs 2024-07-26T10:12:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11814.126919 secs 2024-07-26T10:12:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11796.387519 secs 2024-07-26T10:12:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11814.126919 secs 2024-07-26T10:12:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:47 smithi164 ceph-mon[36970]: pgmap v6825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:47 smithi163 ceph-mon[36873]: pgmap v6825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:47 smithi042 ceph-mon[32775]: pgmap v6825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:49 smithi164 ceph-mon[36970]: pgmap v6826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:12:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:49 smithi163 ceph-mon[36873]: pgmap v6826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:12:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:49 smithi042 ceph-mon[32775]: pgmap v6826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:12:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:51 smithi164 ceph-mon[36970]: pgmap v6827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11801.387635 secs 2024-07-26T10:12:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11819.127011 secs 2024-07-26T10:12:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:51 smithi163 ceph-mon[36873]: pgmap v6827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11801.387635 secs 2024-07-26T10:12:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11819.127011 secs 2024-07-26T10:12:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:51 smithi042 ceph-mon[32775]: pgmap v6827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11801.387635 secs 2024-07-26T10:12:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11819.127011 secs 2024-07-26T10:12:54.222 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:53 smithi042 ceph-mon[32775]: pgmap v6828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:53 smithi164 ceph-mon[36970]: pgmap v6828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:53 smithi163 ceph-mon[36873]: pgmap v6828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:55 smithi164 ceph-mon[36970]: pgmap v6829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:55 smithi163 ceph-mon[36873]: pgmap v6829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:55 smithi042 ceph-mon[32775]: pgmap v6829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:12:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11806.387757 secs 2024-07-26T10:12:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11824.127134 secs 2024-07-26T10:12:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11806.387757 secs 2024-07-26T10:12:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11824.127134 secs 2024-07-26T10:12:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11806.387757 secs 2024-07-26T10:12:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11824.127134 secs 2024-07-26T10:12:57.726 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:12:57.724+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:12:57.726 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:12:57.724+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:12:57.726 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:12:57.724+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:12:58.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:57 smithi164 ceph-mon[36970]: pgmap v6830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:57 smithi163 ceph-mon[36873]: pgmap v6830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:12:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:57 smithi042 ceph-mon[32775]: pgmap v6830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:12:59 smithi164 ceph-mon[36970]: pgmap v6831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:12:59 smithi163 ceph-mon[36873]: pgmap v6831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:12:59 smithi042 ceph-mon[32775]: pgmap v6831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:01 smithi164 ceph-mon[36970]: pgmap v6832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11811.387878 secs 2024-07-26T10:13:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11829.127232 secs 2024-07-26T10:13:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:01 smithi163 ceph-mon[36873]: pgmap v6832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11811.387878 secs 2024-07-26T10:13:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11829.127232 secs 2024-07-26T10:13:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:01 smithi042 ceph-mon[32775]: pgmap v6832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11811.387878 secs 2024-07-26T10:13:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11829.127232 secs 2024-07-26T10:13:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:03 smithi164 ceph-mon[36970]: pgmap v6833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:03 smithi163 ceph-mon[36873]: pgmap v6833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:03 smithi042 ceph-mon[32775]: pgmap v6833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:05 smithi163 ceph-mon[36873]: pgmap v6834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:05 smithi042 ceph-mon[32775]: pgmap v6834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:05 smithi164 ceph-mon[36970]: pgmap v6834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11816.388002 secs 2024-07-26T10:13:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11834.127331 secs 2024-07-26T10:13:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11816.388002 secs 2024-07-26T10:13:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11834.127331 secs 2024-07-26T10:13:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11816.388002 secs 2024-07-26T10:13:07.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11834.127331 secs 2024-07-26T10:13:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:07 smithi164 ceph-mon[36970]: pgmap v6835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:07 smithi163 ceph-mon[36873]: pgmap v6835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:07 smithi042 ceph-mon[32775]: pgmap v6835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:09 smithi163 ceph-mon[36873]: pgmap v6836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:10.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:09 smithi042 ceph-mon[32775]: pgmap v6836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:09 smithi164 ceph-mon[36970]: pgmap v6836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:11 smithi163 ceph-mon[36873]: pgmap v6837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11821.388147 secs 2024-07-26T10:13:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11839.127453 secs 2024-07-26T10:13:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:11 smithi042 ceph-mon[32775]: pgmap v6837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11821.388147 secs 2024-07-26T10:13:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11839.127453 secs 2024-07-26T10:13:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:11 smithi164 ceph-mon[36970]: pgmap v6837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11821.388147 secs 2024-07-26T10:13:12.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11839.127453 secs 2024-07-26T10:13:12.728 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:12.726+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:13:12.728 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:12.726+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:13:12.728 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:12.726+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:13:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:13 smithi163 ceph-mon[36873]: pgmap v6838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:13 smithi042 ceph-mon[32775]: pgmap v6838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:13 smithi164 ceph-mon[36970]: pgmap v6838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:15 smithi163 ceph-mon[36873]: pgmap v6839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:15 smithi042 ceph-mon[32775]: pgmap v6839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:15 smithi164 ceph-mon[36970]: pgmap v6839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11826.388262 secs 2024-07-26T10:13:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11844.127576 secs 2024-07-26T10:13:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11826.388262 secs 2024-07-26T10:13:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11844.127576 secs 2024-07-26T10:13:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11826.388262 secs 2024-07-26T10:13:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11844.127576 secs 2024-07-26T10:13:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:18 smithi163 ceph-mon[36873]: pgmap v6840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:18 smithi042 ceph-mon[32775]: pgmap v6840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:18 smithi164 ceph-mon[36970]: pgmap v6840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:20 smithi042 ceph-mon[32775]: pgmap v6841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:20 smithi164 ceph-mon[36970]: pgmap v6841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:20 smithi163 ceph-mon[36873]: pgmap v6841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:22 smithi042 ceph-mon[32775]: pgmap v6842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11831.388381 secs 2024-07-26T10:13:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11849.127698 secs 2024-07-26T10:13:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:22 smithi164 ceph-mon[36970]: pgmap v6842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11831.388381 secs 2024-07-26T10:13:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11849.127698 secs 2024-07-26T10:13:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:22 smithi163 ceph-mon[36873]: pgmap v6842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11831.388381 secs 2024-07-26T10:13:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11849.127698 secs 2024-07-26T10:13:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:24 smithi042 ceph-mon[32775]: pgmap v6843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:24 smithi164 ceph-mon[36970]: pgmap v6843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:24 smithi163 ceph-mon[36873]: pgmap v6843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:26 smithi042 ceph-mon[32775]: pgmap v6844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:26 smithi164 ceph-mon[36970]: pgmap v6844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:26 smithi163 ceph-mon[36873]: pgmap v6844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:27.298 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11836.388501 secs 2024-07-26T10:13:27.298 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11854.127817 secs 2024-07-26T10:13:27.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11836.388501 secs 2024-07-26T10:13:27.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11854.127817 secs 2024-07-26T10:13:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11836.388501 secs 2024-07-26T10:13:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11854.127817 secs 2024-07-26T10:13:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:27.731+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:13:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:27.731+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:13:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:27.731+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:13:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:28 smithi042 ceph-mon[32775]: pgmap v6845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:28 smithi164 ceph-mon[36970]: pgmap v6845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:28 smithi163 ceph-mon[36873]: pgmap v6845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:30.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:30 smithi042 ceph-mon[32775]: pgmap v6846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:30 smithi164 ceph-mon[36970]: pgmap v6846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:30.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:30 smithi163 ceph-mon[36873]: pgmap v6846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:13:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:13:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:31.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:13:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:32 smithi042 ceph-mon[32775]: pgmap v6847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11841.388573 secs 2024-07-26T10:13:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11859.127886 secs 2024-07-26T10:13:32.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:13:32.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:13:32.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:32 smithi164 ceph-mon[36970]: pgmap v6847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11841.388573 secs 2024-07-26T10:13:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11859.127886 secs 2024-07-26T10:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:32 smithi163 ceph-mon[36873]: pgmap v6847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11841.388573 secs 2024-07-26T10:13:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11859.127886 secs 2024-07-26T10:13:32.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:32.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:13:32.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:13:32.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:13:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:33 smithi042 ceph-mon[32775]: pgmap v6848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:13:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:33 smithi042 ceph-mon[32775]: pgmap v6849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T10:13:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:33 smithi164 ceph-mon[36970]: pgmap v6848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:13:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:33 smithi164 ceph-mon[36970]: pgmap v6849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T10:13:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:33 smithi163 ceph-mon[36873]: pgmap v6848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:13:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:33 smithi163 ceph-mon[36873]: pgmap v6849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T10:13:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:35 smithi042 ceph-mon[32775]: pgmap v6850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T10:13:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:35 smithi164 ceph-mon[36970]: pgmap v6850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T10:13:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:35 smithi163 ceph-mon[36873]: pgmap v6850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T10:13:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:37 smithi042 ceph-mon[32775]: pgmap v6851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T10:13:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11846.388660 secs 2024-07-26T10:13:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11864.128009 secs 2024-07-26T10:13:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:37 smithi164 ceph-mon[36970]: pgmap v6851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T10:13:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11846.388660 secs 2024-07-26T10:13:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11864.128009 secs 2024-07-26T10:13:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:37 smithi163 ceph-mon[36873]: pgmap v6851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T10:13:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11846.388660 secs 2024-07-26T10:13:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11864.128009 secs 2024-07-26T10:13:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:39 smithi042 ceph-mon[32775]: pgmap v6852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T10:13:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:39 smithi164 ceph-mon[36970]: pgmap v6852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T10:13:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:39 smithi163 ceph-mon[36873]: pgmap v6852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T10:13:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:41 smithi042 ceph-mon[32775]: pgmap v6853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T10:13:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:41 smithi164 ceph-mon[36970]: pgmap v6853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T10:13:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:41 smithi163 ceph-mon[36873]: pgmap v6853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T10:13:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11851.388810 secs 2024-07-26T10:13:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11869.128132 secs 2024-07-26T10:13:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11851.388810 secs 2024-07-26T10:13:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11869.128132 secs 2024-07-26T10:13:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11851.388810 secs 2024-07-26T10:13:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11869.128132 secs 2024-07-26T10:13:42.734 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:42.732+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:13:42.734 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:42.732+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:13:42.734 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:42.732+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:13:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:43 smithi042 ceph-mon[32775]: pgmap v6854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:13:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:43 smithi164 ceph-mon[36970]: pgmap v6854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:13:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:43 smithi163 ceph-mon[36873]: pgmap v6854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:13:45.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:45 smithi042 ceph-mon[32775]: pgmap v6855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:45 smithi164 ceph-mon[36970]: pgmap v6855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:45 smithi163 ceph-mon[36873]: pgmap v6855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:47.326 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:47 smithi163 ceph-mon[36873]: pgmap v6856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:47.326 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11856.388953 secs 2024-07-26T10:13:47.326 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11874.128199 secs 2024-07-26T10:13:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:47 smithi042 ceph-mon[32775]: pgmap v6856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11856.388953 secs 2024-07-26T10:13:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11874.128199 secs 2024-07-26T10:13:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:47 smithi164 ceph-mon[36970]: pgmap v6856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11856.388953 secs 2024-07-26T10:13:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11874.128199 secs 2024-07-26T10:13:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:49 smithi042 ceph-mon[32775]: pgmap v6857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:49 smithi164 ceph-mon[36970]: pgmap v6857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:49 smithi163 ceph-mon[36873]: pgmap v6857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:51 smithi042 ceph-mon[32775]: pgmap v6858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:51.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:51 smithi164 ceph-mon[36970]: pgmap v6858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:51 smithi163 ceph-mon[36873]: pgmap v6858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11861.389073 secs 2024-07-26T10:13:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11879.128348 secs 2024-07-26T10:13:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11861.389073 secs 2024-07-26T10:13:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11879.128348 secs 2024-07-26T10:13:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11861.389073 secs 2024-07-26T10:13:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11879.128348 secs 2024-07-26T10:13:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:53 smithi042 ceph-mon[32775]: pgmap v6859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:53 smithi164 ceph-mon[36970]: pgmap v6859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:53 smithi163 ceph-mon[36873]: pgmap v6859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:55 smithi042 ceph-mon[32775]: pgmap v6860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:55 smithi164 ceph-mon[36970]: pgmap v6860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:55 smithi163 ceph-mon[36873]: pgmap v6860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:13:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:57 smithi042 ceph-mon[32775]: pgmap v6861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11866.389211 secs 2024-07-26T10:13:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11884.128488 secs 2024-07-26T10:13:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:57 smithi164 ceph-mon[36970]: pgmap v6861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11866.389211 secs 2024-07-26T10:13:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11884.128488 secs 2024-07-26T10:13:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:57 smithi163 ceph-mon[36873]: pgmap v6861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:13:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11866.389211 secs 2024-07-26T10:13:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11884.128488 secs 2024-07-26T10:13:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:57.738+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:13:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:57.738+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:13:57.740 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:57.738+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:13:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:13:59 smithi042 ceph-mon[32775]: pgmap v6862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:13:59 smithi164 ceph-mon[36970]: pgmap v6862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:13:59 smithi163 ceph-mon[36873]: pgmap v6862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:13:59.720 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:59.717+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:13:59.720 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:59.717+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:13:59.720 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:59.718+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:13:59.720 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:59.718+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:13:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:59.718+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:13:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:59.718+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:13:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:59.718+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:13:59.721 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:13:59.718+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:14:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:01 smithi042 ceph-mon[32775]: pgmap v6863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:01 smithi164 ceph-mon[36970]: pgmap v6863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:01 smithi163 ceph-mon[36873]: pgmap v6863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11871.389331 secs 2024-07-26T10:14:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11889.128649 secs 2024-07-26T10:14:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11871.389331 secs 2024-07-26T10:14:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11889.128649 secs 2024-07-26T10:14:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11871.389331 secs 2024-07-26T10:14:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11889.128649 secs 2024-07-26T10:14:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:03 smithi042 ceph-mon[32775]: pgmap v6864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:03 smithi164 ceph-mon[36970]: pgmap v6864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:03 smithi163 ceph-mon[36873]: pgmap v6864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:05 smithi042 ceph-mon[32775]: pgmap v6865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:14:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:05 smithi164 ceph-mon[36970]: pgmap v6865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:14:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:05 smithi163 ceph-mon[36873]: pgmap v6865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:14:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:07 smithi042 ceph-mon[32775]: pgmap v6866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11876.389431 secs 2024-07-26T10:14:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11894.128771 secs 2024-07-26T10:14:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:07 smithi164 ceph-mon[36970]: pgmap v6866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11876.389431 secs 2024-07-26T10:14:07.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11894.128771 secs 2024-07-26T10:14:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:07 smithi163 ceph-mon[36873]: pgmap v6866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11876.389431 secs 2024-07-26T10:14:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11894.128771 secs 2024-07-26T10:14:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:09 smithi042 ceph-mon[32775]: pgmap v6867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:09 smithi164 ceph-mon[36970]: pgmap v6867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:09 smithi163 ceph-mon[36873]: pgmap v6867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:11 smithi042 ceph-mon[32775]: pgmap v6868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:11 smithi164 ceph-mon[36970]: pgmap v6868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:11 smithi163 ceph-mon[36873]: pgmap v6868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11881.389570 secs 2024-07-26T10:14:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11899.128892 secs 2024-07-26T10:14:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11881.389570 secs 2024-07-26T10:14:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11899.128892 secs 2024-07-26T10:14:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11881.389570 secs 2024-07-26T10:14:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11899.128892 secs 2024-07-26T10:14:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:14:12.739+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:14:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:14:12.739+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:14:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:14:12.739+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:14:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:13 smithi042 ceph-mon[32775]: pgmap v6869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:13 smithi164 ceph-mon[36970]: pgmap v6869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:13 smithi163 ceph-mon[36873]: pgmap v6869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:15 smithi042 ceph-mon[32775]: pgmap v6870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:14:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:15 smithi164 ceph-mon[36970]: pgmap v6870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:14:15.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:15 smithi163 ceph-mon[36873]: pgmap v6870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:14:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:17 smithi042 ceph-mon[32775]: pgmap v6871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11886.389685 secs 2024-07-26T10:14:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11904.128993 secs 2024-07-26T10:14:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:17 smithi164 ceph-mon[36970]: pgmap v6871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11886.389685 secs 2024-07-26T10:14:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11904.128993 secs 2024-07-26T10:14:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:17 smithi163 ceph-mon[36873]: pgmap v6871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11886.389685 secs 2024-07-26T10:14:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11904.128993 secs 2024-07-26T10:14:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:19 smithi042 ceph-mon[32775]: pgmap v6872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:19 smithi164 ceph-mon[36970]: pgmap v6872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:19 smithi163 ceph-mon[36873]: pgmap v6872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:21 smithi042 ceph-mon[32775]: pgmap v6873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:21 smithi164 ceph-mon[36970]: pgmap v6873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:21 smithi163 ceph-mon[36873]: pgmap v6873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11891.389814 secs 2024-07-26T10:14:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11909.129138 secs 2024-07-26T10:14:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11891.389814 secs 2024-07-26T10:14:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11909.129138 secs 2024-07-26T10:14:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11891.389814 secs 2024-07-26T10:14:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11909.129138 secs 2024-07-26T10:14:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:23 smithi164 ceph-mon[36970]: pgmap v6874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:23 smithi163 ceph-mon[36873]: pgmap v6874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:23 smithi042 ceph-mon[32775]: pgmap v6874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:25.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:25 smithi042 ceph-mon[32775]: pgmap v6875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:14:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:25 smithi164 ceph-mon[36970]: pgmap v6875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:14:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:25 smithi163 ceph-mon[36873]: pgmap v6875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:14:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:27 smithi164 ceph-mon[36970]: pgmap v6876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11896.389939 secs 2024-07-26T10:14:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11914.129257 secs 2024-07-26T10:14:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:27 smithi163 ceph-mon[36873]: pgmap v6876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11896.389939 secs 2024-07-26T10:14:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11914.129257 secs 2024-07-26T10:14:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:27 smithi042 ceph-mon[32775]: pgmap v6876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11896.389939 secs 2024-07-26T10:14:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11914.129257 secs 2024-07-26T10:14:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:14:27.745+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:14:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:14:27.745+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:14:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:14:27.745+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:14:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:29 smithi042 ceph-mon[32775]: pgmap v6877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:29 smithi164 ceph-mon[36970]: pgmap v6877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:29 smithi163 ceph-mon[36873]: pgmap v6877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:31 smithi164 ceph-mon[36970]: pgmap v6878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:31 smithi163 ceph-mon[36873]: pgmap v6878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:31 smithi042 ceph-mon[32775]: pgmap v6878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11901.390038 secs 2024-07-26T10:14:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11919.129376 secs 2024-07-26T10:14:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:14:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11901.390038 secs 2024-07-26T10:14:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11919.129376 secs 2024-07-26T10:14:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:14:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11901.390038 secs 2024-07-26T10:14:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11919.129376 secs 2024-07-26T10:14:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:14:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:33 smithi164 ceph-mon[36970]: pgmap v6879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:33.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:14:33.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:14:33.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:14:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:33 smithi163 ceph-mon[36873]: pgmap v6879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:14:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:14:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:14:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:33 smithi042 ceph-mon[32775]: pgmap v6879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:14:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:14:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:14:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:34 smithi164 ceph-mon[36970]: pgmap v6880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:14:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:34 smithi164 ceph-mon[36970]: pgmap v6881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:14:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:34 smithi163 ceph-mon[36873]: pgmap v6880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:14:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:34 smithi163 ceph-mon[36873]: pgmap v6881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:14:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:34 smithi042 ceph-mon[32775]: pgmap v6880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:14:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:34 smithi042 ceph-mon[32775]: pgmap v6881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:14:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:35 smithi164 ceph-mon[36970]: pgmap v6882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:14:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:35 smithi163 ceph-mon[36873]: pgmap v6882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:14:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:35 smithi042 ceph-mon[32775]: pgmap v6882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:14:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11906.390166 secs 2024-07-26T10:14:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11924.129473 secs 2024-07-26T10:14:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:37 smithi042 ceph-mon[32775]: pgmap v6883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:14:37.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11906.390166 secs 2024-07-26T10:14:37.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11924.129473 secs 2024-07-26T10:14:37.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:37 smithi164 ceph-mon[36970]: pgmap v6883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:14:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11906.390166 secs 2024-07-26T10:14:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11924.129473 secs 2024-07-26T10:14:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:37 smithi163 ceph-mon[36873]: pgmap v6883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:14:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:39 smithi042 ceph-mon[32775]: pgmap v6884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:14:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:39 smithi164 ceph-mon[36970]: pgmap v6884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:14:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:39 smithi163 ceph-mon[36873]: pgmap v6884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:14:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:41 smithi042 ceph-mon[32775]: pgmap v6885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:14:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11911.390292 secs 2024-07-26T10:14:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:41 smithi164 ceph-mon[36970]: pgmap v6885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:14:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11911.390292 secs 2024-07-26T10:14:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:41 smithi163 ceph-mon[36873]: pgmap v6885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:14:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11911.390292 secs 2024-07-26T10:14:42.747 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:14:42.746+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:14:42.747 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:14:42.746+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:14:42.747 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:14:42.746+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:14:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11929.129618 secs 2024-07-26T10:14:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11929.129618 secs 2024-07-26T10:14:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11929.129618 secs 2024-07-26T10:14:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:43 smithi042 ceph-mon[32775]: pgmap v6886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:14:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:43 smithi164 ceph-mon[36970]: pgmap v6886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:14:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:43 smithi163 ceph-mon[36873]: pgmap v6886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:14:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:45 smithi042 ceph-mon[32775]: pgmap v6887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:45 smithi164 ceph-mon[36970]: pgmap v6887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:45 smithi163 ceph-mon[36873]: pgmap v6887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11916.390414 secs 2024-07-26T10:14:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11916.390414 secs 2024-07-26T10:14:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11916.390414 secs 2024-07-26T10:14:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11934.129727 secs 2024-07-26T10:14:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:47 smithi042 ceph-mon[32775]: pgmap v6888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:47.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11934.129727 secs 2024-07-26T10:14:47.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:47 smithi163 ceph-mon[36873]: pgmap v6888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11934.129727 secs 2024-07-26T10:14:47.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:47 smithi164 ceph-mon[36970]: pgmap v6888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:49 smithi042 ceph-mon[32775]: pgmap v6889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:14:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:49 smithi164 ceph-mon[36970]: pgmap v6889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:14:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:49 smithi163 ceph-mon[36873]: pgmap v6889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:14:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:51 smithi042 ceph-mon[32775]: pgmap v6890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11921.390538 secs 2024-07-26T10:14:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:51 smithi164 ceph-mon[36970]: pgmap v6890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11921.390538 secs 2024-07-26T10:14:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:51 smithi163 ceph-mon[36873]: pgmap v6890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11921.390538 secs 2024-07-26T10:14:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11939.129834 secs 2024-07-26T10:14:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11939.129834 secs 2024-07-26T10:14:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11939.129834 secs 2024-07-26T10:14:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:53 smithi042 ceph-mon[32775]: pgmap v6891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:53 smithi164 ceph-mon[36970]: pgmap v6891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:53 smithi163 ceph-mon[36873]: pgmap v6891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:55 smithi042 ceph-mon[32775]: pgmap v6892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:55 smithi164 ceph-mon[36970]: pgmap v6892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:55 smithi163 ceph-mon[36873]: pgmap v6892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:14:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11926.390660 secs 2024-07-26T10:14:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11926.390660 secs 2024-07-26T10:14:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11926.390660 secs 2024-07-26T10:14:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:14:57.750+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:14:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:14:57.750+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:14:57.752 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:14:57.750+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:14:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11944.129961 secs 2024-07-26T10:14:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:57 smithi042 ceph-mon[32775]: pgmap v6893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11944.129961 secs 2024-07-26T10:14:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:57 smithi164 ceph-mon[36970]: pgmap v6893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11944.129961 secs 2024-07-26T10:14:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:57 smithi163 ceph-mon[36873]: pgmap v6893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:14:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:14:59 smithi042 ceph-mon[32775]: pgmap v6894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:14:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:14:59 smithi164 ceph-mon[36970]: pgmap v6894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:15:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:14:59 smithi163 ceph-mon[36873]: pgmap v6894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:15:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:01 smithi042 ceph-mon[32775]: pgmap v6895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11931.390775 secs 2024-07-26T10:15:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:01 smithi164 ceph-mon[36970]: pgmap v6895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11931.390775 secs 2024-07-26T10:15:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:01 smithi163 ceph-mon[36873]: pgmap v6895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11931.390775 secs 2024-07-26T10:15:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11949.130083 secs 2024-07-26T10:15:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11949.130083 secs 2024-07-26T10:15:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11949.130083 secs 2024-07-26T10:15:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:03 smithi042 ceph-mon[32775]: pgmap v6896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:03 smithi164 ceph-mon[36970]: pgmap v6896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:03 smithi163 ceph-mon[36873]: pgmap v6896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:05 smithi164 ceph-mon[36970]: pgmap v6897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:05 smithi163 ceph-mon[36873]: pgmap v6897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:05 smithi042 ceph-mon[32775]: pgmap v6897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11936.390855 secs 2024-07-26T10:15:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11936.390855 secs 2024-07-26T10:15:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11936.390855 secs 2024-07-26T10:15:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11954.130200 secs 2024-07-26T10:15:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:07 smithi164 ceph-mon[36970]: pgmap v6898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11954.130200 secs 2024-07-26T10:15:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:07 smithi163 ceph-mon[36873]: pgmap v6898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11954.130200 secs 2024-07-26T10:15:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:07 smithi042 ceph-mon[32775]: pgmap v6898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:09 smithi164 ceph-mon[36970]: pgmap v6899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:15:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:09 smithi163 ceph-mon[36873]: pgmap v6899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:15:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:09 smithi042 ceph-mon[32775]: pgmap v6899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:15:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:11 smithi164 ceph-mon[36970]: pgmap v6900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11941.390985 secs 2024-07-26T10:15:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:11 smithi163 ceph-mon[36873]: pgmap v6900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11941.390985 secs 2024-07-26T10:15:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:11 smithi042 ceph-mon[32775]: pgmap v6900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11941.390985 secs 2024-07-26T10:15:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:15:12.751+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:15:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:15:12.751+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:15:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:15:12.751+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:15:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11959.130322 secs 2024-07-26T10:15:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11959.130322 secs 2024-07-26T10:15:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11959.130322 secs 2024-07-26T10:15:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:13 smithi164 ceph-mon[36970]: pgmap v6901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:13 smithi163 ceph-mon[36873]: pgmap v6901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:13 smithi042 ceph-mon[32775]: pgmap v6901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:15 smithi164 ceph-mon[36970]: pgmap v6902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:15 smithi163 ceph-mon[36873]: pgmap v6902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:15 smithi042 ceph-mon[32775]: pgmap v6902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11946.391069 secs 2024-07-26T10:15:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11946.391069 secs 2024-07-26T10:15:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11946.391069 secs 2024-07-26T10:15:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11964.130442 secs 2024-07-26T10:15:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:17 smithi164 ceph-mon[36970]: pgmap v6903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11964.130442 secs 2024-07-26T10:15:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:17 smithi163 ceph-mon[36873]: pgmap v6903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11964.130442 secs 2024-07-26T10:15:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:17 smithi042 ceph-mon[32775]: pgmap v6903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:19 smithi164 ceph-mon[36970]: pgmap v6904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:15:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:19 smithi163 ceph-mon[36873]: pgmap v6904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:15:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:19 smithi042 ceph-mon[32775]: pgmap v6904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:15:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:21 smithi164 ceph-mon[36970]: pgmap v6905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11951.391193 secs 2024-07-26T10:15:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:21 smithi163 ceph-mon[36873]: pgmap v6905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11951.391193 secs 2024-07-26T10:15:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:21 smithi042 ceph-mon[32775]: pgmap v6905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11951.391193 secs 2024-07-26T10:15:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11969.130566 secs 2024-07-26T10:15:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11969.130566 secs 2024-07-26T10:15:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11969.130566 secs 2024-07-26T10:15:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:23 smithi164 ceph-mon[36970]: pgmap v6906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:23 smithi163 ceph-mon[36873]: pgmap v6906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:23 smithi042 ceph-mon[32775]: pgmap v6906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:25.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:25 smithi042 ceph-mon[32775]: pgmap v6907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:25 smithi164 ceph-mon[36970]: pgmap v6907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:25 smithi163 ceph-mon[36873]: pgmap v6907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11956.391259 secs 2024-07-26T10:15:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11956.391259 secs 2024-07-26T10:15:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11956.391259 secs 2024-07-26T10:15:27.758 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:15:27.756+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:15:27.758 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:15:27.756+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:15:27.759 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:15:27.756+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:15:27.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11974.130663 secs 2024-07-26T10:15:27.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:27 smithi164 ceph-mon[36970]: pgmap v6908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11974.130663 secs 2024-07-26T10:15:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:27 smithi163 ceph-mon[36873]: pgmap v6908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11974.130663 secs 2024-07-26T10:15:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:27 smithi042 ceph-mon[32775]: pgmap v6908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:29.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:29 smithi164 ceph-mon[36970]: pgmap v6909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:15:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:29 smithi163 ceph-mon[36873]: pgmap v6909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:15:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:29 smithi042 ceph-mon[32775]: pgmap v6909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:15:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:31 smithi164 ceph-mon[36970]: pgmap v6910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11961.391353 secs 2024-07-26T10:15:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:31 smithi163 ceph-mon[36873]: pgmap v6910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11961.391353 secs 2024-07-26T10:15:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:31 smithi042 ceph-mon[32775]: pgmap v6910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11961.391353 secs 2024-07-26T10:15:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11979.130788 secs 2024-07-26T10:15:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:15:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11979.130788 secs 2024-07-26T10:15:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:15:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11979.130788 secs 2024-07-26T10:15:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:15:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:33 smithi163 ceph-mon[36873]: pgmap v6911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:15:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:15:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:33 smithi163 ceph-mon[36873]: pgmap v6912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:15:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:33 smithi163 ceph-mon[36873]: pgmap v6913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:15:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:15:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:33 smithi042 ceph-mon[32775]: pgmap v6911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:15:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:15:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:33 smithi042 ceph-mon[32775]: pgmap v6912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:15:34.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:33 smithi042 ceph-mon[32775]: pgmap v6913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:15:34.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:15:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:33 smithi164 ceph-mon[36970]: pgmap v6911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:15:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:15:34.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:33 smithi164 ceph-mon[36970]: pgmap v6912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:15:34.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:33 smithi164 ceph-mon[36970]: pgmap v6913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:15:34.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:15:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:35 smithi164 ceph-mon[36970]: pgmap v6914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:15:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:35 smithi163 ceph-mon[36873]: pgmap v6914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:15:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:35 smithi042 ceph-mon[32775]: pgmap v6914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:15:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11966.391486 secs 2024-07-26T10:15:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11984.130907 secs 2024-07-26T10:15:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11966.391486 secs 2024-07-26T10:15:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11984.130907 secs 2024-07-26T10:15:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11966.391486 secs 2024-07-26T10:15:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11984.130907 secs 2024-07-26T10:15:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:37 smithi164 ceph-mon[36970]: pgmap v6915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:15:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:37 smithi163 ceph-mon[36873]: pgmap v6915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:15:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:37 smithi042 ceph-mon[32775]: pgmap v6915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:15:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:39 smithi164 ceph-mon[36970]: pgmap v6916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:15:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:39 smithi163 ceph-mon[36873]: pgmap v6916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:15:40.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:39 smithi042 ceph-mon[32775]: pgmap v6916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:15:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:41 smithi164 ceph-mon[36970]: pgmap v6917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:15:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11971.391621 secs 2024-07-26T10:15:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11989.131007 secs 2024-07-26T10:15:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:41 smithi163 ceph-mon[36873]: pgmap v6917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:15:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11971.391621 secs 2024-07-26T10:15:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11989.131007 secs 2024-07-26T10:15:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:41 smithi042 ceph-mon[32775]: pgmap v6917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:15:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11971.391621 secs 2024-07-26T10:15:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11989.131007 secs 2024-07-26T10:15:42.759 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:15:42.757+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:15:42.759 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:15:42.757+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:15:42.759 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:15:42.757+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:15:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:43 smithi164 ceph-mon[36970]: pgmap v6918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:15:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:43 smithi163 ceph-mon[36873]: pgmap v6918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:15:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:43 smithi042 ceph-mon[32775]: pgmap v6918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:15:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:45 smithi164 ceph-mon[36970]: pgmap v6919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:45 smithi163 ceph-mon[36873]: pgmap v6919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:45 smithi042 ceph-mon[32775]: pgmap v6919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11976.391719 secs 2024-07-26T10:15:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11994.131112 secs 2024-07-26T10:15:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11976.391719 secs 2024-07-26T10:15:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11994.131112 secs 2024-07-26T10:15:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11976.391719 secs 2024-07-26T10:15:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11994.131112 secs 2024-07-26T10:15:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:47 smithi164 ceph-mon[36970]: pgmap v6920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:47 smithi163 ceph-mon[36873]: pgmap v6920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:47 smithi042 ceph-mon[32775]: pgmap v6920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:49 smithi164 ceph-mon[36970]: pgmap v6921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:15:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:49 smithi163 ceph-mon[36873]: pgmap v6921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:15:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:49 smithi042 ceph-mon[32775]: pgmap v6921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:15:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:51 smithi164 ceph-mon[36970]: pgmap v6922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11981.391843 secs 2024-07-26T10:15:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11999.131233 secs 2024-07-26T10:15:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:51 smithi163 ceph-mon[36873]: pgmap v6922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11981.391843 secs 2024-07-26T10:15:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11999.131233 secs 2024-07-26T10:15:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:51 smithi042 ceph-mon[32775]: pgmap v6922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11981.391843 secs 2024-07-26T10:15:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11999.131233 secs 2024-07-26T10:15:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:53 smithi164 ceph-mon[36970]: pgmap v6923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:53 smithi163 ceph-mon[36873]: pgmap v6923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:53 smithi042 ceph-mon[32775]: pgmap v6923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:55 smithi164 ceph-mon[36970]: pgmap v6924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:55 smithi163 ceph-mon[36873]: pgmap v6924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:55 smithi042 ceph-mon[32775]: pgmap v6924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:15:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11986.391999 secs 2024-07-26T10:15:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12004.131381 secs 2024-07-26T10:15:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11986.391999 secs 2024-07-26T10:15:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12004.131381 secs 2024-07-26T10:15:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11986.391999 secs 2024-07-26T10:15:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12004.131381 secs 2024-07-26T10:15:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:15:57.758+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:15:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:15:57.758+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:15:57.760 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:15:57.758+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:15:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:57 smithi164 ceph-mon[36970]: pgmap v6925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:57 smithi163 ceph-mon[36873]: pgmap v6925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:15:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:57 smithi042 ceph-mon[32775]: pgmap v6925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:15:59 smithi164 ceph-mon[36970]: pgmap v6926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:16:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:15:59 smithi163 ceph-mon[36873]: pgmap v6926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:16:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:15:59 smithi042 ceph-mon[32775]: pgmap v6926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:16:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:01 smithi042 ceph-mon[32775]: pgmap v6927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11991.392120 secs 2024-07-26T10:16:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12009.131501 secs 2024-07-26T10:16:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:01 smithi164 ceph-mon[36970]: pgmap v6927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11991.392120 secs 2024-07-26T10:16:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12009.131501 secs 2024-07-26T10:16:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:01 smithi163 ceph-mon[36873]: pgmap v6927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11991.392120 secs 2024-07-26T10:16:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12009.131501 secs 2024-07-26T10:16:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:03 smithi164 ceph-mon[36970]: pgmap v6928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:03 smithi163 ceph-mon[36873]: pgmap v6928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:03 smithi042 ceph-mon[32775]: pgmap v6928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:05 smithi163 ceph-mon[36873]: pgmap v6929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:05 smithi042 ceph-mon[32775]: pgmap v6929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:05 smithi164 ceph-mon[36970]: pgmap v6929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 11996.392264 secs 2024-07-26T10:16:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12014.131577 secs 2024-07-26T10:16:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 11996.392264 secs 2024-07-26T10:16:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12014.131577 secs 2024-07-26T10:16:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 11996.392264 secs 2024-07-26T10:16:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12014.131577 secs 2024-07-26T10:16:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:07 smithi163 ceph-mon[36873]: pgmap v6930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:07 smithi042 ceph-mon[32775]: pgmap v6930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:07 smithi164 ceph-mon[36970]: pgmap v6930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:09 smithi163 ceph-mon[36873]: pgmap v6931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:16:10.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:09 smithi042 ceph-mon[32775]: pgmap v6931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:16:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:09 smithi164 ceph-mon[36970]: pgmap v6931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:16:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:11 smithi163 ceph-mon[36873]: pgmap v6932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12001.392380 secs 2024-07-26T10:16:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12019.131704 secs 2024-07-26T10:16:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:11 smithi042 ceph-mon[32775]: pgmap v6932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12001.392380 secs 2024-07-26T10:16:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12019.131704 secs 2024-07-26T10:16:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:11 smithi164 ceph-mon[36970]: pgmap v6932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12001.392380 secs 2024-07-26T10:16:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12019.131704 secs 2024-07-26T10:16:12.761 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:16:12.759+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:16:12.761 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:16:12.759+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:16:12.761 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:16:12.759+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:16:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:13 smithi163 ceph-mon[36873]: pgmap v6933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:13 smithi042 ceph-mon[32775]: pgmap v6933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:13 smithi164 ceph-mon[36970]: pgmap v6933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:15 smithi163 ceph-mon[36873]: pgmap v6934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:15 smithi042 ceph-mon[32775]: pgmap v6934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:15 smithi164 ceph-mon[36970]: pgmap v6934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12006.392503 secs 2024-07-26T10:16:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12024.131826 secs 2024-07-26T10:16:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12006.392503 secs 2024-07-26T10:16:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12024.131826 secs 2024-07-26T10:16:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12006.392503 secs 2024-07-26T10:16:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12024.131826 secs 2024-07-26T10:16:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:18 smithi042 ceph-mon[32775]: pgmap v6935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:18 smithi164 ceph-mon[36970]: pgmap v6935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:18 smithi163 ceph-mon[36873]: pgmap v6935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:20 smithi042 ceph-mon[32775]: pgmap v6936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:16:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:20 smithi164 ceph-mon[36970]: pgmap v6936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:16:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:20 smithi163 ceph-mon[36873]: pgmap v6936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:16:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:22 smithi042 ceph-mon[32775]: pgmap v6937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12011.392630 secs 2024-07-26T10:16:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12029.131962 secs 2024-07-26T10:16:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:22 smithi164 ceph-mon[36970]: pgmap v6937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12011.392630 secs 2024-07-26T10:16:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12029.131962 secs 2024-07-26T10:16:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:22 smithi163 ceph-mon[36873]: pgmap v6937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12011.392630 secs 2024-07-26T10:16:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12029.131962 secs 2024-07-26T10:16:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:24 smithi042 ceph-mon[32775]: pgmap v6938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:24 smithi164 ceph-mon[36970]: pgmap v6938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:24 smithi163 ceph-mon[36873]: pgmap v6938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:26 smithi042 ceph-mon[32775]: pgmap v6939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:26 smithi164 ceph-mon[36970]: pgmap v6939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:26 smithi163 ceph-mon[36873]: pgmap v6939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:27.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12016.392758 secs 2024-07-26T10:16:27.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12034.132078 secs 2024-07-26T10:16:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12016.392758 secs 2024-07-26T10:16:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12034.132078 secs 2024-07-26T10:16:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12016.392758 secs 2024-07-26T10:16:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12034.132078 secs 2024-07-26T10:16:27.766 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:16:27.765+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:16:27.767 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:16:27.765+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:16:27.767 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:16:27.765+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:16:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:28 smithi042 ceph-mon[32775]: pgmap v6940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:28 smithi164 ceph-mon[36970]: pgmap v6940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:28.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:28 smithi163 ceph-mon[36873]: pgmap v6940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:30.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:30 smithi042 ceph-mon[32775]: pgmap v6941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:16:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:30 smithi164 ceph-mon[36970]: pgmap v6941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:16:30.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:30 smithi163 ceph-mon[36873]: pgmap v6941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:16:32.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:32 smithi042 ceph-mon[32775]: pgmap v6942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:32.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12021.392912 secs 2024-07-26T10:16:32.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12039.132184 secs 2024-07-26T10:16:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:32 smithi164 ceph-mon[36970]: pgmap v6942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12021.392912 secs 2024-07-26T10:16:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12039.132184 secs 2024-07-26T10:16:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:32 smithi163 ceph-mon[36873]: pgmap v6942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12021.392912 secs 2024-07-26T10:16:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12039.132184 secs 2024-07-26T10:16:33.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:16:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:16:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:16:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:34 smithi042 ceph-mon[32775]: pgmap v6943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:16:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:16:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:16:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:34 smithi164 ceph-mon[36970]: pgmap v6943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:16:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:16:34.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:16:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:34 smithi163 ceph-mon[36873]: pgmap v6943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:16:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:16:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:16:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:35 smithi042 ceph-mon[32775]: pgmap v6944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:16:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:35 smithi042 ceph-mon[32775]: pgmap v6945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:16:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:35 smithi164 ceph-mon[36970]: pgmap v6944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:16:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:35 smithi164 ceph-mon[36970]: pgmap v6945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:16:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:35 smithi163 ceph-mon[36873]: pgmap v6944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:16:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:35 smithi163 ceph-mon[36873]: pgmap v6945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:16:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:37 smithi042 ceph-mon[32775]: pgmap v6946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:16:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12026.393039 secs 2024-07-26T10:16:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12044.132258 secs 2024-07-26T10:16:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:37 smithi164 ceph-mon[36970]: pgmap v6946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:16:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12026.393039 secs 2024-07-26T10:16:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12044.132258 secs 2024-07-26T10:16:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:37 smithi163 ceph-mon[36873]: pgmap v6946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:16:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12026.393039 secs 2024-07-26T10:16:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12044.132258 secs 2024-07-26T10:16:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:39 smithi042 ceph-mon[32775]: pgmap v6947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:16:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:39 smithi164 ceph-mon[36970]: pgmap v6947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:16:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:39 smithi163 ceph-mon[36873]: pgmap v6947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:16:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:41 smithi042 ceph-mon[32775]: pgmap v6948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:16:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:41 smithi164 ceph-mon[36970]: pgmap v6948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:16:41.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:41 smithi163 ceph-mon[36873]: pgmap v6948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:16:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12031.393157 secs 2024-07-26T10:16:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12049.132390 secs 2024-07-26T10:16:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12031.393157 secs 2024-07-26T10:16:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12049.132390 secs 2024-07-26T10:16:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12031.393157 secs 2024-07-26T10:16:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12049.132390 secs 2024-07-26T10:16:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:16:42.766+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:16:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:16:42.766+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:16:42.768 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:16:42.766+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:16:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:43 smithi042 ceph-mon[32775]: pgmap v6949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:16:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:43 smithi164 ceph-mon[36970]: pgmap v6949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:16:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:43 smithi163 ceph-mon[36873]: pgmap v6949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:16:45.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:45 smithi042 ceph-mon[32775]: pgmap v6950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:16:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:45 smithi164 ceph-mon[36970]: pgmap v6950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:16:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:45 smithi163 ceph-mon[36873]: pgmap v6950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:16:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:47 smithi042 ceph-mon[32775]: pgmap v6951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12036.393279 secs 2024-07-26T10:16:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12054.132523 secs 2024-07-26T10:16:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:47 smithi164 ceph-mon[36970]: pgmap v6951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12036.393279 secs 2024-07-26T10:16:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12054.132523 secs 2024-07-26T10:16:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:47 smithi163 ceph-mon[36873]: pgmap v6951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12036.393279 secs 2024-07-26T10:16:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12054.132523 secs 2024-07-26T10:16:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:49 smithi042 ceph-mon[32775]: pgmap v6952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:49 smithi164 ceph-mon[36970]: pgmap v6952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:49 smithi163 ceph-mon[36873]: pgmap v6952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:51 smithi042 ceph-mon[32775]: pgmap v6953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:51 smithi164 ceph-mon[36970]: pgmap v6953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:51 smithi163 ceph-mon[36873]: pgmap v6953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12041.393406 secs 2024-07-26T10:16:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12059.132639 secs 2024-07-26T10:16:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12041.393406 secs 2024-07-26T10:16:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12059.132639 secs 2024-07-26T10:16:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12041.393406 secs 2024-07-26T10:16:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12059.132639 secs 2024-07-26T10:16:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:53 smithi042 ceph-mon[32775]: pgmap v6954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:53 smithi164 ceph-mon[36970]: pgmap v6954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:53 smithi163 ceph-mon[36873]: pgmap v6954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:55 smithi042 ceph-mon[32775]: pgmap v6955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:16:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:55 smithi164 ceph-mon[36970]: pgmap v6955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:16:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:55 smithi163 ceph-mon[36873]: pgmap v6955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:16:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:57 smithi042 ceph-mon[32775]: pgmap v6956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12046.393536 secs 2024-07-26T10:16:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12064.132756 secs 2024-07-26T10:16:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:57 smithi164 ceph-mon[36970]: pgmap v6956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12046.393536 secs 2024-07-26T10:16:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12064.132756 secs 2024-07-26T10:16:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:57 smithi163 ceph-mon[36873]: pgmap v6956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:16:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12046.393536 secs 2024-07-26T10:16:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12064.132756 secs 2024-07-26T10:16:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:16:57.771+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:16:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:16:57.771+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:16:57.773 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:16:57.771+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:16:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:16:59 smithi042 ceph-mon[32775]: pgmap v6957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:16:59 smithi164 ceph-mon[36970]: pgmap v6957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:16:59 smithi163 ceph-mon[36873]: pgmap v6957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:16:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:16:59.747+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:16:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:16:59.747+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:16:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:16:59.747+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:16:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:16:59.747+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:17:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:01 smithi042 ceph-mon[32775]: pgmap v6958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:01 smithi164 ceph-mon[36970]: pgmap v6958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:01 smithi163 ceph-mon[36873]: pgmap v6958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12051.393656 secs 2024-07-26T10:17:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12069.132841 secs 2024-07-26T10:17:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12051.393656 secs 2024-07-26T10:17:02.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12069.132841 secs 2024-07-26T10:17:02.527 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12051.393656 secs 2024-07-26T10:17:02.527 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12069.132841 secs 2024-07-26T10:17:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:03 smithi042 ceph-mon[32775]: pgmap v6959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:03 smithi164 ceph-mon[36970]: pgmap v6959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:03 smithi163 ceph-mon[36873]: pgmap v6959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:05 smithi042 ceph-mon[32775]: pgmap v6960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:17:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:05 smithi164 ceph-mon[36970]: pgmap v6960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:17:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:05 smithi163 ceph-mon[36873]: pgmap v6960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:17:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:07 smithi042 ceph-mon[32775]: pgmap v6961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12056.393781 secs 2024-07-26T10:17:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12074.132964 secs 2024-07-26T10:17:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:07 smithi164 ceph-mon[36970]: pgmap v6961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12056.393781 secs 2024-07-26T10:17:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12074.132964 secs 2024-07-26T10:17:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:07 smithi163 ceph-mon[36873]: pgmap v6961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12056.393781 secs 2024-07-26T10:17:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12074.132964 secs 2024-07-26T10:17:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:09 smithi042 ceph-mon[32775]: pgmap v6962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:09 smithi164 ceph-mon[36970]: pgmap v6962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:09 smithi163 ceph-mon[36873]: pgmap v6962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:11 smithi042 ceph-mon[32775]: pgmap v6963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:11 smithi164 ceph-mon[36970]: pgmap v6963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:11 smithi163 ceph-mon[36873]: pgmap v6963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12061.393905 secs 2024-07-26T10:17:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12079.133080 secs 2024-07-26T10:17:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12061.393905 secs 2024-07-26T10:17:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12079.133080 secs 2024-07-26T10:17:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12061.393905 secs 2024-07-26T10:17:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12079.133080 secs 2024-07-26T10:17:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:17:12.772+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:17:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:17:12.772+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:17:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:17:12.772+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:17:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:13 smithi042 ceph-mon[32775]: pgmap v6964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:13 smithi164 ceph-mon[36970]: pgmap v6964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:13 smithi163 ceph-mon[36873]: pgmap v6964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:15 smithi042 ceph-mon[32775]: pgmap v6965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:17:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:15 smithi164 ceph-mon[36970]: pgmap v6965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:17:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:15 smithi163 ceph-mon[36873]: pgmap v6965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:17:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:17 smithi042 ceph-mon[32775]: pgmap v6966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12066.393973 secs 2024-07-26T10:17:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12084.133202 secs 2024-07-26T10:17:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:17 smithi164 ceph-mon[36970]: pgmap v6966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12066.393973 secs 2024-07-26T10:17:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12084.133202 secs 2024-07-26T10:17:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:17 smithi163 ceph-mon[36873]: pgmap v6966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12066.393973 secs 2024-07-26T10:17:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12084.133202 secs 2024-07-26T10:17:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:19 smithi042 ceph-mon[32775]: pgmap v6967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:19 smithi164 ceph-mon[36970]: pgmap v6967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:19 smithi163 ceph-mon[36873]: pgmap v6967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:21 smithi042 ceph-mon[32775]: pgmap v6968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:21 smithi164 ceph-mon[36970]: pgmap v6968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:21 smithi163 ceph-mon[36873]: pgmap v6968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12071.394099 secs 2024-07-26T10:17:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12089.133316 secs 2024-07-26T10:17:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12071.394099 secs 2024-07-26T10:17:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12089.133316 secs 2024-07-26T10:17:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12071.394099 secs 2024-07-26T10:17:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12089.133316 secs 2024-07-26T10:17:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:23 smithi164 ceph-mon[36970]: pgmap v6969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:23 smithi163 ceph-mon[36873]: pgmap v6969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:23 smithi042 ceph-mon[32775]: pgmap v6969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:25 smithi164 ceph-mon[36970]: pgmap v6970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:17:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:25 smithi163 ceph-mon[36873]: pgmap v6970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:17:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:25 smithi042 ceph-mon[32775]: pgmap v6970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:17:27.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:27 smithi042 ceph-mon[32775]: pgmap v6971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:27.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12076.394237 secs 2024-07-26T10:17:27.449 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12094.133440 secs 2024-07-26T10:17:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:27 smithi164 ceph-mon[36970]: pgmap v6971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12076.394237 secs 2024-07-26T10:17:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12094.133440 secs 2024-07-26T10:17:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:27 smithi163 ceph-mon[36873]: pgmap v6971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12076.394237 secs 2024-07-26T10:17:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12094.133440 secs 2024-07-26T10:17:27.779 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:17:27.777+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:17:27.779 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:17:27.777+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:17:27.779 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:17:27.777+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:17:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:29 smithi164 ceph-mon[36970]: pgmap v6972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:29 smithi163 ceph-mon[36873]: pgmap v6972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:29 smithi042 ceph-mon[32775]: pgmap v6972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:31 smithi164 ceph-mon[36970]: pgmap v6973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:31 smithi163 ceph-mon[36873]: pgmap v6973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:31 smithi042 ceph-mon[32775]: pgmap v6973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12081.394355 secs 2024-07-26T10:17:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12099.133525 secs 2024-07-26T10:17:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12081.394355 secs 2024-07-26T10:17:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12099.133525 secs 2024-07-26T10:17:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12081.394355 secs 2024-07-26T10:17:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12099.133525 secs 2024-07-26T10:17:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:33 smithi164 ceph-mon[36970]: pgmap v6974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:33 smithi163 ceph-mon[36873]: pgmap v6974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:33 smithi042 ceph-mon[32775]: pgmap v6974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:17:34.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:17:34.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:17:34.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:17:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:17:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:17:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:17:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:17:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:17:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:17:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:17:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:17:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:35 smithi042 ceph-mon[32775]: pgmap v6975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:17:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:35 smithi042 ceph-mon[32775]: pgmap v6976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:17:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:35 smithi042 ceph-mon[32775]: pgmap v6977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:17:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:35 smithi164 ceph-mon[36970]: pgmap v6975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:17:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:35 smithi164 ceph-mon[36970]: pgmap v6976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:17:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:35 smithi164 ceph-mon[36970]: pgmap v6977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:17:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:35 smithi163 ceph-mon[36873]: pgmap v6975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:17:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:35 smithi163 ceph-mon[36873]: pgmap v6976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:17:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:35 smithi163 ceph-mon[36873]: pgmap v6977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:17:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12086.394478 secs 2024-07-26T10:17:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:37 smithi164 ceph-mon[36970]: pgmap v6978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:17:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12104.133652 secs 2024-07-26T10:17:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12086.394478 secs 2024-07-26T10:17:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:37 smithi163 ceph-mon[36873]: pgmap v6978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:17:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12104.133652 secs 2024-07-26T10:17:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12086.394478 secs 2024-07-26T10:17:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:37 smithi042 ceph-mon[32775]: pgmap v6978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:17:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12104.133652 secs 2024-07-26T10:17:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:39 smithi164 ceph-mon[36970]: pgmap v6979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:17:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:39 smithi163 ceph-mon[36873]: pgmap v6979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:17:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:39 smithi042 ceph-mon[32775]: pgmap v6979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:17:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:41 smithi164 ceph-mon[36970]: pgmap v6980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:17:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:41 smithi163 ceph-mon[36873]: pgmap v6980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:17:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:41 smithi042 ceph-mon[32775]: pgmap v6980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:17:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12091.394601 secs 2024-07-26T10:17:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12109.133797 secs 2024-07-26T10:17:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12091.394601 secs 2024-07-26T10:17:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12109.133797 secs 2024-07-26T10:17:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12091.394601 secs 2024-07-26T10:17:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12109.133797 secs 2024-07-26T10:17:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:17:42.778+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:17:42.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:17:42.778+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:17:42.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:17:42.778+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:17:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:43 smithi164 ceph-mon[36970]: pgmap v6981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:17:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:43 smithi163 ceph-mon[36873]: pgmap v6981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:17:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:43 smithi042 ceph-mon[32775]: pgmap v6981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:17:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:45 smithi164 ceph-mon[36970]: pgmap v6982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:17:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:45 smithi163 ceph-mon[36873]: pgmap v6982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:17:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:45 smithi042 ceph-mon[32775]: pgmap v6982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:17:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12096.394744 secs 2024-07-26T10:17:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:47 smithi164 ceph-mon[36970]: pgmap v6983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12114.133950 secs 2024-07-26T10:17:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12096.394744 secs 2024-07-26T10:17:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:47 smithi163 ceph-mon[36873]: pgmap v6983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12114.133950 secs 2024-07-26T10:17:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12096.394744 secs 2024-07-26T10:17:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:47 smithi042 ceph-mon[32775]: pgmap v6983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12114.133950 secs 2024-07-26T10:17:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:48 smithi164 ceph-mon[36970]: pgmap v6984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:48 smithi163 ceph-mon[36873]: pgmap v6984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:48 smithi042 ceph-mon[32775]: pgmap v6984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:51 smithi042 ceph-mon[32775]: pgmap v6985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:51 smithi164 ceph-mon[36970]: pgmap v6985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:51 smithi163 ceph-mon[36873]: pgmap v6985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12101.394857 secs 2024-07-26T10:17:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12119.134072 secs 2024-07-26T10:17:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12101.394857 secs 2024-07-26T10:17:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12119.134072 secs 2024-07-26T10:17:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12101.394857 secs 2024-07-26T10:17:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12119.134072 secs 2024-07-26T10:17:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:53 smithi042 ceph-mon[32775]: pgmap v6986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:53 smithi164 ceph-mon[36970]: pgmap v6986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:53 smithi163 ceph-mon[36873]: pgmap v6986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:55 smithi042 ceph-mon[32775]: pgmap v6987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:17:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:55 smithi164 ceph-mon[36970]: pgmap v6987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:17:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:55 smithi163 ceph-mon[36873]: pgmap v6987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:17:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12106.394981 secs 2024-07-26T10:17:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:57 smithi042 ceph-mon[32775]: pgmap v6988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12124.134168 secs 2024-07-26T10:17:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12106.394981 secs 2024-07-26T10:17:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:57 smithi164 ceph-mon[36970]: pgmap v6988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12124.134168 secs 2024-07-26T10:17:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12106.394981 secs 2024-07-26T10:17:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:57 smithi163 ceph-mon[36873]: pgmap v6988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:17:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12124.134168 secs 2024-07-26T10:17:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:17:57.783+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:17:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:17:57.783+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:17:57.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:17:57.783+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:17:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:17:59 smithi042 ceph-mon[32775]: pgmap v6989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:17:59 smithi164 ceph-mon[36970]: pgmap v6989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:17:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:17:59 smithi163 ceph-mon[36873]: pgmap v6989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:01 smithi042 ceph-mon[32775]: pgmap v6990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:01 smithi164 ceph-mon[36970]: pgmap v6990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:01 smithi163 ceph-mon[36873]: pgmap v6990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12111.395079 secs 2024-07-26T10:18:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12129.134297 secs 2024-07-26T10:18:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12111.395079 secs 2024-07-26T10:18:02.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12129.134297 secs 2024-07-26T10:18:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12111.395079 secs 2024-07-26T10:18:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12129.134297 secs 2024-07-26T10:18:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:03 smithi042 ceph-mon[32775]: pgmap v6991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:03 smithi164 ceph-mon[36970]: pgmap v6991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:03 smithi163 ceph-mon[36873]: pgmap v6991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:05 smithi042 ceph-mon[32775]: pgmap v6992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:18:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:05 smithi164 ceph-mon[36970]: pgmap v6992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:18:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:05 smithi163 ceph-mon[36873]: pgmap v6992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:18:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12116.395250 secs 2024-07-26T10:18:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:07 smithi042 ceph-mon[32775]: pgmap v6993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12134.134424 secs 2024-07-26T10:18:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12116.395250 secs 2024-07-26T10:18:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:07 smithi164 ceph-mon[36970]: pgmap v6993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12134.134424 secs 2024-07-26T10:18:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12116.395250 secs 2024-07-26T10:18:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:07 smithi163 ceph-mon[36873]: pgmap v6993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12134.134424 secs 2024-07-26T10:18:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:09 smithi042 ceph-mon[32775]: pgmap v6994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:09 smithi164 ceph-mon[36970]: pgmap v6994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:09 smithi163 ceph-mon[36873]: pgmap v6994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:11 smithi042 ceph-mon[32775]: pgmap v6995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:11 smithi164 ceph-mon[36970]: pgmap v6995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:11 smithi163 ceph-mon[36873]: pgmap v6995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12121.395376 secs 2024-07-26T10:18:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12139.134545 secs 2024-07-26T10:18:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12121.395376 secs 2024-07-26T10:18:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12139.134545 secs 2024-07-26T10:18:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12121.395376 secs 2024-07-26T10:18:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12139.134545 secs 2024-07-26T10:18:12.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:18:12.784+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:18:12.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:18:12.784+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:18:12.785 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:18:12.784+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:18:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:13 smithi042 ceph-mon[32775]: pgmap v6996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:13 smithi164 ceph-mon[36970]: pgmap v6996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:13 smithi163 ceph-mon[36873]: pgmap v6996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:15 smithi042 ceph-mon[32775]: pgmap v6997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:18:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:15 smithi164 ceph-mon[36970]: pgmap v6997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:18:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:15 smithi163 ceph-mon[36873]: pgmap v6997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:18:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12126.395498 secs 2024-07-26T10:18:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:17 smithi042 ceph-mon[32775]: pgmap v6998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12144.134651 secs 2024-07-26T10:18:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12126.395498 secs 2024-07-26T10:18:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:17 smithi164 ceph-mon[36970]: pgmap v6998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12144.134651 secs 2024-07-26T10:18:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12126.395498 secs 2024-07-26T10:18:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:17 smithi163 ceph-mon[36873]: pgmap v6998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12144.134651 secs 2024-07-26T10:18:19.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:19 smithi042 ceph-mon[32775]: pgmap v6999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:19 smithi164 ceph-mon[36970]: pgmap v6999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:19 smithi163 ceph-mon[36873]: pgmap v6999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:21 smithi042 ceph-mon[32775]: pgmap v7000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:21 smithi164 ceph-mon[36970]: pgmap v7000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:21 smithi163 ceph-mon[36873]: pgmap v7000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12131.395622 secs 2024-07-26T10:18:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12149.134751 secs 2024-07-26T10:18:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12131.395622 secs 2024-07-26T10:18:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12149.134751 secs 2024-07-26T10:18:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12131.395622 secs 2024-07-26T10:18:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12149.134751 secs 2024-07-26T10:18:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:23 smithi042 ceph-mon[32775]: pgmap v7001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:23 smithi164 ceph-mon[36970]: pgmap v7001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:23 smithi163 ceph-mon[36873]: pgmap v7001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:25 smithi042 ceph-mon[32775]: pgmap v7002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:18:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:25 smithi164 ceph-mon[36970]: pgmap v7002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:18:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:25 smithi163 ceph-mon[36873]: pgmap v7002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:18:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12136.395741 secs 2024-07-26T10:18:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:27 smithi042 ceph-mon[32775]: pgmap v7003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12154.134872 secs 2024-07-26T10:18:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12136.395741 secs 2024-07-26T10:18:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:27 smithi164 ceph-mon[36970]: pgmap v7003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12154.134872 secs 2024-07-26T10:18:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12136.395741 secs 2024-07-26T10:18:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:27 smithi163 ceph-mon[36873]: pgmap v7003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12154.134872 secs 2024-07-26T10:18:27.791 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:18:27.790+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:18:27.791 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:18:27.790+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:18:27.791 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:18:27.790+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:18:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:29 smithi042 ceph-mon[32775]: pgmap v7004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:29 smithi164 ceph-mon[36970]: pgmap v7004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:29 smithi163 ceph-mon[36873]: pgmap v7004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:31 smithi042 ceph-mon[32775]: pgmap v7005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:31 smithi164 ceph-mon[36970]: pgmap v7005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:31 smithi163 ceph-mon[36873]: pgmap v7005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12141.395824 secs 2024-07-26T10:18:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12159.134994 secs 2024-07-26T10:18:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12141.395824 secs 2024-07-26T10:18:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12159.134994 secs 2024-07-26T10:18:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12141.395824 secs 2024-07-26T10:18:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12159.134994 secs 2024-07-26T10:18:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:33 smithi042 ceph-mon[32775]: pgmap v7006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:33 smithi164 ceph-mon[36970]: pgmap v7006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:33 smithi163 ceph-mon[36873]: pgmap v7006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:18:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:18:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:18:35.209 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:35 smithi042 ceph-mon[32775]: pgmap v7007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:18:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:35 smithi164 ceph-mon[36970]: pgmap v7007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:18:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:35 smithi163 ceph-mon[36873]: pgmap v7007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:18:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12146.395950 secs 2024-07-26T10:18:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:37 smithi042 ceph-mon[32775]: pgmap v7008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12164.135121 secs 2024-07-26T10:18:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12146.395950 secs 2024-07-26T10:18:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:37 smithi164 ceph-mon[36970]: pgmap v7008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12164.135121 secs 2024-07-26T10:18:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12146.395950 secs 2024-07-26T10:18:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:37 smithi163 ceph-mon[36873]: pgmap v7008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12164.135121 secs 2024-07-26T10:18:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:39 smithi042 ceph-mon[32775]: pgmap v7009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:18:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:18:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:18:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:18:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:39 smithi042 ceph-mon[32775]: pgmap v7010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:18:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:39 smithi042 ceph-mon[32775]: pgmap v7011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:18:39.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:18:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:39 smithi164 ceph-mon[36970]: pgmap v7009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:39.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:18:39.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:18:39.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:18:39.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:18:39.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:39 smithi164 ceph-mon[36970]: pgmap v7010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:18:39.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:39 smithi164 ceph-mon[36970]: pgmap v7011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:18:39.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:18:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:39 smithi163 ceph-mon[36873]: pgmap v7009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:18:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:18:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:18:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:18:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:39 smithi163 ceph-mon[36873]: pgmap v7010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:18:39.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:39 smithi163 ceph-mon[36873]: pgmap v7011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:18:39.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:18:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:41 smithi042 ceph-mon[32775]: pgmap v7012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:41 smithi164 ceph-mon[36970]: pgmap v7012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:41 smithi163 ceph-mon[36873]: pgmap v7012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12151.396097 secs 2024-07-26T10:18:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12169.135224 secs 2024-07-26T10:18:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12151.396097 secs 2024-07-26T10:18:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12169.135224 secs 2024-07-26T10:18:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12151.396097 secs 2024-07-26T10:18:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12169.135224 secs 2024-07-26T10:18:42.792 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:18:42.791+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:18:42.792 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:18:42.791+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:18:42.792 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:18:42.791+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:18:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:43 smithi042 ceph-mon[32775]: pgmap v7013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3 op/s 2024-07-26T10:18:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:43 smithi164 ceph-mon[36970]: pgmap v7013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3 op/s 2024-07-26T10:18:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:43 smithi163 ceph-mon[36873]: pgmap v7013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3 op/s 2024-07-26T10:18:45.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:45 smithi042 ceph-mon[32775]: pgmap v7014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:45 smithi164 ceph-mon[36970]: pgmap v7014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:45 smithi163 ceph-mon[36873]: pgmap v7014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12156.396256 secs 2024-07-26T10:18:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:47 smithi042 ceph-mon[32775]: pgmap v7015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12174.135347 secs 2024-07-26T10:18:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12156.396256 secs 2024-07-26T10:18:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:47 smithi164 ceph-mon[36970]: pgmap v7015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12174.135347 secs 2024-07-26T10:18:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12156.396256 secs 2024-07-26T10:18:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:47 smithi163 ceph-mon[36873]: pgmap v7015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12174.135347 secs 2024-07-26T10:18:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:49 smithi042 ceph-mon[32775]: pgmap v7016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:18:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:49 smithi164 ceph-mon[36970]: pgmap v7016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:18:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:49 smithi163 ceph-mon[36873]: pgmap v7016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:18:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:51 smithi042 ceph-mon[32775]: pgmap v7017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:51 smithi164 ceph-mon[36970]: pgmap v7017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:51 smithi163 ceph-mon[36873]: pgmap v7017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12161.396384 secs 2024-07-26T10:18:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12179.135469 secs 2024-07-26T10:18:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12161.396384 secs 2024-07-26T10:18:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12179.135469 secs 2024-07-26T10:18:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12161.396384 secs 2024-07-26T10:18:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12179.135469 secs 2024-07-26T10:18:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:53 smithi042 ceph-mon[32775]: pgmap v7018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:53 smithi164 ceph-mon[36970]: pgmap v7018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:53 smithi163 ceph-mon[36873]: pgmap v7018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:55 smithi042 ceph-mon[32775]: pgmap v7019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:55 smithi164 ceph-mon[36970]: pgmap v7019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:55 smithi163 ceph-mon[36873]: pgmap v7019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:18:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12166.396499 secs 2024-07-26T10:18:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:57 smithi042 ceph-mon[32775]: pgmap v7020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12184.135590 secs 2024-07-26T10:18:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12166.396499 secs 2024-07-26T10:18:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:57 smithi164 ceph-mon[36970]: pgmap v7020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12184.135590 secs 2024-07-26T10:18:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12166.396499 secs 2024-07-26T10:18:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:57 smithi163 ceph-mon[36873]: pgmap v7020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12184.135590 secs 2024-07-26T10:18:57.796 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:18:57.794+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:18:57.796 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:18:57.794+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:18:57.796 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:18:57.794+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:18:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:18:59 smithi042 ceph-mon[32775]: pgmap v7021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:18:59 smithi164 ceph-mon[36970]: pgmap v7021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:18:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:18:59 smithi163 ceph-mon[36873]: pgmap v7021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:01 smithi042 ceph-mon[32775]: pgmap v7022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:01 smithi164 ceph-mon[36970]: pgmap v7022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:01 smithi163 ceph-mon[36873]: pgmap v7022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12171.396625 secs 2024-07-26T10:19:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12189.135729 secs 2024-07-26T10:19:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12171.396625 secs 2024-07-26T10:19:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12189.135729 secs 2024-07-26T10:19:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12171.396625 secs 2024-07-26T10:19:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12189.135729 secs 2024-07-26T10:19:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:03 smithi042 ceph-mon[32775]: pgmap v7023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:03 smithi164 ceph-mon[36970]: pgmap v7023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:03 smithi163 ceph-mon[36873]: pgmap v7023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:05 smithi042 ceph-mon[32775]: pgmap v7024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:05 smithi164 ceph-mon[36970]: pgmap v7024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:05 smithi163 ceph-mon[36873]: pgmap v7024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12176.396747 secs 2024-07-26T10:19:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:07 smithi042 ceph-mon[32775]: pgmap v7025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12194.135873 secs 2024-07-26T10:19:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12176.396747 secs 2024-07-26T10:19:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:07 smithi164 ceph-mon[36970]: pgmap v7025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:07.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12194.135873 secs 2024-07-26T10:19:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12176.396747 secs 2024-07-26T10:19:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:07 smithi163 ceph-mon[36873]: pgmap v7025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12194.135873 secs 2024-07-26T10:19:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:09 smithi042 ceph-mon[32775]: pgmap v7026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:09 smithi164 ceph-mon[36970]: pgmap v7026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:09 smithi163 ceph-mon[36873]: pgmap v7026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:11 smithi042 ceph-mon[32775]: pgmap v7027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:11 smithi164 ceph-mon[36970]: pgmap v7027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:11 smithi163 ceph-mon[36873]: pgmap v7027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:12.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12181.396871 secs 2024-07-26T10:19:12.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12199.136000 secs 2024-07-26T10:19:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12181.396871 secs 2024-07-26T10:19:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12199.136000 secs 2024-07-26T10:19:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12181.396871 secs 2024-07-26T10:19:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12199.136000 secs 2024-07-26T10:19:12.797 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:19:12.795+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:19:12.797 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:19:12.795+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:19:12.797 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:19:12.795+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:19:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:13 smithi042 ceph-mon[32775]: pgmap v7028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:13 smithi164 ceph-mon[36970]: pgmap v7028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:13 smithi163 ceph-mon[36873]: pgmap v7028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:15 smithi164 ceph-mon[36970]: pgmap v7029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:15 smithi163 ceph-mon[36873]: pgmap v7029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:15 smithi042 ceph-mon[32775]: pgmap v7029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12186.397011 secs 2024-07-26T10:19:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:17 smithi164 ceph-mon[36970]: pgmap v7030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12204.136070 secs 2024-07-26T10:19:17.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12186.397011 secs 2024-07-26T10:19:17.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:17 smithi163 ceph-mon[36873]: pgmap v7030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:17.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12204.136070 secs 2024-07-26T10:19:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12186.397011 secs 2024-07-26T10:19:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:17 smithi042 ceph-mon[32775]: pgmap v7030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12204.136070 secs 2024-07-26T10:19:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:19 smithi164 ceph-mon[36970]: pgmap v7031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:19 smithi163 ceph-mon[36873]: pgmap v7031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:19 smithi042 ceph-mon[32775]: pgmap v7031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:21 smithi164 ceph-mon[36970]: pgmap v7032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:21 smithi163 ceph-mon[36873]: pgmap v7032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:21 smithi042 ceph-mon[32775]: pgmap v7032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12191.397105 secs 2024-07-26T10:19:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12209.136194 secs 2024-07-26T10:19:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12191.397105 secs 2024-07-26T10:19:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12209.136194 secs 2024-07-26T10:19:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12191.397105 secs 2024-07-26T10:19:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12209.136194 secs 2024-07-26T10:19:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:23 smithi164 ceph-mon[36970]: pgmap v7033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:23 smithi163 ceph-mon[36873]: pgmap v7033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:23 smithi042 ceph-mon[32775]: pgmap v7033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:25 smithi164 ceph-mon[36970]: pgmap v7034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:25 smithi163 ceph-mon[36873]: pgmap v7034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:25 smithi042 ceph-mon[32775]: pgmap v7034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:27.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12196.397235 secs 2024-07-26T10:19:27.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:27 smithi042 ceph-mon[32775]: pgmap v7035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:27.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12214.136276 secs 2024-07-26T10:19:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12196.397235 secs 2024-07-26T10:19:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:27 smithi164 ceph-mon[36970]: pgmap v7035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12214.136276 secs 2024-07-26T10:19:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12196.397235 secs 2024-07-26T10:19:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:27 smithi163 ceph-mon[36873]: pgmap v7035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12214.136276 secs 2024-07-26T10:19:27.802 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:19:27.800+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:19:27.802 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:19:27.800+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:19:27.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:19:27.800+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:19:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:29 smithi164 ceph-mon[36970]: pgmap v7036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:29 smithi163 ceph-mon[36873]: pgmap v7036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:29 smithi042 ceph-mon[32775]: pgmap v7036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:31 smithi164 ceph-mon[36970]: pgmap v7037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:31 smithi163 ceph-mon[36873]: pgmap v7037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:31 smithi042 ceph-mon[32775]: pgmap v7037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12201.397359 secs 2024-07-26T10:19:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12219.136400 secs 2024-07-26T10:19:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12201.397359 secs 2024-07-26T10:19:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12219.136400 secs 2024-07-26T10:19:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12201.397359 secs 2024-07-26T10:19:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12219.136400 secs 2024-07-26T10:19:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:33 smithi164 ceph-mon[36970]: pgmap v7038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:33 smithi163 ceph-mon[36873]: pgmap v7038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:33 smithi042 ceph-mon[32775]: pgmap v7038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:35 smithi164 ceph-mon[36970]: pgmap v7039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:35 smithi163 ceph-mon[36873]: pgmap v7039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:35 smithi042 ceph-mon[32775]: pgmap v7039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12206.397486 secs 2024-07-26T10:19:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:37 smithi164 ceph-mon[36970]: pgmap v7040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12224.136521 secs 2024-07-26T10:19:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12206.397486 secs 2024-07-26T10:19:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:37 smithi163 ceph-mon[36873]: pgmap v7040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12224.136521 secs 2024-07-26T10:19:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12206.397486 secs 2024-07-26T10:19:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:37 smithi042 ceph-mon[32775]: pgmap v7040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12224.136521 secs 2024-07-26T10:19:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:38 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:19:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:38 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:19:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:38 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:19:39.317 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:39 smithi163 ceph-mon[36873]: pgmap v7041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:39.317 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:39.318 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:39.318 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:39.318 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:39.352 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:39 smithi164 ceph-mon[36970]: pgmap v7041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:39.352 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:39.353 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:39.353 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:39.353 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:39 smithi042 ceph-mon[32775]: pgmap v7041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:40.383 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:40 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T10:19:40.383 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:40 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T10:19:40.430 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:40 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T10:19:40.430 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:40 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T10:19:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:40 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T10:19:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:40 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T10:19:41.456 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:41 smithi163 ceph-mon[36873]: pgmap v7042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:41 smithi164 ceph-mon[36970]: pgmap v7042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:41 smithi042 ceph-mon[32775]: pgmap v7042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:42.467 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12211.397559 secs 2024-07-26T10:19:42.467 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12229.136612 secs 2024-07-26T10:19:42.467 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:19:42.467 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:19:42.467 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.467 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.467 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.467 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.467 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:19:42.468 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:19:42.468 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.468 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.468 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.468 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.468 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:19:42.468 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:19:42.468 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:19:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12211.397559 secs 2024-07-26T10:19:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12229.136612 secs 2024-07-26T10:19:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:19:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:19:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:19:42.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:19:42.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:19:42.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:19:42.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:19:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12211.397559 secs 2024-07-26T10:19:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12229.136612 secs 2024-07-26T10:19:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:19:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:19:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:19:42.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:19:42.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:19:42.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:19:42.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:19:42.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:19:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:19:42.801+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:19:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:19:42.801+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:19:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:19:42.801+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:19:43.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:43 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T10:19:43.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:43 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T10:19:43.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:43 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T10:19:43.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:43 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:19:43.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:43 smithi164 ceph-mon[36970]: pgmap v7043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:43.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:43 smithi164 ceph-mon[36970]: pgmap v7044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:19:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:43 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T10:19:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:43 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T10:19:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:43 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T10:19:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:43 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:19:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:43 smithi163 ceph-mon[36873]: pgmap v7043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:43 smithi163 ceph-mon[36873]: pgmap v7044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:19:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:43 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T10:19:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:43 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T10:19:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:43 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T10:19:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:43 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:19:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:43 smithi042 ceph-mon[32775]: pgmap v7043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:43 smithi042 ceph-mon[32775]: pgmap v7044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:19:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:45 smithi164 ceph-mon[36970]: pgmap v7045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 3 op/s 2024-07-26T10:19:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:45 smithi163 ceph-mon[36873]: pgmap v7045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 3 op/s 2024-07-26T10:19:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:45 smithi042 ceph-mon[32775]: pgmap v7045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 3 op/s 2024-07-26T10:19:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:47 smithi164 ceph-mon[36970]: pgmap v7046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:19:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12216.397690 secs 2024-07-26T10:19:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12234.136757 secs 2024-07-26T10:19:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:47 smithi163 ceph-mon[36873]: pgmap v7046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:19:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12216.397690 secs 2024-07-26T10:19:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12234.136757 secs 2024-07-26T10:19:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:47 smithi042 ceph-mon[32775]: pgmap v7046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:19:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12216.397690 secs 2024-07-26T10:19:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12234.136757 secs 2024-07-26T10:19:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:48 smithi163 ceph-mon[36873]: pgmap v7047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:19:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:48 smithi042 ceph-mon[32775]: pgmap v7047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:19:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:48 smithi164 ceph-mon[36970]: pgmap v7047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:19:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:50 smithi163 ceph-mon[36873]: pgmap v7048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:19:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:50 smithi042 ceph-mon[32775]: pgmap v7048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:19:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:50 smithi164 ceph-mon[36970]: pgmap v7048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:19:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12221.397790 secs 2024-07-26T10:19:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12239.136885 secs 2024-07-26T10:19:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12221.397790 secs 2024-07-26T10:19:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12239.136885 secs 2024-07-26T10:19:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12221.397790 secs 2024-07-26T10:19:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12239.136885 secs 2024-07-26T10:19:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:52 smithi163 ceph-mon[36873]: pgmap v7049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:19:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:52 smithi042 ceph-mon[32775]: pgmap v7049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:19:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:52 smithi164 ceph-mon[36970]: pgmap v7049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:19:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:54 smithi163 ceph-mon[36873]: pgmap v7050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:19:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:54 smithi042 ceph-mon[32775]: pgmap v7050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:19:55.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:54 smithi164 ceph-mon[36970]: pgmap v7050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:19:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:56 smithi163 ceph-mon[36873]: pgmap v7051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12226.397891 secs 2024-07-26T10:19:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12244.137011 secs 2024-07-26T10:19:57.275 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:56 smithi164 ceph-mon[36970]: pgmap v7051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:57.275 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12226.397891 secs 2024-07-26T10:19:57.275 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12244.137011 secs 2024-07-26T10:19:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:56 smithi042 ceph-mon[32775]: pgmap v7051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:19:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12226.397891 secs 2024-07-26T10:19:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12244.137011 secs 2024-07-26T10:19:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:19:57.804+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:19:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:19:57.804+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:19:57.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:19:57.804+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:19:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:19:59 smithi163 ceph-mon[36873]: pgmap v7052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:19:59 smithi042 ceph-mon[32775]: pgmap v7052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:19:59 smithi164 ceph-mon[36970]: pgmap v7052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:19:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:19:59.778+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:19:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:19:59.778+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:19:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:19:59.778+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:19:59.780 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:19:59.778+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:20:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:20:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:20:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:20:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:01 smithi042 ceph-mon[32775]: pgmap v7053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:01 smithi164 ceph-mon[36970]: pgmap v7053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:01 smithi163 ceph-mon[36873]: pgmap v7053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12231.398012 secs 2024-07-26T10:20:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12249.137142 secs 2024-07-26T10:20:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12231.398012 secs 2024-07-26T10:20:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12249.137142 secs 2024-07-26T10:20:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12231.398012 secs 2024-07-26T10:20:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12249.137142 secs 2024-07-26T10:20:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:03 smithi042 ceph-mon[32775]: pgmap v7054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:03 smithi164 ceph-mon[36970]: pgmap v7054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:03 smithi163 ceph-mon[36873]: pgmap v7054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:05 smithi042 ceph-mon[32775]: pgmap v7055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:20:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:05 smithi164 ceph-mon[36970]: pgmap v7055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:20:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:05 smithi163 ceph-mon[36873]: pgmap v7055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:20:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:07 smithi042 ceph-mon[32775]: pgmap v7056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12236.398154 secs 2024-07-26T10:20:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12254.137239 secs 2024-07-26T10:20:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:07 smithi164 ceph-mon[36970]: pgmap v7056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:07.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12236.398154 secs 2024-07-26T10:20:07.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12254.137239 secs 2024-07-26T10:20:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:07 smithi163 ceph-mon[36873]: pgmap v7056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12236.398154 secs 2024-07-26T10:20:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12254.137239 secs 2024-07-26T10:20:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:09 smithi042 ceph-mon[32775]: pgmap v7057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:09 smithi164 ceph-mon[36970]: pgmap v7057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:09 smithi163 ceph-mon[36873]: pgmap v7057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:11 smithi042 ceph-mon[32775]: pgmap v7058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:11 smithi164 ceph-mon[36970]: pgmap v7058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:11 smithi163 ceph-mon[36873]: pgmap v7058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12241.398295 secs 2024-07-26T10:20:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12259.137346 secs 2024-07-26T10:20:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12241.398295 secs 2024-07-26T10:20:12.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12259.137346 secs 2024-07-26T10:20:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12241.398295 secs 2024-07-26T10:20:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12259.137346 secs 2024-07-26T10:20:12.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:20:12.805+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:20:12.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:20:12.805+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:20:12.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:20:12.805+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:20:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:13 smithi042 ceph-mon[32775]: pgmap v7059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:13 smithi164 ceph-mon[36970]: pgmap v7059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:13 smithi163 ceph-mon[36873]: pgmap v7059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:15 smithi042 ceph-mon[32775]: pgmap v7060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:20:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:15 smithi164 ceph-mon[36970]: pgmap v7060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:20:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:15 smithi163 ceph-mon[36873]: pgmap v7060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:20:17.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:17 smithi163 ceph-mon[36873]: pgmap v7061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:20:17.311 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12246.398420 secs 2024-07-26T10:20:17.311 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12264.137468 secs 2024-07-26T10:20:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:17 smithi042 ceph-mon[32775]: pgmap v7061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:20:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12246.398420 secs 2024-07-26T10:20:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12264.137468 secs 2024-07-26T10:20:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:17 smithi164 ceph-mon[36970]: pgmap v7061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:20:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12246.398420 secs 2024-07-26T10:20:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12264.137468 secs 2024-07-26T10:20:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:19 smithi042 ceph-mon[32775]: pgmap v7062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:20:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:19 smithi164 ceph-mon[36970]: pgmap v7062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:20:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:19 smithi163 ceph-mon[36873]: pgmap v7062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:20:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:21 smithi042 ceph-mon[32775]: pgmap v7063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:20:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:21 smithi164 ceph-mon[36970]: pgmap v7063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:20:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:21 smithi163 ceph-mon[36873]: pgmap v7063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:20:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12251.398540 secs 2024-07-26T10:20:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12269.137585 secs 2024-07-26T10:20:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12251.398540 secs 2024-07-26T10:20:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12269.137585 secs 2024-07-26T10:20:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12251.398540 secs 2024-07-26T10:20:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12269.137585 secs 2024-07-26T10:20:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:23 smithi042 ceph-mon[32775]: pgmap v7064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:20:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:23 smithi164 ceph-mon[36970]: pgmap v7064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:20:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:23 smithi163 ceph-mon[36873]: pgmap v7064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:20:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:25 smithi042 ceph-mon[32775]: pgmap v7065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:20:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:25 smithi164 ceph-mon[36970]: pgmap v7065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:20:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:25 smithi163 ceph-mon[36873]: pgmap v7065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:20:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12256.398664 secs 2024-07-26T10:20:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:27 smithi042 ceph-mon[32775]: pgmap v7066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:20:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12274.137705 secs 2024-07-26T10:20:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12256.398664 secs 2024-07-26T10:20:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:27 smithi164 ceph-mon[36970]: pgmap v7066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:20:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12274.137705 secs 2024-07-26T10:20:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12256.398664 secs 2024-07-26T10:20:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:27 smithi163 ceph-mon[36873]: pgmap v7066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:20:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12274.137705 secs 2024-07-26T10:20:27.811 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:20:27.809+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:20:27.811 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:20:27.809+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:20:27.811 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:20:27.809+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:20:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:29 smithi042 ceph-mon[32775]: pgmap v7067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:29 smithi164 ceph-mon[36970]: pgmap v7067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:29 smithi163 ceph-mon[36873]: pgmap v7067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:31 smithi042 ceph-mon[32775]: pgmap v7068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:31 smithi164 ceph-mon[36970]: pgmap v7068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:31 smithi163 ceph-mon[36873]: pgmap v7068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12261.398783 secs 2024-07-26T10:20:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12279.137845 secs 2024-07-26T10:20:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12261.398783 secs 2024-07-26T10:20:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12279.137845 secs 2024-07-26T10:20:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12261.398783 secs 2024-07-26T10:20:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12279.137845 secs 2024-07-26T10:20:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:33 smithi042 ceph-mon[32775]: pgmap v7069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:33 smithi164 ceph-mon[36970]: pgmap v7069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:33 smithi163 ceph-mon[36873]: pgmap v7069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:35 smithi042 ceph-mon[32775]: pgmap v7070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:20:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:35 smithi164 ceph-mon[36970]: pgmap v7070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:20:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:35 smithi163 ceph-mon[36873]: pgmap v7070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:20:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12266.398904 secs 2024-07-26T10:20:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:37 smithi042 ceph-mon[32775]: pgmap v7071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12284.138006 secs 2024-07-26T10:20:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12266.398904 secs 2024-07-26T10:20:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:37 smithi164 ceph-mon[36970]: pgmap v7071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12284.138006 secs 2024-07-26T10:20:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12266.398904 secs 2024-07-26T10:20:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:37 smithi163 ceph-mon[36873]: pgmap v7071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12284.138006 secs 2024-07-26T10:20:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:39 smithi042 ceph-mon[32775]: pgmap v7072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:39 smithi164 ceph-mon[36970]: pgmap v7072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:39 smithi163 ceph-mon[36873]: pgmap v7072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:41 smithi042 ceph-mon[32775]: pgmap v7073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:41 smithi164 ceph-mon[36970]: pgmap v7073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:41 smithi163 ceph-mon[36873]: pgmap v7073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12271.399026 secs 2024-07-26T10:20:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12289.138093 secs 2024-07-26T10:20:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:20:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12271.399026 secs 2024-07-26T10:20:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12289.138093 secs 2024-07-26T10:20:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:20:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12271.399026 secs 2024-07-26T10:20:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12289.138093 secs 2024-07-26T10:20:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:20:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:20:42.810+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:20:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:20:42.810+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:20:42.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:20:42.810+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:20:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:43 smithi042 ceph-mon[32775]: pgmap v7074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:20:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:20:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:20:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:43 smithi164 ceph-mon[36970]: pgmap v7074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:43.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:20:43.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:20:43.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:20:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:43 smithi163 ceph-mon[36873]: pgmap v7074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:20:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:20:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:20:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:44 smithi042 ceph-mon[32775]: pgmap v7075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:20:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:44 smithi042 ceph-mon[32775]: pgmap v7076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:20:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:44 smithi164 ceph-mon[36970]: pgmap v7075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:20:44.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:44 smithi164 ceph-mon[36970]: pgmap v7076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:20:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:44 smithi163 ceph-mon[36873]: pgmap v7075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:20:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:44 smithi163 ceph-mon[36873]: pgmap v7076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:20:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:46 smithi042 ceph-mon[32775]: pgmap v7077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:20:46.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:46 smithi164 ceph-mon[36970]: pgmap v7077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:20:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:46 smithi163 ceph-mon[36873]: pgmap v7077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:20:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12276.399161 secs 2024-07-26T10:20:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12294.138173 secs 2024-07-26T10:20:47.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12276.399161 secs 2024-07-26T10:20:47.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12294.138173 secs 2024-07-26T10:20:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12276.399161 secs 2024-07-26T10:20:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12294.138173 secs 2024-07-26T10:20:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:48 smithi042 ceph-mon[32775]: pgmap v7078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:20:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:48 smithi164 ceph-mon[36970]: pgmap v7078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:20:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:48 smithi163 ceph-mon[36873]: pgmap v7078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:20:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:50 smithi042 ceph-mon[32775]: pgmap v7079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:20:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:50 smithi164 ceph-mon[36970]: pgmap v7079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:20:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:50 smithi163 ceph-mon[36873]: pgmap v7079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:20:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:52 smithi042 ceph-mon[32775]: pgmap v7080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:20:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12281.399282 secs 2024-07-26T10:20:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12299.138296 secs 2024-07-26T10:20:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:52 smithi164 ceph-mon[36970]: pgmap v7080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:20:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12281.399282 secs 2024-07-26T10:20:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12299.138296 secs 2024-07-26T10:20:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:52 smithi163 ceph-mon[36873]: pgmap v7080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:20:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12281.399282 secs 2024-07-26T10:20:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12299.138296 secs 2024-07-26T10:20:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:54 smithi042 ceph-mon[32775]: pgmap v7081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:20:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:54 smithi164 ceph-mon[36970]: pgmap v7081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:20:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:54 smithi163 ceph-mon[36873]: pgmap v7081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:20:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:56 smithi042 ceph-mon[32775]: pgmap v7082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:56 smithi164 ceph-mon[36970]: pgmap v7082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:56 smithi163 ceph-mon[36873]: pgmap v7082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:20:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12286.399404 secs 2024-07-26T10:20:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12304.138425 secs 2024-07-26T10:20:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12286.399404 secs 2024-07-26T10:20:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12304.138425 secs 2024-07-26T10:20:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12286.399404 secs 2024-07-26T10:20:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12304.138425 secs 2024-07-26T10:20:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:20:57.815+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:20:57.817 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:20:57.815+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:20:57.818 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:20:57.815+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:20:58.394 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:20:58 smithi164 ceph-mon[36970]: pgmap v7083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:20:58 smithi042 ceph-mon[32775]: pgmap v7083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:20:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:20:58 smithi163 ceph-mon[36873]: pgmap v7083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:00 smithi042 ceph-mon[32775]: pgmap v7084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:21:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:00 smithi164 ceph-mon[36970]: pgmap v7084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:21:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:00 smithi163 ceph-mon[36873]: pgmap v7084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:21:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:02 smithi164 ceph-mon[36970]: pgmap v7085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12291.399523 secs 2024-07-26T10:21:02.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12309.138547 secs 2024-07-26T10:21:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:02 smithi163 ceph-mon[36873]: pgmap v7085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12291.399523 secs 2024-07-26T10:21:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12309.138547 secs 2024-07-26T10:21:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:02 smithi042 ceph-mon[32775]: pgmap v7085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12291.399523 secs 2024-07-26T10:21:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12309.138547 secs 2024-07-26T10:21:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:04 smithi164 ceph-mon[36970]: pgmap v7086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:04 smithi163 ceph-mon[36873]: pgmap v7086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:04 smithi042 ceph-mon[32775]: pgmap v7086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:06 smithi164 ceph-mon[36970]: pgmap v7087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:06 smithi163 ceph-mon[36873]: pgmap v7087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:06 smithi042 ceph-mon[32775]: pgmap v7087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12296.399618 secs 2024-07-26T10:21:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12314.138673 secs 2024-07-26T10:21:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12296.399618 secs 2024-07-26T10:21:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12314.138673 secs 2024-07-26T10:21:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12296.399618 secs 2024-07-26T10:21:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12314.138673 secs 2024-07-26T10:21:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:08 smithi164 ceph-mon[36970]: pgmap v7088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:08 smithi163 ceph-mon[36873]: pgmap v7088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:08 smithi042 ceph-mon[32775]: pgmap v7088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:10 smithi164 ceph-mon[36970]: pgmap v7089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:21:10.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:10 smithi163 ceph-mon[36873]: pgmap v7089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:21:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:10 smithi042 ceph-mon[32775]: pgmap v7089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:21:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:12 smithi042 ceph-mon[32775]: pgmap v7090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12301.399696 secs 2024-07-26T10:21:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12319.138790 secs 2024-07-26T10:21:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:12 smithi164 ceph-mon[36970]: pgmap v7090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12301.399696 secs 2024-07-26T10:21:12.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12319.138790 secs 2024-07-26T10:21:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:12 smithi163 ceph-mon[36873]: pgmap v7090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12301.399696 secs 2024-07-26T10:21:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12319.138790 secs 2024-07-26T10:21:12.819 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:21:12.817+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:21:12.819 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:21:12.817+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:21:12.819 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:21:12.817+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:21:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:14 smithi164 ceph-mon[36970]: pgmap v7091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:14.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:14 smithi163 ceph-mon[36873]: pgmap v7091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:14 smithi042 ceph-mon[32775]: pgmap v7091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:16 smithi164 ceph-mon[36970]: pgmap v7092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:16 smithi163 ceph-mon[36873]: pgmap v7092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:16 smithi042 ceph-mon[32775]: pgmap v7092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12306.399838 secs 2024-07-26T10:21:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12324.138947 secs 2024-07-26T10:21:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12306.399838 secs 2024-07-26T10:21:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12324.138947 secs 2024-07-26T10:21:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12306.399838 secs 2024-07-26T10:21:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12324.138947 secs 2024-07-26T10:21:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:18 smithi164 ceph-mon[36970]: pgmap v7093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:18 smithi163 ceph-mon[36873]: pgmap v7093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:18 smithi042 ceph-mon[32775]: pgmap v7093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:20 smithi164 ceph-mon[36970]: pgmap v7094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:21:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:20 smithi163 ceph-mon[36873]: pgmap v7094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:21:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:20 smithi042 ceph-mon[32775]: pgmap v7094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:21:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:22 smithi164 ceph-mon[36970]: pgmap v7095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12311.399957 secs 2024-07-26T10:21:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12329.139072 secs 2024-07-26T10:21:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:22 smithi163 ceph-mon[36873]: pgmap v7095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12311.399957 secs 2024-07-26T10:21:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12329.139072 secs 2024-07-26T10:21:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:22 smithi042 ceph-mon[32775]: pgmap v7095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12311.399957 secs 2024-07-26T10:21:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12329.139072 secs 2024-07-26T10:21:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:24 smithi164 ceph-mon[36970]: pgmap v7096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:24 smithi163 ceph-mon[36873]: pgmap v7096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:24 smithi042 ceph-mon[32775]: pgmap v7096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:26 smithi164 ceph-mon[36970]: pgmap v7097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:26 smithi163 ceph-mon[36873]: pgmap v7097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:26 smithi042 ceph-mon[32775]: pgmap v7097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12316.400081 secs 2024-07-26T10:21:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12334.139200 secs 2024-07-26T10:21:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12316.400081 secs 2024-07-26T10:21:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12334.139200 secs 2024-07-26T10:21:27.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12316.400081 secs 2024-07-26T10:21:27.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12334.139200 secs 2024-07-26T10:21:27.824 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:21:27.822+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:21:27.824 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:21:27.822+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:21:27.825 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:21:27.822+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:21:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:28 smithi164 ceph-mon[36970]: pgmap v7098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:28 smithi163 ceph-mon[36873]: pgmap v7098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:28 smithi042 ceph-mon[32775]: pgmap v7098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:30 smithi164 ceph-mon[36970]: pgmap v7099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:21:30.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:30 smithi163 ceph-mon[36873]: pgmap v7099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:21:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:30 smithi042 ceph-mon[32775]: pgmap v7099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:21:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:32 smithi164 ceph-mon[36970]: pgmap v7100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12321.400201 secs 2024-07-26T10:21:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12339.139359 secs 2024-07-26T10:21:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:32 smithi163 ceph-mon[36873]: pgmap v7100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12321.400201 secs 2024-07-26T10:21:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12339.139359 secs 2024-07-26T10:21:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:32 smithi042 ceph-mon[32775]: pgmap v7100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12321.400201 secs 2024-07-26T10:21:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12339.139359 secs 2024-07-26T10:21:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:34 smithi164 ceph-mon[36970]: pgmap v7101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:34 smithi163 ceph-mon[36873]: pgmap v7101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:34 smithi042 ceph-mon[32775]: pgmap v7101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:36 smithi164 ceph-mon[36970]: pgmap v7102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:36 smithi163 ceph-mon[36873]: pgmap v7102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:36 smithi042 ceph-mon[32775]: pgmap v7102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12326.400317 secs 2024-07-26T10:21:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12344.139484 secs 2024-07-26T10:21:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12326.400317 secs 2024-07-26T10:21:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12344.139484 secs 2024-07-26T10:21:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12326.400317 secs 2024-07-26T10:21:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12344.139484 secs 2024-07-26T10:21:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:38 smithi163 ceph-mon[36873]: pgmap v7103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:38 smithi042 ceph-mon[32775]: pgmap v7103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:38 smithi164 ceph-mon[36970]: pgmap v7103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:40 smithi163 ceph-mon[36873]: pgmap v7104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:21:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:40 smithi042 ceph-mon[32775]: pgmap v7104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:21:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:40 smithi164 ceph-mon[36970]: pgmap v7104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:21:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:42 smithi163 ceph-mon[36873]: pgmap v7105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12331.400379 secs 2024-07-26T10:21:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12349.139608 secs 2024-07-26T10:21:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:42 smithi042 ceph-mon[32775]: pgmap v7105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12331.400379 secs 2024-07-26T10:21:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12349.139608 secs 2024-07-26T10:21:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:42 smithi164 ceph-mon[36970]: pgmap v7105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12331.400379 secs 2024-07-26T10:21:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12349.139608 secs 2024-07-26T10:21:42.825 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:21:42.823+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:21:42.825 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:21:42.823+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:21:42.825 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:21:42.823+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:21:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:21:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:21:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:21:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:21:43.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:21:43.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:21:43.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:21:43.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:21:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:21:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:21:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:21:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:21:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:44 smithi163 ceph-mon[36873]: pgmap v7106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:44 smithi163 ceph-mon[36873]: pgmap v7107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:21:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:44 smithi163 ceph-mon[36873]: pgmap v7108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:21:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:44 smithi042 ceph-mon[32775]: pgmap v7106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:44 smithi042 ceph-mon[32775]: pgmap v7107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:21:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:44 smithi042 ceph-mon[32775]: pgmap v7108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:21:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:44 smithi164 ceph-mon[36970]: pgmap v7106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:44 smithi164 ceph-mon[36970]: pgmap v7107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:21:44.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:44 smithi164 ceph-mon[36970]: pgmap v7108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:21:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:45 smithi163 ceph-mon[36873]: pgmap v7109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:21:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:45 smithi042 ceph-mon[32775]: pgmap v7109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:21:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:45 smithi164 ceph-mon[36970]: pgmap v7109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:21:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12336.400512 secs 2024-07-26T10:21:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12336.400512 secs 2024-07-26T10:21:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12336.400512 secs 2024-07-26T10:21:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12354.139753 secs 2024-07-26T10:21:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:47 smithi042 ceph-mon[32775]: pgmap v7110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:21:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12354.139753 secs 2024-07-26T10:21:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:47 smithi164 ceph-mon[36970]: pgmap v7110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:21:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12354.139753 secs 2024-07-26T10:21:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:47 smithi163 ceph-mon[36873]: pgmap v7110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:21:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:50 smithi042 ceph-mon[32775]: pgmap v7111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:21:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:50 smithi164 ceph-mon[36970]: pgmap v7111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:21:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:50 smithi163 ceph-mon[36873]: pgmap v7111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:21:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12341.400609 secs 2024-07-26T10:21:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:52 smithi042 ceph-mon[32775]: pgmap v7112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:21:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12359.139883 secs 2024-07-26T10:21:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12341.400609 secs 2024-07-26T10:21:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:52 smithi164 ceph-mon[36970]: pgmap v7112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:21:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12359.139883 secs 2024-07-26T10:21:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12341.400609 secs 2024-07-26T10:21:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:52 smithi163 ceph-mon[36873]: pgmap v7112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:21:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12359.139883 secs 2024-07-26T10:21:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:54 smithi042 ceph-mon[32775]: pgmap v7113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:21:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:54 smithi164 ceph-mon[36970]: pgmap v7113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:21:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:54 smithi163 ceph-mon[36873]: pgmap v7113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:21:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:56 smithi042 ceph-mon[32775]: pgmap v7114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:56 smithi164 ceph-mon[36970]: pgmap v7114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:56 smithi163 ceph-mon[36873]: pgmap v7114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:21:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12346.400732 secs 2024-07-26T10:21:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12364.140002 secs 2024-07-26T10:21:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12346.400732 secs 2024-07-26T10:21:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12364.140002 secs 2024-07-26T10:21:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12346.400732 secs 2024-07-26T10:21:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12364.140002 secs 2024-07-26T10:21:57.830 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:21:57.829+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:21:57.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:21:57.829+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:21:57.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:21:57.829+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:21:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:21:58 smithi042 ceph-mon[32775]: pgmap v7115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:21:58 smithi164 ceph-mon[36970]: pgmap v7115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:21:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:21:58 smithi163 ceph-mon[36873]: pgmap v7115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:00 smithi042 ceph-mon[32775]: pgmap v7116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:00 smithi164 ceph-mon[36970]: pgmap v7116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:00 smithi163 ceph-mon[36873]: pgmap v7116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12351.400870 secs 2024-07-26T10:22:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:02 smithi042 ceph-mon[32775]: pgmap v7117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12369.140129 secs 2024-07-26T10:22:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12351.400870 secs 2024-07-26T10:22:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:02 smithi164 ceph-mon[36970]: pgmap v7117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12369.140129 secs 2024-07-26T10:22:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12351.400870 secs 2024-07-26T10:22:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:02 smithi163 ceph-mon[36873]: pgmap v7117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12369.140129 secs 2024-07-26T10:22:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:04 smithi042 ceph-mon[32775]: pgmap v7118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:04 smithi164 ceph-mon[36970]: pgmap v7118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:04 smithi163 ceph-mon[36873]: pgmap v7118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:06 smithi164 ceph-mon[36970]: pgmap v7119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:06 smithi163 ceph-mon[36873]: pgmap v7119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:06 smithi042 ceph-mon[32775]: pgmap v7119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12356.401014 secs 2024-07-26T10:22:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12374.140204 secs 2024-07-26T10:22:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12356.401014 secs 2024-07-26T10:22:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12374.140204 secs 2024-07-26T10:22:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12356.401014 secs 2024-07-26T10:22:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12374.140204 secs 2024-07-26T10:22:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:08 smithi164 ceph-mon[36970]: pgmap v7120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:08 smithi163 ceph-mon[36873]: pgmap v7120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:08 smithi042 ceph-mon[32775]: pgmap v7120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:10 smithi164 ceph-mon[36970]: pgmap v7121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:10 smithi163 ceph-mon[36873]: pgmap v7121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:10 smithi042 ceph-mon[32775]: pgmap v7121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12361.401135 secs 2024-07-26T10:22:12.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:12 smithi164 ceph-mon[36970]: pgmap v7122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:12.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12379.140331 secs 2024-07-26T10:22:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12361.401135 secs 2024-07-26T10:22:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:12 smithi163 ceph-mon[36873]: pgmap v7122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12379.140331 secs 2024-07-26T10:22:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12361.401135 secs 2024-07-26T10:22:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:12 smithi042 ceph-mon[32775]: pgmap v7122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12379.140331 secs 2024-07-26T10:22:12.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:12.831+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:22:12.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:12.831+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:22:12.833 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:12.831+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:22:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:14 smithi164 ceph-mon[36970]: pgmap v7123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:14.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:14 smithi163 ceph-mon[36873]: pgmap v7123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:14 smithi042 ceph-mon[32775]: pgmap v7123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:14.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:14.800+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:22:14.801 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:14.800+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:22:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:14.800+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:22:14.802 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:14.800+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:22:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:16 smithi164 ceph-mon[36970]: pgmap v7124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:16 smithi163 ceph-mon[36873]: pgmap v7124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:16 smithi042 ceph-mon[32775]: pgmap v7124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12366.401235 secs 2024-07-26T10:22:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12384.140477 secs 2024-07-26T10:22:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12366.401235 secs 2024-07-26T10:22:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12384.140477 secs 2024-07-26T10:22:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12366.401235 secs 2024-07-26T10:22:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12384.140477 secs 2024-07-26T10:22:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:18 smithi164 ceph-mon[36970]: pgmap v7125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:18 smithi163 ceph-mon[36873]: pgmap v7125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:18 smithi042 ceph-mon[32775]: pgmap v7125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:20 smithi164 ceph-mon[36970]: pgmap v7126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:20 smithi163 ceph-mon[36873]: pgmap v7126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:20 smithi042 ceph-mon[32775]: pgmap v7126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12371.401311 secs 2024-07-26T10:22:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:22 smithi164 ceph-mon[36970]: pgmap v7127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12389.140600 secs 2024-07-26T10:22:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12371.401311 secs 2024-07-26T10:22:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:22 smithi163 ceph-mon[36873]: pgmap v7127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12389.140600 secs 2024-07-26T10:22:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12371.401311 secs 2024-07-26T10:22:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:22 smithi042 ceph-mon[32775]: pgmap v7127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12389.140600 secs 2024-07-26T10:22:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:24 smithi164 ceph-mon[36970]: pgmap v7128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:24 smithi163 ceph-mon[36873]: pgmap v7128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:24 smithi042 ceph-mon[32775]: pgmap v7128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:26 smithi164 ceph-mon[36970]: pgmap v7129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:26 smithi163 ceph-mon[36873]: pgmap v7129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:26 smithi042 ceph-mon[32775]: pgmap v7129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12376.401435 secs 2024-07-26T10:22:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12394.140714 secs 2024-07-26T10:22:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12376.401435 secs 2024-07-26T10:22:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12394.140714 secs 2024-07-26T10:22:27.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12376.401435 secs 2024-07-26T10:22:27.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12394.140714 secs 2024-07-26T10:22:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:27.835+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:22:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:27.835+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:22:27.839 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:27.835+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:22:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:28 smithi164 ceph-mon[36970]: pgmap v7130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:28 smithi163 ceph-mon[36873]: pgmap v7130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:28 smithi042 ceph-mon[32775]: pgmap v7130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:30 smithi164 ceph-mon[36970]: pgmap v7131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:30.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:30 smithi163 ceph-mon[36873]: pgmap v7131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:30 smithi042 ceph-mon[32775]: pgmap v7131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12381.401550 secs 2024-07-26T10:22:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:32 smithi164 ceph-mon[36970]: pgmap v7132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12399.140866 secs 2024-07-26T10:22:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12381.401550 secs 2024-07-26T10:22:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:32 smithi163 ceph-mon[36873]: pgmap v7132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12399.140866 secs 2024-07-26T10:22:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12381.401550 secs 2024-07-26T10:22:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:32 smithi042 ceph-mon[32775]: pgmap v7132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12399.140866 secs 2024-07-26T10:22:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:34 smithi164 ceph-mon[36970]: pgmap v7133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:34 smithi163 ceph-mon[36873]: pgmap v7133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:34 smithi042 ceph-mon[32775]: pgmap v7133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:36 smithi164 ceph-mon[36970]: pgmap v7134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:36 smithi163 ceph-mon[36873]: pgmap v7134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:36 smithi042 ceph-mon[32775]: pgmap v7134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12386.401943 secs 2024-07-26T10:22:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12404.140992 secs 2024-07-26T10:22:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12386.401943 secs 2024-07-26T10:22:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12404.140992 secs 2024-07-26T10:22:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12386.401943 secs 2024-07-26T10:22:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12404.140992 secs 2024-07-26T10:22:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:38 smithi164 ceph-mon[36970]: pgmap v7135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:38 smithi163 ceph-mon[36873]: pgmap v7135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:38 smithi042 ceph-mon[32775]: pgmap v7135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:40 smithi164 ceph-mon[36970]: pgmap v7136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:40 smithi163 ceph-mon[36873]: pgmap v7136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:40 smithi042 ceph-mon[32775]: pgmap v7136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12391.402060 secs 2024-07-26T10:22:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:42 smithi164 ceph-mon[36970]: pgmap v7137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12409.141086 secs 2024-07-26T10:22:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12391.402060 secs 2024-07-26T10:22:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:42 smithi163 ceph-mon[36873]: pgmap v7137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12409.141086 secs 2024-07-26T10:22:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12391.402060 secs 2024-07-26T10:22:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:42 smithi042 ceph-mon[32775]: pgmap v7137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12409.141086 secs 2024-07-26T10:22:42.839 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:42.837+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:22:42.839 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:42.837+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:22:42.839 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:42.837+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:22:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:22:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:22:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:22:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:44 smithi164 ceph-mon[36970]: pgmap v7138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:44 smithi163 ceph-mon[36873]: pgmap v7138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:44 smithi042 ceph-mon[32775]: pgmap v7138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:46.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:45 smithi164 ceph-mon[36970]: pgmap v7139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:46.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:22:46.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:22:46.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:45 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:45 smithi163 ceph-mon[36873]: pgmap v7139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:22:46.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:22:46.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:45 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:45 smithi042 ceph-mon[32775]: pgmap v7139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:22:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:22:46.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:22:46.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:45 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:22:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:46 smithi164 ceph-mon[36970]: pgmap v7140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:22:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:46 smithi164 ceph-mon[36970]: pgmap v7141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:22:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12396.402180 secs 2024-07-26T10:22:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:46 smithi163 ceph-mon[36873]: pgmap v7140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:22:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:46 smithi163 ceph-mon[36873]: pgmap v7141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:22:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12396.402180 secs 2024-07-26T10:22:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:46 smithi042 ceph-mon[32775]: pgmap v7140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:22:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:46 smithi042 ceph-mon[32775]: pgmap v7141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:22:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12396.402180 secs 2024-07-26T10:22:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12414.141206 secs 2024-07-26T10:22:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12414.141206 secs 2024-07-26T10:22:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12414.141206 secs 2024-07-26T10:22:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:48 smithi164 ceph-mon[36970]: pgmap v7142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:22:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:48 smithi163 ceph-mon[36873]: pgmap v7142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:22:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:48 smithi042 ceph-mon[32775]: pgmap v7142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:22:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:50 smithi164 ceph-mon[36970]: pgmap v7143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:22:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:50 smithi163 ceph-mon[36873]: pgmap v7143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:22:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:50 smithi042 ceph-mon[32775]: pgmap v7143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:22:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12401.402300 secs 2024-07-26T10:22:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12401.402300 secs 2024-07-26T10:22:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12401.402300 secs 2024-07-26T10:22:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12419.141328 secs 2024-07-26T10:22:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:52 smithi164 ceph-mon[36970]: pgmap v7144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:22:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12419.141328 secs 2024-07-26T10:22:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:52 smithi163 ceph-mon[36873]: pgmap v7144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:22:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12419.141328 secs 2024-07-26T10:22:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:52 smithi042 ceph-mon[32775]: pgmap v7144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:22:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:54 smithi164 ceph-mon[36970]: pgmap v7145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:22:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:54 smithi163 ceph-mon[36873]: pgmap v7145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:22:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:54 smithi042 ceph-mon[32775]: pgmap v7145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:22:57.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:56 smithi042 ceph-mon[32775]: pgmap v7146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:22:57.199 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12406.402397 secs 2024-07-26T10:22:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:56 smithi164 ceph-mon[36970]: pgmap v7146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:22:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12406.402397 secs 2024-07-26T10:22:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:56 smithi163 ceph-mon[36873]: pgmap v7146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:22:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12406.402397 secs 2024-07-26T10:22:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:57.842+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:22:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:57.842+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:22:57.845 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:57.842+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:22:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12424.141462 secs 2024-07-26T10:22:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12424.141462 secs 2024-07-26T10:22:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12424.141462 secs 2024-07-26T10:22:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:22:58 smithi164 ceph-mon[36970]: pgmap v7147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:22:58 smithi163 ceph-mon[36873]: pgmap v7147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:59.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:22:58 smithi042 ceph-mon[32775]: pgmap v7147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:22:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:59.807+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:22:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:59.807+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:22:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:59.807+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:22:59.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:22:59.807+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:23:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:00 smithi164 ceph-mon[36970]: pgmap v7148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:00 smithi163 ceph-mon[36873]: pgmap v7148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:00 smithi042 ceph-mon[32775]: pgmap v7148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12411.402513 secs 2024-07-26T10:23:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12429.141603 secs 2024-07-26T10:23:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12411.402513 secs 2024-07-26T10:23:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12429.141603 secs 2024-07-26T10:23:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12411.402513 secs 2024-07-26T10:23:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12429.141603 secs 2024-07-26T10:23:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:02 smithi164 ceph-mon[36970]: pgmap v7149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:02 smithi163 ceph-mon[36873]: pgmap v7149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:02 smithi042 ceph-mon[32775]: pgmap v7149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:04 smithi164 ceph-mon[36970]: pgmap v7150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:23:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:04 smithi163 ceph-mon[36873]: pgmap v7150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:23:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:04 smithi042 ceph-mon[32775]: pgmap v7150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:23:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:06 smithi164 ceph-mon[36970]: pgmap v7151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12416.402609 secs 2024-07-26T10:23:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12434.141720 secs 2024-07-26T10:23:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:06 smithi163 ceph-mon[36873]: pgmap v7151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12416.402609 secs 2024-07-26T10:23:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12434.141720 secs 2024-07-26T10:23:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:06 smithi042 ceph-mon[32775]: pgmap v7151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12416.402609 secs 2024-07-26T10:23:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12434.141720 secs 2024-07-26T10:23:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:08 smithi164 ceph-mon[36970]: pgmap v7152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:09.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:08 smithi163 ceph-mon[36873]: pgmap v7152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:08 smithi042 ceph-mon[32775]: pgmap v7152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:10 smithi163 ceph-mon[36873]: pgmap v7153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:10 smithi042 ceph-mon[32775]: pgmap v7153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:10 smithi164 ceph-mon[36970]: pgmap v7153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12421.402727 secs 2024-07-26T10:23:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12439.141789 secs 2024-07-26T10:23:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12421.402727 secs 2024-07-26T10:23:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12439.141789 secs 2024-07-26T10:23:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12421.402727 secs 2024-07-26T10:23:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12439.141789 secs 2024-07-26T10:23:12.846 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:23:12.844+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:23:12.846 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:23:12.844+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:23:12.846 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:23:12.844+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:23:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:12 smithi163 ceph-mon[36873]: pgmap v7154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:12 smithi042 ceph-mon[32775]: pgmap v7154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:12 smithi164 ceph-mon[36970]: pgmap v7154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:14 smithi163 ceph-mon[36873]: pgmap v7155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:23:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:14 smithi042 ceph-mon[32775]: pgmap v7155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:23:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:14 smithi164 ceph-mon[36970]: pgmap v7155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:23:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:16 smithi163 ceph-mon[36873]: pgmap v7156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12426.402821 secs 2024-07-26T10:23:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12444.141913 secs 2024-07-26T10:23:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:16 smithi042 ceph-mon[32775]: pgmap v7156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12426.402821 secs 2024-07-26T10:23:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12444.141913 secs 2024-07-26T10:23:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:16 smithi164 ceph-mon[36970]: pgmap v7156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12426.402821 secs 2024-07-26T10:23:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12444.141913 secs 2024-07-26T10:23:19.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:18 smithi163 ceph-mon[36873]: pgmap v7157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:18 smithi042 ceph-mon[32775]: pgmap v7157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:18 smithi164 ceph-mon[36970]: pgmap v7157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:20 smithi163 ceph-mon[36873]: pgmap v7158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:20 smithi042 ceph-mon[32775]: pgmap v7158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:20 smithi164 ceph-mon[36970]: pgmap v7158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12431.402920 secs 2024-07-26T10:23:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12449.141973 secs 2024-07-26T10:23:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12431.402920 secs 2024-07-26T10:23:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12449.141973 secs 2024-07-26T10:23:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12431.402920 secs 2024-07-26T10:23:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12449.141973 secs 2024-07-26T10:23:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:23 smithi163 ceph-mon[36873]: pgmap v7159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:23 smithi042 ceph-mon[32775]: pgmap v7159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:23 smithi164 ceph-mon[36970]: pgmap v7159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:25.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:25 smithi042 ceph-mon[32775]: pgmap v7160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:23:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:25 smithi164 ceph-mon[36970]: pgmap v7160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:23:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:25 smithi163 ceph-mon[36873]: pgmap v7160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:23:27.276 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:27 smithi164 ceph-mon[36970]: pgmap v7161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:27.276 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12436.403047 secs 2024-07-26T10:23:27.276 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12454.142092 secs 2024-07-26T10:23:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:27 smithi042 ceph-mon[32775]: pgmap v7161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12436.403047 secs 2024-07-26T10:23:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12454.142092 secs 2024-07-26T10:23:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:27 smithi163 ceph-mon[36873]: pgmap v7161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12436.403047 secs 2024-07-26T10:23:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12454.142092 secs 2024-07-26T10:23:27.846 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:23:27.845+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:23:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:23:27.845+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:23:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:23:27.845+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:23:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:29 smithi042 ceph-mon[32775]: pgmap v7162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:29 smithi164 ceph-mon[36970]: pgmap v7162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:29 smithi163 ceph-mon[36873]: pgmap v7162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:31 smithi042 ceph-mon[32775]: pgmap v7163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:31 smithi164 ceph-mon[36970]: pgmap v7163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:31 smithi163 ceph-mon[36873]: pgmap v7163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12441.403190 secs 2024-07-26T10:23:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12459.142211 secs 2024-07-26T10:23:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12441.403190 secs 2024-07-26T10:23:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12459.142211 secs 2024-07-26T10:23:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12441.403190 secs 2024-07-26T10:23:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12459.142211 secs 2024-07-26T10:23:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:33 smithi042 ceph-mon[32775]: pgmap v7164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:33 smithi164 ceph-mon[36970]: pgmap v7164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:33 smithi163 ceph-mon[36873]: pgmap v7164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:35 smithi042 ceph-mon[32775]: pgmap v7165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:23:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:35 smithi164 ceph-mon[36970]: pgmap v7165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:23:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:35 smithi163 ceph-mon[36873]: pgmap v7165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:23:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:37 smithi042 ceph-mon[32775]: pgmap v7166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12446.403318 secs 2024-07-26T10:23:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12464.142332 secs 2024-07-26T10:23:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:37 smithi164 ceph-mon[36970]: pgmap v7166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12446.403318 secs 2024-07-26T10:23:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12464.142332 secs 2024-07-26T10:23:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:37 smithi163 ceph-mon[36873]: pgmap v7166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12446.403318 secs 2024-07-26T10:23:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12464.142332 secs 2024-07-26T10:23:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:39 smithi042 ceph-mon[32775]: pgmap v7167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:39 smithi164 ceph-mon[36970]: pgmap v7167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:39 smithi163 ceph-mon[36873]: pgmap v7167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:41 smithi042 ceph-mon[32775]: pgmap v7168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:41 smithi164 ceph-mon[36970]: pgmap v7168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:41 smithi163 ceph-mon[36873]: pgmap v7168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12451.403448 secs 2024-07-26T10:23:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12469.142460 secs 2024-07-26T10:23:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12451.403448 secs 2024-07-26T10:23:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12469.142460 secs 2024-07-26T10:23:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12451.403448 secs 2024-07-26T10:23:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12469.142460 secs 2024-07-26T10:23:42.850 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:23:42.846+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:23:42.850 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:23:42.846+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:23:42.850 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:23:42.846+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:23:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:43 smithi042 ceph-mon[32775]: pgmap v7169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:43 smithi164 ceph-mon[36970]: pgmap v7169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:43 smithi163 ceph-mon[36873]: pgmap v7169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:23:45.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:45 smithi042 ceph-mon[32775]: pgmap v7170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:23:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:45 smithi164 ceph-mon[36970]: pgmap v7170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:23:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:45 smithi163 ceph-mon[36873]: pgmap v7170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:23:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:23:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:23:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:46.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:46.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:23:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:47 smithi042 ceph-mon[32775]: pgmap v7171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12456.403522 secs 2024-07-26T10:23:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12474.142532 secs 2024-07-26T10:23:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:23:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:23:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:47.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:47 smithi163 ceph-mon[36873]: pgmap v7171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:47.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12456.403522 secs 2024-07-26T10:23:47.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12474.142532 secs 2024-07-26T10:23:47.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:23:47.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:23:47.410 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:47 smithi164 ceph-mon[36970]: pgmap v7171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:23:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12456.403522 secs 2024-07-26T10:23:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12474.142532 secs 2024-07-26T10:23:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:23:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:23:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:23:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:48 smithi042 ceph-mon[32775]: pgmap v7172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:23:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:48 smithi042 ceph-mon[32775]: pgmap v7173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:23:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:48 smithi164 ceph-mon[36970]: pgmap v7172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:23:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:48 smithi164 ceph-mon[36970]: pgmap v7173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:23:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:48 smithi163 ceph-mon[36873]: pgmap v7172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:23:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:48 smithi163 ceph-mon[36873]: pgmap v7173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:23:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:50 smithi042 ceph-mon[32775]: pgmap v7174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T10:23:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:50 smithi164 ceph-mon[36970]: pgmap v7174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T10:23:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:50 smithi163 ceph-mon[36873]: pgmap v7174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T10:23:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:52 smithi042 ceph-mon[32775]: pgmap v7175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:23:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12461.403646 secs 2024-07-26T10:23:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12479.142652 secs 2024-07-26T10:23:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:52 smithi164 ceph-mon[36970]: pgmap v7175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:23:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12461.403646 secs 2024-07-26T10:23:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12479.142652 secs 2024-07-26T10:23:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:52 smithi163 ceph-mon[36873]: pgmap v7175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:23:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12461.403646 secs 2024-07-26T10:23:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12479.142652 secs 2024-07-26T10:23:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:54 smithi042 ceph-mon[32775]: pgmap v7176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:23:54.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:54 smithi164 ceph-mon[36970]: pgmap v7176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:23:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:54 smithi163 ceph-mon[36873]: pgmap v7176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:23:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:56 smithi042 ceph-mon[32775]: pgmap v7177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:23:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:56 smithi164 ceph-mon[36970]: pgmap v7177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:23:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:56 smithi163 ceph-mon[36873]: pgmap v7177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:23:57.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12466.403783 secs 2024-07-26T10:23:57.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12484.142798 secs 2024-07-26T10:23:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12466.403783 secs 2024-07-26T10:23:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12484.142798 secs 2024-07-26T10:23:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12466.403783 secs 2024-07-26T10:23:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12484.142798 secs 2024-07-26T10:23:57.854 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:23:57.852+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:23:57.854 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:23:57.852+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:23:57.854 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:23:57.852+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:23:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:23:58 smithi042 ceph-mon[32775]: pgmap v7178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:23:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:23:58 smithi164 ceph-mon[36970]: pgmap v7178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:23:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:23:58 smithi163 ceph-mon[36873]: pgmap v7178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:24:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:00 smithi042 ceph-mon[32775]: pgmap v7179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:24:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:00 smithi164 ceph-mon[36970]: pgmap v7179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:24:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:00 smithi163 ceph-mon[36873]: pgmap v7179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:24:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:02 smithi042 ceph-mon[32775]: pgmap v7180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12471.403879 secs 2024-07-26T10:24:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12489.142962 secs 2024-07-26T10:24:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:02 smithi164 ceph-mon[36970]: pgmap v7180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12471.403879 secs 2024-07-26T10:24:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12489.142962 secs 2024-07-26T10:24:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:02 smithi163 ceph-mon[36873]: pgmap v7180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12471.403879 secs 2024-07-26T10:24:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12489.142962 secs 2024-07-26T10:24:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:04 smithi042 ceph-mon[32775]: pgmap v7181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:04 smithi164 ceph-mon[36970]: pgmap v7181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:04 smithi163 ceph-mon[36873]: pgmap v7181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:06 smithi042 ceph-mon[32775]: pgmap v7182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:06 smithi164 ceph-mon[36970]: pgmap v7182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:06 smithi163 ceph-mon[36873]: pgmap v7182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12476.403980 secs 2024-07-26T10:24:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12494.143086 secs 2024-07-26T10:24:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12476.403980 secs 2024-07-26T10:24:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12494.143086 secs 2024-07-26T10:24:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12476.403980 secs 2024-07-26T10:24:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12494.143086 secs 2024-07-26T10:24:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:08 smithi042 ceph-mon[32775]: pgmap v7183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:08 smithi164 ceph-mon[36970]: pgmap v7183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:08 smithi163 ceph-mon[36873]: pgmap v7183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:10.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:10 smithi042 ceph-mon[32775]: pgmap v7184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:24:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:10 smithi164 ceph-mon[36970]: pgmap v7184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:24:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:10 smithi163 ceph-mon[36873]: pgmap v7184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:24:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:12 smithi042 ceph-mon[32775]: pgmap v7185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12481.404078 secs 2024-07-26T10:24:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12499.143207 secs 2024-07-26T10:24:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:12 smithi164 ceph-mon[36970]: pgmap v7185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:12.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12481.404078 secs 2024-07-26T10:24:12.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12499.143207 secs 2024-07-26T10:24:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:12 smithi163 ceph-mon[36873]: pgmap v7185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12481.404078 secs 2024-07-26T10:24:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12499.143207 secs 2024-07-26T10:24:12.854 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:24:12.853+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:24:12.854 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:24:12.853+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:24:12.854 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:24:12.853+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:24:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:14 smithi042 ceph-mon[32775]: pgmap v7186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:14 smithi164 ceph-mon[36970]: pgmap v7186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:14 smithi163 ceph-mon[36873]: pgmap v7186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:16 smithi042 ceph-mon[32775]: pgmap v7187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:16 smithi164 ceph-mon[36970]: pgmap v7187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:16 smithi163 ceph-mon[36873]: pgmap v7187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12486.404228 secs 2024-07-26T10:24:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12504.143282 secs 2024-07-26T10:24:17.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12486.404228 secs 2024-07-26T10:24:17.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12504.143282 secs 2024-07-26T10:24:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12486.404228 secs 2024-07-26T10:24:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12504.143282 secs 2024-07-26T10:24:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:18 smithi042 ceph-mon[32775]: pgmap v7188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:18 smithi164 ceph-mon[36970]: pgmap v7188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:18 smithi163 ceph-mon[36873]: pgmap v7188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:20 smithi042 ceph-mon[32775]: pgmap v7189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:24:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:20 smithi164 ceph-mon[36970]: pgmap v7189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:24:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:20 smithi163 ceph-mon[36873]: pgmap v7189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:24:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:22 smithi042 ceph-mon[32775]: pgmap v7190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12491.404338 secs 2024-07-26T10:24:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12509.143412 secs 2024-07-26T10:24:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:22 smithi164 ceph-mon[36970]: pgmap v7190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12491.404338 secs 2024-07-26T10:24:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12509.143412 secs 2024-07-26T10:24:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:22 smithi163 ceph-mon[36873]: pgmap v7190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12491.404338 secs 2024-07-26T10:24:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12509.143412 secs 2024-07-26T10:24:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:24 smithi042 ceph-mon[32775]: pgmap v7191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:24 smithi164 ceph-mon[36970]: pgmap v7191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:24 smithi163 ceph-mon[36873]: pgmap v7191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:26 smithi042 ceph-mon[32775]: pgmap v7192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:26 smithi164 ceph-mon[36970]: pgmap v7192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:26 smithi163 ceph-mon[36873]: pgmap v7192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12496.404464 secs 2024-07-26T10:24:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12514.143554 secs 2024-07-26T10:24:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12496.404464 secs 2024-07-26T10:24:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12514.143554 secs 2024-07-26T10:24:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12496.404464 secs 2024-07-26T10:24:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12514.143554 secs 2024-07-26T10:24:27.860 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:24:27.857+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:24:27.860 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:24:27.858+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:24:27.860 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:24:27.858+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:24:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:28 smithi042 ceph-mon[32775]: pgmap v7193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:28 smithi164 ceph-mon[36970]: pgmap v7193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:28 smithi163 ceph-mon[36873]: pgmap v7193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:30 smithi164 ceph-mon[36970]: pgmap v7194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:24:30.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:30 smithi163 ceph-mon[36873]: pgmap v7194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:24:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:30 smithi042 ceph-mon[32775]: pgmap v7194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:24:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:32 smithi164 ceph-mon[36970]: pgmap v7195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12501.404585 secs 2024-07-26T10:24:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12519.143646 secs 2024-07-26T10:24:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:32 smithi163 ceph-mon[36873]: pgmap v7195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12501.404585 secs 2024-07-26T10:24:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12519.143646 secs 2024-07-26T10:24:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:32 smithi042 ceph-mon[32775]: pgmap v7195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12501.404585 secs 2024-07-26T10:24:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12519.143646 secs 2024-07-26T10:24:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:34 smithi164 ceph-mon[36970]: pgmap v7196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:34 smithi163 ceph-mon[36873]: pgmap v7196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:34 smithi042 ceph-mon[32775]: pgmap v7196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:36 smithi164 ceph-mon[36970]: pgmap v7197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:36 smithi163 ceph-mon[36873]: pgmap v7197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:36 smithi042 ceph-mon[32775]: pgmap v7197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12506.404708 secs 2024-07-26T10:24:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12524.143746 secs 2024-07-26T10:24:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12506.404708 secs 2024-07-26T10:24:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12524.143746 secs 2024-07-26T10:24:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12506.404708 secs 2024-07-26T10:24:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12524.143746 secs 2024-07-26T10:24:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:38 smithi164 ceph-mon[36970]: pgmap v7198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:38 smithi163 ceph-mon[36873]: pgmap v7198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:38 smithi042 ceph-mon[32775]: pgmap v7198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:40 smithi164 ceph-mon[36970]: pgmap v7199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:24:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:40 smithi163 ceph-mon[36873]: pgmap v7199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:24:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:40 smithi042 ceph-mon[32775]: pgmap v7199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:24:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:42 smithi164 ceph-mon[36970]: pgmap v7200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12511.404829 secs 2024-07-26T10:24:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12529.143873 secs 2024-07-26T10:24:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:42 smithi163 ceph-mon[36873]: pgmap v7200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12511.404829 secs 2024-07-26T10:24:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12529.143873 secs 2024-07-26T10:24:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:42 smithi042 ceph-mon[32775]: pgmap v7200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12511.404829 secs 2024-07-26T10:24:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12529.143873 secs 2024-07-26T10:24:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:24:42.859+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:24:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:24:42.859+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:24:42.861 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:24:42.859+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:24:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:44 smithi164 ceph-mon[36970]: pgmap v7201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:44 smithi163 ceph-mon[36873]: pgmap v7201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:44 smithi042 ceph-mon[32775]: pgmap v7201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:46 smithi164 ceph-mon[36970]: pgmap v7202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:46 smithi163 ceph-mon[36873]: pgmap v7202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:46 smithi042 ceph-mon[32775]: pgmap v7202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:24:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12516.404958 secs 2024-07-26T10:24:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12534.143998 secs 2024-07-26T10:24:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:24:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:24:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:24:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:24:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12516.404958 secs 2024-07-26T10:24:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12534.143998 secs 2024-07-26T10:24:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:24:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:24:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:24:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:24:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12516.404958 secs 2024-07-26T10:24:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12534.143998 secs 2024-07-26T10:24:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:24:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:24:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:24:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:24:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:48 smithi164 ceph-mon[36970]: pgmap v7203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:48.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:48 smithi164 ceph-mon[36970]: pgmap v7204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:24:48.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:48 smithi164 ceph-mon[36970]: pgmap v7205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:24:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:48 smithi163 ceph-mon[36873]: pgmap v7203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:48 smithi163 ceph-mon[36873]: pgmap v7204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:24:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:48 smithi163 ceph-mon[36873]: pgmap v7205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:24:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:48 smithi042 ceph-mon[32775]: pgmap v7203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:24:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:48 smithi042 ceph-mon[32775]: pgmap v7204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:24:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:48 smithi042 ceph-mon[32775]: pgmap v7205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:24:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:50 smithi164 ceph-mon[36970]: pgmap v7206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:24:50.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:50 smithi163 ceph-mon[36873]: pgmap v7206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:24:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:50 smithi042 ceph-mon[32775]: pgmap v7206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:24:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12521.405060 secs 2024-07-26T10:24:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:52 smithi164 ceph-mon[36970]: pgmap v7207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:24:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12539.144112 secs 2024-07-26T10:24:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12521.405060 secs 2024-07-26T10:24:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:52 smithi163 ceph-mon[36873]: pgmap v7207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:24:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12539.144112 secs 2024-07-26T10:24:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12521.405060 secs 2024-07-26T10:24:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:52 smithi042 ceph-mon[32775]: pgmap v7207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:24:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12539.144112 secs 2024-07-26T10:24:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:53 smithi042 ceph-mon[32775]: pgmap v7208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:24:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:53 smithi164 ceph-mon[36970]: pgmap v7208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:24:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:53 smithi163 ceph-mon[36873]: pgmap v7208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:24:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:56 smithi042 ceph-mon[32775]: pgmap v7209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:24:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:56 smithi164 ceph-mon[36970]: pgmap v7209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:24:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:56 smithi163 ceph-mon[36873]: pgmap v7209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:24:57.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12526.405209 secs 2024-07-26T10:24:57.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12544.144241 secs 2024-07-26T10:24:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12526.405209 secs 2024-07-26T10:24:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12544.144241 secs 2024-07-26T10:24:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12526.405209 secs 2024-07-26T10:24:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12544.144241 secs 2024-07-26T10:24:57.864 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:24:57.862+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:24:57.864 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:24:57.862+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:24:57.864 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:24:57.862+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:24:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:24:58 smithi042 ceph-mon[32775]: pgmap v7210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:24:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:24:58 smithi164 ceph-mon[36970]: pgmap v7210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:24:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:24:58 smithi163 ceph-mon[36873]: pgmap v7210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:25:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:00 smithi042 ceph-mon[32775]: pgmap v7211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:25:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:00 smithi164 ceph-mon[36970]: pgmap v7211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:25:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:00 smithi163 ceph-mon[36873]: pgmap v7211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:25:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12531.405285 secs 2024-07-26T10:25:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:02 smithi042 ceph-mon[32775]: pgmap v7212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12549.144364 secs 2024-07-26T10:25:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12531.405285 secs 2024-07-26T10:25:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:02 smithi164 ceph-mon[36970]: pgmap v7212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12549.144364 secs 2024-07-26T10:25:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12531.405285 secs 2024-07-26T10:25:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:02 smithi163 ceph-mon[36873]: pgmap v7212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12549.144364 secs 2024-07-26T10:25:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:04 smithi042 ceph-mon[32775]: pgmap v7213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:04 smithi164 ceph-mon[36970]: pgmap v7213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:04 smithi163 ceph-mon[36873]: pgmap v7213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:06 smithi042 ceph-mon[32775]: pgmap v7214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:06 smithi164 ceph-mon[36970]: pgmap v7214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:06 smithi163 ceph-mon[36873]: pgmap v7214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12536.405408 secs 2024-07-26T10:25:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12554.144429 secs 2024-07-26T10:25:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12536.405408 secs 2024-07-26T10:25:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12554.144429 secs 2024-07-26T10:25:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12536.405408 secs 2024-07-26T10:25:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12554.144429 secs 2024-07-26T10:25:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:08 smithi042 ceph-mon[32775]: pgmap v7215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:08 smithi164 ceph-mon[36970]: pgmap v7215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:08 smithi163 ceph-mon[36873]: pgmap v7215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:10.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:10 smithi042 ceph-mon[32775]: pgmap v7216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:25:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:10 smithi164 ceph-mon[36970]: pgmap v7216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:25:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:10 smithi163 ceph-mon[36873]: pgmap v7216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:25:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12541.405528 secs 2024-07-26T10:25:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:12 smithi042 ceph-mon[32775]: pgmap v7217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12559.144575 secs 2024-07-26T10:25:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12541.405528 secs 2024-07-26T10:25:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:12 smithi164 ceph-mon[36970]: pgmap v7217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12559.144575 secs 2024-07-26T10:25:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12541.405528 secs 2024-07-26T10:25:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:12 smithi163 ceph-mon[36873]: pgmap v7217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12559.144575 secs 2024-07-26T10:25:12.865 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:12.863+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:25:12.865 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:12.863+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:25:12.865 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:12.863+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:25:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:14 smithi042 ceph-mon[32775]: pgmap v7218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:14 smithi164 ceph-mon[36970]: pgmap v7218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:14.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:14 smithi163 ceph-mon[36873]: pgmap v7218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:16 smithi042 ceph-mon[32775]: pgmap v7219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:16 smithi164 ceph-mon[36970]: pgmap v7219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:16 smithi163 ceph-mon[36873]: pgmap v7219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12546.405644 secs 2024-07-26T10:25:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12564.144730 secs 2024-07-26T10:25:17.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12546.405644 secs 2024-07-26T10:25:17.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12564.144730 secs 2024-07-26T10:25:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12546.405644 secs 2024-07-26T10:25:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12564.144730 secs 2024-07-26T10:25:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:18 smithi042 ceph-mon[32775]: pgmap v7220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:18 smithi164 ceph-mon[36970]: pgmap v7220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:18 smithi163 ceph-mon[36873]: pgmap v7220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:20 smithi042 ceph-mon[32775]: pgmap v7221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:25:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:20 smithi164 ceph-mon[36970]: pgmap v7221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:25:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:20 smithi163 ceph-mon[36873]: pgmap v7221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:25:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12551.405785 secs 2024-07-26T10:25:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:22 smithi042 ceph-mon[32775]: pgmap v7222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12569.144851 secs 2024-07-26T10:25:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12551.405785 secs 2024-07-26T10:25:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:22 smithi164 ceph-mon[36970]: pgmap v7222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12569.144851 secs 2024-07-26T10:25:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12551.405785 secs 2024-07-26T10:25:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:22 smithi163 ceph-mon[36873]: pgmap v7222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12569.144851 secs 2024-07-26T10:25:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:24 smithi042 ceph-mon[32775]: pgmap v7223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:24 smithi164 ceph-mon[36970]: pgmap v7223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:24 smithi163 ceph-mon[36873]: pgmap v7223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:26 smithi042 ceph-mon[32775]: pgmap v7224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:26 smithi164 ceph-mon[36970]: pgmap v7224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:26 smithi163 ceph-mon[36873]: pgmap v7224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:27.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12556.405929 secs 2024-07-26T10:25:27.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12574.144973 secs 2024-07-26T10:25:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12556.405929 secs 2024-07-26T10:25:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12574.144973 secs 2024-07-26T10:25:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12556.405929 secs 2024-07-26T10:25:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12574.144973 secs 2024-07-26T10:25:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:27.869+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:25:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:27.869+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:25:27.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:27.869+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:25:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:28 smithi042 ceph-mon[32775]: pgmap v7225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:28 smithi164 ceph-mon[36970]: pgmap v7225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:28 smithi163 ceph-mon[36873]: pgmap v7225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:30.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:30 smithi042 ceph-mon[32775]: pgmap v7226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:25:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:30 smithi164 ceph-mon[36970]: pgmap v7226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:25:30.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:30 smithi163 ceph-mon[36873]: pgmap v7226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:25:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12561.406050 secs 2024-07-26T10:25:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:32 smithi042 ceph-mon[32775]: pgmap v7227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12579.145119 secs 2024-07-26T10:25:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12561.406050 secs 2024-07-26T10:25:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:32 smithi164 ceph-mon[36970]: pgmap v7227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12579.145119 secs 2024-07-26T10:25:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12561.406050 secs 2024-07-26T10:25:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:32 smithi163 ceph-mon[36873]: pgmap v7227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12579.145119 secs 2024-07-26T10:25:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:34 smithi042 ceph-mon[32775]: pgmap v7228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:34 smithi164 ceph-mon[36970]: pgmap v7228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:34 smithi163 ceph-mon[36873]: pgmap v7228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:36 smithi042 ceph-mon[32775]: pgmap v7229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:36 smithi164 ceph-mon[36970]: pgmap v7229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:36 smithi163 ceph-mon[36873]: pgmap v7229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12566.406163 secs 2024-07-26T10:25:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12584.145247 secs 2024-07-26T10:25:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12566.406163 secs 2024-07-26T10:25:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12584.145247 secs 2024-07-26T10:25:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12566.406163 secs 2024-07-26T10:25:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12584.145247 secs 2024-07-26T10:25:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:38 smithi042 ceph-mon[32775]: pgmap v7230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:38 smithi164 ceph-mon[36970]: pgmap v7230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:38 smithi163 ceph-mon[36873]: pgmap v7230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:40.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:40 smithi042 ceph-mon[32775]: pgmap v7231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:25:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:40 smithi164 ceph-mon[36970]: pgmap v7231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:25:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:40 smithi163 ceph-mon[36873]: pgmap v7231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:25:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12571.406249 secs 2024-07-26T10:25:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:42 smithi042 ceph-mon[32775]: pgmap v7232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12589.145372 secs 2024-07-26T10:25:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12571.406249 secs 2024-07-26T10:25:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:42 smithi164 ceph-mon[36970]: pgmap v7232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12589.145372 secs 2024-07-26T10:25:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12571.406249 secs 2024-07-26T10:25:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:42 smithi163 ceph-mon[36873]: pgmap v7232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12589.145372 secs 2024-07-26T10:25:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:42.870+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:25:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:42.870+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:25:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:42.870+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:25:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:44 smithi042 ceph-mon[32775]: pgmap v7233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:44 smithi164 ceph-mon[36970]: pgmap v7233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:44 smithi163 ceph-mon[36873]: pgmap v7233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:46 smithi164 ceph-mon[36970]: pgmap v7234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:46 smithi163 ceph-mon[36873]: pgmap v7234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:46 smithi042 ceph-mon[32775]: pgmap v7234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:25:47.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12576.406370 secs 2024-07-26T10:25:47.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12594.145510 secs 2024-07-26T10:25:47.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:25:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12576.406370 secs 2024-07-26T10:25:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12594.145510 secs 2024-07-26T10:25:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:25:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12576.406370 secs 2024-07-26T10:25:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12594.145510 secs 2024-07-26T10:25:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:25:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:48 smithi164 ceph-mon[36970]: pgmap v7235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:25:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:25:48.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:25:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:48 smithi163 ceph-mon[36873]: pgmap v7235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:25:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:25:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:25:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:48 smithi042 ceph-mon[32775]: pgmap v7235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:25:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:25:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:25:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:25:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:49 smithi164 ceph-mon[36970]: pgmap v7236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:25:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:49 smithi164 ceph-mon[36970]: pgmap v7237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:25:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:49 smithi163 ceph-mon[36873]: pgmap v7236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:25:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:49 smithi163 ceph-mon[36873]: pgmap v7237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:25:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:49 smithi042 ceph-mon[32775]: pgmap v7236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:25:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:49 smithi042 ceph-mon[32775]: pgmap v7237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:25:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:51 smithi042 ceph-mon[32775]: pgmap v7238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:25:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:51 smithi164 ceph-mon[36970]: pgmap v7238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:25:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:51 smithi163 ceph-mon[36873]: pgmap v7238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:25:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12581.406468 secs 2024-07-26T10:25:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12599.145623 secs 2024-07-26T10:25:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12581.406468 secs 2024-07-26T10:25:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12599.145623 secs 2024-07-26T10:25:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12581.406468 secs 2024-07-26T10:25:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12599.145623 secs 2024-07-26T10:25:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:53 smithi164 ceph-mon[36970]: pgmap v7239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:25:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:53 smithi163 ceph-mon[36873]: pgmap v7239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:25:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:53 smithi042 ceph-mon[32775]: pgmap v7239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:25:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:55 smithi164 ceph-mon[36970]: pgmap v7240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:25:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:55 smithi163 ceph-mon[36873]: pgmap v7240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:25:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:55 smithi042 ceph-mon[32775]: pgmap v7240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:25:57.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:57 smithi042 ceph-mon[32775]: pgmap v7241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:25:57.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12586.406591 secs 2024-07-26T10:25:57.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12604.145743 secs 2024-07-26T10:25:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:57 smithi164 ceph-mon[36970]: pgmap v7241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:25:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12586.406591 secs 2024-07-26T10:25:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12604.145743 secs 2024-07-26T10:25:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:57 smithi163 ceph-mon[36873]: pgmap v7241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:25:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12586.406591 secs 2024-07-26T10:25:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12604.145743 secs 2024-07-26T10:25:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:57.875+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:25:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:57.875+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:25:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:57.875+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:25:57.881 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:57.879+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T10:25:57.881 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:57.879+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T10:25:57.881 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:57.879+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff678009a60 tx=0x7ff67803e040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:25:57.881 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:57.879+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff678009a60 tx=0x7ff67803e040 comp rx=0 tx=0).stop 2024-07-26T10:25:57.881 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:57.879+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T10:25:57.881 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:57.879+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T10:25:57.882 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:57.879+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:25:57.882 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:57.879+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:25:57.882 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:57.879+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:25:57.882 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:57.880+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff6780317b0 tx=0x7ff678061040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:25:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:25:59 smithi164 ceph-mon[36970]: pgmap v7242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:25:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:25:59 smithi163 ceph-mon[36873]: pgmap v7242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:25:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:25:59 smithi042 ceph-mon[32775]: pgmap v7242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:25:59.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:59.837+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:25:59.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:59.837+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:25:59.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:59.837+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:25:59.838 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:25:59.837+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:26:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:01 smithi164 ceph-mon[36970]: pgmap v7243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:01 smithi163 ceph-mon[36873]: pgmap v7243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:01 smithi042 ceph-mon[32775]: pgmap v7243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12591.406717 secs 2024-07-26T10:26:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12609.145871 secs 2024-07-26T10:26:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12591.406717 secs 2024-07-26T10:26:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12609.145871 secs 2024-07-26T10:26:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12591.406717 secs 2024-07-26T10:26:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12609.145871 secs 2024-07-26T10:26:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:03 smithi164 ceph-mon[36970]: pgmap v7244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:03 smithi163 ceph-mon[36873]: pgmap v7244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:03 smithi042 ceph-mon[32775]: pgmap v7244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:05 smithi164 ceph-mon[36970]: pgmap v7245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:26:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:05 smithi163 ceph-mon[36873]: pgmap v7245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:26:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:05 smithi042 ceph-mon[32775]: pgmap v7245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:26:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:07 smithi042 ceph-mon[32775]: pgmap v7246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12596.406826 secs 2024-07-26T10:26:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12614.145996 secs 2024-07-26T10:26:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:07 smithi164 ceph-mon[36970]: pgmap v7246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12596.406826 secs 2024-07-26T10:26:07.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12614.145996 secs 2024-07-26T10:26:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:07 smithi163 ceph-mon[36873]: pgmap v7246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12596.406826 secs 2024-07-26T10:26:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12614.145996 secs 2024-07-26T10:26:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:09 smithi164 ceph-mon[36970]: pgmap v7247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:09 smithi163 ceph-mon[36873]: pgmap v7247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:09 smithi042 ceph-mon[32775]: pgmap v7247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:11 smithi164 ceph-mon[36970]: pgmap v7248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:11 smithi163 ceph-mon[36873]: pgmap v7248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:11 smithi042 ceph-mon[32775]: pgmap v7248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12601.406890 secs 2024-07-26T10:26:12.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12619.146065 secs 2024-07-26T10:26:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12601.406890 secs 2024-07-26T10:26:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12619.146065 secs 2024-07-26T10:26:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12601.406890 secs 2024-07-26T10:26:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12619.146065 secs 2024-07-26T10:26:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:26:12.876+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:26:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:26:12.876+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:26:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:26:12.876+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:26:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:13 smithi164 ceph-mon[36970]: pgmap v7249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:13 smithi163 ceph-mon[36873]: pgmap v7249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:13 smithi042 ceph-mon[32775]: pgmap v7249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:15 smithi164 ceph-mon[36970]: pgmap v7250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:26:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:15 smithi163 ceph-mon[36873]: pgmap v7250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:26:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:15 smithi042 ceph-mon[32775]: pgmap v7250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:26:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:17 smithi164 ceph-mon[36970]: pgmap v7251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12606.407017 secs 2024-07-26T10:26:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12624.146156 secs 2024-07-26T10:26:17.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:17 smithi163 ceph-mon[36873]: pgmap v7251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:17.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12606.407017 secs 2024-07-26T10:26:17.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12624.146156 secs 2024-07-26T10:26:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:17 smithi042 ceph-mon[32775]: pgmap v7251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12606.407017 secs 2024-07-26T10:26:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12624.146156 secs 2024-07-26T10:26:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:19 smithi163 ceph-mon[36873]: pgmap v7252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:19 smithi042 ceph-mon[32775]: pgmap v7252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:19 smithi164 ceph-mon[36970]: pgmap v7252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:21 smithi163 ceph-mon[36873]: pgmap v7253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:21 smithi042 ceph-mon[32775]: pgmap v7253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:21 smithi164 ceph-mon[36970]: pgmap v7253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12611.407138 secs 2024-07-26T10:26:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12629.146282 secs 2024-07-26T10:26:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12611.407138 secs 2024-07-26T10:26:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12629.146282 secs 2024-07-26T10:26:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12611.407138 secs 2024-07-26T10:26:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12629.146282 secs 2024-07-26T10:26:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:23 smithi163 ceph-mon[36873]: pgmap v7254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:23 smithi042 ceph-mon[32775]: pgmap v7254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:23 smithi164 ceph-mon[36970]: pgmap v7254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:25 smithi163 ceph-mon[36873]: pgmap v7255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:26:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:25 smithi042 ceph-mon[32775]: pgmap v7255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:26:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:25 smithi164 ceph-mon[36970]: pgmap v7255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:26:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:27 smithi163 ceph-mon[36873]: pgmap v7256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12616.407267 secs 2024-07-26T10:26:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12634.146411 secs 2024-07-26T10:26:27.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:27 smithi164 ceph-mon[36970]: pgmap v7256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:27.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12616.407267 secs 2024-07-26T10:26:27.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12634.146411 secs 2024-07-26T10:26:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:27 smithi042 ceph-mon[32775]: pgmap v7256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12616.407267 secs 2024-07-26T10:26:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12634.146411 secs 2024-07-26T10:26:27.883 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:26:27.881+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:26:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:26:27.881+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:26:27.884 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:26:27.881+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:26:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:29 smithi163 ceph-mon[36873]: pgmap v7257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:29 smithi042 ceph-mon[32775]: pgmap v7257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:29 smithi164 ceph-mon[36970]: pgmap v7257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:31 smithi163 ceph-mon[36873]: pgmap v7258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:31 smithi042 ceph-mon[32775]: pgmap v7258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:31 smithi164 ceph-mon[36970]: pgmap v7258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12621.407388 secs 2024-07-26T10:26:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12639.146532 secs 2024-07-26T10:26:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12621.407388 secs 2024-07-26T10:26:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12639.146532 secs 2024-07-26T10:26:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12621.407388 secs 2024-07-26T10:26:32.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12639.146532 secs 2024-07-26T10:26:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:33 smithi163 ceph-mon[36873]: pgmap v7259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:33 smithi042 ceph-mon[32775]: pgmap v7259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:33 smithi164 ceph-mon[36970]: pgmap v7259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:35 smithi042 ceph-mon[32775]: pgmap v7260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:26:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:35 smithi164 ceph-mon[36970]: pgmap v7260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:26:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:35 smithi163 ceph-mon[36873]: pgmap v7260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:26:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:37 smithi042 ceph-mon[32775]: pgmap v7261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12626.407503 secs 2024-07-26T10:26:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12644.146652 secs 2024-07-26T10:26:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:37 smithi164 ceph-mon[36970]: pgmap v7261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12626.407503 secs 2024-07-26T10:26:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12644.146652 secs 2024-07-26T10:26:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:37 smithi163 ceph-mon[36873]: pgmap v7261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12626.407503 secs 2024-07-26T10:26:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12644.146652 secs 2024-07-26T10:26:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:39 smithi042 ceph-mon[32775]: pgmap v7262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:39 smithi164 ceph-mon[36970]: pgmap v7262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:39 smithi163 ceph-mon[36873]: pgmap v7262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:41 smithi042 ceph-mon[32775]: pgmap v7263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:41 smithi164 ceph-mon[36970]: pgmap v7263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:41 smithi163 ceph-mon[36873]: pgmap v7263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12631.407620 secs 2024-07-26T10:26:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12649.146789 secs 2024-07-26T10:26:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12631.407620 secs 2024-07-26T10:26:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12649.146789 secs 2024-07-26T10:26:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12631.407620 secs 2024-07-26T10:26:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12649.146789 secs 2024-07-26T10:26:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:26:42.883+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:26:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:26:42.883+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:26:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:26:42.883+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:26:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:43 smithi042 ceph-mon[32775]: pgmap v7264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:43 smithi164 ceph-mon[36970]: pgmap v7264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:43 smithi163 ceph-mon[36873]: pgmap v7264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:45 smithi042 ceph-mon[32775]: pgmap v7265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:26:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:45 smithi164 ceph-mon[36970]: pgmap v7265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:26:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:45 smithi163 ceph-mon[36873]: pgmap v7265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:26:47.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:47 smithi163 ceph-mon[36873]: pgmap v7266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:47.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12636.407746 secs 2024-07-26T10:26:47.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12654.146856 secs 2024-07-26T10:26:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:47 smithi042 ceph-mon[32775]: pgmap v7266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12636.407746 secs 2024-07-26T10:26:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12654.146856 secs 2024-07-26T10:26:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:47 smithi164 ceph-mon[36970]: pgmap v7266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:26:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12636.407746 secs 2024-07-26T10:26:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12654.146856 secs 2024-07-26T10:26:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:48 smithi042 ceph-mon[32775]: pgmap v7267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:26:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:26:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:26:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:48 smithi042 ceph-mon[32775]: pgmap v7268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:26:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:48 smithi042 ceph-mon[32775]: pgmap v7269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:26:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:26:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:48 smithi164 ceph-mon[36970]: pgmap v7267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:26:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:26:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:26:48.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:48 smithi164 ceph-mon[36970]: pgmap v7268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:26:48.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:48 smithi164 ceph-mon[36970]: pgmap v7269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:26:48.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:26:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:48 smithi163 ceph-mon[36873]: pgmap v7267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:26:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:26:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:26:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:26:48.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:48 smithi163 ceph-mon[36873]: pgmap v7268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:26:48.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:48 smithi163 ceph-mon[36873]: pgmap v7269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:26:48.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:26:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:51 smithi164 ceph-mon[36970]: pgmap v7270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:26:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:51 smithi163 ceph-mon[36873]: pgmap v7270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:26:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:51 smithi042 ceph-mon[32775]: pgmap v7270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:26:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12641.407874 secs 2024-07-26T10:26:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12659.146992 secs 2024-07-26T10:26:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12641.407874 secs 2024-07-26T10:26:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12659.146992 secs 2024-07-26T10:26:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12641.407874 secs 2024-07-26T10:26:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12659.146992 secs 2024-07-26T10:26:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:53 smithi042 ceph-mon[32775]: pgmap v7271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:26:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:53 smithi164 ceph-mon[36970]: pgmap v7271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:26:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:53 smithi163 ceph-mon[36873]: pgmap v7271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:26:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:55 smithi164 ceph-mon[36970]: pgmap v7272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:26:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:55 smithi163 ceph-mon[36873]: pgmap v7272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:26:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:55 smithi042 ceph-mon[32775]: pgmap v7272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:26:57.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12646.407997 secs 2024-07-26T10:26:57.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:57 smithi042 ceph-mon[32775]: pgmap v7273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:26:57.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12664.147116 secs 2024-07-26T10:26:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12646.407997 secs 2024-07-26T10:26:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:57 smithi164 ceph-mon[36970]: pgmap v7273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:26:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12664.147116 secs 2024-07-26T10:26:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12646.407997 secs 2024-07-26T10:26:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:57 smithi163 ceph-mon[36873]: pgmap v7273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:26:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12664.147116 secs 2024-07-26T10:26:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:26:57.888+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:26:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:26:57.888+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:26:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:26:57.888+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:26:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:26:59 smithi164 ceph-mon[36970]: pgmap v7274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:26:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:26:59 smithi163 ceph-mon[36873]: pgmap v7274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:26:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:26:59 smithi042 ceph-mon[32775]: pgmap v7274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:27:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:01 smithi164 ceph-mon[36970]: pgmap v7275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:01 smithi163 ceph-mon[36873]: pgmap v7275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:01 smithi042 ceph-mon[32775]: pgmap v7275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12651.408122 secs 2024-07-26T10:27:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12669.147239 secs 2024-07-26T10:27:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12651.408122 secs 2024-07-26T10:27:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12669.147239 secs 2024-07-26T10:27:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12651.408122 secs 2024-07-26T10:27:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12669.147239 secs 2024-07-26T10:27:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:03 smithi164 ceph-mon[36970]: pgmap v7276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:03 smithi163 ceph-mon[36873]: pgmap v7276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:03 smithi042 ceph-mon[32775]: pgmap v7276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:05 smithi164 ceph-mon[36970]: pgmap v7277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:05 smithi163 ceph-mon[36873]: pgmap v7277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:05 smithi042 ceph-mon[32775]: pgmap v7277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12656.408259 secs 2024-07-26T10:27:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:07 smithi164 ceph-mon[36970]: pgmap v7278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12674.147359 secs 2024-07-26T10:27:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12656.408259 secs 2024-07-26T10:27:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:07 smithi163 ceph-mon[36873]: pgmap v7278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12674.147359 secs 2024-07-26T10:27:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12656.408259 secs 2024-07-26T10:27:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:07 smithi042 ceph-mon[32775]: pgmap v7278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12674.147359 secs 2024-07-26T10:27:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:09 smithi164 ceph-mon[36970]: pgmap v7279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:09 smithi163 ceph-mon[36873]: pgmap v7279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:09 smithi042 ceph-mon[32775]: pgmap v7279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:11 smithi164 ceph-mon[36970]: pgmap v7280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:11 smithi163 ceph-mon[36873]: pgmap v7280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:11 smithi042 ceph-mon[32775]: pgmap v7280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12661.408329 secs 2024-07-26T10:27:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12679.147484 secs 2024-07-26T10:27:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12661.408329 secs 2024-07-26T10:27:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12679.147484 secs 2024-07-26T10:27:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12661.408329 secs 2024-07-26T10:27:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12679.147484 secs 2024-07-26T10:27:12.891 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:27:12.889+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:27:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:27:12.889+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:27:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:27:12.889+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:27:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:13 smithi164 ceph-mon[36970]: pgmap v7281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:13 smithi163 ceph-mon[36873]: pgmap v7281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:13 smithi042 ceph-mon[32775]: pgmap v7281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:15 smithi164 ceph-mon[36970]: pgmap v7282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:15 smithi163 ceph-mon[36873]: pgmap v7282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:15 smithi042 ceph-mon[32775]: pgmap v7282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12666.408454 secs 2024-07-26T10:27:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:17 smithi164 ceph-mon[36970]: pgmap v7283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12684.147627 secs 2024-07-26T10:27:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12666.408454 secs 2024-07-26T10:27:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:17 smithi163 ceph-mon[36873]: pgmap v7283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12684.147627 secs 2024-07-26T10:27:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12666.408454 secs 2024-07-26T10:27:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:17 smithi042 ceph-mon[32775]: pgmap v7283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12684.147627 secs 2024-07-26T10:27:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:19 smithi163 ceph-mon[36873]: pgmap v7284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:19 smithi042 ceph-mon[32775]: pgmap v7284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:19 smithi164 ceph-mon[36970]: pgmap v7284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:21 smithi163 ceph-mon[36873]: pgmap v7285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:21 smithi042 ceph-mon[32775]: pgmap v7285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:21 smithi164 ceph-mon[36970]: pgmap v7285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12671.408575 secs 2024-07-26T10:27:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12689.147772 secs 2024-07-26T10:27:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12671.408575 secs 2024-07-26T10:27:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12689.147772 secs 2024-07-26T10:27:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12671.408575 secs 2024-07-26T10:27:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12689.147772 secs 2024-07-26T10:27:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:23 smithi163 ceph-mon[36873]: pgmap v7286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:23 smithi042 ceph-mon[32775]: pgmap v7286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:23 smithi164 ceph-mon[36970]: pgmap v7286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:25 smithi163 ceph-mon[36873]: pgmap v7287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:25 smithi042 ceph-mon[32775]: pgmap v7287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:25 smithi164 ceph-mon[36970]: pgmap v7287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12676.408698 secs 2024-07-26T10:27:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:27 smithi163 ceph-mon[36873]: pgmap v7288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12694.147919 secs 2024-07-26T10:27:27.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12676.408698 secs 2024-07-26T10:27:27.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:27 smithi164 ceph-mon[36970]: pgmap v7288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:27.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12694.147919 secs 2024-07-26T10:27:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12676.408698 secs 2024-07-26T10:27:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:27 smithi042 ceph-mon[32775]: pgmap v7288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12694.147919 secs 2024-07-26T10:27:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:27:27.894+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:27:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:27:27.894+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:27:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:27:27.894+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:27:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:29 smithi163 ceph-mon[36873]: pgmap v7289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:29 smithi042 ceph-mon[32775]: pgmap v7289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:29.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:29 smithi164 ceph-mon[36970]: pgmap v7289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:31 smithi163 ceph-mon[36873]: pgmap v7290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:31 smithi042 ceph-mon[32775]: pgmap v7290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:31 smithi164 ceph-mon[36970]: pgmap v7290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12681.408839 secs 2024-07-26T10:27:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12699.148077 secs 2024-07-26T10:27:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12681.408839 secs 2024-07-26T10:27:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12699.148077 secs 2024-07-26T10:27:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12681.408839 secs 2024-07-26T10:27:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12699.148077 secs 2024-07-26T10:27:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:33 smithi163 ceph-mon[36873]: pgmap v7291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:33 smithi042 ceph-mon[32775]: pgmap v7291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:33 smithi164 ceph-mon[36970]: pgmap v7291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:35 smithi042 ceph-mon[32775]: pgmap v7292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:35 smithi164 ceph-mon[36970]: pgmap v7292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:35 smithi163 ceph-mon[36873]: pgmap v7292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12686.408970 secs 2024-07-26T10:27:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:37 smithi042 ceph-mon[32775]: pgmap v7293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12704.148203 secs 2024-07-26T10:27:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12686.408970 secs 2024-07-26T10:27:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:37 smithi164 ceph-mon[36970]: pgmap v7293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:37.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12704.148203 secs 2024-07-26T10:27:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12686.408970 secs 2024-07-26T10:27:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:37 smithi163 ceph-mon[36873]: pgmap v7293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12704.148203 secs 2024-07-26T10:27:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:39 smithi042 ceph-mon[32775]: pgmap v7294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:39 smithi164 ceph-mon[36970]: pgmap v7294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:39 smithi163 ceph-mon[36873]: pgmap v7294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:41 smithi042 ceph-mon[32775]: pgmap v7295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:41 smithi164 ceph-mon[36970]: pgmap v7295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:41 smithi163 ceph-mon[36873]: pgmap v7295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12691.409116 secs 2024-07-26T10:27:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12709.148305 secs 2024-07-26T10:27:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12691.409116 secs 2024-07-26T10:27:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12709.148305 secs 2024-07-26T10:27:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12691.409116 secs 2024-07-26T10:27:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12709.148305 secs 2024-07-26T10:27:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:27:42.895+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:27:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:27:42.895+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:27:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:27:42.895+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:27:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:43 smithi042 ceph-mon[32775]: pgmap v7296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:43 smithi164 ceph-mon[36970]: pgmap v7296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:43 smithi163 ceph-mon[36873]: pgmap v7296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:45 smithi042 ceph-mon[32775]: pgmap v7297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:45 smithi164 ceph-mon[36970]: pgmap v7297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:45 smithi163 ceph-mon[36873]: pgmap v7297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:27:47.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12696.409239 secs 2024-07-26T10:27:47.609 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:47 smithi163 ceph-mon[36873]: pgmap v7298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:47.609 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12714.148453 secs 2024-07-26T10:27:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12696.409239 secs 2024-07-26T10:27:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:47 smithi042 ceph-mon[32775]: pgmap v7298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12714.148453 secs 2024-07-26T10:27:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12696.409239 secs 2024-07-26T10:27:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:47 smithi164 ceph-mon[36970]: pgmap v7298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12714.148453 secs 2024-07-26T10:27:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:27:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:27:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:27:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:49 smithi042 ceph-mon[32775]: pgmap v7299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:27:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:27:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:27:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:49 smithi164 ceph-mon[36970]: pgmap v7299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:27:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:27:49.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:27:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:49 smithi163 ceph-mon[36873]: pgmap v7299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:27:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:27:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:27:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:27:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:50 smithi042 ceph-mon[32775]: pgmap v7300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:27:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:50 smithi042 ceph-mon[32775]: pgmap v7301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:27:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:50 smithi164 ceph-mon[36970]: pgmap v7300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:27:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:50 smithi164 ceph-mon[36970]: pgmap v7301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:27:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:50 smithi163 ceph-mon[36873]: pgmap v7300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:27:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:50 smithi163 ceph-mon[36873]: pgmap v7301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:27:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:52 smithi042 ceph-mon[32775]: pgmap v7302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:27:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12701.409365 secs 2024-07-26T10:27:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12719.148574 secs 2024-07-26T10:27:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:52 smithi164 ceph-mon[36970]: pgmap v7302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:27:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12701.409365 secs 2024-07-26T10:27:52.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12719.148574 secs 2024-07-26T10:27:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:52 smithi163 ceph-mon[36873]: pgmap v7302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:27:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12701.409365 secs 2024-07-26T10:27:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12719.148574 secs 2024-07-26T10:27:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:53 smithi042 ceph-mon[32775]: pgmap v7303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:27:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:53 smithi164 ceph-mon[36970]: pgmap v7303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:27:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:53 smithi163 ceph-mon[36873]: pgmap v7303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:27:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:55 smithi164 ceph-mon[36970]: pgmap v7304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:27:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:55 smithi163 ceph-mon[36873]: pgmap v7304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:27:56.017 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:55 smithi042 ceph-mon[32775]: pgmap v7304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:27:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12706.409464 secs 2024-07-26T10:27:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12706.409464 secs 2024-07-26T10:27:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12706.409464 secs 2024-07-26T10:27:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:27:57.899+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:27:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:27:57.899+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:27:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:27:57.899+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:27:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12724.148692 secs 2024-07-26T10:27:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:57 smithi163 ceph-mon[36873]: pgmap v7305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:27:58.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12724.148692 secs 2024-07-26T10:27:58.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:57 smithi164 ceph-mon[36970]: pgmap v7305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:27:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12724.148692 secs 2024-07-26T10:27:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:57 smithi042 ceph-mon[32775]: pgmap v7305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:28:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:27:59 smithi163 ceph-mon[36873]: pgmap v7306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:28:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:27:59 smithi042 ceph-mon[32775]: pgmap v7306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:28:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:27:59 smithi164 ceph-mon[36970]: pgmap v7306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:28:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:01 smithi163 ceph-mon[36873]: pgmap v7307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12711.409589 secs 2024-07-26T10:28:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:01 smithi042 ceph-mon[32775]: pgmap v7307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12711.409589 secs 2024-07-26T10:28:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:01 smithi164 ceph-mon[36970]: pgmap v7307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12711.409589 secs 2024-07-26T10:28:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12729.148813 secs 2024-07-26T10:28:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12729.148813 secs 2024-07-26T10:28:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12729.148813 secs 2024-07-26T10:28:04.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:03 smithi163 ceph-mon[36873]: pgmap v7308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:03 smithi042 ceph-mon[32775]: pgmap v7308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:03 smithi164 ceph-mon[36970]: pgmap v7308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:05 smithi163 ceph-mon[36873]: pgmap v7309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:05 smithi042 ceph-mon[32775]: pgmap v7309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:05 smithi164 ceph-mon[36970]: pgmap v7309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12716.409731 secs 2024-07-26T10:28:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12716.409731 secs 2024-07-26T10:28:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12716.409731 secs 2024-07-26T10:28:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12734.148937 secs 2024-07-26T10:28:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:07 smithi163 ceph-mon[36873]: pgmap v7310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12734.148937 secs 2024-07-26T10:28:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:07 smithi042 ceph-mon[32775]: pgmap v7310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12734.148937 secs 2024-07-26T10:28:08.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:07 smithi164 ceph-mon[36970]: pgmap v7310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:09 smithi163 ceph-mon[36873]: pgmap v7311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:28:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:09 smithi042 ceph-mon[32775]: pgmap v7311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:28:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:09 smithi164 ceph-mon[36970]: pgmap v7311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:28:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:11 smithi042 ceph-mon[32775]: pgmap v7312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12721.409842 secs 2024-07-26T10:28:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:11 smithi164 ceph-mon[36970]: pgmap v7312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12721.409842 secs 2024-07-26T10:28:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:11 smithi163 ceph-mon[36873]: pgmap v7312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12721.409842 secs 2024-07-26T10:28:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:28:12.900+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:28:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:28:12.900+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:28:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:28:12.900+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:28:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12739.149060 secs 2024-07-26T10:28:13.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12739.149060 secs 2024-07-26T10:28:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12739.149060 secs 2024-07-26T10:28:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:13 smithi042 ceph-mon[32775]: pgmap v7313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:13 smithi164 ceph-mon[36970]: pgmap v7313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:13 smithi163 ceph-mon[36873]: pgmap v7313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:15 smithi042 ceph-mon[32775]: pgmap v7314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:15 smithi164 ceph-mon[36970]: pgmap v7314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:15 smithi163 ceph-mon[36873]: pgmap v7314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12726.409991 secs 2024-07-26T10:28:17.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12726.409991 secs 2024-07-26T10:28:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12726.409991 secs 2024-07-26T10:28:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12744.149175 secs 2024-07-26T10:28:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:17 smithi042 ceph-mon[32775]: pgmap v7315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12744.149175 secs 2024-07-26T10:28:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:17 smithi164 ceph-mon[36970]: pgmap v7315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12744.149175 secs 2024-07-26T10:28:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:17 smithi163 ceph-mon[36873]: pgmap v7315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:19 smithi042 ceph-mon[32775]: pgmap v7316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:28:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:19 smithi164 ceph-mon[36970]: pgmap v7316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:28:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:19 smithi163 ceph-mon[36873]: pgmap v7316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:28:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:21 smithi042 ceph-mon[32775]: pgmap v7317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12731.410117 secs 2024-07-26T10:28:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:21 smithi164 ceph-mon[36970]: pgmap v7317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12731.410117 secs 2024-07-26T10:28:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:21 smithi163 ceph-mon[36873]: pgmap v7317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12731.410117 secs 2024-07-26T10:28:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12749.149297 secs 2024-07-26T10:28:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12749.149297 secs 2024-07-26T10:28:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12749.149297 secs 2024-07-26T10:28:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:23 smithi042 ceph-mon[32775]: pgmap v7318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:23 smithi164 ceph-mon[36970]: pgmap v7318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:23 smithi163 ceph-mon[36873]: pgmap v7318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:25 smithi042 ceph-mon[32775]: pgmap v7319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:25 smithi164 ceph-mon[36970]: pgmap v7319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:25 smithi163 ceph-mon[36873]: pgmap v7319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12736.410251 secs 2024-07-26T10:28:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12736.410251 secs 2024-07-26T10:28:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12736.410251 secs 2024-07-26T10:28:27.907 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:28:27.905+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:28:27.907 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:28:27.905+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:28:27.907 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:28:27.905+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:28:28.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12754.149417 secs 2024-07-26T10:28:28.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:27 smithi164 ceph-mon[36970]: pgmap v7320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12754.149417 secs 2024-07-26T10:28:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:27 smithi042 ceph-mon[32775]: pgmap v7320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12754.149417 secs 2024-07-26T10:28:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:27 smithi163 ceph-mon[36873]: pgmap v7320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:29 smithi042 ceph-mon[32775]: pgmap v7321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:28:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:29 smithi164 ceph-mon[36970]: pgmap v7321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:28:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:29 smithi163 ceph-mon[36873]: pgmap v7321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:28:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:31 smithi042 ceph-mon[32775]: pgmap v7322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12741.410372 secs 2024-07-26T10:28:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:31 smithi164 ceph-mon[36970]: pgmap v7322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12741.410372 secs 2024-07-26T10:28:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:31 smithi163 ceph-mon[36873]: pgmap v7322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12741.410372 secs 2024-07-26T10:28:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12759.149545 secs 2024-07-26T10:28:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12759.149545 secs 2024-07-26T10:28:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12759.149545 secs 2024-07-26T10:28:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:33 smithi042 ceph-mon[32775]: pgmap v7323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:33 smithi164 ceph-mon[36970]: pgmap v7323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:33 smithi163 ceph-mon[36873]: pgmap v7323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:35 smithi042 ceph-mon[32775]: pgmap v7324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:35 smithi164 ceph-mon[36970]: pgmap v7324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:35 smithi163 ceph-mon[36873]: pgmap v7324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12746.410469 secs 2024-07-26T10:28:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12746.410469 secs 2024-07-26T10:28:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12746.410469 secs 2024-07-26T10:28:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12764.149669 secs 2024-07-26T10:28:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:37 smithi042 ceph-mon[32775]: pgmap v7325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12764.149669 secs 2024-07-26T10:28:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:37 smithi164 ceph-mon[36970]: pgmap v7325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12764.149669 secs 2024-07-26T10:28:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:37 smithi163 ceph-mon[36873]: pgmap v7325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:39 smithi042 ceph-mon[32775]: pgmap v7326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:28:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:39 smithi164 ceph-mon[36970]: pgmap v7326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:28:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:39 smithi163 ceph-mon[36873]: pgmap v7326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:28:42.097 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:41 smithi042 ceph-mon[32775]: pgmap v7327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:42.097 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12751.410584 secs 2024-07-26T10:28:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:41 smithi164 ceph-mon[36970]: pgmap v7327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12751.410584 secs 2024-07-26T10:28:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:41 smithi163 ceph-mon[36873]: pgmap v7327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12751.410584 secs 2024-07-26T10:28:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:28:42.906+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:28:42.908 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:28:42.906+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:28:42.909 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:28:42.906+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:28:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12769.149785 secs 2024-07-26T10:28:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12769.149785 secs 2024-07-26T10:28:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12769.149785 secs 2024-07-26T10:28:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:43 smithi042 ceph-mon[32775]: pgmap v7328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:43 smithi164 ceph-mon[36970]: pgmap v7328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:43 smithi163 ceph-mon[36873]: pgmap v7328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:45 smithi042 ceph-mon[32775]: pgmap v7329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:45 smithi164 ceph-mon[36970]: pgmap v7329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:45 smithi163 ceph-mon[36873]: pgmap v7329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:28:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12756.410669 secs 2024-07-26T10:28:47.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12756.410669 secs 2024-07-26T10:28:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12756.410669 secs 2024-07-26T10:28:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12774.149924 secs 2024-07-26T10:28:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:47 smithi042 ceph-mon[32775]: pgmap v7330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12774.149924 secs 2024-07-26T10:28:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:47 smithi164 ceph-mon[36970]: pgmap v7330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12774.149924 secs 2024-07-26T10:28:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:47 smithi163 ceph-mon[36873]: pgmap v7330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:28:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:28:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:28:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:28:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:49 smithi042 ceph-mon[32775]: pgmap v7331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:28:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:28:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:28:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:49 smithi042 ceph-mon[32775]: pgmap v7332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:28:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:49 smithi042 ceph-mon[32775]: pgmap v7333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:28:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:28:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:49 smithi164 ceph-mon[36970]: pgmap v7331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:28:50.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:28:50.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:28:50.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:49 smithi164 ceph-mon[36970]: pgmap v7332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:28:50.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:49 smithi164 ceph-mon[36970]: pgmap v7333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:28:50.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:28:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:49 smithi163 ceph-mon[36873]: pgmap v7331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:28:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:28:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:28:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:49 smithi163 ceph-mon[36873]: pgmap v7332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:28:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:49 smithi163 ceph-mon[36873]: pgmap v7333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:28:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:28:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12761.410799 secs 2024-07-26T10:28:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:52 smithi042 ceph-mon[32775]: pgmap v7334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:28:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12779.150044 secs 2024-07-26T10:28:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12761.410799 secs 2024-07-26T10:28:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:52 smithi164 ceph-mon[36970]: pgmap v7334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:28:52.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12779.150044 secs 2024-07-26T10:28:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12761.410799 secs 2024-07-26T10:28:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:52 smithi163 ceph-mon[36873]: pgmap v7334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:28:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12779.150044 secs 2024-07-26T10:28:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:54 smithi042 ceph-mon[32775]: pgmap v7335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:28:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:54 smithi164 ceph-mon[36970]: pgmap v7335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:28:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:54 smithi163 ceph-mon[36873]: pgmap v7335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:28:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:56 smithi042 ceph-mon[32775]: pgmap v7336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:28:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:56 smithi164 ceph-mon[36970]: pgmap v7336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:28:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:56 smithi163 ceph-mon[36873]: pgmap v7336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:28:57.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12766.410922 secs 2024-07-26T10:28:57.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12784.150166 secs 2024-07-26T10:28:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12766.410922 secs 2024-07-26T10:28:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12784.150166 secs 2024-07-26T10:28:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12766.410922 secs 2024-07-26T10:28:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12784.150166 secs 2024-07-26T10:28:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:28:57.912+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:28:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:28:57.912+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:28:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:28:57.912+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:28:58.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:28:58 smithi164 ceph-mon[36970]: pgmap v7337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:28:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:28:58 smithi042 ceph-mon[32775]: pgmap v7337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:28:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:28:58 smithi163 ceph-mon[36873]: pgmap v7337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:28:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:28:59.865+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:28:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:28:59.865+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:28:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:28:59.865+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:28:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:28:59.866+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:29:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:00 smithi042 ceph-mon[32775]: pgmap v7338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:29:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:00 smithi164 ceph-mon[36970]: pgmap v7338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:29:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:00 smithi163 ceph-mon[36873]: pgmap v7338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:29:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12771.411021 secs 2024-07-26T10:29:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:02 smithi042 ceph-mon[32775]: pgmap v7339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12789.150267 secs 2024-07-26T10:29:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12771.411021 secs 2024-07-26T10:29:02.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:02 smithi164 ceph-mon[36970]: pgmap v7339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:02.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12789.150267 secs 2024-07-26T10:29:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12771.411021 secs 2024-07-26T10:29:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:02 smithi163 ceph-mon[36873]: pgmap v7339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12789.150267 secs 2024-07-26T10:29:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:04 smithi042 ceph-mon[32775]: pgmap v7340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:04 smithi164 ceph-mon[36970]: pgmap v7340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:04 smithi163 ceph-mon[36873]: pgmap v7340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:06 smithi042 ceph-mon[32775]: pgmap v7341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:06 smithi164 ceph-mon[36970]: pgmap v7341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:06 smithi163 ceph-mon[36873]: pgmap v7341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12776.411151 secs 2024-07-26T10:29:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12794.150388 secs 2024-07-26T10:29:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12776.411151 secs 2024-07-26T10:29:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12794.150388 secs 2024-07-26T10:29:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12776.411151 secs 2024-07-26T10:29:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12794.150388 secs 2024-07-26T10:29:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:08 smithi042 ceph-mon[32775]: pgmap v7342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:08 smithi164 ceph-mon[36970]: pgmap v7342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:08 smithi163 ceph-mon[36873]: pgmap v7342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:09 smithi042 ceph-mon[32775]: pgmap v7343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:09 smithi164 ceph-mon[36970]: pgmap v7343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:09 smithi163 ceph-mon[36873]: pgmap v7343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12781.411249 secs 2024-07-26T10:29:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:12 smithi042 ceph-mon[32775]: pgmap v7344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12799.150518 secs 2024-07-26T10:29:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12781.411249 secs 2024-07-26T10:29:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:12 smithi164 ceph-mon[36970]: pgmap v7344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12799.150518 secs 2024-07-26T10:29:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12781.411249 secs 2024-07-26T10:29:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:12 smithi163 ceph-mon[36873]: pgmap v7344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12799.150518 secs 2024-07-26T10:29:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:29:12.913+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:29:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:29:12.913+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:29:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:29:12.913+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:29:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:14 smithi042 ceph-mon[32775]: pgmap v7345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:14 smithi164 ceph-mon[36970]: pgmap v7345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:14 smithi163 ceph-mon[36873]: pgmap v7345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:16 smithi042 ceph-mon[32775]: pgmap v7346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:16 smithi164 ceph-mon[36970]: pgmap v7346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:16 smithi163 ceph-mon[36873]: pgmap v7346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12786.411370 secs 2024-07-26T10:29:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12804.150618 secs 2024-07-26T10:29:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12786.411370 secs 2024-07-26T10:29:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12804.150618 secs 2024-07-26T10:29:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12786.411370 secs 2024-07-26T10:29:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12804.150618 secs 2024-07-26T10:29:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:18 smithi042 ceph-mon[32775]: pgmap v7347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:18 smithi164 ceph-mon[36970]: pgmap v7347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:18 smithi163 ceph-mon[36873]: pgmap v7347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:20 smithi042 ceph-mon[32775]: pgmap v7348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:20 smithi164 ceph-mon[36970]: pgmap v7348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:20 smithi163 ceph-mon[36873]: pgmap v7348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12791.411435 secs 2024-07-26T10:29:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:22 smithi042 ceph-mon[32775]: pgmap v7349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12809.150738 secs 2024-07-26T10:29:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12791.411435 secs 2024-07-26T10:29:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:22 smithi164 ceph-mon[36970]: pgmap v7349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12809.150738 secs 2024-07-26T10:29:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12791.411435 secs 2024-07-26T10:29:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:22 smithi163 ceph-mon[36873]: pgmap v7349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12809.150738 secs 2024-07-26T10:29:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:24 smithi042 ceph-mon[32775]: pgmap v7350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:24 smithi164 ceph-mon[36970]: pgmap v7350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:24 smithi163 ceph-mon[36873]: pgmap v7350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:26 smithi042 ceph-mon[32775]: pgmap v7351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:26 smithi164 ceph-mon[36970]: pgmap v7351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:26 smithi163 ceph-mon[36873]: pgmap v7351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12796.411580 secs 2024-07-26T10:29:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12814.150865 secs 2024-07-26T10:29:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12796.411580 secs 2024-07-26T10:29:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12814.150865 secs 2024-07-26T10:29:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12796.411580 secs 2024-07-26T10:29:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12814.150865 secs 2024-07-26T10:29:27.920 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:29:27.918+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:29:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:29:27.918+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:29:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:29:27.919+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:29:28.549 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:28 smithi042 ceph-mon[32775]: pgmap v7352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:28.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:28 smithi164 ceph-mon[36970]: pgmap v7352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:28 smithi163 ceph-mon[36873]: pgmap v7352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:29 smithi042 ceph-mon[32775]: pgmap v7353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:29 smithi164 ceph-mon[36970]: pgmap v7353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:29 smithi163 ceph-mon[36873]: pgmap v7353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12801.411714 secs 2024-07-26T10:29:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:32 smithi042 ceph-mon[32775]: pgmap v7354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12819.150967 secs 2024-07-26T10:29:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12801.411714 secs 2024-07-26T10:29:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:32 smithi164 ceph-mon[36970]: pgmap v7354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12819.150967 secs 2024-07-26T10:29:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12801.411714 secs 2024-07-26T10:29:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:32 smithi163 ceph-mon[36873]: pgmap v7354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12819.150967 secs 2024-07-26T10:29:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:33 smithi042 ceph-mon[32775]: pgmap v7355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:33 smithi164 ceph-mon[36970]: pgmap v7355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:33 smithi163 ceph-mon[36873]: pgmap v7355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:36 smithi042 ceph-mon[32775]: pgmap v7356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:36 smithi164 ceph-mon[36970]: pgmap v7356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:36 smithi163 ceph-mon[36873]: pgmap v7356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12806.411858 secs 2024-07-26T10:29:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12824.151035 secs 2024-07-26T10:29:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12806.411858 secs 2024-07-26T10:29:37.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12824.151035 secs 2024-07-26T10:29:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12806.411858 secs 2024-07-26T10:29:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12824.151035 secs 2024-07-26T10:29:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:38 smithi042 ceph-mon[32775]: pgmap v7357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:38 smithi164 ceph-mon[36970]: pgmap v7357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:38 smithi163 ceph-mon[36873]: pgmap v7357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:40 smithi042 ceph-mon[32775]: pgmap v7358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:40 smithi164 ceph-mon[36970]: pgmap v7358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:40 smithi163 ceph-mon[36873]: pgmap v7358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12811.411923 secs 2024-07-26T10:29:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:42 smithi042 ceph-mon[32775]: pgmap v7359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12829.151158 secs 2024-07-26T10:29:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12811.411923 secs 2024-07-26T10:29:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:42 smithi164 ceph-mon[36970]: pgmap v7359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12829.151158 secs 2024-07-26T10:29:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12811.411923 secs 2024-07-26T10:29:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:42 smithi163 ceph-mon[36873]: pgmap v7359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12829.151158 secs 2024-07-26T10:29:42.922 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:29:42.920+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:29:42.922 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:29:42.920+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:29:42.922 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:29:42.920+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:29:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:44 smithi042 ceph-mon[32775]: pgmap v7360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:44 smithi164 ceph-mon[36970]: pgmap v7360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:44 smithi163 ceph-mon[36873]: pgmap v7360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:46 smithi042 ceph-mon[32775]: pgmap v7361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:46 smithi164 ceph-mon[36970]: pgmap v7361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:46 smithi163 ceph-mon[36873]: pgmap v7361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12816.412026 secs 2024-07-26T10:29:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12834.151249 secs 2024-07-26T10:29:47.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12816.412026 secs 2024-07-26T10:29:47.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12834.151249 secs 2024-07-26T10:29:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12816.412026 secs 2024-07-26T10:29:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12834.151249 secs 2024-07-26T10:29:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:48 smithi042 ceph-mon[32775]: pgmap v7362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:48 smithi164 ceph-mon[36970]: pgmap v7362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:48 smithi163 ceph-mon[36873]: pgmap v7362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:29:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:49 smithi042 ceph-mon[32775]: pgmap v7363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:49 smithi164 ceph-mon[36970]: pgmap v7363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:49 smithi163 ceph-mon[36873]: pgmap v7363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:29:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:29:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T10:29:50.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T10:29:50.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T10:29:50.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:29:50.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T10:29:50.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T10:29:50.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:29:50.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:29:50.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: pgmap v7364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:29:50.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: pgmap v7365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:29:50.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:29:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:29:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T10:29:50.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T10:29:50.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T10:29:50.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:29:50.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T10:29:50.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T10:29:50.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:29:50.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:29:50.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: pgmap v7364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:29:50.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: pgmap v7365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:29:50.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:29:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:29:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:29:50.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T10:29:50.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T10:29:50.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T10:29:50.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:29:50.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T10:29:50.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T10:29:50.760 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:29:50.760 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:29:50.760 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: pgmap v7364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:29:50.760 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: pgmap v7365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:29:50.760 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:29:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12821.412155 secs 2024-07-26T10:29:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12821.412155 secs 2024-07-26T10:29:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12821.412155 secs 2024-07-26T10:29:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12839.151313 secs 2024-07-26T10:29:52.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:52 smithi164 ceph-mon[36970]: pgmap v7366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:29:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12839.151313 secs 2024-07-26T10:29:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:52 smithi163 ceph-mon[36873]: pgmap v7366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:29:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12839.151313 secs 2024-07-26T10:29:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:52 smithi042 ceph-mon[32775]: pgmap v7366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:29:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:54 smithi042 ceph-mon[32775]: pgmap v7367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:29:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:54 smithi164 ceph-mon[36970]: pgmap v7367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:29:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:54 smithi163 ceph-mon[36873]: pgmap v7367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:29:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:56 smithi042 ceph-mon[32775]: pgmap v7368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:29:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12826.412307 secs 2024-07-26T10:29:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:56 smithi164 ceph-mon[36970]: pgmap v7368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:29:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12826.412307 secs 2024-07-26T10:29:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:56 smithi163 ceph-mon[36873]: pgmap v7368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:29:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12826.412307 secs 2024-07-26T10:29:57.927 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:29:57.925+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:29:57.927 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:29:57.925+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:29:57.927 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:29:57.925+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:29:58.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12844.151433 secs 2024-07-26T10:29:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12844.151433 secs 2024-07-26T10:29:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12844.151433 secs 2024-07-26T10:29:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:29:58 smithi042 ceph-mon[32775]: pgmap v7369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:29:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:29:58 smithi164 ceph-mon[36970]: pgmap v7369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:29:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:29:58 smithi163 ceph-mon[36873]: pgmap v7369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:30:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:00 smithi042 ceph-mon[32775]: pgmap v7370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:30:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:30:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:00 smithi164 ceph-mon[36970]: pgmap v7370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:30:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:30:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:00 smithi163 ceph-mon[36873]: pgmap v7370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:30:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:30:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12831.412391 secs 2024-07-26T10:30:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12831.412391 secs 2024-07-26T10:30:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12831.412391 secs 2024-07-26T10:30:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12849.151567 secs 2024-07-26T10:30:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:02 smithi042 ceph-mon[32775]: pgmap v7371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12849.151567 secs 2024-07-26T10:30:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:02 smithi164 ceph-mon[36970]: pgmap v7371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12849.151567 secs 2024-07-26T10:30:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:02 smithi163 ceph-mon[36873]: pgmap v7371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:05.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:04 smithi042 ceph-mon[32775]: pgmap v7372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:30:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:04 smithi164 ceph-mon[36970]: pgmap v7372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:30:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:04 smithi163 ceph-mon[36873]: pgmap v7372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:30:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:06 smithi042 ceph-mon[32775]: pgmap v7373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12836.412488 secs 2024-07-26T10:30:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:06 smithi164 ceph-mon[36970]: pgmap v7373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12836.412488 secs 2024-07-26T10:30:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:06 smithi163 ceph-mon[36873]: pgmap v7373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12836.412488 secs 2024-07-26T10:30:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12854.151710 secs 2024-07-26T10:30:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12854.151710 secs 2024-07-26T10:30:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12854.151710 secs 2024-07-26T10:30:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:08 smithi042 ceph-mon[32775]: pgmap v7374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:08 smithi164 ceph-mon[36970]: pgmap v7374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:08 smithi163 ceph-mon[36873]: pgmap v7374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:10 smithi042 ceph-mon[32775]: pgmap v7375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:10 smithi164 ceph-mon[36970]: pgmap v7375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:10 smithi163 ceph-mon[36873]: pgmap v7375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12841.412608 secs 2024-07-26T10:30:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12841.412608 secs 2024-07-26T10:30:12.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12841.412608 secs 2024-07-26T10:30:12.928 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:30:12.926+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:30:12.928 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:30:12.926+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:30:12.928 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:30:12.926+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:30:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12859.151864 secs 2024-07-26T10:30:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:12 smithi042 ceph-mon[32775]: pgmap v7376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12859.151864 secs 2024-07-26T10:30:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:12 smithi164 ceph-mon[36970]: pgmap v7376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12859.151864 secs 2024-07-26T10:30:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:12 smithi163 ceph-mon[36873]: pgmap v7376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:14 smithi042 ceph-mon[32775]: pgmap v7377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:30:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:14 smithi164 ceph-mon[36970]: pgmap v7377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:30:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:14 smithi163 ceph-mon[36873]: pgmap v7377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:30:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:16 smithi164 ceph-mon[36970]: pgmap v7378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12846.412704 secs 2024-07-26T10:30:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:16 smithi163 ceph-mon[36873]: pgmap v7378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12846.412704 secs 2024-07-26T10:30:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:16 smithi042 ceph-mon[32775]: pgmap v7378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12846.412704 secs 2024-07-26T10:30:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12864.151984 secs 2024-07-26T10:30:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12864.151984 secs 2024-07-26T10:30:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12864.151984 secs 2024-07-26T10:30:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:18 smithi164 ceph-mon[36970]: pgmap v7379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:19.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:18 smithi163 ceph-mon[36873]: pgmap v7379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:18 smithi042 ceph-mon[32775]: pgmap v7379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:20 smithi164 ceph-mon[36970]: pgmap v7380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:20 smithi163 ceph-mon[36873]: pgmap v7380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:20 smithi042 ceph-mon[32775]: pgmap v7380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12851.412772 secs 2024-07-26T10:30:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12851.412772 secs 2024-07-26T10:30:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12851.412772 secs 2024-07-26T10:30:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12869.152084 secs 2024-07-26T10:30:23.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:22 smithi164 ceph-mon[36970]: pgmap v7381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12869.152084 secs 2024-07-26T10:30:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:22 smithi163 ceph-mon[36873]: pgmap v7381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12869.152084 secs 2024-07-26T10:30:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:22 smithi042 ceph-mon[32775]: pgmap v7381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:24 smithi164 ceph-mon[36970]: pgmap v7382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:30:25.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:24 smithi163 ceph-mon[36873]: pgmap v7382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:30:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:24 smithi042 ceph-mon[32775]: pgmap v7382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:30:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:26 smithi164 ceph-mon[36970]: pgmap v7383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12856.412865 secs 2024-07-26T10:30:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:26 smithi163 ceph-mon[36873]: pgmap v7383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12856.412865 secs 2024-07-26T10:30:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:26 smithi042 ceph-mon[32775]: pgmap v7383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12856.412865 secs 2024-07-26T10:30:27.933 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:30:27.931+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:30:27.933 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:30:27.931+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:30:27.933 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:30:27.931+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:30:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12874.152171 secs 2024-07-26T10:30:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12874.152171 secs 2024-07-26T10:30:28.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12874.152171 secs 2024-07-26T10:30:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:28 smithi164 ceph-mon[36970]: pgmap v7384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:28 smithi163 ceph-mon[36873]: pgmap v7384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:28 smithi042 ceph-mon[32775]: pgmap v7384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:29.881 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:30:29.879+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:30:29.881 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:30:29.879+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:30:29.882 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:30:29.880+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:30:29.882 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:30:29.880+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:30:31.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:30 smithi042 ceph-mon[32775]: pgmap v7385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:30 smithi164 ceph-mon[36970]: pgmap v7385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:31.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:30 smithi163 ceph-mon[36873]: pgmap v7385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:32.204 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12861.412990 secs 2024-07-26T10:30:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12861.412990 secs 2024-07-26T10:30:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12861.412990 secs 2024-07-26T10:30:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12879.152292 secs 2024-07-26T10:30:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:32 smithi164 ceph-mon[36970]: pgmap v7386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:30:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12879.152292 secs 2024-07-26T10:30:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:32 smithi163 ceph-mon[36873]: pgmap v7386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:30:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12879.152292 secs 2024-07-26T10:30:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:32 smithi042 ceph-mon[32775]: pgmap v7386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:30:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:34 smithi164 ceph-mon[36970]: pgmap v7387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:30:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:34 smithi163 ceph-mon[36873]: pgmap v7387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:30:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:34 smithi042 ceph-mon[32775]: pgmap v7387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:30:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:36 smithi164 ceph-mon[36970]: pgmap v7388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:30:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12866.413055 secs 2024-07-26T10:30:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:36 smithi163 ceph-mon[36873]: pgmap v7388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:30:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12866.413055 secs 2024-07-26T10:30:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:36 smithi042 ceph-mon[32775]: pgmap v7388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:30:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12866.413055 secs 2024-07-26T10:30:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12884.152375 secs 2024-07-26T10:30:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12884.152375 secs 2024-07-26T10:30:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12884.152375 secs 2024-07-26T10:30:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:38 smithi164 ceph-mon[36970]: pgmap v7389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:30:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:38 smithi163 ceph-mon[36873]: pgmap v7389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:30:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:38 smithi042 ceph-mon[32775]: pgmap v7389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:30:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:40 smithi164 ceph-mon[36970]: pgmap v7390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:41.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:40 smithi163 ceph-mon[36873]: pgmap v7390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:40 smithi042 ceph-mon[32775]: pgmap v7390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12871.413177 secs 2024-07-26T10:30:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12871.413177 secs 2024-07-26T10:30:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12871.413177 secs 2024-07-26T10:30:42.934 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:30:42.932+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:30:42.935 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:30:42.932+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:30:42.935 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:30:42.932+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:30:43.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12889.152516 secs 2024-07-26T10:30:43.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:42 smithi164 ceph-mon[36970]: pgmap v7391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12889.152516 secs 2024-07-26T10:30:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:42 smithi163 ceph-mon[36873]: pgmap v7391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12889.152516 secs 2024-07-26T10:30:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:42 smithi042 ceph-mon[32775]: pgmap v7391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:44 smithi163 ceph-mon[36873]: pgmap v7392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:30:45.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:44 smithi042 ceph-mon[32775]: pgmap v7392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:30:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:44 smithi164 ceph-mon[36970]: pgmap v7392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:30:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:46 smithi163 ceph-mon[36873]: pgmap v7393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12876.413301 secs 2024-07-26T10:30:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:46 smithi042 ceph-mon[32775]: pgmap v7393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12876.413301 secs 2024-07-26T10:30:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:46 smithi164 ceph-mon[36970]: pgmap v7393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12876.413301 secs 2024-07-26T10:30:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12894.152671 secs 2024-07-26T10:30:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12894.152671 secs 2024-07-26T10:30:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12894.152671 secs 2024-07-26T10:30:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:48 smithi163 ceph-mon[36873]: pgmap v7394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:48 smithi042 ceph-mon[32775]: pgmap v7394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:48 smithi164 ceph-mon[36970]: pgmap v7394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:30:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:30:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:30:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:30:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:50 smithi163 ceph-mon[36873]: pgmap v7395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:30:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:30:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:50 smithi163 ceph-mon[36873]: pgmap v7396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:30:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:50 smithi163 ceph-mon[36873]: pgmap v7397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:51.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:30:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:50 smithi042 ceph-mon[32775]: pgmap v7395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:30:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:30:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:50 smithi042 ceph-mon[32775]: pgmap v7396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:30:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:50 smithi042 ceph-mon[32775]: pgmap v7397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:30:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:50 smithi164 ceph-mon[36970]: pgmap v7395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:30:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:30:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:30:51.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:50 smithi164 ceph-mon[36970]: pgmap v7396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:30:51.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:50 smithi164 ceph-mon[36970]: pgmap v7397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:51.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:30:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12881.413411 secs 2024-07-26T10:30:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12881.413411 secs 2024-07-26T10:30:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12881.413411 secs 2024-07-26T10:30:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:53 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12899.152781 secs 2024-07-26T10:30:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:53 smithi163 ceph-mon[36873]: pgmap v7398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:53 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12899.152781 secs 2024-07-26T10:30:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:53 smithi042 ceph-mon[32775]: pgmap v7398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:53 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12899.152781 secs 2024-07-26T10:30:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:53 smithi164 ceph-mon[36970]: pgmap v7398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:54 smithi042 ceph-mon[32775]: pgmap v7399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:54 smithi164 ceph-mon[36970]: pgmap v7399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:54 smithi163 ceph-mon[36873]: pgmap v7399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12886.413493 secs 2024-07-26T10:30:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12904.152901 secs 2024-07-26T10:30:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:57 smithi042 ceph-mon[32775]: pgmap v7400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12886.413493 secs 2024-07-26T10:30:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12904.152901 secs 2024-07-26T10:30:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:57 smithi164 ceph-mon[36970]: pgmap v7400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12886.413493 secs 2024-07-26T10:30:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12904.152901 secs 2024-07-26T10:30:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:57 smithi163 ceph-mon[36873]: pgmap v7400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:30:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:30:57.937+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:30:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:30:57.937+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:30:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:30:57.937+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:30:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:30:59 smithi042 ceph-mon[32775]: pgmap v7401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:30:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:30:59 smithi164 ceph-mon[36970]: pgmap v7401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:30:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:30:59 smithi163 ceph-mon[36873]: pgmap v7401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:31:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:01 smithi042 ceph-mon[32775]: pgmap v7402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:31:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:01 smithi164 ceph-mon[36970]: pgmap v7402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:31:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:01 smithi163 ceph-mon[36873]: pgmap v7402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:31:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12891.413615 secs 2024-07-26T10:31:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12909.153001 secs 2024-07-26T10:31:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12891.413615 secs 2024-07-26T10:31:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12909.153001 secs 2024-07-26T10:31:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12891.413615 secs 2024-07-26T10:31:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12909.153001 secs 2024-07-26T10:31:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:03 smithi042 ceph-mon[32775]: pgmap v7403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:03 smithi164 ceph-mon[36970]: pgmap v7403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:03 smithi163 ceph-mon[36873]: pgmap v7403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:05 smithi164 ceph-mon[36970]: pgmap v7404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:05 smithi163 ceph-mon[36873]: pgmap v7404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:05 smithi042 ceph-mon[32775]: pgmap v7404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12896.413765 secs 2024-07-26T10:31:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12914.153109 secs 2024-07-26T10:31:07.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:07 smithi164 ceph-mon[36970]: pgmap v7405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12896.413765 secs 2024-07-26T10:31:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12914.153109 secs 2024-07-26T10:31:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:07 smithi163 ceph-mon[36873]: pgmap v7405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12896.413765 secs 2024-07-26T10:31:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12914.153109 secs 2024-07-26T10:31:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:07 smithi042 ceph-mon[32775]: pgmap v7405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:09 smithi164 ceph-mon[36970]: pgmap v7406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:31:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:09 smithi163 ceph-mon[36873]: pgmap v7406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:31:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:09 smithi042 ceph-mon[32775]: pgmap v7406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:31:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:11 smithi042 ceph-mon[32775]: pgmap v7407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:11 smithi164 ceph-mon[36970]: pgmap v7407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:11 smithi163 ceph-mon[36873]: pgmap v7407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12901.413891 secs 2024-07-26T10:31:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12919.153191 secs 2024-07-26T10:31:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12901.413891 secs 2024-07-26T10:31:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12919.153191 secs 2024-07-26T10:31:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12901.413891 secs 2024-07-26T10:31:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12919.153191 secs 2024-07-26T10:31:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:31:12.938+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:31:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:31:12.938+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:31:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:31:12.938+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:31:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:13 smithi164 ceph-mon[36970]: pgmap v7408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:13 smithi163 ceph-mon[36873]: pgmap v7408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:13 smithi042 ceph-mon[32775]: pgmap v7408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:15 smithi164 ceph-mon[36970]: pgmap v7409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:15 smithi163 ceph-mon[36873]: pgmap v7409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:15 smithi042 ceph-mon[32775]: pgmap v7409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12906.414016 secs 2024-07-26T10:31:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12924.153311 secs 2024-07-26T10:31:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:17 smithi042 ceph-mon[32775]: pgmap v7410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12906.414016 secs 2024-07-26T10:31:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12924.153311 secs 2024-07-26T10:31:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:17 smithi164 ceph-mon[36970]: pgmap v7410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12906.414016 secs 2024-07-26T10:31:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12924.153311 secs 2024-07-26T10:31:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:17 smithi163 ceph-mon[36873]: pgmap v7410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:19 smithi042 ceph-mon[32775]: pgmap v7411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:31:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:19 smithi164 ceph-mon[36970]: pgmap v7411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:31:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:19 smithi163 ceph-mon[36873]: pgmap v7411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:31:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:21 smithi042 ceph-mon[32775]: pgmap v7412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12911.414167 secs 2024-07-26T10:31:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:21 smithi164 ceph-mon[36970]: pgmap v7412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12911.414167 secs 2024-07-26T10:31:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:21 smithi163 ceph-mon[36873]: pgmap v7412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12911.414167 secs 2024-07-26T10:31:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12929.153438 secs 2024-07-26T10:31:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12929.153438 secs 2024-07-26T10:31:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12929.153438 secs 2024-07-26T10:31:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:23 smithi042 ceph-mon[32775]: pgmap v7413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:23 smithi164 ceph-mon[36970]: pgmap v7413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:23 smithi163 ceph-mon[36873]: pgmap v7413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:25 smithi042 ceph-mon[32775]: pgmap v7414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:25 smithi164 ceph-mon[36970]: pgmap v7414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:25 smithi163 ceph-mon[36873]: pgmap v7414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12916.414320 secs 2024-07-26T10:31:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12916.414320 secs 2024-07-26T10:31:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12916.414320 secs 2024-07-26T10:31:27.945 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:31:27.943+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:31:27.945 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:31:27.943+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:31:27.945 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:31:27.943+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:31:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12934.153554 secs 2024-07-26T10:31:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:27 smithi042 ceph-mon[32775]: pgmap v7415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12934.153554 secs 2024-07-26T10:31:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:27 smithi164 ceph-mon[36970]: pgmap v7415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12934.153554 secs 2024-07-26T10:31:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:27 smithi163 ceph-mon[36873]: pgmap v7415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:29 smithi042 ceph-mon[32775]: pgmap v7416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:31:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:29 smithi164 ceph-mon[36970]: pgmap v7416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:31:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:29 smithi163 ceph-mon[36873]: pgmap v7416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:31:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:31 smithi042 ceph-mon[32775]: pgmap v7417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12921.414452 secs 2024-07-26T10:31:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:31 smithi164 ceph-mon[36970]: pgmap v7417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12921.414452 secs 2024-07-26T10:31:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:31 smithi163 ceph-mon[36873]: pgmap v7417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12921.414452 secs 2024-07-26T10:31:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12939.153679 secs 2024-07-26T10:31:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12939.153679 secs 2024-07-26T10:31:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12939.153679 secs 2024-07-26T10:31:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:33 smithi164 ceph-mon[36970]: pgmap v7418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:33 smithi163 ceph-mon[36873]: pgmap v7418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:33 smithi042 ceph-mon[32775]: pgmap v7418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:35 smithi164 ceph-mon[36970]: pgmap v7419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:35 smithi163 ceph-mon[36873]: pgmap v7419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:35 smithi042 ceph-mon[32775]: pgmap v7419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12926.414562 secs 2024-07-26T10:31:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12926.414562 secs 2024-07-26T10:31:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12926.414562 secs 2024-07-26T10:31:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12944.153797 secs 2024-07-26T10:31:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:37 smithi164 ceph-mon[36970]: pgmap v7420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12944.153797 secs 2024-07-26T10:31:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:37 smithi163 ceph-mon[36873]: pgmap v7420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12944.153797 secs 2024-07-26T10:31:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:37 smithi042 ceph-mon[32775]: pgmap v7420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:39 smithi164 ceph-mon[36970]: pgmap v7421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:31:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:39 smithi163 ceph-mon[36873]: pgmap v7421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:31:40.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:39 smithi042 ceph-mon[32775]: pgmap v7421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:31:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:41 smithi164 ceph-mon[36970]: pgmap v7422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12931.414699 secs 2024-07-26T10:31:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:41 smithi163 ceph-mon[36873]: pgmap v7422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12931.414699 secs 2024-07-26T10:31:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:41 smithi042 ceph-mon[32775]: pgmap v7422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12931.414699 secs 2024-07-26T10:31:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:31:42.943+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:31:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:31:42.943+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:31:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:31:42.943+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:31:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12949.153912 secs 2024-07-26T10:31:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12949.153912 secs 2024-07-26T10:31:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12949.153912 secs 2024-07-26T10:31:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:43 smithi164 ceph-mon[36970]: pgmap v7423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:43 smithi163 ceph-mon[36873]: pgmap v7423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:43 smithi042 ceph-mon[32775]: pgmap v7423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:45 smithi164 ceph-mon[36970]: pgmap v7424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:45 smithi163 ceph-mon[36873]: pgmap v7424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:45 smithi042 ceph-mon[32775]: pgmap v7424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12936.414832 secs 2024-07-26T10:31:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12936.414832 secs 2024-07-26T10:31:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12936.414832 secs 2024-07-26T10:31:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12954.154031 secs 2024-07-26T10:31:48.228 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:47 smithi164 ceph-mon[36970]: pgmap v7425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12954.154031 secs 2024-07-26T10:31:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:47 smithi163 ceph-mon[36873]: pgmap v7425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12954.154031 secs 2024-07-26T10:31:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:47 smithi042 ceph-mon[32775]: pgmap v7425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:31:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:49 smithi164 ceph-mon[36970]: pgmap v7426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:31:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:49 smithi163 ceph-mon[36873]: pgmap v7426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:31:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:49 smithi042 ceph-mon[32775]: pgmap v7426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:31:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:31:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:31:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:31:51.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:31:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:31:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:31:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:31:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:31:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:31:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:31:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:31:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:31:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:51 smithi164 ceph-mon[36970]: pgmap v7427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:51 smithi164 ceph-mon[36970]: pgmap v7428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:31:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:51 smithi164 ceph-mon[36970]: pgmap v7429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:31:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12941.414905 secs 2024-07-26T10:31:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:51 smithi163 ceph-mon[36873]: pgmap v7427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:51 smithi163 ceph-mon[36873]: pgmap v7428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:31:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:51 smithi163 ceph-mon[36873]: pgmap v7429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:31:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12941.414905 secs 2024-07-26T10:31:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:51 smithi042 ceph-mon[32775]: pgmap v7427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:31:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:51 smithi042 ceph-mon[32775]: pgmap v7428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:31:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:51 smithi042 ceph-mon[32775]: pgmap v7429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:31:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12941.414905 secs 2024-07-26T10:31:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12959.154116 secs 2024-07-26T10:31:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12959.154116 secs 2024-07-26T10:31:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12959.154116 secs 2024-07-26T10:31:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:53 smithi164 ceph-mon[36970]: pgmap v7430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:31:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:53 smithi163 ceph-mon[36873]: pgmap v7430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:31:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:53 smithi042 ceph-mon[32775]: pgmap v7430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:31:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:55 smithi164 ceph-mon[36970]: pgmap v7431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:31:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:55 smithi163 ceph-mon[36873]: pgmap v7431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:31:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:55 smithi042 ceph-mon[32775]: pgmap v7431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:31:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12946.414979 secs 2024-07-26T10:31:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12946.414979 secs 2024-07-26T10:31:57.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12946.414979 secs 2024-07-26T10:31:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:31:57.949+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:31:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:31:57.949+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:31:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:31:57.949+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:31:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12964.154272 secs 2024-07-26T10:31:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:57 smithi164 ceph-mon[36970]: pgmap v7432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:31:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12964.154272 secs 2024-07-26T10:31:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:57 smithi163 ceph-mon[36873]: pgmap v7432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:31:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12964.154272 secs 2024-07-26T10:31:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:57 smithi042 ceph-mon[32775]: pgmap v7432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:31:59.896 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:31:59.894+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:31:59.896 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:31:59.894+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:31:59.896 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:31:59.894+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:31:59.896 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:31:59.894+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:32:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:31:59 smithi164 ceph-mon[36970]: pgmap v7433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:32:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:31:59 smithi163 ceph-mon[36873]: pgmap v7433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:32:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:31:59 smithi042 ceph-mon[32775]: pgmap v7433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:32:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:01 smithi164 ceph-mon[36970]: pgmap v7434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:32:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12951.415111 secs 2024-07-26T10:32:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:01 smithi163 ceph-mon[36873]: pgmap v7434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:32:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12951.415111 secs 2024-07-26T10:32:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:01 smithi042 ceph-mon[32775]: pgmap v7434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:32:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12951.415111 secs 2024-07-26T10:32:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12969.154393 secs 2024-07-26T10:32:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12969.154393 secs 2024-07-26T10:32:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12969.154393 secs 2024-07-26T10:32:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:03 smithi042 ceph-mon[32775]: pgmap v7435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:03 smithi163 ceph-mon[36873]: pgmap v7435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:03 smithi164 ceph-mon[36970]: pgmap v7435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:05 smithi163 ceph-mon[36873]: pgmap v7436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:05 smithi042 ceph-mon[32775]: pgmap v7436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:05 smithi164 ceph-mon[36970]: pgmap v7436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12956.415274 secs 2024-07-26T10:32:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12956.415274 secs 2024-07-26T10:32:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12956.415274 secs 2024-07-26T10:32:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12974.154514 secs 2024-07-26T10:32:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:07 smithi163 ceph-mon[36873]: pgmap v7437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12974.154514 secs 2024-07-26T10:32:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:07 smithi042 ceph-mon[32775]: pgmap v7437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12974.154514 secs 2024-07-26T10:32:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:07 smithi164 ceph-mon[36970]: pgmap v7437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:09 smithi163 ceph-mon[36873]: pgmap v7438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:32:10.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:09 smithi042 ceph-mon[32775]: pgmap v7438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:32:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:09 smithi164 ceph-mon[36970]: pgmap v7438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:32:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:11 smithi163 ceph-mon[36873]: pgmap v7439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12961.415416 secs 2024-07-26T10:32:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:11 smithi042 ceph-mon[32775]: pgmap v7439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12961.415416 secs 2024-07-26T10:32:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:11 smithi164 ceph-mon[36970]: pgmap v7439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12961.415416 secs 2024-07-26T10:32:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:32:12.950+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:32:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:32:12.950+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:32:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:32:12.950+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:32:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:13 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12979.154638 secs 2024-07-26T10:32:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:13 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12979.154638 secs 2024-07-26T10:32:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:13 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12979.154638 secs 2024-07-26T10:32:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:14 smithi163 ceph-mon[36873]: pgmap v7440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:14 smithi042 ceph-mon[32775]: pgmap v7440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:14 smithi164 ceph-mon[36970]: pgmap v7440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:16 smithi042 ceph-mon[32775]: pgmap v7441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:16 smithi164 ceph-mon[36970]: pgmap v7441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:16 smithi163 ceph-mon[36873]: pgmap v7441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12966.415487 secs 2024-07-26T10:32:17.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12966.415487 secs 2024-07-26T10:32:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12966.415487 secs 2024-07-26T10:32:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:18 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12984.154759 secs 2024-07-26T10:32:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:18 smithi042 ceph-mon[32775]: pgmap v7442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:18 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12984.154759 secs 2024-07-26T10:32:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:18 smithi164 ceph-mon[36970]: pgmap v7442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:18 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12984.154759 secs 2024-07-26T10:32:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:18 smithi163 ceph-mon[36873]: pgmap v7442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:20 smithi042 ceph-mon[32775]: pgmap v7443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:32:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:20 smithi164 ceph-mon[36970]: pgmap v7443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:32:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:20 smithi163 ceph-mon[36873]: pgmap v7443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:32:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:22 smithi042 ceph-mon[32775]: pgmap v7444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12971.415617 secs 2024-07-26T10:32:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:22 smithi164 ceph-mon[36970]: pgmap v7444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12971.415617 secs 2024-07-26T10:32:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:22 smithi163 ceph-mon[36873]: pgmap v7444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12971.415617 secs 2024-07-26T10:32:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:23 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12989.154883 secs 2024-07-26T10:32:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:23 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12989.154883 secs 2024-07-26T10:32:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:23 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12989.154883 secs 2024-07-26T10:32:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:24 smithi042 ceph-mon[32775]: pgmap v7445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:24 smithi164 ceph-mon[36970]: pgmap v7445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:24 smithi163 ceph-mon[36873]: pgmap v7445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:26 smithi042 ceph-mon[32775]: pgmap v7446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:26 smithi164 ceph-mon[36970]: pgmap v7446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:26 smithi163 ceph-mon[36873]: pgmap v7446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12976.415758 secs 2024-07-26T10:32:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12994.154960 secs 2024-07-26T10:32:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12976.415758 secs 2024-07-26T10:32:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12994.154960 secs 2024-07-26T10:32:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12976.415758 secs 2024-07-26T10:32:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12994.154960 secs 2024-07-26T10:32:27.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:32:27.951+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:32:27.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:32:27.951+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:32:27.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:32:27.951+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:32:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:28 smithi042 ceph-mon[32775]: pgmap v7447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:28 smithi164 ceph-mon[36970]: pgmap v7447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:28 smithi163 ceph-mon[36873]: pgmap v7447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:30.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:30 smithi042 ceph-mon[32775]: pgmap v7448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:32:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:30 smithi164 ceph-mon[36970]: pgmap v7448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:32:30.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:30 smithi163 ceph-mon[36873]: pgmap v7448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:32:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:32 smithi042 ceph-mon[32775]: pgmap v7449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12981.415883 secs 2024-07-26T10:32:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12999.155079 secs 2024-07-26T10:32:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:32 smithi164 ceph-mon[36970]: pgmap v7449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12981.415883 secs 2024-07-26T10:32:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12999.155079 secs 2024-07-26T10:32:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:32 smithi163 ceph-mon[36873]: pgmap v7449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12981.415883 secs 2024-07-26T10:32:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12999.155079 secs 2024-07-26T10:32:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:34 smithi042 ceph-mon[32775]: pgmap v7450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:34 smithi164 ceph-mon[36970]: pgmap v7450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:34 smithi163 ceph-mon[36873]: pgmap v7450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:36 smithi042 ceph-mon[32775]: pgmap v7451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:36 smithi164 ceph-mon[36970]: pgmap v7451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:36 smithi163 ceph-mon[36873]: pgmap v7451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12986.416019 secs 2024-07-26T10:32:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13004.155203 secs 2024-07-26T10:32:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12986.416019 secs 2024-07-26T10:32:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13004.155203 secs 2024-07-26T10:32:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12986.416019 secs 2024-07-26T10:32:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13004.155203 secs 2024-07-26T10:32:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:38 smithi042 ceph-mon[32775]: pgmap v7452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:38 smithi164 ceph-mon[36970]: pgmap v7452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:38 smithi163 ceph-mon[36873]: pgmap v7452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:40.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:40 smithi042 ceph-mon[32775]: pgmap v7453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:32:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:40 smithi164 ceph-mon[36970]: pgmap v7453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:32:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:40 smithi163 ceph-mon[36873]: pgmap v7453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:32:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:42 smithi042 ceph-mon[32775]: pgmap v7454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12991.416171 secs 2024-07-26T10:32:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13009.155327 secs 2024-07-26T10:32:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:42 smithi164 ceph-mon[36970]: pgmap v7454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12991.416171 secs 2024-07-26T10:32:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13009.155327 secs 2024-07-26T10:32:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:42 smithi163 ceph-mon[36873]: pgmap v7454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12991.416171 secs 2024-07-26T10:32:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13009.155327 secs 2024-07-26T10:32:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:32:42.952+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:32:42.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:32:42.952+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:32:42.955 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:32:42.952+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:32:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:44 smithi042 ceph-mon[32775]: pgmap v7455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:44 smithi164 ceph-mon[36970]: pgmap v7455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:44 smithi163 ceph-mon[36873]: pgmap v7455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:46 smithi042 ceph-mon[32775]: pgmap v7456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:46 smithi164 ceph-mon[36970]: pgmap v7456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:46 smithi163 ceph-mon[36873]: pgmap v7456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 12996.416304 secs 2024-07-26T10:32:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13014.155453 secs 2024-07-26T10:32:47.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 12996.416304 secs 2024-07-26T10:32:47.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13014.155453 secs 2024-07-26T10:32:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 12996.416304 secs 2024-07-26T10:32:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13014.155453 secs 2024-07-26T10:32:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:48 smithi042 ceph-mon[32775]: pgmap v7457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:48 smithi164 ceph-mon[36970]: pgmap v7457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:48 smithi163 ceph-mon[36873]: pgmap v7457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:50 smithi042 ceph-mon[32775]: pgmap v7458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:32:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:50 smithi164 ceph-mon[36970]: pgmap v7458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:32:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:50 smithi163 ceph-mon[36873]: pgmap v7458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:32:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:32:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:32:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:32:52.093 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:52 smithi163 ceph-mon[36873]: pgmap v7459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:52.093 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13001.416425 secs 2024-07-26T10:32:52.093 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13019.155522 secs 2024-07-26T10:32:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:52 smithi042 ceph-mon[32775]: pgmap v7459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13001.416425 secs 2024-07-26T10:32:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13019.155522 secs 2024-07-26T10:32:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:52 smithi164 ceph-mon[36970]: pgmap v7459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:32:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13001.416425 secs 2024-07-26T10:32:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13019.155522 secs 2024-07-26T10:32:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:53 smithi042 ceph-mon[32775]: pgmap v7460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:32:53.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:32:53.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:53 smithi164 ceph-mon[36970]: pgmap v7460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:53.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:53.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:53.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:32:53.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:32:53.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:53 smithi163 ceph-mon[36873]: pgmap v7460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:32:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:32:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:32:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:32:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:54 smithi042 ceph-mon[32775]: pgmap v7461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:32:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:54 smithi042 ceph-mon[32775]: pgmap v7462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:32:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:54 smithi164 ceph-mon[36970]: pgmap v7461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:32:54.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:54 smithi164 ceph-mon[36970]: pgmap v7462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:32:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:54 smithi163 ceph-mon[36873]: pgmap v7461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:32:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:54 smithi163 ceph-mon[36873]: pgmap v7462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:32:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:56 smithi042 ceph-mon[32775]: pgmap v7463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:32:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:56 smithi164 ceph-mon[36970]: pgmap v7463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:32:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:56 smithi163 ceph-mon[36873]: pgmap v7463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:32:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13006.416557 secs 2024-07-26T10:32:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13024.155663 secs 2024-07-26T10:32:57.959 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:32:57.958+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:32:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:32:57.958+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:32:57.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:32:57.958+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:32:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13006.416557 secs 2024-07-26T10:32:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13024.155663 secs 2024-07-26T10:32:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13006.416557 secs 2024-07-26T10:32:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13024.155663 secs 2024-07-26T10:32:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:32:58 smithi042 ceph-mon[32775]: pgmap v7464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:32:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:32:58 smithi164 ceph-mon[36970]: pgmap v7464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:32:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:32:58 smithi163 ceph-mon[36873]: pgmap v7464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:33:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:00 smithi042 ceph-mon[32775]: pgmap v7465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:33:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:00 smithi164 ceph-mon[36970]: pgmap v7465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:33:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:00 smithi163 ceph-mon[36873]: pgmap v7465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:33:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13011.416691 secs 2024-07-26T10:33:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13029.155763 secs 2024-07-26T10:33:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:02 smithi042 ceph-mon[32775]: pgmap v7466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:33:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13011.416691 secs 2024-07-26T10:33:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13029.155763 secs 2024-07-26T10:33:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:02 smithi164 ceph-mon[36970]: pgmap v7466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:33:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13011.416691 secs 2024-07-26T10:33:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13029.155763 secs 2024-07-26T10:33:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:02 smithi163 ceph-mon[36873]: pgmap v7466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:33:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:04 smithi164 ceph-mon[36970]: pgmap v7467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:33:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:04 smithi163 ceph-mon[36873]: pgmap v7467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:33:05.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:04 smithi042 ceph-mon[32775]: pgmap v7467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:33:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:06 smithi164 ceph-mon[36970]: pgmap v7468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:06 smithi163 ceph-mon[36873]: pgmap v7468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:06 smithi042 ceph-mon[32775]: pgmap v7468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13016.416825 secs 2024-07-26T10:33:07.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13034.155897 secs 2024-07-26T10:33:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13016.416825 secs 2024-07-26T10:33:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13034.155897 secs 2024-07-26T10:33:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13016.416825 secs 2024-07-26T10:33:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13034.155897 secs 2024-07-26T10:33:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:08 smithi164 ceph-mon[36970]: pgmap v7469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:08 smithi163 ceph-mon[36873]: pgmap v7469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:08 smithi042 ceph-mon[32775]: pgmap v7469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:10 smithi164 ceph-mon[36970]: pgmap v7470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:10 smithi163 ceph-mon[36873]: pgmap v7470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:10 smithi042 ceph-mon[32775]: pgmap v7470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13021.416969 secs 2024-07-26T10:33:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13021.416969 secs 2024-07-26T10:33:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13021.416969 secs 2024-07-26T10:33:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:33:12.959+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:33:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:33:12.959+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:33:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:33:12.959+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:33:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13039.156020 secs 2024-07-26T10:33:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:12 smithi164 ceph-mon[36970]: pgmap v7471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13039.156020 secs 2024-07-26T10:33:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:12 smithi163 ceph-mon[36873]: pgmap v7471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13039.156020 secs 2024-07-26T10:33:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:12 smithi042 ceph-mon[32775]: pgmap v7471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:14 smithi164 ceph-mon[36970]: pgmap v7472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:33:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:14 smithi163 ceph-mon[36873]: pgmap v7472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:33:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:14 smithi042 ceph-mon[32775]: pgmap v7472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:33:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:16 smithi164 ceph-mon[36970]: pgmap v7473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13026.417102 secs 2024-07-26T10:33:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:16 smithi163 ceph-mon[36873]: pgmap v7473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13026.417102 secs 2024-07-26T10:33:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:16 smithi042 ceph-mon[32775]: pgmap v7473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13026.417102 secs 2024-07-26T10:33:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13044.156084 secs 2024-07-26T10:33:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13044.156084 secs 2024-07-26T10:33:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13044.156084 secs 2024-07-26T10:33:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:18 smithi164 ceph-mon[36970]: pgmap v7474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:18 smithi163 ceph-mon[36873]: pgmap v7474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:18 smithi042 ceph-mon[32775]: pgmap v7474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:20 smithi164 ceph-mon[36970]: pgmap v7475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:21.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:20 smithi163 ceph-mon[36873]: pgmap v7475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:21.137 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:20 smithi042 ceph-mon[32775]: pgmap v7475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13031.417249 secs 2024-07-26T10:33:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13031.417249 secs 2024-07-26T10:33:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13031.417249 secs 2024-07-26T10:33:22.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13049.156199 secs 2024-07-26T10:33:22.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:22 smithi164 ceph-mon[36970]: pgmap v7476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13049.156199 secs 2024-07-26T10:33:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:22 smithi163 ceph-mon[36873]: pgmap v7476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13049.156199 secs 2024-07-26T10:33:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:22 smithi042 ceph-mon[32775]: pgmap v7476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:24 smithi164 ceph-mon[36970]: pgmap v7477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:33:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:24 smithi163 ceph-mon[36873]: pgmap v7477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:33:25.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:24 smithi042 ceph-mon[32775]: pgmap v7477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:33:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:26 smithi164 ceph-mon[36970]: pgmap v7478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13036.417384 secs 2024-07-26T10:33:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:26 smithi163 ceph-mon[36873]: pgmap v7478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13036.417384 secs 2024-07-26T10:33:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:26 smithi042 ceph-mon[32775]: pgmap v7478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13036.417384 secs 2024-07-26T10:33:27.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:33:27.963+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:33:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:33:27.963+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:33:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:33:27.963+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:33:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13054.156300 secs 2024-07-26T10:33:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13054.156300 secs 2024-07-26T10:33:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13054.156300 secs 2024-07-26T10:33:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:28 smithi164 ceph-mon[36970]: pgmap v7479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:28 smithi163 ceph-mon[36873]: pgmap v7479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:28 smithi042 ceph-mon[32775]: pgmap v7479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:30 smithi164 ceph-mon[36970]: pgmap v7480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:30 smithi163 ceph-mon[36873]: pgmap v7480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:30 smithi042 ceph-mon[32775]: pgmap v7480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13041.417519 secs 2024-07-26T10:33:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13041.417519 secs 2024-07-26T10:33:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13041.417519 secs 2024-07-26T10:33:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13059.156424 secs 2024-07-26T10:33:32.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:32 smithi164 ceph-mon[36970]: pgmap v7481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13059.156424 secs 2024-07-26T10:33:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:32 smithi163 ceph-mon[36873]: pgmap v7481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13059.156424 secs 2024-07-26T10:33:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:32 smithi042 ceph-mon[32775]: pgmap v7481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:34 smithi164 ceph-mon[36970]: pgmap v7482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:33:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:34 smithi163 ceph-mon[36873]: pgmap v7482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:33:35.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:34 smithi042 ceph-mon[32775]: pgmap v7482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:33:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:36 smithi163 ceph-mon[36873]: pgmap v7483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13046.417617 secs 2024-07-26T10:33:37.009 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:36 smithi042 ceph-mon[32775]: pgmap v7483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:37.009 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13046.417617 secs 2024-07-26T10:33:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:36 smithi164 ceph-mon[36970]: pgmap v7483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13046.417617 secs 2024-07-26T10:33:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13064.156545 secs 2024-07-26T10:33:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13064.156545 secs 2024-07-26T10:33:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13064.156545 secs 2024-07-26T10:33:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:38 smithi163 ceph-mon[36873]: pgmap v7484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:39.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:38 smithi042 ceph-mon[32775]: pgmap v7484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:38 smithi164 ceph-mon[36970]: pgmap v7484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:40 smithi163 ceph-mon[36873]: pgmap v7485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:41.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:40 smithi042 ceph-mon[32775]: pgmap v7485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:40 smithi164 ceph-mon[36970]: pgmap v7485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13051.417755 secs 2024-07-26T10:33:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13051.417755 secs 2024-07-26T10:33:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13051.417755 secs 2024-07-26T10:33:42.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:33:42.965+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:33:42.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:33:42.965+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:33:42.967 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:33:42.965+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:33:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13069.156677 secs 2024-07-26T10:33:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:42 smithi163 ceph-mon[36873]: pgmap v7486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13069.156677 secs 2024-07-26T10:33:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:42 smithi042 ceph-mon[32775]: pgmap v7486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13069.156677 secs 2024-07-26T10:33:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:42 smithi164 ceph-mon[36970]: pgmap v7486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:44 smithi163 ceph-mon[36873]: pgmap v7487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:33:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:44 smithi042 ceph-mon[32775]: pgmap v7487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:33:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:44 smithi164 ceph-mon[36970]: pgmap v7487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:33:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:46 smithi163 ceph-mon[36873]: pgmap v7488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13056.417916 secs 2024-07-26T10:33:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:46 smithi042 ceph-mon[32775]: pgmap v7488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13056.417916 secs 2024-07-26T10:33:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:46 smithi164 ceph-mon[36970]: pgmap v7488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13056.417916 secs 2024-07-26T10:33:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13074.156759 secs 2024-07-26T10:33:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13074.156759 secs 2024-07-26T10:33:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13074.156759 secs 2024-07-26T10:33:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:48 smithi042 ceph-mon[32775]: pgmap v7489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:48 smithi164 ceph-mon[36970]: pgmap v7489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:48 smithi163 ceph-mon[36873]: pgmap v7489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:50 smithi042 ceph-mon[32775]: pgmap v7490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:50 smithi164 ceph-mon[36970]: pgmap v7490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:50 smithi163 ceph-mon[36873]: pgmap v7490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:33:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13061.418066 secs 2024-07-26T10:33:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13061.418066 secs 2024-07-26T10:33:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13061.418066 secs 2024-07-26T10:33:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13079.156855 secs 2024-07-26T10:33:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:52 smithi042 ceph-mon[32775]: pgmap v7491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13079.156855 secs 2024-07-26T10:33:53.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:52 smithi164 ceph-mon[36970]: pgmap v7491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13079.156855 secs 2024-07-26T10:33:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:52 smithi163 ceph-mon[36873]: pgmap v7491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:33:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:33:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:33:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:33:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:54 smithi042 ceph-mon[32775]: pgmap v7492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:33:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:33:55.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:33:55.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:54 smithi042 ceph-mon[32775]: pgmap v7493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:33:55.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:54 smithi042 ceph-mon[32775]: pgmap v7494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:33:55.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:54 smithi164 ceph-mon[36970]: pgmap v7492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:33:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:33:55.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:33:55.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:54 smithi164 ceph-mon[36970]: pgmap v7493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:33:55.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:54 smithi164 ceph-mon[36970]: pgmap v7494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:33:55.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:54 smithi163 ceph-mon[36873]: pgmap v7492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:33:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:55.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:33:55.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:33:55.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:54 smithi163 ceph-mon[36873]: pgmap v7493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:33:55.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:54 smithi163 ceph-mon[36873]: pgmap v7494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:33:55.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:33:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13066.418198 secs 2024-07-26T10:33:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:57 smithi042 ceph-mon[32775]: pgmap v7495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:33:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13084.156974 secs 2024-07-26T10:33:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13066.418198 secs 2024-07-26T10:33:57.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:57 smithi164 ceph-mon[36970]: pgmap v7495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:33:57.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13084.156974 secs 2024-07-26T10:33:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13066.418198 secs 2024-07-26T10:33:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:57 smithi163 ceph-mon[36873]: pgmap v7495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:33:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13084.156974 secs 2024-07-26T10:33:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:33:57.970+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:33:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:33:57.970+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:33:57.972 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:33:57.970+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:33:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:33:59 smithi042 ceph-mon[32775]: pgmap v7496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T10:33:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:33:59 smithi164 ceph-mon[36970]: pgmap v7496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T10:33:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:33:59 smithi163 ceph-mon[36873]: pgmap v7496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T10:34:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:01 smithi042 ceph-mon[32775]: pgmap v7497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:34:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:01 smithi164 ceph-mon[36970]: pgmap v7497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:34:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:01 smithi163 ceph-mon[36873]: pgmap v7497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:34:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13071.418325 secs 2024-07-26T10:34:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13089.157072 secs 2024-07-26T10:34:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13071.418325 secs 2024-07-26T10:34:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13089.157072 secs 2024-07-26T10:34:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13071.418325 secs 2024-07-26T10:34:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13089.157072 secs 2024-07-26T10:34:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:03 smithi042 ceph-mon[32775]: pgmap v7498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:34:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:03 smithi164 ceph-mon[36970]: pgmap v7498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:34:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:03 smithi163 ceph-mon[36873]: pgmap v7498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:34:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:05 smithi042 ceph-mon[32775]: pgmap v7499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:34:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:05 smithi164 ceph-mon[36970]: pgmap v7499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:34:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:05 smithi163 ceph-mon[36873]: pgmap v7499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:34:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13076.418448 secs 2024-07-26T10:34:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:07 smithi042 ceph-mon[32775]: pgmap v7500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13094.157213 secs 2024-07-26T10:34:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13076.418448 secs 2024-07-26T10:34:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:07 smithi164 ceph-mon[36970]: pgmap v7500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:07.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13094.157213 secs 2024-07-26T10:34:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13076.418448 secs 2024-07-26T10:34:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:07 smithi163 ceph-mon[36873]: pgmap v7500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13094.157213 secs 2024-07-26T10:34:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:09 smithi042 ceph-mon[32775]: pgmap v7501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:34:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:09 smithi164 ceph-mon[36970]: pgmap v7501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:34:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:09 smithi163 ceph-mon[36873]: pgmap v7501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:34:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:11 smithi042 ceph-mon[32775]: pgmap v7502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:11 smithi164 ceph-mon[36970]: pgmap v7502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:11 smithi163 ceph-mon[36873]: pgmap v7502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13081.418570 secs 2024-07-26T10:34:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13099.157294 secs 2024-07-26T10:34:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13081.418570 secs 2024-07-26T10:34:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13099.157294 secs 2024-07-26T10:34:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13081.418570 secs 2024-07-26T10:34:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13099.157294 secs 2024-07-26T10:34:12.973 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:34:12.971+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:34:12.973 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:34:12.971+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:34:12.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:34:12.972+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:34:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:13 smithi042 ceph-mon[32775]: pgmap v7503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:13 smithi164 ceph-mon[36970]: pgmap v7503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:13 smithi163 ceph-mon[36873]: pgmap v7503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:14 smithi042 ceph-mon[32775]: pgmap v7504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:14 smithi164 ceph-mon[36970]: pgmap v7504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:14 smithi163 ceph-mon[36873]: pgmap v7504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13086.418636 secs 2024-07-26T10:34:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:17 smithi042 ceph-mon[32775]: pgmap v7505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:17.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13104.157424 secs 2024-07-26T10:34:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13086.418636 secs 2024-07-26T10:34:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:17 smithi164 ceph-mon[36970]: pgmap v7505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13104.157424 secs 2024-07-26T10:34:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13086.418636 secs 2024-07-26T10:34:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:17 smithi163 ceph-mon[36873]: pgmap v7505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13104.157424 secs 2024-07-26T10:34:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:19 smithi042 ceph-mon[32775]: pgmap v7506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:34:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:19 smithi164 ceph-mon[36970]: pgmap v7506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:34:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:19 smithi163 ceph-mon[36873]: pgmap v7506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:34:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:21 smithi042 ceph-mon[32775]: pgmap v7507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:21 smithi164 ceph-mon[36970]: pgmap v7507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:21 smithi163 ceph-mon[36873]: pgmap v7507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13091.418756 secs 2024-07-26T10:34:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13109.157489 secs 2024-07-26T10:34:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13091.418756 secs 2024-07-26T10:34:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13109.157489 secs 2024-07-26T10:34:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13091.418756 secs 2024-07-26T10:34:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13109.157489 secs 2024-07-26T10:34:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:23 smithi042 ceph-mon[32775]: pgmap v7508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:23 smithi164 ceph-mon[36970]: pgmap v7508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:23 smithi163 ceph-mon[36873]: pgmap v7508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:25 smithi042 ceph-mon[32775]: pgmap v7509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:25 smithi164 ceph-mon[36970]: pgmap v7509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:25 smithi163 ceph-mon[36873]: pgmap v7509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13096.418883 secs 2024-07-26T10:34:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:27 smithi042 ceph-mon[32775]: pgmap v7510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13114.157608 secs 2024-07-26T10:34:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13096.418883 secs 2024-07-26T10:34:27.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:27 smithi164 ceph-mon[36970]: pgmap v7510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:27.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13114.157608 secs 2024-07-26T10:34:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13096.418883 secs 2024-07-26T10:34:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:27 smithi163 ceph-mon[36873]: pgmap v7510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13114.157608 secs 2024-07-26T10:34:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:34:27.976+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:34:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:34:27.976+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:34:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:34:27.976+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:34:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:29 smithi042 ceph-mon[32775]: pgmap v7511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:34:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:29 smithi164 ceph-mon[36970]: pgmap v7511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:34:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:29 smithi163 ceph-mon[36873]: pgmap v7511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:34:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:30 smithi164 ceph-mon[36970]: pgmap v7512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:30 smithi163 ceph-mon[36873]: pgmap v7512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:30 smithi042 ceph-mon[32775]: pgmap v7512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13101.418981 secs 2024-07-26T10:34:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13101.418981 secs 2024-07-26T10:34:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13101.418981 secs 2024-07-26T10:34:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13119.157743 secs 2024-07-26T10:34:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:32 smithi163 ceph-mon[36873]: pgmap v7513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13119.157743 secs 2024-07-26T10:34:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:32 smithi042 ceph-mon[32775]: pgmap v7513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13119.157743 secs 2024-07-26T10:34:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:32 smithi164 ceph-mon[36970]: pgmap v7513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:35 smithi042 ceph-mon[32775]: pgmap v7514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:35 smithi164 ceph-mon[36970]: pgmap v7514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:35 smithi163 ceph-mon[36873]: pgmap v7514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13106.419051 secs 2024-07-26T10:34:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:37 smithi042 ceph-mon[32775]: pgmap v7515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13124.157863 secs 2024-07-26T10:34:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13106.419051 secs 2024-07-26T10:34:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:37 smithi164 ceph-mon[36970]: pgmap v7515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13124.157863 secs 2024-07-26T10:34:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13106.419051 secs 2024-07-26T10:34:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:37 smithi163 ceph-mon[36873]: pgmap v7515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13124.157863 secs 2024-07-26T10:34:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:38 smithi164 ceph-mon[36970]: pgmap v7516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:34:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:38 smithi163 ceph-mon[36873]: pgmap v7516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:34:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:38 smithi042 ceph-mon[32775]: pgmap v7516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:34:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:41 smithi042 ceph-mon[32775]: pgmap v7517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:41 smithi164 ceph-mon[36970]: pgmap v7517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:41 smithi163 ceph-mon[36873]: pgmap v7517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13111.419192 secs 2024-07-26T10:34:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13129.157986 secs 2024-07-26T10:34:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13111.419192 secs 2024-07-26T10:34:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13129.157986 secs 2024-07-26T10:34:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13111.419192 secs 2024-07-26T10:34:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13129.157986 secs 2024-07-26T10:34:42.979 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:34:42.977+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:34:42.979 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:34:42.977+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:34:42.979 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:34:42.977+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:34:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:43 smithi042 ceph-mon[32775]: pgmap v7518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:43 smithi164 ceph-mon[36970]: pgmap v7518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:43 smithi163 ceph-mon[36873]: pgmap v7518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:45 smithi042 ceph-mon[32775]: pgmap v7519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:45 smithi164 ceph-mon[36970]: pgmap v7519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:45 smithi163 ceph-mon[36873]: pgmap v7519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13116.419317 secs 2024-07-26T10:34:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:47 smithi042 ceph-mon[32775]: pgmap v7520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13134.158105 secs 2024-07-26T10:34:47.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13116.419317 secs 2024-07-26T10:34:47.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:47 smithi164 ceph-mon[36970]: pgmap v7520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:47.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13134.158105 secs 2024-07-26T10:34:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13116.419317 secs 2024-07-26T10:34:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:47 smithi163 ceph-mon[36873]: pgmap v7520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13134.158105 secs 2024-07-26T10:34:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:49 smithi042 ceph-mon[32775]: pgmap v7521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:34:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:49 smithi164 ceph-mon[36970]: pgmap v7521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:34:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:49 smithi163 ceph-mon[36873]: pgmap v7521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:34:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:51 smithi042 ceph-mon[32775]: pgmap v7522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:51 smithi164 ceph-mon[36970]: pgmap v7522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:51 smithi163 ceph-mon[36873]: pgmap v7522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13121.419414 secs 2024-07-26T10:34:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13139.158224 secs 2024-07-26T10:34:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:52 smithi042 ceph-mon[32775]: pgmap v7523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13121.419414 secs 2024-07-26T10:34:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13139.158224 secs 2024-07-26T10:34:52.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:52 smithi164 ceph-mon[36970]: pgmap v7523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13121.419414 secs 2024-07-26T10:34:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13139.158224 secs 2024-07-26T10:34:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:52 smithi163 ceph-mon[36873]: pgmap v7523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:34:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:55 smithi042 ceph-mon[32775]: pgmap v7524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:34:55.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:34:55.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:34:55.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:34:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:55 smithi164 ceph-mon[36970]: pgmap v7524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:34:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:34:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:34:55.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:34:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:55 smithi163 ceph-mon[36873]: pgmap v7524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:34:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:34:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:34:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:34:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:34:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:56 smithi042 ceph-mon[32775]: pgmap v7525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:34:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:56 smithi042 ceph-mon[32775]: pgmap v7526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:34:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:56 smithi164 ceph-mon[36970]: pgmap v7525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:34:56.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:56 smithi164 ceph-mon[36970]: pgmap v7526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:34:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:56 smithi163 ceph-mon[36873]: pgmap v7525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:34:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:56 smithi163 ceph-mon[36873]: pgmap v7526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:34:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13126.419529 secs 2024-07-26T10:34:57.690 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13144.158326 secs 2024-07-26T10:34:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13126.419529 secs 2024-07-26T10:34:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13144.158326 secs 2024-07-26T10:34:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13126.419529 secs 2024-07-26T10:34:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13144.158326 secs 2024-07-26T10:34:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:34:57.983+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:34:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:34:57.983+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:34:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:34:57.983+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:34:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:34:58 smithi042 ceph-mon[32775]: pgmap v7527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:34:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:34:58 smithi164 ceph-mon[36970]: pgmap v7527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:34:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:34:58 smithi163 ceph-mon[36873]: pgmap v7527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:34:59.924 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:34:59.923+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:34:59.925 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:34:59.923+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:34:59.925 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:34:59.923+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:34:59.925 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:34:59.923+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:35:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:00 smithi042 ceph-mon[32775]: pgmap v7528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:35:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:00 smithi164 ceph-mon[36970]: pgmap v7528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:35:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:00 smithi163 ceph-mon[36873]: pgmap v7528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:35:01.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:01 smithi164 ceph-mon[36970]: pgmap v7529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:35:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:01 smithi042 ceph-mon[32775]: pgmap v7529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:35:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:01 smithi163 ceph-mon[36873]: pgmap v7529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:35:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13131.419606 secs 2024-07-26T10:35:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13149.158388 secs 2024-07-26T10:35:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13131.419606 secs 2024-07-26T10:35:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13149.158388 secs 2024-07-26T10:35:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13131.419606 secs 2024-07-26T10:35:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13149.158388 secs 2024-07-26T10:35:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:03 smithi042 ceph-mon[32775]: pgmap v7530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:35:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:03 smithi164 ceph-mon[36970]: pgmap v7530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:35:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:03 smithi163 ceph-mon[36873]: pgmap v7530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:35:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:05 smithi042 ceph-mon[32775]: pgmap v7531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:35:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:05 smithi164 ceph-mon[36970]: pgmap v7531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:35:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:05 smithi163 ceph-mon[36873]: pgmap v7531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:35:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13136.419710 secs 2024-07-26T10:35:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13136.419710 secs 2024-07-26T10:35:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13136.419710 secs 2024-07-26T10:35:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13154.158525 secs 2024-07-26T10:35:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:07 smithi042 ceph-mon[32775]: pgmap v7532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13154.158525 secs 2024-07-26T10:35:08.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:07 smithi164 ceph-mon[36970]: pgmap v7532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13154.158525 secs 2024-07-26T10:35:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:07 smithi163 ceph-mon[36873]: pgmap v7532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:09 smithi042 ceph-mon[32775]: pgmap v7533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:35:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:09 smithi164 ceph-mon[36970]: pgmap v7533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:35:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:09 smithi163 ceph-mon[36873]: pgmap v7533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:35:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:11 smithi164 ceph-mon[36970]: pgmap v7534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13141.419857 secs 2024-07-26T10:35:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:11 smithi163 ceph-mon[36873]: pgmap v7534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13141.419857 secs 2024-07-26T10:35:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:11 smithi042 ceph-mon[32775]: pgmap v7534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13141.419857 secs 2024-07-26T10:35:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:35:12.984+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:35:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:35:12.984+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:35:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:35:12.984+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:35:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13159.158667 secs 2024-07-26T10:35:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13159.158667 secs 2024-07-26T10:35:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13159.158667 secs 2024-07-26T10:35:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:13 smithi164 ceph-mon[36970]: pgmap v7535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:13 smithi163 ceph-mon[36873]: pgmap v7535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:13 smithi042 ceph-mon[32775]: pgmap v7535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:15 smithi164 ceph-mon[36970]: pgmap v7536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:15 smithi163 ceph-mon[36873]: pgmap v7536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:15 smithi042 ceph-mon[32775]: pgmap v7536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13146.419985 secs 2024-07-26T10:35:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13146.419985 secs 2024-07-26T10:35:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13146.419985 secs 2024-07-26T10:35:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13164.158759 secs 2024-07-26T10:35:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:17 smithi164 ceph-mon[36970]: pgmap v7537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13164.158759 secs 2024-07-26T10:35:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:17 smithi163 ceph-mon[36873]: pgmap v7537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13164.158759 secs 2024-07-26T10:35:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:17 smithi042 ceph-mon[32775]: pgmap v7537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:19 smithi164 ceph-mon[36970]: pgmap v7538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:35:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:19 smithi163 ceph-mon[36873]: pgmap v7538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:35:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:19 smithi042 ceph-mon[32775]: pgmap v7538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:35:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:21 smithi164 ceph-mon[36970]: pgmap v7539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13151.420113 secs 2024-07-26T10:35:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:21 smithi163 ceph-mon[36873]: pgmap v7539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13151.420113 secs 2024-07-26T10:35:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:21 smithi042 ceph-mon[32775]: pgmap v7539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13151.420113 secs 2024-07-26T10:35:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13169.158881 secs 2024-07-26T10:35:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13169.158881 secs 2024-07-26T10:35:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13169.158881 secs 2024-07-26T10:35:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:23 smithi164 ceph-mon[36970]: pgmap v7540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:23 smithi163 ceph-mon[36873]: pgmap v7540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:23 smithi042 ceph-mon[32775]: pgmap v7540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:25 smithi164 ceph-mon[36970]: pgmap v7541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:25 smithi163 ceph-mon[36873]: pgmap v7541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:25 smithi042 ceph-mon[32775]: pgmap v7541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13156.420272 secs 2024-07-26T10:35:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13156.420272 secs 2024-07-26T10:35:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13156.420272 secs 2024-07-26T10:35:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:35:27.989+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:35:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:35:27.990+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:35:27.992 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:35:27.990+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:35:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13174.158998 secs 2024-07-26T10:35:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:27 smithi164 ceph-mon[36970]: pgmap v7542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13174.158998 secs 2024-07-26T10:35:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:27 smithi163 ceph-mon[36873]: pgmap v7542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13174.158998 secs 2024-07-26T10:35:28.414 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:27 smithi042 ceph-mon[32775]: pgmap v7542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:29 smithi164 ceph-mon[36970]: pgmap v7543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:35:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:29 smithi163 ceph-mon[36873]: pgmap v7543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:35:30.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:29 smithi042 ceph-mon[32775]: pgmap v7543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:35:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:31 smithi164 ceph-mon[36970]: pgmap v7544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13161.420404 secs 2024-07-26T10:35:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:31 smithi163 ceph-mon[36873]: pgmap v7544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13161.420404 secs 2024-07-26T10:35:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:31 smithi042 ceph-mon[32775]: pgmap v7544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13161.420404 secs 2024-07-26T10:35:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13179.159119 secs 2024-07-26T10:35:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13179.159119 secs 2024-07-26T10:35:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13179.159119 secs 2024-07-26T10:35:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:33 smithi164 ceph-mon[36970]: pgmap v7545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:33 smithi163 ceph-mon[36873]: pgmap v7545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:33 smithi042 ceph-mon[32775]: pgmap v7545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:35 smithi164 ceph-mon[36970]: pgmap v7546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:35 smithi163 ceph-mon[36873]: pgmap v7546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:35 smithi042 ceph-mon[32775]: pgmap v7546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13166.420470 secs 2024-07-26T10:35:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13166.420470 secs 2024-07-26T10:35:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13166.420470 secs 2024-07-26T10:35:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13184.159249 secs 2024-07-26T10:35:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:37 smithi164 ceph-mon[36970]: pgmap v7547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13184.159249 secs 2024-07-26T10:35:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:37 smithi163 ceph-mon[36873]: pgmap v7547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13184.159249 secs 2024-07-26T10:35:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:37 smithi042 ceph-mon[32775]: pgmap v7547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:39 smithi164 ceph-mon[36970]: pgmap v7548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:35:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:39 smithi163 ceph-mon[36873]: pgmap v7548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:35:40.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:39 smithi042 ceph-mon[32775]: pgmap v7548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:35:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:41 smithi163 ceph-mon[36873]: pgmap v7549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13171.420600 secs 2024-07-26T10:35:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:41 smithi042 ceph-mon[32775]: pgmap v7549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13171.420600 secs 2024-07-26T10:35:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:41 smithi164 ceph-mon[36970]: pgmap v7549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13171.420600 secs 2024-07-26T10:35:42.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:35:42.992+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:35:42.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:35:42.992+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:35:42.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:35:42.992+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:35:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13189.159412 secs 2024-07-26T10:35:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13189.159412 secs 2024-07-26T10:35:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13189.159412 secs 2024-07-26T10:35:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:43 smithi163 ceph-mon[36873]: pgmap v7550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:43 smithi042 ceph-mon[32775]: pgmap v7550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:43 smithi164 ceph-mon[36970]: pgmap v7550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:45 smithi163 ceph-mon[36873]: pgmap v7551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:45 smithi042 ceph-mon[32775]: pgmap v7551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:45 smithi164 ceph-mon[36970]: pgmap v7551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13176.420726 secs 2024-07-26T10:35:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13176.420726 secs 2024-07-26T10:35:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13176.420726 secs 2024-07-26T10:35:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13194.159532 secs 2024-07-26T10:35:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:47 smithi163 ceph-mon[36873]: pgmap v7552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13194.159532 secs 2024-07-26T10:35:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:47 smithi042 ceph-mon[32775]: pgmap v7552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13194.159532 secs 2024-07-26T10:35:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:47 smithi164 ceph-mon[36970]: pgmap v7552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:49 smithi163 ceph-mon[36873]: pgmap v7553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:35:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:49 smithi042 ceph-mon[32775]: pgmap v7553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:35:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:49 smithi164 ceph-mon[36970]: pgmap v7553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:35:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:52 smithi163 ceph-mon[36873]: pgmap v7554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13181.420830 secs 2024-07-26T10:35:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:52 smithi042 ceph-mon[32775]: pgmap v7554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13181.420830 secs 2024-07-26T10:35:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:52 smithi164 ceph-mon[36970]: pgmap v7554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13181.420830 secs 2024-07-26T10:35:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:53 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13199.159674 secs 2024-07-26T10:35:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:53 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13199.159674 secs 2024-07-26T10:35:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:53 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13199.159674 secs 2024-07-26T10:35:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:54 smithi042 ceph-mon[32775]: pgmap v7555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:54 smithi164 ceph-mon[36970]: pgmap v7555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:54 smithi163 ceph-mon[36873]: pgmap v7555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:35:55.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:35:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:35:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:35:56.273 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:56 smithi042 ceph-mon[32775]: pgmap v7556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:56.273 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:35:56.273 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:35:56.273 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:35:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:56 smithi164 ceph-mon[36970]: pgmap v7556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:35:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:35:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:35:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:56 smithi163 ceph-mon[36873]: pgmap v7556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:35:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:35:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:35:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:35:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:57 smithi042 ceph-mon[32775]: pgmap v7557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:35:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:57 smithi042 ceph-mon[32775]: pgmap v7558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:35:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13186.420949 secs 2024-07-26T10:35:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:57 smithi164 ceph-mon[36970]: pgmap v7557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:35:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:57 smithi164 ceph-mon[36970]: pgmap v7558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:35:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13186.420949 secs 2024-07-26T10:35:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:57 smithi163 ceph-mon[36873]: pgmap v7557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:35:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:57 smithi163 ceph-mon[36873]: pgmap v7558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:35:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13186.420949 secs 2024-07-26T10:35:58.000 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:35:57.997+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:35:58.000 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:35:57.997+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:35:58.000 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:35:57.998+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:35:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:58 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13204.159744 secs 2024-07-26T10:35:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:58 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13204.159744 secs 2024-07-26T10:35:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:58 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13204.159744 secs 2024-07-26T10:35:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:35:59 smithi042 ceph-mon[32775]: pgmap v7559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:35:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:35:59 smithi164 ceph-mon[36970]: pgmap v7559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:35:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:35:59 smithi163 ceph-mon[36873]: pgmap v7559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:36:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:01 smithi042 ceph-mon[32775]: pgmap v7560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:36:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:01 smithi164 ceph-mon[36970]: pgmap v7560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:36:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:01 smithi163 ceph-mon[36873]: pgmap v7560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:36:02.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13191.421092 secs 2024-07-26T10:36:02.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13191.421092 secs 2024-07-26T10:36:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13191.421092 secs 2024-07-26T10:36:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:03 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13209.159865 secs 2024-07-26T10:36:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:03 smithi042 ceph-mon[32775]: pgmap v7561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:36:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:03 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13209.159865 secs 2024-07-26T10:36:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:03 smithi164 ceph-mon[36970]: pgmap v7561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:36:03.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:03 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13209.159865 secs 2024-07-26T10:36:03.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:03 smithi163 ceph-mon[36873]: pgmap v7561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:36:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:05 smithi042 ceph-mon[32775]: pgmap v7562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:36:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:05 smithi164 ceph-mon[36970]: pgmap v7562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:36:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:05 smithi163 ceph-mon[36873]: pgmap v7562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:36:07.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:07 smithi042 ceph-mon[32775]: pgmap v7563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:36:07.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13196.421253 secs 2024-07-26T10:36:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:07 smithi164 ceph-mon[36970]: pgmap v7563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:36:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13196.421253 secs 2024-07-26T10:36:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:07 smithi163 ceph-mon[36873]: pgmap v7563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:36:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13196.421253 secs 2024-07-26T10:36:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:08 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13214.160013 secs 2024-07-26T10:36:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:08 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13214.160013 secs 2024-07-26T10:36:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:08 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13214.160013 secs 2024-07-26T10:36:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:09 smithi042 ceph-mon[32775]: pgmap v7564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:09 smithi164 ceph-mon[36970]: pgmap v7564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:09 smithi163 ceph-mon[36873]: pgmap v7564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:11 smithi042 ceph-mon[32775]: pgmap v7565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:11 smithi164 ceph-mon[36970]: pgmap v7565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:11 smithi163 ceph-mon[36873]: pgmap v7565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13201.421376 secs 2024-07-26T10:36:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13201.421376 secs 2024-07-26T10:36:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13201.421376 secs 2024-07-26T10:36:13.002 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:36:13.000+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:36:13.002 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:36:13.000+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:36:13.002 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:36:13.000+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:36:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:13 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13219.160137 secs 2024-07-26T10:36:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:13 smithi042 ceph-mon[32775]: pgmap v7566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:13 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13219.160137 secs 2024-07-26T10:36:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:13 smithi164 ceph-mon[36970]: pgmap v7566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:13 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13219.160137 secs 2024-07-26T10:36:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:13 smithi163 ceph-mon[36873]: pgmap v7566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:15 smithi042 ceph-mon[32775]: pgmap v7567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:36:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:15 smithi164 ceph-mon[36970]: pgmap v7567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:36:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:15 smithi163 ceph-mon[36873]: pgmap v7567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:36:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:17 smithi042 ceph-mon[32775]: pgmap v7568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13206.421455 secs 2024-07-26T10:36:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13224.160236 secs 2024-07-26T10:36:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:17 smithi164 ceph-mon[36970]: pgmap v7568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13206.421455 secs 2024-07-26T10:36:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13224.160236 secs 2024-07-26T10:36:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:17 smithi163 ceph-mon[36873]: pgmap v7568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13206.421455 secs 2024-07-26T10:36:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13224.160236 secs 2024-07-26T10:36:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:19 smithi042 ceph-mon[32775]: pgmap v7569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:19 smithi164 ceph-mon[36970]: pgmap v7569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:19 smithi163 ceph-mon[36873]: pgmap v7569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:21 smithi042 ceph-mon[32775]: pgmap v7570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:21 smithi164 ceph-mon[36970]: pgmap v7570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:21 smithi163 ceph-mon[36873]: pgmap v7570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13211.421602 secs 2024-07-26T10:36:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13229.160358 secs 2024-07-26T10:36:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13211.421602 secs 2024-07-26T10:36:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13229.160358 secs 2024-07-26T10:36:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13211.421602 secs 2024-07-26T10:36:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13229.160358 secs 2024-07-26T10:36:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:23 smithi042 ceph-mon[32775]: pgmap v7571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:23 smithi164 ceph-mon[36970]: pgmap v7571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:23 smithi163 ceph-mon[36873]: pgmap v7571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:25 smithi042 ceph-mon[32775]: pgmap v7572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:36:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:25 smithi164 ceph-mon[36970]: pgmap v7572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:36:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:25 smithi163 ceph-mon[36873]: pgmap v7572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:36:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:27 smithi042 ceph-mon[32775]: pgmap v7573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13216.421706 secs 2024-07-26T10:36:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13234.160502 secs 2024-07-26T10:36:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:27 smithi164 ceph-mon[36970]: pgmap v7573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13216.421706 secs 2024-07-26T10:36:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13234.160502 secs 2024-07-26T10:36:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:27 smithi163 ceph-mon[36873]: pgmap v7573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13216.421706 secs 2024-07-26T10:36:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13234.160502 secs 2024-07-26T10:36:28.007 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:36:28.005+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:36:28.007 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:36:28.005+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:36:28.008 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:36:28.005+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:36:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:29 smithi042 ceph-mon[32775]: pgmap v7574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:29 smithi164 ceph-mon[36970]: pgmap v7574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:29 smithi163 ceph-mon[36873]: pgmap v7574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:31 smithi042 ceph-mon[32775]: pgmap v7575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:31 smithi164 ceph-mon[36970]: pgmap v7575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:31 smithi163 ceph-mon[36873]: pgmap v7575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:32.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13221.421831 secs 2024-07-26T10:36:32.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13239.160641 secs 2024-07-26T10:36:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13221.421831 secs 2024-07-26T10:36:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13239.160641 secs 2024-07-26T10:36:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13221.421831 secs 2024-07-26T10:36:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13239.160641 secs 2024-07-26T10:36:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:33 smithi042 ceph-mon[32775]: pgmap v7576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:33 smithi164 ceph-mon[36970]: pgmap v7576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:33 smithi163 ceph-mon[36873]: pgmap v7576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:35 smithi042 ceph-mon[32775]: pgmap v7577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:36:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:35 smithi164 ceph-mon[36970]: pgmap v7577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:36:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:35 smithi163 ceph-mon[36873]: pgmap v7577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:36:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:37 smithi042 ceph-mon[32775]: pgmap v7578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13226.421953 secs 2024-07-26T10:36:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13244.160707 secs 2024-07-26T10:36:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:37 smithi164 ceph-mon[36970]: pgmap v7578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13226.421953 secs 2024-07-26T10:36:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13244.160707 secs 2024-07-26T10:36:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:37 smithi163 ceph-mon[36873]: pgmap v7578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13226.421953 secs 2024-07-26T10:36:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13244.160707 secs 2024-07-26T10:36:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:39 smithi042 ceph-mon[32775]: pgmap v7579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:39 smithi164 ceph-mon[36970]: pgmap v7579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:39 smithi163 ceph-mon[36873]: pgmap v7579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:41 smithi042 ceph-mon[32775]: pgmap v7580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:41 smithi164 ceph-mon[36970]: pgmap v7580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:41 smithi163 ceph-mon[36873]: pgmap v7580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13231.422082 secs 2024-07-26T10:36:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13249.160834 secs 2024-07-26T10:36:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13231.422082 secs 2024-07-26T10:36:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13249.160834 secs 2024-07-26T10:36:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13231.422082 secs 2024-07-26T10:36:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13249.160834 secs 2024-07-26T10:36:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:36:43.007+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:36:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:36:43.007+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:36:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:36:43.007+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:36:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:43 smithi042 ceph-mon[32775]: pgmap v7581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:43 smithi164 ceph-mon[36970]: pgmap v7581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:43 smithi163 ceph-mon[36873]: pgmap v7581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:45.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:45 smithi042 ceph-mon[32775]: pgmap v7582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:36:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:45 smithi164 ceph-mon[36970]: pgmap v7582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:36:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:45 smithi163 ceph-mon[36873]: pgmap v7582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:36:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:47 smithi042 ceph-mon[32775]: pgmap v7583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13236.422232 secs 2024-07-26T10:36:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13254.160969 secs 2024-07-26T10:36:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:47 smithi164 ceph-mon[36970]: pgmap v7583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13236.422232 secs 2024-07-26T10:36:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13254.160969 secs 2024-07-26T10:36:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:47 smithi163 ceph-mon[36873]: pgmap v7583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13236.422232 secs 2024-07-26T10:36:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13254.160969 secs 2024-07-26T10:36:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:49 smithi042 ceph-mon[32775]: pgmap v7584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:49 smithi164 ceph-mon[36970]: pgmap v7584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:49 smithi163 ceph-mon[36873]: pgmap v7584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:51 smithi042 ceph-mon[32775]: pgmap v7585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:51 smithi164 ceph-mon[36970]: pgmap v7585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:51 smithi163 ceph-mon[36873]: pgmap v7585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13241.422351 secs 2024-07-26T10:36:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13259.161051 secs 2024-07-26T10:36:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13241.422351 secs 2024-07-26T10:36:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13259.161051 secs 2024-07-26T10:36:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13241.422351 secs 2024-07-26T10:36:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13259.161051 secs 2024-07-26T10:36:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:53 smithi164 ceph-mon[36970]: pgmap v7586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:53 smithi163 ceph-mon[36873]: pgmap v7586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:53 smithi042 ceph-mon[32775]: pgmap v7586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:36:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:55 smithi164 ceph-mon[36970]: pgmap v7587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:36:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:55 smithi163 ceph-mon[36873]: pgmap v7587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:36:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:55 smithi042 ceph-mon[32775]: pgmap v7587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:36:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:36:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:36:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:36:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:36:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:36:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:36:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:36:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:36:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:36:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:36:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:36:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:36:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:57 smithi164 ceph-mon[36970]: pgmap v7588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:57 smithi164 ceph-mon[36970]: pgmap v7589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:36:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:57 smithi164 ceph-mon[36970]: pgmap v7590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:36:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13246.422472 secs 2024-07-26T10:36:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13264.161167 secs 2024-07-26T10:36:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:57 smithi163 ceph-mon[36873]: pgmap v7588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:57 smithi163 ceph-mon[36873]: pgmap v7589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:36:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:57 smithi163 ceph-mon[36873]: pgmap v7590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:36:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13246.422472 secs 2024-07-26T10:36:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13264.161167 secs 2024-07-26T10:36:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:57 smithi042 ceph-mon[32775]: pgmap v7588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:36:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:57 smithi042 ceph-mon[32775]: pgmap v7589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:36:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:57 smithi042 ceph-mon[32775]: pgmap v7590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:36:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13246.422472 secs 2024-07-26T10:36:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13264.161167 secs 2024-07-26T10:36:58.014 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:36:58.013+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:36:58.015 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:36:58.013+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:36:58.015 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:36:58.013+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:36:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:36:59 smithi164 ceph-mon[36970]: pgmap v7591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:36:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:36:59 smithi163 ceph-mon[36873]: pgmap v7591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:36:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:36:59 smithi042 ceph-mon[32775]: pgmap v7591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:37:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:01 smithi164 ceph-mon[36970]: pgmap v7592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:37:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:01 smithi163 ceph-mon[36873]: pgmap v7592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:37:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:01 smithi042 ceph-mon[32775]: pgmap v7592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:37:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13251.422568 secs 2024-07-26T10:37:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13269.161263 secs 2024-07-26T10:37:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13251.422568 secs 2024-07-26T10:37:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13269.161263 secs 2024-07-26T10:37:02.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13251.422568 secs 2024-07-26T10:37:02.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13269.161263 secs 2024-07-26T10:37:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:03 smithi164 ceph-mon[36970]: pgmap v7593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:37:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:03 smithi163 ceph-mon[36873]: pgmap v7593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:37:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:03 smithi042 ceph-mon[32775]: pgmap v7593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:37:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:05 smithi164 ceph-mon[36970]: pgmap v7594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:37:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:05 smithi163 ceph-mon[36873]: pgmap v7594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:37:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:05 smithi042 ceph-mon[32775]: pgmap v7594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:37:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:07 smithi164 ceph-mon[36970]: pgmap v7595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:37:07.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13256.422691 secs 2024-07-26T10:37:07.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13274.161391 secs 2024-07-26T10:37:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:07 smithi163 ceph-mon[36873]: pgmap v7595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:37:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13256.422691 secs 2024-07-26T10:37:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13274.161391 secs 2024-07-26T10:37:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:07 smithi042 ceph-mon[32775]: pgmap v7595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:37:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13256.422691 secs 2024-07-26T10:37:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13274.161391 secs 2024-07-26T10:37:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:09 smithi164 ceph-mon[36970]: pgmap v7596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:09 smithi163 ceph-mon[36873]: pgmap v7596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:09 smithi042 ceph-mon[32775]: pgmap v7596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:11 smithi164 ceph-mon[36970]: pgmap v7597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:11 smithi163 ceph-mon[36873]: pgmap v7597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:11 smithi042 ceph-mon[32775]: pgmap v7597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13261.422812 secs 2024-07-26T10:37:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13279.161511 secs 2024-07-26T10:37:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13261.422812 secs 2024-07-26T10:37:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13279.161511 secs 2024-07-26T10:37:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13261.422812 secs 2024-07-26T10:37:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13279.161511 secs 2024-07-26T10:37:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:37:13.014+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:37:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:37:13.014+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:37:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:37:13.014+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:37:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:13 smithi164 ceph-mon[36970]: pgmap v7598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:13 smithi163 ceph-mon[36873]: pgmap v7598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:13 smithi042 ceph-mon[32775]: pgmap v7598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:15 smithi164 ceph-mon[36970]: pgmap v7599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:37:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:15 smithi163 ceph-mon[36873]: pgmap v7599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:37:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:15 smithi042 ceph-mon[32775]: pgmap v7599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:37:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:17 smithi164 ceph-mon[36970]: pgmap v7600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13266.422934 secs 2024-07-26T10:37:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13284.161612 secs 2024-07-26T10:37:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:17 smithi163 ceph-mon[36873]: pgmap v7600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13266.422934 secs 2024-07-26T10:37:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13284.161612 secs 2024-07-26T10:37:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:17 smithi042 ceph-mon[32775]: pgmap v7600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13266.422934 secs 2024-07-26T10:37:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13284.161612 secs 2024-07-26T10:37:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:19 smithi164 ceph-mon[36970]: pgmap v7601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:19 smithi163 ceph-mon[36873]: pgmap v7601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:19 smithi042 ceph-mon[32775]: pgmap v7601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:21 smithi164 ceph-mon[36970]: pgmap v7602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:21 smithi163 ceph-mon[36873]: pgmap v7602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:21 smithi042 ceph-mon[32775]: pgmap v7602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13271.423073 secs 2024-07-26T10:37:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13289.161741 secs 2024-07-26T10:37:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13271.423073 secs 2024-07-26T10:37:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13289.161741 secs 2024-07-26T10:37:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13271.423073 secs 2024-07-26T10:37:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13289.161741 secs 2024-07-26T10:37:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:23 smithi164 ceph-mon[36970]: pgmap v7603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:23 smithi163 ceph-mon[36873]: pgmap v7603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:23 smithi042 ceph-mon[32775]: pgmap v7603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:25 smithi163 ceph-mon[36873]: pgmap v7604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:37:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:25 smithi042 ceph-mon[32775]: pgmap v7604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:37:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:25 smithi164 ceph-mon[36970]: pgmap v7604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:37:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:27 smithi163 ceph-mon[36873]: pgmap v7605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13276.423238 secs 2024-07-26T10:37:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13294.161862 secs 2024-07-26T10:37:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:27 smithi042 ceph-mon[32775]: pgmap v7605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13276.423238 secs 2024-07-26T10:37:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13294.161862 secs 2024-07-26T10:37:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:27 smithi164 ceph-mon[36970]: pgmap v7605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13276.423238 secs 2024-07-26T10:37:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13294.161862 secs 2024-07-26T10:37:28.021 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:37:28.019+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:37:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:37:28.019+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:37:28.022 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:37:28.019+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:37:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:29 smithi163 ceph-mon[36873]: pgmap v7606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:29 smithi042 ceph-mon[32775]: pgmap v7606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:29 smithi164 ceph-mon[36970]: pgmap v7606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:31 smithi163 ceph-mon[36873]: pgmap v7607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:31 smithi042 ceph-mon[32775]: pgmap v7607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:31 smithi164 ceph-mon[36970]: pgmap v7607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13281.423340 secs 2024-07-26T10:37:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13299.162005 secs 2024-07-26T10:37:32.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13281.423340 secs 2024-07-26T10:37:32.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13299.162005 secs 2024-07-26T10:37:32.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13281.423340 secs 2024-07-26T10:37:32.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13299.162005 secs 2024-07-26T10:37:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:33 smithi163 ceph-mon[36873]: pgmap v7608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:33 smithi042 ceph-mon[32775]: pgmap v7608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:33 smithi164 ceph-mon[36970]: pgmap v7608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:35 smithi163 ceph-mon[36873]: pgmap v7609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:37:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:35 smithi042 ceph-mon[32775]: pgmap v7609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:37:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:35 smithi164 ceph-mon[36970]: pgmap v7609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:37:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:37 smithi163 ceph-mon[36873]: pgmap v7610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13286.423470 secs 2024-07-26T10:37:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13304.162154 secs 2024-07-26T10:37:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:37 smithi042 ceph-mon[32775]: pgmap v7610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13286.423470 secs 2024-07-26T10:37:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13304.162154 secs 2024-07-26T10:37:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:37 smithi164 ceph-mon[36970]: pgmap v7610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13286.423470 secs 2024-07-26T10:37:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13304.162154 secs 2024-07-26T10:37:39.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:39 smithi163 ceph-mon[36873]: pgmap v7611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:39 smithi042 ceph-mon[32775]: pgmap v7611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:39 smithi164 ceph-mon[36970]: pgmap v7611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:41 smithi163 ceph-mon[36873]: pgmap v7612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:41 smithi042 ceph-mon[32775]: pgmap v7612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:41 smithi164 ceph-mon[36970]: pgmap v7612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13291.423593 secs 2024-07-26T10:37:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13309.162298 secs 2024-07-26T10:37:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13291.423593 secs 2024-07-26T10:37:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13309.162298 secs 2024-07-26T10:37:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13291.423593 secs 2024-07-26T10:37:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13309.162298 secs 2024-07-26T10:37:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:37:43.021+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:37:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:37:43.021+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:37:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:37:43.021+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:37:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:43 smithi042 ceph-mon[32775]: pgmap v7613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:43 smithi164 ceph-mon[36970]: pgmap v7613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:43 smithi163 ceph-mon[36873]: pgmap v7613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:45 smithi042 ceph-mon[32775]: pgmap v7614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:37:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:45 smithi164 ceph-mon[36970]: pgmap v7614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:37:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:45 smithi163 ceph-mon[36873]: pgmap v7614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:37:47.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:47 smithi163 ceph-mon[36873]: pgmap v7615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:47.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13296.423694 secs 2024-07-26T10:37:47.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13314.162419 secs 2024-07-26T10:37:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:47 smithi042 ceph-mon[32775]: pgmap v7615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13296.423694 secs 2024-07-26T10:37:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13314.162419 secs 2024-07-26T10:37:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:47 smithi164 ceph-mon[36970]: pgmap v7615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13296.423694 secs 2024-07-26T10:37:47.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13314.162419 secs 2024-07-26T10:37:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:49 smithi042 ceph-mon[32775]: pgmap v7616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:49 smithi164 ceph-mon[36970]: pgmap v7616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:49 smithi163 ceph-mon[36873]: pgmap v7616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:51 smithi042 ceph-mon[32775]: pgmap v7617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:51 smithi164 ceph-mon[36970]: pgmap v7617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:51 smithi163 ceph-mon[36873]: pgmap v7617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13301.423820 secs 2024-07-26T10:37:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13319.162543 secs 2024-07-26T10:37:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13301.423820 secs 2024-07-26T10:37:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13319.162543 secs 2024-07-26T10:37:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13301.423820 secs 2024-07-26T10:37:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13319.162543 secs 2024-07-26T10:37:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:53 smithi042 ceph-mon[32775]: pgmap v7618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:53 smithi164 ceph-mon[36970]: pgmap v7618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:53 smithi163 ceph-mon[36873]: pgmap v7618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:37:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:55 smithi042 ceph-mon[32775]: pgmap v7619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:37:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:55 smithi164 ceph-mon[36970]: pgmap v7619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:37:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:55 smithi163 ceph-mon[36873]: pgmap v7619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:37:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:37:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:37:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:37:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:57 smithi042 ceph-mon[32775]: pgmap v7620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13306.423970 secs 2024-07-26T10:37:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13324.162614 secs 2024-07-26T10:37:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:37:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:37:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:37:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:57 smithi164 ceph-mon[36970]: pgmap v7620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13306.423970 secs 2024-07-26T10:37:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13324.162614 secs 2024-07-26T10:37:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:37:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:37:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:37:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:57 smithi163 ceph-mon[36873]: pgmap v7620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:37:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13306.423970 secs 2024-07-26T10:37:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13324.162614 secs 2024-07-26T10:37:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:37:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:37:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:37:58.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:37:58.026+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:37:58.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:37:58.026+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:37:58.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:37:58.026+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:37:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:58 smithi042 ceph-mon[32775]: pgmap v7621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:37:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:37:58 smithi042 ceph-mon[32775]: pgmap v7622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:37:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:58 smithi164 ceph-mon[36970]: pgmap v7621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:37:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:37:58 smithi164 ceph-mon[36970]: pgmap v7622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:37:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:58 smithi163 ceph-mon[36873]: pgmap v7621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:37:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:37:58 smithi163 ceph-mon[36873]: pgmap v7622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:37:59.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:37:59.950+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:37:59.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:37:59.950+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:37:59.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:37:59.951+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:37:59.953 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:37:59.951+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:38:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:00 smithi042 ceph-mon[32775]: pgmap v7623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:38:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:00 smithi164 ceph-mon[36970]: pgmap v7623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:38:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:00 smithi163 ceph-mon[36873]: pgmap v7623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:38:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:02 smithi042 ceph-mon[32775]: pgmap v7624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:38:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13311.424123 secs 2024-07-26T10:38:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13329.162694 secs 2024-07-26T10:38:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:02 smithi164 ceph-mon[36970]: pgmap v7624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:38:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13311.424123 secs 2024-07-26T10:38:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13329.162694 secs 2024-07-26T10:38:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:02 smithi163 ceph-mon[36873]: pgmap v7624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:38:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13311.424123 secs 2024-07-26T10:38:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13329.162694 secs 2024-07-26T10:38:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:04 smithi042 ceph-mon[32775]: pgmap v7625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:38:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:04 smithi164 ceph-mon[36970]: pgmap v7625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:38:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:04 smithi163 ceph-mon[36873]: pgmap v7625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:38:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:06 smithi042 ceph-mon[32775]: pgmap v7626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:38:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:06 smithi164 ceph-mon[36970]: pgmap v7626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:38:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:06 smithi163 ceph-mon[36873]: pgmap v7626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:38:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13316.424283 secs 2024-07-26T10:38:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13334.162813 secs 2024-07-26T10:38:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13316.424283 secs 2024-07-26T10:38:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13334.162813 secs 2024-07-26T10:38:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13316.424283 secs 2024-07-26T10:38:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13334.162813 secs 2024-07-26T10:38:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:08 smithi042 ceph-mon[32775]: pgmap v7627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:38:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:08 smithi164 ceph-mon[36970]: pgmap v7627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:38:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:08 smithi163 ceph-mon[36873]: pgmap v7627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:38:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:10 smithi042 ceph-mon[32775]: pgmap v7628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:38:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:10 smithi164 ceph-mon[36970]: pgmap v7628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:38:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:10 smithi163 ceph-mon[36873]: pgmap v7628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:38:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:12 smithi042 ceph-mon[32775]: pgmap v7629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13321.424359 secs 2024-07-26T10:38:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13339.162956 secs 2024-07-26T10:38:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:12 smithi164 ceph-mon[36970]: pgmap v7629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13321.424359 secs 2024-07-26T10:38:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13339.162956 secs 2024-07-26T10:38:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:12 smithi163 ceph-mon[36873]: pgmap v7629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13321.424359 secs 2024-07-26T10:38:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13339.162956 secs 2024-07-26T10:38:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:38:13.027+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:38:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:38:13.027+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:38:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:38:13.027+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:38:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:14 smithi042 ceph-mon[32775]: pgmap v7630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:14 smithi164 ceph-mon[36970]: pgmap v7630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:14 smithi163 ceph-mon[36873]: pgmap v7630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:16 smithi042 ceph-mon[32775]: pgmap v7631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:16 smithi164 ceph-mon[36970]: pgmap v7631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:16 smithi163 ceph-mon[36873]: pgmap v7631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13326.424445 secs 2024-07-26T10:38:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13344.163047 secs 2024-07-26T10:38:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13326.424445 secs 2024-07-26T10:38:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13344.163047 secs 2024-07-26T10:38:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13326.424445 secs 2024-07-26T10:38:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13344.163047 secs 2024-07-26T10:38:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:18 smithi042 ceph-mon[32775]: pgmap v7632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:18 smithi164 ceph-mon[36970]: pgmap v7632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:18 smithi163 ceph-mon[36873]: pgmap v7632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:20 smithi042 ceph-mon[32775]: pgmap v7633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:38:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:20 smithi164 ceph-mon[36970]: pgmap v7633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:38:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:20 smithi163 ceph-mon[36873]: pgmap v7633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:38:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:22 smithi042 ceph-mon[32775]: pgmap v7634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13331.424568 secs 2024-07-26T10:38:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13349.163171 secs 2024-07-26T10:38:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:22 smithi164 ceph-mon[36970]: pgmap v7634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13331.424568 secs 2024-07-26T10:38:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13349.163171 secs 2024-07-26T10:38:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:22 smithi163 ceph-mon[36873]: pgmap v7634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13331.424568 secs 2024-07-26T10:38:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13349.163171 secs 2024-07-26T10:38:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:24 smithi042 ceph-mon[32775]: pgmap v7635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:24 smithi164 ceph-mon[36970]: pgmap v7635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:24 smithi163 ceph-mon[36873]: pgmap v7635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:26 smithi042 ceph-mon[32775]: pgmap v7636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:26 smithi164 ceph-mon[36970]: pgmap v7636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:26 smithi163 ceph-mon[36873]: pgmap v7636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13336.424654 secs 2024-07-26T10:38:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13354.163299 secs 2024-07-26T10:38:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13336.424654 secs 2024-07-26T10:38:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13354.163299 secs 2024-07-26T10:38:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13336.424654 secs 2024-07-26T10:38:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13354.163299 secs 2024-07-26T10:38:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:38:28.032+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:38:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:38:28.032+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:38:28.034 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:38:28.032+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:38:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:28 smithi042 ceph-mon[32775]: pgmap v7637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:28 smithi164 ceph-mon[36970]: pgmap v7637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:28 smithi163 ceph-mon[36873]: pgmap v7637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:30 smithi042 ceph-mon[32775]: pgmap v7638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:38:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:30 smithi164 ceph-mon[36970]: pgmap v7638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:38:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:30 smithi163 ceph-mon[36873]: pgmap v7638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:38:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:32 smithi042 ceph-mon[32775]: pgmap v7639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13341.424775 secs 2024-07-26T10:38:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13359.163392 secs 2024-07-26T10:38:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:32 smithi164 ceph-mon[36970]: pgmap v7639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13341.424775 secs 2024-07-26T10:38:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13359.163392 secs 2024-07-26T10:38:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:32 smithi163 ceph-mon[36873]: pgmap v7639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13341.424775 secs 2024-07-26T10:38:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13359.163392 secs 2024-07-26T10:38:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:34 smithi042 ceph-mon[32775]: pgmap v7640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:34 smithi164 ceph-mon[36970]: pgmap v7640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:34 smithi163 ceph-mon[36873]: pgmap v7640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:36 smithi042 ceph-mon[32775]: pgmap v7641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:36 smithi164 ceph-mon[36970]: pgmap v7641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:36 smithi163 ceph-mon[36873]: pgmap v7641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13346.424852 secs 2024-07-26T10:38:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13364.163457 secs 2024-07-26T10:38:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13346.424852 secs 2024-07-26T10:38:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13364.163457 secs 2024-07-26T10:38:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13346.424852 secs 2024-07-26T10:38:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13364.163457 secs 2024-07-26T10:38:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:38 smithi042 ceph-mon[32775]: pgmap v7642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:38 smithi164 ceph-mon[36970]: pgmap v7642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:38 smithi163 ceph-mon[36873]: pgmap v7642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:40 smithi042 ceph-mon[32775]: pgmap v7643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:38:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:40 smithi164 ceph-mon[36970]: pgmap v7643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:38:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:40 smithi163 ceph-mon[36873]: pgmap v7643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:38:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:42 smithi042 ceph-mon[32775]: pgmap v7644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13351.424969 secs 2024-07-26T10:38:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13369.163587 secs 2024-07-26T10:38:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:42 smithi164 ceph-mon[36970]: pgmap v7644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13351.424969 secs 2024-07-26T10:38:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13369.163587 secs 2024-07-26T10:38:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:42 smithi163 ceph-mon[36873]: pgmap v7644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13351.424969 secs 2024-07-26T10:38:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13369.163587 secs 2024-07-26T10:38:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:38:43.033+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:38:43.036 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:38:43.033+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:38:43.036 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:38:43.033+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:38:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:44 smithi042 ceph-mon[32775]: pgmap v7645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:44 smithi164 ceph-mon[36970]: pgmap v7645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:44 smithi163 ceph-mon[36873]: pgmap v7645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:44.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:38:44.959+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:38:44.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:38:44.959+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:38:44.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:38:44.959+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:38:44.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:38:44.959+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:38:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:46 smithi164 ceph-mon[36970]: pgmap v7646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:46 smithi163 ceph-mon[36873]: pgmap v7646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:46 smithi042 ceph-mon[32775]: pgmap v7646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13356.425114 secs 2024-07-26T10:38:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13374.163707 secs 2024-07-26T10:38:47.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13356.425114 secs 2024-07-26T10:38:47.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13374.163707 secs 2024-07-26T10:38:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13356.425114 secs 2024-07-26T10:38:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13374.163707 secs 2024-07-26T10:38:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:48 smithi164 ceph-mon[36970]: pgmap v7647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:48 smithi163 ceph-mon[36873]: pgmap v7647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:48 smithi042 ceph-mon[32775]: pgmap v7647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:49 smithi164 ceph-mon[36970]: pgmap v7648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:38:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:49 smithi163 ceph-mon[36873]: pgmap v7648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:38:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:49 smithi042 ceph-mon[32775]: pgmap v7648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:38:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:51 smithi042 ceph-mon[32775]: pgmap v7649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:51 smithi164 ceph-mon[36970]: pgmap v7649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:51 smithi163 ceph-mon[36873]: pgmap v7649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13361.425255 secs 2024-07-26T10:38:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13379.163843 secs 2024-07-26T10:38:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13361.425255 secs 2024-07-26T10:38:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13379.163843 secs 2024-07-26T10:38:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13361.425255 secs 2024-07-26T10:38:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13379.163843 secs 2024-07-26T10:38:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:53 smithi042 ceph-mon[32775]: pgmap v7650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:53 smithi164 ceph-mon[36970]: pgmap v7650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:53 smithi163 ceph-mon[36873]: pgmap v7650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:55 smithi042 ceph-mon[32775]: pgmap v7651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:55 smithi164 ceph-mon[36970]: pgmap v7651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:55 smithi163 ceph-mon[36873]: pgmap v7651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:38:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13366.425349 secs 2024-07-26T10:38:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13384.163986 secs 2024-07-26T10:38:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:57 smithi042 ceph-mon[32775]: pgmap v7652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:38:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:38:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:38:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:57 smithi042 ceph-mon[32775]: pgmap v7653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:38:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:57 smithi042 ceph-mon[32775]: pgmap v7654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:38:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:38:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:38:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13366.425349 secs 2024-07-26T10:38:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13384.163986 secs 2024-07-26T10:38:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:57 smithi164 ceph-mon[36970]: pgmap v7652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:38:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:38:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:38:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:57 smithi164 ceph-mon[36970]: pgmap v7653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:38:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:57 smithi164 ceph-mon[36970]: pgmap v7654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:38:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:38:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:38:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13366.425349 secs 2024-07-26T10:38:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13384.163986 secs 2024-07-26T10:38:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:57 smithi163 ceph-mon[36873]: pgmap v7652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:38:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:38:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:38:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:38:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:57 smithi163 ceph-mon[36873]: pgmap v7653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:38:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:57 smithi163 ceph-mon[36873]: pgmap v7654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:38:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:38:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:38:58.038 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:38:58.036+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:38:58.038 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:38:58.036+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:38:58.038 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:38:58.036+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:39:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:00 smithi042 ceph-mon[32775]: pgmap v7655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:39:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:00 smithi164 ceph-mon[36970]: pgmap v7655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:39:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:00 smithi163 ceph-mon[36873]: pgmap v7655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:39:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13371.425416 secs 2024-07-26T10:39:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:02 smithi042 ceph-mon[32775]: pgmap v7656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:39:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13389.164107 secs 2024-07-26T10:39:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13371.425416 secs 2024-07-26T10:39:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:02 smithi164 ceph-mon[36970]: pgmap v7656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:39:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13389.164107 secs 2024-07-26T10:39:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13371.425416 secs 2024-07-26T10:39:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:02 smithi163 ceph-mon[36873]: pgmap v7656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:39:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13389.164107 secs 2024-07-26T10:39:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:04 smithi042 ceph-mon[32775]: pgmap v7657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:39:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:04 smithi164 ceph-mon[36970]: pgmap v7657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:39:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:04 smithi163 ceph-mon[36873]: pgmap v7657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:39:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:06 smithi042 ceph-mon[32775]: pgmap v7658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:39:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:06 smithi164 ceph-mon[36970]: pgmap v7658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:39:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:06 smithi163 ceph-mon[36873]: pgmap v7658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:39:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13376.425541 secs 2024-07-26T10:39:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13394.164226 secs 2024-07-26T10:39:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13376.425541 secs 2024-07-26T10:39:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13394.164226 secs 2024-07-26T10:39:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13376.425541 secs 2024-07-26T10:39:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13394.164226 secs 2024-07-26T10:39:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:08 smithi042 ceph-mon[32775]: pgmap v7659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:39:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:08 smithi164 ceph-mon[36970]: pgmap v7659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:39:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:08 smithi163 ceph-mon[36873]: pgmap v7659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:39:10.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:10 smithi042 ceph-mon[32775]: pgmap v7660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:10 smithi164 ceph-mon[36970]: pgmap v7660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:10 smithi163 ceph-mon[36873]: pgmap v7660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13381.425639 secs 2024-07-26T10:39:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:12 smithi042 ceph-mon[32775]: pgmap v7661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13399.164298 secs 2024-07-26T10:39:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13381.425639 secs 2024-07-26T10:39:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:12 smithi164 ceph-mon[36970]: pgmap v7661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13399.164298 secs 2024-07-26T10:39:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13381.425639 secs 2024-07-26T10:39:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:12 smithi163 ceph-mon[36873]: pgmap v7661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13399.164298 secs 2024-07-26T10:39:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:39:13.036+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:39:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:39:13.036+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:39:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:39:13.036+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:39:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:14 smithi042 ceph-mon[32775]: pgmap v7662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:14 smithi164 ceph-mon[36970]: pgmap v7662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:14.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:14 smithi163 ceph-mon[36873]: pgmap v7662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:16 smithi164 ceph-mon[36970]: pgmap v7663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:16 smithi163 ceph-mon[36873]: pgmap v7663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:16 smithi042 ceph-mon[32775]: pgmap v7663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:17.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13386.425781 secs 2024-07-26T10:39:17.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13404.164417 secs 2024-07-26T10:39:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13386.425781 secs 2024-07-26T10:39:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13404.164417 secs 2024-07-26T10:39:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13386.425781 secs 2024-07-26T10:39:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13404.164417 secs 2024-07-26T10:39:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:18 smithi164 ceph-mon[36970]: pgmap v7664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:18 smithi163 ceph-mon[36873]: pgmap v7664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:18 smithi042 ceph-mon[32775]: pgmap v7664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:20 smithi164 ceph-mon[36970]: pgmap v7665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:20 smithi163 ceph-mon[36873]: pgmap v7665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:20 smithi042 ceph-mon[32775]: pgmap v7665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13391.425900 secs 2024-07-26T10:39:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:22 smithi164 ceph-mon[36970]: pgmap v7666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13409.164540 secs 2024-07-26T10:39:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13391.425900 secs 2024-07-26T10:39:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:22 smithi163 ceph-mon[36873]: pgmap v7666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13409.164540 secs 2024-07-26T10:39:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13391.425900 secs 2024-07-26T10:39:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:22 smithi042 ceph-mon[32775]: pgmap v7666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13409.164540 secs 2024-07-26T10:39:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:24 smithi164 ceph-mon[36970]: pgmap v7667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:24 smithi163 ceph-mon[36873]: pgmap v7667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:24 smithi042 ceph-mon[32775]: pgmap v7667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:26 smithi164 ceph-mon[36970]: pgmap v7668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:26 smithi163 ceph-mon[36873]: pgmap v7668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:26 smithi042 ceph-mon[32775]: pgmap v7668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13396.426034 secs 2024-07-26T10:39:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13414.164682 secs 2024-07-26T10:39:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13396.426034 secs 2024-07-26T10:39:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13414.164682 secs 2024-07-26T10:39:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13396.426034 secs 2024-07-26T10:39:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13414.164682 secs 2024-07-26T10:39:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:39:28.042+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:39:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:39:28.042+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:39:28.044 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:39:28.042+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:39:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:28 smithi164 ceph-mon[36970]: pgmap v7669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:28 smithi163 ceph-mon[36873]: pgmap v7669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:28 smithi042 ceph-mon[32775]: pgmap v7669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:30 smithi164 ceph-mon[36970]: pgmap v7670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:30.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:30 smithi163 ceph-mon[36873]: pgmap v7670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:30 smithi042 ceph-mon[32775]: pgmap v7670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:32.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13401.426155 secs 2024-07-26T10:39:32.449 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:32 smithi042 ceph-mon[32775]: pgmap v7671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:32.449 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13419.164785 secs 2024-07-26T10:39:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13401.426155 secs 2024-07-26T10:39:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:32 smithi164 ceph-mon[36970]: pgmap v7671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13419.164785 secs 2024-07-26T10:39:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13401.426155 secs 2024-07-26T10:39:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:32 smithi163 ceph-mon[36873]: pgmap v7671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13419.164785 secs 2024-07-26T10:39:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:34 smithi042 ceph-mon[32775]: pgmap v7672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:34 smithi163 ceph-mon[36873]: pgmap v7672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:34 smithi164 ceph-mon[36970]: pgmap v7672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:36 smithi042 ceph-mon[32775]: pgmap v7673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:36 smithi164 ceph-mon[36970]: pgmap v7673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:36 smithi163 ceph-mon[36873]: pgmap v7673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13406.426288 secs 2024-07-26T10:39:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13424.164916 secs 2024-07-26T10:39:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13406.426288 secs 2024-07-26T10:39:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13424.164916 secs 2024-07-26T10:39:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13406.426288 secs 2024-07-26T10:39:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13424.164916 secs 2024-07-26T10:39:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:38 smithi042 ceph-mon[32775]: pgmap v7674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:38 smithi164 ceph-mon[36970]: pgmap v7674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:38 smithi163 ceph-mon[36873]: pgmap v7674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:40.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:40 smithi042 ceph-mon[32775]: pgmap v7675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:40 smithi164 ceph-mon[36970]: pgmap v7675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:40 smithi163 ceph-mon[36873]: pgmap v7675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13411.426433 secs 2024-07-26T10:39:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:42 smithi042 ceph-mon[32775]: pgmap v7676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13429.165042 secs 2024-07-26T10:39:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13411.426433 secs 2024-07-26T10:39:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:42 smithi164 ceph-mon[36970]: pgmap v7676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13429.165042 secs 2024-07-26T10:39:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13411.426433 secs 2024-07-26T10:39:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:42 smithi163 ceph-mon[36873]: pgmap v7676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13429.165042 secs 2024-07-26T10:39:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:39:43.043+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:39:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:39:43.043+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:39:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:39:43.043+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:39:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:44 smithi164 ceph-mon[36970]: pgmap v7677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:44 smithi163 ceph-mon[36873]: pgmap v7677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:44 smithi042 ceph-mon[32775]: pgmap v7677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:46 smithi164 ceph-mon[36970]: pgmap v7678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:46 smithi163 ceph-mon[36873]: pgmap v7678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:46 smithi042 ceph-mon[32775]: pgmap v7678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:47.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13416.426538 secs 2024-07-26T10:39:47.410 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13434.165138 secs 2024-07-26T10:39:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13416.426538 secs 2024-07-26T10:39:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13434.165138 secs 2024-07-26T10:39:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13416.426538 secs 2024-07-26T10:39:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13434.165138 secs 2024-07-26T10:39:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:48 smithi164 ceph-mon[36970]: pgmap v7679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:48 smithi163 ceph-mon[36873]: pgmap v7679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:48 smithi042 ceph-mon[32775]: pgmap v7679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:50 smithi164 ceph-mon[36970]: pgmap v7680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:50 smithi163 ceph-mon[36873]: pgmap v7680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:50 smithi042 ceph-mon[32775]: pgmap v7680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:39:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13421.426664 secs 2024-07-26T10:39:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:52 smithi164 ceph-mon[36970]: pgmap v7681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13439.165286 secs 2024-07-26T10:39:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13421.426664 secs 2024-07-26T10:39:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:52 smithi163 ceph-mon[36873]: pgmap v7681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13439.165286 secs 2024-07-26T10:39:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13421.426664 secs 2024-07-26T10:39:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:52 smithi042 ceph-mon[32775]: pgmap v7681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13439.165286 secs 2024-07-26T10:39:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:54 smithi164 ceph-mon[36970]: pgmap v7682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:54 smithi163 ceph-mon[36873]: pgmap v7682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:54 smithi042 ceph-mon[32775]: pgmap v7682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:39:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:56 smithi164 ceph-mon[36970]: pgmap v7683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:39:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:56 smithi163 ceph-mon[36873]: pgmap v7683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:39:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:56 smithi042 ceph-mon[32775]: pgmap v7683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:39:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13426.426811 secs 2024-07-26T10:39:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13444.165404 secs 2024-07-26T10:39:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13426.426811 secs 2024-07-26T10:39:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13444.165404 secs 2024-07-26T10:39:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13426.426811 secs 2024-07-26T10:39:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13444.165404 secs 2024-07-26T10:39:58.050 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:39:58.048+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:39:58.050 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:39:58.048+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:39:58.050 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:39:58.048+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:39:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:58 smithi164 ceph-mon[36970]: pgmap v7684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:39:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:39:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:39:58.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:39:58.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:39:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:58 smithi163 ceph-mon[36873]: pgmap v7684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:39:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:39:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:39:58.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:39:58.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:39:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:58 smithi042 ceph-mon[32775]: pgmap v7684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:39:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:39:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:39:58.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:39:58.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:39:58.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:39:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:59 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T10:39:59.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:59 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T10:39:59.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:59 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T10:39:59.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:59 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:39:59.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:59 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T10:39:59.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:59 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T10:39:59.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:59 smithi164 ceph-mon[36970]: pgmap v7685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:39:59.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:39:59 smithi164 ceph-mon[36970]: pgmap v7686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 958 B/s rd, 2 op/s 2024-07-26T10:39:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:59 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T10:39:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:59 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T10:39:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:59 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T10:39:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:59 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:39:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:59 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T10:39:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:59 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T10:39:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:59 smithi163 ceph-mon[36873]: pgmap v7685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:39:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:39:59 smithi163 ceph-mon[36873]: pgmap v7686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 958 B/s rd, 2 op/s 2024-07-26T10:39:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:59 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T10:39:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:59 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T10:39:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:59 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T10:39:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:59 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:39:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:59 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T10:39:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:59 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T10:39:59.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:59 smithi042 ceph-mon[32775]: pgmap v7685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:39:59.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:39:59 smithi042 ceph-mon[32775]: pgmap v7686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 958 B/s rd, 2 op/s 2024-07-26T10:40:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:40:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:40:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:40:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:01 smithi164 ceph-mon[36970]: pgmap v7687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T10:40:01.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:01 smithi163 ceph-mon[36873]: pgmap v7687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T10:40:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:01 smithi042 ceph-mon[32775]: pgmap v7687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T10:40:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13431.426935 secs 2024-07-26T10:40:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13449.165527 secs 2024-07-26T10:40:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13431.426935 secs 2024-07-26T10:40:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13449.165527 secs 2024-07-26T10:40:02.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13431.426935 secs 2024-07-26T10:40:02.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13449.165527 secs 2024-07-26T10:40:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:03 smithi164 ceph-mon[36970]: pgmap v7688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T10:40:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:03 smithi163 ceph-mon[36873]: pgmap v7688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T10:40:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:03 smithi042 ceph-mon[32775]: pgmap v7688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T10:40:05.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:05 smithi042 ceph-mon[32775]: pgmap v7689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:40:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:05 smithi164 ceph-mon[36970]: pgmap v7689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:40:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:05 smithi163 ceph-mon[36873]: pgmap v7689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:40:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:07 smithi164 ceph-mon[36970]: pgmap v7690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:40:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13436.427084 secs 2024-07-26T10:40:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13454.165652 secs 2024-07-26T10:40:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:07 smithi163 ceph-mon[36873]: pgmap v7690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:40:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13436.427084 secs 2024-07-26T10:40:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13454.165652 secs 2024-07-26T10:40:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:07 smithi042 ceph-mon[32775]: pgmap v7690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:40:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13436.427084 secs 2024-07-26T10:40:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13454.165652 secs 2024-07-26T10:40:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:09 smithi164 ceph-mon[36970]: pgmap v7691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:40:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:09 smithi163 ceph-mon[36873]: pgmap v7691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:40:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:09 smithi042 ceph-mon[32775]: pgmap v7691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:40:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:11 smithi164 ceph-mon[36970]: pgmap v7692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:11 smithi163 ceph-mon[36873]: pgmap v7692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:11 smithi042 ceph-mon[32775]: pgmap v7692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13441.427254 secs 2024-07-26T10:40:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13459.165783 secs 2024-07-26T10:40:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13441.427254 secs 2024-07-26T10:40:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13459.165783 secs 2024-07-26T10:40:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13441.427254 secs 2024-07-26T10:40:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13459.165783 secs 2024-07-26T10:40:13.051 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:13.049+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:40:13.051 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:13.049+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:40:13.051 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:13.049+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:40:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:13 smithi164 ceph-mon[36970]: pgmap v7693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:13 smithi163 ceph-mon[36873]: pgmap v7693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:13 smithi042 ceph-mon[32775]: pgmap v7693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:15 smithi163 ceph-mon[36873]: pgmap v7694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:40:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:15 smithi042 ceph-mon[32775]: pgmap v7694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:40:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:15 smithi164 ceph-mon[36970]: pgmap v7694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:40:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:17 smithi163 ceph-mon[36873]: pgmap v7695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13446.427356 secs 2024-07-26T10:40:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13464.165849 secs 2024-07-26T10:40:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:17 smithi042 ceph-mon[32775]: pgmap v7695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13446.427356 secs 2024-07-26T10:40:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13464.165849 secs 2024-07-26T10:40:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:17 smithi164 ceph-mon[36970]: pgmap v7695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13446.427356 secs 2024-07-26T10:40:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13464.165849 secs 2024-07-26T10:40:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:19 smithi163 ceph-mon[36873]: pgmap v7696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:19 smithi042 ceph-mon[32775]: pgmap v7696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:19 smithi164 ceph-mon[36970]: pgmap v7696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:21 smithi163 ceph-mon[36873]: pgmap v7697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:21 smithi042 ceph-mon[32775]: pgmap v7697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:21 smithi164 ceph-mon[36970]: pgmap v7697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13451.427481 secs 2024-07-26T10:40:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13469.165970 secs 2024-07-26T10:40:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13451.427481 secs 2024-07-26T10:40:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13469.165970 secs 2024-07-26T10:40:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13451.427481 secs 2024-07-26T10:40:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13469.165970 secs 2024-07-26T10:40:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:23 smithi163 ceph-mon[36873]: pgmap v7698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:23 smithi042 ceph-mon[32775]: pgmap v7698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:23 smithi164 ceph-mon[36970]: pgmap v7698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:25 smithi163 ceph-mon[36873]: pgmap v7699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:40:25.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:25 smithi042 ceph-mon[32775]: pgmap v7699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:40:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:25 smithi164 ceph-mon[36970]: pgmap v7699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:40:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:27 smithi163 ceph-mon[36873]: pgmap v7700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13456.427619 secs 2024-07-26T10:40:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13474.166096 secs 2024-07-26T10:40:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:27 smithi042 ceph-mon[32775]: pgmap v7700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13456.427619 secs 2024-07-26T10:40:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13474.166096 secs 2024-07-26T10:40:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:27 smithi164 ceph-mon[36970]: pgmap v7700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13456.427619 secs 2024-07-26T10:40:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13474.166096 secs 2024-07-26T10:40:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:28.053+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:40:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:28.053+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:40:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:28.053+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:40:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:29 smithi163 ceph-mon[36873]: pgmap v7701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:29 smithi042 ceph-mon[32775]: pgmap v7701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:29 smithi164 ceph-mon[36970]: pgmap v7701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:31.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:31 smithi163 ceph-mon[36873]: pgmap v7702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:31 smithi042 ceph-mon[32775]: pgmap v7702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:31 smithi164 ceph-mon[36970]: pgmap v7702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13461.427746 secs 2024-07-26T10:40:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13479.166217 secs 2024-07-26T10:40:32.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13461.427746 secs 2024-07-26T10:40:32.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13479.166217 secs 2024-07-26T10:40:32.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13461.427746 secs 2024-07-26T10:40:32.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13479.166217 secs 2024-07-26T10:40:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:33 smithi042 ceph-mon[32775]: pgmap v7703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:33 smithi164 ceph-mon[36970]: pgmap v7703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:33 smithi163 ceph-mon[36873]: pgmap v7703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:35 smithi042 ceph-mon[32775]: pgmap v7704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:40:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:35 smithi164 ceph-mon[36970]: pgmap v7704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:40:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:35 smithi163 ceph-mon[36873]: pgmap v7704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:40:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:37 smithi042 ceph-mon[32775]: pgmap v7705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13466.427831 secs 2024-07-26T10:40:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13484.166300 secs 2024-07-26T10:40:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:37 smithi164 ceph-mon[36970]: pgmap v7705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13466.427831 secs 2024-07-26T10:40:37.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13484.166300 secs 2024-07-26T10:40:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:37 smithi163 ceph-mon[36873]: pgmap v7705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13466.427831 secs 2024-07-26T10:40:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13484.166300 secs 2024-07-26T10:40:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:39 smithi042 ceph-mon[32775]: pgmap v7706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:39 smithi164 ceph-mon[36970]: pgmap v7706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:39 smithi163 ceph-mon[36873]: pgmap v7706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:41 smithi042 ceph-mon[32775]: pgmap v7707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:41 smithi164 ceph-mon[36970]: pgmap v7707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:41 smithi163 ceph-mon[36873]: pgmap v7707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13471.427957 secs 2024-07-26T10:40:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13489.166413 secs 2024-07-26T10:40:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13471.427957 secs 2024-07-26T10:40:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13489.166413 secs 2024-07-26T10:40:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13471.427957 secs 2024-07-26T10:40:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13489.166413 secs 2024-07-26T10:40:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:43.054+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:40:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:43.054+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:40:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:43.054+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:40:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:43 smithi042 ceph-mon[32775]: pgmap v7708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:43 smithi164 ceph-mon[36970]: pgmap v7708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:43 smithi163 ceph-mon[36873]: pgmap v7708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:45 smithi042 ceph-mon[32775]: pgmap v7709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:40:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:45 smithi164 ceph-mon[36970]: pgmap v7709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:40:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:45 smithi163 ceph-mon[36873]: pgmap v7709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:40:47.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:47 smithi163 ceph-mon[36873]: pgmap v7710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:47.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13476.428086 secs 2024-07-26T10:40:47.561 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13494.166537 secs 2024-07-26T10:40:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:47 smithi042 ceph-mon[32775]: pgmap v7710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13476.428086 secs 2024-07-26T10:40:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13494.166537 secs 2024-07-26T10:40:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:47 smithi164 ceph-mon[36970]: pgmap v7710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13476.428086 secs 2024-07-26T10:40:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13494.166537 secs 2024-07-26T10:40:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:49 smithi042 ceph-mon[32775]: pgmap v7711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:49 smithi164 ceph-mon[36970]: pgmap v7711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:49 smithi163 ceph-mon[36873]: pgmap v7711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:51.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:51 smithi042 ceph-mon[32775]: pgmap v7712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:51 smithi164 ceph-mon[36970]: pgmap v7712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:51 smithi163 ceph-mon[36873]: pgmap v7712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13481.428209 secs 2024-07-26T10:40:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13499.166603 secs 2024-07-26T10:40:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13481.428209 secs 2024-07-26T10:40:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13499.166603 secs 2024-07-26T10:40:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13481.428209 secs 2024-07-26T10:40:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13499.166603 secs 2024-07-26T10:40:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:53 smithi042 ceph-mon[32775]: pgmap v7713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:53 smithi164 ceph-mon[36970]: pgmap v7713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:53 smithi163 ceph-mon[36873]: pgmap v7713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:55 smithi042 ceph-mon[32775]: pgmap v7714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:40:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:55 smithi164 ceph-mon[36970]: pgmap v7714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:40:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:55 smithi163 ceph-mon[36873]: pgmap v7714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:40:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:57 smithi042 ceph-mon[32775]: pgmap v7715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13486.428359 secs 2024-07-26T10:40:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13504.166745 secs 2024-07-26T10:40:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:57 smithi164 ceph-mon[36970]: pgmap v7715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13486.428359 secs 2024-07-26T10:40:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13504.166745 secs 2024-07-26T10:40:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:57 smithi163 ceph-mon[36873]: pgmap v7715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:40:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13486.428359 secs 2024-07-26T10:40:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13504.166745 secs 2024-07-26T10:40:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:57.880+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T10:40:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:57.880+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T10:40:57.883 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:57.880+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff6780317b0 tx=0x7ff678061040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:40:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:57.880+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff6780317b0 tx=0x7ff678061040 comp rx=0 tx=0).stop 2024-07-26T10:40:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:57.880+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5210 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T10:40:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:57.880+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T10:40:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:57.880+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:40:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:57.880+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:40:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:57.880+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:40:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:57.881+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff6780094b0 tx=0x7ff678073000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:40:58.057 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:58.054+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:40:58.057 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:58.054+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:40:58.057 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:58.054+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:40:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:40:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:40:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:40:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:40:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:40:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:40:58.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:40:58.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:40:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:40:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:40:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:40:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:40:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:59 smithi042 ceph-mon[32775]: pgmap v7716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:59 smithi042 ceph-mon[32775]: pgmap v7717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:40:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:40:59 smithi042 ceph-mon[32775]: pgmap v7718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:40:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:59 smithi164 ceph-mon[36970]: pgmap v7716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:59 smithi164 ceph-mon[36970]: pgmap v7717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:40:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:40:59 smithi164 ceph-mon[36970]: pgmap v7718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:40:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:59 smithi163 ceph-mon[36873]: pgmap v7716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:40:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:59 smithi163 ceph-mon[36873]: pgmap v7717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:40:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:40:59 smithi163 ceph-mon[36873]: pgmap v7718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:40:59.982 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:59.980+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:40:59.983 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:59.980+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:40:59.983 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:59.980+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:40:59.983 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:40:59.980+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:41:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:01 smithi042 ceph-mon[32775]: pgmap v7719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:41:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:01 smithi164 ceph-mon[36970]: pgmap v7719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:41:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:01 smithi163 ceph-mon[36873]: pgmap v7719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:41:02.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13491.428490 secs 2024-07-26T10:41:02.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13509.166886 secs 2024-07-26T10:41:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13491.428490 secs 2024-07-26T10:41:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13509.166886 secs 2024-07-26T10:41:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13491.428490 secs 2024-07-26T10:41:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13509.166886 secs 2024-07-26T10:41:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:03 smithi042 ceph-mon[32775]: pgmap v7720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:41:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:03 smithi164 ceph-mon[36970]: pgmap v7720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:41:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:03 smithi163 ceph-mon[36873]: pgmap v7720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:41:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:05 smithi042 ceph-mon[32775]: pgmap v7721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:41:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:05 smithi164 ceph-mon[36970]: pgmap v7721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:41:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:05 smithi163 ceph-mon[36873]: pgmap v7721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:41:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13496.428621 secs 2024-07-26T10:41:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:07 smithi042 ceph-mon[32775]: pgmap v7722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:41:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13514.167027 secs 2024-07-26T10:41:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13496.428621 secs 2024-07-26T10:41:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:07 smithi164 ceph-mon[36970]: pgmap v7722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:41:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13514.167027 secs 2024-07-26T10:41:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13496.428621 secs 2024-07-26T10:41:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:07 smithi163 ceph-mon[36873]: pgmap v7722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:41:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13514.167027 secs 2024-07-26T10:41:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:09 smithi042 ceph-mon[32775]: pgmap v7723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:41:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:09 smithi164 ceph-mon[36970]: pgmap v7723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:41:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:09 smithi163 ceph-mon[36873]: pgmap v7723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:41:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:11 smithi042 ceph-mon[32775]: pgmap v7724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:11 smithi164 ceph-mon[36970]: pgmap v7724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:11 smithi163 ceph-mon[36873]: pgmap v7724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13501.428747 secs 2024-07-26T10:41:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13519.167092 secs 2024-07-26T10:41:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13501.428747 secs 2024-07-26T10:41:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13519.167092 secs 2024-07-26T10:41:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13501.428747 secs 2024-07-26T10:41:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13519.167092 secs 2024-07-26T10:41:13.058 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:41:13.056+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:41:13.058 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:41:13.056+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:41:13.058 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:41:13.056+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:41:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:13 smithi042 ceph-mon[32775]: pgmap v7725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:13 smithi164 ceph-mon[36970]: pgmap v7725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:13 smithi163 ceph-mon[36873]: pgmap v7725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:15 smithi042 ceph-mon[32775]: pgmap v7726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:15 smithi164 ceph-mon[36970]: pgmap v7726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:15 smithi163 ceph-mon[36873]: pgmap v7726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13506.428871 secs 2024-07-26T10:41:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:17 smithi042 ceph-mon[32775]: pgmap v7727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13524.167211 secs 2024-07-26T10:41:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13506.428871 secs 2024-07-26T10:41:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:17 smithi164 ceph-mon[36970]: pgmap v7727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13524.167211 secs 2024-07-26T10:41:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13506.428871 secs 2024-07-26T10:41:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:17 smithi163 ceph-mon[36873]: pgmap v7727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13524.167211 secs 2024-07-26T10:41:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:19 smithi042 ceph-mon[32775]: pgmap v7728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:19 smithi164 ceph-mon[36970]: pgmap v7728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:19 smithi163 ceph-mon[36873]: pgmap v7728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:21 smithi042 ceph-mon[32775]: pgmap v7729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:21 smithi164 ceph-mon[36970]: pgmap v7729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:21 smithi163 ceph-mon[36873]: pgmap v7729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13511.429015 secs 2024-07-26T10:41:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13529.167326 secs 2024-07-26T10:41:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13511.429015 secs 2024-07-26T10:41:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13529.167326 secs 2024-07-26T10:41:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13511.429015 secs 2024-07-26T10:41:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13529.167326 secs 2024-07-26T10:41:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:23 smithi042 ceph-mon[32775]: pgmap v7730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:23 smithi164 ceph-mon[36970]: pgmap v7730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:23 smithi163 ceph-mon[36873]: pgmap v7730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:25 smithi042 ceph-mon[32775]: pgmap v7731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:25 smithi164 ceph-mon[36970]: pgmap v7731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:25 smithi163 ceph-mon[36873]: pgmap v7731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13516.429129 secs 2024-07-26T10:41:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:27 smithi042 ceph-mon[32775]: pgmap v7732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13534.167466 secs 2024-07-26T10:41:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13516.429129 secs 2024-07-26T10:41:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:27 smithi164 ceph-mon[36970]: pgmap v7732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13534.167466 secs 2024-07-26T10:41:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13516.429129 secs 2024-07-26T10:41:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:27 smithi163 ceph-mon[36873]: pgmap v7732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13534.167466 secs 2024-07-26T10:41:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:41:28.061+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:41:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:41:28.061+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:41:28.063 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:41:28.061+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:41:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:29 smithi042 ceph-mon[32775]: pgmap v7733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:29 smithi164 ceph-mon[36970]: pgmap v7733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:29 smithi163 ceph-mon[36873]: pgmap v7733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:31 smithi042 ceph-mon[32775]: pgmap v7734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:31 smithi164 ceph-mon[36970]: pgmap v7734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:31 smithi163 ceph-mon[36873]: pgmap v7734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13521.429256 secs 2024-07-26T10:41:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13539.167583 secs 2024-07-26T10:41:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13521.429256 secs 2024-07-26T10:41:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13539.167583 secs 2024-07-26T10:41:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13521.429256 secs 2024-07-26T10:41:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13539.167583 secs 2024-07-26T10:41:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:33 smithi164 ceph-mon[36970]: pgmap v7735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:33 smithi163 ceph-mon[36873]: pgmap v7735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:33 smithi042 ceph-mon[32775]: pgmap v7735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:35 smithi164 ceph-mon[36970]: pgmap v7736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:35 smithi163 ceph-mon[36873]: pgmap v7736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:35 smithi042 ceph-mon[32775]: pgmap v7736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13526.429316 secs 2024-07-26T10:41:37.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:37 smithi164 ceph-mon[36970]: pgmap v7737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:37.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13544.167726 secs 2024-07-26T10:41:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13526.429316 secs 2024-07-26T10:41:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:37 smithi163 ceph-mon[36873]: pgmap v7737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13544.167726 secs 2024-07-26T10:41:37.776 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13526.429316 secs 2024-07-26T10:41:37.777 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:37 smithi042 ceph-mon[32775]: pgmap v7737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:37.777 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13544.167726 secs 2024-07-26T10:41:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:39 smithi164 ceph-mon[36970]: pgmap v7738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:39 smithi163 ceph-mon[36873]: pgmap v7738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:39 smithi042 ceph-mon[32775]: pgmap v7738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:41 smithi164 ceph-mon[36970]: pgmap v7739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:41 smithi163 ceph-mon[36873]: pgmap v7739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:41 smithi042 ceph-mon[32775]: pgmap v7739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13531.429443 secs 2024-07-26T10:41:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13549.167826 secs 2024-07-26T10:41:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13531.429443 secs 2024-07-26T10:41:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13549.167826 secs 2024-07-26T10:41:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13531.429443 secs 2024-07-26T10:41:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13549.167826 secs 2024-07-26T10:41:43.063 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:41:43.062+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:41:43.063 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:41:43.062+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:41:43.064 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:41:43.062+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:41:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:43 smithi164 ceph-mon[36970]: pgmap v7740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:43 smithi163 ceph-mon[36873]: pgmap v7740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:43 smithi042 ceph-mon[32775]: pgmap v7740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:45 smithi164 ceph-mon[36970]: pgmap v7741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:45 smithi163 ceph-mon[36873]: pgmap v7741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:45 smithi042 ceph-mon[32775]: pgmap v7741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13536.429567 secs 2024-07-26T10:41:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:47 smithi164 ceph-mon[36970]: pgmap v7742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13554.167978 secs 2024-07-26T10:41:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13536.429567 secs 2024-07-26T10:41:47.810 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:47 smithi163 ceph-mon[36873]: pgmap v7742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:47.810 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13554.167978 secs 2024-07-26T10:41:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13536.429567 secs 2024-07-26T10:41:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:47 smithi042 ceph-mon[32775]: pgmap v7742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13554.167978 secs 2024-07-26T10:41:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:49 smithi164 ceph-mon[36970]: pgmap v7743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:49 smithi163 ceph-mon[36873]: pgmap v7743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:49.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:49 smithi042 ceph-mon[32775]: pgmap v7743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:51 smithi164 ceph-mon[36970]: pgmap v7744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:51 smithi163 ceph-mon[36873]: pgmap v7744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:51 smithi042 ceph-mon[32775]: pgmap v7744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13541.429686 secs 2024-07-26T10:41:52.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13559.168094 secs 2024-07-26T10:41:52.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:52 smithi164 ceph-mon[36970]: pgmap v7745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13541.429686 secs 2024-07-26T10:41:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13559.168094 secs 2024-07-26T10:41:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:52 smithi163 ceph-mon[36873]: pgmap v7745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13541.429686 secs 2024-07-26T10:41:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13559.168094 secs 2024-07-26T10:41:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:52 smithi042 ceph-mon[32775]: pgmap v7745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:55 smithi042 ceph-mon[32775]: pgmap v7746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:55 smithi164 ceph-mon[36970]: pgmap v7746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:55 smithi163 ceph-mon[36873]: pgmap v7746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:41:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13546.429810 secs 2024-07-26T10:41:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:57 smithi042 ceph-mon[32775]: pgmap v7747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13564.168223 secs 2024-07-26T10:41:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13546.429810 secs 2024-07-26T10:41:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:57 smithi164 ceph-mon[36970]: pgmap v7747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13564.168223 secs 2024-07-26T10:41:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13546.429810 secs 2024-07-26T10:41:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:57 smithi163 ceph-mon[36873]: pgmap v7747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13564.168223 secs 2024-07-26T10:41:58.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:41:58.067+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:41:58.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:41:58.067+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:41:58.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:41:58.067+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:41:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:59 smithi042 ceph-mon[32775]: pgmap v7748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:41:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:41:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:41:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:41:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:41:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:59 smithi164 ceph-mon[36970]: pgmap v7748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:41:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:41:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:41:59.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:41:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:41:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:59 smithi163 ceph-mon[36873]: pgmap v7748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:41:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:41:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:41:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:41:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:41:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:42:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:00 smithi042 ceph-mon[32775]: pgmap v7749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:42:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:00 smithi042 ceph-mon[32775]: pgmap v7750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:42:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:00 smithi164 ceph-mon[36970]: pgmap v7749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:42:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:00 smithi164 ceph-mon[36970]: pgmap v7750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:42:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:00 smithi163 ceph-mon[36873]: pgmap v7749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:42:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:00 smithi163 ceph-mon[36873]: pgmap v7750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:42:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:02 smithi042 ceph-mon[32775]: pgmap v7751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:42:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13551.429925 secs 2024-07-26T10:42:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13569.168323 secs 2024-07-26T10:42:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:02 smithi164 ceph-mon[36970]: pgmap v7751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:42:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13551.429925 secs 2024-07-26T10:42:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13569.168323 secs 2024-07-26T10:42:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:02 smithi163 ceph-mon[36873]: pgmap v7751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:42:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13551.429925 secs 2024-07-26T10:42:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13569.168323 secs 2024-07-26T10:42:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:04 smithi042 ceph-mon[32775]: pgmap v7752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:42:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:04 smithi164 ceph-mon[36970]: pgmap v7752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:42:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:04 smithi163 ceph-mon[36873]: pgmap v7752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:42:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:06 smithi042 ceph-mon[32775]: pgmap v7753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:42:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:06 smithi164 ceph-mon[36970]: pgmap v7753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:42:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:06 smithi163 ceph-mon[36873]: pgmap v7753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:42:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13556.430074 secs 2024-07-26T10:42:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13574.168466 secs 2024-07-26T10:42:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13556.430074 secs 2024-07-26T10:42:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13574.168466 secs 2024-07-26T10:42:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13556.430074 secs 2024-07-26T10:42:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13574.168466 secs 2024-07-26T10:42:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:08 smithi042 ceph-mon[32775]: pgmap v7754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:42:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:08 smithi164 ceph-mon[36970]: pgmap v7754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:42:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:08 smithi163 ceph-mon[36873]: pgmap v7754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:42:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:10 smithi042 ceph-mon[32775]: pgmap v7755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:42:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:10 smithi164 ceph-mon[36970]: pgmap v7755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:42:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:10 smithi163 ceph-mon[36873]: pgmap v7755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:42:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:12 smithi042 ceph-mon[32775]: pgmap v7756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13561.430146 secs 2024-07-26T10:42:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13579.168549 secs 2024-07-26T10:42:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:12 smithi164 ceph-mon[36970]: pgmap v7756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13561.430146 secs 2024-07-26T10:42:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13579.168549 secs 2024-07-26T10:42:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:12 smithi163 ceph-mon[36873]: pgmap v7756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13561.430146 secs 2024-07-26T10:42:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13579.168549 secs 2024-07-26T10:42:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:42:13.068+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:42:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:42:13.068+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:42:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:42:13.068+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:42:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:14 smithi042 ceph-mon[32775]: pgmap v7757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:14 smithi164 ceph-mon[36970]: pgmap v7757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:14 smithi163 ceph-mon[36873]: pgmap v7757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:16 smithi042 ceph-mon[32775]: pgmap v7758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:16 smithi164 ceph-mon[36970]: pgmap v7758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:16 smithi163 ceph-mon[36873]: pgmap v7758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13566.430297 secs 2024-07-26T10:42:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13584.168641 secs 2024-07-26T10:42:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13566.430297 secs 2024-07-26T10:42:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13584.168641 secs 2024-07-26T10:42:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13566.430297 secs 2024-07-26T10:42:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13584.168641 secs 2024-07-26T10:42:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:18 smithi042 ceph-mon[32775]: pgmap v7759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:18 smithi164 ceph-mon[36970]: pgmap v7759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:18 smithi163 ceph-mon[36873]: pgmap v7759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:20 smithi042 ceph-mon[32775]: pgmap v7760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:42:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:20 smithi164 ceph-mon[36970]: pgmap v7760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:42:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:20 smithi163 ceph-mon[36873]: pgmap v7760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:42:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:22 smithi042 ceph-mon[32775]: pgmap v7761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13571.430425 secs 2024-07-26T10:42:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13589.168711 secs 2024-07-26T10:42:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:22 smithi164 ceph-mon[36970]: pgmap v7761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13571.430425 secs 2024-07-26T10:42:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13589.168711 secs 2024-07-26T10:42:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:22 smithi163 ceph-mon[36873]: pgmap v7761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13571.430425 secs 2024-07-26T10:42:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13589.168711 secs 2024-07-26T10:42:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:24 smithi042 ceph-mon[32775]: pgmap v7762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:24 smithi164 ceph-mon[36970]: pgmap v7762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:24 smithi163 ceph-mon[36873]: pgmap v7762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:26 smithi042 ceph-mon[32775]: pgmap v7763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:26 smithi164 ceph-mon[36970]: pgmap v7763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:26 smithi163 ceph-mon[36873]: pgmap v7763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13576.430557 secs 2024-07-26T10:42:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13594.168787 secs 2024-07-26T10:42:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13576.430557 secs 2024-07-26T10:42:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13594.168787 secs 2024-07-26T10:42:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13576.430557 secs 2024-07-26T10:42:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13594.168787 secs 2024-07-26T10:42:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:42:28.074+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:42:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:42:28.074+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:42:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:42:28.074+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:42:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:28 smithi042 ceph-mon[32775]: pgmap v7764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:28 smithi164 ceph-mon[36970]: pgmap v7764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:28 smithi163 ceph-mon[36873]: pgmap v7764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:30 smithi042 ceph-mon[32775]: pgmap v7765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:42:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:30 smithi164 ceph-mon[36970]: pgmap v7765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:42:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:30 smithi163 ceph-mon[36873]: pgmap v7765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:42:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:32 smithi042 ceph-mon[32775]: pgmap v7766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13581.430634 secs 2024-07-26T10:42:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13599.168965 secs 2024-07-26T10:42:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:32 smithi164 ceph-mon[36970]: pgmap v7766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13581.430634 secs 2024-07-26T10:42:32.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13599.168965 secs 2024-07-26T10:42:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:32 smithi163 ceph-mon[36873]: pgmap v7766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13581.430634 secs 2024-07-26T10:42:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13599.168965 secs 2024-07-26T10:42:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:34 smithi042 ceph-mon[32775]: pgmap v7767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:34 smithi164 ceph-mon[36970]: pgmap v7767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:34 smithi163 ceph-mon[36873]: pgmap v7767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:36 smithi042 ceph-mon[32775]: pgmap v7768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:36 smithi164 ceph-mon[36970]: pgmap v7768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:36 smithi163 ceph-mon[36873]: pgmap v7768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13586.430717 secs 2024-07-26T10:42:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13604.169082 secs 2024-07-26T10:42:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13586.430717 secs 2024-07-26T10:42:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13604.169082 secs 2024-07-26T10:42:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13586.430717 secs 2024-07-26T10:42:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13604.169082 secs 2024-07-26T10:42:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:38 smithi042 ceph-mon[32775]: pgmap v7769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:38 smithi164 ceph-mon[36970]: pgmap v7769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:38 smithi163 ceph-mon[36873]: pgmap v7769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:40 smithi042 ceph-mon[32775]: pgmap v7770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:42:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:40 smithi164 ceph-mon[36970]: pgmap v7770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:42:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:40 smithi163 ceph-mon[36873]: pgmap v7770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:42:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:42 smithi042 ceph-mon[32775]: pgmap v7771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13591.430843 secs 2024-07-26T10:42:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13609.169181 secs 2024-07-26T10:42:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:42 smithi164 ceph-mon[36970]: pgmap v7771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13591.430843 secs 2024-07-26T10:42:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13609.169181 secs 2024-07-26T10:42:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:42 smithi163 ceph-mon[36873]: pgmap v7771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13591.430843 secs 2024-07-26T10:42:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13609.169181 secs 2024-07-26T10:42:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:42:43.075+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:42:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:42:43.075+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:42:43.078 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:42:43.075+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:42:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:44 smithi042 ceph-mon[32775]: pgmap v7772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:44 smithi164 ceph-mon[36970]: pgmap v7772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:44 smithi163 ceph-mon[36873]: pgmap v7772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:46 smithi042 ceph-mon[32775]: pgmap v7773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:46 smithi164 ceph-mon[36970]: pgmap v7773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:46 smithi163 ceph-mon[36873]: pgmap v7773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13596.430931 secs 2024-07-26T10:42:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13614.169334 secs 2024-07-26T10:42:47.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13596.430931 secs 2024-07-26T10:42:47.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13614.169334 secs 2024-07-26T10:42:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13596.430931 secs 2024-07-26T10:42:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13614.169334 secs 2024-07-26T10:42:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:48 smithi042 ceph-mon[32775]: pgmap v7774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:48 smithi164 ceph-mon[36970]: pgmap v7774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:48 smithi163 ceph-mon[36873]: pgmap v7774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:50 smithi042 ceph-mon[32775]: pgmap v7775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:42:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:50 smithi164 ceph-mon[36970]: pgmap v7775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:42:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:50 smithi163 ceph-mon[36873]: pgmap v7775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:42:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:52 smithi042 ceph-mon[32775]: pgmap v7776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13601.431055 secs 2024-07-26T10:42:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13619.169454 secs 2024-07-26T10:42:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:52 smithi164 ceph-mon[36970]: pgmap v7776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:52.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13601.431055 secs 2024-07-26T10:42:52.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13619.169454 secs 2024-07-26T10:42:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:52 smithi163 ceph-mon[36873]: pgmap v7776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13601.431055 secs 2024-07-26T10:42:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13619.169454 secs 2024-07-26T10:42:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:54 smithi042 ceph-mon[32775]: pgmap v7777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:54 smithi164 ceph-mon[36970]: pgmap v7777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:54 smithi163 ceph-mon[36873]: pgmap v7777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:56 smithi042 ceph-mon[32775]: pgmap v7778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:56 smithi164 ceph-mon[36970]: pgmap v7778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:56 smithi163 ceph-mon[36873]: pgmap v7778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:42:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13606.431244 secs 2024-07-26T10:42:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13624.169592 secs 2024-07-26T10:42:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13606.431244 secs 2024-07-26T10:42:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13624.169592 secs 2024-07-26T10:42:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13606.431244 secs 2024-07-26T10:42:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13624.169592 secs 2024-07-26T10:42:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:42:58.080+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:42:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:42:58.080+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:42:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:42:58.080+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:42:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:58 smithi164 ceph-mon[36970]: pgmap v7779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:58 smithi163 ceph-mon[36873]: pgmap v7779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:58 smithi042 ceph-mon[32775]: pgmap v7779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:42:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:42:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:42:59.730 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:42:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:42:59.748 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:42:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:43:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:00 smithi164 ceph-mon[36970]: pgmap v7780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:00 smithi163 ceph-mon[36873]: pgmap v7780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:00 smithi042 ceph-mon[32775]: pgmap v7780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:01 smithi163 ceph-mon[36873]: pgmap v7781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:01 smithi163 ceph-mon[36873]: pgmap v7782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:43:01.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:01 smithi042 ceph-mon[32775]: pgmap v7781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:43:01.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:43:01.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:01 smithi042 ceph-mon[32775]: pgmap v7782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:43:01.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:01 smithi164 ceph-mon[36970]: pgmap v7781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:43:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:43:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:01 smithi164 ceph-mon[36970]: pgmap v7782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:43:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:43:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13611.431316 secs 2024-07-26T10:43:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13629.169703 secs 2024-07-26T10:43:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:02 smithi163 ceph-mon[36873]: pgmap v7783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:43:02.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13611.431316 secs 2024-07-26T10:43:02.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13629.169703 secs 2024-07-26T10:43:02.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:02 smithi042 ceph-mon[32775]: pgmap v7783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:43:02.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13611.431316 secs 2024-07-26T10:43:02.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13629.169703 secs 2024-07-26T10:43:02.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:02 smithi164 ceph-mon[36970]: pgmap v7783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:43:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:04 smithi163 ceph-mon[36873]: pgmap v7784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:43:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:04 smithi042 ceph-mon[32775]: pgmap v7784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:43:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:04 smithi164 ceph-mon[36970]: pgmap v7784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:43:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:06 smithi042 ceph-mon[32775]: pgmap v7785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:43:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:06 smithi164 ceph-mon[36970]: pgmap v7785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:43:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:06 smithi163 ceph-mon[36873]: pgmap v7785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:43:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13616.431445 secs 2024-07-26T10:43:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13634.169824 secs 2024-07-26T10:43:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13616.431445 secs 2024-07-26T10:43:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13634.169824 secs 2024-07-26T10:43:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13616.431445 secs 2024-07-26T10:43:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13634.169824 secs 2024-07-26T10:43:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:08 smithi042 ceph-mon[32775]: pgmap v7786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:43:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:08 smithi164 ceph-mon[36970]: pgmap v7786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:43:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:08 smithi163 ceph-mon[36873]: pgmap v7786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:43:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:10 smithi042 ceph-mon[32775]: pgmap v7787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:43:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:10 smithi164 ceph-mon[36970]: pgmap v7787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:43:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:10 smithi163 ceph-mon[36873]: pgmap v7787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:43:12.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13621.431507 secs 2024-07-26T10:43:12.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13639.169947 secs 2024-07-26T10:43:12.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:12 smithi042 ceph-mon[32775]: pgmap v7788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:43:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13621.431507 secs 2024-07-26T10:43:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13639.169947 secs 2024-07-26T10:43:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:12 smithi164 ceph-mon[36970]: pgmap v7788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:43:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13621.431507 secs 2024-07-26T10:43:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13639.169947 secs 2024-07-26T10:43:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:12 smithi163 ceph-mon[36873]: pgmap v7788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:43:13.083 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:43:13.081+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:43:13.083 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:43:13.081+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:43:13.083 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:43:13.081+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:43:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:14 smithi042 ceph-mon[32775]: pgmap v7789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:14 smithi164 ceph-mon[36970]: pgmap v7789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:14 smithi163 ceph-mon[36873]: pgmap v7789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:16 smithi042 ceph-mon[32775]: pgmap v7790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:16 smithi164 ceph-mon[36970]: pgmap v7790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:16 smithi163 ceph-mon[36873]: pgmap v7790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13626.431592 secs 2024-07-26T10:43:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13644.170065 secs 2024-07-26T10:43:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13626.431592 secs 2024-07-26T10:43:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13644.170065 secs 2024-07-26T10:43:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13626.431592 secs 2024-07-26T10:43:17.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13644.170065 secs 2024-07-26T10:43:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:18 smithi042 ceph-mon[32775]: pgmap v7791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:18 smithi164 ceph-mon[36970]: pgmap v7791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:18 smithi163 ceph-mon[36873]: pgmap v7791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:20 smithi042 ceph-mon[32775]: pgmap v7792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:20 smithi164 ceph-mon[36970]: pgmap v7792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:20 smithi163 ceph-mon[36873]: pgmap v7792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13631.431656 secs 2024-07-26T10:43:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13649.170172 secs 2024-07-26T10:43:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:22 smithi042 ceph-mon[32775]: pgmap v7793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13631.431656 secs 2024-07-26T10:43:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13649.170172 secs 2024-07-26T10:43:22.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:22 smithi164 ceph-mon[36970]: pgmap v7793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13631.431656 secs 2024-07-26T10:43:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13649.170172 secs 2024-07-26T10:43:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:22 smithi163 ceph-mon[36873]: pgmap v7793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:24 smithi042 ceph-mon[32775]: pgmap v7794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:24 smithi164 ceph-mon[36970]: pgmap v7794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:24 smithi163 ceph-mon[36873]: pgmap v7794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:26 smithi042 ceph-mon[32775]: pgmap v7795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:26 smithi164 ceph-mon[36970]: pgmap v7795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:26 smithi163 ceph-mon[36873]: pgmap v7795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13636.431783 secs 2024-07-26T10:43:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13654.170263 secs 2024-07-26T10:43:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13636.431783 secs 2024-07-26T10:43:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13654.170263 secs 2024-07-26T10:43:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13636.431783 secs 2024-07-26T10:43:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13654.170263 secs 2024-07-26T10:43:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:43:28.086+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:43:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:43:28.086+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:43:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:43:28.086+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:43:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:28 smithi042 ceph-mon[32775]: pgmap v7796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:28 smithi164 ceph-mon[36970]: pgmap v7796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:28 smithi163 ceph-mon[36873]: pgmap v7796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:30 smithi042 ceph-mon[32775]: pgmap v7797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:30 smithi164 ceph-mon[36970]: pgmap v7797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:30 smithi163 ceph-mon[36873]: pgmap v7797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13641.431908 secs 2024-07-26T10:43:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13659.170339 secs 2024-07-26T10:43:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:32 smithi042 ceph-mon[32775]: pgmap v7798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13641.431908 secs 2024-07-26T10:43:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13659.170339 secs 2024-07-26T10:43:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:32 smithi164 ceph-mon[36970]: pgmap v7798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13641.431908 secs 2024-07-26T10:43:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13659.170339 secs 2024-07-26T10:43:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:32 smithi163 ceph-mon[36873]: pgmap v7798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:34 smithi042 ceph-mon[32775]: pgmap v7799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:34 smithi164 ceph-mon[36970]: pgmap v7799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:34 smithi163 ceph-mon[36873]: pgmap v7799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:36 smithi042 ceph-mon[32775]: pgmap v7800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:36 smithi164 ceph-mon[36970]: pgmap v7800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:36 smithi163 ceph-mon[36873]: pgmap v7800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13646.432051 secs 2024-07-26T10:43:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13664.170418 secs 2024-07-26T10:43:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13646.432051 secs 2024-07-26T10:43:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13664.170418 secs 2024-07-26T10:43:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13646.432051 secs 2024-07-26T10:43:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13664.170418 secs 2024-07-26T10:43:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:38 smithi042 ceph-mon[32775]: pgmap v7801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:38 smithi164 ceph-mon[36970]: pgmap v7801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:38 smithi163 ceph-mon[36873]: pgmap v7801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:40 smithi042 ceph-mon[32775]: pgmap v7802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:40 smithi164 ceph-mon[36970]: pgmap v7802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:40 smithi163 ceph-mon[36873]: pgmap v7802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13651.432168 secs 2024-07-26T10:43:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13669.170546 secs 2024-07-26T10:43:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:42 smithi042 ceph-mon[32775]: pgmap v7803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13651.432168 secs 2024-07-26T10:43:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13669.170546 secs 2024-07-26T10:43:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:42 smithi164 ceph-mon[36970]: pgmap v7803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13651.432168 secs 2024-07-26T10:43:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13669.170546 secs 2024-07-26T10:43:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:42 smithi163 ceph-mon[36873]: pgmap v7803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:43.088 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:43:43.087+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:43:43.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:43:43.087+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:43:43.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:43:43.087+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:43:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:44 smithi042 ceph-mon[32775]: pgmap v7804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:44 smithi164 ceph-mon[36970]: pgmap v7804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:44 smithi163 ceph-mon[36873]: pgmap v7804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:46 smithi042 ceph-mon[32775]: pgmap v7805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:46 smithi164 ceph-mon[36970]: pgmap v7805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:46 smithi163 ceph-mon[36873]: pgmap v7805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13656.432294 secs 2024-07-26T10:43:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13674.170697 secs 2024-07-26T10:43:47.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13656.432294 secs 2024-07-26T10:43:47.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13674.170697 secs 2024-07-26T10:43:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13656.432294 secs 2024-07-26T10:43:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13674.170697 secs 2024-07-26T10:43:48.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:48 smithi042 ceph-mon[32775]: pgmap v7806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:48 smithi164 ceph-mon[36970]: pgmap v7806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:48 smithi163 ceph-mon[36873]: pgmap v7806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:50 smithi042 ceph-mon[32775]: pgmap v7807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:50 smithi164 ceph-mon[36970]: pgmap v7807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:50 smithi163 ceph-mon[36873]: pgmap v7807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13661.432359 secs 2024-07-26T10:43:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13679.170838 secs 2024-07-26T10:43:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:52 smithi042 ceph-mon[32775]: pgmap v7808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13661.432359 secs 2024-07-26T10:43:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13679.170838 secs 2024-07-26T10:43:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:52 smithi164 ceph-mon[36970]: pgmap v7808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13661.432359 secs 2024-07-26T10:43:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13679.170838 secs 2024-07-26T10:43:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:52 smithi163 ceph-mon[36873]: pgmap v7808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:54 smithi042 ceph-mon[32775]: pgmap v7809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:54 smithi164 ceph-mon[36970]: pgmap v7809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:54 smithi163 ceph-mon[36873]: pgmap v7809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:43:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:56 smithi042 ceph-mon[32775]: pgmap v7810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:56 smithi164 ceph-mon[36970]: pgmap v7810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:56 smithi163 ceph-mon[36873]: pgmap v7810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:43:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13666.432428 secs 2024-07-26T10:43:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13684.170963 secs 2024-07-26T10:43:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13666.432428 secs 2024-07-26T10:43:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13684.170963 secs 2024-07-26T10:43:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13666.432428 secs 2024-07-26T10:43:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13684.170963 secs 2024-07-26T10:43:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:43:58.092+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:43:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:43:58.092+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:43:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:43:58.092+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:43:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:43:58 smithi042 ceph-mon[32775]: pgmap v7811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:43:58 smithi164 ceph-mon[36970]: pgmap v7811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:43:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:43:58 smithi163 ceph-mon[36873]: pgmap v7811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:00.011 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:44:00.009+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:44:00.011 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:44:00.009+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:44:00.011 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:44:00.009+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:44:00.011 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:44:00.009+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:44:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:00 smithi042 ceph-mon[32775]: pgmap v7812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:00 smithi164 ceph-mon[36970]: pgmap v7812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:00 smithi163 ceph-mon[36873]: pgmap v7812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:01 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:44:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:01 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:44:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:01 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:44:02.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13671.432515 secs 2024-07-26T10:44:02.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13689.171084 secs 2024-07-26T10:44:02.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:02 smithi164 ceph-mon[36970]: pgmap v7813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:02.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:44:02.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:44:02.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13671.432515 secs 2024-07-26T10:44:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13689.171084 secs 2024-07-26T10:44:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:02 smithi042 ceph-mon[32775]: pgmap v7813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:02.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:44:02.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:44:02.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13671.432515 secs 2024-07-26T10:44:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13689.171084 secs 2024-07-26T10:44:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:02 smithi163 ceph-mon[36873]: pgmap v7813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:03.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:03.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:03.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:03.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:03.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:03.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:03.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:03.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:44:03.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:44:03.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:44:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:03 smithi042 ceph-mon[32775]: pgmap v7814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T10:44:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:03 smithi042 ceph-mon[32775]: pgmap v7815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T10:44:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:03 smithi164 ceph-mon[36970]: pgmap v7814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T10:44:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:03 smithi164 ceph-mon[36970]: pgmap v7815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T10:44:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:03 smithi163 ceph-mon[36873]: pgmap v7814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T10:44:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:03 smithi163 ceph-mon[36873]: pgmap v7815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T10:44:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:05 smithi042 ceph-mon[32775]: pgmap v7816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T10:44:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:05 smithi164 ceph-mon[36970]: pgmap v7816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T10:44:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:05 smithi163 ceph-mon[36873]: pgmap v7816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T10:44:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13676.432597 secs 2024-07-26T10:44:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13694.171229 secs 2024-07-26T10:44:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:07 smithi164 ceph-mon[36970]: pgmap v7817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T10:44:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13676.432597 secs 2024-07-26T10:44:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13694.171229 secs 2024-07-26T10:44:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:07 smithi163 ceph-mon[36873]: pgmap v7817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T10:44:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13676.432597 secs 2024-07-26T10:44:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13694.171229 secs 2024-07-26T10:44:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:07 smithi042 ceph-mon[32775]: pgmap v7817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T10:44:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:09 smithi164 ceph-mon[36970]: pgmap v7818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T10:44:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:09 smithi163 ceph-mon[36873]: pgmap v7818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T10:44:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:09 smithi042 ceph-mon[32775]: pgmap v7818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T10:44:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:11 smithi164 ceph-mon[36970]: pgmap v7819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T10:44:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:11 smithi163 ceph-mon[36873]: pgmap v7819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T10:44:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:11 smithi042 ceph-mon[32775]: pgmap v7819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T10:44:12.913 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13681.432717 secs 2024-07-26T10:44:12.913 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13699.171350 secs 2024-07-26T10:44:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13681.432717 secs 2024-07-26T10:44:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13699.171350 secs 2024-07-26T10:44:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13681.432717 secs 2024-07-26T10:44:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13699.171350 secs 2024-07-26T10:44:13.094 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:44:13.093+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:44:13.094 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:44:13.093+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:44:13.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:44:13.093+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:44:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:13 smithi164 ceph-mon[36970]: pgmap v7820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:44:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:13 smithi163 ceph-mon[36873]: pgmap v7820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:44:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:13 smithi042 ceph-mon[32775]: pgmap v7820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:44:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:15 smithi164 ceph-mon[36970]: pgmap v7821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:15 smithi163 ceph-mon[36873]: pgmap v7821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:15 smithi042 ceph-mon[32775]: pgmap v7821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13686.432799 secs 2024-07-26T10:44:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13704.171469 secs 2024-07-26T10:44:17.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:17 smithi164 ceph-mon[36970]: pgmap v7822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13686.432799 secs 2024-07-26T10:44:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13704.171469 secs 2024-07-26T10:44:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:17 smithi163 ceph-mon[36873]: pgmap v7822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13686.432799 secs 2024-07-26T10:44:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13704.171469 secs 2024-07-26T10:44:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:17 smithi042 ceph-mon[32775]: pgmap v7822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:19 smithi164 ceph-mon[36970]: pgmap v7823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:44:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:19 smithi163 ceph-mon[36873]: pgmap v7823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:44:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:19 smithi042 ceph-mon[32775]: pgmap v7823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:44:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:21 smithi164 ceph-mon[36970]: pgmap v7824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:21 smithi163 ceph-mon[36873]: pgmap v7824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:21 smithi042 ceph-mon[32775]: pgmap v7824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13691.432910 secs 2024-07-26T10:44:22.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13709.171544 secs 2024-07-26T10:44:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13691.432910 secs 2024-07-26T10:44:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13709.171544 secs 2024-07-26T10:44:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13691.432910 secs 2024-07-26T10:44:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13709.171544 secs 2024-07-26T10:44:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:23 smithi164 ceph-mon[36970]: pgmap v7825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:23 smithi163 ceph-mon[36873]: pgmap v7825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:23 smithi042 ceph-mon[32775]: pgmap v7825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:25 smithi164 ceph-mon[36970]: pgmap v7826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:25 smithi163 ceph-mon[36873]: pgmap v7826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:25 smithi042 ceph-mon[32775]: pgmap v7826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13696.433041 secs 2024-07-26T10:44:27.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13714.171684 secs 2024-07-26T10:44:27.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:27 smithi164 ceph-mon[36970]: pgmap v7827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13696.433041 secs 2024-07-26T10:44:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13714.171684 secs 2024-07-26T10:44:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:27 smithi163 ceph-mon[36873]: pgmap v7827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:28.100 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:44:28.099+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:44:28.100 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:44:28.099+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:44:28.100 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:44:28.099+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:44:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13696.433041 secs 2024-07-26T10:44:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13714.171684 secs 2024-07-26T10:44:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:27 smithi042 ceph-mon[32775]: pgmap v7827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:29 smithi164 ceph-mon[36970]: pgmap v7828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:44:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:29 smithi163 ceph-mon[36873]: pgmap v7828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:44:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:29 smithi042 ceph-mon[32775]: pgmap v7828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:44:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:31 smithi164 ceph-mon[36970]: pgmap v7829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:31 smithi163 ceph-mon[36873]: pgmap v7829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:31 smithi042 ceph-mon[32775]: pgmap v7829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13701.433166 secs 2024-07-26T10:44:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13719.171803 secs 2024-07-26T10:44:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13701.433166 secs 2024-07-26T10:44:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13719.171803 secs 2024-07-26T10:44:33.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13701.433166 secs 2024-07-26T10:44:33.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13719.171803 secs 2024-07-26T10:44:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:33 smithi164 ceph-mon[36970]: pgmap v7830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:33 smithi163 ceph-mon[36873]: pgmap v7830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:33 smithi042 ceph-mon[32775]: pgmap v7830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:35 smithi164 ceph-mon[36970]: pgmap v7831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:35 smithi163 ceph-mon[36873]: pgmap v7831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:35 smithi042 ceph-mon[32775]: pgmap v7831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13706.433275 secs 2024-07-26T10:44:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13724.171923 secs 2024-07-26T10:44:37.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:37 smithi164 ceph-mon[36970]: pgmap v7832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13706.433275 secs 2024-07-26T10:44:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13724.171923 secs 2024-07-26T10:44:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:37 smithi163 ceph-mon[36873]: pgmap v7832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13706.433275 secs 2024-07-26T10:44:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13724.171923 secs 2024-07-26T10:44:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:37 smithi042 ceph-mon[32775]: pgmap v7832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:40 smithi042 ceph-mon[32775]: pgmap v7833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:44:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:40 smithi164 ceph-mon[36970]: pgmap v7833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:44:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:40 smithi163 ceph-mon[36873]: pgmap v7833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:44:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:42 smithi042 ceph-mon[32775]: pgmap v7834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13711.433346 secs 2024-07-26T10:44:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:42 smithi164 ceph-mon[36970]: pgmap v7834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13711.433346 secs 2024-07-26T10:44:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:42 smithi163 ceph-mon[36873]: pgmap v7834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13711.433346 secs 2024-07-26T10:44:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:44:43.099+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:44:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:44:43.099+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:44:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:44:43.099+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:44:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:43 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13729.172078 secs 2024-07-26T10:44:43.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:43 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13729.172078 secs 2024-07-26T10:44:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:43 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13729.172078 secs 2024-07-26T10:44:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:44 smithi042 ceph-mon[32775]: pgmap v7835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:44 smithi164 ceph-mon[36970]: pgmap v7835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:44 smithi163 ceph-mon[36873]: pgmap v7835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:46 smithi042 ceph-mon[32775]: pgmap v7836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:46 smithi164 ceph-mon[36970]: pgmap v7836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:46 smithi163 ceph-mon[36873]: pgmap v7836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13716.433471 secs 2024-07-26T10:44:47.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13716.433471 secs 2024-07-26T10:44:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13716.433471 secs 2024-07-26T10:44:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:48 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13734.172236 secs 2024-07-26T10:44:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:48 smithi042 ceph-mon[32775]: pgmap v7837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:48 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13734.172236 secs 2024-07-26T10:44:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:48 smithi164 ceph-mon[36970]: pgmap v7837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:48 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13734.172236 secs 2024-07-26T10:44:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:48 smithi163 ceph-mon[36873]: pgmap v7837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:50 smithi042 ceph-mon[32775]: pgmap v7838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:44:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:50 smithi164 ceph-mon[36970]: pgmap v7838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:44:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:50 smithi163 ceph-mon[36873]: pgmap v7838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:44:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:52 smithi042 ceph-mon[32775]: pgmap v7839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13721.433562 secs 2024-07-26T10:44:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:52 smithi164 ceph-mon[36970]: pgmap v7839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13721.433562 secs 2024-07-26T10:44:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:52 smithi163 ceph-mon[36873]: pgmap v7839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13721.433562 secs 2024-07-26T10:44:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:53 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13739.172357 secs 2024-07-26T10:44:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:53 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13739.172357 secs 2024-07-26T10:44:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:53 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13739.172357 secs 2024-07-26T10:44:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:54 smithi042 ceph-mon[32775]: pgmap v7840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:54 smithi164 ceph-mon[36970]: pgmap v7840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:54 smithi163 ceph-mon[36873]: pgmap v7840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:56 smithi042 ceph-mon[32775]: pgmap v7841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:56 smithi164 ceph-mon[36970]: pgmap v7841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:56 smithi163 ceph-mon[36873]: pgmap v7841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:44:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13726.433660 secs 2024-07-26T10:44:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13726.433660 secs 2024-07-26T10:44:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13726.433660 secs 2024-07-26T10:44:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:44:58.104+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:44:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:44:58.104+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:44:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:44:58.104+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:44:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:58 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13744.172479 secs 2024-07-26T10:44:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:44:58 smithi042 ceph-mon[32775]: pgmap v7842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:58 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13744.172479 secs 2024-07-26T10:44:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:44:58 smithi164 ceph-mon[36970]: pgmap v7842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:44:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:58 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13744.172479 secs 2024-07-26T10:44:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:44:58 smithi163 ceph-mon[36873]: pgmap v7842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:00 smithi042 ceph-mon[32775]: pgmap v7843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:45:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:00 smithi164 ceph-mon[36970]: pgmap v7843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:45:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:00 smithi163 ceph-mon[36873]: pgmap v7843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:45:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:02 smithi042 ceph-mon[32775]: pgmap v7844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13731.433785 secs 2024-07-26T10:45:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:02 smithi164 ceph-mon[36970]: pgmap v7844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13731.433785 secs 2024-07-26T10:45:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:02 smithi163 ceph-mon[36873]: pgmap v7844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13731.433785 secs 2024-07-26T10:45:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:03 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13749.172604 secs 2024-07-26T10:45:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:45:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:03 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13749.172604 secs 2024-07-26T10:45:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:45:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:03 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13749.172604 secs 2024-07-26T10:45:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:45:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:04 smithi164 ceph-mon[36970]: pgmap v7845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:45:04.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:45:04.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:04 smithi164 ceph-mon[36970]: pgmap v7846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:45:04.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:04 smithi164 ceph-mon[36970]: pgmap v7847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:45:04.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:45:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:04 smithi163 ceph-mon[36873]: pgmap v7845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:45:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:45:04.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:04 smithi163 ceph-mon[36873]: pgmap v7846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:45:04.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:04 smithi163 ceph-mon[36873]: pgmap v7847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:45:04.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:45:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:04 smithi042 ceph-mon[32775]: pgmap v7845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:45:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:45:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:04 smithi042 ceph-mon[32775]: pgmap v7846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:45:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:04 smithi042 ceph-mon[32775]: pgmap v7847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:45:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:45:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:06 smithi164 ceph-mon[36970]: pgmap v7848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:45:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:06 smithi163 ceph-mon[36873]: pgmap v7848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:45:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:06 smithi042 ceph-mon[32775]: pgmap v7848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:45:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13736.433907 secs 2024-07-26T10:45:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13754.172749 secs 2024-07-26T10:45:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13736.433907 secs 2024-07-26T10:45:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13754.172749 secs 2024-07-26T10:45:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13736.433907 secs 2024-07-26T10:45:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13754.172749 secs 2024-07-26T10:45:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:08 smithi164 ceph-mon[36970]: pgmap v7849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:45:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:08 smithi163 ceph-mon[36873]: pgmap v7849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:45:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:08 smithi042 ceph-mon[32775]: pgmap v7849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:45:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:10 smithi164 ceph-mon[36970]: pgmap v7850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:45:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:10 smithi163 ceph-mon[36873]: pgmap v7850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:45:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:10 smithi042 ceph-mon[32775]: pgmap v7850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:45:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13741.434038 secs 2024-07-26T10:45:12.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:12 smithi164 ceph-mon[36970]: pgmap v7851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:45:12.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13759.172870 secs 2024-07-26T10:45:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13741.434038 secs 2024-07-26T10:45:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:12 smithi163 ceph-mon[36873]: pgmap v7851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:45:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13759.172870 secs 2024-07-26T10:45:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13741.434038 secs 2024-07-26T10:45:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:12 smithi042 ceph-mon[32775]: pgmap v7851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:45:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13759.172870 secs 2024-07-26T10:45:13.107 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:45:13.105+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:45:13.107 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:45:13.105+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:45:13.107 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:45:13.105+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:45:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:14 smithi164 ceph-mon[36970]: pgmap v7852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:45:14.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:14 smithi163 ceph-mon[36873]: pgmap v7852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:45:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:14 smithi042 ceph-mon[32775]: pgmap v7852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:45:16.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:16 smithi164 ceph-mon[36970]: pgmap v7853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:16 smithi163 ceph-mon[36873]: pgmap v7853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:16 smithi042 ceph-mon[32775]: pgmap v7853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13746.434170 secs 2024-07-26T10:45:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13764.172992 secs 2024-07-26T10:45:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13746.434170 secs 2024-07-26T10:45:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13764.172992 secs 2024-07-26T10:45:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13746.434170 secs 2024-07-26T10:45:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13764.172992 secs 2024-07-26T10:45:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:18 smithi164 ceph-mon[36970]: pgmap v7854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:18 smithi163 ceph-mon[36873]: pgmap v7854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:18 smithi042 ceph-mon[32775]: pgmap v7854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:20 smithi164 ceph-mon[36970]: pgmap v7855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:20 smithi163 ceph-mon[36873]: pgmap v7855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:20 smithi042 ceph-mon[32775]: pgmap v7855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13751.434317 secs 2024-07-26T10:45:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:22 smithi164 ceph-mon[36970]: pgmap v7856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13769.173115 secs 2024-07-26T10:45:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13751.434317 secs 2024-07-26T10:45:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:22 smithi163 ceph-mon[36873]: pgmap v7856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13769.173115 secs 2024-07-26T10:45:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13751.434317 secs 2024-07-26T10:45:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:22 smithi042 ceph-mon[32775]: pgmap v7856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13769.173115 secs 2024-07-26T10:45:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:24 smithi164 ceph-mon[36970]: pgmap v7857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:24 smithi163 ceph-mon[36873]: pgmap v7857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:24 smithi042 ceph-mon[32775]: pgmap v7857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:26 smithi164 ceph-mon[36970]: pgmap v7858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:26 smithi163 ceph-mon[36873]: pgmap v7858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:26 smithi042 ceph-mon[32775]: pgmap v7858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13756.434423 secs 2024-07-26T10:45:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13774.173190 secs 2024-07-26T10:45:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13756.434423 secs 2024-07-26T10:45:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13774.173190 secs 2024-07-26T10:45:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13756.434423 secs 2024-07-26T10:45:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13774.173190 secs 2024-07-26T10:45:28.112 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:45:28.110+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:45:28.112 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:45:28.110+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:45:28.112 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:45:28.110+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:45:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:28 smithi164 ceph-mon[36970]: pgmap v7859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:28 smithi163 ceph-mon[36873]: pgmap v7859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:28 smithi042 ceph-mon[32775]: pgmap v7859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:30 smithi164 ceph-mon[36970]: pgmap v7860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:30.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:30 smithi163 ceph-mon[36873]: pgmap v7860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:30 smithi042 ceph-mon[32775]: pgmap v7860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13761.434511 secs 2024-07-26T10:45:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:32 smithi164 ceph-mon[36970]: pgmap v7861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13779.173310 secs 2024-07-26T10:45:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13761.434511 secs 2024-07-26T10:45:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:32 smithi163 ceph-mon[36873]: pgmap v7861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13779.173310 secs 2024-07-26T10:45:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13761.434511 secs 2024-07-26T10:45:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:32 smithi042 ceph-mon[32775]: pgmap v7861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13779.173310 secs 2024-07-26T10:45:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:34 smithi164 ceph-mon[36970]: pgmap v7862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:34 smithi163 ceph-mon[36873]: pgmap v7862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:34 smithi042 ceph-mon[32775]: pgmap v7862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:36 smithi164 ceph-mon[36970]: pgmap v7863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:36 smithi163 ceph-mon[36873]: pgmap v7863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:36 smithi042 ceph-mon[32775]: pgmap v7863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13766.434674 secs 2024-07-26T10:45:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13784.173453 secs 2024-07-26T10:45:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13766.434674 secs 2024-07-26T10:45:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13784.173453 secs 2024-07-26T10:45:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13766.434674 secs 2024-07-26T10:45:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13784.173453 secs 2024-07-26T10:45:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:38 smithi163 ceph-mon[36873]: pgmap v7864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:38 smithi042 ceph-mon[32775]: pgmap v7864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:38 smithi164 ceph-mon[36970]: pgmap v7864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:40 smithi163 ceph-mon[36873]: pgmap v7865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:40 smithi042 ceph-mon[32775]: pgmap v7865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:40 smithi164 ceph-mon[36970]: pgmap v7865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13771.434825 secs 2024-07-26T10:45:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:42 smithi163 ceph-mon[36873]: pgmap v7866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:42.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13789.173569 secs 2024-07-26T10:45:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13771.434825 secs 2024-07-26T10:45:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:42 smithi042 ceph-mon[32775]: pgmap v7866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13789.173569 secs 2024-07-26T10:45:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13771.434825 secs 2024-07-26T10:45:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:42 smithi164 ceph-mon[36970]: pgmap v7866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13789.173569 secs 2024-07-26T10:45:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:45:43.111+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:45:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:45:43.111+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:45:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:45:43.111+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:45:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:44 smithi163 ceph-mon[36873]: pgmap v7867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:44 smithi042 ceph-mon[32775]: pgmap v7867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:44 smithi164 ceph-mon[36970]: pgmap v7867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:46 smithi042 ceph-mon[32775]: pgmap v7868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:46 smithi163 ceph-mon[36873]: pgmap v7868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:46 smithi164 ceph-mon[36970]: pgmap v7868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13776.434906 secs 2024-07-26T10:45:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13794.173707 secs 2024-07-26T10:45:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13776.434906 secs 2024-07-26T10:45:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13794.173707 secs 2024-07-26T10:45:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13776.434906 secs 2024-07-26T10:45:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13794.173707 secs 2024-07-26T10:45:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:48 smithi163 ceph-mon[36873]: pgmap v7869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:48 smithi042 ceph-mon[32775]: pgmap v7869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:48 smithi164 ceph-mon[36970]: pgmap v7869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:50 smithi163 ceph-mon[36873]: pgmap v7870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:50 smithi042 ceph-mon[32775]: pgmap v7870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:50 smithi164 ceph-mon[36970]: pgmap v7870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13781.435035 secs 2024-07-26T10:45:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:52 smithi163 ceph-mon[36873]: pgmap v7871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13799.173805 secs 2024-07-26T10:45:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13781.435035 secs 2024-07-26T10:45:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:52 smithi042 ceph-mon[32775]: pgmap v7871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13799.173805 secs 2024-07-26T10:45:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13781.435035 secs 2024-07-26T10:45:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:52 smithi164 ceph-mon[36970]: pgmap v7871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13799.173805 secs 2024-07-26T10:45:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:54 smithi163 ceph-mon[36873]: pgmap v7872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:54 smithi042 ceph-mon[32775]: pgmap v7872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:54 smithi164 ceph-mon[36970]: pgmap v7872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:56 smithi042 ceph-mon[32775]: pgmap v7873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:56 smithi164 ceph-mon[36970]: pgmap v7873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:56 smithi163 ceph-mon[36873]: pgmap v7873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:45:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13786.435162 secs 2024-07-26T10:45:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13804.173904 secs 2024-07-26T10:45:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13786.435162 secs 2024-07-26T10:45:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13804.173904 secs 2024-07-26T10:45:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13786.435162 secs 2024-07-26T10:45:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13804.173904 secs 2024-07-26T10:45:58.118 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:45:58.117+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:45:58.119 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:45:58.117+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:45:58.119 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:45:58.117+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:45:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:45:58 smithi042 ceph-mon[32775]: pgmap v7874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:45:58 smithi164 ceph-mon[36970]: pgmap v7874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:45:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:45:58 smithi163 ceph-mon[36873]: pgmap v7874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:00 smithi042 ceph-mon[32775]: pgmap v7875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:00 smithi164 ceph-mon[36970]: pgmap v7875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:00 smithi163 ceph-mon[36873]: pgmap v7875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:02.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13791.435245 secs 2024-07-26T10:46:02.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:02 smithi164 ceph-mon[36970]: pgmap v7876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:02.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13809.174027 secs 2024-07-26T10:46:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13791.435245 secs 2024-07-26T10:46:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:02 smithi042 ceph-mon[32775]: pgmap v7876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13809.174027 secs 2024-07-26T10:46:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13791.435245 secs 2024-07-26T10:46:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:02 smithi163 ceph-mon[36873]: pgmap v7876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13809.174027 secs 2024-07-26T10:46:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:46:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:46:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:46:04.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:04 smithi042 ceph-mon[32775]: pgmap v7877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:04.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:46:04.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:46:04.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:46:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:04 smithi164 ceph-mon[36970]: pgmap v7877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:46:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:46:04.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:46:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:04 smithi163 ceph-mon[36873]: pgmap v7877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:46:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:46:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:46:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:05 smithi042 ceph-mon[32775]: pgmap v7878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:46:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:05 smithi042 ceph-mon[32775]: pgmap v7879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:46:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:05 smithi164 ceph-mon[36970]: pgmap v7878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:46:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:05 smithi164 ceph-mon[36970]: pgmap v7879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:46:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:05 smithi163 ceph-mon[36873]: pgmap v7878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:46:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:05 smithi163 ceph-mon[36873]: pgmap v7879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:46:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:07 smithi042 ceph-mon[32775]: pgmap v7880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:46:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13796.435369 secs 2024-07-26T10:46:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13814.174127 secs 2024-07-26T10:46:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:07 smithi164 ceph-mon[36970]: pgmap v7880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:46:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13796.435369 secs 2024-07-26T10:46:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13814.174127 secs 2024-07-26T10:46:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:07 smithi163 ceph-mon[36873]: pgmap v7880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:46:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13796.435369 secs 2024-07-26T10:46:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13814.174127 secs 2024-07-26T10:46:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:09 smithi042 ceph-mon[32775]: pgmap v7881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:46:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:09 smithi164 ceph-mon[36970]: pgmap v7881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:46:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:09 smithi163 ceph-mon[36873]: pgmap v7881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:46:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:11 smithi042 ceph-mon[32775]: pgmap v7882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:46:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:11 smithi164 ceph-mon[36970]: pgmap v7882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:46:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:11 smithi163 ceph-mon[36873]: pgmap v7882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:46:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13801.435502 secs 2024-07-26T10:46:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13819.174249 secs 2024-07-26T10:46:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13801.435502 secs 2024-07-26T10:46:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13819.174249 secs 2024-07-26T10:46:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13801.435502 secs 2024-07-26T10:46:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13819.174249 secs 2024-07-26T10:46:13.119 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:46:13.118+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:46:13.119 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:46:13.118+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:46:13.120 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:46:13.118+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:46:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:13 smithi042 ceph-mon[32775]: pgmap v7883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:46:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:13 smithi164 ceph-mon[36970]: pgmap v7883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:46:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:13 smithi163 ceph-mon[36873]: pgmap v7883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:46:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:15 smithi042 ceph-mon[32775]: pgmap v7884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:46:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:15 smithi164 ceph-mon[36970]: pgmap v7884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:46:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:15 smithi163 ceph-mon[36873]: pgmap v7884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:46:17.561 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:17 smithi163 ceph-mon[36873]: pgmap v7885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:17.561 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13806.435574 secs 2024-07-26T10:46:17.561 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13824.174369 secs 2024-07-26T10:46:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:17 smithi042 ceph-mon[32775]: pgmap v7885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13806.435574 secs 2024-07-26T10:46:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13824.174369 secs 2024-07-26T10:46:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:17 smithi164 ceph-mon[36970]: pgmap v7885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13806.435574 secs 2024-07-26T10:46:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13824.174369 secs 2024-07-26T10:46:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:19 smithi042 ceph-mon[32775]: pgmap v7886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:19 smithi164 ceph-mon[36970]: pgmap v7886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:19 smithi163 ceph-mon[36873]: pgmap v7886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:21 smithi042 ceph-mon[32775]: pgmap v7887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:21 smithi164 ceph-mon[36970]: pgmap v7887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:21 smithi163 ceph-mon[36873]: pgmap v7887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13811.435702 secs 2024-07-26T10:46:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13829.174440 secs 2024-07-26T10:46:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13811.435702 secs 2024-07-26T10:46:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13829.174440 secs 2024-07-26T10:46:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13811.435702 secs 2024-07-26T10:46:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13829.174440 secs 2024-07-26T10:46:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:23 smithi042 ceph-mon[32775]: pgmap v7888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:23 smithi164 ceph-mon[36970]: pgmap v7888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:23 smithi163 ceph-mon[36873]: pgmap v7888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:25 smithi042 ceph-mon[32775]: pgmap v7889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:46:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:25 smithi164 ceph-mon[36970]: pgmap v7889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:46:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:25 smithi163 ceph-mon[36873]: pgmap v7889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:46:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:27 smithi042 ceph-mon[32775]: pgmap v7890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13816.435830 secs 2024-07-26T10:46:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13834.174564 secs 2024-07-26T10:46:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:27 smithi164 ceph-mon[36970]: pgmap v7890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13816.435830 secs 2024-07-26T10:46:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13834.174564 secs 2024-07-26T10:46:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:27 smithi163 ceph-mon[36873]: pgmap v7890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13816.435830 secs 2024-07-26T10:46:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13834.174564 secs 2024-07-26T10:46:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:46:28.123+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:46:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:46:28.123+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:46:28.125 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:46:28.123+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:46:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:29 smithi042 ceph-mon[32775]: pgmap v7891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:29 smithi164 ceph-mon[36970]: pgmap v7891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:29 smithi163 ceph-mon[36873]: pgmap v7891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:31 smithi042 ceph-mon[32775]: pgmap v7892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:31 smithi164 ceph-mon[36970]: pgmap v7892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:31 smithi163 ceph-mon[36873]: pgmap v7892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:32.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13821.435954 secs 2024-07-26T10:46:32.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13839.174650 secs 2024-07-26T10:46:32.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13821.435954 secs 2024-07-26T10:46:32.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13839.174650 secs 2024-07-26T10:46:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13821.435954 secs 2024-07-26T10:46:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13839.174650 secs 2024-07-26T10:46:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:33 smithi042 ceph-mon[32775]: pgmap v7893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:33 smithi164 ceph-mon[36970]: pgmap v7893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:33 smithi163 ceph-mon[36873]: pgmap v7893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:35 smithi042 ceph-mon[32775]: pgmap v7894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:46:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:35 smithi164 ceph-mon[36970]: pgmap v7894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:46:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:35 smithi163 ceph-mon[36873]: pgmap v7894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:46:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:37 smithi042 ceph-mon[32775]: pgmap v7895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13826.436058 secs 2024-07-26T10:46:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13844.174770 secs 2024-07-26T10:46:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:37 smithi164 ceph-mon[36970]: pgmap v7895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13826.436058 secs 2024-07-26T10:46:37.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13844.174770 secs 2024-07-26T10:46:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:37 smithi163 ceph-mon[36873]: pgmap v7895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13826.436058 secs 2024-07-26T10:46:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13844.174770 secs 2024-07-26T10:46:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:39 smithi042 ceph-mon[32775]: pgmap v7896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:39 smithi164 ceph-mon[36970]: pgmap v7896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:39 smithi163 ceph-mon[36873]: pgmap v7896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:41 smithi042 ceph-mon[32775]: pgmap v7897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:41 smithi164 ceph-mon[36970]: pgmap v7897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:41 smithi163 ceph-mon[36873]: pgmap v7897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13831.436198 secs 2024-07-26T10:46:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13849.174850 secs 2024-07-26T10:46:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13831.436198 secs 2024-07-26T10:46:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13849.174850 secs 2024-07-26T10:46:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13831.436198 secs 2024-07-26T10:46:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13849.174850 secs 2024-07-26T10:46:43.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:46:43.124+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:46:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:46:43.124+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:46:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:46:43.124+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:46:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:43 smithi042 ceph-mon[32775]: pgmap v7898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:43 smithi164 ceph-mon[36970]: pgmap v7898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:43 smithi163 ceph-mon[36873]: pgmap v7898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:45 smithi042 ceph-mon[32775]: pgmap v7899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:46:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:45 smithi164 ceph-mon[36970]: pgmap v7899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:46:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:45 smithi163 ceph-mon[36873]: pgmap v7899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:46:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:47 smithi042 ceph-mon[32775]: pgmap v7900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13836.436293 secs 2024-07-26T10:46:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13854.174970 secs 2024-07-26T10:46:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:47 smithi164 ceph-mon[36970]: pgmap v7900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13836.436293 secs 2024-07-26T10:46:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13854.174970 secs 2024-07-26T10:46:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:47 smithi163 ceph-mon[36873]: pgmap v7900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13836.436293 secs 2024-07-26T10:46:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13854.174970 secs 2024-07-26T10:46:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:49 smithi042 ceph-mon[32775]: pgmap v7901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:49 smithi164 ceph-mon[36970]: pgmap v7901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:49 smithi163 ceph-mon[36873]: pgmap v7901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:51 smithi042 ceph-mon[32775]: pgmap v7902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:51 smithi164 ceph-mon[36970]: pgmap v7902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:51 smithi163 ceph-mon[36873]: pgmap v7902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13841.436360 secs 2024-07-26T10:46:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13859.175109 secs 2024-07-26T10:46:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13841.436360 secs 2024-07-26T10:46:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13859.175109 secs 2024-07-26T10:46:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13841.436360 secs 2024-07-26T10:46:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13859.175109 secs 2024-07-26T10:46:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:53 smithi042 ceph-mon[32775]: pgmap v7903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:53 smithi164 ceph-mon[36970]: pgmap v7903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:53 smithi163 ceph-mon[36873]: pgmap v7903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:55 smithi042 ceph-mon[32775]: pgmap v7904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:46:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:55 smithi164 ceph-mon[36970]: pgmap v7904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:46:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:55 smithi163 ceph-mon[36873]: pgmap v7904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:46:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:57 smithi042 ceph-mon[32775]: pgmap v7905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13846.436496 secs 2024-07-26T10:46:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13864.175236 secs 2024-07-26T10:46:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:57 smithi164 ceph-mon[36970]: pgmap v7905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13846.436496 secs 2024-07-26T10:46:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13864.175236 secs 2024-07-26T10:46:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:57 smithi163 ceph-mon[36873]: pgmap v7905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:46:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13846.436496 secs 2024-07-26T10:46:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13864.175236 secs 2024-07-26T10:46:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:46:58.125+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:46:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:46:58.125+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:46:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:46:58.125+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:46:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:46:59 smithi164 ceph-mon[36970]: pgmap v7906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:46:59 smithi163 ceph-mon[36873]: pgmap v7906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:46:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:46:59 smithi042 ceph-mon[32775]: pgmap v7906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:00.037+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:47:00.039 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:00.037+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:47:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:00.038+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T10:47:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:00.038+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:47:00.040 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:00.038+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T10:47:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:00.038+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:47:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:00.038+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:47:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:00.038+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:47:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:01 smithi164 ceph-mon[36970]: pgmap v7907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:01 smithi163 ceph-mon[36873]: pgmap v7907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:01 smithi042 ceph-mon[32775]: pgmap v7907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:02.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13851.436562 secs 2024-07-26T10:47:02.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13869.175357 secs 2024-07-26T10:47:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13851.436562 secs 2024-07-26T10:47:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13869.175357 secs 2024-07-26T10:47:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13851.436562 secs 2024-07-26T10:47:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13869.175357 secs 2024-07-26T10:47:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:03 smithi164 ceph-mon[36970]: pgmap v7908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:03 smithi163 ceph-mon[36873]: pgmap v7908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:03.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:03 smithi042 ceph-mon[32775]: pgmap v7908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:47:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:47:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:47:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:47:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:47:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:47:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:47:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:47:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:47:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:47:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:47:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:47:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:05 smithi164 ceph-mon[36970]: pgmap v7909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:47:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:05 smithi164 ceph-mon[36970]: pgmap v7910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:47:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:05 smithi164 ceph-mon[36970]: pgmap v7911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:47:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:05 smithi163 ceph-mon[36873]: pgmap v7909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:47:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:05 smithi163 ceph-mon[36873]: pgmap v7910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:47:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:05 smithi163 ceph-mon[36873]: pgmap v7911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:47:05.799 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:05 smithi042 ceph-mon[32775]: pgmap v7909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:47:05.799 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:05 smithi042 ceph-mon[32775]: pgmap v7910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:47:05.799 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:05 smithi042 ceph-mon[32775]: pgmap v7911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:47:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13856.436687 secs 2024-07-26T10:47:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:07 smithi164 ceph-mon[36970]: pgmap v7912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:47:07.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13874.175477 secs 2024-07-26T10:47:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13856.436687 secs 2024-07-26T10:47:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:07 smithi163 ceph-mon[36873]: pgmap v7912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:47:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13874.175477 secs 2024-07-26T10:47:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13856.436687 secs 2024-07-26T10:47:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:07 smithi042 ceph-mon[32775]: pgmap v7912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:47:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13874.175477 secs 2024-07-26T10:47:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:09 smithi164 ceph-mon[36970]: pgmap v7913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:47:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:09 smithi163 ceph-mon[36873]: pgmap v7913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:47:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:09 smithi042 ceph-mon[32775]: pgmap v7913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:47:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:11 smithi164 ceph-mon[36970]: pgmap v7914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:47:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:11 smithi163 ceph-mon[36873]: pgmap v7914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:47:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:11 smithi042 ceph-mon[32775]: pgmap v7914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:47:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13861.436782 secs 2024-07-26T10:47:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13879.175595 secs 2024-07-26T10:47:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13861.436782 secs 2024-07-26T10:47:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13879.175595 secs 2024-07-26T10:47:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13861.436782 secs 2024-07-26T10:47:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13879.175595 secs 2024-07-26T10:47:13.128 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:13.126+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:47:13.128 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:13.126+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:47:13.128 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:13.126+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:47:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:13 smithi164 ceph-mon[36970]: pgmap v7915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:47:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:13 smithi163 ceph-mon[36873]: pgmap v7915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:47:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:13 smithi042 ceph-mon[32775]: pgmap v7915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:47:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:15 smithi164 ceph-mon[36970]: pgmap v7916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:47:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:15 smithi163 ceph-mon[36873]: pgmap v7916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:47:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:15 smithi042 ceph-mon[32775]: pgmap v7916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:47:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13866.436904 secs 2024-07-26T10:47:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:17 smithi164 ceph-mon[36970]: pgmap v7917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13884.175717 secs 2024-07-26T10:47:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13866.436904 secs 2024-07-26T10:47:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:17 smithi163 ceph-mon[36873]: pgmap v7917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13884.175717 secs 2024-07-26T10:47:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13866.436904 secs 2024-07-26T10:47:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:17 smithi042 ceph-mon[32775]: pgmap v7917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13884.175717 secs 2024-07-26T10:47:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:19 smithi042 ceph-mon[32775]: pgmap v7918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:47:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:19 smithi164 ceph-mon[36970]: pgmap v7918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:47:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:19 smithi163 ceph-mon[36873]: pgmap v7918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:47:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:21 smithi164 ceph-mon[36970]: pgmap v7919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:21 smithi163 ceph-mon[36873]: pgmap v7919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:21 smithi042 ceph-mon[32775]: pgmap v7919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13871.437025 secs 2024-07-26T10:47:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13889.175794 secs 2024-07-26T10:47:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13871.437025 secs 2024-07-26T10:47:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13889.175794 secs 2024-07-26T10:47:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13871.437025 secs 2024-07-26T10:47:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13889.175794 secs 2024-07-26T10:47:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:23 smithi164 ceph-mon[36970]: pgmap v7920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:23 smithi163 ceph-mon[36873]: pgmap v7920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:23 smithi042 ceph-mon[32775]: pgmap v7920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:25 smithi164 ceph-mon[36970]: pgmap v7921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:25 smithi163 ceph-mon[36873]: pgmap v7921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:25 smithi042 ceph-mon[32775]: pgmap v7921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13876.437167 secs 2024-07-26T10:47:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:27 smithi164 ceph-mon[36970]: pgmap v7922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13894.175957 secs 2024-07-26T10:47:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13876.437167 secs 2024-07-26T10:47:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:27 smithi163 ceph-mon[36873]: pgmap v7922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13894.175957 secs 2024-07-26T10:47:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13876.437167 secs 2024-07-26T10:47:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:27 smithi042 ceph-mon[32775]: pgmap v7922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13894.175957 secs 2024-07-26T10:47:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:28.131+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:47:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:28.131+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:47:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:28.131+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:47:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:29 smithi164 ceph-mon[36970]: pgmap v7923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:47:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:29 smithi163 ceph-mon[36873]: pgmap v7923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:47:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:29 smithi042 ceph-mon[32775]: pgmap v7923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:47:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:31 smithi164 ceph-mon[36970]: pgmap v7924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:31 smithi163 ceph-mon[36873]: pgmap v7924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:31 smithi042 ceph-mon[32775]: pgmap v7924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13881.437287 secs 2024-07-26T10:47:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13899.176073 secs 2024-07-26T10:47:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13881.437287 secs 2024-07-26T10:47:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13899.176073 secs 2024-07-26T10:47:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13881.437287 secs 2024-07-26T10:47:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13899.176073 secs 2024-07-26T10:47:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:33 smithi163 ceph-mon[36873]: pgmap v7925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:33 smithi042 ceph-mon[32775]: pgmap v7925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:33 smithi164 ceph-mon[36970]: pgmap v7925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:35 smithi163 ceph-mon[36873]: pgmap v7926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:35 smithi042 ceph-mon[32775]: pgmap v7926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:35 smithi164 ceph-mon[36970]: pgmap v7926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13886.437369 secs 2024-07-26T10:47:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:37 smithi163 ceph-mon[36873]: pgmap v7927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13904.176165 secs 2024-07-26T10:47:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13886.437369 secs 2024-07-26T10:47:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:37 smithi042 ceph-mon[32775]: pgmap v7927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13904.176165 secs 2024-07-26T10:47:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13886.437369 secs 2024-07-26T10:47:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:37 smithi164 ceph-mon[36970]: pgmap v7927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13904.176165 secs 2024-07-26T10:47:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:39 smithi163 ceph-mon[36873]: pgmap v7928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:47:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:39 smithi042 ceph-mon[32775]: pgmap v7928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:47:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:39 smithi164 ceph-mon[36970]: pgmap v7928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:47:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:40 smithi042 ceph-mon[32775]: pgmap v7929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:40 smithi164 ceph-mon[36970]: pgmap v7929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:40 smithi163 ceph-mon[36873]: pgmap v7929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13891.437447 secs 2024-07-26T10:47:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13891.437447 secs 2024-07-26T10:47:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13891.437447 secs 2024-07-26T10:47:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:43.132+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:47:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:43.132+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:47:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:43.132+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:47:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13909.176250 secs 2024-07-26T10:47:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:42 smithi042 ceph-mon[32775]: pgmap v7930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13909.176250 secs 2024-07-26T10:47:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:42 smithi164 ceph-mon[36970]: pgmap v7930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13909.176250 secs 2024-07-26T10:47:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:42 smithi163 ceph-mon[36873]: pgmap v7930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:45 smithi042 ceph-mon[32775]: pgmap v7931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:45 smithi164 ceph-mon[36970]: pgmap v7931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:45 smithi163 ceph-mon[36873]: pgmap v7931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13896.437542 secs 2024-07-26T10:47:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:47 smithi042 ceph-mon[32775]: pgmap v7932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13914.176319 secs 2024-07-26T10:47:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13896.437542 secs 2024-07-26T10:47:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:47 smithi164 ceph-mon[36970]: pgmap v7932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13914.176319 secs 2024-07-26T10:47:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13896.437542 secs 2024-07-26T10:47:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:47 smithi163 ceph-mon[36873]: pgmap v7932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13914.176319 secs 2024-07-26T10:47:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:49 smithi042 ceph-mon[32775]: pgmap v7933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:47:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:49 smithi164 ceph-mon[36970]: pgmap v7933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:47:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:49 smithi163 ceph-mon[36873]: pgmap v7933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:47:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:51 smithi042 ceph-mon[32775]: pgmap v7934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:51 smithi164 ceph-mon[36970]: pgmap v7934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:51 smithi163 ceph-mon[36873]: pgmap v7934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13901.437668 secs 2024-07-26T10:47:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13919.176464 secs 2024-07-26T10:47:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13901.437668 secs 2024-07-26T10:47:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13919.176464 secs 2024-07-26T10:47:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13901.437668 secs 2024-07-26T10:47:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13919.176464 secs 2024-07-26T10:47:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:53 smithi042 ceph-mon[32775]: pgmap v7935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:53 smithi164 ceph-mon[36970]: pgmap v7935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:53 smithi163 ceph-mon[36873]: pgmap v7935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:55 smithi042 ceph-mon[32775]: pgmap v7936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:55.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:55 smithi164 ceph-mon[36970]: pgmap v7936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:55 smithi163 ceph-mon[36873]: pgmap v7936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:47:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13906.437790 secs 2024-07-26T10:47:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:57 smithi042 ceph-mon[32775]: pgmap v7937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13924.176602 secs 2024-07-26T10:47:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13906.437790 secs 2024-07-26T10:47:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:57 smithi164 ceph-mon[36970]: pgmap v7937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13924.176602 secs 2024-07-26T10:47:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13906.437790 secs 2024-07-26T10:47:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:57 smithi163 ceph-mon[36873]: pgmap v7937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:47:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13924.176602 secs 2024-07-26T10:47:58.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:58.136+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:47:58.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:58.136+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:47:58.138 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:47:58.136+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:47:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:47:59 smithi042 ceph-mon[32775]: pgmap v7938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:47:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:47:59 smithi164 ceph-mon[36970]: pgmap v7938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:47:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:47:59 smithi163 ceph-mon[36873]: pgmap v7938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:48:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:01 smithi042 ceph-mon[32775]: pgmap v7939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:01 smithi164 ceph-mon[36970]: pgmap v7939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:01 smithi163 ceph-mon[36873]: pgmap v7939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:02.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13911.437912 secs 2024-07-26T10:48:02.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13929.176720 secs 2024-07-26T10:48:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13911.437912 secs 2024-07-26T10:48:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13929.176720 secs 2024-07-26T10:48:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13911.437912 secs 2024-07-26T10:48:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13929.176720 secs 2024-07-26T10:48:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:03 smithi042 ceph-mon[32775]: pgmap v7940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:03 smithi164 ceph-mon[36970]: pgmap v7940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:03 smithi163 ceph-mon[36873]: pgmap v7940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:05 smithi042 ceph-mon[32775]: pgmap v7941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:48:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:48:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:48:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:48:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:05 smithi164 ceph-mon[36970]: pgmap v7941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:48:05.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:48:05.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:48:05.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:48:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:05 smithi163 ceph-mon[36873]: pgmap v7941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:48:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:48:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:48:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:48:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:06 smithi042 ceph-mon[32775]: pgmap v7942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:48:06.658 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:06 smithi042 ceph-mon[32775]: pgmap v7943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:48:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:06 smithi164 ceph-mon[36970]: pgmap v7942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:48:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:06 smithi164 ceph-mon[36970]: pgmap v7943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:48:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:06 smithi163 ceph-mon[36873]: pgmap v7942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:48:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:06 smithi163 ceph-mon[36873]: pgmap v7943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:48:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13916.438032 secs 2024-07-26T10:48:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13934.176847 secs 2024-07-26T10:48:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13916.438032 secs 2024-07-26T10:48:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13934.176847 secs 2024-07-26T10:48:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13916.438032 secs 2024-07-26T10:48:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13934.176847 secs 2024-07-26T10:48:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:08 smithi042 ceph-mon[32775]: pgmap v7944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:48:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:08 smithi164 ceph-mon[36970]: pgmap v7944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:48:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:08 smithi163 ceph-mon[36873]: pgmap v7944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:48:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:10 smithi042 ceph-mon[32775]: pgmap v7945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:48:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:10 smithi164 ceph-mon[36970]: pgmap v7945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:48:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:10 smithi163 ceph-mon[36873]: pgmap v7945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:48:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:12 smithi042 ceph-mon[32775]: pgmap v7946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:48:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13921.438153 secs 2024-07-26T10:48:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13939.176945 secs 2024-07-26T10:48:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:12 smithi164 ceph-mon[36970]: pgmap v7946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:48:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13921.438153 secs 2024-07-26T10:48:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13939.176945 secs 2024-07-26T10:48:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:12 smithi163 ceph-mon[36873]: pgmap v7946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:48:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13921.438153 secs 2024-07-26T10:48:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13939.176945 secs 2024-07-26T10:48:13.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:48:13.137+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:48:13.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:48:13.137+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:48:13.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:48:13.137+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:48:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:14 smithi042 ceph-mon[32775]: pgmap v7947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:48:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:14 smithi164 ceph-mon[36970]: pgmap v7947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:48:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:14 smithi163 ceph-mon[36873]: pgmap v7947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:48:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:16 smithi042 ceph-mon[32775]: pgmap v7948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:48:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:16 smithi164 ceph-mon[36970]: pgmap v7948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:48:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:16 smithi163 ceph-mon[36873]: pgmap v7948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:48:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13926.438283 secs 2024-07-26T10:48:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13944.176999 secs 2024-07-26T10:48:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13926.438283 secs 2024-07-26T10:48:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13944.176999 secs 2024-07-26T10:48:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13926.438283 secs 2024-07-26T10:48:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13944.176999 secs 2024-07-26T10:48:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:18 smithi042 ceph-mon[32775]: pgmap v7949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:18 smithi164 ceph-mon[36970]: pgmap v7949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:18 smithi163 ceph-mon[36873]: pgmap v7949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:20 smithi042 ceph-mon[32775]: pgmap v7950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:48:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:20 smithi164 ceph-mon[36970]: pgmap v7950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:48:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:20 smithi163 ceph-mon[36873]: pgmap v7950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:48:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:22 smithi042 ceph-mon[32775]: pgmap v7951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13931.438409 secs 2024-07-26T10:48:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13949.177108 secs 2024-07-26T10:48:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:22 smithi164 ceph-mon[36970]: pgmap v7951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13931.438409 secs 2024-07-26T10:48:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13949.177108 secs 2024-07-26T10:48:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:22 smithi163 ceph-mon[36873]: pgmap v7951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13931.438409 secs 2024-07-26T10:48:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13949.177108 secs 2024-07-26T10:48:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:24 smithi164 ceph-mon[36970]: pgmap v7952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:24 smithi163 ceph-mon[36873]: pgmap v7952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:24 smithi042 ceph-mon[32775]: pgmap v7952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:26 smithi164 ceph-mon[36970]: pgmap v7953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:26 smithi163 ceph-mon[36873]: pgmap v7953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:26 smithi042 ceph-mon[32775]: pgmap v7953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13936.438532 secs 2024-07-26T10:48:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13954.177200 secs 2024-07-26T10:48:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13936.438532 secs 2024-07-26T10:48:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13954.177200 secs 2024-07-26T10:48:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13936.438532 secs 2024-07-26T10:48:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13954.177200 secs 2024-07-26T10:48:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:48:28.142+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:48:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:48:28.142+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:48:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:48:28.142+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:48:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:28 smithi164 ceph-mon[36970]: pgmap v7954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:28 smithi163 ceph-mon[36873]: pgmap v7954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:28 smithi042 ceph-mon[32775]: pgmap v7954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:30 smithi164 ceph-mon[36970]: pgmap v7955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:48:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:30 smithi163 ceph-mon[36873]: pgmap v7955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:48:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:30 smithi042 ceph-mon[32775]: pgmap v7955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:48:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:32 smithi164 ceph-mon[36970]: pgmap v7956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13941.438646 secs 2024-07-26T10:48:32.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13959.177324 secs 2024-07-26T10:48:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:32 smithi163 ceph-mon[36873]: pgmap v7956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13941.438646 secs 2024-07-26T10:48:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13959.177324 secs 2024-07-26T10:48:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:32 smithi042 ceph-mon[32775]: pgmap v7956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13941.438646 secs 2024-07-26T10:48:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13959.177324 secs 2024-07-26T10:48:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:34 smithi164 ceph-mon[36970]: pgmap v7957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:34 smithi163 ceph-mon[36873]: pgmap v7957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:34 smithi042 ceph-mon[32775]: pgmap v7957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:36.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:36 smithi042 ceph-mon[32775]: pgmap v7958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:36 smithi164 ceph-mon[36970]: pgmap v7958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:36 smithi163 ceph-mon[36873]: pgmap v7958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13946.438768 secs 2024-07-26T10:48:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13964.177470 secs 2024-07-26T10:48:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13946.438768 secs 2024-07-26T10:48:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13964.177470 secs 2024-07-26T10:48:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13946.438768 secs 2024-07-26T10:48:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13964.177470 secs 2024-07-26T10:48:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:38 smithi164 ceph-mon[36970]: pgmap v7959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:38 smithi163 ceph-mon[36873]: pgmap v7959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:38 smithi042 ceph-mon[32775]: pgmap v7959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:40 smithi164 ceph-mon[36970]: pgmap v7960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:48:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:40 smithi163 ceph-mon[36873]: pgmap v7960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:48:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:40 smithi042 ceph-mon[32775]: pgmap v7960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:48:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:42 smithi164 ceph-mon[36970]: pgmap v7961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13951.438889 secs 2024-07-26T10:48:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13969.177548 secs 2024-07-26T10:48:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:42 smithi163 ceph-mon[36873]: pgmap v7961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13951.438889 secs 2024-07-26T10:48:42.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13969.177548 secs 2024-07-26T10:48:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:42 smithi042 ceph-mon[32775]: pgmap v7961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13951.438889 secs 2024-07-26T10:48:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13969.177548 secs 2024-07-26T10:48:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:48:43.144+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:48:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:48:43.144+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:48:43.146 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:48:43.144+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:48:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:44 smithi164 ceph-mon[36970]: pgmap v7962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:44 smithi163 ceph-mon[36873]: pgmap v7962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:44 smithi042 ceph-mon[32775]: pgmap v7962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:46.727 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:46 smithi164 ceph-mon[36970]: pgmap v7963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:46 smithi163 ceph-mon[36873]: pgmap v7963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:46 smithi042 ceph-mon[32775]: pgmap v7963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13956.439032 secs 2024-07-26T10:48:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13974.177669 secs 2024-07-26T10:48:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13956.439032 secs 2024-07-26T10:48:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13974.177669 secs 2024-07-26T10:48:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13956.439032 secs 2024-07-26T10:48:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13974.177669 secs 2024-07-26T10:48:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:48 smithi164 ceph-mon[36970]: pgmap v7964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:48 smithi163 ceph-mon[36873]: pgmap v7964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:48 smithi042 ceph-mon[32775]: pgmap v7964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:50 smithi164 ceph-mon[36970]: pgmap v7965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:48:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:50 smithi163 ceph-mon[36873]: pgmap v7965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:48:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:50 smithi042 ceph-mon[32775]: pgmap v7965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:48:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:52 smithi164 ceph-mon[36970]: pgmap v7966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13961.439095 secs 2024-07-26T10:48:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13979.177796 secs 2024-07-26T10:48:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:52 smithi163 ceph-mon[36873]: pgmap v7966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13961.439095 secs 2024-07-26T10:48:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13979.177796 secs 2024-07-26T10:48:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:52 smithi042 ceph-mon[32775]: pgmap v7966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13961.439095 secs 2024-07-26T10:48:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13979.177796 secs 2024-07-26T10:48:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:54 smithi164 ceph-mon[36970]: pgmap v7967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:54 smithi163 ceph-mon[36873]: pgmap v7967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:54 smithi042 ceph-mon[32775]: pgmap v7967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:56 smithi164 ceph-mon[36970]: pgmap v7968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:56 smithi163 ceph-mon[36873]: pgmap v7968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:56 smithi042 ceph-mon[32775]: pgmap v7968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:48:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13966.439227 secs 2024-07-26T10:48:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13984.177915 secs 2024-07-26T10:48:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13966.439227 secs 2024-07-26T10:48:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13984.177915 secs 2024-07-26T10:48:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13966.439227 secs 2024-07-26T10:48:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13984.177915 secs 2024-07-26T10:48:58.151 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:48:58.149+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:48:58.151 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:48:58.149+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:48:58.151 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:48:58.149+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:48:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:48:58 smithi164 ceph-mon[36970]: pgmap v7969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:48:58 smithi163 ceph-mon[36873]: pgmap v7969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:48:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:48:58 smithi042 ceph-mon[32775]: pgmap v7969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:00 smithi163 ceph-mon[36873]: pgmap v7970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:49:00.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:00 smithi042 ceph-mon[32775]: pgmap v7970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:49:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:00 smithi164 ceph-mon[36970]: pgmap v7970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:49:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:02 smithi163 ceph-mon[36873]: pgmap v7971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13971.439298 secs 2024-07-26T10:49:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13989.178035 secs 2024-07-26T10:49:02.775 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:02 smithi164 ceph-mon[36970]: pgmap v7971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:02.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13971.439298 secs 2024-07-26T10:49:02.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13989.178035 secs 2024-07-26T10:49:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:02 smithi042 ceph-mon[32775]: pgmap v7971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13971.439298 secs 2024-07-26T10:49:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13989.178035 secs 2024-07-26T10:49:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:04 smithi163 ceph-mon[36873]: pgmap v7972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:04 smithi042 ceph-mon[32775]: pgmap v7972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:04 smithi164 ceph-mon[36970]: pgmap v7972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:05.787 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:49:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:06 smithi163 ceph-mon[36873]: pgmap v7973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:06.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:06 smithi042 ceph-mon[32775]: pgmap v7973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:06 smithi164 ceph-mon[36970]: pgmap v7973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13976.439363 secs 2024-07-26T10:49:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13994.178152 secs 2024-07-26T10:49:07.826 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13976.439363 secs 2024-07-26T10:49:07.826 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13994.178152 secs 2024-07-26T10:49:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13976.439363 secs 2024-07-26T10:49:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13994.178152 secs 2024-07-26T10:49:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:08 smithi042 ceph-mon[32775]: pgmap v7974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:08 smithi163 ceph-mon[36873]: pgmap v7974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:08 smithi164 ceph-mon[36970]: pgmap v7974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:49:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:49:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:10.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:10.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:09 smithi042 ceph-mon[32775]: pgmap v7975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:49:10.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:09 smithi042 ceph-mon[32775]: pgmap v7976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:49:10.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:49:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:49:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:49:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:10.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:10.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:09 smithi164 ceph-mon[36970]: pgmap v7975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:49:10.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:09 smithi164 ceph-mon[36970]: pgmap v7976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:49:10.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:49:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:49:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:49:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:49:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:49:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:09 smithi163 ceph-mon[36873]: pgmap v7975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:49:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:09 smithi163 ceph-mon[36873]: pgmap v7976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:49:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:49:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:11 smithi042 ceph-mon[32775]: pgmap v7977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:49:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13981.439507 secs 2024-07-26T10:49:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:11 smithi164 ceph-mon[36970]: pgmap v7977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:49:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13981.439507 secs 2024-07-26T10:49:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:11 smithi163 ceph-mon[36873]: pgmap v7977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:49:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13981.439507 secs 2024-07-26T10:49:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13999.178276 secs 2024-07-26T10:49:13.152 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:49:13.150+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:49:13.152 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:49:13.150+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:49:13.152 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:49:13.151+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:49:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13999.178276 secs 2024-07-26T10:49:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13999.178276 secs 2024-07-26T10:49:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:13 smithi042 ceph-mon[32775]: pgmap v7978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:49:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:13 smithi164 ceph-mon[36970]: pgmap v7978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:49:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:13 smithi163 ceph-mon[36873]: pgmap v7978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:49:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:15 smithi042 ceph-mon[32775]: pgmap v7979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:49:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:15 smithi164 ceph-mon[36970]: pgmap v7979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:49:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:15 smithi163 ceph-mon[36873]: pgmap v7979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:49:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13986.439651 secs 2024-07-26T10:49:17.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13986.439651 secs 2024-07-26T10:49:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13986.439651 secs 2024-07-26T10:49:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14004.178421 secs 2024-07-26T10:49:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:17 smithi042 ceph-mon[32775]: pgmap v7980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:49:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14004.178421 secs 2024-07-26T10:49:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:17 smithi164 ceph-mon[36970]: pgmap v7980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:49:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14004.178421 secs 2024-07-26T10:49:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:17 smithi163 ceph-mon[36873]: pgmap v7980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:49:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:19 smithi042 ceph-mon[32775]: pgmap v7981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:49:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:19 smithi164 ceph-mon[36970]: pgmap v7981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:49:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:19 smithi163 ceph-mon[36873]: pgmap v7981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:49:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:21 smithi042 ceph-mon[32775]: pgmap v7982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:49:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13991.439775 secs 2024-07-26T10:49:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:21 smithi164 ceph-mon[36970]: pgmap v7982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:49:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13991.439775 secs 2024-07-26T10:49:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:21 smithi163 ceph-mon[36873]: pgmap v7982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:49:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13991.439775 secs 2024-07-26T10:49:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14009.178541 secs 2024-07-26T10:49:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14009.178541 secs 2024-07-26T10:49:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14009.178541 secs 2024-07-26T10:49:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:23 smithi042 ceph-mon[32775]: pgmap v7983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:24.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:23 smithi164 ceph-mon[36970]: pgmap v7983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:23 smithi163 ceph-mon[36873]: pgmap v7983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:25 smithi042 ceph-mon[32775]: pgmap v7984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:25 smithi164 ceph-mon[36970]: pgmap v7984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:25 smithi163 ceph-mon[36873]: pgmap v7984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 13996.439906 secs 2024-07-26T10:49:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 13996.439906 secs 2024-07-26T10:49:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 13996.439906 secs 2024-07-26T10:49:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14014.178677 secs 2024-07-26T10:49:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:27 smithi042 ceph-mon[32775]: pgmap v7985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:28.158 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:49:28.156+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:49:28.158 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:49:28.156+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:49:28.158 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:49:28.156+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:49:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14014.178677 secs 2024-07-26T10:49:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:27 smithi164 ceph-mon[36970]: pgmap v7985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14014.178677 secs 2024-07-26T10:49:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:27 smithi163 ceph-mon[36873]: pgmap v7985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:29 smithi042 ceph-mon[32775]: pgmap v7986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:49:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:29 smithi164 ceph-mon[36970]: pgmap v7986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:49:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:29 smithi163 ceph-mon[36873]: pgmap v7986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:49:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:31 smithi042 ceph-mon[32775]: pgmap v7987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14001.440027 secs 2024-07-26T10:49:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:31 smithi164 ceph-mon[36970]: pgmap v7987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14001.440027 secs 2024-07-26T10:49:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:31 smithi163 ceph-mon[36873]: pgmap v7987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14001.440027 secs 2024-07-26T10:49:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14019.178832 secs 2024-07-26T10:49:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14019.178832 secs 2024-07-26T10:49:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14019.178832 secs 2024-07-26T10:49:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:33 smithi042 ceph-mon[32775]: pgmap v7988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:33 smithi164 ceph-mon[36970]: pgmap v7988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:33 smithi163 ceph-mon[36873]: pgmap v7988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:35 smithi042 ceph-mon[32775]: pgmap v7989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:35 smithi164 ceph-mon[36970]: pgmap v7989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:35 smithi163 ceph-mon[36873]: pgmap v7989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14006.440155 secs 2024-07-26T10:49:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14006.440155 secs 2024-07-26T10:49:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14006.440155 secs 2024-07-26T10:49:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14024.178966 secs 2024-07-26T10:49:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:37 smithi042 ceph-mon[32775]: pgmap v7990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14024.178966 secs 2024-07-26T10:49:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:37 smithi164 ceph-mon[36970]: pgmap v7990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14024.178966 secs 2024-07-26T10:49:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:37 smithi163 ceph-mon[36873]: pgmap v7990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:39 smithi164 ceph-mon[36970]: pgmap v7991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:49:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:39 smithi163 ceph-mon[36873]: pgmap v7991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:49:40.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:39 smithi042 ceph-mon[32775]: pgmap v7991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:49:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:41 smithi164 ceph-mon[36970]: pgmap v7992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14011.440252 secs 2024-07-26T10:49:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:41 smithi163 ceph-mon[36873]: pgmap v7992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14011.440252 secs 2024-07-26T10:49:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:41 smithi042 ceph-mon[32775]: pgmap v7992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14011.440252 secs 2024-07-26T10:49:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:49:43.157+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:49:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:49:43.157+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:49:43.159 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:49:43.157+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:49:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14029.179087 secs 2024-07-26T10:49:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14029.179087 secs 2024-07-26T10:49:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14029.179087 secs 2024-07-26T10:49:44.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:43 smithi164 ceph-mon[36970]: pgmap v7993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:43 smithi163 ceph-mon[36873]: pgmap v7993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:43 smithi042 ceph-mon[32775]: pgmap v7993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:45 smithi164 ceph-mon[36970]: pgmap v7994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:45 smithi163 ceph-mon[36873]: pgmap v7994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:45 smithi042 ceph-mon[32775]: pgmap v7994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14016.440333 secs 2024-07-26T10:49:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14016.440333 secs 2024-07-26T10:49:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14016.440333 secs 2024-07-26T10:49:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14034.179190 secs 2024-07-26T10:49:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:47 smithi164 ceph-mon[36970]: pgmap v7995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14034.179190 secs 2024-07-26T10:49:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:47 smithi163 ceph-mon[36873]: pgmap v7995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14034.179190 secs 2024-07-26T10:49:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:47 smithi042 ceph-mon[32775]: pgmap v7995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:49 smithi164 ceph-mon[36970]: pgmap v7996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:49:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:49 smithi163 ceph-mon[36873]: pgmap v7996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:49:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:49 smithi042 ceph-mon[32775]: pgmap v7996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:49:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:51 smithi164 ceph-mon[36970]: pgmap v7997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14021.440400 secs 2024-07-26T10:49:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:51 smithi163 ceph-mon[36873]: pgmap v7997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14021.440400 secs 2024-07-26T10:49:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:51 smithi042 ceph-mon[32775]: pgmap v7997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14021.440400 secs 2024-07-26T10:49:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14039.179316 secs 2024-07-26T10:49:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14039.179316 secs 2024-07-26T10:49:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14039.179316 secs 2024-07-26T10:49:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:53 smithi164 ceph-mon[36970]: pgmap v7998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:53 smithi163 ceph-mon[36873]: pgmap v7998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:53 smithi042 ceph-mon[32775]: pgmap v7998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:55 smithi164 ceph-mon[36970]: pgmap v7999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:55 smithi163 ceph-mon[36873]: pgmap v7999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:55 smithi042 ceph-mon[32775]: pgmap v7999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:49:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14026.440520 secs 2024-07-26T10:49:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14026.440520 secs 2024-07-26T10:49:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14026.440520 secs 2024-07-26T10:49:58.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:49:58.162+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:49:58.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:49:58.162+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:49:58.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:49:58.162+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:49:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14044.179396 secs 2024-07-26T10:49:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:57 smithi164 ceph-mon[36970]: pgmap v8000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14044.179396 secs 2024-07-26T10:49:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:57 smithi163 ceph-mon[36873]: pgmap v8000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:49:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14044.179396 secs 2024-07-26T10:49:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:57 smithi042 ceph-mon[32775]: pgmap v8000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:50:00.067+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:50:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:50:00.067+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:50:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:50:00.067+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:50:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:50:00.067+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:50:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:49:59 smithi164 ceph-mon[36970]: pgmap v8001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:50:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:49:59 smithi163 ceph-mon[36873]: pgmap v8001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:50:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:49:59 smithi042 ceph-mon[32775]: pgmap v8001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:50:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:50:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:50:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T10:50:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:01 smithi164 ceph-mon[36970]: pgmap v8002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:02.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14031.440597 secs 2024-07-26T10:50:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:01 smithi163 ceph-mon[36873]: pgmap v8002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14031.440597 secs 2024-07-26T10:50:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:01 smithi042 ceph-mon[32775]: pgmap v8002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14031.440597 secs 2024-07-26T10:50:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14049.179473 secs 2024-07-26T10:50:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14049.179473 secs 2024-07-26T10:50:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14049.179473 secs 2024-07-26T10:50:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:03 smithi164 ceph-mon[36970]: pgmap v8003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:03 smithi163 ceph-mon[36873]: pgmap v8003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:03 smithi042 ceph-mon[32775]: pgmap v8003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:05 smithi164 ceph-mon[36970]: pgmap v8004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:05 smithi163 ceph-mon[36873]: pgmap v8004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:05 smithi042 ceph-mon[32775]: pgmap v8004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14036.440742 secs 2024-07-26T10:50:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14036.440742 secs 2024-07-26T10:50:07.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14036.440742 secs 2024-07-26T10:50:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14054.179597 secs 2024-07-26T10:50:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:07 smithi163 ceph-mon[36873]: pgmap v8005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14054.179597 secs 2024-07-26T10:50:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:07 smithi042 ceph-mon[32775]: pgmap v8005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14054.179597 secs 2024-07-26T10:50:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:07 smithi164 ceph-mon[36970]: pgmap v8005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:09.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:08 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:08 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:08 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:50:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:09 smithi163 ceph-mon[36873]: pgmap v8006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:50:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:09 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:10.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:09 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T10:50:10.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:09 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T10:50:10.021 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:09 smithi164 ceph-mon[36970]: pgmap v8006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:50:10.021 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:10.021 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:10.021 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:10.022 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:09 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:10.022 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:09 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T10:50:10.022 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:09 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T10:50:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:09 smithi042 ceph-mon[32775]: pgmap v8006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:50:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:09 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:09 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T10:50:10.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:09 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T10:50:12.163 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:11 smithi163 ceph-mon[36873]: pgmap v8007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:12.163 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14041.440814 secs 2024-07-26T10:50:12.201 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:11 smithi164 ceph-mon[36970]: pgmap v8007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:12.202 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14041.440814 secs 2024-07-26T10:50:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:11 smithi042 ceph-mon[32775]: pgmap v8007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14041.440814 secs 2024-07-26T10:50:13.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:50:13.162+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:50:13.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:50:13.162+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:50:13.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:50:13.162+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:50:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14059.179689 secs 2024-07-26T10:50:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:50:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:50:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:50:13.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:50:13.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:13.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:13.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:12 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:50:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14059.179689 secs 2024-07-26T10:50:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:50:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:50:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:50:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:50:13.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:13.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:13.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:12 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:50:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14059.179689 secs 2024-07-26T10:50:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:50:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:50:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:50:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:50:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T10:50:13.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:50:13.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:50:13.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:12 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:50:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:13 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T10:50:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:13 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:50:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:13 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T10:50:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:13 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T10:50:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:13 smithi163 ceph-mon[36873]: pgmap v8008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:13 smithi163 ceph-mon[36873]: pgmap v8009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:50:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:13 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T10:50:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:13 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:50:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:13 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T10:50:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:13 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T10:50:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:13 smithi042 ceph-mon[32775]: pgmap v8008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:13 smithi042 ceph-mon[32775]: pgmap v8009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:50:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:13 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T10:50:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:13 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T10:50:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:13 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T10:50:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:13 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T10:50:14.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:13 smithi164 ceph-mon[36970]: pgmap v8008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:14.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:13 smithi164 ceph-mon[36970]: pgmap v8009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:50:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:15 smithi163 ceph-mon[36873]: pgmap v8010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:50:16.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:15 smithi042 ceph-mon[32775]: pgmap v8010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:50:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:15 smithi164 ceph-mon[36970]: pgmap v8010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:50:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14046.440935 secs 2024-07-26T10:50:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14046.440935 secs 2024-07-26T10:50:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14046.440935 secs 2024-07-26T10:50:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14064.179803 secs 2024-07-26T10:50:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:17 smithi163 ceph-mon[36873]: pgmap v8011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:50:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14064.179803 secs 2024-07-26T10:50:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:17 smithi042 ceph-mon[32775]: pgmap v8011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:50:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14064.179803 secs 2024-07-26T10:50:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:17 smithi164 ceph-mon[36970]: pgmap v8011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:50:20.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:20 smithi163 ceph-mon[36873]: pgmap v8012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:50:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:20 smithi042 ceph-mon[32775]: pgmap v8012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:50:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:20 smithi164 ceph-mon[36970]: pgmap v8012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:50:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:22 smithi163 ceph-mon[36873]: pgmap v8013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:50:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14051.441077 secs 2024-07-26T10:50:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:22 smithi042 ceph-mon[32775]: pgmap v8013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:50:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14051.441077 secs 2024-07-26T10:50:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:22 smithi164 ceph-mon[36970]: pgmap v8013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T10:50:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14051.441077 secs 2024-07-26T10:50:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:23 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14069.179964 secs 2024-07-26T10:50:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:23 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14069.179964 secs 2024-07-26T10:50:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:23 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14069.179964 secs 2024-07-26T10:50:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:24 smithi164 ceph-mon[36970]: pgmap v8014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:50:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:24 smithi042 ceph-mon[32775]: pgmap v8014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:50:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:24 smithi163 ceph-mon[36873]: pgmap v8014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:50:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:26 smithi042 ceph-mon[32775]: pgmap v8015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:26 smithi164 ceph-mon[36970]: pgmap v8015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:26 smithi163 ceph-mon[36873]: pgmap v8015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14056.441238 secs 2024-07-26T10:50:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14056.441238 secs 2024-07-26T10:50:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14056.441238 secs 2024-07-26T10:50:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:50:28.168+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:50:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:50:28.168+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:50:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:50:28.168+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:50:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:28 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14074.180088 secs 2024-07-26T10:50:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:28 smithi042 ceph-mon[32775]: pgmap v8016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:28 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14074.180088 secs 2024-07-26T10:50:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:28 smithi164 ceph-mon[36970]: pgmap v8016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:28 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14074.180088 secs 2024-07-26T10:50:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:28 smithi163 ceph-mon[36873]: pgmap v8016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:30.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:30 smithi042 ceph-mon[32775]: pgmap v8017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:50:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:30 smithi164 ceph-mon[36970]: pgmap v8017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:50:30.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:30 smithi163 ceph-mon[36873]: pgmap v8017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:50:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:32 smithi042 ceph-mon[32775]: pgmap v8018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14061.441358 secs 2024-07-26T10:50:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:32 smithi164 ceph-mon[36970]: pgmap v8018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14061.441358 secs 2024-07-26T10:50:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:32 smithi163 ceph-mon[36873]: pgmap v8018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14061.441358 secs 2024-07-26T10:50:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:33 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14079.180210 secs 2024-07-26T10:50:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:33 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14079.180210 secs 2024-07-26T10:50:33.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:33 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14079.180210 secs 2024-07-26T10:50:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:34 smithi042 ceph-mon[32775]: pgmap v8019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:34 smithi164 ceph-mon[36970]: pgmap v8019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:34 smithi163 ceph-mon[36873]: pgmap v8019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:36 smithi042 ceph-mon[32775]: pgmap v8020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:36 smithi164 ceph-mon[36970]: pgmap v8020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:36 smithi163 ceph-mon[36873]: pgmap v8020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14066.441480 secs 2024-07-26T10:50:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14066.441480 secs 2024-07-26T10:50:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14066.441480 secs 2024-07-26T10:50:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:38 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14084.180336 secs 2024-07-26T10:50:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:38 smithi042 ceph-mon[32775]: pgmap v8021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:38 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14084.180336 secs 2024-07-26T10:50:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:38 smithi164 ceph-mon[36970]: pgmap v8021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:38 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14084.180336 secs 2024-07-26T10:50:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:38 smithi163 ceph-mon[36873]: pgmap v8021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:40 smithi164 ceph-mon[36970]: pgmap v8022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:50:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:40 smithi163 ceph-mon[36873]: pgmap v8022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:50:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:40 smithi042 ceph-mon[32775]: pgmap v8022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:50:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:42 smithi164 ceph-mon[36970]: pgmap v8023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14071.441576 secs 2024-07-26T10:50:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:42 smithi163 ceph-mon[36873]: pgmap v8023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14071.441576 secs 2024-07-26T10:50:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:42 smithi042 ceph-mon[32775]: pgmap v8023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14071.441576 secs 2024-07-26T10:50:43.171 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:50:43.169+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:50:43.171 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:50:43.169+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:50:43.171 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:50:43.169+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:50:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:43 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14089.180457 secs 2024-07-26T10:50:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:43 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14089.180457 secs 2024-07-26T10:50:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:43 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14089.180457 secs 2024-07-26T10:50:44.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:44 smithi164 ceph-mon[36970]: pgmap v8024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:44 smithi163 ceph-mon[36873]: pgmap v8024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:44 smithi042 ceph-mon[32775]: pgmap v8024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:46 smithi164 ceph-mon[36970]: pgmap v8025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:46 smithi163 ceph-mon[36873]: pgmap v8025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:46 smithi042 ceph-mon[32775]: pgmap v8025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14076.441723 secs 2024-07-26T10:50:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14076.441723 secs 2024-07-26T10:50:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14076.441723 secs 2024-07-26T10:50:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:48 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14094.180586 secs 2024-07-26T10:50:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:48 smithi164 ceph-mon[36970]: pgmap v8026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:48 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14094.180586 secs 2024-07-26T10:50:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:48 smithi163 ceph-mon[36873]: pgmap v8026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:48 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14094.180586 secs 2024-07-26T10:50:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:48 smithi042 ceph-mon[32775]: pgmap v8026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:50 smithi164 ceph-mon[36970]: pgmap v8027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:50:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:50 smithi163 ceph-mon[36873]: pgmap v8027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:50:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:50 smithi042 ceph-mon[32775]: pgmap v8027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:50:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:52 smithi164 ceph-mon[36970]: pgmap v8028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14081.441849 secs 2024-07-26T10:50:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14099.180707 secs 2024-07-26T10:50:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:52 smithi163 ceph-mon[36873]: pgmap v8028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14081.441849 secs 2024-07-26T10:50:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14099.180707 secs 2024-07-26T10:50:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:52 smithi042 ceph-mon[32775]: pgmap v8028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14081.441849 secs 2024-07-26T10:50:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14099.180707 secs 2024-07-26T10:50:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:54 smithi164 ceph-mon[36970]: pgmap v8029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:54 smithi163 ceph-mon[36873]: pgmap v8029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:54 smithi042 ceph-mon[32775]: pgmap v8029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:56 smithi164 ceph-mon[36970]: pgmap v8030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:56 smithi163 ceph-mon[36873]: pgmap v8030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:56 smithi042 ceph-mon[32775]: pgmap v8030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:50:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14086.441980 secs 2024-07-26T10:50:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14104.180806 secs 2024-07-26T10:50:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14086.441980 secs 2024-07-26T10:50:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14104.180806 secs 2024-07-26T10:50:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14086.441980 secs 2024-07-26T10:50:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14104.180806 secs 2024-07-26T10:50:58.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:50:58.175+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:50:58.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:50:58.175+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:50:58.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:50:58.175+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:50:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:50:58 smithi163 ceph-mon[36873]: pgmap v8031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:50:58 smithi042 ceph-mon[32775]: pgmap v8031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:50:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:50:58 smithi164 ceph-mon[36970]: pgmap v8031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:00 smithi163 ceph-mon[36873]: pgmap v8032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:51:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:00 smithi042 ceph-mon[32775]: pgmap v8032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:51:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:00 smithi164 ceph-mon[36970]: pgmap v8032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:51:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:02 smithi163 ceph-mon[36873]: pgmap v8033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14091.442135 secs 2024-07-26T10:51:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14109.180947 secs 2024-07-26T10:51:02.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:02 smithi164 ceph-mon[36970]: pgmap v8033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:02.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14091.442135 secs 2024-07-26T10:51:02.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14109.180947 secs 2024-07-26T10:51:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:02 smithi042 ceph-mon[32775]: pgmap v8033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14091.442135 secs 2024-07-26T10:51:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14109.180947 secs 2024-07-26T10:51:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:04 smithi163 ceph-mon[36873]: pgmap v8034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:04 smithi042 ceph-mon[32775]: pgmap v8034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:04 smithi164 ceph-mon[36970]: pgmap v8034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:06 smithi163 ceph-mon[36873]: pgmap v8035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:06 smithi042 ceph-mon[32775]: pgmap v8035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:06 smithi164 ceph-mon[36970]: pgmap v8035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14096.442256 secs 2024-07-26T10:51:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14114.181072 secs 2024-07-26T10:51:07.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14096.442256 secs 2024-07-26T10:51:07.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14114.181072 secs 2024-07-26T10:51:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14096.442256 secs 2024-07-26T10:51:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14114.181072 secs 2024-07-26T10:51:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:08 smithi163 ceph-mon[36873]: pgmap v8036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:08 smithi042 ceph-mon[32775]: pgmap v8036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:08 smithi164 ceph-mon[36970]: pgmap v8036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:10 smithi163 ceph-mon[36873]: pgmap v8037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:51:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:10 smithi042 ceph-mon[32775]: pgmap v8037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:51:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:10 smithi164 ceph-mon[36970]: pgmap v8037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:51:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:12 smithi042 ceph-mon[32775]: pgmap v8038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14101.442376 secs 2024-07-26T10:51:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14119.181216 secs 2024-07-26T10:51:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:12 smithi164 ceph-mon[36970]: pgmap v8038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14101.442376 secs 2024-07-26T10:51:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14119.181216 secs 2024-07-26T10:51:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:12 smithi163 ceph-mon[36873]: pgmap v8038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14101.442376 secs 2024-07-26T10:51:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14119.181216 secs 2024-07-26T10:51:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:51:13.177+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:51:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:51:13.177+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:51:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:51:13.177+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:51:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:51:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:51:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:51:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:51:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:51:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:51:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:14 smithi042 ceph-mon[32775]: pgmap v8039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:14 smithi042 ceph-mon[32775]: pgmap v8040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:51:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:14 smithi042 ceph-mon[32775]: pgmap v8041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:51:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:14 smithi164 ceph-mon[36970]: pgmap v8039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:14 smithi164 ceph-mon[36970]: pgmap v8040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:51:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:14 smithi164 ceph-mon[36970]: pgmap v8041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:51:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:14 smithi163 ceph-mon[36873]: pgmap v8039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:14 smithi163 ceph-mon[36873]: pgmap v8040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:51:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:14 smithi163 ceph-mon[36873]: pgmap v8041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:51:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:16 smithi042 ceph-mon[32775]: pgmap v8042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:51:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:16 smithi164 ceph-mon[36970]: pgmap v8042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:51:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:16 smithi163 ceph-mon[36873]: pgmap v8042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:51:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14106.442500 secs 2024-07-26T10:51:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14124.181363 secs 2024-07-26T10:51:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14106.442500 secs 2024-07-26T10:51:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14124.181363 secs 2024-07-26T10:51:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14106.442500 secs 2024-07-26T10:51:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14124.181363 secs 2024-07-26T10:51:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:18 smithi042 ceph-mon[32775]: pgmap v8043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:51:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:18 smithi164 ceph-mon[36970]: pgmap v8043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:51:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:18 smithi163 ceph-mon[36873]: pgmap v8043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:51:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:20 smithi042 ceph-mon[32775]: pgmap v8044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:51:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:20 smithi164 ceph-mon[36970]: pgmap v8044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:51:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:20 smithi163 ceph-mon[36873]: pgmap v8044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:51:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14111.442632 secs 2024-07-26T10:51:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:22 smithi042 ceph-mon[32775]: pgmap v8045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:51:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14129.181486 secs 2024-07-26T10:51:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14111.442632 secs 2024-07-26T10:51:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:22 smithi164 ceph-mon[36970]: pgmap v8045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:51:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14129.181486 secs 2024-07-26T10:51:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14111.442632 secs 2024-07-26T10:51:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:22 smithi163 ceph-mon[36873]: pgmap v8045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:51:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14129.181486 secs 2024-07-26T10:51:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:24 smithi042 ceph-mon[32775]: pgmap v8046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:51:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:24 smithi164 ceph-mon[36970]: pgmap v8046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:51:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:24 smithi163 ceph-mon[36873]: pgmap v8046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:51:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:26 smithi042 ceph-mon[32775]: pgmap v8047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:26 smithi164 ceph-mon[36970]: pgmap v8047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:26 smithi163 ceph-mon[36873]: pgmap v8047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14116.442779 secs 2024-07-26T10:51:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14134.181616 secs 2024-07-26T10:51:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14116.442779 secs 2024-07-26T10:51:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14134.181616 secs 2024-07-26T10:51:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14116.442779 secs 2024-07-26T10:51:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14134.181616 secs 2024-07-26T10:51:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:51:28.182+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:51:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:51:28.182+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:51:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:51:28.182+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:51:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:28 smithi042 ceph-mon[32775]: pgmap v8048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:28 smithi164 ceph-mon[36970]: pgmap v8048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:28 smithi163 ceph-mon[36873]: pgmap v8048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:30 smithi042 ceph-mon[32775]: pgmap v8049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:30 smithi164 ceph-mon[36970]: pgmap v8049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:30 smithi163 ceph-mon[36873]: pgmap v8049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14121.442899 secs 2024-07-26T10:51:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:32 smithi042 ceph-mon[32775]: pgmap v8050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14139.181737 secs 2024-07-26T10:51:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14121.442899 secs 2024-07-26T10:51:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:32 smithi164 ceph-mon[36970]: pgmap v8050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14139.181737 secs 2024-07-26T10:51:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14121.442899 secs 2024-07-26T10:51:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:32 smithi163 ceph-mon[36873]: pgmap v8050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14139.181737 secs 2024-07-26T10:51:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:34 smithi042 ceph-mon[32775]: pgmap v8051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:34 smithi164 ceph-mon[36970]: pgmap v8051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:34 smithi163 ceph-mon[36873]: pgmap v8051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:36 smithi042 ceph-mon[32775]: pgmap v8052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:36.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:36 smithi164 ceph-mon[36970]: pgmap v8052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:36 smithi163 ceph-mon[36873]: pgmap v8052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14126.443041 secs 2024-07-26T10:51:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14144.181879 secs 2024-07-26T10:51:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14126.443041 secs 2024-07-26T10:51:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14144.181879 secs 2024-07-26T10:51:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14126.443041 secs 2024-07-26T10:51:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14144.181879 secs 2024-07-26T10:51:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:38 smithi042 ceph-mon[32775]: pgmap v8053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:38 smithi164 ceph-mon[36970]: pgmap v8053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:38 smithi163 ceph-mon[36873]: pgmap v8053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:40 smithi042 ceph-mon[32775]: pgmap v8054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:40 smithi164 ceph-mon[36970]: pgmap v8054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:40 smithi163 ceph-mon[36873]: pgmap v8054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14131.443161 secs 2024-07-26T10:51:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:42 smithi042 ceph-mon[32775]: pgmap v8055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14149.181993 secs 2024-07-26T10:51:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14131.443161 secs 2024-07-26T10:51:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:42 smithi164 ceph-mon[36970]: pgmap v8055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14149.181993 secs 2024-07-26T10:51:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14131.443161 secs 2024-07-26T10:51:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:42 smithi163 ceph-mon[36873]: pgmap v8055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14149.181993 secs 2024-07-26T10:51:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:51:43.183+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:51:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:51:43.183+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:51:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:51:43.183+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:51:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:44 smithi042 ceph-mon[32775]: pgmap v8056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:44 smithi164 ceph-mon[36970]: pgmap v8056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:44 smithi163 ceph-mon[36873]: pgmap v8056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:46 smithi042 ceph-mon[32775]: pgmap v8057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:46.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:46 smithi164 ceph-mon[36970]: pgmap v8057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:46 smithi163 ceph-mon[36873]: pgmap v8057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14136.443310 secs 2024-07-26T10:51:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14154.182110 secs 2024-07-26T10:51:47.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14136.443310 secs 2024-07-26T10:51:47.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14154.182110 secs 2024-07-26T10:51:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14136.443310 secs 2024-07-26T10:51:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14154.182110 secs 2024-07-26T10:51:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:48 smithi042 ceph-mon[32775]: pgmap v8058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:48 smithi164 ceph-mon[36970]: pgmap v8058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:48 smithi163 ceph-mon[36873]: pgmap v8058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:50 smithi042 ceph-mon[32775]: pgmap v8059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:50 smithi164 ceph-mon[36970]: pgmap v8059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:50 smithi163 ceph-mon[36873]: pgmap v8059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14141.443430 secs 2024-07-26T10:51:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:52 smithi042 ceph-mon[32775]: pgmap v8060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14159.182268 secs 2024-07-26T10:51:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14141.443430 secs 2024-07-26T10:51:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:52 smithi164 ceph-mon[36970]: pgmap v8060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:52.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14159.182268 secs 2024-07-26T10:51:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14141.443430 secs 2024-07-26T10:51:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:52 smithi163 ceph-mon[36873]: pgmap v8060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14159.182268 secs 2024-07-26T10:51:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:54 smithi042 ceph-mon[32775]: pgmap v8061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:54 smithi164 ceph-mon[36970]: pgmap v8061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:54 smithi163 ceph-mon[36873]: pgmap v8061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:56 smithi042 ceph-mon[32775]: pgmap v8062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:56 smithi164 ceph-mon[36970]: pgmap v8062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:56 smithi163 ceph-mon[36873]: pgmap v8062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:51:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14146.443506 secs 2024-07-26T10:51:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14164.182349 secs 2024-07-26T10:51:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14146.443506 secs 2024-07-26T10:51:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14164.182349 secs 2024-07-26T10:51:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14146.443506 secs 2024-07-26T10:51:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14164.182349 secs 2024-07-26T10:51:58.189 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:51:58.188+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:51:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:51:58.188+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:51:58.190 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:51:58.188+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:51:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:51:58 smithi042 ceph-mon[32775]: pgmap v8063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:51:58 smithi164 ceph-mon[36970]: pgmap v8063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:51:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:51:58 smithi163 ceph-mon[36873]: pgmap v8063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:00 smithi042 ceph-mon[32775]: pgmap v8064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:00 smithi164 ceph-mon[36970]: pgmap v8064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:00 smithi163 ceph-mon[36873]: pgmap v8064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14151.443633 secs 2024-07-26T10:52:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:02 smithi042 ceph-mon[32775]: pgmap v8065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14169.182480 secs 2024-07-26T10:52:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14151.443633 secs 2024-07-26T10:52:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:02 smithi164 ceph-mon[36970]: pgmap v8065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:02.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14169.182480 secs 2024-07-26T10:52:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14151.443633 secs 2024-07-26T10:52:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:02 smithi163 ceph-mon[36873]: pgmap v8065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14169.182480 secs 2024-07-26T10:52:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:04 smithi042 ceph-mon[32775]: pgmap v8066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:04 smithi164 ceph-mon[36970]: pgmap v8066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:04 smithi163 ceph-mon[36873]: pgmap v8066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:06 smithi042 ceph-mon[32775]: pgmap v8067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:06 smithi164 ceph-mon[36970]: pgmap v8067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:06 smithi163 ceph-mon[36873]: pgmap v8067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14156.443770 secs 2024-07-26T10:52:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14174.182600 secs 2024-07-26T10:52:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14156.443770 secs 2024-07-26T10:52:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14174.182600 secs 2024-07-26T10:52:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14156.443770 secs 2024-07-26T10:52:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14174.182600 secs 2024-07-26T10:52:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:08 smithi042 ceph-mon[32775]: pgmap v8068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:08 smithi164 ceph-mon[36970]: pgmap v8068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:08 smithi163 ceph-mon[36873]: pgmap v8068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:10 smithi042 ceph-mon[32775]: pgmap v8069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:10 smithi164 ceph-mon[36970]: pgmap v8069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:10 smithi163 ceph-mon[36873]: pgmap v8069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14161.443911 secs 2024-07-26T10:52:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:12 smithi042 ceph-mon[32775]: pgmap v8070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14179.182665 secs 2024-07-26T10:52:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14161.443911 secs 2024-07-26T10:52:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:12 smithi164 ceph-mon[36970]: pgmap v8070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14179.182665 secs 2024-07-26T10:52:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14161.443911 secs 2024-07-26T10:52:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:12 smithi163 ceph-mon[36873]: pgmap v8070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14179.182665 secs 2024-07-26T10:52:13.191 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:52:13.189+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:52:13.191 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:52:13.189+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:52:13.191 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:52:13.189+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:52:13.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:13 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:13 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:13 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:52:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:14 smithi164 ceph-mon[36970]: pgmap v8071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:14.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:14.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:14.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:52:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:14 smithi163 ceph-mon[36873]: pgmap v8071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:52:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:14 smithi042 ceph-mon[32775]: pgmap v8071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:52:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:52:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:52:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:15 smithi164 ceph-mon[36970]: pgmap v8072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:52:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:15 smithi164 ceph-mon[36970]: pgmap v8073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:52:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:15 smithi163 ceph-mon[36873]: pgmap v8072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:52:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:15 smithi163 ceph-mon[36873]: pgmap v8073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:52:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:15 smithi042 ceph-mon[32775]: pgmap v8072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:52:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:15 smithi042 ceph-mon[32775]: pgmap v8073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:52:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:17 smithi164 ceph-mon[36970]: pgmap v8074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:52:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14166.444036 secs 2024-07-26T10:52:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14184.182740 secs 2024-07-26T10:52:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:17 smithi163 ceph-mon[36873]: pgmap v8074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:52:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14166.444036 secs 2024-07-26T10:52:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14184.182740 secs 2024-07-26T10:52:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:17 smithi042 ceph-mon[32775]: pgmap v8074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:52:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14166.444036 secs 2024-07-26T10:52:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14184.182740 secs 2024-07-26T10:52:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:19 smithi164 ceph-mon[36970]: pgmap v8075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:52:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:19 smithi163 ceph-mon[36873]: pgmap v8075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:52:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:19 smithi042 ceph-mon[32775]: pgmap v8075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:52:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:21 smithi164 ceph-mon[36970]: pgmap v8076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:52:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:21 smithi163 ceph-mon[36873]: pgmap v8076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:52:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:21 smithi042 ceph-mon[32775]: pgmap v8076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:52:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14171.444153 secs 2024-07-26T10:52:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14189.182860 secs 2024-07-26T10:52:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14171.444153 secs 2024-07-26T10:52:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14189.182860 secs 2024-07-26T10:52:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14171.444153 secs 2024-07-26T10:52:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14189.182860 secs 2024-07-26T10:52:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:23 smithi164 ceph-mon[36970]: pgmap v8077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:52:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:23 smithi163 ceph-mon[36873]: pgmap v8077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:52:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:23 smithi042 ceph-mon[32775]: pgmap v8077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:52:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:25 smithi164 ceph-mon[36970]: pgmap v8078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:52:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:25 smithi163 ceph-mon[36873]: pgmap v8078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:52:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:25 smithi042 ceph-mon[32775]: pgmap v8078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:52:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:27 smithi164 ceph-mon[36970]: pgmap v8079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14176.444247 secs 2024-07-26T10:52:27.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14194.182978 secs 2024-07-26T10:52:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:27 smithi163 ceph-mon[36873]: pgmap v8079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14176.444247 secs 2024-07-26T10:52:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14194.182978 secs 2024-07-26T10:52:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:27 smithi042 ceph-mon[32775]: pgmap v8079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14176.444247 secs 2024-07-26T10:52:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14194.182978 secs 2024-07-26T10:52:28.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:52:28.194+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:52:28.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:52:28.194+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:52:28.196 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:52:28.194+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:52:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:29 smithi164 ceph-mon[36970]: pgmap v8080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:29 smithi163 ceph-mon[36873]: pgmap v8080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:29 smithi042 ceph-mon[32775]: pgmap v8080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:31 smithi164 ceph-mon[36970]: pgmap v8081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:31 smithi163 ceph-mon[36873]: pgmap v8081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:31 smithi042 ceph-mon[32775]: pgmap v8081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14181.444360 secs 2024-07-26T10:52:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14199.183043 secs 2024-07-26T10:52:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14181.444360 secs 2024-07-26T10:52:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14199.183043 secs 2024-07-26T10:52:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14181.444360 secs 2024-07-26T10:52:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14199.183043 secs 2024-07-26T10:52:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:33 smithi164 ceph-mon[36970]: pgmap v8082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:33 smithi163 ceph-mon[36873]: pgmap v8082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:33 smithi042 ceph-mon[32775]: pgmap v8082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:35 smithi164 ceph-mon[36970]: pgmap v8083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:52:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:35 smithi163 ceph-mon[36873]: pgmap v8083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:52:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:35 smithi042 ceph-mon[32775]: pgmap v8083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:52:37.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:37 smithi042 ceph-mon[32775]: pgmap v8084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:37.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14186.444440 secs 2024-07-26T10:52:37.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14204.183164 secs 2024-07-26T10:52:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:37 smithi164 ceph-mon[36970]: pgmap v8084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14186.444440 secs 2024-07-26T10:52:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14204.183164 secs 2024-07-26T10:52:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:37 smithi163 ceph-mon[36873]: pgmap v8084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14186.444440 secs 2024-07-26T10:52:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14204.183164 secs 2024-07-26T10:52:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:39 smithi164 ceph-mon[36970]: pgmap v8085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:39 smithi163 ceph-mon[36873]: pgmap v8085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:39 smithi042 ceph-mon[32775]: pgmap v8085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:41 smithi164 ceph-mon[36970]: pgmap v8086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:41 smithi163 ceph-mon[36873]: pgmap v8086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:41 smithi042 ceph-mon[32775]: pgmap v8086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14191.444562 secs 2024-07-26T10:52:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14209.183284 secs 2024-07-26T10:52:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14191.444562 secs 2024-07-26T10:52:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14209.183284 secs 2024-07-26T10:52:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14191.444562 secs 2024-07-26T10:52:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14209.183284 secs 2024-07-26T10:52:43.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:52:43.195+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:52:43.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:52:43.195+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:52:43.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:52:43.195+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:52:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:43 smithi164 ceph-mon[36970]: pgmap v8087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:43 smithi163 ceph-mon[36873]: pgmap v8087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:43 smithi042 ceph-mon[32775]: pgmap v8087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:45 smithi164 ceph-mon[36970]: pgmap v8088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:52:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:45 smithi163 ceph-mon[36873]: pgmap v8088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:52:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:45 smithi042 ceph-mon[32775]: pgmap v8088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:52:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:47 smithi164 ceph-mon[36970]: pgmap v8089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14196.444692 secs 2024-07-26T10:52:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14214.183412 secs 2024-07-26T10:52:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:47 smithi163 ceph-mon[36873]: pgmap v8089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14196.444692 secs 2024-07-26T10:52:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14214.183412 secs 2024-07-26T10:52:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:47 smithi042 ceph-mon[32775]: pgmap v8089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14196.444692 secs 2024-07-26T10:52:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14214.183412 secs 2024-07-26T10:52:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:49 smithi164 ceph-mon[36970]: pgmap v8090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:49 smithi163 ceph-mon[36873]: pgmap v8090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:49 smithi042 ceph-mon[32775]: pgmap v8090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:51 smithi163 ceph-mon[36873]: pgmap v8091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:51 smithi042 ceph-mon[32775]: pgmap v8091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:51 smithi164 ceph-mon[36970]: pgmap v8091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14201.444817 secs 2024-07-26T10:52:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14219.183544 secs 2024-07-26T10:52:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14201.444817 secs 2024-07-26T10:52:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14219.183544 secs 2024-07-26T10:52:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14201.444817 secs 2024-07-26T10:52:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14219.183544 secs 2024-07-26T10:52:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:53 smithi163 ceph-mon[36873]: pgmap v8092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:53 smithi042 ceph-mon[32775]: pgmap v8092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:53 smithi164 ceph-mon[36970]: pgmap v8092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:55 smithi163 ceph-mon[36873]: pgmap v8093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:52:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:55 smithi042 ceph-mon[32775]: pgmap v8093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:52:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:55 smithi164 ceph-mon[36970]: pgmap v8093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:52:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:57 smithi163 ceph-mon[36873]: pgmap v8094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14206.444942 secs 2024-07-26T10:52:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14224.183673 secs 2024-07-26T10:52:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:57 smithi042 ceph-mon[32775]: pgmap v8094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14206.444942 secs 2024-07-26T10:52:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14224.183673 secs 2024-07-26T10:52:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:57 smithi164 ceph-mon[36970]: pgmap v8094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:52:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14206.444942 secs 2024-07-26T10:52:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14224.183673 secs 2024-07-26T10:52:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:52:58.198+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:52:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:52:58.198+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:52:58.200 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:52:58.198+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:52:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:52:59 smithi163 ceph-mon[36873]: pgmap v8095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:52:59 smithi042 ceph-mon[32775]: pgmap v8095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:52:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:52:59 smithi164 ceph-mon[36970]: pgmap v8095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:00.096 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:53:00.094+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:53:00.096 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:53:00.094+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:53:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:53:00.095+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:53:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:53:00.095+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:53:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:01 smithi163 ceph-mon[36873]: pgmap v8096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:01 smithi042 ceph-mon[32775]: pgmap v8096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:01 smithi164 ceph-mon[36970]: pgmap v8096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14211.445085 secs 2024-07-26T10:53:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14229.183798 secs 2024-07-26T10:53:02.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14211.445085 secs 2024-07-26T10:53:02.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14229.183798 secs 2024-07-26T10:53:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14211.445085 secs 2024-07-26T10:53:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14229.183798 secs 2024-07-26T10:53:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:03 smithi163 ceph-mon[36873]: pgmap v8097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:03 smithi042 ceph-mon[32775]: pgmap v8097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:03 smithi164 ceph-mon[36970]: pgmap v8097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:05 smithi163 ceph-mon[36873]: pgmap v8098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:53:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:05 smithi042 ceph-mon[32775]: pgmap v8098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:53:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:05 smithi164 ceph-mon[36970]: pgmap v8098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:53:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:07 smithi042 ceph-mon[32775]: pgmap v8099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14216.445234 secs 2024-07-26T10:53:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14234.183916 secs 2024-07-26T10:53:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:07 smithi164 ceph-mon[36970]: pgmap v8099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14216.445234 secs 2024-07-26T10:53:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14234.183916 secs 2024-07-26T10:53:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:07 smithi163 ceph-mon[36873]: pgmap v8099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14216.445234 secs 2024-07-26T10:53:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14234.183916 secs 2024-07-26T10:53:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:09 smithi042 ceph-mon[32775]: pgmap v8100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:09 smithi164 ceph-mon[36970]: pgmap v8100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:09 smithi163 ceph-mon[36873]: pgmap v8100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:11 smithi042 ceph-mon[32775]: pgmap v8101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:11 smithi164 ceph-mon[36970]: pgmap v8101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:11 smithi163 ceph-mon[36873]: pgmap v8101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:12.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14221.445355 secs 2024-07-26T10:53:12.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14239.184035 secs 2024-07-26T10:53:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14221.445355 secs 2024-07-26T10:53:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14239.184035 secs 2024-07-26T10:53:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14221.445355 secs 2024-07-26T10:53:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14239.184035 secs 2024-07-26T10:53:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:53:13.199+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:53:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:53:13.199+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:53:13.201 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:53:13.199+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:53:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:13 smithi042 ceph-mon[32775]: pgmap v8102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:13 smithi164 ceph-mon[36970]: pgmap v8102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:13 smithi163 ceph-mon[36873]: pgmap v8102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:14.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:53:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:15 smithi042 ceph-mon[32775]: pgmap v8103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:53:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:15 smithi164 ceph-mon[36970]: pgmap v8103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:53:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:15 smithi163 ceph-mon[36873]: pgmap v8103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:53:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:16.666 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:16.667 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:16.667 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:16 smithi042 ceph-mon[32775]: pgmap v8104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:16.667 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:16.667 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:16.667 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:16.667 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:16.667 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:16 smithi042 ceph-mon[32775]: pgmap v8105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:53:16.667 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:16 smithi042 ceph-mon[32775]: pgmap v8106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:53:16.667 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:16 smithi164 ceph-mon[36970]: pgmap v8104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:16 smithi164 ceph-mon[36970]: pgmap v8105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:53:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:16 smithi164 ceph-mon[36970]: pgmap v8106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:53:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:16 smithi163 ceph-mon[36873]: pgmap v8104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:53:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:53:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:16 smithi163 ceph-mon[36873]: pgmap v8105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:53:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:16 smithi163 ceph-mon[36873]: pgmap v8106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:53:17.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:53:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14226.445453 secs 2024-07-26T10:53:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14244.184205 secs 2024-07-26T10:53:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14226.445453 secs 2024-07-26T10:53:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14244.184205 secs 2024-07-26T10:53:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14226.445453 secs 2024-07-26T10:53:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14244.184205 secs 2024-07-26T10:53:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:18 smithi042 ceph-mon[32775]: pgmap v8107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:53:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:18 smithi164 ceph-mon[36970]: pgmap v8107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:53:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:18 smithi163 ceph-mon[36873]: pgmap v8107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:53:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:21 smithi163 ceph-mon[36873]: pgmap v8108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:53:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:21 smithi042 ceph-mon[32775]: pgmap v8108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:53:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:21 smithi164 ceph-mon[36970]: pgmap v8108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:53:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14231.445580 secs 2024-07-26T10:53:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14249.184291 secs 2024-07-26T10:53:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14231.445580 secs 2024-07-26T10:53:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14249.184291 secs 2024-07-26T10:53:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14231.445580 secs 2024-07-26T10:53:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14249.184291 secs 2024-07-26T10:53:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:23 smithi163 ceph-mon[36873]: pgmap v8109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T10:53:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:23 smithi042 ceph-mon[32775]: pgmap v8109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T10:53:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:23 smithi164 ceph-mon[36970]: pgmap v8109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T10:53:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:25 smithi163 ceph-mon[36873]: pgmap v8110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:53:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:25 smithi042 ceph-mon[32775]: pgmap v8110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:53:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:25 smithi164 ceph-mon[36970]: pgmap v8110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T10:53:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14236.445665 secs 2024-07-26T10:53:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:27 smithi042 ceph-mon[32775]: pgmap v8111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:53:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14254.184414 secs 2024-07-26T10:53:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14236.445665 secs 2024-07-26T10:53:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:27 smithi164 ceph-mon[36970]: pgmap v8111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:53:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14254.184414 secs 2024-07-26T10:53:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14236.445665 secs 2024-07-26T10:53:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:27 smithi163 ceph-mon[36873]: pgmap v8111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:53:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14254.184414 secs 2024-07-26T10:53:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:53:28.205+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:53:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:53:28.205+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:53:28.207 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:53:28.205+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:53:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:29 smithi042 ceph-mon[32775]: pgmap v8112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:29 smithi164 ceph-mon[36970]: pgmap v8112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:29 smithi163 ceph-mon[36873]: pgmap v8112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:31 smithi042 ceph-mon[32775]: pgmap v8113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:31 smithi164 ceph-mon[36970]: pgmap v8113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:31 smithi163 ceph-mon[36873]: pgmap v8113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:32.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14241.445788 secs 2024-07-26T10:53:32.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14259.184560 secs 2024-07-26T10:53:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14241.445788 secs 2024-07-26T10:53:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14259.184560 secs 2024-07-26T10:53:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14241.445788 secs 2024-07-26T10:53:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14259.184560 secs 2024-07-26T10:53:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:33 smithi042 ceph-mon[32775]: pgmap v8114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:33 smithi164 ceph-mon[36970]: pgmap v8114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:33 smithi163 ceph-mon[36873]: pgmap v8114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:35 smithi042 ceph-mon[32775]: pgmap v8115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:35 smithi164 ceph-mon[36970]: pgmap v8115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:35 smithi163 ceph-mon[36873]: pgmap v8115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14246.445907 secs 2024-07-26T10:53:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:37 smithi042 ceph-mon[32775]: pgmap v8116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14264.184698 secs 2024-07-26T10:53:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14246.445907 secs 2024-07-26T10:53:37.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:37 smithi164 ceph-mon[36970]: pgmap v8116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:37.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14264.184698 secs 2024-07-26T10:53:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14246.445907 secs 2024-07-26T10:53:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:37 smithi163 ceph-mon[36873]: pgmap v8116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14264.184698 secs 2024-07-26T10:53:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:39 smithi042 ceph-mon[32775]: pgmap v8117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:39 smithi164 ceph-mon[36970]: pgmap v8117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:39 smithi163 ceph-mon[36873]: pgmap v8117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:41 smithi042 ceph-mon[32775]: pgmap v8118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:41 smithi164 ceph-mon[36970]: pgmap v8118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:41 smithi163 ceph-mon[36873]: pgmap v8118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14251.446037 secs 2024-07-26T10:53:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14269.184815 secs 2024-07-26T10:53:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14251.446037 secs 2024-07-26T10:53:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14269.184815 secs 2024-07-26T10:53:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14251.446037 secs 2024-07-26T10:53:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14269.184815 secs 2024-07-26T10:53:43.207 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:53:43.206+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:53:43.207 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:53:43.206+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:53:43.208 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:53:43.206+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:53:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:43 smithi042 ceph-mon[32775]: pgmap v8119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:43 smithi164 ceph-mon[36970]: pgmap v8119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:43 smithi163 ceph-mon[36873]: pgmap v8119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:45 smithi042 ceph-mon[32775]: pgmap v8120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:45 smithi164 ceph-mon[36970]: pgmap v8120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:45 smithi163 ceph-mon[36873]: pgmap v8120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:47.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14256.446181 secs 2024-07-26T10:53:47.561 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:47 smithi163 ceph-mon[36873]: pgmap v8121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:47.561 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14274.184947 secs 2024-07-26T10:53:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14256.446181 secs 2024-07-26T10:53:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:47 smithi042 ceph-mon[32775]: pgmap v8121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14274.184947 secs 2024-07-26T10:53:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14256.446181 secs 2024-07-26T10:53:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:47 smithi164 ceph-mon[36970]: pgmap v8121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:47.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14274.184947 secs 2024-07-26T10:53:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:49 smithi042 ceph-mon[32775]: pgmap v8122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:49 smithi164 ceph-mon[36970]: pgmap v8122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:49 smithi163 ceph-mon[36873]: pgmap v8122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:51 smithi042 ceph-mon[32775]: pgmap v8123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:51 smithi164 ceph-mon[36970]: pgmap v8123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:51 smithi163 ceph-mon[36873]: pgmap v8123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14261.446303 secs 2024-07-26T10:53:52.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14279.185048 secs 2024-07-26T10:53:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14261.446303 secs 2024-07-26T10:53:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14279.185048 secs 2024-07-26T10:53:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14261.446303 secs 2024-07-26T10:53:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14279.185048 secs 2024-07-26T10:53:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:53 smithi042 ceph-mon[32775]: pgmap v8124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:53 smithi164 ceph-mon[36970]: pgmap v8124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:53 smithi163 ceph-mon[36873]: pgmap v8124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:55 smithi042 ceph-mon[32775]: pgmap v8125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:55 smithi164 ceph-mon[36970]: pgmap v8125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:55 smithi163 ceph-mon[36873]: pgmap v8125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:53:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14266.446424 secs 2024-07-26T10:53:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:57 smithi042 ceph-mon[32775]: pgmap v8126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14284.185142 secs 2024-07-26T10:53:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14266.446424 secs 2024-07-26T10:53:57.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:57 smithi164 ceph-mon[36970]: pgmap v8126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:57.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14284.185142 secs 2024-07-26T10:53:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14266.446424 secs 2024-07-26T10:53:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:57 smithi163 ceph-mon[36873]: pgmap v8126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14284.185142 secs 2024-07-26T10:53:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:53:58.211+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:53:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:53:58.211+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:53:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:53:58.211+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:53:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:53:59 smithi042 ceph-mon[32775]: pgmap v8127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:53:59 smithi164 ceph-mon[36970]: pgmap v8127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:53:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:53:59 smithi163 ceph-mon[36873]: pgmap v8127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:01 smithi042 ceph-mon[32775]: pgmap v8128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:01 smithi164 ceph-mon[36970]: pgmap v8128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:01 smithi163 ceph-mon[36873]: pgmap v8128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:02.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14271.446545 secs 2024-07-26T10:54:02.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14289.185241 secs 2024-07-26T10:54:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14271.446545 secs 2024-07-26T10:54:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14289.185241 secs 2024-07-26T10:54:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14271.446545 secs 2024-07-26T10:54:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14289.185241 secs 2024-07-26T10:54:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:03 smithi042 ceph-mon[32775]: pgmap v8129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:03 smithi164 ceph-mon[36970]: pgmap v8129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:03 smithi163 ceph-mon[36873]: pgmap v8129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:05 smithi042 ceph-mon[32775]: pgmap v8130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:05 smithi164 ceph-mon[36970]: pgmap v8130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:05 smithi163 ceph-mon[36873]: pgmap v8130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14276.446687 secs 2024-07-26T10:54:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:07 smithi042 ceph-mon[32775]: pgmap v8131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14294.185361 secs 2024-07-26T10:54:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14276.446687 secs 2024-07-26T10:54:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:07 smithi164 ceph-mon[36970]: pgmap v8131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14294.185361 secs 2024-07-26T10:54:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14276.446687 secs 2024-07-26T10:54:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:07 smithi163 ceph-mon[36873]: pgmap v8131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14294.185361 secs 2024-07-26T10:54:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:09 smithi042 ceph-mon[32775]: pgmap v8132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:09 smithi164 ceph-mon[36970]: pgmap v8132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:09 smithi163 ceph-mon[36873]: pgmap v8132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:11 smithi042 ceph-mon[32775]: pgmap v8133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:11 smithi164 ceph-mon[36970]: pgmap v8133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:11 smithi163 ceph-mon[36873]: pgmap v8133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14281.446811 secs 2024-07-26T10:54:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14299.185441 secs 2024-07-26T10:54:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14281.446811 secs 2024-07-26T10:54:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14299.185441 secs 2024-07-26T10:54:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14281.446811 secs 2024-07-26T10:54:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14299.185441 secs 2024-07-26T10:54:13.214 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:54:13.212+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:54:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:54:13.212+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:54:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:54:13.212+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:54:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:13 smithi042 ceph-mon[32775]: pgmap v8134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:13 smithi164 ceph-mon[36970]: pgmap v8134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:13 smithi163 ceph-mon[36873]: pgmap v8134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:15 smithi042 ceph-mon[32775]: pgmap v8135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:15 smithi164 ceph-mon[36970]: pgmap v8135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:15 smithi163 ceph-mon[36873]: pgmap v8135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:54:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14286.446937 secs 2024-07-26T10:54:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:17 smithi042 ceph-mon[32775]: pgmap v8136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14304.185563 secs 2024-07-26T10:54:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:17.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:17.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14286.446937 secs 2024-07-26T10:54:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:17 smithi163 ceph-mon[36873]: pgmap v8136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14304.185563 secs 2024-07-26T10:54:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:17.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:17.660 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14286.446937 secs 2024-07-26T10:54:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:17 smithi164 ceph-mon[36970]: pgmap v8136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14304.185563 secs 2024-07-26T10:54:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:54:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:54:17.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:54:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:18 smithi042 ceph-mon[32775]: pgmap v8137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:54:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:18 smithi042 ceph-mon[32775]: pgmap v8138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:54:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:18 smithi164 ceph-mon[36970]: pgmap v8137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:54:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:18 smithi164 ceph-mon[36970]: pgmap v8138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:54:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:18 smithi163 ceph-mon[36873]: pgmap v8137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T10:54:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:18 smithi163 ceph-mon[36873]: pgmap v8138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:54:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:20 smithi042 ceph-mon[32775]: pgmap v8139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:54:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:20 smithi164 ceph-mon[36970]: pgmap v8139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:54:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:20 smithi163 ceph-mon[36873]: pgmap v8139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:54:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14291.447056 secs 2024-07-26T10:54:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:22 smithi042 ceph-mon[32775]: pgmap v8140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:54:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14309.185683 secs 2024-07-26T10:54:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14291.447056 secs 2024-07-26T10:54:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:22 smithi164 ceph-mon[36970]: pgmap v8140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:54:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14309.185683 secs 2024-07-26T10:54:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14291.447056 secs 2024-07-26T10:54:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:22 smithi163 ceph-mon[36873]: pgmap v8140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:54:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14309.185683 secs 2024-07-26T10:54:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:24 smithi042 ceph-mon[32775]: pgmap v8141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T10:54:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:24 smithi164 ceph-mon[36970]: pgmap v8141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T10:54:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:24 smithi163 ceph-mon[36873]: pgmap v8141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-07-26T10:54:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:26 smithi042 ceph-mon[32775]: pgmap v8142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:54:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:26 smithi164 ceph-mon[36970]: pgmap v8142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:54:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:26 smithi163 ceph-mon[36873]: pgmap v8142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:54:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14296.447180 secs 2024-07-26T10:54:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14314.185778 secs 2024-07-26T10:54:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14296.447180 secs 2024-07-26T10:54:27.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14314.185778 secs 2024-07-26T10:54:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14296.447180 secs 2024-07-26T10:54:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14314.185778 secs 2024-07-26T10:54:28.220 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:54:28.218+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:54:28.220 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:54:28.218+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:54:28.220 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:54:28.218+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:54:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:28 smithi042 ceph-mon[32775]: pgmap v8143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:54:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:28 smithi164 ceph-mon[36970]: pgmap v8143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:54:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:28 smithi163 ceph-mon[36873]: pgmap v8143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:54:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:30 smithi164 ceph-mon[36970]: pgmap v8144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:30 smithi163 ceph-mon[36873]: pgmap v8144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:30 smithi042 ceph-mon[32775]: pgmap v8144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14301.447304 secs 2024-07-26T10:54:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:32 smithi164 ceph-mon[36970]: pgmap v8145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14319.185920 secs 2024-07-26T10:54:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14301.447304 secs 2024-07-26T10:54:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:32 smithi163 ceph-mon[36873]: pgmap v8145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14319.185920 secs 2024-07-26T10:54:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14301.447304 secs 2024-07-26T10:54:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:32 smithi042 ceph-mon[32775]: pgmap v8145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14319.185920 secs 2024-07-26T10:54:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:34 smithi164 ceph-mon[36970]: pgmap v8146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:34 smithi163 ceph-mon[36873]: pgmap v8146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:34 smithi042 ceph-mon[32775]: pgmap v8146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:36 smithi164 ceph-mon[36970]: pgmap v8147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:36 smithi163 ceph-mon[36873]: pgmap v8147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:36 smithi042 ceph-mon[32775]: pgmap v8147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14306.447387 secs 2024-07-26T10:54:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14324.186059 secs 2024-07-26T10:54:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14306.447387 secs 2024-07-26T10:54:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14324.186059 secs 2024-07-26T10:54:37.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14306.447387 secs 2024-07-26T10:54:37.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14324.186059 secs 2024-07-26T10:54:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:38 smithi164 ceph-mon[36970]: pgmap v8148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:38 smithi163 ceph-mon[36873]: pgmap v8148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:38 smithi042 ceph-mon[32775]: pgmap v8148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:40 smithi164 ceph-mon[36970]: pgmap v8149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:40 smithi163 ceph-mon[36873]: pgmap v8149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:40 smithi042 ceph-mon[32775]: pgmap v8149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14311.447501 secs 2024-07-26T10:54:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:42 smithi164 ceph-mon[36970]: pgmap v8150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14329.186212 secs 2024-07-26T10:54:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14311.447501 secs 2024-07-26T10:54:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:42 smithi163 ceph-mon[36873]: pgmap v8150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14329.186212 secs 2024-07-26T10:54:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14311.447501 secs 2024-07-26T10:54:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:42 smithi042 ceph-mon[32775]: pgmap v8150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14329.186212 secs 2024-07-26T10:54:43.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:54:43.219+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:54:43.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:54:43.219+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:54:43.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:54:43.219+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:54:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:44 smithi164 ceph-mon[36970]: pgmap v8151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:44 smithi163 ceph-mon[36873]: pgmap v8151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:44 smithi042 ceph-mon[32775]: pgmap v8151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:46 smithi164 ceph-mon[36970]: pgmap v8152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:46 smithi163 ceph-mon[36873]: pgmap v8152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:46 smithi042 ceph-mon[32775]: pgmap v8152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14316.447644 secs 2024-07-26T10:54:47.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14334.186334 secs 2024-07-26T10:54:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14316.447644 secs 2024-07-26T10:54:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14334.186334 secs 2024-07-26T10:54:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14316.447644 secs 2024-07-26T10:54:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14334.186334 secs 2024-07-26T10:54:48.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:48 smithi164 ceph-mon[36970]: pgmap v8153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:48 smithi163 ceph-mon[36873]: pgmap v8153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:48.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:48 smithi042 ceph-mon[32775]: pgmap v8153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:50 smithi164 ceph-mon[36970]: pgmap v8154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:50 smithi163 ceph-mon[36873]: pgmap v8154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:50 smithi042 ceph-mon[32775]: pgmap v8154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14321.447798 secs 2024-07-26T10:54:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:52 smithi164 ceph-mon[36970]: pgmap v8155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14339.186455 secs 2024-07-26T10:54:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14321.447798 secs 2024-07-26T10:54:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:52 smithi163 ceph-mon[36873]: pgmap v8155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14339.186455 secs 2024-07-26T10:54:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14321.447798 secs 2024-07-26T10:54:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:52 smithi042 ceph-mon[32775]: pgmap v8155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14339.186455 secs 2024-07-26T10:54:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:54 smithi164 ceph-mon[36970]: pgmap v8156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:54 smithi163 ceph-mon[36873]: pgmap v8156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:54 smithi042 ceph-mon[32775]: pgmap v8156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:56 smithi164 ceph-mon[36970]: pgmap v8157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:56 smithi163 ceph-mon[36873]: pgmap v8157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:56 smithi042 ceph-mon[32775]: pgmap v8157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:54:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14326.447932 secs 2024-07-26T10:54:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14344.186583 secs 2024-07-26T10:54:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14326.447932 secs 2024-07-26T10:54:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14344.186583 secs 2024-07-26T10:54:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14326.447932 secs 2024-07-26T10:54:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14344.186583 secs 2024-07-26T10:54:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:54:58.224+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:54:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:54:58.224+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:54:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:54:58.224+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:54:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:54:58 smithi164 ceph-mon[36970]: pgmap v8158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:54:58 smithi163 ceph-mon[36873]: pgmap v8158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:54:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:54:58 smithi042 ceph-mon[32775]: pgmap v8158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:00 smithi164 ceph-mon[36970]: pgmap v8159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:00 smithi163 ceph-mon[36873]: pgmap v8159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:00 smithi042 ceph-mon[32775]: pgmap v8159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14331.448068 secs 2024-07-26T10:55:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:02 smithi164 ceph-mon[36970]: pgmap v8160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14349.186705 secs 2024-07-26T10:55:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14331.448068 secs 2024-07-26T10:55:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:02 smithi163 ceph-mon[36873]: pgmap v8160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14349.186705 secs 2024-07-26T10:55:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14331.448068 secs 2024-07-26T10:55:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:02 smithi042 ceph-mon[32775]: pgmap v8160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14349.186705 secs 2024-07-26T10:55:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:04 smithi164 ceph-mon[36970]: pgmap v8161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:04 smithi163 ceph-mon[36873]: pgmap v8161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:04 smithi042 ceph-mon[32775]: pgmap v8161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:06 smithi164 ceph-mon[36970]: pgmap v8162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:06 smithi163 ceph-mon[36873]: pgmap v8162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:06 smithi042 ceph-mon[32775]: pgmap v8162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14336.448201 secs 2024-07-26T10:55:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14354.186840 secs 2024-07-26T10:55:07.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14336.448201 secs 2024-07-26T10:55:07.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14354.186840 secs 2024-07-26T10:55:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14336.448201 secs 2024-07-26T10:55:07.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14354.186840 secs 2024-07-26T10:55:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:08 smithi163 ceph-mon[36873]: pgmap v8163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:08 smithi042 ceph-mon[32775]: pgmap v8163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:08 smithi164 ceph-mon[36970]: pgmap v8163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:10 smithi163 ceph-mon[36873]: pgmap v8164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:10 smithi042 ceph-mon[32775]: pgmap v8164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:10 smithi164 ceph-mon[36970]: pgmap v8164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14341.448321 secs 2024-07-26T10:55:12.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:12 smithi163 ceph-mon[36873]: pgmap v8165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:12.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14359.186985 secs 2024-07-26T10:55:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14341.448321 secs 2024-07-26T10:55:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:12 smithi042 ceph-mon[32775]: pgmap v8165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14359.186985 secs 2024-07-26T10:55:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14341.448321 secs 2024-07-26T10:55:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:12 smithi164 ceph-mon[36970]: pgmap v8165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:12.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14359.186985 secs 2024-07-26T10:55:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:13.225+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:55:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:13.225+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:55:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:13.225+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:55:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:14 smithi163 ceph-mon[36873]: pgmap v8166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:14 smithi042 ceph-mon[32775]: pgmap v8166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:14 smithi164 ceph-mon[36970]: pgmap v8166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:15.117+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:55:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:15.117+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:55:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:15.117+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:55:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:15.117+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:55:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:16 smithi163 ceph-mon[36873]: pgmap v8167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:16 smithi042 ceph-mon[32775]: pgmap v8167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:16 smithi164 ceph-mon[36970]: pgmap v8167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14346.448390 secs 2024-07-26T10:55:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14364.187104 secs 2024-07-26T10:55:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14346.448390 secs 2024-07-26T10:55:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14364.187104 secs 2024-07-26T10:55:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14346.448390 secs 2024-07-26T10:55:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14364.187104 secs 2024-07-26T10:55:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:55:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:18 smithi163 ceph-mon[36873]: pgmap v8168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:18.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:55:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:18 smithi042 ceph-mon[32775]: pgmap v8168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:55:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:18 smithi164 ceph-mon[36970]: pgmap v8168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:55:18.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:55:18.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:55:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:19 smithi163 ceph-mon[36873]: pgmap v8169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:55:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:19 smithi163 ceph-mon[36873]: pgmap v8170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:55:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:19 smithi042 ceph-mon[32775]: pgmap v8169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:55:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:19 smithi042 ceph-mon[32775]: pgmap v8170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:55:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:19 smithi164 ceph-mon[36970]: pgmap v8169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:55:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:19 smithi164 ceph-mon[36970]: pgmap v8170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:55:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:21 smithi042 ceph-mon[32775]: pgmap v8171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:55:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:21 smithi164 ceph-mon[36970]: pgmap v8171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:55:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:21 smithi163 ceph-mon[36873]: pgmap v8171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:55:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14351.448519 secs 2024-07-26T10:55:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14369.187266 secs 2024-07-26T10:55:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14351.448519 secs 2024-07-26T10:55:22.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14369.187266 secs 2024-07-26T10:55:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14351.448519 secs 2024-07-26T10:55:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14369.187266 secs 2024-07-26T10:55:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:23 smithi042 ceph-mon[32775]: pgmap v8172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:55:23.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:23 smithi164 ceph-mon[36970]: pgmap v8172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:55:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:23 smithi163 ceph-mon[36873]: pgmap v8172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:55:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:25 smithi042 ceph-mon[32775]: pgmap v8173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:55:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:25 smithi164 ceph-mon[36970]: pgmap v8173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:55:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:25 smithi163 ceph-mon[36873]: pgmap v8173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:55:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:27 smithi042 ceph-mon[32775]: pgmap v8174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:55:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14356.448609 secs 2024-07-26T10:55:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14374.187372 secs 2024-07-26T10:55:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:27 smithi164 ceph-mon[36970]: pgmap v8174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:55:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14356.448609 secs 2024-07-26T10:55:27.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14374.187372 secs 2024-07-26T10:55:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:27 smithi163 ceph-mon[36873]: pgmap v8174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:55:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14356.448609 secs 2024-07-26T10:55:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14374.187372 secs 2024-07-26T10:55:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:28.231+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:55:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:28.231+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:55:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:28.231+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:55:29.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:29 smithi042 ceph-mon[32775]: pgmap v8175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:55:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:29 smithi164 ceph-mon[36970]: pgmap v8175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:55:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:29 smithi163 ceph-mon[36873]: pgmap v8175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:55:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:31 smithi042 ceph-mon[32775]: pgmap v8176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:31 smithi164 ceph-mon[36970]: pgmap v8176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:31 smithi163 ceph-mon[36873]: pgmap v8176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:32.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14361.448739 secs 2024-07-26T10:55:32.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14379.187502 secs 2024-07-26T10:55:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14361.448739 secs 2024-07-26T10:55:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14379.187502 secs 2024-07-26T10:55:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14361.448739 secs 2024-07-26T10:55:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14379.187502 secs 2024-07-26T10:55:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:33 smithi042 ceph-mon[32775]: pgmap v8177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:33 smithi164 ceph-mon[36970]: pgmap v8177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:33 smithi163 ceph-mon[36873]: pgmap v8177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:35 smithi042 ceph-mon[32775]: pgmap v8178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:55:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:35 smithi164 ceph-mon[36970]: pgmap v8178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:55:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:35 smithi163 ceph-mon[36873]: pgmap v8178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:55:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:37 smithi042 ceph-mon[32775]: pgmap v8179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14366.448803 secs 2024-07-26T10:55:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14384.187636 secs 2024-07-26T10:55:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:37 smithi164 ceph-mon[36970]: pgmap v8179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14366.448803 secs 2024-07-26T10:55:37.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14384.187636 secs 2024-07-26T10:55:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:37 smithi163 ceph-mon[36873]: pgmap v8179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14366.448803 secs 2024-07-26T10:55:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14384.187636 secs 2024-07-26T10:55:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:39 smithi042 ceph-mon[32775]: pgmap v8180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:39 smithi164 ceph-mon[36970]: pgmap v8180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:39 smithi163 ceph-mon[36873]: pgmap v8180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:41 smithi042 ceph-mon[32775]: pgmap v8181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:41 smithi164 ceph-mon[36970]: pgmap v8181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:41 smithi163 ceph-mon[36873]: pgmap v8181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14371.448929 secs 2024-07-26T10:55:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14389.187761 secs 2024-07-26T10:55:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14371.448929 secs 2024-07-26T10:55:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14389.187761 secs 2024-07-26T10:55:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14371.448929 secs 2024-07-26T10:55:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14389.187761 secs 2024-07-26T10:55:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:43.232+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:55:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:43.232+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:55:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:43.232+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:55:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:43 smithi042 ceph-mon[32775]: pgmap v8182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:43 smithi164 ceph-mon[36970]: pgmap v8182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:43 smithi163 ceph-mon[36873]: pgmap v8182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:45 smithi042 ceph-mon[32775]: pgmap v8183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:55:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:45 smithi164 ceph-mon[36970]: pgmap v8183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:55:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:45 smithi163 ceph-mon[36873]: pgmap v8183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:55:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:47 smithi042 ceph-mon[32775]: pgmap v8184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14376.449024 secs 2024-07-26T10:55:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14394.187883 secs 2024-07-26T10:55:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:47 smithi164 ceph-mon[36970]: pgmap v8184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:47.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14376.449024 secs 2024-07-26T10:55:47.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14394.187883 secs 2024-07-26T10:55:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:47 smithi163 ceph-mon[36873]: pgmap v8184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14376.449024 secs 2024-07-26T10:55:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14394.187883 secs 2024-07-26T10:55:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:48 smithi042 ceph-mon[32775]: pgmap v8185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:48 smithi164 ceph-mon[36970]: pgmap v8185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:48 smithi163 ceph-mon[36873]: pgmap v8185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:50 smithi164 ceph-mon[36970]: pgmap v8186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:50 smithi163 ceph-mon[36873]: pgmap v8186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:50 smithi042 ceph-mon[32775]: pgmap v8186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14381.449168 secs 2024-07-26T10:55:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14399.188007 secs 2024-07-26T10:55:52.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:52 smithi164 ceph-mon[36970]: pgmap v8187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14381.449168 secs 2024-07-26T10:55:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14399.188007 secs 2024-07-26T10:55:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:52 smithi163 ceph-mon[36873]: pgmap v8187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14381.449168 secs 2024-07-26T10:55:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14399.188007 secs 2024-07-26T10:55:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:52 smithi042 ceph-mon[32775]: pgmap v8187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:54 smithi164 ceph-mon[36970]: pgmap v8188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:55:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:54 smithi163 ceph-mon[36873]: pgmap v8188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:55:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:54 smithi042 ceph-mon[32775]: pgmap v8188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:55:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:56 smithi164 ceph-mon[36970]: pgmap v8189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:56 smithi163 ceph-mon[36873]: pgmap v8189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:56 smithi042 ceph-mon[32775]: pgmap v8189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:55:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:57.882+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T10:55:57.884 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:57.882+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T10:55:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:57.882+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff6780094b0 tx=0x7ff678073000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T10:55:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:57.882+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff6780094b0 tx=0x7ff678073000 comp rx=0 tx=0).stop 2024-07-26T10:55:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:57.882+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T10:55:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:57.882+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T10:55:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:57.882+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T10:55:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:57.882+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T10:55:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:57.882+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T10:55:57.885 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:57.883+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7ff678003990 tx=0x7ff67803e070 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T10:55:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14386.449243 secs 2024-07-26T10:55:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14404.188129 secs 2024-07-26T10:55:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14386.449243 secs 2024-07-26T10:55:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14404.188129 secs 2024-07-26T10:55:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14386.449243 secs 2024-07-26T10:55:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14404.188129 secs 2024-07-26T10:55:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:58.232+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:55:58.235 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:58.232+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:55:58.235 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:55:58.232+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:55:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:55:58 smithi164 ceph-mon[36970]: pgmap v8190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:55:58 smithi163 ceph-mon[36873]: pgmap v8190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:55:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:55:58 smithi042 ceph-mon[32775]: pgmap v8190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:56:00.124+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:56:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:56:00.124+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:56:00.127 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:56:00.124+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:56:00.127 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:56:00.124+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:56:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:00 smithi164 ceph-mon[36970]: pgmap v8191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:01.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:00 smithi163 ceph-mon[36873]: pgmap v8191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:00 smithi042 ceph-mon[32775]: pgmap v8191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14391.449364 secs 2024-07-26T10:56:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14409.188211 secs 2024-07-26T10:56:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:02 smithi164 ceph-mon[36970]: pgmap v8192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14391.449364 secs 2024-07-26T10:56:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14409.188211 secs 2024-07-26T10:56:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:02 smithi163 ceph-mon[36873]: pgmap v8192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14391.449364 secs 2024-07-26T10:56:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14409.188211 secs 2024-07-26T10:56:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:02 smithi042 ceph-mon[32775]: pgmap v8192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:05 smithi042 ceph-mon[32775]: pgmap v8193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:56:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:05 smithi164 ceph-mon[36970]: pgmap v8193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:56:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:05 smithi163 ceph-mon[36873]: pgmap v8193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:56:07.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:07 smithi042 ceph-mon[32775]: pgmap v8194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:07.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14396.449486 secs 2024-07-26T10:56:07.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14414.188341 secs 2024-07-26T10:56:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:07 smithi164 ceph-mon[36970]: pgmap v8194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:07.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14396.449486 secs 2024-07-26T10:56:07.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14414.188341 secs 2024-07-26T10:56:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:07 smithi163 ceph-mon[36873]: pgmap v8194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14396.449486 secs 2024-07-26T10:56:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14414.188341 secs 2024-07-26T10:56:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:09 smithi042 ceph-mon[32775]: pgmap v8195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:09 smithi164 ceph-mon[36970]: pgmap v8195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:09 smithi163 ceph-mon[36873]: pgmap v8195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:11 smithi042 ceph-mon[32775]: pgmap v8196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:11 smithi164 ceph-mon[36970]: pgmap v8196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:11 smithi163 ceph-mon[36873]: pgmap v8196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14401.449607 secs 2024-07-26T10:56:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14419.188427 secs 2024-07-26T10:56:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14401.449607 secs 2024-07-26T10:56:12.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14419.188427 secs 2024-07-26T10:56:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14401.449607 secs 2024-07-26T10:56:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14419.188427 secs 2024-07-26T10:56:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:56:13.234+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:56:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:56:13.234+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:56:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:56:13.234+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:56:13.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:13 smithi042 ceph-mon[32775]: pgmap v8197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:13 smithi164 ceph-mon[36970]: pgmap v8197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:13 smithi163 ceph-mon[36873]: pgmap v8197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:15 smithi042 ceph-mon[32775]: pgmap v8198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:56:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:15 smithi164 ceph-mon[36970]: pgmap v8198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:56:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:15 smithi163 ceph-mon[36873]: pgmap v8198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:56:17.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:17 smithi163 ceph-mon[36873]: pgmap v8199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:17.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14406.449733 secs 2024-07-26T10:56:17.561 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14424.188584 secs 2024-07-26T10:56:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:17 smithi042 ceph-mon[32775]: pgmap v8199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14406.449733 secs 2024-07-26T10:56:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14424.188584 secs 2024-07-26T10:56:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:17 smithi164 ceph-mon[36970]: pgmap v8199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14406.449733 secs 2024-07-26T10:56:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14424.188584 secs 2024-07-26T10:56:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:56:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:18.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:56:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:56:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:56:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:56:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:56:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:19 smithi042 ceph-mon[32775]: pgmap v8200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:19 smithi042 ceph-mon[32775]: pgmap v8201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:56:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:19 smithi042 ceph-mon[32775]: pgmap v8202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:56:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:19 smithi164 ceph-mon[36970]: pgmap v8200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:19 smithi164 ceph-mon[36970]: pgmap v8201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:56:19.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:19 smithi164 ceph-mon[36970]: pgmap v8202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:56:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:19 smithi163 ceph-mon[36873]: pgmap v8200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:19 smithi163 ceph-mon[36873]: pgmap v8201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:56:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:19 smithi163 ceph-mon[36873]: pgmap v8202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:56:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:21 smithi042 ceph-mon[32775]: pgmap v8203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:56:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:21 smithi164 ceph-mon[36970]: pgmap v8203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:56:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:21 smithi163 ceph-mon[36873]: pgmap v8203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:56:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14411.449855 secs 2024-07-26T10:56:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14429.188724 secs 2024-07-26T10:56:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14411.449855 secs 2024-07-26T10:56:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14429.188724 secs 2024-07-26T10:56:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14411.449855 secs 2024-07-26T10:56:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14429.188724 secs 2024-07-26T10:56:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:23 smithi042 ceph-mon[32775]: pgmap v8204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:56:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:23 smithi164 ceph-mon[36970]: pgmap v8204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:56:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:23 smithi163 ceph-mon[36873]: pgmap v8204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:56:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:25 smithi042 ceph-mon[32775]: pgmap v8205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:56:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:25 smithi164 ceph-mon[36970]: pgmap v8205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:56:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:25 smithi163 ceph-mon[36873]: pgmap v8205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:56:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14416.449977 secs 2024-07-26T10:56:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:27 smithi042 ceph-mon[32775]: pgmap v8206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:56:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14434.188863 secs 2024-07-26T10:56:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14416.449977 secs 2024-07-26T10:56:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:27 smithi164 ceph-mon[36970]: pgmap v8206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:56:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14434.188863 secs 2024-07-26T10:56:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14416.449977 secs 2024-07-26T10:56:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:27 smithi163 ceph-mon[36873]: pgmap v8206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:56:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14434.188863 secs 2024-07-26T10:56:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:56:28.235+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:56:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:56:28.235+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:56:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:56:28.235+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:56:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:29 smithi042 ceph-mon[32775]: pgmap v8207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:56:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:29 smithi164 ceph-mon[36970]: pgmap v8207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:56:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:29 smithi163 ceph-mon[36873]: pgmap v8207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:56:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:31 smithi042 ceph-mon[32775]: pgmap v8208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:31 smithi164 ceph-mon[36970]: pgmap v8208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:31 smithi163 ceph-mon[36873]: pgmap v8208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:32.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14421.450120 secs 2024-07-26T10:56:32.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14439.188951 secs 2024-07-26T10:56:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14421.450120 secs 2024-07-26T10:56:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14439.188951 secs 2024-07-26T10:56:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14421.450120 secs 2024-07-26T10:56:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14439.188951 secs 2024-07-26T10:56:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:33 smithi042 ceph-mon[32775]: pgmap v8209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:33 smithi164 ceph-mon[36970]: pgmap v8209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:33 smithi163 ceph-mon[36873]: pgmap v8209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:35 smithi042 ceph-mon[32775]: pgmap v8210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:35 smithi164 ceph-mon[36970]: pgmap v8210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:35 smithi163 ceph-mon[36873]: pgmap v8210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14426.450242 secs 2024-07-26T10:56:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:37 smithi042 ceph-mon[32775]: pgmap v8211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14444.189031 secs 2024-07-26T10:56:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14426.450242 secs 2024-07-26T10:56:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:37 smithi164 ceph-mon[36970]: pgmap v8211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14444.189031 secs 2024-07-26T10:56:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14426.450242 secs 2024-07-26T10:56:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:37 smithi163 ceph-mon[36873]: pgmap v8211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14444.189031 secs 2024-07-26T10:56:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:39 smithi042 ceph-mon[32775]: pgmap v8212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:39 smithi164 ceph-mon[36970]: pgmap v8212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:39 smithi163 ceph-mon[36873]: pgmap v8212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:41 smithi042 ceph-mon[32775]: pgmap v8213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:41 smithi164 ceph-mon[36970]: pgmap v8213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:41 smithi163 ceph-mon[36873]: pgmap v8213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14431.450355 secs 2024-07-26T10:56:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14449.189152 secs 2024-07-26T10:56:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14431.450355 secs 2024-07-26T10:56:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14449.189152 secs 2024-07-26T10:56:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14431.450355 secs 2024-07-26T10:56:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14449.189152 secs 2024-07-26T10:56:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:56:43.236+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:56:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:56:43.236+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:56:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:56:43.236+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:56:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:43 smithi042 ceph-mon[32775]: pgmap v8214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:43 smithi164 ceph-mon[36970]: pgmap v8214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:43 smithi163 ceph-mon[36873]: pgmap v8214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:45 smithi042 ceph-mon[32775]: pgmap v8215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:45 smithi164 ceph-mon[36970]: pgmap v8215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:45 smithi163 ceph-mon[36873]: pgmap v8215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14436.450473 secs 2024-07-26T10:56:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:47 smithi042 ceph-mon[32775]: pgmap v8216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14454.189278 secs 2024-07-26T10:56:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14436.450473 secs 2024-07-26T10:56:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:47 smithi164 ceph-mon[36970]: pgmap v8216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14454.189278 secs 2024-07-26T10:56:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14436.450473 secs 2024-07-26T10:56:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:47 smithi163 ceph-mon[36873]: pgmap v8216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14454.189278 secs 2024-07-26T10:56:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:49 smithi042 ceph-mon[32775]: pgmap v8217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:49 smithi164 ceph-mon[36970]: pgmap v8217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:49 smithi163 ceph-mon[36873]: pgmap v8217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:51 smithi042 ceph-mon[32775]: pgmap v8218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:51 smithi164 ceph-mon[36970]: pgmap v8218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:51 smithi163 ceph-mon[36873]: pgmap v8218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14441.450570 secs 2024-07-26T10:56:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14459.189397 secs 2024-07-26T10:56:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14441.450570 secs 2024-07-26T10:56:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14459.189397 secs 2024-07-26T10:56:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14441.450570 secs 2024-07-26T10:56:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14459.189397 secs 2024-07-26T10:56:53.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:53 smithi042 ceph-mon[32775]: pgmap v8219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:53 smithi164 ceph-mon[36970]: pgmap v8219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:53 smithi163 ceph-mon[36873]: pgmap v8219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:55 smithi042 ceph-mon[32775]: pgmap v8220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:55 smithi164 ceph-mon[36970]: pgmap v8220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:55 smithi163 ceph-mon[36873]: pgmap v8220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:56:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14446.450697 secs 2024-07-26T10:56:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:57 smithi042 ceph-mon[32775]: pgmap v8221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14464.189517 secs 2024-07-26T10:56:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14446.450697 secs 2024-07-26T10:56:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:57 smithi164 ceph-mon[36970]: pgmap v8221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14464.189517 secs 2024-07-26T10:56:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14446.450697 secs 2024-07-26T10:56:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:57 smithi163 ceph-mon[36873]: pgmap v8221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14464.189517 secs 2024-07-26T10:56:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:56:58.242+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:56:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:56:58.242+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:56:58.243 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:56:58.242+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:56:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:56:59 smithi042 ceph-mon[32775]: pgmap v8222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:56:59 smithi164 ceph-mon[36970]: pgmap v8222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:56:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:56:59 smithi163 ceph-mon[36873]: pgmap v8222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:01 smithi042 ceph-mon[32775]: pgmap v8223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:01 smithi164 ceph-mon[36970]: pgmap v8223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:01 smithi163 ceph-mon[36873]: pgmap v8223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14451.450840 secs 2024-07-26T10:57:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14469.189638 secs 2024-07-26T10:57:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14451.450840 secs 2024-07-26T10:57:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14469.189638 secs 2024-07-26T10:57:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14451.450840 secs 2024-07-26T10:57:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14469.189638 secs 2024-07-26T10:57:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:03 smithi164 ceph-mon[36970]: pgmap v8224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:03 smithi163 ceph-mon[36873]: pgmap v8224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:03 smithi042 ceph-mon[32775]: pgmap v8224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:05 smithi164 ceph-mon[36970]: pgmap v8225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:05 smithi163 ceph-mon[36873]: pgmap v8225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:05 smithi042 ceph-mon[32775]: pgmap v8225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:07.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14456.450963 secs 2024-07-26T10:57:07.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:07 smithi042 ceph-mon[32775]: pgmap v8226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:07.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14474.189758 secs 2024-07-26T10:57:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14456.450963 secs 2024-07-26T10:57:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:07 smithi164 ceph-mon[36970]: pgmap v8226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14474.189758 secs 2024-07-26T10:57:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14456.450963 secs 2024-07-26T10:57:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:07 smithi163 ceph-mon[36873]: pgmap v8226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14474.189758 secs 2024-07-26T10:57:09.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:09 smithi164 ceph-mon[36970]: pgmap v8227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:09 smithi163 ceph-mon[36873]: pgmap v8227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:09 smithi042 ceph-mon[32775]: pgmap v8227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:11 smithi164 ceph-mon[36970]: pgmap v8228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:11 smithi163 ceph-mon[36873]: pgmap v8228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:11 smithi042 ceph-mon[32775]: pgmap v8228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14461.451091 secs 2024-07-26T10:57:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14479.189881 secs 2024-07-26T10:57:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14461.451091 secs 2024-07-26T10:57:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14479.189881 secs 2024-07-26T10:57:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14461.451091 secs 2024-07-26T10:57:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14479.189881 secs 2024-07-26T10:57:13.246 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:57:13.244+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:57:13.246 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:57:13.244+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:57:13.246 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:57:13.244+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:57:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:13 smithi164 ceph-mon[36970]: pgmap v8229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:13 smithi163 ceph-mon[36873]: pgmap v8229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:13 smithi042 ceph-mon[32775]: pgmap v8229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:15 smithi164 ceph-mon[36970]: pgmap v8230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:15 smithi163 ceph-mon[36873]: pgmap v8230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:15 smithi042 ceph-mon[32775]: pgmap v8230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14466.451234 secs 2024-07-26T10:57:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:17 smithi164 ceph-mon[36970]: pgmap v8231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14484.189999 secs 2024-07-26T10:57:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14466.451234 secs 2024-07-26T10:57:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:17 smithi163 ceph-mon[36873]: pgmap v8231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14484.189999 secs 2024-07-26T10:57:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14466.451234 secs 2024-07-26T10:57:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:17 smithi042 ceph-mon[32775]: pgmap v8231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14484.189999 secs 2024-07-26T10:57:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:57:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:57:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:57:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:19 smithi164 ceph-mon[36970]: pgmap v8232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:57:19.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:57:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:19 smithi163 ceph-mon[36873]: pgmap v8232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:57:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:57:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:19 smithi042 ceph-mon[32775]: pgmap v8232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:57:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:57:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:57:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:20 smithi164 ceph-mon[36970]: pgmap v8233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:57:20.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:20 smithi164 ceph-mon[36970]: pgmap v8234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:57:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:20 smithi163 ceph-mon[36873]: pgmap v8233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:57:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:20 smithi163 ceph-mon[36873]: pgmap v8234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:57:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:20 smithi042 ceph-mon[32775]: pgmap v8233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T10:57:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:20 smithi042 ceph-mon[32775]: pgmap v8234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:57:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:22 smithi164 ceph-mon[36970]: pgmap v8235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:57:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14471.451339 secs 2024-07-26T10:57:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14489.190101 secs 2024-07-26T10:57:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:22 smithi163 ceph-mon[36873]: pgmap v8235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:57:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14471.451339 secs 2024-07-26T10:57:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14489.190101 secs 2024-07-26T10:57:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:22 smithi042 ceph-mon[32775]: pgmap v8235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:57:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14471.451339 secs 2024-07-26T10:57:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14489.190101 secs 2024-07-26T10:57:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:24 smithi164 ceph-mon[36970]: pgmap v8236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:57:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:24 smithi163 ceph-mon[36873]: pgmap v8236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:57:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:24 smithi042 ceph-mon[32775]: pgmap v8236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:57:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:26 smithi164 ceph-mon[36970]: pgmap v8237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:57:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:26 smithi163 ceph-mon[36873]: pgmap v8237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:57:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:26 smithi042 ceph-mon[32775]: pgmap v8237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:57:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14476.451439 secs 2024-07-26T10:57:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14494.190220 secs 2024-07-26T10:57:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14476.451439 secs 2024-07-26T10:57:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14494.190220 secs 2024-07-26T10:57:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14476.451439 secs 2024-07-26T10:57:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14494.190220 secs 2024-07-26T10:57:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:57:28.248+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:57:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:57:28.248+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:57:28.250 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:57:28.248+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:57:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:28 smithi164 ceph-mon[36970]: pgmap v8238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:57:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:28 smithi163 ceph-mon[36873]: pgmap v8238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:57:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:28 smithi042 ceph-mon[32775]: pgmap v8238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:57:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:30 smithi164 ceph-mon[36970]: pgmap v8239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:57:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:30 smithi163 ceph-mon[36873]: pgmap v8239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:57:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:30 smithi042 ceph-mon[32775]: pgmap v8239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:57:32.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:32 smithi164 ceph-mon[36970]: pgmap v8240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:32.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14481.451555 secs 2024-07-26T10:57:32.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14499.190332 secs 2024-07-26T10:57:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:32 smithi163 ceph-mon[36873]: pgmap v8240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14481.451555 secs 2024-07-26T10:57:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14499.190332 secs 2024-07-26T10:57:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:32 smithi042 ceph-mon[32775]: pgmap v8240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14481.451555 secs 2024-07-26T10:57:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14499.190332 secs 2024-07-26T10:57:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:34 smithi164 ceph-mon[36970]: pgmap v8241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:34 smithi163 ceph-mon[36873]: pgmap v8241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:34 smithi042 ceph-mon[32775]: pgmap v8241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:36 smithi164 ceph-mon[36970]: pgmap v8242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:36 smithi163 ceph-mon[36873]: pgmap v8242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:36 smithi042 ceph-mon[32775]: pgmap v8242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14486.451677 secs 2024-07-26T10:57:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14504.190452 secs 2024-07-26T10:57:37.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14486.451677 secs 2024-07-26T10:57:37.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14504.190452 secs 2024-07-26T10:57:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14486.451677 secs 2024-07-26T10:57:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14504.190452 secs 2024-07-26T10:57:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:38 smithi163 ceph-mon[36873]: pgmap v8243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:38 smithi042 ceph-mon[32775]: pgmap v8243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:38 smithi164 ceph-mon[36970]: pgmap v8243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:40 smithi163 ceph-mon[36873]: pgmap v8244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:57:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:40 smithi042 ceph-mon[32775]: pgmap v8244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:57:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:40 smithi164 ceph-mon[36970]: pgmap v8244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:57:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:42 smithi163 ceph-mon[36873]: pgmap v8245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14491.451808 secs 2024-07-26T10:57:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14509.190572 secs 2024-07-26T10:57:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:42 smithi042 ceph-mon[32775]: pgmap v8245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14491.451808 secs 2024-07-26T10:57:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14509.190572 secs 2024-07-26T10:57:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:42 smithi164 ceph-mon[36970]: pgmap v8245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14491.451808 secs 2024-07-26T10:57:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14509.190572 secs 2024-07-26T10:57:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:57:43.248+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:57:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:57:43.248+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:57:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:57:43.248+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:57:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:44 smithi163 ceph-mon[36873]: pgmap v8246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:44 smithi042 ceph-mon[32775]: pgmap v8246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:44 smithi164 ceph-mon[36970]: pgmap v8246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:46 smithi163 ceph-mon[36873]: pgmap v8247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:46 smithi042 ceph-mon[32775]: pgmap v8247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:46 smithi164 ceph-mon[36970]: pgmap v8247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14496.451922 secs 2024-07-26T10:57:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14514.190699 secs 2024-07-26T10:57:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14496.451922 secs 2024-07-26T10:57:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14514.190699 secs 2024-07-26T10:57:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14496.451922 secs 2024-07-26T10:57:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14514.190699 secs 2024-07-26T10:57:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:48 smithi163 ceph-mon[36873]: pgmap v8248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:48 smithi042 ceph-mon[32775]: pgmap v8248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:48 smithi164 ceph-mon[36970]: pgmap v8248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:50 smithi163 ceph-mon[36873]: pgmap v8249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:57:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:50 smithi042 ceph-mon[32775]: pgmap v8249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:57:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:50 smithi164 ceph-mon[36970]: pgmap v8249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:57:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:52 smithi163 ceph-mon[36873]: pgmap v8250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14501.452062 secs 2024-07-26T10:57:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14519.190842 secs 2024-07-26T10:57:52.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:52 smithi042 ceph-mon[32775]: pgmap v8250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:52.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14501.452062 secs 2024-07-26T10:57:52.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14519.190842 secs 2024-07-26T10:57:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:52 smithi164 ceph-mon[36970]: pgmap v8250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14501.452062 secs 2024-07-26T10:57:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14519.190842 secs 2024-07-26T10:57:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:54 smithi163 ceph-mon[36873]: pgmap v8251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:54 smithi042 ceph-mon[32775]: pgmap v8251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:54 smithi164 ceph-mon[36970]: pgmap v8251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:56 smithi042 ceph-mon[32775]: pgmap v8252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:56 smithi164 ceph-mon[36970]: pgmap v8252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:56 smithi163 ceph-mon[36873]: pgmap v8252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:57:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14506.452187 secs 2024-07-26T10:57:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14524.190910 secs 2024-07-26T10:57:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14506.452187 secs 2024-07-26T10:57:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14524.190910 secs 2024-07-26T10:57:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14506.452187 secs 2024-07-26T10:57:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14524.190910 secs 2024-07-26T10:57:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:57:58.254+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:57:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:57:58.254+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:57:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:57:58.254+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:57:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:57:58 smithi042 ceph-mon[32775]: pgmap v8253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:57:58 smithi164 ceph-mon[36970]: pgmap v8253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:57:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:57:58 smithi163 ceph-mon[36873]: pgmap v8253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:00 smithi042 ceph-mon[32775]: pgmap v8254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:00 smithi164 ceph-mon[36970]: pgmap v8254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:00 smithi163 ceph-mon[36873]: pgmap v8254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:02.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:02 smithi164 ceph-mon[36970]: pgmap v8255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:02.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14511.452302 secs 2024-07-26T10:58:02.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14529.191055 secs 2024-07-26T10:58:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:02 smithi042 ceph-mon[32775]: pgmap v8255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14511.452302 secs 2024-07-26T10:58:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14529.191055 secs 2024-07-26T10:58:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:02 smithi163 ceph-mon[36873]: pgmap v8255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14511.452302 secs 2024-07-26T10:58:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14529.191055 secs 2024-07-26T10:58:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:04 smithi042 ceph-mon[32775]: pgmap v8256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:04 smithi164 ceph-mon[36970]: pgmap v8256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:04 smithi163 ceph-mon[36873]: pgmap v8256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:06 smithi042 ceph-mon[32775]: pgmap v8257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:06 smithi164 ceph-mon[36970]: pgmap v8257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:06 smithi163 ceph-mon[36873]: pgmap v8257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:07.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14516.452429 secs 2024-07-26T10:58:07.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14534.191181 secs 2024-07-26T10:58:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14516.452429 secs 2024-07-26T10:58:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14534.191181 secs 2024-07-26T10:58:08.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14516.452429 secs 2024-07-26T10:58:08.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14534.191181 secs 2024-07-26T10:58:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:08 smithi042 ceph-mon[32775]: pgmap v8258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:08 smithi164 ceph-mon[36970]: pgmap v8258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:08 smithi163 ceph-mon[36873]: pgmap v8258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:10 smithi042 ceph-mon[32775]: pgmap v8259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:10 smithi164 ceph-mon[36970]: pgmap v8259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:10 smithi163 ceph-mon[36873]: pgmap v8259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:12 smithi042 ceph-mon[32775]: pgmap v8260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14521.452523 secs 2024-07-26T10:58:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14539.191277 secs 2024-07-26T10:58:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:12 smithi164 ceph-mon[36970]: pgmap v8260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:12.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14521.452523 secs 2024-07-26T10:58:12.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14539.191277 secs 2024-07-26T10:58:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:12 smithi163 ceph-mon[36873]: pgmap v8260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14521.452523 secs 2024-07-26T10:58:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14539.191277 secs 2024-07-26T10:58:13.257 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:58:13.255+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:58:13.257 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:58:13.255+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:58:13.257 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:58:13.255+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:58:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:14 smithi042 ceph-mon[32775]: pgmap v8261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:14 smithi164 ceph-mon[36970]: pgmap v8261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:14 smithi163 ceph-mon[36873]: pgmap v8261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:16 smithi042 ceph-mon[32775]: pgmap v8262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:16 smithi164 ceph-mon[36970]: pgmap v8262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:16 smithi163 ceph-mon[36873]: pgmap v8262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14526.452623 secs 2024-07-26T10:58:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14544.191398 secs 2024-07-26T10:58:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14526.452623 secs 2024-07-26T10:58:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14544.191398 secs 2024-07-26T10:58:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14526.452623 secs 2024-07-26T10:58:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14544.191398 secs 2024-07-26T10:58:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:18 smithi042 ceph-mon[32775]: pgmap v8263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:18 smithi164 ceph-mon[36970]: pgmap v8263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:18 smithi163 ceph-mon[36873]: pgmap v8263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:58:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:58:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:58:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:58:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:58:19.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:58:19.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:58:19.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:58:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:58:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:58:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:58:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:58:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:20 smithi042 ceph-mon[32775]: pgmap v8264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:20 smithi042 ceph-mon[32775]: pgmap v8265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:58:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:20 smithi042 ceph-mon[32775]: pgmap v8266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:58:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:20 smithi164 ceph-mon[36970]: pgmap v8264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:20 smithi164 ceph-mon[36970]: pgmap v8265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:58:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:20 smithi164 ceph-mon[36970]: pgmap v8266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:58:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:20 smithi163 ceph-mon[36873]: pgmap v8264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:20 smithi163 ceph-mon[36873]: pgmap v8265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T10:58:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:20 smithi163 ceph-mon[36873]: pgmap v8266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:58:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14531.452747 secs 2024-07-26T10:58:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:22 smithi042 ceph-mon[32775]: pgmap v8267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:58:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14549.191524 secs 2024-07-26T10:58:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14531.452747 secs 2024-07-26T10:58:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:22 smithi164 ceph-mon[36970]: pgmap v8267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:58:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14549.191524 secs 2024-07-26T10:58:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14531.452747 secs 2024-07-26T10:58:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:22 smithi163 ceph-mon[36873]: pgmap v8267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:58:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14549.191524 secs 2024-07-26T10:58:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:24 smithi042 ceph-mon[32775]: pgmap v8268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:58:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:24 smithi164 ceph-mon[36970]: pgmap v8268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:58:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:24 smithi163 ceph-mon[36873]: pgmap v8268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:58:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:26 smithi042 ceph-mon[32775]: pgmap v8269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:58:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:26 smithi164 ceph-mon[36970]: pgmap v8269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:58:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:26 smithi163 ceph-mon[36873]: pgmap v8269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:58:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14536.452876 secs 2024-07-26T10:58:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14554.191664 secs 2024-07-26T10:58:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14536.452876 secs 2024-07-26T10:58:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14554.191664 secs 2024-07-26T10:58:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14536.452876 secs 2024-07-26T10:58:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14554.191664 secs 2024-07-26T10:58:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:58:28.261+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:58:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:58:28.261+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:58:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:58:28.261+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:58:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:28 smithi042 ceph-mon[32775]: pgmap v8270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:58:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:28 smithi164 ceph-mon[36970]: pgmap v8270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:58:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:28 smithi163 ceph-mon[36873]: pgmap v8270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:58:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:30 smithi042 ceph-mon[32775]: pgmap v8271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:58:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:30 smithi164 ceph-mon[36970]: pgmap v8271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:58:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:30 smithi163 ceph-mon[36873]: pgmap v8271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:58:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:31 smithi042 ceph-mon[32775]: pgmap v8272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:31 smithi164 ceph-mon[36970]: pgmap v8272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:31 smithi163 ceph-mon[36873]: pgmap v8272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14541.452997 secs 2024-07-26T10:58:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14559.191807 secs 2024-07-26T10:58:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14541.452997 secs 2024-07-26T10:58:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14559.191807 secs 2024-07-26T10:58:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14541.452997 secs 2024-07-26T10:58:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14559.191807 secs 2024-07-26T10:58:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:33 smithi164 ceph-mon[36970]: pgmap v8273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:33 smithi163 ceph-mon[36873]: pgmap v8273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:33 smithi042 ceph-mon[32775]: pgmap v8273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:36 smithi042 ceph-mon[32775]: pgmap v8274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:36 smithi164 ceph-mon[36970]: pgmap v8274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:36 smithi163 ceph-mon[36873]: pgmap v8274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14546.453127 secs 2024-07-26T10:58:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14564.191922 secs 2024-07-26T10:58:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14546.453127 secs 2024-07-26T10:58:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14564.191922 secs 2024-07-26T10:58:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14546.453127 secs 2024-07-26T10:58:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14564.191922 secs 2024-07-26T10:58:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:38 smithi042 ceph-mon[32775]: pgmap v8275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:38 smithi164 ceph-mon[36970]: pgmap v8275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:38 smithi163 ceph-mon[36873]: pgmap v8275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:40 smithi042 ceph-mon[32775]: pgmap v8276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:40.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:40 smithi164 ceph-mon[36970]: pgmap v8276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:40 smithi163 ceph-mon[36873]: pgmap v8276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14551.453243 secs 2024-07-26T10:58:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:42 smithi042 ceph-mon[32775]: pgmap v8277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14569.192015 secs 2024-07-26T10:58:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14551.453243 secs 2024-07-26T10:58:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:42 smithi164 ceph-mon[36970]: pgmap v8277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14569.192015 secs 2024-07-26T10:58:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14551.453243 secs 2024-07-26T10:58:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:42 smithi163 ceph-mon[36873]: pgmap v8277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14569.192015 secs 2024-07-26T10:58:43.263 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:58:43.262+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:58:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:58:43.262+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:58:43.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:58:43.262+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:58:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:44 smithi042 ceph-mon[32775]: pgmap v8278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:44 smithi164 ceph-mon[36970]: pgmap v8278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:44 smithi163 ceph-mon[36873]: pgmap v8278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:46 smithi042 ceph-mon[32775]: pgmap v8279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:46 smithi164 ceph-mon[36970]: pgmap v8279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:46 smithi163 ceph-mon[36873]: pgmap v8279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14556.453366 secs 2024-07-26T10:58:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14574.192136 secs 2024-07-26T10:58:47.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14556.453366 secs 2024-07-26T10:58:47.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14574.192136 secs 2024-07-26T10:58:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14556.453366 secs 2024-07-26T10:58:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14574.192136 secs 2024-07-26T10:58:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:48 smithi042 ceph-mon[32775]: pgmap v8280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:48 smithi164 ceph-mon[36970]: pgmap v8280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:48 smithi163 ceph-mon[36873]: pgmap v8280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:50 smithi042 ceph-mon[32775]: pgmap v8281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:50 smithi164 ceph-mon[36970]: pgmap v8281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:50 smithi163 ceph-mon[36873]: pgmap v8281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14561.453448 secs 2024-07-26T10:58:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:52 smithi042 ceph-mon[32775]: pgmap v8282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14579.192235 secs 2024-07-26T10:58:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14561.453448 secs 2024-07-26T10:58:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:52 smithi164 ceph-mon[36970]: pgmap v8282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14579.192235 secs 2024-07-26T10:58:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14561.453448 secs 2024-07-26T10:58:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:52 smithi163 ceph-mon[36873]: pgmap v8282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14579.192235 secs 2024-07-26T10:58:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:54 smithi042 ceph-mon[32775]: pgmap v8283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:54 smithi164 ceph-mon[36970]: pgmap v8283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:54 smithi163 ceph-mon[36873]: pgmap v8283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:58:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:56 smithi042 ceph-mon[32775]: pgmap v8284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:56 smithi164 ceph-mon[36970]: pgmap v8284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:56 smithi163 ceph-mon[36873]: pgmap v8284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:58:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14566.453573 secs 2024-07-26T10:58:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14584.192354 secs 2024-07-26T10:58:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14566.453573 secs 2024-07-26T10:58:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14584.192354 secs 2024-07-26T10:58:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14566.453573 secs 2024-07-26T10:58:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14584.192354 secs 2024-07-26T10:58:58.269 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:58:58.268+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:58:58.269 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:58:58.268+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:58:58.269 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:58:58.268+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:58:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:58:58 smithi042 ceph-mon[32775]: pgmap v8285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:58.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:58:58 smithi164 ceph-mon[36970]: pgmap v8285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:58:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:58:58 smithi163 ceph-mon[36873]: pgmap v8285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:59:00.153+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:59:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:59:00.153+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:59:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:59:00.153+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T10:59:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:59:00.153+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T10:59:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:00 smithi042 ceph-mon[32775]: pgmap v8286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:00 smithi164 ceph-mon[36970]: pgmap v8286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:00 smithi163 ceph-mon[36873]: pgmap v8286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14571.453696 secs 2024-07-26T10:59:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:02 smithi042 ceph-mon[32775]: pgmap v8287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14589.192493 secs 2024-07-26T10:59:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14571.453696 secs 2024-07-26T10:59:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:02 smithi164 ceph-mon[36970]: pgmap v8287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14589.192493 secs 2024-07-26T10:59:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14571.453696 secs 2024-07-26T10:59:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:02 smithi163 ceph-mon[36873]: pgmap v8287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14589.192493 secs 2024-07-26T10:59:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:04 smithi164 ceph-mon[36970]: pgmap v8288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:59:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:04 smithi163 ceph-mon[36873]: pgmap v8288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:59:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:04 smithi042 ceph-mon[32775]: pgmap v8288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:59:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:06 smithi164 ceph-mon[36970]: pgmap v8289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:06 smithi163 ceph-mon[36873]: pgmap v8289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:06 smithi042 ceph-mon[32775]: pgmap v8289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14576.453819 secs 2024-07-26T10:59:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14594.192591 secs 2024-07-26T10:59:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14576.453819 secs 2024-07-26T10:59:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14594.192591 secs 2024-07-26T10:59:07.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14576.453819 secs 2024-07-26T10:59:07.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14594.192591 secs 2024-07-26T10:59:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:08 smithi164 ceph-mon[36970]: pgmap v8290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:08 smithi163 ceph-mon[36873]: pgmap v8290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:08 smithi042 ceph-mon[32775]: pgmap v8290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:10 smithi164 ceph-mon[36970]: pgmap v8291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:10 smithi163 ceph-mon[36873]: pgmap v8291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:10 smithi042 ceph-mon[32775]: pgmap v8291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14581.453964 secs 2024-07-26T10:59:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:12 smithi164 ceph-mon[36970]: pgmap v8292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:12.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14599.192669 secs 2024-07-26T10:59:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14581.453964 secs 2024-07-26T10:59:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:12 smithi163 ceph-mon[36873]: pgmap v8292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14599.192669 secs 2024-07-26T10:59:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14581.453964 secs 2024-07-26T10:59:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:12 smithi042 ceph-mon[32775]: pgmap v8292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14599.192669 secs 2024-07-26T10:59:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:59:13.270+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:59:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:59:13.270+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:59:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:59:13.270+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:59:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:14 smithi164 ceph-mon[36970]: pgmap v8293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:59:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:14 smithi163 ceph-mon[36873]: pgmap v8293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:59:14.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:14 smithi042 ceph-mon[32775]: pgmap v8293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:59:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:16 smithi164 ceph-mon[36970]: pgmap v8294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:16 smithi163 ceph-mon[36873]: pgmap v8294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:16 smithi042 ceph-mon[32775]: pgmap v8294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14586.454109 secs 2024-07-26T10:59:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14604.192807 secs 2024-07-26T10:59:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14586.454109 secs 2024-07-26T10:59:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14604.192807 secs 2024-07-26T10:59:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14586.454109 secs 2024-07-26T10:59:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14604.192807 secs 2024-07-26T10:59:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:18 smithi164 ceph-mon[36970]: pgmap v8295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:18 smithi163 ceph-mon[36873]: pgmap v8295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:18 smithi042 ceph-mon[32775]: pgmap v8295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:59:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:59:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T10:59:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:20 smithi164 ceph-mon[36970]: pgmap v8296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:59:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:59:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:59:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:20 smithi163 ceph-mon[36873]: pgmap v8296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:59:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:59:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:59:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:20 smithi042 ceph-mon[32775]: pgmap v8296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T10:59:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T10:59:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T10:59:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:21 smithi164 ceph-mon[36970]: pgmap v8297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:59:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:21 smithi164 ceph-mon[36970]: pgmap v8298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:59:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:21 smithi163 ceph-mon[36873]: pgmap v8297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:59:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:21 smithi163 ceph-mon[36873]: pgmap v8298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:59:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:21 smithi042 ceph-mon[32775]: pgmap v8297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T10:59:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:21 smithi042 ceph-mon[32775]: pgmap v8298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:59:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14591.454243 secs 2024-07-26T10:59:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14609.192884 secs 2024-07-26T10:59:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14591.454243 secs 2024-07-26T10:59:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14609.192884 secs 2024-07-26T10:59:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14591.454243 secs 2024-07-26T10:59:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14609.192884 secs 2024-07-26T10:59:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:23 smithi164 ceph-mon[36970]: pgmap v8299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:59:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:23 smithi163 ceph-mon[36873]: pgmap v8299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:59:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:23 smithi042 ceph-mon[32775]: pgmap v8299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:59:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:25 smithi164 ceph-mon[36970]: pgmap v8300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:59:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:25 smithi163 ceph-mon[36873]: pgmap v8300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:59:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:25 smithi042 ceph-mon[32775]: pgmap v8300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T10:59:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:27 smithi164 ceph-mon[36970]: pgmap v8301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:59:27.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14596.454364 secs 2024-07-26T10:59:27.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14614.193002 secs 2024-07-26T10:59:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:27 smithi163 ceph-mon[36873]: pgmap v8301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:59:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14596.454364 secs 2024-07-26T10:59:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14614.193002 secs 2024-07-26T10:59:27.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:27 smithi042 ceph-mon[32775]: pgmap v8301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:59:27.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14596.454364 secs 2024-07-26T10:59:27.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14614.193002 secs 2024-07-26T10:59:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:59:28.273+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:59:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:59:28.273+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:59:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:59:28.273+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:59:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:29 smithi164 ceph-mon[36970]: pgmap v8302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:59:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:29 smithi163 ceph-mon[36873]: pgmap v8302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:59:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:29 smithi042 ceph-mon[32775]: pgmap v8302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T10:59:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:31 smithi164 ceph-mon[36970]: pgmap v8303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:59:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:31 smithi163 ceph-mon[36873]: pgmap v8303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:59:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:31 smithi042 ceph-mon[32775]: pgmap v8303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T10:59:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14601.454460 secs 2024-07-26T10:59:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14619.193124 secs 2024-07-26T10:59:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14601.454460 secs 2024-07-26T10:59:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14619.193124 secs 2024-07-26T10:59:32.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14601.454460 secs 2024-07-26T10:59:32.899 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14619.193124 secs 2024-07-26T10:59:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:33 smithi164 ceph-mon[36970]: pgmap v8304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:33 smithi163 ceph-mon[36873]: pgmap v8304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:33 smithi042 ceph-mon[32775]: pgmap v8304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:35 smithi163 ceph-mon[36873]: pgmap v8305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:59:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:35 smithi042 ceph-mon[32775]: pgmap v8305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:59:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:35 smithi164 ceph-mon[36970]: pgmap v8305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:59:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:37 smithi163 ceph-mon[36873]: pgmap v8306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:59:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14606.454612 secs 2024-07-26T10:59:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14624.193223 secs 2024-07-26T10:59:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:37 smithi042 ceph-mon[32775]: pgmap v8306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:59:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14606.454612 secs 2024-07-26T10:59:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14624.193223 secs 2024-07-26T10:59:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:37 smithi164 ceph-mon[36970]: pgmap v8306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:59:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14606.454612 secs 2024-07-26T10:59:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14624.193223 secs 2024-07-26T10:59:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:39 smithi163 ceph-mon[36873]: pgmap v8307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:59:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:39 smithi042 ceph-mon[32775]: pgmap v8307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:59:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:39 smithi164 ceph-mon[36970]: pgmap v8307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:59:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:41 smithi163 ceph-mon[36873]: pgmap v8308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:59:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:41 smithi042 ceph-mon[32775]: pgmap v8308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:59:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:41 smithi164 ceph-mon[36970]: pgmap v8308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T10:59:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14611.454751 secs 2024-07-26T10:59:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14629.193292 secs 2024-07-26T10:59:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14611.454751 secs 2024-07-26T10:59:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14629.193292 secs 2024-07-26T10:59:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14611.454751 secs 2024-07-26T10:59:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14629.193292 secs 2024-07-26T10:59:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:59:43.275+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:59:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:59:43.275+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:59:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:59:43.275+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:59:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:43 smithi163 ceph-mon[36873]: pgmap v8309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:59:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:43 smithi042 ceph-mon[32775]: pgmap v8309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:59:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:43 smithi164 ceph-mon[36970]: pgmap v8309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:59:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:45 smithi163 ceph-mon[36873]: pgmap v8310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:59:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:45 smithi042 ceph-mon[32775]: pgmap v8310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:59:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:45 smithi164 ceph-mon[36970]: pgmap v8310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T10:59:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:47 smithi163 ceph-mon[36873]: pgmap v8311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:59:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14616.454881 secs 2024-07-26T10:59:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14634.193436 secs 2024-07-26T10:59:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:47 smithi042 ceph-mon[32775]: pgmap v8311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:59:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14616.454881 secs 2024-07-26T10:59:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14634.193436 secs 2024-07-26T10:59:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:47 smithi164 ceph-mon[36970]: pgmap v8311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T10:59:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14616.454881 secs 2024-07-26T10:59:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14634.193436 secs 2024-07-26T10:59:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:49 smithi163 ceph-mon[36873]: pgmap v8312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:49 smithi042 ceph-mon[32775]: pgmap v8312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:49 smithi164 ceph-mon[36970]: pgmap v8312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:51 smithi163 ceph-mon[36873]: pgmap v8313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:51 smithi042 ceph-mon[32775]: pgmap v8313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:51 smithi164 ceph-mon[36970]: pgmap v8313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14621.455006 secs 2024-07-26T10:59:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14639.193593 secs 2024-07-26T10:59:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14621.455006 secs 2024-07-26T10:59:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14639.193593 secs 2024-07-26T10:59:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14621.455006 secs 2024-07-26T10:59:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14639.193593 secs 2024-07-26T10:59:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:53 smithi042 ceph-mon[32775]: pgmap v8314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:53 smithi164 ceph-mon[36970]: pgmap v8314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:53 smithi163 ceph-mon[36873]: pgmap v8314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:55 smithi042 ceph-mon[32775]: pgmap v8315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:59:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:55 smithi164 ceph-mon[36970]: pgmap v8315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:59:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:55 smithi163 ceph-mon[36873]: pgmap v8315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T10:59:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:57 smithi042 ceph-mon[32775]: pgmap v8316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14626.455128 secs 2024-07-26T10:59:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14644.193660 secs 2024-07-26T10:59:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:57 smithi164 ceph-mon[36970]: pgmap v8316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14626.455128 secs 2024-07-26T10:59:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14644.193660 secs 2024-07-26T10:59:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:57 smithi163 ceph-mon[36873]: pgmap v8316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T10:59:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14626.455128 secs 2024-07-26T10:59:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14644.193660 secs 2024-07-26T10:59:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:59:58.276+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T10:59:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:59:58.276+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T10:59:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T10:59:58.276+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T10:59:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 10:59:59 smithi042 ceph-mon[32775]: pgmap v8317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T10:59:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 10:59:59 smithi164 ceph-mon[36970]: pgmap v8317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 10:59:59 smithi163 ceph-mon[36873]: pgmap v8317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:00:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:00:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:00:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:01 smithi042 ceph-mon[32775]: pgmap v8318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:01 smithi164 ceph-mon[36970]: pgmap v8318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:01 smithi163 ceph-mon[36873]: pgmap v8318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:02.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14631.455246 secs 2024-07-26T11:00:02.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14649.193783 secs 2024-07-26T11:00:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14631.455246 secs 2024-07-26T11:00:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14649.193783 secs 2024-07-26T11:00:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14631.455246 secs 2024-07-26T11:00:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14649.193783 secs 2024-07-26T11:00:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:03 smithi042 ceph-mon[32775]: pgmap v8319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:03 smithi164 ceph-mon[36970]: pgmap v8319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:03 smithi163 ceph-mon[36873]: pgmap v8319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:05 smithi042 ceph-mon[32775]: pgmap v8320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:00:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:05 smithi164 ceph-mon[36970]: pgmap v8320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:00:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:05 smithi163 ceph-mon[36873]: pgmap v8320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:00:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:07 smithi042 ceph-mon[32775]: pgmap v8321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14636.455376 secs 2024-07-26T11:00:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14654.193884 secs 2024-07-26T11:00:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:07 smithi164 ceph-mon[36970]: pgmap v8321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14636.455376 secs 2024-07-26T11:00:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14654.193884 secs 2024-07-26T11:00:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:07 smithi163 ceph-mon[36873]: pgmap v8321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14636.455376 secs 2024-07-26T11:00:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14654.193884 secs 2024-07-26T11:00:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:09 smithi042 ceph-mon[32775]: pgmap v8322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:09 smithi164 ceph-mon[36970]: pgmap v8322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:09 smithi163 ceph-mon[36873]: pgmap v8322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:11 smithi042 ceph-mon[32775]: pgmap v8323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:11 smithi164 ceph-mon[36970]: pgmap v8323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:11 smithi163 ceph-mon[36873]: pgmap v8323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14641.455517 secs 2024-07-26T11:00:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14659.194008 secs 2024-07-26T11:00:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14641.455517 secs 2024-07-26T11:00:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14659.194008 secs 2024-07-26T11:00:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14641.455517 secs 2024-07-26T11:00:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14659.194008 secs 2024-07-26T11:00:13.279 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:00:13.277+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:00:13.279 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:00:13.277+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:00:13.279 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:00:13.277+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:00:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:13 smithi042 ceph-mon[32775]: pgmap v8324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:13 smithi164 ceph-mon[36970]: pgmap v8324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:13 smithi163 ceph-mon[36873]: pgmap v8324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:15 smithi042 ceph-mon[32775]: pgmap v8325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:00:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:15 smithi164 ceph-mon[36970]: pgmap v8325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:00:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:15 smithi163 ceph-mon[36873]: pgmap v8325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:00:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:17 smithi042 ceph-mon[32775]: pgmap v8326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14646.455638 secs 2024-07-26T11:00:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14664.194129 secs 2024-07-26T11:00:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:17 smithi164 ceph-mon[36970]: pgmap v8326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14646.455638 secs 2024-07-26T11:00:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14664.194129 secs 2024-07-26T11:00:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:17 smithi163 ceph-mon[36873]: pgmap v8326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14646.455638 secs 2024-07-26T11:00:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14664.194129 secs 2024-07-26T11:00:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:19 smithi042 ceph-mon[32775]: pgmap v8327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:19 smithi164 ceph-mon[36970]: pgmap v8327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:19 smithi163 ceph-mon[36873]: pgmap v8327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:00:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:00:20.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:00:20.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:00:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:00:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:00:20.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:00:20.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:00:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:00:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:21.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:21.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:21.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:21.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:21.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:00:21.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:00:21.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:00:21.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:00:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:21 smithi042 ceph-mon[32775]: pgmap v8328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:21 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:00:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:21 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:00:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:21 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:00:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:21 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:00:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:21 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:00:21.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:21 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:00:21.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:21 smithi042 ceph-mon[32775]: pgmap v8329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:00:21.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:21 smithi042 ceph-mon[32775]: pgmap v8330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:00:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:21 smithi164 ceph-mon[36970]: pgmap v8328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:21 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:00:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:21 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:00:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:21 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:00:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:21 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:00:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:21 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:00:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:21 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:00:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:21 smithi164 ceph-mon[36970]: pgmap v8329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:00:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:21 smithi164 ceph-mon[36970]: pgmap v8330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:00:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:21 smithi163 ceph-mon[36873]: pgmap v8328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:21 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:00:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:21 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:00:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:21 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:00:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:21 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:00:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:21 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:00:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:21 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:00:22.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:21 smithi163 ceph-mon[36873]: pgmap v8329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:00:22.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:21 smithi163 ceph-mon[36873]: pgmap v8330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:00:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14651.455785 secs 2024-07-26T11:00:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14669.194279 secs 2024-07-26T11:00:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14651.455785 secs 2024-07-26T11:00:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14669.194279 secs 2024-07-26T11:00:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14651.455785 secs 2024-07-26T11:00:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14669.194279 secs 2024-07-26T11:00:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:23 smithi042 ceph-mon[32775]: pgmap v8331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:00:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:23 smithi164 ceph-mon[36970]: pgmap v8331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:00:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:23 smithi163 ceph-mon[36873]: pgmap v8331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:00:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:25 smithi042 ceph-mon[32775]: pgmap v8332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:00:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:25 smithi164 ceph-mon[36970]: pgmap v8332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:00:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:25 smithi163 ceph-mon[36873]: pgmap v8332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:00:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14656.455909 secs 2024-07-26T11:00:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14674.194401 secs 2024-07-26T11:00:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:27 smithi042 ceph-mon[32775]: pgmap v8333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:00:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14656.455909 secs 2024-07-26T11:00:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14674.194401 secs 2024-07-26T11:00:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:27 smithi164 ceph-mon[36970]: pgmap v8333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:00:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14656.455909 secs 2024-07-26T11:00:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14674.194401 secs 2024-07-26T11:00:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:27 smithi163 ceph-mon[36873]: pgmap v8333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:00:28.282 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:00:28.280+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:00:28.282 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:00:28.280+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:00:28.282 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:00:28.280+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:00:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:29 smithi042 ceph-mon[32775]: pgmap v8334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:00:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:29 smithi164 ceph-mon[36970]: pgmap v8334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:00:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:29 smithi163 ceph-mon[36873]: pgmap v8334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:00:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:31 smithi042 ceph-mon[32775]: pgmap v8335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:00:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:31 smithi164 ceph-mon[36970]: pgmap v8335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:00:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:31 smithi163 ceph-mon[36873]: pgmap v8335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:00:32.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14661.456032 secs 2024-07-26T11:00:32.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14679.194526 secs 2024-07-26T11:00:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14661.456032 secs 2024-07-26T11:00:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14679.194526 secs 2024-07-26T11:00:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14661.456032 secs 2024-07-26T11:00:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14679.194526 secs 2024-07-26T11:00:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:33 smithi042 ceph-mon[32775]: pgmap v8336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:33 smithi164 ceph-mon[36970]: pgmap v8336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:33 smithi163 ceph-mon[36873]: pgmap v8336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:35 smithi042 ceph-mon[32775]: pgmap v8337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:35 smithi164 ceph-mon[36970]: pgmap v8337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:35 smithi163 ceph-mon[36873]: pgmap v8337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:36 smithi164 ceph-mon[36970]: pgmap v8338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:36 smithi163 ceph-mon[36873]: pgmap v8338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:36 smithi042 ceph-mon[32775]: pgmap v8338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14666.456153 secs 2024-07-26T11:00:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14684.194648 secs 2024-07-26T11:00:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14666.456153 secs 2024-07-26T11:00:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14684.194648 secs 2024-07-26T11:00:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14666.456153 secs 2024-07-26T11:00:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14684.194648 secs 2024-07-26T11:00:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:38 smithi164 ceph-mon[36970]: pgmap v8339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:00:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:38 smithi163 ceph-mon[36873]: pgmap v8339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:00:39.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:38 smithi042 ceph-mon[32775]: pgmap v8339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:00:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:41 smithi164 ceph-mon[36970]: pgmap v8340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:41 smithi163 ceph-mon[36873]: pgmap v8340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:41 smithi042 ceph-mon[32775]: pgmap v8340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14671.456301 secs 2024-07-26T11:00:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14689.194770 secs 2024-07-26T11:00:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14671.456301 secs 2024-07-26T11:00:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14689.194770 secs 2024-07-26T11:00:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14671.456301 secs 2024-07-26T11:00:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14689.194770 secs 2024-07-26T11:00:43.283 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:00:43.281+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:00:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:00:43.281+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:00:43.284 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:00:43.281+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:00:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:43 smithi164 ceph-mon[36970]: pgmap v8341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:43 smithi163 ceph-mon[36873]: pgmap v8341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:43 smithi042 ceph-mon[32775]: pgmap v8341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:45 smithi164 ceph-mon[36970]: pgmap v8342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:45.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:45 smithi163 ceph-mon[36873]: pgmap v8342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:45 smithi042 ceph-mon[32775]: pgmap v8342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14676.456428 secs 2024-07-26T11:00:47.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14694.194898 secs 2024-07-26T11:00:47.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:47 smithi164 ceph-mon[36970]: pgmap v8343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14676.456428 secs 2024-07-26T11:00:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14694.194898 secs 2024-07-26T11:00:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:47 smithi163 ceph-mon[36873]: pgmap v8343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14676.456428 secs 2024-07-26T11:00:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14694.194898 secs 2024-07-26T11:00:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:47 smithi042 ceph-mon[32775]: pgmap v8343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:49 smithi164 ceph-mon[36970]: pgmap v8344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:00:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:49 smithi163 ceph-mon[36873]: pgmap v8344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:00:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:49 smithi042 ceph-mon[32775]: pgmap v8344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:00:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:51 smithi163 ceph-mon[36873]: pgmap v8345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:51 smithi042 ceph-mon[32775]: pgmap v8345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:51 smithi164 ceph-mon[36970]: pgmap v8345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14681.456555 secs 2024-07-26T11:00:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14699.194997 secs 2024-07-26T11:00:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14681.456555 secs 2024-07-26T11:00:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14699.194997 secs 2024-07-26T11:00:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14681.456555 secs 2024-07-26T11:00:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14699.194997 secs 2024-07-26T11:00:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:53 smithi163 ceph-mon[36873]: pgmap v8346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:53 smithi042 ceph-mon[32775]: pgmap v8346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:53 smithi164 ceph-mon[36970]: pgmap v8346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:55 smithi163 ceph-mon[36873]: pgmap v8347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:55 smithi042 ceph-mon[32775]: pgmap v8347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:55 smithi164 ceph-mon[36970]: pgmap v8347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:00:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14686.456699 secs 2024-07-26T11:00:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14704.195067 secs 2024-07-26T11:00:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:57 smithi163 ceph-mon[36873]: pgmap v8348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14686.456699 secs 2024-07-26T11:00:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14704.195067 secs 2024-07-26T11:00:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:57 smithi042 ceph-mon[32775]: pgmap v8348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14686.456699 secs 2024-07-26T11:00:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14704.195067 secs 2024-07-26T11:00:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:57 smithi164 ceph-mon[36970]: pgmap v8348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:00:58.288 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:00:58.286+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:00:58.288 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:00:58.286+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:00:58.289 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:00:58.286+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:00:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:00:59 smithi163 ceph-mon[36873]: pgmap v8349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:00:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:00:59 smithi042 ceph-mon[32775]: pgmap v8349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:00:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:00:59 smithi164 ceph-mon[36970]: pgmap v8349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:01:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:01 smithi163 ceph-mon[36873]: pgmap v8350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:01 smithi042 ceph-mon[32775]: pgmap v8350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:01 smithi164 ceph-mon[36970]: pgmap v8350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14691.456819 secs 2024-07-26T11:01:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14709.195189 secs 2024-07-26T11:01:02.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14691.456819 secs 2024-07-26T11:01:02.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14709.195189 secs 2024-07-26T11:01:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14691.456819 secs 2024-07-26T11:01:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14709.195189 secs 2024-07-26T11:01:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:03 smithi163 ceph-mon[36873]: pgmap v8351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:03 smithi042 ceph-mon[32775]: pgmap v8351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:03 smithi164 ceph-mon[36970]: pgmap v8351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:05 smithi163 ceph-mon[36873]: pgmap v8352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:05 smithi042 ceph-mon[32775]: pgmap v8352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:05 smithi164 ceph-mon[36970]: pgmap v8352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14696.456941 secs 2024-07-26T11:01:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14714.195308 secs 2024-07-26T11:01:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:07 smithi042 ceph-mon[32775]: pgmap v8353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14696.456941 secs 2024-07-26T11:01:07.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14714.195308 secs 2024-07-26T11:01:07.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:07 smithi164 ceph-mon[36970]: pgmap v8353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14696.456941 secs 2024-07-26T11:01:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14714.195308 secs 2024-07-26T11:01:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:07 smithi163 ceph-mon[36873]: pgmap v8353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:09 smithi042 ceph-mon[32775]: pgmap v8354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:01:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:09 smithi164 ceph-mon[36970]: pgmap v8354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:01:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:09 smithi163 ceph-mon[36873]: pgmap v8354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:01:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:11 smithi042 ceph-mon[32775]: pgmap v8355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:11 smithi164 ceph-mon[36970]: pgmap v8355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:11 smithi163 ceph-mon[36873]: pgmap v8355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14701.457040 secs 2024-07-26T11:01:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14719.195429 secs 2024-07-26T11:01:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14701.457040 secs 2024-07-26T11:01:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14719.195429 secs 2024-07-26T11:01:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14701.457040 secs 2024-07-26T11:01:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14719.195429 secs 2024-07-26T11:01:13.290 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:01:13.288+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:01:13.290 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:01:13.288+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:01:13.290 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:01:13.288+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:01:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:13 smithi042 ceph-mon[32775]: pgmap v8356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:13 smithi164 ceph-mon[36970]: pgmap v8356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:13 smithi163 ceph-mon[36873]: pgmap v8356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:15 smithi042 ceph-mon[32775]: pgmap v8357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:15 smithi164 ceph-mon[36970]: pgmap v8357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:15 smithi163 ceph-mon[36873]: pgmap v8357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:17.810 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14706.457162 secs 2024-07-26T11:01:17.811 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14724.195554 secs 2024-07-26T11:01:17.811 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:17 smithi163 ceph-mon[36873]: pgmap v8358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14706.457162 secs 2024-07-26T11:01:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14724.195554 secs 2024-07-26T11:01:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:17 smithi042 ceph-mon[32775]: pgmap v8358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14706.457162 secs 2024-07-26T11:01:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14724.195554 secs 2024-07-26T11:01:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:17 smithi164 ceph-mon[36970]: pgmap v8358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:19 smithi042 ceph-mon[32775]: pgmap v8359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:01:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:19 smithi164 ceph-mon[36970]: pgmap v8359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:01:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:19 smithi163 ceph-mon[36873]: pgmap v8359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:01:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:21 smithi042 ceph-mon[32775]: pgmap v8360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:01:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:01:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:01:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:21 smithi164 ceph-mon[36970]: pgmap v8360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:01:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:01:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:01:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:21 smithi163 ceph-mon[36873]: pgmap v8360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:01:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:01:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:01:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:22 smithi042 ceph-mon[32775]: pgmap v8361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:01:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:22 smithi042 ceph-mon[32775]: pgmap v8362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:01:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14711.457282 secs 2024-07-26T11:01:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14729.195701 secs 2024-07-26T11:01:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:22 smithi164 ceph-mon[36970]: pgmap v8361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:01:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:22 smithi164 ceph-mon[36970]: pgmap v8362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:01:22.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14711.457282 secs 2024-07-26T11:01:22.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14729.195701 secs 2024-07-26T11:01:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:22 smithi163 ceph-mon[36873]: pgmap v8361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:01:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:22 smithi163 ceph-mon[36873]: pgmap v8362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:01:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14711.457282 secs 2024-07-26T11:01:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14729.195701 secs 2024-07-26T11:01:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:24 smithi042 ceph-mon[32775]: pgmap v8363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:01:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:24 smithi164 ceph-mon[36970]: pgmap v8363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:01:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:24 smithi163 ceph-mon[36873]: pgmap v8363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:01:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:26 smithi042 ceph-mon[32775]: pgmap v8364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:01:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:26 smithi164 ceph-mon[36970]: pgmap v8364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:01:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:26 smithi163 ceph-mon[36873]: pgmap v8364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:01:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14716.457407 secs 2024-07-26T11:01:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14734.195842 secs 2024-07-26T11:01:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14716.457407 secs 2024-07-26T11:01:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14734.195842 secs 2024-07-26T11:01:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14716.457407 secs 2024-07-26T11:01:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14734.195842 secs 2024-07-26T11:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:01:28.293+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:01:28.293+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:01:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:01:28.293+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:01:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:28 smithi042 ceph-mon[32775]: pgmap v8365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:01:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:28 smithi164 ceph-mon[36970]: pgmap v8365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:01:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:28 smithi163 ceph-mon[36873]: pgmap v8365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:01:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:30 smithi042 ceph-mon[32775]: pgmap v8366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:01:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:30 smithi164 ceph-mon[36970]: pgmap v8366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:01:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:30 smithi163 ceph-mon[36873]: pgmap v8366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:01:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:32 smithi042 ceph-mon[32775]: pgmap v8367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:01:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14721.457518 secs 2024-07-26T11:01:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14739.195962 secs 2024-07-26T11:01:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:32 smithi164 ceph-mon[36970]: pgmap v8367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:01:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14721.457518 secs 2024-07-26T11:01:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14739.195962 secs 2024-07-26T11:01:33.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:32 smithi163 ceph-mon[36873]: pgmap v8367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:01:33.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14721.457518 secs 2024-07-26T11:01:33.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14739.195962 secs 2024-07-26T11:01:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:34 smithi042 ceph-mon[32775]: pgmap v8368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:34 smithi164 ceph-mon[36970]: pgmap v8368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:34 smithi163 ceph-mon[36873]: pgmap v8368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:36.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:36 smithi042 ceph-mon[32775]: pgmap v8369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:36 smithi164 ceph-mon[36970]: pgmap v8369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:36 smithi163 ceph-mon[36873]: pgmap v8369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14726.457640 secs 2024-07-26T11:01:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14744.196054 secs 2024-07-26T11:01:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14726.457640 secs 2024-07-26T11:01:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14744.196054 secs 2024-07-26T11:01:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14726.457640 secs 2024-07-26T11:01:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14744.196054 secs 2024-07-26T11:01:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:38 smithi042 ceph-mon[32775]: pgmap v8370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:38 smithi164 ceph-mon[36970]: pgmap v8370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:38 smithi163 ceph-mon[36873]: pgmap v8370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:40 smithi042 ceph-mon[32775]: pgmap v8371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:01:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:40 smithi164 ceph-mon[36970]: pgmap v8371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:01:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:40 smithi163 ceph-mon[36873]: pgmap v8371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:01:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:42 smithi042 ceph-mon[32775]: pgmap v8372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14731.457705 secs 2024-07-26T11:01:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14749.196176 secs 2024-07-26T11:01:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:42 smithi164 ceph-mon[36970]: pgmap v8372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14731.457705 secs 2024-07-26T11:01:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14749.196176 secs 2024-07-26T11:01:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:42 smithi163 ceph-mon[36873]: pgmap v8372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14731.457705 secs 2024-07-26T11:01:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14749.196176 secs 2024-07-26T11:01:43.296 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:01:43.294+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:01:43.297 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:01:43.294+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:01:43.297 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:01:43.294+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:01:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:44 smithi042 ceph-mon[32775]: pgmap v8373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:44 smithi164 ceph-mon[36970]: pgmap v8373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:44 smithi163 ceph-mon[36873]: pgmap v8373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:46 smithi042 ceph-mon[32775]: pgmap v8374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:46 smithi164 ceph-mon[36970]: pgmap v8374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:46 smithi163 ceph-mon[36873]: pgmap v8374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14736.457849 secs 2024-07-26T11:01:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14754.196287 secs 2024-07-26T11:01:47.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14736.457849 secs 2024-07-26T11:01:47.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14754.196287 secs 2024-07-26T11:01:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14736.457849 secs 2024-07-26T11:01:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14754.196287 secs 2024-07-26T11:01:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:48 smithi042 ceph-mon[32775]: pgmap v8375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:48 smithi164 ceph-mon[36970]: pgmap v8375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:48 smithi163 ceph-mon[36873]: pgmap v8375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:50 smithi042 ceph-mon[32775]: pgmap v8376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:01:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:50 smithi164 ceph-mon[36970]: pgmap v8376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:01:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:50 smithi163 ceph-mon[36873]: pgmap v8376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:01:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:52 smithi042 ceph-mon[32775]: pgmap v8377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14741.457987 secs 2024-07-26T11:01:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14759.196407 secs 2024-07-26T11:01:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:52 smithi164 ceph-mon[36970]: pgmap v8377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14741.457987 secs 2024-07-26T11:01:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14759.196407 secs 2024-07-26T11:01:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:52 smithi163 ceph-mon[36873]: pgmap v8377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14741.457987 secs 2024-07-26T11:01:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14759.196407 secs 2024-07-26T11:01:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:54 smithi042 ceph-mon[32775]: pgmap v8378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:54 smithi164 ceph-mon[36970]: pgmap v8378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:54 smithi163 ceph-mon[36873]: pgmap v8378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:56 smithi042 ceph-mon[32775]: pgmap v8379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:56 smithi164 ceph-mon[36970]: pgmap v8379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:56 smithi163 ceph-mon[36873]: pgmap v8379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:01:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14746.458108 secs 2024-07-26T11:01:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14764.196525 secs 2024-07-26T11:01:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14746.458108 secs 2024-07-26T11:01:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14764.196525 secs 2024-07-26T11:01:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14746.458108 secs 2024-07-26T11:01:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14764.196525 secs 2024-07-26T11:01:58.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:01:58.297+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:01:58.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:01:58.297+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:01:58.299 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:01:58.297+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:01:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:58 smithi042 ceph-mon[32775]: pgmap v8380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:58 smithi164 ceph-mon[36970]: pgmap v8380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:59.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:58 smithi163 ceph-mon[36873]: pgmap v8380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:01:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:01:59 smithi042 ceph-mon[32775]: pgmap v8381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:01:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:01:59 smithi164 ceph-mon[36970]: pgmap v8381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:02:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:01:59 smithi163 ceph-mon[36873]: pgmap v8381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:02:00.184 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:02:00.183+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:02:00.185 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:02:00.183+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:02:00.185 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:02:00.183+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:02:00.185 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:02:00.183+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:02:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:02 smithi042 ceph-mon[32775]: pgmap v8382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14751.458260 secs 2024-07-26T11:02:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:02 smithi164 ceph-mon[36970]: pgmap v8382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14751.458260 secs 2024-07-26T11:02:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:02 smithi163 ceph-mon[36873]: pgmap v8382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14751.458260 secs 2024-07-26T11:02:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:03 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14769.196594 secs 2024-07-26T11:02:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:03 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14769.196594 secs 2024-07-26T11:02:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:03 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14769.196594 secs 2024-07-26T11:02:04.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:04 smithi042 ceph-mon[32775]: pgmap v8383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:04 smithi164 ceph-mon[36970]: pgmap v8383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:04 smithi163 ceph-mon[36873]: pgmap v8383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:06 smithi042 ceph-mon[32775]: pgmap v8384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:06 smithi164 ceph-mon[36970]: pgmap v8384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:06 smithi163 ceph-mon[36873]: pgmap v8384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:07.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14756.458383 secs 2024-07-26T11:02:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14756.458383 secs 2024-07-26T11:02:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14756.458383 secs 2024-07-26T11:02:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:08 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14774.196712 secs 2024-07-26T11:02:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:08 smithi042 ceph-mon[32775]: pgmap v8385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:08 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14774.196712 secs 2024-07-26T11:02:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:08 smithi164 ceph-mon[36970]: pgmap v8385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:08 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14774.196712 secs 2024-07-26T11:02:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:08 smithi163 ceph-mon[36873]: pgmap v8385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:10.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:10 smithi042 ceph-mon[32775]: pgmap v8386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:02:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:10 smithi164 ceph-mon[36970]: pgmap v8386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:02:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:10 smithi163 ceph-mon[36873]: pgmap v8386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:02:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:12 smithi042 ceph-mon[32775]: pgmap v8387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14761.458504 secs 2024-07-26T11:02:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:12 smithi164 ceph-mon[36970]: pgmap v8387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14761.458504 secs 2024-07-26T11:02:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:12 smithi163 ceph-mon[36873]: pgmap v8387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14761.458504 secs 2024-07-26T11:02:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:02:13.299+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:02:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:02:13.299+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:02:13.301 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:02:13.299+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:02:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:13 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14779.196836 secs 2024-07-26T11:02:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:13 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14779.196836 secs 2024-07-26T11:02:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:13 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14779.196836 secs 2024-07-26T11:02:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:14 smithi042 ceph-mon[32775]: pgmap v8388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:14 smithi164 ceph-mon[36970]: pgmap v8388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:14.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:14 smithi163 ceph-mon[36873]: pgmap v8388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:16 smithi042 ceph-mon[32775]: pgmap v8389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:16 smithi164 ceph-mon[36970]: pgmap v8389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:16 smithi163 ceph-mon[36873]: pgmap v8389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14766.458587 secs 2024-07-26T11:02:17.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14766.458587 secs 2024-07-26T11:02:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14766.458587 secs 2024-07-26T11:02:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:18 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14784.196941 secs 2024-07-26T11:02:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:18 smithi042 ceph-mon[32775]: pgmap v8390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:18 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14784.196941 secs 2024-07-26T11:02:18.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:18 smithi164 ceph-mon[36970]: pgmap v8390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:18 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14784.196941 secs 2024-07-26T11:02:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:18 smithi163 ceph-mon[36873]: pgmap v8390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:20 smithi042 ceph-mon[32775]: pgmap v8391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:02:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:20 smithi164 ceph-mon[36970]: pgmap v8391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:02:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:20 smithi163 ceph-mon[36873]: pgmap v8391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:02:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:22 smithi042 ceph-mon[32775]: pgmap v8392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14771.458673 secs 2024-07-26T11:02:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:22 smithi042 ceph-mon[32775]: pgmap v8393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:02:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:22 smithi042 ceph-mon[32775]: pgmap v8394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:02:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:02:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:22 smithi164 ceph-mon[36970]: pgmap v8392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14771.458673 secs 2024-07-26T11:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:22 smithi164 ceph-mon[36970]: pgmap v8393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:22 smithi164 ceph-mon[36970]: pgmap v8394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:02:22.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:22 smithi163 ceph-mon[36873]: pgmap v8392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14771.458673 secs 2024-07-26T11:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:22.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:22 smithi163 ceph-mon[36873]: pgmap v8393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:02:22.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:22 smithi163 ceph-mon[36873]: pgmap v8394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:02:22.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:02:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:23 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14789.196978 secs 2024-07-26T11:02:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:23 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14789.196978 secs 2024-07-26T11:02:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:23 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14789.196978 secs 2024-07-26T11:02:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:24 smithi042 ceph-mon[32775]: pgmap v8395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:02:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:24 smithi164 ceph-mon[36970]: pgmap v8395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:02:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:24 smithi163 ceph-mon[36873]: pgmap v8395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:02:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:26 smithi042 ceph-mon[32775]: pgmap v8396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:02:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:26 smithi164 ceph-mon[36970]: pgmap v8396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:02:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:26 smithi163 ceph-mon[36873]: pgmap v8396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:02:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14776.458796 secs 2024-07-26T11:02:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14794.197121 secs 2024-07-26T11:02:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14776.458796 secs 2024-07-26T11:02:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14794.197121 secs 2024-07-26T11:02:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14776.458796 secs 2024-07-26T11:02:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14794.197121 secs 2024-07-26T11:02:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:02:28.305+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:02:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:02:28.305+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:02:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:02:28.305+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:02:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:28 smithi042 ceph-mon[32775]: pgmap v8397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:02:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:28 smithi164 ceph-mon[36970]: pgmap v8397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:02:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:28 smithi163 ceph-mon[36873]: pgmap v8397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:02:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:30 smithi042 ceph-mon[32775]: pgmap v8398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:02:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:30 smithi164 ceph-mon[36970]: pgmap v8398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:02:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:30 smithi163 ceph-mon[36873]: pgmap v8398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:02:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14781.458939 secs 2024-07-26T11:02:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14799.197234 secs 2024-07-26T11:02:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:32 smithi042 ceph-mon[32775]: pgmap v8399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:02:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14781.458939 secs 2024-07-26T11:02:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14799.197234 secs 2024-07-26T11:02:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:32 smithi164 ceph-mon[36970]: pgmap v8399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:02:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14781.458939 secs 2024-07-26T11:02:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14799.197234 secs 2024-07-26T11:02:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:32 smithi163 ceph-mon[36873]: pgmap v8399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:02:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:34 smithi042 ceph-mon[32775]: pgmap v8400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:02:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:34 smithi164 ceph-mon[36970]: pgmap v8400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:02:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:34 smithi163 ceph-mon[36873]: pgmap v8400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:02:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:36 smithi042 ceph-mon[32775]: pgmap v8401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:36 smithi164 ceph-mon[36970]: pgmap v8401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:36 smithi163 ceph-mon[36873]: pgmap v8401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14786.459081 secs 2024-07-26T11:02:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14804.197366 secs 2024-07-26T11:02:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14786.459081 secs 2024-07-26T11:02:37.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14804.197366 secs 2024-07-26T11:02:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14786.459081 secs 2024-07-26T11:02:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14804.197366 secs 2024-07-26T11:02:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:38 smithi042 ceph-mon[32775]: pgmap v8402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:38 smithi164 ceph-mon[36970]: pgmap v8402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:38 smithi163 ceph-mon[36873]: pgmap v8402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:40 smithi042 ceph-mon[32775]: pgmap v8403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:40 smithi164 ceph-mon[36970]: pgmap v8403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:40 smithi163 ceph-mon[36873]: pgmap v8403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14791.459179 secs 2024-07-26T11:02:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14809.197490 secs 2024-07-26T11:02:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:42 smithi042 ceph-mon[32775]: pgmap v8404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14791.459179 secs 2024-07-26T11:02:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14809.197490 secs 2024-07-26T11:02:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:42 smithi164 ceph-mon[36970]: pgmap v8404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14791.459179 secs 2024-07-26T11:02:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14809.197490 secs 2024-07-26T11:02:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:42 smithi163 ceph-mon[36873]: pgmap v8404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:02:43.306+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:02:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:02:43.306+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:02:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:02:43.306+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:02:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:44 smithi042 ceph-mon[32775]: pgmap v8405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:02:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:44 smithi164 ceph-mon[36970]: pgmap v8405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:02:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:44 smithi163 ceph-mon[36873]: pgmap v8405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:02:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:46 smithi042 ceph-mon[32775]: pgmap v8406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:46 smithi164 ceph-mon[36970]: pgmap v8406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:46 smithi163 ceph-mon[36873]: pgmap v8406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14796.459306 secs 2024-07-26T11:02:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14814.197611 secs 2024-07-26T11:02:47.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14796.459306 secs 2024-07-26T11:02:47.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14814.197611 secs 2024-07-26T11:02:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14796.459306 secs 2024-07-26T11:02:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14814.197611 secs 2024-07-26T11:02:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:48 smithi042 ceph-mon[32775]: pgmap v8407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:48 smithi164 ceph-mon[36970]: pgmap v8407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:48 smithi163 ceph-mon[36873]: pgmap v8407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:50 smithi042 ceph-mon[32775]: pgmap v8408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:50 smithi164 ceph-mon[36970]: pgmap v8408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:50 smithi163 ceph-mon[36873]: pgmap v8408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14801.459373 secs 2024-07-26T11:02:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14819.197731 secs 2024-07-26T11:02:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:52 smithi042 ceph-mon[32775]: pgmap v8409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14801.459373 secs 2024-07-26T11:02:52.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14819.197731 secs 2024-07-26T11:02:52.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:52 smithi164 ceph-mon[36970]: pgmap v8409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14801.459373 secs 2024-07-26T11:02:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14819.197731 secs 2024-07-26T11:02:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:52 smithi163 ceph-mon[36873]: pgmap v8409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:54 smithi042 ceph-mon[32775]: pgmap v8410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:02:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:54 smithi164 ceph-mon[36970]: pgmap v8410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:02:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:54 smithi163 ceph-mon[36873]: pgmap v8410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:02:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:56 smithi042 ceph-mon[32775]: pgmap v8411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:56 smithi164 ceph-mon[36970]: pgmap v8411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:56 smithi163 ceph-mon[36873]: pgmap v8411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:02:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14806.459476 secs 2024-07-26T11:02:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14824.197874 secs 2024-07-26T11:02:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14806.459476 secs 2024-07-26T11:02:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14824.197874 secs 2024-07-26T11:02:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14806.459476 secs 2024-07-26T11:02:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14824.197874 secs 2024-07-26T11:02:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:02:58.312+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:02:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:02:58.312+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:02:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:02:58.312+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:02:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:02:58 smithi042 ceph-mon[32775]: pgmap v8412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:02:58 smithi164 ceph-mon[36970]: pgmap v8412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:02:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:02:58 smithi163 ceph-mon[36873]: pgmap v8412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:00 smithi042 ceph-mon[32775]: pgmap v8413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:00 smithi164 ceph-mon[36970]: pgmap v8413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:00 smithi163 ceph-mon[36873]: pgmap v8413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14811.459599 secs 2024-07-26T11:03:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14829.198003 secs 2024-07-26T11:03:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:02 smithi042 ceph-mon[32775]: pgmap v8414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14811.459599 secs 2024-07-26T11:03:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14829.198003 secs 2024-07-26T11:03:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:02 smithi164 ceph-mon[36970]: pgmap v8414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14811.459599 secs 2024-07-26T11:03:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14829.198003 secs 2024-07-26T11:03:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:02 smithi163 ceph-mon[36873]: pgmap v8414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:04 smithi042 ceph-mon[32775]: pgmap v8415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:03:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:04 smithi164 ceph-mon[36970]: pgmap v8415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:03:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:04 smithi163 ceph-mon[36873]: pgmap v8415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:03:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:06 smithi042 ceph-mon[32775]: pgmap v8416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:06 smithi164 ceph-mon[36970]: pgmap v8416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:06 smithi163 ceph-mon[36873]: pgmap v8416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14816.459728 secs 2024-07-26T11:03:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14834.198067 secs 2024-07-26T11:03:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14816.459728 secs 2024-07-26T11:03:07.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14834.198067 secs 2024-07-26T11:03:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14816.459728 secs 2024-07-26T11:03:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14834.198067 secs 2024-07-26T11:03:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:08 smithi042 ceph-mon[32775]: pgmap v8417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:08 smithi164 ceph-mon[36970]: pgmap v8417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:08 smithi163 ceph-mon[36873]: pgmap v8417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:09 smithi042 ceph-mon[32775]: pgmap v8418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:09 smithi164 ceph-mon[36970]: pgmap v8418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:09 smithi163 ceph-mon[36873]: pgmap v8418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:12.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14821.459849 secs 2024-07-26T11:03:12.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14839.198187 secs 2024-07-26T11:03:12.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:12 smithi042 ceph-mon[32775]: pgmap v8419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14821.459849 secs 2024-07-26T11:03:12.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14839.198187 secs 2024-07-26T11:03:12.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:12 smithi164 ceph-mon[36970]: pgmap v8419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14821.459849 secs 2024-07-26T11:03:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14839.198187 secs 2024-07-26T11:03:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:12 smithi163 ceph-mon[36873]: pgmap v8419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:03:13.312+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:03:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:03:13.312+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:03:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:03:13.312+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:03:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:14 smithi042 ceph-mon[32775]: pgmap v8420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:03:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:14 smithi164 ceph-mon[36970]: pgmap v8420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:03:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:14 smithi163 ceph-mon[36873]: pgmap v8420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:03:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:16 smithi042 ceph-mon[32775]: pgmap v8421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:16 smithi164 ceph-mon[36970]: pgmap v8421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:16 smithi163 ceph-mon[36873]: pgmap v8421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14826.459949 secs 2024-07-26T11:03:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14844.198308 secs 2024-07-26T11:03:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14826.459949 secs 2024-07-26T11:03:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14844.198308 secs 2024-07-26T11:03:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14826.459949 secs 2024-07-26T11:03:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14844.198308 secs 2024-07-26T11:03:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:18 smithi042 ceph-mon[32775]: pgmap v8422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:18 smithi164 ceph-mon[36970]: pgmap v8422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:18 smithi163 ceph-mon[36873]: pgmap v8422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:20 smithi042 ceph-mon[32775]: pgmap v8423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:20 smithi164 ceph-mon[36970]: pgmap v8423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:20 smithi163 ceph-mon[36873]: pgmap v8423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14831.460102 secs 2024-07-26T11:03:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14849.198429 secs 2024-07-26T11:03:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:22 smithi042 ceph-mon[32775]: pgmap v8424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14831.460102 secs 2024-07-26T11:03:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14849.198429 secs 2024-07-26T11:03:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:22 smithi164 ceph-mon[36970]: pgmap v8424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14831.460102 secs 2024-07-26T11:03:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14849.198429 secs 2024-07-26T11:03:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:22 smithi163 ceph-mon[36873]: pgmap v8424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:24 smithi042 ceph-mon[32775]: pgmap v8425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:03:24.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:24.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:24.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:24 smithi164 ceph-mon[36970]: pgmap v8425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:03:24.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:24.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:24.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:24 smithi163 ceph-mon[36873]: pgmap v8425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:03:24.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:24.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:24.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:24.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:03:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:25 smithi163 ceph-mon[36873]: pgmap v8426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:03:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:25 smithi163 ceph-mon[36873]: pgmap v8427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:03:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:25 smithi042 ceph-mon[32775]: pgmap v8426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:03:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:25 smithi042 ceph-mon[32775]: pgmap v8427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:03:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:25 smithi164 ceph-mon[36970]: pgmap v8426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:03:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:25 smithi164 ceph-mon[36970]: pgmap v8427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:03:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14836.460243 secs 2024-07-26T11:03:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:27 smithi042 ceph-mon[32775]: pgmap v8428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:03:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14854.198555 secs 2024-07-26T11:03:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14836.460243 secs 2024-07-26T11:03:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:27 smithi164 ceph-mon[36970]: pgmap v8428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:03:27.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14854.198555 secs 2024-07-26T11:03:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14836.460243 secs 2024-07-26T11:03:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:27 smithi163 ceph-mon[36873]: pgmap v8428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:03:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14854.198555 secs 2024-07-26T11:03:28.320 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:03:28.318+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:03:28.320 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:03:28.318+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:03:28.320 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:03:28.318+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:03:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:29 smithi042 ceph-mon[32775]: pgmap v8429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:03:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:29 smithi164 ceph-mon[36970]: pgmap v8429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:03:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:29 smithi163 ceph-mon[36873]: pgmap v8429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:03:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:03:30.197+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:03:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:03:30.197+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:03:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:03:30.197+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:03:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:03:30.197+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:03:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:31 smithi042 ceph-mon[32775]: pgmap v8430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:03:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:31 smithi164 ceph-mon[36970]: pgmap v8430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:03:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:31 smithi163 ceph-mon[36873]: pgmap v8430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:03:32.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14841.460340 secs 2024-07-26T11:03:32.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14859.198696 secs 2024-07-26T11:03:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14841.460340 secs 2024-07-26T11:03:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14859.198696 secs 2024-07-26T11:03:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14841.460340 secs 2024-07-26T11:03:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14859.198696 secs 2024-07-26T11:03:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:33 smithi042 ceph-mon[32775]: pgmap v8431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T11:03:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:33 smithi164 ceph-mon[36970]: pgmap v8431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T11:03:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:33 smithi163 ceph-mon[36873]: pgmap v8431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T11:03:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:35 smithi042 ceph-mon[32775]: pgmap v8432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:03:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:35 smithi164 ceph-mon[36970]: pgmap v8432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:03:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:35 smithi163 ceph-mon[36873]: pgmap v8432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:03:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14846.460473 secs 2024-07-26T11:03:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:37 smithi042 ceph-mon[32775]: pgmap v8433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14864.198818 secs 2024-07-26T11:03:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14846.460473 secs 2024-07-26T11:03:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:37 smithi164 ceph-mon[36970]: pgmap v8433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14864.198818 secs 2024-07-26T11:03:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14846.460473 secs 2024-07-26T11:03:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:37 smithi163 ceph-mon[36873]: pgmap v8433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14864.198818 secs 2024-07-26T11:03:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:39 smithi042 ceph-mon[32775]: pgmap v8434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:39 smithi164 ceph-mon[36970]: pgmap v8434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:39 smithi163 ceph-mon[36873]: pgmap v8434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:41 smithi042 ceph-mon[32775]: pgmap v8435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:41 smithi164 ceph-mon[36970]: pgmap v8435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:41 smithi163 ceph-mon[36873]: pgmap v8435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14851.460594 secs 2024-07-26T11:03:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14869.198963 secs 2024-07-26T11:03:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14851.460594 secs 2024-07-26T11:03:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14869.198963 secs 2024-07-26T11:03:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14851.460594 secs 2024-07-26T11:03:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14869.198963 secs 2024-07-26T11:03:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:03:43.320+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:03:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:03:43.320+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:03:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:03:43.320+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:03:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:43 smithi042 ceph-mon[32775]: pgmap v8436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:43 smithi164 ceph-mon[36970]: pgmap v8436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:43 smithi163 ceph-mon[36873]: pgmap v8436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:45 smithi042 ceph-mon[32775]: pgmap v8437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:03:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:45 smithi164 ceph-mon[36970]: pgmap v8437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:03:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:45 smithi163 ceph-mon[36873]: pgmap v8437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:03:47.810 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14856.460717 secs 2024-07-26T11:03:47.810 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:47 smithi163 ceph-mon[36873]: pgmap v8438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:47.810 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14874.199090 secs 2024-07-26T11:03:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14856.460717 secs 2024-07-26T11:03:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:47 smithi042 ceph-mon[32775]: pgmap v8438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14874.199090 secs 2024-07-26T11:03:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14856.460717 secs 2024-07-26T11:03:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:47 smithi164 ceph-mon[36970]: pgmap v8438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:47.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14874.199090 secs 2024-07-26T11:03:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:49 smithi042 ceph-mon[32775]: pgmap v8439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:49 smithi164 ceph-mon[36970]: pgmap v8439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:49 smithi163 ceph-mon[36873]: pgmap v8439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:51 smithi042 ceph-mon[32775]: pgmap v8440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:51 smithi164 ceph-mon[36970]: pgmap v8440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:51 smithi163 ceph-mon[36873]: pgmap v8440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14861.460799 secs 2024-07-26T11:03:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14879.199226 secs 2024-07-26T11:03:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14861.460799 secs 2024-07-26T11:03:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14879.199226 secs 2024-07-26T11:03:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14861.460799 secs 2024-07-26T11:03:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14879.199226 secs 2024-07-26T11:03:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:53 smithi042 ceph-mon[32775]: pgmap v8441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:53 smithi164 ceph-mon[36970]: pgmap v8441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:53 smithi163 ceph-mon[36873]: pgmap v8441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:55 smithi042 ceph-mon[32775]: pgmap v8442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:03:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:55 smithi164 ceph-mon[36970]: pgmap v8442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:03:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:55 smithi163 ceph-mon[36873]: pgmap v8442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:03:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14866.460925 secs 2024-07-26T11:03:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:57 smithi042 ceph-mon[32775]: pgmap v8443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14884.199330 secs 2024-07-26T11:03:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14866.460925 secs 2024-07-26T11:03:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:57 smithi164 ceph-mon[36970]: pgmap v8443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14884.199330 secs 2024-07-26T11:03:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14866.460925 secs 2024-07-26T11:03:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:57 smithi163 ceph-mon[36873]: pgmap v8443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:03:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14884.199330 secs 2024-07-26T11:03:58.327 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:03:58.325+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:03:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:03:58.325+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:03:58.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:03:58.325+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:03:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:03:59 smithi042 ceph-mon[32775]: pgmap v8444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:03:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:03:59 smithi164 ceph-mon[36970]: pgmap v8444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:03:59 smithi163 ceph-mon[36873]: pgmap v8444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:01 smithi042 ceph-mon[32775]: pgmap v8445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:01 smithi164 ceph-mon[36970]: pgmap v8445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:01 smithi163 ceph-mon[36873]: pgmap v8445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:02.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14871.461046 secs 2024-07-26T11:04:02.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14889.199469 secs 2024-07-26T11:04:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14871.461046 secs 2024-07-26T11:04:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14889.199469 secs 2024-07-26T11:04:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14871.461046 secs 2024-07-26T11:04:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14889.199469 secs 2024-07-26T11:04:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:03 smithi042 ceph-mon[32775]: pgmap v8446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:03 smithi164 ceph-mon[36970]: pgmap v8446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:03 smithi163 ceph-mon[36873]: pgmap v8446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:05 smithi042 ceph-mon[32775]: pgmap v8447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:04:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:05 smithi164 ceph-mon[36970]: pgmap v8447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:04:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:05 smithi163 ceph-mon[36873]: pgmap v8447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:04:07.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14876.461166 secs 2024-07-26T11:04:07.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:07 smithi042 ceph-mon[32775]: pgmap v8448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:07.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14894.199607 secs 2024-07-26T11:04:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14876.461166 secs 2024-07-26T11:04:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:07 smithi164 ceph-mon[36970]: pgmap v8448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:07.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14894.199607 secs 2024-07-26T11:04:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14876.461166 secs 2024-07-26T11:04:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:07 smithi163 ceph-mon[36873]: pgmap v8448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14894.199607 secs 2024-07-26T11:04:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:09 smithi042 ceph-mon[32775]: pgmap v8449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:09 smithi164 ceph-mon[36970]: pgmap v8449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:10.081 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:09 smithi163 ceph-mon[36873]: pgmap v8449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:11 smithi042 ceph-mon[32775]: pgmap v8450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:11 smithi164 ceph-mon[36970]: pgmap v8450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:11 smithi163 ceph-mon[36873]: pgmap v8450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14881.461294 secs 2024-07-26T11:04:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14899.199726 secs 2024-07-26T11:04:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14881.461294 secs 2024-07-26T11:04:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14899.199726 secs 2024-07-26T11:04:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14881.461294 secs 2024-07-26T11:04:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14899.199726 secs 2024-07-26T11:04:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:04:13.326+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:04:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:04:13.326+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:04:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:04:13.326+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:04:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:13 smithi042 ceph-mon[32775]: pgmap v8451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:13 smithi164 ceph-mon[36970]: pgmap v8451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:13 smithi163 ceph-mon[36873]: pgmap v8451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:15 smithi042 ceph-mon[32775]: pgmap v8452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:04:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:15 smithi164 ceph-mon[36970]: pgmap v8452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:04:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:15 smithi163 ceph-mon[36873]: pgmap v8452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:04:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14886.461363 secs 2024-07-26T11:04:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:17 smithi042 ceph-mon[32775]: pgmap v8453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14904.199853 secs 2024-07-26T11:04:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14886.461363 secs 2024-07-26T11:04:17.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:17 smithi164 ceph-mon[36970]: pgmap v8453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:17.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14904.199853 secs 2024-07-26T11:04:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14886.461363 secs 2024-07-26T11:04:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:17 smithi163 ceph-mon[36873]: pgmap v8453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14904.199853 secs 2024-07-26T11:04:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:19 smithi042 ceph-mon[32775]: pgmap v8454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:19 smithi164 ceph-mon[36970]: pgmap v8454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:19 smithi163 ceph-mon[36873]: pgmap v8454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:21 smithi042 ceph-mon[32775]: pgmap v8455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:21 smithi164 ceph-mon[36970]: pgmap v8455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:21 smithi163 ceph-mon[36873]: pgmap v8455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14891.461470 secs 2024-07-26T11:04:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14909.200009 secs 2024-07-26T11:04:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14891.461470 secs 2024-07-26T11:04:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14909.200009 secs 2024-07-26T11:04:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14891.461470 secs 2024-07-26T11:04:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14909.200009 secs 2024-07-26T11:04:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:23 smithi042 ceph-mon[32775]: pgmap v8456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:23 smithi164 ceph-mon[36970]: pgmap v8456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:23 smithi163 ceph-mon[36873]: pgmap v8456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:24 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:24.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:24.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:24.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:24.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:24 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:24 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:25 smithi042 ceph-mon[32775]: pgmap v8457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:04:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:25 smithi164 ceph-mon[36970]: pgmap v8457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:04:25.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:25.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:25.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:25 smithi163 ceph-mon[36873]: pgmap v8457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:04:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:04:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:26 smithi042 ceph-mon[32775]: pgmap v8458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:04:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:26 smithi042 ceph-mon[32775]: pgmap v8459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:04:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:26 smithi164 ceph-mon[36970]: pgmap v8458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:04:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:26 smithi164 ceph-mon[36970]: pgmap v8459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:04:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:26 smithi163 ceph-mon[36873]: pgmap v8458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:04:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:26 smithi163 ceph-mon[36873]: pgmap v8459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:04:27.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14896.461571 secs 2024-07-26T11:04:27.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14914.200126 secs 2024-07-26T11:04:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14896.461571 secs 2024-07-26T11:04:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14914.200126 secs 2024-07-26T11:04:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14896.461571 secs 2024-07-26T11:04:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14914.200126 secs 2024-07-26T11:04:28.333 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:04:28.332+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:04:28.334 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:04:28.332+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:04:28.334 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:04:28.332+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:04:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:28 smithi164 ceph-mon[36970]: pgmap v8460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:04:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:28 smithi163 ceph-mon[36873]: pgmap v8460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:04:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:28 smithi042 ceph-mon[32775]: pgmap v8460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:04:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:29 smithi164 ceph-mon[36970]: pgmap v8461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:04:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:29 smithi163 ceph-mon[36873]: pgmap v8461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:04:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:29 smithi042 ceph-mon[32775]: pgmap v8461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:04:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:31 smithi164 ceph-mon[36970]: pgmap v8462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:04:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14901.461716 secs 2024-07-26T11:04:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:31 smithi163 ceph-mon[36873]: pgmap v8462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:04:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14901.461716 secs 2024-07-26T11:04:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:31 smithi042 ceph-mon[32775]: pgmap v8462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:04:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14901.461716 secs 2024-07-26T11:04:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14919.200251 secs 2024-07-26T11:04:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14919.200251 secs 2024-07-26T11:04:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14919.200251 secs 2024-07-26T11:04:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:33 smithi164 ceph-mon[36970]: pgmap v8463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:04:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:33 smithi163 ceph-mon[36873]: pgmap v8463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:04:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:33 smithi042 ceph-mon[32775]: pgmap v8463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:04:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:35 smithi164 ceph-mon[36970]: pgmap v8464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:04:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:35 smithi163 ceph-mon[36873]: pgmap v8464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:04:36.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:35 smithi042 ceph-mon[32775]: pgmap v8464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:04:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14906.461796 secs 2024-07-26T11:04:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14906.461796 secs 2024-07-26T11:04:37.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14906.461796 secs 2024-07-26T11:04:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14924.200318 secs 2024-07-26T11:04:38.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:37 smithi164 ceph-mon[36970]: pgmap v8465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14924.200318 secs 2024-07-26T11:04:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:37 smithi163 ceph-mon[36873]: pgmap v8465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14924.200318 secs 2024-07-26T11:04:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:37 smithi042 ceph-mon[32775]: pgmap v8465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:39 smithi164 ceph-mon[36970]: pgmap v8466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:04:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:39 smithi163 ceph-mon[36873]: pgmap v8466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:04:40.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:39 smithi042 ceph-mon[32775]: pgmap v8466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:04:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:41 smithi164 ceph-mon[36970]: pgmap v8467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14911.461918 secs 2024-07-26T11:04:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:41 smithi163 ceph-mon[36873]: pgmap v8467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14911.461918 secs 2024-07-26T11:04:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:41 smithi042 ceph-mon[32775]: pgmap v8467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14911.461918 secs 2024-07-26T11:04:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14929.200463 secs 2024-07-26T11:04:43.335 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:04:43.333+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:04:43.335 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:04:43.333+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:04:43.335 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:04:43.333+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:04:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14929.200463 secs 2024-07-26T11:04:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14929.200463 secs 2024-07-26T11:04:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:43 smithi163 ceph-mon[36873]: pgmap v8468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:43 smithi042 ceph-mon[32775]: pgmap v8468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:43 smithi164 ceph-mon[36970]: pgmap v8468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:45 smithi163 ceph-mon[36873]: pgmap v8469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:45 smithi042 ceph-mon[32775]: pgmap v8469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:45 smithi164 ceph-mon[36970]: pgmap v8469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14916.462044 secs 2024-07-26T11:04:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14916.462044 secs 2024-07-26T11:04:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14916.462044 secs 2024-07-26T11:04:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14934.200572 secs 2024-07-26T11:04:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:47 smithi163 ceph-mon[36873]: pgmap v8470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14934.200572 secs 2024-07-26T11:04:48.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:47 smithi042 ceph-mon[32775]: pgmap v8470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14934.200572 secs 2024-07-26T11:04:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:47 smithi164 ceph-mon[36970]: pgmap v8470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:49 smithi163 ceph-mon[36873]: pgmap v8471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:04:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:49 smithi042 ceph-mon[32775]: pgmap v8471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:04:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:49 smithi164 ceph-mon[36970]: pgmap v8471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:04:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:51 smithi163 ceph-mon[36873]: pgmap v8472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14921.462163 secs 2024-07-26T11:04:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:51 smithi042 ceph-mon[32775]: pgmap v8472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14921.462163 secs 2024-07-26T11:04:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:51 smithi164 ceph-mon[36970]: pgmap v8472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14921.462163 secs 2024-07-26T11:04:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:53 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14939.200693 secs 2024-07-26T11:04:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14939.200693 secs 2024-07-26T11:04:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:53 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14939.200693 secs 2024-07-26T11:04:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:54 smithi163 ceph-mon[36873]: pgmap v8473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:54 smithi042 ceph-mon[32775]: pgmap v8473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:54 smithi164 ceph-mon[36970]: pgmap v8473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:56 smithi163 ceph-mon[36873]: pgmap v8474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:56 smithi042 ceph-mon[32775]: pgmap v8474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:56 smithi164 ceph-mon[36970]: pgmap v8474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:04:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14926.462231 secs 2024-07-26T11:04:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14926.462231 secs 2024-07-26T11:04:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14926.462231 secs 2024-07-26T11:04:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:04:58.339+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:04:58.340 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:04:58.339+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:04:58.341 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:04:58.339+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:04:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:58 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14944.200817 secs 2024-07-26T11:04:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:04:58 smithi042 ceph-mon[32775]: pgmap v8475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:58 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14944.200817 secs 2024-07-26T11:04:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:04:58 smithi164 ceph-mon[36970]: pgmap v8475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:04:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:58 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14944.200817 secs 2024-07-26T11:04:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:04:58 smithi163 ceph-mon[36873]: pgmap v8475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:05:00.212+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:05:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:05:00.212+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:05:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:05:00.212+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:05:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:05:00.212+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:05:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:00 smithi042 ceph-mon[32775]: pgmap v8476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:05:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:00 smithi164 ceph-mon[36970]: pgmap v8476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:05:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:00 smithi163 ceph-mon[36873]: pgmap v8476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:05:02.276 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:02 smithi164 ceph-mon[36970]: pgmap v8477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:02.276 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14931.462353 secs 2024-07-26T11:05:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:02 smithi042 ceph-mon[32775]: pgmap v8477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14931.462353 secs 2024-07-26T11:05:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:02 smithi163 ceph-mon[36873]: pgmap v8477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14931.462353 secs 2024-07-26T11:05:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:03 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14949.200957 secs 2024-07-26T11:05:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:03 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14949.200957 secs 2024-07-26T11:05:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:03 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14949.200957 secs 2024-07-26T11:05:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:04 smithi042 ceph-mon[32775]: pgmap v8478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:04 smithi164 ceph-mon[36970]: pgmap v8478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:04 smithi163 ceph-mon[36873]: pgmap v8478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:06 smithi042 ceph-mon[32775]: pgmap v8479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:06 smithi164 ceph-mon[36970]: pgmap v8479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:06 smithi163 ceph-mon[36873]: pgmap v8479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:07.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14936.462473 secs 2024-07-26T11:05:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14936.462473 secs 2024-07-26T11:05:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14936.462473 secs 2024-07-26T11:05:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:08 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14954.201080 secs 2024-07-26T11:05:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:08 smithi042 ceph-mon[32775]: pgmap v8480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:08 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14954.201080 secs 2024-07-26T11:05:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:08 smithi164 ceph-mon[36970]: pgmap v8480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:08 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14954.201080 secs 2024-07-26T11:05:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:08 smithi163 ceph-mon[36873]: pgmap v8480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:10 smithi042 ceph-mon[32775]: pgmap v8481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:05:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:10 smithi164 ceph-mon[36970]: pgmap v8481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:05:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:10 smithi163 ceph-mon[36873]: pgmap v8481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:05:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:12 smithi042 ceph-mon[32775]: pgmap v8482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14941.462602 secs 2024-07-26T11:05:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:12 smithi164 ceph-mon[36970]: pgmap v8482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14941.462602 secs 2024-07-26T11:05:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:12 smithi163 ceph-mon[36873]: pgmap v8482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14941.462602 secs 2024-07-26T11:05:13.342 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:05:13.340+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:05:13.342 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:05:13.340+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:05:13.342 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:05:13.340+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:05:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:13 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14959.201154 secs 2024-07-26T11:05:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:13 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14959.201154 secs 2024-07-26T11:05:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:13 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14959.201154 secs 2024-07-26T11:05:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:14 smithi042 ceph-mon[32775]: pgmap v8483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:14 smithi164 ceph-mon[36970]: pgmap v8483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:14.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:14 smithi163 ceph-mon[36873]: pgmap v8483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:16 smithi042 ceph-mon[32775]: pgmap v8484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:16 smithi164 ceph-mon[36970]: pgmap v8484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:16 smithi163 ceph-mon[36873]: pgmap v8484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14946.462721 secs 2024-07-26T11:05:17.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14946.462721 secs 2024-07-26T11:05:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14946.462721 secs 2024-07-26T11:05:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:18 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14964.201278 secs 2024-07-26T11:05:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:18 smithi042 ceph-mon[32775]: pgmap v8485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:18 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14964.201278 secs 2024-07-26T11:05:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:18 smithi164 ceph-mon[36970]: pgmap v8485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:18 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14964.201278 secs 2024-07-26T11:05:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:18 smithi163 ceph-mon[36873]: pgmap v8485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:20 smithi042 ceph-mon[32775]: pgmap v8486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:05:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:20 smithi164 ceph-mon[36970]: pgmap v8486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:05:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:20 smithi163 ceph-mon[36873]: pgmap v8486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:05:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:22 smithi042 ceph-mon[32775]: pgmap v8487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14951.462866 secs 2024-07-26T11:05:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:22 smithi164 ceph-mon[36970]: pgmap v8487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14951.462866 secs 2024-07-26T11:05:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:22 smithi163 ceph-mon[36873]: pgmap v8487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14951.462866 secs 2024-07-26T11:05:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:23 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14969.201372 secs 2024-07-26T11:05:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:23 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14969.201372 secs 2024-07-26T11:05:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:23 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14969.201372 secs 2024-07-26T11:05:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:24 smithi042 ceph-mon[32775]: pgmap v8488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:24 smithi164 ceph-mon[36970]: pgmap v8488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:24 smithi163 ceph-mon[36873]: pgmap v8488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:26 smithi042 ceph-mon[32775]: pgmap v8489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:26 smithi042 ceph-mon[32775]: pgmap v8490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:05:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:26 smithi042 ceph-mon[32775]: pgmap v8491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:05:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:05:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:26 smithi164 ceph-mon[36970]: pgmap v8489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:26 smithi164 ceph-mon[36970]: pgmap v8490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:05:26.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:26 smithi164 ceph-mon[36970]: pgmap v8491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:05:26.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:05:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:26 smithi163 ceph-mon[36873]: pgmap v8489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:26 smithi163 ceph-mon[36873]: pgmap v8490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:05:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:26 smithi163 ceph-mon[36873]: pgmap v8491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:05:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:05:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14956.462984 secs 2024-07-26T11:05:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14956.462984 secs 2024-07-26T11:05:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14956.462984 secs 2024-07-26T11:05:28.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:05:28.345+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:05:28.348 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:05:28.345+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:05:28.348 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:05:28.345+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:05:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:28 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14974.201495 secs 2024-07-26T11:05:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:28 smithi042 ceph-mon[32775]: pgmap v8492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:05:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:28 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14974.201495 secs 2024-07-26T11:05:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:28 smithi164 ceph-mon[36970]: pgmap v8492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:05:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:28 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14974.201495 secs 2024-07-26T11:05:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:28 smithi163 ceph-mon[36873]: pgmap v8492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:05:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:30 smithi163 ceph-mon[36873]: pgmap v8493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:05:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:30 smithi042 ceph-mon[32775]: pgmap v8493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:05:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:30 smithi164 ceph-mon[36970]: pgmap v8493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:05:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14961.463105 secs 2024-07-26T11:05:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14979.201624 secs 2024-07-26T11:05:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:32 smithi163 ceph-mon[36873]: pgmap v8494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:05:32.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14961.463105 secs 2024-07-26T11:05:32.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14979.201624 secs 2024-07-26T11:05:32.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:32 smithi164 ceph-mon[36970]: pgmap v8494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:05:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14961.463105 secs 2024-07-26T11:05:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14979.201624 secs 2024-07-26T11:05:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:32 smithi042 ceph-mon[32775]: pgmap v8494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:05:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:34 smithi163 ceph-mon[36873]: pgmap v8495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:05:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:34 smithi042 ceph-mon[32775]: pgmap v8495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:05:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:34 smithi164 ceph-mon[36970]: pgmap v8495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:05:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:36 smithi042 ceph-mon[32775]: pgmap v8496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:05:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:36 smithi164 ceph-mon[36970]: pgmap v8496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:05:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:36 smithi163 ceph-mon[36873]: pgmap v8496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:05:37.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14966.463247 secs 2024-07-26T11:05:37.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14984.201746 secs 2024-07-26T11:05:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14966.463247 secs 2024-07-26T11:05:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14984.201746 secs 2024-07-26T11:05:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14966.463247 secs 2024-07-26T11:05:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14984.201746 secs 2024-07-26T11:05:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:38 smithi042 ceph-mon[32775]: pgmap v8497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:38 smithi164 ceph-mon[36970]: pgmap v8497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:38 smithi163 ceph-mon[36873]: pgmap v8497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:40 smithi042 ceph-mon[32775]: pgmap v8498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:40 smithi164 ceph-mon[36970]: pgmap v8498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:40 smithi163 ceph-mon[36873]: pgmap v8498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14971.463371 secs 2024-07-26T11:05:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14989.201888 secs 2024-07-26T11:05:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:42 smithi042 ceph-mon[32775]: pgmap v8499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14971.463371 secs 2024-07-26T11:05:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14989.201888 secs 2024-07-26T11:05:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:42 smithi164 ceph-mon[36970]: pgmap v8499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:43.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14971.463371 secs 2024-07-26T11:05:43.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14989.201888 secs 2024-07-26T11:05:43.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:42 smithi163 ceph-mon[36873]: pgmap v8499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:43.348 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:05:43.346+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:05:43.348 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:05:43.346+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:05:43.348 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:05:43.346+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:05:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:44 smithi042 ceph-mon[32775]: pgmap v8500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:05:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:44 smithi164 ceph-mon[36970]: pgmap v8500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:05:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:44 smithi163 ceph-mon[36873]: pgmap v8500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:05:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:46 smithi042 ceph-mon[32775]: pgmap v8501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:46 smithi164 ceph-mon[36970]: pgmap v8501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:46 smithi163 ceph-mon[36873]: pgmap v8501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14976.463487 secs 2024-07-26T11:05:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14994.202009 secs 2024-07-26T11:05:47.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14976.463487 secs 2024-07-26T11:05:47.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14994.202009 secs 2024-07-26T11:05:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14976.463487 secs 2024-07-26T11:05:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14994.202009 secs 2024-07-26T11:05:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:48 smithi042 ceph-mon[32775]: pgmap v8502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:48 smithi164 ceph-mon[36970]: pgmap v8502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:48 smithi163 ceph-mon[36873]: pgmap v8502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:50 smithi042 ceph-mon[32775]: pgmap v8503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:50 smithi164 ceph-mon[36970]: pgmap v8503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:50 smithi163 ceph-mon[36873]: pgmap v8503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14981.463604 secs 2024-07-26T11:05:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14999.202094 secs 2024-07-26T11:05:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:52 smithi042 ceph-mon[32775]: pgmap v8504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14981.463604 secs 2024-07-26T11:05:52.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14999.202094 secs 2024-07-26T11:05:52.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:52 smithi164 ceph-mon[36970]: pgmap v8504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14981.463604 secs 2024-07-26T11:05:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14999.202094 secs 2024-07-26T11:05:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:52 smithi163 ceph-mon[36873]: pgmap v8504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:54 smithi042 ceph-mon[32775]: pgmap v8505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:05:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:54 smithi164 ceph-mon[36970]: pgmap v8505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:05:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:54 smithi163 ceph-mon[36873]: pgmap v8505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:05:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:56 smithi042 ceph-mon[32775]: pgmap v8506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:56 smithi164 ceph-mon[36970]: pgmap v8506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:56 smithi163 ceph-mon[36873]: pgmap v8506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:05:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14986.463747 secs 2024-07-26T11:05:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15004.202218 secs 2024-07-26T11:05:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14986.463747 secs 2024-07-26T11:05:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15004.202218 secs 2024-07-26T11:05:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14986.463747 secs 2024-07-26T11:05:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15004.202218 secs 2024-07-26T11:05:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:05:58.350+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:05:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:05:58.350+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:05:58.352 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:05:58.350+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:05:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:05:58 smithi042 ceph-mon[32775]: pgmap v8507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:05:58 smithi164 ceph-mon[36970]: pgmap v8507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:05:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:05:58 smithi163 ceph-mon[36873]: pgmap v8507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:00 smithi042 ceph-mon[32775]: pgmap v8508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:00 smithi164 ceph-mon[36970]: pgmap v8508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:00 smithi163 ceph-mon[36873]: pgmap v8508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14991.463884 secs 2024-07-26T11:06:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15009.202341 secs 2024-07-26T11:06:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:02 smithi042 ceph-mon[32775]: pgmap v8509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14991.463884 secs 2024-07-26T11:06:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15009.202341 secs 2024-07-26T11:06:02.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:02 smithi164 ceph-mon[36970]: pgmap v8509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:03.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14991.463884 secs 2024-07-26T11:06:03.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15009.202341 secs 2024-07-26T11:06:03.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:02 smithi163 ceph-mon[36873]: pgmap v8509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:04 smithi042 ceph-mon[32775]: pgmap v8510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:06:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:04 smithi164 ceph-mon[36970]: pgmap v8510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:06:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:04 smithi163 ceph-mon[36873]: pgmap v8510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:06:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:06 smithi042 ceph-mon[32775]: pgmap v8511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:06 smithi164 ceph-mon[36970]: pgmap v8511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:06 smithi163 ceph-mon[36873]: pgmap v8511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 14996.464009 secs 2024-07-26T11:06:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15014.202405 secs 2024-07-26T11:06:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 14996.464009 secs 2024-07-26T11:06:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15014.202405 secs 2024-07-26T11:06:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 14996.464009 secs 2024-07-26T11:06:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15014.202405 secs 2024-07-26T11:06:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:08 smithi042 ceph-mon[32775]: pgmap v8512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:08 smithi164 ceph-mon[36970]: pgmap v8512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:08 smithi163 ceph-mon[36873]: pgmap v8512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:10 smithi042 ceph-mon[32775]: pgmap v8513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:10 smithi164 ceph-mon[36970]: pgmap v8513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:10 smithi163 ceph-mon[36873]: pgmap v8513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15001.464129 secs 2024-07-26T11:06:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15019.202472 secs 2024-07-26T11:06:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:12 smithi042 ceph-mon[32775]: pgmap v8514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15001.464129 secs 2024-07-26T11:06:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15019.202472 secs 2024-07-26T11:06:12.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:12 smithi164 ceph-mon[36970]: pgmap v8514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15001.464129 secs 2024-07-26T11:06:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15019.202472 secs 2024-07-26T11:06:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:12 smithi163 ceph-mon[36873]: pgmap v8514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:13.353 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:06:13.351+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:06:13.353 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:06:13.351+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:06:13.354 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:06:13.351+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:06:14.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:14 smithi042 ceph-mon[32775]: pgmap v8515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:06:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:14 smithi164 ceph-mon[36970]: pgmap v8515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:06:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:14 smithi163 ceph-mon[36873]: pgmap v8515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:06:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:16 smithi042 ceph-mon[32775]: pgmap v8516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:16 smithi164 ceph-mon[36970]: pgmap v8516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:16 smithi163 ceph-mon[36873]: pgmap v8516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15006.464250 secs 2024-07-26T11:06:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15024.202594 secs 2024-07-26T11:06:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15006.464250 secs 2024-07-26T11:06:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15024.202594 secs 2024-07-26T11:06:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15006.464250 secs 2024-07-26T11:06:17.983 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15024.202594 secs 2024-07-26T11:06:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:18 smithi042 ceph-mon[32775]: pgmap v8517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:18 smithi164 ceph-mon[36970]: pgmap v8517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:18 smithi163 ceph-mon[36873]: pgmap v8517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:20 smithi042 ceph-mon[32775]: pgmap v8518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:20 smithi164 ceph-mon[36970]: pgmap v8518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:20 smithi163 ceph-mon[36873]: pgmap v8518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15011.464374 secs 2024-07-26T11:06:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15029.202722 secs 2024-07-26T11:06:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:22 smithi042 ceph-mon[32775]: pgmap v8519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15011.464374 secs 2024-07-26T11:06:22.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15029.202722 secs 2024-07-26T11:06:22.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:22 smithi164 ceph-mon[36970]: pgmap v8519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15011.464374 secs 2024-07-26T11:06:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15029.202722 secs 2024-07-26T11:06:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:22 smithi163 ceph-mon[36873]: pgmap v8519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:24.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:24 smithi042 ceph-mon[32775]: pgmap v8520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:06:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:24 smithi164 ceph-mon[36970]: pgmap v8520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:06:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:24 smithi163 ceph-mon[36873]: pgmap v8520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:06:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:26 smithi042 ceph-mon[32775]: pgmap v8521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:06:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:26 smithi164 ceph-mon[36970]: pgmap v8521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:06:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:26 smithi163 ceph-mon[36873]: pgmap v8521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:06:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:27 smithi042 ceph-mon[32775]: pgmap v8522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:06:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:27 smithi042 ceph-mon[32775]: pgmap v8523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:06:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15016.464492 secs 2024-07-26T11:06:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15034.202796 secs 2024-07-26T11:06:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:27 smithi164 ceph-mon[36970]: pgmap v8522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:06:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:27 smithi164 ceph-mon[36970]: pgmap v8523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:06:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15016.464492 secs 2024-07-26T11:06:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15034.202796 secs 2024-07-26T11:06:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:27 smithi163 ceph-mon[36873]: pgmap v8522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:06:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:27 smithi163 ceph-mon[36873]: pgmap v8523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:06:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15016.464492 secs 2024-07-26T11:06:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15034.202796 secs 2024-07-26T11:06:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:06:28.354+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:06:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:06:28.354+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:06:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:06:28.354+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:06:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:29 smithi042 ceph-mon[32775]: pgmap v8524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:06:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:29 smithi164 ceph-mon[36970]: pgmap v8524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:06:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:29 smithi163 ceph-mon[36873]: pgmap v8524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:06:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:31 smithi042 ceph-mon[32775]: pgmap v8525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:06:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:31 smithi164 ceph-mon[36970]: pgmap v8525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:06:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:31 smithi163 ceph-mon[36873]: pgmap v8525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:06:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15021.464575 secs 2024-07-26T11:06:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15039.202862 secs 2024-07-26T11:06:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15021.464575 secs 2024-07-26T11:06:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15039.202862 secs 2024-07-26T11:06:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15021.464575 secs 2024-07-26T11:06:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15039.202862 secs 2024-07-26T11:06:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:33 smithi042 ceph-mon[32775]: pgmap v8526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:06:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:33 smithi164 ceph-mon[36970]: pgmap v8526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:06:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:33 smithi163 ceph-mon[36873]: pgmap v8526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:06:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:35 smithi042 ceph-mon[32775]: pgmap v8527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:06:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:35 smithi164 ceph-mon[36970]: pgmap v8527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:06:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:35 smithi163 ceph-mon[36873]: pgmap v8527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:06:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15026.464642 secs 2024-07-26T11:06:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:37 smithi042 ceph-mon[32775]: pgmap v8528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:06:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15044.203002 secs 2024-07-26T11:06:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15026.464642 secs 2024-07-26T11:06:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:37 smithi164 ceph-mon[36970]: pgmap v8528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:06:37.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15044.203002 secs 2024-07-26T11:06:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15026.464642 secs 2024-07-26T11:06:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:37 smithi163 ceph-mon[36873]: pgmap v8528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:06:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15044.203002 secs 2024-07-26T11:06:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:39 smithi164 ceph-mon[36970]: pgmap v8529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:39 smithi163 ceph-mon[36873]: pgmap v8529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:39 smithi042 ceph-mon[32775]: pgmap v8529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:41 smithi164 ceph-mon[36970]: pgmap v8530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:41 smithi163 ceph-mon[36873]: pgmap v8530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:41 smithi042 ceph-mon[32775]: pgmap v8530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15031.464762 secs 2024-07-26T11:06:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15049.203128 secs 2024-07-26T11:06:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15031.464762 secs 2024-07-26T11:06:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15049.203128 secs 2024-07-26T11:06:43.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15031.464762 secs 2024-07-26T11:06:43.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15049.203128 secs 2024-07-26T11:06:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:06:43.355+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:06:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:06:43.355+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:06:43.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:06:43.355+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:06:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:43 smithi164 ceph-mon[36970]: pgmap v8531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:43 smithi163 ceph-mon[36873]: pgmap v8531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:43 smithi042 ceph-mon[32775]: pgmap v8531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:45 smithi164 ceph-mon[36970]: pgmap v8532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:06:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:45 smithi163 ceph-mon[36873]: pgmap v8532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:06:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:45 smithi042 ceph-mon[32775]: pgmap v8532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:06:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15036.464903 secs 2024-07-26T11:06:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:47 smithi164 ceph-mon[36970]: pgmap v8533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15054.203210 secs 2024-07-26T11:06:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15036.464903 secs 2024-07-26T11:06:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:47 smithi163 ceph-mon[36873]: pgmap v8533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15054.203210 secs 2024-07-26T11:06:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15036.464903 secs 2024-07-26T11:06:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:47 smithi042 ceph-mon[32775]: pgmap v8533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15054.203210 secs 2024-07-26T11:06:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:49 smithi164 ceph-mon[36970]: pgmap v8534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:49 smithi163 ceph-mon[36873]: pgmap v8534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:49 smithi042 ceph-mon[32775]: pgmap v8534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:51 smithi164 ceph-mon[36970]: pgmap v8535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15059.203329 secs 2024-07-26T11:06:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:51 smithi163 ceph-mon[36873]: pgmap v8535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15059.203329 secs 2024-07-26T11:06:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:51 smithi042 ceph-mon[32775]: pgmap v8535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15059.203329 secs 2024-07-26T11:06:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15041.465035 secs 2024-07-26T11:06:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15041.465035 secs 2024-07-26T11:06:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15041.465035 secs 2024-07-26T11:06:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:53 smithi164 ceph-mon[36970]: pgmap v8536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:53 smithi163 ceph-mon[36873]: pgmap v8536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:53 smithi042 ceph-mon[32775]: pgmap v8536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:54 smithi164 ceph-mon[36970]: pgmap v8537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:06:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:54 smithi163 ceph-mon[36873]: pgmap v8537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:06:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:54 smithi042 ceph-mon[32775]: pgmap v8537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:06:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15046.465154 secs 2024-07-26T11:06:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:57 smithi042 ceph-mon[32775]: pgmap v8538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15064.203466 secs 2024-07-26T11:06:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15046.465154 secs 2024-07-26T11:06:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:57 smithi164 ceph-mon[36970]: pgmap v8538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15064.203466 secs 2024-07-26T11:06:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15046.465154 secs 2024-07-26T11:06:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:57 smithi163 ceph-mon[36873]: pgmap v8538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:06:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15064.203466 secs 2024-07-26T11:06:58.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:06:58.359+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:06:58.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:06:58.359+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:06:58.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:06:58.359+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:06:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:06:59 smithi042 ceph-mon[32775]: pgmap v8539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:06:59 smithi164 ceph-mon[36970]: pgmap v8539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:06:59.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:06:59 smithi163 ceph-mon[36873]: pgmap v8539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:01 smithi042 ceph-mon[32775]: pgmap v8540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:01.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:01 smithi164 ceph-mon[36970]: pgmap v8540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:01 smithi163 ceph-mon[36873]: pgmap v8540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:02.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15051.465279 secs 2024-07-26T11:07:02.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15069.203576 secs 2024-07-26T11:07:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15051.465279 secs 2024-07-26T11:07:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15069.203576 secs 2024-07-26T11:07:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15051.465279 secs 2024-07-26T11:07:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15069.203576 secs 2024-07-26T11:07:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:03 smithi042 ceph-mon[32775]: pgmap v8541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:03 smithi164 ceph-mon[36970]: pgmap v8541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:03 smithi163 ceph-mon[36873]: pgmap v8541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:05 smithi042 ceph-mon[32775]: pgmap v8542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:07:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:05 smithi164 ceph-mon[36970]: pgmap v8542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:07:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:05 smithi163 ceph-mon[36873]: pgmap v8542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:07:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15056.465385 secs 2024-07-26T11:07:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:07 smithi042 ceph-mon[32775]: pgmap v8543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15074.203730 secs 2024-07-26T11:07:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15056.465385 secs 2024-07-26T11:07:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:07 smithi164 ceph-mon[36970]: pgmap v8543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:07.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15074.203730 secs 2024-07-26T11:07:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15056.465385 secs 2024-07-26T11:07:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:07 smithi163 ceph-mon[36873]: pgmap v8543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15074.203730 secs 2024-07-26T11:07:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:09 smithi042 ceph-mon[32775]: pgmap v8544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:09 smithi164 ceph-mon[36970]: pgmap v8544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:09 smithi163 ceph-mon[36873]: pgmap v8544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:11 smithi042 ceph-mon[32775]: pgmap v8545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:11 smithi164 ceph-mon[36970]: pgmap v8545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:11 smithi163 ceph-mon[36873]: pgmap v8545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15061.465536 secs 2024-07-26T11:07:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15079.203808 secs 2024-07-26T11:07:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15061.465536 secs 2024-07-26T11:07:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15079.203808 secs 2024-07-26T11:07:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15061.465536 secs 2024-07-26T11:07:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15079.203808 secs 2024-07-26T11:07:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:07:13.360+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:07:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:07:13.360+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:07:13.362 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:07:13.360+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:07:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:13 smithi042 ceph-mon[32775]: pgmap v8546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:13 smithi164 ceph-mon[36970]: pgmap v8546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:13 smithi163 ceph-mon[36873]: pgmap v8546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:15 smithi042 ceph-mon[32775]: pgmap v8547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:07:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:15 smithi164 ceph-mon[36970]: pgmap v8547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:07:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:15 smithi163 ceph-mon[36873]: pgmap v8547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:07:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15066.465659 secs 2024-07-26T11:07:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:17 smithi042 ceph-mon[32775]: pgmap v8548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15084.203926 secs 2024-07-26T11:07:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15066.465659 secs 2024-07-26T11:07:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:17 smithi164 ceph-mon[36970]: pgmap v8548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15084.203926 secs 2024-07-26T11:07:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15066.465659 secs 2024-07-26T11:07:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:17 smithi163 ceph-mon[36873]: pgmap v8548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15084.203926 secs 2024-07-26T11:07:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:19 smithi042 ceph-mon[32775]: pgmap v8549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:19 smithi164 ceph-mon[36970]: pgmap v8549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:19 smithi163 ceph-mon[36873]: pgmap v8549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:21 smithi042 ceph-mon[32775]: pgmap v8550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:21 smithi164 ceph-mon[36970]: pgmap v8550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:21 smithi163 ceph-mon[36873]: pgmap v8550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15071.465797 secs 2024-07-26T11:07:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15089.204051 secs 2024-07-26T11:07:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15071.465797 secs 2024-07-26T11:07:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15089.204051 secs 2024-07-26T11:07:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15071.465797 secs 2024-07-26T11:07:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15089.204051 secs 2024-07-26T11:07:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:23 smithi042 ceph-mon[32775]: pgmap v8551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:23 smithi164 ceph-mon[36970]: pgmap v8551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:23 smithi163 ceph-mon[36873]: pgmap v8551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:25 smithi042 ceph-mon[32775]: pgmap v8552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:07:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:25 smithi164 ceph-mon[36970]: pgmap v8552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:07:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:25 smithi163 ceph-mon[36873]: pgmap v8552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:07:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15076.465918 secs 2024-07-26T11:07:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:27 smithi042 ceph-mon[32775]: pgmap v8553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15094.204145 secs 2024-07-26T11:07:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:27 smithi042 ceph-mon[32775]: pgmap v8554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:07:27.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:27 smithi042 ceph-mon[32775]: pgmap v8555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:07:27.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:07:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15076.465918 secs 2024-07-26T11:07:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:27 smithi164 ceph-mon[36970]: pgmap v8553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15094.204145 secs 2024-07-26T11:07:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:27 smithi164 ceph-mon[36970]: pgmap v8554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:07:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:27 smithi164 ceph-mon[36970]: pgmap v8555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:07:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:07:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15076.465918 secs 2024-07-26T11:07:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:27 smithi163 ceph-mon[36873]: pgmap v8553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15094.204145 secs 2024-07-26T11:07:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:27 smithi163 ceph-mon[36873]: pgmap v8554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:07:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:27 smithi163 ceph-mon[36873]: pgmap v8555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:07:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:07:28.367 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:07:28.365+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:07:28.368 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:07:28.365+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:07:28.368 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:07:28.365+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:07:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:29 smithi042 ceph-mon[32775]: pgmap v8556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:07:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:29 smithi164 ceph-mon[36970]: pgmap v8556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:07:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:29 smithi163 ceph-mon[36873]: pgmap v8556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:07:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:31 smithi042 ceph-mon[32775]: pgmap v8557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:07:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15099.204274 secs 2024-07-26T11:07:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:31 smithi164 ceph-mon[36970]: pgmap v8557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:07:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15099.204274 secs 2024-07-26T11:07:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:31 smithi163 ceph-mon[36873]: pgmap v8557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:07:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15099.204274 secs 2024-07-26T11:07:32.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15081.466065 secs 2024-07-26T11:07:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15081.466065 secs 2024-07-26T11:07:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15081.466065 secs 2024-07-26T11:07:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:33 smithi042 ceph-mon[32775]: pgmap v8558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:07:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:33 smithi164 ceph-mon[36970]: pgmap v8558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:07:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:33 smithi163 ceph-mon[36873]: pgmap v8558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:07:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:35 smithi042 ceph-mon[32775]: pgmap v8559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:07:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:35 smithi164 ceph-mon[36970]: pgmap v8559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:07:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:35 smithi163 ceph-mon[36873]: pgmap v8559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:07:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15104.204371 secs 2024-07-26T11:07:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15104.204371 secs 2024-07-26T11:07:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15104.204371 secs 2024-07-26T11:07:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15086.466189 secs 2024-07-26T11:07:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:37 smithi042 ceph-mon[32775]: pgmap v8560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:07:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15086.466189 secs 2024-07-26T11:07:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:37 smithi164 ceph-mon[36970]: pgmap v8560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:07:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15086.466189 secs 2024-07-26T11:07:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:37 smithi163 ceph-mon[36873]: pgmap v8560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:07:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:39 smithi042 ceph-mon[32775]: pgmap v8561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:07:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:39 smithi164 ceph-mon[36970]: pgmap v8561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:07:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:39 smithi163 ceph-mon[36873]: pgmap v8561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:07:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:41 smithi042 ceph-mon[32775]: pgmap v8562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15109.204498 secs 2024-07-26T11:07:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:41 smithi164 ceph-mon[36970]: pgmap v8562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15109.204498 secs 2024-07-26T11:07:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:41 smithi163 ceph-mon[36873]: pgmap v8562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15109.204498 secs 2024-07-26T11:07:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15091.466315 secs 2024-07-26T11:07:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15091.466315 secs 2024-07-26T11:07:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15091.466315 secs 2024-07-26T11:07:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:07:43.367+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:07:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:07:43.367+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:07:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:07:43.367+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:07:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:43 smithi164 ceph-mon[36970]: pgmap v8563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:43 smithi163 ceph-mon[36873]: pgmap v8563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:43 smithi042 ceph-mon[32775]: pgmap v8563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:45 smithi164 ceph-mon[36970]: pgmap v8564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:45 smithi163 ceph-mon[36873]: pgmap v8564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:45 smithi042 ceph-mon[32775]: pgmap v8564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15114.204637 secs 2024-07-26T11:07:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15114.204637 secs 2024-07-26T11:07:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15114.204637 secs 2024-07-26T11:07:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15096.466440 secs 2024-07-26T11:07:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:47 smithi164 ceph-mon[36970]: pgmap v8565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15096.466440 secs 2024-07-26T11:07:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:47 smithi163 ceph-mon[36873]: pgmap v8565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15096.466440 secs 2024-07-26T11:07:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:47 smithi042 ceph-mon[32775]: pgmap v8565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:49 smithi164 ceph-mon[36970]: pgmap v8566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:07:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:49 smithi163 ceph-mon[36873]: pgmap v8566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:07:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:49 smithi042 ceph-mon[32775]: pgmap v8566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:07:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:51 smithi164 ceph-mon[36970]: pgmap v8567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15119.204758 secs 2024-07-26T11:07:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:51 smithi163 ceph-mon[36873]: pgmap v8567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15119.204758 secs 2024-07-26T11:07:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:51 smithi042 ceph-mon[32775]: pgmap v8567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15119.204758 secs 2024-07-26T11:07:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15101.466569 secs 2024-07-26T11:07:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15101.466569 secs 2024-07-26T11:07:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15101.466569 secs 2024-07-26T11:07:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:53 smithi164 ceph-mon[36970]: pgmap v8568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:53 smithi163 ceph-mon[36873]: pgmap v8568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:53 smithi042 ceph-mon[32775]: pgmap v8568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:55 smithi164 ceph-mon[36970]: pgmap v8569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:55 smithi163 ceph-mon[36873]: pgmap v8569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:55 smithi042 ceph-mon[32775]: pgmap v8569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:07:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15124.204854 secs 2024-07-26T11:07:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15124.204854 secs 2024-07-26T11:07:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15124.204854 secs 2024-07-26T11:07:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15106.466649 secs 2024-07-26T11:07:57.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:57 smithi164 ceph-mon[36970]: pgmap v8570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15106.466649 secs 2024-07-26T11:07:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:57 smithi163 ceph-mon[36873]: pgmap v8570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15106.466649 secs 2024-07-26T11:07:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:57 smithi042 ceph-mon[32775]: pgmap v8570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:07:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:07:58.372+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:07:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:07:58.372+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:07:58.375 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:07:58.372+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:07:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:07:58 smithi164 ceph-mon[36970]: pgmap v8571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:07:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:07:58 smithi163 ceph-mon[36873]: pgmap v8571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:07:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:07:58 smithi042 ceph-mon[32775]: pgmap v8571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:08:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:08:00.241+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:08:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:08:00.241+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:08:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:08:00.241+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:08:00.243 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:08:00.241+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:08:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:01 smithi042 ceph-mon[32775]: pgmap v8572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15129.204960 secs 2024-07-26T11:08:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:01 smithi164 ceph-mon[36970]: pgmap v8572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15129.204960 secs 2024-07-26T11:08:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:01 smithi163 ceph-mon[36873]: pgmap v8572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15129.204960 secs 2024-07-26T11:08:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15111.466767 secs 2024-07-26T11:08:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15111.466767 secs 2024-07-26T11:08:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15111.466767 secs 2024-07-26T11:08:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:03 smithi042 ceph-mon[32775]: pgmap v8573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:03 smithi164 ceph-mon[36970]: pgmap v8573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:03 smithi163 ceph-mon[36873]: pgmap v8573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:05 smithi042 ceph-mon[32775]: pgmap v8574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:05 smithi164 ceph-mon[36970]: pgmap v8574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:05 smithi163 ceph-mon[36873]: pgmap v8574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15134.205006 secs 2024-07-26T11:08:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15134.205006 secs 2024-07-26T11:08:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15134.205006 secs 2024-07-26T11:08:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15116.466891 secs 2024-07-26T11:08:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:07 smithi042 ceph-mon[32775]: pgmap v8575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15116.466891 secs 2024-07-26T11:08:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:07 smithi164 ceph-mon[36970]: pgmap v8575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15116.466891 secs 2024-07-26T11:08:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:07 smithi163 ceph-mon[36873]: pgmap v8575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:09 smithi042 ceph-mon[32775]: pgmap v8576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:08:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:09 smithi164 ceph-mon[36970]: pgmap v8576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:08:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:09 smithi163 ceph-mon[36873]: pgmap v8576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:08:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:11 smithi164 ceph-mon[36970]: pgmap v8577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15139.205149 secs 2024-07-26T11:08:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:11 smithi163 ceph-mon[36873]: pgmap v8577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15139.205149 secs 2024-07-26T11:08:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:11 smithi042 ceph-mon[32775]: pgmap v8577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15139.205149 secs 2024-07-26T11:08:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15121.467016 secs 2024-07-26T11:08:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15121.467016 secs 2024-07-26T11:08:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15121.467016 secs 2024-07-26T11:08:13.375 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:08:13.373+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:08:13.375 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:08:13.373+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:08:13.375 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:08:13.373+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:08:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:13 smithi164 ceph-mon[36970]: pgmap v8578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:13 smithi163 ceph-mon[36873]: pgmap v8578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:13 smithi042 ceph-mon[32775]: pgmap v8578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:15 smithi164 ceph-mon[36970]: pgmap v8579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:15 smithi163 ceph-mon[36873]: pgmap v8579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:15 smithi042 ceph-mon[32775]: pgmap v8579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15144.205302 secs 2024-07-26T11:08:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15144.205302 secs 2024-07-26T11:08:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15144.205302 secs 2024-07-26T11:08:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15126.467142 secs 2024-07-26T11:08:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:17 smithi164 ceph-mon[36970]: pgmap v8580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15126.467142 secs 2024-07-26T11:08:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:17 smithi163 ceph-mon[36873]: pgmap v8580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15126.467142 secs 2024-07-26T11:08:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:17 smithi042 ceph-mon[32775]: pgmap v8580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:19 smithi164 ceph-mon[36970]: pgmap v8581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:08:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:19 smithi163 ceph-mon[36873]: pgmap v8581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:08:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:19 smithi042 ceph-mon[32775]: pgmap v8581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:08:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:21 smithi164 ceph-mon[36970]: pgmap v8582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15149.205446 secs 2024-07-26T11:08:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:21 smithi163 ceph-mon[36873]: pgmap v8582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15149.205446 secs 2024-07-26T11:08:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:21 smithi042 ceph-mon[32775]: pgmap v8582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15149.205446 secs 2024-07-26T11:08:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15131.467267 secs 2024-07-26T11:08:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15131.467267 secs 2024-07-26T11:08:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15131.467267 secs 2024-07-26T11:08:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:23 smithi164 ceph-mon[36970]: pgmap v8583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:23 smithi163 ceph-mon[36873]: pgmap v8583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:23 smithi042 ceph-mon[32775]: pgmap v8583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:25 smithi164 ceph-mon[36970]: pgmap v8584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:25 smithi163 ceph-mon[36873]: pgmap v8584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:25 smithi042 ceph-mon[32775]: pgmap v8584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15154.205566 secs 2024-07-26T11:08:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15154.205566 secs 2024-07-26T11:08:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15154.205566 secs 2024-07-26T11:08:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15136.467405 secs 2024-07-26T11:08:27.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:27 smithi164 ceph-mon[36970]: pgmap v8585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:27.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:27.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:27.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:27 smithi164 ceph-mon[36970]: pgmap v8586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:08:27.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:27 smithi164 ceph-mon[36970]: pgmap v8587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:08:27.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:08:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15136.467405 secs 2024-07-26T11:08:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:27 smithi163 ceph-mon[36873]: pgmap v8585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:28.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:27 smithi163 ceph-mon[36873]: pgmap v8586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:08:28.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:27 smithi163 ceph-mon[36873]: pgmap v8587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:08:28.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:08:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15136.467405 secs 2024-07-26T11:08:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:27 smithi042 ceph-mon[32775]: pgmap v8585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:27 smithi042 ceph-mon[32775]: pgmap v8586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:08:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:27 smithi042 ceph-mon[32775]: pgmap v8587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:08:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:08:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:08:28.379+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:08:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:08:28.379+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:08:28.380 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:08:28.379+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:08:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:30 smithi164 ceph-mon[36970]: pgmap v8588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:08:30.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:30 smithi163 ceph-mon[36873]: pgmap v8588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:08:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:30 smithi042 ceph-mon[32775]: pgmap v8588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:08:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15141.467455 secs 2024-07-26T11:08:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:32 smithi164 ceph-mon[36970]: pgmap v8589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:08:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15159.205688 secs 2024-07-26T11:08:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15141.467455 secs 2024-07-26T11:08:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:32 smithi163 ceph-mon[36873]: pgmap v8589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:08:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15159.205688 secs 2024-07-26T11:08:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15141.467455 secs 2024-07-26T11:08:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:32 smithi042 ceph-mon[32775]: pgmap v8589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:08:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15159.205688 secs 2024-07-26T11:08:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:34 smithi164 ceph-mon[36970]: pgmap v8590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:08:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:34 smithi163 ceph-mon[36873]: pgmap v8590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:08:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:34 smithi042 ceph-mon[32775]: pgmap v8590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:08:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:36 smithi164 ceph-mon[36970]: pgmap v8591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:08:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:36 smithi163 ceph-mon[36873]: pgmap v8591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:08:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:36 smithi042 ceph-mon[32775]: pgmap v8591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:08:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15146.467562 secs 2024-07-26T11:08:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15164.205793 secs 2024-07-26T11:08:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15146.467562 secs 2024-07-26T11:08:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15164.205793 secs 2024-07-26T11:08:37.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15146.467562 secs 2024-07-26T11:08:37.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15164.205793 secs 2024-07-26T11:08:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:38 smithi164 ceph-mon[36970]: pgmap v8592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:08:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:38 smithi163 ceph-mon[36873]: pgmap v8592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:08:38.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:38 smithi042 ceph-mon[32775]: pgmap v8592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:08:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:40 smithi164 ceph-mon[36970]: pgmap v8593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:40 smithi163 ceph-mon[36873]: pgmap v8593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:40 smithi042 ceph-mon[32775]: pgmap v8593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15151.467688 secs 2024-07-26T11:08:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:42 smithi164 ceph-mon[36970]: pgmap v8594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:42.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15169.205915 secs 2024-07-26T11:08:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15151.467688 secs 2024-07-26T11:08:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:42 smithi163 ceph-mon[36873]: pgmap v8594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15169.205915 secs 2024-07-26T11:08:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15151.467688 secs 2024-07-26T11:08:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:42 smithi042 ceph-mon[32775]: pgmap v8594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15169.205915 secs 2024-07-26T11:08:43.381 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:08:43.379+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:08:43.382 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:08:43.379+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:08:43.382 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:08:43.379+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:08:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:44 smithi164 ceph-mon[36970]: pgmap v8595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:44 smithi163 ceph-mon[36873]: pgmap v8595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:44 smithi042 ceph-mon[32775]: pgmap v8595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:46 smithi164 ceph-mon[36970]: pgmap v8596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:46 smithi163 ceph-mon[36873]: pgmap v8596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:46 smithi042 ceph-mon[32775]: pgmap v8596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15156.467803 secs 2024-07-26T11:08:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15174.206035 secs 2024-07-26T11:08:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15156.467803 secs 2024-07-26T11:08:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15174.206035 secs 2024-07-26T11:08:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15156.467803 secs 2024-07-26T11:08:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15174.206035 secs 2024-07-26T11:08:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:48 smithi164 ceph-mon[36970]: pgmap v8597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:48 smithi163 ceph-mon[36873]: pgmap v8597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:48 smithi042 ceph-mon[32775]: pgmap v8597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:50 smithi164 ceph-mon[36970]: pgmap v8598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:50 smithi163 ceph-mon[36873]: pgmap v8598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:50 smithi042 ceph-mon[32775]: pgmap v8598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15161.467929 secs 2024-07-26T11:08:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:52 smithi164 ceph-mon[36970]: pgmap v8599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15179.206158 secs 2024-07-26T11:08:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15161.467929 secs 2024-07-26T11:08:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:52 smithi163 ceph-mon[36873]: pgmap v8599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15179.206158 secs 2024-07-26T11:08:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15161.467929 secs 2024-07-26T11:08:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:52 smithi042 ceph-mon[32775]: pgmap v8599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15179.206158 secs 2024-07-26T11:08:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:54 smithi164 ceph-mon[36970]: pgmap v8600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:54 smithi163 ceph-mon[36873]: pgmap v8600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:54 smithi042 ceph-mon[32775]: pgmap v8600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:56 smithi164 ceph-mon[36970]: pgmap v8601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:56 smithi163 ceph-mon[36873]: pgmap v8601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:56 smithi042 ceph-mon[32775]: pgmap v8601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:08:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15166.468071 secs 2024-07-26T11:08:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15184.206286 secs 2024-07-26T11:08:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15166.468071 secs 2024-07-26T11:08:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15184.206286 secs 2024-07-26T11:08:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15166.468071 secs 2024-07-26T11:08:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15184.206286 secs 2024-07-26T11:08:58.386 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:08:58.384+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:08:58.386 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:08:58.384+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:08:58.386 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:08:58.384+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:08:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:08:58 smithi163 ceph-mon[36873]: pgmap v8602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:08:58 smithi042 ceph-mon[32775]: pgmap v8602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:08:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:08:58 smithi164 ceph-mon[36970]: pgmap v8602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:00 smithi163 ceph-mon[36873]: pgmap v8603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:00 smithi042 ceph-mon[32775]: pgmap v8603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:00 smithi164 ceph-mon[36970]: pgmap v8603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15171.468188 secs 2024-07-26T11:09:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:02 smithi163 ceph-mon[36873]: pgmap v8604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15189.206368 secs 2024-07-26T11:09:02.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15171.468188 secs 2024-07-26T11:09:02.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:02 smithi164 ceph-mon[36970]: pgmap v8604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:02.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15189.206368 secs 2024-07-26T11:09:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15171.468188 secs 2024-07-26T11:09:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:02 smithi042 ceph-mon[32775]: pgmap v8604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15189.206368 secs 2024-07-26T11:09:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:04 smithi163 ceph-mon[36873]: pgmap v8605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:04 smithi042 ceph-mon[32775]: pgmap v8605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:04 smithi164 ceph-mon[36970]: pgmap v8605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:06 smithi163 ceph-mon[36873]: pgmap v8606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:06 smithi042 ceph-mon[32775]: pgmap v8606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:06 smithi164 ceph-mon[36970]: pgmap v8606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15176.468311 secs 2024-07-26T11:09:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15194.206481 secs 2024-07-26T11:09:07.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15176.468311 secs 2024-07-26T11:09:07.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15194.206481 secs 2024-07-26T11:09:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15176.468311 secs 2024-07-26T11:09:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15194.206481 secs 2024-07-26T11:09:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:08 smithi163 ceph-mon[36873]: pgmap v8607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:08 smithi042 ceph-mon[32775]: pgmap v8607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:08 smithi164 ceph-mon[36970]: pgmap v8607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:10 smithi163 ceph-mon[36873]: pgmap v8608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:10 smithi042 ceph-mon[32775]: pgmap v8608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:10 smithi164 ceph-mon[36970]: pgmap v8608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15181.468409 secs 2024-07-26T11:09:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:12 smithi163 ceph-mon[36873]: pgmap v8609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:12.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15199.206609 secs 2024-07-26T11:09:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15181.468409 secs 2024-07-26T11:09:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:12 smithi042 ceph-mon[32775]: pgmap v8609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15199.206609 secs 2024-07-26T11:09:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15181.468409 secs 2024-07-26T11:09:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:12 smithi164 ceph-mon[36970]: pgmap v8609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15199.206609 secs 2024-07-26T11:09:13.388 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:09:13.386+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:09:13.388 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:09:13.386+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:09:13.388 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:09:13.386+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:09:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:14 smithi042 ceph-mon[32775]: pgmap v8610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:14 smithi164 ceph-mon[36970]: pgmap v8610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:14 smithi163 ceph-mon[36873]: pgmap v8610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:16 smithi042 ceph-mon[32775]: pgmap v8611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:16 smithi164 ceph-mon[36970]: pgmap v8611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:16 smithi163 ceph-mon[36873]: pgmap v8611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:17.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15186.468537 secs 2024-07-26T11:09:17.548 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15204.206732 secs 2024-07-26T11:09:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15186.468537 secs 2024-07-26T11:09:17.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15204.206732 secs 2024-07-26T11:09:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15186.468537 secs 2024-07-26T11:09:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15204.206732 secs 2024-07-26T11:09:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:18 smithi042 ceph-mon[32775]: pgmap v8612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:18 smithi164 ceph-mon[36970]: pgmap v8612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:18 smithi163 ceph-mon[36873]: pgmap v8612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:20 smithi042 ceph-mon[32775]: pgmap v8613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:20 smithi164 ceph-mon[36970]: pgmap v8613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:20 smithi163 ceph-mon[36873]: pgmap v8613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15191.468618 secs 2024-07-26T11:09:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:22 smithi042 ceph-mon[32775]: pgmap v8614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15209.206854 secs 2024-07-26T11:09:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15191.468618 secs 2024-07-26T11:09:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:22 smithi164 ceph-mon[36970]: pgmap v8614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15209.206854 secs 2024-07-26T11:09:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15191.468618 secs 2024-07-26T11:09:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:22 smithi163 ceph-mon[36873]: pgmap v8614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15209.206854 secs 2024-07-26T11:09:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:24 smithi042 ceph-mon[32775]: pgmap v8615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:24 smithi164 ceph-mon[36970]: pgmap v8615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:24 smithi163 ceph-mon[36873]: pgmap v8615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:26 smithi042 ceph-mon[32775]: pgmap v8616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:26 smithi164 ceph-mon[36970]: pgmap v8616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:26 smithi163 ceph-mon[36873]: pgmap v8616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15196.468762 secs 2024-07-26T11:09:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15214.206980 secs 2024-07-26T11:09:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15196.468762 secs 2024-07-26T11:09:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15214.206980 secs 2024-07-26T11:09:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15196.468762 secs 2024-07-26T11:09:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15214.206980 secs 2024-07-26T11:09:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:28.393 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:09:28.391+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:09:28.393 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:09:28.391+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:09:28.394 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:09:28.391+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:09:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:28 smithi042 ceph-mon[32775]: pgmap v8617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:28 smithi042 ceph-mon[32775]: pgmap v8618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:09:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:28 smithi042 ceph-mon[32775]: pgmap v8619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:09:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:09:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:28 smithi164 ceph-mon[36970]: pgmap v8617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:28.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:28.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:28 smithi164 ceph-mon[36970]: pgmap v8618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:09:28.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:28 smithi164 ceph-mon[36970]: pgmap v8619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:09:28.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:09:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:28 smithi163 ceph-mon[36873]: pgmap v8617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:28 smithi163 ceph-mon[36873]: pgmap v8618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:09:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:28 smithi163 ceph-mon[36873]: pgmap v8619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:09:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:09:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:29 smithi163 ceph-mon[36873]: pgmap v8620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:09:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:29 smithi042 ceph-mon[32775]: pgmap v8620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:09:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:29 smithi164 ceph-mon[36970]: pgmap v8620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:09:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15219.207105 secs 2024-07-26T11:09:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15219.207105 secs 2024-07-26T11:09:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15219.207105 secs 2024-07-26T11:09:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15201.468883 secs 2024-07-26T11:09:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:32 smithi164 ceph-mon[36970]: pgmap v8621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:09:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15201.468883 secs 2024-07-26T11:09:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:32 smithi163 ceph-mon[36873]: pgmap v8621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:09:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15201.468883 secs 2024-07-26T11:09:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:32 smithi042 ceph-mon[32775]: pgmap v8621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:09:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:33 smithi163 ceph-mon[36873]: pgmap v8622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:09:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:33 smithi042 ceph-mon[32775]: pgmap v8622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:09:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:33 smithi164 ceph-mon[36970]: pgmap v8622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:09:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:35 smithi163 ceph-mon[36873]: pgmap v8623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:09:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:35 smithi042 ceph-mon[32775]: pgmap v8623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:09:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:35 smithi164 ceph-mon[36970]: pgmap v8623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:09:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15224.207226 secs 2024-07-26T11:09:37.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15224.207226 secs 2024-07-26T11:09:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15224.207226 secs 2024-07-26T11:09:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15206.469025 secs 2024-07-26T11:09:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:37 smithi042 ceph-mon[32775]: pgmap v8624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:09:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15206.469025 secs 2024-07-26T11:09:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:37 smithi164 ceph-mon[36970]: pgmap v8624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:09:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15206.469025 secs 2024-07-26T11:09:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:37 smithi163 ceph-mon[36873]: pgmap v8624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:09:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:39 smithi163 ceph-mon[36873]: pgmap v8625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:39 smithi042 ceph-mon[32775]: pgmap v8625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:39 smithi164 ceph-mon[36970]: pgmap v8625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15229.207357 secs 2024-07-26T11:09:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15229.207357 secs 2024-07-26T11:09:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15229.207357 secs 2024-07-26T11:09:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15211.469172 secs 2024-07-26T11:09:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:42 smithi164 ceph-mon[36970]: pgmap v8626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15211.469172 secs 2024-07-26T11:09:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:42 smithi163 ceph-mon[36873]: pgmap v8626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15211.469172 secs 2024-07-26T11:09:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:42 smithi042 ceph-mon[32775]: pgmap v8626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:43.394 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:09:43.392+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:09:43.394 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:09:43.392+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:09:43.395 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:09:43.392+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:09:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:43 smithi163 ceph-mon[36873]: pgmap v8627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:09:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:43 smithi042 ceph-mon[32775]: pgmap v8627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:09:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:43 smithi164 ceph-mon[36970]: pgmap v8627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:09:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:45 smithi163 ceph-mon[36873]: pgmap v8628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:45 smithi042 ceph-mon[32775]: pgmap v8628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:45 smithi164 ceph-mon[36970]: pgmap v8628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15234.207480 secs 2024-07-26T11:09:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15234.207480 secs 2024-07-26T11:09:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15234.207480 secs 2024-07-26T11:09:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15216.469283 secs 2024-07-26T11:09:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:47 smithi042 ceph-mon[32775]: pgmap v8629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15216.469283 secs 2024-07-26T11:09:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:47 smithi164 ceph-mon[36970]: pgmap v8629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15216.469283 secs 2024-07-26T11:09:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:47 smithi163 ceph-mon[36873]: pgmap v8629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:49 smithi163 ceph-mon[36873]: pgmap v8630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:49 smithi042 ceph-mon[32775]: pgmap v8630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:49 smithi164 ceph-mon[36970]: pgmap v8630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15239.207607 secs 2024-07-26T11:09:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15239.207607 secs 2024-07-26T11:09:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15239.207607 secs 2024-07-26T11:09:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15221.469382 secs 2024-07-26T11:09:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:52 smithi164 ceph-mon[36970]: pgmap v8631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15221.469382 secs 2024-07-26T11:09:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:52 smithi163 ceph-mon[36873]: pgmap v8631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15221.469382 secs 2024-07-26T11:09:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:52 smithi042 ceph-mon[32775]: pgmap v8631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:53 smithi163 ceph-mon[36873]: pgmap v8632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:09:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:53 smithi042 ceph-mon[32775]: pgmap v8632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:09:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:53 smithi164 ceph-mon[36970]: pgmap v8632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:09:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:55 smithi163 ceph-mon[36873]: pgmap v8633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:55 smithi042 ceph-mon[32775]: pgmap v8633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:55 smithi164 ceph-mon[36970]: pgmap v8633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:09:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15244.207727 secs 2024-07-26T11:09:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15244.207727 secs 2024-07-26T11:09:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15244.207727 secs 2024-07-26T11:09:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15226.469509 secs 2024-07-26T11:09:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:57 smithi042 ceph-mon[32775]: pgmap v8634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15226.469509 secs 2024-07-26T11:09:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:57 smithi164 ceph-mon[36970]: pgmap v8634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15226.469509 secs 2024-07-26T11:09:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:57 smithi163 ceph-mon[36873]: pgmap v8634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:09:58.399 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:09:58.397+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:09:58.399 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:09:58.397+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:09:58.399 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:09:58.397+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:10:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:09:59 smithi163 ceph-mon[36873]: pgmap v8635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:09:59 smithi042 ceph-mon[32775]: pgmap v8635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:09:59 smithi164 ceph-mon[36970]: pgmap v8635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:10:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:10:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:10:02.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15249.207855 secs 2024-07-26T11:10:02.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:01 smithi164 ceph-mon[36970]: pgmap v8636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15249.207855 secs 2024-07-26T11:10:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:01 smithi042 ceph-mon[32775]: pgmap v8636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15249.207855 secs 2024-07-26T11:10:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:01 smithi163 ceph-mon[36873]: pgmap v8636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15231.469631 secs 2024-07-26T11:10:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15231.469631 secs 2024-07-26T11:10:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15231.469631 secs 2024-07-26T11:10:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:03 smithi042 ceph-mon[32775]: pgmap v8637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:10:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:03 smithi164 ceph-mon[36970]: pgmap v8637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:10:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:03 smithi163 ceph-mon[36873]: pgmap v8637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:10:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:05 smithi163 ceph-mon[36873]: pgmap v8638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:10:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:05 smithi042 ceph-mon[32775]: pgmap v8638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:10:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:05 smithi164 ceph-mon[36970]: pgmap v8638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:10:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15254.207940 secs 2024-07-26T11:10:07.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15254.207940 secs 2024-07-26T11:10:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15254.207940 secs 2024-07-26T11:10:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15236.469739 secs 2024-07-26T11:10:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:07 smithi042 ceph-mon[32775]: pgmap v8639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:10:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15236.469739 secs 2024-07-26T11:10:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:07 smithi164 ceph-mon[36970]: pgmap v8639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:10:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15236.469739 secs 2024-07-26T11:10:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:07 smithi163 ceph-mon[36873]: pgmap v8639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:10:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:09 smithi163 ceph-mon[36873]: pgmap v8640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:10:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:09 smithi042 ceph-mon[32775]: pgmap v8640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:10:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:09 smithi164 ceph-mon[36970]: pgmap v8640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:10:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15259.208004 secs 2024-07-26T11:10:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15259.208004 secs 2024-07-26T11:10:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15259.208004 secs 2024-07-26T11:10:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15241.469885 secs 2024-07-26T11:10:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:12 smithi164 ceph-mon[36970]: pgmap v8641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:10:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15241.469885 secs 2024-07-26T11:10:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:12 smithi163 ceph-mon[36873]: pgmap v8641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:10:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15241.469885 secs 2024-07-26T11:10:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:12 smithi042 ceph-mon[32775]: pgmap v8641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:10:13.399 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:13.398+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:10:13.400 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:13.398+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:10:13.400 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:13.398+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:10:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:13 smithi163 ceph-mon[36873]: pgmap v8642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:10:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:13 smithi042 ceph-mon[32775]: pgmap v8642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:10:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:13 smithi164 ceph-mon[36970]: pgmap v8642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:10:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:15 smithi163 ceph-mon[36873]: pgmap v8643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:10:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:15 smithi042 ceph-mon[32775]: pgmap v8643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:10:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:15 smithi164 ceph-mon[36970]: pgmap v8643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:10:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15264.208134 secs 2024-07-26T11:10:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15264.208134 secs 2024-07-26T11:10:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15264.208134 secs 2024-07-26T11:10:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15246.470008 secs 2024-07-26T11:10:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:17 smithi042 ceph-mon[32775]: pgmap v8644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15246.470008 secs 2024-07-26T11:10:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:17 smithi164 ceph-mon[36970]: pgmap v8644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15246.470008 secs 2024-07-26T11:10:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:17 smithi163 ceph-mon[36873]: pgmap v8644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:19 smithi163 ceph-mon[36873]: pgmap v8645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:19 smithi042 ceph-mon[32775]: pgmap v8645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:19 smithi164 ceph-mon[36970]: pgmap v8645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15269.208240 secs 2024-07-26T11:10:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15269.208240 secs 2024-07-26T11:10:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15269.208240 secs 2024-07-26T11:10:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15251.470129 secs 2024-07-26T11:10:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:22 smithi164 ceph-mon[36970]: pgmap v8646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15251.470129 secs 2024-07-26T11:10:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:22 smithi163 ceph-mon[36873]: pgmap v8646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15251.470129 secs 2024-07-26T11:10:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:22 smithi042 ceph-mon[32775]: pgmap v8646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:23 smithi163 ceph-mon[36873]: pgmap v8647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:10:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:23 smithi042 ceph-mon[32775]: pgmap v8647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:10:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:23 smithi164 ceph-mon[36970]: pgmap v8647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:10:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:25 smithi163 ceph-mon[36873]: pgmap v8648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:25 smithi042 ceph-mon[32775]: pgmap v8648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:25 smithi164 ceph-mon[36970]: pgmap v8648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15274.208324 secs 2024-07-26T11:10:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15274.208324 secs 2024-07-26T11:10:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15274.208324 secs 2024-07-26T11:10:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15256.470257 secs 2024-07-26T11:10:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:27 smithi042 ceph-mon[32775]: pgmap v8649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15256.470257 secs 2024-07-26T11:10:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:27 smithi164 ceph-mon[36970]: pgmap v8649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15256.470257 secs 2024-07-26T11:10:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:27 smithi163 ceph-mon[36873]: pgmap v8649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:28.404+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:10:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:28.404+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:10:28.406 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:28.404+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:10:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:10:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:10:29.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:10:29.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:10:29.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:10:29.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:10:29.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:10:29.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:29.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:29.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: pgmap v8650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:10:29.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: pgmap v8651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:10:29.148 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:10:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:10:29.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:10:29.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:10:29.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:10:29.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:10:29.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:10:29.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:10:29.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:29.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:29.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: pgmap v8650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:10:29.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: pgmap v8651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:10:29.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:10:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:10:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:10:29.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:10:29.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:10:29.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:10:29.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:10:29.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:10:29.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:10:29.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:29.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:29.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: pgmap v8650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:10:29.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: pgmap v8651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:10:29.260 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:10:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:31 smithi042 ceph-mon[32775]: pgmap v8652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:10:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:31 smithi164 ceph-mon[36970]: pgmap v8652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:10:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:31 smithi163 ceph-mon[36873]: pgmap v8652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:10:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15261.470315 secs 2024-07-26T11:10:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15279.208452 secs 2024-07-26T11:10:32.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15261.470315 secs 2024-07-26T11:10:32.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15279.208452 secs 2024-07-26T11:10:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15261.470315 secs 2024-07-26T11:10:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15279.208452 secs 2024-07-26T11:10:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:33 smithi042 ceph-mon[32775]: pgmap v8653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:10:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:33 smithi164 ceph-mon[36970]: pgmap v8653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:10:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:33 smithi163 ceph-mon[36873]: pgmap v8653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:10:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:35 smithi042 ceph-mon[32775]: pgmap v8654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:10:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:35 smithi164 ceph-mon[36970]: pgmap v8654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:10:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:35 smithi163 ceph-mon[36873]: pgmap v8654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:10:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15266.470443 secs 2024-07-26T11:10:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:37 smithi042 ceph-mon[32775]: pgmap v8655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:10:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15284.208593 secs 2024-07-26T11:10:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15266.470443 secs 2024-07-26T11:10:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:37 smithi164 ceph-mon[36970]: pgmap v8655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:10:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15284.208593 secs 2024-07-26T11:10:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15266.470443 secs 2024-07-26T11:10:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:37 smithi163 ceph-mon[36873]: pgmap v8655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:10:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15284.208593 secs 2024-07-26T11:10:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:39 smithi042 ceph-mon[32775]: pgmap v8656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:10:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:39 smithi164 ceph-mon[36970]: pgmap v8656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:10:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:39 smithi163 ceph-mon[36873]: pgmap v8656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:10:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:41 smithi042 ceph-mon[32775]: pgmap v8657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15289.208691 secs 2024-07-26T11:10:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:41 smithi164 ceph-mon[36970]: pgmap v8657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15289.208691 secs 2024-07-26T11:10:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:41 smithi163 ceph-mon[36873]: pgmap v8657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15289.208691 secs 2024-07-26T11:10:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15271.470564 secs 2024-07-26T11:10:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15271.470564 secs 2024-07-26T11:10:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15271.470564 secs 2024-07-26T11:10:43.406 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:43.404+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:10:43.406 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:43.404+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:10:43.406 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:43.404+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:10:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:43 smithi042 ceph-mon[32775]: pgmap v8658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:43 smithi164 ceph-mon[36970]: pgmap v8658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:43 smithi163 ceph-mon[36873]: pgmap v8658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:45 smithi042 ceph-mon[32775]: pgmap v8659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:45 smithi164 ceph-mon[36970]: pgmap v8659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:45 smithi163 ceph-mon[36873]: pgmap v8659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15294.208816 secs 2024-07-26T11:10:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15294.208816 secs 2024-07-26T11:10:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15294.208816 secs 2024-07-26T11:10:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15276.470693 secs 2024-07-26T11:10:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:47 smithi042 ceph-mon[32775]: pgmap v8660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:47.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15276.470693 secs 2024-07-26T11:10:47.910 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:47 smithi163 ceph-mon[36873]: pgmap v8660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15276.470693 secs 2024-07-26T11:10:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:47 smithi164 ceph-mon[36970]: pgmap v8660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:49 smithi042 ceph-mon[32775]: pgmap v8661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:49 smithi164 ceph-mon[36970]: pgmap v8661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:49 smithi163 ceph-mon[36873]: pgmap v8661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:51 smithi042 ceph-mon[32775]: pgmap v8662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15299.208967 secs 2024-07-26T11:10:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:51 smithi164 ceph-mon[36970]: pgmap v8662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15299.208967 secs 2024-07-26T11:10:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:51 smithi163 ceph-mon[36873]: pgmap v8662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15299.208967 secs 2024-07-26T11:10:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15281.470815 secs 2024-07-26T11:10:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15281.470815 secs 2024-07-26T11:10:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15281.470815 secs 2024-07-26T11:10:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:53 smithi042 ceph-mon[32775]: pgmap v8663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:53 smithi164 ceph-mon[36970]: pgmap v8663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:53 smithi163 ceph-mon[36873]: pgmap v8663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:55 smithi042 ceph-mon[32775]: pgmap v8664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:55 smithi164 ceph-mon[36970]: pgmap v8664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:55 smithi163 ceph-mon[36873]: pgmap v8664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:10:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15304.209085 secs 2024-07-26T11:10:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15304.209085 secs 2024-07-26T11:10:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15304.209085 secs 2024-07-26T11:10:57.886 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:57.884+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T11:10:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:57.884+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:10:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:57.884+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7ff678003990 tx=0x7ff67803e070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:10:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:57.884+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7ff678003990 tx=0x7ff67803e070 comp rx=0 tx=0).stop 2024-07-26T11:10:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:57.884+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5210 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T11:10:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:57.884+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:10:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:57.884+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5210 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:57.884+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:57.885+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:57.888 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:57.885+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff678031570 tx=0x7ff678055040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15286.470945 secs 2024-07-26T11:10:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:57 smithi042 ceph-mon[32775]: pgmap v8665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15286.470945 secs 2024-07-26T11:10:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:57 smithi164 ceph-mon[36970]: pgmap v8665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15286.470945 secs 2024-07-26T11:10:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:57 smithi163 ceph-mon[36873]: pgmap v8665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:58.406+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:10:58.408 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:58.406+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:10:58.408 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:10:58.406+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:10:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:10:59 smithi042 ceph-mon[32775]: pgmap v8666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:10:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:10:59 smithi164 ceph-mon[36970]: pgmap v8666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:10:59 smithi163 ceph-mon[36873]: pgmap v8666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:00.271 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:00.269+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:11:00.271 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:00.269+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:11:00.271 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:00.269+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:11:00.271 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:00.269+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:11:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:01 smithi042 ceph-mon[32775]: pgmap v8667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15309.209149 secs 2024-07-26T11:11:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:01 smithi164 ceph-mon[36970]: pgmap v8667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15309.209149 secs 2024-07-26T11:11:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:01 smithi163 ceph-mon[36873]: pgmap v8667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15309.209149 secs 2024-07-26T11:11:02.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15291.471066 secs 2024-07-26T11:11:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15291.471066 secs 2024-07-26T11:11:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15291.471066 secs 2024-07-26T11:11:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:03 smithi042 ceph-mon[32775]: pgmap v8668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:03 smithi164 ceph-mon[36970]: pgmap v8668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:03 smithi163 ceph-mon[36873]: pgmap v8668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:05 smithi042 ceph-mon[32775]: pgmap v8669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:05 smithi164 ceph-mon[36970]: pgmap v8669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:05 smithi163 ceph-mon[36873]: pgmap v8669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15314.209241 secs 2024-07-26T11:11:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15314.209241 secs 2024-07-26T11:11:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15314.209241 secs 2024-07-26T11:11:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15296.471192 secs 2024-07-26T11:11:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:07 smithi042 ceph-mon[32775]: pgmap v8670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15296.471192 secs 2024-07-26T11:11:07.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:07 smithi164 ceph-mon[36970]: pgmap v8670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15296.471192 secs 2024-07-26T11:11:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:07 smithi163 ceph-mon[36873]: pgmap v8670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:09 smithi042 ceph-mon[32775]: pgmap v8671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:09 smithi164 ceph-mon[36970]: pgmap v8671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:09 smithi163 ceph-mon[36873]: pgmap v8671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:11 smithi042 ceph-mon[32775]: pgmap v8672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15319.209322 secs 2024-07-26T11:11:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:11 smithi164 ceph-mon[36970]: pgmap v8672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15319.209322 secs 2024-07-26T11:11:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:11 smithi163 ceph-mon[36873]: pgmap v8672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15319.209322 secs 2024-07-26T11:11:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15301.471312 secs 2024-07-26T11:11:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15301.471312 secs 2024-07-26T11:11:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15301.471312 secs 2024-07-26T11:11:13.409 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:13.407+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:11:13.409 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:13.407+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:11:13.409 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:13.407+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:11:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:13 smithi042 ceph-mon[32775]: pgmap v8673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:13 smithi164 ceph-mon[36970]: pgmap v8673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:13 smithi163 ceph-mon[36873]: pgmap v8673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:15 smithi042 ceph-mon[32775]: pgmap v8674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:15 smithi164 ceph-mon[36970]: pgmap v8674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:15 smithi163 ceph-mon[36873]: pgmap v8674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15324.209467 secs 2024-07-26T11:11:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15324.209467 secs 2024-07-26T11:11:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15324.209467 secs 2024-07-26T11:11:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15306.471435 secs 2024-07-26T11:11:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:17 smithi042 ceph-mon[32775]: pgmap v8675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15306.471435 secs 2024-07-26T11:11:17.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:17 smithi163 ceph-mon[36873]: pgmap v8675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15306.471435 secs 2024-07-26T11:11:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:17 smithi164 ceph-mon[36970]: pgmap v8675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:19 smithi042 ceph-mon[32775]: pgmap v8676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:19 smithi164 ceph-mon[36970]: pgmap v8676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:19 smithi163 ceph-mon[36873]: pgmap v8676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:21 smithi042 ceph-mon[32775]: pgmap v8677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15329.209588 secs 2024-07-26T11:11:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:21 smithi164 ceph-mon[36970]: pgmap v8677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15329.209588 secs 2024-07-26T11:11:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:21 smithi163 ceph-mon[36873]: pgmap v8677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15329.209588 secs 2024-07-26T11:11:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15311.471556 secs 2024-07-26T11:11:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15311.471556 secs 2024-07-26T11:11:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15311.471556 secs 2024-07-26T11:11:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:23 smithi042 ceph-mon[32775]: pgmap v8678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:23 smithi164 ceph-mon[36970]: pgmap v8678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:23 smithi163 ceph-mon[36873]: pgmap v8678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:25 smithi042 ceph-mon[32775]: pgmap v8679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:25 smithi164 ceph-mon[36970]: pgmap v8679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:25 smithi163 ceph-mon[36873]: pgmap v8679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15334.209653 secs 2024-07-26T11:11:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15334.209653 secs 2024-07-26T11:11:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15334.209653 secs 2024-07-26T11:11:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15316.471641 secs 2024-07-26T11:11:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:27 smithi042 ceph-mon[32775]: pgmap v8680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15316.471641 secs 2024-07-26T11:11:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:27 smithi164 ceph-mon[36970]: pgmap v8680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15316.471641 secs 2024-07-26T11:11:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:27 smithi163 ceph-mon[36873]: pgmap v8680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:28.414 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:28.412+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:11:28.414 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:28.412+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:11:28.414 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:28.412+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:11:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:11:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:11:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:11:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:29 smithi164 ceph-mon[36970]: pgmap v8681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:11:29.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:11:29.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:11:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:29 smithi163 ceph-mon[36873]: pgmap v8681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:11:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:11:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:11:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:29 smithi042 ceph-mon[32775]: pgmap v8681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:11:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:11:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:11:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:30 smithi164 ceph-mon[36970]: pgmap v8682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:11:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:30 smithi164 ceph-mon[36970]: pgmap v8683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:11:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:30 smithi163 ceph-mon[36873]: pgmap v8682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:11:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:30 smithi163 ceph-mon[36873]: pgmap v8683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:11:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:30 smithi042 ceph-mon[32775]: pgmap v8682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:11:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:30 smithi042 ceph-mon[32775]: pgmap v8683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:11:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15339.209776 secs 2024-07-26T11:11:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15339.209776 secs 2024-07-26T11:11:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15339.209776 secs 2024-07-26T11:11:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:32 smithi164 ceph-mon[36970]: pgmap v8684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:11:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15321.471783 secs 2024-07-26T11:11:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:32 smithi163 ceph-mon[36873]: pgmap v8684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:11:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15321.471783 secs 2024-07-26T11:11:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:32 smithi042 ceph-mon[32775]: pgmap v8684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:11:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15321.471783 secs 2024-07-26T11:11:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:34 smithi164 ceph-mon[36970]: pgmap v8685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:11:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:34 smithi163 ceph-mon[36873]: pgmap v8685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:11:35.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:34 smithi042 ceph-mon[32775]: pgmap v8685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:11:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:36 smithi164 ceph-mon[36970]: pgmap v8686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:11:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15344.209896 secs 2024-07-26T11:11:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:36 smithi163 ceph-mon[36873]: pgmap v8686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:11:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15344.209896 secs 2024-07-26T11:11:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:36 smithi042 ceph-mon[32775]: pgmap v8686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:11:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15344.209896 secs 2024-07-26T11:11:37.969 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15326.471859 secs 2024-07-26T11:11:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15326.471859 secs 2024-07-26T11:11:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15326.471859 secs 2024-07-26T11:11:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:38 smithi164 ceph-mon[36970]: pgmap v8687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:11:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:38 smithi163 ceph-mon[36873]: pgmap v8687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:11:39.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:38 smithi042 ceph-mon[32775]: pgmap v8687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:11:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:40 smithi164 ceph-mon[36970]: pgmap v8688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:11:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:40 smithi163 ceph-mon[36873]: pgmap v8688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:11:41.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:40 smithi042 ceph-mon[32775]: pgmap v8688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:11:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15349.210041 secs 2024-07-26T11:11:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15349.210041 secs 2024-07-26T11:11:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15349.210041 secs 2024-07-26T11:11:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:42 smithi164 ceph-mon[36970]: pgmap v8689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:42.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15331.471982 secs 2024-07-26T11:11:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:42 smithi163 ceph-mon[36873]: pgmap v8689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15331.471982 secs 2024-07-26T11:11:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:42 smithi042 ceph-mon[32775]: pgmap v8689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15331.471982 secs 2024-07-26T11:11:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:43.413+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:11:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:43.413+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:11:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:43.413+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:11:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:44 smithi164 ceph-mon[36970]: pgmap v8690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:44 smithi163 ceph-mon[36873]: pgmap v8690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:44 smithi042 ceph-mon[32775]: pgmap v8690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:45.278 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:45.277+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:11:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:45.277+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:11:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:45.277+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:11:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:45.277+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:11:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:46 smithi164 ceph-mon[36970]: pgmap v8691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15354.210160 secs 2024-07-26T11:11:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:46 smithi163 ceph-mon[36873]: pgmap v8691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15354.210160 secs 2024-07-26T11:11:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:46 smithi042 ceph-mon[32775]: pgmap v8691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15354.210160 secs 2024-07-26T11:11:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15336.472105 secs 2024-07-26T11:11:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15336.472105 secs 2024-07-26T11:11:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15336.472105 secs 2024-07-26T11:11:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:48 smithi164 ceph-mon[36970]: pgmap v8692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:48 smithi163 ceph-mon[36873]: pgmap v8692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:48 smithi042 ceph-mon[32775]: pgmap v8692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:50 smithi164 ceph-mon[36970]: pgmap v8693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:11:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:50 smithi163 ceph-mon[36873]: pgmap v8693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:11:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:50 smithi042 ceph-mon[32775]: pgmap v8693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:11:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15359.210228 secs 2024-07-26T11:11:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15359.210228 secs 2024-07-26T11:11:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15359.210228 secs 2024-07-26T11:11:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:52 smithi164 ceph-mon[36970]: pgmap v8694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15341.472240 secs 2024-07-26T11:11:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:52 smithi163 ceph-mon[36873]: pgmap v8694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15341.472240 secs 2024-07-26T11:11:53.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:52 smithi042 ceph-mon[32775]: pgmap v8694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:53.048 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15341.472240 secs 2024-07-26T11:11:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:54 smithi164 ceph-mon[36970]: pgmap v8695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:54 smithi163 ceph-mon[36873]: pgmap v8695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:54 smithi042 ceph-mon[32775]: pgmap v8695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:56 smithi164 ceph-mon[36970]: pgmap v8696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15364.210353 secs 2024-07-26T11:11:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:56 smithi163 ceph-mon[36873]: pgmap v8696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15364.210353 secs 2024-07-26T11:11:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:56 smithi042 ceph-mon[32775]: pgmap v8696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:11:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15364.210353 secs 2024-07-26T11:11:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15346.472310 secs 2024-07-26T11:11:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:57 smithi163 ceph-mon[36873]: pgmap v8697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15346.472310 secs 2024-07-26T11:11:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:57 smithi042 ceph-mon[32775]: pgmap v8697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15346.472310 secs 2024-07-26T11:11:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:57 smithi164 ceph-mon[36970]: pgmap v8697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:11:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:58.419+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:11:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:58.419+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:11:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:11:58.419+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:12:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:11:59 smithi042 ceph-mon[32775]: pgmap v8698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:11:59 smithi164 ceph-mon[36970]: pgmap v8698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:11:59 smithi163 ceph-mon[36873]: pgmap v8698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:01 smithi042 ceph-mon[32775]: pgmap v8699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15369.210450 secs 2024-07-26T11:12:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:01 smithi164 ceph-mon[36970]: pgmap v8699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15369.210450 secs 2024-07-26T11:12:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:01 smithi163 ceph-mon[36873]: pgmap v8699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15369.210450 secs 2024-07-26T11:12:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15351.472434 secs 2024-07-26T11:12:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15351.472434 secs 2024-07-26T11:12:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15351.472434 secs 2024-07-26T11:12:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:03 smithi042 ceph-mon[32775]: pgmap v8700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:03 smithi164 ceph-mon[36970]: pgmap v8700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:03 smithi163 ceph-mon[36873]: pgmap v8700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:05 smithi042 ceph-mon[32775]: pgmap v8701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:05 smithi164 ceph-mon[36970]: pgmap v8701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:05 smithi163 ceph-mon[36873]: pgmap v8701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15374.210590 secs 2024-07-26T11:12:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15374.210590 secs 2024-07-26T11:12:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15374.210590 secs 2024-07-26T11:12:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15356.472550 secs 2024-07-26T11:12:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:07 smithi042 ceph-mon[32775]: pgmap v8702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15356.472550 secs 2024-07-26T11:12:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:07 smithi164 ceph-mon[36970]: pgmap v8702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15356.472550 secs 2024-07-26T11:12:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:07 smithi163 ceph-mon[36873]: pgmap v8702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:09 smithi042 ceph-mon[32775]: pgmap v8703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:09 smithi164 ceph-mon[36970]: pgmap v8703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:09 smithi163 ceph-mon[36873]: pgmap v8703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:11 smithi042 ceph-mon[32775]: pgmap v8704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15379.210718 secs 2024-07-26T11:12:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:11 smithi164 ceph-mon[36970]: pgmap v8704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15379.210718 secs 2024-07-26T11:12:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:11 smithi163 ceph-mon[36873]: pgmap v8704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15379.210718 secs 2024-07-26T11:12:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15361.472636 secs 2024-07-26T11:12:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15361.472636 secs 2024-07-26T11:12:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15361.472636 secs 2024-07-26T11:12:13.422 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:12:13.420+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:12:13.423 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:12:13.420+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:12:13.423 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:12:13.420+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:12:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:13 smithi042 ceph-mon[32775]: pgmap v8705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:13 smithi164 ceph-mon[36970]: pgmap v8705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:13 smithi163 ceph-mon[36873]: pgmap v8705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:15 smithi042 ceph-mon[32775]: pgmap v8706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:15 smithi164 ceph-mon[36970]: pgmap v8706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:15 smithi163 ceph-mon[36873]: pgmap v8706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15384.210808 secs 2024-07-26T11:12:17.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15384.210808 secs 2024-07-26T11:12:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15384.210808 secs 2024-07-26T11:12:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15366.472740 secs 2024-07-26T11:12:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:17 smithi164 ceph-mon[36970]: pgmap v8707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15366.472740 secs 2024-07-26T11:12:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:17 smithi163 ceph-mon[36873]: pgmap v8707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15366.472740 secs 2024-07-26T11:12:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:17 smithi042 ceph-mon[32775]: pgmap v8707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:19 smithi164 ceph-mon[36970]: pgmap v8708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:19 smithi163 ceph-mon[36873]: pgmap v8708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:19 smithi042 ceph-mon[32775]: pgmap v8708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:21 smithi164 ceph-mon[36970]: pgmap v8709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15389.210948 secs 2024-07-26T11:12:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:21 smithi163 ceph-mon[36873]: pgmap v8709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15389.210948 secs 2024-07-26T11:12:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:21 smithi042 ceph-mon[32775]: pgmap v8709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15389.210948 secs 2024-07-26T11:12:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15371.472899 secs 2024-07-26T11:12:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15371.472899 secs 2024-07-26T11:12:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15371.472899 secs 2024-07-26T11:12:24.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:23 smithi164 ceph-mon[36970]: pgmap v8710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:23 smithi163 ceph-mon[36873]: pgmap v8710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:24.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:23 smithi042 ceph-mon[32775]: pgmap v8710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:25 smithi164 ceph-mon[36970]: pgmap v8711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:25 smithi163 ceph-mon[36873]: pgmap v8711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:25 smithi042 ceph-mon[32775]: pgmap v8711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15394.211057 secs 2024-07-26T11:12:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15394.211057 secs 2024-07-26T11:12:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15394.211057 secs 2024-07-26T11:12:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15376.473024 secs 2024-07-26T11:12:28.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:27 smithi164 ceph-mon[36970]: pgmap v8712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15376.473024 secs 2024-07-26T11:12:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:27 smithi163 ceph-mon[36873]: pgmap v8712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15376.473024 secs 2024-07-26T11:12:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:27 smithi042 ceph-mon[32775]: pgmap v8712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:12:28.425+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:12:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:12:28.425+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:12:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:12:28.425+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:12:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:12:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:12:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:12:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:29 smithi164 ceph-mon[36970]: pgmap v8713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:30.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:12:30.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:12:30.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:29 smithi164 ceph-mon[36970]: pgmap v8714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:12:30.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:29 smithi164 ceph-mon[36970]: pgmap v8715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:12:30.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:12:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:29 smithi163 ceph-mon[36873]: pgmap v8713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:12:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:12:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:29 smithi163 ceph-mon[36873]: pgmap v8714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:12:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:29 smithi163 ceph-mon[36873]: pgmap v8715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:12:30.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:12:30.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:29 smithi042 ceph-mon[32775]: pgmap v8713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:30.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:12:30.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:12:30.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:29 smithi042 ceph-mon[32775]: pgmap v8714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:12:30.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:29 smithi042 ceph-mon[32775]: pgmap v8715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:12:30.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:12:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15381.473148 secs 2024-07-26T11:12:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:32 smithi042 ceph-mon[32775]: pgmap v8716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:12:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15399.211180 secs 2024-07-26T11:12:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15381.473148 secs 2024-07-26T11:12:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:32 smithi164 ceph-mon[36970]: pgmap v8716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:12:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15399.211180 secs 2024-07-26T11:12:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15381.473148 secs 2024-07-26T11:12:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:32 smithi163 ceph-mon[36873]: pgmap v8716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:12:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15399.211180 secs 2024-07-26T11:12:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:34 smithi042 ceph-mon[32775]: pgmap v8717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:12:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:34 smithi164 ceph-mon[36970]: pgmap v8717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:12:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:34 smithi163 ceph-mon[36873]: pgmap v8717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:12:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:36 smithi042 ceph-mon[32775]: pgmap v8718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:12:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15404.211323 secs 2024-07-26T11:12:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:36 smithi164 ceph-mon[36970]: pgmap v8718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:12:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15404.211323 secs 2024-07-26T11:12:37.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:36 smithi163 ceph-mon[36873]: pgmap v8718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:12:37.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15404.211323 secs 2024-07-26T11:12:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15386.473278 secs 2024-07-26T11:12:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15386.473278 secs 2024-07-26T11:12:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15386.473278 secs 2024-07-26T11:12:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:38 smithi042 ceph-mon[32775]: pgmap v8719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:12:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:38 smithi164 ceph-mon[36970]: pgmap v8719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:12:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:38 smithi163 ceph-mon[36873]: pgmap v8719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:12:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:40 smithi042 ceph-mon[32775]: pgmap v8720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:12:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:40 smithi164 ceph-mon[36970]: pgmap v8720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:12:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:40 smithi163 ceph-mon[36873]: pgmap v8720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:12:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15409.211449 secs 2024-07-26T11:12:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15409.211449 secs 2024-07-26T11:12:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15409.211449 secs 2024-07-26T11:12:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15391.473432 secs 2024-07-26T11:12:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:42 smithi042 ceph-mon[32775]: pgmap v8721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15391.473432 secs 2024-07-26T11:12:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:42 smithi164 ceph-mon[36970]: pgmap v8721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15391.473432 secs 2024-07-26T11:12:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:42 smithi163 ceph-mon[36873]: pgmap v8721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:43.428 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:12:43.426+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:12:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:12:43.426+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:12:43.429 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:12:43.426+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:12:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:44 smithi042 ceph-mon[32775]: pgmap v8722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:44 smithi164 ceph-mon[36970]: pgmap v8722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:44 smithi163 ceph-mon[36873]: pgmap v8722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:46 smithi042 ceph-mon[32775]: pgmap v8723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15414.211592 secs 2024-07-26T11:12:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:46 smithi164 ceph-mon[36970]: pgmap v8723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15414.211592 secs 2024-07-26T11:12:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:46 smithi163 ceph-mon[36873]: pgmap v8723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15414.211592 secs 2024-07-26T11:12:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15396.473558 secs 2024-07-26T11:12:47.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15396.473558 secs 2024-07-26T11:12:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15396.473558 secs 2024-07-26T11:12:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:48 smithi042 ceph-mon[32775]: pgmap v8724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:48 smithi164 ceph-mon[36970]: pgmap v8724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:48 smithi163 ceph-mon[36873]: pgmap v8724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:50 smithi042 ceph-mon[32775]: pgmap v8725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:50 smithi164 ceph-mon[36970]: pgmap v8725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:50 smithi163 ceph-mon[36873]: pgmap v8725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15419.211737 secs 2024-07-26T11:12:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15419.211737 secs 2024-07-26T11:12:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15419.211737 secs 2024-07-26T11:12:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15401.473659 secs 2024-07-26T11:12:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:52 smithi042 ceph-mon[32775]: pgmap v8726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15401.473659 secs 2024-07-26T11:12:52.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:52 smithi164 ceph-mon[36970]: pgmap v8726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15401.473659 secs 2024-07-26T11:12:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:52 smithi163 ceph-mon[36873]: pgmap v8726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:54 smithi042 ceph-mon[32775]: pgmap v8727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:54 smithi164 ceph-mon[36970]: pgmap v8727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:54 smithi163 ceph-mon[36873]: pgmap v8727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:12:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:56 smithi042 ceph-mon[32775]: pgmap v8728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15424.211873 secs 2024-07-26T11:12:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:56 smithi164 ceph-mon[36970]: pgmap v8728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15424.211873 secs 2024-07-26T11:12:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:56 smithi163 ceph-mon[36873]: pgmap v8728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:12:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15424.211873 secs 2024-07-26T11:12:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15406.473782 secs 2024-07-26T11:12:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15406.473782 secs 2024-07-26T11:12:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15406.473782 secs 2024-07-26T11:12:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:12:58.431+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:12:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:12:58.431+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:12:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:12:58.431+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:12:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:12:58 smithi042 ceph-mon[32775]: pgmap v8729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:12:58 smithi164 ceph-mon[36970]: pgmap v8729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:12:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:12:58 smithi163 ceph-mon[36873]: pgmap v8729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:00 smithi042 ceph-mon[32775]: pgmap v8730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:00 smithi164 ceph-mon[36970]: pgmap v8730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:00 smithi163 ceph-mon[36873]: pgmap v8730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15429.211965 secs 2024-07-26T11:13:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15429.211965 secs 2024-07-26T11:13:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15429.211965 secs 2024-07-26T11:13:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15411.473923 secs 2024-07-26T11:13:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:02 smithi042 ceph-mon[32775]: pgmap v8731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15411.473923 secs 2024-07-26T11:13:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:02 smithi164 ceph-mon[36970]: pgmap v8731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:03.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15411.473923 secs 2024-07-26T11:13:03.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:02 smithi163 ceph-mon[36873]: pgmap v8731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:04 smithi042 ceph-mon[32775]: pgmap v8732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:13:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:04 smithi164 ceph-mon[36970]: pgmap v8732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:13:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:04 smithi163 ceph-mon[36873]: pgmap v8732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:13:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:06 smithi042 ceph-mon[32775]: pgmap v8733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15434.212085 secs 2024-07-26T11:13:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:06 smithi164 ceph-mon[36970]: pgmap v8733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15434.212085 secs 2024-07-26T11:13:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:06 smithi163 ceph-mon[36873]: pgmap v8733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15434.212085 secs 2024-07-26T11:13:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15416.474046 secs 2024-07-26T11:13:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15416.474046 secs 2024-07-26T11:13:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15416.474046 secs 2024-07-26T11:13:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:08 smithi042 ceph-mon[32775]: pgmap v8734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:08 smithi164 ceph-mon[36970]: pgmap v8734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:08 smithi163 ceph-mon[36873]: pgmap v8734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:10.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:10 smithi042 ceph-mon[32775]: pgmap v8735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:10 smithi164 ceph-mon[36970]: pgmap v8735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:10 smithi163 ceph-mon[36873]: pgmap v8735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15439.212172 secs 2024-07-26T11:13:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15439.212172 secs 2024-07-26T11:13:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15439.212172 secs 2024-07-26T11:13:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15421.474167 secs 2024-07-26T11:13:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:12 smithi164 ceph-mon[36970]: pgmap v8736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15421.474167 secs 2024-07-26T11:13:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:12 smithi163 ceph-mon[36873]: pgmap v8736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15421.474167 secs 2024-07-26T11:13:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:12 smithi042 ceph-mon[32775]: pgmap v8736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:13.434 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:13:13.433+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:13:13.435 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:13:13.433+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:13:13.435 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:13:13.433+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:13:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:14 smithi164 ceph-mon[36970]: pgmap v8737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:13:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:14 smithi163 ceph-mon[36873]: pgmap v8737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:13:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:14 smithi042 ceph-mon[32775]: pgmap v8737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:13:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:16 smithi164 ceph-mon[36970]: pgmap v8738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15444.212291 secs 2024-07-26T11:13:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:16 smithi163 ceph-mon[36873]: pgmap v8738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15444.212291 secs 2024-07-26T11:13:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:16 smithi042 ceph-mon[32775]: pgmap v8738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15444.212291 secs 2024-07-26T11:13:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15426.474295 secs 2024-07-26T11:13:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15426.474295 secs 2024-07-26T11:13:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15426.474295 secs 2024-07-26T11:13:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:18 smithi164 ceph-mon[36970]: pgmap v8739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:18 smithi163 ceph-mon[36873]: pgmap v8739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:18 smithi042 ceph-mon[32775]: pgmap v8739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:20 smithi164 ceph-mon[36970]: pgmap v8740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:20 smithi163 ceph-mon[36873]: pgmap v8740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:20 smithi042 ceph-mon[32775]: pgmap v8740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15449.212390 secs 2024-07-26T11:13:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15449.212390 secs 2024-07-26T11:13:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15449.212390 secs 2024-07-26T11:13:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15431.474415 secs 2024-07-26T11:13:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:22 smithi164 ceph-mon[36970]: pgmap v8741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15431.474415 secs 2024-07-26T11:13:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:22 smithi163 ceph-mon[36873]: pgmap v8741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15431.474415 secs 2024-07-26T11:13:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:22 smithi042 ceph-mon[32775]: pgmap v8741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:24 smithi164 ceph-mon[36970]: pgmap v8742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:13:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:24 smithi163 ceph-mon[36873]: pgmap v8742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:13:25.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:24 smithi042 ceph-mon[32775]: pgmap v8742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:13:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:26 smithi164 ceph-mon[36970]: pgmap v8743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15454.212512 secs 2024-07-26T11:13:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:26 smithi163 ceph-mon[36873]: pgmap v8743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15454.212512 secs 2024-07-26T11:13:27.025 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:26 smithi042 ceph-mon[32775]: pgmap v8743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:27.025 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15454.212512 secs 2024-07-26T11:13:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15436.474514 secs 2024-07-26T11:13:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15436.474514 secs 2024-07-26T11:13:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15436.474514 secs 2024-07-26T11:13:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:13:28.438+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:13:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:13:28.438+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:13:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:13:28.438+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:13:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:28 smithi164 ceph-mon[36970]: pgmap v8744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:28 smithi163 ceph-mon[36873]: pgmap v8744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:28 smithi042 ceph-mon[32775]: pgmap v8744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:13:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:13:30.106 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:13:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:30 smithi164 ceph-mon[36970]: pgmap v8745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:30 smithi163 ceph-mon[36873]: pgmap v8745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:30 smithi042 ceph-mon[32775]: pgmap v8745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15441.474583 secs 2024-07-26T11:13:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:32 smithi042 ceph-mon[32775]: pgmap v8746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15459.212580 secs 2024-07-26T11:13:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:13:32.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:13:32.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15441.474583 secs 2024-07-26T11:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:32 smithi164 ceph-mon[36970]: pgmap v8746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15459.212580 secs 2024-07-26T11:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:13:32.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:13:32.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15441.474583 secs 2024-07-26T11:13:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:32 smithi163 ceph-mon[36873]: pgmap v8746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15459.212580 secs 2024-07-26T11:13:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:32.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:13:32.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:13:32.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:13:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:33 smithi042 ceph-mon[32775]: pgmap v8747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:13:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:33 smithi042 ceph-mon[32775]: pgmap v8748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:13:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:33 smithi164 ceph-mon[36970]: pgmap v8747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:13:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:33 smithi164 ceph-mon[36970]: pgmap v8748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:13:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:33 smithi163 ceph-mon[36873]: pgmap v8747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:13:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:33 smithi163 ceph-mon[36873]: pgmap v8748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:13:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:35 smithi042 ceph-mon[32775]: pgmap v8749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:13:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:35 smithi164 ceph-mon[36970]: pgmap v8749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:13:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:35 smithi163 ceph-mon[36873]: pgmap v8749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:13:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15464.212699 secs 2024-07-26T11:13:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15464.212699 secs 2024-07-26T11:13:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15464.212699 secs 2024-07-26T11:13:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15446.474670 secs 2024-07-26T11:13:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:37 smithi042 ceph-mon[32775]: pgmap v8750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:13:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15446.474670 secs 2024-07-26T11:13:37.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:37 smithi164 ceph-mon[36970]: pgmap v8750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:13:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15446.474670 secs 2024-07-26T11:13:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:37 smithi163 ceph-mon[36873]: pgmap v8750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:13:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:39 smithi042 ceph-mon[32775]: pgmap v8751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:13:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:39 smithi164 ceph-mon[36970]: pgmap v8751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:13:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:39 smithi163 ceph-mon[36873]: pgmap v8751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:13:41.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:41 smithi042 ceph-mon[32775]: pgmap v8752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:13:41.873 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15469.212798 secs 2024-07-26T11:13:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:41 smithi164 ceph-mon[36970]: pgmap v8752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:13:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15469.212798 secs 2024-07-26T11:13:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:41 smithi163 ceph-mon[36873]: pgmap v8752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:13:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15469.212798 secs 2024-07-26T11:13:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15451.474813 secs 2024-07-26T11:13:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15451.474813 secs 2024-07-26T11:13:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15451.474813 secs 2024-07-26T11:13:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:13:43.439+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:13:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:13:43.439+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:13:43.441 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:13:43.439+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:13:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:43 smithi042 ceph-mon[32775]: pgmap v8753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:13:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:43 smithi164 ceph-mon[36970]: pgmap v8753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:13:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:43 smithi163 ceph-mon[36873]: pgmap v8753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:13:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:45 smithi042 ceph-mon[32775]: pgmap v8754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:13:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:45 smithi164 ceph-mon[36970]: pgmap v8754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:13:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:45 smithi163 ceph-mon[36873]: pgmap v8754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:13:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15474.212897 secs 2024-07-26T11:13:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15474.212897 secs 2024-07-26T11:13:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15474.212897 secs 2024-07-26T11:13:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15456.474912 secs 2024-07-26T11:13:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:47 smithi042 ceph-mon[32775]: pgmap v8755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:47.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15456.474912 secs 2024-07-26T11:13:47.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:47 smithi163 ceph-mon[36873]: pgmap v8755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15456.474912 secs 2024-07-26T11:13:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:47 smithi164 ceph-mon[36970]: pgmap v8755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:49 smithi042 ceph-mon[32775]: pgmap v8756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:49 smithi164 ceph-mon[36970]: pgmap v8756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:49 smithi163 ceph-mon[36873]: pgmap v8756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:51 smithi042 ceph-mon[32775]: pgmap v8757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15461.475012 secs 2024-07-26T11:13:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15479.213035 secs 2024-07-26T11:13:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:51 smithi164 ceph-mon[36970]: pgmap v8757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15461.475012 secs 2024-07-26T11:13:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15479.213035 secs 2024-07-26T11:13:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:51 smithi163 ceph-mon[36873]: pgmap v8757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15461.475012 secs 2024-07-26T11:13:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15479.213035 secs 2024-07-26T11:13:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:53 smithi042 ceph-mon[32775]: pgmap v8758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:53 smithi164 ceph-mon[36970]: pgmap v8758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:53 smithi163 ceph-mon[36873]: pgmap v8758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:55 smithi042 ceph-mon[32775]: pgmap v8759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:13:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:55 smithi164 ceph-mon[36970]: pgmap v8759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:13:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:55 smithi163 ceph-mon[36873]: pgmap v8759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:13:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15466.475131 secs 2024-07-26T11:13:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15484.213135 secs 2024-07-26T11:13:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15466.475131 secs 2024-07-26T11:13:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15484.213135 secs 2024-07-26T11:13:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15466.475131 secs 2024-07-26T11:13:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15484.213135 secs 2024-07-26T11:13:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:57 smithi042 ceph-mon[32775]: pgmap v8760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:57 smithi164 ceph-mon[36970]: pgmap v8760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:57 smithi163 ceph-mon[36873]: pgmap v8760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:13:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:13:58.444+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:13:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:13:58.444+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:13:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:13:58.444+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:13:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:13:59 smithi042 ceph-mon[32775]: pgmap v8761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:13:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:13:59 smithi164 ceph-mon[36970]: pgmap v8761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:13:59 smithi163 ceph-mon[36873]: pgmap v8761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:14:00.299+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:14:00.300 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:14:00.299+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:14:00.301 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:14:00.299+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:14:00.301 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:14:00.299+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:14:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:01 smithi042 ceph-mon[32775]: pgmap v8762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15471.475303 secs 2024-07-26T11:14:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15489.213235 secs 2024-07-26T11:14:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:01 smithi164 ceph-mon[36970]: pgmap v8762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15471.475303 secs 2024-07-26T11:14:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15489.213235 secs 2024-07-26T11:14:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:01 smithi163 ceph-mon[36873]: pgmap v8762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15471.475303 secs 2024-07-26T11:14:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15489.213235 secs 2024-07-26T11:14:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:03 smithi042 ceph-mon[32775]: pgmap v8763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:03 smithi164 ceph-mon[36970]: pgmap v8763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:03 smithi163 ceph-mon[36873]: pgmap v8763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:05 smithi042 ceph-mon[32775]: pgmap v8764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:14:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:05 smithi164 ceph-mon[36970]: pgmap v8764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:14:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:05 smithi163 ceph-mon[36873]: pgmap v8764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:14:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15476.475376 secs 2024-07-26T11:14:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15494.213355 secs 2024-07-26T11:14:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15476.475376 secs 2024-07-26T11:14:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15494.213355 secs 2024-07-26T11:14:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15476.475376 secs 2024-07-26T11:14:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15494.213355 secs 2024-07-26T11:14:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:07 smithi042 ceph-mon[32775]: pgmap v8765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:07 smithi164 ceph-mon[36970]: pgmap v8765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:07 smithi163 ceph-mon[36873]: pgmap v8765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:09 smithi164 ceph-mon[36970]: pgmap v8766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:09 smithi163 ceph-mon[36873]: pgmap v8766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:09 smithi042 ceph-mon[32775]: pgmap v8766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:11 smithi164 ceph-mon[36970]: pgmap v8767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15481.475495 secs 2024-07-26T11:14:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15499.213499 secs 2024-07-26T11:14:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:11 smithi163 ceph-mon[36873]: pgmap v8767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15481.475495 secs 2024-07-26T11:14:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15499.213499 secs 2024-07-26T11:14:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:11 smithi042 ceph-mon[32775]: pgmap v8767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15481.475495 secs 2024-07-26T11:14:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15499.213499 secs 2024-07-26T11:14:13.446 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:14:13.444+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:14:13.446 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:14:13.444+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:14:13.446 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:14:13.444+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:14:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:13 smithi164 ceph-mon[36970]: pgmap v8768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:13 smithi163 ceph-mon[36873]: pgmap v8768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:13 smithi042 ceph-mon[32775]: pgmap v8768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:15 smithi164 ceph-mon[36970]: pgmap v8769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:14:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:15 smithi163 ceph-mon[36873]: pgmap v8769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:14:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:15 smithi042 ceph-mon[32775]: pgmap v8769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:14:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15486.475596 secs 2024-07-26T11:14:16.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15504.213622 secs 2024-07-26T11:14:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15486.475596 secs 2024-07-26T11:14:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15504.213622 secs 2024-07-26T11:14:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15486.475596 secs 2024-07-26T11:14:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15504.213622 secs 2024-07-26T11:14:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:17 smithi164 ceph-mon[36970]: pgmap v8770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:17 smithi163 ceph-mon[36873]: pgmap v8770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:17 smithi042 ceph-mon[32775]: pgmap v8770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:19 smithi164 ceph-mon[36970]: pgmap v8771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:19 smithi163 ceph-mon[36873]: pgmap v8771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:19 smithi042 ceph-mon[32775]: pgmap v8771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:21 smithi164 ceph-mon[36970]: pgmap v8772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15491.475726 secs 2024-07-26T11:14:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15509.213741 secs 2024-07-26T11:14:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:21 smithi163 ceph-mon[36873]: pgmap v8772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15491.475726 secs 2024-07-26T11:14:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15509.213741 secs 2024-07-26T11:14:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:21 smithi042 ceph-mon[32775]: pgmap v8772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15491.475726 secs 2024-07-26T11:14:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15509.213741 secs 2024-07-26T11:14:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:23 smithi164 ceph-mon[36970]: pgmap v8773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:23 smithi163 ceph-mon[36873]: pgmap v8773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:23 smithi042 ceph-mon[32775]: pgmap v8773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:25 smithi164 ceph-mon[36970]: pgmap v8774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:14:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:25 smithi163 ceph-mon[36873]: pgmap v8774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:14:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:25 smithi042 ceph-mon[32775]: pgmap v8774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:14:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15496.475850 secs 2024-07-26T11:14:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15514.213864 secs 2024-07-26T11:14:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15496.475850 secs 2024-07-26T11:14:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15514.213864 secs 2024-07-26T11:14:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15496.475850 secs 2024-07-26T11:14:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15514.213864 secs 2024-07-26T11:14:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:27 smithi164 ceph-mon[36970]: pgmap v8775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:27 smithi163 ceph-mon[36873]: pgmap v8775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:27 smithi042 ceph-mon[32775]: pgmap v8775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:28.452 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:14:28.450+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:14:28.452 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:14:28.450+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:14:28.452 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:14:28.450+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:14:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:29 smithi164 ceph-mon[36970]: pgmap v8776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:29 smithi163 ceph-mon[36873]: pgmap v8776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:29 smithi042 ceph-mon[32775]: pgmap v8776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:31 smithi164 ceph-mon[36970]: pgmap v8777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15501.475949 secs 2024-07-26T11:14:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15519.213991 secs 2024-07-26T11:14:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:31 smithi163 ceph-mon[36873]: pgmap v8777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15501.475949 secs 2024-07-26T11:14:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15519.213991 secs 2024-07-26T11:14:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:31 smithi042 ceph-mon[32775]: pgmap v8777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15501.475949 secs 2024-07-26T11:14:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15519.213991 secs 2024-07-26T11:14:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:14:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:32.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:32.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:32.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:32.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:32.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:32 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:14:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:32 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:14:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:32 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:33 smithi164 ceph-mon[36970]: pgmap v8778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:33.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:33.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:33.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:33.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:33.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:14:33.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:14:33.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:33 smithi163 ceph-mon[36873]: pgmap v8778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:14:34.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:14:34.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:33 smithi042 ceph-mon[32775]: pgmap v8778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:14:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:14:34.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:14:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:34 smithi164 ceph-mon[36970]: pgmap v8779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:14:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:34 smithi164 ceph-mon[36970]: pgmap v8780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:14:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:34 smithi163 ceph-mon[36873]: pgmap v8779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:14:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:34 smithi163 ceph-mon[36873]: pgmap v8780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:14:35.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:34 smithi042 ceph-mon[32775]: pgmap v8779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:14:35.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:34 smithi042 ceph-mon[32775]: pgmap v8780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:14:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:36 smithi164 ceph-mon[36970]: pgmap v8781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:14:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15506.476076 secs 2024-07-26T11:14:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15524.214114 secs 2024-07-26T11:14:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:36 smithi163 ceph-mon[36873]: pgmap v8781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:14:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15506.476076 secs 2024-07-26T11:14:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15524.214114 secs 2024-07-26T11:14:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:36 smithi042 ceph-mon[32775]: pgmap v8781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:14:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15506.476076 secs 2024-07-26T11:14:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15524.214114 secs 2024-07-26T11:14:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:38 smithi164 ceph-mon[36970]: pgmap v8782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:14:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:38 smithi163 ceph-mon[36873]: pgmap v8782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:14:39.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:38 smithi042 ceph-mon[32775]: pgmap v8782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:14:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:40 smithi164 ceph-mon[36970]: pgmap v8783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:14:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:40 smithi163 ceph-mon[36873]: pgmap v8783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:14:41.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:40 smithi042 ceph-mon[32775]: pgmap v8783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:14:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15511.476202 secs 2024-07-26T11:14:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15529.214236 secs 2024-07-26T11:14:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15511.476202 secs 2024-07-26T11:14:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15529.214236 secs 2024-07-26T11:14:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15511.476202 secs 2024-07-26T11:14:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15529.214236 secs 2024-07-26T11:14:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:42 smithi164 ceph-mon[36970]: pgmap v8784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:14:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:42 smithi163 ceph-mon[36873]: pgmap v8784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:14:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:42 smithi042 ceph-mon[32775]: pgmap v8784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:14:43.454 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:14:43.452+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:14:43.454 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:14:43.452+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:14:43.454 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:14:43.452+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:14:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:44 smithi164 ceph-mon[36970]: pgmap v8785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:14:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:44 smithi163 ceph-mon[36873]: pgmap v8785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:14:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:44 smithi042 ceph-mon[32775]: pgmap v8785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:14:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:46 smithi163 ceph-mon[36873]: pgmap v8786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15516.476343 secs 2024-07-26T11:14:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15534.214356 secs 2024-07-26T11:14:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:46 smithi042 ceph-mon[32775]: pgmap v8786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15516.476343 secs 2024-07-26T11:14:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15534.214356 secs 2024-07-26T11:14:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:46 smithi164 ceph-mon[36970]: pgmap v8786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15516.476343 secs 2024-07-26T11:14:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15534.214356 secs 2024-07-26T11:14:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:48 smithi163 ceph-mon[36873]: pgmap v8787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:48 smithi042 ceph-mon[32775]: pgmap v8787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:48 smithi164 ceph-mon[36970]: pgmap v8787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:50 smithi163 ceph-mon[36873]: pgmap v8788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:50 smithi042 ceph-mon[32775]: pgmap v8788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:50 smithi164 ceph-mon[36970]: pgmap v8788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15521.476425 secs 2024-07-26T11:14:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15539.214478 secs 2024-07-26T11:14:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15521.476425 secs 2024-07-26T11:14:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15539.214478 secs 2024-07-26T11:14:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15521.476425 secs 2024-07-26T11:14:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15539.214478 secs 2024-07-26T11:14:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:52 smithi163 ceph-mon[36873]: pgmap v8789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:52 smithi042 ceph-mon[32775]: pgmap v8789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:52 smithi164 ceph-mon[36970]: pgmap v8789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:54 smithi163 ceph-mon[36873]: pgmap v8790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:54 smithi042 ceph-mon[32775]: pgmap v8790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:54 smithi164 ceph-mon[36970]: pgmap v8790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:56 smithi163 ceph-mon[36873]: pgmap v8791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15526.476525 secs 2024-07-26T11:14:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15544.214576 secs 2024-07-26T11:14:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:56 smithi042 ceph-mon[32775]: pgmap v8791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15526.476525 secs 2024-07-26T11:14:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15544.214576 secs 2024-07-26T11:14:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:56 smithi164 ceph-mon[36970]: pgmap v8791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:14:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15526.476525 secs 2024-07-26T11:14:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15544.214576 secs 2024-07-26T11:14:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:14:58.457+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:14:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:14:58.457+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:14:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:14:58.457+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:14:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:58 smithi163 ceph-mon[36873]: pgmap v8792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:58 smithi042 ceph-mon[32775]: pgmap v8792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:14:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:58 smithi164 ceph-mon[36970]: pgmap v8792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:14:59 smithi163 ceph-mon[36873]: pgmap v8793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:14:59 smithi042 ceph-mon[32775]: pgmap v8793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:14:59 smithi164 ceph-mon[36970]: pgmap v8793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15531.476647 secs 2024-07-26T11:15:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15531.476647 secs 2024-07-26T11:15:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15531.476647 secs 2024-07-26T11:15:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:02 smithi042 ceph-mon[32775]: pgmap v8794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15549.214694 secs 2024-07-26T11:15:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:02 smithi164 ceph-mon[36970]: pgmap v8794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15549.214694 secs 2024-07-26T11:15:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:02 smithi163 ceph-mon[36873]: pgmap v8794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15549.214694 secs 2024-07-26T11:15:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:04 smithi042 ceph-mon[32775]: pgmap v8795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:04 smithi164 ceph-mon[36970]: pgmap v8795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:04 smithi163 ceph-mon[36873]: pgmap v8795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:06 smithi042 ceph-mon[32775]: pgmap v8796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15536.476751 secs 2024-07-26T11:15:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:06 smithi164 ceph-mon[36970]: pgmap v8796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15536.476751 secs 2024-07-26T11:15:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:06 smithi163 ceph-mon[36873]: pgmap v8796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15536.476751 secs 2024-07-26T11:15:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15554.214837 secs 2024-07-26T11:15:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15554.214837 secs 2024-07-26T11:15:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15554.214837 secs 2024-07-26T11:15:08.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:08 smithi042 ceph-mon[32775]: pgmap v8797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:08 smithi164 ceph-mon[36970]: pgmap v8797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:08 smithi163 ceph-mon[36873]: pgmap v8797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:10 smithi042 ceph-mon[32775]: pgmap v8798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:10 smithi164 ceph-mon[36970]: pgmap v8798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:10 smithi163 ceph-mon[36873]: pgmap v8798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15541.476874 secs 2024-07-26T11:15:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15541.476874 secs 2024-07-26T11:15:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15541.476874 secs 2024-07-26T11:15:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:12 smithi042 ceph-mon[32775]: pgmap v8799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15559.214988 secs 2024-07-26T11:15:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:12 smithi164 ceph-mon[36970]: pgmap v8799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15559.214988 secs 2024-07-26T11:15:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:12 smithi163 ceph-mon[36873]: pgmap v8799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15559.214988 secs 2024-07-26T11:15:13.460 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:15:13.458+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:15:13.460 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:15:13.458+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:15:13.460 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:15:13.458+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:15:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:14 smithi042 ceph-mon[32775]: pgmap v8800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:14 smithi164 ceph-mon[36970]: pgmap v8800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:14 smithi163 ceph-mon[36873]: pgmap v8800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:16 smithi042 ceph-mon[32775]: pgmap v8801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15546.476996 secs 2024-07-26T11:15:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:16 smithi164 ceph-mon[36970]: pgmap v8801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15546.476996 secs 2024-07-26T11:15:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:16 smithi163 ceph-mon[36873]: pgmap v8801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15546.476996 secs 2024-07-26T11:15:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15564.215109 secs 2024-07-26T11:15:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15564.215109 secs 2024-07-26T11:15:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15564.215109 secs 2024-07-26T11:15:18.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:18 smithi042 ceph-mon[32775]: pgmap v8802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:18.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:18 smithi163 ceph-mon[36873]: pgmap v8802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:18 smithi164 ceph-mon[36970]: pgmap v8802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:20 smithi042 ceph-mon[32775]: pgmap v8803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:20 smithi164 ceph-mon[36970]: pgmap v8803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:20 smithi163 ceph-mon[36873]: pgmap v8803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:21.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15551.477114 secs 2024-07-26T11:15:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15551.477114 secs 2024-07-26T11:15:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15551.477114 secs 2024-07-26T11:15:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:22 smithi042 ceph-mon[32775]: pgmap v8804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15569.215227 secs 2024-07-26T11:15:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:22 smithi164 ceph-mon[36970]: pgmap v8804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15569.215227 secs 2024-07-26T11:15:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:22 smithi163 ceph-mon[36873]: pgmap v8804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15569.215227 secs 2024-07-26T11:15:24.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:24 smithi042 ceph-mon[32775]: pgmap v8805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:24 smithi164 ceph-mon[36970]: pgmap v8805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:24 smithi163 ceph-mon[36873]: pgmap v8805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:26 smithi042 ceph-mon[32775]: pgmap v8806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15556.477222 secs 2024-07-26T11:15:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:26 smithi164 ceph-mon[36970]: pgmap v8806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15556.477222 secs 2024-07-26T11:15:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:26 smithi163 ceph-mon[36873]: pgmap v8806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15556.477222 secs 2024-07-26T11:15:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15574.215329 secs 2024-07-26T11:15:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15574.215329 secs 2024-07-26T11:15:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15574.215329 secs 2024-07-26T11:15:28.465 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:15:28.463+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:15:28.465 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:15:28.463+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:15:28.465 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:15:28.463+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:15:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:28 smithi042 ceph-mon[32775]: pgmap v8807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:28 smithi164 ceph-mon[36970]: pgmap v8807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:28 smithi163 ceph-mon[36873]: pgmap v8807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:30 smithi042 ceph-mon[32775]: pgmap v8808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:30 smithi164 ceph-mon[36970]: pgmap v8808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:30 smithi163 ceph-mon[36873]: pgmap v8808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15561.477335 secs 2024-07-26T11:15:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15561.477335 secs 2024-07-26T11:15:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15561.477335 secs 2024-07-26T11:15:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:32 smithi042 ceph-mon[32775]: pgmap v8809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15579.215532 secs 2024-07-26T11:15:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:32 smithi164 ceph-mon[36970]: pgmap v8809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15579.215532 secs 2024-07-26T11:15:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:32 smithi163 ceph-mon[36873]: pgmap v8809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15579.215532 secs 2024-07-26T11:15:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:15:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:15:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:15:34.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:34 smithi164 ceph-mon[36970]: pgmap v8810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:34.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:15:34.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:15:34.625 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:15:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:34 smithi042 ceph-mon[32775]: pgmap v8810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:15:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:15:34.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:15:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:34 smithi163 ceph-mon[36873]: pgmap v8810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:15:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:15:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:15:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:35 smithi042 ceph-mon[32775]: pgmap v8811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:15:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:35 smithi042 ceph-mon[32775]: pgmap v8812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:15:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:35 smithi164 ceph-mon[36970]: pgmap v8811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:15:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:35 smithi164 ceph-mon[36970]: pgmap v8812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:15:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:35 smithi163 ceph-mon[36873]: pgmap v8811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:15:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:35 smithi163 ceph-mon[36873]: pgmap v8812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:15:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15566.477461 secs 2024-07-26T11:15:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15566.477461 secs 2024-07-26T11:15:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15566.477461 secs 2024-07-26T11:15:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:37 smithi042 ceph-mon[32775]: pgmap v8813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:15:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15584.215661 secs 2024-07-26T11:15:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:37 smithi164 ceph-mon[36970]: pgmap v8813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:15:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15584.215661 secs 2024-07-26T11:15:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:37 smithi163 ceph-mon[36873]: pgmap v8813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:15:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15584.215661 secs 2024-07-26T11:15:39.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:39 smithi042 ceph-mon[32775]: pgmap v8814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:15:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:39 smithi164 ceph-mon[36970]: pgmap v8814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:15:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:39 smithi163 ceph-mon[36873]: pgmap v8814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:15:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:41 smithi042 ceph-mon[32775]: pgmap v8815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:15:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15571.477580 secs 2024-07-26T11:15:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:41 smithi164 ceph-mon[36970]: pgmap v8815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:15:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15571.477580 secs 2024-07-26T11:15:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:41 smithi163 ceph-mon[36873]: pgmap v8815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:15:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15571.477580 secs 2024-07-26T11:15:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15589.215787 secs 2024-07-26T11:15:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15589.215787 secs 2024-07-26T11:15:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15589.215787 secs 2024-07-26T11:15:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:15:43.464+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:15:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:15:43.464+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:15:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:15:43.464+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:15:43.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:43 smithi042 ceph-mon[32775]: pgmap v8816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:15:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:43 smithi164 ceph-mon[36970]: pgmap v8816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:15:43.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:43 smithi163 ceph-mon[36873]: pgmap v8816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:15:45.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:45 smithi042 ceph-mon[32775]: pgmap v8817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:15:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:45 smithi164 ceph-mon[36970]: pgmap v8817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:15:45.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:45 smithi163 ceph-mon[36873]: pgmap v8817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:15:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15576.477704 secs 2024-07-26T11:15:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15576.477704 secs 2024-07-26T11:15:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15576.477704 secs 2024-07-26T11:15:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:47 smithi042 ceph-mon[32775]: pgmap v8818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15594.215909 secs 2024-07-26T11:15:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:47 smithi164 ceph-mon[36970]: pgmap v8818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:47.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15594.215909 secs 2024-07-26T11:15:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:47 smithi163 ceph-mon[36873]: pgmap v8818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15594.215909 secs 2024-07-26T11:15:49.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:49 smithi164 ceph-mon[36970]: pgmap v8819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:49 smithi163 ceph-mon[36873]: pgmap v8819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:49 smithi042 ceph-mon[32775]: pgmap v8819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:51 smithi164 ceph-mon[36970]: pgmap v8820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15581.477780 secs 2024-07-26T11:15:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:51 smithi163 ceph-mon[36873]: pgmap v8820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15581.477780 secs 2024-07-26T11:15:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:51 smithi042 ceph-mon[32775]: pgmap v8820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15581.477780 secs 2024-07-26T11:15:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15599.216040 secs 2024-07-26T11:15:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15599.216040 secs 2024-07-26T11:15:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15599.216040 secs 2024-07-26T11:15:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:53 smithi164 ceph-mon[36970]: pgmap v8821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:53 smithi163 ceph-mon[36873]: pgmap v8821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:53 smithi042 ceph-mon[32775]: pgmap v8821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:55 smithi164 ceph-mon[36970]: pgmap v8822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:15:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:55 smithi163 ceph-mon[36873]: pgmap v8822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:15:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:55 smithi042 ceph-mon[32775]: pgmap v8822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:15:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15586.477854 secs 2024-07-26T11:15:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15586.477854 secs 2024-07-26T11:15:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15586.477854 secs 2024-07-26T11:15:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:57 smithi164 ceph-mon[36970]: pgmap v8823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15604.216163 secs 2024-07-26T11:15:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:57 smithi163 ceph-mon[36873]: pgmap v8823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15604.216163 secs 2024-07-26T11:15:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:57 smithi042 ceph-mon[32775]: pgmap v8823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:15:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15604.216163 secs 2024-07-26T11:15:58.471 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:15:58.470+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:15:58.471 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:15:58.470+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:15:58.471 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:15:58.470+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:15:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:15:59 smithi164 ceph-mon[36970]: pgmap v8824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:15:59 smithi163 ceph-mon[36873]: pgmap v8824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:15:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:15:59 smithi042 ceph-mon[32775]: pgmap v8824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:01 smithi164 ceph-mon[36970]: pgmap v8825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:01.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15591.477986 secs 2024-07-26T11:16:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:01 smithi163 ceph-mon[36873]: pgmap v8825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15591.477986 secs 2024-07-26T11:16:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:01 smithi042 ceph-mon[32775]: pgmap v8825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15591.477986 secs 2024-07-26T11:16:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15609.216283 secs 2024-07-26T11:16:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15609.216283 secs 2024-07-26T11:16:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15609.216283 secs 2024-07-26T11:16:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:03 smithi164 ceph-mon[36970]: pgmap v8826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:03 smithi163 ceph-mon[36873]: pgmap v8826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:03 smithi042 ceph-mon[32775]: pgmap v8826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:05 smithi164 ceph-mon[36970]: pgmap v8827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:16:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:05 smithi163 ceph-mon[36873]: pgmap v8827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:16:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:05 smithi042 ceph-mon[32775]: pgmap v8827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:16:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15596.478134 secs 2024-07-26T11:16:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15614.216423 secs 2024-07-26T11:16:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15596.478134 secs 2024-07-26T11:16:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15614.216423 secs 2024-07-26T11:16:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15596.478134 secs 2024-07-26T11:16:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15614.216423 secs 2024-07-26T11:16:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:07 smithi164 ceph-mon[36970]: pgmap v8828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:07 smithi163 ceph-mon[36873]: pgmap v8828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:07 smithi042 ceph-mon[32775]: pgmap v8828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:09 smithi164 ceph-mon[36970]: pgmap v8829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:09 smithi163 ceph-mon[36873]: pgmap v8829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:09 smithi042 ceph-mon[32775]: pgmap v8829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:11 smithi164 ceph-mon[36970]: pgmap v8830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15601.478251 secs 2024-07-26T11:16:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15619.216544 secs 2024-07-26T11:16:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:11 smithi163 ceph-mon[36873]: pgmap v8830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15601.478251 secs 2024-07-26T11:16:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15619.216544 secs 2024-07-26T11:16:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:11 smithi042 ceph-mon[32775]: pgmap v8830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15601.478251 secs 2024-07-26T11:16:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15619.216544 secs 2024-07-26T11:16:13.472 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:16:13.470+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:16:13.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:16:13.471+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:16:13.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:16:13.471+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:16:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:13 smithi164 ceph-mon[36970]: pgmap v8831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:13 smithi163 ceph-mon[36873]: pgmap v8831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:13 smithi042 ceph-mon[32775]: pgmap v8831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:15 smithi164 ceph-mon[36970]: pgmap v8832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:16:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:15 smithi163 ceph-mon[36873]: pgmap v8832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:16:15.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:15 smithi042 ceph-mon[32775]: pgmap v8832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:16:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15606.478376 secs 2024-07-26T11:16:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15624.216683 secs 2024-07-26T11:16:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15606.478376 secs 2024-07-26T11:16:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15624.216683 secs 2024-07-26T11:16:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15606.478376 secs 2024-07-26T11:16:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15624.216683 secs 2024-07-26T11:16:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:17 smithi164 ceph-mon[36970]: pgmap v8833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:17 smithi163 ceph-mon[36873]: pgmap v8833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:17 smithi042 ceph-mon[32775]: pgmap v8833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:19 smithi164 ceph-mon[36970]: pgmap v8834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:19 smithi163 ceph-mon[36873]: pgmap v8834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:19 smithi042 ceph-mon[32775]: pgmap v8834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:21 smithi163 ceph-mon[36873]: pgmap v8835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15611.478472 secs 2024-07-26T11:16:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15629.216819 secs 2024-07-26T11:16:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:21 smithi042 ceph-mon[32775]: pgmap v8835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15611.478472 secs 2024-07-26T11:16:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15629.216819 secs 2024-07-26T11:16:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:21 smithi164 ceph-mon[36970]: pgmap v8835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15611.478472 secs 2024-07-26T11:16:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15629.216819 secs 2024-07-26T11:16:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:23 smithi163 ceph-mon[36873]: pgmap v8836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:23.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:23 smithi042 ceph-mon[32775]: pgmap v8836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:23 smithi164 ceph-mon[36970]: pgmap v8836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:25 smithi163 ceph-mon[36873]: pgmap v8837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:16:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:25 smithi042 ceph-mon[32775]: pgmap v8837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:16:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:25 smithi164 ceph-mon[36970]: pgmap v8837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:16:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15616.478569 secs 2024-07-26T11:16:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15634.216964 secs 2024-07-26T11:16:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15616.478569 secs 2024-07-26T11:16:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15634.216964 secs 2024-07-26T11:16:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15616.478569 secs 2024-07-26T11:16:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15634.216964 secs 2024-07-26T11:16:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:27 smithi163 ceph-mon[36873]: pgmap v8838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:27 smithi042 ceph-mon[32775]: pgmap v8838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:27 smithi164 ceph-mon[36970]: pgmap v8838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:28.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:16:28.477+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:16:28.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:16:28.477+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:16:28.479 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:16:28.477+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:16:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:29 smithi163 ceph-mon[36873]: pgmap v8839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:29.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:29 smithi042 ceph-mon[32775]: pgmap v8839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:29 smithi164 ceph-mon[36970]: pgmap v8839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:31 smithi163 ceph-mon[36873]: pgmap v8840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15621.478717 secs 2024-07-26T11:16:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15639.217120 secs 2024-07-26T11:16:31.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:31 smithi042 ceph-mon[32775]: pgmap v8840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:31.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15621.478717 secs 2024-07-26T11:16:31.857 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15639.217120 secs 2024-07-26T11:16:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:31 smithi164 ceph-mon[36970]: pgmap v8840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15621.478717 secs 2024-07-26T11:16:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15639.217120 secs 2024-07-26T11:16:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:33 smithi163 ceph-mon[36873]: pgmap v8841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:33 smithi042 ceph-mon[32775]: pgmap v8841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:33 smithi164 ceph-mon[36970]: pgmap v8841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:16:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:16:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:16:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:16:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:16:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:16:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:16:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:16:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:16:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:16:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:16:34.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:16:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:35 smithi163 ceph-mon[36873]: pgmap v8842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:16:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:35 smithi163 ceph-mon[36873]: pgmap v8843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:16:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:35 smithi163 ceph-mon[36873]: pgmap v8844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:16:35.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:35 smithi164 ceph-mon[36970]: pgmap v8842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:16:35.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:35 smithi164 ceph-mon[36970]: pgmap v8843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:16:35.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:35 smithi164 ceph-mon[36970]: pgmap v8844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:16:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:35 smithi042 ceph-mon[32775]: pgmap v8842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:16:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:35 smithi042 ceph-mon[32775]: pgmap v8843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:16:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:35 smithi042 ceph-mon[32775]: pgmap v8844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:16:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15626.478839 secs 2024-07-26T11:16:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15644.217285 secs 2024-07-26T11:16:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15626.478839 secs 2024-07-26T11:16:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15644.217285 secs 2024-07-26T11:16:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15626.478839 secs 2024-07-26T11:16:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15644.217285 secs 2024-07-26T11:16:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:37 smithi042 ceph-mon[32775]: pgmap v8845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:16:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:37 smithi164 ceph-mon[36970]: pgmap v8845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:16:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:37 smithi163 ceph-mon[36873]: pgmap v8845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:16:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:39 smithi042 ceph-mon[32775]: pgmap v8846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:16:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:39 smithi164 ceph-mon[36970]: pgmap v8846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:16:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:39 smithi163 ceph-mon[36873]: pgmap v8846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:16:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:41 smithi042 ceph-mon[32775]: pgmap v8847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:16:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15631.478954 secs 2024-07-26T11:16:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15649.217407 secs 2024-07-26T11:16:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:41 smithi164 ceph-mon[36970]: pgmap v8847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:16:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15631.478954 secs 2024-07-26T11:16:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15649.217407 secs 2024-07-26T11:16:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:41 smithi163 ceph-mon[36873]: pgmap v8847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:16:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15631.478954 secs 2024-07-26T11:16:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15649.217407 secs 2024-07-26T11:16:43.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:16:43.478+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:16:43.480 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:16:43.478+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:16:43.481 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:16:43.478+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:16:43.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:43 smithi042 ceph-mon[32775]: pgmap v8848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:16:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:43 smithi164 ceph-mon[36970]: pgmap v8848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:16:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:43 smithi163 ceph-mon[36873]: pgmap v8848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:16:45.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:45 smithi042 ceph-mon[32775]: pgmap v8849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:16:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:45 smithi164 ceph-mon[36970]: pgmap v8849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:16:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:45 smithi163 ceph-mon[36873]: pgmap v8849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:16:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15636.479072 secs 2024-07-26T11:16:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15654.217528 secs 2024-07-26T11:16:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15636.479072 secs 2024-07-26T11:16:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15654.217528 secs 2024-07-26T11:16:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15636.479072 secs 2024-07-26T11:16:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15654.217528 secs 2024-07-26T11:16:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:47 smithi042 ceph-mon[32775]: pgmap v8850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:47 smithi164 ceph-mon[36970]: pgmap v8850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:47 smithi163 ceph-mon[36873]: pgmap v8850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:49.810 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:49 smithi163 ceph-mon[36873]: pgmap v8851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:16:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:49 smithi042 ceph-mon[32775]: pgmap v8851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:16:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:49 smithi164 ceph-mon[36970]: pgmap v8851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:16:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:51 smithi042 ceph-mon[32775]: pgmap v8852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15641.479191 secs 2024-07-26T11:16:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15659.217654 secs 2024-07-26T11:16:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:51 smithi164 ceph-mon[36970]: pgmap v8852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15641.479191 secs 2024-07-26T11:16:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15659.217654 secs 2024-07-26T11:16:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:51 smithi163 ceph-mon[36873]: pgmap v8852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15641.479191 secs 2024-07-26T11:16:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15659.217654 secs 2024-07-26T11:16:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:53 smithi042 ceph-mon[32775]: pgmap v8853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:53 smithi164 ceph-mon[36970]: pgmap v8853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:53 smithi163 ceph-mon[36873]: pgmap v8853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:55 smithi042 ceph-mon[32775]: pgmap v8854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:55 smithi164 ceph-mon[36970]: pgmap v8854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:55 smithi163 ceph-mon[36873]: pgmap v8854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:16:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15646.479320 secs 2024-07-26T11:16:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15664.217779 secs 2024-07-26T11:16:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15646.479320 secs 2024-07-26T11:16:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15664.217779 secs 2024-07-26T11:16:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15646.479320 secs 2024-07-26T11:16:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15664.217779 secs 2024-07-26T11:16:57.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:57 smithi042 ceph-mon[32775]: pgmap v8855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:57 smithi164 ceph-mon[36970]: pgmap v8855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:57 smithi163 ceph-mon[36873]: pgmap v8855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:16:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:16:58.484+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:16:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:16:58.484+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:16:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:16:58.484+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:16:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:16:59 smithi042 ceph-mon[32775]: pgmap v8856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:16:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:16:59 smithi164 ceph-mon[36970]: pgmap v8856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:17:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:16:59 smithi163 ceph-mon[36873]: pgmap v8856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:17:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:17:00.327+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:17:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:17:00.327+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:17:00.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:17:00.327+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:17:00.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:17:00.328+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:17:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:01 smithi042 ceph-mon[32775]: pgmap v8857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15651.479407 secs 2024-07-26T11:17:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15669.217903 secs 2024-07-26T11:17:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:01 smithi164 ceph-mon[36970]: pgmap v8857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15651.479407 secs 2024-07-26T11:17:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15669.217903 secs 2024-07-26T11:17:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:01 smithi163 ceph-mon[36873]: pgmap v8857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15651.479407 secs 2024-07-26T11:17:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15669.217903 secs 2024-07-26T11:17:03.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:03 smithi042 ceph-mon[32775]: pgmap v8858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:03 smithi164 ceph-mon[36970]: pgmap v8858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:03 smithi163 ceph-mon[36873]: pgmap v8858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:05.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:05 smithi042 ceph-mon[32775]: pgmap v8859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:05 smithi164 ceph-mon[36970]: pgmap v8859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:05 smithi163 ceph-mon[36873]: pgmap v8859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15656.479527 secs 2024-07-26T11:17:06.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15674.218035 secs 2024-07-26T11:17:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15656.479527 secs 2024-07-26T11:17:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15674.218035 secs 2024-07-26T11:17:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15656.479527 secs 2024-07-26T11:17:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15674.218035 secs 2024-07-26T11:17:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:07 smithi042 ceph-mon[32775]: pgmap v8860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:07 smithi164 ceph-mon[36970]: pgmap v8860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:07 smithi163 ceph-mon[36873]: pgmap v8860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:09.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:09 smithi042 ceph-mon[32775]: pgmap v8861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:17:09.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:09 smithi164 ceph-mon[36970]: pgmap v8861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:17:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:09 smithi163 ceph-mon[36873]: pgmap v8861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:17:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:11 smithi164 ceph-mon[36970]: pgmap v8862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15661.479649 secs 2024-07-26T11:17:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15679.218162 secs 2024-07-26T11:17:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:11 smithi163 ceph-mon[36873]: pgmap v8862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15661.479649 secs 2024-07-26T11:17:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15679.218162 secs 2024-07-26T11:17:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:11 smithi042 ceph-mon[32775]: pgmap v8862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15661.479649 secs 2024-07-26T11:17:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15679.218162 secs 2024-07-26T11:17:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:17:13.485+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:17:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:17:13.485+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:17:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:17:13.485+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:17:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:13 smithi164 ceph-mon[36970]: pgmap v8863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:13 smithi163 ceph-mon[36873]: pgmap v8863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:13 smithi042 ceph-mon[32775]: pgmap v8863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:15 smithi164 ceph-mon[36970]: pgmap v8864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:15 smithi163 ceph-mon[36873]: pgmap v8864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:15 smithi042 ceph-mon[32775]: pgmap v8864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15666.479776 secs 2024-07-26T11:17:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15684.218262 secs 2024-07-26T11:17:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15666.479776 secs 2024-07-26T11:17:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15684.218262 secs 2024-07-26T11:17:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15666.479776 secs 2024-07-26T11:17:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15684.218262 secs 2024-07-26T11:17:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:17 smithi164 ceph-mon[36970]: pgmap v8865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:17 smithi163 ceph-mon[36873]: pgmap v8865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:17 smithi042 ceph-mon[32775]: pgmap v8865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:19 smithi164 ceph-mon[36970]: pgmap v8866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:17:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:19 smithi163 ceph-mon[36873]: pgmap v8866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:17:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:19 smithi042 ceph-mon[32775]: pgmap v8866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:17:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:21 smithi164 ceph-mon[36970]: pgmap v8867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15671.479903 secs 2024-07-26T11:17:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15689.218385 secs 2024-07-26T11:17:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:21 smithi163 ceph-mon[36873]: pgmap v8867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15671.479903 secs 2024-07-26T11:17:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15689.218385 secs 2024-07-26T11:17:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:21 smithi042 ceph-mon[32775]: pgmap v8867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15671.479903 secs 2024-07-26T11:17:22.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15689.218385 secs 2024-07-26T11:17:23.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:23 smithi164 ceph-mon[36970]: pgmap v8868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:23 smithi163 ceph-mon[36873]: pgmap v8868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:23 smithi042 ceph-mon[32775]: pgmap v8868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:25 smithi164 ceph-mon[36970]: pgmap v8869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:25 smithi163 ceph-mon[36873]: pgmap v8869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:25 smithi042 ceph-mon[32775]: pgmap v8869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15676.480050 secs 2024-07-26T11:17:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15694.218522 secs 2024-07-26T11:17:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15676.480050 secs 2024-07-26T11:17:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15694.218522 secs 2024-07-26T11:17:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15676.480050 secs 2024-07-26T11:17:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15694.218522 secs 2024-07-26T11:17:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:27 smithi164 ceph-mon[36970]: pgmap v8870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:27 smithi163 ceph-mon[36873]: pgmap v8870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:27 smithi042 ceph-mon[32775]: pgmap v8870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:28.492 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:17:28.491+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:17:28.492 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:17:28.491+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:17:28.492 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:17:28.491+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:17:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:29 smithi164 ceph-mon[36970]: pgmap v8871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:17:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:29 smithi163 ceph-mon[36873]: pgmap v8871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:17:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:29 smithi042 ceph-mon[32775]: pgmap v8871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:17:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:31 smithi164 ceph-mon[36970]: pgmap v8872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15681.480172 secs 2024-07-26T11:17:31.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15699.218648 secs 2024-07-26T11:17:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:31 smithi163 ceph-mon[36873]: pgmap v8872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15681.480172 secs 2024-07-26T11:17:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15699.218648 secs 2024-07-26T11:17:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:31 smithi042 ceph-mon[32775]: pgmap v8872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15681.480172 secs 2024-07-26T11:17:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15699.218648 secs 2024-07-26T11:17:33.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:33 smithi042 ceph-mon[32775]: pgmap v8873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:33 smithi164 ceph-mon[36970]: pgmap v8873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:33 smithi163 ceph-mon[36873]: pgmap v8873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:17:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:17:35.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:17:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:35 smithi164 ceph-mon[36970]: pgmap v8874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:35.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:17:35.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:17:35.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:17:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:35 smithi163 ceph-mon[36873]: pgmap v8874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:17:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:17:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:17:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:35 smithi042 ceph-mon[32775]: pgmap v8874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:17:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:17:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:17:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:36 smithi164 ceph-mon[36970]: pgmap v8875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:17:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:36 smithi164 ceph-mon[36970]: pgmap v8876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:17:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15686.480301 secs 2024-07-26T11:17:36.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15704.218769 secs 2024-07-26T11:17:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:36 smithi163 ceph-mon[36873]: pgmap v8875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:17:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:36 smithi163 ceph-mon[36873]: pgmap v8876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:17:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15686.480301 secs 2024-07-26T11:17:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15704.218769 secs 2024-07-26T11:17:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:36 smithi042 ceph-mon[32775]: pgmap v8875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:17:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:36 smithi042 ceph-mon[32775]: pgmap v8876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:17:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15686.480301 secs 2024-07-26T11:17:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15704.218769 secs 2024-07-26T11:17:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:38 smithi164 ceph-mon[36970]: pgmap v8877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:17:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:38 smithi163 ceph-mon[36873]: pgmap v8877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:17:39.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:38 smithi042 ceph-mon[32775]: pgmap v8877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:17:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:40 smithi164 ceph-mon[36970]: pgmap v8878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:17:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:40 smithi163 ceph-mon[36873]: pgmap v8878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:17:41.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:40 smithi042 ceph-mon[32775]: pgmap v8878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:17:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15691.480365 secs 2024-07-26T11:17:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15709.218889 secs 2024-07-26T11:17:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15691.480365 secs 2024-07-26T11:17:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15709.218889 secs 2024-07-26T11:17:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15691.480365 secs 2024-07-26T11:17:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15709.218889 secs 2024-07-26T11:17:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:42 smithi164 ceph-mon[36970]: pgmap v8879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:17:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:42 smithi163 ceph-mon[36873]: pgmap v8879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:17:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:42 smithi042 ceph-mon[32775]: pgmap v8879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:17:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:17:43.492+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:17:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:17:43.492+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:17:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:17:43.492+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:17:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:44 smithi164 ceph-mon[36970]: pgmap v8880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:17:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:44 smithi163 ceph-mon[36873]: pgmap v8880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:17:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:44 smithi042 ceph-mon[32775]: pgmap v8880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:17:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:46 smithi163 ceph-mon[36873]: pgmap v8881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:17:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15696.480438 secs 2024-07-26T11:17:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15714.219054 secs 2024-07-26T11:17:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:46 smithi042 ceph-mon[32775]: pgmap v8881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:17:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15696.480438 secs 2024-07-26T11:17:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15714.219054 secs 2024-07-26T11:17:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:46 smithi164 ceph-mon[36970]: pgmap v8881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:17:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15696.480438 secs 2024-07-26T11:17:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15714.219054 secs 2024-07-26T11:17:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:48 smithi163 ceph-mon[36873]: pgmap v8882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:48 smithi042 ceph-mon[32775]: pgmap v8882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:48 smithi164 ceph-mon[36970]: pgmap v8882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:50 smithi163 ceph-mon[36873]: pgmap v8883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:17:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:50 smithi042 ceph-mon[32775]: pgmap v8883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:17:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:50 smithi164 ceph-mon[36970]: pgmap v8883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:17:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15701.480542 secs 2024-07-26T11:17:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15719.219171 secs 2024-07-26T11:17:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15701.480542 secs 2024-07-26T11:17:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15719.219171 secs 2024-07-26T11:17:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15701.480542 secs 2024-07-26T11:17:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15719.219171 secs 2024-07-26T11:17:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:52 smithi163 ceph-mon[36873]: pgmap v8884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:52 smithi042 ceph-mon[32775]: pgmap v8884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:52 smithi164 ceph-mon[36970]: pgmap v8884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:54 smithi163 ceph-mon[36873]: pgmap v8885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:54 smithi042 ceph-mon[32775]: pgmap v8885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:54 smithi164 ceph-mon[36970]: pgmap v8885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:56 smithi163 ceph-mon[36873]: pgmap v8886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15706.480628 secs 2024-07-26T11:17:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15724.219291 secs 2024-07-26T11:17:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:56 smithi042 ceph-mon[32775]: pgmap v8886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15706.480628 secs 2024-07-26T11:17:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15724.219291 secs 2024-07-26T11:17:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:56 smithi164 ceph-mon[36970]: pgmap v8886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:17:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15706.480628 secs 2024-07-26T11:17:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15724.219291 secs 2024-07-26T11:17:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:17:58.495+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:17:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:17:58.495+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:17:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:17:58.495+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:17:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:17:58 smithi163 ceph-mon[36873]: pgmap v8887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:17:58 smithi042 ceph-mon[32775]: pgmap v8887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:17:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:17:58 smithi164 ceph-mon[36970]: pgmap v8887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:00 smithi163 ceph-mon[36873]: pgmap v8888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:18:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:00 smithi042 ceph-mon[32775]: pgmap v8888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:18:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:00 smithi164 ceph-mon[36970]: pgmap v8888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:18:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15711.480772 secs 2024-07-26T11:18:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15729.219414 secs 2024-07-26T11:18:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15711.480772 secs 2024-07-26T11:18:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15729.219414 secs 2024-07-26T11:18:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15711.480772 secs 2024-07-26T11:18:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15729.219414 secs 2024-07-26T11:18:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:02 smithi042 ceph-mon[32775]: pgmap v8889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:02 smithi164 ceph-mon[36970]: pgmap v8889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:02 smithi163 ceph-mon[36873]: pgmap v8889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:05.026 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:04 smithi164 ceph-mon[36970]: pgmap v8890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:05.041 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:04 smithi042 ceph-mon[32775]: pgmap v8890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:04 smithi163 ceph-mon[36873]: pgmap v8890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:06 smithi042 ceph-mon[32775]: pgmap v8891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15716.480899 secs 2024-07-26T11:18:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15734.219531 secs 2024-07-26T11:18:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:06 smithi164 ceph-mon[36970]: pgmap v8891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15716.480899 secs 2024-07-26T11:18:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15734.219531 secs 2024-07-26T11:18:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:06 smithi163 ceph-mon[36873]: pgmap v8891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15716.480899 secs 2024-07-26T11:18:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15734.219531 secs 2024-07-26T11:18:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:08 smithi042 ceph-mon[32775]: pgmap v8892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:08 smithi164 ceph-mon[36970]: pgmap v8892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:09.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:08 smithi163 ceph-mon[36873]: pgmap v8892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:10 smithi042 ceph-mon[32775]: pgmap v8893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:18:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:10 smithi164 ceph-mon[36970]: pgmap v8893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:18:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:10 smithi163 ceph-mon[36873]: pgmap v8893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:18:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15721.481045 secs 2024-07-26T11:18:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15739.219656 secs 2024-07-26T11:18:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15721.481045 secs 2024-07-26T11:18:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15739.219656 secs 2024-07-26T11:18:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15721.481045 secs 2024-07-26T11:18:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15739.219656 secs 2024-07-26T11:18:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:12 smithi042 ceph-mon[32775]: pgmap v8894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:12 smithi164 ceph-mon[36970]: pgmap v8894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:12 smithi163 ceph-mon[36873]: pgmap v8894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:13.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:18:13.496+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:18:13.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:18:13.496+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:18:13.498 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:18:13.496+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:18:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:13 smithi042 ceph-mon[32775]: pgmap v8895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:13 smithi164 ceph-mon[36970]: pgmap v8895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:13 smithi163 ceph-mon[36873]: pgmap v8895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:15 smithi164 ceph-mon[36970]: pgmap v8896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:15 smithi163 ceph-mon[36873]: pgmap v8896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:15 smithi042 ceph-mon[32775]: pgmap v8896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15726.481183 secs 2024-07-26T11:18:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15744.219721 secs 2024-07-26T11:18:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15726.481183 secs 2024-07-26T11:18:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15744.219721 secs 2024-07-26T11:18:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15726.481183 secs 2024-07-26T11:18:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15744.219721 secs 2024-07-26T11:18:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:17 smithi164 ceph-mon[36970]: pgmap v8897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:18:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:17 smithi163 ceph-mon[36873]: pgmap v8897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:18:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:17 smithi042 ceph-mon[32775]: pgmap v8897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:18:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:19 smithi164 ceph-mon[36970]: pgmap v8898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:18:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:19 smithi163 ceph-mon[36873]: pgmap v8898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:18:20.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:19 smithi042 ceph-mon[32775]: pgmap v8898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:18:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:21 smithi164 ceph-mon[36970]: pgmap v8899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:18:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15731.481280 secs 2024-07-26T11:18:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15749.219871 secs 2024-07-26T11:18:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:21 smithi163 ceph-mon[36873]: pgmap v8899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:18:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15731.481280 secs 2024-07-26T11:18:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15749.219871 secs 2024-07-26T11:18:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:21 smithi042 ceph-mon[32775]: pgmap v8899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:18:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15731.481280 secs 2024-07-26T11:18:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15749.219871 secs 2024-07-26T11:18:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:23 smithi164 ceph-mon[36970]: pgmap v8900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:18:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:23 smithi163 ceph-mon[36873]: pgmap v8900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:18:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:23 smithi042 ceph-mon[32775]: pgmap v8900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:18:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:25 smithi164 ceph-mon[36970]: pgmap v8901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:25 smithi163 ceph-mon[36873]: pgmap v8901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:25 smithi042 ceph-mon[32775]: pgmap v8901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15736.481406 secs 2024-07-26T11:18:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15754.220011 secs 2024-07-26T11:18:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15736.481406 secs 2024-07-26T11:18:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15754.220011 secs 2024-07-26T11:18:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15736.481406 secs 2024-07-26T11:18:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15754.220011 secs 2024-07-26T11:18:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:27 smithi164 ceph-mon[36970]: pgmap v8902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:27 smithi163 ceph-mon[36873]: pgmap v8902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:27 smithi042 ceph-mon[32775]: pgmap v8902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:28.504 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:18:28.502+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:18:28.504 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:18:28.502+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:18:28.504 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:18:28.502+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:18:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:29 smithi164 ceph-mon[36970]: pgmap v8903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:18:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:29 smithi163 ceph-mon[36873]: pgmap v8903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:18:30.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:29 smithi042 ceph-mon[32775]: pgmap v8903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:18:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:31 smithi163 ceph-mon[36873]: pgmap v8904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15741.481538 secs 2024-07-26T11:18:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15759.220136 secs 2024-07-26T11:18:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:31 smithi042 ceph-mon[32775]: pgmap v8904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15741.481538 secs 2024-07-26T11:18:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15759.220136 secs 2024-07-26T11:18:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:31 smithi164 ceph-mon[36970]: pgmap v8904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15741.481538 secs 2024-07-26T11:18:32.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15759.220136 secs 2024-07-26T11:18:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:33 smithi163 ceph-mon[36873]: pgmap v8905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:33 smithi042 ceph-mon[32775]: pgmap v8905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:33 smithi164 ceph-mon[36970]: pgmap v8905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:18:35.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:18:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:18:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:35 smithi163 ceph-mon[36873]: pgmap v8906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:18:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:18:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:35 smithi163 ceph-mon[36873]: pgmap v8907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:18:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:35 smithi163 ceph-mon[36873]: pgmap v8908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:36.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:18:36.273 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:35 smithi042 ceph-mon[32775]: pgmap v8906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:36.273 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:18:36.273 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:18:36.273 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:35 smithi042 ceph-mon[32775]: pgmap v8907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:18:36.273 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:35 smithi042 ceph-mon[32775]: pgmap v8908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:36.273 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:18:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:35 smithi164 ceph-mon[36970]: pgmap v8906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:18:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:18:36.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:35 smithi164 ceph-mon[36970]: pgmap v8907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:18:36.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:35 smithi164 ceph-mon[36970]: pgmap v8908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:36.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:18:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15746.481650 secs 2024-07-26T11:18:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15764.220251 secs 2024-07-26T11:18:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15746.481650 secs 2024-07-26T11:18:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15764.220251 secs 2024-07-26T11:18:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15746.481650 secs 2024-07-26T11:18:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15764.220251 secs 2024-07-26T11:18:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:37 smithi163 ceph-mon[36873]: pgmap v8909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:38.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:37 smithi042 ceph-mon[32775]: pgmap v8909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:37 smithi164 ceph-mon[36970]: pgmap v8909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:40 smithi164 ceph-mon[36970]: pgmap v8910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:40 smithi163 ceph-mon[36873]: pgmap v8910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:41.099 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:40 smithi042 ceph-mon[32775]: pgmap v8910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15751.481756 secs 2024-07-26T11:18:41.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15769.220377 secs 2024-07-26T11:18:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15751.481756 secs 2024-07-26T11:18:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15769.220377 secs 2024-07-26T11:18:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15751.481756 secs 2024-07-26T11:18:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15769.220377 secs 2024-07-26T11:18:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:42 smithi164 ceph-mon[36970]: pgmap v8911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:42 smithi163 ceph-mon[36873]: pgmap v8911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:42 smithi042 ceph-mon[32775]: pgmap v8911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:18:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:18:43.504+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:18:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:18:43.504+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:18:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:18:43.504+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:18:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:44 smithi164 ceph-mon[36970]: pgmap v8912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:18:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:44 smithi163 ceph-mon[36873]: pgmap v8912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:18:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:44 smithi042 ceph-mon[32775]: pgmap v8912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:18:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:46 smithi164 ceph-mon[36970]: pgmap v8913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:18:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15756.481880 secs 2024-07-26T11:18:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15774.220495 secs 2024-07-26T11:18:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:46 smithi163 ceph-mon[36873]: pgmap v8913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:18:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15756.481880 secs 2024-07-26T11:18:47.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15774.220495 secs 2024-07-26T11:18:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:46 smithi042 ceph-mon[32775]: pgmap v8913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:18:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15756.481880 secs 2024-07-26T11:18:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15774.220495 secs 2024-07-26T11:18:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:48 smithi164 ceph-mon[36970]: pgmap v8914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:48 smithi163 ceph-mon[36873]: pgmap v8914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:48 smithi042 ceph-mon[32775]: pgmap v8914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:50 smithi164 ceph-mon[36970]: pgmap v8915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:50 smithi163 ceph-mon[36873]: pgmap v8915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:50 smithi042 ceph-mon[32775]: pgmap v8915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15761.482006 secs 2024-07-26T11:18:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15779.220629 secs 2024-07-26T11:18:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15761.482006 secs 2024-07-26T11:18:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15779.220629 secs 2024-07-26T11:18:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15761.482006 secs 2024-07-26T11:18:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15779.220629 secs 2024-07-26T11:18:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:52 smithi164 ceph-mon[36970]: pgmap v8916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:52 smithi163 ceph-mon[36873]: pgmap v8916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:52 smithi042 ceph-mon[32775]: pgmap v8916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:54 smithi164 ceph-mon[36970]: pgmap v8917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:18:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:54 smithi163 ceph-mon[36873]: pgmap v8917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:18:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:54 smithi042 ceph-mon[32775]: pgmap v8917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:18:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:56 smithi164 ceph-mon[36970]: pgmap v8918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15766.482130 secs 2024-07-26T11:18:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15784.220752 secs 2024-07-26T11:18:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:56 smithi163 ceph-mon[36873]: pgmap v8918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15766.482130 secs 2024-07-26T11:18:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15784.220752 secs 2024-07-26T11:18:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:56 smithi042 ceph-mon[32775]: pgmap v8918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:18:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15766.482130 secs 2024-07-26T11:18:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15784.220752 secs 2024-07-26T11:18:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:18:58.510+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:18:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:18:58.510+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:18:58.512 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:18:58.510+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:18:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:18:58 smithi164 ceph-mon[36970]: pgmap v8919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:18:58 smithi163 ceph-mon[36873]: pgmap v8919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:18:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:18:58 smithi042 ceph-mon[32775]: pgmap v8919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:00 smithi164 ceph-mon[36970]: pgmap v8920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:00 smithi163 ceph-mon[36873]: pgmap v8920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:00 smithi042 ceph-mon[32775]: pgmap v8920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15771.482249 secs 2024-07-26T11:19:01.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15789.220871 secs 2024-07-26T11:19:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15771.482249 secs 2024-07-26T11:19:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15789.220871 secs 2024-07-26T11:19:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15771.482249 secs 2024-07-26T11:19:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15789.220871 secs 2024-07-26T11:19:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:02 smithi164 ceph-mon[36970]: pgmap v8921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:02 smithi163 ceph-mon[36873]: pgmap v8921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:02 smithi042 ceph-mon[32775]: pgmap v8921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:04 smithi164 ceph-mon[36970]: pgmap v8922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:19:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:04 smithi163 ceph-mon[36873]: pgmap v8922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:19:05.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:04 smithi042 ceph-mon[32775]: pgmap v8922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:19:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:06 smithi164 ceph-mon[36970]: pgmap v8923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15776.482399 secs 2024-07-26T11:19:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15794.220995 secs 2024-07-26T11:19:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:06 smithi163 ceph-mon[36873]: pgmap v8923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15776.482399 secs 2024-07-26T11:19:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15794.220995 secs 2024-07-26T11:19:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:06 smithi042 ceph-mon[32775]: pgmap v8923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15776.482399 secs 2024-07-26T11:19:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15794.220995 secs 2024-07-26T11:19:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:08 smithi164 ceph-mon[36970]: pgmap v8924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:08 smithi163 ceph-mon[36873]: pgmap v8924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:08 smithi042 ceph-mon[32775]: pgmap v8924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:10 smithi164 ceph-mon[36970]: pgmap v8925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:10 smithi163 ceph-mon[36873]: pgmap v8925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:10 smithi042 ceph-mon[32775]: pgmap v8925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15781.482517 secs 2024-07-26T11:19:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15799.221153 secs 2024-07-26T11:19:12.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15781.482517 secs 2024-07-26T11:19:12.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15799.221153 secs 2024-07-26T11:19:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15781.482517 secs 2024-07-26T11:19:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15799.221153 secs 2024-07-26T11:19:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:12 smithi163 ceph-mon[36873]: pgmap v8926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:12 smithi042 ceph-mon[32775]: pgmap v8926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:12 smithi164 ceph-mon[36970]: pgmap v8926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:13.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:19:13.512+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:19:13.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:19:13.512+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:19:13.514 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:19:13.512+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:19:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:14 smithi163 ceph-mon[36873]: pgmap v8927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:19:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:14 smithi042 ceph-mon[32775]: pgmap v8927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:19:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:14 smithi164 ceph-mon[36970]: pgmap v8927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:19:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:16 smithi163 ceph-mon[36873]: pgmap v8928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15786.482643 secs 2024-07-26T11:19:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15804.221268 secs 2024-07-26T11:19:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:16 smithi042 ceph-mon[32775]: pgmap v8928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15786.482643 secs 2024-07-26T11:19:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15804.221268 secs 2024-07-26T11:19:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:16 smithi164 ceph-mon[36970]: pgmap v8928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15786.482643 secs 2024-07-26T11:19:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15804.221268 secs 2024-07-26T11:19:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:18 smithi163 ceph-mon[36873]: pgmap v8929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:18 smithi042 ceph-mon[32775]: pgmap v8929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:18 smithi164 ceph-mon[36970]: pgmap v8929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:20 smithi163 ceph-mon[36873]: pgmap v8930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:20 smithi042 ceph-mon[32775]: pgmap v8930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:20 smithi164 ceph-mon[36970]: pgmap v8930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:22.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15791.482774 secs 2024-07-26T11:19:22.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15809.221395 secs 2024-07-26T11:19:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15791.482774 secs 2024-07-26T11:19:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15809.221395 secs 2024-07-26T11:19:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15791.482774 secs 2024-07-26T11:19:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15809.221395 secs 2024-07-26T11:19:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:22 smithi163 ceph-mon[36873]: pgmap v8931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:22 smithi042 ceph-mon[32775]: pgmap v8931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:22 smithi164 ceph-mon[36970]: pgmap v8931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:24 smithi163 ceph-mon[36873]: pgmap v8932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:19:25.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:24 smithi042 ceph-mon[32775]: pgmap v8932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:19:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:24 smithi164 ceph-mon[36970]: pgmap v8932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:19:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:26 smithi163 ceph-mon[36873]: pgmap v8933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15796.482897 secs 2024-07-26T11:19:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15814.221496 secs 2024-07-26T11:19:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:26 smithi042 ceph-mon[32775]: pgmap v8933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15796.482897 secs 2024-07-26T11:19:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15814.221496 secs 2024-07-26T11:19:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:26 smithi164 ceph-mon[36970]: pgmap v8933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15796.482897 secs 2024-07-26T11:19:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15814.221496 secs 2024-07-26T11:19:28.519 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:19:28.518+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:19:28.520 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:19:28.518+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:19:28.520 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:19:28.518+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:19:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:28 smithi042 ceph-mon[32775]: pgmap v8934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:28 smithi164 ceph-mon[36970]: pgmap v8934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:28 smithi163 ceph-mon[36873]: pgmap v8934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:30 smithi042 ceph-mon[32775]: pgmap v8935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:30 smithi164 ceph-mon[36970]: pgmap v8935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:31.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:30 smithi163 ceph-mon[36873]: pgmap v8935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15801.483017 secs 2024-07-26T11:19:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15819.221621 secs 2024-07-26T11:19:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15801.483017 secs 2024-07-26T11:19:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15819.221621 secs 2024-07-26T11:19:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15801.483017 secs 2024-07-26T11:19:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15819.221621 secs 2024-07-26T11:19:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:32 smithi042 ceph-mon[32775]: pgmap v8936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:32 smithi164 ceph-mon[36970]: pgmap v8936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:32 smithi163 ceph-mon[36873]: pgmap v8936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:35.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:34 smithi042 ceph-mon[32775]: pgmap v8937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:19:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:34 smithi164 ceph-mon[36970]: pgmap v8937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:19:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:34 smithi163 ceph-mon[36873]: pgmap v8937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:19:36.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:19:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:19:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:19:37.088 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:36 smithi042 ceph-mon[32775]: pgmap v8938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:37.088 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15806.483144 secs 2024-07-26T11:19:37.088 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15824.221757 secs 2024-07-26T11:19:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:36 smithi164 ceph-mon[36970]: pgmap v8938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15806.483144 secs 2024-07-26T11:19:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15824.221757 secs 2024-07-26T11:19:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:36 smithi163 ceph-mon[36873]: pgmap v8938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15806.483144 secs 2024-07-26T11:19:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15824.221757 secs 2024-07-26T11:19:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:38 smithi042 ceph-mon[32775]: pgmap v8939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:38 smithi164 ceph-mon[36970]: pgmap v8939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:38 smithi163 ceph-mon[36873]: pgmap v8939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:19:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:19:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:19:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:19:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:40 smithi042 ceph-mon[32775]: pgmap v8940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:40.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:40 smithi042 ceph-mon[32775]: pgmap v8941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:40.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:19:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:19:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:19:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:19:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:19:40.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:40 smithi164 ceph-mon[36970]: pgmap v8940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:40.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:40 smithi164 ceph-mon[36970]: pgmap v8941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:40.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:19:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:19:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:19:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:19:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:19:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:40 smithi163 ceph-mon[36873]: pgmap v8940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:40 smithi163 ceph-mon[36873]: pgmap v8941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:19:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15811.483269 secs 2024-07-26T11:19:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15811.483269 secs 2024-07-26T11:19:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15811.483269 secs 2024-07-26T11:19:42.558 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:42 smithi042 ceph-mon[32775]: pgmap v8942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:42.558 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15829.221883 secs 2024-07-26T11:19:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:42 smithi164 ceph-mon[36970]: pgmap v8942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15829.221883 secs 2024-07-26T11:19:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:42 smithi163 ceph-mon[36873]: pgmap v8942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15829.221883 secs 2024-07-26T11:19:43.521 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:19:43.519+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:19:43.521 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:19:43.519+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:19:43.521 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:19:43.519+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:19:44.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:44 smithi042 ceph-mon[32775]: pgmap v8943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:44 smithi164 ceph-mon[36970]: pgmap v8943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:44 smithi163 ceph-mon[36873]: pgmap v8943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:46 smithi042 ceph-mon[32775]: pgmap v8944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15816.483398 secs 2024-07-26T11:19:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:46 smithi164 ceph-mon[36970]: pgmap v8944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15816.483398 secs 2024-07-26T11:19:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:46 smithi163 ceph-mon[36873]: pgmap v8944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15816.483398 secs 2024-07-26T11:19:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15834.222008 secs 2024-07-26T11:19:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15834.222008 secs 2024-07-26T11:19:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15834.222008 secs 2024-07-26T11:19:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:48 smithi042 ceph-mon[32775]: pgmap v8945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:48 smithi164 ceph-mon[36970]: pgmap v8945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:48 smithi163 ceph-mon[36873]: pgmap v8945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:19:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:50 smithi042 ceph-mon[32775]: pgmap v8946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:19:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:50 smithi164 ceph-mon[36970]: pgmap v8946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:19:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:50 smithi163 ceph-mon[36873]: pgmap v8946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:19:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15821.483507 secs 2024-07-26T11:19:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15821.483507 secs 2024-07-26T11:19:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15821.483507 secs 2024-07-26T11:19:52.597 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:52 smithi042 ceph-mon[32775]: pgmap v8947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:52.597 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15839.222111 secs 2024-07-26T11:19:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:52 smithi164 ceph-mon[36970]: pgmap v8947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15839.222111 secs 2024-07-26T11:19:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:52 smithi163 ceph-mon[36873]: pgmap v8947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15839.222111 secs 2024-07-26T11:19:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:54 smithi042 ceph-mon[32775]: pgmap v8948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:19:54.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:54 smithi164 ceph-mon[36970]: pgmap v8948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:19:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:54 smithi163 ceph-mon[36873]: pgmap v8948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:19:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:56 smithi042 ceph-mon[32775]: pgmap v8949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15826.483615 secs 2024-07-26T11:19:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:56 smithi164 ceph-mon[36970]: pgmap v8949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15826.483615 secs 2024-07-26T11:19:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:56 smithi163 ceph-mon[36873]: pgmap v8949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:19:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15826.483615 secs 2024-07-26T11:19:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15844.222175 secs 2024-07-26T11:19:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15844.222175 secs 2024-07-26T11:19:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15844.222175 secs 2024-07-26T11:19:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:19:58.524+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:19:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:19:58.524+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:19:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:19:58.524+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:19:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:19:58 smithi042 ceph-mon[32775]: pgmap v8950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:19:58 smithi164 ceph-mon[36970]: pgmap v8950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:19:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:19:58 smithi163 ceph-mon[36873]: pgmap v8950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:00.357 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:00.355+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:20:00.358 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:00.355+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:20:00.358 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:00.356+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:20:00.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:00.356+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T11:20:00.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:00.356+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:20:00.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:00.356+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:20:00.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:00.356+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T11:20:00.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:00.356+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:20:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:00 smithi042 ceph-mon[32775]: pgmap v8951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:20:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:00 smithi164 ceph-mon[36970]: pgmap v8951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:20:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:00 smithi163 ceph-mon[36873]: pgmap v8951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:20:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15831.483733 secs 2024-07-26T11:20:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15831.483733 secs 2024-07-26T11:20:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15831.483733 secs 2024-07-26T11:20:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:02 smithi042 ceph-mon[32775]: pgmap v8952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15849.222299 secs 2024-07-26T11:20:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:02 smithi164 ceph-mon[36970]: pgmap v8952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15849.222299 secs 2024-07-26T11:20:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:02 smithi163 ceph-mon[36873]: pgmap v8952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15849.222299 secs 2024-07-26T11:20:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:04 smithi042 ceph-mon[32775]: pgmap v8953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:20:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:04 smithi164 ceph-mon[36970]: pgmap v8953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:20:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:04 smithi163 ceph-mon[36873]: pgmap v8953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:20:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:06 smithi042 ceph-mon[32775]: pgmap v8954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15836.483880 secs 2024-07-26T11:20:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:06 smithi164 ceph-mon[36970]: pgmap v8954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15836.483880 secs 2024-07-26T11:20:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:06 smithi163 ceph-mon[36873]: pgmap v8954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15836.483880 secs 2024-07-26T11:20:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15854.222365 secs 2024-07-26T11:20:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15854.222365 secs 2024-07-26T11:20:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15854.222365 secs 2024-07-26T11:20:08.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:08 smithi042 ceph-mon[32775]: pgmap v8955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:08 smithi164 ceph-mon[36970]: pgmap v8955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:08 smithi163 ceph-mon[36873]: pgmap v8955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:10 smithi042 ceph-mon[32775]: pgmap v8956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:10 smithi164 ceph-mon[36970]: pgmap v8956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:10.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:10 smithi163 ceph-mon[36873]: pgmap v8956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:11.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15841.484002 secs 2024-07-26T11:20:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15841.484002 secs 2024-07-26T11:20:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15841.484002 secs 2024-07-26T11:20:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:12 smithi042 ceph-mon[32775]: pgmap v8957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15859.222467 secs 2024-07-26T11:20:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:12 smithi164 ceph-mon[36970]: pgmap v8957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15859.222467 secs 2024-07-26T11:20:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:12 smithi163 ceph-mon[36873]: pgmap v8957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15859.222467 secs 2024-07-26T11:20:13.527 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:13.525+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:20:13.527 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:13.525+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:20:13.527 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:13.525+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:20:14.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:14 smithi042 ceph-mon[32775]: pgmap v8958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:20:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:14 smithi164 ceph-mon[36970]: pgmap v8958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:20:14.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:14 smithi163 ceph-mon[36873]: pgmap v8958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:20:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:16 smithi042 ceph-mon[32775]: pgmap v8959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15846.484127 secs 2024-07-26T11:20:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:16 smithi164 ceph-mon[36970]: pgmap v8959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15846.484127 secs 2024-07-26T11:20:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:16 smithi163 ceph-mon[36873]: pgmap v8959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15846.484127 secs 2024-07-26T11:20:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15864.222603 secs 2024-07-26T11:20:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15864.222603 secs 2024-07-26T11:20:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15864.222603 secs 2024-07-26T11:20:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:18 smithi164 ceph-mon[36970]: pgmap v8960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:18 smithi163 ceph-mon[36873]: pgmap v8960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:18 smithi042 ceph-mon[32775]: pgmap v8960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:20 smithi164 ceph-mon[36970]: pgmap v8961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:20 smithi163 ceph-mon[36873]: pgmap v8961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:20 smithi042 ceph-mon[32775]: pgmap v8961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15851.484294 secs 2024-07-26T11:20:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15851.484294 secs 2024-07-26T11:20:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15851.484294 secs 2024-07-26T11:20:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:22 smithi164 ceph-mon[36970]: pgmap v8962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15869.222725 secs 2024-07-26T11:20:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:22 smithi163 ceph-mon[36873]: pgmap v8962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15869.222725 secs 2024-07-26T11:20:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:22 smithi042 ceph-mon[32775]: pgmap v8962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15869.222725 secs 2024-07-26T11:20:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:24 smithi164 ceph-mon[36970]: pgmap v8963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:20:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:24 smithi163 ceph-mon[36873]: pgmap v8963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:20:24.817 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:24 smithi042 ceph-mon[32775]: pgmap v8963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:20:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:26 smithi164 ceph-mon[36970]: pgmap v8964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15856.484423 secs 2024-07-26T11:20:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:26 smithi163 ceph-mon[36873]: pgmap v8964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15856.484423 secs 2024-07-26T11:20:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:26 smithi042 ceph-mon[32775]: pgmap v8964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15856.484423 secs 2024-07-26T11:20:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15874.222867 secs 2024-07-26T11:20:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15874.222867 secs 2024-07-26T11:20:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15874.222867 secs 2024-07-26T11:20:28.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:28.530+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:20:28.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:28.530+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:20:28.532 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:28.530+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:20:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:28 smithi164 ceph-mon[36970]: pgmap v8965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:28 smithi163 ceph-mon[36873]: pgmap v8965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:28.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:28 smithi042 ceph-mon[32775]: pgmap v8965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:30 smithi164 ceph-mon[36970]: pgmap v8966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:30 smithi163 ceph-mon[36873]: pgmap v8966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:30 smithi042 ceph-mon[32775]: pgmap v8966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15861.484544 secs 2024-07-26T11:20:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15861.484544 secs 2024-07-26T11:20:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15861.484544 secs 2024-07-26T11:20:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:32 smithi164 ceph-mon[36970]: pgmap v8967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15879.222994 secs 2024-07-26T11:20:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:32 smithi163 ceph-mon[36873]: pgmap v8967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15879.222994 secs 2024-07-26T11:20:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:32 smithi042 ceph-mon[32775]: pgmap v8967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15879.222994 secs 2024-07-26T11:20:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:34 smithi164 ceph-mon[36970]: pgmap v8968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:20:34.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:34 smithi163 ceph-mon[36873]: pgmap v8968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:20:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:34 smithi042 ceph-mon[32775]: pgmap v8968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:20:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:36 smithi164 ceph-mon[36970]: pgmap v8969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15866.484685 secs 2024-07-26T11:20:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:36 smithi163 ceph-mon[36873]: pgmap v8969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15866.484685 secs 2024-07-26T11:20:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:36 smithi042 ceph-mon[32775]: pgmap v8969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15866.484685 secs 2024-07-26T11:20:37.721 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15884.223117 secs 2024-07-26T11:20:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15884.223117 secs 2024-07-26T11:20:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15884.223117 secs 2024-07-26T11:20:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:38 smithi164 ceph-mon[36970]: pgmap v8970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:38 smithi163 ceph-mon[36873]: pgmap v8970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:38 smithi042 ceph-mon[32775]: pgmap v8970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:39 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:20:39.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:39 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:20:39.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:39 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:20:40.586 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:40 smithi163 ceph-mon[36873]: pgmap v8971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:40.587 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:40.587 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:40.587 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:40.587 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:40 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:40.594 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:40 smithi164 ceph-mon[36970]: pgmap v8971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:40.594 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:40.594 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:40.594 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:40.594 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:40 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:40 smithi042 ceph-mon[32775]: pgmap v8971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:40 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:41.616 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:41 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:20:41.616 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:41 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:20:41.616 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15871.484804 secs 2024-07-26T11:20:41.641 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:41 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:20:41.642 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:41 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:20:41.642 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15871.484804 secs 2024-07-26T11:20:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:41 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:20:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:41 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:20:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15871.484804 secs 2024-07-26T11:20:42.716 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:42 smithi164 ceph-mon[36970]: pgmap v8972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:42.716 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15889.223242 secs 2024-07-26T11:20:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:42 smithi163 ceph-mon[36873]: pgmap v8972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15889.223242 secs 2024-07-26T11:20:42.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:42 smithi042 ceph-mon[32775]: pgmap v8972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:42.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15889.223242 secs 2024-07-26T11:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:43.531+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:43.531+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:20:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:43.531+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:20:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:20:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:20:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:20:44.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:20:44.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:20:44.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:20:44.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:20:44.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:20:44.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:20:44.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:20:44.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: pgmap v8973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:44.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: pgmap v8974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:20:44.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:20:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:20:44.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:20:44.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:20:44.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:20:44.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:20:44.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:20:44.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:20:44.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:20:44.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:20:44.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:20:44.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: pgmap v8973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:44.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: pgmap v8974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:20:44.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:20:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:20:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:20:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:20:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:20:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:20:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:20:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:20:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:20:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:20:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:20:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:20:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: pgmap v8973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: pgmap v8974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:20:44.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:20:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:46 smithi164 ceph-mon[36970]: pgmap v8975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:20:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15876.484933 secs 2024-07-26T11:20:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:46 smithi163 ceph-mon[36873]: pgmap v8975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:20:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15876.484933 secs 2024-07-26T11:20:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:46 smithi042 ceph-mon[32775]: pgmap v8975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:20:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15876.484933 secs 2024-07-26T11:20:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15894.223305 secs 2024-07-26T11:20:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15894.223305 secs 2024-07-26T11:20:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15894.223305 secs 2024-07-26T11:20:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:48 smithi163 ceph-mon[36873]: pgmap v8976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:20:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:48 smithi042 ceph-mon[32775]: pgmap v8976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:20:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:48 smithi164 ceph-mon[36970]: pgmap v8976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:20:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:50 smithi163 ceph-mon[36873]: pgmap v8977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:20:50.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:50 smithi042 ceph-mon[32775]: pgmap v8977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:20:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:50 smithi164 ceph-mon[36970]: pgmap v8977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:20:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15881.485059 secs 2024-07-26T11:20:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15881.485059 secs 2024-07-26T11:20:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15881.485059 secs 2024-07-26T11:20:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:52 smithi163 ceph-mon[36873]: pgmap v8978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:20:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15899.223391 secs 2024-07-26T11:20:52.511 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:52 smithi164 ceph-mon[36970]: pgmap v8978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:20:52.511 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15899.223391 secs 2024-07-26T11:20:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:52 smithi042 ceph-mon[32775]: pgmap v8978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:20:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15899.223391 secs 2024-07-26T11:20:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:54 smithi164 ceph-mon[36970]: pgmap v8979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:20:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:54 smithi163 ceph-mon[36873]: pgmap v8979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:20:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:54 smithi042 ceph-mon[32775]: pgmap v8979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:20:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:56 smithi164 ceph-mon[36970]: pgmap v8980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15886.485183 secs 2024-07-26T11:20:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15904.223461 secs 2024-07-26T11:20:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:56 smithi163 ceph-mon[36873]: pgmap v8980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15886.485183 secs 2024-07-26T11:20:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15904.223461 secs 2024-07-26T11:20:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:56 smithi042 ceph-mon[32775]: pgmap v8980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:20:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15886.485183 secs 2024-07-26T11:20:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15904.223461 secs 2024-07-26T11:20:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:58.537+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:20:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:58.537+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:20:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:20:58.537+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:20:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:20:58 smithi164 ceph-mon[36970]: pgmap v8981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:20:58 smithi163 ceph-mon[36873]: pgmap v8981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:20:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:20:58 smithi042 ceph-mon[32775]: pgmap v8981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:00 smithi164 ceph-mon[36970]: pgmap v8982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:00 smithi163 ceph-mon[36873]: pgmap v8982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:00 smithi042 ceph-mon[32775]: pgmap v8982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15891.485310 secs 2024-07-26T11:21:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15909.223582 secs 2024-07-26T11:21:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15891.485310 secs 2024-07-26T11:21:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15909.223582 secs 2024-07-26T11:21:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15891.485310 secs 2024-07-26T11:21:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15909.223582 secs 2024-07-26T11:21:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:02 smithi164 ceph-mon[36970]: pgmap v8983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:02 smithi163 ceph-mon[36873]: pgmap v8983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:02 smithi042 ceph-mon[32775]: pgmap v8983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:04 smithi164 ceph-mon[36970]: pgmap v8984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:04 smithi163 ceph-mon[36873]: pgmap v8984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:05.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:04 smithi042 ceph-mon[32775]: pgmap v8984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:06 smithi164 ceph-mon[36970]: pgmap v8985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15896.485431 secs 2024-07-26T11:21:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15914.223703 secs 2024-07-26T11:21:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:06 smithi163 ceph-mon[36873]: pgmap v8985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15896.485431 secs 2024-07-26T11:21:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15914.223703 secs 2024-07-26T11:21:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:06 smithi042 ceph-mon[32775]: pgmap v8985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15896.485431 secs 2024-07-26T11:21:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15914.223703 secs 2024-07-26T11:21:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:08 smithi164 ceph-mon[36970]: pgmap v8986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:08 smithi163 ceph-mon[36873]: pgmap v8986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:08 smithi042 ceph-mon[32775]: pgmap v8986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:10 smithi164 ceph-mon[36970]: pgmap v8987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:10 smithi163 ceph-mon[36873]: pgmap v8987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:10 smithi042 ceph-mon[32775]: pgmap v8987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15901.485554 secs 2024-07-26T11:21:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15919.223825 secs 2024-07-26T11:21:12.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15901.485554 secs 2024-07-26T11:21:12.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15919.223825 secs 2024-07-26T11:21:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15901.485554 secs 2024-07-26T11:21:12.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15919.223825 secs 2024-07-26T11:21:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:12 smithi163 ceph-mon[36873]: pgmap v8988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:12 smithi042 ceph-mon[32775]: pgmap v8988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:12 smithi164 ceph-mon[36970]: pgmap v8988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:13.540 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:21:13.538+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:21:13.540 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:21:13.538+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:21:13.540 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:21:13.538+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:21:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:14 smithi163 ceph-mon[36873]: pgmap v8989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:14 smithi042 ceph-mon[32775]: pgmap v8989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:14 smithi164 ceph-mon[36970]: pgmap v8989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:16 smithi163 ceph-mon[36873]: pgmap v8990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15906.485683 secs 2024-07-26T11:21:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15924.223962 secs 2024-07-26T11:21:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:16 smithi042 ceph-mon[32775]: pgmap v8990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15906.485683 secs 2024-07-26T11:21:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15924.223962 secs 2024-07-26T11:21:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:16 smithi164 ceph-mon[36970]: pgmap v8990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15906.485683 secs 2024-07-26T11:21:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15924.223962 secs 2024-07-26T11:21:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:18 smithi163 ceph-mon[36873]: pgmap v8991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:18 smithi042 ceph-mon[32775]: pgmap v8991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:18 smithi164 ceph-mon[36970]: pgmap v8991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:20 smithi163 ceph-mon[36873]: pgmap v8992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:20 smithi042 ceph-mon[32775]: pgmap v8992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:20 smithi164 ceph-mon[36970]: pgmap v8992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15911.485769 secs 2024-07-26T11:21:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15929.224083 secs 2024-07-26T11:21:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15911.485769 secs 2024-07-26T11:21:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15929.224083 secs 2024-07-26T11:21:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15911.485769 secs 2024-07-26T11:21:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15929.224083 secs 2024-07-26T11:21:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:22 smithi163 ceph-mon[36873]: pgmap v8993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:22 smithi042 ceph-mon[32775]: pgmap v8993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:22 smithi164 ceph-mon[36970]: pgmap v8993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:24 smithi163 ceph-mon[36873]: pgmap v8994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:25.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:24 smithi042 ceph-mon[32775]: pgmap v8994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:24 smithi164 ceph-mon[36970]: pgmap v8994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:26 smithi163 ceph-mon[36873]: pgmap v8995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15916.485893 secs 2024-07-26T11:21:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15934.224208 secs 2024-07-26T11:21:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:26 smithi042 ceph-mon[32775]: pgmap v8995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15916.485893 secs 2024-07-26T11:21:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15934.224208 secs 2024-07-26T11:21:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:26 smithi164 ceph-mon[36970]: pgmap v8995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15916.485893 secs 2024-07-26T11:21:27.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15934.224208 secs 2024-07-26T11:21:28.545 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:21:28.544+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:21:28.545 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:21:28.544+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:21:28.546 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:21:28.544+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:21:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:28 smithi163 ceph-mon[36873]: pgmap v8996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:28 smithi042 ceph-mon[32775]: pgmap v8996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:28 smithi164 ceph-mon[36970]: pgmap v8996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:30 smithi042 ceph-mon[32775]: pgmap v8997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:30 smithi164 ceph-mon[36970]: pgmap v8997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:31.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:30 smithi163 ceph-mon[36873]: pgmap v8997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15921.486042 secs 2024-07-26T11:21:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15939.224310 secs 2024-07-26T11:21:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15921.486042 secs 2024-07-26T11:21:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15939.224310 secs 2024-07-26T11:21:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15921.486042 secs 2024-07-26T11:21:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15939.224310 secs 2024-07-26T11:21:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:32 smithi042 ceph-mon[32775]: pgmap v8998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:32 smithi164 ceph-mon[36970]: pgmap v8998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:32 smithi163 ceph-mon[36873]: pgmap v8998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:35.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:34 smithi042 ceph-mon[32775]: pgmap v8999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:34 smithi164 ceph-mon[36970]: pgmap v8999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:34 smithi163 ceph-mon[36873]: pgmap v8999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:37.026 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:36 smithi164 ceph-mon[36970]: pgmap v9000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:37.026 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15926.486143 secs 2024-07-26T11:21:37.026 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15944.224400 secs 2024-07-26T11:21:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:36 smithi042 ceph-mon[32775]: pgmap v9000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15926.486143 secs 2024-07-26T11:21:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15944.224400 secs 2024-07-26T11:21:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:36 smithi163 ceph-mon[36873]: pgmap v9000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15926.486143 secs 2024-07-26T11:21:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15944.224400 secs 2024-07-26T11:21:39.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:38 smithi042 ceph-mon[32775]: pgmap v9001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:38 smithi164 ceph-mon[36970]: pgmap v9001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:38 smithi163 ceph-mon[36873]: pgmap v9001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:41.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:40 smithi042 ceph-mon[32775]: pgmap v9002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:40 smithi164 ceph-mon[36970]: pgmap v9002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:41.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:40 smithi163 ceph-mon[36873]: pgmap v9002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:42.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15931.486294 secs 2024-07-26T11:21:42.048 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15949.224546 secs 2024-07-26T11:21:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15931.486294 secs 2024-07-26T11:21:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15949.224546 secs 2024-07-26T11:21:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15931.486294 secs 2024-07-26T11:21:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15949.224546 secs 2024-07-26T11:21:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:42 smithi042 ceph-mon[32775]: pgmap v9003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:42 smithi164 ceph-mon[36970]: pgmap v9003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:42 smithi163 ceph-mon[36873]: pgmap v9003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:21:43.544+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:21:43.548 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:21:43.544+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:21:43.548 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:21:43.544+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:21:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:21:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:21:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:21:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:43 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:21:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:21:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:21:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:21:44.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:43 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:21:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:21:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:21:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:21:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:43 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:21:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:44 smithi042 ceph-mon[32775]: pgmap v9004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:44 smithi042 ceph-mon[32775]: pgmap v9005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:21:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:44 smithi042 ceph-mon[32775]: pgmap v9006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:21:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:44 smithi164 ceph-mon[36970]: pgmap v9004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:45.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:44 smithi164 ceph-mon[36970]: pgmap v9005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:21:45.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:44 smithi164 ceph-mon[36970]: pgmap v9006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:21:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:44 smithi163 ceph-mon[36873]: pgmap v9004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:44 smithi163 ceph-mon[36873]: pgmap v9005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:21:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:44 smithi163 ceph-mon[36873]: pgmap v9006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:21:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:46 smithi042 ceph-mon[32775]: pgmap v9007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:21:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15936.486431 secs 2024-07-26T11:21:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15954.224689 secs 2024-07-26T11:21:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:46 smithi164 ceph-mon[36970]: pgmap v9007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:21:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15936.486431 secs 2024-07-26T11:21:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15954.224689 secs 2024-07-26T11:21:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:46 smithi163 ceph-mon[36873]: pgmap v9007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:21:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15936.486431 secs 2024-07-26T11:21:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15954.224689 secs 2024-07-26T11:21:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:48 smithi042 ceph-mon[32775]: pgmap v9008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:21:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:48 smithi164 ceph-mon[36970]: pgmap v9008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:21:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:48 smithi163 ceph-mon[36873]: pgmap v9008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:21:51.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:50 smithi163 ceph-mon[36873]: pgmap v9009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:21:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:50 smithi042 ceph-mon[32775]: pgmap v9009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:21:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:50 smithi164 ceph-mon[36970]: pgmap v9009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:21:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15941.486577 secs 2024-07-26T11:21:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15959.224839 secs 2024-07-26T11:21:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15941.486577 secs 2024-07-26T11:21:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15959.224839 secs 2024-07-26T11:21:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15941.486577 secs 2024-07-26T11:21:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15959.224839 secs 2024-07-26T11:21:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:52 smithi042 ceph-mon[32775]: pgmap v9010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:21:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:52 smithi164 ceph-mon[36970]: pgmap v9010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:21:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:52 smithi163 ceph-mon[36873]: pgmap v9010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:21:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:54 smithi042 ceph-mon[32775]: pgmap v9011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:21:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:54 smithi164 ceph-mon[36970]: pgmap v9011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:21:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:54 smithi163 ceph-mon[36873]: pgmap v9011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:21:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:55 smithi042 ceph-mon[32775]: pgmap v9012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:55 smithi164 ceph-mon[36970]: pgmap v9012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:55 smithi163 ceph-mon[36873]: pgmap v9012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:21:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15946.486679 secs 2024-07-26T11:21:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15964.224965 secs 2024-07-26T11:21:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15946.486679 secs 2024-07-26T11:21:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15964.224965 secs 2024-07-26T11:21:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15946.486679 secs 2024-07-26T11:21:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15964.224965 secs 2024-07-26T11:21:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:21:57 smithi042 ceph-mon[32775]: pgmap v9013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:21:57 smithi164 ceph-mon[36970]: pgmap v9013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:21:57 smithi163 ceph-mon[36873]: pgmap v9013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:21:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:21:58.549+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:21:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:21:58.549+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:21:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:21:58.549+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:22:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:00 smithi042 ceph-mon[32775]: pgmap v9014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:00 smithi164 ceph-mon[36970]: pgmap v9014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:00 smithi163 ceph-mon[36873]: pgmap v9014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15951.486807 secs 2024-07-26T11:22:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15969.225050 secs 2024-07-26T11:22:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15951.486807 secs 2024-07-26T11:22:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15969.225050 secs 2024-07-26T11:22:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15951.486807 secs 2024-07-26T11:22:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15969.225050 secs 2024-07-26T11:22:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:02 smithi042 ceph-mon[32775]: pgmap v9015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:02 smithi164 ceph-mon[36970]: pgmap v9015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:02 smithi163 ceph-mon[36873]: pgmap v9015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:05.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:04 smithi042 ceph-mon[32775]: pgmap v9016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:22:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:04 smithi164 ceph-mon[36970]: pgmap v9016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:22:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:04 smithi163 ceph-mon[36873]: pgmap v9016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:22:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:06 smithi042 ceph-mon[32775]: pgmap v9017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15956.486945 secs 2024-07-26T11:22:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15974.225170 secs 2024-07-26T11:22:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:06 smithi164 ceph-mon[36970]: pgmap v9017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15956.486945 secs 2024-07-26T11:22:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15974.225170 secs 2024-07-26T11:22:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:06 smithi163 ceph-mon[36873]: pgmap v9017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15956.486945 secs 2024-07-26T11:22:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15974.225170 secs 2024-07-26T11:22:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:08 smithi042 ceph-mon[32775]: pgmap v9018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:08 smithi164 ceph-mon[36970]: pgmap v9018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:09.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:08 smithi163 ceph-mon[36873]: pgmap v9018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:10 smithi042 ceph-mon[32775]: pgmap v9019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:10 smithi164 ceph-mon[36970]: pgmap v9019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:10 smithi163 ceph-mon[36873]: pgmap v9019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15961.487061 secs 2024-07-26T11:22:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15979.225298 secs 2024-07-26T11:22:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:11 smithi042 ceph-mon[32775]: pgmap v9020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15961.487061 secs 2024-07-26T11:22:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15979.225298 secs 2024-07-26T11:22:12.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:11 smithi164 ceph-mon[36970]: pgmap v9020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15961.487061 secs 2024-07-26T11:22:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15979.225298 secs 2024-07-26T11:22:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:11 smithi163 ceph-mon[36873]: pgmap v9020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:13.551 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:22:13.550+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:22:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:22:13.550+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:22:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:22:13.550+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:22:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:14 smithi042 ceph-mon[32775]: pgmap v9021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:22:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:14 smithi164 ceph-mon[36970]: pgmap v9021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:22:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:14 smithi163 ceph-mon[36873]: pgmap v9021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:22:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:16 smithi042 ceph-mon[32775]: pgmap v9022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15966.487182 secs 2024-07-26T11:22:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15984.225440 secs 2024-07-26T11:22:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:16 smithi164 ceph-mon[36970]: pgmap v9022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15966.487182 secs 2024-07-26T11:22:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15984.225440 secs 2024-07-26T11:22:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:16 smithi163 ceph-mon[36873]: pgmap v9022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15966.487182 secs 2024-07-26T11:22:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15984.225440 secs 2024-07-26T11:22:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:18 smithi042 ceph-mon[32775]: pgmap v9023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:18 smithi164 ceph-mon[36970]: pgmap v9023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:19.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:18 smithi163 ceph-mon[36873]: pgmap v9023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:21.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:20 smithi163 ceph-mon[36873]: pgmap v9024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:20 smithi042 ceph-mon[32775]: pgmap v9024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:20 smithi164 ceph-mon[36970]: pgmap v9024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15971.487334 secs 2024-07-26T11:22:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15989.225584 secs 2024-07-26T11:22:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15971.487334 secs 2024-07-26T11:22:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15989.225584 secs 2024-07-26T11:22:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15971.487334 secs 2024-07-26T11:22:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15989.225584 secs 2024-07-26T11:22:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:22 smithi042 ceph-mon[32775]: pgmap v9025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:22 smithi164 ceph-mon[36970]: pgmap v9025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:22 smithi163 ceph-mon[36873]: pgmap v9025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:23 smithi042 ceph-mon[32775]: pgmap v9026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:22:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:23 smithi164 ceph-mon[36970]: pgmap v9026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:22:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:23 smithi163 ceph-mon[36873]: pgmap v9026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:22:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:26 smithi042 ceph-mon[32775]: pgmap v9027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15976.487460 secs 2024-07-26T11:22:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15994.225702 secs 2024-07-26T11:22:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:26 smithi164 ceph-mon[36970]: pgmap v9027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15976.487460 secs 2024-07-26T11:22:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15994.225702 secs 2024-07-26T11:22:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:26 smithi163 ceph-mon[36873]: pgmap v9027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15976.487460 secs 2024-07-26T11:22:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15994.225702 secs 2024-07-26T11:22:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:22:28.552+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:22:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:22:28.552+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:22:28.554 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:22:28.552+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:22:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:28 smithi042 ceph-mon[32775]: pgmap v9028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:28 smithi164 ceph-mon[36970]: pgmap v9028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:28 smithi163 ceph-mon[36873]: pgmap v9028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:30 smithi042 ceph-mon[32775]: pgmap v9029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:30 smithi164 ceph-mon[36970]: pgmap v9029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:31.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:30 smithi163 ceph-mon[36873]: pgmap v9029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15981.487582 secs 2024-07-26T11:22:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15999.225826 secs 2024-07-26T11:22:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15981.487582 secs 2024-07-26T11:22:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15999.225826 secs 2024-07-26T11:22:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15981.487582 secs 2024-07-26T11:22:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15999.225826 secs 2024-07-26T11:22:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:32 smithi042 ceph-mon[32775]: pgmap v9030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:32 smithi164 ceph-mon[36970]: pgmap v9030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:32 smithi163 ceph-mon[36873]: pgmap v9030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:35.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:34 smithi042 ceph-mon[32775]: pgmap v9031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:22:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:34 smithi164 ceph-mon[36970]: pgmap v9031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:22:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:34 smithi163 ceph-mon[36873]: pgmap v9031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:22:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:35 smithi042 ceph-mon[32775]: pgmap v9032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:35 smithi164 ceph-mon[36970]: pgmap v9032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:35 smithi163 ceph-mon[36873]: pgmap v9032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:37.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15986.487704 secs 2024-07-26T11:22:37.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16004.225968 secs 2024-07-26T11:22:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15986.487704 secs 2024-07-26T11:22:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16004.225968 secs 2024-07-26T11:22:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15986.487704 secs 2024-07-26T11:22:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16004.225968 secs 2024-07-26T11:22:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:37 smithi042 ceph-mon[32775]: pgmap v9033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:37 smithi164 ceph-mon[36970]: pgmap v9033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:37 smithi163 ceph-mon[36873]: pgmap v9033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:41.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:40 smithi042 ceph-mon[32775]: pgmap v9034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:40 smithi164 ceph-mon[36970]: pgmap v9034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:41.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:40 smithi163 ceph-mon[36873]: pgmap v9034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:22:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15991.487848 secs 2024-07-26T11:22:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16009.226093 secs 2024-07-26T11:22:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15991.487848 secs 2024-07-26T11:22:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16009.226093 secs 2024-07-26T11:22:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15991.487848 secs 2024-07-26T11:22:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16009.226093 secs 2024-07-26T11:22:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:42 smithi042 ceph-mon[32775]: pgmap v9035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:42 smithi164 ceph-mon[36970]: pgmap v9035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:42 smithi163 ceph-mon[36873]: pgmap v9035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:22:43.553+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:22:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:22:43.553+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:22:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:22:43.553+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:22:45.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:44 smithi042 ceph-mon[32775]: pgmap v9036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:22:45.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:22:45.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:22:45.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:22:45.105 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:22:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:44 smithi164 ceph-mon[36970]: pgmap v9036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:22:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:22:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:22:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:22:45.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:22:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:44 smithi163 ceph-mon[36873]: pgmap v9036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:22:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:22:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:22:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:22:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:22:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:45 smithi042 ceph-mon[32775]: pgmap v9037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:22:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:45 smithi042 ceph-mon[32775]: pgmap v9038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:22:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:45 smithi164 ceph-mon[36970]: pgmap v9037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:22:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:45 smithi164 ceph-mon[36970]: pgmap v9038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:22:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:45 smithi163 ceph-mon[36873]: pgmap v9037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:22:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:45 smithi163 ceph-mon[36873]: pgmap v9038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:22:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 15996.487949 secs 2024-07-26T11:22:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:46 smithi042 ceph-mon[32775]: pgmap v9039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:22:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16014.226189 secs 2024-07-26T11:22:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 15996.487949 secs 2024-07-26T11:22:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:46 smithi164 ceph-mon[36970]: pgmap v9039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:22:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16014.226189 secs 2024-07-26T11:22:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 15996.487949 secs 2024-07-26T11:22:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:46 smithi163 ceph-mon[36873]: pgmap v9039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:22:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16014.226189 secs 2024-07-26T11:22:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:49 smithi042 ceph-mon[32775]: pgmap v9040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:22:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:49 smithi164 ceph-mon[36970]: pgmap v9040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:22:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:49 smithi163 ceph-mon[36873]: pgmap v9040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:22:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:51 smithi042 ceph-mon[32775]: pgmap v9041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:22:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16001.488103 secs 2024-07-26T11:22:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:51 smithi164 ceph-mon[36970]: pgmap v9041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:22:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16001.488103 secs 2024-07-26T11:22:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:51 smithi163 ceph-mon[36873]: pgmap v9041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:22:51.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16001.488103 secs 2024-07-26T11:22:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16019.226318 secs 2024-07-26T11:22:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16019.226318 secs 2024-07-26T11:22:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16019.226318 secs 2024-07-26T11:22:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:53 smithi042 ceph-mon[32775]: pgmap v9042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:22:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:53 smithi164 ceph-mon[36970]: pgmap v9042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:22:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:53 smithi163 ceph-mon[36873]: pgmap v9042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:22:55.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:55 smithi042 ceph-mon[32775]: pgmap v9043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:22:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:55 smithi164 ceph-mon[36970]: pgmap v9043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:22:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:55 smithi163 ceph-mon[36873]: pgmap v9043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:22:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16006.488251 secs 2024-07-26T11:22:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16006.488251 secs 2024-07-26T11:22:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16006.488251 secs 2024-07-26T11:22:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:57 smithi042 ceph-mon[32775]: pgmap v9044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16024.226415 secs 2024-07-26T11:22:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:57 smithi164 ceph-mon[36970]: pgmap v9044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16024.226415 secs 2024-07-26T11:22:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:57 smithi163 ceph-mon[36873]: pgmap v9044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:22:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16024.226415 secs 2024-07-26T11:22:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:22:58.558+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:22:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:22:58.558+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:22:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:22:58.558+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:22:59.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:22:59 smithi042 ceph-mon[32775]: pgmap v9045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:22:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:22:59 smithi164 ceph-mon[36970]: pgmap v9045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:22:59.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:22:59 smithi163 ceph-mon[36873]: pgmap v9045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:23:00.387 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:23:00.385+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:23:00.387 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:23:00.385+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T11:23:00.387 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:23:00.385+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:23:00.387 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:23:00.385+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T11:23:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:01 smithi042 ceph-mon[32775]: pgmap v9046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16011.488345 secs 2024-07-26T11:23:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:01 smithi164 ceph-mon[36970]: pgmap v9046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:01.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16011.488345 secs 2024-07-26T11:23:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:01 smithi163 ceph-mon[36873]: pgmap v9046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16011.488345 secs 2024-07-26T11:23:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16029.226537 secs 2024-07-26T11:23:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16029.226537 secs 2024-07-26T11:23:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16029.226537 secs 2024-07-26T11:23:03.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:03 smithi042 ceph-mon[32775]: pgmap v9047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:03 smithi164 ceph-mon[36970]: pgmap v9047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:03.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:03 smithi163 ceph-mon[36873]: pgmap v9047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:05 smithi042 ceph-mon[32775]: pgmap v9048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:05 smithi164 ceph-mon[36970]: pgmap v9048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:05 smithi163 ceph-mon[36873]: pgmap v9048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16016.488435 secs 2024-07-26T11:23:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16016.488435 secs 2024-07-26T11:23:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16016.488435 secs 2024-07-26T11:23:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:07 smithi042 ceph-mon[32775]: pgmap v9049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16034.226662 secs 2024-07-26T11:23:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:07 smithi164 ceph-mon[36970]: pgmap v9049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16034.226662 secs 2024-07-26T11:23:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:07 smithi163 ceph-mon[36873]: pgmap v9049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16034.226662 secs 2024-07-26T11:23:09.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:09 smithi042 ceph-mon[32775]: pgmap v9050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:23:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:09 smithi164 ceph-mon[36970]: pgmap v9050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:23:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:09 smithi163 ceph-mon[36873]: pgmap v9050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:23:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:11 smithi164 ceph-mon[36970]: pgmap v9051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:11.741 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16021.488503 secs 2024-07-26T11:23:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:11 smithi163 ceph-mon[36873]: pgmap v9051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16021.488503 secs 2024-07-26T11:23:11.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:11 smithi042 ceph-mon[32775]: pgmap v9051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:11.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16021.488503 secs 2024-07-26T11:23:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16039.226820 secs 2024-07-26T11:23:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16039.226820 secs 2024-07-26T11:23:12.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16039.226820 secs 2024-07-26T11:23:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:23:13.559+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:23:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:23:13.559+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:23:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:23:13.559+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:23:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:13 smithi164 ceph-mon[36970]: pgmap v9052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:13.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:13 smithi163 ceph-mon[36873]: pgmap v9052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:13.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:13 smithi042 ceph-mon[32775]: pgmap v9052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:15 smithi164 ceph-mon[36970]: pgmap v9053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:15.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:15 smithi163 ceph-mon[36873]: pgmap v9053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:15.825 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:15 smithi042 ceph-mon[32775]: pgmap v9053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16026.488571 secs 2024-07-26T11:23:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16026.488571 secs 2024-07-26T11:23:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16026.488571 secs 2024-07-26T11:23:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:17 smithi164 ceph-mon[36970]: pgmap v9054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:17.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16044.226962 secs 2024-07-26T11:23:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:17 smithi163 ceph-mon[36873]: pgmap v9054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16044.226962 secs 2024-07-26T11:23:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:17 smithi042 ceph-mon[32775]: pgmap v9054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16044.226962 secs 2024-07-26T11:23:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:19 smithi164 ceph-mon[36970]: pgmap v9055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:23:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:19 smithi163 ceph-mon[36873]: pgmap v9055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:23:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:19 smithi042 ceph-mon[32775]: pgmap v9055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:23:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:21 smithi164 ceph-mon[36970]: pgmap v9056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16031.488652 secs 2024-07-26T11:23:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:21 smithi163 ceph-mon[36873]: pgmap v9056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16031.488652 secs 2024-07-26T11:23:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:21 smithi042 ceph-mon[32775]: pgmap v9056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16031.488652 secs 2024-07-26T11:23:22.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16049.227078 secs 2024-07-26T11:23:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16049.227078 secs 2024-07-26T11:23:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16049.227078 secs 2024-07-26T11:23:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:23 smithi164 ceph-mon[36970]: pgmap v9057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:23 smithi163 ceph-mon[36873]: pgmap v9057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:23 smithi042 ceph-mon[32775]: pgmap v9057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:25 smithi164 ceph-mon[36970]: pgmap v9058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:25 smithi163 ceph-mon[36873]: pgmap v9058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:25.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:25 smithi042 ceph-mon[32775]: pgmap v9058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16036.488715 secs 2024-07-26T11:23:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16036.488715 secs 2024-07-26T11:23:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16036.488715 secs 2024-07-26T11:23:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:27 smithi164 ceph-mon[36970]: pgmap v9059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16054.227162 secs 2024-07-26T11:23:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:27 smithi163 ceph-mon[36873]: pgmap v9059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16054.227162 secs 2024-07-26T11:23:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:27 smithi042 ceph-mon[32775]: pgmap v9059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16054.227162 secs 2024-07-26T11:23:28.562 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:23:28.561+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:23:28.562 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:23:28.561+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:23:28.562 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:23:28.561+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:23:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:29 smithi163 ceph-mon[36873]: pgmap v9060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:23:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:29 smithi042 ceph-mon[32775]: pgmap v9060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:23:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:29 smithi164 ceph-mon[36970]: pgmap v9060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:23:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:31 smithi163 ceph-mon[36873]: pgmap v9061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16041.488871 secs 2024-07-26T11:23:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16059.227296 secs 2024-07-26T11:23:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:31 smithi042 ceph-mon[32775]: pgmap v9061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16041.488871 secs 2024-07-26T11:23:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16059.227296 secs 2024-07-26T11:23:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:31 smithi164 ceph-mon[36970]: pgmap v9061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16041.488871 secs 2024-07-26T11:23:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16059.227296 secs 2024-07-26T11:23:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:33 smithi163 ceph-mon[36873]: pgmap v9062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:33 smithi042 ceph-mon[32775]: pgmap v9062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:33 smithi164 ceph-mon[36970]: pgmap v9062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:35 smithi163 ceph-mon[36873]: pgmap v9063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:35 smithi042 ceph-mon[32775]: pgmap v9063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:35 smithi164 ceph-mon[36970]: pgmap v9063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16046.488994 secs 2024-07-26T11:23:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16064.227391 secs 2024-07-26T11:23:37.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16046.488994 secs 2024-07-26T11:23:37.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16064.227391 secs 2024-07-26T11:23:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16046.488994 secs 2024-07-26T11:23:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16064.227391 secs 2024-07-26T11:23:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:37 smithi163 ceph-mon[36873]: pgmap v9064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:37 smithi042 ceph-mon[32775]: pgmap v9064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:37 smithi164 ceph-mon[36970]: pgmap v9064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:40.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:39 smithi163 ceph-mon[36873]: pgmap v9065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:23:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:39 smithi042 ceph-mon[32775]: pgmap v9065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:23:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:39 smithi164 ceph-mon[36970]: pgmap v9065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:23:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:41 smithi163 ceph-mon[36873]: pgmap v9066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16051.489114 secs 2024-07-26T11:23:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16069.227518 secs 2024-07-26T11:23:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:41 smithi042 ceph-mon[32775]: pgmap v9066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16051.489114 secs 2024-07-26T11:23:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16069.227518 secs 2024-07-26T11:23:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:41 smithi164 ceph-mon[36970]: pgmap v9066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16051.489114 secs 2024-07-26T11:23:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16069.227518 secs 2024-07-26T11:23:43.563 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:23:43.561+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:23:43.563 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:23:43.561+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:23:43.563 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:23:43.561+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:23:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:43 smithi042 ceph-mon[32775]: pgmap v9067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:43 smithi164 ceph-mon[36970]: pgmap v9067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:43 smithi163 ceph-mon[36873]: pgmap v9067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:45.025 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:44 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:23:45.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:44 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:23:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:44 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:23:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:45 smithi042 ceph-mon[32775]: pgmap v9068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:45 smithi164 ceph-mon[36970]: pgmap v9068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:45 smithi163 ceph-mon[36873]: pgmap v9068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:23:47.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16056.489182 secs 2024-07-26T11:23:47.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:47.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:47.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:47.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:47.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16074.227610 secs 2024-07-26T11:23:47.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:47.057 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16056.489182 secs 2024-07-26T11:23:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16074.227610 secs 2024-07-26T11:23:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:47.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16056.489182 secs 2024-07-26T11:23:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16074.227610 secs 2024-07-26T11:23:47.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:47.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:47 smithi042 ceph-mon[32775]: pgmap v9069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:23:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:23:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:47 smithi042 ceph-mon[32775]: pgmap v9070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:23:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:47 smithi042 ceph-mon[32775]: pgmap v9071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T11:23:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:47 smithi164 ceph-mon[36970]: pgmap v9069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:23:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:23:48.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:47 smithi164 ceph-mon[36970]: pgmap v9070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:23:48.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:47 smithi164 ceph-mon[36970]: pgmap v9071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T11:23:48.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:47 smithi163 ceph-mon[36873]: pgmap v9069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:23:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:23:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:23:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:47 smithi163 ceph-mon[36873]: pgmap v9070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:23:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:47 smithi163 ceph-mon[36873]: pgmap v9071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T11:23:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:23:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:48 smithi042 ceph-mon[32775]: pgmap v9072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:23:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:48 smithi164 ceph-mon[36970]: pgmap v9072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:23:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:48 smithi163 ceph-mon[36873]: pgmap v9072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:23:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:50 smithi042 ceph-mon[32775]: pgmap v9073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:23:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:50 smithi164 ceph-mon[36970]: pgmap v9073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:23:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:50 smithi163 ceph-mon[36873]: pgmap v9073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:23:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16061.489287 secs 2024-07-26T11:23:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16079.227758 secs 2024-07-26T11:23:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16061.489287 secs 2024-07-26T11:23:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16079.227758 secs 2024-07-26T11:23:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16061.489287 secs 2024-07-26T11:23:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16079.227758 secs 2024-07-26T11:23:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:52 smithi042 ceph-mon[32775]: pgmap v9074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:23:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:52 smithi164 ceph-mon[36970]: pgmap v9074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:23:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:52 smithi163 ceph-mon[36873]: pgmap v9074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:23:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:54 smithi042 ceph-mon[32775]: pgmap v9075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:23:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:54 smithi164 ceph-mon[36970]: pgmap v9075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:23:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:54 smithi163 ceph-mon[36873]: pgmap v9075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:23:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16066.489410 secs 2024-07-26T11:23:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16066.489410 secs 2024-07-26T11:23:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16066.489410 secs 2024-07-26T11:23:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16084.227896 secs 2024-07-26T11:23:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:57 smithi164 ceph-mon[36970]: pgmap v9076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:23:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16084.227896 secs 2024-07-26T11:23:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:57 smithi163 ceph-mon[36873]: pgmap v9076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:23:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16084.227896 secs 2024-07-26T11:23:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:57 smithi042 ceph-mon[32775]: pgmap v9076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:23:58.568 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:23:58.566+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:23:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:23:58.566+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:23:58.569 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:23:58.566+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:23:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:23:58 smithi042 ceph-mon[32775]: pgmap v9077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:23:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:23:58 smithi164 ceph-mon[36970]: pgmap v9077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:23:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:23:58 smithi163 ceph-mon[36873]: pgmap v9077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:24:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:00 smithi042 ceph-mon[32775]: pgmap v9078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:00 smithi164 ceph-mon[36970]: pgmap v9078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:00 smithi163 ceph-mon[36873]: pgmap v9078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16071.489535 secs 2024-07-26T11:24:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16089.227974 secs 2024-07-26T11:24:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16071.489535 secs 2024-07-26T11:24:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16089.227974 secs 2024-07-26T11:24:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16071.489535 secs 2024-07-26T11:24:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16089.227974 secs 2024-07-26T11:24:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:02 smithi042 ceph-mon[32775]: pgmap v9079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:02 smithi164 ceph-mon[36970]: pgmap v9079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:02 smithi163 ceph-mon[36873]: pgmap v9079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:05.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:04 smithi042 ceph-mon[32775]: pgmap v9080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:04 smithi164 ceph-mon[36970]: pgmap v9080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:04 smithi163 ceph-mon[36873]: pgmap v9080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16076.489620 secs 2024-07-26T11:24:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16076.489620 secs 2024-07-26T11:24:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16076.489620 secs 2024-07-26T11:24:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16094.228095 secs 2024-07-26T11:24:07.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:07 smithi164 ceph-mon[36970]: pgmap v9081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16094.228095 secs 2024-07-26T11:24:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:07 smithi163 ceph-mon[36873]: pgmap v9081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16094.228095 secs 2024-07-26T11:24:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:07 smithi042 ceph-mon[32775]: pgmap v9081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:08 smithi164 ceph-mon[36970]: pgmap v9082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:24:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:08 smithi042 ceph-mon[32775]: pgmap v9082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:24:09.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:08 smithi163 ceph-mon[36873]: pgmap v9082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:24:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:10 smithi042 ceph-mon[32775]: pgmap v9083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:10 smithi164 ceph-mon[36970]: pgmap v9083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:10 smithi163 ceph-mon[36873]: pgmap v9083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16081.489777 secs 2024-07-26T11:24:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16099.228199 secs 2024-07-26T11:24:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16081.489777 secs 2024-07-26T11:24:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16099.228199 secs 2024-07-26T11:24:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16081.489777 secs 2024-07-26T11:24:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16099.228199 secs 2024-07-26T11:24:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:12 smithi042 ceph-mon[32775]: pgmap v9084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:12 smithi164 ceph-mon[36970]: pgmap v9084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:12 smithi163 ceph-mon[36873]: pgmap v9084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:13.570 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:24:13.568+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:24:13.570 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:24:13.568+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:24:13.570 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:24:13.568+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:24:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:14 smithi042 ceph-mon[32775]: pgmap v9085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:14 smithi164 ceph-mon[36970]: pgmap v9085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:14 smithi163 ceph-mon[36873]: pgmap v9085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16086.489905 secs 2024-07-26T11:24:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16086.489905 secs 2024-07-26T11:24:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16086.489905 secs 2024-07-26T11:24:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16104.228298 secs 2024-07-26T11:24:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:17 smithi164 ceph-mon[36970]: pgmap v9086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16104.228298 secs 2024-07-26T11:24:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:17 smithi163 ceph-mon[36873]: pgmap v9086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16104.228298 secs 2024-07-26T11:24:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:17 smithi042 ceph-mon[32775]: pgmap v9086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:18 smithi042 ceph-mon[32775]: pgmap v9087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:24:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:18 smithi164 ceph-mon[36970]: pgmap v9087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:24:19.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:18 smithi163 ceph-mon[36873]: pgmap v9087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:24:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:20 smithi042 ceph-mon[32775]: pgmap v9088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:20 smithi164 ceph-mon[36970]: pgmap v9088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:20 smithi163 ceph-mon[36873]: pgmap v9088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16091.490022 secs 2024-07-26T11:24:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16109.228471 secs 2024-07-26T11:24:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16091.490022 secs 2024-07-26T11:24:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16109.228471 secs 2024-07-26T11:24:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16091.490022 secs 2024-07-26T11:24:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16109.228471 secs 2024-07-26T11:24:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:22 smithi042 ceph-mon[32775]: pgmap v9089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:22 smithi164 ceph-mon[36970]: pgmap v9089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:22 smithi163 ceph-mon[36873]: pgmap v9089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:25.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:24 smithi042 ceph-mon[32775]: pgmap v9090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:24 smithi164 ceph-mon[36970]: pgmap v9090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:25.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:24 smithi163 ceph-mon[36873]: pgmap v9090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16096.490147 secs 2024-07-26T11:24:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16096.490147 secs 2024-07-26T11:24:26.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16096.490147 secs 2024-07-26T11:24:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16114.228617 secs 2024-07-26T11:24:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:27 smithi164 ceph-mon[36970]: pgmap v9091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16114.228617 secs 2024-07-26T11:24:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:27 smithi163 ceph-mon[36873]: pgmap v9091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16114.228617 secs 2024-07-26T11:24:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:27 smithi042 ceph-mon[32775]: pgmap v9091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:28.575 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:24:28.573+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:24:28.575 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:24:28.573+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:24:28.575 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:24:28.573+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:24:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:28 smithi042 ceph-mon[32775]: pgmap v9092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:24:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:28 smithi164 ceph-mon[36970]: pgmap v9092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:24:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:28 smithi163 ceph-mon[36873]: pgmap v9092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:24:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:30 smithi042 ceph-mon[32775]: pgmap v9093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:30 smithi164 ceph-mon[36970]: pgmap v9093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:31.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:30 smithi163 ceph-mon[36873]: pgmap v9093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16101.490274 secs 2024-07-26T11:24:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16119.228763 secs 2024-07-26T11:24:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16101.490274 secs 2024-07-26T11:24:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16119.228763 secs 2024-07-26T11:24:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16101.490274 secs 2024-07-26T11:24:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16119.228763 secs 2024-07-26T11:24:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:32 smithi164 ceph-mon[36970]: pgmap v9094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:32 smithi163 ceph-mon[36873]: pgmap v9094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:32 smithi042 ceph-mon[32775]: pgmap v9094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:35.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:34 smithi042 ceph-mon[32775]: pgmap v9095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:34 smithi164 ceph-mon[36970]: pgmap v9095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:34 smithi163 ceph-mon[36873]: pgmap v9095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16106.490399 secs 2024-07-26T11:24:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16106.490399 secs 2024-07-26T11:24:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16106.490399 secs 2024-07-26T11:24:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16124.228859 secs 2024-07-26T11:24:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:37 smithi164 ceph-mon[36970]: pgmap v9096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16124.228859 secs 2024-07-26T11:24:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:37 smithi163 ceph-mon[36873]: pgmap v9096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16124.228859 secs 2024-07-26T11:24:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:37 smithi042 ceph-mon[32775]: pgmap v9096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:39.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:38 smithi042 ceph-mon[32775]: pgmap v9097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:24:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:38 smithi164 ceph-mon[36970]: pgmap v9097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:24:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:38 smithi163 ceph-mon[36873]: pgmap v9097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:24:41.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:40 smithi042 ceph-mon[32775]: pgmap v9098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:40 smithi164 ceph-mon[36970]: pgmap v9098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:41.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:40 smithi163 ceph-mon[36873]: pgmap v9098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16111.490518 secs 2024-07-26T11:24:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16129.228939 secs 2024-07-26T11:24:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16111.490518 secs 2024-07-26T11:24:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16129.228939 secs 2024-07-26T11:24:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16111.490518 secs 2024-07-26T11:24:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16129.228939 secs 2024-07-26T11:24:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:42 smithi164 ceph-mon[36970]: pgmap v9099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:42 smithi163 ceph-mon[36873]: pgmap v9099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:43.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:42 smithi042 ceph-mon[32775]: pgmap v9099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:43.576 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:24:43.574+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:24:43.577 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:24:43.574+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:24:43.577 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:24:43.574+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:24:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:44 smithi042 ceph-mon[32775]: pgmap v9100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:44 smithi164 ceph-mon[36970]: pgmap v9100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:44 smithi163 ceph-mon[36873]: pgmap v9100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:24:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16116.490676 secs 2024-07-26T11:24:46.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16116.490676 secs 2024-07-26T11:24:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16116.490676 secs 2024-07-26T11:24:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16134.229007 secs 2024-07-26T11:24:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:47 smithi164 ceph-mon[36970]: pgmap v9101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:24:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16134.229007 secs 2024-07-26T11:24:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:47 smithi163 ceph-mon[36873]: pgmap v9101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:24:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16134.229007 secs 2024-07-26T11:24:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:47 smithi042 ceph-mon[32775]: pgmap v9101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:24:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:24:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:24:48.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:24:48.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:48 smithi164 ceph-mon[36970]: pgmap v9102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:24:48.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:48 smithi164 ceph-mon[36970]: pgmap v9103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:24:48.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:24:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:24:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:48 smithi163 ceph-mon[36873]: pgmap v9102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:24:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:48 smithi163 ceph-mon[36873]: pgmap v9103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:24:48.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:24:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:24:48.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:48 smithi042 ceph-mon[32775]: pgmap v9102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:24:48.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:48 smithi042 ceph-mon[32775]: pgmap v9103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:24:48.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:24:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:50 smithi164 ceph-mon[36970]: pgmap v9104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:24:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:50 smithi163 ceph-mon[36873]: pgmap v9104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:24:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:50 smithi042 ceph-mon[32775]: pgmap v9104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:24:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16121.490819 secs 2024-07-26T11:24:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16139.229130 secs 2024-07-26T11:24:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16121.490819 secs 2024-07-26T11:24:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16139.229130 secs 2024-07-26T11:24:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16121.490819 secs 2024-07-26T11:24:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16139.229130 secs 2024-07-26T11:24:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:52 smithi164 ceph-mon[36970]: pgmap v9105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:24:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:52 smithi163 ceph-mon[36873]: pgmap v9105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:24:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:52 smithi042 ceph-mon[32775]: pgmap v9105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:24:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:54 smithi164 ceph-mon[36970]: pgmap v9106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:24:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:54 smithi163 ceph-mon[36873]: pgmap v9106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:24:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:54 smithi042 ceph-mon[32775]: pgmap v9106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:24:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:56 smithi164 ceph-mon[36970]: pgmap v9107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:24:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16126.490951 secs 2024-07-26T11:24:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16144.229254 secs 2024-07-26T11:24:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:56 smithi163 ceph-mon[36873]: pgmap v9107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:24:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16126.490951 secs 2024-07-26T11:24:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16144.229254 secs 2024-07-26T11:24:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:56 smithi042 ceph-mon[32775]: pgmap v9107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:24:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16126.490951 secs 2024-07-26T11:24:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16144.229254 secs 2024-07-26T11:24:58.579 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:24:58.577+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:24:58.579 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:24:58.577+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:24:58.579 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:24:58.577+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:24:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:24:58 smithi164 ceph-mon[36970]: pgmap v9108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:24:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:24:58 smithi163 ceph-mon[36873]: pgmap v9108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:24:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:24:58 smithi042 ceph-mon[32775]: pgmap v9108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:25:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:00 smithi164 ceph-mon[36970]: pgmap v9109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:00 smithi163 ceph-mon[36873]: pgmap v9109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:01.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:00 smithi042 ceph-mon[32775]: pgmap v9109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16131.491051 secs 2024-07-26T11:25:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16149.229397 secs 2024-07-26T11:25:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16131.491051 secs 2024-07-26T11:25:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16149.229397 secs 2024-07-26T11:25:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16131.491051 secs 2024-07-26T11:25:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16149.229397 secs 2024-07-26T11:25:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:02 smithi164 ceph-mon[36970]: pgmap v9110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:02 smithi163 ceph-mon[36873]: pgmap v9110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:02 smithi042 ceph-mon[32775]: pgmap v9110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:04 smithi164 ceph-mon[36970]: pgmap v9111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:25:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:04 smithi163 ceph-mon[36873]: pgmap v9111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:25:05.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:04 smithi042 ceph-mon[32775]: pgmap v9111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:25:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:06 smithi164 ceph-mon[36970]: pgmap v9112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16136.491181 secs 2024-07-26T11:25:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16154.229518 secs 2024-07-26T11:25:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:06 smithi163 ceph-mon[36873]: pgmap v9112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16136.491181 secs 2024-07-26T11:25:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16154.229518 secs 2024-07-26T11:25:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:06 smithi042 ceph-mon[32775]: pgmap v9112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16136.491181 secs 2024-07-26T11:25:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16154.229518 secs 2024-07-26T11:25:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:08 smithi164 ceph-mon[36970]: pgmap v9113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:08 smithi163 ceph-mon[36873]: pgmap v9113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:08 smithi042 ceph-mon[32775]: pgmap v9113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:10 smithi164 ceph-mon[36970]: pgmap v9114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:10 smithi163 ceph-mon[36873]: pgmap v9114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:10 smithi042 ceph-mon[32775]: pgmap v9114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16141.491310 secs 2024-07-26T11:25:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16159.229662 secs 2024-07-26T11:25:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16141.491310 secs 2024-07-26T11:25:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16159.229662 secs 2024-07-26T11:25:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16141.491310 secs 2024-07-26T11:25:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16159.229662 secs 2024-07-26T11:25:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:12 smithi163 ceph-mon[36873]: pgmap v9115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:13.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:12 smithi042 ceph-mon[32775]: pgmap v9115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:12 smithi164 ceph-mon[36970]: pgmap v9115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:13.578+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:25:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:13.578+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:25:13.579 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:13.578+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:25:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:14 smithi163 ceph-mon[36873]: pgmap v9116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:25:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:14 smithi042 ceph-mon[32775]: pgmap v9116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:25:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:14 smithi164 ceph-mon[36970]: pgmap v9116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:25:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:16 smithi163 ceph-mon[36873]: pgmap v9117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16146.491444 secs 2024-07-26T11:25:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16164.229785 secs 2024-07-26T11:25:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:16 smithi042 ceph-mon[32775]: pgmap v9117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16146.491444 secs 2024-07-26T11:25:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16164.229785 secs 2024-07-26T11:25:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:16 smithi164 ceph-mon[36970]: pgmap v9117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16146.491444 secs 2024-07-26T11:25:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16164.229785 secs 2024-07-26T11:25:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:18 smithi163 ceph-mon[36873]: pgmap v9118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:18 smithi042 ceph-mon[32775]: pgmap v9118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:18 smithi164 ceph-mon[36970]: pgmap v9118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:20 smithi163 ceph-mon[36873]: pgmap v9119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:20 smithi042 ceph-mon[32775]: pgmap v9119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:20 smithi164 ceph-mon[36970]: pgmap v9119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16151.491567 secs 2024-07-26T11:25:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16169.229907 secs 2024-07-26T11:25:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16151.491567 secs 2024-07-26T11:25:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16169.229907 secs 2024-07-26T11:25:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16151.491567 secs 2024-07-26T11:25:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16169.229907 secs 2024-07-26T11:25:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:22 smithi163 ceph-mon[36873]: pgmap v9120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:22 smithi042 ceph-mon[32775]: pgmap v9120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:22 smithi164 ceph-mon[36970]: pgmap v9120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:24 smithi163 ceph-mon[36873]: pgmap v9121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:25:25.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:24 smithi042 ceph-mon[32775]: pgmap v9121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:25:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:24 smithi164 ceph-mon[36970]: pgmap v9121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:25:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:26 smithi163 ceph-mon[36873]: pgmap v9122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16156.491686 secs 2024-07-26T11:25:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16174.230024 secs 2024-07-26T11:25:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:26 smithi042 ceph-mon[32775]: pgmap v9122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16156.491686 secs 2024-07-26T11:25:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16174.230024 secs 2024-07-26T11:25:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:26 smithi164 ceph-mon[36970]: pgmap v9122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16156.491686 secs 2024-07-26T11:25:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16174.230024 secs 2024-07-26T11:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:28.583+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:28.583+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:25:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:28.583+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:25:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:28 smithi042 ceph-mon[32775]: pgmap v9123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:28 smithi164 ceph-mon[36970]: pgmap v9123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:28 smithi163 ceph-mon[36873]: pgmap v9123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:30 smithi042 ceph-mon[32775]: pgmap v9124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:30 smithi164 ceph-mon[36970]: pgmap v9124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:31.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:30 smithi163 ceph-mon[36873]: pgmap v9124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16161.491785 secs 2024-07-26T11:25:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16179.230148 secs 2024-07-26T11:25:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16161.491785 secs 2024-07-26T11:25:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16179.230148 secs 2024-07-26T11:25:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16161.491785 secs 2024-07-26T11:25:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16179.230148 secs 2024-07-26T11:25:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:32 smithi042 ceph-mon[32775]: pgmap v9125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:32 smithi164 ceph-mon[36970]: pgmap v9125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:32 smithi163 ceph-mon[36873]: pgmap v9125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:35.089 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:34 smithi042 ceph-mon[32775]: pgmap v9126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:25:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:34 smithi164 ceph-mon[36970]: pgmap v9126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:25:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:34 smithi163 ceph-mon[36873]: pgmap v9126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:25:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:36 smithi042 ceph-mon[32775]: pgmap v9127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16166.491880 secs 2024-07-26T11:25:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16184.230266 secs 2024-07-26T11:25:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:36 smithi164 ceph-mon[36970]: pgmap v9127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16166.491880 secs 2024-07-26T11:25:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16184.230266 secs 2024-07-26T11:25:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:36 smithi163 ceph-mon[36873]: pgmap v9127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16166.491880 secs 2024-07-26T11:25:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16184.230266 secs 2024-07-26T11:25:39.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:38 smithi042 ceph-mon[32775]: pgmap v9128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:38 smithi164 ceph-mon[36970]: pgmap v9128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:38 smithi163 ceph-mon[36873]: pgmap v9128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:41.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:40 smithi042 ceph-mon[32775]: pgmap v9129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:40 smithi164 ceph-mon[36970]: pgmap v9129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:41.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:40 smithi163 ceph-mon[36873]: pgmap v9129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16171.492006 secs 2024-07-26T11:25:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16189.230390 secs 2024-07-26T11:25:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16171.492006 secs 2024-07-26T11:25:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16189.230390 secs 2024-07-26T11:25:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16171.492006 secs 2024-07-26T11:25:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16189.230390 secs 2024-07-26T11:25:43.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:42 smithi042 ceph-mon[32775]: pgmap v9130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:42 smithi164 ceph-mon[36970]: pgmap v9130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:42 smithi163 ceph-mon[36873]: pgmap v9130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:43.586 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:43.584+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:25:43.586 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:43.584+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:25:43.587 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:43.585+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:25:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:44 smithi042 ceph-mon[32775]: pgmap v9131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:25:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:44 smithi164 ceph-mon[36970]: pgmap v9131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:25:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:44 smithi163 ceph-mon[36873]: pgmap v9131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:25:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:46 smithi042 ceph-mon[32775]: pgmap v9132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16176.492142 secs 2024-07-26T11:25:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16194.230518 secs 2024-07-26T11:25:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:46 smithi164 ceph-mon[36970]: pgmap v9132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16176.492142 secs 2024-07-26T11:25:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16194.230518 secs 2024-07-26T11:25:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:46 smithi163 ceph-mon[36873]: pgmap v9132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:25:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16176.492142 secs 2024-07-26T11:25:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16194.230518 secs 2024-07-26T11:25:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:25:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:25:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:25:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:48 smithi042 ceph-mon[32775]: pgmap v9133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:25:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:25:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:25:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:48 smithi164 ceph-mon[36970]: pgmap v9133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:25:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:25:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:25:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:48 smithi163 ceph-mon[36873]: pgmap v9133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:25:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:25:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:25:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:25:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:49 smithi042 ceph-mon[32775]: pgmap v9134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:25:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:49 smithi042 ceph-mon[32775]: pgmap v9135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:25:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:49 smithi164 ceph-mon[36970]: pgmap v9134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:25:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:49 smithi164 ceph-mon[36970]: pgmap v9135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:25:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:49 smithi163 ceph-mon[36873]: pgmap v9134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:25:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:49 smithi163 ceph-mon[36873]: pgmap v9135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:25:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:51 smithi042 ceph-mon[32775]: pgmap v9136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:25:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16181.492294 secs 2024-07-26T11:25:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16199.230646 secs 2024-07-26T11:25:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:51 smithi164 ceph-mon[36970]: pgmap v9136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:25:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16181.492294 secs 2024-07-26T11:25:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16199.230646 secs 2024-07-26T11:25:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:51 smithi163 ceph-mon[36873]: pgmap v9136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:25:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16181.492294 secs 2024-07-26T11:25:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16199.230646 secs 2024-07-26T11:25:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:53 smithi042 ceph-mon[32775]: pgmap v9137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:25:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:53 smithi164 ceph-mon[36970]: pgmap v9137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:25:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:53 smithi163 ceph-mon[36873]: pgmap v9137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:25:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:55 smithi042 ceph-mon[32775]: pgmap v9138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:25:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:55 smithi164 ceph-mon[36970]: pgmap v9138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:25:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:55 smithi163 ceph-mon[36873]: pgmap v9138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:25:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16186.492403 secs 2024-07-26T11:25:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16204.230769 secs 2024-07-26T11:25:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16186.492403 secs 2024-07-26T11:25:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16204.230769 secs 2024-07-26T11:25:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16186.492403 secs 2024-07-26T11:25:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16204.230769 secs 2024-07-26T11:25:57.888 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:57.886+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T11:25:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:57.886+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:25:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:57.886+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff678031570 tx=0x7ff678055040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:25:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:57.886+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7ff678031570 tx=0x7ff678055040 comp rx=0 tx=0).stop 2024-07-26T11:25:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:57.886+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T11:25:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:57.886+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:25:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:57.886+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:57.886+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:57.887+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:57.887+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5000 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7ff678009a60 tx=0x7ff678055040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:57 smithi042 ceph-mon[32775]: pgmap v9139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:25:58.227 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:57 smithi164 ceph-mon[36970]: pgmap v9139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:25:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:57 smithi163 ceph-mon[36873]: pgmap v9139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:25:58.588 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:58.586+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:25:58.588 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:58.586+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:25:58.588 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:25:58.586+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:26:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:25:59 smithi042 ceph-mon[32775]: pgmap v9140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:26:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:25:59 smithi164 ceph-mon[36970]: pgmap v9140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:26:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:25:59 smithi163 ceph-mon[36873]: pgmap v9140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:26:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:26:00.413+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:26:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:26:00.413+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T11:26:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:26:00.414+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T11:26:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:26:00.414+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:26:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:01 smithi042 ceph-mon[32775]: pgmap v9141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16191.492526 secs 2024-07-26T11:26:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16209.230894 secs 2024-07-26T11:26:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:01 smithi164 ceph-mon[36970]: pgmap v9141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16191.492526 secs 2024-07-26T11:26:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16209.230894 secs 2024-07-26T11:26:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:01 smithi163 ceph-mon[36873]: pgmap v9141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16191.492526 secs 2024-07-26T11:26:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16209.230894 secs 2024-07-26T11:26:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:03 smithi042 ceph-mon[32775]: pgmap v9142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:03 smithi164 ceph-mon[36970]: pgmap v9142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:03 smithi163 ceph-mon[36873]: pgmap v9142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:05 smithi042 ceph-mon[32775]: pgmap v9143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:05 smithi164 ceph-mon[36970]: pgmap v9143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:05 smithi163 ceph-mon[36873]: pgmap v9143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16196.492675 secs 2024-07-26T11:26:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16214.230992 secs 2024-07-26T11:26:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16196.492675 secs 2024-07-26T11:26:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16214.230992 secs 2024-07-26T11:26:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16196.492675 secs 2024-07-26T11:26:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16214.230992 secs 2024-07-26T11:26:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:07 smithi042 ceph-mon[32775]: pgmap v9144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:07 smithi164 ceph-mon[36970]: pgmap v9144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:07 smithi163 ceph-mon[36873]: pgmap v9144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:09 smithi042 ceph-mon[32775]: pgmap v9145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:09 smithi164 ceph-mon[36970]: pgmap v9145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:09 smithi163 ceph-mon[36873]: pgmap v9145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:11 smithi042 ceph-mon[32775]: pgmap v9146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16201.492819 secs 2024-07-26T11:26:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16219.231145 secs 2024-07-26T11:26:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:11 smithi164 ceph-mon[36970]: pgmap v9146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16201.492819 secs 2024-07-26T11:26:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16219.231145 secs 2024-07-26T11:26:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:11 smithi163 ceph-mon[36873]: pgmap v9146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16201.492819 secs 2024-07-26T11:26:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16219.231145 secs 2024-07-26T11:26:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:12 smithi164 ceph-mon[36970]: pgmap v9147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:12 smithi163 ceph-mon[36873]: pgmap v9147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:12 smithi042 ceph-mon[32775]: pgmap v9147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:13.589 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:26:13.587+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:26:13.589 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:26:13.587+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:26:13.589 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:26:13.587+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:26:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:15 smithi163 ceph-mon[36873]: pgmap v9148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:15 smithi042 ceph-mon[32775]: pgmap v9148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:15 smithi164 ceph-mon[36970]: pgmap v9148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16206.492958 secs 2024-07-26T11:26:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16206.492958 secs 2024-07-26T11:26:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16206.492958 secs 2024-07-26T11:26:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:17 smithi163 ceph-mon[36873]: pgmap v9149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16224.231267 secs 2024-07-26T11:26:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:17 smithi042 ceph-mon[32775]: pgmap v9149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16224.231267 secs 2024-07-26T11:26:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:17 smithi164 ceph-mon[36970]: pgmap v9149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16224.231267 secs 2024-07-26T11:26:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:19 smithi163 ceph-mon[36873]: pgmap v9150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:19.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:19 smithi042 ceph-mon[32775]: pgmap v9150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:19 smithi164 ceph-mon[36970]: pgmap v9150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:21.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:21 smithi042 ceph-mon[32775]: pgmap v9151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:21.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16211.493077 secs 2024-07-26T11:26:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:21 smithi164 ceph-mon[36970]: pgmap v9151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16211.493077 secs 2024-07-26T11:26:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:21 smithi163 ceph-mon[36873]: pgmap v9151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16211.493077 secs 2024-07-26T11:26:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16229.231374 secs 2024-07-26T11:26:22.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16229.231374 secs 2024-07-26T11:26:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16229.231374 secs 2024-07-26T11:26:23.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:23 smithi042 ceph-mon[32775]: pgmap v9152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:23 smithi164 ceph-mon[36970]: pgmap v9152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:23 smithi163 ceph-mon[36873]: pgmap v9152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:25.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:25 smithi042 ceph-mon[32775]: pgmap v9153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:25 smithi164 ceph-mon[36970]: pgmap v9153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:25 smithi163 ceph-mon[36873]: pgmap v9153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16216.493205 secs 2024-07-26T11:26:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16216.493205 secs 2024-07-26T11:26:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16216.493205 secs 2024-07-26T11:26:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:27 smithi042 ceph-mon[32775]: pgmap v9154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16234.231499 secs 2024-07-26T11:26:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:27 smithi164 ceph-mon[36970]: pgmap v9154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16234.231499 secs 2024-07-26T11:26:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:27 smithi163 ceph-mon[36873]: pgmap v9154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16234.231499 secs 2024-07-26T11:26:28.594 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:26:28.592+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:26:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:26:28.593+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:26:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:26:28.593+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:26:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:29 smithi042 ceph-mon[32775]: pgmap v9155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:29 smithi164 ceph-mon[36970]: pgmap v9155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:29 smithi163 ceph-mon[36873]: pgmap v9155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:31 smithi042 ceph-mon[32775]: pgmap v9156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16221.493326 secs 2024-07-26T11:26:31.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:31 smithi164 ceph-mon[36970]: pgmap v9156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:31.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16221.493326 secs 2024-07-26T11:26:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:31 smithi163 ceph-mon[36873]: pgmap v9156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16221.493326 secs 2024-07-26T11:26:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16239.231607 secs 2024-07-26T11:26:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16239.231607 secs 2024-07-26T11:26:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16239.231607 secs 2024-07-26T11:26:33.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:33 smithi042 ceph-mon[32775]: pgmap v9157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:33 smithi164 ceph-mon[36970]: pgmap v9157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:33 smithi163 ceph-mon[36873]: pgmap v9157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:35.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:35 smithi042 ceph-mon[32775]: pgmap v9158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:35 smithi164 ceph-mon[36970]: pgmap v9158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:35 smithi163 ceph-mon[36873]: pgmap v9158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16226.493409 secs 2024-07-26T11:26:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16226.493409 secs 2024-07-26T11:26:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16226.493409 secs 2024-07-26T11:26:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:37 smithi042 ceph-mon[32775]: pgmap v9159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16244.231761 secs 2024-07-26T11:26:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:37 smithi164 ceph-mon[36970]: pgmap v9159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16244.231761 secs 2024-07-26T11:26:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:37 smithi163 ceph-mon[36873]: pgmap v9159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16244.231761 secs 2024-07-26T11:26:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:39 smithi042 ceph-mon[32775]: pgmap v9160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:39 smithi164 ceph-mon[36970]: pgmap v9160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:39 smithi163 ceph-mon[36873]: pgmap v9160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:41 smithi042 ceph-mon[32775]: pgmap v9161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16231.493528 secs 2024-07-26T11:26:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16249.231860 secs 2024-07-26T11:26:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:41 smithi164 ceph-mon[36970]: pgmap v9161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16231.493528 secs 2024-07-26T11:26:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16249.231860 secs 2024-07-26T11:26:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:41 smithi163 ceph-mon[36873]: pgmap v9161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16231.493528 secs 2024-07-26T11:26:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16249.231860 secs 2024-07-26T11:26:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:26:43.594+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:26:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:26:43.594+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:26:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:26:43.594+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:26:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:43 smithi042 ceph-mon[32775]: pgmap v9162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:43 smithi164 ceph-mon[36970]: pgmap v9162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:43 smithi163 ceph-mon[36873]: pgmap v9162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:45 smithi042 ceph-mon[32775]: pgmap v9163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:45 smithi164 ceph-mon[36970]: pgmap v9163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:45 smithi163 ceph-mon[36873]: pgmap v9163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:26:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16236.493670 secs 2024-07-26T11:26:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16254.231980 secs 2024-07-26T11:26:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16236.493670 secs 2024-07-26T11:26:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16254.231980 secs 2024-07-26T11:26:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16236.493670 secs 2024-07-26T11:26:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16254.231980 secs 2024-07-26T11:26:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:47 smithi042 ceph-mon[32775]: pgmap v9164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:47 smithi164 ceph-mon[36970]: pgmap v9164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:47 smithi163 ceph-mon[36873]: pgmap v9164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:26:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:26:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:26:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:26:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:26:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:26:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:26:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:26:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:26:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:26:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:26:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:26:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:49 smithi042 ceph-mon[32775]: pgmap v9165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:49 smithi042 ceph-mon[32775]: pgmap v9166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:26:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:49 smithi042 ceph-mon[32775]: pgmap v9167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:26:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:49 smithi164 ceph-mon[36970]: pgmap v9165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:49 smithi164 ceph-mon[36970]: pgmap v9166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:26:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:49 smithi164 ceph-mon[36970]: pgmap v9167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:26:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:49 smithi163 ceph-mon[36873]: pgmap v9165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:26:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:49 smithi163 ceph-mon[36873]: pgmap v9166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:26:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:49 smithi163 ceph-mon[36873]: pgmap v9167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:26:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:51 smithi042 ceph-mon[32775]: pgmap v9168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:26:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16241.493776 secs 2024-07-26T11:26:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16259.232104 secs 2024-07-26T11:26:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:51 smithi164 ceph-mon[36970]: pgmap v9168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:26:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16241.493776 secs 2024-07-26T11:26:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16259.232104 secs 2024-07-26T11:26:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:51 smithi163 ceph-mon[36873]: pgmap v9168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:26:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16241.493776 secs 2024-07-26T11:26:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16259.232104 secs 2024-07-26T11:26:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:53 smithi042 ceph-mon[32775]: pgmap v9169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:26:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:53 smithi164 ceph-mon[36970]: pgmap v9169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:26:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:53 smithi163 ceph-mon[36873]: pgmap v9169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:26:56.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:55 smithi042 ceph-mon[32775]: pgmap v9170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:26:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:55 smithi164 ceph-mon[36970]: pgmap v9170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:26:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:55 smithi163 ceph-mon[36873]: pgmap v9170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:26:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16246.493887 secs 2024-07-26T11:26:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16264.232229 secs 2024-07-26T11:26:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16246.493887 secs 2024-07-26T11:26:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16264.232229 secs 2024-07-26T11:26:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16246.493887 secs 2024-07-26T11:26:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16264.232229 secs 2024-07-26T11:26:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:57 smithi042 ceph-mon[32775]: pgmap v9171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:26:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:57 smithi164 ceph-mon[36970]: pgmap v9171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:26:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:57 smithi163 ceph-mon[36873]: pgmap v9171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:26:58.600 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:26:58.598+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:26:58.600 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:26:58.598+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:26:58.600 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:26:58.598+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:27:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:26:59 smithi042 ceph-mon[32775]: pgmap v9172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:27:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:26:59 smithi164 ceph-mon[36970]: pgmap v9172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:27:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:26:59 smithi163 ceph-mon[36873]: pgmap v9172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:27:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:01 smithi042 ceph-mon[32775]: pgmap v9173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16251.494031 secs 2024-07-26T11:27:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16269.232358 secs 2024-07-26T11:27:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:01 smithi164 ceph-mon[36970]: pgmap v9173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16251.494031 secs 2024-07-26T11:27:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16269.232358 secs 2024-07-26T11:27:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:01 smithi163 ceph-mon[36873]: pgmap v9173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16251.494031 secs 2024-07-26T11:27:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16269.232358 secs 2024-07-26T11:27:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:03 smithi042 ceph-mon[32775]: pgmap v9174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:03 smithi164 ceph-mon[36970]: pgmap v9174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:03 smithi163 ceph-mon[36873]: pgmap v9174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:05 smithi042 ceph-mon[32775]: pgmap v9175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:05 smithi164 ceph-mon[36970]: pgmap v9175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:05 smithi163 ceph-mon[36873]: pgmap v9175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16256.494161 secs 2024-07-26T11:27:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16274.232482 secs 2024-07-26T11:27:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16256.494161 secs 2024-07-26T11:27:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16274.232482 secs 2024-07-26T11:27:07.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16256.494161 secs 2024-07-26T11:27:07.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16274.232482 secs 2024-07-26T11:27:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:07 smithi042 ceph-mon[32775]: pgmap v9176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:07 smithi164 ceph-mon[36970]: pgmap v9176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:07 smithi163 ceph-mon[36873]: pgmap v9176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:09 smithi042 ceph-mon[32775]: pgmap v9177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:27:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:09 smithi164 ceph-mon[36970]: pgmap v9177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:27:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:09 smithi163 ceph-mon[36873]: pgmap v9177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:27:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:11 smithi042 ceph-mon[32775]: pgmap v9178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16261.494310 secs 2024-07-26T11:27:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16279.232605 secs 2024-07-26T11:27:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:11 smithi164 ceph-mon[36970]: pgmap v9178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16261.494310 secs 2024-07-26T11:27:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16279.232605 secs 2024-07-26T11:27:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:11 smithi163 ceph-mon[36873]: pgmap v9178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16261.494310 secs 2024-07-26T11:27:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16279.232605 secs 2024-07-26T11:27:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:27:13.599+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:27:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:27:13.599+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:13.601 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:27:13.599+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:27:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:13 smithi042 ceph-mon[32775]: pgmap v9179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:13 smithi164 ceph-mon[36970]: pgmap v9179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:13 smithi163 ceph-mon[36873]: pgmap v9179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:14 smithi164 ceph-mon[36970]: pgmap v9180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:14 smithi163 ceph-mon[36873]: pgmap v9180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:14 smithi042 ceph-mon[32775]: pgmap v9180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16266.494435 secs 2024-07-26T11:27:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16266.494435 secs 2024-07-26T11:27:16.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16266.494435 secs 2024-07-26T11:27:17.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16284.232724 secs 2024-07-26T11:27:17.504 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:17 smithi164 ceph-mon[36970]: pgmap v9181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16284.232724 secs 2024-07-26T11:27:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:17 smithi163 ceph-mon[36873]: pgmap v9181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16284.232724 secs 2024-07-26T11:27:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:17 smithi042 ceph-mon[32775]: pgmap v9181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:20.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:19 smithi042 ceph-mon[32775]: pgmap v9182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:27:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:19 smithi164 ceph-mon[36970]: pgmap v9182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:27:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:19 smithi163 ceph-mon[36873]: pgmap v9182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:27:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:21 smithi042 ceph-mon[32775]: pgmap v9183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16271.494565 secs 2024-07-26T11:27:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16289.232849 secs 2024-07-26T11:27:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:21 smithi164 ceph-mon[36970]: pgmap v9183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16271.494565 secs 2024-07-26T11:27:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16289.232849 secs 2024-07-26T11:27:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:21 smithi163 ceph-mon[36873]: pgmap v9183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16271.494565 secs 2024-07-26T11:27:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16289.232849 secs 2024-07-26T11:27:24.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:23 smithi042 ceph-mon[32775]: pgmap v9184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:23 smithi164 ceph-mon[36970]: pgmap v9184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:23 smithi163 ceph-mon[36873]: pgmap v9184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:25 smithi042 ceph-mon[32775]: pgmap v9185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:25 smithi164 ceph-mon[36970]: pgmap v9185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:25 smithi163 ceph-mon[36873]: pgmap v9185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16276.494693 secs 2024-07-26T11:27:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16294.232973 secs 2024-07-26T11:27:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16276.494693 secs 2024-07-26T11:27:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16294.232973 secs 2024-07-26T11:27:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16276.494693 secs 2024-07-26T11:27:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16294.232973 secs 2024-07-26T11:27:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:27 smithi042 ceph-mon[32775]: pgmap v9186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:27 smithi164 ceph-mon[36970]: pgmap v9186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:27 smithi163 ceph-mon[36873]: pgmap v9186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:28.606 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:27:28.604+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:27:28.606 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:27:28.604+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:28.606 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:27:28.604+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:27:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:29 smithi042 ceph-mon[32775]: pgmap v9187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:27:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:29 smithi164 ceph-mon[36970]: pgmap v9187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:27:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:29 smithi163 ceph-mon[36873]: pgmap v9187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:27:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:31 smithi042 ceph-mon[32775]: pgmap v9188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16281.494799 secs 2024-07-26T11:27:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16299.233098 secs 2024-07-26T11:27:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:31 smithi164 ceph-mon[36970]: pgmap v9188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16281.494799 secs 2024-07-26T11:27:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16299.233098 secs 2024-07-26T11:27:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:31 smithi163 ceph-mon[36873]: pgmap v9188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16281.494799 secs 2024-07-26T11:27:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16299.233098 secs 2024-07-26T11:27:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:33 smithi042 ceph-mon[32775]: pgmap v9189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:33 smithi164 ceph-mon[36970]: pgmap v9189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:33 smithi163 ceph-mon[36873]: pgmap v9189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:35 smithi042 ceph-mon[32775]: pgmap v9190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:35 smithi164 ceph-mon[36970]: pgmap v9190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:35 smithi163 ceph-mon[36873]: pgmap v9190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16286.494946 secs 2024-07-26T11:27:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16304.233222 secs 2024-07-26T11:27:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16286.494946 secs 2024-07-26T11:27:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16304.233222 secs 2024-07-26T11:27:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16286.494946 secs 2024-07-26T11:27:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16304.233222 secs 2024-07-26T11:27:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:37 smithi042 ceph-mon[32775]: pgmap v9191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:37 smithi164 ceph-mon[36970]: pgmap v9191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:37 smithi163 ceph-mon[36873]: pgmap v9191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:39 smithi042 ceph-mon[32775]: pgmap v9192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:27:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:39 smithi164 ceph-mon[36970]: pgmap v9192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:27:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:39 smithi163 ceph-mon[36873]: pgmap v9192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:27:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:41 smithi042 ceph-mon[32775]: pgmap v9193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16291.495067 secs 2024-07-26T11:27:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16309.233358 secs 2024-07-26T11:27:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:41 smithi164 ceph-mon[36970]: pgmap v9193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16291.495067 secs 2024-07-26T11:27:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16309.233358 secs 2024-07-26T11:27:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:41 smithi163 ceph-mon[36873]: pgmap v9193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16291.495067 secs 2024-07-26T11:27:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16309.233358 secs 2024-07-26T11:27:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:27:43.606+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:27:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:27:43.606+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:43.608 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:27:43.606+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:27:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:43 smithi042 ceph-mon[32775]: pgmap v9194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:43 smithi164 ceph-mon[36970]: pgmap v9194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:43 smithi163 ceph-mon[36873]: pgmap v9194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:45 smithi042 ceph-mon[32775]: pgmap v9195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:45 smithi164 ceph-mon[36970]: pgmap v9195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:45 smithi163 ceph-mon[36873]: pgmap v9195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:27:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16296.495196 secs 2024-07-26T11:27:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16314.233459 secs 2024-07-26T11:27:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16296.495196 secs 2024-07-26T11:27:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16314.233459 secs 2024-07-26T11:27:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16296.495196 secs 2024-07-26T11:27:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16314.233459 secs 2024-07-26T11:27:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:47 smithi042 ceph-mon[32775]: pgmap v9196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:47 smithi164 ceph-mon[36970]: pgmap v9196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:47 smithi163 ceph-mon[36873]: pgmap v9196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:27:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:48 smithi042 ceph-mon[32775]: pgmap v9197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:27:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:48 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:27:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:48 smithi164 ceph-mon[36970]: pgmap v9197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:27:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:48 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:27:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:48 smithi163 ceph-mon[36873]: pgmap v9197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:27:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:48 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:27:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:27:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:27:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:49 smithi042 ceph-mon[32775]: pgmap v9198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:27:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:49 smithi042 ceph-mon[32775]: pgmap v9199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:27:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:27:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:27:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:27:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:49 smithi164 ceph-mon[36970]: pgmap v9198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:27:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:49 smithi164 ceph-mon[36970]: pgmap v9199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:27:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:27:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:27:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:27:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:49 smithi163 ceph-mon[36873]: pgmap v9198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:27:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:49 smithi163 ceph-mon[36873]: pgmap v9199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:27:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:27:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16301.495294 secs 2024-07-26T11:27:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16301.495294 secs 2024-07-26T11:27:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16301.495294 secs 2024-07-26T11:27:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:52 smithi164 ceph-mon[36970]: pgmap v9200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:27:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16319.233585 secs 2024-07-26T11:27:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:52 smithi163 ceph-mon[36873]: pgmap v9200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:27:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16319.233585 secs 2024-07-26T11:27:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:52 smithi042 ceph-mon[32775]: pgmap v9200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:27:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16319.233585 secs 2024-07-26T11:27:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:54 smithi042 ceph-mon[32775]: pgmap v9201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:27:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:54 smithi164 ceph-mon[36970]: pgmap v9201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:27:54.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:54 smithi163 ceph-mon[36873]: pgmap v9201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:27:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:56 smithi042 ceph-mon[32775]: pgmap v9202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:27:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16306.495418 secs 2024-07-26T11:27:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:56 smithi164 ceph-mon[36970]: pgmap v9202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:27:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16306.495418 secs 2024-07-26T11:27:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:56 smithi163 ceph-mon[36873]: pgmap v9202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:27:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16306.495418 secs 2024-07-26T11:27:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16324.233731 secs 2024-07-26T11:27:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16324.233731 secs 2024-07-26T11:27:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16324.233731 secs 2024-07-26T11:27:58.613 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:27:58.611+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:27:58.613 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:27:58.611+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:58.614 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:27:58.611+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:27:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:27:58 smithi042 ceph-mon[32775]: pgmap v9203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:27:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:27:58 smithi164 ceph-mon[36970]: pgmap v9203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:27:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:27:58 smithi163 ceph-mon[36873]: pgmap v9203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:28:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:00 smithi042 ceph-mon[32775]: pgmap v9204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:28:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:00 smithi164 ceph-mon[36970]: pgmap v9204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:28:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:00 smithi163 ceph-mon[36873]: pgmap v9204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:28:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16311.495536 secs 2024-07-26T11:28:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16311.495536 secs 2024-07-26T11:28:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16311.495536 secs 2024-07-26T11:28:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:02 smithi042 ceph-mon[32775]: pgmap v9205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16329.233865 secs 2024-07-26T11:28:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:02 smithi164 ceph-mon[36970]: pgmap v9205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16329.233865 secs 2024-07-26T11:28:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:02 smithi163 ceph-mon[36873]: pgmap v9205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16329.233865 secs 2024-07-26T11:28:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:04 smithi042 ceph-mon[32775]: pgmap v9206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:28:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:04 smithi164 ceph-mon[36970]: pgmap v9206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:28:04.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:04 smithi163 ceph-mon[36873]: pgmap v9206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:28:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:06 smithi042 ceph-mon[32775]: pgmap v9207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:06.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16316.495636 secs 2024-07-26T11:28:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:06 smithi164 ceph-mon[36970]: pgmap v9207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16316.495636 secs 2024-07-26T11:28:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:06 smithi163 ceph-mon[36873]: pgmap v9207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:06.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16316.495636 secs 2024-07-26T11:28:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16334.233997 secs 2024-07-26T11:28:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16334.233997 secs 2024-07-26T11:28:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16334.233997 secs 2024-07-26T11:28:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:08 smithi164 ceph-mon[36970]: pgmap v9208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:08.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:08 smithi163 ceph-mon[36873]: pgmap v9208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:08.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:08 smithi042 ceph-mon[32775]: pgmap v9208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:10.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:10 smithi042 ceph-mon[32775]: pgmap v9209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:10 smithi164 ceph-mon[36970]: pgmap v9209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:10 smithi163 ceph-mon[36873]: pgmap v9209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16321.495702 secs 2024-07-26T11:28:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16321.495702 secs 2024-07-26T11:28:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16321.495702 secs 2024-07-26T11:28:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:12 smithi164 ceph-mon[36970]: pgmap v9210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:12.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16339.234123 secs 2024-07-26T11:28:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:12 smithi163 ceph-mon[36873]: pgmap v9210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16339.234123 secs 2024-07-26T11:28:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:12 smithi042 ceph-mon[32775]: pgmap v9210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:12.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16339.234123 secs 2024-07-26T11:28:13.614 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:28:13.612+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:28:13.614 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:28:13.612+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:13.614 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:28:13.612+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:28:14.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:14 smithi042 ceph-mon[32775]: pgmap v9211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:28:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:14 smithi164 ceph-mon[36970]: pgmap v9211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:28:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:14 smithi163 ceph-mon[36873]: pgmap v9211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:28:15.437 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:28:15.436+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:28:15.437 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:28:15.436+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T11:28:15.437 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:28:15.436+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:28:15.438 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:28:15.436+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T11:28:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:16 smithi164 ceph-mon[36970]: pgmap v9212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16326.495826 secs 2024-07-26T11:28:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:16 smithi163 ceph-mon[36873]: pgmap v9212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16326.495826 secs 2024-07-26T11:28:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:16 smithi042 ceph-mon[32775]: pgmap v9212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16326.495826 secs 2024-07-26T11:28:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16344.234267 secs 2024-07-26T11:28:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16344.234267 secs 2024-07-26T11:28:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16344.234267 secs 2024-07-26T11:28:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:18 smithi164 ceph-mon[36970]: pgmap v9213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:18.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:18 smithi163 ceph-mon[36873]: pgmap v9213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:18 smithi042 ceph-mon[32775]: pgmap v9213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:20 smithi164 ceph-mon[36970]: pgmap v9214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:20.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:20 smithi163 ceph-mon[36873]: pgmap v9214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:20.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:20 smithi042 ceph-mon[32775]: pgmap v9214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16331.495945 secs 2024-07-26T11:28:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16331.495945 secs 2024-07-26T11:28:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16331.495945 secs 2024-07-26T11:28:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:22 smithi164 ceph-mon[36970]: pgmap v9215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16349.234411 secs 2024-07-26T11:28:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:22 smithi163 ceph-mon[36873]: pgmap v9215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16349.234411 secs 2024-07-26T11:28:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:22 smithi042 ceph-mon[32775]: pgmap v9215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16349.234411 secs 2024-07-26T11:28:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:24 smithi164 ceph-mon[36970]: pgmap v9216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:28:24.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:24 smithi163 ceph-mon[36873]: pgmap v9216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:28:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:24 smithi042 ceph-mon[32775]: pgmap v9216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:28:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:26 smithi164 ceph-mon[36970]: pgmap v9217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16336.496088 secs 2024-07-26T11:28:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:26 smithi163 ceph-mon[36873]: pgmap v9217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16336.496088 secs 2024-07-26T11:28:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:26 smithi042 ceph-mon[32775]: pgmap v9217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16336.496088 secs 2024-07-26T11:28:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16354.234551 secs 2024-07-26T11:28:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16354.234551 secs 2024-07-26T11:28:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16354.234551 secs 2024-07-26T11:28:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:28:28.617+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:28:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:28:28.617+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:28:28.617+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:28:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:28 smithi164 ceph-mon[36970]: pgmap v9218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:28 smithi163 ceph-mon[36873]: pgmap v9218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:28 smithi042 ceph-mon[32775]: pgmap v9218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:30 smithi164 ceph-mon[36970]: pgmap v9219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:30 smithi163 ceph-mon[36873]: pgmap v9219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:30.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:30 smithi042 ceph-mon[32775]: pgmap v9219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16341.496210 secs 2024-07-26T11:28:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16341.496210 secs 2024-07-26T11:28:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16341.496210 secs 2024-07-26T11:28:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:32 smithi164 ceph-mon[36970]: pgmap v9220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16359.234679 secs 2024-07-26T11:28:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:32 smithi163 ceph-mon[36873]: pgmap v9220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16359.234679 secs 2024-07-26T11:28:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:32 smithi042 ceph-mon[32775]: pgmap v9220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:32.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16359.234679 secs 2024-07-26T11:28:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:34 smithi164 ceph-mon[36970]: pgmap v9221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:28:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:34 smithi163 ceph-mon[36873]: pgmap v9221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:28:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:34 smithi042 ceph-mon[32775]: pgmap v9221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:28:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:36 smithi164 ceph-mon[36970]: pgmap v9222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16346.496337 secs 2024-07-26T11:28:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:36 smithi163 ceph-mon[36873]: pgmap v9222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:36.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16346.496337 secs 2024-07-26T11:28:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:36 smithi042 ceph-mon[32775]: pgmap v9222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16346.496337 secs 2024-07-26T11:28:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16364.234804 secs 2024-07-26T11:28:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16364.234804 secs 2024-07-26T11:28:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16364.234804 secs 2024-07-26T11:28:38.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:38 smithi163 ceph-mon[36873]: pgmap v9223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:38.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:38 smithi042 ceph-mon[32775]: pgmap v9223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:38 smithi164 ceph-mon[36970]: pgmap v9223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:40.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:40 smithi163 ceph-mon[36873]: pgmap v9224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:40.775 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:40 smithi164 ceph-mon[36970]: pgmap v9224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:40.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:40 smithi042 ceph-mon[32775]: pgmap v9224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:28:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16351.496425 secs 2024-07-26T11:28:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16351.496425 secs 2024-07-26T11:28:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16351.496425 secs 2024-07-26T11:28:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:42 smithi163 ceph-mon[36873]: pgmap v9225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16369.234958 secs 2024-07-26T11:28:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:42 smithi042 ceph-mon[32775]: pgmap v9225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16369.234958 secs 2024-07-26T11:28:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:42 smithi164 ceph-mon[36970]: pgmap v9225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16369.234958 secs 2024-07-26T11:28:43.619 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:28:43.618+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:28:43.619 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:28:43.618+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:43.619 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:28:43.618+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:28:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:44 smithi163 ceph-mon[36873]: pgmap v9226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:28:44.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:44 smithi042 ceph-mon[32775]: pgmap v9226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:28:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:44 smithi164 ceph-mon[36970]: pgmap v9226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:28:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:46 smithi163 ceph-mon[36873]: pgmap v9227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:28:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16356.496548 secs 2024-07-26T11:28:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:46 smithi042 ceph-mon[32775]: pgmap v9227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:28:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16356.496548 secs 2024-07-26T11:28:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:46 smithi164 ceph-mon[36970]: pgmap v9227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:28:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16356.496548 secs 2024-07-26T11:28:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16374.235076 secs 2024-07-26T11:28:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16374.235076 secs 2024-07-26T11:28:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16374.235076 secs 2024-07-26T11:28:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:48 smithi163 ceph-mon[36873]: pgmap v9228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:28:48.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:48 smithi042 ceph-mon[32775]: pgmap v9228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:28:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:48 smithi164 ceph-mon[36970]: pgmap v9228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:28:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:28:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:28:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:28:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:50 smithi042 ceph-mon[32775]: pgmap v9229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:28:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:28:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:28:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:50 smithi042 ceph-mon[32775]: pgmap v9230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:50 smithi042 ceph-mon[32775]: pgmap v9231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:28:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:50 smithi164 ceph-mon[36970]: pgmap v9229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:28:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:28:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:28:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:50 smithi164 ceph-mon[36970]: pgmap v9230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:50 smithi164 ceph-mon[36970]: pgmap v9231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:50.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:28:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:50 smithi163 ceph-mon[36873]: pgmap v9229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:28:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:28:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:28:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:50 smithi163 ceph-mon[36873]: pgmap v9230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:28:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:50 smithi163 ceph-mon[36873]: pgmap v9231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:28:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:28:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16361.496682 secs 2024-07-26T11:28:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16361.496682 secs 2024-07-26T11:28:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16361.496682 secs 2024-07-26T11:28:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16379.235199 secs 2024-07-26T11:28:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:52 smithi042 ceph-mon[32775]: pgmap v9232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:52.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16379.235199 secs 2024-07-26T11:28:52.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:52 smithi163 ceph-mon[36873]: pgmap v9232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16379.235199 secs 2024-07-26T11:28:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:52 smithi164 ceph-mon[36970]: pgmap v9232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:28:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:53 smithi164 ceph-mon[36970]: pgmap v9233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:28:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:53 smithi163 ceph-mon[36873]: pgmap v9233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:28:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:53 smithi042 ceph-mon[32775]: pgmap v9233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:28:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:55 smithi164 ceph-mon[36970]: pgmap v9234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:28:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:55 smithi163 ceph-mon[36873]: pgmap v9234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:28:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:55 smithi042 ceph-mon[32775]: pgmap v9234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:28:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16366.496827 secs 2024-07-26T11:28:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16384.235321 secs 2024-07-26T11:28:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16366.496827 secs 2024-07-26T11:28:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16384.235321 secs 2024-07-26T11:28:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16366.496827 secs 2024-07-26T11:28:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16384.235321 secs 2024-07-26T11:28:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:57 smithi164 ceph-mon[36970]: pgmap v9235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:28:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:57 smithi163 ceph-mon[36873]: pgmap v9235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:28:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:57 smithi042 ceph-mon[32775]: pgmap v9235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:28:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:28:58.623+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:28:58.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:28:58.623+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:58.625 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:28:58.623+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:29:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:28:59 smithi164 ceph-mon[36970]: pgmap v9236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:29:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:28:59 smithi163 ceph-mon[36873]: pgmap v9236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:29:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:28:59 smithi042 ceph-mon[32775]: pgmap v9236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:29:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:29:00.442+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:29:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:29:00.442+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T11:29:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:29:00.443+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:29:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:29:00.443+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6580064a0 con 0x7ff680102000 2024-07-26T11:29:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16371.496931 secs 2024-07-26T11:29:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16371.496931 secs 2024-07-26T11:29:01.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16371.496931 secs 2024-07-26T11:29:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16389.235452 secs 2024-07-26T11:29:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:02 smithi164 ceph-mon[36970]: pgmap v9237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16389.235452 secs 2024-07-26T11:29:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:02 smithi163 ceph-mon[36873]: pgmap v9237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16389.235452 secs 2024-07-26T11:29:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:02 smithi042 ceph-mon[32775]: pgmap v9237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:03 smithi164 ceph-mon[36970]: pgmap v9238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:29:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:03 smithi163 ceph-mon[36873]: pgmap v9238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:29:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:03 smithi042 ceph-mon[32775]: pgmap v9238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:29:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:05 smithi164 ceph-mon[36970]: pgmap v9239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:05 smithi163 ceph-mon[36873]: pgmap v9239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:05 smithi042 ceph-mon[32775]: pgmap v9239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16376.497074 secs 2024-07-26T11:29:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16394.235576 secs 2024-07-26T11:29:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16376.497074 secs 2024-07-26T11:29:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16394.235576 secs 2024-07-26T11:29:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16376.497074 secs 2024-07-26T11:29:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16394.235576 secs 2024-07-26T11:29:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:07 smithi164 ceph-mon[36970]: pgmap v9240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:07 smithi163 ceph-mon[36873]: pgmap v9240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:07 smithi042 ceph-mon[32775]: pgmap v9240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:10 smithi164 ceph-mon[36970]: pgmap v9241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:10 smithi163 ceph-mon[36873]: pgmap v9241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:10 smithi042 ceph-mon[32775]: pgmap v9241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16381.497149 secs 2024-07-26T11:29:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16399.235702 secs 2024-07-26T11:29:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16381.497149 secs 2024-07-26T11:29:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16399.235702 secs 2024-07-26T11:29:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16381.497149 secs 2024-07-26T11:29:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16399.235702 secs 2024-07-26T11:29:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:12 smithi164 ceph-mon[36970]: pgmap v9242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:12 smithi163 ceph-mon[36873]: pgmap v9242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:13.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:12 smithi042 ceph-mon[32775]: pgmap v9242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:13.626 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:29:13.624+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:29:13.626 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:29:13.624+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:13.626 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:29:13.624+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:29:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:14 smithi164 ceph-mon[36970]: pgmap v9243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:29:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:14 smithi163 ceph-mon[36873]: pgmap v9243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:29:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:14 smithi042 ceph-mon[32775]: pgmap v9243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:29:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:16 smithi164 ceph-mon[36970]: pgmap v9244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16386.497273 secs 2024-07-26T11:29:17.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16404.235823 secs 2024-07-26T11:29:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:16 smithi163 ceph-mon[36873]: pgmap v9244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16386.497273 secs 2024-07-26T11:29:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16404.235823 secs 2024-07-26T11:29:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:16 smithi042 ceph-mon[32775]: pgmap v9244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16386.497273 secs 2024-07-26T11:29:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16404.235823 secs 2024-07-26T11:29:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:18 smithi164 ceph-mon[36970]: pgmap v9245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:19.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:18 smithi163 ceph-mon[36873]: pgmap v9245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:18 smithi042 ceph-mon[32775]: pgmap v9245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:20 smithi164 ceph-mon[36970]: pgmap v9246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:20 smithi163 ceph-mon[36873]: pgmap v9246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:20 smithi042 ceph-mon[32775]: pgmap v9246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16391.497360 secs 2024-07-26T11:29:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16409.235987 secs 2024-07-26T11:29:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16391.497360 secs 2024-07-26T11:29:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16409.235987 secs 2024-07-26T11:29:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16391.497360 secs 2024-07-26T11:29:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16409.235987 secs 2024-07-26T11:29:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:22 smithi164 ceph-mon[36970]: pgmap v9247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:22 smithi163 ceph-mon[36873]: pgmap v9247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:22 smithi042 ceph-mon[32775]: pgmap v9247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:24 smithi164 ceph-mon[36970]: pgmap v9248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:29:25.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:24 smithi163 ceph-mon[36873]: pgmap v9248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:29:25.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:24 smithi042 ceph-mon[32775]: pgmap v9248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:29:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:26 smithi163 ceph-mon[36873]: pgmap v9249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16396.497478 secs 2024-07-26T11:29:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16414.236101 secs 2024-07-26T11:29:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:26 smithi042 ceph-mon[32775]: pgmap v9249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16396.497478 secs 2024-07-26T11:29:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16414.236101 secs 2024-07-26T11:29:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:26 smithi164 ceph-mon[36970]: pgmap v9249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16396.497478 secs 2024-07-26T11:29:27.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16414.236101 secs 2024-07-26T11:29:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:29:28.630+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:29:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:29:28.630+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:29:28.630+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:29:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:28 smithi163 ceph-mon[36873]: pgmap v9250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:29.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:28 smithi042 ceph-mon[32775]: pgmap v9250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:28 smithi164 ceph-mon[36970]: pgmap v9250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:31.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:30 smithi163 ceph-mon[36873]: pgmap v9251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:30 smithi042 ceph-mon[32775]: pgmap v9251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:30 smithi164 ceph-mon[36970]: pgmap v9251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16401.497562 secs 2024-07-26T11:29:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16419.236223 secs 2024-07-26T11:29:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16401.497562 secs 2024-07-26T11:29:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16419.236223 secs 2024-07-26T11:29:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16401.497562 secs 2024-07-26T11:29:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16419.236223 secs 2024-07-26T11:29:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:32 smithi163 ceph-mon[36873]: pgmap v9252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:33.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:32 smithi042 ceph-mon[32775]: pgmap v9252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:32 smithi164 ceph-mon[36970]: pgmap v9252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:34 smithi163 ceph-mon[36873]: pgmap v9253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:29:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:34 smithi042 ceph-mon[32775]: pgmap v9253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:29:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:34 smithi164 ceph-mon[36970]: pgmap v9253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:29:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:36 smithi163 ceph-mon[36873]: pgmap v9254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16406.497708 secs 2024-07-26T11:29:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16424.236344 secs 2024-07-26T11:29:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:36 smithi042 ceph-mon[32775]: pgmap v9254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16406.497708 secs 2024-07-26T11:29:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16424.236344 secs 2024-07-26T11:29:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:36 smithi164 ceph-mon[36970]: pgmap v9254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16406.497708 secs 2024-07-26T11:29:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16424.236344 secs 2024-07-26T11:29:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:38 smithi163 ceph-mon[36873]: pgmap v9255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:39.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:38 smithi042 ceph-mon[32775]: pgmap v9255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:38 smithi164 ceph-mon[36970]: pgmap v9255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:41.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:40 smithi163 ceph-mon[36873]: pgmap v9256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:41.275 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:40 smithi164 ceph-mon[36970]: pgmap v9256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:40 smithi042 ceph-mon[32775]: pgmap v9256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16411.497803 secs 2024-07-26T11:29:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16429.236467 secs 2024-07-26T11:29:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16411.497803 secs 2024-07-26T11:29:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16429.236467 secs 2024-07-26T11:29:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16411.497803 secs 2024-07-26T11:29:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16429.236467 secs 2024-07-26T11:29:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:43 smithi163 ceph-mon[36873]: pgmap v9257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:43 smithi042 ceph-mon[32775]: pgmap v9257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:43 smithi164 ceph-mon[36970]: pgmap v9257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:43.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:29:43.631+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:29:43.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:29:43.631+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:43.633 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:29:43.631+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:29:45.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:45 smithi042 ceph-mon[32775]: pgmap v9258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:29:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:45 smithi164 ceph-mon[36970]: pgmap v9258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:29:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:45 smithi163 ceph-mon[36873]: pgmap v9258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:29:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:47 smithi042 ceph-mon[32775]: pgmap v9259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16416.497936 secs 2024-07-26T11:29:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16434.236589 secs 2024-07-26T11:29:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:47 smithi164 ceph-mon[36970]: pgmap v9259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16416.497936 secs 2024-07-26T11:29:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16434.236589 secs 2024-07-26T11:29:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:47 smithi163 ceph-mon[36873]: pgmap v9259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16416.497936 secs 2024-07-26T11:29:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16434.236589 secs 2024-07-26T11:29:49.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:49 smithi042 ceph-mon[32775]: pgmap v9260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:49 smithi164 ceph-mon[36970]: pgmap v9260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:49 smithi163 ceph-mon[36873]: pgmap v9260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:29:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:29:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:29:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:29:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:51 smithi042 ceph-mon[32775]: pgmap v9261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:29:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:29:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:51 smithi042 ceph-mon[32775]: pgmap v9262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:29:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:51 smithi042 ceph-mon[32775]: pgmap v9263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:29:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:29:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:51 smithi164 ceph-mon[36970]: pgmap v9261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:51.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:29:51.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:29:51.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:51 smithi164 ceph-mon[36970]: pgmap v9262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:29:51.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:51 smithi164 ceph-mon[36970]: pgmap v9263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:29:51.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:29:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:51 smithi163 ceph-mon[36873]: pgmap v9261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:29:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:29:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:29:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:51 smithi163 ceph-mon[36873]: pgmap v9262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:29:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:51 smithi163 ceph-mon[36873]: pgmap v9263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:29:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:29:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16421.498052 secs 2024-07-26T11:29:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16439.236712 secs 2024-07-26T11:29:52.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16421.498052 secs 2024-07-26T11:29:52.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16439.236712 secs 2024-07-26T11:29:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16421.498052 secs 2024-07-26T11:29:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16439.236712 secs 2024-07-26T11:29:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:53 smithi042 ceph-mon[32775]: pgmap v9264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:29:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:53 smithi164 ceph-mon[36970]: pgmap v9264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:29:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:53 smithi163 ceph-mon[36873]: pgmap v9264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:29:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:55 smithi042 ceph-mon[32775]: pgmap v9265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:29:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:55 smithi164 ceph-mon[36970]: pgmap v9265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:29:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:55 smithi163 ceph-mon[36873]: pgmap v9265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:29:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16426.498194 secs 2024-07-26T11:29:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16444.236830 secs 2024-07-26T11:29:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16426.498194 secs 2024-07-26T11:29:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16444.236830 secs 2024-07-26T11:29:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16426.498194 secs 2024-07-26T11:29:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16444.236830 secs 2024-07-26T11:29:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:57 smithi042 ceph-mon[32775]: pgmap v9266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:29:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:57 smithi164 ceph-mon[36970]: pgmap v9266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:29:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:57 smithi163 ceph-mon[36873]: pgmap v9266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:29:58.638 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:29:58.637+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:29:58.639 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:29:58.637+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:58.639 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:29:58.637+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:30:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:29:59 smithi042 ceph-mon[32775]: pgmap v9267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:30:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:29:59 smithi164 ceph-mon[36970]: pgmap v9267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:30:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:29:59 smithi163 ceph-mon[36873]: pgmap v9267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:30:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:30:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:30:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:30:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:01 smithi042 ceph-mon[32775]: pgmap v9268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:30:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16431.498322 secs 2024-07-26T11:30:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16449.236969 secs 2024-07-26T11:30:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:01 smithi164 ceph-mon[36970]: pgmap v9268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:30:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16431.498322 secs 2024-07-26T11:30:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16449.236969 secs 2024-07-26T11:30:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:01 smithi163 ceph-mon[36873]: pgmap v9268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:30:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16431.498322 secs 2024-07-26T11:30:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16449.236969 secs 2024-07-26T11:30:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:03 smithi042 ceph-mon[32775]: pgmap v9269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:03 smithi164 ceph-mon[36970]: pgmap v9269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:03 smithi163 ceph-mon[36873]: pgmap v9269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:05 smithi042 ceph-mon[32775]: pgmap v9270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:05 smithi164 ceph-mon[36970]: pgmap v9270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:05 smithi163 ceph-mon[36873]: pgmap v9270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16436.498450 secs 2024-07-26T11:30:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16454.237029 secs 2024-07-26T11:30:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16436.498450 secs 2024-07-26T11:30:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16454.237029 secs 2024-07-26T11:30:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16436.498450 secs 2024-07-26T11:30:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16454.237029 secs 2024-07-26T11:30:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:07 smithi042 ceph-mon[32775]: pgmap v9271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:07 smithi164 ceph-mon[36970]: pgmap v9271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:07 smithi163 ceph-mon[36873]: pgmap v9271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:09 smithi042 ceph-mon[32775]: pgmap v9272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:30:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:09 smithi164 ceph-mon[36970]: pgmap v9272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:30:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:09 smithi163 ceph-mon[36873]: pgmap v9272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:30:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:11 smithi042 ceph-mon[32775]: pgmap v9273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16441.498577 secs 2024-07-26T11:30:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16459.237130 secs 2024-07-26T11:30:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:11 smithi164 ceph-mon[36970]: pgmap v9273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16441.498577 secs 2024-07-26T11:30:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16459.237130 secs 2024-07-26T11:30:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:11 smithi163 ceph-mon[36873]: pgmap v9273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16441.498577 secs 2024-07-26T11:30:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16459.237130 secs 2024-07-26T11:30:13.640 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:30:13.638+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:30:13.640 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:30:13.638+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:13.640 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:30:13.638+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:30:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:13 smithi042 ceph-mon[32775]: pgmap v9274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:13 smithi164 ceph-mon[36970]: pgmap v9274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:13 smithi163 ceph-mon[36873]: pgmap v9274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:16.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:15 smithi042 ceph-mon[32775]: pgmap v9275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:15 smithi164 ceph-mon[36970]: pgmap v9275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:15 smithi163 ceph-mon[36873]: pgmap v9275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16446.498705 secs 2024-07-26T11:30:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16464.237254 secs 2024-07-26T11:30:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16446.498705 secs 2024-07-26T11:30:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16464.237254 secs 2024-07-26T11:30:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16446.498705 secs 2024-07-26T11:30:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16464.237254 secs 2024-07-26T11:30:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:17 smithi164 ceph-mon[36970]: pgmap v9276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:17 smithi163 ceph-mon[36873]: pgmap v9276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:17 smithi042 ceph-mon[32775]: pgmap v9276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:18 smithi164 ceph-mon[36970]: pgmap v9277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:30:19.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:18 smithi163 ceph-mon[36873]: pgmap v9277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:30:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:18 smithi042 ceph-mon[32775]: pgmap v9277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:30:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:21 smithi163 ceph-mon[36873]: pgmap v9278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16451.498830 secs 2024-07-26T11:30:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:21 smithi042 ceph-mon[32775]: pgmap v9278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16451.498830 secs 2024-07-26T11:30:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:21 smithi164 ceph-mon[36970]: pgmap v9278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16451.498830 secs 2024-07-26T11:30:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16469.237383 secs 2024-07-26T11:30:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16469.237383 secs 2024-07-26T11:30:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16469.237383 secs 2024-07-26T11:30:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:23 smithi163 ceph-mon[36873]: pgmap v9279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:23 smithi042 ceph-mon[32775]: pgmap v9279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:23 smithi164 ceph-mon[36970]: pgmap v9279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:25 smithi163 ceph-mon[36873]: pgmap v9280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:25 smithi042 ceph-mon[32775]: pgmap v9280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:25 smithi164 ceph-mon[36970]: pgmap v9280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16456.498957 secs 2024-07-26T11:30:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16456.498957 secs 2024-07-26T11:30:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16456.498957 secs 2024-07-26T11:30:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16474.237504 secs 2024-07-26T11:30:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:27 smithi042 ceph-mon[32775]: pgmap v9281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16474.237504 secs 2024-07-26T11:30:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:27 smithi164 ceph-mon[36970]: pgmap v9281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16474.237504 secs 2024-07-26T11:30:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:27 smithi163 ceph-mon[36873]: pgmap v9281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:28.645 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:30:28.644+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:30:28.645 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:30:28.644+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:28.646 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:30:28.644+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:30:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:29 smithi042 ceph-mon[32775]: pgmap v9282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:30:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:29 smithi164 ceph-mon[36970]: pgmap v9282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:30:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:29 smithi163 ceph-mon[36873]: pgmap v9282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:30:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:31 smithi042 ceph-mon[32775]: pgmap v9283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16461.499075 secs 2024-07-26T11:30:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:31 smithi164 ceph-mon[36970]: pgmap v9283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16461.499075 secs 2024-07-26T11:30:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:31 smithi163 ceph-mon[36873]: pgmap v9283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16461.499075 secs 2024-07-26T11:30:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16479.237646 secs 2024-07-26T11:30:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16479.237646 secs 2024-07-26T11:30:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16479.237646 secs 2024-07-26T11:30:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:33 smithi042 ceph-mon[32775]: pgmap v9284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:33 smithi164 ceph-mon[36970]: pgmap v9284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:33 smithi163 ceph-mon[36873]: pgmap v9284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:35.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:35 smithi042 ceph-mon[32775]: pgmap v9285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:35 smithi164 ceph-mon[36970]: pgmap v9285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:36.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:35 smithi163 ceph-mon[36873]: pgmap v9285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16466.499225 secs 2024-07-26T11:30:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16466.499225 secs 2024-07-26T11:30:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16466.499225 secs 2024-07-26T11:30:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16484.237742 secs 2024-07-26T11:30:37.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:37 smithi042 ceph-mon[32775]: pgmap v9286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16484.237742 secs 2024-07-26T11:30:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:37 smithi164 ceph-mon[36970]: pgmap v9286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16484.237742 secs 2024-07-26T11:30:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:37 smithi163 ceph-mon[36873]: pgmap v9286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:39 smithi042 ceph-mon[32775]: pgmap v9287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:30:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:39 smithi164 ceph-mon[36970]: pgmap v9287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:30:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:39 smithi163 ceph-mon[36873]: pgmap v9287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:30:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:41 smithi042 ceph-mon[32775]: pgmap v9288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16471.499341 secs 2024-07-26T11:30:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16489.237869 secs 2024-07-26T11:30:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:41 smithi164 ceph-mon[36970]: pgmap v9288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16471.499341 secs 2024-07-26T11:30:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16489.237869 secs 2024-07-26T11:30:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:41 smithi163 ceph-mon[36873]: pgmap v9288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16471.499341 secs 2024-07-26T11:30:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16489.237869 secs 2024-07-26T11:30:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:30:43.645+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:30:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:30:43.645+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:43.647 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:30:43.645+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:30:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:43 smithi042 ceph-mon[32775]: pgmap v9289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:43 smithi164 ceph-mon[36970]: pgmap v9289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:43 smithi163 ceph-mon[36873]: pgmap v9289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:45 smithi042 ceph-mon[32775]: pgmap v9290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:45 smithi164 ceph-mon[36970]: pgmap v9290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:45 smithi163 ceph-mon[36873]: pgmap v9290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16476.499474 secs 2024-07-26T11:30:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16494.237935 secs 2024-07-26T11:30:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16476.499474 secs 2024-07-26T11:30:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16494.237935 secs 2024-07-26T11:30:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16476.499474 secs 2024-07-26T11:30:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16494.237935 secs 2024-07-26T11:30:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:47 smithi042 ceph-mon[32775]: pgmap v9291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:47 smithi164 ceph-mon[36970]: pgmap v9291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:47 smithi163 ceph-mon[36873]: pgmap v9291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:30:50.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:49 smithi042 ceph-mon[32775]: pgmap v9292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:30:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:49 smithi164 ceph-mon[36970]: pgmap v9292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:30:50.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:49 smithi163 ceph-mon[36873]: pgmap v9292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:30:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:30:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:30:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:30:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: pgmap v9293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:30:52.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:30:52.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:30:52.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16481.499602 secs 2024-07-26T11:30:52.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16499.238055 secs 2024-07-26T11:30:52.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: pgmap v9293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:52.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:30:52.160 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:30:52.161 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:30:52.161 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16481.499602 secs 2024-07-26T11:30:52.161 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16499.238055 secs 2024-07-26T11:30:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: pgmap v9293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:30:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:52.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:30:52.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:30:52.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:30:52.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16481.499602 secs 2024-07-26T11:30:52.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16499.238055 secs 2024-07-26T11:30:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:52 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:30:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:52 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:30:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:52 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:30:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:52 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:30:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:52 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:30:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:52 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:30:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:52 smithi042 ceph-mon[32775]: pgmap v9294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:30:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:52 smithi042 ceph-mon[32775]: pgmap v9295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:30:53.175 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:52 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:30:53.175 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:52 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:30:53.175 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:52 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:30:53.175 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:52 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:30:53.175 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:52 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:30:53.175 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:52 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:30:53.175 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:52 smithi163 ceph-mon[36873]: pgmap v9294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:30:53.175 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:52 smithi163 ceph-mon[36873]: pgmap v9295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:30:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:52 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:30:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:52 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:30:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:52 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:30:53.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:52 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:30:53.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:52 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:30:53.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:52 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:30:53.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:52 smithi164 ceph-mon[36970]: pgmap v9294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:30:53.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:52 smithi164 ceph-mon[36970]: pgmap v9295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:30:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:54 smithi042 ceph-mon[32775]: pgmap v9296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:30:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:54 smithi164 ceph-mon[36970]: pgmap v9296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:30:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:54 smithi163 ceph-mon[36873]: pgmap v9296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:30:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:56 smithi164 ceph-mon[36970]: pgmap v9297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:30:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16486.499723 secs 2024-07-26T11:30:57.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16504.238176 secs 2024-07-26T11:30:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:56 smithi163 ceph-mon[36873]: pgmap v9297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:30:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16486.499723 secs 2024-07-26T11:30:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16504.238176 secs 2024-07-26T11:30:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:56 smithi042 ceph-mon[32775]: pgmap v9297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:30:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16486.499723 secs 2024-07-26T11:30:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16504.238176 secs 2024-07-26T11:30:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:30:58.651+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:30:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:30:58.651+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:58.652 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:30:58.651+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:30:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:30:58 smithi164 ceph-mon[36970]: pgmap v9298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:30:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:30:58 smithi163 ceph-mon[36873]: pgmap v9298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:30:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:30:58 smithi042 ceph-mon[32775]: pgmap v9298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:31:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:00 smithi164 ceph-mon[36970]: pgmap v9299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:31:01.232 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:00 smithi042 ceph-mon[32775]: pgmap v9299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:31:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:00 smithi163 ceph-mon[36873]: pgmap v9299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:31:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16491.499852 secs 2024-07-26T11:31:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16509.238285 secs 2024-07-26T11:31:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16491.499852 secs 2024-07-26T11:31:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16509.238285 secs 2024-07-26T11:31:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16491.499852 secs 2024-07-26T11:31:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16509.238285 secs 2024-07-26T11:31:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:02 smithi164 ceph-mon[36970]: pgmap v9300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:31:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:02 smithi163 ceph-mon[36873]: pgmap v9300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:31:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:02 smithi042 ceph-mon[32775]: pgmap v9300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:31:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:03 smithi164 ceph-mon[36970]: pgmap v9301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:03 smithi163 ceph-mon[36873]: pgmap v9301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:03 smithi042 ceph-mon[32775]: pgmap v9301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:06 smithi042 ceph-mon[32775]: pgmap v9302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:06 smithi164 ceph-mon[36970]: pgmap v9302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:06 smithi163 ceph-mon[36873]: pgmap v9302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16496.499979 secs 2024-07-26T11:31:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16514.238406 secs 2024-07-26T11:31:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16496.499979 secs 2024-07-26T11:31:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16514.238406 secs 2024-07-26T11:31:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16496.499979 secs 2024-07-26T11:31:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16514.238406 secs 2024-07-26T11:31:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:08 smithi042 ceph-mon[32775]: pgmap v9303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:08 smithi164 ceph-mon[36970]: pgmap v9303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:08 smithi163 ceph-mon[36873]: pgmap v9303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:10.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:10 smithi042 ceph-mon[32775]: pgmap v9304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:31:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:10 smithi164 ceph-mon[36970]: pgmap v9304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:31:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:10 smithi163 ceph-mon[36873]: pgmap v9304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:31:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:12 smithi042 ceph-mon[32775]: pgmap v9305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16501.500077 secs 2024-07-26T11:31:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16519.238542 secs 2024-07-26T11:31:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:12 smithi164 ceph-mon[36970]: pgmap v9305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16501.500077 secs 2024-07-26T11:31:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16519.238542 secs 2024-07-26T11:31:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:12 smithi163 ceph-mon[36873]: pgmap v9305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16501.500077 secs 2024-07-26T11:31:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16519.238542 secs 2024-07-26T11:31:13.653 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:31:13.651+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:31:13.653 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:31:13.651+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:13.654 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:31:13.651+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:31:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:14 smithi042 ceph-mon[32775]: pgmap v9306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:14 smithi164 ceph-mon[36970]: pgmap v9306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:14.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:14 smithi163 ceph-mon[36873]: pgmap v9306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:16 smithi042 ceph-mon[32775]: pgmap v9307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:16 smithi164 ceph-mon[36970]: pgmap v9307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:16 smithi163 ceph-mon[36873]: pgmap v9307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16506.500155 secs 2024-07-26T11:31:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16524.238687 secs 2024-07-26T11:31:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16506.500155 secs 2024-07-26T11:31:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16524.238687 secs 2024-07-26T11:31:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16506.500155 secs 2024-07-26T11:31:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16524.238687 secs 2024-07-26T11:31:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:18 smithi042 ceph-mon[32775]: pgmap v9308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:18 smithi164 ceph-mon[36970]: pgmap v9308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:18 smithi163 ceph-mon[36873]: pgmap v9308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:20 smithi042 ceph-mon[32775]: pgmap v9309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:31:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:20 smithi164 ceph-mon[36970]: pgmap v9309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:31:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:20 smithi163 ceph-mon[36873]: pgmap v9309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:31:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:22 smithi042 ceph-mon[32775]: pgmap v9310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16511.500277 secs 2024-07-26T11:31:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16529.238834 secs 2024-07-26T11:31:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:22 smithi164 ceph-mon[36970]: pgmap v9310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16511.500277 secs 2024-07-26T11:31:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16529.238834 secs 2024-07-26T11:31:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:22 smithi163 ceph-mon[36873]: pgmap v9310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16511.500277 secs 2024-07-26T11:31:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16529.238834 secs 2024-07-26T11:31:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:24 smithi042 ceph-mon[32775]: pgmap v9311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:24 smithi164 ceph-mon[36970]: pgmap v9311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:24 smithi163 ceph-mon[36873]: pgmap v9311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:26 smithi042 ceph-mon[32775]: pgmap v9312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:26 smithi164 ceph-mon[36970]: pgmap v9312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:26 smithi163 ceph-mon[36873]: pgmap v9312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16516.500401 secs 2024-07-26T11:31:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16534.238957 secs 2024-07-26T11:31:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16516.500401 secs 2024-07-26T11:31:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16534.238957 secs 2024-07-26T11:31:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16516.500401 secs 2024-07-26T11:31:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16534.238957 secs 2024-07-26T11:31:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:28 smithi042 ceph-mon[32775]: pgmap v9313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:28 smithi164 ceph-mon[36970]: pgmap v9313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:28 smithi163 ceph-mon[36873]: pgmap v9313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:31:28.657+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:31:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:31:28.657+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:31:28.657+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:31:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:30 smithi042 ceph-mon[32775]: pgmap v9314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:31:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:30 smithi164 ceph-mon[36970]: pgmap v9314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:31:30.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:30 smithi163 ceph-mon[36873]: pgmap v9314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:31:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:32 smithi042 ceph-mon[32775]: pgmap v9315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16521.500467 secs 2024-07-26T11:31:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16539.239084 secs 2024-07-26T11:31:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:32 smithi164 ceph-mon[36970]: pgmap v9315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16521.500467 secs 2024-07-26T11:31:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16539.239084 secs 2024-07-26T11:31:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:32 smithi163 ceph-mon[36873]: pgmap v9315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16521.500467 secs 2024-07-26T11:31:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16539.239084 secs 2024-07-26T11:31:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:34 smithi042 ceph-mon[32775]: pgmap v9316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:34 smithi164 ceph-mon[36970]: pgmap v9316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:34 smithi163 ceph-mon[36873]: pgmap v9316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:36 smithi042 ceph-mon[32775]: pgmap v9317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:36 smithi164 ceph-mon[36970]: pgmap v9317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:36 smithi163 ceph-mon[36873]: pgmap v9317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16526.500591 secs 2024-07-26T11:31:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16544.239196 secs 2024-07-26T11:31:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16526.500591 secs 2024-07-26T11:31:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16544.239196 secs 2024-07-26T11:31:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16526.500591 secs 2024-07-26T11:31:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16544.239196 secs 2024-07-26T11:31:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:38 smithi042 ceph-mon[32775]: pgmap v9318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:38 smithi164 ceph-mon[36970]: pgmap v9318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:38 smithi163 ceph-mon[36873]: pgmap v9318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:40 smithi042 ceph-mon[32775]: pgmap v9319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:31:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:40 smithi164 ceph-mon[36970]: pgmap v9319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:31:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:40 smithi163 ceph-mon[36873]: pgmap v9319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:31:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:42 smithi042 ceph-mon[32775]: pgmap v9320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16531.500657 secs 2024-07-26T11:31:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16549.239298 secs 2024-07-26T11:31:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:42 smithi164 ceph-mon[36970]: pgmap v9320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16531.500657 secs 2024-07-26T11:31:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16549.239298 secs 2024-07-26T11:31:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:42 smithi163 ceph-mon[36873]: pgmap v9320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16531.500657 secs 2024-07-26T11:31:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16549.239298 secs 2024-07-26T11:31:43.660 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:31:43.658+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:31:43.660 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:31:43.658+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:43.661 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:31:43.658+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:31:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:44 smithi164 ceph-mon[36970]: pgmap v9321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:44 smithi163 ceph-mon[36873]: pgmap v9321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:44 smithi042 ceph-mon[32775]: pgmap v9321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:46 smithi164 ceph-mon[36970]: pgmap v9322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:46 smithi163 ceph-mon[36873]: pgmap v9322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:46 smithi042 ceph-mon[32775]: pgmap v9322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16536.500783 secs 2024-07-26T11:31:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16554.239448 secs 2024-07-26T11:31:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16536.500783 secs 2024-07-26T11:31:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16554.239448 secs 2024-07-26T11:31:47.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16536.500783 secs 2024-07-26T11:31:47.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16554.239448 secs 2024-07-26T11:31:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:48 smithi164 ceph-mon[36970]: pgmap v9323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:48.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:48 smithi163 ceph-mon[36873]: pgmap v9323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:48.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:48 smithi042 ceph-mon[32775]: pgmap v9323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:31:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:50 smithi164 ceph-mon[36970]: pgmap v9324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:31:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:50 smithi163 ceph-mon[36873]: pgmap v9324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:31:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:50 smithi042 ceph-mon[32775]: pgmap v9324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:31:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:31:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:31:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:31:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:52 smithi164 ceph-mon[36970]: pgmap v9325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16541.500907 secs 2024-07-26T11:31:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16559.239521 secs 2024-07-26T11:31:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:31:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:31:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:52 smithi164 ceph-mon[36970]: pgmap v9326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:31:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:52 smithi164 ceph-mon[36970]: pgmap v9327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:31:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:31:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:52 smithi163 ceph-mon[36873]: pgmap v9325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16541.500907 secs 2024-07-26T11:31:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16559.239521 secs 2024-07-26T11:31:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:31:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:31:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:52 smithi163 ceph-mon[36873]: pgmap v9326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:31:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:52 smithi163 ceph-mon[36873]: pgmap v9327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:31:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:31:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:52 smithi042 ceph-mon[32775]: pgmap v9325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:31:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16541.500907 secs 2024-07-26T11:31:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16559.239521 secs 2024-07-26T11:31:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:31:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:31:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:52 smithi042 ceph-mon[32775]: pgmap v9326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:31:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:52 smithi042 ceph-mon[32775]: pgmap v9327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:31:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:31:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:54 smithi042 ceph-mon[32775]: pgmap v9328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:31:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:54 smithi164 ceph-mon[36970]: pgmap v9328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:31:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:54 smithi163 ceph-mon[36873]: pgmap v9328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:31:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:56 smithi042 ceph-mon[32775]: pgmap v9329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:31:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16546.501032 secs 2024-07-26T11:31:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16564.239667 secs 2024-07-26T11:31:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:56 smithi164 ceph-mon[36970]: pgmap v9329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:31:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16546.501032 secs 2024-07-26T11:31:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16564.239667 secs 2024-07-26T11:31:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:56 smithi163 ceph-mon[36873]: pgmap v9329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:31:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16546.501032 secs 2024-07-26T11:31:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16564.239667 secs 2024-07-26T11:31:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:31:58.664+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:31:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:31:58.664+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:31:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:31:58.664+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:58.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:31:58 smithi042 ceph-mon[32775]: pgmap v9330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:31:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:31:58 smithi164 ceph-mon[36970]: pgmap v9330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:31:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:31:58 smithi163 ceph-mon[36873]: pgmap v9330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:32:00.472 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:32:00.471+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006480 con 0x7ff680102000 2024-07-26T11:32:00.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:32:00.471+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:32:00.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:32:00.471+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:32:00.473 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:32:00.472+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006480 con 0x7ff680102000 2024-07-26T11:32:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:00 smithi042 ceph-mon[32775]: pgmap v9331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:32:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:00 smithi164 ceph-mon[36970]: pgmap v9331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:32:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:00 smithi163 ceph-mon[36873]: pgmap v9331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:32:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16551.501158 secs 2024-07-26T11:32:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16569.239789 secs 2024-07-26T11:32:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16551.501158 secs 2024-07-26T11:32:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16569.239789 secs 2024-07-26T11:32:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16551.501158 secs 2024-07-26T11:32:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16569.239789 secs 2024-07-26T11:32:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:02 smithi042 ceph-mon[32775]: pgmap v9332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:32:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:02 smithi164 ceph-mon[36970]: pgmap v9332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:32:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:02 smithi163 ceph-mon[36873]: pgmap v9332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:32:04.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:04 smithi042 ceph-mon[32775]: pgmap v9333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:32:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:04 smithi164 ceph-mon[36970]: pgmap v9333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:32:05.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:04 smithi163 ceph-mon[36873]: pgmap v9333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:32:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:06 smithi164 ceph-mon[36970]: pgmap v9334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16556.501279 secs 2024-07-26T11:32:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16574.239909 secs 2024-07-26T11:32:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:06 smithi163 ceph-mon[36873]: pgmap v9334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16556.501279 secs 2024-07-26T11:32:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16574.239909 secs 2024-07-26T11:32:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:06 smithi042 ceph-mon[32775]: pgmap v9334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16556.501279 secs 2024-07-26T11:32:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16574.239909 secs 2024-07-26T11:32:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:08 smithi164 ceph-mon[36970]: pgmap v9335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:08 smithi163 ceph-mon[36873]: pgmap v9335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:08 smithi042 ceph-mon[32775]: pgmap v9335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:10 smithi164 ceph-mon[36970]: pgmap v9336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:10 smithi163 ceph-mon[36873]: pgmap v9336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:10 smithi042 ceph-mon[32775]: pgmap v9336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16561.501404 secs 2024-07-26T11:32:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16579.240038 secs 2024-07-26T11:32:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16561.501404 secs 2024-07-26T11:32:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16579.240038 secs 2024-07-26T11:32:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16561.501404 secs 2024-07-26T11:32:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16579.240038 secs 2024-07-26T11:32:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:12 smithi164 ceph-mon[36970]: pgmap v9337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:12 smithi163 ceph-mon[36873]: pgmap v9337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:12 smithi042 ceph-mon[32775]: pgmap v9337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:13.667 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:32:13.666+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:32:13.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:32:13.666+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:13.668 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:32:13.666+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:32:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:14 smithi164 ceph-mon[36970]: pgmap v9338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:32:15.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:14 smithi163 ceph-mon[36873]: pgmap v9338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:32:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:14 smithi042 ceph-mon[32775]: pgmap v9338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:32:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:16 smithi164 ceph-mon[36970]: pgmap v9339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16566.501499 secs 2024-07-26T11:32:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16584.240157 secs 2024-07-26T11:32:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:16 smithi163 ceph-mon[36873]: pgmap v9339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16566.501499 secs 2024-07-26T11:32:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16584.240157 secs 2024-07-26T11:32:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:16 smithi042 ceph-mon[32775]: pgmap v9339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16566.501499 secs 2024-07-26T11:32:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16584.240157 secs 2024-07-26T11:32:18.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:18 smithi042 ceph-mon[32775]: pgmap v9340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:18 smithi164 ceph-mon[36970]: pgmap v9340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:18 smithi163 ceph-mon[36873]: pgmap v9340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:20 smithi164 ceph-mon[36970]: pgmap v9341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:21.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:20 smithi163 ceph-mon[36873]: pgmap v9341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:20 smithi042 ceph-mon[32775]: pgmap v9341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16571.501630 secs 2024-07-26T11:32:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16589.240290 secs 2024-07-26T11:32:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16571.501630 secs 2024-07-26T11:32:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16589.240290 secs 2024-07-26T11:32:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16571.501630 secs 2024-07-26T11:32:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16589.240290 secs 2024-07-26T11:32:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:22 smithi164 ceph-mon[36970]: pgmap v9342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:22 smithi163 ceph-mon[36873]: pgmap v9342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:22 smithi042 ceph-mon[32775]: pgmap v9342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:24 smithi164 ceph-mon[36970]: pgmap v9343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:32:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:24 smithi163 ceph-mon[36873]: pgmap v9343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:32:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:24 smithi042 ceph-mon[32775]: pgmap v9343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:32:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:26 smithi164 ceph-mon[36970]: pgmap v9344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16576.501750 secs 2024-07-26T11:32:26.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16594.240375 secs 2024-07-26T11:32:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:26 smithi163 ceph-mon[36873]: pgmap v9344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16576.501750 secs 2024-07-26T11:32:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16594.240375 secs 2024-07-26T11:32:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:26 smithi042 ceph-mon[32775]: pgmap v9344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16576.501750 secs 2024-07-26T11:32:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16594.240375 secs 2024-07-26T11:32:28.673 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:32:28.672+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:32:28.673 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:32:28.672+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:28.674 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:32:28.672+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:32:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:28 smithi164 ceph-mon[36970]: pgmap v9345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:28 smithi163 ceph-mon[36873]: pgmap v9345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:28 smithi042 ceph-mon[32775]: pgmap v9345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:30.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:30 smithi164 ceph-mon[36970]: pgmap v9346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:30 smithi163 ceph-mon[36873]: pgmap v9346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:30 smithi042 ceph-mon[32775]: pgmap v9346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16581.501869 secs 2024-07-26T11:32:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16599.240500 secs 2024-07-26T11:32:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16581.501869 secs 2024-07-26T11:32:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16599.240500 secs 2024-07-26T11:32:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16581.501869 secs 2024-07-26T11:32:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16599.240500 secs 2024-07-26T11:32:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:32 smithi164 ceph-mon[36970]: pgmap v9347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:32 smithi163 ceph-mon[36873]: pgmap v9347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:33.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:32 smithi042 ceph-mon[32775]: pgmap v9347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:34 smithi164 ceph-mon[36970]: pgmap v9348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:32:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:34 smithi163 ceph-mon[36873]: pgmap v9348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:32:35.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:34 smithi042 ceph-mon[32775]: pgmap v9348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:32:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:36 smithi164 ceph-mon[36970]: pgmap v9349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16586.501993 secs 2024-07-26T11:32:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16604.240619 secs 2024-07-26T11:32:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:36 smithi163 ceph-mon[36873]: pgmap v9349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16586.501993 secs 2024-07-26T11:32:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16604.240619 secs 2024-07-26T11:32:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:36 smithi042 ceph-mon[32775]: pgmap v9349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16586.501993 secs 2024-07-26T11:32:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16604.240619 secs 2024-07-26T11:32:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:38 smithi164 ceph-mon[36970]: pgmap v9350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:38 smithi163 ceph-mon[36873]: pgmap v9350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:39.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:38 smithi042 ceph-mon[32775]: pgmap v9350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:40 smithi164 ceph-mon[36970]: pgmap v9351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:40 smithi163 ceph-mon[36873]: pgmap v9351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:40 smithi042 ceph-mon[32775]: pgmap v9351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16591.502117 secs 2024-07-26T11:32:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16609.240777 secs 2024-07-26T11:32:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16591.502117 secs 2024-07-26T11:32:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16609.240777 secs 2024-07-26T11:32:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16591.502117 secs 2024-07-26T11:32:42.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16609.240777 secs 2024-07-26T11:32:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:42 smithi163 ceph-mon[36873]: pgmap v9352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:43.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:42 smithi164 ceph-mon[36970]: pgmap v9352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:42 smithi042 ceph-mon[32775]: pgmap v9352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:32:43.674+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:32:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:32:43.674+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:32:43.674+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:32:45.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:44 smithi163 ceph-mon[36873]: pgmap v9353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:32:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:44 smithi042 ceph-mon[32775]: pgmap v9353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:32:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:44 smithi164 ceph-mon[36970]: pgmap v9353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:32:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:46 smithi163 ceph-mon[36873]: pgmap v9354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16596.502245 secs 2024-07-26T11:32:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16614.240905 secs 2024-07-26T11:32:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:46 smithi042 ceph-mon[32775]: pgmap v9354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16596.502245 secs 2024-07-26T11:32:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16614.240905 secs 2024-07-26T11:32:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:46 smithi164 ceph-mon[36970]: pgmap v9354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16596.502245 secs 2024-07-26T11:32:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16614.240905 secs 2024-07-26T11:32:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:48 smithi163 ceph-mon[36873]: pgmap v9355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:49.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:48 smithi042 ceph-mon[32775]: pgmap v9355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:48 smithi164 ceph-mon[36970]: pgmap v9355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:50 smithi163 ceph-mon[36873]: pgmap v9356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:50 smithi042 ceph-mon[32775]: pgmap v9356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:50 smithi164 ceph-mon[36970]: pgmap v9356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:32:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16601.502388 secs 2024-07-26T11:32:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16619.241034 secs 2024-07-26T11:32:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:51 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:32:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16601.502388 secs 2024-07-26T11:32:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16619.241034 secs 2024-07-26T11:32:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:51 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:32:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16601.502388 secs 2024-07-26T11:32:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16619.241034 secs 2024-07-26T11:32:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:51 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:32:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:52 smithi163 ceph-mon[36873]: pgmap v9357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:32:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:32:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:32:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:52 smithi042 ceph-mon[32775]: pgmap v9357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:32:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:32:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:32:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:52 smithi164 ceph-mon[36970]: pgmap v9357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:32:53.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:32:53.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:32:53.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:32:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:53 smithi163 ceph-mon[36873]: pgmap v9358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:32:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:53 smithi163 ceph-mon[36873]: pgmap v9359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:32:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:53 smithi042 ceph-mon[32775]: pgmap v9358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:32:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:53 smithi042 ceph-mon[32775]: pgmap v9359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:32:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:53 smithi164 ceph-mon[36970]: pgmap v9358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:32:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:53 smithi164 ceph-mon[36970]: pgmap v9359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:32:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:55 smithi163 ceph-mon[36873]: pgmap v9360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:32:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:55 smithi042 ceph-mon[32775]: pgmap v9360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:32:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:55 smithi164 ceph-mon[36970]: pgmap v9360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:32:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16606.502534 secs 2024-07-26T11:32:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16624.241099 secs 2024-07-26T11:32:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16606.502534 secs 2024-07-26T11:32:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16624.241099 secs 2024-07-26T11:32:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16606.502534 secs 2024-07-26T11:32:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16624.241099 secs 2024-07-26T11:32:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:57 smithi042 ceph-mon[32775]: pgmap v9361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:32:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:57 smithi164 ceph-mon[36970]: pgmap v9361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:32:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:57 smithi163 ceph-mon[36873]: pgmap v9361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:32:58.680 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:32:58.677+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:32:58.680 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:32:58.677+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:32:58.680 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:32:58.677+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:32:59 smithi042 ceph-mon[32775]: pgmap v9362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:33:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:32:59 smithi164 ceph-mon[36970]: pgmap v9362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:33:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:32:59 smithi163 ceph-mon[36873]: pgmap v9362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:33:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:01 smithi042 ceph-mon[32775]: pgmap v9363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:33:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16611.502653 secs 2024-07-26T11:33:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16629.241220 secs 2024-07-26T11:33:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:01 smithi164 ceph-mon[36970]: pgmap v9363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:33:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16611.502653 secs 2024-07-26T11:33:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16629.241220 secs 2024-07-26T11:33:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:01 smithi163 ceph-mon[36873]: pgmap v9363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:33:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16611.502653 secs 2024-07-26T11:33:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16629.241220 secs 2024-07-26T11:33:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:03 smithi042 ceph-mon[32775]: pgmap v9364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:33:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:03 smithi164 ceph-mon[36970]: pgmap v9364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:33:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:03 smithi163 ceph-mon[36873]: pgmap v9364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:33:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:05 smithi042 ceph-mon[32775]: pgmap v9365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:05 smithi164 ceph-mon[36970]: pgmap v9365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:05 smithi163 ceph-mon[36873]: pgmap v9365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16616.502779 secs 2024-07-26T11:33:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16634.241368 secs 2024-07-26T11:33:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16616.502779 secs 2024-07-26T11:33:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16634.241368 secs 2024-07-26T11:33:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16616.502779 secs 2024-07-26T11:33:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16634.241368 secs 2024-07-26T11:33:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:07 smithi042 ceph-mon[32775]: pgmap v9366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:07 smithi164 ceph-mon[36970]: pgmap v9366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:07 smithi163 ceph-mon[36873]: pgmap v9366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:09 smithi042 ceph-mon[32775]: pgmap v9367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:09 smithi164 ceph-mon[36970]: pgmap v9367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:09 smithi163 ceph-mon[36873]: pgmap v9367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:11 smithi042 ceph-mon[32775]: pgmap v9368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16621.502895 secs 2024-07-26T11:33:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16639.241510 secs 2024-07-26T11:33:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:11 smithi164 ceph-mon[36970]: pgmap v9368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16621.502895 secs 2024-07-26T11:33:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16639.241510 secs 2024-07-26T11:33:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:11 smithi163 ceph-mon[36873]: pgmap v9368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16621.502895 secs 2024-07-26T11:33:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16639.241510 secs 2024-07-26T11:33:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:33:13.678+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:33:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:33:13.679+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:33:13.679+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:33:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:13 smithi042 ceph-mon[32775]: pgmap v9369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:13 smithi164 ceph-mon[36970]: pgmap v9369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:13 smithi163 ceph-mon[36873]: pgmap v9369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:15 smithi042 ceph-mon[32775]: pgmap v9370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:15 smithi164 ceph-mon[36970]: pgmap v9370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:15 smithi163 ceph-mon[36873]: pgmap v9370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16626.503003 secs 2024-07-26T11:33:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16644.241625 secs 2024-07-26T11:33:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16626.503003 secs 2024-07-26T11:33:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16644.241625 secs 2024-07-26T11:33:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16626.503003 secs 2024-07-26T11:33:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16644.241625 secs 2024-07-26T11:33:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:17 smithi042 ceph-mon[32775]: pgmap v9371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:17 smithi164 ceph-mon[36970]: pgmap v9371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:17 smithi163 ceph-mon[36873]: pgmap v9371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:19 smithi042 ceph-mon[32775]: pgmap v9372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:19 smithi164 ceph-mon[36970]: pgmap v9372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:19 smithi163 ceph-mon[36873]: pgmap v9372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:21 smithi042 ceph-mon[32775]: pgmap v9373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16631.503127 secs 2024-07-26T11:33:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16649.241749 secs 2024-07-26T11:33:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:21 smithi164 ceph-mon[36970]: pgmap v9373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16631.503127 secs 2024-07-26T11:33:22.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16649.241749 secs 2024-07-26T11:33:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:21 smithi163 ceph-mon[36873]: pgmap v9373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16631.503127 secs 2024-07-26T11:33:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16649.241749 secs 2024-07-26T11:33:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:23 smithi042 ceph-mon[32775]: pgmap v9374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:23 smithi164 ceph-mon[36970]: pgmap v9374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:23 smithi163 ceph-mon[36873]: pgmap v9374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:25 smithi042 ceph-mon[32775]: pgmap v9375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:25 smithi164 ceph-mon[36970]: pgmap v9375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:25 smithi163 ceph-mon[36873]: pgmap v9375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16636.503255 secs 2024-07-26T11:33:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16654.241870 secs 2024-07-26T11:33:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16636.503255 secs 2024-07-26T11:33:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16654.241870 secs 2024-07-26T11:33:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16636.503255 secs 2024-07-26T11:33:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16654.241870 secs 2024-07-26T11:33:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:27 smithi042 ceph-mon[32775]: pgmap v9376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:27 smithi164 ceph-mon[36970]: pgmap v9376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:27 smithi163 ceph-mon[36873]: pgmap v9376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:28.686 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:33:28.684+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:33:28.686 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:33:28.684+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:28.686 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:33:28.684+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:33:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:29 smithi042 ceph-mon[32775]: pgmap v9377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:29 smithi164 ceph-mon[36970]: pgmap v9377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:29 smithi163 ceph-mon[36873]: pgmap v9377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:31 smithi042 ceph-mon[32775]: pgmap v9378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16641.503377 secs 2024-07-26T11:33:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16659.242002 secs 2024-07-26T11:33:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:31 smithi164 ceph-mon[36970]: pgmap v9378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16641.503377 secs 2024-07-26T11:33:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16659.242002 secs 2024-07-26T11:33:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:31 smithi163 ceph-mon[36873]: pgmap v9378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16641.503377 secs 2024-07-26T11:33:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16659.242002 secs 2024-07-26T11:33:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:33 smithi042 ceph-mon[32775]: pgmap v9379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:33 smithi164 ceph-mon[36970]: pgmap v9379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:33 smithi163 ceph-mon[36873]: pgmap v9379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:35 smithi042 ceph-mon[32775]: pgmap v9380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:35 smithi164 ceph-mon[36970]: pgmap v9380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:35 smithi163 ceph-mon[36873]: pgmap v9380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16646.503498 secs 2024-07-26T11:33:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16664.242099 secs 2024-07-26T11:33:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16646.503498 secs 2024-07-26T11:33:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16664.242099 secs 2024-07-26T11:33:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16646.503498 secs 2024-07-26T11:33:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16664.242099 secs 2024-07-26T11:33:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:37 smithi042 ceph-mon[32775]: pgmap v9381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:37 smithi164 ceph-mon[36970]: pgmap v9381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:37 smithi163 ceph-mon[36873]: pgmap v9381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:39 smithi042 ceph-mon[32775]: pgmap v9382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:39 smithi164 ceph-mon[36970]: pgmap v9382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:39 smithi163 ceph-mon[36873]: pgmap v9382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:41 smithi042 ceph-mon[32775]: pgmap v9383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16651.503603 secs 2024-07-26T11:33:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16669.242225 secs 2024-07-26T11:33:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:41 smithi164 ceph-mon[36970]: pgmap v9383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16651.503603 secs 2024-07-26T11:33:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16669.242225 secs 2024-07-26T11:33:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:41 smithi163 ceph-mon[36873]: pgmap v9383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16651.503603 secs 2024-07-26T11:33:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16669.242225 secs 2024-07-26T11:33:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:33:43.685+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:33:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:33:43.685+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:43.687 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:33:43.685+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:33:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:43 smithi042 ceph-mon[32775]: pgmap v9384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:43 smithi164 ceph-mon[36970]: pgmap v9384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:43 smithi163 ceph-mon[36873]: pgmap v9384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:46.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:45 smithi042 ceph-mon[32775]: pgmap v9385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:45 smithi164 ceph-mon[36970]: pgmap v9385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:45 smithi163 ceph-mon[36873]: pgmap v9385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16656.503751 secs 2024-07-26T11:33:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16674.242345 secs 2024-07-26T11:33:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16656.503751 secs 2024-07-26T11:33:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16674.242345 secs 2024-07-26T11:33:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16656.503751 secs 2024-07-26T11:33:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16674.242345 secs 2024-07-26T11:33:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:47 smithi042 ceph-mon[32775]: pgmap v9386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:47 smithi164 ceph-mon[36970]: pgmap v9386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:47 smithi163 ceph-mon[36873]: pgmap v9386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:50.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:49 smithi042 ceph-mon[32775]: pgmap v9387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:49 smithi164 ceph-mon[36970]: pgmap v9387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:49 smithi163 ceph-mon[36873]: pgmap v9387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:51 smithi042 ceph-mon[32775]: pgmap v9388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16661.503837 secs 2024-07-26T11:33:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16679.242491 secs 2024-07-26T11:33:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:51 smithi164 ceph-mon[36970]: pgmap v9388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16661.503837 secs 2024-07-26T11:33:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16679.242491 secs 2024-07-26T11:33:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:51 smithi163 ceph-mon[36873]: pgmap v9388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16661.503837 secs 2024-07-26T11:33:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16679.242491 secs 2024-07-26T11:33:53.134 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:52 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:33:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:52 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:33:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:52 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:33:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:53 smithi042 ceph-mon[32775]: pgmap v9389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:54.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:53 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:53 smithi164 ceph-mon[36970]: pgmap v9389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:53 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:53 smithi163 ceph-mon[36873]: pgmap v9389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:33:54.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:53 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:33:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:33:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:33:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:33:55.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:33:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:33:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:33:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:55 smithi164 ceph-mon[36970]: pgmap v9390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:55 smithi164 ceph-mon[36970]: pgmap v9391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:33:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:55 smithi164 ceph-mon[36970]: pgmap v9392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:33:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:55 smithi163 ceph-mon[36873]: pgmap v9390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:55 smithi163 ceph-mon[36873]: pgmap v9391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:33:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:55 smithi163 ceph-mon[36873]: pgmap v9392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:33:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:55 smithi042 ceph-mon[32775]: pgmap v9390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:33:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:55 smithi042 ceph-mon[32775]: pgmap v9391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:33:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:55 smithi042 ceph-mon[32775]: pgmap v9392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:33:57.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16666.503960 secs 2024-07-26T11:33:57.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16684.242633 secs 2024-07-26T11:33:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16666.503960 secs 2024-07-26T11:33:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16684.242633 secs 2024-07-26T11:33:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16666.503960 secs 2024-07-26T11:33:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16684.242633 secs 2024-07-26T11:33:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:57 smithi164 ceph-mon[36970]: pgmap v9393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:33:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:57 smithi163 ceph-mon[36873]: pgmap v9393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:33:58.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:57 smithi042 ceph-mon[32775]: pgmap v9393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:33:58.692 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:33:58.691+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:33:58.692 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:33:58.691+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:33:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:33:58.691+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:33:59 smithi164 ceph-mon[36970]: pgmap v9394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:34:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:33:59 smithi163 ceph-mon[36873]: pgmap v9394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:34:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:33:59 smithi042 ceph-mon[32775]: pgmap v9394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:34:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:01 smithi164 ceph-mon[36970]: pgmap v9395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:34:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16671.504081 secs 2024-07-26T11:34:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16689.242727 secs 2024-07-26T11:34:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:01 smithi163 ceph-mon[36873]: pgmap v9395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:34:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16671.504081 secs 2024-07-26T11:34:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16689.242727 secs 2024-07-26T11:34:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:01 smithi042 ceph-mon[32775]: pgmap v9395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:34:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16671.504081 secs 2024-07-26T11:34:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16689.242727 secs 2024-07-26T11:34:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:03 smithi164 ceph-mon[36970]: pgmap v9396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:34:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:03 smithi163 ceph-mon[36873]: pgmap v9396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:34:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:03 smithi042 ceph-mon[32775]: pgmap v9396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:34:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:05 smithi164 ceph-mon[36970]: pgmap v9397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:34:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:05 smithi163 ceph-mon[36873]: pgmap v9397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:34:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:05 smithi042 ceph-mon[32775]: pgmap v9397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:34:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16676.504203 secs 2024-07-26T11:34:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16694.242853 secs 2024-07-26T11:34:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16676.504203 secs 2024-07-26T11:34:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16694.242853 secs 2024-07-26T11:34:07.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16676.504203 secs 2024-07-26T11:34:07.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16694.242853 secs 2024-07-26T11:34:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:07 smithi164 ceph-mon[36970]: pgmap v9398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:07 smithi163 ceph-mon[36873]: pgmap v9398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:07 smithi042 ceph-mon[32775]: pgmap v9398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:09 smithi164 ceph-mon[36970]: pgmap v9399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:34:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:09 smithi163 ceph-mon[36873]: pgmap v9399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:34:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:09 smithi042 ceph-mon[32775]: pgmap v9399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:34:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:10 smithi163 ceph-mon[36873]: pgmap v9400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:10 smithi042 ceph-mon[32775]: pgmap v9400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:10 smithi164 ceph-mon[36970]: pgmap v9400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16681.504305 secs 2024-07-26T11:34:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16699.242977 secs 2024-07-26T11:34:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16681.504305 secs 2024-07-26T11:34:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16699.242977 secs 2024-07-26T11:34:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16681.504305 secs 2024-07-26T11:34:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16699.242977 secs 2024-07-26T11:34:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:12 smithi163 ceph-mon[36873]: pgmap v9401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:12 smithi042 ceph-mon[32775]: pgmap v9401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:12 smithi164 ceph-mon[36970]: pgmap v9401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:34:13.692+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:34:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:34:13.692+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:34:13.692+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:34:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:15 smithi042 ceph-mon[32775]: pgmap v9402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:16.176 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:15 smithi164 ceph-mon[36970]: pgmap v9402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:15 smithi163 ceph-mon[36873]: pgmap v9402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16686.504431 secs 2024-07-26T11:34:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16704.243102 secs 2024-07-26T11:34:17.192 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16686.504431 secs 2024-07-26T11:34:17.192 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16704.243102 secs 2024-07-26T11:34:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16686.504431 secs 2024-07-26T11:34:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16704.243102 secs 2024-07-26T11:34:18.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:17 smithi042 ceph-mon[32775]: pgmap v9403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:17 smithi164 ceph-mon[36970]: pgmap v9403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:17 smithi163 ceph-mon[36873]: pgmap v9403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:19 smithi042 ceph-mon[32775]: pgmap v9404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:34:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:19 smithi164 ceph-mon[36970]: pgmap v9404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:34:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:19 smithi163 ceph-mon[36873]: pgmap v9404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:34:22.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:21 smithi163 ceph-mon[36873]: pgmap v9405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:22.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16691.504516 secs 2024-07-26T11:34:22.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16709.243166 secs 2024-07-26T11:34:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:21 smithi042 ceph-mon[32775]: pgmap v9405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16691.504516 secs 2024-07-26T11:34:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16709.243166 secs 2024-07-26T11:34:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:21 smithi164 ceph-mon[36970]: pgmap v9405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16691.504516 secs 2024-07-26T11:34:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16709.243166 secs 2024-07-26T11:34:24.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:23 smithi042 ceph-mon[32775]: pgmap v9406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:23 smithi164 ceph-mon[36970]: pgmap v9406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:23 smithi163 ceph-mon[36873]: pgmap v9406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:25 smithi042 ceph-mon[32775]: pgmap v9407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:25 smithi164 ceph-mon[36970]: pgmap v9407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:25 smithi163 ceph-mon[36873]: pgmap v9407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16696.504643 secs 2024-07-26T11:34:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16714.243289 secs 2024-07-26T11:34:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16696.504643 secs 2024-07-26T11:34:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16714.243289 secs 2024-07-26T11:34:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16696.504643 secs 2024-07-26T11:34:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16714.243289 secs 2024-07-26T11:34:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:27 smithi042 ceph-mon[32775]: pgmap v9408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:27 smithi164 ceph-mon[36970]: pgmap v9408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:27 smithi163 ceph-mon[36873]: pgmap v9408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:34:28.698+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:34:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:34:28.698+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:28.700 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:34:28.698+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:34:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:29 smithi042 ceph-mon[32775]: pgmap v9409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:34:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:29 smithi164 ceph-mon[36970]: pgmap v9409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:34:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:29 smithi163 ceph-mon[36873]: pgmap v9409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:34:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:31 smithi042 ceph-mon[32775]: pgmap v9410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16701.504797 secs 2024-07-26T11:34:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16719.243409 secs 2024-07-26T11:34:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:31 smithi164 ceph-mon[36970]: pgmap v9410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16701.504797 secs 2024-07-26T11:34:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16719.243409 secs 2024-07-26T11:34:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:31 smithi163 ceph-mon[36873]: pgmap v9410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16701.504797 secs 2024-07-26T11:34:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16719.243409 secs 2024-07-26T11:34:34.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:33 smithi042 ceph-mon[32775]: pgmap v9411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:33 smithi164 ceph-mon[36970]: pgmap v9411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:33 smithi163 ceph-mon[36873]: pgmap v9411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:36.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:35 smithi042 ceph-mon[32775]: pgmap v9412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:35 smithi164 ceph-mon[36970]: pgmap v9412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:35 smithi163 ceph-mon[36873]: pgmap v9412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16706.504944 secs 2024-07-26T11:34:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16724.243561 secs 2024-07-26T11:34:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16706.504944 secs 2024-07-26T11:34:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16724.243561 secs 2024-07-26T11:34:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16706.504944 secs 2024-07-26T11:34:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16724.243561 secs 2024-07-26T11:34:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:37 smithi042 ceph-mon[32775]: pgmap v9413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:37 smithi164 ceph-mon[36970]: pgmap v9413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:37 smithi163 ceph-mon[36873]: pgmap v9413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:39 smithi164 ceph-mon[36970]: pgmap v9414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:34:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:39 smithi163 ceph-mon[36873]: pgmap v9414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:34:40.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:39 smithi042 ceph-mon[32775]: pgmap v9414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:34:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:41 smithi164 ceph-mon[36970]: pgmap v9415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16711.505085 secs 2024-07-26T11:34:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16729.243686 secs 2024-07-26T11:34:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:41 smithi163 ceph-mon[36873]: pgmap v9415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16711.505085 secs 2024-07-26T11:34:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16729.243686 secs 2024-07-26T11:34:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:41 smithi042 ceph-mon[32775]: pgmap v9415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16711.505085 secs 2024-07-26T11:34:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16729.243686 secs 2024-07-26T11:34:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:34:43.699+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:34:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:34:43.699+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:43.701 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:34:43.699+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:34:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:43 smithi164 ceph-mon[36970]: pgmap v9416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:43 smithi163 ceph-mon[36873]: pgmap v9416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:43 smithi042 ceph-mon[32775]: pgmap v9416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:45 smithi164 ceph-mon[36970]: pgmap v9417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:45 smithi163 ceph-mon[36873]: pgmap v9417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:45 smithi042 ceph-mon[32775]: pgmap v9417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16716.505244 secs 2024-07-26T11:34:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16734.243761 secs 2024-07-26T11:34:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16716.505244 secs 2024-07-26T11:34:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16734.243761 secs 2024-07-26T11:34:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16716.505244 secs 2024-07-26T11:34:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16734.243761 secs 2024-07-26T11:34:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:47 smithi164 ceph-mon[36970]: pgmap v9418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:47 smithi163 ceph-mon[36873]: pgmap v9418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:48.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:47 smithi042 ceph-mon[32775]: pgmap v9418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:49 smithi164 ceph-mon[36970]: pgmap v9419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:34:50.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:49 smithi163 ceph-mon[36873]: pgmap v9419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:34:50.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:49 smithi042 ceph-mon[32775]: pgmap v9419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:34:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:51 smithi164 ceph-mon[36970]: pgmap v9420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16721.505371 secs 2024-07-26T11:34:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16739.243888 secs 2024-07-26T11:34:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:51 smithi163 ceph-mon[36873]: pgmap v9420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16721.505371 secs 2024-07-26T11:34:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16739.243888 secs 2024-07-26T11:34:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:51 smithi042 ceph-mon[32775]: pgmap v9420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16721.505371 secs 2024-07-26T11:34:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16739.243888 secs 2024-07-26T11:34:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:53 smithi164 ceph-mon[36970]: pgmap v9421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:53 smithi163 ceph-mon[36873]: pgmap v9421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:54.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:53 smithi042 ceph-mon[32775]: pgmap v9421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:34:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:54 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:34:55.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:54 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:34:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:54 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:34:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:55 smithi164 ceph-mon[36970]: pgmap v9422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:34:56.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:34:56.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:55 smithi164 ceph-mon[36970]: pgmap v9423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:34:56.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:55 smithi164 ceph-mon[36970]: pgmap v9424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:34:56.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:55 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:55 smithi163 ceph-mon[36873]: pgmap v9422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:34:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:34:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:55 smithi163 ceph-mon[36873]: pgmap v9423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:34:56.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:55 smithi163 ceph-mon[36873]: pgmap v9424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:34:56.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:55 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:55 smithi042 ceph-mon[32775]: pgmap v9422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:34:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:34:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:34:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:55 smithi042 ceph-mon[32775]: pgmap v9423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:34:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:55 smithi042 ceph-mon[32775]: pgmap v9424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:34:56.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:55 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:34:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16726.505489 secs 2024-07-26T11:34:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16744.244018 secs 2024-07-26T11:34:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16726.505489 secs 2024-07-26T11:34:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16744.244018 secs 2024-07-26T11:34:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16726.505489 secs 2024-07-26T11:34:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16744.244018 secs 2024-07-26T11:34:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:34:58 smithi164 ceph-mon[36970]: pgmap v9425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:34:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:34:58 smithi163 ceph-mon[36873]: pgmap v9425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:34:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:34:58 smithi042 ceph-mon[32775]: pgmap v9425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:34:58.701 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:34:58.700+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:34:58.701 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:34:58.700+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:34:58.702 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:34:58.700+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:35:00.500+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006480 con 0x7ff680102000 2024-07-26T11:35:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:35:00.500+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:35:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:35:00.500+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006480 con 0x7ff680102000 2024-07-26T11:35:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:35:00.500+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:35:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:00 smithi042 ceph-mon[32775]: pgmap v9426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:35:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:00 smithi164 ceph-mon[36970]: pgmap v9426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:35:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:00 smithi163 ceph-mon[36873]: pgmap v9426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:35:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16731.505613 secs 2024-07-26T11:35:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16749.244160 secs 2024-07-26T11:35:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16731.505613 secs 2024-07-26T11:35:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16749.244160 secs 2024-07-26T11:35:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16731.505613 secs 2024-07-26T11:35:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16749.244160 secs 2024-07-26T11:35:02.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:02 smithi042 ceph-mon[32775]: pgmap v9427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:35:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:02 smithi164 ceph-mon[36970]: pgmap v9427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:35:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:02 smithi163 ceph-mon[36873]: pgmap v9427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:35:04.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:04 smithi042 ceph-mon[32775]: pgmap v9428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:35:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:04 smithi164 ceph-mon[36970]: pgmap v9428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:35:05.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:04 smithi163 ceph-mon[36873]: pgmap v9428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:35:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:06 smithi164 ceph-mon[36970]: pgmap v9429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:35:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16736.505735 secs 2024-07-26T11:35:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16754.244249 secs 2024-07-26T11:35:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:06 smithi163 ceph-mon[36873]: pgmap v9429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:35:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16736.505735 secs 2024-07-26T11:35:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16754.244249 secs 2024-07-26T11:35:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:06 smithi042 ceph-mon[32775]: pgmap v9429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:35:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16736.505735 secs 2024-07-26T11:35:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16754.244249 secs 2024-07-26T11:35:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:08 smithi164 ceph-mon[36970]: pgmap v9430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:09.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:08 smithi163 ceph-mon[36873]: pgmap v9430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:09.040 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:08 smithi042 ceph-mon[32775]: pgmap v9430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:10 smithi164 ceph-mon[36970]: pgmap v9431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:10 smithi163 ceph-mon[36873]: pgmap v9431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:10 smithi042 ceph-mon[32775]: pgmap v9431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16741.505860 secs 2024-07-26T11:35:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16759.244373 secs 2024-07-26T11:35:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16741.505860 secs 2024-07-26T11:35:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16759.244373 secs 2024-07-26T11:35:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16741.505860 secs 2024-07-26T11:35:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16759.244373 secs 2024-07-26T11:35:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:12 smithi164 ceph-mon[36970]: pgmap v9432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:12 smithi163 ceph-mon[36873]: pgmap v9432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:12 smithi042 ceph-mon[32775]: pgmap v9432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:35:13.702+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:35:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:35:13.702+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:35:13.702+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:35:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:14 smithi164 ceph-mon[36970]: pgmap v9433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:35:15.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:14 smithi163 ceph-mon[36873]: pgmap v9433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:35:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:14 smithi042 ceph-mon[32775]: pgmap v9433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:35:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:16 smithi164 ceph-mon[36970]: pgmap v9434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16746.506016 secs 2024-07-26T11:35:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16764.244496 secs 2024-07-26T11:35:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:16 smithi163 ceph-mon[36873]: pgmap v9434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16746.506016 secs 2024-07-26T11:35:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16764.244496 secs 2024-07-26T11:35:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:16 smithi042 ceph-mon[32775]: pgmap v9434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16746.506016 secs 2024-07-26T11:35:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16764.244496 secs 2024-07-26T11:35:18.950 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:18 smithi042 ceph-mon[32775]: pgmap v9435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:18 smithi164 ceph-mon[36970]: pgmap v9435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:19.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:18 smithi163 ceph-mon[36873]: pgmap v9435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:20 smithi164 ceph-mon[36970]: pgmap v9436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:20 smithi163 ceph-mon[36873]: pgmap v9436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:20 smithi042 ceph-mon[32775]: pgmap v9436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16751.506137 secs 2024-07-26T11:35:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16769.244624 secs 2024-07-26T11:35:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16751.506137 secs 2024-07-26T11:35:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16769.244624 secs 2024-07-26T11:35:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16751.506137 secs 2024-07-26T11:35:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16769.244624 secs 2024-07-26T11:35:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:22 smithi164 ceph-mon[36970]: pgmap v9437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:22 smithi163 ceph-mon[36873]: pgmap v9437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:22 smithi042 ceph-mon[32775]: pgmap v9437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:24 smithi164 ceph-mon[36970]: pgmap v9438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:35:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:24 smithi163 ceph-mon[36873]: pgmap v9438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:35:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:24 smithi042 ceph-mon[32775]: pgmap v9438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:35:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:26 smithi164 ceph-mon[36970]: pgmap v9439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16756.506261 secs 2024-07-26T11:35:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16774.244749 secs 2024-07-26T11:35:27.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:26 smithi163 ceph-mon[36873]: pgmap v9439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:27.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16756.506261 secs 2024-07-26T11:35:27.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16774.244749 secs 2024-07-26T11:35:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:26 smithi042 ceph-mon[32775]: pgmap v9439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16756.506261 secs 2024-07-26T11:35:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16774.244749 secs 2024-07-26T11:35:28.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:35:28.707+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:35:28.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:35:28.707+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:28.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:35:28.707+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:35:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:28 smithi164 ceph-mon[36970]: pgmap v9440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:29.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:28 smithi163 ceph-mon[36873]: pgmap v9440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:28 smithi042 ceph-mon[32775]: pgmap v9440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:30 smithi164 ceph-mon[36970]: pgmap v9441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:30 smithi163 ceph-mon[36873]: pgmap v9441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:31.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:30 smithi042 ceph-mon[32775]: pgmap v9441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16761.506381 secs 2024-07-26T11:35:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16779.244874 secs 2024-07-26T11:35:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16761.506381 secs 2024-07-26T11:35:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16779.244874 secs 2024-07-26T11:35:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16761.506381 secs 2024-07-26T11:35:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16779.244874 secs 2024-07-26T11:35:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:32 smithi164 ceph-mon[36970]: pgmap v9442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:32 smithi163 ceph-mon[36873]: pgmap v9442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:32 smithi042 ceph-mon[32775]: pgmap v9442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:34 smithi164 ceph-mon[36970]: pgmap v9443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:35:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:34 smithi163 ceph-mon[36873]: pgmap v9443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:35:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:34 smithi042 ceph-mon[32775]: pgmap v9443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:35:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:36 smithi164 ceph-mon[36970]: pgmap v9444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16766.506509 secs 2024-07-26T11:35:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16784.244943 secs 2024-07-26T11:35:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:36 smithi163 ceph-mon[36873]: pgmap v9444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16766.506509 secs 2024-07-26T11:35:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16784.244943 secs 2024-07-26T11:35:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:36 smithi042 ceph-mon[32775]: pgmap v9444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16766.506509 secs 2024-07-26T11:35:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16784.244943 secs 2024-07-26T11:35:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:38 smithi164 ceph-mon[36970]: pgmap v9445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:38 smithi163 ceph-mon[36873]: pgmap v9445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:38 smithi042 ceph-mon[32775]: pgmap v9445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:40 smithi164 ceph-mon[36970]: pgmap v9446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:41.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:40 smithi163 ceph-mon[36873]: pgmap v9446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:40 smithi042 ceph-mon[32775]: pgmap v9446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16771.506634 secs 2024-07-26T11:35:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16789.245069 secs 2024-07-26T11:35:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16771.506634 secs 2024-07-26T11:35:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16789.245069 secs 2024-07-26T11:35:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16771.506634 secs 2024-07-26T11:35:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16789.245069 secs 2024-07-26T11:35:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:42 smithi164 ceph-mon[36970]: pgmap v9447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:42 smithi163 ceph-mon[36873]: pgmap v9447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:42 smithi042 ceph-mon[32775]: pgmap v9447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:43.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:35:43.708+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:35:43.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:35:43.708+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:43.709 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:35:43.708+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:35:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:44 smithi164 ceph-mon[36970]: pgmap v9448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:35:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:44 smithi163 ceph-mon[36873]: pgmap v9448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:35:45.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:44 smithi042 ceph-mon[32775]: pgmap v9448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:35:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:45 smithi164 ceph-mon[36970]: pgmap v9449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:45 smithi163 ceph-mon[36873]: pgmap v9449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:45 smithi042 ceph-mon[32775]: pgmap v9449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16776.506774 secs 2024-07-26T11:35:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16794.245188 secs 2024-07-26T11:35:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16776.506774 secs 2024-07-26T11:35:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16794.245188 secs 2024-07-26T11:35:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16776.506774 secs 2024-07-26T11:35:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16794.245188 secs 2024-07-26T11:35:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:47 smithi164 ceph-mon[36970]: pgmap v9450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:47 smithi163 ceph-mon[36873]: pgmap v9450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:48.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:47 smithi042 ceph-mon[32775]: pgmap v9450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:50 smithi164 ceph-mon[36970]: pgmap v9451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:50 smithi163 ceph-mon[36873]: pgmap v9451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:50 smithi042 ceph-mon[32775]: pgmap v9451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16781.506913 secs 2024-07-26T11:35:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16799.245308 secs 2024-07-26T11:35:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16781.506913 secs 2024-07-26T11:35:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16799.245308 secs 2024-07-26T11:35:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16781.506913 secs 2024-07-26T11:35:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16799.245308 secs 2024-07-26T11:35:52.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:52 smithi163 ceph-mon[36873]: pgmap v9452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:52 smithi164 ceph-mon[36970]: pgmap v9452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:52 smithi042 ceph-mon[32775]: pgmap v9452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:35:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:54 smithi164 ceph-mon[36970]: pgmap v9453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:35:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:54 smithi163 ceph-mon[36873]: pgmap v9453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:35:55.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:54 smithi042 ceph-mon[32775]: pgmap v9453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:35:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:56 smithi164 ceph-mon[36970]: pgmap v9454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:35:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16786.507006 secs 2024-07-26T11:35:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:35:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:35:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:35:56.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16804.245428 secs 2024-07-26T11:35:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:56 smithi163 ceph-mon[36873]: pgmap v9454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:35:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16786.507006 secs 2024-07-26T11:35:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:35:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:35:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:35:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16804.245428 secs 2024-07-26T11:35:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:56 smithi042 ceph-mon[32775]: pgmap v9454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:35:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:35:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16786.507006 secs 2024-07-26T11:35:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:35:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:35:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:35:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16804.245428 secs 2024-07-26T11:35:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:57 smithi164 ceph-mon[36970]: pgmap v9455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:35:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:57 smithi164 ceph-mon[36970]: pgmap v9456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:35:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:57 smithi163 ceph-mon[36873]: pgmap v9455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:35:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:57 smithi163 ceph-mon[36873]: pgmap v9456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:35:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:57 smithi042 ceph-mon[32775]: pgmap v9455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:35:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:57 smithi042 ceph-mon[32775]: pgmap v9456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:35:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:35:58.713+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:35:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:35:58.713+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:58.715 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:35:58.713+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:35:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:35:59 smithi164 ceph-mon[36970]: pgmap v9457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:36:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:35:59 smithi163 ceph-mon[36873]: pgmap v9457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:36:00.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:35:59 smithi042 ceph-mon[32775]: pgmap v9457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:36:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:01 smithi164 ceph-mon[36970]: pgmap v9458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:36:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16791.507138 secs 2024-07-26T11:36:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16809.245530 secs 2024-07-26T11:36:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:01 smithi163 ceph-mon[36873]: pgmap v9458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:36:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16791.507138 secs 2024-07-26T11:36:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16809.245530 secs 2024-07-26T11:36:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:01 smithi042 ceph-mon[32775]: pgmap v9458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:36:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16791.507138 secs 2024-07-26T11:36:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16809.245530 secs 2024-07-26T11:36:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:03 smithi164 ceph-mon[36970]: pgmap v9459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:36:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:03 smithi163 ceph-mon[36873]: pgmap v9459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:36:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:03 smithi042 ceph-mon[32775]: pgmap v9459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:36:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:05 smithi164 ceph-mon[36970]: pgmap v9460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:36:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:05 smithi163 ceph-mon[36873]: pgmap v9460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:36:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:05 smithi042 ceph-mon[32775]: pgmap v9460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:36:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16796.507281 secs 2024-07-26T11:36:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16814.245627 secs 2024-07-26T11:36:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16796.507281 secs 2024-07-26T11:36:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16814.245627 secs 2024-07-26T11:36:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16796.507281 secs 2024-07-26T11:36:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16814.245627 secs 2024-07-26T11:36:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:07 smithi164 ceph-mon[36970]: pgmap v9461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:36:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:07 smithi163 ceph-mon[36873]: pgmap v9461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:36:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:07 smithi042 ceph-mon[32775]: pgmap v9461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:36:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:09 smithi164 ceph-mon[36970]: pgmap v9462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:09 smithi163 ceph-mon[36873]: pgmap v9462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:09 smithi042 ceph-mon[32775]: pgmap v9462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:11 smithi164 ceph-mon[36970]: pgmap v9463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16801.507433 secs 2024-07-26T11:36:11.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16819.245749 secs 2024-07-26T11:36:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:11 smithi163 ceph-mon[36873]: pgmap v9463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16801.507433 secs 2024-07-26T11:36:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16819.245749 secs 2024-07-26T11:36:12.016 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:11 smithi042 ceph-mon[32775]: pgmap v9463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:12.017 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16801.507433 secs 2024-07-26T11:36:12.017 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16819.245749 secs 2024-07-26T11:36:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:36:13.714+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:36:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:36:13.714+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:36:13.714+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:36:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:13 smithi164 ceph-mon[36970]: pgmap v9464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:13 smithi163 ceph-mon[36873]: pgmap v9464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:13 smithi042 ceph-mon[32775]: pgmap v9464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:15 smithi164 ceph-mon[36970]: pgmap v9465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:15 smithi163 ceph-mon[36873]: pgmap v9465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:15 smithi042 ceph-mon[32775]: pgmap v9465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16806.507588 secs 2024-07-26T11:36:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16824.245870 secs 2024-07-26T11:36:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16806.507588 secs 2024-07-26T11:36:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16824.245870 secs 2024-07-26T11:36:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16806.507588 secs 2024-07-26T11:36:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16824.245870 secs 2024-07-26T11:36:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:17 smithi164 ceph-mon[36970]: pgmap v9466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:17 smithi163 ceph-mon[36873]: pgmap v9466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:18.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:17 smithi042 ceph-mon[32775]: pgmap v9466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:19 smithi164 ceph-mon[36970]: pgmap v9467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:19 smithi163 ceph-mon[36873]: pgmap v9467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:19 smithi042 ceph-mon[32775]: pgmap v9467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:21 smithi163 ceph-mon[36873]: pgmap v9468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16811.507707 secs 2024-07-26T11:36:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16829.245993 secs 2024-07-26T11:36:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:21 smithi042 ceph-mon[32775]: pgmap v9468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16811.507707 secs 2024-07-26T11:36:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16829.245993 secs 2024-07-26T11:36:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:21 smithi164 ceph-mon[36970]: pgmap v9468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16811.507707 secs 2024-07-26T11:36:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16829.245993 secs 2024-07-26T11:36:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:23 smithi163 ceph-mon[36873]: pgmap v9469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:24.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:23 smithi042 ceph-mon[32775]: pgmap v9469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:23 smithi164 ceph-mon[36970]: pgmap v9469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:25 smithi163 ceph-mon[36873]: pgmap v9470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:25 smithi042 ceph-mon[32775]: pgmap v9470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:25 smithi164 ceph-mon[36970]: pgmap v9470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16816.507850 secs 2024-07-26T11:36:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16834.246117 secs 2024-07-26T11:36:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16816.507850 secs 2024-07-26T11:36:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16834.246117 secs 2024-07-26T11:36:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16816.507850 secs 2024-07-26T11:36:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16834.246117 secs 2024-07-26T11:36:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:27 smithi163 ceph-mon[36873]: pgmap v9471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:27 smithi042 ceph-mon[32775]: pgmap v9471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:27 smithi164 ceph-mon[36970]: pgmap v9471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:28.721 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:36:28.719+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:36:28.721 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:36:28.719+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:28.721 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:36:28.719+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:36:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:29 smithi163 ceph-mon[36873]: pgmap v9472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:29 smithi042 ceph-mon[32775]: pgmap v9472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:29 smithi164 ceph-mon[36970]: pgmap v9472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:36:30.515+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658006480 con 0x7ff680102000 2024-07-26T11:36:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:36:30.515+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:36:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:36:30.515+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658006480 con 0x7ff680102000 2024-07-26T11:36:30.516 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:36:30.515+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:36:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:31 smithi163 ceph-mon[36873]: pgmap v9473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16821.507957 secs 2024-07-26T11:36:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16839.246209 secs 2024-07-26T11:36:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:31 smithi042 ceph-mon[32775]: pgmap v9473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16821.507957 secs 2024-07-26T11:36:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16839.246209 secs 2024-07-26T11:36:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:31 smithi164 ceph-mon[36970]: pgmap v9473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16821.507957 secs 2024-07-26T11:36:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16839.246209 secs 2024-07-26T11:36:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:33 smithi163 ceph-mon[36873]: pgmap v9474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:34.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:33 smithi042 ceph-mon[32775]: pgmap v9474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:33 smithi164 ceph-mon[36970]: pgmap v9474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:35 smithi164 ceph-mon[36970]: pgmap v9475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:35 smithi163 ceph-mon[36873]: pgmap v9475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:35 smithi042 ceph-mon[32775]: pgmap v9475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16826.508113 secs 2024-07-26T11:36:37.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16844.246329 secs 2024-07-26T11:36:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16826.508113 secs 2024-07-26T11:36:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16844.246329 secs 2024-07-26T11:36:37.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16826.508113 secs 2024-07-26T11:36:37.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16844.246329 secs 2024-07-26T11:36:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:37 smithi164 ceph-mon[36970]: pgmap v9476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:37 smithi163 ceph-mon[36873]: pgmap v9476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:37 smithi042 ceph-mon[32775]: pgmap v9476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:39 smithi042 ceph-mon[32775]: pgmap v9477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:39 smithi164 ceph-mon[36970]: pgmap v9477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:39 smithi163 ceph-mon[36873]: pgmap v9477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:41 smithi164 ceph-mon[36970]: pgmap v9478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:41 smithi163 ceph-mon[36873]: pgmap v9478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:41 smithi042 ceph-mon[32775]: pgmap v9478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16831.508268 secs 2024-07-26T11:36:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16849.246464 secs 2024-07-26T11:36:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16831.508268 secs 2024-07-26T11:36:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16849.246464 secs 2024-07-26T11:36:42.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16831.508268 secs 2024-07-26T11:36:42.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16849.246464 secs 2024-07-26T11:36:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:43 smithi164 ceph-mon[36970]: pgmap v9479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:43 smithi163 ceph-mon[36873]: pgmap v9479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:43 smithi042 ceph-mon[32775]: pgmap v9479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:36:43.719+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:36:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:36:43.719+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:36:43.719+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:36:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:45 smithi164 ceph-mon[36970]: pgmap v9480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:45.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:45 smithi163 ceph-mon[36873]: pgmap v9480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:45 smithi042 ceph-mon[32775]: pgmap v9480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16836.508398 secs 2024-07-26T11:36:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:47 smithi164 ceph-mon[36970]: pgmap v9481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16854.246562 secs 2024-07-26T11:36:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16836.508398 secs 2024-07-26T11:36:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:47 smithi163 ceph-mon[36873]: pgmap v9481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16854.246562 secs 2024-07-26T11:36:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16836.508398 secs 2024-07-26T11:36:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:47 smithi042 ceph-mon[32775]: pgmap v9481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16854.246562 secs 2024-07-26T11:36:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:49 smithi164 ceph-mon[36970]: pgmap v9482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:49 smithi163 ceph-mon[36873]: pgmap v9482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:49 smithi042 ceph-mon[32775]: pgmap v9482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:51 smithi163 ceph-mon[36873]: pgmap v9483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:51 smithi042 ceph-mon[32775]: pgmap v9483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:51 smithi164 ceph-mon[36970]: pgmap v9483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16841.508538 secs 2024-07-26T11:36:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16859.246691 secs 2024-07-26T11:36:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16841.508538 secs 2024-07-26T11:36:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16859.246691 secs 2024-07-26T11:36:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16841.508538 secs 2024-07-26T11:36:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16859.246691 secs 2024-07-26T11:36:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:53 smithi163 ceph-mon[36873]: pgmap v9484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:53.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:53 smithi042 ceph-mon[32775]: pgmap v9484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:53 smithi164 ceph-mon[36970]: pgmap v9484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:55 smithi163 ceph-mon[36873]: pgmap v9485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:55 smithi042 ceph-mon[32775]: pgmap v9485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:55 smithi164 ceph-mon[36970]: pgmap v9485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:36:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16846.508688 secs 2024-07-26T11:36:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16846.508688 secs 2024-07-26T11:36:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16846.508688 secs 2024-07-26T11:36:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:57 smithi163 ceph-mon[36873]: pgmap v9486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:36:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16864.246787 secs 2024-07-26T11:36:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:36:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:36:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:57 smithi163 ceph-mon[36873]: pgmap v9487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:36:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:57 smithi163 ceph-mon[36873]: pgmap v9488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:36:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:36:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:57 smithi042 ceph-mon[32775]: pgmap v9486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:36:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16864.246787 secs 2024-07-26T11:36:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:36:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:36:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:57 smithi042 ceph-mon[32775]: pgmap v9487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:36:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:57 smithi042 ceph-mon[32775]: pgmap v9488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:36:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:36:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:57 smithi164 ceph-mon[36970]: pgmap v9486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:36:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:36:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16864.246787 secs 2024-07-26T11:36:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:36:57.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:36:57.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:57 smithi164 ceph-mon[36970]: pgmap v9487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:36:57.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:57 smithi164 ceph-mon[36970]: pgmap v9488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:36:57.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:36:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:36:58.725+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:36:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:36:58.725+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:36:58.725+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:37:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:36:59 smithi163 ceph-mon[36873]: pgmap v9489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:37:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:36:59 smithi042 ceph-mon[32775]: pgmap v9489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:37:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:36:59 smithi164 ceph-mon[36970]: pgmap v9489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:37:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:01 smithi163 ceph-mon[36873]: pgmap v9490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:37:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16851.508820 secs 2024-07-26T11:37:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16869.246906 secs 2024-07-26T11:37:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:01 smithi042 ceph-mon[32775]: pgmap v9490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:37:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16851.508820 secs 2024-07-26T11:37:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16869.246906 secs 2024-07-26T11:37:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:01 smithi164 ceph-mon[36970]: pgmap v9490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:37:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16851.508820 secs 2024-07-26T11:37:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16869.246906 secs 2024-07-26T11:37:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:03 smithi163 ceph-mon[36873]: pgmap v9491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:37:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:03 smithi042 ceph-mon[32775]: pgmap v9491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:37:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:03 smithi164 ceph-mon[36970]: pgmap v9491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:37:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:05 smithi163 ceph-mon[36873]: pgmap v9492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:37:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:05 smithi042 ceph-mon[32775]: pgmap v9492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:37:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:05 smithi164 ceph-mon[36970]: pgmap v9492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:37:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16856.508941 secs 2024-07-26T11:37:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16874.247022 secs 2024-07-26T11:37:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16856.508941 secs 2024-07-26T11:37:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16874.247022 secs 2024-07-26T11:37:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16856.508941 secs 2024-07-26T11:37:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16874.247022 secs 2024-07-26T11:37:08.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:07 smithi042 ceph-mon[32775]: pgmap v9493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:37:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:07 smithi164 ceph-mon[36970]: pgmap v9493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:37:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:07 smithi163 ceph-mon[36873]: pgmap v9493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:37:10.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:09 smithi042 ceph-mon[32775]: pgmap v9494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:37:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:09 smithi164 ceph-mon[36970]: pgmap v9494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:37:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:09 smithi163 ceph-mon[36873]: pgmap v9494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:37:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:11 smithi042 ceph-mon[32775]: pgmap v9495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16861.509087 secs 2024-07-26T11:37:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16879.247158 secs 2024-07-26T11:37:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:11 smithi164 ceph-mon[36970]: pgmap v9495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16861.509087 secs 2024-07-26T11:37:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16879.247158 secs 2024-07-26T11:37:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:11 smithi163 ceph-mon[36873]: pgmap v9495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16861.509087 secs 2024-07-26T11:37:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16879.247158 secs 2024-07-26T11:37:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:37:13.726+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:37:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:37:13.726+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:37:13.726+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:37:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:13 smithi042 ceph-mon[32775]: pgmap v9496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:14.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:13 smithi164 ceph-mon[36970]: pgmap v9496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:13 smithi163 ceph-mon[36873]: pgmap v9496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:15 smithi042 ceph-mon[32775]: pgmap v9497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:15 smithi164 ceph-mon[36970]: pgmap v9497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:15 smithi163 ceph-mon[36873]: pgmap v9497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:17.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16866.509251 secs 2024-07-26T11:37:17.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16884.247295 secs 2024-07-26T11:37:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16866.509251 secs 2024-07-26T11:37:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16884.247295 secs 2024-07-26T11:37:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16866.509251 secs 2024-07-26T11:37:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16884.247295 secs 2024-07-26T11:37:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:17 smithi042 ceph-mon[32775]: pgmap v9498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:17 smithi164 ceph-mon[36970]: pgmap v9498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:17 smithi163 ceph-mon[36873]: pgmap v9498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:19 smithi164 ceph-mon[36970]: pgmap v9499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:37:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:19 smithi163 ceph-mon[36873]: pgmap v9499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:37:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:19 smithi042 ceph-mon[32775]: pgmap v9499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:37:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:21 smithi164 ceph-mon[36970]: pgmap v9500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16871.509378 secs 2024-07-26T11:37:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16889.247411 secs 2024-07-26T11:37:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:21 smithi163 ceph-mon[36873]: pgmap v9500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16871.509378 secs 2024-07-26T11:37:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16889.247411 secs 2024-07-26T11:37:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:21 smithi042 ceph-mon[32775]: pgmap v9500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16871.509378 secs 2024-07-26T11:37:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16889.247411 secs 2024-07-26T11:37:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:23 smithi042 ceph-mon[32775]: pgmap v9501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:23 smithi164 ceph-mon[36970]: pgmap v9501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:23 smithi163 ceph-mon[36873]: pgmap v9501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:25 smithi163 ceph-mon[36873]: pgmap v9502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:25 smithi042 ceph-mon[32775]: pgmap v9502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:25 smithi164 ceph-mon[36970]: pgmap v9502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16876.509517 secs 2024-07-26T11:37:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16894.247530 secs 2024-07-26T11:37:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16876.509517 secs 2024-07-26T11:37:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16894.247530 secs 2024-07-26T11:37:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16876.509517 secs 2024-07-26T11:37:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16894.247530 secs 2024-07-26T11:37:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:27 smithi042 ceph-mon[32775]: pgmap v9503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:27 smithi164 ceph-mon[36970]: pgmap v9503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:27 smithi163 ceph-mon[36873]: pgmap v9503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:28.733 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:37:28.732+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:37:28.733 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:37:28.732+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:37:28.733 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:37:28.732+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:29 smithi042 ceph-mon[32775]: pgmap v9504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:37:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:29 smithi164 ceph-mon[36970]: pgmap v9504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:37:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:29 smithi163 ceph-mon[36873]: pgmap v9504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:37:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:31 smithi042 ceph-mon[32775]: pgmap v9505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16881.509650 secs 2024-07-26T11:37:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16899.247625 secs 2024-07-26T11:37:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:31 smithi164 ceph-mon[36970]: pgmap v9505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16881.509650 secs 2024-07-26T11:37:32.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16899.247625 secs 2024-07-26T11:37:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:31 smithi163 ceph-mon[36873]: pgmap v9505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16881.509650 secs 2024-07-26T11:37:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16899.247625 secs 2024-07-26T11:37:34.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:33 smithi042 ceph-mon[32775]: pgmap v9506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:33 smithi164 ceph-mon[36970]: pgmap v9506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:33 smithi163 ceph-mon[36873]: pgmap v9506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:35 smithi042 ceph-mon[32775]: pgmap v9507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:35 smithi164 ceph-mon[36970]: pgmap v9507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:35 smithi163 ceph-mon[36873]: pgmap v9507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16886.509726 secs 2024-07-26T11:37:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16904.247777 secs 2024-07-26T11:37:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16886.509726 secs 2024-07-26T11:37:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16904.247777 secs 2024-07-26T11:37:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16886.509726 secs 2024-07-26T11:37:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16904.247777 secs 2024-07-26T11:37:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:37 smithi042 ceph-mon[32775]: pgmap v9508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:37 smithi164 ceph-mon[36970]: pgmap v9508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:37 smithi163 ceph-mon[36873]: pgmap v9508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:40.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:39 smithi042 ceph-mon[32775]: pgmap v9509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:37:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:39 smithi164 ceph-mon[36970]: pgmap v9509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:37:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:39 smithi163 ceph-mon[36873]: pgmap v9509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:37:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:41 smithi042 ceph-mon[32775]: pgmap v9510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16891.509844 secs 2024-07-26T11:37:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16909.247900 secs 2024-07-26T11:37:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:41 smithi164 ceph-mon[36970]: pgmap v9510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16891.509844 secs 2024-07-26T11:37:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16909.247900 secs 2024-07-26T11:37:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:41 smithi163 ceph-mon[36873]: pgmap v9510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16891.509844 secs 2024-07-26T11:37:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16909.247900 secs 2024-07-26T11:37:43.735 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:37:43.734+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:37:43.736 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:37:43.734+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:37:43.736 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:37:43.734+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:44.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:43 smithi042 ceph-mon[32775]: pgmap v9511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:43 smithi164 ceph-mon[36970]: pgmap v9511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:43 smithi163 ceph-mon[36873]: pgmap v9511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:46.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:45 smithi042 ceph-mon[32775]: pgmap v9512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:45 smithi164 ceph-mon[36970]: pgmap v9512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:45 smithi163 ceph-mon[36873]: pgmap v9512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:47.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16896.509972 secs 2024-07-26T11:37:47.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16914.247996 secs 2024-07-26T11:37:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16896.509972 secs 2024-07-26T11:37:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16914.247996 secs 2024-07-26T11:37:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16896.509972 secs 2024-07-26T11:37:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16914.247996 secs 2024-07-26T11:37:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:47 smithi042 ceph-mon[32775]: pgmap v9513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:47 smithi164 ceph-mon[36970]: pgmap v9513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:47 smithi163 ceph-mon[36873]: pgmap v9513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:50.147 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:49 smithi042 ceph-mon[32775]: pgmap v9514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:37:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:49 smithi164 ceph-mon[36970]: pgmap v9514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:37:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:49 smithi163 ceph-mon[36873]: pgmap v9514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:37:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:51 smithi042 ceph-mon[32775]: pgmap v9515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16901.510097 secs 2024-07-26T11:37:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16919.248122 secs 2024-07-26T11:37:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:51 smithi164 ceph-mon[36970]: pgmap v9515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16901.510097 secs 2024-07-26T11:37:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16919.248122 secs 2024-07-26T11:37:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:51 smithi163 ceph-mon[36873]: pgmap v9515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16901.510097 secs 2024-07-26T11:37:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16919.248122 secs 2024-07-26T11:37:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:53 smithi042 ceph-mon[32775]: pgmap v9516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:53 smithi164 ceph-mon[36970]: pgmap v9516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:53 smithi163 ceph-mon[36873]: pgmap v9516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:55 smithi042 ceph-mon[32775]: pgmap v9517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:55 smithi164 ceph-mon[36970]: pgmap v9517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:56.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:55 smithi163 ceph-mon[36873]: pgmap v9517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:37:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16906.510161 secs 2024-07-26T11:37:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16924.248214 secs 2024-07-26T11:37:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:56 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:37:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16906.510161 secs 2024-07-26T11:37:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16924.248214 secs 2024-07-26T11:37:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:56 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:37:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16906.510161 secs 2024-07-26T11:37:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16924.248214 secs 2024-07-26T11:37:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:56 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:37:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:57 smithi042 ceph-mon[32775]: pgmap v9518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:37:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:37:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:37:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:57 smithi164 ceph-mon[36970]: pgmap v9518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:37:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:37:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:37:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:57 smithi163 ceph-mon[36873]: pgmap v9518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:37:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:37:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:37:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:37:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:37:58.740+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:37:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:37:58.740+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:58.741 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:37:58.740+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:37:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:58 smithi042 ceph-mon[32775]: pgmap v9519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:37:59.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:37:58 smithi042 ceph-mon[32775]: pgmap v9520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:37:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:58 smithi164 ceph-mon[36970]: pgmap v9519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:37:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:37:58 smithi164 ceph-mon[36970]: pgmap v9520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:37:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:58 smithi163 ceph-mon[36873]: pgmap v9519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:37:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:37:58 smithi163 ceph-mon[36873]: pgmap v9520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:38:00.530 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:38:00.529+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006480 con 0x7ff680102000 2024-07-26T11:38:00.530 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:38:00.529+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:38:00.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:38:00.529+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:38:00.531 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:38:00.529+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006480 con 0x7ff680102000 2024-07-26T11:38:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:00 smithi042 ceph-mon[32775]: pgmap v9521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:38:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:00 smithi164 ceph-mon[36970]: pgmap v9521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:38:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:00 smithi163 ceph-mon[36873]: pgmap v9521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:38:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16911.510283 secs 2024-07-26T11:38:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16929.248336 secs 2024-07-26T11:38:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16911.510283 secs 2024-07-26T11:38:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16929.248336 secs 2024-07-26T11:38:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16911.510283 secs 2024-07-26T11:38:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16929.248336 secs 2024-07-26T11:38:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:02 smithi042 ceph-mon[32775]: pgmap v9522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:38:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:02 smithi164 ceph-mon[36970]: pgmap v9522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:38:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:02 smithi163 ceph-mon[36873]: pgmap v9522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:38:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:04 smithi042 ceph-mon[32775]: pgmap v9523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:38:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:04 smithi164 ceph-mon[36970]: pgmap v9523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:38:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:04 smithi163 ceph-mon[36873]: pgmap v9523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:38:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:06 smithi042 ceph-mon[32775]: pgmap v9524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:38:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16916.510387 secs 2024-07-26T11:38:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16934.248458 secs 2024-07-26T11:38:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:06 smithi164 ceph-mon[36970]: pgmap v9524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:38:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16916.510387 secs 2024-07-26T11:38:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16934.248458 secs 2024-07-26T11:38:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:06 smithi163 ceph-mon[36873]: pgmap v9524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:38:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16916.510387 secs 2024-07-26T11:38:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16934.248458 secs 2024-07-26T11:38:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:08 smithi042 ceph-mon[32775]: pgmap v9525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:38:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:08 smithi164 ceph-mon[36970]: pgmap v9525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:38:09.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:08 smithi163 ceph-mon[36873]: pgmap v9525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:38:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:10 smithi042 ceph-mon[32775]: pgmap v9526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:10 smithi164 ceph-mon[36970]: pgmap v9526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:10 smithi163 ceph-mon[36873]: pgmap v9526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16921.510511 secs 2024-07-26T11:38:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16939.248583 secs 2024-07-26T11:38:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16921.510511 secs 2024-07-26T11:38:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16939.248583 secs 2024-07-26T11:38:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16921.510511 secs 2024-07-26T11:38:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16939.248583 secs 2024-07-26T11:38:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:12 smithi042 ceph-mon[32775]: pgmap v9527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:12 smithi164 ceph-mon[36970]: pgmap v9527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:12 smithi163 ceph-mon[36873]: pgmap v9527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:38:13.741+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:38:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:38:13.741+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:38:13.741+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:38:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:14 smithi042 ceph-mon[32775]: pgmap v9528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:14 smithi164 ceph-mon[36970]: pgmap v9528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:14 smithi163 ceph-mon[36873]: pgmap v9528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:16 smithi042 ceph-mon[32775]: pgmap v9529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16926.510649 secs 2024-07-26T11:38:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16944.248706 secs 2024-07-26T11:38:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:16 smithi164 ceph-mon[36970]: pgmap v9529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16926.510649 secs 2024-07-26T11:38:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16944.248706 secs 2024-07-26T11:38:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:16 smithi163 ceph-mon[36873]: pgmap v9529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16926.510649 secs 2024-07-26T11:38:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16944.248706 secs 2024-07-26T11:38:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:18 smithi042 ceph-mon[32775]: pgmap v9530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:18 smithi164 ceph-mon[36970]: pgmap v9530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:19.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:18 smithi163 ceph-mon[36873]: pgmap v9530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:20 smithi042 ceph-mon[32775]: pgmap v9531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:20 smithi164 ceph-mon[36970]: pgmap v9531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:20 smithi163 ceph-mon[36873]: pgmap v9531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16931.510791 secs 2024-07-26T11:38:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16949.248828 secs 2024-07-26T11:38:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16931.510791 secs 2024-07-26T11:38:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16949.248828 secs 2024-07-26T11:38:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16931.510791 secs 2024-07-26T11:38:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16949.248828 secs 2024-07-26T11:38:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:22 smithi042 ceph-mon[32775]: pgmap v9532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:22 smithi164 ceph-mon[36970]: pgmap v9532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:22 smithi163 ceph-mon[36873]: pgmap v9532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:24 smithi042 ceph-mon[32775]: pgmap v9533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:24 smithi164 ceph-mon[36970]: pgmap v9533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:25.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:24 smithi163 ceph-mon[36873]: pgmap v9533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:26 smithi042 ceph-mon[32775]: pgmap v9534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16936.510915 secs 2024-07-26T11:38:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16954.248970 secs 2024-07-26T11:38:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:26 smithi164 ceph-mon[36970]: pgmap v9534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16936.510915 secs 2024-07-26T11:38:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16954.248970 secs 2024-07-26T11:38:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:26 smithi163 ceph-mon[36873]: pgmap v9534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16936.510915 secs 2024-07-26T11:38:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16954.248970 secs 2024-07-26T11:38:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:38:28.747+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:38:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:38:28.747+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:38:28.747+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:38:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:28 smithi042 ceph-mon[32775]: pgmap v9535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:28 smithi164 ceph-mon[36970]: pgmap v9535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:28 smithi163 ceph-mon[36873]: pgmap v9535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:30 smithi164 ceph-mon[36970]: pgmap v9536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:31.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:30 smithi163 ceph-mon[36873]: pgmap v9536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:31.280 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:30 smithi042 ceph-mon[32775]: pgmap v9536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16941.511020 secs 2024-07-26T11:38:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16959.249112 secs 2024-07-26T11:38:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16941.511020 secs 2024-07-26T11:38:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16959.249112 secs 2024-07-26T11:38:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16941.511020 secs 2024-07-26T11:38:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16959.249112 secs 2024-07-26T11:38:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:32 smithi164 ceph-mon[36970]: pgmap v9537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:32 smithi163 ceph-mon[36873]: pgmap v9537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:32 smithi042 ceph-mon[32775]: pgmap v9537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:34 smithi164 ceph-mon[36970]: pgmap v9538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:34 smithi163 ceph-mon[36873]: pgmap v9538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:34 smithi042 ceph-mon[32775]: pgmap v9538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:36 smithi164 ceph-mon[36970]: pgmap v9539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16946.511144 secs 2024-07-26T11:38:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16964.249225 secs 2024-07-26T11:38:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:36 smithi163 ceph-mon[36873]: pgmap v9539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16946.511144 secs 2024-07-26T11:38:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16964.249225 secs 2024-07-26T11:38:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:36 smithi042 ceph-mon[32775]: pgmap v9539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16946.511144 secs 2024-07-26T11:38:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16964.249225 secs 2024-07-26T11:38:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:38 smithi164 ceph-mon[36970]: pgmap v9540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:38 smithi163 ceph-mon[36873]: pgmap v9540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:38 smithi042 ceph-mon[32775]: pgmap v9540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:39 smithi163 ceph-mon[36873]: pgmap v9541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:39 smithi042 ceph-mon[32775]: pgmap v9541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:39 smithi164 ceph-mon[36970]: pgmap v9541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:41.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16951.511267 secs 2024-07-26T11:38:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16951.511267 secs 2024-07-26T11:38:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16951.511267 secs 2024-07-26T11:38:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:42 smithi042 ceph-mon[32775]: pgmap v9542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16969.249335 secs 2024-07-26T11:38:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:42 smithi164 ceph-mon[36970]: pgmap v9542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16969.249335 secs 2024-07-26T11:38:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:42 smithi163 ceph-mon[36873]: pgmap v9542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16969.249335 secs 2024-07-26T11:38:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:38:43.748+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:38:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:38:43.748+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:38:43.748+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:38:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:44 smithi042 ceph-mon[32775]: pgmap v9543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:44 smithi164 ceph-mon[36970]: pgmap v9543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:44.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:44 smithi163 ceph-mon[36873]: pgmap v9543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:46 smithi042 ceph-mon[32775]: pgmap v9544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:46.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16956.511351 secs 2024-07-26T11:38:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:46 smithi164 ceph-mon[36970]: pgmap v9544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16956.511351 secs 2024-07-26T11:38:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:46 smithi163 ceph-mon[36873]: pgmap v9544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16956.511351 secs 2024-07-26T11:38:47.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16974.249454 secs 2024-07-26T11:38:47.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16974.249454 secs 2024-07-26T11:38:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16974.249454 secs 2024-07-26T11:38:48.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:48 smithi042 ceph-mon[32775]: pgmap v9545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:48 smithi164 ceph-mon[36970]: pgmap v9545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:48.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:48 smithi163 ceph-mon[36873]: pgmap v9545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:50 smithi042 ceph-mon[32775]: pgmap v9546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:50 smithi164 ceph-mon[36970]: pgmap v9546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:50 smithi163 ceph-mon[36873]: pgmap v9546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16961.511474 secs 2024-07-26T11:38:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16961.511474 secs 2024-07-26T11:38:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16961.511474 secs 2024-07-26T11:38:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:52 smithi042 ceph-mon[32775]: pgmap v9547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16979.249580 secs 2024-07-26T11:38:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:52 smithi164 ceph-mon[36970]: pgmap v9547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16979.249580 secs 2024-07-26T11:38:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:52 smithi163 ceph-mon[36873]: pgmap v9547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16979.249580 secs 2024-07-26T11:38:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:54 smithi042 ceph-mon[32775]: pgmap v9548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:54 smithi164 ceph-mon[36970]: pgmap v9548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:54 smithi163 ceph-mon[36873]: pgmap v9548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:56 smithi042 ceph-mon[32775]: pgmap v9549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:56.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16966.511596 secs 2024-07-26T11:38:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:56 smithi164 ceph-mon[36970]: pgmap v9549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16966.511596 secs 2024-07-26T11:38:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:56 smithi163 ceph-mon[36873]: pgmap v9549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:38:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16966.511596 secs 2024-07-26T11:38:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16984.249703 secs 2024-07-26T11:38:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16984.249703 secs 2024-07-26T11:38:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16984.249703 secs 2024-07-26T11:38:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:58 smithi042 ceph-mon[32775]: pgmap v9550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:38:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:38:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:38:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:58 smithi042 ceph-mon[32775]: pgmap v9551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:38:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:58 smithi042 ceph-mon[32775]: pgmap v9552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:38:58.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:38:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:38:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:58 smithi164 ceph-mon[36970]: pgmap v9550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:38:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:38:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:38:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:58 smithi164 ceph-mon[36970]: pgmap v9551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:38:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:58 smithi164 ceph-mon[36970]: pgmap v9552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:38:58.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:38:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:38:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:38:58.753+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:38:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:38:58.753+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:58.755 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:38:58.753+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:38:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:58 smithi163 ceph-mon[36873]: pgmap v9550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:38:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:38:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:38:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:38:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:58 smithi163 ceph-mon[36873]: pgmap v9551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:38:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:58 smithi163 ceph-mon[36873]: pgmap v9552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:38:58.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:38:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:39:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:00 smithi042 ceph-mon[32775]: pgmap v9553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:39:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:00 smithi164 ceph-mon[36970]: pgmap v9553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:39:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:00 smithi163 ceph-mon[36873]: pgmap v9553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:39:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16971.511678 secs 2024-07-26T11:39:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16989.249844 secs 2024-07-26T11:39:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16971.511678 secs 2024-07-26T11:39:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16989.249844 secs 2024-07-26T11:39:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16971.511678 secs 2024-07-26T11:39:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16989.249844 secs 2024-07-26T11:39:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:02 smithi042 ceph-mon[32775]: pgmap v9554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:39:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:02 smithi164 ceph-mon[36970]: pgmap v9554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:39:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:02 smithi163 ceph-mon[36873]: pgmap v9554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:39:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:04 smithi042 ceph-mon[32775]: pgmap v9555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:39:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:04 smithi164 ceph-mon[36970]: pgmap v9555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:39:05.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:04 smithi163 ceph-mon[36873]: pgmap v9555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:39:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:06 smithi042 ceph-mon[32775]: pgmap v9556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16976.511780 secs 2024-07-26T11:39:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16994.249971 secs 2024-07-26T11:39:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:06 smithi164 ceph-mon[36970]: pgmap v9556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16976.511780 secs 2024-07-26T11:39:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16994.249971 secs 2024-07-26T11:39:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:06 smithi163 ceph-mon[36873]: pgmap v9556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:39:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16976.511780 secs 2024-07-26T11:39:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16994.249971 secs 2024-07-26T11:39:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:08 smithi042 ceph-mon[32775]: pgmap v9557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:39:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:08 smithi164 ceph-mon[36970]: pgmap v9557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:39:09.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:08 smithi163 ceph-mon[36873]: pgmap v9557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:39:11.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:10 smithi042 ceph-mon[32775]: pgmap v9558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:39:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:10 smithi164 ceph-mon[36970]: pgmap v9558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:39:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:10 smithi163 ceph-mon[36873]: pgmap v9558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:39:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16981.511904 secs 2024-07-26T11:39:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16999.250054 secs 2024-07-26T11:39:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16981.511904 secs 2024-07-26T11:39:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16999.250054 secs 2024-07-26T11:39:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16981.511904 secs 2024-07-26T11:39:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16999.250054 secs 2024-07-26T11:39:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:12 smithi042 ceph-mon[32775]: pgmap v9559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:39:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:12 smithi164 ceph-mon[36970]: pgmap v9559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:39:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:12 smithi163 ceph-mon[36873]: pgmap v9559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:39:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:39:13.754+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:39:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:39:13.754+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:39:13.754+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:39:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:14 smithi042 ceph-mon[32775]: pgmap v9560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:39:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:14 smithi164 ceph-mon[36970]: pgmap v9560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:39:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:14 smithi163 ceph-mon[36873]: pgmap v9560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:39:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:16 smithi042 ceph-mon[32775]: pgmap v9561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:39:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16986.512026 secs 2024-07-26T11:39:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17004.250210 secs 2024-07-26T11:39:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:16 smithi164 ceph-mon[36970]: pgmap v9561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:39:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16986.512026 secs 2024-07-26T11:39:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17004.250210 secs 2024-07-26T11:39:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:16 smithi163 ceph-mon[36873]: pgmap v9561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:39:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16986.512026 secs 2024-07-26T11:39:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17004.250210 secs 2024-07-26T11:39:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:18 smithi164 ceph-mon[36970]: pgmap v9562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:18 smithi163 ceph-mon[36873]: pgmap v9562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:18 smithi042 ceph-mon[32775]: pgmap v9562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:20 smithi164 ceph-mon[36970]: pgmap v9563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:20 smithi163 ceph-mon[36873]: pgmap v9563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:20 smithi042 ceph-mon[32775]: pgmap v9563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16991.512149 secs 2024-07-26T11:39:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17009.250311 secs 2024-07-26T11:39:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16991.512149 secs 2024-07-26T11:39:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17009.250311 secs 2024-07-26T11:39:22.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16991.512149 secs 2024-07-26T11:39:22.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17009.250311 secs 2024-07-26T11:39:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:22 smithi164 ceph-mon[36970]: pgmap v9564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:22 smithi163 ceph-mon[36873]: pgmap v9564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:22 smithi042 ceph-mon[32775]: pgmap v9564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:24 smithi164 ceph-mon[36970]: pgmap v9565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:39:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:24 smithi163 ceph-mon[36873]: pgmap v9565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:39:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:24 smithi042 ceph-mon[32775]: pgmap v9565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:39:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:26 smithi164 ceph-mon[36970]: pgmap v9566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 16996.512295 secs 2024-07-26T11:39:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17014.250386 secs 2024-07-26T11:39:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:26 smithi163 ceph-mon[36873]: pgmap v9566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 16996.512295 secs 2024-07-26T11:39:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17014.250386 secs 2024-07-26T11:39:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:26 smithi042 ceph-mon[32775]: pgmap v9566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 16996.512295 secs 2024-07-26T11:39:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17014.250386 secs 2024-07-26T11:39:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:39:28.760+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:39:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:39:28.760+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:39:28.760+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:39:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:28 smithi164 ceph-mon[36970]: pgmap v9567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:29.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:28 smithi163 ceph-mon[36873]: pgmap v9567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:28 smithi042 ceph-mon[32775]: pgmap v9567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:30 smithi164 ceph-mon[36970]: pgmap v9568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:30 smithi163 ceph-mon[36873]: pgmap v9568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:30 smithi042 ceph-mon[32775]: pgmap v9568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17001.512439 secs 2024-07-26T11:39:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17019.250512 secs 2024-07-26T11:39:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17001.512439 secs 2024-07-26T11:39:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17019.250512 secs 2024-07-26T11:39:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17001.512439 secs 2024-07-26T11:39:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17019.250512 secs 2024-07-26T11:39:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:32 smithi164 ceph-mon[36970]: pgmap v9569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:33.232 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:32 smithi042 ceph-mon[32775]: pgmap v9569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:32 smithi163 ceph-mon[36873]: pgmap v9569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:34 smithi164 ceph-mon[36970]: pgmap v9570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:39:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:34 smithi163 ceph-mon[36873]: pgmap v9570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:39:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:34 smithi042 ceph-mon[32775]: pgmap v9570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:39:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:36 smithi164 ceph-mon[36970]: pgmap v9571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17006.512561 secs 2024-07-26T11:39:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17024.250631 secs 2024-07-26T11:39:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:36 smithi163 ceph-mon[36873]: pgmap v9571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17006.512561 secs 2024-07-26T11:39:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17024.250631 secs 2024-07-26T11:39:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:36 smithi042 ceph-mon[32775]: pgmap v9571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17006.512561 secs 2024-07-26T11:39:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17024.250631 secs 2024-07-26T11:39:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:38 smithi164 ceph-mon[36970]: pgmap v9572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:38 smithi163 ceph-mon[36873]: pgmap v9572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:38 smithi042 ceph-mon[32775]: pgmap v9572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:40 smithi164 ceph-mon[36970]: pgmap v9573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:40 smithi163 ceph-mon[36873]: pgmap v9573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:40 smithi042 ceph-mon[32775]: pgmap v9573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17011.512641 secs 2024-07-26T11:39:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17029.250753 secs 2024-07-26T11:39:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17011.512641 secs 2024-07-26T11:39:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17029.250753 secs 2024-07-26T11:39:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17011.512641 secs 2024-07-26T11:39:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17029.250753 secs 2024-07-26T11:39:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:42 smithi164 ceph-mon[36970]: pgmap v9574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:42 smithi163 ceph-mon[36873]: pgmap v9574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:42 smithi042 ceph-mon[32775]: pgmap v9574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:43.762 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:39:43.761+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:39:43.762 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:39:43.761+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:43.762 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:39:43.761+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:39:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:44 smithi164 ceph-mon[36970]: pgmap v9575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:39:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:44 smithi163 ceph-mon[36873]: pgmap v9575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:39:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:44 smithi042 ceph-mon[32775]: pgmap v9575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:39:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:46 smithi164 ceph-mon[36970]: pgmap v9576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17016.512726 secs 2024-07-26T11:39:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17034.250793 secs 2024-07-26T11:39:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:46 smithi163 ceph-mon[36873]: pgmap v9576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17016.512726 secs 2024-07-26T11:39:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17034.250793 secs 2024-07-26T11:39:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:46 smithi042 ceph-mon[32775]: pgmap v9576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17016.512726 secs 2024-07-26T11:39:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17034.250793 secs 2024-07-26T11:39:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:48 smithi164 ceph-mon[36970]: pgmap v9577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:49.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:48 smithi163 ceph-mon[36873]: pgmap v9577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:48 smithi042 ceph-mon[32775]: pgmap v9577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:50 smithi164 ceph-mon[36970]: pgmap v9578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:50 smithi163 ceph-mon[36873]: pgmap v9578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:51.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:50 smithi042 ceph-mon[32775]: pgmap v9578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17021.512884 secs 2024-07-26T11:39:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17039.250924 secs 2024-07-26T11:39:52.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17021.512884 secs 2024-07-26T11:39:52.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17039.250924 secs 2024-07-26T11:39:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17021.512884 secs 2024-07-26T11:39:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17039.250924 secs 2024-07-26T11:39:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:52 smithi163 ceph-mon[36873]: pgmap v9579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:52 smithi042 ceph-mon[32775]: pgmap v9579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:52 smithi164 ceph-mon[36970]: pgmap v9579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:54 smithi163 ceph-mon[36873]: pgmap v9580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:39:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:54 smithi042 ceph-mon[32775]: pgmap v9580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:39:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:54 smithi164 ceph-mon[36970]: pgmap v9580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:39:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:56 smithi163 ceph-mon[36873]: pgmap v9581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17026.513008 secs 2024-07-26T11:39:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17044.251039 secs 2024-07-26T11:39:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:56 smithi042 ceph-mon[32775]: pgmap v9581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17026.513008 secs 2024-07-26T11:39:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17044.251039 secs 2024-07-26T11:39:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:56 smithi164 ceph-mon[36970]: pgmap v9581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:39:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17026.513008 secs 2024-07-26T11:39:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17044.251039 secs 2024-07-26T11:39:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:57 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:39:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:57 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:39:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:57 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:39:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:39:58.766+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:39:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:39:58.766+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:39:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:39:58.766+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:58 smithi163 ceph-mon[36873]: pgmap v9582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:39:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:39:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:39:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:58 smithi042 ceph-mon[32775]: pgmap v9582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:39:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:39:59.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:39:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:58 smithi164 ceph-mon[36970]: pgmap v9582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:39:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:39:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:39:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:40:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:59 smithi163 ceph-mon[36873]: pgmap v9583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:40:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:39:59 smithi163 ceph-mon[36873]: pgmap v9584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:40:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:59 smithi042 ceph-mon[32775]: pgmap v9583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:40:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:39:59 smithi042 ceph-mon[32775]: pgmap v9584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:40:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:59 smithi164 ceph-mon[36970]: pgmap v9583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:40:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:39:59 smithi164 ceph-mon[36970]: pgmap v9584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:40:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:01 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:40:01.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:01 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:40:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:01 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:40:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:02 smithi163 ceph-mon[36873]: pgmap v9585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:40:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17031.513150 secs 2024-07-26T11:40:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17049.251171 secs 2024-07-26T11:40:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:02 smithi042 ceph-mon[32775]: pgmap v9585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:40:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17031.513150 secs 2024-07-26T11:40:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17049.251171 secs 2024-07-26T11:40:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:02 smithi164 ceph-mon[36970]: pgmap v9585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:40:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17031.513150 secs 2024-07-26T11:40:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17049.251171 secs 2024-07-26T11:40:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:03 smithi042 ceph-mon[32775]: pgmap v9586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:40:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:03 smithi164 ceph-mon[36970]: pgmap v9586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:40:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:03 smithi163 ceph-mon[36873]: pgmap v9586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:40:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:05 smithi164 ceph-mon[36970]: pgmap v9587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:40:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:05 smithi163 ceph-mon[36873]: pgmap v9587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:40:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:05 smithi042 ceph-mon[32775]: pgmap v9587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:40:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17036.513298 secs 2024-07-26T11:40:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17036.513298 secs 2024-07-26T11:40:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17036.513298 secs 2024-07-26T11:40:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:07 smithi164 ceph-mon[36970]: pgmap v9588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:40:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17054.251269 secs 2024-07-26T11:40:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:07 smithi163 ceph-mon[36873]: pgmap v9588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:40:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17054.251269 secs 2024-07-26T11:40:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:07 smithi042 ceph-mon[32775]: pgmap v9588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:40:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17054.251269 secs 2024-07-26T11:40:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:09 smithi164 ceph-mon[36970]: pgmap v9589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:40:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:09 smithi163 ceph-mon[36873]: pgmap v9589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:40:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:09 smithi042 ceph-mon[32775]: pgmap v9589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:40:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:11 smithi164 ceph-mon[36970]: pgmap v9590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17041.513424 secs 2024-07-26T11:40:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:11 smithi163 ceph-mon[36873]: pgmap v9590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:11.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17041.513424 secs 2024-07-26T11:40:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:11 smithi042 ceph-mon[32775]: pgmap v9590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17041.513424 secs 2024-07-26T11:40:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17059.251392 secs 2024-07-26T11:40:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17059.251392 secs 2024-07-26T11:40:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17059.251392 secs 2024-07-26T11:40:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:13 smithi164 ceph-mon[36970]: pgmap v9591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:13 smithi163 ceph-mon[36873]: pgmap v9591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:13.767+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:40:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:13.767+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:13.767+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:40:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:13 smithi042 ceph-mon[32775]: pgmap v9591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:15 smithi164 ceph-mon[36970]: pgmap v9592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:15 smithi163 ceph-mon[36873]: pgmap v9592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:15.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:15 smithi042 ceph-mon[32775]: pgmap v9592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17046.513514 secs 2024-07-26T11:40:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17046.513514 secs 2024-07-26T11:40:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17046.513514 secs 2024-07-26T11:40:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:17 smithi164 ceph-mon[36970]: pgmap v9593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17064.251530 secs 2024-07-26T11:40:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:17 smithi163 ceph-mon[36873]: pgmap v9593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17064.251530 secs 2024-07-26T11:40:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:17 smithi042 ceph-mon[32775]: pgmap v9593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17064.251530 secs 2024-07-26T11:40:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:19 smithi164 ceph-mon[36970]: pgmap v9594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:40:19.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:19 smithi163 ceph-mon[36873]: pgmap v9594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:40:19.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:19 smithi042 ceph-mon[32775]: pgmap v9594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:40:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:21 smithi164 ceph-mon[36970]: pgmap v9595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17051.513629 secs 2024-07-26T11:40:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:21 smithi163 ceph-mon[36873]: pgmap v9595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:21.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17051.513629 secs 2024-07-26T11:40:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:21 smithi042 ceph-mon[32775]: pgmap v9595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17051.513629 secs 2024-07-26T11:40:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17069.251641 secs 2024-07-26T11:40:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17069.251641 secs 2024-07-26T11:40:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17069.251641 secs 2024-07-26T11:40:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:23 smithi164 ceph-mon[36970]: pgmap v9596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:23.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:23 smithi163 ceph-mon[36873]: pgmap v9596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:23 smithi042 ceph-mon[32775]: pgmap v9596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:25 smithi164 ceph-mon[36970]: pgmap v9597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:25.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:25 smithi163 ceph-mon[36873]: pgmap v9597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:25 smithi042 ceph-mon[32775]: pgmap v9597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:26.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17056.513786 secs 2024-07-26T11:40:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17056.513786 secs 2024-07-26T11:40:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17056.513786 secs 2024-07-26T11:40:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:27 smithi163 ceph-mon[36873]: pgmap v9598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17074.251761 secs 2024-07-26T11:40:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:27 smithi042 ceph-mon[32775]: pgmap v9598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17074.251761 secs 2024-07-26T11:40:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:27 smithi164 ceph-mon[36970]: pgmap v9598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17074.251761 secs 2024-07-26T11:40:28.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:28.771+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:40:28.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:28.771+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:40:28.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:28.771+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:29 smithi164 ceph-mon[36970]: pgmap v9599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:40:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:29 smithi163 ceph-mon[36873]: pgmap v9599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:40:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:29 smithi042 ceph-mon[32775]: pgmap v9599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:40:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:31 smithi163 ceph-mon[36873]: pgmap v9600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17061.513908 secs 2024-07-26T11:40:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:31 smithi042 ceph-mon[32775]: pgmap v9600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17061.513908 secs 2024-07-26T11:40:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:31 smithi164 ceph-mon[36970]: pgmap v9600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17061.513908 secs 2024-07-26T11:40:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17079.251900 secs 2024-07-26T11:40:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17079.251900 secs 2024-07-26T11:40:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17079.251900 secs 2024-07-26T11:40:33.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:33 smithi163 ceph-mon[36873]: pgmap v9601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:33 smithi042 ceph-mon[32775]: pgmap v9601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:33 smithi164 ceph-mon[36970]: pgmap v9601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:35.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:35 smithi163 ceph-mon[36873]: pgmap v9602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:35.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:35 smithi042 ceph-mon[32775]: pgmap v9602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:35 smithi164 ceph-mon[36970]: pgmap v9602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17066.514031 secs 2024-07-26T11:40:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17066.514031 secs 2024-07-26T11:40:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17066.514031 secs 2024-07-26T11:40:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:37 smithi163 ceph-mon[36873]: pgmap v9603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17084.252020 secs 2024-07-26T11:40:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:37 smithi042 ceph-mon[32775]: pgmap v9603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17084.252020 secs 2024-07-26T11:40:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:37 smithi164 ceph-mon[36970]: pgmap v9603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17084.252020 secs 2024-07-26T11:40:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:39 smithi042 ceph-mon[32775]: pgmap v9604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:40:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:39 smithi164 ceph-mon[36970]: pgmap v9604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:40:40.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:39 smithi163 ceph-mon[36873]: pgmap v9604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:40:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:41 smithi042 ceph-mon[32775]: pgmap v9605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17071.514157 secs 2024-07-26T11:40:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:41 smithi164 ceph-mon[36970]: pgmap v9605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17071.514157 secs 2024-07-26T11:40:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:41 smithi163 ceph-mon[36873]: pgmap v9605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17071.514157 secs 2024-07-26T11:40:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17089.252141 secs 2024-07-26T11:40:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17089.252141 secs 2024-07-26T11:40:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17089.252141 secs 2024-07-26T11:40:43.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:43.772+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:40:43.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:43.772+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:43.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:43.772+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:40:43.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:43 smithi042 ceph-mon[32775]: pgmap v9606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:43 smithi164 ceph-mon[36970]: pgmap v9606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:43 smithi163 ceph-mon[36873]: pgmap v9606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:45 smithi042 ceph-mon[32775]: pgmap v9607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:45 smithi164 ceph-mon[36970]: pgmap v9607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:45 smithi163 ceph-mon[36873]: pgmap v9607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17076.514282 secs 2024-07-26T11:40:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17076.514282 secs 2024-07-26T11:40:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17076.514282 secs 2024-07-26T11:40:47.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:47 smithi164 ceph-mon[36970]: pgmap v9608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:47.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17094.252281 secs 2024-07-26T11:40:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:47 smithi042 ceph-mon[32775]: pgmap v9608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17094.252281 secs 2024-07-26T11:40:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:47 smithi163 ceph-mon[36873]: pgmap v9608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17094.252281 secs 2024-07-26T11:40:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:49 smithi042 ceph-mon[32775]: pgmap v9609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:40:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:49 smithi164 ceph-mon[36970]: pgmap v9609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:40:50.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:49 smithi163 ceph-mon[36873]: pgmap v9609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:40:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:51 smithi042 ceph-mon[32775]: pgmap v9610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:51.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17081.514378 secs 2024-07-26T11:40:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:51 smithi164 ceph-mon[36970]: pgmap v9610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:51.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17081.514378 secs 2024-07-26T11:40:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:51 smithi163 ceph-mon[36873]: pgmap v9610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17081.514378 secs 2024-07-26T11:40:52.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17099.252381 secs 2024-07-26T11:40:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17099.252381 secs 2024-07-26T11:40:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17099.252381 secs 2024-07-26T11:40:53.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:53 smithi042 ceph-mon[32775]: pgmap v9611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:53 smithi164 ceph-mon[36970]: pgmap v9611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:53 smithi163 ceph-mon[36873]: pgmap v9611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:55.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:55 smithi042 ceph-mon[32775]: pgmap v9612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:55 smithi164 ceph-mon[36970]: pgmap v9612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:55 smithi163 ceph-mon[36873]: pgmap v9612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:40:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17086.514455 secs 2024-07-26T11:40:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17086.514455 secs 2024-07-26T11:40:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17086.514455 secs 2024-07-26T11:40:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:57.887+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T11:40:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:57.887+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:40:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:57.887+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5000 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7ff678009a60 tx=0x7ff678055040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:40:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:57.887+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5000 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7ff678009a60 tx=0x7ff678055040 comp rx=0 tx=0).stop 2024-07-26T11:40:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:57.887+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5000 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T11:40:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:57.887+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b5000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:40:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:57.887+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b5000 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:40:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:57.887+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:40:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:57.887+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:40:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:57.888+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ff678003990 tx=0x7ff678060000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:40:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:57 smithi042 ceph-mon[32775]: pgmap v9613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17104.252501 secs 2024-07-26T11:40:57.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:57 smithi163 ceph-mon[36873]: pgmap v9613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:57.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17104.252501 secs 2024-07-26T11:40:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:57 smithi164 ceph-mon[36970]: pgmap v9613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:40:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17104.252501 secs 2024-07-26T11:40:58.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:58.773+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:40:58.774 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:58.773+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:40:58.775 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:40:58.773+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:58 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:40:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:58 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:40:59.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:58 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:40:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: pgmap v9614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:40:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:40:59.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:40:59.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:40:59.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:40:59.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:40:59.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:40:59.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:40:59.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:40:59.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: pgmap v9615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:40:59.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: pgmap v9616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:40:59.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:40:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:40:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: pgmap v9614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:40:59.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:59.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:40:59.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:40:59.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:40:59.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:40:59.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:40:59.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:40:59.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:40:59.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:40:59.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: pgmap v9615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:40:59.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: pgmap v9616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:40:59.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:40:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:41:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: pgmap v9614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:41:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:41:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:41:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:41:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:41:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:41:00.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:41:00.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:41:00.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:41:00.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:41:00.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:41:00.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:41:00.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:41:00.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:41:00.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:41:00.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:41:00.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:41:00.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:41:00.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:41:00.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:41:00.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:41:00.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: pgmap v9615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:41:00.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: pgmap v9616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:41:00.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:40:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:41:00.560 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:41:00.558+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006480 con 0x7ff680102000 2024-07-26T11:41:00.560 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:41:00.558+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:41:00.561 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:41:00.558+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6700e9050 con 0x7ff680102000 2024-07-26T11:41:00.561 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:41:00.558+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:41:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:01 smithi163 ceph-mon[36873]: pgmap v9617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:41:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17091.514524 secs 2024-07-26T11:41:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17109.252625 secs 2024-07-26T11:41:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:01 smithi042 ceph-mon[32775]: pgmap v9617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:41:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17091.514524 secs 2024-07-26T11:41:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17109.252625 secs 2024-07-26T11:41:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:01 smithi164 ceph-mon[36970]: pgmap v9617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:41:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17091.514524 secs 2024-07-26T11:41:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17109.252625 secs 2024-07-26T11:41:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:03 smithi163 ceph-mon[36873]: pgmap v9618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:41:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:03 smithi042 ceph-mon[32775]: pgmap v9618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:41:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:03 smithi164 ceph-mon[36970]: pgmap v9618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:41:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:05 smithi163 ceph-mon[36873]: pgmap v9619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:41:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:05 smithi042 ceph-mon[32775]: pgmap v9619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:41:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:05 smithi164 ceph-mon[36970]: pgmap v9619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:41:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17096.514656 secs 2024-07-26T11:41:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17114.252749 secs 2024-07-26T11:41:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17096.514656 secs 2024-07-26T11:41:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17114.252749 secs 2024-07-26T11:41:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17096.514656 secs 2024-07-26T11:41:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17114.252749 secs 2024-07-26T11:41:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:07 smithi163 ceph-mon[36873]: pgmap v9620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:41:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:07 smithi042 ceph-mon[32775]: pgmap v9620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:41:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:07 smithi164 ceph-mon[36970]: pgmap v9620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:41:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:09 smithi163 ceph-mon[36873]: pgmap v9621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:41:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:09 smithi042 ceph-mon[32775]: pgmap v9621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:41:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:09 smithi164 ceph-mon[36970]: pgmap v9621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:41:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:11 smithi163 ceph-mon[36873]: pgmap v9622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17101.514779 secs 2024-07-26T11:41:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17119.252892 secs 2024-07-26T11:41:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:11 smithi042 ceph-mon[32775]: pgmap v9622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17101.514779 secs 2024-07-26T11:41:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17119.252892 secs 2024-07-26T11:41:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:11 smithi164 ceph-mon[36970]: pgmap v9622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:12.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17101.514779 secs 2024-07-26T11:41:12.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17119.252892 secs 2024-07-26T11:41:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:41:13.774+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:41:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:41:13.774+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:41:13.774+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:41:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:14 smithi163 ceph-mon[36873]: pgmap v9623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:14 smithi042 ceph-mon[32775]: pgmap v9623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:14 smithi164 ceph-mon[36970]: pgmap v9623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:16 smithi163 ceph-mon[36873]: pgmap v9624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:16 smithi042 ceph-mon[32775]: pgmap v9624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:16 smithi164 ceph-mon[36970]: pgmap v9624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:17.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17106.514910 secs 2024-07-26T11:41:17.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17124.252949 secs 2024-07-26T11:41:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17106.514910 secs 2024-07-26T11:41:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17124.252949 secs 2024-07-26T11:41:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17106.514910 secs 2024-07-26T11:41:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17124.252949 secs 2024-07-26T11:41:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:18 smithi042 ceph-mon[32775]: pgmap v9625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:18 smithi164 ceph-mon[36970]: pgmap v9625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:18 smithi163 ceph-mon[36873]: pgmap v9625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:19 smithi042 ceph-mon[32775]: pgmap v9626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:41:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:19 smithi164 ceph-mon[36970]: pgmap v9626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:41:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:19 smithi163 ceph-mon[36873]: pgmap v9626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:41:22.099 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:21 smithi042 ceph-mon[32775]: pgmap v9627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:22.099 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17111.515050 secs 2024-07-26T11:41:22.099 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17129.253086 secs 2024-07-26T11:41:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:21 smithi163 ceph-mon[36873]: pgmap v9627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17111.515050 secs 2024-07-26T11:41:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17129.253086 secs 2024-07-26T11:41:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:21 smithi164 ceph-mon[36970]: pgmap v9627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17111.515050 secs 2024-07-26T11:41:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17129.253086 secs 2024-07-26T11:41:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:23 smithi163 ceph-mon[36873]: pgmap v9628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:23 smithi042 ceph-mon[32775]: pgmap v9628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:23 smithi164 ceph-mon[36970]: pgmap v9628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:25 smithi163 ceph-mon[36873]: pgmap v9629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:25 smithi042 ceph-mon[32775]: pgmap v9629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:25 smithi164 ceph-mon[36970]: pgmap v9629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17116.515176 secs 2024-07-26T11:41:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17134.253204 secs 2024-07-26T11:41:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17116.515176 secs 2024-07-26T11:41:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17134.253204 secs 2024-07-26T11:41:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17116.515176 secs 2024-07-26T11:41:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17134.253204 secs 2024-07-26T11:41:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:28 smithi163 ceph-mon[36873]: pgmap v9630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:28 smithi042 ceph-mon[32775]: pgmap v9630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:28 smithi164 ceph-mon[36970]: pgmap v9630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:28.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:41:28.780+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:41:28.781 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:41:28.780+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:28.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:41:28.780+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:41:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:30 smithi163 ceph-mon[36873]: pgmap v9631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:41:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:30 smithi042 ceph-mon[32775]: pgmap v9631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:41:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:30 smithi164 ceph-mon[36970]: pgmap v9631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:41:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:32 smithi042 ceph-mon[32775]: pgmap v9632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17121.515313 secs 2024-07-26T11:41:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17139.253334 secs 2024-07-26T11:41:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:32 smithi164 ceph-mon[36970]: pgmap v9632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17121.515313 secs 2024-07-26T11:41:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17139.253334 secs 2024-07-26T11:41:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:32 smithi163 ceph-mon[36873]: pgmap v9632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17121.515313 secs 2024-07-26T11:41:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17139.253334 secs 2024-07-26T11:41:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:34 smithi042 ceph-mon[32775]: pgmap v9633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:34 smithi164 ceph-mon[36970]: pgmap v9633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:34 smithi163 ceph-mon[36873]: pgmap v9633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:36 smithi042 ceph-mon[32775]: pgmap v9634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:36 smithi164 ceph-mon[36970]: pgmap v9634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:36 smithi163 ceph-mon[36873]: pgmap v9634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17126.515434 secs 2024-07-26T11:41:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17144.253455 secs 2024-07-26T11:41:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17126.515434 secs 2024-07-26T11:41:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17144.253455 secs 2024-07-26T11:41:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17126.515434 secs 2024-07-26T11:41:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17144.253455 secs 2024-07-26T11:41:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:38 smithi042 ceph-mon[32775]: pgmap v9635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:38 smithi164 ceph-mon[36970]: pgmap v9635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:38 smithi163 ceph-mon[36873]: pgmap v9635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:39 smithi042 ceph-mon[32775]: pgmap v9636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:41:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:39 smithi164 ceph-mon[36970]: pgmap v9636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:41:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:39 smithi163 ceph-mon[36873]: pgmap v9636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:41:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:41 smithi163 ceph-mon[36873]: pgmap v9637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17131.515554 secs 2024-07-26T11:41:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17149.253524 secs 2024-07-26T11:41:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:41 smithi042 ceph-mon[32775]: pgmap v9637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17131.515554 secs 2024-07-26T11:41:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17149.253524 secs 2024-07-26T11:41:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:41 smithi164 ceph-mon[36970]: pgmap v9637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17131.515554 secs 2024-07-26T11:41:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17149.253524 secs 2024-07-26T11:41:43.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:41:43.781+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:41:43.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:41:43.781+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:43.782 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:41:43.781+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:41:44.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:43 smithi163 ceph-mon[36873]: pgmap v9638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:43 smithi042 ceph-mon[32775]: pgmap v9638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:43 smithi164 ceph-mon[36970]: pgmap v9638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:45 smithi163 ceph-mon[36873]: pgmap v9639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:45 smithi042 ceph-mon[32775]: pgmap v9639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:45 smithi164 ceph-mon[36970]: pgmap v9639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17136.515686 secs 2024-07-26T11:41:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17154.253646 secs 2024-07-26T11:41:47.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17136.515686 secs 2024-07-26T11:41:47.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17154.253646 secs 2024-07-26T11:41:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17136.515686 secs 2024-07-26T11:41:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17154.253646 secs 2024-07-26T11:41:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:48 smithi042 ceph-mon[32775]: pgmap v9640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:48 smithi164 ceph-mon[36970]: pgmap v9640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:48 smithi163 ceph-mon[36873]: pgmap v9640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:50 smithi042 ceph-mon[32775]: pgmap v9641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:41:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:50 smithi164 ceph-mon[36970]: pgmap v9641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:41:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:50 smithi163 ceph-mon[36873]: pgmap v9641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:41:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:52 smithi042 ceph-mon[32775]: pgmap v9642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17141.515826 secs 2024-07-26T11:41:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17159.253769 secs 2024-07-26T11:41:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:52 smithi164 ceph-mon[36970]: pgmap v9642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17141.515826 secs 2024-07-26T11:41:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17159.253769 secs 2024-07-26T11:41:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:52 smithi163 ceph-mon[36873]: pgmap v9642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17141.515826 secs 2024-07-26T11:41:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17159.253769 secs 2024-07-26T11:41:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:54 smithi042 ceph-mon[32775]: pgmap v9643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:54 smithi164 ceph-mon[36970]: pgmap v9643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:54 smithi163 ceph-mon[36873]: pgmap v9643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:55 smithi042 ceph-mon[32775]: pgmap v9644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:55 smithi164 ceph-mon[36970]: pgmap v9644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:55 smithi163 ceph-mon[36873]: pgmap v9644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:41:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17146.515985 secs 2024-07-26T11:41:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17146.515985 secs 2024-07-26T11:41:56.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17146.515985 secs 2024-07-26T11:41:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17164.253915 secs 2024-07-26T11:41:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:41:57 smithi042 ceph-mon[32775]: pgmap v9645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:57.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17164.253915 secs 2024-07-26T11:41:57.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:41:57 smithi163 ceph-mon[36873]: pgmap v9645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17164.253915 secs 2024-07-26T11:41:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:41:57 smithi164 ceph-mon[36970]: pgmap v9645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:41:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:41:58.787+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:41:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:41:58.787+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:41:58.787+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:42:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:00 smithi163 ceph-mon[36873]: pgmap v9646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:42:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:42:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:42:00.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:42:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:00 smithi042 ceph-mon[32775]: pgmap v9646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:42:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:42:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:42:00.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:42:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:00 smithi164 ceph-mon[36970]: pgmap v9646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:42:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:42:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:42:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:42:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:01 smithi163 ceph-mon[36873]: pgmap v9647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:42:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:01 smithi163 ceph-mon[36873]: pgmap v9648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:42:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:01 smithi042 ceph-mon[32775]: pgmap v9647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:42:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:01 smithi042 ceph-mon[32775]: pgmap v9648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:42:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:01 smithi164 ceph-mon[36970]: pgmap v9647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:42:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:01 smithi164 ceph-mon[36970]: pgmap v9648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:42:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17151.516091 secs 2024-07-26T11:42:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17169.254038 secs 2024-07-26T11:42:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17151.516091 secs 2024-07-26T11:42:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17169.254038 secs 2024-07-26T11:42:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17151.516091 secs 2024-07-26T11:42:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17169.254038 secs 2024-07-26T11:42:03.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:03 smithi042 ceph-mon[32775]: pgmap v9649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:42:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:03 smithi164 ceph-mon[36970]: pgmap v9649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:42:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:03 smithi163 ceph-mon[36873]: pgmap v9649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:42:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:05 smithi042 ceph-mon[32775]: pgmap v9650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:42:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:05 smithi164 ceph-mon[36970]: pgmap v9650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:42:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:05 smithi163 ceph-mon[36873]: pgmap v9650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:42:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:07 smithi042 ceph-mon[32775]: pgmap v9651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:42:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17156.516230 secs 2024-07-26T11:42:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17174.254159 secs 2024-07-26T11:42:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:07 smithi164 ceph-mon[36970]: pgmap v9651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:42:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17156.516230 secs 2024-07-26T11:42:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17174.254159 secs 2024-07-26T11:42:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:07 smithi163 ceph-mon[36873]: pgmap v9651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:42:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17156.516230 secs 2024-07-26T11:42:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17174.254159 secs 2024-07-26T11:42:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:08 smithi042 ceph-mon[32775]: pgmap v9652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:42:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:08 smithi164 ceph-mon[36970]: pgmap v9652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:42:08.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:08 smithi163 ceph-mon[36873]: pgmap v9652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:42:10.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:10 smithi042 ceph-mon[32775]: pgmap v9653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:42:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:10 smithi164 ceph-mon[36970]: pgmap v9653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:42:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:10 smithi163 ceph-mon[36873]: pgmap v9653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:42:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17161.516331 secs 2024-07-26T11:42:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17161.516331 secs 2024-07-26T11:42:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17161.516331 secs 2024-07-26T11:42:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17179.254287 secs 2024-07-26T11:42:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:12 smithi042 ceph-mon[32775]: pgmap v9654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17179.254287 secs 2024-07-26T11:42:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:12 smithi164 ceph-mon[36970]: pgmap v9654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17179.254287 secs 2024-07-26T11:42:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:12 smithi163 ceph-mon[36873]: pgmap v9654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:42:13.788+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:42:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:42:13.788+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:42:13.788+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:42:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:14 smithi042 ceph-mon[32775]: pgmap v9655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:14 smithi164 ceph-mon[36970]: pgmap v9655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:15.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:14 smithi163 ceph-mon[36873]: pgmap v9655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:16 smithi042 ceph-mon[32775]: pgmap v9656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:16.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17166.516457 secs 2024-07-26T11:42:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:16 smithi164 ceph-mon[36970]: pgmap v9656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17166.516457 secs 2024-07-26T11:42:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:16 smithi163 ceph-mon[36873]: pgmap v9656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17166.516457 secs 2024-07-26T11:42:17.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17184.254405 secs 2024-07-26T11:42:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17184.254405 secs 2024-07-26T11:42:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17184.254405 secs 2024-07-26T11:42:18.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:18 smithi042 ceph-mon[32775]: pgmap v9657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:18 smithi164 ceph-mon[36970]: pgmap v9657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:19.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:18 smithi163 ceph-mon[36873]: pgmap v9657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:20 smithi042 ceph-mon[32775]: pgmap v9658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:20 smithi164 ceph-mon[36970]: pgmap v9658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:20 smithi163 ceph-mon[36873]: pgmap v9658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17171.516575 secs 2024-07-26T11:42:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17171.516575 secs 2024-07-26T11:42:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17171.516575 secs 2024-07-26T11:42:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17189.254535 secs 2024-07-26T11:42:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:22 smithi042 ceph-mon[32775]: pgmap v9659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17189.254535 secs 2024-07-26T11:42:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:22 smithi164 ceph-mon[36970]: pgmap v9659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17189.254535 secs 2024-07-26T11:42:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:22 smithi163 ceph-mon[36873]: pgmap v9659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:24 smithi042 ceph-mon[32775]: pgmap v9660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:24 smithi164 ceph-mon[36970]: pgmap v9660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:24 smithi163 ceph-mon[36873]: pgmap v9660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:26 smithi042 ceph-mon[32775]: pgmap v9661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:26.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17176.516699 secs 2024-07-26T11:42:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:26 smithi164 ceph-mon[36970]: pgmap v9661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17176.516699 secs 2024-07-26T11:42:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:26 smithi163 ceph-mon[36873]: pgmap v9661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17176.516699 secs 2024-07-26T11:42:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17194.254678 secs 2024-07-26T11:42:27.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17194.254678 secs 2024-07-26T11:42:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17194.254678 secs 2024-07-26T11:42:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:42:28.793+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:42:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:42:28.793+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:42:28.793+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:42:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:28 smithi042 ceph-mon[32775]: pgmap v9662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:28 smithi164 ceph-mon[36970]: pgmap v9662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:29.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:28 smithi163 ceph-mon[36873]: pgmap v9662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:30 smithi042 ceph-mon[32775]: pgmap v9663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:30 smithi164 ceph-mon[36970]: pgmap v9663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:31.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:30 smithi163 ceph-mon[36873]: pgmap v9663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17181.516818 secs 2024-07-26T11:42:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17181.516818 secs 2024-07-26T11:42:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17181.516818 secs 2024-07-26T11:42:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17199.254819 secs 2024-07-26T11:42:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:32 smithi042 ceph-mon[32775]: pgmap v9664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17199.254819 secs 2024-07-26T11:42:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:32 smithi164 ceph-mon[36970]: pgmap v9664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17199.254819 secs 2024-07-26T11:42:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:32 smithi163 ceph-mon[36873]: pgmap v9664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:34 smithi042 ceph-mon[32775]: pgmap v9665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:34 smithi164 ceph-mon[36970]: pgmap v9665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:35.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:34 smithi163 ceph-mon[36873]: pgmap v9665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:36 smithi042 ceph-mon[32775]: pgmap v9666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:36.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17186.516946 secs 2024-07-26T11:42:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:36 smithi164 ceph-mon[36970]: pgmap v9666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17186.516946 secs 2024-07-26T11:42:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:36 smithi163 ceph-mon[36873]: pgmap v9666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17186.516946 secs 2024-07-26T11:42:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17204.254920 secs 2024-07-26T11:42:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17204.254920 secs 2024-07-26T11:42:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17204.254920 secs 2024-07-26T11:42:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:38 smithi042 ceph-mon[32775]: pgmap v9667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:38 smithi164 ceph-mon[36970]: pgmap v9667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:38 smithi163 ceph-mon[36873]: pgmap v9667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:40 smithi042 ceph-mon[32775]: pgmap v9668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:40 smithi164 ceph-mon[36970]: pgmap v9668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:40 smithi163 ceph-mon[36873]: pgmap v9668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17191.517078 secs 2024-07-26T11:42:41.791 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17191.517078 secs 2024-07-26T11:42:41.791 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17191.517078 secs 2024-07-26T11:42:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17209.255048 secs 2024-07-26T11:42:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:42 smithi042 ceph-mon[32775]: pgmap v9669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17209.255048 secs 2024-07-26T11:42:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:42 smithi164 ceph-mon[36970]: pgmap v9669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17209.255048 secs 2024-07-26T11:42:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:42 smithi163 ceph-mon[36873]: pgmap v9669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:42:43.794+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:42:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:42:43.794+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:43.796 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:42:43.794+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:42:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:44 smithi042 ceph-mon[32775]: pgmap v9670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:44 smithi164 ceph-mon[36970]: pgmap v9670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:44 smithi163 ceph-mon[36873]: pgmap v9670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:46 smithi042 ceph-mon[32775]: pgmap v9671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17196.517239 secs 2024-07-26T11:42:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:46 smithi164 ceph-mon[36970]: pgmap v9671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17196.517239 secs 2024-07-26T11:42:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:46 smithi163 ceph-mon[36873]: pgmap v9671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17196.517239 secs 2024-07-26T11:42:47.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17214.255179 secs 2024-07-26T11:42:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17214.255179 secs 2024-07-26T11:42:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17214.255179 secs 2024-07-26T11:42:48.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:48 smithi042 ceph-mon[32775]: pgmap v9672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:48.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:48 smithi164 ceph-mon[36970]: pgmap v9672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:48 smithi163 ceph-mon[36873]: pgmap v9672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:50 smithi042 ceph-mon[32775]: pgmap v9673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:50 smithi164 ceph-mon[36970]: pgmap v9673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:50 smithi163 ceph-mon[36873]: pgmap v9673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17201.517322 secs 2024-07-26T11:42:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17201.517322 secs 2024-07-26T11:42:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17201.517322 secs 2024-07-26T11:42:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17219.255311 secs 2024-07-26T11:42:52.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:52 smithi042 ceph-mon[32775]: pgmap v9674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17219.255311 secs 2024-07-26T11:42:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:52 smithi164 ceph-mon[36970]: pgmap v9674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17219.255311 secs 2024-07-26T11:42:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:52 smithi163 ceph-mon[36873]: pgmap v9674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:54.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:54 smithi042 ceph-mon[32775]: pgmap v9675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:54 smithi164 ceph-mon[36970]: pgmap v9675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:54 smithi163 ceph-mon[36873]: pgmap v9675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:42:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:56 smithi042 ceph-mon[32775]: pgmap v9676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:56.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17206.517440 secs 2024-07-26T11:42:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:56 smithi164 ceph-mon[36970]: pgmap v9676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17206.517440 secs 2024-07-26T11:42:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:56 smithi163 ceph-mon[36873]: pgmap v9676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:42:57.111 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17206.517440 secs 2024-07-26T11:42:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17224.255433 secs 2024-07-26T11:42:57.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17224.255433 secs 2024-07-26T11:42:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17224.255433 secs 2024-07-26T11:42:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:42:58.798+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:42:58.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:42:58.798+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:42:58.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:42:58.798+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:58 smithi042 ceph-mon[32775]: pgmap v9677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:58.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:58 smithi164 ceph-mon[36970]: pgmap v9677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:59.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:58 smithi163 ceph-mon[36873]: pgmap v9677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:42:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:42:59 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:42:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:42:59 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:43:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:42:59 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:43:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:00 smithi042 ceph-mon[32775]: pgmap v9678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:00.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:43:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:43:00.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:43:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:00 smithi164 ceph-mon[36970]: pgmap v9678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:43:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:43:00.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:43:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:00 smithi163 ceph-mon[36873]: pgmap v9678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:43:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:43:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:43:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:01 smithi042 ceph-mon[32775]: pgmap v9679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:43:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:01 smithi042 ceph-mon[32775]: pgmap v9680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:43:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17211.517590 secs 2024-07-26T11:43:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:01 smithi164 ceph-mon[36970]: pgmap v9679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:43:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:01 smithi164 ceph-mon[36970]: pgmap v9680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:43:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17211.517590 secs 2024-07-26T11:43:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:01 smithi163 ceph-mon[36873]: pgmap v9679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:43:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:01 smithi163 ceph-mon[36873]: pgmap v9680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:43:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17211.517590 secs 2024-07-26T11:43:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17229.255552 secs 2024-07-26T11:43:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17229.255552 secs 2024-07-26T11:43:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17229.255552 secs 2024-07-26T11:43:03.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:03 smithi042 ceph-mon[32775]: pgmap v9681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:43:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:03 smithi164 ceph-mon[36970]: pgmap v9681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:43:04.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:03 smithi163 ceph-mon[36873]: pgmap v9681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:43:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:05 smithi042 ceph-mon[32775]: pgmap v9682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:43:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:05 smithi164 ceph-mon[36970]: pgmap v9682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:43:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:05 smithi163 ceph-mon[36873]: pgmap v9682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:43:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17216.517689 secs 2024-07-26T11:43:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17216.517689 secs 2024-07-26T11:43:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17216.517689 secs 2024-07-26T11:43:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:07 smithi042 ceph-mon[32775]: pgmap v9683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:43:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17234.255671 secs 2024-07-26T11:43:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:07 smithi164 ceph-mon[36970]: pgmap v9683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:43:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17234.255671 secs 2024-07-26T11:43:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:07 smithi163 ceph-mon[36873]: pgmap v9683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:43:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17234.255671 secs 2024-07-26T11:43:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:09 smithi042 ceph-mon[32775]: pgmap v9684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:43:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:09 smithi164 ceph-mon[36970]: pgmap v9684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:43:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:09 smithi163 ceph-mon[36873]: pgmap v9684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:43:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:11 smithi042 ceph-mon[32775]: pgmap v9685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:43:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17221.517809 secs 2024-07-26T11:43:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:11 smithi164 ceph-mon[36970]: pgmap v9685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:43:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17221.517809 secs 2024-07-26T11:43:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:11 smithi163 ceph-mon[36873]: pgmap v9685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:43:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17221.517809 secs 2024-07-26T11:43:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17239.255796 secs 2024-07-26T11:43:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17239.255796 secs 2024-07-26T11:43:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17239.255796 secs 2024-07-26T11:43:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:43:13.799+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:43:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:43:13.799+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:43:13.799+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:43:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:13 smithi042 ceph-mon[32775]: pgmap v9686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:13 smithi164 ceph-mon[36970]: pgmap v9686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:13 smithi163 ceph-mon[36873]: pgmap v9686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:15 smithi042 ceph-mon[32775]: pgmap v9687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:43:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:15 smithi164 ceph-mon[36970]: pgmap v9687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:43:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:15 smithi163 ceph-mon[36873]: pgmap v9687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:43:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17226.517939 secs 2024-07-26T11:43:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17226.517939 secs 2024-07-26T11:43:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17226.517939 secs 2024-07-26T11:43:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:17 smithi164 ceph-mon[36970]: pgmap v9688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17244.255914 secs 2024-07-26T11:43:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:17 smithi163 ceph-mon[36873]: pgmap v9688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17244.255914 secs 2024-07-26T11:43:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:17 smithi042 ceph-mon[32775]: pgmap v9688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:18.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17244.255914 secs 2024-07-26T11:43:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:19 smithi164 ceph-mon[36970]: pgmap v9689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:20.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:19 smithi163 ceph-mon[36873]: pgmap v9689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:19 smithi042 ceph-mon[32775]: pgmap v9689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:21 smithi164 ceph-mon[36970]: pgmap v9690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17231.518053 secs 2024-07-26T11:43:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:21 smithi163 ceph-mon[36873]: pgmap v9690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17231.518053 secs 2024-07-26T11:43:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:21 smithi042 ceph-mon[32775]: pgmap v9690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17231.518053 secs 2024-07-26T11:43:22.661 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17249.256027 secs 2024-07-26T11:43:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17249.256027 secs 2024-07-26T11:43:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17249.256027 secs 2024-07-26T11:43:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:23 smithi164 ceph-mon[36970]: pgmap v9691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:24.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:23 smithi163 ceph-mon[36873]: pgmap v9691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:24.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:23 smithi042 ceph-mon[32775]: pgmap v9691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:25 smithi164 ceph-mon[36970]: pgmap v9692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:43:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:25 smithi163 ceph-mon[36873]: pgmap v9692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:43:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:25 smithi042 ceph-mon[32775]: pgmap v9692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:43:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17236.518177 secs 2024-07-26T11:43:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17236.518177 secs 2024-07-26T11:43:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17236.518177 secs 2024-07-26T11:43:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:27 smithi164 ceph-mon[36970]: pgmap v9693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17254.256123 secs 2024-07-26T11:43:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:27 smithi163 ceph-mon[36873]: pgmap v9693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:28.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17254.256123 secs 2024-07-26T11:43:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:27 smithi042 ceph-mon[32775]: pgmap v9693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17254.256123 secs 2024-07-26T11:43:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:43:28.804+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:43:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:43:28.804+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:43:28.804+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:43:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:29 smithi164 ceph-mon[36970]: pgmap v9694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:29 smithi163 ceph-mon[36873]: pgmap v9694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:29 smithi042 ceph-mon[32775]: pgmap v9694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:31 smithi164 ceph-mon[36970]: pgmap v9695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17241.518270 secs 2024-07-26T11:43:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:31 smithi163 ceph-mon[36873]: pgmap v9695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17241.518270 secs 2024-07-26T11:43:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:31 smithi042 ceph-mon[32775]: pgmap v9695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17241.518270 secs 2024-07-26T11:43:32.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17259.256222 secs 2024-07-26T11:43:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17259.256222 secs 2024-07-26T11:43:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17259.256222 secs 2024-07-26T11:43:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:33 smithi164 ceph-mon[36970]: pgmap v9696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:33 smithi163 ceph-mon[36873]: pgmap v9696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:34.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:33 smithi042 ceph-mon[32775]: pgmap v9696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:35 smithi164 ceph-mon[36970]: pgmap v9697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:43:36.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:35 smithi163 ceph-mon[36873]: pgmap v9697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:43:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:35 smithi042 ceph-mon[32775]: pgmap v9697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:43:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17246.518408 secs 2024-07-26T11:43:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17246.518408 secs 2024-07-26T11:43:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17246.518408 secs 2024-07-26T11:43:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:37 smithi164 ceph-mon[36970]: pgmap v9698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:38.028 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17264.256295 secs 2024-07-26T11:43:38.031 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:37 smithi163 ceph-mon[36873]: pgmap v9698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:38.032 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17264.256295 secs 2024-07-26T11:43:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:37 smithi042 ceph-mon[32775]: pgmap v9698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17264.256295 secs 2024-07-26T11:43:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:39 smithi164 ceph-mon[36970]: pgmap v9699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:40.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:39 smithi163 ceph-mon[36873]: pgmap v9699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:39 smithi042 ceph-mon[32775]: pgmap v9699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:41 smithi164 ceph-mon[36970]: pgmap v9700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17251.518500 secs 2024-07-26T11:43:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17269.256451 secs 2024-07-26T11:43:42.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:41 smithi163 ceph-mon[36873]: pgmap v9700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:42.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17251.518500 secs 2024-07-26T11:43:42.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17269.256451 secs 2024-07-26T11:43:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:41 smithi042 ceph-mon[32775]: pgmap v9700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17251.518500 secs 2024-07-26T11:43:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17269.256451 secs 2024-07-26T11:43:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:43:43.806+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:43:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:43:43.806+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:43:43.806+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:43:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:43 smithi164 ceph-mon[36970]: pgmap v9701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:44.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:43 smithi163 ceph-mon[36873]: pgmap v9701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:43 smithi042 ceph-mon[32775]: pgmap v9701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:45 smithi164 ceph-mon[36970]: pgmap v9702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:43:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:45 smithi163 ceph-mon[36873]: pgmap v9702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:43:46.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:45 smithi042 ceph-mon[32775]: pgmap v9702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:43:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17256.518629 secs 2024-07-26T11:43:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17274.256548 secs 2024-07-26T11:43:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17256.518629 secs 2024-07-26T11:43:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17274.256548 secs 2024-07-26T11:43:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17256.518629 secs 2024-07-26T11:43:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17274.256548 secs 2024-07-26T11:43:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:47 smithi164 ceph-mon[36970]: pgmap v9703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:47 smithi163 ceph-mon[36873]: pgmap v9703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:47 smithi042 ceph-mon[32775]: pgmap v9703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:49 smithi164 ceph-mon[36970]: pgmap v9704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:49 smithi163 ceph-mon[36873]: pgmap v9704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:50.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:49 smithi042 ceph-mon[32775]: pgmap v9704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:51 smithi163 ceph-mon[36873]: pgmap v9705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17261.518752 secs 2024-07-26T11:43:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17279.256692 secs 2024-07-26T11:43:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:51 smithi042 ceph-mon[32775]: pgmap v9705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17261.518752 secs 2024-07-26T11:43:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17279.256692 secs 2024-07-26T11:43:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:51 smithi164 ceph-mon[36970]: pgmap v9705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17261.518752 secs 2024-07-26T11:43:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17279.256692 secs 2024-07-26T11:43:54.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:53 smithi163 ceph-mon[36873]: pgmap v9706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:53 smithi042 ceph-mon[32775]: pgmap v9706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:53 smithi164 ceph-mon[36970]: pgmap v9706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:43:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:55 smithi163 ceph-mon[36873]: pgmap v9707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:43:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:55 smithi042 ceph-mon[32775]: pgmap v9707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:43:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:55 smithi164 ceph-mon[36970]: pgmap v9707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:43:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17266.518891 secs 2024-07-26T11:43:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17284.256791 secs 2024-07-26T11:43:57.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17266.518891 secs 2024-07-26T11:43:57.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17284.256791 secs 2024-07-26T11:43:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17266.518891 secs 2024-07-26T11:43:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17284.256791 secs 2024-07-26T11:43:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:57 smithi163 ceph-mon[36873]: pgmap v9708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:57 smithi042 ceph-mon[32775]: pgmap v9708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:57 smithi164 ceph-mon[36970]: pgmap v9708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:43:58.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:43:58.807+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:43:58.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:43:58.807+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:58.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:43:58.807+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:44:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:43:59 smithi163 ceph-mon[36873]: pgmap v9709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:43:59 smithi042 ceph-mon[32775]: pgmap v9709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:43:59 smithi164 ceph-mon[36970]: pgmap v9709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:00.588 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:00.587+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006480 con 0x7ff680102000 2024-07-26T11:44:00.588 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:00.587+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:44:00.588 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:00.587+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:44:00.588 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:00.587+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6700e9050 con 0x7ff680102000 2024-07-26T11:44:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:00 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:44:01.030 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:00 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:44:01.030 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:00 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:44:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:01 smithi163 ceph-mon[36873]: pgmap v9710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17271.518986 secs 2024-07-26T11:44:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17289.256864 secs 2024-07-26T11:44:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:01 smithi042 ceph-mon[32775]: pgmap v9710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17271.518986 secs 2024-07-26T11:44:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17289.256864 secs 2024-07-26T11:44:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:01 smithi164 ceph-mon[36970]: pgmap v9710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17271.518986 secs 2024-07-26T11:44:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17289.256864 secs 2024-07-26T11:44:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:03.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:02 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:03.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:03.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:03.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:03.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:02 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:02 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:03 smithi042 ceph-mon[32775]: pgmap v9711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:44:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:44:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:03 smithi042 ceph-mon[32775]: pgmap v9712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:44:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:03 smithi042 ceph-mon[32775]: pgmap v9713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:44:04.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:03 smithi164 ceph-mon[36970]: pgmap v9711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:04.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:44:04.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:44:04.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:03 smithi164 ceph-mon[36970]: pgmap v9712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:44:04.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:03 smithi164 ceph-mon[36970]: pgmap v9713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:44:04.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:03 smithi163 ceph-mon[36873]: pgmap v9711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:44:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:44:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:03 smithi163 ceph-mon[36873]: pgmap v9712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:44:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:03 smithi163 ceph-mon[36873]: pgmap v9713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:44:04.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:44:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:05 smithi164 ceph-mon[36970]: pgmap v9714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:44:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:05 smithi163 ceph-mon[36873]: pgmap v9714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:44:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:05 smithi042 ceph-mon[32775]: pgmap v9714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:44:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17276.519111 secs 2024-07-26T11:44:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17294.257008 secs 2024-07-26T11:44:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17276.519111 secs 2024-07-26T11:44:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17294.257008 secs 2024-07-26T11:44:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17276.519111 secs 2024-07-26T11:44:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17294.257008 secs 2024-07-26T11:44:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:07 smithi164 ceph-mon[36970]: pgmap v9715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:44:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:07 smithi163 ceph-mon[36873]: pgmap v9715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:44:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:07 smithi042 ceph-mon[32775]: pgmap v9715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:44:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:09 smithi164 ceph-mon[36970]: pgmap v9716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:44:10.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:09 smithi163 ceph-mon[36873]: pgmap v9716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:44:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:09 smithi042 ceph-mon[32775]: pgmap v9716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:44:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:11 smithi164 ceph-mon[36970]: pgmap v9717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:44:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17281.519250 secs 2024-07-26T11:44:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17299.257115 secs 2024-07-26T11:44:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:11 smithi163 ceph-mon[36873]: pgmap v9717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:44:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17281.519250 secs 2024-07-26T11:44:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17299.257115 secs 2024-07-26T11:44:12.272 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:11 smithi042 ceph-mon[32775]: pgmap v9717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:44:12.272 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17281.519250 secs 2024-07-26T11:44:12.273 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17299.257115 secs 2024-07-26T11:44:13.809 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:13.807+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:44:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:13.807+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:13.807+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:44:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:13 smithi164 ceph-mon[36970]: pgmap v9718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:44:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:13 smithi163 ceph-mon[36873]: pgmap v9718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:44:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:13 smithi042 ceph-mon[32775]: pgmap v9718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:44:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:15 smithi164 ceph-mon[36970]: pgmap v9719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:15 smithi163 ceph-mon[36873]: pgmap v9719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:15 smithi042 ceph-mon[32775]: pgmap v9719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17286.519369 secs 2024-07-26T11:44:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17304.257233 secs 2024-07-26T11:44:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17286.519369 secs 2024-07-26T11:44:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17304.257233 secs 2024-07-26T11:44:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17286.519369 secs 2024-07-26T11:44:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17304.257233 secs 2024-07-26T11:44:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:17 smithi164 ceph-mon[36970]: pgmap v9720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:17 smithi163 ceph-mon[36873]: pgmap v9720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:18.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:17 smithi042 ceph-mon[32775]: pgmap v9720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:19 smithi164 ceph-mon[36970]: pgmap v9721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:44:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:19 smithi163 ceph-mon[36873]: pgmap v9721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:44:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:19 smithi042 ceph-mon[32775]: pgmap v9721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:44:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:21 smithi164 ceph-mon[36970]: pgmap v9722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17291.519467 secs 2024-07-26T11:44:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17309.257362 secs 2024-07-26T11:44:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:21 smithi163 ceph-mon[36873]: pgmap v9722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17291.519467 secs 2024-07-26T11:44:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17309.257362 secs 2024-07-26T11:44:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:21 smithi042 ceph-mon[32775]: pgmap v9722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17291.519467 secs 2024-07-26T11:44:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17309.257362 secs 2024-07-26T11:44:24.197 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:23 smithi042 ceph-mon[32775]: pgmap v9723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:24.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:23 smithi164 ceph-mon[36970]: pgmap v9723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:23 smithi163 ceph-mon[36873]: pgmap v9723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:25 smithi164 ceph-mon[36970]: pgmap v9724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:26.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:25 smithi163 ceph-mon[36873]: pgmap v9724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:25 smithi042 ceph-mon[32775]: pgmap v9724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:27.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17296.519617 secs 2024-07-26T11:44:27.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17314.257482 secs 2024-07-26T11:44:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17296.519617 secs 2024-07-26T11:44:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17314.257482 secs 2024-07-26T11:44:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17296.519617 secs 2024-07-26T11:44:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17314.257482 secs 2024-07-26T11:44:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:27 smithi164 ceph-mon[36970]: pgmap v9725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:27 smithi163 ceph-mon[36873]: pgmap v9725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:28.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:27 smithi042 ceph-mon[32775]: pgmap v9725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:28.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:28.812+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:44:28.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:28.812+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:28.813 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:28.812+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:44:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:29 smithi164 ceph-mon[36970]: pgmap v9726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:44:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:29 smithi163 ceph-mon[36873]: pgmap v9726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:44:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:29 smithi042 ceph-mon[32775]: pgmap v9726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:44:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:31 smithi164 ceph-mon[36970]: pgmap v9727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17301.519755 secs 2024-07-26T11:44:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17319.257580 secs 2024-07-26T11:44:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:31 smithi163 ceph-mon[36873]: pgmap v9727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17301.519755 secs 2024-07-26T11:44:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17319.257580 secs 2024-07-26T11:44:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:31 smithi042 ceph-mon[32775]: pgmap v9727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17301.519755 secs 2024-07-26T11:44:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17319.257580 secs 2024-07-26T11:44:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:33 smithi164 ceph-mon[36970]: pgmap v9728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:34.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:33 smithi163 ceph-mon[36873]: pgmap v9728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:34.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:33 smithi042 ceph-mon[32775]: pgmap v9728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:35 smithi164 ceph-mon[36970]: pgmap v9729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:35 smithi163 ceph-mon[36873]: pgmap v9729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:35 smithi042 ceph-mon[32775]: pgmap v9729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17306.519865 secs 2024-07-26T11:44:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17324.257703 secs 2024-07-26T11:44:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17306.519865 secs 2024-07-26T11:44:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17324.257703 secs 2024-07-26T11:44:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17306.519865 secs 2024-07-26T11:44:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17324.257703 secs 2024-07-26T11:44:38.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:37 smithi163 ceph-mon[36873]: pgmap v9730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:37 smithi042 ceph-mon[32775]: pgmap v9730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:37 smithi164 ceph-mon[36970]: pgmap v9730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:39 smithi163 ceph-mon[36873]: pgmap v9731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:44:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:39 smithi042 ceph-mon[32775]: pgmap v9731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:44:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:39 smithi164 ceph-mon[36970]: pgmap v9731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:44:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:41 smithi163 ceph-mon[36873]: pgmap v9732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17311.519987 secs 2024-07-26T11:44:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17329.257824 secs 2024-07-26T11:44:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:41 smithi042 ceph-mon[32775]: pgmap v9732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17311.519987 secs 2024-07-26T11:44:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17329.257824 secs 2024-07-26T11:44:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:41 smithi164 ceph-mon[36970]: pgmap v9732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17311.519987 secs 2024-07-26T11:44:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17329.257824 secs 2024-07-26T11:44:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:43.813+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:44:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:43.813+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:43.813+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:44:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:43 smithi163 ceph-mon[36873]: pgmap v9733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:43 smithi042 ceph-mon[32775]: pgmap v9733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:43 smithi164 ceph-mon[36970]: pgmap v9733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:45.594+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:44:45.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:45.594+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658006480 con 0x7ff680102000 2024-07-26T11:44:45.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:45.594+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:44:45.596 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:45.594+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6700e9050 con 0x7ff680102000 2024-07-26T11:44:46.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:45 smithi163 ceph-mon[36873]: pgmap v9734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:45 smithi042 ceph-mon[32775]: pgmap v9734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:45 smithi164 ceph-mon[36970]: pgmap v9734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17316.520130 secs 2024-07-26T11:44:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17334.257962 secs 2024-07-26T11:44:47.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17316.520130 secs 2024-07-26T11:44:47.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17334.257962 secs 2024-07-26T11:44:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17316.520130 secs 2024-07-26T11:44:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17334.257962 secs 2024-07-26T11:44:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:47 smithi163 ceph-mon[36873]: pgmap v9735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:47 smithi042 ceph-mon[32775]: pgmap v9735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:47 smithi164 ceph-mon[36970]: pgmap v9735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:50.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:50 smithi163 ceph-mon[36873]: pgmap v9736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:44:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:50 smithi042 ceph-mon[32775]: pgmap v9736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:44:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:50 smithi164 ceph-mon[36970]: pgmap v9736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:44:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:52 smithi163 ceph-mon[36873]: pgmap v9737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17321.520288 secs 2024-07-26T11:44:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17339.258082 secs 2024-07-26T11:44:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:52 smithi042 ceph-mon[32775]: pgmap v9737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17321.520288 secs 2024-07-26T11:44:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17339.258082 secs 2024-07-26T11:44:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:52 smithi164 ceph-mon[36970]: pgmap v9737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17321.520288 secs 2024-07-26T11:44:52.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17339.258082 secs 2024-07-26T11:44:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:54 smithi042 ceph-mon[32775]: pgmap v9738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:54 smithi164 ceph-mon[36970]: pgmap v9738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:54 smithi163 ceph-mon[36873]: pgmap v9738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:56 smithi042 ceph-mon[32775]: pgmap v9739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:56 smithi164 ceph-mon[36970]: pgmap v9739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:56 smithi163 ceph-mon[36873]: pgmap v9739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:44:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17326.520379 secs 2024-07-26T11:44:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17344.258200 secs 2024-07-26T11:44:57.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17326.520379 secs 2024-07-26T11:44:57.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17344.258200 secs 2024-07-26T11:44:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17326.520379 secs 2024-07-26T11:44:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17344.258200 secs 2024-07-26T11:44:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:44:58 smithi042 ceph-mon[32775]: pgmap v9740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:44:58 smithi164 ceph-mon[36970]: pgmap v9740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:44:58 smithi163 ceph-mon[36873]: pgmap v9740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:44:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:58.819+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:44:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:58.819+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:44:58.819+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:45:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:00 smithi042 ceph-mon[32775]: pgmap v9741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:45:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:00 smithi164 ceph-mon[36970]: pgmap v9741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:45:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:00 smithi163 ceph-mon[36873]: pgmap v9741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:45:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:02 smithi042 ceph-mon[32775]: pgmap v9742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17331.520497 secs 2024-07-26T11:45:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17349.258305 secs 2024-07-26T11:45:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:02 smithi164 ceph-mon[36970]: pgmap v9742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17331.520497 secs 2024-07-26T11:45:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17349.258305 secs 2024-07-26T11:45:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:02 smithi163 ceph-mon[36873]: pgmap v9742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17331.520497 secs 2024-07-26T11:45:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17349.258305 secs 2024-07-26T11:45:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:03 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:45:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:03 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:45:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:03 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:45:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:04 smithi042 ceph-mon[32775]: pgmap v9743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:45:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:45:04.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:04 smithi164 ceph-mon[36970]: pgmap v9743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:45:04.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:45:04.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:04 smithi163 ceph-mon[36873]: pgmap v9743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:04.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:45:04.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:45:04.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:45:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:05 smithi042 ceph-mon[32775]: pgmap v9744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:45:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:05 smithi042 ceph-mon[32775]: pgmap v9745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:45:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:05 smithi164 ceph-mon[36970]: pgmap v9744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:45:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:05 smithi164 ceph-mon[36970]: pgmap v9745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:45:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:05 smithi163 ceph-mon[36873]: pgmap v9744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:45:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:05 smithi163 ceph-mon[36873]: pgmap v9745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:45:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:07 smithi042 ceph-mon[32775]: pgmap v9746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17336.520617 secs 2024-07-26T11:45:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17354.258426 secs 2024-07-26T11:45:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:07 smithi164 ceph-mon[36970]: pgmap v9746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17336.520617 secs 2024-07-26T11:45:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17354.258426 secs 2024-07-26T11:45:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:07 smithi163 ceph-mon[36873]: pgmap v9746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17336.520617 secs 2024-07-26T11:45:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17354.258426 secs 2024-07-26T11:45:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:09 smithi042 ceph-mon[32775]: pgmap v9747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:45:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:09 smithi164 ceph-mon[36970]: pgmap v9747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:45:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:09 smithi163 ceph-mon[36873]: pgmap v9747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:45:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:11 smithi042 ceph-mon[32775]: pgmap v9748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:11 smithi164 ceph-mon[36970]: pgmap v9748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:11.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:11 smithi163 ceph-mon[36873]: pgmap v9748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T11:45:12.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17341.520686 secs 2024-07-26T11:45:12.399 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17359.258550 secs 2024-07-26T11:45:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17341.520686 secs 2024-07-26T11:45:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17359.258550 secs 2024-07-26T11:45:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17341.520686 secs 2024-07-26T11:45:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17359.258550 secs 2024-07-26T11:45:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:13 smithi042 ceph-mon[32775]: pgmap v9749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:45:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:13 smithi164 ceph-mon[36970]: pgmap v9749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:45:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:13 smithi163 ceph-mon[36873]: pgmap v9749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T11:45:13.822 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:45:13.821+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:45:13.822 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:45:13.821+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:13.822 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:45:13.821+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:45:14.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:14 smithi042 ceph-mon[32775]: pgmap v9750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:45:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:14 smithi164 ceph-mon[36970]: pgmap v9750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:45:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:14 smithi163 ceph-mon[36873]: pgmap v9750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:45:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:17 smithi042 ceph-mon[32775]: pgmap v9751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17346.520792 secs 2024-07-26T11:45:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17364.258702 secs 2024-07-26T11:45:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:17 smithi164 ceph-mon[36970]: pgmap v9751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17346.520792 secs 2024-07-26T11:45:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17364.258702 secs 2024-07-26T11:45:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:17 smithi163 ceph-mon[36873]: pgmap v9751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17346.520792 secs 2024-07-26T11:45:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17364.258702 secs 2024-07-26T11:45:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:19 smithi042 ceph-mon[32775]: pgmap v9752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:19 smithi164 ceph-mon[36970]: pgmap v9752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:19 smithi163 ceph-mon[36873]: pgmap v9752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:21 smithi042 ceph-mon[32775]: pgmap v9753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:21 smithi164 ceph-mon[36970]: pgmap v9753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:21 smithi163 ceph-mon[36873]: pgmap v9753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17351.520911 secs 2024-07-26T11:45:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17369.258854 secs 2024-07-26T11:45:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17351.520911 secs 2024-07-26T11:45:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17369.258854 secs 2024-07-26T11:45:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17351.520911 secs 2024-07-26T11:45:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17369.258854 secs 2024-07-26T11:45:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:23 smithi042 ceph-mon[32775]: pgmap v9754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:23 smithi164 ceph-mon[36970]: pgmap v9754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:23 smithi163 ceph-mon[36873]: pgmap v9754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:24 smithi042 ceph-mon[32775]: pgmap v9755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:45:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:24 smithi164 ceph-mon[36970]: pgmap v9755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:45:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:24 smithi163 ceph-mon[36873]: pgmap v9755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:45:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:27 smithi042 ceph-mon[32775]: pgmap v9756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17356.521057 secs 2024-07-26T11:45:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17374.258974 secs 2024-07-26T11:45:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:27 smithi164 ceph-mon[36970]: pgmap v9756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17356.521057 secs 2024-07-26T11:45:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17374.258974 secs 2024-07-26T11:45:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:27 smithi163 ceph-mon[36873]: pgmap v9756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17356.521057 secs 2024-07-26T11:45:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17374.258974 secs 2024-07-26T11:45:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:45:28.827+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:45:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:45:28.827+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:45:28.827+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:45:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:29 smithi042 ceph-mon[32775]: pgmap v9757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:29.404 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:29 smithi163 ceph-mon[36873]: pgmap v9757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:29 smithi164 ceph-mon[36970]: pgmap v9757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:30 smithi042 ceph-mon[32775]: pgmap v9758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:30 smithi164 ceph-mon[36970]: pgmap v9758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:30 smithi163 ceph-mon[36873]: pgmap v9758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:31.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17361.521175 secs 2024-07-26T11:45:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17361.521175 secs 2024-07-26T11:45:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17361.521175 secs 2024-07-26T11:45:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17379.259044 secs 2024-07-26T11:45:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:32 smithi042 ceph-mon[32775]: pgmap v9759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17379.259044 secs 2024-07-26T11:45:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:32 smithi164 ceph-mon[36970]: pgmap v9759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17379.259044 secs 2024-07-26T11:45:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:32 smithi163 ceph-mon[36873]: pgmap v9759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:35 smithi042 ceph-mon[32775]: pgmap v9760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:45:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:35 smithi164 ceph-mon[36970]: pgmap v9760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:45:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:35 smithi163 ceph-mon[36873]: pgmap v9760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:45:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:37 smithi042 ceph-mon[32775]: pgmap v9761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:37.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17366.521299 secs 2024-07-26T11:45:37.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17384.259137 secs 2024-07-26T11:45:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:37 smithi164 ceph-mon[36970]: pgmap v9761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17366.521299 secs 2024-07-26T11:45:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17384.259137 secs 2024-07-26T11:45:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:37 smithi163 ceph-mon[36873]: pgmap v9761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17366.521299 secs 2024-07-26T11:45:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17384.259137 secs 2024-07-26T11:45:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:39 smithi163 ceph-mon[36873]: pgmap v9762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:39 smithi042 ceph-mon[32775]: pgmap v9762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:39 smithi164 ceph-mon[36970]: pgmap v9762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:41.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:41 smithi042 ceph-mon[32775]: pgmap v9763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:41 smithi164 ceph-mon[36970]: pgmap v9763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:41 smithi163 ceph-mon[36873]: pgmap v9763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17371.521369 secs 2024-07-26T11:45:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17389.259270 secs 2024-07-26T11:45:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17371.521369 secs 2024-07-26T11:45:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17389.259270 secs 2024-07-26T11:45:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17371.521369 secs 2024-07-26T11:45:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17389.259270 secs 2024-07-26T11:45:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:43 smithi042 ceph-mon[32775]: pgmap v9764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:43 smithi164 ceph-mon[36970]: pgmap v9764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:43 smithi163 ceph-mon[36873]: pgmap v9764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:43.830 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:45:43.829+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:45:43.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:45:43.829+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:45:43.831 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:45:43.829+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:44 smithi042 ceph-mon[32775]: pgmap v9765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:44 smithi164 ceph-mon[36970]: pgmap v9765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:44.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:44 smithi163 ceph-mon[36873]: pgmap v9765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:47 smithi042 ceph-mon[32775]: pgmap v9766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17376.521498 secs 2024-07-26T11:45:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17394.259390 secs 2024-07-26T11:45:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:47 smithi164 ceph-mon[36970]: pgmap v9766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17376.521498 secs 2024-07-26T11:45:47.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17394.259390 secs 2024-07-26T11:45:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:47 smithi163 ceph-mon[36873]: pgmap v9766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17376.521498 secs 2024-07-26T11:45:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17394.259390 secs 2024-07-26T11:45:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:49 smithi042 ceph-mon[32775]: pgmap v9767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:49 smithi164 ceph-mon[36970]: pgmap v9767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:49 smithi163 ceph-mon[36873]: pgmap v9767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:51 smithi042 ceph-mon[32775]: pgmap v9768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:51 smithi164 ceph-mon[36970]: pgmap v9768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:51 smithi163 ceph-mon[36873]: pgmap v9768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17381.521577 secs 2024-07-26T11:45:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17399.259514 secs 2024-07-26T11:45:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17381.521577 secs 2024-07-26T11:45:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17399.259514 secs 2024-07-26T11:45:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17381.521577 secs 2024-07-26T11:45:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17399.259514 secs 2024-07-26T11:45:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:53 smithi042 ceph-mon[32775]: pgmap v9769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:53 smithi164 ceph-mon[36970]: pgmap v9769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:53.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:53 smithi163 ceph-mon[36873]: pgmap v9769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:54.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:54 smithi042 ceph-mon[32775]: pgmap v9770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:54 smithi164 ceph-mon[36970]: pgmap v9770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:54 smithi163 ceph-mon[36873]: pgmap v9770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:45:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:57 smithi042 ceph-mon[32775]: pgmap v9771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17386.521634 secs 2024-07-26T11:45:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17404.259640 secs 2024-07-26T11:45:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:57 smithi164 ceph-mon[36970]: pgmap v9771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17386.521634 secs 2024-07-26T11:45:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17404.259640 secs 2024-07-26T11:45:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:57 smithi163 ceph-mon[36873]: pgmap v9771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17386.521634 secs 2024-07-26T11:45:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17404.259640 secs 2024-07-26T11:45:58.836 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:45:58.835+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:45:58.836 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:45:58.835+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:58.836 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:45:58.835+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:45:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:45:59 smithi042 ceph-mon[32775]: pgmap v9772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:45:59 smithi164 ceph-mon[36970]: pgmap v9772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:45:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:45:59 smithi163 ceph-mon[36873]: pgmap v9772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:01 smithi042 ceph-mon[32775]: pgmap v9773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:01 smithi164 ceph-mon[36970]: pgmap v9773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:01 smithi163 ceph-mon[36873]: pgmap v9773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17391.521737 secs 2024-07-26T11:46:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17409.259761 secs 2024-07-26T11:46:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:02 smithi042 ceph-mon[32775]: pgmap v9774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17391.521737 secs 2024-07-26T11:46:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17409.259761 secs 2024-07-26T11:46:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:02 smithi164 ceph-mon[36970]: pgmap v9774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17391.521737 secs 2024-07-26T11:46:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17409.259761 secs 2024-07-26T11:46:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:02 smithi163 ceph-mon[36873]: pgmap v9774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:05 smithi042 ceph-mon[32775]: pgmap v9775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:46:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:46:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:46:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:46:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:05 smithi164 ceph-mon[36970]: pgmap v9775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:46:05.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:46:05.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:46:05.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:46:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:05 smithi163 ceph-mon[36873]: pgmap v9775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:46:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:46:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:46:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:46:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:06 smithi042 ceph-mon[32775]: pgmap v9776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:46:06.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:06 smithi042 ceph-mon[32775]: pgmap v9777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:46:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:06 smithi164 ceph-mon[36970]: pgmap v9776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:46:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:06 smithi164 ceph-mon[36970]: pgmap v9777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:46:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:06 smithi163 ceph-mon[36873]: pgmap v9776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:46:06.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:06 smithi163 ceph-mon[36873]: pgmap v9777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:46:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17396.521840 secs 2024-07-26T11:46:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17414.259886 secs 2024-07-26T11:46:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:07 smithi042 ceph-mon[32775]: pgmap v9778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:46:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17396.521840 secs 2024-07-26T11:46:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17414.259886 secs 2024-07-26T11:46:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:07 smithi164 ceph-mon[36970]: pgmap v9778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:46:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17396.521840 secs 2024-07-26T11:46:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17414.259886 secs 2024-07-26T11:46:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:07 smithi163 ceph-mon[36873]: pgmap v9778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:46:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:09 smithi042 ceph-mon[32775]: pgmap v9779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:46:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:09 smithi164 ceph-mon[36970]: pgmap v9779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:46:10.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:09 smithi163 ceph-mon[36873]: pgmap v9779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:46:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:11 smithi042 ceph-mon[32775]: pgmap v9780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:46:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17401.521958 secs 2024-07-26T11:46:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:11 smithi164 ceph-mon[36970]: pgmap v9780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:46:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17401.521958 secs 2024-07-26T11:46:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:11 smithi163 ceph-mon[36873]: pgmap v9780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:46:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17401.521958 secs 2024-07-26T11:46:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17419.260007 secs 2024-07-26T11:46:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17419.260007 secs 2024-07-26T11:46:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17419.260007 secs 2024-07-26T11:46:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:46:13.835+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:46:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:46:13.835+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:46:13.835+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:46:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:13 smithi042 ceph-mon[32775]: pgmap v9781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:46:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:13 smithi164 ceph-mon[36970]: pgmap v9781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:46:14.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:13 smithi163 ceph-mon[36873]: pgmap v9781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:46:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:15 smithi042 ceph-mon[32775]: pgmap v9782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:15 smithi164 ceph-mon[36970]: pgmap v9782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:16.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:15 smithi163 ceph-mon[36873]: pgmap v9782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:46:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17406.522064 secs 2024-07-26T11:46:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17406.522064 secs 2024-07-26T11:46:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17406.522064 secs 2024-07-26T11:46:17.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17424.260129 secs 2024-07-26T11:46:17.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:17 smithi164 ceph-mon[36970]: pgmap v9783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17424.260129 secs 2024-07-26T11:46:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:17 smithi042 ceph-mon[32775]: pgmap v9783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17424.260129 secs 2024-07-26T11:46:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:17 smithi163 ceph-mon[36873]: pgmap v9783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:19.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:19 smithi042 ceph-mon[32775]: pgmap v9784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:46:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:19 smithi164 ceph-mon[36970]: pgmap v9784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:46:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:19 smithi163 ceph-mon[36873]: pgmap v9784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:46:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:21 smithi042 ceph-mon[32775]: pgmap v9785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:21.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17411.522205 secs 2024-07-26T11:46:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:21 smithi164 ceph-mon[36970]: pgmap v9785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:21.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17411.522205 secs 2024-07-26T11:46:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:21 smithi163 ceph-mon[36873]: pgmap v9785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17411.522205 secs 2024-07-26T11:46:22.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17429.260252 secs 2024-07-26T11:46:22.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17429.260252 secs 2024-07-26T11:46:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17429.260252 secs 2024-07-26T11:46:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:23 smithi042 ceph-mon[32775]: pgmap v9786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:23.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:23 smithi164 ceph-mon[36970]: pgmap v9786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:24.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:23 smithi163 ceph-mon[36873]: pgmap v9786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:25 smithi042 ceph-mon[32775]: pgmap v9787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:25.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:25 smithi164 ceph-mon[36970]: pgmap v9787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:25 smithi163 ceph-mon[36873]: pgmap v9787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17416.522329 secs 2024-07-26T11:46:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17416.522329 secs 2024-07-26T11:46:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17416.522329 secs 2024-07-26T11:46:27.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17434.260373 secs 2024-07-26T11:46:27.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:27 smithi163 ceph-mon[36873]: pgmap v9788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17434.260373 secs 2024-07-26T11:46:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:27 smithi164 ceph-mon[36970]: pgmap v9788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17434.260373 secs 2024-07-26T11:46:28.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:27 smithi042 ceph-mon[32775]: pgmap v9788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:46:28.841+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:46:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:46:28.841+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:46:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:46:28.841+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:29 smithi164 ceph-mon[36970]: pgmap v9789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:46:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:29 smithi163 ceph-mon[36873]: pgmap v9789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:46:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:29 smithi042 ceph-mon[32775]: pgmap v9789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:46:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:31 smithi164 ceph-mon[36970]: pgmap v9790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17421.522449 secs 2024-07-26T11:46:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:31 smithi163 ceph-mon[36873]: pgmap v9790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17421.522449 secs 2024-07-26T11:46:32.048 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:31 smithi042 ceph-mon[32775]: pgmap v9790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:32.048 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17421.522449 secs 2024-07-26T11:46:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17439.260517 secs 2024-07-26T11:46:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17439.260517 secs 2024-07-26T11:46:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17439.260517 secs 2024-07-26T11:46:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:33 smithi164 ceph-mon[36970]: pgmap v9791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:34.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:33 smithi163 ceph-mon[36873]: pgmap v9791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:34.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:33 smithi042 ceph-mon[32775]: pgmap v9791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:35 smithi042 ceph-mon[32775]: pgmap v9792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:35 smithi164 ceph-mon[36970]: pgmap v9792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:35 smithi163 ceph-mon[36873]: pgmap v9792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17426.522576 secs 2024-07-26T11:46:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17426.522576 secs 2024-07-26T11:46:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17426.522576 secs 2024-07-26T11:46:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17444.260635 secs 2024-07-26T11:46:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:37 smithi042 ceph-mon[32775]: pgmap v9793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17444.260635 secs 2024-07-26T11:46:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:37 smithi164 ceph-mon[36970]: pgmap v9793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17444.260635 secs 2024-07-26T11:46:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:37 smithi163 ceph-mon[36873]: pgmap v9793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:39 smithi042 ceph-mon[32775]: pgmap v9794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:46:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:39 smithi164 ceph-mon[36970]: pgmap v9794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:46:40.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:39 smithi163 ceph-mon[36873]: pgmap v9794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:46:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:41 smithi042 ceph-mon[32775]: pgmap v9795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17431.522703 secs 2024-07-26T11:46:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:41 smithi164 ceph-mon[36970]: pgmap v9795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17431.522703 secs 2024-07-26T11:46:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:41 smithi163 ceph-mon[36873]: pgmap v9795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17431.522703 secs 2024-07-26T11:46:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17449.260773 secs 2024-07-26T11:46:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17449.260773 secs 2024-07-26T11:46:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17449.260773 secs 2024-07-26T11:46:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:46:43.842+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:46:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:46:43.842+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:46:43.842+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:46:43.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:43 smithi042 ceph-mon[32775]: pgmap v9796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:43 smithi164 ceph-mon[36970]: pgmap v9796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:43 smithi163 ceph-mon[36873]: pgmap v9796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:45 smithi042 ceph-mon[32775]: pgmap v9797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:45 smithi164 ceph-mon[36970]: pgmap v9797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:46.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:45 smithi163 ceph-mon[36873]: pgmap v9797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17436.522805 secs 2024-07-26T11:46:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17436.522805 secs 2024-07-26T11:46:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17436.522805 secs 2024-07-26T11:46:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17454.260870 secs 2024-07-26T11:46:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:47 smithi042 ceph-mon[32775]: pgmap v9798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17454.260870 secs 2024-07-26T11:46:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:47 smithi164 ceph-mon[36970]: pgmap v9798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17454.260870 secs 2024-07-26T11:46:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:47 smithi163 ceph-mon[36873]: pgmap v9798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:49.898 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:49 smithi042 ceph-mon[32775]: pgmap v9799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:46:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:49 smithi164 ceph-mon[36970]: pgmap v9799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:46:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:49 smithi163 ceph-mon[36873]: pgmap v9799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:46:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:51 smithi042 ceph-mon[32775]: pgmap v9800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17441.522927 secs 2024-07-26T11:46:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:51 smithi164 ceph-mon[36970]: pgmap v9800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17441.522927 secs 2024-07-26T11:46:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:51 smithi163 ceph-mon[36873]: pgmap v9800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17441.522927 secs 2024-07-26T11:46:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17459.261005 secs 2024-07-26T11:46:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17459.261005 secs 2024-07-26T11:46:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17459.261005 secs 2024-07-26T11:46:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:53 smithi042 ceph-mon[32775]: pgmap v9801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:53 smithi164 ceph-mon[36970]: pgmap v9801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:53 smithi163 ceph-mon[36873]: pgmap v9801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:55 smithi042 ceph-mon[32775]: pgmap v9802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:55 smithi164 ceph-mon[36970]: pgmap v9802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:55 smithi163 ceph-mon[36873]: pgmap v9802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:46:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17446.523068 secs 2024-07-26T11:46:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17446.523068 secs 2024-07-26T11:46:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17446.523068 secs 2024-07-26T11:46:57.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17464.261125 secs 2024-07-26T11:46:57.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:57 smithi163 ceph-mon[36873]: pgmap v9803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17464.261125 secs 2024-07-26T11:46:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:57 smithi164 ceph-mon[36970]: pgmap v9803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17464.261125 secs 2024-07-26T11:46:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:57 smithi042 ceph-mon[32775]: pgmap v9803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:46:58.849 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:46:58.848+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:46:58.849 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:46:58.848+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:46:58.849 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:46:58.848+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:46:59 smithi164 ceph-mon[36970]: pgmap v9804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:47:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:46:59 smithi163 ceph-mon[36873]: pgmap v9804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:47:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:46:59 smithi042 ceph-mon[32775]: pgmap v9804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:47:00.617 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:47:00.615+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:47:00.617 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:47:00.615+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006480 con 0x7ff680102000 2024-07-26T11:47:00.617 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:47:00.615+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:47:00.617 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:47:00.615+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6700e9050 con 0x7ff680102000 2024-07-26T11:47:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:01 smithi164 ceph-mon[36970]: pgmap v9805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17451.523194 secs 2024-07-26T11:47:02.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:01 smithi163 ceph-mon[36873]: pgmap v9805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:02.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17451.523194 secs 2024-07-26T11:47:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:01 smithi042 ceph-mon[32775]: pgmap v9805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17451.523194 secs 2024-07-26T11:47:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17469.261252 secs 2024-07-26T11:47:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17469.261252 secs 2024-07-26T11:47:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17469.261252 secs 2024-07-26T11:47:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:03 smithi164 ceph-mon[36970]: pgmap v9806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:03 smithi163 ceph-mon[36873]: pgmap v9806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:03 smithi042 ceph-mon[32775]: pgmap v9806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:04.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:04 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:47:05.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:04 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:47:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:04 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:47:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:05 smithi164 ceph-mon[36970]: pgmap v9807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:47:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:47:05.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:47:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:05 smithi163 ceph-mon[36873]: pgmap v9807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:47:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:47:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:47:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:05 smithi042 ceph-mon[32775]: pgmap v9807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:47:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:47:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:47:06.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:06 smithi164 ceph-mon[36970]: pgmap v9808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:47:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:06 smithi164 ceph-mon[36970]: pgmap v9809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:47:06.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17456.523319 secs 2024-07-26T11:47:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:06 smithi163 ceph-mon[36873]: pgmap v9808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:47:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:06 smithi163 ceph-mon[36873]: pgmap v9809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:47:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17456.523319 secs 2024-07-26T11:47:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:06 smithi042 ceph-mon[32775]: pgmap v9808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:47:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:06 smithi042 ceph-mon[32775]: pgmap v9809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:47:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17456.523319 secs 2024-07-26T11:47:07.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17474.261371 secs 2024-07-26T11:47:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17474.261371 secs 2024-07-26T11:47:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17474.261371 secs 2024-07-26T11:47:08.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:08 smithi164 ceph-mon[36970]: pgmap v9810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:47:09.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:08 smithi163 ceph-mon[36873]: pgmap v9810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:47:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:08 smithi042 ceph-mon[32775]: pgmap v9810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:47:10.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:10 smithi164 ceph-mon[36970]: pgmap v9811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:11.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:10 smithi163 ceph-mon[36873]: pgmap v9811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:10 smithi042 ceph-mon[32775]: pgmap v9811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-07-26T11:47:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17461.523433 secs 2024-07-26T11:47:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17461.523433 secs 2024-07-26T11:47:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17461.523433 secs 2024-07-26T11:47:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:12 smithi164 ceph-mon[36970]: pgmap v9812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17479.261493 secs 2024-07-26T11:47:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:12 smithi163 ceph-mon[36873]: pgmap v9812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:13.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17479.261493 secs 2024-07-26T11:47:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:12 smithi042 ceph-mon[32775]: pgmap v9812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3 op/s 2024-07-26T11:47:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17479.261493 secs 2024-07-26T11:47:13.850 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:47:13.849+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:47:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:47:13.849+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:47:13.849+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:47:14.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:14 smithi164 ceph-mon[36970]: pgmap v9813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 959 B/s rd, 2 op/s 2024-07-26T11:47:15.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:14 smithi163 ceph-mon[36873]: pgmap v9813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 959 B/s rd, 2 op/s 2024-07-26T11:47:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:14 smithi042 ceph-mon[32775]: pgmap v9813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 959 B/s rd, 2 op/s 2024-07-26T11:47:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:16 smithi164 ceph-mon[36970]: pgmap v9814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:47:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17466.523560 secs 2024-07-26T11:47:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:16 smithi163 ceph-mon[36873]: pgmap v9814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:47:17.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17466.523560 secs 2024-07-26T11:47:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:16 smithi042 ceph-mon[32775]: pgmap v9814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:47:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17466.523560 secs 2024-07-26T11:47:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17484.261615 secs 2024-07-26T11:47:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17484.261615 secs 2024-07-26T11:47:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17484.261615 secs 2024-07-26T11:47:18.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:18 smithi164 ceph-mon[36970]: pgmap v9815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:47:19.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:18 smithi163 ceph-mon[36873]: pgmap v9815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:47:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:18 smithi042 ceph-mon[32775]: pgmap v9815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:47:20.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:20 smithi164 ceph-mon[36970]: pgmap v9816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:47:21.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:20 smithi163 ceph-mon[36873]: pgmap v9816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:47:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:20 smithi042 ceph-mon[32775]: pgmap v9816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:47:22.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17471.523694 secs 2024-07-26T11:47:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17471.523694 secs 2024-07-26T11:47:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17471.523694 secs 2024-07-26T11:47:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:22 smithi163 ceph-mon[36873]: pgmap v9817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17489.261742 secs 2024-07-26T11:47:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:22 smithi042 ceph-mon[32775]: pgmap v9817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17489.261742 secs 2024-07-26T11:47:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:22 smithi164 ceph-mon[36970]: pgmap v9817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17489.261742 secs 2024-07-26T11:47:25.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:24 smithi163 ceph-mon[36873]: pgmap v9818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:24 smithi042 ceph-mon[32775]: pgmap v9818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:24 smithi164 ceph-mon[36970]: pgmap v9818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:26 smithi163 ceph-mon[36873]: pgmap v9819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17476.523815 secs 2024-07-26T11:47:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17494.261860 secs 2024-07-26T11:47:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:26 smithi042 ceph-mon[32775]: pgmap v9819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17476.523815 secs 2024-07-26T11:47:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17494.261860 secs 2024-07-26T11:47:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:26 smithi164 ceph-mon[36970]: pgmap v9819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17476.523815 secs 2024-07-26T11:47:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17494.261860 secs 2024-07-26T11:47:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:47:28.855+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:47:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:47:28.855+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:47:28.855+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:47:29.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:28 smithi163 ceph-mon[36873]: pgmap v9820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:28 smithi042 ceph-mon[32775]: pgmap v9820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:28 smithi164 ceph-mon[36970]: pgmap v9820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:30 smithi163 ceph-mon[36873]: pgmap v9821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:30 smithi042 ceph-mon[32775]: pgmap v9821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:30 smithi164 ceph-mon[36970]: pgmap v9821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17481.523939 secs 2024-07-26T11:47:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17499.261984 secs 2024-07-26T11:47:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17481.523939 secs 2024-07-26T11:47:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17499.261984 secs 2024-07-26T11:47:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17481.523939 secs 2024-07-26T11:47:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17499.261984 secs 2024-07-26T11:47:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:32 smithi163 ceph-mon[36873]: pgmap v9822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:32 smithi042 ceph-mon[32775]: pgmap v9822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:32 smithi164 ceph-mon[36970]: pgmap v9822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:34 smithi163 ceph-mon[36873]: pgmap v9823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:35.024 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:34 smithi042 ceph-mon[32775]: pgmap v9823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:34 smithi164 ceph-mon[36970]: pgmap v9823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:36 smithi163 ceph-mon[36873]: pgmap v9824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17486.524067 secs 2024-07-26T11:47:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17504.262084 secs 2024-07-26T11:47:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:36 smithi042 ceph-mon[32775]: pgmap v9824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17486.524067 secs 2024-07-26T11:47:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17504.262084 secs 2024-07-26T11:47:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:36 smithi164 ceph-mon[36970]: pgmap v9824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17486.524067 secs 2024-07-26T11:47:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17504.262084 secs 2024-07-26T11:47:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:38 smithi163 ceph-mon[36873]: pgmap v9825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:38 smithi042 ceph-mon[32775]: pgmap v9825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:38 smithi164 ceph-mon[36970]: pgmap v9825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:40 smithi042 ceph-mon[32775]: pgmap v9826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:40 smithi164 ceph-mon[36970]: pgmap v9826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:41.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:40 smithi163 ceph-mon[36873]: pgmap v9826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17491.524158 secs 2024-07-26T11:47:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17509.262206 secs 2024-07-26T11:47:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17491.524158 secs 2024-07-26T11:47:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17509.262206 secs 2024-07-26T11:47:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17491.524158 secs 2024-07-26T11:47:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17509.262206 secs 2024-07-26T11:47:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:42 smithi042 ceph-mon[32775]: pgmap v9827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:42 smithi164 ceph-mon[36970]: pgmap v9827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:42 smithi163 ceph-mon[36873]: pgmap v9827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:43.857 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:47:43.856+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:47:43.857 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:47:43.856+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:43.857 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:47:43.856+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:47:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:44 smithi042 ceph-mon[32775]: pgmap v9828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:44 smithi164 ceph-mon[36970]: pgmap v9828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:44 smithi163 ceph-mon[36873]: pgmap v9828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:47.024 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:46 smithi164 ceph-mon[36970]: pgmap v9829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:47.024 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17496.524264 secs 2024-07-26T11:47:47.024 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17514.262303 secs 2024-07-26T11:47:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:46 smithi042 ceph-mon[32775]: pgmap v9829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17496.524264 secs 2024-07-26T11:47:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17514.262303 secs 2024-07-26T11:47:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:46 smithi163 ceph-mon[36873]: pgmap v9829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17496.524264 secs 2024-07-26T11:47:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17514.262303 secs 2024-07-26T11:47:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:48 smithi042 ceph-mon[32775]: pgmap v9830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:48 smithi164 ceph-mon[36970]: pgmap v9830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:48 smithi163 ceph-mon[36873]: pgmap v9830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:50 smithi042 ceph-mon[32775]: pgmap v9831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:50 smithi164 ceph-mon[36970]: pgmap v9831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:50 smithi163 ceph-mon[36873]: pgmap v9831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17501.524392 secs 2024-07-26T11:47:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17519.262449 secs 2024-07-26T11:47:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17501.524392 secs 2024-07-26T11:47:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17519.262449 secs 2024-07-26T11:47:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17501.524392 secs 2024-07-26T11:47:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17519.262449 secs 2024-07-26T11:47:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:52 smithi042 ceph-mon[32775]: pgmap v9832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:53.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:52 smithi164 ceph-mon[36970]: pgmap v9832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:53.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:52 smithi163 ceph-mon[36873]: pgmap v9832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:54 smithi042 ceph-mon[32775]: pgmap v9833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:54 smithi164 ceph-mon[36970]: pgmap v9833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:54 smithi163 ceph-mon[36873]: pgmap v9833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:56 smithi042 ceph-mon[32775]: pgmap v9834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17506.524491 secs 2024-07-26T11:47:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17524.262596 secs 2024-07-26T11:47:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:56 smithi164 ceph-mon[36970]: pgmap v9834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17506.524491 secs 2024-07-26T11:47:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17524.262596 secs 2024-07-26T11:47:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:56 smithi163 ceph-mon[36873]: pgmap v9834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:47:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17506.524491 secs 2024-07-26T11:47:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17524.262596 secs 2024-07-26T11:47:58.862 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:47:58.860+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:47:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:47:58.860+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:47:58.860+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:47:59.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:47:58 smithi163 ceph-mon[36873]: pgmap v9835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:47:58 smithi042 ceph-mon[32775]: pgmap v9835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:47:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:47:58 smithi164 ceph-mon[36970]: pgmap v9835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:00 smithi042 ceph-mon[32775]: pgmap v9836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:00 smithi164 ceph-mon[36970]: pgmap v9836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:00 smithi163 ceph-mon[36873]: pgmap v9836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17511.524574 secs 2024-07-26T11:48:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17529.262718 secs 2024-07-26T11:48:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17511.524574 secs 2024-07-26T11:48:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17529.262718 secs 2024-07-26T11:48:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17511.524574 secs 2024-07-26T11:48:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17529.262718 secs 2024-07-26T11:48:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:02 smithi042 ceph-mon[32775]: pgmap v9837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:02 smithi164 ceph-mon[36970]: pgmap v9837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:02 smithi163 ceph-mon[36873]: pgmap v9837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:04 smithi042 ceph-mon[32775]: pgmap v9838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:04 smithi164 ceph-mon[36970]: pgmap v9838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:04 smithi163 ceph-mon[36873]: pgmap v9838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:48:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:48:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:48:06.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:48:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:48:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:48:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:48:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:48:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:48:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:48:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:48:06.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:48:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:06 smithi042 ceph-mon[32775]: pgmap v9839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:06 smithi042 ceph-mon[32775]: pgmap v9840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:48:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:06 smithi042 ceph-mon[32775]: pgmap v9841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:48:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17516.524700 secs 2024-07-26T11:48:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17534.262816 secs 2024-07-26T11:48:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:06 smithi164 ceph-mon[36970]: pgmap v9839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:06 smithi164 ceph-mon[36970]: pgmap v9840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:48:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:06 smithi164 ceph-mon[36970]: pgmap v9841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:48:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17516.524700 secs 2024-07-26T11:48:07.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17534.262816 secs 2024-07-26T11:48:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:06 smithi163 ceph-mon[36873]: pgmap v9839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:06 smithi163 ceph-mon[36873]: pgmap v9840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:48:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:06 smithi163 ceph-mon[36873]: pgmap v9841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:48:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17516.524700 secs 2024-07-26T11:48:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17534.262816 secs 2024-07-26T11:48:09.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:08 smithi042 ceph-mon[32775]: pgmap v9842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:48:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:08 smithi164 ceph-mon[36970]: pgmap v9842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:48:09.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:08 smithi163 ceph-mon[36873]: pgmap v9842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:48:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:10 smithi042 ceph-mon[32775]: pgmap v9843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:48:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:10 smithi164 ceph-mon[36970]: pgmap v9843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:48:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:10 smithi163 ceph-mon[36873]: pgmap v9843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:48:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17521.524824 secs 2024-07-26T11:48:12.295 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17539.262960 secs 2024-07-26T11:48:12.296 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17521.524824 secs 2024-07-26T11:48:12.296 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17539.262960 secs 2024-07-26T11:48:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17521.524824 secs 2024-07-26T11:48:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17539.262960 secs 2024-07-26T11:48:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:12 smithi042 ceph-mon[32775]: pgmap v9844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:48:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:12 smithi164 ceph-mon[36970]: pgmap v9844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:48:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:12 smithi163 ceph-mon[36873]: pgmap v9844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:48:13.863 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:48:13.861+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:48:13.863 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:48:13.862+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:13.863 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:48:13.862+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:48:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:14 smithi042 ceph-mon[32775]: pgmap v9845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:48:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:14 smithi164 ceph-mon[36970]: pgmap v9845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:48:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:14 smithi163 ceph-mon[36873]: pgmap v9845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:48:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:16 smithi042 ceph-mon[32775]: pgmap v9846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:48:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17526.524974 secs 2024-07-26T11:48:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17544.263082 secs 2024-07-26T11:48:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:16 smithi164 ceph-mon[36970]: pgmap v9846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:48:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17526.524974 secs 2024-07-26T11:48:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17544.263082 secs 2024-07-26T11:48:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:16 smithi163 ceph-mon[36873]: pgmap v9846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:48:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17526.524974 secs 2024-07-26T11:48:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17544.263082 secs 2024-07-26T11:48:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:18 smithi042 ceph-mon[32775]: pgmap v9847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:18 smithi164 ceph-mon[36970]: pgmap v9847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:19.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:18 smithi163 ceph-mon[36873]: pgmap v9847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:21.104 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:20 smithi042 ceph-mon[32775]: pgmap v9848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:20 smithi164 ceph-mon[36970]: pgmap v9848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:20 smithi163 ceph-mon[36873]: pgmap v9848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17531.525096 secs 2024-07-26T11:48:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17549.263206 secs 2024-07-26T11:48:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17531.525096 secs 2024-07-26T11:48:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17549.263206 secs 2024-07-26T11:48:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17531.525096 secs 2024-07-26T11:48:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17549.263206 secs 2024-07-26T11:48:23.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:22 smithi042 ceph-mon[32775]: pgmap v9849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:22 smithi164 ceph-mon[36970]: pgmap v9849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:22 smithi163 ceph-mon[36873]: pgmap v9849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:24 smithi042 ceph-mon[32775]: pgmap v9850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:48:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:24 smithi164 ceph-mon[36970]: pgmap v9850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:48:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:24 smithi163 ceph-mon[36873]: pgmap v9850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:48:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:26 smithi042 ceph-mon[32775]: pgmap v9851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17536.525250 secs 2024-07-26T11:48:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17554.263340 secs 2024-07-26T11:48:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:26 smithi164 ceph-mon[36970]: pgmap v9851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17536.525250 secs 2024-07-26T11:48:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17554.263340 secs 2024-07-26T11:48:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:26 smithi163 ceph-mon[36873]: pgmap v9851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17536.525250 secs 2024-07-26T11:48:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17554.263340 secs 2024-07-26T11:48:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:48:28.868+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:48:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:48:28.868+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:28.870 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:48:28.868+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:48:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:28 smithi042 ceph-mon[32775]: pgmap v9852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:28 smithi164 ceph-mon[36970]: pgmap v9852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:28 smithi163 ceph-mon[36873]: pgmap v9852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:30 smithi042 ceph-mon[32775]: pgmap v9853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:30 smithi164 ceph-mon[36970]: pgmap v9853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:30 smithi163 ceph-mon[36873]: pgmap v9853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17541.525349 secs 2024-07-26T11:48:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17559.263493 secs 2024-07-26T11:48:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17541.525349 secs 2024-07-26T11:48:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17559.263493 secs 2024-07-26T11:48:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17541.525349 secs 2024-07-26T11:48:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17559.263493 secs 2024-07-26T11:48:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:32 smithi042 ceph-mon[32775]: pgmap v9854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:32 smithi164 ceph-mon[36970]: pgmap v9854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:32 smithi163 ceph-mon[36873]: pgmap v9854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:35.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:34 smithi042 ceph-mon[32775]: pgmap v9855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:48:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:34 smithi164 ceph-mon[36970]: pgmap v9855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:48:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:34 smithi163 ceph-mon[36873]: pgmap v9855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:48:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:36 smithi042 ceph-mon[32775]: pgmap v9856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17546.525476 secs 2024-07-26T11:48:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17564.263617 secs 2024-07-26T11:48:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:36 smithi164 ceph-mon[36970]: pgmap v9856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17546.525476 secs 2024-07-26T11:48:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17564.263617 secs 2024-07-26T11:48:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:36 smithi163 ceph-mon[36873]: pgmap v9856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17546.525476 secs 2024-07-26T11:48:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17564.263617 secs 2024-07-26T11:48:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:38 smithi042 ceph-mon[32775]: pgmap v9857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:38 smithi164 ceph-mon[36970]: pgmap v9857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:38 smithi163 ceph-mon[36873]: pgmap v9857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:40 smithi042 ceph-mon[32775]: pgmap v9858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:40 smithi164 ceph-mon[36970]: pgmap v9858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:40 smithi163 ceph-mon[36873]: pgmap v9858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17551.525604 secs 2024-07-26T11:48:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17569.263743 secs 2024-07-26T11:48:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17551.525604 secs 2024-07-26T11:48:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17569.263743 secs 2024-07-26T11:48:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17551.525604 secs 2024-07-26T11:48:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17569.263743 secs 2024-07-26T11:48:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:42 smithi164 ceph-mon[36970]: pgmap v9859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:42 smithi163 ceph-mon[36873]: pgmap v9859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:42 smithi042 ceph-mon[32775]: pgmap v9859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:43.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:48:43.869+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:48:43.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:48:43.869+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:43.871 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:48:43.869+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:48:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:44 smithi164 ceph-mon[36970]: pgmap v9860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:48:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:44 smithi163 ceph-mon[36873]: pgmap v9860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:48:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:44 smithi042 ceph-mon[32775]: pgmap v9860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:48:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:46 smithi164 ceph-mon[36970]: pgmap v9861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17556.525733 secs 2024-07-26T11:48:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17574.263868 secs 2024-07-26T11:48:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:46 smithi163 ceph-mon[36873]: pgmap v9861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17556.525733 secs 2024-07-26T11:48:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17574.263868 secs 2024-07-26T11:48:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:46 smithi042 ceph-mon[32775]: pgmap v9861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17556.525733 secs 2024-07-26T11:48:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17574.263868 secs 2024-07-26T11:48:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:48 smithi164 ceph-mon[36970]: pgmap v9862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:48 smithi163 ceph-mon[36873]: pgmap v9862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:48 smithi042 ceph-mon[32775]: pgmap v9862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:50 smithi164 ceph-mon[36970]: pgmap v9863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:50 smithi163 ceph-mon[36873]: pgmap v9863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:50 smithi042 ceph-mon[32775]: pgmap v9863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17561.525875 secs 2024-07-26T11:48:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17579.264005 secs 2024-07-26T11:48:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17561.525875 secs 2024-07-26T11:48:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17579.264005 secs 2024-07-26T11:48:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17561.525875 secs 2024-07-26T11:48:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17579.264005 secs 2024-07-26T11:48:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:52 smithi164 ceph-mon[36970]: pgmap v9864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:52 smithi163 ceph-mon[36873]: pgmap v9864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:52 smithi042 ceph-mon[32775]: pgmap v9864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:54 smithi164 ceph-mon[36970]: pgmap v9865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:48:55.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:54 smithi163 ceph-mon[36873]: pgmap v9865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:48:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:54 smithi042 ceph-mon[32775]: pgmap v9865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:48:57.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:56 smithi042 ceph-mon[32775]: pgmap v9866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:57.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17566.526003 secs 2024-07-26T11:48:57.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17584.264116 secs 2024-07-26T11:48:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:56 smithi164 ceph-mon[36970]: pgmap v9866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17566.526003 secs 2024-07-26T11:48:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17584.264116 secs 2024-07-26T11:48:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:56 smithi163 ceph-mon[36873]: pgmap v9866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:48:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17566.526003 secs 2024-07-26T11:48:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17584.264116 secs 2024-07-26T11:48:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:48:58.871+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:48:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:48:58.871+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:48:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:48:58.871+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:48:58 smithi164 ceph-mon[36970]: pgmap v9867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:59.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:48:58 smithi163 ceph-mon[36873]: pgmap v9867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:48:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:48:58 smithi042 ceph-mon[32775]: pgmap v9867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:00 smithi164 ceph-mon[36970]: pgmap v9868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:00 smithi163 ceph-mon[36873]: pgmap v9868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:00 smithi042 ceph-mon[32775]: pgmap v9868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17571.526104 secs 2024-07-26T11:49:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17589.264212 secs 2024-07-26T11:49:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17571.526104 secs 2024-07-26T11:49:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17589.264212 secs 2024-07-26T11:49:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17571.526104 secs 2024-07-26T11:49:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17589.264212 secs 2024-07-26T11:49:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:02 smithi164 ceph-mon[36970]: pgmap v9869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:02 smithi163 ceph-mon[36873]: pgmap v9869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:02 smithi042 ceph-mon[32775]: pgmap v9869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:04 smithi164 ceph-mon[36970]: pgmap v9870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:49:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:04 smithi163 ceph-mon[36873]: pgmap v9870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:49:05.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:04 smithi042 ceph-mon[32775]: pgmap v9870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:49:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:05 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:49:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:05 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:49:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:05 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:49:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:06 smithi164 ceph-mon[36970]: pgmap v9871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17576.526176 secs 2024-07-26T11:49:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:49:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:49:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:49:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17594.264370 secs 2024-07-26T11:49:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:06 smithi163 ceph-mon[36873]: pgmap v9871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17576.526176 secs 2024-07-26T11:49:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:49:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:49:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:49:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17594.264370 secs 2024-07-26T11:49:07.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:06 smithi042 ceph-mon[32775]: pgmap v9871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:07.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17576.526176 secs 2024-07-26T11:49:07.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:49:07.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:49:07.298 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:49:07.298 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17594.264370 secs 2024-07-26T11:49:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:07 smithi164 ceph-mon[36970]: pgmap v9872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:49:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:07 smithi164 ceph-mon[36970]: pgmap v9873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:49:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:07 smithi163 ceph-mon[36873]: pgmap v9872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:49:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:07 smithi163 ceph-mon[36873]: pgmap v9873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:49:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:07 smithi042 ceph-mon[32775]: pgmap v9872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:49:08.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:07 smithi042 ceph-mon[32775]: pgmap v9873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:49:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:09 smithi164 ceph-mon[36970]: pgmap v9874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:49:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:09 smithi163 ceph-mon[36873]: pgmap v9874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:49:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:09 smithi042 ceph-mon[32775]: pgmap v9874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:49:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:11 smithi164 ceph-mon[36970]: pgmap v9875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:49:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17581.526302 secs 2024-07-26T11:49:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17599.264475 secs 2024-07-26T11:49:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:11 smithi163 ceph-mon[36873]: pgmap v9875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:49:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17581.526302 secs 2024-07-26T11:49:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17599.264475 secs 2024-07-26T11:49:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:11 smithi042 ceph-mon[32775]: pgmap v9875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:49:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17581.526302 secs 2024-07-26T11:49:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17599.264475 secs 2024-07-26T11:49:13.873 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:49:13.872+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:49:13.873 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:49:13.872+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:13.873 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:49:13.872+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:49:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:13 smithi164 ceph-mon[36970]: pgmap v9876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:49:14.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:13 smithi163 ceph-mon[36873]: pgmap v9876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:49:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:13 smithi042 ceph-mon[32775]: pgmap v9876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:49:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:15 smithi164 ceph-mon[36970]: pgmap v9877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:49:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:15 smithi163 ceph-mon[36873]: pgmap v9877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:49:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:15 smithi042 ceph-mon[32775]: pgmap v9877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:49:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17586.526429 secs 2024-07-26T11:49:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17604.264596 secs 2024-07-26T11:49:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17586.526429 secs 2024-07-26T11:49:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17604.264596 secs 2024-07-26T11:49:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17586.526429 secs 2024-07-26T11:49:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17604.264596 secs 2024-07-26T11:49:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:17 smithi163 ceph-mon[36873]: pgmap v9878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:49:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:17 smithi042 ceph-mon[32775]: pgmap v9878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:49:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:17 smithi164 ceph-mon[36970]: pgmap v9878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:49:20.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:19 smithi163 ceph-mon[36873]: pgmap v9879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:49:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:19 smithi042 ceph-mon[32775]: pgmap v9879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:49:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:19 smithi164 ceph-mon[36970]: pgmap v9879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:49:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:21 smithi163 ceph-mon[36873]: pgmap v9880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17591.526546 secs 2024-07-26T11:49:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17609.264720 secs 2024-07-26T11:49:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:21 smithi042 ceph-mon[32775]: pgmap v9880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17591.526546 secs 2024-07-26T11:49:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17609.264720 secs 2024-07-26T11:49:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:21 smithi164 ceph-mon[36970]: pgmap v9880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17591.526546 secs 2024-07-26T11:49:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17609.264720 secs 2024-07-26T11:49:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:23 smithi163 ceph-mon[36873]: pgmap v9881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:23 smithi042 ceph-mon[32775]: pgmap v9881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:23 smithi164 ceph-mon[36970]: pgmap v9881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:25 smithi163 ceph-mon[36873]: pgmap v9882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:25 smithi042 ceph-mon[32775]: pgmap v9882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:25 smithi164 ceph-mon[36970]: pgmap v9882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17596.526674 secs 2024-07-26T11:49:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17614.264859 secs 2024-07-26T11:49:27.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17596.526674 secs 2024-07-26T11:49:27.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17614.264859 secs 2024-07-26T11:49:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17596.526674 secs 2024-07-26T11:49:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17614.264859 secs 2024-07-26T11:49:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:27 smithi163 ceph-mon[36873]: pgmap v9883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:27 smithi042 ceph-mon[32775]: pgmap v9883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:27 smithi164 ceph-mon[36970]: pgmap v9883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:28.879 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:49:28.878+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:49:28.879 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:49:28.878+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:28.879 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:49:28.878+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:49:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:29 smithi163 ceph-mon[36873]: pgmap v9884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:49:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:29 smithi042 ceph-mon[32775]: pgmap v9884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:49:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:29 smithi164 ceph-mon[36970]: pgmap v9884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:49:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:32 smithi163 ceph-mon[36873]: pgmap v9885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17601.526793 secs 2024-07-26T11:49:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17619.264989 secs 2024-07-26T11:49:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:32 smithi042 ceph-mon[32775]: pgmap v9885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17601.526793 secs 2024-07-26T11:49:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17619.264989 secs 2024-07-26T11:49:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:32 smithi164 ceph-mon[36970]: pgmap v9885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17601.526793 secs 2024-07-26T11:49:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17619.264989 secs 2024-07-26T11:49:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:34 smithi163 ceph-mon[36873]: pgmap v9886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:34 smithi042 ceph-mon[32775]: pgmap v9886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:34 smithi164 ceph-mon[36970]: pgmap v9886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:35 smithi164 ceph-mon[36970]: pgmap v9887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:35 smithi163 ceph-mon[36873]: pgmap v9887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:35.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:35 smithi042 ceph-mon[32775]: pgmap v9887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17606.526944 secs 2024-07-26T11:49:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:37 smithi042 ceph-mon[32775]: pgmap v9888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17624.265111 secs 2024-07-26T11:49:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17606.526944 secs 2024-07-26T11:49:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:37 smithi164 ceph-mon[36970]: pgmap v9888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17624.265111 secs 2024-07-26T11:49:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17606.526944 secs 2024-07-26T11:49:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:37 smithi163 ceph-mon[36873]: pgmap v9888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17624.265111 secs 2024-07-26T11:49:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:39 smithi042 ceph-mon[32775]: pgmap v9889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:49:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:39 smithi164 ceph-mon[36970]: pgmap v9889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:49:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:39 smithi163 ceph-mon[36873]: pgmap v9889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:49:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:41 smithi042 ceph-mon[32775]: pgmap v9890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:41.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:41 smithi164 ceph-mon[36970]: pgmap v9890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:41.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:41 smithi163 ceph-mon[36873]: pgmap v9890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17611.527090 secs 2024-07-26T11:49:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17629.265236 secs 2024-07-26T11:49:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17611.527090 secs 2024-07-26T11:49:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17629.265236 secs 2024-07-26T11:49:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17611.527090 secs 2024-07-26T11:49:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17629.265236 secs 2024-07-26T11:49:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:43 smithi042 ceph-mon[32775]: pgmap v9891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:43 smithi164 ceph-mon[36970]: pgmap v9891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:43 smithi163 ceph-mon[36873]: pgmap v9891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:43.880 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:49:43.879+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:49:43.880 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:49:43.879+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:43.880 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:49:43.879+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:49:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:45 smithi042 ceph-mon[32775]: pgmap v9892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:45 smithi164 ceph-mon[36970]: pgmap v9892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:45 smithi163 ceph-mon[36873]: pgmap v9892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17616.527262 secs 2024-07-26T11:49:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:47 smithi042 ceph-mon[32775]: pgmap v9893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17634.265352 secs 2024-07-26T11:49:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17616.527262 secs 2024-07-26T11:49:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:47 smithi164 ceph-mon[36970]: pgmap v9893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17634.265352 secs 2024-07-26T11:49:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17616.527262 secs 2024-07-26T11:49:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:47 smithi163 ceph-mon[36873]: pgmap v9893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17634.265352 secs 2024-07-26T11:49:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:49 smithi042 ceph-mon[32775]: pgmap v9894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:49:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:49 smithi164 ceph-mon[36970]: pgmap v9894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:49:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:49 smithi163 ceph-mon[36873]: pgmap v9894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:49:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:51 smithi042 ceph-mon[32775]: pgmap v9895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:51 smithi164 ceph-mon[36970]: pgmap v9895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:51 smithi163 ceph-mon[36873]: pgmap v9895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17621.527405 secs 2024-07-26T11:49:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17639.265487 secs 2024-07-26T11:49:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17621.527405 secs 2024-07-26T11:49:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17639.265487 secs 2024-07-26T11:49:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17621.527405 secs 2024-07-26T11:49:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17639.265487 secs 2024-07-26T11:49:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:53 smithi042 ceph-mon[32775]: pgmap v9896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:53 smithi164 ceph-mon[36970]: pgmap v9896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:53.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:53 smithi163 ceph-mon[36873]: pgmap v9896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:55 smithi042 ceph-mon[32775]: pgmap v9897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:55 smithi164 ceph-mon[36970]: pgmap v9897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:55.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:55 smithi163 ceph-mon[36873]: pgmap v9897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:49:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17626.527530 secs 2024-07-26T11:49:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:57 smithi042 ceph-mon[32775]: pgmap v9898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17644.265619 secs 2024-07-26T11:49:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17626.527530 secs 2024-07-26T11:49:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:57 smithi164 ceph-mon[36970]: pgmap v9898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17644.265619 secs 2024-07-26T11:49:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17626.527530 secs 2024-07-26T11:49:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:57 smithi163 ceph-mon[36873]: pgmap v9898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:49:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17644.265619 secs 2024-07-26T11:49:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:49:58.884+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:49:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:49:58.884+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:49:58.884+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:49:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:49:59 smithi042 ceph-mon[32775]: pgmap v9899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:49:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:49:59 smithi164 ceph-mon[36970]: pgmap v9899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:49:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:49:59 smithi163 ceph-mon[36873]: pgmap v9899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:50:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:50:00.644+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006480 con 0x7ff680102000 2024-07-26T11:50:00.645 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:50:00.644+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:50:00.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:50:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:50:00.644+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:50:00.646 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:50:00.644+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6700e9050 con 0x7ff680102000 2024-07-26T11:50:00.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:50:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T11:50:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:01 smithi042 ceph-mon[32775]: pgmap v9900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:01 smithi164 ceph-mon[36970]: pgmap v9900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:01.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:01 smithi163 ceph-mon[36873]: pgmap v9900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17631.527600 secs 2024-07-26T11:50:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17649.265745 secs 2024-07-26T11:50:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17631.527600 secs 2024-07-26T11:50:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17649.265745 secs 2024-07-26T11:50:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17631.527600 secs 2024-07-26T11:50:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17649.265745 secs 2024-07-26T11:50:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:03 smithi042 ceph-mon[32775]: pgmap v9901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:03 smithi164 ceph-mon[36970]: pgmap v9901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:03 smithi163 ceph-mon[36873]: pgmap v9901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:05 smithi042 ceph-mon[32775]: pgmap v9902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:05 smithi164 ceph-mon[36970]: pgmap v9902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:05 smithi163 ceph-mon[36873]: pgmap v9902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:06 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:50:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:06 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:50:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:06 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:50:07.424 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17636.527731 secs 2024-07-26T11:50:07.424 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:07 smithi042 ceph-mon[32775]: pgmap v9903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:07.425 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17654.265838 secs 2024-07-26T11:50:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17636.527731 secs 2024-07-26T11:50:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:07 smithi164 ceph-mon[36970]: pgmap v9903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17654.265838 secs 2024-07-26T11:50:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17636.527731 secs 2024-07-26T11:50:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:07 smithi163 ceph-mon[36873]: pgmap v9903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17654.265838 secs 2024-07-26T11:50:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:09 smithi042 ceph-mon[32775]: pgmap v9904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:50:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:09 smithi164 ceph-mon[36970]: pgmap v9904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:50:09.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:09 smithi163 ceph-mon[36873]: pgmap v9904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:50:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:50:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:50:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:50:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:50:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:11 smithi042 ceph-mon[32775]: pgmap v9905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:11.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:11 smithi042 ceph-mon[32775]: pgmap v9906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:11.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:50:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:50:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:50:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:50:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:50:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:11 smithi164 ceph-mon[36970]: pgmap v9905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:11.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:11 smithi164 ceph-mon[36970]: pgmap v9906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:11.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:50:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:50:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:50:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:50:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:50:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:11 smithi163 ceph-mon[36873]: pgmap v9905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:11 smithi163 ceph-mon[36873]: pgmap v9906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:11.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:50:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17641.527861 secs 2024-07-26T11:50:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17659.265975 secs 2024-07-26T11:50:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17641.527861 secs 2024-07-26T11:50:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17659.265975 secs 2024-07-26T11:50:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17641.527861 secs 2024-07-26T11:50:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17659.265975 secs 2024-07-26T11:50:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:13 smithi042 ceph-mon[32775]: pgmap v9907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:50:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:13 smithi164 ceph-mon[36970]: pgmap v9907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:50:13.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:13 smithi163 ceph-mon[36873]: pgmap v9907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:50:13.887 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:50:13.886+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:50:13.887 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:50:13.886+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:50:13.887 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:50:13.886+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:15 smithi042 ceph-mon[32775]: pgmap v9908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 3 op/s 2024-07-26T11:50:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:15 smithi164 ceph-mon[36970]: pgmap v9908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 3 op/s 2024-07-26T11:50:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:15 smithi163 ceph-mon[36873]: pgmap v9908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 3 op/s 2024-07-26T11:50:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:17 smithi042 ceph-mon[32775]: pgmap v9909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17646.528005 secs 2024-07-26T11:50:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17664.266102 secs 2024-07-26T11:50:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:17 smithi164 ceph-mon[36970]: pgmap v9909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17646.528005 secs 2024-07-26T11:50:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17664.266102 secs 2024-07-26T11:50:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:17 smithi163 ceph-mon[36873]: pgmap v9909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:50:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17646.528005 secs 2024-07-26T11:50:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17664.266102 secs 2024-07-26T11:50:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:19 smithi042 ceph-mon[32775]: pgmap v9910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:50:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:19 smithi164 ceph-mon[36970]: pgmap v9910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:50:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:19 smithi163 ceph-mon[36873]: pgmap v9910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:50:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:21 smithi042 ceph-mon[32775]: pgmap v9911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:50:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:21 smithi164 ceph-mon[36970]: pgmap v9911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:50:21.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:21 smithi163 ceph-mon[36873]: pgmap v9911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:50:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17651.528128 secs 2024-07-26T11:50:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17669.266223 secs 2024-07-26T11:50:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17651.528128 secs 2024-07-26T11:50:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17669.266223 secs 2024-07-26T11:50:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17651.528128 secs 2024-07-26T11:50:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17669.266223 secs 2024-07-26T11:50:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:23 smithi042 ceph-mon[32775]: pgmap v9912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:23 smithi164 ceph-mon[36970]: pgmap v9912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:23 smithi163 ceph-mon[36873]: pgmap v9912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:25 smithi042 ceph-mon[32775]: pgmap v9913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:50:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:25 smithi164 ceph-mon[36970]: pgmap v9913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:50:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:25 smithi163 ceph-mon[36873]: pgmap v9913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:50:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:27 smithi042 ceph-mon[32775]: pgmap v9914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17656.528249 secs 2024-07-26T11:50:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17674.266346 secs 2024-07-26T11:50:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:27 smithi164 ceph-mon[36970]: pgmap v9914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17656.528249 secs 2024-07-26T11:50:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17674.266346 secs 2024-07-26T11:50:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:27 smithi163 ceph-mon[36873]: pgmap v9914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17656.528249 secs 2024-07-26T11:50:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17674.266346 secs 2024-07-26T11:50:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:50:28.892+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:50:28.894 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:50:28.892+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:28.894 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:50:28.892+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:50:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:29 smithi042 ceph-mon[32775]: pgmap v9915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:29 smithi164 ceph-mon[36970]: pgmap v9915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:29 smithi163 ceph-mon[36873]: pgmap v9915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:31.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:31 smithi163 ceph-mon[36873]: pgmap v9916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:31 smithi042 ceph-mon[32775]: pgmap v9916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:31 smithi164 ceph-mon[36970]: pgmap v9916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17661.528372 secs 2024-07-26T11:50:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17679.266471 secs 2024-07-26T11:50:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17661.528372 secs 2024-07-26T11:50:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17679.266471 secs 2024-07-26T11:50:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17661.528372 secs 2024-07-26T11:50:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17679.266471 secs 2024-07-26T11:50:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:33 smithi042 ceph-mon[32775]: pgmap v9917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:33 smithi164 ceph-mon[36970]: pgmap v9917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:33 smithi163 ceph-mon[36873]: pgmap v9917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:35 smithi042 ceph-mon[32775]: pgmap v9918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:50:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:35 smithi164 ceph-mon[36970]: pgmap v9918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:50:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:35 smithi163 ceph-mon[36873]: pgmap v9918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:50:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:37 smithi042 ceph-mon[32775]: pgmap v9919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17666.528440 secs 2024-07-26T11:50:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17684.266594 secs 2024-07-26T11:50:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:37 smithi164 ceph-mon[36970]: pgmap v9919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17666.528440 secs 2024-07-26T11:50:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17684.266594 secs 2024-07-26T11:50:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:37 smithi163 ceph-mon[36873]: pgmap v9919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17666.528440 secs 2024-07-26T11:50:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17684.266594 secs 2024-07-26T11:50:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:39 smithi042 ceph-mon[32775]: pgmap v9920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:39 smithi164 ceph-mon[36970]: pgmap v9920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:39 smithi163 ceph-mon[36873]: pgmap v9920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:40 smithi164 ceph-mon[36970]: pgmap v9921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:40 smithi163 ceph-mon[36873]: pgmap v9921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:40 smithi042 ceph-mon[32775]: pgmap v9921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17671.528556 secs 2024-07-26T11:50:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17671.528556 secs 2024-07-26T11:50:41.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17671.528556 secs 2024-07-26T11:50:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17689.266717 secs 2024-07-26T11:50:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:42 smithi164 ceph-mon[36970]: pgmap v9922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17689.266717 secs 2024-07-26T11:50:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:42 smithi163 ceph-mon[36873]: pgmap v9922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17689.266717 secs 2024-07-26T11:50:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:42 smithi042 ceph-mon[32775]: pgmap v9922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:50:43.893+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:50:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:50:43.893+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:50:43.893+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:50:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:45 smithi042 ceph-mon[32775]: pgmap v9923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:45 smithi164 ceph-mon[36970]: pgmap v9923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:45 smithi163 ceph-mon[36873]: pgmap v9923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:47 smithi042 ceph-mon[32775]: pgmap v9924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17676.528646 secs 2024-07-26T11:50:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17694.266837 secs 2024-07-26T11:50:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:47 smithi164 ceph-mon[36970]: pgmap v9924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17676.528646 secs 2024-07-26T11:50:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17694.266837 secs 2024-07-26T11:50:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:47 smithi163 ceph-mon[36873]: pgmap v9924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17676.528646 secs 2024-07-26T11:50:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17694.266837 secs 2024-07-26T11:50:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:49 smithi042 ceph-mon[32775]: pgmap v9925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:49 smithi164 ceph-mon[36970]: pgmap v9925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:49 smithi163 ceph-mon[36873]: pgmap v9925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:51 smithi042 ceph-mon[32775]: pgmap v9926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:51 smithi164 ceph-mon[36970]: pgmap v9926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:51 smithi163 ceph-mon[36873]: pgmap v9926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17681.528708 secs 2024-07-26T11:50:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17699.266969 secs 2024-07-26T11:50:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17681.528708 secs 2024-07-26T11:50:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17699.266969 secs 2024-07-26T11:50:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17681.528708 secs 2024-07-26T11:50:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17699.266969 secs 2024-07-26T11:50:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:53 smithi042 ceph-mon[32775]: pgmap v9927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:53 smithi164 ceph-mon[36970]: pgmap v9927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:53 smithi163 ceph-mon[36873]: pgmap v9927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:55 smithi042 ceph-mon[32775]: pgmap v9928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:55 smithi164 ceph-mon[36970]: pgmap v9928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:55 smithi163 ceph-mon[36873]: pgmap v9928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:50:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:57 smithi042 ceph-mon[32775]: pgmap v9929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17686.528836 secs 2024-07-26T11:50:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17704.267081 secs 2024-07-26T11:50:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:57 smithi164 ceph-mon[36970]: pgmap v9929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17686.528836 secs 2024-07-26T11:50:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17704.267081 secs 2024-07-26T11:50:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:57 smithi163 ceph-mon[36873]: pgmap v9929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17686.528836 secs 2024-07-26T11:50:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17704.267081 secs 2024-07-26T11:50:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:50:58 smithi164 ceph-mon[36970]: pgmap v9930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:50:58 smithi163 ceph-mon[36873]: pgmap v9930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:50:58 smithi042 ceph-mon[32775]: pgmap v9930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:50:58.899 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:50:58.898+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:50:58.900 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:50:58.898+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:58.900 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:50:58.898+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:51:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:01 smithi042 ceph-mon[32775]: pgmap v9931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:01 smithi164 ceph-mon[36970]: pgmap v9931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:01 smithi163 ceph-mon[36873]: pgmap v9931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17691.528955 secs 2024-07-26T11:51:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17709.267147 secs 2024-07-26T11:51:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17691.528955 secs 2024-07-26T11:51:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17709.267147 secs 2024-07-26T11:51:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17691.528955 secs 2024-07-26T11:51:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17709.267147 secs 2024-07-26T11:51:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:03 smithi042 ceph-mon[32775]: pgmap v9932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:03 smithi164 ceph-mon[36970]: pgmap v9932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:03 smithi163 ceph-mon[36873]: pgmap v9932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:05 smithi042 ceph-mon[32775]: pgmap v9933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:05 smithi164 ceph-mon[36970]: pgmap v9933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:05.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:05 smithi163 ceph-mon[36873]: pgmap v9933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:07 smithi042 ceph-mon[32775]: pgmap v9934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17696.529056 secs 2024-07-26T11:51:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17714.267270 secs 2024-07-26T11:51:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:07 smithi164 ceph-mon[36970]: pgmap v9934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17696.529056 secs 2024-07-26T11:51:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17714.267270 secs 2024-07-26T11:51:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:07 smithi163 ceph-mon[36873]: pgmap v9934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17696.529056 secs 2024-07-26T11:51:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17714.267270 secs 2024-07-26T11:51:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:09 smithi042 ceph-mon[32775]: pgmap v9935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:09 smithi164 ceph-mon[36970]: pgmap v9935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:09.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:09 smithi163 ceph-mon[36873]: pgmap v9935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:11 smithi163 ceph-mon[36873]: pgmap v9936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:51:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:11.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:11 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:11.345 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:11 smithi164 ceph-mon[36970]: pgmap v9936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:11.345 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:51:11.346 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:11.346 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:11.346 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:11.346 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:11 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:11 smithi042 ceph-mon[32775]: pgmap v9936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:51:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:11.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:11 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:12 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:51:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:12 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:51:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17701.529169 secs 2024-07-26T11:51:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17719.267421 secs 2024-07-26T11:51:12.421 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:12 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:51:12.421 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:12 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:51:12.421 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17701.529169 secs 2024-07-26T11:51:12.421 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17719.267421 secs 2024-07-26T11:51:12.467 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:12 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T11:51:12.467 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:12 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T11:51:12.467 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17701.529169 secs 2024-07-26T11:51:12.467 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17719.267421 secs 2024-07-26T11:51:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:13 smithi042 ceph-mon[32775]: pgmap v9937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:13 smithi164 ceph-mon[36970]: pgmap v9937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:13 smithi163 ceph-mon[36873]: pgmap v9937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:51:13.899+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:51:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:51:13.899+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:51:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:51:13.899+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:51:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:51:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:51:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:51:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:51:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:51:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:51:15.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:51:15.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:51:15.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:51:15.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: pgmap v9938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:51:15.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: pgmap v9939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 3 op/s 2024-07-26T11:51:15.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:51:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:51:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:51:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:51:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:51:15.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:51:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:51:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:51:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:51:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:51:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:51:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: pgmap v9938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:51:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: pgmap v9939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 3 op/s 2024-07-26T11:51:15.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:51:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:51:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:51:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T11:51:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T11:51:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:51:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:51:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:15.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T11:51:15.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T11:51:15.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:51:15.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:51:15.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: pgmap v9938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:51:15.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: pgmap v9939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 3 op/s 2024-07-26T11:51:15.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:51:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:17 smithi042 ceph-mon[32775]: pgmap v9940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:51:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17706.529269 secs 2024-07-26T11:51:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17724.267540 secs 2024-07-26T11:51:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:17 smithi164 ceph-mon[36970]: pgmap v9940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:51:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17706.529269 secs 2024-07-26T11:51:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17724.267540 secs 2024-07-26T11:51:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:17 smithi163 ceph-mon[36873]: pgmap v9940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:51:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17706.529269 secs 2024-07-26T11:51:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17724.267540 secs 2024-07-26T11:51:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:19 smithi042 ceph-mon[32775]: pgmap v9941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:51:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:19 smithi164 ceph-mon[36970]: pgmap v9941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:51:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:19 smithi163 ceph-mon[36873]: pgmap v9941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:51:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:21 smithi042 ceph-mon[32775]: pgmap v9942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:51:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:21 smithi164 ceph-mon[36970]: pgmap v9942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:51:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:21 smithi163 ceph-mon[36873]: pgmap v9942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:51:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17711.529389 secs 2024-07-26T11:51:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17729.267662 secs 2024-07-26T11:51:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17711.529389 secs 2024-07-26T11:51:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17729.267662 secs 2024-07-26T11:51:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17711.529389 secs 2024-07-26T11:51:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17729.267662 secs 2024-07-26T11:51:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:23 smithi042 ceph-mon[32775]: pgmap v9943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:51:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:23 smithi164 ceph-mon[36970]: pgmap v9943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:51:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:23 smithi163 ceph-mon[36873]: pgmap v9943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T11:51:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:25 smithi042 ceph-mon[32775]: pgmap v9944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:51:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:25 smithi164 ceph-mon[36970]: pgmap v9944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:51:25.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:25 smithi163 ceph-mon[36873]: pgmap v9944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:51:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:27 smithi042 ceph-mon[32775]: pgmap v9945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17716.529512 secs 2024-07-26T11:51:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17734.267783 secs 2024-07-26T11:51:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:27 smithi164 ceph-mon[36970]: pgmap v9945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17716.529512 secs 2024-07-26T11:51:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17734.267783 secs 2024-07-26T11:51:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:27 smithi163 ceph-mon[36873]: pgmap v9945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17716.529512 secs 2024-07-26T11:51:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17734.267783 secs 2024-07-26T11:51:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:51:28.905+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:51:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:51:28.905+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:51:28.905+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:51:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:29 smithi042 ceph-mon[32775]: pgmap v9946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:29 smithi164 ceph-mon[36970]: pgmap v9946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:29 smithi163 ceph-mon[36873]: pgmap v9946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:31 smithi042 ceph-mon[32775]: pgmap v9947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:31 smithi164 ceph-mon[36970]: pgmap v9947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:31 smithi163 ceph-mon[36873]: pgmap v9947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17721.529629 secs 2024-07-26T11:51:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17739.267906 secs 2024-07-26T11:51:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17721.529629 secs 2024-07-26T11:51:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17739.267906 secs 2024-07-26T11:51:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17721.529629 secs 2024-07-26T11:51:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17739.267906 secs 2024-07-26T11:51:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:33 smithi042 ceph-mon[32775]: pgmap v9948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:33 smithi164 ceph-mon[36970]: pgmap v9948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:33 smithi163 ceph-mon[36873]: pgmap v9948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:34 smithi164 ceph-mon[36970]: pgmap v9949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:51:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:34 smithi163 ceph-mon[36873]: pgmap v9949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:51:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:34 smithi042 ceph-mon[32775]: pgmap v9949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:51:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:37 smithi042 ceph-mon[32775]: pgmap v9950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17726.529755 secs 2024-07-26T11:51:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17744.268026 secs 2024-07-26T11:51:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:37 smithi164 ceph-mon[36970]: pgmap v9950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17726.529755 secs 2024-07-26T11:51:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17744.268026 secs 2024-07-26T11:51:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:37 smithi163 ceph-mon[36873]: pgmap v9950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17726.529755 secs 2024-07-26T11:51:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17744.268026 secs 2024-07-26T11:51:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:39 smithi042 ceph-mon[32775]: pgmap v9951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:39 smithi164 ceph-mon[36970]: pgmap v9951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:39.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:39 smithi163 ceph-mon[36873]: pgmap v9951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:41 smithi042 ceph-mon[32775]: pgmap v9952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:41 smithi164 ceph-mon[36970]: pgmap v9952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:41.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:41 smithi163 ceph-mon[36873]: pgmap v9952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17731.529875 secs 2024-07-26T11:51:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17749.268170 secs 2024-07-26T11:51:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17731.529875 secs 2024-07-26T11:51:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17749.268170 secs 2024-07-26T11:51:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17731.529875 secs 2024-07-26T11:51:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17749.268170 secs 2024-07-26T11:51:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:43 smithi042 ceph-mon[32775]: pgmap v9953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:43 smithi164 ceph-mon[36970]: pgmap v9953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:43 smithi163 ceph-mon[36873]: pgmap v9953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:51:43.905+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:51:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:51:43.905+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:51:43.905+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:51:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:45 smithi042 ceph-mon[32775]: pgmap v9954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:45 smithi164 ceph-mon[36970]: pgmap v9954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:45 smithi163 ceph-mon[36873]: pgmap v9954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:47 smithi042 ceph-mon[32775]: pgmap v9955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17736.530018 secs 2024-07-26T11:51:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17754.268308 secs 2024-07-26T11:51:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:47 smithi164 ceph-mon[36970]: pgmap v9955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17736.530018 secs 2024-07-26T11:51:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17754.268308 secs 2024-07-26T11:51:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:47 smithi163 ceph-mon[36873]: pgmap v9955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17736.530018 secs 2024-07-26T11:51:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17754.268308 secs 2024-07-26T11:51:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:49 smithi042 ceph-mon[32775]: pgmap v9956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:49 smithi164 ceph-mon[36970]: pgmap v9956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:49 smithi163 ceph-mon[36873]: pgmap v9956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:51 smithi042 ceph-mon[32775]: pgmap v9957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:51 smithi164 ceph-mon[36970]: pgmap v9957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:51.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:51 smithi163 ceph-mon[36873]: pgmap v9957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17741.530156 secs 2024-07-26T11:51:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17759.268452 secs 2024-07-26T11:51:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17741.530156 secs 2024-07-26T11:51:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17759.268452 secs 2024-07-26T11:51:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17741.530156 secs 2024-07-26T11:51:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17759.268452 secs 2024-07-26T11:51:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:53 smithi042 ceph-mon[32775]: pgmap v9958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:53 smithi164 ceph-mon[36970]: pgmap v9958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:53 smithi163 ceph-mon[36873]: pgmap v9958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:54 smithi042 ceph-mon[32775]: pgmap v9959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:54 smithi164 ceph-mon[36970]: pgmap v9959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:54 smithi163 ceph-mon[36873]: pgmap v9959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:51:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:57 smithi042 ceph-mon[32775]: pgmap v9960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17746.530304 secs 2024-07-26T11:51:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17764.268593 secs 2024-07-26T11:51:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:57 smithi164 ceph-mon[36970]: pgmap v9960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17746.530304 secs 2024-07-26T11:51:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17764.268593 secs 2024-07-26T11:51:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:57 smithi163 ceph-mon[36873]: pgmap v9960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17746.530304 secs 2024-07-26T11:51:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17764.268593 secs 2024-07-26T11:51:58.912 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:51:58.910+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:51:58.912 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:51:58.911+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:51:58.912 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:51:58.911+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:51:59 smithi042 ceph-mon[32775]: pgmap v9961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:51:59 smithi164 ceph-mon[36970]: pgmap v9961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:51:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:51:59 smithi163 ceph-mon[36873]: pgmap v9961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:01 smithi042 ceph-mon[32775]: pgmap v9962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:01 smithi164 ceph-mon[36970]: pgmap v9962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:01 smithi163 ceph-mon[36873]: pgmap v9962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:02.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17751.530424 secs 2024-07-26T11:52:02.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17769.268750 secs 2024-07-26T11:52:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17751.530424 secs 2024-07-26T11:52:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17769.268750 secs 2024-07-26T11:52:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17751.530424 secs 2024-07-26T11:52:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17769.268750 secs 2024-07-26T11:52:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:03 smithi042 ceph-mon[32775]: pgmap v9963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:03 smithi164 ceph-mon[36970]: pgmap v9963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:03 smithi163 ceph-mon[36873]: pgmap v9963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:05 smithi042 ceph-mon[32775]: pgmap v9964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:05 smithi164 ceph-mon[36970]: pgmap v9964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:05 smithi163 ceph-mon[36873]: pgmap v9964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:07 smithi042 ceph-mon[32775]: pgmap v9965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17756.530506 secs 2024-07-26T11:52:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17774.268872 secs 2024-07-26T11:52:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:07 smithi164 ceph-mon[36970]: pgmap v9965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17756.530506 secs 2024-07-26T11:52:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17774.268872 secs 2024-07-26T11:52:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:07 smithi163 ceph-mon[36873]: pgmap v9965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17756.530506 secs 2024-07-26T11:52:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17774.268872 secs 2024-07-26T11:52:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:09 smithi042 ceph-mon[32775]: pgmap v9966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:09 smithi164 ceph-mon[36970]: pgmap v9966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:09 smithi163 ceph-mon[36873]: pgmap v9966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:11.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:11 smithi042 ceph-mon[32775]: pgmap v9967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:11 smithi164 ceph-mon[36970]: pgmap v9967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:11 smithi163 ceph-mon[36873]: pgmap v9967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17761.530638 secs 2024-07-26T11:52:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17779.268997 secs 2024-07-26T11:52:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:12 smithi042 ceph-mon[32775]: pgmap v9968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17761.530638 secs 2024-07-26T11:52:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17779.268997 secs 2024-07-26T11:52:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:12 smithi164 ceph-mon[36970]: pgmap v9968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17761.530638 secs 2024-07-26T11:52:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17779.268997 secs 2024-07-26T11:52:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:12 smithi163 ceph-mon[36873]: pgmap v9968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:52:13.912+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:52:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:52:13.912+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:52:13.912+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:52:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:15 smithi042 ceph-mon[32775]: pgmap v9969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:52:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:52:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:52:15.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:52:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:15 smithi164 ceph-mon[36970]: pgmap v9969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:52:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:52:15.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:52:15.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:52:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:15 smithi163 ceph-mon[36873]: pgmap v9969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:52:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:52:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:52:15.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:52:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:16 smithi042 ceph-mon[32775]: pgmap v9970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:52:16.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:16 smithi042 ceph-mon[32775]: pgmap v9971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:52:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:16 smithi164 ceph-mon[36970]: pgmap v9970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:52:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:16 smithi164 ceph-mon[36970]: pgmap v9971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:52:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:16 smithi163 ceph-mon[36873]: pgmap v9970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:52:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:16 smithi163 ceph-mon[36873]: pgmap v9971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:52:17.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17766.530762 secs 2024-07-26T11:52:17.375 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17784.269123 secs 2024-07-26T11:52:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17766.530762 secs 2024-07-26T11:52:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17784.269123 secs 2024-07-26T11:52:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17766.530762 secs 2024-07-26T11:52:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17784.269123 secs 2024-07-26T11:52:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:18 smithi042 ceph-mon[32775]: pgmap v9972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:52:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:18 smithi164 ceph-mon[36970]: pgmap v9972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:52:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:18 smithi163 ceph-mon[36873]: pgmap v9972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:52:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:20 smithi042 ceph-mon[32775]: pgmap v9973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:52:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:20 smithi164 ceph-mon[36970]: pgmap v9973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:52:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:20 smithi163 ceph-mon[36873]: pgmap v9973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:52:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:22 smithi042 ceph-mon[32775]: pgmap v9974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:52:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17771.530887 secs 2024-07-26T11:52:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17789.269266 secs 2024-07-26T11:52:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:22 smithi164 ceph-mon[36970]: pgmap v9974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:52:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17771.530887 secs 2024-07-26T11:52:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17789.269266 secs 2024-07-26T11:52:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:22 smithi163 ceph-mon[36873]: pgmap v9974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:52:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17771.530887 secs 2024-07-26T11:52:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17789.269266 secs 2024-07-26T11:52:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:24 smithi042 ceph-mon[32775]: pgmap v9975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:52:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:24 smithi164 ceph-mon[36970]: pgmap v9975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:52:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:24 smithi163 ceph-mon[36873]: pgmap v9975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:52:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:25 smithi042 ceph-mon[32775]: pgmap v9976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:25 smithi164 ceph-mon[36970]: pgmap v9976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:25 smithi163 ceph-mon[36873]: pgmap v9976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:52:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17776.531016 secs 2024-07-26T11:52:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17776.531016 secs 2024-07-26T11:52:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17776.531016 secs 2024-07-26T11:52:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17794.269382 secs 2024-07-26T11:52:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:27 smithi164 ceph-mon[36970]: pgmap v9977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17794.269382 secs 2024-07-26T11:52:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:27 smithi163 ceph-mon[36873]: pgmap v9977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:27.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17794.269382 secs 2024-07-26T11:52:27.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:27 smithi042 ceph-mon[32775]: pgmap v9977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:28.918 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:52:28.917+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:52:28.918 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:52:28.917+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:28.918 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:52:28.917+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:52:29.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:29 smithi164 ceph-mon[36970]: pgmap v9978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:52:30.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:29 smithi163 ceph-mon[36873]: pgmap v9978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:52:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:29 smithi042 ceph-mon[32775]: pgmap v9978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:52:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:31 smithi164 ceph-mon[36970]: pgmap v9979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17781.531156 secs 2024-07-26T11:52:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:31 smithi163 ceph-mon[36873]: pgmap v9979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17781.531156 secs 2024-07-26T11:52:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:31 smithi042 ceph-mon[32775]: pgmap v9979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17781.531156 secs 2024-07-26T11:52:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17799.269505 secs 2024-07-26T11:52:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17799.269505 secs 2024-07-26T11:52:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17799.269505 secs 2024-07-26T11:52:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:33 smithi164 ceph-mon[36970]: pgmap v9980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:33 smithi163 ceph-mon[36873]: pgmap v9980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:34.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:33 smithi042 ceph-mon[32775]: pgmap v9980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:35 smithi164 ceph-mon[36970]: pgmap v9981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:36.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:35 smithi163 ceph-mon[36873]: pgmap v9981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:35 smithi042 ceph-mon[32775]: pgmap v9981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17786.531279 secs 2024-07-26T11:52:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17786.531279 secs 2024-07-26T11:52:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17786.531279 secs 2024-07-26T11:52:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17804.269580 secs 2024-07-26T11:52:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:37 smithi164 ceph-mon[36970]: pgmap v9982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17804.269580 secs 2024-07-26T11:52:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:37 smithi163 ceph-mon[36873]: pgmap v9982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17804.269580 secs 2024-07-26T11:52:38.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:37 smithi042 ceph-mon[32775]: pgmap v9982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:39 smithi164 ceph-mon[36970]: pgmap v9983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:52:40.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:39 smithi163 ceph-mon[36873]: pgmap v9983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:52:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:39 smithi042 ceph-mon[32775]: pgmap v9983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:52:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:41 smithi164 ceph-mon[36970]: pgmap v9984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17791.531381 secs 2024-07-26T11:52:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:41 smithi163 ceph-mon[36873]: pgmap v9984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:42.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17791.531381 secs 2024-07-26T11:52:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:41 smithi042 ceph-mon[32775]: pgmap v9984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17791.531381 secs 2024-07-26T11:52:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17809.269716 secs 2024-07-26T11:52:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17809.269716 secs 2024-07-26T11:52:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17809.269716 secs 2024-07-26T11:52:43.919 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:52:43.918+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:52:43.919 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:52:43.918+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:43.919 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:52:43.918+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:52:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:43 smithi164 ceph-mon[36970]: pgmap v9985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:43 smithi163 ceph-mon[36873]: pgmap v9985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:43 smithi042 ceph-mon[32775]: pgmap v9985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:45 smithi164 ceph-mon[36970]: pgmap v9986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:45 smithi163 ceph-mon[36873]: pgmap v9986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:46.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:45 smithi042 ceph-mon[32775]: pgmap v9986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17796.531486 secs 2024-07-26T11:52:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17796.531486 secs 2024-07-26T11:52:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17796.531486 secs 2024-07-26T11:52:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17814.269813 secs 2024-07-26T11:52:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:47 smithi164 ceph-mon[36970]: pgmap v9987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17814.269813 secs 2024-07-26T11:52:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:47 smithi163 ceph-mon[36873]: pgmap v9987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17814.269813 secs 2024-07-26T11:52:48.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:47 smithi042 ceph-mon[32775]: pgmap v9987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:49 smithi164 ceph-mon[36970]: pgmap v9988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:52:50.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:49 smithi163 ceph-mon[36873]: pgmap v9988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:52:50.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:49 smithi042 ceph-mon[32775]: pgmap v9988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:52:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:51 smithi164 ceph-mon[36970]: pgmap v9989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17801.531609 secs 2024-07-26T11:52:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:51 smithi163 ceph-mon[36873]: pgmap v9989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17801.531609 secs 2024-07-26T11:52:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:51 smithi042 ceph-mon[32775]: pgmap v9989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17801.531609 secs 2024-07-26T11:52:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17819.269973 secs 2024-07-26T11:52:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17819.269973 secs 2024-07-26T11:52:53.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17819.269973 secs 2024-07-26T11:52:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:53 smithi163 ceph-mon[36873]: pgmap v9990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:53 smithi042 ceph-mon[32775]: pgmap v9990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:53 smithi164 ceph-mon[36970]: pgmap v9990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:56.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:55 smithi163 ceph-mon[36873]: pgmap v9991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:55 smithi042 ceph-mon[32775]: pgmap v9991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:55 smithi164 ceph-mon[36970]: pgmap v9991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:52:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17806.531759 secs 2024-07-26T11:52:57.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17806.531759 secs 2024-07-26T11:52:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17806.531759 secs 2024-07-26T11:52:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17824.270090 secs 2024-07-26T11:52:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:57 smithi163 ceph-mon[36873]: pgmap v9992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17824.270090 secs 2024-07-26T11:52:58.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:57 smithi042 ceph-mon[32775]: pgmap v9992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17824.270090 secs 2024-07-26T11:52:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:57 smithi164 ceph-mon[36970]: pgmap v9992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:52:58.925 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:52:58.923+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:52:58.925 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:52:58.923+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:52:58.925 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:52:58.923+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:52:59 smithi163 ceph-mon[36873]: pgmap v9993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:53:00.144 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:52:59 smithi042 ceph-mon[32775]: pgmap v9993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:53:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:52:59 smithi164 ceph-mon[36970]: pgmap v9993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:53:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:00.673+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:53:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:00.673+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658006480 con 0x7ff680102000 2024-07-26T11:53:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:00.673+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff658006480 con 0x7ff680102000 2024-07-26T11:53:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:00.673+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:53:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:00.673+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T11:53:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:00.673+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:53:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:00.673+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:53:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:00.673+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:53:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:01 smithi163 ceph-mon[36873]: pgmap v9994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:02.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17811.531905 secs 2024-07-26T11:53:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:01 smithi042 ceph-mon[32775]: pgmap v9994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17811.531905 secs 2024-07-26T11:53:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:01 smithi164 ceph-mon[36970]: pgmap v9994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17811.531905 secs 2024-07-26T11:53:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17829.270223 secs 2024-07-26T11:53:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17829.270223 secs 2024-07-26T11:53:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17829.270223 secs 2024-07-26T11:53:04.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:03 smithi163 ceph-mon[36873]: pgmap v9995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:03 smithi042 ceph-mon[32775]: pgmap v9995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:03 smithi164 ceph-mon[36970]: pgmap v9995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:06.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:05 smithi163 ceph-mon[36873]: pgmap v9996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:05 smithi042 ceph-mon[32775]: pgmap v9996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:05 smithi164 ceph-mon[36970]: pgmap v9996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17816.532037 secs 2024-07-26T11:53:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17816.532037 secs 2024-07-26T11:53:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17816.532037 secs 2024-07-26T11:53:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17834.270302 secs 2024-07-26T11:53:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:07 smithi042 ceph-mon[32775]: pgmap v9997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17834.270302 secs 2024-07-26T11:53:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:07 smithi164 ceph-mon[36970]: pgmap v9997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17834.270302 secs 2024-07-26T11:53:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:07 smithi163 ceph-mon[36873]: pgmap v9997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:09 smithi042 ceph-mon[32775]: pgmap v9998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:53:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:09 smithi164 ceph-mon[36970]: pgmap v9998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:53:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:09 smithi163 ceph-mon[36873]: pgmap v9998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:53:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:11 smithi042 ceph-mon[32775]: pgmap v9999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17821.532159 secs 2024-07-26T11:53:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:11 smithi164 ceph-mon[36970]: pgmap v9999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17821.532159 secs 2024-07-26T11:53:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:11 smithi163 ceph-mon[36873]: pgmap v9999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17821.532159 secs 2024-07-26T11:53:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17839.270429 secs 2024-07-26T11:53:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17839.270429 secs 2024-07-26T11:53:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17839.270429 secs 2024-07-26T11:53:13.926 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:13.924+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:53:13.926 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:13.924+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:13.926 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:13.924+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:53:14.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:13 smithi042 ceph-mon[32775]: pgmap v10000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:13 smithi164 ceph-mon[36970]: pgmap v10000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:13 smithi163 ceph-mon[36873]: pgmap v10000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:14 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:53:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:14 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:53:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:14 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:53:16.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:15 smithi042 ceph-mon[32775]: pgmap v10001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:16.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:53:16.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:53:16.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:53:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:15 smithi164 ceph-mon[36970]: pgmap v10001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:53:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:53:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:53:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:15 smithi163 ceph-mon[36873]: pgmap v10001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:53:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:53:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:53:17.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:16 smithi164 ceph-mon[36970]: pgmap v10002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:53:17.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:16 smithi164 ceph-mon[36970]: pgmap v10003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:53:17.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17826.532286 secs 2024-07-26T11:53:17.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17844.270570 secs 2024-07-26T11:53:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:16 smithi042 ceph-mon[32775]: pgmap v10002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:53:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:16 smithi042 ceph-mon[32775]: pgmap v10003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:53:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17826.532286 secs 2024-07-26T11:53:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17844.270570 secs 2024-07-26T11:53:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:16 smithi163 ceph-mon[36873]: pgmap v10002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:53:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:16 smithi163 ceph-mon[36873]: pgmap v10003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:53:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17826.532286 secs 2024-07-26T11:53:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17844.270570 secs 2024-07-26T11:53:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:18 smithi042 ceph-mon[32775]: pgmap v10004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:53:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:18 smithi164 ceph-mon[36970]: pgmap v10004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:53:19.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:18 smithi163 ceph-mon[36873]: pgmap v10004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:53:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:20 smithi042 ceph-mon[32775]: pgmap v10005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:53:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:20 smithi164 ceph-mon[36970]: pgmap v10005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:53:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:20 smithi163 ceph-mon[36873]: pgmap v10005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:53:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17831.532415 secs 2024-07-26T11:53:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17849.270717 secs 2024-07-26T11:53:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17831.532415 secs 2024-07-26T11:53:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17849.270717 secs 2024-07-26T11:53:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17831.532415 secs 2024-07-26T11:53:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17849.270717 secs 2024-07-26T11:53:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:22 smithi042 ceph-mon[32775]: pgmap v10006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:53:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:22 smithi164 ceph-mon[36970]: pgmap v10006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:53:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:22 smithi163 ceph-mon[36873]: pgmap v10006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:53:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:24 smithi042 ceph-mon[32775]: pgmap v10007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:53:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:24 smithi164 ceph-mon[36970]: pgmap v10007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:53:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:24 smithi163 ceph-mon[36873]: pgmap v10007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:53:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:26 smithi042 ceph-mon[32775]: pgmap v10008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:53:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17836.532538 secs 2024-07-26T11:53:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17854.270836 secs 2024-07-26T11:53:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:26 smithi164 ceph-mon[36970]: pgmap v10008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:53:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17836.532538 secs 2024-07-26T11:53:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17854.270836 secs 2024-07-26T11:53:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:26 smithi163 ceph-mon[36873]: pgmap v10008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:53:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17836.532538 secs 2024-07-26T11:53:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17854.270836 secs 2024-07-26T11:53:28.934 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:28.930+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:53:28.934 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:28.930+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:28.934 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:28.930+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:53:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:28 smithi042 ceph-mon[32775]: pgmap v10009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:28 smithi164 ceph-mon[36970]: pgmap v10009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:28 smithi163 ceph-mon[36873]: pgmap v10009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:30 smithi042 ceph-mon[32775]: pgmap v10010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:30 smithi164 ceph-mon[36970]: pgmap v10010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:30 smithi163 ceph-mon[36873]: pgmap v10010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17841.532661 secs 2024-07-26T11:53:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17859.271003 secs 2024-07-26T11:53:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17841.532661 secs 2024-07-26T11:53:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17859.271003 secs 2024-07-26T11:53:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17841.532661 secs 2024-07-26T11:53:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17859.271003 secs 2024-07-26T11:53:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:32 smithi042 ceph-mon[32775]: pgmap v10011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:32 smithi164 ceph-mon[36970]: pgmap v10011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:32 smithi163 ceph-mon[36873]: pgmap v10011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:35.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:34 smithi042 ceph-mon[32775]: pgmap v10012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:34 smithi164 ceph-mon[36970]: pgmap v10012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:35.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:34 smithi163 ceph-mon[36873]: pgmap v10012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:36 smithi042 ceph-mon[32775]: pgmap v10013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17846.532767 secs 2024-07-26T11:53:37.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17864.271067 secs 2024-07-26T11:53:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:36 smithi164 ceph-mon[36970]: pgmap v10013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17846.532767 secs 2024-07-26T11:53:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17864.271067 secs 2024-07-26T11:53:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:36 smithi163 ceph-mon[36873]: pgmap v10013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17846.532767 secs 2024-07-26T11:53:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17864.271067 secs 2024-07-26T11:53:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:38 smithi042 ceph-mon[32775]: pgmap v10014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:38 smithi164 ceph-mon[36970]: pgmap v10014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:39.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:38 smithi163 ceph-mon[36873]: pgmap v10014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:40 smithi042 ceph-mon[32775]: pgmap v10015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:40 smithi164 ceph-mon[36970]: pgmap v10015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:41.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:40 smithi163 ceph-mon[36873]: pgmap v10015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17851.532891 secs 2024-07-26T11:53:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17869.271186 secs 2024-07-26T11:53:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17851.532891 secs 2024-07-26T11:53:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17869.271186 secs 2024-07-26T11:53:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17851.532891 secs 2024-07-26T11:53:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17869.271186 secs 2024-07-26T11:53:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:42 smithi042 ceph-mon[32775]: pgmap v10016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:42 smithi164 ceph-mon[36970]: pgmap v10016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:42 smithi163 ceph-mon[36873]: pgmap v10016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:43.931+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:53:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:43.931+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:43.931+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:53:45.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:44 smithi042 ceph-mon[32775]: pgmap v10017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:44 smithi164 ceph-mon[36970]: pgmap v10017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:45.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:44 smithi163 ceph-mon[36873]: pgmap v10017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:46 smithi042 ceph-mon[32775]: pgmap v10018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17856.533040 secs 2024-07-26T11:53:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17874.271305 secs 2024-07-26T11:53:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:46 smithi164 ceph-mon[36970]: pgmap v10018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17856.533040 secs 2024-07-26T11:53:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17874.271305 secs 2024-07-26T11:53:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:46 smithi163 ceph-mon[36873]: pgmap v10018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17856.533040 secs 2024-07-26T11:53:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17874.271305 secs 2024-07-26T11:53:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:48 smithi042 ceph-mon[32775]: pgmap v10019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:48 smithi164 ceph-mon[36970]: pgmap v10019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:48 smithi163 ceph-mon[36873]: pgmap v10019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:50 smithi042 ceph-mon[32775]: pgmap v10020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:50 smithi164 ceph-mon[36970]: pgmap v10020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:50 smithi163 ceph-mon[36873]: pgmap v10020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17861.533110 secs 2024-07-26T11:53:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17879.271414 secs 2024-07-26T11:53:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17861.533110 secs 2024-07-26T11:53:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17879.271414 secs 2024-07-26T11:53:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17861.533110 secs 2024-07-26T11:53:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17879.271414 secs 2024-07-26T11:53:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:52 smithi042 ceph-mon[32775]: pgmap v10021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:52 smithi164 ceph-mon[36970]: pgmap v10021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:52 smithi163 ceph-mon[36873]: pgmap v10021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:54 smithi042 ceph-mon[32775]: pgmap v10022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:54 smithi164 ceph-mon[36970]: pgmap v10022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:55.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:54 smithi163 ceph-mon[36873]: pgmap v10022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:56 smithi042 ceph-mon[32775]: pgmap v10023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17866.533256 secs 2024-07-26T11:53:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17884.271527 secs 2024-07-26T11:53:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:56 smithi164 ceph-mon[36970]: pgmap v10023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17866.533256 secs 2024-07-26T11:53:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17884.271527 secs 2024-07-26T11:53:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:56 smithi163 ceph-mon[36873]: pgmap v10023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:53:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17866.533256 secs 2024-07-26T11:53:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17884.271527 secs 2024-07-26T11:53:58.939 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:58.937+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:53:58.939 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:58.937+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:58.939 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:53:58.937+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:53:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:53:58 smithi042 ceph-mon[32775]: pgmap v10024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:53:58 smithi164 ceph-mon[36970]: pgmap v10024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:53:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:53:58 smithi163 ceph-mon[36873]: pgmap v10024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:00 smithi042 ceph-mon[32775]: pgmap v10025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:00 smithi164 ceph-mon[36970]: pgmap v10025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:00 smithi163 ceph-mon[36873]: pgmap v10025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17871.533358 secs 2024-07-26T11:54:02.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17889.271650 secs 2024-07-26T11:54:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17871.533358 secs 2024-07-26T11:54:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17889.271650 secs 2024-07-26T11:54:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17871.533358 secs 2024-07-26T11:54:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17889.271650 secs 2024-07-26T11:54:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:02 smithi042 ceph-mon[32775]: pgmap v10026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:02 smithi164 ceph-mon[36970]: pgmap v10026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:02 smithi163 ceph-mon[36873]: pgmap v10026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:04 smithi042 ceph-mon[32775]: pgmap v10027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:04 smithi164 ceph-mon[36970]: pgmap v10027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:04 smithi163 ceph-mon[36873]: pgmap v10027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:06 smithi042 ceph-mon[32775]: pgmap v10028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17876.533482 secs 2024-07-26T11:54:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17894.271788 secs 2024-07-26T11:54:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:06 smithi164 ceph-mon[36970]: pgmap v10028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17876.533482 secs 2024-07-26T11:54:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17894.271788 secs 2024-07-26T11:54:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:06 smithi163 ceph-mon[36873]: pgmap v10028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17876.533482 secs 2024-07-26T11:54:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17894.271788 secs 2024-07-26T11:54:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:08 smithi164 ceph-mon[36970]: pgmap v10029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:08 smithi163 ceph-mon[36873]: pgmap v10029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:09.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:08 smithi042 ceph-mon[32775]: pgmap v10029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:10 smithi164 ceph-mon[36970]: pgmap v10030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:10 smithi163 ceph-mon[36873]: pgmap v10030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:10 smithi042 ceph-mon[32775]: pgmap v10030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17881.533605 secs 2024-07-26T11:54:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17899.271912 secs 2024-07-26T11:54:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17881.533605 secs 2024-07-26T11:54:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17899.271912 secs 2024-07-26T11:54:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17881.533605 secs 2024-07-26T11:54:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17899.271912 secs 2024-07-26T11:54:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:12 smithi164 ceph-mon[36970]: pgmap v10031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:12 smithi163 ceph-mon[36873]: pgmap v10031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:12 smithi042 ceph-mon[32775]: pgmap v10031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:13.940 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:54:13.938+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:54:13.940 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:54:13.938+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:13.940 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:54:13.938+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:54:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:14 smithi164 ceph-mon[36970]: pgmap v10032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:14 smithi163 ceph-mon[36873]: pgmap v10032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:14 smithi042 ceph-mon[32775]: pgmap v10032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:15 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:54:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:15 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:54:16.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:15 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:54:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:16 smithi164 ceph-mon[36970]: pgmap v10033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17886.533698 secs 2024-07-26T11:54:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17904.271984 secs 2024-07-26T11:54:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:16 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:16 smithi163 ceph-mon[36873]: pgmap v10033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17886.533698 secs 2024-07-26T11:54:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17904.271984 secs 2024-07-26T11:54:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:16 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:16 smithi042 ceph-mon[32775]: pgmap v10033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17886.533698 secs 2024-07-26T11:54:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17904.271984 secs 2024-07-26T11:54:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:16 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:18.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:18.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:18.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:18.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:18.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:54:18.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:54:18.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:18 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:18.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:18.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:54:18.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:54:18.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:18 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:54:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:54:18.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:18 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:54:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:19 smithi164 ceph-mon[36970]: pgmap v10034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:19 smithi164 ceph-mon[36970]: pgmap v10035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:54:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:19 smithi164 ceph-mon[36970]: pgmap v10036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:54:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:19 smithi163 ceph-mon[36873]: pgmap v10034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:19 smithi163 ceph-mon[36873]: pgmap v10035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:54:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:19 smithi163 ceph-mon[36873]: pgmap v10036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:54:19.552 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:19 smithi042 ceph-mon[32775]: pgmap v10034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:19.552 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:19 smithi042 ceph-mon[32775]: pgmap v10035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T11:54:19.552 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:19 smithi042 ceph-mon[32775]: pgmap v10036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:54:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:20 smithi164 ceph-mon[36970]: pgmap v10037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:54:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:20 smithi163 ceph-mon[36873]: pgmap v10037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:54:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:20 smithi042 ceph-mon[32775]: pgmap v10037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:54:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17891.533818 secs 2024-07-26T11:54:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17891.533818 secs 2024-07-26T11:54:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17891.533818 secs 2024-07-26T11:54:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17909.272050 secs 2024-07-26T11:54:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:22 smithi164 ceph-mon[36970]: pgmap v10038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:54:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17909.272050 secs 2024-07-26T11:54:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:22 smithi163 ceph-mon[36873]: pgmap v10038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:54:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17909.272050 secs 2024-07-26T11:54:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:22 smithi042 ceph-mon[32775]: pgmap v10038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:54:24.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:24 smithi164 ceph-mon[36970]: pgmap v10039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:54:25.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:24 smithi163 ceph-mon[36873]: pgmap v10039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:54:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:24 smithi042 ceph-mon[32775]: pgmap v10039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:54:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:26 smithi164 ceph-mon[36970]: pgmap v10040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:54:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17896.533946 secs 2024-07-26T11:54:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:26 smithi163 ceph-mon[36873]: pgmap v10040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:54:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17896.533946 secs 2024-07-26T11:54:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:26 smithi042 ceph-mon[32775]: pgmap v10040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T11:54:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17896.533946 secs 2024-07-26T11:54:28.010 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17914.272171 secs 2024-07-26T11:54:28.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17914.272171 secs 2024-07-26T11:54:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17914.272171 secs 2024-07-26T11:54:28.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:54:28.944+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:54:28.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:54:28.944+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:28.946 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:54:28.944+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:54:29.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:28 smithi163 ceph-mon[36873]: pgmap v10041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:54:29.036 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:28 smithi042 ceph-mon[32775]: pgmap v10041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:54:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:28 smithi164 ceph-mon[36970]: pgmap v10041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:54:31.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:30 smithi163 ceph-mon[36873]: pgmap v10042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:30 smithi042 ceph-mon[32775]: pgmap v10042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:30 smithi164 ceph-mon[36970]: pgmap v10042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:32.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17901.534078 secs 2024-07-26T11:54:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17901.534078 secs 2024-07-26T11:54:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17901.534078 secs 2024-07-26T11:54:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17919.272299 secs 2024-07-26T11:54:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:32 smithi163 ceph-mon[36873]: pgmap v10043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17919.272299 secs 2024-07-26T11:54:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:32 smithi042 ceph-mon[32775]: pgmap v10043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17919.272299 secs 2024-07-26T11:54:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:32 smithi164 ceph-mon[36970]: pgmap v10043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:34 smithi163 ceph-mon[36873]: pgmap v10044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:54:35.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:34 smithi042 ceph-mon[32775]: pgmap v10044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:54:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:34 smithi164 ceph-mon[36970]: pgmap v10044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:54:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:36 smithi163 ceph-mon[36873]: pgmap v10045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:37.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17906.534206 secs 2024-07-26T11:54:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:36 smithi042 ceph-mon[32775]: pgmap v10045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17906.534206 secs 2024-07-26T11:54:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:36 smithi164 ceph-mon[36970]: pgmap v10045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17906.534206 secs 2024-07-26T11:54:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17924.272403 secs 2024-07-26T11:54:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17924.272403 secs 2024-07-26T11:54:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17924.272403 secs 2024-07-26T11:54:39.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:38 smithi163 ceph-mon[36873]: pgmap v10046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:38 smithi042 ceph-mon[32775]: pgmap v10046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:38 smithi164 ceph-mon[36970]: pgmap v10046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:40 smithi163 ceph-mon[36873]: pgmap v10047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:40 smithi042 ceph-mon[32775]: pgmap v10047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:40 smithi164 ceph-mon[36970]: pgmap v10047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17911.534304 secs 2024-07-26T11:54:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17911.534304 secs 2024-07-26T11:54:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17911.534304 secs 2024-07-26T11:54:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17929.272502 secs 2024-07-26T11:54:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:42 smithi164 ceph-mon[36970]: pgmap v10048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17929.272502 secs 2024-07-26T11:54:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:42 smithi163 ceph-mon[36873]: pgmap v10048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17929.272502 secs 2024-07-26T11:54:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:42 smithi042 ceph-mon[32775]: pgmap v10048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:54:43.945+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:54:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:54:43.945+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:54:43.945+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:54:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:44 smithi164 ceph-mon[36970]: pgmap v10049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:54:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:44 smithi163 ceph-mon[36873]: pgmap v10049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:54:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:44 smithi042 ceph-mon[32775]: pgmap v10049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:54:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:46 smithi163 ceph-mon[36873]: pgmap v10050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17916.534429 secs 2024-07-26T11:54:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:46 smithi042 ceph-mon[32775]: pgmap v10050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17916.534429 secs 2024-07-26T11:54:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:46 smithi164 ceph-mon[36970]: pgmap v10050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17916.534429 secs 2024-07-26T11:54:48.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17934.272643 secs 2024-07-26T11:54:48.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17934.272643 secs 2024-07-26T11:54:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17934.272643 secs 2024-07-26T11:54:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:48 smithi163 ceph-mon[36873]: pgmap v10051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:48 smithi042 ceph-mon[32775]: pgmap v10051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:48 smithi164 ceph-mon[36970]: pgmap v10051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:51.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:50 smithi163 ceph-mon[36873]: pgmap v10052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:50 smithi042 ceph-mon[32775]: pgmap v10052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:50 smithi164 ceph-mon[36970]: pgmap v10052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17921.534535 secs 2024-07-26T11:54:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17921.534535 secs 2024-07-26T11:54:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17921.534535 secs 2024-07-26T11:54:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17939.272761 secs 2024-07-26T11:54:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:52 smithi163 ceph-mon[36873]: pgmap v10053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17939.272761 secs 2024-07-26T11:54:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:52 smithi042 ceph-mon[32775]: pgmap v10053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17939.272761 secs 2024-07-26T11:54:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:52 smithi164 ceph-mon[36970]: pgmap v10053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:54 smithi163 ceph-mon[36873]: pgmap v10054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:54:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:54 smithi042 ceph-mon[32775]: pgmap v10054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:54:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:54 smithi164 ceph-mon[36970]: pgmap v10054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:54:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:56 smithi163 ceph-mon[36873]: pgmap v10055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17926.534662 secs 2024-07-26T11:54:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:56 smithi042 ceph-mon[32775]: pgmap v10055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17926.534662 secs 2024-07-26T11:54:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:56 smithi164 ceph-mon[36970]: pgmap v10055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:54:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17926.534662 secs 2024-07-26T11:54:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17944.272892 secs 2024-07-26T11:54:58.047 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17944.272892 secs 2024-07-26T11:54:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17944.272892 secs 2024-07-26T11:54:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:54:58.951+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:54:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:54:58.951+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:54:58.951+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:54:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:54:58 smithi042 ceph-mon[32775]: pgmap v10056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:54:58 smithi164 ceph-mon[36970]: pgmap v10056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:54:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:54:58 smithi163 ceph-mon[36873]: pgmap v10056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:01.060 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:00 smithi163 ceph-mon[36873]: pgmap v10057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:00 smithi042 ceph-mon[32775]: pgmap v10057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:00 smithi164 ceph-mon[36970]: pgmap v10057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17931.534799 secs 2024-07-26T11:55:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17931.534799 secs 2024-07-26T11:55:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17931.534799 secs 2024-07-26T11:55:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17949.273022 secs 2024-07-26T11:55:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:02 smithi042 ceph-mon[32775]: pgmap v10058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17949.273022 secs 2024-07-26T11:55:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:02 smithi164 ceph-mon[36970]: pgmap v10058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17949.273022 secs 2024-07-26T11:55:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:02 smithi163 ceph-mon[36873]: pgmap v10058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:04 smithi042 ceph-mon[32775]: pgmap v10059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:55:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:04 smithi164 ceph-mon[36970]: pgmap v10059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:55:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:04 smithi163 ceph-mon[36873]: pgmap v10059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:55:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:06 smithi042 ceph-mon[32775]: pgmap v10060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17936.534876 secs 2024-07-26T11:55:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:06 smithi164 ceph-mon[36970]: pgmap v10060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17936.534876 secs 2024-07-26T11:55:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:06 smithi163 ceph-mon[36873]: pgmap v10060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17936.534876 secs 2024-07-26T11:55:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17954.273140 secs 2024-07-26T11:55:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17954.273140 secs 2024-07-26T11:55:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17954.273140 secs 2024-07-26T11:55:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:08 smithi042 ceph-mon[32775]: pgmap v10061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:08 smithi164 ceph-mon[36970]: pgmap v10061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:08 smithi163 ceph-mon[36873]: pgmap v10061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:10 smithi042 ceph-mon[32775]: pgmap v10062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:10 smithi164 ceph-mon[36970]: pgmap v10062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:10 smithi163 ceph-mon[36873]: pgmap v10062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17941.535000 secs 2024-07-26T11:55:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17941.535000 secs 2024-07-26T11:55:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17941.535000 secs 2024-07-26T11:55:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17959.273262 secs 2024-07-26T11:55:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:12 smithi042 ceph-mon[32775]: pgmap v10063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17959.273262 secs 2024-07-26T11:55:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:12 smithi164 ceph-mon[36970]: pgmap v10063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17959.273262 secs 2024-07-26T11:55:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:12 smithi163 ceph-mon[36873]: pgmap v10063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:13.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:13.952+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:55:13.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:13.952+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:13.954 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:13.952+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:55:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:14 smithi042 ceph-mon[32775]: pgmap v10064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:55:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:14 smithi164 ceph-mon[36970]: pgmap v10064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:55:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:14 smithi163 ceph-mon[36873]: pgmap v10064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:55:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:16 smithi042 ceph-mon[32775]: pgmap v10065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17946.535103 secs 2024-07-26T11:55:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17964.273405 secs 2024-07-26T11:55:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:16 smithi164 ceph-mon[36970]: pgmap v10065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17946.535103 secs 2024-07-26T11:55:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17964.273405 secs 2024-07-26T11:55:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:16 smithi163 ceph-mon[36873]: pgmap v10065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17946.535103 secs 2024-07-26T11:55:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17964.273405 secs 2024-07-26T11:55:18.123 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:17 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:55:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:17 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:55:18.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:17 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:55:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:19 smithi042 ceph-mon[32775]: pgmap v10066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:55:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:55:19.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:19 smithi164 ceph-mon[36970]: pgmap v10066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:55:19.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:55:19.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:19 smithi163 ceph-mon[36873]: pgmap v10066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:19.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:55:19.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:55:19.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:55:20.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:20 smithi042 ceph-mon[32775]: pgmap v10067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:55:20.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:20 smithi042 ceph-mon[32775]: pgmap v10068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:55:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:20 smithi164 ceph-mon[36970]: pgmap v10067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:55:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:20 smithi164 ceph-mon[36970]: pgmap v10068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:55:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:20 smithi163 ceph-mon[36873]: pgmap v10067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 4 op/s 2024-07-26T11:55:20.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:20 smithi163 ceph-mon[36873]: pgmap v10068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T11:55:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:22 smithi042 ceph-mon[32775]: pgmap v10069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:55:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17951.535239 secs 2024-07-26T11:55:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17969.273523 secs 2024-07-26T11:55:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:22 smithi164 ceph-mon[36970]: pgmap v10069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:55:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17951.535239 secs 2024-07-26T11:55:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17969.273523 secs 2024-07-26T11:55:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:22 smithi163 ceph-mon[36873]: pgmap v10069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:55:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17951.535239 secs 2024-07-26T11:55:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17969.273523 secs 2024-07-26T11:55:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:24 smithi042 ceph-mon[32775]: pgmap v10070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:55:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:24 smithi164 ceph-mon[36970]: pgmap v10070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:55:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:24 smithi163 ceph-mon[36873]: pgmap v10070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:55:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:26 smithi042 ceph-mon[32775]: pgmap v10071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:55:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:26 smithi164 ceph-mon[36970]: pgmap v10071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:55:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:26 smithi163 ceph-mon[36873]: pgmap v10071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:55:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17956.535313 secs 2024-07-26T11:55:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17974.273649 secs 2024-07-26T11:55:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17956.535313 secs 2024-07-26T11:55:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17974.273649 secs 2024-07-26T11:55:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17956.535313 secs 2024-07-26T11:55:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17974.273649 secs 2024-07-26T11:55:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:28 smithi042 ceph-mon[32775]: pgmap v10072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:55:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:28 smithi164 ceph-mon[36970]: pgmap v10072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:55:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:28 smithi163 ceph-mon[36873]: pgmap v10072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:55:28.958 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:28.957+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:55:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:28.957+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:28.957+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:55:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:30 smithi042 ceph-mon[32775]: pgmap v10073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:55:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:30 smithi164 ceph-mon[36970]: pgmap v10073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:55:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:30 smithi163 ceph-mon[36873]: pgmap v10073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:55:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:32 smithi042 ceph-mon[32775]: pgmap v10074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17961.535439 secs 2024-07-26T11:55:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17979.273776 secs 2024-07-26T11:55:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:32 smithi164 ceph-mon[36970]: pgmap v10074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17961.535439 secs 2024-07-26T11:55:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17979.273776 secs 2024-07-26T11:55:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:32 smithi163 ceph-mon[36873]: pgmap v10074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17961.535439 secs 2024-07-26T11:55:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17979.273776 secs 2024-07-26T11:55:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:34 smithi042 ceph-mon[32775]: pgmap v10075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:34 smithi164 ceph-mon[36970]: pgmap v10075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:34 smithi163 ceph-mon[36873]: pgmap v10075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:36 smithi042 ceph-mon[32775]: pgmap v10076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:36 smithi164 ceph-mon[36970]: pgmap v10076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:36 smithi163 ceph-mon[36873]: pgmap v10076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17966.535559 secs 2024-07-26T11:55:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17984.273907 secs 2024-07-26T11:55:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17966.535559 secs 2024-07-26T11:55:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17984.273907 secs 2024-07-26T11:55:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17966.535559 secs 2024-07-26T11:55:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17984.273907 secs 2024-07-26T11:55:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:38 smithi164 ceph-mon[36970]: pgmap v10077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:38 smithi163 ceph-mon[36873]: pgmap v10077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:38 smithi042 ceph-mon[32775]: pgmap v10077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:40 smithi164 ceph-mon[36970]: pgmap v10078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:55:40.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:40 smithi163 ceph-mon[36873]: pgmap v10078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:55:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:40 smithi042 ceph-mon[32775]: pgmap v10078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:55:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:42 smithi164 ceph-mon[36970]: pgmap v10079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17971.535683 secs 2024-07-26T11:55:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17989.274034 secs 2024-07-26T11:55:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:42 smithi163 ceph-mon[36873]: pgmap v10079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17971.535683 secs 2024-07-26T11:55:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17989.274034 secs 2024-07-26T11:55:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:42 smithi042 ceph-mon[32775]: pgmap v10079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17971.535683 secs 2024-07-26T11:55:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17989.274034 secs 2024-07-26T11:55:43.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:43.959+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:55:43.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:43.959+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:43.960 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:43.959+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:55:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:44 smithi164 ceph-mon[36970]: pgmap v10080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:44 smithi163 ceph-mon[36873]: pgmap v10080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:44 smithi042 ceph-mon[32775]: pgmap v10080: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:45 smithi164 ceph-mon[36970]: pgmap v10081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:45 smithi163 ceph-mon[36873]: pgmap v10081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:45 smithi042 ceph-mon[32775]: pgmap v10081: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17976.535807 secs 2024-07-26T11:55:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17976.535807 secs 2024-07-26T11:55:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17976.535807 secs 2024-07-26T11:55:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17994.274128 secs 2024-07-26T11:55:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:47 smithi164 ceph-mon[36970]: pgmap v10082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17994.274128 secs 2024-07-26T11:55:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:47 smithi163 ceph-mon[36873]: pgmap v10082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17994.274128 secs 2024-07-26T11:55:47.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:47 smithi042 ceph-mon[32775]: pgmap v10082: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:49 smithi042 ceph-mon[32775]: pgmap v10083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:55:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:49 smithi164 ceph-mon[36970]: pgmap v10083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:55:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:49 smithi163 ceph-mon[36873]: pgmap v10083: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:55:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:51 smithi042 ceph-mon[32775]: pgmap v10084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17981.535935 secs 2024-07-26T11:55:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:51 smithi164 ceph-mon[36970]: pgmap v10084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17981.535935 secs 2024-07-26T11:55:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:51 smithi163 ceph-mon[36873]: pgmap v10084: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:52.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17981.535935 secs 2024-07-26T11:55:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17999.274210 secs 2024-07-26T11:55:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17999.274210 secs 2024-07-26T11:55:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17999.274210 secs 2024-07-26T11:55:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:53 smithi042 ceph-mon[32775]: pgmap v10085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:53 smithi164 ceph-mon[36970]: pgmap v10085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:53 smithi163 ceph-mon[36873]: pgmap v10085: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:55 smithi042 ceph-mon[32775]: pgmap v10086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:55 smithi164 ceph-mon[36970]: pgmap v10086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:55 smithi163 ceph-mon[36873]: pgmap v10086: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:55:56.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17986.536060 secs 2024-07-26T11:55:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17986.536060 secs 2024-07-26T11:55:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17986.536060 secs 2024-07-26T11:55:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:57.889+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T11:55:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:57.889+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:55:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:57.889+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ff678003990 tx=0x7ff678060000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:55:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:57.889+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ff678003990 tx=0x7ff678060000 comp rx=0 tx=0).stop 2024-07-26T11:55:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:57.889+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T11:55:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:57.889+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:55:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:57.889+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:55:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:57.889+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:55:57.892 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:57.889+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:55:57.892 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:57.890+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b4fe0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff67802f970 tx=0x7ff67806c070 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:55:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18004.274330 secs 2024-07-26T11:55:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:57 smithi042 ceph-mon[32775]: pgmap v10087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18004.274330 secs 2024-07-26T11:55:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:57 smithi164 ceph-mon[36970]: pgmap v10087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18004.274330 secs 2024-07-26T11:55:58.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:57 smithi163 ceph-mon[36873]: pgmap v10087: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:55:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:58.960+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:55:58.962 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:58.960+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:58.962 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:55:58.960+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:55:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:55:59 smithi042 ceph-mon[32775]: pgmap v10088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:55:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:55:59 smithi164 ceph-mon[36970]: pgmap v10088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:56:00.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:55:59 smithi163 ceph-mon[36873]: pgmap v10088: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:56:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:56:00.703+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:56:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:56:00.703+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:56:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:56:00.703+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:56:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:56:00.703+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:56:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:01 smithi042 ceph-mon[32775]: pgmap v10089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:01.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17991.536179 secs 2024-07-26T11:56:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:01 smithi164 ceph-mon[36970]: pgmap v10089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17991.536179 secs 2024-07-26T11:56:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:01 smithi163 ceph-mon[36873]: pgmap v10089: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17991.536179 secs 2024-07-26T11:56:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18009.274456 secs 2024-07-26T11:56:02.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18009.274456 secs 2024-07-26T11:56:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18009.274456 secs 2024-07-26T11:56:03.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:03 smithi042 ceph-mon[32775]: pgmap v10090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:03.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:03 smithi164 ceph-mon[36970]: pgmap v10090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:04.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:03 smithi163 ceph-mon[36873]: pgmap v10090: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:05 smithi042 ceph-mon[32775]: pgmap v10091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:05 smithi164 ceph-mon[36970]: pgmap v10091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:05 smithi163 ceph-mon[36873]: pgmap v10091: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 17996.536296 secs 2024-07-26T11:56:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 17996.536296 secs 2024-07-26T11:56:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 17996.536296 secs 2024-07-26T11:56:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18014.274579 secs 2024-07-26T11:56:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:07 smithi164 ceph-mon[36970]: pgmap v10092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18014.274579 secs 2024-07-26T11:56:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:07 smithi163 ceph-mon[36873]: pgmap v10092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18014.274579 secs 2024-07-26T11:56:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:07 smithi042 ceph-mon[32775]: pgmap v10092: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:09 smithi042 ceph-mon[32775]: pgmap v10093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:56:09.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:09 smithi164 ceph-mon[36970]: pgmap v10093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:56:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:09 smithi163 ceph-mon[36873]: pgmap v10093: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:56:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:11 smithi042 ceph-mon[32775]: pgmap v10094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:11.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18001.536362 secs 2024-07-26T11:56:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:11 smithi164 ceph-mon[36970]: pgmap v10094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18001.536362 secs 2024-07-26T11:56:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:11 smithi163 ceph-mon[36873]: pgmap v10094: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:12.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18001.536362 secs 2024-07-26T11:56:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18019.274698 secs 2024-07-26T11:56:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18019.274698 secs 2024-07-26T11:56:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18019.274698 secs 2024-07-26T11:56:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:13 smithi042 ceph-mon[32775]: pgmap v10095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:56:13.961+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:56:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:56:13.961+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:56:13.961+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:56:13.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:13 smithi164 ceph-mon[36970]: pgmap v10095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:14.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:13 smithi163 ceph-mon[36873]: pgmap v10095: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:15.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:15 smithi042 ceph-mon[32775]: pgmap v10096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:15.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:15 smithi164 ceph-mon[36970]: pgmap v10096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:16.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:15 smithi163 ceph-mon[36873]: pgmap v10096: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:16.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18006.536509 secs 2024-07-26T11:56:16.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18006.536509 secs 2024-07-26T11:56:17.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18006.536509 secs 2024-07-26T11:56:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18024.274822 secs 2024-07-26T11:56:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:17 smithi042 ceph-mon[32775]: pgmap v10097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18024.274822 secs 2024-07-26T11:56:17.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:17 smithi164 ceph-mon[36970]: pgmap v10097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:18.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18024.274822 secs 2024-07-26T11:56:18.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:17 smithi163 ceph-mon[36873]: pgmap v10097: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:19.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:19 smithi042 ceph-mon[32775]: pgmap v10098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:56:19.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:56:19.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:56:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:56:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:19 smithi042 ceph-mon[32775]: pgmap v10099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:56:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:19 smithi042 ceph-mon[32775]: pgmap v10100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:56:19.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:56:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:19 smithi164 ceph-mon[36970]: pgmap v10098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:56:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:56:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:56:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:56:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:19 smithi164 ceph-mon[36970]: pgmap v10099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:56:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:19 smithi164 ceph-mon[36970]: pgmap v10100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:56:19.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:56:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:19 smithi163 ceph-mon[36873]: pgmap v10098: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:56:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:56:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:56:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:56:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:19 smithi163 ceph-mon[36873]: pgmap v10099: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T11:56:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:19 smithi163 ceph-mon[36873]: pgmap v10100: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:56:20.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:56:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:22 smithi042 ceph-mon[32775]: pgmap v10101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:56:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18011.536637 secs 2024-07-26T11:56:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18029.274966 secs 2024-07-26T11:56:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:22 smithi164 ceph-mon[36970]: pgmap v10101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:56:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18011.536637 secs 2024-07-26T11:56:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18029.274966 secs 2024-07-26T11:56:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:22 smithi163 ceph-mon[36873]: pgmap v10101: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:56:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18011.536637 secs 2024-07-26T11:56:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18029.274966 secs 2024-07-26T11:56:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:24 smithi042 ceph-mon[32775]: pgmap v10102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:56:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:24 smithi164 ceph-mon[36970]: pgmap v10102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:56:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:24 smithi163 ceph-mon[36873]: pgmap v10102: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:56:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:26 smithi042 ceph-mon[32775]: pgmap v10103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:56:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:26 smithi164 ceph-mon[36970]: pgmap v10103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:56:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:26 smithi163 ceph-mon[36873]: pgmap v10103: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:56:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18016.536784 secs 2024-07-26T11:56:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18034.275085 secs 2024-07-26T11:56:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18016.536784 secs 2024-07-26T11:56:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18034.275085 secs 2024-07-26T11:56:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18016.536784 secs 2024-07-26T11:56:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18034.275085 secs 2024-07-26T11:56:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:28 smithi042 ceph-mon[32775]: pgmap v10104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:56:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:28 smithi164 ceph-mon[36970]: pgmap v10104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:56:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:28 smithi163 ceph-mon[36873]: pgmap v10104: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:56:28.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:56:28.966+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:56:28.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:56:28.966+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:28.968 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:56:28.966+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:56:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:30 smithi042 ceph-mon[32775]: pgmap v10105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:30 smithi164 ceph-mon[36970]: pgmap v10105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:30 smithi163 ceph-mon[36873]: pgmap v10105: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:56:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:32 smithi164 ceph-mon[36970]: pgmap v10106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18021.536910 secs 2024-07-26T11:56:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18039.275210 secs 2024-07-26T11:56:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:32 smithi163 ceph-mon[36873]: pgmap v10106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18021.536910 secs 2024-07-26T11:56:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18039.275210 secs 2024-07-26T11:56:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:32 smithi042 ceph-mon[32775]: pgmap v10106: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18021.536910 secs 2024-07-26T11:56:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18039.275210 secs 2024-07-26T11:56:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:34 smithi164 ceph-mon[36970]: pgmap v10107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:34 smithi163 ceph-mon[36873]: pgmap v10107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:34 smithi042 ceph-mon[32775]: pgmap v10107: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:36 smithi164 ceph-mon[36970]: pgmap v10108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:36 smithi163 ceph-mon[36873]: pgmap v10108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:36 smithi042 ceph-mon[32775]: pgmap v10108: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18026.537033 secs 2024-07-26T11:56:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18044.275328 secs 2024-07-26T11:56:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18026.537033 secs 2024-07-26T11:56:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18044.275328 secs 2024-07-26T11:56:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18026.537033 secs 2024-07-26T11:56:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18044.275328 secs 2024-07-26T11:56:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:38 smithi164 ceph-mon[36970]: pgmap v10109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:38 smithi163 ceph-mon[36873]: pgmap v10109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:38 smithi042 ceph-mon[32775]: pgmap v10109: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:40 smithi164 ceph-mon[36970]: pgmap v10110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:40 smithi163 ceph-mon[36873]: pgmap v10110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:40 smithi042 ceph-mon[32775]: pgmap v10110: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:41 smithi164 ceph-mon[36970]: pgmap v10111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:41 smithi163 ceph-mon[36873]: pgmap v10111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:41 smithi042 ceph-mon[32775]: pgmap v10111: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18031.537133 secs 2024-07-26T11:56:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18049.275416 secs 2024-07-26T11:56:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18031.537133 secs 2024-07-26T11:56:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18049.275416 secs 2024-07-26T11:56:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18031.537133 secs 2024-07-26T11:56:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18049.275416 secs 2024-07-26T11:56:43.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:43 smithi163 ceph-mon[36873]: pgmap v10112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:43 smithi042 ceph-mon[32775]: pgmap v10112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:43 smithi164 ceph-mon[36970]: pgmap v10112: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:43.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:56:43.967+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:56:43.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:56:43.967+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:43.969 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:56:43.967+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:56:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:46 smithi042 ceph-mon[32775]: pgmap v10113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:46 smithi164 ceph-mon[36970]: pgmap v10113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:46 smithi163 ceph-mon[36873]: pgmap v10113: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18036.537293 secs 2024-07-26T11:56:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18054.275554 secs 2024-07-26T11:56:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18036.537293 secs 2024-07-26T11:56:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18054.275554 secs 2024-07-26T11:56:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18036.537293 secs 2024-07-26T11:56:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18054.275554 secs 2024-07-26T11:56:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:48 smithi042 ceph-mon[32775]: pgmap v10114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:48 smithi164 ceph-mon[36970]: pgmap v10114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:48 smithi163 ceph-mon[36873]: pgmap v10114: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:50 smithi164 ceph-mon[36970]: pgmap v10115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:50 smithi163 ceph-mon[36873]: pgmap v10115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:50 smithi042 ceph-mon[32775]: pgmap v10115: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18041.537416 secs 2024-07-26T11:56:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:52 smithi164 ceph-mon[36970]: pgmap v10116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18059.275714 secs 2024-07-26T11:56:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18041.537416 secs 2024-07-26T11:56:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:52 smithi163 ceph-mon[36873]: pgmap v10116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18059.275714 secs 2024-07-26T11:56:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18041.537416 secs 2024-07-26T11:56:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:52 smithi042 ceph-mon[32775]: pgmap v10116: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18059.275714 secs 2024-07-26T11:56:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:54 smithi164 ceph-mon[36970]: pgmap v10117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:54.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:54 smithi163 ceph-mon[36873]: pgmap v10117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:54.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:54 smithi042 ceph-mon[32775]: pgmap v10117: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:56 smithi164 ceph-mon[36970]: pgmap v10118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:56 smithi163 ceph-mon[36873]: pgmap v10118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:56 smithi042 ceph-mon[32775]: pgmap v10118: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:56:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18046.537540 secs 2024-07-26T11:56:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18064.275797 secs 2024-07-26T11:56:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18046.537540 secs 2024-07-26T11:56:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18064.275797 secs 2024-07-26T11:56:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18046.537540 secs 2024-07-26T11:56:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18064.275797 secs 2024-07-26T11:56:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:56:58 smithi164 ceph-mon[36970]: pgmap v10119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:58.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:56:58 smithi163 ceph-mon[36873]: pgmap v10119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:56:58 smithi042 ceph-mon[32775]: pgmap v10119: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:56:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:56:58.972+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:56:58.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:56:58.972+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:58.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:56:58.972+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:57:00.448 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:00 smithi042 ceph-mon[32775]: pgmap v10120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:00 smithi164 ceph-mon[36970]: pgmap v10120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:00 smithi163 ceph-mon[36873]: pgmap v10120: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18051.537660 secs 2024-07-26T11:57:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:02 smithi164 ceph-mon[36970]: pgmap v10121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18069.275923 secs 2024-07-26T11:57:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18051.537660 secs 2024-07-26T11:57:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:02 smithi163 ceph-mon[36873]: pgmap v10121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18069.275923 secs 2024-07-26T11:57:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18051.537660 secs 2024-07-26T11:57:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:02 smithi042 ceph-mon[32775]: pgmap v10121: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18069.275923 secs 2024-07-26T11:57:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:04 smithi164 ceph-mon[36970]: pgmap v10122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:04.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:04 smithi163 ceph-mon[36873]: pgmap v10122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:04.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:04 smithi042 ceph-mon[32775]: pgmap v10122: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:05.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:05 smithi163 ceph-mon[36873]: pgmap v10123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:05 smithi042 ceph-mon[32775]: pgmap v10123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:05 smithi164 ceph-mon[36970]: pgmap v10123: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18056.537758 secs 2024-07-26T11:57:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18056.537758 secs 2024-07-26T11:57:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18056.537758 secs 2024-07-26T11:57:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18074.276021 secs 2024-07-26T11:57:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:07 smithi164 ceph-mon[36970]: pgmap v10124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18074.276021 secs 2024-07-26T11:57:07.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:07 smithi163 ceph-mon[36873]: pgmap v10124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18074.276021 secs 2024-07-26T11:57:07.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:07 smithi042 ceph-mon[32775]: pgmap v10124: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:10 smithi042 ceph-mon[32775]: pgmap v10125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:10 smithi164 ceph-mon[36970]: pgmap v10125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:10.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:10 smithi163 ceph-mon[36873]: pgmap v10125: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18061.537898 secs 2024-07-26T11:57:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:12 smithi164 ceph-mon[36970]: pgmap v10126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18079.276155 secs 2024-07-26T11:57:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18061.537898 secs 2024-07-26T11:57:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:12 smithi163 ceph-mon[36873]: pgmap v10126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18079.276155 secs 2024-07-26T11:57:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18061.537898 secs 2024-07-26T11:57:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:12 smithi042 ceph-mon[32775]: pgmap v10126: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18079.276155 secs 2024-07-26T11:57:13.973 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:57:13.972+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:57:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:57:13.972+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:13.974 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:57:13.972+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:57:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:14 smithi164 ceph-mon[36970]: pgmap v10127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:14 smithi163 ceph-mon[36873]: pgmap v10127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:14 smithi042 ceph-mon[32775]: pgmap v10127: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:16 smithi164 ceph-mon[36970]: pgmap v10128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:16 smithi163 ceph-mon[36873]: pgmap v10128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:16 smithi042 ceph-mon[32775]: pgmap v10128: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18066.537994 secs 2024-07-26T11:57:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18084.276268 secs 2024-07-26T11:57:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18066.537994 secs 2024-07-26T11:57:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18084.276268 secs 2024-07-26T11:57:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18066.537994 secs 2024-07-26T11:57:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18084.276268 secs 2024-07-26T11:57:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:18 smithi164 ceph-mon[36970]: pgmap v10129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:18 smithi163 ceph-mon[36873]: pgmap v10129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:18 smithi042 ceph-mon[32775]: pgmap v10129: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:20 smithi164 ceph-mon[36970]: pgmap v10130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:57:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:57:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:57:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:57:20.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:20 smithi163 ceph-mon[36873]: pgmap v10130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:20.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:57:20.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:57:20.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:57:20.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:57:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:20 smithi042 ceph-mon[32775]: pgmap v10130: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:57:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:57:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:57:20.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:57:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:21 smithi164 ceph-mon[36970]: pgmap v10131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:57:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:21 smithi164 ceph-mon[36970]: pgmap v10132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:57:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:21 smithi163 ceph-mon[36873]: pgmap v10131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:57:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:21 smithi163 ceph-mon[36873]: pgmap v10132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:57:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:21 smithi042 ceph-mon[32775]: pgmap v10131: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:57:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:21 smithi042 ceph-mon[32775]: pgmap v10132: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:57:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18071.538111 secs 2024-07-26T11:57:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18089.276418 secs 2024-07-26T11:57:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18071.538111 secs 2024-07-26T11:57:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18089.276418 secs 2024-07-26T11:57:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18071.538111 secs 2024-07-26T11:57:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18089.276418 secs 2024-07-26T11:57:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:23 smithi164 ceph-mon[36970]: pgmap v10133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:57:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:23 smithi163 ceph-mon[36873]: pgmap v10133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:57:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:23 smithi042 ceph-mon[32775]: pgmap v10133: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:57:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:24 smithi164 ceph-mon[36970]: pgmap v10134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:57:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:24 smithi163 ceph-mon[36873]: pgmap v10134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:57:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:24 smithi042 ceph-mon[32775]: pgmap v10134: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:57:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:26 smithi164 ceph-mon[36970]: pgmap v10135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:57:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18076.538255 secs 2024-07-26T11:57:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:26 smithi163 ceph-mon[36873]: pgmap v10135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:57:27.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18076.538255 secs 2024-07-26T11:57:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:26 smithi042 ceph-mon[32775]: pgmap v10135: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:57:27.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18076.538255 secs 2024-07-26T11:57:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18094.276538 secs 2024-07-26T11:57:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18094.276538 secs 2024-07-26T11:57:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18094.276538 secs 2024-07-26T11:57:28.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:28 smithi164 ceph-mon[36970]: pgmap v10136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:57:28.979 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:57:28.978+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:57:28.979 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:57:28.978+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:57:28.980 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:57:28.978+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:29.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:28 smithi163 ceph-mon[36873]: pgmap v10136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:57:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:28 smithi042 ceph-mon[32775]: pgmap v10136: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:57:30.948 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:30 smithi042 ceph-mon[32775]: pgmap v10137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:57:30.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:30 smithi164 ceph-mon[36970]: pgmap v10137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:57:31.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:30 smithi163 ceph-mon[36873]: pgmap v10137: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T11:57:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18081.538378 secs 2024-07-26T11:57:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18081.538378 secs 2024-07-26T11:57:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18081.538378 secs 2024-07-26T11:57:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18099.276671 secs 2024-07-26T11:57:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:32 smithi164 ceph-mon[36970]: pgmap v10138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:57:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18099.276671 secs 2024-07-26T11:57:33.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:32 smithi163 ceph-mon[36873]: pgmap v10138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:57:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18099.276671 secs 2024-07-26T11:57:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:32 smithi042 ceph-mon[32775]: pgmap v10138: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:57:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:34 smithi164 ceph-mon[36970]: pgmap v10139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:57:35.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:34 smithi163 ceph-mon[36873]: pgmap v10139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:57:35.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:34 smithi042 ceph-mon[32775]: pgmap v10139: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:57:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:36 smithi164 ceph-mon[36970]: pgmap v10140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:57:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18086.538479 secs 2024-07-26T11:57:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:36 smithi163 ceph-mon[36873]: pgmap v10140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:57:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18086.538479 secs 2024-07-26T11:57:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:36 smithi042 ceph-mon[32775]: pgmap v10140: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:57:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18086.538479 secs 2024-07-26T11:57:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18104.276790 secs 2024-07-26T11:57:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18104.276790 secs 2024-07-26T11:57:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18104.276790 secs 2024-07-26T11:57:38.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:38 smithi164 ceph-mon[36970]: pgmap v10141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:57:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:38 smithi163 ceph-mon[36873]: pgmap v10141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:57:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:38 smithi042 ceph-mon[32775]: pgmap v10141: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T11:57:41.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:40 smithi163 ceph-mon[36873]: pgmap v10142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:57:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:40 smithi042 ceph-mon[32775]: pgmap v10142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:57:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:40 smithi164 ceph-mon[36970]: pgmap v10142: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:57:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18091.538606 secs 2024-07-26T11:57:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18091.538606 secs 2024-07-26T11:57:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18091.538606 secs 2024-07-26T11:57:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18109.276954 secs 2024-07-26T11:57:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:42 smithi163 ceph-mon[36873]: pgmap v10143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18109.276954 secs 2024-07-26T11:57:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:42 smithi042 ceph-mon[32775]: pgmap v10143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18109.276954 secs 2024-07-26T11:57:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:42 smithi164 ceph-mon[36970]: pgmap v10143: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:43.981 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:57:43.980+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:57:43.982 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:57:43.980+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:43.982 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:57:43.980+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:57:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:44 smithi163 ceph-mon[36873]: pgmap v10144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:57:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:44 smithi042 ceph-mon[32775]: pgmap v10144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:57:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:44 smithi164 ceph-mon[36970]: pgmap v10144: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:57:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:46 smithi163 ceph-mon[36873]: pgmap v10145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:47.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18096.538754 secs 2024-07-26T11:57:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:46 smithi042 ceph-mon[32775]: pgmap v10145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18096.538754 secs 2024-07-26T11:57:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:46 smithi164 ceph-mon[36970]: pgmap v10145: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18096.538754 secs 2024-07-26T11:57:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18114.277072 secs 2024-07-26T11:57:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18114.277072 secs 2024-07-26T11:57:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18114.277072 secs 2024-07-26T11:57:49.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:48 smithi163 ceph-mon[36873]: pgmap v10146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:48 smithi042 ceph-mon[32775]: pgmap v10146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:48 smithi164 ceph-mon[36970]: pgmap v10146: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:50 smithi163 ceph-mon[36873]: pgmap v10147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:51.025 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:50 smithi164 ceph-mon[36970]: pgmap v10147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:50 smithi042 ceph-mon[32775]: pgmap v10147: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18101.538886 secs 2024-07-26T11:57:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18101.538886 secs 2024-07-26T11:57:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18101.538886 secs 2024-07-26T11:57:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18119.277219 secs 2024-07-26T11:57:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:52 smithi163 ceph-mon[36873]: pgmap v10148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18119.277219 secs 2024-07-26T11:57:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:52 smithi042 ceph-mon[32775]: pgmap v10148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18119.277219 secs 2024-07-26T11:57:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:52 smithi164 ceph-mon[36970]: pgmap v10148: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:55.056 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:54 smithi042 ceph-mon[32775]: pgmap v10149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:57:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:54 smithi164 ceph-mon[36970]: pgmap v10149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:57:55.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:54 smithi163 ceph-mon[36873]: pgmap v10149: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:57:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:56 smithi042 ceph-mon[32775]: pgmap v10150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:57.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18106.539020 secs 2024-07-26T11:57:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:56 smithi164 ceph-mon[36970]: pgmap v10150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18106.539020 secs 2024-07-26T11:57:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:56 smithi163 ceph-mon[36873]: pgmap v10150: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:57:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18106.539020 secs 2024-07-26T11:57:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18124.277338 secs 2024-07-26T11:57:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18124.277338 secs 2024-07-26T11:57:58.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18124.277338 secs 2024-07-26T11:57:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:57:58.985+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:57:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:57:58.985+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:57:58.985+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:57:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:57:58 smithi042 ceph-mon[32775]: pgmap v10151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:57:58 smithi164 ceph-mon[36970]: pgmap v10151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:57:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:57:58 smithi163 ceph-mon[36873]: pgmap v10151: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:00 smithi042 ceph-mon[32775]: pgmap v10152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:01.225 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:00 smithi164 ceph-mon[36970]: pgmap v10152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:00 smithi163 ceph-mon[36873]: pgmap v10152: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18111.539139 secs 2024-07-26T11:58:02.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18111.539139 secs 2024-07-26T11:58:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18111.539139 secs 2024-07-26T11:58:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18129.277417 secs 2024-07-26T11:58:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:02 smithi042 ceph-mon[32775]: pgmap v10153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18129.277417 secs 2024-07-26T11:58:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:02 smithi164 ceph-mon[36970]: pgmap v10153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18129.277417 secs 2024-07-26T11:58:03.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:02 smithi163 ceph-mon[36873]: pgmap v10153: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:04 smithi042 ceph-mon[32775]: pgmap v10154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:58:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:04 smithi164 ceph-mon[36970]: pgmap v10154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:58:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:04 smithi163 ceph-mon[36873]: pgmap v10154: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:58:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:06 smithi042 ceph-mon[32775]: pgmap v10155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:07.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18116.539263 secs 2024-07-26T11:58:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:06 smithi164 ceph-mon[36970]: pgmap v10155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18116.539263 secs 2024-07-26T11:58:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:06 smithi163 ceph-mon[36873]: pgmap v10155: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18116.539263 secs 2024-07-26T11:58:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18134.277538 secs 2024-07-26T11:58:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18134.277538 secs 2024-07-26T11:58:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18134.277538 secs 2024-07-26T11:58:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:08 smithi042 ceph-mon[32775]: pgmap v10156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:08 smithi164 ceph-mon[36970]: pgmap v10156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:08 smithi163 ceph-mon[36873]: pgmap v10156: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:10 smithi042 ceph-mon[32775]: pgmap v10157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:10 smithi164 ceph-mon[36970]: pgmap v10157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:11.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:10 smithi163 ceph-mon[36873]: pgmap v10157: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18121.539387 secs 2024-07-26T11:58:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18121.539387 secs 2024-07-26T11:58:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18121.539387 secs 2024-07-26T11:58:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18139.277665 secs 2024-07-26T11:58:13.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:12 smithi042 ceph-mon[32775]: pgmap v10158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18139.277665 secs 2024-07-26T11:58:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:12 smithi164 ceph-mon[36970]: pgmap v10158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18139.277665 secs 2024-07-26T11:58:13.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:12 smithi163 ceph-mon[36873]: pgmap v10158: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:58:13.986+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:58:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:58:13.986+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:13.987 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:58:13.986+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:58:15.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:14 smithi042 ceph-mon[32775]: pgmap v10159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:58:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:14 smithi164 ceph-mon[36970]: pgmap v10159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:58:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:14 smithi163 ceph-mon[36873]: pgmap v10159: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:58:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:16 smithi042 ceph-mon[32775]: pgmap v10160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18126.539516 secs 2024-07-26T11:58:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:16 smithi164 ceph-mon[36970]: pgmap v10160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18126.539516 secs 2024-07-26T11:58:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:16 smithi163 ceph-mon[36873]: pgmap v10160: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18126.539516 secs 2024-07-26T11:58:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18144.277765 secs 2024-07-26T11:58:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18144.277765 secs 2024-07-26T11:58:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18144.277765 secs 2024-07-26T11:58:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:18 smithi042 ceph-mon[32775]: pgmap v10161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:18 smithi164 ceph-mon[36970]: pgmap v10161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:18 smithi163 ceph-mon[36873]: pgmap v10161: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:19 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:58:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:19 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:58:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:19 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:58:21.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:20 smithi164 ceph-mon[36970]: pgmap v10162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:21.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:58:21.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:58:21.125 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:58:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:20 smithi042 ceph-mon[32775]: pgmap v10162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:58:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:58:21.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:58:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:20 smithi163 ceph-mon[36873]: pgmap v10162: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:58:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:58:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:58:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:21 smithi042 ceph-mon[32775]: pgmap v10163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:58:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:21 smithi042 ceph-mon[32775]: pgmap v10164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:58:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18131.539597 secs 2024-07-26T11:58:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18149.277895 secs 2024-07-26T11:58:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:21 smithi164 ceph-mon[36970]: pgmap v10163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:58:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:21 smithi164 ceph-mon[36970]: pgmap v10164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:58:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18131.539597 secs 2024-07-26T11:58:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18149.277895 secs 2024-07-26T11:58:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:21 smithi163 ceph-mon[36873]: pgmap v10163: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:58:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:21 smithi163 ceph-mon[36873]: pgmap v10164: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:58:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18131.539597 secs 2024-07-26T11:58:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18149.277895 secs 2024-07-26T11:58:24.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:23 smithi042 ceph-mon[32775]: pgmap v10165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:58:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:23 smithi164 ceph-mon[36970]: pgmap v10165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:58:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:23 smithi163 ceph-mon[36873]: pgmap v10165: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T11:58:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:25 smithi042 ceph-mon[32775]: pgmap v10166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:58:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:25 smithi164 ceph-mon[36970]: pgmap v10166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:58:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:25 smithi163 ceph-mon[36873]: pgmap v10166: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:58:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18136.539724 secs 2024-07-26T11:58:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18154.278014 secs 2024-07-26T11:58:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18136.539724 secs 2024-07-26T11:58:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18154.278014 secs 2024-07-26T11:58:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18136.539724 secs 2024-07-26T11:58:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18154.278014 secs 2024-07-26T11:58:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:27 smithi042 ceph-mon[32775]: pgmap v10167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:58:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:27 smithi164 ceph-mon[36970]: pgmap v10167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:58:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:27 smithi163 ceph-mon[36873]: pgmap v10167: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:58:28.992 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:58:28.991+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:58:28.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:58:28.991+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:28.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:58:28.991+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:58:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:29 smithi042 ceph-mon[32775]: pgmap v10168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:58:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:29 smithi164 ceph-mon[36970]: pgmap v10168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:58:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:29 smithi163 ceph-mon[36873]: pgmap v10168: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:58:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:31 smithi042 ceph-mon[32775]: pgmap v10169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:58:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18141.539872 secs 2024-07-26T11:58:32.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18159.278160 secs 2024-07-26T11:58:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:31 smithi164 ceph-mon[36970]: pgmap v10169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:58:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18141.539872 secs 2024-07-26T11:58:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18159.278160 secs 2024-07-26T11:58:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:31 smithi163 ceph-mon[36873]: pgmap v10169: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:58:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18141.539872 secs 2024-07-26T11:58:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18159.278160 secs 2024-07-26T11:58:34.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:33 smithi042 ceph-mon[32775]: pgmap v10170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:34.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:33 smithi164 ceph-mon[36970]: pgmap v10170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:33 smithi163 ceph-mon[36873]: pgmap v10170: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:35 smithi042 ceph-mon[32775]: pgmap v10171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:35 smithi164 ceph-mon[36970]: pgmap v10171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:35 smithi163 ceph-mon[36873]: pgmap v10171: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18146.540018 secs 2024-07-26T11:58:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18164.278300 secs 2024-07-26T11:58:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18146.540018 secs 2024-07-26T11:58:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18164.278300 secs 2024-07-26T11:58:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18146.540018 secs 2024-07-26T11:58:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18164.278300 secs 2024-07-26T11:58:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:37 smithi042 ceph-mon[32775]: pgmap v10172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:37 smithi164 ceph-mon[36970]: pgmap v10172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:37 smithi163 ceph-mon[36873]: pgmap v10172: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:39 smithi042 ceph-mon[32775]: pgmap v10173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:39 smithi164 ceph-mon[36970]: pgmap v10173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:39 smithi163 ceph-mon[36873]: pgmap v10173: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:41 smithi042 ceph-mon[32775]: pgmap v10174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18151.540156 secs 2024-07-26T11:58:42.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18169.278454 secs 2024-07-26T11:58:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:41 smithi164 ceph-mon[36970]: pgmap v10174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18151.540156 secs 2024-07-26T11:58:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18169.278454 secs 2024-07-26T11:58:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:41 smithi163 ceph-mon[36873]: pgmap v10174: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18151.540156 secs 2024-07-26T11:58:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18169.278454 secs 2024-07-26T11:58:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:58:43.992+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:58:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:58:43.992+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:58:43.992+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:58:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:43 smithi042 ceph-mon[32775]: pgmap v10175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:43 smithi164 ceph-mon[36970]: pgmap v10175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:43 smithi163 ceph-mon[36873]: pgmap v10175: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:46.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:45 smithi042 ceph-mon[32775]: pgmap v10176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:45 smithi164 ceph-mon[36970]: pgmap v10176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:45 smithi163 ceph-mon[36873]: pgmap v10176: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18156.540241 secs 2024-07-26T11:58:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18174.278596 secs 2024-07-26T11:58:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18156.540241 secs 2024-07-26T11:58:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18174.278596 secs 2024-07-26T11:58:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18156.540241 secs 2024-07-26T11:58:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18174.278596 secs 2024-07-26T11:58:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:47 smithi042 ceph-mon[32775]: pgmap v10177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:47 smithi164 ceph-mon[36970]: pgmap v10177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:47 smithi163 ceph-mon[36873]: pgmap v10177: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:49 smithi164 ceph-mon[36970]: pgmap v10178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:50.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:49 smithi163 ceph-mon[36873]: pgmap v10178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:49 smithi042 ceph-mon[32775]: pgmap v10178: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:51 smithi164 ceph-mon[36970]: pgmap v10179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18161.540364 secs 2024-07-26T11:58:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18179.278722 secs 2024-07-26T11:58:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:51 smithi163 ceph-mon[36873]: pgmap v10179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18161.540364 secs 2024-07-26T11:58:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18179.278722 secs 2024-07-26T11:58:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:51 smithi042 ceph-mon[32775]: pgmap v10179: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:52.432 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18161.540364 secs 2024-07-26T11:58:52.433 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18179.278722 secs 2024-07-26T11:58:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:53 smithi164 ceph-mon[36970]: pgmap v10180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:53 smithi163 ceph-mon[36873]: pgmap v10180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:53 smithi042 ceph-mon[32775]: pgmap v10180: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:55 smithi164 ceph-mon[36970]: pgmap v10181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:56.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:55 smithi163 ceph-mon[36873]: pgmap v10181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:55 smithi042 ceph-mon[32775]: pgmap v10181: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:58:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18166.540487 secs 2024-07-26T11:58:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18184.278866 secs 2024-07-26T11:58:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18166.540487 secs 2024-07-26T11:58:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18184.278866 secs 2024-07-26T11:58:57.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18166.540487 secs 2024-07-26T11:58:57.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18184.278866 secs 2024-07-26T11:58:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:57 smithi164 ceph-mon[36970]: pgmap v10182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:57 smithi163 ceph-mon[36873]: pgmap v10182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:57 smithi042 ceph-mon[32775]: pgmap v10182: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:58:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:58:58.997+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:58:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:58:58.997+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:58.998 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:58:58.997+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:59:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:58:59 smithi164 ceph-mon[36970]: pgmap v10183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:00.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:58:59 smithi163 ceph-mon[36873]: pgmap v10183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:58:59 smithi042 ceph-mon[32775]: pgmap v10183: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:00.733 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:59:00.731+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:59:00.733 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:59:00.731+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:59:00.733 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:59:00.731+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T11:59:00.733 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:59:00.731+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T11:59:02.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:01 smithi042 ceph-mon[32775]: pgmap v10184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:02.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18171.540611 secs 2024-07-26T11:59:02.198 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18189.278958 secs 2024-07-26T11:59:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:01 smithi164 ceph-mon[36970]: pgmap v10184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18171.540611 secs 2024-07-26T11:59:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18189.278958 secs 2024-07-26T11:59:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:01 smithi163 ceph-mon[36873]: pgmap v10184: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18171.540611 secs 2024-07-26T11:59:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18189.278958 secs 2024-07-26T11:59:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:03 smithi164 ceph-mon[36970]: pgmap v10185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:03 smithi163 ceph-mon[36873]: pgmap v10185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:03 smithi042 ceph-mon[32775]: pgmap v10185: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:05 smithi164 ceph-mon[36970]: pgmap v10186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:05 smithi163 ceph-mon[36873]: pgmap v10186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:05 smithi042 ceph-mon[32775]: pgmap v10186: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18176.540737 secs 2024-07-26T11:59:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18194.279020 secs 2024-07-26T11:59:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18176.540737 secs 2024-07-26T11:59:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18194.279020 secs 2024-07-26T11:59:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18176.540737 secs 2024-07-26T11:59:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18194.279020 secs 2024-07-26T11:59:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:07 smithi164 ceph-mon[36970]: pgmap v10187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:07 smithi163 ceph-mon[36873]: pgmap v10187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:07 smithi042 ceph-mon[32775]: pgmap v10187: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:09 smithi164 ceph-mon[36970]: pgmap v10188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:09 smithi163 ceph-mon[36873]: pgmap v10188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:09 smithi042 ceph-mon[32775]: pgmap v10188: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:11 smithi042 ceph-mon[32775]: pgmap v10189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18181.540871 secs 2024-07-26T11:59:12.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18199.279124 secs 2024-07-26T11:59:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:11 smithi164 ceph-mon[36970]: pgmap v10189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18181.540871 secs 2024-07-26T11:59:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18199.279124 secs 2024-07-26T11:59:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:11 smithi163 ceph-mon[36873]: pgmap v10189: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18181.540871 secs 2024-07-26T11:59:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18199.279124 secs 2024-07-26T11:59:13.999 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:59:13.997+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:59:13.999 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:59:13.997+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:13.999 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:59:13.997+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:59:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:13 smithi164 ceph-mon[36970]: pgmap v10190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:13 smithi163 ceph-mon[36873]: pgmap v10190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:13 smithi042 ceph-mon[32775]: pgmap v10190: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:15 smithi164 ceph-mon[36970]: pgmap v10191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:15 smithi163 ceph-mon[36873]: pgmap v10191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:15 smithi042 ceph-mon[32775]: pgmap v10191: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18186.540995 secs 2024-07-26T11:59:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18204.279249 secs 2024-07-26T11:59:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18186.540995 secs 2024-07-26T11:59:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18204.279249 secs 2024-07-26T11:59:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18186.540995 secs 2024-07-26T11:59:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18204.279249 secs 2024-07-26T11:59:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:17 smithi164 ceph-mon[36970]: pgmap v10192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:17 smithi163 ceph-mon[36873]: pgmap v10192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:17 smithi042 ceph-mon[32775]: pgmap v10192: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:19 smithi164 ceph-mon[36970]: pgmap v10193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:19 smithi163 ceph-mon[36873]: pgmap v10193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:19 smithi042 ceph-mon[32775]: pgmap v10193: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:59:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:59:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:59:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:20 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:59:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:59:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:59:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:59:21.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:20 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:59:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:59:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:59:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:59:21.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:20 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T11:59:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:21 smithi164 ceph-mon[36970]: pgmap v10194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:21 smithi164 ceph-mon[36970]: pgmap v10195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:59:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:21 smithi164 ceph-mon[36970]: pgmap v10196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:59:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18191.541112 secs 2024-07-26T11:59:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18209.279372 secs 2024-07-26T11:59:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:21 smithi163 ceph-mon[36873]: pgmap v10194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:21 smithi163 ceph-mon[36873]: pgmap v10195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:59:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:21 smithi163 ceph-mon[36873]: pgmap v10196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:59:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18191.541112 secs 2024-07-26T11:59:22.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18209.279372 secs 2024-07-26T11:59:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:21 smithi042 ceph-mon[32775]: pgmap v10194: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:21 smithi042 ceph-mon[32775]: pgmap v10195: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T11:59:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:21 smithi042 ceph-mon[32775]: pgmap v10196: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:59:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18191.541112 secs 2024-07-26T11:59:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18209.279372 secs 2024-07-26T11:59:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:23 smithi164 ceph-mon[36970]: pgmap v10197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:59:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:23 smithi163 ceph-mon[36873]: pgmap v10197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:59:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:23 smithi042 ceph-mon[32775]: pgmap v10197: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T11:59:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:25 smithi164 ceph-mon[36970]: pgmap v10198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:59:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:25 smithi163 ceph-mon[36873]: pgmap v10198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:59:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:25 smithi042 ceph-mon[32775]: pgmap v10198: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:59:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18196.541260 secs 2024-07-26T11:59:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18214.279491 secs 2024-07-26T11:59:27.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18196.541260 secs 2024-07-26T11:59:27.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18214.279491 secs 2024-07-26T11:59:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18196.541260 secs 2024-07-26T11:59:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18214.279491 secs 2024-07-26T11:59:28.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:27 smithi163 ceph-mon[36873]: pgmap v10199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:59:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:27 smithi042 ceph-mon[32775]: pgmap v10199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:59:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:27 smithi164 ceph-mon[36970]: pgmap v10199: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T11:59:29.004 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:59:29.002+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:59:29.004 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:59:29.002+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:29.004 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:59:29.002+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:59:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:29 smithi163 ceph-mon[36873]: pgmap v10200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:59:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:29 smithi042 ceph-mon[32775]: pgmap v10200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:59:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:29 smithi164 ceph-mon[36970]: pgmap v10200: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T11:59:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:31 smithi163 ceph-mon[36873]: pgmap v10201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:59:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18201.541375 secs 2024-07-26T11:59:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18219.279616 secs 2024-07-26T11:59:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:31 smithi042 ceph-mon[32775]: pgmap v10201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:59:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18201.541375 secs 2024-07-26T11:59:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18219.279616 secs 2024-07-26T11:59:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:31 smithi164 ceph-mon[36970]: pgmap v10201: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T11:59:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18201.541375 secs 2024-07-26T11:59:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18219.279616 secs 2024-07-26T11:59:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:33 smithi163 ceph-mon[36873]: pgmap v10202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:33 smithi042 ceph-mon[32775]: pgmap v10202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:33 smithi164 ceph-mon[36970]: pgmap v10202: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:36.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:35 smithi163 ceph-mon[36873]: pgmap v10203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:35 smithi042 ceph-mon[32775]: pgmap v10203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:35 smithi164 ceph-mon[36970]: pgmap v10203: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18206.541525 secs 2024-07-26T11:59:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18224.279751 secs 2024-07-26T11:59:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18206.541525 secs 2024-07-26T11:59:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18224.279751 secs 2024-07-26T11:59:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18206.541525 secs 2024-07-26T11:59:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18224.279751 secs 2024-07-26T11:59:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:38 smithi163 ceph-mon[36873]: pgmap v10204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:38 smithi042 ceph-mon[32775]: pgmap v10204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:38 smithi164 ceph-mon[36970]: pgmap v10204: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:40.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:40 smithi163 ceph-mon[36873]: pgmap v10205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:59:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:40 smithi042 ceph-mon[32775]: pgmap v10205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:59:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:40 smithi164 ceph-mon[36970]: pgmap v10205: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:59:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:42 smithi163 ceph-mon[36873]: pgmap v10206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18211.541687 secs 2024-07-26T11:59:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18229.279876 secs 2024-07-26T11:59:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:42 smithi042 ceph-mon[32775]: pgmap v10206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18211.541687 secs 2024-07-26T11:59:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18229.279876 secs 2024-07-26T11:59:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:42 smithi164 ceph-mon[36970]: pgmap v10206: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18211.541687 secs 2024-07-26T11:59:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18229.279876 secs 2024-07-26T11:59:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:59:44.003+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:59:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:59:44.004+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:59:44.004+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T11:59:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:44 smithi042 ceph-mon[32775]: pgmap v10207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:44 smithi164 ceph-mon[36970]: pgmap v10207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:44 smithi163 ceph-mon[36873]: pgmap v10207: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:46 smithi042 ceph-mon[32775]: pgmap v10208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:46 smithi164 ceph-mon[36970]: pgmap v10208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:46 smithi163 ceph-mon[36873]: pgmap v10208: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18216.541831 secs 2024-07-26T11:59:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18234.279999 secs 2024-07-26T11:59:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18216.541831 secs 2024-07-26T11:59:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18234.279999 secs 2024-07-26T11:59:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18216.541831 secs 2024-07-26T11:59:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18234.279999 secs 2024-07-26T11:59:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:48 smithi042 ceph-mon[32775]: pgmap v10209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:48 smithi164 ceph-mon[36970]: pgmap v10209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:48 smithi163 ceph-mon[36873]: pgmap v10209: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:50 smithi042 ceph-mon[32775]: pgmap v10210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:59:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:50 smithi164 ceph-mon[36970]: pgmap v10210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:59:50.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:50 smithi163 ceph-mon[36873]: pgmap v10210: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T11:59:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:52 smithi042 ceph-mon[32775]: pgmap v10211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18221.541962 secs 2024-07-26T11:59:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18239.280115 secs 2024-07-26T11:59:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:52 smithi164 ceph-mon[36970]: pgmap v10211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18221.541962 secs 2024-07-26T11:59:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18239.280115 secs 2024-07-26T11:59:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:52 smithi163 ceph-mon[36873]: pgmap v10211: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18221.541962 secs 2024-07-26T11:59:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18239.280115 secs 2024-07-26T11:59:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:54 smithi042 ceph-mon[32775]: pgmap v10212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:54 smithi164 ceph-mon[36970]: pgmap v10212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:54 smithi163 ceph-mon[36873]: pgmap v10212: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:56 smithi042 ceph-mon[32775]: pgmap v10213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:56 smithi164 ceph-mon[36970]: pgmap v10213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:56 smithi163 ceph-mon[36873]: pgmap v10213: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T11:59:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18226.542094 secs 2024-07-26T11:59:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18244.280235 secs 2024-07-26T11:59:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18226.542094 secs 2024-07-26T11:59:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18244.280235 secs 2024-07-26T11:59:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18226.542094 secs 2024-07-26T11:59:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18244.280235 secs 2024-07-26T11:59:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 11:59:58 smithi042 ceph-mon[32775]: pgmap v10214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 11:59:58 smithi164 ceph-mon[36970]: pgmap v10214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 11:59:58 smithi163 ceph-mon[36873]: pgmap v10214: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T11:59:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:59:59.010+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T11:59:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:59:59.010+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T11:59:59.010+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:00:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:00 smithi042 ceph-mon[32775]: pgmap v10215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T12:00:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:00 smithi164 ceph-mon[36970]: pgmap v10215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T12:00:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:00 smithi163 ceph-mon[36873]: pgmap v10215: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T12:00:02.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:02 smithi163 ceph-mon[36873]: pgmap v10216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:02.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18231.542235 secs 2024-07-26T12:00:02.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18249.280361 secs 2024-07-26T12:00:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:02 smithi042 ceph-mon[32775]: pgmap v10216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18231.542235 secs 2024-07-26T12:00:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18249.280361 secs 2024-07-26T12:00:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:02 smithi164 ceph-mon[36970]: pgmap v10216: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18231.542235 secs 2024-07-26T12:00:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18249.280361 secs 2024-07-26T12:00:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:04 smithi042 ceph-mon[32775]: pgmap v10217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:04 smithi164 ceph-mon[36970]: pgmap v10217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:04.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:04 smithi163 ceph-mon[36873]: pgmap v10217: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:06 smithi042 ceph-mon[32775]: pgmap v10218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:06 smithi164 ceph-mon[36970]: pgmap v10218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:06 smithi163 ceph-mon[36873]: pgmap v10218: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18236.542361 secs 2024-07-26T12:00:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18254.280501 secs 2024-07-26T12:00:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18236.542361 secs 2024-07-26T12:00:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18254.280501 secs 2024-07-26T12:00:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18236.542361 secs 2024-07-26T12:00:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18254.280501 secs 2024-07-26T12:00:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:08 smithi042 ceph-mon[32775]: pgmap v10219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:08 smithi164 ceph-mon[36970]: pgmap v10219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:08 smithi163 ceph-mon[36873]: pgmap v10219: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:10 smithi042 ceph-mon[32775]: pgmap v10220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:10 smithi164 ceph-mon[36970]: pgmap v10220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:10 smithi163 ceph-mon[36873]: pgmap v10220: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:12 smithi042 ceph-mon[32775]: pgmap v10221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18241.542476 secs 2024-07-26T12:00:12.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18259.280573 secs 2024-07-26T12:00:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:12 smithi164 ceph-mon[36970]: pgmap v10221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18241.542476 secs 2024-07-26T12:00:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18259.280573 secs 2024-07-26T12:00:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:12 smithi163 ceph-mon[36873]: pgmap v10221: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18241.542476 secs 2024-07-26T12:00:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18259.280573 secs 2024-07-26T12:00:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:00:14.011+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:00:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:00:14.011+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:14.013 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:00:14.011+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:00:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:14 smithi042 ceph-mon[32775]: pgmap v10222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:14 smithi164 ceph-mon[36970]: pgmap v10222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:14 smithi163 ceph-mon[36873]: pgmap v10222: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:16 smithi042 ceph-mon[32775]: pgmap v10223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:16 smithi164 ceph-mon[36970]: pgmap v10223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:16.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:16 smithi163 ceph-mon[36873]: pgmap v10223: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18246.542547 secs 2024-07-26T12:00:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18264.280693 secs 2024-07-26T12:00:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18246.542547 secs 2024-07-26T12:00:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18264.280693 secs 2024-07-26T12:00:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18246.542547 secs 2024-07-26T12:00:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18264.280693 secs 2024-07-26T12:00:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:18 smithi042 ceph-mon[32775]: pgmap v10224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:18 smithi164 ceph-mon[36970]: pgmap v10224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:18 smithi163 ceph-mon[36873]: pgmap v10224: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:20 smithi042 ceph-mon[32775]: pgmap v10225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:20 smithi164 ceph-mon[36970]: pgmap v10225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:20 smithi163 ceph-mon[36873]: pgmap v10225: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:21 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:00:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:21 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:00:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:21 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:00:22.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:22 smithi164 ceph-mon[36970]: pgmap v10226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:22.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18251.542641 secs 2024-07-26T12:00:22.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:00:22.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:00:22.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:00:22.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18269.280781 secs 2024-07-26T12:00:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:22 smithi042 ceph-mon[32775]: pgmap v10226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18251.542641 secs 2024-07-26T12:00:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:00:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:00:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:00:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18269.280781 secs 2024-07-26T12:00:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:22 smithi163 ceph-mon[36873]: pgmap v10226: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18251.542641 secs 2024-07-26T12:00:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:00:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:00:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:00:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18269.280781 secs 2024-07-26T12:00:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:23 smithi042 ceph-mon[32775]: pgmap v10227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:00:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:23 smithi042 ceph-mon[32775]: pgmap v10228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:00:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:23 smithi164 ceph-mon[36970]: pgmap v10227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:00:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:23 smithi164 ceph-mon[36970]: pgmap v10228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:00:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:23 smithi163 ceph-mon[36873]: pgmap v10227: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:00:23.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:23 smithi163 ceph-mon[36873]: pgmap v10228: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:00:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:25 smithi042 ceph-mon[32775]: pgmap v10229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:00:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:25 smithi164 ceph-mon[36970]: pgmap v10229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:00:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:25 smithi163 ceph-mon[36873]: pgmap v10229: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:00:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:27 smithi042 ceph-mon[32775]: pgmap v10230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:00:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18256.542789 secs 2024-07-26T12:00:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18274.280896 secs 2024-07-26T12:00:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:27 smithi164 ceph-mon[36970]: pgmap v10230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:00:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18256.542789 secs 2024-07-26T12:00:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18274.280896 secs 2024-07-26T12:00:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:27 smithi163 ceph-mon[36873]: pgmap v10230: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:00:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18256.542789 secs 2024-07-26T12:00:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18274.280896 secs 2024-07-26T12:00:29.017 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:00:29.016+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:00:29.018 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:00:29.016+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:00:29.018 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:00:29.016+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:29 smithi042 ceph-mon[32775]: pgmap v10231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:00:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:29 smithi164 ceph-mon[36970]: pgmap v10231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:00:29.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:29 smithi163 ceph-mon[36873]: pgmap v10231: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:00:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:31 smithi042 ceph-mon[32775]: pgmap v10232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:00:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:31 smithi164 ceph-mon[36970]: pgmap v10232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:00:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:31 smithi163 ceph-mon[36873]: pgmap v10232: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:00:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18261.542868 secs 2024-07-26T12:00:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18279.281021 secs 2024-07-26T12:00:32.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18261.542868 secs 2024-07-26T12:00:32.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18279.281021 secs 2024-07-26T12:00:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18261.542868 secs 2024-07-26T12:00:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18279.281021 secs 2024-07-26T12:00:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:33 smithi042 ceph-mon[32775]: pgmap v10233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:00:33.427 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:33 smithi163 ceph-mon[36873]: pgmap v10233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:00:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:33 smithi164 ceph-mon[36970]: pgmap v10233: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:00:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:35 smithi042 ceph-mon[32775]: pgmap v10234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:35 smithi164 ceph-mon[36970]: pgmap v10234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:35 smithi163 ceph-mon[36873]: pgmap v10234: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:37 smithi042 ceph-mon[32775]: pgmap v10235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18266.542996 secs 2024-07-26T12:00:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18284.281133 secs 2024-07-26T12:00:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:37 smithi164 ceph-mon[36970]: pgmap v10235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18266.542996 secs 2024-07-26T12:00:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18284.281133 secs 2024-07-26T12:00:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:37 smithi163 ceph-mon[36873]: pgmap v10235: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18266.542996 secs 2024-07-26T12:00:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18284.281133 secs 2024-07-26T12:00:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:39 smithi042 ceph-mon[32775]: pgmap v10236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:39 smithi164 ceph-mon[36970]: pgmap v10236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:39 smithi163 ceph-mon[36873]: pgmap v10236: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:41 smithi042 ceph-mon[32775]: pgmap v10237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:41 smithi164 ceph-mon[36970]: pgmap v10237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:41 smithi163 ceph-mon[36873]: pgmap v10237: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18271.543101 secs 2024-07-26T12:00:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18289.281230 secs 2024-07-26T12:00:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18271.543101 secs 2024-07-26T12:00:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18289.281230 secs 2024-07-26T12:00:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18271.543101 secs 2024-07-26T12:00:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18289.281230 secs 2024-07-26T12:00:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:43 smithi164 ceph-mon[36970]: pgmap v10238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:43 smithi163 ceph-mon[36873]: pgmap v10238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:43 smithi042 ceph-mon[32775]: pgmap v10238: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:00:44.017+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:00:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:00:44.017+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:00:44.017+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:00:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:45 smithi042 ceph-mon[32775]: pgmap v10239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:45 smithi164 ceph-mon[36970]: pgmap v10239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:45 smithi163 ceph-mon[36873]: pgmap v10239: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:47 smithi164 ceph-mon[36970]: pgmap v10240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18276.543248 secs 2024-07-26T12:00:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18294.281350 secs 2024-07-26T12:00:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:47 smithi163 ceph-mon[36873]: pgmap v10240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18276.543248 secs 2024-07-26T12:00:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18294.281350 secs 2024-07-26T12:00:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:47 smithi042 ceph-mon[32775]: pgmap v10240: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18276.543248 secs 2024-07-26T12:00:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18294.281350 secs 2024-07-26T12:00:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:49 smithi164 ceph-mon[36970]: pgmap v10241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:49 smithi163 ceph-mon[36873]: pgmap v10241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:49 smithi042 ceph-mon[32775]: pgmap v10241: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:51 smithi164 ceph-mon[36970]: pgmap v10242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:51 smithi163 ceph-mon[36873]: pgmap v10242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:51 smithi042 ceph-mon[32775]: pgmap v10242: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18281.543311 secs 2024-07-26T12:00:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18299.281417 secs 2024-07-26T12:00:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18281.543311 secs 2024-07-26T12:00:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18299.281417 secs 2024-07-26T12:00:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18281.543311 secs 2024-07-26T12:00:52.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18299.281417 secs 2024-07-26T12:00:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:53 smithi164 ceph-mon[36970]: pgmap v10243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:53 smithi163 ceph-mon[36873]: pgmap v10243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:53 smithi042 ceph-mon[32775]: pgmap v10243: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:55 smithi164 ceph-mon[36970]: pgmap v10244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:55.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:55 smithi163 ceph-mon[36873]: pgmap v10244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:55 smithi042 ceph-mon[32775]: pgmap v10244: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:00:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:57 smithi164 ceph-mon[36970]: pgmap v10245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18286.543442 secs 2024-07-26T12:00:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18304.281539 secs 2024-07-26T12:00:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:57 smithi163 ceph-mon[36873]: pgmap v10245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18286.543442 secs 2024-07-26T12:00:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18304.281539 secs 2024-07-26T12:00:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:57 smithi042 ceph-mon[32775]: pgmap v10245: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:00:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18286.543442 secs 2024-07-26T12:00:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18304.281539 secs 2024-07-26T12:00:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:00:59.022+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:00:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:00:59.022+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:00:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:00:59.022+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:00:59 smithi164 ceph-mon[36970]: pgmap v10246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:00:59 smithi163 ceph-mon[36873]: pgmap v10246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:00:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:00:59 smithi042 ceph-mon[32775]: pgmap v10246: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:01 smithi164 ceph-mon[36970]: pgmap v10247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:01 smithi163 ceph-mon[36873]: pgmap v10247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:01 smithi042 ceph-mon[32775]: pgmap v10247: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18291.543562 secs 2024-07-26T12:01:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18309.281689 secs 2024-07-26T12:01:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18291.543562 secs 2024-07-26T12:01:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18309.281689 secs 2024-07-26T12:01:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18291.543562 secs 2024-07-26T12:01:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18309.281689 secs 2024-07-26T12:01:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:03 smithi164 ceph-mon[36970]: pgmap v10248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:03 smithi163 ceph-mon[36873]: pgmap v10248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:03 smithi042 ceph-mon[32775]: pgmap v10248: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:05 smithi164 ceph-mon[36970]: pgmap v10249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:01:05.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:05 smithi163 ceph-mon[36873]: pgmap v10249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:01:05.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:05 smithi042 ceph-mon[32775]: pgmap v10249: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:01:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:07 smithi164 ceph-mon[36970]: pgmap v10250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18296.543648 secs 2024-07-26T12:01:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18314.281828 secs 2024-07-26T12:01:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:07 smithi163 ceph-mon[36873]: pgmap v10250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18296.543648 secs 2024-07-26T12:01:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18314.281828 secs 2024-07-26T12:01:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:07 smithi042 ceph-mon[32775]: pgmap v10250: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18296.543648 secs 2024-07-26T12:01:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18314.281828 secs 2024-07-26T12:01:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:09 smithi164 ceph-mon[36970]: pgmap v10251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:09 smithi163 ceph-mon[36873]: pgmap v10251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:09 smithi042 ceph-mon[32775]: pgmap v10251: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:11 smithi164 ceph-mon[36970]: pgmap v10252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:11 smithi163 ceph-mon[36873]: pgmap v10252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:11 smithi042 ceph-mon[32775]: pgmap v10252: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18301.543772 secs 2024-07-26T12:01:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18319.281959 secs 2024-07-26T12:01:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18301.543772 secs 2024-07-26T12:01:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18319.281959 secs 2024-07-26T12:01:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18301.543772 secs 2024-07-26T12:01:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18319.281959 secs 2024-07-26T12:01:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:13 smithi164 ceph-mon[36970]: pgmap v10253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:13.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:13 smithi163 ceph-mon[36873]: pgmap v10253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:13 smithi042 ceph-mon[32775]: pgmap v10253: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:14.024 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:01:14.023+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:01:14.025 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:01:14.023+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:01:14.025 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:01:14.023+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:15 smithi164 ceph-mon[36970]: pgmap v10254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:01:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:15 smithi163 ceph-mon[36873]: pgmap v10254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:01:15.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:15 smithi042 ceph-mon[32775]: pgmap v10254: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:01:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:01:15.753+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:01:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:01:15.753+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T12:01:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:01:15.753+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T12:01:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:01:15.753+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:01:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:16 smithi042 ceph-mon[32775]: pgmap v10255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:16 smithi164 ceph-mon[36970]: pgmap v10255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:16 smithi163 ceph-mon[36873]: pgmap v10255: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18306.543894 secs 2024-07-26T12:01:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18324.282049 secs 2024-07-26T12:01:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18306.543894 secs 2024-07-26T12:01:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18324.282049 secs 2024-07-26T12:01:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18306.543894 secs 2024-07-26T12:01:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18324.282049 secs 2024-07-26T12:01:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:18 smithi042 ceph-mon[32775]: pgmap v10256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:18 smithi164 ceph-mon[36970]: pgmap v10256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:18 smithi163 ceph-mon[36873]: pgmap v10256: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:20 smithi042 ceph-mon[32775]: pgmap v10257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:20 smithi164 ceph-mon[36970]: pgmap v10257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:20 smithi163 ceph-mon[36873]: pgmap v10257: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18311.544021 secs 2024-07-26T12:01:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: pgmap v10258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:01:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18329.282125 secs 2024-07-26T12:01:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T12:01:22.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T12:01:22.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T12:01:22.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T12:01:22.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:01:22.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T12:01:22.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:01:22.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:01:22.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: pgmap v10259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:01:22.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: pgmap v10260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:01:22.648 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:01:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18311.544021 secs 2024-07-26T12:01:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: pgmap v10258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:01:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18329.282125 secs 2024-07-26T12:01:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T12:01:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T12:01:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T12:01:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T12:01:22.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:01:22.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T12:01:22.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:01:22.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:01:22.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: pgmap v10259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:01:22.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: pgmap v10260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:01:22.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:01:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18311.544021 secs 2024-07-26T12:01:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: pgmap v10258: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:01:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18329.282125 secs 2024-07-26T12:01:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:22.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T12:01:22.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T12:01:22.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T12:01:22.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T12:01:22.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:01:22.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T12:01:22.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:01:22.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:01:22.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: pgmap v10259: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:01:22.760 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: pgmap v10260: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:01:22.760 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:01:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:24 smithi164 ceph-mon[36970]: pgmap v10261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:01:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:24 smithi163 ceph-mon[36873]: pgmap v10261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:01:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:24 smithi042 ceph-mon[32775]: pgmap v10261: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:01:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:26 smithi164 ceph-mon[36970]: pgmap v10262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:01:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18316.544147 secs 2024-07-26T12:01:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18334.282275 secs 2024-07-26T12:01:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:26 smithi163 ceph-mon[36873]: pgmap v10262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:01:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18316.544147 secs 2024-07-26T12:01:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18334.282275 secs 2024-07-26T12:01:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:26 smithi042 ceph-mon[32775]: pgmap v10262: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:01:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18316.544147 secs 2024-07-26T12:01:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18334.282275 secs 2024-07-26T12:01:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:01:29.027+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:01:29.028 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:01:29.027+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:01:29.029 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:01:29.027+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:28 smithi164 ceph-mon[36970]: pgmap v10263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:01:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:28 smithi163 ceph-mon[36873]: pgmap v10263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:01:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:28 smithi042 ceph-mon[32775]: pgmap v10263: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:01:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:30 smithi164 ceph-mon[36970]: pgmap v10264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:01:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:30 smithi163 ceph-mon[36873]: pgmap v10264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:01:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:30 smithi042 ceph-mon[32775]: pgmap v10264: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:01:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18321.544248 secs 2024-07-26T12:01:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18339.282391 secs 2024-07-26T12:01:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18321.544248 secs 2024-07-26T12:01:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18339.282391 secs 2024-07-26T12:01:32.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18321.544248 secs 2024-07-26T12:01:32.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18339.282391 secs 2024-07-26T12:01:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:32 smithi164 ceph-mon[36970]: pgmap v10265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:01:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:32 smithi163 ceph-mon[36873]: pgmap v10265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:01:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:32 smithi042 ceph-mon[32775]: pgmap v10265: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:01:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:34 smithi164 ceph-mon[36970]: pgmap v10266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:01:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:34 smithi163 ceph-mon[36873]: pgmap v10266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:01:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:34 smithi042 ceph-mon[32775]: pgmap v10266: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:01:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:36 smithi164 ceph-mon[36970]: pgmap v10267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18326.544370 secs 2024-07-26T12:01:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18344.282511 secs 2024-07-26T12:01:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:36 smithi163 ceph-mon[36873]: pgmap v10267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18326.544370 secs 2024-07-26T12:01:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18344.282511 secs 2024-07-26T12:01:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:36 smithi042 ceph-mon[32775]: pgmap v10267: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18326.544370 secs 2024-07-26T12:01:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18344.282511 secs 2024-07-26T12:01:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:38 smithi164 ceph-mon[36970]: pgmap v10268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:38 smithi163 ceph-mon[36873]: pgmap v10268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:38 smithi042 ceph-mon[32775]: pgmap v10268: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:40 smithi164 ceph-mon[36970]: pgmap v10269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:40 smithi163 ceph-mon[36873]: pgmap v10269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:40 smithi042 ceph-mon[32775]: pgmap v10269: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18331.544497 secs 2024-07-26T12:01:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18349.282630 secs 2024-07-26T12:01:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18331.544497 secs 2024-07-26T12:01:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18349.282630 secs 2024-07-26T12:01:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18331.544497 secs 2024-07-26T12:01:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18349.282630 secs 2024-07-26T12:01:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:42 smithi163 ceph-mon[36873]: pgmap v10270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:42 smithi042 ceph-mon[32775]: pgmap v10270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:42 smithi164 ceph-mon[36970]: pgmap v10270: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:01:44.029+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:01:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:01:44.029+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:01:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:01:44.029+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:44 smithi163 ceph-mon[36873]: pgmap v10271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:01:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:44 smithi042 ceph-mon[32775]: pgmap v10271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:01:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:44 smithi164 ceph-mon[36970]: pgmap v10271: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:01:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:46 smithi163 ceph-mon[36873]: pgmap v10272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18336.544622 secs 2024-07-26T12:01:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18354.282725 secs 2024-07-26T12:01:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:46 smithi042 ceph-mon[32775]: pgmap v10272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18336.544622 secs 2024-07-26T12:01:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18354.282725 secs 2024-07-26T12:01:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:46 smithi164 ceph-mon[36970]: pgmap v10272: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18336.544622 secs 2024-07-26T12:01:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18354.282725 secs 2024-07-26T12:01:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:48 smithi163 ceph-mon[36873]: pgmap v10273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:48 smithi042 ceph-mon[32775]: pgmap v10273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:48 smithi164 ceph-mon[36970]: pgmap v10273: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:50 smithi163 ceph-mon[36873]: pgmap v10274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:50 smithi042 ceph-mon[32775]: pgmap v10274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:50 smithi164 ceph-mon[36970]: pgmap v10274: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18341.544746 secs 2024-07-26T12:01:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18359.282800 secs 2024-07-26T12:01:52.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18341.544746 secs 2024-07-26T12:01:52.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18359.282800 secs 2024-07-26T12:01:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18341.544746 secs 2024-07-26T12:01:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18359.282800 secs 2024-07-26T12:01:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:52 smithi163 ceph-mon[36873]: pgmap v10275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:53.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:52 smithi042 ceph-mon[32775]: pgmap v10275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:52 smithi164 ceph-mon[36970]: pgmap v10275: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:55.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:54 smithi163 ceph-mon[36873]: pgmap v10276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:01:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:54 smithi042 ceph-mon[32775]: pgmap v10276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:01:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:54 smithi164 ceph-mon[36970]: pgmap v10276: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:01:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:56 smithi163 ceph-mon[36873]: pgmap v10277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18346.544854 secs 2024-07-26T12:01:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18364.282960 secs 2024-07-26T12:01:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:56 smithi042 ceph-mon[32775]: pgmap v10277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18346.544854 secs 2024-07-26T12:01:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18364.282960 secs 2024-07-26T12:01:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:56 smithi164 ceph-mon[36970]: pgmap v10277: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:01:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18346.544854 secs 2024-07-26T12:01:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18364.282960 secs 2024-07-26T12:01:59.033 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:01:59.031+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:01:59.033 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:01:59.031+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:01:59.033 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:01:59.031+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:01:59 smithi163 ceph-mon[36873]: pgmap v10278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:01:59 smithi042 ceph-mon[32775]: pgmap v10278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:01:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:01:59 smithi164 ceph-mon[36970]: pgmap v10278: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:00 smithi042 ceph-mon[32775]: pgmap v10279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:00 smithi164 ceph-mon[36970]: pgmap v10279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:00 smithi163 ceph-mon[36873]: pgmap v10279: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:02:00.760+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:02:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:02:00.760+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T12:02:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:02:00.760+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658003690 con 0x7ff680102000 2024-07-26T12:02:00.761 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:02:00.760+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:02:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18351.544976 secs 2024-07-26T12:02:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18351.544976 secs 2024-07-26T12:02:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18351.544976 secs 2024-07-26T12:02:02.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18369.283052 secs 2024-07-26T12:02:02.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:02 smithi042 ceph-mon[32775]: pgmap v10280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18369.283052 secs 2024-07-26T12:02:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:02 smithi164 ceph-mon[36970]: pgmap v10280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18369.283052 secs 2024-07-26T12:02:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:02 smithi163 ceph-mon[36873]: pgmap v10280: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:04 smithi164 ceph-mon[36970]: pgmap v10281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:02:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:04 smithi163 ceph-mon[36873]: pgmap v10281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:02:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:04 smithi042 ceph-mon[32775]: pgmap v10281: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:02:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:06 smithi164 ceph-mon[36970]: pgmap v10282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18356.545102 secs 2024-07-26T12:02:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18374.283173 secs 2024-07-26T12:02:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:06 smithi163 ceph-mon[36873]: pgmap v10282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18356.545102 secs 2024-07-26T12:02:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18374.283173 secs 2024-07-26T12:02:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:06 smithi042 ceph-mon[32775]: pgmap v10282: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18356.545102 secs 2024-07-26T12:02:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18374.283173 secs 2024-07-26T12:02:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:08 smithi164 ceph-mon[36970]: pgmap v10283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:08 smithi163 ceph-mon[36873]: pgmap v10283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:08 smithi042 ceph-mon[32775]: pgmap v10283: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:10 smithi164 ceph-mon[36970]: pgmap v10284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:10 smithi163 ceph-mon[36873]: pgmap v10284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:10 smithi042 ceph-mon[32775]: pgmap v10284: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18361.545252 secs 2024-07-26T12:02:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18379.283296 secs 2024-07-26T12:02:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18361.545252 secs 2024-07-26T12:02:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18379.283296 secs 2024-07-26T12:02:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18361.545252 secs 2024-07-26T12:02:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18379.283296 secs 2024-07-26T12:02:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:12 smithi164 ceph-mon[36970]: pgmap v10285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:12 smithi163 ceph-mon[36873]: pgmap v10285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:12 smithi042 ceph-mon[32775]: pgmap v10285: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:02:14.032+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:02:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:02:14.032+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:02:14.033 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:02:14.032+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:14 smithi164 ceph-mon[36970]: pgmap v10286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:02:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:14 smithi163 ceph-mon[36873]: pgmap v10286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:02:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:14 smithi042 ceph-mon[32775]: pgmap v10286: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:02:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:16 smithi164 ceph-mon[36970]: pgmap v10287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18366.545368 secs 2024-07-26T12:02:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18384.283415 secs 2024-07-26T12:02:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:16 smithi163 ceph-mon[36873]: pgmap v10287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18366.545368 secs 2024-07-26T12:02:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18384.283415 secs 2024-07-26T12:02:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:16 smithi042 ceph-mon[32775]: pgmap v10287: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18366.545368 secs 2024-07-26T12:02:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18384.283415 secs 2024-07-26T12:02:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:18 smithi164 ceph-mon[36970]: pgmap v10288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:18 smithi163 ceph-mon[36873]: pgmap v10288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:18 smithi042 ceph-mon[32775]: pgmap v10288: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:20 smithi163 ceph-mon[36873]: pgmap v10289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:20 smithi042 ceph-mon[32775]: pgmap v10289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:20 smithi164 ceph-mon[36970]: pgmap v10289: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:22.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18371.545470 secs 2024-07-26T12:02:22.259 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18389.283533 secs 2024-07-26T12:02:22.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18371.545470 secs 2024-07-26T12:02:22.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18389.283533 secs 2024-07-26T12:02:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18371.545470 secs 2024-07-26T12:02:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18389.283533 secs 2024-07-26T12:02:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:22 smithi163 ceph-mon[36873]: pgmap v10290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:02:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:02:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:02:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:22 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:02:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:22 smithi042 ceph-mon[32775]: pgmap v10290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:02:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:02:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:02:23.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:22 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:02:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:22 smithi164 ceph-mon[36970]: pgmap v10290: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:02:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:02:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:02:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:22 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:02:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:23 smithi163 ceph-mon[36873]: pgmap v10291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T12:02:24.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:23 smithi163 ceph-mon[36873]: pgmap v10292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:02:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:23 smithi042 ceph-mon[32775]: pgmap v10291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T12:02:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:23 smithi042 ceph-mon[32775]: pgmap v10292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:02:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:23 smithi164 ceph-mon[36970]: pgmap v10291: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T12:02:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:23 smithi164 ceph-mon[36970]: pgmap v10292: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:02:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:25 smithi163 ceph-mon[36873]: pgmap v10293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:02:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:25 smithi042 ceph-mon[32775]: pgmap v10293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:02:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:25 smithi164 ceph-mon[36970]: pgmap v10293: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:02:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18376.545599 secs 2024-07-26T12:02:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18394.283679 secs 2024-07-26T12:02:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18376.545599 secs 2024-07-26T12:02:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18394.283679 secs 2024-07-26T12:02:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18376.545599 secs 2024-07-26T12:02:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18394.283679 secs 2024-07-26T12:02:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:27 smithi163 ceph-mon[36873]: pgmap v10294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:02:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:27 smithi042 ceph-mon[32775]: pgmap v10294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:02:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:27 smithi164 ceph-mon[36970]: pgmap v10294: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:02:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:02:29.038+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:02:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:02:29.038+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:02:29.039 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:02:29.038+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:30 smithi163 ceph-mon[36873]: pgmap v10295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:02:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:30 smithi042 ceph-mon[32775]: pgmap v10295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:02:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:30 smithi164 ceph-mon[36970]: pgmap v10295: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:02:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:32 smithi163 ceph-mon[36873]: pgmap v10296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:02:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18381.545723 secs 2024-07-26T12:02:32.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18399.283800 secs 2024-07-26T12:02:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:32 smithi042 ceph-mon[32775]: pgmap v10296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:02:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18381.545723 secs 2024-07-26T12:02:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18399.283800 secs 2024-07-26T12:02:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:32 smithi164 ceph-mon[36970]: pgmap v10296: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:02:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18381.545723 secs 2024-07-26T12:02:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18399.283800 secs 2024-07-26T12:02:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:34 smithi042 ceph-mon[32775]: pgmap v10297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:02:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:34 smithi164 ceph-mon[36970]: pgmap v10297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:02:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:34 smithi163 ceph-mon[36873]: pgmap v10297: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:02:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:36 smithi042 ceph-mon[32775]: pgmap v10298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:36 smithi164 ceph-mon[36970]: pgmap v10298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:36 smithi163 ceph-mon[36873]: pgmap v10298: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18386.545851 secs 2024-07-26T12:02:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18404.283904 secs 2024-07-26T12:02:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18386.545851 secs 2024-07-26T12:02:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18404.283904 secs 2024-07-26T12:02:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18386.545851 secs 2024-07-26T12:02:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18404.283904 secs 2024-07-26T12:02:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:38 smithi042 ceph-mon[32775]: pgmap v10299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:38 smithi164 ceph-mon[36970]: pgmap v10299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:38 smithi163 ceph-mon[36873]: pgmap v10299: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:40 smithi042 ceph-mon[32775]: pgmap v10300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:02:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:40 smithi164 ceph-mon[36970]: pgmap v10300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:02:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:40 smithi163 ceph-mon[36873]: pgmap v10300: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:02:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:42 smithi042 ceph-mon[32775]: pgmap v10301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18391.545930 secs 2024-07-26T12:02:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18409.284025 secs 2024-07-26T12:02:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:42 smithi164 ceph-mon[36970]: pgmap v10301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18391.545930 secs 2024-07-26T12:02:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18409.284025 secs 2024-07-26T12:02:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:42 smithi163 ceph-mon[36873]: pgmap v10301: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18391.545930 secs 2024-07-26T12:02:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18409.284025 secs 2024-07-26T12:02:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:02:44.040+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:02:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:02:44.040+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:02:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:02:44.040+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:44 smithi042 ceph-mon[32775]: pgmap v10302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:44 smithi164 ceph-mon[36970]: pgmap v10302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:44 smithi163 ceph-mon[36873]: pgmap v10302: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:46 smithi042 ceph-mon[32775]: pgmap v10303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:46 smithi164 ceph-mon[36970]: pgmap v10303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:46 smithi163 ceph-mon[36873]: pgmap v10303: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18396.546051 secs 2024-07-26T12:02:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18414.284143 secs 2024-07-26T12:02:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18396.546051 secs 2024-07-26T12:02:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18414.284143 secs 2024-07-26T12:02:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18396.546051 secs 2024-07-26T12:02:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18414.284143 secs 2024-07-26T12:02:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:48 smithi042 ceph-mon[32775]: pgmap v10304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:48 smithi164 ceph-mon[36970]: pgmap v10304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:48 smithi163 ceph-mon[36873]: pgmap v10304: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:50 smithi042 ceph-mon[32775]: pgmap v10305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:02:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:50 smithi164 ceph-mon[36970]: pgmap v10305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:02:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:50 smithi163 ceph-mon[36873]: pgmap v10305: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:02:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:52 smithi042 ceph-mon[32775]: pgmap v10306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18401.546179 secs 2024-07-26T12:02:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18419.284226 secs 2024-07-26T12:02:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:52 smithi164 ceph-mon[36970]: pgmap v10306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18401.546179 secs 2024-07-26T12:02:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18419.284226 secs 2024-07-26T12:02:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:52 smithi163 ceph-mon[36873]: pgmap v10306: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18401.546179 secs 2024-07-26T12:02:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18419.284226 secs 2024-07-26T12:02:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:54 smithi042 ceph-mon[32775]: pgmap v10307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:54 smithi164 ceph-mon[36970]: pgmap v10307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:54 smithi163 ceph-mon[36873]: pgmap v10307: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:56 smithi042 ceph-mon[32775]: pgmap v10308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:56 smithi164 ceph-mon[36970]: pgmap v10308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:56 smithi163 ceph-mon[36873]: pgmap v10308: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:02:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18406.546279 secs 2024-07-26T12:02:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18424.284346 secs 2024-07-26T12:02:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18406.546279 secs 2024-07-26T12:02:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18424.284346 secs 2024-07-26T12:02:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18406.546279 secs 2024-07-26T12:02:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18424.284346 secs 2024-07-26T12:02:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:02:58 smithi042 ceph-mon[32775]: pgmap v10309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:02:58 smithi164 ceph-mon[36970]: pgmap v10309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:02:58 smithi163 ceph-mon[36873]: pgmap v10309: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:02:59.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:02:59.045+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:02:59.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:02:59.045+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:02:59.047 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:02:59.045+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:00 smithi042 ceph-mon[32775]: pgmap v10310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:03:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:00 smithi164 ceph-mon[36970]: pgmap v10310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:03:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:00 smithi163 ceph-mon[36873]: pgmap v10310: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:03:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:02 smithi042 ceph-mon[32775]: pgmap v10311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18411.546440 secs 2024-07-26T12:03:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18429.284466 secs 2024-07-26T12:03:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:02 smithi164 ceph-mon[36970]: pgmap v10311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18411.546440 secs 2024-07-26T12:03:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18429.284466 secs 2024-07-26T12:03:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:02 smithi163 ceph-mon[36873]: pgmap v10311: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18411.546440 secs 2024-07-26T12:03:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18429.284466 secs 2024-07-26T12:03:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:04 smithi042 ceph-mon[32775]: pgmap v10312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:04 smithi164 ceph-mon[36970]: pgmap v10312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:04.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:04 smithi163 ceph-mon[36873]: pgmap v10312: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:06.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:06 smithi042 ceph-mon[32775]: pgmap v10313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:06 smithi164 ceph-mon[36970]: pgmap v10313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:06 smithi163 ceph-mon[36873]: pgmap v10313: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18416.546590 secs 2024-07-26T12:03:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18434.284566 secs 2024-07-26T12:03:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18416.546590 secs 2024-07-26T12:03:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18434.284566 secs 2024-07-26T12:03:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18416.546590 secs 2024-07-26T12:03:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18434.284566 secs 2024-07-26T12:03:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:08 smithi042 ceph-mon[32775]: pgmap v10314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:08 smithi164 ceph-mon[36970]: pgmap v10314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:08 smithi163 ceph-mon[36873]: pgmap v10314: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:10 smithi042 ceph-mon[32775]: pgmap v10315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:03:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:10 smithi164 ceph-mon[36970]: pgmap v10315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:03:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:10 smithi163 ceph-mon[36873]: pgmap v10315: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:03:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:12 smithi042 ceph-mon[32775]: pgmap v10316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18421.546654 secs 2024-07-26T12:03:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18439.284690 secs 2024-07-26T12:03:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:12 smithi164 ceph-mon[36970]: pgmap v10316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18421.546654 secs 2024-07-26T12:03:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18439.284690 secs 2024-07-26T12:03:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:12 smithi163 ceph-mon[36873]: pgmap v10316: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18421.546654 secs 2024-07-26T12:03:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18439.284690 secs 2024-07-26T12:03:14.048 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:03:14.047+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:03:14.048 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:03:14.047+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:03:14.049 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:03:14.047+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:14 smithi042 ceph-mon[32775]: pgmap v10317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:14 smithi164 ceph-mon[36970]: pgmap v10317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:14 smithi163 ceph-mon[36873]: pgmap v10317: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:16 smithi042 ceph-mon[32775]: pgmap v10318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:16 smithi164 ceph-mon[36970]: pgmap v10318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:16 smithi163 ceph-mon[36873]: pgmap v10318: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18426.546754 secs 2024-07-26T12:03:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18444.284821 secs 2024-07-26T12:03:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18426.546754 secs 2024-07-26T12:03:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18444.284821 secs 2024-07-26T12:03:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18426.546754 secs 2024-07-26T12:03:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18444.284821 secs 2024-07-26T12:03:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:18 smithi042 ceph-mon[32775]: pgmap v10319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:18 smithi164 ceph-mon[36970]: pgmap v10319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:18 smithi163 ceph-mon[36873]: pgmap v10319: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:20 smithi042 ceph-mon[32775]: pgmap v10320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:03:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:20 smithi164 ceph-mon[36970]: pgmap v10320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:03:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:20 smithi163 ceph-mon[36873]: pgmap v10320: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:03:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:22 smithi042 ceph-mon[32775]: pgmap v10321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18431.546874 secs 2024-07-26T12:03:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18449.284967 secs 2024-07-26T12:03:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:22 smithi164 ceph-mon[36970]: pgmap v10321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18431.546874 secs 2024-07-26T12:03:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18449.284967 secs 2024-07-26T12:03:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:22 smithi163 ceph-mon[36873]: pgmap v10321: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18431.546874 secs 2024-07-26T12:03:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18449.284967 secs 2024-07-26T12:03:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:03:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:03:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:03:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:03:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:03:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:03:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:03:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:03:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:03:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:03:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:03:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:03:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:24 smithi042 ceph-mon[32775]: pgmap v10322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:24 smithi042 ceph-mon[32775]: pgmap v10323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:03:24.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:24 smithi042 ceph-mon[32775]: pgmap v10324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:03:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:24 smithi164 ceph-mon[36970]: pgmap v10322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:24 smithi164 ceph-mon[36970]: pgmap v10323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:03:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:24 smithi164 ceph-mon[36970]: pgmap v10324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:03:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:24 smithi163 ceph-mon[36873]: pgmap v10322: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:24 smithi163 ceph-mon[36873]: pgmap v10323: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:03:24.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:24 smithi163 ceph-mon[36873]: pgmap v10324: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:03:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:26 smithi042 ceph-mon[32775]: pgmap v10325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:03:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:26 smithi164 ceph-mon[36970]: pgmap v10325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:03:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:26 smithi163 ceph-mon[36873]: pgmap v10325: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:03:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18436.547001 secs 2024-07-26T12:03:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18454.285101 secs 2024-07-26T12:03:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18436.547001 secs 2024-07-26T12:03:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18454.285101 secs 2024-07-26T12:03:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18436.547001 secs 2024-07-26T12:03:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18454.285101 secs 2024-07-26T12:03:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:28 smithi042 ceph-mon[32775]: pgmap v10326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:03:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:28 smithi164 ceph-mon[36970]: pgmap v10326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:03:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:28 smithi163 ceph-mon[36873]: pgmap v10326: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:03:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:03:29.053+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:03:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:03:29.053+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:03:29.054 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:03:29.053+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:30 smithi042 ceph-mon[32775]: pgmap v10327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:03:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:30 smithi164 ceph-mon[36970]: pgmap v10327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:03:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:30 smithi163 ceph-mon[36873]: pgmap v10327: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:03:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:32 smithi042 ceph-mon[32775]: pgmap v10328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:03:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18441.547061 secs 2024-07-26T12:03:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18459.285211 secs 2024-07-26T12:03:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:32 smithi164 ceph-mon[36970]: pgmap v10328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:03:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18441.547061 secs 2024-07-26T12:03:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18459.285211 secs 2024-07-26T12:03:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:32 smithi163 ceph-mon[36873]: pgmap v10328: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:03:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18441.547061 secs 2024-07-26T12:03:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18459.285211 secs 2024-07-26T12:03:34.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:34 smithi042 ceph-mon[32775]: pgmap v10329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:34 smithi164 ceph-mon[36970]: pgmap v10329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:34 smithi163 ceph-mon[36873]: pgmap v10329: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:03:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:36 smithi042 ceph-mon[32775]: pgmap v10330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:36 smithi164 ceph-mon[36970]: pgmap v10330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:36 smithi163 ceph-mon[36873]: pgmap v10330: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18446.547203 secs 2024-07-26T12:03:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18464.285329 secs 2024-07-26T12:03:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18446.547203 secs 2024-07-26T12:03:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18464.285329 secs 2024-07-26T12:03:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18446.547203 secs 2024-07-26T12:03:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18464.285329 secs 2024-07-26T12:03:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:38 smithi164 ceph-mon[36970]: pgmap v10331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:38 smithi163 ceph-mon[36873]: pgmap v10331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:38 smithi042 ceph-mon[32775]: pgmap v10331: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:40 smithi164 ceph-mon[36970]: pgmap v10332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:03:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:40 smithi163 ceph-mon[36873]: pgmap v10332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:03:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:40 smithi042 ceph-mon[32775]: pgmap v10332: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:03:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:42 smithi164 ceph-mon[36970]: pgmap v10333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18451.547357 secs 2024-07-26T12:03:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18469.285455 secs 2024-07-26T12:03:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:42 smithi163 ceph-mon[36873]: pgmap v10333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18451.547357 secs 2024-07-26T12:03:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18469.285455 secs 2024-07-26T12:03:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:42 smithi042 ceph-mon[32775]: pgmap v10333: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18451.547357 secs 2024-07-26T12:03:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18469.285455 secs 2024-07-26T12:03:44.055 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:03:44.054+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:03:44.055 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:03:44.054+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:03:44.055 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:03:44.054+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:44 smithi164 ceph-mon[36970]: pgmap v10334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:44 smithi163 ceph-mon[36873]: pgmap v10334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:44 smithi042 ceph-mon[32775]: pgmap v10334: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:46 smithi164 ceph-mon[36970]: pgmap v10335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:46 smithi163 ceph-mon[36873]: pgmap v10335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:46 smithi042 ceph-mon[32775]: pgmap v10335: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18456.547480 secs 2024-07-26T12:03:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18474.285575 secs 2024-07-26T12:03:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18456.547480 secs 2024-07-26T12:03:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18474.285575 secs 2024-07-26T12:03:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18456.547480 secs 2024-07-26T12:03:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18474.285575 secs 2024-07-26T12:03:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:48 smithi164 ceph-mon[36970]: pgmap v10336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:48 smithi163 ceph-mon[36873]: pgmap v10336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:48.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:48 smithi042 ceph-mon[32775]: pgmap v10336: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:50 smithi164 ceph-mon[36970]: pgmap v10337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:03:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:50 smithi163 ceph-mon[36873]: pgmap v10337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:03:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:50 smithi042 ceph-mon[32775]: pgmap v10337: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:03:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:52 smithi164 ceph-mon[36970]: pgmap v10338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18461.547601 secs 2024-07-26T12:03:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18479.285698 secs 2024-07-26T12:03:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:52 smithi163 ceph-mon[36873]: pgmap v10338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18461.547601 secs 2024-07-26T12:03:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18479.285698 secs 2024-07-26T12:03:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:52 smithi042 ceph-mon[32775]: pgmap v10338: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18461.547601 secs 2024-07-26T12:03:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18479.285698 secs 2024-07-26T12:03:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:54 smithi164 ceph-mon[36970]: pgmap v10339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:54 smithi163 ceph-mon[36873]: pgmap v10339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:54 smithi042 ceph-mon[32775]: pgmap v10339: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:56 smithi164 ceph-mon[36970]: pgmap v10340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:56.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:56 smithi163 ceph-mon[36873]: pgmap v10340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:56 smithi042 ceph-mon[32775]: pgmap v10340: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:03:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18466.547723 secs 2024-07-26T12:03:57.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18484.285819 secs 2024-07-26T12:03:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18466.547723 secs 2024-07-26T12:03:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18484.285819 secs 2024-07-26T12:03:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18466.547723 secs 2024-07-26T12:03:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18484.285819 secs 2024-07-26T12:03:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:03:58 smithi164 ceph-mon[36970]: pgmap v10341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:03:58 smithi163 ceph-mon[36873]: pgmap v10341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:03:58 smithi042 ceph-mon[32775]: pgmap v10341: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:03:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:03:59.060+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:03:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:03:59.060+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:03:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:03:59.060+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:00 smithi164 ceph-mon[36970]: pgmap v10342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:04:00.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:00 smithi163 ceph-mon[36873]: pgmap v10342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:04:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:00 smithi042 ceph-mon[32775]: pgmap v10342: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:04:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:02 smithi164 ceph-mon[36970]: pgmap v10343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18471.547861 secs 2024-07-26T12:04:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18489.285959 secs 2024-07-26T12:04:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:02 smithi163 ceph-mon[36873]: pgmap v10343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18471.547861 secs 2024-07-26T12:04:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18489.285959 secs 2024-07-26T12:04:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:02 smithi042 ceph-mon[32775]: pgmap v10343: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18471.547861 secs 2024-07-26T12:04:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18489.285959 secs 2024-07-26T12:04:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:04 smithi164 ceph-mon[36970]: pgmap v10344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:04.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:04 smithi163 ceph-mon[36873]: pgmap v10344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:04.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:04 smithi042 ceph-mon[32775]: pgmap v10344: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:06 smithi164 ceph-mon[36970]: pgmap v10345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:06 smithi163 ceph-mon[36873]: pgmap v10345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:06 smithi042 ceph-mon[32775]: pgmap v10345: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18476.547983 secs 2024-07-26T12:04:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18494.286015 secs 2024-07-26T12:04:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18476.547983 secs 2024-07-26T12:04:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18494.286015 secs 2024-07-26T12:04:07.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18476.547983 secs 2024-07-26T12:04:07.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18494.286015 secs 2024-07-26T12:04:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:08 smithi164 ceph-mon[36970]: pgmap v10346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:08 smithi163 ceph-mon[36873]: pgmap v10346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:08.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:08 smithi042 ceph-mon[32775]: pgmap v10346: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:10 smithi164 ceph-mon[36970]: pgmap v10347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:04:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:10 smithi163 ceph-mon[36873]: pgmap v10347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:04:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:10 smithi042 ceph-mon[32775]: pgmap v10347: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:04:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:12 smithi164 ceph-mon[36970]: pgmap v10348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18481.548102 secs 2024-07-26T12:04:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18499.286140 secs 2024-07-26T12:04:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:12 smithi163 ceph-mon[36873]: pgmap v10348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18481.548102 secs 2024-07-26T12:04:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18499.286140 secs 2024-07-26T12:04:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:12 smithi042 ceph-mon[32775]: pgmap v10348: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18481.548102 secs 2024-07-26T12:04:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18499.286140 secs 2024-07-26T12:04:14.063 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:04:14.061+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:04:14.063 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:04:14.061+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:14.063 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:04:14.061+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:04:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:14 smithi163 ceph-mon[36873]: pgmap v10349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:14 smithi042 ceph-mon[32775]: pgmap v10349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:14 smithi164 ceph-mon[36970]: pgmap v10349: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:16 smithi163 ceph-mon[36873]: pgmap v10350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:16 smithi042 ceph-mon[32775]: pgmap v10350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:16 smithi164 ceph-mon[36970]: pgmap v10350: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18486.548243 secs 2024-07-26T12:04:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18504.286282 secs 2024-07-26T12:04:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18486.548243 secs 2024-07-26T12:04:17.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18504.286282 secs 2024-07-26T12:04:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18486.548243 secs 2024-07-26T12:04:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18504.286282 secs 2024-07-26T12:04:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:18 smithi163 ceph-mon[36873]: pgmap v10351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:18 smithi042 ceph-mon[32775]: pgmap v10351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:18 smithi164 ceph-mon[36970]: pgmap v10351: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:20 smithi163 ceph-mon[36873]: pgmap v10352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:04:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:20 smithi042 ceph-mon[32775]: pgmap v10352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:04:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:20 smithi164 ceph-mon[36970]: pgmap v10352: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:04:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:22 smithi163 ceph-mon[36873]: pgmap v10353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18491.548368 secs 2024-07-26T12:04:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18509.286405 secs 2024-07-26T12:04:22.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:22 smithi164 ceph-mon[36970]: pgmap v10353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:22.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18491.548368 secs 2024-07-26T12:04:22.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18509.286405 secs 2024-07-26T12:04:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:22 smithi042 ceph-mon[32775]: pgmap v10353: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18491.548368 secs 2024-07-26T12:04:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18509.286405 secs 2024-07-26T12:04:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:23 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:04:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:23 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:04:23.714 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:23 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:04:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:24 smithi163 ceph-mon[36873]: pgmap v10354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:24 smithi042 ceph-mon[32775]: pgmap v10354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:24 smithi164 ceph-mon[36970]: pgmap v10354: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:25 smithi163 ceph-mon[36873]: pgmap v10355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:04:26.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:04:26.009 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:25 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:25 smithi042 ceph-mon[32775]: pgmap v10355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:04:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:04:26.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:25 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:25 smithi164 ceph-mon[36970]: pgmap v10355: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:26.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:04:26.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:04:26.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:25 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:04:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:27 smithi163 ceph-mon[36873]: pgmap v10356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:04:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:27 smithi163 ceph-mon[36873]: pgmap v10357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:04:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18496.548527 secs 2024-07-26T12:04:27.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18514.286530 secs 2024-07-26T12:04:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:27 smithi042 ceph-mon[32775]: pgmap v10356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:04:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:27 smithi042 ceph-mon[32775]: pgmap v10357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:04:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18496.548527 secs 2024-07-26T12:04:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18514.286530 secs 2024-07-26T12:04:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:27 smithi164 ceph-mon[36970]: pgmap v10356: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:04:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:27 smithi164 ceph-mon[36970]: pgmap v10357: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:04:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18496.548527 secs 2024-07-26T12:04:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18514.286530 secs 2024-07-26T12:04:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:04:29.067+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:04:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:04:29.067+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:29.069 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:04:29.067+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:04:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:29 smithi163 ceph-mon[36873]: pgmap v10358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:04:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:29 smithi042 ceph-mon[32775]: pgmap v10358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:04:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:29 smithi164 ceph-mon[36970]: pgmap v10358: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:04:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:31 smithi163 ceph-mon[36873]: pgmap v10359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:04:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:31 smithi042 ceph-mon[32775]: pgmap v10359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:04:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:31 smithi164 ceph-mon[36970]: pgmap v10359: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:04:32.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18501.548656 secs 2024-07-26T12:04:32.297 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18519.286678 secs 2024-07-26T12:04:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18501.548656 secs 2024-07-26T12:04:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18519.286678 secs 2024-07-26T12:04:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18501.548656 secs 2024-07-26T12:04:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18519.286678 secs 2024-07-26T12:04:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:33 smithi042 ceph-mon[32775]: pgmap v10360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:04:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:33 smithi164 ceph-mon[36970]: pgmap v10360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:04:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:33 smithi163 ceph-mon[36873]: pgmap v10360: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:04:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:35 smithi042 ceph-mon[32775]: pgmap v10361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:04:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:35 smithi164 ceph-mon[36970]: pgmap v10361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:04:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:35 smithi163 ceph-mon[36873]: pgmap v10361: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:04:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:37 smithi042 ceph-mon[32775]: pgmap v10362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:04:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18506.548782 secs 2024-07-26T12:04:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18524.286819 secs 2024-07-26T12:04:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:37 smithi164 ceph-mon[36970]: pgmap v10362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:04:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18506.548782 secs 2024-07-26T12:04:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18524.286819 secs 2024-07-26T12:04:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:37 smithi163 ceph-mon[36873]: pgmap v10362: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:04:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18506.548782 secs 2024-07-26T12:04:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18524.286819 secs 2024-07-26T12:04:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:39 smithi042 ceph-mon[32775]: pgmap v10363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:39 smithi164 ceph-mon[36970]: pgmap v10363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:39 smithi163 ceph-mon[36873]: pgmap v10363: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:41 smithi042 ceph-mon[32775]: pgmap v10364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:41 smithi164 ceph-mon[36970]: pgmap v10364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:41 smithi163 ceph-mon[36873]: pgmap v10364: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18511.548906 secs 2024-07-26T12:04:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18529.286966 secs 2024-07-26T12:04:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18511.548906 secs 2024-07-26T12:04:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18529.286966 secs 2024-07-26T12:04:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18511.548906 secs 2024-07-26T12:04:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18529.286966 secs 2024-07-26T12:04:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:43 smithi042 ceph-mon[32775]: pgmap v10365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:43 smithi164 ceph-mon[36970]: pgmap v10365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:43 smithi163 ceph-mon[36873]: pgmap v10365: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:44.070 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:04:44.068+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:04:44.070 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:04:44.068+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:44.071 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:04:44.068+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:04:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:45 smithi042 ceph-mon[32775]: pgmap v10366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:04:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:45 smithi164 ceph-mon[36970]: pgmap v10366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:04:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:45 smithi163 ceph-mon[36873]: pgmap v10366: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:04:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:47 smithi042 ceph-mon[32775]: pgmap v10367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18516.549029 secs 2024-07-26T12:04:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18534.287011 secs 2024-07-26T12:04:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:47 smithi164 ceph-mon[36970]: pgmap v10367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18516.549029 secs 2024-07-26T12:04:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18534.287011 secs 2024-07-26T12:04:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:47 smithi163 ceph-mon[36873]: pgmap v10367: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18516.549029 secs 2024-07-26T12:04:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18534.287011 secs 2024-07-26T12:04:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:49 smithi042 ceph-mon[32775]: pgmap v10368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:49 smithi164 ceph-mon[36970]: pgmap v10368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:49 smithi163 ceph-mon[36873]: pgmap v10368: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:51 smithi042 ceph-mon[32775]: pgmap v10369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:51 smithi164 ceph-mon[36970]: pgmap v10369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:51 smithi163 ceph-mon[36873]: pgmap v10369: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:52.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18521.549116 secs 2024-07-26T12:04:52.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18539.287113 secs 2024-07-26T12:04:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18521.549116 secs 2024-07-26T12:04:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18539.287113 secs 2024-07-26T12:04:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18521.549116 secs 2024-07-26T12:04:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18539.287113 secs 2024-07-26T12:04:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:53 smithi042 ceph-mon[32775]: pgmap v10370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:53 smithi164 ceph-mon[36970]: pgmap v10370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:53 smithi163 ceph-mon[36873]: pgmap v10370: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:55 smithi042 ceph-mon[32775]: pgmap v10371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:04:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:55 smithi164 ceph-mon[36970]: pgmap v10371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:04:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:55 smithi163 ceph-mon[36873]: pgmap v10371: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:04:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:57 smithi042 ceph-mon[32775]: pgmap v10372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18526.549254 secs 2024-07-26T12:04:57.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18544.287198 secs 2024-07-26T12:04:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:57 smithi164 ceph-mon[36970]: pgmap v10372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18526.549254 secs 2024-07-26T12:04:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18544.287198 secs 2024-07-26T12:04:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:57 smithi163 ceph-mon[36873]: pgmap v10372: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:04:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18526.549254 secs 2024-07-26T12:04:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18544.287198 secs 2024-07-26T12:04:59.074 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:04:59.073+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:04:59.075 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:04:59.073+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:59.075 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:04:59.073+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:04:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:04:59 smithi042 ceph-mon[32775]: pgmap v10373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:04:59 smithi164 ceph-mon[36970]: pgmap v10373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:04:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:04:59 smithi163 ceph-mon[36873]: pgmap v10373: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:00.790 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:05:00.789+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:05:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:05:00.789+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T12:05:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:05:00.789+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6700e9050 con 0x7ff680102000 2024-07-26T12:05:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:05:00.789+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:05:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:01 smithi042 ceph-mon[32775]: pgmap v10374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:01 smithi164 ceph-mon[36970]: pgmap v10374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:01.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:01 smithi163 ceph-mon[36873]: pgmap v10374: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18531.549386 secs 2024-07-26T12:05:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18549.287319 secs 2024-07-26T12:05:02.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18531.549386 secs 2024-07-26T12:05:02.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18549.287319 secs 2024-07-26T12:05:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18531.549386 secs 2024-07-26T12:05:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18549.287319 secs 2024-07-26T12:05:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:03 smithi042 ceph-mon[32775]: pgmap v10375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:03 smithi164 ceph-mon[36970]: pgmap v10375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:03 smithi163 ceph-mon[36873]: pgmap v10375: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:05 smithi042 ceph-mon[32775]: pgmap v10376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:05:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:05 smithi164 ceph-mon[36970]: pgmap v10376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:05:05.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:05 smithi163 ceph-mon[36873]: pgmap v10376: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:05:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:07 smithi042 ceph-mon[32775]: pgmap v10377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18536.549523 secs 2024-07-26T12:05:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18554.287400 secs 2024-07-26T12:05:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:07 smithi164 ceph-mon[36970]: pgmap v10377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18536.549523 secs 2024-07-26T12:05:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18554.287400 secs 2024-07-26T12:05:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:07 smithi163 ceph-mon[36873]: pgmap v10377: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18536.549523 secs 2024-07-26T12:05:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18554.287400 secs 2024-07-26T12:05:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:09 smithi042 ceph-mon[32775]: pgmap v10378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:09 smithi164 ceph-mon[36970]: pgmap v10378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:09 smithi163 ceph-mon[36873]: pgmap v10378: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:11 smithi042 ceph-mon[32775]: pgmap v10379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:11 smithi164 ceph-mon[36970]: pgmap v10379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:11 smithi163 ceph-mon[36873]: pgmap v10379: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18541.549596 secs 2024-07-26T12:05:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18559.287527 secs 2024-07-26T12:05:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18541.549596 secs 2024-07-26T12:05:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18559.287527 secs 2024-07-26T12:05:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18541.549596 secs 2024-07-26T12:05:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18559.287527 secs 2024-07-26T12:05:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:13 smithi042 ceph-mon[32775]: pgmap v10380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:13 smithi164 ceph-mon[36970]: pgmap v10380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:13 smithi163 ceph-mon[36873]: pgmap v10380: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:14.076 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:05:14.075+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:05:14.076 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:05:14.075+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:14.076 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:05:14.075+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:05:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:15 smithi042 ceph-mon[32775]: pgmap v10381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:05:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:15 smithi164 ceph-mon[36970]: pgmap v10381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:05:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:15 smithi163 ceph-mon[36873]: pgmap v10381: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:05:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:17 smithi042 ceph-mon[32775]: pgmap v10382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18546.549737 secs 2024-07-26T12:05:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18564.287627 secs 2024-07-26T12:05:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:17 smithi164 ceph-mon[36970]: pgmap v10382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18546.549737 secs 2024-07-26T12:05:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18564.287627 secs 2024-07-26T12:05:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:17 smithi163 ceph-mon[36873]: pgmap v10382: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18546.549737 secs 2024-07-26T12:05:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18564.287627 secs 2024-07-26T12:05:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:19 smithi042 ceph-mon[32775]: pgmap v10383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:19 smithi164 ceph-mon[36970]: pgmap v10383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:19 smithi163 ceph-mon[36873]: pgmap v10383: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:21 smithi042 ceph-mon[32775]: pgmap v10384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:21 smithi164 ceph-mon[36970]: pgmap v10384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:21 smithi163 ceph-mon[36873]: pgmap v10384: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:22.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18551.549856 secs 2024-07-26T12:05:22.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18569.287769 secs 2024-07-26T12:05:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18551.549856 secs 2024-07-26T12:05:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18569.287769 secs 2024-07-26T12:05:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18551.549856 secs 2024-07-26T12:05:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18569.287769 secs 2024-07-26T12:05:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:23 smithi042 ceph-mon[32775]: pgmap v10385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:23 smithi164 ceph-mon[36970]: pgmap v10385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:23 smithi163 ceph-mon[36873]: pgmap v10385: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:25 smithi042 ceph-mon[32775]: pgmap v10386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:05:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:25 smithi164 ceph-mon[36970]: pgmap v10386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:05:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:25 smithi163 ceph-mon[36873]: pgmap v10386: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:05:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:05:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:26.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:26 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:05:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:26.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:26 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:05:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:26.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:26 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:27 smithi042 ceph-mon[32775]: pgmap v10387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18556.549931 secs 2024-07-26T12:05:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:05:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:05:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18574.287913 secs 2024-07-26T12:05:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:27 smithi164 ceph-mon[36970]: pgmap v10387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18556.549931 secs 2024-07-26T12:05:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:05:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:05:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18574.287913 secs 2024-07-26T12:05:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:27 smithi163 ceph-mon[36873]: pgmap v10387: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18556.549931 secs 2024-07-26T12:05:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:05:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:05:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:05:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18574.287913 secs 2024-07-26T12:05:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:28 smithi042 ceph-mon[32775]: pgmap v10388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:05:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:28 smithi042 ceph-mon[32775]: pgmap v10389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:05:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:28 smithi164 ceph-mon[36970]: pgmap v10388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:05:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:28 smithi164 ceph-mon[36970]: pgmap v10389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:05:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:28 smithi163 ceph-mon[36873]: pgmap v10388: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-07-26T12:05:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:28 smithi163 ceph-mon[36873]: pgmap v10389: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:05:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:05:29.081+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:05:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:05:29.081+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:05:29.081+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:05:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:30 smithi042 ceph-mon[32775]: pgmap v10390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T12:05:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:30 smithi164 ceph-mon[36970]: pgmap v10390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T12:05:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:30 smithi163 ceph-mon[36873]: pgmap v10390: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T12:05:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:32 smithi042 ceph-mon[32775]: pgmap v10391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:05:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18561.550062 secs 2024-07-26T12:05:32.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18579.288005 secs 2024-07-26T12:05:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:32 smithi164 ceph-mon[36970]: pgmap v10391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:05:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18561.550062 secs 2024-07-26T12:05:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18579.288005 secs 2024-07-26T12:05:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:32 smithi163 ceph-mon[36873]: pgmap v10391: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:05:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18561.550062 secs 2024-07-26T12:05:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18579.288005 secs 2024-07-26T12:05:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:34 smithi164 ceph-mon[36970]: pgmap v10392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:05:34.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:34 smithi163 ceph-mon[36873]: pgmap v10392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:05:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:34 smithi042 ceph-mon[32775]: pgmap v10392: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:05:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:36 smithi164 ceph-mon[36970]: pgmap v10393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:05:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:36 smithi163 ceph-mon[36873]: pgmap v10393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:05:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:36 smithi042 ceph-mon[32775]: pgmap v10393: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:05:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18566.550206 secs 2024-07-26T12:05:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18584.288067 secs 2024-07-26T12:05:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18566.550206 secs 2024-07-26T12:05:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18584.288067 secs 2024-07-26T12:05:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18566.550206 secs 2024-07-26T12:05:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18584.288067 secs 2024-07-26T12:05:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:38 smithi164 ceph-mon[36970]: pgmap v10394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:05:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:38 smithi163 ceph-mon[36873]: pgmap v10394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:05:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:38 smithi042 ceph-mon[32775]: pgmap v10394: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:05:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:40 smithi164 ceph-mon[36970]: pgmap v10395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:05:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:40 smithi163 ceph-mon[36873]: pgmap v10395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:05:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:40 smithi042 ceph-mon[32775]: pgmap v10395: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:05:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:42 smithi042 ceph-mon[32775]: pgmap v10396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18571.550353 secs 2024-07-26T12:05:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18589.288192 secs 2024-07-26T12:05:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:42 smithi164 ceph-mon[36970]: pgmap v10396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18571.550353 secs 2024-07-26T12:05:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18589.288192 secs 2024-07-26T12:05:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:42 smithi163 ceph-mon[36873]: pgmap v10396: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18571.550353 secs 2024-07-26T12:05:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18589.288192 secs 2024-07-26T12:05:44.084 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:05:44.082+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:05:44.084 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:05:44.083+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:44.084 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:05:44.083+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:05:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:44 smithi164 ceph-mon[36970]: pgmap v10397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:44 smithi163 ceph-mon[36873]: pgmap v10397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:44 smithi042 ceph-mon[32775]: pgmap v10397: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:46 smithi164 ceph-mon[36970]: pgmap v10398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:46 smithi163 ceph-mon[36873]: pgmap v10398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:46 smithi042 ceph-mon[32775]: pgmap v10398: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18576.550476 secs 2024-07-26T12:05:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18594.288311 secs 2024-07-26T12:05:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18576.550476 secs 2024-07-26T12:05:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18594.288311 secs 2024-07-26T12:05:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18576.550476 secs 2024-07-26T12:05:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18594.288311 secs 2024-07-26T12:05:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:48 smithi164 ceph-mon[36970]: pgmap v10399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:48 smithi163 ceph-mon[36873]: pgmap v10399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:48.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:48 smithi042 ceph-mon[32775]: pgmap v10399: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:50 smithi164 ceph-mon[36970]: pgmap v10400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:05:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:50 smithi163 ceph-mon[36873]: pgmap v10400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:05:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:50 smithi042 ceph-mon[32775]: pgmap v10400: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:05:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:52 smithi164 ceph-mon[36970]: pgmap v10401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18581.550600 secs 2024-07-26T12:05:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18599.288387 secs 2024-07-26T12:05:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:52 smithi163 ceph-mon[36873]: pgmap v10401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18581.550600 secs 2024-07-26T12:05:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18599.288387 secs 2024-07-26T12:05:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:52 smithi042 ceph-mon[32775]: pgmap v10401: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18581.550600 secs 2024-07-26T12:05:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18599.288387 secs 2024-07-26T12:05:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:54 smithi164 ceph-mon[36970]: pgmap v10402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:54 smithi163 ceph-mon[36873]: pgmap v10402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:54 smithi042 ceph-mon[32775]: pgmap v10402: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:56 smithi164 ceph-mon[36970]: pgmap v10403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:56 smithi163 ceph-mon[36873]: pgmap v10403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:56 smithi042 ceph-mon[32775]: pgmap v10403: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:05:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18586.550719 secs 2024-07-26T12:05:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18604.288506 secs 2024-07-26T12:05:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18586.550719 secs 2024-07-26T12:05:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18604.288506 secs 2024-07-26T12:05:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18586.550719 secs 2024-07-26T12:05:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18604.288506 secs 2024-07-26T12:05:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:05:58 smithi164 ceph-mon[36970]: pgmap v10404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:05:58 smithi163 ceph-mon[36873]: pgmap v10404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:05:58 smithi042 ceph-mon[32775]: pgmap v10404: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:05:59.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:05:59.088+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:05:59.089 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:05:59.088+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:59.090 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:05:59.088+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:06:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:00 smithi164 ceph-mon[36970]: pgmap v10405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:06:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:00 smithi163 ceph-mon[36873]: pgmap v10405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:06:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:00 smithi042 ceph-mon[32775]: pgmap v10405: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:06:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:02 smithi164 ceph-mon[36970]: pgmap v10406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18591.550846 secs 2024-07-26T12:06:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18609.288634 secs 2024-07-26T12:06:02.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:02 smithi163 ceph-mon[36873]: pgmap v10406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:02.511 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18591.550846 secs 2024-07-26T12:06:02.511 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18609.288634 secs 2024-07-26T12:06:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:02 smithi042 ceph-mon[32775]: pgmap v10406: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18591.550846 secs 2024-07-26T12:06:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18609.288634 secs 2024-07-26T12:06:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:04 smithi164 ceph-mon[36970]: pgmap v10407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:04.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:04 smithi163 ceph-mon[36873]: pgmap v10407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:04.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:04 smithi042 ceph-mon[32775]: pgmap v10407: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:06 smithi164 ceph-mon[36970]: pgmap v10408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:06 smithi163 ceph-mon[36873]: pgmap v10408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:06 smithi042 ceph-mon[32775]: pgmap v10408: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18596.550966 secs 2024-07-26T12:06:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18614.288751 secs 2024-07-26T12:06:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18596.550966 secs 2024-07-26T12:06:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18614.288751 secs 2024-07-26T12:06:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18596.550966 secs 2024-07-26T12:06:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18614.288751 secs 2024-07-26T12:06:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:08 smithi164 ceph-mon[36970]: pgmap v10409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:08 smithi163 ceph-mon[36873]: pgmap v10409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:08.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:08 smithi042 ceph-mon[32775]: pgmap v10409: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:10 smithi163 ceph-mon[36873]: pgmap v10410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:06:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:10 smithi042 ceph-mon[32775]: pgmap v10410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:06:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:10 smithi164 ceph-mon[36970]: pgmap v10410: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:06:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:12 smithi163 ceph-mon[36873]: pgmap v10411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18601.551098 secs 2024-07-26T12:06:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18619.288903 secs 2024-07-26T12:06:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:12 smithi042 ceph-mon[32775]: pgmap v10411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18601.551098 secs 2024-07-26T12:06:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18619.288903 secs 2024-07-26T12:06:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:12 smithi164 ceph-mon[36970]: pgmap v10411: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18601.551098 secs 2024-07-26T12:06:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18619.288903 secs 2024-07-26T12:06:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:06:14.089+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:06:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:06:14.089+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:14.090 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:06:14.089+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:06:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:14 smithi163 ceph-mon[36873]: pgmap v10412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:14 smithi042 ceph-mon[32775]: pgmap v10412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:14 smithi164 ceph-mon[36970]: pgmap v10412: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:16 smithi163 ceph-mon[36873]: pgmap v10413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:16 smithi042 ceph-mon[32775]: pgmap v10413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:16 smithi164 ceph-mon[36970]: pgmap v10413: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18606.551249 secs 2024-07-26T12:06:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18624.288999 secs 2024-07-26T12:06:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18606.551249 secs 2024-07-26T12:06:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18624.288999 secs 2024-07-26T12:06:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18606.551249 secs 2024-07-26T12:06:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18624.288999 secs 2024-07-26T12:06:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:18 smithi163 ceph-mon[36873]: pgmap v10414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:18 smithi042 ceph-mon[32775]: pgmap v10414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:18 smithi164 ceph-mon[36970]: pgmap v10414: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:20 smithi163 ceph-mon[36873]: pgmap v10415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:06:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:20 smithi042 ceph-mon[32775]: pgmap v10415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:06:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:20 smithi164 ceph-mon[36970]: pgmap v10415: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:06:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:22 smithi163 ceph-mon[36873]: pgmap v10416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18611.551391 secs 2024-07-26T12:06:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18629.289145 secs 2024-07-26T12:06:22.524 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:22 smithi164 ceph-mon[36970]: pgmap v10416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:22.524 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18611.551391 secs 2024-07-26T12:06:22.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18629.289145 secs 2024-07-26T12:06:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:22 smithi042 ceph-mon[32775]: pgmap v10416: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18611.551391 secs 2024-07-26T12:06:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18629.289145 secs 2024-07-26T12:06:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:24 smithi163 ceph-mon[36873]: pgmap v10417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:24 smithi042 ceph-mon[32775]: pgmap v10417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:24 smithi164 ceph-mon[36970]: pgmap v10417: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:26 smithi042 ceph-mon[32775]: pgmap v10418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:26 smithi164 ceph-mon[36970]: pgmap v10418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:26 smithi163 ceph-mon[36873]: pgmap v10418: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18616.551513 secs 2024-07-26T12:06:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:06:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18634.289239 secs 2024-07-26T12:06:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:06:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:06:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:06:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18616.551513 secs 2024-07-26T12:06:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:06:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18634.289239 secs 2024-07-26T12:06:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:06:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:06:27.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:06:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18616.551513 secs 2024-07-26T12:06:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:06:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18634.289239 secs 2024-07-26T12:06:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:06:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:06:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:06:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:28 smithi042 ceph-mon[32775]: pgmap v10419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:28 smithi042 ceph-mon[32775]: pgmap v10420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:06:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:28 smithi042 ceph-mon[32775]: pgmap v10421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:06:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:28 smithi164 ceph-mon[36970]: pgmap v10419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:28 smithi164 ceph-mon[36970]: pgmap v10420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:06:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:28 smithi164 ceph-mon[36970]: pgmap v10421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:06:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:28 smithi163 ceph-mon[36873]: pgmap v10419: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:28 smithi163 ceph-mon[36873]: pgmap v10420: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:06:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:28 smithi163 ceph-mon[36873]: pgmap v10421: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:06:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:06:29.094+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:06:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:06:29.094+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:06:29.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:06:29.094+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:30 smithi042 ceph-mon[32775]: pgmap v10422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:06:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:30 smithi164 ceph-mon[36970]: pgmap v10422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:06:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:30 smithi163 ceph-mon[36873]: pgmap v10422: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:06:32.559 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:32 smithi163 ceph-mon[36873]: pgmap v10423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:06:32.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18621.551641 secs 2024-07-26T12:06:32.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18639.289368 secs 2024-07-26T12:06:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:32 smithi042 ceph-mon[32775]: pgmap v10423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:06:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18621.551641 secs 2024-07-26T12:06:32.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18639.289368 secs 2024-07-26T12:06:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:32 smithi164 ceph-mon[36970]: pgmap v10423: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:06:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18621.551641 secs 2024-07-26T12:06:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18639.289368 secs 2024-07-26T12:06:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:34 smithi042 ceph-mon[32775]: pgmap v10424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:06:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:34 smithi164 ceph-mon[36970]: pgmap v10424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:06:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:34 smithi163 ceph-mon[36873]: pgmap v10424: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:06:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:36 smithi042 ceph-mon[32775]: pgmap v10425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:06:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:36 smithi164 ceph-mon[36970]: pgmap v10425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:06:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:36 smithi163 ceph-mon[36873]: pgmap v10425: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:06:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18626.551762 secs 2024-07-26T12:06:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18644.289489 secs 2024-07-26T12:06:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:37 smithi042 ceph-mon[32775]: pgmap v10426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:06:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18626.551762 secs 2024-07-26T12:06:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18644.289489 secs 2024-07-26T12:06:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:37 smithi164 ceph-mon[36970]: pgmap v10426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:06:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18626.551762 secs 2024-07-26T12:06:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18644.289489 secs 2024-07-26T12:06:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:37 smithi163 ceph-mon[36873]: pgmap v10426: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:06:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:39 smithi164 ceph-mon[36970]: pgmap v10427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:06:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:39 smithi163 ceph-mon[36873]: pgmap v10427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:06:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:39 smithi042 ceph-mon[32775]: pgmap v10427: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:06:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:41 smithi164 ceph-mon[36970]: pgmap v10428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18631.551913 secs 2024-07-26T12:06:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18649.289594 secs 2024-07-26T12:06:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:41 smithi163 ceph-mon[36873]: pgmap v10428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18631.551913 secs 2024-07-26T12:06:42.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18649.289594 secs 2024-07-26T12:06:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:41 smithi042 ceph-mon[32775]: pgmap v10428: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18631.551913 secs 2024-07-26T12:06:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18649.289594 secs 2024-07-26T12:06:44.095 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:06:44.094+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:06:44.096 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:06:44.094+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:44.096 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:06:44.094+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:06:44.197 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:43 smithi042 ceph-mon[32775]: pgmap v10429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:44.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:43 smithi164 ceph-mon[36970]: pgmap v10429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:43 smithi163 ceph-mon[36873]: pgmap v10429: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:45 smithi164 ceph-mon[36970]: pgmap v10430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:45 smithi163 ceph-mon[36873]: pgmap v10430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:45 smithi042 ceph-mon[32775]: pgmap v10430: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18636.552046 secs 2024-07-26T12:06:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18654.289716 secs 2024-07-26T12:06:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18636.552046 secs 2024-07-26T12:06:47.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18654.289716 secs 2024-07-26T12:06:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18636.552046 secs 2024-07-26T12:06:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18654.289716 secs 2024-07-26T12:06:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:47 smithi164 ceph-mon[36970]: pgmap v10431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:47 smithi163 ceph-mon[36873]: pgmap v10431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:47 smithi042 ceph-mon[32775]: pgmap v10431: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:50.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:49 smithi164 ceph-mon[36970]: pgmap v10432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:06:50.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:49 smithi163 ceph-mon[36873]: pgmap v10432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:06:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:49 smithi042 ceph-mon[32775]: pgmap v10432: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:06:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:51 smithi164 ceph-mon[36970]: pgmap v10433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18641.552168 secs 2024-07-26T12:06:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18659.289837 secs 2024-07-26T12:06:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:51 smithi163 ceph-mon[36873]: pgmap v10433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18641.552168 secs 2024-07-26T12:06:52.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18659.289837 secs 2024-07-26T12:06:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:51 smithi042 ceph-mon[32775]: pgmap v10433: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18641.552168 secs 2024-07-26T12:06:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18659.289837 secs 2024-07-26T12:06:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:53 smithi164 ceph-mon[36970]: pgmap v10434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:53 smithi163 ceph-mon[36873]: pgmap v10434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:53 smithi042 ceph-mon[32775]: pgmap v10434: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:55 smithi164 ceph-mon[36970]: pgmap v10435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:56.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:55 smithi163 ceph-mon[36873]: pgmap v10435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:55 smithi042 ceph-mon[32775]: pgmap v10435: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:06:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18646.552295 secs 2024-07-26T12:06:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18664.289956 secs 2024-07-26T12:06:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18646.552295 secs 2024-07-26T12:06:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18664.289956 secs 2024-07-26T12:06:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18646.552295 secs 2024-07-26T12:06:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18664.289956 secs 2024-07-26T12:06:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:57 smithi163 ceph-mon[36873]: pgmap v10436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:57 smithi042 ceph-mon[32775]: pgmap v10436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:57 smithi164 ceph-mon[36970]: pgmap v10436: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:06:59.101 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:06:59.099+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:06:59.101 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:06:59.099+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:59.101 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:06:59.099+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:07:00.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:06:59 smithi163 ceph-mon[36873]: pgmap v10437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:07:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:06:59 smithi042 ceph-mon[32775]: pgmap v10437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:07:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:06:59 smithi164 ceph-mon[36970]: pgmap v10437: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:07:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:01 smithi163 ceph-mon[36873]: pgmap v10438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18651.552380 secs 2024-07-26T12:07:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18669.290037 secs 2024-07-26T12:07:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:01 smithi042 ceph-mon[32775]: pgmap v10438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18651.552380 secs 2024-07-26T12:07:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18669.290037 secs 2024-07-26T12:07:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:01 smithi164 ceph-mon[36970]: pgmap v10438: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18651.552380 secs 2024-07-26T12:07:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18669.290037 secs 2024-07-26T12:07:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:03 smithi163 ceph-mon[36873]: pgmap v10439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:03 smithi042 ceph-mon[32775]: pgmap v10439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:03 smithi164 ceph-mon[36970]: pgmap v10439: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:05 smithi163 ceph-mon[36873]: pgmap v10440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:05 smithi042 ceph-mon[32775]: pgmap v10440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:06.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:05 smithi164 ceph-mon[36970]: pgmap v10440: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18656.552450 secs 2024-07-26T12:07:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18674.290129 secs 2024-07-26T12:07:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18656.552450 secs 2024-07-26T12:07:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18674.290129 secs 2024-07-26T12:07:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18656.552450 secs 2024-07-26T12:07:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18674.290129 secs 2024-07-26T12:07:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:08 smithi163 ceph-mon[36873]: pgmap v10441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:08 smithi042 ceph-mon[32775]: pgmap v10441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:08 smithi164 ceph-mon[36970]: pgmap v10441: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:10 smithi163 ceph-mon[36873]: pgmap v10442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:07:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:10 smithi042 ceph-mon[32775]: pgmap v10442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:07:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:10 smithi164 ceph-mon[36970]: pgmap v10442: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:07:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:12 smithi163 ceph-mon[36873]: pgmap v10443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18661.552585 secs 2024-07-26T12:07:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18679.290212 secs 2024-07-26T12:07:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:12 smithi042 ceph-mon[32775]: pgmap v10443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18661.552585 secs 2024-07-26T12:07:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18679.290212 secs 2024-07-26T12:07:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:12 smithi164 ceph-mon[36970]: pgmap v10443: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18661.552585 secs 2024-07-26T12:07:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18679.290212 secs 2024-07-26T12:07:14.101 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:07:14.100+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:07:14.101 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:07:14.100+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:14.101 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:07:14.100+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:07:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:14 smithi042 ceph-mon[32775]: pgmap v10444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:14 smithi164 ceph-mon[36970]: pgmap v10444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:14 smithi163 ceph-mon[36873]: pgmap v10444: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:16 smithi042 ceph-mon[32775]: pgmap v10445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:16 smithi164 ceph-mon[36970]: pgmap v10445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:16 smithi163 ceph-mon[36873]: pgmap v10445: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18666.552715 secs 2024-07-26T12:07:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18684.290293 secs 2024-07-26T12:07:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18666.552715 secs 2024-07-26T12:07:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18684.290293 secs 2024-07-26T12:07:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18666.552715 secs 2024-07-26T12:07:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18684.290293 secs 2024-07-26T12:07:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:18 smithi042 ceph-mon[32775]: pgmap v10446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:18 smithi164 ceph-mon[36970]: pgmap v10446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:18 smithi163 ceph-mon[36873]: pgmap v10446: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:20 smithi042 ceph-mon[32775]: pgmap v10447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:07:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:20 smithi164 ceph-mon[36970]: pgmap v10447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:07:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:20 smithi163 ceph-mon[36873]: pgmap v10447: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:07:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:22 smithi042 ceph-mon[32775]: pgmap v10448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18671.552785 secs 2024-07-26T12:07:22.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18689.290417 secs 2024-07-26T12:07:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:22 smithi164 ceph-mon[36970]: pgmap v10448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18671.552785 secs 2024-07-26T12:07:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18689.290417 secs 2024-07-26T12:07:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:22 smithi163 ceph-mon[36873]: pgmap v10448: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18671.552785 secs 2024-07-26T12:07:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18689.290417 secs 2024-07-26T12:07:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:24 smithi042 ceph-mon[32775]: pgmap v10449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:24 smithi164 ceph-mon[36970]: pgmap v10449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:24 smithi163 ceph-mon[36873]: pgmap v10449: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:26 smithi042 ceph-mon[32775]: pgmap v10450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:26 smithi164 ceph-mon[36970]: pgmap v10450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:26 smithi163 ceph-mon[36873]: pgmap v10450: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18676.552935 secs 2024-07-26T12:07:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18694.290536 secs 2024-07-26T12:07:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:27 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:07:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18676.552935 secs 2024-07-26T12:07:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18694.290536 secs 2024-07-26T12:07:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:27 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:07:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18676.552935 secs 2024-07-26T12:07:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18694.290536 secs 2024-07-26T12:07:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:27 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:07:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:28 smithi042 ceph-mon[32775]: pgmap v10451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:07:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:07:28.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:07:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:28 smithi164 ceph-mon[36970]: pgmap v10451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:07:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:07:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:07:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:28 smithi163 ceph-mon[36873]: pgmap v10451: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:07:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:07:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:07:29.107 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:07:29.106+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:07:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:07:29.106+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:07:29.106+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:07:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:29 smithi042 ceph-mon[32775]: pgmap v10452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T12:07:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:29 smithi042 ceph-mon[32775]: pgmap v10453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:07:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:29 smithi164 ceph-mon[36970]: pgmap v10452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T12:07:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:29 smithi164 ceph-mon[36970]: pgmap v10453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:07:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:29 smithi163 ceph-mon[36873]: pgmap v10452: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T12:07:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:29 smithi163 ceph-mon[36873]: pgmap v10453: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:07:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:31 smithi042 ceph-mon[32775]: pgmap v10454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:07:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:31 smithi164 ceph-mon[36970]: pgmap v10454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:07:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:31 smithi163 ceph-mon[36873]: pgmap v10454: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:07:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18681.553060 secs 2024-07-26T12:07:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18699.290641 secs 2024-07-26T12:07:32.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18681.553060 secs 2024-07-26T12:07:32.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18699.290641 secs 2024-07-26T12:07:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18681.553060 secs 2024-07-26T12:07:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18699.290641 secs 2024-07-26T12:07:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:33 smithi042 ceph-mon[32775]: pgmap v10455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:07:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:33 smithi164 ceph-mon[36970]: pgmap v10455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:07:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:33 smithi163 ceph-mon[36873]: pgmap v10455: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:07:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:35 smithi042 ceph-mon[32775]: pgmap v10456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:07:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:35 smithi164 ceph-mon[36970]: pgmap v10456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:07:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:35 smithi163 ceph-mon[36873]: pgmap v10456: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:07:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:37 smithi042 ceph-mon[32775]: pgmap v10457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:07:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18686.553188 secs 2024-07-26T12:07:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18704.290791 secs 2024-07-26T12:07:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:37 smithi164 ceph-mon[36970]: pgmap v10457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:07:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18686.553188 secs 2024-07-26T12:07:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18704.290791 secs 2024-07-26T12:07:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:37 smithi163 ceph-mon[36873]: pgmap v10457: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:07:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18686.553188 secs 2024-07-26T12:07:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18704.290791 secs 2024-07-26T12:07:39.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:39 smithi042 ceph-mon[32775]: pgmap v10458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:07:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:39 smithi164 ceph-mon[36970]: pgmap v10458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:07:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:39 smithi163 ceph-mon[36873]: pgmap v10458: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:07:41.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:41 smithi042 ceph-mon[32775]: pgmap v10459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:41 smithi164 ceph-mon[36970]: pgmap v10459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:41 smithi163 ceph-mon[36873]: pgmap v10459: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18691.553262 secs 2024-07-26T12:07:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18709.290890 secs 2024-07-26T12:07:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18691.553262 secs 2024-07-26T12:07:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18709.290890 secs 2024-07-26T12:07:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18691.553262 secs 2024-07-26T12:07:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18709.290890 secs 2024-07-26T12:07:43.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:43 smithi042 ceph-mon[32775]: pgmap v10460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:43 smithi164 ceph-mon[36970]: pgmap v10460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:43 smithi163 ceph-mon[36873]: pgmap v10460: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:44.108 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:07:44.107+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:07:44.108 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:07:44.107+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:44.108 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:07:44.107+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:07:45.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:45 smithi042 ceph-mon[32775]: pgmap v10461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:07:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:45 smithi164 ceph-mon[36970]: pgmap v10461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:07:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:45 smithi163 ceph-mon[36873]: pgmap v10461: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:07:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:47 smithi042 ceph-mon[32775]: pgmap v10462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18696.553365 secs 2024-07-26T12:07:47.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18714.291017 secs 2024-07-26T12:07:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:47 smithi164 ceph-mon[36970]: pgmap v10462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18696.553365 secs 2024-07-26T12:07:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18714.291017 secs 2024-07-26T12:07:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:47 smithi163 ceph-mon[36873]: pgmap v10462: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18696.553365 secs 2024-07-26T12:07:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18714.291017 secs 2024-07-26T12:07:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:49 smithi042 ceph-mon[32775]: pgmap v10463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:49 smithi164 ceph-mon[36970]: pgmap v10463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:49 smithi163 ceph-mon[36873]: pgmap v10463: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:51.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:51 smithi042 ceph-mon[32775]: pgmap v10464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:51.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:51 smithi164 ceph-mon[36970]: pgmap v10464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:51 smithi163 ceph-mon[36873]: pgmap v10464: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:52.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18701.553488 secs 2024-07-26T12:07:52.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18719.291143 secs 2024-07-26T12:07:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18701.553488 secs 2024-07-26T12:07:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18719.291143 secs 2024-07-26T12:07:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18701.553488 secs 2024-07-26T12:07:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18719.291143 secs 2024-07-26T12:07:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:53 smithi042 ceph-mon[32775]: pgmap v10465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:53.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:53 smithi164 ceph-mon[36970]: pgmap v10465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:53 smithi163 ceph-mon[36873]: pgmap v10465: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:55.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:55 smithi042 ceph-mon[32775]: pgmap v10466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:07:55.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:55 smithi164 ceph-mon[36970]: pgmap v10466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:07:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:55 smithi163 ceph-mon[36873]: pgmap v10466: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:07:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:57 smithi042 ceph-mon[32775]: pgmap v10467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18706.553635 secs 2024-07-26T12:07:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18724.291276 secs 2024-07-26T12:07:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:57 smithi164 ceph-mon[36970]: pgmap v10467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18706.553635 secs 2024-07-26T12:07:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18724.291276 secs 2024-07-26T12:07:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:57 smithi163 ceph-mon[36873]: pgmap v10467: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:07:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18706.553635 secs 2024-07-26T12:07:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18724.291276 secs 2024-07-26T12:07:59.113 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:07:59.112+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:07:59.114 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:07:59.112+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:59.114 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:07:59.112+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:07:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:07:59 smithi042 ceph-mon[32775]: pgmap v10468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:59.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:07:59 smithi164 ceph-mon[36970]: pgmap v10468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:07:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:07:59 smithi163 ceph-mon[36873]: pgmap v10468: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:08:00.818+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:08:00.819 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:08:00.818+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658004740 con 0x7ff680102000 2024-07-26T12:08:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:08:00.818+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6700e9050 con 0x7ff680102000 2024-07-26T12:08:00.820 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:08:00.818+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:08:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:01 smithi042 ceph-mon[32775]: pgmap v10469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:08:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:01 smithi164 ceph-mon[36970]: pgmap v10469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:08:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:01 smithi163 ceph-mon[36873]: pgmap v10469: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:08:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18711.553764 secs 2024-07-26T12:08:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18729.291400 secs 2024-07-26T12:08:02.408 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18711.553764 secs 2024-07-26T12:08:02.409 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18729.291400 secs 2024-07-26T12:08:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18711.553764 secs 2024-07-26T12:08:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18729.291400 secs 2024-07-26T12:08:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:03 smithi042 ceph-mon[32775]: pgmap v10470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:08:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:03 smithi164 ceph-mon[36970]: pgmap v10470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:08:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:03 smithi163 ceph-mon[36873]: pgmap v10470: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:08:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:05 smithi042 ceph-mon[32775]: pgmap v10471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:08:05.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:05 smithi164 ceph-mon[36970]: pgmap v10471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:08:05.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:05 smithi163 ceph-mon[36873]: pgmap v10471: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:08:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:07 smithi042 ceph-mon[32775]: pgmap v10472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:08:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18716.553886 secs 2024-07-26T12:08:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18734.291521 secs 2024-07-26T12:08:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:07 smithi164 ceph-mon[36970]: pgmap v10472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:08:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18716.553886 secs 2024-07-26T12:08:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18734.291521 secs 2024-07-26T12:08:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:07 smithi163 ceph-mon[36873]: pgmap v10472: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:08:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18716.553886 secs 2024-07-26T12:08:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18734.291521 secs 2024-07-26T12:08:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:09 smithi042 ceph-mon[32775]: pgmap v10473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:08:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:09 smithi164 ceph-mon[36970]: pgmap v10473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:08:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:09 smithi163 ceph-mon[36873]: pgmap v10473: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:08:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:11 smithi042 ceph-mon[32775]: pgmap v10474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:08:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:11 smithi164 ceph-mon[36970]: pgmap v10474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:08:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:11 smithi163 ceph-mon[36873]: pgmap v10474: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:08:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18721.554006 secs 2024-07-26T12:08:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18739.291650 secs 2024-07-26T12:08:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18721.554006 secs 2024-07-26T12:08:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18739.291650 secs 2024-07-26T12:08:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18721.554006 secs 2024-07-26T12:08:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18739.291650 secs 2024-07-26T12:08:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:13 smithi164 ceph-mon[36970]: pgmap v10475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:13 smithi163 ceph-mon[36873]: pgmap v10475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:13 smithi042 ceph-mon[32775]: pgmap v10475: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:14.114 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:08:14.113+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:08:14.114 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:08:14.113+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:14.114 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:08:14.113+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:08:15.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:15 smithi164 ceph-mon[36970]: pgmap v10476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:08:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:15 smithi163 ceph-mon[36873]: pgmap v10476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:08:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:15 smithi042 ceph-mon[32775]: pgmap v10476: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:08:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:17 smithi042 ceph-mon[32775]: pgmap v10477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18726.554134 secs 2024-07-26T12:08:17.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18744.291768 secs 2024-07-26T12:08:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:17 smithi164 ceph-mon[36970]: pgmap v10477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18726.554134 secs 2024-07-26T12:08:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18744.291768 secs 2024-07-26T12:08:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:17 smithi163 ceph-mon[36873]: pgmap v10477: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18726.554134 secs 2024-07-26T12:08:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18744.291768 secs 2024-07-26T12:08:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:19 smithi164 ceph-mon[36970]: pgmap v10478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:19 smithi163 ceph-mon[36873]: pgmap v10478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:19.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:19 smithi042 ceph-mon[32775]: pgmap v10478: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:21.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:21 smithi164 ceph-mon[36970]: pgmap v10479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:21.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:21 smithi163 ceph-mon[36873]: pgmap v10479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:21 smithi042 ceph-mon[32775]: pgmap v10479: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18731.554263 secs 2024-07-26T12:08:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18749.291890 secs 2024-07-26T12:08:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18731.554263 secs 2024-07-26T12:08:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18749.291890 secs 2024-07-26T12:08:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18731.554263 secs 2024-07-26T12:08:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18749.291890 secs 2024-07-26T12:08:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:23 smithi164 ceph-mon[36970]: pgmap v10480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:23 smithi163 ceph-mon[36873]: pgmap v10480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:23 smithi042 ceph-mon[32775]: pgmap v10480: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:25 smithi164 ceph-mon[36970]: pgmap v10481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:08:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:25 smithi163 ceph-mon[36873]: pgmap v10481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:08:25.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:25 smithi042 ceph-mon[32775]: pgmap v10481: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:08:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:27 smithi164 ceph-mon[36970]: pgmap v10482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18736.554392 secs 2024-07-26T12:08:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18754.292011 secs 2024-07-26T12:08:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:27 smithi163 ceph-mon[36873]: pgmap v10482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18736.554392 secs 2024-07-26T12:08:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18754.292011 secs 2024-07-26T12:08:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:27 smithi042 ceph-mon[32775]: pgmap v10482: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18736.554392 secs 2024-07-26T12:08:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18754.292011 secs 2024-07-26T12:08:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:08:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:08:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:08:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:08:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:08:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:08:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:08:28.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:08:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:08:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:08:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:08:28.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:08:29.119 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:08:29.118+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:08:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:08:29.118+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:29.120 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:08:29.118+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:08:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:29 smithi164 ceph-mon[36970]: pgmap v10483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:29 smithi164 ceph-mon[36970]: pgmap v10484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:08:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:29 smithi164 ceph-mon[36970]: pgmap v10485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:08:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:29 smithi163 ceph-mon[36873]: pgmap v10483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:29 smithi163 ceph-mon[36873]: pgmap v10484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:08:29.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:29 smithi163 ceph-mon[36873]: pgmap v10485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:08:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:29 smithi042 ceph-mon[32775]: pgmap v10483: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:29 smithi042 ceph-mon[32775]: pgmap v10484: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:08:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:29 smithi042 ceph-mon[32775]: pgmap v10485: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:08:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:31 smithi164 ceph-mon[36970]: pgmap v10486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:08:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:31 smithi163 ceph-mon[36873]: pgmap v10486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:08:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:31 smithi042 ceph-mon[32775]: pgmap v10486: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:08:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18741.554507 secs 2024-07-26T12:08:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18759.292136 secs 2024-07-26T12:08:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18741.554507 secs 2024-07-26T12:08:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18759.292136 secs 2024-07-26T12:08:32.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18741.554507 secs 2024-07-26T12:08:32.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18759.292136 secs 2024-07-26T12:08:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:33 smithi042 ceph-mon[32775]: pgmap v10487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:08:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:33 smithi164 ceph-mon[36970]: pgmap v10487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:08:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:33 smithi163 ceph-mon[36873]: pgmap v10487: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:08:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:35 smithi164 ceph-mon[36970]: pgmap v10488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:08:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:35 smithi163 ceph-mon[36873]: pgmap v10488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:08:35.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:35 smithi042 ceph-mon[32775]: pgmap v10488: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:08:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:37 smithi164 ceph-mon[36970]: pgmap v10489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:08:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18746.554604 secs 2024-07-26T12:08:37.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18764.292247 secs 2024-07-26T12:08:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:37 smithi163 ceph-mon[36873]: pgmap v10489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:08:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18746.554604 secs 2024-07-26T12:08:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18764.292247 secs 2024-07-26T12:08:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:37 smithi042 ceph-mon[32775]: pgmap v10489: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:08:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18746.554604 secs 2024-07-26T12:08:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18764.292247 secs 2024-07-26T12:08:39.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:39 smithi164 ceph-mon[36970]: pgmap v10490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:08:39.510 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:39 smithi163 ceph-mon[36873]: pgmap v10490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:08:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:39 smithi042 ceph-mon[32775]: pgmap v10490: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:08:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:41 smithi164 ceph-mon[36970]: pgmap v10491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:41.515 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:41 smithi163 ceph-mon[36873]: pgmap v10491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:41 smithi042 ceph-mon[32775]: pgmap v10491: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18751.554755 secs 2024-07-26T12:08:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18769.292375 secs 2024-07-26T12:08:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18751.554755 secs 2024-07-26T12:08:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18769.292375 secs 2024-07-26T12:08:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18751.554755 secs 2024-07-26T12:08:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18769.292375 secs 2024-07-26T12:08:43.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:43 smithi164 ceph-mon[36970]: pgmap v10492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:43 smithi163 ceph-mon[36873]: pgmap v10492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:43 smithi042 ceph-mon[32775]: pgmap v10492: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:08:44.119+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:08:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:08:44.119+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:08:44.119+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:08:45.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:45 smithi164 ceph-mon[36970]: pgmap v10493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:08:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:45 smithi163 ceph-mon[36873]: pgmap v10493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:08:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:45 smithi042 ceph-mon[32775]: pgmap v10493: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:08:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:47 smithi164 ceph-mon[36970]: pgmap v10494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18756.554864 secs 2024-07-26T12:08:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18774.292502 secs 2024-07-26T12:08:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:47 smithi163 ceph-mon[36873]: pgmap v10494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18756.554864 secs 2024-07-26T12:08:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18774.292502 secs 2024-07-26T12:08:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:47 smithi042 ceph-mon[32775]: pgmap v10494: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18756.554864 secs 2024-07-26T12:08:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18774.292502 secs 2024-07-26T12:08:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:49 smithi163 ceph-mon[36873]: pgmap v10495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:49 smithi042 ceph-mon[32775]: pgmap v10495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:49 smithi164 ceph-mon[36970]: pgmap v10495: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:51 smithi163 ceph-mon[36873]: pgmap v10496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:51 smithi042 ceph-mon[32775]: pgmap v10496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:51 smithi164 ceph-mon[36970]: pgmap v10496: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18761.554985 secs 2024-07-26T12:08:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18779.292654 secs 2024-07-26T12:08:52.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18761.554985 secs 2024-07-26T12:08:52.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18779.292654 secs 2024-07-26T12:08:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18761.554985 secs 2024-07-26T12:08:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18779.292654 secs 2024-07-26T12:08:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:53 smithi163 ceph-mon[36873]: pgmap v10497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:53 smithi042 ceph-mon[32775]: pgmap v10497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:53 smithi164 ceph-mon[36970]: pgmap v10497: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:55 smithi163 ceph-mon[36873]: pgmap v10498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:08:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:55 smithi042 ceph-mon[32775]: pgmap v10498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:08:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:55 smithi164 ceph-mon[36970]: pgmap v10498: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:08:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:57 smithi163 ceph-mon[36873]: pgmap v10499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18766.555110 secs 2024-07-26T12:08:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18784.292792 secs 2024-07-26T12:08:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:57 smithi042 ceph-mon[32775]: pgmap v10499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18766.555110 secs 2024-07-26T12:08:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18784.292792 secs 2024-07-26T12:08:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:57 smithi164 ceph-mon[36970]: pgmap v10499: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:08:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18766.555110 secs 2024-07-26T12:08:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18784.292792 secs 2024-07-26T12:08:59.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:08:59.125+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:08:59.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:08:59.125+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:08:59.126 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:08:59.125+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:59.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:08:59 smithi163 ceph-mon[36873]: pgmap v10500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:08:59 smithi042 ceph-mon[32775]: pgmap v10500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:08:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:08:59 smithi164 ceph-mon[36970]: pgmap v10500: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:01.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:01 smithi163 ceph-mon[36873]: pgmap v10501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:01 smithi042 ceph-mon[32775]: pgmap v10501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:01 smithi164 ceph-mon[36970]: pgmap v10501: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18771.555233 secs 2024-07-26T12:09:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18789.292940 secs 2024-07-26T12:09:02.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18771.555233 secs 2024-07-26T12:09:02.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18789.292940 secs 2024-07-26T12:09:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18771.555233 secs 2024-07-26T12:09:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18789.292940 secs 2024-07-26T12:09:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:03 smithi042 ceph-mon[32775]: pgmap v10502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:03 smithi164 ceph-mon[36970]: pgmap v10502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:03 smithi163 ceph-mon[36873]: pgmap v10502: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:05 smithi042 ceph-mon[32775]: pgmap v10503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:09:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:05 smithi164 ceph-mon[36970]: pgmap v10503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:09:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:05 smithi163 ceph-mon[36873]: pgmap v10503: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:09:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:07 smithi042 ceph-mon[32775]: pgmap v10504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18776.555359 secs 2024-07-26T12:09:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18794.293068 secs 2024-07-26T12:09:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:07 smithi164 ceph-mon[36970]: pgmap v10504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18776.555359 secs 2024-07-26T12:09:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18794.293068 secs 2024-07-26T12:09:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:07 smithi163 ceph-mon[36873]: pgmap v10504: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18776.555359 secs 2024-07-26T12:09:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18794.293068 secs 2024-07-26T12:09:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:09 smithi042 ceph-mon[32775]: pgmap v10505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:09 smithi164 ceph-mon[36970]: pgmap v10505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:09 smithi163 ceph-mon[36873]: pgmap v10505: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:11 smithi042 ceph-mon[32775]: pgmap v10506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:11 smithi164 ceph-mon[36970]: pgmap v10506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:11 smithi163 ceph-mon[36873]: pgmap v10506: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18781.555441 secs 2024-07-26T12:09:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18799.293188 secs 2024-07-26T12:09:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18781.555441 secs 2024-07-26T12:09:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18799.293188 secs 2024-07-26T12:09:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18781.555441 secs 2024-07-26T12:09:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18799.293188 secs 2024-07-26T12:09:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:13 smithi042 ceph-mon[32775]: pgmap v10507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:13 smithi164 ceph-mon[36970]: pgmap v10507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:13 smithi163 ceph-mon[36873]: pgmap v10507: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:09:14.126+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:09:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:09:14.126+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:14.128 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:09:14.126+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:09:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:15 smithi042 ceph-mon[32775]: pgmap v10508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:09:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:15 smithi164 ceph-mon[36970]: pgmap v10508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:09:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:15 smithi163 ceph-mon[36873]: pgmap v10508: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:09:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:17 smithi042 ceph-mon[32775]: pgmap v10509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18786.555540 secs 2024-07-26T12:09:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18804.293310 secs 2024-07-26T12:09:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:17 smithi164 ceph-mon[36970]: pgmap v10509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18786.555540 secs 2024-07-26T12:09:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18804.293310 secs 2024-07-26T12:09:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:17 smithi163 ceph-mon[36873]: pgmap v10509: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18786.555540 secs 2024-07-26T12:09:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18804.293310 secs 2024-07-26T12:09:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:19 smithi042 ceph-mon[32775]: pgmap v10510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:19 smithi164 ceph-mon[36970]: pgmap v10510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:19 smithi163 ceph-mon[36873]: pgmap v10510: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:21 smithi042 ceph-mon[32775]: pgmap v10511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:21 smithi164 ceph-mon[36970]: pgmap v10511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:21 smithi163 ceph-mon[36873]: pgmap v10511: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:22.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18791.555664 secs 2024-07-26T12:09:22.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18809.293439 secs 2024-07-26T12:09:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18791.555664 secs 2024-07-26T12:09:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18809.293439 secs 2024-07-26T12:09:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18791.555664 secs 2024-07-26T12:09:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18809.293439 secs 2024-07-26T12:09:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:23 smithi042 ceph-mon[32775]: pgmap v10512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:23 smithi164 ceph-mon[36970]: pgmap v10512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:23 smithi163 ceph-mon[36873]: pgmap v10512: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:25.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:25 smithi042 ceph-mon[32775]: pgmap v10513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:09:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:25 smithi164 ceph-mon[36970]: pgmap v10513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:09:25.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:25 smithi163 ceph-mon[36873]: pgmap v10513: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:09:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:27 smithi042 ceph-mon[32775]: pgmap v10514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18796.555785 secs 2024-07-26T12:09:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18814.293563 secs 2024-07-26T12:09:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:27 smithi164 ceph-mon[36970]: pgmap v10514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18796.555785 secs 2024-07-26T12:09:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18814.293563 secs 2024-07-26T12:09:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:27 smithi163 ceph-mon[36873]: pgmap v10514: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18796.555785 secs 2024-07-26T12:09:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18814.293563 secs 2024-07-26T12:09:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:28 smithi042 ceph-mon[32775]: pgmap v10515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:09:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:28 smithi164 ceph-mon[36970]: pgmap v10515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:09:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:28 smithi163 ceph-mon[36873]: pgmap v10515: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:09:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:09:29.132+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:09:29.133 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:09:29.132+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:29.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:09:29.132+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:09:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:09:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:09:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:29 smithi042 ceph-mon[32775]: pgmap v10516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T12:09:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:29 smithi042 ceph-mon[32775]: pgmap v10517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:09:29.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:09:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:09:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:09:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:29 smithi164 ceph-mon[36970]: pgmap v10516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T12:09:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:29 smithi164 ceph-mon[36970]: pgmap v10517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:09:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:09:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:09:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:09:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:29 smithi163 ceph-mon[36873]: pgmap v10516: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T12:09:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:29 smithi163 ceph-mon[36873]: pgmap v10517: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:09:29.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:09:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:09:30.833+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T12:09:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:09:30.833+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:09:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:09:30.833+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6700e9050 con 0x7ff680102000 2024-07-26T12:09:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:09:30.833+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:09:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:31 smithi042 ceph-mon[32775]: pgmap v10518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:09:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18801.555905 secs 2024-07-26T12:09:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:31 smithi164 ceph-mon[36970]: pgmap v10518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:09:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18801.555905 secs 2024-07-26T12:09:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:31 smithi163 ceph-mon[36873]: pgmap v10518: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:09:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18801.555905 secs 2024-07-26T12:09:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18819.293689 secs 2024-07-26T12:09:33.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18819.293689 secs 2024-07-26T12:09:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18819.293689 secs 2024-07-26T12:09:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:33 smithi164 ceph-mon[36970]: pgmap v10519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:09:34.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:33 smithi163 ceph-mon[36873]: pgmap v10519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:09:34.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:33 smithi042 ceph-mon[32775]: pgmap v10519: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:09:35.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:35 smithi164 ceph-mon[36970]: pgmap v10520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:09:36.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:35 smithi163 ceph-mon[36873]: pgmap v10520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:09:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:35 smithi042 ceph-mon[32775]: pgmap v10520: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:09:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18806.556003 secs 2024-07-26T12:09:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18806.556003 secs 2024-07-26T12:09:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18806.556003 secs 2024-07-26T12:09:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18824.293767 secs 2024-07-26T12:09:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:37 smithi164 ceph-mon[36970]: pgmap v10521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:09:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18824.293767 secs 2024-07-26T12:09:38.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:37 smithi163 ceph-mon[36873]: pgmap v10521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:09:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18824.293767 secs 2024-07-26T12:09:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:37 smithi042 ceph-mon[32775]: pgmap v10521: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:09:39.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:39 smithi164 ceph-mon[36970]: pgmap v10522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:09:40.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:39 smithi163 ceph-mon[36873]: pgmap v10522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:09:40.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:39 smithi042 ceph-mon[32775]: pgmap v10522: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:09:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:41 smithi164 ceph-mon[36970]: pgmap v10523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18811.556152 secs 2024-07-26T12:09:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:41 smithi163 ceph-mon[36873]: pgmap v10523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18811.556152 secs 2024-07-26T12:09:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:41 smithi042 ceph-mon[32775]: pgmap v10523: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18811.556152 secs 2024-07-26T12:09:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18829.293884 secs 2024-07-26T12:09:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18829.293884 secs 2024-07-26T12:09:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18829.293884 secs 2024-07-26T12:09:43.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:43 smithi164 ceph-mon[36970]: pgmap v10524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:44.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:43 smithi163 ceph-mon[36873]: pgmap v10524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:09:44.133+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:09:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:09:44.133+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:09:44.133+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:09:44.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:43 smithi042 ceph-mon[32775]: pgmap v10524: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:45.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:45 smithi164 ceph-mon[36970]: pgmap v10525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:46.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:45 smithi163 ceph-mon[36873]: pgmap v10525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:46.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:45 smithi042 ceph-mon[32775]: pgmap v10525: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:46.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18816.556256 secs 2024-07-26T12:09:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18816.556256 secs 2024-07-26T12:09:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18816.556256 secs 2024-07-26T12:09:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18834.293980 secs 2024-07-26T12:09:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:47 smithi164 ceph-mon[36970]: pgmap v10526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18834.293980 secs 2024-07-26T12:09:48.088 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:47 smithi163 ceph-mon[36873]: pgmap v10526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18834.293980 secs 2024-07-26T12:09:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:47 smithi042 ceph-mon[32775]: pgmap v10526: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:49.947 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:49 smithi042 ceph-mon[32775]: pgmap v10527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:09:49.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:49 smithi164 ceph-mon[36970]: pgmap v10527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:09:50.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:49 smithi163 ceph-mon[36873]: pgmap v10527: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:09:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:51 smithi164 ceph-mon[36970]: pgmap v10528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18821.556379 secs 2024-07-26T12:09:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:51 smithi163 ceph-mon[36873]: pgmap v10528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18821.556379 secs 2024-07-26T12:09:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:51 smithi042 ceph-mon[32775]: pgmap v10528: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:52.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18821.556379 secs 2024-07-26T12:09:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18839.294106 secs 2024-07-26T12:09:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18839.294106 secs 2024-07-26T12:09:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18839.294106 secs 2024-07-26T12:09:53.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:53 smithi164 ceph-mon[36970]: pgmap v10529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:54.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:53 smithi163 ceph-mon[36873]: pgmap v10529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:54.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:53 smithi042 ceph-mon[32775]: pgmap v10529: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:55.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:55 smithi164 ceph-mon[36970]: pgmap v10530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:56.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:55 smithi163 ceph-mon[36873]: pgmap v10530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:56.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:55 smithi042 ceph-mon[32775]: pgmap v10530: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:09:56.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18826.556526 secs 2024-07-26T12:09:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18826.556526 secs 2024-07-26T12:09:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18826.556526 secs 2024-07-26T12:09:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18844.294226 secs 2024-07-26T12:09:57.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:57 smithi164 ceph-mon[36970]: pgmap v10531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18844.294226 secs 2024-07-26T12:09:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:57 smithi163 ceph-mon[36873]: pgmap v10531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18844.294226 secs 2024-07-26T12:09:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:57 smithi042 ceph-mon[32775]: pgmap v10531: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:09:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:09:59.138+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:09:59.139 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:09:59.138+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:59.140 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:09:59.138+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:09:59.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:09:59 smithi164 ceph-mon[36970]: pgmap v10532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:10:00.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:09:59 smithi163 ceph-mon[36873]: pgmap v10532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:10:00.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:09:59 smithi042 ceph-mon[32775]: pgmap v10532: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:10:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T12:10:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T12:10:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T12:10:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:01 smithi163 ceph-mon[36873]: pgmap v10533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18831.556606 secs 2024-07-26T12:10:02.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:01 smithi042 ceph-mon[32775]: pgmap v10533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:02.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18831.556606 secs 2024-07-26T12:10:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:01 smithi164 ceph-mon[36970]: pgmap v10533: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18831.556606 secs 2024-07-26T12:10:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18849.294346 secs 2024-07-26T12:10:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18849.294346 secs 2024-07-26T12:10:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18849.294346 secs 2024-07-26T12:10:04.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:03 smithi163 ceph-mon[36873]: pgmap v10534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:04.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:03 smithi042 ceph-mon[32775]: pgmap v10534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:03 smithi164 ceph-mon[36970]: pgmap v10534: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:06.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:05 smithi163 ceph-mon[36873]: pgmap v10535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:06.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:05 smithi042 ceph-mon[32775]: pgmap v10535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:06.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:05 smithi164 ceph-mon[36970]: pgmap v10535: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18836.556729 secs 2024-07-26T12:10:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18836.556729 secs 2024-07-26T12:10:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18836.556729 secs 2024-07-26T12:10:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18854.294472 secs 2024-07-26T12:10:08.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:07 smithi163 ceph-mon[36873]: pgmap v10536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18854.294472 secs 2024-07-26T12:10:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:07 smithi042 ceph-mon[32775]: pgmap v10536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18854.294472 secs 2024-07-26T12:10:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:07 smithi164 ceph-mon[36970]: pgmap v10536: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:10.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:09 smithi163 ceph-mon[36873]: pgmap v10537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:10:10.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:09 smithi042 ceph-mon[32775]: pgmap v10537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:10:10.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:09 smithi164 ceph-mon[36970]: pgmap v10537: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:10:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:11 smithi163 ceph-mon[36873]: pgmap v10538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18841.556846 secs 2024-07-26T12:10:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:11 smithi042 ceph-mon[32775]: pgmap v10538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18841.556846 secs 2024-07-26T12:10:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:11 smithi164 ceph-mon[36970]: pgmap v10538: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18841.556846 secs 2024-07-26T12:10:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18859.294594 secs 2024-07-26T12:10:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18859.294594 secs 2024-07-26T12:10:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18859.294594 secs 2024-07-26T12:10:14.141 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:14.139+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:10:14.141 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:14.139+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:14.141 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:14.139+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:10:14.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:13 smithi042 ceph-mon[32775]: pgmap v10539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:14.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:13 smithi164 ceph-mon[36970]: pgmap v10539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:13 smithi163 ceph-mon[36873]: pgmap v10539: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:16.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:15 smithi042 ceph-mon[32775]: pgmap v10540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:16.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:15 smithi164 ceph-mon[36970]: pgmap v10540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:15 smithi163 ceph-mon[36873]: pgmap v10540: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18846.556970 secs 2024-07-26T12:10:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18846.556970 secs 2024-07-26T12:10:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18846.556970 secs 2024-07-26T12:10:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18864.294674 secs 2024-07-26T12:10:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:17 smithi042 ceph-mon[32775]: pgmap v10541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18864.294674 secs 2024-07-26T12:10:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:17 smithi164 ceph-mon[36970]: pgmap v10541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18864.294674 secs 2024-07-26T12:10:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:17 smithi163 ceph-mon[36873]: pgmap v10541: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:20.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:19 smithi042 ceph-mon[32775]: pgmap v10542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:10:20.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:19 smithi164 ceph-mon[36970]: pgmap v10542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:10:20.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:19 smithi163 ceph-mon[36873]: pgmap v10542: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:10:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:21 smithi042 ceph-mon[32775]: pgmap v10543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:22.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18851.557044 secs 2024-07-26T12:10:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:21 smithi164 ceph-mon[36970]: pgmap v10543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18851.557044 secs 2024-07-26T12:10:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:21 smithi163 ceph-mon[36873]: pgmap v10543: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18851.557044 secs 2024-07-26T12:10:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18869.294794 secs 2024-07-26T12:10:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18869.294794 secs 2024-07-26T12:10:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18869.294794 secs 2024-07-26T12:10:24.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:23 smithi042 ceph-mon[32775]: pgmap v10544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:24.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:23 smithi164 ceph-mon[36970]: pgmap v10544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:24.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:23 smithi163 ceph-mon[36873]: pgmap v10544: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:25 smithi042 ceph-mon[32775]: pgmap v10545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:26.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:25 smithi164 ceph-mon[36970]: pgmap v10545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:26.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:25 smithi163 ceph-mon[36873]: pgmap v10545: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18856.557187 secs 2024-07-26T12:10:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18856.557187 secs 2024-07-26T12:10:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18856.557187 secs 2024-07-26T12:10:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18874.294947 secs 2024-07-26T12:10:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:27 smithi042 ceph-mon[32775]: pgmap v10546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18874.294947 secs 2024-07-26T12:10:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:27 smithi164 ceph-mon[36970]: pgmap v10546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18874.294947 secs 2024-07-26T12:10:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:27 smithi163 ceph-mon[36873]: pgmap v10546: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:29.144+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:10:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:29.144+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:29.145 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:29.144+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:10:29.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:28 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:10:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:28 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:10:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:28 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:10:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:29 smithi042 ceph-mon[32775]: pgmap v10547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:10:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:10:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:10:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:29 smithi042 ceph-mon[32775]: pgmap v10548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:10:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:29 smithi042 ceph-mon[32775]: pgmap v10549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:10:30.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:10:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:29 smithi164 ceph-mon[36970]: pgmap v10547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:10:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:10:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:10:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:29 smithi164 ceph-mon[36970]: pgmap v10548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:10:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:29 smithi164 ceph-mon[36970]: pgmap v10549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:10:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:10:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:29 smithi163 ceph-mon[36873]: pgmap v10547: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:10:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:10:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:10:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:29 smithi163 ceph-mon[36873]: pgmap v10548: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:10:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:29 smithi163 ceph-mon[36873]: pgmap v10549: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:10:30.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:10:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18861.557323 secs 2024-07-26T12:10:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:32 smithi164 ceph-mon[36970]: pgmap v10550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:10:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18879.295044 secs 2024-07-26T12:10:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18861.557323 secs 2024-07-26T12:10:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:32 smithi163 ceph-mon[36873]: pgmap v10550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:10:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18879.295044 secs 2024-07-26T12:10:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18861.557323 secs 2024-07-26T12:10:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:32 smithi042 ceph-mon[32775]: pgmap v10550: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:10:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18879.295044 secs 2024-07-26T12:10:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:34 smithi164 ceph-mon[36970]: pgmap v10551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:10:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:34 smithi163 ceph-mon[36873]: pgmap v10551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:10:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:34 smithi042 ceph-mon[32775]: pgmap v10551: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:10:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:36 smithi164 ceph-mon[36970]: pgmap v10552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:10:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:36 smithi163 ceph-mon[36873]: pgmap v10552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:10:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:36 smithi042 ceph-mon[32775]: pgmap v10552: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:10:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18866.557408 secs 2024-07-26T12:10:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18884.295111 secs 2024-07-26T12:10:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18866.557408 secs 2024-07-26T12:10:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18884.295111 secs 2024-07-26T12:10:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18866.557408 secs 2024-07-26T12:10:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18884.295111 secs 2024-07-26T12:10:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:38 smithi164 ceph-mon[36970]: pgmap v10553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:10:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:38 smithi163 ceph-mon[36873]: pgmap v10553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:10:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:38 smithi042 ceph-mon[32775]: pgmap v10553: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:10:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:40 smithi164 ceph-mon[36970]: pgmap v10554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:10:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:40 smithi163 ceph-mon[36873]: pgmap v10554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:10:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:40 smithi042 ceph-mon[32775]: pgmap v10554: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:10:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18871.557480 secs 2024-07-26T12:10:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:42 smithi163 ceph-mon[36873]: pgmap v10555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18889.295177 secs 2024-07-26T12:10:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18871.557480 secs 2024-07-26T12:10:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:42 smithi042 ceph-mon[32775]: pgmap v10555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18889.295177 secs 2024-07-26T12:10:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18871.557480 secs 2024-07-26T12:10:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:42 smithi164 ceph-mon[36970]: pgmap v10555: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18889.295177 secs 2024-07-26T12:10:44.146 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:44.145+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:10:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:44.145+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:10:44.147 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:44.145+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:44 smithi163 ceph-mon[36873]: pgmap v10556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:44 smithi042 ceph-mon[32775]: pgmap v10556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:44 smithi164 ceph-mon[36970]: pgmap v10556: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:46 smithi163 ceph-mon[36873]: pgmap v10557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:46 smithi042 ceph-mon[32775]: pgmap v10557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:46 smithi164 ceph-mon[36970]: pgmap v10557: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18876.557563 secs 2024-07-26T12:10:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18894.295264 secs 2024-07-26T12:10:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18876.557563 secs 2024-07-26T12:10:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18894.295264 secs 2024-07-26T12:10:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18876.557563 secs 2024-07-26T12:10:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18894.295264 secs 2024-07-26T12:10:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:48 smithi163 ceph-mon[36873]: pgmap v10558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:48.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:48 smithi042 ceph-mon[32775]: pgmap v10558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:48 smithi164 ceph-mon[36970]: pgmap v10558: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:50 smithi163 ceph-mon[36873]: pgmap v10559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:50 smithi042 ceph-mon[32775]: pgmap v10559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:50 smithi164 ceph-mon[36970]: pgmap v10559: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18881.557677 secs 2024-07-26T12:10:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:52 smithi163 ceph-mon[36873]: pgmap v10560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18899.295389 secs 2024-07-26T12:10:52.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18881.557677 secs 2024-07-26T12:10:52.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:52 smithi164 ceph-mon[36970]: pgmap v10560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:52.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18899.295389 secs 2024-07-26T12:10:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18881.557677 secs 2024-07-26T12:10:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:52 smithi042 ceph-mon[32775]: pgmap v10560: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18899.295389 secs 2024-07-26T12:10:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:54 smithi163 ceph-mon[36873]: pgmap v10561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:54 smithi042 ceph-mon[32775]: pgmap v10561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:54 smithi164 ceph-mon[36970]: pgmap v10561: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:56 smithi163 ceph-mon[36873]: pgmap v10562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:56 smithi042 ceph-mon[32775]: pgmap v10562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:56 smithi164 ceph-mon[36970]: pgmap v10562: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:10:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18886.557800 secs 2024-07-26T12:10:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18904.295547 secs 2024-07-26T12:10:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18886.557800 secs 2024-07-26T12:10:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18904.295547 secs 2024-07-26T12:10:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18886.557800 secs 2024-07-26T12:10:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18904.295547 secs 2024-07-26T12:10:57.892 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:57.891+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T12:10:57.893 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:57.891+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:10:57.893 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:57.891+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b4fe0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff67802f970 tx=0x7ff67806c070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:10:57.893 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:57.891+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b4fe0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7ff67802f970 tx=0x7ff67806c070 comp rx=0 tx=0).stop 2024-07-26T12:10:57.893 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:57.891+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b4fe0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T12:10:57.893 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:57.891+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 msgr2=0x7ff66c0b4fe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:10:57.893 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:57.891+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b4fe0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:10:57.893 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:57.891+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:10:57.893 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:57.891+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:10:57.894 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:57.892+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff678009a60 tx=0x7ff678055070 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:10:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:10:58 smithi042 ceph-mon[32775]: pgmap v10563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:10:58 smithi164 ceph-mon[36970]: pgmap v10563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:10:58 smithi163 ceph-mon[36873]: pgmap v10563: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:10:59.148 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:59.147+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:10:59.148 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:59.147+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:59.148 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:10:59.147+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:11:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:00 smithi042 ceph-mon[32775]: pgmap v10564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:00 smithi164 ceph-mon[36970]: pgmap v10564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:00 smithi163 ceph-mon[36873]: pgmap v10564: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:00.848 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:11:00.847+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:11:00.848 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:11:00.847+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658004740 con 0x7ff680102000 2024-07-26T12:11:00.849 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:11:00.847+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:11:00.849 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:11:00.847+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6700e9050 con 0x7ff680102000 2024-07-26T12:11:02.559 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18891.557884 secs 2024-07-26T12:11:02.559 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:02 smithi163 ceph-mon[36873]: pgmap v10565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:02.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18909.295670 secs 2024-07-26T12:11:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18891.557884 secs 2024-07-26T12:11:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:02 smithi042 ceph-mon[32775]: pgmap v10565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18909.295670 secs 2024-07-26T12:11:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18891.557884 secs 2024-07-26T12:11:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:02 smithi164 ceph-mon[36970]: pgmap v10565: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18909.295670 secs 2024-07-26T12:11:04.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:04 smithi042 ceph-mon[32775]: pgmap v10566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:04.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:04 smithi164 ceph-mon[36970]: pgmap v10566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:04 smithi163 ceph-mon[36873]: pgmap v10566: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:06 smithi042 ceph-mon[32775]: pgmap v10567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:06 smithi164 ceph-mon[36970]: pgmap v10567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:06 smithi163 ceph-mon[36873]: pgmap v10567: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18896.558028 secs 2024-07-26T12:11:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18914.295814 secs 2024-07-26T12:11:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18896.558028 secs 2024-07-26T12:11:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18914.295814 secs 2024-07-26T12:11:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18896.558028 secs 2024-07-26T12:11:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18914.295814 secs 2024-07-26T12:11:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:08 smithi042 ceph-mon[32775]: pgmap v10568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:08 smithi164 ceph-mon[36970]: pgmap v10568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:08 smithi163 ceph-mon[36873]: pgmap v10568: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:10 smithi042 ceph-mon[32775]: pgmap v10569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:10 smithi164 ceph-mon[36970]: pgmap v10569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:10 smithi163 ceph-mon[36873]: pgmap v10569: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18901.558154 secs 2024-07-26T12:11:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:12 smithi042 ceph-mon[32775]: pgmap v10570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18919.295992 secs 2024-07-26T12:11:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18901.558154 secs 2024-07-26T12:11:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:12 smithi164 ceph-mon[36970]: pgmap v10570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18919.295992 secs 2024-07-26T12:11:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18901.558154 secs 2024-07-26T12:11:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:12 smithi163 ceph-mon[36873]: pgmap v10570: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18919.295992 secs 2024-07-26T12:11:14.149 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:11:14.148+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:11:14.149 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:11:14.148+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:14.149 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:11:14.148+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:11:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:14 smithi042 ceph-mon[32775]: pgmap v10571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:14.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:14 smithi164 ceph-mon[36970]: pgmap v10571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:14 smithi163 ceph-mon[36873]: pgmap v10571: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:16 smithi042 ceph-mon[32775]: pgmap v10572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:16 smithi164 ceph-mon[36970]: pgmap v10572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:16 smithi163 ceph-mon[36873]: pgmap v10572: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18906.558276 secs 2024-07-26T12:11:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18924.296116 secs 2024-07-26T12:11:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18906.558276 secs 2024-07-26T12:11:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18924.296116 secs 2024-07-26T12:11:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18906.558276 secs 2024-07-26T12:11:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18924.296116 secs 2024-07-26T12:11:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:18 smithi042 ceph-mon[32775]: pgmap v10573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:18.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:18 smithi164 ceph-mon[36970]: pgmap v10573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:18 smithi163 ceph-mon[36873]: pgmap v10573: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:20 smithi042 ceph-mon[32775]: pgmap v10574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:20 smithi164 ceph-mon[36970]: pgmap v10574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:20 smithi163 ceph-mon[36873]: pgmap v10574: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18911.558374 secs 2024-07-26T12:11:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:22 smithi042 ceph-mon[32775]: pgmap v10575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18929.296204 secs 2024-07-26T12:11:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18911.558374 secs 2024-07-26T12:11:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:22 smithi164 ceph-mon[36970]: pgmap v10575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18929.296204 secs 2024-07-26T12:11:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18911.558374 secs 2024-07-26T12:11:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:22 smithi163 ceph-mon[36873]: pgmap v10575: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18929.296204 secs 2024-07-26T12:11:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:24 smithi042 ceph-mon[32775]: pgmap v10576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:24.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:24 smithi164 ceph-mon[36970]: pgmap v10576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:24 smithi163 ceph-mon[36873]: pgmap v10576: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:26 smithi042 ceph-mon[32775]: pgmap v10577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:26.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:26 smithi164 ceph-mon[36970]: pgmap v10577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:26 smithi163 ceph-mon[36873]: pgmap v10577: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18916.558499 secs 2024-07-26T12:11:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18934.296321 secs 2024-07-26T12:11:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18916.558499 secs 2024-07-26T12:11:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18934.296321 secs 2024-07-26T12:11:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18916.558499 secs 2024-07-26T12:11:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18934.296321 secs 2024-07-26T12:11:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:28 smithi042 ceph-mon[32775]: pgmap v10578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:28 smithi164 ceph-mon[36970]: pgmap v10578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:28 smithi163 ceph-mon[36873]: pgmap v10578: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:11:29.154+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:11:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:11:29.154+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:29.155 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:11:29.154+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:11:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:29 smithi042 ceph-mon[32775]: pgmap v10579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:11:29.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:29 smithi164 ceph-mon[36970]: pgmap v10579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:11:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:29 smithi163 ceph-mon[36873]: pgmap v10579: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:11:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T12:11:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T12:11:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T12:11:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:11:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T12:11:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T12:11:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:11:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:11:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: pgmap v10580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T12:11:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: pgmap v10581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:11:30.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:11:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T12:11:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T12:11:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T12:11:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:11:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T12:11:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T12:11:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:11:30.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:11:30.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: pgmap v10580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T12:11:30.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: pgmap v10581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:11:30.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:11:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T12:11:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T12:11:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T12:11:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:11:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T12:11:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T12:11:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:11:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:11:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: pgmap v10580: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T12:11:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: pgmap v10581: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:11:30.759 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:11:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18921.558617 secs 2024-07-26T12:11:31.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18921.558617 secs 2024-07-26T12:11:32.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18921.558617 secs 2024-07-26T12:11:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18939.296439 secs 2024-07-26T12:11:32.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:32 smithi042 ceph-mon[32775]: pgmap v10582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:11:32.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18939.296439 secs 2024-07-26T12:11:32.909 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:32 smithi163 ceph-mon[36873]: pgmap v10582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:11:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18939.296439 secs 2024-07-26T12:11:32.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:32 smithi164 ceph-mon[36970]: pgmap v10582: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:11:35.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:34 smithi042 ceph-mon[32775]: pgmap v10583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:11:35.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:34 smithi164 ceph-mon[36970]: pgmap v10583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:11:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:34 smithi163 ceph-mon[36873]: pgmap v10583: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:11:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:36 smithi042 ceph-mon[32775]: pgmap v10584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:11:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18926.558746 secs 2024-07-26T12:11:37.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:36 smithi164 ceph-mon[36970]: pgmap v10584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:11:37.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18926.558746 secs 2024-07-26T12:11:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:36 smithi163 ceph-mon[36873]: pgmap v10584: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:11:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18926.558746 secs 2024-07-26T12:11:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18944.296567 secs 2024-07-26T12:11:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18944.296567 secs 2024-07-26T12:11:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18944.296567 secs 2024-07-26T12:11:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:38 smithi042 ceph-mon[32775]: pgmap v10585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:11:39.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:38 smithi164 ceph-mon[36970]: pgmap v10585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:11:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:38 smithi163 ceph-mon[36873]: pgmap v10585: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:11:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:40 smithi042 ceph-mon[32775]: pgmap v10586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:11:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:40 smithi164 ceph-mon[36970]: pgmap v10586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:11:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:40 smithi163 ceph-mon[36873]: pgmap v10586: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:11:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18931.558902 secs 2024-07-26T12:11:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18931.558902 secs 2024-07-26T12:11:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18931.558902 secs 2024-07-26T12:11:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18949.296686 secs 2024-07-26T12:11:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:42 smithi042 ceph-mon[32775]: pgmap v10587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18949.296686 secs 2024-07-26T12:11:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:42 smithi164 ceph-mon[36970]: pgmap v10587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18949.296686 secs 2024-07-26T12:11:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:42 smithi163 ceph-mon[36873]: pgmap v10587: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:11:44.155+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:11:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:11:44.155+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:44.156 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:11:44.155+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:11:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:44 smithi042 ceph-mon[32775]: pgmap v10588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:11:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:44 smithi164 ceph-mon[36970]: pgmap v10588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:11:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:44 smithi163 ceph-mon[36873]: pgmap v10588: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:11:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:46 smithi042 ceph-mon[32775]: pgmap v10589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:47.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18936.559055 secs 2024-07-26T12:11:47.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:46 smithi164 ceph-mon[36970]: pgmap v10589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:47.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18936.559055 secs 2024-07-26T12:11:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:46 smithi163 ceph-mon[36873]: pgmap v10589: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18936.559055 secs 2024-07-26T12:11:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18954.296815 secs 2024-07-26T12:11:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18954.296815 secs 2024-07-26T12:11:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18954.296815 secs 2024-07-26T12:11:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:48 smithi042 ceph-mon[32775]: pgmap v10590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:48 smithi164 ceph-mon[36970]: pgmap v10590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:48 smithi163 ceph-mon[36873]: pgmap v10590: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:50 smithi042 ceph-mon[32775]: pgmap v10591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:50 smithi164 ceph-mon[36970]: pgmap v10591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:50 smithi163 ceph-mon[36873]: pgmap v10591: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:52.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18941.559171 secs 2024-07-26T12:11:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18941.559171 secs 2024-07-26T12:11:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18941.559171 secs 2024-07-26T12:11:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18959.296957 secs 2024-07-26T12:11:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:52 smithi042 ceph-mon[32775]: pgmap v10592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18959.296957 secs 2024-07-26T12:11:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:52 smithi164 ceph-mon[36970]: pgmap v10592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18959.296957 secs 2024-07-26T12:11:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:52 smithi163 ceph-mon[36873]: pgmap v10592: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:54 smithi042 ceph-mon[32775]: pgmap v10593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:11:55.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:54 smithi164 ceph-mon[36970]: pgmap v10593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:11:55.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:54 smithi163 ceph-mon[36873]: pgmap v10593: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:11:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:56 smithi042 ceph-mon[32775]: pgmap v10594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18946.559316 secs 2024-07-26T12:11:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:56 smithi164 ceph-mon[36970]: pgmap v10594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18946.559316 secs 2024-07-26T12:11:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:56 smithi163 ceph-mon[36873]: pgmap v10594: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:11:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18946.559316 secs 2024-07-26T12:11:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18964.296983 secs 2024-07-26T12:11:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18964.296983 secs 2024-07-26T12:11:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18964.296983 secs 2024-07-26T12:11:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:11:58 smithi042 ceph-mon[32775]: pgmap v10595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:11:59.161+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:11:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:11:59.161+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:11:59.161+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:11:59.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:11:58 smithi164 ceph-mon[36970]: pgmap v10595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:11:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:11:58 smithi163 ceph-mon[36873]: pgmap v10595: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:00 smithi042 ceph-mon[32775]: pgmap v10596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:01.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:00 smithi164 ceph-mon[36970]: pgmap v10596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:01.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:00 smithi163 ceph-mon[36873]: pgmap v10596: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18951.559383 secs 2024-07-26T12:12:02.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18951.559383 secs 2024-07-26T12:12:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18951.559383 secs 2024-07-26T12:12:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18969.297062 secs 2024-07-26T12:12:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:02 smithi042 ceph-mon[32775]: pgmap v10597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18969.297062 secs 2024-07-26T12:12:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:02 smithi164 ceph-mon[36970]: pgmap v10597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18969.297062 secs 2024-07-26T12:12:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:02 smithi163 ceph-mon[36873]: pgmap v10597: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:04 smithi042 ceph-mon[32775]: pgmap v10598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:12:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:04 smithi164 ceph-mon[36970]: pgmap v10598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:12:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:04 smithi163 ceph-mon[36873]: pgmap v10598: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:12:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:06 smithi042 ceph-mon[32775]: pgmap v10599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18956.559509 secs 2024-07-26T12:12:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:06 smithi164 ceph-mon[36970]: pgmap v10599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18956.559509 secs 2024-07-26T12:12:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:06 smithi163 ceph-mon[36873]: pgmap v10599: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:07.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18956.559509 secs 2024-07-26T12:12:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18974.297184 secs 2024-07-26T12:12:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18974.297184 secs 2024-07-26T12:12:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18974.297184 secs 2024-07-26T12:12:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:08 smithi042 ceph-mon[32775]: pgmap v10600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:08 smithi164 ceph-mon[36970]: pgmap v10600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:08 smithi163 ceph-mon[36873]: pgmap v10600: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:10 smithi042 ceph-mon[32775]: pgmap v10601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:11.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:10 smithi164 ceph-mon[36970]: pgmap v10601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:10 smithi163 ceph-mon[36873]: pgmap v10601: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18961.559603 secs 2024-07-26T12:12:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18961.559603 secs 2024-07-26T12:12:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18961.559603 secs 2024-07-26T12:12:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18979.297266 secs 2024-07-26T12:12:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:12 smithi042 ceph-mon[32775]: pgmap v10602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18979.297266 secs 2024-07-26T12:12:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:12 smithi164 ceph-mon[36970]: pgmap v10602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18979.297266 secs 2024-07-26T12:12:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:12 smithi163 ceph-mon[36873]: pgmap v10602: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:14.163 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:12:14.162+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:12:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:12:14.162+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:12:14.162+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:12:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:14 smithi042 ceph-mon[32775]: pgmap v10603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:12:15.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:14 smithi164 ceph-mon[36970]: pgmap v10603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:12:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:14 smithi163 ceph-mon[36873]: pgmap v10603: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:12:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:16 smithi042 ceph-mon[32775]: pgmap v10604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:17.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18966.559669 secs 2024-07-26T12:12:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:16 smithi164 ceph-mon[36970]: pgmap v10604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18966.559669 secs 2024-07-26T12:12:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:16 smithi163 ceph-mon[36873]: pgmap v10604: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18966.559669 secs 2024-07-26T12:12:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18984.297413 secs 2024-07-26T12:12:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18984.297413 secs 2024-07-26T12:12:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18984.297413 secs 2024-07-26T12:12:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:18 smithi042 ceph-mon[32775]: pgmap v10605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:18 smithi164 ceph-mon[36970]: pgmap v10605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:18 smithi163 ceph-mon[36873]: pgmap v10605: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:20 smithi164 ceph-mon[36970]: pgmap v10606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:20 smithi163 ceph-mon[36873]: pgmap v10606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:20 smithi042 ceph-mon[32775]: pgmap v10606: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:22.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18971.559797 secs 2024-07-26T12:12:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18971.559797 secs 2024-07-26T12:12:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18971.559797 secs 2024-07-26T12:12:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18989.297532 secs 2024-07-26T12:12:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:22 smithi164 ceph-mon[36970]: pgmap v10607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18989.297532 secs 2024-07-26T12:12:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:22 smithi163 ceph-mon[36873]: pgmap v10607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18989.297532 secs 2024-07-26T12:12:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:22 smithi042 ceph-mon[32775]: pgmap v10607: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:24 smithi164 ceph-mon[36970]: pgmap v10608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:12:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:24 smithi163 ceph-mon[36873]: pgmap v10608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:12:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:24 smithi042 ceph-mon[32775]: pgmap v10608: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:12:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:26 smithi164 ceph-mon[36970]: pgmap v10609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18976.559901 secs 2024-07-26T12:12:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:26 smithi163 ceph-mon[36873]: pgmap v10609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18976.559901 secs 2024-07-26T12:12:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:26 smithi042 ceph-mon[32775]: pgmap v10609: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18976.559901 secs 2024-07-26T12:12:28.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18994.297654 secs 2024-07-26T12:12:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18994.297654 secs 2024-07-26T12:12:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18994.297654 secs 2024-07-26T12:12:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:12:29.168+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:12:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:12:29.168+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:12:29.169 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:12:29.168+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:28 smithi164 ceph-mon[36970]: pgmap v10610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:28 smithi163 ceph-mon[36873]: pgmap v10610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:28 smithi042 ceph-mon[32775]: pgmap v10610: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:30.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:29 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:12:30.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:29 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:12:30.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:29 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:12:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:30 smithi164 ceph-mon[36970]: pgmap v10611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:12:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:12:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:30 smithi164 ceph-mon[36970]: pgmap v10612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:12:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:30 smithi164 ceph-mon[36970]: pgmap v10613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:12:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:12:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:30 smithi163 ceph-mon[36873]: pgmap v10611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:12:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:12:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:30 smithi163 ceph-mon[36873]: pgmap v10612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:12:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:30 smithi163 ceph-mon[36873]: pgmap v10613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:12:31.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:12:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:30 smithi042 ceph-mon[32775]: pgmap v10611: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:12:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:12:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:30 smithi042 ceph-mon[32775]: pgmap v10612: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:12:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:30 smithi042 ceph-mon[32775]: pgmap v10613: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:12:31.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:12:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18981.560024 secs 2024-07-26T12:12:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18981.560024 secs 2024-07-26T12:12:32.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18981.560024 secs 2024-07-26T12:12:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18999.297754 secs 2024-07-26T12:12:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:32 smithi164 ceph-mon[36970]: pgmap v10614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:12:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18999.297754 secs 2024-07-26T12:12:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:32 smithi163 ceph-mon[36873]: pgmap v10614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:12:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18999.297754 secs 2024-07-26T12:12:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:32 smithi042 ceph-mon[32775]: pgmap v10614: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:12:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:34 smithi042 ceph-mon[32775]: pgmap v10615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:12:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:34 smithi164 ceph-mon[36970]: pgmap v10615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:12:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:34 smithi163 ceph-mon[36873]: pgmap v10615: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:12:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:36 smithi042 ceph-mon[32775]: pgmap v10616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:12:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:36 smithi164 ceph-mon[36970]: pgmap v10616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:12:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:36 smithi163 ceph-mon[36873]: pgmap v10616: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:12:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18986.560183 secs 2024-07-26T12:12:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19004.297879 secs 2024-07-26T12:12:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18986.560183 secs 2024-07-26T12:12:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19004.297879 secs 2024-07-26T12:12:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18986.560183 secs 2024-07-26T12:12:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19004.297879 secs 2024-07-26T12:12:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:38 smithi042 ceph-mon[32775]: pgmap v10617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:12:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:38 smithi164 ceph-mon[36970]: pgmap v10617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:12:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:38 smithi163 ceph-mon[36873]: pgmap v10617: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:12:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:40 smithi042 ceph-mon[32775]: pgmap v10618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:12:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:40 smithi164 ceph-mon[36970]: pgmap v10618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:12:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:40 smithi163 ceph-mon[36873]: pgmap v10618: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:12:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18991.560310 secs 2024-07-26T12:12:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18991.560310 secs 2024-07-26T12:12:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18991.560310 secs 2024-07-26T12:12:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19009.297997 secs 2024-07-26T12:12:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:42 smithi042 ceph-mon[32775]: pgmap v10619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19009.297997 secs 2024-07-26T12:12:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:42 smithi164 ceph-mon[36970]: pgmap v10619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19009.297997 secs 2024-07-26T12:12:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:42 smithi163 ceph-mon[36873]: pgmap v10619: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:12:44.169+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:12:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:12:44.169+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:12:44.169+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:12:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:44 smithi042 ceph-mon[32775]: pgmap v10620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:44 smithi164 ceph-mon[36970]: pgmap v10620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:44 smithi163 ceph-mon[36873]: pgmap v10620: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:46 smithi042 ceph-mon[32775]: pgmap v10621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:46 smithi164 ceph-mon[36970]: pgmap v10621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:46 smithi163 ceph-mon[36873]: pgmap v10621: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 18996.560403 secs 2024-07-26T12:12:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19014.298123 secs 2024-07-26T12:12:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 18996.560403 secs 2024-07-26T12:12:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19014.298123 secs 2024-07-26T12:12:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 18996.560403 secs 2024-07-26T12:12:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19014.298123 secs 2024-07-26T12:12:48.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:48 smithi042 ceph-mon[32775]: pgmap v10622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:12:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:48 smithi164 ceph-mon[36970]: pgmap v10622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:12:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:48 smithi163 ceph-mon[36873]: pgmap v10622: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:12:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:50 smithi042 ceph-mon[32775]: pgmap v10623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:50 smithi164 ceph-mon[36970]: pgmap v10623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:50 smithi163 ceph-mon[36873]: pgmap v10623: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:51.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19001.560526 secs 2024-07-26T12:12:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19001.560526 secs 2024-07-26T12:12:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19001.560526 secs 2024-07-26T12:12:52.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19019.298244 secs 2024-07-26T12:12:52.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:52 smithi164 ceph-mon[36970]: pgmap v10624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19019.298244 secs 2024-07-26T12:12:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:52 smithi042 ceph-mon[32775]: pgmap v10624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19019.298244 secs 2024-07-26T12:12:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:52 smithi163 ceph-mon[36873]: pgmap v10624: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:54 smithi042 ceph-mon[32775]: pgmap v10625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:54 smithi164 ceph-mon[36970]: pgmap v10625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:54 smithi163 ceph-mon[36873]: pgmap v10625: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:12:56.447 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:56 smithi042 ceph-mon[32775]: pgmap v10626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:56 smithi164 ceph-mon[36970]: pgmap v10626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:56 smithi163 ceph-mon[36873]: pgmap v10626: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:12:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19006.560628 secs 2024-07-26T12:12:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19024.298367 secs 2024-07-26T12:12:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19006.560628 secs 2024-07-26T12:12:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19024.298367 secs 2024-07-26T12:12:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19006.560628 secs 2024-07-26T12:12:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19024.298367 secs 2024-07-26T12:12:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:12:58 smithi164 ceph-mon[36970]: pgmap v10627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:12:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:12:58 smithi163 ceph-mon[36873]: pgmap v10627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:12:58.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:12:58 smithi042 ceph-mon[32775]: pgmap v10627: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:12:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:12:59.175+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:12:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:12:59.175+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:59.176 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:12:59.175+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:13:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:00 smithi042 ceph-mon[32775]: pgmap v10628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:00 smithi164 ceph-mon[36970]: pgmap v10628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:00 smithi163 ceph-mon[36873]: pgmap v10628: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19011.560751 secs 2024-07-26T12:13:01.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19011.560751 secs 2024-07-26T12:13:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19011.560751 secs 2024-07-26T12:13:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19029.298486 secs 2024-07-26T12:13:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:02 smithi042 ceph-mon[32775]: pgmap v10629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:02.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19029.298486 secs 2024-07-26T12:13:02.908 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:02 smithi163 ceph-mon[36873]: pgmap v10629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19029.298486 secs 2024-07-26T12:13:02.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:02 smithi164 ceph-mon[36970]: pgmap v10629: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:04.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:04 smithi042 ceph-mon[32775]: pgmap v10630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:04.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:04 smithi164 ceph-mon[36970]: pgmap v10630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:04 smithi163 ceph-mon[36873]: pgmap v10630: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:06 smithi042 ceph-mon[32775]: pgmap v10631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:06.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:06 smithi164 ceph-mon[36970]: pgmap v10631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:06 smithi163 ceph-mon[36873]: pgmap v10631: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19016.560876 secs 2024-07-26T12:13:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19034.298618 secs 2024-07-26T12:13:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19016.560876 secs 2024-07-26T12:13:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19034.298618 secs 2024-07-26T12:13:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19016.560876 secs 2024-07-26T12:13:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19034.298618 secs 2024-07-26T12:13:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:08 smithi164 ceph-mon[36970]: pgmap v10632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:13:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:08 smithi163 ceph-mon[36873]: pgmap v10632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:13:08.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:08 smithi042 ceph-mon[32775]: pgmap v10632: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:13:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:10 smithi042 ceph-mon[32775]: pgmap v10633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:10 smithi164 ceph-mon[36970]: pgmap v10633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:10 smithi163 ceph-mon[36873]: pgmap v10633: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:11.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19021.561007 secs 2024-07-26T12:13:11.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19021.561007 secs 2024-07-26T12:13:12.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19021.561007 secs 2024-07-26T12:13:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19039.298738 secs 2024-07-26T12:13:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:12 smithi042 ceph-mon[32775]: pgmap v10634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19039.298738 secs 2024-07-26T12:13:12.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:12 smithi164 ceph-mon[36970]: pgmap v10634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19039.298738 secs 2024-07-26T12:13:13.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:12 smithi163 ceph-mon[36873]: pgmap v10634: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:14.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:13:14.176+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:13:14.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:13:14.176+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:14.177 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:13:14.176+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:13:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:14 smithi042 ceph-mon[32775]: pgmap v10635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:14 smithi164 ceph-mon[36970]: pgmap v10635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:14 smithi163 ceph-mon[36873]: pgmap v10635: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:16 smithi042 ceph-mon[32775]: pgmap v10636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:16 smithi164 ceph-mon[36970]: pgmap v10636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:16 smithi163 ceph-mon[36873]: pgmap v10636: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19026.561137 secs 2024-07-26T12:13:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19044.298846 secs 2024-07-26T12:13:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19026.561137 secs 2024-07-26T12:13:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19044.298846 secs 2024-07-26T12:13:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19026.561137 secs 2024-07-26T12:13:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19044.298846 secs 2024-07-26T12:13:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:18 smithi164 ceph-mon[36970]: pgmap v10637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:13:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:18 smithi163 ceph-mon[36873]: pgmap v10637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:13:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:18 smithi042 ceph-mon[32775]: pgmap v10637: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:13:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:20 smithi042 ceph-mon[32775]: pgmap v10638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:20 smithi164 ceph-mon[36970]: pgmap v10638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:20 smithi163 ceph-mon[36873]: pgmap v10638: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19031.561276 secs 2024-07-26T12:13:21.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19031.561276 secs 2024-07-26T12:13:22.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19031.561276 secs 2024-07-26T12:13:22.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19049.298967 secs 2024-07-26T12:13:22.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:22 smithi164 ceph-mon[36970]: pgmap v10639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19049.298967 secs 2024-07-26T12:13:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:22 smithi042 ceph-mon[32775]: pgmap v10639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:23.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19049.298967 secs 2024-07-26T12:13:23.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:22 smithi163 ceph-mon[36873]: pgmap v10639: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:24 smithi042 ceph-mon[32775]: pgmap v10640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:24.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:24 smithi164 ceph-mon[36970]: pgmap v10640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:24 smithi163 ceph-mon[36873]: pgmap v10640: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:26 smithi042 ceph-mon[32775]: pgmap v10641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:26 smithi164 ceph-mon[36970]: pgmap v10641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:26 smithi163 ceph-mon[36873]: pgmap v10641: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19036.561382 secs 2024-07-26T12:13:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19054.299065 secs 2024-07-26T12:13:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19036.561382 secs 2024-07-26T12:13:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19054.299065 secs 2024-07-26T12:13:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19036.561382 secs 2024-07-26T12:13:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19054.299065 secs 2024-07-26T12:13:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:28 smithi164 ceph-mon[36970]: pgmap v10642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:13:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:28 smithi163 ceph-mon[36873]: pgmap v10642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:13:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:28 smithi042 ceph-mon[32775]: pgmap v10642: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:13:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:13:29.182+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:13:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:13:29.182+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:29.183 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:13:29.182+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:13:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:30 smithi042 ceph-mon[32775]: pgmap v10643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:30 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:13:30.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:30 smithi164 ceph-mon[36970]: pgmap v10643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:30 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:13:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:30 smithi163 ceph-mon[36873]: pgmap v10643: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:30 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:13:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:13:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:13:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:31 smithi164 ceph-mon[36970]: pgmap v10644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:13:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:31 smithi164 ceph-mon[36970]: pgmap v10645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:13:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:13:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:13:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:13:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:31 smithi163 ceph-mon[36873]: pgmap v10644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:13:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:31 smithi163 ceph-mon[36873]: pgmap v10645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:13:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:13:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:13:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:13:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:31 smithi042 ceph-mon[32775]: pgmap v10644: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:13:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:31 smithi042 ceph-mon[32775]: pgmap v10645: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:13:31.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:13:32.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19041.561495 secs 2024-07-26T12:13:32.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19059.299185 secs 2024-07-26T12:13:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19041.561495 secs 2024-07-26T12:13:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19059.299185 secs 2024-07-26T12:13:32.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19041.561495 secs 2024-07-26T12:13:32.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19059.299185 secs 2024-07-26T12:13:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:33 smithi164 ceph-mon[36970]: pgmap v10646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:13:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:33 smithi163 ceph-mon[36873]: pgmap v10646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:13:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:33 smithi042 ceph-mon[32775]: pgmap v10646: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:13:36.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:35 smithi042 ceph-mon[32775]: pgmap v10647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:13:36.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:35 smithi164 ceph-mon[36970]: pgmap v10647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:13:36.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:35 smithi163 ceph-mon[36873]: pgmap v10647: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:13:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19046.561617 secs 2024-07-26T12:13:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19046.561617 secs 2024-07-26T12:13:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19046.561617 secs 2024-07-26T12:13:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19064.299307 secs 2024-07-26T12:13:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:37 smithi164 ceph-mon[36970]: pgmap v10648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:13:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19064.299307 secs 2024-07-26T12:13:38.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:37 smithi163 ceph-mon[36873]: pgmap v10648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:13:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19064.299307 secs 2024-07-26T12:13:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:37 smithi042 ceph-mon[32775]: pgmap v10648: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:13:40.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:39 smithi164 ceph-mon[36970]: pgmap v10649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:13:40.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:39 smithi163 ceph-mon[36873]: pgmap v10649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:13:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:39 smithi042 ceph-mon[32775]: pgmap v10649: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:13:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:41 smithi164 ceph-mon[36970]: pgmap v10650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:13:42.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19051.561737 secs 2024-07-26T12:13:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:41 smithi163 ceph-mon[36873]: pgmap v10650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:13:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19051.561737 secs 2024-07-26T12:13:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:41 smithi042 ceph-mon[32775]: pgmap v10650: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:13:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19051.561737 secs 2024-07-26T12:13:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19069.299443 secs 2024-07-26T12:13:43.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19069.299443 secs 2024-07-26T12:13:43.280 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19069.299443 secs 2024-07-26T12:13:44.184 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:13:44.183+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:13:44.184 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:13:44.183+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:44.184 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:13:44.183+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:13:44.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:43 smithi164 ceph-mon[36970]: pgmap v10651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:44.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:43 smithi163 ceph-mon[36873]: pgmap v10651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:43 smithi042 ceph-mon[32775]: pgmap v10651: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:45 smithi164 ceph-mon[36970]: pgmap v10652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:46.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:45 smithi163 ceph-mon[36873]: pgmap v10652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:45 smithi042 ceph-mon[32775]: pgmap v10652: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19056.561861 secs 2024-07-26T12:13:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19056.561861 secs 2024-07-26T12:13:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19056.561861 secs 2024-07-26T12:13:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19074.299566 secs 2024-07-26T12:13:48.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:47 smithi164 ceph-mon[36970]: pgmap v10653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:48.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19074.299566 secs 2024-07-26T12:13:48.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:47 smithi163 ceph-mon[36873]: pgmap v10653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19074.299566 secs 2024-07-26T12:13:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:47 smithi042 ceph-mon[32775]: pgmap v10653: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:50.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:49 smithi164 ceph-mon[36970]: pgmap v10654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:13:50.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:49 smithi163 ceph-mon[36873]: pgmap v10654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:13:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:49 smithi042 ceph-mon[32775]: pgmap v10654: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:13:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:51 smithi164 ceph-mon[36970]: pgmap v10655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:52.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19061.561990 secs 2024-07-26T12:13:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:51 smithi163 ceph-mon[36873]: pgmap v10655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19061.561990 secs 2024-07-26T12:13:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:51 smithi042 ceph-mon[32775]: pgmap v10655: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19061.561990 secs 2024-07-26T12:13:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19079.299702 secs 2024-07-26T12:13:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19079.299702 secs 2024-07-26T12:13:53.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19079.299702 secs 2024-07-26T12:13:54.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:53 smithi164 ceph-mon[36970]: pgmap v10656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:54.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:53 smithi163 ceph-mon[36873]: pgmap v10656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:53 smithi042 ceph-mon[32775]: pgmap v10656: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:56.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:55 smithi164 ceph-mon[36970]: pgmap v10657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:56.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:55 smithi163 ceph-mon[36873]: pgmap v10657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:55 smithi042 ceph-mon[32775]: pgmap v10657: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:13:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19066.562121 secs 2024-07-26T12:13:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19066.562121 secs 2024-07-26T12:13:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19066.562121 secs 2024-07-26T12:13:58.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19084.299853 secs 2024-07-26T12:13:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:57 smithi164 ceph-mon[36970]: pgmap v10658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19084.299853 secs 2024-07-26T12:13:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:57 smithi163 ceph-mon[36873]: pgmap v10658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19084.299853 secs 2024-07-26T12:13:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:57 smithi042 ceph-mon[32775]: pgmap v10658: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:13:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:13:59.189+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:13:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:13:59.189+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:59.190 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:13:59.189+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:14:00.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:13:59 smithi164 ceph-mon[36970]: pgmap v10659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:14:00.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:13:59 smithi163 ceph-mon[36873]: pgmap v10659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:14:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:13:59 smithi042 ceph-mon[32775]: pgmap v10659: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:14:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:14:00.876+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658003690 con 0x7ff680102000 2024-07-26T12:14:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:14:00.876+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:14:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:14:00.876+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6700e9050 con 0x7ff680102000 2024-07-26T12:14:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:14:00.876+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:14:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:01 smithi164 ceph-mon[36970]: pgmap v10660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19071.562265 secs 2024-07-26T12:14:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:01 smithi163 ceph-mon[36873]: pgmap v10660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:02.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19071.562265 secs 2024-07-26T12:14:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:01 smithi042 ceph-mon[32775]: pgmap v10660: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19071.562265 secs 2024-07-26T12:14:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19089.299957 secs 2024-07-26T12:14:03.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19089.299957 secs 2024-07-26T12:14:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19089.299957 secs 2024-07-26T12:14:04.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:03 smithi164 ceph-mon[36970]: pgmap v10661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:04.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:03 smithi163 ceph-mon[36873]: pgmap v10661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:03 smithi042 ceph-mon[32775]: pgmap v10661: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:06.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:05 smithi164 ceph-mon[36970]: pgmap v10662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:06.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:05 smithi163 ceph-mon[36873]: pgmap v10662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:05 smithi042 ceph-mon[32775]: pgmap v10662: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19076.562382 secs 2024-07-26T12:14:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19076.562382 secs 2024-07-26T12:14:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19076.562382 secs 2024-07-26T12:14:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19094.300081 secs 2024-07-26T12:14:08.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:07 smithi163 ceph-mon[36873]: pgmap v10663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19094.300081 secs 2024-07-26T12:14:08.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:07 smithi042 ceph-mon[32775]: pgmap v10663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19094.300081 secs 2024-07-26T12:14:08.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:07 smithi164 ceph-mon[36970]: pgmap v10663: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:10.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:09 smithi163 ceph-mon[36873]: pgmap v10664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:14:10.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:09 smithi042 ceph-mon[32775]: pgmap v10664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:14:10.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:09 smithi164 ceph-mon[36970]: pgmap v10664: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:14:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:11 smithi163 ceph-mon[36873]: pgmap v10665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19081.562507 secs 2024-07-26T12:14:12.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19099.300203 secs 2024-07-26T12:14:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:11 smithi042 ceph-mon[32775]: pgmap v10665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19081.562507 secs 2024-07-26T12:14:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19099.300203 secs 2024-07-26T12:14:12.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:11 smithi164 ceph-mon[36970]: pgmap v10665: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19081.562507 secs 2024-07-26T12:14:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19099.300203 secs 2024-07-26T12:14:14.191 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:14:14.190+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:14:14.191 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:14:14.190+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:14:14.191 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:14:14.190+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:14.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:13 smithi163 ceph-mon[36873]: pgmap v10666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:14.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:13 smithi042 ceph-mon[32775]: pgmap v10666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:13 smithi164 ceph-mon[36970]: pgmap v10666: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:16.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:15 smithi163 ceph-mon[36873]: pgmap v10667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:16.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:15 smithi042 ceph-mon[32775]: pgmap v10667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:15 smithi164 ceph-mon[36970]: pgmap v10667: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19086.562634 secs 2024-07-26T12:14:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19104.300312 secs 2024-07-26T12:14:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19086.562634 secs 2024-07-26T12:14:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19104.300312 secs 2024-07-26T12:14:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19086.562634 secs 2024-07-26T12:14:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19104.300312 secs 2024-07-26T12:14:18.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:18 smithi042 ceph-mon[32775]: pgmap v10668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:18 smithi164 ceph-mon[36970]: pgmap v10668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:18 smithi163 ceph-mon[36873]: pgmap v10668: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:20.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:20 smithi042 ceph-mon[32775]: pgmap v10669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:14:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:20 smithi164 ceph-mon[36970]: pgmap v10669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:14:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:20 smithi163 ceph-mon[36873]: pgmap v10669: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:14:22.275 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:22 smithi164 ceph-mon[36970]: pgmap v10670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:22.275 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19091.562757 secs 2024-07-26T12:14:22.275 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19109.300436 secs 2024-07-26T12:14:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:22 smithi042 ceph-mon[32775]: pgmap v10670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19091.562757 secs 2024-07-26T12:14:22.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19109.300436 secs 2024-07-26T12:14:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:22 smithi163 ceph-mon[36873]: pgmap v10670: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19091.562757 secs 2024-07-26T12:14:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19109.300436 secs 2024-07-26T12:14:24.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:24 smithi042 ceph-mon[32775]: pgmap v10671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:24 smithi164 ceph-mon[36970]: pgmap v10671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:24 smithi163 ceph-mon[36873]: pgmap v10671: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:26.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:26 smithi042 ceph-mon[32775]: pgmap v10672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:26 smithi164 ceph-mon[36970]: pgmap v10672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:26 smithi163 ceph-mon[36873]: pgmap v10672: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19096.562891 secs 2024-07-26T12:14:27.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19114.300559 secs 2024-07-26T12:14:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19096.562891 secs 2024-07-26T12:14:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19114.300559 secs 2024-07-26T12:14:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19096.562891 secs 2024-07-26T12:14:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19114.300559 secs 2024-07-26T12:14:28.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:28 smithi042 ceph-mon[32775]: pgmap v10673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:28 smithi164 ceph-mon[36970]: pgmap v10673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:28 smithi163 ceph-mon[36873]: pgmap v10673: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:14:29.196+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:14:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:14:29.196+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:14:29.196+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:14:30.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:30 smithi042 ceph-mon[32775]: pgmap v10674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:14:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:30 smithi164 ceph-mon[36970]: pgmap v10674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:14:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:30 smithi163 ceph-mon[36873]: pgmap v10674: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:14:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:31 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:14:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:31 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:14:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:31 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:14:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:32 smithi042 ceph-mon[32775]: pgmap v10675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19101.562985 secs 2024-07-26T12:14:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19119.300659 secs 2024-07-26T12:14:32.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:32 smithi164 ceph-mon[36970]: pgmap v10675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19101.562985 secs 2024-07-26T12:14:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19119.300659 secs 2024-07-26T12:14:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:32 smithi163 ceph-mon[36873]: pgmap v10675: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19101.562985 secs 2024-07-26T12:14:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19119.300659 secs 2024-07-26T12:14:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:33 smithi042 ceph-mon[32775]: pgmap v10676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:33.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:33 smithi164 ceph-mon[36970]: pgmap v10676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:33.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:34.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:33 smithi163 ceph-mon[36873]: pgmap v10676: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:34.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:14:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:14:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:34 smithi042 ceph-mon[32775]: pgmap v10677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T12:14:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:34 smithi042 ceph-mon[32775]: pgmap v10678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T12:14:34.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:34 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:14:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:14:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:34 smithi164 ceph-mon[36970]: pgmap v10677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T12:14:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:34 smithi164 ceph-mon[36970]: pgmap v10678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T12:14:34.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:34 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:35.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:14:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:14:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:34 smithi163 ceph-mon[36873]: pgmap v10677: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-07-26T12:14:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:34 smithi163 ceph-mon[36873]: pgmap v10678: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T12:14:35.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:34 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:14:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:36 smithi164 ceph-mon[36970]: pgmap v10679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:14:36.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19106.563126 secs 2024-07-26T12:14:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:36 smithi163 ceph-mon[36873]: pgmap v10679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:14:37.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19106.563126 secs 2024-07-26T12:14:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:36 smithi042 ceph-mon[32775]: pgmap v10679: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:14:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19106.563126 secs 2024-07-26T12:14:37.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19124.300787 secs 2024-07-26T12:14:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19124.300787 secs 2024-07-26T12:14:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19124.300787 secs 2024-07-26T12:14:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:38 smithi164 ceph-mon[36970]: pgmap v10680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:14:39.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:38 smithi163 ceph-mon[36873]: pgmap v10680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:14:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:38 smithi042 ceph-mon[32775]: pgmap v10680: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:14:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:40 smithi042 ceph-mon[32775]: pgmap v10681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:14:40.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:40 smithi164 ceph-mon[36970]: pgmap v10681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:14:41.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:40 smithi163 ceph-mon[36873]: pgmap v10681: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:14:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19111.563245 secs 2024-07-26T12:14:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19111.563245 secs 2024-07-26T12:14:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19111.563245 secs 2024-07-26T12:14:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19129.300912 secs 2024-07-26T12:14:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:42 smithi042 ceph-mon[32775]: pgmap v10682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:14:42.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19129.300912 secs 2024-07-26T12:14:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:42 smithi164 ceph-mon[36970]: pgmap v10682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:14:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19129.300912 secs 2024-07-26T12:14:43.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:42 smithi163 ceph-mon[36873]: pgmap v10682: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:14:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:14:44.197+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:14:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:14:44.197+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:14:44.197+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:14:44.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:44 smithi164 ceph-mon[36970]: pgmap v10683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:14:45.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:44 smithi163 ceph-mon[36873]: pgmap v10683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:14:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:44 smithi042 ceph-mon[32775]: pgmap v10683: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:14:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:46 smithi164 ceph-mon[36970]: pgmap v10684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:46.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19116.563369 secs 2024-07-26T12:14:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:46 smithi163 ceph-mon[36873]: pgmap v10684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:47.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19116.563369 secs 2024-07-26T12:14:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:46 smithi042 ceph-mon[32775]: pgmap v10684: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19116.563369 secs 2024-07-26T12:14:47.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19134.301012 secs 2024-07-26T12:14:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19134.301012 secs 2024-07-26T12:14:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19134.301012 secs 2024-07-26T12:14:49.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:48 smithi163 ceph-mon[36873]: pgmap v10685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:48 smithi042 ceph-mon[32775]: pgmap v10685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:49.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:48 smithi164 ceph-mon[36970]: pgmap v10685: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:51.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:50 smithi163 ceph-mon[36873]: pgmap v10686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:50 smithi042 ceph-mon[32775]: pgmap v10686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:50 smithi164 ceph-mon[36970]: pgmap v10686: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19121.563460 secs 2024-07-26T12:14:52.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19121.563460 secs 2024-07-26T12:14:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19121.563460 secs 2024-07-26T12:14:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19139.301141 secs 2024-07-26T12:14:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:52 smithi163 ceph-mon[36873]: pgmap v10687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:53.138 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19139.301141 secs 2024-07-26T12:14:53.138 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:52 smithi164 ceph-mon[36970]: pgmap v10687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19139.301141 secs 2024-07-26T12:14:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:52 smithi042 ceph-mon[32775]: pgmap v10687: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:55.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:54 smithi163 ceph-mon[36873]: pgmap v10688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:14:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:54 smithi042 ceph-mon[32775]: pgmap v10688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:14:55.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:54 smithi164 ceph-mon[36970]: pgmap v10688: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:14:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:56 smithi163 ceph-mon[36873]: pgmap v10689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:57.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19126.563549 secs 2024-07-26T12:14:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:56 smithi042 ceph-mon[32775]: pgmap v10689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19126.563549 secs 2024-07-26T12:14:57.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:56 smithi164 ceph-mon[36970]: pgmap v10689: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:14:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19126.563549 secs 2024-07-26T12:14:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19144.301266 secs 2024-07-26T12:14:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19144.301266 secs 2024-07-26T12:14:58.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19144.301266 secs 2024-07-26T12:14:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:14:58 smithi163 ceph-mon[36873]: pgmap v10690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:14:58 smithi042 ceph-mon[32775]: pgmap v10690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:14:59.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:14:59.203+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:14:59.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:14:59.203+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:59.204 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:14:59.203+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:14:59.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:14:58 smithi164 ceph-mon[36970]: pgmap v10690: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:01.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:00 smithi163 ceph-mon[36873]: pgmap v10691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:01.104 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:00 smithi042 ceph-mon[32775]: pgmap v10691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:00 smithi164 ceph-mon[36970]: pgmap v10691: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:02.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19131.563688 secs 2024-07-26T12:15:02.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19131.563688 secs 2024-07-26T12:15:02.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19131.563688 secs 2024-07-26T12:15:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19149.301389 secs 2024-07-26T12:15:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:02 smithi042 ceph-mon[32775]: pgmap v10692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:03.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19149.301389 secs 2024-07-26T12:15:03.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:02 smithi163 ceph-mon[36873]: pgmap v10692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:03.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19149.301389 secs 2024-07-26T12:15:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:02 smithi164 ceph-mon[36970]: pgmap v10692: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:04 smithi042 ceph-mon[32775]: pgmap v10693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:15:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:04 smithi164 ceph-mon[36970]: pgmap v10693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:15:05.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:04 smithi163 ceph-mon[36873]: pgmap v10693: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:15:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:06 smithi042 ceph-mon[32775]: pgmap v10694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19136.563832 secs 2024-07-26T12:15:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:06 smithi164 ceph-mon[36970]: pgmap v10694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19136.563832 secs 2024-07-26T12:15:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:06 smithi163 ceph-mon[36873]: pgmap v10694: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:07.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19136.563832 secs 2024-07-26T12:15:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19154.301518 secs 2024-07-26T12:15:08.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19154.301518 secs 2024-07-26T12:15:08.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19154.301518 secs 2024-07-26T12:15:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:08 smithi042 ceph-mon[32775]: pgmap v10695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:08 smithi164 ceph-mon[36970]: pgmap v10695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:09.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:08 smithi163 ceph-mon[36873]: pgmap v10695: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:10 smithi042 ceph-mon[32775]: pgmap v10696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:11.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:10 smithi164 ceph-mon[36970]: pgmap v10696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:10 smithi163 ceph-mon[36873]: pgmap v10696: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19141.563947 secs 2024-07-26T12:15:12.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19141.563947 secs 2024-07-26T12:15:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19141.563947 secs 2024-07-26T12:15:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19159.301647 secs 2024-07-26T12:15:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:12 smithi042 ceph-mon[32775]: pgmap v10697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19159.301647 secs 2024-07-26T12:15:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:12 smithi164 ceph-mon[36970]: pgmap v10697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19159.301647 secs 2024-07-26T12:15:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:12 smithi163 ceph-mon[36873]: pgmap v10697: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:14.205 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:15:14.204+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:15:14.205 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:15:14.204+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:14.205 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:15:14.204+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:15:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:14 smithi042 ceph-mon[32775]: pgmap v10698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:15:15.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:14 smithi164 ceph-mon[36970]: pgmap v10698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:15:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:14 smithi163 ceph-mon[36873]: pgmap v10698: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:15:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:16 smithi042 ceph-mon[32775]: pgmap v10699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19146.564077 secs 2024-07-26T12:15:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:16 smithi164 ceph-mon[36970]: pgmap v10699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19146.564077 secs 2024-07-26T12:15:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:16 smithi163 ceph-mon[36873]: pgmap v10699: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:17.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19146.564077 secs 2024-07-26T12:15:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19164.301769 secs 2024-07-26T12:15:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19164.301769 secs 2024-07-26T12:15:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19164.301769 secs 2024-07-26T12:15:19.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:18 smithi042 ceph-mon[32775]: pgmap v10700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:18 smithi164 ceph-mon[36970]: pgmap v10700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:18 smithi163 ceph-mon[36873]: pgmap v10700: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:20 smithi042 ceph-mon[32775]: pgmap v10701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:21.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:20 smithi164 ceph-mon[36970]: pgmap v10701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:20 smithi163 ceph-mon[36873]: pgmap v10701: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:22.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19151.564243 secs 2024-07-26T12:15:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19151.564243 secs 2024-07-26T12:15:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19151.564243 secs 2024-07-26T12:15:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19169.301875 secs 2024-07-26T12:15:23.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:22 smithi042 ceph-mon[32775]: pgmap v10702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19169.301875 secs 2024-07-26T12:15:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:22 smithi164 ceph-mon[36970]: pgmap v10702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19169.301875 secs 2024-07-26T12:15:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:22 smithi163 ceph-mon[36873]: pgmap v10702: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:24 smithi042 ceph-mon[32775]: pgmap v10703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:15:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:24 smithi164 ceph-mon[36970]: pgmap v10703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:15:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:24 smithi163 ceph-mon[36873]: pgmap v10703: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:15:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:26 smithi042 ceph-mon[32775]: pgmap v10704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19156.564311 secs 2024-07-26T12:15:27.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:26 smithi164 ceph-mon[36970]: pgmap v10704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19156.564311 secs 2024-07-26T12:15:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:26 smithi163 ceph-mon[36873]: pgmap v10704: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19156.564311 secs 2024-07-26T12:15:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19174.301998 secs 2024-07-26T12:15:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19174.301998 secs 2024-07-26T12:15:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19174.301998 secs 2024-07-26T12:15:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:28 smithi042 ceph-mon[32775]: pgmap v10705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:15:29.206+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:15:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:15:29.206+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:15:29.206+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:15:29.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:28 smithi164 ceph-mon[36970]: pgmap v10705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:28 smithi163 ceph-mon[36873]: pgmap v10705: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:30 smithi042 ceph-mon[32775]: pgmap v10706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:31.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:30 smithi164 ceph-mon[36970]: pgmap v10706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:30 smithi163 ceph-mon[36873]: pgmap v10706: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19161.564448 secs 2024-07-26T12:15:32.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19161.564448 secs 2024-07-26T12:15:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19161.564448 secs 2024-07-26T12:15:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19179.302123 secs 2024-07-26T12:15:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:32 smithi042 ceph-mon[32775]: pgmap v10707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:33.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19179.302123 secs 2024-07-26T12:15:33.159 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:32 smithi163 ceph-mon[36873]: pgmap v10707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19179.302123 secs 2024-07-26T12:15:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:32 smithi164 ceph-mon[36970]: pgmap v10707: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:34.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:33 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:15:34.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:33 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:15:34.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:33 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:15:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:35 smithi042 ceph-mon[32775]: pgmap v10708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:15:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:15:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:15:35.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:35 smithi164 ceph-mon[36970]: pgmap v10708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:15:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:15:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:15:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:35 smithi163 ceph-mon[36873]: pgmap v10708: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:15:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:15:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:15:35.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:15:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:36 smithi042 ceph-mon[32775]: pgmap v10709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-07-26T12:15:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:36 smithi042 ceph-mon[32775]: pgmap v10710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:36 smithi164 ceph-mon[36970]: pgmap v10709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-07-26T12:15:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:36 smithi164 ceph-mon[36970]: pgmap v10710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:36 smithi163 ceph-mon[36873]: pgmap v10709: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-07-26T12:15:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:36 smithi163 ceph-mon[36873]: pgmap v10710: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19166.564581 secs 2024-07-26T12:15:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19184.302280 secs 2024-07-26T12:15:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19166.564581 secs 2024-07-26T12:15:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19184.302280 secs 2024-07-26T12:15:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19166.564581 secs 2024-07-26T12:15:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19184.302280 secs 2024-07-26T12:15:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:38 smithi042 ceph-mon[32775]: pgmap v10711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:15:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:38 smithi164 ceph-mon[36970]: pgmap v10711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:15:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:38 smithi163 ceph-mon[36873]: pgmap v10711: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:15:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:40 smithi042 ceph-mon[32775]: pgmap v10712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T12:15:40.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:40 smithi164 ceph-mon[36970]: pgmap v10712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T12:15:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:40 smithi163 ceph-mon[36873]: pgmap v10712: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T12:15:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:42 smithi042 ceph-mon[32775]: pgmap v10713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19171.564713 secs 2024-07-26T12:15:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19189.302435 secs 2024-07-26T12:15:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:42 smithi164 ceph-mon[36970]: pgmap v10713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19171.564713 secs 2024-07-26T12:15:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19189.302435 secs 2024-07-26T12:15:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:42 smithi163 ceph-mon[36873]: pgmap v10713: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-07-26T12:15:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19171.564713 secs 2024-07-26T12:15:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19189.302435 secs 2024-07-26T12:15:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:15:44.207+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:15:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:15:44.207+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:44.209 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:15:44.207+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:15:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:44 smithi042 ceph-mon[32775]: pgmap v10714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:15:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:44 smithi164 ceph-mon[36970]: pgmap v10714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:15:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:44 smithi163 ceph-mon[36873]: pgmap v10714: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s 2024-07-26T12:15:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:46 smithi042 ceph-mon[32775]: pgmap v10715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:15:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:46 smithi164 ceph-mon[36970]: pgmap v10715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:15:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:46 smithi163 ceph-mon[36873]: pgmap v10715: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:15:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19176.564783 secs 2024-07-26T12:15:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19194.302559 secs 2024-07-26T12:15:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19176.564783 secs 2024-07-26T12:15:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19194.302559 secs 2024-07-26T12:15:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19176.564783 secs 2024-07-26T12:15:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19194.302559 secs 2024-07-26T12:15:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:48 smithi042 ceph-mon[32775]: pgmap v10716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:48.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:48 smithi164 ceph-mon[36970]: pgmap v10716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:48 smithi163 ceph-mon[36873]: pgmap v10716: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:50 smithi042 ceph-mon[32775]: pgmap v10717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:15:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:50 smithi164 ceph-mon[36970]: pgmap v10717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:15:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:50 smithi163 ceph-mon[36873]: pgmap v10717: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:15:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:52 smithi042 ceph-mon[32775]: pgmap v10718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19181.564911 secs 2024-07-26T12:15:52.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19199.302700 secs 2024-07-26T12:15:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:52 smithi164 ceph-mon[36970]: pgmap v10718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19181.564911 secs 2024-07-26T12:15:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19199.302700 secs 2024-07-26T12:15:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:52 smithi163 ceph-mon[36873]: pgmap v10718: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19181.564911 secs 2024-07-26T12:15:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19199.302700 secs 2024-07-26T12:15:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:54 smithi042 ceph-mon[32775]: pgmap v10719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:54 smithi164 ceph-mon[36970]: pgmap v10719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:54 smithi163 ceph-mon[36873]: pgmap v10719: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:56 smithi042 ceph-mon[32775]: pgmap v10720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:56 smithi164 ceph-mon[36970]: pgmap v10720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:56 smithi163 ceph-mon[36873]: pgmap v10720: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:15:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19186.565042 secs 2024-07-26T12:15:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19204.302825 secs 2024-07-26T12:15:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19186.565042 secs 2024-07-26T12:15:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19204.302825 secs 2024-07-26T12:15:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19186.565042 secs 2024-07-26T12:15:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19204.302825 secs 2024-07-26T12:15:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:15:58 smithi042 ceph-mon[32775]: pgmap v10721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:15:58 smithi164 ceph-mon[36970]: pgmap v10721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:15:58 smithi163 ceph-mon[36873]: pgmap v10721: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:15:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:15:59.213+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:15:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:15:59.213+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:59.214 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:15:59.213+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:16:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:00 smithi042 ceph-mon[32775]: pgmap v10722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:16:00.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:00 smithi164 ceph-mon[36970]: pgmap v10722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:16:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:00 smithi163 ceph-mon[36873]: pgmap v10722: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:16:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:02 smithi042 ceph-mon[32775]: pgmap v10723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19191.565165 secs 2024-07-26T12:16:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19209.302987 secs 2024-07-26T12:16:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:02 smithi164 ceph-mon[36970]: pgmap v10723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19191.565165 secs 2024-07-26T12:16:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19209.302987 secs 2024-07-26T12:16:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:02 smithi163 ceph-mon[36873]: pgmap v10723: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19191.565165 secs 2024-07-26T12:16:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19209.302987 secs 2024-07-26T12:16:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:04 smithi042 ceph-mon[32775]: pgmap v10724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:04.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:04 smithi164 ceph-mon[36970]: pgmap v10724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:04.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:04 smithi163 ceph-mon[36873]: pgmap v10724: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:06.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:06 smithi042 ceph-mon[32775]: pgmap v10725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:06 smithi164 ceph-mon[36970]: pgmap v10725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:06 smithi163 ceph-mon[36873]: pgmap v10725: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19196.565303 secs 2024-07-26T12:16:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19214.303111 secs 2024-07-26T12:16:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19196.565303 secs 2024-07-26T12:16:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19214.303111 secs 2024-07-26T12:16:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19196.565303 secs 2024-07-26T12:16:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19214.303111 secs 2024-07-26T12:16:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:08 smithi164 ceph-mon[36970]: pgmap v10726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:08 smithi163 ceph-mon[36873]: pgmap v10726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:08.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:08 smithi042 ceph-mon[32775]: pgmap v10726: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:10 smithi164 ceph-mon[36970]: pgmap v10727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:16:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:10 smithi163 ceph-mon[36873]: pgmap v10727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:16:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:10 smithi042 ceph-mon[32775]: pgmap v10727: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:16:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:12 smithi164 ceph-mon[36970]: pgmap v10728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19201.565431 secs 2024-07-26T12:16:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19219.303264 secs 2024-07-26T12:16:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:12 smithi163 ceph-mon[36873]: pgmap v10728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19201.565431 secs 2024-07-26T12:16:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19219.303264 secs 2024-07-26T12:16:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:12 smithi042 ceph-mon[32775]: pgmap v10728: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19201.565431 secs 2024-07-26T12:16:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19219.303264 secs 2024-07-26T12:16:14.215 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:16:14.214+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:16:14.215 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:16:14.214+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:14.216 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:16:14.214+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:16:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:14 smithi164 ceph-mon[36970]: pgmap v10729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:14 smithi163 ceph-mon[36873]: pgmap v10729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:14 smithi042 ceph-mon[32775]: pgmap v10729: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:16 smithi164 ceph-mon[36970]: pgmap v10730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:16 smithi163 ceph-mon[36873]: pgmap v10730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:16 smithi042 ceph-mon[32775]: pgmap v10730: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19206.565547 secs 2024-07-26T12:16:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19224.303385 secs 2024-07-26T12:16:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19206.565547 secs 2024-07-26T12:16:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19224.303385 secs 2024-07-26T12:16:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19206.565547 secs 2024-07-26T12:16:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19224.303385 secs 2024-07-26T12:16:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:18 smithi164 ceph-mon[36970]: pgmap v10731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:16:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:18 smithi163 ceph-mon[36873]: pgmap v10731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:16:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:18 smithi042 ceph-mon[32775]: pgmap v10731: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:16:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:20 smithi164 ceph-mon[36970]: pgmap v10732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:16:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:20 smithi163 ceph-mon[36873]: pgmap v10732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:16:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:20 smithi042 ceph-mon[32775]: pgmap v10732: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:16:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:22 smithi164 ceph-mon[36970]: pgmap v10733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:16:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19211.565636 secs 2024-07-26T12:16:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19229.303511 secs 2024-07-26T12:16:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:22 smithi163 ceph-mon[36873]: pgmap v10733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:16:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19211.565636 secs 2024-07-26T12:16:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19229.303511 secs 2024-07-26T12:16:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:22 smithi042 ceph-mon[32775]: pgmap v10733: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:16:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19211.565636 secs 2024-07-26T12:16:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19229.303511 secs 2024-07-26T12:16:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:24 smithi164 ceph-mon[36970]: pgmap v10734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:16:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:24 smithi163 ceph-mon[36873]: pgmap v10734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:16:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:24 smithi042 ceph-mon[32775]: pgmap v10734: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:16:26.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:26 smithi164 ceph-mon[36970]: pgmap v10735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:26 smithi163 ceph-mon[36873]: pgmap v10735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:26 smithi042 ceph-mon[32775]: pgmap v10735: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:27.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19216.565797 secs 2024-07-26T12:16:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19234.303633 secs 2024-07-26T12:16:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19216.565797 secs 2024-07-26T12:16:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19234.303633 secs 2024-07-26T12:16:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19216.565797 secs 2024-07-26T12:16:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19234.303633 secs 2024-07-26T12:16:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:28 smithi164 ceph-mon[36970]: pgmap v10736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:28 smithi163 ceph-mon[36873]: pgmap v10736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:28 smithi042 ceph-mon[32775]: pgmap v10736: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:29.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:16:29.219+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:16:29.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:16:29.220+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:29.221 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:16:29.220+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:16:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:30 smithi164 ceph-mon[36970]: pgmap v10737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:16:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:30 smithi163 ceph-mon[36873]: pgmap v10737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:16:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:30 smithi042 ceph-mon[32775]: pgmap v10737: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:16:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:32 smithi164 ceph-mon[36970]: pgmap v10738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19221.565927 secs 2024-07-26T12:16:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19239.303775 secs 2024-07-26T12:16:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:32 smithi163 ceph-mon[36873]: pgmap v10738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19221.565927 secs 2024-07-26T12:16:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19239.303775 secs 2024-07-26T12:16:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:32 smithi042 ceph-mon[32775]: pgmap v10738: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19221.565927 secs 2024-07-26T12:16:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19239.303775 secs 2024-07-26T12:16:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:34 smithi164 ceph-mon[36970]: pgmap v10739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:34 smithi163 ceph-mon[36873]: pgmap v10739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:34 smithi042 ceph-mon[32775]: pgmap v10739: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:35 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:16:35.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:35 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:16:35.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:35 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:16:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:36 smithi164 ceph-mon[36970]: pgmap v10740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:16:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:16:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:36 smithi164 ceph-mon[36970]: pgmap v10741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:16:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:36 smithi164 ceph-mon[36970]: pgmap v10742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:36.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:16:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:36 smithi163 ceph-mon[36873]: pgmap v10740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:16:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:16:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:36 smithi163 ceph-mon[36873]: pgmap v10741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:16:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:36 smithi163 ceph-mon[36873]: pgmap v10742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:36.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:16:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:36 smithi042 ceph-mon[32775]: pgmap v10740: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:16:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:16:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:36 smithi042 ceph-mon[32775]: pgmap v10741: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:16:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:36 smithi042 ceph-mon[32775]: pgmap v10742: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:36.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:16:37.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19226.565988 secs 2024-07-26T12:16:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19244.303897 secs 2024-07-26T12:16:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19226.565988 secs 2024-07-26T12:16:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19244.303897 secs 2024-07-26T12:16:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19226.565988 secs 2024-07-26T12:16:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19244.303897 secs 2024-07-26T12:16:39.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:39 smithi164 ceph-mon[36970]: pgmap v10743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:39.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:39 smithi163 ceph-mon[36873]: pgmap v10743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:39 smithi042 ceph-mon[32775]: pgmap v10743: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:41.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:41 smithi164 ceph-mon[36970]: pgmap v10744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:41.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:41 smithi163 ceph-mon[36873]: pgmap v10744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:41 smithi042 ceph-mon[32775]: pgmap v10744: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19231.566113 secs 2024-07-26T12:16:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19249.304056 secs 2024-07-26T12:16:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19231.566113 secs 2024-07-26T12:16:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19249.304056 secs 2024-07-26T12:16:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19231.566113 secs 2024-07-26T12:16:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19249.304056 secs 2024-07-26T12:16:43.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:43 smithi163 ceph-mon[36873]: pgmap v10745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:43 smithi042 ceph-mon[32775]: pgmap v10745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:43 smithi164 ceph-mon[36970]: pgmap v10745: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:16:44.222 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:16:44.221+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:16:44.222 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:16:44.221+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:44.222 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:16:44.221+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:16:45.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:45 smithi163 ceph-mon[36873]: pgmap v10746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:16:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:45 smithi042 ceph-mon[32775]: pgmap v10746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:16:45.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:45 smithi164 ceph-mon[36970]: pgmap v10746: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:16:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:47 smithi163 ceph-mon[36873]: pgmap v10747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:16:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19236.566264 secs 2024-07-26T12:16:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19254.304138 secs 2024-07-26T12:16:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:47 smithi042 ceph-mon[32775]: pgmap v10747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:16:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19236.566264 secs 2024-07-26T12:16:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19254.304138 secs 2024-07-26T12:16:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:47 smithi164 ceph-mon[36970]: pgmap v10747: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:16:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19236.566264 secs 2024-07-26T12:16:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19254.304138 secs 2024-07-26T12:16:49.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:49 smithi042 ceph-mon[32775]: pgmap v10748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:49 smithi163 ceph-mon[36873]: pgmap v10748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:49 smithi164 ceph-mon[36970]: pgmap v10748: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:51.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:51 smithi163 ceph-mon[36873]: pgmap v10749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:51.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:51 smithi042 ceph-mon[32775]: pgmap v10749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:51 smithi164 ceph-mon[36970]: pgmap v10749: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19241.566391 secs 2024-07-26T12:16:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19259.304207 secs 2024-07-26T12:16:52.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19241.566391 secs 2024-07-26T12:16:52.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19259.304207 secs 2024-07-26T12:16:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19241.566391 secs 2024-07-26T12:16:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19259.304207 secs 2024-07-26T12:16:53.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:53 smithi163 ceph-mon[36873]: pgmap v10750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:53 smithi042 ceph-mon[32775]: pgmap v10750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:53 smithi164 ceph-mon[36970]: pgmap v10750: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:55.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:55 smithi163 ceph-mon[36873]: pgmap v10751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:16:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:55 smithi042 ceph-mon[32775]: pgmap v10751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:16:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:55 smithi164 ceph-mon[36970]: pgmap v10751: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:16:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:57 smithi042 ceph-mon[32775]: pgmap v10752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19246.566526 secs 2024-07-26T12:16:57.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19264.304350 secs 2024-07-26T12:16:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:57 smithi164 ceph-mon[36970]: pgmap v10752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19246.566526 secs 2024-07-26T12:16:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19264.304350 secs 2024-07-26T12:16:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:57 smithi163 ceph-mon[36873]: pgmap v10752: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:16:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19246.566526 secs 2024-07-26T12:16:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19264.304350 secs 2024-07-26T12:16:59.227 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:16:59.226+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:16:59.228 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:16:59.226+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:59.228 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:16:59.226+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:16:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:16:59 smithi042 ceph-mon[32775]: pgmap v10753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:16:59 smithi164 ceph-mon[36970]: pgmap v10753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:16:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:16:59 smithi163 ceph-mon[36873]: pgmap v10753: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:00.906 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:00.905+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:17:00.906 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:00.905+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658004740 con 0x7ff680102000 2024-07-26T12:17:00.906 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:00.905+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 150 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6700e9050 con 0x7ff680102000 2024-07-26T12:17:00.907 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:00.905+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 150 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:17:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:01 smithi042 ceph-mon[32775]: pgmap v10754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:01 smithi164 ceph-mon[36970]: pgmap v10754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:01 smithi163 ceph-mon[36873]: pgmap v10754: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:02.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19251.566655 secs 2024-07-26T12:17:02.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19269.304479 secs 2024-07-26T12:17:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19251.566655 secs 2024-07-26T12:17:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19269.304479 secs 2024-07-26T12:17:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19251.566655 secs 2024-07-26T12:17:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19269.304479 secs 2024-07-26T12:17:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:03 smithi042 ceph-mon[32775]: pgmap v10755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:03 smithi164 ceph-mon[36970]: pgmap v10755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:03 smithi163 ceph-mon[36873]: pgmap v10755: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:05.562 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:05 smithi163 ceph-mon[36873]: pgmap v10756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:17:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:05 smithi042 ceph-mon[32775]: pgmap v10756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:17:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:05 smithi164 ceph-mon[36970]: pgmap v10756: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:17:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:07 smithi042 ceph-mon[32775]: pgmap v10757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19256.566721 secs 2024-07-26T12:17:07.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19274.304604 secs 2024-07-26T12:17:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:07 smithi164 ceph-mon[36970]: pgmap v10757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19256.566721 secs 2024-07-26T12:17:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19274.304604 secs 2024-07-26T12:17:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:07 smithi163 ceph-mon[36873]: pgmap v10757: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19256.566721 secs 2024-07-26T12:17:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19274.304604 secs 2024-07-26T12:17:09.649 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:09 smithi042 ceph-mon[32775]: pgmap v10758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:09 smithi164 ceph-mon[36970]: pgmap v10758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:09 smithi163 ceph-mon[36873]: pgmap v10758: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:11 smithi042 ceph-mon[32775]: pgmap v10759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:11 smithi164 ceph-mon[36970]: pgmap v10759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:11 smithi163 ceph-mon[36873]: pgmap v10759: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19261.566788 secs 2024-07-26T12:17:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19279.304748 secs 2024-07-26T12:17:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19261.566788 secs 2024-07-26T12:17:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19279.304748 secs 2024-07-26T12:17:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19261.566788 secs 2024-07-26T12:17:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19279.304748 secs 2024-07-26T12:17:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:13 smithi042 ceph-mon[32775]: pgmap v10760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:13 smithi164 ceph-mon[36970]: pgmap v10760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:13 smithi163 ceph-mon[36873]: pgmap v10760: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:14.227+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:17:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:14.227+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:17:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:14.227+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:15 smithi042 ceph-mon[32775]: pgmap v10761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:17:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:15 smithi164 ceph-mon[36970]: pgmap v10761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:17:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:15 smithi163 ceph-mon[36873]: pgmap v10761: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:17:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:17 smithi042 ceph-mon[32775]: pgmap v10762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19266.566924 secs 2024-07-26T12:17:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19284.304872 secs 2024-07-26T12:17:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:17 smithi164 ceph-mon[36970]: pgmap v10762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19266.566924 secs 2024-07-26T12:17:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19284.304872 secs 2024-07-26T12:17:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:17 smithi163 ceph-mon[36873]: pgmap v10762: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19266.566924 secs 2024-07-26T12:17:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19284.304872 secs 2024-07-26T12:17:19.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:19 smithi042 ceph-mon[32775]: pgmap v10763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:19.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:19 smithi164 ceph-mon[36970]: pgmap v10763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:19 smithi163 ceph-mon[36873]: pgmap v10763: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:21 smithi042 ceph-mon[32775]: pgmap v10764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:21 smithi164 ceph-mon[36970]: pgmap v10764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:21 smithi163 ceph-mon[36873]: pgmap v10764: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:22.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19271.567052 secs 2024-07-26T12:17:22.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19289.304995 secs 2024-07-26T12:17:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19271.567052 secs 2024-07-26T12:17:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19289.304995 secs 2024-07-26T12:17:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19271.567052 secs 2024-07-26T12:17:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19289.304995 secs 2024-07-26T12:17:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:23 smithi042 ceph-mon[32775]: pgmap v10765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:23 smithi164 ceph-mon[36970]: pgmap v10765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:23 smithi163 ceph-mon[36873]: pgmap v10765: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:25.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:25 smithi042 ceph-mon[32775]: pgmap v10766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:17:25.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:25 smithi164 ceph-mon[36970]: pgmap v10766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:17:25.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:25 smithi163 ceph-mon[36873]: pgmap v10766: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:17:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:27 smithi042 ceph-mon[32775]: pgmap v10767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19276.567184 secs 2024-07-26T12:17:27.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19294.305120 secs 2024-07-26T12:17:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:27 smithi164 ceph-mon[36970]: pgmap v10767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19276.567184 secs 2024-07-26T12:17:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19294.305120 secs 2024-07-26T12:17:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:27 smithi163 ceph-mon[36873]: pgmap v10767: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19276.567184 secs 2024-07-26T12:17:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19294.305120 secs 2024-07-26T12:17:29.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:29.233+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:17:29.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:29.233+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:17:29.234 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:29.233+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:29.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:29 smithi042 ceph-mon[32775]: pgmap v10768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:29 smithi164 ceph-mon[36970]: pgmap v10768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:29 smithi163 ceph-mon[36873]: pgmap v10768: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:31.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:31 smithi042 ceph-mon[32775]: pgmap v10769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:31 smithi164 ceph-mon[36970]: pgmap v10769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:31 smithi163 ceph-mon[36873]: pgmap v10769: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19281.567313 secs 2024-07-26T12:17:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19299.305249 secs 2024-07-26T12:17:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19281.567313 secs 2024-07-26T12:17:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19299.305249 secs 2024-07-26T12:17:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19281.567313 secs 2024-07-26T12:17:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19299.305249 secs 2024-07-26T12:17:33.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:33 smithi042 ceph-mon[32775]: pgmap v10770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:33.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:33 smithi164 ceph-mon[36970]: pgmap v10770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:33 smithi163 ceph-mon[36873]: pgmap v10770: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:35.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:35 smithi042 ceph-mon[32775]: pgmap v10771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:17:35.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:35 smithi164 ceph-mon[36970]: pgmap v10771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:17:35.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:35 smithi163 ceph-mon[36873]: pgmap v10771: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:17:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:17:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:17:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:17:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:17:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:17:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:17:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:17:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:17:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:17:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:17:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:17:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:17:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:37 smithi042 ceph-mon[32775]: pgmap v10772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:37 smithi042 ceph-mon[32775]: pgmap v10773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:17:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:37 smithi042 ceph-mon[32775]: pgmap v10774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:17:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19286.567436 secs 2024-07-26T12:17:37.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19304.305338 secs 2024-07-26T12:17:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:37 smithi164 ceph-mon[36970]: pgmap v10772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:37 smithi164 ceph-mon[36970]: pgmap v10773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:17:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:37 smithi164 ceph-mon[36970]: pgmap v10774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:17:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19286.567436 secs 2024-07-26T12:17:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19304.305338 secs 2024-07-26T12:17:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:37 smithi163 ceph-mon[36873]: pgmap v10772: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:37 smithi163 ceph-mon[36873]: pgmap v10773: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:17:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:37 smithi163 ceph-mon[36873]: pgmap v10774: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:17:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19286.567436 secs 2024-07-26T12:17:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19304.305338 secs 2024-07-26T12:17:39.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:39 smithi042 ceph-mon[32775]: pgmap v10775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:17:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:39 smithi164 ceph-mon[36970]: pgmap v10775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:17:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:39 smithi163 ceph-mon[36873]: pgmap v10775: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:17:41.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:41 smithi042 ceph-mon[32775]: pgmap v10776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:17:41.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:41 smithi164 ceph-mon[36970]: pgmap v10776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:17:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:41 smithi163 ceph-mon[36873]: pgmap v10776: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:17:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19291.567584 secs 2024-07-26T12:17:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19309.305470 secs 2024-07-26T12:17:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19291.567584 secs 2024-07-26T12:17:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19309.305470 secs 2024-07-26T12:17:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19291.567584 secs 2024-07-26T12:17:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19309.305470 secs 2024-07-26T12:17:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:43 smithi042 ceph-mon[32775]: pgmap v10777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:17:43.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:43 smithi164 ceph-mon[36970]: pgmap v10777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:17:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:43 smithi163 ceph-mon[36873]: pgmap v10777: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:17:44.235 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:44.234+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:17:44.235 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:44.234+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:44.235 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:44.234+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:17:45.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:45 smithi042 ceph-mon[32775]: pgmap v10778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:17:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:45 smithi164 ceph-mon[36970]: pgmap v10778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:17:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:45 smithi163 ceph-mon[36873]: pgmap v10778: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:17:45.913 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:45.913+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:17:45.914 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:45.913+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6580036c0 con 0x7ff680102000 2024-07-26T12:17:45.914 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:45.913+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 151 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6700e9050 con 0x7ff680102000 2024-07-26T12:17:45.914 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:45.913+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 151 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:17:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:47 smithi042 ceph-mon[32775]: pgmap v10779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:17:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19296.567718 secs 2024-07-26T12:17:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19314.305595 secs 2024-07-26T12:17:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:47 smithi164 ceph-mon[36970]: pgmap v10779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:17:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19296.567718 secs 2024-07-26T12:17:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19314.305595 secs 2024-07-26T12:17:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:47 smithi163 ceph-mon[36873]: pgmap v10779: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:17:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19296.567718 secs 2024-07-26T12:17:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19314.305595 secs 2024-07-26T12:17:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:49 smithi042 ceph-mon[32775]: pgmap v10780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:49 smithi164 ceph-mon[36970]: pgmap v10780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:49 smithi163 ceph-mon[36873]: pgmap v10780: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:51 smithi042 ceph-mon[32775]: pgmap v10781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:51 smithi164 ceph-mon[36970]: pgmap v10781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:51 smithi163 ceph-mon[36873]: pgmap v10781: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19301.567845 secs 2024-07-26T12:17:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19319.305676 secs 2024-07-26T12:17:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19301.567845 secs 2024-07-26T12:17:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19319.305676 secs 2024-07-26T12:17:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19301.567845 secs 2024-07-26T12:17:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19319.305676 secs 2024-07-26T12:17:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:53 smithi042 ceph-mon[32775]: pgmap v10782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:53.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:53 smithi164 ceph-mon[36970]: pgmap v10782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:53 smithi163 ceph-mon[36873]: pgmap v10782: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:54 smithi164 ceph-mon[36970]: pgmap v10783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:17:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:54 smithi163 ceph-mon[36873]: pgmap v10783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:17:54.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:54 smithi042 ceph-mon[32775]: pgmap v10783: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:17:57.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:56 smithi163 ceph-mon[36873]: pgmap v10784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:57.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19306.567983 secs 2024-07-26T12:17:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:56 smithi042 ceph-mon[32775]: pgmap v10784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19306.567983 secs 2024-07-26T12:17:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:56 smithi164 ceph-mon[36970]: pgmap v10784: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:17:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19306.567983 secs 2024-07-26T12:17:58.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19324.305804 secs 2024-07-26T12:17:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19324.305804 secs 2024-07-26T12:17:58.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19324.305804 secs 2024-07-26T12:17:59.237 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:59.236+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:17:59.237 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:59.236+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:17:59.237 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:17:59.236+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:59.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:17:58 smithi163 ceph-mon[36873]: pgmap v10785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:59.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:17:58 smithi042 ceph-mon[32775]: pgmap v10785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:17:59.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:17:58 smithi164 ceph-mon[36970]: pgmap v10785: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:01.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:01 smithi163 ceph-mon[36873]: pgmap v10786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:01.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:01 smithi042 ceph-mon[32775]: pgmap v10786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:01.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:01 smithi164 ceph-mon[36970]: pgmap v10786: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:02.296 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19311.568128 secs 2024-07-26T12:18:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19311.568128 secs 2024-07-26T12:18:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19311.568128 secs 2024-07-26T12:18:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:03 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19329.305949 secs 2024-07-26T12:18:03.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:03 smithi042 ceph-mon[32775]: pgmap v10787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:03.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:03 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19329.305949 secs 2024-07-26T12:18:03.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:03 smithi164 ceph-mon[36970]: pgmap v10787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:03.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:03 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19329.305949 secs 2024-07-26T12:18:03.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:03 smithi163 ceph-mon[36873]: pgmap v10787: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:05.310 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:05 smithi163 ceph-mon[36873]: pgmap v10788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:18:05.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:05 smithi042 ceph-mon[32775]: pgmap v10788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:18:05.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:05 smithi164 ceph-mon[36970]: pgmap v10788: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:18:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:07 smithi042 ceph-mon[32775]: pgmap v10789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:07.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19316.568259 secs 2024-07-26T12:18:07.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19334.306062 secs 2024-07-26T12:18:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:07 smithi164 ceph-mon[36970]: pgmap v10789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19316.568259 secs 2024-07-26T12:18:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19334.306062 secs 2024-07-26T12:18:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:07 smithi163 ceph-mon[36873]: pgmap v10789: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19316.568259 secs 2024-07-26T12:18:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19334.306062 secs 2024-07-26T12:18:09.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:09 smithi042 ceph-mon[32775]: pgmap v10790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:09.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:09 smithi164 ceph-mon[36970]: pgmap v10790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:09.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:09 smithi163 ceph-mon[36873]: pgmap v10790: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:11.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:11 smithi042 ceph-mon[32775]: pgmap v10791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:11.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:11 smithi164 ceph-mon[36970]: pgmap v10791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:11.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:11 smithi163 ceph-mon[36873]: pgmap v10791: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19321.568366 secs 2024-07-26T12:18:12.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19339.306164 secs 2024-07-26T12:18:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19321.568366 secs 2024-07-26T12:18:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19339.306164 secs 2024-07-26T12:18:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19321.568366 secs 2024-07-26T12:18:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19339.306164 secs 2024-07-26T12:18:13.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:13 smithi042 ceph-mon[32775]: pgmap v10792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:13.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:13 smithi164 ceph-mon[36970]: pgmap v10792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:13.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:13 smithi163 ceph-mon[36873]: pgmap v10792: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:18:14.237+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:18:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:18:14.237+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:18:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:18:14.237+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:15.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:15 smithi042 ceph-mon[32775]: pgmap v10793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:18:15.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:15 smithi164 ceph-mon[36970]: pgmap v10793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:18:15.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:15 smithi163 ceph-mon[36873]: pgmap v10793: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:18:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:17 smithi042 ceph-mon[32775]: pgmap v10794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19326.568472 secs 2024-07-26T12:18:17.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19344.306290 secs 2024-07-26T12:18:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:17 smithi164 ceph-mon[36970]: pgmap v10794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19326.568472 secs 2024-07-26T12:18:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19344.306290 secs 2024-07-26T12:18:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:17 smithi163 ceph-mon[36873]: pgmap v10794: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:17.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19326.568472 secs 2024-07-26T12:18:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19344.306290 secs 2024-07-26T12:18:19.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:19 smithi042 ceph-mon[32775]: pgmap v10795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:19.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:19 smithi164 ceph-mon[36970]: pgmap v10795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:19.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:19 smithi163 ceph-mon[36873]: pgmap v10795: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:21.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:21 smithi042 ceph-mon[32775]: pgmap v10796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:21.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:21 smithi164 ceph-mon[36970]: pgmap v10796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:21.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:21 smithi163 ceph-mon[36873]: pgmap v10796: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19331.568551 secs 2024-07-26T12:18:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19349.306414 secs 2024-07-26T12:18:22.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19331.568551 secs 2024-07-26T12:18:22.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19349.306414 secs 2024-07-26T12:18:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19331.568551 secs 2024-07-26T12:18:22.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19349.306414 secs 2024-07-26T12:18:23.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:23 smithi042 ceph-mon[32775]: pgmap v10797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:23.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:23 smithi164 ceph-mon[36970]: pgmap v10797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:23.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:23 smithi163 ceph-mon[36873]: pgmap v10797: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:25.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:25 smithi042 ceph-mon[32775]: pgmap v10798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:18:25.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:25 smithi164 ceph-mon[36970]: pgmap v10798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:18:25.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:25 smithi163 ceph-mon[36873]: pgmap v10798: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:18:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:27 smithi042 ceph-mon[32775]: pgmap v10799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19336.568630 secs 2024-07-26T12:18:27.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19354.306539 secs 2024-07-26T12:18:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:27 smithi164 ceph-mon[36970]: pgmap v10799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19336.568630 secs 2024-07-26T12:18:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19354.306539 secs 2024-07-26T12:18:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:27 smithi163 ceph-mon[36873]: pgmap v10799: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:27.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19336.568630 secs 2024-07-26T12:18:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19354.306539 secs 2024-07-26T12:18:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:18:29.243+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:18:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:18:29.243+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:18:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:18:29.243+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:29.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:29 smithi042 ceph-mon[32775]: pgmap v10800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:29.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:29 smithi164 ceph-mon[36970]: pgmap v10800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:29.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:29 smithi163 ceph-mon[36873]: pgmap v10800: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:31.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:31 smithi042 ceph-mon[32775]: pgmap v10801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:31.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:31 smithi164 ceph-mon[36970]: pgmap v10801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:31.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:31 smithi163 ceph-mon[36873]: pgmap v10801: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19341.568772 secs 2024-07-26T12:18:32.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19359.306684 secs 2024-07-26T12:18:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19341.568772 secs 2024-07-26T12:18:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19359.306684 secs 2024-07-26T12:18:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19341.568772 secs 2024-07-26T12:18:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19359.306684 secs 2024-07-26T12:18:33.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:33 smithi042 ceph-mon[32775]: pgmap v10802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:33.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:33 smithi164 ceph-mon[36970]: pgmap v10802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:33.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:33 smithi163 ceph-mon[36873]: pgmap v10802: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:35.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:35 smithi042 ceph-mon[32775]: pgmap v10803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:18:35.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:35 smithi164 ceph-mon[36970]: pgmap v10803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:18:35.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:35 smithi163 ceph-mon[36873]: pgmap v10803: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:18:36.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:36 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:18:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:36 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:18:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:36 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:18:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:37 smithi042 ceph-mon[32775]: pgmap v10804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19346.568919 secs 2024-07-26T12:18:37.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19364.306811 secs 2024-07-26T12:18:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:18:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:18:37.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:18:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:37 smithi164 ceph-mon[36970]: pgmap v10804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19346.568919 secs 2024-07-26T12:18:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19364.306811 secs 2024-07-26T12:18:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:18:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:18:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:18:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:37 smithi163 ceph-mon[36873]: pgmap v10804: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19346.568919 secs 2024-07-26T12:18:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19364.306811 secs 2024-07-26T12:18:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:18:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:18:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:18:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:38 smithi042 ceph-mon[32775]: pgmap v10805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:18:38.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:38 smithi042 ceph-mon[32775]: pgmap v10806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:18:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:38 smithi164 ceph-mon[36970]: pgmap v10805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:18:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:38 smithi164 ceph-mon[36970]: pgmap v10806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:18:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:38 smithi163 ceph-mon[36873]: pgmap v10805: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:18:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:38 smithi163 ceph-mon[36873]: pgmap v10806: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:18:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:40 smithi042 ceph-mon[32775]: pgmap v10807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:18:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:40 smithi164 ceph-mon[36970]: pgmap v10807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:18:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:40 smithi163 ceph-mon[36873]: pgmap v10807: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:18:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:42 smithi042 ceph-mon[32775]: pgmap v10808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:18:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19351.569074 secs 2024-07-26T12:18:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19369.306915 secs 2024-07-26T12:18:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:42 smithi164 ceph-mon[36970]: pgmap v10808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:18:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19351.569074 secs 2024-07-26T12:18:42.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19369.306915 secs 2024-07-26T12:18:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:42 smithi163 ceph-mon[36873]: pgmap v10808: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:18:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19351.569074 secs 2024-07-26T12:18:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19369.306915 secs 2024-07-26T12:18:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:18:44.244+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:18:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:18:44.244+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:18:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:18:44.244+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:44.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:44 smithi042 ceph-mon[32775]: pgmap v10809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:18:44.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:44 smithi164 ceph-mon[36970]: pgmap v10809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:18:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:44 smithi163 ceph-mon[36873]: pgmap v10809: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:18:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:46 smithi042 ceph-mon[32775]: pgmap v10810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:18:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:46 smithi164 ceph-mon[36970]: pgmap v10810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:18:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:46 smithi163 ceph-mon[36873]: pgmap v10810: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:18:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19356.569200 secs 2024-07-26T12:18:47.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19374.307039 secs 2024-07-26T12:18:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19356.569200 secs 2024-07-26T12:18:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19374.307039 secs 2024-07-26T12:18:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19356.569200 secs 2024-07-26T12:18:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19374.307039 secs 2024-07-26T12:18:48.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:48 smithi042 ceph-mon[32775]: pgmap v10811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:18:48.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:48 smithi164 ceph-mon[36970]: pgmap v10811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:18:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:48 smithi163 ceph-mon[36873]: pgmap v10811: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:18:50.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:50 smithi042 ceph-mon[32775]: pgmap v10812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:18:50.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:50 smithi164 ceph-mon[36970]: pgmap v10812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:18:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:50 smithi163 ceph-mon[36873]: pgmap v10812: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:18:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:52 smithi042 ceph-mon[32775]: pgmap v10813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19361.569322 secs 2024-07-26T12:18:52.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19379.307185 secs 2024-07-26T12:18:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:52 smithi164 ceph-mon[36970]: pgmap v10813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19361.569322 secs 2024-07-26T12:18:52.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19379.307185 secs 2024-07-26T12:18:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:52 smithi163 ceph-mon[36873]: pgmap v10813: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19361.569322 secs 2024-07-26T12:18:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19379.307185 secs 2024-07-26T12:18:54.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:54 smithi042 ceph-mon[32775]: pgmap v10814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:54.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:54 smithi164 ceph-mon[36970]: pgmap v10814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:54 smithi163 ceph-mon[36873]: pgmap v10814: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:56 smithi042 ceph-mon[32775]: pgmap v10815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:56.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:56 smithi164 ceph-mon[36970]: pgmap v10815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:56 smithi163 ceph-mon[36873]: pgmap v10815: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:18:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19366.569425 secs 2024-07-26T12:18:57.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19384.307261 secs 2024-07-26T12:18:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19366.569425 secs 2024-07-26T12:18:57.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19384.307261 secs 2024-07-26T12:18:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19366.569425 secs 2024-07-26T12:18:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19384.307261 secs 2024-07-26T12:18:58.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:18:58 smithi042 ceph-mon[32775]: pgmap v10816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:58.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:18:58 smithi164 ceph-mon[36970]: pgmap v10816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:58.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:18:58 smithi163 ceph-mon[36873]: pgmap v10816: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:18:59.251 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:18:59.250+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:18:59.251 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:18:59.250+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:18:59.251 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:18:59.250+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:00.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:00 smithi042 ceph-mon[32775]: pgmap v10817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:19:00.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:00 smithi164 ceph-mon[36970]: pgmap v10817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:19:00.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:00 smithi163 ceph-mon[36873]: pgmap v10817: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:19:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:02 smithi042 ceph-mon[32775]: pgmap v10818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19371.569544 secs 2024-07-26T12:19:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19389.307416 secs 2024-07-26T12:19:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:02 smithi164 ceph-mon[36970]: pgmap v10818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19371.569544 secs 2024-07-26T12:19:02.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19389.307416 secs 2024-07-26T12:19:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:02 smithi163 ceph-mon[36873]: pgmap v10818: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:02.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19371.569544 secs 2024-07-26T12:19:02.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19389.307416 secs 2024-07-26T12:19:04.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:04 smithi042 ceph-mon[32775]: pgmap v10819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:04.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:04 smithi164 ceph-mon[36970]: pgmap v10819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:04.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:04 smithi163 ceph-mon[36873]: pgmap v10819: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:06.398 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:06 smithi042 ceph-mon[32775]: pgmap v10820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:06.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:06 smithi164 ceph-mon[36970]: pgmap v10820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:06.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:06 smithi163 ceph-mon[36873]: pgmap v10820: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:07.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19376.569678 secs 2024-07-26T12:19:07.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19394.307543 secs 2024-07-26T12:19:07.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19376.569678 secs 2024-07-26T12:19:07.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19394.307543 secs 2024-07-26T12:19:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19376.569678 secs 2024-07-26T12:19:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19394.307543 secs 2024-07-26T12:19:08.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:08 smithi164 ceph-mon[36970]: pgmap v10821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:08.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:08 smithi163 ceph-mon[36873]: pgmap v10821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:08.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:08 smithi042 ceph-mon[32775]: pgmap v10821: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:10.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:10 smithi164 ceph-mon[36970]: pgmap v10822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:19:10.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:10 smithi163 ceph-mon[36873]: pgmap v10822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:19:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:10 smithi042 ceph-mon[32775]: pgmap v10822: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:19:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:12 smithi164 ceph-mon[36970]: pgmap v10823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19381.569804 secs 2024-07-26T12:19:12.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19399.307611 secs 2024-07-26T12:19:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:12 smithi163 ceph-mon[36873]: pgmap v10823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:12.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19381.569804 secs 2024-07-26T12:19:12.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19399.307611 secs 2024-07-26T12:19:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:12 smithi042 ceph-mon[32775]: pgmap v10823: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19381.569804 secs 2024-07-26T12:19:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19399.307611 secs 2024-07-26T12:19:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:19:14.251+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:19:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:19:14.251+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:19:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:19:14.251+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:14.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:14 smithi164 ceph-mon[36970]: pgmap v10824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:14.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:14 smithi163 ceph-mon[36873]: pgmap v10824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:14 smithi042 ceph-mon[32775]: pgmap v10824: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:16.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:16 smithi164 ceph-mon[36970]: pgmap v10825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:16.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:16 smithi163 ceph-mon[36873]: pgmap v10825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:16 smithi042 ceph-mon[32775]: pgmap v10825: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19386.569950 secs 2024-07-26T12:19:17.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19404.307736 secs 2024-07-26T12:19:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19386.569950 secs 2024-07-26T12:19:17.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19404.307736 secs 2024-07-26T12:19:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19386.569950 secs 2024-07-26T12:19:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19404.307736 secs 2024-07-26T12:19:18.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:18 smithi164 ceph-mon[36970]: pgmap v10826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:18.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:18 smithi163 ceph-mon[36873]: pgmap v10826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:18 smithi042 ceph-mon[32775]: pgmap v10826: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:20.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:20 smithi164 ceph-mon[36970]: pgmap v10827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:19:20.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:20 smithi163 ceph-mon[36873]: pgmap v10827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:19:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:20 smithi042 ceph-mon[32775]: pgmap v10827: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:19:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:22 smithi164 ceph-mon[36970]: pgmap v10828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19391.570094 secs 2024-07-26T12:19:22.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19409.307865 secs 2024-07-26T12:19:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:22 smithi163 ceph-mon[36873]: pgmap v10828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19391.570094 secs 2024-07-26T12:19:22.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19409.307865 secs 2024-07-26T12:19:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:22 smithi042 ceph-mon[32775]: pgmap v10828: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19391.570094 secs 2024-07-26T12:19:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19409.307865 secs 2024-07-26T12:19:24.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:24 smithi164 ceph-mon[36970]: pgmap v10829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:24.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:24 smithi163 ceph-mon[36873]: pgmap v10829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:24 smithi042 ceph-mon[32775]: pgmap v10829: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:26.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:26 smithi164 ceph-mon[36970]: pgmap v10830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:26.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:26 smithi163 ceph-mon[36873]: pgmap v10830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:26 smithi042 ceph-mon[32775]: pgmap v10830: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19396.570169 secs 2024-07-26T12:19:27.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19414.308022 secs 2024-07-26T12:19:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19396.570169 secs 2024-07-26T12:19:27.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19414.308022 secs 2024-07-26T12:19:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19396.570169 secs 2024-07-26T12:19:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19414.308022 secs 2024-07-26T12:19:28.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:28 smithi164 ceph-mon[36970]: pgmap v10831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:28.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:28 smithi163 ceph-mon[36873]: pgmap v10831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:28 smithi042 ceph-mon[32775]: pgmap v10831: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:19:29.257+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:19:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:19:29.257+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:19:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:19:29.257+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:30.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:30 smithi164 ceph-mon[36970]: pgmap v10832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:19:30.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:30 smithi163 ceph-mon[36873]: pgmap v10832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:19:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:30 smithi042 ceph-mon[32775]: pgmap v10832: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:19:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:32 smithi164 ceph-mon[36970]: pgmap v10833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19401.570288 secs 2024-07-26T12:19:32.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19419.308146 secs 2024-07-26T12:19:32.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:32 smithi163 ceph-mon[36873]: pgmap v10833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19401.570288 secs 2024-07-26T12:19:32.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19419.308146 secs 2024-07-26T12:19:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:32 smithi042 ceph-mon[32775]: pgmap v10833: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19401.570288 secs 2024-07-26T12:19:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19419.308146 secs 2024-07-26T12:19:34.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:34 smithi164 ceph-mon[36970]: pgmap v10834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:34.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:34 smithi163 ceph-mon[36873]: pgmap v10834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:34 smithi042 ceph-mon[32775]: pgmap v10834: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:36.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:36 smithi164 ceph-mon[36970]: pgmap v10835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:36.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:36 smithi163 ceph-mon[36873]: pgmap v10835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:36 smithi042 ceph-mon[32775]: pgmap v10835: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19406.570410 secs 2024-07-26T12:19:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19424.308232 secs 2024-07-26T12:19:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:19:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:19:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:19:37.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:19:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19406.570410 secs 2024-07-26T12:19:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19424.308232 secs 2024-07-26T12:19:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:19:37.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:19:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:19:37.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:19:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19406.570410 secs 2024-07-26T12:19:37.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19424.308232 secs 2024-07-26T12:19:37.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:19:37.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:19:37.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:19:37.669 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:19:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:38 smithi164 ceph-mon[36970]: pgmap v10836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:38 smithi164 ceph-mon[36970]: pgmap v10837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:19:38.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:38 smithi164 ceph-mon[36970]: pgmap v10838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:19:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:38 smithi163 ceph-mon[36873]: pgmap v10836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:38.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:38 smithi163 ceph-mon[36873]: pgmap v10837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:19:38.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:38 smithi163 ceph-mon[36873]: pgmap v10838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:19:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:38 smithi042 ceph-mon[32775]: pgmap v10836: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:38 smithi042 ceph-mon[32775]: pgmap v10837: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:19:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:38 smithi042 ceph-mon[32775]: pgmap v10838: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:19:40.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:40 smithi042 ceph-mon[32775]: pgmap v10839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:19:40.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:40 smithi164 ceph-mon[36970]: pgmap v10839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:19:40.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:40 smithi163 ceph-mon[36873]: pgmap v10839: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:19:42.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:42 smithi163 ceph-mon[36873]: pgmap v10840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:19:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19411.570531 secs 2024-07-26T12:19:42.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19429.308353 secs 2024-07-26T12:19:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:42 smithi042 ceph-mon[32775]: pgmap v10840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:19:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19411.570531 secs 2024-07-26T12:19:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19429.308353 secs 2024-07-26T12:19:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:42 smithi164 ceph-mon[36970]: pgmap v10840: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:19:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19411.570531 secs 2024-07-26T12:19:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19429.308353 secs 2024-07-26T12:19:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:19:44.257+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:19:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:19:44.257+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:19:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:19:44.257+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:44.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:44 smithi163 ceph-mon[36873]: pgmap v10841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:19:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:44 smithi042 ceph-mon[32775]: pgmap v10841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:19:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:44 smithi164 ceph-mon[36970]: pgmap v10841: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:19:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:46 smithi163 ceph-mon[36873]: pgmap v10842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:19:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:46 smithi042 ceph-mon[32775]: pgmap v10842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:19:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:46 smithi164 ceph-mon[36970]: pgmap v10842: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:19:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19416.570667 secs 2024-07-26T12:19:47.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19434.308480 secs 2024-07-26T12:19:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19416.570667 secs 2024-07-26T12:19:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19434.308480 secs 2024-07-26T12:19:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19416.570667 secs 2024-07-26T12:19:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19434.308480 secs 2024-07-26T12:19:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:48 smithi163 ceph-mon[36873]: pgmap v10843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:19:48.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:48 smithi042 ceph-mon[32775]: pgmap v10843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:19:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:48 smithi164 ceph-mon[36970]: pgmap v10843: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:19:50.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:50 smithi163 ceph-mon[36873]: pgmap v10844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:50 smithi042 ceph-mon[32775]: pgmap v10844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:50 smithi164 ceph-mon[36970]: pgmap v10844: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:52.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:52 smithi163 ceph-mon[36873]: pgmap v10845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19421.570808 secs 2024-07-26T12:19:52.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19439.308578 secs 2024-07-26T12:19:52.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:52 smithi164 ceph-mon[36970]: pgmap v10845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:52.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19421.570808 secs 2024-07-26T12:19:52.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19439.308578 secs 2024-07-26T12:19:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:52 smithi042 ceph-mon[32775]: pgmap v10845: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19421.570808 secs 2024-07-26T12:19:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19439.308578 secs 2024-07-26T12:19:54.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:54 smithi163 ceph-mon[36873]: pgmap v10846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:54 smithi042 ceph-mon[32775]: pgmap v10846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:54 smithi164 ceph-mon[36970]: pgmap v10846: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:56.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:56 smithi042 ceph-mon[32775]: pgmap v10847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:56.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:56 smithi163 ceph-mon[36873]: pgmap v10847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:56 smithi164 ceph-mon[36970]: pgmap v10847: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:19:57.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19426.570936 secs 2024-07-26T12:19:57.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19444.308722 secs 2024-07-26T12:19:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19426.570936 secs 2024-07-26T12:19:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19444.308722 secs 2024-07-26T12:19:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19426.570936 secs 2024-07-26T12:19:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19444.308722 secs 2024-07-26T12:19:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:19:58 smithi042 ceph-mon[32775]: pgmap v10848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:58.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:19:58 smithi164 ceph-mon[36970]: pgmap v10848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:19:58 smithi163 ceph-mon[36873]: pgmap v10848: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:19:59.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:19:59.263+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:19:59.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:19:59.263+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:59.264 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:19:59.263+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:20:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:00 smithi042 ceph-mon[32775]: pgmap v10849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:00 smithi042 ceph-mon[32775]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T12:20:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:00 smithi164 ceph-mon[36970]: pgmap v10849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:00 smithi164 ceph-mon[36970]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T12:20:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:00 smithi163 ceph-mon[36873]: pgmap v10849: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:00.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:00 smithi163 ceph-mon[36873]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 filesystem is degraded; 2 MDSs report slow requests 2024-07-26T12:20:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:20:00.933+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580036c0 con 0x7ff680102000 2024-07-26T12:20:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:20:00.933+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:20:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:20:00.934+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 152 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6700e9050 con 0x7ff680102000 2024-07-26T12:20:00.935 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:20:00.934+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 152 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:20:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:02 smithi042 ceph-mon[32775]: pgmap v10850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19431.571073 secs 2024-07-26T12:20:02.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19449.308827 secs 2024-07-26T12:20:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:02 smithi164 ceph-mon[36970]: pgmap v10850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19431.571073 secs 2024-07-26T12:20:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19449.308827 secs 2024-07-26T12:20:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:02 smithi163 ceph-mon[36873]: pgmap v10850: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19431.571073 secs 2024-07-26T12:20:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19449.308827 secs 2024-07-26T12:20:04.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:04 smithi042 ceph-mon[32775]: pgmap v10851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:04 smithi164 ceph-mon[36970]: pgmap v10851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:04 smithi163 ceph-mon[36873]: pgmap v10851: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:06.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:06 smithi163 ceph-mon[36873]: pgmap v10852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:06 smithi042 ceph-mon[32775]: pgmap v10852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:06 smithi164 ceph-mon[36970]: pgmap v10852: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19436.571194 secs 2024-07-26T12:20:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19454.308972 secs 2024-07-26T12:20:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19436.571194 secs 2024-07-26T12:20:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19454.308972 secs 2024-07-26T12:20:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19436.571194 secs 2024-07-26T12:20:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19454.308972 secs 2024-07-26T12:20:08.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:08 smithi042 ceph-mon[32775]: pgmap v10853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:08 smithi164 ceph-mon[36970]: pgmap v10853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:08 smithi163 ceph-mon[36873]: pgmap v10853: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:10 smithi042 ceph-mon[32775]: pgmap v10854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:10.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:10 smithi164 ceph-mon[36970]: pgmap v10854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:10 smithi163 ceph-mon[36873]: pgmap v10854: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:12 smithi042 ceph-mon[32775]: pgmap v10855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19441.571343 secs 2024-07-26T12:20:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19459.309102 secs 2024-07-26T12:20:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:12 smithi164 ceph-mon[36970]: pgmap v10855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19441.571343 secs 2024-07-26T12:20:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19459.309102 secs 2024-07-26T12:20:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:12 smithi163 ceph-mon[36873]: pgmap v10855: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19441.571343 secs 2024-07-26T12:20:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19459.309102 secs 2024-07-26T12:20:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:20:14.265+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:20:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:20:14.265+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:14.266 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:20:14.265+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:20:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:14 smithi042 ceph-mon[32775]: pgmap v10856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:14 smithi164 ceph-mon[36970]: pgmap v10856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:14 smithi163 ceph-mon[36873]: pgmap v10856: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:16 smithi042 ceph-mon[32775]: pgmap v10857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:16.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:16 smithi164 ceph-mon[36970]: pgmap v10857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:16 smithi163 ceph-mon[36873]: pgmap v10857: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19446.571469 secs 2024-07-26T12:20:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19464.309197 secs 2024-07-26T12:20:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19446.571469 secs 2024-07-26T12:20:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19464.309197 secs 2024-07-26T12:20:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19446.571469 secs 2024-07-26T12:20:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19464.309197 secs 2024-07-26T12:20:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:18 smithi042 ceph-mon[32775]: pgmap v10858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:18.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:18 smithi164 ceph-mon[36970]: pgmap v10858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:18 smithi163 ceph-mon[36873]: pgmap v10858: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:20 smithi042 ceph-mon[32775]: pgmap v10859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:20.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:20 smithi164 ceph-mon[36970]: pgmap v10859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:20 smithi163 ceph-mon[36873]: pgmap v10859: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:22 smithi042 ceph-mon[32775]: pgmap v10860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19451.571569 secs 2024-07-26T12:20:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19469.309313 secs 2024-07-26T12:20:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:22 smithi164 ceph-mon[36970]: pgmap v10860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19451.571569 secs 2024-07-26T12:20:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19469.309313 secs 2024-07-26T12:20:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:22 smithi163 ceph-mon[36873]: pgmap v10860: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19451.571569 secs 2024-07-26T12:20:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19469.309313 secs 2024-07-26T12:20:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:24 smithi042 ceph-mon[32775]: pgmap v10861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:24.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:24 smithi164 ceph-mon[36970]: pgmap v10861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:24 smithi163 ceph-mon[36873]: pgmap v10861: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:26.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:26 smithi042 ceph-mon[32775]: pgmap v10862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:26 smithi164 ceph-mon[36970]: pgmap v10862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:26 smithi163 ceph-mon[36873]: pgmap v10862: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19456.571727 secs 2024-07-26T12:20:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19474.309383 secs 2024-07-26T12:20:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19456.571727 secs 2024-07-26T12:20:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19474.309383 secs 2024-07-26T12:20:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19456.571727 secs 2024-07-26T12:20:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19474.309383 secs 2024-07-26T12:20:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:28 smithi042 ceph-mon[32775]: pgmap v10863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:28.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:28 smithi164 ceph-mon[36970]: pgmap v10863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:28 smithi163 ceph-mon[36873]: pgmap v10863: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:20:29.271+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:20:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:20:29.271+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:20:29.271+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:20:30.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:30 smithi042 ceph-mon[32775]: pgmap v10864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:30 smithi164 ceph-mon[36970]: pgmap v10864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:30.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:30 smithi163 ceph-mon[36873]: pgmap v10864: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:32 smithi042 ceph-mon[32775]: pgmap v10865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19461.571870 secs 2024-07-26T12:20:32.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19479.309532 secs 2024-07-26T12:20:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:32 smithi164 ceph-mon[36970]: pgmap v10865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19461.571870 secs 2024-07-26T12:20:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19479.309532 secs 2024-07-26T12:20:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:32 smithi163 ceph-mon[36873]: pgmap v10865: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19461.571870 secs 2024-07-26T12:20:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19479.309532 secs 2024-07-26T12:20:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:34 smithi042 ceph-mon[32775]: pgmap v10866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:34 smithi164 ceph-mon[36970]: pgmap v10866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:34 smithi163 ceph-mon[36873]: pgmap v10866: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:36 smithi042 ceph-mon[32775]: pgmap v10867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:36 smithi164 ceph-mon[36970]: pgmap v10867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:36 smithi163 ceph-mon[36873]: pgmap v10867: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19466.571998 secs 2024-07-26T12:20:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19484.309678 secs 2024-07-26T12:20:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:37 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:20:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19466.571998 secs 2024-07-26T12:20:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19484.309678 secs 2024-07-26T12:20:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:37 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:20:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19466.571998 secs 2024-07-26T12:20:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19484.309678 secs 2024-07-26T12:20:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:37 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:20:38.617 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:38 smithi042 ceph-mon[32775]: pgmap v10868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:38.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:38 smithi164 ceph-mon[36970]: pgmap v10868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:38 smithi163 ceph-mon[36873]: pgmap v10868: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:40.605 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:40 smithi042 ceph-mon[32775]: pgmap v10869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:40 smithi164 ceph-mon[36970]: pgmap v10869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:40 smithi163 ceph-mon[36873]: pgmap v10869: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:42.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:42 smithi042 ceph-mon[32775]: pgmap v10870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19471.572115 secs 2024-07-26T12:20:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:20:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:20:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:20:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:20:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:42 smithi042 ceph-mon[32775]: pgmap v10871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:20:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:42 smithi042 ceph-mon[32775]: pgmap v10872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:20:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:20:42.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19489.309826 secs 2024-07-26T12:20:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:42 smithi164 ceph-mon[36970]: pgmap v10870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19471.572115 secs 2024-07-26T12:20:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:20:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:20:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:20:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:20:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:42 smithi164 ceph-mon[36970]: pgmap v10871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:20:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:42 smithi164 ceph-mon[36970]: pgmap v10872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:20:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:20:42.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19489.309826 secs 2024-07-26T12:20:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:42 smithi163 ceph-mon[36873]: pgmap v10870: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19471.572115 secs 2024-07-26T12:20:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:20:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:20:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:20:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:20:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:42 smithi163 ceph-mon[36873]: pgmap v10871: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:20:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:42 smithi163 ceph-mon[36873]: pgmap v10872: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:20:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:20:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19489.309826 secs 2024-07-26T12:20:44.274 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:20:44.273+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:20:44.274 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:20:44.273+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:44.274 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:20:44.273+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:20:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:44 smithi042 ceph-mon[32775]: pgmap v10873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:20:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:44 smithi164 ceph-mon[36970]: pgmap v10873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:20:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:44 smithi163 ceph-mon[36873]: pgmap v10873: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:20:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:46 smithi042 ceph-mon[32775]: pgmap v10874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:20:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:46 smithi164 ceph-mon[36970]: pgmap v10874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:20:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:46 smithi163 ceph-mon[36873]: pgmap v10874: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:20:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19476.572245 secs 2024-07-26T12:20:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19494.309936 secs 2024-07-26T12:20:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19476.572245 secs 2024-07-26T12:20:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19494.309936 secs 2024-07-26T12:20:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19476.572245 secs 2024-07-26T12:20:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19494.309936 secs 2024-07-26T12:20:48.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:48 smithi042 ceph-mon[32775]: pgmap v10875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:20:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:48 smithi164 ceph-mon[36970]: pgmap v10875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:20:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:48 smithi163 ceph-mon[36873]: pgmap v10875: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:20:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:50 smithi042 ceph-mon[32775]: pgmap v10876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:20:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:50 smithi164 ceph-mon[36970]: pgmap v10876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:20:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:50 smithi163 ceph-mon[36873]: pgmap v10876: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:20:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19481.572365 secs 2024-07-26T12:20:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:52 smithi042 ceph-mon[32775]: pgmap v10877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:20:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19499.310058 secs 2024-07-26T12:20:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19481.572365 secs 2024-07-26T12:20:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:52 smithi164 ceph-mon[36970]: pgmap v10877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:20:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19499.310058 secs 2024-07-26T12:20:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19481.572365 secs 2024-07-26T12:20:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:52 smithi163 ceph-mon[36873]: pgmap v10877: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:20:52.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19499.310058 secs 2024-07-26T12:20:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:54 smithi042 ceph-mon[32775]: pgmap v10878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:20:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:54 smithi164 ceph-mon[36970]: pgmap v10878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:20:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:54 smithi163 ceph-mon[36873]: pgmap v10878: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:20:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:56 smithi042 ceph-mon[32775]: pgmap v10879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:56 smithi164 ceph-mon[36970]: pgmap v10879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:56 smithi163 ceph-mon[36873]: pgmap v10879: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:20:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19486.572465 secs 2024-07-26T12:20:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19504.310148 secs 2024-07-26T12:20:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19486.572465 secs 2024-07-26T12:20:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19504.310148 secs 2024-07-26T12:20:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19486.572465 secs 2024-07-26T12:20:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19504.310148 secs 2024-07-26T12:20:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:20:58 smithi042 ceph-mon[32775]: pgmap v10880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:58.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:20:58 smithi164 ceph-mon[36970]: pgmap v10880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:20:58 smithi163 ceph-mon[36873]: pgmap v10880: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:20:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:20:59.279+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:20:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:20:59.279+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:59.280 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:20:59.279+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:21:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:00 smithi042 ceph-mon[32775]: pgmap v10881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:00 smithi164 ceph-mon[36970]: pgmap v10881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:00 smithi163 ceph-mon[36873]: pgmap v10881: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19491.572589 secs 2024-07-26T12:21:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:02 smithi164 ceph-mon[36970]: pgmap v10882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19509.310271 secs 2024-07-26T12:21:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19491.572589 secs 2024-07-26T12:21:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:02 smithi163 ceph-mon[36873]: pgmap v10882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:02.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19509.310271 secs 2024-07-26T12:21:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19491.572589 secs 2024-07-26T12:21:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:02 smithi042 ceph-mon[32775]: pgmap v10882: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:02.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19509.310271 secs 2024-07-26T12:21:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:04 smithi164 ceph-mon[36970]: pgmap v10883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:21:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:04 smithi163 ceph-mon[36873]: pgmap v10883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:21:04.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:04 smithi042 ceph-mon[32775]: pgmap v10883: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:21:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:06 smithi164 ceph-mon[36970]: pgmap v10884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:06 smithi163 ceph-mon[36873]: pgmap v10884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:06.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:06 smithi042 ceph-mon[32775]: pgmap v10884: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19496.572691 secs 2024-07-26T12:21:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19514.310391 secs 2024-07-26T12:21:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19496.572691 secs 2024-07-26T12:21:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19514.310391 secs 2024-07-26T12:21:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19496.572691 secs 2024-07-26T12:21:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19514.310391 secs 2024-07-26T12:21:08.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:08 smithi164 ceph-mon[36970]: pgmap v10885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:08 smithi163 ceph-mon[36873]: pgmap v10885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:08.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:08 smithi042 ceph-mon[32775]: pgmap v10885: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:09 smithi164 ceph-mon[36970]: pgmap v10886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:09 smithi163 ceph-mon[36873]: pgmap v10886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:09 smithi042 ceph-mon[32775]: pgmap v10886: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19501.572813 secs 2024-07-26T12:21:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:12 smithi042 ceph-mon[32775]: pgmap v10887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19519.310499 secs 2024-07-26T12:21:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19501.572813 secs 2024-07-26T12:21:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:12 smithi164 ceph-mon[36970]: pgmap v10887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19519.310499 secs 2024-07-26T12:21:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19501.572813 secs 2024-07-26T12:21:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:12 smithi163 ceph-mon[36873]: pgmap v10887: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19519.310499 secs 2024-07-26T12:21:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:21:14.280+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:21:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:21:14.280+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:21:14.280+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:21:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:14 smithi042 ceph-mon[32775]: pgmap v10888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:21:14.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:14 smithi164 ceph-mon[36970]: pgmap v10888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:21:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:14 smithi163 ceph-mon[36873]: pgmap v10888: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:21:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:16 smithi042 ceph-mon[32775]: pgmap v10889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:16 smithi164 ceph-mon[36970]: pgmap v10889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:16 smithi163 ceph-mon[36873]: pgmap v10889: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19506.572943 secs 2024-07-26T12:21:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19524.310585 secs 2024-07-26T12:21:17.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19506.572943 secs 2024-07-26T12:21:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19524.310585 secs 2024-07-26T12:21:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19506.572943 secs 2024-07-26T12:21:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19524.310585 secs 2024-07-26T12:21:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:18 smithi042 ceph-mon[32775]: pgmap v10890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:18.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:18 smithi164 ceph-mon[36970]: pgmap v10890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:18 smithi163 ceph-mon[36873]: pgmap v10890: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:20 smithi042 ceph-mon[32775]: pgmap v10891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:20.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:20 smithi164 ceph-mon[36970]: pgmap v10891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:20 smithi163 ceph-mon[36873]: pgmap v10891: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19511.573044 secs 2024-07-26T12:21:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:22 smithi042 ceph-mon[32775]: pgmap v10892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19529.310706 secs 2024-07-26T12:21:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19511.573044 secs 2024-07-26T12:21:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:22 smithi164 ceph-mon[36970]: pgmap v10892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19529.310706 secs 2024-07-26T12:21:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19511.573044 secs 2024-07-26T12:21:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:22 smithi163 ceph-mon[36873]: pgmap v10892: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19529.310706 secs 2024-07-26T12:21:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:24 smithi042 ceph-mon[32775]: pgmap v10893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:21:24.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:24 smithi164 ceph-mon[36970]: pgmap v10893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:21:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:24 smithi163 ceph-mon[36873]: pgmap v10893: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:21:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:26 smithi164 ceph-mon[36970]: pgmap v10894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:26 smithi163 ceph-mon[36873]: pgmap v10894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:26 smithi042 ceph-mon[32775]: pgmap v10894: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:27.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19516.573173 secs 2024-07-26T12:21:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19534.310829 secs 2024-07-26T12:21:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19516.573173 secs 2024-07-26T12:21:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19534.310829 secs 2024-07-26T12:21:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19516.573173 secs 2024-07-26T12:21:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19534.310829 secs 2024-07-26T12:21:28.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:28 smithi042 ceph-mon[32775]: pgmap v10895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:28 smithi164 ceph-mon[36970]: pgmap v10895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:28.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:28 smithi163 ceph-mon[36873]: pgmap v10895: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:21:29.287+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:21:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:21:29.287+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:21:29.287+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:21:30.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:30 smithi164 ceph-mon[36970]: pgmap v10896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:30 smithi163 ceph-mon[36873]: pgmap v10896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:30 smithi042 ceph-mon[32775]: pgmap v10896: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:32.697 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19521.573260 secs 2024-07-26T12:21:32.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:32 smithi042 ceph-mon[32775]: pgmap v10897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:32.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19539.310980 secs 2024-07-26T12:21:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19521.573260 secs 2024-07-26T12:21:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:32 smithi164 ceph-mon[36970]: pgmap v10897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19539.310980 secs 2024-07-26T12:21:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19521.573260 secs 2024-07-26T12:21:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:32 smithi163 ceph-mon[36873]: pgmap v10897: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19539.310980 secs 2024-07-26T12:21:34.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:34 smithi164 ceph-mon[36970]: pgmap v10898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:21:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:34 smithi163 ceph-mon[36873]: pgmap v10898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:21:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:34 smithi042 ceph-mon[32775]: pgmap v10898: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:21:36.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:36 smithi164 ceph-mon[36970]: pgmap v10899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:36 smithi163 ceph-mon[36873]: pgmap v10899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:36 smithi042 ceph-mon[32775]: pgmap v10899: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19526.573420 secs 2024-07-26T12:21:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19544.311100 secs 2024-07-26T12:21:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19526.573420 secs 2024-07-26T12:21:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19544.311100 secs 2024-07-26T12:21:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19526.573420 secs 2024-07-26T12:21:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19544.311100 secs 2024-07-26T12:21:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:38 smithi164 ceph-mon[36970]: pgmap v10900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:38 smithi163 ceph-mon[36873]: pgmap v10900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:38 smithi042 ceph-mon[32775]: pgmap v10900: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:39 smithi164 ceph-mon[36970]: pgmap v10901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:39 smithi163 ceph-mon[36873]: pgmap v10901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:39 smithi042 ceph-mon[32775]: pgmap v10901: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:42.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19531.573472 secs 2024-07-26T12:21:42.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:42 smithi163 ceph-mon[36873]: pgmap v10902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:42.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:21:42.561 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19549.311178 secs 2024-07-26T12:21:42.561 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:42.561 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:42.561 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:42.561 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:42 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:42.569 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19531.573472 secs 2024-07-26T12:21:42.569 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:42 smithi164 ceph-mon[36970]: pgmap v10902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:42.569 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:21:42.569 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19549.311178 secs 2024-07-26T12:21:42.570 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:42.570 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:42.570 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:42.570 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:42 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19531.573472 secs 2024-07-26T12:21:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:42 smithi042 ceph-mon[32775]: pgmap v10902: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:21:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19549.311178 secs 2024-07-26T12:21:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:42.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:42 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:43.615 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:43 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T12:21:43.615 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:43 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T12:21:43.622 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:43 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T12:21:43.622 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:43 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T12:21:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:43 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi042 to 177.1M 2024-07-26T12:21:43.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:43 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi042 to 185704960: error parsing value: Value '185704960' is below minimum 939524096 2024-07-26T12:21:44.288 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:21:44.287+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:21:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:21:44.287+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:21:44.287+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:21:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:44 smithi042 ceph-mon[32775]: pgmap v10903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:21:44.698 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:44 smithi164 ceph-mon[36970]: pgmap v10903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:21:44.739 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:44 smithi163 ceph-mon[36873]: pgmap v10903: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:21:46.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:21:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:21:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T12:21:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:21:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:21:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:21:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T12:21:46.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T12:21:46.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:21:46.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:21:46.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: pgmap v10904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:46.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: pgmap v10905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:46.397 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:21:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:21:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:21:46.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T12:21:46.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:21:46.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:21:46.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:21:46.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T12:21:46.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T12:21:46.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:21:46.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:21:46.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: pgmap v10904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:46.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: pgmap v10905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:46.475 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:21:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:21:46.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:21:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi164 to 176.2M 2024-07-26T12:21:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi164 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-07-26T12:21:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:21:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:21:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:46.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-07-26T12:21:46.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: Unable to set osd_memory_target on smithi163 to 184779571: error parsing value: Value '184779571' is below minimum 939524096 2024-07-26T12:21:46.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:21:46.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:21:46.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: pgmap v10904: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:21:46.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: pgmap v10905: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:46.509 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:21:47.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19536.573578 secs 2024-07-26T12:21:47.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19554.311308 secs 2024-07-26T12:21:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19536.573578 secs 2024-07-26T12:21:47.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19554.311308 secs 2024-07-26T12:21:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19536.573578 secs 2024-07-26T12:21:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19554.311308 secs 2024-07-26T12:21:48.472 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:48 smithi164 ceph-mon[36970]: pgmap v10906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:48.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:48 smithi163 ceph-mon[36873]: pgmap v10906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:48.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:48 smithi042 ceph-mon[32775]: pgmap v10906: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:50 smithi042 ceph-mon[32775]: pgmap v10907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:50 smithi164 ceph-mon[36970]: pgmap v10907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:50 smithi163 ceph-mon[36873]: pgmap v10907: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:52.270 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19541.573707 secs 2024-07-26T12:21:52.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19541.573707 secs 2024-07-26T12:21:52.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:52 smithi164 ceph-mon[36970]: pgmap v10908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:52.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19559.311436 secs 2024-07-26T12:21:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19541.573707 secs 2024-07-26T12:21:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:52 smithi042 ceph-mon[32775]: pgmap v10908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19559.311436 secs 2024-07-26T12:21:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:52 smithi163 ceph-mon[36873]: pgmap v10908: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-07-26T12:21:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19559.311436 secs 2024-07-26T12:21:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:54 smithi042 ceph-mon[32775]: pgmap v10909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:21:54.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:54 smithi164 ceph-mon[36970]: pgmap v10909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:21:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:54 smithi163 ceph-mon[36873]: pgmap v10909: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:21:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:56 smithi042 ceph-mon[32775]: pgmap v10910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:21:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:56 smithi164 ceph-mon[36970]: pgmap v10910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:21:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:56 smithi163 ceph-mon[36873]: pgmap v10910: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:21:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19546.573806 secs 2024-07-26T12:21:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19564.311583 secs 2024-07-26T12:21:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19546.573806 secs 2024-07-26T12:21:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19564.311583 secs 2024-07-26T12:21:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19546.573806 secs 2024-07-26T12:21:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19564.311583 secs 2024-07-26T12:21:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:21:58 smithi042 ceph-mon[32775]: pgmap v10911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:58.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:21:58 smithi164 ceph-mon[36970]: pgmap v10911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:21:58 smithi163 ceph-mon[36873]: pgmap v10911: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:21:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:21:59.293+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:21:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:21:59.293+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:21:59.293+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:22:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:00 smithi042 ceph-mon[32775]: pgmap v10912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:00.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:00 smithi164 ceph-mon[36970]: pgmap v10912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:00 smithi163 ceph-mon[36873]: pgmap v10912: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19551.573905 secs 2024-07-26T12:22:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:02 smithi042 ceph-mon[32775]: pgmap v10913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19569.311705 secs 2024-07-26T12:22:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19551.573905 secs 2024-07-26T12:22:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:02 smithi164 ceph-mon[36970]: pgmap v10913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19569.311705 secs 2024-07-26T12:22:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19551.573905 secs 2024-07-26T12:22:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:02 smithi163 ceph-mon[36873]: pgmap v10913: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19569.311705 secs 2024-07-26T12:22:04.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:04 smithi042 ceph-mon[32775]: pgmap v10914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:04.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:04 smithi164 ceph-mon[36970]: pgmap v10914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:04 smithi163 ceph-mon[36873]: pgmap v10914: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:06.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:06 smithi163 ceph-mon[36873]: pgmap v10915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:06 smithi042 ceph-mon[32775]: pgmap v10915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:06 smithi164 ceph-mon[36970]: pgmap v10915: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19556.574046 secs 2024-07-26T12:22:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19574.311806 secs 2024-07-26T12:22:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19556.574046 secs 2024-07-26T12:22:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19574.311806 secs 2024-07-26T12:22:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19556.574046 secs 2024-07-26T12:22:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19574.311806 secs 2024-07-26T12:22:08.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:08 smithi042 ceph-mon[32775]: pgmap v10916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:08 smithi164 ceph-mon[36970]: pgmap v10916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:08 smithi163 ceph-mon[36873]: pgmap v10916: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:10 smithi042 ceph-mon[32775]: pgmap v10917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:10 smithi164 ceph-mon[36970]: pgmap v10917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:10 smithi163 ceph-mon[36873]: pgmap v10917: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19561.574169 secs 2024-07-26T12:22:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:12 smithi042 ceph-mon[32775]: pgmap v10918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:12.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19579.311951 secs 2024-07-26T12:22:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19561.574169 secs 2024-07-26T12:22:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:12 smithi164 ceph-mon[36970]: pgmap v10918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19579.311951 secs 2024-07-26T12:22:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19561.574169 secs 2024-07-26T12:22:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:12 smithi163 ceph-mon[36873]: pgmap v10918: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19579.311951 secs 2024-07-26T12:22:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:22:14.294+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:22:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:22:14.294+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:14.296 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:22:14.294+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:22:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:14 smithi042 ceph-mon[32775]: pgmap v10919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:14.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:14 smithi164 ceph-mon[36970]: pgmap v10919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:14 smithi163 ceph-mon[36873]: pgmap v10919: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:16 smithi042 ceph-mon[32775]: pgmap v10920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:16 smithi164 ceph-mon[36970]: pgmap v10920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:16.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:16 smithi163 ceph-mon[36873]: pgmap v10920: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19566.574293 secs 2024-07-26T12:22:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19584.312075 secs 2024-07-26T12:22:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19566.574293 secs 2024-07-26T12:22:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19584.312075 secs 2024-07-26T12:22:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19566.574293 secs 2024-07-26T12:22:17.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19584.312075 secs 2024-07-26T12:22:18.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:18 smithi042 ceph-mon[32775]: pgmap v10921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:18 smithi164 ceph-mon[36970]: pgmap v10921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:18 smithi163 ceph-mon[36873]: pgmap v10921: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:20.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:20 smithi042 ceph-mon[32775]: pgmap v10922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:20.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:20 smithi164 ceph-mon[36970]: pgmap v10922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:20 smithi163 ceph-mon[36873]: pgmap v10922: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19571.574412 secs 2024-07-26T12:22:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:22 smithi042 ceph-mon[32775]: pgmap v10923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:22.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19589.312221 secs 2024-07-26T12:22:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19571.574412 secs 2024-07-26T12:22:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:22 smithi164 ceph-mon[36970]: pgmap v10923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19589.312221 secs 2024-07-26T12:22:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19571.574412 secs 2024-07-26T12:22:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:22 smithi163 ceph-mon[36873]: pgmap v10923: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19589.312221 secs 2024-07-26T12:22:24.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:24 smithi042 ceph-mon[32775]: pgmap v10924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:24.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:24 smithi164 ceph-mon[36970]: pgmap v10924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:24 smithi163 ceph-mon[36873]: pgmap v10924: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:25.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:25 smithi164 ceph-mon[36970]: pgmap v10925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:26.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:25 smithi163 ceph-mon[36873]: pgmap v10925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:26.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:25 smithi042 ceph-mon[32775]: pgmap v10925: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:26.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19576.574541 secs 2024-07-26T12:22:27.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19576.574541 secs 2024-07-26T12:22:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19576.574541 secs 2024-07-26T12:22:27.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19594.312344 secs 2024-07-26T12:22:27.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:27 smithi164 ceph-mon[36970]: pgmap v10926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19594.312344 secs 2024-07-26T12:22:28.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:27 smithi163 ceph-mon[36873]: pgmap v10926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19594.312344 secs 2024-07-26T12:22:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:27 smithi042 ceph-mon[32775]: pgmap v10926: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:22:29.300+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:22:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:22:29.300+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:22:29.300+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:22:30.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:30 smithi042 ceph-mon[32775]: pgmap v10927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:30.725 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:30 smithi164 ceph-mon[36970]: pgmap v10927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:30 smithi163 ceph-mon[36873]: pgmap v10927: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:32.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19581.574641 secs 2024-07-26T12:22:32.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:32 smithi042 ceph-mon[32775]: pgmap v10928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:32.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19599.312426 secs 2024-07-26T12:22:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19581.574641 secs 2024-07-26T12:22:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:32 smithi164 ceph-mon[36970]: pgmap v10928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19599.312426 secs 2024-07-26T12:22:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19581.574641 secs 2024-07-26T12:22:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:32 smithi163 ceph-mon[36873]: pgmap v10928: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19599.312426 secs 2024-07-26T12:22:34.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:34 smithi042 ceph-mon[32775]: pgmap v10929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:34 smithi164 ceph-mon[36970]: pgmap v10929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:34 smithi163 ceph-mon[36873]: pgmap v10929: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:36.559 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:36 smithi163 ceph-mon[36873]: pgmap v10930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:36.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:36 smithi042 ceph-mon[32775]: pgmap v10930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:36.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:36 smithi164 ceph-mon[36970]: pgmap v10930: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19586.574706 secs 2024-07-26T12:22:37.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19604.312565 secs 2024-07-26T12:22:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19586.574706 secs 2024-07-26T12:22:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19604.312565 secs 2024-07-26T12:22:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19586.574706 secs 2024-07-26T12:22:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19604.312565 secs 2024-07-26T12:22:38.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:38 smithi042 ceph-mon[32775]: pgmap v10931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:38 smithi164 ceph-mon[36970]: pgmap v10931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:38 smithi163 ceph-mon[36873]: pgmap v10931: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:40.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:40 smithi042 ceph-mon[32775]: pgmap v10932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:40.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:40 smithi164 ceph-mon[36970]: pgmap v10932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:40 smithi163 ceph-mon[36873]: pgmap v10932: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19591.574803 secs 2024-07-26T12:22:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:42 smithi042 ceph-mon[32775]: pgmap v10933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:42.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19609.312676 secs 2024-07-26T12:22:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19591.574803 secs 2024-07-26T12:22:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:42 smithi164 ceph-mon[36970]: pgmap v10933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19609.312676 secs 2024-07-26T12:22:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19591.574803 secs 2024-07-26T12:22:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:42 smithi163 ceph-mon[36873]: pgmap v10933: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:42.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19609.312676 secs 2024-07-26T12:22:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:22:44.301+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:22:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:22:44.301+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:22:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:22:44.301+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:44.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:44 smithi042 ceph-mon[32775]: pgmap v10934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:44.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:44 smithi164 ceph-mon[36970]: pgmap v10934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:44 smithi163 ceph-mon[36873]: pgmap v10934: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:46 smithi042 ceph-mon[32775]: pgmap v10935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:22:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:22:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:22:46.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:22:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:46 smithi164 ceph-mon[36970]: pgmap v10935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:22:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:22:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:22:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:22:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:46 smithi163 ceph-mon[36873]: pgmap v10935: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:22:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:22:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:22:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:22:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:22:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:47 smithi042 ceph-mon[32775]: pgmap v10936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:22:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:47 smithi042 ceph-mon[32775]: pgmap v10937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:22:47.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19596.574923 secs 2024-07-26T12:22:47.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19614.312775 secs 2024-07-26T12:22:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:47 smithi164 ceph-mon[36970]: pgmap v10936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:22:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:47 smithi164 ceph-mon[36970]: pgmap v10937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:22:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19596.574923 secs 2024-07-26T12:22:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19614.312775 secs 2024-07-26T12:22:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:47 smithi163 ceph-mon[36873]: pgmap v10936: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:22:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:47 smithi163 ceph-mon[36873]: pgmap v10937: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:22:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19596.574923 secs 2024-07-26T12:22:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19614.312775 secs 2024-07-26T12:22:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:49 smithi042 ceph-mon[32775]: pgmap v10938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:22:49.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:49 smithi164 ceph-mon[36970]: pgmap v10938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:22:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:49 smithi163 ceph-mon[36873]: pgmap v10938: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:22:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:51 smithi042 ceph-mon[32775]: pgmap v10939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:22:51.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:51 smithi164 ceph-mon[36970]: pgmap v10939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:22:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:51 smithi163 ceph-mon[36873]: pgmap v10939: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:22:52.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19601.575049 secs 2024-07-26T12:22:52.624 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19619.312884 secs 2024-07-26T12:22:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19601.575049 secs 2024-07-26T12:22:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19619.312884 secs 2024-07-26T12:22:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19601.575049 secs 2024-07-26T12:22:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19619.312884 secs 2024-07-26T12:22:53.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:53 smithi042 ceph-mon[32775]: pgmap v10940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:22:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:53 smithi164 ceph-mon[36970]: pgmap v10940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:22:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:53 smithi163 ceph-mon[36873]: pgmap v10940: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:22:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:55 smithi042 ceph-mon[32775]: pgmap v10941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:22:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:55 smithi164 ceph-mon[36970]: pgmap v10941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:22:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:55 smithi163 ceph-mon[36873]: pgmap v10941: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:22:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:57 smithi042 ceph-mon[32775]: pgmap v10942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19606.575154 secs 2024-07-26T12:22:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19624.313009 secs 2024-07-26T12:22:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:57 smithi164 ceph-mon[36970]: pgmap v10942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19606.575154 secs 2024-07-26T12:22:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19624.313009 secs 2024-07-26T12:22:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:57 smithi163 ceph-mon[36873]: pgmap v10942: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:22:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19606.575154 secs 2024-07-26T12:22:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19624.313009 secs 2024-07-26T12:22:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:22:59.307+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:22:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:22:59.307+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:22:59.307+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:22:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:22:59 smithi042 ceph-mon[32775]: pgmap v10943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:22:59 smithi164 ceph-mon[36970]: pgmap v10943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:22:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:22:59 smithi163 ceph-mon[36873]: pgmap v10943: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:00.964 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:23:00.963+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6580036c0 con 0x7ff680102000 2024-07-26T12:23:00.964 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:23:00.963+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:23:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:23:00.963+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 153 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:23:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:23:00.963+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 153 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff6700e9050 con 0x7ff680102000 2024-07-26T12:23:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:01 smithi042 ceph-mon[32775]: pgmap v10944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:01 smithi164 ceph-mon[36970]: pgmap v10944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:01 smithi163 ceph-mon[36873]: pgmap v10944: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19611.575288 secs 2024-07-26T12:23:02.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19629.313136 secs 2024-07-26T12:23:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19611.575288 secs 2024-07-26T12:23:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19629.313136 secs 2024-07-26T12:23:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19611.575288 secs 2024-07-26T12:23:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19629.313136 secs 2024-07-26T12:23:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:03 smithi042 ceph-mon[32775]: pgmap v10945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:03 smithi164 ceph-mon[36970]: pgmap v10945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:03 smithi163 ceph-mon[36873]: pgmap v10945: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:05.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:05 smithi042 ceph-mon[32775]: pgmap v10946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:23:05.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:05 smithi164 ceph-mon[36970]: pgmap v10946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:23:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:05 smithi163 ceph-mon[36873]: pgmap v10946: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:23:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:07 smithi042 ceph-mon[32775]: pgmap v10947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19616.575405 secs 2024-07-26T12:23:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19634.313237 secs 2024-07-26T12:23:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:07 smithi164 ceph-mon[36970]: pgmap v10947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19616.575405 secs 2024-07-26T12:23:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19634.313237 secs 2024-07-26T12:23:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:07 smithi163 ceph-mon[36873]: pgmap v10947: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19616.575405 secs 2024-07-26T12:23:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19634.313237 secs 2024-07-26T12:23:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:09 smithi042 ceph-mon[32775]: pgmap v10948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:09 smithi164 ceph-mon[36970]: pgmap v10948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:09 smithi163 ceph-mon[36873]: pgmap v10948: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:11 smithi042 ceph-mon[32775]: pgmap v10949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:11.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:11 smithi164 ceph-mon[36970]: pgmap v10949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:11 smithi163 ceph-mon[36873]: pgmap v10949: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19621.575550 secs 2024-07-26T12:23:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19639.313360 secs 2024-07-26T12:23:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19621.575550 secs 2024-07-26T12:23:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19639.313360 secs 2024-07-26T12:23:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19621.575550 secs 2024-07-26T12:23:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19639.313360 secs 2024-07-26T12:23:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:13 smithi042 ceph-mon[32775]: pgmap v10950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:13.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:13 smithi164 ceph-mon[36970]: pgmap v10950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:13 smithi163 ceph-mon[36873]: pgmap v10950: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:23:14.308+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:23:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:23:14.308+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:23:14.308+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:23:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:15 smithi042 ceph-mon[32775]: pgmap v10951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:23:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:15 smithi164 ceph-mon[36970]: pgmap v10951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:23:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:15 smithi163 ceph-mon[36873]: pgmap v10951: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:23:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:17 smithi042 ceph-mon[32775]: pgmap v10952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19626.575696 secs 2024-07-26T12:23:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19644.313497 secs 2024-07-26T12:23:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:17 smithi164 ceph-mon[36970]: pgmap v10952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19626.575696 secs 2024-07-26T12:23:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19644.313497 secs 2024-07-26T12:23:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:17 smithi163 ceph-mon[36873]: pgmap v10952: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19626.575696 secs 2024-07-26T12:23:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19644.313497 secs 2024-07-26T12:23:19.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:19 smithi042 ceph-mon[32775]: pgmap v10953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:19.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:19 smithi164 ceph-mon[36970]: pgmap v10953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:19 smithi163 ceph-mon[36873]: pgmap v10953: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:21.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:21 smithi042 ceph-mon[32775]: pgmap v10954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:21.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:21 smithi164 ceph-mon[36970]: pgmap v10954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:21 smithi163 ceph-mon[36873]: pgmap v10954: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19631.575816 secs 2024-07-26T12:23:22.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19649.313624 secs 2024-07-26T12:23:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19631.575816 secs 2024-07-26T12:23:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19649.313624 secs 2024-07-26T12:23:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19631.575816 secs 2024-07-26T12:23:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19649.313624 secs 2024-07-26T12:23:23.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:23 smithi042 ceph-mon[32775]: pgmap v10955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:23.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:23 smithi164 ceph-mon[36970]: pgmap v10955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:23 smithi163 ceph-mon[36873]: pgmap v10955: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:25.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:25 smithi042 ceph-mon[32775]: pgmap v10956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:23:25.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:25 smithi164 ceph-mon[36970]: pgmap v10956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:23:25.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:25 smithi163 ceph-mon[36873]: pgmap v10956: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:23:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:27 smithi042 ceph-mon[32775]: pgmap v10957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19636.575915 secs 2024-07-26T12:23:27.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19654.313747 secs 2024-07-26T12:23:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:27 smithi164 ceph-mon[36970]: pgmap v10957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19636.575915 secs 2024-07-26T12:23:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19654.313747 secs 2024-07-26T12:23:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:27 smithi163 ceph-mon[36873]: pgmap v10957: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19636.575915 secs 2024-07-26T12:23:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19654.313747 secs 2024-07-26T12:23:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:23:29.313+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:23:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:23:29.313+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:23:29.313+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:23:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:29 smithi164 ceph-mon[36970]: pgmap v10958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:29 smithi163 ceph-mon[36873]: pgmap v10958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:29 smithi042 ceph-mon[32775]: pgmap v10958: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:31.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:31 smithi164 ceph-mon[36970]: pgmap v10959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:31.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:31 smithi163 ceph-mon[36873]: pgmap v10959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:31 smithi042 ceph-mon[32775]: pgmap v10959: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19641.576038 secs 2024-07-26T12:23:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19659.313869 secs 2024-07-26T12:23:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19641.576038 secs 2024-07-26T12:23:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19659.313869 secs 2024-07-26T12:23:32.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19641.576038 secs 2024-07-26T12:23:32.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19659.313869 secs 2024-07-26T12:23:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:33 smithi164 ceph-mon[36970]: pgmap v10960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:33 smithi163 ceph-mon[36873]: pgmap v10960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:33 smithi042 ceph-mon[32775]: pgmap v10960: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:35.726 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:35 smithi164 ceph-mon[36970]: pgmap v10961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:23:35.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:35 smithi163 ceph-mon[36873]: pgmap v10961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:23:35.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:35 smithi042 ceph-mon[32775]: pgmap v10961: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:23:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:37 smithi164 ceph-mon[36970]: pgmap v10962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19646.576138 secs 2024-07-26T12:23:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19664.314030 secs 2024-07-26T12:23:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:37 smithi163 ceph-mon[36873]: pgmap v10962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19646.576138 secs 2024-07-26T12:23:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19664.314030 secs 2024-07-26T12:23:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:37 smithi042 ceph-mon[32775]: pgmap v10962: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19646.576138 secs 2024-07-26T12:23:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19664.314030 secs 2024-07-26T12:23:39.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:39 smithi164 ceph-mon[36970]: pgmap v10963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:39 smithi163 ceph-mon[36873]: pgmap v10963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:39 smithi042 ceph-mon[32775]: pgmap v10963: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:41.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:41 smithi164 ceph-mon[36970]: pgmap v10964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:41.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:41 smithi163 ceph-mon[36873]: pgmap v10964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:41.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:41 smithi042 ceph-mon[32775]: pgmap v10964: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19651.576282 secs 2024-07-26T12:23:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19669.314150 secs 2024-07-26T12:23:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19651.576282 secs 2024-07-26T12:23:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19669.314150 secs 2024-07-26T12:23:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19651.576282 secs 2024-07-26T12:23:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19669.314150 secs 2024-07-26T12:23:43.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:43 smithi164 ceph-mon[36970]: pgmap v10965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:43.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:43 smithi163 ceph-mon[36873]: pgmap v10965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:43.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:43 smithi042 ceph-mon[32775]: pgmap v10965: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:23:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:23:44.315+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:23:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:23:44.315+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:23:44.315+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:23:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:45 smithi164 ceph-mon[36970]: pgmap v10966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:23:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:45 smithi163 ceph-mon[36873]: pgmap v10966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:23:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:45 smithi042 ceph-mon[32775]: pgmap v10966: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:23:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:23:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:23:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:23:46.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:46 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:23:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:23:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:23:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:23:46.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:46 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:23:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:23:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:23:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:23:46.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:46 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:23:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:47 smithi164 ceph-mon[36970]: pgmap v10967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19656.576377 secs 2024-07-26T12:23:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:47 smithi164 ceph-mon[36970]: pgmap v10968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:23:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:47 smithi164 ceph-mon[36970]: pgmap v10969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:23:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19674.314248 secs 2024-07-26T12:23:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:47 smithi163 ceph-mon[36873]: pgmap v10967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19656.576377 secs 2024-07-26T12:23:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:47 smithi163 ceph-mon[36873]: pgmap v10968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:23:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:47 smithi163 ceph-mon[36873]: pgmap v10969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:23:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19674.314248 secs 2024-07-26T12:23:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:47 smithi042 ceph-mon[32775]: pgmap v10967: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:23:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19656.576377 secs 2024-07-26T12:23:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:47 smithi042 ceph-mon[32775]: pgmap v10968: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:23:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:47 smithi042 ceph-mon[32775]: pgmap v10969: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:23:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19674.314248 secs 2024-07-26T12:23:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:48 smithi163 ceph-mon[36873]: pgmap v10970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:23:48.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:48 smithi042 ceph-mon[32775]: pgmap v10970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:23:48.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:48 smithi164 ceph-mon[36970]: pgmap v10970: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:23:51.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:51 smithi042 ceph-mon[32775]: pgmap v10971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:23:51.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:51 smithi164 ceph-mon[36970]: pgmap v10971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:23:51.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:51 smithi163 ceph-mon[36873]: pgmap v10971: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:23:52.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19661.576494 secs 2024-07-26T12:23:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19679.314382 secs 2024-07-26T12:23:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19661.576494 secs 2024-07-26T12:23:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19679.314382 secs 2024-07-26T12:23:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19661.576494 secs 2024-07-26T12:23:52.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19679.314382 secs 2024-07-26T12:23:53.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:53 smithi164 ceph-mon[36970]: pgmap v10972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:23:53.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:53 smithi163 ceph-mon[36873]: pgmap v10972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:23:53.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:53 smithi042 ceph-mon[32775]: pgmap v10972: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:23:55.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:55 smithi164 ceph-mon[36970]: pgmap v10973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:23:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:55 smithi163 ceph-mon[36873]: pgmap v10973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:23:55.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:55 smithi042 ceph-mon[32775]: pgmap v10973: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:23:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19666.576623 secs 2024-07-26T12:23:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:57 smithi164 ceph-mon[36970]: pgmap v10974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:23:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19684.314524 secs 2024-07-26T12:23:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19666.576623 secs 2024-07-26T12:23:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:57 smithi163 ceph-mon[36873]: pgmap v10974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:23:57.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19684.314524 secs 2024-07-26T12:23:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19666.576623 secs 2024-07-26T12:23:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:57 smithi042 ceph-mon[32775]: pgmap v10974: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:23:57.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19684.314524 secs 2024-07-26T12:23:59.322 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:23:59.321+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:23:59.323 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:23:59.321+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:59.323 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:23:59.321+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:23:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:23:59 smithi164 ceph-mon[36970]: pgmap v10975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:23:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:23:59 smithi163 ceph-mon[36873]: pgmap v10975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:23:59.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:23:59 smithi042 ceph-mon[32775]: pgmap v10975: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:24:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:01 smithi164 ceph-mon[36970]: pgmap v10976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:01 smithi163 ceph-mon[36873]: pgmap v10976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:01.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:01 smithi042 ceph-mon[32775]: pgmap v10976: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19671.576747 secs 2024-07-26T12:24:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19689.314654 secs 2024-07-26T12:24:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19671.576747 secs 2024-07-26T12:24:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19689.314654 secs 2024-07-26T12:24:02.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19671.576747 secs 2024-07-26T12:24:02.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19689.314654 secs 2024-07-26T12:24:03.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:03 smithi164 ceph-mon[36970]: pgmap v10977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:03 smithi163 ceph-mon[36873]: pgmap v10977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:03.824 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:03 smithi042 ceph-mon[32775]: pgmap v10977: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:05 smithi164 ceph-mon[36970]: pgmap v10978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:05.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:05 smithi163 ceph-mon[36873]: pgmap v10978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:05.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:05 smithi042 ceph-mon[32775]: pgmap v10978: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19676.576890 secs 2024-07-26T12:24:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:07 smithi164 ceph-mon[36970]: pgmap v10979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19694.314768 secs 2024-07-26T12:24:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19676.576890 secs 2024-07-26T12:24:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:07 smithi163 ceph-mon[36873]: pgmap v10979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:07.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19694.314768 secs 2024-07-26T12:24:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19676.576890 secs 2024-07-26T12:24:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:07 smithi042 ceph-mon[32775]: pgmap v10979: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:07.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19694.314768 secs 2024-07-26T12:24:09.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:09 smithi164 ceph-mon[36970]: pgmap v10980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:24:09.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:09 smithi163 ceph-mon[36873]: pgmap v10980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:24:09.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:09 smithi042 ceph-mon[32775]: pgmap v10980: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:24:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:11 smithi164 ceph-mon[36970]: pgmap v10981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:11 smithi163 ceph-mon[36873]: pgmap v10981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:11.897 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:11 smithi042 ceph-mon[32775]: pgmap v10981: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19681.577031 secs 2024-07-26T12:24:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19699.314889 secs 2024-07-26T12:24:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19681.577031 secs 2024-07-26T12:24:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19699.314889 secs 2024-07-26T12:24:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19681.577031 secs 2024-07-26T12:24:12.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19699.314889 secs 2024-07-26T12:24:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:13 smithi164 ceph-mon[36970]: pgmap v10982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:13 smithi163 ceph-mon[36873]: pgmap v10982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:13.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:13 smithi042 ceph-mon[32775]: pgmap v10982: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:24:14.322+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:24:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:24:14.323+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:24:14.323+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:24:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:14 smithi163 ceph-mon[36873]: pgmap v10983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:14.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:14 smithi042 ceph-mon[32775]: pgmap v10983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:14.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:14 smithi164 ceph-mon[36970]: pgmap v10983: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19686.577175 secs 2024-07-26T12:24:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:17 smithi164 ceph-mon[36970]: pgmap v10984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19704.315013 secs 2024-07-26T12:24:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19686.577175 secs 2024-07-26T12:24:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:17 smithi163 ceph-mon[36873]: pgmap v10984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19704.315013 secs 2024-07-26T12:24:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19686.577175 secs 2024-07-26T12:24:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:17 smithi042 ceph-mon[32775]: pgmap v10984: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:17.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19704.315013 secs 2024-07-26T12:24:19.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:19 smithi042 ceph-mon[32775]: pgmap v10985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:24:19.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:19 smithi164 ceph-mon[36970]: pgmap v10985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:24:19.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:19 smithi163 ceph-mon[36873]: pgmap v10985: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:24:21.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:21 smithi164 ceph-mon[36970]: pgmap v10986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:21.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:21 smithi163 ceph-mon[36873]: pgmap v10986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:21.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:21 smithi042 ceph-mon[32775]: pgmap v10986: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19691.577288 secs 2024-07-26T12:24:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19709.315133 secs 2024-07-26T12:24:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19691.577288 secs 2024-07-26T12:24:22.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19709.315133 secs 2024-07-26T12:24:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19691.577288 secs 2024-07-26T12:24:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19709.315133 secs 2024-07-26T12:24:23.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:23 smithi164 ceph-mon[36970]: pgmap v10987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:23.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:23 smithi163 ceph-mon[36873]: pgmap v10987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:23.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:23 smithi042 ceph-mon[32775]: pgmap v10987: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:25.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:25 smithi164 ceph-mon[36970]: pgmap v10988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:25.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:25 smithi163 ceph-mon[36873]: pgmap v10988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:25.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:25 smithi042 ceph-mon[32775]: pgmap v10988: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19696.577374 secs 2024-07-26T12:24:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:27 smithi164 ceph-mon[36970]: pgmap v10989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19714.315252 secs 2024-07-26T12:24:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19696.577374 secs 2024-07-26T12:24:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:27 smithi163 ceph-mon[36873]: pgmap v10989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:27.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19714.315252 secs 2024-07-26T12:24:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19696.577374 secs 2024-07-26T12:24:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:27 smithi042 ceph-mon[32775]: pgmap v10989: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:27.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19714.315252 secs 2024-07-26T12:24:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:24:29.329+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:24:29.330 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:24:29.329+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:29.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:24:29.329+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:24:29.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:29 smithi164 ceph-mon[36970]: pgmap v10990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:24:29.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:29 smithi163 ceph-mon[36873]: pgmap v10990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:24:29.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:29 smithi042 ceph-mon[32775]: pgmap v10990: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:24:31.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:31 smithi164 ceph-mon[36970]: pgmap v10991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:31.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:31 smithi163 ceph-mon[36873]: pgmap v10991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:31.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:31 smithi042 ceph-mon[32775]: pgmap v10991: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19701.577461 secs 2024-07-26T12:24:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19719.315373 secs 2024-07-26T12:24:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19701.577461 secs 2024-07-26T12:24:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19719.315373 secs 2024-07-26T12:24:32.796 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19701.577461 secs 2024-07-26T12:24:32.797 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19719.315373 secs 2024-07-26T12:24:33.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:33 smithi164 ceph-mon[36970]: pgmap v10992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:33.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:33 smithi163 ceph-mon[36873]: pgmap v10992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:33.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:33 smithi042 ceph-mon[32775]: pgmap v10992: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:35.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:35 smithi164 ceph-mon[36970]: pgmap v10993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:35.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:35 smithi163 ceph-mon[36873]: pgmap v10993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:35.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:35 smithi042 ceph-mon[32775]: pgmap v10993: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19706.577587 secs 2024-07-26T12:24:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:37 smithi164 ceph-mon[36970]: pgmap v10994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19724.315490 secs 2024-07-26T12:24:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19706.577587 secs 2024-07-26T12:24:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:37 smithi163 ceph-mon[36873]: pgmap v10994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19724.315490 secs 2024-07-26T12:24:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19706.577587 secs 2024-07-26T12:24:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:37 smithi042 ceph-mon[32775]: pgmap v10994: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19724.315490 secs 2024-07-26T12:24:39.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:39 smithi164 ceph-mon[36970]: pgmap v10995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:24:39.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:39 smithi163 ceph-mon[36873]: pgmap v10995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:24:39.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:39 smithi042 ceph-mon[32775]: pgmap v10995: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:24:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:40 smithi163 ceph-mon[36873]: pgmap v10996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:40 smithi042 ceph-mon[32775]: pgmap v10996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:40.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:40 smithi164 ceph-mon[36970]: pgmap v10996: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:41.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19711.577705 secs 2024-07-26T12:24:42.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19711.577705 secs 2024-07-26T12:24:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19711.577705 secs 2024-07-26T12:24:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19729.315626 secs 2024-07-26T12:24:42.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:42 smithi164 ceph-mon[36970]: pgmap v10997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19729.315626 secs 2024-07-26T12:24:43.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:42 smithi163 ceph-mon[36873]: pgmap v10997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19729.315626 secs 2024-07-26T12:24:43.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:42 smithi042 ceph-mon[32775]: pgmap v10997: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:24:44.330+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:24:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:24:44.330+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:24:44.330+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:24:45.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:45 smithi164 ceph-mon[36970]: pgmap v10998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:45.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:45 smithi163 ceph-mon[36873]: pgmap v10998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:45.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:45 smithi042 ceph-mon[32775]: pgmap v10998: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:24:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19716.577859 secs 2024-07-26T12:24:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:47 smithi164 ceph-mon[36970]: pgmap v10999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:47 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:24:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19734.315721 secs 2024-07-26T12:24:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19716.577859 secs 2024-07-26T12:24:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:47 smithi163 ceph-mon[36873]: pgmap v10999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:47 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:24:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19734.315721 secs 2024-07-26T12:24:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19716.577859 secs 2024-07-26T12:24:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:47 smithi042 ceph-mon[32775]: pgmap v10999: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:24:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:47 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:24:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19734.315721 secs 2024-07-26T12:24:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:49 smithi164 ceph-mon[36970]: pgmap v11000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:24:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.473 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:24:49.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:24:49.474 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.507 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:49 smithi163 ceph-mon[36873]: pgmap v11000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:24:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:24:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:24:49.508 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:49 smithi042 ceph-mon[32775]: pgmap v11000: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:24:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:49.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:24:49.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:24:49.647 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:24:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:50 smithi042 ceph-mon[32775]: pgmap v11001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:24:50.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:50 smithi042 ceph-mon[32775]: pgmap v11002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:24:50.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:50 smithi164 ceph-mon[36970]: pgmap v11001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:24:50.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:50 smithi164 ceph-mon[36970]: pgmap v11002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:24:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:50 smithi163 ceph-mon[36873]: pgmap v11001: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:24:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:50 smithi163 ceph-mon[36873]: pgmap v11002: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 3 op/s 2024-07-26T12:24:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:52 smithi042 ceph-mon[32775]: pgmap v11003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:24:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19721.578005 secs 2024-07-26T12:24:52.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19739.315849 secs 2024-07-26T12:24:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:52 smithi164 ceph-mon[36970]: pgmap v11003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:24:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19721.578005 secs 2024-07-26T12:24:52.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19739.315849 secs 2024-07-26T12:24:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:52 smithi163 ceph-mon[36873]: pgmap v11003: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:24:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19721.578005 secs 2024-07-26T12:24:52.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19739.315849 secs 2024-07-26T12:24:54.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:54 smithi042 ceph-mon[32775]: pgmap v11004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:24:54.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:54 smithi164 ceph-mon[36970]: pgmap v11004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:24:54.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:54 smithi163 ceph-mon[36873]: pgmap v11004: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:24:56.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:56 smithi042 ceph-mon[32775]: pgmap v11005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:24:56.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:56 smithi164 ceph-mon[36970]: pgmap v11005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:24:56.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:56 smithi163 ceph-mon[36873]: pgmap v11005: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-07-26T12:24:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19726.578149 secs 2024-07-26T12:24:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19744.315974 secs 2024-07-26T12:24:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19726.578149 secs 2024-07-26T12:24:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19744.315974 secs 2024-07-26T12:24:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19726.578149 secs 2024-07-26T12:24:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19744.315974 secs 2024-07-26T12:24:58.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:24:58 smithi042 ceph-mon[32775]: pgmap v11006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T12:24:58.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:24:58 smithi164 ceph-mon[36970]: pgmap v11006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T12:24:58.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:24:58 smithi163 ceph-mon[36873]: pgmap v11006: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 3 op/s 2024-07-26T12:24:59.337 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:24:59.336+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:24:59.337 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:24:59.336+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:59.337 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:24:59.336+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:25:00.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:00 smithi042 ceph-mon[32775]: pgmap v11007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:25:00.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:00 smithi164 ceph-mon[36970]: pgmap v11007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:25:00.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:00 smithi163 ceph-mon[36873]: pgmap v11007: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:25:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:02 smithi042 ceph-mon[32775]: pgmap v11008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:02.395 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19731.578249 secs 2024-07-26T12:25:02.396 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19749.316093 secs 2024-07-26T12:25:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:02 smithi164 ceph-mon[36970]: pgmap v11008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19731.578249 secs 2024-07-26T12:25:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19749.316093 secs 2024-07-26T12:25:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:02 smithi163 ceph-mon[36873]: pgmap v11008: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19731.578249 secs 2024-07-26T12:25:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19749.316093 secs 2024-07-26T12:25:04.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:04 smithi042 ceph-mon[32775]: pgmap v11009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:04.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:04 smithi164 ceph-mon[36970]: pgmap v11009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:04.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:04 smithi163 ceph-mon[36873]: pgmap v11009: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:06.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:06 smithi042 ceph-mon[32775]: pgmap v11010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:06.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:06 smithi164 ceph-mon[36970]: pgmap v11010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:06.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:06 smithi163 ceph-mon[36873]: pgmap v11010: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19736.578378 secs 2024-07-26T12:25:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19754.316217 secs 2024-07-26T12:25:07.658 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19736.578378 secs 2024-07-26T12:25:07.659 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19754.316217 secs 2024-07-26T12:25:07.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19736.578378 secs 2024-07-26T12:25:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19754.316217 secs 2024-07-26T12:25:08.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:08 smithi042 ceph-mon[32775]: pgmap v11011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:08.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:08 smithi164 ceph-mon[36970]: pgmap v11011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:08.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:08 smithi163 ceph-mon[36873]: pgmap v11011: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:10.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:10 smithi042 ceph-mon[32775]: pgmap v11012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:25:10.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:10 smithi164 ceph-mon[36970]: pgmap v11012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:25:10.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:10 smithi163 ceph-mon[36873]: pgmap v11012: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:25:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:12 smithi042 ceph-mon[32775]: pgmap v11013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19741.578503 secs 2024-07-26T12:25:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19759.316282 secs 2024-07-26T12:25:12.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:12 smithi164 ceph-mon[36970]: pgmap v11013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19741.578503 secs 2024-07-26T12:25:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19759.316282 secs 2024-07-26T12:25:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:12 smithi163 ceph-mon[36873]: pgmap v11013: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19741.578503 secs 2024-07-26T12:25:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19759.316282 secs 2024-07-26T12:25:14.338 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:14.337+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:25:14.338 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:14.337+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:14.338 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:14.337+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:25:14.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:14 smithi042 ceph-mon[32775]: pgmap v11014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:14.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:14 smithi164 ceph-mon[36970]: pgmap v11014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:14.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:14 smithi163 ceph-mon[36873]: pgmap v11014: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:16.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:16 smithi042 ceph-mon[32775]: pgmap v11015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:16.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:16 smithi164 ceph-mon[36970]: pgmap v11015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:16.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:16 smithi163 ceph-mon[36873]: pgmap v11015: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19746.578626 secs 2024-07-26T12:25:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19764.316408 secs 2024-07-26T12:25:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19746.578626 secs 2024-07-26T12:25:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19764.316408 secs 2024-07-26T12:25:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19746.578626 secs 2024-07-26T12:25:17.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19764.316408 secs 2024-07-26T12:25:18.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:18 smithi164 ceph-mon[36970]: pgmap v11016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:18.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:18 smithi163 ceph-mon[36873]: pgmap v11016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:18.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:18 smithi042 ceph-mon[32775]: pgmap v11016: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:20.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:20 smithi164 ceph-mon[36970]: pgmap v11017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:25:20.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:20 smithi163 ceph-mon[36873]: pgmap v11017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:25:20.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:20 smithi042 ceph-mon[32775]: pgmap v11017: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:25:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:22 smithi164 ceph-mon[36970]: pgmap v11018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19751.578753 secs 2024-07-26T12:25:22.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19769.316561 secs 2024-07-26T12:25:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:22 smithi163 ceph-mon[36873]: pgmap v11018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19751.578753 secs 2024-07-26T12:25:22.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19769.316561 secs 2024-07-26T12:25:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:22 smithi042 ceph-mon[32775]: pgmap v11018: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19751.578753 secs 2024-07-26T12:25:22.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19769.316561 secs 2024-07-26T12:25:24.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:24 smithi164 ceph-mon[36970]: pgmap v11019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:24.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:24 smithi163 ceph-mon[36873]: pgmap v11019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:24.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:24 smithi042 ceph-mon[32775]: pgmap v11019: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:26.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:26 smithi164 ceph-mon[36970]: pgmap v11020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:26.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:26 smithi163 ceph-mon[36873]: pgmap v11020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:26.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:26 smithi042 ceph-mon[32775]: pgmap v11020: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19756.578884 secs 2024-07-26T12:25:27.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19774.316687 secs 2024-07-26T12:25:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19756.578884 secs 2024-07-26T12:25:27.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19774.316687 secs 2024-07-26T12:25:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19756.578884 secs 2024-07-26T12:25:27.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19774.316687 secs 2024-07-26T12:25:28.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:28 smithi164 ceph-mon[36970]: pgmap v11021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:28.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:28 smithi163 ceph-mon[36873]: pgmap v11021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:28.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:28 smithi042 ceph-mon[32775]: pgmap v11021: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:29.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:29.342+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:25:29.343 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:29.342+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:29.344 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:29.342+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:25:30.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:30 smithi164 ceph-mon[36970]: pgmap v11022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:25:30.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:30 smithi163 ceph-mon[36873]: pgmap v11022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:25:30.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:30 smithi042 ceph-mon[32775]: pgmap v11022: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:25:32.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:32 smithi042 ceph-mon[32775]: pgmap v11023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:32.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19761.578946 secs 2024-07-26T12:25:32.698 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19779.316813 secs 2024-07-26T12:25:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:32 smithi164 ceph-mon[36970]: pgmap v11023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19761.578946 secs 2024-07-26T12:25:32.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19779.316813 secs 2024-07-26T12:25:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:32 smithi163 ceph-mon[36873]: pgmap v11023: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:32.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19761.578946 secs 2024-07-26T12:25:32.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19779.316813 secs 2024-07-26T12:25:34.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:34 smithi164 ceph-mon[36970]: pgmap v11024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:34.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:34 smithi163 ceph-mon[36873]: pgmap v11024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:34.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:34 smithi042 ceph-mon[32775]: pgmap v11024: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:36.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:36 smithi164 ceph-mon[36970]: pgmap v11025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:36.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:36 smithi163 ceph-mon[36873]: pgmap v11025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:36.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:36 smithi042 ceph-mon[32775]: pgmap v11025: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:37.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19766.579072 secs 2024-07-26T12:25:37.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19784.316963 secs 2024-07-26T12:25:37.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19766.579072 secs 2024-07-26T12:25:37.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19784.316963 secs 2024-07-26T12:25:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19766.579072 secs 2024-07-26T12:25:37.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19784.316963 secs 2024-07-26T12:25:38.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:38 smithi164 ceph-mon[36970]: pgmap v11026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:38.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:38 smithi163 ceph-mon[36873]: pgmap v11026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:38.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:38 smithi042 ceph-mon[32775]: pgmap v11026: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:40.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:40 smithi164 ceph-mon[36970]: pgmap v11027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:25:40.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:40 smithi163 ceph-mon[36873]: pgmap v11027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:25:40.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:40 smithi042 ceph-mon[32775]: pgmap v11027: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:25:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:42 smithi164 ceph-mon[36970]: pgmap v11028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19771.579195 secs 2024-07-26T12:25:42.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19789.317107 secs 2024-07-26T12:25:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:42 smithi163 ceph-mon[36873]: pgmap v11028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19771.579195 secs 2024-07-26T12:25:42.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19789.317107 secs 2024-07-26T12:25:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:42 smithi042 ceph-mon[32775]: pgmap v11028: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:42.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19771.579195 secs 2024-07-26T12:25:42.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19789.317107 secs 2024-07-26T12:25:44.344 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:44.343+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:25:44.344 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:44.343+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:44.344 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:44.343+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:25:44.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:44 smithi164 ceph-mon[36970]: pgmap v11029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:44.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:44 smithi163 ceph-mon[36873]: pgmap v11029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:44.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:44 smithi042 ceph-mon[32775]: pgmap v11029: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:46.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:46 smithi164 ceph-mon[36970]: pgmap v11030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:46.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:46 smithi163 ceph-mon[36873]: pgmap v11030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:46.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:46 smithi042 ceph-mon[32775]: pgmap v11030: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:25:47.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19776.579328 secs 2024-07-26T12:25:47.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19794.317174 secs 2024-07-26T12:25:47.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19776.579328 secs 2024-07-26T12:25:47.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19794.317174 secs 2024-07-26T12:25:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19776.579328 secs 2024-07-26T12:25:47.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19794.317174 secs 2024-07-26T12:25:48.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:48 smithi164 ceph-mon[36970]: pgmap v11031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:48.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:48 smithi163 ceph-mon[36873]: pgmap v11031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:48.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:48 smithi042 ceph-mon[32775]: pgmap v11031: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:25:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:49 smithi164 ceph-mon[36970]: pgmap v11032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:25:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:25:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:49.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:49 smithi163 ceph-mon[36873]: pgmap v11032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:25:49.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:25:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:49.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:49 smithi042 ceph-mon[32775]: pgmap v11032: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:25:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:25:49.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:49.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:50.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:25:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:25:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:50 smithi163 ceph-mon[36873]: pgmap v11033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T12:25:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:50 smithi163 ceph-mon[36873]: pgmap v11034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:25:50.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:50.895 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:25:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:25:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:50 smithi042 ceph-mon[32775]: pgmap v11033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T12:25:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:50 smithi042 ceph-mon[32775]: pgmap v11034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:25:50.896 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:50.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:25:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:25:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:50 smithi164 ceph-mon[36970]: pgmap v11033: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-07-26T12:25:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:50 smithi164 ceph-mon[36970]: pgmap v11034: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:25:50.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:25:51.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19781.579453 secs 2024-07-26T12:25:52.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19781.579453 secs 2024-07-26T12:25:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19781.579453 secs 2024-07-26T12:25:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19799.317295 secs 2024-07-26T12:25:52.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:52 smithi164 ceph-mon[36970]: pgmap v11035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:25:53.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19799.317295 secs 2024-07-26T12:25:53.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:52 smithi163 ceph-mon[36873]: pgmap v11035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:25:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19799.317295 secs 2024-07-26T12:25:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:52 smithi042 ceph-mon[32775]: pgmap v11035: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:25:54.973 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:54 smithi164 ceph-mon[36970]: pgmap v11036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T12:25:55.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:54 smithi163 ceph-mon[36873]: pgmap v11036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T12:25:55.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:54 smithi042 ceph-mon[32775]: pgmap v11036: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 3 op/s 2024-07-26T12:25:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:56 smithi163 ceph-mon[36873]: pgmap v11037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:25:57.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:56 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19786.579575 secs 2024-07-26T12:25:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:56 smithi042 ceph-mon[32775]: pgmap v11037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:25:57.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:56 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19786.579575 secs 2024-07-26T12:25:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:56 smithi164 ceph-mon[36970]: pgmap v11037: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:25:57.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:56 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19786.579575 secs 2024-07-26T12:25:57.894 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:57.893+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T12:25:57.894 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:57.893+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:25:57.895 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:57.893+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff678009a60 tx=0x7ff678055070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:25:57.895 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:57.893+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff678009a60 tx=0x7ff678055070 comp rx=0 tx=0).stop 2024-07-26T12:25:57.895 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:57.893+0000 7ff67ffff640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.42:6800/2719631589, fault. 2024-07-26T12:25:57.895 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:57.893+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 msgr2=0x7ff66c0bfeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:25:57.895 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:57.893+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0bda90 0x7ff66c0bfeb0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:25:57.895 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:57.893+0000 7ff67cff9640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:25:57.895 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:57.893+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:25:57.895 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:57.894+0000 7ff67ffff640 1 --2- 172.21.15.42:0/2402090969 >> [v2:172.21.15.42:6800/2719631589,v1:172.21.15.42:6801/2719631589] conn(0x7ff66c0b4840 0x7ff66c0b4fe0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff6780094b0 tx=0x7ff67803d000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:25:58.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19804.317394 secs 2024-07-26T12:25:58.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19804.317394 secs 2024-07-26T12:25:58.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19804.317394 secs 2024-07-26T12:25:59.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:25:58 smithi163 ceph-mon[36873]: pgmap v11038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:25:59.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:25:58 smithi042 ceph-mon[32775]: pgmap v11038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:25:59.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:25:58 smithi164 ceph-mon[36970]: pgmap v11038: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:25:59.346 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:59.345+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:25:59.346 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:59.345+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:59.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:25:59.345+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:26:00.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:00.992+0000 7ff6777fe640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6580036c0 con 0x7ff680102000 2024-07-26T12:26:00.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:00.992+0000 7ff65dffb640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:26:00.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:00.992+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 154 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7ff6500064b0 con 0x55db6e0564d0 2024-07-26T12:26:00.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:00.992+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T12:26:00.994 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:00.992+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 154 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7ff6700e9050 con 0x7ff680102000 2024-07-26T12:26:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:00.992+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff658006480 con 0x7ff680102000 2024-07-26T12:26:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:00.992+0000 7ff65effd640 1 -- 172.21.15.42:0/1831263443 <== mon.0 v2:172.21.15.42:3300/0 155 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff650003690 con 0x55db6e0564d0 2024-07-26T12:26:00.995 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:00.992+0000 7ff67cff9640 1 -- 172.21.15.42:0/2402090969 <== mon.0 v2:172.21.15.42:3300/0 155 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff658006480 con 0x7ff680102000 2024-07-26T12:26:01.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:00 smithi163 ceph-mon[36873]: pgmap v11039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:26:01.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:00 smithi042 ceph-mon[32775]: pgmap v11039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:26:01.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:00 smithi164 ceph-mon[36970]: pgmap v11039: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:26:02.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:01 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19791.579703 secs 2024-07-26T12:26:02.046 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:01 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19791.579703 secs 2024-07-26T12:26:02.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:01 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19791.579703 secs 2024-07-26T12:26:03.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19809.317521 secs 2024-07-26T12:26:03.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:02 smithi163 ceph-mon[36873]: pgmap v11040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19809.317521 secs 2024-07-26T12:26:03.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:02 smithi042 ceph-mon[32775]: pgmap v11040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19809.317521 secs 2024-07-26T12:26:03.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:02 smithi164 ceph-mon[36970]: pgmap v11040: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:05.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:04 smithi163 ceph-mon[36873]: pgmap v11041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:26:05.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:04 smithi042 ceph-mon[32775]: pgmap v11041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:26:05.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:04 smithi164 ceph-mon[36970]: pgmap v11041: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:26:07.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:06 smithi163 ceph-mon[36873]: pgmap v11042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:07.008 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:06 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19796.579833 secs 2024-07-26T12:26:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:06 smithi042 ceph-mon[32775]: pgmap v11042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:07.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:06 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19796.579833 secs 2024-07-26T12:26:07.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:06 smithi164 ceph-mon[36970]: pgmap v11042: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:07.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:06 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19796.579833 secs 2024-07-26T12:26:08.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19814.317647 secs 2024-07-26T12:26:08.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19814.317647 secs 2024-07-26T12:26:08.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19814.317647 secs 2024-07-26T12:26:09.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:08 smithi163 ceph-mon[36873]: pgmap v11043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:09.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:08 smithi042 ceph-mon[32775]: pgmap v11043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:09.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:08 smithi164 ceph-mon[36970]: pgmap v11043: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:10.763 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:10 smithi164 ceph-mon[36970]: pgmap v11044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:11.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:10 smithi042 ceph-mon[32775]: pgmap v11044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:11.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:10 smithi163 ceph-mon[36873]: pgmap v11044: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:12.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:11 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19801.579915 secs 2024-07-26T12:26:12.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:11 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19801.579915 secs 2024-07-26T12:26:12.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:11 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19801.579915 secs 2024-07-26T12:26:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19819.317766 secs 2024-07-26T12:26:13.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:12 smithi042 ceph-mon[32775]: pgmap v11045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19819.317766 secs 2024-07-26T12:26:13.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:12 smithi164 ceph-mon[36970]: pgmap v11045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19819.317766 secs 2024-07-26T12:26:13.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:12 smithi163 ceph-mon[36873]: pgmap v11045: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:14.346+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:26:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:14.346+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:14.346+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:26:15.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:14 smithi042 ceph-mon[32775]: pgmap v11046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:26:15.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:14 smithi164 ceph-mon[36970]: pgmap v11046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:26:15.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:14 smithi163 ceph-mon[36873]: pgmap v11046: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:26:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:16 smithi042 ceph-mon[32775]: pgmap v11047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:17.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:16 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19806.580043 secs 2024-07-26T12:26:17.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:16 smithi164 ceph-mon[36970]: pgmap v11047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:17.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:16 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19806.580043 secs 2024-07-26T12:26:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:16 smithi163 ceph-mon[36873]: pgmap v11047: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:17.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:16 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19806.580043 secs 2024-07-26T12:26:18.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19824.317863 secs 2024-07-26T12:26:18.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19824.317863 secs 2024-07-26T12:26:18.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19824.317863 secs 2024-07-26T12:26:19.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:18 smithi042 ceph-mon[32775]: pgmap v11048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:19.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:18 smithi164 ceph-mon[36970]: pgmap v11048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:19.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:18 smithi163 ceph-mon[36873]: pgmap v11048: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:21.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:20 smithi042 ceph-mon[32775]: pgmap v11049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:21.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:20 smithi164 ceph-mon[36970]: pgmap v11049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:21.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:20 smithi163 ceph-mon[36873]: pgmap v11049: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:22.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:21 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19811.580142 secs 2024-07-26T12:26:22.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:21 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19811.580142 secs 2024-07-26T12:26:22.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:21 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19811.580142 secs 2024-07-26T12:26:23.088 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:22 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19829.317989 secs 2024-07-26T12:26:23.088 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:22 smithi042 ceph-mon[32775]: pgmap v11050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:23.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:22 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19829.317989 secs 2024-07-26T12:26:23.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:22 smithi164 ceph-mon[36970]: pgmap v11050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:23.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:22 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19829.317989 secs 2024-07-26T12:26:23.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:22 smithi163 ceph-mon[36873]: pgmap v11050: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:25.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:24 smithi042 ceph-mon[32775]: pgmap v11051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:26:25.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:24 smithi164 ceph-mon[36970]: pgmap v11051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:26:25.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:24 smithi163 ceph-mon[36873]: pgmap v11051: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:26:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:26 smithi042 ceph-mon[32775]: pgmap v11052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:27.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:26 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19816.580273 secs 2024-07-26T12:26:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:26 smithi164 ceph-mon[36970]: pgmap v11052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:27.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:26 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19816.580273 secs 2024-07-26T12:26:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:26 smithi163 ceph-mon[36873]: pgmap v11052: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:27.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:26 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19816.580273 secs 2024-07-26T12:26:28.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:27 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19834.318056 secs 2024-07-26T12:26:28.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:27 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19834.318056 secs 2024-07-26T12:26:28.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:27 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19834.318056 secs 2024-07-26T12:26:29.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:28 smithi042 ceph-mon[32775]: pgmap v11053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:29.226 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:28 smithi164 ceph-mon[36970]: pgmap v11053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:29.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:28 smithi163 ceph-mon[36873]: pgmap v11053: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:29.353 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:29.352+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:26:29.353 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:29.352+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:29.353 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:29.352+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:26:31.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:30 smithi042 ceph-mon[32775]: pgmap v11054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:31.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:30 smithi164 ceph-mon[36970]: pgmap v11054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:31.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:30 smithi163 ceph-mon[36873]: pgmap v11054: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:32.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:31 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19821.580427 secs 2024-07-26T12:26:32.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:31 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19821.580427 secs 2024-07-26T12:26:32.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:31 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19821.580427 secs 2024-07-26T12:26:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:32 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19839.318172 secs 2024-07-26T12:26:33.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:32 smithi042 ceph-mon[32775]: pgmap v11055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:33.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:32 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19839.318172 secs 2024-07-26T12:26:33.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:32 smithi164 ceph-mon[36970]: pgmap v11055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:33.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:32 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19839.318172 secs 2024-07-26T12:26:33.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:32 smithi163 ceph-mon[36873]: pgmap v11055: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:35.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:34 smithi042 ceph-mon[32775]: pgmap v11056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:26:35.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:34 smithi164 ceph-mon[36970]: pgmap v11056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:26:35.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:34 smithi163 ceph-mon[36873]: pgmap v11056: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-07-26T12:26:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:36 smithi042 ceph-mon[32775]: pgmap v11057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:26:37.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:36 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19826.580518 secs 2024-07-26T12:26:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:36 smithi164 ceph-mon[36970]: pgmap v11057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:26:37.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:36 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19826.580518 secs 2024-07-26T12:26:37.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:36 smithi163 ceph-mon[36873]: pgmap v11057: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:26:37.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:36 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19826.580518 secs 2024-07-26T12:26:38.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:37 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19844.318255 secs 2024-07-26T12:26:38.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:37 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19844.318255 secs 2024-07-26T12:26:38.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:37 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19844.318255 secs 2024-07-26T12:26:39.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:38 smithi042 ceph-mon[32775]: pgmap v11058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:26:39.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:38 smithi164 ceph-mon[36970]: pgmap v11058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:26:39.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:38 smithi163 ceph-mon[36873]: pgmap v11058: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:26:41.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:40 smithi042 ceph-mon[32775]: pgmap v11059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:26:41.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:40 smithi164 ceph-mon[36970]: pgmap v11059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:26:41.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:40 smithi163 ceph-mon[36873]: pgmap v11059: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:26:42.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:41 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19831.580666 secs 2024-07-26T12:26:42.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:41 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19831.580666 secs 2024-07-26T12:26:42.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:41 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19831.580666 secs 2024-07-26T12:26:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:42 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19849.318382 secs 2024-07-26T12:26:43.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:42 smithi042 ceph-mon[32775]: pgmap v11060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:26:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:42 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19849.318382 secs 2024-07-26T12:26:43.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:42 smithi164 ceph-mon[36970]: pgmap v11060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:26:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:42 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19849.318382 secs 2024-07-26T12:26:43.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:42 smithi163 ceph-mon[36873]: pgmap v11060: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:26:44.354 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:44.353+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:26:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:44.353+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:26:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:44.353+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:45.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:44 smithi042 ceph-mon[32775]: pgmap v11061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:26:45.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:44 smithi164 ceph-mon[36970]: pgmap v11061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:26:45.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:44 smithi163 ceph-mon[36873]: pgmap v11061: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 3 op/s 2024-07-26T12:26:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:46 smithi042 ceph-mon[32775]: pgmap v11062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:26:47.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:46 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19836.580795 secs 2024-07-26T12:26:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:46 smithi164 ceph-mon[36970]: pgmap v11062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:26:47.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:46 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19836.580795 secs 2024-07-26T12:26:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:46 smithi163 ceph-mon[36873]: pgmap v11062: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:26:47.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:46 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19836.580795 secs 2024-07-26T12:26:48.007 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:47 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19854.318496 secs 2024-07-26T12:26:48.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:47 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19854.318496 secs 2024-07-26T12:26:48.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:47 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19854.318496 secs 2024-07-26T12:26:49.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:48 smithi042 ceph-mon[32775]: pgmap v11063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:49.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:48 smithi164 ceph-mon[36970]: pgmap v11063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:49.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:48 smithi163 ceph-mon[36873]: pgmap v11063: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:26:50.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:49 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:26:50.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:49 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:26:50.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:49 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:26:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:50 smithi042 ceph-mon[32775]: pgmap v11064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:51.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:26:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:26:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:50 smithi042 ceph-mon[32775]: pgmap v11065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:26:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:50 smithi042 ceph-mon[32775]: pgmap v11066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:26:51.146 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:50 smithi042 ceph-mon[32775]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:26:51.222 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:50 smithi164 ceph-mon[36970]: pgmap v11064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:26:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:26:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:50 smithi164 ceph-mon[36970]: pgmap v11065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:26:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:50 smithi164 ceph-mon[36970]: pgmap v11066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:26:51.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:50 smithi164 ceph-mon[36970]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:26:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:50 smithi163 ceph-mon[36873]: pgmap v11064: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:26:51.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:26:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:26:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:50 smithi163 ceph-mon[36873]: pgmap v11065: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-07-26T12:26:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:50 smithi163 ceph-mon[36873]: pgmap v11066: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:26:51.258 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:50 smithi163 ceph-mon[36873]: from='mgr.14148 172.21.15.42:0/2088340532' entity='mgr.x' 2024-07-26T12:26:52.124 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:51 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19841.580921 secs 2024-07-26T12:26:52.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:51 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19841.580921 secs 2024-07-26T12:26:52.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:51 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19841.580921 secs 2024-07-26T12:26:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:52 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19859.318636 secs 2024-07-26T12:26:53.145 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:52 smithi042 ceph-mon[32775]: pgmap v11067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:26:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:52 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19859.318636 secs 2024-07-26T12:26:53.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:52 smithi164 ceph-mon[36970]: pgmap v11067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:26:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:52 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19859.318636 secs 2024-07-26T12:26:53.257 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:52 smithi163 ceph-mon[36873]: pgmap v11067: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 3 op/s 2024-07-26T12:26:55.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:55 smithi042 ceph-mon[32775]: pgmap v11068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:26:55.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:55 smithi164 ceph-mon[36970]: pgmap v11068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:26:55.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:55 smithi163 ceph-mon[36873]: pgmap v11068: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:26:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19846.581023 secs 2024-07-26T12:26:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:57 smithi042 ceph-mon[32775]: pgmap v11069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:26:57.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:57 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19864.318781 secs 2024-07-26T12:26:57.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19846.581023 secs 2024-07-26T12:26:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:57 smithi164 ceph-mon[36970]: pgmap v11069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:26:57.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:57 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19864.318781 secs 2024-07-26T12:26:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19846.581023 secs 2024-07-26T12:26:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:57 smithi163 ceph-mon[36873]: pgmap v11069: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-07-26T12:26:57.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:57 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19864.318781 secs 2024-07-26T12:26:59.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:59.358+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:26:59.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:59.358+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:59.359 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:26:59.358+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:26:59.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:26:59 smithi042 ceph-mon[32775]: pgmap v11070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:26:59.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:26:59 smithi164 ceph-mon[36970]: pgmap v11070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:26:59.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:26:59 smithi163 ceph-mon[36873]: pgmap v11070: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-07-26T12:27:01.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:27:01 smithi042 ceph-mon[32775]: pgmap v11071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:27:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:27:01 smithi164 ceph-mon[36970]: pgmap v11071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:27:01.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:27:01 smithi163 ceph-mon[36873]: pgmap v11071: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s 2024-07-26T12:27:02.546 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:27:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19851.581106 secs 2024-07-26T12:27:02.547 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:27:02 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19869.318945 secs 2024-07-26T12:27:02.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:27:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19851.581106 secs 2024-07-26T12:27:02.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:27:02 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19869.318945 secs 2024-07-26T12:27:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:27:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19851.581106 secs 2024-07-26T12:27:02.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:27:02 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19869.318945 secs 2024-07-26T12:27:03.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:27:03 smithi042 ceph-mon[32775]: pgmap v11072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:03.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:27:03 smithi164 ceph-mon[36970]: pgmap v11072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:03.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:27:03 smithi163 ceph-mon[36873]: pgmap v11072: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:05.646 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:27:05 smithi042 ceph-mon[32775]: pgmap v11073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:27:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:27:05 smithi164 ceph-mon[36970]: pgmap v11073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:27:05.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:27:05 smithi163 ceph-mon[36873]: pgmap v11073: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:27:07.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:27:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19856.581261 secs 2024-07-26T12:27:07.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:27:07 smithi163 ceph-mon[36873]: pgmap v11074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:07.560 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:27:07 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19874.319049 secs 2024-07-26T12:27:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:27:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19856.581261 secs 2024-07-26T12:27:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:27:07 smithi042 ceph-mon[32775]: pgmap v11074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:07.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:27:07 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19874.319049 secs 2024-07-26T12:27:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:27:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19856.581261 secs 2024-07-26T12:27:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:27:07 smithi164 ceph-mon[36970]: pgmap v11074: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:07.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:27:07 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19874.319049 secs 2024-07-26T12:27:09.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:27:09 smithi042 ceph-mon[32775]: pgmap v11075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:09.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:27:09 smithi164 ceph-mon[36970]: pgmap v11075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:09.754 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:27:09 smithi163 ceph-mon[36873]: pgmap v11075: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:11.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:27:11 smithi042 ceph-mon[32775]: pgmap v11076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:27:11.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:27:11 smithi164 ceph-mon[36970]: pgmap v11076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:27:11.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:27:11 smithi163 ceph-mon[36873]: pgmap v11076: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:27:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:27:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19861.581387 secs 2024-07-26T12:27:12.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:27:12 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19879.319173 secs 2024-07-26T12:27:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:27:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19861.581387 secs 2024-07-26T12:27:12.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:27:12 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19879.319173 secs 2024-07-26T12:27:12.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:27:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19861.581387 secs 2024-07-26T12:27:12.758 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:27:12 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19879.319173 secs 2024-07-26T12:27:13.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:27:13 smithi042 ceph-mon[32775]: pgmap v11077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:13.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:27:13 smithi164 ceph-mon[36970]: pgmap v11077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:13.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:27:13 smithi163 ceph-mon[36873]: pgmap v11077: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:14.360 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:27:14.359+0000 7ff67f7fe640 1 -- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 msgr2=0x55db6e432d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.163:6838/563658453 2024-07-26T12:27:14.361 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:27:14.359+0000 7ff65effd640 1 client.16083 ms_handle_refused on v2:172.21.15.163:6838/563658453 2024-07-26T12:27:14.361 INFO:teuthology.task.background_exec.ubuntu@smithi042.front.sepia.ceph.com.smithi042.stderr:2024-07-26T12:27:14.359+0000 7ff67f7fe640 1 --2- 172.21.15.42:0/1831263443 >> [v2:172.21.15.163:6838/563658453,v1:172.21.15.163:6839/563658453] conn(0x55db6e035d80 0x55db6e432d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:15.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:27:15 smithi042 ceph-mon[32775]: pgmap v11078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:27:15.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:27:15 smithi164 ceph-mon[36970]: pgmap v11078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:27:15.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:27:15 smithi163 ceph-mon[36873]: pgmap v11078: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-07-26T12:27:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:27:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19866.581541 secs 2024-07-26T12:27:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:27:17 smithi042 ceph-mon[32775]: pgmap v11079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:17.645 INFO:journalctl@ceph.mon.a.smithi042.stdout:Jul 26 12:27:17 smithi042 ceph-mon[32775]: 1 slow requests, 0 included below; oldest blocked for > 19884.319277 secs 2024-07-26T12:27:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:27:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19866.581541 secs 2024-07-26T12:27:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:27:17 smithi164 ceph-mon[36970]: pgmap v11079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:17.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Jul 26 12:27:17 smithi164 ceph-mon[36970]: 1 slow requests, 0 included below; oldest blocked for > 19884.319277 secs 2024-07-26T12:27:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:27:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19866.581541 secs 2024-07-26T12:27:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:27:17 smithi163 ceph-mon[36873]: pgmap v11079: 97 pgs: 97 active+clean; 61 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-07-26T12:27:17.757 INFO:journalctl@ceph.mon.b.smithi163.stdout:Jul 26 12:27:17 smithi163 ceph-mon[36873]: 1 slow requests, 0 included below; oldest blocked for > 19884.319277 secs 2024-07-26T12:27:18.789 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-07-26T12:27:18.789 DEBUG:teuthology.task.console_log:Killing console logger for smithi042 2024-07-26T12:27:18.790 DEBUG:teuthology.task.console_log:Killing console logger for smithi163 2024-07-26T12:27:18.790 DEBUG:teuthology.task.console_log:Killing console logger for smithi164 2024-07-26T12:27:18.790 DEBUG:teuthology.exit:Finished running handlers